From eaf62b39230d01780e77dfc8d90ce3976c832ccb Mon Sep 17 00:00:00 2001 From: Matt Cossins Date: Tue, 18 Aug 2026 12:01:21 +0100 Subject: [PATCH 1/6] Add ExecuTorch CMSIS fallback learning path --- .../1-prepare.md | 310 ++++++++++++++++++ .../2-run-cortex-m.md | 119 +++++++ .../3-add-ethosu-fallback.md | 139 ++++++++ .../4-inspect-cmsis-pack.md | 97 ++++++ .../5-prepare-cmsis-runner.md | 251 ++++++++++++++ .../6-consume-cmsis-pack.md | 158 +++++++++ .../_index.md | 83 +++++ .../_next-steps.md | 8 + .../files/export_cortex_m_fallback.py | 202 ++++++++++++ .../files/verify_pack_sources.py | 42 +++ 10 files changed, 1409 insertions(+) create mode 100644 content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/1-prepare.md create mode 100644 content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/2-run-cortex-m.md create mode 100644 content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/3-add-ethosu-fallback.md create mode 100644 content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/4-inspect-cmsis-pack.md create mode 100644 content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/5-prepare-cmsis-runner.md create mode 100644 content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/6-consume-cmsis-pack.md create mode 100644 content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/_index.md create mode 100644 content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/_next-steps.md create mode 100644 content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/export_cortex_m_fallback.py create mode 100644 content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/verify_pack_sources.py diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/1-prepare.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/1-prepare.md new file mode 100644 index 0000000000..32e45aef48 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/1-prepare.md @@ -0,0 +1,310 @@ +--- +title: Prepare the ExecuTorch CMSIS environment +weight: 2 +description: Set up the pinned ExecuTorch 1.4.0, CMSIS-NN, Ethos-U, Arm GNU, FVP, and Docker environment used by this Learning Path. + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Understand what you will build + +Before preparing the environment, it helps to understand how the model will run. +You will use the ExecuTorch ahead-of-time export flow to divide one model across +three execution paths: + +- A stride-one convolution that Ethos-U55 accepts. +- A stride-four convolution that Ethos-U55 rejects but CMSIS-NN supports. +- A `topk` operation that remains an ExecuTorch portable FP32 operation. + +After Ethos-U partitioning and Cortex-M lowering, the graph follows this path: + +```text +input + -> Ethos-U55 delegated convolution + -> Cortex-M quantized operator using CMSIS-NN + -> portable FP32 topk + -> output +``` + +The three execution paths and the ExecuTorch CMSIS Pack have different roles: + +| Technology | Role in this Learning Path | Expected performance | +| --- | --- | --- | +| Ethos-U | Runs supported graph regions on the NPU. | Expected to provide the best performance and energy efficiency for supported neural-network operations. | +| CMSIS-NN | Supplies optimized Cortex-M kernels for supported quantized CPU operations that remain outside the NPU delegate. | Preferred CPU fallback when a matching CMSIS-NN-backed operator is available. | +| ExecuTorch portable kernels | Supply general CPU implementations for operations without an accelerated backend implementation. | Compatibility fallback rather than the preferred performance path. | +| ExecuTorch CMSIS Pack | Supplies versioned, selectable runtime, backend, and operator source components. | Packages the implementation; it does not provide acceleration by itself. | + +The intended order is Ethos-U first, CMSIS-NN second, and portable kernels +last. This is an expected hierarchy rather than a guaranteed benchmark result. +Operator shapes, graph boundaries, memory movement, and quantize and dequantize +operations can affect end-to-end performance. + +You will install the published source-based `PyTorch::ExecuTorch` CMSIS Pack, +select the matching backend and operator components in a CMSIS-Toolbox project, +and run the resulting executable on a Corstone-300 Fixed Virtual Platform +(FVP). The FVP validates functional execution, but it does not provide final +latency, memory, or power measurements for a physical device. + +The following setup creates the pinned export, build, and simulation +environment used for that workflow. + +## Check the host requirements + +This Learning Path uses an ExecuTorch CMSIS environment pinned to ExecuTorch +`1.4.0` and Python `3.12`. The published ExecuTorch CMSIS Pack and the Python +exporter must use the same ExecuTorch version. + +Use an arm64 Linux computer with at least `30 GB` of free storage. The host runs +the exporter and FVP, while Docker provides the CMSIS build environment. Install +[Docker](/install-guides/docker/) before continuing. + +Ubuntu 24.04 LTS on arm64 is a good host option because it provides Python +`3.12` packages and supports Docker Engine on `arm64`. Other arm64 Linux +distributions can also work if they provide the required versions and tools. + +Install the required host packages: + +```bash +sudo apt update +sudo apt install -y \ + build-essential \ + pkg-config \ + libzstd-dev \ + python3.12 \ + python3.12-dev \ + python3.12-venv \ + git \ + curl \ + unzip +``` + +Check the host architecture, Python version, Docker server, and available disk +space: + +```bash +uname -m +python3.12 --version +docker version --format '{{.Server.Arch}} {{.Server.Version}}' +df -h "$HOME" +``` + +Confirm the following results before continuing: + +- `uname -m` reports `aarch64`. +- Python reports version `3.12`. +- The Docker server architecture is `arm64`. +- The home filesystem has at least `30 GB` available. + +## Clone ExecuTorch 1.4.0 + +The checkout directory must be named `executorch`. Define the paths used by the +remaining setup stages, then clone the release tag: + +```bash +export ET_ROOT=$HOME/executorch-1.4.0/executorch +export LP_WORKSPACE=$HOME/executorch-cmsis-lp +export RUNNER_DIR=$LP_WORKSPACE/pack-runner +export LP_ASSETS=https://raw.githubusercontent.com/ArmDeveloperEcosystem/arm-learning-paths/main/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files + +mkdir -p "$(dirname "$ET_ROOT")" "$LP_WORKSPACE/artifacts" + +git clone \ + --branch v1.4.0 \ + --depth 1 \ + --recurse-submodules \ + --shallow-submodules \ + https://github.com/pytorch/executorch.git \ + "$ET_ROOT" + +git -C "$ET_ROOT" describe --tags --exact-match +``` + +The expected output is: + +```output +v1.4.0 +``` + +## Build the Python export environment + +Create a Python `3.12` virtual environment inside the checkout: + +```bash +cd "$ET_ROOT" +python3.12 -m venv .venv +source .venv/bin/activate +python -m pip install --upgrade pip setuptools-scm +``` + +The exporter needs the Python backend modules and FlatBuffer schema tool. It +does not need a complete host runtime build, so the following options disable +unrelated backends, extensions, kernels, and tests. + +```bash +export CMAKE_ARGS="\ +-DEXECUTORCH_BUILD_COREML=OFF \ +-DEXECUTORCH_BUILD_CUDA=OFF \ +-DEXECUTORCH_BUILD_DEVTOOLS=OFF \ +-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=OFF \ +-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=OFF \ +-DEXECUTORCH_BUILD_EXTENSION_LLM=OFF \ +-DEXECUTORCH_BUILD_EXTENSION_LLM_RUNNER=OFF \ +-DEXECUTORCH_BUILD_EXTENSION_MODULE=OFF \ +-DEXECUTORCH_BUILD_EXTENSION_NAMED_DATA_MAP=OFF \ +-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=OFF \ +-DEXECUTORCH_BUILD_EXTENSION_TENSOR=OFF \ +-DEXECUTORCH_BUILD_EXTENSION_TRAINING=OFF \ +-DEXECUTORCH_BUILD_KERNELS_CUSTOM_AOT=OFF \ +-DEXECUTORCH_BUILD_KERNELS_LLM=OFF \ +-DEXECUTORCH_BUILD_KERNELS_LLM_AOT=OFF \ +-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=OFF \ +-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=OFF \ +-DEXECUTORCH_BUILD_KERNELS_QUANTIZED_AOT=OFF \ +-DEXECUTORCH_BUILD_MLX=OFF \ +-DEXECUTORCH_BUILD_OPENVINO=OFF \ +-DEXECUTORCH_BUILD_PORTABLE_OPS=OFF \ +-DEXECUTORCH_BUILD_PYBIND=OFF \ +-DEXECUTORCH_BUILD_QNN=OFF \ +-DEXECUTORCH_BUILD_TESTS=OFF \ +-DEXECUTORCH_BUILD_VULKAN=OFF \ +-DEXECUTORCH_BUILD_XNNPACK=OFF \ +-DEXECUTORCH_BUILD_CMSIS_NN_PYBINDS=OFF" +export CMAKE_BUILD_ARGS="--target flatbuffers_ep" + +CMAKE_BUILD_PARALLEL_LEVEL=2 \ + ./install_executorch.sh --optional-dependency ethos_u + +unset CMAKE_ARGS CMAKE_BUILD_ARGS +``` + +Install the CMSIS-NN Python binding at the revision pinned by ExecuTorch +`1.4.0`: + +```bash +python -m pip install \ + "scikit-build-core<0.8" \ + pybind11 \ + pyproject-metadata \ + pathspec + +python -m pip install --no-build-isolation \ + "git+https://github.com/ARM-software/CMSIS-NN.git@dbf45dbfcc515421dd6099037d3e2637b90748c8" +``` + +Confirm that the exporter dependencies can be imported: + +```bash +python - <<'PY' +import cmsis_nn +import executorch.backends.arm +import executorch.backends.cortex_m +from executorch.exir import to_edge + +print("ExecuTorch 1.4.0 CMSIS Python imports passed") +PY +``` + +The expected output is: + +```output +ExecuTorch 1.4.0 CMSIS Python imports passed +``` + +## Install the Arm tools + +The ExecuTorch Arm setup installs the Arm GNU toolchain, Ethos-U software, and +Corstone-300 FVP. Review the Arm license terms before using the acceptance +option: + +```bash +cd "$ET_ROOT" +source .venv/bin/activate +examples/arm/setup.sh --i-agree-to-the-contained-eula +source examples/arm/arm-scratch/setup_path.sh +``` + +Confirm that the compiler and FVP are available: + +```bash +arm-none-eabi-gcc --version | head -n 1 +command -v FVP_Corstone_SSE-300_Ethos-U55 +``` + +## Fetch the Corstone platform sources + +The ExecuTorch CMSIS Pack supplies the runtime, backends, and operators. The +runner also needs the Corstone target, UART, mailbox, timing, and startup +sources from the Ethos-U SDK. These are board and simulator support files, not +ExecuTorch runtime components. + +Fetch the `26.02` Ethos-U release used by ExecuTorch `1.4.0`, then apply the +platform patches supplied in the ExecuTorch checkout. `fetch_externals.py` +retrieves the SDK repositories listed by that release, while `patch_repo` +applies the compatibility changes expected by the ExecuTorch example: + +```bash +cd "$ET_ROOT" +source .venv/bin/activate + +ETHOS_SDK="$ET_ROOT/examples/arm/arm-scratch/ethos-u" +PATCH_DIR="$ET_ROOT/examples/arm/ethos-u-setup" + +if [ ! -d "$ETHOS_SDK/.git" ]; then + git clone \ + --branch 26.02 \ + --depth 1 \ + https://git.gitlab.arm.com/artificial-intelligence/ethos-u/ethos-u.git \ + "$ETHOS_SDK" +fi + +source backends/arm/scripts/utils.sh +patch_repo "$ETHOS_SDK" 26.02 "$PATCH_DIR" + +cd "$ETHOS_SDK" +python fetch_externals.py -c 26.02.json fetch + +cd "$ET_ROOT" +patch_repo "$ETHOS_SDK/core_software" 26.02 "$PATCH_DIR" +patch_repo "$ETHOS_SDK/core_platform" 26.02 "$PATCH_DIR" + +test -f "$ETHOS_SDK/core_platform/targets/corstone-300/target.cpp" +test -f "$ETHOS_SDK/core_software/Cortex_DFP/Device/ARMCM55/Source/startup_ARMCM55.c" +echo "Ethos-U Corstone platform sources prepared" +``` + +The expected final output is: + +```output +Ethos-U Corstone platform sources prepared +``` + +This stage downloads platform support only. The embedded application still +compiles the ExecuTorch runtime, Ethos-U backend, and selected operators from +the published ExecuTorch CMSIS Pack. + +## Save the workspace settings + +Save the paths so each later page can restore them in a new shell: + +```bash +cat > "$LP_WORKSPACE/env.sh" <<'EOF' +export ET_ROOT="$HOME/executorch-1.4.0/executorch" +export LP_WORKSPACE="$HOME/executorch-cmsis-lp" +export RUNNER_DIR="$HOME/executorch-cmsis-lp/pack-runner" +export LP_ASSETS="https://raw.githubusercontent.com/ArmDeveloperEcosystem/arm-learning-paths/main/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files" +EOF + +source "$LP_WORKSPACE/env.sh" +cd "$LP_WORKSPACE" +curl -O "$LP_ASSETS/export_cortex_m_fallback.py" +``` + +## What you've accomplished and what's next + +You have prepared the pinned ExecuTorch CMSIS environment and a separate model +export workspace. + +Next, you will inspect how CMSIS-NN and portable kernels divide Cortex-M work +before adding the Ethos-U delegate. diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/2-run-cortex-m.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/2-run-cortex-m.md new file mode 100644 index 0000000000..0656a2b8f3 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/2-run-cortex-m.md @@ -0,0 +1,119 @@ +--- +title: Lower a model to the Cortex-M backend +weight: 3 +description: Export a Cortex-M baseline and identify CMSIS-NN accelerated convolutions and a portable FP32 operation. + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Use a simple convolution example + +With the ExecuTorch CMSIS environment and export workspace ready, start by +creating the simple model introduced at the start of the previous page. We will then lower to Cortex-M, without any Ethos-U backend. This use-case is for a device that does not contain an NPU. + +The model contains two convolutions followed by `topk`: + +```python +self.conv1 = torch.nn.Conv2d(3, 4, kernel_size=3, stride=1, padding=1, bias=False) +self.conv2 = torch.nn.Conv2d(4, 1, kernel_size=3, stride=4, padding=1, bias=False) + +values, _ = torch.topk( + self.conv2(torch.relu(self.conv1(inputs))), + k=1, + dim=-1, +) +``` + +## See how the exporter lowers the graph + +The exporter first prepares and converts the model with `CortexMQuantizer`: + +```python +quantizer = CortexMQuantizer() + +prepared = prepare_pt2e(exported_program.module(), quantizer) +prepared(*example_inputs) +quantized_model = convert_pt2e(prepared) +quantized_exported_program = torch.export.export(quantized_model, example_inputs) +``` + +The prepared model is run once with the example input to calibrate activation +ranges. `convert_pt2e()` uses those ranges to choose quantization parameters and +insert the operations that move values between FP32 and INT8. + +The exporter converts the quantized program to the ExecuTorch Edge dialect and +applies the Cortex-M pass manager: + +```python +edge_program_manager = to_edge( + quantized_exported_program, + compile_config=EdgeCompileConfig(_check_ir_validity=False), +) + +edge_program_manager._edge_programs["forward"] = CortexMPassManager( + edge_program_manager.exported_program() +).transform() +``` + +This step rewrites supported patterns as `cortex_m.*` operators. Unsupported +operations, including `aten.topk`, remain in the portable graph. + +The Cortex-M backend is broader than CMSIS-NN. In this model, the quantized +convolutions use CMSIS-NN, while quantize and dequantize use Cortex-M operators +with Arm Helium vector implementations. A `cortex_m.*` target therefore means +that the Cortex-M backend owns the operator, not necessarily that CMSIS-NN +implements it. + +{{% notice Important %}} +The Cortex-M backend is not a delegate. Delegation replaces a graph region with +a call to a separately compiled backend. Cortex-M lowering instead rewrites +supported patterns as registered `cortex_m.*` operators that the ExecuTorch +runtime dispatches normally. You will not see an `executorch_call_delegate` +node until the Ethos-U partitioner is added on the next page. +{{% /notice %}} + +## Export the model for Cortex-M + +Run the exporter from the separate workspace: + +```bash +source ~/executorch-cmsis-lp/env.sh +source "$ET_ROOT/.venv/bin/activate" +cd "$LP_WORKSPACE" + +python export_cortex_m_fallback.py \ + --mode cortex-m \ + --output-dir artifacts +``` + +The relevant output is similar to: + +```output +Relevant targets before Cortex-M lowering: +aten.convolution.default: 2 +aten.topk.default: 1 + +Relevant targets after Cortex-M lowering: +aten.topk.default: 1 +cortex_m.dequantize_per_tensor.default: 2 +cortex_m.quantize_per_tensor.default: 2 +cortex_m.quantized_conv2d.default: 2 +``` + +Both convolutions become `cortex_m.quantized_conv2d` operations. The +`aten.topk` operation remains portable FP32. Quantize and dequantize operations +mark where tensors move between FP32 and INT8 regions. These optimized Cortex-M +boundary operations are part of the executable graph and can matter when +evaluating the cost of a fallback path. + +## What you've accomplished and what's next + +You have seen how Cortex-M lowering replaces supported patterns with optimized +Cortex-M operators. The convolutions use CMSIS-NN, quantize and dequantize use +Helium vector implementations, and `topk` remains an ExecuTorch portable +kernel. + +Next, you will see how CMSIS-NN provides optimized CPU fallback when deploying a +model to an Ethos-U NPU. The resulting graph combines Ethos-U delegation, +CMSIS-NN-backed Cortex-M operators, and a portable CPU kernel. diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/3-add-ethosu-fallback.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/3-add-ethosu-fallback.md new file mode 100644 index 0000000000..cb0be3adbc --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/3-add-ethosu-fallback.md @@ -0,0 +1,139 @@ +--- +title: CMSIS-NN provides optimized fallback for Ethos-U NPUs +weight: 4 +description: Partition the model for Ethos-U55, then lower the remaining supported convolution to CMSIS-NN. + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Deploying across Cortex-M and Ethos-U NPU + +We can re-use the same script as on the previous page, this time using 'Fallback mode'. This uses `EthosUPartitioner` to +create an Ethos-U55 delegate region, represented by `executorch_call_delegate`. +The exporter then applies `CortexMPassManager` to operations left outside that +delegate, lowering its supported convolution to a registered Cortex-M operator +implemented by CMSIS-NN. + +This is planned fallback, not recovery from a runtime NPU error. During export, +the partitioner checks operator support and assigns graph regions to Ethos-U. +Known unsupported operations remain in the graph for another implementation. +Here, the stride-four convolution is deliberately used to demonstrate that +path. + +{{% notice Why this example uses Ethos-U55 %}} +Ethos-U55 is used because its unsupported stride-four convolution creates a +predictable fallback region. Ethos-U85 supports more operator configurations, +so will typically have less CPU fallback than U55. The same workflow still +applies when a model contains operations that remain outside the U85 delegate. +{{% /notice %}} + +For supported neural-network operations, the expected order is Ethos-U NPU, +then CMSIS-NN on the Cortex-M CPU, then a portable CPU kernel. This keeps as much +work as possible on the NPU and gives unsupported quantized work an optimized +CPU route before using a general implementation. Measure the complete graph on +the target device because tensor conversions and graph boundaries can change +the end-to-end result. + +The exporter first creates an Ethos-U55 compile specification and configures +the target-specific quantizer: + +```python +compile_spec = EthosUCompileSpec(target=args.target) +quantizer = EthosUQuantizer(compile_spec) +quantizer.set_global( + get_symmetric_quantization_config(is_per_channel=True) +) +``` + +After quantization, `EthosUPartitioner` identifies supported regions and lowers +them into an Ethos-U delegate: + +```python +edge_program_manager = to_edge_transform_and_lower( + quantized_exported_program, + partitioner=[EthosUPartitioner(compile_spec)], + compile_config=EdgeCompileConfig(_check_ir_validity=False), +) +``` + +The exporter then applies Cortex-M lowering to the operations that remain +outside the delegate: + +```python +edge_program_manager._edge_programs["forward"] = CortexMPassManager( + edge_program_manager.exported_program() +).transform() +``` + +The order is important: Ethos-U partitioning runs first, so +`CortexMPassManager` only transforms the non-delegated remainder. + +## Export the partially delegated program + +Run the exporter in Ethos-U fallback mode: + +```bash +source ~/executorch-cmsis-lp/env.sh +source "$ET_ROOT/.venv/bin/activate" +cd "$LP_WORKSPACE" + +python export_cortex_m_fallback.py \ + --mode ethos-u-fallback \ + --target ethos-u55-128 \ + --output-dir artifacts +``` + +The relevant output is similar to: + +```output +Relevant targets before Cortex-M lowering: +aten.convolution.default: 1 +aten.topk.default: 1 +executorch_call_delegate: 1 + +Relevant targets after Cortex-M lowering: +aten.topk.default: 1 +cortex_m.dequantize_per_tensor.default: 2 +cortex_m.quantize_per_tensor.default: 2 +cortex_m.quantized_conv2d.default: 1 +executorch_call_delegate: 1 +``` + +The final graph contains all three execution paths: + +- `executorch_call_delegate` contains the stride-one convolution accepted by + Ethos-U55. +- `cortex_m.quantized_conv2d` is the stride-four convolution rejected by + Ethos-U55 and lowered to CMSIS-NN. +- `aten.topk` remains a portable FP32 operation. + +The graph contains two quantize and two dequantize operations because values +cross between delegated INT8 work, Cortex-M INT8 work, and portable FP32 work. +Keeping these conversions visible helps you judge the whole fallback region, +rather than considering only the convolution kernel. + +The lowered targets determine which components the CMSIS project needs: + +| Model operation | Lowered graph target | CMSIS project component | Execution engine | +| --- | --- | --- | --- | +| Stride-one convolution | `executorch_call_delegate` | `Backend EthosU` and `Ethos-U Driver Generic U55` | Ethos-U55 | +| Stride-four convolution | `cortex_m.quantized_conv2d` | `Cortex-M quantized_conv2d` and `CMSIS:NN Lib` | Cortex-M55 with CMSIS-NN | +| Quantized boundaries | `cortex_m.quantize_per_tensor` and `cortex_m.dequantize_per_tensor` | Matching Cortex-M operator components | Cortex-M55 | +| Output selection | `aten.topk` | `Portable topk` | Cortex-M55 portable FP32 kernel | + +The CMSIS project also needs the ExecuTorch runtime, kernel utilities, and +operator registration components. You will inspect these components when you +open the published pack. + +## What you've accomplished and what's next + +You have exported a program that distributes work across Ethos-U55, +CMSIS-NN, and a portable FP32 kernel. The unsupported convolution does not need +to fall back to a general FP32 implementation: it remains quantized and uses a +CMSIS-NN-backed Cortex-M operator. This is designed to reduce the CPU cost of +known unsupported NPU work, although performance must still be measured on the +target device, including quantize and dequantize boundaries. + +Next, you will download and inspect the published ExecuTorch CMSIS Pack that +supplies these runtime, backend, and operator components. diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/4-inspect-cmsis-pack.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/4-inspect-cmsis-pack.md new file mode 100644 index 0000000000..04bbd752cd --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/4-inspect-cmsis-pack.md @@ -0,0 +1,97 @@ +--- +title: Inspect the published ExecuTorch CMSIS Pack +weight: 5 +description: Download the published ExecuTorch 1.4.0 CMSIS Pack and inspect the runtime, backend, and operator components used by the model. + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Understand the ExecuTorch CMSIS Pack benefits + +The ExecuTorch CMSIS Pack is a source pack rather than a prebuilt runtime +library. It supplies the ExecuTorch runtime, registration code, backends, and +operators as selectable CMSIS components. + +A `.pack` file is a ZIP-compatible archive. Its Pack Description file, or +PDSC, records component names, source files, dependencies, versions, and +conditions. CMSIS-Toolbox reads this metadata to turn component selections in a +project into a concrete build. + +A CMSIS project can select only the components required by its lowered graph +instead of maintaining one large source list. The selected source files then +compile with the application's target compiler and options. + +The published pack also gives CMSIS-Toolbox and Keil Studio a versioned package +identifier: `PyTorch::ExecuTorch@1.4.0`. This allows the tools to download the +same package, resolve its dependencies, expose its components in an IDE, and +record the selected version without a local pack-building stage. + +## Download the published pack + +Download the pack attached to the ExecuTorch `1.4.0` release: + +```bash +source ~/executorch-cmsis-lp/env.sh + +PACK_DIR="$LP_WORKSPACE/packs" +PACK_FILE="$PACK_DIR/PyTorch.ExecuTorch.1.4.0.pack" + +mkdir -p "$PACK_DIR" + +curl --location --fail \ + --output "$PACK_FILE" \ + https://github.com/pytorch/executorch/releases/download/v1.4.0/PyTorch.ExecuTorch.1.4.0.pack + +unzip -t "$PACK_FILE" >/dev/null && \ + echo "ExecuTorch CMSIS Pack archive passed" +``` + +The expected final output is: + +```output +ExecuTorch CMSIS Pack archive passed +``` + +This copy is used only to inspect the archive and its component metadata. The +final build installs the same pack through the public pack index into a clean +pack root. That separate installation checks the normal CMSIS dependency and +component-resolution workflow. + +## Inspect the components used by the model + +Display the PDSC components that correspond to the partially delegated graph: + +```bash +unzip -p "$PACK_FILE" PyTorch.ExecuTorch.pdsc | \ +grep ' uart_config.h <<'EOF' +#pragma once + +#define UART0_BASE (0x49303000) +#define UART0_BAUDRATE (115200) +#define SYSTEM_CORE_CLOCK (25000000) +EOF +``` + +## Define the CMSIS solution and project + +Create `pack_runner.csolution.yml`. This file selects the compiler, target, +build type, project, and pack versions: + +```bash +cat > pack_runner.csolution.yml <<'YAML' +solution: + created-for: CMSIS-Toolbox@2.13.0 + cdefault: + compiler: GCC@14.3.1 + + packs: + - pack: ARM::CMSIS@>=6.0.0 + - pack: ARM::CMSIS-NN@7.0.0 + - pack: ARM::Cortex_DFP@>=1.1.0 + - pack: ARM::ethos-u-core-driver@1.26.2 + - pack: PyTorch::ExecuTorch@1.4.0 + + target-types: + - type: Corstone-300 + device: ARM::ARMCM55 + + build-types: + - type: Release + optimize: speed + + projects: + - project: ./pack_runner.cproject.yml +YAML +``` + +Create `pack_runner.cproject.yml`. The `components` section selects the +ExecuTorch, CMSIS-NN, Ethos-U, Cortex-M, and portable components used by the +lowered model. These selections come directly from the target names observed on +the previous pages. The remaining sections configure and build the runner +around those components: + +```bash +cat > pack_runner.cproject.yml <<'YAML' +project: + packs: + - pack: PyTorch::ExecuTorch@1.4.0 + + output: + base-name: pack_runner + type: + - elf + - map + + linker: + - script: ./platform.ld + for-compiler: GCC + + components: + - component: ARM::CMSIS:CORE + - component: ARM::CMSIS:NN Lib + - component: ARM::Machine Learning:NPU Support:Ethos-U Driver&Generic U55 + - component: Machine Learning:ExecuTorch:Runtime + - component: Machine Learning:ExecuTorch:Kernel Utils + - component: Machine Learning:ExecuTorch:Kernel Registration + - component: Machine Learning:ExecuTorch:Backend EthosU + - component: Machine Learning:ExecuTorch Operators:Cortex-M quantize_per_tensor + - component: Machine Learning:ExecuTorch Operators:Cortex-M dequantize_per_tensor + - component: Machine Learning:ExecuTorch Operators:Cortex-M quantized_conv2d + - component: Machine Learning:ExecuTorch Operators:Portable topk + + add-path: + - . + - ./executorch/examples/arm/executor_runner + - ./executorch/examples/arm/arm-scratch/ethos-u/core_platform/targets/common/include + - ./executorch/examples/arm/arm-scratch/ethos-u/core_platform/drivers/mpu/include + - ./executorch/examples/arm/arm-scratch/ethos-u/core_platform/drivers/timing_adapter/include + - ./executorch/examples/arm/arm-scratch/ethos-u/core_platform/drivers/uart/include + - ./executorch/examples/arm/arm-scratch/ethos-u/core_platform/drivers/mailbox/include + - ./executorch/examples/arm/arm-scratch/ethos-u/core_platform/drivers/mhu_dummy/include + - ./executorch/examples/arm/arm-scratch/ethos-u/core_software/Cortex_DFP/Device/ARMCM55/Include + - ./executorch/examples/arm/arm-scratch/ethos-u/core_software/cmsis_6/CMSIS/Core/Include + + define: + - ETHOSU + - ETHOSU55 + - ETHOSU_HAS_TA + - ET_COMPILED_PTE + - ET_LOG_DUMP_OUTPUT + + misc: + - for-compiler: GCC + C-CPP: + - -includeARMCM55.h + - -DARM_NN_ENABLE_F16=0 + - -DARM_NN_ENABLE_F32=0 + - -DCMSIS_VER=6 + - -DETHOSU_ARENA=1 + - -DETHOSU_MACS=128 + - -DETHOSU_MODEL=1 + - -DET_ARM_BAREMETAL_SCRATCH_TEMP_ALLOCATOR_POOL_SIZE=0x200000 + - -DET_NUM_INFERENCES=1 + C: + - -std=gnu11 + CPP: + - -std=gnu++17 + - -fno-unwind-tables + - -fno-rtti + - -fno-exceptions + Link: + - -Wl,-u,_printf_float + + groups: + - group: Application + files: + - file: ./executorch/examples/arm/executor_runner/arm_executor_runner.cpp + - file: ./executorch/examples/arm/executor_runner/arm_memory_allocator.cpp + - file: ./executorch/examples/arm/executor_runner/arm_perf_monitor.cpp + - file: ./executorch/extension/runner_util/inputs.cpp + - file: ./executorch/extension/runner_util/inputs_portable.cpp + - group: Corstone-300 + files: + - file: ./executorch/examples/arm/arm-scratch/ethos-u/core_platform/targets/common/src/init.cpp + - file: ./executorch/examples/arm/arm-scratch/ethos-u/core_platform/targets/corstone-300/target.cpp + - file: ./executorch/examples/arm/arm-scratch/ethos-u/core_platform/targets/corstone-300/retarget.c + - file: ./executorch/examples/arm/arm-scratch/ethos-u/core_platform/drivers/mpu/src/mpu.cpp + - file: ./executorch/examples/arm/arm-scratch/ethos-u/core_platform/drivers/timing_adapter/src/timing_adapter.c + - file: ./executorch/examples/arm/arm-scratch/ethos-u/core_platform/drivers/uart/src/uart_cmsdk_apb.c + - file: ./executorch/examples/arm/arm-scratch/ethos-u/core_platform/drivers/mailbox/src/mailbox.cpp + - file: ./executorch/examples/arm/arm-scratch/ethos-u/core_platform/drivers/mhu_dummy/src/mhu_dummy.cpp + - file: ./executorch/examples/arm/arm-scratch/ethos-u/core_software/Cortex_DFP/Device/ARMCM55/Source/system_ARMCM55.c + - file: ./executorch/examples/arm/arm-scratch/ethos-u/core_software/Cortex_DFP/Device/ARMCM55/Source/startup_ARMCM55.c +YAML +``` + +### How the CMSIS project file is structured + +The CMSIS project file combines components installed from packs with application +and platform sources from the ExecuTorch checkout: + +| YAML section | Purpose | +| --- | --- | +| `packs` | Pins the project to `PyTorch::ExecuTorch@1.4.0`. | +| `output` and `linker` | Select the ELF and map outputs and the generated Corstone-300 linker script. | +| `components` | Select CMSIS Core, CMSIS-NN, the Ethos-U55 driver, ExecuTorch runtime support, and only the operators required by the lowered graph. | +| `add-path` | Adds headers for the runner and Corstone-300 platform sources mounted from the ExecuTorch checkout. | +| `define` and `misc` | Configure the Ethos-U55 target, embedded `.pte` data, logging, memory allocation, inference count, and compiler options. | +| `groups` | Adds the runner and Corstone-300 source files that form the application around the pack components. | + +The `components` section connects the lowered graph to the embedded build. It +selects the required CPU and NPU implementations from the pinned ExecuTorch +`1.4.0` pack instead of maintaining a source list manually. The entries under +`groups` remain application and platform code; they are not supplied by the +pack. + +## Embed the exported program + +Convert the partially delegated `.pte` file to a C array. The linker places this +array in the Corstone-300 DDR model section: + +```bash +cd "$RUNNER_DIR" +source "$ET_ROOT/.venv/bin/activate" + +python "$ET_ROOT/examples/arm/executor_runner/pte_to_header.py" \ + --pte "$LP_WORKSPACE/artifacts/stride4_ethos_u_fallback.pte" \ + --outdir "$RUNNER_DIR" +``` + +The expected output includes: + +```output +Section: network_model_sec. +``` + +The `.pte` file is the serialized ExecuTorch program produced by the exporter. +Converting it to `model_pte.h` makes the program part of the firmware image, so +the bare-metal runner does not need a filesystem or external model loader. + +## Generate the Corstone-300 linker script + +Preprocess the ExecuTorch linker template with the Arm GNU compiler: + +```bash +source "$ET_ROOT/examples/arm/arm-scratch/setup_path.sh" + +arm-none-eabi-gcc -E -x c -P \ + -o "$RUNNER_DIR/platform.ld" \ + "$ET_ROOT/examples/arm/executor_runner/Corstone-300.ld" + +ls -lh "$RUNNER_DIR/model_pte.h" "$RUNNER_DIR/platform.ld" +``` + +`model_pte.h` embeds the model. `platform.ld` places code, the model, the method +allocator, and the CMSIS-NN scratch arena in memory regions available on the +Corstone-300 FVP. Preprocessing is required because the linker template uses C +preprocessor directives to produce the final GNU linker script. + +The CMSIS project mounts the v1.4.0 checkout read-only to compile the existing +runner and Corstone platform sources. ExecuTorch runtime, backend, registration, +and operator sources come from the installed ExecuTorch CMSIS Pack. + +## What you've accomplished and what's next + +You have prepared a Corstone-300 CMSIS project containing the exported model +and the platform files needed to execute it. The project selects the versioned +ExecuTorch, CMSIS-NN, Ethos-U, and portable components that match the graph. + +Next, you will build the application from the selected pack components, verify +their source paths, and run the model on the FVP. diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/6-consume-cmsis-pack.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/6-consume-cmsis-pack.md new file mode 100644 index 0000000000..f3bba87d70 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/6-consume-cmsis-pack.md @@ -0,0 +1,158 @@ +--- +title: Build and run the ExecuTorch CMSIS Pack application +weight: 7 +description: Build a Corstone-300 executable from ExecuTorch, CMSIS-NN, and Ethos-U CMSIS components, then run the model on an Ethos-U55 FVP. + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Build the executable from ExecuTorch CMSIS Pack components + +Build the application in the arm64 AVH-MLOps container. The command: + +- Mounts the ExecuTorch checkout as a read-only input. +- Activates the CMSIS-Toolbox utilities supplied by the container. +- Installs `PyTorch::ExecuTorch@1.4.0` from the pack index into a clean pack root. +- Runs `cbuild` for the Corstone-300 release context. + +The temporary `CMSIS_PACK_ROOT` is intentionally empty at the start. A +successful build therefore shows that the project can resolve its declared +packs, rather than relying on packages left by an earlier build. + +```bash +source ~/executorch-cmsis-lp/env.sh +cd "$RUNNER_DIR" + +DOCKER_IMAGE="ghcr.io/arm-software/avh-mlops/arm-mlops-docker-licensed-community:latest-arm64" + +chmod a+rwx "$RUNNER_DIR" + +docker run --rm \ + -v "$RUNNER_DIR:/workspace" \ + -v "$ET_ROOT:/workspace/executorch:ro" \ + "$DOCKER_IMAGE" \ + bash -lc ' +set -euo pipefail +cd /workspace + +export Z_VCPKG_POSTSCRIPT="$(mktemp /tmp/vcpkg.XXXXXX.sh)" +vcpkg activate +source "$Z_VCPKG_POSTSCRIPT" + +export CMSIS_PACK_ROOT=/tmp/cmsis-pack-root +cpackget init https://www.keil.com/pack/index.pidx +cpackget add --agree-embedded-license PyTorch::ExecuTorch@1.4.0 + +cbuild pack_runner.csolution.yml \ + --packs \ + --update-rte \ + --context pack_runner.Release+Corstone-300 +' +``` + +In the command, `cpackget init` configures the public pack index and +`cpackget add` installs the selected ExecuTorch pack. `cbuild --packs` resolves +declared pack dependencies, `--update-rte` regenerates the run-time environment +files, and `--context` selects the project, build type, and target combination +defined by the solution. + +{{% notice Note %}} +The generated `pack_runner.csolution.yml` and `pack_runner.cproject.yml` files +can also be opened with the Arm CMSIS Solution extension included in the +[Arm Keil Studio Pack](https://marketplace.visualstudio.com/items?itemName=Arm.keil-studio-pack) +for Visual Studio Code. Use the graphical view to inspect the selected target, +build context, packs, and software components. +{{% /notice %}} + +The first build downloads the arm64 AVH-MLOps container, CMSIS-Toolbox compiler +tools, the ExecuTorch CMSIS Pack, and its dependent packs. The final build +summary is: + +```output +Build summary: 1 succeeded, 0 failed +``` + +{{% notice Note %}} +GNU `ld` can print allocation warnings for the Corstone multi-segment linker +script before the successful build summary. The generated ELF is validated in +the next section on the FVP. +{{% /notice %}} + +The executable is written to: + +```text +$RUNNER_DIR/out/pack_runner/Corstone-300/Release/pack_runner.elf +``` + +## Confirm that the build used the pack + +Run the verification helper against the generated compilation database. It +checks that the runtime, CMSIS-NN-backed Cortex-M operator, Ethos-U backend, and +portable `topk` sources came from the installed pack: + +```bash +COMPILE_DB=out/pack_runner/Corstone-300/Release/compile_commands.json +python verify_pack_sources.py "$COMPILE_DB" +``` + +The expected output is: + +```output +PASS: ExecuTorch runtime came from the ExecuTorch CMSIS Pack +PASS: CMSIS-NN Cortex-M convolution came from the ExecuTorch CMSIS Pack +PASS: Ethos-U backend came from the ExecuTorch CMSIS Pack +PASS: Portable topk came from the ExecuTorch CMSIS Pack +``` + +Each `PASS` confirms that one required ExecuTorch source file appears in the +compilation database under the installed pack path. The application +and Corstone platform sources come from the ExecuTorch checkout. A +successful build confirms that an ELF was produced; this additional check +confirms that the selected ExecuTorch components were resolved from the pack. + +## Run the pack-built model on the FVP + +Run the `cbuild` output on the Corstone-300 Ethos-U55 FVP: + +```bash +source "$ET_ROOT/examples/arm/arm-scratch/setup_path.sh" + +"$ET_ROOT/backends/arm/scripts/run_fvp.sh" \ + --elf="$RUNNER_DIR/out/pack_runner/Corstone-300/Release/pack_runner.elf" \ + --target=ethos-u55-128 +``` + +The relevant output is: + +```output +PTE Model data loaded. Size: 5476 bytes. +NPU delegations: 1 (1.00 per inference) +Output[0][0]: (float) 0.030337 +Output[0][1]: (float) -0.034964 +Program complete, exiting. +No problems found! +``` + +The graph summary from the export step identifies the stride-four convolution +as `cortex_m.quantized_conv2d` and `topk` as portable. The compilation database +shows that the matching Cortex-M and portable sources came from the ExecuTorch +CMSIS Pack. The FVP output confirms that the complete pack-built application +loaded and ran the same partially delegated program with one Ethos-U delegation. + +## What you've accomplished + +You have run an ExecuTorch model built from the ExecuTorch CMSIS Pack on a +Corstone-300 system. The application delegates the supported convolution to +Ethos-U55, executes the unsupported stride-four convolution through the +CMSIS-NN-backed Cortex-M operator, and finishes with a portable FP32 operator. + +CMSIS-NN provides the optimized CPU implementation for the quantized fallback +convolution. The ExecuTorch CMSIS Pack provides versioned, component-based +integration for the matching runtime, backend, and operator sources. The clean +CMSIS build and compilation-database checks make that software composition +visible and repeatable. + +For another model, inspect its lowered graph, select the matching components +from the ExecuTorch CMSIS Pack, regenerate `model_pte.h`, and rebuild the CMSIS +project. diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/_index.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/_index.md new file mode 100644 index 0000000000..cd0fd9ca78 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/_index.md @@ -0,0 +1,83 @@ +--- +title: Deploy ML models across Cortex-M and Ethos-U with CMSIS-NN optimized kernels and the ExecuTorch CMSIS Pack + +description: Build and run an ExecuTorch 1.4.0 application from the published ExecuTorch CMSIS Pack using Ethos-U, CMSIS-NN, and portable Cortex-M kernels. + +minutes_to_complete: 60 + +who_is_this_for: This is an introductory topic for embedded machine learning developers building ExecuTorch applications for Cortex-M CPUs and Ethos-U NPUs. + +learning_objectives: + - Export an ExecuTorch graph with Ethos-U, CMSIS-NN, and portable-kernel execution regions + - Lower quantized CPU fallback operations to optimized CMSIS-NN-backed Cortex-M operators, and understand the benefits in doing so + - Install and consume the published ExecuTorch CMSIS Pack for version 1.4.0 with selected runtime and operator components + - Run an application built from the ExecuTorch CMSIS Pack on a Corstone-300 Ethos-U55 FVP + - Understand when and why to use the ExecuTorch CMSIS Pack + +prerequisites: + - An arm64 Linux computer with Docker and at least 30 GB of free storage; Ubuntu 24.04 LTS is a recommended option + - Python 3.12 with support for virtual environments + - Familiarity with PyTorch export and post-training quantization + - Familiarity with CMake and command-line development tools + - Awareness of Ethos-U NPU, and Cortex-M CPU + +author: +- Matt Cossins + +generate_summary_faq: false +rerun_summary: false +rerun_faqs: false +test_maintenance: false + +### Tags +skilllevels: Introductory +subjects: ML +armips: + - Cortex-M + - Ethos-U +operatingsystems: + - Linux +tools_software_languages: + - Python + - PyTorch + - ExecuTorch + - CMake + - CMSIS + - CMSIS Pack + - CMSIS-NN + - Docker + - FVP + - GCC + +further_reading: + - resource: + title: ExecuTorch 1.4.0 release + link: https://github.com/pytorch/executorch/releases/tag/v1.4.0 + type: documentation + - resource: + title: Published ExecuTorch CMSIS Pack + link: https://www.keil.arm.com/packs/executorch-pytorch/overview/ + type: documentation + - resource: + title: ExecuTorch Arm Cortex-M backend + link: https://docs.pytorch.org/executorch/stable/backends/arm-cortex-m/arm-cortex-m-overview.html + type: documentation + - resource: + title: ExecuTorch Ethos-U with CMSIS-NN fallback example + link: https://github.com/pytorch/executorch/blob/v1.4.0/examples/arm/ethos_u_cmsis_nn_fallback_example.ipynb + type: documentation + - resource: + title: CMSIS-Toolbox install guide + link: /install-guides/cmsis-toolbox/ + type: install_guide + - resource: + title: CMSIS-NN documentation + link: https://arm-software.github.io/CMSIS-NN/latest/ + type: documentation + +### FIXED, DO NOT MODIFY +# ================================================================================ +weight: 1 +layout: "learningpathall" +learning_path_main_page: "yes" +--- diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/_next-steps.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/_next-steps.md new file mode 100644 index 0000000000..e20dfa6d43 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/_next-steps.md @@ -0,0 +1,8 @@ +--- +# ================================================================================ +# FIXED, DO NOT MODIFY THIS FILE +# ================================================================================ +weight: 21 +title: "Next Steps" +layout: "learningpathall" +--- diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/export_cortex_m_fallback.py b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/export_cortex_m_fallback.py new file mode 100644 index 0000000000..2898bdb77d --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/export_cortex_m_fallback.py @@ -0,0 +1,202 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import json +from pathlib import Path +from typing import Any + + +class ThreePathModel: + def __new__(cls) -> Any: + import torch + + class Model(torch.nn.Module): + def __init__(self) -> None: + super().__init__() + self.conv1 = torch.nn.Conv2d( + in_channels=3, + out_channels=4, + kernel_size=3, + stride=1, + padding=1, + bias=False, + ) + self.conv2 = torch.nn.Conv2d( + in_channels=4, + out_channels=1, + kernel_size=3, + stride=4, + padding=1, + bias=False, + ) + + def forward(self, inputs: torch.Tensor) -> torch.Tensor: + values, _ = torch.topk( + self.conv2(torch.relu(self.conv1(inputs))), + k=1, + dim=-1, + ) + return values + + return Model() + + +def main() -> None: + parser = argparse.ArgumentParser( + description=( + "Export a Cortex-M baseline or a partially delegated Ethos-U55, " + "CMSIS-NN, and portable-kernel model." + ) + ) + parser.add_argument( + "--mode", + choices=("cortex-m", "ethos-u-fallback"), + required=True, + help="Select Cortex-M-only lowering or Ethos-U55 followed by Cortex-M lowering.", + ) + parser.add_argument( + "--target", + default="ethos-u55-128", + help="Ethos-U target used by fallback mode and the Corstone FVP runner.", + ) + parser.add_argument( + "--output-dir", + type=Path, + default=Path("artifacts"), + help="Directory for the bundled program and graph summary.", + ) + args = parser.parse_args() + + import torch + from executorch.backends.cortex_m.passes.cortex_m_pass_manager import ( + CortexMPassManager, + ) + from executorch.exir import ( + EdgeCompileConfig, + ExecutorchBackendConfig, + to_edge, + to_edge_transform_and_lower, + ) + from executorch.extension.export_util.utils import save_pte_program + from torchao.quantization.pt2e.quantize_pt2e import convert_pt2e, prepare_pt2e + + torch.manual_seed(0) + model = ThreePathModel().eval().to(memory_format=torch.channels_last) + example_inputs = ( + torch.ones(1, 3, 8, 8, dtype=torch.float32).to( + memory_format=torch.channels_last + ), + ) + exported_program = torch.export.export(model, example_inputs) + + if args.mode == "cortex-m": + from executorch.backends.cortex_m.quantizer.quantizer import CortexMQuantizer + + quantizer = CortexMQuantizer() + else: + from executorch.backends.arm.ethosu import EthosUCompileSpec + from executorch.backends.arm.quantizer import ( + EthosUQuantizer, + get_symmetric_quantization_config, + ) + + compile_spec = EthosUCompileSpec(target=args.target) + quantizer = EthosUQuantizer(compile_spec) + quantizer.set_global(get_symmetric_quantization_config(is_per_channel=True)) + + prepared = prepare_pt2e(exported_program.module(), quantizer) + prepared(*example_inputs) + quantized_model = convert_pt2e(prepared) + quantized_exported_program = torch.export.export(quantized_model, example_inputs) + reference_output = quantized_exported_program.module()(*example_inputs) + + if args.mode == "cortex-m": + edge_program_manager = to_edge( + quantized_exported_program, + compile_config=EdgeCompileConfig(_check_ir_validity=False), + ) + else: + from executorch.backends.arm.ethosu import EthosUPartitioner + + edge_program_manager = to_edge_transform_and_lower( + quantized_exported_program, + partitioner=[EthosUPartitioner(compile_spec)], + compile_config=EdgeCompileConfig(_check_ir_validity=False), + ) + + targets_before_cortex_m = count_targets( + edge_program_manager.exported_program().graph_module.graph + ) + + edge_program_manager._edge_programs["forward"] = CortexMPassManager( + edge_program_manager.exported_program() + ).transform() + + targets_after_cortex_m = count_targets( + edge_program_manager.exported_program().graph_module.graph + ) + executorch_program = edge_program_manager.to_executorch( + config=ExecutorchBackendConfig(extract_delegate_segments=False) + ) + + args.output_dir.mkdir(parents=True, exist_ok=True) + artifact_name = args.mode.replace("-", "_") + artifact_path = args.output_dir / f"stride4_{artifact_name}.pte" + save_pte_program(executorch_program, str(artifact_path)) + + summary = { + "mode": args.mode, + "target": args.target, + "artifact": str(artifact_path), + "artifact_size_bytes": artifact_path.stat().st_size, + "reference_output": reference_output.detach().flatten().tolist(), + "targets_before_cortex_m": targets_before_cortex_m, + "targets_after_cortex_m": targets_after_cortex_m, + } + summary_path = artifact_path.with_suffix(".json") + summary_path.write_text(json.dumps(summary, indent=2) + "\n") + + print(f"Wrote {artifact_path}") + print(f"Wrote {summary_path}") + print("\nRelevant targets before Cortex-M lowering:") + print_relevant_targets(targets_before_cortex_m) + print("\nRelevant targets after Cortex-M lowering:") + print_relevant_targets(targets_after_cortex_m) + + +def count_targets(graph: Any) -> dict[str, int]: + counts: dict[str, int] = {} + for node in graph.nodes: + if node.op != "call_function": + continue + target = display_target(str(node.target)) + if target.startswith("<") and hasattr(node.target, "__name__"): + target = node.target.__name__ + counts[target] = counts.get(target, 0) + 1 + return dict(sorted(counts.items())) + + +def print_relevant_targets(targets: dict[str, int]) -> None: + found = False + for target, count in targets.items(): + if ( + "cortex_m" in target + or "delegate" in target + or "convolution" in target + or "topk" in target + ): + print(f"{display_target(target)}: {count}") + found = True + if not found: + print("No delegate or Cortex-M targets found.") + + +def display_target(target: str) -> str: + if target.startswith("", 1)[0] + return target + + +if __name__ == "__main__": + main() diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/verify_pack_sources.py b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/verify_pack_sources.py new file mode 100644 index 0000000000..07dbd2f791 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/verify_pack_sources.py @@ -0,0 +1,42 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import json +from pathlib import Path + + +EXPECTED_SOURCES = { + "ExecuTorch runtime": "src/runtime/executor/program.cpp", + "CMSIS-NN Cortex-M convolution": "src/backends/cortex_m/ops/op_quantized_conv2d.cpp", + "Ethos-U backend": "src/backends/arm/runtime/EthosUBackend.cpp", + "Portable topk": "src/kernels/portable/cpu/op_topk.cpp", +} + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("compile_database", type=Path) + args = parser.parse_args() + + entries = json.loads(args.compile_database.read_text()) + source_files = { + entry["file"].replace("\\", "/") + for entry in entries + if "file" in entry + } + + missing = [] + for label, source_path in EXPECTED_SOURCES.items(): + pack_source = f"/PyTorch/ExecuTorch/1.4.0/{source_path}" + if any(source_file.endswith(pack_source) for source_file in source_files): + print(f"PASS: {label} came from the ExecuTorch CMSIS Pack") + else: + missing.append(label) + + if missing: + raise SystemExit(f"Missing pack sources: {', '.join(missing)}") + + +if __name__ == "__main__": + main() From 590361dd14c7bb78ae01758c7611bb4f969e7bd2 Mon Sep 17 00:00:00 2001 From: Kwashie Andoh Date: Fri, 16 Oct 2026 16:54:51 +0100 Subject: [PATCH 2/6] CMSIS LP review draft 1 complete --- .../1-prepare.md | 381 ++++++------------ .../2-run-cortex-m.md | 146 ++++--- .../3-add-ethosu-fallback.md | 226 ++++++----- .../4-inspect-cmsis-pack.md | 168 ++++---- .../5-prepare-cmsis-runner.md | 302 ++++---------- .../6-consume-cmsis-pack.md | 191 ++++----- .../_index.md | 44 +- .../files/enable_cortex_m_fallback.patch | 70 ++++ .../files/export_cortex_m_fallback.py | 72 ++-- .../files/mixed_backend_cnn.py | 58 +++ .../files/verify_pack_sources.py | 57 ++- 11 files changed, 790 insertions(+), 925 deletions(-) create mode 100644 content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/enable_cortex_m_fallback.patch create mode 100644 content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/mixed_backend_cnn.py diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/1-prepare.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/1-prepare.md index 32e45aef48..ff4b88cf32 100644 --- a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/1-prepare.md +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/1-prepare.md @@ -1,16 +1,20 @@ --- -title: Prepare the ExecuTorch CMSIS environment +title: Prepare the Corstone-300 U55 project weight: 2 -description: Set up the pinned ExecuTorch 1.4.0, CMSIS-NN, Ethos-U, Arm GNU, FVP, and Docker environment used by this Learning Path. +description: Set up the reusable CMSIS-Executorch project and its pinned Python and Arm tool environments. ### FIXED, DO NOT MODIFY layout: learningpathall --- -## Understand what you will build +## Understand the execution plan -Before preparing the environment, it helps to understand how the model will run. -You will use the ExecuTorch ahead-of-time export flow to divide one model across +You will start with a small PyTorch model and export it as an ExecuTorch +program (`.pte`). + +On the target, the Cortex-M CPU runs the ExecuTorch runtime and coordinates +the whole program. The CPU reads the `.pte` and dispatches different parts +to either the Ethos-U NPU or to CPU kernels. You will use the ExecuTorch ahead-of-time export flow to divide one model across three execution paths: - A stride-one convolution that Ethos-U55 accepts. @@ -24,287 +28,158 @@ input -> Ethos-U55 delegated convolution -> Cortex-M quantized operator using CMSIS-NN -> portable FP32 topk - -> output +output ``` -The three execution paths and the ExecuTorch CMSIS Pack have different roles: +The three execution paths have different roles: | Technology | Role in this Learning Path | Expected performance | | --- | --- | --- | -| Ethos-U | Runs supported graph regions on the NPU. | Expected to provide the best performance and energy efficiency for supported neural-network operations. | -| CMSIS-NN | Supplies optimized Cortex-M kernels for supported quantized CPU operations that remain outside the NPU delegate. | Preferred CPU fallback when a matching CMSIS-NN-backed operator is available. | -| ExecuTorch portable kernels | Supply general CPU implementations for operations without an accelerated backend implementation. | Compatibility fallback rather than the preferred performance path. | -| ExecuTorch CMSIS Pack | Supplies versioned, selectable runtime, backend, and operator source components. | Packages the implementation; it does not provide acceleration by itself. | - -The intended order is Ethos-U first, CMSIS-NN second, and portable kernels -last. This is an expected hierarchy rather than a guaranteed benchmark result. -Operator shapes, graph boundaries, memory movement, and quantize and dequantize -operations can affect end-to-end performance. - -You will install the published source-based `PyTorch::ExecuTorch` CMSIS Pack, -select the matching backend and operator components in a CMSIS-Toolbox project, -and run the resulting executable on a Corstone-300 Fixed Virtual Platform -(FVP). The FVP validates functional execution, but it does not provide final -latency, memory, or power measurements for a physical device. - -The following setup creates the pinned export, build, and simulation -environment used for that workflow. - -## Check the host requirements - -This Learning Path uses an ExecuTorch CMSIS environment pinned to ExecuTorch -`1.4.0` and Python `3.12`. The published ExecuTorch CMSIS Pack and the Python -exporter must use the same ExecuTorch version. - -Use an arm64 Linux computer with at least `30 GB` of free storage. The host runs -the exporter and FVP, while Docker provides the CMSIS build environment. Install -[Docker](/install-guides/docker/) before continuing. - -Ubuntu 24.04 LTS on arm64 is a good host option because it provides Python -`3.12` packages and supports Docker Engine on `arm64`. Other arm64 Linux -distributions can also work if they provide the required versions and tools. - -Install the required host packages: - -```bash -sudo apt update -sudo apt install -y \ - build-essential \ - pkg-config \ - libzstd-dev \ - python3.12 \ - python3.12-dev \ - python3.12-venv \ - git \ - curl \ - unzip -``` - -Check the host architecture, Python version, Docker server, and available disk -space: - -```bash -uname -m -python3.12 --version -docker version --format '{{.Server.Arch}} {{.Server.Version}}' -df -h "$HOME" -``` +| Ethos-U | Runs supported graph regions on the NPU. (eg: Stride-1 convolution) | Expected to provide the best performance and energy efficiency for supported neural-network operations. | +| [CMSIS-NN](https://arm-software.github.io/CMSIS-NN/latest/index.html) | Supplies optimized Cortex-M kernels for quantized CPU operations that are unsupported by the NPU. (eg: Stride-4 convolution) | Preferred CPU fallback when a matching CMSIS-NN-backed operator is available. | +| ExecuTorch portable kernels | Supply general CPU implementations for operations without an accelerated backend implementation. (eg: `topk` operation) | Compatibility fallback rather than the preferred performance path. | -Confirm the following results before continuing: +Fallback here is planned during export. It isn't recovery from an NPU failure +at runtime. The exporter determines which regions the target Ethos-U delegate +supports, then assigns known remaining patterns to the Cortex-M backend. -- `uname -m` reports `aarch64`. -- Python reports version `3.12`. -- The Docker server architecture is `arm64`. -- The home filesystem has at least `30 GB` available. +{{% notice Why this example uses Ethos-U55 %}} +Ethos-U55 is used because its unsupported stride-four convolution creates a +predictable fallback region. Ethos-U85 supports more operator configurations, +so it will typically have less CPU fallback than U55. The same workflow still +applies when a model contains operations that remain outside the U85 delegate. +{{% /notice %}} -## Clone ExecuTorch 1.4.0 +In this Learning Path, you will clone a CMSIS-Executorch example that already contains the needed board support, +runner, memory configuration, and Fixed Virtual Platform (FVP) settings. You +will then inspect and modify that to observe Cortex-M fallback using optimized CMSIS-NN microkernels. -The checkout directory must be named `executorch`. Define the paths used by the -remaining setup stages, then clone the release tag: +## Clone the example project -```bash -export ET_ROOT=$HOME/executorch-1.4.0/executorch -export LP_WORKSPACE=$HOME/executorch-cmsis-lp -export RUNNER_DIR=$LP_WORKSPACE/pack-runner -export LP_ASSETS=https://raw.githubusercontent.com/ArmDeveloperEcosystem/arm-learning-paths/main/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files +Clone the CMSIS-Executorch project used by this Learning Path: -mkdir -p "$(dirname "$ET_ROOT")" "$LP_WORKSPACE/artifacts" - -git clone \ - --branch v1.4.0 \ - --depth 1 \ - --recurse-submodules \ - --shallow-submodules \ - https://github.com/pytorch/executorch.git \ - "$ET_ROOT" - -git -C "$ET_ROOT" describe --tags --exact-match +```console +git clone --branch add-corstone-300-u55 https://github.com/Kwash45/CMSIS-Executorch.git +cd CMSIS-Executorch +git checkout 173c251496446b8f564c178c415e6f7f28081070 ``` -The expected output is: +The explicit checkout pins the project to a revision that has been tested and validated. -```output -v1.4.0 -``` +## Install vcpkg and Activate the project tools -## Build the Python export environment +The repository's `vcpkg-configuration.json` declares CMSIS-Toolbox, Arm +Compiler for Embedded, and the Arm FVP. Install `vcpkg` with the command for your host: -Create a Python `3.12` virtual environment inside the checkout: +{{< tabpane code=true >}} + {{< tab header="Linux" language="bash">}} +. <(curl https://github.com/microsoft/vcpkg-tool/releases/download/2026-04-08/vcpkg-init.sh -L) + {{< /tab >}} + {{< tab header="Windows PowerShell" language="powershell">}} +iex (iwr -useb https://github.com/microsoft/vcpkg-tool/releases/download/2026-04-08/vcpkg-init.ps1) + {{< /tab >}} +{{< /tabpane >}} -```bash -cd "$ET_ROOT" -python3.12 -m venv .venv -source .venv/bin/activate -python -m pip install --upgrade pip setuptools-scm -``` +Initialize `vcpkg`, then activate it from the repository root. Repeat these commands in each new terminal you open. -The exporter needs the Python backend modules and FlatBuffer schema tool. It -does not need a complete host runtime build, so the following options disable -unrelated backends, extensions, kernels, and tests. - -```bash -export CMAKE_ARGS="\ --DEXECUTORCH_BUILD_COREML=OFF \ --DEXECUTORCH_BUILD_CUDA=OFF \ --DEXECUTORCH_BUILD_DEVTOOLS=OFF \ --DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=OFF \ --DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=OFF \ --DEXECUTORCH_BUILD_EXTENSION_LLM=OFF \ --DEXECUTORCH_BUILD_EXTENSION_LLM_RUNNER=OFF \ --DEXECUTORCH_BUILD_EXTENSION_MODULE=OFF \ --DEXECUTORCH_BUILD_EXTENSION_NAMED_DATA_MAP=OFF \ --DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=OFF \ --DEXECUTORCH_BUILD_EXTENSION_TENSOR=OFF \ --DEXECUTORCH_BUILD_EXTENSION_TRAINING=OFF \ --DEXECUTORCH_BUILD_KERNELS_CUSTOM_AOT=OFF \ --DEXECUTORCH_BUILD_KERNELS_LLM=OFF \ --DEXECUTORCH_BUILD_KERNELS_LLM_AOT=OFF \ --DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=OFF \ --DEXECUTORCH_BUILD_KERNELS_QUANTIZED=OFF \ --DEXECUTORCH_BUILD_KERNELS_QUANTIZED_AOT=OFF \ --DEXECUTORCH_BUILD_MLX=OFF \ --DEXECUTORCH_BUILD_OPENVINO=OFF \ --DEXECUTORCH_BUILD_PORTABLE_OPS=OFF \ --DEXECUTORCH_BUILD_PYBIND=OFF \ --DEXECUTORCH_BUILD_QNN=OFF \ --DEXECUTORCH_BUILD_TESTS=OFF \ --DEXECUTORCH_BUILD_VULKAN=OFF \ --DEXECUTORCH_BUILD_XNNPACK=OFF \ --DEXECUTORCH_BUILD_CMSIS_NN_PYBINDS=OFF" -export CMAKE_BUILD_ARGS="--target flatbuffers_ep" - -CMAKE_BUILD_PARALLEL_LEVEL=2 \ - ./install_executorch.sh --optional-dependency ethos_u - -unset CMAKE_ARGS CMAKE_BUILD_ARGS -``` - -Install the CMSIS-NN Python binding at the revision pinned by ExecuTorch -`1.4.0`: - -```bash -python -m pip install \ - "scikit-build-core<0.8" \ - pybind11 \ - pyproject-metadata \ - pathspec - -python -m pip install --no-build-isolation \ - "git+https://github.com/ARM-software/CMSIS-NN.git@dbf45dbfcc515421dd6099037d3e2637b90748c8" -``` +{{< tabpane code=true >}} + {{< tab header="Linux" language="bash">}} +. ~/.vcpkg/vcpkg-init +vcpkg-shell activate + {{< /tab >}} + {{< tab header="Windows PowerShell" language="powershell">}} +. "$env:USERPROFILE\.vcpkg\vcpkg-init.ps1" +vcpkg-shell activate + {{< /tab >}} +{{< /tabpane >}} -Confirm that the exporter dependencies can be imported: +The first activation downloads any missing tools. Later activations reuse the +installed versions. -```bash -python - <<'PY' -import cmsis_nn -import executorch.backends.arm -import executorch.backends.cortex_m -from executorch.exir import to_edge +If you don't already have a suitable Arm Compiler for Embedded license, +activate the free, non-commercial Keil MDK Community license: -print("ExecuTorch 1.4.0 CMSIS Python imports passed") -PY +```console +armlm activate -product KEMDK-COM0 -server https://mdk-preview.keil.arm.com ``` -The expected output is: +## Create the Python environment + +Use the repository's setup wrapper to create `.venv` and install its pinned +export dependencies: + +{{< tabpane code=true >}} + {{< tab header="Linux" language="bash">}} +./setup_venv.sh + {{< /tab >}} + {{< tab header="Windows PowerShell" language="powershell">}} +.\setup_venv.bat + {{< /tab >}} +{{< /tabpane >}} + +## Verify the environment + +The Python package and CMSIS Pack versions must match because the embedded +runtime loads the program produced by the exporter. Check the host tools, Arm +build tools, and Python packages: + +{{< tabpane code=true >}} + {{< tab header="Linux" language="bash">}} +git --version +python3 --version +cbuild --version +armclang --version +.venv/bin/python -c "from importlib.metadata import version; print('ExecuTorch', version('executorch')); print('PyTorch', version('torch'))" + {{< /tab >}} + {{< tab header="Windows PowerShell" language="powershell">}} +git --version +python --version +cbuild --version +armclang --version +.\.venv\Scripts\python.exe -c "from importlib.metadata import version; print('ExecuTorch', version('executorch')); print('PyTorch', version('torch'))" + {{< /tab >}} +{{< /tabpane >}} + +Your Git version can vary. Confirm that Python is between 3.10 and 3.13, +`cbuild` reports 2.14.0, Arm Compiler for Embedded reports 6.24, and the final +two lines are: ```output -ExecuTorch 1.4.0 CMSIS Python imports passed +ExecuTorch 1.5.1 +PyTorch 2.14.0 ``` -## Install the Arm tools +## Download the Learning Path assets -The ExecuTorch Arm setup installs the Arm GNU toolchain, Ethos-U software, and -Corstone-300 FVP. Review the Arm license terms before using the acceptance -option: +Download the fallback model, focused integration patch, and two inspection +helpers into the repository root: -```bash -cd "$ET_ROOT" -source .venv/bin/activate -examples/arm/setup.sh --i-agree-to-the-contained-eula -source examples/arm/arm-scratch/setup_path.sh -``` - -Confirm that the compiler and FVP are available: - -```bash -arm-none-eabi-gcc --version | head -n 1 -command -v FVP_Corstone_SSE-300_Ethos-U55 -``` +{{< tabpane code=true >}} + {{< tab header="Linux" language="bash">}} +LP_ASSETS=https://raw.githubusercontent.com/ArmDeveloperEcosystem/arm-learning-paths/main/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files -## Fetch the Corstone platform sources +curl -fLO "$LP_ASSETS/mixed_backend_cnn.py" +curl -fLO "$LP_ASSETS/enable_cortex_m_fallback.patch" +curl -fLO "$LP_ASSETS/export_cortex_m_fallback.py" +curl -fLO "$LP_ASSETS/verify_pack_sources.py" + {{< /tab >}} + {{< tab header="Windows PowerShell" language="powershell">}} +$lpAssets = "https://raw.githubusercontent.com/ArmDeveloperEcosystem/arm-learning-paths/main/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files" -The ExecuTorch CMSIS Pack supplies the runtime, backends, and operators. The -runner also needs the Corstone target, UART, mailbox, timing, and startup -sources from the Ethos-U SDK. These are board and simulator support files, not -ExecuTorch runtime components. +curl.exe -fLO "$lpAssets/mixed_backend_cnn.py" +curl.exe -fLO "$lpAssets/enable_cortex_m_fallback.patch" +curl.exe -fLO "$lpAssets/export_cortex_m_fallback.py" +curl.exe -fLO "$lpAssets/verify_pack_sources.py" + {{< /tab >}} +{{< /tabpane >}} -Fetch the `26.02` Ethos-U release used by ExecuTorch `1.4.0`, then apply the -platform patches supplied in the ExecuTorch checkout. `fetch_externals.py` -retrieves the SDK repositories listed by that release, while `patch_repo` -applies the compatibility changes expected by the ExecuTorch example: - -```bash -cd "$ET_ROOT" -source .venv/bin/activate - -ETHOS_SDK="$ET_ROOT/examples/arm/arm-scratch/ethos-u" -PATCH_DIR="$ET_ROOT/examples/arm/ethos-u-setup" - -if [ ! -d "$ETHOS_SDK/.git" ]; then - git clone \ - --branch 26.02 \ - --depth 1 \ - https://git.gitlab.arm.com/artificial-intelligence/ethos-u/ethos-u.git \ - "$ETHOS_SDK" -fi - -source backends/arm/scripts/utils.sh -patch_repo "$ETHOS_SDK" 26.02 "$PATCH_DIR" - -cd "$ETHOS_SDK" -python fetch_externals.py -c 26.02.json fetch - -cd "$ET_ROOT" -patch_repo "$ETHOS_SDK/core_software" 26.02 "$PATCH_DIR" -patch_repo "$ETHOS_SDK/core_platform" 26.02 "$PATCH_DIR" - -test -f "$ETHOS_SDK/core_platform/targets/corstone-300/target.cpp" -test -f "$ETHOS_SDK/core_software/Cortex_DFP/Device/ARMCM55/Source/startup_ARMCM55.c" -echo "Ethos-U Corstone platform sources prepared" -``` - -The expected final output is: - -```output -Ethos-U Corstone platform sources prepared -``` - -This stage downloads platform support only. The embedded application still -compiles the ExecuTorch runtime, Ethos-U backend, and selected operators from -the published ExecuTorch CMSIS Pack. - -## Save the workspace settings - -Save the paths so each later page can restore them in a new shell: - -```bash -cat > "$LP_WORKSPACE/env.sh" <<'EOF' -export ET_ROOT="$HOME/executorch-1.4.0/executorch" -export LP_WORKSPACE="$HOME/executorch-cmsis-lp" -export RUNNER_DIR="$HOME/executorch-cmsis-lp/pack-runner" -export LP_ASSETS="https://raw.githubusercontent.com/ArmDeveloperEcosystem/arm-learning-paths/main/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files" -EOF - -source "$LP_WORKSPACE/env.sh" -cd "$LP_WORKSPACE" -curl -O "$LP_ASSETS/export_cortex_m_fallback.py" -``` +Each file will be explained in greater detail in subsequent pages. ## What you've accomplished and what's next -You have prepared the pinned ExecuTorch CMSIS environment and a separate model -export workspace. +You now have a pinned Corstone-300 U55 project, its matching Python packages, +and its versioned Arm tools. The project already owns the board and application +integration that previously needed many manual setup steps. -Next, you will inspect how CMSIS-NN and portable kernels divide Cortex-M work -before adding the Ethos-U delegate. +Next, you will establish a CPU-only baseline before adding +Ethos-U delegation on the U55 demonstration target. diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/2-run-cortex-m.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/2-run-cortex-m.md index 0656a2b8f3..17f5c0c3dc 100644 --- a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/2-run-cortex-m.md +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/2-run-cortex-m.md @@ -1,7 +1,7 @@ --- -title: Lower a model to the Cortex-M backend +title: Establish the Cortex-M baseline weight: 3 -description: Export a Cortex-M baseline and identify CMSIS-NN accelerated convolutions and a portable FP32 operation. +description: Adapt TinyCNN and inspect how its convolutions and topk operation run without Ethos-U delegation. ### FIXED, DO NOT MODIFY layout: learningpathall @@ -9,85 +9,69 @@ layout: learningpathall ## Use a simple convolution example -With the ExecuTorch CMSIS environment and export workspace ready, start by -creating the simple model introduced at the start of the previous page. We will then lower to Cortex-M, without any Ethos-U backend. This use-case is for a device that does not contain an NPU. +The CMSIS-Executorch repository starts with a TinyCNN model that can be delegated completely to +Ethos-U. To make fallback visible, you will retain its input and output +interface while changing the operations happening inside it. -The model contains two convolutions followed by `topk`: +Replace the original model: -```python -self.conv1 = torch.nn.Conv2d(3, 4, kernel_size=3, stride=1, padding=1, bias=False) -self.conv2 = torch.nn.Conv2d(4, 1, kernel_size=3, stride=4, padding=1, bias=False) - -values, _ = torch.topk( - self.conv2(torch.relu(self.conv1(inputs))), - k=1, - dim=-1, -) -``` - -## See how the exporter lowers the graph - -The exporter first prepares and converts the model with `CortexMQuantizer`: - -```python -quantizer = CortexMQuantizer() - -prepared = prepare_pt2e(exported_program.module(), quantizer) -prepared(*example_inputs) -quantized_model = convert_pt2e(prepared) -quantized_exported_program = torch.export.export(quantized_model, example_inputs) -``` +{{< tabpane code=true >}} + {{< tab header="Linux" language="bash">}} +cp mixed_backend_cnn.py model/model.py + {{< /tab >}} + {{< tab header="Windows PowerShell" language="powershell">}} +Copy-Item .\mixed_backend_cnn.py .\model\model.py + {{< /tab >}} +{{< /tabpane >}} -The prepared model is run once with the example input to calibrate activation -ranges. `convert_pt2e()` uses those ranges to choose quantization parameters and -insert the operations that move values between FP32 and INT8. +The model still accepts one `16 x 16` RGB image in NHWC order and returns ten +values. Keeping that interface means the supplied bare-metal runner needs no +changes. -The exporter converts the quantized program to the ExecuTorch Edge dialect and -applies the Cortex-M pass manager: +The important part of `model/model.py` now looks like this: ```python -edge_program_manager = to_edge( - quantized_exported_program, - compile_config=EdgeCompileConfig(_check_ir_validity=False), +self.conv1 = nn.Conv2d( + channels, 4, kernel_size=3, stride=1, padding=1, bias=False +) +self.conv2 = nn.Conv2d( + 4, num_classes, kernel_size=3, stride=4, padding=1, bias=False ) -edge_program_manager._edge_programs["forward"] = CortexMPassManager( - edge_program_manager.exported_program() -).transform() +values = torch.relu(self.conv1(values)) +values = self.conv2(values) +values = torch.flatten(values, start_dim=2) +values, _ = torch.topk(values, k=1, dim=-1) ``` -This step rewrites supported patterns as `cortex_m.*` operators. Unsupported -operations, including `aten.topk`, remain in the portable graph. - -The Cortex-M backend is broader than CMSIS-NN. In this model, the quantized -convolutions use CMSIS-NN, while quantize and dequantize use Cortex-M operators -with Arm Helium vector implementations. A `cortex_m.*` target therefore means -that the Cortex-M backend owns the operator, not necessarily that CMSIS-NN -implements it. - -{{% notice Important %}} -The Cortex-M backend is not a delegate. Delegation replaces a graph region with -a call to a separately compiled backend. Cortex-M lowering instead rewrites -supported patterns as registered `cortex_m.*` operators that the ExecuTorch -runtime dispatches normally. You will not see an `executorch_call_delegate` -node until the Ethos-U partitioner is added on the next page. -{{% /notice %}} - -## Export the model for Cortex-M +The *stride* controls how far a convolution filter moves between positions. A +stride of one visits adjacent positions. A stride of four moves four positions +at a time and produces a smaller output. For this U55 target and tensor shape, +the first convolution can be delegated to the NPU, while the stride-four convolution +remains outside the delegate. `topk` selects the largest spatial response for +each of the ten output channels. -Run the exporter from the separate workspace: +## Export the Cortex-M baseline -```bash -source ~/executorch-cmsis-lp/env.sh -source "$ET_ROOT/.venv/bin/activate" -cd "$LP_WORKSPACE" +Before we demonstrate fallback, it is useful to know which operations could be optimized by the Cortex-M backend. +Therefore, as a useful first step, we will specify `--mode cortex-m`, such that no operation is offered to the NPU. Run the diagnostic exporter from the repository root: -python export_cortex_m_fallback.py \ +{{< tabpane code=true >}} + {{< tab header="Linux" language="bash">}} +.venv/bin/python export_cortex_m_fallback.py \ --mode cortex-m \ + --model-file model/model.py \ --output-dir artifacts -``` + {{< /tab >}} + {{< tab header="Windows PowerShell" language="powershell">}} +.\.venv\Scripts\python.exe .\export_cortex_m_fallback.py ` + --mode cortex-m ` + --model-file model/model.py ` + --output-dir artifacts + {{< /tab >}} +{{< /tabpane >}} -The relevant output is similar to: +The relevant output is: ```output Relevant targets before Cortex-M lowering: @@ -98,22 +82,28 @@ Relevant targets after Cortex-M lowering: aten.topk.default: 1 cortex_m.dequantize_per_tensor.default: 2 cortex_m.quantize_per_tensor.default: 2 -cortex_m.quantized_conv2d.default: 2 +cortex_m.quantized_conv2d_nhwc.default: 2 +cortex_m.transpose.default: 1 ``` -Both convolutions become `cortex_m.quantized_conv2d` operations. The -`aten.topk` operation remains portable FP32. Quantize and dequantize operations -mark where tensors move between FP32 and INT8 regions. These optimized Cortex-M -boundary operations are part of the executable graph and can matter when -evaluating the cost of a fallback path. +Before lowering, the graph contained two general convolution operations (Stride-1 and Stride-4). After +lowering, both become `cortex_m.quantized_conv2d_nhwc` operations. The `_nhwc` +suffix identifies the explicit channel-last layout used by the Cortex-M +operator. + +`aten.topk` is unchanged, so it will use the portable CPU kernel. The `cortex_m.quantize_*` +and `cortex_m.dequantize_*` operators mark boundaries between floating-point and INT8 +regions. They are real work in the final graph, not annotation-only metadata. + +The helper writes `artifacts/stride4_cortex_m.pte` and a JSON summary. This PTE +is an inspection baseline; the project generator will create the program that +is embedded in the final firmware. ## What you've accomplished and what's next -You have seen how Cortex-M lowering replaces supported patterns with optimized -Cortex-M operators. The convolutions use CMSIS-NN, quantize and dequantize use -Helium vector implementations, and `topk` remains an ExecuTorch portable -kernel. +You have separated two CPU execution routes without involving the NPU. Both +convolutions use Cortex-M operators backed by CMSIS-NN, while `topk` remains a +portable ExecuTorch operation. -Next, you will see how CMSIS-NN provides optimized CPU fallback when deploying a -model to an Ethos-U NPU. The resulting graph combines Ethos-U delegation, -CMSIS-NN-backed Cortex-M operators, and a portable CPU kernel. +Next, you will offer the same graph to the Ethos-U delegate first and inspect +what remains for the CPU. diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/3-add-ethosu-fallback.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/3-add-ethosu-fallback.md index cb0be3adbc..9b17a09d59 100644 --- a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/3-add-ethosu-fallback.md +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/3-add-ethosu-fallback.md @@ -1,90 +1,59 @@ --- -title: CMSIS-NN provides optimized fallback for Ethos-U NPUs +title: Add Ethos-U delegation and CPU fallback weight: 4 -description: Partition the model for Ethos-U55, then lower the remaining supported convolution to CMSIS-NN. +description: Partition TinyCNN for Ethos-U, lower the remaining convolution to Cortex-M, and generate the model-specific AI layer on the U55 demonstration target. ### FIXED, DO NOT MODIFY layout: learningpathall --- -## Deploying across Cortex-M and Ethos-U NPU - -We can re-use the same script as on the previous page, this time using 'Fallback mode'. This uses `EthosUPartitioner` to -create an Ethos-U55 delegate region, represented by `executorch_call_delegate`. -The exporter then applies `CortexMPassManager` to operations left outside that -delegate, lowering its supported convolution to a registered Cortex-M operator -implemented by CMSIS-NN. - -This is planned fallback, not recovery from a runtime NPU error. During export, -the partitioner checks operator support and assigns graph regions to Ethos-U. -Known unsupported operations remain in the graph for another implementation. -Here, the stride-four convolution is deliberately used to demonstrate that -path. - -{{% notice Why this example uses Ethos-U55 %}} -Ethos-U55 is used because its unsupported stride-four convolution creates a -predictable fallback region. Ethos-U85 supports more operator configurations, -so will typically have less CPU fallback than U55. The same workflow still -applies when a model contains operations that remain outside the U85 delegate. -{{% /notice %}} - -For supported neural-network operations, the expected order is Ethos-U NPU, -then CMSIS-NN on the Cortex-M CPU, then a portable CPU kernel. This keeps as much -work as possible on the NPU and gives unsupported quantized work an optimized -CPU route before using a general implementation. Measure the complete graph on -the target device because tensor conversions and graph boundaries can change -the end-to-end result. - -The exporter first creates an Ethos-U55 compile specification and configures -the target-specific quantizer: - -```python -compile_spec = EthosUCompileSpec(target=args.target) -quantizer = EthosUQuantizer(compile_spec) -quantizer.set_global( - get_symmetric_quantization_config(is_per_channel=True) -) -``` - -After quantization, `EthosUPartitioner` identifies supported regions and lowers -them into an Ethos-U delegate: - -```python -edge_program_manager = to_edge_transform_and_lower( - quantized_exported_program, - partitioner=[EthosUPartitioner(compile_spec)], - compile_config=EdgeCompileConfig(_check_ir_validity=False), -) +## Partition the same model for Ethos-U + +The mixed export changes the order of decisions, not the model. It first gives +`EthosUPartitioner` the opportunity to claim supported graph regions. It then +applies Cortex-M lowering only to the operations left outside the delegate: + +```text +quantize the graph + | + v +partition supported work for Ethos-U + | + v +lower supported remainder to Cortex-M operators + | + v +leave the final remainder as portable operations ``` -The exporter then applies Cortex-M lowering to the operations that remain -outside the delegate: - -```python -edge_program_manager._edge_programs["forward"] = CortexMPassManager( - edge_program_manager.exported_program() -).transform() -``` - -The order is important: Ethos-U partitioning runs first, so -`CortexMPassManager` only transforms the non-delegated remainder. - -## Export the partially delegated program +Ordering matters. If Cortex-M lowering ran first, it could rewrite a +convolution that should have been offered to the NPU. -Run the exporter in Ethos-U fallback mode: +This run uses the `ethos-u55-128` target to create the predictable fallback +described earlier. The partition-then-lower order also applies to +other Ethos-U targets; the operations left outside their delegates can differ. -```bash -source ~/executorch-cmsis-lp/env.sh -source "$ET_ROOT/.venv/bin/activate" -cd "$LP_WORKSPACE" +Run the diagnostic exporter in mixed mode: -python export_cortex_m_fallback.py \ +{{< tabpane code=true >}} + {{< tab header="Linux" language="bash">}} +.venv/bin/python export_cortex_m_fallback.py \ --mode ethos-u-fallback \ --target ethos-u55-128 \ + --model-file model/model.py \ --output-dir artifacts -``` + {{< /tab >}} + {{< tab header="Windows PowerShell" language="powershell">}} +.\.venv\Scripts\python.exe .\export_cortex_m_fallback.py ` + --mode ethos-u-fallback ` + --target ethos-u55-128 ` + --model-file model/model.py ` + --output-dir artifacts + {{< /tab >}} +{{< /tabpane >}} -The relevant output is similar to: +Vela prints a summary for the region it compiles. The graph inspection at the +end shows the complete program: ```output Relevant targets before Cortex-M lowering: @@ -96,44 +65,101 @@ Relevant targets after Cortex-M lowering: aten.topk.default: 1 cortex_m.dequantize_per_tensor.default: 2 cortex_m.quantize_per_tensor.default: 2 -cortex_m.quantized_conv2d.default: 1 +cortex_m.quantized_conv2d_nhwc.default: 1 +cortex_m.transpose.default: 2 executorch_call_delegate: 1 ``` -The final graph contains all three execution paths: +Compare this result with the baseline: + +| Model work | Final graph target | Execution engine | +| --- | --- | --- | +| Stride-one convolution and supported surrounding work | `executorch_call_delegate` | Ethos-U55 | +| Stride-four convolution | `cortex_m.quantized_conv2d_nhwc` | Cortex-M55 using CMSIS-NN | +| Output selection | `aten.topk` | Cortex-M55 using the portable ExecuTorch kernel | + +One of the two convolution nodes has disappeared into the delegate. The other +becomes a Cortex-M operator. `topk` remains portable. + +{{% notice Note %}} +Vela can report `CPU operators = 0` for the region submitted to it. That line +doesn't describe the complete ExecuTorch graph. The graph summary shows the +Cortex-M convolution and portable `topk` that remain outside the delegate. +{{% /notice %}} + +## Connect fallback to the CMSIS project + +The PTE you have just produced by runnning the diagnostic exporter is not used by the final CMSIS project. It's simply an inspection result that proves the graph split. +The CMSIS-Executorch project uses `create_ai_layer.py` to export the model again, generate the final PTE, and select the source components required to execute it. -- `executorch_call_delegate` contains the stride-one convolution accepted by - Ethos-U55. -- `cortex_m.quantized_conv2d` is the stride-four convolution rejected by - Ethos-U55 and lowered to CMSIS-NN. -- `aten.topk` remains a portable FP32 operation. +However, some patches have to be made. The project generator within that project currently handles Ethos-U delegation, but it does not lower the remaining convolution to Cortex-M or select CMSIS-NN. The focused +integration patch makes four related changes: -The graph contains two quantize and two dequantize operations because values -cross between delegated INT8 work, Cortex-M INT8 work, and portable FP32 work. -Keeping these conversions visible helps you judge the whole fallback region, -rather than considering only the convolution kernel. +- It declares `ARM::CMSIS-NN@8.0.0` as a project pack. +- It applies Cortex-M lowering after Ethos-U partitioning. +- It maps explicit-layout names such as `quantized_conv2d_nhwc` to their Pack + component names. +- It selects the CMSIS-NN library when the exported program contains a + CMSIS-NN-backed Cortex-M operator. -The lowered targets determine which components the CMSIS project needs: +Check and apply the patch, then inspect its small project diff: -| Model operation | Lowered graph target | CMSIS project component | Execution engine | -| --- | --- | --- | --- | -| Stride-one convolution | `executorch_call_delegate` | `Backend EthosU` and `Ethos-U Driver Generic U55` | Ethos-U55 | -| Stride-four convolution | `cortex_m.quantized_conv2d` | `Cortex-M quantized_conv2d` and `CMSIS:NN Lib` | Cortex-M55 with CMSIS-NN | -| Quantized boundaries | `cortex_m.quantize_per_tensor` and `cortex_m.dequantize_per_tensor` | Matching Cortex-M operator components | Cortex-M55 | -| Output selection | `aten.topk` | `Portable topk` | Cortex-M55 portable FP32 kernel | +```console +git apply --check enable_cortex_m_fallback.patch +git apply enable_cortex_m_fallback.patch +git diff -- cmsis-executorch-u55.csolution.yml create_ai_layer.py +``` + +No board, runner, or linker changes are needed. + +## Generate the target description + +If you opened a new terminal, initialize `vcpkg` and run +`vcpkg-shell activate` again. Then resolve the declared packs and active target: + +```console +cbuild setup cmsis-executorch-u55.csolution.yml --active SSE-300-U55 --packs +``` + +This one command installs missing packs through the public Pack index and +writes three useful descriptions: + +- `cmsis-executorch-u55.cbuild-pack.yml` locks the resolved Pack versions. +- `cmsis-executorch-u55.cbuild-mlops.yml` describes the Cortex-M55, Ethos-U55, + Vela settings, and destination AI layer. +- `cmsis-executorch-u55.cbuild-idx.yml` indexes the resolved build context. + +You need `--packs` on a fresh checkout. There is no separate manual Pack +download or `cpackget add` step. + +## Generate the AI layer + +Use the MLOps description to export the model and write its CMSIS AI layer: + +{{< tabpane code=true >}} + {{< tab header="Linux" language="bash">}} +.venv/bin/python create_ai_layer.py cmsis-executorch-u55.cbuild-mlops.yml + {{< /tab >}} + {{< tab header="Windows PowerShell" language="powershell">}} +.\.venv\Scripts\python.exe .\create_ai_layer.py .\cmsis-executorch-u55.cbuild-mlops.yml + {{< /tab >}} +{{< /tabpane >}} + +The relevant final output is similar to: + +```output +[ai_layer] 7608 byte program, operators: ['Cortex-M dequantize_per_tensor', 'Cortex-M quantize_per_tensor', 'Cortex-M quantized_conv2d', 'Cortex-M transpose', 'Portable topk'] +[ai_layer] wrote .../ai_layer/U55/ai_layer.clayer.yml +``` -The CMSIS project also needs the ExecuTorch runtime, kernel utilities, and -operator registration components. You will inspect these components when you -open the published pack. +The generator writes the final `model.pte`, converts it to C data for the +firmware, and records the runtime and operator components needed to execute it. ## What you've accomplished and what's next -You have exported a program that distributes work across Ethos-U55, -CMSIS-NN, and a portable FP32 kernel. The unsupported convolution does not need -to fall back to a general FP32 implementation: it remains quantized and uses a -CMSIS-NN-backed Cortex-M operator. This is designed to reduce the CPU cost of -known unsupported NPU work, although performance must still be measured on the -target device, including quantize and dequantize boundaries. +You have produced a model-specific AI layer with one Ethos-U55 delegate, one +CMSIS-NN-backed fallback convolution, and one portable operation. The split is +decided ahead of time and stored in the ExecuTorch program. -Next, you will download and inspect the published ExecuTorch CMSIS Pack that -supplies these runtime, backend, and operator components. +Next, you will inspect how the two published source Packs supply the code behind +those graph targets. diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/4-inspect-cmsis-pack.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/4-inspect-cmsis-pack.md index 04bbd752cd..eb628b5765 100644 --- a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/4-inspect-cmsis-pack.md +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/4-inspect-cmsis-pack.md @@ -1,97 +1,113 @@ --- -title: Inspect the published ExecuTorch CMSIS Pack +title: Inspect the generated CMSIS Pack selection weight: 5 -description: Download the published ExecuTorch 1.4.0 CMSIS Pack and inspect the runtime, backend, and operator components used by the model. +description: Trace the mixed ExecuTorch graph to versioned runtime, backend, operator, and CMSIS-NN source components. ### FIXED, DO NOT MODIFY layout: learningpathall --- -## Understand the ExecuTorch CMSIS Pack benefits +## See which Packs were resolved -The ExecuTorch CMSIS Pack is a source pack rather than a prebuilt runtime -library. It supplies the ExecuTorch runtime, registration code, backends, and -operators as selectable CMSIS components. +CMSIS-Toolbox has already acquired the software Packs needed by the project. +Inspect the generated Pack lock: -A `.pack` file is a ZIP-compatible archive. Its Pack Description file, or -PDSC, records component names, source files, dependencies, versions, and -conditions. CMSIS-Toolbox reads this metadata to turn component selections in a -project into a concrete build. +{{< tabpane code=true >}} + {{< tab header="Linux" language="bash">}} +cat cmsis-executorch-u55.cbuild-pack.yml + {{< /tab >}} + {{< tab header="Windows PowerShell" language="powershell">}} +Get-Content .\cmsis-executorch-u55.cbuild-pack.yml + {{< /tab >}} +{{< /tabpane >}} -A CMSIS project can select only the components required by its lowered graph -instead of maintaining one large source list. The selected source files then -compile with the application's target compiler and options. +Among the resolved entries, you can see the two Packs responsible for the +mixed model: -The published pack also gives CMSIS-Toolbox and Keil Studio a versioned package -identifier: `PyTorch::ExecuTorch@1.4.0`. This allows the tools to download the -same package, resolve its dependencies, expose its components in an IDE, and -record the selected version without a local pack-building stage. - -## Download the published pack - -Download the pack attached to the ExecuTorch `1.4.0` release: - -```bash -source ~/executorch-cmsis-lp/env.sh - -PACK_DIR="$LP_WORKSPACE/packs" -PACK_FILE="$PACK_DIR/PyTorch.ExecuTorch.1.4.0.pack" - -mkdir -p "$PACK_DIR" - -curl --location --fail \ - --output "$PACK_FILE" \ - https://github.com/pytorch/executorch/releases/download/v1.4.0/PyTorch.ExecuTorch.1.4.0.pack - -unzip -t "$PACK_FILE" >/dev/null && \ - echo "ExecuTorch CMSIS Pack archive passed" +```yaml +- resolved-pack: ARM::CMSIS-NN@8.0.0 +- resolved-pack: PyTorch::ExecuTorch@1.5.1 ``` -The expected final output is: - -```output -ExecuTorch CMSIS Pack archive passed +They have different responsibilities: + +- `PyTorch::ExecuTorch@1.5.1` supplies the ExecuTorch runtime, Ethos-U backend, + Cortex-M operator wrappers, registration code, and portable operators. +- `ARM::CMSIS-NN@8.0.0` supplies the optimized neural-network kernels called by + the Cortex-M convolution wrapper. + +The Pack lock records exact resolved versions so later builds don't silently +move to newer installed Packs. + +## Understand what a source Pack provides + +The ExecuTorch CMSIS Pack is a *source Pack*, not a prebuilt runtime library. +Its Pack Description file lists named components, their source files, their +dependencies, and the conditions under which they can be selected. + +CMSIS-Toolbox uses that metadata to add only the chosen components to the +application build. Arm Compiler then compiles those sources for the Cortex-M55 +target with the rest of the project. This is why one published Pack can support +different CMSIS projects without shipping a separate binary runtime for every +target configuration. + +`cbuild setup --packs` stores each installed Pack under the CMSIS Pack root in +a vendor, name, and version hierarchy. Both `cbuild` and other CMSIS tools can +then resolve `PyTorch::ExecuTorch@1.5.1` by its package identity. Manually +downloading the same archive with `curl` would create an unrelated second copy +and wouldn't improve the build. + +## Trace the graph into the AI layer + +Open the AI layer generated from the `.pte` operator list: + +{{< tabpane code=true >}} + {{< tab header="Linux" language="bash">}} +cat ai_layer/U55/ai_layer.clayer.yml + {{< /tab >}} + {{< tab header="Windows PowerShell" language="powershell">}} +Get-Content .\ai_layer\U55\ai_layer.clayer.yml + {{< /tab >}} +{{< /tabpane >}} + +Its component list includes: + +```yaml +components: + - component: Machine Learning:ExecuTorch:Runtime + - component: Machine Learning:ExecuTorch:Kernel Utils + - component: Machine Learning:ExecuTorch:Kernel Registration + - component: Machine Learning:ExecuTorch:Backend EthosU + - component: Machine Learning:ExecuTorch Operators:Cortex-M dequantize_per_tensor + - component: Machine Learning:ExecuTorch Operators:Cortex-M quantize_per_tensor + - component: Machine Learning:ExecuTorch Operators:Cortex-M quantized_conv2d + - component: Machine Learning:ExecuTorch Operators:Cortex-M transpose + - component: Machine Learning:ExecuTorch Operators:Portable topk + - component: ARM::CMSIS:NN Lib ``` -This copy is used only to inspect the archive and its component metadata. The -final build installs the same pack through the public pack index into a clean -pack root. That separate installation checks the normal CMSIS dependency and -component-resolution workflow. - -## Inspect the components used by the model +This file is the handover between model export and firmware build: -Display the PDSC components that correspond to the partially delegated graph: +| Export result | Selected component | +| --- | --- | +| Ethos-U delegate | `Backend EthosU` | +| Cortex-M convolution | `Cortex-M quantized_conv2d` and `ARM::CMSIS:NN Lib` | +| Quantized boundaries and layout changes | Matching Cortex-M components | +| Portable `topk` | `Portable topk` | -```bash -unzip -p "$PACK_FILE" PyTorch.ExecuTorch.pdsc | \ -grep ' uart_config.h <<'EOF' -#pragma once - -#define UART0_BASE (0x49303000) -#define UART0_BAUDRATE (115200) -#define SYSTEM_CORE_CLOCK (25000000) -EOF +`SSE-300-U55` is the project's readable target name. The `ARM::` values identify +specific board and device records supplied by installed Packs. CMSIS-Toolbox +uses those identities to resolve startup files, memory information, and device +conditions. + +The solution's `mlops:` node is the single source for the NPU and Vela target +settings: + +```yaml +mlops: + npu: + type: Ethos-U55 + macs: 128 + vela: + system: Ethos_U55_High_End_Embedded + memory: Shared_Sram + model: + clayer: $AI-Layer$ + name: TinyCNN ``` -## Define the CMSIS solution and project - -Create `pack_runner.csolution.yml`. This file selects the compiler, target, -build type, project, and pack versions: - -```bash -cat > pack_runner.csolution.yml <<'YAML' -solution: - created-for: CMSIS-Toolbox@2.13.0 - cdefault: - compiler: GCC@14.3.1 +`cbuild setup` resolves this into the `*.cbuild-mlops.yml` file consumed by +`create_ai_layer.py`. The exporter therefore doesn't duplicate the project's +NPU configuration. - packs: - - pack: ARM::CMSIS@>=6.0.0 - - pack: ARM::CMSIS-NN@7.0.0 - - pack: ARM::Cortex_DFP@>=1.1.0 - - pack: ARM::ethos-u-core-driver@1.26.2 - - pack: PyTorch::ExecuTorch@1.4.0 +## Follow the layers into the project - target-types: - - type: Corstone-300 - device: ARM::ARMCM55 +`cmsis-executorch.cproject.yml` owns the application and consumes two layers: - build-types: - - type: Release - optimize: speed - - projects: - - project: ./pack_runner.cproject.yml -YAML -``` +```yaml +groups: + - group: App + files: + - file: ./src/app_main.cpp + - file: ./src/arm_embedded_module.cpp + - file: ./src/arm_embedded_module.hpp -Create `pack_runner.cproject.yml`. The `components` section selects the -ExecuTorch, CMSIS-NN, Ethos-U, Cortex-M, and portable components used by the -lowered model. These selections come directly from the target names observed on -the previous pages. The remaining sections configure and build the runner -around those components: - -```bash -cat > pack_runner.cproject.yml <<'YAML' -project: - packs: - - pack: PyTorch::ExecuTorch@1.4.0 - - output: - base-name: pack_runner - type: - - elf - - map - - linker: - - script: ./platform.ld - for-compiler: GCC - - components: - - component: ARM::CMSIS:CORE - - component: ARM::CMSIS:NN Lib - - component: ARM::Machine Learning:NPU Support:Ethos-U Driver&Generic U55 - - component: Machine Learning:ExecuTorch:Runtime - - component: Machine Learning:ExecuTorch:Kernel Utils - - component: Machine Learning:ExecuTorch:Kernel Registration - - component: Machine Learning:ExecuTorch:Backend EthosU - - component: Machine Learning:ExecuTorch Operators:Cortex-M quantize_per_tensor - - component: Machine Learning:ExecuTorch Operators:Cortex-M dequantize_per_tensor - - component: Machine Learning:ExecuTorch Operators:Cortex-M quantized_conv2d - - component: Machine Learning:ExecuTorch Operators:Portable topk - - add-path: - - . - - ./executorch/examples/arm/executor_runner - - ./executorch/examples/arm/arm-scratch/ethos-u/core_platform/targets/common/include - - ./executorch/examples/arm/arm-scratch/ethos-u/core_platform/drivers/mpu/include - - ./executorch/examples/arm/arm-scratch/ethos-u/core_platform/drivers/timing_adapter/include - - ./executorch/examples/arm/arm-scratch/ethos-u/core_platform/drivers/uart/include - - ./executorch/examples/arm/arm-scratch/ethos-u/core_platform/drivers/mailbox/include - - ./executorch/examples/arm/arm-scratch/ethos-u/core_platform/drivers/mhu_dummy/include - - ./executorch/examples/arm/arm-scratch/ethos-u/core_software/Cortex_DFP/Device/ARMCM55/Include - - ./executorch/examples/arm/arm-scratch/ethos-u/core_software/cmsis_6/CMSIS/Core/Include - - define: - - ETHOSU - - ETHOSU55 - - ETHOSU_HAS_TA - - ET_COMPILED_PTE - - ET_LOG_DUMP_OUTPUT - - misc: - - for-compiler: GCC - C-CPP: - - -includeARMCM55.h - - -DARM_NN_ENABLE_F16=0 - - -DARM_NN_ENABLE_F32=0 - - -DCMSIS_VER=6 - - -DETHOSU_ARENA=1 - - -DETHOSU_MACS=128 - - -DETHOSU_MODEL=1 - - -DET_ARM_BAREMETAL_SCRATCH_TEMP_ALLOCATOR_POOL_SIZE=0x200000 - - -DET_NUM_INFERENCES=1 - C: - - -std=gnu11 - CPP: - - -std=gnu++17 - - -fno-unwind-tables - - -fno-rtti - - -fno-exceptions - Link: - - -Wl,-u,_printf_float - - groups: - - group: Application - files: - - file: ./executorch/examples/arm/executor_runner/arm_executor_runner.cpp - - file: ./executorch/examples/arm/executor_runner/arm_memory_allocator.cpp - - file: ./executorch/examples/arm/executor_runner/arm_perf_monitor.cpp - - file: ./executorch/extension/runner_util/inputs.cpp - - file: ./executorch/extension/runner_util/inputs_portable.cpp - - group: Corstone-300 - files: - - file: ./executorch/examples/arm/arm-scratch/ethos-u/core_platform/targets/common/src/init.cpp - - file: ./executorch/examples/arm/arm-scratch/ethos-u/core_platform/targets/corstone-300/target.cpp - - file: ./executorch/examples/arm/arm-scratch/ethos-u/core_platform/targets/corstone-300/retarget.c - - file: ./executorch/examples/arm/arm-scratch/ethos-u/core_platform/drivers/mpu/src/mpu.cpp - - file: ./executorch/examples/arm/arm-scratch/ethos-u/core_platform/drivers/timing_adapter/src/timing_adapter.c - - file: ./executorch/examples/arm/arm-scratch/ethos-u/core_platform/drivers/uart/src/uart_cmsdk_apb.c - - file: ./executorch/examples/arm/arm-scratch/ethos-u/core_platform/drivers/mailbox/src/mailbox.cpp - - file: ./executorch/examples/arm/arm-scratch/ethos-u/core_platform/drivers/mhu_dummy/src/mhu_dummy.cpp - - file: ./executorch/examples/arm/arm-scratch/ethos-u/core_software/Cortex_DFP/Device/ARMCM55/Source/system_ARMCM55.c - - file: ./executorch/examples/arm/arm-scratch/ethos-u/core_software/Cortex_DFP/Device/ARMCM55/Source/startup_ARMCM55.c -YAML +layers: + - layer: $Board-Layer$ + type: Board + - layer: $AI-Layer$ + type: AI ``` -### How the CMSIS project file is structured +Each part has one job: -The CMSIS project file combines components installed from packs with application -and platform sources from the ExecuTorch checkout: +- The Board layer supplies Cortex-M55 startup, the memory layout, standard + output, the Ethos-U55 driver, and Corstone-300 initialization. +- The generated AI layer supplies the embedded model and its exact ExecuTorch + and CMSIS-NN component selection. +- The App group initializes ExecuTorch, creates an input tensor, runs one + inference, and prints the ten output values. -| YAML section | Purpose | -| --- | --- | -| `packs` | Pins the project to `PyTorch::ExecuTorch@1.4.0`. | -| `output` and `linker` | Select the ELF and map outputs and the generated Corstone-300 linker script. | -| `components` | Select CMSIS Core, CMSIS-NN, the Ethos-U55 driver, ExecuTorch runtime support, and only the operators required by the lowered graph. | -| `add-path` | Adds headers for the runner and Corstone-300 platform sources mounted from the ExecuTorch checkout. | -| `define` and `misc` | Configure the Ethos-U55 target, embedded `.pte` data, logging, memory allocation, inference count, and compiler options. | -| `groups` | Adds the runner and Corstone-300 source files that form the application around the pack components. | +The FVP configuration in `board/Corstone-300/fvp_config.txt` describes the +simulated hardware. It is separate from the Board layer because one configures +the simulator while the other builds software for the simulated target. -The `components` section connects the lowered graph to the embedded build. It -selects the required CPU and NPU implementations from the pinned ExecuTorch -`1.4.0` pack instead of maintaining a source list manually. The entries under -`groups` remain application and platform code; they are not supplied by the -pack. +## See how the PTE enters the firmware -## Embed the exported program +The AI-layer directory now contains these related model files: -Convert the partially delegated `.pte` file to a C array. The linker places this -array in the Corstone-300 DDR model section: - -```bash -cd "$RUNNER_DIR" -source "$ET_ROOT/.venv/bin/activate" - -python "$ET_ROOT/examples/arm/executor_runner/pte_to_header.py" \ - --pte "$LP_WORKSPACE/artifacts/stride4_ethos_u_fallback.pte" \ - --outdir "$RUNNER_DIR" -``` - -The expected output includes: - -```output -Section: network_model_sec. +```text +ai_layer/U55/model.pte +ai_layer/U55/model_pte.c +ai_layer/U55/model_pte.h ``` -The `.pte` file is the serialized ExecuTorch program produced by the exporter. -Converting it to `model_pte.h` makes the program part of the firmware image, so -the bare-metal runner does not need a filesystem or external model loader. - -## Generate the Corstone-300 linker script - -Preprocess the ExecuTorch linker template with the Arm GNU compiler: - -```bash -source "$ET_ROOT/examples/arm/arm-scratch/setup_path.sh" - -arm-none-eabi-gcc -E -x c -P \ - -o "$RUNNER_DIR/platform.ld" \ - "$ET_ROOT/examples/arm/executor_runner/Corstone-300.ld" - -ls -lh "$RUNNER_DIR/model_pte.h" "$RUNNER_DIR/platform.ld" -``` +`model.pte` is the serialized ExecuTorch program. `model_pte.c` contains the +same bytes represented as a C array, which is why it appears to be a long list +of numbers. Embedding the bytes lets a bare-metal system load the program +without a filesystem. -`model_pte.h` embeds the model. `platform.ld` places code, the model, the method -allocator, and the CMSIS-NN scratch arena in memory regions available on the -Corstone-300 FVP. Preprocessing is required because the linker template uses C -preprocessor directives to produce the final GNU linker script. +`src/app_main.cpp` includes `model_pte.h`, wraps a deterministic +`1 x 16 x 16 x 3` input buffer as an ExecuTorch tensor, and calls the runtime. +The runtime follows the PTE execution plan: it invokes the Ethos-U backend for +the delegate and CPU kernels for the remaining operations. -The CMSIS project mounts the v1.4.0 checkout read-only to compile the existing -runner and Corstone platform sources. ExecuTorch runtime, backend, registration, -and operator sources come from the installed ExecuTorch CMSIS Pack. +Because the adapted model preserves the original input shape and ten-value +output, the runner doesn't need model-specific edits. A different model shape +or output contract would need corresponding changes in `app_main.cpp`. -## What you've accomplished and what's next +## What you've learned and what's next -You have prepared a Corstone-300 CMSIS project containing the exported model -and the platform files needed to execute it. The project selects the versioned -ExecuTorch, CMSIS-NN, Ethos-U, and portable components that match the graph. +You have traced the ready-made project from target selection to the embedded +PTE. The repository removes board bring-up work from the lesson while keeping +the CMSIS structure visible and reusable. -Next, you will build the application from the selected pack components, verify -their source paths, and run the model on the FVP. +Next, you will compile these parts, verify which Pack sources were used, and run +the image on the Corstone-300 FVP. diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/6-consume-cmsis-pack.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/6-consume-cmsis-pack.md index f3bba87d70..c63abb1a84 100644 --- a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/6-consume-cmsis-pack.md +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/6-consume-cmsis-pack.md @@ -1,158 +1,121 @@ --- -title: Build and run the ExecuTorch CMSIS Pack application +title: Build and run the mixed application weight: 7 -description: Build a Corstone-300 executable from ExecuTorch, CMSIS-NN, and Ethos-U CMSIS components, then run the model on an Ethos-U55 FVP. +description: Compile the selected Pack sources, verify their provenance, and run the application on the Corstone-300 Ethos-U55 FVP. ### FIXED, DO NOT MODIFY layout: learningpathall --- -## Build the executable from ExecuTorch CMSIS Pack components +## Build the application -Build the application in the arm64 AVH-MLOps container. The command: +Make sure the `vcpkg` tools are active in your current terminal, then build the +U55 target from the repository root: -- Mounts the ExecuTorch checkout as a read-only input. -- Activates the CMSIS-Toolbox utilities supplied by the container. -- Installs `PyTorch::ExecuTorch@1.4.0` from the pack index into a clean pack root. -- Runs `cbuild` for the Corstone-300 release context. - -The temporary `CMSIS_PACK_ROOT` is intentionally empty at the start. A -successful build therefore shows that the project can resolve its declared -packs, rather than relying on packages left by an earlier build. - -```bash -source ~/executorch-cmsis-lp/env.sh -cd "$RUNNER_DIR" - -DOCKER_IMAGE="ghcr.io/arm-software/avh-mlops/arm-mlops-docker-licensed-community:latest-arm64" - -chmod a+rwx "$RUNNER_DIR" - -docker run --rm \ - -v "$RUNNER_DIR:/workspace" \ - -v "$ET_ROOT:/workspace/executorch:ro" \ - "$DOCKER_IMAGE" \ - bash -lc ' -set -euo pipefail -cd /workspace - -export Z_VCPKG_POSTSCRIPT="$(mktemp /tmp/vcpkg.XXXXXX.sh)" -vcpkg activate -source "$Z_VCPKG_POSTSCRIPT" - -export CMSIS_PACK_ROOT=/tmp/cmsis-pack-root -cpackget init https://www.keil.com/pack/index.pidx -cpackget add --agree-embedded-license PyTorch::ExecuTorch@1.4.0 - -cbuild pack_runner.csolution.yml \ - --packs \ - --update-rte \ - --context pack_runner.Release+Corstone-300 -' +```console +cbuild cmsis-executorch-u55.csolution.yml --active SSE-300-U55 ``` -In the command, `cpackget init` configures the public pack index and -`cpackget add` installs the selected ExecuTorch pack. `cbuild --packs` resolves -declared pack dependencies, `--update-rte` regenerates the run-time environment -files, and `--context` selects the project, build type, and target combination -defined by the solution. - -{{% notice Note %}} -The generated `pack_runner.csolution.yml` and `pack_runner.cproject.yml` files -can also be opened with the Arm CMSIS Solution extension included in the -[Arm Keil Studio Pack](https://marketplace.visualstudio.com/items?itemName=Arm.keil-studio-pack) -for Visual Studio Code. Use the graphical view to inspect the selected target, -build context, packs, and software components. -{{% /notice %}} - -The first build downloads the arm64 AVH-MLOps container, CMSIS-Toolbox compiler -tools, the ExecuTorch CMSIS Pack, and its dependent packs. The final build -summary is: +CMSIS-Toolbox combines the App, Board, and AI layers, resolves their selected +components, and invokes Arm Compiler for Embedded. The final output includes: ```output Build summary: 1 succeeded, 0 failed ``` -{{% notice Note %}} -GNU `ld` can print allocation warnings for the Corstone multi-segment linker -script before the successful build summary. The generated ELF is validated in -the next section on the FVP. -{{% /notice %}} - -The executable is written to: +The generated images are written under: ```text -$RUNNER_DIR/out/pack_runner/Corstone-300/Release/pack_runner.elf +out/cmsis-executorch/SSE-300-U55/Debug/ ``` -## Confirm that the build used the pack +The `.axf` file contains the linked executable and debug information. The +`.hex` file contains the image loaded into the FVP. -Run the verification helper against the generated compilation database. It -checks that the runtime, CMSIS-NN-backed Cortex-M operator, Ethos-U backend, and -portable `topk` sources came from the installed pack: +## Verify the compiled source paths -```bash -COMPILE_DB=out/pack_runner/Corstone-300/Release/compile_commands.json -python verify_pack_sources.py "$COMPILE_DB" -``` +A successful build proves that an image was produced, but it doesn't by itself +show which implementation supplied each operation. Run the verification helper +against the compilation database: + +{{< tabpane code=true >}} + {{< tab header="Linux" language="bash">}} +.venv/bin/python verify_pack_sources.py \ + out/cmsis-executorch/SSE-300-U55/Debug/compile_commands.json + {{< /tab >}} + {{< tab header="Windows PowerShell" language="powershell">}} +.\.venv\Scripts\python.exe .\verify_pack_sources.py ` + .\out\cmsis-executorch\SSE-300-U55\Debug\compile_commands.json + {{< /tab >}} +{{< /tabpane >}} The expected output is: ```output -PASS: ExecuTorch runtime came from the ExecuTorch CMSIS Pack -PASS: CMSIS-NN Cortex-M convolution came from the ExecuTorch CMSIS Pack -PASS: Ethos-U backend came from the ExecuTorch CMSIS Pack -PASS: Portable topk came from the ExecuTorch CMSIS Pack +PASS: ExecuTorch runtime was compiled from its declared CMSIS Pack +PASS: ExecuTorch Cortex-M convolution wrapper was compiled from its declared CMSIS Pack +PASS: CMSIS-NN convolution kernel was compiled from its declared CMSIS Pack +PASS: Ethos-U backend was compiled from its declared CMSIS Pack +PASS: ExecuTorch portable topk kernel was compiled from its declared CMSIS Pack ``` -Each `PASS` confirms that one required ExecuTorch source file appears in the -compilation database under the installed pack path. The application -and Corstone platform sources come from the ExecuTorch checkout. A -successful build confirms that an ELF was produced; this additional check -confirms that the selected ExecuTorch components were resolved from the pack. +These checks distinguish the two layers of the optimized fallback. The +ExecuTorch Pack provides the `quantized_conv2d` wrapper, while the CMSIS-NN +Pack provides the convolution implementation that wrapper calls. -## Run the pack-built model on the FVP +Together with `cmsis-executorch-u55.cbuild-pack.yml`, the compilation database +shows both the resolved Pack versions and the source files actually compiled. -Run the `cbuild` output on the Corstone-300 Ethos-U55 FVP: +## Run the image on the FVP -```bash -source "$ET_ROOT/examples/arm/arm-scratch/setup_path.sh" +Start the Corstone-300 model and load the generated HEX image: -"$ET_ROOT/backends/arm/scripts/run_fvp.sh" \ - --elf="$RUNNER_DIR/out/pack_runner/Corstone-300/Release/pack_runner.elf" \ - --target=ethos-u55-128 +```console +FVP_Corstone_SSE-300_Ethos-U55 -f board/Corstone-300/fvp_config.txt --simlimit 60 -a cpu0=out/cmsis-executorch/SSE-300-U55/Debug/cmsis-executorch.hex ``` The relevant output is: ```output -PTE Model data loaded. Size: 5476 bytes. -NPU delegations: 1 (1.00 per inference) -Output[0][0]: (float) 0.030337 -Output[0][1]: (float) -0.034964 -Program complete, exiting. -No problems found! +Ethos-U version info: + Arch: v1.1.0 + MACs/cc: 128 + Cmd stream: v0 +ExecuTorch Ethos-U example: 7608 byte model +Output: 10 element(s): 0.0532 -0.0239 0.0541 0.1064 0.0706 0.0395 0.0881 0.1340 0.0807 0.0064 +Test_result: PASS ``` +{{% notice Note %}} +On Windows, the FVP can print warnings that `python3.dll` is unavailable and +that the VIO or VSI bridges are disabled. This headless example doesn't use +those bridges. If the run reaches `Test_result: PASS`, inference completed +successfully. +{{% /notice %}} + + -The graph summary from the export step identifies the stride-four convolution -as `cortex_m.quantized_conv2d` and `topk` as portable. The compilation database -shows that the matching Cortex-M and portable sources came from the ExecuTorch -CMSIS Pack. The FVP output confirms that the complete pack-built application -loaded and ran the same partially delegated program with one Ethos-U delegation. +The FVP result proves functional execution, not physical-device latency or +power. Measure the complete graph on your target hardware before drawing +performance conclusions because delegation boundaries and tensor conversions +also have a cost. ## What you've accomplished -You have run an ExecuTorch model built from the ExecuTorch CMSIS Pack on a -Corstone-300 system. The application delegates the supported convolution to -Ethos-U55, executes the unsupported stride-four convolution through the -CMSIS-NN-backed Cortex-M operator, and finishes with a portable FP32 operator. +You have built and run one ExecuTorch program across three execution paths: + +- The supported convolution runs in an Ethos-U55 delegate. +- The unsupported stride-four convolution runs on Cortex-M55 through the + ExecuTorch wrapper and optimized CMSIS-NN kernel. +- `topk` runs through the portable ExecuTorch CPU kernel. + +You also verified the implementation rather than inferring it from a successful +run. The exported graph shows ownership, the compilation database shows the +compiled Pack sources, and the FVP confirms that the combined firmware runs. -CMSIS-NN provides the optimized CPU implementation for the quantized fallback -convolution. The ExecuTorch CMSIS Pack provides versioned, component-based -integration for the matching runtime, backend, and operator sources. The clean -CMSIS build and compilation-database checks make that software composition -visible and repeatable. +This build uses U55 to make one unsupported convolution predictable. The same +Pack workflow applies to another Ethos-U target when its delegated graph leaves +operations for Cortex-M execution. -For another model, inspect its lowered graph, select the matching components -from the ExecuTorch CMSIS Pack, regenerate `model_pte.h`, and rebuild the CMSIS -project. +To adapt the project, replace `model/model.py`, provide representative +calibration inputs, preserve or update the runner's tensor interface, and +regenerate the AI layer before rebuilding. diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/_index.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/_index.md index cd0fd9ca78..5d244c0a21 100644 --- a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/_index.md +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/_index.md @@ -1,28 +1,27 @@ --- -title: Deploy ML models across Cortex-M and Ethos-U with CMSIS-NN optimized kernels and the ExecuTorch CMSIS Pack +title: Deploy ML models across Cortex-M and Ethos-U with CMSIS-NN fallback -description: Build and run an ExecuTorch 1.4.0 application from the published ExecuTorch CMSIS Pack using Ethos-U, CMSIS-NN, and portable Cortex-M kernels. +description: Build an ExecuTorch 1.5.1 application that delegates supported work to Ethos-U and uses CMSIS-NN for optimized Cortex-M fallback, demonstrated on Ethos-U55. minutes_to_complete: 60 -who_is_this_for: This is an introductory topic for embedded machine learning developers building ExecuTorch applications for Cortex-M CPUs and Ethos-U NPUs. +who_is_this_for: This Learning Path is for embedded machine learning developers who want to understand and build an ExecuTorch application split across an Ethos-U NPU and Cortex-M CPU. learning_objectives: - - Export an ExecuTorch graph with Ethos-U, CMSIS-NN, and portable-kernel execution regions - - Lower quantized CPU fallback operations to optimized CMSIS-NN-backed Cortex-M operators, and understand the benefits in doing so - - Install and consume the published ExecuTorch CMSIS Pack for version 1.4.0 with selected runtime and operator components - - Run an application built from the ExecuTorch CMSIS Pack on a Corstone-300 Ethos-U55 FVP - - Understand when and why to use the ExecuTorch CMSIS Pack + - Explain how ExecuTorch assigns model operations to Ethos-U, CMSIS-NN-backed Cortex-M operators, and portable kernels + - Compare a Cortex-M baseline with a partially delegated Ethos-U program + - Generate a model-specific CMSIS AI layer from published software packs + - Build and run the mixed application on a Corstone-300 Ethos-U55 FVP prerequisites: - - An arm64 Linux computer with Docker and at least 30 GB of free storage; Ubuntu 24.04 LTS is a recommended option - - Python 3.12 with support for virtual environments - - Familiarity with PyTorch export and post-training quantization - - Familiarity with CMake and command-line development tools - - Awareness of Ethos-U NPU, and Cortex-M CPU + - A Windows or Linux computer with Git and Python 3.10 to 3.13 + - An internet connection and command-line access + - Familiarity with PyTorch models and post-training quantization + - Basic familiarity with Cortex-M processors and Ethos-U NPUs author: - Matt Cossins +- Kwashie Andoh generate_summary_faq: false rerun_summary: false @@ -37,22 +36,21 @@ armips: - Ethos-U operatingsystems: - Linux + - Windows tools_software_languages: - Python - PyTorch - ExecuTorch - - CMake - CMSIS - CMSIS Pack - CMSIS-NN - - Docker - FVP - - GCC + - vcpkg further_reading: - resource: - title: ExecuTorch 1.4.0 release - link: https://github.com/pytorch/executorch/releases/tag/v1.4.0 + title: CMSIS-Executorch example project + link: https://github.com/Arm-Examples/CMSIS-Executorch type: documentation - resource: title: Published ExecuTorch CMSIS Pack @@ -63,13 +61,13 @@ further_reading: link: https://docs.pytorch.org/executorch/stable/backends/arm-cortex-m/arm-cortex-m-overview.html type: documentation - resource: - title: ExecuTorch Ethos-U with CMSIS-NN fallback example - link: https://github.com/pytorch/executorch/blob/v1.4.0/examples/arm/ethos_u_cmsis_nn_fallback_example.ipynb + title: ExecuTorch Arm Ethos-U backend + link: https://docs.pytorch.org/executorch/stable/backends-arm-ethos-u.html type: documentation - resource: - title: CMSIS-Toolbox install guide - link: /install-guides/cmsis-toolbox/ - type: install_guide + title: CMSIS-Toolbox MLOps integration + link: https://open-cmsis-pack.github.io/cmsis-toolbox/build-overview/#mlops-information + type: documentation - resource: title: CMSIS-NN documentation link: https://arm-software.github.io/CMSIS-NN/latest/ diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/enable_cortex_m_fallback.patch b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/enable_cortex_m_fallback.patch new file mode 100644 index 0000000000..f3fa9d0fa2 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/enable_cortex_m_fallback.patch @@ -0,0 +1,70 @@ +diff --git a/cmsis-executorch-u55.csolution.yml b/cmsis-executorch-u55.csolution.yml +--- a/cmsis-executorch-u55.csolution.yml ++++ b/cmsis-executorch-u55.csolution.yml +@@ -21,5 +21,6 @@ solution: + # to load. `cbuild setup --packs` fetches it from the public index. + - pack: PyTorch::ExecuTorch@1.5.1 + - pack: ARM::CMSIS@6.3.0 ++ - pack: ARM::CMSIS-NN@8.0.0 + - pack: ARM::CMSIS-Compiler + - pack: ARM::Cortex_DFP +diff --git a/create_ai_layer.py b/create_ai_layer.py +--- a/create_ai_layer.py ++++ b/create_ai_layer.py +@@ -173,6 +173,9 @@ def export_model(spec) -> bytes: + """Quantize model/model.py, delegate it to the Ethos-U and return the .pte.""" + import torch + from executorch.backends.arm.ethosu import EthosUPartitioner ++ from executorch.backends.cortex_m.passes.cortex_m_pass_manager import ( ++ CortexMPassManager, ++ ) + from executorch.backends.arm.quantizer import EthosUQuantizer, get_symmetric_quantization_config + from executorch.exir import EdgeCompileConfig, ExecutorchBackendConfig, to_edge_transform_and_lower + from torchao.quantization.pt2e.quantize_pt2e import convert_pt2e, prepare_pt2e +@@ -194,5 +197,8 @@ def export_model(spec) -> bytes: + partitioner=[EthosUPartitioner(spec)], + compile_config=EdgeCompileConfig(_check_ir_validity=False), + ) ++ edge._edge_programs["forward"] = CortexMPassManager( ++ edge.exported_program(), use_explicit_layout=True ++ ).transform() + program = edge.to_executorch(ExecutorchBackendConfig(extract_delegate_segments=False)) + return bytes(program.buffer) +@@ -214,7 +220,7 @@ def components(pte: bytes, pack: Path) -> tuple[list[str], list[str]]: + ns, op = ns.decode(), op.decode() + candidates = [ + f"{family[ns]} {op}", +- f"{family[ns]} {re.sub(r'_(per_tensor|per_channel|byte|copy)$', '', op)}", ++ f"{family[ns]} {re.sub(r'_(per_tensor|per_channel|byte|copy|nhwc)$', '', op)}", + ] + if match := next((c for c in candidates if c in available), None): + selected.add(match) +@@ -250,6 +256,8 @@ extern const unsigned long {SYMBOL}_size; + + + def clayer(mlops: dict, runtime: list[str], operators: list[str], mlops_file: Path, version: str) -> str: ++ # CMSIS-NN 8.0 builds optional MVE float sources that need both switches. ++ uses_cmsis_nn = any(op.startswith("Cortex-M quantized_") for op in operators) + lines = [ + f"# Generated by create_ai_layer.py from {mlops_file.name} -- do not edit.", + f"# Re-run `python create_ai_layer.py {mlops_file.name}` after changing", +@@ -263,6 +271,11 @@ def clayer(mlops: dict, runtime: list[str], operators: list[str], mlops_file: Pa + "", + " define:", + " - ET_LOG_ENABLED: 0", ++ *( ++ [" - ARM_NN_ENABLE_F32: 1", " - ARM_NN_ENABLE_F16: 1"] ++ if uses_cmsis_nn ++ else [] ++ ), + "", + " add-path:", + " - .", +@@ -270,6 +283,7 @@ def clayer(mlops: dict, runtime: list[str], operators: list[str], mlops_file: Pa + " components:", + *[f" - component: Machine Learning:ExecuTorch:{c}" for c in runtime], + *[f" - component: Machine Learning:ExecuTorch Operators:{c}" for c in operators], ++ *([" - component: ARM::CMSIS:NN Lib"] if uses_cmsis_nn else []), + "", + " groups:", + f" - group: {mlops['model'].get('name', 'Model')}", diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/export_cortex_m_fallback.py b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/export_cortex_m_fallback.py index 2898bdb77d..39aa9a36a8 100644 --- a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/export_cortex_m_fallback.py +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/export_cortex_m_fallback.py @@ -2,50 +2,27 @@ from __future__ import annotations import argparse +import importlib.util import json from pathlib import Path +import sys from typing import Any -class ThreePathModel: - def __new__(cls) -> Any: - import torch - - class Model(torch.nn.Module): - def __init__(self) -> None: - super().__init__() - self.conv1 = torch.nn.Conv2d( - in_channels=3, - out_channels=4, - kernel_size=3, - stride=1, - padding=1, - bias=False, - ) - self.conv2 = torch.nn.Conv2d( - in_channels=4, - out_channels=1, - kernel_size=3, - stride=4, - padding=1, - bias=False, - ) - - def forward(self, inputs: torch.Tensor) -> torch.Tensor: - values, _ = torch.topk( - self.conv2(torch.relu(self.conv1(inputs))), - k=1, - dim=-1, - ) - return values - - return Model() +def load_model_module(path: Path) -> Any: + spec = importlib.util.spec_from_file_location("lp_model", path) + if spec is None or spec.loader is None: + raise SystemExit(f"Cannot load model from {path}") + module = importlib.util.module_from_spec(spec) + sys.modules[spec.name] = module + spec.loader.exec_module(module) + return module def main() -> None: parser = argparse.ArgumentParser( description=( - "Export a Cortex-M baseline or a partially delegated Ethos-U55, " + "Export a Cortex-M baseline or a partially delegated Ethos-U, " "CMSIS-NN, and portable-kernel model." ) ) @@ -53,7 +30,7 @@ def main() -> None: "--mode", choices=("cortex-m", "ethos-u-fallback"), required=True, - help="Select Cortex-M-only lowering or Ethos-U55 followed by Cortex-M lowering.", + help="Select Cortex-M-only lowering or Ethos-U followed by Cortex-M lowering.", ) parser.add_argument( "--target", @@ -66,6 +43,12 @@ def main() -> None: default=Path("artifacts"), help="Directory for the bundled program and graph summary.", ) + parser.add_argument( + "--model-file", + type=Path, + default=Path("model/model.py"), + help="Model module providing get_model() and get_calibration_inputs().", + ) args = parser.parse_args() import torch @@ -81,14 +64,12 @@ def main() -> None: from executorch.extension.export_util.utils import save_pte_program from torchao.quantization.pt2e.quantize_pt2e import convert_pt2e, prepare_pt2e - torch.manual_seed(0) - model = ThreePathModel().eval().to(memory_format=torch.channels_last) - example_inputs = ( - torch.ones(1, 3, 8, 8, dtype=torch.float32).to( - memory_format=torch.channels_last - ), - ) + model_module = load_model_module(args.model_file.resolve()) + model = model_module.get_model() + calibration_inputs = model_module.get_calibration_inputs() + example_inputs = (calibration_inputs[0],) exported_program = torch.export.export(model, example_inputs) + captured_graph = exported_program.module(check_guards=False) if args.mode == "cortex-m": from executorch.backends.cortex_m.quantizer.quantizer import CortexMQuantizer @@ -105,8 +86,9 @@ def main() -> None: quantizer = EthosUQuantizer(compile_spec) quantizer.set_global(get_symmetric_quantization_config(is_per_channel=True)) - prepared = prepare_pt2e(exported_program.module(), quantizer) - prepared(*example_inputs) + prepared = prepare_pt2e(captured_graph, quantizer) + for calibration_input in calibration_inputs: + prepared(calibration_input) quantized_model = convert_pt2e(prepared) quantized_exported_program = torch.export.export(quantized_model, example_inputs) reference_output = quantized_exported_program.module()(*example_inputs) @@ -130,7 +112,7 @@ def main() -> None: ) edge_program_manager._edge_programs["forward"] = CortexMPassManager( - edge_program_manager.exported_program() + edge_program_manager.exported_program(), use_explicit_layout=True ).transform() targets_after_cortex_m = count_targets( diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/mixed_backend_cnn.py b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/mixed_backend_cnn.py new file mode 100644 index 0000000000..a28427f789 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/mixed_backend_cnn.py @@ -0,0 +1,58 @@ +# Copyright 2026 Arm Limited and/or its affiliates. +# SPDX-License-Identifier: Apache-2.0 +"""TinyCNN variant with Ethos-U, CMSIS-NN, and portable execution paths.""" + +import torch +from torch import nn + +# NHWC, the layout of a camera frame, which the Ethos-U reads as it is. An NCHW +# input costs one more NPU operation that reorders it before the first +# convolution. +INPUT_SHAPE = (1, 16, 16, 3) + + +class TinyCNN(nn.Module): + def __init__(self, input_shape: tuple[int, ...] = INPUT_SHAPE, num_classes: int = 10) -> None: + super().__init__() + channels = input_shape[-1] + self.conv1 = nn.Conv2d( + channels, 4, kernel_size=3, stride=1, padding=1, bias=False + ) + self.conv2 = nn.Conv2d( + 4, num_classes, kernel_size=3, stride=4, padding=1, bias=False + ) + + def forward(self, inputs: torch.Tensor) -> torch.Tensor: + # Camera input is NHWC; convolutions use NCHW. + values = inputs.permute(0, 3, 1, 2) + values = torch.relu(self.conv1(values)) + values = self.conv2(values) + + # Keep one maximum spatial response for each of the ten output channels. + values = torch.flatten(values, start_dim=2) + values, _ = torch.topk(values, k=1, dim=-1) + return values.squeeze(-1) + + +def get_model(input_shape: tuple[int, ...] = INPUT_SHAPE) -> nn.Module: + # Fixed seed: the example uses untrained (random) weights, and a fixed seed + # keeps the exported program identical from one export to the next (the + # C array also embeds the Vela options, so the same working directory is + # part of that guarantee). + torch.manual_seed(0) + return TinyCNN(input_shape).eval() + + +def get_calibration_inputs( + input_shape: tuple[int, ...] = INPUT_SHAPE, calibration_samples: int = 2 +) -> list[torch.Tensor]: + """Float samples for quantization calibration; the first one is also the export example. + + Fixed values keep the quantization parameters, and with them the output + logits in the README, the same from one export to the next. Replace them + with representative, preprocessed data when calibrating a trained model. + """ + samples = [torch.full(input_shape, 0.5), torch.full(input_shape, -0.5)][:calibration_samples] + for i in range(len(samples), calibration_samples): + samples.append(torch.randn(input_shape, generator=torch.Generator().manual_seed(i))) + return samples diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/verify_pack_sources.py b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/verify_pack_sources.py index 07dbd2f791..4c5bf84d84 100644 --- a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/verify_pack_sources.py +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/verify_pack_sources.py @@ -6,12 +6,33 @@ from pathlib import Path -EXPECTED_SOURCES = { - "ExecuTorch runtime": "src/runtime/executor/program.cpp", - "CMSIS-NN Cortex-M convolution": "src/backends/cortex_m/ops/op_quantized_conv2d.cpp", - "Ethos-U backend": "src/backends/arm/runtime/EthosUBackend.cpp", - "Portable topk": "src/kernels/portable/cpu/op_topk.cpp", -} +EXPECTED_SOURCES = ( + ( + "ExecuTorch runtime", + "src/runtime/executor/program.cpp", + ("/PyTorch/ExecuTorch/1.5.1/", "PyTorch_Machine_Learning_ExecuTorch_"), + ), + ( + "ExecuTorch Cortex-M convolution wrapper", + "src/backends/cortex_m/ops/op_quantized_conv2d.cpp", + ("/PyTorch/ExecuTorch/1.5.1/", "PyTorch_Machine_Learning_ExecuTorch_"), + ), + ( + "CMSIS-NN convolution kernel", + "Source/ConvolutionFunctions/arm_convolve_s8.c", + ("/ARM/CMSIS-NN/8.0.0/", "ARM_CMSIS_NN_Lib_8_0_0"), + ), + ( + "Ethos-U backend", + "src/backends/arm/runtime/EthosUBackend.cpp", + ("/PyTorch/ExecuTorch/1.5.1/", "PyTorch_Machine_Learning_ExecuTorch_"), + ), + ( + "ExecuTorch portable topk kernel", + "src/kernels/portable/cpu/op_topk.cpp", + ("/PyTorch/ExecuTorch/1.5.1/", "PyTorch_Machine_Learning_ExecuTorch_"), + ), +) def main() -> None: @@ -20,17 +41,21 @@ def main() -> None: args = parser.parse_args() entries = json.loads(args.compile_database.read_text()) - source_files = { - entry["file"].replace("\\", "/") - for entry in entries - if "file" in entry - } - missing = [] - for label, source_path in EXPECTED_SOURCES.items(): - pack_source = f"/PyTorch/ExecuTorch/1.4.0/{source_path}" - if any(source_file.endswith(pack_source) for source_file in source_files): - print(f"PASS: {label} came from the ExecuTorch CMSIS Pack") + for label, source_path, pack_markers in EXPECTED_SOURCES: + matched = False + for entry in entries: + source_file = entry.get("file", "").replace("\\", "/") + command = entry.get("command", " ".join(entry.get("arguments", []))).replace( + "\\", "/" + ) + if source_file.endswith(source_path) and any( + marker in f"{source_file} {command}" for marker in pack_markers + ): + matched = True + break + if matched: + print(f"PASS: {label} was compiled from its declared CMSIS Pack") else: missing.append(label) From 384b06ee472775641aa227f15bfa55922ab8510c Mon Sep 17 00:00:00 2001 From: Kwashie Andoh Date: Tue, 3 Nov 2026 15:14:21 +0000 Subject: [PATCH 3/6] new changes --- .../1-prepare.md | 52 ++- .../2-run-cortex-m.md | 35 +- .../3-add-ethosu-fallback.md | 43 +- .../4-inspect-cmsis-pack.md | 34 +- .../5-prepare-cmsis-runner.md | 23 +- .../6-consume-cmsis-pack.md | 11 +- .../files/enable_cortex_m_fallback.patch | 7 +- .../images/cortex-m-execution-picture.svg | 369 ++++++++++++++++++ .../images/executorch-export-flow.svg | 76 ++++ 9 files changed, 540 insertions(+), 110 deletions(-) create mode 100644 content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/images/cortex-m-execution-picture.svg create mode 100644 content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/images/executorch-export-flow.svg diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/1-prepare.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/1-prepare.md index ff4b88cf32..069ced99e6 100644 --- a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/1-prepare.md +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/1-prepare.md @@ -9,12 +9,11 @@ layout: learningpathall ## Understand the execution plan -You will start with a small PyTorch model and export it as an ExecuTorch -program (`.pte`). +This learning path involves exporting a small PyTorch model into an ExecuTorch +program (`.pte`). The CMSIS project then embeds the PTE bytes in the firmware as a C array. -On the target, the Cortex-M CPU runs the ExecuTorch runtime and coordinates -the whole program. The CPU reads the `.pte` and dispatches different parts -to either the Ethos-U NPU or to CPU kernels. You will use the ExecuTorch ahead-of-time export flow to divide one model across +On the target, the Cortex-M CPU runs the ExecuTorch runtime and loads the embedded program. It then dispatches different parts of the operation +to either the Ethos-U NPU or to CPU kernels. In this learning path, you will use the ExecuTorch ahead-of-time export flow to divide one model across three execution paths: - A stride-one convolution that Ethos-U55 accepts. @@ -23,19 +22,21 @@ three execution paths: After Ethos-U partitioning and Cortex-M lowering, the graph follows this path: -```text + + +![Diagram of the simulated Corstone-300 FVP target showing the Cortex-M CPU running the ExecuTorch runtime and dispatching a stride-one convolution to Ethos-U55, while CMSIS-NN executes the stride-four convolution and a portable kernel executes topk on Cortex-M#center](images/cortex-m-execution-picture.svg "ExecuTorch execution paths") The three execution paths have different roles: | Technology | Role in this Learning Path | Expected performance | | --- | --- | --- | -| Ethos-U | Runs supported graph regions on the NPU. (eg: Stride-1 convolution) | Expected to provide the best performance and energy efficiency for supported neural-network operations. | +| [Ethos-U55](https://www.arm.com/products/silicon-ip-cpu/ethos/ethos-u55) | Runs supported graph regions on the NPU. (eg: Stride-1 convolution) | Expected to provide the best performance and energy efficiency for supported neural-network operations. | | [CMSIS-NN](https://arm-software.github.io/CMSIS-NN/latest/index.html) | Supplies optimized Cortex-M kernels for quantized CPU operations that are unsupported by the NPU. (eg: Stride-4 convolution) | Preferred CPU fallback when a matching CMSIS-NN-backed operator is available. | | ExecuTorch portable kernels | Supply general CPU implementations for operations without an accelerated backend implementation. (eg: `topk` operation) | Compatibility fallback rather than the preferred performance path. | @@ -50,18 +51,18 @@ so it will typically have less CPU fallback than U55. The same workflow still applies when a model contains operations that remain outside the U85 delegate. {{% /notice %}} -In this Learning Path, you will clone a CMSIS-Executorch example that already contains the needed board support, +In this Learning Path, you will clone a [CMSIS-Executorch example](https://github.com/Arm-Examples/cmsis-executorch) that already contains the needed board support, runner, memory configuration, and Fixed Virtual Platform (FVP) settings. You -will then inspect and modify that to observe Cortex-M fallback using optimized CMSIS-NN microkernels. +will then inspect and modify it to observe Cortex-M fallback using optimized CMSIS-NN microkernels. ## Clone the example project -Clone the CMSIS-Executorch project used by this Learning Path: +Begin by cloning the CMSIS-Executorch project used by this Learning Path: ```console -git clone --branch add-corstone-300-u55 https://github.com/Kwash45/CMSIS-Executorch.git +git clone https://github.com/Arm-Examples/CMSIS-Executorch.git cd CMSIS-Executorch -git checkout 173c251496446b8f564c178c415e6f7f28081070 +git checkout 4002b63dcc3e27eb7444946a0dbf69b9a74d6fc9 ``` The explicit checkout pins the project to a revision that has been tested and validated. @@ -103,17 +104,19 @@ activate the free, non-commercial Keil MDK Community license: armlm activate -product KEMDK-COM0 -server https://mdk-preview.keil.arm.com ``` -## Create the Python environment +## Create and activate the Python environment -Use the repository's setup wrapper to create `.venv` and install its pinned -export dependencies: +Use the repository's setup wrapper to create `.venv`, install its pinned +export dependencies, and activate it in the current terminal: {{< tabpane code=true >}} {{< tab header="Linux" language="bash">}} ./setup_venv.sh +source .venv/bin/activate {{< /tab >}} {{< tab header="Windows PowerShell" language="powershell">}} .\setup_venv.bat +.\.venv\Scripts\Activate.ps1 {{< /tab >}} {{< /tabpane >}} @@ -125,23 +128,21 @@ build tools, and Python packages: {{< tabpane code=true >}} {{< tab header="Linux" language="bash">}} -git --version -python3 --version +python --version cbuild --version armclang --version -.venv/bin/python -c "from importlib.metadata import version; print('ExecuTorch', version('executorch')); print('PyTorch', version('torch'))" +python -c "from importlib.metadata import version; print('ExecuTorch', version('executorch')); print('PyTorch', version('torch'))" {{< /tab >}} {{< tab header="Windows PowerShell" language="powershell">}} -git --version python --version cbuild --version armclang --version -.\.venv\Scripts\python.exe -c "from importlib.metadata import version; print('ExecuTorch', version('executorch')); print('PyTorch', version('torch'))" +python -c "from importlib.metadata import version; print('ExecuTorch', version('executorch')); print('PyTorch', version('torch'))" {{< /tab >}} {{< /tabpane >}} -Your Git version can vary. Confirm that Python is between 3.10 and 3.13, -`cbuild` reports 2.14.0, Arm Compiler for Embedded reports 6.24, and the final +Confirm that Python is between 3.10 and 3.13, +`cbuild` reports 2.15.0, Arm Compiler for Embedded reports 6.24, and the final two lines are: ```output @@ -178,8 +179,5 @@ Each file will be explained in greater detail in subsequent pages. ## What you've accomplished and what's next You now have a pinned Corstone-300 U55 project, its matching Python packages, -and its versioned Arm tools. The project already owns the board and application -integration that previously needed many manual setup steps. - -Next, you will establish a CPU-only baseline before adding +and its versioned Arm tools. Next, you will establish a CPU-only baseline before adding Ethos-U delegation on the U55 demonstration target. diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/2-run-cortex-m.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/2-run-cortex-m.md index 17f5c0c3dc..501f12a712 100644 --- a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/2-run-cortex-m.md +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/2-run-cortex-m.md @@ -53,18 +53,18 @@ each of the ten output channels. ## Export the Cortex-M baseline -Before we demonstrate fallback, it is useful to know which operations could be optimized by the Cortex-M backend. +Before we demonstrate fallback, it is useful to know which operations could be optimized by the Cortex-M backend itself. Therefore, as a useful first step, we will specify `--mode cortex-m`, such that no operation is offered to the NPU. Run the diagnostic exporter from the repository root: {{< tabpane code=true >}} {{< tab header="Linux" language="bash">}} -.venv/bin/python export_cortex_m_fallback.py \ +python export_cortex_m_fallback.py \ --mode cortex-m \ --model-file model/model.py \ --output-dir artifacts {{< /tab >}} {{< tab header="Windows PowerShell" language="powershell">}} -.\.venv\Scripts\python.exe .\export_cortex_m_fallback.py ` +python .\export_cortex_m_fallback.py ` --mode cortex-m ` --model-file model/model.py ` --output-dir artifacts @@ -86,18 +86,29 @@ cortex_m.quantized_conv2d_nhwc.default: 2 cortex_m.transpose.default: 1 ``` -Before lowering, the graph contained two general convolution operations (Stride-1 and Stride-4). After -lowering, both become `cortex_m.quantized_conv2d_nhwc` operations. The `_nhwc` -suffix identifies the explicit channel-last layout used by the Cortex-M -operator. +Before lowering, the graph contained two general convolution operations (Stride-1 and Stride-4). The `aten` prefix identifies PyTorch’s ATen operator namespace. These operators describe the computation, but not the hardware or kernel that will perform it. Lowering replaces both with `cortex_m.quantized_conv2d_nhwc`, an INT8 convolution using NHWC layout. `aten.topk` is unchanged, so it will use the portable CPU kernel. The `cortex_m.quantize_*` -and `cortex_m.dequantize_*` operators mark boundaries between floating-point and INT8 -regions. They are real work in the final graph, not annotation-only metadata. +and `cortex_m.dequantize_*` operators mark boundaries between floating-point (FP32) and INT8 +regions. -The helper writes `artifacts/stride4_cortex_m.pte` and a JSON summary. This PTE -is an inspection baseline; the project generator will create the program that -is embedded in the final firmware. +Confirm the generated artifacts from the export script: + +{{< tabpane code=true >}} + {{< tab header="Linux" language="bash">}} +ls -lh artifacts/stride4_cortex_m.* + {{< /tab >}} + {{< tab header="Windows PowerShell" language="powershell">}} +Get-ChildItem .\artifacts\stride4_cortex_m.* + {{< /tab >}} +{{< /tabpane >}} + +The expected output contains: + +```output +stride4_cortex_m.json +stride4_cortex_m.pte +``` ## What you've accomplished and what's next diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/3-add-ethosu-fallback.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/3-add-ethosu-fallback.md index 9b17a09d59..b143207f6b 100644 --- a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/3-add-ethosu-fallback.md +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/3-add-ethosu-fallback.md @@ -9,22 +9,11 @@ layout: learningpathall ## Partition the same model for Ethos-U -The mixed export changes the order of decisions, not the model. It first gives +The mixed export changes the order of decisions. It first gives `EthosUPartitioner` the opportunity to claim supported graph regions. It then applies Cortex-M lowering only to the operations left outside the delegate: -```text -quantize the graph - | - v -partition supported work for Ethos-U - | - v -lower supported remainder to Cortex-M operators - | - v -leave the final remainder as portable operations -``` +![Flow diagram showing that the graph is quantized, supported work is partitioned for Ethos-U, the supported remainder is lowered to Cortex-M operators, and the final remainder stays portable. This order preserves eligible NPU work before CPU lowering.#center](images/executorch-export-flow.svg "ExecuTorch partitioning and lowering order") Ordering matters. If Cortex-M lowering ran first, it could rewrite a convolution that should have been offered to the NPU. @@ -37,14 +26,14 @@ Run the diagnostic exporter in mixed mode: {{< tabpane code=true >}} {{< tab header="Linux" language="bash">}} -.venv/bin/python export_cortex_m_fallback.py \ +python export_cortex_m_fallback.py \ --mode ethos-u-fallback \ --target ethos-u55-128 \ --model-file model/model.py \ --output-dir artifacts {{< /tab >}} {{< tab header="Windows PowerShell" language="powershell">}} -.\.venv\Scripts\python.exe .\export_cortex_m_fallback.py ` +python .\export_cortex_m_fallback.py ` --mode ethos-u-fallback ` --target ethos-u55-128 ` --model-file model/model.py ` @@ -52,8 +41,7 @@ Run the diagnostic exporter in mixed mode: {{< /tab >}} {{< /tabpane >}} -Vela prints a summary for the region it compiles. The graph inspection at the -end shows the complete program: +Vela prints a summary for the region it compiles: ```output Relevant targets before Cortex-M lowering: @@ -70,9 +58,9 @@ cortex_m.transpose.default: 2 executorch_call_delegate: 1 ``` -Compare this result with the baseline: +The above summary is interpreted as follows: -| Model work | Final graph target | Execution engine | +| Model work | Final Graph target | Execution engine | | --- | --- | --- | | Stride-one convolution and supported surrounding work | `executorch_call_delegate` | Ethos-U55 | | Stride-four convolution | `cortex_m.quantized_conv2d_nhwc` | Cortex-M55 using CMSIS-NN | @@ -89,13 +77,12 @@ Cortex-M convolution and portable `topk` that remain outside the delegate. ## Connect fallback to the CMSIS project -The PTE you have just produced by runnning the diagnostic exporter is not used by the final CMSIS project. It's simply an inspection result that proves the graph split. The CMSIS-Executorch project uses `create_ai_layer.py` to export the model again, generate the final PTE, and select the source components required to execute it. However, some patches have to be made. The project generator within that project currently handles Ethos-U delegation, but it does not lower the remaining convolution to Cortex-M or select CMSIS-NN. The focused integration patch makes four related changes: -- It declares `ARM::CMSIS-NN@8.0.0` as a project pack. +- It adds `ARM::CMSIS-NN@8.0.0` to the solution's Pack list. - It applies Cortex-M lowering after Ethos-U partitioning. - It maps explicit-layout names such as `quantized_conv2d_nhwc` to their Pack component names. @@ -107,15 +94,14 @@ Check and apply the patch, then inspect its small project diff: ```console git apply --check enable_cortex_m_fallback.patch git apply enable_cortex_m_fallback.patch -git diff -- cmsis-executorch-u55.csolution.yml create_ai_layer.py +git --no-pager diff -- cmsis-executorch-u55.csolution.yml create_ai_layer.py ``` No board, runner, or linker changes are needed. ## Generate the target description -If you opened a new terminal, initialize `vcpkg` and run -`vcpkg-shell activate` again. Then resolve the declared packs and active target: +Resolve the declared Packs and active target: ```console cbuild setup cmsis-executorch-u55.csolution.yml --active SSE-300-U55 --packs @@ -129,19 +115,16 @@ writes three useful descriptions: Vela settings, and destination AI layer. - `cmsis-executorch-u55.cbuild-idx.yml` indexes the resolved build context. -You need `--packs` on a fresh checkout. There is no separate manual Pack -download or `cpackget add` step. - ## Generate the AI layer Use the MLOps description to export the model and write its CMSIS AI layer: {{< tabpane code=true >}} {{< tab header="Linux" language="bash">}} -.venv/bin/python create_ai_layer.py cmsis-executorch-u55.cbuild-mlops.yml +python create_ai_layer.py cmsis-executorch-u55.cbuild-mlops.yml {{< /tab >}} {{< tab header="Windows PowerShell" language="powershell">}} -.\.venv\Scripts\python.exe .\create_ai_layer.py .\cmsis-executorch-u55.cbuild-mlops.yml +python .\create_ai_layer.py .\cmsis-executorch-u55.cbuild-mlops.yml {{< /tab >}} {{< /tabpane >}} @@ -149,7 +132,7 @@ The relevant final output is similar to: ```output [ai_layer] 7608 byte program, operators: ['Cortex-M dequantize_per_tensor', 'Cortex-M quantize_per_tensor', 'Cortex-M quantized_conv2d', 'Cortex-M transpose', 'Portable topk'] -[ai_layer] wrote .../ai_layer/U55/ai_layer.clayer.yml +[ai_layer] wrote .../ai_layer/ai_layer.clayer.yml ``` The generator writes the final `model.pte`, converts it to C data for the diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/4-inspect-cmsis-pack.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/4-inspect-cmsis-pack.md index eb628b5765..2d4acc3e06 100644 --- a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/4-inspect-cmsis-pack.md +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/4-inspect-cmsis-pack.md @@ -7,6 +7,14 @@ description: Trace the mixed ExecuTorch graph to versioned runtime, backend, ope layout: learningpathall --- +## Understand what the CMSIS Pack provides + +The [ExecuTorch CMSIS Pack](https://www.keil.arm.com/packs/executorch-pytorch/overview/) is a *source Pack*, not a prebuilt runtime library. It contains the ExecuTorch sources plus metadata defining selectable components and their dependencies. + +CMSIS-Toolbox then uses that metadata to add only the chosen components to the +application build. Arm Compiler then compiles those sources for the Cortex-M55 +target with the rest of the project. + ## See which Packs were resolved CMSIS-Toolbox has already acquired the software Packs needed by the project. @@ -24,7 +32,7 @@ Get-Content .\cmsis-executorch-u55.cbuild-pack.yml Among the resolved entries, you can see the two Packs responsible for the mixed model: -```yaml +```output - resolved-pack: ARM::CMSIS-NN@8.0.0 - resolved-pack: PyTorch::ExecuTorch@1.5.1 ``` @@ -39,34 +47,18 @@ They have different responsibilities: The Pack lock records exact resolved versions so later builds don't silently move to newer installed Packs. -## Understand what a source Pack provides - -The ExecuTorch CMSIS Pack is a *source Pack*, not a prebuilt runtime library. -Its Pack Description file lists named components, their source files, their -dependencies, and the conditions under which they can be selected. - -CMSIS-Toolbox uses that metadata to add only the chosen components to the -application build. Arm Compiler then compiles those sources for the Cortex-M55 -target with the rest of the project. This is why one published Pack can support -different CMSIS projects without shipping a separate binary runtime for every -target configuration. - -`cbuild setup --packs` stores each installed Pack under the CMSIS Pack root in -a vendor, name, and version hierarchy. Both `cbuild` and other CMSIS tools can -then resolve `PyTorch::ExecuTorch@1.5.1` by its package identity. Manually -downloading the same archive with `curl` would create an unrelated second copy -and wouldn't improve the build. - ## Trace the graph into the AI layer +The AI layer is a generated CMSIS software layer that embeds the model and declares the runtime, backends, operators, and libraries needed to execute it. + Open the AI layer generated from the `.pte` operator list: {{< tabpane code=true >}} {{< tab header="Linux" language="bash">}} -cat ai_layer/U55/ai_layer.clayer.yml +cat ai_layer/ai_layer.clayer.yml {{< /tab >}} {{< tab header="Windows PowerShell" language="powershell">}} -Get-Content .\ai_layer\U55\ai_layer.clayer.yml +Get-Content .\ai_layer\ai_layer.clayer.yml {{< /tab >}} {{< /tabpane >}} diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/5-prepare-cmsis-runner.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/5-prepare-cmsis-runner.md index 7f4db09621..0d7b4aeea3 100644 --- a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/5-prepare-cmsis-runner.md +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/5-prepare-cmsis-runner.md @@ -15,17 +15,16 @@ Open it and inspect: ```yaml target-types: - type: SSE-300-U55 - board: ARM::V2M-MPS3-SSE-300-FVP - device: ARM::SSE-300-MPS3 + board: V2M-MPS3-SSE-300-FVP + device: SSE-300-MPS3 variables: - Board-Layer: $SolutionDir()$/board/Corstone-300/Board-U55.clayer.yml - - AI-Layer: $SolutionDir()$/ai_layer/U55/ai_layer.clayer.yml + - AI-Layer: $SolutionDir()$/ai_layer/ai_layer.clayer.yml ``` -`SSE-300-U55` is the project's readable target name. The `ARM::` values identify -specific board and device records supplied by installed Packs. CMSIS-Toolbox -uses those identities to resolve startup files, memory information, and device -conditions. +`SSE-300-U55` is the project's readable target name. The board and device +values identify records supplied by installed Packs. CMSIS-Toolbox uses those +identities to resolve startup files, memory information, and device conditions. The solution's `mlops:` node is the single source for the NPU and Vela target settings: @@ -41,6 +40,8 @@ mlops: model: clayer: $AI-Layer$ name: TinyCNN + input-shape: 1x16x16x3 + calibration-samples: 2 ``` `cbuild setup` resolves this into the `*.cbuild-mlops.yml` file consumed by @@ -75,7 +76,7 @@ Each part has one job: - The App group initializes ExecuTorch, creates an input tensor, runs one inference, and prints the ten output values. -The FVP configuration in `board/Corstone-300/fvp_config.txt` describes the +The FVP configuration in `board/Corstone-300/fvp_config_u55.txt` describes the simulated hardware. It is separate from the Board layer because one configures the simulator while the other builds software for the simulated target. @@ -84,9 +85,9 @@ the simulator while the other builds software for the simulated target. The AI-layer directory now contains these related model files: ```text -ai_layer/U55/model.pte -ai_layer/U55/model_pte.c -ai_layer/U55/model_pte.h +ai_layer/model.pte +ai_layer/model_pte.c +ai_layer/model_pte.h ``` `model.pte` is the serialized ExecuTorch program. `model_pte.c` contains the diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/6-consume-cmsis-pack.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/6-consume-cmsis-pack.md index c63abb1a84..9a822ecab7 100644 --- a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/6-consume-cmsis-pack.md +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/6-consume-cmsis-pack.md @@ -40,16 +40,16 @@ against the compilation database: {{< tabpane code=true >}} {{< tab header="Linux" language="bash">}} -.venv/bin/python verify_pack_sources.py \ +python verify_pack_sources.py \ out/cmsis-executorch/SSE-300-U55/Debug/compile_commands.json {{< /tab >}} {{< tab header="Windows PowerShell" language="powershell">}} -.\.venv\Scripts\python.exe .\verify_pack_sources.py ` +python .\verify_pack_sources.py ` .\out\cmsis-executorch\SSE-300-U55\Debug\compile_commands.json {{< /tab >}} {{< /tabpane >}} -The expected output is: +The expected output is: ```output PASS: ExecuTorch runtime was compiled from its declared CMSIS Pack @@ -71,7 +71,7 @@ shows both the resolved Pack versions and the source files actually compiled. Start the Corstone-300 model and load the generated HEX image: ```console -FVP_Corstone_SSE-300_Ethos-U55 -f board/Corstone-300/fvp_config.txt --simlimit 60 -a cpu0=out/cmsis-executorch/SSE-300-U55/Debug/cmsis-executorch.hex +FVP_Corstone_SSE-300_Ethos-U55 -f board/Corstone-300/fvp_config_u55.txt -a out/cmsis-executorch/SSE-300-U55/Debug/cmsis-executorch.hex --simlimit 60 ``` The relevant output is: @@ -81,10 +81,11 @@ Ethos-U version info: Arch: v1.1.0 MACs/cc: 128 Cmd stream: v0 -ExecuTorch Ethos-U example: 7608 byte model +ExecuTorch Ethos-U55-128 application: 7608 byte model Output: 10 element(s): 0.0532 -0.0239 0.0541 0.1064 0.0706 0.0395 0.0881 0.1340 0.0807 0.0064 Test_result: PASS ``` + {{% notice Note %}} On Windows, the FVP can print warnings that `python3.dll` is unavailable and that the VIO or VSI bridges are disabled. This headless example doesn't use diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/enable_cortex_m_fallback.patch b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/enable_cortex_m_fallback.patch index f3fa9d0fa2..9738f53935 100644 --- a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/enable_cortex_m_fallback.patch +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/enable_cortex_m_fallback.patch @@ -1,12 +1,11 @@ diff --git a/cmsis-executorch-u55.csolution.yml b/cmsis-executorch-u55.csolution.yml --- a/cmsis-executorch-u55.csolution.yml +++ b/cmsis-executorch-u55.csolution.yml -@@ -21,5 +21,6 @@ solution: - # to load. `cbuild setup --packs` fetches it from the public index. +@@ -20,4 +20,5 @@ solution: + # The PyTorch::ExecuTorch pack version must match the ExecuTorch Python + # exporter in requirements.txt because its runtime consumes the generated .pte. - pack: PyTorch::ExecuTorch@1.5.1 - - pack: ARM::CMSIS@6.3.0 + - pack: ARM::CMSIS-NN@8.0.0 - - pack: ARM::CMSIS-Compiler - pack: ARM::Cortex_DFP diff --git a/create_ai_layer.py b/create_ai_layer.py --- a/create_ai_layer.py diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/images/cortex-m-execution-picture.svg b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/images/cortex-m-execution-picture.svg new file mode 100644 index 0000000000..1b1aa5e2a0 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/images/cortex-m-execution-picture.svg @@ -0,0 +1,369 @@ + + + One program. Three execution paths. + A simplified view of the simulated Corstone-300 target in the FVP. The Cortex-M CPU runs the ExecuTorch runtime and executes model.pte. It dispatches the stride-one convolution to the separate Ethos-U55 NPU. On the same Cortex-M core, CMSIS-NN runs the stride-four convolution and the portable FP32 kernel runs topk. CPU and NPU both connect to shared SRAM. The connections are schematic and execution routes are chosen during export. + Architecture references: https://arm-software.github.io/CMSIS-Ethos-U/main/general/index.html ; https://docs.pytorch.org/executorch/stable/backends/arm-cortex-m/arm-cortex-m-overview.html + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Target + + + + + + Cortex-M CPU + ExecuTorch runtime + model.pte + + + 2 + + 3 + + + + Ethos-U55 NPU + + + 1 + Stride-1 convolution + Stride-4 convolution + topk operation + Supported by CMSIS-NN + Supported by Executorch Portable kernels + Dispatch + + + + + Shared SRAM + Input, intermediate tensors, output + + diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/images/executorch-export-flow.svg b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/images/executorch-export-flow.svg new file mode 100644 index 0000000000..2af601cb4f --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/images/executorch-export-flow.svg @@ -0,0 +1,76 @@ + + + ExecuTorch export and lowering flow + A four-stage flow from quantizing the graph, through Ethos-U partitioning and Cortex-M lowering, to the portable operations that remain. + + + + + + + + + + + + + Quantize the graph + + + + + + Ethos-U partition + + + + + + Cortex-M lowering + + + + + + + Portable operations + remain + + From 8b1b47e0d778aabdf6065851592584ebbc75493d Mon Sep 17 00:00:00 2001 From: Matt Cossins Date: Thu, 5 Nov 2026 13:29:48 +0000 Subject: [PATCH 4/6] initial changes --- .../1-prepare.md | 150 +++++++++++------- .../2-run-cortex-m.md | 113 ++++++------- .../3-add-ethosu-fallback.md | 90 +++++++---- .../4-inspect-cmsis-pack.md | 16 +- .../5-prepare-cmsis-runner.md | 2 +- .../6-consume-cmsis-pack.md | 31 ++-- .../_index.md | 21 +-- 7 files changed, 243 insertions(+), 180 deletions(-) diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/1-prepare.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/1-prepare.md index 069ced99e6..c238252bc9 100644 --- a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/1-prepare.md +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/1-prepare.md @@ -1,5 +1,5 @@ --- -title: Prepare the Corstone-300 U55 project +title: Understand and prepare the environment weight: 2 description: Set up the reusable CMSIS-Executorch project and its pinned Python and Arm tool environments. @@ -7,28 +7,26 @@ description: Set up the reusable CMSIS-Executorch project and its pinned Python layout: learningpathall --- -## Understand the execution plan +## ExecuTorch for Cortex-M and Ethos-U -This learning path involves exporting a small PyTorch model into an ExecuTorch -program (`.pte`). The CMSIS project then embeds the PTE bytes in the firmware as a C array. +ExecuTorch is PyTorch’s runtime for deploying machine learning models on edge devices, including embedded systems. It converts trained models into programs that can run within the limited memory and compute resources of a microcontroller. -On the target, the Cortex-M CPU runs the ExecuTorch runtime and loads the embedded program. It then dispatches different parts of the operation -to either the Ethos-U NPU or to CPU kernels. In this learning path, you will use the ExecuTorch ahead-of-time export flow to divide one model across +Many microcontrollers use Arm Cortex-M CPUs, and others integrate both Cortex-M CPUs, and Arm Ethos-U NPUs. + +For combined Cortex-M and Ethos-U systems, ExecuTorch lets one model use both processors: supported operations are dispatched to the Ethos-U NPU, while remaining operations run on the Cortex-M CPU. + +This Learning Path will demonstrate how developers can better optimize ExecuTorch deployment, either when running models directly on a Cortex-M CPU, or in a scenario where unsupported operators fallback from an Ethos-U to the Cortex-M CPU. + +## Ethos-U vs CMSIS-NN vs Portable Kernels + +You will use the ExecuTorch ahead-of-time export flow to divide one model across three execution paths: - A stride-one convolution that Ethos-U55 accepts. - A stride-four convolution that Ethos-U55 rejects but CMSIS-NN supports. - A `topk` operation that remains an ExecuTorch portable FP32 operation. -After Ethos-U partitioning and Cortex-M lowering, the graph follows this path: - - +This is illustrated in the following diagram: ![Diagram of the simulated Corstone-300 FVP target showing the Cortex-M CPU running the ExecuTorch runtime and dispatching a stride-one convolution to Ethos-U55, while CMSIS-NN executes the stride-four convolution and a portable kernel executes topk on Cortex-M#center](images/cortex-m-execution-picture.svg "ExecuTorch execution paths") @@ -47,7 +45,7 @@ supports, then assigns known remaining patterns to the Cortex-M backend. {{% notice Why this example uses Ethos-U55 %}} Ethos-U55 is used because its unsupported stride-four convolution creates a predictable fallback region. Ethos-U85 supports more operator configurations, -so it will typically have less CPU fallback than U55. The same workflow still +so it will typically have less CPU fallback than Ethos-U55. The same workflow still applies when a model contains operations that remain outside the U85 delegate. {{% /notice %}} @@ -67,58 +65,50 @@ git checkout 4002b63dcc3e27eb7444946a0dbf69b9a74d6fc9 The explicit checkout pins the project to a revision that has been tested and validated. -## Install vcpkg and Activate the project tools +## Set up the project in VS Code -The repository's `vcpkg-configuration.json` declares CMSIS-Toolbox, Arm -Compiler for Embedded, and the Arm FVP. Install `vcpkg` with the command for your host: +Install [Visual Studio Code](https://code.visualstudio.com/), the +[Keil Studio Pack](https://marketplace.visualstudio.com/items?itemName=Arm.keil-studio-pack), +and the [Python extension](https://marketplace.visualstudio.com/items?itemName=ms-python.python). +Install Python 3.10–3.13 on your host; Python 3.14 is not supported by the +pinned export dependencies. On Windows, enable +[long-path support](https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation). -{{< tabpane code=true >}} - {{< tab header="Linux" language="bash">}} -. <(curl https://github.com/microsoft/vcpkg-tool/releases/download/2026-04-08/vcpkg-init.sh -L) - {{< /tab >}} - {{< tab header="Windows PowerShell" language="powershell">}} -iex (iwr -useb https://github.com/microsoft/vcpkg-tool/releases/download/2026-04-08/vcpkg-init.ps1) - {{< /tab >}} -{{< /tabpane >}} +1. Open the cloned `CMSIS-Executorch` folder in VS Code. +2. Allow Arm Environment Manager to install the tools declared in + `vcpkg-configuration.json` and wait for installation to finish. +3. In the **CMSIS** view, select **... > Open Solution in Workspace** + and choose `cmsis-executorch-u55.csolution.yml`. +4. Select **Terminal > Run Task > Create Python environment (pip)**. Wait for the + task to create `.venv` and install the pinned dependencies. +5. Open a new VS Code terminal in the project folder so it receives the Arm tool + environment. Run all subsequent commands in this terminal. -Initialize `vcpkg`, then activate it from the repository root. Repeat these commands in each new terminal you open. +Activate the Python environment in each new terminal: {{< tabpane code=true >}} - {{< tab header="Linux" language="bash">}} -. ~/.vcpkg/vcpkg-init -vcpkg-shell activate + {{< tab header="Linux and macOS" language="bash">}} +source .venv/bin/activate {{< /tab >}} {{< tab header="Windows PowerShell" language="powershell">}} -. "$env:USERPROFILE\.vcpkg\vcpkg-init.ps1" -vcpkg-shell activate +.\.venv\Scripts\Activate.ps1 {{< /tab >}} {{< /tabpane >}} -The first activation downloads any missing tools. Later activations reuse the -installed versions. +### Additional macOS setup -If you don't already have a suitable Arm Compiler for Embedded license, -activate the free, non-commercial Keil MDK Community license: +Install and start [Docker Desktop](https://docs.docker.com/desktop/setup/install/mac-install/). +The FVP runs inside a Linux container because there is no native macOS FVP. +Keep the project under your home directory so the wrapper can mount it into +Docker. Confirm Docker is running and select the Ethos-U55 model in each new terminal: -```console -armlm activate -product KEMDK-COM0 -server https://mdk-preview.keil.arm.com +```bash +docker info +export FVP_MODEL=FVP_Corstone_SSE-300_Ethos-U55 ``` -## Create and activate the Python environment - -Use the repository's setup wrapper to create `.venv`, install its pinned -export dependencies, and activate it in the current terminal: - -{{< tabpane code=true >}} - {{< tab header="Linux" language="bash">}} -./setup_venv.sh -source .venv/bin/activate - {{< /tab >}} - {{< tab header="Windows PowerShell" language="powershell">}} -.\setup_venv.bat -.\.venv\Scripts\Activate.ps1 - {{< /tab >}} -{{< /tabpane >}} +The final chapter uses the repository's `.vscode/fvp.sh` wrapper to run the +application on macOS. Its first run builds the container image. ## Verify the environment @@ -127,7 +117,7 @@ runtime loads the program produced by the exporter. Check the host tools, Arm build tools, and Python packages: {{< tabpane code=true >}} - {{< tab header="Linux" language="bash">}} + {{< tab header="Linux and macOS" language="bash">}} python --version cbuild --version armclang --version @@ -156,7 +146,7 @@ Download the fallback model, focused integration patch, and two inspection helpers into the repository root: {{< tabpane code=true >}} - {{< tab header="Linux" language="bash">}} + {{< tab header="Linux and macOS" language="bash">}} LP_ASSETS=https://raw.githubusercontent.com/ArmDeveloperEcosystem/arm-learning-paths/main/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files curl -fLO "$LP_ASSETS/mixed_backend_cnn.py" @@ -174,10 +164,50 @@ curl.exe -fLO "$lpAssets/verify_pack_sources.py" {{< /tab >}} {{< /tabpane >}} -Each file will be explained in greater detail in subsequent pages. +## Use a simple convolution example + +The CMSIS-Executorch repository starts with a TinyCNN model that can be delegated completely to +Ethos-U. You will change this model to one that will demonstrate fallback by retaining its input and output +interface, but adjusting the operations happening inside it. + +Replace the original model with the one you have downloaded: + +{{< tabpane code=true >}} + {{< tab header="Linux and macOS" language="bash">}} +cp mixed_backend_cnn.py model/model.py + {{< /tab >}} + {{< tab header="Windows PowerShell" language="powershell">}} +Copy-Item .\mixed_backend_cnn.py .\model\model.py + {{< /tab >}} +{{< /tabpane >}} + +The model still accepts one `16 x 16` RGB image in NHWC order and returns ten +values. Keeping that interface means the supplied bare-metal runner needs no +changes. + +The important part of `model/model.py` now looks like this: + +```python +self.conv1 = nn.Conv2d( + channels, 4, kernel_size=3, stride=1, padding=1, bias=False +) +self.conv2 = nn.Conv2d( + 4, num_classes, kernel_size=3, stride=4, padding=1, bias=False +) + +values = torch.relu(self.conv1(values)) +values = self.conv2(values) +values = torch.flatten(values, start_dim=2) +values, _ = torch.topk(values, k=1, dim=-1) +``` + +The *stride* controls how far a convolution filter moves between positions. A +stride of one visits adjacent positions. A stride of four moves four positions +at a time and produces a smaller output. For this Ethos-U55 target and tensor shape, +the first convolution can be delegated to the NPU, while the stride-four convolution +remains outside the delegate. `topk` selects the largest spatial response for +each of the ten output channels. ## What you've accomplished and what's next -You now have a pinned Corstone-300 U55 project, its matching Python packages, -and its versioned Arm tools. Next, you will establish a CPU-only baseline before adding -Ethos-U delegation on the U55 demonstration target. +You now have a pinned Corstone-300 Ethos-U55 project, its matching Python packages, its versioned Arm tools, and a model that will only partially run on an Ethos-U55. Next, you will understand how to optimize supported operators with CMSIS-NN on Cortex-M CPUs in a CPU-only example. diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/2-run-cortex-m.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/2-run-cortex-m.md index 501f12a712..6e88fd7bf4 100644 --- a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/2-run-cortex-m.md +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/2-run-cortex-m.md @@ -1,63 +1,70 @@ --- -title: Establish the Cortex-M baseline +title: Lower a model to the Cortex-M backend weight: 3 -description: Adapt TinyCNN and inspect how its convolutions and topk operation run without Ethos-U delegation. +description: Export a Cortex-M model and identify CMSIS-NN accelerated convolutions ### FIXED, DO NOT MODIFY layout: learningpathall --- -## Use a simple convolution example +## First use only Cortex-M -The CMSIS-Executorch repository starts with a TinyCNN model that can be delegated completely to -Ethos-U. To make fallback visible, you will retain its input and output -interface while changing the operations happening inside it. +Before we demonstrate fallback from an Ethos-U, we will briefly look at running a model exclusively on Cortex-M. Not every microcontroller has an onboard NPU, so some use-cases will involve running TinyML directly on the CPU. In this case, any optimizations can be the vital difference between functional performance, and an impractical application. -Replace the original model: +## See how the export script lowers the graph -{{< tabpane code=true >}} - {{< tab header="Linux" language="bash">}} -cp mixed_backend_cnn.py model/model.py - {{< /tab >}} - {{< tab header="Windows PowerShell" language="powershell">}} -Copy-Item .\mixed_backend_cnn.py .\model\model.py - {{< /tab >}} -{{< /tabpane >}} +The exporter first prepares and converts the model with `CortexMQuantizer`: + +```python +quantizer = CortexMQuantizer() + +prepared = prepare_pt2e(exported_program.module(), quantizer) +prepared(*example_inputs) +quantized_model = convert_pt2e(prepared) +quantized_exported_program = torch.export.export(quantized_model, example_inputs) +``` -The model still accepts one `16 x 16` RGB image in NHWC order and returns ten -values. Keeping that interface means the supplied bare-metal runner needs no -changes. +The prepared model is run once with the example input to calibrate activation +ranges. `convert_pt2e()` uses those ranges to choose quantization parameters and +insert the operations that move values between FP32 and INT8. -The important part of `model/model.py` now looks like this: +The exporter converts the quantized program to the ExecuTorch Edge dialect and +applies the Cortex-M pass manager: ```python -self.conv1 = nn.Conv2d( - channels, 4, kernel_size=3, stride=1, padding=1, bias=False -) -self.conv2 = nn.Conv2d( - 4, num_classes, kernel_size=3, stride=4, padding=1, bias=False +edge_program_manager = to_edge( + quantized_exported_program, + compile_config=EdgeCompileConfig(_check_ir_validity=False) ) -values = torch.relu(self.conv1(values)) -values = self.conv2(values) -values = torch.flatten(values, start_dim=2) -values, _ = torch.topk(values, k=1, dim=-1) +edge_program_manager._edge_programs["forward"] = CortexMPassManager( + edge_program_manager.exported_program() +).transform() ``` -The *stride* controls how far a convolution filter moves between positions. A -stride of one visits adjacent positions. A stride of four moves four positions -at a time and produces a smaller output. For this U55 target and tensor shape, -the first convolution can be delegated to the NPU, while the stride-four convolution -remains outside the delegate. `topk` selects the largest spatial response for -each of the ten output channels. +This step rewrites supported patterns as `cortex_m.*` operators. Unsupported +operations, including `aten.topk`, remain in the portable graph. + +The Cortex-M backend is broader than CMSIS-NN. In this model, the quantized +convolutions use CMSIS-NN, while quantize and dequantize use Cortex-M operators +with Arm Helium vector implementations. A `cortex_m.*` target therefore means +that the Cortex-M backend owns the operator, not necessarily that CMSIS-NN +implements it. + +{{% notice Important %}} +The Cortex-M backend is not a delegate. Delegation replaces a graph region with +a call to a separately compiled backend. Cortex-M lowering instead rewrites +supported patterns as registered `cortex_m.*` operators that the ExecuTorch +runtime dispatches normally. You will not see an `executorch_call_delegate` +node until the Ethos-U partitioner is added on the next page. +{{% /notice %}} -## Export the Cortex-M baseline +## Export the model for Cortex-M -Before we demonstrate fallback, it is useful to know which operations could be optimized by the Cortex-M backend itself. -Therefore, as a useful first step, we will specify `--mode cortex-m`, such that no operation is offered to the NPU. Run the diagnostic exporter from the repository root: +Use the provided script to lower the model to Cortex-M only, by specifying `-mode cortex-m`: {{< tabpane code=true >}} - {{< tab header="Linux" language="bash">}} + {{< tab header="Linux and macOS" language="bash">}} python export_cortex_m_fallback.py \ --mode cortex-m \ --model-file model/model.py \ @@ -92,29 +99,13 @@ Before lowering, the graph contained two general convolution operations (Stride- and `cortex_m.dequantize_*` operators mark boundaries between floating-point (FP32) and INT8 regions. -Confirm the generated artifacts from the export script: - -{{< tabpane code=true >}} - {{< tab header="Linux" language="bash">}} -ls -lh artifacts/stride4_cortex_m.* - {{< /tab >}} - {{< tab header="Windows PowerShell" language="powershell">}} -Get-ChildItem .\artifacts\stride4_cortex_m.* - {{< /tab >}} -{{< /tabpane >}} - -The expected output contains: - -```output -stride4_cortex_m.json -stride4_cortex_m.pte -``` - ## What you've accomplished and what's next -You have separated two CPU execution routes without involving the NPU. Both -convolutions use Cortex-M operators backed by CMSIS-NN, while `topk` remains a -portable ExecuTorch operation. +You have seen how Cortex-M lowering replaces supported patterns with optimized +Cortex-M operators. The convolutions use CMSIS-NN, quantize and dequantize use +Helium vector implementations, and `topk` remains an ExecuTorch portable +kernel. -Next, you will offer the same graph to the Ethos-U delegate first and inspect -what remains for the CPU. +Next, you will see how CMSIS-NN provides optimized CPU fallback when deploying a +model to an Ethos-U NPU. The resulting graph combines Ethos-U delegation, +CMSIS-NN-backed Cortex-M operators, and a portable CPU kernel. diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/3-add-ethosu-fallback.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/3-add-ethosu-fallback.md index b143207f6b..5ec1716660 100644 --- a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/3-add-ethosu-fallback.md +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/3-add-ethosu-fallback.md @@ -1,31 +1,68 @@ --- title: Add Ethos-U delegation and CPU fallback weight: 4 -description: Partition TinyCNN for Ethos-U, lower the remaining convolution to Cortex-M, and generate the model-specific AI layer on the U55 demonstration target. +description: Partition TinyCNN for Ethos-U, lower the remaining convolution to Cortex-M, and generate the model-specific AI layer on the Ethos-U55 demonstration target. ### FIXED, DO NOT MODIFY layout: learningpathall --- -## Partition the same model for Ethos-U +## Deploying across Cortex-M and Ethos-U NPU -The mixed export changes the order of decisions. It first gives -`EthosUPartitioner` the opportunity to claim supported graph regions. It then -applies Cortex-M lowering only to the operations left outside the delegate: +We can re-use the same script as on the previous page, this time using 'Fallback mode'. This uses `EthosUPartitioner` to +create an Ethos-U55 delegate region, represented by `executorch_call_delegate`. +The exporter then applies `CortexMPassManager` to operations left outside that +delegate, lowering its supported convolution to a registered Cortex-M operator +implemented by CMSIS-NN. + +For supported neural-network operations, the expected order is Ethos-U NPU, +then CMSIS-NN on the Cortex-M CPU, then a portable CPU kernel. This keeps as much +work as possible on the NPU and gives unsupported quantized work an optimized +CPU route before using a general implementation. ![Flow diagram showing that the graph is quantized, supported work is partitioned for Ethos-U, the supported remainder is lowered to Cortex-M operators, and the final remainder stays portable. This order preserves eligible NPU work before CPU lowering.#center](images/executorch-export-flow.svg "ExecuTorch partitioning and lowering order") -Ordering matters. If Cortex-M lowering ran first, it could rewrite a -convolution that should have been offered to the NPU. +Ordering matters. If Cortex-M lowering runs first, it could rewrite a +convolution that should be offered to the NPU. + +The exporter first creates an Ethos-U55 compile specification and configures +the target-specific quantizer: + +```python +compile_spec = EthosUCompileSpec(target=args.target) +quantizer = EthosUQuantizer(compile_spec) +quantizer.set_global( + get_symmetric_quantization_config(is_per_channel=True) +) +``` + +After quantization, `EthosUPartitioner` identifies supported regions and lowers +them into an Ethos-U delegate: + +```python +edge_program_manager = to_edge_transform_and_lower( + quantized_exported_program, + partitioner=[EthosUPartitioner(compile_spec)], + compile_config=EdgeCompileConfig(_check_ir_validity=False), +) +``` + +The exporter then applies Cortex-M lowering to the operations that remain +outside the delegate: + +```python +edge_program_manager._edge_programs["forward"] = CortexMPassManager( + edge_program_manager.exported_program() +).transform() +``` This run uses the `ethos-u55-128` target to create the predictable fallback -described earlier. The partition-then-lower order also applies to -other Ethos-U targets; the operations left outside their delegates can differ. +described earlier. -Run the diagnostic exporter in mixed mode: +Run the export script in mixed Cortex-M + Ethos-U mode: {{< tabpane code=true >}} - {{< tab header="Linux" language="bash">}} + {{< tab header="Linux and macOS" language="bash">}} python export_cortex_m_fallback.py \ --mode ethos-u-fallback \ --target ethos-u55-128 \ @@ -58,29 +95,20 @@ cortex_m.transpose.default: 2 executorch_call_delegate: 1 ``` -The above summary is interpreted as follows: - -| Model work | Final Graph target | Execution engine | -| --- | --- | --- | -| Stride-one convolution and supported surrounding work | `executorch_call_delegate` | Ethos-U55 | -| Stride-four convolution | `cortex_m.quantized_conv2d_nhwc` | Cortex-M55 using CMSIS-NN | -| Output selection | `aten.topk` | Cortex-M55 using the portable ExecuTorch kernel | - -One of the two convolution nodes has disappeared into the delegate. The other -becomes a Cortex-M operator. `topk` remains portable. +The final graph contains all three execution paths: +- `executorch_call_delegate` contains the stride-one convolution accepted by + Ethos-U55. +- `cortex_m.quantized_conv2d` is the stride-four convolution rejected by + Ethos-U55 and lowered to CMSIS-NN. +- `aten.topk` remains a portable FP32 operation. -{{% notice Note %}} -Vela can report `CPU operators = 0` for the region submitted to it. That line -doesn't describe the complete ExecuTorch graph. The graph summary shows the -Cortex-M convolution and portable `topk` that remain outside the delegate. -{{% /notice %}} +## Using the CMSIS-ExecuTorch project workflow -## Connect fallback to the CMSIS project +The CMSIS-Executorch project workflow uses `create_ai_layer.py` to export a model, generate a `.pte`, and select the source components required to execute it. -The CMSIS-Executorch project uses `create_ai_layer.py` to export the model again, generate the final PTE, and select the source components required to execute it. +We have instead used a separate script for the previous examples. By using `create_ai_layer.py` instead, we will benefit from the rest of the setup. -However, some patches have to be made. The project generator within that project currently handles Ethos-U delegation, but it does not lower the remaining convolution to Cortex-M or select CMSIS-NN. The focused -integration patch makes four related changes: +The pre-existing `create_ai_layer.py` does not include CMSIS-NN lowering, so we will add this through a targeted patch that makes four key changes: - It adds `ARM::CMSIS-NN@8.0.0` to the solution's Pack list. - It applies Cortex-M lowering after Ethos-U partitioning. @@ -120,7 +148,7 @@ writes three useful descriptions: Use the MLOps description to export the model and write its CMSIS AI layer: {{< tabpane code=true >}} - {{< tab header="Linux" language="bash">}} + {{< tab header="Linux and macOS" language="bash">}} python create_ai_layer.py cmsis-executorch-u55.cbuild-mlops.yml {{< /tab >}} {{< tab header="Windows PowerShell" language="powershell">}} diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/4-inspect-cmsis-pack.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/4-inspect-cmsis-pack.md index 2d4acc3e06..32f9a1ea06 100644 --- a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/4-inspect-cmsis-pack.md +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/4-inspect-cmsis-pack.md @@ -9,10 +9,16 @@ layout: learningpathall ## Understand what the CMSIS Pack provides -The [ExecuTorch CMSIS Pack](https://www.keil.arm.com/packs/executorch-pytorch/overview/) is a *source Pack*, not a prebuilt runtime library. It contains the ExecuTorch sources plus metadata defining selectable components and their dependencies. +The [ExecuTorch CMSIS Pack](https://www.keil.arm.com/packs/executorch-pytorch/overview/) is a *source Pack*, not a prebuilt runtime library. It supplies the ExecuTorch runtime, registration code, backends, and +operators as selectable CMSIS components. -CMSIS-Toolbox then uses that metadata to add only the chosen components to the -application build. Arm Compiler then compiles those sources for the Cortex-M55 +A `.pack` file is a ZIP-compatible archive. Its Pack Description file, or +PDSC, records component names, source files, dependencies, versions, and +conditions. + +CMSIS-Toolbox uses that metadata to add only the chosen components to the +application build. A CMSIS project can select only the components required by its lowered graph +instead of maintaining one large source list. Arm Compiler then compiles those sources for the Cortex-M55 target with the rest of the project. ## See which Packs were resolved @@ -21,7 +27,7 @@ CMSIS-Toolbox has already acquired the software Packs needed by the project. Inspect the generated Pack lock: {{< tabpane code=true >}} - {{< tab header="Linux" language="bash">}} + {{< tab header="Linux and macOS" language="bash">}} cat cmsis-executorch-u55.cbuild-pack.yml {{< /tab >}} {{< tab header="Windows PowerShell" language="powershell">}} @@ -54,7 +60,7 @@ The AI layer is a generated CMSIS software layer that embeds the model and decla Open the AI layer generated from the `.pte` operator list: {{< tabpane code=true >}} - {{< tab header="Linux" language="bash">}} + {{< tab header="Linux and macOS" language="bash">}} cat ai_layer/ai_layer.clayer.yml {{< /tab >}} {{< tab header="Windows PowerShell" language="powershell">}} diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/5-prepare-cmsis-runner.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/5-prepare-cmsis-runner.md index 0d7b4aeea3..4f3a3eaa5d 100644 --- a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/5-prepare-cmsis-runner.md +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/5-prepare-cmsis-runner.md @@ -9,7 +9,7 @@ layout: learningpathall ## Understand the solution file -`cmsis-executorch-u55.csolution.yml` defines the complete U55 build context. +`cmsis-executorch-u55.csolution.yml` defines the complete Ethos-U55 build context. Open it and inspect: ```yaml diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/6-consume-cmsis-pack.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/6-consume-cmsis-pack.md index 9a822ecab7..02ab0eb844 100644 --- a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/6-consume-cmsis-pack.md +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/6-consume-cmsis-pack.md @@ -9,12 +9,10 @@ layout: learningpathall ## Build the application -Make sure the `vcpkg` tools are active in your current terminal, then build the -U55 target from the repository root: - -```console -cbuild cmsis-executorch-u55.csolution.yml --active SSE-300-U55 -``` +In VS Code's **CMSIS** view, confirm that +`cmsis-executorch-u55.csolution.yml` is the active solution and +`SSE-300-U55` is the selected target. Select **Build** to compile the application +using the AI layer generated in chapter 3. CMSIS-Toolbox combines the App, Board, and AI layers, resolves their selected components, and invokes Arm Compiler for Embedded. The final output includes: @@ -35,11 +33,12 @@ The `.axf` file contains the linked executable and debug information. The ## Verify the compiled source paths A successful build proves that an image was produced, but it doesn't by itself -show which implementation supplied each operation. Run the verification helper -against the compilation database: +show which implementation supplied each operation. In the VS Code terminal, +with `.venv` activated and the project root as the current directory, run the +verification helper against the compilation database: {{< tabpane code=true >}} - {{< tab header="Linux" language="bash">}} + {{< tab header="Linux and macOS" language="bash">}} python verify_pack_sources.py \ out/cmsis-executorch/SSE-300-U55/Debug/compile_commands.json {{< /tab >}} @@ -68,11 +67,17 @@ shows both the resolved Pack versions and the source files actually compiled. ## Run the image on the FVP -Start the Corstone-300 model and load the generated HEX image: +In the same VS Code terminal, start the Corstone-300 model and load the generated +HEX image. On macOS, use the Docker wrapper configured during setup: -```console +{{< tabpane code=true >}} + {{< tab header="Linux and Windows" language="console">}} FVP_Corstone_SSE-300_Ethos-U55 -f board/Corstone-300/fvp_config_u55.txt -a out/cmsis-executorch/SSE-300-U55/Debug/cmsis-executorch.hex --simlimit 60 -``` + {{< /tab >}} + {{< tab header="macOS" language="bash">}} +FVP_MODEL=FVP_Corstone_SSE-300_Ethos-U55 bash .vscode/fvp.sh -f board/Corstone-300/fvp_config_u55.txt -a out/cmsis-executorch/SSE-300-U55/Debug/cmsis-executorch.hex --simlimit 60 + {{< /tab >}} +{{< /tabpane >}} The relevant output is: @@ -113,7 +118,7 @@ You also verified the implementation rather than inferring it from a successful run. The exported graph shows ownership, the compilation database shows the compiled Pack sources, and the FVP confirms that the combined firmware runs. -This build uses U55 to make one unsupported convolution predictable. The same +This build uses Ethos-U55 to make one unsupported convolution predictable. The same Pack workflow applies to another Ethos-U target when its delegated graph leaves operations for Cortex-M execution. diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/_index.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/_index.md index 5d244c0a21..b9664bd427 100644 --- a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/_index.md +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/_index.md @@ -1,22 +1,24 @@ --- -title: Deploy ML models across Cortex-M and Ethos-U with CMSIS-NN fallback +title: Optimize ML model deployment on microcontrollers with CMSIS-NN kernels -description: Build an ExecuTorch 1.5.1 application that delegates supported work to Ethos-U and uses CMSIS-NN for optimized Cortex-M fallback, demonstrated on Ethos-U55. +description: Build an ExecuTorch 1.5.1 application from the published ExecuTorch CMSIS Pack, that uses CMSIS-NN for optimized Cortex-M fallback. minutes_to_complete: 60 -who_is_this_for: This Learning Path is for embedded machine learning developers who want to understand and build an ExecuTorch application split across an Ethos-U NPU and Cortex-M CPU. +who_is_this_for: This Learning Path is for embedded machine learning developers building ExecuTorch applications for Cortex-M CPUs and Ethos-U NPUs. learning_objectives: - - Explain how ExecuTorch assigns model operations to Ethos-U, CMSIS-NN-backed Cortex-M operators, and portable kernels - - Compare a Cortex-M baseline with a partially delegated Ethos-U program - - Generate a model-specific CMSIS AI layer from published software packs - - Build and run the mixed application on a Corstone-300 Ethos-U55 FVP + - Export an ExecuTorch graph with Ethos-U, CMSIS-NN, and portable-kernel execution regions + - Lower quantized CPU fallback operations to optimized CMSIS-NN-backed Cortex-M operators, and understand the benefits in doing so + - Install and consume the published ExecuTorch CMSIS Pack for version 1.5.1 with selected runtime and operator components + - Run an application built from the ExecuTorch CMSIS Pack on a Corstone-300 Ethos-U55 FVP + - Understand when and why to use the ExecuTorch CMSIS Pack prerequisites: - - A Windows or Linux computer with Git and Python 3.10 to 3.13 + - A Windows, Linux, or macOS machine with Git, VS Code, and Python 3.10 to 3.13 + - Docker Desktop for FVP simulation on macOS - An internet connection and command-line access - - Familiarity with PyTorch models and post-training quantization + - Familiarity with PyTorch - Basic familiarity with Cortex-M processors and Ethos-U NPUs author: @@ -37,6 +39,7 @@ armips: operatingsystems: - Linux - Windows + - macOS tools_software_languages: - Python - PyTorch From 67026eca9cf99d07a35c1fa718546343d5fb88b5 Mon Sep 17 00:00:00 2001 From: Matt Cossins Date: Fri, 6 Nov 2026 09:00:36 +0000 Subject: [PATCH 5/6] Review changes --- .../1-prepare.md | 35 ++-- .../2-run-cortex-m.md | 9 + .../6-consume-cmsis-pack.md | 15 +- .../files/enable_cortex_m_fallback.patch | 69 ------- .../files/export_cortex_m_fallback.py | 184 ------------------ .../files/mixed_backend_cnn.py | 58 ------ .../files/verify_pack_sources.py | 67 ------- 7 files changed, 34 insertions(+), 403 deletions(-) delete mode 100644 content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/enable_cortex_m_fallback.patch delete mode 100644 content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/export_cortex_m_fallback.py delete mode 100644 content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/mixed_backend_cnn.py delete mode 100644 content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/verify_pack_sources.py diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/1-prepare.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/1-prepare.md index c238252bc9..bc8752328d 100644 --- a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/1-prepare.md +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/1-prepare.md @@ -132,7 +132,7 @@ python -c "from importlib.metadata import version; print('ExecuTorch', version(' {{< /tabpane >}} Confirm that Python is between 3.10 and 3.13, -`cbuild` reports 2.15.0, Arm Compiler for Embedded reports 6.24, and the final +`cbuild` reports 2.15.0 or later, Arm Compiler for Embedded reports 6.24, and the final two lines are: ```output @@ -140,27 +140,32 @@ ExecuTorch 1.5.1 PyTorch 2.14.0 ``` +{{% notice Note %}} +CMSIS-Toolbox 2.15.0 or later is required for the commands and MLOps workflow in +this Learning Path. If `cbuild --version` reports an older version, update the +toolbox or activate the project's 2.15.0 installation through Arm Environment +Manager, then open a new VS Code terminal. +{{% /notice %}} + ## Download the Learning Path assets -Download the fallback model, focused integration patch, and two inspection -helpers into the repository root: +Download this Learning Path's example folder from the code examples repository. +The archive is pinned to a commit and includes the fallback model, integration +patch, inspection helpers, and their license. Extract it into `lp-assets`, then +copy the four working files into the CMSIS-Executorch project root: {{< tabpane code=true >}} {{< tab header="Linux and macOS" language="bash">}} -LP_ASSETS=https://raw.githubusercontent.com/ArmDeveloperEcosystem/arm-learning-paths/main/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files - -curl -fLO "$LP_ASSETS/mixed_backend_cnn.py" -curl -fLO "$LP_ASSETS/enable_cortex_m_fallback.patch" -curl -fLO "$LP_ASSETS/export_cortex_m_fallback.py" -curl -fLO "$LP_ASSETS/verify_pack_sources.py" +curl -fL "https://gitlab.arm.com/learning-code-examples/code-examples/-/archive/e234c59f4b64b7b41e3d82b90dca21775572363f/code-examples-e234c59f4b64b7b41e3d82b90dca21775572363f.tar.gz?path=learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m" -o lp-assets.tar.gz +mkdir -p lp-assets +tar -xzf lp-assets.tar.gz -C lp-assets --strip-components=4 +cp lp-assets/*.py lp-assets/*.patch . {{< /tab >}} {{< tab header="Windows PowerShell" language="powershell">}} -$lpAssets = "https://raw.githubusercontent.com/ArmDeveloperEcosystem/arm-learning-paths/main/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files" - -curl.exe -fLO "$lpAssets/mixed_backend_cnn.py" -curl.exe -fLO "$lpAssets/enable_cortex_m_fallback.patch" -curl.exe -fLO "$lpAssets/export_cortex_m_fallback.py" -curl.exe -fLO "$lpAssets/verify_pack_sources.py" +curl.exe -fL "https://gitlab.arm.com/learning-code-examples/code-examples/-/archive/e234c59f4b64b7b41e3d82b90dca21775572363f/code-examples-e234c59f4b64b7b41e3d82b90dca21775572363f.tar.gz?path=learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m" -o lp-assets.tar.gz +New-Item -ItemType Directory -Force lp-assets | Out-Null +tar -xzf lp-assets.tar.gz -C lp-assets --strip-components=4 +Copy-Item lp-assets\*.py, lp-assets\*.patch . {{< /tab >}} {{< /tabpane >}} diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/2-run-cortex-m.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/2-run-cortex-m.md index 6e88fd7bf4..3ee648cdf1 100644 --- a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/2-run-cortex-m.md +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/2-run-cortex-m.md @@ -13,6 +13,15 @@ Before we demonstrate fallback from an Ethos-U, we will briefly look at running ## See how the export script lowers the graph +{{% notice Note %}} +The ExecuTorch Cortex-M backend is in **beta** in the 1.5 release used here. +Validation covers MLPerf Tiny models and MobileNetV2. +It combines optimized CMSIS-NN implementations with portable FP32 operators +for operations outside its accelerated coverage. See the +[Cortex-M backend documentation](https://docs.pytorch.org/executorch/stable/backends/arm-cortex-m/arm-cortex-m-overview.html) +for supported operators and limitations. +{{% /notice %}} + The exporter first prepares and converts the model with `CortexMQuantizer`: ```python diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/6-consume-cmsis-pack.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/6-consume-cmsis-pack.md index 02ab0eb844..89e6dd7139 100644 --- a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/6-consume-cmsis-pack.md +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/6-consume-cmsis-pack.md @@ -15,13 +15,9 @@ In VS Code's **CMSIS** view, confirm that using the AI layer generated in chapter 3. CMSIS-Toolbox combines the App, Board, and AI layers, resolves their selected -components, and invokes Arm Compiler for Embedded. The final output includes: - -```output -Build summary: 1 succeeded, 0 failed -``` - -The generated images are written under: +components, and invokes Arm Compiler for Embedded. Check that the build task +finishes successfully without errors. Confirm that the build produces `cmsis-executorch.axf` and +`cmsis-executorch.hex` under: ```text out/cmsis-executorch/SSE-300-U55/Debug/ @@ -107,15 +103,14 @@ also have a cost. ## What you've accomplished -You have built and run one ExecuTorch program across three execution paths: +You have built and run one ExecuTorch program across three execution paths, and used CMSIS-ExecuTorch to make setup easier: - The supported convolution runs in an Ethos-U55 delegate. - The unsupported stride-four convolution runs on Cortex-M55 through the ExecuTorch wrapper and optimized CMSIS-NN kernel. - `topk` runs through the portable ExecuTorch CPU kernel. -You also verified the implementation rather than inferring it from a successful -run. The exported graph shows ownership, the compilation database shows the +You also verified the implementation. The exported graph shows ownership, the compilation database shows the compiled Pack sources, and the FVP confirms that the combined firmware runs. This build uses Ethos-U55 to make one unsupported convolution predictable. The same diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/enable_cortex_m_fallback.patch b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/enable_cortex_m_fallback.patch deleted file mode 100644 index 9738f53935..0000000000 --- a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/enable_cortex_m_fallback.patch +++ /dev/null @@ -1,69 +0,0 @@ -diff --git a/cmsis-executorch-u55.csolution.yml b/cmsis-executorch-u55.csolution.yml ---- a/cmsis-executorch-u55.csolution.yml -+++ b/cmsis-executorch-u55.csolution.yml -@@ -20,4 +20,5 @@ solution: - # The PyTorch::ExecuTorch pack version must match the ExecuTorch Python - # exporter in requirements.txt because its runtime consumes the generated .pte. - - pack: PyTorch::ExecuTorch@1.5.1 -+ - pack: ARM::CMSIS-NN@8.0.0 - - pack: ARM::Cortex_DFP -diff --git a/create_ai_layer.py b/create_ai_layer.py ---- a/create_ai_layer.py -+++ b/create_ai_layer.py -@@ -173,6 +173,9 @@ def export_model(spec) -> bytes: - """Quantize model/model.py, delegate it to the Ethos-U and return the .pte.""" - import torch - from executorch.backends.arm.ethosu import EthosUPartitioner -+ from executorch.backends.cortex_m.passes.cortex_m_pass_manager import ( -+ CortexMPassManager, -+ ) - from executorch.backends.arm.quantizer import EthosUQuantizer, get_symmetric_quantization_config - from executorch.exir import EdgeCompileConfig, ExecutorchBackendConfig, to_edge_transform_and_lower - from torchao.quantization.pt2e.quantize_pt2e import convert_pt2e, prepare_pt2e -@@ -194,5 +197,8 @@ def export_model(spec) -> bytes: - partitioner=[EthosUPartitioner(spec)], - compile_config=EdgeCompileConfig(_check_ir_validity=False), - ) -+ edge._edge_programs["forward"] = CortexMPassManager( -+ edge.exported_program(), use_explicit_layout=True -+ ).transform() - program = edge.to_executorch(ExecutorchBackendConfig(extract_delegate_segments=False)) - return bytes(program.buffer) -@@ -214,7 +220,7 @@ def components(pte: bytes, pack: Path) -> tuple[list[str], list[str]]: - ns, op = ns.decode(), op.decode() - candidates = [ - f"{family[ns]} {op}", -- f"{family[ns]} {re.sub(r'_(per_tensor|per_channel|byte|copy)$', '', op)}", -+ f"{family[ns]} {re.sub(r'_(per_tensor|per_channel|byte|copy|nhwc)$', '', op)}", - ] - if match := next((c for c in candidates if c in available), None): - selected.add(match) -@@ -250,6 +256,8 @@ extern const unsigned long {SYMBOL}_size; - - - def clayer(mlops: dict, runtime: list[str], operators: list[str], mlops_file: Path, version: str) -> str: -+ # CMSIS-NN 8.0 builds optional MVE float sources that need both switches. -+ uses_cmsis_nn = any(op.startswith("Cortex-M quantized_") for op in operators) - lines = [ - f"# Generated by create_ai_layer.py from {mlops_file.name} -- do not edit.", - f"# Re-run `python create_ai_layer.py {mlops_file.name}` after changing", -@@ -263,6 +271,11 @@ def clayer(mlops: dict, runtime: list[str], operators: list[str], mlops_file: Pa - "", - " define:", - " - ET_LOG_ENABLED: 0", -+ *( -+ [" - ARM_NN_ENABLE_F32: 1", " - ARM_NN_ENABLE_F16: 1"] -+ if uses_cmsis_nn -+ else [] -+ ), - "", - " add-path:", - " - .", -@@ -270,6 +283,7 @@ def clayer(mlops: dict, runtime: list[str], operators: list[str], mlops_file: Pa - " components:", - *[f" - component: Machine Learning:ExecuTorch:{c}" for c in runtime], - *[f" - component: Machine Learning:ExecuTorch Operators:{c}" for c in operators], -+ *([" - component: ARM::CMSIS:NN Lib"] if uses_cmsis_nn else []), - "", - " groups:", - f" - group: {mlops['model'].get('name', 'Model')}", diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/export_cortex_m_fallback.py b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/export_cortex_m_fallback.py deleted file mode 100644 index 39aa9a36a8..0000000000 --- a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/export_cortex_m_fallback.py +++ /dev/null @@ -1,184 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import annotations - -import argparse -import importlib.util -import json -from pathlib import Path -import sys -from typing import Any - - -def load_model_module(path: Path) -> Any: - spec = importlib.util.spec_from_file_location("lp_model", path) - if spec is None or spec.loader is None: - raise SystemExit(f"Cannot load model from {path}") - module = importlib.util.module_from_spec(spec) - sys.modules[spec.name] = module - spec.loader.exec_module(module) - return module - - -def main() -> None: - parser = argparse.ArgumentParser( - description=( - "Export a Cortex-M baseline or a partially delegated Ethos-U, " - "CMSIS-NN, and portable-kernel model." - ) - ) - parser.add_argument( - "--mode", - choices=("cortex-m", "ethos-u-fallback"), - required=True, - help="Select Cortex-M-only lowering or Ethos-U followed by Cortex-M lowering.", - ) - parser.add_argument( - "--target", - default="ethos-u55-128", - help="Ethos-U target used by fallback mode and the Corstone FVP runner.", - ) - parser.add_argument( - "--output-dir", - type=Path, - default=Path("artifacts"), - help="Directory for the bundled program and graph summary.", - ) - parser.add_argument( - "--model-file", - type=Path, - default=Path("model/model.py"), - help="Model module providing get_model() and get_calibration_inputs().", - ) - args = parser.parse_args() - - import torch - from executorch.backends.cortex_m.passes.cortex_m_pass_manager import ( - CortexMPassManager, - ) - from executorch.exir import ( - EdgeCompileConfig, - ExecutorchBackendConfig, - to_edge, - to_edge_transform_and_lower, - ) - from executorch.extension.export_util.utils import save_pte_program - from torchao.quantization.pt2e.quantize_pt2e import convert_pt2e, prepare_pt2e - - model_module = load_model_module(args.model_file.resolve()) - model = model_module.get_model() - calibration_inputs = model_module.get_calibration_inputs() - example_inputs = (calibration_inputs[0],) - exported_program = torch.export.export(model, example_inputs) - captured_graph = exported_program.module(check_guards=False) - - if args.mode == "cortex-m": - from executorch.backends.cortex_m.quantizer.quantizer import CortexMQuantizer - - quantizer = CortexMQuantizer() - else: - from executorch.backends.arm.ethosu import EthosUCompileSpec - from executorch.backends.arm.quantizer import ( - EthosUQuantizer, - get_symmetric_quantization_config, - ) - - compile_spec = EthosUCompileSpec(target=args.target) - quantizer = EthosUQuantizer(compile_spec) - quantizer.set_global(get_symmetric_quantization_config(is_per_channel=True)) - - prepared = prepare_pt2e(captured_graph, quantizer) - for calibration_input in calibration_inputs: - prepared(calibration_input) - quantized_model = convert_pt2e(prepared) - quantized_exported_program = torch.export.export(quantized_model, example_inputs) - reference_output = quantized_exported_program.module()(*example_inputs) - - if args.mode == "cortex-m": - edge_program_manager = to_edge( - quantized_exported_program, - compile_config=EdgeCompileConfig(_check_ir_validity=False), - ) - else: - from executorch.backends.arm.ethosu import EthosUPartitioner - - edge_program_manager = to_edge_transform_and_lower( - quantized_exported_program, - partitioner=[EthosUPartitioner(compile_spec)], - compile_config=EdgeCompileConfig(_check_ir_validity=False), - ) - - targets_before_cortex_m = count_targets( - edge_program_manager.exported_program().graph_module.graph - ) - - edge_program_manager._edge_programs["forward"] = CortexMPassManager( - edge_program_manager.exported_program(), use_explicit_layout=True - ).transform() - - targets_after_cortex_m = count_targets( - edge_program_manager.exported_program().graph_module.graph - ) - executorch_program = edge_program_manager.to_executorch( - config=ExecutorchBackendConfig(extract_delegate_segments=False) - ) - - args.output_dir.mkdir(parents=True, exist_ok=True) - artifact_name = args.mode.replace("-", "_") - artifact_path = args.output_dir / f"stride4_{artifact_name}.pte" - save_pte_program(executorch_program, str(artifact_path)) - - summary = { - "mode": args.mode, - "target": args.target, - "artifact": str(artifact_path), - "artifact_size_bytes": artifact_path.stat().st_size, - "reference_output": reference_output.detach().flatten().tolist(), - "targets_before_cortex_m": targets_before_cortex_m, - "targets_after_cortex_m": targets_after_cortex_m, - } - summary_path = artifact_path.with_suffix(".json") - summary_path.write_text(json.dumps(summary, indent=2) + "\n") - - print(f"Wrote {artifact_path}") - print(f"Wrote {summary_path}") - print("\nRelevant targets before Cortex-M lowering:") - print_relevant_targets(targets_before_cortex_m) - print("\nRelevant targets after Cortex-M lowering:") - print_relevant_targets(targets_after_cortex_m) - - -def count_targets(graph: Any) -> dict[str, int]: - counts: dict[str, int] = {} - for node in graph.nodes: - if node.op != "call_function": - continue - target = display_target(str(node.target)) - if target.startswith("<") and hasattr(node.target, "__name__"): - target = node.target.__name__ - counts[target] = counts.get(target, 0) + 1 - return dict(sorted(counts.items())) - - -def print_relevant_targets(targets: dict[str, int]) -> None: - found = False - for target, count in targets.items(): - if ( - "cortex_m" in target - or "delegate" in target - or "convolution" in target - or "topk" in target - ): - print(f"{display_target(target)}: {count}") - found = True - if not found: - print("No delegate or Cortex-M targets found.") - - -def display_target(target: str) -> str: - if target.startswith("", 1)[0] - return target - - -if __name__ == "__main__": - main() diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/mixed_backend_cnn.py b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/mixed_backend_cnn.py deleted file mode 100644 index a28427f789..0000000000 --- a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/mixed_backend_cnn.py +++ /dev/null @@ -1,58 +0,0 @@ -# Copyright 2026 Arm Limited and/or its affiliates. -# SPDX-License-Identifier: Apache-2.0 -"""TinyCNN variant with Ethos-U, CMSIS-NN, and portable execution paths.""" - -import torch -from torch import nn - -# NHWC, the layout of a camera frame, which the Ethos-U reads as it is. An NCHW -# input costs one more NPU operation that reorders it before the first -# convolution. -INPUT_SHAPE = (1, 16, 16, 3) - - -class TinyCNN(nn.Module): - def __init__(self, input_shape: tuple[int, ...] = INPUT_SHAPE, num_classes: int = 10) -> None: - super().__init__() - channels = input_shape[-1] - self.conv1 = nn.Conv2d( - channels, 4, kernel_size=3, stride=1, padding=1, bias=False - ) - self.conv2 = nn.Conv2d( - 4, num_classes, kernel_size=3, stride=4, padding=1, bias=False - ) - - def forward(self, inputs: torch.Tensor) -> torch.Tensor: - # Camera input is NHWC; convolutions use NCHW. - values = inputs.permute(0, 3, 1, 2) - values = torch.relu(self.conv1(values)) - values = self.conv2(values) - - # Keep one maximum spatial response for each of the ten output channels. - values = torch.flatten(values, start_dim=2) - values, _ = torch.topk(values, k=1, dim=-1) - return values.squeeze(-1) - - -def get_model(input_shape: tuple[int, ...] = INPUT_SHAPE) -> nn.Module: - # Fixed seed: the example uses untrained (random) weights, and a fixed seed - # keeps the exported program identical from one export to the next (the - # C array also embeds the Vela options, so the same working directory is - # part of that guarantee). - torch.manual_seed(0) - return TinyCNN(input_shape).eval() - - -def get_calibration_inputs( - input_shape: tuple[int, ...] = INPUT_SHAPE, calibration_samples: int = 2 -) -> list[torch.Tensor]: - """Float samples for quantization calibration; the first one is also the export example. - - Fixed values keep the quantization parameters, and with them the output - logits in the README, the same from one export to the next. Replace them - with representative, preprocessed data when calibrating a trained model. - """ - samples = [torch.full(input_shape, 0.5), torch.full(input_shape, -0.5)][:calibration_samples] - for i in range(len(samples), calibration_samples): - samples.append(torch.randn(input_shape, generator=torch.Generator().manual_seed(i))) - return samples diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/verify_pack_sources.py b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/verify_pack_sources.py deleted file mode 100644 index 4c5bf84d84..0000000000 --- a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/files/verify_pack_sources.py +++ /dev/null @@ -1,67 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import annotations - -import argparse -import json -from pathlib import Path - - -EXPECTED_SOURCES = ( - ( - "ExecuTorch runtime", - "src/runtime/executor/program.cpp", - ("/PyTorch/ExecuTorch/1.5.1/", "PyTorch_Machine_Learning_ExecuTorch_"), - ), - ( - "ExecuTorch Cortex-M convolution wrapper", - "src/backends/cortex_m/ops/op_quantized_conv2d.cpp", - ("/PyTorch/ExecuTorch/1.5.1/", "PyTorch_Machine_Learning_ExecuTorch_"), - ), - ( - "CMSIS-NN convolution kernel", - "Source/ConvolutionFunctions/arm_convolve_s8.c", - ("/ARM/CMSIS-NN/8.0.0/", "ARM_CMSIS_NN_Lib_8_0_0"), - ), - ( - "Ethos-U backend", - "src/backends/arm/runtime/EthosUBackend.cpp", - ("/PyTorch/ExecuTorch/1.5.1/", "PyTorch_Machine_Learning_ExecuTorch_"), - ), - ( - "ExecuTorch portable topk kernel", - "src/kernels/portable/cpu/op_topk.cpp", - ("/PyTorch/ExecuTorch/1.5.1/", "PyTorch_Machine_Learning_ExecuTorch_"), - ), -) - - -def main() -> None: - parser = argparse.ArgumentParser() - parser.add_argument("compile_database", type=Path) - args = parser.parse_args() - - entries = json.loads(args.compile_database.read_text()) - missing = [] - for label, source_path, pack_markers in EXPECTED_SOURCES: - matched = False - for entry in entries: - source_file = entry.get("file", "").replace("\\", "/") - command = entry.get("command", " ".join(entry.get("arguments", []))).replace( - "\\", "/" - ) - if source_file.endswith(source_path) and any( - marker in f"{source_file} {command}" for marker in pack_markers - ): - matched = True - break - if matched: - print(f"PASS: {label} was compiled from its declared CMSIS Pack") - else: - missing.append(label) - - if missing: - raise SystemExit(f"Missing pack sources: {', '.join(missing)}") - - -if __name__ == "__main__": - main() From 1dac3f98306fea43037bd76f20db15554541fadb Mon Sep 17 00:00:00 2001 From: Matt Cossins Date: Fri, 6 Nov 2026 15:09:31 +0000 Subject: [PATCH 6/6] Nits --- .../1-prepare.md | 4 ++-- .../2-run-cortex-m.md | 11 ++++++----- .../3-add-ethosu-fallback.md | 6 +++--- .../optimize-executorch-fallback-cortex-m/_index.md | 2 +- 4 files changed, 12 insertions(+), 11 deletions(-) diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/1-prepare.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/1-prepare.md index bc8752328d..074d56321f 100644 --- a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/1-prepare.md +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/1-prepare.md @@ -156,13 +156,13 @@ copy the four working files into the CMSIS-Executorch project root: {{< tabpane code=true >}} {{< tab header="Linux and macOS" language="bash">}} -curl -fL "https://gitlab.arm.com/learning-code-examples/code-examples/-/archive/e234c59f4b64b7b41e3d82b90dca21775572363f/code-examples-e234c59f4b64b7b41e3d82b90dca21775572363f.tar.gz?path=learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m" -o lp-assets.tar.gz +curl -fL "https://gitlab.arm.com/learning-code-examples/code-examples/-/archive/5eb4e50676d4085c90e926abec1c978f257d1bde/code-examples-5eb4e50676d4085c90e926abec1c978f257d1bde.tar.gz?path=learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m" -o lp-assets.tar.gz mkdir -p lp-assets tar -xzf lp-assets.tar.gz -C lp-assets --strip-components=4 cp lp-assets/*.py lp-assets/*.patch . {{< /tab >}} {{< tab header="Windows PowerShell" language="powershell">}} -curl.exe -fL "https://gitlab.arm.com/learning-code-examples/code-examples/-/archive/e234c59f4b64b7b41e3d82b90dca21775572363f/code-examples-e234c59f4b64b7b41e3d82b90dca21775572363f.tar.gz?path=learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m" -o lp-assets.tar.gz +curl.exe -fL "https://gitlab.arm.com/learning-code-examples/code-examples/-/archive/5eb4e50676d4085c90e926abec1c978f257d1bde/code-examples-5eb4e50676d4085c90e926abec1c978f257d1bde.tar.gz?path=learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m" -o lp-assets.tar.gz New-Item -ItemType Directory -Force lp-assets | Out-Null tar -xzf lp-assets.tar.gz -C lp-assets --strip-components=4 Copy-Item lp-assets\*.py, lp-assets\*.patch . diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/2-run-cortex-m.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/2-run-cortex-m.md index 3ee648cdf1..6b1eda6e63 100644 --- a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/2-run-cortex-m.md +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/2-run-cortex-m.md @@ -27,13 +27,14 @@ The exporter first prepares and converts the model with `CortexMQuantizer`: ```python quantizer = CortexMQuantizer() -prepared = prepare_pt2e(exported_program.module(), quantizer) -prepared(*example_inputs) +prepared = prepare_pt2e(captured_graph, quantizer) +for calibration_input in calibration_inputs: + prepared(calibration_input) quantized_model = convert_pt2e(prepared) quantized_exported_program = torch.export.export(quantized_model, example_inputs) ``` -The prepared model is run once with the example input to calibrate activation +The prepared model is run with both calibration samples to measure activation ranges. `convert_pt2e()` uses those ranges to choose quantization parameters and insert the operations that move values between FP32 and INT8. @@ -47,7 +48,7 @@ edge_program_manager = to_edge( ) edge_program_manager._edge_programs["forward"] = CortexMPassManager( - edge_program_manager.exported_program() + edge_program_manager.exported_program(), use_explicit_layout=True ).transform() ``` @@ -70,7 +71,7 @@ node until the Ethos-U partitioner is added on the next page. ## Export the model for Cortex-M -Use the provided script to lower the model to Cortex-M only, by specifying `-mode cortex-m`: +Use the provided script to lower the model to Cortex-M only, by specifying `--mode cortex-m`: {{< tabpane code=true >}} {{< tab header="Linux and macOS" language="bash">}} diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/3-add-ethosu-fallback.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/3-add-ethosu-fallback.md index 5ec1716660..c9e4ced73a 100644 --- a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/3-add-ethosu-fallback.md +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/3-add-ethosu-fallback.md @@ -52,7 +52,7 @@ outside the delegate: ```python edge_program_manager._edge_programs["forward"] = CortexMPassManager( - edge_program_manager.exported_program() + edge_program_manager.exported_program(), use_explicit_layout=True ).transform() ``` @@ -78,7 +78,7 @@ python .\export_cortex_m_fallback.py ` {{< /tab >}} {{< /tabpane >}} -Vela prints a summary for the region it compiles: +The export script prints a summary of the graph before and after Cortex-M lowering: ```output Relevant targets before Cortex-M lowering: @@ -98,7 +98,7 @@ executorch_call_delegate: 1 The final graph contains all three execution paths: - `executorch_call_delegate` contains the stride-one convolution accepted by Ethos-U55. -- `cortex_m.quantized_conv2d` is the stride-four convolution rejected by +- `cortex_m.quantized_conv2d_nhwc` is the stride-four convolution rejected by Ethos-U55 and lowered to CMSIS-NN. - `aten.topk` remains a portable FP32 operation. diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/_index.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/_index.md index b9664bd427..db909cbdc8 100644 --- a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/_index.md +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/_index.md @@ -15,7 +15,7 @@ learning_objectives: - Understand when and why to use the ExecuTorch CMSIS Pack prerequisites: - - A Windows, Linux, or macOS machine with Git, VS Code, and Python 3.10 to 3.13 + - Git, VS Code, and Python 3.10 to 3.13 on Windows, macOS, or x86-64 Linux - Docker Desktop for FVP simulation on macOS - An internet connection and command-line access - Familiarity with PyTorch