diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/1-prepare.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/1-prepare.md new file mode 100644 index 0000000000..074d56321f --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/1-prepare.md @@ -0,0 +1,218 @@ +--- +title: Understand and prepare the environment +weight: 2 +description: Set up the reusable CMSIS-Executorch project and its pinned Python and Arm tool environments. + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## ExecuTorch for Cortex-M and Ethos-U + +ExecuTorch is PyTorch’s runtime for deploying machine learning models on edge devices, including embedded systems. It converts trained models into programs that can run within the limited memory and compute resources of a microcontroller. + +Many microcontrollers use Arm Cortex-M CPUs, and others integrate both Cortex-M CPUs, and Arm Ethos-U NPUs. + +For combined Cortex-M and Ethos-U systems, ExecuTorch lets one model use both processors: supported operations are dispatched to the Ethos-U NPU, while remaining operations run on the Cortex-M CPU. + +This Learning Path will demonstrate how developers can better optimize ExecuTorch deployment, either when running models directly on a Cortex-M CPU, or in a scenario where unsupported operators fallback from an Ethos-U to the Cortex-M CPU. + +## Ethos-U vs CMSIS-NN vs Portable Kernels + +You will use the ExecuTorch ahead-of-time export flow to divide one model across +three execution paths: + +- A stride-one convolution that Ethos-U55 accepts. +- A stride-four convolution that Ethos-U55 rejects but CMSIS-NN supports. +- A `topk` operation that remains an ExecuTorch portable FP32 operation. + +This is illustrated in the following diagram: + +![Diagram of the simulated Corstone-300 FVP target showing the Cortex-M CPU running the ExecuTorch runtime and dispatching a stride-one convolution to Ethos-U55, while CMSIS-NN executes the stride-four convolution and a portable kernel executes topk on Cortex-M#center](images/cortex-m-execution-picture.svg "ExecuTorch execution paths") + +The three execution paths have different roles: + +| Technology | Role in this Learning Path | Expected performance | +| --- | --- | --- | +| [Ethos-U55](https://www.arm.com/products/silicon-ip-cpu/ethos/ethos-u55) | Runs supported graph regions on the NPU. (eg: Stride-1 convolution) | Expected to provide the best performance and energy efficiency for supported neural-network operations. | +| [CMSIS-NN](https://arm-software.github.io/CMSIS-NN/latest/index.html) | Supplies optimized Cortex-M kernels for quantized CPU operations that are unsupported by the NPU. (eg: Stride-4 convolution) | Preferred CPU fallback when a matching CMSIS-NN-backed operator is available. | +| ExecuTorch portable kernels | Supply general CPU implementations for operations without an accelerated backend implementation. (eg: `topk` operation) | Compatibility fallback rather than the preferred performance path. | + +Fallback here is planned during export. It isn't recovery from an NPU failure +at runtime. The exporter determines which regions the target Ethos-U delegate +supports, then assigns known remaining patterns to the Cortex-M backend. + +{{% notice Why this example uses Ethos-U55 %}} +Ethos-U55 is used because its unsupported stride-four convolution creates a +predictable fallback region. Ethos-U85 supports more operator configurations, +so it will typically have less CPU fallback than Ethos-U55. The same workflow still +applies when a model contains operations that remain outside the U85 delegate. +{{% /notice %}} + +In this Learning Path, you will clone a [CMSIS-Executorch example](https://github.com/Arm-Examples/cmsis-executorch) that already contains the needed board support, +runner, memory configuration, and Fixed Virtual Platform (FVP) settings. You +will then inspect and modify it to observe Cortex-M fallback using optimized CMSIS-NN microkernels. + +## Clone the example project + +Begin by cloning the CMSIS-Executorch project used by this Learning Path: + +```console +git clone https://github.com/Arm-Examples/CMSIS-Executorch.git +cd CMSIS-Executorch +git checkout 4002b63dcc3e27eb7444946a0dbf69b9a74d6fc9 +``` + +The explicit checkout pins the project to a revision that has been tested and validated. + +## Set up the project in VS Code + +Install [Visual Studio Code](https://code.visualstudio.com/), the +[Keil Studio Pack](https://marketplace.visualstudio.com/items?itemName=Arm.keil-studio-pack), +and the [Python extension](https://marketplace.visualstudio.com/items?itemName=ms-python.python). +Install Python 3.10–3.13 on your host; Python 3.14 is not supported by the +pinned export dependencies. On Windows, enable +[long-path support](https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation). + +1. Open the cloned `CMSIS-Executorch` folder in VS Code. +2. Allow Arm Environment Manager to install the tools declared in + `vcpkg-configuration.json` and wait for installation to finish. +3. In the **CMSIS** view, select **... > Open Solution in Workspace** + and choose `cmsis-executorch-u55.csolution.yml`. +4. Select **Terminal > Run Task > Create Python environment (pip)**. Wait for the + task to create `.venv` and install the pinned dependencies. +5. Open a new VS Code terminal in the project folder so it receives the Arm tool + environment. Run all subsequent commands in this terminal. + +Activate the Python environment in each new terminal: + +{{< tabpane code=true >}} + {{< tab header="Linux and macOS" language="bash">}} +source .venv/bin/activate + {{< /tab >}} + {{< tab header="Windows PowerShell" language="powershell">}} +.\.venv\Scripts\Activate.ps1 + {{< /tab >}} +{{< /tabpane >}} + +### Additional macOS setup + +Install and start [Docker Desktop](https://docs.docker.com/desktop/setup/install/mac-install/). +The FVP runs inside a Linux container because there is no native macOS FVP. +Keep the project under your home directory so the wrapper can mount it into +Docker. Confirm Docker is running and select the Ethos-U55 model in each new terminal: + +```bash +docker info +export FVP_MODEL=FVP_Corstone_SSE-300_Ethos-U55 +``` + +The final chapter uses the repository's `.vscode/fvp.sh` wrapper to run the +application on macOS. Its first run builds the container image. + +## Verify the environment + +The Python package and CMSIS Pack versions must match because the embedded +runtime loads the program produced by the exporter. Check the host tools, Arm +build tools, and Python packages: + +{{< tabpane code=true >}} + {{< tab header="Linux and macOS" language="bash">}} +python --version +cbuild --version +armclang --version +python -c "from importlib.metadata import version; print('ExecuTorch', version('executorch')); print('PyTorch', version('torch'))" + {{< /tab >}} + {{< tab header="Windows PowerShell" language="powershell">}} +python --version +cbuild --version +armclang --version +python -c "from importlib.metadata import version; print('ExecuTorch', version('executorch')); print('PyTorch', version('torch'))" + {{< /tab >}} +{{< /tabpane >}} + +Confirm that Python is between 3.10 and 3.13, +`cbuild` reports 2.15.0 or later, Arm Compiler for Embedded reports 6.24, and the final +two lines are: + +```output +ExecuTorch 1.5.1 +PyTorch 2.14.0 +``` + +{{% notice Note %}} +CMSIS-Toolbox 2.15.0 or later is required for the commands and MLOps workflow in +this Learning Path. If `cbuild --version` reports an older version, update the +toolbox or activate the project's 2.15.0 installation through Arm Environment +Manager, then open a new VS Code terminal. +{{% /notice %}} + +## Download the Learning Path assets + +Download this Learning Path's example folder from the code examples repository. +The archive is pinned to a commit and includes the fallback model, integration +patch, inspection helpers, and their license. Extract it into `lp-assets`, then +copy the four working files into the CMSIS-Executorch project root: + +{{< tabpane code=true >}} + {{< tab header="Linux and macOS" language="bash">}} +curl -fL "https://gitlab.arm.com/learning-code-examples/code-examples/-/archive/5eb4e50676d4085c90e926abec1c978f257d1bde/code-examples-5eb4e50676d4085c90e926abec1c978f257d1bde.tar.gz?path=learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m" -o lp-assets.tar.gz +mkdir -p lp-assets +tar -xzf lp-assets.tar.gz -C lp-assets --strip-components=4 +cp lp-assets/*.py lp-assets/*.patch . + {{< /tab >}} + {{< tab header="Windows PowerShell" language="powershell">}} +curl.exe -fL "https://gitlab.arm.com/learning-code-examples/code-examples/-/archive/5eb4e50676d4085c90e926abec1c978f257d1bde/code-examples-5eb4e50676d4085c90e926abec1c978f257d1bde.tar.gz?path=learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m" -o lp-assets.tar.gz +New-Item -ItemType Directory -Force lp-assets | Out-Null +tar -xzf lp-assets.tar.gz -C lp-assets --strip-components=4 +Copy-Item lp-assets\*.py, lp-assets\*.patch . + {{< /tab >}} +{{< /tabpane >}} + +## Use a simple convolution example + +The CMSIS-Executorch repository starts with a TinyCNN model that can be delegated completely to +Ethos-U. You will change this model to one that will demonstrate fallback by retaining its input and output +interface, but adjusting the operations happening inside it. + +Replace the original model with the one you have downloaded: + +{{< tabpane code=true >}} + {{< tab header="Linux and macOS" language="bash">}} +cp mixed_backend_cnn.py model/model.py + {{< /tab >}} + {{< tab header="Windows PowerShell" language="powershell">}} +Copy-Item .\mixed_backend_cnn.py .\model\model.py + {{< /tab >}} +{{< /tabpane >}} + +The model still accepts one `16 x 16` RGB image in NHWC order and returns ten +values. Keeping that interface means the supplied bare-metal runner needs no +changes. + +The important part of `model/model.py` now looks like this: + +```python +self.conv1 = nn.Conv2d( + channels, 4, kernel_size=3, stride=1, padding=1, bias=False +) +self.conv2 = nn.Conv2d( + 4, num_classes, kernel_size=3, stride=4, padding=1, bias=False +) + +values = torch.relu(self.conv1(values)) +values = self.conv2(values) +values = torch.flatten(values, start_dim=2) +values, _ = torch.topk(values, k=1, dim=-1) +``` + +The *stride* controls how far a convolution filter moves between positions. A +stride of one visits adjacent positions. A stride of four moves four positions +at a time and produces a smaller output. For this Ethos-U55 target and tensor shape, +the first convolution can be delegated to the NPU, while the stride-four convolution +remains outside the delegate. `topk` selects the largest spatial response for +each of the ten output channels. + +## What you've accomplished and what's next + +You now have a pinned Corstone-300 Ethos-U55 project, its matching Python packages, its versioned Arm tools, and a model that will only partially run on an Ethos-U55. Next, you will understand how to optimize supported operators with CMSIS-NN on Cortex-M CPUs in a CPU-only example. diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/2-run-cortex-m.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/2-run-cortex-m.md new file mode 100644 index 0000000000..6b1eda6e63 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/2-run-cortex-m.md @@ -0,0 +1,121 @@ +--- +title: Lower a model to the Cortex-M backend +weight: 3 +description: Export a Cortex-M model and identify CMSIS-NN accelerated convolutions + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## First use only Cortex-M + +Before we demonstrate fallback from an Ethos-U, we will briefly look at running a model exclusively on Cortex-M. Not every microcontroller has an onboard NPU, so some use-cases will involve running TinyML directly on the CPU. In this case, any optimizations can be the vital difference between functional performance, and an impractical application. + +## See how the export script lowers the graph + +{{% notice Note %}} +The ExecuTorch Cortex-M backend is in **beta** in the 1.5 release used here. +Validation covers MLPerf Tiny models and MobileNetV2. +It combines optimized CMSIS-NN implementations with portable FP32 operators +for operations outside its accelerated coverage. See the +[Cortex-M backend documentation](https://docs.pytorch.org/executorch/stable/backends/arm-cortex-m/arm-cortex-m-overview.html) +for supported operators and limitations. +{{% /notice %}} + +The exporter first prepares and converts the model with `CortexMQuantizer`: + +```python +quantizer = CortexMQuantizer() + +prepared = prepare_pt2e(captured_graph, quantizer) +for calibration_input in calibration_inputs: + prepared(calibration_input) +quantized_model = convert_pt2e(prepared) +quantized_exported_program = torch.export.export(quantized_model, example_inputs) +``` + +The prepared model is run with both calibration samples to measure activation +ranges. `convert_pt2e()` uses those ranges to choose quantization parameters and +insert the operations that move values between FP32 and INT8. + +The exporter converts the quantized program to the ExecuTorch Edge dialect and +applies the Cortex-M pass manager: + +```python +edge_program_manager = to_edge( + quantized_exported_program, + compile_config=EdgeCompileConfig(_check_ir_validity=False) +) + +edge_program_manager._edge_programs["forward"] = CortexMPassManager( + edge_program_manager.exported_program(), use_explicit_layout=True +).transform() +``` + +This step rewrites supported patterns as `cortex_m.*` operators. Unsupported +operations, including `aten.topk`, remain in the portable graph. + +The Cortex-M backend is broader than CMSIS-NN. In this model, the quantized +convolutions use CMSIS-NN, while quantize and dequantize use Cortex-M operators +with Arm Helium vector implementations. A `cortex_m.*` target therefore means +that the Cortex-M backend owns the operator, not necessarily that CMSIS-NN +implements it. + +{{% notice Important %}} +The Cortex-M backend is not a delegate. Delegation replaces a graph region with +a call to a separately compiled backend. Cortex-M lowering instead rewrites +supported patterns as registered `cortex_m.*` operators that the ExecuTorch +runtime dispatches normally. You will not see an `executorch_call_delegate` +node until the Ethos-U partitioner is added on the next page. +{{% /notice %}} + +## Export the model for Cortex-M + +Use the provided script to lower the model to Cortex-M only, by specifying `--mode cortex-m`: + +{{< tabpane code=true >}} + {{< tab header="Linux and macOS" language="bash">}} +python export_cortex_m_fallback.py \ + --mode cortex-m \ + --model-file model/model.py \ + --output-dir artifacts + {{< /tab >}} + {{< tab header="Windows PowerShell" language="powershell">}} +python .\export_cortex_m_fallback.py ` + --mode cortex-m ` + --model-file model/model.py ` + --output-dir artifacts + {{< /tab >}} +{{< /tabpane >}} + +The relevant output is: + +```output +Relevant targets before Cortex-M lowering: +aten.convolution.default: 2 +aten.topk.default: 1 + +Relevant targets after Cortex-M lowering: +aten.topk.default: 1 +cortex_m.dequantize_per_tensor.default: 2 +cortex_m.quantize_per_tensor.default: 2 +cortex_m.quantized_conv2d_nhwc.default: 2 +cortex_m.transpose.default: 1 +``` + +Before lowering, the graph contained two general convolution operations (Stride-1 and Stride-4). The `aten` prefix identifies PyTorch’s ATen operator namespace. These operators describe the computation, but not the hardware or kernel that will perform it. Lowering replaces both with `cortex_m.quantized_conv2d_nhwc`, an INT8 convolution using NHWC layout. + +`aten.topk` is unchanged, so it will use the portable CPU kernel. The `cortex_m.quantize_*` +and `cortex_m.dequantize_*` operators mark boundaries between floating-point (FP32) and INT8 +regions. + +## What you've accomplished and what's next + +You have seen how Cortex-M lowering replaces supported patterns with optimized +Cortex-M operators. The convolutions use CMSIS-NN, quantize and dequantize use +Helium vector implementations, and `topk` remains an ExecuTorch portable +kernel. + +Next, you will see how CMSIS-NN provides optimized CPU fallback when deploying a +model to an Ethos-U NPU. The resulting graph combines Ethos-U delegation, +CMSIS-NN-backed Cortex-M operators, and a portable CPU kernel. diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/3-add-ethosu-fallback.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/3-add-ethosu-fallback.md new file mode 100644 index 0000000000..c9e4ced73a --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/3-add-ethosu-fallback.md @@ -0,0 +1,176 @@ +--- +title: Add Ethos-U delegation and CPU fallback +weight: 4 +description: Partition TinyCNN for Ethos-U, lower the remaining convolution to Cortex-M, and generate the model-specific AI layer on the Ethos-U55 demonstration target. + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Deploying across Cortex-M and Ethos-U NPU + +We can re-use the same script as on the previous page, this time using 'Fallback mode'. This uses `EthosUPartitioner` to +create an Ethos-U55 delegate region, represented by `executorch_call_delegate`. +The exporter then applies `CortexMPassManager` to operations left outside that +delegate, lowering its supported convolution to a registered Cortex-M operator +implemented by CMSIS-NN. + +For supported neural-network operations, the expected order is Ethos-U NPU, +then CMSIS-NN on the Cortex-M CPU, then a portable CPU kernel. This keeps as much +work as possible on the NPU and gives unsupported quantized work an optimized +CPU route before using a general implementation. + +![Flow diagram showing that the graph is quantized, supported work is partitioned for Ethos-U, the supported remainder is lowered to Cortex-M operators, and the final remainder stays portable. This order preserves eligible NPU work before CPU lowering.#center](images/executorch-export-flow.svg "ExecuTorch partitioning and lowering order") + +Ordering matters. If Cortex-M lowering runs first, it could rewrite a +convolution that should be offered to the NPU. + +The exporter first creates an Ethos-U55 compile specification and configures +the target-specific quantizer: + +```python +compile_spec = EthosUCompileSpec(target=args.target) +quantizer = EthosUQuantizer(compile_spec) +quantizer.set_global( + get_symmetric_quantization_config(is_per_channel=True) +) +``` + +After quantization, `EthosUPartitioner` identifies supported regions and lowers +them into an Ethos-U delegate: + +```python +edge_program_manager = to_edge_transform_and_lower( + quantized_exported_program, + partitioner=[EthosUPartitioner(compile_spec)], + compile_config=EdgeCompileConfig(_check_ir_validity=False), +) +``` + +The exporter then applies Cortex-M lowering to the operations that remain +outside the delegate: + +```python +edge_program_manager._edge_programs["forward"] = CortexMPassManager( + edge_program_manager.exported_program(), use_explicit_layout=True +).transform() +``` + +This run uses the `ethos-u55-128` target to create the predictable fallback +described earlier. + +Run the export script in mixed Cortex-M + Ethos-U mode: + +{{< tabpane code=true >}} + {{< tab header="Linux and macOS" language="bash">}} +python export_cortex_m_fallback.py \ + --mode ethos-u-fallback \ + --target ethos-u55-128 \ + --model-file model/model.py \ + --output-dir artifacts + {{< /tab >}} + {{< tab header="Windows PowerShell" language="powershell">}} +python .\export_cortex_m_fallback.py ` + --mode ethos-u-fallback ` + --target ethos-u55-128 ` + --model-file model/model.py ` + --output-dir artifacts + {{< /tab >}} +{{< /tabpane >}} + +The export script prints a summary of the graph before and after Cortex-M lowering: + +```output +Relevant targets before Cortex-M lowering: +aten.convolution.default: 1 +aten.topk.default: 1 +executorch_call_delegate: 1 + +Relevant targets after Cortex-M lowering: +aten.topk.default: 1 +cortex_m.dequantize_per_tensor.default: 2 +cortex_m.quantize_per_tensor.default: 2 +cortex_m.quantized_conv2d_nhwc.default: 1 +cortex_m.transpose.default: 2 +executorch_call_delegate: 1 +``` + +The final graph contains all three execution paths: +- `executorch_call_delegate` contains the stride-one convolution accepted by + Ethos-U55. +- `cortex_m.quantized_conv2d_nhwc` is the stride-four convolution rejected by + Ethos-U55 and lowered to CMSIS-NN. +- `aten.topk` remains a portable FP32 operation. + +## Using the CMSIS-ExecuTorch project workflow + +The CMSIS-Executorch project workflow uses `create_ai_layer.py` to export a model, generate a `.pte`, and select the source components required to execute it. + +We have instead used a separate script for the previous examples. By using `create_ai_layer.py` instead, we will benefit from the rest of the setup. + +The pre-existing `create_ai_layer.py` does not include CMSIS-NN lowering, so we will add this through a targeted patch that makes four key changes: + +- It adds `ARM::CMSIS-NN@8.0.0` to the solution's Pack list. +- It applies Cortex-M lowering after Ethos-U partitioning. +- It maps explicit-layout names such as `quantized_conv2d_nhwc` to their Pack + component names. +- It selects the CMSIS-NN library when the exported program contains a + CMSIS-NN-backed Cortex-M operator. + +Check and apply the patch, then inspect its small project diff: + +```console +git apply --check enable_cortex_m_fallback.patch +git apply enable_cortex_m_fallback.patch +git --no-pager diff -- cmsis-executorch-u55.csolution.yml create_ai_layer.py +``` + +No board, runner, or linker changes are needed. + +## Generate the target description + +Resolve the declared Packs and active target: + +```console +cbuild setup cmsis-executorch-u55.csolution.yml --active SSE-300-U55 --packs +``` + +This one command installs missing packs through the public Pack index and +writes three useful descriptions: + +- `cmsis-executorch-u55.cbuild-pack.yml` locks the resolved Pack versions. +- `cmsis-executorch-u55.cbuild-mlops.yml` describes the Cortex-M55, Ethos-U55, + Vela settings, and destination AI layer. +- `cmsis-executorch-u55.cbuild-idx.yml` indexes the resolved build context. + +## Generate the AI layer + +Use the MLOps description to export the model and write its CMSIS AI layer: + +{{< tabpane code=true >}} + {{< tab header="Linux and macOS" language="bash">}} +python create_ai_layer.py cmsis-executorch-u55.cbuild-mlops.yml + {{< /tab >}} + {{< tab header="Windows PowerShell" language="powershell">}} +python .\create_ai_layer.py .\cmsis-executorch-u55.cbuild-mlops.yml + {{< /tab >}} +{{< /tabpane >}} + +The relevant final output is similar to: + +```output +[ai_layer] 7608 byte program, operators: ['Cortex-M dequantize_per_tensor', 'Cortex-M quantize_per_tensor', 'Cortex-M quantized_conv2d', 'Cortex-M transpose', 'Portable topk'] +[ai_layer] wrote .../ai_layer/ai_layer.clayer.yml +``` + +The generator writes the final `model.pte`, converts it to C data for the +firmware, and records the runtime and operator components needed to execute it. + +## What you've accomplished and what's next + +You have produced a model-specific AI layer with one Ethos-U55 delegate, one +CMSIS-NN-backed fallback convolution, and one portable operation. The split is +decided ahead of time and stored in the ExecuTorch program. + +Next, you will inspect how the two published source Packs supply the code behind +those graph targets. diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/4-inspect-cmsis-pack.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/4-inspect-cmsis-pack.md new file mode 100644 index 0000000000..32f9a1ea06 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/4-inspect-cmsis-pack.md @@ -0,0 +1,111 @@ +--- +title: Inspect the generated CMSIS Pack selection +weight: 5 +description: Trace the mixed ExecuTorch graph to versioned runtime, backend, operator, and CMSIS-NN source components. + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Understand what the CMSIS Pack provides + +The [ExecuTorch CMSIS Pack](https://www.keil.arm.com/packs/executorch-pytorch/overview/) is a *source Pack*, not a prebuilt runtime library. It supplies the ExecuTorch runtime, registration code, backends, and +operators as selectable CMSIS components. + +A `.pack` file is a ZIP-compatible archive. Its Pack Description file, or +PDSC, records component names, source files, dependencies, versions, and +conditions. + +CMSIS-Toolbox uses that metadata to add only the chosen components to the +application build. A CMSIS project can select only the components required by its lowered graph +instead of maintaining one large source list. Arm Compiler then compiles those sources for the Cortex-M55 +target with the rest of the project. + +## See which Packs were resolved + +CMSIS-Toolbox has already acquired the software Packs needed by the project. +Inspect the generated Pack lock: + +{{< tabpane code=true >}} + {{< tab header="Linux and macOS" language="bash">}} +cat cmsis-executorch-u55.cbuild-pack.yml + {{< /tab >}} + {{< tab header="Windows PowerShell" language="powershell">}} +Get-Content .\cmsis-executorch-u55.cbuild-pack.yml + {{< /tab >}} +{{< /tabpane >}} + +Among the resolved entries, you can see the two Packs responsible for the +mixed model: + +```output +- resolved-pack: ARM::CMSIS-NN@8.0.0 +- resolved-pack: PyTorch::ExecuTorch@1.5.1 +``` + +They have different responsibilities: + +- `PyTorch::ExecuTorch@1.5.1` supplies the ExecuTorch runtime, Ethos-U backend, + Cortex-M operator wrappers, registration code, and portable operators. +- `ARM::CMSIS-NN@8.0.0` supplies the optimized neural-network kernels called by + the Cortex-M convolution wrapper. + +The Pack lock records exact resolved versions so later builds don't silently +move to newer installed Packs. + +## Trace the graph into the AI layer + +The AI layer is a generated CMSIS software layer that embeds the model and declares the runtime, backends, operators, and libraries needed to execute it. + +Open the AI layer generated from the `.pte` operator list: + +{{< tabpane code=true >}} + {{< tab header="Linux and macOS" language="bash">}} +cat ai_layer/ai_layer.clayer.yml + {{< /tab >}} + {{< tab header="Windows PowerShell" language="powershell">}} +Get-Content .\ai_layer\ai_layer.clayer.yml + {{< /tab >}} +{{< /tabpane >}} + +Its component list includes: + +```yaml +components: + - component: Machine Learning:ExecuTorch:Runtime + - component: Machine Learning:ExecuTorch:Kernel Utils + - component: Machine Learning:ExecuTorch:Kernel Registration + - component: Machine Learning:ExecuTorch:Backend EthosU + - component: Machine Learning:ExecuTorch Operators:Cortex-M dequantize_per_tensor + - component: Machine Learning:ExecuTorch Operators:Cortex-M quantize_per_tensor + - component: Machine Learning:ExecuTorch Operators:Cortex-M quantized_conv2d + - component: Machine Learning:ExecuTorch Operators:Cortex-M transpose + - component: Machine Learning:ExecuTorch Operators:Portable topk + - component: ARM::CMSIS:NN Lib +``` + +This file is the handover between model export and firmware build: + +| Export result | Selected component | +| --- | --- | +| Ethos-U delegate | `Backend EthosU` | +| Cortex-M convolution | `Cortex-M quantized_conv2d` and `ARM::CMSIS:NN Lib` | +| Quantized boundaries and layout changes | Matching Cortex-M components | +| Portable `topk` | `Portable topk` | + +The runtime, kernel utilities, and registration components provide the common +machinery needed to load the `.pte` and dispatch its operators. + +You will also see `ARM_NN_ENABLE_F32` and `ARM_NN_ENABLE_F16` definitions in +the generated layer. CMSIS-NN 8.0.0 needs them because its Pack component +compiles optional floating-point source files. The fallback convolution in +this model remains INT8; these definitions don't change its data type. + +## What you've learned and what's next + +You have followed graph targets into two versioned source Packs and a concrete +component list. Pack acquisition, dependency resolution, and component +selection are now one connected workflow rather than separate downloads. + +Next, you will see how the reusable CMSIS project combines this generated AI +layer with the Corstone-300 board and application layers. diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/5-prepare-cmsis-runner.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/5-prepare-cmsis-runner.md new file mode 100644 index 0000000000..4f3a3eaa5d --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/5-prepare-cmsis-runner.md @@ -0,0 +1,114 @@ +--- +title: Understand the reusable CMSIS project +weight: 6 +description: See how the solution, application project, board layer, and generated AI layer form the Corstone-300 runner. + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Understand the solution file + +`cmsis-executorch-u55.csolution.yml` defines the complete Ethos-U55 build context. +Open it and inspect: + +```yaml +target-types: + - type: SSE-300-U55 + board: V2M-MPS3-SSE-300-FVP + device: SSE-300-MPS3 + variables: + - Board-Layer: $SolutionDir()$/board/Corstone-300/Board-U55.clayer.yml + - AI-Layer: $SolutionDir()$/ai_layer/ai_layer.clayer.yml +``` + +`SSE-300-U55` is the project's readable target name. The board and device +values identify records supplied by installed Packs. CMSIS-Toolbox uses those +identities to resolve startup files, memory information, and device conditions. + +The solution's `mlops:` node is the single source for the NPU and Vela target +settings: + +```yaml +mlops: + npu: + type: Ethos-U55 + macs: 128 + vela: + system: Ethos_U55_High_End_Embedded + memory: Shared_Sram + model: + clayer: $AI-Layer$ + name: TinyCNN + input-shape: 1x16x16x3 + calibration-samples: 2 +``` + +`cbuild setup` resolves this into the `*.cbuild-mlops.yml` file consumed by +`create_ai_layer.py`. The exporter therefore doesn't duplicate the project's +NPU configuration. + +## Follow the layers into the project + +`cmsis-executorch.cproject.yml` owns the application and consumes two layers: + +```yaml +groups: + - group: App + files: + - file: ./src/app_main.cpp + - file: ./src/arm_embedded_module.cpp + - file: ./src/arm_embedded_module.hpp + +layers: + - layer: $Board-Layer$ + type: Board + - layer: $AI-Layer$ + type: AI +``` + +Each part has one job: + +- The Board layer supplies Cortex-M55 startup, the memory layout, standard + output, the Ethos-U55 driver, and Corstone-300 initialization. +- The generated AI layer supplies the embedded model and its exact ExecuTorch + and CMSIS-NN component selection. +- The App group initializes ExecuTorch, creates an input tensor, runs one + inference, and prints the ten output values. + +The FVP configuration in `board/Corstone-300/fvp_config_u55.txt` describes the +simulated hardware. It is separate from the Board layer because one configures +the simulator while the other builds software for the simulated target. + +## See how the PTE enters the firmware + +The AI-layer directory now contains these related model files: + +```text +ai_layer/model.pte +ai_layer/model_pte.c +ai_layer/model_pte.h +``` + +`model.pte` is the serialized ExecuTorch program. `model_pte.c` contains the +same bytes represented as a C array, which is why it appears to be a long list +of numbers. Embedding the bytes lets a bare-metal system load the program +without a filesystem. + +`src/app_main.cpp` includes `model_pte.h`, wraps a deterministic +`1 x 16 x 16 x 3` input buffer as an ExecuTorch tensor, and calls the runtime. +The runtime follows the PTE execution plan: it invokes the Ethos-U backend for +the delegate and CPU kernels for the remaining operations. + +Because the adapted model preserves the original input shape and ten-value +output, the runner doesn't need model-specific edits. A different model shape +or output contract would need corresponding changes in `app_main.cpp`. + +## What you've learned and what's next + +You have traced the ready-made project from target selection to the embedded +PTE. The repository removes board bring-up work from the lesson while keeping +the CMSIS structure visible and reusable. + +Next, you will compile these parts, verify which Pack sources were used, and run +the image on the Corstone-300 FVP. diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/6-consume-cmsis-pack.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/6-consume-cmsis-pack.md new file mode 100644 index 0000000000..89e6dd7139 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/6-consume-cmsis-pack.md @@ -0,0 +1,122 @@ +--- +title: Build and run the mixed application +weight: 7 +description: Compile the selected Pack sources, verify their provenance, and run the application on the Corstone-300 Ethos-U55 FVP. + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Build the application + +In VS Code's **CMSIS** view, confirm that +`cmsis-executorch-u55.csolution.yml` is the active solution and +`SSE-300-U55` is the selected target. Select **Build** to compile the application +using the AI layer generated in chapter 3. + +CMSIS-Toolbox combines the App, Board, and AI layers, resolves their selected +components, and invokes Arm Compiler for Embedded. Check that the build task +finishes successfully without errors. Confirm that the build produces `cmsis-executorch.axf` and +`cmsis-executorch.hex` under: + +```text +out/cmsis-executorch/SSE-300-U55/Debug/ +``` + +The `.axf` file contains the linked executable and debug information. The +`.hex` file contains the image loaded into the FVP. + +## Verify the compiled source paths + +A successful build proves that an image was produced, but it doesn't by itself +show which implementation supplied each operation. In the VS Code terminal, +with `.venv` activated and the project root as the current directory, run the +verification helper against the compilation database: + +{{< tabpane code=true >}} + {{< tab header="Linux and macOS" language="bash">}} +python verify_pack_sources.py \ + out/cmsis-executorch/SSE-300-U55/Debug/compile_commands.json + {{< /tab >}} + {{< tab header="Windows PowerShell" language="powershell">}} +python .\verify_pack_sources.py ` + .\out\cmsis-executorch\SSE-300-U55\Debug\compile_commands.json + {{< /tab >}} +{{< /tabpane >}} + +The expected output is: + +```output +PASS: ExecuTorch runtime was compiled from its declared CMSIS Pack +PASS: ExecuTorch Cortex-M convolution wrapper was compiled from its declared CMSIS Pack +PASS: CMSIS-NN convolution kernel was compiled from its declared CMSIS Pack +PASS: Ethos-U backend was compiled from its declared CMSIS Pack +PASS: ExecuTorch portable topk kernel was compiled from its declared CMSIS Pack +``` + +These checks distinguish the two layers of the optimized fallback. The +ExecuTorch Pack provides the `quantized_conv2d` wrapper, while the CMSIS-NN +Pack provides the convolution implementation that wrapper calls. + +Together with `cmsis-executorch-u55.cbuild-pack.yml`, the compilation database +shows both the resolved Pack versions and the source files actually compiled. + +## Run the image on the FVP + +In the same VS Code terminal, start the Corstone-300 model and load the generated +HEX image. On macOS, use the Docker wrapper configured during setup: + +{{< tabpane code=true >}} + {{< tab header="Linux and Windows" language="console">}} +FVP_Corstone_SSE-300_Ethos-U55 -f board/Corstone-300/fvp_config_u55.txt -a out/cmsis-executorch/SSE-300-U55/Debug/cmsis-executorch.hex --simlimit 60 + {{< /tab >}} + {{< tab header="macOS" language="bash">}} +FVP_MODEL=FVP_Corstone_SSE-300_Ethos-U55 bash .vscode/fvp.sh -f board/Corstone-300/fvp_config_u55.txt -a out/cmsis-executorch/SSE-300-U55/Debug/cmsis-executorch.hex --simlimit 60 + {{< /tab >}} +{{< /tabpane >}} + +The relevant output is: + +```output +Ethos-U version info: + Arch: v1.1.0 + MACs/cc: 128 + Cmd stream: v0 +ExecuTorch Ethos-U55-128 application: 7608 byte model +Output: 10 element(s): 0.0532 -0.0239 0.0541 0.1064 0.0706 0.0395 0.0881 0.1340 0.0807 0.0064 +Test_result: PASS +``` + +{{% notice Note %}} +On Windows, the FVP can print warnings that `python3.dll` is unavailable and +that the VIO or VSI bridges are disabled. This headless example doesn't use +those bridges. If the run reaches `Test_result: PASS`, inference completed +successfully. +{{% /notice %}} + + + +The FVP result proves functional execution, not physical-device latency or +power. Measure the complete graph on your target hardware before drawing +performance conclusions because delegation boundaries and tensor conversions +also have a cost. + +## What you've accomplished + +You have built and run one ExecuTorch program across three execution paths, and used CMSIS-ExecuTorch to make setup easier: + +- The supported convolution runs in an Ethos-U55 delegate. +- The unsupported stride-four convolution runs on Cortex-M55 through the + ExecuTorch wrapper and optimized CMSIS-NN kernel. +- `topk` runs through the portable ExecuTorch CPU kernel. + +You also verified the implementation. The exported graph shows ownership, the compilation database shows the +compiled Pack sources, and the FVP confirms that the combined firmware runs. + +This build uses Ethos-U55 to make one unsupported convolution predictable. The same +Pack workflow applies to another Ethos-U target when its delegated graph leaves +operations for Cortex-M execution. + +To adapt the project, replace `model/model.py`, provide representative +calibration inputs, preserve or update the runner's tensor interface, and +regenerate the AI layer before rebuilding. diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/_index.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/_index.md new file mode 100644 index 0000000000..db909cbdc8 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/_index.md @@ -0,0 +1,84 @@ +--- +title: Optimize ML model deployment on microcontrollers with CMSIS-NN kernels + +description: Build an ExecuTorch 1.5.1 application from the published ExecuTorch CMSIS Pack, that uses CMSIS-NN for optimized Cortex-M fallback. + +minutes_to_complete: 60 + +who_is_this_for: This Learning Path is for embedded machine learning developers building ExecuTorch applications for Cortex-M CPUs and Ethos-U NPUs. + +learning_objectives: + - Export an ExecuTorch graph with Ethos-U, CMSIS-NN, and portable-kernel execution regions + - Lower quantized CPU fallback operations to optimized CMSIS-NN-backed Cortex-M operators, and understand the benefits in doing so + - Install and consume the published ExecuTorch CMSIS Pack for version 1.5.1 with selected runtime and operator components + - Run an application built from the ExecuTorch CMSIS Pack on a Corstone-300 Ethos-U55 FVP + - Understand when and why to use the ExecuTorch CMSIS Pack + +prerequisites: + - Git, VS Code, and Python 3.10 to 3.13 on Windows, macOS, or x86-64 Linux + - Docker Desktop for FVP simulation on macOS + - An internet connection and command-line access + - Familiarity with PyTorch + - Basic familiarity with Cortex-M processors and Ethos-U NPUs + +author: +- Matt Cossins +- Kwashie Andoh + +generate_summary_faq: false +rerun_summary: false +rerun_faqs: false +test_maintenance: false + +### Tags +skilllevels: Introductory +subjects: ML +armips: + - Cortex-M + - Ethos-U +operatingsystems: + - Linux + - Windows + - macOS +tools_software_languages: + - Python + - PyTorch + - ExecuTorch + - CMSIS + - CMSIS Pack + - CMSIS-NN + - FVP + - vcpkg + +further_reading: + - resource: + title: CMSIS-Executorch example project + link: https://github.com/Arm-Examples/CMSIS-Executorch + type: documentation + - resource: + title: Published ExecuTorch CMSIS Pack + link: https://www.keil.arm.com/packs/executorch-pytorch/overview/ + type: documentation + - resource: + title: ExecuTorch Arm Cortex-M backend + link: https://docs.pytorch.org/executorch/stable/backends/arm-cortex-m/arm-cortex-m-overview.html + type: documentation + - resource: + title: ExecuTorch Arm Ethos-U backend + link: https://docs.pytorch.org/executorch/stable/backends-arm-ethos-u.html + type: documentation + - resource: + title: CMSIS-Toolbox MLOps integration + link: https://open-cmsis-pack.github.io/cmsis-toolbox/build-overview/#mlops-information + type: documentation + - resource: + title: CMSIS-NN documentation + link: https://arm-software.github.io/CMSIS-NN/latest/ + type: documentation + +### FIXED, DO NOT MODIFY +# ================================================================================ +weight: 1 +layout: "learningpathall" +learning_path_main_page: "yes" +--- diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/_next-steps.md b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/_next-steps.md new file mode 100644 index 0000000000..e20dfa6d43 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/_next-steps.md @@ -0,0 +1,8 @@ +--- +# ================================================================================ +# FIXED, DO NOT MODIFY THIS FILE +# ================================================================================ +weight: 21 +title: "Next Steps" +layout: "learningpathall" +--- diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/images/cortex-m-execution-picture.svg b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/images/cortex-m-execution-picture.svg new file mode 100644 index 0000000000..1b1aa5e2a0 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/images/cortex-m-execution-picture.svg @@ -0,0 +1,369 @@ + + + One program. Three execution paths. + A simplified view of the simulated Corstone-300 target in the FVP. The Cortex-M CPU runs the ExecuTorch runtime and executes model.pte. It dispatches the stride-one convolution to the separate Ethos-U55 NPU. On the same Cortex-M core, CMSIS-NN runs the stride-four convolution and the portable FP32 kernel runs topk. CPU and NPU both connect to shared SRAM. The connections are schematic and execution routes are chosen during export. + Architecture references: https://arm-software.github.io/CMSIS-Ethos-U/main/general/index.html ; https://docs.pytorch.org/executorch/stable/backends/arm-cortex-m/arm-cortex-m-overview.html + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Target + + + + + + Cortex-M CPU + ExecuTorch runtime + model.pte + + + 2 + + 3 + + + + Ethos-U55 NPU + + + 1 + Stride-1 convolution + Stride-4 convolution + topk operation + Supported by CMSIS-NN + Supported by Executorch Portable kernels + Dispatch + + + + + Shared SRAM + Input, intermediate tensors, output + + diff --git a/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/images/executorch-export-flow.svg b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/images/executorch-export-flow.svg new file mode 100644 index 0000000000..2af601cb4f --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/optimize-executorch-fallback-cortex-m/images/executorch-export-flow.svg @@ -0,0 +1,76 @@ + + + ExecuTorch export and lowering flow + A four-stage flow from quantizing the graph, through Ethos-U partitioning and Cortex-M lowering, to the portable operations that remain. + + + + + + + + + + + + + Quantize the graph + + + + + + Ethos-U partition + + + + + + Cortex-M lowering + + + + + + + Portable operations + remain + +