Deploy MobileSAM on Alif Ensemble E8 with ExecuTorch

content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/_index.md

@@ -0,0 +1,65 @@
+---
+title: Deploy MobileSAM on Alif Ensemble E8 with ExecuTorch
+description: Export a quantized MobileSAM model and build a Zephyr application that segments live camera frames on the Alif Ensemble E8.
+minutes_to_complete: 120
+draft: true
+cascade:
+  draft: true
+who_is_this_for: This Learning Path is for embedded ML developers who want to deploy live MobileSAM segmentation on an Alif Ensemble E8 using ExecuTorch and Zephyr.
+learning_objectives:
+  - Trace the fixed-prompt and quantized tensor contracts from model export to live camera inference
+  - Export a MobileSAM model for the E8 HP core and Ethos-U85-256
+  - Build a Zephyr application within its scratch and application RAM budgets
+  - Program the model and firmware and validate live camera capture, inference, and display output
+prerequisites:
+  - An [Alif Ensemble E8 Development Kit](https://alifsemi.com/support/kits/ensemble-e8devkit/) with its MT9M114 selfie camera and display attached
+  - An RGB calibration image representative of the camera scenes you want to segment
+  - Experience with PyTorch model export, C++, and Zephyr builds
+  - An Apple silicon Mac with Python 3.12, Git, CMake, Ninja, and a C++ build toolchain
+  - Zephyr SDK 0.17.0, west, Alif Security Toolkit, SEGGER J-Link, and Expect
+author: Usamah Zaheer
+generate_summary_faq: false
+rerun_summary: false
+rerun_faqs: false
+skilllevels: Advanced
+subjects: ML
+armips:
+  - Ethos-U
+tools_software_languages:
+  - ExecuTorch
+  - PyTorch
+  - MobileSAM
+  - Zephyr
+  - Python
+  - CMake
+operatingsystems:
+  - macOS
+  - RTOS
+further_reading:
+  - resource:
+      title: Run MobileSAM prompt segmentation on Arm Ethos-U85 with ExecuTorch
+      link: /learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-ethos-u/
+      type: documentation
+  - resource:
+      title: Set up the Alif Ensemble E8 board
+      link: /learning-paths/embedded-and-microcontrollers/alif-image-classification/board-setup/
+      type: documentation
+  - resource:
+      title: Alif Ensemble E8 DevKit support page
+      link: https://alifsemi.com/support/kits/ensemble-e8devkit/
+      type: website
+  - resource:
+      title: ExecuTorch Arm Ethos-U backend documentation
+      link: https://docs.pytorch.org/executorch/stable/embedded-arm-ethos-u.html
+      type: documentation
+  - resource:
+      title: MobileSAM source repository
+      link: https://github.com/ChaoningZhang/MobileSAM
+      type: repository
+
+### FIXED, DO NOT MODIFY
+# ================================================================================
+weight: 1
+layout: learningpathall
+learning_path_main_page: "yes"
+---

content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/_next-steps.md

@@ -0,0 +1,8 @@
+---
+# ================================================================================
+#       FIXED, DO NOT MODIFY THIS FILE
+# ================================================================================
+weight: 21                  # The weight controls the order of the pages. _index.md always has weight 1.
+title: "Next Steps"         # Always the same, html page title.
+layout: "learningpathall"   # All files under learning paths have this same wrapper for Hugo processing.
+---

content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/build-the-firmware.md

@@ -0,0 +1,61 @@
+---
+title: Build the Zephyr firmware within the E8 memory budget
+description: Build for the correct HP and U85 target and check model metadata, scratch capacity, and linker memory usage.
+weight: 5
+layout: learningpathall
+---
+
+## Build the matching application
+
+Build the supplied Zephyr sample with the pinned Alif SDK and the PTE/metadata pair from your export:
+
+```bash
+cd "$EXECUTORCH_ROOT"
+west build \
+  -d "$MOBILE_SAM_OUTPUT/build-mobilesam-live" \
+  -p always \
+  -b alif_e8_dk/ae822fa0e5597xx0/rtss_hp \
+  "$EXECUTORCH_ROOT/zephyr/samples/mobilesam-ethosu" -- \
+  -DZEPHYR_EXTRA_MODULES="$EXECUTORCH_ROOT;$SDK_ALIF" \
+  -DSDK_ALIF="$SDK_ALIF" \
+  -DET_PTE_FILE_PATH="$MOBILE_SAM_OUTPUT/mobilesam_live_384_grid4_multimask.pte" \
+  -DMODEL_METADATA_PATH="$MOBILE_SAM_OUTPUT/mobilesam_live_384_grid4_multimask.json" \
+  -DPython3_EXECUTABLE="$(command -v python)"
+```
+
+The board identifier is tied to the pinned vendor tree. Replacing it with a similar identifier from a newer Zephyr tutorial can select a different configuration. This sample sets `CONFIG_ARM_ETHOS_U85_256=y` and `ETHOSU_TARGET_NPU_CONFIG=ethos-u85-256` explicitly.
+
+CMake finds the official `serial_camera_mt9m114_selfie.overlay` in `sdk-alif` and adds the sample's deferred-initialization overlay. It generates model metadata and display-logo headers, inspects the PTE for unsupported CPU operators, and extracts the Ethos-U scratch requirement.
+
+## Understand the three allocator pools
+
+The supplied `prj.conf` sets:
+
+| Setting | Bytes | Purpose |
+| --- | ---: | --- |
+| `CONFIG_EXECUTORCH_METHOD_ALLOCATOR_POOL_SIZE` | 655,360 | Method state and planned buffers allocated by the sample |
+| `CONFIG_EXECUTORCH_TEMP_ALLOCATOR_POOL_SIZE` | 6,488,064 | Main Ethos-U tensor/scratch arena |
+| `CONFIG_EXECUTORCH_FAST_TEMP_ALLOCATOR_POOL_SIZE` | 393,216 | Fast scratch arena |
+
+CMake rejects a PTE whose extracted scratch requirement exceeds the configured main scratch pool. Passing that check does not prove the complete application fits: camera, display, stacks, method allocations, and linker regions also consume memory.
+
+The final documented clean build reported:
+
+| Linker region | Used bytes | Capacity bytes | Remaining bytes |
+| --- | ---: | ---: | ---: |
+| Application RAM | 7,580,784 | 7,602,176 | 21,392 |
+| SRAM0 | 768,064 | 786,432 | 18,368 |
+
+These figures describe the reference build. Read your own build's memory report and `zephyr.map`; changing toolchains, model shape, prompt count, logging, or buffer placement can exhaust the remaining space.
+
+## Keep model storage separate from firmware storage
+
+The PTE remains in memory-mapped external OSPI flash at `0xC0000000`. The firmware records its size and address and opens it with `BufferDataLoader`. It does not link the 13 MB model into the firmware image.
+
+Changing the external-flash address alone does not enlarge RAM or fix a linker overflow. Changing the model size also requires rebuilding the firmware so its compiled PTE size and generated metadata remain consistent.
+
+Before flashing, locate `build-mobilesam-live/zephyr/zephyr.elf`, inspect the linker report, and confirm that the build accepted your PTE. The next lesson programs model data and application firmware as separate operations.
+
+## What you've accomplished
+
+You've built the HP-core application for Ethos-U85-256, generated matching model metadata, and checked scratch and linker capacity. Next, program the model and firmware as separate artifacts.

content/learning-paths/embedded-and-microcontrollers/deploy-mobilesam-on-alif-e8/export-the-model.md

@@ -0,0 +1,152 @@
+---
+title: Export and inspect the live MobileSAM model
+description: Freeze four independent prompts, export int8 model I/O, and inspect delegation and metadata before building firmware.
+weight: 4
+layout: learningpathall
+---
+
+## Prepare the pinned MobileSAM source
+
+Keep the environment from the previous lesson active and run these commands from the patched ExecuTorch checkout. Its exporter has live-camera options that are absent from some later versions of the public FVP example.
+
+```bash
+cd "$EXECUTORCH_ROOT"
+python examples/arm/mobilesam_prompt_segmentation_example_ethos_u/model_export/prepare_mobilesam.py \
+  --source-dir "$MOBILE_SAM_SOURCE"
+```
+
+The preparation step uses MobileSAM revision `f706ad9c4eb7f219c00d9050e46328518ffb65d2` and applies the configurable-image-size patch. The exporter verifies the official checkpoint SHA256:
+
+```output
+6dbb90523a35330fedd7f1d3dfc66f995213d81b29a5ca8108dbcdd4e37d6c2f
+```
+
+## Select a calibration image
+
+Choose an RGB image representative of the camera scenes you want to segment. The exporter uses this image for both quantization calibration and its host comparison. Set its path:
+
+```bash
+export MOBILE_SAM_CALIBRATION_IMAGE=/absolute/path/to/calibration-image.jpg
+test -f "$MOBILE_SAM_CALIBRATION_IMAGE"
+```
+
+The original deployment image passed the unchanged quality gate with the released dependencies: selected-mask IoU `0.8060` and pixel agreement `0.9673`. The bundled `examples/models/dinov2/dog.jpg` failed this four-prompt configuration with selected IoU `0.7448` and agreement `0.9315`. Calibration input affects the result; require your own export to pass the gate before continuing.
+
+## Export four independent prompts
+
+Use the live-camera configuration:
+
+```bash
+python examples/arm/mobilesam_prompt_segmentation_example_ethos_u/model_export/export_mobilesam.py \
+  --output-path "$MOBILE_SAM_OUTPUT/mobilesam_live_384_grid4_multimask.pte" \
+  --mobile-sam-source "$MOBILE_SAM_SOURCE" \
+  --image "$MOBILE_SAM_CALIBRATION_IMAGE" \
+  --input-size 384 \
+  --point 96 96 \
+  --point 288 96 \
+  --point 96 288 \
+  --point 288 288 \
+  --multimask-output \
+  --quantized-io
+```
+
+The points sample the four image quadrants. In this multimask mode they are independent prompt batches, giving three candidate masks and associated quality scores per point. This is different from asking SAM for one mask constrained jointly by all four points.
+
+The image encoder and decoder run on the NPU. The prompt encoder runs during export, and its fixed embeddings are stored in the model. Re-export when you change prompt coordinates, input size, or the output contract.
+
+The export applies A8W8 globally and A16W8 to TinyViT attention, then lowers through the Arm backend and Vela for `ethos-u85-256`. The compiled configuration uses `Ethos_U85_SYS_DRAM_Mid` and `Dedicated_Sram_384KB`. Those are compiler settings; actual firmware storage and scratch placement still need to match the E8 memory map.
+
+## Inspect artifacts before continuing
+
+The exporter writes the `.pte` and matching `.json` metadata beside it, together with mask-agreement metrics, a delegation summary, and TOSA/Vela debug artifacts. Inspect them:
+
+```bash
+ls -lh "$MOBILE_SAM_OUTPUT"
+python -m json.tool "$MOBILE_SAM_OUTPUT/mobilesam_live_384_grid4_multimask.json"
+cat "$MOBILE_SAM_OUTPUT/mobilesam_live_384_grid4_multimask_metrics.json"
+cat "$MOBILE_SAM_OUTPUT/mobilesam_live_384_grid4_multimask_delegation.txt"
+```
+
+Require successful export and inspect the generated masks and both quality metrics. This exporter accepts selected-mask IoU at least `0.9` **or** pixel agreement at least `0.96`. It compares selected candidates at prompts whose predicted quantized quality reaches `0.9`, using all prompts as fallback when none reaches that threshold. This does not require `0.9` IoU for every candidate or every prompt. The device's later mask-selection threshold is `0.8`.
+
+Using the same image for calibration and evaluation does not measure segmentation accuracy on new scenes. Inspect the generated masks, then validate the programmed model on representative camera scenes. Your calibration image and released dependencies can produce a different artifact from the retained board binary; record your own model and metadata hashes.
+
+The live application expects this tensor interface:
+
+| Tensor | Shape | Type |
+| --- | --- | --- |
+| Image input | `[1,3,384,384]` | int8 |
+| Mask output | `[4,3,96,96]` | int8 |
+| Quality output | `[4,3]` | int8 |
+
+CMake rejects known CPU operators, while the runtime checks input type/byte count and some output dimensions. The metadata-only header generator does not validate `output_shape`. Check the actual PTE tensor interface against the JSON and this configuration before building:
+
+```bash
+python - <<'PY'
+import hashlib
+import json
+import math
+import os
+from pathlib import Path
+
+from executorch.exir._serialize._program import deserialize_pte_binary
+from executorch.exir.scalar_type import ScalarType
+from executorch.exir.schema import Tensor
+
+pte_path = Path(os.environ["MOBILE_SAM_OUTPUT"]) / "mobilesam_live_384_grid4_multimask.pte"
+json_path = pte_path.with_suffix(".json")
+metadata = json.loads(json_path.read_text())
+program = deserialize_pte_binary(pte_path.read_bytes()).program
+plan, = [p for p in program.execution_plan if p.name == "forward"]
+shapes = [[1, 3, 384, 384], [4, 3, 96, 96], [4, 3]]
+assert len(plan.inputs) == 1 and len(plan.outputs) == 2
+for index, shape in zip(plan.inputs + plan.outputs, shapes):
+    tensor = plan.values[index].val
+    assert isinstance(tensor, Tensor) and tensor.scalar_type == ScalarType.CHAR
+    assert tensor.sizes == shape, (tensor.sizes, shape)
+assert metadata["input_shape"] == shapes[0]
+assert metadata["output_shape"] == shapes[1]
+assert metadata["input_size"] == 384 and metadata["multimask_output"] is True
+assert metadata["point_prompts_xy"] == [[96, 96], [288, 96], [96, 288], [288, 288]]
+preprocessing = metadata["preprocessing"]
+assert preprocessing["resize"] == "longest_side_then_zero_pad"
+for key, expected in (("pixel_mean", [123.675, 116.28, 103.53]),
+                      ("pixel_std", [58.395, 57.12, 57.375])):
+    values = preprocessing[key]
+    assert len(values) == 3 and all(
+        math.isclose(value, reference, rel_tol=0, abs_tol=0.0001)
+        for value, reference in zip(values, expected)
+    )
+for name in ("input", "output", "quality"):
+    qparams = metadata["io_quantization"][name]
+    assert qparams["dtype"] == "int8"
+    assert math.isfinite(qparams["scale"]) and qparams["scale"] > 0
+    assert isinstance(qparams["zero_point"], int) and -128 <= qparams["zero_point"] <= 127
+print("PTE tensor interface and JSON configuration checks passed")
+for path in (pte_path, json_path):
+    print(hashlib.sha256(path.read_bytes()).hexdigest(), path.name)
+PY
+```
+
+Require the success message and retain both printed hashes with your export. An assertion failure means the files do not satisfy this configuration; resolve it before continuing. The PTE's tensor schema does not store quantization parameters, so this check cannot prove that the JSON scales and zero points belong to that PTE. Keep the matching PTE and JSON from the same export, alongside their hashes, and check target output during execution.
+
+An old `[1,1,112,112]` floating-point FVP result belongs to the earlier 448×448 single-mask configuration. It is not the output contract of this 384×384 live model.
+
+## Prepare an identical model file for programming
+
+J-Link and the recovery writer consume a raw `.bin` file. This is an unchanged copy of the PTE, without a new container or header:
+
+```bash
+cp "$MOBILE_SAM_OUTPUT/mobilesam_live_384_grid4_multimask.pte" \
+  "$MOBILE_SAM_OUTPUT/mobilesam_live_384_grid4_multimask.bin"
+cmp "$MOBILE_SAM_OUTPUT/mobilesam_live_384_grid4_multimask.pte" \
+  "$MOBILE_SAM_OUTPUT/mobilesam_live_384_grid4_multimask.bin"
+```
+
+The validated final binary is 13,293,120 bytes, with SHA256 `895098c246cfcf0ca1a20e9dd9834dd9f4cfc3906edcb1dc901c47139e4f779d`. A fresh export can differ with dependency versions or calibration. Record the size and hash of your own PTE rather than assuming the reference hash must match.
+
+Before proceeding, explain why `--quantized-io` matters: it removes the host-style floating-point I/O boundary from the device program, while the application performs camera normalization, quantization, and mask post-processing.
+
+## What you've accomplished
+
+You've exported the four-prompt model, checked delegation and quantized tensor metadata, and prepared an identical binary for programming. Next, build firmware with this PTE and JSON and check its memory use.