[metascroy]

This PR mainly adds testing for the Core AI backend: a multithreaded runtime stress test, and Core AI flows in the shared ExecuTorch operator and model test suites. It also fixes the smaller issues that testing found.

Multithreaded stress test

• runtime/test/coreai_multithread_test.cpp (CTest coreai_multithread_test) starts N threads. Each thread repeatedly creates an executorch::extension::Module, loads it, runs it and checks the output against expected values. Threads wait at a shared start gate so their loads overlap.
• Each run has two phases sharing a private cache directory: cold, where the cache starts empty, then warm, which reuses the cache. This covers both concurrent first-time setup and concurrent cache hits.
• runtime/test/export_stress_fixture.py exports the test model at build time: an fp16 residual MLP with a forced graph break, so the program has two Core AI delegates.
• COREAI_STRESS_THREADS / COREAI_STRESS_ITERATIONS set the load (default 4 × 10). CI runs 50 × 100.

Shared operator and model suites

• test/tester.py adds CoreAITester, with Core AI stages for quantization (CoreAIQuantize, through coreai_opt), lowering (CoreAILower, applying the default passes) and serialization. Each run gets a temporary coreai_cache_dir, so runs don't share state.
• backends/test/suite/flows/coreai.py registers four flows: coreai_fp32, coreai_fp16, coreai_w8a8 and coreai_w8. Quantized flows skip models with nothing to quantize.
• Skip lists cover cases that can't produce a valid result:
  • all flows: index_put writing the same position twice with accumulate=False, which PyTorch documents as undefined
  • fp16 only: avg_pool3d, which has no CPU float16 reference
  • fp16 only: inception_v3, whose randomly initialized weights overflow float16, so the reference is itself NaN
• setup.py / CMakeLists.txt build and package the Core AI delegate into the Python runtime when EXECUTORCH_BUILD_COREAI=ON, so the suites can run models on Core AI.

CI failure thresholds

.github/workflows/coreai.yml runs every suite on every flow (-n auto). The job reads the JSON report and fails if any test errors, no test passes, or more than the allowed number of tests fail:

Flow	Operators	Models
coreai_fp32	0	0
coreai_fp16	0	0
coreai_w8	0	0
coreai_w8a8	23	7

The zero-failure jobs stop at the first failure (-x). The w8a8 jobs run to completion so every failure is counted; lower their limits as the failures are fixed. The backend's Python unit tests now also run in parallel.

Fixes found by the new tests

• add/sub with alpha: the installed converter ignores a non-1 alpha on add/sub. The new DecomposeAddSubAlphaPass rewrites x ± alpha*y as a multiply followed by a plain add/sub. alpha becomes a lifted constant in the output dtype, so fp16 stays fp16.
• Implicit op arguments: the new NormalizeCoreAIOpArgsPass writes out arguments the converter misreads or crashes on when they're implicit:
  • mean.dim / amax / amin with dims omitted or None
  • avg_pool defaults, plus ceil_mode when it doesn't change the output shape
  • squeeze naming a dim whose size isn't 1, which PyTorch treats as a no-op
• Partitioner variant checks: the new is_coreai_supported_variant keeps ops the converter would crash on or compute wrongly out of the delegate (they run on CPU instead):
  • transposed 3D convolution
  • index_put with accumulate=True, dynamic non-batch dims, or integer indices that leave input dims unindexed (aborts the process in MPS scatter_nd)
  • non-1 alpha
  • reductions without explicit dims
  • avg_pool ceil_mode / count_include_pad=False cases the converter can't express
  • squeezes the converter can't lower
  • Python scalar operands or results
• Runtime: the delegate now passes INT32 and BOOL tensors in and out, not just FP16/FP32.
• Asset keys: the asset namespace (export_hash) is now a hash of the full bundle contents, not the SDK program hash. Two delegates with the same program but different metadata (for example creation timestamps) no longer collide in the NamedDataStore.
• I/O compatibility: the check now accepts dimensions that are dynamic on the Core AI side.

Test plan

• python -m pytest backends/apple/coreai -n auto: 151 passed, including new unit tests for both passes, the variant checks, asset deduplication and dynamic I/O dims.
• Native stress CTest passed locally, at both the default load and CI's 50 × 100.
• Operator and model suites run in CI on this PR under the thresholds above.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24374

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 2 Pending, 1 Unclassified Failure

As of commit 6009fe6 with merge base 822cb60 ():

NEW FAILURE - The following job has failed:

• pull / nxp-mcuxpresso-test / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1

UNCLASSIFIED FAILURE - DrCI could not classify the following job because the workflow did not run on the merge base. The failure may be pre-existing on trunk or introduced by this PR:

• Test ROCm AOTI / unittest-rocm-gfx950-rocm7.2 / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  backends/cuda/tests/test_antialiased_resize.py::TestAntialiasedResizeDecomposition::test_align_corners_compiles_with_dynamic_sizes

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[metascroy]

@claude review this PR

[metascroy]

@claude review this code

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[claude]

Claude finished @metascroy's task in 4m 26s —— View job

---

This adds a good test layer for Core AI and I found no blockers, but the README is out of date in three places, the suite CI has no per-test timeout, and two avg_pool partitioner cases may still be delegated wrongly.

Detailed review

PR Review: #24374

Summary

This PR adds a multithreaded runtime stress test, CoreAITester, four Core AI flows for the shared operator and model suites, and fixes for problems the tests found: two new passes, partitioner variant checks, INT32/BOOL I/O and content-hashed asset keys. Three reviewers went through the diff independently and I re-read each finding below against the code. None of the problems blocks merging. Most are documentation drift, CI hardening, and a few partitioner edge cases.

Code Quality

• backends/apple/coreai/README.md:481-484: the README lists coreai_w4 and says "the last two" flows are weight-only. backends/test/suite/flows/coreai.py registers only coreai_fp32, coreai_fp16, coreai_w8a8 and coreai_w8, so -m flow_coreai_w4 selects no tests. CoreAIQuantize in tester.py also still accepts "w4", and no flow uses it.

• backends/apple/coreai/README.md:495: the README tells readers to run COREAI_TEST_RUNTIME=1 python -m pytest backends/apple/coreai/test/test_tester.py -x. Commit 710078b removed that file, and COREAI_TEST_RUNTIME isn't referenced anywhere else.

• backends/apple/coreai/README.md:504-505: the README says CI "propagates test failures without a failure allowance". The workflow allows 23 operator failures and 7 model failures for coreai_w8a8.

• backends/apple/coreai/test/tester.py:53: getattr(QuantizerConfig.presets, self.mode)() is the dynamic getattr pattern that CLAUDE.md asks us to avoid. An explicit {"w8a8": ..., "w8": ...} mapping would also reject unknown modes with a clear error.

• backends/apple/coreai/partition/partitioner.py:169-179 (_avg_pool_supported): divisor_override (args[6]) is never checked. The comment above says the converter always pools as if count_include_pad=True, which suggests it ignores divisor_override too. If so, F.avg_pool2d(x, 2, divisor_override=3) is delegated and returns wrongly scaled output. Low confidence, because I didn't check the converter. Please confirm, and either reject the case or add a test.

• backends/apple/coreai/passes/normalize_op_args.py:85-87 (_floor_mode_shape_matches): n = len(kernel_size) is not the spatial rank. If the edge graph keeps a length-1 kernel such as avg_pool2d(x, (2,), ceil_mode=True) without expanding it, only the last spatial dim is compared:

  • Example input (1, C, 5, 4): W matches under floor mode, H doesn't (3 vs 2).
  • ceil_mode is cleared anyway and the node is delegated with the wrong shape.

  Low confidence, because it depends on whether [2] survives to the edge graph. Using the op's spatial rank (2 or 3) and _expand(kernel_size, n) would remove the doubt.

• backends/apple/coreai/passes/decompose_add_sub_alpha.py:92: the cast condition is torch.result_type(other_val, alpha_val) != out_dtype. With an f32 x and an int32 y (the int_other_float_result case), promotion already gives f32, so no cast is inserted and the emitted op is mul(i32, f32). That contradicts the "Core AI never sees a mixed-dtype binary op" invariant stated in test_fp16_operands_share_result_dtype. Checking other_val.dtype != out_dtype is simpler and covers this case. Minor.

Infrastructure

• .github/workflows/coreai.yml:141-167: pytest-timeout is installed, but no --timeout is passed, and -c /dev/null discards any config default. A single hung Core AI compile or MPS execute stalls its xdist worker until the job's 120-minute limit. With 8 matrix jobs at max-parallel: 2, one hang can hold up PR signal for hours. Please add --timeout=<N>.
• .github/workflows/coreai.yml:159-170: the case lists all 8 matrix combinations, so the *) MAX_FAILED="" branch and the "informational" path behind it can never run. They also fail open: a new flow or suite added to the matrix without a case entry would always pass. I'd make the default fail the job, or drop the branch.
• .github/workflows/coreai.yml:151,167: GOLDEN_ARTIFACTS_DIR and --junitxml=coreai-results.xml are set, but no step uploads or reads them. GOLDEN_ARTIFACTS_DIR makes the suite serialize a .pte and reference outputs for every passing test, which is extra time and disk on the macOS runner for no benefit. Either drop both, or upload them if they're meant to be kept. This doesn't use GitHub Actions storage; it's only wasted work.
• backends/apple/coreai/CMakeLists.txt:495: TIMEOUT 240 looks sized for the 4 × 10 default. CI runs 2 phases × 50 threads × 100 iterations, about 10,000 two-delegate inferences, and the cold phase has 50 threads filling the cache at once. You mention it passed locally at 50 × 100. On a slower shared runner it could time out intermittently. Consider having the workflow set the timeout, or scaling it with the env values. Low confidence; I didn't time it.

Testing

• backends/apple/coreai/runtime/test/coreai_multithread_test.cpp (main, the two phases): nothing verifies that the warm phase actually hits the cache. If the cache stops being written, or coreai_cache_dir stops reaching the delegate through LoadBackendOptionsMap, both phases compile cold and the test still passes, so the cold/warm coverage the description promises isn't enforced. A check that the cache directory is non-empty after phase 1, or that phase 2 adds no new entries, would close that gap.
• Same file: each worker builds its own Module, so concurrent execute on one shared Module/method is never exercised. Only concurrent load and cache fill are stressed. If sharing a loaded program across threads is meant to be supported, it needs its own case. Otherwise, say in the test header that it isn't covered.

Recommendation

Approve

Nothing here blocks merging. Before or soon after merge I'd fix the README drift, the missing pytest --timeout, and the fail-open MAX_FAILED default. The two avg_pool partitioner items are worth confirming against the converter.