[baominghelly]

Current head: 8804b5495ed5979b5f6890039bfd699786a8bbdb. See the head-specific CI results below.

Summary

• Install declared build dependencies before non-isolated builds and select MetaX/Moore compilers before enabling CXX, preserving the InfiniRT prefix.
• Correct Iluvatar dispatch include paths, Ascend SOC discovery, and NVIDIA QuickGelu compilation for SM52 while retaining native FP16 arithmetic on SM53+.
• Reuse master's operator dispatch workaround unchanged: src/operator.h is identical to master.
• Align Moore's image with its MUSA 4.3.6 driver, propagate allocated GPUs to Torch, and disable TF32 only for the FP32 GEMM reference.
• Explicitly set gpu_ids: auto on all seven platforms. Retain device counts, idle thresholds, locks and execution budgets; Ascend/Iluvatar retain 600-second queue budgets.
• Align Shadow's workflow/tools with ordinary CI at 901566229509ac54f520210c6856bf163209f907, including the Ascend910 physical-chip/process parser fix.
• For Iluvatar Torch 2.4, use ATen math SDPA for FP32 or unequal query/value head widths, converting boolean masks to additive masks. Preserve caller streams and global backend flags.
• Compare the FP32 SDPA regression against CPU FP64 with temporary full-FP32 test settings. Cover equal and unequal head widths; skip only muDNN's unsupported unequal-width combinations, retaining equal-width coverage on Moore.
• Merge master 673ca669, retaining its stream/provider changes and TASK_QUEUE_ENABLE=0.

Motivation

Accelerator CI exposed dependency, compiler initialization, generated-header, SOC discovery and NVIDIA compilation failures. Moore also required a compatible runtime image and full-FP32 reference. Shadow's older Ascend detector could not parse the host's Ascend910 label, while fixed device IDs unnecessarily constrained allocation.

Master's new external-stream test exposed a separate SDK bug on iluvatar13 (tg200): in the exact CI image, pure Torch 2.4.1 SDPA produces incorrect FP32 output on default and side streams. FP16/BF16 also produce incorrect output or NaNs when query/value head widths differ. ATen math matches an independent CPU FP64 reference. The compatibility path is restricted to this backend, version and input domain; equal-width FP16/BF16 retain fused dispatch.

Type of Change

• feat — new feature / new operator / new platform
• fix — bug fix
• perf — performance improvement (no behavioral change)
• refactor — code restructuring without behavior change
• test — adding or fixing tests only
• docs — documentation only
• build / ci — build system or CI configuration
• chore — tooling, formatting, or other non-code changes
• Breaking change

Platforms Affected

• CPU (WITH_CPU)
• NVIDIA (WITH_NVIDIA)
• Iluvatar (WITH_ILUVATAR)
• MetaX (WITH_METAX)
• Cambricon (WITH_CAMBRICON)
• Moore (WITH_MOORE)
• Ascend (WITH_ASCEND)
• PyTorch C++ bindings (WITH_TORCH)
• Build system / CMake / CI
• Python bindings / user-facing API

Hygon also has an explicit automatic-device setting but is outside the active CI matrix.

Smoke Test Result

The final source was built and installed in the actual CI images on iluvatar13, metax49 and moore97, each with one idle GPU held under the same host device locks as CI. Images were infiniops-ci/iluvatar:df-f3dee09e167e, infiniops-ci/metax:df-287c3ccc968a, and infiniops-ci/moore:df-53201cdeba16. Validation used InfiniRT cd1d48258f0a6b8d9f66111d5d6cb1664205e78d and nlohmann/json v3.12.0, supplied as local archives after GitHub dependency downloads timed out.

python3 -m pip install . --no-build-isolation --no-deps \
  --config-settings=cmake.define.INFINI_OPS_SMOKE_BUILD=ON \
  --config-settings=cmake.define.INFINI_RT_ROOT=/tmp/validation-rt-prefix \
  --config-settings=cmake.define.FETCHCONTENT_SOURCE_DIR_NLOHMANN_JSON=/tmp/validation-json
# Successfully built and installed InfiniOps-0.1.0
python3 -m pytest tests/test_scaled_dot_product_attention.py -m smoke -q --tb=short
# Iluvatar: 26 passed, 2 skipped, 36 deselected
# MetaX:    26 passed, 2 skipped, 36 deselected
# Moore:    20 passed, 8 skipped, 36 deselected
python3 -m pytest tests/ -m smoke -n 4 -q --tb=short
# Iluvatar: 172 passed, 66 skipped in 18.82s
# MetaX:    168 passed, 68 skipped in 22.70s
# Moore:    167 passed, 79 skipped in 35.83s

Both formerly failing stream cases pass on all three platforms. New FP32 cases compare mask types, scale, GQA and equal/unequal head widths against independent CPU FP64 softmax/matmul, without relaxing tolerances. The test reference avoids the Iluvatar SDK's unsupported fused domain and handles the pre-Torch-2.5 GQA signature. The preceding source passed the complete Iluvatar SDPA module (50 passed, 2 skipped); the final source was validated with the SDPA smoke subset and complete smoke suite above.

The first independent FP64 test exposed TF32 rounding on MetaX and a muDNN shape restriction on Moore. In pure Torch inside the MetaX CI image, disabling TF32 reduced maximum error from 5.5e-4 to 2.8e-7. The regression temporarily disables TF32 and restores the prior precision setting. In the Moore CI image, pure Torch rejects unequal query/value head widths; these six unsupported cases are explicitly skipped and six equal-width accelerator cases are retained. Neither finding expands the production Iluvatar workaround to other platforms.

An isolated harness using the pinned CI tools passed idle selection, locked-device exclusion, device visibility and no-idle rejection for all six active platforms. Hygon's explicit auto setting passed matrix conversion. Hardware queries were mocked with temporary locks for this harness; the Iluvatar tests above used an actual leased GPU.

Local Clang Format 21.1.8 and Ruff 0.15.22 checks passed. Final-head GitHub Clang Format, Ruff and documentation checks also passed. The earlier merge source passed 103 build/configuration/code-generation regressions, including real CUDA 12.6 compilation for SM52/SM80; that result predates the final SDPA change.

Test Results on Supported Platforms

Platform	Affected	Build / Smoke Result	Full Result / Notes
NVIDIA	Yes	Final-head ordinary/Shadow CI running; allocation harness passed	Earlier merged-source NVCC regressions passed; full suite not run
Iluvatar	Yes	Final-head ordinary/Shadow CI passed; local smoke: 172 passed, 66 skipped	SDPA smoke: 26 passed, 2 skipped; whole operator suite not run
MetaX	Yes	Final-head ordinary/Shadow CI passed; local smoke: 168 passed, 68 skipped	SDPA smoke: 26 passed, 2 skipped; whole operator suite not run
Cambricon	Yes	Final-head ordinary/Shadow CI passed; allocation harness passed	Full suite not run
Moore	Yes	Final-head ordinary/Shadow CI passed; local smoke: 167 passed, 79 skipped	SDPA smoke: 20 passed, 8 skipped; unequal-width muDNN cases unsupported
Ascend	Yes	Final-head ordinary/Shadow CI passed; parser/allocation harness passed	Full suite not run
CPU	Yes	Included in all three final-source smoke/SDPA validations	Whole operator suite not run
Hygon	Yes	Matrix conversion accepted; no hardware test	SDK/hardware unavailable; outside active CI matrix

Current-head workflow links: ordinary CI and Shadow CI. All five non-NVIDIA platforms have completed successfully in both workflows; only NVIDIA remains running. Historical results are not counted as new-head passes.

Benchmark / Performance Impact

No performance claim or benchmark. The Iluvatar Torch 2.4 math path can use more memory and run slower than fused attention; it is limited to inputs that produced incorrect fused output in the validated SDK. Other backends and equal-width FP16/BF16 retain existing dispatch. Moore GEMM tolerances remain unchanged.

Notes for Reviewers

• Keep this PR in draft until new-head hardware CI is reviewed.
• src/operator.h is identical to master. Generated dispatch includes the internal operator definition before the public generated header.
• The Iluvatar workaround does not change global SDPA flags or synchronize the whole device. Stream restoration and unrelated-stream delay assertions remain enabled.
• Automatic allocation retains resource checks and host locks. If every suitable device is busy, resource waiting remains expected.
• No runner was restarted or current job manually canceled. Proxy configuration is outside this source change. Repository cancel-in-progress policies may replace old-head jobs on push.
• Final logs under /tmp/infiniops-pr1007-precision-l6c6yY/results on iluvatar13, /tmp/infiniops-pr1007-precision-4noKsw/results on metax49, and /tmp/infiniops-pr1007-precision-Jxg3TJ/results on moore97. Earlier merge regressions: /tmp/infiniops-pr1007-nov2-hciI25 on the development host.