Repository navigation
[executorch][native] Record dynamic-shape linears as chained bundled samples - #24549
Closed
pytorchbot wants to merge 128 commits into
Closed
pytorchbot wants to merge 128 commits into
pytorchbot wants to merge 128 commits into
Conversation
…convolution Pull Request resolved: #24433 ## Ulterior Motive Closing the latency gap between ET-VK and LiteRT (TFLite GPU) on the DSR denoising model on mobile GPUs. The model's 80 1x1 convolutions were the largest per-operator gap on Mali. ## Rationale **What**: Reworks `conv2d_pw_tiled` around a 2x2-pixel x 8-channel tile with buffer weights, branch-free bounds handling and per-GPU local sizes. The 2x1-pixel x 4-channel tile stays for small Half grids and for Float on Mali, and Float on Mali uses texture weights. **Why**: Pointwise convolutions are load/store bound on Mali. The old tile issued 6 texel loads per 8 vec4 FMAs, read weights through a texture, and guarded every load with a bounds check. The 2x2 x 8 tile issues half as many loads per FMA. ## Details - Tile - **Before**: one tile, 2 consecutive output positions x 4 output channels per invocation. - **After**: a 2x2 output pixel block x 8 output channels (`_4x8` variants) for Float on every GPU except Mali, and for Half when the grid has at least 4096 invocations. Smaller Half grids keep the 2x1 x 4 tile, which fills the GPU better there, and so does Float on Mali. - Weight storage - **Before**: texture2d, unless the packed weights exceed texture limits. - **After**: buffer (`_bufw` variants) for Float on every GPU except Mali, and for Half when the device supports 16-bit storage buffers; texture2d otherwise. On a Mali-G715 in CI, Float buffer weights produced wrong results for padded and strided convolutions with either tile. The same cases passed with the parent's texture weights. - Bounds handling - **Before**: every input and weight load is guarded. - **After**: pixels and channel tiles past the edge read the nearest in-range address and are discarded at the store, so weight loads have no branches. One redundant K check stays on the input load; without any branch in the K loop, the Adreno compiler reads the weight buffer through a much slower path. - Local size - **Before**: 64-invocation square for every dispatch. - **After**: wide tile work groups span the channel tiles of a row of pixel tiles, with 512 threads on Mali, 128 on Adreno and 64 elsewhere, halved until they fit the device limits. The narrow tile uses 1x64 along pixel tiles on Mali when there are at least 64 of them, and the 64-invocation square otherwise. - Tests - The pointwise reference supports Half. `pw_correctness` runs Half too, and adds partial-tile shapes below and above the Half tile switch. - New timing-only `dsr_pw_set` with the DSR model's 1x1 convolution shapes. The tile switch and local sizes come from a sweep over 12 local sizes and both tiles on 57 shapes (see Test Plan). Float on Mali keeps the 2x1 x 4 tile: in the Mali offline compiler the 2x2 x 8 Float tile spills 88 bytes per thread while the 2x1 x 4 Float tile does not spill, and the wide Float tile was 2-31% slower in the sweep. ghstack-source-id: 462948223 @exported-using-ghexport Differential Revision: [D128786678](https://our.internmc.facebook.com/intern/diff/D128786678/)
### Summary See [PR #23299](#23299) for the root-cause writeup — `ConcatObserver` is PTQ-only (plain observer, pokes other observers' `min_val`/`max_val` directly), which breaks under QAT since the output never gets fake-quantized and the write lands on a dead attribute on `FusedMovingAvgObsFakeQuantize` inputs. This PR takes a different fix than #23299's `SharedQuantizationSpec` approach: that shares qparams only with the first cat input, which clips any other input with a wider range (verified with a `cat([x+1, y*4])` toy model — the first input's calibrated range doesn't cover the second). Instead, `concat_observer_ctr()` keeps `ConcatObserver` unchanged for PTQ, and adds `ConcatFakeQuantize` for QAT — a `FakeQuantize` paired with `ConcatObserver`, same pattern as the existing `PerBlockParamObserver`/`PerBlockParamFakeQuantize` — so the output is still fake-quantized while `ConcatObserver` keeps aggregating the true max range across every input and the output, not just the first. ### Test plan Added to `backends/qualcomm/tests/rework/utils/test.py` (no device required): - `test_qat_concat` — QAT-prepared `cat([x+1, y*4])`: output is `ConcatFakeQuantize` wrapping `ConcatObserver`, its observed range covers both inputs within one quant step, and `convert_pt2e` gives inputs and output one shared qparam. - `test_concat_observer_deepcopy` — cat at the end of a 300-node chain: `copy.deepcopy` + `convert_pt2e` doesn't raise `RecursionError`, and the copy's `input_observers` point at the copy's own submodules.
…on_backward (#23610) ### Summary Adds Qualcomm AI Engine Direct composite lowering for `aten.convolution_backward.default` when only the input gradient is requested. This PR builds on work originally implemented by Rajeswaran C Ravindran @rajeswar277 and Mohit Gupta @quic-mohitgupta of the Qualcomm On-Device Learning team. The lowering reuses the static filter wrapper shared with the forward Conv2d, emits the required runtime HWIO→HWOI QNN Transpose, and lowers the operation to QNN TransposeConv2d. This avoids requiring a separate duplicated backward weight. This is implemented as a composite visitor rather than an ATen decomposition: the visitor must reuse the forward Conv2d QNN wrapper and retain the HWOI conversion as a runtime operation. An ATen decomposition would otherwise need another transformed static weight or changes to the forward Conv2d visitor. Supported scope is HTP V69+, floating-point static 2D convolution filters with `groups=1`, `transposed=False`, `dilation=[1, 1]`, and `output_mask=[True, False, False]`. The gradient input/output and filter must be rank-4. Quantized and custom-dequant ConvBackward lowering is intentionally out of scope for this PR and will be added separately. ### Test plan - Structural lowering: - `python -m pytest -q backends/qualcomm/tests/test_passes.py -k "skip_decomp_table_preserves_conv_backward or conv_backward_composite_lowering or conv_backward_quantized_inputs_fallback or layout_transform_tuple_outputs"` - Covers decomposition preservation, composite TransposeConv2d lowering and HWOI-wrapper reuse, quantized fallback, and synthetic tuple-layout edge cases (multiple outputs, `None` outputs, and malformed users). - HTP rework: - `test_conv_backward` passed on V68, V69, V73, V75, V79, and V81. - V68 verifies expected fallback; V69 through V81 verify successful delegation. - Rework pass coverage: - `python -m pytest -q backends/qualcomm/tests/rework/passes/test.py -k "test_layout_transform or test_i64_to_i32"` - Covers FP ConvBackward tuple-output layout placement through the preprocess pipeline, plus optional tuple metadata `(grad_input, None, None)`. - On-device: - `TestQNNFloatingPointOperator.test_qnn_backend_conv_backward` passed on SM8850. - Tuple-output layout regression coverage: - Ran BatchNorm2d, MaxPool2d, GroupNorm, InstanceNorm2d, and AdaptiveMaxPool2d-with-indices on V69 and result matched clean latest `main`. --------- Co-authored-by: rajeswar277 <rajeswar@qti.qualcomm.com> Co-authored-by: quic-mohitgupta <guptam@qti.qualcomm.com>
- Add a CONST dialect operation to TOSA dialect + helper function - Add explicit CONST shift to tosa.MUL - Add explicit CONST zero shift to tosa.NEGATE - Add explicit CONST pad values to tosa.PAD This will be used for 1-1 tosa graph serialization from exir graph, for now ignore dialect CONST nodes in the existing serializer flow --------- Signed-off-by: Adrian Lundell <adrian.lundell@arm.com>
In the Ethos-U Backend, create a delegate context that is passed to ethosu_invoke_v3 as user_args. When ethosu_execution_stats is linked in, it registers ethosu callbacks that use the context to write to an EthosUStatsBuffer that is owned by the application and registered using `set_ethosu_stats_buffer`. The buffer consists of EthosUExecutionRecords that contain data for one delegate call, including metadata to identify what npu ran the delegate. Additional introduced API: - EthosUBackend_npu_begin() callback - EthosUBackend_npu_end() callback These can be used for measuring CPU time during npu dispatch. - ethosu_execution_metrics_enabled() -> bool Use this infra in arm_perf_monitor. cc @digantdesai @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell @rascani --------- Signed-off-by: Erik Lundell <erik.lundell@arm.com>
Fix for provenance metadata for DecomposeTopK. cc @digantdesai @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell @rascani Signed-off-by: Elena Zhelezina <elena.zhelezina@arm.com>
Plan exact integer adaptive windows as shared pooling patches and recognize equivalent padded pools within target restrictions. Signed-off-by: Adrian Lundell <adrian.lundell@arm.com>
### Summary conda-forge's pip 26.2.2 build 1 (conda-forge/pip-feedstock#151, published 2026-11-02) removes pip's bundled `cacert.pem` and makes `certifi.where()` return `sys.prefix/ssl/cacert.pem`. `backends/native/python/setup.py` builds with `python -m build`, which installs build dependencies by running the outer pip with `--python` pointed at its isolated venv. That venv's `sys.prefix` has no `ssl/cacert.pem`, so pip fails with `FileNotFoundError` in `load_verify_locations` ([example](https://github.com/pytorch/executorch/actions/runs/41501634985/job/125470594908)). Every "Build Aarch64 Linux Native Wheels" PR run has failed this way since about 17:17 UTC on 2026-11-02, because the aarch64 image resolves pip from conda-forge. The x86 image gets pip from Anaconda's defaults channel and is unaffected. Reinstalling pip from PyPI, which bundles its own certificates, before installing `build` avoids this. The line can be removed once conda-forge's pip works inside venvs again. Authored with Claude Code. ### Test plan Reproduced locally (osx-arm64) in a conda env created with `-c conda-forge pip=26.2.2=*_1`: `python -m build --wheel` on a project with the native wheel's build requirements fails with the same traceback. After `python -m pip install --force-reinstall pip`, the same command installs `certifi==2026.10.23`, `pybind11==3.1.0`, `pyyaml==6.0.3`, `scikit-build-core==1.1.1` and `zstd==1.5.7.2` into the isolated env and succeeds. This PR touches `.ci/`, so it runs "Build Aarch64 Linux Native Wheels" itself.
…bounds (#24483) Summary: `test_vulkan_backend_max_pool2d` and `test_vulkan_backend_avg_pool2d` export with a `(5, 13, 55, 68)` sample input and mark only the batch dimension as dynamic, so dimension 1 has an upper bound of 13. Their first test input, `(3, 14, 15, 9)`, exceeds that bound. Since #24203, `set_input` enforces per-dimension upper bounds for `DYNAMIC_BOUND` inputs, so both tests now fail with `Input 0 dimension 1 is 14 but its upper bound is 13`. Use 13 for dimension 1 of that input so every test input stays within the exported bounds. The other two inputs already do. Authored with AI assistance. Reviewed By: shoumikhin Differential Revision: D129264803 Pull Request resolved: #24483
Extended automatic list of supported ops to include FP8 formats. cc @digantdesai @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell @rascani Signed-off-by: Elena Zhelezina <elena.zhelezina@arm.com>
### Summary Every push to `main` currently puts the ESP32 workflow in the same concurrency group, so a new commit cancels the previous commit's jobs. In [this job](https://github.com/pytorch/executorch/actions/runs/41514087917/job/125514346213), the smoke test and all 16 optimized model cases passed, but the job was marked cancelled when another push arrived. Use the concurrency group already shared by the other backend test workflows. Including the commit SHA for branch runs lets different `main` commits finish independently. The group also separates scheduled and manually triggered runs. Authored with AI assistance using OpenAI Codex. ### Test plan `git diff --check` passed. Compared the complete `concurrency` block with all ten `test-backend-*.yml` workflows and verified that it matches each exactly. No emulator tests were rerun for this workflow scheduling change.
### Summary Run the shared Espressif operator-library implementation cases in the existing optimized ESP32-S3 CI job, using its SDK setup and runtime build. Include the portable sine kernel used by the multiply tests' float branches and verify that the operator firmware selects optimized ESP-NN kernels. Stop at the first operator failure so subsequent cases cannot overwrite its firmware and logs. Collect the bundle, SDK configuration, ELF, merged image, and test logs with the existing CI artifacts. This builds on the multiply implementation and ESP-IDF test harness in #24311. ### Test plan The complete optimized S3 script passed with a non-editable installed Python package: three smoke inferences, sixteen delegate fixtures, and all eleven shared operator implementation cases. SDK configuration confirmed optimized kernels, and the workflow collected its firmware and test artifacts. The preceding kernel PR supplies the explicit checkout path fix. This PR remains the one-file CI integration change. The full rebased stack also passed 153 frontend tests. ```bash ESPRESSIF_NN_KERNELS=optimized bash .ci/scripts/test_espressif.sh ``` Authored with AI assistance using Codex.
…out (#24081) ### Summary A delegate input whose dim order is neither contiguous nor channels-last (e.g. `(1, 0, 2)` from a transposed example input) is serialized with its logical sizes, but `XNNExecutor` reshapes it with sizes in dim order, so execution fails with `xnn_status_invalid_parameter` or reads swapped axes. Define such inputs in memory order and transpose back at the start of the graph. #14735 was the constant-tensor side of this. ### Test plan New `ops/test_io_dim_order.py` tests fail before, pass after. `backends/xnnpack/test/{ops,passes,serialization}` unchanged. cc @GregoryComer @digantdesai @cbilgin @JakeStevens @nil-is-all
### Summary Fixes #24169. The portable `div.out_mode` and `div.Scalar_mode_out` kernels compute the integer quotient with `/` and then call `std::trunc` on it, which converts it to `double` and back. Integer division already truncates, so `std::trunc` now only runs for floating-point types. For int64 values above 2^53, `div(a, 3, rounding_mode="trunc")` gave `1537228672809129216` instead of `1537228672809129302`. `fmod.Tensor_out` and `fmod.Scalar_out` switched every integer compute type to `Double`. Integer types now use `%` through a new `utils::integer_fmod` in `math_util.h`, which returns 0 for a divisor of -1 so the most negative value can't overflow. Float types keep the existing `std::fmod` path, including the vectorized one for the Scalar variant. ### Test plan Added `OpDivOutTest.TruncModeKeepsInt64Precision` (Tensor and Scalar divisor) and `OpFmodTest.Int64KeepsFullPrecision` (Tensor, Scalar, and a divisor of -1 with `INT64_MIN`). The expected values come from eager PyTorch. I built `portable_kernels_test` with CMake (`EXECUTORCH_BUILD_TESTS=ON`, Release) on Linux x86_64 (WSL2) at 5b14a2f. Before this change both new tests fail; after it, the full `portable_kernels_test` run passes (1696 tests). `clang-format` 18.1.3 reports no changes on the changed files. This PR was authored with AI assistance (Claude Code). cc @nil-is-all @larryliu0820 @manuelcandales @JakeStevens
Main-branch counterpart of release/1.6 fix #24509. Fixes the NXP MCUXpresso failure seen on #24490; main has the same incomplete CMake target. The [failed job](https://github.com/pytorch/executorch/actions/runs/41601038988/job/125799646998) reaches the final CIFAR firmware link, then fails with `undefined reference to SYS_INFO_PrintAll`. The SDK example calls this helper, but our standalone CMake target omits `examples/eiq_examples/common/sys_info/sys_info.c`. Add that source to the firmware target. The helper also needs `MCUXSDK_VERSION_FULL_STR`, which the SDK's normal build supplies through a generated header. Our standalone target now uses the SDK's `mcux_version.cmake` parser and sets the equivalent compile definition from the checked-out SDK version. ## Verification A focused host CMake build reused the actual example target's source list and include paths, the real SDK helper/header, and the SDK version parser. The board/runtime sources were replaced with empty fixtures and the application with a small C++ caller. The original source list reproduced the unresolved `SYS_INFO_PrintAll` link error. Adding only the helper reproduced the missing `MCUXSDK_VERSION_FULL_STR` compile error. With both changes, the executable linked and printed `MCUX SDK version: 2026.12.00-pvw1`. `git diff --check` passed. The full ARM firmware build and execution on hardware were not run locally; the NXP CI job must validate the complete build. Only the example's CMakeLists.txt changes (eight added lines). The resulting file is byte-identical to the verified release fix in #24509. Land this on main; #24509 is the corresponding release/1.6 pick. Prepared with OpenAI Codex.
… delegates share (#24404) ### What is wrong **Zero strides.** ExecuTorch cannot store a zero stride, and `contiguous()` keeps one on a size-1 dimension: ```python t = torch.tensor(2.0).expand(1) t.stride() # (0,) t.is_contiguous() # True t.contiguous().stride() # still (0,) ``` So export fails with `0 in strides is not supported for ExecuTorch.` when `constant_prop_pass` folds `self.scale.expand(1)`, a parameter, buffer or constant method holds such a tensor, or an example input is expanded. **Strides of size-1 dimensions.** PyTorch also ignores the stride of a size-1 dimension when it checks contiguity, but ExecuTorch derives the dim order (the order of dimensions in memory) from every stride: ```python q = torch.randn(1, 8, 4).transpose(0, 1) q.stride() # (4, 32, 1) q.is_contiguous() # True, yet the program stores dim order (1, 0, 2) ``` An op that reads such a constant fails at run time: portable `bmm` with `3 input tensors have different dim orders`, XNNPACK with `Propagating input shapes failed`. This happens for a parameter or buffer like `q`, and when `constant_prop_pass` folds `self.queries.transpose(0, 1) * 0.5`. **Constants that ops in different delegates read.** `duplicate_constant_node` gives each such op its own copy. - An op that read it from a list, such as `cat`, kept the original, and lowering failed. - `tag_mutated_buffer` gave a mutated buffer to a delegate even when some readers or writers stayed outside. A reader outside got a stale copy, or lowering failed (new value computed outside, read outside, or also returned). ### What this change does - `without_zero_strides()` in `exir/tensor.py` copies a tensor with a zero stride, keeping channels-last. `constant_prop_pass`, `convert_constant_dim_order_pass` (run by `to_edge`) and `TensorSpec.from_tensor(const=True)` use it. That pass also removes one from method inputs, so an input with a zero stride on a dimension larger than 1 must be passed contiguous. - `convert_constant_dim_order_pass` also checks the dim order that a constant's strides give. If it is neither the default nor channels-last, the pass copies the constant into the layout PyTorch reports. Constants whose strides already give one of those orders do not change, so a channels-last weight with a size-1 channel stays channels-last. - `duplicate_constant_node` copies the constant for every op that reads it, including from a list, and the graph output keeps the original. It raises instead of copying a mutated buffer. - `tag_mutated_buffer` gives a mutated buffer to a delegate only when these hold: - every op that reads it (old or new value) or writes it is in that delegate; - its new value is not returned; - some op reads it, unless the partitioner already tagged it for its writer. Otherwise it stays in the owning program and loses any tag, with a warning. **Behavior change.** A mutated buffer used by ops in different partitions, which used to give stale reads, now stays in the owning program and is passed to each delegate: correct, but copied on every call. A buffer whose new value is a graph input, as in `self.state.copy_(x)`, is no longer tagged, so an existing test in `test_partitioner.py` now expects no tag. With the reader delegated, that model failed to lower on main, and now it lowers and runs. When `constant_prop_pass` runs before `to_edge` and folds a zero-stride constant that PyTorch calls channels-last but not contiguous, the constant now stays channels-last instead of becoming contiguous. Like any other channels-last constant, it then fails at run time (`different dim orders`) in a portable op that also reads a contiguous tensor. ### Testing - New tests under `exir/emit`, `exir/tests` and `exir/backend/test` each fail without their fix. - Existing tests in those folders pass as before. - MobileNetV2 and a small Conv2d model, default and channels-last, give the same `.pte` bytes or the same export error as before, with portable, XNNPACK, and XNNPACK after `constant_prop_pass`. Tested on Linux with torch 2.14.0 (CPU).
## What is wrong today coremltools converts some ops differently from PyTorch, so lowering fails or outputs are wrong: - `sqrt`, `exp`, `sin` and similar ops fail to lower on an int or bool tensor, which PyTorch computes in float. So do `floor`, `ceil`, `round`, `relu`, `relu6` and `hardtanh` on an int tensor, though PyTorch keeps their result int. - Copying an int or bool tensor into a float one, as `y[:, i] = v` does, keeps its dtype, so a later float-only op fails to lower. - `clamp_min` and `clamp_max` fail to lower when the tensor and the bound differ in dtype, as in `x.clamp_min(0)` on a float `x`. - Fill ops such as `full` and `zeros_like` get the wrong dtype: `torch.full((1, 16), 0)` is int64 in PyTorch and float in Core ML, so later ops fail or lose precision. `nn.MultiheadAttention` hits this through its bool mask. - `gelu(approximate="tanh")` runs as the exact GELU: coremltools ignores the keyword `torch.export` passes. - `nn.RMSNorm(eps=None)` fails to lower: coremltools requires `eps`. - A Core ML program that returns an input, one value twice, or a same-shape `expand` of another output can silently return one of them wrong. - `y[:, i] = v` with `i = -1` writes to an empty slice: lowering fails for a constant `i`, and for an input `i` Core ML fails at run time or, from iOS 18, silently skips the write. ## What this change does - `compiler/torch_ops.py` overrides the converters of these ops; each override can go once coremltools fixes its bug. Inputs get the dtype PyTorch computes in: float for the ops that need it, int kept for the rounding and clipping ops, the dtype of `y` for a value copied into it, and the result's dtype for `clamp_min` and `clamp_max`. Fills get PyTorch's dtype, `gelu` reads its keyword, `rms_norm` gets PyTorch's default eps, and a negative index next to a slice gets the dim size added. The input fixes come first, since the fill fix sends int fills to them. - `compiler/coreml_preprocess.py` turns each same-shape `tile` output into a copy of its input, and copies an output that repeats an input or another output. ## Testing - The Core ML tests (`test_torch_ops.py`, `test_coreml_partitioner.py`, `test_coreml_recipes.py`) pass on Linux and on macOS, where the new tests also compare the Core ML output with eager PyTorch. - Each new test fails when its fix is removed.
Summary: The Vulkan engine, its graph passes and the vision runner were only defined in fbcode, so the runner could not be built for Android. This defines them in xplat too. `backends/native/runtime/vulkan/targets.bzl` no longer returns early in xplat, `backends/native/runtime/vulkan/BUCK` defines the xplat `xplat_vulkan_runtime` next to the fbcode one, and `run_classification` and `run_classification_vulkan` are declared for both cells. The fbcode targets are unchanged. `vulkan_passes_test` needs neither a GPU nor SwiftShader, so it now also builds and runs in xplat. The xplat `run_classification` carries a CI label that builds it for Android arm64, which covers the Vulkan engine, its passes and ET-VK on Android. The new xplat targets use the default xplat platform set, which matches ET-VK's `get_platforms()`. Review fixes authored with Claude Code. Differential Revision: D129370846
Summary: These 23 passes in `backends/cadence/aot` are not Cadence-specific in what they do, so they move to `backends/transforms` where any backend can use them: - Three (`FuseCascadedTransposeOrPermuteOps`, `FuseCascadedViewOps`, `ReplaceNopTransposeOrPermuteWithViewPass`) were already empty subclasses of a `backends/transforms` pass, and stay as they are here. - The other 20 (including `RemoveNopAsStridedCopyOpPass`) move to `backends/transforms`, one module and target each. The ones that listed Cadence's q/dq ops alongside `quantized_decomposed`'s now take those op sets as class attributes holding only the standard ops. Cadence keeps each pass under its old name as a thin subclass that adds its own ops back, so Cadence's behavior is unchanged. - `FuseQuantDequantToRequantizePass` needs a requantize op to fuse a pair whose qparams differ, and there is no backend-neutral one. The shared pass takes a `requantize_op` (default `None`) and defaults to `allow_requantize=False`; Cadence sets its op and keeps `True`. - `permute_pass_utils` gains `get_overload_packet` and `get_placeholders`, and `quantize_reorder_utils` holds the two op sets the reorder passes share. Reviewed By: JakeStevens Differential Revision: D129173244 Pull Request resolved: #24469
## What is wrong today 2 threads can hang forever when a backend logs while holding a lock, as XNNPACK does when a load fails. A log line from a thread that holds the GIL (Python's global interpreter lock) goes to `sys.stderr` at once, and that Python write can let another thread take the GIL. If that thread then waits for the lock, neither thread moves again: ``` thread A: load_method -> takes lock L -> logs -> sys.stderr.write releases the GIL thread B: takes the GIL -> execute -> waits for lock L, holding the GIL thread A: waits for the GIL forever ``` ## What this change does The log function never runs Python code. While `sys.stderr` is the process's stderr (fd 2), each line goes straight to fd 2. When Python has replaced `sys.stderr` (`capsys`, `redirect_stderr`, a Jupyter notebook), the lines wait: - A binding call's lines are written as it returns, with no backend lock held. - A line that Python's main thread logs outside a call while holding the GIL (for example while the LLM runner loads) is written before the next Python code runs, so it still lands in a short `redirect_stderr` block. - Other threads' lines go to a thread that the bindings start. It never writes from inside another thread's `sys.stderr.write`, so a stream whose `write` takes a lock cannot hang. Outside a binding call, threads that do not hold the GIL write to fd 2, as today. Up to 1 MiB waits, keeping the newest. On Linux and macOS, a line also waits while fd 2 is a pipe or socket that stays full, so a GIL holder does not block while a Python thread drains it (`wurlitzer.pipes()`). A terminal is written at once, since its reader is normally another process. A fatal line goes to fd 2 at once while there is room. On Linux with Python before 3.13.8, a daemon thread never writes through Python. Today it does, and Python can end it mid-write at exit, which crashes the process. ## Tests New tests cover the hang (also from the LLM runner), a stream whose `write` takes a lock, a full pipe or socket drained by a thread, lines inside a short redirect, daemon threads at exit, and cut error messages. Ran them on Linux x86_64 and macOS, and the logging cases in Jupyter with fd capture on and off. Also fixed: an error message with invalid UTF-8, such as a corrupt program's identifier, raised `UnicodeDecodeError`, not `RuntimeError`. The docs on registering a PAL (platform abstraction layer) now use the right namespace and a complete example.
Summary: Lower matrix row gathers from `aten.index.Tensor` through the existing embedding path and lower positive bounded `aten.repeat` shapes with `ggml_repeat`. Preserve snapshots when repeat aliases its input so later cache writes cannot change previously computed values. Allow row-index validation to follow repeats and validate integer `aten.arange.start_step` bounds when building the graph. This supports Whisper's encoder position embeddings and decoder cache-position gathers while retaining input bounds checks on cached graph replay. Review row-index validation and gather lowering first, then repeat lowering and the operator regressions. Authored with assistance from Codex. Differential Revision: D125479769
…24403) ### What is wrong today Rotary position embeddings are often written with complex numbers: `torch.polar`, `torch.view_as_complex`, a product, then `torch.view_as_real`. Such a model exports, but the CUDA backend cannot lower it: ``` RuntimeError: Method forward missing fallback kernels (5 total): - aoti_torch_cuda_mul_Tensor - aoti_torch_cuda_polar - aten::view_as_complex_copy - aten::view_as_real_copy - prims::convert_element_type ``` Inductor generates no code for complex numbers, so it calls ATen kernels, which this backend does not ship. Torch-TensorRT leaves these ops to the CUDA backend, so its programs stop at the same error. ### What this change does A new pass, `ReplaceComplexWithRealPass`, stores each complex tensor as a real tensor with a last dim of size 2 (the layout of `view_as_real`) and rewrites the ops on it as real math: - `polar` becomes cosine and sine times the magnitude; a complex product becomes four real products. - `view_as_complex` and `view_as_real` go away. - Views, slices, splits, indexing, repeats, joins and casts keep their op, with the dims and dtype of the pair. This includes `_reshape_copy`, which Torch-TensorRT makes of every view, such as Llama's `freqs_cis.view(...)`. - Complex constants, buffers and parameters are stored in the real layout. If the pass cannot rewrite every op on a complex tensor, for example a complex add, a complex input or output of the partition, a `torch.cond` branch, or complex32, it leaves the program as it is and logs why. So the complex math has to stay inside one CUDA partition. Complex128 becomes float64, which the CUDA runtime supports. Also in the CUDA backend: - With CUDA graphs on, a copy or clone of a CUDA tensor during the capture aborted the process. AOTI returns a part of a constant through such a clone. These shims now refuse inside a capture, as `.item()` does, and the call runs again without a graph. - On ROCm, `test_varlen_attn` compared no case. It now skips only windows over query and key sequences that differ, which AOTriton lines up differently. ### What was tested With torch 2.14 on an A100: - `test_complex_rotary_embedding` (GPU) and `test_replace_complex_with_real.py` (CPU) compare rewritten programs with eager, in float32, bfloat16 and complex128, with a dynamic length, `_reshape_copy` and a buffer with two placeholders. Unsupported programs stay as they are. - `test_constant_copy_runs_without_a_graph` and `test_aoti_torch_copy_` cover the copy and clone refusal. - A Llama-style rotary embedding lowers through Torch-TensorRT 2.14 and the CUDA backend. The new tests fail without their fixes, and the other CUDA tests pass. The ROCm change was not run on ROCm.
…f16 matmul load crash, and skipped bmm tests (#24504) XNNPACK lowering can give silently wrong results when one Linear or Conv is used twice and each result goes into its own BatchNorm: def forward(self, x, y): return self.bn1(self.linear(x)), self.bn2(self.linear(y)) When the BatchNorms are folded into the Linear, both folds ask for a constant with the same name, built only from the Linear weight name. The helper that creates constants returns the existing one when a name is taken, so the second output uses the first BatchNorm's statistics. The same thing happens when XNNPACK turns standalone BatchNorms with affine=False into depthwise convolutions: the names came from the BatchNorm weight, which they do not have, so all of them shared one name. The commits: 1. Fold names now include the BatchNorm node, so each fold gets its own weight and bias. Models with one BatchNorm per Linear or Conv give the same .pte as before. Vulkan uses the same pass and gets the fix too. 2. Depthwise convolution names for a standalone BatchNorm now come from the BatchNorm node, which is unique. 3. Loading a bf16 batch matrix multiply or linear whose output tensor has no dims vector no longer crashes. The dims are read the same way defineTensor() reads them. 4. backends/xnnpack/test/ops/test_bmm.py was skipped in pytest.ini and pytest-windows.ini. The file passes now, so the skip is removed and test_bf16_bmm runs in CI. Tested on Linux x86_64: - New tests for 1 and 2 fail before the fix and pass after it, including a lowered and serialized run. - A new C++ test for 3 crashes before the fix and passes after it. - test_bmm.py passes; test_bf16_bmm fails on a build from before the bf16 bmm fix. - backends/xnnpack/test/passes and backends/xnnpack/test/ops show no new failures. - resnet18, mobilenet_v2, mobilenet_v3_small and efficientnet_b0 lower to byte-identical .pte files before and after. macOS and Windows were not run locally; the unit test jobs here cover them.
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24549
Note: Links to docs will display an error until the docs builds have been completed. ❌ 14 New Failures, 17 Cancelled Jobs, 2 Unrelated Failures, 12 Unclassified FailuresAs of commit 3016b89 with merge base 0f8e096 ( NEW FAILURES - The following jobs have failed:
UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:
CANCELLED JOBS - The following jobs were cancelled. Please retry:
FLAKY - The following job failed but was likely due to flakiness present on trunk:
BROKEN TRUNK - The following job failed but was present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
Summary: Drop the `//third-party/cuda:windows-cuda-11` arm from the Vulkan platform flags. No CUDA 11 toolkit or mode remains, so it never matches; the `ovr_config//os:windows` arm already sets the same define. Part of removing CUDA versions below 13 (T294426402). ___ landed-with-radar-review Reviewed By: dtolnay Differential Revision: D129309847 Pull Request resolved: #24513
…-block scales in the op samples Pull Request resolved: #24368 GGUF samples used fixed positive power-of-two super-block scales, so engines' decoders never saw negative, rounded or subnormal ones. A model's i-th `nn.Linear` now takes the i-th scale family, and a new `scale_families` case gives each family its own output. A zero-d family waits behind a TODO. With q4_k's d at 0 every weight is -dmin·m, so a token's 32 outputs share one activation-quantization error (correlation about 0.74) and `OutputCheck`'s bias standard error, which assumes independent outputs, comes out about 4.9 times too small; with q6_k's d at 0 every weight is zero and the outputs are just the bias. Authored with Claude Code. @bypass-github-export-checks @bypass-github-pytorch-ci-checks @bypass-github-executorch-ci-checks ghstack-source-id: 463657756 @exported-using-ghexport Differential Revision: [D128254896](https://our.internmc.facebook.com/intern/diff/D128254896/)
Pull Request resolved: #24369 `linear_dq8ca_q4gsw` failed on rank-1 activations: `is_2d_matrix` rejected them, and `get_quantized_input_num_blocks` read the size at `ndim - 2`. Both now treat a 1-D tensor as a single row. Authored with Claude Code. @bypass-github-export-checks @bypass-github-pytorch-ci-checks @bypass-github-executorch-ci-checks ghstack-source-id: 463657751 @exported-using-ghexport Differential Revision: [D128254882](https://our.internmc.facebook.com/intern/diff/D128254882/)
… 1 to 4 Pull Request resolved: #24370 `linear_dq8ca_q4gsw_tiled` strode each group's weight sums by `div_up(N, 4)` texels, but the sums are laid out `{groups, align_up(N, 8)}`, as both the ET-VK AOT pass (`compute_per_group_sums`) and Native's `FuseQuantizedLinear` write them. When `div_up(N, 4)` is odd, every group after the first read another row's sums; the shader now strides by `2 * div_up(N, 8)`, which equals the old stride for every other N. Authored with Claude Code. @bypass-github-export-checks @bypass-github-pytorch-ci-checks @bypass-github-executorch-ci-checks ghstack-source-id: 463657747 @exported-using-ghexport Differential Revision: [D128254883](https://our.internmc.facebook.com/intern/diff/D128254883/)
…near gaps Pull Request resolved: #24371 The Vulkan engine leaves int4 weight-only linears unlowered when N isn't a multiple of 4, so op-sample cases with N of 1 and 43 fail. It also computes a linear on a rank-4 input wrong, while rank-3 inputs pass. `TODO.md` records both, with the fix for the first and where to look for the second. Authored with Claude Code. @bypass-github-export-checks @bypass-github-pytorch-ci-checks @bypass-github-executorch-ci-checks ghstack-source-id: 463657741 @exported-using-ghexport Differential Revision: [D128254884](https://our.internmc.facebook.com/intern/diff/D128254884/)
Pull Request resolved: #24372 Vulkan lets the device choose how stores to rgba16f images round, and devices that round toward zero bias f16 outputs; shader float controls don't govern that conversion. `gen_vulkan_spv.py` now wraps `imageStore` on writable rgba16f images in a round-to-nearest-even helper, and six shaders that declared such images by hand, two of them test shaders, now use the codegen. On devices that round toward zero, stores of 65520 and above now give inf, as the CPU conversion does, instead of 65504. The four texture shaders that already call `round_to_half_rte` on their outputs (`binary_scalar_texture`, `full_texture`, `reduce`, `scalar_tensor`) now round twice, with identical results; removing those calls is a follow-up. `aten._to_copy` from f32 to f16 textures is a `vkCmdBlitImage`, which the codegen doesn't reach, so `test_to_copy` keeps its one-ulp tolerance. Authored with Claude Code. @bypass-github-export-checks @bypass-github-pytorch-ci-checks @bypass-github-executorch-ci-checks ghstack-source-id: 463657749 @exported-using-ghexport Differential Revision: [D128254886](https://our.internmc.facebook.com/intern/diff/D128254886/)
Pull Request resolved: #24373 `ggml_sample_test_act_quant` (CPU) is the one sample test that passes every package, so it leaves the manual set and runs in internal CI. The OSS ggml job now runs it as `native_ggml_sample_test_act_quant`, along with `native_output_check_test`. The rest stay manual with a TODO: plain `ggml_sample_test` fails the weight-only packages by design, and the Vulkan, CUDA and HIP ones have gaps. The OSS ggml job installs `gguf` for the op sample generator, and the workflow also runs on changes to FACTO and `test_linear.py`, which the samples draw from. Authored with Claude Code. @bypass-github-export-checks @bypass-github-pytorch-ci-checks @bypass-github-executorch-ci-checks ghstack-source-id: 463657737 @exported-using-ghexport Differential Revision: [D128668496](https://our.internmc.facebook.com/intern/diff/D128668496/)
…samples Pull Request resolved: #24405 Engines resize dynamic inputs between inferences, so the op samples now cover a shape changing mid-run. A case built with `chain` is exported with its `Dim`s and recorded as one sample of four calls. Each `Dim` starts mid-range, where export traces it, visits both bounds and returns, every other `Dim` in the opposite order. The harness already runs a sample's calls in order on one executable from one context, resizing inputs between them, so only its header comment changes. The new linear cases are the XNNPACK delegate's dynamic linear tests, each also followed by a ReLU: one and two batch dims, one `Dim` on two dims or two inputs, and sequential and parallel linears, with batch dims down to 1, which tracing mid-range keeps export from specializing. The cases every quantization scheme fits take 256 input features over up to 64 tokens: `dynamic_tokens` in 2-D and, with a static batch of one as in an LLM, in 3-D, two batch dims sharing one `Dim` of up to 16, and the sequential and parallel linears. Authored with Claude Code. Differential Revision: [D128770754](https://our.internmc.facebook.com/intern/diff/D128770754/) **NOTE FOR REVIEWERS**: This PR has internal Meta-specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D128770754/)! ghstack-source-id: 463657750
digantdesai
force-pushed
the
gh/digantdesai/201/orig
branch
from
November 5, 2026 22:52
3016b89 to
f67eb1a
Compare
digantdesai
requested review from
JacobSzwejbka,
mcremon-meta,
mergennachin,
metascroy,
psiddh,
rascani,
robert-kalmar and
shoumikhin
as code owners
November 5, 2026 22:52
One or more co-authors of this pull request were not found. You must specify co-authors in commit message trailer via: Supported
Alternatively, if the co-author should not be included, remove the Please update your commit message(s) by doing |
github-actions
Bot
added
ciflow/trunk
module: arm
Contributor
|
Superseded by #24698. |
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR was created by the merge bot to help merge the original PR into the main branch.
ghstack PR number: #24405 by @digantdesai
^ Please use this as the source of truth for the PR details, comments, and reviews
ghstack PR base: https://github.com/pytorch/executorch/tree/gh/digantdesai/201/base
ghstack PR head: https://github.com/pytorch/executorch/tree/gh/digantdesai/201/head
Merge bot PR base: https://github.com/pytorch/executorch/tree/gh/digantdesai/200/orig
Merge bot PR head: https://github.com/pytorch/executorch/tree/gh/digantdesai/201/orig
@diff-train-skip-merge
cc @digantdesai @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell @rascani