Repository navigation
Fix 0-d tensors, dynamic output sizes and integer reductions, and default to float32 on CPU, in the OpenVINO backend - #24294
Open
shoumikhin wants to merge 15 commits into
Open
shoumikhin wants to merge 15 commits into
shoumikhin wants to merge 15 commits into
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24294
Note: Links to docs will display an error until the docs builds have been completed. ❌ 1 New FailureAs of commit bd59ee0 with merge base 7586c16 ( NEW FAILURE - The following job has failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
shoumikhin
added
release notes: openvino
shoumikhin
force-pushed
the
openvino-precision-and-scalars
branch
from
October 29, 2026 18:00
ca054b6 to
b9d75ec
Compare
shoumikhin
force-pushed
the
openvino-precision-and-scalars
branch
from
October 29, 2026 21:40
b9d75ec to
3fb1360
Compare
shoumikhin
changed the title
shoumikhin
force-pushed
the
openvino-precision-and-scalars
branch
from
October 29, 2026 23:23
3fb1360 to
bc88265
Compare
shoumikhin
changed the title
shoumikhin
force-pushed
the
openvino-precision-and-scalars
branch
from
October 29, 2026 23:50
bc88265 to
03e2941
Compare
shoumikhin
force-pushed
the
openvino-precision-and-scalars
branch
from
October 30, 2026 01:27
03e2941 to
8969a7b
Compare
shoumikhin
force-pushed
the
openvino-precision-and-scalars
branch
from
October 30, 2026 01:42
8969a7b to
89f775c
Compare
Contributor
Author
|
@claude /pr-review |
|
Claude finished @shoumikhin's task in 38m 1s —— View job Summary: The new float32-on-CPU default slows down programs that were exported before it on x86 CPUs with bf16 units, and neither a load-time option nor the repo's own export tools can switch it off; the other findings are minor issues in the cumsum-folding pass, the output-resize loop, the tests and the docs. Detailed reviewPR Review: #24294SummaryThis PR fixes 0-d tensors, dynamic output sizes and integer reductions in the OpenVINO backend, and makes float32 the default inference precision on CPU. That default needs discussion before merge, for three reasons:
All other findings are minor. Code Quality
Infrastructure
Testing
API Design
Backward Compatibility
Performance
RecommendationNeeds Discussion. Approval waits on decisions about the precision default:
All other findings are minor.
Not done: nothing was built or run. The claims about OpenVINO's behavior come from reading the code, and the ones I couldn't confirm are marked low confidence. |
Contributor
Author
|
@suryasidd @daniil-lyakhov @anzr299 could you take a look? This changes the OpenVINO backend. |
The OpenVINO backend built every input and output tensor's shape with ov_shape_create, which refuses rank 0. So a delegate with a 0-d input or output, such as (x * 2).sum(), failed every run with an Internal error and nothing in the log. Keep the empty shape for a 0-d tensor; ov_tensor_create_from_host_ptr takes it as a scalar. Also log both error returns of create_ov_tensor. The 0-d cases of the existing OpenVINO mean tests failed this way and now pass: the op tests go from 25 failures to 21, with no new failure.
…ption
OpenVINO picks the inference precision when it imports a compiled
model, and on CPUs with bfloat16 units it picks bfloat16. The backend
set no precision, so a float32 program ran in bfloat16 there. A small
float32 MLP came out with a relative error of 4e-3 against eager
PyTorch instead of 1e-6, and no compile spec could change it.
On a CPU the backend now asks for float32 by default. A program can opt
into the faster, less precise mode when it is lowered:
OpenvinoPartitioner(
[CompileSpec("device", b"CPU"), CompileSpec("inference_precision", b"bf16")]
)
The value is OpenVINO's INFERENCE_PRECISION_HINT, and it works for any
device. Other devices, such as GPUs, keep OpenVINO's default unless the
option is set, since their compiled kernels pick their own precision. A
value OpenVINO rejects fails the load.
The trade-off: on CPUs with bfloat16 units the default is slower than
before, for float16 programs too, which now compute in float32. On a
Xeon with bfloat16 units, 8 cores, a 4 x Linear(1024) MLP at batch 64
takes 2.4 ms per call in float32 and 1.5 ms in bfloat16. A program that
wants the old speed sets inference_precision to bf16.
The OpenVINO README describes the option. test_addmm.py gains a 256 x
256 case checked to 1e-4, which fails on that CPU without this change,
and a case where an unknown precision fails the load. On that CPU, 21
existing OpenVINO op tests failed their 1e-3 tolerance before this
change; all of them pass now.
On ARM CPUs OpenVINO fixes the inference precision when it compiles the model, which happens during lowering. preprocess called compile_model without a config, so the model was compiled in float16 there, while the runtime later asked for float32 on import. The compiled model reported float32 and still computed in float16: test_addmm_computes_in_float32 missed by 2e-3 on a Linux aarch64 board. preprocess now passes INFERENCE_PRECISION_HINT to compile_model, with the same rule as the runtime: the inference_precision compile spec, or float32 on a CPU. On x86, where OpenVINO picks the precision on import, nothing changes. On ARM CPUs a program now computes in float32 by default too, about 1.8x slower than float16 on an Orin Nano; it can opt back in with inference_precision set to f16.
OpenVINO does not check INFERENCE_PRECISION_HINT the same way on every release this backend supports. 2025.1.0 accepts a value such as f7 and runs the model. 2026.4.1 accepts it in set_property and only fails the import, so the load reports "failed to import model" without naming the precision. Check the value against f32, f16 and bf16 ourselves: at lowering, so a typo fails at export with a ValueError that names it, and at load, so a program exported with a bad value fails with InvalidArgument and a log line that names it. The test now asserts the lowering error, which no OpenVINO release produces without this check. The load-time check has no test of its own: lowering now rejects such a value before it can reach the runtime.
Only CPUs compute in float32 by default. OpenVINO's GPU default is float16, and at 256 x 256 a float16 result is far outside the test's 1e-4 tolerance, so a documented --device GPU run of the op suite would fail it. The test now skips on other devices.
… the docs The README's Precision section described only x86. On ARM CPUs OpenVINO would compute in float16, and the float32 default now applies there too, so the section says how to opt back into float16. It also says that bfloat16 keeps two to three significant digits, not about three, what a CPU without float16 or bfloat16 support does with each value, that other values fail at lowering, and the snippet imports what it uses. docs/source/build-run-openvino.md mirrors this README and gets the same section.
An output whose size depends on an int input, or on a dynamic input
size, is planned at its upper bound. The backend handed OpenVINO that
buffer and never read back the shape OpenVINO actually produced, so the
output kept its upper-bound shape. For example, with x of shape (4, 8)
and n = 3:
x[:n].view(n, -1) # eager: (3, 8); OpenVINO delegate: (4, 8), one extra row
After inference, the backend now reads each output's shape from
OpenVINO and resizes the ExecuTorch output to it. The data is already
laid out for that shape at the start of the buffer. A 0-d output is
skipped: it has no size to change, and newer OpenVINO releases cannot
return a rank-0 shape from their C API.
Once an output is smaller, the next call would hand OpenVINO the smaller
buffer, and OpenVINO treats a buffer's shape as its capacity, so a larger
result would fail. The backend records each output's size at the first
call, which is its upper bound, and restores it before every later call.
The sizes live in a small list from the runtime allocator, 16 int64
values per output.
test_view_data_dependent_size returned x[:n].view(n, -1).sum(0), which
hid the extra row. It now returns the view itself and runs n = 3 and
then n = 4 on the same loaded method, through a new more_inputs option
of the shared test helper. It fails without the resize, and also
without the restore.
PyTorch computes sum, cumsum and prod of a bool or integer tensor in
int64. OpenVINO's PyTorch frontend keeps a narrower type for some of
them. A cumsum over a bool mask is computed in 8 bits, so the count
wraps after 255. A cumsum or prod over int32 returns int32. When such a
result is a delegate output, the runtime rejects it at execute:
m = torch.rand(2, 7) > 0.5
torch.cumsum(m, 1) # CALL_DELEGATE execute failed: 0x1
Inside a graph the wrong values flow on silently. pi0 and pi0.5
compute their position ids as cumsum(pad_masks) - 1 over more than
255 tokens, so every position from 255 on was wrong and the policies
gave wrong actions.
Cast the input of these reductions to their result dtype before the
graph goes to OpenVINO. The open pytorch#24024 proposes a similar cast for
bool sum and cumsum.
The cast exposes an OpenVINO CPU bug, seen on 2025.4.1, 2026.0.0 and
2026.4.1 but not on the nightly: with an int64 cumsum over a constant,
a loaded program gives wrong results from the second inference on.
pi0 builds its attention masks from such a cumsum, and each call's mask
kept only what earlier calls allowed. So the backend now replaces an
int64 cumsum that reads only constants with its result before
converting. Buffers the graph writes to, random ops and higher-order ops
are not treated as constants, and other constants, such as compressed
weights, are left as they are. The fold computes on the CPU, so lowering
does not need the GPU, or the pinned memory, a constant was created
with. If the CPU cannot compute such a constant, the cumsum is left to
OpenVINO as before.
The new tests run cumsum over bool and int32, prod over int32, and
pi0-style attention masks and positions over 816 tokens through one
loaded program twice, with more valid tokens the second time. Pass tests
check that mutated buffers, random draws (direct and inside cond) and
float cumsums are not folded, that cumsums over a split constant and
over CUDA and pinned constants without a GPU are, and that a folded
result does not replace a buffer.
A bad inference_precision value was only reported after the whole graph had been converted, which takes a while for a large model. The check now runs first. The test makes conversion fail if it starts, so it fails when the check comes after conversion.
FoldConstantCumsumPass evaluated every constant int64 cumsum in one go. If any of them could not be evaluated on the CPU, none were folded, and lowering only logged a generic warning. The unfolded cumsums then give wrong results from the second inference on with the affected OpenVINO CPU plugin. The pass now evaluates each cumsum separately, so one failure no longer keeps the others from being folded, and the warning names the cumsum and says that its results can be wrong.
Add the cases the earlier tests left out: - sum over a bool mask with more than 255 true values in a row, and over int32, which must come out in int64 as in PyTorch; - a full prod (prod.default) next to prod over one dimension; - 0-d tensors as inputs of the delegate, not only as outputs; - the earlier static output computed from a data dependent view, kept next to the new dynamic output case. The float32 addmm test now skips on CPUs where OpenVINO computes in float32 anyway, so a runner that cannot check the default reports a skip instead of a pass.
The Precision section listed which x86 instruction sets give bfloat16 and how CPUs without bfloat16 units simulate it. That is OpenVINO's behavior and changes with its releases, so link to OpenVINO's Precision Control page instead, and keep only what the f16 and bf16 values do on CPUs without float16 and on ARM. Also say: - that the float32 default applies when the device is CPU, and also to programs exported in float16; - that bf16 brings back the old speed of an older program on x86, and that on ARM an older program keeps float16 until it is exported again; - that OpenVINO's CPU plugin computes int64 in 32 bits, so an integer result past the int32 range wraps without an error, including the int64 result of cumsum and prod over smaller integer types.
CastReductionInputPass also cast the input of sum to int64. OpenVINO's PyTorch frontend already does that for sum, so the extra cast changed nothing. Only cumsum and prod need it. test_sum, over a bool mask with more than 255 true values in a row and over int32, passes with and without the cast, and keeps checking that a sum comes out in int64 as in PyTorch.
To let an output come back smaller than its upper bound, the backend recorded each output's size at the first call on a delegate handle, and restored it before every later call. Programs from ExecuTorch 0.7 and earlier run every call of a lowered module on one handle, each with its own output tensors, so a second call with a larger output than the first failed: OpenVINO cannot write more than a borrowed buffer's shape holds. An output whose size can change now goes to a buffer that OpenVINO owns and grows as needed, one per output, made on first use and reused by every call. After inference, the backend resizes the ExecuTorch output to OpenVINO's shape and copies the data in. Each call overwrites that buffer, so nothing from one call reaches the next. Static outputs still borrow the ExecuTorch buffer, with no copy. The new test lowers one module, calls it from two places with different output sizes on one shared handle, and runs the program twice.
FoldConstantCumsumPass only folded cumsums with an int64 result. OpenVINO's CPU plugin gives the same wrong results from the second inference on for an int32 cumsum over a constant, for example cumsum(mask, dtype=int32), so such a model still got wrong attention masks. The pass now folds any integer cumsum that reads only constants. Float cumsums are still left alone. The pi0-style mask test also runs with an int32 cumsum.
shoumikhin
force-pushed
the
openvino-precision-and-scalars
branch
from
October 30, 2026 05:05
89f775c to
bd59ee0
Compare
Contributor
Author
|
@claude /pr-review |
|
Claude finished @shoumikhin's task in 29m 47s —— View job Summary: Several points from the last review are fixed, but two remain: the float32 default still makes already-exported programs compute in float32 on x86, with no load-time way back, and the repo's export tools still can't pass Detailed reviewPR Review: #24294SummaryThis PR fixes 0-d tensors, dynamic output sizes and integer Backward Compatibility
API Design
Infrastructure
Testing
Performance
Code Quality
RecommendationNeeds Discussion. Approval waits on two decisions from the OpenVINO maintainers:
All other findings are minor.
Not done: nothing was built or run. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is wrong today
The OpenVINO backend has these problems:
ov_shape_createrefuses rank 0.x[:n].view(n, -1)with x of shape (4, 8) and n = 3 returns (4, 8).cumsumandprodof a bool or integer tensor are not computed in int64, as PyTorch does. A cumsum over a bool mask wraps after 255, so pi0 and pi0.5 get wrong position ids and wrong actions.What this change does
0-d tensors: a rank-0 tensor keeps its empty shape, which OpenVINO accepts.
Precision: when the device is
CPU, a program computes in float32 by default, both when it is lowered and when it is loaded. A program can ask for a faster precision:The value is
f32,f16orbf16, checked before conversion starts. Other devices keep OpenVINO's default.Dynamic output sizes: an output whose size can change goes to a buffer that OpenVINO owns and grows, reused by every call. After inference the backend resizes the ExecuTorch output to OpenVINO's shape and copies the data in, so one delegate can also serve several calls, as programs from ExecuTorch 0.7 and earlier do. Static outputs still use the ExecuTorch buffer directly.
Integer reductions: the input of
cumsumandprodis cast to int64 before conversion. An integer cumsum that reads only constants is computed during lowering, because OpenVINO's CPU plugin gives it wrong results from the second inference on. Each one is folded on its own, and one that cannot be folded is named in a warning.Trade-offs:
inference_precisionset tobf16.cumsumandprodover smaller integer types.Tests
Each new test fails without its fix: 0-d inputs and outputs; a 256 x 256 addmm checked to 1e-4 (skipped where OpenVINO already computes in float32); an unknown precision refused before conversion; dynamic output sizes, on one program and on one delegate shared by two calls; bool and int32 cumsum and prod; pi0-style masks with int64 and int32 cumsums over two calls; and folding when one cumsum cannot be evaluated. Two guards pass either way: integer
sum, which OpenVINO already computes in int64, and a static output after a dynamic view.The full OpenVINO op suite passes on Linux x86_64 (a CPU with bfloat16 units) with OpenVINO 2026.4.1, through both pytest and the CI test runner. With OpenVINO 2025.1.0 it also passes, except that the prod test is skipped because that release does not delegate prod.