From ea4a814e3e08801050e7284f24fd97de17918958 Mon Sep 17 00:00:00 2001 From: Anthony Shoumikhin Date: Wed, 4 Nov 2026 17:25:16 -0800 Subject: [PATCH 01/13] OpenVINO: require OpenVINO 2025.4 or later The backend allowed OpenVINO 2025.1 and later. The backend's maintainers asked to support only recent OpenVINO releases, with 2025.4, the last 2025 release, as the oldest. 2025.4 is also the first release whose CPU plugin has Identity, which is needed to fix programs that return an input unchanged. The openvino extra, the docs and the runtime's install hint now ask for openvino>=2025.4.0, and the build-from-source steps check out the 2026.4 release, the one CI tests. OpenVINO's Linux aarch64 builds need glibc 2.35 from 2025.3 on, while the ExecuTorch wheel needs only 2.28. The extra used to skip aarch64 only with Python 3.14, where every aarch64 build needs glibc 2.35. Every OpenVINO release the extra now allows has aarch64 builds, all needing glibc 2.35, and those from 2026.2 on also cover Python 3.14. So the extra now keeps aarch64 for every Python. On an older aarch64 system, pip then installs an older ExecuTorch, such as 1.6.0, without an error. The docs and the wheel README say that aarch64 needs glibc 2.35 or newer, as on Ubuntu 22.04 or Raspberry Pi OS Bookworm and later, and what to do on an older system. --- README-wheel.md | 7 +++++-- backends/openvino/README.md | 6 +++--- backends/openvino/runtime/OpenvinoBackend.cpp | 2 +- docs/source/build-run-openvino.md | 13 ++++++++----- docs/source/using-executorch-cpp.md | 8 +++++--- pyproject.toml | 8 ++++---- 6 files changed, 26 insertions(+), 18 deletions(-) diff --git a/README-wheel.md b/README-wheel.md index 5245e312bcd..efc089dee84 100644 --- a/README-wheel.md +++ b/README-wheel.md @@ -37,8 +37,11 @@ to run ExecuTorch `.pte` files, with some restrictions: * \[Linux and Windows] [OpenVINO](docs/source/build-run-openvino.md) backend is also linked into the prebuilt module. OpenVINO requires the runtime to be installed separately: `pip install executorch[openvino]`. For now the extra installs no runtime on Python 3.15, - or on Linux aarch64 with Python 3.14. OpenVINO publishes no build for Python 3.15, and its - aarch64 builds for Python 3.14 need glibc 2.35 or newer. + where OpenVINO publishes no build. On Linux aarch64, OpenVINO needs glibc 2.35 or newer, + as on Ubuntu 22.04 or Raspberry Pi OS Bookworm and later. On an older aarch64 system, such as + RHEL 9 or Amazon Linux 2023, `pip install executorch[openvino]` does not fail but installs an + older ExecuTorch release; see the [OpenVINO docs](docs/source/build-run-openvino.md) for what + to do there. On Linux, macOS and Windows the runtime, kernels and backends also ship as shared libraries with headers and a CMake package, so a C++ application can link them without building diff --git a/backends/openvino/README.md b/backends/openvino/README.md index 883e1e285f6..76f834a59f7 100644 --- a/backends/openvino/README.md +++ b/backends/openvino/README.md @@ -20,7 +20,7 @@ On Linux and Windows, the OpenVINO backend is included in the ExecuTorch pip whe pip install executorch[openvino] ``` -For now the extra installs no runtime on Python 3.15, or on Linux aarch64 with Python 3.14. OpenVINO publishes no build for Python 3.15, and its aarch64 builds for Python 3.14 need glibc 2.35 or newer. +For now the extra installs no runtime on Python 3.15, where OpenVINO publishes no build. On Linux aarch64, OpenVINO needs glibc 2.35 or newer, as on Ubuntu 22.04 or Raspberry Pi OS Bookworm and later. On an older aarch64 system, `pip install executorch[openvino]` does not fail: pip finds no OpenVINO 2025.4 or newer for that system, so it installs an older ExecuTorch release instead. On such a system, install `executorch` without the extra, then build OpenVINO from source as described below. The backend automatically discovers the OpenVINO C library from the pip-installed package, so no library search path setup is needed. @@ -139,7 +139,7 @@ executorch ### Prerequisites -Before you begin, ensure you have openvino installed and configured on your system. +Before you begin, ensure you have OpenVINO 2025.4 or newer installed and configured on your system. ### Use OpenVINO from Release Packages @@ -157,7 +157,7 @@ Before you begin, ensure you have openvino installed and configured on your syst ```bash git clone https://github.com/openvinotoolkit/openvino.git -cd openvino +cd openvino && git checkout releases/2026/4 git submodule update --init --recursive sudo ./install_build_dependencies.sh mkdir build && cd build diff --git a/backends/openvino/runtime/OpenvinoBackend.cpp b/backends/openvino/runtime/OpenvinoBackend.cpp index ff7c33aa4d9..15af84040bf 100644 --- a/backends/openvino/runtime/OpenvinoBackend.cpp +++ b/backends/openvino/runtime/OpenvinoBackend.cpp @@ -214,7 +214,7 @@ bool OpenvinoBackend::ensure_loaded() const { Error, "OpenVINO runtime not found (loading '%s' failed: %s). " "Ensure '%s' can be found (%s), or install with: " - "pip install \"openvino>=2025.1.0,<2027.0.0\"", + "pip install \"openvino>=2025.4.0,<2027.0.0\"", to_utf8(effective_path).c_str(), error.c_str(), to_utf8(kDefaultLibName).c_str(), diff --git a/docs/source/build-run-openvino.md b/docs/source/build-run-openvino.md index 2d842ed2bcd..a6303158b49 100644 --- a/docs/source/build-run-openvino.md +++ b/docs/source/build-run-openvino.md @@ -43,9 +43,12 @@ The easiest way to use the OpenVINO backend is via the prebuilt pip wheel: pip install executorch[openvino] ``` -For now the extra installs no runtime on Python 3.15, or on Linux aarch64 with Python -3.14. OpenVINO publishes no build for Python 3.15, and its aarch64 builds for Python 3.14 -need glibc 2.35 or newer. +For now the extra installs no runtime on Python 3.15, where OpenVINO publishes no build. +On Linux aarch64, OpenVINO needs glibc 2.35 or newer, as on Ubuntu 22.04 or Raspberry Pi OS +Bookworm and later. On an older aarch64 system, `pip install executorch[openvino]` does not +fail: pip finds no OpenVINO 2025.4 or newer for that system, so it installs an older +ExecuTorch release instead. On such a system, install `executorch` without the extra, then +build OpenVINO from source as described below. The backend automatically discovers the OpenVINO C library from the pip-installed package — no environment variable setup is needed. @@ -61,14 +64,14 @@ If you need a custom build (different platform, custom configuration, etc.), fol Before you begin, ensure you have the OpenVINO runtime installed and configured on your system. For most users, we recommend installing it via pip: ```bash -pip install "openvino>=2025.1.0,<2027.0.0" +pip install "openvino>=2025.4.0,<2027.0.0" ``` Alternatively, you can build OpenVINO from source or follow the official [OpenVINO installation guide](https://docs.openvino.ai/2025/get-started/install-openvino.html) for platform-specific options: ```bash git clone https://github.com/openvinotoolkit/openvino.git -cd openvino && git checkout releases/2025/1 +cd openvino && git checkout releases/2026/4 git submodule update --init --recursive sudo ./install_build_dependencies.sh mkdir build && cd build diff --git a/docs/source/using-executorch-cpp.md b/docs/source/using-executorch-cpp.md index 2391584196f..015b286243a 100644 --- a/docs/source/using-executorch-cpp.md +++ b/docs/source/using-executorch-cpp.md @@ -272,9 +272,11 @@ $env:OPENVINO_LIB_PATH = python -c "import openvino, os; print(os.path.join(os.p Without it the delegate still registers and the program still links, and the failure arrives later, when the model is loaded. -For now the openvino extra installs no runtime on Python 3.15, or on Linux aarch64 with -Python 3.14. OpenVINO publishes no build for Python 3.15, and its aarch64 builds for Python -3.14 need glibc 2.35 or newer. +For now the openvino extra installs no runtime on Python 3.15, where OpenVINO publishes no +build. On Linux aarch64, OpenVINO needs glibc 2.35 or newer, as on Ubuntu 22.04 or Raspberry +Pi OS Bookworm and later. On an older aarch64 system, such as RHEL 9 or Amazon Linux 2023, +`pip install executorch[openvino]` does not fail but installs an older ExecuTorch release; see +[OpenVINO Backend](build-run-openvino.md) for what to do there. The MLX delegate has a similar requirement. Its Metal kernels live in a separate `mlx.metallib` file, which MLX looks for next to whichever library holds MLX code. The wheel ships it beside the diff --git a/pyproject.toml b/pyproject.toml index 116552df1ed..f1d717bc1eb 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -86,10 +86,10 @@ ethos_u = [ "tosa-tools==2026.5.0", ] openvino = [ - # OpenVINO publishes no build for Python 3.15, and its Linux aarch64 builds for Python - # 3.14 need glibc 2.35 while the wheel needs only 2.28. Without these bounds pip fails to - # install the extra on Python 3.15 and on those older aarch64 systems. - "openvino>=2025.1.0,<2027.0.0; (platform_system == 'Linux' or (platform_system == 'Windows' and platform_machine == 'AMD64')) and python_version < '3.15' and (platform_machine != 'aarch64' or python_version < '3.14')", + # OpenVINO publishes no build for Python 3.15. Its Linux aarch64 builds need glibc 2.35 or + # newer while the wheel needs only 2.28, so on an older aarch64 system pip installs an older + # ExecuTorch for this extra instead; the docs say what to do there. + "openvino>=2025.4.0,<2027.0.0; (platform_system == 'Linux' or (platform_system == 'Windows' and platform_machine == 'AMD64')) and python_version < '3.15'", ] [project.urls] From fb5c5891944763869522cb6f5d7330192f215110 Mon Sep 17 00:00:00 2001 From: Anthony Shoumikhin Date: Tue, 3 Nov 2026 06:57:49 -0800 Subject: [PATCH 02/13] OpenVINO: keep copies of constants inside the partition The partitioner moves a copy out of an OpenVINO partition when the copy would return one of the partition's inputs unchanged, because OpenVINO gives stale values for such an output. It also did this for copies of weights and buffers, but those never had the problem: they are stored inside the partition as constants, not passed in as inputs. Moving them out cost something. A buffer that is returned as a copy and also used inside the partition was stored twice in the program, such a buffer marked no_copy failed to export, and with dim order turned off a clone(memory_format=torch.preserve_format) of a buffer failed at run time in the portable clone kernel, which accepts only contiguous. Now the partitioner leaves copies of constants in place. Copies of real inputs, including mutated buffers, still move out, so such a clone of an input still fails in that kernel. execute_layer_test now returns the program, so a test can check what ended up outside the delegate. --- backends/openvino/partitioner.py | 32 +++++++++++-- .../tests/ops/base_openvino_op_test.py | 2 + backends/openvino/tests/ops/test_copy.py | 48 +++++++++++++++++++ 3 files changed, 79 insertions(+), 3 deletions(-) diff --git a/backends/openvino/partitioner.py b/backends/openvino/partitioner.py index 4d6cfd2f93e..4941c2b9548 100644 --- a/backends/openvino/partitioner.py +++ b/backends/openvino/partitioner.py @@ -56,7 +56,31 @@ def _copy_source(node: torch.fx.Node) -> Optional[torch.fx.Node]: return source if isinstance(source, torch.fx.Node) else None -def _remove_copies_of_inputs(partition: Partition) -> None: +def _constants(program: ExportedProgram) -> set[torch.fx.Node]: + """The placeholders that each partition reading them stores as its own + data (see tag_constant_data), so OpenVINO gets them as constants, not as + inputs.""" + signature = program.graph_signature + mutated = set(signature.buffers_to_mutate.values()) + names = { + *signature.inputs_to_parameters, + *signature.inputs_to_lifted_tensor_constants, + *( + name + for name, buffer in signature.inputs_to_buffers.items() + if buffer not in mutated + ), + } + return { + node + for node in program.graph.nodes + if node.op == "placeholder" and node.name in names + } + + +def _remove_copies_of_inputs( + partition: Partition, constants: set[torch.fx.Node] +) -> None: """Leave out of the partition each copy that would return one of the partition's inputs, unchanged, as one of its outputs. @@ -65,12 +89,13 @@ def _remove_copies_of_inputs(partition: Partition) -> None: output and can overwrite the input with the output buffer's old contents. Copies used only inside the partition stay, because OpenVINO reads every input in the default dim order, which a copy left out could have changed. + Copies of constants stay too, since constants are not partition inputs. """ def copies_input(node: torch.fx.Node) -> bool: while (source := _copy_source(node)) is not None: if source not in partition.nodes: - return True + return source not in constants node = source return False @@ -294,8 +319,9 @@ def partition(self, exported_program: ExportedProgram) -> PartitionResult: partition_list = partitioner.propose_partitions() partition_tags = {} + constants = _constants(exported_program) for partition in partition_list: - _remove_copies_of_inputs(partition) + _remove_copies_of_inputs(partition, constants) for node in partition.nodes: tag = f"tag{partition.id}" node.meta["delegation_tag"] = tag diff --git a/backends/openvino/tests/ops/base_openvino_op_test.py b/backends/openvino/tests/ops/base_openvino_op_test.py index df921c4c4ac..c7a00bdcbf6 100644 --- a/backends/openvino/tests/ops/base_openvino_op_test.py +++ b/backends/openvino/tests/ops/base_openvino_op_test.py @@ -105,3 +105,5 @@ def execute_layer_test( ), msg=f"ref_output:\n{ref_output[i]}\n\ntest_output:\n{outputs[i]}", ) + + return exec_prog diff --git a/backends/openvino/tests/ops/test_copy.py b/backends/openvino/tests/ops/test_copy.py index c65b1d35272..c9ba131f138 100644 --- a/backends/openvino/tests/ops/test_copy.py +++ b/backends/openvino/tests/ops/test_copy.py @@ -42,3 +42,51 @@ def forward(self, x): # Run outside, the clone would hand OpenVINO a channels-last input, # which it reads as if it were contiguous. self.execute_layer_test(ChannelsLastMean(), (torch.randn(2, 3, 4, 5),)) + + def test_copy_of_constant_as_output(self): + class CopyOfConstant(torch.nn.Module): + def __init__(self, kind): + super().__init__() + value = torch.randn(2, 3, 4, 5) + if kind == "parameter": + self.constant = torch.nn.Parameter(value) + elif kind == "buffer": + self.register_buffer("constant", value) + else: + self.constant = value + + def forward(self, x): + return self.constant.clone(), x + self.constant + + for kind in ("parameter", "buffer", "lifted tensor"): + with self.subTest(kind=kind): + program = self.execute_layer_test( + CopyOfConstant(kind), (torch.randn(2, 3, 4, 5),) + ) + # OpenVINO gets a constant as its own data, not as an input, so + # the copy stays. Run outside, it would store the constant twice. + plan = program.executorch_program.execution_plan[0] + self.assertEqual(plan.operators, []) + + def test_copy_of_mutated_buffer_as_output(self): + class CopyOfMutatedBuffer(torch.nn.Module): + def __init__(self): + super().__init__() + self.register_buffer("buffer", torch.randn(2, 3, 4, 5)) + + def forward(self, x): + copy = self.buffer.clone() + self.buffer.add_(x) + return copy, torch.sin(copy) + + # The program does not keep the starting value of a mutated buffer, so + # there is no reference output to compare with. + program = self.execute_layer_test( + CopyOfMutatedBuffer(), + (torch.randn(2, 3, 4, 5),), + assert_output_equal=False, + ) + # A mutated buffer is passed to the partition as an input, so its copy + # must run outside. + operators = program.executorch_program.execution_plan[0].operators + self.assertIn("dim_order_ops::_clone_dim_order", [op.name for op in operators]) From c8958bd349b49d16a314a64b7429b16e2e4d0077 Mon Sep 17 00:00:00 2001 From: Anthony Shoumikhin Date: Tue, 3 Nov 2026 07:28:51 -0800 Subject: [PATCH 03/13] OpenVINO: copy inputs that a whole program returns unchanged When a program returns a copy of its input, like `return x.clone()`, the OpenVINO converter turns the copy into nothing, so the converted model returns its input directly. The CPU plugin does not write such an output, and it can overwrite the input, so the program gives stale values. The partitioner already avoids this by running such copies on the CPU. Lowering a whole program with to_backend, without a partitioner, has no CPU part to run them on, so it still gave wrong values. Now preprocess puts an OpenVINO Identity, which copies its input, between such an input and the output. OpenVINO's shared graph passes do not remove an operation that connects an input straight to an output, and the CPU plugin keeps the Identity as a real copy for every type and shape tried. A Convert to the same type is not safe: the CPU plugin removes it for some types and for dynamic shapes. --- backends/openvino/preprocess.py | 18 ++++++++++++++- backends/openvino/tests/ops/test_copy.py | 28 +++++++++++++++++++++++- 2 files changed, 44 insertions(+), 2 deletions(-) diff --git a/backends/openvino/preprocess.py b/backends/openvino/preprocess.py index e11ac05aac1..9090d4df6ad 100644 --- a/backends/openvino/preprocess.py +++ b/backends/openvino/preprocess.py @@ -22,7 +22,7 @@ ) from executorch.exir.backend.compile_spec_schema import CompileSpec from executorch.exir.passes.memory_format_ops_pass import DimOrderOpsRevertPass -from openvino import compile_model, opset13 # type: ignore[import-untyped] +from openvino import compile_model, opset13, opset16 # type: ignore[import-untyped] from openvino.frontend import FrontEndManager # type: ignore[import-untyped] from openvino.frontend.pytorch.fx_decoder import ( # type: ignore[import-untyped] TorchFXPythonDecoder, @@ -63,6 +63,21 @@ def _match_partition_inputs( ) +def _copy_returned_inputs(model) -> None: + """Make the converted model compute each output that is one of its inputs. + + The converter turns copies like aten.clone into nothing, so such an output + is the input itself. The CPU plugin does not write it, and can overwrite + the input with the output buffer's old contents. OpenVINO keeps an + Identity between an input and an output, and the CPU plugin runs it as a + copy, while it can remove a Convert to the same type. + """ + for result in model.get_results(): + source = result.input_value(0) + if source.get_node().get_type_name() == "Parameter": + result.input(0).replace_source_output(opset16.identity(source).output(0)) + + @final class OpenvinoBackend(BackendDetails): @@ -142,6 +157,7 @@ def preprocess( frontend = FrontEndManager().load_by_framework("pytorch") model = frontend.convert(frontend.load(decoder)) _match_partition_inputs(model, module, input_shapes, input_types) + _copy_returned_inputs(model) ppp = PrePostProcessor(model) # The converter takes an int input, like a SymInt size, as a scalar, but # the runtime passes it as an int64 tensor of shape [1]. diff --git a/backends/openvino/tests/ops/test_copy.py b/backends/openvino/tests/ops/test_copy.py index c9ba131f138..a9b0060c2ff 100644 --- a/backends/openvino/tests/ops/test_copy.py +++ b/backends/openvino/tests/ops/test_copy.py @@ -1,8 +1,13 @@ import torch +from executorch.backends.openvino.preprocess import OpenvinoBackend from executorch.backends.openvino.tests.ops.base_openvino_op_test import ( BaseOpenvinoOpTest, ) -from executorch.exir import EdgeCompileConfig +from executorch.exir import EdgeCompileConfig, to_edge +from executorch.exir.backend.backend_api import to_backend +from executorch.exir.backend.backend_details import CompileSpec +from executorch.runtime import Runtime +from torch.export import export class TestCopyOperator(BaseOpenvinoOpTest): @@ -90,3 +95,24 @@ def forward(self, x): # must run outside. operators = program.executorch_program.execution_plan[0].operators self.assertIn("dim_order_ops::_clone_dim_order", [op.name for op in operators]) + + def test_whole_program_returning_copy_of_input(self): + class CopyAndSin(torch.nn.Module): + def forward(self, x): + copy = x.clone() + return copy, torch.sin(copy) + + x = torch.randn(2, 3, 4, 5) + # Without a partitioner the copy cannot run on the CPU, so OpenVINO + # has to make it. + lowered = to_backend( + OpenvinoBackend.__name__, + to_edge(export(CopyAndSin(), (x,))).exported_program(), + [CompileSpec("device", self.device.encode())], + ) + program = Runtime.get().load_program(lowered.buffer()) + method = program.load_method("forward") + assert method is not None + torch.testing.assert_close( + method.execute((x,)), [x, torch.sin(x)], atol=self.atol, rtol=self.rtol + ) From d7844b633690f65be36647c1d19a36bc586db7f7 Mon Sep 17 00:00:00 2001 From: Anthony Shoumikhin Date: Tue, 3 Nov 2026 08:49:09 -0800 Subject: [PATCH 04/13] OpenVINO: move a copy of a constant out when OpenVINO cannot keep it "OpenVINO: keep copies of constants inside the partition" kept every copy of a constant inside the OpenVINO partition. That is wrong in the cases below, and the copy now moves out to the CPU again for them: - The copy changes the dim order, like `buffer.clone(memory_format=torch.channels_last)`. OpenVINO writes every output in the default dim order, so the values came back permuted. - The CPU also reads the constant, for example through an operator the partitioner does not take. The CPU stores the constant anyway, and a copy kept in the partition made the partition store it a second time. In these cases the costs of a copy made on the CPU come back for it: a no_copy constant read by both sides fails to export, and with dim order turned off a preserve_format clone fails in the portable clone kernel. A copy left out that is not in the default dim order is still read wrong by ops inside the partition, the same limit inputs already have. The mutated buffer test now also runs the program and checks the copy against the call before it. The constant test runs a second input. --- backends/openvino/partitioner.py | 13 +++++- backends/openvino/tests/ops/test_copy.py | 58 ++++++++++++++++++++++-- 2 files changed, 66 insertions(+), 5 deletions(-) diff --git a/backends/openvino/partitioner.py b/backends/openvino/partitioner.py index 4941c2b9548..ff827c00744 100644 --- a/backends/openvino/partitioner.py +++ b/backends/openvino/partitioner.py @@ -89,13 +89,22 @@ def _remove_copies_of_inputs( output and can overwrite the input with the output buffer's old contents. Copies used only inside the partition stay, because OpenVINO reads every input in the default dim order, which a copy left out could have changed. - Copies of constants stay too, since constants are not partition inputs. + A copy of a constant only this partition reads stays too, since such a + constant is not a partition input, unless OpenVINO would write the copy in + the wrong dim order: it writes every output in the default one. Like any + input, a copy left out that is not in the default dim order is still read + wrong by the ops that use it inside. """ def copies_input(node: torch.fx.Node) -> bool: + copy = node while (source := _copy_source(node)) is not None: if source not in partition.nodes: - return source not in constants + return not ( + source in constants + and all(user in partition.nodes for user in source.users) + and copy.meta["val"].is_contiguous() + ) node = source return False diff --git a/backends/openvino/tests/ops/test_copy.py b/backends/openvino/tests/ops/test_copy.py index a9b0060c2ff..039a1c105f9 100644 --- a/backends/openvino/tests/ops/test_copy.py +++ b/backends/openvino/tests/ops/test_copy.py @@ -1,3 +1,6 @@ +import os +import tempfile + import torch from executorch.backends.openvino.preprocess import OpenvinoBackend from executorch.backends.openvino.tests.ops.base_openvino_op_test import ( @@ -66,13 +69,46 @@ def forward(self, x): for kind in ("parameter", "buffer", "lifted tensor"): with self.subTest(kind=kind): program = self.execute_layer_test( - CopyOfConstant(kind), (torch.randn(2, 3, 4, 5),) + CopyOfConstant(kind), + (torch.randn(2, 3, 4, 5),), + more_inputs=((torch.randn(2, 3, 4, 5),),), ) # OpenVINO gets a constant as its own data, not as an input, so # the copy stays. Run outside, it would store the constant twice. plan = program.executorch_program.execution_plan[0] self.assertEqual(plan.operators, []) + def test_copy_of_constant_that_runs_outside(self): + class CopyOfBuffer(torch.nn.Module): + def __init__(self, copy, read): + super().__init__() + self.copy = copy + self.read = read + self.register_buffer("buffer", torch.randn(2, 3, 4, 5)) + + def forward(self, x): + return self.copy(self.buffer), self.read(self.buffer) + x + + cases = { + # OpenVINO writes every output in the default dim order. + "channels last": ( + lambda b: b.clone(memory_format=torch.channels_last), + lambda b: b, + ), + # The partitioner does not take atan, so the CPU stores the buffer + # anyway, and a copy made there keeps it stored once. + "also read outside": (lambda b: b.clone(), torch.atan), + } + for name, (copy, read) in cases.items(): + with self.subTest(case=name): + program = self.execute_layer_test( + CopyOfBuffer(copy, read), (torch.randn(2, 3, 4, 5),) + ) + operators = program.executorch_program.execution_plan[0].operators + self.assertIn( + "dim_order_ops::_clone_dim_order", [op.name for op in operators] + ) + def test_copy_of_mutated_buffer_as_output(self): class CopyOfMutatedBuffer(torch.nn.Module): def __init__(self): @@ -84,8 +120,6 @@ def forward(self, x): self.buffer.add_(x) return copy, torch.sin(copy) - # The program does not keep the starting value of a mutated buffer, so - # there is no reference output to compare with. program = self.execute_layer_test( CopyOfMutatedBuffer(), (torch.randn(2, 3, 4, 5),), @@ -96,6 +130,24 @@ def forward(self, x): operators = program.executorch_program.execution_plan[0].operators self.assertIn("dim_order_ops::_clone_dim_order", [op.name for op in operators]) + # The program does not keep the starting value of a mutated buffer, so + # check each call against the one before it. + x1, x2 = torch.randn(2, 3, 4, 5), torch.randn(2, 3, 4, 5) + with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmp_dir: + pte_path = os.path.join(tmp_dir, "model.pte") + with open(pte_path, "wb") as f: + program.write_to_file(f) + loaded = Runtime.get().load_program(pte_path) + method = loaded.load_method("forward") + assert method is not None + first = method.execute((x1,)) + second = method.execute((x2,)) + del method, loaded + tolerance = {"atol": self.atol, "rtol": self.rtol} + torch.testing.assert_close(second[0], first[0] + x1, **tolerance) + for copy, sine in (first, second): + torch.testing.assert_close(sine, torch.sin(copy), **tolerance) + def test_whole_program_returning_copy_of_input(self): class CopyAndSin(torch.nn.Module): def forward(self, x): From f73e6b3344065e5a46f0864f5965bc487da7ba6b Mon Sep 17 00:00:00 2001 From: Anthony Shoumikhin Date: Tue, 3 Nov 2026 08:49:09 -0800 Subject: [PATCH 05/13] OpenVINO: insert the Identity copy only for the CPU device The Identity that preprocess puts between an input and an output fixes a problem seen only in the CPU plugin, and it was only checked there. Other devices now get the converted model unchanged, as before, so a device without Identity cannot start failing. This follows the CPU check preprocess already makes for the inference precision. The partitioner path still runs such copies on the CPU, which is exact on every device. The whole program test now covers float, int64 and bool inputs, a dynamic shape, and a second call on the same loaded method, and it loads the program from a file like the other tests. Its input also feeds a reshape, so the copied output is not the input's only user. --- backends/openvino/preprocess.py | 9 ++-- backends/openvino/tests/ops/test_copy.py | 62 +++++++++++++++++------- 2 files changed, 50 insertions(+), 21 deletions(-) diff --git a/backends/openvino/preprocess.py b/backends/openvino/preprocess.py index 9090d4df6ad..df5f237368e 100644 --- a/backends/openvino/preprocess.py +++ b/backends/openvino/preprocess.py @@ -63,15 +63,18 @@ def _match_partition_inputs( ) -def _copy_returned_inputs(model) -> None: +def _copy_returned_inputs(model, device: str) -> None: """Make the converted model compute each output that is one of its inputs. The converter turns copies like aten.clone into nothing, so such an output is the input itself. The CPU plugin does not write it, and can overwrite the input with the output buffer's old contents. OpenVINO keeps an Identity between an input and an output, and the CPU plugin runs it as a - copy, while it can remove a Convert to the same type. + copy, while it can remove a Convert to the same type. Other devices are + left as they are, since neither the problem nor Identity was checked there. """ + if not device.startswith("CPU"): + return for result in model.get_results(): source = result.input_value(0) if source.get_node().get_type_name() == "Parameter": @@ -157,7 +160,7 @@ def preprocess( frontend = FrontEndManager().load_by_framework("pytorch") model = frontend.convert(frontend.load(decoder)) _match_partition_inputs(model, module, input_shapes, input_types) - _copy_returned_inputs(model) + _copy_returned_inputs(model, device) ppp = PrePostProcessor(model) # The converter takes an int input, like a SymInt size, as a scalar, but # the runtime passes it as an int64 tensor of shape [1]. diff --git a/backends/openvino/tests/ops/test_copy.py b/backends/openvino/tests/ops/test_copy.py index 039a1c105f9..48033e9a0f6 100644 --- a/backends/openvino/tests/ops/test_copy.py +++ b/backends/openvino/tests/ops/test_copy.py @@ -10,7 +10,7 @@ from executorch.exir.backend.backend_api import to_backend from executorch.exir.backend.backend_details import CompileSpec from executorch.runtime import Runtime -from torch.export import export +from torch.export import Dim, export class TestCopyOperator(BaseOpenvinoOpTest): @@ -149,22 +149,48 @@ def forward(self, x): torch.testing.assert_close(sine, torch.sin(copy), **tolerance) def test_whole_program_returning_copy_of_input(self): - class CopyAndSin(torch.nn.Module): + class CopyAndReshape(torch.nn.Module): def forward(self, x): - copy = x.clone() - return copy, torch.sin(copy) + return x.clone(), x.reshape(-1) - x = torch.randn(2, 3, 4, 5) - # Without a partitioner the copy cannot run on the CPU, so OpenVINO - # has to make it. - lowered = to_backend( - OpenvinoBackend.__name__, - to_edge(export(CopyAndSin(), (x,))).exported_program(), - [CompileSpec("device", self.device.encode())], - ) - program = Runtime.get().load_program(lowered.buffer()) - method = program.load_method("forward") - assert method is not None - torch.testing.assert_close( - method.execute((x,)), [x, torch.sin(x)], atol=self.atol, rtol=self.rtol - ) + class Lowered(torch.nn.Module): + def __init__(self, lowered): + super().__init__() + self.lowered = lowered + + def forward(self, x): + return self.lowered(x) + + inputs = { + "float": lambda n: torch.randn(n, 3), + "int64": lambda n: torch.randint(-100, 100, (n, 3)), + "bool": lambda n: torch.randn(n, 3) > 0, + } + dynamic_shapes = ({0: Dim("rows", min=1, max=4)},) + for name, make in inputs.items(): + with self.subTest(dtype=name): + x = make(4) + # Without a partitioner the copy cannot run on the CPU, so + # OpenVINO has to make it. + lowered = to_backend( + OpenvinoBackend.__name__, + to_edge( + export(CopyAndReshape(), (x,), dynamic_shapes=dynamic_shapes) + ).exported_program(), + [CompileSpec("device", self.device.encode())], + ) + program = to_edge( + export(Lowered(lowered), (x,), dynamic_shapes=dynamic_shapes) + ).to_executorch() + with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmp: + pte_path = os.path.join(tmp, "model.pte") + with open(pte_path, "wb") as f: + program.write_to_file(f) + loaded = Runtime.get().load_program(pte_path) + method = loaded.load_method("forward") + assert method is not None + for value in (x, make(2)): + copy, flat = method.execute((value,)) + torch.testing.assert_close(copy, value) + torch.testing.assert_close(flat, value.reshape(-1)) + del method, loaded From 687959a7370586bdf4349cd9ae2c1147e76ce589 Mon Sep 17 00:00:00 2001 From: Anthony Shoumikhin Date: Tue, 3 Nov 2026 10:50:07 -0800 Subject: [PATCH 06/13] OpenVINO: move whole copy chains and 64-bit copies of constants out A copy of a constant stayed in the partition when only the partition read the constant and the copy kept the default dim order. That still broke in these cases: - A chain like `buffer.clone().clone(memory_format=torch.channels_last)` whose result is also read inside. The outer copy left, but the inner one stayed, so the partition fed the copy taken out and also read it. Export failed with "Invalid partition, found dependency cycles". Now a copy that leaves takes the copies it reads with it. Such a model now exports as on main. A channels-last copy read inside is still read in the default dim order, the limit every partition input has. - A copy of an int64 or float64 constant. The CPU plugin computes 64-bit types in 32 bits, so the copy came back changed. Such copies now run on the CPU again. The test for copies that run outside covers both, and the case where the CPU also reads the constant now checks that atan runs there. --- backends/openvino/partitioner.py | 16 +++++++--- backends/openvino/tests/ops/test_copy.py | 40 +++++++++++++++--------- 2 files changed, 37 insertions(+), 19 deletions(-) diff --git a/backends/openvino/partitioner.py b/backends/openvino/partitioner.py index ff827c00744..2796104e98d 100644 --- a/backends/openvino/partitioner.py +++ b/backends/openvino/partitioner.py @@ -89,9 +89,10 @@ def _remove_copies_of_inputs( output and can overwrite the input with the output buffer's old contents. Copies used only inside the partition stay, because OpenVINO reads every input in the default dim order, which a copy left out could have changed. - A copy of a constant only this partition reads stays too, since such a - constant is not a partition input, unless OpenVINO would write the copy in - the wrong dim order: it writes every output in the default one. Like any + A copy of a constant that only this partition reads stays too, since the + constant is not a partition input. Such a copy still leaves when it changes + the dim order, since OpenVINO writes every output in the default one, or + has a 64-bit type, since the CPU plugin computes those in 32 bits. Like any input, a copy left out that is not in the default dim order is still read wrong by the ops that use it inside. """ @@ -104,18 +105,23 @@ def copies_input(node: torch.fx.Node) -> bool: source in constants and all(user in partition.nodes for user in source.users) and copy.meta["val"].is_contiguous() + and copy.meta["val"].dtype.itemsize < 8 ) node = source return False - # Taking out a copy can make the copy it reads an output. + # Taking out a copy can make another copy leave. while leaving := [ node for node in partition.nodes if copies_input(node) and any(u not in partition.nodes for u in node.users) ]: for node in leaving: - partition.remove_node(node) + # The copies it reads leave with it. A copy of a constant left inside + # could feed it while the partition also reads it, a cycle. + while (inner := _copy_source(node)) is not None and node in partition.nodes: + partition.remove_node(node) + node = inner class OpenvinoOperatorsSupport(OperatorSupportBase): diff --git a/backends/openvino/tests/ops/test_copy.py b/backends/openvino/tests/ops/test_copy.py index 48033e9a0f6..24a07f472fb 100644 --- a/backends/openvino/tests/ops/test_copy.py +++ b/backends/openvino/tests/ops/test_copy.py @@ -79,35 +79,47 @@ def forward(self, x): self.assertEqual(plan.operators, []) def test_copy_of_constant_that_runs_outside(self): - class CopyOfBuffer(torch.nn.Module): - def __init__(self, copy, read): + class Buffered(torch.nn.Module): + def __init__(self, body, buffer): super().__init__() - self.copy = copy - self.read = read - self.register_buffer("buffer", torch.randn(2, 3, 4, 5)) + self.body = body + self.register_buffer("buffer", buffer) def forward(self, x): - return self.copy(self.buffer), self.read(self.buffer) + x + return self.body(self.buffer, x) + + def copy_of_copy_and_sin(buffer, x): + copy = buffer.clone().clone(memory_format=torch.channels_last) + return copy, torch.sin(copy) + floats = torch.randn(2, 3, 4, 5) cases = { # OpenVINO writes every output in the default dim order. "channels last": ( - lambda b: b.clone(memory_format=torch.channels_last), - lambda b: b, + lambda b, x: (b.clone(memory_format=torch.channels_last), b + x), + floats, + ), + # The inner copy must leave too, or export finds a cycle. + "copy of a copy, channels last": (copy_of_copy_and_sin, floats), + # The CPU plugin computes 64-bit types in 32 bits. + "int64": ( + lambda b, x: (b.clone(), x + 1), + torch.tensor([2**40, -(2**40)]), ), # The partitioner does not take atan, so the CPU stores the buffer # anyway, and a copy made there keeps it stored once. - "also read outside": (lambda b: b.clone(), torch.atan), + "also read outside": (lambda b, x: (b.clone(), torch.atan(b) + x), floats), } - for name, (copy, read) in cases.items(): + for name, (body, buffer) in cases.items(): with self.subTest(case=name): program = self.execute_layer_test( - CopyOfBuffer(copy, read), (torch.randn(2, 3, 4, 5),) + Buffered(body, buffer), (torch.randn(2, 3, 4, 5),) ) operators = program.executorch_program.execution_plan[0].operators - self.assertIn( - "dim_order_ops::_clone_dim_order", [op.name for op in operators] - ) + names = [op.name for op in operators] + self.assertIn("dim_order_ops::_clone_dim_order", names) + if name == "also read outside": + self.assertIn("aten::atan", names) def test_copy_of_mutated_buffer_as_output(self): class CopyOfMutatedBuffer(torch.nn.Module): From 642cfaa980fe2240525dffb3060978af6a59c342 Mon Sep 17 00:00:00 2001 From: Anthony Shoumikhin Date: Tue, 3 Nov 2026 10:50:29 -0800 Subject: [PATCH 07/13] OpenVINO: test where copies of inputs run, and static whole programs On the CPU, preprocess now copies an input that a partition returns unchanged, so a copy of an input that wrongly stayed in the partition would still give right values. test_copy_of_input_as_output now also checks that each copy runs on the CPU, which is the only safeguard on other devices. The whole program test runs each input type with static shapes too. For a static output the runtime binds ExecuTorch's own buffer, which is the case the bug hit, while a dynamic output goes through a tensor OpenVINO owns. --- backends/openvino/tests/ops/test_copy.py | 39 ++++++++++++++---------- 1 file changed, 23 insertions(+), 16 deletions(-) diff --git a/backends/openvino/tests/ops/test_copy.py b/backends/openvino/tests/ops/test_copy.py index 24a07f472fb..ab4d29fc719 100644 --- a/backends/openvino/tests/ops/test_copy.py +++ b/backends/openvino/tests/ops/test_copy.py @@ -1,3 +1,4 @@ +import itertools import os import tempfile @@ -25,22 +26,28 @@ def forward(self, x): copy = self.copy(x) return copy, torch.sin(copy) + clone = "dim_order_ops::_clone_dim_order" copies = { - "clone": lambda x: x.clone(), - "full slice": lambda x: x[:, :], - "clone of a clone": lambda x: x.clone().clone(), + "clone": (lambda x: x.clone(), clone, None), + "full slice": (lambda x: x[:, :], "aten::alias_copy", None), + "clone of a clone": (lambda x: x.clone().clone(), clone, None), + # LLM exports turn dim order off, which makes a clone aten.clone. + "clone, dim order off": ( + lambda x: x.clone(), + "aten::clone", + EdgeCompileConfig(_skip_dim_order=True), + ), } inputs = (torch.randn(2, 3, 4, 5),) - for name, copy in copies.items(): + for name, (copy, operator, config) in copies.items(): with self.subTest(copy=name): - self.execute_layer_test(CopyAndSin(copy), inputs) - - # LLM exports turn dim order off, which makes a clone aten.clone. - self.execute_layer_test( - CopyAndSin(lambda x: x.clone()), - inputs, - compile_config=EdgeCompileConfig(_skip_dim_order=True), - ) + program = self.execute_layer_test( + CopyAndSin(copy), inputs, compile_config=config + ) + # On the CPU, preprocess would also give right values, so check + # where the copy runs. + operators = program.executorch_program.execution_plan[0].operators + self.assertIn(operator, [op.name for op in operators]) def test_copy_of_input_used_inside(self): class ChannelsLastMean(torch.nn.Module): @@ -178,9 +185,9 @@ def forward(self, x): "int64": lambda n: torch.randint(-100, 100, (n, 3)), "bool": lambda n: torch.randn(n, 3) > 0, } - dynamic_shapes = ({0: Dim("rows", min=1, max=4)},) - for name, make in inputs.items(): - with self.subTest(dtype=name): + for (name, make), dynamic in itertools.product(inputs.items(), (False, True)): + dynamic_shapes = ({0: Dim("rows", min=1, max=4)},) if dynamic else None + with self.subTest(dtype=name, dynamic=dynamic): x = make(4) # Without a partitioner the copy cannot run on the CPU, so # OpenVINO has to make it. @@ -201,7 +208,7 @@ def forward(self, x): loaded = Runtime.get().load_program(pte_path) method = loaded.load_method("forward") assert method is not None - for value in (x, make(2)): + for value in (x, make(2 if dynamic else 4)): copy, flat = method.execute((value,)) torch.testing.assert_close(copy, value) torch.testing.assert_close(flat, value.reshape(-1)) From 8b21f66d98be273fcb103a333585ef1442c98465 Mon Sep 17 00:00:00 2001 From: Anthony Shoumikhin Date: Tue, 3 Nov 2026 12:24:56 -0800 Subject: [PATCH 08/13] OpenVINO: share the load-and-run block of the tests Two copy tests repeated the block from execute_layer_test that writes the program to a file, loads it, runs inputs on one method and releases the file for Windows. A later fix to one copy, like the Windows one, was easy to miss in the others. The block is now BaseOpenvinoOpTest.execute_program, and all three places call it. --- .../tests/ops/base_openvino_op_test.py | 40 +++++++++++-------- backends/openvino/tests/ops/test_copy.py | 31 +++----------- 2 files changed, 30 insertions(+), 41 deletions(-) diff --git a/backends/openvino/tests/ops/base_openvino_op_test.py b/backends/openvino/tests/ops/base_openvino_op_test.py index c7a00bdcbf6..1be9cb38fe3 100644 --- a/backends/openvino/tests/ops/base_openvino_op_test.py +++ b/backends/openvino/tests/ops/base_openvino_op_test.py @@ -1,7 +1,7 @@ import os import tempfile import unittest -from typing import Optional +from typing import Any, Optional, Sequence import executorch import torch @@ -10,6 +10,7 @@ from executorch.exir import ( EdgeCompileConfig, EdgeProgramManager, + ExecutorchProgramManager, to_edge_transform_and_lower, ) from executorch.exir.backend.backend_details import CompileSpec @@ -24,6 +25,27 @@ class BaseOpenvinoOpTest(unittest.TestCase): atol = 1e-3 rtol = 1e-3 + @staticmethod + def execute_program( + exec_prog: ExecutorchProgramManager, + all_inputs: Sequence[Sequence[torch.Tensor]], + ) -> list[Sequence[Any]]: + """Run each set of inputs, in order, on one loaded method, as an app + would.""" + # Load from a file: unlike a buffer, it really frees the delegate data + # when the backend frees it. + with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmp_dir: + pte_path = os.path.join(tmp_dir, "model.pte") + with open(pte_path, "wb") as f: + exec_prog.write_to_file(f) + program = Runtime.get().load_program(pte_path) + method = program.load_method("forward") + assert method is not None + all_outputs = [method.execute(inputs) for inputs in all_inputs] + # Windows cannot delete a mapped file, so release it first. + del method, program + return all_outputs + def execute_layer_test( self, module: torch.nn.Module, @@ -68,22 +90,8 @@ def execute_layer_test( # Execute the model and compare the outputs with the reference outputs if assert_output_equal: - # Later inputs run on the same loaded method, as an app would. all_inputs = (sample_inputs, *more_inputs) - - # Load from a file: unlike a buffer, it really frees the delegate - # data when the backend frees it. - with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmp_dir: - pte_path = os.path.join(tmp_dir, "model.pte") - with open(pte_path, "wb") as f: - exec_prog.write_to_file(f) - program = Runtime.get().load_program(pte_path) - method = program.load_method("forward") - assert method is not None - all_outputs = [method.execute(inputs) for inputs in all_inputs] - # Windows cannot delete a mapped file, so release it first. - del method, program - + all_outputs = self.execute_program(exec_prog, all_inputs) for inputs, outputs in zip(all_inputs, all_outputs): # Execute the module in eager mode to calculate the reference outputs ref_output = module(*inputs) diff --git a/backends/openvino/tests/ops/test_copy.py b/backends/openvino/tests/ops/test_copy.py index ab4d29fc719..6441d28de93 100644 --- a/backends/openvino/tests/ops/test_copy.py +++ b/backends/openvino/tests/ops/test_copy.py @@ -1,6 +1,4 @@ import itertools -import os -import tempfile import torch from executorch.backends.openvino.preprocess import OpenvinoBackend @@ -10,7 +8,6 @@ from executorch.exir import EdgeCompileConfig, to_edge from executorch.exir.backend.backend_api import to_backend from executorch.exir.backend.backend_details import CompileSpec -from executorch.runtime import Runtime from torch.export import Dim, export @@ -152,16 +149,7 @@ def forward(self, x): # The program does not keep the starting value of a mutated buffer, so # check each call against the one before it. x1, x2 = torch.randn(2, 3, 4, 5), torch.randn(2, 3, 4, 5) - with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmp_dir: - pte_path = os.path.join(tmp_dir, "model.pte") - with open(pte_path, "wb") as f: - program.write_to_file(f) - loaded = Runtime.get().load_program(pte_path) - method = loaded.load_method("forward") - assert method is not None - first = method.execute((x1,)) - second = method.execute((x2,)) - del method, loaded + first, second = self.execute_program(program, ((x1,), (x2,))) tolerance = {"atol": self.atol, "rtol": self.rtol} torch.testing.assert_close(second[0], first[0] + x1, **tolerance) for copy, sine in (first, second): @@ -201,15 +189,8 @@ def forward(self, x): program = to_edge( export(Lowered(lowered), (x,), dynamic_shapes=dynamic_shapes) ).to_executorch() - with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmp: - pte_path = os.path.join(tmp, "model.pte") - with open(pte_path, "wb") as f: - program.write_to_file(f) - loaded = Runtime.get().load_program(pte_path) - method = loaded.load_method("forward") - assert method is not None - for value in (x, make(2 if dynamic else 4)): - copy, flat = method.execute((value,)) - torch.testing.assert_close(copy, value) - torch.testing.assert_close(flat, value.reshape(-1)) - del method, loaded + values = (x, make(2 if dynamic else 4)) + outputs = self.execute_program(program, [(v,) for v in values]) + for value, (copy, flat) in zip(values, outputs): + torch.testing.assert_close(copy, value) + torch.testing.assert_close(flat, value.reshape(-1)) From 4ce996cf6d5257973279725b0d287931193b5d40 Mon Sep 17 00:00:00 2001 From: Anthony Shoumikhin Date: Tue, 3 Nov 2026 12:24:57 -0800 Subject: [PATCH 09/13] OpenVINO: test a copy of a constant with dim order off With dim order off, `buffer.clone(memory_format=torch.preserve_format)` stays aten.clone. Run on the CPU, as before this change, it failed in the portable clone kernel, which accepts only contiguous_format. Now it stays in the partition. test_copy_of_constant_as_output covers it, and the case fails on main with the kernel's error. The whole program test now skips devices other than the CPU, since preprocess adds the copy only there, so elsewhere it would not test the fix. And when the CPU also reads the constant, the test checks that atan runs there before it checks the clone, so a release that delegates atan fails with that reason. --- backends/openvino/tests/ops/test_copy.py | 29 +++++++++++++++++++----- 1 file changed, 23 insertions(+), 6 deletions(-) diff --git a/backends/openvino/tests/ops/test_copy.py b/backends/openvino/tests/ops/test_copy.py index 6441d28de93..647fbd7fec5 100644 --- a/backends/openvino/tests/ops/test_copy.py +++ b/backends/openvino/tests/ops/test_copy.py @@ -57,8 +57,9 @@ def forward(self, x): def test_copy_of_constant_as_output(self): class CopyOfConstant(torch.nn.Module): - def __init__(self, kind): + def __init__(self, kind, memory_format=None): super().__init__() + self.memory_format = memory_format value = torch.randn(2, 3, 4, 5) if kind == "parameter": self.constant = torch.nn.Parameter(value) @@ -68,14 +69,27 @@ def __init__(self, kind): self.constant = value def forward(self, x): - return self.constant.clone(), x + self.constant + copy = self.constant.clone(memory_format=self.memory_format) + return copy, x + self.constant - for kind in ("parameter", "buffer", "lifted tensor"): - with self.subTest(kind=kind): + cases = { + "parameter": (CopyOfConstant("parameter"), None), + "buffer": (CopyOfConstant("buffer"), None), + "lifted tensor": (CopyOfConstant("lifted tensor"), None), + # With dim order off this stays aten.clone, and the portable clone + # kernel rejects preserve_format. + "buffer, dim order off": ( + CopyOfConstant("buffer", torch.preserve_format), + EdgeCompileConfig(_skip_dim_order=True), + ), + } + for name, (module, config) in cases.items(): + with self.subTest(case=name): program = self.execute_layer_test( - CopyOfConstant(kind), + module, (torch.randn(2, 3, 4, 5),), more_inputs=((torch.randn(2, 3, 4, 5),),), + compile_config=config, ) # OpenVINO gets a constant as its own data, not as an input, so # the copy stays. Run outside, it would store the constant twice. @@ -121,9 +135,9 @@ def copy_of_copy_and_sin(buffer, x): ) operators = program.executorch_program.execution_plan[0].operators names = [op.name for op in operators] - self.assertIn("dim_order_ops::_clone_dim_order", names) if name == "also read outside": self.assertIn("aten::atan", names) + self.assertIn("dim_order_ops::_clone_dim_order", names) def test_copy_of_mutated_buffer_as_output(self): class CopyOfMutatedBuffer(torch.nn.Module): @@ -156,6 +170,9 @@ def forward(self, x): torch.testing.assert_close(sine, torch.sin(copy), **tolerance) def test_whole_program_returning_copy_of_input(self): + if not self.device.startswith("CPU"): + self.skipTest("preprocess adds the copy only for the CPU") + class CopyAndReshape(torch.nn.Module): def forward(self, x): return x.clone(), x.reshape(-1) From 336b6d987da08989206475eb84a4b373a6a0fc23 Mon Sep 17 00:00:00 2001 From: Anthony Shoumikhin Date: Tue, 3 Nov 2026 18:16:48 -0800 Subject: [PATCH 10/13] OpenVINO: run copies of a constant on the CPU when a partition returns it twice When one partition returned 2 copies of the same constant, both stayed in the partition. OpenVINO's PyTorch frontend then cannot convert 2 outputs of one constant, and export failed with "Tensor name is not a number". For example: return self.b.clone(), self.b.clone(), x + self.b A copy of a constant now stays only when it is the only partition output that returns that constant, counting copies of copies. Otherwise the copies run on the CPU, as on main. The test for copies that run outside covers a copy and a copy of it returned together. --- backends/openvino/partitioner.py | 51 +++++++++++++++--------- backends/openvino/tests/ops/test_copy.py | 6 +++ 2 files changed, 38 insertions(+), 19 deletions(-) diff --git a/backends/openvino/partitioner.py b/backends/openvino/partitioner.py index 2796104e98d..d4de013afc8 100644 --- a/backends/openvino/partitioner.py +++ b/backends/openvino/partitioner.py @@ -6,6 +6,7 @@ # mypy: disable-error-code=import-not-found +from collections import Counter from typing import Callable, final, List, Optional, Tuple import torch @@ -81,8 +82,8 @@ def _constants(program: ExportedProgram) -> set[torch.fx.Node]: def _remove_copies_of_inputs( partition: Partition, constants: set[torch.fx.Node] ) -> None: - """Leave out of the partition each copy that would return one of the - partition's inputs, unchanged, as one of its outputs. + """Leave out of the partition each copy that would return, unchanged, one + of the partition's inputs or a constant OpenVINO cannot return. OpenVINO would return the input itself, and when the runtime binds the output buffers after the inputs, its CPU plugin does not write such an @@ -90,32 +91,44 @@ def _remove_copies_of_inputs( Copies used only inside the partition stay, because OpenVINO reads every input in the default dim order, which a copy left out could have changed. A copy of a constant that only this partition reads stays too, since the - constant is not a partition input. Such a copy still leaves when it changes - the dim order, since OpenVINO writes every output in the default one, or - has a 64-bit type, since the CPU plugin computes those in 32 bits. Like any + constant is not a partition input. Such a copy still leaves when it is not + in the default dim order, since OpenVINO writes every output in the default + one, when it has a 64-bit type, since the CPU plugin computes those in 32 + bits, or when another output also returns that constant, since OpenVINO's + PyTorch frontend cannot convert two outputs of one constant. Like any input, a copy left out that is not in the default dim order is still read wrong by the ops that use it inside. """ - def copies_input(node: torch.fx.Node) -> bool: - copy = node + def copied(node: torch.fx.Node) -> Optional[torch.fx.Node]: while (source := _copy_source(node)) is not None: if source not in partition.nodes: - return not ( - source in constants - and all(user in partition.nodes for user in source.users) - and copy.meta["val"].is_contiguous() - and copy.meta["val"].dtype.itemsize < 8 - ) + return source node = source - return False + return None # Taking out a copy can make another copy leave. - while leaving := [ - node - for node in partition.nodes - if copies_input(node) and any(u not in partition.nodes for u in node.users) - ]: + while True: + outputs = { + node: copied(node) + for node in partition.nodes + if any(user not in partition.nodes for user in node.users) + } + returns = Counter(outputs.values()) + leaving = [ + node + for node, source in outputs.items() + if source is not None + and not ( + source in constants + and returns[source] == 1 + and all(user in partition.nodes for user in source.users) + and node.meta["val"].is_contiguous() + and node.meta["val"].dtype.itemsize < 8 + ) + ] + if not leaving: + return for node in leaving: # The copies it reads leave with it. A copy of a constant left inside # could feed it while the partition also reads it, a cycle. diff --git a/backends/openvino/tests/ops/test_copy.py b/backends/openvino/tests/ops/test_copy.py index 647fbd7fec5..eeb6e50b638 100644 --- a/backends/openvino/tests/ops/test_copy.py +++ b/backends/openvino/tests/ops/test_copy.py @@ -110,6 +110,10 @@ def copy_of_copy_and_sin(buffer, x): copy = buffer.clone().clone(memory_format=torch.channels_last) return copy, torch.sin(copy) + def copy_and_copy_of_it(buffer, x): + copy = buffer.clone() + return copy, copy.clone(), buffer + x + floats = torch.randn(2, 3, 4, 5) cases = { # OpenVINO writes every output in the default dim order. @@ -124,6 +128,8 @@ def copy_of_copy_and_sin(buffer, x): lambda b, x: (b.clone(), x + 1), torch.tensor([2**40, -(2**40)]), ), + # OpenVINO's PyTorch frontend cannot return one constant twice. + "two copies": (copy_and_copy_of_it, floats), # The partitioner does not take atan, so the CPU stores the buffer # anyway, and a copy made there keeps it stored once. "also read outside": (lambda b, x: (b.clone(), torch.atan(b) + x), floats), From b459290c26ff22c8aaea733c7ad51b709f9baee5 Mon Sep 17 00:00:00 2001 From: Anthony Shoumikhin Date: Tue, 3 Nov 2026 18:16:48 -0800 Subject: [PATCH 11/13] OpenVINO: keep a copy of a constant only in a type the runtime binds A uint16 copy of a constant stayed in the partition, but the runtime cannot bind a uint16 output, so the program failed at run time with "unsupported scalar type 27". A copy of a constant now stays only in a type the runtime binds, leaving out the 64-bit types the CPU plugin computes in 32 bits: bool, uint8, int8, int16, int32, float16, bfloat16 and float32. This list replaces the check for 64-bit types. The test for copies that run outside covers uint16. --- backends/openvino/partitioner.py | 25 ++++++++++++++++++------ backends/openvino/tests/ops/test_copy.py | 5 +++++ 2 files changed, 24 insertions(+), 6 deletions(-) diff --git a/backends/openvino/partitioner.py b/backends/openvino/partitioner.py index d4de013afc8..b201779146a 100644 --- a/backends/openvino/partitioner.py +++ b/backends/openvino/partitioner.py @@ -33,6 +33,19 @@ "dim_order_ops._clone_dim_order.default", } +# The runtime binds outputs of these types, and the CPU plugin computes 64-bit +# types in 32 bits, so only copies of these types can return a constant. +_CONSTANT_COPY_DTYPES = { + torch.bool, + torch.uint8, + torch.int8, + torch.int16, + torch.int32, + torch.float16, + torch.bfloat16, + torch.float32, +} + class PatternNode: op_types: dict[str, Optional[list]] = {} @@ -93,11 +106,11 @@ def _remove_copies_of_inputs( A copy of a constant that only this partition reads stays too, since the constant is not a partition input. Such a copy still leaves when it is not in the default dim order, since OpenVINO writes every output in the default - one, when it has a 64-bit type, since the CPU plugin computes those in 32 - bits, or when another output also returns that constant, since OpenVINO's - PyTorch frontend cannot convert two outputs of one constant. Like any - input, a copy left out that is not in the default dim order is still read - wrong by the ops that use it inside. + one, when its type is not in _CONSTANT_COPY_DTYPES, or when another output + also returns that constant, since OpenVINO's PyTorch frontend cannot + convert two outputs of one constant. Like any input, a copy left out that is + not in the default dim order is still read wrong by the ops that use it + inside. """ def copied(node: torch.fx.Node) -> Optional[torch.fx.Node]: @@ -124,7 +137,7 @@ def copied(node: torch.fx.Node) -> Optional[torch.fx.Node]: and returns[source] == 1 and all(user in partition.nodes for user in source.users) and node.meta["val"].is_contiguous() - and node.meta["val"].dtype.itemsize < 8 + and node.meta["val"].dtype in _CONSTANT_COPY_DTYPES ) ] if not leaving: diff --git a/backends/openvino/tests/ops/test_copy.py b/backends/openvino/tests/ops/test_copy.py index eeb6e50b638..659384e7416 100644 --- a/backends/openvino/tests/ops/test_copy.py +++ b/backends/openvino/tests/ops/test_copy.py @@ -128,6 +128,11 @@ def copy_and_copy_of_it(buffer, x): lambda b, x: (b.clone(), x + 1), torch.tensor([2**40, -(2**40)]), ), + # The runtime cannot bind a uint16 output. + "uint16": ( + lambda b, x: (b.clone(), x + 1), + torch.tensor([0, 40000, 65535], dtype=torch.uint16), + ), # OpenVINO's PyTorch frontend cannot return one constant twice. "two copies": (copy_and_copy_of_it, floats), # The partitioner does not take atan, so the CPU stores the buffer From a722e943ee37414d6c702b0469fc0209cf215184 Mon Sep 17 00:00:00 2001 From: Anthony Shoumikhin Date: Tue, 3 Nov 2026 19:23:04 -0800 Subject: [PATCH 12/13] OpenVINO: keep float copies of constants only in float32 precision With inference_precision set to f16 or bf16, OpenVINO can return a float copy of a constant rounded: on an ARM CPU at f16, 3.14159265 came back as 3.140625. The README suggests f16 on ARM CPUs. A float copy of a constant now stays in the partition only when OpenVINO computes in float32, which is the default on the CPU. Otherwise it runs on the CPU, as on main. Integer and bool copies are not affected. On other devices the backend sets no default precision, so unless f32 is set, float copies of constants run on the CPU there too, and test_copy_of_constant_as_output skips them. The partitioner and preprocess share the rule that picks the precision. At f16, the tests check that a float copy runs on the CPU and an int32 copy stays. The tests also check a copy of each type that stays, use integers above 2048, where float16 cannot hold every value, and compare each kept copy exactly, since the default tolerance accepts a float16 rounding. A program lowered whole, with no partitioner, has no CPU side to run such a copy, so preprocess still copies a returned input with Identity. At f16 on an ARM CPU, OpenVINO rounds that copy of a float input like any other float output, and the docstring says so. --- backends/openvino/partitioner.py | 34 +++++++++++---- backends/openvino/preprocess.py | 15 +++++-- backends/openvino/tests/ops/test_copy.py | 54 +++++++++++++++++++++--- 3 files changed, 84 insertions(+), 19 deletions(-) diff --git a/backends/openvino/partitioner.py b/backends/openvino/partitioner.py index b201779146a..b78a114b822 100644 --- a/backends/openvino/partitioner.py +++ b/backends/openvino/partitioner.py @@ -10,7 +10,10 @@ from typing import Callable, final, List, Optional, Tuple import torch -from executorch.backends.openvino.preprocess import OpenvinoBackend +from executorch.backends.openvino.preprocess import ( + _inference_precision, + OpenvinoBackend, +) from executorch.exir.backend.backend_details import CompileSpec from executorch.exir.backend.partitioner import ( DelegationSpec, @@ -93,7 +96,7 @@ def _constants(program: ExportedProgram) -> set[torch.fx.Node]: def _remove_copies_of_inputs( - partition: Partition, constants: set[torch.fx.Node] + partition: Partition, constants: set[torch.fx.Node], dtypes: set[torch.dtype] ) -> None: """Leave out of the partition each copy that would return, unchanged, one of the partition's inputs or a constant OpenVINO cannot return. @@ -106,11 +109,11 @@ def _remove_copies_of_inputs( A copy of a constant that only this partition reads stays too, since the constant is not a partition input. Such a copy still leaves when it is not in the default dim order, since OpenVINO writes every output in the default - one, when its type is not in _CONSTANT_COPY_DTYPES, or when another output - also returns that constant, since OpenVINO's PyTorch frontend cannot - convert two outputs of one constant. Like any input, a copy left out that is - not in the default dim order is still read wrong by the ops that use it - inside. + one, when its type is not in dtypes, the types OpenVINO can return, or when + another output also returns that constant, since OpenVINO's PyTorch + frontend cannot convert two outputs of one constant. Like any input, a copy + left out that is not in the default dim order is still read wrong by the + ops that use it inside. """ def copied(node: torch.fx.Node) -> Optional[torch.fx.Node]: @@ -137,7 +140,7 @@ def copied(node: torch.fx.Node) -> Optional[torch.fx.Node]: and returns[source] == 1 and all(user in partition.nodes for user in source.users) and node.meta["val"].is_contiguous() - and node.meta["val"].dtype in _CONSTANT_COPY_DTYPES + and node.meta["val"].dtype in dtypes ) ] if not leaving: @@ -361,8 +364,21 @@ def partition(self, exported_program: ExportedProgram) -> PartitionResult: partition_tags = {} constants = _constants(exported_program) + options = { + spec.key: spec.value.decode() for spec in self.delegation_spec.compile_specs + } + precision = _inference_precision( + options.get("device", "CPU"), options.get("inference_precision") + ) + # OpenVINO can round a float copy in any precision but f32, as an ARM + # CPU does in f16. + dtypes = { + dtype + for dtype in _CONSTANT_COPY_DTYPES + if precision == "f32" or not dtype.is_floating_point + } for partition in partition_list: - _remove_copies_of_inputs(partition, constants) + _remove_copies_of_inputs(partition, constants, dtypes) for node in partition.nodes: tag = f"tag{partition.id}" node.meta["delegation_tag"] = tag diff --git a/backends/openvino/preprocess.py b/backends/openvino/preprocess.py index df5f237368e..1a17d120d3c 100644 --- a/backends/openvino/preprocess.py +++ b/backends/openvino/preprocess.py @@ -6,7 +6,7 @@ # mypy: disable-error-code=import-not-found -from typing import final, List +from typing import final, List, Optional import torch from executorch.backends.openvino._passes import ( @@ -63,6 +63,11 @@ def _match_partition_inputs( ) +def _inference_precision(device: str, requested: Optional[str]) -> Optional[str]: + """The INFERENCE_PRECISION_HINT to pass, or None for the device's default.""" + return requested or ("f32" if device.startswith("CPU") else None) + + def _copy_returned_inputs(model, device: str) -> None: """Make the converted model compute each output that is one of its inputs. @@ -70,8 +75,10 @@ def _copy_returned_inputs(model, device: str) -> None: is the input itself. The CPU plugin does not write it, and can overwrite the input with the output buffer's old contents. OpenVINO keeps an Identity between an input and an output, and the CPU plugin runs it as a - copy, while it can remove a Convert to the same type. Other devices are - left as they are, since neither the problem nor Identity was checked there. + copy, while it can remove a Convert to the same type. On an ARM CPU at + f16, it rounds this copy of a float input, like its other float outputs. + Other devices are left as they are, since neither the problem nor Identity + was checked there. """ if not device.startswith("CPU"): return @@ -169,7 +176,7 @@ def preprocess( ppp.input(i).preprocess().custom(lambda x: opset13.squeeze(x, 0).output(0)) # On ARM CPUs OpenVINO fixes the precision here, not when the runtime # imports the model, so this must follow the same rule as the runtime. - precision = requested_precision or ("f32" if device.startswith("CPU") else None) + precision = _inference_precision(device, requested_precision) config = {"INFERENCE_PRECISION_HINT": precision} if precision else {} compiled = compile_model(ppp.build(), device, config) model_bytes = compiled.export_model() diff --git a/backends/openvino/tests/ops/test_copy.py b/backends/openvino/tests/ops/test_copy.py index 659384e7416..586a7ec3422 100644 --- a/backends/openvino/tests/ops/test_copy.py +++ b/backends/openvino/tests/ops/test_copy.py @@ -56,11 +56,24 @@ def forward(self, x): self.execute_layer_test(ChannelsLastMean(), (torch.randn(2, 3, 4, 5),)) def test_copy_of_constant_as_output(self): + if not self.device.startswith("CPU"): + self.skipTest("float copies stay only in float32, the CPU default") + class CopyOfConstant(torch.nn.Module): - def __init__(self, kind, memory_format=None): + def __init__(self, kind, memory_format=None, dtype=torch.float32): super().__init__() self.memory_format = memory_format - value = torch.randn(2, 3, 4, 5) + if dtype.is_floating_point: + value = torch.randn(2, 3, 4, 5).to(dtype) + elif dtype == torch.bool: + value = torch.randn(2, 3, 4, 5) > 0 + else: + # float16 holds every integer only up to 2048, so these + # show a copy rounded through it, and x + constant stays + # in its range. + info = torch.iinfo(dtype) + low, high = max(info.min, -(2**15)), min(info.max, 2**15) + value = torch.randint(low, high, (2, 3, 4, 5), dtype=dtype) if kind == "parameter": self.constant = torch.nn.Parameter(value) elif kind == "buffer": @@ -82,19 +95,40 @@ def forward(self, x): CopyOfConstant("buffer", torch.preserve_format), EdgeCompileConfig(_skip_dim_order=True), ), + # The precision changes only how OpenVINO computes floats. + "int32 buffer, f16 precision": ( + CopyOfConstant("buffer", dtype=torch.int32), + None, + CompileSpec("inference_precision", b"f16"), + ), } - for name, (module, config) in cases.items(): + for dtype in ( + torch.bool, + torch.uint8, + torch.int8, + torch.int16, + torch.int32, + torch.float16, + torch.bfloat16, + ): + cases[f"{dtype} buffer"] = (CopyOfConstant("buffer", dtype=dtype), None) + for name, (module, config, *compile_specs) in cases.items(): with self.subTest(case=name): + inputs = (torch.randn(2, 3, 4, 5),) program = self.execute_layer_test( module, - (torch.randn(2, 3, 4, 5),), + inputs, more_inputs=((torch.randn(2, 3, 4, 5),),), + compile_specs=tuple(compile_specs), compile_config=config, ) # OpenVINO gets a constant as its own data, not as an input, so # the copy stays. Run outside, it would store the constant twice. plan = program.executorch_program.execution_plan[0] self.assertEqual(plan.operators, []) + # The tolerance above lets a copy rounded through float16 pass. + copy = self.execute_program(program, [inputs])[0][0] + torch.testing.assert_close(copy, module.constant, rtol=0, atol=0) def test_copy_of_constant_that_runs_outside(self): class Buffered(torch.nn.Module): @@ -135,14 +169,22 @@ def copy_and_copy_of_it(buffer, x): ), # OpenVINO's PyTorch frontend cannot return one constant twice. "two copies": (copy_and_copy_of_it, floats), + # In a lower precision OpenVINO can round a float copy. + "float16 precision": ( + lambda b, x: (b.clone(), x + 1), + floats, + CompileSpec("inference_precision", b"f16"), + ), # The partitioner does not take atan, so the CPU stores the buffer # anyway, and a copy made there keeps it stored once. "also read outside": (lambda b, x: (b.clone(), torch.atan(b) + x), floats), } - for name, (body, buffer) in cases.items(): + for name, (body, buffer, *compile_specs) in cases.items(): with self.subTest(case=name): program = self.execute_layer_test( - Buffered(body, buffer), (torch.randn(2, 3, 4, 5),) + Buffered(body, buffer), + (torch.randn(2, 3, 4, 5),), + compile_specs=tuple(compile_specs), ) operators = program.executorch_program.execution_plan[0].operators names = [op.name for op in operators] From 030831cca53775049c2ead762bcf697a4ffd03ff Mon Sep 17 00:00:00 2001 From: Anthony Shoumikhin Date: Wed, 4 Nov 2026 20:47:07 -0800 Subject: [PATCH 13/13] OpenVINO: say that an f16 program can run in bfloat16 on an x86 CPU The README and the OpenVINO guide said that a CPU without float16 support computes an f16 program in float32. On an x86 CPU that has bfloat16 but not float16, such as an Intel Xeon with AVX512_BF16, OpenVINO 2025.4 and 2026.4 report bfloat16 as the precision they use for an f16 request, and a matrix multiply comes back with bfloat16 error. The docs now say that such a CPU computes in bfloat16. --- backends/openvino/README.md | 3 ++- docs/source/build-run-openvino.md | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/backends/openvino/README.md b/backends/openvino/README.md index 76f834a59f7..743a40ba314 100644 --- a/backends/openvino/README.md +++ b/backends/openvino/README.md @@ -70,7 +70,8 @@ float16, as measured on an Orin Nano. The value is OpenVINO's `INFERENCE_PRECISION_HINT`: `f32`, `f16` or `bf16`. An empty value means the default, and any other value fails at lowering. A CPU -without float16 support computes an `f16` program in float32. OpenVINO has no +without float16 support computes an `f16` program in bfloat16 if it has bfloat16 +units, as some x86 CPUs do, and in float32 otherwise. OpenVINO has no bfloat16 on ARM CPUs, so there `bf16` keeps the ARM default, float16 where the CPU supports it. Which x86 CPUs compute in bfloat16 is up to OpenVINO; see its [Precision Control](https://docs.openvino.ai/2026/openvino-workflow/running-inference/optimize-inference/precision-control.html) diff --git a/docs/source/build-run-openvino.md b/docs/source/build-run-openvino.md index a6303158b49..1aecb1b48c0 100644 --- a/docs/source/build-run-openvino.md +++ b/docs/source/build-run-openvino.md @@ -127,7 +127,8 @@ float16, as measured on an Orin Nano. The value is OpenVINO's `INFERENCE_PRECISION_HINT`: `f32`, `f16` or `bf16`. An empty value means the default, and any other value fails at lowering. A CPU -without float16 support computes an `f16` program in float32. OpenVINO has no +without float16 support computes an `f16` program in bfloat16 if it has bfloat16 +units, as some x86 CPUs do, and in float32 otherwise. OpenVINO has no bfloat16 on ARM CPUs, so there `bf16` keeps the ARM default, float16 where the CPU supports it. Which x86 CPUs compute in bfloat16 is up to OpenVINO; see its [Precision Control](https://docs.openvino.ai/2026/openvino-workflow/running-inference/optimize-inference/precision-control.html)