OpenVINO: keep copies of constants in the partition, and copy inputs a whole program returns

README-wheel.md

@@ -37,8 +37,11 @@ to run ExecuTorch `.pte` files, with some restrictions:
 * \[Linux and Windows] [OpenVINO](docs/source/build-run-openvino.md) backend is also linked into the
   prebuilt module. OpenVINO requires the runtime to be installed separately:
   `pip install executorch[openvino]`. For now the extra installs no runtime on Python 3.15,
-  or on Linux aarch64 with Python 3.14. OpenVINO publishes no build for Python 3.15, and its
-  aarch64 builds for Python 3.14 need glibc 2.35 or newer.
+  where OpenVINO publishes no build. On Linux aarch64, OpenVINO needs glibc 2.35 or newer,
+  as on Ubuntu 22.04 or Raspberry Pi OS Bookworm and later. On an older aarch64 system, such as
+  RHEL 9 or Amazon Linux 2023, `pip install executorch[openvino]` does not fail but installs an
+  older ExecuTorch release; see the [OpenVINO docs](docs/source/build-run-openvino.md) for what
+  to do there.
 
 On Linux, macOS and Windows the runtime, kernels and backends also ship as shared libraries
 with headers and a CMake package, so a C++ application can link them without building

backends/openvino/README.md

@@ -20,7 +20,7 @@ On Linux and Windows, the OpenVINO backend is included in the ExecuTorch pip whe
 pip install executorch[openvino]
 ```
 
-For now the extra installs no runtime on Python 3.15, or on Linux aarch64 with Python 3.14. OpenVINO publishes no build for Python 3.15, and its aarch64 builds for Python 3.14 need glibc 2.35 or newer.
+For now the extra installs no runtime on Python 3.15, where OpenVINO publishes no build. On Linux aarch64, OpenVINO needs glibc 2.35 or newer, as on Ubuntu 22.04 or Raspberry Pi OS Bookworm and later. On an older aarch64 system, `pip install executorch[openvino]` does not fail: pip finds no OpenVINO 2025.4 or newer for that system, so it installs an older ExecuTorch release instead. On such a system, install `executorch` without the extra, then build OpenVINO from source as described below.
 
 The backend automatically discovers the OpenVINO C library from the pip-installed package, so no library search path setup is needed.
 
@@ -70,7 +70,8 @@ float16, as measured on an Orin Nano.
 
 The value is OpenVINO's `INFERENCE_PRECISION_HINT`: `f32`, `f16` or `bf16`. An
 empty value means the default, and any other value fails at lowering. A CPU
-without float16 support computes an `f16` program in float32. OpenVINO has no
+without float16 support computes an `f16` program in bfloat16 if it has bfloat16
+units, as some x86 CPUs do, and in float32 otherwise. OpenVINO has no
 bfloat16 on ARM CPUs, so there `bf16` keeps the ARM default, float16 where the
 CPU supports it. Which x86 CPUs compute in bfloat16 is up to OpenVINO; see its
 [Precision Control](https://docs.openvino.ai/2026/openvino-workflow/running-inference/optimize-inference/precision-control.html)
@@ -139,7 +140,7 @@ executorch
 
 ### Prerequisites
 
-Before you begin, ensure you have openvino installed and configured on your system.
+Before you begin, ensure you have OpenVINO 2025.4 or newer installed and configured on your system.
 
 ### Use OpenVINO from Release Packages
 
@@ -157,7 +158,7 @@ Before you begin, ensure you have openvino installed and configured on your syst
 
 ```bash
 git clone https://github.com/openvinotoolkit/openvino.git
-cd openvino
+cd openvino && git checkout releases/2026/4
 git submodule update --init --recursive
 sudo ./install_build_dependencies.sh
 mkdir build && cd build

backends/openvino/partitioner.py

@@ -6,10 +6,14 @@
 
 # mypy: disable-error-code=import-not-found
 
+from collections import Counter
 from typing import Callable, final, List, Optional, Tuple
 
 import torch
-from executorch.backends.openvino.preprocess import OpenvinoBackend
+from executorch.backends.openvino.preprocess import (
+    _inference_precision,
+    OpenvinoBackend,
+)
 from executorch.exir.backend.backend_details import CompileSpec
 from executorch.exir.backend.partitioner import (
     DelegationSpec,
@@ -32,6 +36,19 @@
     "dim_order_ops._clone_dim_order.default",
 }
 
+# The runtime binds outputs of these types, and the CPU plugin computes 64-bit
+# types in 32 bits, so only copies of these types can return a constant.
+_CONSTANT_COPY_DTYPES = {
+    torch.bool,
+    torch.uint8,
+    torch.int8,
+    torch.int16,
+    torch.int32,
+    torch.float16,
+    torch.bfloat16,
+    torch.float32,
+}
+
 
 class PatternNode:
     op_types: dict[str, Optional[list]] = {}
@@ -56,32 +73,84 @@ def _copy_source(node: torch.fx.Node) -> Optional[torch.fx.Node]:
     return source if isinstance(source, torch.fx.Node) else None
 
 
-def _remove_copies_of_inputs(partition: Partition) -> None:
-    """Leave out of the partition each copy that would return one of the
-    partition's inputs, unchanged, as one of its outputs.
+def _constants(program: ExportedProgram) -> set[torch.fx.Node]:
+    """The placeholders that each partition reading them stores as its own
+    data (see tag_constant_data), so OpenVINO gets them as constants, not as
+    inputs."""
+    signature = program.graph_signature
+    mutated = set(signature.buffers_to_mutate.values())
+    names = {
+        *signature.inputs_to_parameters,
+        *signature.inputs_to_lifted_tensor_constants,
+        *(
+            name
+            for name, buffer in signature.inputs_to_buffers.items()
+            if buffer not in mutated
+        ),
+    }
+    return {
+        node
+        for node in program.graph.nodes
+        if node.op == "placeholder" and node.name in names
+    }
+
+
+def _remove_copies_of_inputs(
+    partition: Partition, constants: set[torch.fx.Node], dtypes: set[torch.dtype]
+) -> None:
+    """Leave out of the partition each copy that would return, unchanged, one
+    of the partition's inputs or a constant OpenVINO cannot return.
 
     OpenVINO would return the input itself, and when the runtime binds the
     output buffers after the inputs, its CPU plugin does not write such an
     output and can overwrite the input with the output buffer's old contents.
     Copies used only inside the partition stay, because OpenVINO reads every
     input in the default dim order, which a copy left out could have changed.
+    A copy of a constant that only this partition reads stays too, since the
+    constant is not a partition input. Such a copy still leaves when it is not
+    in the default dim order, since OpenVINO writes every output in the default
+    one, when its type is not in dtypes, the types OpenVINO can return, or when
+    another output also returns that constant, since OpenVINO's PyTorch
+    frontend cannot convert two outputs of one constant. Like any input, a copy
+    left out that is not in the default dim order is still read wrong by the
+    ops that use it inside.
     """
 
-    def copies_input(node: torch.fx.Node) -> bool:
+    def copied(node: torch.fx.Node) -> Optional[torch.fx.Node]:
         while (source := _copy_source(node)) is not None:
             if source not in partition.nodes:
-                return True
+                return source
             node = source
-        return False
+        return None
 
-    # Taking out a copy can make the copy it reads an output.
-    while leaving := [
-        node
-        for node in partition.nodes
-        if copies_input(node) and any(u not in partition.nodes for u in node.users)
-    ]:
+    # Taking out a copy can make another copy leave.
+    while True:
+        outputs = {
+            node: copied(node)
+            for node in partition.nodes
+            if any(user not in partition.nodes for user in node.users)
+        }
+        returns = Counter(outputs.values())
+        leaving = [
+            node
+            for node, source in outputs.items()
+            if source is not None
+            and not (
+                source in constants
+                and returns[source] == 1
+                and all(user in partition.nodes for user in source.users)
+                and node.meta["val"].is_contiguous()
+                and node.meta["val"].dtype in dtypes
+            )
+        ]
+        if not leaving:
+            return
         for node in leaving:
-            partition.remove_node(node)
+            # The copies it reads leave with it. A copy of a constant left inside
+            # could feed it while the partition also reads it, a cycle.
+            while (inner := _copy_source(node)) is not None and node in partition.nodes:
+                partition.remove_node(node)
+                node = inner
 
 
 class OpenvinoOperatorsSupport(OperatorSupportBase):
@@ -294,8 +363,22 @@ def partition(self, exported_program: ExportedProgram) -> PartitionResult:
         partition_list = partitioner.propose_partitions()
 
         partition_tags = {}
+        constants = _constants(exported_program)
+        options = {
+            spec.key: spec.value.decode() for spec in self.delegation_spec.compile_specs
+        }
+        precision = _inference_precision(
+            options.get("device", "CPU"), options.get("inference_precision")
+        )
+        # OpenVINO can round a float copy in any precision but f32, as an ARM
+        # CPU does in f16.
+        dtypes = {
+            dtype
+            for dtype in _CONSTANT_COPY_DTYPES
+            if precision == "f32" or not dtype.is_floating_point
+        }
         for partition in partition_list:
-            _remove_copies_of_inputs(partition)
+            _remove_copies_of_inputs(partition, constants, dtypes)
             for node in partition.nodes:
                 tag = f"tag{partition.id}"
                 node.meta["delegation_tag"] = tag

backends/openvino/preprocess.py

@@ -6,7 +6,7 @@
 
 # mypy: disable-error-code=import-not-found
 
-from typing import final, List
+from typing import final, List, Optional
 
 import torch
 from executorch.backends.openvino._passes import (
@@ -22,7 +22,7 @@
 )
 from executorch.exir.backend.compile_spec_schema import CompileSpec
 from executorch.exir.passes.memory_format_ops_pass import DimOrderOpsRevertPass
-from openvino import compile_model, opset13  # type: ignore[import-untyped]
+from openvino import compile_model, opset13, opset16  # type: ignore[import-untyped]
 from openvino.frontend import FrontEndManager  # type: ignore[import-untyped]
 from openvino.frontend.pytorch.fx_decoder import (  # type: ignore[import-untyped]
     TorchFXPythonDecoder,
@@ -63,6 +63,31 @@ def _match_partition_inputs(
         )
 
 
+def _inference_precision(device: str, requested: Optional[str]) -> Optional[str]:
+    """The INFERENCE_PRECISION_HINT to pass, or None for the device's default."""
+    return requested or ("f32" if device.startswith("CPU") else None)
+
+
+def _copy_returned_inputs(model, device: str) -> None:
+    """Make the converted model compute each output that is one of its inputs.
+
+    The converter turns copies like aten.clone into nothing, so such an output
+    is the input itself. The CPU plugin does not write it, and can overwrite
+    the input with the output buffer's old contents. OpenVINO keeps an
+    Identity between an input and an output, and the CPU plugin runs it as a
+    copy, while it can remove a Convert to the same type. On an ARM CPU at
+    f16, it rounds this copy of a float input, like its other float outputs.
+    Other devices are left as they are, since neither the problem nor Identity
+    was checked there.
+    """
+    if not device.startswith("CPU"):
+        return
+    for result in model.get_results():
+        source = result.input_value(0)
+        if source.get_node().get_type_name() == "Parameter":
+            result.input(0).replace_source_output(opset16.identity(source).output(0))
+
+
 @final
 class OpenvinoBackend(BackendDetails):
 
@@ -142,6 +167,7 @@ def preprocess(
         frontend = FrontEndManager().load_by_framework("pytorch")
         model = frontend.convert(frontend.load(decoder))
         _match_partition_inputs(model, module, input_shapes, input_types)
+        _copy_returned_inputs(model, device)
         ppp = PrePostProcessor(model)
         # The converter takes an int input, like a SymInt size, as a scalar, but
         # the runtime passes it as an int64 tensor of shape [1].
@@ -150,7 +176,7 @@ def preprocess(
             ppp.input(i).preprocess().custom(lambda x: opset13.squeeze(x, 0).output(0))
         # On ARM CPUs OpenVINO fixes the precision here, not when the runtime
         # imports the model, so this must follow the same rule as the runtime.
-        precision = requested_precision or ("f32" if device.startswith("CPU") else None)
+        precision = _inference_precision(device, requested_precision)
         config = {"INFERENCE_PRECISION_HINT": precision} if precision else {}
         compiled = compile_model(ppp.build(), device, config)
         model_bytes = compiled.export_model()

backends/openvino/runtime/OpenvinoBackend.cpp

@@ -214,7 +214,7 @@ bool OpenvinoBackend::ensure_loaded() const {
           Error,
           "OpenVINO runtime not found (loading '%s' failed: %s). "
           "Ensure '%s' can be found (%s), or install with: "
-          "pip install \"openvino>=2025.1.0,<2027.0.0\"",
+          "pip install \"openvino>=2025.4.0,<2027.0.0\"",
           to_utf8(effective_path).c_str(),
           error.c_str(),
           to_utf8(kDefaultLibName).c_str(),

backends/openvino/tests/ops/base_openvino_op_test.py

@@ -1,7 +1,7 @@
 import os
 import tempfile
 import unittest
-from typing import Optional
+from typing import Any, Optional, Sequence
 
 import executorch
 import torch
@@ -10,6 +10,7 @@
 from executorch.exir import (
     EdgeCompileConfig,
     EdgeProgramManager,
+    ExecutorchProgramManager,
     to_edge_transform_and_lower,
 )
 from executorch.exir.backend.backend_details import CompileSpec
@@ -24,6 +25,27 @@ class BaseOpenvinoOpTest(unittest.TestCase):
     atol = 1e-3
     rtol = 1e-3
 
+    @staticmethod
+    def execute_program(
+        exec_prog: ExecutorchProgramManager,
+        all_inputs: Sequence[Sequence[torch.Tensor]],
+    ) -> list[Sequence[Any]]:
+        """Run each set of inputs, in order, on one loaded method, as an app
+        would."""
+        # Load from a file: unlike a buffer, it really frees the delegate data
+        # when the backend frees it.
+        with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmp_dir:
+            pte_path = os.path.join(tmp_dir, "model.pte")
+            with open(pte_path, "wb") as f:
+                exec_prog.write_to_file(f)
+            program = Runtime.get().load_program(pte_path)
+            method = program.load_method("forward")
+            assert method is not None
+            all_outputs = [method.execute(inputs) for inputs in all_inputs]
+            # Windows cannot delete a mapped file, so release it first.
+            del method, program
+        return all_outputs
+
     def execute_layer_test(
         self,
         module: torch.nn.Module,
@@ -68,22 +90,8 @@ def execute_layer_test(
 
         # Execute the model and compare the outputs with the reference outputs
         if assert_output_equal:
-            # Later inputs run on the same loaded method, as an app would.
             all_inputs = (sample_inputs, *more_inputs)
-
-            # Load from a file: unlike a buffer, it really frees the delegate
-            # data when the backend frees it.
-            with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmp_dir:
-                pte_path = os.path.join(tmp_dir, "model.pte")
-                with open(pte_path, "wb") as f:
-                    exec_prog.write_to_file(f)
-                program = Runtime.get().load_program(pte_path)
-                method = program.load_method("forward")
-                assert method is not None
-                all_outputs = [method.execute(inputs) for inputs in all_inputs]
-                # Windows cannot delete a mapped file, so release it first.
-                del method, program
-
+            all_outputs = self.execute_program(exec_prog, all_inputs)
             for inputs, outputs in zip(all_inputs, all_outputs):
                 # Execute the module in eager mode to calculate the reference outputs
                 ref_output = module(*inputs)
@@ -105,3 +113,5 @@ def execute_layer_test(
                         ),
                         msg=f"ref_output:\n{ref_output[i]}\n\ntest_output:\n{outputs[i]}",
                     )
+
+        return exec_prog