diff --git a/README-wheel.md b/README-wheel.md index 5245e312bcd..efc089dee84 100644 --- a/README-wheel.md +++ b/README-wheel.md @@ -37,8 +37,11 @@ to run ExecuTorch `.pte` files, with some restrictions: * \[Linux and Windows] [OpenVINO](docs/source/build-run-openvino.md) backend is also linked into the prebuilt module. OpenVINO requires the runtime to be installed separately: `pip install executorch[openvino]`. For now the extra installs no runtime on Python 3.15, - or on Linux aarch64 with Python 3.14. OpenVINO publishes no build for Python 3.15, and its - aarch64 builds for Python 3.14 need glibc 2.35 or newer. + where OpenVINO publishes no build. On Linux aarch64, OpenVINO needs glibc 2.35 or newer, + as on Ubuntu 22.04 or Raspberry Pi OS Bookworm and later. On an older aarch64 system, such as + RHEL 9 or Amazon Linux 2023, `pip install executorch[openvino]` does not fail but installs an + older ExecuTorch release; see the [OpenVINO docs](docs/source/build-run-openvino.md) for what + to do there. On Linux, macOS and Windows the runtime, kernels and backends also ship as shared libraries with headers and a CMake package, so a C++ application can link them without building diff --git a/backends/openvino/README.md b/backends/openvino/README.md index 883e1e285f6..743a40ba314 100644 --- a/backends/openvino/README.md +++ b/backends/openvino/README.md @@ -20,7 +20,7 @@ On Linux and Windows, the OpenVINO backend is included in the ExecuTorch pip whe pip install executorch[openvino] ``` -For now the extra installs no runtime on Python 3.15, or on Linux aarch64 with Python 3.14. OpenVINO publishes no build for Python 3.15, and its aarch64 builds for Python 3.14 need glibc 2.35 or newer. +For now the extra installs no runtime on Python 3.15, where OpenVINO publishes no build. On Linux aarch64, OpenVINO needs glibc 2.35 or newer, as on Ubuntu 22.04 or Raspberry Pi OS Bookworm and later. On an older aarch64 system, `pip install executorch[openvino]` does not fail: pip finds no OpenVINO 2025.4 or newer for that system, so it installs an older ExecuTorch release instead. On such a system, install `executorch` without the extra, then build OpenVINO from source as described below. The backend automatically discovers the OpenVINO C library from the pip-installed package, so no library search path setup is needed. @@ -70,7 +70,8 @@ float16, as measured on an Orin Nano. The value is OpenVINO's `INFERENCE_PRECISION_HINT`: `f32`, `f16` or `bf16`. An empty value means the default, and any other value fails at lowering. A CPU -without float16 support computes an `f16` program in float32. OpenVINO has no +without float16 support computes an `f16` program in bfloat16 if it has bfloat16 +units, as some x86 CPUs do, and in float32 otherwise. OpenVINO has no bfloat16 on ARM CPUs, so there `bf16` keeps the ARM default, float16 where the CPU supports it. Which x86 CPUs compute in bfloat16 is up to OpenVINO; see its [Precision Control](https://docs.openvino.ai/2026/openvino-workflow/running-inference/optimize-inference/precision-control.html) @@ -139,7 +140,7 @@ executorch ### Prerequisites -Before you begin, ensure you have openvino installed and configured on your system. +Before you begin, ensure you have OpenVINO 2025.4 or newer installed and configured on your system. ### Use OpenVINO from Release Packages @@ -157,7 +158,7 @@ Before you begin, ensure you have openvino installed and configured on your syst ```bash git clone https://github.com/openvinotoolkit/openvino.git -cd openvino +cd openvino && git checkout releases/2026/4 git submodule update --init --recursive sudo ./install_build_dependencies.sh mkdir build && cd build diff --git a/backends/openvino/partitioner.py b/backends/openvino/partitioner.py index 4d6cfd2f93e..b78a114b822 100644 --- a/backends/openvino/partitioner.py +++ b/backends/openvino/partitioner.py @@ -6,10 +6,14 @@ # mypy: disable-error-code=import-not-found +from collections import Counter from typing import Callable, final, List, Optional, Tuple import torch -from executorch.backends.openvino.preprocess import OpenvinoBackend +from executorch.backends.openvino.preprocess import ( + _inference_precision, + OpenvinoBackend, +) from executorch.exir.backend.backend_details import CompileSpec from executorch.exir.backend.partitioner import ( DelegationSpec, @@ -32,6 +36,19 @@ "dim_order_ops._clone_dim_order.default", } +# The runtime binds outputs of these types, and the CPU plugin computes 64-bit +# types in 32 bits, so only copies of these types can return a constant. +_CONSTANT_COPY_DTYPES = { + torch.bool, + torch.uint8, + torch.int8, + torch.int16, + torch.int32, + torch.float16, + torch.bfloat16, + torch.float32, +} + class PatternNode: op_types: dict[str, Optional[list]] = {} @@ -56,32 +73,84 @@ def _copy_source(node: torch.fx.Node) -> Optional[torch.fx.Node]: return source if isinstance(source, torch.fx.Node) else None -def _remove_copies_of_inputs(partition: Partition) -> None: - """Leave out of the partition each copy that would return one of the - partition's inputs, unchanged, as one of its outputs. +def _constants(program: ExportedProgram) -> set[torch.fx.Node]: + """The placeholders that each partition reading them stores as its own + data (see tag_constant_data), so OpenVINO gets them as constants, not as + inputs.""" + signature = program.graph_signature + mutated = set(signature.buffers_to_mutate.values()) + names = { + *signature.inputs_to_parameters, + *signature.inputs_to_lifted_tensor_constants, + *( + name + for name, buffer in signature.inputs_to_buffers.items() + if buffer not in mutated + ), + } + return { + node + for node in program.graph.nodes + if node.op == "placeholder" and node.name in names + } + + +def _remove_copies_of_inputs( + partition: Partition, constants: set[torch.fx.Node], dtypes: set[torch.dtype] +) -> None: + """Leave out of the partition each copy that would return, unchanged, one + of the partition's inputs or a constant OpenVINO cannot return. OpenVINO would return the input itself, and when the runtime binds the output buffers after the inputs, its CPU plugin does not write such an output and can overwrite the input with the output buffer's old contents. Copies used only inside the partition stay, because OpenVINO reads every input in the default dim order, which a copy left out could have changed. + A copy of a constant that only this partition reads stays too, since the + constant is not a partition input. Such a copy still leaves when it is not + in the default dim order, since OpenVINO writes every output in the default + one, when its type is not in dtypes, the types OpenVINO can return, or when + another output also returns that constant, since OpenVINO's PyTorch + frontend cannot convert two outputs of one constant. Like any input, a copy + left out that is not in the default dim order is still read wrong by the + ops that use it inside. """ - def copies_input(node: torch.fx.Node) -> bool: + def copied(node: torch.fx.Node) -> Optional[torch.fx.Node]: while (source := _copy_source(node)) is not None: if source not in partition.nodes: - return True + return source node = source - return False + return None - # Taking out a copy can make the copy it reads an output. - while leaving := [ - node - for node in partition.nodes - if copies_input(node) and any(u not in partition.nodes for u in node.users) - ]: + # Taking out a copy can make another copy leave. + while True: + outputs = { + node: copied(node) + for node in partition.nodes + if any(user not in partition.nodes for user in node.users) + } + returns = Counter(outputs.values()) + leaving = [ + node + for node, source in outputs.items() + if source is not None + and not ( + source in constants + and returns[source] == 1 + and all(user in partition.nodes for user in source.users) + and node.meta["val"].is_contiguous() + and node.meta["val"].dtype in dtypes + ) + ] + if not leaving: + return for node in leaving: - partition.remove_node(node) + # The copies it reads leave with it. A copy of a constant left inside + # could feed it while the partition also reads it, a cycle. + while (inner := _copy_source(node)) is not None and node in partition.nodes: + partition.remove_node(node) + node = inner class OpenvinoOperatorsSupport(OperatorSupportBase): @@ -294,8 +363,22 @@ def partition(self, exported_program: ExportedProgram) -> PartitionResult: partition_list = partitioner.propose_partitions() partition_tags = {} + constants = _constants(exported_program) + options = { + spec.key: spec.value.decode() for spec in self.delegation_spec.compile_specs + } + precision = _inference_precision( + options.get("device", "CPU"), options.get("inference_precision") + ) + # OpenVINO can round a float copy in any precision but f32, as an ARM + # CPU does in f16. + dtypes = { + dtype + for dtype in _CONSTANT_COPY_DTYPES + if precision == "f32" or not dtype.is_floating_point + } for partition in partition_list: - _remove_copies_of_inputs(partition) + _remove_copies_of_inputs(partition, constants, dtypes) for node in partition.nodes: tag = f"tag{partition.id}" node.meta["delegation_tag"] = tag diff --git a/backends/openvino/preprocess.py b/backends/openvino/preprocess.py index e11ac05aac1..1a17d120d3c 100644 --- a/backends/openvino/preprocess.py +++ b/backends/openvino/preprocess.py @@ -6,7 +6,7 @@ # mypy: disable-error-code=import-not-found -from typing import final, List +from typing import final, List, Optional import torch from executorch.backends.openvino._passes import ( @@ -22,7 +22,7 @@ ) from executorch.exir.backend.compile_spec_schema import CompileSpec from executorch.exir.passes.memory_format_ops_pass import DimOrderOpsRevertPass -from openvino import compile_model, opset13 # type: ignore[import-untyped] +from openvino import compile_model, opset13, opset16 # type: ignore[import-untyped] from openvino.frontend import FrontEndManager # type: ignore[import-untyped] from openvino.frontend.pytorch.fx_decoder import ( # type: ignore[import-untyped] TorchFXPythonDecoder, @@ -63,6 +63,31 @@ def _match_partition_inputs( ) +def _inference_precision(device: str, requested: Optional[str]) -> Optional[str]: + """The INFERENCE_PRECISION_HINT to pass, or None for the device's default.""" + return requested or ("f32" if device.startswith("CPU") else None) + + +def _copy_returned_inputs(model, device: str) -> None: + """Make the converted model compute each output that is one of its inputs. + + The converter turns copies like aten.clone into nothing, so such an output + is the input itself. The CPU plugin does not write it, and can overwrite + the input with the output buffer's old contents. OpenVINO keeps an + Identity between an input and an output, and the CPU plugin runs it as a + copy, while it can remove a Convert to the same type. On an ARM CPU at + f16, it rounds this copy of a float input, like its other float outputs. + Other devices are left as they are, since neither the problem nor Identity + was checked there. + """ + if not device.startswith("CPU"): + return + for result in model.get_results(): + source = result.input_value(0) + if source.get_node().get_type_name() == "Parameter": + result.input(0).replace_source_output(opset16.identity(source).output(0)) + + @final class OpenvinoBackend(BackendDetails): @@ -142,6 +167,7 @@ def preprocess( frontend = FrontEndManager().load_by_framework("pytorch") model = frontend.convert(frontend.load(decoder)) _match_partition_inputs(model, module, input_shapes, input_types) + _copy_returned_inputs(model, device) ppp = PrePostProcessor(model) # The converter takes an int input, like a SymInt size, as a scalar, but # the runtime passes it as an int64 tensor of shape [1]. @@ -150,7 +176,7 @@ def preprocess( ppp.input(i).preprocess().custom(lambda x: opset13.squeeze(x, 0).output(0)) # On ARM CPUs OpenVINO fixes the precision here, not when the runtime # imports the model, so this must follow the same rule as the runtime. - precision = requested_precision or ("f32" if device.startswith("CPU") else None) + precision = _inference_precision(device, requested_precision) config = {"INFERENCE_PRECISION_HINT": precision} if precision else {} compiled = compile_model(ppp.build(), device, config) model_bytes = compiled.export_model() diff --git a/backends/openvino/runtime/OpenvinoBackend.cpp b/backends/openvino/runtime/OpenvinoBackend.cpp index ff7c33aa4d9..15af84040bf 100644 --- a/backends/openvino/runtime/OpenvinoBackend.cpp +++ b/backends/openvino/runtime/OpenvinoBackend.cpp @@ -214,7 +214,7 @@ bool OpenvinoBackend::ensure_loaded() const { Error, "OpenVINO runtime not found (loading '%s' failed: %s). " "Ensure '%s' can be found (%s), or install with: " - "pip install \"openvino>=2025.1.0,<2027.0.0\"", + "pip install \"openvino>=2025.4.0,<2027.0.0\"", to_utf8(effective_path).c_str(), error.c_str(), to_utf8(kDefaultLibName).c_str(), diff --git a/backends/openvino/tests/ops/base_openvino_op_test.py b/backends/openvino/tests/ops/base_openvino_op_test.py index df921c4c4ac..1be9cb38fe3 100644 --- a/backends/openvino/tests/ops/base_openvino_op_test.py +++ b/backends/openvino/tests/ops/base_openvino_op_test.py @@ -1,7 +1,7 @@ import os import tempfile import unittest -from typing import Optional +from typing import Any, Optional, Sequence import executorch import torch @@ -10,6 +10,7 @@ from executorch.exir import ( EdgeCompileConfig, EdgeProgramManager, + ExecutorchProgramManager, to_edge_transform_and_lower, ) from executorch.exir.backend.backend_details import CompileSpec @@ -24,6 +25,27 @@ class BaseOpenvinoOpTest(unittest.TestCase): atol = 1e-3 rtol = 1e-3 + @staticmethod + def execute_program( + exec_prog: ExecutorchProgramManager, + all_inputs: Sequence[Sequence[torch.Tensor]], + ) -> list[Sequence[Any]]: + """Run each set of inputs, in order, on one loaded method, as an app + would.""" + # Load from a file: unlike a buffer, it really frees the delegate data + # when the backend frees it. + with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmp_dir: + pte_path = os.path.join(tmp_dir, "model.pte") + with open(pte_path, "wb") as f: + exec_prog.write_to_file(f) + program = Runtime.get().load_program(pte_path) + method = program.load_method("forward") + assert method is not None + all_outputs = [method.execute(inputs) for inputs in all_inputs] + # Windows cannot delete a mapped file, so release it first. + del method, program + return all_outputs + def execute_layer_test( self, module: torch.nn.Module, @@ -68,22 +90,8 @@ def execute_layer_test( # Execute the model and compare the outputs with the reference outputs if assert_output_equal: - # Later inputs run on the same loaded method, as an app would. all_inputs = (sample_inputs, *more_inputs) - - # Load from a file: unlike a buffer, it really frees the delegate - # data when the backend frees it. - with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmp_dir: - pte_path = os.path.join(tmp_dir, "model.pte") - with open(pte_path, "wb") as f: - exec_prog.write_to_file(f) - program = Runtime.get().load_program(pte_path) - method = program.load_method("forward") - assert method is not None - all_outputs = [method.execute(inputs) for inputs in all_inputs] - # Windows cannot delete a mapped file, so release it first. - del method, program - + all_outputs = self.execute_program(exec_prog, all_inputs) for inputs, outputs in zip(all_inputs, all_outputs): # Execute the module in eager mode to calculate the reference outputs ref_output = module(*inputs) @@ -105,3 +113,5 @@ def execute_layer_test( ), msg=f"ref_output:\n{ref_output[i]}\n\ntest_output:\n{outputs[i]}", ) + + return exec_prog diff --git a/backends/openvino/tests/ops/test_copy.py b/backends/openvino/tests/ops/test_copy.py index c65b1d35272..586a7ec3422 100644 --- a/backends/openvino/tests/ops/test_copy.py +++ b/backends/openvino/tests/ops/test_copy.py @@ -1,8 +1,14 @@ +import itertools + import torch +from executorch.backends.openvino.preprocess import OpenvinoBackend from executorch.backends.openvino.tests.ops.base_openvino_op_test import ( BaseOpenvinoOpTest, ) -from executorch.exir import EdgeCompileConfig +from executorch.exir import EdgeCompileConfig, to_edge +from executorch.exir.backend.backend_api import to_backend +from executorch.exir.backend.backend_details import CompileSpec +from torch.export import Dim, export class TestCopyOperator(BaseOpenvinoOpTest): @@ -17,22 +23,28 @@ def forward(self, x): copy = self.copy(x) return copy, torch.sin(copy) + clone = "dim_order_ops::_clone_dim_order" copies = { - "clone": lambda x: x.clone(), - "full slice": lambda x: x[:, :], - "clone of a clone": lambda x: x.clone().clone(), + "clone": (lambda x: x.clone(), clone, None), + "full slice": (lambda x: x[:, :], "aten::alias_copy", None), + "clone of a clone": (lambda x: x.clone().clone(), clone, None), + # LLM exports turn dim order off, which makes a clone aten.clone. + "clone, dim order off": ( + lambda x: x.clone(), + "aten::clone", + EdgeCompileConfig(_skip_dim_order=True), + ), } inputs = (torch.randn(2, 3, 4, 5),) - for name, copy in copies.items(): + for name, (copy, operator, config) in copies.items(): with self.subTest(copy=name): - self.execute_layer_test(CopyAndSin(copy), inputs) - - # LLM exports turn dim order off, which makes a clone aten.clone. - self.execute_layer_test( - CopyAndSin(lambda x: x.clone()), - inputs, - compile_config=EdgeCompileConfig(_skip_dim_order=True), - ) + program = self.execute_layer_test( + CopyAndSin(copy), inputs, compile_config=config + ) + # On the CPU, preprocess would also give right values, so check + # where the copy runs. + operators = program.executorch_program.execution_plan[0].operators + self.assertIn(operator, [op.name for op in operators]) def test_copy_of_input_used_inside(self): class ChannelsLastMean(torch.nn.Module): @@ -42,3 +54,213 @@ def forward(self, x): # Run outside, the clone would hand OpenVINO a channels-last input, # which it reads as if it were contiguous. self.execute_layer_test(ChannelsLastMean(), (torch.randn(2, 3, 4, 5),)) + + def test_copy_of_constant_as_output(self): + if not self.device.startswith("CPU"): + self.skipTest("float copies stay only in float32, the CPU default") + + class CopyOfConstant(torch.nn.Module): + def __init__(self, kind, memory_format=None, dtype=torch.float32): + super().__init__() + self.memory_format = memory_format + if dtype.is_floating_point: + value = torch.randn(2, 3, 4, 5).to(dtype) + elif dtype == torch.bool: + value = torch.randn(2, 3, 4, 5) > 0 + else: + # float16 holds every integer only up to 2048, so these + # show a copy rounded through it, and x + constant stays + # in its range. + info = torch.iinfo(dtype) + low, high = max(info.min, -(2**15)), min(info.max, 2**15) + value = torch.randint(low, high, (2, 3, 4, 5), dtype=dtype) + if kind == "parameter": + self.constant = torch.nn.Parameter(value) + elif kind == "buffer": + self.register_buffer("constant", value) + else: + self.constant = value + + def forward(self, x): + copy = self.constant.clone(memory_format=self.memory_format) + return copy, x + self.constant + + cases = { + "parameter": (CopyOfConstant("parameter"), None), + "buffer": (CopyOfConstant("buffer"), None), + "lifted tensor": (CopyOfConstant("lifted tensor"), None), + # With dim order off this stays aten.clone, and the portable clone + # kernel rejects preserve_format. + "buffer, dim order off": ( + CopyOfConstant("buffer", torch.preserve_format), + EdgeCompileConfig(_skip_dim_order=True), + ), + # The precision changes only how OpenVINO computes floats. + "int32 buffer, f16 precision": ( + CopyOfConstant("buffer", dtype=torch.int32), + None, + CompileSpec("inference_precision", b"f16"), + ), + } + for dtype in ( + torch.bool, + torch.uint8, + torch.int8, + torch.int16, + torch.int32, + torch.float16, + torch.bfloat16, + ): + cases[f"{dtype} buffer"] = (CopyOfConstant("buffer", dtype=dtype), None) + for name, (module, config, *compile_specs) in cases.items(): + with self.subTest(case=name): + inputs = (torch.randn(2, 3, 4, 5),) + program = self.execute_layer_test( + module, + inputs, + more_inputs=((torch.randn(2, 3, 4, 5),),), + compile_specs=tuple(compile_specs), + compile_config=config, + ) + # OpenVINO gets a constant as its own data, not as an input, so + # the copy stays. Run outside, it would store the constant twice. + plan = program.executorch_program.execution_plan[0] + self.assertEqual(plan.operators, []) + # The tolerance above lets a copy rounded through float16 pass. + copy = self.execute_program(program, [inputs])[0][0] + torch.testing.assert_close(copy, module.constant, rtol=0, atol=0) + + def test_copy_of_constant_that_runs_outside(self): + class Buffered(torch.nn.Module): + def __init__(self, body, buffer): + super().__init__() + self.body = body + self.register_buffer("buffer", buffer) + + def forward(self, x): + return self.body(self.buffer, x) + + def copy_of_copy_and_sin(buffer, x): + copy = buffer.clone().clone(memory_format=torch.channels_last) + return copy, torch.sin(copy) + + def copy_and_copy_of_it(buffer, x): + copy = buffer.clone() + return copy, copy.clone(), buffer + x + + floats = torch.randn(2, 3, 4, 5) + cases = { + # OpenVINO writes every output in the default dim order. + "channels last": ( + lambda b, x: (b.clone(memory_format=torch.channels_last), b + x), + floats, + ), + # The inner copy must leave too, or export finds a cycle. + "copy of a copy, channels last": (copy_of_copy_and_sin, floats), + # The CPU plugin computes 64-bit types in 32 bits. + "int64": ( + lambda b, x: (b.clone(), x + 1), + torch.tensor([2**40, -(2**40)]), + ), + # The runtime cannot bind a uint16 output. + "uint16": ( + lambda b, x: (b.clone(), x + 1), + torch.tensor([0, 40000, 65535], dtype=torch.uint16), + ), + # OpenVINO's PyTorch frontend cannot return one constant twice. + "two copies": (copy_and_copy_of_it, floats), + # In a lower precision OpenVINO can round a float copy. + "float16 precision": ( + lambda b, x: (b.clone(), x + 1), + floats, + CompileSpec("inference_precision", b"f16"), + ), + # The partitioner does not take atan, so the CPU stores the buffer + # anyway, and a copy made there keeps it stored once. + "also read outside": (lambda b, x: (b.clone(), torch.atan(b) + x), floats), + } + for name, (body, buffer, *compile_specs) in cases.items(): + with self.subTest(case=name): + program = self.execute_layer_test( + Buffered(body, buffer), + (torch.randn(2, 3, 4, 5),), + compile_specs=tuple(compile_specs), + ) + operators = program.executorch_program.execution_plan[0].operators + names = [op.name for op in operators] + if name == "also read outside": + self.assertIn("aten::atan", names) + self.assertIn("dim_order_ops::_clone_dim_order", names) + + def test_copy_of_mutated_buffer_as_output(self): + class CopyOfMutatedBuffer(torch.nn.Module): + def __init__(self): + super().__init__() + self.register_buffer("buffer", torch.randn(2, 3, 4, 5)) + + def forward(self, x): + copy = self.buffer.clone() + self.buffer.add_(x) + return copy, torch.sin(copy) + + program = self.execute_layer_test( + CopyOfMutatedBuffer(), + (torch.randn(2, 3, 4, 5),), + assert_output_equal=False, + ) + # A mutated buffer is passed to the partition as an input, so its copy + # must run outside. + operators = program.executorch_program.execution_plan[0].operators + self.assertIn("dim_order_ops::_clone_dim_order", [op.name for op in operators]) + + # The program does not keep the starting value of a mutated buffer, so + # check each call against the one before it. + x1, x2 = torch.randn(2, 3, 4, 5), torch.randn(2, 3, 4, 5) + first, second = self.execute_program(program, ((x1,), (x2,))) + tolerance = {"atol": self.atol, "rtol": self.rtol} + torch.testing.assert_close(second[0], first[0] + x1, **tolerance) + for copy, sine in (first, second): + torch.testing.assert_close(sine, torch.sin(copy), **tolerance) + + def test_whole_program_returning_copy_of_input(self): + if not self.device.startswith("CPU"): + self.skipTest("preprocess adds the copy only for the CPU") + + class CopyAndReshape(torch.nn.Module): + def forward(self, x): + return x.clone(), x.reshape(-1) + + class Lowered(torch.nn.Module): + def __init__(self, lowered): + super().__init__() + self.lowered = lowered + + def forward(self, x): + return self.lowered(x) + + inputs = { + "float": lambda n: torch.randn(n, 3), + "int64": lambda n: torch.randint(-100, 100, (n, 3)), + "bool": lambda n: torch.randn(n, 3) > 0, + } + for (name, make), dynamic in itertools.product(inputs.items(), (False, True)): + dynamic_shapes = ({0: Dim("rows", min=1, max=4)},) if dynamic else None + with self.subTest(dtype=name, dynamic=dynamic): + x = make(4) + # Without a partitioner the copy cannot run on the CPU, so + # OpenVINO has to make it. + lowered = to_backend( + OpenvinoBackend.__name__, + to_edge( + export(CopyAndReshape(), (x,), dynamic_shapes=dynamic_shapes) + ).exported_program(), + [CompileSpec("device", self.device.encode())], + ) + program = to_edge( + export(Lowered(lowered), (x,), dynamic_shapes=dynamic_shapes) + ).to_executorch() + values = (x, make(2 if dynamic else 4)) + outputs = self.execute_program(program, [(v,) for v in values]) + for value, (copy, flat) in zip(values, outputs): + torch.testing.assert_close(copy, value) + torch.testing.assert_close(flat, value.reshape(-1)) diff --git a/docs/source/build-run-openvino.md b/docs/source/build-run-openvino.md index 2d842ed2bcd..1aecb1b48c0 100644 --- a/docs/source/build-run-openvino.md +++ b/docs/source/build-run-openvino.md @@ -43,9 +43,12 @@ The easiest way to use the OpenVINO backend is via the prebuilt pip wheel: pip install executorch[openvino] ``` -For now the extra installs no runtime on Python 3.15, or on Linux aarch64 with Python -3.14. OpenVINO publishes no build for Python 3.15, and its aarch64 builds for Python 3.14 -need glibc 2.35 or newer. +For now the extra installs no runtime on Python 3.15, where OpenVINO publishes no build. +On Linux aarch64, OpenVINO needs glibc 2.35 or newer, as on Ubuntu 22.04 or Raspberry Pi OS +Bookworm and later. On an older aarch64 system, `pip install executorch[openvino]` does not +fail: pip finds no OpenVINO 2025.4 or newer for that system, so it installs an older +ExecuTorch release instead. On such a system, install `executorch` without the extra, then +build OpenVINO from source as described below. The backend automatically discovers the OpenVINO C library from the pip-installed package — no environment variable setup is needed. @@ -61,14 +64,14 @@ If you need a custom build (different platform, custom configuration, etc.), fol Before you begin, ensure you have the OpenVINO runtime installed and configured on your system. For most users, we recommend installing it via pip: ```bash -pip install "openvino>=2025.1.0,<2027.0.0" +pip install "openvino>=2025.4.0,<2027.0.0" ``` Alternatively, you can build OpenVINO from source or follow the official [OpenVINO installation guide](https://docs.openvino.ai/2025/get-started/install-openvino.html) for platform-specific options: ```bash git clone https://github.com/openvinotoolkit/openvino.git -cd openvino && git checkout releases/2025/1 +cd openvino && git checkout releases/2026/4 git submodule update --init --recursive sudo ./install_build_dependencies.sh mkdir build && cd build @@ -124,7 +127,8 @@ float16, as measured on an Orin Nano. The value is OpenVINO's `INFERENCE_PRECISION_HINT`: `f32`, `f16` or `bf16`. An empty value means the default, and any other value fails at lowering. A CPU -without float16 support computes an `f16` program in float32. OpenVINO has no +without float16 support computes an `f16` program in bfloat16 if it has bfloat16 +units, as some x86 CPUs do, and in float32 otherwise. OpenVINO has no bfloat16 on ARM CPUs, so there `bf16` keeps the ARM default, float16 where the CPU supports it. Which x86 CPUs compute in bfloat16 is up to OpenVINO; see its [Precision Control](https://docs.openvino.ai/2026/openvino-workflow/running-inference/optimize-inference/precision-control.html) diff --git a/docs/source/using-executorch-cpp.md b/docs/source/using-executorch-cpp.md index 2391584196f..015b286243a 100644 --- a/docs/source/using-executorch-cpp.md +++ b/docs/source/using-executorch-cpp.md @@ -272,9 +272,11 @@ $env:OPENVINO_LIB_PATH = python -c "import openvino, os; print(os.path.join(os.p Without it the delegate still registers and the program still links, and the failure arrives later, when the model is loaded. -For now the openvino extra installs no runtime on Python 3.15, or on Linux aarch64 with -Python 3.14. OpenVINO publishes no build for Python 3.15, and its aarch64 builds for Python -3.14 need glibc 2.35 or newer. +For now the openvino extra installs no runtime on Python 3.15, where OpenVINO publishes no +build. On Linux aarch64, OpenVINO needs glibc 2.35 or newer, as on Ubuntu 22.04 or Raspberry +Pi OS Bookworm and later. On an older aarch64 system, such as RHEL 9 or Amazon Linux 2023, +`pip install executorch[openvino]` does not fail but installs an older ExecuTorch release; see +[OpenVINO Backend](build-run-openvino.md) for what to do there. The MLX delegate has a similar requirement. Its Metal kernels live in a separate `mlx.metallib` file, which MLX looks for next to whichever library holds MLX code. The wheel ships it beside the diff --git a/pyproject.toml b/pyproject.toml index 116552df1ed..f1d717bc1eb 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -86,10 +86,10 @@ ethos_u = [ "tosa-tools==2026.5.0", ] openvino = [ - # OpenVINO publishes no build for Python 3.15, and its Linux aarch64 builds for Python - # 3.14 need glibc 2.35 while the wheel needs only 2.28. Without these bounds pip fails to - # install the extra on Python 3.15 and on those older aarch64 systems. - "openvino>=2025.1.0,<2027.0.0; (platform_system == 'Linux' or (platform_system == 'Windows' and platform_machine == 'AMD64')) and python_version < '3.15' and (platform_machine != 'aarch64' or python_version < '3.14')", + # OpenVINO publishes no build for Python 3.15. Its Linux aarch64 builds need glibc 2.35 or + # newer while the wheel needs only 2.28, so on an older aarch64 system pip installs an older + # ExecuTorch for this extra instead; the docs say what to do there. + "openvino>=2025.4.0,<2027.0.0; (platform_system == 'Linux' or (platform_system == 'Windows' and platform_machine == 'AMD64')) and python_version < '3.15'", ] [project.urls]