Add a desktop mirror to OVXR teleoperation

README.md

@@ -91,10 +91,10 @@ Third-party materials may carry their own licenses.
 ### Separately retrieved optional materials
 
 The optional `isaaccapture[ovxr]` extra retrieves OVXR and its OVRTX/OVStage
-dependencies.
+dependencies. The separate `isaaccapture[desktop-mirror]` extra retrieves OVUI.
 These packages are downloaded separately, rather than bundled in the Isaac Capture
 wheel. Their licenses apply independently of Isaac Capture's Apache-2.0 license.
-OVXR, OVRTX and OVStage are NVIDIA proprietary software.
+OVXR, OVRTX and OVStage are NVIDIA proprietary software. OVUI is Apache-2.0.
 See [package sources and license details](docs/source/getting_started/ovxr.rst#optional-packages-and-licenses).
 Review the applicable package licenses and terms before installing or using these
 optional dependencies.

docs/source/getting_started/ovxr.rst

@@ -37,6 +37,9 @@ from the Isaac Capture wheel. These packages retain their own licenses:
   `OVRTX <https://pypi.org/project/ovrtx/>`__ and
   `OVStage <https://pypi.org/project/ovstage/>`__ are NVIDIA proprietary software.
   Review the license included with each distribution before use.
+* `OVUI <https://pypi.org/project/ovui/>`__ is Apache-2.0 and is installed by
+  the independent ``desktop-mirror`` extra. Review its package metadata for
+  supported Python versions and platforms.
 
 CloudXR native libraries are bundled in Isaac Capture wheels and remain subject
 to their separate NVIDIA license and CloudXR EULA acceptance.
@@ -72,6 +75,32 @@ Use ``CUDAToolkit_ROOT`` to select a CUDA Toolkit installation and ``VULKAN_SDK`
 to help locate a Vulkan SDK. The GPU driver and Vulkan runtime loader are required
 on the host.
 
+Optional desktop mirror
+-----------------------
+
+The desktop mirror requires **Python 3.12 or newer**. Isaac Capture's core
+continues to support Python 3.11; on that version the extra does not install OVUI
+and the desktop mirror is unavailable. Use a Python 3.12+ environment and select
+both extras with ``isaaccapture[ovxr,desktop-mirror]`` in the installation
+requirement or the application's optional dependencies to include the OVUI
+mirror. The mirror requires CUDA 13 Python bindings, a compatible NVIDIA driver
+as described above, and a desktop display associated with a CUDA device.
+The native camera helper is required only when camera panels are also enabled.
+
+Initialize the renderer before starting the mirror. OVUI uses its default Vulkan
+backend; no ``OMNIUI_BACKEND`` setting is required. Capture does not change the
+process-wide backend. GPU images stay on their producing CUDA device, and OVUI
+copies them into its Vulkan image. The mirror requires the active OVUI backend
+to be Vulkan. Remove conflicting ``OMNIUI_BACKEND`` settings before OVUI starts.
+This selection does not change headset rendering.
+
+The pinned OVUI 0.3.0 Vulkan upload copies on its import device's default CUDA
+stream. The mirror drains that stream before reusing the source buffer, including
+upload failures; the source image can reside on a different GPU. This depends on
+the synchronous ``ByteImageProvider.set_bytes_data_from_gpu`` implementation,
+not OVUI's separate asynchronous upload API. See the `OVUI upload implementation
+<https://github.com/NVIDIA-Omniverse/ovui/blob/60fc40daf56d010ccbc9734f64e6463b1017f74a/ovui/standalone/src/VulkanByteImageGpu.cpp#L1069-L1124>`_.
+
 Use from another application
 ----------------------------
 
@@ -83,9 +112,11 @@ resolved its render-product paths. Capture does not depend on an Isaac Lab task
 or scene configuration. ``camera_paths`` is required and must contain both stereo
 cameras. Set ``up_axis="Y"`` for a Y-up Stage; the default is ``"Z"``.
 
-``CameraPanelConsumer`` implements the ``OvxrFrameConsumer`` lifecycle. Configure panels with a tuple of
+``CameraPanelConsumer`` and ``OwnedDesktopMirror`` implement the same
+``OvxrFrameConsumer`` lifecycle. Configure panels with a tuple of
 ``CameraPanelConfig`` values, each pairing a render-product path with its placement
-and image bound.
+and image bound. Configure the mirror's camera choices with ``MirrorProduct``
+values. Either consumer can be enabled independently.
 
 Camera-panel attachment failure disables panels and logs the reason without
 blocking teleoperation input. An incomplete GPU copy disables the affected panel;

pyproject.toml

@@ -54,6 +54,11 @@ ovxr = [
     "ovxr==0.1.0.5842",
     "cuda-bindings>=13,<14",
 ]
+desktop-mirror = [
+    # OVUI requires 3.12; core Capture continues to support 3.11.
+    "ovui==0.3.0; python_version >= '3.12'",
+    "cuda-bindings>=13,<14",
+]
 retargeters = [
     "dex-retargeting>=0.4.6,<0.6.0",
     "scipy>=1.15.0",

scripts/check_camera_quad_wheel.py

@@ -1,7 +1,7 @@
 # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 # SPDX-License-Identifier: Apache-2.0
 
-"""Check optional camera-helper delivery in an actual project wheel.
+"""Check camera-helper and desktop-mirror delivery in an actual project wheel.
 
 Run against wheels built with BUILD_OVXR_CAMERA_QUADS ON and OFF, including
 ON -> OFF -> ON in the same build directory to detect stale packaged data.
@@ -26,6 +26,15 @@ def main():
         prefix + "libisaaccapture_camera_panels.so",
     }
     with ZipFile(args.wheel) as wheel:
+        mirror_modules = {
+            "isaaccapture/ovxr_session_manager/" + name + ".py"
+            for name in ("owned_mirror", "desktop_mirror", "gpu_output")
+        }
+        missing = mirror_modules.difference(wheel.namelist())
+        if missing:
+            raise RuntimeError(
+                f"Wheel is missing desktop mirror modules: {sorted(missing)}"
+            )
         members = {
             name
             for name in wheel.namelist()

src/compat/pyproject.toml

@@ -23,6 +23,7 @@ Homepage = "https://github.com/NVIDIA/IsaacCapture"
 dev = ["isaaccapture[dev]>=1.6.dev0,<1.9"]
 ui = ["isaaccapture[ui]>=1.6.dev0,<1.9"]
 ovxr = ["isaaccapture[ovxr]>=1.6.dev0,<1.9"]
+desktop-mirror = ["isaaccapture[desktop-mirror]>=1.6.dev0,<1.9"]
 retargeters = ["isaaccapture[retargeters]>=1.6.dev0,<1.9"]
 retargeters-lite = ["isaaccapture[retargeters-lite]>=1.6.dev0,<1.9"]
 grounding = ["isaaccapture[grounding]>=1.6.dev0,<1.9"]

src/core/python/CMakeLists.txt

@@ -84,6 +84,7 @@ add_custom_target(python_package ALL
     COMMAND ${CMAKE_COMMAND} -E copy "${CMAKE_CURRENT_SOURCE_DIR}/requirements.txt" "${CMAKE_BINARY_DIR}/python_package/$<CONFIG>/"
     COMMAND ${CMAKE_COMMAND} -E copy "${CMAKE_CURRENT_SOURCE_DIR}/requirements-dev.txt" "${CMAKE_BINARY_DIR}/python_package/$<CONFIG>/"
     COMMAND ${CMAKE_COMMAND} -E copy "${CMAKE_CURRENT_SOURCE_DIR}/requirements-ui.txt" "${CMAKE_BINARY_DIR}/python_package/$<CONFIG>/"
+    COMMAND ${CMAKE_COMMAND} -E copy "${CMAKE_CURRENT_SOURCE_DIR}/requirements-desktop-mirror.txt" "${CMAKE_BINARY_DIR}/python_package/$<CONFIG>/"
     COMMAND ${CMAKE_COMMAND} -E copy "${CMAKE_CURRENT_SOURCE_DIR}/requirements-ovxr.txt" "${CMAKE_BINARY_DIR}/python_package/$<CONFIG>/"
     COMMAND ${CMAKE_COMMAND} -E copy "${CMAKE_CURRENT_SOURCE_DIR}/requirements-cloudxr.txt" "${CMAKE_BINARY_DIR}/python_package/$<CONFIG>/"
     COMMAND ${CMAKE_COMMAND} -E copy "${CMAKE_CURRENT_SOURCE_DIR}/requirements-retargeters.txt" "${CMAKE_BINARY_DIR}/python_package/$<CONFIG>/"

src/core/python/pyproject.toml.in

@@ -31,6 +31,7 @@ dynamic = ["dependencies", "optional-dependencies"]
 dependencies = {file = ["requirements.txt"]}
 optional-dependencies.dev = {file = ["requirements-dev.txt"]}
 optional-dependencies.ui = {file = ["requirements-ui.txt"]}
+optional-dependencies.desktop-mirror = {file = ["requirements-desktop-mirror.txt"]}
 optional-dependencies.ovxr = {file = ["requirements-ovxr.txt"]}
 optional-dependencies.retargeters = {file = ["requirements-retargeters.txt"]}
 optional-dependencies.retargeters-lite = {file = ["requirements-retargeters-lite.txt"]}

src/core/python/requirements-desktop-mirror.txt

@@ -0,0 +1,3 @@
+# Optional standalone desktop mirror (Python 3.12+)
+ovui==0.3.0; python_version >= "3.12"
+cuda-bindings>=13,<14

src/python/isaaccapture/ovxr_session_manager/__init__.py

@@ -13,6 +13,7 @@
     OvxrSession,
     StageOrdinalLedger,
 )
+from .desktop_mirror import DesktopMirrorConfig, DesktopMirrorStatus
 
 __all__ = [
     "OvxrSessionBinding",
@@ -21,6 +22,8 @@
     "OvxrFrameConsumer",
     "OvxrSession",
     "StageOrdinalLedger",
+    "DesktopMirrorConfig",
+    "DesktopMirrorStatus",
 ]
 
 from .camera_panels import CameraPanelConfig, CameraPanelConsumer

src/python/isaaccapture/ovxr_session_manager/desktop_mirror.py

@@ -0,0 +1,97 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Shared configuration, cadence and copy completion for the owned desktop mirror."""
+
+from __future__ import annotations
+
+import math
+from collections import deque
+from dataclasses import dataclass
+
+
+@dataclass(frozen=True)
+class DesktopMirrorConfig:
+    """Configuration for the optional, application-side OVUI mirror."""
+
+    max_fps: float = 30.0
+    window_width: int = 960
+    title: str = "Isaac Teleop — render products"
+    initial_status_text: str = ""
+
+    def __post_init__(self) -> None:
+        if not math.isfinite(self.max_fps) or self.max_fps <= 0.0:
+            raise ValueError("desktop mirror max_fps must be finite and positive")
+        if self.window_width <= 0:
+            raise ValueError("desktop mirror window_width must be positive")
+        if not self.title:
+            raise ValueError("desktop mirror title must be non-empty")
+
+
+@dataclass(frozen=True)
+class DesktopMirrorStatus:
+    """Thread-safe snapshot of mirror lifecycle and latest-frame behavior."""
+
+    thread_id: int | None = None
+    window_opened: bool = False
+    closed: bool = False
+    error: str | None = None
+    presented_frames: int = 0
+    last_sequence: int = 0
+    main_thread_fps: float = 0.0
+    status_text: str = ""
+    render_thread_fps: float = 0.0
+
+
+class _CadenceSampler:
+    """Measure recent host and render cadence from monotonic samples."""
+
+    def __init__(self, window_s: float = 1.0) -> None:
+        self._window_ns = int(window_s * 1_000_000_000)
+        self._host_samples: deque[int] = deque()
+        self._render_samples: deque[int] = deque()
+
+    def record_host_frame(self, now_ns: int) -> None:
+        self._host_samples.append(now_ns)
+        while (
+            len(self._host_samples) > 2
+            and self._host_samples[1] <= now_ns - self._window_ns
+        ):
+            self._host_samples.popleft()
+
+    def record_render_frame(self, now_ns: int) -> None:
+        self._render_samples.append(now_ns)
+        while (
+            len(self._render_samples) > 2
+            and self._render_samples[1] <= now_ns - self._window_ns
+        ):
+            self._render_samples.popleft()
+
+    @staticmethod
+    def _event_rate(samples: deque[int] | list[int]) -> float:
+        if len(samples) < 2:
+            return 0.0
+        elapsed_s = (samples[-1] - samples[0]) / 1_000_000_000
+        return 0.0 if elapsed_s <= 0.0 else (len(samples) - 1) / elapsed_s
+
+    def rates(self, now_ns: int | None = None) -> tuple[float, float]:
+        def recent_rate(samples: deque[int]) -> float:
+            if now_ns is not None and (
+                not samples or now_ns - samples[-1] > self._window_ns
+            ):
+                return 0.0
+            return self._event_rate(samples)
+
+        return recent_rate(self._host_samples), recent_rate(self._render_samples)
+
+
+def _wait_input_copy() -> None:
+    """Drain the current OVUI Vulkan upload device's legacy CUDA stream."""
+    from cuda.bindings import runtime
+
+    # OVUI 0.3.0 VulkanByteImageGpu::updateImageImpl copies on its import
+    # device's default stream. Also drain on failure before reusing the source;
+    # the image's source device may differ from the current upload device.
+    result = runtime.cudaStreamSynchronize(runtime.cudaStreamLegacy)
+    if int(result[0]):
+        raise RuntimeError(f"OVUI input-copy completion failed: CUDA {result[0]}")

src/python/isaaccapture/ovxr_session_manager/gpu_output.py

@@ -126,6 +126,128 @@ class MappedGpuImage(MappedOutput):
     is_array: bool = False
 
 
+@dataclass
+class OwnedGpuImage:
+    """One reusable CUDA allocation, leased until the UI's source read completes."""
+
+    width: int
+    height: int
+    pointer: int
+    pitch: int
+    device: int
+    owner: int
+    closed: bool = True
+
+    def close(self) -> None:
+        if threading.get_ident() != self.owner:
+            raise RuntimeError("GPU image must be retired on its allocation owner")
+        self.closed = True
+
+
+class OwnedCudaImagePool:
+    """At most two mirror copies; no borrowed OVRTX map escapes a frame."""
+
+    def __init__(self) -> None:
+        self._images: list[OwnedGpuImage] = []
+        self._source: MappedGpuImage | None = None
+        self._ready = False
+
+    def enable(self) -> None:
+        """UI owner, after OVUI has initialized its Vulkan image provider."""
+        self._ready = True
+
+    def __call__(
+        self, renderer: int, handle: int, max_bytes: int
+    ) -> OwnedGpuImage | None:
+        if not self._ready:
+            return None  # GUI startup is an expected optional-output omission.
+        self.finish_frame()
+        from cuda.bindings import runtime as cuda
+
+        try:
+            source = map_cuda_array_rgba(renderer, handle, max_bytes)
+        except OutputMapCleanupError as error:
+            self._source = error.image
+            raise
+        self._source = source
+        destination = source.device
+        try:
+            image = next((item for item in self._images if item.closed), None)
+            if image is not None and (image.width, image.height, image.device) != (
+                source.width,
+                source.height,
+                destination,
+            ):
+                _check_cuda(cuda.cudaSetDevice(image.device))
+                _check_cuda(cuda.cudaFree(image.pointer))
+                self._images.remove(image)
+                image = None
+            _check_cuda(cuda.cudaSetDevice(destination))
+            if image is None:
+                if len(self._images) >= 2:
+                    raise RuntimeError("both owned mirror images are in use")
+                result = cuda.cudaMallocPitch(source.width * 4, source.height)
+                _check_cuda(result)
+                image = OwnedGpuImage(
+                    source.width,
+                    source.height,
+                    int(result[1]),
+                    int(result[2]),
+                    destination,
+                    threading.get_ident(),
+                )
+                self._images.append(image)
+            # Check reused allocations too: CUDA may pad a new allocation past
+            # the limit, and a rejected allocation remains owned until cleanup.
+            if image.pitch * image.height > max_bytes:
+                raise RuntimeError("owned mirror allocation exceeds the byte bound")
+            # The source map and the destination have separate lifetimes. Prove
+            # the read complete before unmapping even when enqueue reports error.
+            source.busy = True
+            _check_cuda(
+                cuda.cudaMemcpy2DFromArrayAsync(
+                    image.pointer,
+                    image.pitch,
+                    source.pointer,
+                    0,
+                    0,
+                    source.width * 4,
+                    source.height,
+                    cuda.cudaMemcpyKind.cudaMemcpyDeviceToDevice,
+                    0,
+                )
+            )
+        finally:
+            _check_cuda(cuda.cudaSetDevice(destination))
+            if source.busy:
+                _check_cuda(cuda.cudaStreamSynchronize(0))
+                source.busy = False
+            source.close()
+            self._source = None
+        image.closed = False
+        return image
+
+    def finish_frame(self) -> None:
+        """Fail closed before submit/cancel if a source read/unmap was not proved."""
+        if self._source is not None:
+            raise RuntimeError(
+                "mirror still holds a borrowed frame output; retain its owner"
+            )
+
+    def close(self) -> None:
+        self.finish_frame()
+        if not self._images:
+            return
+        from cuda.bindings import runtime as cuda
+
+        for image in list(self._images):
+            if not image.closed:
+                raise RuntimeError("owned mirror image still has a consumer")
+            _check_cuda(cuda.cudaSetDevice(image.device))
+            _check_cuda(cuda.cudaFree(image.pointer))
+            self._images.remove(image)
+
+
 def map_cuda_array_rgba(renderer: int, handle: int, max_bytes: int) -> MappedGpuImage:
     """Retain the native image array, avoiding OVRTX array-to-linear conversion."""
     from cuda.bindings import runtime as cuda
@@ -164,7 +286,7 @@ def map_cuda_array_rgba(renderer: int, handle: int, max_bytes: int) -> MappedGpu
             or len(shape) != 3
             or shape[2] != 4
         ):
-            raise ValueError("mirror GPU output must be an opaque RGBA CUDA array")
+            raise ValueError("camera GPU output must be an opaque RGBA CUDA array")
         info = cuda.cudaArrayGetInfo(int(dl.data))
         _check_cuda(info)
         channel, extent = info[1:3]