Optimize shared camera image observations

[StafaH]

Summary

• move Kuka Allegro camera observations onto the shared image MDP term
• fuse image normalization with channel-layout conversion and infer single camera output types
• remove redundant camera, observation-manager, and clone work across camera tasks

Performance

Measured with 4096 Isaac-Lift-KukaAllegro-Camera environments using Newton physics, the Newton renderer, and 128x128 RGB:

• shared image MDP: 1.84 ms -> 1.39 ms (~24% faster)
• full environment: ~215 ms/step before and after (renderer-bound; difference is within noise)

Validation

• 160 targeted tests passed
• 2-environment Kuka Allegro Newton/newton-renderer smoke test passed
• formatting and code/style hooks passed

The repository-wide changelog hook remains blocked by a pre-existing modified immutable isaaclab_ov fragment and unrelated missing package fragments.

[greptile-apps]

Greptile Summary

The PR consolidates camera observations onto the shared image MDP path and removes redundant image, manager, and sensor-update work.

• Adds fused image normalization and channel-layout conversion.
• Infers the output type for single-output cameras.
• Avoids redundant camera-buffer, observation concatenation, and task-level copies.
• Migrates Kuka Allegro and several contributed camera configurations to the optimized path.

Confidence Score: 4/5

The PR should not merge until the Kuka Allegro camera migration preserves the established RGB and depth observation semantics or explicitly migrates the policy contract.

The shared image path applies per-frame mean subtraction to RGB and leaves finite depth in raw distance units, whereas the removed Kuka term used fixed RGB centering and bounded tanh depth values consumed directly by its CNN.

Files Needing Attention: source/isaaclab_tasks/isaaclab_tasks/core/lift/config/kuka_allegro/camera_cfg.py; source/isaaclab_tasks/isaaclab_tasks/core/lift/mdp/observations.py

Important Files Changed

Filename	Overview
source/isaaclab/isaaclab/envs/mdp/observations.py	Extends the shared image term with output inference, zero-copy reads, cloning control, and fused layout conversion.
source/isaaclab/isaaclab/utils/images.py	Adds output-layout conversion while preserving the shared utility's existing datatype-specific normalization.
source/isaaclab/isaaclab/utils/warp/kernels.py	Updates the uint8 normalization kernel to write either BHWC or BCHW output directly.
source/isaaclab/isaaclab/utils/warp/ops.py	Validates and allocates the requested output layout before launching the fused Warp kernel.
source/isaaclab/isaaclab/managers/observation_manager.py	Returns an already-isolated single observation term directly instead of concatenating it.
source/isaaclab/isaaclab/sensors/camera/camera.py	Avoids a device-to-host mask check for cameras configured to update every simulation step.
source/isaaclab_tasks/isaaclab_tasks/core/lift/config/kuka_allegro/camera_cfg.py	Migrates Kuka camera observations to the shared image term but changes the policy-visible RGB and depth normalization semantics.
source/isaaclab_tasks/isaaclab_tasks/core/lift/mdp/observations.py	Removes the task-specific camera term and its stationary RGB and bounded depth transformations.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  R[Renderer] --> C[Camera ProxyArray]
  C --> I[Shared image MDP term]
  I --> N[Normalize and convert layout]
  N --> O[Observation manager]
  O --> P[Policy CNN]

Loading

_{Reviews (1): Last reviewed commit: "Optimize shared camera image observation..." | Re-trigger Greptile}

[greptile-apps]

Camera normalization contract changes

When a Kuka Allegro task selects an RGB or depth camera preset, the shared mdp.image term replaces the previous fixed RGB centering and bounded depth transformation with per-frame RGB mean subtraction or raw finite depth values. The policy CNN consumes these values directly, so existing policies and checkpoints receive a different observation distribution and produce incorrect behavior.

Knowledge Base Used:

• Manager-based environment orchestration
• Task library

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[isaaclab-review-bot]

Isaac Lab Review Bot

The optimization direction is sound, but the Kuka Allegro migration changes established camera normalization semantics, removes the exported vision_camera API without deprecation, and breaks positional callers of normalize_image_uint8 by inserting a parameter before out. These compatibility issues should be resolved before merge.

• Design and architecture: Consolidating task-specific camera observations onto the shared image MDP term reduces duplicated work, but the shared term does not preserve the stationary RGB/depth normalization and NaN handling required by the migrated Kuka Allegro path. The migration therefore changes model inputs rather than serving as a behavior-preserving consolidation.
• API: Deleting the publicly exported isaaclab_tasks.core.lift.mdp.vision_camera violates the repository’s deprecation requirement and lacks migration guidance. Additionally, inserting output_channel_dim before the existing out parameter in normalize_image_uint8 rebinds valid positional calls; the new parameter should be appended or made keyword-only.
• Implementation: The fused Warp layout conversion and single-term observation shortcut are supported by the shown implementation and tests. However, replacing vision_camera with mdp.image changes RGB normalization from x/255 - 0.5 to per-image mean subtraction, changes depth normalization from tanh(depth/2) - 0.5 to infinity replacement only, and no longer performs the same NaN sanitization. The prior transforms should be preserved or introduced through an explicit compatibility path.

Significant concerns. Posted 3 actionable findings inline.

Automated review; human maintainers own approval decisions.

[isaaclab-review-bot]

🟡 Warning · Api — Removed public vision_camera lacks deprecation path

vision_camera is exported from isaaclab_tasks.core.lift.mdp (__all__ and __init__.pyi) and is deleted outright, so downstream configs importing it now fail with AttributeError. Repository rules require a prior deprecation and migration path, and changelog fragments must mark breaking changes with migration guidance; this fragment mentions only reuse of the shared term. Keep a deprecated alias or document the removal and replacement.

[isaaclab-review-bot]

🟡 Warning · Api — Kuka camera normalization semantics silently changed

The replaced vision_camera applied nan_to_num_ plus stationary scaling (x/255 - 0.5, and tanh(d/2) - 0.5 for the depth/raycaster presets). mdp.image with normalize=True instead subtracts a per-frame spatial mean for RGB and only maps inf -> 0 for depth, leaving NaNs and unbounded metres. Existing Isaac-Lift-KukaAllegro-Camera configs and checkpoints therefore see a different observation distribution. Preserve the prior transforms or make the break explicit.

[isaaclab-review-bot]

🔵 Suggestion · Api — New parameter inserted before existing out argument

normalize_image_uint8 previously had the signature (src, channel_dim=-1, out=None). Placing output_channel_dim third rebinds an existing positional call such as normalize_image_uint8(src, 1, out_buf), so the tensor is treated as a channel index and fails during resolution. Appending the new parameter after out, or making it keyword-only, preserves the existing positional contract.