[Docker] Install the cuRobo image from uv.lock - #7442
Closed
hujc7 wants to merge 25 commits into
Closed
Conversation
Adds a composable teleop-no-isaacsim extra so the lock path selects the teleop requirements without pulling the isaacsim wheel into an image that already ships Kit. Derives the submodule extras from OPTIONAL_SUBMODULE_ROOT_EXTRAS instead of a second hardcoded table, and falls back to pip rather than installing less than was asked for. Pins the uv pip calls to the venv: UV_PYTHON selects the target environment, so they were writing into Isaac Sim's site-packages.
Reverts the install CLI lock path: it also caught Conda environments, where uv sync targets the project environment rather than the active one, breaking the install_ci Conda tests. The images control VIRTUAL_ENV and UV_PROJECT_ENVIRONMENT themselves, so Dockerfile.base syncs the lock directly through a named kit-image extra. imageio-ffmpeg is excluded during the sync instead of uninstalled after.
The extras set gained teleop-no-isaacsim and kit-image. teleop now composes from the former, so asserting on its literal requirement list no longer sees isaaclab-mimic; flatten the isaaclab-dev[...] self-reference first so the contract stays on what the extra resolves to.
run_tests.sh launches the test container with --user ${host_uid}:${host_gid}, so a
venv owned by the image's own user is read-only there and the test-time extras
install (pytetwild for tetrahedralization) fails with EACCES. Widen the mode
instead of chowning to a uid the image cannot know.
The teleop extra now composes through an isaaclab-dev[teleop-no-isaacsim] self-reference. _root_extra_dependencies strips every isaaclab* requirement as a workspace self-reference and isaacsim separately, so the list came back empty and ./isaaclab.sh -i teleop installed none of the teleop stack. Expand root self-references before the workspace filter runs, guarding against a cycle with a seen set.
The test extra pulls the whole Sphinx docs stack, so the images installed a bare pytest and CI layered pytest-mock, junitparser, flaky and coverage on top in a separate post-build docker build. Split those runners into a test-runtime extra that test composes from, and pull it into kit-image so both images carry them from the lock. In-image pytest goes from 238 passed / 23 failed to 257 passed / 4 failed; the 4 remaining fail identically without this change.
Restore the uv cache mount and drop UV_NO_CACHE=1, which disabled the only working build cache. Resolve the third-party packages from workspace manifests above the source copy so a source edit re-runs only the workspace install (240 packages in 6.0s -> 14 in 4ms). Set ownership in the layer that creates the venv instead of a recursive chown afterwards: on overlayfs a chown copies up every file it touches, so the base image stored the 11.5GB venv twice. That layer drops from 1003.9s to 36.1s and the image from 44.7GB to 33.2GB, with 0 of 98562 common files differing in owner or mode. Kit-less sets the modes in its builder stage so the stage copy carries them rather than adding an equivalent layer. Also hoist the layers that depend on no repo content above the copies, give isaaclab.sh an explicit 755 (sed -i rewrote it under the build umask, leaving it unrunnable by other uids), and keep local ruff/pytest caches out of the context.
test_dockerfile_nonroot.py only globs Dockerfile.* and parses their text, but build.yaml ran it inside the image via /isaac-sim/python.sh in both the isaaclab_ov and curobo jobs. Kit's interpreter has no pytest once the install moves to the venv, so those steps failed, and a text lint held a self-hosted GPU runner in two places. Move it to a ubuntu-latest job using the invocation kitless-docker.yml already uses. The steps that verify the running container's uid stay where they are.
Two assumptions broke when the install moved into a venv. The uv path was derived from site.getuserbase() and bootstrapped with pip install uv, which only landed there because Isaac Sim's site-packages was read-only; pip installs into the venv's own bin instead, so $HOME/.local/bin/uv never appeared and record-video died with No such file or directory. Prefer uv from PATH, which the image ships, and keep the user-base bootstrap for images that do not. The [INFO] banner isaaclab.sh writes to stdout now follows the command output rather than preceding it, so tail -n 1 captured the banner for the interpreter, user-site and uv paths. Filter the banner instead of skipping by position.
Isaac Sim ships prebundled copies under pip_prebundle that setup_python_env.sh puts on PYTHONPATH. isaaclab.sh deprioritizes them, but launch paths that never import isaaclab (runheadless.sh, isaac-sim.streaming.sh) do not, and the prebundled torch loads a libtorch_cuda.so needing an NCCL symbol the bundled NCCL does not export (nvbugs 6343978). The pip install path repointed them itself; installing into a venv leaves them untouched, so test_no_shadowing_prebundled_torch_in_isaac_sim failed. Reuse the same repoint after the workspace sync, while the build is still root: 17 packages become symlinks into the venv and the shadowing directory count drops to 0.
The image installed with pip into Isaac Sim's site-packages, which forced three workarounds: deleting the prebundled torch so a fresh one would install, re- bootstrapping pip afterwards because that deletion corrupts its vendored _structures.py, and uninstalling quadprog. Resolving into the venv removes the cause of all three - quadprog is not a lock entry at all - and drops the separate toml/pytest and editable isaaclab_teleop installs, which the sync now provides. cuRobo stays out of the lock so the kit-less and base images do not resolve it; it is installed from its pinned commit afterwards, with build isolation off so its CUDA extensions build against the venv's torch. Carries the base image's caching and ordering: uv cache mounts, third-party resolution above the source copy, ownership set in the creating layer, and the prebundle repoint that keeps Isaac Sim's torch from shadowing the venv. Cold build 1014.4s -> 630.6s; 30 passed / 1 skipped across test_curobo_planner_franka, test_curobo_planner_cube_stack and test_pink_ik.
github-actions
Bot
added
documentation
hujc7
added
ci:run-docker
The images excluded imageio-ffmpeg, so moviepy.editor - which isaaclab/envs/utils/video_recorder.py imports - fell through to imageio's TIFF writer and raised TypeError on write. Video recording only worked because CI installs moviepy into the container at test time. Pull the existing video extra into kit-image and drop the exclusion. Costs 31.6 MB (moviepy, imageio-ffmpeg, proglog) on a 33 GB image; verified in-image that ImageSequenceClip.write_videofile produces a real mp4.
Mirrors the base image: the video extra now comes from kit-image, so the imageio-ffmpeg exclusion would strip the backend moviepy.editor needs.
hujc7
added
ci:run-docker
This reverts commit 8cab65f.
hujc7
added
ci:run-docker
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
cuRobo image cold build 1014.4 s → 630.6 s (−38%) and three long-standing workarounds
disappear, by resolving into the venv instead of Isaac Sim's site-packages.
Description
Dockerfile.curobowas the last CI-built image still installing with pip into Isaac Sim'ssite-packages. That shared prefix forced three workarounds, each caused by the previous one:
rm -rf .../pip_prebundle/torch*get-pip.pyre-bootstrap (+ a network download)torch*corrupts pip's vendored_structures.pypip uninstall -y quadprogquadprogis not auv.lockentry, so it is never installedpip install toml pytestinstall_deps.pyusestomllib; pytest comes from thetest-runtimeextrapip install --editable .../isaaclab_teleopcuRobo itself is deliberately kept out of
uv.lock: the lock is shared with the base andkit-less images, and neither should resolve it. It is installed from its pinned commit after
the sync, with build isolation off so its CUDA extensions build against the environment's torch.
The image also inherits the base image's build changes from #7405 — uv cache mounts,
third-party resolution above the source copy, ownership set in the layer that creates the venv,
and the prebundle repoint that stops Isaac Sim's torch shadowing the venv for launch paths that
never import
isaaclab(nvbugs 6343978).Validation
Built and tested locally on an RTX PRO 6000; CI's exact
test-curoboselection:--no-cache)test_curobo_planner_franka.pytest_curobo_planner_cube_stack.pytest_pink_ik.pytest_no_shadowing_prebundled_torch_in_isaac_simType of change
Release backport
developChecklist
pre-commitchecks with./isaaclab.sh --formattest_curobo_planner_*,test_pink_ik, and the Dockerfile contract tests); no new test was neededsource/<pkg>/changelog.d/for every touched packageCONTRIBUTORS.mdor my name already exists there