Skip to content

fix(apps): bind volume mounts from RUNPOD_MOUNTS when the runtime did not - #626

Open
justinwlin wants to merge 3 commits into
mainfrom
fix/volume-path-self-configure
Open

justinwlin wants to merge 3 commits into
mainfrom
fix/volume-path-self-configure

Conversation

@justinwlin

@justinwlin justinwlin commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Flash → SDK 2.0 fix stack

PRs 1–9 are independent PRs off main, and 10 is stacked on 9. Merge in this order. Hover a link for its status.

# PR Fixes Needs
1 #627 ✅ merged: CI e2e fixed on main —
2 #623 GPU crash loop: torch pulled in by peft/accelerate gets bundled nothing
3 #624 Workers land on GPUs you didn't request (whole pools) nothing
4 #625 rp flash dev leaks endpoints when stopped by a signal nothing
5 👉 #626 (this PR) vol.path says "not mounted" on deployed workers nothing
6 #628 Account API key copied into every endpoint nothing
7 — Release runpod 2.0.0rc2 with 1–6 1–6 merged
8 runpod-workers/sdk-runtime#1 Rebuild runtime images (dev sessions run an old SDK) independent
9 runpod/runpod-plugins-official#89 Plugin docs for the fixed SDK (flash lane → SDK 2.0) 2.0.0rc2 on PyPI
10 runpod/runpod-plugins-official#90 Plugin VolumeCache guidance: global volumes, measured numbers, golden paths 08/09 (stacked on #89) runpod/runpod-plugins-official#89

In deployed apps, NetworkVolume.path raised volume '...' is not mounted in the current execution even though the volume was attached at /runpod-volume.

Cause

vol.path reads bindings that the worker runtime installs by calling _configure_mounts before user code (sdk-runtime mounts.py, added 2026-09-29). The published runtime images (runpod/queue:py3.12-latest and the rest) were pushed on 2026-09-22 and don't contain that hook. docker run against the image shows no mounts.py, and its queue/worker.py never calls configure_mounts. The bindings are never installed.

Change

Deployed workers run the client's vendored runpod (unless RUNPOD_PACKAGE_SPEC overrides the package), so the first vol.path lookup now installs the bindings from the RUNPOD_MOUNTS provisioning env itself when the runtime hasn't. A runtime that configured mounts still takes precedence. Invalid JSON raises VolumeError.

This removes the SDK's dependence on the image for vol.path. The images still need a rebuild for dev sessions, which run on the image's own runpod.

Testing

CPU queue writing through vol.path / "probe.txt" main runpod-flash 1.21 This PR
Deployed on today's runtime images VolumeError: volume '...' is not mounted in the current execution no vol.path; apps used the literal /runpod-volume {'path': '/runpod-volume', 'read': 'ok'}
How we tested and compared
  • Root cause. docker run runpod/queue:py3.12-latest shows no mounts.py, and its queue/worker.py never calls configure_mounts. The images predate the hook.
  • Live, before and after. The same app, with a network volume attached, was deployed from main (raised the error above) and from this branch (returned the path). The app and volume were deleted afterward.
  • Unit tests cover binding from RUNPOD_MOUNTS when the runtime did not, the runtime's configuration winning, and invalid JSON. pytest tests: 1482 passed, coverage 91.66%.
Review (Codex, two rounds)
Finding Resolution
The flag was set before the bindings were published, so a concurrent first lookup could read empty bindings Fixed in dee2d20 by publishing the bindings first.
"Deployed workers run the client's vendored runpod" has an exception: RUNPOD_PACKAGE_SPEC skips the overlay Description corrected below.
No concurrent-lookup test, and no automated old-image worker test Open. Old-image behavior was checked live.

Comment thread tests/test_apps/test_volume.py Fixed
Comment thread tests/test_apps/test_volume.py Fixed
Comment thread tests/test_apps/test_volume.py Fixed
Comment thread tests/test_apps/test_volume.py Fixed
… not

vol.path reads bindings the worker runtime installs through
_configure_mounts before user code. Runtime images published before
that hook never call it, so on them every deployed NetworkVolume
attached but vol.path raised "not mounted in the current execution".

Deployed workers run the client's vendored runpod package, so the
first vol.path lookup now installs the bindings from the RUNPOD_MOUNTS
provisioning env itself when the runtime has not. A runtime that
configured mounts still takes precedence.
@justinwlin
justinwlin force-pushed the fix/volume-path-self-configure branch from 317e3fd to dee2d20 Compare October 20, 2026 15:48
@justinwlin
justinwlin marked this pull request as ready for review October 20, 2026 15:48

import pytest

import runpod.apps.volume as volume_mod

import pytest

import runpod.apps.volume as volume_mod

@KAJdev KAJdev left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants