feat(slurm): local-disk tiering for confined targets - #10
Merged
Merged
Conversation
Step (2) of docs/local-disk-tiering.org, for confined (sbatch) targets.
With slurm.local_disk set (true = /local, or a path), the batch body runs
a staged prepare script that clones the shared mirror to
<root>/job$SLURM_JOB_ID/mirrors/<token>, installs a post-commit hook that
publishes every commit back to the shared mirror (never forced; a push
refused because the laptop pushed first is reported for the agent to
git pull --ff-only), restores the last WIP snapshot when its parent is
the branch tip and the tree is clean, and starts the agent in the clone
with UV_CACHE_DIR / PIP_CACHE_DIR / npm_config_cache / TMPDIR under
<root>/job$SLURM_JOB_ID. The deadline timer snapshots the clone (runtime
path, via a new snapshot_dir_shell parameter) instead of the shared
mirror. The mirror root stays on the shared filesystem throughout: it is
the laptop's push/pull target and the staging area for prelude and batch
script, so _sync_remote / _pull_from_remote / scaffolding are untouched.
Slurm's epilog wipes <root>/job<ID>; nothing is orphaned and --node
pinning is unnecessary.
New module sucoder/local_tier.py: path helpers with literal and runtime
(${SLURM_JOB_ID}) flavours so the batch body, timer, and prepare script
agree on paths without passing them around; the prepare script is
executed, not sourced, so its early exits cannot take the batch body with
it. It warns loudly when the shared mirror has uncommitted tracked
changes (updateInstead would refuse every publish), when the clone and
mirror have diverged, and when a foreign post-commit hook is left alone.
Config: slurm.local_disk now also accepts true/false (README already
showed local_disk: true, which the parser rejected). RemoteExecutor
gains local_disk_root; cli._build_executor sets it for confined targets
(config or --local-disk override) while keeping remote_mirror_root
shared. Unconfined (salloc) targets keep the legacy all-on-/local layout
for now; README says so.
Smoke-tested on n0036.savio4 (job 38661192) under the batch job's
environment: prepare 4.7s (4 MB .git), a commit in the clone published to
the shared mirror by the hook without moving the mirror's checkout, the
dirty-mirror warning fired (the mirror held these very edits), and the
timer with a 1-minute cadence put refs/sucoder/wip/SuCoder on the shared
mirror with the dirty files and parent = the clone's HEAD. A fresh
sbatch launch could not be exercised from inside a job.
Tests: tests/test_local_tier.py drives the prepare script under bash
against a real temporary shared mirror (fresh clone + hook publishes;
idempotent re-run keeps dirty files and fast-forwards; WIP restored only
onto its parent, kept afterwards; foreign hook left alone; dirty mirror
warns; missing/detached mirror fails loudly). Batch-script, confined
launch (three staged files before sbatch, runtime snapshot path), config,
timer, and executor-kwargs tests added. 707 passed CI-style
(GITHUB_ACTIONS=true, lock-matched versions); mypy unchanged (new module
clean).
Impact analysis by hand (gitnexus cannot load here, libssl.so.3):
_build_batch_script <- _launch_confined + tests/test_batch_script.py;
build_timer_script <- cli._start_slurm_timer, mirror._launch_confined;
_parse_slurm_config <- _parse_targets; RemoteExecutor <- cli._build_executor.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01StcQgXQDE4F6eJer1sEXRb
…oc example indent - _launch_confined waits up to 100 x 3s (was 20 x 3s) for the tmux session when tiering is on: the batch body now clones the mirror before new-session, and a fat mirror on slow Lustre can take minutes. - The prepare script refuses to `rm -rf` a work path that is not of the form <root>/job<ID>/mirrors/<token>, documenting the invariant at the only destructive step. - docs/persistent-presence.org example had system_prompt_extra indented under slurm:, the exact misplacement config.py warns about. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01StcQgXQDE4F6eJer1sEXRb
The global --local-disk/--no-local-disk flag predates tiering; pin that on a confined target with no slurm.local_disk in config it now resolves to the default /local root while the mirror root stays shared. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01StcQgXQDE4F6eJer1sEXRb
A global option beside --local-disk: picks the node-local root instead of slurm.local_disk's path or /local. Giving a root implies --local-disk; combining it with --no-local-disk is rejected as a BadParameter. Precedence: --local-disk-root > slurm.local_disk > /local. The value is normalised (trailing slash stripped) and travels on ctx.obj["local_disk_root"]; _build_executor now also falls back to the context for both overrides, so callers that hand over only cli_ctx (e.g. `list models`) see them too. Tests: root alone enables tiering with that root; CLI root beats the config root; --no-local-disk + --local-disk-root exits non-zero with the hint; the parsed value reaches _build_executor's context. 712 passed CI-style; mypy unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01StcQgXQDE4F6eJer1sEXRb
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step (2) of
docs/local-disk-tiering.org, for confined (sbatch) targets. Unconfined (salloc) targets keep the legacy all-on-/locallayout for now (README says so); that is the next PR.What it does. With
slurm.local_disk(true=/local, or a path) on a confined target, the batch body runs a staged prepare script (sucoder/local_tier.py) that clones the shared mirror to<root>/job$SLURM_JOB_ID/mirrors/<name>, installs apost-commithook publishing every commit back to the shared mirror (never forced), restores the last WIP snapshot when its parent is the branch tip, and starts the agent in the clone withUV_CACHE_DIR/PIP_CACHE_DIR/npm_config_cache/TMPDIRon local disk. The deadline timer snapshots the clone (runtime path via a newsnapshot_dir_shell). The mirror root stays shared: laptop push/pull, prelude/batch staging, and scaffolding are untouched. SLURM's epilog cleans up.Config.
slurm.local_disknow acceptstrue/falsetoo (the README already showedtrue; the parser rejected it).RemoteExecutor.local_disk_rootcarries the resolved choice for confined targets.Smoke test on n0036.savio4 under the batch job's environment: prepare 4.7 s; a commit in the clone landed on the shared mirror via the hook without moving its checkout; the dirty-mirror warning fired correctly; the timer put
refs/sucoder/wip/<name>on the shared mirror with the dirty files. A freshsbatchlaunch cannot be exercised from inside a job — first real run issucoder -T carleton-htc collaborate SuCoderwithlocal_disk: truein the target.Tests.
tests/test_local_tier.pydrives the prepare script under bash against a real temporary shared mirror (7 scenarios); batch-script, confined-launch, config, timer, and executor-kwargs tests added. 707 passed CI-style; mypy unchanged.Impact analysis by hand (gitnexus cannot load here):
_build_batch_script<-_launch_confined(+ tests);build_timer_script<-_start_slurm_timer,_launch_confined;RemoteExecutor<-_build_executor.🤖 Generated with Claude Code
https://claude.ai/code/session_01StcQgXQDE4F6eJer1sEXRb