feat(slurm): local-disk tiering for salloc targets; retire the all-on-/local layout - #11
Merged
Merged
Conversation
…-/local layout Second half of step (2) in docs/local-disk-tiering.org. slurm.local_disk (and --local-disk / --local-disk-root) now mean the same thing on an unconfined (salloc) target as on a confined one: the mirror root stays on the shared filesystem and the agent works in a clone under <root>/job<ID>/mirrors/<token> on the compute node, published back by the post-commit hook and snapshotted by the deadline timer. - cli._build_executor resolves local disk BEFORE _ensure_slurm_node and passes the root down so _start_slurm_timer renders the clone's literal path as the timer's snapshot dir (snapshot_wip skips it until the clone exists). remote_mirror_root is always the shared root; the /local/mirrors branch, the node-changed stale-root logic, and the compute-node scaffolding routing are gone. local_disk_root goes on the executor for both modes. - mirror.launch_agent (remote, unconfined) stages and runs the same prepare script the confined batch body uses, via a new _prepare_local_tier helper, then starts tmux with cwd = the clone and the cache exports prepended to the window command. A non-zero prepare exit raises MirrorError carrying the script's SUCODER: lines. - _ensure_slurm_node no longer carries a dead job's node as --nodelist affinity (the only reason was data on that node's /local); explicit --node still works. The "unpulled work may be orphaned" fallback message is gone with it. - _start_slurm_timer writes a per-mirror slurm-timer-<token>.sh and pkill -f '[s]lurm-timer-<token>.sh' before nohup, so the timer that every attach/pull/status restarts no longer piles up. - A session that still records a /local/mirrors root from the retired layout gets one warning naming the root and node: commits made there were never published to the shared mirror, so push them from that node before trusting `sucoder pull`. The session then records the shared root. - _forget_allocation drops its /local special case; --local-disk help, session.py, and stale comments updated. README, config.example.yaml, connection-faq.org, and the design note's status updated. Not verified here: a real salloc launch (this session is inside a confined job). The prepare script and timer are the files the confined smoke test already ran on n0036.savio4. Tests: unconfined launch clones then launches in the clone (staged file, `bash <path>` before tmux, cwd, exports, exec bash -l kept); prepare failure raises; launch without tiering unchanged; _build_executor unconfined tiering kwargs and allocation kwargs; retired-layout warning once with the old node; _start_slurm_timer snapshot dir + per-mirror file + pkill. The two test_remote tests that reimplemented the node-changed discard logic inline are removed with that logic. 716 passed CI-style; mypy unchanged. Impact analysis by hand (gitnexus cannot load here, libssl.so.3): _ensure_slurm_node <- _build_executor (+ test spy); _start_slurm_timer <- _ensure_slurm_node (+ guard tests); _forget_allocation local to release; launch_agent's remote branch <- cli collaborate/renew via _build_manager. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01StcQgXQDE4F6eJer1sEXRb
This was referenced Sep 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second half of step (2) in
docs/local-disk-tiering.org(PR B).slurm.local_disk/--local-disk/--local-disk-rootnow mean the same onsalloctargets as on confined ones: mirror root stays shared, the agent works in<root>/job<ID>/mirrors/<name>on the node, published by the post-commit hook and snapshotted by the timer.Changes.
_build_executorresolves local disk before allocation and hands the root to_ensure_slurm_node→_start_slurm_timer(snapshot dir = the clone's literal path).remote_mirror_rootis always the shared root; the/local/mirrorsbranch, node-changed stale-root logic, and compute-node scaffolding routing are removed.launch_agent(remote, unconfined) stages and runs the same prepare script the confined batch body uses (_prepare_local_tier), then starts tmux withcwd= the clone and the cache exports prepended. No more dead-job--nodelistaffinity (explicit--nodestill works). Timer is per-mirror (slurm-timer-<name>.sh) and retires its predecessor withpkill -f '[s]lurm-timer-<name>.sh'— everyattach/pull/statusrestarted it and they piled up. A session still recording a/local/mirrorsroot from the retired layout gets one warning naming that node (commits there were never published).Not verified here: a real
salloclaunch — this session runs inside a confined job. The prepare script and timer are the files already smoke-tested on n0036.savio4 under PR #10. First real run:sucoder --local-disk -T savio-node collaborate <mirror>, then check~/.cache/sucoder/on the node forlocal-tier-<name>.shandslurm-deadline-<name>.warn, andgit -C ~/mirrors/<name> show-ref refs/sucoder/wip/<name>after ten minutes of dirty work.Known gap (step 3):
_maybe_run_poetry_auto_install/_maybe_suggest_mcp_serversstill act on the shared mirror path, not the clone.Tests. 716 passed CI-style; mypy unchanged. The two
test_remotetests that reimplemented the node-changed discard logic inline are removed with it.🤖 Generated with Claude Code
https://claude.ai/code/session_01StcQgXQDE4F6eJer1sEXRb