fix: harden startup and Slurm timer supervision

.coder/ledger.md

@@ -0,0 +1,69 @@
+# Prior-Art Ledger - Slurm timer integration
+
+Search tier: GitNexus query/context/impact plus source and test inspection.
+Baseline: PR #13 (`2c24354`) and local main (`258accb`).
+
+## 1. Task
+
+Integrate PR #13's deadline warnings and regression coverage with main's
+timer supervision. Keep monitoring through scheduler errors.
+
+## 2. Existing machinery
+
+| Machinery | Source at baseline | Tests | Decision |
+|---|---|---|---|
+| SSH timer staging and readiness | `sucoder/cli.py:1129` on main | `tests/test_timer_lifecycle.py::test_timer_ssh_timeout_is_advisory` | Reuse main |
+| Allocation-scoped locks and readiness handshake | `sucoder/timer_lifecycle.py:3` on main | `test_timer_survives_starter_and_reuses_owner`, `test_concurrent_starters_create_one_timer`, `test_missing_executable_reports_failure` | Reuse main |
+| Confined timer launch | `sucoder/mirror.py:2391` on main | `tests/test_batch_script.py`, `tests/test_mirror.py::test_launch_confined_stages_and_starts_deadline_timer` | Keep `bash --ensure`; extend atomic staging |
+| Shared warning loop and builder | `sucoder/slurm_timer.py:102` on PR #13 | `tests/test_slurm_timer_script.py` warning-chain tests | Extend scheduler error handling |
+| Snapshot implementation | `sucoder/slurm_timer.py:65` on PR #13 | Real repository-pair tests and `test_watchdog_warns_and_snapshots_without_changing_index` | Reuse |
+| Scheduler observation contract | `sucoder/cli.py:765` on PR #13 | Slurm state-query tests in `tests/test_cli.py` | Reuse distinction between errors and empty successful queries |
+
+## 3. Definitions and conventions
+
+- Scheduler errors are unknown state: "An ssh/squeue failure is NOT evidence
+  the job is dead" (`sucoder/cli.py:765`, PR #13).
+- Successful empty queries indicate disappearance; retain PR #13's three
+  consecutive observations before ending the watchdog.
+- `snapshot_minutes` is a periodic cadence; zero disables periodic snapshots,
+  while threshold snapshots remain enabled (`build_timer_script` docstring).
+- `--ensure` reports `SUCODER_TIMER_STARTED` or `SUCODER_TIMER_REUSED`; a
+  successful SSH return alone does not establish readiness.
+
+## 4. Invariants
+
+- Preserve main's target/node/allocation locks, immutable SSH script names,
+  readiness diagnostics, and advisory timer failures. Do not restore `pkill`.
+- Never cancel the user's allocation from the timer.
+- Preserve the agent's real Git index and existing snapshot implementation.
+- Warning urgency only increases; skipped thresholds must not fire later.
+- Scheduler failures must not stop monitoring or suppress periodic snapshots.
+- Keep unrelated startup safety fixes from `258accb` intact.
+
+## 5. Reuse decisions
+
+- Reuse main's supervision rather than implementing another startup check.
+- Extend the existing warning loop and bash test driver to distinguish failed
+  queries from successful empty queries, including recovery and snapshot tests.
+- Retain PR #13's confined atomic staging, warning order, lifecycle wording,
+  and mutation-derived regression tests; adapt assertions to supervision.
+- Keep compatibility warning files cleared when a new watchdog starts.
+
+Verification: OK against sections 3-5. `python -m pytest -q` passed all 759
+tests on Python 3.12. Four scheduler-error regression cases failed before the
+loop fix. A real-file staging test checks the open inode and quoted paths.
+The SSH supervisor, lifecycle module, remote bootstrap, and their regression
+tests are byte-for-byte unchanged from `258accb`. GitNexus's comparison with
+main reports the expected confined-launch and warning-loop scope.
+
+Reproduce from the repository root with `pytest -q`; the focused checks are
+`pytest -q tests/test_slurm_timer_script.py tests/test_timer_lifecycle.py
+tests/test_batch_script.py tests/test_cli.py tests/test_mirror.py`.
+
+## 6. Open questions
+
+No implementation decision is blocked. Cluster smoke testing and issue #15's
+batch-environment Git availability need a real allocation. Issues #14 and #16
+remain outside this integration.
+
+Prepared by Sue (2026-09-12).

.sucoder/handoffs/2026-08-25-savio-harness-bootstrap.org

@@ -0,0 +1,191 @@
+#+title: Savio user-level harness bootstrap
+#+date: 2026-08-25
+#+status: READY
+
+* Mission
+
+Install and validate SuCoder's supported harnesses in the Savio account
+=ligon= without root access.  Work only in user-owned locations under
+=$HOME=, preserve existing configuration, and leave a reproducible report in
+this file.
+
+This is an empirical bootstrap pass.  Do not redesign SuCoder unless a real
+cluster-side incompatibility requires it; record such an incompatibility for a
+follow-up instead.
+
+* Important architecture
+
+- Remote SuCoder sessions intentionally run as the SSH user =ligon=.  There is
+  no remote =coder= account and no remote sudo boundary.  SSH/SLURM and the
+  remote mirror provide the isolation boundary.
+- SuCoder itself runs on the laptop.  It does not need to be installed on
+  Savio.  Only harness binaries and their non-secret user configuration belong
+  on the cluster.
+- Do *not* copy =~/.sucoder/config.yaml=, the password store, GPG keys, or an
+  OpenRouter API key to Savio.  The laptop-side SuCoder process reads the key
+  from =pass= and stages it transiently into the remote harness environment.
+- Never print an API key, inspect its value, put it in argv, or write it to a
+  cluster configuration file.
+- =$HOME= is shared across Savio login and compute nodes.  Install there, not
+  in a node-local directory.
+
+* Entry checks and stop conditions
+
+First read =AGENTS.md= and this entire handoff.  Then collect:
+
+#+begin_src sh
+date
+hostname -f
+printf 'SLURM_JOB_ID=%s\n' "${SLURM_JOB_ID:-}"
+nproc
+free -h
+squeue --me -o '%i %P %T %L %D %C %m'
+bash -lc 'printf "HOME=%s\nPATH=%s\n" "$HOME" "$PATH"'
+#+end_src
+
+If =SLURM_JOB_ID= is empty, this session is on a login node.  Do not run the
+installers there.  Report that the human should relaunch with =-T savio-htc=
+or =-T carleton-htc= and stop.
+
+Before changing anything, inventory the executable path and version of:
+=claude=, =codex=, =aider=, =opencode=, =goose=, =kimi=, =gitnexus=, =uv=,
+=node=, and =npm=.  Also record =npm config get prefix= and the effective uv
+tool directory.
+
+Known state from a login-shell probe on 2026-08-25:
+
+- =$HOME= is =/global/home/users/ligon=.
+- =claude= 2.1.245 is present at =~/.local/bin/claude= and is the only
+  installed harness.
+- =node= 20.20.2 and npm 10.8.2 are under =~/.nvm=; that npm prefix is
+  user-writable.
+- =uv= 0.11.29 is at =~/.local/bin/uv=.
+- =gitnexus= 1.6.3 is installed under the current nvm Node prefix.
+- =~/.sucoder/skills= exists and resolves to the shared skills checkout.
+- The current environment sets =UV_TOOL_DIR=/tmp/uv-tools-ligon=.  This is
+  unsuitable: =/tmp= may be node-local or disposable.  Do not install Aider
+  there.  Find where this setting comes from and propose the smallest safe
+  correction.  Until corrected, override it explicitly with a persistent
+  tool root under =$HOME=.
+
+* Installation policy
+
+- Install the latest *stable* releases from official upstream sources.  Do not
+  install beta/nightly builds.
+- Prefer isolated or native distributions.  Do not install Python tools into
+  system Python 3.6 or a project environment.
+- Keep executable shims in =~/.local/bin= or the existing user-owned nvm
+  prefix.  A fresh =bash -lc= must resolve every harness without manual setup.
+- Download installer scripts from official locations, inspect their relevant
+  install-path behavior before execution, and do not use sudo.
+- Preserve existing config files.  Before editing one, make a timestamped,
+  mode-preserving backup beside it.  Never commit home-directory config or
+  backups to this repository.
+- Do not delete stale installations during this pass.  Report shadowed or
+  duplicate copies and recommend cleanup separately.
+
+Install these harnesses:
+
+1. *Aider*: use uv's isolated tool installation with Python 3.12 and
+   =aider-chat@latest=.  Explicitly set a persistent =UV_TOOL_DIR= under
+   =~/.local/share/uv= and =UV_TOOL_BIN_DIR=$HOME/.local/bin= for the command.
+2. *Codex*: install the stable =@openai/codex= package using the existing
+   user-owned npm prefix, unless the current official standalone installer is
+   materially better on this host.
+3. *OpenCode*: install stable OpenCode 1 (=opencode-ai=, executable
+   =opencode=).  Do not substitute the OpenCode 2 beta executable
+   =opencode2=; SuCoder's current profile targets =opencode=.
+4. *Goose*: install the CLI, not the desktop application, using the official
+   =aaif-goose/goose= stable installer with configuration disabled during
+   install.  Its executable should land in =~/.local/bin=.
+5. *Kimi*: install current stable *Kimi Code CLI* from MoonshotAI's official
+   installer.  Prefer its current native/single-binary distribution rather
+   than adding a Node-version dependency merely for installation.  Confirm
+   that =kimi --help= still provides the flags SuCoder uses: =--agent-file=,
+   =--auto=, =--add-dir=, and model/provider support.
+
+Inspect the existing Claude and GitNexus installations.  Update them only via
+their native/official update path when a newer stable release is actually
+available.  Do not replace a working install merely to make all tools use the
+same package manager.
+
+Official starting points (verify them at execution time):
+
+- Aider: https://aider.chat/docs/install.html
+- Codex: https://github.com/openai/codex
+- OpenCode: https://opencode.ai/docs/
+- Goose: https://github.com/aaif-goose/goose
+- Kimi Code: https://github.com/MoonshotAI/kimi-code
+
+* Provider and harness configuration
+
+Configure only non-secret provider metadata on Savio.  The target model for
+smoke testing is =openrouter/moonshotai/kimi-k3=.
+
+- Aider and OpenCode should consume =OPENROUTER_API_KEY= from the launch
+  environment and accept the provider-prefixed model name.
+- Native Kimi is adapted by SuCoder through temporary =KIMI_MODEL_*= variables;
+  do not persist the key in Kimi's config.
+- Goose natively recognizes =OPENROUTER_API_KEY=, but it may also require
+  non-secret provider/model selection (=openrouter= and
+  =moonshotai/kimi-k3=).  Configure that metadata without a key.  If SuCoder's
+  generic =--model openrouter/...= spelling does not match Goose's CLI, record
+  the exact error as a local SuCoder adapter gap rather than storing a secret
+  to work around it.
+- Codex uses custom model-provider configuration for OpenAI-compatible
+  endpoints.  Add a key-free OpenRouter provider stanza only if current Codex
+  supports sourcing its key from =OPENROUTER_API_KEY=.  If SuCoder's model
+  prefix needs translation for Codex, record the exact required mapping rather
+  than embedding credentials.
+- Leave Claude's existing authentication and configuration intact.
+
+* Verification
+
+After installation, start a fresh login shell and record, for every harness:
+
+1. =command -v= result;
+2. version;
+3. whether the path is under =$HOME= and visible on both the allocated compute
+   node and a login shell;
+4. whether its SuCoder-required flags are present;
+5. whether native Agent Skills, shell execution, file editing, MCP, and
+   subagents are available as expected.
+
+Do not claim provider success from =--version=.  Perform a minimal live
+OpenRouter smoke test only when the launch environment already contains the
+transient key; do not ask the human to paste it into the cluster.  Keep API use
+small.  If this bootstrap Claude session has no OpenRouter environment, leave
+the live tests for the laptop-side relaunch commands below.
+
+The human will validate each completed harness from the laptop with commands
+of this form:
+
+#+begin_src sh
+sucoder -T savio-htc collaborate SuCoder --harness aider --model openrouter/moonshotai/kimi-k3
+sucoder -T savio-htc collaborate SuCoder --harness opencode --model openrouter/moonshotai/kimi-k3
+sucoder -T savio-htc collaborate SuCoder --harness goose --model openrouter/moonshotai/kimi-k3
+sucoder -T savio-htc collaborate SuCoder --harness kimi --model openrouter/moonshotai/kimi-k3
+#+end_src
+
+Test Codex separately after confirming its OpenRouter provider mapping.
+
+* Deliverable
+
+Replace the =Outcome= section below with:
+
+- a table of harness, exact version, resolved path, install/update method, and
+  smoke-test result;
+- any persistent shell/config changes, with backup paths;
+- unresolved SuCoder adapter gaps, including exact reproduction commands and
+  sanitized errors;
+- the exact idempotent commands that should become a future target-bootstrap
+  script;
+- a recommendation about the bad =UV_TOOL_DIR=/tmp/uv-tools-ligon= setting.
+
+Do not record secrets or secret-derived output.  Commit only repository files
+that document or implement the bootstrap; never commit files copied from
+=$HOME=.
+
+* Outcome
+
+Pending remote execution.

README.org

@@ -229,9 +229,11 @@ sucoder status project
   prompt when the two histories have diverged.
 - sync :: an alias for =push=, kept for compatibility.
 
-=agents-clone= is for creating the mirror, not for routine pushing: on a
-remote mirror it will rebuild the repository from scratch if it decides
-the remote copy is a half-initialised husk.
+=agents-clone= creates the mirror. Remote startup initializes absent or
+empty directories and recovers valid repositories in place. It never deletes
+a directory because a probe failed or a repository has no default branch.
+Unreadable, invalid, symlinked, or non-repository nonempty paths stop startup
+with a diagnostic; inspect them before retrying.
 
 ** Mirror safety: the pull must succeed before the push
 =push= (and =agents-clone=) send to the mirror with =git push --all
@@ -243,6 +245,10 @@ push is refused rather than allowed to overwrite unretrieved commits.
 An empty or half-initialised mirror is *not* treated as unreadable, so
 first-time bootstrap is unaffected: sucoder asks the mirror whether it
 holds any commits instead of guessing from the error text.
+Any ref (including feature branches, tags, and WIP refs) counts as content,
+even with an unborn HEAD. A repository initialized by this invocation skips
+the initial fetch and receives a non-forcing first push. The
+=--allow-unverified-mirror= override does not bypass initialization checks.
 
 - --allow-unverified-mirror :: Push anyway, discarding any unpulled
   mirror commits.  Use when the mirror is known to be expendable.
@@ -783,11 +789,31 @@ is the right behaviour for whole-node partitions.
 Every SLURM-backed session (=salloc= or =confined= =sbatch=) starts a
 small watchdog on the compute node.  It warns at 30, 15, and 5 minutes
 before the allocation's =--time= via =tmux display-message= and by
-writing =$HOME/.cache/sucoder/slurm-deadline-<mirror>.warn= (the
-un-suffixed =slurm-deadline.warn= is also written for prompts that
-poll the older path).  It never cancels the job; that stays with
+writing a warning under
+=/tmp/sucoder-<uid>/timers/<scope>/<node>-<job>/= (the
+un-suffixed =slurm-deadline.warn= and per-mirror
+=slurm-deadline-<mirror>.warn= are also written in =$HOME/.cache/sucoder/= for
+older prompts; these compatibility copies show the last writer).
+It never cancels the job; that stays with
 =sucoder release=.
 
+The scope is a digest of the mirror and target names. The timer uses Linux
+=/proc= and =flock= to verify/reuse a live watchdog for that allocation;
+repeated startup does not restart it or reset warning state. Startup reports
+whether it started or reused a timer. Failures explicitly report that
+deadline warnings and periodic snapshots are unavailable; diagnostics remain
+in =timer.log= beside its =owner= and =status= files. Timers from the older
+implementation are not killed by a broad process-name match; they exit with
+their old allocation.
+
+Locks, owner/readiness records, threshold markers, and timer logs live in a
+private, owner-verified mode-700 directory on node-local =/tmp=. This transient
+state only needs to survive for the watchdog lifetime. =TMPDIR= is deliberately
+not used: it may point at shared storage. Staged scripts and compatibility
+warnings stay on NFS; NFS need not support =flock=. Working clones may use
+=/local/job<jobid>/=, with snapshots pushed to durable storage. Lustre is not
+used for the timer's frequent small-file operations.
+
 The same script snapshots the mirror's dirty working tree (tracked and
 untracked files, not ignored ones) to =refs/sucoder/wip/<mirror>= on
 the mirror's =origin= at each warning and every