Skip to content

Re-read shards before acting; locks never ask you to judge their holder - #6

Merged
vmogilev merged 3 commits into
mainfrom
fix/two-instance-safety
Oct 28, 2026
Merged

vmogilev merged 3 commits into
mainfrom
fix/two-instance-safety

Conversation

@vmogilev

Copy link
Copy Markdown
Member

What and why

A rolling restart had two ways to act on a fleet it no longer understood. This PR closes both, and fixes a third problem found along the way.

  1. A stale plan could restart a live primary. A plan is a snapshot taken at startup. If a second copy of the roll ran, or something failed a shard over mid-roll, a worker could take the shard lock and restart the new primary because its plan still said "replica". Shard locks made the two copies take turns; they didn't make the second one correct.
  2. The lock asked the operator to judge its holder. lock is held ... remove <file> if stale hands the riskiest call in the roll to whoever just lost their SSH session and believes the old one is dead. If it isn't, there are now two rolls.
  3. A hangup killed the roll mid-operation. SIGHUP wasn't handled. A dropped session killed hammerdown on the spot, possibly mid-failover, and left both locks behind: exactly the stale-looking lock that (2) invites you to delete.

No issue: found while auditing hammerdown against the two-instance failure mode.

Changes

Re-read before you act (dbb54cc)

  • Batch.Refresher re-reads each shard after taking its shard lock. If any member's role, state or running build differs from the plan, the shard is skipped untouched with roll.ErrStale (skipped untouched; rerun to replan).
  • Job.CheckRoles(ctx) re-reads roles right before every restart and the failover in LeaderAware. A replica that became a primary is never restarted. This restores a per-node check from an earlier implementation of this tool that was lost in the port.
  • New optional fleet.ShardInventory (ShardNodes) for cheap one-shard reads. roll.InventoryRefresher falls back to a full listing. The sim implements it, and the CLI wires the refresher in for every backend.

Locks never ask you to judge their holder (5f62d78)

  • lock.Dir checks the holder itself:

    • Same host: kill(pid, 0) reports running or not running.
    • Another host: unknown. The lock file's mtime is a heartbeat refreshed by every Validate.
  • Lock returns *lock.HeldError (wraps ErrHeld) carrying the holder. The message no longer suggests removing anything.

  • New lock.Breaker (Holder, Break), implemented by Dir. Break refuses in any of these cases:

    • the holder is running
    • the lock changed hands since it was inspected
    • no reason was given

    Every release is appended to unlock.log.

  • New hammerdown unlock. On its own it only inspects. With --reason it releases a lock only when the holder is provably gone:

    • a dead pid on this host, or
    • a holder on another host with a stale heartbeat, plus --confirm-host <that host>

    There is no --force.

  • roll explains a held lock: running → reattach, Ctrl-C there, or wait; dead → the exact unlock command; other host → check there first.

  • SIGHUP now drains like the first Ctrl-C and releases the locks. SIGPIPE is ignored, so a dead terminal can't kill the process either.

Docs (12295a7)

  • README: two new safety rules, plus what to do about "lock is held".
  • CLAUDE.md: invariants 7 (re-read before you act) and 8 (a lock never asks the operator to judge its holder).
  • docs/extending.md and the add-backend skill: implement ShardNodes, back the process lock with a lease, and test that a stale plan is refused.

How I tested it

New tests (all under -race):

Test Proves
TestStalePlanWithoutRefresherRestartsPrimaries Control: reproduces the bug. With a copy-on-read fake topo, a stale second instance restarts every primary the first one promoted, even with shard locks working.
TestStalePlanIsSkippedUnderShardLock With the re-read, all 3 stale shards are skipped with ErrStale, zero node operations, and nothing counted as a node failure.
TestReplicaBecamePrimaryMidShard A replica promoted behind the roll's back mid-shard is not restarted, and the error says was replica, now primary.
TestRefresherIsQuietWhenNothingChanged No false positives on an unchanged fleet.
TestDirHeldByRunningHolder A held lock reports running, never says "remove", and Break refuses it.
TestDirBreakDeadHolder / RefusesChangedHands / OtherHostIsUnknown / ValidateHeartbeats Liveness, token check, audit log, heartbeat.
TestMayBreak, TestCLIUnlock Every unlock rule, end to end through the CLI.

By hand, with the shell backend and real processes:

  • A lock held by a live pid: roll refuses and explains, and unlock --reason ... refuses. Once the pid exits, unlock releases the lock and writes unlock.log.
  • kill -HUP on a live roll: the in-flight restart finished, the failover never started, both locks were released, and the process exited cleanly. Before this change the process died and both lock files stayed behind.

Regression and performance:

  • make demo, then a fresh full roll: 2,829 shards, 11,000 nodes, 0 failures, 0 stale shards, about 11 s with --workers 64. That's almost all simulated restart latency; CPU time was under 1 s, so the extra reads cost nothing measurable.
  • GOOS=windows go build ./lock builds (the pid probe is //go:build unix; elsewhere liveness is reported as unknown, never dead).

Checklist

  • make fmt lint test passes (golangci-lint 2.5.0 locally; CI will confirm against its pinned version)
  • Dry run still touches nothing; node operations check job.Stopped() (see CLAUDE.md invariants). The new shard re-read also runs in dry runs: it only reads, takes no locks and makes no NodeOps calls.
  • No new dependencies in the core packages. Standard library only (syscall for the pid probe).
  • Changed a default (workers, fail-fast, dry-run)? No flag defaults changed. Two behavior changes are explained above: every roll now re-reads shards before acting, and SIGHUP now drains instead of killing the process.
  • Docs updated (README / docs/extending.md / flag help) if behaviour changed
  • Added myself to AUTHORS (first contribution): n/a, maintainer

For reviewers: judgment calls worth a second look

  • Stale shards don't count toward fail-fast. They make the run exit non-zero and show up under SHARD failures, but they don't drain the batch. An automatic failover elsewhere shouldn't stop a whole roll; skipping the one shard is the safe response.
  • Liveness is same-host only. Across hosts, Dir reports unknown and leans on the heartbeat plus --confirm-host. A reused pid reads as running, which is the safe direction. Production backends should prefer a lease that expires with its holder.
  • No --force on unlock, deliberately. The escape hatch is the lock file itself, which is the step this PR stops recommending. If we ever need one, it should require --reason and land in unlock.log.
  • Not in this PR: a default blackout window. --blackout is still opt-in.

🤖 Generated with Claude Code

https://claude.ai/code/session_01A29B3qtLVy4Eo9f9zxH5Eo

vmogilev and others added 3 commits October 27, 2026 22:12
A plan is a snapshot. Two hammerdown processes each hold their own, and
locks alone don't make the second one correct: by the time it takes a
shard lock, the first may have failed that shard over, and the stale
instance then restarts the new primary as if it were a replica.

- Batch.Refresher re-reads the shard after taking its shard lock (and in
  dry runs) and skips it untouched with ErrStale if any member's role,
  state or running build differs from the plan.
- LeaderAware calls Job.CheckRoles right before every restart and the
  failover, restoring the per-node "refresh metadata and check this isn't
  a primary" step from an earlier implementation of this tool that was lost in
  the port.
- fleet.ShardInventory lets an inventory read one shard cheaply;
  roll.InventoryRefresher falls back to a full listing. The sim
  implements it; the CLI wires the refresher in for every backend.

Tests reproduce the two-instance scenario with a copy-on-read fake topo:
a control test shows the stale instance restarting live primaries without
the re-read, and the fixed path skips every stale shard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A29B3qtLVy4Eo9f9zxH5Eo
The Dir lock answered "lock is held" with "remove <file> if stale",
handing the riskiest call to the person least able to make it: someone
whose SSH session just dropped, looking at a lock held by a process they
believe is dead. That is how two copies of the roll end up running.

- Dir checks the holder itself: same host -> kill(pid, 0) says running
  or not running; another host -> unknown. The lock file's mtime is a
  heartbeat refreshed by every Validate. Lock returns a *HeldError
  (wraps ErrHeld) carrying the holder; the message no longer suggests
  removing anything.
- New lock.Breaker (Holder, Break), implemented by Dir: Break refuses a
  running holder, refuses if the lock changed hands since inspection,
  requires a reason, and appends every release to unlock.log.
- New `hammerdown unlock`: inspects by default; releases only a provably
  gone holder (dead pid on this host, or another host with a stale
  heartbeat plus --confirm-host <that host>). No --force.
- `roll` explains a held lock: running -> reattach/Ctrl-C/wait; dead ->
  the exact unlock command; other host -> check there first.
- SIGHUP (the SSH session drops) now drains and releases the locks
  instead of killing the process mid-roll and leaving them behind.
  SIGPIPE is ignored so a dead terminal can't kill it either.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A29B3qtLVy4Eo9f9zxH5Eo
README: two new rules in the safety list, what to do about "lock is
held" (`hammerdown unlock`, no --force), tmux/screen for long rolls.
CLAUDE.md: invariants 7 and 8, map entries for refresh.go and unlock.go.
docs/extending.md and the add-backend skill: implement ShardNodes, back
the process lock with a lease, never tell the operator to delete a lock,
and test that a stale plan is refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A29B3qtLVy4Eo9f9zxH5Eo
@vmogilev
vmogilev merged commit 93dc6bb into main Oct 28, 2026
4 checks passed
@vmogilev
vmogilev deleted the fix/two-instance-safety branch October 28, 2026 05:38
@vmogilev vmogilev changed the title Fix/two instance safety Re-read shards before acting; locks never ask you to judge their holder Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant