Repository navigation
Re-read shards before acting; locks never ask you to judge their holder - #6
Merged
Merged
Conversation
A plan is a snapshot. Two hammerdown processes each hold their own, and locks alone don't make the second one correct: by the time it takes a shard lock, the first may have failed that shard over, and the stale instance then restarts the new primary as if it were a replica. - Batch.Refresher re-reads the shard after taking its shard lock (and in dry runs) and skips it untouched with ErrStale if any member's role, state or running build differs from the plan. - LeaderAware calls Job.CheckRoles right before every restart and the failover, restoring the per-node "refresh metadata and check this isn't a primary" step from an earlier implementation of this tool that was lost in the port. - fleet.ShardInventory lets an inventory read one shard cheaply; roll.InventoryRefresher falls back to a full listing. The sim implements it; the CLI wires the refresher in for every backend. Tests reproduce the two-instance scenario with a copy-on-read fake topo: a control test shows the stale instance restarting live primaries without the re-read, and the fixed path skips every stale shard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A29B3qtLVy4Eo9f9zxH5Eo
The Dir lock answered "lock is held" with "remove <file> if stale", handing the riskiest call to the person least able to make it: someone whose SSH session just dropped, looking at a lock held by a process they believe is dead. That is how two copies of the roll end up running. - Dir checks the holder itself: same host -> kill(pid, 0) says running or not running; another host -> unknown. The lock file's mtime is a heartbeat refreshed by every Validate. Lock returns a *HeldError (wraps ErrHeld) carrying the holder; the message no longer suggests removing anything. - New lock.Breaker (Holder, Break), implemented by Dir: Break refuses a running holder, refuses if the lock changed hands since inspection, requires a reason, and appends every release to unlock.log. - New `hammerdown unlock`: inspects by default; releases only a provably gone holder (dead pid on this host, or another host with a stale heartbeat plus --confirm-host <that host>). No --force. - `roll` explains a held lock: running -> reattach/Ctrl-C/wait; dead -> the exact unlock command; other host -> check there first. - SIGHUP (the SSH session drops) now drains and releases the locks instead of killing the process mid-roll and leaving them behind. SIGPIPE is ignored so a dead terminal can't kill it either. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A29B3qtLVy4Eo9f9zxH5Eo
README: two new rules in the safety list, what to do about "lock is held" (`hammerdown unlock`, no --force), tmux/screen for long rolls. CLAUDE.md: invariants 7 and 8, map entries for refresh.go and unlock.go. docs/extending.md and the add-backend skill: implement ShardNodes, back the process lock with a lease, never tell the operator to delete a lock, and test that a stale plan is refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A29B3qtLVy4Eo9f9zxH5Eo
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
A rolling restart had two ways to act on a fleet it no longer understood. This PR closes both, and fixes a third problem found along the way.
lock is held ... remove <file> if stalehands the riskiest call in the roll to whoever just lost their SSH session and believes the old one is dead. If it isn't, there are now two rolls.SIGHUPwasn't handled. A dropped session killed hammerdown on the spot, possibly mid-failover, and left both locks behind: exactly the stale-looking lock that (2) invites you to delete.No issue: found while auditing hammerdown against the two-instance failure mode.
Changes
Re-read before you act (
dbb54cc)Batch.Refresherre-reads each shard after taking its shard lock. If any member's role, state or running build differs from the plan, the shard is skipped untouched withroll.ErrStale(skipped untouched; rerun to replan).Job.CheckRoles(ctx)re-reads roles right before every restart and the failover inLeaderAware. A replica that became a primary is never restarted. This restores a per-node check from an earlier implementation of this tool that was lost in the port.fleet.ShardInventory(ShardNodes) for cheap one-shard reads.roll.InventoryRefresherfalls back to a full listing. The sim implements it, and the CLI wires the refresher in for every backend.Locks never ask you to judge their holder (
5f62d78)lock.Dirchecks the holder itself:kill(pid, 0)reports running or not running.Validate.Lockreturns*lock.HeldError(wrapsErrHeld) carrying the holder. The message no longer suggests removing anything.New
lock.Breaker(Holder,Break), implemented byDir.Breakrefuses in any of these cases:Every release is appended to
unlock.log.New
hammerdown unlock. On its own it only inspects. With--reasonit releases a lock only when the holder is provably gone:--confirm-host <that host>There is no
--force.rollexplains a held lock: running → reattach, Ctrl-C there, or wait; dead → the exactunlockcommand; other host → check there first.SIGHUPnow drains like the first Ctrl-C and releases the locks.SIGPIPEis ignored, so a dead terminal can't kill the process either.Docs (
12295a7)docs/extending.mdand theadd-backendskill: implementShardNodes, back the process lock with a lease, and test that a stale plan is refused.How I tested it
New tests (all under
-race):TestStalePlanWithoutRefresherRestartsPrimariesTestStalePlanIsSkippedUnderShardLockErrStale, zero node operations, and nothing counted as a node failure.TestReplicaBecamePrimaryMidShardwas replica, now primary.TestRefresherIsQuietWhenNothingChangedTestDirHeldByRunningHolderrunning, never says "remove", andBreakrefuses it.TestDirBreakDeadHolder/RefusesChangedHands/OtherHostIsUnknown/ValidateHeartbeatsTestMayBreak,TestCLIUnlockunlockrule, end to end through the CLI.By hand, with the shell backend and real processes:
rollrefuses and explains, andunlock --reason ...refuses. Once the pid exits,unlockreleases the lock and writesunlock.log.kill -HUPon a live roll: the in-flight restart finished, the failover never started, both locks were released, and the process exited cleanly. Before this change the process died and both lock files stayed behind.Regression and performance:
make demo, then a fresh full roll: 2,829 shards, 11,000 nodes, 0 failures, 0 stale shards, about 11 s with--workers 64. That's almost all simulated restart latency; CPU time was under 1 s, so the extra reads cost nothing measurable.GOOS=windows go build ./lockbuilds (the pid probe is//go:build unix; elsewhere liveness is reported as unknown, never dead).Checklist
make fmt lint testpasses (golangci-lint 2.5.0 locally; CI will confirm against its pinned version)job.Stopped()(see CLAUDE.md invariants). The new shard re-read also runs in dry runs: it only reads, takes no locks and makes no NodeOps calls.syscallfor the pid probe).SIGHUPnow drains instead of killing the process.For reviewers: judgment calls worth a second look
SHARDfailures, but they don't drain the batch. An automatic failover elsewhere shouldn't stop a whole roll; skipping the one shard is the safe response.Dirreports unknown and leans on the heartbeat plus--confirm-host. A reused pid reads as running, which is the safe direction. Production backends should prefer a lease that expires with its holder.--forceonunlock, deliberately. The escape hatch is the lock file itself, which is the step this PR stops recommending. If we ever need one, it should require--reasonand land inunlock.log.--blackoutis still opt-in.🤖 Generated with Claude Code
https://claude.ai/code/session_01A29B3qtLVy4Eo9f9zxH5Eo