Skip to content

cut: three mechanisms the rest of the system already covers - #424

Merged
CMGS merged 7 commits into
mainfrom
cut/overdesign-low
Nov 3, 2026
Merged

CMGS merged 7 commits into
mainfrom
cut/overdesign-low

Conversation

@CMGS

@CMGS CMGS commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Three mechanisms whose justifying scenario came from review, not from an incident, and whose failure the rest of the system already handles. Each was attacked first: read every caller and test, then try to build an interleaving that loses a claim, serves two owners, leaves a free NIC or breaks a documented status code. None held, so they go, one commit each. Net production change is −44 lines.

Commit What goes Net Claim and release cost What changes
2f6387df adoptHeld and the duplicate branch: a held claim answers before reaching Take −19 none, runs only after an AgentSocket miss A late /owner during a release answers 200 where it answered 404, the same race as an AgentSocket hit just before a release
afb2cc33 replicaState.skipLogged and its log line −13 none, the 5 s tick only The "holds volumes and is not replicated" log line is gone. The rule stays documented, and GET /v1/sandboxes/{id}/volumes shows what a claim holds
8d2a837e Prebound door closing in Prebind and Reloaded −12 one lock and one map lookup per claim on an unguarded node, measured 3.8–5.6 ns to about 15 ns serial and about 120 ns at 12-way contention, against a warm claim of 0.2–0.7 ms ArmProxy closes them instead. After a reload to no policy, an unclaimed warm VM keeps two listeners until it is claimed or removed

Tests: rewritten where they drove a sequence production cannot produce, and named in each commit. Mutants confirm the replacements:

  • Removing the adoption precheck: moved the row from epoch 3 to 4 under the wake.
  • No door close: a prebound door is left open.

Not cut, tried and reverted:

The tap lookup (b3f0964e, reverted). runRecord swallows a parse failure of cocoon's output and returns an empty record. Without the lookup every egress claim on such a node answers 500 and burns its warm VM: 0 of 5 claims succeeded, against 5 of 5 with it. No cocoon version read so far triggers it, but a test guaranteed the behaviour, and 19 lines off a branch that never runs buy nothing.

The retry inside confirmOwnership (413724c5, reverted in the last commit). It looked redundant, because the next renewal retries. But without it, one release landing on a claim not yet confirmed aborts the whole unfence pass, and every replicated claim on the node stays at 503. Fake-clock simulation with 1.3 ms per Take, 20 trials per row:

replicated claims releases/s fence, with the retry fence, without it (mean / max)
100 5 130 ms 130 ms / 130 ms
100 20 130 ms 4.35 s / 5.13 s
1000 1 1.3 s 4.03 s / 19.6 s
1000 5 1.3 s 1m57s / 2m36s
1000 20 1.3 s 49.7 s / 52.6 s

The rates above 1/s end early only because the simulation runs out of claims to release.

Evidence:

  • go build, go vet on both GOOS, asl on both GOOS and gofmt are clean.
  • go test ./... and -race on pool, outbound and server pass, as do the e2e module's tests.
  • pool, outbound and server pass in a Linux arm64 container as root.

Hardware: two bare-metal nodes, a real PostgreSQL and S3-compatible storage, n=1 per step, at 8d2a837, which still carried the two later-reverted cuts:

  • After an owner died, one adoption moved the row from epoch 32 to 33. Then 30 repeated /owner calls answered 200 with the row byte-identical and no sandboxd_archives statement in that window.
  • A warm VM's doors stay bound after a reload that drops every policy, close at its claim, and the guest behaves.
  • An egress claim's tap locks with the recorded name.

Not run: that hardware stage at this head. The full hardware round on the tree with every open PR merged covers it. An independent review reran each remaining cut against the base with differential tests on darwin, and kept all three.

Stacked on #420. holdStaleVolumes stays: its 409 is part of the API.

@CMGS CMGS changed the title cut: five mechanisms the rest of the system already covers cut: four mechanisms the rest of the system already covers Nov 2, 2026
@CMGS CMGS changed the title cut: four mechanisms the rest of the system already covers cut: three mechanisms the rest of the system already covers Nov 2, 2026
CMGS added 7 commits November 3, 2026 16:22
…ng a second take

handleOwner reaches AdoptArchived for a held claim only when its AgentSocket miss predates the flight that adopted it. A check under m.mu at the top of the flight answers that case before any registry write, so the duplicate branch and adoptHeld cannot run: no other path adds an id a live row names while the server serves.
… inside it

A refused take no longer re-reads the row and retries within the pass. Any error leaves the node fenced and the next renewal, 5 s later, reads the rows again. An own claim is still never dropped on a refusal, and a contested pass costs replicated claims about 5 s more of 503.
Arm runs only from finalizeBatch on a sandbox startVM just built, and every claim path goes through startVM, which copies the tap from cocoon's clone or run record. Restart uses readoptEgressTap and Relock. An empty tap now fails the claim closed, which is the Arm contract, and outbound.Engine no longer needs Inspect.
…ogged

The line only repeated a documented rule: a claim that holds volumes is not replicated. GET /v1/sandboxes/{id}/volumes shows what a claim holds, so the state stays visible. replicaState collapses to the time of the last try.
ArmProxy's early return now closes what refill bound, so Prebind's re-check and the reload sweep have nothing left to cover. After a reload that drops every egress policy, an unclaimed warm VM keeps its listeners until it is claimed or removed. No tenant code runs in it before the claim, and removeVM, the only removal path, closes them too. An unguarded claim gains one uncontended lock and one map lookup, about 20 ns.
This reverts 413724c. Without the retry, one release landing on a claim not yet confirmed aborts the whole unfence pass, and every replicated claim on the node stays at 503 for another renewal. Simulated with 1.3 ms per Take: with 1000 replicated claims and 5 releases/s the fence lasted a mean 1m57s against 1.3 s with the retry, and with 1/s a mean 4.0 s and a max 19.6 s against 1.3 s. 16 lines do not pay for that.
This reverts b3f0964. runRecord swallows a parse failure of cocoon's output and returns an empty record, so without the fallback every egress claim on such a node answers 500 and burns its warm VM, where before it took one inspect and a warning: 5 of 5 claims succeeded with the lookup, 0 of 5 without. No cocoon version read today triggers it, but a test guaranteed the behaviour and 19 lines off a branch that never runs buy nothing.
@CMGS
CMGS force-pushed the cut/overdesign-low branch from 5d66f91 to 36bfa20 Compare November 3, 2026 08:22
@CMGS
CMGS changed the base branch from review/post-417 to main November 3, 2026 08:22
@CMGS
CMGS merged commit c609c5c into main Nov 3, 2026
1 check passed
@CMGS
CMGS deleted the cut/overdesign-low branch November 3, 2026 09:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant