Skip to content

pool: a live move aborts a paused final round that overruns max_downtime_ms, and reports its phases - #553

Merged
CMGS merged 6 commits into
mainfrom
move/downtime-guard
Nov 10, 2026
Merged

CMGS merged 6 commits into
mainfrom
move/downtime-guard

Conversation

@CMGS

@CMGS CMGS commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Part of v0.1.31, per CMGS: option A+C from the live-move downtime report.

Problem. On v0.1.30, cocoonbox ran 12 serial moves with max_downtime_ms 2000, from a loaded source (load 4×, cpu psi 81) to an idle target, cross-zone. 5 of the 11 successful moves paused the guest for 2.3–17.2 s.

The pre-pause estimate (finalPause) counts only the last delta's bytes on the wire at the rounds' rate, plus a fixed 400 ms. It leaves out:

  • the final capture, which runs with the guest paused and was starved on that source;
  • the pages dirtied since the last delta;
  • any part of the commit beyond the 400 ms.

Change.

  • Once paused (A):
    • After the final capture, the source checks capture time plus the final round's estimated wire bytes against max_downtime_ms. The estimate is the last delta's wire bytes plus the final round's growth in apparent size; delta files hold only dirty ranges, so apparent size tracks dirty bytes, and the fixed per-round transfer overhead is not scaled.
    • If it doesn't fit, the move aborts before the final round is sent. The move record is still pending, so the existing settle path takes over: the record ends aborted, cocoon vm migrate abort resumes the guest on the source, the target drops its reservation and staged rounds, and the move answers 503 ErrMoveDowntime.
  • Worst case: the check cannot give back a pause already spent. With the meta store up, a final capture that alone overruns the budget still pauses the guest for the capture plus the abort and its record write. Once past the check, a slow transfer or commit still lengthens the pause, and downtime_ms reports it. The docs state both.
  • Phases (C): the move answer gains capture_ms, send_ms and commit_ms next to downtime_ms.
  • Dropped during /simplify: a second check after the final transfer. By then the pause is spent, and an abort ends it no sooner than the commit, so the check would only turn a late success into a failure.
  • Dropped after hardware: adding the last delta capture's time to the pre-pause estimate. On test1, delta captures averaged 121 ms against 55 ms for final ones, because a delta runs with the guest live. With that term the head refused every move at 600 ms, while main completed the same moves within budget.

Cost. Moves are off the claim path. The new work is a few time.Now calls and float arithmetic per move, under the move's own transition lock.

Tests.

  • TestMoveWhosePausedRoundOverrunsTheBudgetResumesTheGuestOnTheSource covers a slow final capture and a final round grown 1 TiB over a 1 KiB delta. Each case asserts:
    • the final capture happened;
    • no final round was sent and no commit was made;
    • the guest is running and unmarked on the source;
    • the target holds no reservation, staging dir or received VM;
    • no move record is left.
  • TestMoveWhoseFinalRoundOutgrowsATinyDeltaStillFits: a 1-byte delta sent as 4 KiB, then a 16 MiB final round, still fits.
  • The main move test checks that Capture, Send and Commit are all > 0 and that capture + send + commit ≤ downtime. The server test covers the new fields.
  • TestMoveGivesEachRoundADeadlineFromItsSize gets explicit wire sizes (a 1 GiB baseline, so its rate leaves room under -race), since its 1 TiB final round would otherwise trip the check.
  • Mutants, all killed:
    • no post-capture check;
    • capture not timed;
    • send not timed;
    • commit not timed;
    • response drops the phases;
    • final round growth ignored;
    • ratio scaling instead of growth;
    • capture missing from the result.

/simplify (four lenses):

  • Fixed:
    • the post-send check dropped (altitude);
    • one size across the loop instead of lastWire;
    • the final round's size bound once;
    • a finalCaptured test helper;
    • an unread wire size dropped;
    • a test field derived instead of stored.
  • Skipped: refusing moves up front on node psi. It needs a config field, and it would block the moves most wanted, those that take load off a hot node.
  • Kept as is: pausedAt before the dump-slot wait. It matches the documented downtime_ms, and is revisited only if hardware shows large slot waits.

Gates.

  • golangci-lint 0 issues. on darwin and linux; fmt clean; asl clean.
  • go test -race ./... green on darwin; linux/arm64 green except the container-only utils test that also fails on main.
  • No new comments.

Hardware. test1 (load 164, the source) → test2, warm-0 pool, a 4G guest rewriting 256 MiB of /dev/shm from urandom; every success moved back.

budget head ad3953e base 8a45f70
600 ms 8/8 ok, downtime 531–554 ms 8/8 ok, 546–591 ms
800 ms 8/8 ok, 595–629 ms 10/10 ok, 649–710 ms
520 ms 6/6 ok, 441–540 ms —
500 ms 3 ok at 511–523 ms, 5 refused before any pause —
480 ms 6 refused before any pause —
  • Phases (600 ms arm): capture 36–43 ms, send 74–92 ms, commit 411–428 ms. The small overruns at 500 and 520 ms come from commit at 421–442 ms against the 400 ms allowance, which is the documented unguarded tail.
  • Post-capture abort: never fired here, because final captures took 31–56 ms. It is covered by the unit tests.
  • Paused abort cost, timed at the cocoon CLI on test1: final capture ~145 ms plus vm migrate abort 47–50 ms, and the guest ran commands after each abort.
  • Target clean: after every 503, the target showed 0 VMs, 0 staging dirs and 0 sandboxd_moves rows.
  • Revert evidence: with the delta-capture term, the head (9dad147) refused 8/8 moves at 600 ms.

CMGS added 6 commits November 10, 2026 19:53
…ime_ms, and reports its phases

The pre-copy estimate covers only the last delta's transfer plus 400 ms,
so a final capture starved on a busy source, or pages dirtied since the
last delta, paused guests well past the budget (17 s against 2 s on a
loaded node). With the guest paused, the source now checks twice before
the commit: after the final capture, its time plus the final round's
scaled wire bytes; after the final round arrived, the pause so far. Past
the budget it aborts through the existing settle path, so the guest
resumes on the source and the target drops what it received. The answer
splits downtime_ms into capture_ms, send_ms and commit_ms.
…s one check once paused

The pre-pause estimate now adds the last delta capture's time, so a
source too slow to capture answers 503 without pausing the guest. The
check after the final round's transfer is dropped: by then the pause is
spent and an abort ends it no sooner than the commit, so it only turned
a late success into a failure; send_ms still reports it. Simplify: one
size variable across the loop, the final round's size bound once, a
finalCaptured test helper, and an unread wire size dropped.
…t delta's wire bytes instead of scaling them

Scaling multiplied each round's fixed transfer overhead along with its
data, so a tiny last delta followed by a larger final round looked far
too large and aborted moves that fit. The estimate is now the last
delta's wire bytes plus the final round's growth. The docs name the
abort's record write in the worst case and describe the phase times as
parts of downtime_ms, and the move test checks the capture time too.
…ture time

On test1 (load 164) delta captures averaged 121 ms against 55 ms for
final ones, since a delta runs with the guest live. Counting it refused
every move at a 600 ms budget while main completed the same moves in
546-591 ms. The check after the final capture already measures the
capture that pauses the guest, so it alone covers a slow final capture.
…r the last delta

A final round smaller than the last delta, as zero-heavy rounds can be,
cancelled the delta's wire bytes and let a slow capture through: 32 MiB
on the wire with a 256 MiB shrink read as zero. The estimate now keeps
the last delta's bytes and adds only growth.
@CMGS
CMGS merged commit 93fc22b into main Nov 10, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant