Repository navigation
pool: a live move aborts a paused final round that overruns max_downtime_ms, and reports its phases - #553
Merged
Conversation
…ime_ms, and reports its phases The pre-copy estimate covers only the last delta's transfer plus 400 ms, so a final capture starved on a busy source, or pages dirtied since the last delta, paused guests well past the budget (17 s against 2 s on a loaded node). With the guest paused, the source now checks twice before the commit: after the final capture, its time plus the final round's scaled wire bytes; after the final round arrived, the pause so far. Past the budget it aborts through the existing settle path, so the guest resumes on the source and the target drops what it received. The answer splits downtime_ms into capture_ms, send_ms and commit_ms.
…s one check once paused The pre-pause estimate now adds the last delta capture's time, so a source too slow to capture answers 503 without pausing the guest. The check after the final round's transfer is dropped: by then the pause is spent and an abort ends it no sooner than the commit, so it only turned a late success into a failure; send_ms still reports it. Simplify: one size variable across the loop, the final round's size bound once, a finalCaptured test helper, and an unread wire size dropped.
…t delta's wire bytes instead of scaling them Scaling multiplied each round's fixed transfer overhead along with its data, so a tiny last delta followed by a larger final round looked far too large and aborted moves that fit. The estimate is now the last delta's wire bytes plus the final round's growth. The docs name the abort's record write in the worst case and describe the phase times as parts of downtime_ms, and the move test checks the capture time too.
…ture time On test1 (load 164) delta captures averaged 121 ms against 55 ms for final ones, since a delta runs with the guest live. Counting it refused every move at a 600 ms budget while main completed the same moves in 546-591 ms. The check after the final capture already measures the capture that pauses the guest, so it alone covers a slow final capture.
… leaves room under -race
…r the last delta A final round smaller than the last delta, as zero-heavy rounds can be, cancelled the delta's wire bytes and let a slow capture through: 32 MiB on the wire with a 256 MiB shrink read as zero. The estimate now keeps the last delta's bytes and adds only growth.
This was referenced Nov 10, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of v0.1.31, per CMGS: option A+C from the live-move downtime report.
Problem. On v0.1.30, cocoonbox ran 12 serial moves with
max_downtime_ms2000, from a loaded source (load 4×, cpu psi 81) to an idle target, cross-zone. 5 of the 11 successful moves paused the guest for 2.3–17.2 s.The pre-pause estimate (
finalPause) counts only the last delta's bytes on the wire at the rounds' rate, plus a fixed 400 ms. It leaves out:Change.
max_downtime_ms. The estimate is the last delta's wire bytes plus the final round's growth in apparent size; delta files hold only dirty ranges, so apparent size tracks dirty bytes, and the fixed per-round transfer overhead is not scaled.pending, so the existing settle path takes over: the record endsaborted,cocoon vm migrate abortresumes the guest on the source, the target drops its reservation and staged rounds, and the move answers 503ErrMoveDowntime.downtime_msreports it. The docs state both.capture_ms,send_msandcommit_msnext todowntime_ms.Cost. Moves are off the claim path. The new work is a few
time.Nowcalls and float arithmetic per move, under the move's own transition lock.Tests.
TestMoveWhosePausedRoundOverrunsTheBudgetResumesTheGuestOnTheSourcecovers a slow final capture and a final round grown 1 TiB over a 1 KiB delta. Each case asserts:TestMoveWhoseFinalRoundOutgrowsATinyDeltaStillFits: a 1-byte delta sent as 4 KiB, then a 16 MiB final round, still fits.Capture,SendandCommitare all > 0 and that capture + send + commit ≤ downtime. The server test covers the new fields.TestMoveGivesEachRoundADeadlineFromItsSizegets explicit wire sizes (a 1 GiB baseline, so its rate leaves room under-race), since its 1 TiB final round would otherwise trip the check./simplify (four lenses):
sizeacross the loop instead oflastWire;finalCapturedtest helper;pausedAtbefore the dump-slot wait. It matches the documenteddowntime_ms, and is revisited only if hardware shows large slot waits.Gates.
0 issues.on darwin and linux; fmt clean; asl clean.go test -race ./...green on darwin; linux/arm64 green except the container-onlyutilstest that also fails on main.Hardware. test1 (load 164, the source) → test2, warm-0 pool, a 4G guest rewriting 256 MiB of
/dev/shmfrom urandom; every success moved back.vm migrate abort47–50 ms, and the guest ran commands after each abort.sandboxd_movesrows.