feat(box): deliberate user-triggered box image update - #149
Open
pythonlearner1025 wants to merge 7 commits into
Open
feat(box): deliberate user-triggered box image update#149pythonlearner1025 wants to merge 7 commits into
pythonlearner1025 wants to merge 7 commits into
Conversation
The box-config contract gains three outcomes and one optional field, both sides and the fixture corpus together. The outcomes name the stages the manifest install path can fail at: `download-failed` (a manifest or part did not arrive), `digest-mismatch` (a part or the whole archive failed its SHA-256, so nothing was loaded), `load-failed` (the archive verified and docker load still refused it). Like `pull-failed` beside them, every one of these means the running container was never touched. `tag` on the update result is the CONCRETE image the container runs once the attempt has settled — the tag from inside the manifest under an R2 pin, the ref itself under a registry pin, and the OLD image whenever the attempt left the container alone. It exists because `ref` alone cannot answer "is an update available" under a manifest pin: that URL is byte-identical across rebakes while the tag inside it moves. It is optional because a host emitted before this change never sends it. The box-config wire block moves to core/wire-box-config.ts, mirroring the split schema/src/box-config.ts already had, so wire-machines.ts can name BoxUpdateOutcome without importing back through core/wire.ts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013voh4zgczP2jmPrS2ERrav
…token Two live failures on blitzos-dev, both of which made the update flag a dead letter on canary. The first: the emitted updater refused every https ref outright — "a tarball ref cannot be pulled in place". Canary pins BOX_IMAGE_REF to an https R2 manifest, so no canary box could ever update in place. The first boot already knew how to install that image, so the updater was about to carry a second copy of the same pipeline: download every part, check each digest, concatenate, check the whole, docker load. Two copies of a verification pipeline are two chances to verify differently, and both would ride in cloud-init user-data, which Hetzner caps at a hard 32 KiB. So the pipeline becomes one host script, /usr/local/sbin/blitz-box-image, that both callers invoke; its exit codes are the interface, and the updater turns them into the contract's outcome vocabulary. The pull-first invariant now holds on both paths and is pinned on both. The second: the updater read /var/lib/blitz/box-credential.json and could not rotate it. A box access token lives 15 minutes and this timer runs every 5, but nothing on the VM keeps that file fresh — the Go client inside the container rotates only in reaction to its own 401, which needs somebody to run blitz-cred. On a quiet box the on-disk token expires and every later poll 401s for good. Measured: file mtime 02:03, 401s from 02:20 onward, while blitz-cred and the gateway kept working. So the updater now spends the refresh token itself, under the same flock the Go client takes, and writes the rotation back — which keeps the file fresh for every other reader on the box too. It must be able to do this while the container it is about to replace is broken, which is exactly when nothing inside the box can help. Emitted-size budget: a heavy manifest-mode create was 25.4 KiB and is now 30.1 KiB against the 32 KiB cap. bootstrap.test.ts pins a 2 KiB floor so the next feature that emits bash finds out there, not as a 413 on a real create. Buying that headroom back means shipping the host scripts in the box image instead of in user-data. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013voh4zgczP2jmPrS2ERrav
…judge it `POST /machines/:machineId/box-update` is the route the My machine dialog calls. It names ONE machine whoever calls it, and takes the same shape and the same gate as the lifecycle verbs beside it (scope `own`: a member may act on their own machine, an admin on any in the workspace). The existing `POST /workspaces/:id/box-update` stays the deliberate workspace-wide fan-out — a user who clicked a button in their own machine dialog did not ask to restart a colleague's work, and this route can never become that by accident. It answers with the machine, so the dialog can show the pending flag without waiting for the next poll. That gate was a closure inside addMachineRoutes and is now core/machine-access.ts. Two copies of an authorization rule are how the two drift apart, and core/machines.ts sits on the 700-line warn, so the split pays twice. MachineView gains what the UI needs to be honest about update state: `boxImage` (the concrete image the machine reports), `boxImageTarget` (the one this deployment installs now), `boxUpdateRequested`, and `boxUpdateOutcome`. Comparing the first two is what answers "is an update available" — and it has to be the concrete image on both sides, because under a manifest pin the ref never changes while the tag inside it does. `boxImageTarget` is a deployment fact, so it is passed into machineView rather than read off the row; projectWorkspaces now takes the runtime it was already being handed the db of. The remaining question was what a machine that has never been asked to update should report. Migration 0046 adds the two columns, and armPhoneHome seeds the image at every VM provision: that is the one moment the answer is known exactly, since the same runtime.vars renders the user-data that boot installs. Without it the answer would stay unknown until somebody made an update attempt, which is the question the button is supposed to answer beforehand. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013voh4zgczP2jmPrS2ERrav
… answer A Box image section in the My machine dialog, in the settings-surface vocabulary (cfg- classes only, no new CSS, and the divider comes free from .cfg-section ~ .cfg-section). It shows both images — the one running and the one available — so "update available" is checkable rather than asserted, then one line of status and one button. The button confirms first, in the words that matter: "This restarts your machine. Running terminals and agents stop." The judgement lives in src/box-update-state.ts rather than in the component, because the interesting part is which of seven things is true, not the markup. Telling a user "up to date" about a machine that is behind, or offering a button to a host that can never install an image, is worse than telling them nothing: - pending — asked for, waiting on the host's five-minute poll - unsupported — the host reported it cannot update in place. This is every box created before the new updater shipped, blitzos-dev included, and it says so plainly: recreate it to move to the new image - not-running, no-machine — nothing to update - unknown — never reported an image. Asking is still offered, because the attempt is what makes it report; guessing "up to date" here is how a stale box looks current forever - up-to-date / available — the concrete images differ, or they do not A failed attempt keeps the retry on offer and says the machine was left untouched, because that is the updater's invariant and the first thing a worried user wants to know. The dialog takes refreshWorkspaces so the pending state appears on the click rather than up to 15 seconds later. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013voh4zgczP2jmPrS2ERrav
plans/MEMBER-MACHINES.md §5a covers the whole path: the flag and why its three routes differ, the two install modes behind one host script, how "is an update available" is answered when a manifest ref never changes, why old boxes can never self-update and what the UI says about it, and why the updater has to be able to rotate its own credential. CLAUDE.md gets the box-config row's new fields, and a new rule under the VM provider section: the emitted script has a hard 32 KiB budget, reasoning belongs in the TS comment that never ships, and the way to buy headroom back is to stop shipping the host scripts in user-data at all. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013voh4zgczP2jmPrS2ERrav
…the manifest The updater verified the archive against the digest the MANIFEST declares, which is self-certifying: whoever serves the manifest serves the digest beside it, so on its own it proves only that the parts were reassembled correctly. The first boot never had that weakness — its BOX_IMAGE_SHA256 is baked in at create time and arrives from the control plane. So box-config carries `boxImageSha256` too, and the updater passes it to the installer as the same optional fourth argument the first boot uses. An archive that is internally consistent and still not the image this deployment pinned is now refused, with the container untouched. Empty under a registry pin, where the ref carries its own digest and docker checks it. Two things fell out of the change: The parsed config moved from a tab-separated line to one field per line, read with mapfile. TAB is IFS whitespace, so `read -r` collapses an empty column — and `boxImageSha256` is empty on every registry deployment, which would have shifted the origin and the update flag one field left. The emitted parser and its fixtures moved together. `retry` is only called by the registry branch of the image setup and the installer only by the tarball one, so `boxImageSetupPreamble` now emits what each mode actually runs. One function for the bootstrap and the bake, so neither can drift into emitting a helper the other does not. A heavy manifest create is 30.4 KiB of the 32 KiB cap, 2.4 KiB clear. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013voh4zgczP2jmPrS2ERrav
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013voh4zgczP2jmPrS2ERrav
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A user can now click a button in My machine to deliberately update their
box to the deployment's current image. Three layers, one PR.
Two live bugs found on
blitzos-devwhile building this made the flag a deadletter on canary; both are fixed here.
1. The host updater could never update a canary box
The emitted updater refused every https ref outright — "a tarball ref cannot
be pulled in place". Canary pins
BOX_IMAGE_REFto an https R2manifest.json, so no canary box could ever update in place.The first boot already knew how to install that image, so the updater was
about to carry a second copy of the same pipeline: download every part, check
each digest, concatenate, check the whole,
docker load. Two copies of averification pipeline are two chances to verify differently. So the pipeline
became one host script,
/usr/local/sbin/blitz-box-image, written once by thebootstrap and invoked by both callers. Its exit codes are the interface —
10download,11digest,12load,13bad manifest — and the updaterturns them into the contract's outcomes.
The pull-first invariant now holds on both paths and is pinned on both: the
image is acquired and verified before the running container is touched, so
every acquire failure leaves the workspace exactly as it was. A new image that
will not start still rolls back.
2. The updater could not keep its own credential fresh
A box access token lives 15 minutes (
ACCESS_LIFETIME_MS) and the timer runsevery 5, but nothing on the VM keeps
/var/lib/blitz/box-credential.jsonfresh. The Go client inside the container rotates only in reaction to its own
401, which needs somebody to run
blitz-cred. On a quiet box the on-disk tokenexpires and every later poll 401s for good.
Measured live: file mtime 02:03, 401
invalid box access tokenon every pollfrom 02:20 onward, while
blitz-credand the gateway kept working — becauseblitz-credself-heals on its own 401 and the gateway never reads thecredential at all.
The updater now spends the refresh token itself against
POST /oauth/token,under the same flock (
box-credential.lock) the Go client takes, and writesthe rotation back. It must be able to do this while the container it is
about to replace is broken — which is exactly when nothing inside the box can
help. Writing the file also keeps it fresh for every other reader on the
machine as a side effect.
3. Contract, route, UI
Contract (
box config v1, fixtures + both conformance suites together).Three outcomes for the stages the manifest path can fail at —
download-failed,digest-mismatch,load-failed— an optionaltagon theupdate result, and
boxImageSha256on the config.tagis the load-bearing addition. Under a manifest pin the ref isbyte-identical across rebakes while the tag inside it moves, so comparing
refs would read every box as current forever.
tagis the CONCRETE image thecontainer runs once the attempt settled — which is the OLD image whenever the
attempt changed nothing.
boxImageSha256closes a gap I found on the way: the updater was checking thearchive against the digest the manifest declares, which is self-certifying —
whoever serves the manifest serves the digest beside it. The first boot never
had that weakness, because its digest is baked in from the control plane. Now
the updater gets the same pin over the same separate connection and passes it
to the installer as the fourth argument the first boot already used. An archive
that is internally consistent and still not the image this deployment pinned is
refused, container untouched.
One subtlety that bit: the parsed config moved from a tab-separated line to one
field per line. TAB is IFS whitespace, so
read -rcollapses an empty column —and
boxImageSha256is empty on every registry deployment, which silentlyshifted the origin and the update flag one field left. Caught by the host
suite, and it is why that suite runs real bash.
Route.
POST /machines/:machineId/box-update, the machine-verb shape andgate of the lifecycle verbs beside it (
core/machine-access.ts, scopeown).It names ONE machine whoever calls it. The existing
POST /workspaces/:id/box-updatestays the deliberate admin fan-out — a userwho clicked a button in their own machine dialog did not ask to restart a
colleague's work, and this route can never become that by accident.
MachineViewgainsboxImage,boxImageTarget,boxUpdateRequestedandboxUpdateOutcome. Migration0046adds the two columns, andarmPhoneHomeseeds the image at every VM provision — the one moment the answer is known
exactly, since the same
runtime.varsrenders the user-data that bootinstalls. Without it, a machine nobody had ever asked to update would have no
reported image, and the button's own label would have no answer.
UI. A
Box imagesection in the My machine dialog: settings-surfacecfg-classes only, no new CSS. It shows both images so "update available" is
checkable rather than asserted, then one status line and one button that
confirms first — "This restarts your machine. Running terminals and agents
stop."
The judgement is a pure module (
src/box-update-state.ts) rather than markup,because the interesting part is which of seven things is true. Telling a user
"up to date" about a machine that is behind, or handing a button to a host that
can never install an image, is worse than telling them nothing.
Honest limitations
user-data at create time. Only machines created after this deploy can
self-update. Every pre-existing canary box,
blitzos-devincluded, answersunsupportedto the first request it is given — and the UI renders thatverdict rather than hiding it: "This machine's host cannot update in place.
Recreate it to move to the new image." The flag, the route and the UI work
against old boxes today; that message is what they get.
hard 32 KiB and does not compress. A heavy manifest-mode create was 25.4 KiB
and is now 30.4 KiB — 2.4 KiB of headroom, down from 7.3 KiB.
test/bootstrap.test.tspins a 2 KiB floor so the next feature that emitsbash finds out there rather than as a 413 on a real create. Buying the
headroom back means shipping the host scripts in the box image instead of in
user-data (
plans/MEMBER-MACHINES.md§5a) — a box-image change and arebake, so it is the next step and not this one.
packages/microvm-host/has its own guest lifecycle andno update path; this is the cloud-VM providers only.
Tests
test/box-update-host.test.mjsruns the emitted updater and the emittedinstaller in real bash, against a real control plane over real curl, with a
scripted docker and a stand-in R2 serving a real gzip archive split into
digest-checked parts. New cases: the manifest happy path (asserting the exact
bytes that reach
docker load), a part whose digest does not match, a partthat 404s, a
docker loadthat refuses, an image already in the store, andboth credential-rotation paths, and an archive that passes its own manifest but
not the deployment's pin.
test/box-update-conformance.test.mjsadditionally pins that exactly one copyof the manifest parser is emitted and that both callers reach it by one path.
Gates
npm run typecheck✅ ·npm run lint:gate✅ (74 findings, baseline 74) ·npm test✅ — control-plane 672 passed, webapp 707 passed.Caveat worth naming: on a contended box, four
packages/webappLody suitestime out in
startLodyHarness(beforeAll), which is the known daemon-suiteflake. Re-run isolated they pass or skip; this branch touches nothing under
webapp/src/lody.🤖 Generated with Claude Code
https://claude.ai/code/session_013voh4zgczP2jmPrS2ERrav