Skip to content

feat(box): deliberate user-triggered box image update - #149

Open
pythonlearner1025 wants to merge 7 commits into
mainfrom
feat/box-update-ui
Open

feat(box): deliberate user-triggered box image update#149
pythonlearner1025 wants to merge 7 commits into
mainfrom
feat/box-update-ui

Conversation

@pythonlearner1025

Copy link
Copy Markdown
Member

A user can now click a button in My machine to deliberately update their
box to the deployment's current image. Three layers, one PR.

Two live bugs found on blitzos-dev while building this made the flag a dead
letter on canary; both are fixed here.

1. The host updater could never update a canary box

The emitted updater refused every https ref outright — "a tarball ref cannot
be pulled in place"
. Canary pins BOX_IMAGE_REF to an https R2
manifest.json, so no canary box could ever update in place.

The first boot already knew how to install that image, so the updater was
about to carry a second copy of the same pipeline: download every part, check
each digest, concatenate, check the whole, docker load. Two copies of a
verification pipeline are two chances to verify differently. So the pipeline
became one host script, /usr/local/sbin/blitz-box-image, written once by the
bootstrap and invoked by both callers. Its exit codes are the interface —
10 download, 11 digest, 12 load, 13 bad manifest — and the updater
turns them into the contract's outcomes.

The pull-first invariant now holds on both paths and is pinned on both: the
image is acquired and verified before the running container is touched, so
every acquire failure leaves the workspace exactly as it was. A new image that
will not start still rolls back.

2. The updater could not keep its own credential fresh

A box access token lives 15 minutes (ACCESS_LIFETIME_MS) and the timer runs
every 5, but nothing on the VM keeps /var/lib/blitz/box-credential.json
fresh
. The Go client inside the container rotates only in reaction to its own
401, which needs somebody to run blitz-cred. On a quiet box the on-disk token
expires and every later poll 401s for good.

Measured live: file mtime 02:03, 401 invalid box access token on every poll
from 02:20 onward, while blitz-cred and the gateway kept working — because
blitz-cred self-heals on its own 401 and the gateway never reads the
credential at all.

The updater now spends the refresh token itself against POST /oauth/token,
under the same flock (box-credential.lock) the Go client takes, and writes
the rotation back. It must be able to do this while the container it is
about to replace is broken — which is exactly when nothing inside the box can
help. Writing the file also keeps it fresh for every other reader on the
machine as a side effect.

3. Contract, route, UI

Contract (box config v1, fixtures + both conformance suites together).
Three outcomes for the stages the manifest path can fail at —
download-failed, digest-mismatch, load-failed — an optional tag on the
update result, and boxImageSha256 on the config.

tag is the load-bearing addition. Under a manifest pin the ref is
byte-identical across rebakes while the tag inside it moves, so comparing
refs would read every box as current forever. tag is the CONCRETE image the
container runs once the attempt settled — which is the OLD image whenever the
attempt changed nothing.

boxImageSha256 closes a gap I found on the way: the updater was checking the
archive against the digest the manifest declares, which is self-certifying —
whoever serves the manifest serves the digest beside it. The first boot never
had that weakness, because its digest is baked in from the control plane. Now
the updater gets the same pin over the same separate connection and passes it
to the installer as the fourth argument the first boot already used. An archive
that is internally consistent and still not the image this deployment pinned is
refused, container untouched.

One subtlety that bit: the parsed config moved from a tab-separated line to one
field per line. TAB is IFS whitespace, so read -r collapses an empty column —
and boxImageSha256 is empty on every registry deployment, which silently
shifted the origin and the update flag one field left. Caught by the host
suite, and it is why that suite runs real bash.

Route. POST /machines/:machineId/box-update, the machine-verb shape and
gate of the lifecycle verbs beside it (core/machine-access.ts, scope own).
It names ONE machine whoever calls it. The existing
POST /workspaces/:id/box-update stays the deliberate admin fan-out — a user
who clicked a button in their own machine dialog did not ask to restart a
colleague's work, and this route can never become that by accident.

MachineView gains boxImage, boxImageTarget, boxUpdateRequested and
boxUpdateOutcome. Migration 0046 adds the two columns, and armPhoneHome
seeds the image at every VM provision — the one moment the answer is known
exactly, since the same runtime.vars renders the user-data that boot
installs. Without it, a machine nobody had ever asked to update would have no
reported image, and the button's own label would have no answer.

UI. A Box image section in the My machine dialog: settings-surface cfg-
classes only, no new CSS. It shows both images so "update available" is
checkable rather than asserted, then one status line and one button that
confirms first — "This restarts your machine. Running terminals and agents
stop."

The judgement is a pure module (src/box-update-state.ts) rather than markup,
because the interesting part is which of seven things is true. Telling a user
"up to date" about a machine that is behind, or handing a button to a host that
can never install an image, is worse than telling them nothing.

Honest limitations

  • Existing VMs keep their old emitted updater. The script is baked into
    user-data at create time. Only machines created after this deploy can
    self-update. Every pre-existing canary box, blitzos-dev included, answers
    unsupported to the first request it is given — and the UI renders that
    verdict rather than hiding it: "This machine's host cannot update in place.
    Recreate it to move to the new image."
    The flag, the route and the UI work
    against old boxes today; that message is what they get.
  • The user-data budget got tighter. Hetzner caps cloud-init user-data at a
    hard 32 KiB and does not compress. A heavy manifest-mode create was 25.4 KiB
    and is now 30.4 KiB — 2.4 KiB of headroom, down from 7.3 KiB.
    test/bootstrap.test.ts pins a 2 KiB floor so the next feature that emits
    bash finds out there rather than as a 413 on a real create. Buying the
    headroom back means shipping the host scripts in the box image instead of in
    user-data (plans/MEMBER-MACHINES.md §5a) — a box-image change and a
    rebake, so it is the next step and not this one.
  • No microVM path. packages/microvm-host/ has its own guest lifecycle and
    no update path; this is the cloud-VM providers only.

Tests

test/box-update-host.test.mjs runs the emitted updater and the emitted
installer in real bash, against a real control plane over real curl, with a
scripted docker and a stand-in R2 serving a real gzip archive split into
digest-checked parts. New cases: the manifest happy path (asserting the exact
bytes that reach docker load), a part whose digest does not match, a part
that 404s, a docker load that refuses, an image already in the store, and
both credential-rotation paths, and an archive that passes its own manifest but
not the deployment's pin.

test/box-update-conformance.test.mjs additionally pins that exactly one copy
of the manifest parser is emitted and that both callers reach it by one path.

Gates

npm run typecheck ✅ · npm run lint:gate ✅ (74 findings, baseline 74) ·
npm test ✅ — control-plane 672 passed, webapp 707 passed.

Caveat worth naming: on a contended box, four packages/webapp Lody suites
time out in startLodyHarness (beforeAll), which is the known daemon-suite
flake. Re-run isolated they pass or skip; this branch touches nothing under
webapp/src/lody.

🤖 Generated with Claude Code

https://claude.ai/code/session_013voh4zgczP2jmPrS2ERrav

The box-config contract gains three outcomes and one optional field, both
sides and the fixture corpus together.

The outcomes name the stages the manifest install path can fail at:
`download-failed` (a manifest or part did not arrive), `digest-mismatch`
(a part or the whole archive failed its SHA-256, so nothing was loaded),
`load-failed` (the archive verified and docker load still refused it).
Like `pull-failed` beside them, every one of these means the running
container was never touched.

`tag` on the update result is the CONCRETE image the container runs once
the attempt has settled — the tag from inside the manifest under an R2
pin, the ref itself under a registry pin, and the OLD image whenever the
attempt left the container alone. It exists because `ref` alone cannot
answer "is an update available" under a manifest pin: that URL is
byte-identical across rebakes while the tag inside it moves. It is
optional because a host emitted before this change never sends it.

The box-config wire block moves to core/wire-box-config.ts, mirroring the
split schema/src/box-config.ts already had, so wire-machines.ts can name
BoxUpdateOutcome without importing back through core/wire.ts.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013voh4zgczP2jmPrS2ERrav
…token

Two live failures on blitzos-dev, both of which made the update flag a
dead letter on canary.

The first: the emitted updater refused every https ref outright — "a
tarball ref cannot be pulled in place". Canary pins BOX_IMAGE_REF to an
https R2 manifest, so no canary box could ever update in place.

The first boot already knew how to install that image, so the updater was
about to carry a second copy of the same pipeline: download every part,
check each digest, concatenate, check the whole, docker load. Two copies
of a verification pipeline are two chances to verify differently, and both
would ride in cloud-init user-data, which Hetzner caps at a hard 32 KiB.
So the pipeline becomes one host script, /usr/local/sbin/blitz-box-image,
that both callers invoke; its exit codes are the interface, and the
updater turns them into the contract's outcome vocabulary. The pull-first
invariant now holds on both paths and is pinned on both.

The second: the updater read /var/lib/blitz/box-credential.json and could
not rotate it. A box access token lives 15 minutes and this timer runs
every 5, but nothing on the VM keeps that file fresh — the Go client
inside the container rotates only in reaction to its own 401, which needs
somebody to run blitz-cred. On a quiet box the on-disk token expires and
every later poll 401s for good. Measured: file mtime 02:03, 401s from
02:20 onward, while blitz-cred and the gateway kept working. So the
updater now spends the refresh token itself, under the same flock the Go
client takes, and writes the rotation back — which keeps the file fresh
for every other reader on the box too. It must be able to do this while
the container it is about to replace is broken, which is exactly when
nothing inside the box can help.

Emitted-size budget: a heavy manifest-mode create was 25.4 KiB and is now
30.1 KiB against the 32 KiB cap. bootstrap.test.ts pins a 2 KiB floor so
the next feature that emits bash finds out there, not as a 413 on a real
create. Buying that headroom back means shipping the host scripts in the
box image instead of in user-data.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013voh4zgczP2jmPrS2ERrav
…judge it

`POST /machines/:machineId/box-update` is the route the My machine dialog
calls. It names ONE machine whoever calls it, and takes the same shape and
the same gate as the lifecycle verbs beside it (scope `own`: a member may
act on their own machine, an admin on any in the workspace). The existing
`POST /workspaces/:id/box-update` stays the deliberate workspace-wide
fan-out — a user who clicked a button in their own machine dialog did not
ask to restart a colleague's work, and this route can never become that by
accident. It answers with the machine, so the dialog can show the pending
flag without waiting for the next poll.

That gate was a closure inside addMachineRoutes and is now
core/machine-access.ts. Two copies of an authorization rule are how the
two drift apart, and core/machines.ts sits on the 700-line warn, so the
split pays twice.

MachineView gains what the UI needs to be honest about update state:
`boxImage` (the concrete image the machine reports), `boxImageTarget` (the
one this deployment installs now), `boxUpdateRequested`, and
`boxUpdateOutcome`. Comparing the first two is what answers "is an update
available" — and it has to be the concrete image on both sides, because
under a manifest pin the ref never changes while the tag inside it does.

`boxImageTarget` is a deployment fact, so it is passed into machineView
rather than read off the row; projectWorkspaces now takes the runtime it
was already being handed the db of.

The remaining question was what a machine that has never been asked to
update should report. Migration 0046 adds the two columns, and
armPhoneHome seeds the image at every VM provision: that is the one moment
the answer is known exactly, since the same runtime.vars renders the
user-data that boot installs. Without it the answer would stay unknown
until somebody made an update attempt, which is the question the button is
supposed to answer beforehand.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013voh4zgczP2jmPrS2ERrav
… answer

A Box image section in the My machine dialog, in the settings-surface
vocabulary (cfg- classes only, no new CSS, and the divider comes free from
.cfg-section ~ .cfg-section).

It shows both images — the one running and the one available — so "update
available" is checkable rather than asserted, then one line of status and
one button. The button confirms first, in the words that matter: "This
restarts your machine. Running terminals and agents stop."

The judgement lives in src/box-update-state.ts rather than in the
component, because the interesting part is which of seven things is true,
not the markup. Telling a user "up to date" about a machine that is behind,
or offering a button to a host that can never install an image, is worse
than telling them nothing:

- pending — asked for, waiting on the host's five-minute poll
- unsupported — the host reported it cannot update in place. This is every
  box created before the new updater shipped, blitzos-dev included, and it
  says so plainly: recreate it to move to the new image
- not-running, no-machine — nothing to update
- unknown — never reported an image. Asking is still offered, because the
  attempt is what makes it report; guessing "up to date" here is how a
  stale box looks current forever
- up-to-date / available — the concrete images differ, or they do not

A failed attempt keeps the retry on offer and says the machine was left
untouched, because that is the updater's invariant and the first thing a
worried user wants to know.

The dialog takes refreshWorkspaces so the pending state appears on the
click rather than up to 15 seconds later.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013voh4zgczP2jmPrS2ERrav
plans/MEMBER-MACHINES.md §5a covers the whole path: the flag and why its
three routes differ, the two install modes behind one host script, how "is
an update available" is answered when a manifest ref never changes, why old
boxes can never self-update and what the UI says about it, and why the
updater has to be able to rotate its own credential.

CLAUDE.md gets the box-config row's new fields, and a new rule under the VM
provider section: the emitted script has a hard 32 KiB budget, reasoning
belongs in the TS comment that never ships, and the way to buy headroom
back is to stop shipping the host scripts in user-data at all.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013voh4zgczP2jmPrS2ERrav
…the manifest

The updater verified the archive against the digest the MANIFEST declares,
which is self-certifying: whoever serves the manifest serves the digest
beside it, so on its own it proves only that the parts were reassembled
correctly. The first boot never had that weakness — its BOX_IMAGE_SHA256 is
baked in at create time and arrives from the control plane.

So box-config carries `boxImageSha256` too, and the updater passes it to
the installer as the same optional fourth argument the first boot uses. An
archive that is internally consistent and still not the image this
deployment pinned is now refused, with the container untouched. Empty under
a registry pin, where the ref carries its own digest and docker checks it.

Two things fell out of the change:

The parsed config moved from a tab-separated line to one field per line,
read with mapfile. TAB is IFS whitespace, so `read -r` collapses an empty
column — and `boxImageSha256` is empty on every registry deployment, which
would have shifted the origin and the update flag one field left. The
emitted parser and its fixtures moved together.

`retry` is only called by the registry branch of the image setup and the
installer only by the tarball one, so `boxImageSetupPreamble` now emits what
each mode actually runs. One function for the bootstrap and the bake, so
neither can drift into emitting a helper the other does not. A heavy
manifest create is 30.4 KiB of the 32 KiB cap, 2.4 KiB clear.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013voh4zgczP2jmPrS2ERrav
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants