diff --git a/CLAUDE.md b/CLAUDE.md index 6bb52b5f..2725b601 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -82,7 +82,7 @@ conformance tests on BOTH sides. Never hand-edit one side of a contract. | lody local-project registration | box node `box/rootfs/usr/local/libexec/blitz-lody-projects` (registers each `/workspace/` clone) ↔ browser `webapp/src/lody/local-bridge.ts` + `rpc-client.ts` + `local-projects.ts` (`registerWorkspaceRepositories`, the same sweep driven from the tab) ↔ the `lody` daemon's `/project-control` (not in this tree). The SCHEMA stays Lody's (`vendor/lody/packages/shared/src/message-schemas.ts`, `LocalProjectControlRequest`/`Response`); what is ours is that two BlitzOS producers keep agreeing with it | `fixtures/lody-project-registration/` (responses captured from a real `lody@0.88.1`) | `box/guest-tests/test/lody-projects-registration.test.ts` (runs the real registrar against a stand-in daemon socket), `webapp/test/lody-project-control-frames.test.ts` (browser producer/parser) | | lody session-control stream | browser `webapp/src/lody/rpc-client.ts` (`sendSessionControl`) ↔ node `box/rootfs/usr/local/libexec/blitz-lody-bridge` ↔ the `lody` daemon's `/session-control` (not in this tree). The daemon picks NDJSON-per-response or one buffered envelope from the request's `Accept`; ours is the browser that negotiates and reads it frame by frame, and the bridge decision that carries the negotiation upstream. The FRAME UNION stays Lody's (`vendor/lody/packages/shared/src/node/local-ipc.ts:80`, `{kind:'response'\|'complete'\|'error'}`) — it is not exported and its module is node-only, so `rpc-client.ts` re-states it and the corpus keeps the copy honest | `fixtures/lody-session-control-stream/` (bodies captured from a real `lody@0.88.1` through the real bridge) | `webapp/test/lody-session-control-stream.test.ts` (browser consumer: frames emitted before the promise settles, at adversarial chunk boundaries), `box/guest-tests/test/lody-bridge-control-stream.test.ts` (runs the real bridge against a stand-in daemon that holds its stream open), `webapp/test/lody-acp-authentication.test.ts` (whole chain against a real daemon; skips without the bundle) | | lody share claim | Go gateway `box/gateway/main.go` (verifies the webApp ticket's `share` claim and forwards it on `X-Blitz-Lody-Share`, stripping any inbound copy) ↔ node `box/rootfs/usr/local/libexec/blitz-lody-bridge` (room ACL on `/sync`, session scoping on `/rpc` and `/project`, `/control` refused, `/platform` narrowed). The claim's OWN wire format is pinned by the webApp-ticket corpus on three runtimes; what this pins is the hand-off and the decisions the bridge makes from it | `fixtures/lody-share-claim/` | `gateway/main_test.go` (producer: the header bytes + the path allowlist), `box/guest-tests/test/lody-bridge-share.test.ts` (consumer: runs the real bridge against a stand-in daemon over the whole decision table) | -| box config v1 | CP `core/box-config.ts` producer (`GET /workspaces/self/box-config`) and consumer (`POST /workspaces/self/box-update-result`) ↔ host updater bash/python emitted by `core/bootstrap.ts` (`blitz-box-update`; cloud-VM path only — the microVM provider has its own guest lifecycle and no update path yet) | `fixtures/box-config/` | `test/box-config-conformance.test.ts` (CP), `test/box-update-conformance.test.mjs` (runs real `python3` over the emitted parser/producer, `bash -n` over the emitted scripts), `test/box-update-host.test.mjs` (runs the emitted updater in real bash against a live CP over real curl) | +| box config v1 | CP `core/box-config.ts` producer (`GET /workspaces/self/box-config`) and consumer (`POST /workspaces/self/box-update-result`) ↔ host updater bash/python emitted by `core/bootstrap.ts` (`blitz-box-update`, which shells out to the emitted `blitz-box-image` for the manifest install; cloud-VM path only — the microVM provider has its own guest lifecycle and no update path yet). The config carries `boxImageSha256`, the deployment's own pin of the whole archive, so the host's check does not rest on the digest the manifest declares about itself. The result body carries an optional `tag`: the CONCRETE image the container runs once the attempt settled, because `ref` alone cannot answer "is an update available" under a manifest pin whose URL never changes between rebakes | `fixtures/box-config/` | `test/box-config-conformance.test.ts` (CP), `test/box-update-conformance.test.mjs` (runs real `python3` over the emitted parser/producer, `bash -n` over the emitted scripts), `test/box-update-host.test.mjs` (runs the emitted updater in real bash against a live CP over real curl, including the manifest install and the credential rotation) | Retired 2026-08-29: the `ACP` contract (box actor ↔ ui chat reducer, `fixtures/acp/`). The native-chat surface, the box actor on port 7444, its @@ -203,6 +203,14 @@ The four rules a change must not break: read files at runtime. Its emitted bytes are a contract pinned by tests. Do not edit the emitted script casually. Extraction to build-time text imports is an approved future direction (see issue #1 discussion). +- **The emitted script has a hard 32 KiB budget.** Hetzner caps cloud-init + user-data at `HETZNER_USER_DATA_MAX_BYTES` and does not compress (AWS + gzips and has room to spare), so every byte of emitted bash costs. A heavy + manifest-mode create is ~30.4 KiB of that today and `test/bootstrap.test.ts` + pins a 2 KiB floor. Reasoning belongs in the TS comment, which never ships; + the emitted script carries only what bash must read. Buying real headroom + back means shipping the host scripts in the box image instead of in + user-data — see `plans/MEMBER-MACHINES.md` §5a. ## Box image: canary from R2, client prod from GHCR diff --git a/packages/control-plane/core/bootstrap.ts b/packages/control-plane/core/bootstrap.ts index 1e7197ce..0c98cc8c 100644 --- a/packages/control-plane/core/bootstrap.ts +++ b/packages/control-plane/core/bootstrap.ts @@ -85,14 +85,14 @@ export function recipeInvocationEnvFile(recipe: RecipeBootstrap): string { * segments pinned by `test/recipe-invocation-fixtures.test.ts`; a create * without a recipe or usage capture emits byte-identical output. */ /** - * The shell helpers `boxImageSetupScript` calls. Emitting that setup without - * these gives `retry: command not found`, and under `set -e` the script dies - * where it stands. The golden-image bake hit exactly that on its first real - * run: the builder never powered off, and the bake waited 30 minutes for a - * shutdown that could not come. + * The retry helper the REGISTRY branch of `boxImageSetupScript` calls. + * Emitting that setup without it gives `retry: command not found`, and under + * `set -e` the script dies where it stands. The golden-image bake hit exactly + * that on its first real run: the builder never powered off, and the bake + * waited 30 minutes for a shutdown that could not come. * - * `buildBootstrapScript` emits these in its own preamble. Any other caller - * that embeds the setup has to emit them first. + * Emit `boxImageSetupPreamble` rather than reaching for this directly; it is + * exported because the bake's own tests pin the bytes. */ export const BOX_IMAGE_SETUP_HELPERS = `retry() { local attempt=1 @@ -108,62 +108,43 @@ export const BOX_IMAGE_SETUP_HELPERS = `retry() { } `; -/** The three variables that name one box image build. */ -export interface BoxImageRef { - boxImageRef: string; - boxImageTag: string; - boxImageSha256: string; -} - -/** - * The bash that puts the box image into the host's docker store: download and - * `docker load` for an HTTPS tarball ref, or `docker pull` for a registry ref. - * Both branches guard on `docker image inspect`, so a host that already holds - * the image does no work at all. That guard is what makes a golden snapshot - * fast: the image is already there and the whole block is skipped. +/** `download` and `verify_sha256`, shared verbatim by the bootstrap's own + * image setup and by the host updater's manifest branch. * - * Exported so the golden-image bake script bakes the SAME bytes a workspace - * would download. Two copies of this would be two sides of one contract, and - * drift between them would produce snapshots holding the wrong image. - */ -export function boxImageSetupScript(options: BoxImageRef): string { - const isTarball = options.boxImageRef.startsWith("https://"); - if (isTarball && options.boxImageTag.trim() === "") { - throw new Error("BOX_IMAGE_TAG is required when BOX_IMAGE_REF is an HTTPS URL"); - } - if (isTarball && !/^[a-fA-F0-9]{64}$/u.test(options.boxImageSha256)) { - throw new Error( - "BOX_IMAGE_SHA256 must be a 64-character hexadecimal digest when BOX_IMAGE_REF is an HTTPS URL", - ); - } - return isTarball - ? String.raw`download() { - curl --fail --location --retry 10 --retry-all-errors --retry-delay 3 \ + * `verify_sha256` RETURNS non-zero instead of calling `fail`, because that is + * the one thing the two callers genuinely disagree about. The bootstrap has + * nothing to protect — no container is running yet — so a mismatch is fatal + * and it dies where it stands. The updater has a live workspace to protect, so + * a mismatch has to become a reported outcome with the old container still + * running. Each caller says `|| fail` or `|| report` for itself; the + * arithmetic that decides the verdict exists once. */ +export const BOX_IMAGE_DOWNLOAD_HELPERS = `download() { + curl --fail --location --retry 10 --retry-all-errors --retry-delay 3 \\ --silent --show-error --output "$2" "$1" } verify_sha256() { local path="$1" - local expected="$2" + local expected local actual actual=$(sha256sum "$path" | cut -d ' ' -f 1) - expected=$(printf '%s' "$expected" | tr 'A-F' 'a-f') - [ "$actual" = "$expected" ] || fail "SHA-256 mismatch for $path" + expected=$(printf '%s' "$2" | tr 'A-F' 'a-f') + [ "$actual" = "$expected" ] } +`; -if ! docker image inspect "$BOX_IMAGE_TAG" >/dev/null 2>&1; then -image_tmp_dir=$(mktemp -d /var/lib/blitz/.bootstrap-image.XXXXXX) -trap 'rm -rf "$image_tmp_dir"' EXIT -image_archive="$image_tmp_dir/image.tar.gz" - -case "$BOX_IMAGE_REF" in - */manifest.json) - manifest_path="$image_tmp_dir/manifest.json" - manifest_parts_path="$image_tmp_dir/parts.tsv" - manifest_metadata_path="$image_tmp_dir/metadata.tsv" - download "$BOX_IMAGE_REF" "$manifest_path" - python3 - "$manifest_path" "$manifest_parts_path" >"$manifest_metadata_path" <<'PYTHON' -import json +/** The box-image manifest validator, embedded in Python. + * + * Reads the manifest at argv[1], writes one `name\\tsha256` line per part to + * argv[2], and prints `totalSha256\\timageTag` on stdout. It is the consumer + * side of the `box-image manifest` contract + * (`packages/schema/fixtures/box-image-manifest/`). + * + * The bootstrap and the host updater embed these SAME bytes under two heredoc + * markers (`PYTHON` and `MANIFEST_PARSER`), because a manifest that two + * readers disagree about is two contracts. `test/box-update-conformance.test.mjs` + * pins the two copies equal. */ +export const BOX_IMAGE_MANIFEST_PARSER = `import json import re import sys @@ -191,33 +172,134 @@ with open(parts_path, "w", encoding="utf-8") as parts_file: raise ValueError("manifest part name is invalid") if not isinstance(sha256, str) or re.fullmatch(r"[a-fA-F0-9]{64}", sha256) is None: raise ValueError("manifest part sha256 must be a SHA-256 digest") - parts_file.write(f"{name}\t{sha256.lower()}\n") + parts_file.write(f"{name}\\t{sha256.lower()}\\n") -print(f"{total_sha256.lower()}\t{image_tag}") -PYTHON - IFS=$'\t' read -r manifest_total_sha256 manifest_image_tag <"$manifest_metadata_path" - [ "$manifest_image_tag" = "$BOX_IMAGE_TAG" ] || fail "manifest imageTag does not match BOX_IMAGE_TAG" - manifest_base=${"${BOX_IMAGE_REF%/*}"} - : >"$image_archive" - while IFS=$'\t' read -r part_name part_sha256; do - part_path="$image_tmp_dir/$part_name" - download "$manifest_base/$part_name" "$part_path" - verify_sha256 "$part_path" "$part_sha256" - cat "$part_path" >>"$image_archive" - rm -f "$part_path" - done <"$manifest_parts_path" - verify_sha256 "$image_archive" "$manifest_total_sha256" - ;; - *) - download "$BOX_IMAGE_REF" "$image_archive" - ;; -esac +print(f"{total_sha256.lower()}\\t{image_tag}") +`; -verify_sha256 "$image_archive" "$BOX_IMAGE_SHA256" -gunzip -c "$image_archive" | docker load -rm -rf "$image_tmp_dir" -trap - EXIT +/** The one host script that installs a box image from an R2 manifest. + * + * Two callers needed this job: the first boot, and the host updater when a + * user asks for an update. They were the same pipeline written twice — + * download every part, check each digest, concatenate, check the whole, load — + * and two copies of a verification pipeline are two chances to verify + * differently. Worse, both copies rode in cloud-init user-data, which Hetzner + * caps at 32 KiB (`HETZNER_USER_DATA_MAX_BYTES`); the duplicate parser and + * helpers alone cost about 1.8 KiB of that budget. + * + * So it is written to the host once and both callers invoke it. The exit codes + * are the interface: the updater turns them into the outcome vocabulary the + * box-config contract defines, and the bootstrap just dies. + * + * `BOX_IMAGE_SETUP_HELPERS` and this must both be emitted before + * `boxImageSetupScript` — `buildBootstrapScript` and the golden-image bake + * each do so in their own preamble. */ +export const BOX_IMAGE_INSTALLER = String.raw`cat >/usr/local/sbin/blitz-box-image <<'BOX_IMAGE_INSTALL' +#!/bin/bash +# resolve print the tag the manifest names +# install [sha256] multi-part manifest archive +# fetch one whole archive +# A tag already in the store is a no-op. Exit: 10 download, 11 digest, +# 12 load, 13 the manifest is invalid or names another tag. +set -Eeuo pipefail +${BOX_IMAGE_DOWNLOAD_HELPERS} +load_archive() { + gunzip -c "$archive" | docker load || exit 12 + docker image inspect "$1" >/dev/null 2>&1 || exit 12 +} + +action="${"${1:?usage: blitz-box-image [tag] [sha256]}"}" +url="${"${2:?}"}" +tmp=$(mktemp -d /var/lib/blitz/.box-image.XXXXXX) +trap 'rm -rf "$tmp"' EXIT +archive="$tmp/image.tar.gz" + +if [ "$action" = fetch ]; then + if docker image inspect "${"${3:?}"}" >/dev/null 2>&1; then exit 0; fi + download "$url" "$archive" || exit 10 + verify_sha256 "$archive" "${"${4:?}"}" || exit 11 + load_archive "$3" + exit 0 +fi + +download "$url" "$tmp/manifest.json" || exit 10 +python3 - "$tmp/manifest.json" "$tmp/parts.tsv" >"$tmp/meta.tsv" <<'MANIFEST_PARSER' || exit 13 +${BOX_IMAGE_MANIFEST_PARSER}MANIFEST_PARSER +IFS=$'\t' read -r manifest_total manifest_tag <"$tmp/meta.tsv" +if [ "$action" = resolve ]; then + printf '%s\n' "$manifest_tag" + exit 0 fi +[ "$manifest_tag" = "${"${3:?}"}" ] || exit 13 +if docker image inspect "$manifest_tag" >/dev/null 2>&1; then exit 0; fi +: >"$archive" +while IFS=$'\t' read -r part_name part_sha256; do + download "${"${url%/*}"}/$part_name" "$tmp/$part_name" || exit 10 + verify_sha256 "$tmp/$part_name" "$part_sha256" || exit 11 + cat "$tmp/$part_name" >>"$archive" + rm -f "$tmp/$part_name" +done <"$tmp/parts.tsv" +verify_sha256 "$archive" "$manifest_total" || exit 11 +# A caller carrying its own pinned digest checks that too, so a swapped +# manifest cannot redirect a boot to another image. +[ -z "${"${4:-}"}" ] || verify_sha256 "$archive" "$4" || exit 11 +load_archive "$manifest_tag" +BOX_IMAGE_INSTALL +chmod 0755 /usr/local/sbin/blitz-box-image +`; + +/** + * Everything that must be emitted before `boxImageSetupScript`, for the mode + * it is about to run in. `buildBootstrapScript` and the golden-image bake both + * call this, so neither can drift into emitting a helper the other does not. + * + * The tarball branch never calls `retry` and the registry branch never calls + * the installer, and every byte here is cloud-init user-data against a hard + * 32 KiB Hetzner cap — so each mode emits only what it runs. The installer + * stays in BOTH, because the host updater reaches for it whenever the control + * plane hands it a manifest ref, which can happen on a box whose deployment + * later moves from a registry pin to an R2 one. + */ +export function boxImageSetupPreamble(options: BoxImageRef): string { + const helpers = options.boxImageRef.startsWith("https://") ? "" : BOX_IMAGE_SETUP_HELPERS; + return `${helpers}${BOX_IMAGE_INSTALLER}`; +} + +/** The three variables that name one box image build. */ +export interface BoxImageRef { + boxImageRef: string; + boxImageTag: string; + boxImageSha256: string; +} + +/** + * The bash that puts the box image into the host's docker store: download and + * `docker load` for an HTTPS tarball ref, or `docker pull` for a registry ref. + * Both branches guard on `docker image inspect`, so a host that already holds + * the image does no work at all. That guard is what makes a golden snapshot + * fast: the image is already there and the whole block is skipped. + * + * Exported so the golden-image bake script bakes the SAME bytes a workspace + * would download. Two copies of this would be two sides of one contract, and + * drift between them would produce snapshots holding the wrong image. + */ +export function boxImageSetupScript(options: BoxImageRef): string { + const isTarball = options.boxImageRef.startsWith("https://"); + if (isTarball && options.boxImageTag.trim() === "") { + throw new Error("BOX_IMAGE_TAG is required when BOX_IMAGE_REF is an HTTPS URL"); + } + if (isTarball && !/^[a-fA-F0-9]{64}$/u.test(options.boxImageSha256)) { + throw new Error( + "BOX_IMAGE_SHA256 must be a 64-character hexadecimal digest when BOX_IMAGE_REF is an HTTPS URL", + ); + } + return isTarball + ? String.raw`case "$BOX_IMAGE_REF" in + */manifest.json) box_image_action=install ;; + *) box_image_action=fetch ;; +esac +/usr/local/sbin/blitz-box-image "$box_image_action" "$BOX_IMAGE_REF" "$BOX_IMAGE_TAG" "$BOX_IMAGE_SHA256" || + fail "box image install failed with exit $?" docker image inspect "$BOX_IMAGE_TAG" >/dev/null box_image="$BOX_IMAGE_TAG"` : String.raw`if ! docker image inspect "$BOX_IMAGE_REF" >/dev/null 2>&1; then @@ -446,7 +528,7 @@ touch "$BOOTSTRAP_LOG" chmod 0600 "$BOOTSTRAP_LOG" exec >>"$BOOTSTRAP_LOG" 2>&1 -${BOX_IMAGE_SETUP_HELPERS} +${boxImageSetupPreamble(options)} fail() { bootstrap_error="$*" echo "blitz bootstrap failed: $*" @@ -768,40 +850,96 @@ set -Eeuo pipefail readonly STATE_DIR=/var/lib/blitz readonly ORIGIN_PATH="$STATE_DIR/origin" readonly CREDENTIAL_PATH="$STATE_DIR/box-credential.json" +readonly CREDENTIAL_LOCK="$STATE_DIR/box-credential.lock" readonly UPDATE_LOG="$STATE_DIR/box-update.log" log() { printf '%s %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*" >>"$UPDATE_LOG" } -# A box that has not registered yet has neither file; that is the normal -# pre-enrollment state, not an error. [ -s "$CREDENTIAL_PATH" ] || exit 0 [ -s "$ORIGIN_PATH" ] || exit 0 current_origin=$(sed -n '1p' "$ORIGIN_PATH") [ -n "$current_origin" ] || exit 0 -access_token=$(python3 - "$CREDENTIAL_PATH" <<'CREDENTIAL_READER' -import json -import sys +# One non-empty string field out of the credential file, or a non-zero exit. +credential_field() { + python3 -c 'import json,sys +v=json.load(open(sys.argv[1]))[sys.argv[2]] +assert isinstance(v,str) and v +sys.stdout.write(v)' "$CREDENTIAL_PATH" "$1" +} -with open(sys.argv[1], encoding="utf-8") as credential_file: - value = json.load(credential_file) -token = value.get("access_token") if isinstance(value, dict) else None -if not isinstance(token, str) or token == "": - raise SystemExit("box credential has no access_token") -sys.stdout.write(token) -CREDENTIAL_READER -) || { log "skip: box credential is unreadable"; exit 0; } +access_token=$(credential_field access_token) || + { log "skip: box credential is unreadable"; exit 0; } + +rotate_credential() { + local refresh_token rotated_tmp + refresh_token=$(credential_field refresh_token) || return 1 + rotated_tmp=$(mktemp "$STATE_DIR/.box-credential.XXXXXX") + if ! curl --fail --silent --show-error --max-time 30 \ + --request POST \ + --data-urlencode "grant_type=refresh_token" \ + --data-urlencode "refresh_token=$refresh_token" \ + --output "$rotated_tmp" \ + "$current_origin/oauth/token"; then + rm -f "$rotated_tmp" + return 1 + fi + # The token response, narrowed to the three fields the credential file holds. + if ! python3 -c 'import json,sys +p=sys.argv[1];d=json.load(open(p)) +json.dump({k:d[k] for k in ("box_id","access_token","refresh_token")}, + open(p,"w"),separators=(",",":"))' "$rotated_tmp" + then + rm -f "$rotated_tmp" + return 1 + fi + chown 1000:1000 "$rotated_tmp" + chmod 0600 "$rotated_tmp" + mv "$rotated_tmp" "$CREDENTIAL_PATH" +} + +# Under the lock the in-container Go client also takes. +refresh_access_token() { + local before="$access_token" + ( + flock --exclusive --timeout 30 9 || exit 1 + current=$(credential_field access_token) || exit 1 + [ "$current" = "$before" ] || exit 0 + rotate_credential || exit 1 + ) 9>"$CREDENTIAL_LOCK" || { log "credential refresh failed; the next poll retries"; return 1; } + access_token=$(credential_field access_token) || return 1 + [ "$access_token" != "$before" ] || return 1 + log "credential refresh: the box access token was rotated" +} + +box_curl() { + curl --silent --show-error --max-time 30 --write-out '%{http_code}' \ + --header "Authorization: Bearer $access_token" \ + --output "$1" "${"${@:2}"}" +} + +# Rotates the credential once if the first attempt is refused. +authed_curl() { + local status + status=$(box_curl "$@") || return 1 + if [ "$status" = 401 ]; then + refresh_access_token || return 1 + status=$(box_curl "$@") || return 1 + fi + case "$status" in + 2??) return 0 ;; + *) return 1 ;; + esac +} config_tmp=$(mktemp "$STATE_DIR/.box-config.XXXXXX") result_tmp=$(mktemp "$STATE_DIR/.box-update-result.XXXXXX") trap 'rm -f "$config_tmp" "$result_tmp"' EXIT -if ! curl --fail --silent --show-error --max-time 30 \ - --header "Authorization: Bearer $access_token" \ +if ! authed_curl "$config_tmp" \ --header "Accept: application/json" \ - --output "$config_tmp" \ "$current_origin/workspaces/self/box-config"; then log "poll failed: $current_origin/workspaces/self/box-config did not answer" exit 0 @@ -817,21 +955,30 @@ with open(sys.argv[1], encoding="utf-8") as config_file: if not isinstance(value, dict): raise ValueError("box-config must be an object") ref = value.get("boxImageRef") +sha256 = value.get("boxImageSha256") origin = value.get("controlPlaneOrigin") update_requested = value.get("updateRequested") if not isinstance(ref, str) or re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9._/:@-]*", ref) is None: raise ValueError("box-config boxImageRef is invalid") +if not isinstance(sha256, str) or re.fullmatch(r"([a-fA-F0-9]{64})?", sha256) is None: + raise ValueError("box-config boxImageSha256 must be a SHA-256 digest or empty") if not isinstance(origin, str) or re.fullmatch(r"https?://[A-Za-z0-9.-]+(:[0-9]+)?", origin) is None: raise ValueError("box-config controlPlaneOrigin is not an origin") if not isinstance(update_requested, bool): raise ValueError("box-config updateRequested must be a boolean") -print(f"{ref}\t{origin}\t{'true' if update_requested else 'false'}") +print(f"{ref}\n{sha256.lower()}\n{origin}\n{'true' if update_requested else 'false'}") BOX_CONFIG_PARSER ) || { log "poll rejected: box-config response failed validation"; exit 0; } -IFS=$'\t' read -r next_ref next_origin update_requested <<<"$parsed" - -# Origin refresh, every poll. Safe with no restart: the gateway re-reads the -# file per request. This closes the stale-origin outage class for new boxes. +# One field per line, read with mapfile rather than read -r over a TSV: +# boxImageSha256 is empty under a registry pin, and TAB is IFS whitespace, so +# read would collapse the empty column and shift every field after it. +mapfile -t config_fields <<<"$parsed" +next_ref=${"${config_fields[0]}"} +next_sha256=${"${config_fields[1]}"} +next_origin=${"${config_fields[2]}"} +update_requested=${"${config_fields[3]}"} + +# No restart needed: the gateway re-reads this file per request. if [ "$next_origin" != "$current_origin" ]; then origin_tmp=$(mktemp "$STATE_DIR/.origin.XXXXXX") printf '%s\n' "$next_origin" >"$origin_tmp" @@ -840,33 +987,38 @@ if [ "$next_origin" != "$current_origin" ]; then mv "$origin_tmp" "$ORIGIN_PATH" log "origin refreshed: the box gateway now trusts $next_origin" fi +current_origin="$next_origin" [ "$update_requested" = true ] || exit 0 report_result() { - python3 - "$1" "$2" <<'RESULT_WRITER' >"$result_tmp" + local ref="$1" + local outcome="$2" + local running + # The image running NOW; empty means none, and the key is omitted. + running=$(docker inspect --format '{{.Config.Image}}' blitz-box 2>/dev/null || true) + python3 - "$ref" "$outcome" "$running" <<'RESULT_WRITER' >"$result_tmp" import json import sys -ref, outcome = sys.argv[1:] -json.dump({"ref": ref, "outcome": outcome}, sys.stdout, separators=(",", ":")) +ref, outcome, tag = sys.argv[1:] +body = {"ref": ref, "outcome": outcome} +if tag: + body["tag"] = tag +json.dump(body, sys.stdout, separators=(",", ":")) RESULT_WRITER - if curl --fail --silent --show-error --max-time 30 \ + if authed_curl /dev/null \ --request POST \ - --header "Authorization: Bearer $access_token" \ --header "Content-Type: application/json" \ --data-binary @"$result_tmp" \ - --output /dev/null \ - "$next_origin/workspaces/self/box-update-result"; then - log "reported outcome $2 for $1" + "$current_origin/workspaces/self/box-update-result"; then + log "reported outcome $outcome for $ref" else - log "outcome report failed for $1 ($2); the update flag stays set until a report lands" + log "outcome report failed for $ref ($outcome); the update flag stays set until a report lands" fi } start_box() { - # blitz-box-run owns the whole start, including refreshing the container env - # from the image it is about to run. /usr/local/bin/blitz-box-run "$1" >>"$UPDATE_LOG" 2>&1 || return 1 local deadline=$((SECONDS + 60)) while (( SECONDS < deadline )); do @@ -879,31 +1031,59 @@ start_box() { } current_image=$(docker inspect --format '{{.Config.Image}}' blitz-box 2>/dev/null || true) -if [ "$next_ref" = "$current_image" ]; then - log "update requested but the requested ref is already running; clearing the request" - report_result "$next_ref" up-to-date - exit 0 -fi + +# A manifest URL names its image inside itself, so the tag is only known after +# resolving it — which is why the up-to-date check sits below, not here. +manifest_mode=false case "$next_ref" in + https://*/manifest.json) + manifest_mode=true + next_image=$(/usr/local/sbin/blitz-box-image resolve "$next_ref") || { + log "update failed: the manifest did not resolve; the container is untouched" + report_result "$next_ref" download-failed + exit 0 + } + ;; https://*) - # Tarball pins ride the bootstrap's manifest download path, which this - # updater does not carry. Report it so the flag clears. - log "update refused: a tarball ref cannot be pulled in place" + log "update refused: an https ref that is not a manifest cannot be installed" report_result "$next_ref" unsupported exit 0 ;; + *) + next_image="$next_ref" + ;; esac -log "update start: [$current_image] -> [$next_ref]" -# Pull FIRST: a failed pull must leave the old container running untouched. -if ! docker pull "$next_ref" >>"$UPDATE_LOG" 2>&1; then +if [ "$next_image" = "$current_image" ]; then + log "update requested but [$next_image] is already running; clearing the request" + report_result "$next_ref" up-to-date + exit 0 +fi + +log "update start: [$current_image] -> [$next_image]" +# Install FIRST: a failed install must leave the old container running. +if [ "$manifest_mode" = true ]; then + install_status=0 + /usr/local/sbin/blitz-box-image install "$next_ref" "$next_image" "$next_sha256" \ + >>"$UPDATE_LOG" 2>&1 || install_status=$? + if [ "$install_status" != 0 ]; then + case "$install_status" in + 11) install_outcome=digest-mismatch ;; + 12) install_outcome=load-failed ;; + *) install_outcome=download-failed ;; + esac + log "update failed: image install exited $install_status; the container is untouched" + report_result "$next_ref" "$install_outcome" + exit 0 + fi +elif ! docker pull "$next_image" >>"$UPDATE_LOG" 2>&1; then log "update failed: pull did not complete; the running container is untouched" report_result "$next_ref" pull-failed exit 0 fi docker rm -f blitz-box >>"$UPDATE_LOG" 2>&1 || true -if start_box "$next_ref"; then - log "update complete: blitz-box now runs [$next_ref]" +if start_box "$next_image"; then + log "update complete: blitz-box now runs [$next_image]" report_result "$next_ref" updated exit 0 fi diff --git a/packages/control-plane/core/box-config.ts b/packages/control-plane/core/box-config.ts index 5c172420..db804a11 100644 --- a/packages/control-plane/core/box-config.ts +++ b/packages/control-plane/core/box-config.ts @@ -5,17 +5,19 @@ import type { BoxIdentity } from "./types.js"; import type { Principal } from "./principals.js"; import type { CoreContext, CoreRouter, CoreRuntime, RuntimeFactory } from "./runtime.js"; import { machineFor } from "./machines.js"; +import { machineById, machineTarget } from "./machine-access.js"; import { isWorkspaceAdmin, isWorkspaceMember, workspaceAccess, } from "./workspace-access.js"; -import { workspaceById } from "./workspace-records.js"; +import { machineView, workspaceById } from "./workspace-records.js"; import { BOX_UPDATE_OUTCOMES, type BoxConfigResponse, type BoxUpdateOutcome, type BoxUpdateResultRequest, + type MachineResponse, } from "./wire.js"; // The box-config contract (see wire.ts and packages/schema/fixtures/box-config/): @@ -51,6 +53,23 @@ export function controlPlaneOriginFromEnv(value: string | null | undefined): str } } +/** The CONCRETE box image this deployment installs now. + * + * Two modes, one answer. Under a registry pin (`BOX_IMAGE_REF` is a + * `ghcr.io/...` ref) the ref IS the image and `BOX_IMAGE_TAG` is empty. Under + * an R2 manifest pin the ref is a `https://.../manifest.json` URL that is + * byte-identical across rebakes, and `BOX_IMAGE_TAG` is the tag that actually + * moves — so the tag is the only thing worth comparing a machine against. + * This is the value the host reports back as `tag`, from the other end of the + * same two modes. */ +export function deploymentBoxImage(vars: { + boxImageRef: string; + boxImageTag: string; +}): string { + const tag = vars.boxImageTag.trim(); + return tag === "" ? vars.boxImageRef : tag; +} + function boxUpdateOutcome(value: string): BoxUpdateOutcome { const outcome = BOX_UPDATE_OUTCOMES.find((known) => known === value); if (outcome === undefined) { @@ -62,12 +81,20 @@ function boxUpdateOutcome(value: string): BoxUpdateOutcome { /** Accepts iff `ref` is one image-reference token and `outcome` is a known * verdict. Unknown extra keys are tolerated on purpose: hosts only update by * shipping new images, so an older control plane must keep accepting a newer - * host's report or the update flag would stay set forever. */ + * host's report or the update flag would stay set forever. + * + * `tag` is optional for the same reason in the other direction: a host emitted + * before the manifest branch reports only `ref`. Present, it must be one image + * reference token like `ref` is — it is the image the container now runs. */ export function parseBoxUpdateResult(value: JsonValue): BoxUpdateResultRequest { if (!isRecord(value)) throw new HttpError(400, "request body must be an object"); const ref = requiredString(value.ref, "ref", 512); if (!IMAGE_REF.test(ref)) throw new HttpError(400, "ref must be an image reference"); - return { ref, outcome: boxUpdateOutcome(requiredString(value.outcome, "outcome", 64)) }; + const outcome = boxUpdateOutcome(requiredString(value.outcome, "outcome", 64)); + if (value.tag === undefined) return { ref, outcome }; + const tag = requiredString(value.tag, "tag", 512); + if (!IMAGE_REF.test(tag)) throw new HttpError(400, "tag must be an image reference"); + return { ref, outcome, tag }; } /** Arm the flag the host's next poll reads. The flag is per MACHINE now: a @@ -112,6 +139,7 @@ export function addBoxConfigRoutes( if (row === null) throw new HttpError(404, "machine not found"); const response: BoxConfigResponse = { boxImageRef: runtime.vars.boxImageRef, + boxImageSha256: runtime.vars.boxImageSha256, // The configured public origin when the deployment has one; otherwise // the origin this request arrived on. The fallback keeps a fresh // self-host working before APP_URL is filled in, but only the @@ -130,11 +158,16 @@ export function addBoxConfigRoutes( router.post("/workspaces/self/box-update-result", async (context) => { const box = await requireWorkspaceBox(context, runtimeFactory); const input = parseBoxUpdateResult(await readJson(context.req.raw, 4 * 1024)); + // A host that sends no `tag` leaves the stored one alone rather than + // nulling it: it predates the field, and what the row already holds — the + // image the boot was armed with — stays the best answer available. await changed(runtimeFactory(context).db, { q: `UPDATE machines - SET box_update_requested = 0, box_image_reported = ?1, updated_at = ?2 + SET box_update_requested = 0, box_image_reported = ?1, + box_image_tag_reported = COALESCE(?4, box_image_tag_reported), + box_update_outcome = ?5, updated_at = ?2 WHERE id = ?3 RETURNING id`, - v: [input.ref, Date.now(), box.id], + v: [input.ref, Date.now(), box.id, input.tag ?? null, input.outcome], }); return context.body(null, 204); }); @@ -147,8 +180,43 @@ export function addBoxConfigRoutes( return context.body(null, 204); }); - // Session-authenticated request path for the UI/API. No webapp UI consumes - // it yet; the gate is the same one destroy uses. + // The "Update machine" button in the My machine dialog. + // + // It names ONE machine, and it is registered here rather than in + // `core/machines.ts` because the flag and its contract live here — but it + // takes the same `/machines/:machineId/` shape and the same gate as + // the lifecycle verbs beside it (`core/machine-access.ts`, scope `own`), so + // a member may update their own machine and nobody else's. + // + // The workspace-level route below is the fan-out an admin asks for + // deliberately. This one can never become that by accident, which matters: + // replacing a container kills every process inside it, and a user who + // clicked a button in their own machine dialog did not ask to restart their + // colleagues' work. It answers with the machine, so the dialog can render + // the pending flag without waiting for the next poll. + router.post("/machines/:machineId/box-update", async (context) => { + const principal = await requirePrincipal(context); + const runtime = runtimeFactory(context); + const { machine } = await machineTarget( + runtime.db, + principal, + context.req.param("machineId"), + "own", + ); + if (machine.state === "destroyed" || machine.state === "destroying") { + throw new HttpError(409, `machine is ${machine.state}`); + } + await requestBoxUpdate(runtime.db, machine.id); + const updated = await machineById(runtime.db, machine.id); + if (updated === null) throw new HttpError(404, "machine not found"); + return context.json({ + machine: machineView(updated, deploymentBoxImage(runtime.vars)), + }); + }); + + // Session-authenticated request path for the UI/API. The webapp uses the + // per-machine route above; this one stays the deliberate workspace-wide + // fan-out, and the gate is the same one destroy uses. router.post("/workspaces/:id/box-update", async (context) => { const principal = await requirePrincipal(context); const runtime = runtimeFactory(context); diff --git a/packages/control-plane/core/machine-access.ts b/packages/control-plane/core/machine-access.ts new file mode 100644 index 00000000..4515b90c --- /dev/null +++ b/packages/control-plane/core/machine-access.ts @@ -0,0 +1,61 @@ +/** Resolving the machine a route names, and grading the caller against + * plans/MEMBER-MACHINES.md §3. + * + * Split out of `core/machines.ts` rather than added to it: that file sits on + * the 700-line warn, and the house rule is to split on touch. The split earns + * itself twice over — `core/box-config.ts` registers a machine verb of its own + * (`POST /machines/:machineId/box-update`) and needs the identical gate, and a + * second copy of an authorization rule is how the two drift apart. */ + +import { HttpError } from "./http.js"; +import type { Db } from "./db.js"; +import type { Principal } from "./principals.js"; +import { isWorkspaceAdmin, workspaceAccess } from "./workspace-access.js"; +import { workspaceById } from "./workspace-records.js"; +import type { MachineRow, WorkspaceRow } from "./workspace-records.js"; +import { first } from "./db.js"; + +export interface MachineTarget { + workspace: WorkspaceRow; + machine: MachineRow; + admin: boolean; +} + +export async function machineById(db: Db, id: string): Promise { + return first(db, { + q: "SELECT * FROM machines WHERE id = ?1 LIMIT 1", + v: [id], + }); +} + +/** Resolves the machine a verb names and grades the caller. + * + * `own` is what a plain member may do to their own machine (stop, start, ask + * for a box update). Everything else is workspace-admin work, and an org admin + * passes through implicit reach. + * + * A machine in another org reads as 404, not 403: whether an id exists + * elsewhere is not this caller's business. + */ +export async function machineTarget( + db: Db, + principal: Principal, + machineId: string, + scope: "admin" | "own", +): Promise { + const machine = await machineById(db, machineId); + if (machine === null) throw new HttpError(404, "machine not found"); + const workspace = await workspaceById(db, machine.workspace_id); + if (workspace === null || workspace.org_id !== principal.orgId) { + throw new HttpError(404, "machine not found"); + } + const access = await workspaceAccess(db, principal, workspace); + const admin = isWorkspaceAdmin(access); + if (!admin) { + if (scope === "admin") throw new HttpError(403, "workspace admin required"); + if (access.stored !== "member" || machine.membership_id !== principal.membershipId) { + throw new HttpError(403, "forbidden"); + } + } + return { workspace, machine, admin }; +} diff --git a/packages/control-plane/core/machines.ts b/packages/control-plane/core/machines.ts index d8fcc5d8..bf6b73c2 100644 --- a/packages/control-plane/core/machines.ts +++ b/packages/control-plane/core/machines.ts @@ -1,4 +1,6 @@ import { boxHostname, type RecipeBootstrap } from "./bootstrap.js"; +import { deploymentBoxImage } from "./box-config.js"; +import { machineById, machineTarget, type MachineTarget } from "./machine-access.js"; import { buildUserData, type BootShaping } from "./cloud-init.js"; import { revokeMachineLeasesQuery } from "./connections/leases.js"; import { hashSecret, randomToken } from "./crypto.js"; @@ -49,10 +51,6 @@ export function providerOperationError(error: unknown): string { return detail === "" ? "provider operation failed" : `provider operation failed: ${detail}`; } -export async function machineById(db: Db, id: string): Promise { - return first(db, { q: "SELECT * FROM machines WHERE id = ?1 LIMIT 1", v: [id] }); -} - export async function machineFor( db: Db, workspaceId: string, @@ -76,14 +74,29 @@ function revokeMachineTokensQuery(machineId: string): Query { /** Arms a fresh phone-home capability for the next boot. The capability is * per machine and re-armed at every VM provision, so a stale URL from a - * previous incarnation cannot enroll a new one. */ -async function armPhoneHome(db: Db, machineId: string, now: number): Promise { + * previous incarnation cannot enroll a new one. + * + * It also records the box image that boot will install. This is the one moment + * the answer is known exactly: the same `runtime.vars` that renders the + * user-data a few lines below decides both, so the row cannot claim an image + * the boot was never handed. Recording it here is what lets the UI answer "is + * an update available" for a machine that has never been asked to update — the + * host only reports after an attempt, and without this the answer would stay + * unknown until someone made one. The previous incarnation's update verdict is + * cleared with it: a fresh VM has attempted nothing. */ +async function armPhoneHome( + db: Db, + machineId: string, + now: number, + boxImage: string, +): Promise { const capability = randomToken(); await rows(db, { q: `UPDATE machines - SET phone_home_hash = ?1, phone_home_used = 0, updated_at = ?2 + SET phone_home_hash = ?1, phone_home_used = 0, + box_image_tag_reported = ?4, box_update_outcome = NULL, updated_at = ?2 WHERE id = ?3`, - v: [await hashSecret(capability), now, machineId], + v: [await hashSecret(capability), now, machineId, boxImage], }); return capability; } @@ -216,7 +229,7 @@ export async function provisionMachine( }); } - const capability = await armPhoneHome(runtime.db, id, now); + const capability = await armPhoneHome(runtime.db, id, now, deploymentBoxImage(runtime.vars)); // The URL keeps its workspace shape. Every deployed guest holds one it was // handed at creation and never updates it, and the route resolves the // machine by matching the capability hash, so one route serves both. @@ -522,43 +535,26 @@ function reprovisionInput( return input; } -interface MachineTarget { - workspace: WorkspaceRow; - machine: MachineRow; - admin: boolean; -} - export function addMachineRoutes( router: CoreRouter, runtimeFactory: RuntimeFactory, requirePrincipal: (context: CoreContext) => Promise, ): void { - /** Resolves the machine a verb names and grades the caller against §3. - * - * `own` is what a plain member may do to their own machine (stop, start). - * Everything else is workspace-admin work, and an org admin passes through - * implicit reach. */ + /** The shared gate in `core/machine-access.ts`, with the principal this + * request resolved to carried alongside it. */ async function target( context: CoreContext, runtime: CoreRuntime, scope: "admin" | "own", ): Promise { const principal = await requirePrincipal(context); - const machine = await machineById(runtime.db, context.req.param("machineId")); - if (machine === null) throw new HttpError(404, "machine not found"); - const workspace = await workspaceById(runtime.db, machine.workspace_id); - if (workspace === null || workspace.org_id !== principal.orgId) { - throw new HttpError(404, "machine not found"); - } - const access = await workspaceAccess(runtime.db, principal, workspace); - const admin = isWorkspaceAdmin(access); - if (!admin) { - if (scope === "admin") throw new HttpError(403, "workspace admin required"); - if (access.stored !== "member" || machine.membership_id !== principal.membershipId) { - throw new HttpError(403, "forbidden"); - } - } - return { workspace, machine, admin, principal }; + const resolved = await machineTarget( + runtime.db, + principal, + context.req.param("machineId"), + scope, + ); + return { ...resolved, principal }; } /** Brings up a machine that has no VM. A member whose workspace has @@ -576,7 +572,7 @@ export function addMachineRoutes( machine.machine_type_id, new URL(context.req.url).origin, )); - return context.json({ machine: machineView(provisioned) }); + return context.json({ machine: machineView(provisioned, deploymentBoxImage(runtime.vars)) }); }); /** Stop keeps the disk and the machine row. The VM goes, because a stopped @@ -586,20 +582,20 @@ export function addMachineRoutes( const runtime = runtimeFactory(context); const { machine } = await target(context, runtime, "own"); if (machine.state === "stopped") { - return context.json({ machine: machineView(machine) }); + return context.json({ machine: machineView(machine, deploymentBoxImage(runtime.vars)) }); } if (!LIVE_STATES.includes(machine.state)) { throw new HttpError(409, `machine is ${machine.state}`); } const stopped = await destroyMachine(runtime, machine, { keepRow: true }); - return context.json({ machine: machineView(stopped) }); + return context.json({ machine: machineView(stopped, deploymentBoxImage(runtime.vars)) }); }); router.post("/machines/:machineId/start", async (context) => { const runtime = runtimeFactory(context); const { workspace, machine } = await target(context, runtime, "own"); if (machine.vm_id !== null) { - return context.json({ machine: machineView(machine) }); + return context.json({ machine: machineView(machine, deploymentBoxImage(runtime.vars)) }); } if (machine.state === "destroying" || machine.state === "destroyed") { throw new HttpError(409, `machine is ${machine.state}`); @@ -610,7 +606,7 @@ export function addMachineRoutes( machine.machine_type_id, new URL(context.req.url).origin, )); - return context.json({ machine: machineView(started) }); + return context.json({ machine: machineView(started, deploymentBoxImage(runtime.vars)) }); }); /** Replaces the VM on the same volume. Sessions restart; disk state @@ -627,7 +623,7 @@ export function addMachineRoutes( machine.machine_type_id, new URL(context.req.url).origin, )); - return context.json({ machine: machineView(recreated) }); + return context.json({ machine: machineView(recreated, deploymentBoxImage(runtime.vars)) }); }); /** @@ -643,7 +639,7 @@ export function addMachineRoutes( const { workspace, machine } = await target(context, runtime, "admin"); const input = parseSetMachineType(await readJson(context.req.raw, 4 * 1024)); if (input.machineTypeId === machine.machine_type_id) { - return context.json({ machine: machineView(machine) }); + return context.json({ machine: machineView(machine, deploymentBoxImage(runtime.vars)) }); } if (machine.state === "destroying" || machine.state === "destroyed") { throw new HttpError(409, `machine is ${machine.state}`); @@ -672,18 +668,20 @@ export function addMachineRoutes( input.machineTypeId, new URL(context.req.url).origin, )); - return context.json({ machine: machineView(changed) }); + return context.json({ machine: machineView(changed, deploymentBoxImage(runtime.vars)) }); }); router.delete("/machines/:machineId", async (context) => { const runtime = runtimeFactory(context); const { machine } = await target(context, runtime, "admin"); if (machine.state === "destroyed") { - return context.json({ machine: machineView(machine) }); + return context.json({ machine: machineView(machine, deploymentBoxImage(runtime.vars)) }); } const destroyed = await destroyMachine(runtime, machine); - return context.json({ machine: machineView(destroyed) }); + return context.json({ machine: machineView(destroyed, deploymentBoxImage(runtime.vars)) }); }); } export { requireWorkspaceAdmin }; + +export { machineById } from "./machine-access.js"; diff --git a/packages/control-plane/core/wire-box-config.ts b/packages/control-plane/core/wire-box-config.ts new file mode 100644 index 00000000..b0026d62 --- /dev/null +++ b/packages/control-plane/core/wire-box-config.ts @@ -0,0 +1,85 @@ +/** The `box config v1` wire vocabulary (CLAUDE.md cross-runtime contracts). + * + * Split out of `core/wire.ts` for the same reason `wire-machines.ts` was: the + * machine view now names a `BoxUpdateOutcome`, and a type-only import back + * from `wire.ts` would close a cycle between the two halves of one mirror. + * `core/wire.ts` re-exports every name here, so nothing else has to know about + * the seam. Its mirror is `packages/schema/src/box-config.ts`, held equal by + * `test/wire-drift.test.ts`. */ + +/** The envelope `GET /workspaces/self/box-config` returns to the VM host. + * + * This crosses a runtime boundary: the producer is the control-plane Worker + * and the consumer is the host-side updater bash/python emitted by + * `core/bootstrap.ts` (`blitz-box-update`). Both are pinned to + * `packages/schema/fixtures/box-config/`. + * + * `boxImageRef` is the deployment's current pin (`runtime.vars.boxImageRef`). + * `controlPlaneOrigin` is the one origin the box gateway should trust; the + * host rewrites `/var/lib/blitz/origin` on every poll when it differs, which + * needs no restart because the gateway re-reads the file per request. + * `updateRequested` is the per-machine flag; image updates are request-gated + * because replacing the container kills every process inside it. + * + * `boxImageSha256` is the digest of the whole image archive, and it is what + * makes the host's verification as strong as the first boot's. Without it the + * updater could only check the archive against the digest the MANIFEST + * declares, which is self-certifying: whoever serves the manifest serves the + * digest beside it. This one arrives from the control plane instead, over a + * separate connection, exactly as the bootstrap's baked-in `BOX_IMAGE_SHA256` + * does. Empty under a registry pin, where the ref carries its own digest and + * docker checks it. */ +export interface BoxConfigResponse { + boxImageRef: string; + boxImageSha256: string; + controlPlaneOrigin: string; + updateRequested: boolean; +} + +/** What the host reports after an update attempt. + * + * Acquiring the image comes first and never touches the running container, so + * every acquire verdict means "nothing changed": `pull-failed` (a registry + * pull failed), `download-failed` (a manifest or archive part did not arrive), + * `digest-mismatch` (a part or the whole archive failed its SHA-256 — + * a corrupt or tampered archive, never loaded), `load-failed` (the archive + * verified but `docker load` refused it). + * + * The rest describe the swap: `up-to-date` (the wanted image already runs, + * nothing replaced), `updated` (the new container runs), `rolled-back` (the + * new container did not start and the old image runs again), `start-failed` + * (neither started), `unsupported` (this host's updater cannot install the + * ref at all — an https ref that is not a manifest, or any https ref on a host + * whose updater predates the manifest branch). */ +export const BOX_UPDATE_OUTCOMES = [ + "updated", + "up-to-date", + "rolled-back", + "pull-failed", + "download-failed", + "digest-mismatch", + "load-failed", + "start-failed", + "unsupported", +] as const; + +export type BoxUpdateOutcome = (typeof BOX_UPDATE_OUTCOMES)[number]; + +/** The body of `POST /workspaces/self/box-update-result`: the host is the + * producer (bash/python in the emitted updater), the control plane is the + * consumer. The control plane clears the machine's update flag and stores + * `ref` on the row (`box_image_reported`) whatever the outcome, so a failed + * attempt never leaves the flag re-triggering forever. + * + * `tag` is the CONCRETE image the `blitz-box` container runs once the attempt + * has settled — the tag under an R2 manifest ref, the ref itself under a + * registry ref, and the OLD image whenever the attempt left the container + * alone. It is optional because a host emitted before the manifest branch + * never sends it; `ref` alone cannot answer "is an update available" under a + * manifest ref, whose URL is identical across rebakes while the tag inside it + * moves. */ +export interface BoxUpdateResultRequest { + ref: string; + outcome: BoxUpdateOutcome; + tag?: string; +} diff --git a/packages/control-plane/core/wire-machines.ts b/packages/control-plane/core/wire-machines.ts index 53dc4aeb..5106ccf3 100644 --- a/packages/control-plane/core/wire-machines.ts +++ b/packages/control-plane/core/wire-machines.ts @@ -6,6 +6,8 @@ * Its mirror is `packages/schema/src/workspace.ts`, held equal by * `test/wire-drift.test.ts`. */ +import type { BoxUpdateOutcome } from "./wire-box-config.js"; + /** The stored workspace role (plans/MEMBER-MACHINES.md §3). `admin` here is * workspace admin, which is not the org role of the same name: an org admin * reaches every workspace of the org implicitly without holding a row. */ @@ -40,6 +42,23 @@ export interface MachineView { volumeUsedPercent: number | null; membershipId: string; error: string | null; + /** The CONCRETE box image this machine runs, as its host last reported it + * (or as the deployment pinned it when the machine was created). Null means + * unknown: a machine created before the host started reporting a tag, which + * has not attempted an update since. Never compare `boxImage` to a manifest + * URL — under an R2 manifest ref the URL is identical across rebakes while + * the tag inside it moves, and the tag is what this field holds. */ + boxImage: string | null; + /** The CONCRETE box image this deployment installs now. Equal to `boxImage` + * means up to date; different means an update is available. */ + boxImageTarget: string; + /** An update has been asked for and the host has not reported back yet. The + * host polls every five minutes. */ + boxUpdateRequested: boolean; + /** How the host's last update attempt ended, or null if it never made one. + * `unsupported` is the honest signal that this host's updater predates the + * manifest branch and can never self-update. */ + boxUpdateOutcome: BoxUpdateOutcome | null; createdAt: number; updatedAt: number; } diff --git a/packages/control-plane/core/wire.ts b/packages/control-plane/core/wire.ts index 2c38ff30..092c44dd 100644 --- a/packages/control-plane/core/wire.ts +++ b/packages/control-plane/core/wire.ts @@ -104,51 +104,15 @@ export interface PutAgentRuleResponse { rule: AgentRuleView; } -/** The envelope `GET /workspaces/self/box-config` returns to the VM host. - * - * This crosses a runtime boundary: the producer is the control-plane Worker - * and the consumer is the host-side updater bash/python emitted by - * `core/bootstrap.ts` (`blitz-box-update`). Both are pinned to - * `packages/schema/fixtures/box-config/`. - * - * `boxImageRef` is the deployment's current pin (`runtime.vars.boxImageRef`). - * `controlPlaneOrigin` is the one origin the box gateway should trust; the - * host rewrites `/var/lib/blitz/origin` on every poll when it differs, which - * needs no restart because the gateway re-reads the file per request. - * `updateRequested` is the per-workspace flag; image updates are request-gated - * because replacing the container kills every process inside it. */ -export interface BoxConfigResponse { - boxImageRef: string; - controlPlaneOrigin: string; - updateRequested: boolean; -} - -/** What the host reports after an update attempt, in the order it tries: - * `up-to-date` (the requested ref already runs, nothing replaced), - * `unsupported` (a tarball https ref, which the updater cannot pull), - * `pull-failed` (the pull failed; the old container was never touched), - * `updated` (the new container runs), `rolled-back` (the new container did - * not start and the old ref runs again), `start-failed` (neither started). */ -export const BOX_UPDATE_OUTCOMES = [ - "updated", - "up-to-date", - "rolled-back", - "pull-failed", - "start-failed", - "unsupported", -] as const; - -export type BoxUpdateOutcome = (typeof BOX_UPDATE_OUTCOMES)[number]; - -/** The body of `POST /workspaces/self/box-update-result`: the host is the - * producer (bash/python in the emitted updater), the control plane is the - * consumer. The control plane clears the workspace's update flag and stores - * `ref` on the row (`box_image_reported`) whatever the outcome, so a failed - * attempt never leaves the flag re-triggering forever. */ -export interface BoxUpdateResultRequest { - ref: string; - outcome: BoxUpdateOutcome; -} +// The box-config vocabulary lives in its own module and is re-exported here, +// so `wire-machines.ts` can name `BoxUpdateOutcome` without importing back +// through this file. See `core/wire-box-config.ts`. +export { + BOX_UPDATE_OUTCOMES, + type BoxConfigResponse, + type BoxUpdateOutcome, + type BoxUpdateResultRequest, +} from "./wire-box-config.js"; export const PHASES = [ "creating", diff --git a/packages/control-plane/core/workspace-members.ts b/packages/control-plane/core/workspace-members.ts index b20ad337..534b7e75 100644 --- a/packages/control-plane/core/workspace-members.ts +++ b/packages/control-plane/core/workspace-members.ts @@ -1,3 +1,4 @@ +import { deploymentBoxImage } from "./box-config.js"; import { first, rows } from "./db.js"; import { HttpError, @@ -167,7 +168,7 @@ export async function addWorkspaceMember( name: member.name, avatarUrl: member.avatar_url, role: input.role, - machine: machine === null || machine.state === "destroyed" ? null : machineView(machine), + machine: machine === null || machine.state === "destroyed" ? null : machineView(machine, deploymentBoxImage(runtime.vars)), }; } @@ -242,7 +243,7 @@ export function addWorkspaceMemberRoutes( name: member.name, avatarUrl: member.avatar_url, role: input.role, - machine: machine === null || machine.state === "destroyed" ? null : machineView(machine), + machine: machine === null || machine.state === "destroyed" ? null : machineView(machine, deploymentBoxImage(runtime.vars)), }, }); }); @@ -293,7 +294,7 @@ export function addWorkspaceMemberRoutes( name: member.name, avatarUrl: member.avatar_url, role, - machine: machineView(machine), + machine: machineView(machine, deploymentBoxImage(runtime.vars)), }, }, 201); }); diff --git a/packages/control-plane/core/workspace-projection.ts b/packages/control-plane/core/workspace-projection.ts index f622cd28..83a1cde3 100644 --- a/packages/control-plane/core/workspace-projection.ts +++ b/packages/control-plane/core/workspace-projection.ts @@ -1,6 +1,7 @@ -import type { Db } from "./db.js"; import { rows } from "./db.js"; +import { deploymentBoxImage } from "./box-config.js"; import type { Principal } from "./principals.js"; +import type { CoreRuntime } from "./runtime.js"; import { accessFor, legacyRole } from "./workspace-access.js"; import { machinesForWorkspaces, @@ -31,13 +32,20 @@ function groupBy(items: readonly T[], key: (item: T) => string): Map { if (workspaces.length === 0) return []; + const db = runtime.db; + const boxImageTarget = deploymentBoxImage(runtime.vars); const ids = workspaces.map(({ id }) => id); const placeholders = ids.map((_id, index) => `?${String(index + 1)}`).join(", "); const [members, machines, credentials] = await Promise.all([ @@ -72,17 +80,18 @@ export async function projectWorkspaces( membershipId: principal.membershipId, myRole: stored, role: legacyRole(access), + boxImageTarget, })); } return views; } export async function projectWorkspace( - db: Db, + runtime: CoreRuntime, principal: Principal, workspace: WorkspaceRow, ): Promise { - const [view] = await projectWorkspaces(db, principal, [workspace]); + const [view] = await projectWorkspaces(runtime, principal, [workspace]); if (view === undefined) throw new Error("workspace projection produced no view"); return view; } diff --git a/packages/control-plane/core/workspace-records.ts b/packages/control-plane/core/workspace-records.ts index 0aa3d938..a29a69de 100644 --- a/packages/control-plane/core/workspace-records.ts +++ b/packages/control-plane/core/workspace-records.ts @@ -2,7 +2,9 @@ import type { Db } from "./db.js"; import { first, rows } from "./db.js"; import { isMicrovmProviderId } from "./compute/microvm.js"; import { manifestConnectionNames } from "./connections/manifest.js"; +import { BOX_UPDATE_OUTCOMES } from "./wire.js"; import type { + BoxUpdateOutcome, MachineState, MachineView, Phase, @@ -60,6 +62,8 @@ export interface MachineRow { broker_box_id: string | null; box_update_requested: number; box_image_reported: string | null; + box_image_tag_reported: string | null; + box_update_outcome: string | null; disk_used_percent: number | null; disk_reported_at: number | null; error: string | null; @@ -122,7 +126,21 @@ function volumeUsedPercentForRow(row: MachineRow): number | null { return row.volume_id === null ? null : row.disk_used_percent; } -export function machineView(row: MachineRow): MachineView { +/** The host's last update verdict, or null when it never reported one. + * + * The column is plain TEXT, so a row written by a NEWER control plane than the + * one reading it can hold an outcome this build has no name for. Answering + * null there is the honest reading: "no verdict I understand" is closer to the + * truth than inventing one, and the UI already has to handle a machine that + * has never reported. */ +function boxUpdateOutcomeForRow(row: MachineRow): BoxUpdateOutcome | null { + return BOX_UPDATE_OUTCOMES.find((known) => known === row.box_update_outcome) ?? null; +} + +/** `boxImageTarget` is the deployment's pin, not a machine fact, so it is + * passed in rather than read from the row: the row records what the machine + * reports, and comparing the two is what answers "is an update available". */ +export function machineView(row: MachineRow, boxImageTarget: string): MachineView { return { id: row.id, state: row.state, @@ -131,6 +149,10 @@ export function machineView(row: MachineRow): MachineView { volumeUsedPercent: volumeUsedPercentForRow(row), membershipId: row.membership_id, error: row.error, + boxImage: row.box_image_tag_reported, + boxImageTarget, + boxUpdateRequested: row.box_update_requested === 1, + boxUpdateOutcome: boxUpdateOutcomeForRow(row), createdAt: row.created_at, updatedAt: row.updated_at, }; @@ -148,6 +170,8 @@ export interface WorkspaceProjection { myRole: WorkspaceMemberRole | null; /** The legacy four-value access role the webApp and the ticket both use. */ role: WorkspaceView["role"]; + /** The deployment's current box image, for every machine view built here. */ + boxImageTarget: string; } export function workspaceView(projection: WorkspaceProjection): WorkspaceView { @@ -170,7 +194,7 @@ export function workspaceView(projection: WorkspaceProjection): WorkspaceView { role: member.role, machine: machine === undefined || machine.state === "destroyed" ? null - : machineView(machine), + : machineView(machine, projection.boxImageTarget), }; }); const view: WorkspaceView = { diff --git a/packages/control-plane/core/workspace-settings.ts b/packages/control-plane/core/workspace-settings.ts index 07ac3e42..78b0710b 100644 --- a/packages/control-plane/core/workspace-settings.ts +++ b/packages/control-plane/core/workspace-settings.ts @@ -177,7 +177,7 @@ export function addWorkspaceSettingsRoutes( const updated = await workspaceById(runtime.db, workspace.id); if (updated === null) throw new Error("workspace disappeared during update"); return context.json({ - workspace: await projectWorkspace(runtime.db, principal, updated), + workspace: await projectWorkspace(runtime, principal, updated), }); }); diff --git a/packages/control-plane/core/workspaces.ts b/packages/control-plane/core/workspaces.ts index 71d8fbe7..983fc8db 100644 --- a/packages/control-plane/core/workspaces.ts +++ b/packages/control-plane/core/workspaces.ts @@ -732,7 +732,7 @@ export function addWorkspaceRoutes( input, ); return context.json( - { workspace: await projectWorkspace(runtime.db, principal, row) }, + { workspace: await projectWorkspace(runtime, principal, row) }, 201, ); }); @@ -744,7 +744,7 @@ export function addWorkspaceRoutes( } const runtime = runtimeFactory(context); const all = await workspacesForOrg(runtime.db, principal.orgId); - const views = await projectWorkspaces(runtime.db, principal, all); + const views = await projectWorkspaces(runtime, principal, all); return context.json({ // A member sees the workspaces they are in; an org admin sees every one // of the organization's, which is the reach they already held. @@ -784,7 +784,7 @@ export function addWorkspaceRoutes( v: [principal.orgId], }); return context.json({ - workspaces: await projectWorkspaces(runtime.db, principal, deleted), + workspaces: await projectWorkspaces(runtime, principal, deleted), }); }); @@ -847,7 +847,7 @@ export function addWorkspaceRoutes( request, ); return context.json( - { workspace: await projectWorkspace(runtime.db, principal, created) }, + { workspace: await projectWorkspace(runtime, principal, created) }, 201, ); } catch (error) { @@ -896,7 +896,7 @@ export function addWorkspaceRoutes( if (row === null || row.org_id !== principal.orgId || row.deleted_at !== null) { throw new HttpError(404, "workspace not found"); } - const view = await projectWorkspace(runtime.db, principal, row); + const view = await projectWorkspace(runtime, principal, row); if (view.role === null) throw new HttpError(403, "forbidden"); return context.json({ workspace: view }); }); @@ -1074,7 +1074,7 @@ export function addWorkspaceRoutes( await requireWorkspaceAdmin(runtime.db, principal, row); if (row.deleted_at !== null) { return context.json({ - workspace: await projectWorkspace(runtime.db, principal, row), + workspace: await projectWorkspace(runtime, principal, row), }); } let pending = false; @@ -1100,7 +1100,7 @@ export function addWorkspaceRoutes( const after = await workspaceById(runtime.db, id); if (after === null) throw new Error("workspace disappeared during destroy"); return context.json({ - workspace: await projectWorkspace(runtime.db, principal, after), + workspace: await projectWorkspace(runtime, principal, after), }); }); diff --git a/packages/control-plane/migrations/0046_machine_box_image.sql b/packages/control-plane/migrations/0046_machine_box_image.sql new file mode 100644 index 00000000..92751d10 --- /dev/null +++ b/packages/control-plane/migrations/0046_machine_box_image.sql @@ -0,0 +1,17 @@ +-- What image a machine actually runs, and how its last update attempt ended. +-- +-- `box_image_reported` already stores the REF the host was asked to install. +-- Under the R2 manifest mode that canary runs, the ref is a manifest URL that +-- never changes across rebakes, so it cannot answer "is an update available". +-- `box_image_tag_reported` stores the CONCRETE image instead — the tag from +-- inside the manifest, or the ref itself under a registry pin — which is what +-- the deployment's own pin can be compared against. +-- +-- `box_update_outcome` keeps the host's last verdict. It is what lets the UI +-- say the honest thing about a machine whose emitted updater predates the +-- manifest branch: that host reports `unsupported` and can never self-update. +-- +-- Neither column takes a default. NULL means "never reported", which is a +-- different fact from any value either column can hold. +ALTER TABLE machines ADD COLUMN box_image_tag_reported TEXT; +ALTER TABLE machines ADD COLUMN box_update_outcome TEXT; diff --git a/packages/control-plane/scripts/bake-golden-image.mjs b/packages/control-plane/scripts/bake-golden-image.mjs index f6321f67..8164992a 100644 --- a/packages/control-plane/scripts/bake-golden-image.mjs +++ b/packages/control-plane/scripts/bake-golden-image.mjs @@ -23,7 +23,7 @@ import { execFileSync, spawnSync } from "node:child_process"; import { mkdtempSync, readFileSync, rmSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; -import { BOX_IMAGE_SETUP_HELPERS, boxImageSetupScript } from "../dist/core/bootstrap.js"; +import { boxImageSetupPreamble, boxImageSetupScript } from "../dist/core/bootstrap.js"; const API = "https://api.hetzner.cloud/v1"; const POLL_INTERVAL_MS = 5_000; @@ -104,9 +104,9 @@ export BOX_IMAGE_SHA256=${JSON.stringify(image.boxImageSha256)} # broken bake costs two minutes instead of thirty. trap 'echo "bake: FAILED at line $LINENO"; shutdown -h now' ERR -# The emitted image setup calls these. Without them the setup dies on a -# "retry: command not found", and set -e stops the builder where it stands. -${BOX_IMAGE_SETUP_HELPERS} +# Whatever the emitted image setup calls, for the mode it runs in. Without it +# the setup dies on a "command not found" and set -e stops the builder. +${boxImageSetupPreamble(image)} apt-get update apt-get install -y docker.io curl systemctl enable --now docker diff --git a/packages/control-plane/scripts/lib/worker-source.mjs b/packages/control-plane/scripts/lib/worker-source.mjs index 0159132b..0e08e3ae 100644 --- a/packages/control-plane/scripts/lib/worker-source.mjs +++ b/packages/control-plane/scripts/lib/worker-source.mjs @@ -27,6 +27,7 @@ export const CORE_MANIFEST = Object.freeze([ "core/runtime.ts", "core/db.ts", "core/blobs.ts", + "core/wire-box-config.ts", "core/wire.ts", "core/wire-machines.ts", "core/wire-sharing.ts", "core/agent-rules.ts", "core/bootstrap.ts", @@ -45,6 +46,7 @@ export const CORE_MANIFEST = Object.freeze([ "core/identity/google.ts", "core/identity/invites.ts", "core/identity/members.ts", "core/identity/orgs.ts", "core/identity/routes.ts", "core/janitors.ts", "core/machines.ts", + "core/machine-access.ts", "core/machine-stats.ts", "core/oauth-state.ts", "core/oauth.ts", diff --git a/packages/control-plane/test/bootstrap.test.ts b/packages/control-plane/test/bootstrap.test.ts index a0d1cb59..f451edeb 100644 --- a/packages/control-plane/test/bootstrap.test.ts +++ b/packages/control-plane/test/bootstrap.test.ts @@ -4,6 +4,7 @@ import { boxHostname, buildBootstrapScript } from "../core/bootstrap.js"; import { buildUserData } from "../core/cloud-init.js"; import { AwsProvider } from "../core/compute/aws.js"; import { HetznerProvider } from "../core/compute/hetzner.js"; +import { HETZNER_USER_DATA_MAX_BYTES } from "../core/compute/hetzner-config.js"; import { appRequest, appWithProviders, @@ -451,9 +452,9 @@ describe("production VM bootstrap", () => { // Origin refresh happens on every poll, before the update gate. const refresh = userData.indexOf('mv "$origin_tmp" "$ORIGIN_PATH"'); const gate = userData.indexOf('[ "$update_requested" = true ] || exit 0'); - const pull = userData.indexOf('docker pull "$next_ref"', gate); + const pull = userData.indexOf('docker pull "$next_image"', gate); const remove = userData.indexOf("docker rm -f blitz-box", pull); - const startNew = userData.indexOf('if start_box "$next_ref"; then', remove); + const startNew = userData.indexOf('if start_box "$next_image"; then', remove); const rollback = userData.indexOf('start_box "$current_image"', startNew); expect(refresh).toBeGreaterThan(-1); expect(gate).toBeGreaterThan(refresh); @@ -550,19 +551,26 @@ write_files: expect(userData).toContain(`readonly BOX_IMAGE_TAG='${BOX_IMAGE_TAG}'`); expect(userData).toContain(`readonly BOX_IMAGE_SHA256='${BOX_IMAGE_SHA256}'`); expect(userData).toContain('curl --fail --location --retry 10 --retry-all-errors'); - expect(userData).toContain('verify_sha256 "$image_archive" "$BOX_IMAGE_SHA256"'); - expect(userData).toContain('gunzip -c "$image_archive" | docker load'); - expect(userData).toContain('"$BOX_IMAGE_TAG"'); expect(userData).not.toContain('docker pull "$BOX_IMAGE_REF"'); + // A non-manifest archive takes the installer's `fetch` action, and the + // three variables are handed to it in the order it names them. + expect(userData).toContain('*) box_image_action=fetch ;;'); + expect(userData).toContain( + '/usr/local/sbin/blitz-box-image "$box_image_action" "$BOX_IMAGE_REF" "$BOX_IMAGE_TAG" "$BOX_IMAGE_SHA256"', + ); - const download = userData.indexOf('download "$BOX_IMAGE_REF" "$image_archive"'); - const checksum = userData.indexOf('verify_sha256 "$image_archive" "$BOX_IMAGE_SHA256"'); - const load = userData.indexOf('gunzip -c "$image_archive" | docker load'); - const run = userData.indexOf("docker run", load); + // Inside the fetch branch: download, then verify, then load — and the + // load is the shared helper, so the archive can only reach docker after + // its digest matched. (`load_archive` is defined above its callers, which + // is why the ordering is asserted on the call, not on the gunzip.) + const download = userData.indexOf('download "$url" "$archive"'); + const checksum = userData.indexOf('verify_sha256 "$archive" "${4:?}"', download); + const load = userData.indexOf('load_archive "$3"', checksum); expect(download).toBeGreaterThan(-1); expect(checksum).toBeGreaterThan(download); expect(load).toBeGreaterThan(checksum); - expect(run).toBeGreaterThan(load); + expect(userData).toContain('gunzip -c "$archive" | docker load'); + expect(userData.indexOf("docker run", load)).toBeGreaterThan(load); }); it("keys archive image setup only to image availability on the current daemon", () => { @@ -576,51 +584,22 @@ write_files: BOX_IMAGE_SHA256, ); - const inspectGuard = userData.indexOf( - 'if ! docker image inspect "$BOX_IMAGE_TAG" >/dev/null 2>&1; then', - ); - const download = userData.indexOf('download "$BOX_IMAGE_REF" "$image_archive"'); - const load = userData.indexOf('gunzip -c "$image_archive" | docker load'); - const guardEnd = userData.indexOf("\nfi\n", load); - const provenPresent = userData.indexOf( - 'docker image inspect "$BOX_IMAGE_TAG" >/dev/null', - guardEnd + 1, - ); - const run = userData.indexOf("docker run", provenPresent); - - expect(inspectGuard).toBeGreaterThan(-1); - expect(download).toBeGreaterThan(inspectGuard); + // The skip lives in the installer now, and it asks the daemon and nothing + // else: an image the store already holds costs no bytes off the network. + // This is what makes a golden snapshot fast. + const skip = userData.indexOf('if docker image inspect "${3:?}" >/dev/null 2>&1; then exit 0; fi'); + const download = userData.indexOf('download "$url" "$archive"', skip); + const load = userData.indexOf('load_archive "$3"', download); + expect(skip).toBeGreaterThan(-1); + expect(download).toBeGreaterThan(skip); expect(load).toBeGreaterThan(download); - expect(guardEnd).toBeGreaterThan(load); - expect(provenPresent).toBeGreaterThan(guardEnd); - expect(run).toBeGreaterThan(provenPresent); - expect(userData.slice(inspectGuard, provenPresent)).not.toMatch( - /(?:if|elif)[^\n]*\/var\/lib\/blitz/u, - ); - }); - - it("keys registry image setup only to image availability on the current daemon", () => { - const userData = registryUserData(); + expect(userData.slice(skip, load)).not.toMatch(/(?:if|elif)[^\n]*\/var\/lib\/blitz/u); - const inspectGuard = userData.indexOf( - 'if ! docker image inspect "$BOX_IMAGE_REF" >/dev/null 2>&1; then', - ); - const pull = userData.indexOf('retry docker pull "$BOX_IMAGE_REF"'); - const guardEnd = userData.indexOf("\nfi\n", pull); - const provenPresent = userData.indexOf( - 'docker image inspect "$BOX_IMAGE_REF" >/dev/null', - guardEnd + 1, - ); + // The bootstrap proves the tag is present before it runs anything. + const provenPresent = userData.indexOf('docker image inspect "$BOX_IMAGE_TAG" >/dev/null\nbox_image='); const run = userData.indexOf("docker run", provenPresent); - - expect(inspectGuard).toBeGreaterThan(-1); - expect(pull).toBeGreaterThan(inspectGuard); - expect(guardEnd).toBeGreaterThan(pull); - expect(provenPresent).toBeGreaterThan(guardEnd); + expect(provenPresent).toBeGreaterThan(load); expect(run).toBeGreaterThan(provenPresent); - expect(userData.slice(inspectGuard, provenPresent)).not.toMatch( - /(?:if|elif)[^\n]*\/var\/lib\/blitz/u, - ); }); it("validates multipart manifest parts, total digest, and image tag before loading", () => { @@ -634,17 +613,20 @@ write_files: BOX_IMAGE_SHA256, ); - expect(userData).toContain('download "$BOX_IMAGE_REF" "$manifest_path"'); + expect(userData).toContain('*/manifest.json) box_image_action=install ;;'); + expect(userData).toContain('download "$url" "$tmp/manifest.json"'); expect(userData).toContain('value.get("parts")'); expect(userData).toContain('value.get("totalSha256")'); expect(userData).toContain('value.get("imageTag")'); - expect(userData).toContain('download "$manifest_base/$part_name" "$part_path"'); - expect(userData).toContain('verify_sha256 "$part_path" "$part_sha256"'); - expect(userData).toContain('cat "$part_path" >>"$image_archive"'); - expect(userData).toContain('verify_sha256 "$image_archive" "$manifest_total_sha256"'); - expect(userData).toContain('verify_sha256 "$image_archive" "$BOX_IMAGE_SHA256"'); - expect(userData).toContain('[ "$manifest_image_tag" = "$BOX_IMAGE_TAG" ]'); - expect(userData).toContain('gunzip -c "$image_archive" | docker load'); + expect(userData).toContain('download "${url%/*}/$part_name" "$tmp/$part_name"'); + expect(userData).toContain('verify_sha256 "$tmp/$part_name" "$part_sha256"'); + expect(userData).toContain('cat "$tmp/$part_name" >>"$archive"'); + expect(userData).toContain('verify_sha256 "$archive" "$manifest_total"'); + // The caller's own pinned digest is checked on top of the manifest's, so + // a swapped manifest cannot redirect a boot to another image. + expect(userData).toContain('[ -z "${4:-}" ] || verify_sha256 "$archive" "$4"'); + expect(userData).toContain('[ "$manifest_tag" = "${3:?}" ] || exit 13'); + expect(userData).toContain('gunzip -c "$archive" | docker load'); expect(userData).not.toContain('docker pull "$BOX_IMAGE_REF"'); }); @@ -1060,3 +1042,43 @@ write_files: }); }); }); + +// Cloud-init user-data on Hetzner is a hard 32 KiB, uncompressed (AWS gzips +// and has room to spare, so Hetzner is the binding constraint for every +// emitted byte). The script is what it is: this pins that a realistic create +// keeps a working margin, so the next feature that wants to emit bash finds +// out here rather than as a 413 on a real create. +describe("the emitted bootstrap fits the cloud-init budget", () => { + const manifest = { + boxImageSha256: "a".repeat(64), + boxImageRef: "https://r2.example/box-image/manifest.json", + boxImageTag: "blitz-box:2026-08-31", + phoneHomeUrl: "https://cp.example/workspaces/workspace/phone-home/token", + sshPublicKey: "ssh-ed25519 AAAAcaller", + }; + + /** The most expensive shape a real create emits: the manifest install path + * (canary), a hostname, repos to clone and usage capture. */ + const heaviest = { + ...manifest, + boxHostname: "a-workspace-with-a-long-enough-name", + repos: ["blitzdotdev/BlitzOS", "blitzdotdev/another-repo"], + usageCapture: true, + }; + + it("leaves a working margin under the Hetzner cap", () => { + const bytes = new TextEncoder().encode(buildBootstrapScript(heaviest)).byteLength; + // 2 KiB is the floor this change accepted, not a target to spend down. + // The way to buy real headroom back is to stop shipping the host scripts + // in user-data at all and extract them from the box image, which is the + // direction plans/MEMBER-MACHINES.md records. + expect(bytes).toBeLessThan(HETZNER_USER_DATA_MAX_BYTES - 2 * 1024); + }); + + it("emits the host installer once, whichever mode the deployment pins", () => { + for (const options of [manifest, { ...manifest, boxImageRef: "ghcr.io/o/box:v3", boxImageTag: "", boxImageSha256: "" }]) { + const script = buildBootstrapScript(options); + expect(script.split("cat >/usr/local/sbin/blitz-box-image <<").length - 1).toBe(1); + } + }); +}); diff --git a/packages/control-plane/test/box-config-conformance.test.ts b/packages/control-plane/test/box-config-conformance.test.ts index f16be62d..0642c3cb 100644 --- a/packages/control-plane/test/box-config-conformance.test.ts +++ b/packages/control-plane/test/box-config-conformance.test.ts @@ -1,7 +1,7 @@ import { env } from "cloudflare:workers"; import { beforeEach, describe, expect, it } from "vitest"; -import { controlPlaneOriginFromEnv } from "../core/box-config.js"; -import type { BoxConfigResponse } from "../core/wire.js"; +import { controlPlaneOriginFromEnv, deploymentBoxImage } from "../core/box-config.js"; +import type { BoxConfigResponse, MachineView, WorkspaceMemberView } from "../core/wire.js"; import { appRequest, harness, @@ -77,13 +77,20 @@ function boxHeaders(box: BoxCredential): Record { return { Authorization: `Bearer ${box.access_token}` }; } -async function workspaceUpdateColumns( - workspaceId: string, -): Promise<{ box_update_requested: number; box_image_reported: string | null }> { +interface UpdateColumns { + box_update_requested: number; + box_image_reported: string | null; + box_image_tag_reported: string | null; + box_update_outcome: string | null; +} + +async function workspaceUpdateColumns(workspaceId: string): Promise { const row = await env.DB - .prepare("SELECT box_update_requested, box_image_reported FROM machines WHERE workspace_id = ?1") + .prepare(`SELECT box_update_requested, box_image_reported, + box_image_tag_reported, box_update_outcome + FROM machines WHERE workspace_id = ?1`) .bind(workspaceId) - .first<{ box_update_requested: number; box_image_reported: string | null }>(); + .first(); if (row === null) throw new Error("machine row missing"); return row; } @@ -95,6 +102,7 @@ describe("box-config control-plane conformance", () => { it("pins the shared box-config fixture corpus", () => { expect(fixtures("config-").map(([name]) => name)).toEqual([ + "config-bad-image-sha256.json", "config-image-ref-with-space.json", "config-missing-image-ref.json", "config-non-boolean-update-requested.json", @@ -108,8 +116,13 @@ describe("box-config control-plane conformance", () => { "result-missing-outcome.json", "result-missing-ref.json", "result-ref-with-space.json", + "result-tag-with-space.json", "result-unknown-outcome.json", + "result-valid-digest-mismatch.json", + "result-valid-download-failed.json", "result-valid-extra-key.json", + "result-valid-load-failed.json", + "result-valid-manifest-tag.json", "result-valid-rolled-back.json", "result-valid-updated.json", ]); @@ -128,6 +141,9 @@ describe("box-config control-plane conformance", () => { const body = await response.json(); expect(body).toEqual({ boxImageRef: env.BOX_IMAGE_REF, + // The digest the deployment pins, so the host's check is as strong as + // the first boot's rather than trusting the manifest about itself. + boxImageSha256: env.BOX_IMAGE_SHA256, // No APP_URL binding on this request, so the fallback answers with the // origin the poll arrived on. controlPlaneOrigin: "https://cp.example", @@ -206,7 +222,10 @@ describe("box-config control-plane conformance", () => { for (const [name, fixture] of fixtures("result-")) { await env.DB - .prepare("UPDATE machines SET box_update_requested = 1, box_image_reported = NULL WHERE workspace_id = ?1") + .prepare(`UPDATE machines + SET box_update_requested = 1, box_image_reported = NULL, + box_image_tag_reported = 'seeded-at-boot', box_update_outcome = NULL + WHERE workspace_id = ?1`) .bind(workspaceId) .run(); const response = await appRequest(h.app, "/workspaces/self/box-update-result", { @@ -222,14 +241,113 @@ describe("box-config control-plane conformance", () => { // operation on its own. expect(columns.box_update_requested, name).toBe(0); expect(columns.box_image_reported, name).toBe(fixture.request.ref); + expect(columns.box_update_outcome, name).toBe(fixture.request.outcome); + // A host that sends no `tag` predates the field, and the value the + // row already holds stays the best answer there is — nulling it would + // throw away the image the boot was armed with. + expect(columns.box_image_tag_reported, name) + .toBe(fixture.request.tag ?? "seeded-at-boot"); } else { expect(response.status, name).toBe(400); expect(columns.box_update_requested, name).toBe(1); expect(columns.box_image_reported, name).toBeNull(); + expect(columns.box_update_outcome, name).toBeNull(); + expect(columns.box_image_tag_reported, name).toBe("seeded-at-boot"); } } }); + // The "Update machine" button. It names ONE machine whoever calls it, which + // is the whole difference from the workspace route below: a user who clicked + // a button in their own machine dialog did not ask to restart a colleague's + // work, and an admin pressing it must not accidentally fan out. + it("lets a member request an update for their own machine and nobody else's", async () => { + const h = harness(); + const cookie = await operatorSession(); + const { workspaceId } = await readyWorkspaceBox(h, cookie); + const machine = await env.DB + .prepare("SELECT id FROM machines WHERE workspace_id = ?1") + .bind(workspaceId) + .first<{ id: string }>(); + if (machine === null) throw new Error("machine row missing"); + const stranger = await sameOrgSession("member-nobody"); + + const denied = await appRequest(h.app, `/machines/${machine.id}/box-update`, { + method: "POST", + headers: { Cookie: stranger.cookie }, + }); + expect(denied.status).toBe(403); + expect((await workspaceUpdateColumns(workspaceId)).box_update_requested).toBe(0); + + const requested = await appRequest(h.app, `/machines/${machine.id}/box-update`, { + method: "POST", + headers: { Cookie: cookie }, + }); + expect(requested.status).toBe(200); + // It answers with the machine, so the dialog renders the pending flag + // without waiting for the next poll. + const { machine: view } = await requested.json<{ machine: MachineView }>(); + expect(view.id).toBe(machine.id); + expect(view.boxUpdateRequested).toBe(true); + expect((await workspaceUpdateColumns(workspaceId)).box_update_requested).toBe(1); + + const missing = await appRequest(h.app, "/machines/not-a-machine/box-update", { + method: "POST", + headers: { Cookie: cookie }, + }); + expect(missing.status).toBe(404); + }); + + // Under an R2 manifest pin the ref never changes between rebakes, so the + // machine's own reported TAG is the only thing that can answer the question + // the button's label depends on. + it("projects the reported image against the deployment's own pin", async () => { + const h = harness(); + const cookie = await operatorSession(); + const { workspaceId } = await readyWorkspaceBox(h, cookie); + + const beforeReport = await appRequest(h.app, `/workspaces/${workspaceId}`, { + headers: { Cookie: cookie }, + }); + const before = await beforeReport.json<{ workspace: { members: WorkspaceMemberView[] } }>(); + const seeded = before.workspace.members[0]?.machine; + // A freshly armed boot already knows its image, so "is an update + // available" has an answer before any update is ever attempted. + expect(seeded?.boxImage).toBe(env.BOX_IMAGE_TAG); + expect(seeded?.boxImageTarget).toBe(env.BOX_IMAGE_TAG); + expect(seeded?.boxUpdateOutcome).toBeNull(); + + await env.DB + .prepare("UPDATE machines SET box_image_tag_reported = ?2, box_update_outcome = ?3 WHERE workspace_id = ?1") + .bind(workspaceId, "blitz-box:2026-08-01", "unsupported") + .run(); + + const stale = await appRequest(h.app, `/workspaces/${workspaceId}`, { + headers: { Cookie: cookie }, + }); + const view = await stale.json<{ workspace: { members: WorkspaceMemberView[] } }>(); + const machine = view.workspace.members[0]?.machine; + expect(machine?.boxImage).toBe("blitz-box:2026-08-01"); + // The deployment pins a manifest URL, so the concrete image is the tag + // inside it — never the URL, which is the same across every rebake. + expect(machine?.boxImageTarget).toBe(env.BOX_IMAGE_TAG); + expect(machine?.boxImageTarget).not.toBe(env.BOX_IMAGE_REF); + // The honest signal for a host whose emitted updater predates the manifest + // branch: it reported `unsupported`, and it can never self-update. + expect(machine?.boxUpdateOutcome).toBe("unsupported"); + }); + + it("names the tag rather than the ref when the deployment pins a manifest", () => { + // Registry mode: the ref IS the image and the tag var is empty. + expect(deploymentBoxImage({ boxImageRef: "ghcr.io/o/box:v3", boxImageTag: "" })) + .toBe("ghcr.io/o/box:v3"); + // Manifest mode: the URL is identical across rebakes, the tag is not. + expect(deploymentBoxImage({ + boxImageRef: "https://cp.example/box-image/manifest.json", + boxImageTag: "blitz-box:2026-08-31", + })).toBe("blitz-box:2026-08-31"); + }); + it("gates the session route on canControlWorkspace", async () => { const h = harness(); const cookie = await operatorSession(); diff --git a/packages/control-plane/test/box-update-conformance.test.mjs b/packages/control-plane/test/box-update-conformance.test.mjs index 93f3861c..dc595f2b 100644 --- a/packages/control-plane/test/box-update-conformance.test.mjs +++ b/packages/control-plane/test/box-update-conformance.test.mjs @@ -5,7 +5,10 @@ import { tmpdir } from "node:os"; import path from "node:path"; import test from "node:test"; import { fileURLToPath } from "node:url"; -import { buildBootstrapScript } from "../dist/core/bootstrap.js"; +import { + BOX_IMAGE_MANIFEST_PARSER, + buildBootstrapScript, +} from "../dist/core/bootstrap.js"; import { embeddedSection } from "./emitted-script.mjs"; // Host side of the `box config v1` cross-runtime contract. The updater the @@ -48,7 +51,7 @@ function python3Available() { } test("the emitted host scripts are valid bash", () => { - for (const marker of ["BOX_RUN", "BOX_UPDATER"]) { + for (const marker of ["BOX_RUN", "BOX_UPDATER", "BOX_IMAGE_INSTALL"]) { const result = spawnSync("bash", ["-n"], { input: embeddedSection(bootstrap, marker), encoding: "utf8", @@ -61,12 +64,57 @@ test("the updater and the initial start share the one blitz-box-run path", () => const updater = embeddedSection(bootstrap, "BOX_UPDATER"); assert.match(updater, /\/usr\/local\/bin\/blitz-box-run "\$1"/u); assert.ok(bootstrap.includes('/usr/local/bin/blitz-box-run "$box_image"')); - // Pull first, remove second: a failed pull must leave the old container - // running. - assert.ok( - updater.indexOf('docker pull "$next_ref"') < updater.indexOf("docker rm -f blitz-box"), - "the updater must pull before it removes the running container", + // Acquire first, remove second, on BOTH install paths: a registry pull or a + // manifest install that fails must leave the old container running. + const removal = updater.indexOf("docker rm -f blitz-box"); + assert.ok(removal > 0, "the updater no longer removes the container"); + for (const acquire of ['docker pull "$next_image"', "blitz-box-image install"]) { + const at = updater.indexOf(acquire); + assert.ok(at > 0, `the updater no longer runs: ${acquire}`); + assert.ok(at < removal, `the updater must run ${acquire} before removing the container`); + } +}); + +// The first boot and the host updater install the same image the same way, so +// they run the SAME host script rather than two copies of one pipeline. Two +// copies would be two chances to verify a digest differently — and both used +// to ride in cloud-init user-data, which Hetzner caps at 32 KiB. +test("one host installer serves both the first boot and the updater", () => { + const manifestBootstrap = buildBootstrapScript({ + boxImageSha256: "a".repeat(64), + boxImageRef: "https://r2.example/box-image/manifest.json", + boxImageTag: "blitz-box:2026-08-31", + phoneHomeUrl: "https://cp.example/workspaces/workspace/phone-home/token", + }); + // Exactly one copy of the manifest parser is emitted, and it is the shared + // constant. + const copies = manifestBootstrap.split("manifest totalSha256 must be a SHA-256 digest").length - 1; + assert.equal(copies, 1, "the manifest parser is emitted more than once"); + assert.equal( + `${embeddedSection(manifestBootstrap, "MANIFEST_PARSER")}\n`, + BOX_IMAGE_MANIFEST_PARSER, ); + // Both callers reach it by the one path. + assert.match(manifestBootstrap, /\/usr\/local\/sbin\/blitz-box-image "\$box_image_action"/u); + assert.match( + embeddedSection(manifestBootstrap, "BOX_UPDATER"), + /\/usr\/local\/sbin\/blitz-box-image resolve "\$next_ref"/u, + ); +}); + +// The failure this closes: a 15-minute box token, a 5-minute timer, and a +// consumer that could not rotate. Every poll after the first expiry 401d for +// good, so the update flag never cleared and the button that set it died +// silently. The updater has to hold its own credential. +test("the updater can rotate the box credential it reads", () => { + const updater = embeddedSection(bootstrap, "BOX_UPDATER"); + assert.match(updater, /"\$current_origin\/oauth\/token"/u); + assert.match(updater, /grant_type=refresh_token/u); + // Under the same lock the in-container Go client takes, beside the file, + // because the file is replaced by rename. + assert.match(updater, /flock --exclusive --timeout 30 9/u); + assert.match(updater, /9>"\$CREDENTIAL_LOCK"/u); + assert.ok(updater.includes('readonly CREDENTIAL_LOCK="$STATE_DIR/box-credential.lock"')); }); test("embedded box-config parser matches every config fixture", (context) => { @@ -87,11 +135,12 @@ test("embedded box-config parser matches every config fixture", (context) => { }); if (fixture.accepts) { assert.equal(result.status, 0, `${name}: ${result.stderr}`); - const { boxImageRef, controlPlaneOrigin, updateRequested } = fixture.response; + const { boxImageRef, boxImageSha256, controlPlaneOrigin, updateRequested } = + fixture.response; assert.equal( result.stdout, - `${boxImageRef}\t${controlPlaneOrigin}\t${updateRequested}\n`, - `${name}: parsed TSV mismatch`, + `${boxImageRef}\n${boxImageSha256.toLowerCase()}\n${controlPlaneOrigin}\n${updateRequested}\n`, + `${name}: parsed field mismatch`, ); } else { assert.notEqual(result.status, 0, `${name} unexpectedly passed`); @@ -112,15 +161,30 @@ test("embedded update-result producer emits bytes the control plane accepts", (c const accepted = fixtures("result-").filter(([, fixture]) => fixture.accepts); assert.ok(accepted.length > 0, "no accepted update-result fixtures"); for (const [name, fixture] of accepted) { - const { ref, outcome } = fixture.request; - const result = spawnSync("python3", ["-", ref, outcome], { + const { ref, outcome, tag } = fixture.request; + const result = spawnSync("python3", ["-", ref, outcome, tag ?? ""], { input: writer, encoding: "utf8", }); assert.equal(result.status, 0, `${name}: ${result.stderr}`); - // The producer emits exactly the two contract keys; extra keys in a - // fixture exist to pin the CONSUMER's tolerance, never this producer. - assert.equal(result.stdout, JSON.stringify({ ref, outcome }), `${name}: body mismatch`); + // The producer emits the two required contract keys, plus `tag` when the + // host has an image to name. Other extra keys in a fixture exist to pin + // the CONSUMER's tolerance, never this producer. + const expected = tag === undefined ? { ref, outcome } : { ref, outcome, tag }; + assert.equal(result.stdout, JSON.stringify(expected), `${name}: body mismatch`); } + + // No running container means no image to name, and the producer omits the + // key rather than reporting an empty one. + const noContainer = spawnSync( + "python3", + ["-", "ghcr.io/blitzdotdev/blitz-box:v2", "start-failed", ""], + { input: writer, encoding: "utf8" }, + ); + assert.equal(noContainer.status, 0, noContainer.stderr); + assert.equal( + noContainer.stdout, + JSON.stringify({ ref: "ghcr.io/blitzdotdev/blitz-box:v2", outcome: "start-failed" }), + ); console.log(`update-result producer conformance: ${accepted.length} accepted fixtures`); }); diff --git a/packages/control-plane/test/box-update-host.test.mjs b/packages/control-plane/test/box-update-host.test.mjs index f3a99bf6..744c00c1 100644 --- a/packages/control-plane/test/box-update-host.test.mjs +++ b/packages/control-plane/test/box-update-host.test.mjs @@ -1,10 +1,12 @@ import assert from "node:assert/strict"; import { spawn } from "node:child_process"; +import { createHash } from "node:crypto"; import { chmodSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; import { createServer } from "node:http"; import { tmpdir } from "node:os"; import path from "node:path"; import test from "node:test"; +import { gzipSync } from "node:zlib"; import { buildBootstrapScript } from "../dist/core/bootstrap.js"; import { embeddedSection } from "./emitted-script.mjs"; @@ -33,16 +35,43 @@ const NEXT_REF = "ghcr.io/blitzdotdev/blitz-box:v2"; function relocate(body, root) { return body .replaceAll("/usr/local/bin/blitz-box-run", path.join(root, "bin/blitz-box-run")) + .replaceAll("/usr/local/sbin/blitz-box-image", path.join(root, "bin/blitz-box-image")) .replaceAll("/etc/blitz", path.join(root, "etc/blitz")) .replaceAll("/var/lib/blitz", path.join(root, "state")); } +/** Widens the two ref globs from `https://` to `http*://`, and nothing else. + * + * R2 is https and the emitted script is right to say so, but a test cannot + * hand curl a trusted certificate for 127.0.0.1. Both globs move together, so + * the branch this suite exercises is still the real one: manifest URL versus + * any other https ref versus a registry ref, in that order. The `https?://` + * inside the box-config parser's regex does not match this literal and is + * left alone. */ +function relocateImageHost(body) { + return body + .replace(" https://*/manifest.json)", " http*://*/manifest.json)") + .replace(" https://*)", " http*://*)"); +} + /** A scratch VM host: the emitted blitz-box-run, a docker whose every call is * recorded and whose outcomes the test dictates, and a chown that records the * ownership the updater asks for (a test does not run as root). `refuseRun` * names the image refs whose `docker run` fails the way a real one does when - * the image cannot start — a bad platform, or host port 22 already bound. */ -function scratchHost({ pullStatus = 0, refuseRun = [], runningRef = RUNNING_REF } = {}) { + * the image cannot start — a bad platform, or host port 22 already bound. + * + * The manifest path adds a local image store: `storedImages` is what + * `docker image inspect ` already answers to, `loadProduces` is the tag a + * successful `docker load` puts there, and `refuseLoad` makes the load fail + * the way a truncated archive does. */ +function scratchHost({ + pullStatus = 0, + refuseRun = [], + runningRef = RUNNING_REF, + storedImages = [], + loadProduces = "", + refuseLoad = false, +} = {}) { const root = mkdtempSync(path.join(tmpdir(), "blitz-box-update-")); mkdirSync(path.join(root, "bin"), { recursive: true }); mkdirSync(path.join(root, "state"), { recursive: true }); @@ -54,12 +83,22 @@ function scratchHost({ pullStatus = 0, refuseRun = [], runningRef = RUNNING_REF } writeFileSync(path.join(root, "bin/blitz-box-run"), relocate(embeddedSection(BOOTSTRAP, "BOX_RUN"), root)); chmodSync(path.join(root, "bin/blitz-box-run"), 0o755); + // The one host image installer, the same bytes the first boot uses. The + // updater shells out to it, so the manifest path under test here is the + // production one rather than a second copy written for the test. + writeFileSync( + path.join(root, "bin/blitz-box-image"), + relocate(embeddedSection(BOOTSTRAP, "BOX_IMAGE_INSTALL"), root), + ); + chmodSync(path.join(root, "bin/blitz-box-image"), 0o755); writeFileSync(path.join(root, "refuse-run"), `${refuseRun.join("\n")}\n`); + writeFileSync(path.join(root, "stored.images"), `${storedImages.join("\n")}\n`); + writeFileSync(path.join(root, "load-produces"), loadProduces); // `docker run --detach` is the container start and its image is the last // argument; `docker run --rm --entrypoint cat IMAGE PATH` is the // env.defaults read. Everything else the updater calls is answered from the - // two files that stand in for the daemon's view of blitz-box. + // files that stand in for the daemon's view of blitz-box and of its store. const docker = `#!/bin/bash printf '%s\\n' "$*" >>"${root}/docker.argv" case "$*" in @@ -69,6 +108,11 @@ case "$*" in "inspect --format {{.State.Running}} blitz-box") [ -f "${root}/container.running" ] || exit 1 cat "${root}/container.running" ;; + "image inspect "*) + grep -qxF "$3" "${root}/stored.images" ;; + "load") + cat >"${root}/loaded.archive" + ${refuseLoad ? 'echo "docker: unexpected EOF" >&2; exit 1' : `cat "${root}/load-produces" >>"${root}/stored.images"`} ;; "pull "*) exit ${pullStatus} ;; "rm -f blitz-box") rm -f "${root}/container.image" "${root}/container.running" ;; @@ -98,11 +142,58 @@ esac /** A control plane the emitted curl really talks to. `boxConfig` is called * with the plane's own origin, so a test can serve either that origin (the * steady state) or a different one (the domain move). Every update-result - * report is recorded with its Authorization header. */ -async function controlPlane(boxConfig) { + * report is recorded with its Authorization header. + * + * `assets` serves the box-image manifest and its parts as plain bytes, which + * is what R2 is to the host. `token` makes the plane demand a live box access + * token and rotate it at `/oauth/token`, which is the only way to exercise the + * updater's own credential refresh: a real box token lives 15 minutes and this + * timer runs every 5, so the 401 is a state every long-lived box reaches. */ +async function controlPlane(boxConfig, { assets = new Map(), token = null } = {}) { const reports = []; + const grants = []; + const live = token === null ? null : { access: token.access, refresh: token.refresh }; + const refused = []; const server = createServer((request, response) => { const origin = `http://127.0.0.1:${server.address().port}`; + const asset = assets.get(request.url); + if (asset !== undefined) { + response.statusCode = asset.status ?? 200; + response.end(asset.body); + return; + } + if (request.url === "/oauth/token") { + const chunks = []; + request.on("data", (chunk) => chunks.push(chunk)); + request.on("end", () => { + const form = new URLSearchParams(Buffer.concat(chunks).toString("utf8")); + grants.push(Object.fromEntries(form)); + if (live === null || form.get("refresh_token") !== live.refresh) { + response.statusCode = 400; + response.end(JSON.stringify({ error: "invalid_grant" })); + return; + } + live.access = `${live.access}-rotated`; + live.refresh = `${live.refresh}-rotated`; + response.setHeader("Content-Type", "application/json"); + response.end(JSON.stringify({ + box_id: "box", + access_token: live.access, + refresh_token: live.refresh, + token_type: "Bearer", + expires_in: 900, + })); + }); + return; + } + // Everything below is box-authenticated. A plane with a `token` refuses a + // stale one exactly as the real oauth check does. + if (live !== null && request.headers.authorization !== `Bearer ${live.access}`) { + refused.push(request.url); + response.statusCode = 401; + response.end(JSON.stringify({ error: "invalid box access token" })); + return; + } if (request.url === "/workspaces/self/box-config") { response.setHeader("Content-Type", "application/json"); response.end(JSON.stringify(boxConfig(origin))); @@ -128,24 +219,67 @@ async function controlPlane(boxConfig) { return { origin: `http://127.0.0.1:${server.address().port}`, reports, + grants, + refused, close: () => new Promise((resolve) => server.close(resolve)), }; } +/** A box image served the way canary serves it: one gzip archive split into + * parts, each with its own SHA-256, behind a manifest that names the tag the + * archive loads as. Returns the assets to serve and the manifest URL to pin. */ +function manifestAssets({ imageTag, parts = 2, corruptPart = null, missingPart = null }) { + const payload = Buffer.from(`box image payload for ${imageTag}`.repeat(64)); + const archive = gzipSync(payload); + const size = Math.ceil(archive.length / parts); + const assets = new Map(); + const entries = []; + for (let index = 0; index < parts; index += 1) { + const name = `part-${String(index)}`; + const bytes = archive.subarray(index * size, (index + 1) * size); + entries.push({ name, sha256: digest(bytes) }); + if (name === missingPart) assets.set(`/box-image/${name}`, { status: 404, body: "" }); + // A corrupt part keeps the digest the manifest promises and serves other + // bytes, which is exactly what a truncated or tampered object looks like. + else if (name === corruptPart) assets.set(`/box-image/${name}`, { body: Buffer.from("tampered") }); + else assets.set(`/box-image/${name}`, { body: bytes }); + } + assets.set("/box-image/manifest.json", { + body: JSON.stringify({ imageTag, totalSha256: digest(archive), parts: entries }), + }); + // `payload` is what docker load actually receives: the updater pipes the + // reassembled archive through gunzip, so asserting on it proves the parts + // were concatenated in manifest order and decompressed whole. + return { + assets, + ref: "/box-image/manifest.json", + archive, + payload, + totalSha256: digest(archive), + }; +} + +function digest(bytes) { + return createHash("sha256").update(bytes).digest("hex"); +} + /** Runs the emitted updater under `bash -x`, asynchronously: the control * plane it curls lives in this process, so a synchronous spawn would block * the very event loop that has to answer. A run that dies under `set -e` * prints nothing of its own, so the trace names the command that killed it. */ -async function runUpdater(root, { origin, token = "box-access-token" } = {}) { +async function runUpdater( + root, + { origin, token = "box-access-token", refresh = "box-refresh-token" } = {}, +) { if (origin !== null) writeFileSync(path.join(root, "state/origin"), `${origin}\n`); if (token !== null) { writeFileSync( path.join(root, "state/box-credential.json"), - `${JSON.stringify({ box_id: "box", access_token: token, refresh_token: "r" })}\n`, + `${JSON.stringify({ box_id: "box", access_token: token, refresh_token: refresh })}\n`, ); } const scriptPath = path.join(root, "blitz-box-update"); - writeFileSync(scriptPath, relocate(embeddedSection(BOOTSTRAP, "BOX_UPDATER"), root)); + writeFileSync(scriptPath, relocateImageHost(relocate(embeddedSection(BOOTSTRAP, "BOX_UPDATER"), root))); const result = await new Promise((resolve) => { const child = spawn("bash", ["-x", scriptPath], { env: { ...process.env, PATH: `${path.join(root, "bin")}:${process.env.PATH ?? ""}` }, @@ -171,9 +305,12 @@ async function runUpdater(root, { origin, token = "box-access-token" } = {}) { }; } -/** The steady-state box-config: this plane's own origin, update requested. */ -function configFor(boxImageRef, planeOrigin) { - return { boxImageRef, controlPlaneOrigin: planeOrigin, updateRequested: true }; +/** The steady-state box-config: this plane's own origin, update requested. + * `boxImageSha256` is the deployment's pinned digest of the whole archive; + * empty is what a registry pin sends, and the manifest tests below pass the + * real one. */ +function configFor(boxImageRef, planeOrigin, boxImageSha256 = "") { + return { boxImageRef, boxImageSha256, controlPlaneOrigin: planeOrigin, updateRequested: true }; } function readOptional(file) { @@ -185,9 +322,9 @@ function readOptional(file) { } /** A scratch host and a live control plane for the length of one body. */ -async function withHost(hostOptions, boxConfig, body) { +async function withHost(hostOptions, boxConfig, body, planeOptions = {}) { const root = scratchHost(hostOptions); - const plane = await controlPlane(boxConfig); + const plane = await controlPlane(boxConfig, planeOptions); try { await body(root, plane, (options) => runUpdater(root, { origin: plane.origin, ...options })); } finally { @@ -196,6 +333,11 @@ async function withHost(hostOptions, boxConfig, body) { } } +/** The credential file as the updater reads it back after a run. */ +function credential(root) { + return JSON.parse(readOptional(path.join(root, "state/box-credential.json"))); +} + // The domain move that caused the fleet-wide websocket outage: the box-config // names an origin the file on disk does not carry. The refresh is unconditional // on every poll and needs no restart, because the gateway re-reads the file. @@ -203,7 +345,12 @@ test("a poll with no update requested refreshes the origin and touches no contai const moved = "https://blitzos.example"; await withHost( {}, - () => ({ boxImageRef: NEXT_REF, controlPlaneOrigin: moved, updateRequested: false }), + () => ({ + boxImageRef: NEXT_REF, + boxImageSha256: "", + controlPlaneOrigin: moved, + updateRequested: false, + }), async (root, plane, run) => { const result = await run(); assert.equal(result.status, 0, result.report); @@ -233,7 +380,13 @@ test("a requested update pulls, replaces, and reports updated", async () => { assert.equal(result.envDefaults, `BLITZ_FROM=${NEXT_REF}\n`); assert.equal(plane.reports.length, 1); assert.equal(plane.reports[0].authorization, "Bearer box-access-token"); - assert.equal(plane.reports[0].body, JSON.stringify({ ref: NEXT_REF, outcome: "updated" })); + // `tag` is the image the container runs NOW. On a clean update that is the + // new ref; the failure cases below prove it is the OLD one when nothing + // was replaced, which is what makes it answer "is an update available". + assert.equal( + plane.reports[0].body, + JSON.stringify({ ref: NEXT_REF, outcome: "updated", tag: NEXT_REF }), + ); assert.equal(readOptional(path.join(root, "state/origin")).trim(), plane.origin); }); }); @@ -251,7 +404,10 @@ test("a failed pull leaves the running container untouched", async () => { ); assert.equal(result.image, RUNNING_REF); assert.equal(readOptional(path.join(root, "container.running")), "true"); - assert.equal(plane.reports[0].body, JSON.stringify({ ref: NEXT_REF, outcome: "pull-failed" })); + assert.equal( + plane.reports[0].body, + JSON.stringify({ ref: NEXT_REF, outcome: "pull-failed", tag: RUNNING_REF }), + ); }, ); }); @@ -275,7 +431,10 @@ test("a new image that refuses to start rolls back to the old ref", async () => assert.equal(result.running, "true"); // Rolling back restores the old image's env file too. assert.equal(result.envDefaults, `BLITZ_FROM=${RUNNING_REF}\n`); - assert.equal(plane.reports[0].body, JSON.stringify({ ref: NEXT_REF, outcome: "rolled-back" })); + assert.equal( + plane.reports[0].body, + JSON.stringify({ ref: NEXT_REF, outcome: "rolled-back", tag: RUNNING_REF }), + ); assert.match(result.log, /rollback complete/u); assert.ok(root); }, @@ -293,7 +452,12 @@ test("a rollback that also fails reports start-failed and still clears the flag" const result = await run(); assert.equal(result.status, 0, result.report); assert.equal(result.image, ""); - assert.equal(plane.reports[0].body, JSON.stringify({ ref: NEXT_REF, outcome: "start-failed" })); + // Nothing is running, so there is no image to name and the producer omits + // the key rather than reporting an empty one. + assert.equal( + plane.reports[0].body, + JSON.stringify({ ref: NEXT_REF, outcome: "start-failed" }), + ); assert.match(result.log, /rollback failed/u); assert.ok(root); }, @@ -304,20 +468,35 @@ test("a requested update to the ref already running clears the flag without pull await withHost({}, (planeOrigin) => configFor(RUNNING_REF, planeOrigin), async (root, plane, run) => { const result = await run(); assert.equal(result.status, 0, result.report); - assert.deepEqual(result.dockerCalls, ["inspect --format {{.Config.Image}} blitz-box"]); - assert.equal(plane.reports[0].body, JSON.stringify({ ref: RUNNING_REF, outcome: "up-to-date" })); + // Two reads and nothing else: the up-to-date check, then the report's own + // read of the image still running. No pull, no removal, no restart. + assert.deepEqual(result.dockerCalls, [ + "inspect --format {{.Config.Image}} blitz-box", + "inspect --format {{.Config.Image}} blitz-box", + ]); + assert.equal( + plane.reports[0].body, + JSON.stringify({ ref: RUNNING_REF, outcome: "up-to-date", tag: RUNNING_REF }), + ); assert.ok(root); }); }); -test("a tarball ref reports unsupported and still refreshes the origin", async () => { - const tarball = "https://cp.example/box-image/manifest.json"; +// An https ref the host cannot resolve to an image is the one case that stays +// unsupported. Every emitted updater before the manifest branch reported this +// for EVERY https ref, which is how the UI recognises a machine that can never +// update itself in place. +test("an https ref that is not a manifest reports unsupported and still refreshes the origin", async () => { + const tarball = "https://cp.example/box-image/image.tar.gz"; await withHost({}, (planeOrigin) => configFor(tarball, planeOrigin), async (root, plane, run) => { const result = await run(); assert.equal(result.status, 0, result.report); assert.ok(!result.dockerCalls.some((call) => call.startsWith("pull"))); assert.equal(result.image, RUNNING_REF); - assert.equal(plane.reports[0].body, JSON.stringify({ ref: tarball, outcome: "unsupported" })); + assert.equal( + plane.reports[0].body, + JSON.stringify({ ref: tarball, outcome: "unsupported", tag: RUNNING_REF }), + ); assert.equal(readOptional(path.join(root, "state/origin")).trim(), plane.origin); }); }); @@ -348,3 +527,277 @@ test("a box-config the contract rejects changes nothing on the host", async () = }, ); }); + +// ---- the manifest branch (canary's mode B) ---- +// +// Canary pins BOX_IMAGE_REF to an https R2 manifest, and every updater before +// this branch refused it outright: no canary box could ever update in place. +// The manifest names the tag, so the host learns what it is being asked for +// only after fetching it — which is also why the ref alone can never answer +// "is an update available" here. + +const MANIFEST_TAG = "blitz-box:2026-08-31"; + +test("a manifest ref downloads, verifies, loads and replaces the container", async () => { + const image = manifestAssets({ imageTag: MANIFEST_TAG }); + await withHost( + { loadProduces: MANIFEST_TAG }, + (planeOrigin) => configFor(`${planeOrigin}${image.ref}`, planeOrigin, image.totalSha256), + async (root, plane, run) => { + const result = await run(); + assert.equal(result.status, 0, result.report); + // The parts are concatenated in manifest order and the whole archive is + // what reaches docker load, byte for byte. + assert.deepEqual(readFileSync(path.join(root, "loaded.archive")), image.payload); + assert.equal(result.image, MANIFEST_TAG); + assert.equal(result.running, "true"); + // The new image's env defaults replaced the old file, same as a pull. + assert.equal(result.envDefaults, `BLITZ_FROM=${MANIFEST_TAG}\n`); + assert.ok(!result.dockerCalls.some((call) => call.startsWith("pull"))); + assert.match(result.log, /update complete/u); + // `ref` is the manifest URL the deployment pins; `tag` is the image that + // URL resolved to. Only the second can be compared against a machine. + assert.equal(plane.reports.length, 1); + assert.equal( + plane.reports[0].body, + JSON.stringify({ + ref: `${plane.origin}${image.ref}`, + outcome: "updated", + tag: MANIFEST_TAG, + }), + ); + }, + { assets: image.assets }, + ); +}); + +// The security case. A part whose bytes do not match the digest the manifest +// promises is a corrupt or tampered archive, and it must never be loaded — nor +// may it disturb the container that is running fine. +test("a part that fails its digest is never loaded and leaves the container running", async () => { + const image = manifestAssets({ imageTag: MANIFEST_TAG, corruptPart: "part-1" }); + await withHost( + { loadProduces: MANIFEST_TAG }, + (planeOrigin) => configFor(`${planeOrigin}${image.ref}`, planeOrigin, image.totalSha256), + async (root, plane, run) => { + const result = await run(); + assert.equal(result.status, 0, result.report); + assert.ok( + !result.dockerCalls.includes("load"), + `a mismatched archive reached docker load: ${result.dockerCalls.join(" | ")}`, + ); + assert.ok(!result.dockerCalls.includes("rm -f blitz-box")); + assert.equal(result.image, RUNNING_REF); + assert.equal(result.running, "true"); + // The installer's exit code is what the updater turns into the outcome. + assert.match(result.log, /image install exited 11/u); + assert.equal( + plane.reports[0].body, + JSON.stringify({ + ref: `${plane.origin}${image.ref}`, + outcome: "digest-mismatch", + tag: RUNNING_REF, + }), + ); + assert.ok(root); + }, + { assets: image.assets }, + ); +}); + +test("a part that does not download reports download-failed and touches nothing", async () => { + const image = manifestAssets({ imageTag: MANIFEST_TAG, missingPart: "part-0" }); + await withHost( + { loadProduces: MANIFEST_TAG }, + (planeOrigin) => configFor(`${planeOrigin}${image.ref}`, planeOrigin, image.totalSha256), + async (root, plane, run) => { + const result = await run(); + assert.equal(result.status, 0, result.report); + assert.ok(!result.dockerCalls.includes("load")); + assert.ok(!result.dockerCalls.includes("rm -f blitz-box")); + assert.equal(result.image, RUNNING_REF); + assert.equal( + plane.reports[0].body, + JSON.stringify({ + ref: `${plane.origin}${image.ref}`, + outcome: "download-failed", + tag: RUNNING_REF, + }), + ); + assert.ok(root); + }, + { assets: image.assets }, + ); +}); + +test("an archive docker load refuses reports load-failed and leaves the container running", async () => { + const image = manifestAssets({ imageTag: MANIFEST_TAG }); + await withHost( + { refuseLoad: true }, + (planeOrigin) => configFor(`${planeOrigin}${image.ref}`, planeOrigin, image.totalSha256), + async (root, plane, run) => { + const result = await run(); + assert.equal(result.status, 0, result.report); + assert.ok(!result.dockerCalls.includes("rm -f blitz-box")); + assert.equal(result.image, RUNNING_REF); + assert.equal( + plane.reports[0].body, + JSON.stringify({ + ref: `${plane.origin}${image.ref}`, + outcome: "load-failed", + tag: RUNNING_REF, + }), + ); + assert.ok(root); + }, + { assets: image.assets }, + ); +}); + +// The ref does not move between rebakes under a manifest pin, so this is the +// case a five-minute timer hits over and over once a box is current. It must +// cost one manifest fetch and no download at all. +test("a manifest whose tag already runs reports up-to-date without downloading parts", async () => { + const image = manifestAssets({ imageTag: MANIFEST_TAG }); + await withHost( + { runningRef: MANIFEST_TAG }, + (planeOrigin) => configFor(`${planeOrigin}${image.ref}`, planeOrigin, image.totalSha256), + async (root, plane, run) => { + const result = await run(); + assert.equal(result.status, 0, result.report); + assert.ok(!result.dockerCalls.includes("load")); + assert.ok(!result.dockerCalls.includes("rm -f blitz-box")); + assert.equal(result.image, MANIFEST_TAG); + assert.equal( + plane.reports[0].body, + JSON.stringify({ + ref: `${plane.origin}${image.ref}`, + outcome: "up-to-date", + tag: MANIFEST_TAG, + }), + ); + assert.ok(root); + }, + { assets: image.assets }, + ); +}); + +// A previous attempt can have loaded the image and then failed to start it. +// Re-downloading gigabytes to reach layers the store already holds helps +// nobody, so the store is checked before the network is. +test("an image already in the local store is not downloaded again", async () => { + const image = manifestAssets({ imageTag: MANIFEST_TAG, missingPart: "part-0" }); + await withHost( + { storedImages: [MANIFEST_TAG] }, + (planeOrigin) => configFor(`${planeOrigin}${image.ref}`, planeOrigin, image.totalSha256), + async (root, plane, run) => { + const result = await run(); + assert.equal(result.status, 0, result.report); + // part-0 would 404 if it were fetched, so reaching `updated` at all + // proves the store was consulted before the network; and nothing was + // loaded, because there was nothing to load. + assert.equal(result.image, MANIFEST_TAG); + assert.ok(!result.dockerCalls.includes("load")); + assert.equal( + plane.reports[0].body, + JSON.stringify({ + ref: `${plane.origin}${image.ref}`, + outcome: "updated", + tag: MANIFEST_TAG, + }), + ); + assert.ok(root); + }, + { assets: image.assets }, + ); +}); + +// ---- the credential the updater holds ---- +// +// A box access token lives 15 minutes; this timer runs every 5. Nothing else +// on the VM keeps the on-disk file fresh — the Go client inside the container +// rotates only in reaction to its own 401, which needs somebody to run +// blitz-cred. A quiet box therefore reaches a state where every poll 401s +// forever, the update flag never clears, and the button that set it dies +// silently. That was live on blitzos-dev: file mtime 02:03, 401s from 02:20 on. + +test("an expired access token is rotated and the poll retried", async () => { + await withHost( + {}, + (planeOrigin) => configFor(NEXT_REF, planeOrigin), + async (root, plane, run) => { + const result = await run({ token: "stale-access-token" }); + assert.equal(result.status, 0, result.report); + // The plane refused the stale token first, then the rotated one worked. + assert.deepEqual(plane.refused, ["/workspaces/self/box-config"]); + assert.deepEqual(plane.grants, [ + { grant_type: "refresh_token", refresh_token: "box-refresh-token" }, + ]); + // The rotation was written back to the file every other reader on the + // box shares, not just held in this process. + assert.deepEqual(credential(root), { + box_id: "box", + access_token: "live-access-token-rotated", + refresh_token: "box-refresh-token-rotated", + }); + // The update itself went through on the rotated token. + assert.equal(result.image, NEXT_REF); + assert.equal(plane.reports[0].authorization, "Bearer live-access-token-rotated"); + assert.match(result.log, /credential refresh/u); + }, + { token: { access: "live-access-token", refresh: "box-refresh-token" } }, + ); +}); + +test("a refresh token the control plane rejects leaves the credential alone", async () => { + await withHost( + {}, + (planeOrigin) => configFor(NEXT_REF, planeOrigin), + async (root, plane, run) => { + const result = await run({ token: "stale-access-token", refresh: "revoked-refresh-token" }); + // A box whose family was revoked cannot recover on its own, and it must + // say so rather than replace the container on a guess. + assert.equal(result.status, 0, result.report); + assert.deepEqual(result.dockerCalls, []); + assert.deepEqual(plane.reports, []); + assert.match(result.log, /credential refresh failed/u); + assert.match(result.log, /poll failed/u); + assert.equal(credential(root).refresh_token, "revoked-refresh-token"); + }, + { token: { access: "live-access-token", refresh: "box-refresh-token" } }, + ); +}); + +// The digest the MANIFEST declares is self-certifying: whoever serves the +// manifest serves the digest beside it. The control plane pins its own copy, +// which arrives over a different connection, and the host checks both. This is +// what makes the updater's verification as strong as the first boot's. +test("an archive that does not match the control plane's pinned digest is refused", async () => { + const image = manifestAssets({ imageTag: MANIFEST_TAG }); + await withHost( + { loadProduces: MANIFEST_TAG }, + // A manifest that is internally consistent — every part digest and the + // total agree — and still is not the image this deployment pinned. + (planeOrigin) => configFor(`${planeOrigin}${image.ref}`, planeOrigin, "c".repeat(64)), + async (root, plane, run) => { + const result = await run(); + assert.equal(result.status, 0, result.report); + assert.ok( + !result.dockerCalls.includes("load"), + `an unpinned archive reached docker load: ${result.dockerCalls.join(" | ")}`, + ); + assert.ok(!result.dockerCalls.includes("rm -f blitz-box")); + assert.equal(result.image, RUNNING_REF); + assert.equal( + plane.reports[0].body, + JSON.stringify({ + ref: `${plane.origin}${image.ref}`, + outcome: "digest-mismatch", + tag: RUNNING_REF, + }), + ); + assert.ok(root); + }, + { assets: image.assets }, + ); +}); diff --git a/packages/control-plane/test/core-imports.test.ts b/packages/control-plane/test/core-imports.test.ts index 0d098d9e..dcc03735 100644 --- a/packages/control-plane/test/core-imports.test.ts +++ b/packages/control-plane/test/core-imports.test.ts @@ -64,6 +64,7 @@ const expected = [ "identity/routes.ts", "index.ts", "janitors.ts", + "machine-access.ts", "machine-stats.ts", "machines.ts", "oauth.ts", @@ -112,6 +113,7 @@ const expected = [ "webapp-proxy.ts", "webapp-surface.ts", "webapp-tickets.ts", + "wire-box-config.ts", "wire-machines.ts", "wire-sharing.ts", "wire.ts", @@ -134,6 +136,6 @@ describe("portable core imports", () => { (values: string[]) => values.every((value) => value.startsWith("./") || value.startsWith("../")), ); } - expect(expected).toHaveLength(109); + expect(expected).toHaveLength(111); }); }); diff --git a/packages/control-plane/test/wire-drift.test.ts b/packages/control-plane/test/wire-drift.test.ts index 06d1aa04..8c9dfce0 100644 --- a/packages/control-plane/test/wire-drift.test.ts +++ b/packages/control-plane/test/wire-drift.test.ts @@ -68,6 +68,8 @@ const boxConfigResponse: SharedShape< schema.BoxConfigResponse > = { boxImageRef: "ghcr.io/blitzdotdev/blitz-box:v2", + // Empty is what a registry pin sends: the ref carries its own digest there. + boxImageSha256: "", controlPlaneOrigin: "https://cp.example", updateRequested: true, }; @@ -145,16 +147,25 @@ const machine: SharedShape = { volumeUsedPercent: 62, membershipId: "membership", error: null, + boxImage: "blitz-box:2026-08-01", + boxImageTarget: "blitz-box:2026-08-31", + boxUpdateRequested: true, + boxUpdateOutcome: "digest-mismatch", createdAt: 1_700_000_000_000, updatedAt: 1_700_000_005_000, }; // A machine whose guest has not reported yet answers null, which covers -// different ground than an integer does. +// different ground than an integer does. The same is true of its image and +// its last update verdict: "never reported" is not "up to date", and not +// "no update was attempted successfully" either. const unreportedMachine: SharedShape = { ...machine, id: "machine-unreported", volumeUsedPercent: null, + boxImage: null, + boxUpdateRequested: false, + boxUpdateOutcome: null, }; const machineStats: SharedShape< diff --git a/packages/schema/fixtures/box-config/README.md b/packages/schema/fixtures/box-config/README.md index 92e8572c..f6d06321 100644 --- a/packages/schema/fixtures/box-config/README.md +++ b/packages/schema/fixtures/box-config/README.md @@ -11,23 +11,40 @@ consumer and the update-result producer. This corpus pins both directions. with whether the host consumer must accept it (`accepts`). The accept rule: a JSON object whose `boxImageRef` is one image-reference token (`[A-Za-z0-9][A-Za-z0-9._/:@-]*` — a registry ref or the R2 tarball https -URL), whose `controlPlaneOrigin` is exactly an http(s) origin (scheme, host, +URL), whose `boxImageSha256` is a 64-character SHA-256 digest or empty (empty +under a registry pin, where the ref carries its own digest; it is what makes +the host's verification as strong as the first boot's, because the digest the +MANIFEST declares is self-certifying), whose `controlPlaneOrigin` is exactly +an http(s) origin (scheme, host, optional port, nothing after — the host writes it verbatim into `/var/lib/blitz/origin`, which the box gateway compares against the browser Origin header), and whose `updateRequested` is a boolean. Unknown extra keys -are tolerated on both sides for forward compatibility. A tarball https -`boxImageRef` is accepted by the parser; the updater then reports the attempt -`unsupported` rather than rejecting the poll, so the origin refresh still -happens. On a rejected envelope the host changes nothing and keeps polling. +are tolerated on both sides for forward compatibility. An https `boxImageRef` +is accepted by the parser; what the updater then does with it depends on the +shape — a `.../manifest.json` URL is fetched, verified and loaded, and any +other https ref is reported `unsupported` rather than rejecting the poll, so +the origin refresh still happens either way. On a rejected envelope the host +changes nothing and keeps polling. `result-*.json` fixtures pair a candidate update-result request body (`request`) with whether the control-plane consumer must accept it (`accepts`): `ref` is one image-reference token and `outcome` is one of -`updated`, `up-to-date`, `rolled-back`, `pull-failed`, `start-failed`, -`unsupported` (`BOX_UPDATE_OUTCOMES` in `packages/schema/src/box-config.ts`). +`updated`, `up-to-date`, `rolled-back`, `pull-failed`, `download-failed`, +`digest-mismatch`, `load-failed`, `start-failed`, `unsupported` +(`BOX_UPDATE_OUTCOMES` in `packages/schema/src/box-config.ts`). + +`tag` is optional and, when present, is one image-reference token too. It is +the CONCRETE image the container runs once the attempt has settled — the tag +from inside the manifest under an R2 pin, the ref itself under a registry pin, +and the OLD image whenever the attempt left the container alone. It exists +because `ref` alone cannot answer "is an update available" under a manifest +pin, whose URL is byte-identical across rebakes while the tag inside it moves. +A host emitted before the manifest branch sends no `tag` at all, and the +control plane then leaves the stored one alone rather than nulling it. + Extra keys are tolerated on purpose: hosts only update by shipping new images, so an older control plane must keep accepting a newer host's report -or the workspace's update flag would stay set forever. +or the machine's update flag would stay set forever. Conformance: the control-plane side is `packages/control-plane/test/box-config-conformance.test.ts`; the host side diff --git a/packages/schema/fixtures/box-config/config-bad-image-sha256.json b/packages/schema/fixtures/box-config/config-bad-image-sha256.json new file mode 100644 index 00000000..f03431c0 --- /dev/null +++ b/packages/schema/fixtures/box-config/config-bad-image-sha256.json @@ -0,0 +1,9 @@ +{ + "response": { + "boxImageRef": "https://cp.example/box-image/manifest.json", + "boxImageSha256": "not-a-digest", + "controlPlaneOrigin": "https://cp.example", + "updateRequested": true + }, + "accepts": false +} diff --git a/packages/schema/fixtures/box-config/config-image-ref-with-space.json b/packages/schema/fixtures/box-config/config-image-ref-with-space.json index 166bcd58..e34dfbff 100644 --- a/packages/schema/fixtures/box-config/config-image-ref-with-space.json +++ b/packages/schema/fixtures/box-config/config-image-ref-with-space.json @@ -1,6 +1,7 @@ { "response": { "boxImageRef": "ghcr.io/blitzdotdev/blitz-box:v2 --privileged", + "boxImageSha256": "", "controlPlaneOrigin": "https://cp.example", "updateRequested": true }, diff --git a/packages/schema/fixtures/box-config/config-missing-image-ref.json b/packages/schema/fixtures/box-config/config-missing-image-ref.json index 6faedbbe..fddd75a5 100644 --- a/packages/schema/fixtures/box-config/config-missing-image-ref.json +++ b/packages/schema/fixtures/box-config/config-missing-image-ref.json @@ -1,7 +1,8 @@ { "response": { "controlPlaneOrigin": "https://cp.example", - "updateRequested": false + "updateRequested": false, + "boxImageSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" }, "accepts": false } diff --git a/packages/schema/fixtures/box-config/config-non-boolean-update-requested.json b/packages/schema/fixtures/box-config/config-non-boolean-update-requested.json index a117d0ba..9920f8e3 100644 --- a/packages/schema/fixtures/box-config/config-non-boolean-update-requested.json +++ b/packages/schema/fixtures/box-config/config-non-boolean-update-requested.json @@ -1,6 +1,7 @@ { "response": { "boxImageRef": "ghcr.io/blitzdotdev/blitz-box:v2", + "boxImageSha256": "", "controlPlaneOrigin": "https://cp.example", "updateRequested": "yes" }, diff --git a/packages/schema/fixtures/box-config/config-origin-with-path.json b/packages/schema/fixtures/box-config/config-origin-with-path.json index 6e93b190..dc765984 100644 --- a/packages/schema/fixtures/box-config/config-origin-with-path.json +++ b/packages/schema/fixtures/box-config/config-origin-with-path.json @@ -1,6 +1,7 @@ { "response": { "boxImageRef": "ghcr.io/blitzdotdev/blitz-box:v2", + "boxImageSha256": "", "controlPlaneOrigin": "https://cp.example/app", "updateRequested": false }, diff --git a/packages/schema/fixtures/box-config/config-valid-extra-key.json b/packages/schema/fixtures/box-config/config-valid-extra-key.json index ab571ae7..1adf7bef 100644 --- a/packages/schema/fixtures/box-config/config-valid-extra-key.json +++ b/packages/schema/fixtures/box-config/config-valid-extra-key.json @@ -1,6 +1,7 @@ { "response": { "boxImageRef": "ghcr.io/blitzdotdev/blitz-box:v2", + "boxImageSha256": "", "controlPlaneOrigin": "https://cp.example", "updateRequested": false, "note": "unknown keys are tolerated for forward compatibility" diff --git a/packages/schema/fixtures/box-config/config-valid-minimal.json b/packages/schema/fixtures/box-config/config-valid-minimal.json index 32e16a50..20fdfbbf 100644 --- a/packages/schema/fixtures/box-config/config-valid-minimal.json +++ b/packages/schema/fixtures/box-config/config-valid-minimal.json @@ -1,6 +1,7 @@ { "response": { "boxImageRef": "ghcr.io/blitzdotdev/blitz-box@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "boxImageSha256": "", "controlPlaneOrigin": "https://cp.example", "updateRequested": false }, diff --git a/packages/schema/fixtures/box-config/config-valid-tarball-ref.json b/packages/schema/fixtures/box-config/config-valid-tarball-ref.json index 7ba39b37..2b2e4c11 100644 --- a/packages/schema/fixtures/box-config/config-valid-tarball-ref.json +++ b/packages/schema/fixtures/box-config/config-valid-tarball-ref.json @@ -1,6 +1,7 @@ { "response": { "boxImageRef": "https://cp.example/box-image/manifest.json", + "boxImageSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", "controlPlaneOrigin": "https://cp.example", "updateRequested": true }, diff --git a/packages/schema/fixtures/box-config/config-valid-update-requested.json b/packages/schema/fixtures/box-config/config-valid-update-requested.json index 0d2f01b2..a5e3e27b 100644 --- a/packages/schema/fixtures/box-config/config-valid-update-requested.json +++ b/packages/schema/fixtures/box-config/config-valid-update-requested.json @@ -1,6 +1,7 @@ { "response": { "boxImageRef": "ghcr.io/blitzdotdev/blitz-box:v2", + "boxImageSha256": "", "controlPlaneOrigin": "https://blitzos.example:8443", "updateRequested": true }, diff --git a/packages/schema/fixtures/box-config/result-tag-with-space.json b/packages/schema/fixtures/box-config/result-tag-with-space.json new file mode 100644 index 00000000..627bbe84 --- /dev/null +++ b/packages/schema/fixtures/box-config/result-tag-with-space.json @@ -0,0 +1,8 @@ +{ + "request": { + "ref": "https://cp.example/box-image/manifest.json", + "outcome": "updated", + "tag": "blitz-box:2026-08-31 --privileged" + }, + "accepts": false +} diff --git a/packages/schema/fixtures/box-config/result-valid-digest-mismatch.json b/packages/schema/fixtures/box-config/result-valid-digest-mismatch.json new file mode 100644 index 00000000..8ba94b09 --- /dev/null +++ b/packages/schema/fixtures/box-config/result-valid-digest-mismatch.json @@ -0,0 +1,8 @@ +{ + "request": { + "ref": "https://cp.example/box-image/manifest.json", + "outcome": "digest-mismatch", + "tag": "blitz-box:2026-08-30" + }, + "accepts": true +} diff --git a/packages/schema/fixtures/box-config/result-valid-download-failed.json b/packages/schema/fixtures/box-config/result-valid-download-failed.json new file mode 100644 index 00000000..2057f096 --- /dev/null +++ b/packages/schema/fixtures/box-config/result-valid-download-failed.json @@ -0,0 +1,8 @@ +{ + "request": { + "ref": "https://cp.example/box-image/manifest.json", + "outcome": "download-failed", + "tag": "blitz-box:2026-08-30" + }, + "accepts": true +} diff --git a/packages/schema/fixtures/box-config/result-valid-load-failed.json b/packages/schema/fixtures/box-config/result-valid-load-failed.json new file mode 100644 index 00000000..fcf3de5b --- /dev/null +++ b/packages/schema/fixtures/box-config/result-valid-load-failed.json @@ -0,0 +1,8 @@ +{ + "request": { + "ref": "https://cp.example/box-image/manifest.json", + "outcome": "load-failed", + "tag": "blitz-box:2026-08-30" + }, + "accepts": true +} diff --git a/packages/schema/fixtures/box-config/result-valid-manifest-tag.json b/packages/schema/fixtures/box-config/result-valid-manifest-tag.json new file mode 100644 index 00000000..0cbe00fe --- /dev/null +++ b/packages/schema/fixtures/box-config/result-valid-manifest-tag.json @@ -0,0 +1,8 @@ +{ + "request": { + "ref": "https://cp.example/box-image/manifest.json", + "outcome": "updated", + "tag": "blitz-box:2026-08-31" + }, + "accepts": true +} diff --git a/packages/schema/src/box-config.ts b/packages/schema/src/box-config.ts index 1f930e38..3d085513 100644 --- a/packages/schema/src/box-config.ts +++ b/packages/schema/src/box-config.ts @@ -9,25 +9,47 @@ * `controlPlaneOrigin` is the one origin the box gateway should trust; the * host rewrites `/var/lib/blitz/origin` on every poll when it differs, which * needs no restart because the gateway re-reads the file per request. - * `updateRequested` is the per-workspace flag; image updates are request-gated - * because replacing the container kills every process inside it. */ + * `updateRequested` is the per-machine flag; image updates are request-gated + * because replacing the container kills every process inside it. + * + * `boxImageSha256` is the digest of the whole image archive, and it is what + * makes the host's verification as strong as the first boot's. Without it the + * updater could only check the archive against the digest the MANIFEST + * declares, which is self-certifying: whoever serves the manifest serves the + * digest beside it. This one arrives from the control plane instead, over a + * separate connection, exactly as the bootstrap's baked-in `BOX_IMAGE_SHA256` + * does. Empty under a registry pin, where the ref carries its own digest and + * docker checks it. */ export interface BoxConfigResponse { boxImageRef: string; + boxImageSha256: string; controlPlaneOrigin: string; updateRequested: boolean; } -/** What the host reports after an update attempt, in the order it tries: - * `up-to-date` (the requested ref already runs, nothing replaced), - * `unsupported` (a tarball https ref, which the updater cannot pull), - * `pull-failed` (the pull failed; the old container was never touched), - * `updated` (the new container runs), `rolled-back` (the new container did - * not start and the old ref runs again), `start-failed` (neither started). */ +/** What the host reports after an update attempt. + * + * Acquiring the image comes first and never touches the running container, so + * every acquire verdict means "nothing changed": `pull-failed` (a registry + * pull failed), `download-failed` (a manifest or archive part did not arrive), + * `digest-mismatch` (a part or the whole archive failed its SHA-256 — + * a corrupt or tampered archive, never loaded), `load-failed` (the archive + * verified but `docker load` refused it). + * + * The rest describe the swap: `up-to-date` (the wanted image already runs, + * nothing replaced), `updated` (the new container runs), `rolled-back` (the + * new container did not start and the old image runs again), `start-failed` + * (neither started), `unsupported` (this host's updater cannot install the + * ref at all — an https ref that is not a manifest, or any https ref on a host + * whose updater predates the manifest branch). */ export const BOX_UPDATE_OUTCOMES = [ "updated", "up-to-date", "rolled-back", "pull-failed", + "download-failed", + "digest-mismatch", + "load-failed", "start-failed", "unsupported", ] as const; @@ -36,10 +58,19 @@ export type BoxUpdateOutcome = (typeof BOX_UPDATE_OUTCOMES)[number]; /** The body of `POST /workspaces/self/box-update-result`: the host is the * producer (bash/python in the emitted updater), the control plane is the - * consumer. The control plane clears the workspace's update flag and stores + * consumer. The control plane clears the machine's update flag and stores * `ref` on the row (`box_image_reported`) whatever the outcome, so a failed - * attempt never leaves the flag re-triggering forever. */ + * attempt never leaves the flag re-triggering forever. + * + * `tag` is the CONCRETE image the `blitz-box` container runs once the attempt + * has settled — the tag under an R2 manifest ref, the ref itself under a + * registry ref, and the OLD image whenever the attempt left the container + * alone. It is optional because a host emitted before the manifest branch + * never sends it; `ref` alone cannot answer "is an update available" under a + * manifest ref, whose URL is identical across rebakes while the tag inside it + * moves. */ export interface BoxUpdateResultRequest { ref: string; outcome: BoxUpdateOutcome; + tag?: string; } diff --git a/packages/schema/src/workspace.ts b/packages/schema/src/workspace.ts index 833d4b55..ac13eded 100644 --- a/packages/schema/src/workspace.ts +++ b/packages/schema/src/workspace.ts @@ -1,3 +1,4 @@ +import type { BoxUpdateOutcome } from "./box-config.js"; import type { WorkspaceEnvironment } from "./environment.js"; export const PHASES = [ @@ -66,6 +67,23 @@ export interface MachineView { volumeUsedPercent: number | null; membershipId: string; error: string | null; + /** The CONCRETE box image this machine runs, as its host last reported it + * (or as the deployment pinned it when the machine was created). Null means + * unknown: a machine created before the host started reporting a tag, which + * has not attempted an update since. Never compare `boxImage` to a manifest + * URL — under an R2 manifest ref the URL is identical across rebakes while + * the tag inside it moves, and the tag is what this field holds. */ + boxImage: string | null; + /** The CONCRETE box image this deployment installs now. Equal to `boxImage` + * means up to date; different means an update is available. */ + boxImageTarget: string; + /** An update has been asked for and the host has not reported back yet. The + * host polls every five minutes. */ + boxUpdateRequested: boolean; + /** How the host's last update attempt ended, or null if it never made one. + * `unsupported` is the honest signal that this host's updater predates the + * manifest branch and can never self-update. */ + boxUpdateOutcome: BoxUpdateOutcome | null; createdAt: number; updatedAt: number; } diff --git a/packages/webapp/src/MyMachineDialog.tsx b/packages/webapp/src/MyMachineDialog.tsx index ae03dd7a..558356b9 100644 --- a/packages/webapp/src/MyMachineDialog.tsx +++ b/packages/webapp/src/MyMachineDialog.tsx @@ -7,6 +7,7 @@ import type { } from '@blitzos/schema'; import type { ControlPlaneClient } from './api'; import { caughtErrorMessage } from './error-message'; +import { boxUpdateStatus } from './box-update-state'; import { ConfirmationDialog } from './ConfirmationDialog'; import { monthlyPriceLabel } from './MachineCatalogGrid'; import { MachineTypeSelect } from './MachineTypeSelect'; @@ -122,6 +123,7 @@ export function MyMachineDialog({ workspace, membershipId, listMachineTypes, + refreshWorkspaces, onClose, }: { client: ControlPlaneClient; @@ -129,6 +131,9 @@ export function MyMachineDialog({ /** The requesting member's membership, which is what keys a machine. */ membershipId: string | null; listMachineTypes: () => Promise; + /** Runs the workspace poll now. An update request only shows as pending once + * the machine row comes back, and the poll is 15 seconds otherwise. */ + refreshWorkspaces: () => void; onClose: () => void; }) { const closeButton = useRef(null); @@ -138,6 +143,7 @@ export function MyMachineDialog({ const [catalog, setCatalog] = useState(NO_CATALOG); const [error, setError] = useState(null); const [pendingTypeId, setPendingTypeId] = useState(null); + const [confirmingUpdate, setConfirmingUpdate] = useState(false); const machines: MachineType[] = catalog.machineTypes; useEffect(() => { closeButton.current?.focus(); }, []); @@ -180,6 +186,7 @@ export function MyMachineDialog({ if (action === 'destroy') run(client.destroyMachine(machine.id)); }; + const updateStatus = boxUpdateStatus(machine); const actions = member === undefined || member.role === 'viewer' ? [] : machineActionsFor(machine); @@ -273,6 +280,35 @@ export function MyMachineDialog({ )} +
+
+

Box image

+

+ The software your machine runs. Updating installs the + current image and restarts the machine. +

+
+
+ + +
+

{updateStatus.summary}

+ {updateStatus.lastAttempt !== null && ( +

{updateStatus.lastAttempt}

+ )} +
+ +
+
+

Lifecycle

@@ -311,6 +347,18 @@ export function MyMachineDialog({ )}
+ {confirmingUpdate && machine !== null && ( + setConfirmingUpdate(false)} + onConfirm={() => { + setConfirmingUpdate(false); + run(client.requestMachineBoxUpdate(machine.id).then(refreshWorkspaces)); + }} + /> + )} {pendingTypeId !== null && machine !== null && ( ; startMachine(machineId: string): Promise; recreateMachine(machineId: string): Promise; + /** Ask THIS machine's host to install the deployment's current box image on + * its next poll. It names one machine: the workspace-wide route beside it on + * the server is a deliberate admin fan-out, and replacing a container kills + * every process inside it. */ + requestMachineBoxUpdate(machineId: string): Promise; /** Same-location only: the VM is replaced and the volume — the disk — stays. * Another location is refused until the volume move lands (§5). */ setMachineType(machineId: string, input: SetMachineTypeRequest): Promise; @@ -687,6 +692,10 @@ export function createControlPlaneClient(baseUrl = ""): ControlPlaneClient { stopMachine: (machineId) => machineAction(machineId, "stop"), startMachine: (machineId) => machineAction(machineId, "start"), recreateMachine: (machineId) => machineAction(machineId, "recreate"), + requestMachineBoxUpdate: (machineId) => request( + `/machines/${encodeURIComponent(machineId)}/box-update`, + { method: "POST" }, + ), setMachineType: (machineId, input) => request( `/machines/${encodeURIComponent(machineId)}/machine-type`, { method: "POST", headers: jsonHeaders, body: JSON.stringify(input) }, diff --git a/packages/webapp/src/box-update-state.ts b/packages/webapp/src/box-update-state.ts new file mode 100644 index 00000000..b49641eb --- /dev/null +++ b/packages/webapp/src/box-update-state.ts @@ -0,0 +1,132 @@ +import type { BoxUpdateOutcome, MachineView } from '@blitzos/schema'; + +/** + * What the My machine dialog can honestly say about a machine's box image. + * + * Kept out of the component because the interesting part is the judgement, not + * the markup: a machine can be up to date, behind, unable to update at all, or + * simply unknown, and telling a user the wrong one of those is worse than + * telling them nothing. It is pure, so `test/box-update-state.test.ts` covers + * the whole table without rendering anything. + */ +export type BoxUpdateKind = + /** No machine row, so there is nothing to update. */ + | 'no-machine' + /** An update is requested and the host has not reported back yet. */ + | 'pending' + /** This machine's host updater predates the manifest branch and reported + * `unsupported`. It can never install this deployment's image in place. */ + | 'unsupported' + /** The machine is not running, so replacing its container is not on offer. */ + | 'not-running' + /** The machine has never reported an image, so the comparison has no answer. */ + | 'unknown' + | 'up-to-date' + | 'available'; + +export interface BoxUpdateStatus { + kind: BoxUpdateKind; + /** The one-line summary shown beside the button. */ + summary: string; + /** What the last attempt did, when the state above does not already say it. + * Null when there was no attempt, or when it told us nothing new. */ + lastAttempt: string | null; + /** Whether asking for an update now could accomplish anything. */ + canRequest: boolean; +} + +/** + * What the host's last verdict means for the person reading it. + * + * Only the outcomes that leave something worth saying return a sentence. + * `updated`, `up-to-date` and `unsupported` are already carried by the kind, + * so repeating them under the status line would just be noise. + * + * Every acquire failure says the machine was left alone, because that is the + * updater's invariant and it is the thing a worried user wants to know first. + */ +function lastAttemptSentence(outcome: BoxUpdateOutcome | null): string | null { + switch (outcome) { + case 'rolled-back': + return 'The last attempt could not start the new image, so your machine went back to the one it was running.'; + case 'start-failed': + return 'The last attempt left no container running. Recreate the machine if it is still down.'; + case 'pull-failed': + return 'The last attempt could not fetch the new image. Your machine was left untouched.'; + case 'download-failed': + return 'The last attempt could not download the new image. Your machine was left untouched.'; + case 'digest-mismatch': + return 'The last attempt downloaded a damaged image and refused it. Your machine was left untouched.'; + case 'load-failed': + return 'The last attempt could not load the new image. Your machine was left untouched.'; + default: + return null; + } +} + +export function boxUpdateStatus(machine: MachineView | null): BoxUpdateStatus { + if (machine === null) { + return { + kind: 'no-machine', + summary: 'You have no machine yet.', + lastAttempt: null, + canRequest: false, + }; + } + const lastAttempt = lastAttemptSentence(machine.boxUpdateOutcome); + + // Pending outranks everything: an update is already on its way, and the + // image the row still reports is the one it is on its way from. + if (machine.boxUpdateRequested) { + return { + kind: 'pending', + summary: 'Update requested. Your machine picks it up within five minutes and restarts.', + lastAttempt: null, + canRequest: false, + }; + } + // A host that answered `unsupported` will answer it again. Saying "update + // available" and handing over a button that cannot work would be a lie, and + // this covers every box created before the manifest updater shipped. + if (machine.boxUpdateOutcome === 'unsupported') { + return { + kind: 'unsupported', + summary: 'This machine’s host cannot update in place. Recreate it to move to the new image.', + lastAttempt: null, + canRequest: false, + }; + } + if (machine.state !== 'running') { + return { + kind: 'not-running', + summary: `A machine that is ${machine.state} cannot update. Start it first.`, + lastAttempt, + canRequest: false, + }; + } + // Never reported. Asking is still worth doing — the attempt is what makes + // the machine report — but the answer is not known yet, and guessing + // "up to date" here is how a stale box looks current forever. + if (machine.boxImage === null) { + return { + kind: 'unknown', + summary: 'This machine has not reported which image it runs.', + lastAttempt, + canRequest: true, + }; + } + if (machine.boxImage === machine.boxImageTarget) { + return { + kind: 'up-to-date', + summary: 'Up to date.', + lastAttempt, + canRequest: false, + }; + } + return { + kind: 'available', + summary: 'Update available.', + lastAttempt, + canRequest: true, + }; +} diff --git a/packages/webapp/src/shell/ShellDialogs.tsx b/packages/webapp/src/shell/ShellDialogs.tsx index b757a97b..8824b7fc 100644 --- a/packages/webapp/src/shell/ShellDialogs.tsx +++ b/packages/webapp/src/shell/ShellDialogs.tsx @@ -142,6 +142,7 @@ export function ShellDialogs({ workspace={machineWorkspace} membershipId={viewer?.membership.id ?? null} listMachineTypes={listMachineTypes} + refreshWorkspaces={refreshWorkspaces} onClose={onCloseMachine} /> )} diff --git a/packages/webapp/test/WorkspaceDetailsDialog.test.tsx b/packages/webapp/test/WorkspaceDetailsDialog.test.tsx index fe63f73a..2fd0062d 100644 --- a/packages/webapp/test/WorkspaceDetailsDialog.test.tsx +++ b/packages/webapp/test/WorkspaceDetailsDialog.test.tsx @@ -55,6 +55,10 @@ const ada: WorkspaceMemberView = { machineTypeId: 'cx23@fsn1', volumeId: 'volume-one', volumeUsedPercent: 62, + boxImage: 'blitz-box:2026-08-31', + boxImageTarget: 'blitz-box:2026-08-31', + boxUpdateRequested: false, + boxUpdateOutcome: null, membershipId: 'membership-1', error: null, createdAt: 1_700_000_000_000, @@ -693,6 +697,10 @@ describe('machineActionsFor', () => { machineTypeId: 'cx23@fsn1', volumeId: 'volume-one', volumeUsedPercent: null, + boxImage: 'blitz-box:2026-08-31', + boxImageTarget: 'blitz-box:2026-08-31', + boxUpdateRequested: false, + boxUpdateOutcome: null, membershipId: 'membership-1', error: null, createdAt: 1, diff --git a/packages/webapp/test/admin-connections.test.tsx b/packages/webapp/test/admin-connections.test.tsx index 08aa77c1..3d3cf92f 100644 --- a/packages/webapp/test/admin-connections.test.tsx +++ b/packages/webapp/test/admin-connections.test.tsx @@ -44,6 +44,7 @@ function client(overrides: Partial = {}): ControlPlaneClient stopMachine: vi.fn(async () => { throw new Error('unused'); }), startMachine: vi.fn(async () => { throw new Error('unused'); }), recreateMachine: vi.fn(async () => { throw new Error('unused'); }), + requestMachineBoxUpdate: vi.fn(async () => { throw new Error('unused'); }), setMachineType: vi.fn(async () => { throw new Error('unused'); }), destroyMachine: vi.fn(async () => { throw new Error('unused'); }), putWorkspaceCredential: vi.fn(async () => undefined), diff --git a/packages/webapp/test/api-adapter.test.ts b/packages/webapp/test/api-adapter.test.ts index 580458e8..ce3d853b 100644 --- a/packages/webapp/test/api-adapter.test.ts +++ b/packages/webapp/test/api-adapter.test.ts @@ -53,6 +53,7 @@ function client(overrides: Partial = {}): ControlPlaneClient stopMachine: vi.fn(async () => { throw new Error("unused"); }), startMachine: vi.fn(async () => { throw new Error("unused"); }), recreateMachine: vi.fn(async () => { throw new Error("unused"); }), + requestMachineBoxUpdate: vi.fn(async () => { throw new Error("unused"); }), setMachineType: vi.fn(async () => { throw new Error("unused"); }), destroyMachine: vi.fn(async () => { throw new Error("unused"); }), putWorkspaceCredential: vi.fn(async () => undefined), diff --git a/packages/webapp/test/box-update-state.test.ts b/packages/webapp/test/box-update-state.test.ts new file mode 100644 index 00000000..527bc917 --- /dev/null +++ b/packages/webapp/test/box-update-state.test.ts @@ -0,0 +1,142 @@ +import { describe, expect, it } from 'vitest'; +import type { MachineView } from '@blitzos/schema'; +import { boxUpdateStatus } from '../src/box-update-state'; + +// The whole judgement table. Telling a user "up to date" about a machine that +// is behind, or "update available" about a host that can never install one, is +// worse than telling them nothing — so every branch is named here. + +function machine(overrides: Partial = {}): MachineView { + return { + id: 'machine-one', + state: 'running', + machineTypeId: 'cx23@fsn1', + volumeId: 'volume-one', + volumeUsedPercent: 62, + membershipId: 'membership-1', + error: null, + boxImage: 'blitz-box:2026-08-31', + boxImageTarget: 'blitz-box:2026-08-31', + boxUpdateRequested: false, + boxUpdateOutcome: null, + createdAt: 1_700_000_000_000, + updatedAt: 1_700_000_005_000, + ...overrides, + }; +} + +describe('boxUpdateStatus', () => { + it('offers nothing when there is no machine', () => { + const status = boxUpdateStatus(null); + expect(status.kind).toBe('no-machine'); + expect(status.canRequest).toBe(false); + }); + + it('reads a machine on the deployment image as up to date', () => { + const status = boxUpdateStatus(machine()); + expect(status.kind).toBe('up-to-date'); + expect(status.summary).toBe('Up to date.'); + // Nothing to do, so nothing is offered. + expect(status.canRequest).toBe(false); + }); + + it('offers the update when the machine runs an older image', () => { + const status = boxUpdateStatus(machine({ boxImage: 'blitz-box:2026-08-01' })); + expect(status.kind).toBe('available'); + expect(status.canRequest).toBe(true); + }); + + // The manifest pin is why the comparison is on the concrete image: the ref + // is byte-identical across rebakes, so comparing refs would read every box + // as current forever. + it('compares the concrete image, not the ref', () => { + const behind = machine({ + boxImage: 'blitz-box:2026-08-01', + boxImageTarget: 'blitz-box:2026-08-31', + }); + expect(boxUpdateStatus(behind).kind).toBe('available'); + }); + + it('says so plainly when the machine has never reported an image', () => { + const status = boxUpdateStatus(machine({ boxImage: null })); + expect(status.kind).toBe('unknown'); + expect(status.summary).toContain('has not reported'); + // Asking is still worth doing: the attempt is what makes it report. + expect(status.canRequest).toBe(true); + }); + + it('shows a requested update as pending and offers no second click', () => { + const status = boxUpdateStatus(machine({ + boxImage: 'blitz-box:2026-08-01', + boxUpdateRequested: true, + })); + expect(status.kind).toBe('pending'); + expect(status.summary).toContain('five minutes'); + expect(status.canRequest).toBe(false); + }); + + // Every box created before the manifest updater shipped — which is every + // pre-existing canary box, blitzos-dev included — lands here after one + // request. The honest answer is that it can never do this in place. + it('refuses to offer an update a host reported it cannot do', () => { + const status = boxUpdateStatus(machine({ + boxImage: 'blitz-box:2026-08-01', + boxUpdateOutcome: 'unsupported', + })); + expect(status.kind).toBe('unsupported'); + expect(status.summary).toContain('cannot update in place'); + expect(status.summary).toContain('Recreate it'); + expect(status.canRequest).toBe(false); + }); + + it('will not offer an update to a machine that is not running', () => { + const status = boxUpdateStatus(machine({ + state: 'stopped', + boxImage: 'blitz-box:2026-08-01', + })); + expect(status.kind).toBe('not-running'); + expect(status.summary).toContain('stopped'); + expect(status.canRequest).toBe(false); + }); + + // Every acquire failure leaves the container alone, and that is the first + // thing a worried user wants to know. The update is still on offer, because + // a failed fetch is worth retrying. + it.each([ + ['pull-failed', 'could not fetch'], + ['download-failed', 'could not download'], + ['digest-mismatch', 'damaged image and refused it'], + ['load-failed', 'could not load'], + ] as const)('reports %s as leaving the machine untouched', (outcome, phrase) => { + const status = boxUpdateStatus(machine({ + boxImage: 'blitz-box:2026-08-01', + boxUpdateOutcome: outcome, + })); + expect(status.kind).toBe('available'); + expect(status.canRequest).toBe(true); + expect(status.lastAttempt).toContain(phrase); + expect(status.lastAttempt).toContain('left untouched'); + }); + + it('reports a rollback as the machine going back to what it had', () => { + const status = boxUpdateStatus(machine({ + boxImage: 'blitz-box:2026-08-01', + boxUpdateOutcome: 'rolled-back', + })); + expect(status.lastAttempt).toContain('went back to the one it was running'); + expect(status.canRequest).toBe(true); + }); + + it('tells a user whose machine is down that recreating is the way out', () => { + const status = boxUpdateStatus(machine({ + boxImage: 'blitz-box:2026-08-01', + boxUpdateOutcome: 'start-failed', + })); + expect(status.lastAttempt).toContain('Recreate the machine'); + }); + + // A verdict the state line already carries is not repeated underneath it. + it.each(['updated', 'up-to-date'] as const)('says nothing extra after %s', (outcome) => { + expect(boxUpdateStatus(machine({ boxUpdateOutcome: outcome })).lastAttempt).toBeNull(); + }); +}); diff --git a/packages/webapp/test/credentials-surfaces-v2.test.tsx b/packages/webapp/test/credentials-surfaces-v2.test.tsx index 841fa3bc..d74417cd 100644 --- a/packages/webapp/test/credentials-surfaces-v2.test.tsx +++ b/packages/webapp/test/credentials-surfaces-v2.test.tsx @@ -47,6 +47,7 @@ function client(overrides: Partial = {}): ControlPlaneClient stopMachine: vi.fn(async () => { throw new Error('unused'); }), startMachine: vi.fn(async () => { throw new Error('unused'); }), recreateMachine: vi.fn(async () => { throw new Error('unused'); }), + requestMachineBoxUpdate: vi.fn(async () => { throw new Error('unused'); }), setMachineType: vi.fn(async () => { throw new Error('unused'); }), destroyMachine: vi.fn(async () => { throw new Error('unused'); }), putWorkspaceCredential: vi.fn(async () => undefined), diff --git a/packages/webapp/test/my-machine.test.tsx b/packages/webapp/test/my-machine.test.tsx index fe6cbb8d..25748521 100644 --- a/packages/webapp/test/my-machine.test.tsx +++ b/packages/webapp/test/my-machine.test.tsx @@ -2,6 +2,7 @@ import { act } from 'react'; import type { ListMachineTypesResponse, MachineType, + MachineView, WorkspaceMemberView, } from '@blitzos/schema'; import { describe, expect, it, vi } from 'vitest'; @@ -57,22 +58,30 @@ const ada: WorkspaceMemberView = { machine: null, }; +// Named so the box-image tests below can vary one field of it without +// spreading `me.machine`, which is nullable on the member row. +const myMachine: MachineView = { + id: 'machine-mo', + state: 'running', + machineTypeId: 'cx23@fsn1', + volumeId: 'volume-one', + volumeUsedPercent: 62, + boxImage: 'blitz-box:2026-08-31', + boxImageTarget: 'blitz-box:2026-08-31', + boxUpdateRequested: false, + boxUpdateOutcome: null, + membershipId: 'membership-2', + error: null, + createdAt: 1_700_000_000_000, + updatedAt: 1_700_000_000_000, +}; + const me: WorkspaceMemberView = { membershipId: 'membership-2', name: 'Mo Member', avatarUrl: null, role: 'member', - machine: { - id: 'machine-mo', - state: 'running', - machineTypeId: 'cx23@fsn1', - volumeId: 'volume-one', - volumeUsedPercent: 62, - membershipId: 'membership-2', - error: null, - createdAt: 1_700_000_000_000, - updatedAt: 1_700_000_000_000, - }, + machine: myMachine, }; const workspace = workspaceModelFixture({ @@ -93,16 +102,33 @@ function dialog(overrides: Partial[0]> = {}) workspace={workspace} membershipId="membership-2" listMachineTypes={async () => ({ machineTypes, failures: [] })} + refreshWorkspaces={() => undefined} onClose={() => undefined} {...overrides} /> ); } +/** The one settings section whose title matches, so a helper that asks for + * the lifecycle verbs cannot pick up the Box image section's button too. */ +function section(container: HTMLElement, title: string): HTMLElement { + const found = [...container.querySelectorAll('.cfg-section')] + .find((node) => node.querySelector('.cfg-title')?.textContent === title); + if (found === undefined) throw new Error(`no settings section titled ${title}`); + return found; +} + function buttons(container: HTMLElement): HTMLButtonElement[] { - // The lifecycle verbs sit in the settings-surface actions row - // (src/settings-surface.css); it is the only one in this dialog. - return [...container.querySelectorAll('.cfg-actions button')]; + // The lifecycle verbs sit in that section's settings-surface actions row + // (src/settings-surface.css). + return [...section(container, 'Lifecycle').querySelectorAll('.cfg-actions button')]; +} + +/** The Box image section's single button. */ +function updateButton(container: HTMLElement): HTMLButtonElement { + const found = section(container, 'Box image').querySelector('.cfg-actions button'); + if (found === null) throw new Error('no update button'); + return found; } describe('MyMachineDialog', () => { @@ -287,6 +313,98 @@ describe('MyMachineDialog', () => { await view.unmount(); }); + // The deliberate, user-triggered update. It restarts the machine, so it is + // confirmed in plain language first and shows as pending afterwards. + it('confirms an update in plain language, then asks for it', async () => { + const requestMachineBoxUpdate = vi.fn().mockResolvedValue({ machine: me.machine }); + const refreshWorkspaces = vi.fn(); + const behind = { ...me, machine: { ...myMachine, boxImage: 'blitz-box:2026-08-01' } }; + const view = await render(dialog({ + client: client({ requestMachineBoxUpdate }), + refreshWorkspaces, + workspace: { ...workspace, members: [ada, behind] }, + })); + await settle(); + + expect(view.container.textContent).toContain('Update available'); + // Both images are shown, so "available" is checkable rather than asserted. + expect(view.container.textContent).toContain('blitz-box:2026-08-01'); + expect(view.container.textContent).toContain('blitz-box:2026-08-31'); + const button = updateButton(view.container); + expect(button.disabled).toBe(false); + + await act(async () => button.click()); + await settle(); + // Nothing has been asked for yet: the click opens the confirmation. + expect(requestMachineBoxUpdate).not.toHaveBeenCalled(); + const dialogText = document.body.textContent ?? ''; + expect(dialogText).toContain('This restarts your machine'); + expect(dialogText).toContain('Running terminals and agents stop'); + + const confirm = [...document.querySelectorAll('button')] + .find((node) => node.textContent === 'Yes, update it'); + await act(async () => confirm?.click()); + await settle(); + + expect(requestMachineBoxUpdate).toHaveBeenCalledWith('machine-mo'); + // The poll runs now, so the pending state appears without a 15s wait. + expect(refreshWorkspaces).toHaveBeenCalled(); + await view.unmount(); + }); + + it('shows a requested update as pending and offers no second click', async () => { + const pending = { ...me, machine: { ...myMachine, boxUpdateRequested: true } }; + const view = await render(dialog({ + workspace: { ...workspace, members: [ada, pending] }, + })); + await settle(); + + expect(view.container.textContent).toContain('Update requested'); + expect(updateButton(view.container).disabled).toBe(true); + await view.unmount(); + }); + + // Every box created before the manifest updater shipped reports this, which + // is every pre-existing canary box. The dialog must not pretend otherwise. + it('tells a machine whose host cannot self-update to recreate instead', async () => { + const legacy = { + ...me, + machine: { + ...myMachine, + boxImage: 'blitz-box:2026-08-01', + boxUpdateOutcome: 'unsupported' as const, + }, + }; + const view = await render(dialog({ + workspace: { ...workspace, members: [ada, legacy] }, + })); + await settle(); + + expect(view.container.textContent).toContain('cannot update in place'); + expect(view.container.textContent).toContain('Recreate it'); + expect(updateButton(view.container).disabled).toBe(true); + await view.unmount(); + }); + + it('says a failed attempt left the machine untouched, and still offers a retry', async () => { + const failed = { + ...me, + machine: { + ...myMachine, + boxImage: 'blitz-box:2026-08-01', + boxUpdateOutcome: 'digest-mismatch' as const, + }, + }; + const view = await render(dialog({ + workspace: { ...workspace, members: [ada, failed] }, + })); + await settle(); + + expect(view.container.textContent).toContain('left untouched'); + expect(updateButton(view.container).disabled).toBe(false); + await view.unmount(); + }); + it('tells a viewer they hold no machine', async () => { const view = await render(dialog({ workspace: { @@ -298,7 +416,10 @@ describe('MyMachineDialog', () => { await settle(); expect(view.container.textContent).toContain('A viewer holds no machine'); - expect(buttons(view.container)).toHaveLength(0); + // No settings sections at all, so no verb of any kind — lifecycle or + // box image — is on offer. + expect(view.container.querySelectorAll('.cfg-section')).toHaveLength(0); + expect(view.container.querySelectorAll('.cfg-actions button')).toHaveLength(0); await view.unmount(); }); }); diff --git a/packages/webapp/test/recipes.test.tsx b/packages/webapp/test/recipes.test.tsx index 03e9fae3..eb2db98c 100644 --- a/packages/webapp/test/recipes.test.tsx +++ b/packages/webapp/test/recipes.test.tsx @@ -424,6 +424,7 @@ function client(overrides: Partial = {}): ControlPlaneClient stopMachine: vi.fn(async () => { throw new Error('unused'); }), startMachine: vi.fn(async () => { throw new Error('unused'); }), recreateMachine: vi.fn(async () => { throw new Error('unused'); }), + requestMachineBoxUpdate: vi.fn(async () => { throw new Error('unused'); }), setMachineType: vi.fn(async () => { throw new Error('unused'); }), destroyMachine: vi.fn(async () => { throw new Error('unused'); }), putWorkspaceCredential: vi.fn(async () => undefined), diff --git a/packages/webapp/test/shell-smoke.test.tsx b/packages/webapp/test/shell-smoke.test.tsx index 3ebb8c2f..6633445d 100644 --- a/packages/webapp/test/shell-smoke.test.tsx +++ b/packages/webapp/test/shell-smoke.test.tsx @@ -207,6 +207,7 @@ function client(): ControlPlaneClient { stopMachine: vi.fn(async () => { throw new Error("unused"); }), startMachine: vi.fn(async () => { throw new Error("unused"); }), recreateMachine: vi.fn(async () => { throw new Error("unused"); }), + requestMachineBoxUpdate: vi.fn(async () => { throw new Error("unused"); }), setMachineType: vi.fn(async () => { throw new Error("unused"); }), destroyMachine: vi.fn(async () => { throw new Error("unused"); }), putWorkspaceCredential: vi.fn(async () => undefined), diff --git a/plans/MEMBER-MACHINES.md b/plans/MEMBER-MACHINES.md index 58551c48..73e9d3b9 100644 --- a/plans/MEMBER-MACHINES.md +++ b/plans/MEMBER-MACHINES.md @@ -360,6 +360,112 @@ Injection-at-use (values out of transcripts) rides the existing seams refuses a type whose location differs from the volume's. - Per-session credential audit dimension: lands with sessions (Build 2+). +## 5a. Updating a machine's box image + +A machine runs the box image its VM was created with, and it keeps running it +forever. Nothing upgrades a box on its own, and nothing should: replacing the +container kills every process inside it, so the trigger is a person clicking a +button in their own machine's dialog. + +**The flag and its three routes.** `machines.box_update_requested` is the +request. `GET /workspaces/self/box-config` is the host's five-minute poll, +`POST /workspaces/self/box-update-result` is its report, and either one +clears the flag on the way through — a failed attempt was still the answer to +this request, and a flag that re-arms itself would retry a kill-everything +operation nobody asked for twice. The shapes are the `box config v1` +cross-runtime contract; edit them with `packages/schema/fixtures/box-config/` +and both conformance suites, never one side alone. + +Three ways to set it, and the difference between them is deliberate: + +- `POST /machines/:machineId/box-update` — the My machine dialog. Names ONE + machine whoever calls it, and takes the machine-verb shape and gate of the + lifecycle verbs beside it (`core/machine-access.ts`, scope `own`). +- `POST /workspaces/:id/box-update` — the workspace-wide fan-out. An admin + asking here means every machine in the workspace, which is the point of + asking at the workspace level. A user who clicked a button in their own + dialog did not ask to restart a colleague's work, which is why the two are + separate routes rather than one route with a role branch. +- `POST /workspaces/self/box-update` — the guest verb, `blitz box update`. + +**Two install modes, one host script.** Client prod pins a GHCR ref, which is +the image, and `docker pull` installs it. Canary pins an https R2 +`manifest.json`, which NAMES the image inside itself, so the host has to fetch +and validate the manifest before it even learns which tag it was asked for. +Both modes converge on `/usr/local/sbin/blitz-box-image`, written once by the +bootstrap and invoked by the first boot and by the updater. Its exit codes are +the interface (10 download, 11 digest, 12 load, 13 bad manifest), and +`blitz-box-update` turns them into the contract's outcomes. + +The pull-first invariant holds on both paths: the image is acquired and +verified before the running container is touched, so every acquire failure +(`pull-failed`, `download-failed`, `digest-mismatch`, `load-failed`) leaves the +workspace exactly as it was. A new image that will not start rolls back. + +**Two digests, and why both.** The manifest declares the archive's +`totalSha256`, but that is self-certifying: whoever serves the manifest serves +the digest beside it, so on its own it proves only that the parts were +reassembled correctly. `box-config` therefore carries `boxImageSha256`, the +deployment's own pin, which reaches the host over a different connection from +a different origin. The updater passes it to the installer as the same +optional fourth argument the first boot uses, so an archive that is internally +consistent and still not the image this deployment pinned is refused. That is +what makes the updater's verification as strong as the bootstrap's, which has +always had its digest baked in. + +**Answering "is an update available".** Under a manifest pin the ref is +byte-identical across rebakes while the tag inside it moves, so comparing refs +would read every box as current forever. The comparison is on the CONCRETE +image on both sides: `machines.box_image_tag_reported` against +`deploymentBoxImage(runtime.vars)`, surfaced as `MachineView.boxImage` and +`.boxImageTarget`. The host reports its concrete image as `tag` on every +update result — the image it runs once the attempt settled, which is the OLD +one whenever the attempt changed nothing. + +A machine that has never been asked to update would otherwise have no reported +image at all, so `armPhoneHome` records the image at every VM provision. That +is the one moment the answer is known exactly: the same `runtime.vars` renders +the user-data that boot installs. + +**Old boxes cannot self-update, and the UI says so.** The emitted updater is +baked into a VM's user-data at create time. A machine created before this +change keeps its old script forever, and that script refuses every https ref — +so every pre-existing canary box, `blitzos-dev` included, answers `unsupported` +to the first request it is given. The dialog renders that verdict honestly +("This machine's host cannot update in place. Recreate it to move to the new +image.") and disables the button, rather than offering one that cannot work. +Only machines created after this deploy get the new updater. + +**The updater holds its own credential.** A box access token lives 15 minutes +(`ACCESS_LIFETIME_MS`) and the timer runs every 5, but nothing on the VM keeps +`/var/lib/blitz/box-credential.json` fresh on its own: the Go client inside the +container rotates only in reaction to its own 401, which needs somebody to run +`blitz-cred`. On a quiet box the on-disk token expires and every later poll +401s for good — measured live on `blitzos-dev`, file mtime 02:03 and 401s from +02:20 onward, while `blitz-cred` and the gateway kept working. So the updater +spends the refresh token itself against `POST /oauth/token`, under the same +flock (`box-credential.lock`) the Go client takes, and writes the rotation +back. It has to be able to do this while the container it is about to replace +is broken, which is exactly when nothing inside the box can help — and writing +the file keeps it fresh for every other reader on the machine as a side effect. + +**The size budget, and the way out of it.** All of this rides in cloud-init +user-data, which Hetzner caps at a hard 32 KiB +(`HETZNER_USER_DATA_MAX_BYTES`; AWS gzips and has room to spare). A heavy +manifest-mode create was 25.4 KiB before this work and is 30.4 KiB after. +`test/bootstrap.test.ts` pins a 2 KiB floor so the next feature that wants to +emit bash finds out there rather than as a 413 on a real create. + +That floor is thin, and the way to buy the headroom back is to stop shipping +the host scripts in user-data at all: `blitz-box-run`, `blitz-box-update` and +most of `blitz-box-image` could be extracted from the box image the way +`blitz-box-run` already extracts `/etc/blitz/env.defaults`, leaving only enough +in user-data to fetch the image. That is a box-image change and a rebake, so it +is the next step and not this one. + +**Not the microVM path.** `packages/microvm-host/` has its own guest lifecycle +and no update path. The routes and the flag exist for the cloud-VM providers. + ## 6. The workspace details page (revamp target) The dialog at `/workspaces/:id` ("Workspace details", annotated for revamp)