Skip to content

fix(bake): the builder script has been truncated since it was written - #101

Open
pythonlearner1025 wants to merge 1 commit into
mainfrom
fix/bake-sshd-test-and-root-expiry
Open

fix(bake): the builder script has been truncated since it was written#101
pythonlearner1025 wants to merge 1 commit into
mainfrom
fix/bake-sshd-test-and-root-expiry

Conversation

@pythonlearner1025

Copy link
Copy Markdown
Member

No golden snapshot has ever been a complete build — including 425047509, which canary and client prod boot today. Three bugs, stacked so each hid the next.

1. A backtick ends a template literal, even inside a shell comment

# `|| true` throughout: an absent unit is not a bake failure.

That line closed builderUserData's template 49 lines early. What follows parses as template || true`tagged-template` — valid JavaScript that short-circuits to the first string. So node --check passes, the function returns, nothing throws, and the builder receives a script a third shorter than the file looks.

Lost with it: lever 2, the root-password fix, the identity strip, the marker, and shutdown -h now.

2. sshd -t has no privilege separation directory

Missing privilege separation directory: /run/sshd
bake: FAILED at line 49

/run is a tmpfs that only ssh.service populates, and socket activation has not started it at first boot. set -Eeuo pipefail plus the ERR trap power the builder off there.

This masked bug 1 completely. The truncated script had no shutdown of its own, so the ERR trap's shutdown -h now was the only reason a bake ever finished — and the bake reads "powered off" as "finished".

3. chage was never passed -d

sp_lstchg stays at Hetzner's 0. That does not mean "old", it means must be changed at next login, so PAM refuses a key login with "Your password has expired" however open -I -m -M -E leave the other fields.

Evidence

Booted both snapshots and read /var/log/blitz-bake.log, which survives only because the cleanup never ran:

Snapshot Result
425047509 — live on canary and client prod FAILED at line 49
425198277 — baked today, pre-fix FAILED at line 49

Fixing bug 2 alone turned the symptom into builder never powered off after the full 30-minute timeout — bug 1 with nothing left to hide behind.

Bug 3 was verified in isolation: booting 425047509 and running only chage -d "$(date +%F)" root turns a refused login into a working one.

What was actually shipping

Every workspace on 425047509 carries the builder's /etc/machine-id, because the strip never ran. The bake's own comment says why that matters. Lever 2 never applied either, so those boots pay for snapd, unattended-upgrades and the apt timers.

Verification — a real bake, end to end

425198627, probed independently after the bake's own probe passed:

root aging          Password expires: never    (key login works with no workaround)
marker              ghcr.io/blitzdotdev/blitz-box@sha256:b48f7a2a…
docker image        PRESENT
sshd                0.0.0.0:2222, owned by sshd — ssh.socket masked, ssh.service enabled
machine-id          dd05e9bd… — freshly generated, not the builder's
lever 2             snapd / unattended-upgrades / apt-daily all `disabled`
size                2.40 GB, x86, ~$0.05/month

npm run lint:gate passes, baseline unchanged.

The guard

assertWholeScript refuses to spend a VM on a script missing any of its five landmarks, or not ending in shutdown -h now. Bug 1's failure mode is silence, and #98's probe only catches it after a builder and a snapshot have been paid for. This catches it before.

Pinning 425198627 is #102.

🤖 Generated with Claude Code

https://claude.ai/code/session_01J6fUBY1B27EzvDwbhfBf52

No golden snapshot has ever been a complete build. Three bugs, stacked so
that each hid the next.

1. A backtick ends a template literal, including inside a shell comment.

       # `|| true` throughout: an absent unit is not a bake failure.

   That line closed builderUserData's template 49 lines early. What follows
   parses as `template || true`tagged-template``, which is valid JavaScript
   and short-circuits to the first string -- so `node --check` passes, the
   function returns, nothing throws, and the builder is handed a script a
   third shorter than the file looks. Lost with it: lever 2, the
   root-password fix, the identity strip, the marker, and `shutdown -h now`.

2. `/usr/sbin/sshd -t` fails with "Missing privilege separation directory:
   /run/sshd". /run is a tmpfs that only ssh.service populates, and socket
   activation has not started it yet at first boot. `set -Eeuo pipefail` and
   the ERR trap then power the builder off at line 49.

   This masked bug 1 completely. The truncated script had no shutdown of its
   own, so the trap's `shutdown -h now` was the only reason a bake ever
   finished at all -- and the bake reads "powered off" as "finished".

3. `chage` was never passed -d, so sp_lstchg stays at Hetzner's 0. That does
   not mean "old", it means "must be changed at next login", and PAM refuses
   a key login with "Your password has expired" however open the other aging
   fields are.

Confirmed by booting both snapshots and reading /var/log/blitz-bake.log,
which survives only because the cleanup never ran:

    425047509 -- live on canary and client prod -- FAILED at line 49
    425198277 -- baked today                    -- FAILED at line 49

Fixing 2 alone turned the symptom into "builder never powered off" after the
full 30-minute timeout, which is bug 1 with nothing left to hide behind.

Fixes: escape the backticks; create /run/sshd before the test; pass
`chage -d "$(date +%F)"`. Verified for 3 by booting 425047509 and running
only that command -- a refused login becomes a working one.

assertWholeScript then refuses to spend a VM on a script missing any of its
five landmarks or not ending in `shutdown -h now`, because the failure mode
of bug 1 is silence.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J6fUBY1B27EzvDwbhfBf52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant