fix(bake): the builder script has been truncated since it was written - #101
Open
pythonlearner1025 wants to merge 1 commit into
Open
fix(bake): the builder script has been truncated since it was written#101pythonlearner1025 wants to merge 1 commit into
pythonlearner1025 wants to merge 1 commit into
Conversation
No golden snapshot has ever been a complete build. Three bugs, stacked so
that each hid the next.
1. A backtick ends a template literal, including inside a shell comment.
# `|| true` throughout: an absent unit is not a bake failure.
That line closed builderUserData's template 49 lines early. What follows
parses as `template || true`tagged-template``, which is valid JavaScript
and short-circuits to the first string -- so `node --check` passes, the
function returns, nothing throws, and the builder is handed a script a
third shorter than the file looks. Lost with it: lever 2, the
root-password fix, the identity strip, the marker, and `shutdown -h now`.
2. `/usr/sbin/sshd -t` fails with "Missing privilege separation directory:
/run/sshd". /run is a tmpfs that only ssh.service populates, and socket
activation has not started it yet at first boot. `set -Eeuo pipefail` and
the ERR trap then power the builder off at line 49.
This masked bug 1 completely. The truncated script had no shutdown of its
own, so the trap's `shutdown -h now` was the only reason a bake ever
finished at all -- and the bake reads "powered off" as "finished".
3. `chage` was never passed -d, so sp_lstchg stays at Hetzner's 0. That does
not mean "old", it means "must be changed at next login", and PAM refuses
a key login with "Your password has expired" however open the other aging
fields are.
Confirmed by booting both snapshots and reading /var/log/blitz-bake.log,
which survives only because the cleanup never ran:
425047509 -- live on canary and client prod -- FAILED at line 49
425198277 -- baked today -- FAILED at line 49
Fixing 2 alone turned the symptom into "builder never powered off" after the
full 30-minute timeout, which is bug 1 with nothing left to hide behind.
Fixes: escape the backticks; create /run/sshd before the test; pass
`chage -d "$(date +%F)"`. Verified for 3 by booting 425047509 and running
only that command -- a refused login becomes a working one.
assertWholeScript then refuses to spend a VM on a script missing any of its
five landmarks or not ending in `shutdown -h now`, because the failure mode
of bug 1 is silence.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J6fUBY1B27EzvDwbhfBf52
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No golden snapshot has ever been a complete build — including
425047509, which canary and client prod boot today. Three bugs, stacked so each hid the next.1. A backtick ends a template literal, even inside a shell comment
That line closed
builderUserData's template 49 lines early. What follows parses astemplate || true`tagged-template`— valid JavaScript that short-circuits to the first string. Sonode --checkpasses, the function returns, nothing throws, and the builder receives a script a third shorter than the file looks.Lost with it: lever 2, the root-password fix, the identity strip, the marker, and
shutdown -h now.2.
sshd -thas no privilege separation directory/runis a tmpfs that onlyssh.servicepopulates, and socket activation has not started it at first boot.set -Eeuo pipefailplus the ERR trap power the builder off there.This masked bug 1 completely. The truncated script had no
shutdownof its own, so the ERR trap'sshutdown -h nowwas the only reason a bake ever finished — and the bake reads "powered off" as "finished".3.
chagewas never passed-dsp_lstchgstays at Hetzner's0. That does not mean "old", it means must be changed at next login, so PAM refuses a key login with "Your password has expired" however open-I -m -M -Eleave the other fields.Evidence
Booted both snapshots and read
/var/log/blitz-bake.log, which survives only because the cleanup never ran:425047509— live on canary and client prodFAILED at line 49425198277— baked today, pre-fixFAILED at line 49Fixing bug 2 alone turned the symptom into
builder never powered offafter the full 30-minute timeout — bug 1 with nothing left to hide behind.Bug 3 was verified in isolation: booting
425047509and running onlychage -d "$(date +%F)" rootturns a refused login into a working one.What was actually shipping
Every workspace on
425047509carries the builder's/etc/machine-id, because the strip never ran. The bake's own comment says why that matters. Lever 2 never applied either, so those boots pay forsnapd,unattended-upgradesand the apt timers.Verification — a real bake, end to end
425198627, probed independently after the bake's own probe passed:npm run lint:gatepasses, baseline unchanged.The guard
assertWholeScriptrefuses to spend a VM on a script missing any of its five landmarks, or not ending inshutdown -h now. Bug 1's failure mode is silence, and #98's probe only catches it after a builder and a snapshot have been paid for. This catches it before.Pinning
425198627is#102.🤖 Generated with Claude Code
https://claude.ai/code/session_01J6fUBY1B27EzvDwbhfBf52