docs: a provisioning runbook written from what actually broke - #69
Merged
Conversation
The docs described a pipeline we no longer run. The fresh-host golden path still said "move the passing hosts into the bootstrap group" by hand, assumed SIP was disabled at the rack, and predated `add-to-group` and `validate` entirely -- so following it would have produced a manual group click, an unnecessary Recovery trip, and no fitness gate before release. New docs/RUNBOOK.md is the operator field guide: copy-paste sequences for both paths, then the failure modes, each written SYMPTOM-FIRST because that is the order you meet them in. Everything in it cost a real debugging session: - the 75Hz KVM trap -- 100% task failure in ~43s with the worker exiting code 0, every other signal green, and system_profiler unable to confirm or deny it. 31 of 33 hosts on 2026-08-14. Reprovisioning cannot fix it. - the two ways Safari automation wedges: MiniBuddy (the PER-USER first-login assistant, which Skip Setup Assistant does not suppress) stealing focus, and Safari running with zero windows so `window 1` fails forever. - hostname is not a SimpleMDM device key -- a fresh DEP arrival is named "Mac mini" and the API answers HTTP 200 with zero hits, silently. - `ok` != the work happened, per action, with how to actually verify each. - ADD-never-MOVE, and why the client ships no move verb at all. - zsh not word-splitting, and `host`/`nslookup` bypassing the VPN split-DNS -- both of which produced confidently wrong readings during the wave. Also documents the `-j` table, which is the single easiest way to break a wave: `os-update` ignores `-j` entirely (launch-and-return, so the hosts-file length is the real concurrency) and `add-to-group` needs `-j2` or SimpleMDM 429s. orchestrator/README.md: rewrote both golden paths, added `add-to-group` and `validate` to the command table, documented the new config knobs, and corrected the batch section with the per-action concurrency guidance and the verify-don't-trust-ok point. README.md: status now reflects wave scale (37 hosts in a day, SIP-on throughout), and the two known bugs found at that scale are written up under Open work rather than living only in someone's head -- the `-j` coupling in `add-to-group --quarantine-on-register`, and the AppleScript's window assumption. Cross-links use explicit HTML anchors: GitHub's emoji-to-anchor slugging is not reliably predictable, and a heading rename should not silently rot the links. Ref: RELOPS-2515 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The docs described a pipeline we no longer run. The fresh-host golden path still said "move the passing hosts into the bootstrap group" by hand, assumed SIP was disabled at the rack, and predated
add-to-groupandvalidateentirely. Following it would have produced a manual group click, an unnecessary Recovery trip, and no fitness gate before release.🧑🚀 New:
docs/RUNBOOK.mdThe operator field guide. Copy-paste sequences for both paths, then the failure modes — each written symptom-first, because that is the order you meet them in. Everything in it cost a real debugging session:
system_profilerover SSH can neither confirm nor deny it. 31 of 33 hosts. Reprovisioning cannot fix it.Skip Setup Assistant - All Screensdoes not suppress it. Steals focus from the UI scripting.window 1exists. Once a session is disturbed, every retry fails forever — andopen -a Safaridoesn't help.Mac mini. The API answers HTTP 200 with zero hits — silent and confident.ok≠ the work happenedhostbypasses split-DNSPlus the
-jtable, which is the single easiest way to break a wave:os-updateignores-jentirely — launch-and-return, so the hosts-file length is the real concurrencyadd-to-groupneeds-j2— 3 SimpleMDM calls per host, and the limiter is unforgivingUpdated
orchestrator/README.md— rewrote both golden paths, addedadd-to-groupandvalidateto the command table, documented the new config knobs (including why the quarantine watch needs 5400s not the 900s default), and corrected the batch section.README.md— status reflects wave scale (37 hosts in a day, SIP-on throughout), and the two known bugs found at that scale are written up under Open work rather than living in someone's head: the-jcoupling inadd-to-group --quarantine-on-register, and the AppleScript's window assumption.Note
Cross-links use explicit HTML anchors. GitHub's emoji-to-anchor slugging isn't reliably predictable, and a heading rename shouldn't silently rot the links.
Docs only — no code changed.
Ref: RELOPS-2515
🤖 Generated with Claude Code