add-to-group: --quarantine-on-register, because the bootstrap doesn't wait

[rcurranmoz]

provision --quarantine-on-register cannot hold back a fresh host, and wave 1 proved it.

Adding a host to the bootstrap group triggers a fully autonomous bootstrap — pkg, puppet, autologin, Safari, sentinel, worker. All four wave-1 hosts finished, registered, and began claiming autoland tasks before anyone ran provision. The flag can only quarantine a registration it is actively watching for; it cannot retroactively hold a worker that is already taking work.

macmini-m4-242 failed 15 production tasks in that window — mochitest, jsreftest, web-platform-tests — with no live_backing.log on any of them. The cycle was: claim → reboot ~90s later → fail with no output → repeat.

So the watch has to start where the bootstrap is triggered:

reprovision add-to-group <host> --quarantine-on-register
reprovision batch <file> --action add-to-group --quarantine-on-register

Details that matter

The watch budget must span the entire bootstrap. step_quarantine_on_register's default is 900s, sized for the provision path where the worker is ~a minute away. Started from here it also covers pkg install, puppet, several reboots and the sentinel — ~30 min measured on wave 1. A default-sized budget would expire before there was anything to quarantine, and the host would go live unheld: precisely the failure the flag exists to prevent. This caller passes bootstrap_max_wait + quarantine_on_register_max_wait.

The batch per-host timeout scales with it, 5:00 → 80:00. A child killed mid-watch leaves the host live and unheld.

without: · no OS/SIP gate … · per-host timeout 5:00
         · NOT quarantining: the bootstrap is autonomous, so these hosts will go live…
with:    · no OS/SIP gate … · per-host timeout 80:00
         · watching for registration and quarantining on sight (blocks for the whole bootstrap)

The watch also runs on the already-a-member path. That path returns early, but an already-member host can still be mid-bootstrap and about to register.

Without the flag, the banner now says so rather than staying silent.

Bonus fix

Membership does not prove the pkg was ever pushed — the already-member path skips push_apps, so a host added by hand in the UI can sit in the group with nothing installed (m4-241 was in exactly that state). It now warns when the payload is missing.

It deliberately does not push: push_apps hits every member of the group and would re-run the postinstall on hosts that are mid-bootstrap.

Testing

216 pass, ruff check . clean on the pinned 0.15.19. 6 new tests, including one asserting the budget exceeds the default (the bug that would silently reintroduce the whole failure) and one asserting the batch forwards the flag.

Ref: RELOPS-2515

🤖 Generated with Claude Code