diff --git a/docs/docs/index.md b/docs/docs/index.md index 42df0c7..6f4127a 100644 --- a/docs/docs/index.md +++ b/docs/docs/index.md @@ -46,6 +46,9 @@ and runbook lives here. - [Change the Tailscale policy](runbooks/tailscale-policy-change.md) covers changing, verifying, reverting, and emergency-editing the tailnet policy. +- [Rebuild nas01](runbooks/rebuild-nas01.md) reinstalls IncusOS on `nas01` + from git-defined install media and restores it as the cluster bootstrap + node. - [Deploy the VyOS gateway configuration](runbooks/vyos-gateway-deployment.md) covers validation, inspection, guarded deployment, verification, and console recovery for `gw01`. diff --git a/docs/docs/runbooks/rebuild-nas01.md b/docs/docs/runbooks/rebuild-nas01.md new file mode 100644 index 0000000..4bc39df --- /dev/null +++ b/docs/docs/runbooks/rebuild-nas01.md @@ -0,0 +1,128 @@ +--- +title: Rebuild nas01 +description: Reinstall IncusOS on nas01 from git-defined install media and restore it as the cluster bootstrap node. +--- + +# Rebuild nas01 + +Rebuild `nas01` from scratch: seeded IncusOS install media built from +`GilmanLab/fleet`, written to USB, installed onto the 128GB system drive, and +the machine restored as the Incus cluster's first member. This procedure was +proven live on 2026-08-20. + +## Preconditions and required access + +- `GilmanLab/fleet` checkout (`nodes/nas01/config.yaml` is the seed source). +- `incusos-builder` (build from source per the fleet README until a release + exists). +- A sacrificial USB stick (raw `dd` target) and brief physical access to + `nas01`. The PiKVM chain (`pikvm01` → `kvm01` input `nas01`) provides + remote console and keyboard; mass-storage emulation through `kvm01` does + NOT work (USB 1.1 enumeration) and ATX is not wired — media insertion and + power-cycling are physical. +- `lab-admin` AWS SSO for SOPS (recovery-key escrow in `GilmanLab/secrets`). +- The trusted Incus client keypair (`bootstrap-admin`); its public half is in + the seed, its private half escrowed at `fleet/shared/bootstrap-client` in + `GilmanLab/secrets`. + +## Safety impact + +- Destroys everything on `nas01`'s 128GB system drive (RS128GSSD510). The + seed's `install.target` (`bus: NVME`, `max_size: 200GiB`) cannot select the + 1TB data drives. +- The machine's previous disk-encryption recovery keys become invalid; new + ones are generated at first boot and MUST be re-escrowed (step 8). +- If `nas01` hosts cluster workloads, they are lost unless recovered by their + own means — this runbook covers the node, not workload data. + +## Procedure + +1. Build the media (any machine with the builder): + + ```sh + incusos-builder validate -f nodes/nas01/config.yaml + incusos-builder build -f nodes/nas01/config.yaml -o nas01.img + ``` + + Record the reported IncusOS version and sha256. Pin `image.release` in the + config if updating it. +2. Write the image raw to a USB stick (macOS shown; never write through + Ventoy — chain-loading alters the measured boot path): + + ```sh + diskutil list # identify the stick, carefully + diskutil unmountDisk /dev/diskN + sudo dd if=nas01.img of=/dev/rdiskN bs=16m + diskutil eject /dev/diskN + ``` + +3. If the previous install is being replaced because of TPM/unseal problems, + or the TPM's history is unknown: **clear the fTPM in BIOS first** + (Security → fTPM → clear; the machine reboots). A stale TPM state causes + the first boot to demand a recovery key it may not have. +4. If a previous IncusOS install exists on the system drive and this rebuild + is intentional, wipe it first (boot the Ventoy recovery stick, then + `wipefs -af` + `sgdisk --zap-all` the RS128 device) — otherwise the + installer refuses (`force_install` is deliberately not set in the seed). + Never leave install media written to an internal drive: it pollutes the + measured boot path (PCR4) and leaves a rogue boot entry. +5. Insert the stick, boot the machine (F7 for the boot menu if needed). The + install is unattended: it selects the 128GB drive, installs, and displays + "remove the install media to complete the installation". +6. Remove the stick. The machine reboots into IncusOS: the console (PiKVM) + shows `nas01.glab.lol`, recovery-key generation, and + `Network configuration: mgmt(10.10.10.14)`. +7. Trust and re-cluster from an operator machine (the seeded client + certificate is already trusted by the node): + + ```sh + # Verify the fingerprint against the console's + # "Application TLS certificate fingerprint" line before accepting. + incus remote add nas01 10.10.10.14 + incus config set nas01: cluster.https_address=10.10.10.14:8443 + incus cluster enable nas01: nas01 + # Enabling clustering regenerates the server certificate: + incus remote remove nas01 && incus remote add nas01 10.10.10.14 + ``` + +8. **Escrow the new recovery keys immediately** (before any workload or + reboot): + + ```sh + incus admin os system security show nas01: + ``` + + Write `config.encryption_recovery_keys[0]` and + `state.pool_recovery_keys.local` into + `GilmanLab/secrets` `fleet/nas01/incusos.sops.yaml` (SOPS; `fleet` scope) + and merge through the normal PR flow. + +## Verification + +- `incus cluster list nas01:` shows `nas01` as `database-leader`, `ONLINE`, + "Fully operational". +- `incus storage list nas01:` shows the `local` ZFS pool; `incus launch + images:alpine/3.22 nas01:smoke1`, exec into it, ping `10.10.10.1`, delete. +- Reboot survival: `incus admin os system reboot nas01:` (confirm), machine + returns within ~3 minutes, `security show` reports both volumes + `unlocked (TPM)` — no passphrase prompt on console. + +## Rollback / recovery + +- Boot prompts for a passphrase/recovery key: enter the escrowed + `encryption_recovery_key` from `GilmanLab/secrets`, then force-reset the + TPM bindings via the system security API. If no valid key exists (e.g. + first boot never completed), treat the install as failed: clear the fTPM + and reinstall from step 3. +- Installer refuses ("already installed"): expected safety behavior; see + step 4. +- Console access at any point: PiKVM at `https://10.10.70.20/` (input + `nas01` on the KVM), including a snapshot API and HID keyboard injection. + +## Escalation + +- TPM unseal failures that survive an fTPM clear + clean reinstall: check + the IncusOS forum/issues (fTPM quirks are hardware-specific) before + changing Secure Boot key state. Do not reset Secure Boot to Setup Mode + casually — the firmware's vendor certificates are involved in the boot + chain. diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index c59f590..c9048da 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -47,6 +47,7 @@ nav: - Tailscale policy: reference/networking/tailscale-policy.md - Runbooks: - Change the Tailscale policy: runbooks/tailscale-policy-change.md + - Rebuild nas01: runbooks/rebuild-nas01.md - Deploy the VyOS gateway configuration: runbooks/vyos-gateway-deployment.md plugins: