diff --git a/README.md b/README.md index 5f9800f..32c67af 100644 --- a/README.md +++ b/README.md @@ -79,6 +79,24 @@ HA), see [Hardware bring-up](docs/hardware.md). > **Hardened by default, reachable by your key.** The installer image is the hardened profile: no known password, key-only SSH (the `keepadmin` account, your enrolled key), `debugAccess` off, Vaultwarden bound to localhost, signups default-deny. During bring-up SSH is reachable on the LAN (`keepNode.adminAccess.lanBringup`) because a fresh node has no mesh yet; once it joins the mesh you redeploy mesh-only. See [Deployment](docs/deployment.md) for the declarative multi-node + admin-access how-to. The legacy `keepnode-debug` profile (known root password, password SSH, open signups) still exists as an explicit opt-in for keyless evaluation, but is never the installed default. `frost-gate` is off, so Vaultwarden data sits on the plain disk with no TPM unlock yet. +### Console status display (optional) + +A hardened node boots to a login prompt where every account is password-locked, so an operator standing +in front of the box learns nothing about why the vault didn't unlock or why the mesh didn't form. The +status display paints a read-only screen on a virtual terminal instead: node label, vault and gate +state, service health, anti-lockout, and (opt-in) mesh facts. It is **off by default**; enable it with: + +```nix +keepNode.statusDisplay.enable = true; # optional: tty = 2, nodeLabel = "vault-rack-3" +``` + +It is a pane of glass, not a console. It accepts **no input** (the renderer runs unprivileged with +`StandardInput=null`, so it holds no descriptor on the keyboard), reaches no shell, and grants nothing +that physical presence didn't already grant. The mesh address is a separate opt-in +(`showMeshAddress`) because it is network topology shown to whoever is standing +there. Operator detail in [Hardware bring-up](docs/hardware.md#console-status-display-optional); +threat model in [Security](docs/SECURITY.md). + ## License [MIT](LICENSE) diff --git a/docs/SECURITY.md b/docs/SECURITY.md index 0b1ec78..f7db7a9 100644 --- a/docs/SECURITY.md +++ b/docs/SECURITY.md @@ -89,6 +89,37 @@ true of all full-disk encryption. The quorum protects the key at rest and gates unlock on a second holder; it does not protect a live, compromised, running system. Keep Node does not claim to. +### The physical console + +Physical console access grants nothing today: every account on a hardened node is password-locked, so +the login prompt on tty1 is not a way in. The opt-in **status display** +(`keepNode.statusDisplay.enable`, off by default) paints a read-only status screen on one virtual +terminal instead of that prompt, and it is built so that this stays true. + +The module is split in two so that the process holding a terminal has no privilege to lose. The +**collector** runs as root on a timer, probes the node, and publishes one world-readable JSON snapshot +under `/run`; it has no terminal. The **renderer** reads that file and paints it, and runs as +`keep-status` , a system user with `nologin` as its shell and `!` as its password hash, so no `su` or +PAM path will accept it. The single load-bearing setting is `StandardInput=null`: the renderer holds no +file descriptor on the keyboard (not `tty`, not `tty-force`), so there is no read path to escape from, +no line discipline to poke and no "press any key" affordance to find. It parses no input because it is +given none. Around that: `PrivateNetwork=true` with `RestrictAddressFamilies=AF_UNIX` (a screen painter +has no business on a network in either direction), an empty `CapabilityBoundingSet`, `NoNewPrivileges`, +`ProtectSystem=strict`, and `DevicePolicy=closed` with a `DeviceAllow` naming exactly the one configured +VT , so enabling the display on tty3 does not also hand the renderer tty1-tty12. + +A Wayland kiosk was evaluated and rejected for this reason: `foot` binds `ctrl+shift+n` to spawning +`$SHELL`, which would hand a shell to anyone standing at the box, and a compositor adds a seat and a +keyboard input consumer to an appliance that deliberately has neither. + +What the screen does change is what an attacker at the box can *read*. Vault state, service health and +the node label are visible to anyone in the room. The mesh address is therefore a +separate opt-in (`showMeshAddress`, default false): that is topology intelligence rather than a +health fact. Enable it only where the console itself is in a trusted location. + +This display does not touch the threshold model, adds no password path, opens no port, and introduces +no network-reachable daemon. It has been tested in NixOS VMs, not on hardware. + ### Duress and coercion The quorum protects against theft and remote compromise. It does not, on its own, protect diff --git a/docs/hardware.md b/docs/hardware.md index bab21ff..968be82 100644 --- a/docs/hardware.md +++ b/docs/hardware.md @@ -56,6 +56,102 @@ That's a hardened node on real metal: key-only SSH, no known password, signups d the Vaultwarden web vault before the mesh exists, tunnel it: `ssh -L 8222:localhost:8222 keepadmin@`, then open `http://localhost:8222`. +### Console status display (optional) + +Everything above assumes you can already reach the box over SSH. When you can't, the only thing on the +monitor is a login prompt for accounts that all have locked passwords. `keepNode.statusDisplay` replaces +that with a read-only status screen, which is most useful exactly here in Tier 0 , before the mesh +exists and before SSH is proven. + +```nix +keepNode.statusDisplay = { + enable = true; # off by default + tty = 1; # which VT it owns (1-12); 1 is what a plugged-in monitor shows on boot + nodeLabel = "vault-rack-3"; # defaults to config.networking.hostName + # showMeshAddress = true; # defaults false, see below +}; +``` + +It is two units: `keep-node-status-collect` (root, on a timer, writes one JSON snapshot to +`/run/keep-node-status/status.json`) and `keep-node-status-render` (unprivileged, reads that file and +paints the VT). Neither binary is on your `$PATH`; to debug the screen from an SSH session use +`systemctl status keep-node-status-render`, `journalctl -u keep-node-status-collect`, and +`cat /run/keep-node-status/status.json`. + +**The rows.** `VAULT` is the vault volume (`unlocked` / `locked` / `wrong-device`, the last meaning the +data dir is mounted from something that is not the gate's mapper). `GATE`, `VAULTWARDEN`, `MESH`, +`WISP` and `ANTI-LOCKOUT` are the systemd states of the corresponding units. `MESH LINK` appears only +when `keepNode.mesh.enable` is on, and reads the interface's `UP` flag (not its `state` word, which +sits at `UNKNOWN` on a healthy tun device). `MESH ADDR` appears only when you opt in. +`REPLICATION` appears only when `keepNode.vaultReplication.role` is set, and shows the role plus lag in +seconds when a lag number is available. + +**`n/a` vs `unknown` vs `failed`** are three different facts and the screen never conflates them: + +- `n/a` (`·`, or `[ -- ]` in ASCII) , the owning module is not enabled on this node. Normal. A node + without `frostGate` shows `VAULT n/a`, not `locked`; that is a node that was never gated, not a node + whose vault failed to open. +- `unknown` (`○`, `[ ?? ]`) , the probe did not return, or the value could not be determined. `MESH + LINK unknown` means the interface could not be queried, which is not the same claim as the link + being down. +- `failed` (`×`, `[FAIL]`) , the unit actually failed. An absent unit is never reported this way. + +Every state is triple-coded (glyph + word + colour), so the reading survives a monochrome panel, a +colourblind reader, a serial capture and a photograph. + +**STALE.** If the newest snapshot is older than `staleSeconds` (default 20, against a `refreshSeconds` +collection cadence of 5), a red full-width banner appears and **every collected value on the screen is +replaced by `??`** , not dimmed, not greyed, not left showing its last reading. A greyed-out `UNLOCKED` +is still read as "unlocked" by someone glancing across the room, and if the collector has stopped, the +screen no longer knows whether that is true. Only the last-known timestamp (in the footer) and the SSH +line survive, because those are static config rather than collected facts. The node label is a +collected value and blanks with everything else, so a stale screen does not tell you which box you are +looking at , the SSH line at the foot does, since it carries the hostname (when admin SSH is +configured). Seeing `??` everywhere means the collector is not producing snapshots; check +`journalctl -u keep-node-status-collect`. + +**Anti-lockout renders as a full-width alarm**, not just a row. `nixos/admin-access.nix:198` writes its +"NO authorized SSH key" warning directly to `/dev/console`. On a default install that is the same VT +the renderer owns, so a repaint would erase it within a second; with a serial console, or with the +display moved off whatever VT `/dev/console` maps to, the warning simply lands somewhere the operator +at the monitor never sees. It is re-raised here as a banner pinned above the fold, where the row-budget +fitter can never drop or scroll it. If you see it, the box has no admin key and remote access is +impossible; provision one before you walk away. + +The anti-lockout row carries **the age of its own verdict** (`active (checked 3d ago)`), and you should +read that age, not just the colour. `keep-node-admin-key-check` is a `Type=oneshot` + +`RemainAfterExit=true` unit with no timer: it runs once at boot and latches its result. A key deleted +*after* boot does not re-trigger it, so a green anti-lockout row on a long-uptime node means "no lockout +as of that many days ago", not "no lockout now". To re-verify on demand: + +```bash +systemctl restart keep-node-admin-key-check.service +``` + +The screen picks the new verdict, and the reset age, up within one collector cycle. + +**If the status screen is blank or dead, use another VT.** The display only ever takes over the VT named +by `keepNode.statusDisplay.tty`; every other VT keeps its getty. At the default `tty = 1`, press +**Alt+F2** (through Alt+F6) for a normal login prompt , if you moved the display, that VT is the one to +avoid and any other Alt+F works. This matters because the display's VT has *no* getty to fall back +to , `getty@` and `autovt@` for that terminal are disabled, which NixOS implements by masking them, so +`systemctl start getty@tty` for that VT will refuse. Switching to another VT is the recovery path at +the keyboard. (Those prompts grant nothing by themselves: every account is password-locked, so you +still need a key-based SSH session, or physical media, to do anything.) From there, +`systemctl status keep-node-status-render` and +`journalctl -u keep-node-status-render` say why the screen is not painting. + +**Conflict with `debugAccess`.** `keepNode.debugAccess` sets `services.getty.autologinUser = "root"`, +which claims tty1. Enabling both with `statusDisplay.tty = 1` is a build-time assertion failure, not a +runtime surprise. Remedy: either disable `keepNode.debugAccess` (the production posture), or move the +screen to a free VT with `keepNode.statusDisplay.tty = 2`. The status display takes over only its own +VT , every other VT keeps its normal getty, which grants nothing anyway since every account is +password-locked. + +Untested on real hardware: this is CI-tested in NixOS VMs like the rest of the tree, so treat the first +boot on metal (console font, VT geometry, whether your firmware hands you a usable tty1) as something +to confirm, not assume. + ## Phase 2 (Tier 0, cont.) , mesh onboarding (declarative) From here you leave the stock image and deploy **your own config** (a flake that imports keep-node's diff --git a/flake.nix b/flake.nix index 479f12b..ea52758 100644 --- a/flake.nix +++ b/flake.nix @@ -378,6 +378,112 @@ authorizedKeysFile = ""; }).success; + # Pure-eval guard for the statusDisplay cadence and console-ownership assertions. The screen is + # the only diagnostic an operator standing at the box gets, so a config that would make it lie + # (a permanently-lit STALE banner, a healthy-looking frozen frame, mesh fields stuck on "n/a") + # or that would silently steal the bring-up console must fail the BUILD, not surface at 3am on a + # rack. Forcing the system toplevel triggers assertions; tryEval turns a fired assertion into + # success=false. + statusDisplayToplevelEvals = + { + statusDisplay, + debugAccess ? false, + }: + builtins.tryEval + (nixpkgs.lib.nixosSystem { + inherit system; + modules = [ + ./nixos/keep-node.nix + ./nixos/debug-access.nix + { + fileSystems."/" = { + device = "/dev/disk/by-label/root"; + fsType = "ext4"; + }; + boot.loader.grub.enable = false; + keepNode.debugAccess.enable = debugAccess; + keepNode.statusDisplay = { + enable = true; + } + // statusDisplay; + } + ]; + }).config.system.build.toplevel.drvPath; + # Controls: a valid config must still evaluate, otherwise a broken fixture (or a base-config + # regression) is indistinguishable from the guard doing its job. + statusDisplayControlsEvaluate = + (statusDisplayToplevelEvals { statusDisplay = { }; }).success # all defaults + # The tty1 contention is tty1-SPECIFIC: debugAccess's autologin getty and a renderer on a + # different VT coexist fine, and that is the documented bring-up remedy. + && (statusDisplayToplevelEvals { + statusDisplay.tty = 2; + debugAccess = true; + }).success + # The allowed staleness boundary: exactly 3x the collector cadence, the recommended minimum. + && (statusDisplayToplevelEvals { + statusDisplay = { + staleSeconds = 15; + refreshSeconds = 5; + }; + }).success; + statusDisplayBadConfigsRejected = + # tty1 claimed by debugAccess's autologin getty: two owners, one VT. + !(statusDisplayToplevelEvals { + statusDisplay.tty = 1; + debugAccess = true; + }).success + # staleSeconds == refreshSeconds: every snapshot is stale on arrival, banner permanently lit. + && !(statusDisplayToplevelEvals { + statusDisplay = { + staleSeconds = 5; + refreshSeconds = 5; + }; + }).success + # repaintSeconds >= staleSeconds: a frozen frame keeps looking healthy after the collector dies. + && !(statusDisplayToplevelEvals { + statusDisplay = { + repaintSeconds = 30; + staleSeconds = 20; + }; + }).success + # The mesh address requested with no mesh: permanent "n/a" that reads as a fault. + && !(statusDisplayToplevelEvals { statusDisplay.showMeshAddress = true; }).success; + + # Pure-eval guard for the console brand mark. The status display and the installer MOTD frame a + # box sized from brand.markWidth, so a mark line that is not exactly that many COLUMNS wide + # leaves every border below it ragged. brand.displayWidth counts columns (not bytes) and returns + # null on any glyph outside its known alphabet, so an unmeasurable mark fails rather than + # silently reporting a byte count -- see nixos/lib/brand.nix for why byte length is not enough. + brand = import ./nixos/lib/brand.nix { inherit (nixpkgs) lib; }; + brandWidths = map brand.displayWidth brand.markUnicode; + brandAsciiWidths = map brand.displayWidth brand.markAscii; + brandUniform = + ws: ws != [ ] && !(builtins.elem null ws) && nixpkgs.lib.all (w: w == builtins.head ws) ws; + brandLineCountsOk = builtins.length brand.markUnicode == 3 && builtins.length brand.markAscii == 3; + brandUnicodeUniform = brandUniform brandWidths; + brandAsciiUniform = brandUniform brandAsciiWidths; + brandMarksAgree = brandUnicodeUniform && brandAsciiUniform && brandWidths == brandAsciiWidths; + brandWidthDeclared = brandUnicodeUniform && builtins.head brandWidths == brand.markWidth; + # The wordmark sits directly under the mark inside the same frame sized from markWidth, so it + # must be measurable and must not overhang it. Null-guarded: comparing null with an int throws. + brandWordmarkWidthOk = + let + ws = [ + (brand.displayWidth brand.wordmark) + (brand.displayWidth brand.wordmarkAscii) + ]; + in + !(builtins.elem null ws) && nixpkgs.lib.all (w: w <= brand.markWidth) ws; + # The fallback exists for serial/vt100/non-UTF-8 consoles; a stray multibyte byte defeats it. + brandAsciiPure = + nixpkgs.lib.all brand.isPrintableAscii brand.markAscii + && brand.isPrintableAscii brand.wordmarkAscii; + brandBlockSwitches = brand.block { } != brand.block { ascii = true; }; + brandBlockIndents = + brand.block { indent = 0; } != brand.block { indent = 4; } + && nixpkgs.lib.hasPrefix (builtins.head brand.markUnicode) (brand.block { indent = 0; }) + && nixpkgs.lib.hasPrefix " ${builtins.head brand.markUnicode}" (brand.block { indent = 4; }); + # Pure-eval guard for the mesh-interface single source of truth: setting keepNode.mesh.interface # once must propagate to every mesh-scoped service's meshInterface, so they cannot drift apart. # A refactor that reverts one service to a hardcoded default flips this red. @@ -472,6 +578,14 @@ enable = true; authorizedKeys = [ "ssh-ed25519 AAAAeval-only-fixture-key keepadmin-eval" ]; }; + # The console status screen is enabled here specifically because it is option-clash-prone in + # a way no VM test can catch: it writes systemd.services."getty@tty1".enable and + # "autovt@tty1".enable, defines users.users/groups.keep-status, and sets + # boot.consoleLogLevel. Any other module in this stack asserting ownership of the same + # attributes (a second getty definition, a conflicting consoleLogLevel set with mkForce) + # would collide only when all of them are composed into one system, which is exactly what + # this fixture is. + keepNode.statusDisplay.enable = true; } ]; }; @@ -654,6 +768,63 @@ else "echo 'adminAccess anti-lockout regression: the module either built with no usable key (empty or whitespace-only authorizedKeys and no authorizedKeysFile, a permanent key-only-SSH remote lockout) or refused a valid config (a real inline key, or the installer authorizedKeysFile escape)' >&2; exit 1" ); + statusdisplay-assertions = pkgs.runCommand "statusdisplay-assertions" { } ( + if !statusDisplayControlsEvaluate then + "echo 'statusdisplay-assertions control broke: a VALID statusDisplay config (defaults, or tty=2 alongside debugAccess, or staleSeconds=15 with refreshSeconds=5) unexpectedly failed to evaluate -- base config or nixpkgs regression, or the assertions now over-reject' >&2; exit 1" + else if + (statusDisplayToplevelEvals { + statusDisplay.tty = 1; + debugAccess = true; + }).success + then + "echo 'statusDisplay tty guard regression: tty=1 was accepted alongside keepNode.debugAccess.enable, whose services.getty.autologinUser also claims tty1 -- the renderer and the autologin getty would contend for one VT and the bring-up console is lost' >&2; exit 1" + else if + (statusDisplayToplevelEvals { + statusDisplay = { + staleSeconds = 5; + refreshSeconds = 5; + }; + }).success + then + "echo 'statusDisplay staleness guard regression: staleSeconds was accepted at or below refreshSeconds -- every freshly-collected snapshot is already stale, so the STALE banner is permanently lit and the operator learns to ignore the one banner that matters' >&2; exit 1" + else if + (statusDisplayToplevelEvals { + statusDisplay = { + repaintSeconds = 30; + staleSeconds = 20; + }; + }).success + then + "echo 'statusDisplay repaint guard regression: repaintSeconds was accepted at or above staleSeconds -- the renderer draws the STALE banner, so a healthy-looking frozen screen would persist for a full repaint period after the collector died' >&2; exit 1" + else if (statusDisplayToplevelEvals { statusDisplay.showMeshAddress = true; }).success then + "echo 'statusDisplay mesh-field guard regression: showMeshAddress was accepted with keepNode.mesh.enable false -- the mesh address row would render a permanent \"n/a\" that reads as a fault and sends the operator chasing a non-existent mesh problem' >&2; exit 1" + else if !statusDisplayBadConfigsRejected then + "echo 'statusDisplay guard regression: a config the fixtures expect to be rejected evaluated successfully, but no chained case above named it -- a fixture was added to statusDisplayBadConfigsRejected without its own diagnostic' >&2; exit 1" + else + "touch $out" + ); + brand-shapes = pkgs.runCommand "brand-shapes" { } ( + if !brandLineCountsOk then + "echo 'brand mark shape regression: markUnicode and markAscii must each be exactly 3 lines' >&2; exit 1" + else if !brandUnicodeUniform then + "echo 'brand mark shape regression: markUnicode lines are not all the same COLUMN width (or a line contains a glyph outside the measurable alphabet, so displayWidth returned null) -- every console box border drawn under the mark would be ragged' >&2; exit 1" + else if !brandAsciiUniform then + "echo 'brand mark shape regression: markAscii lines are not all the same COLUMN width (or a line is unmeasurable)' >&2; exit 1" + else if !brandMarksAgree then + "echo 'brand mark shape regression: markAscii is not the same width as markUnicode -- a renderer swapping the fallback in on a serial/non-UTF-8 console would resize the frame' >&2; exit 1" + else if !brandWidthDeclared then + "echo 'brand mark shape regression: brand.markWidth disagrees with the measured column width of markUnicode -- consumers size their frame from markWidth, so the declared number must track the art' >&2; exit 1" + else if !brandWordmarkWidthOk then + "echo 'brand mark shape regression: wordmark or wordmarkAscii is unmeasurable or wider than markWidth columns -- it would overhang the frame drawn around the mark' >&2; exit 1" + else if !brandAsciiPure then + "echo 'brand mark shape regression: markAscii or wordmarkAscii contains a byte outside printable ASCII (0x20-0x7E) -- the fallback exists precisely for consoles that cannot render multibyte glyphs' >&2; exit 1" + else if !brandBlockSwitches then + "echo 'brand mark shape regression: block { ascii = true; } returned the same string as block { } -- the ascii argument is being ignored' >&2; exit 1" + else if !brandBlockIndents then + "echo 'brand mark shape regression: block does not honour its indent argument (lines are not prefixed with exactly that many spaces)' >&2; exit 1" + else + "touch $out" + ); mesh-interface-consolidation = pkgs.runCommand "mesh-interface-consolidation" { } ( if meshInterfaceInheritance then "touch $out" @@ -689,6 +860,13 @@ imports = [ ./tests/adminaccess-bringup.nix ]; _module.args = { inherit adminKeyFixture; }; }; + status-display = pkgs.testers.runNixOSTest { + imports = [ ./tests/status-display.nix ]; + _module.args = { + inherit vaultRsaKeyFixture; + nvpnPackage = nvpn; + }; + }; installer-guards = pkgs.testers.runNixOSTest { imports = [ ./tests/installer-guards.nix ]; _module.args = { diff --git a/nixos/installer.nix b/nixos/installer.nix index c197d74..ad3aa03 100644 --- a/nixos/installer.nix +++ b/nixos/installer.nix @@ -3,11 +3,17 @@ # ISO store via system.extraDependencies, so `install-keepnode` installs with no network and no # rebuild on the target machine. { + lib, pkgs, keepnodeToplevel, ... }: let + # Same mark the installed node's status screen paints, so the ISO console and the appliance console + # are recognisably the same product. indent = 0 because the instructions below are flush left and a + # mark floating two columns off them reads as a misalignment rather than as a frame. + brand = import ./lib/brand.nix { inherit lib; }; + install-keepnode = pkgs.writeShellScriptBin "install-keepnode" '' set -euo pipefail # Partitioning needs root; the installer logs in as an unprivileged user, so re-exec via sudo. @@ -118,10 +124,14 @@ in # Bake the appliance into the ISO so the install is fully offline. system.extraDependencies = [ keepnodeToplevel ]; + # Unicode, not ascii: this MOTD is only ever seen over SSH or on the ISO's own UTF-8 console, both of + # which render the block glyphs. The installed node's renderer has to decide that at runtime (serial + # consoles, non-UTF-8 locales); here it is known at build time. users.motd = '' - keep-node installer - =================== + ${brand.block { indent = 0; }} + installer + ========= 1) See your disks: lsblk 2) Install (ERASES disk): install-keepnode /dev/nvme0n1 --ssh-key "ssh-ed25519 AAAA... you@host" --ssh-key takes your PUBLIC key inline (paste it), or a file path if you first copy the key diff --git a/nixos/keep-node.nix b/nixos/keep-node.nix index cbc3d66..b00aa0c 100644 --- a/nixos/keep-node.nix +++ b/nixos/keep-node.nix @@ -15,6 +15,7 @@ ./vault-replication.nix ./mesh.nix ./admin-access.nix + ./status-display.nix ]; # Appliance defaults (overridable). Hostname is left to the host/VM/test layer so it does diff --git a/nixos/lib/brand.nix b/nixos/lib/brand.nix new file mode 100644 index 0000000..7f77496 --- /dev/null +++ b/nixos/lib/brand.nix @@ -0,0 +1,80 @@ +# The keep-node brand mark: a pure typographic asset, no config, no options, no pkgs. +# +# Everything the box prints to a console -- the boot status display, the installer MOTD -- draws a +# framed block underneath this mark. The frame is sized from `markWidth`, so if the mark's lines are +# not all EXACTLY that many terminal columns wide, every border below it is ragged and the whole +# display looks broken on the one screen an operator actually looks at during bring-up. That is the +# invariant this file exists to hold, and `checks.brand-shapes` in flake.nix enforces it. +# +# Measuring it is the hard part. Nix has no codepoint-aware length: `lib.stringLength "█▀█"` is 9 +# (bytes), and `lib.stringToCharacters` splits bytes too. So byte equality across lines is necessary +# but NOT sufficient -- a genuinely ragged unicode mark passes a byte check whenever the raggedness +# happens to be byte-balanced (one dropped 3-byte block plus three added spaces, say). `displayWidth` +# below therefore counts real columns: it folds each known wide glyph down to a single ASCII byte and +# then requires every remaining byte to be printable ASCII. An unrecognised multibyte glyph leaves a +# non-ASCII byte behind and yields `null`, which callers must treat as a failure -- the measurement +# refuses to guess rather than silently returning a byte count. +# +# `markAscii` is the fallback for serial consoles, vt100, and any terminal not in a UTF-8 locale, +# where the block glyphs render as mojibake or question marks. It must be the same line count and the +# same column width as the unicode mark so a renderer can swap one for the other without resizing. +{ lib }: +let + # Printable ASCII (0x20-0x7E) as single bytes. Used both to prove the fallback mark is transport + # safe and, in displayWidth, to prove nothing unmeasured is left in a string. + printableAscii = lib.stringToCharacters " !\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~"; + + # Every non-ASCII glyph this file is allowed to contain, each occupying exactly one column. + wideGlyphs = [ + "█" + "▀" + "▄" + "·" + ]; + + indentOf = n: lib.concatStrings (lib.replicate n " "); +in +rec { + isPrintableAscii = s: lib.replaceStrings printableAscii (map (_: "") printableAscii) s == ""; + + # Terminal columns, or null if the string holds a glyph this file cannot measure. + displayWidth = + s: + let + folded = lib.replaceStrings wideGlyphs (map (_: "#") wideGlyphs) s; + in + if isPrintableAscii folded then lib.stringLength folded else null; + + # Declared, not derived: flake.nix checks the mark against this number, so deriving it from the + # mark itself would make that check vacuous. + markWidth = 18; + + markUnicode = [ + "█▄▀ █▀▀ █▀▀ █▀█" + "██ █▀ █▀ █▀▀" + "█▀▄ █▄▄ █▄▄ █ " + ]; + + markAscii = [ + "| / |-- |-- |-\\" + "|< |- |- |-/" + "| \\ |__ |__ | " + ]; + + wordmark = "k e e p · n o d e"; + wordmarkAscii = "k e e p . n o d e"; + + block = + { + ascii ? false, + indent ? 2, + }: + let + pad = indentOf indent; + lines = (if ascii then markAscii else markUnicode) ++ [ + "" + (if ascii then wordmarkAscii else wordmark) + ]; + in + lib.concatStringsSep "\n" (map (l: if l == "" then "" else pad + l) lines); +} diff --git a/nixos/status-display.nix b/nixos/status-display.nix new file mode 100644 index 0000000..cf9047f --- /dev/null +++ b/nixos/status-display.nix @@ -0,0 +1,1463 @@ +# A read-only, Keep-branded status screen on the physical console. It paints what the node is doing , +# vault state, service health, and (opt-in) mesh facts , onto a virtual terminal, and accepts nothing +# back. +# +# The problem it solves: an installed node boots to a bare kernel log dump followed by a `keepnode +# login:` prompt where every account is password-locked. An operator standing in front of the box +# therefore learns NOTHING about why the vault did not unlock or why the mesh did not form. The only +# diagnostic channel is SSH over the mesh , which means you need working SSH to diagnose why SSH might +# not be working. That is the gap: the one moment the operator has physical presence is the one moment +# the appliance tells them the least. +# +# Scope fence: physical console access grants NOTHING today, and it must continue to grant nothing. +# This display accepts NO input, reaches NO shell, and confers NO privilege. It does not touch the +# FROST threshold model, adds no password path, opens no listening port, and introduces no network- +# reachable daemon. It is a one-way pane of glass: information flows out to the operator's eyes and +# nothing flows back in. Anything that would make the console actionable belongs in a different module +# with a different threat model. +# +# Backward compatibility: off by default. Enabling it is an explicit operator decision, and leaving it +# off changes nothing whatsoever for existing deployments. +# +# The module is two halves that must stay two halves. The COLLECTOR runs as root on a timer, probes the +# node, and publishes one JSON snapshot under /run. The RENDERER runs unprivileged, reads that snapshot, +# and paints it. Nothing but a file crosses between them, so the thing holding the terminal has no +# privilege to lose and the thing holding the privilege has no terminal to be attacked through. +# +# The single most important line in this file is `StandardInput = "null"` on the renderer. The renderer +# holds NO file descriptor on the keyboard: not `tty`, not `tty-force`. There is therefore no read path +# to escape from, no line discipline to poke, and no "press a key" affordance to find. That is what +# makes "physical console grants nothing" a structural property rather than a promise about how +# carefully the script parses input , it parses no input at all. +{ + config, + lib, + pkgs, + ... +}: +let + cfg = config.keepNode.statusDisplay; + + # Probe targets are read out of the config of OTHER modules, every one of which is optional and may + # not even be imported on a given node. `x.y or default` tolerates a missing attribute at any level + # of the path, so a minimal node that imports neither frost-gate nor vault-replication still + # evaluates -- and reports those subsystems as absent rather than as broken. + gateEnabled = config.keepNode.frostGate.enable or false; + gateDataDir = config.keepNode.frostGate.dataDir or "/var/lib/vaultwarden"; + gateMapper = config.keepNode.frostGate.mapperName or "keep-vault"; + meshEnabled = config.keepNode.mesh.enable or false; + meshInterface = config.keepNode.mesh.interface or ""; + replRole = config.keepNode.vaultReplication.role or null; + + statusDir = "/run/keep-node-status"; + statusFile = "${statusDir}/status.json"; + + brand = import ./lib/brand.nix { inherit lib; }; + + ttyPath = "/dev/tty${toString cfg.tty}"; + gettyUnit = "getty@tty${toString cfg.tty}.service"; + + # The account the renderer runs as. It exists only to be a uid that owns a terminal and can read one + # world-readable file, which is exactly the amount of authority the console is allowed to represent. + renderUser = "keep-status"; + + # Static config, so it survives the staleness blanking below: this line is not a collected fact, and + # it is precisely what an operator standing at a screen full of `??` needs in order to go fix it. + sshCommand = + if (config.keepNode.adminAccess.enable or false) then + "ssh keepadmin@${config.networking.hostName}" + else + "admin SSH not configured on this node"; + + # Appended only on the widest tier, and only when it FITS (see build_frame). Truncating a reassurance + # mid-word is worse than not showing it: the longest static command text plus the older, wordier note + # came to 79 columns against the 78 a two_col left field gets at w=80, so the note lost its closing + # paren and read as an unterminated aside. The command itself is the part that must survive at every + # width, so the note is both shorter than that budget and dropped whole rather than cut. + sshNote = " (key-only; no console login)"; + + shellList = xs: lib.concatMapStringsSep " " lib.escapeShellArg xs; + + collector = pkgs.writeShellScriptBin "keep-node-status-collect" '' + set -euo pipefail + + # Operator-supplied strings (paths, interface and mapper names, the node label) are bound to shell + # VARIABLES via escapeShellArg and only ever referenced as "$var". A variable's contents are never + # re-scanned for command substitution, so no config value can execute as root here -- the same + # discipline as nixos/admin-access.nix's key check. + probe_timeout=${toString cfg.probeTimeoutSeconds} + dir=${lib.escapeShellArg statusDir} + node_label=${lib.escapeShellArg cfg.nodeLabel} + vault_dir=${lib.escapeShellArg gateDataDir} + vault_mapper=${lib.escapeShellArg "/dev/mapper/${gateMapper}"} + mesh_iface=${lib.escapeShellArg (if meshEnabled then meshInterface else "")} + repl_role=${lib.escapeShellArg (if replRole == null then "" else replRole)} + + # EVERY output variable is pre-initialised. The invariant this module lives or dies by is that the + # write block at the bottom is reached on every run: a collector that exits early leaves the + # PREVIOUS snapshot in place, and a renderer then paints stale numbers with nothing on the glass + # admitting why. "unknown" is the honest default for a probe that never got to run. + vault_state=unknown + gate_state=unknown + svc_vaultwarden=unknown + svc_mesh=unknown + svc_wisp=unknown + anti_lockout=unknown + anti_lockout_at_json=null + repl_lag_check=unknown + mesh_up=false + mesh_addr="" + lag_json=null + + is_uint() { + case "''${1:-}" in + "" | *[!0-9]*) return 1 ;; + *) return 0 ;; + esac + } + + # Every probe runs under a wall-clock timeout: one wedged systemctl or findmnt must not stall the + # round, because a stalled round ages the snapshot and would eventually mark the WHOLE screen STALE + # when in truth a single subsystem is hung. + run() { + ${pkgs.coreutils}/bin/timeout "$probe_timeout" "$@" + } + + # The three-way unit state. Conflating these is the single most damaging thing this collector could + # do, because most probe targets belong to OPTIONAL modules and on a minimal node legitimately do + # not exist: + # LoadState=not-found (or masked) -> "n/a" the owning module is off; this is normal + # probe timed out / returned nothing -> "unknown" we could not tell + # otherwise -> the real ActiveState + # An absent unit must NEVER render as "failed": that sends an operator chasing a subsystem the node + # was never configured to run. + # + # ONE systemctl call, parsed as KEY=value rather than with --value, because `systemctl show` does + # not promise to echo multiple --property requests back in the order they were asked. Sets `load` + # and `active` for the two wrappers below, each of which owns only its final mapping -- the shared + # probe is identical for both and duplicating it invites the two readings to drift apart. + unit_load_active() { + local out + load="" + active="" + out="$(run ${pkgs.systemd}/bin/systemctl show --property=LoadState --property=ActiveState -- "$1" 2>/dev/null || true)" + load="$(printf '%s\n' "$out" | ${pkgs.gnused}/bin/sed -n 's/^LoadState=//p' || true)" + active="$(printf '%s\n' "$out" | ${pkgs.gnused}/bin/sed -n 's/^ActiveState=//p' || true)" + } + + unit_state() { + local load active + unit_load_active "$1" + case "$load" in + "") echo unknown ;; + not-found | masked) echo "n/a" ;; + *) + if [ -z "$active" ]; then echo unknown; else echo "$active"; fi + ;; + esac + } + + # Timer-driven Type=oneshot units WITHOUT RemainAfterExit report nothing on success: the unit falls + # back to "inactive" the moment the run ends, so ActiveState carries a fail signal but no pass + # signal. Read through unit_state above, "inactive" maps to warn and a perfectly healthy node shows + # a permanent amber row -- the cry-wolf failure the staleSeconds assertion exists to prevent, + # arriving by another door. nixos/vault-replication.nix:628 states the contract for such a unit: + # `systemctl is-failed keep-node-vault-lag-check` IS the monitoring signal. So: failed -> "failed", + # anything else -> "ok". The three-way discipline is unchanged -- an absent unit is still "n/a" and + # never "failed", and a probe that did not return is still "unknown". + check_unit_state() { + local load active + unit_load_active "$1" + case "$load" in + "") echo unknown ;; + not-found | masked) echo "n/a" ;; + *) + case "$active" in + "") echo unknown ;; + failed) echo failed ;; + *) echo ok ;; + esac + ;; + esac + } + + # WHEN the anti-lockout backstop last actually ran. keep-node-admin-key-check is Type=oneshot + + # RemainAfterExit with NO timer anywhere in the tree: it runs once at boot and LATCHES "active" + # forever. A key deleted after boot -- named at nixos/admin-access.nix:158 as precisely the + # scenario the backstop exists for -- never re-triggers it, so the row would paint a green + # ANTI-LOCKOUT ok on a node whose admin SSH is already gone. + # + # That is worse than ordinary staleness because this display's contract trains the operator to + # trust it: every other field blanks to "??" the moment it stops being current, so a non-"??" + # value reads as "known right now". A boot-latched verdict wearing live-data clothing is exactly + # the failure this feature exists to prevent, so the age of the verdict is carried alongside it + # and rendered ("active (checked 41d ago)") rather than left implied. + # + # --timestamp=unix yields "@"; a never-activated unit yields "@0" or an empty value, both + # of which fall through to "" and simply omit the age rather than claiming a bogus one. + unit_active_since() { + local ts + ts="$(run ${pkgs.systemd}/bin/systemctl show --timestamp=unix --property=ActiveEnterTimestamp --value -- "$1" 2>/dev/null || true)" + ts="''${ts#@}" + if is_uint "$ts" && [ "$ts" -gt 0 ]; then printf '%s' "$ts"; fi + } + + now="$(run ${pkgs.coreutils}/bin/date +%s 2>/dev/null || echo unknown)" + is_uint "$now" || now=0 + + # Type=oneshot + RemainAfterExit units (the anti-lockout check at nixos/admin-access.nix:170-171, + # the frost gate) LATCH "active" after a successful run and "failed" after a bad one -- that is the + # intended reading, not a liveness claim. The frost gate's "activating" means an unlock is genuinely + # in flight. keep-node-vault-lag-check does NOT latch and so must not be read this way. + gate_state="$(unit_state keep-node-frost-gate.service || echo unknown)" + svc_vaultwarden="$(unit_state vaultwarden.service || echo unknown)" + svc_mesh="$(unit_state keep-node-mesh.service || echo unknown)" + svc_wisp="$(unit_state wisp.service || echo unknown)" + anti_lockout="$(unit_state keep-node-admin-key-check.service || echo unknown)" + anti_lockout_at="$(unit_active_since keep-node-admin-key-check.service || true)" + if is_uint "$anti_lockout_at"; then + anti_lockout_at_json="$anti_lockout_at" + fi + repl_lag_check="$(check_unit_state keep-node-vault-lag-check.service || echo unknown)" + + ${ + if !gateEnabled then + '' + # No gate configured, so there is no encrypted volume whose state could be reported. "n/a", + # not "locked": a node that was never gated is not a node whose vault failed to open. + vault_state="n/a" + '' + else + '' + # Mirrors the fail-closed mount guard in nixos/frost-gate.nix:118-123. A dataDir backed by a + # source that is NOT our mapper is the plaintext-exposure case the gate itself refuses to + # proceed on, so it gets its own alarm state and is never folded into "locked" or "unlocked". + if src="$(run ${pkgs.util-linux}/bin/findmnt -no SOURCE -- "$vault_dir" 2>/dev/null)"; then + findmnt_rc=0 + else + findmnt_rc=$? + src="" + fi + if [ "$findmnt_rc" -gt 1 ]; then + # Exit 1 is findmnt's ordinary "no match" (not a mountpoint). Anything above it, including + # timeout's 124, means the probe itself failed and we genuinely do not know. + vault_state=unknown + elif [ -z "$src" ]; then + vault_state=locked + elif [ "$src" = "$vault_mapper" ]; then + vault_state=unlocked + else + vault_state=wrong-device + fi + '' + } + + ${lib.optionalString meshEnabled '' + # tun devices sit at "state UNKNOWN" even when carrying traffic, so operational state is read + # from the IFF_UP flag inside <...> rather than from the state word, which would read every + # healthy mesh interface as down. + if link="$(run ${pkgs.iproute2}/bin/ip -o link show dev "$mesh_iface" 2>/dev/null)"; then + flags="$(printf '%s' "$link" | ${pkgs.gnused}/bin/sed -n 's/.*<\([^>]*\)>.*/\1/p' || true)" + case ",$flags," in + *,UP,*) mesh_up=true ;; + *) mesh_up=false ;; + esac + fi + ''} + + ${lib.optionalString (meshEnabled && cfg.showMeshAddress) '' + # awk (not `head`) consumes the whole stream: under `set -o pipefail` a SIGPIPE'd producer would + # be reported as a failed probe on a perfectly healthy interface. + mesh_addr="$(run ${pkgs.iproute2}/bin/ip -o -4 addr show dev "$mesh_iface" 2>/dev/null | ${pkgs.gawk}/bin/awk 'NR==1{split($4,a,"/"); print a[1]}' || true)" + ''} + + ${lib.optionalString (replRole == "standby") '' + # keep-node-vault-lag-check logs exactly one line per run; its unit state is the pass/fail signal + # and this recovers the NUMBER behind it so the screen can show how far behind the standby is. + lag_line="$(run ${pkgs.systemd}/bin/journalctl -u keep-node-vault-lag-check.service -n 50 -o cat --no-pager 2>/dev/null | ${pkgs.gawk}/bin/awk '/^vault replication lag: /{ l = $0 } END { print l }' || true)" + lag="$(printf '%s' "$lag_line" | ${pkgs.gnused}/bin/sed -n 's/^vault replication lag: \([0-9][0-9]*\)s .*/\1/p' || true)" + if is_uint "$lag"; then + lag_json="$lag" + fi + ''} + + # Atomic publish: same-directory temp, mode set BEFORE any content exists, then rename. The + # world-readable window therefore never holds a half-written file, and a reader either sees the + # previous snapshot or the new one -- never a torn one. + tmp="$dir/.status.json.$$" + trap 'rm -f "$tmp"' EXIT + # The `|| : >` fallback keeps a broken/unavailable `install` from being the one thing that stops a + # snapshot from ever being published. Plain redirection creates the file at 0644 under this unit's + # UMask=0022, so the fallback lands on the same mode rather than a laxer one. + ${pkgs.coreutils}/bin/install -m 0644 /dev/null "$tmp" || : > "$tmp" + + # Schema 1 field vocabularies. Every unit field carries a raw systemd ActiveState ("active", + # "inactive", "failed", ...) plus "n/a" and "unknown" -- EXCEPT replication.lag_check, which carries + # a verdict: "ok" | "failed" | "n/a" | "unknown" (see check_unit_state above; its unit's ActiveState + # is not a pass/fail signal). The schema stays at 1 because nothing has shipped, so no consumer of + # an older vocabulary exists to keep working. + # + # Emitted through jq rather than concatenated by hand: a probe result containing a quote or a + # backslash cannot produce unparseable output, and the renderer's only input stays machine-readable + # no matter what the probes returned. --argjson values are all either a validated unsigned integer + # or the literal null/true/false, so they cannot inject structure either. + if ! ${pkgs.jq}/bin/jq -n \ + --argjson schema 1 \ + --argjson generated_at "$now" \ + --arg node "$node_label" \ + --arg vault_state "$vault_state" \ + --arg gate_state "$gate_state" \ + --arg svc_vaultwarden "$svc_vaultwarden" \ + --arg svc_mesh "$svc_mesh" \ + --arg svc_wisp "$svc_wisp" \ + --arg mesh_iface "$mesh_iface" \ + --argjson mesh_up "$mesh_up" \ + --arg mesh_addr "$mesh_addr" \ + --arg repl_role "$repl_role" \ + --arg repl_lag_check "$repl_lag_check" \ + --argjson repl_lag "$lag_json" \ + --arg anti_lockout "$anti_lockout" \ + --argjson anti_lockout_checked_at "$anti_lockout_at_json" \ + '{ + schema: $schema, + generated_at: $generated_at, + node: $node, + vault: { state: $vault_state, gate: $gate_state }, + services: { vaultwarden: $svc_vaultwarden, mesh: $svc_mesh, wisp: $svc_wisp }, + mesh: { + interface: (if $mesh_iface == "" then null else $mesh_iface end), + up: $mesh_up, + address: (if $mesh_addr == "" then null else $mesh_addr end) + }, + replication: { + role: (if $repl_role == "" then null else $repl_role end), + lag_check: $repl_lag_check, + lag_seconds: $repl_lag + }, + anti_lockout: $anti_lockout, + anti_lockout_checked_at: $anti_lockout_checked_at + }' > "$tmp"; then + # Last-ditch: even a broken jq must leave PARSEABLE json, so the renderer can BANNER a collector + # fault instead of falling over on a truncated file and showing nothing at all. `error` is a + # real degrade reason in the renderer (read_status), not decoration: generated_at here is + # CURRENT, so staleness would never fire and without that read the frame would look entirely + # normal -- hostname painted, every row "unknown", no banner at all. + printf '{"schema":1,"generated_at":%s,"node":null,"error":"collector-emit-failed"}\n' "$now" > "$tmp" + fi + + ${pkgs.coreutils}/bin/mv -f "$tmp" "$dir/status.json" + trap - EXIT + ''; + + # ONE jq call per repaint, emitting a FIXED, ORDERED, newline-separated value list that a fixed + # sequence of `read -r` consumes into named variables. No eval, no source, no dynamic variable names: + # the snapshot cannot introduce a name, only fill a slot that already exists. `s` additionally folds + # control characters to spaces so that no single value can ever span two lines and shift every + # subsequent field into the wrong variable. + jqProgram = '' + def s: if . == null then "" else tostring end | gsub("[[:cntrl:]]"; " "); + [ (.schema | s), + (.generated_at | s), + (.node | s), + (.vault.state | s), + (.vault.gate | s), + (.services.vaultwarden | s), + (.services.mesh | s), + (.services.wisp | s), + (.mesh.interface | s), + (.mesh.up | s), + (.mesh.address | s), + (.replication.role | s), + (.replication.lag_check | s), + (.replication.lag_seconds | s), + (.anti_lockout | s), + (.anti_lockout_checked_at | s), + (.error | s), + "." + ] | .[] + ''; + + renderer = pkgs.writeShellScriptBin "keep-node-status-render" '' + set -euo pipefail + + # Same discipline as the collector: every operator-supplied string is bound to a shell VARIABLE via + # escapeShellArg and only ever referenced as "$var", so no config value is ever re-scanned as code. + status_file=${lib.escapeShellArg statusFile} + stale_seconds=${toString cfg.staleSeconds} + repaint_seconds=${toString cfg.repaintSeconds} + node_label=${lib.escapeShellArg cfg.nodeLabel} + host_name=${lib.escapeShellArg config.networking.hostName} + ssh_command=${lib.escapeShellArg sshCommand} + ssh_note=${lib.escapeShellArg sshNote} + show_mesh_link=${if meshEnabled then "1" else "0"} + show_mesh_addr=${if cfg.showMeshAddress then "1" else "0"} + show_repl=${if replRole != null then "1" else "0"} + + mark_unicode=(${shellList brand.markUnicode}) + mark_ascii=(${shellList brand.markAscii}) + wordmark=${lib.escapeShellArg brand.wordmark} + wordmark_ascii=${lib.escapeShellArg brand.wordmarkAscii} + + # --once renders exactly ONE frame to stdout and exits 0. It is not a parallel implementation: it + # runs build_frame(), the same function the loop runs, and differs only in that it does not wrap the + # frame in cursor-positioning control codes (there is no terminal to position on). The VM test + # asserts real frame CONTENT through this path, so content and loop output cannot drift apart. + once=0 + while [ "$#" -gt 0 ]; do + case "$1" in + --once) + once=1 + shift + ;; + --status-file) + if [ "$#" -lt 2 ]; then + echo "keep-node-status-render: --status-file needs a path" >&2 + exit 2 + fi + status_file="$2" + shift 2 + ;; + *) + echo "usage: keep-node-status-render [--once] [--status-file PATH]" >&2 + exit 2 + ;; + esac + done + + is_uint() { + case "''${1:-}" in + "" | *[!0-9]*) return 1 ;; + *) return 0 ;; + esac + } + + # EVERY value taken from the snapshot passes through here before it reaches the screen. tr in the C + # locale keeps only 0x20-0x7E, so an ESC, a CSI, an OSC title-set, a CR or a stray newline are gone + # before any of them can reach the terminal's parser; the caller then hard-truncates to the field + # width so a long value cannot push the layout apart either. The collector is trusted and this is + # still done: it is the difference between "the screen is safe because the producer behaves" and + # "the screen is safe regardless of what the producer emits". + clean() { + printf '%s' "''${1:-}" | LC_ALL=C ${pkgs.coreutils}/bin/tr -cd '[:print:]' + } + + # Bound a snapshot-derived string to $1 characters. clean() sanitises CONTENT; this sanitises + # LENGTH, which the banner no longer does for itself since it word-wraps instead of hard-cutting. + # Callers that put a snapshot value into a banner must clip it, or one oversized field can grow the + # header past the whole screen. + clip() { + local n=$1 s=''${2:-} + if [ "''${#s}" -le "$n" ]; then + printf '%s' "$s" + else + printf '%s...' "''${s:0:$n}" + fi + } + + repeat_char() { + local n=$1 c=$2 out="" + while [ "''${#out}" -lt "$n" ]; do + out="$out$c" + done + printf '%s' "''${out:0:$n}" + } + + # Capability detection runs per repaint, not once at startup: a serial console can be attached to a + # box that has been up for months, and the frame after that must already be in the right alphabet. + detect_caps() { + ascii=0 + color=1 + case "''${TERM:-}" in + "" | dumb | unknown) + # No terminfo worth trusting: drop to the lowest common denominator on both axes. + ascii=1 + color=0 + ;; + vt100* | vt102* | vt220* | vt320*) + ascii=1 + ;; + esac + case "''${LC_ALL:-''${LC_CTYPE:-''${LANG:-}}}" in + *UTF-8* | *utf-8* | *UTF8* | *utf8*) : ;; + # Not a UTF-8 locale: the block-glyph mark would arrive as mojibake, so use brand.markAscii. + *) ascii=1 ;; + esac + if [ -n "''${NO_COLOR:-}" ]; then color=0; fi + if [ "''${KEEPNODE_STATUS_ASCII:-0}" != 0 ]; then ascii=1; fi + if [ "''${KEEPNODE_STATUS_NOCOLOR:-0}" != 0 ]; then color=0; fi + } + + detect_geometry() { + local size="" + term_cols="" + term_rows="" + # stty is asked first because it reads the real kernel window size, where `tput cols` may only + # know terminfo's static 80x24. It is pointed at fd 1, NOT at the renderer's own fd 0: fd 0 is + # /dev/null (StandardInput=null) and knows nothing about any terminal. The `0<&1` is a redirection + # scoped to this one command -- it duplicates the ALREADY-OPEN, write-only terminal descriptor + # onto fd 0 for the lifetime of the stty process alone. It opens nothing, grants no read access + # (a dup shares the original's O_WRONLY access mode, so a read would fail with EBADF), and leaves + # the renderer's own fd 0 still on /dev/null. The no-input property is untouched. + if size="$(${pkgs.coreutils}/bin/stty size 0<&1 2>/dev/null)"; then + term_rows="''${size%% *}" + term_cols="''${size##* }" + fi + if ! is_uint "$term_cols"; then + term_cols="$(${pkgs.ncurses}/bin/tput cols 2>/dev/null || true)" + fi + if ! is_uint "$term_rows"; then + term_rows="$(${pkgs.ncurses}/bin/tput lines 2>/dev/null || true)" + fi + if [ -n "''${KEEPNODE_STATUS_WIDTH:-}" ]; then term_cols="$KEEPNODE_STATUS_WIDTH"; fi + if [ -n "''${KEEPNODE_STATUS_ROWS:-}" ]; then term_rows="$KEEPNODE_STATUS_ROWS"; fi + is_uint "$term_cols" || term_cols=80 + is_uint "$term_rows" || term_rows=24 + + # Clamped to [40,100] and CENTRED. A 4K VT reports 240 columns and a status pane stretched across + # all of them reads as broken furniture rather than as an instrument: the eye cannot associate a + # label on the far left with a value on the far right. + if [ "$term_cols" -lt 40 ]; then + w=40 + elif [ "$term_cols" -gt 100 ]; then + w=100 + else + w="$term_cols" + fi + pad_n=$(( (term_cols - w) / 2 )) + if [ "$pad_n" -lt 0 ]; then pad_n=0; fi + pad="$(repeat_char "$pad_n" ' ')" + + # Three layout tiers. Narrow does not reflow into columns; it keeps ONE LINE PER ITEM and spends + # its remaining budget on the value, because a wrapped status line is a misread status line. + if [ "$w" -ge 80 ]; then + tier=full + labelw=16 + elif [ "$w" -ge 60 ]; then + tier=plain + labelw=14 + else + tier=narrow + # 12, not 10: 12 is the longest label this screen uses, so no tier ever cuts a label mid-word. + # A truncated label is a guessing game ("VAULTWARDE"), and the value is what the extra columns + # would have bought -- values are already truncated last and are the part that carries news. + labelw=12 + fi + } + + colorize() { + if [ "$color" -eq 1 ]; then + printf -v _c '\033[1;%sm%s\033[0m' "$1" "$2" + else + _c="$2" + fi + } + + # Triple-coded: GLYPH + WORD + COLOUR, so the reading survives colourblindness, a monochrome LCD, a + # serial capture and a photograph. Colour is never the only carrier of a state. + # + # `n/a` (a module that is switched off -- entirely normal) and `unknown`/`failed` (we could not tell + # / it broke) get DIFFERENT glyphs. Collapsing them is how an operator ends up chasing a subsystem + # this node was never configured to run. + # + # U+00D7 MULTIPLICATION SIGN for fail, deliberately NOT U+2715: U+2715 is absent from the Linux + # console fonts (CP437, Lat2-Terminus16) and paints as an empty box -- an alarm state that renders + # as nothing at all is worse than no alarm state. + glyph_for() { + case "''${1:-}" in + ok) + code=32 + if [ "$ascii" -eq 1 ]; then glyph="[ ok ]"; else glyph="●"; fi + ;; + warn) + code=33 + if [ "$ascii" -eq 1 ]; then glyph="[warn]"; else glyph="▲"; fi + ;; + fail) + code=31 + if [ "$ascii" -eq 1 ]; then glyph="[FAIL]"; else glyph="×"; fi + ;; + na) + code=90 + if [ "$ascii" -eq 1 ]; then glyph="[ -- ]"; else glyph="·"; fi + ;; + *) + code=90 + if [ "$ascii" -eq 1 ]; then glyph="[ ?? ]"; else glyph="○"; fi + ;; + esac + if [ "$ascii" -eq 1 ]; then gw=6; else gw=1; fi + } + + # Both mappers fall through to `unknown` for anything they do not recognise, which is what makes the + # staleness blanking below a one-liner: set every collected value to "??" and every state reads + # unknown automatically, with no second code path to keep in sync. + unit_status() { + case "''${1:-}" in + active) echo ok ;; + activating | reloading | deactivating) echo warn ;; + failed) echo fail ;; + inactive) echo warn ;; + "n/a") echo na ;; + *) echo unknown ;; + esac + } + + # replication.lag_check is a VERDICT, not an ActiveState: its unit is a timer-driven oneshot whose + # success latches nothing, so the collector resolves it against `is-failed` (the contract stated at + # nixos/vault-replication.nix:628) and emits ok/failed rather than active/inactive. Feeding it to + # unit_status would paint a healthy standby amber forever. + check_status() { + case "''${1:-}" in + ok) echo ok ;; + failed) echo fail ;; + "n/a") echo na ;; + *) echo unknown ;; + esac + } + + vault_status() { + case "''${1:-}" in + unlocked) echo ok ;; + locked) echo warn ;; + wrong-device) echo fail ;; + "n/a") echo na ;; + *) echo unknown ;; + esac + } + + fmt_time() { + if is_uint "''${1:-}" && [ "$1" -gt 0 ]; then + ${pkgs.coreutils}/bin/date -u -d "@$1" +'%Y-%m-%d %H:%M:%SZ' 2>/dev/null || printf 'unknown' + else + printf 'unknown' + fi + } + + # Coarse, single-unit age. Deliberately imprecise: the reader needs "is this verdict minutes or + # months old", and a wider string would be the first thing truncated out of a narrow value column. + fmt_age() { + local secs=''${1:-0} + if [ "$secs" -lt 60 ]; then + printf '%ds' "$secs" + elif [ "$secs" -lt 3600 ]; then + printf '%dm' $(( secs / 60 )) + elif [ "$secs" -lt 86400 ]; then + printf '%dh' $(( secs / 3600 )) + else + printf '%dd' $(( secs / 86400 )) + fi + } + + read_status() { + local json line_count gen mtime + degraded=0 + degrade_reason="" + last_ts=unknown + + now="$(${pkgs.coreutils}/bin/date +%s 2>/dev/null || echo 0)" + is_uint "$now" || now=0 + + v_schema="" + v_generated="" + v_node="" + v_vault="" + v_gate="" + v_vw="" + v_mesh="" + v_wisp="" + v_iface="" + v_up="" + v_addr="" + v_role="" + v_lagchk="" + v_lag="" + v_al="" + v_al_at="" + v_error="" + + if [ ! -r "$status_file" ]; then + degraded=1 + degrade_reason="NO STATUS SNAPSHOT - COLLECTOR HAS NOT RUN" + return 0 + fi + + mtime="$(${pkgs.coreutils}/bin/stat -c %Y -- "$status_file" 2>/dev/null || true)" + is_uint "$mtime" || mtime=0 + + if ! json="$(${pkgs.jq}/bin/jq -r ${lib.escapeShellArg jqProgram} < "$status_file" 2>/dev/null)"; then + degraded=1 + degrade_reason="STATUS SNAPSHOT UNREADABLE - MALFORMED JSON" + last_ts="$(fmt_time "$mtime")" + return 0 + fi + # A short list means the filter did not produce the shape this reader expects. Reading it anyway + # would silently slide every field into the wrong slot, which is the one failure that looks + # completely plausible on screen. The LAST element is a "." sentinel purely so that a trailing + # EMPTY field cannot be eaten by command substitution's newline stripping and fake a short list. + line_count="$(printf '%s\n' "$json" | ${pkgs.coreutils}/bin/wc -l)" + if [ "$line_count" -ne 18 ]; then + degraded=1 + degrade_reason="STATUS SNAPSHOT UNREADABLE - UNEXPECTED SHAPE" + last_ts="$(fmt_time "$mtime")" + return 0 + fi + + { + read -r v_schema + read -r v_generated + read -r v_node + read -r v_vault + read -r v_gate + read -r v_vw + read -r v_mesh + read -r v_wisp + read -r v_iface + read -r v_up + read -r v_addr + read -r v_role + read -r v_lagchk + read -r v_lag + read -r v_al + read -r v_al_at + read -r v_error + } <<< "$json" + + if [ "$v_schema" != "1" ]; then + degraded=1 + # Bounded before it reaches the banner. clean() strips control bytes but not LENGTH, and the + # banner word-wraps rather than hard-cutting, so an unbounded snapshot value would grow the + # header without limit -- and `head` is the one array the row-budget fitter never trades away, + # so a long enough value pushes every status row and the SSH line off the screen. + degrade_reason="STATUS SCHEMA $(clip 24 "$(clean "$v_schema")") UNSUPPORTED - EXPECTED 1" + last_ts="$(fmt_time "$mtime")" + return 0 + fi + + # The collector's own emergency payload (its jq emit failed) carries `error` and a CURRENT + # generated_at, so staleness can never catch it. Read here it becomes a banner; unread it was a + # perfectly normal-looking frame with every row "unknown" and nothing on the glass admitting a + # fault -- the exact "plausible but wrong" screen this module refuses to show. + if [ -n "$v_error" ]; then + degraded=1 + degrade_reason="COLLECTOR FAULT - $(clip 60 "$(clean "$v_error")")" + last_ts="$(fmt_time "$mtime")" + return 0 + fi + + # age = now - MIN(mtime, generated_at). The minimum is the pessimistic reading and the correct + # one: mtime alone trusts a collector that keeps touching the file while re-emitting a cached + # payload, and generated_at alone trusts a clock. Either one being old is enough to be old. + gen="$v_generated" + is_uint "$gen" || gen=0 + oldest="$mtime" + if [ "$gen" -lt "$oldest" ]; then oldest="$gen"; fi + age=$(( now - oldest )) + if [ "$age" -lt 0 ]; then age=0; fi + last_ts="$(fmt_time "$oldest")" + + if [ "$age" -gt "$stale_seconds" ]; then + degraded=1 + degrade_reason="STALE - DATA ''${age}s OLD (LIMIT ''${stale_seconds}s)" + fi + } + + # Banner text WRAPS on whitespace; it is never sliced mid-sentence. A hard cut to w-6 (34 columns at + # the narrow tier) turned "VALUES BELOW ARE NOT KNOWN AND SHOW AS ??" -- 41 characters -- into + # "...AND SHO", losing the very "??" the sentence exists to explain, and truncated a long + # "STALE - DATA s OLD (LIMIT s)" into an unbalanced open paren. That is the same + # cut-mid-token failure already fixed in the SSH footer, which drops its note whole rather than + # slicing; a banner cannot be dropped (it is the alarm), so it wraps instead. + # + # Banners live in `head`, which the row-budget fitter never trades away, so an extra wrapped line + # costs a body row on a very short screen rather than the alarm itself. + add_banner() { + local bcode=$1 inner text word cur bannerMaxLines=6 + shift + inner=$(( w - 6 )) + if [ "$inner" -lt 1 ]; then inner=1; fi + colorize "$bcode" "$(repeat_char "$w" '!')" + head+=("$_c") + # Hard ceiling on wrapped lines. Callers clip their snapshot values, so this should never bind; + # it is the backstop that keeps a missed clip from costing the operator the whole screen rather + # than a few words. `head` is never traded away by the row-budget fitter, so an unbounded banner + # pushes every status row AND the SSH line off the glass -- strictly worse than a cut sentence. + local emitted=0 + for text in "$@"; do + text="$(clean "$text")" + # `set -f` for the word split: clean() permits `*` and `?`, and an unguarded split would let a + # degrade reason glob against the working directory and paint filenames into the alarm. + set -f + cur="" + for word in $text; do + if [ "$emitted" -ge "$bannerMaxLines" ]; then break; fi + if [ -z "$cur" ]; then + cur="$word" + elif [ $(( ''${#cur} + 1 + ''${#word} )) -le "$inner" ]; then + cur="$cur $word" + else + printf -v line '!! %-*s !!' "$inner" "''${cur:0:$inner}" + colorize "$bcode" "$line" + head+=("$_c") + emitted=$(( emitted + 1 )) + cur="$word" + fi + done + set +f + if [ "$emitted" -ge "$bannerMaxLines" ]; then cur=""; fi + # A single token longer than the whole banner is the one case still hard-cut: there is no + # whitespace to break it at, and it carries no sentence to leave half-finished. + if [ -n "$cur" ]; then + printf -v line '!! %-*s !!' "$inner" "''${cur:0:$inner}" + colorize "$bcode" "$line" + head+=("$_c") + fi + done + colorize "$bcode" "$(repeat_char "$w" '!')" + head+=("$_c") + } + + add_item() { + local st=$1 label=$2 value=$3 valw rest + glyph_for "$st" + valw=$(( w - 3 - gw - labelw )) + if [ "$valw" -lt 1 ]; then valw=1; fi + value="$(clean "$value")" + if [ -z "$value" ]; then value=unknown; fi + printf -v rest ' %-*s %s' "$labelw" "''${label:0:$labelw}" "''${value:0:$valw}" + colorize "$code" "$glyph" + body+=(" $_c$rest") + } + + two_col() { + local left right space + left="$(clean "$1")" + right="$(clean "$2")" + space=$(( w - 2 - ''${#right} )) + if [ "$space" -lt 1 ]; then space=1; fi + printf -v line ' %-*s %s' "$space" "''${left:0:$space}" "$right" + printf '%s' "''${line:0:$w}" + } + + build_frame() { + local i n budget kept sep al_value al_age al_valw al_uptime al_long al_short + detect_caps + detect_geometry + read_status + + # THE staleness rule. Every collected value is REPLACED by "??" -- not dimmed, not greyed, not + # kept as a last-known reading. A greyed-out "UNLOCKED" is still read as "unlocked" by someone + # glancing at the screen from across the room, and the whole point of this display is what it + # tells that person. Only the last-known TIMESTAMP (in the footer) and the SSH line survive, being + # static config rather than collected facts. The node label is NOT among them -- it is collected, + # so it blanks with everything else and a stale screen cannot tell you which box you are looking + # at. The SSH line is what identifies the node then. + if [ "$degraded" -eq 1 ]; then + v_node="??" + v_vault="??" + v_gate="??" + v_vw="??" + v_mesh="??" + v_wisp="??" + v_iface="??" + v_up="??" + v_addr="??" + v_role="??" + v_lagchk="??" + v_lag="??" + v_al="??" + v_al_at="??" + fi + + brandl=() + head=() + body=() + foot=() + + if [ "$ascii" -eq 1 ]; then + for i in "''${mark_ascii[@]}"; do brandl+=(" $i"); done + brandl+=("" " $wordmark_ascii" "") + else + for i in "''${mark_unicode[@]}"; do brandl+=(" $i"); done + brandl+=("" " $wordmark" "") + fi + + sep="$(repeat_char "$w" '-')" + if [ "$tier" = full ]; then head+=("$sep"); fi + # Minutes, not seconds. The header clock is part of the frame comparison key, so a ticking seconds + # field would make every frame differ from the last and turn the diff-repaint below into an + # unconditional 1 Hz full repaint. Sub-minute liveness is the heartbeat's job. + # The node label is a COLLECTED value, so it blanks to ?? with everything else when the snapshot + # goes stale. The static hostname stays reachable in the SSH line at the foot of the frame, and + # that pair -- "I cannot tell you anything" plus "here is how to come and find out" -- is what the + # operator actually needs. The empty case is the collector's own emergency fallback payload, which + # carries no label at all. + if [ "$degraded" -eq 0 ] && [ -z "$v_node" ]; then v_node="$host_name"; fi + now_hm="$(${pkgs.coreutils}/bin/date -u +'%H:%MZ' 2>/dev/null || printf "unknown")" + head+=("$(two_col "NODE $v_node" "$now_hm")") + if [ "$tier" = full ]; then head+=("$sep"); fi + + # Banner precedence. nixos/admin-access.nix:198 writes its KEEP NODE ANTI-LOCKOUT message straight + # to /dev/console -- this very VT -- and a 1 Hz repaint erases it inside a second. Losing that + # message means an operator never learns the node is unreachable until they try to reach it, so it + # is re-raised here as the most visually dominant element on the screen, above the fold, and it is + # placed in `head` where the row-budget fitter can never drop or scroll it. + # + # When the snapshot is degraded the anti-lockout field is "??" and no claim is made either way: + # the staleness banner is already saying that nothing on the screen is known. + if [ "$degraded" -eq 0 ] && [ "$v_al" = "failed" ]; then + add_banner 31 "ANTI-LOCKOUT TRIPPED: NO ADMIN SSH KEY" "REMOTE ACCESS IS IMPOSSIBLE - PROVISION A KEY" + fi + if [ "$degraded" -eq 1 ]; then + add_banner 31 "$degrade_reason" "VALUES BELOW ARE NOT KNOWN AND SHOW AS ??" + fi + + add_item "$(vault_status "$v_vault")" VAULT "$v_vault" + add_item "$(unit_status "$v_gate")" GATE "$v_gate" + add_item "$(unit_status "$v_vw")" VAULTWARDEN "$v_vw" + add_item "$(unit_status "$v_mesh")" MESH "$v_mesh" + add_item "$(unit_status "$v_wisp")" WISP "$v_wisp" + if [ "$show_mesh_link" = 1 ]; then + case "$v_up" in + true) add_item ok "MESH LINK" "up ($v_iface)" ;; + false) add_item warn "MESH LINK" "down ($v_iface)" ;; + *) add_item unknown "MESH LINK" "$v_up" ;; + esac + fi + # The `degraded` arm has to come FIRST in each of these. Without it a blanked "??" is a non-empty + # string / a non-integer and would fall into the healthy branch or the "unknown" branch, painting + # a green dot next to an unknown address, or the word "unknown" where the frame has just promised + # that every collected value reads "??". + if [ "$show_mesh_addr" = 1 ]; then + if [ "$degraded" -eq 1 ]; then + add_item unknown "MESH ADDR" "$v_addr" + elif [ -n "$v_addr" ]; then + add_item ok "MESH ADDR" "$v_addr" + else + add_item unknown "MESH ADDR" unknown + fi + fi + if [ "$show_repl" = 1 ]; then + if is_uint "$v_lag"; then + add_item "$(check_status "$v_lagchk")" REPLICATION "$v_role lag ''${v_lag}s" + else + add_item "$(check_status "$v_lagchk")" REPLICATION "$v_role $v_lagchk" + fi + fi + # The anti-lockout verdict is BOOT-LATCHED (Type=oneshot + RemainAfterExit, no timer), so unlike + # every other row it can be arbitrarily old while the snapshot carrying it is perfectly fresh. + # Showing the bare word would be a green dot asserting "no lockout" about a node whose key was + # deleted weeks ago. The age is therefore part of the value, not metadata. + # + # Two forms, longest-that-fits, then dropped whole -- the SSH footer's discipline. Slicing this + # is what produces "active (checked 41d" with an open paren, and the ascii tier's six-column + # glyph leaves as little as 19 columns for the value at the narrow tier. + al_value="$v_al" + # Clamped to uptime, not just to "not in the future". ActiveEnterTimestamp is CLOCK_REALTIME at + # activation, so on a box with no RTC (or with NTP landing late) the clock jumps FORWARD after + # boot and a check that ran seconds ago computes as decades. The unit cannot have activated + # before this boot, so an age exceeding uptime is impossible and the age is dropped rather than + # shown -- a fabricated reading on the one row added to stop fabricated readings would be the + # worst possible place to have one. + # KEEPNODE_STATUS_UPTIME is a test seam, like KEEPNODE_STATUS_WIDTH/ASCII/NOCOLOR above. A long + # age is only PHYSICALLY possible on a long-lived node, so a VM three minutes into its life + # cannot produce the "checked 41d ago" string whose width behaviour needs testing. Overriding the + # plausibility bound is the only way to exercise that path; systemd never sets this, and the worst + # a wrong value can do is mis-display an age -- it gates no security property. + al_uptime="''${KEEPNODE_STATUS_UPTIME:-}" + if [ -z "$al_uptime" ] && [ -r /proc/uptime ]; then + al_uptime="$(${pkgs.coreutils}/bin/cut -d' ' -f1 /proc/uptime 2>/dev/null || true)" + al_uptime="''${al_uptime%%.*}" + fi + if [ "$degraded" -eq 0 ] && is_uint "$v_al_at" && [ "$v_al_at" -gt 0 ] && [ "$now" -ge "$v_al_at" ] && + { ! is_uint "$al_uptime" || [ $(( now - v_al_at )) -le "$al_uptime" ]; }; then + al_age="$(fmt_age $(( now - v_al_at )))" + al_valw=$(( w - 3 - labelw )) + if [ "$ascii" -eq 1 ]; then al_valw=$(( al_valw - 6 )); else al_valw=$(( al_valw - 1 )); fi + # Measured off the candidate strings themselves rather than off hand-counted widths, so the + # fit test cannot drift away from the text it is meant to be measuring. + al_long="$v_al (checked $al_age ago)" + al_short="$v_al ($al_age ago)" + if [ "''${#al_long}" -le "$al_valw" ]; then + al_value="$al_long" + elif [ "''${#al_short}" -le "$al_valw" ]; then + al_value="$al_short" + fi + fi + add_item "$(unit_status "$v_al")" "ANTI-LOCKOUT" "$al_value" + + if [ "$tier" = full ]; then foot+=("$sep"); fi + # two_col gives its left field w-2 columns. The note is appended only if the WHOLE line fits in + # that budget, so a long hostname (up to 63 chars in "ssh keepadmin@") drops the note + # instead of having it sliced mid-token. The command may still be hard-truncated at 40 columns, + # but it carries no punctuation that can be left unbalanced. + ssh_line="SSH $ssh_command" + if [ "$tier" = full ] && [ $(( ''${#ssh_line} + ''${#ssh_note} )) -le $(( w - 2 )) ]; then + ssh_line="$ssh_line$ssh_note" + fi + foot+=("$(two_col "$ssh_line" "")") + # The heartbeat placeholder. \001 is not survivable through clean(), so no collected value can + # ever introduce a second one and move the cursor target. + foot+=(" "$'\001'" last update: $last_ts") + + # Row budget. Decoration goes first, then items from the end; the header (which carries the + # banners) and the footer are never traded away. + all=() + if [ $(( ''${#brandl[@]} + ''${#head[@]} + ''${#body[@]} + ''${#foot[@]} )) -le "$term_rows" ] && + [ "$tier" != narrow ]; then + all+=("''${brandl[@]}") + fi + all+=("''${head[@]}") + budget=$(( term_rows - ''${#all[@]} - ''${#foot[@]} )) + if [ "$budget" -lt 0 ]; then budget=0; fi + n=''${#body[@]} + if [ "$n" -gt "$budget" ]; then n="$budget"; fi + i=0 + while [ "$i" -lt "$n" ]; do + all+=("''${body[$i]}") + i=$(( i + 1 )) + done + if [ "$n" -lt "''${#body[@]}" ] && [ "$n" -gt 0 ]; then + kept=$(( ''${#body[@]} - n + 1 )) + all[$(( ''${#all[@]} - 1 ))]=" ... $kept more items (screen too short)" + fi + all+=("''${foot[@]}") + if [ "''${#all[@]}" -gt "$term_rows" ]; then + all=("''${all[@]:0:$term_rows}") + fi + + # ONE variable holding the WHOLE frame. Emitting line by line tears visibly: the collector's next + # snapshot can land mid-frame and the screen shows half of one reading and half of another. + frame_tpl="" + for i in "''${!all[@]}"; do + if [ "$i" -gt 0 ]; then frame_tpl="$frame_tpl"$'\n'; fi + frame_tpl="$frame_tpl$pad''${all[$i]}" + done + } + + # Where the heartbeat cell sits, in 1-based screen coordinates, derived from the assembled template. + # Everything to its left on that line is ASCII spaces, so a byte offset is a column offset. + locate_heartbeat() { + local pre nls lastline + hb_row=0 + hb_col=0 + case "$frame_tpl" in + *$'\001'*) : ;; + *) return 0 ;; + esac + pre="''${frame_tpl%%$'\001'*}" + nls="''${pre//[!$'\n']/}" + hb_row=$(( ''${#nls} + 1 )) + lastline="''${pre##*$'\n'}" + hb_col=$(( ''${#lastline} + 1 )) + } + + hb_chars='-\|/' + hb_i=0 + + heartbeat_char() { + hb="''${hb_chars:$(( hb_i % 4 )):1}" + hb_i=$(( hb_i + 1 )) + } + + if [ "$once" -eq 1 ]; then + build_frame + heartbeat_char + frame="''${frame_tpl/$'\001'/$hb}" + # printf '%s', NEVER printf "$frame": a node label or hostname containing %s or %n would otherwise + # be interpreted as a conversion and corrupt (or crash) the output. + printf '%s\n' "$frame" + exit 0 + fi + + # Console blanking would hide the one screen this feature exists to show, on exactly the timescale + # an unattended appliance is left alone for. + ${pkgs.util-linux}/bin/setterm --blank 0 --powersave off 2>/dev/null || true + printf '\033[?25l' + + # A frozen renderer cannot draw its own staleness banner -- the banner is painted by the thing that + # died. The rotating heartbeat is the only cue that survives that failure, and this trap covers the + # case where the process at least gets to exit: it leaves an explicit statement on the glass instead + # of a plausible-looking last frame that nobody can date. + on_exit() { + local msg bar + bar="$(repeat_char "''${w:-80}" '!')" + printf '\033[H\033[J\033[?25h' + printf '%s\n' "$bar" + printf -v msg '!! %-*s !!' $(( ''${w:-80} - 6 )) "KEEP NODE STATUS DISPLAY STOPPED - NO DATA IS SHOWN" + printf '%s\n' "''${msg:0:''${w:-80}}" + printf '%s\n' "$bar" + } + # EXIT alone carries the painting; INT/TERM only convert the signal into an ordinary exit. Trapping + # on_exit for all three would run it TWICE on a systemctl stop -- once for the signal and once for + # the exit it causes -- leaving two stacked STOPPED banners on the terminal. + trap on_exit EXIT + trap 'exit 143' INT TERM + + prev_key="" + last_full=0 + while :; do + build_frame + locate_heartbeat + heartbeat_char + + # Serial efficiency: at 9600 baud a full 24x80 frame is about 2.5 seconds of wire time, so a 1 Hz + # unconditional repaint would never finish one before starting the next. The comparison key is the + # template, heartbeat placeholder and all, so the heartbeat's own rotation does not count as a + # change; when nothing else moved, only the single heartbeat cell is rewritten. The forced full + # repaint at most once per 60s is how a line garbled by kernel output heals itself. + if [ "$frame_tpl" != "$prev_key" ] || [ $(( now - last_full )) -ge 60 ] || [ "$hb_row" -eq 0 ]; then + # ESC[H + per-line ESC[K, then ESC[J. NEVER ESC[2J: clearing the whole screen before redrawing + # it shows a black flash on every repaint, which at 1 Hz is a strobing screen. + out=$'\033[H' + for i in "''${!all[@]}"; do + if [ "$i" -gt 0 ]; then out="$out"$'\r\n'; fi + out="$out$pad''${all[$i]}"$'\033[K' + done + # Substituting into the assembled string rather than re-rendering keeps the emitted frame + # byte-identical to the one the key was computed from. + out="''${out/$'\001'/$hb}"$'\033[J' + printf '%s' "$out" + prev_key="$frame_tpl" + last_full="$now" + else + printf '\033[%d;%dH%s' "$hb_row" "$hb_col" "$hb" + fi + + ${pkgs.coreutils}/bin/sleep "$repaint_seconds" + done + ''; +in +{ + options.keepNode.statusDisplay = { + enable = lib.mkEnableOption "the read-only Keep status screen on the physical console"; + + backend = lib.mkOption { + type = lib.types.enum [ "console" ]; + default = "console"; + description = '' + How the status screen is painted. Only `console` , writing frames directly to a Linux virtual + terminal , is implemented, and it is the only value the enum currently accepts. + + A Wayland kiosk (`cage` running `foot`) was evaluated and REJECTED. `foot` binds + `ctrl+shift+n` to spawning `$SHELL`, which would hand a shell to anyone standing at the box and + blow straight through the "console grants nothing" fence this module is built around. Beyond + that single binding, a compositor adds a seat and a keyboard input consumer to an appliance + that deliberately has neither, which is a large new attack surface bought for a nicer font. + + The option is an enum rather than a bare implementation detail so that a future framebuffer or + DRM backend arrives as an added enum value rather than as a breaking rename of this option. + ''; + }; + + tty = lib.mkOption { + type = lib.types.ints.between 1 12; + default = 1; + example = 3; + description = '' + Which Linux virtual terminal the status screen owns. Defaults to tty1 because that is the VT + the machine shows on boot, and so the only one an operator with a monitor plugged in will + actually see without knowing to press Alt+F-something. Move it to a free VT when something else + needs tty1 (for example the bring-up autologin console from `keepNode.debugAccess`). + ''; + }; + + refreshSeconds = lib.mkOption { + type = lib.types.ints.positive; + default = 5; + description = '' + How often the collector re-samples node state and rewrites its status snapshot. This is the + rate at which the displayed FACTS can change; the repaint cadence below only controls how + quickly a new snapshot reaches the glass. Lower values cost more probing of the services being + sampled, so keep this comfortably above the time a full collection round takes. + ''; + }; + + repaintSeconds = lib.mkOption { + type = lib.types.ints.positive; + default = 1; + description = '' + How often the renderer redraws the screen from the latest collector snapshot. Painting is + cheap and touches nothing but the VT, so this runs faster than collection: it keeps the clock + and the staleness banner honest even when the collector has stopped producing snapshots. + ''; + }; + + staleSeconds = lib.mkOption { + type = lib.types.ints.positive; + default = 20; + description = '' + How old the collector's snapshot may be before the screen is marked STALE. Past this age the + renderer stops presenting the numbers as current, because a frozen-but-plausible screen is + worse than an obviously broken one , it invites an operator to make decisions from data that + stopped tracking reality minutes ago. Should be several multiples of `refreshSeconds` so + ordinary collection jitter never trips it. + ''; + }; + + probeTimeoutSeconds = lib.mkOption { + type = lib.types.ints.positive; + default = 2; + description = '' + Per-probe wall-clock timeout inside the collector. A single wedged probe must not be able to + stall a whole collection round, because a stalled round ages the snapshot and would eventually + show STALE for the entire screen when in fact one subsystem is hung. Keep it well under + `refreshSeconds` so that even several slow probes fit inside one round. + ''; + }; + + showMeshAddress = lib.mkOption { + type = lib.types.bool; + default = false; + description = '' + Show this node's mesh address on the status screen. Off by default: a mesh address is topology + intelligence, and this screen renders it to whoever is physically standing in front of the box + , which includes anyone who should not be learning the shape of the private network. Turn it + on when the console is in a trusted location and the address genuinely helps bring-up. + ''; + }; + + nodeLabel = lib.mkOption { + type = lib.types.str; + default = config.networking.hostName; + defaultText = lib.literalExpression "config.networking.hostName"; + example = "vault-rack-3"; + description = '' + The human-facing name shown in the status screen header. Defaults to the system hostname, + which is what an operator already correlates with the box. Override it when the hostname is an + opaque identifier and a physical label (rack position, site name) is what the person standing + in front of the machine actually needs to match against. + ''; + }; + }; + + config = lib.mkIf cfg.enable { + assertions = [ + { + # nixos/debug-access.nix:19 sets services.getty.autologinUser = "root", which claims tty1. + # Two owners, one VT: the status renderer and the autologin getty would fight over the same + # terminal and the operator loses the bring-up console they turned debugAccess on to get. + assertion = !(cfg.tty == 1 && (config.keepNode.debugAccess.enable or false)); + message = "keepNode.statusDisplay.tty is 1 but keepNode.debugAccess.enable is true: debug-access sets services.getty.autologinUser = \"root\", which claims tty1, so the status renderer and the autologin getty would contend for the same virtual terminal and the bring-up console would be lost. Disable keepNode.debugAccess (production), or move keepNode.statusDisplay.tty to a free VT such as 2 (bring-up)."; + } + { + # Staleness must trigger LATER than the collector produces snapshots. At or below the + # collection cadence, every snapshot is already stale by the time it is painted, so the screen + # reads STALE permanently , which trains the operator to ignore the one banner that matters. + # A multiple of at least 3x leaves room for ordinary collection jitter. + assertion = cfg.staleSeconds > cfg.refreshSeconds; + message = "keepNode.statusDisplay.staleSeconds (${toString cfg.staleSeconds}) must be greater than refreshSeconds (${toString cfg.refreshSeconds}): staleness that triggers at or below the collector cadence marks every freshly-collected snapshot STALE, so the banner is permanently lit and the operator learns to ignore it. Set staleSeconds to at least 3x refreshSeconds (e.g. ${ + toString (3 * cfg.refreshSeconds) + })."; + } + { + # The renderer is what draws the STALE banner, so it must repaint at least once within the + # staleness window. Otherwise a healthy-looking frozen screen persists for a full repaint + # period after the collector has already died, with nothing on the glass admitting it. + assertion = cfg.repaintSeconds < cfg.staleSeconds; + message = "keepNode.statusDisplay.repaintSeconds (${toString cfg.repaintSeconds}) must be less than staleSeconds (${toString cfg.staleSeconds}): the renderer draws the STALE banner, so if it repaints less often than the staleness window it keeps showing a healthy-looking frozen screen for a full repaint period after the collector dies. Lower repaintSeconds below staleSeconds (the default 1 is fine)."; + } + { + # The mesh address has nothing to read when the mesh module is off. It would render a + # permanent "n/a", which on a status screen looks like a FAULT rather than a disabled feature, + # and sends the operator chasing a mesh problem that does not exist. + assertion = cfg.showMeshAddress -> (config.keepNode.mesh.enable or false); + message = "keepNode.statusDisplay.showMeshAddress is true but keepNode.mesh.enable is false: there is no mesh to report on, so the field would render a permanent \"n/a\" that reads as a fault and sends the operator chasing a non-existent mesh problem. Enable keepNode.mesh, or leave showMeshAddress off."; + } + ]; + + systemd.services.keep-node-status-collect = { + description = "Collect the Keep node status snapshot for the console display"; + # One eager run at boot so the first frame the renderer paints is real rather than a missing-file + # placeholder; the timer takes over from there. NOTHING may `require` this unit: it is a reporting + # side-channel, and boot must never be able to block on it. + wantedBy = [ "multi-user.target" ]; + after = [ "local-fs.target" ]; + serviceConfig = { + # NOT RemainAfterExit: this repeats on a timer, and a unit that stays "active" after its run + # would make every subsequent trigger a no-op. + Type = "oneshot"; + ExecStart = lib.getExe collector; + RuntimeDirectory = "keep-node-status"; + RuntimeDirectoryMode = "0755"; + # The snapshot must outlive each oneshot run, otherwise systemd removes the directory the + # instant the collector exits and the renderer reads a file that exists for milliseconds + # per cycle. + RuntimeDirectoryPreserve = "yes"; + UMask = "0022"; + # Sandboxing mirrors nixos/mesh.nix:440-468. This unit runs as root only because it reads + # unit state and the vault gate's device mapping; it needs no capability at all, writes + # exactly one directory, and must not be a lever if anything it shells out to misbehaves. + # No PrivateNetwork: `ip` must observe the HOST network namespace or every mesh fact it + # reports would describe an empty private netns instead of the real interface. + NoNewPrivileges = true; + CapabilityBoundingSet = [ ]; + ProtectSystem = "strict"; + ProtectHome = true; + PrivateTmp = true; + # The collector cannot use PrivateNetwork (see above), so the socket families it may open are + # narrowed instead. Its baseline needs are AF_UNIX (systemctl and journalctl talk to PID 1 and + # the journal over unix sockets) and AF_NETLINK (how ip(8) actually asks the kernel about + # links and addresses -- without it every mesh fact reads as unknown). + # + # Nothing else is admitted: no probe here talks to a network, so this root unit cannot open an + # internet socket at all. + RestrictAddressFamilies = [ + "AF_UNIX" + "AF_NETLINK" + ]; + SystemCallFilter = [ "@system-service" ]; + SystemCallArchitectures = "native"; + RestrictNamespaces = true; + LockPersonality = true; + RestrictRealtime = true; + RestrictSUIDSGID = true; + MemoryDenyWriteExecute = true; + }; + }; + + systemd.timers.keep-node-status-collect = { + description = "Re-collect the Keep node status snapshot"; + wantedBy = [ "timers.target" ]; + timerConfig = { + OnBootSec = "5s"; + # OnUnitInactiveSec, NOT OnUnitActiveSec: the interval is measured from the END of the previous + # run, so a collection round that runs long simply delays the next one instead of stacking a + # second collector on top of a still-running first. + OnUnitInactiveSec = "${toString cfg.refreshSeconds}s"; + AccuracySec = "1s"; + Unit = "keep-node-status-collect.service"; + }; + }; + + # Writing users.users directly is normally something this tree avoids, but this account is owned by + # nothing else: it exists solely so the process holding a terminal is not root. It has no password + # (`!` is unmatchable, so `su` and every PAM password path refuse it), no shell, and no home, so + # there is nothing to log into even if a login path to it were somehow found. + users.users.${renderUser} = { + isSystemUser = true; + group = renderUser; + description = "Keep node console status renderer"; + shell = "${pkgs.shadow}/bin/nologin"; + hashedPassword = "!"; + home = "/var/empty"; + createHome = false; + }; + users.groups.${renderUser} = { }; + + systemd.services.keep-node-status-render = { + description = "Paint the Keep node status screen on ${ttyPath}"; + wantedBy = [ "multi-user.target" ]; + after = [ "keep-node-status-collect.service" ]; + # Belt and braces alongside the getty/autovt disabling below: even if something re-enables the + # getty for this VT, systemd will not run both owners on the same terminal at once. + conflicts = [ gettyUnit ]; + # NO start rate limit. With systemd's defaults (StartLimitBurst=5 in StartLimitIntervalSec=10s) + # and RestartSec=2 below, a renderer that fails fast burns its five attempts in about ten + # seconds and is then given up on PERMANENTLY -- the 209/STDOUT device-cgroup bug this module + # already hit once looked exactly like that. + # + # There is no recovery from that state at the console. getty@ and autovt@ for this VT are + # `enable = false`, which NixOS implements as a /dev/null symlink -- i.e. MASKED -- so + # `systemctl start getty@tty1` refuses and the operator is left with a permanently dead terminal + # on the one screen this feature exists to provide. Retrying forever is strictly better: the + # failure is loud in the journal either way, and a transient cause (a device that appears late, + # a VT briefly held by something else) heals itself instead of latching. + # + # This is a [Unit] directive, not [Service]: putting it in serviceConfig silently does nothing. + unitConfig.StartLimitIntervalSec = 0; + # tput needs a terminfo name, and setterm's blanking controls are linux-console specific. + environment = { + TERM = "linux"; + LANG = "C.UTF-8"; + }; + serviceConfig = { + Type = "simple"; + ExecStart = "${lib.getExe renderer}"; + Restart = "always"; + RestartSec = 2; + + User = renderUser; + Group = renderUser; + + # THE security property of this module. The renderer holds NO descriptor on the keyboard -- + # not "tty", not "tty-force", which would both give it fd 0 on the console. With no read path + # there is nothing to escape from: no shell-out, no pager, no "press any key". Everything else + # in this block is defence in depth around a process that already cannot be talked to. + StandardInput = "null"; + StandardOutput = "tty"; + # Diagnostics go to the journal, never to the terminal: a stray warning painted onto the VT + # would corrupt the frame and, worse, would be the one line on screen with no timestamp. + StandardError = "journal"; + TTYPath = ttyPath; + TTYReset = true; + TTYVHangup = true; + TTYVTDisallocate = true; + + # Deliberately NOT PrivateDevices: that substitutes a private /dev and fights TTYPath and + # TTYVTDisallocate, which need the real VT. + # + # DevicePolicy=closed narrows the unit to the standard pseudo-devices (/dev/null, zero, full, + # random, urandom). That set does NOT include a virtual terminal: /dev/tty1 is an ordinary + # character device, so opening it for StandardOutput=tty is refused by the device cgroup with + # EPERM and the unit dies at step STDOUT (209/STDOUT) in a restart loop -- the screen never + # paints at all. The terminal is opened with this unit's cgroup already applied, so the policy + # very much does reach it. + # + # DeviceAllow re-admits exactly the one VT this module owns, and nothing else: it names the + # configured tty path rather than the whole `char-tty` class, so enabling the display on tty3 + # does not also hand the renderer tty1..tty12. + # + # `rw` IS THE MINIMUM -- do not "harden" it to `w`. The read bit is NOT for StandardOutput: + # that open is O_WRONLY and `w` alone satisfies it. It is required because TTYVHangup and + # TTYReset below reopen the terminal O_RDWR|O_NOCTTY in the child, after the cgroup has been + # attached, in order to issue their ioctls. Dropping to `w` reintroduces the 209/STDOUT + # restart loop. The renderer process itself still never holds a readable descriptor on the VT + # (StandardInput=null; asserted mechanically in tests/status-display.nix), so this bit costs + # nothing against the no-input property -- it is a device-cgroup permission, not an open fd. + DevicePolicy = "closed"; + DeviceAllow = [ "${ttyPath} rw" ]; + # A screen painter has no business on any network, in any direction. + PrivateNetwork = true; + RestrictAddressFamilies = [ "AF_UNIX" ]; + NoNewPrivileges = true; + CapabilityBoundingSet = [ ]; + ProtectSystem = "strict"; + ProtectHome = true; + PrivateTmp = true; + ProtectProc = "invisible"; + ProcSubset = "pid"; + ProtectKernelTunables = true; + ProtectKernelModules = true; + ProtectKernelLogs = true; + ProtectClock = true; + ProtectControlGroups = true; + SystemCallFilter = [ "@system-service" ]; + SystemCallArchitectures = "native"; + RestrictNamespaces = true; + LockPersonality = true; + MemoryDenyWriteExecute = true; + RestrictRealtime = true; + RestrictSUIDSGID = true; + UMask = "0077"; + }; + }; + + # Displacing the getty is not cosmetic: two processes writing one VT interleave their output, and + # the login prompt would be repainted over once per second anyway. Both units must go. + # + # autovt@ is the one that is easy to miss. logind instantiates it on demand, so leaving it enabled + # means the moment anyone presses Alt+F (or anything calls chvt) a fresh getty is spawned onto + # the terminal the renderer already owns, and the two race for the device. + systemd.services."getty@tty${toString cfg.tty}".enable = false; + systemd.services."autovt@tty${toString cfg.tty}".enable = false; + + # Kernel messages above KERN_ERR would otherwise scribble across the frame. mkDefault, NOT mkForce: + # an operator debugging a boot problem must still be able to raise the level, and the cost of doing + # so is only a garbled frame that the next repaint (or the 60s forced full repaint) cleans up. + boot.consoleLogLevel = lib.mkDefault 3; + + # tty2-6 keep their gettys ON PURPOSE. Every account on the node is password-locked, so those + # prompts grant exactly nothing to someone at the keyboard, and `keepNode.debugAccess` uses them + # during bring-up. Taking them away would remove a bring-up affordance while adding no security. + }; +} diff --git a/tests/status-display.nix b/tests/status-display.nix new file mode 100644 index 0000000..f5c6ff3 --- /dev/null +++ b/tests/status-display.nix @@ -0,0 +1,748 @@ +# The read-only console status screen, on a booted machine. Everything here needs a real kernel VT, a +# real systemd, and real units to probe, which is why none of it can be an eval check: the collector's +# probes, the renderer's ownership of tty1, and the getty displacement only mean anything once something +# has actually booted. +# +# The load-bearing assertion is the STALE one. This display's entire justification is that it must never +# show a plausible-but-frozen reading, so the test stops the collector, waits past staleSeconds, and +# proves BOTH that the banner lights AND that no healthy value survived anywhere on the glass. Then it +# restarts the collector and proves recovery, because a STALE banner that is simply always-on would pass +# the first half of that and be worthless. +# +# Screen content is read from /dev/vcs1 -- the kernel's live character cells for tty1 -- not from OCR. +# That is exact: it is the bytes the VT actually holds. OCR against the box-drawing brand mark is flaky +# and asserting on it would be theatre. +# +# Run: nix build .#checks.x86_64-linux.status-display +{ + nvpnPackage, + vaultRsaKeyFixture, + ... +}: +let + statusFile = "/run/keep-node-status/status.json"; + + # Cadences fast enough that the staleness window is tens of seconds rather than minutes, but NOT as + # tight as they could be on real hardware. A collection round spawns a good number of short-lived + # probes, and on two contending VMs those rounds were measured taking up to 25s of wall clock (against + # ~2s of CPU) -- so with staleSeconds down at 6 the snapshot was legitimately stale the instant it was + # written and the screen read STALE permanently. That is the module's assertion documentation being + # exactly right about wanting several multiples of refreshSeconds; it just needs more headroom under a + # loaded hypervisor than on the appliance. Still satisfies both module assertions: + # staleSeconds (30) > refreshSeconds (5), and repaintSeconds (1) < staleSeconds (30). + display = { + enable = true; + refreshSeconds = 5; + staleSeconds = 30; + repaintSeconds = 1; + }; + + # Generous, because the bound is one collection round on a contended host, not the product's cadence. + slow = 120; +in +{ + name = "keep-node-status-display"; + + # The plain node: no mesh, no wisp, no replication. Their units are therefore genuinely NOT PRESENT, + # which is what makes this the right place to prove absent-unit -> "n/a" rather than "failed". + nodes.box = + { pkgs, ... }: + { + imports = [ ../nixos/keep-node.nix ]; + keepNode.statusDisplay = display; + environment.systemPackages = [ + pkgs.jq + pkgs.kbd + ]; + }; + + # A second node purely to execute the REPLICATION row and the collector's lag_seconds journal parsing, + # neither of which any test has ever run. Both are gated on vaultReplication.role, so they need a node + # that has one. meshReplication (and therefore keepNode.mesh) is required because the lag-check unit + # lives inside that block -- but no mesh has to actually FORM: the test writes the heartbeat file the + # active would have pushed and runs the check directly, which is enough to make it log the line the + # collector parses. Standing up real litestream replication between two VMs to observe one screen row + # would be disproportionate, and tests/mesh-replication.nix already covers that transport. + nodes.standby = + { pkgs, ... }: + { + imports = [ ../nixos/keep-node.nix ]; + keepNode.statusDisplay = display; + keepNode.mesh = { + enable = true; + package = nvpnPackage; + }; + keepNode.vaultReplication = { + # Test-only anti-pattern (exactly what rsaKeyFile warns against): a Nix-store path leaves the key world-readable in /nix/store. Safe only because this is an ephemeral per-build fixture, never a real cluster signing key; a real deploy must pass an out-of-band path. + rsaKeyFile = "${vaultRsaKeyFixture}/rsa_key.pem"; + role = "standby"; + meshReplication.enable = true; + }; + environment.systemPackages = [ pkgs.jq ]; + }; + + testScript = + { nodes, ... }: + let + render = nodes.box.systemd.services.keep-node-status-render.serviceConfig.ExecStart; + renderStandby = nodes.standby.systemd.services.keep-node-status-render.serviceConfig.ExecStart; + replicaDir = nodes.standby.keepNode.vaultReplication.litestream.replicaDir; + in + '' + start_all() + + for node in [box, standby]: + node.wait_for_unit("multi-user.target") + node.wait_for_unit("keep-node-status-render.service") + + # The renderer must be genuinely RUNNING, not crash-looping. wait_for_unit alone does not catch + # that: a unit with Restart=always is "active" for a moment on every respawn. DevicePolicy=closed + # once refused the manager's own open() of /dev/tty1 (EPERM at step STDOUT, 209/STDOUT) and the + # only symptom was a restart counter climbing behind a blank screen. Comparing NRestarts across a + # few seconds tolerates one benign restart during boot but fails a loop. + for node in [box, standby]: + node.succeed("systemctl show -p NRestarts --value keep-node-status-render.service > /tmp/n1") + box.sleep(5) + for node in [box, standby]: + node.succeed("systemctl show -p NRestarts --value keep-node-status-render.service > /tmp/n2") + node.succeed("cmp /tmp/n1 /tmp/n2") + node.succeed("systemctl is-active --quiet keep-node-status-render.service") + + # --------------------------------------------------------------------------------------------- + # The collector at runtime. Nothing in CI has ever exercised it; its only prior verification was a + # throwaway harness. The snapshot must EXIST, be world-readable (the renderer runs as an + # unprivileged user and can read nothing else), parse as JSON, and declare the schema the renderer + # is built against. + # --------------------------------------------------------------------------------------------- + box.wait_for_file("${statusFile}") + + box.succeed("stat -c %a ${statusFile} > /tmp/mode") + box.succeed("grep -qx 644 /tmp/mode") + + # jq -e exits non-zero on a false/null result, so this IS the assertion -- no pipe into grep, which + # under the driver's `set -o pipefail` could never have passed anyway. + box.succeed("jq -e '.schema == 1' ${statusFile} >/dev/null") + box.succeed("jq -e 'has(\"generated_at\") and has(\"vault\") and has(\"services\")' ${statusFile} >/dev/null") + + # --------------------------------------------------------------------------------------------- + # Absent unit -> "n/a", never "failed". This node runs neither mesh nor wisp, so those units do not + # exist at all. Reporting a non-existent subsystem as failed is the single most damaging lie this + # screen could tell: it sends an operator chasing a service the node was never configured to run. + # --------------------------------------------------------------------------------------------- + for field in ["mesh", "wisp"]: + box.succeed(f"jq -r '.services.{field}' ${statusFile} > /tmp/svc") + box.succeed("grep -qx 'n/a' /tmp/svc") + box.fail("grep -qx 'failed' /tmp/svc") + + # --------------------------------------------------------------------------------------------- + # Can we read the real screen? /dev/vcs1 is the kernel's live character-cell buffer for tty1. Under + # some headless configurations the kernel binds dummycon and this device is absent. This first + # read only proves it is READABLE -- vcs1 always returns a full screen of cells, so a non-empty + # result says nothing about content. The grep for painted text just below is the real proof, and + # every screen assertion in this test goes through the same device. + # --------------------------------------------------------------------------------------------- + box.wait_until_succeeds("cat /dev/vcs1 > /tmp/screen_probe; test -s /tmp/screen_probe", timeout=${toString slow}) + print("=== /dev/vcs1 initial contents ===") + print(box.succeed("cat /tmp/screen_probe")) + + def screen(path): + """Snapshot tty1's live character cells to a file inside the VM.""" + box.succeed(f"cat /dev/vcs1 > {path}") + + # The renderer paints the frame from the top of the screen, so a HEALTHY, non-stale frame must be + # visibly present before the stale test can prove it was REPLACED rather than merely absent. Both + # halves are waited on together so that one slow collection round early on is tolerated rather than + # latched as a failure. + box.wait_until_succeeds( + "cat /dev/vcs1 > /tmp/screen_healthy; " + "grep -q 'VAULT' /tmp/screen_healthy && ! grep -q 'STALE' /tmp/screen_healthy", + timeout=${toString slow}, + ) + box.succeed("grep -q 'ANTI-LOCKOUT' /tmp/screen_healthy") + + # --------------------------------------------------------------------------------------------- + # The anti-lockout verdict carries its own AGE. keep-node-admin-key-check is Type=oneshot + + # RemainAfterExit with no timer anywhere in the tree: it runs once at boot and latches "active" + # forever, so a key deleted after boot never re-triggers it. Without the age beside it, the row + # paints a green "ok" on a node whose admin SSH is already gone -- and on a display where every + # other field blanks to ?? the moment it stops being current, a non-?? value reads as "true right + # now". The age is what stops the row from wearing live-data clothing. + # + # keepNode.adminAccess is OFF on these nodes, so the unit is genuinely absent here. That is worth + # asserting in its own right: the field must be null rather than fabricating an age for a check + # that never ran. + box.succeed("jq -e 'has(\"anti_lockout_checked_at\")' ${statusFile} >/dev/null") + box.succeed("jq -e '.anti_lockout == \"n/a\" and .anti_lockout_checked_at == null' ${statusFile} >/dev/null") + box.succeed( + "KEEPNODE_STATUS_WIDTH=80 KEEPNODE_STATUS_ASCII=1 KEEPNODE_STATUS_NOCOLOR=1 TERM=linux " + "${render} --once --status-file ${statusFile} > /tmp/frame_al_absent" + ) + # No timestamp, no age claim -- and specifically no "checked 0s ago", which would be a lie about + # a check that has never run. + box.fail("grep -q 'checked' /tmp/frame_al_absent") + + # The rendering of a REAL, old verdict, driven from a fixture so the age is exact rather than + # whatever the VM's uptime happens to be. 41 days is chosen to be unmistakably not-now. + box.succeed( + "jq -n --argjson generated_at \"$(date +%s)\" " + "--argjson checked_at \"$(( $(date +%s) - 41 * 86400 ))\" " + "'{schema:1, generated_at:$generated_at, node:\"agetest\", " + "vault:{state:\"unlocked\",gate:\"active\"}, " + "services:{vaultwarden:\"active\",mesh:\"n/a\",wisp:\"n/a\"}, " + "mesh:{interface:null,up:false,address:null}, " + "replication:{role:null,lag_check:\"n/a\",lag_seconds:null}, " + "anti_lockout:\"active\", anti_lockout_checked_at:$checked_at}' > /tmp/al_old.json" + ) + # KEEPNODE_STATUS_UPTIME: the renderer refuses to show an age exceeding this boot's uptime, since + # the unit cannot have activated before the machine started. A 41-day age is therefore physically + # impossible on a VM minutes into its life, and without this seam the fixture exercises the clamp + # instead of the width behaviour it exists to test. The clamp itself is asserted separately below. + box.succeed( + "KEEPNODE_STATUS_UPTIME=9999999 " + "KEEPNODE_STATUS_WIDTH=80 KEEPNODE_STATUS_ASCII=1 KEEPNODE_STATUS_NOCOLOR=1 TERM=linux " + "${render} --once --status-file /tmp/al_old.json > /tmp/frame_al_old" + ) + print("=== frame with a 41-day-old anti-lockout verdict ===") + print(box.succeed("cat /tmp/frame_al_old")) + # The frame is otherwise healthy, so the age is the row's doing and not the staleness blanking. + box.fail("grep -q 'STALE' /tmp/frame_al_old") + box.succeed("grep -qE '\\[ ok \\] +ANTI-LOCKOUT +active \\(checked 41d ago\\)' /tmp/frame_al_old") + + # The age must be DROPPED WHOLE when it does not fit, never sliced into an unbalanced paren -- + # the SSH footer's discipline. At w=40 in the ascii tier the value column is 19 columns, which + # fits the short form but not the long one. + box.succeed( + "KEEPNODE_STATUS_UPTIME=9999999 " + "KEEPNODE_STATUS_WIDTH=40 KEEPNODE_STATUS_ASCII=1 KEEPNODE_STATUS_NOCOLOR=1 TERM=linux " + "${render} --once --status-file /tmp/al_old.json > /tmp/frame_al_narrow" + ) + print("=== narrow-tier anti-lockout age ===") + print(box.succeed("cat /tmp/frame_al_narrow")) + box.succeed( + "awk -v w=40 'length($0) > w { print \"OVERWIDE \" length($0) \": \" $0; bad=1 } " + "END { exit bad }' /tmp/frame_al_narrow" + ) + # Positive assertion FIRST. The three negative checks below all pass vacuously if the age feature + # broke entirely or the fixture aged past staleSeconds -- v_al_at blanks, no parenthetical is + # emitted, and "no half-written parenthetical" becomes trivially true. This pins the short form + # that is supposed to survive here, which is the actual "drop whole, do not slice" property. + box.succeed("grep -qF 'active (41d ago)' /tmp/frame_al_narrow") + # Whatever form survived, it is not a half-written parenthetical. + box.fail("grep -qE '\\((checked )?41d( ago)?$' /tmp/frame_al_narrow") + box.fail("grep -q '(checked$' /tmp/frame_al_narrow") + + # --------------------------------------------------------------------------------------------- + # The uptime clamp. ActiveEnterTimestamp is CLOCK_REALTIME at activation, so a box with no RTC + # (or with NTP landing after boot) jumps the clock forward and a check that ran seconds ago + # computes as decades. The unit cannot have activated before this boot, so an age exceeding + # uptime is impossible and must be DROPPED rather than painted -- a fabricated reading on the one + # row added to stop fabricated readings is the worst possible place to have one. + # + # Same 41-day fixture, real /proc/uptime this time (VM is minutes old), so the age is impossible. + box.succeed( + "KEEPNODE_STATUS_WIDTH=80 KEEPNODE_STATUS_ASCII=1 KEEPNODE_STATUS_NOCOLOR=1 TERM=linux " + "${render} --once --status-file /tmp/al_old.json > /tmp/frame_al_skew" + ) + print("=== impossible age (exceeds uptime) must be dropped ===") + print(box.succeed("cat /tmp/frame_al_skew")) + # The row still renders and still reads healthy -- only the implausible AGE is withheld. + box.succeed("grep -qE '\\[ ok \\] +ANTI-LOCKOUT +active *$' /tmp/frame_al_skew") + box.fail("grep -q '41d' /tmp/frame_al_skew") + box.fail("grep -q 'checked' /tmp/frame_al_skew") + + # --------------------------------------------------------------------------------------------- + # THE staleness property. Stop the collector and let the snapshot age past staleSeconds. Two + # separate things must hold, and the second is the one that matters: the banner lights, AND every + # collected value is REPLACED by "??". A greyed-out or last-known "unlocked" left on the glass is + # still read as "unlocked" by someone glancing across the room. + # --------------------------------------------------------------------------------------------- + box.succeed("systemctl stop keep-node-status-collect.timer") + box.succeed("systemctl stop keep-node-status-collect.service") + + box.wait_until_succeeds("cat /dev/vcs1 > /tmp/screen_stale; grep -q 'STALE' /tmp/screen_stale", timeout=${toString slow}) + + # No healthy reading survived anywhere on the screen. This is the assertion the whole feature + # exists to satisfy. + box.fail("grep -qE '(unlocked|active|RUNNING)' /tmp/screen_stale") + box.succeed("grep -q '??' /tmp/screen_stale") + + # The static config lines are NOT collected facts and must survive: they are what the operator + # standing at a screen full of ?? needs in order to go and fix it. + box.succeed("grep -q 'SSH' /tmp/screen_stale") + + # --------------------------------------------------------------------------------------------- + # Recovery. Without this, a STALE banner that was simply always-on would have passed everything + # above. + # --------------------------------------------------------------------------------------------- + box.succeed("systemctl start keep-node-status-collect.timer") + box.succeed("systemctl start keep-node-status-collect.service") + box.wait_until_succeeds("cat /dev/vcs1 > /tmp/screen_recovered; grep -q 'VAULT' /tmp/screen_recovered", timeout=${toString slow}) + box.wait_until_succeeds("cat /dev/vcs1 > /tmp/screen_recovered; ! grep -q 'STALE' /tmp/screen_recovered", timeout=${toString slow}) + + # --------------------------------------------------------------------------------------------- + # Degraded state propagates. A service going down must reach the glass within a couple of collector + # cycles, otherwise the screen is decorative. + # --------------------------------------------------------------------------------------------- + box.wait_until_succeeds("jq -e '.services.vaultwarden == \"active\"' ${statusFile} >/dev/null", timeout=${toString slow}) + box.succeed("systemctl stop vaultwarden.service") + box.wait_until_succeeds("jq -e '.services.vaultwarden == \"inactive\"' ${statusFile} >/dev/null", timeout=${toString slow}) + box.wait_until_succeeds("cat /dev/vcs1 > /tmp/screen_degraded; grep -q 'inactive' /tmp/screen_degraded", timeout=${toString slow}) + box.succeed("systemctl start vaultwarden.service") + + # --------------------------------------------------------------------------------------------- + # ActiveState=failed, end to end. Until now NO unit in this test was ever driven to `failed`, so + # the collector's unit_state -> "failed" mapping, the renderer's unit_status -> `fail`, and the + # `fail` glyph itself were entirely unexercised -- despite state mapping being the exact bug class + # that already regressed here once (a non-latching oneshot read as a raw ActiveState painted a + # healthy node permanently amber). + # + # A runtime drop-in under /run replaces a PROBED unit's ExecStart with a command that exits 1. + # It has to be a probed unit: the collector asks about a fixed list of unit names, so a standalone + # fixture unit would fail invisibly and prove nothing. `Restart=no` is part of the fixture -- with + # the packaged restart policy left in place the unit sits in "activating" (auto-restart) instead + # of settling at "failed", which is a different state and not the one under test. + box.succeed("mkdir -p /run/systemd/system/vaultwarden.service.d") + box.succeed( + "printf '[Service]\\nExecStart=\\nExecStart=/bin/sh -c \"exit 1\"\\nRestart=no\\n' " + "> /run/systemd/system/vaultwarden.service.d/99-fail-fixture.conf" + ) + box.succeed("systemctl daemon-reload") + box.succeed("systemctl restart vaultwarden.service || true") + box.wait_until_succeeds("systemctl is-failed --quiet vaultwarden.service", timeout=${toString slow}) + + # The collector maps it to "failed" -- and specifically NOT to "n/a", which is the mapping an + # absent unit gets and would quietly hide a genuinely broken service. + box.wait_until_succeeds("jq -e '.services.vaultwarden == \"failed\"' ${statusFile} >/dev/null", timeout=${toString slow}) + box.succeed("jq -r '.services.vaultwarden' ${statusFile} > /tmp/svc_failed") + box.fail("grep -qx 'n/a' /tmp/svc_failed") + + # ...and the renderer turns that into the `fail` glyph on the row. Read through --once in the + # ascii tier so the assertion is on plain text ("[FAIL]") rather than on the U+00D7 byte, which + # reading /dev/vcs1 would make brittle. The frame must be otherwise healthy, so this is the state + # mapping's doing and not the staleness blanking painting everything unknown. + box.succeed( + "KEEPNODE_STATUS_WIDTH=80 KEEPNODE_STATUS_ASCII=1 KEEPNODE_STATUS_NOCOLOR=1 TERM=linux " + "${render} --once --status-file ${statusFile} > /tmp/frame_failed" + ) + print("=== frame with a failed unit ===") + print(box.succeed("cat /tmp/frame_failed")) + box.fail("grep -q 'STALE' /tmp/frame_failed") + box.succeed("grep -qE '\\[FAIL\\] +VAULTWARDEN' /tmp/frame_failed") + box.fail("grep -qE '\\[ -- \\] +VAULTWARDEN' /tmp/frame_failed") + + # Restore, and prove the fail state was not sticky -- a `fail` glyph that never clears is as + # useless as one that never lights. + box.succeed("rm -f /run/systemd/system/vaultwarden.service.d/99-fail-fixture.conf") + box.succeed("systemctl daemon-reload") + box.succeed("systemctl reset-failed vaultwarden.service") + box.succeed("systemctl restart vaultwarden.service") + box.wait_until_succeeds("jq -e '.services.vaultwarden == \"active\"' ${statusFile} >/dev/null", timeout=${toString slow}) + + # --------------------------------------------------------------------------------------------- + # The no-input guarantee, checked MECHANICALLY rather than by reading the unit file. StandardInput + # = "null" is the single line that makes "physical console grants nothing" structural: with fd 0 on + # /dev/null there is no read path to escape from, no line discipline to poke, no "press any key". + # --------------------------------------------------------------------------------------------- + pid = box.succeed("systemctl show -p MainPID --value keep-node-status-render.service").strip() + assert pid.isdigit() and int(pid) > 0, f"renderer has no MainPID: {pid!r}" + box.succeed(f"readlink /proc/{pid}/fd/0 > /tmp/fd0") + box.succeed("grep -qx /dev/null /tmp/fd0") + + # The TIOCSTI question, VERIFIED rather than inferred. TIOCSTI ("push this byte back into the + # terminal's input queue") is the classic way a process holding a VT fabricates keystrokes for + # whatever reads that terminal next. The kernel requires a descriptor open for READING to issue + # it. The security review concluded by reading the unit file that the renderer holds no such + # descriptor; that conclusion is asserted here against the running process instead. + # + # Two independent properties, because either one alone leaves a gap: + # 1. NO descriptor on the VT is open in a readable mode. Not just fd 0: fd 1 is the terminal + # (StandardOutput=tty), and this proves the manager opened it O_WRONLY rather than O_RDWR. + # Access mode is the low two bits of the octal `flags:` field in fdinfo (0=RDONLY, 1=WRONLY, + # 2=RDWR), so a readable VT fd is anything whose accmode is not 1. + # 2. The process has NO controlling terminal, i.e. field 7 (tty_nr) of /proc/PID/stat is 0. + # Without one there is no /dev/tty to reopen -- which is how a process that was given only a + # write-only fd could otherwise get a fresh read-write one back. + vt = "/dev/tty1" + fds = box.succeed(f"ls /proc/{pid}/fd").split() + assert fds, f"renderer {pid} has no open descriptors at all" + saw_vt = False + for fd in fds: + # Target and flags in ONE round trip, both tolerant of the fd vanishing. The renderer runs + # command substitutions every repaintSeconds, each transiently opening a pipe fd, so an entry + # present in the `ls` above can be gone by the time a separate driver round trip reaches it. + # An unguarded awk exits 2 there and fails the test on a race rather than on a defect. + probe = box.succeed( + f"readlink /proc/{pid}/fd/{fd} 2>/dev/null || true; echo '|'; " + f"awk '/^flags:/{{print $2}}' /proc/{pid}/fdinfo/{fd} 2>/dev/null || true" + ) + target, _, flags = probe.partition("|") + target, flags = target.strip(), flags.strip() + if not target or not flags: + continue + accmode = int(flags, 8) & 3 + print(f"fd {fd} -> {target} flags={flags} accmode={accmode}") + if target == vt: + saw_vt = True + assert accmode == 1, ( + f"renderer fd {fd} is open on {vt} in a READABLE mode (accmode={accmode}, " + f"flags={flags}): TIOCSTI keystroke injection into this VT becomes possible, and " + f"'the console accepts no input' stops being a structural property" + ) + assert not (target == "/dev/tty"), f"renderer fd {fd} is on /dev/tty (a controlling terminal)" + + # Non-vacuity: if the renderer held NO descriptor on the VT at all, the loop above would pass + # while proving nothing. It must hold one (that is how it paints) -- just never a readable one. + assert saw_vt, f"renderer holds no descriptor on {vt}; the fd-mode assertion above was vacuous" + + # No controlling terminal. comm (field 2) can itself contain spaces and parens, so fields are + # counted from the LAST ')' -- after which stat's field 3 (state) is index 0, making tty_nr + # (field 7) index 4. + stat_line = box.succeed(f"cat /proc/{pid}/stat") + tty_nr = int(stat_line[stat_line.rindex(")") + 1:].split()[4]) + assert tty_nr == 0, ( + f"renderer has a controlling terminal (tty_nr={tty_nr}); it could reopen /dev/tty read-write " + f"and regain exactly the read path StandardInput=null exists to deny" + ) + + # The account holding the terminal cannot be logged into even if a path to it were found: no + # password (`!` is unmatchable) and a nologin shell. + # + # `su -s /bin/sh keep-status` is NOT a test of this and was wrong here first time round: the driver + # runs as root, so su skips authentication altogether, and -s overrides the very nologin shell that + # is doing the work. That command succeeds for ANY account on the box and asserts nothing. The two + # properties are checked directly instead. + box.succeed("getent passwd keep-status > /tmp/pw") + box.succeed("grep -q ':[^:]*/nologin$' /tmp/pw") + + # `!` cannot be produced by any hash function, so no supplied password can ever match it. This is + # what closes every PAM password path to the account, su included. + box.succeed("getent shadow keep-status > /tmp/shadow_ent") + box.succeed("cut -d: -f2 /tmp/shadow_ent > /tmp/hash") + box.succeed("grep -qx '!' /tmp/hash") + + # And with the account's OWN shell (no -s override), even root's passwordless su gets no session. + box.fail("su keep-status -c true") + + # --------------------------------------------------------------------------------------------- + # The getty is displaced AND does not come back. autovt@ is the easy one to miss: logind + # instantiates it on demand, so with it left enabled the moment anything calls chvt a fresh getty + # is spawned onto the terminal the renderer already owns and the two race for the device. Only a + # booted machine can show this. + # --------------------------------------------------------------------------------------------- + box.fail("systemctl is-active --quiet getty@tty1.service") + + box.succeed("chvt 1") + box.sleep(5) + box.fail("systemctl is-active --quiet getty@tty1.service") + box.fail("systemctl is-active --quiet autovt@tty1.service") + + # The renderer survived the VT switch and still owns the screen. + box.succeed("systemctl is-active --quiet keep-node-status-render.service") + box.wait_until_succeeds("cat /dev/vcs1 > /tmp/screen_postchvt; grep -q 'VAULT' /tmp/screen_postchvt", timeout=${toString slow}) + box.fail("grep -qi 'login:' /tmp/screen_postchvt") + + # tty2 keeps its getty on purpose (every account is password-locked, so it grants nothing, and + # debugAccess uses it during bring-up). Displacing tty1 must not have taken the others with it, + # and Alt+F2 is the ONLY console recovery path if the renderer dies -- tty1 has no getty left to + # fall back to. + # + # `systemctl cat getty@tty2.service` was the previous check and proved nothing: cat succeeds on + # any template unit regardless of enablement, and for a MASKED instance it happily prints the + # /dev/null symlink target. LoadState is the property that actually distinguishes them, because + # NixOS implements `enable = false` for a unit as exactly that /dev/null symlink, which systemd + # reports as "masked". So the two VTs must disagree here, and that disagreement is the assertion. + box.succeed("systemctl show -p LoadState --value getty@tty1.service > /tmp/getty1_load") + box.succeed("systemctl show -p LoadState --value getty@tty2.service > /tmp/getty2_load") + print("=== getty LoadState: tty1 / tty2 ===") + print(box.succeed("cat /tmp/getty1_load /tmp/getty2_load")) + box.succeed("grep -qx 'masked' /tmp/getty1_load") + box.fail("grep -qx 'masked' /tmp/getty2_load") + box.succeed("grep -qx 'loaded' /tmp/getty2_load") + + # And the same for autovt@, the one that is easy to miss: logind instantiates it on demand, so if + # tty2's were masked too, Alt+F2 would silently produce nothing at the keyboard. + box.succeed("systemctl show -p LoadState --value autovt@tty1.service > /tmp/autovt1_load") + box.succeed("systemctl show -p LoadState --value autovt@tty2.service > /tmp/autovt2_load") + box.succeed("grep -qx 'masked' /tmp/autovt1_load") + box.fail("grep -qx 'masked' /tmp/autovt2_load") + + # --------------------------------------------------------------------------------------------- + # Escape injection, end to end. The collector is trusted and the renderer sanitises anyway: the + # difference between "the screen is safe because the producer behaves" and "the screen is safe + # whatever the producer emits". The fixture carries a REAL ESC byte, not the text "\\033". + # --------------------------------------------------------------------------------------------- + # The fixture is built BY jq from a shell string holding a real ESC, so the file is VALID JSON that + # encodes the escape as . A raw ESC byte written straight into the file would instead be + # illegal JSON (control characters below 0x20 may not appear unescaped in a JSON string): jq would + # reject the whole snapshot and the renderer would blank to ?? via the malformed-JSON path, which + # tests the wrong thing entirely and never exercises the sanitiser at all. + box.succeed( + "jq -n --argjson generated_at \"$(date +%s)\" " + "--arg node \"$(printf 'AAA\\033[2JBBB')\" " + "'{schema:1, generated_at:$generated_at, node:$node, " + "vault:{state:\"unlocked\",gate:\"active\"}, " + "services:{vaultwarden:\"active\",mesh:\"n/a\",wisp:\"n/a\"}, " + "mesh:{interface:null,up:false,address:null}, " + "replication:{role:null,lag_check:\"n/a\",lag_seconds:null}, " + "anti_lockout:\"active\"}' > /tmp/evil.json" + ) + + # Vacuity guards on the fixture itself. It must parse... + box.succeed("jq -e . /tmp/evil.json >/dev/null") + # ...must carry no raw ESC on disk (it is -encoded)... + box.succeed("LC_ALL=C tr -cd '\\033' < /tmp/evil.json > /tmp/file_esc") + box.succeed("test ! -s /tmp/file_esc") + # ...and must nonetheless DECODE to a value containing a real ESC byte, which is what actually + # reaches the renderer's variable. Without this check a fixture that quietly lost its escape would + # make the assertion below pass while proving nothing. + box.succeed("jq -r '.node' /tmp/evil.json > /tmp/node_raw") + box.succeed("LC_ALL=C tr -cd '\\033' < /tmp/node_raw > /tmp/node_esc") + box.succeed("test -s /tmp/node_esc") + + box.succeed( + "KEEPNODE_STATUS_WIDTH=80 KEEPNODE_STATUS_NOCOLOR=1 TERM=linux LANG=C.UTF-8 " + "${render} --once --status-file /tmp/evil.json > /tmp/frame_evil" + ) + # Not one ESC byte reaches the glass. + box.succeed("LC_ALL=C tr -cd '\\033' < /tmp/frame_evil > /tmp/frame_esc") + box.succeed("test ! -s /tmp/frame_esc") + + # The frame is otherwise HEALTHY (so this is the sanitiser's doing, not the blanking path) and the + # payload around the escape survived as inert text: the clear-screen sequence was defanged into + # characters, not obeyed and not silently swallowed whole. + box.fail("grep -q 'STALE' /tmp/frame_evil") + box.succeed("grep -q 'AAA' /tmp/frame_evil") + box.succeed("grep -q 'BBB' /tmp/frame_evil") + box.succeed("grep -qF '[2J' /tmp/frame_evil") + + # --------------------------------------------------------------------------------------------- + # Degradation tiers, through the --once path (the SAME build_frame the loop runs). + # --------------------------------------------------------------------------------------------- + + # ASCII tier: not one byte >= 0x80. This is the fallback for serial consoles, vt100 and non-UTF-8 + # locales, where the block-glyph brand mark would arrive as mojibake. + box.succeed( + "KEEPNODE_STATUS_WIDTH=80 KEEPNODE_STATUS_ASCII=1 KEEPNODE_STATUS_NOCOLOR=1 TERM=linux " + "${render} --once --status-file /tmp/evil.json > /tmp/frame_ascii" + ) + box.succeed("LC_ALL=C tr -d '\\000-\\177' < /tmp/frame_ascii > /tmp/frame_high") + box.succeed("test ! -s /tmp/frame_high") + + # No-colour tier: no ESC[ sequences at all, for a monochrome LCD, a serial capture or a photograph. + # State is triple-coded (glyph + word + colour) precisely so it survives this. + box.succeed("LC_ALL=C tr -cd '\\033' < /tmp/frame_ascii > /tmp/ascii_esc") + box.succeed("test ! -s /tmp/ascii_esc") + + # Width budget at every tier. 80 is the tier the SSH footer used to overflow by exactly one column, + # which cut its closing paren off mid-token. + for width in [40, 60, 80, 100]: + box.succeed( + f"KEEPNODE_STATUS_WIDTH={width} KEEPNODE_STATUS_NOCOLOR=1 TERM=linux LANG=C.UTF-8 " + f"${render} --once --status-file /tmp/evil.json > /tmp/frame_{width}" + ) + box.succeed( + f"awk -v w={width} 'length($0) > w {{ print \"OVERWIDE \" length($0) \": \" $0; bad=1 }} " + f"END {{ exit bad }}' /tmp/frame_{width}" + ) + + # The full tier carries the SSH note, and it must be INTACT -- balanced parens, not a mid-word cut. + box.succeed("grep -q 'no console login)' /tmp/frame_80") + box.succeed("grep -q 'no console login)' /tmp/frame_100") + + # --------------------------------------------------------------------------------------------- + # The DEGRADED paths, through --once --status-file. Only the timer-driven STALE path had ever been + # covered; the missing-file, malformed-JSON, wrong-schema and collector-fault paths were all + # unexercised, even though each one is a distinct banner and each one is what stands between the + # operator and a plausible-looking frame full of numbers that are not true. + # + # Every one of them must satisfy the SAME two properties as the STALE path: the banner names the + # reason, and every collected value is blanked to "??". + # --------------------------------------------------------------------------------------------- + def degraded_frame(name, setup, expect, width=80): + if setup: + box.succeed(setup) + box.succeed( + f"KEEPNODE_STATUS_WIDTH={width} KEEPNODE_STATUS_ASCII=1 KEEPNODE_STATUS_NOCOLOR=1 " + f"TERM=linux ${render} --once --status-file /tmp/deg_{name}.json > /tmp/frame_deg_{name}" + ) + print(f"=== degraded frame: {name} ===") + print(box.succeed(f"cat /tmp/frame_deg_{name}")) + # The banner names the reason... + box.succeed(f"grep -qF '{expect}' /tmp/frame_deg_{name}") + # ...and nothing collected survived. The unlocked/active greps are the ones that matter: a + # last-known reading left on the glass is read as current by anyone glancing at it. + box.succeed(f"grep -q '??' /tmp/frame_deg_{name}") + box.fail(f"grep -qE '(unlocked|active)' /tmp/frame_deg_{name}") + # The static SSH line is not a collected fact and must survive -- it is how the operator + # standing at a screen full of ?? gets to a shell to fix it. + box.succeed(f"grep -q 'SSH' /tmp/frame_deg_{name}") + + # 1. No snapshot at all: the collector has never run, or /run was wiped. + box.succeed("rm -f /tmp/deg_missing.json") + degraded_frame("missing", None, "NO STATUS SNAPSHOT") + + # 2. Unparseable bytes where JSON should be (a truncated write, a half-written file). + degraded_frame("malformed", "printf 'this is not json{' > /tmp/deg_malformed.json", "MALFORMED JSON") + + # 3. A snapshot from a FUTURE schema. Parsing it against schema 1's field vocabulary would slide + # values into the wrong rows, which is the failure mode that looks entirely plausible. + degraded_frame( + "schema", + "jq -n --argjson generated_at \"$(date +%s)\" " + "'{schema:2, generated_at:$generated_at, node:\"future\"}' > /tmp/deg_schema.json", + "UNSUPPORTED", + ) + + # 4. The collector's OWN emit failure. This is the subtle one: the fallback payload carries a + # CURRENT generated_at, so staleness can never fire on it, and until the renderer read `.error` + # the result was a completely normal-looking frame -- hostname painted, every row "unknown", + # no banner anywhere. The comment in the collector claimed a fault was reported; nothing read + # the field. + degraded_frame( + "collector", + "jq -n --argjson generated_at \"$(date +%s)\" " + "'{schema:1, generated_at:$generated_at, node:null, error:\"collector-emit-failed\"}' " + "> /tmp/deg_collector.json", + "COLLECTOR FAULT", + ) + + # Non-vacuity for the collector-fault case specifically: prove the snapshot is NOT stale, so the + # banner above came from reading .error and not from the age check firing by luck. The bound is + # 15s, not a tight few: several renders run between the fixture's `date +%s` stamp and this + # check, which on a contended hypervisor overran a 5s window. staleSeconds is 30 on this node, + # so 15 still proves staleness did not fire. + box.succeed("jq -e '(now - .generated_at) < 15' /tmp/deg_collector.json >/dev/null") + + # --------------------------------------------------------------------------------------------- + # Banner text WRAPS, never slices. At w=40 the inner width is 34 and the notice is 41 characters, + # so a hard cut produced "...AND SHO" -- dropping the very "??" the sentence exists to explain. + # Asserted at the narrowest tier, which is the only one where it bites. + # --------------------------------------------------------------------------------------------- + box.succeed( + "KEEPNODE_STATUS_WIDTH=40 KEEPNODE_STATUS_ASCII=1 KEEPNODE_STATUS_NOCOLOR=1 TERM=linux " + "${render} --once --status-file /tmp/deg_malformed.json > /tmp/frame_banner40" + ) + print("=== narrow-tier banner wrap ===") + print(box.succeed("cat /tmp/frame_banner40")) + # Still inside the width budget after wrapping... + box.succeed( + "awk -v w=40 'length($0) > w { print \"OVERWIDE \" length($0) \": \" $0; bad=1 } " + "END { exit bad }' /tmp/frame_banner40" + ) + # ...the sentence survived WHOLE across the wrap, including its trailing ?? ... + box.succeed("grep -q 'SHOW AS ??' /tmp/frame_banner40") + # ...and no line ends mid-word on the word the old hard cut split. + box.fail("grep -q 'AND SHO *!!' /tmp/frame_banner40") + + # --------------------------------------------------------------------------------------------- + # The REPLICATION row, which has never been rendered, and the collector's lag_seconds journal + # parsing, which has never been exercised. Both are gated on vaultReplication.role. + # --------------------------------------------------------------------------------------------- + standby.wait_for_file("${statusFile}") + standby.succeed("jq -e '.replication.role == \"standby\"' ${statusFile} >/dev/null") + + # Stand in for the heartbeat the active would have pushed into replicaDir, then run the check so it + # logs the one line the collector's journal parse is looking for. + standby.succeed("install -d -o vaultwarden -g vaultwarden -m 0700 ${replicaDir}") + standby.succeed("date +%s > ${replicaDir}/.push-heartbeat") + standby.succeed("chown vaultwarden:vaultwarden ${replicaDir}/.push-heartbeat") + standby.succeed("systemctl start keep-node-vault-lag-check.service") + + # The unit logged a lag line... + standby.succeed("journalctl -u keep-node-vault-lag-check.service -o cat --no-pager > /tmp/lag_log") + standby.succeed("grep -q '^vault replication lag: ' /tmp/lag_log") + + # ...and the collector recovered the NUMBER behind it into the snapshot. This is the parse that has + # never run before. + standby.wait_until_succeeds("jq -e '.replication.lag_seconds != null' ${statusFile} >/dev/null", timeout=${toString slow}) + standby.succeed("jq -e '.replication.lag_seconds >= 0' ${statusFile} >/dev/null") + + # A HEALTHY standby must read OK. keep-node-vault-lag-check is Type=oneshot with NO + # RemainAfterExit, so a SUCCESSFUL run leaves the unit "inactive" and its ActiveState carries no + # pass signal at all; nixos/vault-replication.nix:628 states that `systemctl is-failed` is the + # monitoring signal instead. Reading it as a raw ActiveState painted a permanent amber + # REPLICATION row on a healthy node, which is the cry-wolf failure this must never regress into. + # + # The heartbeat is RE-STAMPED on every attempt rather than written once: it goes stale after + # maxLagSeconds (90s), so a single fixture stamp would make this a race against the 30s timer on a + # contended builder. Each attempt refreshes the stamp, runs the check, and then waits one + # collector round (refreshSeconds=5) for the verdict to reach the snapshot. + def restamp_and_check(extra): + standby.wait_until_succeeds( + "date +%s > ${replicaDir}/.push-heartbeat && " + "chown vaultwarden:vaultwarden ${replicaDir}/.push-heartbeat && " + "systemctl start keep-node-vault-lag-check.service && " + "sleep 7 && " + extra, + timeout=${toString slow}, + ) + + restamp_and_check("jq -e '.replication.lag_check == \"ok\"' ${statusFile} >/dev/null") + + # Non-vacuity guard on the assertion above: it only proves anything if the unit really is sitting + # at the non-latched "inactive" that used to render amber. If systemd ever started latching this + # unit "active", the ok verdict would be trivially true and this row would stop testing the bug. + standby.succeed("systemctl show -p ActiveState --value keep-node-vault-lag-check.service > /tmp/lagstate") + print("=== observed lag-check ActiveState ===") + print(standby.succeed("cat /tmp/lagstate")) + standby.succeed("grep -qx 'inactive' /tmp/lagstate") + + # ...and the OK state reaches the RENDERED row, not just the JSON. Asserted through --once in the + # ascii/no-colour tier so the verdict is plain text ("[ ok ]") rather than a colour code or a + # console-mapped glyph byte that reading /dev/vcs1 would make brittle to grep for. + restamp_and_check( + "KEEPNODE_STATUS_WIDTH=80 KEEPNODE_STATUS_ASCII=1 KEEPNODE_STATUS_NOCOLOR=1 TERM=linux " + "${renderStandby} --once --status-file ${statusFile} > /tmp/frame_repl && " + "grep -qE '\\[ ok \\] +REPLICATION' /tmp/frame_repl" + ) + print("=== rendered standby frame ===") + print(standby.succeed("cat /tmp/frame_repl")) + standby.fail("grep -qE '\\[warn\\] +REPLICATION' /tmp/frame_repl") + + # And the row actually reaches the glass, with the lag rendered into it. + # Wait on the LAG text, not on the word REPLICATION: the row is painted either way (it reads + # "standby " when no lag number is available), so waiting on the label alone matches a + # frame rendered before the heartbeat existed and proves nothing about the parse. + standby.wait_until_succeeds( + "cat /dev/vcs1 > /tmp/screen_repl; grep -qE 'standby lag [0-9]+s' /tmp/screen_repl", + timeout=${toString slow}, + ) + standby.succeed("grep -q 'REPLICATION' /tmp/screen_repl") + + # --------------------------------------------------------------------------------------------- + # The OTHER half of the verdict mapping: check_unit_state -> "failed" -> check_status -> `fail`. + # Everything above only ever drove this unit through its healthy "inactive" state, so the failing + # branch of the non-latching-oneshot reading had never executed. That branch is the whole reason + # this unit is read via `is-failed` semantics instead of as a raw ActiveState, so leaving it + # untested left the interesting half of the fix unguarded. + # + # Same runtime drop-in technique as the vaultwarden fixture on box: replace ExecStart with a + # command that exits 1 and let the unit settle at failed. + standby.succeed("mkdir -p /run/systemd/system/keep-node-vault-lag-check.service.d") + standby.succeed( + "printf '[Service]\\nExecStart=\\nExecStart=/bin/sh -c \"exit 1\"\\n' " + "> /run/systemd/system/keep-node-vault-lag-check.service.d/99-fail-fixture.conf" + ) + standby.succeed("systemctl daemon-reload") + standby.succeed("systemctl start keep-node-vault-lag-check.service || true") + standby.wait_until_succeeds("systemctl is-failed --quiet keep-node-vault-lag-check.service", timeout=${toString slow}) + + # The collector emits the VERDICT vocabulary here ("failed"), not a raw ActiveState. + standby.wait_until_succeeds("jq -e '.replication.lag_check == \"failed\"' ${statusFile} >/dev/null", timeout=${toString slow}) + + standby.succeed( + "KEEPNODE_STATUS_WIDTH=80 KEEPNODE_STATUS_ASCII=1 KEEPNODE_STATUS_NOCOLOR=1 TERM=linux " + "${renderStandby} --once --status-file ${statusFile} > /tmp/frame_repl_failed" + ) + print("=== standby frame with a failed lag check ===") + print(standby.succeed("cat /tmp/frame_repl_failed")) + standby.fail("grep -q 'STALE' /tmp/frame_repl_failed") + standby.succeed("grep -qE '\\[FAIL\\] +REPLICATION' /tmp/frame_repl_failed") + # Not amber and not "absent": those are the two mappings this row must never collapse into. + standby.fail("grep -qE '\\[warn\\] +REPLICATION' /tmp/frame_repl_failed") + standby.fail("grep -qE '\\[ -- \\] +REPLICATION' /tmp/frame_repl_failed") + + standby.succeed("rm -f /run/systemd/system/keep-node-vault-lag-check.service.d/99-fail-fixture.conf") + standby.succeed("systemctl daemon-reload") + standby.succeed("systemctl reset-failed keep-node-vault-lag-check.service") + ''; +}