box: measure and reduce protected memory overhead

[pythonlearner1025]

Problem

The box memory boundary deliberately protects availability, but its current defaults leave a meaningful amount of VM RAM unavailable to user workloads, while some always-on box infrastructure may be paying avoidable runtime overhead.

For VM RAM R, the current hierarchy is approximately:

• host reserve: 512 MiB
• box container: R - 512 MiB
• blitz-system.slice: memory.min=256 MiB, memory.low=384 MiB
• internal headroom: 256 MiB
• blitz-user.slice memory.max: R - 1024 MiB
• memory.high: another 500 MiB below the user hard limit

The sizing rationale says the measured protected set was about 85 MiB and that 256 MiB was selected as roughly 3x that set. That safety margin was appropriate when introducing the boundary, but we should determine whether implementation improvements and new load measurements can safely return 128–256 MiB or more to user work.

One visible candidate is the base-owned blitz-payload updater. It is a permanent Node process even while idle. Most of its robustness comes from its transaction protocol—staged extraction, hash verification, atomic renames, durable pending state, health-gated commit, rollback, and s6 supervision—not specifically from Node. The repo already has a pinned, static Go build chain for blitz-cred and blitz-box-gateway, so a behavior-compatible Go updater may reduce idle RSS without adding a new toolchain.

Other candidates worth measuring include persistent Node helpers such as blitz-lody-projects, plus the aggregate baseline of cloudflared, ttyd, dufs, gateway, s6, and Lody processes. We currently lack a checked-in idle PSS/RSS budget by process and cgroup, so optimization decisions are estimates rather than evidence.

Proposed solution

Treat this as a measurement-first memory-budget project rather than immediately lowering cgroup limits.

1. Establish a reproducible baseline

   • Record RSS and preferably PSS for every persistent process on representative Hetzner machine sizes.
   • Record memory.current, memory.stat, memory.events, and pressure data for the system and user slices.
   • Separate host, protected system, idle user infrastructure, and actual user workload consumption.
   • Add a repeatable report to packages/box/test/ so future changes can be compared.

2. Remove avoidable always-on runtime overhead

   • Prototype blitz-payload in Go while preserving the existing wire formats, on-disk state, lock path, crash boundaries, rollback semantics, and base-owned recovery floor.
   • Run the existing conformance and crash-injection suite against both implementations before switching.
   • Evaluate whether blitz-lody-projects should be a smaller binary or a supervised periodic one-shot rather than a permanently resident Node process.
   • Prioritize changes by measured PSS saved, not executable size.

3. Re-size the boundary using the existing Hetzner load campaign

   • First test BLITZ_CG_SYSTEM_MIN=192M and BLITZ_CG_HEADROOM=128M (returns 192 MiB to user.max).
   • If green, test SYSTEM_MIN=128M and HEADROOM=128M (returns 256 MiB).
   • Do not reduce the separate 512 MiB host reserve until host kernel, sshd, dockerd, and blitz-box-update usage is independently measured under pressure.
   • Revisit the 500 MiB memory.high gap separately: moving the throttle point and moving the hard ceiling solve different problems.

4. Ratchet the result

   • Document an explicit protected-system memory budget.
   • Add a regression threshold for idle system memory.
   • Update docs/MEMORY-BOUNDARY.md with measured results and final defaults.

Acceptance criteria

• Reproducible idle and pressured memory reports exist for supported Hetzner sizes.
• The largest persistent consumers are identified using PSS/RSS rather than estimates.
• Any updater rewrite passes the existing payload fixture, health, rollback, and crash-point tests with the same state format.
• The real Hetzner memory-load campaign remains green with the proposed lower reserves.
• At least 128 MiB is safely returned to blitz-user.slice memory.max, or the measurements document why current defaults remain necessary.
• System services remain responsive during reclaim and no system-slice OOM kills occur.