feat(backup): fleet backup engine

AUDIT_OPEN.md

@@ -1,5 +1,130 @@
 # Open audit items
 
+## 2026-11-05 — Dash auto-config Slice 4 CLI: fleet backup engine (G1, G3, G4, G5)
+
+Plan: `../_internal/DASH_AUTOCONFIG_PLAN_2026-11-03.md` (sections 2.1-2.3, 5). Branch
+`slice4-fleet-engine`. Closes the Slice 2 opens on G5/one-object-per-database, role
+hashes unencrypted in sidecars, and `backup list` having no rows for `backup_databases:`.
+
+Decisions worth reviewing:
+- `max_age_days` is a keep rule (restic keep-within), unioned with the rest, not a cap:
+  it is what it already was with `keep_last`, and a cap could not express deploy-ovh's
+  "hourly for 48h, then daily/weekly/monthly". `max_age 2 + daily 7 + weekly 4 + monthly 12`
+  is that policy, with no separate hourly rule.
+- The recovery key lives in the policy, not the target: Dash forgets bucket credentials,
+  and setting a target needs them.
+- Upload commit is `.incoming/` + verify + server-side copy, sidecars first, primary last,
+  rather than a manifest: every existing reader already treats "the primary object exists"
+  as the backup existing, and legacy objects have no manifest to find.
+- `--all` skips targets with a `run`/`legacy` cron line (override) but not `custom` ones:
+  a hand-written job is not evidence of anything, and policy coverage is how they retire.
+- `.owners/<fingerprint>` markers refuse a prune in a namespace another server wrote to,
+  because a reused prefix or a same-named app is the failure a prefix alone cannot catch.
+
+Independent review (2026-11-05, same day) reproduced real holes and all are fixed on the
+branch: purge and upgrade backed up only the primary (now every database, refusing what
+it cannot cover); a leftover backup-schedule.json decided a policy run's retention; a run
+that stored nothing still pruned and reported ok; a database's objects followed the
+primary's ids (each series is selected on its own now); the .owners guard failed open on
+a listing error, an empty machine-id and a late marker; status never looked at the
+bucket; owner-key recovery could not work through normal commands; a non-root user reading
+a 0700 /etc/teploy saw "no policy" and uploaded plaintext; per-database rows reached Dash,
+whose restore by id would have replaced the primary. Medium/low: refused prunes invisible,
+pre-restore not forwarding --ignore-space-check, zero targets reading as ok, .incoming only
+at one depth, a docker blip reading as no container, a wrong clock deleting real backups,
+--previous-prefix-owned marking every earlier prefix, AWS's bundle unverified, a 6h deadline
+with no override, legacy lines reported encrypted, ClickHouse and friends in no report, a
+--no-credentials target never working, --stdin dropping the recipient silently, bucket-wide
+multipart abort, TEPLOY_BACKUP_TMPDIR workspaces invisible, postgresql:// names, a stale
+holder's children, self-referential checksums above 256 MiB, safety copies prunable.
+
+Verified live (colima, MinIO via the real aws CLI, real age, the real linux binary):
+`go test -tags integration -run TestFleetIntegration ./internal/cli`, 25 scenarios, on real
+multi-database accessories; `TEPLOY_IT_INSTALL_AWS=1 go test -tags integration -run
+TestAWSCLIInstallScriptLive ./internal/backup` installs AWS's bundle for real and refuses a wrong
+pin. Only unit-tested: the deploy hook, healthchecks.io's /fail form, the janitors' thresholds
+(the expiry check ran in a real shell, not on a live run) and multipart abort, the space preflights'
+refusal, a bucket refusing deletes, a store without upload checksums, a musl host, the clock checks
+and an empty machine-id, an old aws CLI on a server and an old distribution package being replaced,
+a previous prefix that does not answer, the stuck-holder kill order (the kill itself ran live),
+setup's stage itself (the script it runs is the live one), the /etc/teploy root path (the fixture
+is non-root; the non-root paths, both with sudo and without, ran live).
+Regressions: `go test -tags integration ./internal/backup`, the full engine matrix after merging
+v0.1.51 (postgres, pgvector, timescale, mysql, mariadb, mongo, redis, valkey, dragonfly). The DR
+fixture still needs a 022 umask.
+
+Re-review (2026-11-05, 16 items), all on the branch. Blocker and highs: a publish copy of 8 MB
+or more becomes UploadPartCopy, which B2 documents cannot change a checksum algorithm, and R2
+cannot do with its default tag read (`--copy-props metadata-directive`, then a re-upload of the
+verified local file on any refusal); pending uploads at every depth billed forever on a store
+that keeps versions (one `.incoming/` tree, doctor reads the lifecycle rule, the marker is not
+rewritten); a sudoer read /etc and wrote ~/.config, so rotation did nothing (one path, a read-back
+in `WriteFleetPolicy`, and the lock falls back). Medium and low: `owners --clear` on a live marker,
+a prune trusting a same-host clock and skipping an unreadable one, a janitor fixed at 7h, uncapped
+safety copies, an unreachable previous prefix staling everything, the wrong chmod 711 remedy,
+future-dated objects, the checksum head with no fallback, `--exclude` unnamed, no minimum aws
+version, the kill order on a stuck holder. Decisions: the clone and renamed-host limits of the
+fingerprint are documented, not papered over (an id file is copied by a clone too); marker recency
+is the namespace's newest write, since a marker that is not rewritten says nothing about liveness;
+the minimum aws is 2.23.0/1.37.0 from aws-cli's own changelog (first default checksum on upload).
+
+Proven only against MinIO, not real B2 or R2: every publish path. The refusal is simulated by a
+shim, with an assumed error text, and a re-upload follows anything shaped like a store's error
+reply rather than one exact string for that reason. Also unverified on B2: whether its S3 API
+answers `GetBucketLifecycleConfiguration` (if not, doctor warns "unknown", which is true and
+permanent there), whether `--copy-props metadata-directive` carries metadata through its multipart
+copy, and Object Lock with aws 2.23+. The real-B2 smoke test is the coordinator's.
+
+Open / deferred:
+- The run lock falls back per user when the policy's directory is not writable, so a root run
+  and a sudoer's run do not exclude each other. Only one of them has the cron line.
+- Legacy objects from a server that has not upgraded yet carry no marker. A first prune in
+  a shared root by an upgraded server cannot tell them from its own: only markers from the
+  moment each server upgrades protect it. Per-server prefixes close this (decided: Dash always
+  sets one): the set command warns when there is none, and doctor flags a policy with no prefix
+  once another server's marker is at the bucket root. Attributing unmarked objects would stop a
+  single server's own legacy history ever ageing out.
+- A dropped database in an accessory whose other databases are fine keeps its series (its
+  newest good object forever) and raises nothing. It reads as history, not as a problem.
+- An external kill -9 of a run leaves its children (own process groups) running until they
+  finish; they cannot block the next run, and a stale holder's are killed. A Pdeathsig
+  would reach only the shell.
+- AWS's signing key is pinned (expires 2027-07-01; old signatures stay valid). A rotated
+  key makes setup refuse the bundle until the pin is updated: the safe way to fail.
+- Unsupported databases are recognised by image name only; a renamed image reads as generic
+  (and a labelled accessory still hits the live-data guard).
+- A schema name holding `'` or `\`: `mysqldump` mangles it in its own `USE` (seen live on
+  mysql:8), so it is reported as not backed up in omissions and status, never silently.
+  A Postgres or MySQL name with `=`, a leading `-`, a control character or over 64 bytes
+  likewise. Fixing needs dumping through `--databases`-free stdin tricks; not worth it yet.
+- Adopted MySQL dumps schemas, not `mysql.user`: per-site database users must be recreated
+  when recovering `wordpress-shared-mysql` onto a new server. The postgres side has globals.
+- An adopted container's password comes from its env. A root password changed after creation
+  (deploy-ovh's was stale) fails the dump loudly. There is no policy credential yet.
+- Mongo with auth still fails loudly (Slice 2 item), now as a failing target.
+- `aws` stays the transport (G8 deferred). Credential charset is restricted as before.
+- Restore and verify of an adopted database stop nothing; the operator stops its users.
+- Volume and generic estimates use the live tree's size, so a small disk can refuse an
+  archive that would have compressed to fit (`--ignore-space-check`, or a larger
+  `TEPLOY_BACKUP_TMPDIR`). The previous object's size would sharpen this.
+- A provider's own checksum is not verified beyond the 256 MiB read-back: above that it is
+  stored size plus the recorded SHA-256 plus the aws CLI's transfer checksums.
+- Heartbeat redirects are followed by Go's default client. Dash's push monitor does not
+  redirect; a receiver that does would see the token on the redirect target.
+- Dash must read `backup-fleet-v1` before using any of this (plan: version-gated). Behavior
+  Dash and scripts see: `remove --purge` now needs a target or `--no-backup`; restores take a
+  backup first; a `backup_databases:` name that is missing now fails that accessory's backup.
+- Pre-existing, seen live: `accessory upgrade` stops and removes the container before
+  `EnsureRunning`, so a failure there (an unsafe directory mode, here) leaves the accessory
+  down until re-run. Its data and the new safety backup are intact. Not changed in this slice.
+- `AGENTS.md` points at a repo `CLAUDE.md` that does not exist.
+- `backup status` runs one bucket listing per app and adopted namespace, and reads each
+  app's event log. Fine at fleet sizes here; a very large bucket would want a cache.
+- Encrypted objects cannot be restored by a CLI older than this one: the `.age` suffix is
+  invisible to its listing. Release order: CLI on servers first (plan section 4).
+- `TEPLOY_SERVER_BINARY` was used in the live test; on real servers `ensureFleetBinary`
+  upgrades via the release pipeline, so the policy cannot be set until this ships.
+
 ## 2026-11-05 — Dash auto-config Slice 5: domain health (CLI side)
 
 Plan: `../_internal/DASH_AUTOCONFIG_PLAN_2026-11-03.md` section 8. Branch
@@ -105,8 +230,8 @@ its marker table; checks run before the app stops; `database` in the restore
 event; `backup list` works without cron.
 
 Open / deferred:
-- G5 adoption of unmanaged containers and MySQL one-object-per-database: not
-  in this slice. MySQL/MariaDB backups cover one schema and warn about the rest.
+- ~~G5 adoption of unmanaged containers and MySQL one-object-per-database~~ closed
+  by Slice 4.
 - A scheduled backup alerts on coverage only where `backup schedule` has
   uploaded the current script (rescheduling updates it). Dash alerting: Slice 4.
 - Safety copies are not rolled back automatically for mysql/mongo/dragonfly:
@@ -117,10 +242,9 @@ Open / deferred:
 - Mongo restore (--drop) leaves collections created after the backup.
 - verify-backup covers the primary database only, not backup_databases: dumps;
   DR bundles carry the primary only (their manifest notes what is missing).
-- `backup list` shows one row per id; backup_databases: dumps have no rows of
-  their own.
-- Role password hashes are in the postgres globals sidecar and DR member,
-  unencrypted until G1 (the dump beside them is too).
+- ~~`backup list` one row per id~~ closed by Slice 4 (`database` on each row).
+- ~~Role password hashes unencrypted in sidecars~~ closed by G1 when a recovery key
+  is configured.
 - Redis/valkey restore and verify assume /data/dump.rdb (Bitnami redis differs).
 - Dragonfly's pre-restore snapshot in /var/tmp has no free-space preflight
   (its size is the in-memory dataset).

CHANGELOG.md

@@ -4,6 +4,99 @@ All notable changes to teploy are documented here. Format follows [Keep a Change
 
 ## [Unreleased]
 
+### Added
+
+- Server backup policy: `teploy server backup-policy set/show/unset`, one cron
+  line, `teploy backup run --all`, enumerating at run time every labelled
+  accessory and each database in it, app volumes and adopted containers. A new
+  database is covered with no command. An overlap lock (an `flock`: a killed run
+  cannot block the next), per-target and per-run deadlines, one target failing
+  does not stop the others, and the exit status and `backup.all` event report
+  `ok`/`partial`/`failed` plus every omission. A per-app schedule stays an override.
+- Adopted containers: policy entries `{container, engine, databases: all|[...]}`
+  bring unlabelled containers (`omni_postgres`, `wordpress-shared-mysql`) under the
+  policy, addressed by docker name (`--adopted`) in `backup restore/list/create/prune`
+  and `accessory verify-backup`.
+- One object per database. MySQL/MariaDB schemas and Postgres databases each get
+  their own series (`databases/<name>/`), restored and verified alone with
+  `--database`; `backup_databases:` now works on MySQL too.
+- GFS retention: `--keep-daily/-weekly/-monthly` beside `--keep-last` and
+  `--max-age-days`, a union of rules with restic `forget` semantics. `--max-age-days`
+  is restic's keep-within, which is also what it always was with `--keep-last`.
+- Payload encryption: with a recovery key (`teploy backup keygen`; `backup-policy set
+  --recipient`), every object is age-encrypted to the owner key and the server's
+  identity, sidecars and DR members included. `teploy backup fetch <key> --identity`
+  downloads and decrypts locally, for the recovery drill and a lost server.
+- Heartbeats: `heartbeat_urls` are pinged after every `--all` run, with
+  `?status=down` on anything but a clean run. Policy-managed backups do not use the
+  per-app alert script.
+- `teploy backup status --host S --json`: every DB-like container, labelled or not,
+  and every app's volumes, with coverage state, last success and failure, newest
+  object, last verify, uncovered databases, target and policy. No secrets.
+- Automatic backup before `backup restore`, `accessory restore`, `accessory upgrade`
+  and `remove --purge` (refused if it fails, `--no-backup` overrides), and optionally
+  before a deploy with a `pre_deploy` hook (`backup_before_hooks: true`).
+- Volume size guard: `volume_max_gb` warns and excludes, visible in status and events.
+- `teploy doctor` check `backup-policy`; `teploy setup` installs the aws CLI (AWS's
+  bundle with its PGP signature checked, where the distribution has none, as on Ubuntu
+  24.04) and age.
+- `backup restore`, `accessory restore` and `accessory verify-backup` take `--identity`,
+  `dr` commands take `--prefix`: a rebuilt host recovers with only the owner key and the
+  bucket credentials, through the normal commands. `--deadline` on manual backups,
+  restores and verifies; `--tmp-dir`, `--target-deadline`, `--allow-empty` on the policy.
+- `teploy server backup-target owners [--clear <fingerprint>]`.
+
+### Changed
+
+- Volume backups leave `.env` out when no recovery key is configured, with a warning:
+  it holds resolved secrets. Restoring such a backup keeps the server's current `.env`.
+- `dr create --include-age-key` needs a recovery key and wraps the server's private
+  key to it (a plaintext key in a bucket is the keys to everything); `dr restore
+  --identity <file>` unwraps it. Bundles made before this keep restoring.
+- `remove --purge` needs a backup target or `--no-backup`, names what it will delete,
+  and removes containers with `docker rm -v`.
+- Backups and accessory restores build in `/var/tmp` (`TEPLOY_BACKUP_TMPDIR`), not a
+  `/tmp` that may be tmpfs, and refuse a dump that would not fit: plaintext, ciphertext
+  and a reserve, measured against what the backup will occupy.
+- `backup list` and status no longer treat below-floor objects as restorable, and a
+  newest good backup is never pruned even with only `--max-age-days` set.
+- `backup list` shows per-database rows only with `--databases`: a caller restoring a
+  row by id and accessory alone would replace the primary with another database's dump.
+- Pending uploads live in one `<prefix>/.incoming/` tree, so a 1-day lifecycle rule can
+  expire them on a store that keeps versions; doctor reads the rule and warns without one.
+  Owner markers are written once, not on every prune.
+- A publish copy names `--copy-props metadata-directive`. A store that refuses the copy
+  gets the verified file uploaded straight to its final name.
+- The policy is written and removed through passwordless sudo when that is how it is
+  read, and `set`/`unset` refuse when a policy would stay in force. The run lock falls
+  back to `~/.config/teploy` when the policy's directory is not writable.
+- `owners --clear` refuses a namespace written to in the last week without `--force`,
+  and records every clear or refusal (`backup_target.owners_clear`).
+- A prune refuses when it would delete most of a series (`--allow-mass-delete`) and when
+  the store's clock cannot be read. A future-dated object no longer stops retention: it
+  is left out of selection, never deleted, and warned about.
+- `target_deadline` and `--deadline` over 23h are refused rather than capped. A workspace
+  records its own deadline for the janitor; a pending upload is a leftover after 24h.
+- Safety copies are capped at the newest 5 per target. An unreachable previous prefix is
+  a warning, not a failed listing and a stale status.
+- The aws CLI must be at least 2.23.0 (or 1.37.0), the first to send a checksum on every
+  upload, which Object Lock requires. `teploy setup` replaces an older one.
+
+### Fixed
+
+- `server backup-target set --prefix` broke listing: keys were matched without the
+  prefix and rebuilt with it twice. Previous prefixes stay searchable after a change,
+  and are pruned only when marked owned; a namespace another server has written to is
+  never pruned.
+- A backup's sidecars could outlive it and a volume backup's server-data sibling was
+  never pruned; a pruned backup now takes every object with its id.
+- Upload integrity: a final key only ever holds an object whose size and checksum were
+  verified, so a killed run can never leave a partial upload that counts as a backup.
+- `aws s3 ls` exiting 1 on an empty prefix read as a failure on a first backup.
+- Cron's minimal PATH hid an `aws` or `age` in `/usr/local/bin` from scheduled runs.
+- `backup run` volume archives failed on GNU tar's "file changed" from a live app; the
+  artifact is already labelled crash-consistent.
+
 ## [0.1.51] - 2026-11-05
 
 ### Added