feat(backup): server preflight, detached runs and ranged reads for Dash setup

AUDIT_OPEN.md

@@ -1,5 +1,25 @@
 # Open audit items
 
+## 2026-11-06 — Dash simple backup setup CLI
+
+Plan: `../_internal/DASH_SIMPLE_SETUP_PLAN_2026-11-06.md`. Branch `backup-setup`.
+`server backup-preflight`, `backup head`, `backup run --all --detach`, owners
+fixes, capability `backup-setup-v1`, contracts rev 21.
+
+Open / deferred:
+- `ensureFleetBinary` still installs the latest release, unasked, when
+  `backup-policy set` finds a binary with no `--all`. Left alone: it only fires when
+  the policy could not run at all. `--upgrade-binary` is the consented path for
+  everything else, and a binary that has `--all` is still never touched implicitly.
+- "Managed by something else" has no marker to read, so it is evidence: a policy
+  file that is a symlink, owned by another user, not replaceable by this user, or an
+  interval with no cron line in this user's crontab. Heartbeat hosts are reported so
+  a consumer can tell its own from someone else's.
+- A run that stands down later than the 3 seconds the start script watches reports
+  `started`. A run is in progress either way, and it does not start a second one.
+- teploy-dash `backupstorage.go:35` says the CLI's `set` may install age and the aws
+  CLI. It installs age only. Not this repo's comment; for the Dash side of this work.
+
 ## 2026-11-05 — Dash auto-config Programme B CLI: server siblings
 
 Plan: `../_internal/DASH_AUTOCONFIG_PLAN_2026-11-03.md` section 10. Branch

CHANGELOG.md

@@ -4,6 +4,49 @@ All notable changes to teploy are documented here. Format follows [Keep a Change
 
 ## [Unreleased]
 
+### Added
+
+- `teploy server backup-preflight --host S [--json]`: what a server needs before
+  backups can start. Whether it answers, its teploy binary's version and whether that
+  has the fleet engine, aws against its minimum version, age, python3, free disk where
+  backups are built, whether a target and policy exist, and whether something else
+  looks after the policy. Read-only. A probe that fails reads as unknown, never as fine,
+  and an unreachable server is a report rather than an error.
+- `backup-preflight --install-tools`: installs aws (the same script `teploy setup` runs,
+  signature check included), age and python3 where missing, and nothing else. Idempotent,
+  reports each result, and checks each tool reads as fine afterwards. Recorded as a
+  `server.tools_install` event.
+- `backup-preflight --upgrade-binary`: the consented way for engine fixes to reach a
+  server. Installs the release matching this CLI's version through the verified install
+  path and checks it afterwards. Refuses a development CLI and a downgrade. Recorded as a
+  `server.binary_upgrade` event. Nothing upgrades without the flag.
+- `teploy backup head <key> --bytes N --host S [--json]`: a ranged read of an object as
+  stored, through the server's stored target, as base64. At most 256 KiB, only under the
+  server's own or previous prefixes, and it never decrypts.
+- `teploy backup run --all --detach --host S [--json]`: starts the same `backup run --all`
+  on the server so that it survives a dropped connection, under the same lock, and returns
+  within seconds. A second one reports `already_running` and exits 0. Results reach status
+  and events as a normal run's do, so it counts for coverage. Recorded as a `backup.detach`
+  event. Too old a server binary is a refusal naming `--upgrade-binary`.
+- Contracts: `backup-preflight-envelope`, `backup-head-envelope` and
+  `backup-detach-envelope` (schemas + generated fixtures, corpus rev 21); version-handshake
+  advertises `backup-setup-v1`.
+
+### Fixed
+
+- `server backup-target owners` no longer fails outright on a previous location it cannot
+  reach, which is every previous location for a key limited to the current prefix: each is
+  skipped with a warning and listed in `skipped_locations`. `--clear` asks every location
+  first, so a refusal removes nothing anywhere, then tries all of them instead of stopping
+  at the first failure, and now takes `--json`.
+- `backup status` always says why a configured target is not reachable. With no aws CLI the
+  reason was empty, which a consumer showed as a bare "cannot reach"; a listing that failed
+  silently now names its exit status too.
+- Messages and comments no longer imply that `backup-target set` installs the aws CLI: it
+  checks and warns. The remedy now names `backup-preflight --install-tools`.
+- AWS's bundle installer installs curl when it is missing, on a server `teploy setup` never
+  prepared.
+
 ## [0.1.54] - 2026-11-05
 
 ### Added

README.md

@@ -938,7 +938,9 @@ server backup-target owners` lists the markers; `--clear <fingerprint>` removes
 a server that is gone, or this one before a rebuild. Without its marker a prune here deletes
 that server's backups, so a namespace written to in the last week needs `--force`, and every
 clear or refusal is an event in the `_fleet` log. A policy with no prefix in a bucket another
-server writes to is flagged by `teploy doctor`.
+server writes to is flagged by `teploy doctor`. A previous prefix it cannot
+reach, which is all of them for a key limited to the current prefix, is skipped with a
+warning and listed in `skipped_locations`.
 
 The fingerprint is a hash of `/etc/machine-id` and the hostname, plus the server's age public
 key in the marker. Two things follow, on purpose rather than papered over with an id file a
@@ -966,7 +968,27 @@ stale runs, no heartbeat, a missing `aws` or `age`, a target that does not answe
 leftover workspaces and uploads, a refused prune, a legacy shell cron line (which
 uploads plaintext whatever the policy says), and databases nothing backs up. `teploy
 setup` installs `aws` (AWS's bundle, signature checked, where the distribution has no
-package) and `age`.
+package) and `age`. `backup-target set` does not: it checks and warns.
+
+```bash
+teploy server backup-preflight --host prod [--json]    # read-only: what this server needs
+teploy server backup-preflight --host prod --install-tools    # aws, age, python3, nothing else
+teploy server backup-preflight --host prod --upgrade-binary   # this CLI's version on the server
+teploy backup run --all --detach --host prod [--json]  # survives a dropped connection
+teploy backup head s3://b/prod/app/volumes/20261104-030000.tar.gz.age --bytes 131072 --host prod --json
+```
+
+`backup-preflight` reports the server's teploy binary, `aws` against its minimum version,
+`age`, `python3`, free disk where backups are built, and whether a target and a policy
+exist and something else looks after the policy. A probe that fails reads as unknown,
+never as fine. It changes nothing unless a flag asks: `--install-tools` does none of
+`teploy setup`'s hardening, and `--upgrade-binary` is the only way an engine fix reaches a
+server whose binary already supports `--all`. It never downgrades.
+
+`backup run --all --detach` starts the same run the cron line does, under the same lock: a
+second one reports `already_running`. `backup head` reads the start of an object as stored,
+at most 256 KiB, under the server's own or previous prefixes. It never decrypts: it is how a
+recovery key gets proved in a browser.
 
 Accessory backups are engine-aware. The engine is detected from the
 container — the accessory's `engine:` in `teploy.yml` (set on the container

contracts/MANIFEST.md

@@ -10,6 +10,7 @@ Neutron/Nucleus dependency and a public mirror.
 
 | Corpus rev | Emitting CLI | Machine Interface | Notes |
 |---|---|---|---|
+| 21 | main (Dash simple backup setup) | 2 | Additive. New artifacts backup-preflight-envelope (`teploy server backup-preflight --host S [--install-tools] [--upgrade-binary] --json`: reachable, the server binary's version, fleet engine and match with this CLI, aws against its floor, age, python3, free disk at the backup tmp dir, target and policy state, whether something else manages the policy; a failed probe is `unknown` or null, never fine; an unreachable server is a report with exit 0), backup-head-envelope (`teploy backup head <key> --bytes N --host S --json`: ranged read as stored, base64, at most 256 KiB, own and previous prefixes only, never decrypts) and backup-detach-envelope (`teploy backup run --all --detach --host S --json`: started, already_running, finished with the run's own result, or failed; the run is the cron line's, under its flock). Fixtures GENERATED from the real `collectBackupPreflight`, `backupPreflightOn` and `startDetachedRun` over mock executors and the real backupHeadDTO. Event kinds `backup.detach`, `server.tools_install` and `server.binary_upgrade` under `_fleet`; `backup_target.owners_clear` gains `skipped_locations`. `server backup-target owners --json` gains `skipped_locations`, and `--clear` gains `--json` ({machine_interface, host, fingerprint, removed, skipped_locations}, printed even when it exits nonzero); neither is a corpus artifact. version-handshake advertises `backup-setup-v1` (regenerated mi2.json). No MI bump. |
 | 20 | main (Dash auto-config Programme B: siblings) | 2 | Additive. New artifact siblings-envelope (`teploy server siblings <server> --json`, read-only sibling discovery: teploy products by the `dev.teploy.product` label — the container's own, else its image's — and the forgejo family by the OCI `org.opencontainers.image.title`/`source` labels; per row version hints (dev.teploy.version, the teploy release label for a teploy app, else the image's `org.opencontainers.image.version`), the teploy app name, and `port_hints` from docker inspect's published binds; unrecognised `dev.teploy.product` values pass through verbatim under `unknown_labeled`; state ok/partial/unreachable so an unobserved server is never a clean "no siblings"). Observation only — no probing, no DNS, no env/credentials; at most two remote commands (the shared batched `docker ps`, then one batched inspect for port binds + image labels), pinned read-only by test. Fixtures GENERATED from the real `collectServerSiblings` over a mock executor (mixed: one of each detection class + pass-through; none). version-handshake advertises `siblings-v1` (regenerated mi2.json). No MI bump. |
 | 19 | main (Dash auto-config Slice 6: cleanup / audit) | 2 | Additive. New artifact server-audit-envelope (`teploy server audit <server> --json`, read-only: findings with stable `id` = kind:subject, severity, evidence, and remedy {argv (null when nothing the CLI can do safely), destructive, confirm}, per-source `checks` where unknown means findings depending on it are missing — never clean — and disk attribution), fixture GENERATED from the real `auditServer` over a mock executor (one of each finding incl. null-remedy, one unknown check). New remedy commands `route remove --app X` (exact-block CAS), `releases prune --keep N` (pins/current/previous/running never pruned) and `server prune --build-cache --dangling`; events `route.remove`, `releases.prune`, `server.prune` (the last under `_fleet`). keep_versions becomes optional: unset → 5 pruned only after a 7-day notice recorded server-side (`deploy/retention.go`), `init` writes 3, explicit 0 keeps everything and records `keep_all_versions` in the applied manifest (unset and N record what they always did, so existing digests hold). version-handshake advertises `server-audit-v1` (regenerated mi2.json). No MI bump. |
 | 18 | main (Dash auto-config Slice 4: fleet backup engine) | 2 | Additive. New artifacts backup-status-envelope (`backup status --json`: every DB-like container and app volume with coverage state, last success/failure, newest object, last verify, plus target and policy summaries, the aws CLI's `aws_cli_problem` and the target's `incoming_lifecycle`; no secrets), backup-run-envelope (`backup run --all --json`) and backup-policy-envelope (`server backup-policy show/set --json`), fixtures GENERATED from the real DTOs. backup-list rows gain optional `encrypted`, `previous_prefix`, and `database` only with `--databases` or `--adopted` (a caller restoring by id alone would replace the primary); backup-create gains `encrypted` and `objects`; backup-restore gains `pre_backup_id`; backup-target gains `previous_prefixes`. Event kinds `backup.all`, `backup_policy.set` and `backup_target.owners_clear` (a marker removal or refusal) under the pseudo-app `_fleet`; adopted containers log under `_adopted`. version-handshake advertises `backup-fleet-v1` (regenerated mi2.json). No MI bump. |
@@ -64,6 +65,9 @@ Neutron/Nucleus dependency and a public mirror.
 | domains-check-envelope | yes (rev 17) | valid (one server with each finding, one unreachable; real `sweepDomains`) | teploy-cli |
 | server-audit-envelope | yes (rev 19) | valid (one of each finding incl. null-remedy, one unknown check; real `auditServer` over a mock executor) | teploy-cli |
 | siblings-envelope | yes (rev 20) | valid x2 (mixed detection classes + pass-through, none; real `collectServerSiblings` over a mock executor) | teploy-cli |
+| backup-preflight-envelope | yes (rev 21) | valid x5 (ready, needs-setup, unknown, unreachable, after-actions; real `collectBackupPreflight`/`backupPreflightOn` over a mock executor) | teploy-cli |
+| backup-head-envelope | yes (rev 21) | valid (encrypted; real backupHeadDTO) | teploy-cli |
+| backup-detach-envelope | yes (rev 21) | valid x4 (started, already-running, finished, failed; real `startDetachedRun` over a mock executor) | teploy-cli |
 | releases-envelope | yes (rev 14) | valid x2 (container app, static app; real releasesDTO) | teploy-cli |
 | rollback-result | yes (rev 14) | valid (real rollbackResult) | teploy-cli |
 | redeploy-result | yes (rev 14) | valid x2 (teploy-home, folder-home; real redeployDTO) | teploy-cli |

contracts/fixtures/backup-detach-envelope/valid/already-running.json

@@ -0,0 +1,6 @@
+{
+  "machine_interface": 2,
+  "host": "192.0.2.10",
+  "requested_at": "2026-11-06T12:00:00Z",
+  "status": "already_running"
+}

contracts/fixtures/backup-detach-envelope/valid/failed.json

@@ -0,0 +1,9 @@
+{
+  "machine_interface": 2,
+  "host": "192.0.2.10",
+  "requested_at": "2026-11-06T12:00:00Z",
+  "status": "failed",
+  "pid": 4245,
+  "log": "/root/.config/teploy/backup-runs/run.MnOp3456.log",
+  "error": "the run ended without a result: Error: docker: permission denied"
+}

contracts/fixtures/backup-detach-envelope/valid/finished.json

@@ -0,0 +1,16 @@
+{
+  "machine_interface": 2,
+  "host": "192.0.2.10",
+  "requested_at": "2026-11-06T12:00:00Z",
+  "status": "finished",
+  "pid": 4244,
+  "log": "/root/.config/teploy/backup-runs/run.IjKl9012.log",
+  "result": {
+    "status": "failed",
+    "started_at": "2026-11-06T12:00:00Z",
+    "duration_ms": 0,
+    "encrypted": false,
+    "targets": [],
+    "error": "no backup policy on this server: set one with `teploy server backup-policy set`"
+  }
+}

contracts/fixtures/backup-detach-envelope/valid/started.json

@@ -0,0 +1,8 @@
+{
+  "machine_interface": 2,
+  "host": "192.0.2.10",
+  "requested_at": "2026-11-06T12:00:00Z",
+  "status": "started",
+  "pid": 4242,
+  "log": "/root/.config/teploy/backup-runs/run.AbCd1234.log"
+}

contracts/fixtures/backup-head-envelope/valid/encrypted.json

@@ -0,0 +1,10 @@
+{
+  "machine_interface": 2,
+  "host": "192.0.2.10",
+  "key": "s3://teploy-backups/teploy/srv1/blog/accessories/db/20261104-031704-1f2e3d4c5b6a7988.pgdump.age",
+  "encrypted": true,
+  "requested_bytes": 22,
+  "length": 22,
+  "total_size": 1048576,
+  "data_base64": "YWdlLWVuY3J5cHRpb24ub3JnL3YxCg=="
+}

contracts/fixtures/backup-preflight-envelope/valid/after-actions.json

@@ -0,0 +1,61 @@
+{
+  "machine_interface": 2,
+  "host": "h",
+  "observed_at": "2026-11-06T12:00:00Z",
+  "reachable": true,
+  "binary": {
+    "path": "/usr/local/bin/teploy",
+    "state": "ok",
+    "version": "0.1.55",
+    "fleet_engine": true,
+    "cli_version": "0.1.55",
+    "same_as_cli": true
+  },
+  "tools": {
+    "aws": {
+      "state": "ok",
+      "version": "2.30.1"
+    },
+    "age": {
+      "state": "ok",
+      "version": "v1.2.1"
+    },
+    "python3": {
+      "state": "ok",
+      "version": "Python 3.12.3"
+    }
+  },
+  "disk": {
+    "path": "/var/tmp",
+    "free_bytes": 107374182400
+  },
+  "target": {
+    "state": "absent",
+    "has_credentials": false
+  },
+  "policy": {
+    "state": "absent",
+    "encrypted": false,
+    "cron_installed": null,
+    "managed_elsewhere": "no"
+  },
+  "tools_install": [
+    {
+      "tool": "python3",
+      "action": "done"
+    },
+    {
+      "tool": "age",
+      "action": "done"
+    },
+    {
+      "tool": "aws",
+      "action": "done"
+    }
+  ],
+  "binary_upgrade": {
+    "action": "done",
+    "from": "0.1.20",
+    "to": "0.1.55"
+  }
+}

contracts/fixtures/backup-preflight-envelope/valid/needs-setup.json

@@ -0,0 +1,41 @@
+{
+  "machine_interface": 2,
+  "host": "192.0.2.10",
+  "observed_at": "2026-11-06T12:00:00Z",
+  "reachable": true,
+  "binary": {
+    "path": "/usr/local/bin/teploy",
+    "state": "no_fleet_engine",
+    "version": "0.1.20",
+    "fleet_engine": false,
+    "cli_version": "0.1.55",
+    "same_as_cli": false
+  },
+  "tools": {
+    "aws": {
+      "state": "outdated",
+      "version": "2.15.0",
+      "detail": "aws CLI 2.15.0 is older than 2.23.0 (or 1.37.0), the first to send a checksum on every upload, which a bucket with Object Lock requires"
+    },
+    "age": {
+      "state": "missing"
+    },
+    "python3": {
+      "state": "missing"
+    }
+  },
+  "disk": {
+    "path": "/var/tmp",
+    "free_bytes": 2147483648
+  },
+  "target": {
+    "state": "absent",
+    "has_credentials": false
+  },
+  "policy": {
+    "state": "absent",
+    "encrypted": false,
+    "cron_installed": null,
+    "managed_elsewhere": "no"
+  }
+}

contracts/fixtures/backup-preflight-envelope/valid/ready.json

@@ -0,0 +1,49 @@
+{
+  "machine_interface": 2,
+  "host": "192.0.2.10",
+  "observed_at": "2026-11-06T12:00:00Z",
+  "reachable": true,
+  "binary": {
+    "path": "/usr/local/bin/teploy",
+    "state": "ok",
+    "version": "0.1.55",
+    "fleet_engine": true,
+    "cli_version": "0.1.55",
+    "same_as_cli": true
+  },
+  "tools": {
+    "aws": {
+      "state": "ok",
+      "version": "2.30.1"
+    },
+    "age": {
+      "state": "ok",
+      "version": "v1.2.1"
+    },
+    "python3": {
+      "state": "ok",
+      "version": "Python 3.12.3"
+    }
+  },
+  "disk": {
+    "path": "/var/tmp",
+    "free_bytes": 107374182400
+  },
+  "target": {
+    "state": "configured",
+    "bucket": "b",
+    "prefix": "teploy/srv1",
+    "has_credentials": true
+  },
+  "policy": {
+    "state": "configured",
+    "path": "/etc/teploy/backup-policy.json",
+    "interval": "1h",
+    "encrypted": true,
+    "heartbeat_hosts": [
+      "https://dash.example.internal"
+    ],
+    "cron_installed": true,
+    "managed_elsewhere": "no"
+  }
+}

contracts/fixtures/backup-preflight-envelope/valid/unknown.json

@@ -0,0 +1,44 @@
+{
+  "machine_interface": 2,
+  "host": "192.0.2.10",
+  "observed_at": "2026-11-06T12:00:00Z",
+  "reachable": true,
+  "error": "the probe did not finish, so what it did not reach is unknown",
+  "binary": {
+    "path": "/usr/local/bin/teploy",
+    "state": "unknown",
+    "fleet_engine": null,
+    "cli_version": "0.1.55",
+    "same_as_cli": null,
+    "detail": "the binary is there and did not run"
+  },
+  "tools": {
+    "aws": {
+      "state": "unknown",
+      "detail": "could not read its version: some wrapper"
+    },
+    "age": {
+      "state": "unknown"
+    },
+    "python3": {
+      "state": "unknown"
+    }
+  },
+  "disk": {
+    "path": "/var/tmp",
+    "free_bytes": null,
+    "detail": "could not read the free space there"
+  },
+  "target": {
+    "state": "unknown",
+    "has_credentials": false,
+    "detail": "reading the backup target: unrecognized output framing"
+  },
+  "policy": {
+    "state": "unknown",
+    "encrypted": false,
+    "cron_installed": null,
+    "managed_elsewhere": "unknown",
+    "detail": "the backup policy cannot be read: exit status 3: cannot tell whether this server's backups must be encrypted"
+  }
+}