diff --git a/AUDIT_OPEN.md b/AUDIT_OPEN.md index 350ddf9..2b3e311 100644 --- a/AUDIT_OPEN.md +++ b/AUDIT_OPEN.md @@ -1,5 +1,25 @@ # Open audit items +## 2026-11-06 — Dash simple backup setup CLI + +Plan: `../_internal/DASH_SIMPLE_SETUP_PLAN_2026-11-06.md`. Branch `backup-setup`. +`server backup-preflight`, `backup head`, `backup run --all --detach`, owners +fixes, capability `backup-setup-v1`, contracts rev 21. + +Open / deferred: +- `ensureFleetBinary` still installs the latest release, unasked, when + `backup-policy set` finds a binary with no `--all`. Left alone: it only fires when + the policy could not run at all. `--upgrade-binary` is the consented path for + everything else, and a binary that has `--all` is still never touched implicitly. +- "Managed by something else" has no marker to read, so it is evidence: a policy + file that is a symlink, owned by another user, not replaceable by this user, or an + interval with no cron line in this user's crontab. Heartbeat hosts are reported so + a consumer can tell its own from someone else's. +- A run that stands down later than the 3 seconds the start script watches reports + `started`. A run is in progress either way, and it does not start a second one. +- teploy-dash `backupstorage.go:35` says the CLI's `set` may install age and the aws + CLI. It installs age only. Not this repo's comment; for the Dash side of this work. + ## 2026-11-05 — Dash auto-config Programme B CLI: server siblings Plan: `../_internal/DASH_AUTOCONFIG_PLAN_2026-11-03.md` section 10. Branch diff --git a/CHANGELOG.md b/CHANGELOG.md index 615f269..fb0de71 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,49 @@ All notable changes to teploy are documented here. Format follows [Keep a Change ## [Unreleased] +### Added + +- `teploy server backup-preflight --host S [--json]`: what a server needs before + backups can start. Whether it answers, its teploy binary's version and whether that + has the fleet engine, aws against its minimum version, age, python3, free disk where + backups are built, whether a target and policy exist, and whether something else + looks after the policy. Read-only. A probe that fails reads as unknown, never as fine, + and an unreachable server is a report rather than an error. +- `backup-preflight --install-tools`: installs aws (the same script `teploy setup` runs, + signature check included), age and python3 where missing, and nothing else. Idempotent, + reports each result, and checks each tool reads as fine afterwards. Recorded as a + `server.tools_install` event. +- `backup-preflight --upgrade-binary`: the consented way for engine fixes to reach a + server. Installs the release matching this CLI's version through the verified install + path and checks it afterwards. Refuses a development CLI and a downgrade. Recorded as a + `server.binary_upgrade` event. Nothing upgrades without the flag. +- `teploy backup head --bytes N --host S [--json]`: a ranged read of an object as + stored, through the server's stored target, as base64. At most 256 KiB, only under the + server's own or previous prefixes, and it never decrypts. +- `teploy backup run --all --detach --host S [--json]`: starts the same `backup run --all` + on the server so that it survives a dropped connection, under the same lock, and returns + within seconds. A second one reports `already_running` and exits 0. Results reach status + and events as a normal run's do, so it counts for coverage. Recorded as a `backup.detach` + event. Too old a server binary is a refusal naming `--upgrade-binary`. +- Contracts: `backup-preflight-envelope`, `backup-head-envelope` and + `backup-detach-envelope` (schemas + generated fixtures, corpus rev 21); version-handshake + advertises `backup-setup-v1`. + +### Fixed + +- `server backup-target owners` no longer fails outright on a previous location it cannot + reach, which is every previous location for a key limited to the current prefix: each is + skipped with a warning and listed in `skipped_locations`. `--clear` asks every location + first, so a refusal removes nothing anywhere, then tries all of them instead of stopping + at the first failure, and now takes `--json`. +- `backup status` always says why a configured target is not reachable. With no aws CLI the + reason was empty, which a consumer showed as a bare "cannot reach"; a listing that failed + silently now names its exit status too. +- Messages and comments no longer imply that `backup-target set` installs the aws CLI: it + checks and warns. The remedy now names `backup-preflight --install-tools`. +- AWS's bundle installer installs curl when it is missing, on a server `teploy setup` never + prepared. + ## [0.1.54] - 2026-11-05 ### Added diff --git a/README.md b/README.md index b7ef020..13779b9 100644 --- a/README.md +++ b/README.md @@ -938,7 +938,9 @@ server backup-target owners` lists the markers; `--clear ` removes a server that is gone, or this one before a rebuild. Without its marker a prune here deletes that server's backups, so a namespace written to in the last week needs `--force`, and every clear or refusal is an event in the `_fleet` log. A policy with no prefix in a bucket another -server writes to is flagged by `teploy doctor`. +server writes to is flagged by `teploy doctor`. A previous prefix it cannot +reach, which is all of them for a key limited to the current prefix, is skipped with a +warning and listed in `skipped_locations`. The fingerprint is a hash of `/etc/machine-id` and the hostname, plus the server's age public key in the marker. Two things follow, on purpose rather than papered over with an id file a @@ -966,7 +968,27 @@ stale runs, no heartbeat, a missing `aws` or `age`, a target that does not answe leftover workspaces and uploads, a refused prune, a legacy shell cron line (which uploads plaintext whatever the policy says), and databases nothing backs up. `teploy setup` installs `aws` (AWS's bundle, signature checked, where the distribution has no -package) and `age`. +package) and `age`. `backup-target set` does not: it checks and warns. + +```bash +teploy server backup-preflight --host prod [--json] # read-only: what this server needs +teploy server backup-preflight --host prod --install-tools # aws, age, python3, nothing else +teploy server backup-preflight --host prod --upgrade-binary # this CLI's version on the server +teploy backup run --all --detach --host prod [--json] # survives a dropped connection +teploy backup head s3://b/prod/app/volumes/20261104-030000.tar.gz.age --bytes 131072 --host prod --json +``` + +`backup-preflight` reports the server's teploy binary, `aws` against its minimum version, +`age`, `python3`, free disk where backups are built, and whether a target and a policy +exist and something else looks after the policy. A probe that fails reads as unknown, +never as fine. It changes nothing unless a flag asks: `--install-tools` does none of +`teploy setup`'s hardening, and `--upgrade-binary` is the only way an engine fix reaches a +server whose binary already supports `--all`. It never downgrades. + +`backup run --all --detach` starts the same run the cron line does, under the same lock: a +second one reports `already_running`. `backup head` reads the start of an object as stored, +at most 256 KiB, under the server's own or previous prefixes. It never decrypts: it is how a +recovery key gets proved in a browser. Accessory backups are engine-aware. The engine is detected from the container — the accessory's `engine:` in `teploy.yml` (set on the container diff --git a/contracts/MANIFEST.md b/contracts/MANIFEST.md index 3c64024..746e1e5 100644 --- a/contracts/MANIFEST.md +++ b/contracts/MANIFEST.md @@ -10,6 +10,7 @@ Neutron/Nucleus dependency and a public mirror. | Corpus rev | Emitting CLI | Machine Interface | Notes | |---|---|---|---| +| 21 | main (Dash simple backup setup) | 2 | Additive. New artifacts backup-preflight-envelope (`teploy server backup-preflight --host S [--install-tools] [--upgrade-binary] --json`: reachable, the server binary's version, fleet engine and match with this CLI, aws against its floor, age, python3, free disk at the backup tmp dir, target and policy state, whether something else manages the policy; a failed probe is `unknown` or null, never fine; an unreachable server is a report with exit 0), backup-head-envelope (`teploy backup head --bytes N --host S --json`: ranged read as stored, base64, at most 256 KiB, own and previous prefixes only, never decrypts) and backup-detach-envelope (`teploy backup run --all --detach --host S --json`: started, already_running, finished with the run's own result, or failed; the run is the cron line's, under its flock). Fixtures GENERATED from the real `collectBackupPreflight`, `backupPreflightOn` and `startDetachedRun` over mock executors and the real backupHeadDTO. Event kinds `backup.detach`, `server.tools_install` and `server.binary_upgrade` under `_fleet`; `backup_target.owners_clear` gains `skipped_locations`. `server backup-target owners --json` gains `skipped_locations`, and `--clear` gains `--json` ({machine_interface, host, fingerprint, removed, skipped_locations}, printed even when it exits nonzero); neither is a corpus artifact. version-handshake advertises `backup-setup-v1` (regenerated mi2.json). No MI bump. | | 20 | main (Dash auto-config Programme B: siblings) | 2 | Additive. New artifact siblings-envelope (`teploy server siblings --json`, read-only sibling discovery: teploy products by the `dev.teploy.product` label — the container's own, else its image's — and the forgejo family by the OCI `org.opencontainers.image.title`/`source` labels; per row version hints (dev.teploy.version, the teploy release label for a teploy app, else the image's `org.opencontainers.image.version`), the teploy app name, and `port_hints` from docker inspect's published binds; unrecognised `dev.teploy.product` values pass through verbatim under `unknown_labeled`; state ok/partial/unreachable so an unobserved server is never a clean "no siblings"). Observation only — no probing, no DNS, no env/credentials; at most two remote commands (the shared batched `docker ps`, then one batched inspect for port binds + image labels), pinned read-only by test. Fixtures GENERATED from the real `collectServerSiblings` over a mock executor (mixed: one of each detection class + pass-through; none). version-handshake advertises `siblings-v1` (regenerated mi2.json). No MI bump. | | 19 | main (Dash auto-config Slice 6: cleanup / audit) | 2 | Additive. New artifact server-audit-envelope (`teploy server audit --json`, read-only: findings with stable `id` = kind:subject, severity, evidence, and remedy {argv (null when nothing the CLI can do safely), destructive, confirm}, per-source `checks` where unknown means findings depending on it are missing — never clean — and disk attribution), fixture GENERATED from the real `auditServer` over a mock executor (one of each finding incl. null-remedy, one unknown check). New remedy commands `route remove --app X` (exact-block CAS), `releases prune --keep N` (pins/current/previous/running never pruned) and `server prune --build-cache --dangling`; events `route.remove`, `releases.prune`, `server.prune` (the last under `_fleet`). keep_versions becomes optional: unset → 5 pruned only after a 7-day notice recorded server-side (`deploy/retention.go`), `init` writes 3, explicit 0 keeps everything and records `keep_all_versions` in the applied manifest (unset and N record what they always did, so existing digests hold). version-handshake advertises `server-audit-v1` (regenerated mi2.json). No MI bump. | | 18 | main (Dash auto-config Slice 4: fleet backup engine) | 2 | Additive. New artifacts backup-status-envelope (`backup status --json`: every DB-like container and app volume with coverage state, last success/failure, newest object, last verify, plus target and policy summaries, the aws CLI's `aws_cli_problem` and the target's `incoming_lifecycle`; no secrets), backup-run-envelope (`backup run --all --json`) and backup-policy-envelope (`server backup-policy show/set --json`), fixtures GENERATED from the real DTOs. backup-list rows gain optional `encrypted`, `previous_prefix`, and `database` only with `--databases` or `--adopted` (a caller restoring by id alone would replace the primary); backup-create gains `encrypted` and `objects`; backup-restore gains `pre_backup_id`; backup-target gains `previous_prefixes`. Event kinds `backup.all`, `backup_policy.set` and `backup_target.owners_clear` (a marker removal or refusal) under the pseudo-app `_fleet`; adopted containers log under `_adopted`. version-handshake advertises `backup-fleet-v1` (regenerated mi2.json). No MI bump. | @@ -64,6 +65,9 @@ Neutron/Nucleus dependency and a public mirror. | domains-check-envelope | yes (rev 17) | valid (one server with each finding, one unreachable; real `sweepDomains`) | teploy-cli | | server-audit-envelope | yes (rev 19) | valid (one of each finding incl. null-remedy, one unknown check; real `auditServer` over a mock executor) | teploy-cli | | siblings-envelope | yes (rev 20) | valid x2 (mixed detection classes + pass-through, none; real `collectServerSiblings` over a mock executor) | teploy-cli | +| backup-preflight-envelope | yes (rev 21) | valid x5 (ready, needs-setup, unknown, unreachable, after-actions; real `collectBackupPreflight`/`backupPreflightOn` over a mock executor) | teploy-cli | +| backup-head-envelope | yes (rev 21) | valid (encrypted; real backupHeadDTO) | teploy-cli | +| backup-detach-envelope | yes (rev 21) | valid x4 (started, already-running, finished, failed; real `startDetachedRun` over a mock executor) | teploy-cli | | releases-envelope | yes (rev 14) | valid x2 (container app, static app; real releasesDTO) | teploy-cli | | rollback-result | yes (rev 14) | valid (real rollbackResult) | teploy-cli | | redeploy-result | yes (rev 14) | valid x2 (teploy-home, folder-home; real redeployDTO) | teploy-cli | diff --git a/contracts/fixtures/backup-detach-envelope/valid/already-running.json b/contracts/fixtures/backup-detach-envelope/valid/already-running.json new file mode 100644 index 0000000..6c64337 --- /dev/null +++ b/contracts/fixtures/backup-detach-envelope/valid/already-running.json @@ -0,0 +1,6 @@ +{ + "machine_interface": 2, + "host": "192.0.2.10", + "requested_at": "2026-11-06T12:00:00Z", + "status": "already_running" +} diff --git a/contracts/fixtures/backup-detach-envelope/valid/failed.json b/contracts/fixtures/backup-detach-envelope/valid/failed.json new file mode 100644 index 0000000..13d57c5 --- /dev/null +++ b/contracts/fixtures/backup-detach-envelope/valid/failed.json @@ -0,0 +1,9 @@ +{ + "machine_interface": 2, + "host": "192.0.2.10", + "requested_at": "2026-11-06T12:00:00Z", + "status": "failed", + "pid": 4245, + "log": "/root/.config/teploy/backup-runs/run.MnOp3456.log", + "error": "the run ended without a result: Error: docker: permission denied" +} diff --git a/contracts/fixtures/backup-detach-envelope/valid/finished.json b/contracts/fixtures/backup-detach-envelope/valid/finished.json new file mode 100644 index 0000000..3ae97a0 --- /dev/null +++ b/contracts/fixtures/backup-detach-envelope/valid/finished.json @@ -0,0 +1,16 @@ +{ + "machine_interface": 2, + "host": "192.0.2.10", + "requested_at": "2026-11-06T12:00:00Z", + "status": "finished", + "pid": 4244, + "log": "/root/.config/teploy/backup-runs/run.IjKl9012.log", + "result": { + "status": "failed", + "started_at": "2026-11-06T12:00:00Z", + "duration_ms": 0, + "encrypted": false, + "targets": [], + "error": "no backup policy on this server: set one with `teploy server backup-policy set`" + } +} diff --git a/contracts/fixtures/backup-detach-envelope/valid/started.json b/contracts/fixtures/backup-detach-envelope/valid/started.json new file mode 100644 index 0000000..86b0709 --- /dev/null +++ b/contracts/fixtures/backup-detach-envelope/valid/started.json @@ -0,0 +1,8 @@ +{ + "machine_interface": 2, + "host": "192.0.2.10", + "requested_at": "2026-11-06T12:00:00Z", + "status": "started", + "pid": 4242, + "log": "/root/.config/teploy/backup-runs/run.AbCd1234.log" +} diff --git a/contracts/fixtures/backup-head-envelope/valid/encrypted.json b/contracts/fixtures/backup-head-envelope/valid/encrypted.json new file mode 100644 index 0000000..2ee40c8 --- /dev/null +++ b/contracts/fixtures/backup-head-envelope/valid/encrypted.json @@ -0,0 +1,10 @@ +{ + "machine_interface": 2, + "host": "192.0.2.10", + "key": "s3://teploy-backups/teploy/srv1/blog/accessories/db/20261104-031704-1f2e3d4c5b6a7988.pgdump.age", + "encrypted": true, + "requested_bytes": 22, + "length": 22, + "total_size": 1048576, + "data_base64": "YWdlLWVuY3J5cHRpb24ub3JnL3YxCg==" +} diff --git a/contracts/fixtures/backup-preflight-envelope/valid/after-actions.json b/contracts/fixtures/backup-preflight-envelope/valid/after-actions.json new file mode 100644 index 0000000..10e77f6 --- /dev/null +++ b/contracts/fixtures/backup-preflight-envelope/valid/after-actions.json @@ -0,0 +1,61 @@ +{ + "machine_interface": 2, + "host": "h", + "observed_at": "2026-11-06T12:00:00Z", + "reachable": true, + "binary": { + "path": "/usr/local/bin/teploy", + "state": "ok", + "version": "0.1.55", + "fleet_engine": true, + "cli_version": "0.1.55", + "same_as_cli": true + }, + "tools": { + "aws": { + "state": "ok", + "version": "2.30.1" + }, + "age": { + "state": "ok", + "version": "v1.2.1" + }, + "python3": { + "state": "ok", + "version": "Python 3.12.3" + } + }, + "disk": { + "path": "/var/tmp", + "free_bytes": 107374182400 + }, + "target": { + "state": "absent", + "has_credentials": false + }, + "policy": { + "state": "absent", + "encrypted": false, + "cron_installed": null, + "managed_elsewhere": "no" + }, + "tools_install": [ + { + "tool": "python3", + "action": "done" + }, + { + "tool": "age", + "action": "done" + }, + { + "tool": "aws", + "action": "done" + } + ], + "binary_upgrade": { + "action": "done", + "from": "0.1.20", + "to": "0.1.55" + } +} diff --git a/contracts/fixtures/backup-preflight-envelope/valid/needs-setup.json b/contracts/fixtures/backup-preflight-envelope/valid/needs-setup.json new file mode 100644 index 0000000..ec13e27 --- /dev/null +++ b/contracts/fixtures/backup-preflight-envelope/valid/needs-setup.json @@ -0,0 +1,41 @@ +{ + "machine_interface": 2, + "host": "192.0.2.10", + "observed_at": "2026-11-06T12:00:00Z", + "reachable": true, + "binary": { + "path": "/usr/local/bin/teploy", + "state": "no_fleet_engine", + "version": "0.1.20", + "fleet_engine": false, + "cli_version": "0.1.55", + "same_as_cli": false + }, + "tools": { + "aws": { + "state": "outdated", + "version": "2.15.0", + "detail": "aws CLI 2.15.0 is older than 2.23.0 (or 1.37.0), the first to send a checksum on every upload, which a bucket with Object Lock requires" + }, + "age": { + "state": "missing" + }, + "python3": { + "state": "missing" + } + }, + "disk": { + "path": "/var/tmp", + "free_bytes": 2147483648 + }, + "target": { + "state": "absent", + "has_credentials": false + }, + "policy": { + "state": "absent", + "encrypted": false, + "cron_installed": null, + "managed_elsewhere": "no" + } +} diff --git a/contracts/fixtures/backup-preflight-envelope/valid/ready.json b/contracts/fixtures/backup-preflight-envelope/valid/ready.json new file mode 100644 index 0000000..f391dc3 --- /dev/null +++ b/contracts/fixtures/backup-preflight-envelope/valid/ready.json @@ -0,0 +1,49 @@ +{ + "machine_interface": 2, + "host": "192.0.2.10", + "observed_at": "2026-11-06T12:00:00Z", + "reachable": true, + "binary": { + "path": "/usr/local/bin/teploy", + "state": "ok", + "version": "0.1.55", + "fleet_engine": true, + "cli_version": "0.1.55", + "same_as_cli": true + }, + "tools": { + "aws": { + "state": "ok", + "version": "2.30.1" + }, + "age": { + "state": "ok", + "version": "v1.2.1" + }, + "python3": { + "state": "ok", + "version": "Python 3.12.3" + } + }, + "disk": { + "path": "/var/tmp", + "free_bytes": 107374182400 + }, + "target": { + "state": "configured", + "bucket": "b", + "prefix": "teploy/srv1", + "has_credentials": true + }, + "policy": { + "state": "configured", + "path": "/etc/teploy/backup-policy.json", + "interval": "1h", + "encrypted": true, + "heartbeat_hosts": [ + "https://dash.example.internal" + ], + "cron_installed": true, + "managed_elsewhere": "no" + } +} diff --git a/contracts/fixtures/backup-preflight-envelope/valid/unknown.json b/contracts/fixtures/backup-preflight-envelope/valid/unknown.json new file mode 100644 index 0000000..8bbaeaf --- /dev/null +++ b/contracts/fixtures/backup-preflight-envelope/valid/unknown.json @@ -0,0 +1,44 @@ +{ + "machine_interface": 2, + "host": "192.0.2.10", + "observed_at": "2026-11-06T12:00:00Z", + "reachable": true, + "error": "the probe did not finish, so what it did not reach is unknown", + "binary": { + "path": "/usr/local/bin/teploy", + "state": "unknown", + "fleet_engine": null, + "cli_version": "0.1.55", + "same_as_cli": null, + "detail": "the binary is there and did not run" + }, + "tools": { + "aws": { + "state": "unknown", + "detail": "could not read its version: some wrapper" + }, + "age": { + "state": "unknown" + }, + "python3": { + "state": "unknown" + } + }, + "disk": { + "path": "/var/tmp", + "free_bytes": null, + "detail": "could not read the free space there" + }, + "target": { + "state": "unknown", + "has_credentials": false, + "detail": "reading the backup target: unrecognized output framing" + }, + "policy": { + "state": "unknown", + "encrypted": false, + "cron_installed": null, + "managed_elsewhere": "unknown", + "detail": "the backup policy cannot be read: exit status 3: cannot tell whether this server's backups must be encrypted" + } +} diff --git a/contracts/fixtures/backup-preflight-envelope/valid/unreachable.json b/contracts/fixtures/backup-preflight-envelope/valid/unreachable.json new file mode 100644 index 0000000..3e74984 --- /dev/null +++ b/contracts/fixtures/backup-preflight-envelope/valid/unreachable.json @@ -0,0 +1,39 @@ +{ + "machine_interface": 2, + "host": "192.0.2.10", + "observed_at": "2026-11-06T12:00:00Z", + "reachable": false, + "error": "dial tcp 192.0.2.10:22: i/o timeout", + "binary": { + "path": "/usr/local/bin/teploy", + "state": "unknown", + "fleet_engine": null, + "cli_version": "0.1.55", + "same_as_cli": null + }, + "tools": { + "aws": { + "state": "unknown" + }, + "age": { + "state": "unknown" + }, + "python3": { + "state": "unknown" + } + }, + "disk": { + "path": "/var/tmp", + "free_bytes": null + }, + "target": { + "state": "unknown", + "has_credentials": false + }, + "policy": { + "state": "unknown", + "encrypted": false, + "cron_installed": null, + "managed_elsewhere": "unknown" + } +} diff --git a/contracts/fixtures/version-handshake/valid/mi2.json b/contracts/fixtures/version-handshake/valid/mi2.json index 8a99edf..3c2e13e 100644 --- a/contracts/fixtures/version-handshake/valid/mi2.json +++ b/contracts/fixtures/version-handshake/valid/mi2.json @@ -5,6 +5,7 @@ "autodeploy-redeploy", "backup-fleet-v1", "backup-json-v1", + "backup-setup-v1", "backup-target-v1", "config-home-v1", "config-merge-v1", diff --git a/contracts/schema/backup-detach-envelope.schema.json b/contracts/schema/backup-detach-envelope.schema.json new file mode 100644 index 0000000..fb84971 --- /dev/null +++ b/contracts/schema/backup-detach-envelope.schema.json @@ -0,0 +1,20 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://teploy.github.io/contracts/schema/backup-detach-envelope.schema.json", + "title": "`teploy backup run --all --detach --host S --json` (backup-setup-v1, rev 21): starts the cron line's `backup run --all` on the server, under its flock, and looks at it for about 3 seconds. Later results reach `backup status` and `events` as a normal run's do. Exit 0 for started and already_running; finished exits as the run would have; failed exits nonzero. A binary too old to run it, or a failure to start, uses the error envelope.", + "type": "object", + "required": ["machine_interface", "host", "requested_at", "status"], + "properties": { + "machine_interface": {"type": "integer", "minimum": 1}, + "host": {"type": "string"}, + "requested_at": {"type": "string", "format": "date-time"}, + "status": { + "enum": ["started", "already_running", "finished", "failed"], + "description": "started: still running when it looked. already_running: another run holds the lock and this one stood down. finished: it ended within the first seconds, see result. failed: it ended without a result, see error." + }, + "pid": {"type": "integer", "minimum": 1}, + "log": {"type": "string", "description": "the run's progress on the server (0600, removed after 14 days)"}, + "result": {"type": "object", "description": "with finished: the run's own result, as in backup-run-envelope without machine_interface", "required": ["status", "started_at", "duration_ms", "encrypted", "targets"]}, + "error": {"type": "string"} + } +} diff --git a/contracts/schema/backup-head-envelope.schema.json b/contracts/schema/backup-head-envelope.schema.json new file mode 100644 index 0000000..a916665 --- /dev/null +++ b/contracts/schema/backup-head-envelope.schema.json @@ -0,0 +1,17 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://teploy.github.io/contracts/schema/backup-head-envelope.schema.json", + "title": "`teploy backup head --bytes N --host S --json` (backup-setup-v1, rev 21): the first bytes of a stored object, as stored, through the server's stored target. N is 1 to 262144; a larger N is refused, not clamped. A key outside the server's own and previous prefixes is refused. It never decrypts, so an encrypted object's data_base64 is ciphertext. Refusals and store failures use the error envelope.", + "type": "object", + "required": ["machine_interface", "host", "key", "encrypted", "requested_bytes", "length", "total_size", "data_base64"], + "properties": { + "machine_interface": {"type": "integer", "minimum": 1}, + "host": {"type": "string"}, + "key": {"type": "string", "pattern": "^s3://"}, + "encrypted": {"type": "boolean", "description": "the key ends in .age"}, + "requested_bytes": {"type": "integer", "minimum": 1, "maximum": 262144}, + "length": {"type": "integer", "minimum": 1, "maximum": 262144, "description": "bytes returned; less than requested_bytes when the object is shorter"}, + "total_size": {"type": ["integer", "null"], "minimum": 0, "description": "the whole object's size; null when the store did not say"}, + "data_base64": {"type": "string", "contentEncoding": "base64"} + } +} diff --git a/contracts/schema/backup-preflight-envelope.schema.json b/contracts/schema/backup-preflight-envelope.schema.json new file mode 100644 index 0000000..1bde892 --- /dev/null +++ b/contracts/schema/backup-preflight-envelope.schema.json @@ -0,0 +1,107 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://teploy.github.io/contracts/schema/backup-preflight-envelope.schema.json", + "title": "`teploy server backup-preflight --host S [--install-tools] [--upgrade-binary] --json` (backup-setup-v1, rev 21). Read-only without a flag. A probe that fails reads as unknown or null, never as fine. An unreachable server is a report with exit 0. With a flag, the exit status is nonzero when that action failed or was refused, and the report is printed either way, describing the server after the action.", + "type": "object", + "required": ["machine_interface", "host", "observed_at", "reachable", "binary", "tools", "disk", "target", "policy"], + "properties": { + "machine_interface": {"type": "integer", "minimum": 1}, + "host": {"type": "string"}, + "observed_at": {"type": "string", "format": "date-time"}, + "reachable": {"type": "boolean"}, + "error": {"type": "string", "description": "why the server did not answer, or why the probe did not finish"}, + "binary": { + "type": "object", + "required": ["path", "state", "fleet_engine", "cli_version", "same_as_cli"], + "properties": { + "path": {"type": "string"}, + "state": {"enum": ["ok", "missing", "no_fleet_engine", "unknown"]}, + "version": {"type": "string"}, + "fleet_engine": {"type": ["boolean", "null"], "description": "whether it speaks `backup run --all`; null is unknown"}, + "cli_version": {"type": "string"}, + "same_as_cli": {"type": ["boolean", "null"]}, + "detail": {"type": "string"} + } + }, + "tools": { + "type": "object", + "required": ["aws", "age", "python3"], + "properties": { + "aws": {"$ref": "#/$defs/tool"}, + "age": {"$ref": "#/$defs/tool"}, + "python3": {"$ref": "#/$defs/tool"} + } + }, + "disk": { + "type": "object", + "required": ["path", "free_bytes"], + "properties": { + "path": {"type": "string", "description": "where backups are built: the policy's tmp_dir, else /var/tmp"}, + "free_bytes": {"type": ["integer", "null"], "minimum": 0}, + "detail": {"type": "string"} + } + }, + "target": { + "type": "object", + "required": ["state", "has_credentials"], + "properties": { + "state": {"enum": ["configured", "absent", "unknown"]}, + "bucket": {"type": "string"}, + "endpoint": {"type": "string"}, + "prefix": {"type": "string"}, + "has_credentials": {"type": "boolean"}, + "detail": {"type": "string"} + } + }, + "policy": { + "type": "object", + "required": ["state", "encrypted", "cron_installed", "managed_elsewhere"], + "properties": { + "state": {"enum": ["configured", "absent", "unknown"]}, + "path": {"type": "string"}, + "interval": {"type": "string"}, + "encrypted": {"type": "boolean"}, + "heartbeat_hosts": {"type": "array", "items": {"type": "string"}, "description": "hosts only, never tokens: lets a consumer tell its own heartbeat from someone else's"}, + "cron_installed": {"type": ["boolean", "null"]}, + "managed_elsewhere": {"enum": ["no", "yes", "unknown"], "description": "yes: the policy file is a symlink, belongs to another user or cannot be replaced by this one, or it has an interval and this user's crontab has no line for it. See managed_reasons."}, + "managed_reasons": {"type": "array", "items": {"type": "string"}}, + "detail": {"type": "string"} + } + }, + "tools_install": { + "type": "array", + "description": "only with --install-tools; one row per tool, python3 first", + "items": { + "type": "object", + "required": ["tool", "action"], + "properties": { + "tool": {"enum": ["python3", "age", "aws"]}, + "action": {"enum": ["already_done", "done", "failed"]}, + "detail": {"type": "string"} + } + } + }, + "binary_upgrade": { + "type": "object", + "description": "only with --upgrade-binary", + "required": ["action"], + "properties": { + "action": {"enum": ["already_done", "done", "failed", "refused"]}, + "from": {"type": "string"}, + "to": {"type": "string"}, + "detail": {"type": "string"} + } + } + }, + "$defs": { + "tool": { + "type": "object", + "required": ["state"], + "properties": { + "state": {"enum": ["ok", "missing", "outdated", "unknown"], "description": "outdated: aws below the minimum version"}, + "version": {"type": "string"}, + "detail": {"type": "string"} + } + } + } +} diff --git a/internal/backup/accessory.go b/internal/backup/accessory.go index 7a28f01..b629a0b 100644 --- a/internal/backup/accessory.go +++ b/internal/backup/accessory.go @@ -63,13 +63,16 @@ func (c *Client) backupDeadline() time.Duration { return BackupDeadline } +// DefaultTmpDir is where a backup builds with no tmp_dir in its policy. +const DefaultTmpDir = "/var/tmp" + // backupTmpDir is where a backup builds its artifacts: disk, not the tmpfs /tmp // is on many hosts, which a database dump would fill along with the RAM. func backupTmpDir() string { if d := strings.TrimSpace(os.Getenv("TEPLOY_BACKUP_TMPDIR")); d != "" { return strings.TrimRight(d, "/") } - return "/var/tmp" + return DefaultTmpDir } func (c *Client) newBackupWorkspace(ctx context.Context) (string, func(), error) { diff --git a/internal/backup/backup.go b/internal/backup/backup.go index a18e86c..f18b96b 100644 --- a/internal/backup/backup.go +++ b/internal/backup/backup.go @@ -799,9 +799,10 @@ func (c *Client) SetSchedule(ctx context.Context, schedule, command, marker stri return nil } -// AWSCLIRemedy is the exact way to give a server the aws CLI. `teploy setup` -// runs AWSCLIInstallScript; Ubuntu 24.04+ has no apt package for it. -const AWSCLIRemedy = "run `teploy setup` against the server (it installs the distribution's package if that is new enough, else AWS's bundle with its signature checked), or on the server: AWS's installer as documented at https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html" +// AWSCLIRemedy is the exact way to give a server the aws CLI. `backup-preflight +// --install-tools` and `teploy setup` run AWSCLIInstallScript, and nothing else does: +// `backup-target set` and a run only check for it. Ubuntu 24.04+ has no apt package. +const AWSCLIRemedy = "run `teploy server backup-preflight --host --install-tools` (backup tools only) or `teploy setup` against the server (either installs the distribution's package if that is new enough, else AWS's bundle with its signature checked), or on the server: AWS's installer as documented at https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html" // The oldest aws CLIs teploy runs on: the first to send a checksum header on every // upload by default. A store with Object Lock (B2's included) refuses an upload with @@ -816,6 +817,15 @@ var awsCLIMinimum = fmt.Sprintf("2.%d.0 (or 1.%d.0)", awsCLIMinV2Minor, awsCLIMi var awsVersionRE = regexp.MustCompile(`aws-cli/(\d+)\.(\d+)\.(\d+)`) +// AWSCLIVersion reads the version out of `aws --version` output, "" when it is not one. +func AWSCLIVersion(versionOutput string) string { + m := awsVersionRE.FindStringSubmatch(versionOutput) + if m == nil { + return "" + } + return m[1] + "." + m[2] + "." + m[3] +} + // AWSCLIVersionProblem reads `aws --version` output and says what is wrong with it, // "" when nothing is, or when it is not a version (a wrapper): the first upload would // answer for that. @@ -888,6 +898,7 @@ export DEBIAN_FRONTEND=noninteractive if aws_ok; then exit 0; fi if apt-get install -y -qq awscli >/dev/null 2>&1 && aws_ok; then exit 0; fi apt-get remove -y -qq awscli >/dev/null 2>&1 || true +command -v curl >/dev/null 2>&1 || apt-get install -y -qq curl >/dev/null 2>&1 || { echo "curl is needed to download AWS's bundle" >&2; exit 1; } command -v gpg >/dev/null 2>&1 || apt-get install -y -qq gnupg >/dev/null 2>&1 || { echo "gpg is needed to verify AWS's bundle" >&2; exit 1; } d=$(mktemp -d /tmp/teploy-awscli.XXXXXXXX) trap 'rm -rf "$d"' EXIT diff --git a/internal/backup/head.go b/internal/backup/head.go new file mode 100644 index 0000000..efc91a7 --- /dev/null +++ b/internal/backup/head.go @@ -0,0 +1,139 @@ +package backup + +import ( + "context" + "encoding/base64" + "encoding/json" + "errors" + "fmt" + "strconv" + "strings" + "unicode" + + "github.com/useteploy/teploy/internal/ssh" +) + +// MaxObjectStartBytes caps a ranged read: enough for an age header and its first +// chunk, which is what proving a recovery key needs, and small enough to pass through +// a JSON envelope. +const MaxObjectStartBytes = 256 << 10 + +// ErrKeyOutsideTarget refuses a key that is not under this server's own prefix or one +// of its previous prefixes. +var ErrKeyOutsideTarget = errors.New("the key is outside this server's backup target") + +const objectStartBodyMark = "TEPLOY-OBJECT-START-BODY" + +// ObjectStart is the first bytes of a stored object, exactly as stored: nothing here +// decrypts. +type ObjectStart struct { + Key string + Data []byte + // TotalSize is the whole object's size, nil when the store's reply did not say. + TotalSize *int64 + Encrypted bool +} + +// CheckOwnKey refuses a key outside this config's bucket and its current and previous +// prefixes, and any key with a segment a store or client might normalize away. +func (s3 S3Config) CheckOwnKey(key string) error { + bucket, k, err := splitS3URI(key) + if err != nil { + return fmt.Errorf("%w: %v", ErrKeyOutsideTarget, err) + } + if bucket != s3.Bucket { + return fmt.Errorf("%w: it is in bucket %q, and this server's target is %q", ErrKeyOutsideTarget, bucket, s3.Bucket) + } + if strings.IndexFunc(k, unicode.IsControl) >= 0 { + return fmt.Errorf("%w: it contains control characters", ErrKeyOutsideTarget) + } + for _, seg := range strings.Split(k, "/") { + if seg == "" || seg == "." || seg == ".." { + return fmt.Errorf("%w: it has an empty, . or .. segment", ErrKeyOutsideTarget) + } + } + for _, loc := range s3.locations() { + if loc.Prefix == "" || strings.HasPrefix(k, loc.Prefix+"/") { + return nil + } + } + return fmt.Errorf("%w: it is not under the prefix %q or a previous prefix", ErrKeyOutsideTarget, s3.Prefix) +} + +// ReadObjectStart reads the first n bytes of an object through the stored target. The +// body sits in a private temp file only long enough to encode it, because aws writes +// its reply to stdout and the object to a file. The name is one the janitor knows, if +// a kill leaves it behind. +func (c *Client) ReadObjectStart(ctx context.Context, s3 S3Config, key string, n int) (ObjectStart, error) { + var none ObjectStart + if n < 1 || n > MaxObjectStartBytes { + return none, fmt.Errorf("--bytes must be between 1 and %d (256 KiB)", MaxObjectStartBytes) + } + if err := s3.CheckOwnKey(key); err != nil { + return none, err + } + if err := c.ensureAWSCLI(ctx); err != nil { + return none, err + } + bucket, k, err := splitS3URI(key) + if err != nil { + return none, err + } + q := ssh.ShellQuote + get := s3.AWS("s3api get-object --bucket " + q(bucket) + " --key " + q(k) + " --range " + q(fmt.Sprintf("bytes=0-%d", n-1)) + ` --output json "$d/o"`) + script := "umask 077\n" + + "d=$(mktemp -d " + q(backupTmpDir()+"/teploy-backup.XXXXXXXX") + ") || exit 3\n" + + `trap 'rm -rf -- "$d"' EXIT` + "\n" + + get + ` > "$d/m" || exit $?` + "\n" + + fmt.Sprintf(`[ "$(wc -c < "$d/o")" -le %d ] || { echo "the store ignored the requested range" >&2; exit 4; }`, n) + "\n" + + `cat "$d/m"` + "\n" + + `printf '\n%s\n' ` + objectStartBodyMark + "\n" + + `base64 < "$d/o"` + // Base64 of the cap, plus aws's reply: well under this. + res := ssh.RunDetailedLimit(ctx, c.exec, "sh -c "+q(script), 1<<20) + if res.Failed() { + reason := oneLine(res.ExitErrorText()) + if reason == "" { + reason = fmt.Sprintf("the command exited with status %d and gave no reason", res.ExitCode) + } + return none, fmt.Errorf("reading %s: %s", key, reason) + } + if res.Truncated { + return none, fmt.Errorf("reading %s: the reply was larger than the requested range", key) + } + meta, body, ok := strings.Cut(string(res.Stdout), objectStartBodyMark) + if !ok { + return none, fmt.Errorf("reading %s: unrecognized output", key) + } + data, err := base64.StdEncoding.DecodeString(strings.Join(strings.Fields(body), "")) + if err != nil { + return none, fmt.Errorf("reading %s: unreadable body: %w", key, err) + } + var reply struct { + ContentLength *int64 `json:"ContentLength"` + ContentRange string `json:"ContentRange"` + } + if err := json.Unmarshal([]byte(meta), &reply); err != nil { + return none, fmt.Errorf("reading %s: unexpected reply: %w", key, err) + } + if reply.ContentLength != nil && *reply.ContentLength != int64(len(data)) { + return none, fmt.Errorf("reading %s: the store said %d bytes and sent %d", key, *reply.ContentLength, len(data)) + } + if len(data) == 0 { + return none, fmt.Errorf("reading %s: the store sent no bytes", key) + } + return ObjectStart{Key: key, Data: data, TotalSize: contentRangeTotal(reply.ContentRange), Encrypted: strings.HasSuffix(k, ageSuffix)}, nil +} + +// contentRangeTotal reads "bytes 0-9/1234". +func contentRangeTotal(s string) *int64 { + i := strings.LastIndex(s, "/") + if i < 0 { + return nil + } + total, err := strconv.ParseInt(s[i+1:], 10, 64) + if err != nil { + return nil + } + return &total +} diff --git a/internal/backup/head_test.go b/internal/backup/head_test.go new file mode 100644 index 0000000..fb7cfcd --- /dev/null +++ b/internal/backup/head_test.go @@ -0,0 +1,168 @@ +package backup + +import ( + "bytes" + "context" + "errors" + "io" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/useteploy/teploy/internal/ssh" +) + +func TestCheckOwnKey(t *testing.T) { + s3 := S3Config{Bucket: "b", Prefix: "teploy/srv1", PreviousPrefixes: []PreviousPrefix{{Prefix: "old"}}} + for _, ok := range []string{ + "s3://b/teploy/srv1/app/accessories/db/20261104-030000.pgdump.age", + "s3://b/old/app/volumes/20261104-030000.tar.gz", + } { + if err := s3.CheckOwnKey(ok); err != nil { + t.Errorf("%s: %v", ok, err) + } + } + for _, bad := range []string{ + "s3://other/teploy/srv1/app/x", + "s3://b/teploy/srv2/app/x", + "s3://b/teploy/srv1-evil/app/x", + "s3://b/oldish/app/x", + "s3://b/teploy/srv1/../srv2/x", + "s3://b/teploy/srv1/./x", + "s3://b/teploy/srv1//x", + "s3://b/teploy/srv1/x\nrm", + "s3://b/", + "b/teploy/srv1/x", + "/etc/passwd", + } { + if err := s3.CheckOwnKey(bad); !errors.Is(err, ErrKeyOutsideTarget) { + t.Errorf("%q: %v", bad, err) + } + } + // A target with no prefix has the whole bucket. + if err := (S3Config{Bucket: "b"}).CheckOwnKey("s3://b/app/x"); err != nil { + t.Fatal(err) + } +} + +func TestReadObjectStartRefusesBeforeAnyCommand(t *testing.T) { + s3 := S3Config{Bucket: "b", Region: "r", Prefix: "mine"} + for _, tc := range []struct { + key string + n int + }{ + {"s3://b/mine/x", 0}, + {"s3://b/mine/x", -1}, + {"s3://b/mine/x", MaxObjectStartBytes + 1}, + {"s3://b/other/x", 1024}, + } { + mock := ssh.NewMockExecutor("h") + if _, err := NewClient(mock, io.Discard).ReadObjectStart(context.Background(), s3, tc.key, tc.n); err == nil { + t.Errorf("%+v: no refusal", tc) + } + if len(mock.Calls) != 0 { + t.Errorf("%+v ran %v", tc, mock.Calls) + } + } +} + +// fakeAWS is an aws on PATH whose get-object runs the given shell; "$out" is the +// outfile and "$range" the requested range. +func fakeAWS(t *testing.T, getObject string) { + t.Helper() + dir := t.TempDir() + script := `#!/bin/sh +if [ "$1" = "--version" ]; then echo "aws-cli/2.30.0 Python/3"; exit 0; fi +out= +range= +while [ $# -gt 0 ]; do + case "$1" in + --range) range=$2; shift;; + */o) out=$1;; + esac + shift +done +` + getObject + "\n" + if err := os.WriteFile(filepath.Join(dir, "aws"), []byte(script), 0o755); err != nil { + t.Fatal(err) + } + t.Setenv("PATH", dir+string(os.PathListSeparator)+os.Getenv("PATH")) + t.Setenv("TEPLOY_BACKUP_TMPDIR", t.TempDir()) +} + +func tmpEntries(t *testing.T) []os.DirEntry { + t.Helper() + entries, err := os.ReadDir(os.Getenv("TEPLOY_BACKUP_TMPDIR")) + if err != nil { + t.Fatal(err) + } + return entries +} + +// The real script through a real shell: binary bytes round trip, the range reaches +// aws, the temp file does not outlive the command. +func TestReadObjectStartReadsTheRange(t *testing.T) { + fakeAWS(t, `[ "$range" = "bytes=0-7" ] || { echo "wrong range $range" >&2; exit 9; } +printf 'age\377\000\001\002\003' > "$out" +echo '{"ContentLength": 8, "ContentRange": "bytes 0-7/4096"}'`) + s3 := S3Config{Bucket: "b", Region: "r", Prefix: "mine"} + got, err := NewClient(ssh.NewLocalExecutor(), io.Discard).ReadObjectStart(context.Background(), s3, "s3://b/mine/app/x.pgdump.age", 8) + if err != nil { + t.Fatal(err) + } + if !bytes.Equal(got.Data, []byte("age\xff\x00\x01\x02\x03")) { + t.Fatalf("%q", got.Data) + } + if got.TotalSize == nil || *got.TotalSize != 4096 || !got.Encrypted || got.Key != "s3://b/mine/app/x.pgdump.age" { + t.Fatalf("%+v", got) + } + if e := tmpEntries(t); len(e) != 0 { + t.Fatalf("left behind: %v", e) + } +} + +func TestReadObjectStartFailures(t *testing.T) { + s3 := S3Config{Bucket: "b", Region: "r"} + for _, tc := range []struct{ name, fake, want string }{ + {"store refuses", `echo "An error occurred (AccessDenied)" >&2; exit 254`, "AccessDenied"}, + {"silent failure still has a reason", `exit 2`, "no reason"}, + {"store ignored the range", `head -c 64 /dev/zero > "$out"; echo '{"ContentLength": 64}'`, "ignored the requested range"}, + {"length disagrees", `printf abc > "$out"; echo '{"ContentLength": 8}'`, "said 8 bytes and sent 3"}, + {"no bytes", `: > "$out"; echo '{"ContentLength": 0}'`, "no bytes"}, + } { + t.Run(tc.name, func(t *testing.T) { + fakeAWS(t, tc.fake) + _, err := NewClient(ssh.NewLocalExecutor(), io.Discard).ReadObjectStart(context.Background(), s3, "s3://b/x", 8) + if err == nil || !strings.Contains(err.Error(), tc.want) { + t.Fatalf("%v", err) + } + if e := tmpEntries(t); len(e) != 0 { + t.Fatalf("left behind: %v", e) + } + }) + } +} + +// It reads through the stored credentials, decrypting only those, and never asks age +// about the object. +func TestReadObjectStartCommand(t *testing.T) { + mock := ssh.NewMockExecutor("h", + ssh.MockCommand{Match: "which aws", Output: "aws-cli/2.30.0\n"}, + ssh.MockCommand{Match: "sh -c", Output: `{"ContentLength":3}` + "\n" + objectStartBodyMark + "\nYWJj\n"}, + ) + s3 := S3Config{Bucket: "b", Region: "r", StoredTarget: "/etc/teploy/backup-target.json"} + if _, err := NewClient(mock, io.Discard).ReadObjectStart(context.Background(), s3, "s3://b/x.age", 3); err != nil { + t.Fatal(err) + } + cmd := mock.Calls[len(mock.Calls)-1] + if !strings.Contains(cmd, "exec aws s3api get-object") || !strings.Contains(cmd, "/etc/teploy/backup-target.json") { + t.Fatalf("not through the stored target: %s", cmd) + } + if strings.Contains(cmd, "AWS_ACCESS_KEY_ID") || strings.Contains(cmd, "AWS_SECRET_ACCESS_KEY") { + t.Fatalf("credentials in the command: %s", cmd) + } + if strings.Count(cmd, "age -d") != 1 || !strings.Contains(cmd, `sys.stdout.write(json.load(open(sys.argv[1])).get("credentials")`) { + t.Fatalf("age is for the credentials and nothing else: %s", cmd) + } +} diff --git a/internal/backup/status.go b/internal/backup/status.go index 9fadfd2..898c93a 100644 --- a/internal/backup/status.go +++ b/internal/backup/status.go @@ -200,7 +200,9 @@ func ReadFleetStatus(ctx context.Context, exec ssh.Executor, now time.Time) (Fle Configured: target.Bucket != "", Bucket: target.Bucket, Region: target.Region, Endpoint: target.Endpoint, Prefix: target.Prefix, HasCredentials: target.HasCredentials, PreviousPrefixes: target.PreviousPrefixes, } + toolsRead := false if out, err := exec.Run(ctx, `command -v aws >/dev/null 2>&1 && v=$(aws --version 2>&1) && echo "aws $v"; command -v age >/dev/null 2>&1 && echo age; true`); err == nil { + toolsRead = true for _, t := range splitLines(out) { if t == "aws" || strings.HasPrefix(t, "aws ") { st.AWSCLI = true @@ -251,6 +253,10 @@ func ReadFleetStatus(ctx context.Context, exec ssh.Executor, now time.Time) (Fle } else { st.Target.Reachable = true } + } else if target.Bucket != "" { + // Not reachable has to say why: nothing tried the target, and an empty reason + // reads as the target failing. + st.Target.Error = awsAbsentCause(toolsRead) } if st.Target.Reachable { st.Target.IncomingLifecycle = c.incomingLifecycle(ctx, cfg) @@ -482,6 +488,14 @@ func ReadFleetStatus(ctx context.Context, exec ssh.Executor, now time.Time) (Fle return st, nil } +// awsAbsentCause is the plain reason a target was never tried. +func awsAbsentCause(toolsRead bool) string { + if !toolsRead { + return "could not check whether the aws CLI is on this server, so the target was not tried" + } + return "the aws CLI is not installed on this server, or does not run, so the target was not tried" +} + // incomingLifecycle reads the bucket's lifecycle rules for one that expires // /.incoming/. Read-only. func (c *Client) incomingLifecycle(ctx context.Context, s3 S3Config) string { diff --git a/internal/backup/status_cause_test.go b/internal/backup/status_cause_test.go new file mode 100644 index 0000000..7575f57 --- /dev/null +++ b/internal/backup/status_cause_test.go @@ -0,0 +1,79 @@ +package backup + +import ( + "context" + "errors" + "strings" + "testing" + "time" + + "github.com/useteploy/teploy/internal/events" + "github.com/useteploy/teploy/internal/ssh" +) + +// Not reachable always says why: this is what a consumer shows after "cannot reach". +func TestFleetStatusSaysWhyTheTargetWasNotReached(t *testing.T) { + base := []ssh.MockCommand{ + {Match: "if [ -d /etc/teploy ]", Output: "absent\n"}, + {Match: "for p in /etc/teploy/backup-target.json", Output: "path /etc/teploy/backup-target.json\n" + `{"v":"1","bucket":"b","region":"r"}`}, + {Match: "crontab -l", Output: ""}, + {Match: "ids=$(docker ps -aq", Output: ""}, + {Match: "for d in /deployments/*/volumes", Output: ""}, + {Match: "for d in '/var/tmp'/teploy-backup.", Output: ""}, + {Match: "python3 -c", Output: ""}, + } + now := time.Date(2026, 11, 6, 0, 0, 0, 0, time.UTC) + for _, tc := range []struct { + name string + extra ssh.MockCommand + want string + }{ + {"no aws", ssh.MockCommand{Match: "command -v aws >/dev/null", Output: "age\n"}, "aws CLI is not installed"}, + {"could not check", ssh.MockCommand{Match: "command -v aws >/dev/null", Err: errors.New("ssh: connection reset")}, "could not check"}, + } { + t.Run(tc.name, func(t *testing.T) { + st, err := ReadFleetStatus(context.Background(), ssh.NewMockExecutor("h", append(base, tc.extra)...), now) + if err != nil { + t.Fatal(err) + } + if st.Target.Reachable || !strings.Contains(st.Target.Error, tc.want) { + t.Fatalf("%+v", st.Target) + } + }) + } + // aws ran and said nothing: still a reason. + cmds := append(append([]ssh.MockCommand{}, base...), + ssh.MockCommand{Match: "command -v aws >/dev/null", Output: "aws aws-cli/2.30.0\n"}, + ssh.MockCommand{Match: "aws s3 ls", Err: errors.New("exit status 2")}, + ) + mock := ssh.NewMockExecutor("h", cmds...) + st, err := ReadFleetStatus(context.Background(), mock, now) + if err != nil { + t.Fatal(err) + } + if st.Target.Reachable || strings.HasSuffix(st.Target.Error, ": ") || st.Target.Error == "" { + t.Fatalf("%+v", st.Target) + } +} + +// A detached run is `backup run --all`: what makes its events count for coverage is +// policy_managed, and it is what a policy entry requires. +func TestPolicyManagedRunCountsForCoverage(t *testing.T) { + at := time.Date(2026, 11, 6, 1, 0, 0, 0, time.UTC) + ev := func(managed bool) events.Event { + d := map[string]any{"accessory": "db", "kind": KindAccessory} + if managed { + d["policy_managed"] = true + } + return events.Event{TS: at, Kind: events.KindBackupRun, Outcome: events.Succeeded, Detail: d} + } + e := StatusEntry{Schedule: "policy"} + fillEntryEvents(&e, []events.Event{ev(false)}, "db") + if e.LastSuccess != nil { + t.Fatal("a manual run must not count") + } + fillEntryEvents(&e, []events.Event{ev(true)}, "db") + if e.LastSuccess == nil || !e.LastSuccess.Equal(at) { + t.Fatalf("%+v", e) + } +} diff --git a/internal/backup/store.go b/internal/backup/store.go index a76ae48..69cbfc4 100644 --- a/internal/backup/store.go +++ b/internal/backup/store.go @@ -98,7 +98,7 @@ func (c *Client) serverRecipient(ctx context.Context) (string, error) { return c.serverRecip, nil } if out, err := c.exec.Run(ctx, "command -v age >/dev/null 2>&1 && command -v age-keygen >/dev/null 2>&1 && echo ok"); err != nil || strings.TrimSpace(out) != "ok" { - return "", fmt.Errorf("the age tool is not installed on this server, so nothing can be encrypted (run `teploy setup`, or apt-get install age)") + return "", fmt.Errorf("the age tool is not installed on this server, so nothing can be encrypted (run `teploy server backup-preflight --host --install-tools` or `teploy setup`, or apt-get install age)") } id, err := ensureAgeIdentity(ctx, c.exec) if err != nil { @@ -623,7 +623,11 @@ func (c *Client) listTree(ctx context.Context, s3 S3Config, rel string) ([]store if res.ExitCode == 1 && strings.TrimSpace(string(res.Stdout)) == "" && strings.TrimSpace(string(res.Stderr)) == "" { return nil, nil } - return nil, fmt.Errorf("listing %s: %s", root, oneLine(res.ExitErrorText())) + reason := oneLine(res.ExitErrorText()) + if reason == "" { + reason = fmt.Sprintf("aws exited with status %d and gave no reason", res.ExitCode) + } + return nil, fmt.Errorf("listing %s: %s", root, reason) } var out []storedKey for _, line := range strings.Split(string(res.Stdout), "\n") { @@ -1085,6 +1089,38 @@ const markerRecentWindow = 7 * 24 * time.Hour // ErrMarkerRecent marks a clear refused because the namespace is in use. var ErrMarkerRecent = errors.New("recently active backup namespace") +// CheckMarkersClearable says whether a clear here would be refused, removing nothing: +// ErrMarkerRecent if so, a plain error if this location did not answer. A clear across +// several locations asks every one first, so that a refusal removes nothing anywhere. +func (c *Client) CheckMarkersClearable(ctx context.Context, s3 S3Config, fingerprint string, now time.Time) error { + if !ownerMarkerRE.MatchString(fingerprint) { + return fmt.Errorf("invalid fingerprint %q", fingerprint) + } + markers, err := c.ListNamespaceMarkers(ctx, s3) + if err != nil { + return err + } + return c.checkMarkersIdle(ctx, s3, markers, fingerprint, now) +} + +func (c *Client) checkMarkersIdle(ctx context.Context, s3 S3Config, markers []NamespaceMarker, fingerprint string, now time.Time) error { + for _, m := range markers { + if m.Fingerprint != fingerprint { + continue + } + keys, err := c.listTree(ctx, s3, m.Namespace+"/") + if err != nil { + return fmt.Errorf("%w: could not tell how recently %s was written to, so nothing was removed (--force to remove it anyway): %v", ErrMarkerRecent, m.Namespace, err) + } + for _, k := range keys { + if !k.Modified.IsZero() && now.Sub(k.Modified) < markerRecentWindow { + return fmt.Errorf("%w: %s was written to on %s, so %s may be a live server, and without its marker a prune here deletes its backups; nothing was removed. If that server is really gone: --force", ErrMarkerRecent, m.Namespace, k.Modified.Format("2006-01-02"), fingerprint) + } + } + } + return nil +} + // ClearNamespaceMarkers removes a fingerprint's markers (a server that no longer // exists, a rebuild's old self), returning the keys removed. The marker is all that // stops another server's prune from deleting that server's objects, so a namespace @@ -1100,19 +1136,8 @@ func (c *Client) ClearNamespaceMarkers(ctx context.Context, s3 S3Config, fingerp return nil, err } if !force { - for _, m := range markers { - if m.Fingerprint != fingerprint { - continue - } - keys, err := c.listTree(ctx, s3, m.Namespace+"/") - if err != nil { - return nil, fmt.Errorf("%w: could not tell how recently %s was written to, so nothing was removed (--force to remove it anyway): %v", ErrMarkerRecent, m.Namespace, err) - } - for _, k := range keys { - if !k.Modified.IsZero() && now.Sub(k.Modified) < markerRecentWindow { - return nil, fmt.Errorf("%w: %s was written to on %s, so %s may be a live server, and without its marker a prune here deletes its backups; nothing was removed. If that server is really gone: --force", ErrMarkerRecent, m.Namespace, k.Modified.Format("2006-01-02"), fingerprint) - } - } + if err := c.checkMarkersIdle(ctx, s3, markers, fingerprint, now); err != nil { + return nil, err } } var removed []string diff --git a/internal/cli/backup.go b/internal/cli/backup.go index ae978bb..9acf798 100644 --- a/internal/cli/backup.go +++ b/internal/cli/backup.go @@ -46,6 +46,7 @@ func newBackupCmd(flags *Flags) *cobra.Command { cmd.AddCommand(newBackupRunCmd(flags)) cmd.AddCommand(newBackupStatusCmd(flags)) cmd.AddCommand(newBackupFetchCmd(flags)) + cmd.AddCommand(newBackupHeadCmd(flags)) cmd.AddCommand(newBackupKeygenCmd(flags)) return cmd diff --git a/internal/cli/backup_detach.go b/internal/cli/backup_detach.go new file mode 100644 index 0000000..419d42f --- /dev/null +++ b/internal/cli/backup_detach.go @@ -0,0 +1,234 @@ +package cli + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "os" + "os/signal" + "strconv" + "strings" + "time" + + "github.com/useteploy/teploy/internal/backup" + "github.com/useteploy/teploy/internal/events" + "github.com/useteploy/teploy/internal/ssh" +) + +// `backup run --all --detach`: the same `backup run --all` a cron line runs, started +// on the server so that it outlives this connection. Its flock, heartbeat, events and +// status are that run's own; this only starts it and looks once, briefly, at what it +// did. + +// Detached run states. +const ( + DetachStarted = "started" + DetachAlreadyRunning = "already_running" + DetachFinished = "finished" + DetachFailed = "failed" +) + +// backupDetachDTO is `backup run --all --detach --json` (backup-setup-v1). +type backupDetachDTO struct { + MachineInterface int `json:"machine_interface"` + Host string `json:"host"` + RequestedAt time.Time `json:"requested_at"` + // Status: started (still running when this looked), already_running (another run + // holds the lock; this one stood down), finished (it ended within the first + // seconds; see result), failed (it ended without a result; see error). + Status string `json:"status"` + PID int `json:"pid,omitempty"` + // Log is the run's progress on the server. + Log string `json:"log,omitempty"` + Result *backup.FleetRunResult `json:"result,omitempty"` + Error string `json:"error,omitempty"` +} + +const ( + detachMarker = "# teploy-backup-detach" + detachResultMark = "TEPLOY-DETACH-RESULT" + detachLogMark = "TEPLOY-DETACH-LOG" +) + +// detachWait is how many seconds the start script watches the run: long enough for +// one that finds the lock held, or no policy, to say so. +const detachWait = 3 + +// detachScript starts cmd in its own session with no descriptor on this connection, +// so ending the connection neither signals it nor waits for it. The exit status goes +// through a file: a shell cannot ask about a child in another session reliably. +func detachScript(cmd string) string { + return detachMarker + ` +umask 077 +[ -n "${HOME:-}" ] || { echo "this user has no home directory for the run's log" >&2; exit 3; } +d="$HOME/.config/teploy/backup-runs" +mkdir -p "$d" || exit 3 +find "$d" -type f -mtime +14 -exec rm -f {} + 2>/dev/null +b=$(mktemp "$d/run.XXXXXXXX") || exit 3 +W='b=$1; shift; "$@" >"$b.json" 2>"$b.log"; echo $? >"$b.rc.t"; mv -f "$b.rc.t" "$b.rc"' +if command -v setsid >/dev/null 2>&1; then + setsid nohup sh -c "$W" sh "$b" ` + cmd + ` /dev/null 2>&1 & +else + nohup sh -c "$W" sh "$b" ` + cmd + ` /dev/null 2>&1 & +fi +pid=$! +i=0 +while [ "$i" -lt ` + strconv.Itoa(detachWait) + ` ] && [ ! -e "$b.rc" ]; do sleep 1; i=$((i+1)); done +if [ ! -e "$b.rc" ]; then printf 'started %s %s\n' "$pid" "$b"; exit 0; fi +printf 'finished %s %s\n' "$pid" "$b" +echo ` + detachResultMark + ` +cat "$b.json" +echo ` + detachLogMark + ` +tail -n 5 "$b.log"` +} + +// parseDetachReply reads the start script's report into the DTO, and returns the base +// path of the run's files. +func parseDetachReply(out string, dto *backupDetachDTO) (string, error) { + head, rest, _ := strings.Cut(out, "\n") + // The path comes last: a home directory may have spaces in it. + f := strings.SplitN(head, " ", 3) + if len(f) < 3 || (f[0] != "started" && f[0] != "finished") { + return "", fmt.Errorf("starting the run: unrecognized reply from the server") + } + pid, err := strconv.Atoi(f[1]) + if err != nil { + return "", fmt.Errorf("starting the run: unrecognized reply from the server") + } + base := f[2] + dto.PID, dto.Log = pid, base+".log" + if f[0] == "started" { + dto.Status = DetachStarted + return base, nil + } + _, rest, _ = strings.Cut(rest, detachResultMark+"\n") + body, logTail, _ := strings.Cut(rest, detachLogMark) + var env backupRunEnvelope + if err := json.Unmarshal([]byte(body), &env); err != nil || env.Status == "" { + dto.Status = DetachFailed + dto.Error = "the run ended without a result" + if reason := lastNonEmptyLine(logTail); reason != "" { + dto.Error += ": " + reason + } + return base, nil + } + if env.Status == "skipped" { + dto.Status = DetachAlreadyRunning + return base, nil + } + dto.Status = DetachFinished + dto.Result = &env.FleetRunResult + return base, nil +} + +func lastNonEmptyLine(s string) string { + lines := strings.Split(strings.TrimSpace(s), "\n") + return oneLine(lines[len(lines)-1]) +} + +// fleetBinarySupportsAll is whether bin speaks `backup run --all`. +func fleetBinarySupportsAll(ctx context.Context, exec ssh.Executor, bin string) bool { + return !ssh.RunDetailed(ctx, exec, ssh.ShellQuote(bin)+" backup run --help 2>/dev/null | grep -q -- --all").Failed() +} + +// startDetachedRun starts the run and records that it did. It never upgrades the +// server's binary: one too old to run this is a refusal naming the consented path. +func startDetachedRun(ctx context.Context, exec ssh.Executor, ignoreSpace bool, actor events.Actor, now time.Time) (backupDetachDTO, error) { + dto := backupDetachDTO{MachineInterface: MachineInterface, Host: exec.Host(), RequestedAt: now} + bin := scheduledTeployPath() + if !fleetBinarySupportsAll(ctx, exec, bin) { + return dto, fmt.Errorf("%s on this server is missing or does not support `backup run --all`, so nothing was started: upgrade it with `teploy server backup-preflight --host %s --upgrade-binary`", bin, exec.Host()) + } + q := ssh.ShellQuote + cmd := q(bin) + " backup run --all --json" + if ignoreSpace { + cmd += " --ignore-space-check" + } + if actor.Client != "" { + who := actor.Client + if actor.Label != "" { + who += ":" + actor.Label + } + cmd += " --actor " + q(who) + } + if actor.Op != "" { + cmd += " --actor-op " + q(actor.Op) + } + res := ssh.RunDetailed(ctx, exec, "sh -c "+q(detachScript(cmd))) + if res.Failed() { + reason := oneLine(res.ExitErrorText()) + if reason == "" { + reason = fmt.Sprintf("the command exited with status %d and gave no reason", res.ExitCode) + } + err := fmt.Errorf("starting the run: %s", reason) + events.Record(ctx, exec, events.Event{App: backup.FleetApp, Kind: events.KindBackupDetach, Outcome: events.Failed, Detail: events.ErrDetail(map[string]any{}, err)}) + return dto, err + } + base, err := parseDetachReply(string(res.Stdout), &dto) + if err != nil { + return dto, err + } + detail := map[string]any{"status": dto.Status, "pid": dto.PID, "log": dto.Log} + outcome := events.Succeeded + switch dto.Status { + case DetachAlreadyRunning: + // Nothing of this one is worth keeping. + _, _ = exec.Run(ctx, "rm -f -- "+q(base)+" "+q(base+".json")+" "+q(base+".log")+" "+q(base+".rc")) + dto.PID, dto.Log = 0, "" + detail = map[string]any{"status": dto.Status} + outcome = events.Refused + case DetachFailed: + outcome = events.Failed + detail["error"] = dto.Error + } + events.Record(ctx, exec, events.Event{App: backup.FleetApp, Kind: events.KindBackupDetach, Outcome: outcome, Detail: detail}) + return dto, nil +} + +// detachExit is the command's exit status: an already-running run is an answer, not a +// failure, and one that has finished exits as `backup run --all` would have. +func detachExit(dto backupDetachDTO) error { + switch dto.Status { + case DetachFailed: + return errors.New(dto.Error) + case DetachFinished: + return fleetFailed(*dto.Result) + } + return nil +} + +func runBackupRunDetach(flags *Flags, ignoreSpace bool) error { + ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt) + defer cancel() + host, user, key, err := configResolveServerForTarget(flags) + if err != nil { + return err + } + executor, err := ssh.Connect(ctx, ssh.ConnectConfig{Host: host, User: user, KeyPath: key}) + if err != nil { + return err + } + defer executor.Close() + sctx, scancel := context.WithTimeout(ctx, 2*time.Minute) + defer scancel() + dto, err := startDetachedRun(sctx, executor, ignoreSpace, resolveActor(flags, cliBuildVersion), time.Now().UTC()) + if err != nil { + return err + } + if flags.JSON { + if encErr := json.NewEncoder(os.Stdout).Encode(dto); encErr != nil { + return errors.Join(detachExit(dto), encErr) + } + return detachExit(dto) + } + switch dto.Status { + case DetachStarted: + fmt.Printf("Backup run started on %s (pid %d). Progress: %s. Results reach `teploy backup status` and `teploy events` as a normal run's do.\n", dto.Host, dto.PID, dto.Log) + case DetachAlreadyRunning: + fmt.Printf("A backup run is already running on %s; this one stood down.\n", dto.Host) + case DetachFinished: + fmt.Printf("The run ended straight away: %s\n", dto.Result.Status) + } + return detachExit(dto) +} diff --git a/internal/cli/backup_detach_integration_test.go b/internal/cli/backup_detach_integration_test.go new file mode 100644 index 0000000..428882e --- /dev/null +++ b/internal/cli/backup_detach_integration_test.go @@ -0,0 +1,166 @@ +//go:build integration + +// The detached run with the real binary and its real flock, and the preflight's real +// probes, on this machine as an unprivileged user: policy and target under a temp HOME, +// aws and docker as stubs (docker is what holds a run open while the lock is tested). +// Nothing outside the temp directories is touched. A host with no setsid gets a real +// one through perl, and dash is used where there is one. +// +// go build -o /tmp/teploy-e2e ./cmd/teploy +// TEPLOY_DETACH_E2E=/tmp/teploy-e2e go test -tags integration -run TestDetachE2E -v ./internal/cli +package cli + +import ( + "context" + "encoding/json" + "os" + "os/exec" + "path/filepath" + "runtime" + "strconv" + "strings" + "testing" + "time" + + "github.com/useteploy/teploy/internal/backup" + "github.com/useteploy/teploy/internal/events" + "github.com/useteploy/teploy/internal/ssh" +) + +func writeE2E(t *testing.T, path, body string, mode os.FileMode) { + t.Helper() + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(path, []byte(body), mode); err != nil { + t.Fatal(err) + } +} + +func TestDetachE2E(t *testing.T) { + bin := os.Getenv("TEPLOY_DETACH_E2E") + if bin == "" { + t.Skip("set TEPLOY_DETACH_E2E to a built teploy binary") + } + if _, err := os.Stat("/etc/teploy"); err == nil { + t.Skip("this machine has /etc/teploy, which would shadow the temp policy") + } + home, stubs := t.TempDir(), t.TempDir() + release := filepath.Join(home, "release-docker") + writeE2E(t, filepath.Join(stubs, "aws"), "#!/bin/sh\nif [ \"$1\" = --version ]; then echo aws-cli/2.30.1 Python/3; exit 0; fi\nexit 1\n", 0o755) + // The run's first docker call waits here, holding the run lock, until released. + writeE2E(t, filepath.Join(stubs, "docker"), "#!/bin/sh\nwhile [ ! -e "+ssh.ShellQuote(release)+" ]; do sleep 1; done\nexit 1\n", 0o755) + if _, err := exec.LookPath("setsid"); err != nil { + writeE2E(t, filepath.Join(stubs, "setsid"), "#!/usr/bin/perl\nuse POSIX;\nPOSIX::setsid();\nexec @ARGV;\n", 0o755) + } + if _, err := os.Stat("/bin/dash"); err == nil { + if err := os.Symlink("/bin/dash", filepath.Join(stubs, "sh")); err != nil { + t.Fatal(err) + } + } + policy := `{"v":"1","retention":{"keep_last":3},"volumes":false}` + writeE2E(t, filepath.Join(home, ".config/teploy/backup-policy.json"), policy, 0o600) + writeE2E(t, filepath.Join(home, ".config/teploy/backup-target.json"), `{"v":"1","bucket":"b","region":"r"}`, 0o600) + t.Setenv("HOME", home) + t.Setenv("PATH", stubs+string(os.PathListSeparator)+os.Getenv("PATH")) + t.Setenv("TEPLOY_SERVER_BINARY", bin) + + ctx := context.Background() + local := localNoEvents{ssh.NewLocalExecutor()} + now := time.Now().UTC() + + // The preflight's real probes. + d := collectBackupPreflight(ctx, local, bin, "0.1.55", now) + t.Logf("preflight: %+v", d) + if !d.Reachable || d.Error != "" || d.Binary.FleetEngine == nil || !*d.Binary.FleetEngine || d.Binary.Version == "" { + t.Fatalf("%+v", d) + } + if d.Tools.AWS.State != PreflightOK || d.Tools.AWS.Version != "2.30.1" || d.Tools.Python3.State != PreflightOK { + t.Fatalf("%+v", d.Tools) + } + if d.Disk.FreeBytes == nil || *d.Disk.FreeBytes <= 0 || d.Target.State != PreflightConfigured || d.Policy.State != PreflightConfigured { + t.Fatalf("%+v %+v %+v", d.Disk, d.Target, d.Policy) + } + // stat -c is GNU's: elsewhere the owner is unreadable, which must read as unknown, + // and so must a crontab that could not be read. + want := ManagedNo + if runtime.GOOS != "linux" || d.Policy.CronInstalled == nil { + want = PreflightUnknown + } + if d.Policy.ManagedElsewhere != want { + t.Fatalf("%+v", d.Policy) + } + + // The first run takes the lock and stays in docker. That this returned at all says + // the run holds nothing of the connection: the executor waits for its pipes. + first, err := startDetachedRun(ctx, local, false, events.Actor{Client: "dash", Label: "e2e"}, now) + if err != nil { + t.Fatal(err) + } + t.Logf("first: %+v", first) + if first.Status != DetachStarted || first.PID == 0 { + t.Fatalf("%+v", first) + } + pgid, err := exec.Command("ps", "-o", "pgid=", "-p", strconv.Itoa(first.PID)).Output() + if err != nil { + t.Fatal(err) + } + if strings.TrimSpace(string(pgid)) != strconv.Itoa(first.PID) { + t.Fatalf("not a session of its own: pgid %s, pid %d", pgid, first.PID) + } + + // What a dropped connection sends, to the run's whole group. + if err := exec.Command("kill", "-HUP", "-"+strconv.Itoa(first.PID)).Run(); err != nil { + t.Fatal(err) + } + time.Sleep(time.Second) + if err := exec.Command("kill", "-0", strconv.Itoa(first.PID)).Run(); err != nil { + t.Fatal("the run did not survive a hangup") + } + + // A second one finds the real flock held. + second, err := startDetachedRun(ctx, local, false, events.Actor{Client: "dash", Label: "e2e"}, now) + if err != nil { + t.Fatal(err) + } + t.Logf("second: %+v", second) + if second.Status != DetachAlreadyRunning || detachExit(second) != nil { + t.Fatalf("%+v", second) + } + + // Released, the first ends as a normal run does, with its result and exit status. + writeE2E(t, release, "", 0o644) + base := strings.TrimSuffix(first.Log, ".log") + deadline := time.Now().Add(60 * time.Second) + for { + if _, err := os.Stat(base + ".rc"); err == nil { + break + } + if time.Now().After(deadline) { + runlog, _ := os.ReadFile(first.Log) + t.Fatalf("the run did not end: %s", runlog) + } + time.Sleep(time.Second) + } + body, err := os.ReadFile(base + ".json") + if err != nil { + t.Fatal(err) + } + var env backupRunEnvelope + if err := json.Unmarshal(body, &env); err != nil { + t.Fatalf("%v: %s", err, body) + } + if env.Status != backup.RunFailed || env.MachineInterface != MachineInterface { + t.Fatalf("%s", body) + } + + // The lock is free again: a third run is its own, and ends within the look. + third, err := startDetachedRun(ctx, local, false, events.Actor{}, now) + if err != nil { + t.Fatal(err) + } + t.Logf("third: %+v", third) + if third.Status != DetachFinished || third.Result == nil || detachExit(third) == nil { + t.Fatalf("%+v", third) + } +} diff --git a/internal/cli/backup_detach_test.go b/internal/cli/backup_detach_test.go new file mode 100644 index 0000000..627aef4 --- /dev/null +++ b/internal/cli/backup_detach_test.go @@ -0,0 +1,235 @@ +package cli + +import ( + "context" + "encoding/json" + "errors" + "github.com/useteploy/teploy/internal/backup" + "github.com/useteploy/teploy/internal/events" + "github.com/useteploy/teploy/internal/ssh" + "io" + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +func TestBackupRunDetachFlagRefusals(t *testing.T) { + flags := &Flags{Host: "h"} + for _, args := range [][]string{{"--detach"}, {"--detach", "--app", "x"}, {"--all", "--detach", "--app", "x"}, {"--all", "--detach", "--accessory", "x"}} { + cmd := newBackupRunCmd(flags) + cmd.SetArgs(args) + cmd.SetOut(io.Discard) + cmd.SetErr(io.Discard) + if err := cmd.Execute(); err == nil || !strings.Contains(err.Error(), "--detach") { + t.Fatalf("%v: %v", args, err) + } + } + // --host is not optional: there is no server to start it on. + cmd := newBackupRunCmd(&Flags{}) + cmd.SetArgs([]string{"--all", "--detach"}) + cmd.SetOut(io.Discard) + cmd.SetErr(io.Discard) + if err := cmd.Execute(); err == nil || !strings.Contains(err.Error(), "--host") { + t.Fatalf("%v", err) + } +} + +// Too old a binary is a refusal naming the consented way: never an implicit upgrade. +func TestDetachNeverUpgrades(t *testing.T) { + mock := ssh.NewMockExecutor("h", ssh.MockCommand{Match: "'/usr/local/bin/teploy' backup run --help", Err: errors.New("exit status 1")}) + _, err := startDetachedRun(context.Background(), mock, false, events.Actor{}, setupNow) + if err == nil || !strings.Contains(err.Error(), "--upgrade-binary") { + t.Fatalf("%v", err) + } + if len(mock.Calls) != 1 { + t.Fatalf("%v", mock.Calls) + } +} + +func detachMock(reply string, extra ...ssh.MockCommand) *ssh.MockExecutor { + cmds := []ssh.MockCommand{ + {Match: "'/usr/local/bin/teploy' backup run --help", Output: ""}, + {Match: "sh -c '" + detachMarker, Output: reply}, + {Match: "rm -f -- ", Output: ""}, + {Match: "python3 -c", Output: `{"seq":1,"ts":"2026-01-01T00:00:00Z"}`}, + } + return ssh.NewMockExecutor("192.0.2.10", append(cmds, extra...)...) +} + +func TestDetachCommandIsTheCronRunAndIsRecorded(t *testing.T) { + mock := detachMock("started 4242 /root/.config/teploy/backup-runs/run.abc\n") + d, err := startDetachedRun(context.Background(), mock, true, events.Actor{Client: "dash", Label: "ada", Op: "op1"}, setupNow) + if err != nil { + t.Fatal(err) + } + if d.Status != DetachStarted || d.PID != 4242 || d.Log != "/root/.config/teploy/backup-runs/run.abc.log" || d.Result != nil { + t.Fatalf("%+v", d) + } + if detachExit(d) != nil { + t.Fatal("started is not a failure") + } + var script string + for _, c := range mock.Calls { + if strings.Contains(c, detachMarker) { + script = c + } + } + // ShellQuote nests: the words are there, quoted twice. + for _, want := range []string{"/usr/local/bin/teploy", " backup run --all --json --ignore-space-check", "--actor", "dash:ada", "--actor-op", "op1", "setsid nohup"} { + if !strings.Contains(script, want) { + t.Fatalf("no %q in %s", want, script) + } + } + if len(mock.Inputs) != 1 || !strings.Contains(mock.Inputs[0], `"kind":"backup.detach"`) || !strings.Contains(mock.Inputs[0], `"outcome":"succeeded"`) || !strings.Contains(mock.Inputs[0], `"pid":4242`) { + t.Fatalf("%v", mock.Inputs) + } +} + +func TestDetachAlreadyRunningIsNotAFailure(t *testing.T) { + skipped, _ := json.Marshal(backupRunEnvelope{MachineInterface: MachineInterface, FleetRunResult: backup.FleetRunResult{Status: "skipped", StartedAt: setupNow, Targets: []backup.FleetTargetResult{}, Error: "another run holds the lock"}}) + mock := detachMock("finished 4243 /root/run.abc\n" + detachResultMark + "\n" + string(skipped) + "\n" + detachLogMark + "\nAnother backup run --all is running\n") + d, err := startDetachedRun(context.Background(), mock, false, events.Actor{}, setupNow) + if err != nil { + t.Fatal(err) + } + if d.Status != DetachAlreadyRunning || d.PID != 0 || d.Log != "" || detachExit(d) != nil { + t.Fatalf("%+v", d) + } + if len(mock.Inputs) != 1 || !strings.Contains(mock.Inputs[0], `"outcome":"refused"`) { + t.Fatalf("%v", mock.Inputs) + } + if !strings.Contains(strings.Join(mock.Calls, "\n"), "rm -f -- '/root/run.abc' '/root/run.abc.json'") { + t.Fatalf("%v", mock.Calls) + } +} + +func TestDetachRunThatEndsAtOnce(t *testing.T) { + failed, _ := json.Marshal(backupRunEnvelope{MachineInterface: MachineInterface, FleetRunResult: backup.FleetRunResult{Status: backup.RunFailed, StartedAt: setupNow, Targets: []backup.FleetTargetResult{}, Error: "no backup policy on this server"}}) + d, err := startDetachedRun(context.Background(), detachMock("finished 1 /r/run.a\n"+detachResultMark+"\n"+string(failed)+"\n"+detachLogMark+"\n"), false, events.Actor{}, setupNow) + if err != nil { + t.Fatal(err) + } + if d.Status != DetachFinished || d.Result == nil || d.Result.Error == "" { + t.Fatalf("%+v", d) + } + if e := detachExit(d); e == nil || !strings.Contains(e.Error(), "no backup policy") { + t.Fatalf("%v", e) + } + // Ended with no result at all: the log's last line is the reason. + mock := detachMock("finished 1 /r/run.a\n" + detachResultMark + "\n" + detachLogMark + "\nunknown flag: --json\n") + d, err = startDetachedRun(context.Background(), mock, false, events.Actor{}, setupNow) + if err != nil { + t.Fatal(err) + } + if d.Status != DetachFailed || !strings.Contains(d.Error, "unknown flag") || detachExit(d) == nil { + t.Fatalf("%+v", d) + } + if !strings.Contains(mock.Inputs[0], `"outcome":"failed"`) { + t.Fatalf("%v", mock.Inputs) + } + // Refused garbage rather than guessing. + if _, err := startDetachedRun(context.Background(), detachMock("what\n"), false, events.Actor{}, setupNow); err == nil { + t.Fatal("no error") + } +} + +// fakeServerBinary is a teploy that answers --help and runs body for backup run --all. +func fakeServerBinary(t *testing.T, body string) string { + t.Helper() + dir := t.TempDir() + path := filepath.Join(dir, "teploy") + script := "#!/bin/sh\ncase \"$*\" in\n*--help*) echo ' --all back up everything';;\n*)\n" + body + "\n;;\nesac\n" + if err := os.WriteFile(path, []byte(script), 0o755); err != nil { + t.Fatal(err) + } + t.Setenv("TEPLOY_SERVER_BINARY", path) + home := t.TempDir() + t.Setenv("HOME", home) + return dir +} + +// The real start script through a real shell: it returns while the run goes on, with +// nothing of the run attached to the connection. +func TestDetachScriptRunOutlivesTheStart(t *testing.T) { + marker := filepath.Join(t.TempDir(), "finished") + fakeServerBinary(t, `sleep 6; echo "$*" > `+ssh.ShellQuote(marker)) + started := time.Now() + d, err := startDetachedRun(context.Background(), localNoEvents{ssh.NewLocalExecutor()}, false, events.Actor{Client: "dash"}, setupNow) + if err != nil { + t.Fatal(err) + } + if d.Status != DetachStarted || d.PID == 0 || !strings.HasSuffix(d.Log, ".log") { + t.Fatalf("%+v", d) + } + if took := time.Since(started); took > 5500*time.Millisecond { + t.Fatalf("waited for the run: %s", took) + } + if _, err := os.Stat(marker); err == nil { + t.Fatal("already finished") + } + deadline := time.Now().Add(15 * time.Second) + for { + if b, err := os.ReadFile(marker); err == nil { + if !strings.Contains(string(b), "backup run --all --json") || !strings.Contains(string(b), "--actor dash") { + t.Fatalf("%s", b) + } + break + } + if time.Now().After(deadline) { + t.Fatal("the run did not outlive the start") + } + time.Sleep(200 * time.Millisecond) + } +} + +func TestDetachScriptSeesTheStandDown(t *testing.T) { + fakeServerBinary(t, `echo '{"machine_interface":2,"status":"skipped","started_at":"2026-11-06T00:00:00Z","duration_ms":0,"encrypted":false,"targets":[],"error":"another run holds the lock"}'; echo "Another backup run --all is running" >&2`) + d, err := startDetachedRun(context.Background(), localNoEvents{ssh.NewLocalExecutor()}, false, events.Actor{}, setupNow) + if err != nil { + t.Fatal(err) + } + if d.Status != DetachAlreadyRunning { + t.Fatalf("%+v", d) + } + entries, err := os.ReadDir(filepath.Join(os.Getenv("HOME"), ".config/teploy/backup-runs")) + if err != nil { + t.Fatal(err) + } + if len(entries) != 0 { + t.Fatalf("a stood-down run left %v", entries) + } +} + +func TestDetachScriptReportsARunThatDied(t *testing.T) { + fakeServerBinary(t, `echo "boom: cannot start" >&2; exit 2`) + d, err := startDetachedRun(context.Background(), localNoEvents{ssh.NewLocalExecutor()}, false, events.Actor{}, setupNow) + if err != nil { + t.Fatal(err) + } + if d.Status != DetachFailed || !strings.Contains(d.Error, "boom: cannot start") { + t.Fatalf("%+v", d) + } +} + +// localNoEvents runs commands locally and answers the event helper itself: the real +// one writes under /deployments. +type localNoEvents struct{ *ssh.LocalExecutor } + +func (l localNoEvents) RunStream(ctx context.Context, cmd string, stdout, stderr io.Writer) error { + if strings.HasPrefix(cmd, "python3 -c") { + _, err := io.WriteString(stdout, `{"seq":1,"ts":"2026-01-01T00:00:00Z"}`) + return err + } + return l.LocalExecutor.RunStream(ctx, cmd, stdout, stderr) +} + +// What a run under `--detach` records for a target is what coverage reads. +func TestFleetTargetEventIsPolicyManaged(t *testing.T) { + mock := ssh.NewMockExecutor("h", ssh.MockCommand{Match: "python3 -c", Output: `{"seq":1,"ts":"2026-01-01T00:00:00Z"}`}) + recordFleetTargetEvent(context.Background(), mock, backup.FleetTargetResult{Kind: backup.TargetAccessory, App: "blog", Accessory: "db", Status: backup.TargetOK}) + if len(mock.Inputs) != 1 || !strings.Contains(mock.Inputs[0], `"policy_managed":true`) { + t.Fatalf("%v", mock.Inputs) + } +} diff --git a/internal/cli/backup_fleet.go b/internal/cli/backup_fleet.go index 35caadd..af32556 100644 --- a/internal/cli/backup_fleet.go +++ b/internal/cli/backup_fleet.go @@ -753,7 +753,7 @@ func orDetected(engine string) string { func ensureFleetBinary(ctx context.Context, executor ssh.Executor) (string, error) { bin := scheduledTeployPath() supports := func() bool { - return !ssh.RunDetailed(ctx, executor, ssh.ShellQuote(bin)+" backup run --help 2>/dev/null | grep -q -- --all").Failed() + return fleetBinarySupportsAll(ctx, executor, bin) } if supports() { return bin, nil diff --git a/internal/cli/backup_head.go b/internal/cli/backup_head.go new file mode 100644 index 0000000..df42ff8 --- /dev/null +++ b/internal/cli/backup_head.go @@ -0,0 +1,95 @@ +package cli + +import ( + "context" + "encoding/base64" + "encoding/json" + "fmt" + "io" + "os" + "os/signal" + "time" + + "github.com/spf13/cobra" + "github.com/useteploy/teploy/internal/backup" + "github.com/useteploy/teploy/internal/ssh" +) + +// backupHeadDTO is `backup head --json` (backup-setup-v1). Data is the object's first +// bytes exactly as stored: ciphertext for an encrypted backup. Nothing here decrypts. +type backupHeadDTO struct { + MachineInterface int `json:"machine_interface"` + Host string `json:"host"` + Key string `json:"key"` + Encrypted bool `json:"encrypted"` + RequestedBytes int `json:"requested_bytes"` + Length int `json:"length"` + // TotalSize is null when the store's reply did not say. + TotalSize *int64 `json:"total_size"` + DataBase64 string `json:"data_base64"` +} + +func newBackupHeadDTO(host string, requested int, o backup.ObjectStart) backupHeadDTO { + return backupHeadDTO{ + MachineInterface: MachineInterface, Host: host, Key: o.Key, Encrypted: o.Encrypted, RequestedBytes: requested, + Length: len(o.Data), TotalSize: o.TotalSize, DataBase64: base64.StdEncoding.EncodeToString(o.Data), + } +} + +func newBackupHeadCmd(flags *Flags) *cobra.Command { + var n int + cmd := &cobra.Command{ + Use: "head ", + Short: "Read the first bytes of a backup object through the server's stored target", + Long: "A ranged read of one object, as stored, printed as base64: it never decrypts.\n" + + "Used to prove a recovery key somewhere the key is, and the server's credentials\n" + + "are not. The key is the s3:// URI from `teploy backup list` or `backup status`,\n" + + "and must be under this server's prefix or a previous prefix. --bytes is at most\n" + + "262144 (256 KiB). --host names the server.", + Args: cobra.ExactArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt) + defer cancel() + if n < 1 || n > backup.MaxObjectStartBytes { + return fmt.Errorf("--bytes must be between 1 and %d (256 KiB)", backup.MaxObjectStartBytes) + } + host, user, key, err := configResolveServerForTarget(flags) + if err != nil { + return err + } + executor, err := ssh.Connect(ctx, ssh.ConnectConfig{Host: host, User: user, KeyPath: key}) + if err != nil { + return err + } + defer executor.Close() + rctx, rcancel := context.WithTimeout(ctx, 2*time.Minute) + defer rcancel() + dto, err := readBackupHead(rctx, executor, args[0], n) + if err != nil { + return err + } + if flags.JSON { + return json.NewEncoder(os.Stdout).Encode(dto) + } + fmt.Println(dto.DataBase64) + return nil + }, + } + cmd.Flags().IntVar(&n, "bytes", 0, "how many bytes to read from the start of the object (at most 262144)") + return cmd +} + +func readBackupHead(ctx context.Context, exec ssh.Executor, key string, n int) (backupHeadDTO, error) { + target, err := backup.ReadTarget(ctx, exec) + if err != nil { + return backupHeadDTO{}, err + } + if !TargetConfigured(target) { + return backupHeadDTO{}, fmt.Errorf("no backup target is configured for this server, so no key belongs to it") + } + o, err := backup.NewClient(exec, io.Discard).ReadObjectStart(ctx, TargetS3(target), key, n) + if err != nil { + return backupHeadDTO{}, err + } + return newBackupHeadDTO(exec.Host(), n, o), nil +} diff --git a/internal/cli/backup_head_test.go b/internal/cli/backup_head_test.go new file mode 100644 index 0000000..2ee36e2 --- /dev/null +++ b/internal/cli/backup_head_test.go @@ -0,0 +1,46 @@ +package cli + +import ( + "context" + "encoding/base64" + "encoding/json" + "github.com/useteploy/teploy/internal/backup" + "github.com/useteploy/teploy/internal/ssh" + "strings" + "testing" +) + +func TestBackupHeadThroughTheStoredTarget(t *testing.T) { + body := base64.StdEncoding.EncodeToString([]byte("age-encryption.org/v1\n")) + mock := ssh.NewMockExecutor("192.0.2.10", + ssh.MockCommand{Match: "for p in /etc/teploy/backup-target.json", Output: targetStored}, + ssh.MockCommand{Match: "which aws", Output: "aws-cli/2.30.0\n"}, + ssh.MockCommand{Match: "sh -c", Output: `{"ContentLength":22,"ContentRange":"bytes 0-21/1048576"}` + "\nTEPLOY-OBJECT-START-BODY\n" + body + "\n"}, + ) + key := "s3://b/teploy/srv1/app/accessories/db/20261104-030000.pgdump.age" + d, err := readBackupHead(context.Background(), mock, key, 22) + if err != nil { + t.Fatal(err) + } + if d.DataBase64 != body || d.Length != 22 || *d.TotalSize != 1048576 || !d.Encrypted || d.RequestedBytes != 22 { + t.Fatalf("%+v", d) + } + out, _ := json.Marshal(d) + if strings.Contains(string(out), `"x"`) { + t.Fatalf("credentials in output: %s", out) + } + // Refusals. + for _, tc := range []struct { + key string + n int + }{{"s3://b/teploy/srv2/app/x", 22}, {"s3://other/teploy/srv1/app/x", 22}, {key, backup.MaxObjectStartBytes + 1}, {key, 0}} { + mock := ssh.NewMockExecutor("h", ssh.MockCommand{Match: "for p in /etc/teploy/backup-target.json", Output: targetStored}) + if _, err := readBackupHead(context.Background(), mock, tc.key, tc.n); err == nil { + t.Fatalf("%+v was read", tc) + } + } + mock = ssh.NewMockExecutor("h", ssh.MockCommand{Match: "for p in /etc/teploy/backup-target.json", Output: "absent\n"}) + if _, err := readBackupHead(context.Background(), mock, key, 22); err == nil { + t.Fatal("no target, and it read anyway") + } +} diff --git a/internal/cli/backup_owners_test.go b/internal/cli/backup_owners_test.go new file mode 100644 index 0000000..e2a2c25 --- /dev/null +++ b/internal/cli/backup_owners_test.go @@ -0,0 +1,155 @@ +package cli + +import ( + "context" + "errors" + "github.com/useteploy/teploy/internal/backup" + "github.com/useteploy/teploy/internal/ssh" + "io" + "strings" + "testing" + "time" +) + +var ownersNow = time.Date(2026, 11, 6, 12, 0, 0, 0, time.UTC) + +// owners with a key limited to the current prefix: previous locations answer +// AccessDenied. +func ownersMock(extra ...ssh.MockCommand) *ssh.MockExecutor { + cmds := []ssh.MockCommand{{Match: "{ cat /etc/machine-id", Output: "0123456789abcdef0123456789abcdef -\n"}} + cmds = append(cmds, extra...) + cmds = append(cmds, ssh.MockCommand{Match: "python3 -c", Output: `{"seq":1,"ts":"2026-01-01T00:00:00Z"}`}) + return ssh.NewMockExecutor("h", cmds...) +} + +func ownersLocs() []backup.S3Config { + return []backup.S3Config{{Bucket: "b", Region: "r", Prefix: "cur"}, {Bucket: "b", Region: "r", Prefix: ""}, {Bucket: "b", Region: "r", Prefix: "old"}} +} + +const fp = "aaaaaaaaaaaaaaaa" + +var ( + lsCur = "aws s3 ls 's3://b/cur/" + lsRoot = "aws s3 ls 's3://b/'" + lsOld = "aws s3 ls 's3://b/old/" + denied = errors.New("exit status 254: An error occurred (AccessDenied) when calling ListObjectsV2") +) + +func TestOwnersSkipsPreviousLocationsItCannotReach(t *testing.T) { + stale := ownersNow.Add(-30 * 24 * time.Hour).Format("2006-01-02 15:04:05") + mock := ownersMock( + ssh.MockCommand{Match: lsCur, Output: stale + " 20 cur/app/.owners/" + fp + "\n"}, + ssh.MockCommand{Match: lsRoot, Err: denied}, + ssh.MockCommand{Match: lsOld, Output: stale + " 20 old/app/.owners/" + fp + "\n"}, + ) + markers, skipped, err := listOwnerMarkers(context.Background(), backup.NewClient(mock, io.Discard), ownersLocs()) + if err != nil { + t.Fatal(err) + } + if len(markers) != 2 || len(skipped) != 1 || skipped[0].Prefix != "" || !strings.Contains(skipped[0].Reason, "AccessDenied") { + t.Fatalf("%+v %+v", markers, skipped) + } + // The current location not answering is not "no owners". + mock = ownersMock(ssh.MockCommand{Match: lsCur, Err: denied}) + if _, _, err := listOwnerMarkers(context.Background(), backup.NewClient(mock, io.Discard), ownersLocs()); err == nil { + t.Fatal("no error") + } +} + +func TestOwnersClearDoesNotStopPartway(t *testing.T) { + now := ownersNow + stale := now.Add(-30 * 24 * time.Hour).Format("2006-01-02 15:04:05") + t.Run("key limited to the current prefix", func(t *testing.T) { + mock := ownersMock( + ssh.MockCommand{Match: lsCur, Output: stale + " 20 cur/app/.owners/" + fp + "\n"}, + ssh.MockCommand{Match: lsRoot, Err: denied}, + ssh.MockCommand{Match: lsOld, Err: denied}, + ssh.MockCommand{Match: "aws s3 rm", Output: ""}, + ) + n, skipped, err := clearOwnerMarkers(context.Background(), mock, backup.NewClient(mock, io.Discard), ownersLocs(), fp, false, now) + if err != nil || n != 1 || len(skipped) != 2 { + t.Fatalf("%d %+v %v", n, skipped, err) + } + if !strings.Contains(mock.Inputs[len(mock.Inputs)-1], `"skipped_locations"`) { + t.Fatalf("%v", mock.Inputs) + } + }) + t.Run("an unreachable location in the middle does not stop the last", func(t *testing.T) { + mock := ownersMock( + ssh.MockCommand{Match: lsCur, Output: stale + " 20 cur/app/.owners/" + fp + "\n"}, + ssh.MockCommand{Match: lsRoot, Err: denied}, + ssh.MockCommand{Match: lsOld, Output: stale + " 20 old/app/.owners/" + fp + "\n"}, + ssh.MockCommand{Match: "aws s3 rm", Output: ""}, + ) + n, skipped, err := clearOwnerMarkers(context.Background(), mock, backup.NewClient(mock, io.Discard), ownersLocs(), fp, true, now) + if err != nil || n != 2 || len(skipped) != 1 { + t.Fatalf("%d %+v %v", n, skipped, err) + } + }) + t.Run("the current location failing still lets the others go, and is the error", func(t *testing.T) { + mock := ownersMock( + ssh.MockCommand{Match: lsCur, Err: denied}, + ssh.MockCommand{Match: lsRoot, Output: ""}, + ssh.MockCommand{Match: lsOld, Output: stale + " 20 old/app/.owners/" + fp + "\n"}, + ssh.MockCommand{Match: "aws s3 rm", Output: ""}, + ) + n, _, err := clearOwnerMarkers(context.Background(), mock, backup.NewClient(mock, io.Discard), ownersLocs(), fp, false, now) + if err == nil || !strings.Contains(err.Error(), "AccessDenied") || n != 1 { + t.Fatalf("%d %v", n, err) + } + }) + t.Run("a removal failing in a previous location is reported and the rest go on", func(t *testing.T) { + mock := ownersMock( + ssh.MockCommand{Match: lsCur, Output: stale + " 20 cur/app/.owners/" + fp + "\n"}, + ssh.MockCommand{Match: lsRoot, Output: ""}, + ssh.MockCommand{Match: lsOld, Output: stale + " 20 old/app/.owners/" + fp + "\n"}, + ssh.MockCommand{Match: "aws s3 rm 's3://b/old/", Err: denied}, + ssh.MockCommand{Match: "aws s3 rm", Output: ""}, + ) + n, skipped, err := clearOwnerMarkers(context.Background(), mock, backup.NewClient(mock, io.Discard), ownersLocs(), fp, false, now) + if err != nil || n != 1 || len(skipped) != 1 || skipped[0].Prefix != "old" { + t.Fatalf("%d %+v %v", n, skipped, err) + } + }) + t.Run("a refusal anywhere removes nothing anywhere", func(t *testing.T) { + recent := now.Add(-time.Hour).Format("2006-01-02 15:04:05") + mock := ownersMock( + ssh.MockCommand{Match: lsCur, Output: stale + " 20 cur/app/.owners/" + fp + "\n"}, + ssh.MockCommand{Match: lsRoot, Err: denied}, + ssh.MockCommand{Match: lsOld, Output: recent + " 20 old/app/.owners/" + fp + "\n"}, + ) + n, _, err := clearOwnerMarkers(context.Background(), mock, backup.NewClient(mock, io.Discard), ownersLocs(), fp, false, now) + if !errors.Is(err, backup.ErrMarkerRecent) || n != 0 { + t.Fatalf("%d %v", n, err) + } + for _, c := range mock.Calls { + if strings.HasPrefix(c, "aws s3 rm") { + t.Fatalf("removed after a refusal: %s", c) + } + } + if !strings.Contains(mock.Inputs[len(mock.Inputs)-1], `"outcome":"refused"`) { + t.Fatalf("%v", mock.Inputs) + } + }) +} + +func TestAWSInstallScriptStaysVerified(t *testing.T) { + for _, want := range []string{"VALIDSIG", "curl -fsSL"} { + if !strings.Contains(backup.AWSCLIInstallScript, want) { + t.Fatalf("no %q", want) + } + } + if strings.Index(backup.AWSCLIInstallScript, "command -v curl") > strings.Index(backup.AWSCLIInstallScript, "curl -fsSL") { + t.Fatal("curl is installed after it is used") + } +} + +func TestAWSRemedyNamesTheToolsOnlyPath(t *testing.T) { + if !strings.Contains(backup.AWSCLIRemedy, "backup-preflight") || !strings.Contains(backup.AWSCLIRemedy, "--install-tools") { + t.Fatal(backup.AWSCLIRemedy) + } + h := newServerBackupTargetSetCmd(&Flags{}).Long + if !strings.Contains(h, "does not install the aws CLI") { + t.Fatal(h) + } +} diff --git a/internal/cli/backup_preflight.go b/internal/cli/backup_preflight.go new file mode 100644 index 0000000..3bea730 --- /dev/null +++ b/internal/cli/backup_preflight.go @@ -0,0 +1,719 @@ +package cli + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "os" + "os/signal" + "strconv" + "strings" + "time" + + "github.com/spf13/cobra" + "github.com/useteploy/teploy/internal/backup" + "github.com/useteploy/teploy/internal/events" + "github.com/useteploy/teploy/internal/secret" + "github.com/useteploy/teploy/internal/ssh" +) + +// `teploy server backup-preflight`: what a server needs before backups can start, +// asked before anyone is asked for a key. A probe that fails reads as unknown, never +// as fine. --install-tools and --upgrade-binary are the only things here that change +// the server, each only on request. + +// Preflight states. +const ( + PreflightOK = "ok" + PreflightMissing = "missing" + PreflightOld = "outdated" + PreflightNoFleet = "no_fleet_engine" + PreflightUnknown = "unknown" + + PreflightConfigured = "configured" + PreflightAbsent = "absent" + + ManagedNo = "no" + ManagedYes = "yes" +) + +// Action results of --install-tools and --upgrade-binary. +const ( + ActionAlreadyDone = "already_done" + ActionDone = "done" + ActionFailed = "failed" + ActionRefused = "refused" +) + +// preflightTool is one tool. State: ok, missing, outdated (aws below its floor) or +// unknown. +type preflightTool struct { + State string `json:"state"` + Version string `json:"version,omitempty"` + Detail string `json:"detail,omitempty"` +} + +type preflightTools struct { + AWS preflightTool `json:"aws"` + Age preflightTool `json:"age"` + Python3 preflightTool `json:"python3"` +} + +// preflightBinary is the server's teploy, at the path cron lines run. State: ok, +// missing, no_fleet_engine or unknown. Nulls mean unknown. +type preflightBinary struct { + Path string `json:"path"` + State string `json:"state"` + Version string `json:"version,omitempty"` + FleetEngine *bool `json:"fleet_engine"` + CLIVersion string `json:"cli_version"` + SameAsCLI *bool `json:"same_as_cli"` + Detail string `json:"detail,omitempty"` +} + +// preflightDisk is the free space where backups are built. FreeBytes null: unknown. +type preflightDisk struct { + Path string `json:"path"` + FreeBytes *int64 `json:"free_bytes"` + Detail string `json:"detail,omitempty"` +} + +// preflightTarget is the stored target. State: configured, absent or unknown. +type preflightTarget struct { + State string `json:"state"` + Bucket string `json:"bucket,omitempty"` + Endpoint string `json:"endpoint,omitempty"` + Prefix string `json:"prefix,omitempty"` + HasCredentials bool `json:"has_credentials"` + Detail string `json:"detail,omitempty"` +} + +// preflightPolicy is the stored policy. State: configured, absent or unknown. +// ManagedElsewhere says whether something other than this user looks after it (no, +// yes or unknown), with the evidence in ManagedReasons; HeartbeatHosts lets a +// consumer tell its own heartbeat from someone else's. +type preflightPolicy struct { + State string `json:"state"` + Path string `json:"path,omitempty"` + Interval string `json:"interval,omitempty"` + Encrypted bool `json:"encrypted"` + HeartbeatHosts []string `json:"heartbeat_hosts,omitempty"` + CronInstalled *bool `json:"cron_installed"` + ManagedElsewhere string `json:"managed_elsewhere"` + ManagedReasons []string `json:"managed_reasons,omitempty"` + Detail string `json:"detail,omitempty"` +} + +// preflightToolAction is one tool's result under --install-tools. +type preflightToolAction struct { + Tool string `json:"tool"` + Action string `json:"action"` + Detail string `json:"detail,omitempty"` +} + +// preflightUpgrade is the result of --upgrade-binary. +type preflightUpgrade struct { + Action string `json:"action"` + From string `json:"from,omitempty"` + To string `json:"to,omitempty"` + Detail string `json:"detail,omitempty"` +} + +// backupPreflightDTO is `server backup-preflight --json` (backup-setup-v1). The +// command exits nonzero only when a requested action failed or was refused; the report +// is printed either way. +type backupPreflightDTO struct { + MachineInterface int `json:"machine_interface"` + Host string `json:"host"` + ObservedAt time.Time `json:"observed_at"` + Reachable bool `json:"reachable"` + Error string `json:"error,omitempty"` + Binary preflightBinary `json:"binary"` + Tools preflightTools `json:"tools"` + Disk preflightDisk `json:"disk"` + Target preflightTarget `json:"target"` + Policy preflightPolicy `json:"policy"` + // ToolsInstall and BinaryUpgrade appear only when asked for. + ToolsInstall []preflightToolAction `json:"tools_install,omitempty"` + BinaryUpgrade *preflightUpgrade `json:"binary_upgrade,omitempty"` +} + +func newServerBackupPreflightCmd(flags *Flags) *cobra.Command { + var installTools, upgradeBinary bool + cmd := &cobra.Command{ + Use: "backup-preflight", + Short: "Report what this server needs before backups can start", + Long: "Read-only by default: whether the server answers, its teploy binary's version\n" + + "and whether that has the fleet engine, aws against its minimum version, age,\n" + + "python3, free disk where backups are built, and whether a target and policy\n" + + "exist and something else looks after the policy. A probe that fails reads as\n" + + "unknown, never as fine. An unreachable server is a report, not an error.\n\n" + + "--install-tools installs aws, age and python3 where missing, and nothing else\n" + + "(no hardening, no other `teploy setup` steps). --upgrade-binary installs the\n" + + "release matching this CLI's version as the server's teploy, verified before it\n" + + "replaces the old one; it never downgrades. Nothing upgrades without this flag.\n" + + "Either exits nonzero when it fails or refuses, after printing the report.", + Args: cobra.NoArgs, + RunE: func(cmd *cobra.Command, args []string) error { + return runServerBackupPreflight(flags, installTools, upgradeBinary) + }, + } + cmd.Flags().BoolVar(&installTools, "install-tools", false, "install aws, age and python3 where missing (nothing else)") + cmd.Flags().BoolVar(&upgradeBinary, "upgrade-binary", false, "install this CLI's version as the server's teploy binary") + return cmd +} + +func runServerBackupPreflight(flags *Flags, installTools, upgradeBinary bool) error { + ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt) + defer cancel() + host, user, key, err := configResolveServerForTarget(flags) + if err != nil { + return err + } + now := time.Now().UTC() + var dto backupPreflightDTO + var actionErr error + executor, err := ssh.Connect(ctx, ssh.ConnectConfig{Host: host, User: user, KeyPath: key}) + if err != nil { + dto = unreachablePreflight(host, scheduledTeployPath(), cliBuildVersion, err, now) + if installTools || upgradeBinary { + actionErr = fmt.Errorf("this server did not answer, so nothing was installed: %w", err) + } + } else { + defer executor.Close() + dto, actionErr = backupPreflightOn(ctx, executor, preflightOptions{ + InstallTools: installTools, UpgradeBinary: upgradeBinary, Binary: scheduledTeployPath(), CLIVersion: cliBuildVersion, Now: now, InstallRelease: installTeployRelease, + }) + } + if flags.JSON { + return errors.Join(actionErr, json.NewEncoder(os.Stdout).Encode(dto)) + } + printBackupPreflight(dto) + return actionErr +} + +type preflightOptions struct { + InstallTools, UpgradeBinary bool + Binary, CLIVersion string + Now time.Time + // InstallRelease is the release pipeline (a seam: tests have no network). + InstallRelease func(ctx context.Context, exec ssh.Executor, remotePath, tag string, capabilities ...string) (string, error) +} + +// backupPreflightOn reports, acts if asked, and reports again after acting so the +// report describes the server as it now is. +func backupPreflightOn(ctx context.Context, exec ssh.Executor, o preflightOptions) (backupPreflightDTO, error) { + pctx, cancel := context.WithTimeout(ctx, 2*time.Minute) + dto := collectBackupPreflight(pctx, exec, o.Binary, o.CLIVersion, o.Now) + cancel() + if !o.InstallTools && !o.UpgradeBinary { + return dto, nil + } + var errs []error + var actions []preflightToolAction + var upgrade *preflightUpgrade + if o.InstallTools { + actx, acancel := context.WithTimeout(ctx, 20*time.Minute) + var err error + actions, err = installBackupTools(actx, exec, dto.Tools, o.Binary, o.CLIVersion) + acancel() + errs = append(errs, err) + } + if o.UpgradeBinary { + actx, acancel := context.WithTimeout(ctx, 10*time.Minute) + u, err := upgradeServerBinary(actx, exec, dto.Binary, o) + acancel() + upgrade = &u + errs = append(errs, err) + } + pctx, cancel = context.WithTimeout(ctx, 2*time.Minute) + defer cancel() + dto = collectBackupPreflight(pctx, exec, o.Binary, o.CLIVersion, o.Now) + dto.ToolsInstall, dto.BinaryUpgrade = actions, upgrade + return dto, errors.Join(errs...) +} + +// unreachablePreflight is the report for a server that did not answer: everything is +// unknown, which is what the tri-state exists for. +func unreachablePreflight(host, bin, cliVersion string, err error, now time.Time) backupPreflightDTO { + dto := unknownPreflight(host, bin, cliVersion, now) + dto.Error = oneLine(err.Error()) + return dto +} + +func unknownPreflight(host, bin, cliVersion string, now time.Time) backupPreflightDTO { + unknown := preflightTool{State: PreflightUnknown} + return backupPreflightDTO{ + MachineInterface: MachineInterface, Host: host, ObservedAt: now, + Binary: preflightBinary{Path: bin, State: PreflightUnknown, CLIVersion: cliVersion}, + Tools: preflightTools{AWS: unknown, Age: unknown, Python3: unknown}, + Disk: preflightDisk{Path: backup.DefaultTmpDir}, + Target: preflightTarget{State: PreflightUnknown}, + Policy: preflightPolicy{State: PreflightUnknown, ManagedElsewhere: PreflightUnknown}, + } +} + +const ( + preflightProbeMarker = "# teploy-preflight-probe" + preflightManagedMarker = "# teploy-preflight-managed" +) + +// preflightProbeScript prints tab-separated facts and, last, "end": a run cut short +// leaves the rest unknown. It sees the PATH a run sees, not this session's. +func preflightProbeScript(bin, tmpDir string) string { + q := ssh.ShellQuote + return preflightProbeMarker + ` +PATH="$PATH:` + strings.Join(serverToolPaths, ":") + `" +b=` + q(bin) + ` +if [ -e "$b" ]; then + if v=$("$b" version 2>/dev/null); then printf 'binary_version\t%s\n' "$v"; else printf 'binary_broken\t1\n'; fi + if h=$("$b" backup run --help 2>/dev/null); then case "$h" in *--all*) printf 'fleet\tyes\n';; *) printf 'fleet\tno\n';; esac; fi +else + printf 'binary_missing\t1\n' +fi +if command -v aws >/dev/null 2>&1; then + if v=$(aws --version 2>&1); then printf 'aws\t%s\n' "$v"; else printf 'aws_broken\t%s\n' "$v"; fi +else + printf 'aws_missing\t1\n' +fi +if command -v age >/dev/null 2>&1 && command -v age-keygen >/dev/null 2>&1; then + if v=$(age --version 2>&1); then printf 'age\t%s\n' "$v"; else printf 'age_broken\t%s\n' "$v"; fi +else + printf 'age_missing\t1\n' +fi +if command -v python3 >/dev/null 2>&1; then + if v=$(python3 --version 2>&1); then printf 'python3\t%s\n' "$v"; else printf 'python3_broken\t%s\n' "$v"; fi +else + printf 'python3_missing\t1\n' +fi +if v=$(df -Pk -- ` + q(tmpDir) + ` 2>/dev/null); then printf 'disk\t%s\n' "$(printf '%s\n' "$v" | awk 'NR==2{print $4}')"; fi +printf 'end\t1\n'` +} + +// collectBackupPreflight only reads. Each part fails to its own unknown. +func collectBackupPreflight(ctx context.Context, exec ssh.Executor, bin, cliVersion string, now time.Time) backupPreflightDTO { + dto := unknownPreflight(exec.Host(), bin, cliVersion, now) + + policy, policyErr := backup.ReadFleetPolicy(ctx, exec) + if policy.TmpDir != "" { + dto.Disk.Path = policy.TmpDir + } + res := ssh.RunDetailed(ctx, exec, "sh -c "+ssh.ShellQuote(preflightProbeScript(bin, dto.Disk.Path))) + if res.Err != nil { + dto.Error = oneLine(res.Err.Error()) + return dto + } + dto.Reachable = true + facts := map[string]string{} + for _, line := range strings.Split(string(res.Stdout), "\n") { + if k, v, ok := strings.Cut(line, "\t"); ok { + facts[k] = v + } + } + if _, done := facts["end"]; !done || res.ExitCode != 0 { + dto.Error = "the probe did not finish, so what it did not reach is unknown" + } + dto.Binary = readBinaryFacts(facts, bin, cliVersion) + dto.Tools = preflightTools{AWS: readAWSFacts(facts), Age: readToolFacts(facts, "age"), Python3: readToolFacts(facts, "python3")} + dto.Disk = readDiskFacts(facts, dto.Disk.Path) + dto.Target = readPreflightTarget(ctx, exec) + dto.Policy = readPreflightPolicy(ctx, exec, policy, policyErr) + return dto +} + +func readBinaryFacts(f map[string]string, bin, cliVersion string) preflightBinary { + b := preflightBinary{Path: bin, State: PreflightUnknown, CLIVersion: cliVersion} + if _, missing := f["binary_missing"]; missing { + no := false + b.State, b.FleetEngine = PreflightMissing, &no + return b + } + if _, broken := f["binary_broken"]; broken { + b.Detail = "the binary is there and did not run" + } + if v, ok := f["binary_version"]; ok { + b.Version = strings.TrimSpace(strings.TrimPrefix(strings.TrimSpace(v), "teploy")) + same := normalizeVersion(b.Version) == normalizeVersion(cliVersion) + b.SameAsCLI = &same + } + if v, ok := f["fleet"]; ok { + yes := v == "yes" + b.FleetEngine = &yes + } + switch { + case b.Version == "" || b.FleetEngine == nil: + if b.Detail == "" && b.Version == "" { + b.Detail = "could not read its version" + } + case *b.FleetEngine: + b.State = PreflightOK + default: + b.State = PreflightNoFleet + } + return b +} + +func readAWSFacts(f map[string]string) preflightTool { + t := readToolFacts(f, "aws") + if t.State != PreflightOK { + return t + } + version := backup.AWSCLIVersion(t.Version) + if version == "" { + return preflightTool{State: PreflightUnknown, Detail: "could not read its version: " + oneLine(t.Version)} + } + t.Version = version + if problem := backup.AWSCLIVersionProblem("aws-cli/" + version); problem != "" { + t.State, t.Detail = PreflightOld, problem + } + return t +} + +func readToolFacts(f map[string]string, name string) preflightTool { + if _, missing := f[name+"_missing"]; missing { + return preflightTool{State: PreflightMissing} + } + if v, broken := f[name+"_broken"]; broken { + return preflightTool{State: PreflightUnknown, Detail: "it is there and did not run: " + oneLine(v)} + } + if v, ok := f[name]; ok { + return preflightTool{State: PreflightOK, Version: strings.TrimSpace(v)} + } + return preflightTool{State: PreflightUnknown} +} + +func readDiskFacts(f map[string]string, path string) preflightDisk { + d := preflightDisk{Path: path} + v, ok := f["disk"] + if !ok { + d.Detail = "could not read the free space there" + return d + } + kb, err := strconv.ParseInt(strings.TrimSpace(v), 10, 64) + if err != nil { + d.Detail = "could not read the free space there" + return d + } + free := kb * 1024 + d.FreeBytes = &free + return d +} + +func readPreflightTarget(ctx context.Context, exec ssh.Executor) preflightTarget { + t, err := backup.ReadTarget(ctx, exec) + if err != nil { + return preflightTarget{State: PreflightUnknown, Detail: oneLine(err.Error())} + } + if !TargetConfigured(t) { + return preflightTarget{State: PreflightAbsent} + } + return preflightTarget{State: PreflightConfigured, Bucket: t.Bucket, Endpoint: t.Endpoint, Prefix: t.Prefix, HasCredentials: t.HasCredentials} +} + +func readPreflightPolicy(ctx context.Context, exec ssh.Executor, p backup.FleetPolicy, err error) preflightPolicy { + if err != nil { + return preflightPolicy{State: PreflightUnknown, ManagedElsewhere: PreflightUnknown, Detail: oneLine(err.Error())} + } + if !p.Configured() { + return preflightPolicy{State: PreflightAbsent, ManagedElsewhere: ManagedNo} + } + out := preflightPolicy{State: PreflightConfigured, Path: p.Path, Interval: p.Interval, Encrypted: p.Recipient != "", ManagedElsewhere: ManagedNo} + for _, h := range backup.MaskedHeartbeats(p.HeartbeatURLs) { + out.HeartbeatHosts = append(out.HeartbeatHosts, strings.TrimSuffix(h, "/...")) + } + if lines, cerr := backup.ReadCrontab(ctx, exec); cerr == nil { + found := false + for _, line := range lines { + if m := backup.FindCronMarker(line); m != nil && m.Kind == backup.KindFleet { + found = true + } + } + out.CronInstalled = &found + if !found && p.Interval != "" { + out.ManagedReasons = append(out.ManagedReasons, "the policy has an interval and this user's crontab has no line for it, so something else may run it") + } + } + reasons, known := policyFileManagedReasons(ctx, exec, p.Path) + out.ManagedReasons = append(out.ManagedReasons, reasons...) + switch { + case len(out.ManagedReasons) > 0: + out.ManagedElsewhere = ManagedYes + case !known || out.CronInstalled == nil: + out.ManagedElsewhere = PreflightUnknown + } + return out +} + +// policyFileManagedReasons looks at the policy file itself. teploy writes it as a +// regular file, owned by root or the connecting user, that this user can replace. +// Anything else got there another way, such as configuration management. +func policyFileManagedReasons(ctx context.Context, exec ssh.Executor, path string) ([]string, bool) { + script := preflightManagedMarker + ` +p=` + ssh.ShellQuote(path) + ` +S= +if [ "$(id -u)" != 0 ] && sudo -n true 2>/dev/null; then S="sudo -n"; fi +if $S test -L "$p"; then printf 'symlink\t1\n'; fi +if o=$($S stat -c %U "$p" 2>/dev/null); then printf 'owner\t%s\n' "$o"; fi +if [ -n "$S" ] || [ -w "$p" ]; then printf 'writable\t1\n'; fi +printf 'me\t%s\n' "$(id -un)" +printf 'end\t1\n'` + res := ssh.RunDetailed(ctx, exec, "sh -c "+ssh.ShellQuote(script)) + if res.Failed() { + return nil, false + } + f := map[string]string{} + for _, line := range strings.Split(string(res.Stdout), "\n") { + if k, v, ok := strings.Cut(line, "\t"); ok { + f[k] = strings.TrimSpace(v) + } + } + _, known := f["end"] + if !known { + return nil, false + } + var reasons []string + if _, ok := f["symlink"]; ok { + reasons = append(reasons, path+" is a symlink") + } + owner, haveOwner := f["owner"] + if haveOwner && owner != "root" && owner != f["me"] { + reasons = append(reasons, path+" is owned by "+owner) + } + if _, ok := f["writable"]; !ok { + reasons = append(reasons, "this user cannot replace "+path) + } + return reasons, haveOwner +} + +// installBackupTools installs aws, age and python3 where the report says they are not +// fine, through what `teploy setup` uses and nothing else of it. python3 first: AWS's +// bundle is unpacked with it. One failing does not stop the next. Recorded only when +// something was tried. +func installBackupTools(ctx context.Context, exec ssh.Executor, tools preflightTools, bin, cliVersion string) ([]preflightToolAction, error) { + sudo := sudoPrefixFor(ctx, exec) + const noninteractive = "DEBIAN_FRONTEND=noninteractive " + steps := []struct { + name string + state string + run func() error + }{ + {"python3", tools.Python3.State, func() error { + out, err := exec.Run(ctx, sudo+"sh -c "+ssh.ShellQuote(noninteractive+"apt-get update -qq && "+noninteractive+"apt-get install -y -qq python3")) + return commandError(out, err) + }}, + {"age", tools.Age.State, func() error { return secret.NewManager(exec).EnsureAge(ctx) }}, + {"aws", tools.AWS.State, func() error { + // The package index may be stale on a server `teploy setup` never saw. + _, _ = exec.Run(ctx, sudo+"sh -c "+ssh.ShellQuote(noninteractive+"apt-get update -qq")) + out, err := exec.Run(ctx, sudo+"sh -c "+ssh.ShellQuote(backup.AWSCLIInstallScript)) + return commandError(out, err) + }}, + } + var actions []preflightToolAction + var errs []error + tried := false + for _, s := range steps { + if s.state == PreflightOK { + actions = append(actions, preflightToolAction{Tool: s.name, Action: ActionAlreadyDone}) + continue + } + tried = true + if err := s.run(); err != nil { + actions = append(actions, preflightToolAction{Tool: s.name, Action: ActionFailed, Detail: oneLine(err.Error())}) + errs = append(errs, fmt.Errorf("installing %s: %w", s.name, err)) + continue + } + actions = append(actions, preflightToolAction{Tool: s.name, Action: ActionDone}) + } + // Exit status alone is not success: an install that ran and left a tool that does + // not run, or an aws still below its floor, did not work. + after := collectBackupToolStates(ctx, exec, bin, cliVersion) + for i, a := range actions { + if a.Action != ActionDone { + continue + } + if st := after[a.Tool]; st != PreflightOK { + actions[i].Action = ActionFailed + actions[i].Detail = "the install finished and the tool still reads as " + st + errs = append(errs, fmt.Errorf("installing %s: it still reads as %s", a.Tool, st)) + } + } + err := errors.Join(errs...) + if tried { + var done, failed []string + for _, a := range actions { + switch a.Action { + case ActionDone: + done = append(done, a.Tool) + case ActionFailed: + failed = append(failed, a.Tool) + } + } + events.Record(ctx, exec, events.Event{App: backup.FleetApp, Kind: events.KindServerTools, Outcome: events.Outcome(err), Detail: events.ErrDetail(map[string]any{"installed": done, "failed": failed}, err)}) + } + return actions, err +} + +func collectBackupToolStates(ctx context.Context, exec ssh.Executor, bin, cliVersion string) map[string]string { + res := ssh.RunDetailed(ctx, exec, "sh -c "+ssh.ShellQuote(preflightProbeScript(bin, backup.DefaultTmpDir))) + f := map[string]string{} + if !res.Failed() { + for _, line := range strings.Split(string(res.Stdout), "\n") { + if k, v, ok := strings.Cut(line, "\t"); ok { + f[k] = v + } + } + } + return map[string]string{"aws": readAWSFacts(f).State, "age": readToolFacts(f, "age").State, "python3": readToolFacts(f, "python3").State} +} + +func commandError(out string, err error) error { + if err == nil { + return nil + } + if s := strings.TrimSpace(out); s != "" { + return fmt.Errorf("%w: %s", err, s) + } + return err +} + +// upgradeServerBinary installs the release matching the running CLI, through the +// verified path every other server install uses. It refuses what cannot be that: a +// development CLI, a binary path that is not the managed one, a downgrade. +func upgradeServerBinary(ctx context.Context, exec ssh.Executor, b preflightBinary, o preflightOptions) (preflightUpgrade, error) { + want := normalizeVersion(o.CLIVersion) + u := preflightUpgrade{From: b.Version, To: want} + refuse := func(format string, args ...any) (preflightUpgrade, error) { + u.Action, u.Detail = ActionRefused, fmt.Sprintf(format, args...) + err := errors.New("refusing to upgrade the server's teploy: " + u.Detail) + events.Record(ctx, exec, events.Event{App: backup.FleetApp, Kind: events.KindServerBinary, Outcome: events.Refused, Detail: events.ErrDetail(map[string]any{"from": u.From, "to": u.To}, err)}) + return u, err + } + wantParts, ok := parseVersion(want) + if !ok { + return refuse("this CLI is version %q, which is not a release", o.CLIVersion) + } + if o.Binary != serverTeployPath { + return refuse("TEPLOY_SERVER_BINARY names %s, which this path does not manage", o.Binary) + } + if b.Version != "" { + if have, ok := parseVersion(normalizeVersion(b.Version)); ok { + switch compareVersions(have, wantParts) { + case 0: + if b.State == PreflightOK { + u.Action = ActionAlreadyDone + return u, nil + } + case 1: + return refuse("the server has %s, newer than this CLI's %s; upgrade this CLI instead", b.Version, want) + } + } + } + record := func(err error) { + events.Record(ctx, exec, events.Event{App: backup.FleetApp, Kind: events.KindServerBinary, Outcome: events.Outcome(err), Detail: events.ErrDetail(map[string]any{"from": u.From, "to": u.To}, err)}) + } + fail := func(err error) (preflightUpgrade, error) { + u.Action, u.Detail = ActionFailed, oneLine(err.Error()) + record(err) + return u, err + } + if _, err := o.InstallRelease(ctx, exec, serverTeployPath, "v"+want, "backup run"); err != nil { + return fail(fmt.Errorf("installing teploy %s: %w", want, err)) + } + after := collectBackupPreflight(ctx, exec, o.Binary, o.CLIVersion, o.Now).Binary + if normalizeVersion(after.Version) != want { + return fail(fmt.Errorf("after installing, the server's teploy reads as %q, not %s", after.Version, want)) + } + if after.State != PreflightOK { + return fail(fmt.Errorf("after installing %s, the server's teploy reads as %s", want, after.State)) + } + u.Action = ActionDone + record(nil) + return u, nil +} + +func normalizeVersion(v string) string { + return strings.TrimPrefix(strings.TrimSpace(v), "v") +} + +// parseVersion reads a plain release version: 0.1.54. +func parseVersion(v string) ([3]int, bool) { + var out [3]int + parts := strings.Split(v, ".") + if len(parts) != 3 { + return out, false + } + for i, p := range parts { + n, err := strconv.Atoi(p) + if err != nil || n < 0 { + return out, false + } + out[i] = n + } + return out, true +} + +func compareVersions(a, b [3]int) int { + for i := range a { + switch { + case a[i] < b[i]: + return -1 + case a[i] > b[i]: + return 1 + } + } + return 0 +} + +func printBackupPreflight(d backupPreflightDTO) { + if !d.Reachable { + fmt.Printf("%s did not answer: %s\nEverything below it is unknown.\n", d.Host, d.Error) + return + } + if d.Error != "" { + fmt.Printf("Warning: %s\n", d.Error) + } + version := d.Binary.Version + if version == "" { + version = "version unknown" + } + fmt.Printf("teploy on server: %s (%s, %s; this CLI is %s)\n", d.Binary.State, version, d.Binary.Path, d.Binary.CLIVersion) + tool := func(name string, t preflightTool) { + fmt.Printf("%-16s %s", name+":", t.State) + if t.Version != "" { + fmt.Printf(" (%s)", t.Version) + } + if t.Detail != "" { + fmt.Printf(": %s", t.Detail) + } + fmt.Println() + } + tool("aws", d.Tools.AWS) + tool("age", d.Tools.Age) + tool("python3", d.Tools.Python3) + if d.Disk.FreeBytes != nil { + fmt.Printf("%-16s %.1f GB free at %s\n", "disk:", float64(*d.Disk.FreeBytes)/(1<<30), d.Disk.Path) + } else { + fmt.Printf("%-16s unknown at %s\n", "disk:", d.Disk.Path) + } + fmt.Printf("%-16s %s\n", "target:", d.Target.State) + fmt.Printf("%-16s %s (managed elsewhere: %s)\n", "policy:", d.Policy.State, d.Policy.ManagedElsewhere) + for _, r := range d.Policy.ManagedReasons { + fmt.Printf(" %s\n", r) + } + for _, a := range d.ToolsInstall { + fmt.Printf("install %s: %s", a.Tool, a.Action) + if a.Detail != "" { + fmt.Printf(": %s", a.Detail) + } + fmt.Println() + } + if u := d.BinaryUpgrade; u != nil { + fmt.Printf("upgrade teploy to %s: %s", u.To, u.Action) + if u.Detail != "" { + fmt.Printf(": %s", u.Detail) + } + fmt.Println() + } +} diff --git a/internal/cli/backup_preflight_test.go b/internal/cli/backup_preflight_test.go new file mode 100644 index 0000000..d18457d --- /dev/null +++ b/internal/cli/backup_preflight_test.go @@ -0,0 +1,415 @@ +package cli + +import ( + "context" + "encoding/json" + "errors" + "github.com/useteploy/teploy/internal/ssh" + "strings" + "testing" + "time" +) + +var setupNow = time.Date(2026, 11, 6, 12, 0, 0, 0, time.UTC) + +const ( + probeReady = "binary_version\tteploy 0.1.55\nfleet\tyes\naws\taws-cli/2.30.1 Python/3.13 Linux\nage\tv1.2.1\npython3\tPython 3.12.3\ndisk\t104857600\nend\t1\n" + probeBare = "binary_version\tteploy 0.1.20\nfleet\tno\naws\taws-cli/2.15.0 Python/3.11 Linux\nage_missing\t1\npython3_missing\t1\ndisk\t2097152\nend\t1\n" + targetStored = "path /etc/teploy/backup-target.json\n" + `{"v":"1","bucket":"b","region":"r","prefix":"teploy/srv1","credentials":"x"}` + policyStored = "path /etc/teploy/backup-policy.json\n" + `{"v":"1","interval":"1h","recipient":"` + testRecipient + `","heartbeat_urls":["https://dash.example.internal/api/push/secret-token"]}` + cronFleet = "5 * * * * /usr/local/bin/teploy backup run --all # [teploy-backup-all]\n" + managedPlain = "owner\troot\nwritable\t1\nme\troot\nend\t1\n" +) + +func preflightCommands(probe, target, policy, cron, managed string) []ssh.MockCommand { + return []ssh.MockCommand{ + {Match: "sh -c '" + preflightProbeMarker, Output: probe}, + {Match: "sh -c '" + preflightManagedMarker, Output: managed}, + {Match: "for p in /etc/teploy/backup-target.json", Output: target}, + {Match: "if [ -d /etc/teploy ]", Output: policy}, + {Match: "crontab -l", Output: cron}, + } +} + +func collectFrom(mock *ssh.MockExecutor) backupPreflightDTO { + return collectBackupPreflight(context.Background(), mock, serverTeployPath, "0.1.55", setupNow) +} + +func TestPreflightReportsAReadyServer(t *testing.T) { + mock := ssh.NewMockExecutor("192.0.2.10", preflightCommands(probeReady, targetStored, policyStored, cronFleet, managedPlain)...) + d := collectFrom(mock) + if !d.Reachable || d.Error != "" { + t.Fatalf("%+v", d) + } + if d.Binary.State != PreflightOK || d.Binary.Version != "0.1.55" || d.Binary.FleetEngine == nil || !*d.Binary.FleetEngine || d.Binary.SameAsCLI == nil || !*d.Binary.SameAsCLI { + t.Fatalf("%+v", d.Binary) + } + if d.Tools.AWS.State != PreflightOK || d.Tools.AWS.Version != "2.30.1" || d.Tools.Age.State != PreflightOK || d.Tools.Python3.State != PreflightOK { + t.Fatalf("%+v", d.Tools) + } + if d.Disk.FreeBytes == nil || *d.Disk.FreeBytes != 104857600*1024 { + t.Fatalf("%+v", d.Disk) + } + if d.Target.State != PreflightConfigured || d.Target.Bucket != "b" || !d.Target.HasCredentials { + t.Fatalf("%+v", d.Target) + } + if d.Policy.State != PreflightConfigured || d.Policy.ManagedElsewhere != ManagedNo || !d.Policy.Encrypted || d.Policy.CronInstalled == nil || !*d.Policy.CronInstalled { + t.Fatalf("%+v", d.Policy) + } + // Tokens stay out: the host alone. + body, _ := json.Marshal(d) + if strings.Contains(string(body), "secret-token") { + t.Fatalf("heartbeat token in the report: %s", body) + } +} + +// Read-only: the plain report runs nothing that changes the server. +func TestPreflightIsReadOnly(t *testing.T) { + mock := ssh.NewMockExecutor("h", preflightCommands(probeBare, targetStored, policyStored, cronFleet, managedPlain)...) + d, err := backupPreflightOn(context.Background(), mock, preflightOptions{Binary: serverTeployPath, CLIVersion: "0.1.55", Now: setupNow, InstallRelease: func(context.Context, ssh.Executor, string, string, ...string) (string, error) { + t.Fatal("installed without being asked") + return "", nil + }}) + if err != nil { + t.Fatal(err) + } + if d.ToolsInstall != nil || d.BinaryUpgrade != nil { + t.Fatalf("%+v", d) + } + for _, c := range mock.Calls { + for _, verb := range []string{"apt-get", "yum ", "crontab -\n", "mv ", "python3 -c", "aws s3", "ufw", "systemctl"} { + if strings.Contains(c, verb) && !strings.Contains(c, preflightProbeMarker) && !strings.Contains(c, "python3 -c 'import") { + t.Fatalf("%q ran: %s", verb, c) + } + } + } +} + +// A probe that fails reads as unknown, never as fine. +func TestPreflightFailedProbesAreUnknown(t *testing.T) { + cases := []struct { + name string + probe string + check func(backupPreflightDTO) bool + }{ + {"cut short", "binary_version\tteploy 0.1.55\n", func(d backupPreflightDTO) bool { + return d.Error != "" && d.Binary.State == PreflightUnknown && d.Binary.FleetEngine == nil && d.Tools.AWS.State == PreflightUnknown && d.Tools.Age.State == PreflightUnknown && d.Tools.Python3.State == PreflightUnknown && d.Disk.FreeBytes == nil + }}, + {"binary does not run", "binary_broken\t1\naws\taws-cli/2.30.0\nend\t1\n", func(d backupPreflightDTO) bool { + return d.Binary.State == PreflightUnknown && d.Binary.FleetEngine == nil && d.Binary.SameAsCLI == nil && d.Binary.Detail != "" + }}, + {"help does not run", "binary_version\tteploy 0.1.55\nend\t1\n", func(d backupPreflightDTO) bool { + return d.Binary.State == PreflightUnknown && d.Binary.FleetEngine == nil && d.Binary.Version == "0.1.55" + }}, + {"aws wrapper with no version", "aws\tsome wrapper\nend\t1\n", func(d backupPreflightDTO) bool { + return d.Tools.AWS.State == PreflightUnknown && d.Tools.AWS.Detail != "" + }}, + {"aws does not run", "aws_broken\tcannot execute\nend\t1\n", func(d backupPreflightDTO) bool { + return d.Tools.AWS.State == PreflightUnknown && strings.Contains(d.Tools.AWS.Detail, "cannot execute") + }}, + {"disk not a number", "disk\t\nend\t1\n", func(d backupPreflightDTO) bool { + return d.Disk.FreeBytes == nil && d.Disk.Detail != "" + }}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + d := collectFrom(ssh.NewMockExecutor("h", preflightCommands(tc.probe, "absent\n", "absent\n", "", "")...)) + if !d.Reachable || !tc.check(d) { + t.Fatalf("%+v", d) + } + }) + } + t.Run("old and missing are facts, not unknowns", func(t *testing.T) { + d := collectFrom(ssh.NewMockExecutor("h", preflightCommands(probeBare, "absent\n", "absent\n", "", "")...)) + if d.Binary.State != PreflightNoFleet || *d.Binary.FleetEngine || d.Tools.AWS.State != PreflightOld || d.Tools.AWS.Detail == "" || d.Tools.Age.State != PreflightMissing || d.Tools.Python3.State != PreflightMissing { + t.Fatalf("%+v", d) + } + if d.Target.State != PreflightAbsent || d.Policy.State != PreflightAbsent { + t.Fatalf("%+v", d) + } + }) + t.Run("no binary", func(t *testing.T) { + d := collectFrom(ssh.NewMockExecutor("h", preflightCommands("binary_missing\t1\nend\t1\n", "absent\n", "absent\n", "", "")...)) + if d.Binary.State != PreflightMissing || d.Binary.FleetEngine == nil || *d.Binary.FleetEngine { + t.Fatalf("%+v", d.Binary) + } + }) + t.Run("connection lost", func(t *testing.T) { + d := collectFrom(ssh.NewMockExecutor("h", ssh.MockCommand{Match: "if [ -d /etc/teploy ]", Err: errors.New("ssh: connection reset")}, ssh.MockCommand{Match: "sh -c", Err: errors.New("ssh: connection reset")})) + if d.Reachable || d.Error == "" || d.Binary.State != PreflightUnknown || d.Target.State != PreflightUnknown || d.Policy.State != PreflightUnknown || d.Policy.ManagedElsewhere != PreflightUnknown { + t.Fatalf("%+v", d) + } + }) + t.Run("target and policy unreadable", func(t *testing.T) { + cmds := []ssh.MockCommand{ + {Match: "sh -c '" + preflightProbeMarker, Output: probeReady}, + {Match: "for p in /etc/teploy/backup-target.json", Output: "garbage"}, + {Match: "if [ -d /etc/teploy ]", Err: errors.New("exit status 3: cannot tell")}, + } + d := collectFrom(ssh.NewMockExecutor("h", cmds...)) + if d.Target.State != PreflightUnknown || d.Policy.State != PreflightUnknown || d.Policy.ManagedElsewhere != PreflightUnknown || d.Policy.Detail == "" { + t.Fatalf("%+v %+v", d.Target, d.Policy) + } + }) + t.Run("unreachable", func(t *testing.T) { + d := unreachablePreflight("h", serverTeployPath, "0.1.55", errors.New("dial tcp: i/o timeout"), setupNow) + if d.Reachable || !strings.Contains(d.Error, "timeout") || d.Tools.AWS.State != PreflightUnknown || d.Disk.FreeBytes != nil || d.Target.State != PreflightUnknown { + t.Fatalf("%+v", d) + } + }) +} + +func TestPreflightPolicyManagedElsewhere(t *testing.T) { + for _, tc := range []struct { + name, cron, managed, want, reason string + }{ + {"teploy's own", cronFleet, managedPlain, ManagedNo, ""}, + {"symlink", cronFleet, "symlink\t1\nowner\troot\nwritable\t1\nme\troot\nend\t1\n", ManagedYes, "symlink"}, + {"another owner", cronFleet, "owner\tansible\nwritable\t1\nme\tdeploy\nend\t1\n", ManagedYes, "owned by ansible"}, + {"cannot replace", cronFleet, "owner\troot\nme\tdeploy\nend\t1\n", ManagedYes, "cannot replace"}, + {"no cron line for its interval", "", managedPlain, ManagedYes, "crontab"}, + {"probe cut short", cronFleet, "owner\troot\n", PreflightUnknown, ""}, + {"owner unreadable", cronFleet, "writable\t1\nme\tdeploy\nend\t1\n", PreflightUnknown, ""}, + } { + t.Run(tc.name, func(t *testing.T) { + d := collectFrom(ssh.NewMockExecutor("h", preflightCommands(probeReady, targetStored, policyStored, tc.cron, tc.managed)...)) + if d.Policy.ManagedElsewhere != tc.want { + t.Fatalf("%+v", d.Policy) + } + if tc.reason != "" && !strings.Contains(strings.Join(d.Policy.ManagedReasons, ";"), tc.reason) { + t.Fatalf("%+v", d.Policy) + } + }) + } + // Hidden from this user: unknown, not "no". + d := collectFrom(ssh.NewMockExecutor("h", + ssh.MockCommand{Match: "sh -c '" + preflightProbeMarker, Output: probeReady}, + ssh.MockCommand{Match: "for p in /etc/teploy/backup-target.json", Output: "absent\n"}, + ssh.MockCommand{Match: "if [ -d /etc/teploy ]", Err: errors.New("exit status 3: cannot tell")}, + )) + if d.Policy.ManagedElsewhere != PreflightUnknown { + t.Fatalf("%+v", d.Policy) + } + // Probes that could not run mean unknown too. + d = collectFrom(ssh.NewMockExecutor("h", + ssh.MockCommand{Match: "sh -c '" + preflightProbeMarker, Output: probeReady}, + ssh.MockCommand{Match: "for p in /etc/teploy/backup-target.json", Output: "absent\n"}, + ssh.MockCommand{Match: "if [ -d /etc/teploy ]", Output: policyStored}, + ssh.MockCommand{Match: "crontab -l", Err: errors.New("ssh: connection reset")}, + ssh.MockCommand{Match: "sh -c '" + preflightManagedMarker, Err: errors.New("ssh: connection reset")}, + )) + if d.Policy.ManagedElsewhere != PreflightUnknown || d.Policy.CronInstalled != nil { + t.Fatalf("%+v", d.Policy) + } +} + +func installMock(probeBefore, probeAfter string, extra ...ssh.MockCommand) *ssh.MockExecutor { + cmds := []ssh.MockCommand{{Match: "sh -c '" + preflightProbeMarker, Output: probeBefore, Once: true}, {Match: "sh -c '" + preflightProbeMarker, Output: probeAfter}} + cmds = append(cmds, extra...) + cmds = append(cmds, + ssh.MockCommand{Match: "id -u", Output: "0\n"}, + ssh.MockCommand{Match: "which age", Output: "/usr/bin/age\n"}, + ssh.MockCommand{Match: "sh -c '" + "DEBIAN_FRONTEND", Output: ""}, + ssh.MockCommand{Match: "sh -c 'set -e", Output: ""}, + ssh.MockCommand{Match: "for p in /etc/teploy/backup-target.json", Output: "absent\n"}, + ssh.MockCommand{Match: "if [ -d /etc/teploy ]", Output: "absent\n"}, + ssh.MockCommand{Match: "python3 -c", Output: `{"seq":1,"ts":"2026-01-01T00:00:00Z"}`}, + ) + return ssh.NewMockExecutor("h", cmds...) +} + +func toolsOpts() preflightOptions { + return preflightOptions{InstallTools: true, Binary: serverTeployPath, CLIVersion: "0.1.55", Now: setupNow} +} + +// Tools and nothing else: the aws step is setup's script with its signature check, and +// no hardening or other setup step runs. +func TestInstallToolsInstallsOnlyTheTools(t *testing.T) { + mock := installMock(probeBare, probeReady) + d, err := backupPreflightOn(context.Background(), mock, toolsOpts()) + if err != nil { + t.Fatal(err) + } + if len(d.ToolsInstall) != 3 { + t.Fatalf("%+v", d.ToolsInstall) + } + for _, a := range d.ToolsInstall { + if a.Action != ActionDone { + t.Fatalf("%+v", d.ToolsInstall) + } + } + if d.ToolsInstall[0].Tool != "python3" { + t.Fatalf("python3 goes first, AWS's bundle is unpacked with it: %+v", d.ToolsInstall) + } + all := strings.Join(mock.Calls, "\n") + for _, must := range []string{"VALIDSIG", "apt-get install -y -qq python3"} { + if !strings.Contains(all, must) { + t.Fatalf("no %q in %s", must, all) + } + } + for _, never := range []string{"ufw", "fail2ban", "sshd", "get.docker", "tailscale", "docker network", "PasswordAuthentication"} { + if strings.Contains(all, never) { + t.Fatalf("%q ran", never) + } + } + if !strings.Contains(all, "append '_fleet'") { + t.Fatal("no event") + } + if d.Tools.AWS.State != PreflightOK { + t.Fatalf("the report must describe the server after the install: %+v", d.Tools) + } +} + +func TestInstallToolsIsIdempotent(t *testing.T) { + mock := installMock(probeReady, probeReady) + d, err := backupPreflightOn(context.Background(), mock, toolsOpts()) + if err != nil { + t.Fatal(err) + } + for _, a := range d.ToolsInstall { + if a.Action != ActionAlreadyDone { + t.Fatalf("%+v", d.ToolsInstall) + } + } + all := strings.Join(mock.Calls, "\n") + if strings.Contains(all, "apt-get") || strings.Contains(all, "append '_fleet'") { + t.Fatalf("nothing to do, and it did: %s", all) + } +} + +// One failing does not stop the next, and the report says which. +func TestInstallToolsReportsEachResult(t *testing.T) { + failing := ssh.MockCommand{Match: "sh -c 'set -e", Output: "boom", Err: errors.New("exit status 1")} + after := strings.Replace(probeReady, "aws\taws-cli/2.30.1 Python/3.13 Linux", "aws_missing\t1", 1) + mock := installMock(probeBare, after, failing) + d, err := backupPreflightOn(context.Background(), mock, toolsOpts()) + if err == nil || !strings.Contains(err.Error(), "installing aws") { + t.Fatalf("%v", err) + } + got := map[string]preflightToolAction{} + for _, a := range d.ToolsInstall { + got[a.Tool] = a + } + if got["aws"].Action != ActionFailed || !strings.Contains(got["aws"].Detail, "boom") || got["python3"].Action != ActionDone || got["age"].Action != ActionDone { + t.Fatalf("%+v", d.ToolsInstall) + } + if !strings.Contains(strings.Join(mock.Calls, "\n"), "append '_fleet'") { + t.Fatal("no event") + } +} + +// An exit status of 0 is not success: the tool has to read as fine afterwards. +func TestInstallToolsChecksTheResult(t *testing.T) { + mock := installMock(probeBare, strings.Replace(probeReady, "aws-cli/2.30.1", "aws-cli/2.15.0", 1)) + d, err := backupPreflightOn(context.Background(), mock, toolsOpts()) + if err == nil { + t.Fatal("no error") + } + for _, a := range d.ToolsInstall { + if a.Tool == "aws" && (a.Action != ActionFailed || !strings.Contains(a.Detail, PreflightOld)) { + t.Fatalf("%+v", a) + } + } +} + +func upgradeOpts(cli string, install func(context.Context, ssh.Executor, string, string, ...string) (string, error)) preflightOptions { + return preflightOptions{UpgradeBinary: true, Binary: serverTeployPath, CLIVersion: cli, Now: setupNow, InstallRelease: install} +} + +func TestUpgradeBinary(t *testing.T) { + noInstall := func(t *testing.T) func(context.Context, ssh.Executor, string, string, ...string) (string, error) { + return func(context.Context, ssh.Executor, string, string, ...string) (string, error) { + t.Helper() + t.Fatal("must not install") + return "", nil + } + } + t.Run("installs the CLI's own version and verifies it", func(t *testing.T) { + mock := installMock(probeBare, probeReady) + var tag string + var caps []string + d, err := backupPreflightOn(context.Background(), mock, upgradeOpts("v0.1.55", func(_ context.Context, _ ssh.Executor, path, tg string, c ...string) (string, error) { + if path != serverTeployPath { + t.Fatal(path) + } + tag, caps = tg, c + return "0.1.55", nil + })) + if err != nil { + t.Fatal(err) + } + if tag != "v0.1.55" || len(caps) != 1 || caps[0] != "backup run" { + t.Fatalf("%q %v", tag, caps) + } + if u := d.BinaryUpgrade; u == nil || u.Action != ActionDone || u.From != "0.1.20" || u.To != "0.1.55" { + t.Fatalf("%+v", d.BinaryUpgrade) + } + if !strings.Contains(strings.Join(mock.Calls, "\n"), "append '_fleet'") { + t.Fatal("no event") + } + }) + t.Run("already there", func(t *testing.T) { + mock := installMock(probeReady, probeReady) + d, err := backupPreflightOn(context.Background(), mock, upgradeOpts("0.1.55", noInstall(t))) + if err != nil || d.BinaryUpgrade.Action != ActionAlreadyDone { + t.Fatalf("%+v %v", d.BinaryUpgrade, err) + } + if strings.Contains(strings.Join(mock.Calls, "\n"), "append '_fleet'") { + t.Fatal("nothing changed, and an event says something did") + } + }) + for _, tc := range []struct { + name, cli, probe, want string + }{ + {"a development build has no release", "dev", probeBare, "not a release"}, + {"never downgrades", "0.1.54", probeReady, "newer"}, + } { + t.Run(tc.name, func(t *testing.T) { + mock := installMock(tc.probe, tc.probe) + d, err := backupPreflightOn(context.Background(), mock, upgradeOpts(tc.cli, noInstall(t))) + if err == nil || d.BinaryUpgrade.Action != ActionRefused || !strings.Contains(d.BinaryUpgrade.Detail, tc.want) { + t.Fatalf("%+v %v", d.BinaryUpgrade, err) + } + if !strings.Contains(strings.Join(mock.Calls, "\n"), "append '_fleet'") { + t.Fatal("a refusal is on the record too") + } + }) + } + t.Run("a binary path it does not manage", func(t *testing.T) { + o := upgradeOpts("0.1.55", noInstall(t)) + o.Binary = "/opt/test/teploy" + mock := installMock(probeBare, probeBare) + d, err := backupPreflightOn(context.Background(), mock, o) + if err == nil || d.BinaryUpgrade.Action != ActionRefused { + t.Fatalf("%+v %v", d.BinaryUpgrade, err) + } + }) + t.Run("the install fails", func(t *testing.T) { + mock := installMock(probeBare, probeBare) + d, err := backupPreflightOn(context.Background(), mock, upgradeOpts("0.1.55", func(context.Context, ssh.Executor, string, string, ...string) (string, error) { + return "", errors.New("checksum mismatch") + })) + if err == nil || d.BinaryUpgrade.Action != ActionFailed || !strings.Contains(d.BinaryUpgrade.Detail, "checksum mismatch") { + t.Fatalf("%+v %v", d.BinaryUpgrade, err) + } + }) + t.Run("the server still reads old after the install", func(t *testing.T) { + mock := installMock(probeBare, probeBare) + d, err := backupPreflightOn(context.Background(), mock, upgradeOpts("0.1.55", func(context.Context, ssh.Executor, string, string, ...string) (string, error) { + return "0.1.55", nil + })) + if err == nil || d.BinaryUpgrade.Action != ActionFailed || !strings.Contains(d.BinaryUpgrade.Detail, "0.1.20") { + t.Fatalf("%+v %v", d.BinaryUpgrade, err) + } + }) + t.Run("same version without the engine is not done", func(t *testing.T) { + same := strings.Replace(probeReady, "fleet\tyes", "fleet\tno", 1) + mock := installMock(same, same) + d, err := backupPreflightOn(context.Background(), mock, upgradeOpts("0.1.55", func(context.Context, ssh.Executor, string, string, ...string) (string, error) { + return "0.1.55", nil + })) + if err == nil || d.BinaryUpgrade.Action != ActionFailed { + t.Fatalf("%+v %v", d.BinaryUpgrade, err) + } + }) +} diff --git a/internal/cli/backup_run.go b/internal/cli/backup_run.go index 6623aa2..19f671e 100644 --- a/internal/cli/backup_run.go +++ b/internal/cli/backup_run.go @@ -24,6 +24,7 @@ func newBackupRunCmd(flags *Flags) *cobra.Command { appName string accessory string all bool + detach bool ignoreSpace bool ) cmd := &cobra.Command{ @@ -35,9 +36,19 @@ func newBackupRunCmd(flags *Flags) *cobra.Command { "and each of its databases, app volumes when enabled, adopted containers), one\n" + "target at a time. One target failing does not stop the others; the exit status\n" + "is nonzero for anything but a clean run, and nothing of a failed target is\n" + - "pruned. A run that finds another one running stands down.", + "pruned. A run that finds another one running stands down.\n\n" + + "--all --detach --host S runs from anywhere: it starts that same run on the\n" + + "server so that it survives this connection, under the same lock, and returns\n" + + "within seconds. A second one reports already_running. Results reach status and\n" + + "events as a normal run's do.", Args: cobra.NoArgs, RunE: func(cmd *cobra.Command, args []string) error { + if detach { + if !all || appName != "" || accessory != "" { + return fmt.Errorf("--detach starts `backup run --all`: it needs --all, and takes no --app or --accessory") + } + return runBackupRunDetach(flags, ignoreSpace) + } if all { if appName != "" || accessory != "" { return fmt.Errorf("--all takes no --app or --accessory: it works out its own targets") @@ -50,6 +61,7 @@ func newBackupRunCmd(flags *Flags) *cobra.Command { cmd.Flags().StringVar(&appName, "app", "", "app name (required without --all)") cmd.Flags().StringVar(&accessory, "accessory", "", "back up this accessory instead of the app's volumes") cmd.Flags().BoolVar(&all, "all", false, "back up everything the server's backup policy covers") + cmd.Flags().BoolVar(&detach, "detach", false, "with --all and --host: start the run on the server and return; it survives this connection") cmd.Flags().BoolVar(&ignoreSpace, "ignore-space-check", false, "with --all: skip the free-space preflights") return cmd } diff --git a/internal/cli/backup_target.go b/internal/cli/backup_target.go index 5468f2b..76bae52 100644 --- a/internal/cli/backup_target.go +++ b/internal/cli/backup_target.go @@ -66,7 +66,11 @@ func newServerBackupTargetOwnersCmd(flags *Flags) *cobra.Command { "no longer exists, or this one before a rebuild. Backups are not touched. Do not\n" + "clear a server that still writes here: give it its own --prefix. A namespace\n" + "written to in the last week is refused without --force, and every clear is\n" + - "recorded in the server's event log.", + "recorded in the server's event log. A refusal anywhere removes nothing anywhere.\n\n" + + "A previous prefix that does not answer (a key limited to the current prefix\n" + + "cannot reach one) is skipped with a warning and reported; its markers stay. The\n" + + "current prefix not answering is an error, after every other location has been\n" + + "tried.", Args: cobra.NoArgs, RunE: func(cmd *cobra.Command, args []string) error { ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt) @@ -96,24 +100,25 @@ func newServerBackupTargetOwnersCmd(flags *Flags) *cobra.Command { locs = append(locs, l) } if clear != "" { - total, err := clearOwnerMarkers(ctx, executor, client, locs, clear, force, time.Now()) + total, skipped, err := clearOwnerMarkers(ctx, executor, client, locs, clear, force, time.Now()) + if flags.JSON { + return errors.Join(err, json.NewEncoder(os.Stdout).Encode(ownersClearDTO{MachineInterface: MachineInterface, Host: executor.Host(), Fingerprint: clear, Removed: total, SkippedLocations: skipped})) + } + warnSkippedLocations(skipped) if err != nil { return err } fmt.Printf("Removed %d marker(s) of %s\n", total, clear) return nil } - markers := []backup.NamespaceMarker{} - for _, l := range locs { - m, err := client.ListNamespaceMarkers(ctx, l) - if err != nil { - return err - } - markers = append(markers, m...) + markers, skipped, err := listOwnerMarkers(ctx, client, locs) + if err != nil { + return err } if flags.JSON { - return json.NewEncoder(os.Stdout).Encode(map[string]any{"machine_interface": MachineInterface, "host": executor.Host(), "markers": markers}) + return json.NewEncoder(os.Stdout).Encode(ownersListDTO{MachineInterface: MachineInterface, Host: executor.Host(), Markers: markers, SkippedLocations: skipped}) } + warnSkippedLocations(skipped) for _, m := range markers { who := "another server" if m.Self { @@ -129,26 +134,125 @@ func newServerBackupTargetOwnersCmd(flags *Flags) *cobra.Command { return cmd } +// skippedLocation is a previous prefix that could not be read or changed. +type skippedLocation struct { + Prefix string `json:"prefix"` + Reason string `json:"reason"` +} + +// ownersListDTO is `server backup-target owners --json`. +type ownersListDTO struct { + MachineInterface int `json:"machine_interface"` + Host string `json:"host"` + Markers []backup.NamespaceMarker `json:"markers"` + SkippedLocations []skippedLocation `json:"skipped_locations"` +} + +// ownersClearDTO is `server backup-target owners --clear --json`, printed whether or +// not the clear succeeded: part of it may have. +type ownersClearDTO struct { + MachineInterface int `json:"machine_interface"` + Host string `json:"host"` + Fingerprint string `json:"fingerprint"` + Removed int `json:"removed"` + SkippedLocations []skippedLocation `json:"skipped_locations"` +} + +func skipLocation(l backup.S3Config, err error) skippedLocation { + return skippedLocation{Prefix: l.Prefix, Reason: oneLine(err.Error())} +} + +func warnSkippedLocations(skipped []skippedLocation) { + for _, s := range skipped { + where := "the bucket root" + if s.Prefix != "" { + where = "prefix " + s.Prefix + } + fmt.Fprintf(os.Stderr, "Warning: previous location %s was skipped: %s\n", where, s.Reason) + } +} + +// listOwnerMarkers lists every location. locs[0] is the current prefix: it failing is +// an error, since an empty list would read as no owners. A previous one failing is a +// warning, because a key limited to the current prefix cannot reach it. +func listOwnerMarkers(ctx context.Context, client *backup.Client, locs []backup.S3Config) ([]backup.NamespaceMarker, []skippedLocation, error) { + markers := []backup.NamespaceMarker{} + skipped := []skippedLocation{} + for i, l := range locs { + m, err := client.ListNamespaceMarkers(ctx, l) + if err != nil { + if i == 0 { + return nil, nil, err + } + skipped = append(skipped, skipLocation(l, err)) + continue + } + markers = append(markers, m...) + } + return markers, skipped, nil +} + // clearOwnerMarkers removes a fingerprint's markers from every location, and records // it: the marker is all that stops a prune deleting that server's objects, so its // removal is an event whether it went through or was refused. -func clearOwnerMarkers(ctx context.Context, exec ssh.Executor, client *backup.Client, locs []backup.S3Config, fingerprint string, force bool, now time.Time) (int, error) { +// +// Every location is asked first, so a refusal removes nothing anywhere. After that +// nothing stops it partway: each location is tried, a previous one failing is skipped +// and reported, and the current one failing is the error returned at the end. +func clearOwnerMarkers(ctx context.Context, exec ssh.Executor, client *backup.Client, locs []backup.S3Config, fingerprint string, force bool, now time.Time) (int, []skippedLocation, error) { total := 0 - var err error - for _, l := range locs { - removed, cerr := client.ClearNamespaceMarkers(ctx, l, fingerprint, force, now) + skipped := []skippedLocation{} + var errs []error + finish := func() (int, []skippedLocation, error) { + err := errors.Join(errs...) + events.Record(ctx, exec, ownerClearEvent(fingerprint, force, total, skipped, err)) + return total, skipped, err + } + failed := func(i int, l backup.S3Config, err error) { + if i == 0 { + errs = append(errs, err) + } else { + skipped = append(skipped, skipLocation(l, err)) + } + } + tryable := make([]bool, len(locs)) + for i, l := range locs { + tryable[i] = true + if force { + continue + } + if err := client.CheckMarkersClearable(ctx, l, fingerprint, now); err != nil { + if errors.Is(err, backup.ErrMarkerRecent) { + errs = []error{err} + return finish() + } + tryable[i] = false + failed(i, l, err) + } + } + for i, l := range locs { + if !tryable[i] { + continue + } + // Already checked above, or --force: this step only removes. + removed, err := client.ClearNamespaceMarkers(ctx, l, fingerprint, true, now) total += len(removed) - if cerr != nil { - err = cerr - break + if err != nil { + failed(i, l, err) } } - events.Record(ctx, exec, ownerClearEvent(fingerprint, force, total, err)) - return total, err + return finish() } -func ownerClearEvent(fingerprint string, force bool, removed int, err error) events.Event { +func ownerClearEvent(fingerprint string, force bool, removed int, skipped []skippedLocation, err error) events.Event { ev := events.Event{App: backup.FleetApp, Kind: events.KindBackupOwners, Outcome: events.Succeeded, Detail: map[string]any{"fingerprint": fingerprint, "force": force, "removed": removed}} + if len(skipped) > 0 { + prefixes := make([]string, 0, len(skipped)) + for _, s := range skipped { + prefixes = append(prefixes, s.Prefix) + } + ev.Detail["skipped_locations"] = prefixes + } if err != nil { ev.Outcome = events.Failed if errors.Is(err, backup.ErrMarkerRecent) { @@ -202,7 +306,9 @@ func newServerBackupTargetSetCmd(flags *Flags) *cobra.Command { "--previous-prefix-owned says only this server ever wrote under the prefix\n" + "being left, and --claim-previous-prefix says it of an earlier one, which lets\n" + "them age out under the policy's retention (never where another server's\n" + - "marker says otherwise).", + "marker says otherwise).\n\n" + + "This does not install the aws CLI. It checks for it and warns; install it with\n" + + "`teploy server backup-preflight --install-tools` or `teploy setup`.", Args: cobra.NoArgs, RunE: func(cmd *cobra.Command, args []string) error { return runServerBackupTargetSet(flags, bucket, region, endpoint, prefix, noCredentials, prevOwned, dropPrevious, claimPrevious) @@ -383,7 +489,7 @@ func checkReachability(ctx context.Context, executor ssh.Executor, t backup.Targ client := backup.NewClient(executor, io.Discard) if err := client.EnsureAWSCLIPublic(ctx); err != nil { // The target is stored, and every backup through it will fail until this - // is fixed: say so in the terms that fix it. + // is fixed: say so in the terms that fix it. This does not install aws. fmt.Fprintf(os.Stderr, "Warning: backups through this target will fail: %v\n", err) return } diff --git a/internal/cli/backup_target_test.go b/internal/cli/backup_target_test.go index 7bd2d74..b49bf5a 100644 --- a/internal/cli/backup_target_test.go +++ b/internal/cli/backup_target_test.go @@ -184,23 +184,23 @@ func TestClearOwnerMarkersIsRecorded(t *testing.T) { t.Fatalf("no event: %v", mock.Calls) } mock := mk() - n, err := clearOwnerMarkers(context.Background(), mock, backup.NewClient(mock, io.Discard), s3, "aaaaaaaaaaaaaaaa", false, now) + n, _, err := clearOwnerMarkers(context.Background(), mock, backup.NewClient(mock, io.Discard), s3, "aaaaaaaaaaaaaaaa", false, now) if !errors.Is(err, backup.ErrMarkerRecent) || n != 0 { t.Fatalf("%d %v", n, err) } recorded(mock) mock = mk() - if n, err = clearOwnerMarkers(context.Background(), mock, backup.NewClient(mock, io.Discard), s3, "aaaaaaaaaaaaaaaa", true, now); err != nil || n != 1 { + if n, _, err = clearOwnerMarkers(context.Background(), mock, backup.NewClient(mock, io.Discard), s3, "aaaaaaaaaaaaaaaa", true, now); err != nil || n != 1 { t.Fatalf("%d %v", n, err) } recorded(mock) // Through Sanitize, which is what reaches the log: a kind's detail is allowlisted. - ev := ownerClearEvent("f", false, 0, backup.ErrMarkerRecent) + ev := ownerClearEvent("f", false, 0, nil, backup.ErrMarkerRecent) events.Sanitize(&ev) if ev.Outcome != events.Refused || ev.Detail["fingerprint"] != "f" || ev.Detail["error"] == nil { t.Fatalf("%+v", ev) } - ev = ownerClearEvent("f", true, 1, nil) + ev = ownerClearEvent("f", true, 1, nil, nil) events.Sanitize(&ev) if ev.Outcome != events.Succeeded || ev.Detail["force"] != true || ev.Detail["removed"] != 1 { t.Fatalf("%+v", ev) diff --git a/internal/cli/binarydist.go b/internal/cli/binarydist.go index 3b3190d..8a1ff94 100644 --- a/internal/cli/binarydist.go +++ b/internal/cli/binarydist.go @@ -21,16 +21,20 @@ import ( // extractBinary), just targeting the server's platform instead of the // operator's local one, and uploading instead of self-replacing. func deployTeployBinaryToServer(ctx context.Context, exec ssh.Executor, remotePath string, capabilities ...string) (version string, err error) { - goos, goarch, err := serverPlatform(ctx, exec) + latest, err := fetchLatestRelease(ctx) if err != nil { - return "", fmt.Errorf("detecting server platform: %w", err) + return "", fmt.Errorf("checking latest teploy release: %w", err) } + return installTeployRelease(ctx, exec, remotePath, latest.TagName, capabilities...) +} - latest, err := fetchLatestRelease(ctx) +// installTeployRelease is that pipeline for one named release tag. +func installTeployRelease(ctx context.Context, exec ssh.Executor, remotePath, tag string, capabilities ...string) (version string, err error) { + goos, goarch, err := serverPlatform(ctx, exec) if err != nil { - return "", fmt.Errorf("checking latest teploy release: %w", err) + return "", fmt.Errorf("detecting server platform: %w", err) } - latestVersion := strings.TrimPrefix(latest.TagName, "v") + version = strings.TrimPrefix(tag, "v") // goreleaser publishes archives named teploy_{os}_{arch}.tar.gz plus a // checksums.txt — matches update.go's runUpdate exactly, just with the @@ -41,8 +45,8 @@ func deployTeployBinaryToServer(ctx context.Context, exec ssh.Executor, remotePa const binName = "teploy" assetName := fmt.Sprintf("teploy_%s_%s.%s", goos, goarch, ext) - archiveURL := fmt.Sprintf("%s/%s/%s", downloadBase, latest.TagName, assetName) - checksumURL := fmt.Sprintf("%s/%s/checksums.txt", downloadBase, latest.TagName) + archiveURL := fmt.Sprintf("%s/%s/%s", downloadBase, tag, assetName) + checksumURL := fmt.Sprintf("%s/%s/checksums.txt", downloadBase, tag) archive, err := downloadToBytes(ctx, archiveURL) if err != nil { @@ -66,11 +70,11 @@ func deployTeployBinaryToServer(ctx context.Context, exec ssh.Executor, remotePa return "", fmt.Errorf("extracting binary: %w", err) } - if err := installServerBinary(ctx, exec, remotePath, binData, latestVersion, capabilities...); err != nil { + if err := installServerBinary(ctx, exec, remotePath, binData, version, capabilities...); err != nil { return "", err } - return latestVersion, nil + return version, nil } // serverPlatform runs uname on the target and maps the result to Go's diff --git a/internal/cli/contracts_golden_test.go b/internal/cli/contracts_golden_test.go index 131d62d..59c6b3f 100644 --- a/internal/cli/contracts_golden_test.go +++ b/internal/cli/contracts_golden_test.go @@ -645,6 +645,69 @@ func TestContractsSiblingsGolden(t *testing.T) { writeFixture(t, "siblings-envelope/valid/none.json", none.collect(t)) } +// TestContractsBackupSetupGolden pins what Dash's simple backup setup calls (rev 21, +// backup-setup-v1). Preflight fixtures come from the real collector and actions over a +// mock executor, the detached run's from the real start path, head's from the real DTO. +func TestContractsBackupSetupGolden(t *testing.T) { + ctx := context.Background() + const host = "192.0.2.10" + + ready := collectBackupPreflight(ctx, ssh.NewMockExecutor(host, preflightCommands(probeReady, targetStored, policyStored, cronFleet, managedPlain)...), serverTeployPath, "0.1.55", setupNow) + writeFixture(t, "backup-preflight-envelope/valid/ready.json", ready) + + bare := collectBackupPreflight(ctx, ssh.NewMockExecutor(host, preflightCommands(probeBare, "absent\n", "absent\n", "", "")...), serverTeployPath, "0.1.55", setupNow) + writeFixture(t, "backup-preflight-envelope/valid/needs-setup.json", bare) + + unknown := collectBackupPreflight(ctx, ssh.NewMockExecutor(host, + ssh.MockCommand{Match: "sh -c '" + preflightProbeMarker, Output: "binary_broken\t1\naws\tsome wrapper\n"}, + ssh.MockCommand{Match: "for p in /etc/teploy/backup-target.json", Output: "garbage"}, + ssh.MockCommand{Match: "if [ -d /etc/teploy ]", Err: errors.New("exit status 3: cannot tell whether this server's backups must be encrypted")}, + ), serverTeployPath, "0.1.55", setupNow) + writeFixture(t, "backup-preflight-envelope/valid/unknown.json", unknown) + + writeFixture(t, "backup-preflight-envelope/valid/unreachable.json", unreachablePreflight(host, serverTeployPath, "0.1.55", errors.New("dial tcp 192.0.2.10:22: i/o timeout"), setupNow)) + + acted, err := backupPreflightOn(ctx, installMock(probeBare, probeReady), preflightOptions{ + InstallTools: true, UpgradeBinary: true, Binary: serverTeployPath, CLIVersion: "0.1.55", Now: setupNow, + InstallRelease: func(context.Context, ssh.Executor, string, string, ...string) (string, error) { return "0.1.55", nil }, + }) + if err != nil { + t.Fatal(err) + } + writeFixture(t, "backup-preflight-envelope/valid/after-actions.json", acted) + + total := int64(1048576) + writeFixture(t, "backup-head-envelope/valid/encrypted.json", newBackupHeadDTO(host, 22, backup.ObjectStart{ + Key: "s3://teploy-backups/teploy/srv1/blog/accessories/db/20261104-031704-1f2e3d4c5b6a7988.pgdump.age", Data: []byte("age-encryption.org/v1\n"), TotalSize: &total, Encrypted: true, + })) + + started, err := startDetachedRun(ctx, detachMock("started 4242 /root/.config/teploy/backup-runs/run.AbCd1234\n"), false, events.Actor{}, setupNow) + if err != nil { + t.Fatal(err) + } + writeFixture(t, "backup-detach-envelope/valid/started.json", started) + + skipped, _ := json.Marshal(backupRunEnvelope{MachineInterface: MachineInterface, FleetRunResult: backup.FleetRunResult{Status: "skipped", StartedAt: setupNow, Targets: []backup.FleetTargetResult{}, Error: "another run holds the lock"}}) + running, err := startDetachedRun(ctx, detachMock("finished 4243 /root/.config/teploy/backup-runs/run.EfGh5678\n"+detachResultMark+"\n"+string(skipped)+"\n"+detachLogMark+"\n"), false, events.Actor{}, setupNow) + if err != nil { + t.Fatal(err) + } + writeFixture(t, "backup-detach-envelope/valid/already-running.json", running) + + failedRun, _ := json.Marshal(backupRunEnvelope{MachineInterface: MachineInterface, FleetRunResult: backup.FleetRunResult{Status: backup.RunFailed, StartedAt: setupNow, Targets: []backup.FleetTargetResult{}, Error: "no backup policy on this server: set one with `teploy server backup-policy set`"}}) + finished, err := startDetachedRun(ctx, detachMock("finished 4244 /root/.config/teploy/backup-runs/run.IjKl9012\n"+detachResultMark+"\n"+string(failedRun)+"\n"+detachLogMark+"\n"), false, events.Actor{}, setupNow) + if err != nil { + t.Fatal(err) + } + writeFixture(t, "backup-detach-envelope/valid/finished.json", finished) + + died, err := startDetachedRun(ctx, detachMock("finished 4245 /root/.config/teploy/backup-runs/run.MnOp3456\n"+detachResultMark+"\n"+detachLogMark+"\nError: docker: permission denied\n"), false, events.Actor{}, setupNow) + if err != nil { + t.Fatal(err) + } + writeFixture(t, "backup-detach-envelope/valid/failed.json", died) +} + func TestContractsBackupFleetGolden(t *testing.T) { at := time.Date(2026, 11, 4, 3, 17, 4, 0, time.UTC) failedAt := at.Add(-time.Hour) diff --git a/internal/cli/machineinterface.go b/internal/cli/machineinterface.go index 67748c9..8e3856c 100644 --- a/internal/cli/machineinterface.go +++ b/internal/cli/machineinterface.go @@ -208,6 +208,16 @@ const ( // confirms; unrecognised dev.teploy.product values pass through // verbatim. siblings-envelope pins the JSON. CapServerSiblingsV1 = "siblings-v1" + // CapBackupSetupV1: what Dash's simple backup setup calls. + // `server backup-preflight [--install-tools] [--upgrade-binary] --json` + // (read-only unless a flag says otherwise; failed probes read as unknown), + // `backup head --bytes N --json` (ranged read, never decrypts, own + // prefixes only) and `backup run --all --detach --json` (survives the + // connection, same lock). `backup-target owners` skips previous locations + // it cannot reach and `--clear` takes --json and no longer stops partway. + // backup-preflight-envelope, backup-head-envelope and backup-detach-envelope + // pin the JSON. + CapBackupSetupV1 = "backup-setup-v1" ) // MachineCapabilities returns every capability token this build @@ -254,7 +264,8 @@ func MachineCapabilities() []string { CapEnvConfigGuardV1, CapDomainsCheckV1, CapServerAuditV1, - CapServerSiblingsV1} + CapServerSiblingsV1, + CapBackupSetupV1} sort.Strings(tokens) return tokens } diff --git a/internal/cli/machineinterface_test.go b/internal/cli/machineinterface_test.go index 25bae82..7cfd51a 100644 --- a/internal/cli/machineinterface_test.go +++ b/internal/cli/machineinterface_test.go @@ -106,7 +106,7 @@ func TestCapabilityTokenRegistry(t *testing.T) { "trigger-convergence-v1", } want = append(want, "actor-flag-v1", "events-v1", "config-home-v1", "undeployed-list-v1", "lock-wait-v1", "config-merge-v1", "config-unprivileged-v1", "deploy-home-v1", "backup-target-v1", "backup-json-v1", "backup-fleet-v1") - want = append(want, "secret-stdin-v1", "releases-json-v1", "rollback-json-v1", "redeploy-v1", "env-config-guard-v1", "git-source-v1", "trigger-poll-v1", "server-audit-v1", "siblings-v1") + want = append(want, "secret-stdin-v1", "releases-json-v1", "rollback-json-v1", "redeploy-v1", "env-config-guard-v1", "git-source-v1", "trigger-poll-v1", "server-audit-v1", "siblings-v1", "backup-setup-v1") sort.Strings(want) got := MachineCapabilities() if len(got) != len(want) { diff --git a/internal/cli/server.go b/internal/cli/server.go index 0b368ba..16dd87e 100644 --- a/internal/cli/server.go +++ b/internal/cli/server.go @@ -28,6 +28,7 @@ func newServerCmd(flags *Flags) *cobra.Command { cmd.AddCommand(newServerListCmd(flags)) cmd.AddCommand(newServerStatusCmd(flags)) cmd.AddCommand(newServerBackupTargetCmd(flags)) + cmd.AddCommand(newServerBackupPreflightCmd(flags)) cmd.AddCommand(newServerBackupPolicyCmd(flags)) cmd.AddCommand(newServerAuditCmd(flags)) cmd.AddCommand(newServerSiblingsCmd(flags)) diff --git a/internal/events/events.go b/internal/events/events.go index d4d514b..fe7e159 100644 --- a/internal/events/events.go +++ b/internal/events/events.go @@ -120,6 +120,9 @@ const ( KindRouteRemove = "route.remove" KindReleasesPrune = "releases.prune" KindServerPrune = "server.prune" + KindBackupDetach = "backup.detach" + KindServerTools = "server.tools_install" + KindServerBinary = "server.binary_upgrade" ) // detailAllow is the per-kind allowlist of detail keys. Env and secret @@ -154,12 +157,15 @@ var detailAllow = map[string][]string{ KindBackupVerify: {"accessory", "database", "date", "ok", "engine", "kind", "metric", "key", "size_bytes", "duration_ms", "encrypted", "error"}, KindBackupRestore: {"id", "kind", "accessory", "action", "database", "released", "pre_backup_id", "error"}, KindBackupTarget: {"id", "kind", "accessory", "error"}, - KindBackupOwners: {"fingerprint", "force", "removed", "error"}, + KindBackupOwners: {"fingerprint", "force", "removed", "skipped_locations", "error"}, KindBackupAll: {"status", "targets_ok", "targets_failed", "omissions", "encrypted", "duration_ms", "heartbeats_ok", "heartbeats_failed", "warnings", "error"}, KindBackupPolicy: {"interval", "encrypted", "volumes", "adopted", "error"}, KindRouteRemove: {"route", "hosts", "upstreams", "reason", "error"}, KindReleasesPrune: {"keep", "pruned", "protected", "trigger", "error"}, KindServerPrune: {"build_cache", "dangling", "output", "error"}, + KindBackupDetach: {"status", "pid", "log", "error"}, + KindServerTools: {"installed", "failed", "error"}, + KindServerBinary: {"from", "to", "error"}, } // maxErrorDetail bounds the "error" detail so a command's full stderr