Repository navigation
feat(backup): fleet backup engine - #65
Merged
Merged
Conversation
…ies, GFS, prefix safety Every artifact and sidecar now goes through one store path (G1, G3, G5): - put: age-encrypt to the owner key and the server identity, prove the ciphertext decrypts, upload to .incoming/, check size, recorded SHA-256 and (to 256 MiB) the bytes, then copy server-side to the final key. A final key only ever holds a verified object; sidecars precede the primary that commits a backup; a failure to encrypt never uploads plaintext. - fetch: checks the recorded checksum and free space, decrypts on the server with its own identity. Legacy objects keep restoring. - One object per database for postgres and mysql/mariadb, in a series of its own; hostile names are shell-quoted and key-encoded, and a name the tools cannot address is reported, never skipped silently. - Adopted containers: the _adopted namespace, addressed by docker name. - Retention: restic forget semantics (keep_last, daily, weekly, monthly, and max_age_days as keep-within). The newest good backup is never pruned, below-floor objects never count, every sidecar goes with its backup, refused deletes do not stop the rest. - Prefixes: listing no longer breaks under a prefix; previous prefixes are searched and only pruned when owned; .owners markers refuse a prune in a namespace another server wrote to. - Backups build in /var/tmp with a space preflight that measures the backup and its ciphertext, and have deadlines. - DR bundles: members encrypted, the age key wrapped to the owner only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lap lock, heartbeats, status The engine behind `teploy backup run --all` (G4): - /etc/teploy/backup-policy.json, 0600, with the non-root fallback; an unreadable policy fails closed instead of reading as no encryption. - Enumeration at run time: labelled accessories, adopted containers, app volumes; a per-app schedule overrides, a hand-written job under a marker does not; docker failing is an error, not an empty server. - flock overlap lock: a killed run cannot block the next, a live holder past every deadline is killed after checking the pid is teploy's. - One target failing does not stop the next, nothing of a failed target is pruned, and every omission is named, so ok never hides one. - Heartbeats with status=down semantics; no URL or token in any result. - Volume size guard, killed-run janitor, multipart abort. - ReadFleetStatus: one read-only report of every DB-like container, no secrets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, backup before restore - `teploy server backup-policy set/show/unset`: exactly one cron line, marker [teploy-backup-all]; flags change only what they name; --stdin is the machine path; heartbeat URLs arrive on stdin only and show back as hosts. - `teploy backup run --all`, with a nonzero exit for anything but a clean run. - `teploy backup status --host S --json`. - `teploy backup keygen`, printed once; `teploy backup fetch <key> --identity`, local download and decrypt with nothing else, no partial result left behind. - --adopted and --database on restore, verify-backup, list, create, prune. - backup restore and accessory restore/upgrade store a fresh backup first and refuse if that fails (--no-backup); a database that is already gone has nothing to protect and is not a refusal. - backup-target set keeps the prefix history. - dr create wraps --include-age-key; dr restore --identity unwraps it. Contracts rev 17: backup-status, backup-run and backup-policy envelopes, additive fields on backup-list/create/restore/target, capability backup-fleet-v1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… what it deletes remove --purge stores a fresh backup of every accessory and the volumes first, refuses if that fails or no target exists (--no-backup), names what it will delete in its prompt, and removes containers with docker rm -v. backup_before_hooks (default off) does the same for a deploy with a pre_deploy hook. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
doctor flags a policy with no target, no cron line, no recipient, stale runs, no heartbeat, a missing aws or age, a target that does not answer, leftover workspaces and uploads, and databases nothing backs up. setup installs the aws CLI, through AWS's bundle where the distribution has none (Ubuntu 24.04), with modes every user can execute; server backup-target set names the exact remedy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the aws CLI TestFleetIntegration runs the real linux binary on the colima fixture: mixed engines with an adopted multi-schema mysql and hostile names, a database appearing between runs, one schema restored alone, server-identity verify and a corrupt object, a legacy object after encryption, recovery with only the owner key, kills mid-upload and before publishing, a stuck lock holder, overlap, partial failure, heartbeats, exact GFS sets in a real bucket, prefix moves, shared namespaces, status, destructive operations refusing and backing up, a wrapped DR bundle, doctor. The engine suite's fake aws gains head-object, stdin/stdout copies and recursive listing, and its executor routes RunStream through the fake too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Blockers: - purge, upgrade and the deploy hook backed up only the primary: remove --purge said "backed up", stored appdb and destroyed analytics. They now store every database and refuse what they cannot cover; only a restore is scoped to one. - A leftover backup-schedule.json decided a policy run's retention (keep_last 2 deleted seeded history, reporting ok). The policy applies; the leftover warns. High: - A run that stored nothing still pruned, and read as ok. No objects, no prune; a target with earlier backups and nothing now is failed, heartbeat down. - Per-database objects followed the primary's ids. Each series is selected by the policy on its own; the primary's kept ids are an extra keep only. - .owners failed open. A listing error refuses; empty machine-id refuses; a marker is written before the first prune; machine-id + hostname name it and the age recipient in its body tells clones apart; previous prefixes are checked too. - status never looked at the bucket. Covered needs a restorable recent object, and only policy runs count: a pre-op copy cannot mask a failing --all. - Owner-key recovery: -i only when the identity exists, dr --prefix, --identity on backup restore, accessory restore and verify-backup. - A non-root user reading a 0700 /etc/teploy saw no policy and uploaded plaintext. It asks through passwordless sudo, follows its own ~/.config policy, or refuses. - backup list hides per-database rows unless --databases: Dash restores by id and accessory alone, which would replace the primary. Medium and low: refused prunes reach run warnings, events and doctor, with `backup-target owners [--clear]`; --ignore-space-check reaches the pre-restore backup; zero targets fail unless allow_empty; .incoming at any depth; an inspect error is not "no container"; prune refuses a clock an hour off the store's or a future-dated backup (--ignore-clock); --previous-prefix-owned marks only the prefix being left (--claim-previous-prefix for others); AWS's bundle is PGP-checked against a pinned key and aws must actually run; --deadline and target_deadline; legacy cron lines reported per entry and warned about; ClickHouse and friends reported uncovered; --no-credentials targets work; policy --stdin cannot silently drop the recipient and --host no longer counts as a policy flag; multipart abort scoped to owned namespaces; tmp_dir and TEPLOY_BACKUP_TMPDIR workspaces in the janitor and doctor; postgresql:// names refused; a stale holder's child groups killed; --checksum-algorithm above the read-back limit; safety copies protected for 7 days. Contracts rev 17 regenerated: per-entry encrypted and schedule_form, last_run.warnings, allow_empty, tmp_dir, target_deadline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Purge and upgrade across databases; a stale schedule store; every database dropped; each series' own retention in a real bucket; the real --previous-prefix-owned and a foreign marker under it; a wiped namespace; owners --clear; a real cron firing with cron's PATH; a non-root user behind a 0700 /etc/teploy, with sudo and without; zero targets; a stuck holder's children; a target over the read-back limit; policy --stdin; clean-host recovery through `backup restore --identity` and `dr restore --identity --bucket --prefix` with no server identity; AWS's installer, live, with a wrong pin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts, docs only: contracts/MANIFEST.md (#63 took rev 17 for domains-check-envelope; the fleet engine's artifacts are renumbered rev 18, here and in the backup-* schemas), CHANGELOG.md (this branch stays under Unreleased, v0.1.51 is released) and AUDIT_OPEN.md (both sections kept). Fleet test fixtures use teploy-fixture.dev rather than a reserved name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…write to Per-server prefixes are the answer to objects no marker can attribute, and Dash always sets one. backup status reports other_servers when there is no prefix, from the markers at the bucket root; doctor warns with the exact remedy. set keeps its warning. Servers that have not upgraded leave no marker, so they stay invisible here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Publish: the copy names --copy-props metadata-directive (R2 has no GetObjectTagging), and a store that refuses it gets the verified local file uploaded straight to its final name and verified there, since B2 documents that a copy cannot change a checksum algorithm. A final object that lost the recorded checksum (aws CLI v1's multipart copy) is replaced the same way. Pending uploads move to one <prefix>/.incoming/ tree so a 1-day lifecycle rule can expire them on a store that keeps versions; doctor and status read the rule. Markers are written once; the store's clock comes from a probe under .incoming/. The policy is written and unset through the same sudo path reads use, a policy this user cannot replace is a refusal, WriteFleetPolicy reads itself back, and the run lock falls back to ~/.config when the policy's directory is not writable. Prune refuses when it would delete most of a series (--allow-mass-delete) and when the store's clock cannot be read. Future-dated objects leave selection instead of stopping it. Safety copies are capped at 5 per target. owners --clear refuses a namespace written to in the last week without --force, and is recorded. target_deadline and --deadline over 23h are refused. A workspace carries its own expiry for the janitor; a pending upload is a leftover after 24h. An unreachable previous prefix only warns. aws CLI is held to 2.23.0/1.37.0, the head-object checksum read falls back through checksumUnsupported, the stuck holder dies before its children, and the chmod 711 remedy is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…print limits Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MinIO needs MINIO_CI_CD to erasure-code directories on one disk, and a home policy left by an earlier scenario is not the second copy the sudo scenario looks for. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The fleet backup engine (Dash auto-config Slice 4): one policy on each server, one cron line, and coverage that can be checked.
server backup-policyandbackup run --all, with adopted containers, GFS retention and heartbeats.backup keygenandbackup fetchcover recovery with only the owner key and bucket credentials.--databaseon create, restore and verify;--databaseson list.backup status --jsonreports every DB-like container and volume. Contracts rev 18 pins the JSON.--no-backupoverrides), a mass-delete brake, owner markers per prefix, and a.incomingpublish path with a re-upload fallback.Upgrade note: backups now refuse aws CLI older than 2.23.0 / 1.37.0, and
teploy setupreplaces an old package. On the live fleet, infra-home has 2.23.6 and the servers checked otherwise have no aws yet.Testing: two review rounds. The integration suite passes on all 9 engines, and the fleet suite passes 25/25, all against MinIO.
Not proven on real B2 or R2: the copy refusal is simulated with a shim, metadata through multipart copy is untested on R2, and lifecycle expiry was never observed. Slice 1 is blocked on a real-B2 smoke test for this reason.
🤖 Generated with Claude Code