[robinnsc]

Backup/restore series (design: #396): #384 restore fidelity (merged) · #393 store · #394 format · #395 SnapshotEngine · #666 catalog migration framework (merged) · PR 6 the catalog index · PR 7 configuration, the store identity file and backup purge · #700 the CreateBackup job · #740 the restore job, the cross-backend matrix and backup remove-target · this PR: extenddb backup sync, --prune-missing, --adopt-store and the sqlite-memory startup sync · next: #820 the store-maintenance options, #821 migrate-legacy.
Closes #746 (part of #624).

Stacked on #740 (feat/restore-job, head 9803ab7f), itself on #700 (c461442a), #672 (ba6b82d4) and #671 (6ff37487) on main at 94b5f689. Rebased from #740's earlier head 6bdd1177 onto that stack after its review rounds and main's eight commits under it: PR 10 now meets #740's bounded store calls, its DeleteBackup that answers DELETED once the DELETING write commits, its CreateBackup that checks the maintenance claim before any row, its signal listener opened before every command's catalog, its purge_rows and its MongoDB deployment of tests/test_backup_job.py, and is adapted to each (below). The rebase conflicted in 14 files, each resolved by keeping both sides' intent: crates/app/src/cmd_backup.rs (#740's run with sync added; sync takes the run's signal handle), crates/engine/src/backup.rs (DeleteBackup's second fence logged and answered DELETED), backup/job.rs, backup/maintenance.rs (the fence threaded through purge_rows; the skip of FAILED maintenance rows dropped, #700 writes no such row), backup/reconciler.rs, backup/runtime.rs (both sides' ConfiguredStore fields; the fence's read through BoundedCalls), backup/tests.rs, crates/storage-cassandra/src/test_hooks.rs, crates/storage-postgres/src/backup_catalog.rs, docs/design/15-backup-restore.md (this PR's revision item renumbered 34 after #740's 28 to 33), docs/differences-from-dynamodb.md, docs/manuals/05-admin-guide.md, docs/manuals/06-developer-test-guide.md and tests/test_backup_job.py (the sync cells skip on the MongoDB deployment). The design's PR 10 is split in three (revision item 34): this PR is the catalog-loss recovery, #820 the options that rewrite or delete rows of other stores, #821 the legacy-backup migration. No migration, no schema change, no new dependency; Cargo.lock is unchanged. The identity file stays format version 1: its one new member is #[serde(default)], so this release and #740's read each other's files.

What

Storage trait (crates/storage). Two BackupCatalog methods, with no default body. insert_backup_row(&BackupRow) -> bool, beside claim_backup: insert the row as given, unowned (owner_generation 0), its status as given, completed_at and heartbeat_at on the database's clock; false when a row with its key exists, or with its backup id on the three backends that index it. The trait doc says that Cassandra's check is per ARN spelling and that sync closes the gap, and that on Cassandra a stored row equal to this one counts as inserted. forget_available_backup_row(&BackupKey) -> ForgetRow: remove an AVAILABLE row outright under the row's lock with start_backup_delete's in-use rule, leaving no tombstone and touching no object; InUse(table) and Status(status) leave the row. StorageConfig::database_location keeps the reason a location names no file.

Core (crates/core). layout.rs gains EXPORTS_COMPONENT, CHANGELOG_COMPONENT, StoreKey and classify_store_key(key), which tells a key outside every backup prefix, a key under a reserved component (exports/ first, changelog third), three components that are not a backup prefix (another id shape, a component no ARN carries) with the reason, and a backup prefix with its account, table and id; parse_store_prefix(key) is its Option form. Only the 14-digit id shape is a backup prefix, and the triple is validated through BackupArn::new.

PostgreSQL, SQLite, MongoDB, Cassandra. insert_backup_row: PostgreSQL INSERT ... ON CONFLICT DO NOTHING with no conflict target (the key and the unique backup_id both refuse it), on the pool, with no advisory lock (a job that holds the id's lock and has not inserted its row yet finds this one and is refused). SQLite INSERT ... ON CONFLICT DO NOTHING (so a NOT NULL or CHECK violation is an error, not a false) under the write lock and BEGIN IMMEDIATE, without a job slot. MongoDB the claim's pipeline upsert filtered on owner_generation: {$exists: false}, writing owner_generation: 0 and $$NOW for the two stamps, E11000 (the unique backup_id) meaning a row exists; row_fields takes the status, and claim_backup passes CREATING. Cassandra the backups_by_arn lightweight transaction IF NOT EXISTS, then the LOCAL_QUORUM batch of backups_by_account, backups_by_table and the backup_index_by_status entry under the row's status (write_index_entries, the batch claim_backup wrote inline, now shared); an unknown outcome or a NotMatched answer is re-read at LOCAL_SERIAL and settled as inserted when the row read is this one (its entries then written, which completes an insert that stopped between its two writes), as existing when it is another; no row after a lost answer is sent once more (then an error), and no row after NotMatched answers false (a legacy in-catalog backup holds the ARN); a failed entry batch is sent once more. forget_available_backup_row shares one body with start_backup_delete on every backend, with a remove flag: PostgreSQL and SQLite start_backup_delete_tx(key, remove) (the row lock, the status check, the in-use rule, then UPDATE ... DELETING or DELETE); MongoDB start_backup_delete_tx with delete_one, its exhaustion error naming DeleteBackup or the --prune-missing removal; Cassandra start_backup_delete_lwt(key, remove), one loop under one condition (IF backup_status = 'AVAILABLE' AND active_restores = null) through conditional_write, where a removal's unknown outcome is settled by a LOCAL_SERIAL read (no row: removed, then the entries; a row: the loop runs again) and an entry batch that fails after the removal is logged and answered Removed. SQLite's claim_backup keeps INSERT OR IGNORE, with a comment saying why it ignores only a key conflict.

Engine (crates/engine/src/backup/). New sync.rs, written against BackupCatalog and &dyn BackupStore, which the CLI and the server both call:

• sync reads the configured root's rows of every status and every store id first, then lists the whole root and handles each prefix as the listing passes it (the store lists in key order; sync holds one prefix's objects and the prefix of each manifest it listed, and keeps of the rows only each one's status, store id and table and the keys of the configured store's AVAILABLE rows), checking it against the rows before it reads anything under it. A listing that fails part-way keeps the rows inserted before it and makes no missing-manifest report; a stop asked through SyncTarget::stop ends the run before the next prefix. A prefix with no row is verified by reconciler::verify_prefix (split out of probe_manifest, which keeps its comparison of the whole ARN and maps to StatusFields; the dead-owner tests pass unchanged): the manifest and sidecar within their bounds, the digest, the reader's acceptance, and every data file the manifest lists at its recorded size, from the listing already held. The manifest's ARN must name the listed account, table and backup id; another region is adopted and logged, another partition skipped (no request can address it). Before the insert the row is read under both spellings of the ARN (this server's region and the manifest's) through the engine's BackupArnSpellings, and a backup the other spelling has a row for is skipped; a row under the manifest's own spelling is left to the insert, which answers false for it or, on Cassandra, completes its own insert that stopped before its entries. The adopted row is the one a CreateBackup leaves: the manifest's name, ARN, size, item total and source_table, the id's instant as created_at, the configured store's identity. An insert that answers false is re-read so the report says which row it met. A failed read of one prefix's manifest or sidecar (SkipReason::Store) or a failed catalog call (SkipReason::Catalog) is reported for that prefix and the sync goes on; SyncReport::transient_skips counts them. The report also lists the AVAILABLE rows whose manifest the listing lacks.
• prune_missing heads each such row's manifest once more and removes the row outright (forget_available_backup_row), deleting no object; it reports InUse and a row another writer moved, and checks the claim and the signal before each row. A head that fails keeps the row (PruneOutcome::Failed): the store has not said whether the manifest is back. PruneOutcome::needs_another_run names the outcomes a later run may change (Failed, InUse), which the command exits 1 for.
• startup_sync is the in-memory catalog's sync: when the store is healthy and the sync pending, bounded by STARTUP_SYNC_TIMEOUT (60 s); a run that stops before it listed the whole root, reaches the bound, or skipped a prefix for a store or catalog error stays pending, those skips logged at WARN. Every run lists the root from its first key, so the startup sync completes only while listing the whole root fits in the bound; past it the sync stays pending and is retried every cycle.

store_identity.rs: the file's ephemeral_catalog member; adopt(store, kind, root, catalog, backend, ephemeral, rule) rewrites the file's catalog_id and backend with a plain put, keeping store_id and created_at_ms, reads it back (AdoptRefusal::RaceLost when another write landed over it), refuses when this catalog's rows for the root carry another store id, and logs an extenddb::audit intent line before the put and the audit line after the re-read with its outcome; AdoptRefusal::NoFile carries the rows' store ids, so a root that lost its file under rows is identity file missing and a fresh one identity file unreadable rather than unreachable; establish of an in-memory catalog adopts a root whose file names another catalog and carries the marker (AdoptRule::EphemeralOwner), and recheck never adopts. runtime.rs: ConfiguredStore::with_ephemeral_catalog and the pending startup sync (take, retry); BackupRuntime::with_region; and StoreFence (ConfiguredStore::fence), the deletion fence: it shares the store's health and reads the catalog id once, and its check re-reads extenddb-store.json with one get under the store's check bound, refuses with no read on a store that is not healthy, and when the file names another catalog records ForeignCatalog on the store and refuses, and when it names this catalog under another store id than the one the store's check settled (a root replaced by another root of the catalog; the store's id is now shared with its fences) records IdMismatch and refuses; a file it cannot read, or a root with no file, refuses the call and leaves the health as it was. It runs before DeleteBackup's DELETING write and before purge_row removes a dead CREATING row or moves an AVAILABLE one, so a refusal there for a root another catalog owns leaves the row as read (DescribeBackup still answers AVAILABLE; a file DeleteBackup cannot read there is a store failure, and the request goes on to the DELETING write), and again right before each delete_prefix of a backup's objects: finish_backup_delete (now taking the fence and answering FinishError; an adoption between the two reads leaves the row DELETING with nothing deleted and DeleteBackup answering DELETED, as for any deletion that fails after the DELETING write, which this catalog's reconciler finishes only if this catalog owns the root again), the job's fail_row (FailurePath::Fenced), the reconciler's FAILED-prefix and DELETED-prefix duties, and purge_row; before the reconciler removes a tombstone; and in CreateBackup before claim_backup, so a fenced create writes no row and no object. The restore deletes nothing in the store and has no fence, and neither has --prune-missing's removal of a row, which deletes no object; StoreFence's type doc names every call site and says so. reconciler.rs: StoreCalls and verify_prefix are pub(super), the cycle runs a pending startup sync after a healthy store check, within its bound, and it re-tests the store's health after each store duty, so a fence refusal stops the duties after it, the dead-owner duty included; the SQLite command's prelude logs a line of its own instead of "a maintenance claim is held". maintenance.rs: purge_row takes the fence, and its comment says why nothing adopts a prefix between its two steps (every sync takes the claim); restore.rs: OTHER_STORE and the missing-manifest error line say what an operator can do in this release; a missing sidecar's line says to put the prefix back, since --prune-missing removes only a row whose manifest is gone.

Server (crates/server). backup_store.rs: Caller::SyncCommand, as Command except that a defaulted SQLite root whose file names another catalog, while the catalog has no rows for it, is configured and unhealthy rather than the warning that leaves it unconfigured (PR 7's leniency, which the server and purge keep); the ephemeral flag from backend == "sqlite" and database_file().is_none(); under the exception the unhealthy line says the root is the shared default one with no rows, configured for backup sync; the warning for a defaulted root another catalog owns names backup sync --adopt-store; a command's unhealthy-store line is its own, without the server's sentence about starting and /health. serve.rs: the region on the runtime, and the startup sync of a filesystem store before the routes serve, within the engine's bound.

App (crates/app). extenddb backup sync [--adopt-store] [--prune-missing]: opened as serve opens the catalog (on SQLite under the serve lock, after resolve_dead_owners), its serving registration named extenddb backup sync; prints a header naming the store, the adopted ARNs, each skipped prefix with its reason, the AVAILABLE rows whose manifest is gone (with "run with --prune-missing to remove the rows, which deletes no object" when the option is absent), and a count line; exits 0 once it ran, 1 on a refusal (an unhealthy store, a root another catalog owns without --adopt-store, a held or lost claim), a listing that failed part-way, a stop, a row --prune-missing left for a later run (a store or catalog call failed, a failed head included, or a restore is reading the backup, with "run the command again once the restore ends"; a row whose manifest is back or that another writer moved is reported with exit 0), or a prefix skipped for a store or catalog error, 6 and 3 on the catalog refusals. The missing-manifest heading says the rows were AVAILABLE when the catalog was read, since a DeleteBackup on another instance can move one during the run. On PostgreSQL, MongoDB and Cassandra every sync takes the maintenance claim and heartbeats it, plain sync without the wait; --prune-missing and --adopt-store also wait for running jobs as purge does. Every sync installs the signal listener and stops before its next prefix. An adopting sync reads the identity file once more at the end and fails naming the owner if it changed (owner_after_sync); a run with a store or catalog skip exits 1 (complete_run); app unit tests cover both decisions, and the wire cells test_an_adopter_whose_put_lands_during_the_sync_ends_it_with_exit_1 and test_a_prefix_the_store_cannot_read_leaves_the_sync_incomplete pin their call sites. An adoption whose check then fails says the identity file was rewritten, so the root is taken from its previous owner. A stopped or failed run prints "(not examined)" under the missing-manifest header, and a stopped prune's count line says "stopped". Plain sync's refusal of a root another catalog owns, --adopt-store's output and its help say to stop every server of the catalog the file names before adopting (a running one acts on the root until a later check that can read its catalog, its fence excepted), and the refusal that every database file of a directory shares the default root beside it. purge passes the store's fence to purge_row. Every backup subcommand refuses an in-memory SQLite database before it opens a catalog, and reports a location that does not parse with its parse error. The clap help names sync. destroy's output names backup sync --adopt-store.

Harness and matrix. tests/test_backup_job.py, on the SQLite and PostgreSQL deployments: test_sync_rebuilds_dropped_rows_and_adopts_only_verified_prefixes, test_prune_missing_and_the_unhealthy_store_refusals, test_a_second_maintenance_command_fails_at_once (PostgreSQL; on SQLite the serve lock refuses a second writing command first; --adopt-store on a root the catalog owns waits for the same live job, "Waiting for running jobs to finish", and a signal stops it before any change), test_prune_missing_on_a_host_local_root_deletes_no_object and test_sync_is_refused_while_purge_holds_the_claim (PostgreSQL), test_adopt_store_makes_this_catalog_the_owner (a second catalog of the build's backend, its PostgreSQL databases dropped after), and test_the_previous_owner_deletes_nothing_under_an_adopted_root (PostgreSQL: the previous owner's server keeps running with its next check an hour away, its DeleteBackup of the adopted backup is refused, the bytes stay and DescribeBackup there still answers AVAILABLE, its /health then reports owned by another catalog, and the adopting catalog's server is healthy and restores the backup), and test_sigterm_during_the_prefix_loop_stops_sync_and_keeps_its_rows (SIGTERM after the first two prefixes' rows are back, 10,000 skipped copies still to go: exit 1, "stopped", the rows kept, and the next sync runs at once and adopts nothing more), test_a_prefix_the_store_cannot_read_leaves_the_sync_incomplete (a row-less prefix whose manifest is mode 000: the "store could not be read" skip, exit 1 "the run is incomplete", and the next sync adopts it once the mode is restored), and test_an_adopter_whose_put_lands_during_the_sync_ends_it_with_exit_1 (a second catalog's --adopt-store with 10,000 skipped copies under the root; once "Adopted the root:" is printed and while the sync is in its prefix loop the identity file is rewritten to name a third catalog, and the command exits 1 naming it and the store id). tests/test_backup_store_config.py: test_an_in_memory_server_restarted_onto_its_root_serves_the_previous_starts_backups, and backup sync in both tuples of test_a_refused_catalog_ends_the_backup_commands_with_the_serve_exit_codes (exit 6 while a migration holds the serving key, 3 on a catalog behind the binary). devtools/backup-restore-matrix copies the source's prefix and runs extenddb backup sync on the target (on SQLite with the target's server stopped) in place of seed_row, checks the adopted row's name and size against the manifest, and gains --adopt-store (no copy: the target's store is the source's root; after the adoption the restarted source must report owned by another catalog); matrix-nightly.yml runs it from SQLite to PostgreSQL and back, and its comments and requirements-matrix.txt's say what the script now does.

Docs and texts. The admin guide: a backup sync paragraph (what it adopts and never adopts, its output and memory (the rows' read, measured), its exit codes, its refusals, the claim and what a killed sync costs, the signal, the SQLite serve lock, the partial report of other roots, the case with no way out but putting the prefix back), the identity paragraph (--adopt-store, the default-root exception and the default root every database file of a directory shares, the marker, the deletion fence (before the row change and before the deletion, what a late refusal leaves, a root with no file, the S3 gap, a stale check, the failed job's orphan prefix), the previous owner's window and the rule to stop every server of the catalog a file names before adopting its root), the --adopt-store paragraph (the same rule), the :memory: row (the 60 s bound and the listing it must fit, an abandoned run's filesystem walk that runs to its end while the next cycle starts another, /health silent until a filesystem startup sync ends and a signal then ending the process at once, the BackupNotFound window until the startup sync's log line and, for a prefix that met a store or catalog error, until a later cycle's line, the backup a server killed inside a DeleteBackup serves again, the partial prefix of a killed CreateBackup no command removes, and a new in-memory server started only after the old one has exited) and the in-memory refusal, the registrations list. The sync paragraph also gives the per-prefix memory term (about 350 bytes per prefix at one object per prefix), says the report lists every prefix the catalog already has a row for, that the objects a pruned row's prefix still holds stay until an operator deletes the prefix by hand (purge deletes only the prefixes of rows), and that on PostgreSQL the next maintenance command takes a killed command's claim over at once; the identity paragraph gives the fence's store-id rule. The upgrade manual's rollback paragraph (--adopt-store after the re-upgrade, the defaulted SQLite root's warning until then, the serve lock or the claim for sync), the differences ("Where backups live"), usage (cross-backend restore, where --adopt-store of the source's root moves the root to the target's catalog: stop the source's servers first) and limits (backup contents) rows, the developer test guide's matrix paragraph. Design: revision item 34 (the fence included), the PR 10 section's note of the split, the PR 9b paragraph and the doc-row table's migrate-legacy row; 4(c) (the previous owner acts until a later check that can read its catalog unless the fence stops it), the PR 10 section's host-local acceptance line (amended to the rule implemented), 4(e), 4(f), 4(g), decision 6, section 7's memory paragraph and the PR 10 contract edited with item 34. The comments that gave the Cassandra orphan backup_index_by_status sweep and the MongoDB orphan target-collection sweep to backup sync say no sweep removes them yet, and the MongoDB error line in remove_restore_target, which an operator reads, now says the collections it names can be dropped by hand. Cassandra's module doc says that a status entry under an earlier status stays until a sweep exists, and names the path that writes one by design (a DeleteBackup between insert_backup_row's transaction and its entry batch); the trait doc, the sync module doc and item 34 qualify "no two rows of one backup id" as under the manifest's account, table and id; sync's sentence for an insert that met a row it cannot re-read names an earlier release's in-catalog backup at the ARN; STARTUP_SYNC_TIMEOUT's doc states the abandoned filesystem walk; destroy's sentence says that on the default SQLite root the re-created catalog's server logs the warning and leaves the store unconfigured.

Why

Design 4(c), 4(f), 4(g) and the PR 10 section. The store is the source of truth and the row a cache; until this PR nothing rebuilt the cache, so a lost or re-created catalog lost its view of every store-backed backup, and sqlite-memory lost it at every restart. The split keeps this PR to the commands that touch no row of another store: plain sync and --prune-missing act only on the configured store's rows, --adopt-store rewrites one file and then syncs, and the startup sync is the same sync. --move-store, --forget-store and --forget-backup rewrite or delete rows of other stores, a different set of invariants with two more trait methods per backend (#820); migrate-legacy is a separate data path (#821).

Choices to weigh

• Caller::SyncCommand, the one exception to PR 7's defaulted-root leniency. On SQLite's defaulted root a catalog re-created by destroy and init finds the root's file naming the old catalog and no rows of its own, and PR 7 (item 29) makes that a warning that leaves the store unconfigured, so CreateBackup keeps working on a server that never configured backups. Without the exception backup sync --adopt-store would answer "no store" on exactly the root it exists for. The exception is scoped to the backup sync command and to a root another catalog owns; the server and purge keep the warning. The alternative is to require an explicit [backup.store] naming <dir>/backups for the adoption: that is a configured root, unhealthy owned by another catalog, which --adopt-store adopts, so it is a remedy too, at the cost of one configuration line the operator has to know to write. The default root is shared by every database file in its directory, so with either choice --adopt-store there takes the root from whichever catalog wrote its file; the refusal, the server's warning and the admin guide say so.
• The startup sync runs on every start of an in-memory catalog with a store, not only after an adoption. The catalog is new at every start, so there is no row to compare with and nothing to decide; on a fresh root the sync is one listing. A filesystem store syncs before the listener, an S3 store in the reconciler's cycle after its first healthy check, since an S3 check never holds the listener; every run is bounded at 60 s, and one past the bound, or one that met a store or catalog error on a prefix, runs again in the next cycle. Until a run ends, requests for the earlier backups answer BackupNotFound (with /health possibly already healthy); the :memory: row says so and names the log line that ends it. The adoption that precedes it is inside establish and only for a file carrying ephemeral_catalog: true; recheck never adopts, so two in-memory servers on one root (a non-goal) do not take it from each other every cycle, and a durable catalog's root stays unhealthy for an in-memory server.
• --prune-missing removes rows and deletes no object. The manifest probe sees only the newest AVAILABLE row's manifest, so a host-local root that holds the newest backup is healthy while it lacks older ones. A prune that moved those rows to DELETING or DELETED would let the other host's deletion duties delete their bytes. Removing the row outright, as 4(g) does for --forget-store, leaves nothing for a duty to act on; the row is gone for every instance, so the backup is BackupNotFound on every instance until a sync on the host that holds the bytes adopts it again; the design's host-local acceptance line is amended to that rule (no reconciler deletes a prefix because its own root lacks the bytes, and --prune-missing there removes the row and deletes nothing), and test_prune_missing_on_a_host_local_root_deletes_no_object pins it. The cost: a row pruned because its prefix was deleted by hand leaves that prefix's other objects for the operator to delete by hand, since purge deletes only the prefixes of rows and every later sync reports the prefix as not a complete backup. The alternative was to prune only ARNs the operator names; the option would then need an argument, which the design does not give it.
• The dead end that follows. When the newest AVAILABLE row's own manifest is gone, the probe marks the store unhealthy, so plain sync and --prune-missing both refuse, and the row cannot be pruned. The way out in this PR is to put the prefix (or its manifest) back; Backup/restore: backup sync --move-store, --forget-store, --forget-backup #820's --forget-backup will be the other. The admin guide states it; test_prune_missing_and_the_unhealthy_store_refusals pins both refusals and the recovery once the manifest is back. This predates the PR (the probe is PR 7's) and is the same condition the host-local case shows, so the PR does not change it.
• Plain sync takes the maintenance claim beside servers, with no wait. The insert-if-absent orders it against a CreateBackup on any instance (an_insert_that_meets_a_row_leaves_it_and_says_which), but not against purge, which removes a dead CREATING row before it deletes the prefix, and that prefix can hold a complete manifest (the job died after writing it); a sync between the two steps would adopt bytes purge then deletes. With the claim the two never overlap (test_sync_is_refused_while_purge_holds_the_claim), and 4(f) now names plain sync. The cost is that CreateBackup and RestoreTableFromBackup are refused while a sync runs, as during purge (a refused CreateBackup writes no row: the request checks the claim before any). A sync killed while it holds the claim keeps every instance's jobs refused until stale_job_after_secs passes (300 s by default); the admin guide says so. A prefix purge leaves after a failed deletion is a backup with no row: when it holds a complete manifest a later sync adopts it and the next purge deletes it; an incomplete one stays too, for the operator to delete by hand. On SQLite it runs under the serve lock, after the dead-owner resolutions, as the design's every-writing-command rule requires; that prelude is purge's (ReconcilerState::for_maintenance_holder), and that prelude now logs a line of its own ("run by a maintenance command before its own work").
• The rows are read once, before the listing; the listing streams. The design's "checks each prefix against the catalog as the listing reaches it" holds with one read of the root's rows up front (read_backup_row needs a full ARN, which sync has only after reading the manifest, and Cassandra keys by ARN); the read under the ARN's two spellings and the insert-if-absent then decide against rows that land later and rows of other roots, on Cassandra too. The listing is consumed as it streams with no store trait change, but the filesystem store builds its whole listing in memory first, so on a filesystem root the peak barely moved (98 MiB at 300,000 objects against 102 before). Sync keeps of each row only what it uses, but every backend's list_backup_rows collects its whole result first, so the read of the rows is the larger cost on a catalog with many rows of the root, tombstones included: about 22.5 KiB of peak RSS per row of a table with five GSIs (1,452 bytes of source_table_details_json), 610 MiB at 25,000 rows and 1,144 MiB at 50,000 against 47 MiB with none, measured on the SQLite debug build of this tree on a 16-core x86_64 Linux host with glibc's default arenas; backup list peaks the same, and with a one-object prefix per row sync now peaks at 1,145 MiB where it held the rows beside the listing at 1,181. Streaming the rows would change every backend's list_backup_rows; the design, the admin guide and section 7 state the cost instead.
• The deletion fence narrows the previous owner's window; it does not close it. After an adoption the previous owner's store stays healthy until a later check that can read its catalog finds it, at the earliest one reconciler interval (60 s by default) after its cycle in progress ends, and in that window its DeleteBackup, its reconciler and its failed jobs would delete bytes under the new owner's AVAILABLE rows, which would then leave the new owner's store unhealthy with no way out in this release. The fence re-reads the identity file right before each deletion (and before a CreateBackup's first write), so the window shrinks to the gap between one get and the call after it (on S3 a listing and a DeleteObjects under the SDK's retries, seconds when throttled), and the first refusal marks the store unhealthy for later requests (a check that read the file before the adoption and ends after the refusal records it healthy until the next cycle: deletions stay fenced, restores are accepted). DeleteBackup and purge run it before they change the row as well as before the deletion, so a DeleteBackup refused for a root another catalog owns leaves the row AVAILABLE and DescribeBackup never answers that refusal with DELETED; the cost is one small get per CreateBackup and two per deletion. A put already past the read still lands, which is why the texts state the precondition (stop every server of the previous catalog first) rather than promise the fence enforces it. A file the fence cannot read refuses the deletion (it is retried, as a failed delete_prefix is) and changes no health, so a transient error does not take the store down; at DeleteBackup's first read it is a store failure, so the request writes DELETING and answers DELETED and the second read refuses the deletion. A refusal at the second read (an adoption between the two) leaves the previous owner's row DELETING with nothing deleted, and DeleteBackup answers DELETED, as for any deletion that fails after its DELETING write; that catalog's reconciler acts on no row of its now unhealthy store, and finishes the deletion only if the catalog owns the root again. A root with no identity file also refuses the fenced call. A failed job of the previous owner leaves its partial prefix under the adopted root, which nothing of the new owner removes; the admin guide says so. The fence keeps the catalog id it read first: a catalog re-created beside a running process is then refused every deletion and CreateBackup until the process restarts, the safe direction, which the comment states.
• Rows of other roots are reported in part. Sync reports the prefixes whose backup id a row of another root holds. The rows of the configured root under another store id cannot reach it (such a store is unhealthy, which sync refuses), so the report has no section for them. A listing of every root's rows needs list_backup_rows_by_root, which Backup/restore: backup sync --move-store, --forget-store, --forget-backup #820 adds with the options that act on them.

Testing done

Re-measured on the tree rebased onto #740's 94cd324e, debug builds (the move to 9803ab7f, whose two conflicts were text in runtime.rs and the design, was checked with fmt, clippy and the engine, server and app suites: engine 466, server 310, app 49): cargo fmt --all -- --check and cargo +1.97.0 clippy --locked --workspace --all-targets --features extenddb-storage-cassandra/test-hooks -- -D warnings clean; engine --lib 464 passed (with and without backup-test-hooks: #740's 423 plus this PR's 42, less the dropped wait cell), server 310, app 49, core 801; PostgreSQL backup_restore 58; SQLite --lib 278 (1 ignored), its -- backup restore_job_tests 72, backup_job_panic and restore_job_resume 1 + 1; MongoDB live live_ 67 (--test-threads=1); Cassandra direct_integration (test-hooks, --exact, under the shared node's lock) the five insert and three forget cells named below, 8 passed; the wire suites as below (35 passed, 8 skipped on SQLite; 27 passed, 16 skipped on PostgreSQL). The mutant at the rebase's DeleteBackup change (its first fence never refusing) is killed by delete_backup_on_a_root_another_catalog_adopted_deletes_nothing. Everything else below is carried from the tree before the rebase, as each line says.

The counts below were measured on the final tree, and the binaries are debug builds of it, except where a line says the earlier tree (the first commit of this PR, before the changes to sync's claim, prune, partition and spelling rules, the streaming listing and the startup bound) or b58d879 (the tree before the last changes: the prune exit for a row in use, the fence's store-id rule, and the cells and texts listed above; they change no code of the core and storage crates, where the trait's change is a doc comment, so the core, PostgreSQL, SQLite and MongoDB counts below are b58d879's).

• cargo fmt --all -- --check clean; cargo +1.97.0 clippy --locked --workspace --all-targets --features extenddb-storage-cassandra/test-hooks -- -D warnings clean; cargo +1.88.0 check --locked --workspace --all-targets clean with no warning on b58d879.
• Engine --lib: 431 passed before the rebase (464 on it), among them the deletion fence at each call site, each on a store whose last check passed and a root another catalog adopted since (DeleteBackup refused before its DELETING write with no delete_prefix, the row left AVAILABLE, and a second request refused at the gate with no read; an adoption between DeleteBackup's two reads leaving the row DELETING with nothing deleted and DELETED answered; purge refused before it changes the row; the reconciler's DELETING, FAILED-prefix, DELETED-prefix and tombstone duties with the file rewritten right after the cycle's check read it, each acting on nothing; CreateBackup refused before any row or put; the job's failure path writing FAILED and deleting nothing; purge deleting nothing), each leaving the store unhealthy owned by another catalog; the fence's read failing alone (a MemStore failure of the identity key) at DeleteBackup, CreateBackup, the job's failure path, purge and each reconciler deletion, each refused with nothing deleted and the health left, no row moved but DeleteBackup's, which writes DELETING and answers DELETED; a stalled read refused at the bound and a root with no identity file refused (DeleteBackup answering DELETED with the row DELETING in both); a fence refusal in the DELETING duty stopping the dead-owner duty before it reads the foreign root; the startup sync through reconciler::cycle on a root listing that never answers, cut at 60 s with the DELETING row finished in the same cycle and the sync pending; the next sync completing its own torn insert (a Fake knob for Cassandra's rule) while a row under the other spelling still skips; a key ending in the manifest's name below a prefix not hiding a missing manifest; the marker cell (a durable --adopt-store of a marked file writes ephemeral_catalog: false, an in-memory establish on that file answers ForeignCatalog and writes nothing, and an in-memory recheck of a lost file writes it back marked); and the sync cells (every adopt and skip reason, another account, table or partition in the manifest, a row of another root under the canonical spelling, a prefix under another table with a row's backup id, a CreateBackup racing the insert, an insert that fails, a stop before a prefix, a listing failing part-way, a failed row read, only AVAILABLE rows reported missing, --prune-missing's outcomes with no object deleted and no tombstone, the adoption and its lost race, the in-memory adoption with the marker true and false, the startup sync through reconciler::cycle with a store error keeping it pending and with a recheck finding the root foreign, the abandoned startup sync, the dead-owner probe refusing a manifest of another region), and: four manifests whose sidecar matches and that the reader refuses (a source_table of another table, a creation time that is not the id's instant, a ../ data-file key, a size total that is not the files' sum), each skipped as "the manifest is refused"; a failed read of the row under the ARN's spellings (a Fake knob) skipping the prefix as a catalog error with no insert, and the next sync adopting it; a Fake mode with rows unique by ARN alone (Cassandra's rule) and two prefixes of one id under two tables, the second skipped on the row check; --prune-missing with a failed head (a MemStore knob) keeping the row as Failed, and needs_another_run true for Failed and InUse only; the startup sync with an incomplete prefix left done; a cycle with a pending startup sync on a stalled listing returning at the shutdown, well before the bound, with the sync pending; the fence refusal in the DELETING duty also leaving the prefixes of a stale FAILED row and a DELETED row unlisted (a MemStore list record); adopt(.., AdoptRule::EphemeralOwner) on a durable file refused with nothing written, and an in-memory establish on it reading the file once; the fence refusing a file of this catalog under another store id and recording IdMismatch; server --lib 305 before the rebase, 310 on it (backup_store with the SyncCommand cells, the exception's scope among them); app 49 (the adopting sync's last read, the transient-skip exit); core --lib 800 before the rebase, 801 on it, with the layout cells (every reserved component, the id grid is_backup_id accepts and refuses, traversal, separators, control and non-ASCII characters).
• On the live services: PostgreSQL backup_restore 51 passed (58 on the rebased tree); SQLite's backup_job_panic and restore_job_resume, which drive the engine's job and reconciler, pass; SQLite --lib 268 (measured before the deletion fence, as MongoDB's and Cassandra's below are, and 278 on the rebased tree; the fence changes no storage crate, and none of them depends on the engine); MongoDB live live_ 63 with --test-threads=1 (67 on the rebased tree); Cassandra direct_integration (--features test-hooks, under the shared node's lock, --exact) test_an_insert_over_an_in_catalog_backup_answers_false_and_writes_no_entry (a backups_by_arn row with no store kind at the ARN: false, no list or status entry) on the final tree, and the insert cells (with the completed torn insert, the unknown-outcome cells through a test-hooks seam on conditional_write, and the failed entry batch) and the start_backup_delete cell, which on each backend also covers forget_available_backup_row (removed and listed nowhere, the in-use and status refusals, a missing row; on Cassandra the three index entries read directly).
• Mutants, each a targeted edit, run, and the file restored (those of the earlier rules on the earlier tree): 6 in layout.rs; the four backends' inserts (the insert that overwrites, the one that ignores backup_id uniqueness, a wrong generation or stamp, on Cassandra the list rows and the status entry skipped or misplaced, the settlement ignoring the generation, the store or the status, any row read after a lost answer counted as inserted, the single attempt, NotMatched not completing a torn insert, the batch not re-sent); forget_available_backup_row without the in-use rule (PostgreSQL, SQLite), marking DELETING (MongoDB), leaving its index entries (Cassandra); the engine's sync, adoption and startup-sync rules (the account, table and partition checks, the spelling read, missing manifests of any status, the startup sync on an unhealthy store or with a transient skip marked done, an insert error, the stop, the prune deleting bytes, the other-table reason, the probe comparing only the backup id, and the earlier rules); the SyncCommand exception removed; the deletion fence removed at each of its six call sites (finish_backup_delete, the reconciler's prefix deletion and its tombstone removal, fail_row, CreateBackup, purge_row), the fence not recording ForeignCatalog, the fence reading the file on a store already unhealthy, adopt keeping the file's marker, and recheck rewriting a lost file without the marker; and at the binary level against the wire cells: finish_backup_delete's fence removed (the previous owner's DeleteBackup then succeeds), the row check before the manifest read removed, the startup sync before the listener removed, the in-memory CLI refusal removed, --adopt-store taking no claim, plain sync taking no claim beside servers, and the prune through start_backup_delete and finish_backup_delete. Each was caught at the assertion it targets. On the final tree, each against the whole crate suite: the spelling skip for either spelling; no fence before DeleteBackup's DELETING write, or before purge's row change; the dead-owner duty after a refusal; missing-manifest keys of any status; any key ending in the manifest's name; the startup sync's timeout removed; the fence's read error passing, its read unbounded, a root with no file passing; the adopting sync's last read accepting another owner and a transient skip exiting 0 (app cells); stop: None in the CLI's sync (the wire SIGTERM cell); PostgreSQL's shared body without FOR UPDATE, Cassandra's shared condition as IF EXISTS and without the in-use pre-check, and Cassandra's forget with the entry error returned, Unknown never or always counted as removed, and no lost-answer exhaustion check. All 21 were killed (a 22nd, the wait on a FAILED maintenance row, targeted code the rebase dropped). On the final tree, 16 more, each a targeted edit, run, and the file restored with its md5 checked: in the engine, against its whole --lib suite, the reader's validation replaced by a plain parse, a failed spelling read treated as no row, a failed prune head falling through to the removal, Incomplete counted as transient, the row check's insert after an adoption removed, InUse not counted for another run, the cycle's shutdown select replaced by awaiting the sync, the three health re-tests before the FAILED-prefix, DELETED-prefix and tombstone duties removed, the marker guard in establish and the one in adopt each removed, and the fence's store-id comparison removed; Cassandra's NotMatched arm with is_none_or (killed by the in-catalog backup cell, which the four earlier insert cells let pass); and at the binary level (PostgreSQL debug builds, the targeted wire cell) the adopting sync's last read removed, complete_run answering Ok, --adopt-store's wait gated on --prune-missing alone, and sync's catalog refusal turned into a plain error (exit 1 where 6 is expected). All 16 were killed at the assertion each targets.
• Wire, by name with the moto variables, on Python 3.10: tests/test_backup_store_config.py tests/test_backup_job.py on the SQLite build 35 passed, 8 skipped (the PostgreSQL-only cells; pytest -n 4), on the PostgreSQL build 27 passed, 16 skipped (the SQLite-only cells; pytest -n 2, since -n 4 exhausts the server's connection slots).
• Storage, for the shared delete and forget body and Cassandra's forget: PostgreSQL backup_restore's two row-lock cells (start_backup_delete_waits_... and the new forget_available_backup_row_waits_for_the_row_lock_and_then_sees_the_registration) and start_backup_delete_checks_...; SQLite --lib -- backup restore_job_tests 63 passed (72 on the rebased tree); MongoDB's two live_ cells of the shared body; Cassandra --exact under the lock, 6 passed, among them test_an_unknown_forget_outcome_is_settled_by_a_serial_read and test_a_forget_whose_entry_removal_fails_answers_removed (seams fail_forget_writes and fail_index_entry_removals) and test_a_forget_clears_a_finished_restores_registration_and_removes_the_row.
• Binary checks on b58d879 (SQLite): SIGTERM one second into a sync of 2,000 prefixes: exit 1, "stopped"; the in-memory refusal and the parse errors of file://example.com/... and file:x.sqlite?bogus=1; peak RSS 45 MiB empty, 66 MiB at 100,000 objects, 98 MiB at 300,000 (no rows); with rows of the root, 610 MiB at 25,000 and 1,144 MiB at 50,000 rows of a five-GSI table, 148 MiB at 50,000 rows whose source description is {}. On the earlier tree: SQLite (a writing sync beside a server exits 6; destroy and init, then plain sync refusing and --adopt-store adopting the root and its backup, which the restarted server describes, lists and restores; a prefix copied into another deployment's root adopted and restored there), the sqlite-memory shape on a filesystem and on an S3 store (moto: the reconciler's first cycle adopts the bucket's root and syncs; Describe and Restore of the previous start's ARN), and PostgreSQL beside its server (a SQLite deployment's backup copied in, adopted and restored; the volume unmounted refusing sync and --prune-missing with nothing written; --adopt-store of a SQLite root and of an in-memory catalog's root).
• devtools/backup-restore-matrix, one invocation per cell: sqlite→sqlite, sqlite→postgres, postgres→sqlite, postgres→postgres, the two --duplicate-key cells into SQLite and PostgreSQL, sqlite→postgres and postgres→sqlite --vector, and sqlite→postgres and postgres→sqlite --adopt-store: 10 of 10 PASSED (25 to 33 s each) on the earlier tree; on b58d879 sqlite→postgres, postgres→postgres, sqlite→postgres --adopt-store and postgres→sqlite PASSED again (28 to 33 s), the PostgreSQL targets' syncs now under the maintenance claim beside the server. On b58d879 sqlite→mongodb also PASSED, the target's sync taking the claim beside its server and adopting the backup. postgres→cassandra could not run on this host: the shared Cassandra node refuses CREATE ROLE ... PASSWORD (CassandraRoleManager doesn't support PASSWORD), which is the environment's, not the change's; it passed on an earlier tree of this PR. The nightly runs the MongoDB and Cassandra target cells on a schedule, not on this PR.

Not in this PR

• Backup/restore: backup sync --move-store, --forget-store, --forget-backup #820: backup sync --move-store, --forget-store and --forget-backup, with BackupCatalog::list_backup_rows_by_root and rewrite_store, and the full report of the rows of other roots, with their acceptance cells (--move-store's resolutions and refusals, --forget-backup's refusals, the store's return after --forget-store, a CreateBackup racing --move-store). restore.rs's OTHER_STORE reason and the reconciler's module doc now say what an operator can do without --move-store (configure the store the record names and restore again).
• Backup/restore: backup migrate-legacy #821: extenddb backup migrate-legacy <arn>|--all, with its acceptance cells.
• The Cassandra sweep of orphan backup_index_by_status entries and the MongoDB sweep of the data collections a failed restore-target removal leaves; the code says no sweep removes them yet.
• The acceptance line "a maintenance command started while a DELETING row and a RUNNING restore with a dead owner exist proceeds once the reconciler has finished the one and failed the other" is PR 9's wait (wait_for_quiet) and its engine tests; this PR's claimed options reuse it unchanged.

Breaking changes

None. The identity file's new member defaults to false, so #740's binary reads a file this release wrote and this release reads #740's. extenddb backup list, purge and remove-target now refuse an in-memory SQLite database (exit 1), where they opened a new, empty catalog.

DCO

Signed-off-by: Scott Robinson robinnsc@amazon.com