[robinnsc]

Backup/restore series (design: #396): #384 restore fidelity (merged) · #393 store · #394 format · #395 SnapshotEngine · #666 catalog migration framework · this PR: the catalog index · next: configuration and the store identity file (PR 7), the CreateBackup job and worker (PR 8), the restore job (PR 9a).
Closes #669 (part of #624).

Stacked on #666 (base branch feat/catalog-migrations, head d4187ea, which carries main through #395, #638, #412, #511 and #471): this PR adds migration id 5 to the MigrationId registry #666 introduces, and its PostgreSQL and SQLite bodies run through #666's runner (Kind::Breaking, convergent, one transaction). The diff shows this PR's single commit on top of #666. Where this head meets what main brought in: StorageEngine requires BackupCatalog + LegacyBackupReader + SnapshotEngine (#395); SQLite's backup_server_slot_count slots are shared by backups and #395's table snapshots, with the formula stated once on SqliteConfig::pool_size and one refusal text; the PostgreSQL control-plane pass keeps #395's drop_unless_held and loses only sweep_abandoned_restores; Cassandra's V006 runs through #638's keyspace-qualifying runner, and the backfill step and the version write name the keyspace in their own statements. When #666 merges, rebasing onto main changes nothing but the base.

What

The catalog index of design section 4 on every backend, the engine's Describe, List and Delete over it, the reconciler, and the legacy in-catalog paths kept beside it as a frozen reader. No request or response shape changes; the one visible API change is that ListBackups accepts a table ARN in TableName, as CreateBackup already does.

Traits (crates/storage/src/backup_catalog.rs). BackupEngine is replaced by BackupCatalog and LegacyBackupReader, both supertraits of StorageEngine, implemented by all four backends. BackupCatalog addresses every backup row by BackupKey (account id, backup id, ARN) and every restore target by RestoreTargetKey (account id, table name, table id); no method takes a bare id, and read_backup_row returns a row only when its account, table name and ARN equal the request's. Its methods: catalog_id, claim_backup (the row is inserted CREATING under a fresh owner_generation; a second claim is AlreadyExists), list_backup_rows (by store identity and status) and list_account_backup_rows (by account, table and time bounds), both ordered (created_at, backup_arn), read_backup_row, start_backup_delete (the in-use check and the AVAILABLE-to-DELETING write in one transaction under the row's lock; Deleting, InUse(table) or Status), set_backup_deleted, remove_backup_row (DELETED or FAILED only), backup_in_use, remove_restore_target (conditional on a CREATING restore target with this table id, so it and the ACTIVE flip exclude each other), list_dead_restore_jobs(stale_after) (RUNNING records whose heartbeat is older than the window on the database's clock, and FAILED records whose table is still a CREATING restore target, or on Cassandra a DELETING one with its registration, once the record is that old), fail_restore_job (RUNNING to FAILED, conditioned), check_maintenance_claim. JobClaim is a job's handle on its row: generation, heartbeat, set_status(from, to, fields) conditioned on the generation and from (Matched, NotMatched, or Unknown when the connection failed and the write may have landed), read. LegacyBackupReader is #384's create, describe, list, delete, restore-source and restore, frozen apart from the restore's registration, which writes the job fields; create and restore take the job's JobPermit and hold it for the job's life, and abandoned_creating_backups(stale_after) reports a CREATING row whose create died. BackupRow derives BackupDescription and BackupSummary from source_table_details_json (the manifest's source_table; the summary reads only the table id and ARN, so a later build's required member does not break a listing); BackupStatus reports DELETING as DELETED and hides DELETED and FAILED. The in-use text ("Backup is being used to restore table : "), the not-found signal and the LimitExceededException text (limit_exceeded_message(JobBound), naming the bound met: [backup] max_concurrent_jobs, the [storage.sqlite] pool_size snapshot-slot bound, or the PostgreSQL server's connection slots) are one implementation every backend and the engine bind, as is JOB_HEARTBEAT_INTERVAL (15 s), the period every backend's jobs heartbeat at. StorageError::AlreadyExists is new.

Config (crates/config/src/backup.rs). [backup] with data_file_target_bytes (64 MiB, minimum 32 MiB), heartbeat_interval_secs (15), stale_job_after_secs (300, at least three times the period the jobs heartbeat at, max(heartbeat_interval_secs, 15): 45 at the least), failed_row_retention_secs (7 days), reconcile_interval_secs (60), max_concurrent_jobs (4, at most 1024), spool_path; deny_unknown_fields, every key optional, validated at load. init writes the commented block; extenddb.sample.toml and the admin guide carry it. ServerComponentsOptions.max_concurrent_jobs lets the PostgreSQL factory raise pool_size and catalog_pool_size to 2 * max_concurrent_jobs + 10 with a warning naming both numbers.

Engine (crates/engine/src/backup.rs, backup/runtime.rs, backup/reconciler.rs). BackupRuntime holds the job slots (JobPermit::try_acquire; a request over the bound is LimitExceededException at once, no wait) and the configured store, which is None on this head, so every backup_index row is a row of another store: Delete and Restore of one return the ValidationException that names no root. Describe, List and Delete read the catalog index first and fall back to the reader; List merges list_account_backup_rows (the time range as bounds, is_listed statuses) with the reader's list under the existing paging, filtering and cursor exceptions. CreateBackup runs the design's step 1 order: the handler's ACTIVE probe, check_backup_arn_fits (the region first, refusing with a message naming server.region when it is outside [a-z0-9-]; then the store-prefix rules on the name, over 255 bytes or dot-only, before the ARN is formed so the limit is named whatever the length), the vector refusal, the permit, the deprecation warning, then LegacyBackupReader::create(permit); restore checks the target name through describe_table after the ARN (a table of the name TableAlreadyExistsException, a CREATING restore target TableInUseException naming its source backup), then the backup, then takes the permit last and passes it in; the backend's fenced name check under the registration stays. Both spellings of a backup ARN (as written and re-spelled in this region) parse into BackupArn where the store-prefix rules allow; a legacy backup of a name they refuse (dot-only, over 255 bytes) skips the catalog index and describes, restores and deletes through the legacy reader as on main. Delete over a row runs the in-use check before the other-store check, as a restore does, and builds the description before the DELETING write; a row whose recorded source table this build cannot read is left out of ListBackups (logged) rather than failing the listing; DeleteStart::Status(AVAILABLE) is an internal error. The reconciler runs a cycle at startup and every reconcile_interval_secs, spawned by serve beside the backend workers and drained with them: the maintenance-claim check, each dead legacy restore failed (fail_restore_job) and its target removed (remove_restore_target; a target the removal finds no longer CREATING is left to its owner and reported as such), the target of a FAILED record whose removal did not land removed once the record is stale, with no second FAILED write (the FAILED-target duty of design 4(e), on every backend; on Cassandra, where the removal is a claim to DELETING and then a drop, a target left DELETING with its registration is listed on the same rule and its removal resumed from the drop), a dead manifest restore logged once per record and left to PR 9a, each abandoned in-catalog CREATING backup deleted through the reader; it records the worker metrics (WorkerLastSuccess, WorkerCycleLatency, WorkerErrorCount under BackupReconciler). It replaces the four backends' sweeps and their slot semaphores.

PostgreSQL. Migration 5 (migrations/0005_backup_catalog_index.sql): backup_index (primary key (account_id, backup_id), backup_id unique, backup_arn COLLATE "C" so the listings' (created_at, backup_arn) order is byte order whatever the database's collation, account/table and store/status indexes), twelve columns on table_restores (status, kind, store_kind, store_root, store_id, owner_generation, heartbeat_at, spool_host, file_index, items_applied, failure_reason, completed_at) with the backfill of the registrations that exist (kind = legacy; DONE with completed_at = creation_date_time, or RUNNING with the epoch heartbeat and generation 0 where the table is CREATING, then NOT NULL on the four claim columns), backup_maintenance, and settings.catalog_id written once. The claim is pg_try_advisory_lock(hashtextextended('backup:' || backup_id, RESTORE_LOCK_SEED)) on a dedicated, non-pooled connection (ClaimSession) that the backup job's heartbeats and status writes travel through (the legacy restore heartbeats through it and writes DONE and FAILED on pooled connections, each conditioned on RUNNING and the generation), every write WHERE owner_generation = $gen; the reconciler's dead rule is the heartbeat alone, the lock being the claim's exclusion, as on the other backends. The legacy restore registers RUNNING, kind = legacy, owner_generation, heartbeat_at = now() in the transaction that creates the target, marks DONE in the transaction that flips it ACTIVE, and on failure marks FAILED (conditioned on RUNNING and the generation) before abort_restore, removing the target only when that matched. The per-account and per-server restore slots, sweep_abandoned_restores and the control-plane pass's call are removed; RESTORE_LOCK_SEED and abort_restore are pub and #[doc(hidden)] so the tests bind them. docs/schema/postgres.txt is regenerated from a fresh init.

SQLite. Migration 5 as the code body m0005_backup_catalog_index.rs (fingerprint 0005_backup_catalog_index/sqlite/1, golden fixture blessed): the same tables and indexes, column-checked ADD COLUMNs (nullable; ADD COLUMN on a populated table cannot add NOT NULL without a default, and the application writes every one), the backfill, the catalog id. One process serves a file (the serve lock), so the claim is an in-process slot (JobSlots, a set behind a mutex with a drop guard) and a restart is proof of death: expire_restore_job_heartbeats ages every RUNNING record to the epoch at startup so the reconciler's first cycle removes an interrupted restore, and abandoned_creating_backups reports a CREATING backups row whose slot no task holds, ignoring stale_after (there is no heartbeat column on backups). The reader's delete of such a row removes it whole, items and definition, with no DELETED tombstone (a backup that never finished was never visible as one). The restore's registration carries the job fields in the same write transaction as the CREATING target; the copy marks DONE with the ACTIVE flip, FAILED before removal, and heartbeats every JOB_HEARTBEAT_INTERVAL (15 s) under the engine write lock. WriteOutcome::Unknown is never produced. The two startup sweeps are removed; backup_server_slot_count stays until PR 8 folds it into max_concurrent_jobs: its slots are shared by backups and table snapshots (#395), and the legacy create tries a slot without waiting, as a snapshot open does, answering the one LimitExceededException text (limit_exceeded_message), naming the [storage.sqlite] pool_size snapshot-slot bound, for both. docs/schema/sqlite.txt regenerated.

MongoDB. The backup_index collection (top-level account_id/backup_id under a unique compound index, backup_id unique, account/table/created and store/status indexes), backup_maintenance, and a restore_job subdocument (status, kind, owner_generation, heartbeat_at, failure_reason, completed_at) on a restore target's tables document beside the provenance fields DescribeTable reads; the versioned step 002_backup_catalog_index, run by run_catalog_migrations after the initial step and recorded in schema_history, creates them, backfills the registrations (DONE or RUNNING with the epoch heartbeat as on PostgreSQL, restore_operation_id and restore_heartbeat_at removed), writes catalog_id and sets catalog_version to 0.0.5; the expected version is one constant, CATALOG_VERSION, where bootstrapper.rs held two copies of 0.0.2 and operations.rs a third. The claim is the row: an update_one upsert filtered on owner_generation: {$exists: false} with heartbeat_at: "$$NOW", so a second claim trips the unique index. The lock for DeleteBackup and a restore's registration is a session transaction that $incs a lock_seq field on the backup's document while it is AVAILABLE, so the two conflict and order wholly; this replaces the restore_in_progress/delete_in_progress claim markers and the sweep that cleared stale ones. Both in-use checks read at the transaction read concern (majority-committed data) while DescribeTable reads at local, so a RestoreTableFromBackup or DeleteBackup issued within the majority-commit lag after DescribeTable first shows the target ACTIVE can still answer BackupInUseException and succeeds when retried; the differences document's concurrent-restores row states it with the measured lag. The legacy DeleteBackup writes DELETING before the drop and DELETED after, and describe/list hide DELETING as they hide DELETED; a retry of DeleteBackup on a DELETING backup finishes the drop. The restore's hand-over to the GSI backfill worker and the record's DONE are one update_one conditioned on the generation. A restore request that outlives the 60 s registration bound answers InternalServerError while its task goes on; the target it leaves (CREATING, holding the backup in use and the name until the copy ends, found through DescribeTable) is stated in the differences document's restore row. WriteOutcome::Unknown is the driver's Io and ConnectionPoolCleared. serve refuses a catalog whose catalog_version is not CATALOG_VERSION with a message naming extenddb migrate --yes (require_served_catalog). The restore slots and sweep_stale_backup_claims are removed. backup_protocol_model.rs is rewritten for the new steps (registration, copy, hand-over, decide, drop, mark, stop, resume, the reconciler's removal).

Cassandra. V006__backup_catalog_index.cql, after which the runner writes 0.0.5 (once the backfill step and the file's schema_history row are in, so a run that fails in the step leaves 0.0.4 and migrate --yes runs the file again): backups_by_arn gains backup_id, store_kind, store_root, store_id, owner_generation, heartbeat_at, completed_at, failure_reason, source_table_details_json; backup_index_by_status (PRIMARY KEY ((store_root, backup_status), backup_arn), with account_id, store_kind, store_id so a listing can re-read the row and scope to the store); tables gains the twelve restore_* job columns; backup_maintenance (every statement an ALTER TABLE ... ADD IF NOT EXISTS or a CREATE TABLE IF NOT EXISTS, which #638's runner qualifies with the keyspace; the file does not write catalog_version); a Rust backfill step after the file (backfill_backup_catalog_index, LWT IF restore_status = null so a second run changes nothing) writes the job fields and the catalog id. An index row is a backups_by_arn row with store_kind set; the legacy reader treats such a row as not found and the catalog ignores rows without it. Every claim is a lightweight transaction IF owner_generation = ?; backups_by_arn is written first and the list and status entries follow, every reader re-reading the authoritative row. The legacy create holds a claim on its CREATING row, heartbeats it and polls the claim once per 64-item bucket, so a copy that lost its row writes at most one bucket after the loss; abandoned_creating_backups fences a stale CREATING row to DELETING under a fresh owner_generation before reporting it, and reports stale DELETING rows (scanned first, so a row it just fenced is reported once). A job stops after three heartbeat errors in a row (a beat that hangs is not counted; the per-bucket poll of the claim bounds the copy then), as after a heartbeat that does not apply, and touches neither its row nor its data again; one error does not stop it. WriteOutcome::Unknown is the driver's Io and Timeout and the server's WriteTimeout and WriteFailure. An orphan backup_index_by_status entry left by a crash between the row's LWT and the batch is skipped by every reader and costs a wasted read until the row is removed under its current status (remove_backup_row deletes the entry under the status it read); the sweep that removes orphans is backup sync's (PR 10). serve refuses a catalog whose version is not CATALOG_VERSION, as on MongoDB (CassandraEngine::require_served_catalog); require_migrated_catalog stays for an engine built without the factory. poll_abandoned_restores, sweep_abandoned_restores and the worker are removed. The crate gains a test-hooks feature (as MongoDB's), compiled only for the direct_integration binary, which CI runs with --features test-hooks: a per-account heartbeat period and heartbeat-error injection, and a pause between a delete's in-use read and its claim.

Versions. One change, one number on every backend: migration id 5 on PostgreSQL and SQLite; 0.0.5 written by Cassandra's migration runner after V006 and by MongoDB's step 002_backup_catalog_index. The design's PR 6 section and section 4 now state these numbers (revision log items 26 and 27).

Docs. The upgrade manual (migration 5 and its rollback: the pre-upgrade dump restored, the previous binary refusing the migrated catalog with exit 4, and the [backup] section removed from the configuration, which the previous binary refuses as an unknown field; a "Cassandra and MongoDB: catalog 0.0.5" section with the commands, the stop-first rule, the first reconciler pass, the serve-time refusal and the Cassandra resume sentence (a run failing in the step leaves 0.0.4 and migrate --yes runs V006 again); the 0.0.4 section's 15 s sweep line now attributed to the 0.1.x server), the admin guide (the [backup] table; the job bound in place of the per-account and half-pool restore slots; the pooler note), the deployment guide (the job bound in place of the restore slots; the connection formulas gain + max_concurrent_jobs), the high-level design and getting-started connection formulas, the limits document's concurrent-restores row, the architecture guide, the MongoDB design document's version check, the differences document (the ListBackups paging row, the DeleteBackup-during-restore row, the interrupted-restore row, the two-servers row, the RestoreTableFromBackup row's MongoDB registration bound, the concurrent-restores row's read-concern window and its restore-during-delete answer (BackupNotFoundException on every backend), the restore-order row (the name checked before the job permit), the ListBackups cursor row under an invalid region, the BackupArn and backup creation rows: CreateBackup refused under a server.region outside [a-z0-9-] and for a name no backup ARN can carry), the admin guide's region row, the limits document's BackupName/TableName row, the upgrade manual's server.region paragraph, CONTRIBUTING.md's migration example renumbered to 6, the sample config, docs/schema/postgres.txt regenerated for the collation, the backup/restore design's version literals and revision log items 26 and 27.

Why

#396 section 4 and the PR 6 section. The engine-owned backup path (PR 8) needs a row per backup it can list without the store, a claim that excludes a second owner and lets a successor detect a dead one, and one place every backend's rules meet; the legacy in-catalog paths of #384 stay reachable so backups taken before the series restore, but as a reader whose only unfrozen part is the restore's registration through the new job record. Replacing the four backends' sweeps and slot semaphores with one reconciler and one max_concurrent_jobs bound is what makes the next PRs backend-agnostic. The migration and version step land first, in their own PR, because every later PR writes the rows this one creates.

Decisions taken while integrating the per-backend work, each stated in the module doc of the backend it concerns: JobPermit travels as a trait-method argument (keeps RestoreRequest Clone); the MongoDB and Cassandra version is 0.0.5 so one number names one change; MongoDB and Cassandra get the serve-time version guard the SQL backends have through their ledgers; SQLite's abandoned_creating_backups ignores stale_after and its dead-create delete leaves no tombstone; MongoDB's lock replaces the claim markers and DELETING precedes the drop; the WriteOutcome::Unknown classes differ per driver; the 15 s heartbeat is one constant, extenddb_storage::backup_catalog::JOB_HEARTBEAT_INTERVAL, that every backend's heartbeat loop reads and the config floor is computed from, until PR 8 wires heartbeat_interval_secs; a Cassandra job stops after three heartbeat errors in a row (a beat that hangs is not counted; the per-bucket poll of the claim bounds the copy then), as after a heartbeat that does not apply, and touches neither its row nor its data again, one error not stopping it (design PR 6 section); the in-catalog create on Cassandra polls its claim once per 64-item bucket; Cassandra's orphan status entries wait for PR 10's sweep; the [backup] doc says why the stale floor is a floor on SQLite.

Testing done

• extenddb-storage: 196 (14 new; 197 beside a crate that enables its test-hooks feature, as cargo test --workspace does: the LimitExceededException texts naming their bound, BackupStatus, BackupKey::from_arn, TimeBounds, the description and summary derived from the manifest fixture crates/core/testdata/extenddb-manifest/v1.json, a summary needing only the source table id and ARN, pool_size_for_jobs, the serve-time version rule).
• extenddb-config: 14 (6 new: defaults, the stale floor as three times the period the jobs heartbeat at, the data-file floor, the max_concurrent_jobs upper bound, unknown keys refused).
• extenddb-engine: 208 (25 new: the target name checked before the backup and the permit, the region refusal naming server.region, a removal that finds the target moved left to its owner, Describe, List and Delete over the catalog with the legacy fallback, Delete and Restore of a row of another store, Delete of a configured store's row, the in-use check before the other-store check, an unreadable source table left out of the listing and stopping a delete before the write, the permit taken last for restore and create, the ARN-fits rule, a name the store prefix refuses resolved through the legacy reader, TableName as an ARN, the time range reaching the catalog as bounds, the reconciler's cycle, FAILED-target duty, once-per-record manifest log, worker metrics and worker with the Fake implementing both traits).
• extenddb-storage-postgres (PostgreSQL 16 at 127.0.0.1:55432, EXTENDDB_TEST_PG_REQUIRED=1): lib 69; tests/backup_restore 37 (28 kept with renames, abandoned_restore_is_removed_only_when_its_heartbeat_is_stale rewritten over the reconciler's methods, 2 slot tests removed, 9 new: a copy whose record the reconciler failed stopping at its next heartbeat; a claim excludes a second owner, fences its writes and holds the advisory lock while it lives; start_backup_delete's status, in-use and tombstone outcomes; start_backup_delete waiting for the row lock and then seeing the registration; listings scoped and ordered; migration 5's backfill plus the first reconciler pass; a legacy restore registers RUNNING and finishes DONE; a FAILED record whose target removal did not land listed once stale and removed; a target removed only while CREATING and only under its full key); doc_version_literals 3, migrations_fresh 5, migrations_released 12, migrations_golden 5, migrations_crash 9, migrations_guards 6, migrations_runs 5, migrations_owners 3, migrations_tag_blobs 1, vector_control_plane 42, gsi_online_build 12, gsi_stale_writer 1, key_collation 3, migrate_data_db_name 2, put_create_race 9, stream_records_page_order 1, stream_shard_ownership 6, stream_shard_retention 1, twi_conflict 8.
• extenddb-storage-sqlite: 225 (3 rewritten from the startup sweeps to the reconciler's methods and expire_restore_job_heartbeats; 11 new: the PostgreSQL catalog-index module ported, a dead CREATING backup reported and removed through delete, a restart aging an interrupted restore, a legacy restore's record RUNNING mid-copy then DONE, a failed restore marking FAILED before removal, a create refused at once when no snapshot slot is free, a FAILED record whose target removal did not land listed once stale, slots and generations; 1 ignored benchmark).
• extenddb-storage-mongodb (replica set at mongodb://127.0.0.1:27117/?directConnection=true, EXTENDDB_TEST_MONGODB_REQUIRED=1): lib 141 (with EXTENDDB_TEST_MONGODB_STANDALONE_URI naming a standalone mongod, as CI's workflow does; without it feat(storage): SnapshotEngine, a consistent table snapshot on every backend [backup/restore 4/4] #395's a_standalone_server_is_unsupported is the one failure, 140 passed); -- --ignored live_ 28 (22 backup, 6 stream): a heartbeat lost while $out runs leaving no orphan collection, the PostgreSQL catalog-index module ported, the migration step's backfill and the reconciler's first pass, a legacy restore's record, a delete and three restores ordered both ways over six rounds (accounted by the catalog), a FAILED record whose target removal did not land listed once stale, DELETING left by a failed or dropped delete and finished by a retry, a restore of another instance not dead while it heartbeats, a slow-clock process not dead, and the serve-time refusal of a 0.0.2 catalog; interpreter_selftests 18, pushdown_parity 20. The 7 protocol-model tests run by default (1.5 billion interleavings memoised, under a second).
• extenddb-storage-cassandra (Cassandra 4.1 at 127.0.0.1:9042, EXTENDDB_TEST_CASSANDRA=required, one binary at a time): lib 78 (4 new: the unknown-outcome classes from a synthetic error body, generations, listing order, the consecutive heartbeat-error rule); direct_integration 192 with --features test-hooks (12 new: a FAILED record whose target the removal's claim left DELETING listed once stale and its removal resumed from the drop, claim, start_backup_delete, scoped listings, V006's backfill and the first reconciler pass on an isolated catalog, a legacy restore's record, an abandoned in-catalog create reported once stale and finished (counting the report and showing the dead generation fenced out), a FAILED record whose target removal did not land listed once stale and removed, a V006 run failing in its step leaving 0.0.4 and resuming, the create's heartbeat observed mid-copy, three heartbeat errors stopping a create within a bucket and two not, a restore registering between a delete's in-use read and its claim refusing the delete on both delete paths; test_table_operations_work_on_a_catalog_without_v006 now also asserts serve's refusal of the 0.0.4 catalog and its acceptance after V006); ttl_integration 26 passed, 1 ignored, 1 pre-existing failure (below).
• tests/test_resource_tag_migration.py's released-catalog fixture strips migration 5's objects (the twelve table_restores columns and their index, backup_index, backup_maintenance, catalog_id) before it clears the ledger, since the check after migration 1 refuses a table_restores carrying the job columns; both cells pass on the SQLite build.
• Wire: tests/test_backup_restore_fidelity.py (37) and tests/test_table_operations.py (27) against a serve of this tree (extenddb init on scratch databases, provision-test-credentials, control_plane_delay_seconds 0.05): 64 passed on PostgreSQL with EXTENDDB_TEST_BACKEND=postgres EXTENDDB_CONFIG=<config> (the DELETING test reads the catalog), 63 and 1 skipped without them; on SQLite 63 and the DELETING test skipped without them (with them it fails on the host's sqlite3 3.7.17 module, which cannot read a partial index); on Cassandra (with index_propagation_delay_ms 0) 62 passed and 2 skipped (the vector test and the DELETING test); a server killed mid-CreateBackup and one killed mid-restore were each cleaned by the next server's first reconciler cycle after the stale window. The PostgreSQL and SQLite servers drained their workers at shutdown; the scratch databases and keyspaces were dropped.

cargo fmt --all -- --check
cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings
CARGO_TARGET_DIR=target/t188 cargo +1.88.0 check --locked --workspace --all-targets
cargo doc --no-deps --workspace                                          # no warning in a file this PR adds or at a line it changes
EXTENDDB_TEST_PG_REQUIRED=1 EXTENDDB_TEST_PG_CONNECTION_STRING=... cargo test --locked --workspace --exclude extenddb-storage-cassandra --exclude extenddb-storage-mongodb
EXTENDDB_TEST_MONGODB_URI=... EXTENDDB_TEST_MONGODB_STANDALONE_URI=... EXTENDDB_TEST_MONGODB_REQUIRED=1 cargo test --locked -p extenddb-storage-mongodb
EXTENDDB_TEST_MONGODB_URI=... EXTENDDB_TEST_MONGODB_REQUIRED=1 cargo test --locked -p extenddb-storage-mongodb --lib -- --ignored live_
EXTENDDB_TEST_CASSANDRA=required cargo test --locked -p extenddb-storage-cassandra --lib
EXTENDDB_TEST_CASSANDRA=required cargo test --locked -p extenddb-storage-cassandra --features test-hooks --test direct_integration
EXTENDDB_TEST_CASSANDRA=required cargo test --locked -p extenddb-storage-cassandra --test ttl_integration   # 26 passed, 1 failed (pre-existing), 1 ignored

Not changed here

• The backup store: [backup.store], the store identity file (extenddb-store.json), backup purge (PR 7). BackupRuntime.configured_store is None, so on this head Delete and Restore of a backup_index row return the other-store ValidationException, and claim_backup, set_backup_deleted and remove_backup_row have no production caller yet; each backend's tests drive them.
• The store-backed CreateBackup writer and worker, and the reconciler duties on backup rows (resolving a dead CREATING row, finishing DELETING, deleting prefixes, tombstone retention) (PR 8). heartbeat_interval_secs reaches no job yet: every in-catalog job heartbeats at JOB_HEARTBEAT_INTERVAL (15 s), which the stale floor is computed from.
• The manifest restore job (PR 9a): a dead kind = manifest record is logged and left; remove_restore_target deletes restore_progress rows only where that table exists.
• The maintenance claim's holder and expiry (PR 10): check_maintenance_claim reports any backup_maintenance row. Cassandra's orphan backup_index_by_status entries wait for backup sync.
• describe_continuous_backups and update_continuous_backups stay on LegacyBackupReader until PR 1 replaces them.
• The Cassandra ttl_integration::test_ttl_enable_rejects_asynchronous_gsi failure predates this PR: without this commit (checked on a clean checkout of bffa088) it fails the same assertion (the test expects "asynchronously propagated GSIs" in the refusal; metadata_engine::async_gsi_ttl_refusal says "asynchronously propagated global secondary index"). Neither file is touched here.
• PR 1's line in the design still names Cassandra V006 and 0.0.5 for its own migration; whichever of PR 1 and this PR lands second takes the next numbers.
• The design's PR 6 acceptance cell running the two DeleteBackup/RestoreTableFromBackup orderings over a backup_index row is deferred to PR 9a, which adds the first restore of such a row; this PR observes the row lock with a two-transaction test of start_backup_delete on PostgreSQL and a paused-delete test on Cassandra.
• The upgrade manual's migration 5 section states the rollback: the pre-upgrade dump restored (the previous binary refuses the migrated catalog, exit 4) and the [backup] section removed from the configuration, which the previous binary refuses as an unknown field; no store exists on this head, so the dump loses nothing.

Checklist

• I have read CONTRIBUTING.md
• All tests pass (cargo test --workspace), the pre-existing Cassandra TTL failure above excepted
• Code is formatted (cargo fmt --check)
• Clippy is clean (cargo clippy --workspace --all-targets -- -D warnings)
• I have added or updated tests for new functionality
• I have updated documentation if behavior changed
• Breaking changes are noted below (if any)
• If this changes the wire protocol, Storage trait, auth model, on-disk format, or public CLI surface, an RFC has been accepted or is linked below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: #396, sections 3 and 4 and the PR 6 section; ADR-0023 (catalog migrations, the Cassandra and MongoDB bootstrapper mechanisms kept). Storage trait: BackupEngine is replaced by BackupCatalog and LegacyBackupReader on the StorageEngine supertrait bound; StorageError gains AlreadyExists.

Breaking changes

• Catalog migration 5 on PostgreSQL and SQLite, breaking: stop every server of the previous release before extenddb migrate --yes; a server with pending migrations refuses to start. A restore in flight at the stop is failed and its target removed on the new server's first reconciler pass.
• MongoDB's catalog version moves from 0.0.2 to 0.0.5 and Cassandra's from 0.0.4 to 0.0.5, each by one bootstrapper step run by extenddb migrate --yes with pre-upgrade servers stopped; on both backends serve now refuses a catalog at another version (exit status 1, naming the command), where the previous release's Cassandra server started and refused only the backup operations and the MongoDB server started regardless.
• The per-backend restore slot semaphores are removed (PostgreSQL and MongoDB: 4 per account and half the pool per server with a 30 s wait); the bound is [backup] max_concurrent_jobs (default 4, at most 1024) per instance for backups and restores together, refused at once with LimitExceededException whose text names the bound met beside its number ([backup] max_concurrent_jobs, the [storage.sqlite] pool_size snapshot-slot bound, or the PostgreSQL server's connection slots when the job's claim connection is refused). SQLite's backup_server_slot_count stays until PR 8 folds it into max_concurrent_jobs, its slots shared by backups and table snapshots (feat(storage): SnapshotEngine, a consistent table snapshot on every backend [backup/restore 4/4] #395); a CreateBackup that finds no slot free is refused at once with the same text where it waited up to 30 s, so snapshots holding every slot refuse every CreateBackup with no backup running. On PostgreSQL pool_size and catalog_pool_size are raised to at least 2 * max_concurrent_jobs + 10 (18 at the defaults), logged.
• A new [backup] section, all keys optional with defaults; a config file with an unknown key under it is refused at load, and stale_job_after_secs under three times the heartbeat period (45) or max_concurrent_jobs over 1024 is refused at load.
• CreateBackup refuses a table whose name no backup ARN can carry: over 255 bytes (reachable with [limits] max_table_name_length raised) or made only of dots, with ValidationException "TableName cannot be backed up: a backup ARN cannot be formed for it: invalid backup ARN: ...", where the previous release backed such a table up.
• Under a server.region outside [a-z0-9-] CreateBackup is refused with a ValidationException naming server.region (no backup ARN can be formed), where the previous release created a backup that DescribeBackup, DeleteBackup and RestoreTableFromBackup then refused as not valid; backups taken before the upgrade under such a region answer as before. The startup warning says so.
• Out-of-tree storage backends must implement BackupCatalog and LegacyBackupReader in place of BackupEngine, and handle StorageError::AlreadyExists in an exhaustive match.
• ListBackups accepts a table ARN in TableName (it was a ValidationException).

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0 and I agree to the Developer Certificate of Origin (DCO). See CONTRIBUTING.md for details.