[robinnsc]

Backup/restore series (design: #396): #384 restore fidelity (merged) · #393 store · #394 format · #395 SnapshotEngine · #666 catalog migration framework (merged) · PR 6 the catalog index · this PR: configuration, the store identity file and backup purge · next: the CreateBackup job and worker (PR 8), the restore job (PR 9a).
Closes #670 (part of #624).

On main at 6ab74d57, which carries PR 6 (#671, merged as ff6f74eb); rebased onto it after #671 merged, with no conflicts (the rebase adds only main's #704 and #541 lines). The text below still says "PR 6" for #671's code: this PR adds nothing to the catalog schema (PR 6's migration 6 created backup_maintenance, the store_* columns and catalog_id); it binds them. The diff shows this PR's single commit on top of PR 6's. The rebase onto main conflicted only in the four SOFTWARE-LICENSE-NOTICES*.html files, regenerated with devtools/generate-software-license-notices --backend {postgres,mongodb,cassandra} and devtools/generate-dev-license-notices (all four --check pass). The earlier rebase onto PR 6's rebased head changed three things: MongoDB's and Cassandra's factories now register through #681's numbered runner, so the backup commands register there too, named after the command, as on PostgreSQL, and release the registration when they exit; MongoDB's startup key repair starts in main's build_components, after register_and_check, for a server only (start_key_repair), where it started in PR 6's open_data_engine, which #681 made moot; and the reconciler, which this PR gives the backup runtime, runs in #666's restartable worker set. It conflicted in seven files: serve.rs (the worker set), Cassandra's and MongoDB's factories, MongoDB's key repair doc and its live test, the MongoDB design's 5.10 and the design's revision log (item 29, after PR 6's items 27 and 28). Where this PR meets PR 6 and main: the reconciler's cycle carries PR 6's ReconcilerState (here with the shutdown token), worker metrics and FAILED-target duty, whose three-way Removal reports a target left to its owner (targets_left_to_owner, which backup purge's zero-window pass prints), beside this PR's store check and DELETING duty; the permit refusal is PR 6's limit_exceeded_message(JobBound) naming [backup] max_concurrent_jobs, while the maintenance refusal keeps its own text naming the command; CreateBackup's request checks run in PR 6's order (check_backup_arn_fits: the region alone, PR 6's review having moved the name rules to the store-backed writer), then the maintenance claim, then the permit, which PR 6's review made a bounded 30 s wait; RestoreTableFromBackup's run in one order too: PR 6's target-name check through TableEngine::describe_table (check_restore_target_name), the backup row, this PR's store-health refusal on a row of the configured store, the maintenance claim, the permit (the engine's restore doc and the design's 5.3 step 1 with revision item 29 state it); the SQLite snapshot-slot refusal names PR 6's [storage.sqlite] pool_size snapshot-slot bound, and the limits doc's bound-naming sentence carries this PR's exception for the maintenance claim; DeleteBackup over a row builds its description before the DELETING write (PR 6) and then deletes the objects (this PR); backup purge's claim heartbeats at PR 6's JOB_HEARTBEAT_INTERVAL; the SQLite beside_server path builds the engine through the one constructor, so the backup slots it shares with #395's snapshots and PR 6's synchronous permit are the server's; Cassandra's list_dead_restore_jobs lists PR 6's DELETING targets with their registration and reads the coordinator clock after its scans (this PR); the engine's test double carries PR 6's record_still_running and its TableEngine (tables) beside this PR's row and store knobs, and PR 6's other-store cell builds its runtime through this PR's ConfiguredStore; every Cassandra statement here names its keyspace, as #638 requires.

What

The [backup.store] configuration with its per-backend defaults and validation, the store identity file and the health component, put_if_absent on the store trait, the maintenance claim on every backend, DeleteBackup running against the configured store, and extenddb backup list and backup purge. No request or response shape changes; the API-visible changes are the refusals of a store-backed row's restore and delete while a configured store is unhealthy or its first check pending, the LimitExceededException of CreateBackup and RestoreTableFromBackup while a maintenance claim is held (its message says a backup maintenance command holds the claim, without naming one), and the /health body (reason is the problem's category alone; checking before an S3 store's first check). backup list and backup purge say why there is no store (the default root failed its check, the section absent, disabled, the root not yet created); a defaulted root with rows for it is reported and refused as a configured one; the refusal before the first check says the first check is pending. backup list and the dry run give up on a store that does not answer after the check's bound; serve and the commands exit within 5 s of their last work with a thread parked on a stalled mount; a FIFO at the identity key is reported as neither readable nor creatable; a root with edge whitespace is quoted in the banner and the startup error.

Store (crates/backup-store). BackupStore::put_if_absent(key, body) -> Option<ObjectMeta> creates the object only if none is under the key and returns None, leaving the existing object untouched, otherwise. S3 sends PutObject (or CompleteMultipartUpload for a body over one part) with If-None-Match: *; a 412 is "present", a 409 ConditionalRequestConflict is re-sent after a back-off of 100 ms doubling to 1.6 s, eight waits in all, and a conditional complete that finds the key present aborts its upload. The filesystem store writes and syncs its temporary as put does (created O_CREAT | O_EXCL) and links it to the key, which fails if anything is there, so the object appears complete or not at all; the temporary is unlinked on every path; the trait doc says a failed parent sync after link leaves the object. put's temporary write and ancestor sync are factored out and shared; put_impl's unreachable arm is gone. BackupStoreKind::Disabled and open_described (the store with its kind and root label: the canonical directory, or s3://<bucket>/<prefix> with ?endpoint=<url> for the endpoint the client uses, from the config or the SDK's environment and shared config, spelled one way: lowercase scheme and host, no default port, no / for an empty path, no user information, query or fragment) are new; validate keeps a transport failure's class (the bucket joins the error chain) so a caller can tell it from a configuration error. check_private_directory(dir, what) is exported for the spool. README and module docs updated; the README names the S3-compatible-server caveat (a server that ignores If-None-Match makes the create a plain put) and the hard-link requirement (CIFS and FUSE mounts that refuse link with EPERM cannot hold a root).

Config (crates/config/src/backup.rs). store: Option<BackupStoreConfig> under [backup] (the crate's type, nested under its own key so deny_unknown_fields holds on both tables). resolve_store(database_file) applies the per-backend rule: an absent section on a file-backed SQLite database is a filesystem store at backups/ beside the file, marked defaulted (created 0700 by the server; a failure of its open, validation or identity check is a warning that leaves the store unconfigured); on a server database or an in-memory one it is unconfigured; kind = "disabled" is unconfigured on purpose. The spool: required with an S3 store except beside a database file (backup-spool/); an explicit spool_path is refused inside or containing a filesystem root, and when it is or contains the database directory, compared on normalized and, where they exist, canonicalized paths; a filesystem root that is or contains the database directory is refused beside the spool rule; a placement refusal spells both paths as compared. A relative [backup.store.filesystem] path or spool_path is made absolute against the working directory at load (make_paths_absolute), as storage.path is; an empty path is compared with nothing in resolve_store and left for open to refuse. StorageConfig::database_file (default None; SQLite answers its file through the one database_file rule it already has) is how the server learns the backend's file. The init template and extenddb.sample.toml carry the commented block.

Storage (crates/storage/src/backup_catalog.rs). read_store_ids(store_kind, store_root): the distinct ids of a root's rows, every status. newest_available_backup_row(store): the last AVAILABLE row in the listing's order, for the manifest probe. list_running_restore_jobs(): every RUNNING restore record by status alone, whatever its heartbeat, for the purge's wait set. maintenance_claim(holder, stale_after) -> Box<dyn MaintenanceClaim> and check_maintenance_claim(stale_after): the claim is backup_maintenance's one record, born owned with a fresh generation, heartbeated and released through the handle; a holder whose heartbeat is older than the stale window is dead, its claim is not reported held and is taken over; a second live holder is AlreadyExists naming the first; on PostgreSQL the check tests the advisory lock first (a transaction-scoped shared try-lock in one autocommit statement, so a cancelled request leaves no lock on the pool), so a live session is held whatever its heartbeat says; on SQLite a server's startup under the serve lock removes the record outright, and a command under the serve lock takes over a record whatever its heartbeat says (the lock is the proof that its holder is dead), and a zero stale window makes the dead-owner listings and removals (list_backup_rows_older_than, remove_dead_creating_backup_row, list_dead_restore_jobs) select by status alone, so a heartbeat stamped ahead of the clock by a backward clock step is not a live owner under the lock; on PostgreSQL the take tries the exclusive lock up to six times 2 ms apart on its claim connection, so a job's one-statement shared lock does not refuse it with no holder to name. remove_dead_creating_backup_row(key, stale_after): a CREATING row whose heartbeat is older than the window is removed outright (the maintenance command's write). list_backup_rows_older_than(status, store, clock, age) with StaleClock::{Heartbeat, Completed}: the rows of a store with a status whose heartbeat_at, or completed_at (never older when unset), is older than the age, compared in the backend's own query on the database's clock (Cassandra: the coordinator's, read once before the scan); purge's test of a live owner, and the listing PR 8's reconciler duties read. ServerComponentsOptions::beside_server: a read beside a server takes no SQLite serve lock and runs no startup recovery (the catalog's status is still read, on a read-only connection). ServerComponentsOptions::startup_recovery (default true) is the recovery alone: every backup command sets it false, purge --confirm included, which still takes SQLite's serve lock, so on PostgreSQL the vector index rebuild, the control-plane recovery and the GSI queue workers, on MongoDB the repair of documents stored without their key (#473), on Cassandra the control-plane transitions and on SQLite the interrupted-restore, claim and index passes run for a server only, and no command exits with that work half done. ServerComponentsOptions::command names the command that opens the catalog (None is serve): the serving registration every caller takes carries it, kept when the registration is taken again, so migrate --yes names the command while it waits for it, and a refusal while a migration runs names the migration's holder without the server's advice to start the server. On PostgreSQL the name is the session's application_name (ServerRegistration::register_as); on MongoDB a command field of the extenddb_servers document (register_and_check_as), which the wait prints before the pid; on Cassandra, whose extenddb_servers table has no column for it (a column would be a catalog migration), the release cell after the release and as (register_server_as; 1.0.0 as extenddb backup purge), which the wait prints in place of extenddb serve. Per backend: PostgreSQL takes an advisory lock (backup_maintenance, the shared seed) on a dedicated connection as the proof of a live holder and upserts the row, and release deletes the row, unlocks the advisory lock (pg_advisory_unlock, since PgConnection::close returns before the backend drops the session's locks) and closes the session, so a check on another connection right after the release reads the claim free; SQLite takes the row under the engine write lock; MongoDB takes over a stale record with a conditional update and inserts a missing one with an upsert whose filter a live record never matches ($expr is not allowed in an upsert's filter, so the two are two writes), and an upsert that matched a document with no owner_generation (an operator's insert) is taken, not refused; Cassandra inserts IF NOT EXISTS and takes over a stale row IF owner_generation = <old>, its claim-row reads run at LOCAL_SERIAL, and an insert or a takeover whose answer was lost is settled by re-reading the generation; its list_dead_restore_jobs reads the coordinator clock after its scans. MaintenanceClaim is a new trait (generation, heartbeat, release).

Engine (crates/engine/src/backup/). BackupRuntime.configured_store is a ConfiguredStore: kind, root label, the store handle, the spool (path, spool_host = <hostname>:<path>), the id once a check has read it, and the health as the last check left it: Checking until an S3 store's first check, Healthy, or Unhealthy; a transition between categories is logged once; a check that cannot read the catalog changes nothing. check_configured_store runs validate (until it has passed) then establish or recheck, with the store's calls under the store's bound (5 s for a filesystem root, 30 s for a bucket, under the SDK's per-attempt timeout) through one adapter (BoundedCalls, ConfiguredStore::calls) and the catalog's reads beside them, not under the bound, so a slow catalog never marks the store unhealthy; a call that does not finish is recorded as unreachable naming the bound; a filesystem root's calls run as tasks of their own, so an abandoned call stays parked on one thread until the mount answers and no call starts while it is (recorded unreachable again), and they run one at a time (ConfiguredStore::call_slot): a call that finds another running waits for it under its own bound, so calls started before the first reached its bound (two DeleteBackups together, a DeleteBackup beside a check) park no thread of their own and answer at the bound with the same transport failure; an S3 store's abandoned request is dropped with the call; a store whose startup validate failed is built unvalidated, so the next check runs validate again rather than reporting the store healthy on a re-read of the identity file; read_identity_bounded is the command's identity read under the same bound; the reconciler's cycle, which selects it against the shutdown token, and the backup command call it. A transport problem's text is the store error's whole cause chain; every other store error of a check is failed validation. store_identity runs the checks: establish reads extenddb-store.json, creates it with put_if_absent when the root has no file and the catalog no rows for the root, re-reads it, and compares it with catalog_id and the rows' ids; recheck re-reads and compares, rewriting a lost file with the known id when no rows exist; both end with the manifest probe (head of the newest AVAILABLE row's manifest, read as one row through BackupCatalog::newest_available_backup_row). The identity file is read up to 64 KiB. The problems: another catalog's file, an id the rows do not carry, more than one id on the rows, a missing file with rows, a create race lost to a store that ignored the condition (the re-read holds another id), a missing manifest, the store unreachable, a failed validation, the file unreadable, the catalog unreadable; each has an operator line naming the ids and a /health reason that is the category alone. require_healthy_store is the ValidationException naming the kind and the category, or, before the first check, that the first check is pending. CreateBackup takes the in-catalog path whatever the store's health (the gate behind STORE_BACKED_CREATE is PR 8's) and logs that the store is configured; the in-catalog create and the restore refuse with LimitExceededException while a maintenance claim is held, after every other check and before the permit (refuse_while_maintenance_held; the check runs before the job's row, which the backends commit and detach from), the message saying that a backup maintenance command holds the claim (every maintenance command holds the one claim, so it names none). HeldClaim heartbeats a maintenance claim on a timer for its life (Drop stops the timer; release waits for a heartbeat in flight). DeleteBackup of a configured store's row runs finish_backup_delete after start_backup_delete: the manifest, the prefix, a listing that must be empty, set_backup_deleted, each store call through the store's bounded calls (ConfiguredStore::calls: the check's bound and in-flight rule); the request answers the description with BackupStatus DELETED once the DELETING write has committed, whatever the deletion does (every reader reports the backup deleted from that write on, and the service has no counterpart for a failed object deletion), and a failure leaves the row DELETING, logged, for the reconciler. The reconciler's cycle runs the store's check each cycle, records the outcome, and while healthy finishes the configured store's DELETING rows (through the store's bounded calls); while unhealthy it touches no row of the store; while a maintenance claim is held it runs only the duties the command waits on or finishes itself (the branch the later duties go in is empty and says so); check_maintenance_claim takes the stale window. The defaulted SQLite root keeps its rule with the store (ConfiguredStore::defaulted): a root whose startup checks did not finish starts checking, and the cycle applies the rows rule to the first check's outcome, setting a failing root with no rows aside (set_aside: the runtime then answers as a server with no store) as the startup check would have. A caller that holds the claim says so through its state (ReconcilerState::for_maintenance_holder) and the cycle takes the held branch without reading the claim; a cycle run with a zero dead-owner window would otherwise read its caller's own claim through that window, and see it held only within one millisecond of its heartbeat. wait_for_quiet tells a live owner from a dead one through list_backup_rows_older_than on the heartbeat clock, never with this host's clock. maintenance also carries wait_for_quiet (WaitOutcome; RUNNING restores read by status through list_running_restore_jobs, with no age, so a record stamped in the listing's instant holds the wait), purge_row, purge_rows (the deletion loop, which stops before its next row at a lost claim or a signal, PurgeOutcome), Interrupt, and HeldClaim::release bounded by RELEASE_TIMEOUT (10 s): the purge's steps against the catalog trait, tested against the engine's double.

Server (crates/server/src/backup_store.rs, lib.rs). Startup resolves and opens the store, takes the spool (0700, .lock 0600 under flock; another live holder is a startup error naming the path; an existing spool must be owned by the server's user with mode 0700, parents included), and for a filesystem store runs validate and establish under a 5 s bound (LOCAL_CHECK_TIMEOUT, in the engine beside S3_CHECK_TIMEOUT, 30 s; past it, a warning and the store checking, a defaulted root keeping its rule so the reconciler applies it to the first check's outcome; a root that does not answer before open holds startup, as the database directory would); an S3 store's validate and establish run in the reconciler's first cycle, after the listener is up, so a stalled endpoint never holds the data plane (the store is checking until then; a validate refusal there makes it unhealthy). A configuration error that needs no request (open refusing the configuration, the spool rule, an S3 store without a spool on a backend with no default) stops startup with the path or bucket named; the defaulted SQLite root turns a validation or identity-check failure into a warning that leaves the store unconfigured while the catalog has no rows for it (read_store_ids); with rows for it the root is handled as a configured one, and configure_described carries the reason for an absent store out (NotConfigured) for backup list and backup purge to print. A filesystem root's I/O failure at validate's probe write degrades the store rather than stopping startup; the owner-only and existence refusals stay startup errors. A transport, identity or probe failure, or a check that does not finish within its bound, leaves the server running with the store unhealthy and logs it. /health is {"status": "healthy" | "degraded", "components": {...}}, HTTP 200 either way, with backup_store (status healthy, checking or unhealthy, kind, and when unhealthy reason, the category) when a store is configured; the console banner names the store beside the storage line, a filesystem root in the spelling the log line and the rows use (the canonical directory) with the configured spelling beside it only where the two differ (a root reached through a symlink or written relative), and the log names its id and the spool. catalog_refusal builds the message and exit code serve gives for a catalog the backend refuses, for serve and the backup commands alike.

App (crates/app/src/cmd_backup.rs). extenddb backup list [--failed] prints the catalog's rows for the configured store's root, every store id they carry, with each row's store_root and store_id; it is a read beside a server (no SQLite serve lock, no startup recovery), reads the identity file without creating one, under the store's check bound (5 s for a root, 30 s for a bucket), runs on an unhealthy store naming the problem in its header, and prints (none) with exit 0 when the defaulted root does not exist yet. extenddb backup purge --confirm opens the catalog as serve does (on SQLite under the serve lock) without the startup recovery a server runs, refuses an unhealthy store, prints the plan, takes the maintenance claim and heartbeats it for its whole run (HeldClaim at JOB_HEARTBEAT_INTERVAL; a lost claim stops the deletion loop before its next row), on SQLite runs the reconciler's duties with a zero window before the wait, as the claim's holder (a legacy restore the stopped server left RUNNING is failed and its target removed (a manifest one, PR 9a's, is left for its resumer and waited on), the CREATING target of a FAILED restore removed, an in-catalog CREATING backup removed, a DELETING row finished, a target that is no longer the restore target its record names printed as left), waits up to 2 * stale_job_after_secs until no CREATING or FAILED row has a heartbeat within the window (on the catalog's clock, through list_backup_rows_older_than) and no restore job is RUNNING, refuses and releases if they have not gone (and releases on a catalog read failure), then deletes each row: a CREATING row with a dead owner through remove_dead_creating_backup_row first and its prefix after, every other row's prefix (manifest first) and then the row outright through the status machine and remove_backup_row, leaving extenddb-store.json, exports/ and prefixes with no row; each deletion is logged under extenddb::audit. A row a DeleteBackup moves to DELETING or DELETED, or removes, between the command's read and its turn is finished and counted as deleted; on SQLite the owner window is zero (under the serve lock every owner is dead), so a CREATING or FAILED row with a young heartbeat is deleted, not waited on, and a server starting on the file clears a maintenance claim a killed purge left, as a purge run right after the killed one takes it over; SIGINT or SIGTERM releases the claim at once during the wait and before the next row during the deletion, and the wait's failures (jobs still running, the claim lost, the catalog unreadable, a signal) are each reported as themselves. Without --confirm it is a read like list: it prints the plan, changes nothing and exits 0, with no store configured as with one (an empty plan naming the reason), and on an unhealthy store says that --confirm refuses until the store's check passes. The signal listener is up before the open in every subcommand, so a signal during the open or the identity read ends the command once the open returns, with its registration released. --config is accepted before or after the subcommand. On PostgreSQL, MongoDB and Cassandra list, the dry run and purge --confirm each hold a serving registration for their run named extenddb backup list or extenddb backup purge, so migrate --yes waits for them and names them, and release it when they exit, whatever the outcome (under a 10 s bound), where a registration left to the runtime hooks the command drops is deleted only by a task the command's exit may never run, so migrate --yes waited out its 45 s; a catalog the backend refuses ends the command with serve's message and exit code (6 while a migration runs, naming its sessions; 3 for a catalog behind the binary, with the migrate --yes advice), where it exited 1 with a message telling the operator to start a server. On SQLite purge --confirm beside a running server is refused on the serve lock with exit 6, as a second serve is, where it exited 1. Every command and serve end their tokio runtime with a 5 s bounded shutdown, so a thread parked on a mount does not hold the process open past its drain. extenddb destroy's help and output say the store is left untouched and that a re-created catalog needs the old root's extenddb-store.json removed or a new root.

Docs. docs/differences-from-dynamodb.md gains two rows: the maintenance claim's LimitExceededException from CreateBackup and RestoreTableFromBackup, and the unhealthy store's ValidationException from DeleteBackup and RestoreTableFromBackup, both HTTP 400, with DeleteBackup's DELETED answer on a failed object deletion. Admin guide ([backup.store] section: the rules, the identity file, the health component (present whenever the server has a store, the SQLite default root included), the check's bound (store calls only; the catalog's reads beside it; a root's calls one at a time), DeleteBackup's answer, the commands running no startup recovery, backup list/purge (their serving registration on PostgreSQL, MongoDB and Cassandra and the refusals' exit codes, on SQLite the serve lock's exit 6), the shared-root rule, the re-init paragraph; /health examples and the reason list with catalog unreadable; the spool_path row), the limits doc's backup store row, the upgrade manual (configure a store before the release that stops the in-catalog create; disabled as the opt-out; the SQLite default root rule and what the warning means; the re-init sentence; the rollback paragraph's store note), the deployment guide's multi-instance notes (/health degraded, one shared root, one spool per instance, the hard-link requirement, two host-local roots on an empty catalog), the sample config and init template ("will live, from a later release"), the store crate's README, the four backends' and the engine's module docs, design revision items 29 and 30 (PR 6's review added 28, so this PR's two items moved up by one). The four licence notice files are regenerated: the server binaries now link the store crate's AWS SDK tree (Apache-2.0, MIT, ISC), which cargo about adds.

Why

#396 sections 4(c), 4(f), 7, 8, 9 and the PR 7 section. The store-backed CreateBackup (PR 8) and the restore job (PR 9a) need a configured, validated store whose identity the catalog's rows carry, a health state that refuses work when the root is not the one the rows describe, and a maintenance claim that excludes jobs on every instance while an operator deletes, which this PR already applies to the in-catalog create and restore, before their row. Landing them first, with the identity file and the health component, lets PR 8 write rows under an id that is already checked at startup and each cycle.

Decisions taken while implementing, each stated in the module doc of the code it concerns: maintenance_claim takes the holder's label and the stale window and returns a MaintenanceClaim (heartbeat and release) rather than a JobClaim, and check_maintenance_claim takes the stale window (the design's signatures had no duration to measure a dead holder by, and no release); a configured store whose startup check failed keeps its id unknown, and a row of its kind and root is a row of this store for the refusals until a check reads the id; CreateBackup takes the in-catalog path whatever the store's health (no store-backed writer exists on this head); an S3 store's checks run after the listener so an endpoint cannot hold startup; purge's "live owner" for a CREATING or FAILED row is the backend's list_backup_rows_older_than on the database's clock (an earlier version of this PR compared heartbeat_at with this host's clock and said that could only lengthen the wait, which is false: a host running ahead of the database shortens it; the method is the listing PR 8's reconciler needs, brought into this PR so the wait is right on its own); the purge's own reconciler cycle is told that its caller holds the claim through a ReconcilerState constructor rather than a second duration, so every other caller keeps the one-window call; purge waits on FAILED rows with a live heartbeat and deletes CREATING rows with a dead owner, the window being zero on SQLite; the restore part of its wait set is the RUNNING records alone: the FAILED records the zero-window listing also returns are the reconciler's duty; purge does not wait on DELETING rows but finishes them itself (nothing heartbeats one, the sequence is idempotent, and on SQLite no reconciler runs while the command holds the file); the jobs' claim check runs before the row on this head; the MongoDB claim is two writes.

Testing done

The first CI run of the second-round head failed run-integration-sqlite-s3 on #740 at the end of tests/test_backup_store_config.py's purge cell: the server started after the killed purges runs its reconciler, which deletes the objects of the stale FAILED row the cell left as its lever (the FAILED-prefix duty, by design on any head where the duty runs in the first cycle), and the cell's own cleanup then failed on the missing prefix. The cleanup now tolerates the prefix being gone; nothing else in the cell changes.

The first CI run of the rebased head failed run-integration and run-integration-sqlite in the tests/test_backup_store_config.py cells of the CLI lifecycle step: the SQLite deployment runs init with cwd set to its own directory (the relative-root cell needs the shell's directory), and CI names the binary as target/release/extenddb, which subprocess then resolved against that directory (FileNotFoundError). The suite now takes EXTENDDB_BINARY as an absolute path (os.path.abspath), which #700's tests/test_backup_job.py inherits; the cells skip as before where the binary is not a SQLite build.

Fourth round (Lee's third review; one inline Minor and the mark race from the review body): BoundedCalls::run starts a call's bound once the call holds the root's slot, not over the slot wait and the call together, so two 3 s DeleteBackups on one root both finish and the root stays healthy where the earlier bound abandoned the second. The in-flight mark is now set under the task's own running lock and only while the task still runs; the task's ClearOnDrop takes the same lock, so a task that ends between the timeout's last poll and the mark leaves no mark set (the race Lee read at runtime.rs:692). review_round_2::a_calls_bound_starts_when_it_holds_the_slot runs two deletions whose store call each takes most of the bound (MemStore::set_delay) through one slot and reads both rows DELETED and the root healthy; with the bound over the wait restored it fails ("both deletions finished", the second row left DELETING). Design section 7 and item 30 and the admin guide's bound sentence say where the bound starts. Engine backup:: 153 passed on this head, clippy 1.97 clean.

The third round (Lee's second review) changed the engine's BoundedCalls, its test, the three catalog-lock scripts and the SQLite job's moto step; extenddb-engine's 303 ran on this head (all passed), cargo check --workspace --tests and cargo +1.97.0 clippy -p extenddb-engine --all-targets -- -D warnings are clean, and the MongoDB catalog-lock script ran as described below. The moto step's new body passes shellcheck; this host has no Python that can build a virtualenv, so the step itself is read, not run.

Counts marked re-measured are of this head (cargo test ... -- --list | grep -c ': test'); the others are carried from the previous head, whose crates this round did not touch. "New" is against PR 6's head.

• extenddb-backup-store: lib 37 (4 new: the conditional outcomes from synthetic replies, the root label in one spelling from four spellings of an endpoint, a validation transport failure keeping its class, disabled); conformance 123 (7 new: put_if_absent creates the object and answers None for a key that exists on both stores, the second call on the large key multipart so the aborted upload on S3 is exercised (on the filesystem store the existence check before the link already answers a second sequential call, so this test does not reach the conditional step there); eight concurrent callers on one key leave the winner's body, three times over, both stores, the test that catches a create that is last-writer-wins (with rename in place of link the filesystem store fails it); the filesystem refusals; an empty root path refused by name; a scripted HTTP/1.1 server answering 409 twice then 200, 409 then 412, and 409 forever, with the back-off measured), against moto at 127.0.0.1:9200; the SDK-environment cells also assert store_root() names the env-, profile- and services-sourced endpoint.
• extenddb-config: 23 (re-measured; 9 new, 1 rewritten: the section parses as the crate's shape and refuses unknown keys at every level, the per-backend default, the spool required with S3 except beside a database file, every cell of the spool placement rule, path comparison through a symlink, relative paths made absolute with the empty-path cell, an empty root compared with nothing and left for open, a root that is or contains the database directory refused and a child, a sibling and the default accepted, a placement refusal spelling both paths as compared).
• extenddb-engine: 303 (re-measured; 46 new: the identity file's shape and refusals, the public reason naming no root, compare's rules; an empty root creates the file and a second startup reads it; two instances on one empty root end with one id; another catalog's file; ids that disagree, a missing file with rows, rows with two ids; the manifest probe on the newest AVAILABLE row; a create race lost on a store ignoring the condition; an unreachable store unhealthy and recovering through the reconciler, with the id settled on recovery and both links of the transport cause chain in the reason; a stalled store's check abandoned at its bound (5 s for a filesystem root, 30 s for a bucket) with the dead-owner duties run and the store unreachable, the next cycle on a stalled root starting no check and waiting no bound while the abandoned call is parked, and on a stalled bucket each cycle sending a fresh request; a command's identity read returning at the same bound; a FIFO at the identity key neither readable nor creatable; a shutdown during the check ending the cycle without waiting the bound; a transition logged across categories only; a recheck rewriting a lost file; a store checked after startup checking until its first check, its refusal saying the first check is pending; the health reason the category only; an oversized identity file unreadable; a catalog read failure leaving the health unchanged; the refusals naming the kind on delete and the restore check with no slot taken, and the in-catalog create not refused; a healthy store leaving create on the in-catalog path; jobs refused while a maintenance claim is held, the window passed through; the heartbeat timer keeping a claim held through a long run and a dropped handle stopping it, on tokio's paused clock; delete of a configured store's row deleting manifest, prefix, then marking DELETED; a failed deletion left DELETING and finished by the reconciler once the store answers; another store's DELETING row untouched; a prefix that still lists an object left DELETING; purge finishing a row another deleter moved to DELETING or DELETED, or removed, between its read and its turn, and leaving a FAILED one; a young CREATING row left under a 300 s window and removed under the zero one; the wait's five outcomes (quiet on a FAILED record and a dead FAILED row, busy naming a live CREATING and FAILED row and a RUNNING restore), a signal ending its poll sleep at once; a DELETING row finished while the claim is held; the purge's own cycle, run with a zero window by the claim's holder, taking the held branch without reading the claim, where a default state reads the same claim through the window as free; the wait reading a live owner on the catalog's clock, with that clock 400 s behind this host's (a heartbeat this host reads as past the window is live) and 400 s ahead (one stamped with this host's now is dead)). The engine's MemStore double implements BackupStore in memory with a failure switch, a stall switch, a validate-failure switch, a late-put-after-delete switch and a post-conditional-put overwrite hook; its catalog double implements the purge's row writes, newest_available_backup_row and list_running_restore_jobs. The second review round's eleven: a store whose startup validate failed rechecked by validate and unhealthy until the probe passes (healthy after one cycle with validated true); purge_row's manifest-first order, the listing after the deletion leaving a row over a late put, and DeleteStart::InUse stopping it; a foreign identity file with a store that answers deletes leaving the DELETING row untouched (deleted with the is_healthy filter gone); the maintenance refusal after check_backup_arn_fits on the create and after validate_override_for_source on the restore, the claim not read for a request refused on its own terms; recheck on rows arriving with another id, the file deleted while rows exist (not rewritten), the file replaced by another catalog's; the wait holding on a RUNNING restore with a fresh heartbeat (quiet with the age-based listing in place); the deletion loop stopping before its next row at a lost claim and at a signal, and skipping a row left for a reason; a defaulted root set aside after its first failed check with no rows, unhealthy with rows, never set aside when not defaulted; DeleteBackup and the reconciler's deletion returning at the store's bound on a stalled store with the row left DELETING, two DeleteBackups together parking one thread between them (the store counts one stalled call; with the slot widened to two it counts two and the test fails, verified); the manifest probe reading one row; and HeldClaim::release ending at its bound on a catalog that does not answer. Each mutant Lee named was applied, the test run and seen to fail, and the mutant reverted (fourteen in the engine).
• extenddb-server: 303 (re-measured; 10 new: the spool lock refusing a second instance, narrowing a permissive lock file, and an existing spool refused unless owner-only, a permissive parent included; a defaulted root with rows for it configured and unhealthy rather than unconfigured, and a foreign identity file a warning without rows and owned by another catalog with them; a defaulted root whose open refuses a warning without rows and a startup error naming the rows with them; the startup checks returning at their 5 s bound on a stalled validate and an I/O failure of the probe degrading where a permission refusal stops startup; NotConfigured reading as the commands print it; an unreadable catalog keeping a failing defaulted root configured; a root with edge whitespace quoted in the banner; the banner showing a filesystem root in its canonical spelling, with the configured spelling beside it for a root reached through a symlink and for one not created yet under a symlinked parent).
• extenddb-app: 46 (re-measured; 2 new: the tokio runtime, multi-thread and current-thread, ending within its 5 s shutdown bound with a blocking thread parked forever; a command's registration released, and a release that does not answer abandoned at its bound).
• extenddb-storage-postgres (127.0.0.1:55432): lib 75; backup_restore 50 (re-measured; 13 new: read_store_ids; the maintenance claim excluding a second holder, released, held by a dead session's heartbeat until it ages, then taken over, kept by a live session whatever its heartbeat, reported Lost to a holder whose generation moved with its row left and still reported held through the heartbeat arm, and leaving nothing but the holder's lock in pg_locks after a check; a dead CREATING row removed and a live or moved one kept; two PostgresEngines on one catalog with two FilesystemStores: one empty root gives one id and both read it back; host-local roots under one label leave the second unhealthy, missing file first and then, with the file copied, the manifest probe; a root owned by another catalog; the purge wait quiet on a FAILED record whose target is CREATING with no duty run, busy naming a FAILED row with a live heartbeat and quiet once it has aged; the factory with beside_server leaving a gsi_pending row for the server and applying it as a server; list_backup_rows_older_than on the heartbeat and completed clocks, a row crossing the window by waiting on the database's clock; a maintenance take refused with no holder to name while a check's shared lock is held across its tries, and taking the lock on a later try once the shared lock ends; maintenance_claim itself passing a shared lock released after its first failed try (refused with one try in place of six, verified); a released claim reading free at once on another connection, twenty rounds with no wait; a RUNNING record stamped 60 s ahead of the clock listed by list_running_restore_jobs and missed by the zero-window dead-job listing); migrations_serving 6 (1 new: a command's registration named after it, the migration's wait naming it, the name kept when the registration is taken again, and a refusal while a migration runs not advising to start a server); table_snapshot 81 and the other 19 binaries green.
• extenddb-storage-sqlite: 255, 1 ignored (re-measured; 7 new: read_store_ids, the maintenance claim with the stale takeover, a dead CREATING row removed, a server start clearing a maintenance claim a stopped command left, list_backup_rows_older_than on both clocks, a command under the serve lock taking over a claim whose heartbeat is young, the holder it took it from finding its claim lost; a zero window listing and removing FAILED and CREATING rows whose heartbeat is a day ahead of the clock, which a 300 s window reads as live (fails with the zero-window rule removed, verified)).
• extenddb-storage-mongodb (replica set at 127.0.0.1:27117): lib 166, 53 ignored (re-measured; 1 new: the older-than filter naming one clock column and requiring it set), interpreter_selftests 18, pushdown_parity 20, migrations_fresh 8, migrations_guards 7, migrations_holders 10, migrations_locks 11 (1 new: a command registering under its own name: the document's command field, the migration's wait naming it, the name kept when taken again, its refusal under a migration's lock without the server's advice and a server's with it), migrations_released 5; -- --ignored --test-threads=1 live_ 53 (7 new: read_store_ids; the maintenance claim with the stale takeover and a heartbeat-less record of an older release treated as stale; a dead CREATING row removed; list_backup_rows_older_than on both clocks; an open beside a server starting no key repair of a document stored without its key, where a server's open then repairs it, through start_key_repair, the call build_components makes; a claim document with a fresh heartbeat and no owner_generation taken, not refused, and a live holder's still refused (refused with upserted_id alone, verified); newest_available_backup_row the last AVAILABLE row of the listing and list_running_restore_jobs listing a record stamped 60 s ahead of the clock that the zero-window dead-job listing misses). Of the live tests this round ran the four claim and listing ones against the replica set at 127.0.0.1:27117; the rest are carried.
• extenddb-storage-cassandra (127.0.0.1:9042, one binary at a time): lib 129 (1 new: a command's release cell carrying its name and read back, the wait's line naming the command); migrations 22 (1 new: a command registering under its own name, the migration's wait naming it and not extenddb serve, the name kept when taken again, its refusal under the lock without the server's advice and a server's with it, no row left); migrations_stopped 8; direct_integration 212 with --features test-hooks (re-measured; 7 new: read_store_ids over the status partitions, the maintenance claim with the LWT takeover, a dead CREATING row removed with its status partition checked, list_backup_rows_older_than on the coordinator's clock; two takers of one stale claim row held by a test-hooks pause after their reads, both writing, one winning (both won with IF EXISTS, verified); an insert whose answer the test hook loses after it committed settled by the re-read (refused with the re-read skipped, verified); newest_available_backup_row and list_running_restore_jobs with a record stamped a minute ahead of the coordinator); of this binary the backup_engine:: module (33) ran this round, the rest is carried; ttl_integration 26 passed, 1 failed, 1 ignored, test_ttl_enable_rejects_asynchronous_gsi being main's failure (it expects "asynchronously propagated GSIs" where the refusal says "global secondary index"; neither the test nor the refusal is touched by PR 6 or this PR) (carried).
• Wire: tests/test_backup_store_config.py (new; excluded from the backend-agnostic pytest run and run after it by devtools/run-tests: in its own step on a SQLite build, with the S3 cells against moto, which .github/workflows/integration.yml's SQLite job now starts on port 9200 from a virtualenv of its own under $RUNNER_TEMP (requirements-moto.txt pins boto3 and botocore 1.43.120 where requirements.txt pins 1.43.14, so installing it into the job's environment would have run the suite on moto's pins), and in the PostgreSQL CLI lifecycle list; the 12 SQLite cells ran on this head's debug SQLite binary against moto at 127.0.0.1:9200, 12 passed): on the SQLite build 12 cells (the default root created 0700 with the identity file and reported healthy, the banner line in the log line's spelling, CreateBackup logging the configured store; disabled with no component and the deprecation warning; an in-memory database ([storage.sqlite] path = ":memory:", sqlite-memory's default) with [backup.store.filesystem] path set healthy with its identity file and a backup created there restoring with its item, and without [backup.store] no default root, no component and CreateBackup logging the warning; a group-writable parent making the default root a warning and the same root explicit a startup error; a missing explicit root and a spool in the database directory stopping startup; an S3 endpoint that refuses connections starting the server with the component unhealthy, reason unreachable, CreateBackup still served, the restore and delete refused naming s3 and not the bucket, recovering when a forwarder to moto starts listening, creating the identity file in the bucket, and degrading again when it stops; a stalled S3 endpoint holding neither the listener nor the reconciler: /health answered with the component checking, a seeded row's restore refused as pending, then after the 30 s bound unreachable, the next cycle's check opening a new connection within seconds and shutdown not waiting; one AVAILABLE row under the default root with no manifest making it a configured store, degraded with manifest probe failed, the row's restore and delete refused naming the kind, CreateBackup served, healthy again once the manifest is back, backup list listing the row and purge --confirm refusing on the unhealthy store; a relative root in daemon mode resolving against the shell's directory; a catalog re-created after destroy warning about the old default root; backup purge --confirm with stale_job_after_secs = 45 over seeded AVAILABLE, FAILED, DELETING and dead-owner CREATING rows, a FAILED and a CREATING row with heartbeats 5 s old, and a RUNNING legacy restore with a CREATING target, resolving the restore without waiting, deleting each prefix and row outright with no wait on the young rows, leaving a prefix with no row, exports/ and the identity file, logging six audit lines, releasing the claim, then the wait's limit and refusal on a RUNNING restore of kind manifest (which the zero-window pass leaves for its resumer and the wait lists by status): 90 s, refusing naming its table and deleting nothing, with a FAILED row stamped 120 s ahead of the clock beside it not named as a live owner and, once the restore record is gone, deleted outright with no wait; a waiting purge (held by the manifest restore) sent SIGTERM exiting within 5 s with the claim released and nothing deleted, one SIGKILLed leaving its claim, a second purge run right after it taking the claim over and reaching its wait, and the next server start clearing it with CreateBackup accepted at once; and the server starting healthy against the purged root afterwards; backup list and the dry run before any server creating nothing, the dry run naming the missing default root and exiting 0, the dry run and list beside a running server, backup purge --confirm on an empty store and refused with exit 6 while a server holds the SQLite serve lock, --config after the subcommand, and destroy leaving the store); on the PostgreSQL build 4 cells (no store: no component and the warning; an S3 store without spool_path stopping startup; a filesystem store healthy with the identity file naming postgres, and an S3 Express bucket name stopping startup naming the bucket; backup list, the dry run and purge --confirm exiting 6 while a session holds the serving key as a migration does, naming that session without advising to start the server, then list and purge --confirm exiting 3 with the migrate --yes advice on a catalog migrate --forget 6 put behind the binary, and list exiting 0 once migrate --yes has run; with the command's refusal mapping removed it fails at the first exit code, 1 for 6). On the tree before its rebases onto 7f79f87e and ee6e119d (not re-run since), tests/test_backup_restore_fidelity.py and tests/test_table_operations.py on three served legs: SQLite with the default store (63 passed, 1 skipped: the DELETING-table test needs EXTENDDB_TEST_BACKEND and a host SQLite newer than this one's), PostgreSQL without a store and PostgreSQL with an S3 store on moto (64 passed each with EXTENDDB_TEST_BACKEND=postgres EXTENDDB_CONFIG=<toml>); every server drained its workers at shutdown; the deprecation warning appears on every CreateBackup without a store and never with one; the S3 leg's store is checking at the listener line and passed its check 140 ms later.
• The CI jobs, run their way on this head (release builds of this tree; on this host's Python 3.10 with the pinned requirements): the SQLite integration job (devtools/check-sqlite-catalog-lock, all checks passed; devtools/check-sqlite-older-binary against extenddb/extenddb-dev:0.1.14, all checks passed; the schema check, no difference; devtools/run-tests --extenddb --pytest --comprehensive --parallel --filter "not import_export" against a server initialised in the checkout's root, 1666 passed, 21 skipped, 1 xfailed, and comprehensive 502 passed); the PostgreSQL job's schema check (no difference) and devtools/check-postgres-catalog-lock, all checks passed (its pass through a connection pooler skipped: no POOLER_URL); MongoDB's mongodb-catalog (check-mongodb-schema, check-mongodb-catalog-lock 29 checks, check-mongodb-auth 16, check-mongodb-released 53, check-mongodb-older-binary against extenddb/extenddb-mongodb:0.1.14 28, each all passed) and mongodb-rust (devtools/run-mongodb-tests --rust-live -- --rust-integration with the mongodb-test-hooks release build: tests/rust 514, the records' live tests 6, the migration suites, the live Rust tests 19); Cassandra's cassandra-storage (the rows above), cassandra-rust (devtools/run-cassandra-tests -- --rust-integration: tests/rust 514), and from cassandra-catalog-checks the schema check (no difference) and check-cassandra-catalog-lock (42 checks, all passed, on a node of its own configured as the job's service). devtools/check-{postgres,mongodb,cassandra}-catalog-lock gain a cell that runs backup list and the dry run with no server, first with no store configured and then with a filesystem store (a 0700 root under the script's work directory, through catalog_checks.with_filesystem_store, so the commands open the store and release their registration when they close the catalog rather than on open's no-store path), and then migrate --yes, which must run at once (on MongoDB the registration count is read first). The MongoDB script ran on this head's release binary (36 checks, all passed, in a container of its own); on a build with the release removed from Opened::close it fails the backup commands released their registration and migrate --yes right after the backup commands (exit 6 naming extenddb backup list, pid <pid>), and passes again once the release is back. The PostgreSQL and Cassandra scripts are compiled and read, not run here (the one needs psycopg2, the other pauses its node, which is shared here). catalog_checks.start_mongo compared its readiness ping's output with endswith("1"), which a refused connection's error naming port 27041 passed, so the set was initiated before mongod listened and never became primary on this host; it now requires the output to be 1. Through the binary on MongoDB and Cassandra: backup list, the dry run and purge --confirm leave no registration once they exit, and migrate --yes right after them does not wait (with the release removed, MongoDB's documents stay and migrate --yes exits 6 after its 20 s lock timeout); under a migration's lock they exit 6 naming the holder with "Retry once it has finished" and without the server's advice; on a catalog migrate --forget 6 put behind they exit 3 with the migrate --yes advice. Not run: mongodb-pytest, cassandra-pytest, the PostgreSQL pytest job, cassandra-two-nodes, cassandra-authorizer and check-cassandra-older-binary (which builds v0.1.14 from its tag); this rebase changes no request path on those backends, and the server's registration they start is exercised by the Rust suites and the catalog-lock checks above.

cargo fmt --all -- --check
cargo +1.97.0 clippy -p extenddb-storage -p extenddb-engine -p extenddb-app -p extenddb-server -p extenddb-backup-store -p extenddb-storage-{postgres,sqlite,mongodb} --all-targets -- -D warnings
cargo +1.97.0 clippy -p extenddb-storage-cassandra --all-targets [--features test-hooks] -- -D warnings
CARGO_TARGET_DIR=target/t188 cargo +1.88.0 check --workspace --all-targets
cargo check --workspace --tests
EXTENDDB_TEST_PG_REQUIRED=1 EXTENDDB_TEST_S3_REQUIRED=1 ... cargo test --locked --workspace --exclude extenddb-storage-cassandra --exclude extenddb-storage-mongodb
EXTENDDB_TEST_MONGODB_REQUIRED=1 cargo test --locked -p extenddb-storage-mongodb; ... --lib -- --ignored live_
EXTENDDB_TEST_CASSANDRA=required cargo test --locked -p extenddb-storage-cassandra --lib | --features test-hooks --test direct_integration | --test ttl_integration -- --test-threads=1
devtools/generate-software-license-notices --check [--backend mongodb|cassandra]; devtools/generate-dev-license-notices --check

Not changed here

• The store-backed CreateBackup job and worker (PR 8): with a store configured, CreateBackup writes the in-catalog format and says so at info; the catalog has no CREATING or AVAILABLE rows of a store until then, so the manifest probe passes on an empty root and purge finds nothing to delete in a fresh deployment. The jobs' claim check after the row exists is PR 8's store-backed job; the in-catalog paths check before their row.
• The manifest restore (PR 9a): the restore check of a configured store's AVAILABLE row still answers "this release restores in-catalog backups only", after the health check.
• backup sync and its options, --adopt-store included (PR 10): a root whose file names another catalog stays unhealthy until an operator changes the file.
• sqlite-memory's startup sync (PR 10).
• The audit log target extenddb::audit is new with backup purge; CreateBackup, DeleteBackup and RestoreTableFromBackup are not yet logged under it.
• The reconciler's other store duties (resolving a dead CREATING row, deleting a FAILED row's prefix, tombstone retention) are PR 8's.

Checklist

• I have read CONTRIBUTING.md
• All tests pass (cargo test --workspace), the pre-existing Cassandra TTL failure above excepted
• Code is formatted (cargo fmt --check)
• Clippy is clean (cargo clippy --workspace --all-targets -- -D warnings)
• I have added or updated tests for new functionality
• I have updated documentation if behavior changed
• Breaking changes are noted below (if any)
• If this changes the wire protocol, Storage trait, auth model, on-disk format, or public CLI surface, an RFC has been accepted or is linked below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: #396, sections 4(c), 4(f), 7, 8, 9 and the PR 7 section, revision item 29. Storage trait: BackupCatalog gains read_store_ids, maintenance_claim, remove_dead_creating_backup_row, list_backup_rows_older_than (with StaleClock), newest_available_backup_row and list_running_restore_jobs, check_maintenance_claim takes the stale window, MaintenanceClaim is new; ServerComponentsOptions gains beside_server, startup_recovery and command; StorageConfig gains database_file with a default. BackupStore gains put_if_absent.

Breaking changes

• /health answers {"status": ..., "components": {...}} instead of {"status": "healthy"}; status is degraded while a component is unhealthy, with HTTP 200 kept; reason is a category; an S3 store's component is checking right after startup. A probe that parsed the body for "healthy" alone now sees degraded during a store outage.
• A configuration with [backup.store] is validated at startup: a bad root, bucket, part size, prefix, endpoint or spool placement, an S3 store without spool_path on PostgreSQL, MongoDB or Cassandra, and a spool another instance holds, stop the server with the path or bucket named. A relative filesystem.path or spool_path now resolves against the working directory at load; an existing spool with a mode other than 0700, or under a parent others can write to, stops startup.
• On file-backed SQLite a server without [backup.store] now creates backups/ beside the database file (0700) and extenddb-store.json in it; a root that fails the owner-only rule or whose identity file names another catalog is a logged warning, not an error.
• With a configured store that is unhealthy or not yet checked, the restore and delete of a store-backed backup answer ValidationException naming the store's kind; CreateBackup is not refused. DeleteBackup of a store-backed backup answers the DELETED description once its DELETING write has committed, whatever the object deletion does; a failed deletion is logged and left to the reconciler, never an error to the client.
• backup purge holding the maintenance claim makes CreateBackup and RestoreTableFromBackup answer LimitExceededException on every instance.
• Out-of-tree storage backends must implement read_store_ids, maintenance_claim, remove_dead_creating_backup_row, list_backup_rows_older_than, newest_available_backup_row, list_running_restore_jobs and the new check_maintenance_claim signature, and honour ServerComponentsOptions::beside_server and startup_recovery (and may name their serving registrations after ServerComponentsOptions::command) (on PostgreSQL it also decides whether the GSI queue workers start, a note for backends with their own); out-of-tree BackupStore implementations must implement put_if_absent.
• New CLI surface: extenddb backup list [--failed] and extenddb backup purge [--confirm]. On PostgreSQL, MongoDB and Cassandra they hold a serving registration for their run, so migrate --yes waits for them, and they exit 6 while a migration runs and 3 on a catalog behind the binary, as serve does. None runs a server's startup recovery. On SQLite purge --confirm beside a running server exits 6 (it exited 1).

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0 and I agree to the Developer Certificate of Origin (DCO). See CONTRIBUTING.md for details.

[LeeroyHannigan]

I built the SQLite and PostgreSQL servers at this head and at #671's head, and ran this head's servers in containers with a filesystem store. I ran 77 mutants of the new code against the engine, app, server, config, store and backend suites, and tests catch 55 of them. The backend suites ran against PostgreSQL 16, a MongoDB 7 replica set, Cassandra 4.1 and moto 5.2.4. I compared the CreateBackup and RestoreTableFromBackup errors and the LimitExceededException envelope with Amazon DynamoDB in us-east-1, twice each.

All 36 CI checks pass on this head. The test counts match the description: engine 290, server 303, app 46, config 23, SQLite 254, PostgreSQL backup_restore 47 and Cassandra direct_integration 209. I did not run a MongoDB or Cassandra server, only their backend suites.

The identity file and its check against another catalog's file matched the description. So did the /health shape (HTTP 200, degraded with the category and no root) and the refusal of store-backed deletes and restores while the store is unhealthy. The purge's claim, wait, deletions and audit lines matched too, and it leaves the identity file, exports/ and prefixes with no row in place. While a purge holds the claim, CreateBackup and RestoreTableFromBackup answer LimitExceededException in the service's envelope on SQLite and PostgreSQL, and every earlier refusal answers first. A second purge fails at once naming the holder, and SIGTERM during the wait releases the claim.

The in-catalog flow and the error cases answer as on #671's head. The TableNotFoundException, TableAlreadyExistsException, BackupNotFoundException and BackupName ValidationException texts match the service byte for byte.

Twelve Major and seventeen Minor items are inline, each starting with its severity. Smaller items:

• docs/differences-from-dynamodb.md has no row for two answers this PR adds, and the service has neither. One is LimitExceededException A backup maintenance command holds this catalog's maintenance claim; backup and restore jobs are refused until it finishes from CreateBackup and RestoreTableFromBackup while a purge holds the claim, where main and #671's head serve both. The other is ValidationException The filesystem backup store configured on this server is unhealthy (owned by another catalog); the operation is refused until the store's check passes from DeleteBackup and RestoreTableFromBackup of a store-backed backup. Please add both rows with HTTP 400, and the DeleteBackup 500 if it stays.
• From #671: a store row's description omits SourceTableFeatureDetails, where the service answers {} for a table without TTL. A DELETING row of a store other than the configured one is described as DELETED, while DeleteBackup answers that it is in another store.

Once you fix the twelve Major items, I will approve.

[LeeroyHannigan]

Major: this sets validated to true even for a store whose startup validate failed, so check_unbounded (line 462) skips validate and the next identity check reports it healthy. I ran a SQLite server whose validate probe write fails with EIO: startup logged the store failed its validation: write probe ...: Input/output error (os error 5) ... the check runs again each reconciler cycle, and 2 s later, without running validate again, it logged is healthy again and /health answered {"components":{"backup_store":{"kind":"filesystem","status":"healthy"}},"status":"healthy"}. A MemStore test whose validate fails also gets store_healthy == Some(true) after one cycle. The 290 engine tests pass with validated false here. Please build the store with validated false when its outcome is a failed validate (the I/O arm at crates/server/src/backup_store.rs line 461). Add that test too.

[LeeroyHannigan]

Major: this excludes tests/test_backup_store_config.py from the main pytest run, and no CI job runs it, the CLI lifecycle list at line 563 included. So CI never runs its purge, SIGTERM, SIGKILL, S3 /health and PostgreSQL exit-code cells, the only tests of three properties in crates/app/src/cmd_backup.rs. The 46 app tests pass with stale_job_after_secs as SQLite's owner window, no release after a failed wait, or purge --confirm beside a server (lines 618, 648 and 289). Please run the file in CI on SQLite (with moto) and in the PostgreSQL CLI lifecycle list. Otherwise, move its cells into Rust tests.

[LeeroyHannigan]

Major: CI never runs this test or the other three backup live tests this PR adds, because the only --ignored step filters data_engine::live_tests:: (devtools/run-mongodb-tests line 354). Against a replica set the module passes, 26 of 26. This test fails when the upsert filter at backup_catalog.rs line 922 loses owner_generation: {$exists: false}, or when the takeover filter at lines 905 to 911 matches a live record. Please add backup_engine::live_tests:: to that step.

[LeeroyHannigan]

Major: no purge_row test checks the manifest-first order here, the listing after the deletion at line 365, or the InUse refusal at line 346. With the deletes swapped, the listing gone, or purge continuing past InUse, the 290 engine tests still pass, while DeleteBackup's tests catch the first two. Without the listing, a purge that overlaps a put leaves objects under a prefix no later purge deletes. Without the InUse refusal, a restore that started after the wait would lose its source once restores read the store (PR 9a). Please add cases for the order (MemStore records the deletes), an object left under the prefix, and DeleteStart::InUse.

[LeeroyHannigan]

Major: with the is_healthy filter gone, the 290 engine tests still pass. The existing test makes the store unhealthy through the failure switch, so its deletion fails with or without the filter. The description says the reconciler touches no row of an unhealthy store, and this filter prevents it from deleting under a root another catalog owns. Please add a case with a foreign extenddb-store.json and a store that answers deletes. It should assert that no delete runs and the row stays DELETING.

[LeeroyHannigan]

Minor: with [backup.store] absent, a SQLite server on a database file still reports backup_store for its default root: /health answered {"components": {"backup_store": {"kind": "filesystem", "status": "healthy"}}, "status": "healthy"}. Please say the component is present whenever the server has a store, the SQLite default root included.

[LeeroyHannigan]

Minor: when a defaulted SQLite root's startup checks time out, this returns the store configured and checking whatever defaulted says. A later failing check, such as another catalog's identity file, then makes a defaulted root with no rows an unhealthy configured store. The module doc at lines 27 to 33 and docs/manuals/05-admin-guide.md line 165 promise a warning that leaves such a root unconfigured. Please keep defaulted with the store and apply the rows rule to the first check's outcome.

[LeeroyHannigan]

Minor: on an unhealthy store the dry run lists the rows under Backups to delete and ends Run again with --confirm to permanently delete every backup of the configured store listed above. --confirm refuses an unhealthy store (line 580), so please say on Identity::Problem that --confirm refuses until the store's check passes.

[LeeroyHannigan]

Minor: this says jobs check the claim after their own claim and row exist. On this head they check before the row, as the doc at crates/engine/src/backup.rs line 488 says. Please name both orders here and the release each belongs to.

[LeeroyHannigan]

Minor: this commit inserted store_root_label and normalized_endpoint between normalize_store_prefix's doc comment and the function, so store_root_label's rustdoc starts with Normalize the configured bucket prefix and normalize_store_prefix at line 1418 has none. Please move the three lines above this one to sit above normalize_store_prefix.

[robinnsc]

Thanks for review, addressed the feedback below, in your order, with the test that catches each mutant you named.

• crates/engine/src/backup/runtime.rs:211, validated true after a failed validate. ConfiguredStore::new now reads it from the outcome: a Transport, Denied or Validate problem (the three validate can end in) builds the store unvalidated, so the next check runs validate again; an identity problem still builds it validated. Test review_round_2::a_store_whose_startup_validate_failed_is_rechecked_by_validate (engine): a MemStore whose validate fails stays unhealthy through a cycle and is healthy once the probe passes; with validated true it reported healthy after one cycle and the test fails.

• devtools/run-tests:471, the wire file never runs in CI. tests/test_backup_store_config.py now runs after the main pytest run: in the PostgreSQL CLI lifecycle list, and in a new step on a SQLite build (detected from extenddb version), with EXTENDDB_TEST_S3_ENDPOINT passed through; .github/workflows/integration.yml's SQLite job starts moto on port 9200 (the unit-test workflow's pattern, requirements-moto.txt) and sets the variable. The 12 SQLite cells ran here on this head's binary against moto. The purge cells changed with the SQLite item below: the wait's lever is now a RUNNING manifest restore, and the FAILED row stamped ahead of the clock is deleted outright.

• crates/storage-mongodb/src/backup_live_tests.rs:972, the live tests not in CI. The backup_engine::live_tests:: step in devtools/run-mongodb-tests is feat(backup): catalog index, BackupCatalog, JobClaim, legacy reader [backup/restore 6] #671's change (the same step serves both PRs); this PR adds two live tests to that module, which the step then runs.

• crates/engine/src/backup/maintenance.rs:358, purge_row's order, listing and InUse. Test review_round_2::purge_row_deletes_manifest_first_lists_after_and_stops_at_in_use (engine): MemStore records the deletes (manifest key, then prefix), a late put after the delete leaves the row DELETING with the reason naming the object, and DeleteStart::InUse returns the restore's name with no delete run and the row AVAILABLE. Each of the three mutants fails it.

• crates/engine/src/backup/reconciler.rs:346, the is_healthy filter. Test review_round_2::an_unhealthy_store_whose_deletes_would_answer_has_no_row_touched (engine): a foreign extenddb-store.json, a store that answers deletes, a DELETING row; no delete runs and the row stays DELETING. With the filter gone the row is deleted and the test fails.

• crates/engine/src/backup.rs:463, the maintenance refusal's place. Test review_round_2::the_maintenance_refusal_comes_after_the_requests_own_checks (engine): with the claim held, a create under a server.region outside [a-z0-9-] gets check_backup_arn_fits's ValidationException (the dot-only name is backed up since feat(backup): catalog index, BackupCatalog, JobClaim, legacy reader [backup/restore 6] #671's review, so the cell uses the region rule) and a restore with ProvisionedThroughputOverride on a PAY_PER_REQUEST source gets validate_override_for_source's, and the claim is not read for either; the same requests without their fault are refused on the claim. Moving the check before check_backup_arn_fits or before validate_override_for_source fails it.

• crates/engine/src/backup/store_identity.rs:478, recheck's untested rules. Test review_round_2::a_recheck_catches_rows_with_another_id_a_lost_file_and_a_foreign_file (engine), through the reconciler as a running server runs it: rows arriving with another id are identity mismatch, a file deleted while rows exist is identity file missing and is not rewritten (the put count is unchanged), and a file replaced with another catalog's is owned by another catalog; the store is healthy again once each is undone. Removing each of the three rules fails it.

• crates/engine/src/backup/maintenance.rs:278, RUNNING restores by age. BackupCatalog::list_running_restore_jobs is new, by status alone, on the four backends (PostgreSQL and SQLite WHERE status = 'RUNNING', MongoDB restore_job.status, Cassandra the RUNNING scan with no cutoff), and busy_jobs reads it; the comment is gone. Tests: review_round_2::the_wait_sees_a_running_restore_with_a_fresh_heartbeat on the double (a record stamped this instant holds the wait; with the zero-window listing in place it is quiet and the test fails); on PostgreSQL running_restores_are_listed_by_status_whatever_their_heartbeat and on MongoDB live_the_newest_available_row_and_the_running_restores_are_read_by_status, each with a record stamped 60 s ahead of the clock that the zero-window dead-job listing misses; Cassandra's test_the_newest_available_row_and_the_running_restores_are_read_by_status likewise.

• crates/app/src/cmd_backup.rs:691, the lost-claim and signal checks untested. The loop is maintenance::purge_rows now (next to purge_row, returning PurgeOutcome { deleted, skipped, stopped: Option<Stop> }), and the command prints what it returns. Test review_round_2::the_deletion_loop_stops_at_a_lost_claim_and_at_a_signal (engine): a claim already lost deletes nothing; a signal sent as the first row is reported stops the loop before the second; a row left for a reason is skipped and the loop goes on. Removing either check fails it.

• crates/app/src/cmd_backup.rs:265, close releasing the registration. The three catalog-lock scripts (devtools/check-{postgres,mongodb,cassandra}-catalog-lock) gain a cell that runs backup list and the dry run with no server and then migrate --yes with a short lock timeout, which must exit 0 at once; the MongoDB cell also reads the registration count after the commands. A recording registrar through the subcommands would need a fake StorageEngine behind create_server_components, which the app crate has no seam for; the script cell exercises the real path on the real binary. I could not run the scripts here (they build release binaries and start their own containers); they are compiled and read.

• crates/storage-cassandra/src/backup_catalog.rs:1423, the takeover's condition and the lost-answer re-read untested. Tests in tests/direct/backup_engine.rs: test_two_takers_of_one_stale_maintenance_claim_see_one_win holds both takers after their read of the stale row with a new test-hooks pause (pause_before_maintenance_takeover), releases both, and asserts one Ok and one AlreadyExists; with IF EXISTS both win and it fails. test_a_maintenance_insert_whose_answer_is_lost_is_settled_by_the_re_read arms a test-hooks fault that turns the committed insert's answer into Unknown (lose_next_maintenance_insert_answer) and asserts the call holds the claim, heartbeats and releases it; with the re-read skipped it is refused and the test fails.

• crates/storage-postgres/src/backup_catalog.rs:820, the take's tries untested. Test a_maintenance_take_passes_a_shared_lock_shorter_than_its_span (backup_restore): the check's shared lock is held until maintenance_claim's first try has failed (seen in pg_stat_activity) and then released, and the take succeeds on a later try; with MAINTENANCE_TAKE_TRIES = 1 it is refused with no holder and the test fails.

• crates/engine/src/backup/reconciler.rs:479, the swapped arguments. Restored to target.account_id, target.table_id.

• crates/storage-sqlite/src/backup_catalog.rs:769, a future heartbeat under the zero window. A zero window now selects by status alone (zero_window_clause) in list_backup_rows_older_than, remove_dead_creating_backup_row and list_dead_restore_jobs: under the serve lock every owner is dead, as the guide says. Test catalog_index::a_zero_window_selects_dead_owners_by_status_alone (SQLite): FAILED and CREATING rows stamped a day ahead read as live under 300 s and are listed and removed under zero; without the rule it fails. The wire file's purge cell now seeds the FAILED row 120 s ahead beside a RUNNING manifest restore: the wait names the restore only, and once the restore is gone the row is deleted with no wait.

• crates/engine/src/backup/maintenance.rs:316, the deletes carrying no generation. Not changed: the row writes the loop makes (start_backup_delete, set_backup_deleted, remove_backup_row) are the ones DeleteBackup shares and carry no generation on any backend, so fencing them is a trait change across the four backends that the store-backed job's own row ownership (PR 8) is the place for; as you say, two deleters in the heartbeat window run one idempotent sequence and lose nothing. The loop now stops before its next row once the heartbeat reports the loss, which purge_rows's test checks.

• crates/engine/src/backup/reconciler.rs:383, store calls outside the bound and the in-flight rule. Every store call of a check, a command's identity read and the deletions goes through one adapter (ConfiguredStore::calls, BoundedCalls): each call under the store's bound, the catalog's reads beside it, a filesystem call abandoned at the bound marked in flight until it returns and every later call refused at once while it is. DeleteBackup's and the reconciler's finish_backup_delete use it. Test review_round_2::the_deletions_run_under_the_stores_bound (engine, paused clock): DeleteBackup on a stalled store returns at 5 s with the row DELETING, the next cycle starts no call, and once the mount answers the deletion finishes. This also covers the feat(backup): CreateBackup job, worker and reconciler duties [backup/restore 8] #700 note on the catalog reads: a slow catalog no longer counts against the bound.

• crates/app/src/cmd_backup.rs:598, the listener after open. run starts the listener before open in every subcommand and, when a signal arrived during the open, closes the opened catalog (releasing the registration) and exits; HeldClaim::release is bounded by RELEASE_TIMEOUT (10 s), tested in maintenance::tests::a_release_the_catalog_does_not_answer_ends_at_its_bound.

• crates/engine/src/backup/store_identity.rs:501, the probe listing every AVAILABLE row. BackupCatalog::newest_available_backup_row(store) is new on the four backends (ORDER BY created_at DESC, backup_arn DESC LIMIT 1 on PostgreSQL and SQLite, find_one with the reverse sort on MongoDB, the status index's AVAILABLE partition ordered by the instant minted into the backup id and one row read on Cassandra), and manifest_probe reads it. Test review_round_2::the_manifest_probe_reads_one_row (engine) and the backend tests above.

• crates/storage-cassandra/src/backup_catalog.rs:1436, "held by" on an Unknown takeover. The takeover re-reads at LOCAL_SERIAL on Unknown, as the insert does, and a row carrying this call's generation is its claim; the module doc says so.

• crates/storage-mongodb/src/backup_catalog.rs:930, a matched document without owner_generation. matched_count == 1 is taken. Test live_a_claim_document_without_a_generation_is_taken_not_refused (MongoDB live): an operator's document with a fresh heartbeat and no generation is taken and carries the taker's generation, a live holder's is still refused; with upserted_id alone it fails.

• crates/app/src/cmd_backup.rs:289, purge --confirm running the startup recovery. ServerComponentsOptions::startup_recovery (default true) is split from beside_server: every backup command sets it false, purge --confirm included, which still takes SQLite's serve lock. PostgreSQL skips the vector index rebuild, the control-plane recovery and the GSI workers, MongoDB the key repair, Cassandra the control-plane transitions, SQLite its recovery passes (the command resolves what it needs itself). The admin guide says so.

• crates/engine/src/backup/runtime.rs:736, the unreachable arm. Deleted; require_healthy_store reads the store with an expect naming the callers' check.

• crates/storage-postgres/src/backup_catalog.rs:893, the lock outliving close. release runs pg_advisory_unlock before close. Test a_released_claim_reads_free_at_once (backup_restore): twenty rounds of take, release, check with no wait. The race did not reproduce here without the unlock in three runs, so the test documents the property rather than killing that mutant.

• crates/engine/src/backup.rs:1230, the 500 after DELETING committed. DeleteBackup answers the DELETED description once start_backup_delete commits; a failed object deletion is logged and left to the reconciler (a_failed_deletion_stays_deleting_and_the_reconciler_finishes_it asserts the DELETED answer now). The engine doc, the design (5.2, item 29), the admin guide and the differences doc say so; no 500 row is needed.

• docs/manuals/05-admin-guide.md:67, the component's presence. Reworded: present whenever the server has a store, the SQLite default root included, absent with no store, disabled, or a default root that failed its check with no rows.

• crates/server/src/backup_store.rs:394, defaulted lost on a timed-out startup check. ConfiguredStore::checking(...).defaulted(defaulted) keeps it, and the reconciler applies the rows rule to the first check's outcome: a failure with no rows for the root logs the warning and sets the store aside (set_aside; the runtime then answers as a server with no store, /health has no component), with rows it is an unhealthy configured store. Test review_round_2::a_defaulted_root_set_aside_after_its_first_failed_check (engine); the warning at startup names the rule.

• crates/app/src/cmd_backup.rs:562, the dry run's closing line on an unhealthy store. On Identity::Problem it says the store is unhealthy (the category) and that --confirm refuses until the store's check passes.

• crates/storage/src/backup_catalog.rs:873, the order the doc states. The doc names both: this release's in-catalog create and restore check before their row (the backends commit it and detach), the store-backed job of a later release after its own claim and row exist (5.1 step 2).

• crates/backup-store/src/s3.rs:1365, the misplaced rustdoc. The three lines sit above normalize_store_prefix again.

Review body: the two docs/differences-from-dynamodb.md rows are added, with HTTP 400 and the texts byte for byte; the DeleteBackup 500 does not stay, so no row for it. SourceTableFeatureDetails {} and the DELETING-row wording are #671's and arrive by rebase.

[LeeroyHannigan]

I checked every item from my last review at 9765cdd. I ran the round-one reproductions again on the backends each item named, with the store in a container where an item needed one. I applied the round-one mutants and ran the tests your reply names. I also ran the MongoDB catalog-lock script on a build without the registration release.

27 of the 29 inline items are fixed, and so is the first item in the review body. Each reproduction now gives the right answer, and each named test fails on its mutant and passes after the revert.

Two items have replies on their threads. The registration release in Opened::close is still untested: the new script cell passes on a build without the release. I agree with your reply on the purge overlap for this release.

From #671, a store row's description now carries "SourceTableFeatureDetails": {}. A DELETING row of a store other than the configured one is still described as DELETED, while DeleteBackup answers that it is in another store.

The new code brings two Minor items, both inline.

Once the release cell fails without the release, I will approve.

[LeeroyHannigan]

Minor: calls that start before the first one reaches its bound all run, so two concurrent DeleteBackup requests on a stalled mount park two threads. A test with two delete_backup calls on a stalled MemStore counted two parked calls, and the doc above says no second thread is parked. Please bound the parked calls or correct the doc before #700 writes store rows, and add the concurrent case to the_deletions_run_under_the_stores_bound.

[robinnsc]

Bounded. A filesystem root's calls now run one at a time (ConfiguredStore::call_slot, a one-permit semaphore the call's task holds until the store answers): a call that finds another running waits for it under its own bound, so calls started before the first reached its bound park no thread of their own and answer at the bound with the same transport failure. The in-flight mark keeps its meaning (an abandoned call has not returned; later calls refused at once). the_deletions_run_under_the_stores_bound now sends two delete_backup calls together on the stalled MemStore: both answer DELETED at 5 s, both rows are left DELETING, the store counts one stalled call, and the reconciler finishes both once the mount answers. With the slot widened to two permits the store counts two and the test fails; it passes after the revert. The module doc, the admin guide's store paragraph and the design (section 7, item 30) state the rule.

[LeeroyHannigan]

Minor: this installs moto's pins into the job's only Python environment, so the integration suite then runs on boto3 1.43.120 and botocore 1.43.120. In a scratch venv, requirements.txt gave boto3 1.43.14, and a dry run of this file would replace it with 1.43.120. requirements-moto.txt keeps these pins apart for that reason, so please start moto_server from a virtualenv of its own.

[robinnsc]

moto now runs from a virtualenv of its own: python -m venv "$RUNNER_TEMP/moto-venv", its pip install -r requirements-moto.txt, and "$RUNNER_TEMP/moto-venv/bin/moto_server" -p 9200, so the suite keeps the requirements.txt environment installed above it. test.yml also installs requirements-moto.txt, but that job installs no requirements.txt and its moto client is the backup-store crate's Rust SDK, so nothing there is displaced and it is left as is; no devtools/ script starts moto. The step body passes shellcheck; I have no Python here that builds a virtualenv, so the step is read, not run.

[robinnsc]

Pushed fb12f6b1 over the rebased head: the first CI run's run-integration and run-integration-sqlite jobs failed at the setup of tests/test_backup_store_config.py's SQLite cells with FileNotFoundError: 'target/release/extenddb'. The SQLite deployment runs init with cwd set to its own directory (the relative-root cell needs the shell's directory), and CI names the binary relatively, so subprocess resolved it against that directory. The suite now takes EXTENDDB_BINARY as an absolute path; #700's tests/test_backup_job.py imports it and gets the same fix (its MongoDB deployment met the same error). cassandra-storage's first run failed on main's test_ttl_enable_rejects_asynchronous_gsi and a GSI-backfill timing cell and passed when rerun.

[robinnsc]

The DELETING row of another store stands as replied: DescribeBackup answers the row's lifecycle state (DELETED, with the warning logged), DeleteBackup refuses to act on another store's row, and the design's 4(g) says so. SourceTableFeatureDetails: {} is #671's. The release cell now fails without the release, as above; the engine's 303 tests, cargo check --workspace --tests and clippy 1.97 are green on this head.

[robinnsc]

Pushed eb15f901 over the second-round head: one test fix. On #740's S3 job the purge cell's final cleanup failed on a prefix the started server's reconciler had already deleted (the stale FAILED row the cell uses as its lever, deleted by the FAILED-prefix duty in the first cycle, as designed); the cleanup now tolerates the prefix being gone. No code change.

[LeeroyHannigan]

I checked the four items from my last review at eb15f90. You fixed all four.

I ran devtools/check-mongodb-catalog-lock with each extenddb command in a container, so the filesystem store opens. On a build where Opened::close does not release, three checks fail. With the release, they pass. the_deletions_run_under_the_stores_bound fails with two permits and with the earlier code, and it passes on this head. The moto step leaves the job's boto3 at 1.43.14 and runs moto 5.2.4 from its own virtualenv. I agree with your reply on the purge overlap.

The slot change brings one Minor item, inline.

I also found a defect in code that did not change since my last review, so it is not a condition for this PR. BoundedCalls::run sets the in-flight mark at runtime.rs:692, after the timeout returns and outside the call's task. If the task ends between the timeout's last poll and that line, its ClearOnDrop runs first. The mark then stays set with no task to clear it, and the server refuses every later check until it restarts. I found this by reading the code and did not reproduce it.

The DELETING row of another store is in #671's code, so my answer to your comment is on #671.

[LeeroyHannigan]

Minor: the bound now includes the slot wait, so the adapter abandons a 3 s call after it waits 3 s for the slot. In an engine test on a healthy root, two DeleteBackups together left both rows DELETING, and the next check marked the store unreachable. Please start a call's bound when it gets the slot, before #700 writes store rows.

[robinnsc]

Fixed. BoundedCalls::run waits for the slot under the bound and, once the call holds the slot, gives the call the bound for itself (tokio::time::timeout(bound, task) after the slot is taken, where it was bound.saturating_sub(started.elapsed())), so a 3 s call behind a 3 s call finishes. review_round_2::a_calls_bound_starts_when_it_holds_the_slot runs two DeleteBackups whose store call each takes most of the bound (MemStore::set_delay) through one slot and reads both rows DELETED and the root healthy; with the earlier bound restored it fails with "both deletions finished" and the second row left DELETING. The doc on BoundedCalls and the comment at the wait say where the bound starts, and design section 7, item 30 and the admin guide's bound sentence now say the bound covers the call once it holds the slot.

[robinnsc]

The Minor is fixed as above (head ba6b82d4), and the mark race you read at runtime.rs:692 with it: the call's task now carries a running: Arc<Mutex<bool>> beside the in-flight mark, its ClearOnDrop sets running false and clears the mark under that lock, and the caller that abandons the call at its bound takes the same lock and sets the mark only while running is still true. A task that ended between the timeout's last poll and the mark is therefore never marked, so no mark can be left with nothing to clear it. I did not reproduce the race either; the fix closes the window by construction and the existing abandoned-call cells (a call parked past the bound marked, cleared when it returns) still pass. Engine backup:: 153 passed on this head, clippy 1.97 clean.

[LeeroyHannigan]

I checked the slot-bound item and the in-flight mark at ba6b82d. You fixed both.

Two DeleteBackups whose store calls take 3 s each now both end DELETED, and the store stays healthy. With the earlier bound restored, a_calls_bound_starts_when_it_holds_the_slot fails. For the mark, I opened a 100 ms window before it in a scratch build, with a call that ends 50 ms after its bound. This head leaves no mark, and the earlier code leaves it set. The lock is never held across an await, and the abandoned-call cells pass.

One Minor item is inline: no test fails with the earlier mark.

[LeeroyHannigan]

Minor: with the mark set unconditionally here, as at eb15f90, all 153 backup:: tests still pass. The race needs the task to end between the timeout and the mark, and no test ends a call there. Please add a test that fails with the unconditional mark.

The base branch was changed.

[LeeroyHannigan]

#671 merged with my Minor on design section 4(g) still open. So two statements in 4(g) on main now differ from the code:

• 4(g) says DescribeBackup of a DELETED row answers DELETED. On SQLite the code answers BackupNotFoundException, as section 4(a) says.
• 4(g) says the other-store ValidationException answers what the state check leaves. The in-use rule and a restore's name check run first. So an AVAILABLE row of another store, restored into an existing table, answers TableAlreadyExistsException.

This PR edits the same design doc. Please fix 4(g) when you restack this PR onto main.