[robinnsc]

Backup/restore series (design: #396): #384 restore fidelity (merged) · #393 store · #394 format · #395 SnapshotEngine · #666 catalog migration framework (merged) · PR 6 the catalog index · PR 7 configuration, the store identity file and backup purge · this PR: the CreateBackup job · next: the restore job (PR 9a).
Closes #699 (part of #624).

On main at 71bcf55a, which carries #671 (merged as ff6f74eb) and #672 (merged as 4a42c5e0); rebased onto it after each merge, with no conflicts (the last two rebases add main's #466, #754, #756, #819 and #875). The text below still says "PR 6" and "PR 7" for #671's and #672's code: this PR adds nothing to the catalog schema (every column the job writes exists from PR 6's migration 6; docs/schema/*.txt are unchanged, checked against a fresh init on PostgreSQL and SQLite); it binds PR 7's store, identity file and health to the job, and PR 6's claim, row and reconciler to the duties. Where the PRs meet: the job takes PR 7's require_healthy_store and the maintenance check before the permit (as the in-catalog paths do; the review moved the check in front of the row, so a refusal writes none) and PR 6's claim_backup after the snapshot; the reconciler's cycle carries PR 7's store check and DELETING duty and PR 6's dead-restore duties beside this PR's dead-CREATING, FAILED-prefix, DELETED-prefix and tombstone duties, with the maintenance flag gating the last three; backup purge's zero-window pass on SQLite runs as the claim's holder (PR 7's ReconcilerState::for_maintenance_holder), so on this tree its cycle skips the three deleting duties and the registrations sweep, which the held-branch test now pins; the reconciler lists its FAILED-prefix and tombstone rows through PR 7's list_backup_rows_older_than, which PR 7 added for the purge's live-owner test; the jobs' maintenance refusal is PR 7's text ("A backup maintenance command holds this catalog's maintenance claim; ..."); the PostgreSQL snapshot slot's refusal now goes through PR 6's limit_exceeded_message; StatusFields (PR 6) gains one member every backend's set_status writes; BackupCatalog gains two methods beside PR 7's. Since the last push main merged #335, #666 and #681 under the stack: PR 6's migration is 6 (#335 took 5 for dropping continuous backups), Cassandra and MongoDB run it through #681's numbered runner, and the job drain now runs in #666's shutdown path, before the registration supervisor's return and the stop of the restartable worker set the reconciler belongs to, so this PR's own worker drain is gone, a job still running at the drain's bound keeps the serving registration as a worker that could not be stopped does, and extenddb stop waits for the sum of that path's bounds (76 s). The earlier rebase onto #672's head conflicted in nine files: serve.rs (that order), the Cassandra workflow (this PR's none on #681's cassandra-rust job), the engine tests' imports, and the differences, limits, usage and upgrade documents and the design, where #335's PITR rows and PR 6's migration 6 text meet this PR's text (main's rows kept beside this PR's). The rebase onto the review rounds of #671 and #672 conflicted in fifteen files, resolved so that both PRs' review changes stand beside this PR's: JobPermit::acquire_within (the 30 s wait before LimitExceededException) is what the store-backed path's take_permit().await takes; fail_restore_job carries the owner generation and heartbeat #671's reconciler passes; the reconciler's state is the Arc<Mutex<..>> #671's serve.rs shares; the in-catalog create keeps #671's name rule (a dot-only or over-long table name is backed up in-catalog, as the service backs up any table it can create), and the store-prefix rules on the name that #671 removed from that path move to the store-backed path as check_store_prefix_fits(region, account_id, table_name), run before the store's health check, the maintenance check and the permit, so a name no prefix can carry is refused with the same ValidationException texts and writes no row (a_store_backed_create_refuses_a_name_no_store_prefix_can_carry, new); #672's unhealthy-store row in the differences document now names the store-backed create beside the delete and the restore; the clippy job keeps #671's run with the Cassandra and MongoDB test-hooks features and adds this PR's run with extenddb-engine/backup-test-hooks; the TableName ARN region refusal, the 30 s permit wait, the catalog reads beside the store check's bound, the DELETED answer and the defaulted-root rule from the two review rounds are re-applied to this PR's rows of the differences document and the admin guide. #671's review item took revision item 28 and #672's are 29 and 30, so this PR's are 31 and 32 (every reference in the tree renumbered). On the store-backed row's path the target name is checked once the row is known to exist and AVAILABLE (check_catalog_row_status), before the backup's in-use rule and before this release's refusal of a store-backed restore (check_catalog_row_for_restore): the service's order, measured (a restore of a backup into its own CREATING target is TableInUseException; a missing backup into any target BackupNotFoundException), which the first CI run of this head caught on the S3 and MongoDB store jobs, where test_restore_into_an_existing_name_is_table_already_exists and the auth suite's test_an_existing_target_is_evaluated_with_its_tags met the store refusal before TableAlreadyExistsException (a_store_backed_restore_checks_the_target_name_after_the_row_and_before_its_refusal, new, pins the three answers and that a row that is not AVAILABLE answers before the name is read); design 5.3 step 1 states the order.

Merge order. This PR and #740 (the restore job) merge together, with no main commit between them: this PR makes CreateBackup store-backed wherever a store is configured, which on file-backed SQLite is the default, and RestoreTableFromBackup of such a backup is refused until #740. A main that carried this PR alone could not restore the backups it takes (the maintainer measured 8 of 8 restores refused where #672's head and the service restored). The store-backed create is not disabled here; the two PRs are one release.

What

With a configured [backup.store] on PostgreSQL, SQLite and MongoDB, CreateBackup writes the backup to the store and returns it CREATING; a job of the server uploads it and marks the row AVAILABLE. Without a store, and on Cassandra always, the in-catalog create of PR 7 runs as before. The reconciler runs its full duty list. No request or response shape changes; the API-visible changes are the asynchronous create (clients poll DescribeBackup, as against the service), CreateBackup of a vector table succeeding on the store-backed path, CreateBackup refused with ValidationException while a configured store is unhealthy or its first check pending, LimitExceededException for a second CreateBackup over max_concurrent_jobs while one job runs and for a CreateBackup while backup purge holds the maintenance claim (checked before the permit, as on the in-catalog path, so no row is written for the refusal), ServiceUnavailable for a CreateBackup whose snapshot open or row insert meets the cancellation of the jobs' token, which a client sees only after a registration loss (on SIGTERM the listener's 5 s drain closes the connection before the token is cancelled, so the maintainer's client saw a closed connection 8.2 s after sending; the row is not written either way), and a backup of a table holding an N value the wire did not validate (#618) failing on the store-backed path (CREATING, then BackupNotFound; the reason in backup list --failed and the log). DescribeBackup's ItemCount and TableSizeBytes are the stats pass's figures while CREATING and the stream's (the items written, the sum of their DynamoDB sizes) once AVAILABLE. On SQLite on a platform where the serve lock is not taken (not Unix) CreateBackup stays in-catalog whatever [backup.store] says (backup_store::in_catalog_create_only): the job's claim is an in-process slot, sound only with one process per file.

Core (crates/core). validation::validate_restorable_item(item, &RestorableKeySchema, &RestorableLimits): one walk of the item, every N and NS member parsed once on the accepting path with the wire's number rule, the nesting depth counted on the way, the key attributes (RestorableKeySchema::resolve from the key schema and attribute definitions, multi-part keys included) checked first with the format's own check_key_value (the declared type compared before the value is parsed), then the key sizes against the instance's hash and range key limits, every attribute name against max_attribute_name_bytes, then the size (RestorableLimits from &LimitsConfig); RestorableItemError { key, path, reason } with Display key {key}, attribute {path}: {reason}, every key value, name and path segment bounded on its own so a legal key appears whole whatever its bytes (a key value at SHOWN_KEY_VALUE_BYTES, 12,288 escaped bytes, six times the hash key limit; names, path segments and echoed numbers at SHOWN_VALUE_BYTES, 4096), and every such text written as a JSON string's content with U+0085, U+2028 and U+2029 escaped beside the control characters, so neither a control character nor a Unicode line separator in a tenant's item can add a line to the server's log (a reader using str.splitlines sees one line too); RestorableItemReason::{InvalidNumber, KeyShape, AttributeNameTooLong, TooLarge, TooDeep, LineTooLong}, the last carrying the cap. backup_format::encode_item_line_len_bounded(item, cap) measures an item's line through a counting writer that stops at the cap, so the feeder's measurement of an oversized item holds no line; limits::DEFAULT_MAX_ITEM_SIZE_BYTES is the feeder's threshold. backup_format::check_key_value and shown are pub. Metrics: MetricName::{BackupJobs, BackupJobDuration, BackupBytesUploaded, BackupSnapshotAge, BackupStoreErrors}, BackupJobOutcome, QuerySource::BackupJob, the recorders on MetricsCollector; /metrics emits them under those names as JSON like every other metric (the endpoint is not Prometheus exposition format), BackupJobDuration in microseconds from the snapshot's open to the job's last catalog or store call (for a FAILED job its FAILED write and delete_prefix included), BackupBytesUploaded for a job that ended AVAILABLE, WorkerErrorCount under BackupJob for a FAILED job (a cancelled one records neither worker metric).

Storage (crates/storage). JobClaim::heartbeat_within(bound), a provided method that bounds heartbeat over its whole call and answers None at the bound; the engine's timer and its claim re-check call it instead of wrapping heartbeat in a timeout, so a backend can place the bound (SQLite, below). BackupCatalog::claim_dead_backup(key, stale_after) -> Option<Box<dyn JobClaim>> (the reconciler's takeover of a dead CREATING row under a fresh generation, heartbeat_at untouched; None for a live owner, a fresh heartbeat, a row that is not CREATING or missing, or a claim another task holds), sweep_restore_registrations (a default empty body), StatusFields.source_table_details_json; the trait doc of #672's list_backup_rows_older_than names the reconciler's two listings beside the purge's. snapshot::paging::SNAPSHOT_FETCH_BYTES (64 MiB) with fetch_bytes(), record_page_bytes, record_rows_past_cut and the test-hooks readers and override; snapshot::source_table_schema(description, ttl) builds the manifest's source_table from a snapshot's description (round-trips through the manifest reader, absent members as explicit nulls); snapshot_conformance::run_page_bounds (a one-byte budget measures the row and proves the knob took effect, then an eight-row budget: every row once, pages at most the budget plus one row, the cut fired; a stream that has yielded more items than the table holds fails the check at once, naming the key yielded twice, so an inclusive seek fails in seconds rather than running until the hooks lock's wait gives out).

PostgreSQL (crates/storage-postgres). The cursor is replaced by keyset pages in the same REPEATABLE READ transaction: three layers (LIMIT n after the last key, the running byte sum and the row count, the rows under the budget in key order), the key columns under qualified aliases and bound as their column types, COLLATE "C" only where the probe found it, the rows pulled from the connection one at a time and parsed as they arrive (the page's text is never held whole), n 16 for the first page and then the budget divided by the largest row of the last pages that together hold at least a row bound's worth of rows (RowBoundMemory), never under 4 (MIN_NEXT_PAGE_ROWS) nor over 1000: the server serializes and spills every row the inner LIMIT reads, so a page costs it n rows of text and not the rows it returns, and sized from the average row a page of 984 small rows and 3 large ones asked the next page for 810 rows of a table that was then large rows only (5.9 GB of server temp files, 18 to 44 s for the 3 rows it returned), where the largest row asks for 4 (208 MiB, 1.1 s); a page of 27 MB rows reads one row past its cut rather than thirteen; sized from the previous page alone the bound oscillated on a table whose large rows are scattered among small ones (4 after a large row, 1000 after four small ones, the 1000-row page meeting large rows again every other page: 5,555 rows read for 1,000 kept and 13.4 GB of temp files on a 1,000-row table with 30 such rows), where the memory keeps it at 4 while large rows keep arriving within a row bound's worth of rows (1,901 read, 3.9 GB; a memory of four pages read the same 5,555 as one); the bound that remains, stated as it is: the second page, and then the page that follows a row bound's worth of small rows, reads up to n rows of up to the line cap, at most once per row bound's worth of small rows, and nothing in this form stops the server at the budget (a walk that does was measured and not taken, below); the floor's cost is 250 pages of four after every large row of a table whose large rows are rarer than one per row bound (2,585 page queries against 38 with one page of memory on a 20,000-row table with a 27 MB row at every 1,100th key, 25.6 against 25.1 s, the record); the stream ends when fewer rows than asked came back and none was cut; page_sql_for (taking the collation flag open_stream decides), MIN_PAGE_ROWS and MIN_NEXT_PAGE_ROWS exported for the tests; the slot refusal through limit_exceeded_message naming [storage.postgres] pool_size snapshot-slot. claim_dead_backup takes the row's advisory lock on a dedicated connection and then UPDATE ... WHERE status = 'CREATING' AND heartbeat_at < now() - make_interval(secs => $stale); set_status writes source_table_details_json = COALESCE($10, ...); try_claim sets tcp_keepalives_idle = 30, tcp_keepalives_interval = 10, tcp_keepalives_count = 3 (CLAIM_KEEPALIVES), each failure a warning, before #672's retried lock.

SQLite (crates/storage-sqlite). SqliteJobClaim::heartbeat_within takes the engine write lock first and bounds only the UPDATE: the beat queues behind every writer (a GSI backfill, a large write), and that queue is this process's own, not a sign the claim is lost, so a writer holding the lock past a period delays the beat and the claim re-check and ends no job (the maintainer saw two backups fail beside a GSI build on a 400,000-item table; the end-to-end test holds the lock for three periods and the backup ends AVAILABLE; a second cell, a_put_during_a_writers_hold_passes_on_the_beat_parked_behind_it, puts a data file per item through a store whose first put takes four periods and holds the lock until the second data file's put has started, so that put's check runs with the first beat parked and no beat confirmed since the insert, and the backup ends AVAILABLE with its three data files and the manifest; with the in-flight clause removed the second data file is never put and the row is left CREATING). store::connect_options builds every pool connection's options (row_buffer_size four), so the run-ahead test opens a connection with the engine's options and a counting collation. claim_dead_backup is the slot rule: JobSlots::take is the liveness test and the exclusion in one step, stale_after unread, then a conditional UPDATE under BEGIN IMMEDIATE; the byte cut in fetch_page as a row stream (the stream pulls no row past the cut; the driver's worker reads up to five ahead of it, below); backup::pool_size_for_jobs(pool_size, max_concurrent_jobs) and SqliteEngine::new_for_server: a file-backed pool is raised to 2 * (min(jobs, 16) + 1) with a warning naming both numbers so the snapshot slot refuses no job the permit admitted (10 at the defaults, the default pool; over 16 jobs the warning names the jobs the slot cap refuses, 17 to max_concurrent_jobs, pool_raise_message); the rule is stated once on SqliteConfig::pool_size. The pool's connections are built with sqlx's row_buffer_size at four (store.rs): the driver's worker runs each statement ahead of the task reading it, and its default of 50 buffered rows let a snapshot of large rows hold up to 50 lines the page never asked for; the rows ahead of the reader are five (the four the channel holds and the one the worker has stepped and copied while its send waits), 160 MiB at the line cap.

MongoDB (crates/storage-mongodb). claim_dead_backup is one find_one_and_update on status: CREATING and heartbeat_at < $$NOW - stale, the generation set through a $literal pipeline (literal_set, now one function for every pipeline write); set_status writes the member; the snapshot stream applies no cut (the 16 MiB reply bound is under the budget), stated in its module doc.

Cassandra (crates/storage-cassandra). claim_dead_backup: a LOCAL_SERIAL read, staleness against the coordinator clock, then UPDATE ... SET owner_generation = <fresh> IF owner_generation = <read> AND backup_status = 'CREATING' AND heartbeat_at = <read> (the heartbeat in the condition puts the liveness test inside the write), an Unknown outcome settled by a re-read; set_status writes the member; a test-hooks pause between the read and the LWT (test_hooks::pause_before_takeover_write) lets the direct test land the owner's beat in that gap and see the LWT refuse; sweep_restore_registrations removes active_restores entries whose target is not a CREATING restore target of that backup, by LWT on the entry as read. Cassandra has no SnapshotEngine, so these run only on rows the reconciler meets.

Engine (crates/engine/src/backup/). create_backup branches on configured_store().is_some() && !legacy_create_only(): the store-backed path requires a healthy store, checks the maintenance claim (refuse_while_maintenance_held, before the permit as on the in-catalog path: a row written FAILED maintenance for the refusal carried the insert's heartbeat and backup purge's wait counted it as a live job for the stale window; the maintainer's eight purge attempts beside four creating clients all gave up, leaving 16,054 such rows), takes the permit, and spawns job::start through the runtime's task tracker, awaiting its one-shot reply; refuse_backup_of_vector_table stays in front of the in-catalog create alone. job.rs: start opens the snapshot and inserts the CREATING row through claim_backup, both selected against the shutdown token (an open waiting behind a table's ACCESS EXCLUSIVE lock no longer holds the drain for its whole bound; the request answers ServiceUnavailable, which reaches a client only when the token is cancelled while the listener still serves, a registration loss: on SIGTERM the listener's 5 s drain closes the connection first), mints the id from the open's instant, reads the TTL, builds the source_table (AlreadyExists on the insert is an internal error), answers CREATING, and runs the job: writer.rs (a feeder task validating every item into a four-item channel and, for an item over the default item limit, validating it and measuring its line against MAX_LINE_BYTES on the blocking pool (the walk's cost grows with the item) through a counting writer that holds no line, a line over the cap refused through the same path with a reason naming the cap; one spawn_blocking encoder per data file writing 1 MiB chunks into a four-chunk channel that is the put's body; a write into the sink after the pipeline stopped is BrokenPipe, which the encoder does not retry, so a job cancelled mid-file ends within milliseconds rather than spinning through the drain; a file closes at the first item that takes it to or past data_file_target_bytes; KeyRangeTracker per item on a key-ordered table; the first refusal stops the encoding and the uploads but not the validation, each refused item logged with key, path and reason up to 1,000, the rest counted; a put that fails ends the feeder at once, the feeder's token cancelled before the encoder is awaited (an encoder waiting on the item channel returns only once the feeder closes it, so the old order left a mid-file failure running for as long as the stream was held), so the FAILED write does not wait for the snapshot stream's next page; a data file whose stored size (meta.size) differs from the encoder's compressed count fails the job, since the reconciler's dead-owner probe refuses that difference and a live job must not commit a manifest the probe would call incomplete; before every put the job reads the store as the runtime holds it (Pipeline::store_check: healthy, and its identity the row's), and a store the reconciler marked unhealthy mid-job, or whose identity file now names another store (a root swapped under the job), fails the row with the store's reason and makes no store call, the prefix's deletion included (the FAILED-prefix duty deletes it once the store is healthy again); a stream that fails after a refusal gives the reason both, the refusals' count and first item and then the snapshot then failed: ..., so backup list --failed still names the item; a put is started only when a file's first chunk exists and only when a heartbeat confirmed the claim within two periods, the claim's insert standing for the first beat (the watch starts from the instant taken before claim_backup); one info line at the start with the table's count as the stats pass last counted it and one at the end with the items written, the bytes and the duration), then BackupDocuments::build with the stream's count and the sum of its items' sizes (the manifest's item_count and table_size_bytes), the three documents, a claim re-check, the manifest and the AVAILABLE write with the manifest's size, count and source_table; heartbeat.rs (a timer through heartbeat_within every max(heartbeat_interval_secs, 15) s; start takes the claim's instant; a lost, failed or unreturned beat raises the abort; confirms_the_claim is the pre-put check, inclusive at exactly two periods, and passes as well while a beat is in flight and has not failed (the maintainer's second round: a 180,000-item backup beside a GSI build stopped twice at the check before data file 3 or 4, its beat parked on the SQLite write lock past two periods, and the reconciler then failed each row and deleted its objects; the flag is set around every beat, the timer's and the re-check's, and cleared on the way out by a drop guard, so a timer that stops mid-beat leaves the two-period clause; the module doc argues why the clause is safe: on SQLite the beat parks only on this process's lock, one process serves the file and the reconciler's liveness test is the in-process slot, so no takeover coincides with a parked beat; on PostgreSQL, MongoDB and Cassandra the bound covers the whole call, so the two-period rule already covers a beat in flight and the clause does not apply (JobClaim::heartbeat_parks_before_its_bound, true on SQLite only): a process stopped mid-beat or between beats and resumed would otherwise pass checks while its beat runs, with another owner possibly holding the row; a loss is read before the flight); recheck is the claim re-check before the manifest, whose confirmation counts as a beat; the module doc states the outcome for a process stopped and resumed: on PostgreSQL no takeover while its session lives, a job with no beat in flight resumes under its own generation once its next beat succeeds however long the stop was (a job frozen 160 s finished AVAILABLE), and it stops as a lost claim only when a beat in flight outlives its bound or a put comes before the next beat); failure.rs (write_status: each unknown outcome followed by a re-read, the write sent once more at most, and after the second write a re-read that shows the row in the target status counted as matched and anything else as matching no row, the second write being for a backend whose claim connection survives the unknown outcome (MongoDB, Cassandra; on PostgreSQL it fails with the session and the row is the reconciler's); fail_row: FAILED first, delete_prefix only after it matched). A lost claim ends the job with no catalog write. KillPoint::{AfterDataFile, BeforeManifest, AfterManifest} pause the job under backup-test-hooks (EXTENDDB_BACKUP_PAUSE, released by the file EXTENDDB_BACKUP_PAUSE_RELEASE names). runtime.rs: JobSettings, with_metrics, with_shutdown (the jobs run on a child of the server's token), cancel_jobs (cancels the jobs running now and gives the next ones a fresh child token; the server's workers' stop calls it, so a registration lost while serving cancels the jobs rather than letting their heartbeats and status writes go on beside a migration), with_source, spawn_job (public, for the server's shutdown test), running_jobs, drain_jobs(bound), whose give-up line counts job tasks, whether or not a row exists for one yet; the configured store is held in an Arc the jobs share (into_configured_store returns it); ConfiguredStore gains bucket and key_prefix for the manifests. reconciler.rs: after the maintenance check, the dead restore jobs, the abandoned in-catalog backups and the store check, and while the store is healthy: the DELETING rows finished, dead CREATING rows taken over (also under a maintenance claim) and resolved to AVAILABLE when a complete, verified manifest with every data file at its recorded size is under the prefix, else FAILED (owner dead ...), nothing deleted there; then, without a maintenance claim: FAILED prefixes deleted once the heartbeat is stale (the same cycle's for a row it just failed: in that cycle on PostgreSQL, MongoDB and Cassandra, where the stale heartbeat made the owner dead, and once the dead owner's last beat is that old on SQLite, where the slot did), DELETED prefixes swept, tombstones past failed_row_retention_secs removed once their prefix is empty, the registrations swept; CycleReport and the cycle's log line carry the counts, a dead-CREATING resolution whose write matched no row counted as left to its owner; a duty's store call that fails (get and list of the manifest probe and the deleting duties, delete_prefix; not an absent object, not the store's check) records BackupStoreErrors under the call's name, so a store that refuses one row's calls is counted though the cycle completes.

Server and app. serve.rs creates the shutdown token before the runtime, gives it to the runtime with the metrics and the build's backend and version, and runs the shutdown sequence (drain_then_release: drain_jobs (30 s) after the listener stops, then the registration supervisor's return, then stop_then_release (#666), which takes whether every job ended and keeps the serving registration, to expire by itself, while a job may still run; one function, which the server's test drives with a job that outlives a 100 ms drain and asserts no release); BackgroundWorkers::with_jobs makes the workers' stop (the supervisor's, at a registration loss) cancel the jobs through cancel_jobs; lib.rs bounds the plain-HTTP listener's drain at CONNECTION_DRAIN_TIMEOUT as the TLS listener's is, so stop's 76 s sum holds for a dev-mode build too; backup_store::in_catalog_create_only(backend) names why a backend's CreateBackup stays in-catalog (Cassandra; SQLite where the serve lock is not taken) and the startup log says so; cmd_backup.rs's resolve_dead_owners prints the CREATING rows the purge's cycle resolved, with their new status, and the ones another owner took over; JOB_DRAIN_TIMEOUT, WORKER_STOP_TIMEOUT (the supervisor's 10 s, the workers' stop's 16 s and the release's 10 s) and the listener's CONNECTION_DRAIN_TIMEOUT are exported, and extenddb stop (cmd_stop.rs) waits for their sum and the runtime's RUNTIME_SHUTDOWN_TIMEOUT, 76 s, where it gave up at 10 s while the server was still inside the job drain, its timeout message naming the four parts; backup_store.rs builds JobSettings from [backup] and the manifest names from the store configuration (S3: bucket and prefix; filesystem: the root label and ""), and its startup lines say CreateBackup writes to the store. crates/app/Cargo.toml and crates/server/Cargo.toml chain the backup-test-hooks feature, and extenddb version prints backup-test-hooks enabled on such a build (the test-hooks line stays the MongoDB hooks' line that devtools/run-mongodb-tests reads). crates/config/src/backup.rs docs and the floor's error text name the in-catalog jobs' constant, and failed_row_retention_secs is refused under stale_job_after_secs (with a retention of 0 one cycle failed a killed job's row and removed row and prefix within 29 ms, and an object the stopped owner landed afterwards was never deleted: the FAILED-prefix duty needs the row).

Harness and CI. tests/test_backup_restore_fidelity.py: wait_for_backup_available, used by _create_backup, _cleanup and every cell that reads a backup it just made; a session fixture reads /health once and checks that the server's backup_store component is the kind EXTENDDB_TEST_BACKUP_STORE names and healthy (absent for none), so the variable describes the server under test rather than being taken on trust; the vector test reads the variable (filesystem, s3, none), fails unset ("is unset"), prints which branch ran, and runs the refusal (capturing the ARN of a create accepted against expectation) or the store-backed create (CREATING, AVAILABLE with the GSI in SourceTableFeatureDetails, listed, the restore's ValidationException naming the store until PR 9a); the 24 restore cells skip on a server with a store, naming PR 9a. tests/test_auth_permissions.py: the backup fixture's cleanup waits for AVAILABLE before DeleteBackup, and the restore-authorization cell skips on a server with a store for the same reason; tests/test_abac.py's backup cell waits the same way before its DeleteBackup (deleting at once answered BackupInUseException and left an AVAILABLE backup on every store-backed run), and tests/test_authz_resource_arn.py's module fixture, whose cells deny DeleteBackup to other users, deletes its backup once AVAILABLE and its table when the module ends (it left a complete backup under the store on every store-backed run); the fidelity suite's skip of its 24 restore cells is gated on the target as well as the variable, so a stale EXTENDDB_TEST_BACKUP_STORE in a real-DynamoDB run skips nothing. tests/test_backup_job.py (new, its own servers, run by name): a second CreateBackup over max_concurrent_jobs = 1 waits for the slot (#671's 30 s wait) and is answered only once the first job has ended, the first backup AVAILABLE at its answer (one answered while the first is still CREATING fails the cell; a first backup AVAILABLE before the second is sent asks for more items; a wait that outlives the bound on a slow host is LimitExceededException with the bound's text, which the cell accepts and reports), and a third is accepted at once afterwards; one cell per #618 write path (BatchWriteItem, TransactWriteItems, UpdateItem on an absent item in the key it creates, ImportTable) with three refused items: three log lines with the keys, the FAILED reason's count and first item naming #618, BackupNotFound, no manifest; a refused item with a UUID hash key and a ten-digit epoch range key named by its whole key, and the wire's reason for its value, in the log line and in backup list --failed (whose Rust Debug escapes the harness decodes); an item over max_item_size_bytes after a restart with a lower limit: a reason naming the size and the limit and not #618. tests/test_backup_store_config.py: the create refused on an unhealthy or checking store, the objects under the root and in the bucket once healthy, the start and completion log lines with their two counts. tests/test_backup_job.py's deployment fixture gains a MongoDB branch (a replica set of the suite's own through EXTENDDB_TEST_MONGODB_URI; the catalog's database names are fixed, so each test destroys and initializes the catalog), tests/test_mongodb_backup_restore.py's three cells skip on a server with a store (they read and restore the in-catalog create's documents; PR 9a brings them back), devtools/run-mongodb-tests --backup-store-root DIR configures a filesystem store on the server it starts, and integration-mongodb.yml gains mongodb-backup-store: the backup selection of the pytest suite with filesystem (the fidelity suite's store branch) and the job suite on a second replica set. .github/workflows/clippy.yml lints --features extenddb-engine/backup-test-hooks beside the default build, and the kill-point scripts are committed as devtools/backup-kill-points.sh and devtools/backup-kill-points.py (the API half, which #740 extends with the restore points), run from a checkout against a build with the hooks. .github/workflows/integration.yml: the PostgreSQL job appends a filesystem store after init and sets EXTENDDB_TEST_BACKUP_STORE=filesystem, the SQLite job sets filesystem (the defaulted root), a new run-integration-sqlite-s3 job runs the backup tests against an S3 store on moto with s3 (its rust-cache step saves only on main, as #680 made every other job's, its actions are pinned to the commits main's jobs use (#752), and it dumps moto's log on failure; its by-name step runs the store configuration suite's S3 cells and the job suite against the build's defaulted filesystem root, which the step says), and the PostgreSQL and SQLite jobs run the two store suites by name; the three start steps wait for the backup_store health component to be healthy rather than for the word anywhere in the reply, so a store that failed its check or is still checking is named at startup; the pytest lines carry -rP so a passed cell's printed branch reaches the job log; integration-mongodb.yml and integration-cassandra.yml set none; the Rust integration suite (tests/rust/src/backup_restore.rs) waits for AVAILABLE after every create that goes on to describe, delete or restore (create_available_backup, wait_for_backup_available), deletes the backups its tests make, and reads EXTENDDB_TEST_BACKUP_STORE as the Python suite does (backup_store: filesystem and s3 expect the restore's ValidationException naming the store, none the round trip, unset fails with the fixture's text), backup_arn_scoping.rs and restore_active_completeness.rs through it, since the SQLite Rust integration job's file-backed server has the defaulted root and its CreateBackup is the asynchronous one (five tests assumed the synchronous create and the suite left 12 complete backups under that root); the five Rust integration jobs set the variable (filesystem on run-rust-integration-sqlite, none on the PostgreSQL, MongoDB and Cassandra jobs); devtools/run-tests --pytest refuses when the variable is unset (its --help, CONTRIBUTING.md and the developer test guide name the variable, its three values and the refusal), naming the three values, ignores the two store suites in its line, and its health_check waits for the backup_store component to be healthy when the variable is filesystem or s3 (the server it restarts for [import] is the one the suites run against); the start steps' python3 reads curl's reply with its stderr quiet, so the seconds before the listener answers print no traceback; tests/test_backup_job.py's _started stops a server whose health wait or provisioning failed. crates/storage-sqlite/tests/backup_job_panic.rs (new; extenddb-engine and extenddb-backup-store as dev-dependencies): a job whose store put panics leaves a CREATING row that one reconciler cycle in the same process resolves to FAILED.

Docs. The differences document (the vector row, the restore-interruption row with the backup duties and the measured kill points, where backups live, backup creation with the record's durations, the CREATING ItemCount and TableSizeBytes that can be 0 and 0 and the stream's figures once AVAILABLE, the BackupInUseException "Backup is being created: " that DeleteBackup and RestoreTableFromBackup answer while CREATING lasts (the whole upload; the service's backups measured were AVAILABLE 12 to 90 ms after CreateBackup), and the snapshot's instant stated as the table at the snapshot's open, after the request's checks and after any wait for a permit (up to 30 s), so a write acknowledged during that wait is in the backup (the maintainer's second round: a PutItem acknowledged 1.01 s after a CreateBackup that waited for a permit was in it, 2 of 2 runs, where the row said no write acknowledged after the request was received), against the maintainer's measurement of the service (every write acknowledged before CreateBackup was sent and the 18 in the 0.1 s up to BackupCreationDateTime, none of the 11,754 in the next 75 s) in place of the "one-minute window" clause, the shutdown refusal's reach (a registration loss, not SIGTERM) in the same row and in the admin guide's Stopping paragraph, a new #618 row), the limits document (the SQLite pool raise and PostgreSQL's slot, the store row, a data-file row with the line cap and the file size, the contents row), the usage guide's backups row (Partial: CreateBackup to the store, RestoreTableFromBackup for in-catalog backups only this release), the upgrade manual (the asynchronous create, a store-backed backup listed, described and deleted but not restored until the restore job lands, the #618 paragraph, the rollback paragraph), the admin guide's Stopping paragraph (the connection drain, on the TLS and the plain-HTTP listener alike, the job drain's 30 s, the supervisor's 10 s, the workers' stop's 16 s, the release's 10 s, the runtime's 5 s, and stop's 76 s), its [backup.store] sentence on a parked store check and its statement of the different-roots gap (two host-local roots at one path with a copied identity file are not told apart before the first AVAILABLE backup, and one instance's reconciler fails the other's complete backup; one shared root from the first backup on), its failed_row_retention_secs row (at least stale_job_after_secs), the troubleshooting entry for stop's timeout, the deployment guide's container paragraph (a grace period under 35 s kills the process during the job drain; the three compose files set stop_grace_period: 80s, above the 76 s sum, where they set 15 s) and the architecture guide's shutdown step, the admin guide's [backup] rows (the memory figures, their scope at the default max_item_size_bytes, and MALLOC_ARENA_MAX=1 with its measured cost beside max_concurrent_jobs, the SQLite pool raise, the job's two log lines quoted whole, the /health examples' key order as the build's) and [backup.store] section, the deployment guide's service unit (MALLOC_ARENA_MAX=1 as a commented line with its condition and cost) and its SQLite sentence pointing at the admin guide's row, extenddb.sample.toml and the init template, the measurement record committed as docs/design/15-backup-restore-bench-pr8.md, design revision items 31 and 32 (the review's changes: the backend-placed heartbeat bound and, from the second round, the pre-put check's in-flight clause with its safety argument in 5.1 step 4, the step 2 text on the snapshot's open after the permit wait and on the shutdown refusal's reach, the maintenance check before the permit in 5.1 step 2 and 4(f), the store check before every put, 4(c)'s gap before the first AVAILABLE row, TableSizeBytes, the retention floor, the kill-point scripts in devtools/, the MongoDB job) with the body sentences they correct (section 3's byte budget as the three-layer page with up to n - 1 rows read past the cut, 4(e)'s unknown-outcome rule as write_status has it (the second write for a backend whose claim connection survives; on PostgreSQL it fails with the session) and its PostgreSQL liveness rule (a stale heartbeat and a free advisory lock; a stopped owner whose session lives keeps its row), 5.1 step 4's SIGSTOP outcome per backend (on PostgreSQL no takeover while the session lives; on resume the job completes, or after more than two periods stops as a lost claim with no catalog write and the reconciler resolves the row) and step 6's cancelled job completing AVAILABLE after its manifest put, 5.1 step 1's store-health check and step 2's unreadable maintenance claim, section 7 as a sizing rule scoped to the default item limit with the one-arena factor measured on both backends and the setting's request-path cost, section 9's metrics as emitted (the reconciler's BackupStoreErrors with the job's), section 10's table with PR 8's rows, the duty list's resume marked PR 9a's, the testing summary's kill points run by hand, the PR 8 acceptance's bounds and its paging cells).

Why

#396 section 5.1, 4(a) and 4(e), section 7's memory bound and pool rule, section 9, and the PR 8 section. A job without its reconciler leaves rows no one resolves, so both ship together (item 24). Decisions taken while implementing, each stated in the module doc of the code it concerns and in revision item 31: the job and the reconciler live in the engine's backup module and nothing crosses a channel between the request and the job but the response; the takeover is a method of its own, claim_dead_backup, since claim_backup inserts; the ages are backend predicates (list_backup_rows_older_than, which #672 now carries for its purge as well), so the engine holds no clock; the pre-put check is two heartbeat periods, since a beat sent at the tick returns up to a period later, or a beat in flight, since on SQLite a beat parked on the write lock behind a long writer is this process's own queue and not a loss; a lost heartbeat ends the job with no catalog write, the claim connection being the one that failed; the pipeline ends its puts before the FAILED write, so the failure path awaits nothing; the FAILED-prefix duty of the same cycle deletes the prefix of a row the cycle just failed once the dead owner's beat is stale; SQLite's snapshot slot stays the backend's admission bound and the pool is raised for it; the PostgreSQL page is three layers because a window function runs before LIMIT; the vector round trip's create half lands here and its restore half with PR 9a; the restore cells of the fidelity suite skip on a server with a store until then; EXTENDDB_TEST_BACKUP_STORE is set by every CI job that runs a suite creating backups (the pytest jobs and the Rust integration jobs: filesystem on the SQLite jobs, s3 on the moto job, none elsewhere) and refused unset. Three decisions came from the measurements: section 7's memory paragraph is the sizing rule the worst-expansion runs measured (the terms at the line cap, about 490 MiB per job on PostgreSQL and 650 MiB on SQLite: the page's decoded rows, the items in the feeder, the channel and the encoder, one row's parse, the line buffer, the chunks and SQLite's five-row run-ahead, times the allocator's factor, times max_concurrent_jobs), since the earlier "budget plus one row plus its decoded form" did not describe the 880 MiB peak of 27.7 MB rows; the allocator's factor is documented rather than set from the binary: with glibc's default arenas the peak grows with the number of large rows a job reads (466, 588, 760 MiB over idle at 12, 24, 48 rows on a quiet host; 707, 700, 1,281 MiB on a loaded one) and under MALLOC_ARENA_MAX=1 it is flat (452 to 481 MiB in both series), so the admin guide carries that setting for hosts that back up large items and the service unit carries the line commented out, since one arena costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%); and the item channel stays at four items after a one-item channel was measured 75% slower on a 1M-row backup for 50 MiB less at that peak, while the SQLite pool's row_buffer_size goes from sqlx's 50 to four, which bounded the driver's run-ahead at five rows at no cost to the 1M-row duration. The PostgreSQL page's row bound follows the largest row of the last pages rather than the average because the server serializes every row the inner LIMIT reads: the average sized a page after a mixed page at 810 large rows for the 3 it returned; the memory covers a row bound's worth of rows because one page of memory oscillated on a table whose large rows are scattered among small ones (and four pages read the same rows as one); a page query that stops at the budget on the server, a recursive walk of the primary key one index probe per row, read exactly the rows it returned (the interleaved table in 38.8 s with 2.5 GB of temp files against 54.6 s and 3.2 GB for the memory and 72.0 s and 9.2 GB for one page) and was not taken because the probe per row cost the 1M-row backup 95% of its duration (23.1 s against 11.9 s); the record carries both.

Testing done

Fourth review round: the in-flight clause applies only where the beat parks before its bound (in_flight_confirms is parks && in flight; the handle's in-flight mark is a flag again, the beat's start no longer read). Bounded to one period of the beat's start (the third round's fix), the clause still passed checks after a stop between beats: on resume the missed tick starts a beat at once, and every check in its round trip passed (Lee measured 5.2 to 6.2 ms), though the last confirmed beat was eight periods old. On PostgreSQL, MongoDB and Cassandra the two-period rule already covers a beat in flight. a_process_stopped_between_beats::a_stop_between_beats_passes_no_check_while_the_resumed_beat_runs confirms three beats, moves the clock eight periods with no task running, lets the missed tick start a beat that has not returned, and checks under both declarations: no check passes where the beat does not park, the check passes on SQLite's, and the beat's Lost answer ends both. It fails with the third round's bounded clause and with the unbounded one. The third round's cell is now a_beat_in_flight_confirms_the_claim_only_where_the_beat_parks (a beat within its bound no longer confirms where it does not park). The module doc's takeover sentence is now the two-period rule's, with the SQLite clause argued in its own paragraph; the trait doc, design 5.1 step 4 and item 32 and the admin guide's heartbeat_interval_secs row say the clause applies on SQLite only. Engine backup:: 199 passed on this head, storage-sqlite write_lock_tests 15, clippy 1.97 clean with the three test-hooks features.

Third review round: the in-flight clause of confirms_the_claim is bounded where the beat does not park before its bound. JobClaim gains a provided heartbeat_parks_before_its_bound (false; SQLite's claim answers true, its beat waiting behind the engine write lock outside the bound), the handle records the beat's start, and a beat in flight confirms the claim for as long as it waits where the beat parks and within one period of its start otherwise, so a process stopped mid-beat and resumed on PostgreSQL, MongoDB or Cassandra passes no check in the window before its timer raises the loss. review_round_3::a_beat_in_flight_past_its_bound_confirms_the_claim_only_where_the_beat_parks runs the same parked beat under both declarations (within the bound both confirm; past it only the parking one; the loss read first when the beat returns Lost); with the clause unbounded again its parks=false half fails. The module doc, design 5.1 step 4 and item 32, and the admin guide's heartbeat_interval_secs row say so. Engine backup:: 198 passed, storage-sqlite write_lock_tests 15, clippy 1.97 clean. The SQLite write-lock cell's doc says what the cell itself sees without the clause (one put started, the row left CREATING: no reconciler runs there); the job-bound cell's comment describes the slot-wait branches, and its admitted branch asserts that the first backup was seen CREATING while the second request waited.

The store jobs' runs of tests/test_backup_job.py's job-bound cell found the first backup AVAILABLE once the second CreateBackup was answered, on MongoDB and SQLite alike: since #671's review round a request that finds every slot held waits up to 30 s for one rather than refusing at once, so the second request was admitted when the first job ended, and the cell's expectation (an immediate LimitExceededException) was the earlier contract. The cell (test_second_create_backup_over_the_job_bound_waits_for_the_slot) now sends the second request from its own thread while watching the first backup, and asserts that it is answered only once the first is AVAILABLE and then succeeds; the refusal at the bound is kept as the branch for a first job that outlives the wait. The design's PR 8 acceptance says the same.

The maintainer's second round (three inline items), on this head: cargo test -p extenddb-engine --lib -- backup:: 194 passed (re-measured, --list: 194; review_round_3::a_beat_parked_on_the_write_lock_confirms_the_claim_until_it_returns_lost new), cargo test -p extenddb-storage-sqlite --lib 267 passed, 1 ignored (re-measured: 268; write_lock_tests::a_put_during_a_writers_hold_passes_on_the_beat_parked_behind_it new, the two write-lock cells run five more times, 1.7 s each run) and backup_job_panic 1 passed; cargo check --workspace --tests; cargo fmt --all -- --check; cargo +1.97.0 clippy -p extenddb-engine -p extenddb-storage-sqlite --all-targets --features extenddb-engine/backup-test-hooks -- -D warnings, clean. The mutant the maintainer's item implies, the in-flight clause removed from confirms_the_claim: the new SQLite cell fails (one put started under the hold, the row left CREATING: the second data file's check refused the put) and the new engine cell fails at its first assertion (the parked beat does not confirm the claim); both pass after the revert. The other counts and runs below are carried from the previous head; nothing else in the code changed.

After the rebase onto the review rounds of #671 and #672, on this head: cargo clippy --all-targets with extenddb-storage-cassandra/test-hooks,extenddb-storage-mongodb/test-hooks,extenddb-engine/backup-test-hooks under -D warnings, clean; cargo test -p extenddb-engine --lib -- backup:: 193 passed (a_store_backed_create_refuses_a_name_no_store_prefix_can_carry and a_store_backed_restore_checks_the_target_name_after_the_row_and_before_its_refusal among them); the extenddb-app (47), extenddb-config (24), extenddb-server (307), extenddb-storage (233) and extenddb-storage-sqlite (266 passed, 1 ignored) lib suites; backup_restore on PostgreSQL at 127.0.0.1:55432, 56 passed; backup_engine::live_tests:: on the MongoDB replica set at 127.0.0.1:27117, 31 passed; direct_integration -- backup_engine:: with test-hooks on a fresh Cassandra 4.1 node, 41 passed (a first run against the node earlier rounds had filled with test keyspaces failed 24 cells on connection timeouts while the node sat in old-generation collections; recreated, every cell passed). The wire, kill-point and memory runs below are from the previous heads and were not repeated; CI runs the wire suites.

The review round (the maintainer's 30 inline items). Re-measured on the previous head (--list): extenddb-core lib 797, extenddb-storage 230, extenddb-engine 331 (backup::tests 161), extenddb-server 307, extenddb-app 47, extenddb-config 24, extenddb-storage-sqlite lib 260 and backup_job_panic 1, extenddb-storage-postgres lib 78 and backup_restore 50, extenddb-storage-cassandra lib 129 and direct_integration 212 with test-hooks, extenddb-storage-mongodb lib 220; the other counts below are carried from the previous head. Run on this head: cargo test -p extenddb-engine --lib -- backup::tests (161 passed), the extenddb-core, extenddb-storage, extenddb-config, extenddb-app, extenddb-server and extenddb-storage-sqlite lib suites and backup_job_panic (all passed), backup_restore::backup_job::the_claim_session_sets_tcp_keepalives on PostgreSQL 16 at 127.0.0.1:55432, backup_engine::test_a_beat_between_the_takeovers_read_and_its_write_refuses_the_takeover on Cassandra at 127.0.0.1:9042 with test-hooks (7.4 s), tests/test_backup_job.py on a MongoDB release build against a replica set of its own (7 passed, 63 s), and devtools/run-mongodb-tests --backup-store-root ... -- --pytest --parallel --filter backup with EXTENDDB_TEST_BACKUP_STORE=filesystem on that build (40 passed, 29 skipped: the fidelity suite's 25 restore cells, the three MongoDB in-catalog cells, one authorization cell; the two collection errors were this host's Python 3.7 on unrelated files that use :=). Each test the maintainer asked for was run against the mutation he named and fails there: the SQLite write-lock test with the beat bounded over its whole call (the row stays CREATING); the mid-file put failure with the encoder awaited before the feeder's cancel (the job did not end within 10 s); the mid-job store check with store_check removed; the key-ordered stream with the tracker's refusal ignored; the BackupStoreErrors cells with the delete_prefix recording removed from failure.rs and the list recording removed from list_sizes; the re-read cell with the generation check removed; the two-period sample with <= changed to <; the nesting cell with the list depth's >= changed to >; the keepalive cell with tcp_keepalives_idle at 7200; the run-ahead cell with ROW_BUFFER_ROWS at 50 (51 rows stepped for one pulled); the Cassandra cell with AND heartbeat_at = ? removed from the LWT; pool_size_for_jobs(11, 4) with >= changed to >; the shutdown-sequence cell with jobs_left == 0 replaced by true. cargo fmt --all -- --check; cargo +1.97.0 clippy --all-targets -- -D warnings on the touched crates with extenddb-engine/backup-test-hooks and on extenddb-storage-cassandra with test-hooks (the host's 1.98 toolchain flags a pre-existing result_large_err in the console pages, which CI's 1.97.0 does not); cargo +1.88.0 check --workspace --all-targets --features extenddb-engine/backup-test-hooks; cargo check --workspace --tests; shellcheck on devtools/backup-kill-points.sh (clean) and devtools/run-mongodb-tests (no warnings, as before). The MongoDB CI job, the clippy step and the compose grace periods are not run here.

The rounds before. Counts are of the whole tree on the previous head, measured after the rebase onto #672's head 9434f34c and re-measured after the move onto main at e8ca7df (the workspace run on this tree; the MongoDB and Cassandra counts listed on this tree, --list, with their suites run on the top of the stack, #740); "new" is against #672's head. Runs marked "before the rebase" were made on this PR before the stack's rebases onto #666 and then main, and were not repeated.

• extenddb-core: lib 795 passed, 1 ignored (30 new: validate_restorable_item per shape, 1E126 and abc in a non-key attribute with the wire's reason text, the nested path a.b[2].c and the NS member position and member, validation past the first valid number, size at and over the limit, depth at and under the limit agreeing with validate_item_nesting_depth, a missing, mistyped and empty key attribute, abc and 1E126 in the range key as InvalidNumber and 1.0 as KeyShape, N("abc") under an S key as KeyShape, multi-part keys attribute by attribute, key attributes before the others, RestorableKeySchema::resolve's refusals, every key attribute of a legal key shown whole and only an oversized value cut, the attribute path and the echoed number bounded segment by segment, the key sizes and attribute name sizes at and over their limits, RestorableLimits from LimitsConfig, LineTooLong's text naming the cap, a control character in a name or a number escaped in the message, a legal key of control characters shown whole, the bounded line measurement stopping at the cap without a buffer; check_key_value comparing the type before parsing; the backup job recorders landing under the outcome and the store call; the three line separators escaped in a key, a name and a number; a 4097-byte key attribute name cut at SHOWN_VALUE_BYTES and a 20,000-byte list under a key attribute cut to a JSON string); data_file_line_bomb 1, data_file_nested_decode_memory 1, data_file_decode_peak 1 ignored.
• extenddb-storage: 220 (4 new: source_table_schema round-trips through ExtenddbManifest::from_json_bytes, absent members as explicit nulls, on-demand drops the throughputs, the TableClassSummary error); snapshot_conformance::run_page_bounds is run by the PostgreSQL and SQLite crates and fails in 3 s on a stream that repeats rows (an inclusive-seek mutant, which ran until another test's 600 s lock wait before).
• extenddb-engine: 325 (35 new: a snapshot stream that fails after a refused item keeping both in the FAILED reason, and the plain stream error keeping its own; the reconciler's failed get, list and delete_prefix each recorded once as BackupStoreErrors under its name while the cycle completes, and nothing recorded once the store answers; a tracing callsite first met by a thread without a scoped subscriber still reaching the test's LogLines (the suite's global default answers "sometimes" for every callsite, so a test running beside a log-asserting test cannot cache the job's callsites as "never"; the CI run of the previous push lost the lost-claim line that way); a 400 ms maintenance check at a 100 ms period delaying the request and not the first put, the backup AVAILABLE; the store-backed create answering CREATING and its job writing the backup with the put order, the manifest's instant, count, size, order and key range, the AVAILABLE row and the metrics; the maintenance claim refusing the request before any row; an open that fails leaving nothing; a healthy store making the create store-backed and a vector table backed up; an unhealthy store refusing create, delete and restore unless bound to the in-catalog create; the data-file close rule at the knob and at its 32 MiB minimum; the manifest recording the snapshot's order; the validator failing the backup after the rest is validated, with and without BatchWriteItem, TransactWriteItems and other operations store N values that fail number validation as written #618; a put failure failing the row then deleting the prefix; a FAILED write that does not match making no store call; an Unknown write re-read, applied and lost; a lost, stalled or erroring heartbeat stopping the job with no put; the handle confirming the claim within two periods; cancellation running the failure path; a second create over the bound; the reconciler resolving a dead CREATING row with a manifest to AVAILABLE and one without to FAILED; the FAILED and DELETED prefixes and the tombstones on their clocks under and without the maintenance claim; a write after the stop answering BrokenPipe and an encoder closing at once after the stop; a put failure ending the job while the stream is held, the start and completion lines' counts, 1,001 refusals through the log cap, a line over the cap refused and named, a status write error leaving the manifest and making no store call, a second unknown outcome, a stopped heartbeat timer starting no put, the claim re-check before the manifest, the probe's seven incomplete shapes and a store error during the probe, a row failed by the cycle losing its prefix in that cycle, a takeover lost to another reconciler left to it, the tombstone guard (an old AVAILABLE row with an empty prefix kept) and the registrations sweep gate, a second unknown outcome whose write landed matched, a dead CREATING row whose status write errors left for the next cycle, a FAILED job's duration covering its FAILED write and deletion, an eighth incomplete-manifest shape (an unparseable manifest), the resolved row's source_table_details_json equal to the manifest's; the validate-the-rest cell fed one item per 20 ms; BackupSnapshotAge asserted recorded per confirmed beat; the probe-error cell asserting no row left to an owner). feat(backup): [backup.store], the store identity file, backup purge [backup/restore 7] #672's held-branch test (a_cycle_run_by_the_claim_holder_takes_the_held_branch_whatever_its_window) also asserts here that the holder's zero-window cycle leaves a stale FAILED row's prefix and runs no registrations sweep, and that the same cycle reading the claim runs both; it fails on a held branch that runs the deleting duties. The two slow job tests wait under a bound of their own (wait_for_jobs_within: 300 s for the line-cap test, about 20 s alone, 22.8 s on two cores beside a busy loop; 120 s for the data-file test, 5 s alone, 10.1 s so loaded), where the 30 s guard failed the line-cap test three times in forty loaded runs; cargo test -p extenddb-engine --lib -- backup::tests under taskset -c 0-1 nice -n 5 beside one busy loop passed 20 runs of 20 (155 tests each, 26 to 27 s).
• extenddb-server: 305 (2 new: the backup job metric names render from the store; a backup job still running at the drain's bound keeps the serving registration while the workers stop, which fails when the release ignores the jobs). extenddb-app 47 (1 new: stop waits for the sum of the server's shutdown bounds, 76 s, the bounds after the job drain 36 s; it fails on the old 10 s and on a sum without the supervisor's and the release's bounds). extenddb-config 23, extenddb 3, extenddb-auth 274, extenddb-backup-store lib 37 and conformance 123, extenddb-cache 27: unchanged.
• extenddb-storage-postgres (127.0.0.1:55432): lib 78 (3 new: the_row_bound_adapts_within_its_clamps, sizing a page of 984 small rows and 3 large ones from the large row (under 8, not about 800) and pinning the floor at four, the_memory_covers_a_row_bound_of_rows, the_page_query_binds_the_key_by_type); backup_restore 50 (3 new: claim_dead_backup taking over a stale row on the database's clock and refusing a live one, with the successor's AVAILABLE write carrying source_table_details_json; a terminated claim session reporting Unknown and the row re-read; the claim session's TCP keepalives; the listing on the named clock is feat(backup): [backup.store], the store identity file, backup purge [backup/restore 7] #672's cell); table_snapshot 89 passed, 2 ignored (4 new, each on both database layouts: pages_are_cut_at_the_byte_budget with run_page_bounds and the rows past the cut asserted exactly, 8 over 77 pages, all on the first page; the_row_bound_remembers_large_rows_among_small_ones: a 300-row table with a 27 MB row at every 33rd key streamed whole in order, 201 rows read past the cut, all on the second page, over 53 round trips (306 over 7 with one page of memory, 103 round trips with a floor of 2); the_page_query_is_an_index_scan_on_every_key_shape: EXPLAIN of the first page and a page after a key at n 16 and 1000 on 20,000-row tables of the six key shapes (hash only, S, N and B range keys, a multi-part range key, a legacy table recollated to the database collation), Limit over one Index Scan on the primary key with no sort on each, then each table streamed whole in 21 round trips; the_last_page_sorts_the_rows_after_the_seek: the index scan ten rows from the end of the 100k table and the bitmap scan with a Sort near the end of a 1,060-row table, the sorted page's rows in key order); the other 20 binaries as before.
• extenddb-storage-sqlite: lib 257 passed, 1 ignored (3 new: the takeover of a row whose slot no task holds, the pool raise and its slots, the pool-raise warning's text at 8 and 20 jobs; the listing on the named clock is feat(backup): [backup.store], the store identity file, backup purge [backup/restore 7] #672's); backup_job_panic 1 (new: a job whose store put panics leaves a CREATING row one reconciler cycle in the same process resolves to FAILED).
• extenddb-storage-mongodb (replica set at 127.0.0.1:27117): lib 167 passed, 52 ignored (1 filtered), interpreter_selftests 18, pushdown_parity 20, the migration binaries migrations_fresh 8, migrations_guards 7, migrations_holders 10, migrations_locks 11, migrations_released 5 (1 new unit: the $literal set; the older-than filter's cell is feat(backup): [backup.store], the store identity file, backup purge [backup/restore 7] #672's); --lib -- --ignored --test-threads=1 live_ 52 (1 new: the takeover on the server's clock with the fence on the old claim's write, its window 3 s; the listing on the named clock is feat(backup): [backup.store], the store identity file, backup purge [backup/restore 7] #672's). Before the rebase two runs under a host running a 1M-item load beside them failed 5 and 3 restore cells on their 90 s ACTIVE wait and the takeover cell on its then 1 s window, and passed when rerun.
• extenddb-storage-cassandra (127.0.0.1:9042, one binary at a time): lib 129, migrations 22, migrations_stopped 8; direct_integration 211 with --features test-hooks (2 new: the takeover with the heartbeat in the LWT's condition, the registrations sweep keeping the live entry; the listing on the coordinator's clock is feat(backup): [backup.store], the store identity file, backup purge [backup/restore 7] #672's), run on 03baa321 in three parts of 53, 72 and 73 tests named --exact (the second part's first run met connection timeouts while the node was still dropping the first part's 45 keyspaces, and passed when rerun alone); a container holding the keyspaces earlier runs of the TTL suite left (165 extenddb_ttl_test_account_*) stalled two runs before the rebase in old-generation collections, and the runs here drop what they leave; ttl_integration 26 passed, 1 failed, 1 ignored, test_ttl_enable_rejects_asynchronous_gsi being main's failure PR 6 and PR 7 recorded (the file is not touched here).
• Wire, the way CI's jobs run it, on this head: the run-integration-sqlite job's steps on the SQLite release build (check-sqlite-catalog-lock and check-sqlite-older-binary against extenddb/extenddb-dev:0.1.14: all checks passed; init in the checkout's root and the schema check: no difference from docs/schema/sqlite.txt; serve --foreground --write-pid-file waited for a healthy store; devtools/run-tests --extenddb --pytest --comprehensive --parallel --filter "not import_export" with EXTENDDB_TEST_BACKUP_STORE=filesystem: pytest 1641 passed, 46 skipped, 1 xfailed, comprehensive 502 passed; the two store suites by name: 18 passed, 5 skipped); the run-integration-sqlite-s3 job's steps (a bucket on moto, the job's [backup.store.s3] block, the wait for a healthy S3 store, the backup selection --pytest --parallel --filter "backup" with s3: 40 passed, 28 skipped; the two store suites with the S3 endpoint: 19 passed, 4 skipped; extenddb stop of that server: extenddb stopped in 0.26 s); the catalog-lock checks of the PostgreSQL, MongoDB and Cassandra jobs (check-postgres-catalog-lock 16 ok without its pooler pass, check-mongodb-catalog-lock 29 ok, check-cassandra-catalog-lock 42 ok on a node of its own: all checks passed), which stop servers and wait for their registrations; cassandra-rust and mongodb-rust (run-cassandra-tests -- --rust-integration and run-mongodb-tests --rust-live -- --rust-integration with none: tests/rust 514 passed on each, MongoDB's live and migration binaries green); docs/schema/*.txt against a fresh init of each backend: identical; a catalog created by main's release build at migration 5, on SQLite and PostgreSQL: this build's serve refuses it (exit 3, naming migration 6), migrate --yes applies 6, serve starts and extenddb stop of it on PostgreSQL ends it in 0.26 s with its 16 workers drained and no session left, and main's serve refuses the migrated catalog (exit 4). Before the rebase onto 1234179b, devtools/run-tests --extenddb --pytest --comprehensive --parallel --filter "not import_export" on the PostgreSQL release build with a filesystem store appended after init and EXTENDDB_TEST_BACKUP_STORE=filesystem: 1439 passed, 52 skipped, 6 failed; with kind = "disabled" and none: 1464 passed, 27 skipped, 6 failed; on the SQLite release build with the defaulted root and filesystem: 1446 passed, 45 skipped, 6 failed; the comprehensive suite 336 passed, 4 failed on each. (These three full runs are on the binaries before the sink's BrokenPipe change and the SQLite row buffer, neither of which a wire test reaches; the backup selection and the two store suites were rerun on the final binaries.) The ten failures are the same on every run and on 62d32f0f: the host's botocore (Python 3.7) refuses ReturnValuesOnConditionCheckFailure (test_conditional_writes.py::TestConditionalPutItem, 4 cells; tests/python/test_item_edge_cases.py, the 4 rv_on_ccf cells) and OnDemandThroughput (test_config_fields.py::TestOnDemandThroughput, 2 cells) as unknown parameters before a request is sent; none is a backup test. The skipped cells include the 24 restore cells of test_backup_restore_fidelity.py on the servers with a store (PR 9a) and test_auth_permissions.py's restore-authorization cell for the same reason; test_create_backup_of_a_vector_table_is_refused ran its filesystem branch there and its none branch on the PostgreSQL run without a store. The S3 job's selection (--pytest --parallel --filter "backup") on the SQLite build against an S3 store on moto (127.0.0.1:9200) with s3: 27 passed, 28 skipped; the same selection on the SQLite build with the defaulted root: 27 passed, 28 skipped.
• Before the rebase, tests/rust, the Rust integration suite as the jobs run it (devtools/run-tests --extenddb --rust-integration --release): the whole suite on the SQLite release build with the defaulted root and EXTENDDB_TEST_BACKUP_STORE=filesystem, 512 passed, and afterwards no complete backup under the root (find <root> -name extenddb-manifest.json finds none; extenddb-store.json alone); the backup and restore_active_completeness filters on the PostgreSQL release build without a store and none: 18 and 2 passed, the restore round trip taken. With the variable unset the restore cell fails with EXTENDDB_TEST_BACKUP_STORE is unset; set it to one of ["filesystem", "s3", "none"] ..., and with none against the filesystem server it fails on the store's ValidationException.
• tests/test_backup_store_config.py and tests/test_backup_job.py (7 cells) by name on this head: on the PostgreSQL release build 11 passed (feat(backup): [backup.store], the store identity file, backup purge [backup/restore 7] #672's exit-code cell among them), 12 skipped (the SQLite class); on the SQLite release build 19 passed, 4 skipped (the PostgreSQL class) with moto reachable at 127.0.0.1:9200 for the S3 cells. feat(backup): [backup.store], the store identity file, backup purge [backup/restore 7] #672's in-memory cell asserts this PR's create there: with [backup.store.filesystem] path set the backup is CREATING, then AVAILABLE with its manifest under the root, and its restore is refused naming the store until PR 9a; without [backup.store] the in-catalog create warns. Before the rebase, the fidelity, ABAC and auth-permissions suites against a server of each release build with a filesystem store and EXTENDDB_TEST_BACKUP_STORE=filesystem: 144 passed, 25 skipped on each, the fixture printing backup store under test: {'kind': 'filesystem', 'status': 'healthy'} and the vector cell its filesystem branch; the ABAC cell's backup deleted.
• Before the rebase, kill points (kill-points.sh, a server built with --features extenddb-app/backup-test-hooks, reconcile_interval_secs = 2, stale_job_after_secs = 45, a 100k-item composite-key table with two GSIs; the SIGTERM point on a 300k-item table without GSIs), on SQLite with the defaulted root, on PostgreSQL with a filesystem store, and on the SQLite build with an S3 store on moto: after_data_file: after the SIGKILL the prefix holds data/000001.json.gz and no manifest, the row CREATING; after the restart the row is FAILED owner dead ... after 0 s on SQLite and the S3 store (the slot rule, the first cycle) and 46 s on PostgreSQL (the 45 s window), DescribeBackup BackupNotFound, the prefix deleted 45 s after the kill on SQLite and S3 (the dead owner's heartbeat stale) and in the same cycle on PostgreSQL; before_manifest (run on SQLite and PostgreSQL): the data file and the three documents (manifest-files.json, manifest-summary.json, extenddb-manifest.sha256) under the prefix after the kill and no manifest, the row CREATING, then FAILED owner dead ... after 0 s on SQLite and 47 s on PostgreSQL, the prefix deleted 46 s after the kill on SQLite and 48 s on PostgreSQL (its cycle), so a prefix without extenddb-manifest.json is never resolved AVAILABLE whatever else is under it; after_manifest: the five objects under the prefix after the kill, the row AVAILABLE after 0 s on SQLite and S3 and 47 to 48 s on PostgreSQL with the manifest's size (847,689) and count (100,000), the prefix whole, DescribeBackup AVAILABLE; sigterm: waiting up to 30 s for 1 backup job(s) to end, FAILED (shutdown); its objects were deleted within 15 ms, drained, the row FAILED shutdown, the prefix empty, the process gone 55 ms after the signal (polled at 50 ms; the first record's "5 s" was the script's own sleep), and on the S3 store list-multipart-uploads empty. The first SIGTERM runs on SQLite found the job spinning through the whole 30 s drain with its row left CREATING: the chunk sink answered the encoder's write after the stop with Interrupted, which flate2's writers retry; it answers BrokenPipe now (writer.rs, two unit tests), and the three stores pass on the rebuilt binary.
• Before the rebase, memory (bench-rss.sh, the release binary under a VmHWM sampler at 200 ms, restarted after each load so the high-water mark covers the backup alone; 16 cores, 30 GiB, PostgreSQL 16.15, bundled SQLite 3.46 via libsqlite3-sys 0.30.1, MongoDB 7.0.43; BENCH-pr8.md has every figure and caveat): 100k and 1M items of a composite-key table with two GSIs: peak over idle 11.3 and 20.6 MiB on PostgreSQL (+9.5 MiB), 45.0 and 50.5 MiB on SQLite (+7.9 MiB; the 45 is the pool's SQLite page caches), 9.6 MiB at 100k and 15.5 MiB at 500k on MongoDB (+4.8 MiB; the 1M load would have taken close to an hour through the wire, so 500k stands in its place), each under the 112 MiB bound. Durations: 1.85 s and 36.1 s on PostgreSQL (a loaded host), 1.44 s and 12.45 s on SQLite, 1.85 s and 8.81 s (500k) on MongoDB. The incompressible run (64 MiB of random B attributes, a 68 MB backup gzip did not shrink): 120.5, 74.8 and 90.2 MiB over idle, so the encoder streamed the file through its 4 MiB chunk window. The worst-expansion run (1,000 small rows then 12 of feat(core): backup format v1: streaming data files, sidecar manifest hash, typed backup ARN [backup/restore 3/4] #394's 27.7 MB worst lines, three to a page) at the first head: 880 MiB over idle on PostgreSQL, 582 to 762 MiB on SQLite across runs, 474 MiB on MongoDB with 13.5 MB lines (its 16 MiB document cap refuses the full shape). These peaks are not the design's earlier formula ("the budget plus one row of text, plus its decoded form"): the page's decoded rows, the items in the feeder, the four-item channel and the encoder, one row's parse through the deserializer's intermediate value, the line buffer, the chunks and on SQLite the driver's five-row run-ahead are terms of their own (545 MiB for this shape on SQLite, about 490 MiB per job on PostgreSQL and 650 MiB on SQLite at the 32 MiB line cap), and what glibc's allocator keeps on top of them depends on how many large rows the job reads: the series at 12, 24 and 48 such rows on one SQLite binary peaked at 466, 588 and 760 MiB over idle on a quiet host and 707, 700 and 1,281 MiB on the final build with the kill points running beside it (0.86 to 2.35 of the terms, growing with the rows), and under MALLOC_ARENA_MAX=1 at 481, 452, 480 and 452, 480, 455 MiB (0.83 to 0.88 of the terms, flat), so section 7 states the terms as a sizing rule (the terms at the line cap, times 1.2 under one arena, times max_concurrent_jobs, at the default max_item_size_bytes), the admin guide carries MALLOC_ARENA_MAX=1 for hosts that back up large items and the service unit the line commented out, and the acceptance bounds the one-arena peak at 1.2 times the terms and records the default-arena peak beside it; the record is docs/design/15-backup-restore-bench-pr8.md. On PostgreSQL the build with the row bound from the largest row and the rows parsed as they arrive took the worst shape to a VmHWM of 790 MiB in 16.5 s against 905 MiB in 18.2 s, its 1M run +4.0 MiB over the 100k run (13.3 s), and the rows read past the cut in the paging test are 8 over 77 pages, all on the first page (602 at the head); the same shape under MALLOC_ARENA_MAX=1 on the final PostgreSQL build peaked at 340 MiB VmHWM against 750 to 788 MiB under the default arenas (0.82 of the terms at that row on PostgreSQL, 413 MiB), so the one-arena factor is measured on both backends. The setting's cost on the request path (a 50k-item table, one boto3 client per thread, three alternating repeats of 20 s per operation): GetItem −3.8%, Query −3.5%, Scan −6.9%, paged Scan −9.2% on SQLite; +1.1%, −0.6%, −2.1%, −2.1% on PostgreSQL. The row bound's memory on the release binary (two 20,000-row tables with 30 worst-expansion items, scattered among the first 1,000 keys and grouped after them, both binaries back to back): the interleaved table's backup 54.6 s with 3.18 GB of server temp files against 72.0 s and 9.21 GB with one page of memory, the grouped table 38.9 against 39.8 s and 2.85 GB both; the 1M run 13.7 against 12.1 s (VmHWM 36.7 against 36.3 MiB, +4.1 MiB over the 100k run; the memory changes no page of a small-row table, and the tree's binary carries the validator's and feeder's changes of the same day). The server-side walk that was measured and not taken: 38.8 s and 2.51 GB interleaved, 38.0 s and 2.51 GB grouped, 23.07 s against 11.85 s for the 1M run, 13.4 s and 658 MiB against 17.2 s and 553 MiB for the worst shape. A one-item channel was measured and not taken: 50 MiB less at the worst-shape peak for a 1M-row backup 75% slower (12.4 s to 21.6 s, both orders). The SQLite pool's row_buffer_size is set to four (store.rs; sqlx's default of 50 let the driver's worker read every row of a 24-row worst-shape table ahead of the first page, 844 MiB over idle against 696 with four, back to back; a 1M-row backup took 11.4 s with either).

cargo fmt --all -- --check
cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings
cargo +1.97.0 clippy --locked --workspace --all-targets --features extenddb-engine/backup-test-hooks -- -D warnings
cargo +1.97.0 clippy --locked -p extenddb-storage-cassandra --all-targets --features test-hooks -- -D warnings
CARGO_TARGET_DIR=target/t188 cargo +1.88.0 check --locked --workspace --all-targets
cargo doc --no-deps --workspace
cargo check --locked -p extenddb-storage --lib
devtools/generate-software-license-notices --check [--backend mongodb|cassandra]; devtools/generate-dev-license-notices --check
EXTENDDB_TEST_PG_REQUIRED=1 ... cargo test --locked --workspace --exclude extenddb-storage-cassandra --exclude extenddb-storage-mongodb --exclude extenddb-backup-store
EXTENDDB_TEST_S3_REQUIRED=1 ... cargo test --locked -p extenddb-backup-store
EXTENDDB_TEST_MONGODB_REQUIRED=1 cargo test --locked -p extenddb-storage-mongodb -- --skip a_standalone_server_is_unsupported; ... --lib -- --ignored --test-threads=1 live_
EXTENDDB_TEST_CASSANDRA=required cargo test --locked -p extenddb-storage-cassandra --lib | --features test-hooks --test direct_integration
taskset -c 0-1 nice -n 5 cargo test --locked -p extenddb-engine --lib -- backup::tests   # 20 runs, beside one busy loop

Not changed here

• The restore of a store-backed backup (PR 9a): RestoreTableFromBackup of a row of the configured store answers ValidationException "Backup is in a backup store; this release restores in-catalog backups only"; the fidelity suite's restore cells skip on a server with a store and come back with it; the vector wire test asserts that refusal where PR 9a asserts the round trip.
• The spool sweep and the restore_progress sweep (PR 9a adds the records they read); the duty that removed a CREATING restore target with no job row is withdrawn (design item 31).
• backup sync and its options, remove-target, migrate-legacy (PR 10); export and import through the store (PR 13); the legacy reader's removal (PR 14).
• The restore metrics of section 9 (restore_jobs_total, restore_job_duration_seconds) are PR 9a's.
• The in-catalog create (no store, Cassandra) is unchanged: synchronous, vector tables refused, the deprecation warning on each call.
• The PostgreSQL page's plan near the end of a small table is the planner's (a bitmap scan and a Sort of the rows after the seek, pinned by a test and stated in the module doc); the query is not changed to force the index scan. A page query that stops at the budget on the server (a recursive walk of the primary key, one index probe per row) was measured and not taken: it reads no row past the cut on any shape and costs the 1M-row backup 95% of its duration; the record carries its figures, and the window form's bound is stated as it is.

Checklist

• I have read CONTRIBUTING.md
• All tests pass (cargo test --workspace), the pre-existing Cassandra TTL failure excepted (the review round ran the touched crates' suites and the named backend tests; CI runs the rest on push)
• Code is formatted (cargo fmt --check)
• Clippy is clean (cargo clippy --workspace --all-targets -- -D warnings)
• I have added or updated tests for new functionality
• I have updated documentation if behavior changed
• Breaking changes are noted below (if any)
• If this changes the wire protocol, Storage trait, auth model, on-disk format, or public CLI surface, an RFC has been accepted or is linked below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: #396, sections 4(a), 4(c), 4(e), 4(f), 5.1, 7, 9 and the PR 8 section, revision items 31 and 32. Storage trait: BackupCatalog gains claim_dead_backup and sweep_restore_registrations (the second with a default body); JobClaim gains heartbeat_within (a provided method); StatusFields gains source_table_details_json; SnapshotEngine is unchanged. No wire protocol or on-disk format change (the data files and manifests are #394's).

Breaking changes

• With a configured [backup.store] on PostgreSQL, SQLite and MongoDB, CreateBackup returns the backup CREATING and it becomes AVAILABLE once the upload ends, as on the service; while it is CREATING, which lasts the whole upload, DeleteBackup and RestoreTableFromBackup of it answer BackupInUseException "Backup is being created: " (the service's backups measured were AVAILABLE 12 to 90 ms after CreateBackup), so a client that deletes or restores a backup right after creating it polls DescribeBackup first. The backup is the table at its snapshot's open, which comes after the request's checks and after any wait for a job permit (up to 30 s), so a write acknowledged during that wait is in the backup. On file-backed SQLite the defaulted root makes this the default behaviour. The restore of a store-backed backup is feat(backup): RestoreTableFromBackup job and cross-backend matrix [backup/restore 9] #740's, which merges with this PR (Merge order above): a main carrying this PR alone would refuse RestoreTableFromBackup of the backups it takes with ValidationException, so the two land together; the usage guide's backups row and the upgrade manual describe the pair.
• DescribeBackup's ItemCount and TableSizeBytes are the stats pass's figures while CREATING (0 and 0 for a table it has not reached) and the stream's once AVAILABLE; the previous release reported the stats pass's for both.
• CreateBackup is refused with LimitExceededException while backup purge holds the maintenance claim, checked before the permit, so no FAILED row is written for the refusal (the previous head wrote one with reason maintenance).
• CreateBackup writes no row when the server's shutdown meets its snapshot open or row insert; its ServiceUnavailable answer reaches the client only when the jobs' token is cancelled while the listener still serves (a registration loss), since on SIGTERM the listener's 5 s drain closes the connection first.
• A registration lost while serving cancels the running backup jobs (each fails its row with reason shutdown and deletes its objects); the jobs started under the new registration run on.
• [backup] failed_row_retention_secs must be at least stale_job_after_secs; a smaller value is refused at startup.
• On SQLite on a platform where the serve lock is not taken (not Unix), CreateBackup writes the in-catalog format whatever [backup.store] says, and the store serves restores and deletes.
• CreateBackup is refused with ValidationException naming the store's kind while a configured store is unhealthy or its first check is pending (PR 7 refused only the restore and delete).
• CreateBackup of a table with vector indexes succeeds on the store-backed path (refused as before without a store).
• A backup of a table holding an N value the wire did not validate (BatchWriteItem, TransactWriteItems and other operations store N values that fail number validation as written #618) fails on the store-backed path where the previous release backed it up; the reason reaches the operator through extenddb backup list --failed and the log.
• On SQLite a server raises [storage.sqlite] pool_size to 2 * (max_concurrent_jobs + 1) with a warning when it is smaller (no change at the defaults).
• Out-of-tree storage backends must implement claim_dead_backup, write StatusFields.source_table_details_json in JobClaim::set_status, and may override sweep_restore_registrations and JobClaim::heartbeat_within (a backend whose beat queues behind a lock of its own process places the bound after the lock, as SQLite does). BackupRuntime::shutdown returns a token (the jobs' current one) rather than a reference, and into_configured_store returns the Arc the jobs share.
• devtools/run-tests --extenddb --pytest requires EXTENDDB_TEST_BACKUP_STORE (filesystem, s3 or none).
• extenddb stop waits up to 76 s for the server to exit (it waited 10 s); a supervisor that runs it, or a container runtime's grace period, should allow that long for a server that may be running backup jobs. The repository's compose files set stop_grace_period: 80s (they set 15 s). A dev-mode build's plain-HTTP listener drains for at most 5 s, as the TLS listener does.

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0 and I agree to the Developer Certificate of Origin (DCO). See CONTRIBUTING.md for details.

[LeeroyHannigan]

I built this head with extenddb-app/backup-test-hooks for SQLite, PostgreSQL and MongoDB and ran it in containers with a filesystem store. The runs included SIGKILL after a job's first data file and after its manifest, SIGTERM during jobs and a 160 s freeze. Two servers also raced for the same dead rows, backup purge ran beside clients creating backups, and I swapped the store root during one job. I sent the backup requests to DynamoDB in us-east-1, to this head on the three backends and to #672's head. Mutants of the job, the reconciler, the item validator and the backends ran against the PR's tests. CI passes on this head (37 checks in 10 workflow runs), and every new Rust test runs there except the MongoDB takeover test.

The CreateBackup response, and DescribeBackup and ListBackups of a CREATING backup, match the service key for key on all three backends. A SIGTERM during a job failed its row and deleted its objects within 10 ms, and the server exited 0.15 s after the signal. Two servers racing for a killed server's 12 rows resolved each row once, on MongoDB and on PostgreSQL. On PostgreSQL, client timeouts of 1 to 43 ms on 1,200 creates left no CREATING row and no FAILED row with objects.

A PostgreSQL backup whose job ran while four writers put and deleted keys contained exactly the 2,500 items the table held before the request. A backup #672's head made on a SQLite file still restores after the upgrade to this head. DeleteTable and UpdateTable during a job, the item validator's limits and log, the SQLite pool raise and the job bound behave as the description says.

Two Blocking items and 28 more are inline. One item comes from #672:

• The store check's 5 s bound covers its catalog reads, so a slow catalog marks the store unhealthy for one cycle (60 s by default). With this PR the server also refuses CreateBackup while the store is unhealthy. I saw the store unhealthy for 2.4 s on MongoDB at a host load of 45.

Once you fix the inline items, I will approve.

[LeeroyHannigan]

Blocking: this makes CreateBackup store-backed whenever a store is configured, and file-backed SQLite has one by default, but RestoreTableFromBackup refuses those backups until #740. On SQLite by default, and on PostgreSQL and MongoDB with a store, a restore right after CreateBackup answered ValidationException Backup <arn> is in a backup store; this release restores in-catalog backups only in all 8 tries. DynamoDB accepted that restore 0.6 s after CreateBackup, and #672's head, like main, restored all 100 items. Breaking changes lists this. Please merge this PR with #740, with no main commit between them, or disable the store-backed create until #740.

[LeeroyHannigan]

Blocking: on SQLite this beat is an UPDATE under the engine write lock, so a write holding that lock past one period ends a running backup. Beside UpdateTable building a GSI on a 400,000-item table, one backup failed here (the heartbeat did not return within 15 s). Another backup failed at the claim re-check in job.rs line 515 (the claim re-check did not return within one heartbeat period), after its uploads, and the reconciler then deleted its 5 objects. The in-catalog create that main runs, here with kind = "disabled", finished AVAILABLE in 98.2 s beside the GSI build. Please bound the beat and the re-check only once they hold the lock. Add a test whose writer holds the lock past two periods.

[LeeroyHannigan]

The 60,000-item race from my comment now ends AVAILABLE. With 180,000 items the backup stopped twice at the check before data file 3 or 4. Both times a GSI build on the 400,000-item table held the write lock. The reconciler then failed each row and deleted its objects, and the in-catalog create of the same table ended AVAILABLE. Please let confirms_the_claim pass while the beat waits for the write lock, and add a test that puts a data file during the hold.

[robinnsc]

Fixed. HeartbeatHandle::confirms_the_claim has a third clause: a beat that is in flight and has not failed confirms the claim, however old the last confirmed beat is. The timer and recheck set an in-flight flag around every heartbeat_within call and clear it through a drop guard on the way out (a beat that returns Lost, fails or outlives its bound raises the loss first, and the check reads the loss before the flight; a timer that stops mid-beat clears the flag, so the two-period clause remains for it). The module doc states why it is safe: on SQLite the beat parks only on this process's write lock, one process serves the file (backup purge refuses while a server runs on it) and the reconciler's liveness test is the in-process slot, so a parked beat cannot coincide with a takeover; on PostgreSQL, MongoDB and Cassandra no process-local lock sits in front of the beat and the bound covers the whole call, so a beat is in flight for at most one period after its tick and the clause admits no put the two-period rule refuses. Tests: write_lock_tests::a_put_during_a_writers_hold_passes_on_the_beat_parked_behind_it (storage-sqlite) backs up three items at a data_file_target_bytes of one byte through a store whose first put takes four periods, takes the write lock as the row appears and holds it until the second data file's put has started, so that put's check runs with the first beat parked and no beat confirmed since the insert (asserted: heartbeat_at unchanged under the hold); after the release the backup is AVAILABLE with its three data files and the manifest, and heartbeat_at has moved. review_round_3::a_beat_parked_on_the_write_lock_confirms_the_claim_until_it_returns_lost (engine) drives the handle with a fake claim whose heartbeat_within parks outside the bound: the check stays true at two and a half and at twelve and a half periods, turns false the moment the released beat returns Lost, and after a parked beat released with a confirmation and the timer stopped, turns false two and a half periods later. Mutant: with the in-flight clause removed, the SQLite cell fails (one put started under the hold, the row left CREATING) and the engine cell fails at its first assertion; both pass after the revert.

[LeeroyHannigan]

This awaits the encoder before line 562 cancels failed, and an encoder in items.blocking_recv() returns only when the feeder reads the stream's next item. A test that holds the stream between items and fails a mid-file put found the row CREATING and the job running 3 s later. With failed.cancel() before this await and the one at line 495, as the module doc describes, the row was FAILED and the job ended. With that order, 156 backup::tests passed. Please make that change. Add the mid-file case to a_put_failure_ends_the_job_without_waiting_for_the_streams_next_item, which passes either way because MemStore::put fails after reading the whole body.

[LeeroyHannigan]

I paused a job after its first data file and swapped the root for one whose extenddb-store.json names another catalog. The reconciler marked the store unhealthy, yet the job wrote its documents and this manifest into that root, and the row went AVAILABLE. With the original root back, the store stayed unhealthy after a restart, and CreateBackup and DeleteBackup answered ValidationException The filesystem backup store configured on this server is unhealthy (manifest probe failed); the operation is refused until the store's check passes, so no request clears it. Please check is_healthy() and that the identity equals env.identity before each put. When either check fails, fail the row with the store's reason and no store call.

[LeeroyHannigan]

A create refused here leaves a FAILED maintenance row with the insert's heartbeat, and backup purge counts such a row as a live job for stale_job_after_secs. With four clients creating backups and stale_job_after_secs = 45, all 8 purge attempts exited 1 after 90 to 93 s. Each attempt named such rows or rows inserted seconds before. Those runs left 16,054 FAILED maintenance rows, kept 7 days by default, and with them a restarted server's first reconciler cycle lasted over 4 minutes. Please refuse before the insert as the in-catalog path does, or write the refused row so the wait ignores it.

[LeeroyHannigan]

With >= changed to >, both pool tests pass, because the pools they size exactly are even. pool_size_for_jobs(11, 4) would then return 10, a pool lowered by one. Please add that case.

[LeeroyHannigan]

No CI job builds or lints this feature, and kill-points.sh is outside the repository (design line 467). The pause code in job.rs lines 607 to 640 and the description's kill-point results depend on hand runs that CI cannot repeat. Clippy on 1.97.0 and a 1.88.0 check with the feature pass on this head today. Please add the feature to the clippy job. Commit the script under devtools/.

[LeeroyHannigan]

This says the backup is stricter than the service's one-minute window. In one DynamoDB run, the restored backup contained every write acknowledged before I sent CreateBackup. It also contained the 18 writes acknowledged in the 0.1 s up to BackupCreationDateTime. It contained none of the 11,754 acknowledged in the next 75 s, so the service took the table at an instant inside the request. Please replace the clause with that measurement, or cite the source of the one-minute window.

[LeeroyHannigan]

The AVAILABLE write sets ItemCount from the stream and leaves TableSizeBytes at the statistics figure. On SQLite the backup of a fresh 1,000-item table reports 1000 and 0, as on MongoDB, and on PostgreSQL it reports 1000 and 1359872. DynamoDB reported 0 and 0 in every fresh-table backup I measured, and #672's head on SQLite, like main, reported 100 and 5989 for 100 items. Please set TableSizeBytes from the stream as well, or keep both at the snapshot's figures. Also name TableSizeBytes in the differences row.

[LeeroyHannigan]

While a backup is CREATING, which now lasts the whole upload, DeleteBackup and RestoreTableFromBackup of it answer BackupInUseException Backup is being created: <arn> on all three backends. All 16 DynamoDB backups I measured were AVAILABLE 12 to 90 ms after CreateBackup answered, and #672's head, like main, answered both requests at once. Please add both refusals and how long CREATING lasts to this row.

[robinnsc]

Pushed b20e19ec: the job-bound cell of tests/test_backup_job.py rewritten to the contract #671's review round set. A second CreateBackup while the one slot is held now waits up to 30 s for it rather than refusing at once, so on every store job the second request was admitted when the first job ended and the cell's immediate LimitExceededException expectation was the earlier contract. The cell (test_second_create_backup_over_the_job_bound_waits_for_the_slot) sends the second request from its own thread while watching the first backup, asserts it is answered only once the first is AVAILABLE and then succeeds, and keeps the refusal at the bound as the branch for a first job that outlives the wait; the design's PR 8 acceptance says the same. The MongoDB item factor from the previous push is withdrawn (it was not the cause). Ran on a SQLite release build here: the second waited 0.4 s, the first seen CREATING 8 times meanwhile.

[LeeroyHannigan]

I checked the items from my last review at 141f911. I also read the diff to b20e19e, which changes only the job-bound cell and one design line.

You fixed the write-lock case. I ran three SQLite races, each a 180,000-item backup beside a GSI build on the 400,000-item table. All three ended AVAILABLE, and the reconciler kept every object. Without the in-flight clause, both new cells fail.

You also fixed the snapshot row. The code docs, the differences row and the admin guide now state the ServiceUnavailable case. My reply on its thread covers the one sentence left in the description.

The in-flight clause brings one Major item, inline. On PostgreSQL, MongoDB and Cassandra it admits puts for a short time after a process stop, and the module doc says it never does. The other new items are Minor.

At 141f911 the job-bound cell failed in three CI jobs, because the second request waits for the slot. Your rewrite passed in mongodb-backup-store and run-integration-sqlite-s3. The other jobs that run it were still running when I checked.

Once you bound the clause on those three backends, I will approve.

[LeeroyHannigan]

Major: the module doc says that on PostgreSQL, MongoDB and Cassandra this clause admits no put that the two-period rule refuses. The clause reads only the flag, and a beat in flight when the process stops keeps it set after the resume.

I stopped a test process for 1.5 s with kill -STOP while a bounded beat was in flight. After kill -CONT, checks passed for 64 µs and 85 µs in two runs, until the timer raised the loss. Without the clause, no check passed.

On MongoDB and Cassandra another owner can claim the row during the stop. Please apply the clause there only while the beat is inside its bound.

[robinnsc]

Fixed: the in-flight clause is bounded where the beat does not park. JobClaim gains a provided heartbeat_parks_before_its_bound (default false; SQLite's claim answers true, its beat waiting behind the engine write lock of its own process, outside heartbeat_within's bound), the handle records the beat's start (in_flight: Mutex<Option<Instant>> instead of the flag), and in_flight_confirms is parks || since.elapsed() <= period: a beat in flight confirms the claim for as long as it waits on SQLite, and within one period of its start on PostgreSQL, MongoDB and Cassandra, so a process stopped mid-beat and resumed there passes no check in the window before its timer raises the loss (your 64 and 85 µs). review_round_3::a_beat_in_flight_past_its_bound_confirms_the_claim_only_where_the_beat_parks runs the same parked beat under both declarations: within the bound both confirm; past it only the parking one; and when the beat returns Lost the loss is read first. With the clause unbounded again (parks || true) its parks=false half fails. The module doc now states the rule per backend, and design 5.1 step 4, item 32 and the admin guide's heartbeat_interval_secs row say the same. #740's ClaimCheck takes the same parks so the restore handle's clause is bounded there too.

[LeeroyHannigan]

The Engine paragraph of the PR description still says that on the other backends the clause admits no put that the two-period rule refuses. The module doc no longer says this, so please update that sentence too.

[robinnsc]

The Engine paragraph's sentence is updated. It now says that on PostgreSQL, MongoDB and Cassandra the bound covers the whole call, so the two-period rule already covers a beat in flight and the clause does not apply, and that a process stopped mid-beat or between beats would otherwise pass checks while its beat runs.

[LeeroyHannigan]

The Engine paragraph now says the clause doesn't apply on PostgreSQL, MongoDB and Cassandra, where the two-period rule covers a beat in flight. At 57cfe20, in_flight_confirms reads the flight only when the claim's beat parks, and only SQLite's claim declares that.

[LeeroyHannigan]

Minor: without the in-flight clause, this cell fails with one put started and the row left CREATING, because no reconciler runs here. The doc says the row ends FAILED and its prefix is empty, so please describe what the cell sees.

[robinnsc]

Fixed. The cell's doc now says what the cell sees without the clause: one put started and the row left CREATING, since no reconciler runs there (on a server the reconciler fails the row and deletes its prefix).

[LeeroyHannigan]

Minor: this comment still describes the LimitExceeded cell. In the new cell, a second request admitted with the first backup AVAILABLE is the passing branch, and a refusal is the slow-host branch. Please update it to match.

[robinnsc]

Fixed. The comment now describes the slot-wait cell: the second request is sent while the first job holds the one slot and waits for it; the passing branch is the second admitted once the first job ended (the first AVAILABLE at its answer and seen CREATING while it waited); a second answered while the first is still CREATING is a broken bound; a refusal at the wait's bound is the slow-host branch; a first backup already AVAILABLE before the second is sent is a probe that came too late, where the cell asks for more items.

[LeeroyHannigan]

Minor: the cell reads the first backup's status before the thread builds the second client and connects. If the first job ends in that gap, this branch passes and the second request never waited. Please assert that seen_creating is above zero here, so a late probe fails as it did before.

[robinnsc]

Fixed. The admitted branch asserts seen_creating > 0, with the message naming EXTENDDB_TEST_BACKUP_BUSY_ITEMS, so a first job that ended before the thread connected fails the cell as the late probe did before.

[robinnsc]

The Major and the four Minors are fixed as above (head c461442a, on #672's ba6b82d4 and #671's 6ff37487). Engine backup:: 198 passed on this head, storage-sqlite write_lock_tests 15, clippy 1.97 clean with the Cassandra and MongoDB test-hooks and the engine's backup-test-hooks features.

[LeeroyHannigan]

I checked the items from my last review at c461442. You fixed all of them.

I reran the stop probe with a beat in flight at the stop, and no check passed after the resume in three runs. With the clause unbounded, 15 and 114 checks passed. With SQLite's declaration, the check still passes while a beat waits, and the SQLite cell passes in a container. The parks || true mutant fails a_beat_in_flight_past_its_bound_confirms_the_claim_only_where_the_beat_parks. The three Minor texts read as asked, you qualified the Engine paragraph, and the job-bound cell passed in all four CI jobs that run it.

One Major item is inline. A stop that falls between beats still lets checks pass after the resume, for the round trip of the beat that the missed tick starts. That path was already there at b20e19e, and the bound I asked for does not cover it. So it is not a condition for this approval. The simpler rule is to use the clause only where the beat parks. Please fix it in this PR, or open an issue for it before the stack merges.

My reply on the clause's thread covers one sentence left in the description.

[LeeroyHannigan]

Major: after a stop that falls between beats, this clause still passes checks that the two-period rule refuses. On resume, the missed tick starts a beat at once, and every check passes during its round trip.

I stopped a test process for 1.5 s between two 5 ms beats, and the first beat after the resume answered Lost. Checks then passed for 5.2 to 6.2 ms after the resume, though the previous beat returned eight periods before them. Without the clause, no check passed.

The module doc still says that a row that passes the check "is never one a reconciler may take over". On PostgreSQL, MongoDB and Cassandra the two-period rule already covers a beat in flight, so only a claim whose beat parks needs the clause.

[robinnsc]

Fixed as you suggest: the clause applies only where the beat parks. in_flight_confirms is now parks && <a beat is in flight>, and the handle's mark is a flag again, since the beat's start is no longer read. On PostgreSQL, MongoDB and Cassandra a beat in flight confirms nothing, and the check is the two-period rule alone.

New cell a_process_stopped_between_beats::a_stop_between_beats_passes_no_check_while_the_resumed_beat_runs, under both declarations:

• Three beats confirm on time.
• The clock moves eight periods with no task of the runtime running (tokio::time::advance), so the timer misses its ticks.
• On resume, the missed tick starts a beat that has not returned. Where the beat does not park, no check passes at 1 ms or 6 ms into it. On SQLite's declaration the check passes.
• The beat answers Lost, and both refuse.

It fails with the third round's bounded clause (parks=false: after a stop between beats, the beat the missed tick started confirms the claim ...) and with the unbounded one. The third round's cell is now a_beat_in_flight_confirms_the_claim_only_where_the_beat_parks: a beat within its bound no longer confirms where it does not park, and it fails with the bounded clause too.

The module doc's takeover sentence now says that a row passing the check by the two-period rule is never one a reconciler may take over, and that the third clause applies only on SQLite, where no reconciler takes over a row whose job runs in its process. The clause's paragraph names both windows: a stop mid-beat, and a stop between beats with your 5 to 6 ms. The trait doc on heartbeat_parks_before_its_bound, design 5.1 step 4 and item 32, and the admin guide's heartbeat_interval_secs row now say the same.

[LeeroyHannigan]

I stopped a test process for 1.5 s between two beats at 57cfe20, and no check passed after the resume in five runs. With in_flight_confirms reading the flag alone, checks passed for 5.2 to 6.2 ms. The new cell fails with the flag alone and with heartbeat.rs from 33a8e9c.

[LeeroyHannigan]

My last review approved this PR with a Major item open. That item blocks approval, so I am changing my review to request changes.

After a stop that falls between beats, the clause at heartbeat.rs:129 passes checks for 5 to 6 ms after the resume. Another owner may hold the row by then. On PostgreSQL, MongoDB and Cassandra the two-period rule already covers a beat in flight, so only a claim whose beat parks needs the clause. Please apply the clause only there. Then add a test that stops the process between beats and fails without that change.

The restore checks in #740 call the same function, so this fix covers them too. Once it is in, I will approve.

[robinnsc]

The Major is fixed as above (head 57cfe208, now on main at 71bcf55a, with #671 and #672 merged under it). Engine backup:: 199 passed on this head, storage-sqlite write_lock_tests 15, cargo check --workspace --tests and clippy 1.97 clean with the three test-hooks features. #740's ClaimCheck calls the same in_flight_confirms, so the restore checks take this rule when #740 is rebased onto this head.

[LeeroyHannigan]

I checked the two items from my last review at 57cfe20. You fixed both. in_flight_confirms applies the clause only where the beat parks, which is SQLite's claim alone.

I stopped a test process for 1.5 s between two beats. No check passed after the resume in five runs, with 5 ms and 20 ms beats. With the flag alone, checks passed for 5.2 to 6.2 ms.

When I stopped the process mid-beat, no check passed after the resume in three runs. The flag alone let 11 and 48 checks pass within 80 µs.

I ran beats that take 0.9 of a period, and the check refused nothing. That held over 40 periods of 50 ms in real time and 12 periods in paused time.

The new cell fails with the flag alone and with heartbeat.rs from 33a8e9c. With no clause, the parks = true halves fail, and so does SQLite's parked-beat cell. SQLite's write_lock_tests pass at 57cfe20. The engine's 199 backup tests pass on 1.88.0, rustfmt and clippy on 1.97 are clean, and CI shows 50 of 50.

Design 5.1 step 4 (docs/design/15-backup-restore.md:120) still says the check refuses a first put more than two periods after the insert "with no beat in flight". On PostgreSQL, MongoDB and Cassandra it refuses that put with a beat in flight too. That's not a condition for approval.