From 88d445dad9288048a20921fbce51b1b1b1e55fba Mon Sep 17 00:00:00 2001 From: Scott Robinson Date: Fri, 6 Nov 2026 05:19:42 +0000 Subject: [PATCH] docs(backup): the backup and restore manual, the open measurements The integration workflow's PostgreSQL and SQLite jobs ran the wire suite with `--filter "not import_export"`. The filter came with the PostgreSQL job's first version (687e46ff) and was copied into the SQLite job (556c501e); neither commit gives a reason. A foreground server wrote a PID file in every tree up to e0d6a0af, which made foreground mode write none by default and added `--write-pid-file` to the job in the same commit; 556c501e came after e0d6a0af. So the restart `devtools/run-tests` makes for the import and export tests could stop the job's server in every tree, and the filter was not working around it. The MongoDB and Cassandra jobs already run `tests/test_import_export.py` unfiltered. The expression selects out that file's ten tests and nothing else. Run the job's way on release builds of this tree they pass 10 of 10 on SQLite and on PostgreSQL, so both filters go and no narrower one stays. `docs/design/15-backup-restore-bench-pr11.md` (row 15d of the design index) records the cells the testing summary left open, in the PR 9 record's method: release builds, a filesystem store, the server restarted before each phase, `VmHWM` sampled every 200 ms, on the 16-core host class of the earlier records. 1M items of 200 bytes restore in 33.8 s on MongoDB (its own backup) and 49.0 s on Cassandra (a SQLite backup adopted with `backup sync`), 52 and 33 MiB over idle. A 1M-item table with a 128-dimension vector index backs up in 243 s on PostgreSQL and 210 s on SQLite and restores in 246 s and 214 s, 111 MiB and 552 MiB over idle; SQLite's figure is higher than at 100k under glibc's default arenas and is 204 MiB, flat from 100k, under `MALLOC_ARENA_MAX=1`, where the backup took 222 s and the restore 283 s. The Cassandra cells ran on a node of their own with the Cassandra workflow's service settings. No cell is left open. Later runs on the same host, also in the record, reproduce the cells and show that SQLite's restore memory is its write batch and decode chunks, not the vector index: it grows with the item's size up to the 32 MiB batch bound whatever the attribute types (253 MiB, 130 MiB under one arena, for 100k items of 2.5 KB strings), and lists of numbers decode larger (397 to 424 MiB for the same 100k items of 128 numbers with no index, 301 to 316 MiB under one arena for 20,000 items of 1,536 numbers). Two restores at once under one arena took 57.5 s against 24.9 s. Five pairs of PostgreSQL runs with and without the vector index, each pair from one harness, put its share of the 100k restore at 27 to 53 MiB and 33 to 47% of the duration. A MongoDB restore becomes ACTIVE at the backfill worker's next 5 s cycle after its load, so it takes the load's time plus up to 5 s. `docs/manuals/14-backup-and-restore.md` is the operator's manual, derived from the design: the strategy (its first sentence: a filesystem store keeps the host's failure domain, S3 is the production recommendation), RPO and RTO per store kind with the three records' figures and host class, the configuration (pointing at the admin guide's rows and the store crate's README), securing the store, what a running backup costs each database, a restore drill, catalog-loss recovery with `backup sync`, `--adopt-store` and the in-memory startup sync, legacy backups and what to do with them while `migrate-legacy` (#821) is not shipped, what a restore preserves, and moving data to and from Amazon DynamoDB with ImportTable and export (a store-prefix import is #748). It states each deployment statement the design's PR 11 acceptance lists. Figures measured for it, with their method in the record's "Figures measured for the manual": a 25 s PostgreSQL backup beside 12,156 overwrites of 200-byte items grew the written table from 0.4 to 4.4 MiB, its dead row versions not removable by a `VACUUM` during the backup (0.5 MiB with no backup); on SQLite the WAL grows by the pages each write changes, from 4.0 to 61.1 MiB over 15,472 writes to a table with no index during a 22.8 s backup and from 4.1 to 1,294 MiB over 13,996 writes to a table with two GSIs during a 25.6 s backup (about 20 to 24 pages per write over four runs). The manual's process-level statements were checked on release builds of this tree, SQLite and PostgreSQL: the root placement refusals and their texts, the drill, a re-created catalog's store health with an explicit store and with SQLite's default root (unconfigured, with CreateBackup writing in-catalog backups until the adoption), the accounts re-created with their old ids and old key pairs imported, the adoption, the in-memory startup sync and, on the in-memory dev build, the adopted backups refused until their account is re-created, the in-catalog create's warning and the restore of its backup through a store-configured server, an S3 store at moto with the default spool, `backup sync` exiting 6 beside a SQLite server, ImportTable refusing a gzip file and both kinds of symbolic link and importing a decompressed backup data file, an off-host copy with loose modes refused by `backup sync`, and on PostgreSQL the claim session found by its lock key, whose termination stops the job at its next heartbeat and leaves the row to the reconciler, the registration session's termination failing the server's running backups, the wait of `--prune-missing` on a recently failed backup, a catalog restored alone beside a newer data database, and the catalog and data dumps restored into new databases. On PostgreSQL 16 containers the restore the admin guide now gives ran end to end, on the old server and on a new, empty one, and an access key was imported through the management API from a request body read from a file. The admin guide's Data Recovery entry named the PostgreSQL data database `extenddb_catalog_data`; `init` names it after the catalog without its `_catalog` suffix (`extenddb` by default). The entry now lists each backend's databases, says a catalog name without the suffix needs `--data-db`, and treats the two PostgreSQL databases as one copy: both dumps with the server stopped, under `umask 077` since the catalog dump holds the encryption key and the application role's password, or a base backup; the restore goes into new, empty databases created with `createdb -O` by the admin role, since one over the existing databases restores nothing and one into databases the admin role owns leaves the server unable to create tables. The old databases are renamed aside until the restored server has read every table; the dumps use the server's major version of `pg_dump` and are checked non-empty, since an older one exits 1 leaving an empty file that `psql` restores without an error; and on a new server the application role is created first with the dump's password and, at another address, the catalog's recorded data-database address is set, or the server keeps using the old one. The SQLite row gives the restore with the current `-wal` and `-shm` removed. The security model and the deployment guide's checklist say the same, and the upgrade manual's dumps take `umask 077` and its rollback restores both databases. The architecture, design, quickstart and security-model guides, AGENTS.md, the troubleshooting guide, the technical-debt register and the storage-extension guide carry the right name, the quickstart's `init` table gives the PostgreSQL admin and application roles' flags with their defaults, and `init --help` states both database defaults. The admin guide's `[backup]` section and the upgrade manual's backup paragraph point at the manual; the upgrade manual's shared-root and re-created-catalog sentences move there. The admin guide's `max_concurrent_jobs` row says that SQLite restores of items larger than a few hundred bytes hold more and points at the manual, its backup commands' paragraph gives the exit codes 3, 4 and 5 on a catalog `serve` refuses, and the deployment guide's checklist has a `[backup.store]` line. ADR-0019's Status line says it is superseded in part by the design, whose section 10 states what that covers, a form `docs/adr/README.md` now names; the Cassandra RFC's "Backup and restore" section carries a note, superseded in part, since CreateBackup on Cassandra still writes their in-catalog format, and its lines on synchronous operations and on restored indexes carry notes too. The differences document gives the 1M MongoDB and Cassandra restore figures. `docs/build-docs.py` lists the manual as document 19 (the build renders all 19), and the README's and AGENTS.md's documentation tables have a row for it (AGENTS.md's Extending Storage row now names its file). The quickstart and the `extenddb` skill say that `init` refuses an existing config file without `--overwrite`. The design's testing summary states the measured cells in place of "still to be measured", section 7 says what a SQLite restore of items larger than a few hundred bytes holds, the PR 11 section's line says what shipped, and revision item 35 records the filter history, the measurements, the manual and the database-name correction. Co-authored-by: Lee Signed-off-by: Scott Robinson --- .../references/setup/04-init-stage.md | 2 +- .../troubleshooting/07-runtime-symptoms.md | 2 +- .github/workflows/integration.yml | 4 +- AGENTS.md | 5 +- README.md | 1 + crates/app/src/cmd_init.rs | 4 +- .../0019-cassandra-logical-backup-restore.md | 2 +- docs/adr/README.md | 4 +- docs/build-docs.py | 2 + docs/design/15-backup-restore-bench-pr11.md | 381 ++++++++++ docs/design/15-backup-restore.md | 12 +- docs/design/README.md | 1 + docs/differences-from-dynamodb.md | 2 +- docs/manuals/01-architecture-guide.md | 2 +- docs/manuals/02-design-guide.md | 4 +- docs/manuals/04-quickstart-setup-guide.md | 15 +- docs/manuals/05-admin-guide.md | 44 +- docs/manuals/07-upgrade-manual.md | 13 +- docs/manuals/10-security-model.md | 5 +- docs/manuals/11-deployment-guide.md | 12 +- docs/manuals/12-extending-extenddb-storage.md | 4 +- docs/manuals/14-backup-and-restore.md | 678 ++++++++++++++++++ docs/rfcs/draft-cassandra-backend.md | 18 + docs/technical-debt.md | 2 +- docs/troubleshooting.md | 2 +- 25 files changed, 1173 insertions(+), 48 deletions(-) create mode 100644 docs/design/15-backup-restore-bench-pr11.md create mode 100644 docs/manuals/14-backup-and-restore.md diff --git a/.agents/skills/extenddb/references/setup/04-init-stage.md b/.agents/skills/extenddb/references/setup/04-init-stage.md index b17d74d40..ff4688a83 100644 --- a/.agents/skills/extenddb/references/setup/04-init-stage.md +++ b/.agents/skills/extenddb/references/setup/04-init-stage.md @@ -14,7 +14,7 @@ First, check whether a config file is already present at the repo root. test -f extenddb.toml && echo "extenddb.toml already exists" || echo "extenddb.toml absent" ``` -`extenddb.toml` on its own does not block `extenddb init`. If `extenddb.toml` exists, `init` loads defaults from it and continues. The real blocker is whether the `extenddb_catalog` and `extenddb` data databases exist in PostgreSQL. +`extenddb.toml` on its own blocks `extenddb init`: if it exists, `init` refuses with `Error: Config file "extenddb.toml" already exists. Use --overwrite to delete and regenerate it.` and exits 255. With `--overwrite`, `init` loads the PostgreSQL connection defaults from the file and continues. The other blocker is whether the `extenddb_catalog` and `extenddb` data databases exist in PostgreSQL. Second, check whether a prior deployment is present in PostgreSQL. diff --git a/.agents/skills/extenddb/references/troubleshooting/07-runtime-symptoms.md b/.agents/skills/extenddb/references/troubleshooting/07-runtime-symptoms.md index 86c85875f..a7b14f69f 100644 --- a/.agents/skills/extenddb/references/troubleshooting/07-runtime-symptoms.md +++ b/.agents/skills/extenddb/references/troubleshooting/07-runtime-symptoms.md @@ -21,7 +21,7 @@ HTTP 500 on all requests under heavy load pool_size = 50 # default is 20 ``` -If the problem persists, check for long-running queries or connection leaks with `SELECT * FROM pg_stat_activity WHERE datname = 'extenddb_data';`. +If the problem persists, check for long-running queries or connection leaks with `SELECT * FROM pg_stat_activity WHERE datname = 'extenddb';` (the data database's default name). **Known limitation:** The HTTP status code should be 503 with a `Retry-After` header. This is tracked as technical debt. diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml index 124e64e50..90c4fe6ed 100644 --- a/.github/workflows/integration.yml +++ b/.github/workflows/integration.yml @@ -173,7 +173,7 @@ jobs: EXTENDDB_ADMIN_PASSWORD: ${{ steps.init.outputs.admin_password }} EXTENDDB_TEST_PG_ADMIN_CONNECTION_STRING: postgresql://postgres:devpass@127.0.0.1:5432 EXTENDDB_TEST_BACKUP_STORE: filesystem - run: devtools/run-tests --extenddb --pytest --comprehensive --parallel --filter "not import_export" + run: devtools/run-tests --extenddb --pytest --comprehensive --parallel # moto for the job suite's store-outage cell, the one cell that # exercises `spool_path` on PostgreSQL (an S3 download into the spool, @@ -332,7 +332,7 @@ jobs: # A file-backed SQLite server without [backup.store] writes to the # defaulted root beside the database file. EXTENDDB_TEST_BACKUP_STORE: filesystem - run: devtools/run-tests --extenddb --pytest --comprehensive --parallel --filter "not import_export" + run: devtools/run-tests --extenddb --pytest --comprehensive --parallel - name: Run the backup store and job suites run: python3 -m pytest tests/test_backup_store_config.py tests/test_backup_job.py -v -rP -p no:cacheprovider diff --git a/AGENTS.md b/AGENTS.md index 97dbf65bd..b6aca4056 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -151,7 +151,7 @@ cargo clippy --all-targets -- -D warnings ``` This creates: -- Databases (`extenddb_catalog`, `extenddb_data`) +- Databases (`extenddb_catalog` and `extenddb` on PostgreSQL; `extenddb_catalog` and `extenddb_data` on MongoDB) - Admin user credentials (printed to stdout — save the password!) - Self-signed TLS certificate at `~/.extenddb/tls/cert.pem` - Config file `extenddb.toml` @@ -395,6 +395,7 @@ Expression parsing lives in `crates/core/src/expression/`. This is pure sync Rus | Admin Guide | `docs/manuals/05-admin-guide.md` | Server lifecycle, configuration, IAM | | Security Model | `docs/manuals/10-security-model.md` | Threat model, auth, authz | | Deployment Guide | `docs/manuals/11-deployment-guide.md` | Self-hosted, multi-cloud, air-gapped | +| Backup and Restore | `docs/manuals/14-backup-and-restore.md` | Backup strategy, drills, recovery | | Differences from DynamoDB | `docs/differences-from-dynamodb.md` | Behavioral differences | | Troubleshooting | `docs/troubleshooting.md` | Common errors and solutions | | Storage Component Design | `docs/design/04-component-storage.md` | Storage trait design | @@ -404,7 +405,7 @@ Expression parsing lives in `crates/core/src/expression/`. This is pure sync Rus | Local MongoDB Setup | `docs/local-mongodb-setup.md` | MongoDB installation and config | | Cassandra Setup | `docs/cassandra-setup.md` | Cassandra install, config, and deployment (experimental) | | Testing Design | `docs/design/09-testing.md` | Test strategy and infrastructure | -| Extending Storage | `12-backend-plugin-architecture.md` | Guide to implementing storage backends | +| Extending Storage | `docs/manuals/12-extending-extenddb-storage.md` | Guide to implementing storage backends | Build PDFs: diff --git a/README.md b/README.md index 9f5e9f664..79acfcbe2 100755 --- a/README.md +++ b/README.md @@ -259,6 +259,7 @@ docs/ | [Admin Guide](docs/manuals/05-admin-guide.md) | Server lifecycle, configuration, IAM management | | [Security Model](docs/manuals/10-security-model.md) | Threat model, authentication, authorization | | [Deployment Guide](docs/manuals/11-deployment-guide.md) | Self-hosted, multi-cloud, air-gapped deployments | +| [Backup and Restore](docs/manuals/14-backup-and-restore.md) | Backup strategy, recovery figures, restore drill, catalog-loss recovery | | [Differences from DynamoDB](docs/differences-from-dynamodb.md) | Behavioral differences and unsupported operations | | [Troubleshooting](docs/troubleshooting.md) | Common errors and solutions | diff --git a/crates/app/src/cmd_init.rs b/crates/app/src/cmd_init.rs index 5ec78a390..54f4cde6f 100755 --- a/crates/app/src/cmd_init.rs +++ b/crates/app/src/cmd_init.rs @@ -21,11 +21,11 @@ pub struct InitArgs { #[arg(long)] backend: Option, - /// Data database name (default: extenddb) + /// Data database name (default: the catalog's name without its `_catalog` suffix) #[arg(long)] data_db: Option, - /// Catalog database name (default: _catalog) + /// Catalog database name (default: extenddb_catalog, or the config file's) #[arg(long)] catalog_db: Option, diff --git a/docs/adr/0019-cassandra-logical-backup-restore.md b/docs/adr/0019-cassandra-logical-backup-restore.md index 7e247743c..29696813e 100644 --- a/docs/adr/0019-cassandra-logical-backup-restore.md +++ b/docs/adr/0019-cassandra-logical-backup-restore.md @@ -1,6 +1,6 @@ # ADR-0019: Logical backup and restore for Cassandra -**Status:** Accepted +**Status:** Superseded in part by [the backup and restore design](../design/15-backup-restore.md) (section 10) **Date:** 2026-08-03 ## Context diff --git a/docs/adr/README.md b/docs/adr/README.md index 1fe3fe104..82c2f34ea 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -22,6 +22,8 @@ ADRs are *records*, not proposals. If you are still soliciting input, write an 3. Open a PR. Discussion happens inline. 4. On merge, the ADR is Accepted. Subsequent decisions that override it should create a new ADR and update the original's Status to "Superseded by ADR-NNNN". + A decision that a later design document replaces in part updates its Status + to "Superseded in part by [document]", and that document states which parts. ADRs are never edited after acceptance except to update Status. To change a decision, write a new ADR. @@ -45,7 +47,7 @@ decision, write a new ADR. | [0016](0016-cassandra-index-table-primary-key-structure.md) | Cassandra: index table primary-key structure | Accepted | | [0017](0017-cassandra-transaction-implementation.md) | Cassandra: TransactWriteItems / TransactGetItems implementation | Accepted | | [0018](0018-cassandra-stream-implementation.md) | Cassandra: DynamoDB Streams implementation | Accepted | -| [0019](0019-cassandra-logical-backup-restore.md) | Cassandra: logical backup and restore | Accepted | +| [0019](0019-cassandra-logical-backup-restore.md) | Cassandra: logical backup and restore | Superseded in part by [the backup and restore design](../design/15-backup-restore.md) (section 10) | | [0020](0020-cassandra-occ-update-item.md) | Cassandra: optimistic concurrency control for UpdateItem and PutItem | Accepted | | [0021](0021-cassandra-lwt-delete-atomicity-gap.md) | Cassandra: LWT delete atomicity gap and required worker/saga pattern | Accepted | | [0022](0022-trust-policy-deny-over-allow.md) | Role trust-policy evaluation follows IAM Deny-over-Allow order | Accepted | diff --git a/docs/build-docs.py b/docs/build-docs.py index 1a43bae74..41409e13e 100755 --- a/docs/build-docs.py +++ b/docs/build-docs.py @@ -89,6 +89,8 @@ "docs/local-postgres-setup.md"), (18, "event-ticketing-demo", "Event Ticketing Platform Demo", "getting-started", "docs/manuals/13-event-ticketing-demo.md"), + (19, "backup-and-restore", "Backup and Restore", "usage", + "docs/manuals/14-backup-and-restore.md"), ] diff --git a/docs/design/15-backup-restore-bench-pr11.md b/docs/design/15-backup-restore-bench-pr11.md new file mode 100644 index 000000000..b6b521b45 --- /dev/null +++ b/docs/design/15-backup-restore-bench-pr11.md @@ -0,0 +1,381 @@ +# PR 11 backup and restore record: the open cells (2026-11-05) + +This is the record the testing summary of `15-backup-restore.md` cites for +the cells the PR 8 and PR 9 records left open: a 1M-item restore on MongoDB +and on Cassandra, and the backup and restore of a 1M-item table carrying a +vector index on PostgreSQL and on SQLite. The script (`bench-pr11.py`) and +the raw samples (`/proc//status` every 200 ms per phase, the servers' +logs, one JSON result per run) are kept beside the PR outside the +repository, on the measuring host; the figures below are what they +produced. The earlier records are `15-backup-restore-bench-pr8.md` (the +backup) and `15-backup-restore-bench-pr9.md` (the restore). + +## Method + +The PR 9 record's method. The release binaries (`cargo build --release -p +extenddb` for PostgreSQL, `--no-default-features --features sqlite`, +`mongodb` and `cassandra` for the others), built from this PR's tree, whose +Rust sources are PR 10's, running `serve --foreground` with a filesystem +backup store on local disk. Each run: a fresh deployment (`init`, the port +patched, `[backup] max_concurrent_jobs = 2`, and, after provisioning, +`control_plane_delay_seconds = 0.05` and `index_propagation_delay_ms = 0` +set with `extenddb settings set`), the table loaded through the wire, then +for each measured phase the server stopped and started again so the +process's high-water mark covers that phase alone; `VmHWM` read one second +after the start (the idle figure) and one second after the phase ended (the +peak), sampled every 200 ms in between; "over idle" is the difference. A +backup ends when `DescribeBackup` reports `AVAILABLE`, a restore when +`DescribeTable` reports the table and every vector index `ACTIVE`, each +timed from the request as the client saw it; the job's own log line gives +the same figure within 0.3 s (on MongoDB the job's line is its `DONE`, +below). After each restore the target was counted with a paged `Scan` +(`Select = COUNT`), equal to the items written in every run, and on a vector +table searched with `SearchVectors` for item 7's own vector, which came +first in every run. + +The table: a 200-byte `S` pad of one repeated character (`"x" * 200`) under +a composite key (`pk` cycling over 1,000 values, `sk` the item number), no +secondary index, loaded with BatchWriteItem from four client processes of +eight threads each (the load is not measured). The vector shape adds an +`emb` attribute to every item, a list of 128 numbers (seeded uniform values +in [-1, 1], six decimals), and a `COSINE` vector index on it with no hash +element; its data files are about 2.5 KB of line text per item, 9.5 times +the plain shape's 265 bytes, and 532 bytes compressed against 3.0 to 5.9 +bytes for the plain shape (90 to 176 times; 176 for the 1M key-ordered +backup, 3.0 bytes per item, 145 at 100k, and 90 for MongoDB's unordered 1M +backup: the plain pad is one repeated character, so its 265-byte lines +compress to 3 to 6 bytes). The vector cells' durations are not the index's +cost alone. + +MongoDB restores its own backup, whose data files are unordered (the job's +log line says so). Cassandra's own CreateBackup is +in-catalog, so the Cassandra cell restores the SQLite build's store-backed +backup of the same table (key-ordered lines): the prefix is copied under the +root the Cassandra deployment owns and adopted with `extenddb backup sync`, +as `devtools/backup-restore-matrix` does, and the server is started again +before the restore. + +Host: 16 cores (Xeon Platinum 8124M), 30 GiB, Linux 5.10 (amzn2int); +PostgreSQL 16.15 with pgvector 0.8.7 in a container on loopback (55432); the +SQLite the binary bundles (libsqlite3-sys 0.30.1); MongoDB 7.0.43, +single-node replica set in a container on loopback; Cassandra 4.1.12, a +single node in a container on loopback started as the Cassandra integration +workflow starts its service (`MAX_HEAP_SIZE=1024M`, `HEAP_NEWSIZE=128M`, +`PasswordAuthenticator`), used by these runs alone. What was not held +constant: the host ran other work; the 1-minute load average at each phase's +start is beside each figure (1.3 to 6.4, and 12.6 at the start of the +PostgreSQL 1M vector backup, which began as its table's load ended and ended +at 1.7), so the durations are upper bounds, and each figure is one run. The +peaks varied by up to about 20% between runs of one cell, idle or loaded: +the SQLite 100k vector restore's peak over idle was 344.6 to 409.0 MiB over +nine runs (this record's cell and the runs below, two of them pinned to two +cores, idle and beside four busy loops), and the PostgreSQL 100k restore of +the same items with no vector index 60.0 to 71.4 MiB over five. Everything is on +loopback and on a filesystem store: nothing here says what a network +database or an S3 store's upload or download does to the duration, and the +spool's disk is not in these figures. Every run is under glibc's default +arenas (glibc 2.26) except the four rows marked `MALLOC_ARENA_MAX=1`. + +## Restores + +| backend | shape | data files | idle `VmHWM` | peak `VmHWM` | peak − idle | `ACTIVE` after | downloaded | load | +|---|---|---|---|---|---|---|---|---| +| MongoDB | 100k × 200 B, unordered lines | 1 | 23.5 MiB | 58.6 MiB | **35.1 MiB** | 3.92 s | 573,993 B | 2.7 | +| MongoDB | 1M × 200 B, unordered lines | 4 | 23.5 MiB | 75.3 MiB | **51.8 MiB** | 33.81 s | 5,932,752 B | 3.4 | +| Cassandra | 100k × 200 B, key-ordered lines | 1 | 22.5 MiB | 42.4 MiB | **19.9 MiB** | 6.12 s | 366,631 B | 1.3 | +| Cassandra | 1M × 200 B, key-ordered lines | 4 | 23.2 MiB | 56.3 MiB | **33.1 MiB** | 48.99 s | 3,017,864 B | 3.2 | +| PostgreSQL | 100k, 128-dimension vector | 4 | 21.4 MiB | 122.0 MiB | **100.6 MiB** | 25.15 s | 53,202,407 B | 4.9 | +| PostgreSQL | 1M, 128-dimension vector | 38 | 21.5 MiB | 132.6 MiB | **111.1 MiB** | 246.41 s | 531,977,239 B | 1.7 | +| SQLite | 100k, 128-dimension vector | 4 | 22.8 MiB | 376.8 MiB | **354.0 MiB** | 21.78 s | 53,202,407 B | 3.1 | +| SQLite | 1M, 128-dimension vector | 38 | 21.7 MiB | 573.8 MiB | **552.1 MiB** | 214.23 s | 531,977,239 B | 1.9 | +| SQLite | 100k, 128-dimension vector, `MALLOC_ARENA_MAX=1` | 4 | 22.8 MiB | 218.1 MiB | **195.3 MiB** | 27.93 s | 53,202,407 B | 2.8 | +| SQLite | 1M, 128-dimension vector, `MALLOC_ARENA_MAX=1` | 38 | 23.1 MiB | 227.0 MiB | **203.9 MiB** | 282.86 s | 531,977,239 B | 2.4 | + +The restore request answered in 0.014 to 0.181 s in every run (0.05 s on +MongoDB, 0.17 to 0.18 s on Cassandra, 0.06 s on PostgreSQL, 0.014 to +0.016 s on SQLite). On MongoDB the job's `DONE` ends the item load (33.6 s +at 1M), and the backend's backfill worker flips the table `ACTIVE` at its +next cycle: it runs every `GSI_BACKFILL_INTERVAL`, 5 s +(`crates/storage-mongodb/src/ttl_worker.rs`), and the end of a restore does +not wake it. A MongoDB restore's duration is therefore the job's plus up to +5 s: 3.92 s here, and 8.94 and 8.92 s in two runs at 100k whose job took +3.9 s (below). On Cassandra the 1M high-water mark reached 90% of its +final value at 11.5 s of 49.9, on MongoDB at 18.9 s of 34.7; those times +include the idle figure, and over idle they are 11.7 s and 21.1 s. + +From 100k to 1M the peak over idle moved +16.7 MiB on MongoDB, +13.2 MiB on +Cassandra and +10.5 MiB on PostgreSQL with a vector index. On PostgreSQL the +1M run's mark reached 90% of its final value at 51 s of 247 (57.8 s over +idle) and moved by 1 MiB after 150 s. + +SQLite with a vector index holds more, and under the default arenas its peak +is higher at 1M than at 100k (+198 MiB, 354 to 552 MiB over idle); the 1M +run levelled off after about 150 s, and no other size was measured. The 1M +run's `VmHWM` was 363 MiB at 15 s, 484 MiB at 90 s and 574 MiB at 150 s, +flat from there to the end at 215 s, with `VmRSS` 516 MiB at the end. Under +`MALLOC_ARENA_MAX=1` the same restore holds 195 MiB over idle at 100k and +204 MiB at 1M (+8.6 MiB; `VmHWM` 193 MiB at 20 s and 225 MiB at 60 s, +within 2 MiB of that to the end), so the growth is the allocator's, the +pattern the PR 9 record explains for its worst-expansion shape (items +allocated on the decoder's thread and freed on the backend's write, the +per-thread arenas keeping that memory). What stays is the restore's terms +on this shape: a write batch of up to `RESTORE_BATCH_ITEMS` (10,000) items, +here about 25 MB of line text with each item's 128 numbers decoded beside +it, and, as the PR 9 record found for indexed tables, the write +connection's page cache (`cache_size`, 32 MiB per connection) over the base +and index tables. The decoded size of one such item was not measured. One +arena cost this restore 32% in duration (282.9 s against 214.2 s, one run +each); with two 100k restores running at once it cost 131% (57.5 s against +24.9 s for the pair, 278.9 against 696.7 MiB over idle; the later runs +below). + +## Backups of the same tables + +| backend | shape | idle `VmHWM` | peak `VmHWM` | peak − idle | `AVAILABLE` after | backup size | load | +|---|---|---|---|---|---|---|---| +| MongoDB | 100k × 200 B | 24.5 MiB | 33.8 MiB | **9.3 MiB** | 1.45 s | 573,993 B | 3.0 | +| MongoDB | 1M × 200 B | 24.3 MiB | 40.3 MiB | **16.0 MiB** | 13.49 s | 5,932,752 B | 3.8 | +| PostgreSQL | 100k, 128-dimension vector | 20.8 MiB | 103.8 MiB | **83.0 MiB** | 24.19 s | 53,202,407 B | 6.4 | +| PostgreSQL | 1M, 128-dimension vector | 21.4 MiB | 125.6 MiB | **104.2 MiB** | 243.17 s | 531,977,239 B | 12.6 | +| SQLite | 100k, 128-dimension vector | 21.4 MiB | 119.2 MiB | **97.8 MiB** | 21.42 s | 53,202,407 B | 3.0 | +| SQLite | 1M, 128-dimension vector | 21.5 MiB | 159.1 MiB | **137.6 MiB** | 209.72 s | 531,977,239 B | 3.8 | +| SQLite | 100k, 128-dimension vector, `MALLOC_ARENA_MAX=1` | 22.3 MiB | 75.6 MiB | **53.3 MiB** | 22.19 s | 53,202,407 B | 3.3 | +| SQLite | 1M, 128-dimension vector, `MALLOC_ARENA_MAX=1` | 22.3 MiB | 80.1 MiB | **57.8 MiB** | 222.48 s | 531,977,239 B | 3.1 | + +The backup's growth from 100k to 1M is +6.7 MiB on MongoDB, +21.2 MiB on +PostgreSQL and +39.8 MiB on SQLite with a vector index under the default +arenas (+4.5 MiB under one arena), each under the PR 8 record's bound of +112 MiB. The MongoDB rows are of a table without secondary indexes, so they +do not compare with the PR 8 record's MongoDB rows (two GSIs, 500k in the +1M row's place); they are the first 1M-item MongoDB backup measured. A +backup stores no index contents (the design's testing summary), so the +vector rows measure the backup of the items that carry the vectors. + +## Durations (the RTO baseline the testing summary names) + +On this host, loopback, filesystem store, one run each: 1M items of 200 +bytes restore in 33.8 s on MongoDB (its own backup, unordered lines; another +run may take up to one 5 s cycle of the backfill worker more) and in +49.0 s on Cassandra (a key-ordered backup adopted from SQLite); 100k in +3.9 s (3.9 to 8.9 s over five runs: the job's time plus up to one cycle) +and 6.1 s. A 1M-item table whose items carry a 128-dimension vector, +with a vector index, backs up in 243 s on PostgreSQL and 210 s on SQLite +and restores in 246 s and 214 s; under `MALLOC_ARENA_MAX=1` the SQLite +backup took 222 s and the SQLite restore 283 s. At 100k: 24 s and 21 to +22 s to back up, 25 s and 22 to 28 s to restore. The 1M MongoDB backup took 13.5 s. The PR 9 record's 1M +restores of the plain shape were 9.5 s on PostgreSQL and 26.8 s on SQLite +on the same host class under another build and load, so they compare with +these as an order of magnitude only. + +After each vector restore one `SearchVectors` over the whole table (no hash +element, so every vector is scored) answered in 0.24 s on PostgreSQL and in +5.5 to 15.8 s on SQLite at 1M over seven runs (14.7 s in this run, 5.8 s in +the one-arena run, 5.5 s in a later run, 6.9 s and 5.7 s, one arena, on +04c29155, and 15.8 and 15.2 s in the last section's runs), 0.04 s and 0.6 s +at 100k: SQLite's vector search is an exact scan +(`crates/storage-sqlite/src/vector_search.rs`), which an index with a hash +element bounds to one partition. + +## Later runs on the same host + +These runs used release builds of 30553b38 and, for the second 1,536-number +one-arena restore and the third two-GSI WAL run, of 04c29155, two earlier +versions of this change: 04c29155's Rust sources are this commit's, and +30553b38's differ from them only in the help text of two `init` options. On +the host above, with a separate harness: a fresh deployment per +run, a filesystem store, `max_concurrent_jobs = 2`, the item shapes above +loaded from four processes of eight threads, the server's default settings +(no `settings set`), the server restarted before each measured phase, +`VmHWM` read right before the request and at the end, `/proc//status` +sampled every 200 ms. Every restore was counted and every vector restore +searched for item 7's vector, as above. The 1-minute load average at each +phase's start is in the last column. + +They reproduce the cells above in kind (within 12% on the peak, within 20% +on the durations): the SQLite 100k vector restore 373.3 and 376.5 MiB over +idle in 23.3 and 22.7 s (load 3.0, 2.9), 192.3 MiB in 28.6 s under one arena +(2.8); the SQLite 1M vector backup 141.2 MiB in 210.7 s (3.1), its restore +522.8 MiB in 218.7 s (2.5) and 204.3 MiB in 286.7 s under one arena (1.7); +the PostgreSQL 100k vector backup 83.0 MiB in 26.5 s (10.8) and restore 112.5 +MiB in 29.7 s (8.1); the MongoDB 100k restore 36.2 and 33.2 MiB in 3.9 s; +the Cassandra 100k restore 19.7 and 20.1 MiB in 6.4 and 4.9 s. The vector +data files were byte-identical with the runs above (53,202,407 bytes at +100k). + +What the vector restore's memory belongs to (SQLite unless stated): + +| restore | default arenas | `MALLOC_ARENA_MAX=1` | load | +|---|---|---|---| +| 100k, 128 numbers per item, vector index | 373.3 / 376.5 MiB, 23.3 / 22.7 s | 192.3 MiB, 28.6 s | 3.0, 2.9 / 2.8 | +| 100k, the same items, no vector index | 407.5 MiB, 18.3 s | 184.0 MiB, 20.5 s | 3.4 / 2.6 | +| 20,000 items of 1,536 numbers, vector index | 668.6 MiB, 44.7 s | 301.0 MiB, 48.9 s; 315.9 MiB, 48.4 s | 2.3 / 1.6, 2.8 | +| two 100k restores at once (128 numbers, vector index) | 696.7 MiB, 24.9 s | 278.9 MiB, 57.5 s | 2.0 / 2.7 | +| 100k, 128 numbers, pinned to two cores: idle / beside four busy loops | 409.0 MiB, 22.5 s / 376.0 MiB, 60.6 s | | 2.2 / 2.7 | +| PostgreSQL 100k, 128 numbers, vector index | 112.5 MiB, 29.7 s | 48.8 MiB, 31.5 s | 8.1 / 6.2 | +| PostgreSQL 100k, the same items, no vector index | 60.0 MiB, 15.6 s | | 7.0 | + +The backup of the 1,536-number table held 520.9 MiB over idle and took +47.0 s (load 3.3); its data files carry 26,987 bytes of line text per item. On +SQLite the items without the index held as much as those with it, so the +vector index is not what the memory belongs to. The string control of the +last section shows what is: the restore's write batch (up to +`RESTORE_BATCH_ITEMS`, 10,000 items, or `RESTORE_BATCH_BYTES`, 32 MiB of +item text, `crates/storage/src/restore_engine.rs`) and the decode chunks in +front of it (up to 1,000 items each), which grow with the item's size up to +the batch bound whatever the attribute types. Lists of numbers decode larger +than strings of the same text size, and the term grows with the numbers per +item. The vector index is about 20% of the 100k restore's duration (18.3 +against 22.7 to 23.3 s). On PostgreSQL the index's share is given by the pairs below. The second +1,536-number one-arena run (315.9 MiB) was taken on the same build to +confirm the first. + +Two more pairs on PostgreSQL, on release builds of 04c29155 with a third +harness written from the Method paragraph (a fresh deployment per cell, +`max_concurrent_jobs = 2`, the two `settings set` lines, the server +restarted before each phase, `VmHWM` read one second after the start and +sampled every 200 ms), each 100k vector restore followed at once by the +restore of the same items with no vector index: 99.9 MiB in 24.84 s (load +6.8) against 66.5 MiB in 16.26 s (6.0), and 98.5 MiB in 25.23 s (5.8) +against 71.4 MiB in 15.72 s (3.0). The vector index's share comes from +pairs whose two runs share a harness: these two, the later runs' pair in the +table above (112.5 MiB in 29.7 s against 60.0 MiB in 15.6 s) and the last +section's two (104.5 against 63.9 MiB in 26.99 against 18.05 s, and 99.8 +against 69.8 MiB in 28.30 against 17.34 s). The five put the index at 33.4, +27.1, 52.5, 40.6 and 30.0 MiB and at 35, 38, 47, 33 and 39% of the restore's +duration: 27 to 53 MiB and 33 to 47%. This record's cell has no no-index run +from its own harness, and its pair with the later runs' no-index run (100.6 +against 60.0 MiB), whose harness read the idle figure differently and kept +the server's default settings, is left out. On SQLite the same harness measured the 100k vector restore at +395.8 MiB in 21.86 s (load 4.1) and 358.2 MiB in 21.96 s (2.7), and the same items +with no index at 397.3 MiB in 18.01 s (5.5): the index 18% of the duration. + +SQLite's WAL beside a backup, on the same build: the writer of the manual's +figure (`PutItem` overwrites of 1,000 200-byte items from four threads, no +secondary index) beside the 100k vector backup grew a 51 MiB `-wal` file to +58.7 MiB over 15,051 writes, the frames reusing the file until they pass its +end. The same writer on a table with two GSIs (hash keys `g0` and `g1`, each +changed by every overwrite, `ALL` projection, the default +`index_propagation_delay_ms`) grew it from 58.7 to 1,277.6 MiB over 13,953 +writes in 23.4 s (91.6 KB per write) and, on fresh deployments, from 4.1 to +1,294.1 MiB over 13,996 writes in 25.6 s (96.6 KB, load 4.2) and from 4.1 to +1,261.8 MiB over 13,891 writes in 26.0 s (94.9 KB, load 3.7). With no backup +the same writer for 15 s left it at 4.1 MiB. Each file kept its size after +the server stopped. An asynchronous GSI write is queued in `gsi_pending` +inside the put's transaction and applied and removed by later transactions, +so one such overwrite appends about 22 to 24 pages of 4 KiB during a backup +(about 20 in a fourth run, below). + +## Figures measured for the manual + +The two database-cost figures of `docs/manuals/14-backup-and-restore.md` +("What a backup costs the database") for a table with no secondary index. +Release builds of 358ff8bd's Rust sources (those of 30553b38) on the host +above, a fresh deployment per run, a filesystem store, +`max_concurrent_jobs = 2`, the Method paragraph's two `settings set` lines (the reproductions +below used the defaults and measured the same in kind). The backup's source +is this record's 100k-item, 128-dimension vector table, loaded through the +wire, the server restarted after the load. The writer: four client threads +of `PutItem` overwrites of 1,000 items (a hash key and a 200-byte `S` pad) +in a second table with no secondary index, started with the `CreateBackup` +request and stopped when `DescribeBackup` reported `AVAILABLE`; the control +is the same writer for the same time with no backup. + +- PostgreSQL 16.15. The relation measured is the writer table's + `pg_total_relation_size` (with its indexes and TOAST), after a `VACUUM` + before the run. A 25.2 s backup beside 12,156 overwrites grew it from 0.4 to + 4.4 MiB, about 345 bytes per write. A `VACUUM (VERBOSE)` of that table 10 s + into the backup, 4,947 writes in, reported "4948 are dead but not yet + removable"; `pg_stat_user_tables` counted 12,114 dead after the backup, and + the first `VACUUM` after it left "0 are dead but not yet removable". The + control, after a `VACUUM FULL`: 17,478 overwrites in 25.2 s, 0.4 to 0.5 MiB. + Load 5.4 at the backup's start, 4.3 at its end. + +- SQLite. The file measured is the database's `-wal`, sampled every 200 ms. + The control first: 15,762 writes in 22.2 s left it at 4.0 MiB. Then a 22.8 s + backup beside 15,472 writes grew it from 4.0 to 61.1 MiB, about 3.9 KB per + write; it was 61.1 MiB 5 s after the backup, after 10.2 s more of the writer + with no backup, and after the server stopped. Load 3.2 at the backup's end. + +Reproduced on release builds of 04c29155 with the third harness above and +the server's default settings: + +- PostgreSQL: a 28.1 s backup beside 12,623 overwrites grew the relation + from 0.49 to 4.91 MiB (367 bytes per write); `VACUUM (VERBOSE)` 10 s in: + "3952 are dead but not yet removable"; the `VACUUM` after the backup: 0. The + same writer with no backup for the same time (17,042 writes), the relation + not compacted in between, left it at 4.91 MiB: the space was reused. Load + 9.2 at the end. + +- SQLite, the table with no index: 15,267 writes during a 22.5 s backup grew + the `-wal` file from 4.1 to 63.0 MiB (4.0 KB per write); 15,165 writes with + no backup left it at 4.1 MiB; 63.0 MiB 5 s after the backup and after a + `SIGTERM` stop. Load 5.2. + +- SQLite, the two-GSI writer of "Later runs": 14,212 writes during a 25.9 s + backup grew it from 4.2 to 1,117.7 MiB (82.2 KB per write, about 20 frames), + kept after the stop. Load 3.9. + +## Runs on this commit's Rust sources + +Release builds of this commit's Rust sources on the host above, with a +fourth harness written from the Method paragraph: a fresh deployment per +invocation, a filesystem store, `max_concurrent_jobs = 2`, the two +`settings set` lines, the shapes above loaded from four processes of eight +threads, the server stopped and started before each phase, `VmHWM` read one +second after the start and one second after the phase and sampled every +200 ms, every restore counted with a paged `Scan` (equal to the items in +every run) and every vector restore searched for item 7's vector (first in +every run). The host ran other work, and two cells often ran at once; the +1-minute load at each phase's start is in the last column. The string shape +replaces the 128-number list with a random 2,290-character `S` attribute and +has no vector index: 2,569 bytes of line text per item in the decompressed +data files, against 2,503 for the 128-number items. + +| restore | default arenas | `MALLOC_ARENA_MAX=1` | load | +|---|---|---|---| +| SQLite 100k, 128 numbers, vector index | 367.2 MiB, 22.26 s; 344.6 MiB, 36.58 s | 185.4 MiB, 37.13 s | 13.1, 5.4 / 7.8 | +| SQLite 100k, the same items, no vector index | 399.2 MiB, 20.53 s; 424.5 MiB, 22.36 s | 186.2 MiB, 23.25 s | 7.0, 7.3 / 6.5 | +| SQLite 100k, a 2,290-character string per item, no index | 253.4 MiB, 10.78 s; 252.5 MiB, 9.20 s | 129.6 MiB, 11.66 s; 129.8 MiB, 9.44 s | 6.2, 2.8 / 6.2, 2.7 | +| SQLite 1M, 128 numbers, vector index | 500.8 MiB, 226.47 s | 208.0 MiB | 4.5 / 1.6 | +| PostgreSQL 100k, 128 numbers, vector index | 104.5 MiB, 26.99 s; 99.8 MiB, 28.30 s | 52.0 MiB, 35.40 s | 14.5, 6.0 / 8.5 | +| PostgreSQL 100k, the same items, no vector index | 63.9 MiB, 18.05 s; 69.8 MiB, 17.34 s | | 7.1, 6.1 | +| MongoDB 100k × 200 B | 35.5 MiB, 8.94 s; 35.4 MiB, 8.92 s | | 7.3, not read | + +The 1M one-arena restore's duration is left out: it started while the +server was still dropping the previous 1M target. + +The string items carry no number and held 253 MiB over idle (130 MiB under +one arena), 6.7 times the PR 9 record's 38 MiB for 100k items of 200 bytes. +So on SQLite a restore of items of more than a few hundred bytes holds its +write batch and decode chunks ("Later runs"), which grow with the item's +size up to the batch bound whatever the attribute types. At the same text +size the 128-number items held more, 344.6 to 424.5 MiB at 100k over the +measured runs with or without the index and 184.0 to 195.3 MiB under one +arena, since each number decodes to an attribute value with a string of its +own. Under one arena the 1M restore measured 203.9 to 208.0 MiB over three +runs, so that figure is flat from 100k to 1M within about 25 MiB over the +measured runs. + +On MongoDB each of the two 100k restores logged the job's `DONE` "in 3.9 s; +index backfill pending", and the backfill worker's `ACTIVE` line came about +4.8 s later: the worker's cycle ("Restores"). The whole-table +`SearchVectors` over the 1M restored vectors took 15.78 s after the +default-arena restore (load 2.7) and 15.23 s after the one-arena one. + +## The sizing sentence + +A restore of 200-byte items held under 60 MiB over idle on MongoDB and under +40 MiB on Cassandra at 1M items. With a 128-dimension vector on every item, +PostgreSQL held about 110 MiB over idle to restore and about 105 MiB to back +up at 1M, growing about 10 and 20 MiB from 100k. SQLite held 552 MiB over +idle to restore and 138 MiB to back up at 1M under the default arenas, +higher than at 100k, and 204 and 58 MiB under `MALLOC_ARENA_MAX=1`, flat +from 100k to 1M within about 25 MiB over the measured runs. The runs above +show that on SQLite this memory is the restore's write batch and decode +chunks, not the vector index, and that it grows with the item's size up to +the 32 MiB batch bound whatever the attribute types: size a SQLite host that +restores tables of items of more than a few hundred bytes from the measured +points under one arena (130 MiB per running restore for 100k items of +2.5 KB strings, 184 to 208 MiB for items of 128 numbers, 301 to 316 MiB for +20,000 items of 1,536 numbers), times `max_concurrent_jobs`, more for larger +items, and treat the default-arena figures as unbounded by this record. diff --git a/docs/design/15-backup-restore.md b/docs/design/15-backup-restore.md index 7d6b12ed3..050c47e51 100644 --- a/docs/design/15-backup-restore.md +++ b/docs/design/15-backup-restore.md @@ -169,7 +169,7 @@ spool_path = "" # restore download area for an S3 store; d On SQLite (file-backed) a missing `[backup]` section is equivalent to `kind = "filesystem"` with `path` = `backups/` beside the database file; a `validate` failure of that defaulted root is a logged warning that leaves the store unconfigured, not a startup error, so an upgrade never stops a server that never configured backups, while an explicitly configured root that fails `validate` is still a startup error. The warning holds only while the catalog has no backup rows for the defaulted root (`read_store_ids`); once it has, the root has been used and is handled as an explicitly configured one (revision item 29). On PostgreSQL and MongoDB a missing `[backup.store]` leaves the off-host path unconfigured and startup logs that; `kind = "disabled"` leaves it unconfigured on purpose on any backend. Without a configured store, CreateBackup on every backend uses the legacy in-catalog create (`LegacyBackupReader::create`, section 3) and logs a deprecation warning naming `[backup.store]` on each call, until PR 14 removes that path; after it CreateBackup without a configured store returns a ValidationException naming `[backup.store]`. Legacy backups stay readable, listable, restorable and deletable either way. `sqlite-memory` (any `:memory:` database path) has no database file and so no default root: CreateBackup there takes the off-host path only with `[backup.store]` set and the legacy create without it. Cassandra has no `SnapshotEngine`, so it uses the legacy create even with a store configured (section 3); a configured store serves restores of off-host backups into Cassandra and nothing else there. -Memory per running job, which operators size hosts from. A backup's terms, for a row of `r` bytes of item text whose decoded `Item` is `d` bytes (`d` is about 1.2 × `r` for a row of many short strings, #394's worst shape, where every `N` is a 133-character string): the read path holds the page's decoded rows, `d` each, while they wait for the pipeline, at most the rows of the budget PR 8 adds to the PostgreSQL and SQLite streams (`SNAPSHOT_FETCH_BYTES`, 64 MiB, a constant beside `DEFAULT_FETCH_BATCH`, not a `[backup]` key in v1) plus one row, and one row's parse, `r + 2d`, since the deserializer builds an intermediate value before the item; PostgreSQL pulls the page's rows from the connection one at a time and parses each as it arrives, so it holds no page text beyond the row being parsed, and SQLite pulls a row at a time with the driver's worker at most five rows ahead of the pull, `r` each (`row_buffer_size` on the pool, four, plus the row the worker holds while its `send` waits; 160 MiB at the line cap; the default of 50 made the run-ahead, not the page, the largest term on a table of large rows). The pipeline holds the feeder's item, the item channel's items (`ITEM_CHANNEL`, four) and the encoder's item, `d` each; the encoder's line buffer (one line, at most 32 MiB, the line cap) and gzip window; the feeder's measurement of an item over the default `max_item_size_bytes` holds no line (it counts the line's bytes through a writer that stops at the cap), so the terms are the same on an instance that raises the limit, at the largest stored row; the chunk sink's 1 MiB buffer and the chunk channel (four 1 MiB chunks); the upload window (`upload_part_size_bytes * max_concurrent_uploads` on an S3 store, one write buffer on a filesystem store); and the manifest's data-file entries until the manifest is written. No term grows with `data_file_target_bytes` or with the table, except on `sqlite-memory`, whose open copies the whole table (section 3); every term but the budget and the chunks grows with the row, which `max_item_size_bytes` bounds (the line cap bounds the line the job writes and names the item it refuses; the row is read and decoded before the feeder measures it). The sizing rule, per running job, is the terms at the line cap times the allocator's factor, times `max_concurrent_jobs`; the rule is stated at the default `max_item_size_bytes`, where no stored item's line exceeds the cap (27.7 MB is the largest line a 400 KB item expands to, `MAX_LINE_BYTES`'s doc); on an instance that raises it the page's decoded rows, one row's parse and the pipeline's items are at the largest stored row, which only `max_item_size_bytes` bounds (about 68 times the item's size as a line, for #394's worst shape). The terms at the line cap (`r` = 32 MiB, `d` = 1.2 `r`): the page's rows, 3 `d` = 115 MiB; one row's parse, `r` + 2 `d` = 109 MiB; the pipeline's six items, 6 `d` = 230 MiB; the line buffer, 32 MiB; the chunks, 5 MiB; on SQLite the run-ahead, 5 `r` = 160 MiB: about 490 MiB on PostgreSQL and 650 MiB on SQLite, an upper bound, since the run-ahead and a full pipeline are not both at their largest at one instant. The allocator's factor is glibc's: the row's 200,000 short strings are allocated on the tokio worker that parses the row and freed on the blocking thread that encodes it, and with glibc's default per-thread arenas that memory stays in the arenas between pages, so the process's high-water mark grows with the number of large rows the job reads and varies by about 200 MiB between identical runs; under `MALLOC_ARENA_MAX=1` (one arena, set in the server's environment on a host that backs up tables of large items; the admin guide's `max_concurrent_jobs` row carries it and the deployment guide's service unit carries the line commented out with that condition, since one arena serializes the worker threads' allocations on one lock and costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%), measured on GetItem, Query and Scan with a Python client, the record) the peak is the terms' and does not grow with the table. Measured on PR 8's 16-core host at 27.7 MB rows (`r` = 26.4 MiB, three rows to a page; the terms' sum 545 MiB on SQLite), 1,000 small rows then 12, 24 and 48 such rows, each series six runs back to back on one SQLite binary (`docs/design/15-backup-restore-bench-pr8.md`; revision item 31): with the default arenas the peak over idle was 466, 588 and 760 MiB on a quiet host and 707, 700 and 1,281 MiB on the final build with the kill-point runs beside it (0.86 to 2.35 of the terms, growing with the rows and varying by hundreds of MiB between series; runs of the PR's first build on a loaded host reached 1,096 MiB at 48 rows; no plateau was measured), and under `MALLOC_ARENA_MAX=1` 481, 452 and 480 MiB and then 452, 480 and 455 MiB (0.83 to 0.88 of the terms, flat from 12 to 48 rows in both series; the SQLite ratios are of the peak over idle), and on PostgreSQL at 12 rows 340 MiB `VmHWM` under one arena against 750 to 788 MiB under the default arenas (0.82 of the terms at that row on PostgreSQL, 413 MiB, which has no run-ahead; that ratio is of the `VmHWM`, the process's whole high-water mark, and about 0.47 over its idle of 144 MiB); so the bound an operator can hold the process to is the terms at the line cap times 1.2 per job under `MALLOC_ARENA_MAX=1`, and under the default arenas a figure that grows with the large rows a job reads, 2.35 of the terms at 48 rows being the largest measured. A one-item channel saved about 50 MiB of the peak and cost a 1M-row backup 75% of its duration, so the channel stays at four. With 200-byte rows every term that grows with the row is under 1 MiB, the chunk channel and the sink hold their 5 MiB whatever the row, and the high-water mark of a 1M-row backup is within 10 MiB of a 100k-row backup's on PostgreSQL and SQLite (the bound the acceptance asserts is 112 MiB: the budget, the line cap and the channels) and the 500k-row backup's within 5 MiB on MongoDB, where the 500k run stands in for the 1M run (its load alone took 27 minutes through the wire). A running restore holds the parsed manifest, the decoder's line (at most `MAX_LINE_BYTES`) and the deserializer's intermediate value for it, the decoded items of `LOAD_CHUNK_BYTES` (32 MiB) of data-file text plus one line per chunk (the chunk closes with the item that reaches the bound) at four chunk positions (two in the channel to `bulk_load`, the one being filled, the one being drained), and the backend's write batch, times `max_concurrent_jobs`; the chunk bound counts the bytes the decoder read for the chunk's items, the unit the backup side budgets in, not the service's item sizing, and a chunk also closes at `LOAD_CHUNK_ITEMS` (1,000) items. The backend's write batch: at most 32 MiB of item text on SQLite and Cassandra (10,000 items when they are smaller), held beside the decoded items of that text, 4 MiB per INSERT statement on PostgreSQL (500 rows when they are smaller; one transaction per file; the item and its text, two copies), 1,000 documents per `insert_many` on MongoDB (the driver splits the call; the documents are held until it returns, bounded in bytes only by the data file, since `bulk_load` is per file). A `bulk_load` is one data file at a time, so the chunks in flight never hold more than one file's items, at most the source's `data_file_target_bytes` plus one line of text. None of these grows with the table: they grow with the item, which `max_item_size_bytes` bounds, and SQLite's `COUNT(*)` after the last file costs the counting connection's page cache, bounded by `cache_size` (32 MiB; 22 to 28 MiB measured at 1M and at 3M rows, the record). The input order of the backup's lines is a term of SQLite's beside these: a key-ordered backup (the server's own, of a hash-only or single-range-key table) appends to the base b-tree's right edge, an unordered one (every MongoDB-sourced backup and every multi-part-key table) dirties pages across the whole tree, and the write connection's page cache plus the allocator's arenas hold the difference (the record's input-order table: 1M items of 200 bytes 87 to 98 MiB key-ordered against 144 to 164 MiB unordered on SQLite, in 27 to 56 s against 66 to 90 s; PostgreSQL 16 MiB either way). A table with secondary or vector indexes adds no term of the restore's: on SQLite it adds the backend's own per-connection page cache for the index b-trees (`cache_size`, 32 MiB), bounded by that setting and growing sub-linearly with the table (the record's index table), on PostgreSQL a few MiB. Measured (`docs/design/15-backup-restore-bench-pr9.md`; revision item 33), `VmHWM` over idle from a fresh start, 24 worst-shape items of 27.7 MB: 848 MiB on SQLite and 778 MiB on PostgreSQL under the default arenas (848 and 1,013 MiB at 12 and 48 items on SQLite: the peak grows with the large items a restore reads, as the backup's does, and no bound was measured), 392 MiB on SQLite and 427 MiB on PostgreSQL under `MALLOC_ARENA_MAX=1`; 1,200 items of 350 KB in one file 270 MiB on SQLite and 156 MiB on PostgreSQL; 1M items of 200 bytes under a composite key with no secondary index, the backup's lines in key order, 89 MiB on SQLite and 16 MiB on PostgreSQL (100k: 38 and 10.7 MiB), restored in 26.8 s and 9.5 s; the same items under the same key with two full-projection GSIs 143 to 225 MiB on SQLite at 100,000 items in three runs and 165 to 172 MiB at 300,000, and 23 to 34 MiB on PostgreSQL at both: the excess on SQLite is the backend's own per-connection page cache (`cache_size`, 32 MiB) holding the index tables' b-tree pages plus the allocator's arenas (about a third less under `MALLOC_ARENA_MAX=1`), not the restore's buffers, which the 32 MiB batch bound holds whatever the indexes; it is bounded by `cache_size` and grows sub-linearly with the table. A restore of a table of large items is of the same order as its backup, not less; the sizing rule is the record's: about 450 MiB per running restore of large items under one arena, times `max_concurrent_jobs`. Each running backup also holds one connection from the PostgreSQL data pool for its duration, and on the separate layout one catalog-pool connection while its snapshot opens (section 3); the backend admits at most half the pool (after the raise below) as open snapshots and refuses more with `LimitExceeded`; a running restore holds one data connection and has no slot of its own after PR 6; and each PostgreSQL job holds one dedicated claim connection outside the pool. `pool_size` sizes both pools, and the rule is `pool_size >= max(MIN_POOL_SIZE, 2 * max_concurrent_jobs + 10)`: twice the jobs, so the half-pool bound admits every job `max_concurrent_jobs` allows, plus ten for requests; 18 at the defaults, under the default `pool_size` of 20. It applies whether or not `[backup.store]` is configured, because legacy restores are jobs too. Startup raises a smaller `pool_size` to that value with a warning naming both numbers, as it already raises a `pool_size` under `MIN_POOL_SIZE` (10) today. On SQLite the snapshot slot (`backup_server_slot_count`, `pool_size / 2 - 1` capped at 16, shared by the in-catalog create and the snapshot open) stays the backend's admission bound, as PostgreSQL's half-pool slot does, and a server raises `[storage.sqlite] pool_size` to `2 * (max_concurrent_jobs + 1)` (at most 34) with a warning naming both numbers so the slot refuses no job the permit admitted; 10 at the defaults, the default pool; above 16 jobs the slot refuses the seventeenth. `backup sync`, which is not a job, keeps of the configured root's rows (every status) each row's status, store id and table and the key of each AVAILABLE row of the configured store id, dropping the rest of each row as it is read, and holds the objects of the one prefix the listing is in with their sizes and the prefix of each manifest it listed; every backend's `list_backup_rows` collects its whole result before it yields the first row, so the read of the rows is sync's largest term on a catalog with many rows of the root (tombstones within their retention included): about 22.5 KiB of peak RSS per row of a table with five GSIs (1,452 bytes of `source_table_details_json`; 610 MiB at 25,000 rows and 1,144 MiB at 50,000, against 47 MiB with none, and 148 MiB at 50,000 rows whose JSON is `{}`, so nearly all of it is the parsed description), measured on a SQLite debug build on a 16-core x86_64 Linux host under glibc's default arenas, the same peak `backup list` reaches over those rows; the filesystem store builds its listing of the whole root in one `Vec` before it yields the first key (about 72 bytes plus the key per object, the store trait's doc), and the S3 store lists a page of 1,000 keys at a time. Measured on a filesystem store with 100 objects per prefix and 60-byte keys: 45 MiB of peak RSS for an empty root, 66 MiB for 100,000 objects and 98 MiB for 300,000, about 190 bytes per object, nearly all of it the store's listing (revision item 34). The report keeps an entry per prefix for the whole run (it lists every prefix the catalog already has a row for among the skipped ones), so at one object per prefix the per-prefix terms double that: about 350 bytes per prefix, 62 MiB of peak RSS for 50,000 such prefixes and 112 MiB for 200,000 against 45 MiB for an empty root (and about 165 bytes per object at 100 objects per prefix), measured on a SQLite debug build of PR 10 with a filesystem root and no rows. On an in-memory SQLite server that cost is paid in the serving process at every start. +Memory per running job, which operators size hosts from. A backup's terms, for a row of `r` bytes of item text whose decoded `Item` is `d` bytes (`d` is about 1.2 × `r` for a row of many short strings, #394's worst shape, where every `N` is a 133-character string): the read path holds the page's decoded rows, `d` each, while they wait for the pipeline, at most the rows of the budget PR 8 adds to the PostgreSQL and SQLite streams (`SNAPSHOT_FETCH_BYTES`, 64 MiB, a constant beside `DEFAULT_FETCH_BATCH`, not a `[backup]` key in v1) plus one row, and one row's parse, `r + 2d`, since the deserializer builds an intermediate value before the item; PostgreSQL pulls the page's rows from the connection one at a time and parses each as it arrives, so it holds no page text beyond the row being parsed, and SQLite pulls a row at a time with the driver's worker at most five rows ahead of the pull, `r` each (`row_buffer_size` on the pool, four, plus the row the worker holds while its `send` waits; 160 MiB at the line cap; the default of 50 made the run-ahead, not the page, the largest term on a table of large rows). The pipeline holds the feeder's item, the item channel's items (`ITEM_CHANNEL`, four) and the encoder's item, `d` each; the encoder's line buffer (one line, at most 32 MiB, the line cap) and gzip window; the feeder's measurement of an item over the default `max_item_size_bytes` holds no line (it counts the line's bytes through a writer that stops at the cap), so the terms are the same on an instance that raises the limit, at the largest stored row; the chunk sink's 1 MiB buffer and the chunk channel (four 1 MiB chunks); the upload window (`upload_part_size_bytes * max_concurrent_uploads` on an S3 store, one write buffer on a filesystem store); and the manifest's data-file entries until the manifest is written. No term grows with `data_file_target_bytes` or with the table, except on `sqlite-memory`, whose open copies the whole table (section 3); every term but the budget and the chunks grows with the row, which `max_item_size_bytes` bounds (the line cap bounds the line the job writes and names the item it refuses; the row is read and decoded before the feeder measures it). The sizing rule, per running job, is the terms at the line cap times the allocator's factor, times `max_concurrent_jobs`; the rule is stated at the default `max_item_size_bytes`, where no stored item's line exceeds the cap (27.7 MB is the largest line a 400 KB item expands to, `MAX_LINE_BYTES`'s doc); on an instance that raises it the page's decoded rows, one row's parse and the pipeline's items are at the largest stored row, which only `max_item_size_bytes` bounds (about 68 times the item's size as a line, for #394's worst shape). The terms at the line cap (`r` = 32 MiB, `d` = 1.2 `r`): the page's rows, 3 `d` = 115 MiB; one row's parse, `r` + 2 `d` = 109 MiB; the pipeline's six items, 6 `d` = 230 MiB; the line buffer, 32 MiB; the chunks, 5 MiB; on SQLite the run-ahead, 5 `r` = 160 MiB: about 490 MiB on PostgreSQL and 650 MiB on SQLite, an upper bound, since the run-ahead and a full pipeline are not both at their largest at one instant. The allocator's factor is glibc's: the row's 200,000 short strings are allocated on the tokio worker that parses the row and freed on the blocking thread that encodes it, and with glibc's default per-thread arenas that memory stays in the arenas between pages, so the process's high-water mark grows with the number of large rows the job reads and varies by about 200 MiB between identical runs; under `MALLOC_ARENA_MAX=1` (one arena, set in the server's environment on a host that backs up tables of large items; the admin guide's `max_concurrent_jobs` row carries it and the deployment guide's service unit carries the line commented out with that condition, since one arena serializes the worker threads' allocations on one lock and costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%), measured on GetItem, Query and Scan with a Python client, the record) the peak is the terms' and does not grow with the table. Measured on PR 8's 16-core host at 27.7 MB rows (`r` = 26.4 MiB, three rows to a page; the terms' sum 545 MiB on SQLite), 1,000 small rows then 12, 24 and 48 such rows, each series six runs back to back on one SQLite binary (`docs/design/15-backup-restore-bench-pr8.md`; revision item 31): with the default arenas the peak over idle was 466, 588 and 760 MiB on a quiet host and 707, 700 and 1,281 MiB on the final build with the kill-point runs beside it (0.86 to 2.35 of the terms, growing with the rows and varying by hundreds of MiB between series; runs of the PR's first build on a loaded host reached 1,096 MiB at 48 rows; no plateau was measured), and under `MALLOC_ARENA_MAX=1` 481, 452 and 480 MiB and then 452, 480 and 455 MiB (0.83 to 0.88 of the terms, flat from 12 to 48 rows in both series; the SQLite ratios are of the peak over idle), and on PostgreSQL at 12 rows 340 MiB `VmHWM` under one arena against 750 to 788 MiB under the default arenas (0.82 of the terms at that row on PostgreSQL, 413 MiB, which has no run-ahead; that ratio is of the `VmHWM`, the process's whole high-water mark, and about 0.47 over its idle of 144 MiB); so the bound an operator can hold the process to is the terms at the line cap times 1.2 per job under `MALLOC_ARENA_MAX=1`, and under the default arenas a figure that grows with the large rows a job reads, 2.35 of the terms at 48 rows being the largest measured. A one-item channel saved about 50 MiB of the peak and cost a 1M-row backup 75% of its duration, so the channel stays at four. With 200-byte rows every term that grows with the row is under 1 MiB, the chunk channel and the sink hold their 5 MiB whatever the row, and the high-water mark of a 1M-row backup is within 10 MiB of a 100k-row backup's on PostgreSQL and SQLite (the bound the acceptance asserts is 112 MiB: the budget, the line cap and the channels) and the 500k-row backup's within 5 MiB on MongoDB, where the 500k run stands in for the 1M run (its load alone took 27 minutes through the wire). A running restore holds the parsed manifest, the decoder's line (at most `MAX_LINE_BYTES`) and the deserializer's intermediate value for it, the decoded items of `LOAD_CHUNK_BYTES` (32 MiB) of data-file text plus one line per chunk (the chunk closes with the item that reaches the bound) at four chunk positions (two in the channel to `bulk_load`, the one being filled, the one being drained), and the backend's write batch, times `max_concurrent_jobs`; the chunk bound counts the bytes the decoder read for the chunk's items, the unit the backup side budgets in, not the service's item sizing, and a chunk also closes at `LOAD_CHUNK_ITEMS` (1,000) items. The backend's write batch: at most 32 MiB of item text on SQLite and Cassandra (10,000 items when they are smaller), held beside the decoded items of that text, 4 MiB per INSERT statement on PostgreSQL (500 rows when they are smaller; one transaction per file; the item and its text, two copies), 1,000 documents per `insert_many` on MongoDB (the driver splits the call; the documents are held until it returns, bounded in bytes only by the data file, since `bulk_load` is per file). A `bulk_load` is one data file at a time, so the chunks in flight never hold more than one file's items, at most the source's `data_file_target_bytes` plus one line of text. None of these grows with the table: they grow with the item, which `max_item_size_bytes` bounds, and SQLite's `COUNT(*)` after the last file costs the counting connection's page cache, bounded by `cache_size` (32 MiB; 22 to 28 MiB measured at 1M and at 3M rows, the record). The input order of the backup's lines is a term of SQLite's beside these: a key-ordered backup (the server's own, of a hash-only or single-range-key table) appends to the base b-tree's right edge, an unordered one (every MongoDB-sourced backup and every multi-part-key table) dirties pages across the whole tree, and the write connection's page cache plus the allocator's arenas hold the difference (the record's input-order table: 1M items of 200 bytes 87 to 98 MiB key-ordered against 144 to 164 MiB unordered on SQLite, in 27 to 56 s against 66 to 90 s; PostgreSQL 16 MiB either way). A table with secondary indexes adds no term of the restore's: on SQLite it adds the backend's own per-connection page cache for the index b-trees (`cache_size`, 32 MiB), bounded by that setting and growing sub-linearly with the table (the record's index table), on PostgreSQL a few MiB. On SQLite the restore's write batch (`RESTORE_BATCH_ITEMS`, 10,000 items, or `RESTORE_BATCH_BYTES`, 32 MiB of item text) and the decode chunks in front of it grow with the item's size up to the batch bound, whatever the attribute types, so a restore of items of more than a few hundred bytes holds more: 253 MiB over idle for 100k items of about 2.5 KB of strings, 130 MiB under one arena (two runs each), and more for lists of numbers of the same text size, which decode larger (345 to 424 MiB, 184 to 195 MiB under one arena, at 100k items of 128 numbers, with or without a vector index); at 1M items of 128 numbers 552 MiB under the default arenas and 204 MiB under one arena, and 301 to 316 MiB under one arena for 20,000 items of 1,536 numbers (`15-backup-restore-bench-pr11.md`). A PostgreSQL vector index adds 27 to 53 MiB to the restore at 100k items (five pairs of runs with and without it). Measured (`docs/design/15-backup-restore-bench-pr9.md`; revision item 33), `VmHWM` over idle from a fresh start, 24 worst-shape items of 27.7 MB: 848 MiB on SQLite and 778 MiB on PostgreSQL under the default arenas (848 and 1,013 MiB at 12 and 48 items on SQLite: the peak grows with the large items a restore reads, as the backup's does, and no bound was measured), 392 MiB on SQLite and 427 MiB on PostgreSQL under `MALLOC_ARENA_MAX=1`; 1,200 items of 350 KB in one file 270 MiB on SQLite and 156 MiB on PostgreSQL; 1M items of 200 bytes under a composite key with no secondary index, the backup's lines in key order, 89 MiB on SQLite and 16 MiB on PostgreSQL (100k: 38 and 10.7 MiB), restored in 26.8 s and 9.5 s; the same items under the same key with two full-projection GSIs 143 to 225 MiB on SQLite at 100,000 items in three runs and 165 to 172 MiB at 300,000, and 23 to 34 MiB on PostgreSQL at both: the excess on SQLite is the backend's own per-connection page cache (`cache_size`, 32 MiB) holding the index tables' b-tree pages plus the allocator's arenas (about a third less under `MALLOC_ARENA_MAX=1`), not the restore's buffers, which the 32 MiB batch bound holds whatever the indexes; it is bounded by `cache_size` and grows sub-linearly with the table. A restore of a table of large items is of the same order as its backup, not less; the sizing rule is the record's: about 450 MiB per running restore of large items under one arena, times `max_concurrent_jobs`. Each running backup also holds one connection from the PostgreSQL data pool for its duration, and on the separate layout one catalog-pool connection while its snapshot opens (section 3); the backend admits at most half the pool (after the raise below) as open snapshots and refuses more with `LimitExceeded`; a running restore holds one data connection and has no slot of its own after PR 6; and each PostgreSQL job holds one dedicated claim connection outside the pool. `pool_size` sizes both pools, and the rule is `pool_size >= max(MIN_POOL_SIZE, 2 * max_concurrent_jobs + 10)`: twice the jobs, so the half-pool bound admits every job `max_concurrent_jobs` allows, plus ten for requests; 18 at the defaults, under the default `pool_size` of 20. It applies whether or not `[backup.store]` is configured, because legacy restores are jobs too. Startup raises a smaller `pool_size` to that value with a warning naming both numbers, as it already raises a `pool_size` under `MIN_POOL_SIZE` (10) today. On SQLite the snapshot slot (`backup_server_slot_count`, `pool_size / 2 - 1` capped at 16, shared by the in-catalog create and the snapshot open) stays the backend's admission bound, as PostgreSQL's half-pool slot does, and a server raises `[storage.sqlite] pool_size` to `2 * (max_concurrent_jobs + 1)` (at most 34) with a warning naming both numbers so the slot refuses no job the permit admitted; 10 at the defaults, the default pool; above 16 jobs the slot refuses the seventeenth. `backup sync`, which is not a job, keeps of the configured root's rows (every status) each row's status, store id and table and the key of each AVAILABLE row of the configured store id, dropping the rest of each row as it is read, and holds the objects of the one prefix the listing is in with their sizes and the prefix of each manifest it listed; every backend's `list_backup_rows` collects its whole result before it yields the first row, so the read of the rows is sync's largest term on a catalog with many rows of the root (tombstones within their retention included): about 22.5 KiB of peak RSS per row of a table with five GSIs (1,452 bytes of `source_table_details_json`; 610 MiB at 25,000 rows and 1,144 MiB at 50,000, against 47 MiB with none, and 148 MiB at 50,000 rows whose JSON is `{}`, so nearly all of it is the parsed description), measured on a SQLite debug build on a 16-core x86_64 Linux host under glibc's default arenas, the same peak `backup list` reaches over those rows; the filesystem store builds its listing of the whole root in one `Vec` before it yields the first key (about 72 bytes plus the key per object, the store trait's doc), and the S3 store lists a page of 1,000 keys at a time. Measured on a filesystem store with 100 objects per prefix and 60-byte keys: 45 MiB of peak RSS for an empty root, 66 MiB for 100,000 objects and 98 MiB for 300,000, about 190 bytes per object, nearly all of it the store's listing (revision item 34). The report keeps an entry per prefix for the whole run (it lists every prefix the catalog already has a row for among the skipped ones), so at one object per prefix the per-prefix terms double that: about 350 bytes per prefix, 62 MiB of peak RSS for 50,000 such prefixes and 112 MiB for 200,000 against 45 MiB for an empty root (and about 165 bytes per object at 100 objects per prefix), measured on a SQLite debug build of PR 10 with a filesystem root and no rows. On an in-memory SQLite server that cost is paid in the serving process at every start. A configuration error that needs no request to the store (a part size, prefix or key-length problem, an insecure endpoint, a root that fails the owner-only rule or whose ancestors others can write to, a filesystem root that is or contains the database directory, a spool rule) is a startup error naming the path or bucket, except for the defaulted SQLite root above while the catalog has no rows for it, where it and a failure of the root's identity check are a warning that leaves the store unconfigured. A filesystem root's `validate` also writes a probe file, a request to the store: an I/O failure there or in resolving the root is a store condition that degrades the store (`failed validation`), as the same condition one reconciler cycle later does, not a startup error. A filesystem store's `validate` and identity check run before the listener, bounded by a timeout, so a mount that stops answering after the root is opened leaves the server starting with the store `checking` (a root that does not answer at startup holds the server before the listener, as the database directory would); an S3 store's are requests to the endpoint and run after the listener is up, in the reconciler's first cycle, with the store `checking` until the first result reports, so a stalled endpoint never holds the data plane (revision item 29). Every later check runs under the same bound (5 s for a filesystem root; 30 s for a bucket, under the SDK's per-attempt timeout so no retry outlives the check) and against the shutdown token: a store that has stopped answering costs a reconciler cycle its bound and nothing more, the cycle's other duties run, the check is recorded as a transport failure, and neither the drain at shutdown nor the process's exit waits on it (a filesystem root's abandoned call stays parked on a thread until the mount answers, no later check starts while it is, the root's calls run one at a time so that calls started before the first reached its bound wait for it under the bound and then have the bound for their own call, rather than parking a thread each, and the binary ends its runtime with a 5 s bound). A transport failure (no region, no credentials, an unreachable endpoint, a check that did not finish within its bound), a missing or denied bucket found after the listener, and a failure of section 4's identity check or manifest probe, do not stop startup, because a store problem must not take the data plane out of rotation (section 9): the failure is logged with its cause, the store component is reported unhealthy with the problem's category as its reason, the operations that touch the store refuse with a ValidationException that names the store's kind and the category and not its root (CreateBackup on PostgreSQL, SQLite and MongoDB, and RestoreTableFromBackup and DeleteBackup of an off-host backup; Describe and List, legacy backups and the in-catalog CreateBackup are unaffected), the reconciler acts on no row of the store (its legacy duty, which touches no store, still runs) and resumes no restore, and the check runs again each reconciler cycle until it passes. @@ -226,11 +226,11 @@ Which document and which row each PR changes; what each row says is in the ownin | upgrade manual: PR 14's migration refusal, `--force-drop-legacy-backups`, CreateBackup without a store after PR 14 | PR 14 | | upgrade manual: the #618 regression on the off-host path and the way out; the `[backup.store]` paragraph's asynchronous create and the restore of a store-backed backup deferred to PR 9a | PR 8 | | `docs/troubleshooting.md` DeleteTable entry and admin guide `control_plane_delay_seconds` row | #395 (in its diff) | -| ADR-0019 and the Cassandra RFC's "Backup and restore" section marked as superseded; the admin and upgrade manuals' backup sections | PR 11 | +| ADR-0019 and the Cassandra RFC's "Backup and restore" section marked as superseded in part; the admin and upgrade manuals' backup sections | PR 11 | | `docs/design/README.md` line 22 | PR 2 | | RFC-0003 backup criterion | a separate docs change (section 3) | -- This document is the design record; `docs/manuals/14-backup-and-restore.md` is derived from it for operators. Related records: ADR-0019 (Cassandra logical backup) and the "Backup and restore" section of `docs/rfcs/draft-cassandra-backend.md`, which still describe backup and restore as synchronous and are superseded on that point by #384 and this document; the admin and upgrade manuals' backup sections (#384) describe the in-catalog path this series replaces and are rewritten in PR 11. +- This document is the design record; `docs/manuals/14-backup-and-restore.md` is derived from it for operators. Related records: ADR-0019 (Cassandra logical backup) and the "Backup and restore" section of `docs/rfcs/draft-cassandra-backend.md` describe the first Cassandra implementation, with a synchronous restore that dropped the secondary indexes. Both are superseded in part (PR 11): the RFC section carries a note at its top, and the ADR its Status line, "Superseded in part by" this document (section 10), since `docs/adr/README.md` allows no other edit of an accepted ADR; what that Status covers is stated here. What still holds of ADR-0019: CreateBackup on Cassandra writes the in-catalog backup it describes, before it returns. What changed: RestoreTableFromBackup runs as a background job that answers `CREATING` and restores the secondary indexes (#384), a reconciler resolves an interrupted restore, a backup in a configured backup store restores into Cassandra, and the continuous-backup status is no longer stored. The reconciler also removes an in-catalog backup whose CreateBackup did not finish and that no process owns, so the ADR's Negative consequence that interrupted operations need a future reconciliation worker no longer holds for the create. The admin guide keeps the `[backup]` and `[backup.store]` reference and the `extenddb backup` commands, and the upgrade manual what an upgrade changes; both point at the manual for the rest (PR 11). ## Phasing and work breakdown @@ -468,7 +468,7 @@ Documentation rows it corrects: upgrade manual: on SQLite the backup CLI command ### PR 11: CI and documentation -- PR 11: CI and documentation: drop the `not import_export` filters; measure the 1M-item restore on MongoDB and Cassandra and the 1M-item vector cell on PostgreSQL and SQLite (the testing summary's two open cells; the nightly schedule for the matrix shipped with PR 9); the documentation per section 10 with RPO and RTO numbers from the PR 8, PR 9 and these runs on a stated host class. Depends on PR 3, 9a and 9b. +- PR 11: CI and documentation: drop the `not import_export` filters; measure the 1M-item restore on MongoDB and Cassandra and the 1M-item vector cell on PostgreSQL and SQLite (the testing summary's two open cells; the nightly schedule for the matrix shipped with PR 9); the documentation per section 10 with RPO and RTO numbers from the PR 8, PR 9 and these runs on a stated host class. Depends on PR 3, 9a and 9b. Shipped as revision item 35 says: both filters dropped; the four cells measured at 1M, none left open (`15-backup-restore-bench-pr11.md`); `docs/manuals/14-backup-and-restore.md` with every statement of the acceptance below, the PostgreSQL dead-tuple and SQLite WAL figures measured for it (the record's "Figures measured for the manual"); the admin guide's database names; ADR-0019's Status line and a note on the Cassandra RFC's section, superseded in part; the admin and upgrade manuals' backup sections pointing at the manual. Documentation it writes (section 10's table): New `docs/manuals/14-backup-and-restore.md`: strategy, RPO and RTO statements per store kind (RPO equals backup interval, no PITR; RTO measured in the acceptance tests and published), configuration, IAM policy example, restore drill procedure, catalog loss recovery with `backup sync`, migrating legacy backups, what a restore preserves and what it does not (the table from the readiness discussion), moving a backup to or from Amazon DynamoDB with ImportTable and export. The IAM and S3-compatible-server material points at `crates/backup-store/README.md` rather than copying it; the admin guide's database names; ADR-0019 and the Cassandra RFC's "Backup and restore" section marked as superseded; the admin and upgrade manuals' backup sections rewritten. Acceptance: the statements the series defers to the deployment guide, each stated in the manual: a filesystem store keeps the same-host failure domain, said in the first sentence of the backup section, with S3 as the production recommendation; encryption at rest is the store's job; a pickup agent runs as the server's user or as root, and a root under a storage administrator's directory, an NFS mount owned by `nobody` or a group-writable parent without the sticky bit is refused; a bind mount or a filesystem without Unix permission semantics is outside what the store can observe; an `AbortIncompleteMultipartUpload` lifecycle rule, and for a versioned bucket the `NoncurrentVersionExpiration` and `ExpiredObjectDeleteMarker` rules; the least-privilege IAM policy example; a PostgreSQL snapshot retains every dead tuple written during the backup and the SQLite WAL grows by the write volume during it, with the sizing figures the manual measures; the claim connection's keepalive bound (about a minute after the loss of the owner's host) and `pg_terminate_backend` on the claim's session as the way to end it sooner; a transaction-mode pooler is unsupported for the claim; the SDK's `trace` logs carry the session token and are not enabled in production; write access to an account's prefix is backup-administrator authority for that account, with per-account IAM prefix policies; MongoDB's `minSnapshotHistoryWindowInSeconds` as the operator setting a slow backup needs; a root that reports `0700` while enforcing nothing. @@ -502,7 +502,7 @@ Phase 3, later (v1.x): - Distribution matrix for the filesystem default path: unprivileged local launch, the npm launcher, the container image with and without a volume, `sqlite-memory`. - Cross-backend matrix: nightly. - Live DynamoDB: Phase 2, measured. -- Performance: backup and restore of a 1M-item table per backend, and of a 1M-item table carrying a vector index on the backends that support one, duration and peak memory recorded as the published RTO baseline; memory must stay flat with table size (streaming, never materializing the table) in the restore's own terms: on SQLite a table with secondary indexes costs SQLite's page cache for the index b-trees above the no-index figure, an unordered backup costs its page cache across the base b-tree above the key-ordered figure, and the final `COUNT(*)` costs the counting connection's page cache, each bounded by `cache_size` (section 7, the record). The backup side's 1M figures are PR 8's record; the restore's are PR 9a's (`docs/design/15-backup-restore-bench-pr9.md`): 1M items of 200 bytes in key order in 9.5 s on PostgreSQL and 26.8 s on SQLite, the peak over idle 16 and 89 MiB, the growth from 100k to 1M 5.5 MiB on PostgreSQL and 51 MiB on SQLite (the write connection's page cache and the final `COUNT(*)`, each bounded by `cache_size`, not the items; from 1M to 3M rows the SQLite peak moved 11 MiB), and the same 1M items as an unordered backup 16 MiB on PostgreSQL and 144 to 164 MiB on SQLite in 66 to 90 s (SQLite's base b-tree under scattered inserts, not the restore's buffers). The MongoDB and Cassandra 1M restore and the vector cell are still to be measured (PR 11). Index contents are not stored in the backup; every index is rebuilt from the items on restore, so the vector cell measures the dominant restore cost. +- Performance: backup and restore of a 1M-item table per backend, and of a 1M-item table carrying a vector index on the backends that support one, duration and peak memory recorded as the published RTO baseline; memory must stay flat with table size (streaming, never materializing the table) in the restore's own terms: on SQLite a table with secondary indexes costs SQLite's page cache for the index b-trees above the no-index figure, an unordered backup costs its page cache across the base b-tree above the key-ordered figure, and the final `COUNT(*)` costs the counting connection's page cache, each bounded by `cache_size` (section 7, the record). The backup side's 1M figures are PR 8's record; the restore's are PR 9a's (`docs/design/15-backup-restore-bench-pr9.md`): 1M items of 200 bytes in key order in 9.5 s on PostgreSQL and 26.8 s on SQLite, the peak over idle 16 and 89 MiB, the growth from 100k to 1M 5.5 MiB on PostgreSQL and 51 MiB on SQLite (the write connection's page cache and the final `COUNT(*)`, each bounded by `cache_size`, not the items; from 1M to 3M rows the SQLite peak moved 11 MiB), and the same 1M items as an unordered backup 16 MiB on PostgreSQL and 144 to 164 MiB on SQLite in 66 to 90 s (SQLite's base b-tree under scattered inserts, not the restore's buffers). The MongoDB and Cassandra 1M restores and the vector cell are the PR 11 record's (`docs/design/15-backup-restore-bench-pr11.md`; revision item 35), on the same host class with no cell left open: 1M items of 200 bytes restore in 33.8 s on MongoDB (its own backup, unordered) and 49.0 s on Cassandra (a key-ordered SQLite backup adopted with `backup sync`), the peak over idle 52 and 33 MiB, 16.7 and 13.2 MiB above the 100k figure; a 1M-item table whose items carry a 128-dimension vector, with a vector index, backs up in 243 s on PostgreSQL and 210 s on SQLite and restores in 246 s and 214 s, the restore's peak over idle 111 MiB on PostgreSQL (10.5 MiB above 100k) and 552 MiB on SQLite under the default arenas (198 MiB above 100k; the 1M run levelled off after about 150 s, and no other size was measured), 204 MiB under `MALLOC_ARENA_MAX=1` (8.6 MiB above 100k), where the backup took 222 s and the restore 283 s. On SQLite that memory is the restore's write batch and decode chunks, not the vector index: they grow with the item's size up to the 32 MiB batch bound whatever the attribute types (100k items of about 2.5 KB of strings held 253 MiB over idle, 130 MiB under one arena), and lists of numbers decode larger at the same text size (the same 100k items of 128 numbers with no vector index held 397 to 424 MiB, and 20,000 items of 1,536 numbers 669 MiB, 301 to 316 MiB under one arena). Against this item's criterion the vector cell is flat from 100k to 1M under `MALLOC_ARENA_MAX=1` (within about 25 MiB over the measured runs) and is not shown bounded under the default arenas. On PostgreSQL the index adds 27 to 53 MiB at 100k (five pairs of runs with and without it, each pair from one harness). On MongoDB the table becomes `ACTIVE` at the backfill worker's next 5 s cycle after the load, so the restore's duration is the load's plus up to 5 s (3.9 to 8.9 s at 100k). Index contents are not stored in the backup; every index is rebuilt from the items on restore, and the vector index is about 20% of the 100k restore's duration on SQLite (18.3 s without it against 22.7 to 23.3 s with it) and 33 to 47% on PostgreSQL (the same five pairs, 15.6 to 18.1 s without it against 24.8 to 29.7 s with it). ## Review checklist for implementation PRs @@ -593,3 +593,5 @@ Changed after a review of the draft against main `9f43d03` and PRs #384, #335, # 33. (2026-11-01, PR 9 as implemented — the design's PR 9a and PR 9b as one PR, the maintainer's decision.) The restore job runs from the store on PostgreSQL, SQLite, MongoDB and Cassandra, same-backend and cross-backend, from a filesystem or an S3 store; the matrix and the nightly are PR 9b's, shipped here. `RestoreTableFromBackup` of a `backup_index` row no longer answers the "in a backup store; this release restores in-catalog backups only" refusal `check_catalog_row_for_restore` carried; a healthy configured store's AVAILABLE row is restored, and the refusal is gone from the engine and the differences, usage and upgrade docs (5.3, section 10, PR 9a). The interface: `RestoreEngine`, a new trait in `StorageEngine`'s bound with `bulk_load(target, claim, BulkLoadFile { file_index, start_at, reapplying }, items) -> applied`, `finish_restore(target, claim, item_count, size_bytes) -> Finished { Active, Pending, NotMatched }`, `count_restore_target(target) -> u64`, `read_restore_progress` (default `[]`) and `sweep_restore_progress(running) -> u64` (default 0); `StorageEngine` has no methods of its own, so `bulk_load` is not "a new `StorageEngine` method" (section 3's PR 9a interface list). `BackupCatalog` gains `claim_restore(account_id, input: CreateTableInput, &RestoreRegistrationRow) -> (TableDescription, Box)` (the registration creates the CREATING target and returns the description the request answers; the record is born owned under the backend's generation, `RestoreRegistrationRow.owner_generation` being `UNOWNED_GENERATION` and the backend minting its own), `claim_restore_dead(key, stale_after, spool_host) -> Option>` (the restore analogue of `claim_dead_backup`, writing the claimant's `spool_host`, a fresh generation and a fresh `heartbeat_at` on the database's clock in one conditional write, where the backup takeover leaves the heartbeat: nothing waits on a restore record's old heartbeat after a takeover, and a second taker must find the record live at once, 4(e)), `read_restore_job`, `list_restore_jobs(Option)` and `remove_restore_target`/`fail_restore_job` as the reconciler uses them; `RestoreJobRow` carries `store`, `spool_host`, `file_index`, `items_applied`, `failure_reason`, `completed_at`, so the FAILED-target duty reads a row's reason and the sweeps read its store and `spool_host` (section 3, 4, 5.3, PR 9a, PR 9b). The design's `claim_restore(&RestoreJobRow) -> Box` and `record_progress(&dyn JobClaim, file_index, items_applied)` are gone: there is no `record_progress`; every backend records progress inside `bulk_load`, in the write that applied the items (PostgreSQL a `restore_progress` row in the file's data transaction, SQLite and Cassandra a `tables`-row update per batch, MongoDB a `restore_job.file_index` update per file), and the design's "conditioned on the generation" holds on SQLite, MongoDB and Cassandra but on PostgreSQL the progress row lives in the data database, which holds no job record, so the fence there is the base table's primary key (a file applied twice fails on its first key, a recorded file on the progress row's own key) and the claim's liveness, which the engine confirms before each file and whose loss, the heartbeat's, aborts a file in flight; the generation fences only the record's own writes in the catalog (DONE, FAILED, the takeover) (5.3 step 5, PR 9a). `bulk_load` carries `reapplying` so MongoDB's duplicate suppression is explicit: on a resume the first file the engine meets is `reapplying = true` and every later file false; MongoDB suppresses an all-E11000 `insert_many` only when `reapplying`, else "the backup holds a key twice"; PostgreSQL and SQLite never suppress (their base rows are plain `INSERT`s, a duplicate fails the file whatever the flag, and SQLite has no unrecorded partial write, its progress being in the batch's transaction); Cassandra's `put_restored_item` is an upsert, so a duplicate collapses and `count_restore_target` one short of the manifest's total is the only duplicate check there (5.3 step 6, PR 9a). The restore ends by counting the target against the manifest's item total on every backend, so an unordered backup holding one key in two files and a key-range check that missed are both caught; a key-ordered file runs #394's per-file `KeyRange` check, which ends the file's stream at an item outside its range. The migration for PostgreSQL's `restore_progress` is id 6 (`MigrationId::RestoreProgress`), a `StoreKind::Data` body on PostgreSQL (`data_migrations/0006_restore_progress.sql`), `NotApplicable` on SQLite, and nothing on MongoDB and Cassandra, whose bootstrappers have no `MigrationId` table; this corrects "data migration 005 for `restore_progress`" in the PR 9a section (its acceptance line and its load paragraph) and the PR 6 line on reserving numbers; item 14's "data migration 005 reserved" stands as the history it is (4(i), PR 6, PR 9a). The manifest registration locks the `backup_index` row (the lock `start_backup_delete` takes: PostgreSQL `SELECT ... FOR UPDATE`, SQLite the `BEGIN IMMEDIATE` write transaction, MongoDB `$inc lock_seq` on the document, Cassandra the `active_restores` lightweight transaction on the backup's partition), while the legacy registration keeps the `backups` row; a legacy and a store-backed restore into one name exclude each other (PostgreSQL the advisory `restore_lock_key`, which both paths take; on SQLite the store-backed path's `restore:/` slot and, for a legacy restore, the `tables` row under the write transaction), so a `DeleteBackup` of a `backup_index` row meets only store-backed registrations and the two orderings resolve one before the other (DeleteBackup wins → the registration is `BackupNotFound` and no target; the registration wins → DeleteBackup is `BackupInUse` and the row stays AVAILABLE) (5.2, 5.3, PR 9a). The restore's ACTIVE flip sets the table and every secondary and vector index of the target in one catalog transaction on PostgreSQL and SQLite (an index of a restore target is created ACTIVE, and `activate_restore_target` sets ACTIVE any `indexes` or `vector_indexes` row that is not), on MongoDB the job writes DONE and `restore_backfill_pending` and the backfill worker flips the table ACTIVE once every index has left CREATING (`finish_restore` returns `Pending` there), and on Cassandra the one lightweight transaction flips the table alone, its index rows created ACTIVE (`Active`, never `Pending`), so a Query on each GSI and `SearchVectors` answer as soon as the target reports ACTIVE (5.3 step 6, PR 9a). The request bounds its wait for the detached registration at `RESTORE_REGISTRATION_TIMEOUT` (60 s) on every backend; past it it answers `InternalServerError` while the task goes on and the target appears through `DescribeTable`, so the differences document's MongoDB-only sentence becomes every backend's (5.3 step 4, PR 9a). The maintenance check runs after the registration on the manifest path (5.3 step 4 as written) and before the permit on the legacy path (item 29), both orders in `restore_table_from_backup` and `restore::start`; the restore's first confirmed beat stands at the instant `claim_restore` returned (the record's `heartbeat_at` was stamped inside the registration, before the return, so the instant is at or after the beat it stands for by the registration's remaining duration, which each backend bounds, 4(e)), where the backup job takes the instant before its claim's insert (5.1 step 4), so a registration that took longer than two heartbeat periods does not end the job at its first check; a held claim on the manifest path leaves a FAILED `maintenance` record whose target the job removes before it answers `LimitExceededException`. Only `StoreError::Transport` is retried on a data-file download, for up to `DOWNLOAD_RETRY_BOUND` (15 min, counted from the job's first download attempt, every file's retries against it) with `min(2^n s, 60 s)` plus up to a second of jitter, so the retry covers S3 stores (a filesystem store's `get` errors are `Io` and fail at once); `NotFound`, a denial, and a digest or size mismatch fail at once naming the file. On a filesystem store "verified in place" is two `get` passes, a hashing pass before any load and a decoding pass through `ExtenddbManifest::reader` that hashes again (so a file replaced between the passes fails in the load) with no spool; on an S3 store the files are spooled as `restore-
/.json.gz` with `.partial` → `sync_all` → rename → `.verified` (the marker holding the file's sha256, compared with the manifest's before the file is trusted; an entry left by a run of this host for a record another host then owned is removed before use) and read through `reader_prehashed`, the spool at `spool_path` (the defaulted `backup-spool/` on file-backed SQLite). A restore the shutdown stops records `RestoreJobs{cancelled}` and no worker metric, keeps its spool and its progress and writes no status (the `SHUTDOWN` reason is a backup's alone); a lost claim or a DONE write that matched no record records `RestoreJobs{failed}` and `WorkerErrorCount{RestoreJob}` with no catalog or table access; `fail_restore` writes FAILED then runs `remove_restore_target` only when that write matched. A job that learnt the record is another's (the lost claim, the DONE write's `NotMatched`, `fail_restore`'s no-table-access arm) leaves its spool entry to the sweep, since a same-host successor may be reading it (the entry is keyed by table id); every other end but the shutdown's removes it (4(e), 5.3 step 7, PR 9a). The load hands the backend one `bulk_load` per file over a stream fed in chunks of at most `LOAD_CHUNK_ITEMS` (1000) items or `LOAD_CHUNK_BYTES` (32 MiB of data-file line text) through a channel of two chunks; a restore holds the parsed manifest, one decoded line and its parse, the decoded items of four chunk positions and the backend's batch (32 MiB of item text on SQLite and Cassandra, 4 MiB per INSERT on PostgreSQL, 1,000 documents on MongoDB), which is section 7's restore sentence, with the figures measured on this PR's final tree (`docs/design/15-backup-restore-bench-pr9.md`: 848 MiB over idle on SQLite and 778 MiB on PostgreSQL at 24 worst-shape items under the default arenas, 392 and 427 MiB under one arena; 89 and 16 MiB at 1M key-ordered items of 200 bytes, in 26.8 s and 9.5 s, 144 to 164 and 16 MiB for the same items unordered), which replace the earlier sentence, "A restore holds one line's decode plus one insert batch plus the parsed manifest for the whole job; neither grows with the table": the terms stand, the figures are measured, and a restore of large items is of the backup's order. The metrics are `MetricName::{RestoreJobs, RestoreJobDuration, RestoreBytesDownloaded}` with the outcome in the operation dimension, `QuerySource::RestoreJob` for the worker metrics, and `BackupStoreErrors{get}` on every failed `get` of a restore job (the request's and a resume's two document reads and each download attempt the store failed, retried or not; a digest mismatch and the spool's own errors are not the store's); section 9's `restore_jobs_total{outcome}` and `restore_job_duration_seconds` are a converter's spellings, as item 31 said for the backup names (section 9, PR 9a). A backend's refusal of an item in `bulk_load` is not a wire exception: on MongoDB an `N` sort-key value past Decimal128's 34 significant digits is the write path's `ValidationException`, which becomes the restore's FAILED reason, and the client sees the target disappear; a 38-digit non-key number is stored as text and restored intact on every backend. The reconciler resumes a dead store-backed restore from its recorded progress (per file on PostgreSQL and MongoDB, per batch on SQLite and Cassandra) under `claim_restore_dead` once the heartbeat is stale (on SQLite at once after a restart, by the record's slot; on PostgreSQL once the owner's claim session has ended as well), re-downloading to its own spool when the dead owner's `spool_host` is not this host's; it fails a dead manifest record under a maintenance claim before it looks at the record's store (`OWNER_DEAD`, the target removed; `target_not_empty`, the target left for `remove-target`, when it recorded no progress and the target holds items), fails a record of a store other than the configured one or of no store with `OTHER_STORE` (naming `backup sync --move-store`, the target left to the FAILED-target duty), skips a resume without a permit, and FAILs a resume with no recorded progress into a non-empty target as `target_not_empty`, left for `extenddb backup remove-target --account
` (PR 10's command, shipped with PR 9, below); its two sweeps (`restore_progress` rows of tables that are not RUNNING restore targets; spool entries of restores not RUNNING with this host's `spool_host`) run only without the maintenance claim and read `list_restore_jobs(Some(RUNNING))`, a backend that cannot list them logging once and skipping both (section 4, 5.3 step 7, PR 9a). A store-backed backup of a multi-part-key table is written `unordered` and restores on every backend, the target laid out as a natively created table (`bulk_load` keys rows with the backend's own key functions), while a legacy backup of one keeps the reader's refusal; the #384 override tests run through both the legacy and the manifest paths for every case with a definition and assert the same input or the same refusal, the manifest path deriving the SSE specification from the manifest's `sse_description` where the legacy path keeps the specification as written. The cross-backend matrix (PR 9b): a cell copies the source's backup prefix into a root the target catalog owns and inserts the target's `backup_index` row as `backup sync --adopt-store` will (PR 10), so the cell proves the restore and not the adoption; it runs from the committed `devtools/backup-restore-matrix`, and `.github/workflows/matrix-nightly.yml` (a `schedule` cron and `workflow_dispatch`) runs the twelve plain cells and the duplicate-key and vector cells nightly, building the four backend binaries and starting PostgreSQL, a MongoDB replica set and Cassandra as the integration workflows do; the seed stands in for `backup sync --adopt-store` until PR 10 replaces it. The restore kill points are run by hand with the `kill-points.py` posted with PR 9 (outside the repository, as PR 8's was), one mode per backend where it applies: `restore_after_download`, `restore_after_file_commit`, `restore_sqlite_mid_batch`, `restore_cassandra_before_progress`, `restore_mongodb_after_insert_many`, `restore_postgres_before_progress_delete`, `restore_mongodb_before_worker_flip`, `restore_between_failed_and_removal`, `restore_fenced_out`, `pg_terminate_backend` and `sigterm`, each a SIGKILL or a signal at the named point with a restart and the outcome asserted (the exact count after a resume, or the target gone for a failing restore). Two points pause before the first progress record of a file or batch (`restore_mongodb_after_insert_many`, `restore_cassandra_before_progress`), so a kill at their first reach leaves written items and no progress and the resume answers `target_not_empty` (the rule of 5.3 step 7, asserted); the re-application path (MongoDB's E11000 suppression of the one unrecorded file, Cassandra's re-applied batch) is run by killing at the point's second reach, after an earlier file's or batch's progress is recorded (the script's `--second-reach`): on MongoDB a 24,000-item backup in three files resumed from file 1 with file 0 recorded and reached ACTIVE with the exact count (file 1's 8,227 items re-applied as duplicates, 7,544 applied by the resume); on Cassandra a one-file backup of 12,000 items killed before the second batch's progress write resumed from item 10,000 and reached ACTIVE with 12,000 (2,000 applied by the resume), and `restore_after_file_commit` resumed from file 1 to the exact count. A Cassandra server's CreateBackup is the in-catalog format (section 3), so a Cassandra kill point takes its backup on a SQLite source, copies the prefix under the Cassandra target's root and seeds the target's `backup_index` row as the matrix does; the single-node Cassandra of the harness needs `replication_factor = 1` in the server's configuration, as `init` writes it, since under the default of 3 every lightweight transaction and quorum read on a keyspace the server creates answers "No nodes available in query plan", which is what had blocked these points. The feature chain for the backends' pause points: `extenddb-app/backup-test-hooks` → `extenddb-server/backup-test-hooks` now pulls `extenddb-storage/test-hooks` beside `extenddb-engine/backup-test-hooks`, so a hooks build compiles `extenddb_storage::backup_catalog::restore_pause_point` and each backend's own point (`restore_sqlite_mid_batch`, `restore_cassandra_before_progress`, `restore_mongodb_after_insert_many`, `restore_postgres_before_progress_delete`, each a `pub const` beside `restore_pause_point` in `extenddb_storage::backup_catalog`, the one spelling the backends and the kill-point script use) in without naming a backend crate (the server depends on none directly); the engine's five restore points are its `KillPoint` variants (`RestoreAfterDownload`, `RestoreAfterFileCommit`, `RestoreMongodbBeforeWorkerFlip`, `RestoreBetweenFailedAndRemoval`, `RestoreFencedOut`), and that enum is the engine's list, not the whole list. Corrections made before PR 9 merged: (1) the restore takeover writes `heartbeat_at = now()` in the one conditional write with the generation and `spool_host`, on every backend, so a second taker finds the record live at once; the backup takeover keeps leaving it (4(e) states why the rules differ); (2) `LOAD_CHUNK_BYTES` is summed from the decoded line's length (the decoder's `bytes_decoded` across one item), not `item_size_bytes`; section 7's terms are restated in that unit; (3) the download retry bound is per job: `download_all` takes the job's download start once and every file's retries count against it; the module doc, 5.3 step 3 and this item say so; (4) a job that learnt the record is another's (a lost claim, a DONE or FAILED write that matched no record) leaves its spool entry to the reconciler's sweep; (5) the `.verified` marker holds the file's sha256 and `is_verified` compares it with the manifest's; `Entry::open` removes an entry whose record's `spool_host` was another host's before use; the resume reads the manifest from the spool only when the record named this host before the takeover (it read the record after the takeover, which had just written this host's name); (6) the decoder checks the heartbeat's pre-write rule (`confirms_the_claim`) at every chunk boundary beside the shutdown token; MongoDB and Cassandra re-read the record per batch; (7) a restore's `claimed_at` is the instant `claim_restore` returned, the record's `heartbeat_at` having been stamped inside the registration. Cassandra's takeover re-applies the `active_restores` entry when absent; the SQLite and Cassandra batch is bounded by `RESTORE_BATCH_BYTES` (32 MiB of item text) beside `RESTORE_BATCH_ITEMS` (10,000), both in `extenddb_storage::restore_engine`; MongoDB and Cassandra read the record before each write batch; `sweep_restore_progress` deletes from a positive list; the backends' pause-point names are `pub const`s beside `restore_pause_point` in `extenddb_storage::backup_catalog`, one spelling each (`RESTORE_SQLITE_MID_BATCH`, `RESTORE_MONGODB_AFTER_INSERT_MANY`, `RESTORE_CASSANDRA_BEFORE_PROGRESS`, `RESTORE_POSTGRES_BEFORE_PROGRESS_DELETE`), and the engine's `KillPoint` is its own list; Cassandra's `count_restore_target` reads the catalog first and answers `TableNotFound` for a key that is not a CREATING restore target with that id, as the other three do; the MongoDB registration stamps the record's heartbeat from the server's clock inside its transaction (the registering process's clock is replaced; the test registers from a process an hour behind); a duplicate primary key fails the file with `data file : holds a primary key the target already has (the backup holds a key twice): `, the engine naming the file by its key once and the backends naming no file; SQLite's startup line for an aged RUNNING record names the reconciler's duty per kind. The reconciler's store check runs first in the cycle, so the manifest arm reads this cycle's health, and the skip line names `not yet checked` for a store whose first check has not recorded; a maintenance-held dead manifest restore is counted once, in `manifest_restores_failed_and_removed`; `RestorableKeySchema::resolve` runs in the request with step 2's refusals; the request's second table-name check and second `check_supported` are gone: the parser's identity rule and version gate run first; the engine's kill points log `job reached kill point `; each backend's own point is named beside `restore_pause_point`; `finish_restore`'s `Pending`, the DONE write's `NotMatched`, the pre-write claim checks of `run`, the download's pre-attempt check and `WorkerErrorCount{RestoreJob}` on a FAILED job have engine tests; the #384 override tests run through both paths for every case with a definition. The wire test `test_create_backup_of_a_vector_table_is_refused` is `test_vector_table_backup_refused_in_catalog_and_restored_from_store`, since it asserts the round trip on a server with a store; the multi-part-key wire cell lives in `tests/test_backup_job.py` on a deployment that enables `allow_multipart_table_keys`, compares the restored item set with the source's `Scan` and makes no `GetItem` by the whole key (the preview's item paths address such a table by its first HASH attribute on every backend: a `GetItem` by the whole key finds nothing on PostgreSQL and the first HASH key's one item on SQLite, restored or not). Corrections made before PR 9 merged, second set: `extenddb backup remove-target --account
` ships with PR 9 rather than PR 10 (the PR 10 section's clause says so): the `target_not_empty` rule was written for a catalog restored from a dump that names a restore which finished and was written to since, but its common trigger on MongoDB and Cassandra, which record progress after the write, is an ordinary crash before the first progress record (the whole first data file on MongoDB, the first 10,000 items or 32 MiB on Cassandra: the kill leaves written items and no progress), and without the command this release had no way to remove such a target (DeleteTable refuses a restore target, the FAILED-target duty skips the reason, DeleteBackup and a second restore of the backup are `BackupInUseException`); the first-reach runs of `restore_mongodb_after_insert_many` and `restore_cassandra_before_progress` are that state, so the PR 9a acceptance line, the PR 9a load paragraph, the testing summary and the differences doc's restore-interruption row name it as the third outcome beside "finishes with the right count" and "gone", where they had named two; the command is the one section 4's duty list describes (the maintenance claim, a FAILED record's target whatever the reason, a RUNNING record's dead target, a legacy record's target, the three refusals, no store needed, the audit line), its tests are the engine's maintenance tests and one wire cell on SQLite seeded into the state at the catalog, and the two first-reach kill points run it and assert the target gone and DeleteBackup allowed. The spool sweep lists the spool before the records (the design's rule, which the shipped order broke); the spooled manifest and sidecar are written as the data files are and a spooled pair that fails verification yields to the store's; the resume's first confirmed beat is the takeover's return, taken beside `claim_restore_dead`; a `finish_restore` error is an unknown outcome re-read through the claim; `BackupStoreErrors{get}` counts every `get` the store failed; the request's and the resume's document reads are bounded (4 KiB, 16 MiB); the restored table's size is the source table's; `RestorePostgresBeforeProgressDelete` is gone from the engine's `KillPoint` (one spelling each holds: the four backend names are the storage crate's constants and nothing else); a registration that fails after the request stopped waiting is logged. The restore's re-applied batch on Cassandra relies on the write path rewriting an unchanged item's GSI/LSI row instead of deleting it (#742, fixed on main by #424), and the `restore_cassandra_before_progress` second-reach run reaches ACTIVE with the GSI count exact where before #742 was corrected it was one batch short. PostgreSQL's `finish_restore` deletes the progress rows after the flip's commit and does not fail the restore if that delete errors (logged and swept). A file's PostgreSQL progress row is a plain INSERT from `start_at == 0` and an upsert from `start_at > 0`. Two concurrent registrations of one backup leave one target on Cassandra (the registration LWT's `active_restores = null` is the fence). What bounds a registration's duration per backend (the advisory/`FOR UPDATE` lock, the slot, `transactionLifetimeLimitSeconds`, the LWT timeouts), so the `claimed_at`-minus-`heartbeat_at` gap can never reach the stale window, replaces the earlier "conservative" in 4(e) and in this item, which had the direction wrong: an instant later than the stamp makes the pre-write check admit a beat older than two periods by the gap, so the gap needs a bound, not a name. The SQLite restore's published memory figures are the no-index shape; a table with indexes costs SQLite's page cache (`cache_size`) plus arena fragmentation, bounded, not the restore's buffers (the record's index table: 143 to 225 MiB at 100k and 165 to 172 MiB at 300k with two GSIs on SQLite, 23 to 34 MiB on PostgreSQL), and the 60,000-item shape of the differences doc's in-catalog timings was measured store-backed on PostgreSQL, SQLite and MongoDB (0.67, 1.85 and 3.74 s to ACTIVE), the row now labelling which path each figure is. The MongoDB CI jobs' `EXTENDDB_TEST_BACKUP_STORE` is `filesystem` from this PR (`devtools/run-mongodb-tests` serves a filesystem root), where it was `none`; the Cassandra job's stays `none`; the runner's `--backup-store none` is the one way to run the legacy LSI fallback cell, the in-catalog restore path's one wire cell on MongoDB, which no CI job runs. The PR 11 line owns the two unmeasured cells (the MongoDB and Cassandra 1M restores, the vector cell) and no longer the nightly schedule, shipped here. The kill-point cells as run on the pushed head's hooks binaries (4c353191), every ACTIVE outcome asserting the base table's and the GSI's `Scan` counts: `restore_after_download`, `restore_after_file_commit`, `restore_between_failed_and_removal` and `sigterm` on all four backends; `restore_sqlite_mid_batch`; `restore_postgres_before_progress_delete`, `restore_fenced_out` and `pg_terminate_backend` on PostgreSQL; `restore_mongodb_after_insert_many` at its first reach (`target_not_empty`, then `remove-target`) and its second (the re-application, 24,000 items in three files); `restore_mongodb_before_worker_flip`; `restore_cassandra_before_progress` at its first reach (`target_not_empty`, then `remove-target`) and its second (12,000 items, the GSI exact); the Cassandra points with a SQLite source. Rerun on the build after the last corrections (the first beat before the reply, the Cassandra re-read and fence, the PostgreSQL vector drop): `pg_terminate_backend` as reworked (the lost claim at the next check 20 s after the termination, the record RUNNING with its `(0, 8228)` progress row, the takeover 45.6 s after the termination and 46.7 s after the last beat, ACTIVE 24,000 on base and GSI), `restore_after_file_commit` and `restore_mongodb_before_worker_flip` on MongoDB (resumed from data file 1 after 0 of its items, 3,771 applied by the resume, DONE then the worker's flip, 12,000 exact; DONE before the kill and the restarted worker's flip after it), `restore_cassandra_before_progress` at its second reach (resumed after 10,000, 2,000 applied, 12,000 on base and GSI) and `restore_between_failed_and_removal` on PostgreSQL with a vector backup (the target gone after the stale window, its data table and `_ddb_vec_` relation gone with it, where the pushed head's build left the relation). The matrix's four `--duplicate-key` and eight `--vector` cells ran on the final binaries. Corrections made before PR 9 merged, third set. The job's first heartbeat is sent right after `claim_restore` returns and before the reply, and the pre-write check counts its two periods from that beat's return; a `Lost` or failed first beat fails the record ("the claim was lost before the job started") and answers `InternalServerError`. `extenddb backup remove-target` decides a RUNNING owner dead through `list_dead_restore_jobs(owner_window)` on the database's clock, and the purge's wait decides a backup row's owner live through `list_backup_rows_older_than` on the heartbeat clock; neither reads a host clock. The reconciler takes nothing over and writes no FAILED on its manifest arm once the server's shutdown has begun. Under a maintenance claim a dead record with no progress over a target that holds items is failed `target_not_empty` through `fail_restore_job`, not `OWNER_DEAD`. The reconciler's `target_not_empty` lines print `extenddb backup remove-target --account
`. The restore job's stop line names the heartbeat timer's reason when the timer raised the loss, and the timer logs the reason at WARN. A DONE write whose answer was lost and whose record re-reads DONE under the job's generation ends with "complete (DONE)", not "ACTIVE". A spool document write that fails and a resume that fails before it runs remove this host's entry. The engine's tests of the 60 s bound, the slot held to the job's end and the re-reads' generation checks are cells of their own. Cassandra's progress lightweight transaction re-reads an unknown outcome (applied, one retry, or the lost claim; a second unknown outcome fails the restore), where the earlier text said the restore fails and the reconciler resumes it, which the FAILED write's removal made false. The Cassandra registration's refusal path conditions the target's removal on its own generation; a record a reconciler took over during the DDL keeps its target. PostgreSQL's `abort_restore` drops a vector index's relation with the data table. The PostgreSQL record's `heartbeat_at` is the registration transaction's start. Tests: `test_an_unknown_progress_write_outcome_is_decided_by_a_re_read`, `test_a_registration_taken_over_during_its_ddl_leaves_the_successors_target`, `test_a_mid_file_resume_records_start_at_in_every_batch_progress`, `remove_restore_target_drops_the_vector_relation_with_the_data_table`. The engine's `KillPoint` has five restore variants, named above. SQLite's takeover clock is `strftime('%Y-%m-%dT%H:%M:%fZ', 'now')`, the `NOW` constant of its catalog (4(e) names it). `extenddb backup remove-target` refuses a RUNNING record with a live owner by its own text ("stop it or wait for it"); the DONE and no-record refusals are the two "not a restore target" ones. The published 1M figures are the key-ordered shape's and the record says so beside them, with the same items measured as an unordered backup (SQLite 144 to 164 MiB over idle in 66 to 90 s against 87 to 98 MiB in 27 to 56 s key-ordered; PostgreSQL 16 MiB either way), and SQLite's final `COUNT(*)` is the counting connection's page cache, bounded by `cache_size` (32 MiB) and the same size at 1M and at 3M rows (22 to 28 MiB), where the record had called it the one term that grows with the table from a single measurement. The `pg_terminate_backend` cell of `kill-points.py` pauses the job at `restore_after_file_commit`, terminates the claim session while it is paused (psql must answer a row), waits past one heartbeat period, releases, and asserts the lost-claim stop line, the record RUNNING with its progress row, the takeover no earlier than `stale_job_after_secs` after the owner's last beat and the exact counts; the shipped cell had terminated the session after a fixed sleep and asserted the final count alone, which a 2.6 s restore satisfied without the case ever running. The script's `--vector` switch gives the source table a vector index, and the tamper cell on PostgreSQL then asserts the removed target's data table and `_ddb_vec_` relation gone. `.github/workflows/matrix-nightly.yml` installs `postgresql-client` and checks `psql --version` and the two drivers' import before the cells; it has had no run, since `workflow_dispatch` needs the file on the default branch, so its first run after the merge is the proof of the runner image. The upgrade manual counts migrations 1 to 6 as breaking and migration 7 as declared breaking on PostgreSQL for want of the additive-declaration job (on SQLite, MongoDB and Cassandra it does not apply). The body sentences each of these corrects are edited with this item (section 1's duplicate rule and Decimal128 bullets, section 3's PR 9a interface list, 4(b)'s progress members and metric name, 4(e) and 4(i), 5.2, 5.3's preamble and steps 1 and 3 to 7, section 7's restore memory sentence, section 9, section 10's table, the PR 6, PR 8, PR 9a and PR 9b sections, the PR 10 clause naming what the seed stands in for and the command shipped here, the PR 11 line, the testing summary and the review checklist); earlier items keep their wording, as item 22 requires. With the rebase onto #666's head (2026-11-03): the jobs and the reconciler are not gated by the PostgreSQL serving registration, and 4(e) says what they do in the registration's retry window, GitHub issue #755 being the gate; `backup purge`'s own cycle under the SQLite serve lock fails a dead store-backed restore and removes its target rather than resuming it (4(f)), since the command tells the cycle it holds the claim instead of the cycle reading it through a zero window; the PR 9a load paragraph gives PostgreSQL's batch as 500 rows or 4 MiB, as 5.3 step 5 does; and MongoDB's snapshot open waits, at most 2 s, for the majority-committed snapshot to show a table a `local` read shows ACTIVE, so a store-backed CreateBackup sent as soon as DescribeTable says ACTIVE is not refused as a table that does not exist. With the rebase onto `main` (2026-11-04), after #335, #666 and #681 merged: `restore_progress` is migration 7 (4(i)), declared not applicable in the MongoDB and Cassandra registries #681 added, as on SQLite; 4(e) cites the module doc of `crates/server/src/registration.rs` for what a lost serving registration does, which replaces this item's 2026-11-03 clause on it: the reconciler stops with the server's workers and spawns no resume once its own token has fired, and the running jobs go on, #755 being the gate for them; a restore job joins PR 8's job drain as a backup job does; and `remove-target` releases its serving registration when it exits and is named on MongoDB and Cassandra as on PostgreSQL. With the move onto `main` at e8ca7df3 (2026-11-04), after #424 merged: the Cassandra write path's rule for an unchanged index row (rewritten by its INSERT alone, #742) and its version stamps on every index row are #424's, so this PR no longer changes `sync_indexes` or the GSI worker; the PR 9a load paragraph, 5.3 steps 5 and 6 and this item cite #424 for what a re-applied batch relies on, and a restored item's index rows carry its base row's `version` as #424's write path gives them (`put_restored_item` is that path). After the second review round (2026-11-05): the Cassandra load no longer goes through `put_restored_item`'s lightweight transaction but writes each item's base and index rows in one logged batch `USING TIMESTAMP` a reading taken before the batch's claim check (`put_restored_item_at`), the fence against a batch a stalled writer lands after a takeover (the PR 9a load paragraph); the index rows still carry the base row's `version`, which is now that timestamp. A resume with no recorded progress fails `target_not_empty` only when the target holds more items than the backup's row counts; at or below the total it re-applies the first file with upserts (step 7), so a Cassandra crash before the first progress record resumes to ACTIVE. One instance's sixth takeover of a record fails it (`RESTORE_TAKEOVERS`, step 7). `restore_target_ready` on Cassandra checks each GSI's and LSI's table as well as the data table. The nightly workflow runs every restore kill point of every backend in four jobs (the testing summary); the kill-point script's first-reach cells assert ACTIVE with the exact count and its `--remove-target` flag is gone (the `target_not_empty` state is seeded by `tests/test_backup_job.py`'s `remove-target` cell). 34. (2026-11-04, PR 10 as implemented, #746.) PR 10 is split in three: #746 ships plain `backup sync`, `--prune-missing`, `--adopt-store`, the `sqlite-memory` startup sync and `BackupCatalog::insert_backup_row`; #820 carries `--move-store`, `--forget-store` and `--forget-backup` with `list_backup_rows_by_root` and `rewrite_store`, which rewrite or delete rows of other stores, and the full report of the rows of other roots; #821 carries `migrate-legacy`, whose data path (the frozen reader, `DataFileEncoder`, the item validator) shares nothing with sync but the insert; `remove-target` shipped with PR 9 (item 33). The PR 10 section's line, the PR 9b paragraph (the matrix now adopts with `backup sync`) and the doc-row table's `migrate-legacy` row say so (PR 9b, PR 10, section 10). Neither the Cassandra sweep of orphan `backup_index_by_status` entries nor the MongoDB sweep of the data collections a failed restore-target removal leaves is in any of the three: the code comments that gave them to `backup sync` now say no sweep removes them yet, and the MongoDB error line names the collections for an operator to drop. `insert_backup_row(row) -> bool` inserts the row as given, unowned (`owner_generation` 0), with `completed_at` and `heartbeat_at` on the database's clock, and answers `false` when a row with its key exists: PostgreSQL `INSERT ... ON CONFLICT DO NOTHING` with no conflict target, so both unique indexes (the key and `backup_id` alone) refuse it, on the pool and with no advisory lock; SQLite `INSERT ... ON CONFLICT DO NOTHING` under the write lock and `BEGIN IMMEDIATE`, without a job slot; MongoDB the claim's pipeline upsert filtered on `owner_generation: {$exists: false}` with the row's status, `owner_generation: 0`, and E11000 (the unique `backup_id`) meaning a row exists; Cassandra the `backups_by_arn` lightweight transaction `IF NOT EXISTS`, then the LOCAL_QUORUM batch of the two list rows and the `backup_index_by_status` entry under the row's status (the batch `claim_backup` wrote inline, now shared), an unknown outcome or a `NotMatched` answer re-read at `LOCAL_SERIAL` and settled as inserted when the row read is this one (its entries then written, which completes an insert that stopped between its two writes), as existing when it is another; no row after a lost answer is sent once more, and no row after `NotMatched` answers `false` (a legacy in-catalog backup holds the ARN); a failed batch of the entries is sent once more before the error. The row is AVAILABLE from the transaction on, so a DeleteBackup between the transaction and the batch moves a status entry the batch has not written yet, and the batch then writes an AVAILABLE entry no later write removes; every reader of the status table re-reads the row and skips it, one wasted read each, until a sweep of orphan entries exists. Cassandra has no `backup_id` uniqueness, so its insert-if-absent is per ARN spelling: the same id under another region is another row there, as the trait doc says, and sync closes the gap itself by reading the row under both spellings of the manifest's ARN (this server's region and the manifest's) before the insert and skipping a backup the other spelling has a row for, while a row under the manifest's own spelling is left to the insert (which answers `false` for it or, on Cassandra, completes its own torn insert), so through sync no account holds two rows of one backup id under the manifest's account, table and id on any backend (section 3, 4); on Cassandra a row of that id under another table that the rows read did not return (a row of another root, or one whose list entries are missing) is not seen by that read, and within one run the row check keeps a second prefix of an id the run adopted out. Sync reads the configured root's rows first, every status and every store id the catalog records for the root (`read_store_ids`, then `list_backup_rows` per id), keeping of each only the status, store id and table by account and backup id and the key of each AVAILABLE row of the configured store id (the backends collect the rows whole first, so the read peaks at about 22.5 KiB per row of a five-GSI table, section 7), and checks each listed prefix against that set before it reads anything under the prefix; the sync contract's "checks each prefix against the catalog as the listing reaches it" holds with the rows read once up front, and the insert-if-absent then decides against a CreateBackup that landed after the read and against rows of other roots. The listing is consumed as it streams: the store lists in key order, so a prefix is complete once the listing passes it, and sync holds that prefix's objects and the prefix of each manifest it listed; the filesystem store's own listing is the larger cost, measured in section 7. A listing that fails part-way keeps the rows inserted before it, does not read the prefix it was in, and makes no missing-manifest report. An insert that answers `false` is re-read to say which row it met: one that landed after the read, one of another store, account or table holding the backup id, or on Cassandra a row whose list entries are missing (a CreateBackup's claim that stopped between its two writes; sync's own insert that stopped there is completed by the next sync of the prefix). The prefix check is the dead-owner duty's: `reconciler::verify_prefix` reads the manifest and sidecar within their bounds, checks the digest and the reader's acceptance and every listed data file at its size from the listing sync already holds; the dead-owner probe keeps its comparison of the whole ARN, and sync compares the manifest's account, table and backup id with the path, adopts an ARN of another region, logging it (requests resolve it by the ARN as written), and skips one of another partition than `aws`, since every request refuses such an ARN before it reads a row. `layout.rs` gains `EXPORTS_COMPONENT`, `CHANGELOG_COMPONENT`, `classify_store_key` and `parse_store_prefix`: keys outside every backup prefix are passed over, the reserved components and a third component of another id shape (only the 14-digit shape carries a manifest) are reported as skipped and never read. The adopted row carries the manifest's name, ARN, size and item total and `source_table`, the id's instant as `created_at`, and the configured store's identity (section 4, PR 10). Rows of other roots are reported in part: sync prints the prefixes whose backup id a row of another root holds (the read under the ARN's spellings, or the insert's re-read). The rows of the configured root under another store id are not a case sync meets: a store whose rows carry another id than its file, or more than one id, is unhealthy, and sync refuses an unhealthy store. A listing of every root's rows needs `list_backup_rows_by_root`, which is #820's, so until then 4(g)'s "rows for another store are reported" holds for that one case (4(g)). Every sync takes the maintenance claim where it runs beside servers (PostgreSQL, MongoDB, Cassandra), plain sync with no wait for running jobs: its insert-if-absent orders it against a CreateBackup, and the claim orders it against `purge`, whose `purge_row` removes a dead CREATING row before it deletes the prefix, which can hold a complete manifest (a job that died after its manifest); a sync between the two steps would adopt bytes purge then deletes. While the claim is held the jobs refuse, as during purge (4(f)), and a sync killed while it holds the claim keeps the jobs of every instance refused until its heartbeat is `stale_job_after_secs` old. The restore's line for a row whose prefix lacks its sidecar says to put the prefix back; only a missing manifest names `--prune-missing`, the one case it removes. On SQLite it opens the catalog as `serve` does, under the serve lock (exit 6 beside a server, the refusal purge gives), after the dead-owner resolutions purge runs, so a DELETING row a stopped server left is finished and a dead CREATING row resolved before sync reads the rows. `--prune-missing` and `--adopt-store` also wait for running jobs as purge does; every sync installs purge's signal handling, checks it before each prefix and exits 1 "stopped" on a signal, releasing the claim it holds and its serving registration. `--prune-missing` removes each AVAILABLE row whose manifest is gone outright, through `BackupCatalog::forget_available_backup_row` (added here on the four backends: under the row's lock as `start_backup_delete` takes it, the status check and the in-use rule with a delete of the row, and on Cassandra its index entries after it), and deletes nothing in the store. The manifest probe sees only the newest AVAILABLE row's manifest, so a host-local root that holds the newest backup passes it while it lacks older ones; with no tombstone left, no deletion duty on the host that holds those bytes acts on them, but the row is gone for every instance: the backup is BackupNotFound on every instance until a sync on the host that holds the bytes adopts it again, which is the host-local acceptance line as amended here. The objects a removed row's prefix still holds stay until an operator deletes the prefix by hand: `purge` deletes only the prefixes of rows, and every later sync reports the prefix as not a complete backup. `--prune-missing` exits 1 when it leaves a row a later run may remove (a store or catalog call failed, the manifest's `head` included, which keeps the row; or a restore is reading the backup, "run the command again once the restore ends") and 0 when the rows it leaves have their manifest back or were moved by another writer. An unhealthy or `checking` store refuses plain sync and `--prune-missing`, exit 1, naming the problem and changing no row. Sync exits 1 when a prefix was skipped for a store read or catalog call that failed (the run is incomplete; the next run adopts it), and a startup sync that met one stays pending, its skips logged at WARN. So when the newest AVAILABLE row's own manifest is gone the store is unhealthy and neither command runs; the way out in this release is to put the prefix back (#820's `--forget-backup` is the other). The serving registration is named `extenddb backup sync`, and the catalog refusals exit 6 and 3 as the other commands' do (4(f), PR 10). `--adopt-store` is `store_identity::adopt`: with the claim held it rewrites `extenddb-store.json` with a plain `put`, the file's `catalog_id` and `backend` changed and its `store_id` and `created_at_ms` kept, reads it back, and syncs with the adopted identity; it logs an `extenddb::audit` intent line before the put and the audit line after the re-read with its outcome (adopted, or race lost), so a put whose caller stopped waiting is on record too. The re-read detects another adopter's put that landed between this put and the re-read (the store trait has no conditional replace); the command reads the file once more after the sync and fails naming the owner when it changed, which detects one that landed during the sync; a put after both reads is found by this catalog's next check. It refuses when this catalog's rows for the root carry another store id, and on a root this catalog owns it says so and syncs. A later check of the previous owner that can read its catalog finds the file naming another catalog, at the earliest one reconciler interval after its cycle in progress ends; until then it acts on the root, except where its deletion fence stops it. `StoreFence` re-reads the file (one `get` through the store's bounded calls, under its check bound and its one call slot on a filesystem root) right before each `delete_prefix` of a backup's objects (`finish_backup_delete`, which DeleteBackup and the reconciler's DELETING duty run, the backup job's failure path, the FAILED-prefix and DELETED-prefix duties, purge), before DeleteBackup's DELETING write and before purge's removal of a dead CREATING row or its DELETING write, before the reconciler removes a tombstone, and before a CreateBackup's row and first put; when the file names another catalog it refuses, records `ForeignCatalog` so the store is unhealthy at once (and a file naming this catalog under another store id than the store's check settled, a root replaced by another root of the catalog, is refused and recorded as `IdMismatch`; DeleteBackup and CreateBackup answer the unhealthy store's `ValidationException`, and the cycle re-tests the store's health after each store duty and runs none after the refusal, the dead-owner duty included), and a file it cannot read, or a root with no file, refuses the call and leaves the health to the next check, except at DeleteBackup's read before the DELETING write, where it is a store failure: the request writes DELETING and answers DELETED, as for a deletion the store does not answer (revision item 30), and the read before the deletion refuses it. A refusal before the row is changed leaves it as read, so DescribeBackup does not answer a refused DeleteBackup with DELETED; one at the read before the deletion (an adoption between the two reads) leaves the row DELETING with nothing deleted and DeleteBackup answering DELETED (revision item 30), which this catalog's reconciler finishes only if this catalog owns the root again. The restore deletes nothing in the store, so it has no fence, and neither has `--prune-missing`'s removal of a row, which deletes no object. A write already past the fence's read lands, so every server of the catalog the file names is stopped before an adoption, and a new in-memory server is started only after the old one has exited; the admin guide, the refusal and the command's output say so (4(c)). For `backup sync` a defaulted SQLite root whose file names another catalog while this catalog has no rows for it is configured and unhealthy (`Caller::SyncCommand`) instead of the warning that leaves it unconfigured (item 29), so `--adopt-store` can adopt the root a catalog re-created by `destroy` and `init` finds; the server and purge keep the warning (4(c), section 7, PR 7). The identity file gains `ephemeral_catalog` (`#[serde(default)]`, so the format stays version 1 both ways), written `true` by a server whose SQLite database is in memory (`database_file()` names no file: the state, not the `sqlite-memory` cargo feature) and `false` by every other writer, an adoption included. The adoption of a root an in-memory catalog owned is part of `store_identity::establish` when this catalog is itself in memory and the file carries the marker, and never of `recheck`, so two in-memory servers on one root do not take it from each other each cycle; a root whose file names a durable catalog stays unhealthy for an in-memory server, as 4(c) requires. The startup sync runs on every start of an in-memory catalog with a store configured, not only after an adoption (on a fresh root it is one listing): for a filesystem store in `serve` before the routes serve; for an S3 store, and for a startup sync not finished, in the reconciler's cycle after a healthy check. Every run is bounded at 60 s (`sync::STARTUP_SYNC_TIMEOUT`), in `serve` and in the cycle, where the duties after it wait at most that bound; a run that stops before it listed the whole root, reaches the bound (the rows it inserted stay, but every run lists the root from its first key, so the sync completes only while listing the whole root fits in the bound; past it the sync stays pending and is retried every cycle), or skipped a prefix for a store or catalog error stays pending. On a filesystem store a run abandoned at the bound does not stop its listing (the walk has no cancel flag): it runs to its end on the blocking pool, holding the listing it builds, and the next cycle starts another walk; and `serve` answers nothing, `/health` included, until the startup sync before the routes ends, while a signal in that time ends the process at once. Until a run finishes, requests for the earlier backups answer `BackupNotFoundException`, `/health` possibly already healthy; the log line that reports the adopted count ends that window, and for a prefix that line reports as having met a store or catalog error, a later cycle's line does. It takes no claim: the process is the catalog's only user (4(c), PR 7, PR 10). Every `extenddb backup` command refuses an in-memory SQLite database before it opens a catalog, exit 1, with a sentence saying why (a location that does not parse is reported with its parse error instead, as `serve` reports it): it would open a new, empty catalog that ends with the command, and a writing command would hand an adopted root to a catalog nothing can open again; this is the sync contract's "the backup CLI commands do not exist there" (PR 10). The matrix (`devtools/backup-restore-matrix`) copies the source's prefix under the target's root and runs `extenddb backup sync` on the target (on SQLite with the target's server stopped) in place of the hand-written row, so a cell proves the adoption and the restore; `--adopt-store` points the target's store at the source's root, adopts it and checks that the source's server then reports the store owned by another catalog, and the nightly runs that cell from SQLite to PostgreSQL and back (PR 9b). 4(c), 4(e), 4(f), 4(g), decision 6, section 7's memory paragraph and the PR 10 contract are edited with this item. + +35. (2026-11-05, PR 11 as implemented, #747.) The CI filters: `--filter "not import_export"` came with the PostgreSQL integration job's first version (687e46ff) and was copied into the SQLite job (556c501e); neither commit gives a reason. A foreground server wrote a PID file in every tree up to e0d6a0af, which made foreground mode write none by default and added `--write-pid-file` to the job's start in the same commit; 556c501e came after e0d6a0af, and its SQLite job passes the flag. So the restart `devtools/run-tests` makes for the import and export tests could stop the job's server in every tree, and the filter was not working around it. The MongoDB and Cassandra jobs already run `tests/test_import_export.py` unfiltered (`--pytest --comprehensive --parallel`). The expression selects out the ten tests of that file and nothing else (none of the comprehensive suite, the CLI lifecycle list or the backup-store step); run the job's way on release builds of this tree they passed 10 of 10 on SQLite and on PostgreSQL, so both filters are dropped and no narrower one is kept. The open cells (testing summary) are measured in `15-backup-restore-bench-pr11.md`, in `-pr9.md`'s method, none left open; the Cassandra cells ran on a single 4.1 node of their own with the Cassandra integration workflow's service settings (`PasswordAuthenticator`, a 1 GiB heap), since a node running `AllowAllAuthenticator` cannot hold the role `extenddb init` creates. Later runs on the same host with other harnesses, on release builds of this change's sources (04c29155, an earlier version of this commit with the same Rust sources, and 30553b38, whose Rust sources differ only in the help text of two `init` options), reproduce the cells in kind and are in the record, with a control of items of 2.5 KB strings run on this commit's sources. Findings beyond the figures: SQLite's vector restore under glibc's default arenas is higher at 1M than at 100k (552 against 354 MiB over idle; the 1M run levelled off after about 150 s, and no other size was measured) and is 195 and 204 MiB under `MALLOC_ARENA_MAX=1` (flat within about 25 MiB over the measured runs), the allocator's term section 7 describes for large items; that memory is the restore's write batch (up to 10,000 items or 32 MiB of item text) and decode chunks, not the vector index, and grows with the item's size up to the batch bound whatever the attribute types (100k items of about 2.5 KB of strings held 253 MiB, 130 MiB under one arena), lists of numbers decoding larger at the same text size (345 to 424 MiB and 184 to 195 MiB at 100k items of 128 numbers, with or without the index; 20,000 items of 1,536 numbers 669 MiB and 301 to 316 MiB under one arena), while on PostgreSQL the index adds 27 to 53 MiB (five pairs of runs with and without it, each from one harness; a pair whose two runs came from different harnesses is left out); one arena cost the 1M restore 32% in duration and two 100k restores running at once 131% (57.5 against 24.9 s); the vector index is about 20% of the 100k restore's duration on SQLite and 33 to 47% on PostgreSQL (the same pairs); MongoDB's restore becomes `ACTIVE` at the backfill worker's next cycle after the load (`GSI_BACKFILL_INTERVAL`, 5 s, which the restore's end does not wake), so its duration is the load's plus up to 5 s, 3.9 to 8.9 s at 100k over five runs; the peaks of one cell varied by up to about 20% between runs; and a whole-table `SearchVectors` on SQLite is the exact scan of `crates/storage-sqlite/src/vector_search.rs` (5.5 to 15.8 s over 1M 128-dimension vectors in seven runs, 0.24 s on PostgreSQL). `docs/manuals/14-backup-and-restore.md` is the operator's account derived from this document, which remains the record (section 10's first row): the strategy, RPO and RTO per store kind with the three records' figures and their host class, the configuration (pointing at the admin guide's rows and the store crate), securing the store, what a running backup costs each database, the restore drill, catalog-loss recovery with `backup sync`, `--adopt-store` and the in-memory startup sync, legacy backups (with `migrate-legacy`, #821, not shipped, the restore-and-back-up-again path), what a restore preserves, and ImportTable and export interchange with Amazon DynamoDB (a store-prefix import is #748). Each PR 11 acceptance statement is in it. Figures measured for it on the same host class: a 25 s PostgreSQL backup beside 12,156 overwrites of 200-byte items in another table grew that table from 0.4 to 4.4 MiB, its dead row versions "not yet removable" by a `VACUUM` during the backup (0.5 MiB for the same writer with no backup); on SQLite the WAL grows by the pages each write changes, which depends on the indexes: a 22.8 s backup beside 15,472 overwrites of a table with no secondary index grew the `-wal` file from 4.0 to 61.1 MiB (about 3.9 KB per write), and 13,996 overwrites of a table with two GSIs during a 25.6 s backup from 4.1 to 1,294 MiB (82 to 97 KB per write over four runs, about 20 to 24 pages: an asynchronous GSI write goes through `gsi_pending` and is applied in later transactions); the same writers with no backup left it at 4.0 and 4.1 MiB, and the file keeps its grown size after the backup and after a stop. The record's "Figures measured for the manual" section carries these runs' method (the writer, the relation and file measured, the `VACUUM (VERBOSE)` 10 s in), build and load, with their reproductions on 04c29155. Its process-level statements were checked against release builds of this tree on SQLite and PostgreSQL: the root placement refusals and their texts and an absent root's, the drill, a re-created catalog's `owned by another catalog` health with `[backup.store]` configured and, on file-backed SQLite with the default root, the logged warning, no `backup_store` component and in-catalog `CreateBackup` until the adoption, the accounts re-created with `manage create-account --account-id` and the old key pairs kept with `manage import-access-key`, `--adopt-store` exiting 6 beside a SQLite server and adopting once it stopped, the in-memory startup sync line and, on the `sqlite-memory,dev-mode` build, the adopted backups answering `AccessDeniedException` until their account was re-created with its old id, the in-catalog create's warning and the restore of its backup through a store-configured server, an S3 store at moto with the default spool, `backup sync` exiting 6 beside a SQLite server, ImportTable refusing a gzip file and a `FileSource.Path` through a symbolic link and importing a decompressed backup data file, the export's `FilePath` and `S3Prefix`, and on PostgreSQL the claim session found by its lock key (`hashtextextended('backup:', RESTORE_LOCK_SEED)` in the catalog database, distinct from the `extenddb serve` registration session) whose termination stopped the job at its next heartbeat and left the row to the reconciler, which failed it once stale. Checked on release builds of 04c29155 (this change's Rust sources) as well: an off-host copy with loose modes refused by `backup sync` (exit 1, naming the directory) and its restore failing, the store unhealthy only when the copy is the newest backup; the registration session's termination failing every store-backed backup the server ran (reason `shutdown`); `--prune-missing` waiting on a backup that failed within `stale_job_after_secs`; a catalog database restored alone beside a newer data database (tables created since absent and empty when created again, tables deleted since listed and answering `ResourceNotFoundException`); both symbolic-link refusals of `FileSource.Path`; an on-demand table's GSIs reported with 0 and 0 throughput in `SourceTableFeatureDetails`; and the PostgreSQL pair dumped with the server stopped under `umask 077` (files 0600) and restored into new databases created with `createdb -O` by the admin role, after which `CreateTable` and `PutItem` of a new table work, while a restore over the existing databases restores nothing and one into databases the admin role owns breaks `CreateTable` ("permission denied for schema public"). On release builds of this commit's sources, on PostgreSQL 16 containers: the entry's procedure as it now stands (an older `pg_dump` refused, "aborting because of server version mismatch", exit 1, leaving an empty file, which `psql -v ON_ERROR_STOP=1` restores with exit 0, after which the server refuses the catalog as not initialized; both dumps exit 0 and non-empty; the old databases renamed aside, the restored server reading the table and its GSI and creating a table, then the old databases dropped), and on a new, empty server `createdb -O` refused until the application role was created with the dump's password, the restore, and the server, whose catalog still named the old server's address, connecting its data pool there: it put a new table's items into the old server's data database while that server ran, failed to start once it was gone, and served the restored table, with `CreateTable` and `PutItem` landing on the new server, once `data_database_connection_string` was set to the new address; and an access key imported through the management API from a request body read from a file. A catalog restored from a dump whose newest `AVAILABLE` backup was deleted after the dump keeps its store unhealthy, and this release has no command that clears that without a copy of the prefix (`--prune-missing` refuses while the store is unhealthy); the manual says to take the dump after deletions. The admin guide's Data Recovery entry named the PostgreSQL data database `extenddb_catalog_data`; `init` names it after the catalog without its `_catalog` suffix (`extenddb` by default) and records it as `data_database_name` (the connection string as `data_database_connection_string`), and the entry, the architecture, design, quickstart and security-model guides, `AGENTS.md`, the troubleshooting guide and the `extenddb` skill's runtime-symptoms reference (the `pg_stat_activity` query), the technical-debt register's A-1 entry, the storage-extension guide's database list and `init --help` now say so, with each backend's databases; the quickstart's `init` table gives `--pg-user` (the admin role, `$USER` then `postgres`), `--pg-pass`, `--extenddb-user` and `--extenddb-pass` with their defaults, and its sample lines show migration 7; a catalog name without the suffix needs `--data-db`, since `init` otherwise derives the catalog's own name, creates it once and fails on the second create, leaving an empty database (not changed here). On PostgreSQL the entry, the security model, the deployment guide's checklist and the manual treat the catalog and data databases as one copy (both dumps with the server stopped, or a base backup), with `umask 077` before the dumps, since the catalog dump holds the encryption key and the application role's password, and the restore goes into new, empty databases owned by the application role, with the old databases renamed aside until the restored server has read every table, the `pg_dump` of the server's major version and both dump files checked, and on a new PostgreSQL server the application role created first and, at another address, the recorded data-database address set; the entry's SQLite row gives the restore (the current `-wal` and `-shm` removed first), and the upgrade manual's dumps take `umask 077` and its rollback restores the pair; the manual's catalog-loss section says what a catalog older than the tables gives and that tables backed up since restore once `backup sync` lists their backups. ADR-0019 and the RFC section are superseded in part, not whole: CreateBackup on Cassandra still writes their in-catalog format, before it returns (section 3). ADR-0019 changes only its Status line, as `docs/adr/README.md` allows, to the "Superseded in part by [document]" form the README's process now names for a decision a later design replaces in part, and section 10's closing paragraph states what that Status covers, including that the reconciler now removes an interrupted in-catalog create; the RFC's section carries a note, and its section 4 and Known Gaps lines on synchronous operations and on restored indexes carry notes. Edited with this item: section 7's memory paragraph (the restore-memory and vector-index sentences), section 10's table row and closing paragraph, the PR 11 line and the testing summary's performance paragraph. diff --git a/docs/design/README.md b/docs/design/README.md index f0ee934ed..870933166 100755 --- a/docs/design/README.md +++ b/docs/design/README.md @@ -22,6 +22,7 @@ This directory contains the authoritative design documents for ExtendDB (extendd | 15 | [Backup and Restore](15-backup-restore.md) | Overview of the off-host backup series: the invariants, the per-operation contracts and a short section per later PR; the format, store and snapshot documentation lives with their crates (`crates/core/src/backup_format`, `crates/backup-store`, `crates/storage/src/snapshot.rs`) | | 15b | [Backup memory and duration record, PR 8](15-backup-restore-bench-pr8.md) | The measurements section 7 and revision item 31 of the design cite: peak RSS and duration of the 100k, 1M, worst-expansion and incompressible backups per backend, the allocator series, the PostgreSQL page's cost | | 15c | [Restore memory and duration record, PR 9](15-backup-restore-bench-pr9.md) | The measurements section 7 and revision item 33 of the design cite for the restore: peak RSS and duration of the 100k, 1M, worst-expansion and large-item restores on SQLite and PostgreSQL, with and without one allocator arena, the terms the code bounds and the sizing sentence | +| 15d | [Backup and restore record, PR 11](15-backup-restore-bench-pr11.md) | The measurements the testing summary cites for the MongoDB, Cassandra and vector cells: peak RSS and duration of the 100k and 1M restores on MongoDB and Cassandra and of the backup and restore of 100k- and 1M-item tables with a 128-dimension vector index on PostgreSQL and SQLite, with and without one allocator arena; the 100k and 1M MongoDB backups; later runs that attribute SQLite's restore memory to the write batch, which grows with the item's size whatever the attribute types (the same items with no vector index, items of a 2,290-character string, 20,000 items of 1,536 numbers, two restores at once, the peak's spread over nine runs), give the PostgreSQL vector index's share from five same-harness pairs and MongoDB's restore duration with the backfill worker's cycle; SQLite's WAL growth beside a backup with and without two GSIs; and the figures measured for the backup manual (PostgreSQL dead row versions and SQLite WAL growth during a backup) with their reproductions | ## How to Use These Docs diff --git a/docs/differences-from-dynamodb.md b/docs/differences-from-dynamodb.md index 077062530..1f41a44a8 100755 --- a/docs/differences-from-dynamodb.md +++ b/docs/differences-from-dynamodb.md @@ -108,7 +108,7 @@ Behaviour stated as measured, and every number below, was observed with release | Area | DynamoDB | ExtendDB | |------|----------|------| | What a restore carries across | Key schema, attribute definitions, items, secondary indexes, billing mode and provisioned throughput, on-demand throughput (table and GSI level), SSE settings. Table class is not carried: a restore is `STANDARD` whatever the source's class (measured). Stream settings, TTL, tags, auto scaling, IAM policies, and CloudWatch settings must be reapplied by hand. `DescribeTable` on a freshly restored table reports `ItemCount` 0 and `TableSizeBytes` 0 (measured on 15 restored tables, one of them ten minutes after `ACTIVE`) | The same set on all four backends, with one gap: the table's on-demand throughput is carried, a GSI's is not (ExtendDB records no per-index on-demand maximum, and `OnDemandThroughput` is not a member of its GSI definition). Table class is not restored: a `STANDARD_INFREQUENT_ACCESS` source restores as `STANDARD`, as on the service. Streams, TTL, tags, and deletion protection are not restored. `DescribeTable` on a restored table reports `ItemCount` 0 and `TableSizeBytes` 0 when the table goes `ACTIVE`, as the service does (measured: 0 and 0 on 15 restored tables, one of them ten minutes after it went `ACTIVE`), until the statistics pass next rewrites both from the restored rows *(from the code)*. A backup written before catalog 0.0.4 (PostgreSQL, SQLite, Cassandra) has no recorded table definition and restores with its keys, items, and billing mode but no secondary indexes. A provisioned one restores with 5/5 throughput on PostgreSQL and SQLite and with its recorded throughput on Cassandra | -| How `RestoreTableFromBackup` runs | Returns at once with the target `CREATING`; the service copies in the background | The same on all four backends: the response carries the target `CREATING`, and the items are copied by a background task in the server, so a client that times out or disconnects does not stop it. Once the request has been authenticated and authorized, the rest of it, from reading the backup to registering the restore, runs in a task of its own, so a client that disconnects after that does not cancel it either: the restore runs and the client finds the target with `DescribeTable`. A client that disconnects while the request is still being authenticated or authorized cancels it, and nothing is created *(from the code)*. Measured with a client whose 1 ms read timeout closed its connection before any response arrived: on all four backends the restore was registered and finished. Measured *(in-catalog restore, an earlier build)* with a 60,000-item backup of a table with no secondary indexes: response in 0.032 s (PostgreSQL), 0.012 s (SQLite), 0.053 s (MongoDB), 0.151 s (Cassandra); `ACTIVE` after 1.5 s, 2.1 s, 3.6 s, and 11.2 s. Measured *(store-backed restore, this release)* with the same shape (60,000 items of 200 bytes, composite key, no secondary index) from a filesystem store on loopback: response in 0.051 s (PostgreSQL), 0.016 s (SQLite), 0.055 s (MongoDB); `ACTIVE` after 0.67 s, 1.85 s and 3.74 s (on MongoDB the job's DONE at 1.8 s and the index backfill worker's flip after); 1M items of 200 bytes in key order (the server's own backups) `ACTIVE` after 9.5 s on PostgreSQL and 26.8 s on SQLite, the same items as an unordered backup (a MongoDB source, a multi-part-key table) after 17.5 s and 66 to 90 s (`docs/design/15-backup-restore-bench-pr9.md`, which has the host and its load); the store-backed response includes the sidecar and manifest reads and its `ACTIVE` time the download and the per-file verification, and Cassandra as a store-backed target was measured at 12,000 items only (the kill points: a resumed restore of 12,000 items of 4,000 bytes in two data files reached `ACTIVE` 8.3 s after its takeover on a single-node container). On Cassandra the response waits for the target table's schema to be created: 0.75 to 1.3 s with five restores running at once *(earlier build)*, which is longer than a client read timeout of 1 s. On every backend the request waits at most 60 s (`RESTORE_REGISTRATION_TIMEOUT`) for its background task to register the `CREATING` target of a store-backed restore (a bound against a connection pool that has stopped answering; the manifest and sidecar reads that precede the registration are not under it); past that it answers `InternalServerError` while the task goes on, so the target can appear after the error, holding the backup in use (`BackupInUseException` to `DeleteBackup`) and the name until the restore ends; `DescribeTable` is where a client finds it *(from the code)*. Because that 60 s bound equals botocore's default read timeout, a client whose first attempt times out while the registration is still in flight (a backup's `backup_index` row held under a slow lock) retries: a retry that names the same backup and target waits for the in-flight registration and is answered its `CREATING` description rather than registering a second target, and a restore of a *different* backup into the same name while the first is in flight is `TableInUseException` *(from the code)*. The wait is per server process: each server keeps its own list of in-flight registrations, so a retry waits for the first attempt's registration only when it reaches the server that holds it; a retry that reaches another server behind a load balancer finds the name free until the registration commits and then `TableInUseException` (the registration took the name) or, once the target exists, the same `CREATING` description from the catalog *(from the code)*. The job permit is the last of the request's checks: a restore that finds every `[backup] max_concurrent_jobs` slot held (default 4; backups and restores together, per server process) waits up to 30 s for one and is then `LimitExceededException` ("Too many backup and restore jobs are running on this server (at most N at once, the [backup] max_concurrent_jobs bound); retry later"), creating nothing, where the service accepted five restores sent together (measured on DynamoDB); on the store-backed path the permit is taken before the manifest is read, after the catalog row's state, the target name and the backup's in-use rule are checked, in that order (a restore of a backup into its own CREATING target is `TableInUseException`, measured on DynamoDB). A restore runs inside the server process; if the process stops, see the abandoned-restore row | +| How `RestoreTableFromBackup` runs | Returns at once with the target `CREATING`; the service copies in the background | The same on all four backends: the response carries the target `CREATING`, and the items are copied by a background task in the server, so a client that times out or disconnects does not stop it. Once the request has been authenticated and authorized, the rest of it, from reading the backup to registering the restore, runs in a task of its own, so a client that disconnects after that does not cancel it either: the restore runs and the client finds the target with `DescribeTable`. A client that disconnects while the request is still being authenticated or authorized cancels it, and nothing is created *(from the code)*. Measured with a client whose 1 ms read timeout closed its connection before any response arrived: on all four backends the restore was registered and finished. Measured *(in-catalog restore, an earlier build)* with a 60,000-item backup of a table with no secondary indexes: response in 0.032 s (PostgreSQL), 0.012 s (SQLite), 0.053 s (MongoDB), 0.151 s (Cassandra); `ACTIVE` after 1.5 s, 2.1 s, 3.6 s, and 11.2 s. Measured *(store-backed restore, this release)* with the same shape (60,000 items of 200 bytes, composite key, no secondary index) from a filesystem store on loopback: response in 0.051 s (PostgreSQL), 0.016 s (SQLite), 0.055 s (MongoDB); `ACTIVE` after 0.67 s, 1.85 s and 3.74 s (on MongoDB the job's DONE at 1.8 s and the index backfill worker's flip after); 1M items of 200 bytes in key order (the server's own backups) `ACTIVE` after 9.5 s on PostgreSQL and 26.8 s on SQLite, the same items as an unordered backup (a MongoDB source, a multi-part-key table) after 17.5 s and 66 to 90 s (`docs/design/15-backup-restore-bench-pr9.md`, which has the host and its load); the store-backed response includes the sidecar and manifest reads and its `ACTIVE` time the download and the per-file verification, and Cassandra as a store-backed target was measured at 12,000 items (the kill points: a resumed restore of 12,000 items of 4,000 bytes in two data files reached `ACTIVE` 8.3 s after its takeover on a single-node container) and at 1M items of 200 bytes in key order, `ACTIVE` after 49.0 s, and MongoDB's restore of its own unordered backup of 1M such items `ACTIVE` after 33.8 s, which another run may exceed by up to 5 s, since MongoDB's backfill worker flips the table at its next 5 s cycle after the load (`docs/design/15-backup-restore-bench-pr11.md`, which has the host and its load). On Cassandra the response waits for the target table's schema to be created: 0.75 to 1.3 s with five restores running at once *(earlier build)*, which is longer than a client read timeout of 1 s. On every backend the request waits at most 60 s (`RESTORE_REGISTRATION_TIMEOUT`) for its background task to register the `CREATING` target of a store-backed restore (a bound against a connection pool that has stopped answering; the manifest and sidecar reads that precede the registration are not under it); past that it answers `InternalServerError` while the task goes on, so the target can appear after the error, holding the backup in use (`BackupInUseException` to `DeleteBackup`) and the name until the restore ends; `DescribeTable` is where a client finds it *(from the code)*. Because that 60 s bound equals botocore's default read timeout, a client whose first attempt times out while the registration is still in flight (a backup's `backup_index` row held under a slow lock) retries: a retry that names the same backup and target waits for the in-flight registration and is answered its `CREATING` description rather than registering a second target, and a restore of a *different* backup into the same name while the first is in flight is `TableInUseException` *(from the code)*. The wait is per server process: each server keeps its own list of in-flight registrations, so a retry waits for the first attempt's registration only when it reaches the server that holds it; a retry that reaches another server behind a load balancer finds the name free until the registration commits and then `TableInUseException` (the registration took the name) or, once the target exists, the same `CREATING` description from the catalog *(from the code)*. The job permit is the last of the request's checks: a restore that finds every `[backup] max_concurrent_jobs` slot held (default 4; backups and restores together, per server process) waits up to 30 s for one and is then `LimitExceededException` ("Too many backup and restore jobs are running on this server (at most N at once, the [backup] max_concurrent_jobs bound); retry later"), creating nothing, where the service accepted five restores sent together (measured on DynamoDB); on the store-backed path the permit is taken before the manifest is read, after the catalog row's state, the target name and the backup's in-use rule are checked, in that order (a restore of a backup into its own CREATING target is `TableInUseException`, measured on DynamoDB). A restore runs inside the server process; if the process stops, see the abandoned-restore row | | `RestoreSummary` | The restore response and `DescribeTable` while `CREATING` report `SourceBackupArn`, `SourceTableArn`, `RestoreDateTime` (the backup's `BackupCreationDateTime`, not the time the restore ran), and `RestoreInProgress=true`. Once the table is `ACTIVE`, `DescribeTable` omits `RestoreSummary` | The same on all four backends. No response that describes a table (`CreateTable`, `UpdateTable`, `DeleteTable`, `DescribeTable`, `RestoreTableFromBackup`) carries `RestoreSummary` unless the table is `CREATING`; what the service's `UpdateTable` returns for a restored table was not measured | | Reads on a restore target | A `GetItem` or `Scan` on the target while `CREATING` is `ResourceNotFoundException`; the first read after `DescribeTable` reports `ACTIVE` succeeds (measured) | The same on all four backends. The server's key-info cache stores a negative entry (5 s TTL by default) for a name the catalog does not have, and does not store one for a name the catalog has in a state other than `ACTIVE`: on a not-found answer it asks the catalog once more with `DescribeTable`, and authorization records the name as missing in the key-info snapshot it hands the engine, so the engine does not look it up again. A `GetItem`, `PutItem`, `UpdateItem`, `DeleteItem`, `Query`, or `Scan` on the target while it is `CREATING` therefore costs exactly one `table_key_info` and one `describe_table` catalog load, uncached, per request (on SQLite five SQL statements: one for the key info, four for `DescribeTable`; on PostgreSQL `describe_table` is two queries), plus one `describe_table` on the first such request for the throttle registry; the first read after `ACTIVE` loads once more and is then cached. Measured on SQLite with a `GetItem` loop on a `CREATING` restore target of 100,000 items: 40 requests refused, 40 key-info and 41 `DescribeTable` loads in the server's statement log; with the hand-off to the engine removed, 80 and 81. In an earlier run of the fidelity test the first `GetItem` after `ACTIVE` answered in 3 ms after 72 refused reads. | | `RestoreTableFromBackup` overrides | Current request members are `BillingModeOverride`, `ProvisionedThroughputOverride`, `GlobalSecondaryIndexOverride`, `LocalSecondaryIndexOverride`, `SSESpecificationOverride`, `OnDemandThroughputOverride`, and `VectorIndexOverride`; DynamoDB applies them to the restored table | See the overrides table below. A refused override is refused before the target table is created | diff --git a/docs/manuals/01-architecture-guide.md b/docs/manuals/01-architecture-guide.md index 93ccf3ba9..53f67a497 100755 --- a/docs/manuals/01-architecture-guide.md +++ b/docs/manuals/01-architecture-guide.md @@ -203,7 +203,7 @@ commands at the cost of needing a writable `run_dir`. extenddb uses a dual-database architecture: - **Catalog database** (e.g., `extenddb_catalog`): Stores table metadata, account/user/group/role/policy definitions, access keys, settings, stream metadata, and metrics. Shared across all accounts. -- **Data database** (e.g., `extenddb_catalog_data`): Stores user items, GSI/LSI data, and stream records. Each table gets its own PostgreSQL table. +- **Data database** (`extenddb` for the default catalog `extenddb_catalog`: the catalog's name without its `_catalog` suffix): Stores user items, GSI/LSI data, and stream records. Each table gets its own PostgreSQL table. On PostgreSQL, SQLite, Cassandra and MongoDB the catalog is versioned by its migration ledger, the `extenddb_migrations` table (on MongoDB a collection, in the catalog database and in the data database): the catalog version is the highest migration recorded in every store, and `serve` reads the catalog's status at startup and refuses a catalog that is behind it or carries a breaking migration it does not know. On Cassandra every account keyspace has a ledger beside the catalog keyspace's. A server that refuses its catalog prints the status and the command that resolves it; for pending migrations that is `extenddb migrate --yes`. diff --git a/docs/manuals/02-design-guide.md b/docs/manuals/02-design-guide.md index 5d7b7ea2f..bef8226e5 100755 --- a/docs/manuals/02-design-guide.md +++ b/docs/manuals/02-design-guide.md @@ -11,9 +11,9 @@ examples below name the PostgreSQL databases; other backends map the same two roles onto their own structures (see backend-specific documentation). - **Catalog** (e.g., the `extenddb_catalog` PostgreSQL database): All metadata — table definitions, indexes, accounts, IAM entities, settings, stream metadata, and schema history. -- **Data** (e.g., the `extenddb_catalog_data` PostgreSQL database): User item data. Each DynamoDB table maps to a PostgreSQL table. GSI and LSI data are stored in separate PostgreSQL tables. +- **Data** (the `extenddb` PostgreSQL database for the default catalog `extenddb_catalog`): User item data. Each DynamoDB table maps to a PostgreSQL table. GSI and LSI data are stored in separate PostgreSQL tables. -On PostgreSQL, the data database connection string is stored in the catalog's `settings` table under the key `data_database_url`. This allows the catalog and data databases to live on different PostgreSQL instances. +On PostgreSQL, the data database connection string is stored in the catalog's `settings` table under the key `data_database_connection_string`, and its name under `data_database_name`. This allows the catalog and data databases to live on different PostgreSQL instances. ### Catalog Tables diff --git a/docs/manuals/04-quickstart-setup-guide.md b/docs/manuals/04-quickstart-setup-guide.md index 08333d9dd..66440a3ea 100755 --- a/docs/manuals/04-quickstart-setup-guide.md +++ b/docs/manuals/04-quickstart-setup-guide.md @@ -130,9 +130,10 @@ Check the version: ```bash ./target/release/extenddb version # extenddb 0.1.14 -# catalog migration 6 (postgres) +# catalog migration 7 (postgres) # commit abc1234 # built 2026-04-17T12:00:00Z +# test-hooks disabled ``` ### Initialization @@ -148,13 +149,15 @@ Options: | Flag | Default | Description | |------|---------|-------------| | `--catalog-db` | `extenddb_catalog` | Catalog database name | -| `--data-db` | `_data` | Data database name | +| `--data-db` | the catalog's name without `_catalog` (`extenddb` for the default catalog) | Data database name | | `--pg-host` | `localhost` | PostgreSQL host | | `--pg-port` | `5432` | PostgreSQL port | -| `--pg-user` | `extenddb` | PostgreSQL user | -| `--pg-password` | `extenddb-local-dev` | PostgreSQL password | +| `--pg-user` | `$USER`, then `postgres` | PostgreSQL admin role, used for `CREATE DATABASE` and `CREATE USER` | +| `--pg-pass` | none (or `EXTENDDB_PG_PASSWORD`) | The admin role's password | +| `--extenddb-user` | `extenddb` | The application role the server connects as | +| `--extenddb-pass` | `extenddb-local-dev` (or `EXTENDDB_APP_PASSWORD`) | The application role's password | -The command generates `extenddb.toml` with the connection details. If `extenddb.toml` already exists, `init` loads defaults from it. +The command generates `extenddb.toml` with the connection details. If `extenddb.toml` already exists, `init` refuses (`Error: Config file "extenddb.toml" already exists. Use --overwrite to delete and regenerate it.`, exit 255); with `--overwrite` it loads the PostgreSQL connection defaults from the existing file and writes a new one. **Save the admin credentials** printed during init. They are shown once and cannot be retrieved later. @@ -172,7 +175,7 @@ Config: extenddb.toml Catalog database: extenddb_catalog --- Checking catalog migrations... - OK: Current: the catalog is at migration 6 + OK: Current: the catalog is at migration 7 --- Checking catalog connection... OK: Connected to catalog. --- Checking data database... diff --git a/docs/manuals/05-admin-guide.md b/docs/manuals/05-admin-guide.md index 5d283c7dc..18bc2c1d9 100755 --- a/docs/manuals/05-admin-guide.md +++ b/docs/manuals/05-admin-guide.md @@ -145,7 +145,7 @@ Backup and restore jobs are bounded per server process by `[backup] max_concurre #### [backup] -The knobs backup and restore jobs and the reconciler read; every key has a default, and the section may be left out. +The knobs backup and restore jobs and the reconciler read; every key has a default, and the section may be left out. This section and `[backup.store]` below are the reference for the keys and the `extenddb backup` commands; the [Backup and Restore manual](14-backup-and-restore.md) covers the strategy, the recovery point and time, securing the store, the restore drill and catalog-loss recovery. | Key | Default | Description | |-----|---------|-------------| @@ -154,7 +154,7 @@ The knobs backup and restore jobs and the reconciler read; every key has a defau | `stale_job_after_secs` | `300` | Heartbeat age after which a job's owner is dead (on PostgreSQL, once the owner's claim session has ended as well) and the reconciler resolves the job: a dead backup job with a complete manifest under its prefix is AVAILABLE, one without is FAILED and its prefix deleted; a dead in-catalog restore job is FAILED and its target removed; a dead store-backed restore is resumed from its recorded progress (the differences document's restore-interruption row). Must be at least three heartbeat periods, where the period is the larger of `heartbeat_interval_secs` and the 15 s the in-catalog jobs heartbeat at (so 45 at the least, whatever `heartbeat_interval_secs` says); on SQLite a long write transaction delays a heartbeat, so keep a margin above that floor | | `failed_row_retention_secs` | `604800` | How long FAILED and DELETED backup rows of the configured store are kept once their objects are gone (7 days); `extenddb backup list --failed` prints the FAILED ones with their reason. At least `stale_job_after_secs`: the reconciler deletes a stopped job's late object only under its row, so the row outlives the window in which that job can still write; a smaller value is refused at startup | | `reconcile_interval_secs` | `60` | The reconciler's cycle; it also runs once at startup. Each cycle checks the store, resolves backup and restore jobs whose owner is dead, finishes deletions, deletes the objects of FAILED backups once their heartbeat is older than `stale_job_after_secs`, and removes tombstones past `failed_row_retention_secs` (the differences document's restore-interruption row lists the duties) | -| `max_concurrent_jobs` | `4` | Backup and restore jobs this instance runs at once (at most 1024); a request over it waits up to 30 s for a slot, then `LimitExceededException`. On PostgreSQL the pool sizes are raised to `2 * max_concurrent_jobs + 10` when smaller; on SQLite `[storage.sqlite] pool_size` is raised to `2 * (max_concurrent_jobs + 1)`, at most 34, when its snapshot slots (`pool_size / 2 - 1`, between 1 and 16) would refuse a job this knob admits. Memory: a running backup holds about 490 MiB (PostgreSQL) or 650 MiB (SQLite) at most in the pipeline's own terms for a table of items at the 32 MiB line cap, a few tens of MiB for 200-byte items (the figures are for the default `max_item_size_bytes`, 400 KB, where no stored item's line exceeds the cap; on an instance that raises it the terms are at the largest line a stored item expands to, about 68 times the item's size for the worst shape); with glibc's default allocator the process's high-water mark grows with the number of large items a job reads (measured 760 MiB over idle at 48 items of 27.7 MB on a quiet host and 1,281 MiB on a loaded one, 1.39 and 2.35 times the terms at that item size) because the items' strings are allocated on one thread and freed on another and the per-thread arenas keep that memory. Set `MALLOC_ARENA_MAX=1` in the server's environment (the deployment guide's service unit carries the line, commented out) on a host that backs up large items: the peak is then the terms' and does not grow with the table (measured 452 to 481 MiB over idle at 12, 24 and 48 items of 27.7 MB on SQLite, in two series on two hosts' loads, and 340 MiB `VmHWM` (about 200 MiB over idle) at 12 items on PostgreSQL); one arena serializes the worker threads' allocations on one lock and costs the request path a few percent (measured 3.5 to 9.2% on GetItem, Query and Scan on SQLite and about 2% on PostgreSQL (the scans, 2.1%), where the request path waits on the database on PostgreSQL). Size the host for 1.2 times the terms (about 590 MiB on PostgreSQL, 780 MiB on SQLite) times this knob under `MALLOC_ARENA_MAX=1`; without it the peak grows with the large rows a job reads and no bound was measured (1,281 MiB over idle at 48 items of 27.7 MB is the largest seen). A running restore holds the parsed manifest, one decoded data-file line and its parse, the decoded items of 32 MiB of data-file text plus one line per chunk (the chunk closes with the item that reaches the bound) at four chunk positions between the decoder and the load (two in the channel, the one being filled, the one being drained; a chunk also closes at 1,000 items, and the chunks never hold more than one data file's items), and the backend's write batch: at most 32 MiB of item text on SQLite and Cassandra (10,000 items when they are smaller), held beside the decoded items of that text; 500 items (or 4 MiB) per INSERT statement inside one transaction per data file on PostgreSQL; 1,000 documents per `insert_many` on MongoDB (the driver splits the call); times this knob. For a table of large items that is the same order as the backup, not less: measured `VmHWM` over idle from a fresh start (`docs/design/15-backup-restore-bench-pr9.md`), 24 items of 27.7 MB restore with a peak of 848 MiB on SQLite and 778 MiB on PostgreSQL under the default allocator (848 and 1,013 MiB at 12 and 48 such items on SQLite: the peak grows with the large items a restore reads, as the backup's does, and no bound was measured), and 392 MiB on SQLite and 427 MiB on PostgreSQL under `MALLOC_ARENA_MAX=1`, the same setting as above; 1,200 items of 350 KB in one data file 270 MiB on SQLite and 156 MiB on PostgreSQL; 1M items of 200 bytes under a composite key with no secondary index, the backup's lines in key order as the server's own backups are, 89 MiB on SQLite (the write connection's page cache and the final `COUNT(*)`, each bounded by `cache_size`; the count's share was 22 to 28 MiB at 1M and at 3M rows) and 16 MiB on PostgreSQL, and the same 1M items as an unordered backup (every MongoDB-sourced backup and every multi-part-key table) 144 to 164 MiB on SQLite in 66 to 90 s against 27 to 56 s key-ordered (scattered inserts dirty the whole base b-tree; the restore's buffers are the same) and 16 MiB on PostgreSQL; the same items under the same key with two full-projection GSIs 143 to 225 MiB on SQLite at 100,000 items in three runs and 165 to 172 MiB at 300,000, and 23 to 34 MiB on PostgreSQL at both: the excess on SQLite is the backend's own per-connection page cache (`cache_size`, 32 MiB) holding the index tables' b-tree pages plus the allocator's arenas (about a third less under `MALLOC_ARENA_MAX=1`), not the restore's buffers, which the 32 MiB batch bound holds whatever the indexes; it is bounded by `cache_size` and grows sub-linearly with the table. Size the host for about 450 MiB per running restore of large items under `MALLOC_ARENA_MAX=1`, times this knob, plus about 32 MiB of page cache per running restore of an indexed table on SQLite, and read the 1M figure for 200-byte items as the key-ordered shape's: an unordered backup of them restores in about 1.5 to 1.8 times that on SQLite | +| `max_concurrent_jobs` | `4` | Backup and restore jobs this instance runs at once (at most 1024); a request over it waits up to 30 s for a slot, then `LimitExceededException`. On PostgreSQL the pool sizes are raised to `2 * max_concurrent_jobs + 10` when smaller; on SQLite `[storage.sqlite] pool_size` is raised to `2 * (max_concurrent_jobs + 1)`, at most 34, when its snapshot slots (`pool_size / 2 - 1`, between 1 and 16) would refuse a job this knob admits. Memory: a running backup holds about 490 MiB (PostgreSQL) or 650 MiB (SQLite) at most in the pipeline's own terms for a table of items at the 32 MiB line cap, a few tens of MiB for 200-byte items (the figures are for the default `max_item_size_bytes`, 400 KB, where no stored item's line exceeds the cap; on an instance that raises it the terms are at the largest line a stored item expands to, about 68 times the item's size for the worst shape); with glibc's default allocator the process's high-water mark grows with the number of large items a job reads (measured 760 MiB over idle at 48 items of 27.7 MB on a quiet host and 1,281 MiB on a loaded one, 1.39 and 2.35 times the terms at that item size) because the items' strings are allocated on one thread and freed on another and the per-thread arenas keep that memory. Set `MALLOC_ARENA_MAX=1` in the server's environment (the deployment guide's service unit carries the line, commented out) on a host that backs up large items: the peak is then the terms' and does not grow with the table (measured 452 to 481 MiB over idle at 12, 24 and 48 items of 27.7 MB on SQLite, in two series on two hosts' loads, and 340 MiB `VmHWM` (about 200 MiB over idle) at 12 items on PostgreSQL); one arena serializes the worker threads' allocations on one lock and costs the request path a few percent (measured 3.5 to 9.2% on GetItem, Query and Scan on SQLite and about 2% on PostgreSQL (the scans, 2.1%), where the request path waits on the database on PostgreSQL). Size the host for 1.2 times the terms (about 590 MiB on PostgreSQL, 780 MiB on SQLite) times this knob under `MALLOC_ARENA_MAX=1`; without it the peak grows with the large rows a job reads and no bound was measured (1,281 MiB over idle at 48 items of 27.7 MB is the largest seen). A running restore holds the parsed manifest, one decoded data-file line and its parse, the decoded items of 32 MiB of data-file text plus one line per chunk (the chunk closes with the item that reaches the bound) at four chunk positions between the decoder and the load (two in the channel, the one being filled, the one being drained; a chunk also closes at 1,000 items, and the chunks never hold more than one data file's items), and the backend's write batch: at most 32 MiB of item text on SQLite and Cassandra (10,000 items when they are smaller), held beside the decoded items of that text; 500 items (or 4 MiB) per INSERT statement inside one transaction per data file on PostgreSQL; 1,000 documents per `insert_many` on MongoDB (the driver splits the call); times this knob. For a table of large items that is the same order as the backup, not less: measured `VmHWM` over idle from a fresh start (`docs/design/15-backup-restore-bench-pr9.md`), 24 items of 27.7 MB restore with a peak of 848 MiB on SQLite and 778 MiB on PostgreSQL under the default allocator (848 and 1,013 MiB at 12 and 48 such items on SQLite: the peak grows with the large items a restore reads, as the backup's does, and no bound was measured), and 392 MiB on SQLite and 427 MiB on PostgreSQL under `MALLOC_ARENA_MAX=1`, the same setting as above; 1,200 items of 350 KB in one data file 270 MiB on SQLite and 156 MiB on PostgreSQL; 1M items of 200 bytes under a composite key with no secondary index, the backup's lines in key order as the server's own backups are, 89 MiB on SQLite (the write connection's page cache and the final `COUNT(*)`, each bounded by `cache_size`; the count's share was 22 to 28 MiB at 1M and at 3M rows) and 16 MiB on PostgreSQL, and the same 1M items as an unordered backup (every MongoDB-sourced backup and every multi-part-key table) 144 to 164 MiB on SQLite in 66 to 90 s against 27 to 56 s key-ordered (scattered inserts dirty the whole base b-tree; the restore's buffers are the same) and 16 MiB on PostgreSQL; the same items under the same key with two full-projection GSIs 143 to 225 MiB on SQLite at 100,000 items in three runs and 165 to 172 MiB at 300,000, and 23 to 34 MiB on PostgreSQL at both: the excess on SQLite is the backend's own per-connection page cache (`cache_size`, 32 MiB) holding the index tables' b-tree pages plus the allocator's arenas (about a third less under `MALLOC_ARENA_MAX=1`), not the restore's buffers, which the 32 MiB batch bound holds whatever the indexes; it is bounded by `cache_size` and grows sub-linearly with the table. Size the host for about 450 MiB per running restore of large items under `MALLOC_ARENA_MAX=1`, times this knob, plus about 32 MiB of page cache per running restore of an indexed table on SQLite, and read the 1M figure for 200-byte items as the key-ordered shape's: an unordered backup of them restores in about 1.5 to 1.8 times that on SQLite. The page-cache term is for B-tree indexes. On SQLite a restore of items of more than a few hundred bytes, of any attribute types, holds more: its write batch (up to 10,000 items or 32 MiB of item text) and the decode chunks in front of it grow with the item's size up to that bound, and lists of numbers decode larger than strings of the same size; the [Backup and Restore manual](14-backup-and-restore.md#memory-while-a-job-runs) has the measured points | | `spool_path` | (unset) | Restore download area for an S3 store, created `0700` with files `0600`, holding a `.lock` another instance on the host refuses. A store-backed restore from an S3 store downloads each data file here under `restore-
/` (`.json.gz`, `` zero-padded to six digits, with a `.verified` marker holding the file's sha256, written only after the file's digests matched and trusted only while it names the digest the manifest records, plus the manifest and sidecar), reads it from there, and removes the entry when the restore ends, except when it ended because the record had become another instance's (its claim lost, a status write that matched no record), where the sweep removes it; an entry left by an earlier run of this host for a record another host then owned is removed before a new run of the same restore uses it; the reconciler sweeps the `restore-` entry of a restore no longer RUNNING with this host's `spool_host`. Its disk cost is the compressed backup of the restore in flight. A filesystem store needs no spool (files are hashed in place and read again). Required with an S3 store, except on file-backed SQLite where it defaults to `backup-spool/` beside the database file. Refused inside or containing a filesystem store root, and when it is or contains the database file's directory | #### [backup.store] @@ -173,7 +173,7 @@ Startup opens the configured store and, for a filesystem store, validates it and The store identity. The root carries `extenddb-store.json`: a random store id and the id of the catalog that owns the root. Startup creates it when the root has no file and the catalog has no rows for the root, and otherwise reads it and compares: a file naming another catalog, a store id that differs from the one the catalog's rows for the root carry, a root with no file while rows exist, or rows carrying more than one id each mark the store unhealthy. Instances that share a catalog must share one store root (one S3 bucket and prefix, or one shared mount): host-local roots configured at the same path give each instance different bytes under one root label, so each check also `head`s the manifest of the newest AVAILABLE backup and marks the store unhealthy when it is absent, which is what a second instance with a local copy of the directory sees. The file is left by `backup purge` and by every deletion; remove it only when retiring the root, or after `extenddb destroy`: a catalog re-created with `init` has a new id, and the old root's file names the old one, so the re-initialised server reports the store owned by another catalog until `extenddb backup sync --adopt-store` makes the new catalog its owner (below), the file is removed or a new root configured (on the default SQLite root, that is the logged warning above for the server, which keeps CreateBackup on the in-catalog path, while `backup sync` handles that root as configured, so `--adopt-store` adopts it). The default root, `backups` beside the database file, is shared by every database file in that directory: `--adopt-store` on it takes the root from whichever catalog wrote its file, so give each database its own `[backup.store]` when several share a directory. A server on an in-memory SQLite database writes the file with `ephemeral_catalog: true`; at its next start it adopts a root whose file names another catalog and carries that marker, and no other, so it never takes a durable catalog's root, and then syncs the root's backups into its new catalog (the `path` row above: until that sync's log line, the earlier backups answer `BackupNotFoundException`). A catalog whose root another catalog adopted (by `--adopt-store`, or a second in-memory server's start) keeps acting on the root until a later check that can read the catalog finds it owned by another catalog, at the earliest one reconciler interval after the cycle in progress ends (a check whose catalog read fails leaves the health as it was), with one exception, its deletion fence: right before it deletes a backup's objects (DeleteBackup, the reconciler's deletions, a failed backup job's cleanup, `purge`), removes a FAILED or DELETED row and writes a CreateBackup's first object, and also before DeleteBackup and `purge` change the row, a server or `purge` re-reads `extenddb-store.json`, and when the file names another catalog, names this catalog under another store id than the one the store's check settled (a root replaced by another root of this catalog), or the root has none, it deletes nothing and refuses the request; a file naming another catalog also marks the store unhealthy (`owned by another catalog`) at once, and one of another store id `identity mismatch`. DeleteBackup takes a file it cannot read, or a root with no file, at the read before the row is changed as it takes a store that does not answer: it writes DELETING and answers `DELETED`, the read before the deletion refuses it, and the row stays DELETING for the reconciler, whose deletion runs the fence again. `--prune-missing`'s removal of a row is not fenced: it deletes no object. A refusal at the read before the row is changed (the file names another catalog or another store id) leaves the row as it was, so DescribeBackup still answers AVAILABLE. One at the read before the deletion (an adoption landed between the two) leaves an AVAILABLE row DELETING with nothing deleted (DeleteBackup and DescribeBackup there answer `DELETED`), which this catalog's reconciler finishes only if this catalog owns the root again, and a dead CREATING row that `purge` already removed has its prefix kept. The read narrows the window to the gap between it and the deletion; on S3 that gap is a listing and a `DeleteObjects`, each under the SDK's retries, so seconds when the store throttles. A check that read the file before the adoption and ends after a fence's refusal records the store healthy again until the next cycle's check: each deletion stays fenced in that time, but a restore is accepted. A backup job of the previous owner that fails after the adoption leaves its partial prefix (no manifest, and no row of the new owner) under the root, which no duty or command of the new owner removes. A write that has already passed that read still lands, so stop every server of the catalog a file names before adopting its root. While the catalog has no AVAILABLE backup for a root, two host-local directories at one path cannot be told apart (each instance creates its own file and id, and a file copied from one directory to the other gives both one id), and the manifest probe has nothing to probe: a backup one instance completed under its own directory looks, to the other instance's reconciler, like a dead owner's row with no manifest, which it marks FAILED, and the first instance's restart then deletes the only copy. Point every instance at one shared root before the first backup; copying `extenddb-store.json` between directories defeats the check rather than satisfying it. The identity file is read up to 64 KiB; a larger object under its key is reported unreadable. -`extenddb destroy` removes the databases and leaves the store untouched: it is the copy that survives destroy. `extenddb backup list [--failed]` prints the catalog's rows for the configured store's root, every store id they carry, with each row's `store_root` and `store_id`; it is a read that runs beside a server (on SQLite it takes no serve lock), reads the identity file without creating one, and runs on an unhealthy store, naming the problem in its header. It lists backups only, never restore jobs: a restore's record is the job's log line, not a `backup list` row. `extenddb backup purge --confirm` deletes every backup of the configured store, each backup's objects (its own prefix, never a whole account prefix, so a prefix the catalog has no row for survives) and its catalog row outright, leaving no tombstone, and leaves `extenddb-store.json` and the `exports/` component; it needs the store healthy. It takes the catalog-level maintenance claim first and heartbeats it for the command's whole run, waits up to `2 * stale_job_after_secs` until no store-backed CreateBackup is running on any instance (a CREATING or FAILED row with a heartbeat within `stale_job_after_secs` on the catalog database's clock) and no restore is RUNNING, refuses and releases the claim if they have not finished, deletes a CREATING row the wait saw no live owner for as it deletes any other row, finishes any DeleteBackup left in progress as part of the purge (a backup a DeleteBackup finishes while the purge runs is counted as deleted, not left), and logs each deletion under the `extenddb::audit` target with the account and the ARN. While the claim is held, CreateBackup and RestoreTableFromBackup on every instance answer `LimitExceededException` saying that a backup maintenance command holds the catalog's maintenance claim (the message does not say which command; the second command's refusal and the claim's row do), and a second maintenance command fails at once naming the holder. Without `--confirm` it prints the plan, changes nothing and exits 0. `--config` is accepted before or after the subcommand. None of the backup commands runs the startup recovery a server runs (on PostgreSQL the vector index rebuild, the control-plane recovery and the GSI queue workers; on MongoDB the key-repair pass; on Cassandra the control-plane transitions; on SQLite the interrupted-restore and index passes): that work belongs to the servers, and a command that ran it could exit with it half done. RUNNING restores are read by status, with no age, so a restore whose record was stamped this instant holds the wait too. On SQLite `purge --confirm` opens the database as `serve` does and so takes the serve lock: stop the server first (while one runs on the file the command refuses with exit 6, as a second `serve` does); under that lock it first resolves what the stopped server left (a RUNNING restore is failed and its target removed, the CREATING target of a FAILED restore removed, an in-catalog CREATING backup removed, a DELETING row finished) and deletes a CREATING or FAILED row whatever its heartbeat says, a heartbeat stamped ahead of the clock by a backward clock step included (the server that wrote it is stopped, so the zero window the command passes selects by status alone), so such a job never makes it wait or refuse. That includes a RUNNING restore from a backup store, which a server would resume: under the serve lock the command fails it and removes its target, as for an in-catalog restore (a resume is a job, which the maintenance claim excludes), except one that recorded no progress while its target holds items, which is failed `target_not_empty` and its target left for `extenddb backup remove-target`. A server starting on the file removes a claim a killed purge left, so no job is refused for the stale window, as a purge run again after the killed one takes its claim over. On the other backends it runs beside live servers, and a claim whose holder died expires after `stale_job_after_secs`. On PostgreSQL, MongoDB and Cassandra the backup commands (`list`, `purge`, and `sync` and `remove-target` below) open the catalog as a server does and hold a serving registration for their run, named after the command (`extenddb backup list`, `extenddb backup purge`, `extenddb backup sync`, `extenddb backup remove-target`: on PostgreSQL the session's `application_name`, on MongoDB the `command` field of its `extenddb_servers` document, on Cassandra the `release` column of its `extenddb_servers` row, after the release and ` as `), and release it when they exit: `extenddb migrate --yes` waits for them as for a server and names them while it waits, and while a migration runs they refuse at once (exit 6), as they do on a catalog at another migration than the binary's (exit 3, with the `extenddb migrate --yes` advice `serve` prints). On every backend Ctrl-C or `SIGTERM` makes the command release what it holds and exit: the serving registration after the open, if the signal came during it; the claim at once while it waits, and before its next backup while it deletes (the release itself is bounded by 10 s, past which the claim is left to the stale rule), so an interrupted purge leaves no half-deleted backup and no claim to wait out; run it again for the rest. The dry run on an unhealthy store lists the rows and says that `--confirm` refuses until the store's check passes. +`extenddb destroy` removes the databases and leaves the store untouched: it is the copy that survives destroy. `extenddb backup list [--failed]` prints the catalog's rows for the configured store's root, every store id they carry, with each row's `store_root` and `store_id`; it is a read that runs beside a server (on SQLite it takes no serve lock), reads the identity file without creating one, and runs on an unhealthy store, naming the problem in its header. It lists backups only, never restore jobs: a restore's record is the job's log line, not a `backup list` row. `extenddb backup purge --confirm` deletes every backup of the configured store, each backup's objects (its own prefix, never a whole account prefix, so a prefix the catalog has no row for survives) and its catalog row outright, leaving no tombstone, and leaves `extenddb-store.json` and the `exports/` component; it needs the store healthy. It takes the catalog-level maintenance claim first and heartbeats it for the command's whole run, waits up to `2 * stale_job_after_secs` until no store-backed CreateBackup is running on any instance (a CREATING or FAILED row with a heartbeat within `stale_job_after_secs` on the catalog database's clock) and no restore is RUNNING, refuses and releases the claim if they have not finished, deletes a CREATING row the wait saw no live owner for as it deletes any other row, finishes any DeleteBackup left in progress as part of the purge (a backup a DeleteBackup finishes while the purge runs is counted as deleted, not left), and logs each deletion under the `extenddb::audit` target with the account and the ARN. While the claim is held, CreateBackup and RestoreTableFromBackup on every instance answer `LimitExceededException` saying that a backup maintenance command holds the catalog's maintenance claim (the message does not say which command; the second command's refusal and the claim's row do), and a second maintenance command fails at once naming the holder. Without `--confirm` it prints the plan, changes nothing and exits 0. `--config` is accepted before or after the subcommand. None of the backup commands runs the startup recovery a server runs (on PostgreSQL the vector index rebuild, the control-plane recovery and the GSI queue workers; on MongoDB the key-repair pass; on Cassandra the control-plane transitions; on SQLite the interrupted-restore and index passes): that work belongs to the servers, and a command that ran it could exit with it half done. RUNNING restores are read by status, with no age, so a restore whose record was stamped this instant holds the wait too. On SQLite `purge --confirm` opens the database as `serve` does and so takes the serve lock: stop the server first (while one runs on the file the command refuses with exit 6, as a second `serve` does); under that lock it first resolves what the stopped server left (a RUNNING restore is failed and its target removed, the CREATING target of a FAILED restore removed, an in-catalog CREATING backup removed, a DELETING row finished) and deletes a CREATING or FAILED row whatever its heartbeat says, a heartbeat stamped ahead of the clock by a backward clock step included (the server that wrote it is stopped, so the zero window the command passes selects by status alone), so such a job never makes it wait or refuse. That includes a RUNNING restore from a backup store, which a server would resume: under the serve lock the command fails it and removes its target, as for an in-catalog restore (a resume is a job, which the maintenance claim excludes), except one that recorded no progress while its target holds items, which is failed `target_not_empty` and its target left for `extenddb backup remove-target`. A server starting on the file removes a claim a killed purge left, so no job is refused for the stale window, as a purge run again after the killed one takes its claim over. On the other backends it runs beside live servers, and a claim whose holder died expires after `stale_job_after_secs`. On PostgreSQL, MongoDB and Cassandra the backup commands (`list`, `purge`, and `sync` and `remove-target` below) open the catalog as a server does and hold a serving registration for their run, named after the command (`extenddb backup list`, `extenddb backup purge`, `extenddb backup sync`, `extenddb backup remove-target`: on PostgreSQL the session's `application_name`, on MongoDB the `command` field of its `extenddb_servers` document, on Cassandra the `release` column of its `extenddb_servers` row, after the release and ` as `), and release it when they exit: `extenddb migrate --yes` waits for them as for a server and names them while it waits, and while a migration runs they refuse at once (exit 6). On a catalog `serve` would refuse they exit with `serve`'s code and message: 3 with migrations pending (with the `extenddb migrate --yes` advice), 4 on a catalog a later release migrated with a breaking migration, and 5 on one that needs attention (the upgrade manual's [catalog status](07-upgrade-manual.md#the-catalog-status) table). On every backend Ctrl-C or `SIGTERM` makes the command release what it holds and exit: the serving registration after the open, if the signal came during it; the claim at once while it waits, and before its next backup while it deletes (the release itself is bounded by 10 s, past which the claim is left to the stale rule), so an interrupted purge leaves no half-deleted backup and no claim to wait out; run it again for the rest. The dry run on an unhealthy store lists the rows and says that `--confirm` refuses until the store's check passes. `extenddb backup sync [--adopt-store] [--prune-missing]` rebuilds the catalog's rows from the configured store: after a lost or re-created catalog, or after a backup's prefix was copied under the root from another deployment or another backend. It reads the catalog's rows for the root (of every status and every store id they carry), then lists the whole root and handles each `/
//` prefix as the listing passes it, checking it against those rows before it reads anything under it. Sync holds one prefix's object list at a time and the prefix of each manifest it saw; a filesystem store builds its listing of the whole root in memory first (about 72 bytes plus the key per object), an S3 store lists a page at a time. The report keeps an entry per prefix for the whole run, and it lists every prefix the catalog already has a row for among the skipped ones, so on a root of small backups the per-prefix term dominates: about 350 bytes per prefix at one object per prefix (62 MiB of peak RSS at 50,000 such prefixes and 112 MiB at 200,000, against 45 MiB for an empty root; about 165 bytes per object at 100 objects per prefix), measured on a SQLite debug build with a filesystem root and no rows. Of the catalog's rows it keeps each row's status, store id and table, and the key of each AVAILABLE row of the configured store, but every backend reads the rows whole before sync sees the first, so the read is the larger cost and grows with the rows of the root, tombstones within their retention included: about 22.5 KiB per row of a table with five GSIs (1,452 bytes of source table description per row; 610 MiB of peak RSS at 25,000 rows and 1,144 MiB at 50,000, against 47 MiB with none; 148 MiB at 50,000 rows whose description is `{}`), measured on a SQLite debug build on a 16-core x86_64 Linux host with glibc's default arenas. `backup list` peaks the same over those rows. A prefix with no row gets an AVAILABLE row under the configured store's identity when its manifest and sidecar verify, every data file the manifest lists is under the prefix at its recorded size, and the manifest's ARN names the prefix's account, table and backup id; an ARN of another region is adopted and logged (requests find it by its ARN as written), and one of another partition than `aws` is skipped, since every request refuses such an ARN. The row carries the manifest's backup name, size, item count and source table, and the backup id's instant as its creation time. It never adopts a prefix with a row of any status (CREATING, DELETING, DELETED, FAILED or AVAILABLE), one whose manifest, sidecar or data files are missing or do not match, one whose manifest names another backup, or anything under `exports/` or a table's `changelog/`, and prints each with its reason. Before the insert it reads the row under both spellings of the ARN (this server's region and the manifest's), and the insert happens only if no row exists: a CreateBackup whose row landed after sync read the rows keeps its row, as does a row of another root, account or table holding the backup id, and sync changes no existing row. It also prints the AVAILABLE rows whose manifest the listing does not hold. It exits 0 once it ran, skipped prefixes included, and 1 when a prefix was skipped because a store read or a catalog call failed (the run is incomplete; a second run adopts it), when the listing failed part-way (the backups adopted before the failure keep their rows), when a signal stopped it, or when `--prune-missing` left a row it may remove on a later run (a store or catalog call failed, its manifest's `head` included, or a restore is reading the backup: run the command again once the restore ends). A row `--prune-missing` leaves because its manifest is back or another writer moved it is reported, and the exit stays 0: @@ -1049,18 +1049,42 @@ Increase `pool_size` in `extenddb.toml` or check for connection leaks. ### Data Recovery -extenddb stores all data in PostgreSQL. Use standard PostgreSQL backup and recovery tools: +Table backups (`CreateBackup`) hold one table's items and definition; they do not hold the catalog's accounts, IAM users, keys, policies and settings. Back the databases up with the backend's own tools as well; the [Backup and Restore manual](14-backup-and-restore.md) covers table backups, the backup store and recovery from a lost catalog. The databases each backend uses: + +| Backend | Databases | +|---------|-----------| +| PostgreSQL | The catalog database, `extenddb_catalog` unless `init --catalog-db` names another, and the data database, whose name `init` derives from the catalog's by dropping the `_catalog` suffix (`extenddb` for the default; `init --data-db` names another). A catalog name without the `_catalog` suffix needs `--data-db`, since the derived name would be the catalog's own. The catalog records the data database's name in its `settings` table under `data_database_name` | +| SQLite | One database file (`[storage.sqlite] path`) with its `-wal` and `-shm` files beside it; stop the server before copying the three. To restore, stop the server, remove the current `-wal` and `-shm` files, and put the copied files in their place | +| MongoDB | `extenddb_catalog` and `extenddb_data`, fixed names | +| Cassandra | The catalog keyspace `_catalog` and one keyspace per account, `_account_` (`keyspace_prefix` defaults to `extenddb`) | + +On PostgreSQL the catalog and data databases are one copy: dump both with every server of the catalog stopped, or, to back up without stopping the server, take a base backup (`pg_basebackup`) of the PostgreSQL server that holds both. Each `pg_dump` takes its own snapshot of one database, so two dumps taken while a server runs disagree when a table is created or deleted between them. The catalog dump holds the encryption key and the application role's password (in the `data_database_connection_string` setting), so create both files readable by their owner alone, with `umask 077`. With the default names: ```bash -# Backup -pg_dump extenddb_catalog > catalog_backup.sql -pg_dump extenddb_catalog_data > data_backup.sql +# Backup, with the server stopped, with the pg_dump of the server's major version +umask 077 +pg_dump extenddb_catalog > catalog_backup.sql && pg_dump extenddb > data_backup.sql \ + && test -s catalog_backup.sql && test -s data_backup.sql && echo "both dumps written" +``` + +Use the `pg_dump` of the PostgreSQL server's major version (for a server in a container, the container's own: `docker exec pg_dump ...`). An older `pg_dump` refuses, "aborting because of server version mismatch", and exits 1, and the shell has already created its file, empty. Go on only when both commands exited 0 and neither file is empty: `psql -v ON_ERROR_STOP=1` restores an empty file without an error, and the server then refuses to start ("the catalog at ... is not initialized"). -# Restore -psql -f catalog_backup.sql extenddb_catalog -psql -f data_backup.sql extenddb_catalog_data +Restore into new, empty databases owned by the application role (`extenddb` unless `init --extenddb-user` named another), created by the PostgreSQL admin role, with the server stopped. Restoring over the existing databases restores nothing: the dump's statements fail on the objects already there (`psql` prints an `ERROR` line for each and exits 0), and the tables keep their current items. Rename the old databases aside first, and drop them only once the restored server has read every table, so that the old copy stays until the new one is proven. The owner matters: on PostgreSQL 15 and later the `public` schema belongs to the database's owner and the dump does not carry it, so in a database the admin role owns the server cannot create tables (`CreateTable` fails and the server logs "permission denied for schema public"). `-v ON_ERROR_STOP=1` makes `psql` stop at the first error, with exit 3: + +```bash +# Restore, as the PostgreSQL admin role, with the server stopped +psql -c 'ALTER DATABASE extenddb_catalog RENAME TO extenddb_catalog_before_restore' postgres +psql -c 'ALTER DATABASE extenddb RENAME TO extenddb_before_restore' postgres +createdb -O extenddb extenddb_catalog +createdb -O extenddb extenddb +psql -v ON_ERROR_STOP=1 -f catalog_backup.sql extenddb_catalog +psql -v ON_ERROR_STOP=1 -f data_backup.sql extenddb +# Start the server and read every table, then +psql -c 'DROP DATABASE extenddb_catalog_before_restore' -c 'DROP DATABASE extenddb_before_restore' postgres ``` +On a new PostgreSQL server (the old one lost with its databases) create the application role first: a dump holds no roles, and without it `createdb -O extenddb` fails, `role "extenddb" does not exist`. Give it the password that the catalog dump's `data_database_connection_string` row carries (`postgresql://extenddb:@:/extenddb`), the one the config file's `[storage.postgres] connection_string` names as well: `psql -c "CREATE USER extenddb WITH PASSWORD ''" postgres`. That row also records the data database's host and port, and the server connects to the data database there. On a server at another address, point the config file's `connection_string` at it and, after the restore and before the server starts, set the row: `psql -c "UPDATE settings SET value = 'postgresql://extenddb:@:/extenddb' WHERE key = 'data_database_connection_string'" extenddb_catalog`. Without that the server connects to the old address: it fails to start once nothing answers there, and while the old server still answers, the tables created then keep their items in the old server's data database while the new catalog lists them. + --- ## License diff --git a/docs/manuals/07-upgrade-manual.md b/docs/manuals/07-upgrade-manual.md index 09835c730..4ad610031 100755 --- a/docs/manuals/07-upgrade-manual.md +++ b/docs/manuals/07-upgrade-manual.md @@ -6,7 +6,7 @@ This release versions the PostgreSQL, SQLite, Cassandra and MongoDB catalogs by a migration ledger and carries catalog migrations 1 to 7, which `extenddb migrate --list` prints. **Every deployment created by a 0.1.x release runs `extenddb migrate --yes` once** before a server of this release starts on it: the server refuses a catalog with pending migrations. Migration 1 brings the catalog of every 0.1.x release to the v1.0 schema, migration 5 drops `continuous_backups`, a table and on MongoDB a collection (see [Catalog 0.0.5](#catalog-005)), migration 6 adds the backup catalog index and the restore job records ([below](#migration-6-the-backup-catalog-index)), and migration 7 adds the `restore_progress` table to the PostgreSQL data database (it does not apply on the other backends). -Configure `[backup.store]` before the release that stops creating in-catalog backups. From this release CreateBackup writes to the configured store on PostgreSQL, SQLite and MongoDB, and is asynchronous there as on the service: the response is `CREATING` and the backup becomes `AVAILABLE` once the upload ends, so a client that deletes a backup it just created has to poll DescribeBackup first (the previous release returned `AVAILABLE`). A store-backed backup is listed, described, deleted and restored like any other. `RestoreTableFromBackup` of a backup in the configured store runs the restore job from the store on every backend (PostgreSQL, SQLite, MongoDB and Cassandra); in-catalog backups, those of the previous release included, restore as before. An S3 store needs `spool_path` set (the restore's download area) on PostgreSQL, MongoDB and Cassandra, and that must be a directory of its own, not the database directory; a filesystem store needs no spool. Migration 7 adds the `restore_progress` table to the PostgreSQL data database; run `extenddb migrate --yes` before starting the new binary. The progress rows of a restore in flight live in the data database, so a data database restored from a dump taken before migration 7 has none of them and no `restore_progress` table: the new binary refuses to serve it until `extenddb migrate --yes` has run again, and its reconciler then finds no progress for the restore and resumes it from its first data file into the (empty) target, or `FAILED`s it as `target_not_empty` if the target already holds items, in which case `extenddb backup remove-target --account
` (the admin guide's `backup` section) removes the target and its record and the backup restores again. The same `target_not_empty` state is what a server stopped inside a restore's first data file on MongoDB, or inside its first batch of 10,000 items or 32 MiB on Cassandra, leaves on restart, since both record progress after the write; the command is the remedy there too. `backup remove-target` writes the catalog, so on SQLite the server is stopped for it, as for `purge`. The previous binary resumes no restore (a store-backed restore in flight at a rollback is lost with the catalog rows, as the rollback paragraph below says). Quiesce restores before a downgrade. `kind = "disabled"` is no longer needed to keep a deployment able to restore the backups it takes. Without a store CreateBackup writes the in-catalog format and logs a deprecation warning naming `[backup.store]` on each call; a later release's CreateBackup refuses without one. Cassandra writes the in-catalog format with or without a store. `kind = "disabled"` under `[backup.store]` is the opt-out that keeps a deployment on the in-catalog format on purpose and silences the warning. On file-backed SQLite an absent section is a filesystem store at `backups/` beside the database file, created `0700` at startup; that root must be owned by the extenddb user with mode `0700`, under directories owned by root or that user which others cannot write to. A default root that breaks the rule (a database file in a group-writable directory, say), or whose `extenddb-store.json` names another catalog (a catalog re-created by `destroy` and `init` under the old root), is logged at startup as a warning (`backup store: the default root ... failed its check`) and leaves the store unconfigured, not unhealthy, so the upgrade never stops a server that never configured backups and CreateBackup keeps working on it; the same root configured explicitly is a startup error or an unhealthy store. The warning applies only while the catalog has no backup rows for the default root (an upgrade, a fresh deployment, a catalog re-created by `destroy` and `init`): once a server has written rows for it, the default root is handled as a configured one, so its backups never vanish behind a warning (a failed identity check is an unhealthy store, reported on `/health`; a root that can no longer be opened is a startup error naming it). Instances that share a catalog must share one store root, an S3 bucket and prefix or a shared mount: a second instance pointed at a host-local directory under the same configured path starts with the store unhealthy (`/health` reports `backup_store` unhealthy; the admin guide's `[backup.store]` section says what the identity file and the manifest probe check). `extenddb destroy` leaves the store; `extenddb backup purge --confirm` is what deletes its backups, and a catalog re-created with `init` needs the old root's `extenddb-store.json` removed or a new root, since the file names the old catalog. +Configure `[backup.store]` before the release that stops creating in-catalog backups. The [Backup and Restore manual](14-backup-and-restore.md) covers choosing and securing a store, the recovery figures, the restore drill, catalog-loss recovery and what to do with in-catalog backups; this section says what the upgrade changes. From this release CreateBackup writes to the configured store on PostgreSQL, SQLite and MongoDB, and is asynchronous there as on the service: the response is `CREATING` and the backup becomes `AVAILABLE` once the upload ends, so a client that deletes a backup it just created has to poll DescribeBackup first (the previous release returned `AVAILABLE`). A store-backed backup is listed, described, deleted and restored like any other. `RestoreTableFromBackup` of a backup in the configured store runs the restore job from the store on every backend (PostgreSQL, SQLite, MongoDB and Cassandra); in-catalog backups, those of the previous release included, restore as before. An S3 store needs `spool_path` set (the restore's download area) on PostgreSQL, MongoDB and Cassandra, and that must be a directory of its own, not the database directory; a filesystem store needs no spool. Migration 7 adds the `restore_progress` table to the PostgreSQL data database; run `extenddb migrate --yes` before starting the new binary. The progress rows of a restore in flight live in the data database, so a data database restored from a dump taken before migration 7 has none of them and no `restore_progress` table: the new binary refuses to serve it until `extenddb migrate --yes` has run again, and its reconciler then finds no progress for the restore and resumes it from its first data file into the (empty) target, or `FAILED`s it as `target_not_empty` if the target already holds items, in which case `extenddb backup remove-target --account
` (the admin guide's `backup` section) removes the target and its record and the backup restores again. The same `target_not_empty` state is what a server stopped inside a restore's first data file on MongoDB, or inside its first batch of 10,000 items or 32 MiB on Cassandra, leaves on restart, since both record progress after the write; the command is the remedy there too. `backup remove-target` writes the catalog, so on SQLite the server is stopped for it, as for `purge`. The previous binary resumes no restore (a store-backed restore in flight at a rollback is lost with the catalog rows, as the rollback paragraph below says). Quiesce restores before a downgrade. `kind = "disabled"` is no longer needed to keep a deployment able to restore the backups it takes. Without a store CreateBackup writes the in-catalog format and logs a deprecation warning naming `[backup.store]` on each call; a later release's CreateBackup refuses without one. Cassandra writes the in-catalog format with or without a store. `kind = "disabled"` under `[backup.store]` is the opt-out that keeps a deployment on the in-catalog format on purpose and silences the warning. On file-backed SQLite an absent section is a filesystem store at `backups/` beside the database file, created `0700` at startup; that root must be owned by the extenddb user with mode `0700`, under directories owned by root or that user which others cannot write to. A default root that breaks the rule (a database file in a group-writable directory, say), or whose `extenddb-store.json` names another catalog (a catalog re-created by `destroy` and `init` under the old root), is logged at startup as a warning (`backup store: the default root ... failed its check`) and leaves the store unconfigured, not unhealthy, so the upgrade never stops a server that never configured backups and CreateBackup keeps working on it; the same root configured explicitly is a startup error or an unhealthy store. The warning applies only while the catalog has no backup rows for the default root (an upgrade, a fresh deployment, a catalog re-created by `destroy` and `init`): once a server has written rows for it, the default root is handled as a configured one, so its backups never vanish behind a warning (a failed identity check is an unhealthy store, reported on `/health`; a root that can no longer be opened is a startup error naming it). Instances that share a catalog must share one store root, and a catalog re-created with `init` finds the old root's `extenddb-store.json` naming the old catalog: the [Backup and Restore manual](14-backup-and-restore.md) gives both rules and the recovery with `extenddb backup sync --adopt-store`. A backup that a 0.1.x release took can restore without its global and local secondary indexes: a SQLite backup taken by 0.1.14 records only the table's key schema and attribute definitions, and its restore creates the table with its items and no index. Take a new backup of each table after upgrading to v1.0, and keep the 0.1.x backups for their items only. @@ -68,10 +68,13 @@ extenddb stop --config extenddb.toml 2. **Back up the databases** (PostgreSQL; for SQLite see [SQLite](#sqlite) below, for Cassandra see the [rollback procedure](#rollback-procedure)) ```bash +umask 077 pg_dump -h -U > catalog_backup_$(date +%Y%m%d).sql pg_dump -h -U > data_backup_$(date +%Y%m%d).sql ``` +The catalog dump holds the encryption key and the application role's password, so `umask 077` creates both files readable by their owner alone. + `pg_dump` refuses to dump a server of a newer major version than itself ("aborting because of server version mismatch"); use the client of the server's version. 3. **Build the new version** @@ -167,15 +170,18 @@ SQLite deployments follow the same sequence, backing up the database files as [S The upgrade is not reversible in place when it applies a breaking migration, and every migration in this release is breaking (see [The Catalog Status](#the-catalog-status)). If an upgrade fails or must be undone (PostgreSQL; for SQLite see [SQLite](#sqlite) below): 1. Stop the server -2. Restore the catalog from the backup, connected as `` to the `postgres` database: +2. Restore both databases from the backups, connected as `` to the `postgres` database: ```bash psql -h -U -d postgres -c "DROP DATABASE ;" psql -h -U -d postgres -c "CREATE DATABASE OWNER ;" psql -h -U -d -f catalog_backup_YYYYMMDD.sql +psql -h -U -d postgres -c "DROP DATABASE ;" +psql -h -U -d postgres -c "CREATE DATABASE OWNER ;" +psql -h -U -d -f data_backup_YYYYMMDD.sql ``` -To discard data written after the upgrade as well, restore `` from its dump the same way. +The catalog and data databases are one copy, so restore both; every write made after the dumps is lost. A catalog restored alone beside the newer data database is a catalog older than the tables; the Backup and Restore manual's [catalog restore](14-backup-and-restore.md#the-catalog-is-restored-from-a-database-backup) section says what that gives. 3. Rebuild the previous version and start it @@ -216,6 +222,7 @@ The upgrade is not reversible in place: a 0.0.3 PostgreSQL or SQLite binary refu ```bash extenddb stop --config extenddb.toml +umask 077 pg_dump -h -U > catalog_backup_$(date +%Y%m%d).sql export EXTENDDB_PG_PASSWORD='' extenddb migrate --config extenddb.toml --pg-user # exit 3, prints the plan diff --git a/docs/manuals/10-security-model.md b/docs/manuals/10-security-model.md index 8558c0681..b5e71a330 100755 --- a/docs/manuals/10-security-model.md +++ b/docs/manuals/10-security-model.md @@ -327,11 +327,12 @@ Import/export file paths are validated: ### Backup and Recovery -extenddb stores all state in PostgreSQL. Use standard PostgreSQL tools for backup and recovery: +Back the databases up with the backend's own tools (the admin guide's [Data Recovery](05-admin-guide.md#data-recovery) entry names each backend's databases), and keep table backups in a backup store whose access you restrict (the [Backup and Restore manual](14-backup-and-restore.md#securing-the-store)). On PostgreSQL the catalog and data databases are one copy: dump both with the server stopped (the admin guide's Data Recovery entry restores the dumps), or, when both databases are on one PostgreSQL server, take a base backup of that server and restore it with PostgreSQL's own procedure (the design guide allows the two databases on different PostgreSQL instances, which one base backup does not cover). With the default database names, and `umask 077` so that only the owner can read the files: ```bash +umask 077 pg_dump extenddb_catalog > catalog_backup.sql -pg_dump extenddb_catalog_data > data_backup.sql +pg_dump extenddb > data_backup.sql ``` Encryption keys are stored in the catalog database. A catalog backup includes the encryption key needed to decrypt access key secrets. diff --git a/docs/manuals/11-deployment-guide.md b/docs/manuals/11-deployment-guide.md index 198c6e781..35b506768 100755 --- a/docs/manuals/11-deployment-guide.md +++ b/docs/manuals/11-deployment-guide.md @@ -17,7 +17,7 @@ This guide covers deploying extenddb in various environments beyond local develo extenddb is a single Rust binary that connects to a storage backend. All durable state lives in the backend; extenddb keeps only short-lived credential, policy, and table-metadata caches in process. This means: - Multiple extenddb instances can share one backend catalog (with caveats — see Multi-Instance below) -- The backend's own HA, backup, and replication tools provide durability +- The backend's own HA, backup, and replication tools provide durability; with `[backup.store]` configured (file-backed SQLite has one by default), table backups go to the store on PostgreSQL, SQLite and MongoDB; without one, and always on Cassandra, they are in-catalog backups inside the database; the [Backup and Restore manual](14-backup-and-restore.md) covers both - extenddb can run anywhere the storage backend is reachable ## Deployment Models @@ -229,7 +229,8 @@ extenddb init \ - [ ] Use a dedicated PostgreSQL user for extenddb with minimal privileges - [ ] Configure PostgreSQL `max_connections` ≥ `2 * pool_size + catalog_pool_size + max_concurrent_jobs`, plus one per overlapping vector index build and one during a schema migration - [ ] Enable PostgreSQL TLS -- [ ] Set up automated backups (pg_dump, WAL archiving, or managed service snapshots) +- [ ] Set up automated backups that copy the catalog and data databases as one copy: a base backup with WAL archiving, managed service snapshots of the instance, or `pg_dump` of both with the server stopped, under `umask 077` (the admin guide's [Data Recovery](05-admin-guide.md#data-recovery) entry) +- [ ] Configure `[backup.store]` (S3 for production) per the [Backup and Restore manual](14-backup-and-restore.md) - [ ] Monitor PostgreSQL disk usage, connection count, and query performance ### Monitoring @@ -272,7 +273,10 @@ RestartSec=5 # large items it reads (the admin guide's [backup] max_concurrent_jobs row # gives the figures). It costs the request path a few percent (3.5 to 9.2% # on GetItem, Query and Scan on SQLite, about 2% on PostgreSQL (the scans, -# 2.1%)), so the line is left to the operator: +# 2.1%)), so the line is left to the operator. On SQLite it also lowers +# the peak of a restore of tables whose items are larger than a few +# hundred bytes, of any attribute types (the Backup and Restore manual +# gives the figures): #Environment=MALLOC_ARENA_MAX=1 [Install] @@ -291,7 +295,7 @@ Run one extenddb instance per catalog. Multiple instances behind a load balancer - Credentials, policies, and table metadata are cached in each process (see the cache TTLs in the admin guide). Nothing invalidates a cache on another instance, so a key revoked or a policy changed through one instance stays in force on the others until the TTL expires. - Every instance runs every background worker (TTL expiry, GSI backfill, stream retention, control-plane transitions). The workers are safe to run twice, but the work is duplicated. - `/health` is a liveness check that does not query the database, so a load balancer cannot use it to take an instance with a broken database connection out of rotation. It does report the backup store: `components.backup_store` is `unhealthy` and `status` is `degraded` while the configured store cannot be reached or fails its identity check, with HTTP 200 kept, so a store outage never drains a replica. -- Instances that share a catalog must share one backup store root (`[backup.store]`: one S3 bucket and prefix, or one shared mount that supports hard links, which the store's conditional create needs; some CIFS and FUSE mounts refuse `link(2)` with `EPERM`, and the server then reports the store as having failed validation, with the `link` error in the log line). A host-local directory configured at the same path on each host is not one root: once the catalog has rows for the root, the second instance finds the root's identity file missing or the newest backup's manifest absent and starts with the store unhealthy. While the catalog has no rows for the root (a fresh deployment, or one whose backups were purged), nothing tells the two directories apart: each instance creates its own identity file and store id and both start healthy, and the first backups then split the rows between two ids, which the next check reports as an identity mismatch that only `backup purge` resolves. Point every instance at one root before the first backup. Each instance on a host needs its own `spool_path`; the spool's `.lock` refuses a second instance. An existing spool directory must be owned by the server's user with mode `0700`, as the root must; a spool created `0755` stops startup naming it. +- Instances that share a catalog must share one backup store root (`[backup.store]`: one S3 bucket and prefix, or one shared mount that supports hard links, which the store's conditional create needs; some CIFS and FUSE mounts refuse `link(2)` with `EPERM`, and the server then reports the store as having failed validation, with the `link` error in the log line). A host-local directory configured at the same path on each host is not one root: once the catalog has rows for the root, the second instance finds the root's identity file missing or the newest backup's manifest absent and starts with the store unhealthy. While the catalog has no rows for the root (a fresh deployment, or one whose backups were purged), nothing tells the two directories apart: each instance creates its own identity file and store id and both start healthy, and the first backups then split the rows between two ids, which the next check reports as an identity mismatch that only `backup purge` resolves. Point every instance at one root before the first backup ([Backup and Restore manual](14-backup-and-restore.md#configuration)). Each instance on a host needs its own `spool_path`; the spool's `.lock` refuses a second instance. An existing spool directory must be owned by the server's user with mode `0700`, as the root must; a spool created `0755` stops startup naming it. On SQLite this limit is enforced: `extenddb serve` holds an exclusive lock on `.lock` and a second server on the same file refuses to start. On PostgreSQL, MongoDB, and Cassandra it is not enforced. On PostgreSQL, if you start a second instance anyway, size `pool_size × instance_count + 3 × instance_count ≤ PostgreSQL max_connections`. diff --git a/docs/manuals/12-extending-extenddb-storage.md b/docs/manuals/12-extending-extenddb-storage.md index 93db24814..70b395d4a 100755 --- a/docs/manuals/12-extending-extenddb-storage.md +++ b/docs/manuals/12-extending-extenddb-storage.md @@ -288,8 +288,8 @@ The `storage-postgres` crate provides the default implementation: ### Database Architecture extenddb uses a dual-database architecture: -- **Catalog database** (`extenddb`) — table metadata, IAM data, settings, metrics, stream records, tags, idempotency tokens, login attempts -- **Data database** (`extenddb_data`) — DynamoDB items and index rows +- **Catalog database** (`extenddb_catalog` by default) — table metadata, IAM data, settings, metrics, stream records, tags, idempotency tokens, login attempts +- **Data database** (`extenddb` by default, the catalog's name without `_catalog`) — DynamoDB items and index rows ### Migrations diff --git a/docs/manuals/14-backup-and-restore.md b/docs/manuals/14-backup-and-restore.md new file mode 100644 index 000000000..7f3424abf --- /dev/null +++ b/docs/manuals/14-backup-and-restore.md @@ -0,0 +1,678 @@ +# Backup and Restore + +> See [NOTICE](../NOTICE.md) for important disclaimers. + +A filesystem backup store keeps every backup in the same failure domain as the host that holds it, so S3 is +the production recommendation: a lost host or disk loses a filesystem store's backups with it unless +something copies the root off the host, while an S3 bucket (or an S3-compatible server on other hardware) +survives the loss of the ExtendDB host and of its database. + +This manual is for operators who plan, run and test backups. The admin guide is the reference for every +configuration key and command named here; the +[differences document](../differences-from-dynamodb.md#backup-and-restore) records how each backup operation +compares with Amazon DynamoDB's. + +## Strategy + +What a table backup is. `CreateBackup` takes one table's items and definition at one instant and writes +them to the configured backup store, the `[backup.store]` section: a directory (`kind = "filesystem"`) or +an S3 bucket and prefix (`kind = "s3"`). The request answers `CREATING` and a job of the server uploads the +backup; it becomes `AVAILABLE` when the upload ends. The backup is a consistent snapshot of the table on +PostgreSQL, SQLite and MongoDB. The catalog keeps a row per backup as the index that `ListBackups` and +`DescribeBackup` read, and the store holds the bytes, so a lost catalog loses the index, not the backups +([Recovering from catalog loss](#recovering-from-catalog-loss)). + +Cassandra has no table snapshot. `CreateBackup` there writes the in-catalog format (inside the Cassandra +cluster, in the catalog keyspace and the account's keyspace) whatever the store, and that copy pages the live +table, so it is not point-in-time. A configured store serves restores into Cassandra of backups that another +backend wrote to the store. For off-host copies of a Cassandra deployment use the cluster's own snapshot +tooling. + +What a table backup is not: + +- It does not hold the catalog: accounts, IAM users, groups, roles, policies, access keys, runtime settings + and the encryption key live in the catalog database, which no table backup contains. Back the catalog up + with the database's own tools (the admin guide's [Data Recovery](05-admin-guide.md#data-recovery) entry + names the databases and gives the PostgreSQL commands), and keep that copy off the host as well. On + PostgreSQL the catalog and data databases are one copy: dump both with the server stopped, or take a base + backup of the PostgreSQL server that holds both. A catalog backup holds the encryption key that decrypts the + access key secrets, and on PostgreSQL the application role's password: protect it as a secret, and create + the dump files with `umask 077`. +- It is not scheduled. ExtendDB has no backup scheduler: run `CreateBackup` from cron or your scheduler. As a + crontab line, where every `%` must be escaped because cron turns an unescaped `%` into a newline: + + ``` + 0 * * * * AWS_CA_BUNDLE=/path/to/cert.pem AWS_DEFAULT_REGION=us-east-1 aws dynamodb create-backup --table-name orders --backup-name orders-$(date +\%Y\%m\%d\%H\%M) --endpoint-url https://:18443 + ``` + + or put the command, with its `%` unescaped, in a script that cron runs. `AWS_CA_BUNDLE` (or the CLI's + `--ca-bundle`) names the self-signed certificate `init` generates, `~/.extenddb/tls/cert.pem` of the user + who ran it; without it the CLI fails certificate verification. Cron's environment names no region; + `AWS_DEFAULT_REGION` gives the CLI one, which it otherwise needs from the user's `~/.aws/config`. +- It has no retention. A backup stays until `DeleteBackup` removes it. Your scheduler deletes the backups it + no longer wants. +- It is not point-in-time recovery, which ExtendDB does not provide (below). + +A plan that covers host loss, database loss and operator error therefore has three parts: table backups on a +schedule to an S3 store in another failure domain, database-level backups of the catalog, and a restore drill +run on a schedule ([Restore drill](#restore-drill)). + +## RPO and RTO + +### Recovery point + +The recovery point objective equals the interval between two backups of a table. There is no point-in-time +recovery: `RestoreTableToPointInTime` answers `PointInTimeRecoveryUnavailableException`, and +`UpdateContinuousBackups` with `PointInTimeRecoveryEnabled` true answers +`ContinuousBackupsUnavailableException`. A restore brings the table back as it was at the backup's +`BackupCreationDateTime`, the instant the request opened the table's snapshot; every write acknowledged after +that instant is lost. + +Per store kind: + +| Store | Loss it survives | Recovery point | +|---|---|---| +| S3, a bucket outside the ExtendDB host and the database host | the ExtendDB host, the database, the catalog | the backup interval | +| filesystem, a root on the ExtendDB host | the database and the catalog, while the host's disk survives | the backup interval for database loss; for host loss, the interval of whatever copies the root off the host | +| filesystem, a root on a shared mount | as the mount's own storage | as the mount's own storage | +| in-catalog, inside the database (Cassandra always; PostgreSQL, MongoDB and in-memory SQLite without `[backup.store]`; `kind = "disabled"`) | the loss of a table, not of the database or its host | the backup interval while the database survives; none once it is lost, which for in-memory SQLite is at every restart | + +### Recovery time + +The recovery time is the restore's duration plus the time to bring a server back, and, after catalog loss, +the time to rebuild the catalog and run `extenddb backup sync`. A restore answers `CREATING` at once and the +table accepts reads and writes when it reports `ACTIVE`; every index, vector indexes included, is rebuilt from +the items during the restore, so the table is complete when it is `ACTIVE`. + +Measured restore durations, from the request to `ACTIVE` as the client saw it. Host: 16 cores (Xeon +Platinum 8124M), 30 GiB, Linux 5.10; PostgreSQL 16 (with pgvector for the vector rows), MongoDB 7.0 as a +single-node replica set and Cassandra 4.1 as a single node, each in a container on loopback; SQLite as the +binary bundles it; release builds; a filesystem store on local disk; items of a 200-byte string under a +composite key with no secondary index unless the row says otherwise. Each figure is one run (a range is +several) on a host that ran other work, so the durations are upper bounds. The figures come from the +measurement records under `docs/design/` (`15-backup-restore-bench-*.md`), each on the release build it +names: + +| Table | PostgreSQL | SQLite | MongoDB | Cassandra | +|---|---|---|---|---| +| 100k items | 1.1 s | 2.9 s | 3.9 to 8.9 s (unordered) | 6.1 s | +| 1M items, backup in key order | 9.5 s | 26.8 to 56.3 s | | 49.0 s | +| 1M items, unordered backup | 17.5 s | 66 to 90 s | 33.8 s (up to 5 s more in another run) | | +| 100k items, two GSIs | 6.2 to 9.9 s | 10.2 to 13.9 s | | | +| 300k items, two GSIs | 19.7 to 21.4 s | 51.7 to 51.9 s | | | +| 100k items with a 128-dimension vector and a vector index | 25 s | 22 s (28 s under `MALLOC_ARENA_MAX=1`) | | | +| 1M items with a 128-dimension vector and a vector index | 246 s | 214 s (283 s under `MALLOC_ARENA_MAX=1`) | | | + +A server writes its own backups in key order for hash-only and single-range-key tables on PostgreSQL and +SQLite; MongoDB's backups and those of multi-part-key tables are unordered. On MongoDB the table becomes +`ACTIVE` at the next cycle of the backend's backfill worker after the restore's load ends, and the worker +runs every 5 s, so a MongoDB restore takes the load's time plus up to 5 s: 3.9 to 8.9 s at 100k items over +five runs, and at 1M items 33.8 s in one run, which another run may exceed by up to one cycle. The Cassandra rows restore a +SQLite backup copied into the Cassandra deployment's store, since Cassandra's own backups are in-catalog. The +vector rows' data files carry about 2.5 KB of line text per item, 9.5 times the plain rows' 265 bytes (the +plain rows' pad is one repeated character, so their files compress far better: 90 to 176 times smaller than +the vector rows' compressed files, 176 for the 1M key-ordered backup); their duration is not the index's cost +alone. In the runs paired with an indexed restore, the same 100k items without the vector index restored in +18.0 to 18.3 s on SQLite and 15.6 to 18.1 s on PostgreSQL: the index is about 20% of the 100k restore's +duration on SQLite (18 to 22% in two sets of runs) and 33 to 47% on PostgreSQL (five pairs), the runs of each +set or pair from one measurement harness (the measurement record's "Later runs"). + +Measured backup durations, `CreateBackup` to `AVAILABLE`, on the same host class: 1M items with two GSIs +36.1 s on PostgreSQL (the host loaded by two other 1M-item loads) and 12.45 s on SQLite; 500k items with two +GSIs 8.81 s on MongoDB; 1M items without a secondary index 13.5 s on MongoDB; 1M items with a 128-dimension +vector 243 s on PostgreSQL and 210 s on SQLite (222 s under `MALLOC_ARENA_MAX=1`). + +Not measured: a database on another host, an S3 store's upload and download, the spool's disk, and the restore +of a Cassandra deployment's own in-catalog backups beyond 60,000 items (11.2 s for 60,000 on an earlier build, +the differences document's [Backup and Restore](../differences-from-dynamodb.md#backup-and-restore) section). +Time a restore drill on your own hardware and store; that figure, not this table, is your recovery time. + +### Memory while a job runs + +Every backup and restore job is bounded per server by `[backup] max_concurrent_jobs` (default 4); a request +over it waits up to 30 s for a slot and is then refused with `LimitExceededException`. Size the host for the +jobs it runs at once. The admin guide's `max_concurrent_jobs` row has the figures for tables of large items. +For the tables above and two more shapes, the process's peak over its idle figure while one job ran: + +| Job | PostgreSQL | SQLite | MongoDB | Cassandra | +|---|---|---|---|---| +| restore, 1M items, key order | 16 MiB | 87 to 98 MiB | | 33 MiB | +| restore, 1M items, unordered | 16 MiB | 144 to 164 MiB | 52 MiB | | +| restore, 1M items with a 128-number vector and a vector index | 111 MiB | 552 MiB; 204 MiB under `MALLOC_ARENA_MAX=1` | | | +| backup, 1M items with a 128-number vector and a vector index | 104 MiB | 138 MiB; 58 MiB under `MALLOC_ARENA_MAX=1` | | | +| restore, 100k items with a 128-number list and no vector index | 60 to 71 MiB | 397 to 424 MiB; 184 to 186 MiB under `MALLOC_ARENA_MAX=1` | | | +| restore, 100k items with a 2,290-character string and no index | | 253 MiB; 130 MiB under `MALLOC_ARENA_MAX=1` | | | +| restore, 20,000 items with a 1,536-number vector and a vector index | | 669 MiB; 301 to 316 MiB under `MALLOC_ARENA_MAX=1` | | | +| backup, 20,000 items with a 1,536-number vector and a vector index | | 521 MiB | | | + +On SQLite a restore holds its write batch, up to 10,000 items or 32 MiB of item text, and the decode chunks in +front of it, so for items of more than a few hundred bytes its memory grows with the item's size up to that +bound, whatever the attribute types, and the vector index is not what it belongs to. 100k items of about +2.5 KB of strings with no index held 253 MiB over idle, and 130 MiB under `MALLOC_ARENA_MAX=1` (two runs each), +against 38 MiB for 100k items of 200 bytes. Lists of numbers decode larger at the same text size: 100k items +of 128 numbers held 345 to 424 MiB over the measured runs with or without a vector index, and 184 to 195 MiB +under one arena; the figure also grows with the numbers per item (the 1,536-number row). On PostgreSQL the +vector index adds 27 to 53 MiB at 100k items (five pairs of runs with and without it). Under glibc's default allocator the SQLite vector restore's +peak is higher at 1M items than at 100k (552 against 354 MiB), and the 1M run levelled off after about 150 s; +no other size was measured. Under `MALLOC_ARENA_MAX=1` the same restores held 195 MiB at 100k and 204 MiB at +1M, at a cost in duration: 32% for one 1M restore (283 s against 214 s), and for two 100k restores running at +once 57.5 s against 24.9 s under the default arenas (279 against 697 MiB over idle for the pair). The +deployment guide's service unit carries `MALLOC_ARENA_MAX=1` commented out, because one allocator arena costs +the request path a few percent (3.5 to 9.2% on GetItem, Query and Scan on SQLite). Set it on a SQLite host +that restores tables of items of more than a few hundred bytes, or backs up tables of large items, and size +such a host per running restore from the points measured under it: 130 MiB for 100k items of 2.5 KB strings, +184 to 208 MiB for items of 128 numbers (100k and 1M), 301 to 316 MiB for 20,000 items of 1,536 numbers, and +more for larger items (the admin guide's `max_concurrent_jobs` row has the large-item figures). Without it, +size for the default-allocator figures, which these measurements do not bound. The peaks varied by up to about +20% between runs of one cell, idle or loaded (345 to 409 MiB over nine runs of the SQLite 100k vector restore). + +## Configuration + +The `[backup]` section's keys (the job bound, the heartbeat and reconciler intervals, `spool_path`) and the +`[backup.store]` section are documented in the admin guide's [`[backup]`](05-admin-guide.md#backup) and +[`[backup.store]`](05-admin-guide.md#backupstore) sections. The store's keys, defaults and refusals are the +backup store crate's, `crates/backup-store/src/config.rs`. + +A filesystem store: + +```toml +[backup.store] +kind = "filesystem" + +[backup.store.filesystem] +path = "/var/lib/extenddb/backups" +``` + +The configured root must exist: the server does not create it and refuses to start, "backup root is +not usable: No such file or directory (os error 2)". Create it owned by the server's user with mode `0700`, for example +`install -d -m 0700 -o /var/lib/extenddb/backups` ([A filesystem root](#a-filesystem-root) has +the rules for the directories above it). + +An S3 store; a restore from S3 downloads to `spool_path`, which needs disk for the compressed size of the +backups being restored at once (on file-backed SQLite it defaults to `backup-spool/` beside the database file): + +```toml +[backup] +spool_path = "/var/lib/extenddb/backup-spool" + +[backup.store] +kind = "s3" + +[backup.store.s3] +bucket = "my-extenddb-backups" +prefix = "extenddb/" +region = "us-east-1" +``` + +The spool is created `0700` when it is absent. An existing one must be owned by the server's user with mode +`0700`, under directories that follow the root's rules ([A filesystem root](#a-filesystem-root)), or the +server refuses to start, naming it (a spool left `0755` by `mkdir -p` under a `022` umask is refused, +"backup.spool_path ... is not private"). + +Credentials for S3 come from the AWS SDK's default chain (environment, shared config, instance or container +role) and are never in the file. An S3-compatible server takes `endpoint`, and an `http://` endpoint needs +`force_path_style = true` and a loopback host unless `allow_insecure_endpoint = true`. The S3-compatible +material (the request checksums some servers reject, TLS trust, conditional writes) is in +[`crates/backup-store/README.md`](../../crates/backup-store/README.md). + +Defaults: on file-backed SQLite an absent section is a filesystem store at `backups/` beside the database +file. On PostgreSQL, MongoDB, Cassandra and an in-memory SQLite database there is no default, and without a +store `CreateBackup` writes the in-catalog format with a deprecation warning ([Legacy +backups](#legacy-backups)). `kind = "disabled"` keeps a deployment on the in-catalog format on purpose. + +Instances that share a catalog must share one store root: one S3 bucket and prefix, or one shared mount. A +host-local directory configured at one path on each host is not one root: once the catalog has backups there, +a second instance pointed at its own directory under that path starts with the store unhealthy. Before the +first backup nothing tells the directories apart: each instance creates its own identity file and store id, +the first backups split the rows between the two ids, and the next check reports an identity mismatch that +only `backup purge` clears (the deployment guide's [Multi-Instance +Considerations](11-deployment-guide.md#multi-instance-considerations)); with an `extenddb-store.json` copied +from one directory to the other nothing detects it, and a restart can delete the only copy of a backup (the +admin guide's [`[backup.store]`](05-admin-guide.md#backupstore) section). Point every instance at the shared +root before the first backup, and never copy `extenddb-store.json` between roots. + +`extenddb destroy` removes the databases and leaves the store: the store is the copy that survives it. +`extenddb backup purge --confirm` is what deletes a store's backups. A catalog re-created with `init` finds +the old root's `extenddb-store.json` naming the old catalog; `extenddb backup sync --adopt-store` makes the +new catalog its owner, and until then the store is unhealthy, or, on file-backed SQLite with the default root, +unconfigured, with `CreateBackup` writing in-catalog backups ([The catalog is lost and +re-created](#the-catalog-is-lost-and-re-created)). Or configure a new root. + +`/health` reports the store as the `backup_store` component, for example +`{"components":{"backup_store":{"kind":"filesystem","status":"healthy"}},"status":"healthy"}` (on MongoDB +`status` comes first). An unhealthy store (unreachable, or failing its identity check) is `status` `degraded` +with HTTP 200 kept, so a store outage never takes the data plane out of rotation; `CreateBackup`, and the +restore and delete of a backup in that store, are refused with a `ValidationException` until the next check +passes. + +## Securing the store + +### Encryption and integrity + +Backup contents are DynamoDB JSON in gzip, in plaintext. Encryption at rest is the store's job: SSE-S3 or +SSE-KMS on the bucket (SSE-KMS needs `kms:GenerateDataKey` and `kms:Decrypt` on the key for the server's +role), or an encrypted filesystem for a filesystem root. Bucket versioning and Object Lock add protection +against deletion and ransomware; ExtendDB depends on neither. A restore verifies every data file's checksums +and the manifest's before it writes any item, so a corrupted file fails the restore, naming the file, and +leaves no table behind. + +### A filesystem root + +The root must be owned by the server's user with mode `0700`, and every directory above it must be owned by +root or by that user and not writable by group or others unless it carries the sticky bit. The server +refuses anything else at startup, naming the path, for example: + +``` +[backup.store] (filesystem path /srv/backups/extenddb) cannot be opened: permission denied: /srv/backups (an +ancestor of the backup root) has mode 0775 and is writable by group or others without the sticky bit; another +user could replace the backup root. chmod it to remove group and world write, or move the backup root +``` + +So a root under a directory a storage administrator owns, under an NFS mount owned by `nobody`, or under a +group-writable parent without the sticky bit (a `root:staff 0775` `/srv`) is refused; put the root under a +directory owned by root or by the server's user. A root of mode `0755` is refused with "chmod it to 0700". +The root itself must exist before the server starts; the directories the store creates under it are `0700` +and its files `0600`. + +A copy tool that takes the root off the host (rsync, restic, borg) cannot read the tree as a member of a +group: run it as the server's user or as root, and have it copy the root as a whole while no backup job runs, +or copy whole backup prefixes whose `extenddb-manifest.json` is present (a prefix without that file is a +backup still being written). + +The owner-only checks see the tree as the kernel reports it. They cannot observe a bind mount of the root or +of a directory under it, nor a filesystem without Unix permission semantics: a root on such a filesystem +mounted with a mask that makes it report `0700` passes the check while enforcing nothing. Both are the +operator's responsibility. The root also needs hard links, which the store's create-if-absent uses; some CIFS +and FUSE mounts refuse `link(2)`, and the server then reports the store as having failed validation. + +### An S3 bucket + +The least-privilege IAM policy for the server's role, its actions and the resources each applies to, is in +[`crates/backup-store/README.md`](../../crates/backup-store/README.md) (the IAM item of its Behavior section). +The store's check probes each grant the policy lists and names a missing one. On an SSE-KMS bucket it does not +probe `kms:Decrypt`, which an upload's parts (`UploadPart`) and a restore's download (`GetObject`) use, so a +role without it passes the check and fails at the first job that needs it. Run the restore drill after any +change to the key policy. + +Lifecycle rules for the bucket: + +- `AbortIncompleteMultipartUpload` (one day): a process killed during an upload leaves a multipart upload + that only a later deletion of the same prefix would sweep; the rule removes the rest. +- On a versioned bucket (an Object Lock bucket always is), deletions add delete markers and keep the + noncurrent versions, which are billed: add `NoncurrentVersionExpiration` and `Expiration` with + `ExpiredObjectDeleteMarker: true`, or use an unversioned bucket. + +Objects are keyed `/
//`, so each account's backups sit under a +prefix of their own. Write access to an account's prefix is backup-administrator authority for that account: +whoever can write there can delete the account's backups, or replace one with a backup of other items whose +checksums and manifest agree, which a restore then accepts. Grant the server's role the whole prefix, and +grant any other principal (a tenant's tooling, a copy job) at most its own account's prefix, +`arn:aws:s3::://*`, with IAM prefix policies. + +The AWS SDK's `trace` logs carry credentials: at `trace`, `aws_sigv4` logs the canonical request with the +session token and `aws_smithy_runtime` the `authorization` header. Do not enable those targets at `trace` in +production. + +## What a backup costs the database + +A backup holds its snapshot open from the request until its last data file is written, and each backend pays +for that while it runs. The figures below were measured for this manual; the measurement record +`docs/design/15-backup-restore-bench-pr11.md` ("Figures measured for the manual") has their method, build, +load and repeat runs. + +- PostgreSQL. The snapshot is a read-only `REPEATABLE READ` transaction on the data database. While it is + open, `VACUUM` cannot remove any row version that became dead after it started, in any table of the data + database: every update and delete during the backup leaves its old row version in place until the backup + ends. Measured on the host above: during a 25 s backup (100k items with a 128-dimension vector), 12,156 + `PutItem` overwrites of 1,000 200-byte items in another table grew that table from 0.4 to 4.4 MiB, about + 345 bytes per write, and a `VACUUM` run 10 s into the backup reported its 4,948 dead row versions as "dead + but not yet removable"; the same writer for the same time with no backup left the table at 0.5 MiB. The + first `VACUUM` after the backup removed every dead version, and the table keeps the space for later writes. + Size the data database's disk for the update and delete volume of the longest backup. +- SQLite. The snapshot is a read transaction, which stops the WAL from being checkpointed past it, so the + `-wal` file grows by the pages written during the backup, and how many pages a write changes depends on the + table's indexes. Measured on the host above, beside a 100k-item backup (items with a 128-dimension vector), + with `PutItem` overwrites of 200-byte items in another table: + - a table with no secondary index: 15,472 writes during a 22.8 s backup grew the `-wal` file from 4.0 to + 61.1 MiB, about 3.9 KB per write; + - a table with two GSIs (each overwrite changing both index keys, `ALL` projection): 13,996 writes during + a 25.6 s backup grew it from 4.1 to 1,294 MiB, about 82 to 97 KB per write over four runs, some 20 + to 24 pages per write. A GSI is asynchronous at the default `index_propagation_delay_ms`: the put queues each + index write in `gsi_pending` inside the put's transaction, and later transactions apply and remove it, + each appending every page it touched. + + The same writers for the same time with no backup left the file at 4.0 and 4.1 MiB. The file keeps its size + after the backup, and after the server stops; later writes reuse it. Size the disk for the pages each write + changes, about one 4 KiB page for a table with no index and about 24 for a table with two GSIs, times the + writes during the longest backup. +- MongoDB. The server keeps snapshot history for `minSnapshotHistoryWindowInSeconds` (default 300). A backup + whose read stays open longer may fail with `SnapshotTooOld`; the backup is then `FAILED` and must be taken + again. A table that takes longer than the window to back up needs the parameter raised on the replica set + (`setParameter`), which costs the server memory for the longer history. On the host above, 1M items of 200 + bytes backed up in 13.5 s. + +On PostgreSQL each running job also holds a dedicated connection outside the pool, whose session advisory +lock is the job's claim; startup raises `pool_size` and `catalog_pool_size` to at least +`2 * max_concurrent_jobs + 10`, and `max_connections` must leave room for it (the deployment guide's +connection pool section). The claim's session sets TCP keepalives (30 s idle, then three probes 10 s apart), +so after the loss of the owner's host the database ends the session about a minute later. Another instance's +reconciler takes the job over only once that session has ended and the job's heartbeat is older than +`[backup] stale_job_after_secs` (default 300 s). An owner whose host still answers but whose process is +stopped (a paused VM, `SIGSTOP`) keeps its session, and with it the job, until it resumes. To end the +session, find it by its lock key, `backup:` for a backup (the last component of the backup ARN) +and `/
` for a restore into that table, and terminate it: + +```sql +-- in the catalog database +SELECT l.pid FROM pg_locks l + WHERE l.locktype = 'advisory' AND l.objsubid = 1 AND l.granted + AND (l.classid::bigint << 32 | l.objid::bigint) = hashtextextended('backup:', 4539474) + AND l.database = (SELECT oid FROM pg_database WHERE datname = current_database()); +SELECT pg_terminate_backend(); +``` + +The job keeps writing until its next heartbeat (within 15 s), then stops with no further store or catalog +access, and once its heartbeat is older than `stale_job_after_secs` the reconciler resolves it: a backup with +no complete manifest becomes `FAILED` (`DescribeBackup` answers `BackupNotFoundException`) and its partial +objects are deleted; a store-backed restore is resumed from its recorded progress (the differences document's +restore-interruption row). Do not terminate the session whose `application_name` is `extenddb serve`: that is +the server's own registration. The server then refuses requests with `ServiceUnavailable` and stops its +background work while it registers again, and every store-backed backup it is running fails (reason +`shutdown`) and must be taken again. + +A transaction-mode pooler (PgBouncer with `pool_mode = transaction`) is unsupported for the +claim: the lock lands on a pooled server connection and is not released with the job. Reach the catalog +database directly or through a session-mode pooler. + +## Restore drill + +Run a drill on a schedule and after every change to the store, the hosts or the release. It proves that the +backups restore, and its duration is your recovery time. + +Every `aws` command below is for the ExtendDB server: give each one `--endpoint-url https://:18443`, with +`AWS_CA_BUNDLE` naming the server's certificate, as the crontab line above does. Without the flag the CLI sends +the request to Amazon DynamoDB, with whatever credentials the host's profile holds. + +1. Pick a backup: `aws dynamodb list-backups --table-name orders`, or take one with `create-backup` and poll + `aws dynamodb describe-backup --backup-arn ` until `BackupStatus` is `AVAILABLE`. On the server, + `extenddb backup list --config extenddb.toml` prints the store-backed backups with their size, item count + and store; it runs beside a server. +2. Restore it under a new name: + `aws dynamodb restore-table-from-backup --backup-arn --target-table-name orders-drill`. The answer is + `CREATING` at once. +3. Poll `aws dynamodb describe-table --table-name orders-drill` until `TableStatus` is `ACTIVE` and every + index reports `ACTIVE`. Until then the table answers reads with `ResourceNotFoundException`, and + `DeleteTable` on it is refused. Note the elapsed time. +4. Compare: `aws dynamodb scan --table-name orders-drill --select COUNT` (follow `LastEvaluatedKey`) against + the backup's `SourceTableDetails.ItemCount` from `describe-backup`, and read a few known keys with `get-item`. + `DescribeTable` reports `ItemCount` 0 until the statistics pass next runs, as the service does, so count + with a scan. +5. Delete the drill table: `aws dynamodb delete-table --table-name orders-drill --endpoint-url https://:18443`. + +A drill on the same server proves the backup. To prove the off-host copy, run it on a separate deployment: +copy the backup's prefix (`/
//`, every object under it) under the drill +deployment's own store root, with the same relative path, as the drill server's user, keeping directories +`0700` and files `0600` (`cp -rp`, `rsync -a`, or `chmod -R u=rwX,go=` on the copy afterwards). Then run +`extenddb backup sync` there (below), and restore. The drill deployment needs the account the backup's ARN +names. A copy that lands readable by other users, as `aws s3 cp --recursive`, `tar` or `rsync` without +permissions under a `022` umask leave it, is refused: `backup sync` exits 1 naming the directory ("permission +denied: has mode 0755 and is accessible to other users; chmod it to 0700"), a restore of such a +backup is refused (`InternalServerError`; the server log names the directory), and when the copy is the newest backup the store's next check marks the whole store unhealthy. +Run `chmod -R u=rwX,go=` on the copy and `extenddb backup sync` again. + +A restore that fails (a checksum mismatch, found before any item is written, or an item the target refuses +during the load) removes its target, +and the server logs the reason with the backup ARN. A store-backed restore stopped by the server stopping is +resumed by a server's reconciler from its recorded progress; an in-catalog one is failed and its target +removed. If a resume finds a target holding more +items than the backup and no recorded progress, it fails as `target_not_empty` and leaves the target +`CREATING`; `extenddb backup remove-target --account
` removes it (the admin guide's +`backup` section). + +## Recovering from catalog loss + +The store holds every store-backed backup; the catalog holds the rows that index them, the accounts and IAM. +After a catalog is lost, bring the catalog back, then give it a row for each backup with `extenddb backup +sync`. + +### The catalog is restored from a database backup + +Restore the catalog from its database backup (the admin guide's [Data +Recovery](05-admin-guide.md#data-recovery) entry). On PostgreSQL the catalog and data databases are one copy: +restore both, either from dumps taken together with the server stopped, into new, empty databases owned by the +application role as the entry gives it (restoring over the existing databases restores nothing), or from one +base backup of the PostgreSQL server that held both, with PostgreSQL's own procedure. On a new PostgreSQL +server create the application role before the dumps' restore and, when the server's address changed, set the +data database's address in the restored catalog, as the entry's last paragraph says. On SQLite the database +file holds the catalog and the tables; restore it as the entry's SQLite row says. Either way every table returns to the copy's moment, and a table created after it is gone. The restored catalog has the same catalog id, so it still +owns the store root, and its accounts and IAM are back. Then run, with the servers stopped on SQLite and +beside them on the other backends: + +```bash +extenddb backup sync --config extenddb.toml +``` + +Sync lists the root and inserts an `AVAILABLE` row for each complete backup the catalog has no row for (the +backups taken after the catalog's backup). It reports the prefixes it skipped (an incomplete backup, one the +catalog already has) and the `AVAILABLE` rows whose manifest is no longer in the store (backups deleted after +the catalog's backup); `extenddb backup sync --prune-missing` removes those rows and deletes nothing in the +store. When the newest of those rows is the catalog's newest `AVAILABLE` backup, the store is unhealthy (each +check looks for that backup's manifest, and `/health` reports `"reason":"manifest probe failed"`): both +commands refuse ("sync changes no row while the store is unhealthy"), as do `CreateBackup` and the restore and +delete of every backup in that store, and the state survives a restart. Putting that backup's prefix back from +a copy clears it. This release has no command that clears it without such a copy: `--prune-missing` refuses +while the store is unhealthy. To stay out of this state, take each catalog backup right after the deletions +your scheduler makes, so that the newest backup a catalog backup names is not one deleted later. + +Tables backed up after the copy can be restored, under their own names or others, once `backup sync` lists +their backups; their writes after their last backup are lost. + +A PostgreSQL catalog database restored alone, beside a data database that survived, is a catalog older than +the tables, and gives this: a table created after the copy is not listed, and creating it again gives an empty +table while its old items stay in a table of the data database that nothing references; a table deleted after +the copy is listed and answers `ResourceNotFoundException` until it is deleted again. Restore the pair +instead. + +### The catalog is lost and re-created + +When no catalog backup exists, `extenddb init` creates a new catalog with a new catalog id. The store's +`extenddb-store.json` still names the old catalog, so the new server starts with the store unhealthy: + +``` +{"components":{"backup_store":{"kind":"filesystem","reason":"owned by another catalog","status":"unhealthy"}},"status":"degraded"} +``` + +On file-backed SQLite with the default root (no `[backup.store]` section) the server instead logs a warning, +"the default root ... failed its check (extenddb-store.json names catalog ; this catalog is ); the catalog has no backup rows for it, so the store is unconfigured and CreateBackup writes in-catalog +backups", and `/health` shows no `backup_store` component (`{"components":{},"status":"healthy"}`). Until +the adoption below, `CreateBackup` answers `AVAILABLE` at once and writes an in-catalog backup into the +database file, the failure domain this manual's first paragraph warns about; such a backup stays in-catalog +after the adoption ([Legacy backups](#legacy-backups)). Pause scheduled backups until the adoption, or +configure `[backup.store]` explicitly, which gives the unhealthy store above instead. + +In both shapes plain `backup sync` refuses (exit 1, "owned by another catalog"). Then: + +1. Stop every server of the old catalog, on every host. A server that still runs keeps acting on the root + until a later check finds it adopted. +2. Start a server of the new catalog: `extenddb manage` works through its management API. Re-create the + accounts with their old account ids, `extenddb manage --user admin create-account --account-id + --account-name ` (the admin guide's Accounts section), and their IAM users and policies. A backup is + reachable only by callers of the account its ARN names. `create-access-key` makes new key pairs; to keep + the clients' existing ones, import each with `extenddb manage --user admin import-access-key --account-id + --user-name --access-key-id --secret-access-key --yes`. While that command + runs, the secret is in its arguments, which the process list shows to every user of the host. On a shared + host send the pair to the management API instead, in a request body read from a file: + `curl --cacert --netrc-file -H 'Content-Type: application/json' --data @key.json + https://:18443/management/accounts//users//access-keys/import`, where `key.json` holds + `{"access_key_id": "", "secret_access_key": ""}` and the netrc file the admin user's name + and password, both created under `umask 077`; the answer is `Access key imported`. On SQLite stop the + server before step 3. +3. Adopt the root and sync it into the new catalog, on SQLite with the server stopped and on the other + backends beside the running servers: + + ```bash + extenddb backup sync --adopt-store --config extenddb.toml + ``` + + It rewrites `extenddb-store.json` to name the new catalog (keeping the store id), logs the change under + `extenddb::audit`, and syncs: + + ``` + Adopted the root: extenddb-store.json named catalog and now names this catalog (store id ). + ... + --- Adopted (an AVAILABLE row inserted for each): + arn:aws:dynamodb:us-east-1::table/orders/backup/ + ... + === extenddb backup sync complete: 1 backup(s) adopted, 0 prefix(es) skipped, 0 AVAILABLE row(s) whose manifest is gone === + ``` + +4. On SQLite start the server again; on the other backends the running servers find the store adopted at a + later check. `/health` reports the `backup_store` component healthy (on SQLite's default root as well), the + backups are `AVAILABLE`, and each restores as in the drill. The new catalog knows none of the old tables; restore each + from its backup. + +On SQLite every `backup sync` needs the server stopped, since it opens the database as the server does. On +PostgreSQL, MongoDB and Cassandra it runs beside servers and holds the catalog's maintenance claim while it +runs, during which `CreateBackup` and `RestoreTableFromBackup` on every instance answer +`LimitExceededException`; `--adopt-store` and `--prune-missing` first wait, as `purge` does, until no backup +or restore job is running and no backup failed within `stale_job_after_secs` (a job that failed may still be +writing to its prefix), at most `2 * stale_job_after_secs`, and then refuse. Sync exits 0 when it completed +and 1 when it refused or skipped a prefix for a store or catalog error (run it again). It exits 6 when it +cannot open the catalog: beside a running server on SQLite ("another extenddb process is already using ..."), +or on the other backends while a migration runs. On a catalog `serve` would refuse it exits with `serve`'s +code: 3 with migrations pending (run `extenddb migrate`), 4 on a catalog a later release migrated with a +breaking migration, and 5 on one that needs attention (the upgrade manual's [catalog +status](07-upgrade-manual.md#the-catalog-status)). +The admin guide's `backup sync` paragraph has its options and output in full. + +### An in-memory SQLite server + +A server on an in-memory SQLite database (`[storage.sqlite] path = ":memory:"`) starts with a new, empty +catalog every time. With `[backup.store]` configured it adopts, at startup, a root that an earlier in-memory +server owned, and syncs the root's backups into its new catalog before it serves (on a filesystem store; an +S3 store in the reconciler's first cycle). The log line + +``` +backup store (filesystem) startup sync of the in-memory catalog: 1 backup(s) adopted, 0 prefix(es) skipped (0 of them reserved for exports or the change-log archive) +``` + +ends the window in which the earlier backups answer `BackupNotFoundException`. Tables, accounts and IAM do not +survive the restart; only the backups do. A backup is reachable only by callers of the account its ARN names, +and the in-memory server creates its default account with a new id at every start, so until that account +exists again the adopted backups answer `AccessDeniedException` and `ListBackups` lists none of them. After +each start, re-create the accounts with their old ids (`extenddb manage --user admin create-account +--account-id --account-name `, as the admin user, whose password is the +`EXTENDDB_ADMIN_PASSWORD` the server started with or, without it, the one it printed at start), then their +users, policies and keys; the backups then answer `DescribeBackup` and restore. Start a new in-memory server +only after the old one has exited. A root that a durable catalog owns is never adopted by an in-memory server. +Every `extenddb backup` command refuses an in-memory database (exit 1), since it would open a new, empty +catalog that ends with the command. + +## Legacy backups + +A backup written in the in-catalog format (by an earlier release, by a server without a configured store or +with `kind = "disabled"`, and on Cassandra always) lives in the database the server uses: on PostgreSQL in +the catalog database, on SQLite in the database file, on MongoDB in the catalog and data databases, on +Cassandra in the catalog keyspace and the account's keyspace. It shares the database's failure domain, and a +lost database loses it. + +Legacy backups stay listable, describable, restorable and deletable whether or not a store is configured: +`ListBackups` lists both kinds side by side. `extenddb backup list` lists only the store's rows. The +in-catalog format is deprecated: without a store `CreateBackup` logs a warning on each call ("is writing the +in-catalog backup format, which a later release stops creating; configure [backup.store] ..."), and a later +release stops creating it. + +This release has no command that converts a legacy backup to the store format (`backup migrate-legacy` is +planned, #821). To move one into the store today, with the store configured: + +1. Restore it under a new name: `aws dynamodb restore-table-from-backup --backup-arn + --target-table-name orders-moved --endpoint-url https://:18443`, and wait for `ACTIVE`. +2. `aws dynamodb create-backup --table-name orders-moved --backup-name --endpoint-url + https://:18443` and wait for `AVAILABLE`. This is a store-backed backup with a new ARN and a new + `BackupCreationDateTime`, of a table named `orders-moved`. +3. Check the new backup with a drill, then `aws dynamodb delete-backup` the legacy one and `aws dynamodb + delete-table` the interim table, each with `--endpoint-url https://:18443`. + +To keep the original table name, restore into it only once that table is gone: deleting a live table to make +room loses every write since the legacy backup. While the table exists, `create-backup` of it is a +store-backed backup of that name with its current items; keep the legacy backup only for its older contents. +A backup taken by an earlier release may restore without its secondary indexes +(the upgrade manual's Current Status section); take a new backup of each table after an upgrade rather than +moving the old ones. A table with a vector index has no legacy backup: the in-catalog format refuses it. + +## What a restore preserves + +A restore reproduces what Amazon DynamoDB's does, on every backend, with one gap (a GSI's on-demand throughput): + +| Carried across | Not carried across | +|---|---| +| items, with every number's 38 digits | table class (a restore is `STANDARD`) | +| key schema and attribute definitions | stream settings | +| global and local secondary indexes, rebuilt from the items | TTL settings | +| billing mode and provisioned throughput, table and GSI | tags | +| the table's on-demand throughput | a GSI's on-demand throughput (ExtendDB records none) | +| SSE settings | deletion protection | +| vector indexes, into PostgreSQL and SQLite | auto scaling, IAM policies, CloudWatch settings | + +Reapply what is not carried by hand after the restore. The overrides of `RestoreTableFromBackup` +(`BillingModeOverride`, `GlobalSecondaryIndexOverride`, `LocalSecondaryIndexOverride`, +`SSESpecificationOverride`, `OnDemandThroughputOverride`, `ProvisionedThroughputOverride`) apply as on the +service; `VectorIndexOverride` is refused. + +Across backends: a store-backed backup restores into any backend, with these refusals, each before any table +is created or with the target removed: + +- A backup with vector indexes is refused into MongoDB and Cassandra, which have no vector support. +- Into MongoDB, an `N` sort-key value of more than 34 significant digits fails the restore (the sort-key + column is Decimal128); other numbers keep their 38 digits. + +`DescribeTable` on a restored table reports `ItemCount` and `TableSizeBytes` 0 until the statistics pass +rewrites them. The differences document's [Backup and Restore](../differences-from-dynamodb.md#backup-and-restore) +section has each of these rows in full, with the multi-part-key preview and the restore's error texts. + +## Moving data to and from Amazon DynamoDB + +The backup format is the DynamoDB S3 export layout: each backup's `data/` directory holds gzip files of +DynamoDB JSON, one `{"Item": {...}}` object per line, beside the export's `manifest-summary.json` and +`manifest-files.json`, plus ExtendDB's own `extenddb-manifest.json` and its checksum file. + +From ExtendDB to Amazon DynamoDB: upload a backup's `data/` files to an S3 bucket and run the service's +`ImportTable` with `InputFormat` `DYNAMODB_JSON`, `InputCompressionType` `GZIP`, an `S3BucketSource` whose +`S3KeyPrefix` is that directory, and `TableCreationParameters` built from the backup: the key schema from +`describe-backup`'s `SourceTableDetails`, the GSIs from its `SourceTableFeatureDetails` (for an on-demand +table drop each GSI's `ProvisionedThroughput`, which the backup reports as 0 and 0 and the SDK's parameter +validation refuses), and the attribute types (`AttributeDefinitions`, which neither carries) from the +`source_table` object of the backup's `extenddb-manifest.json`. `TableCreationParameters` has no member for +local secondary indexes, so the service's import creates none; a table with LSIs arrives without them. That is +the layout the service's import reads; this release has not run the import against the service. + +From Amazon DynamoDB to ExtendDB: export the table to S3 with `DYNAMODB_JSON`, download the export's data +files, decompress them and concatenate them into one file under an `[import]` path, in the calling account's +directory (`//`), then call ExtendDB's `ImportTable` with a `FileSource` naming that +file and `InputFormat` `DYNAMODB_JSON`. Name the file by its canonical path in `FileSource.Path`: a path +through a symbolically linked directory is refused, "Path must resolve under one of the configured allowed +paths", whether or not the `[import] paths` entry is itself written through that link, and a file that is +itself a symbolic link is refused with `ValidationException` "Symbolic links are not allowed in import/export +paths". ExtendDB's `ImportTable` reads one uncompressed local file (a gzip file is refused, "I/O error reading +import file at line 1"), creates the table from `TableCreationParameters`, reads the whole file before it +writes the first item, completes before it answers, and skips the items that fail validation (keys that do not +match the key schema, an item over the size limit), counting them in `ErrorCount`; the file is bounded by +`max_import_file_bytes` (10 GiB) and `max_import_item_count` (10 million). The same works for one of +ExtendDB's own backup data files. + +ExtendDB's `ExportTableToPointInTime` writes one uncompressed DynamoDB JSON file under an `[export]` path, in +the calling account's directory, at the path the request's `FilePath` member names (an ExtendDB member; a +request without it reads `S3Prefix` as the path, and `S3Bucket` is accepted and ignored, so a request naming +neither path is refused, "Either FilePath or S3Prefix must be provided"). It reads the table as it changes, so it is not a consistent snapshot. Reading a store prefix or a +DynamoDB export directly, with `ImportTable` or a restore, is planned (#748) and not in this release; `backup +sync` adopts only prefixes with an ExtendDB manifest. + +## License + +Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. +See [LICENSE](../../LICENSE) for the full text. + +This software is provided "as is" without warranty of any kind. ExtendDB is not +affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark +of Amazon.com, Inc. diff --git a/docs/rfcs/draft-cassandra-backend.md b/docs/rfcs/draft-cassandra-backend.md index d40f74118..4433af568 100644 --- a/docs/rfcs/draft-cassandra-backend.md +++ b/docs/rfcs/draft-cassandra-backend.md @@ -190,6 +190,12 @@ high expiration rates will accumulate backlog. ### Backup and restore +> Superseded in part. This section describes the first implementation. Restores now run as background jobs that +> restore the secondary indexes, and a backup in a configured backup store restores into Cassandra; +> `CreateBackup` on Cassandra still writes the in-catalog backup, before it returns. See the +> [Backup and Restore manual](../manuals/14-backup-and-restore.md) and the design, +> [15-backup-restore.md](../design/15-backup-restore.md). + Cassandra's native snapshot mechanism (`nodetool snapshot`) produces node-local SSTable artifacts managed outside CQL. Creating, enumerating, or restoring a distributed snapshot requires cluster-level orchestration that is not available through the driver session, so the backend implements on-demand backups as logical snapshots instead. @@ -234,6 +240,11 @@ writes can race the scan. `RestoreTableToPointInTime` is not supported. and `ImportTable` execute synchronously and block the API response until complete. Amazon DynamoDB performs these asynchronously. +> Superseded in part: `RestoreTableFromBackup` now answers `CREATING` and copies in the background on every backend, +> and `CreateBackup` with a backup store does so on PostgreSQL, SQLite and MongoDB. On Cassandra `CreateBackup` still +> completes before it returns, as do `ExportTableToPointInTime` and `ImportTable` on every backend. See the note under +> "Backup and restore". + **Transaction recovery is conservative.** Transactions found in PREPARING state during recovery are always rolled back, even if they could theoretically have been committed. This is safe but means a transaction that completed PREPARE and then encountered a server crash will be rolled back rather than committed. @@ -252,8 +263,15 @@ path is suitable at scale. `ImportTable` to background jobs with status polling is planned but not yet implemented. This applies to all current ExtendDB backends. +> Superseded in part: the restore, and `CreateBackup` with a backup store on PostgreSQL, SQLite and MongoDB, now run +> as background jobs (the note under "Data movement operations are synchronous" above). + **Restore does not recover GSI/LSI definitions, tags, TTL configuration, or stream history.** +> Superseded in part: a restore now recovers the GSI and LSI definitions, rebuilt from the items, on every backend, of a backup +> that recorded them; a backup an earlier release took may not have (the upgrade manual's Current Status section). +> Tags, TTL configuration and stream history are still not restored. + **Transaction fence uses item data equality.** The TTL deletion protocol conditions the final base item delete on the stored item image rather than a version counter. A monotonic version column would be a more robust fence and remove the dependency on stable JSON encoding. diff --git a/docs/technical-debt.md b/docs/technical-debt.md index 44d79f8e8..f2656616b 100755 --- a/docs/technical-debt.md +++ b/docs/technical-debt.md @@ -76,7 +76,7 @@ Last updated: 2026-09-16 ### A-1: Catalog/Data Database Separation -**Design requirement:** Two databases — catalog (`extenddb`) for metadata, data (`extenddb_data`) for user items (REQ-CAT-001, REQ-CAT-002). +**Design requirement:** Two databases — a catalog for metadata and a data database for user items (REQ-CAT-001, REQ-CAT-002); on PostgreSQL `init` names them `extenddb_catalog` and `extenddb` by default. **Current state:** Implemented on PostgreSQL. `extenddb init` creates both databases, records the data connection string in the catalog `settings` table (`storage/src/bootstrapper.rs:139`, called at `app/src/cmd_init.rs:334`), and initialises the data schema. The runtime reads that setting and opens a second pool (`storage-postgres/src/lib.rs:238-256`). Table creation, the GSI queue, the item-write paths, vector index data, and the backup and stream engines all take that pool (`create_table.rs:311`, `gsi_queue.rs`, `data/put_item.rs`, `data/vector_index.rs`, `backup_engine.rs`, `stream_engine.rs`), and `create_table.rs:300` commits catalog metadata before creating data tables. `extenddb destroy` drops both databases (`storage-postgres/src/bootstrapper.rs:725` and `:734`). SQLite co-locates catalog and data by design, which its worker module states (`storage-sqlite/src/workers.rs:11`). diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 2151440b3..8296c0ea9 100755 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -910,7 +910,7 @@ The error log line names the exact relations for one leaked table and gives this pool_size = 50 # default is 20 ``` -If the problem persists, check for long-running queries or connection leaks with `SELECT * FROM pg_stat_activity WHERE datname = 'extenddb_data';`. +If the problem persists, check for long-running queries or connection leaks with `SELECT * FROM pg_stat_activity WHERE datname = 'extenddb';` (the data database's default name). **Known limitation:** The HTTP status code should be 503 with a `Retry-After` header. This is tracked as technical debt.