fix(postgres): build a GSI added by UpdateTable correctly while writes continue

[robinnsc]

What

Adding a GSI with UpdateTable on PostgreSQL now produces a correct index when writes keep going during the build. Before this change the index ended up wrong or the call failed. The same cached-definitions bug on SQLite is fixed alongside.

The new module crates/storage-postgres/src/data/gsi_build.rs documents the ordering argument. The build:

• update_table.rs creates the GSI's data table before it commits the indexes row, and commits that row as CREATING instead of ACTIVE. Every write reads the catalog, so once the row commits, writes maintain the index exactly as they maintain an ACTIVE one, inline or through the propagation queue.
• After the commit, UpdateTable waits 1 s. It then takes LOCK TABLE <base> IN SHARE MODE and releases it at once; the lock waits for writes already in flight. Writes arriving meanwhile queue behind the request, each holding a data pool connection, so the request has a lock timeout (one second over deadlock_timeout, which is what lets it cancel an autovacuum) and is retried up to six times with growing pauses: with the default deadlock_timeout, about 12 s of write stalls spread over 20 s in the worst case before the build is left to recovery.
• The backfill runs in keyset pages of 256 over the full base key, up to the last key present when it starts. Each page locks its rows FOR SHARE SKIP LOCKED and writes each item's current state into the index. Rows a writer held are then done one at a time with a waiting lock, so a page never holds row locks while it waits on another one. Pages are retried on 40P01 and 40001.
• A last pass deletes index rows whose item no longer exists. It locks the rows first and then deletes those exact rows by ctid with a fresh snapshot, so a writer re-creating the item at that moment keeps its row.
• When the backfill finishes, the index flips to ACTIVE. If the build fails for a transient reason (a conflict, a lock timeout, a lost connection or an empty pool, a shutdown), the index stays CREATING and the request still succeeds. A new background worker (recovery_worker, every 60 s, with backoff per index) finishes such indexes, and also those left behind by a crash. Any other error, including a row that does not decode, counts as one that would recur: the index is removed and the request fails, as before, and the recovery worker does the same when it meets one. The removal only touches a row still CREATING, so a second build of the same index (the recovery worker next to an UpdateTable that could not take build ownership) cannot remove an index the first one has published. The data table is dropped a second after the catalog row, and the synchronous write path treats a table that is gone as an index that is gone, so a write that read the catalog just before the removal does not fail.

Queue and write path:

• gsi_queue.rs: a queued GSI update now applies the item's current state, read under its row lock (resync_base_key), instead of the image the row carried. A row applied late, twice, out of order by a second server process, or after the backfill already wrote a newer state can no longer leave the index behind the table.
• index.rs: the inline (synchronous) path writes a GSI the way the backfill and the queue do (write_state): delete the item's rows by base key, then upsert the current state on the index primary key. LSIs keep the old image-based delete.
• delete_table.rs: the base data table is dropped before the index tables. The other order could deadlock with a backfill page (share lock on base rows, then index writes) and, with DeleteTable as the victim, leave data tables behind after the catalog rows were gone.
• Index attribute definitions now come from the write's catalog read, which now joins tables, instead of the cached TableKeyInfo. The engine drops that cache only when UpdateTable returns. Until then, writes built rows for a new GSI with a RANGE key on a new attribute without the sort key column, and skipped the type check on that attribute.

The remaining gap. The catalog and the data are separate databases, so a write that read the catalog just before the index row committed cannot learn about the index in its own transaction. Two things cover that write:

• Each write stamps the time of its catalog read (after the catalog connection is acquired, so pool wait is not counted), on CLOCK_BOOTTIME so suspend time is counted.
• If more than 500 ms has passed when it is about to commit, it re-reads the catalog and, in the same transaction, queues a repair row for any GSI it did not know about.

A write whose base-table statements all returned within 500 ms holds the table's ROW EXCLUSIVE lock from the first of them until it commits, and the barrier comes at least 1 s after the catalog commit, so the barrier waits for it and the backfill sees it. A slower write repairs itself. Every source of measurement error makes a write look slower or the wait longer, which is the safe direction.

A write with no transaction (a plain put or delete on a table the write believed had no index) takes the transaction path instead if it is already past 500 ms when it decides. One that becomes slow only after that queues its repair in a separate transaction after the write; if the process dies between the two, that item stays out of the index (or, for a delete, its old row stays in) until the item is next written. The module doc records this.

SQLite: SQLite took a new GSI's attribute definitions from the cached TableKeyInfo in the same way. A write landing after the build but before the cache was dropped wrote the index row without its sort key column, so a Query on the full index key missed the item. CI found this with the new pytest. SQLite now reads the definitions with its index read (fetch_indexes_for_table joins tables) and uses them to build rows, queue rows, and validate keys. SQLite already reads its index set under its write lock, so it does not have the cross-database window.

Other changes:

• The OFFSET-based backfill_gsi in table_helpers.rs is removed.
• Docs: docs/design/04-component-storage.md §6.2 describes the build as implemented.
• CI: the two new storage-level test files are added to the PostgreSQL storage-level step in integration.yml.

Why

No issue filed. Found in the 1.0 readiness review (P0-4: adding a GSI under concurrent writes fails on PostgreSQL). The readiness document puts the cause in the index table's primary key. That is not it: the PK already includes the base key (ddl.rs, (pk, base_pk, base_sk...)). The causes on main:

1. The indexes row committed ACTIVE before the data table existed. Writes in between failed with 42P01 (synchronous GSIs). Asynchronous ones queued rows that the worker skipped as "table dropped", so those rows were lost.
2. backfill_gsi paged ORDER BY key LIMIT 500 OFFSET n while writes landed. Rows were read twice, which hit duplicate key value violates unique constraint and failed UpdateTable with InternalServerError, or were skipped.
3. Queue rows applied their own old/new images, and deleted the old row only if the old image had the index keys. A row applied after the backfill wrote the same item collided (23505, retried forever, stalling its partition) or left a second row for the item.
4. Writes that read the catalog before the index existed maintained nothing. The backfill could pass their item before they committed.

Measured with a script that loads 2,000 items, runs 4 writer threads (put, update of the key, REMOVE of the key, delete, TransactWriteItems), and adds a GSI mid-traffic. On main, 3 of 3 runs ended ACTIVE but wrong: 15 to 18 rows missing, 33 to 38 extra and 40 to 50 stale out of about 1,770. With a RANGE key on a new attribute, UpdateTable failed with InternalServerError. On this branch every run matches the table: hash-only and hash+range GSIs, asynchronous and synchronous propagation, 4 and 8 writers. SQLite is not affected, because it holds its write lock for the whole backfill, and it passes the same test on main.

Not closed here

• Query and Scan on a CREATING GSI are not refused; DynamoDB refuses them with a ValidationException. Main returned the partial index while reporting it ACTIVE. This change makes the status honest but does not add the read gate.
• UpdateTable still returns after the build rather than right away with CREATING, as before.
• Vector indexes have the same cross-database window for writes that read the catalog just before an index was created. They are left as they are.
• On MongoDB, the new pytest is marked xfail. With these writers, concurrent UpdateItem and TransactWriteItems calls that create the same item return HTTP 500 (ImmutableField on _id). That also happens on a table with no index at all, so it is the MongoDB write-race issue that test_put_item_create_race.py already marks, not this change.
• On Cassandra, the new pytest is skipped. In CI its writers got HTTP 500 from DeleteItem under contention on one item, and the online build had not finished after 10 minutes of them. Both are Cassandra backend issues worth their own issue.
• If a client disconnects mid-build, the build still finishes, because it runs in its own task. The engine's cached TableKeyInfo on that node is not invalidated, because that happens after the storage call returns. This was already true on main.

Cost

• No extra SQL on the write path unless a write takes more than 500 ms. Above that, it costs one catalog read, once per table for TransactWriteItems.
• On tables with a synchronous GSI, each write does a DELETE by base key before its upsert (an overwrite already did; a new item did not), and the GSI maintenance runs inside one SAVEPOINT/RELEASE pair per write so that an index removed meanwhile does not fail the write (the pass is redone without that index). With N synchronous GSIs that is 2N + 2 data round trips per write where there were N to 2N. Asynchronous GSIs, the default, add nothing on the write path.
• A queue apply now reads the base row FOR SHARE (three statements for a present item instead of one or two). That lock means a queued update and a concurrent write to the same item now wait for each other, where before the worker only touched the index table; a long foreground transaction holding a row holds up its queue partition for that long.
• The backfill writes two statements per row (delete by base key, upsert) in one transaction per page, against main's one INSERT per row, and ends with an anti-join of the index against the table.
• Write throughput and queue drain time showed no measurable difference with 8 client threads on one async-GSI table (about 610 writes/s on both, queue drained within 20 ms), on distinct and on 20 hot keys. The client was the limit there, so this is not a ceiling measurement.
• Each UpdateTable that creates a GSI takes at least 1 s longer.

Testing done

New tests/test_gsi_online_backfill.py: adds a GSI while four writer threads run puts, key updates, key removals, deletes and TransactWriteItems, then compares a Scan of the index with a Scan of the table (missing, extra, stale, and items indexed twice), for a hash-only GSI and a hash+range GSI on a new attribute. On main both fail on PostgreSQL. On this branch both pass on PostgreSQL and SQLite. On main, SQLite fails the hash+range case only intermittently (it did in CI); the deterministic test below covers that case. On MongoDB both are xfail and on Cassandra both are skipped (see above).

New crates/storage-postgres/tests/gsi_online_build.rs. On main every test fails except the recovery test, which needs code this PR adds.

• queue_rows_applied_out_of_order_converge_to_the_item: on main, the item is indexed under both its old and new key.

• queue_row_for_a_row_the_backfill_wrote_is_applied: on main, 23505 and the queue never drains.

• sync_write_over_an_existing_index_row_succeeds: on main, the put fails with 23505.

• sync_write_survives_a_gsi_whose_table_is_gone: two synchronous GSIs, one whose data table is gone; the write succeeds and the other index is right.

• concurrent_writes_during_a_build_leave_the_index_equal_to_the_table: 1,200 items and 4 writers, synchronous then asynchronous. On main, writes fail with 42P01.

• recovery_finishes_a_gsi_left_creating: missing, stale and orphan rows are put right, and the index becomes ACTIVE.

• write_with_key_info_older_than_a_new_gsi_indexes_its_sort_key: a put carrying key info from before the GSI was added. On main, the index row has no sort key. The same test is added for SQLite in crates/storage-sqlite/src/update_table.rs, where it also fails without this change.

New crates/storage-postgres/tests/gsi_stale_writer.rs: a hidden test hook (pause_writes_after_index_read_for_tests) makes writes pause 3 s after their catalog read, so they write after the build has finished. It covers a new item, a key change and a delete, on the autocommit path and the transaction path. With the repair turned off by hand, the test fails: all three items are wrong in the index.

Full PostgreSQL pytest run (tests/, -n 6) against servers built from this branch and from main: no test fails on the branch that passes on main. The failures that are common to both come from the local environment (CLI lifecycle without a connection string, import/export paths not configured). The same comparison for the full SQLite pytest run: no new failures. The Rust integration suite (tests/rust, --test-threads=1) has the same 3 failures on both.

Seven rounds of independent review of the design and the implementation. Each round's findings are fixed here or listed under "Not closed here".

cargo fmt --all -- --check
cargo +1.97.0 clippy --all-targets -- -D warnings
cargo +1.88.0 check --workspace --locked
EXTENDDB_TEST_PG_CONNECTION_STRING=... cargo test --workspace --exclude extenddb-storage-cassandra
EXTENDDB_TEST_PG_CONNECTION_STRING=... cargo test -p extenddb-storage-postgres --test gsi_online_build --test gsi_stale_writer

Checklist

• I have read CONTRIBUTING.md
• All tests pass (cargo test --workspace)
• Code is formatted (cargo fmt --check)
• Clippy is clean (cargo clippy -- -W clippy::pedantic)
• I have added or updated tests for new functionality
• I have updated documentation if behavior changed
• Breaking changes are noted below (if any)
• If this changes the wire protocol, Storage trait, auth model, on-disk
  format, or public CLI surface, an RFC has been accepted or is linked
  below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: n/a. There is no schema migration, no wire change and no trait change.

Breaking changes

None on the wire. On PostgreSQL, DescribeTable reports a GSI that is being built as CREATING instead of ACTIVE.

---

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.