Repository navigation
fix: answer requests during a table's DELETING window as the service does - #550
LeeroyHannigan wants to merge 16 commits into
Conversation
7075440 to
b0ca085
Compare
b0ca085 to
360f822
Compare
…does A second DeleteTable and an UpdateTable during a table's DELETING window answer 400 ResourceInUseException with "Attempt to change a resource which is still in use: Table is being deleted: <name>" on all four backends. PutItem, GetItem, Query and Scan are served during the window on PostgreSQL and SQLite and answer ResourceNotFoundException once the table is gone; a write made in the window goes away with the table. Two concurrent DeleteTable calls on one ACTIVE table get one description and one ResourceInUseException on PostgreSQL, SQLite and MongoDB. All measured against the service on 2026-10-24 and 2026-10-25. DeleteTable on PostgreSQL and SQLite looked the name up with a status filter that excluded DELETING rows and answered ResourceNotFoundException. On MongoDB and Cassandra it had no status filter and ran the delete again. UpdateTable on every backend answered "Table <name> is not in ACTIVE state", the sentence for the other non-ACTIVE statuses. The data plane on PostgreSQL and SQLite answered ResourceNotFoundException for the whole window, because the key-info read that authorization and the handlers share mapped a DELETING row to not found. On SQLite the DeleteTable status lookup ran outside the write transaction and on MongoDB the DELETING mark was unconditioned, so two concurrent calls both proceeded. New StorageError::TableBeingDeleted, worded once in the engine's message table and mapped to ResourceInUseException by the DeleteTable, UpdateTable and TTL error mappers. The key-info read on PostgreSQL and SQLite reads a DELETING row as found; a statement or transaction that reaches a data table the control plane has dropped is answered as not found instead of as an internal error. SQLite's DeleteTable looks the row up inside its write transaction and fences the DELETING mark on the table id; MongoDB's mark is an update_one conditioned on the table id and status, and its final delete_one is keyed on both. On PostgreSQL and SQLite the DELETING row exists only on the asynchronous DeleteTable path (control_plane_delay_seconds of 1 or more; the default is 0.25), so on a default deployment the new answers are reachable after #504, which marks the row on both paths. On Cassandra DeleteTable never marks the row, so the check is reachable only for an abandoned restore target, and the data plane there is unchanged. MongoDB drops the collections inside the DeleteTable call a few milliseconds after the mark, so its data plane keeps answering not found during that window. The differences document carries the per-backend shape. Tests: engine unit tests for the three mappers; storage-level tests on each backend for both control-plane requests and for a CREATING table keeping its answer; on PostgreSQL and SQLite, for the four data-plane requests during the window, for a put, a get and the two transaction calls after the data table is dropped under a DELETING row, and for ten rounds of two concurrent DeleteTable calls (also on MongoDB); a dual-target wire test for the PostgreSQL case that holds the data table locked from a second connection, with the data-plane rows, which skips on a server whose delay is under 1 s until #504 lands. The PostgreSQL storage-level CI step runs the new binary. The tests #395 added for that call now expect the service's refusal (12 of 12 runs on 2026-10-30). Closes #505
…ne sweep The three deleting_window tests set a fresh table's catalog row to DELETING to produce the window. A fresh table carries its activation deadline in status_transition_at, and process_control_plane_transitions removes any DELETING row whose deadline has passed: it clears the table's TTL entries in the account keyspace, drops the indexes and the data table, and deletes the row. The tests left their rows in place, so the next test in the binary to run the sweep after their account keyspaces were dropped failed on the missing keyspace. On CI that was index::test_ttl_on_a_table_with_an_async_gsi_is_refused_as_a_validation_error in the cassandra-storage job. The sequence reproduces locally with --test-threads=1 on deleting_window, groups, then that test. set_status now clears status_transition_at together with the status, the column activation clears, so the sweep's deadline filter excludes the row for the rest of the test. Each test deletes its row before it returns. The same sequence passes with the change, the full direct_integration binary passes in parallel (152 of 152), and the catalog holds no DeletingWindowTable row afterwards.
360f822 to
5b4c9ff
Compare
|
@robinnsc, rebasing this onto #395 meant picking one answer for a DeleteTable of a table that's already DELETING. I kept the refusal, since the service gave it in 12 of 12 runs this morning, and changed your three tests of that call to expect it. The details are in the description under "Rebase on #395". |
robinnsc
left a comment
robinnsc
left a comment
There was a problem hiding this comment.
Most of this works as described. With the window held (a reader on the data table on PostgreSQL, a long delay on SQLite), the second DeleteTable and every UpdateTable kind get the measured ResourceInUseException sentence, and the data plane, batch and transact calls are served on both backends. After the drop nothing answers 500 and no data table is recreated, and N concurrent DeleteTable calls had exactly one winner on all three backends. Writes made in the window never showed up in a new table of the same name, and a crash mid-delete is recovered on PostgreSQL and SQLite. Head and main came out the same for PutItem/GetItem/Query latency on PostgreSQL and SQLite. The new PostgreSQL wire tests fail on main and pass here.
The one I'd hold this for is the MongoDB stuck-DELETING case in the first inline comment. The rest are smaller.
Other notes:
- TagResource and UntagResource now answer 200 during the window on PostgreSQL and SQLite, where main refused them and, per the description, the service refuses them too. It might be worth keeping those two refused on a DELETING row rather than letting the key-info change widen them.
- On PostgreSQL, the abandoned-restore cleanup (
abort_restore,backup_engine.rs:702) also marks rows DELETING, so a partly copied restore target is now served for that window. The description gives that same reason for Cassandra answering not found. It's a small window, but it might be worth telling the two cases apart infetch_table_key_info. - None of the MongoDB
deleting_window_testsrun in CI: they're#[ignore]and the MongoDB workflow only runssnapshot_engine. deleting_window_data_tests.rs:236needs<crate>/../../targetor an absoluteCARGO_TARGET_DIR;snapshot_engine.rs:489already derives the path fromcurrent_exe().- On SQLite the lookup now runs inside
BEGIN IMMEDIATE, so therows_affected() == 0arm after the fenced mark can't be reached. - Description: the "Change" bullet about looking the name up without the status filter describes the pre-rebase mechanism; the "Rebase on #395" section and the code say otherwise.
| // below. A second DeleteTable in that window is refused the way the | ||
| // service refuses it (measured 2026-10-24), before the deletion | ||
| // protection check, as the SQL backends order it. | ||
| if desc.table_status == TableStatus::Deleting { |
There was a problem hiding this comment.
I think this needs a way to finish a delete that stopped partway. On MongoDB the mark, the drops, the tag and stream cleanup and the final delete_one all run inside the request, and nothing else removes a DELETING document. On main, the next DeleteTable re-ran those idempotent steps and finished. With this refusal, anything that stops between the mark and the delete_one leaves the table DELETING for good. That includes a client timeout, a crash, or an error from one of the drops. CreateTable of the name then answers Table already exists.
Repro: a GSI + stream table, with a transaction holding the data collection for 10 s so the drop waits, and DeleteTable with read_timeout=2. botocore's own retry of the timed-out call got Table is being deleted, and the table was still DELETING after 60 s and after a server restart. kill -9 mid-drop gave the same result. On main the retry answered 200 and the name was reusable 3 s later.
Could consider letting a DeleteTable that finds DELETING run the remaining steps and still answer ResourceInUseException, or giving the Mongo control-plane worker a pass over DELETING documents.
There was a problem hiding this comment.
Fixed in e583912. The steps after the DELETING mark now run in a task of their own that the request awaits, so a client that gives up on the call no longer cancels them. The mark also records a time 60 s ahead, and the control-plane worker finishes any delete whose document is still DELETING past that time, which covers a step that failed and a process that died. A second DeleteTable during the window still answers ResourceInUseException.
I ran your repro against a build of the old head and against this one: a GSI and stream table, a transaction holding the data collection for 10 s, and DeleteTable with read_timeout=2. On the old head botocore's retry got Table is being deleted, the table was still DELETING 75 s later, and CreateTable of the name answered Table already exists. With kill -9 mid-drop and a restart it was still DELETING 100 s later. On this build the timed-out delete finished on its own once the transaction ended, 11 s after the call. In the kill case the control plane removed the table 61 s after the mark. CreateTable of the name answered 200 both times.
a_delete_table_whose_caller_gives_up_still_finishes and the_control_plane_finishes_a_delete_left_deleting fail with the task and the worker pass taken out. The MongoDB DELETING window tests are no longer #[ignore]d and run in the MongoDB workflow's replica set job.
The request's task and the control plane can run the steps at once, and a table created into the name has the same ARN, so 34a9c9a removes the tags in one transaction with the document, and only when that run removed it. a_late_second_run_of_a_delete_leaves_a_new_table_of_the_name_whole fails without it, with the new table's tag gone.
| // plane drops them, so the data plane reads a DELETING row as found; | ||
| // a statement that reaches a dropped data table is answered as not | ||
| // found by the `DataEngine` implementation. | ||
| "ACTIVE" | "DELETING" => {} |
There was a problem hiding this comment.
With DELETING read as found here (and at storage-sqlite/src/data/ddl.rs:367), one data-plane read during the window puts the DELETING key info in the key-info cache. Nothing invalidates it when the control plane removes the table. Base-table calls then fall through to the dropped-table mapping and answer ResourceNotFoundException. A Query or Scan with IndexName, though, answers ValidationException: The table does not have the specified index for about 30 s until the soft TTL refreshes. PostgreSQL showed this at delay 1 and at the default 0.25 with a reader holding the table, and SQLite showed it at delay 1. On main a DELETING row is never cached, so it was ResourceNotFoundException from t=0. Maybe don't cache a DELETING answer, or re-check the catalog on IndexNotFound with a cached entry.
There was a problem hiding this comment.
Fixed in f2a777a, with your second suggestion. When an index name does not resolve, Query and Scan now ask the catalog for the table again, uncached. A table it no longer has is answered ResourceNotFoundException and its cache entry is dropped; a table it still has keeps the ValidationException. The extra read runs only on that error path.
I kept the DELETING answer cacheable. The key info carries no status, so not caching it would need a status field on TableKeyInfo or a describe on every load.
test_index_reads_after_the_table_is_gone_are_not_found (PostgreSQL, lock held, delay 0.05) does a GetItem in the window, waits for the table to go, then runs Query and Scan on the index. Both answer ResourceNotFoundException on this branch. With the engine change reverted the test fails at the Query, which answers ValidationException.
| delay_secs = float(delay) if delay is not None else 0.25 | ||
| except ValueError: | ||
| delay_secs = 0.25 | ||
| if delay_secs < 1.0: |
There was a problem hiding this comment.
On PostgreSQL the window already opens at the default delay. Since #395, a DeleteTable that finds an ACCESS SHARE holder takes the async branch with schedule_secs = 0.0 and leaves the row DELETING. At 0.25 with the lock held, DeleteTable answered 200 DELETING on both main and head. With this check disabled, the three tests passed on head and failed on main at 0.25, so they'd run and discriminate in CI, which pins 0.05. Could consider dropping the delay condition for PostgreSQL. The docstring and the description's "exists only on the asynchronous path / once #504 lands" would need the same correction.
There was a problem hiding this comment.
Fixed in 657e1c0. The tests no longer read the delay, and the docstring now says that a DeleteTable that finds another session holding the data table marks the row DELETING and leaves the drop to the control plane at any delay. At 0.05, the CI job's value, all five tests in the file ran and passed on this branch, none skipped. The description no longer says the window exists only on the asynchronous path.
| /// unapplied migration shows as `ResourceNotFoundException` on the wire and as | ||
| /// the named table in the server log. | ||
| fn dropped_table_as_not_found(key_info: &TableKeyInfo, e: StorageError) -> StorageError { | ||
| if super::index::is_no_such_table(&e) { |
There was a problem hiding this comment.
This matches any no such table, not just the table's own _ddb_<id> and index tables. With stream_records renamed away on a stream-enabled ACTIVE table, PutItem answered 400 ResourceNotFoundException with no log line. Main answered 500 and logged no such table: stream_records at ERROR. So an unapplied migration would show up as "not found" on a table DescribeTable reports ACTIVE. Matching by name, the way PostgreSQL's data_table_missing does, would keep the intended case and leave the rest as Internal. On PostgreSQL a missing GSI table is also IndexNotFound, while here it comes out as TableNotFound.
There was a problem hiding this comment.
Fixed in acb03bf. The funnel now matches only no such table: _ddb_ (with or without main.), the names of this backend's data and index tables, the way PostgreSQL's data_table_missing matches its own. A missing stream_shards, stream_records, gsi_pending or vector_index_holds stays an internal error and is logged. a_missing_shared_table_stays_internal fails with the old matcher.
I left a missing index table on an index read as TableNotFound on SQLite. In the DELETING window that error means the whole table is being dropped, and ResourceNotFoundException is the service's answer once the table is gone. The case where IndexNotFound is right on its own is a GSI dropped by UpdateTable under a running read. If you would rather have the two backends match, I can move SQLite to PostgreSQL's mapping here.
| ) | ||
|
|
||
| def test_delete_of_a_deleting_table_returns_its_description( | ||
| def test_delete_of_a_deleting_table_is_refused_as_being_deleted( |
There was a problem hiding this comment.
create_and_cleanup_table in conftest.py treats any ResourceInUseException at teardown as UPDATING and calls wait_for_active. With delay 1 on PostgreSQL, test_delete_existing_table now errors at teardown: ResourceInUse "Table is being deleted", then ResourceNotFound from DescribeTable. Main passes that test. It only shows at delay ≥ 1 today, but once #504 lands it will hit the default path. Might be worth having the fixture go straight to wait_for_deleted on that sentence.
There was a problem hiding this comment.
Fixed in 3a0ee7d. On Table is being deleted the fixture now goes straight to wait_for_deleted. With delay 1 on PostgreSQL, test_delete_existing_table passes with the change and errors at teardown without it, with ResourceNotFoundException from DescribeTable, as you saw.
|
|
||
| /// Two concurrent DeleteTable calls on one ACTIVE table: the first locks the | ||
| /// row `FOR UPDATE` and marks it DELETING; the second, blocked on that lock, | ||
| /// re-reads the row after the commit, finds it excluded by the status filter, |
There was a problem hiding this comment.
nit: the select now fetches DELETING rows and answers them directly (delete_table.rs:70), so the "excluded by the status filter ... from the unfiltered lookup" wording is left over from before the rebase.
There was a problem hiding this comment.
Fixed in 1de1816. The comment now says the second call reads the row again after the first commits, finds it DELETING and answers TableBeingDeleted.
A second DeleteTable during the DELETING window is refused, as the service refuses it, so on MongoDB nothing re-ran the steps of a delete that stopped between the mark and the removal of the table document. A client read timeout dropped the request future and the steps with it, and a crash or a failed drop did the same. The document then stayed DELETING and CreateTable of the name answered "Table already exists". The steps after the mark now run in a task of their own that the request awaits. Dropping the request detaches the task instead of cancelling it, so a delete stalled behind a transaction finishes once the transaction ends. The mark also writes a status_transition_at 60 s ahead, and the control-plane worker finishes every DELETING document whose time has passed, or that has none (a document an earlier build left). The worker claims a document by moving its time forward, conditioned on the value it read, so one process runs the steps at a time and a step that keeps failing is retried once a minute. The steps are idempotent: drop the data and index collections, remove the tags and the stream state, then delete only this table id's DELETING document. The DELETING window tests no longer carry #[ignore]: they skip without EXTENDDB_TEST_MONGODB_URI unless EXTENDDB_TEST_MONGODB_REQUIRED is set, and the MongoDB workflow runs them against its replica set.
…runs The second DeleteTable reads the row again after the first commits, finds it DELETING and answers TableBeingDeleted. Since the rebase on #395 there is no status filter on that select and no second lookup.
… removed The restore sweep claims an abandoned restore target by marking its row DELETING, then drops its data tables. Once the data plane read a DELETING row as found, a partly copied target was served for as long as the drop waited. The claim now sets the row's transition time to the epoch, a time no DeleteTable mark carries: DeleteTable schedules the removal at or after the time it runs. The key-info and index lookups read a DELETING row with that transition time as CREATING, so the target stays ResourceNotFoundException, as a table the service is still restoring is. The control plane still finds the row due, so a drop that fails is finished later as before.
The funnel that answers a statement reaching a dropped data table as TableNotFound matched any "no such table". With stream_records missing on a stream-enabled ACTIVE table, PutItem answered 400 ResourceNotFoundException with no log line, where main answered 500 and logged the missing table at ERROR. The funnel now matches only this backend's data and index tables, all named _ddb_<id>, as PostgreSQL's data_table_missing matches its own. A missing shared table stays an internal error.
…mark DeleteTable's status select and the DELETING mark run in one BEGIN IMMEDIATE transaction, so the mark always updates the row the select read, and the rows_affected() == 0 arm after it could not be reached.
…path The concurrent DeleteTable test read CARGO_TARGET_DIR and fell back to <crate>/../../target, so a relative CARGO_TARGET_DIR or a --target-dir put its database file in the wrong place or failed the existence check. It now takes the target directory from the test binary's own path, as the file engine in snapshot_engine does.
A DeleteTable that finds another session holding the data table marks the row DELETING and leaves the drop to the control plane, whatever control_plane_delay_seconds is, so the held lock keeps the row DELETING at the CI job's 0.05 s as well. The tests no longer skip below 1 s.
create_and_cleanup_table treated every ResourceInUseException from its DeleteTable as an UPDATING table and waited for ACTIVE. A test that had deleted its table itself got "Table is being deleted", then ResourceNotFoundException from DescribeTable inside wait_for_active, so test_delete_existing_table errored at teardown whenever the table was still DELETING. On that sentence the fixture now waits for the table to go.
On PostgreSQL and SQLite a data-plane read during the DELETING window stores the table's key info in the key-info cache, and nothing drops the entry when the control plane removes the table. A base-table read then reaches the dropped data table and answers ResourceNotFoundException. A Query or Scan with IndexName looked the index up by the cached table id instead, found no index rows, and answered "The table does not have the specified index" until the entry expired, about 30 s later. When the index name does not resolve, Query and Scan now ask the catalog for the table again, uncached. A table it no longer has is answered ResourceNotFoundException and its cache entry is dropped. A table it still has keeps the ValidationException. The extra read runs only for an index name that does not resolve.
…LETING Amazon DynamoDB answers TagResource and UntagResource on a DELETING table with ResourceInUseException "Attempt to change a resource which is still in use: Table is being deleted: <name>", and serves ListTagsOfResource (measured 2026-11-03). Before this branch ExtendDB answered the first two ResourceNotFoundException. Once the data plane read a DELETING table as found, PostgreSQL and SQLite answered them 200, since the tag path checks existence through the same key-info read. The tag handlers now read the table's status with describe_table first and refuse both calls on a DELETING table with that sentence. ListTagsOfResource keeps the existence check alone.
Two runs of one delete's steps can overlap: the DeleteTable request's own task, still waiting on a drop when the mark's due time passes, and the control plane's pass, which finishes a document still DELETING 60 s after its mark. The tag step deleted by resource_arn, and a table created into the same name has the same ARN. A run that reached the tag step after the other run had removed the document and the name had been created again deleted the new table's tags. The other steps are keyed by the table id and leave the new table alone. The last step is now one transaction that deletes this table id's DELETING document and, only when that removed it, the tags of its ARN. As in the old order, no tag outlives the document: both go in one commit. A write conflict on the transaction means another run is claiming or removing the document, so the run answers Ok(()) and leaves the document to that run, or to the control plane if it stays DELETING. Any other error is internal, as before. The doc comments of finish_table_delete and finish_due_deletes, and the message of e583912, said the claim lets one process run the steps at a time and that a second run changes nothing else. The comments now say that passes claim a document one at a time, that a DeleteTable whose own task is still running runs the same steps alongside the pass, and that the tags are the one step keyed by the ARN. a_late_second_run_of_a_delete_leaves_a_new_table_of_the_name_whole runs the steps of a removed table again after the name is created again and checks the new table's document, collections, index document, tag and stream shards. a_run_that_meets_another_removing_the_document_leaves_it_to_that_run holds another run's transaction open with the document and the tag deleted, and checks that the run answers Ok(()) and that nothing is left once the other run commits.
test_tag_changes_are_being_deleted_and_tags_are_listed checked that ListTagsOfResource answered an empty list for a table it never tagged, which also passes when the window loses the table's tags. The test now tags the table with k1/v1 before DeleteTable, asks UntagResource to remove k1 during the window, and checks that ListTagsOfResource still lists k1/v1, as the service did on 2026-11-03. PostgreSQL and SQLite keep the tag rows until the control plane removes the table.
|
@robinnsc, thanks for the review. I merged main into the branch (b060b0e) and addressed each point; the inline comments have their own replies. The hold item, MongoDB leaving a table DELETING for good, is fixed in e583912. Your timeout and kill -9 repros leave the table DELETING on the old head. On this build the timed-out delete finishes once the transaction ends, and the control plane finishes the killed one 61 s after its mark. The name is reusable afterwards in both cases. The request's task and a control-plane pass can run a delete's steps at once, so 34a9c9a removes the tags in one transaction with the document, and only when that run removed it: a table created into the name has the same ARN and keeps its tags. The notes in the review body:
Checks on the merged head: fmt and clippy ( |
Summary
While a table is
DELETING, ExtendDB answered four requests differently from Amazon DynamoDB (#505): a secondDeleteTableand anUpdateTablereported the table as absent or with the wrong sentence, and the data plane refused reads and writes the service serves. Every expected answer was measured on the service on 2026-10-24 and 2026-10-25, with the window held open by a lock on the data table, and the tag answers on 2026-11-03.Closes #505. This also fixes the
TagResourceandUntagResourcerows of #551, and itsListTagsOfResourcerow on PostgreSQL and SQLite; on MongoDBListTagsOfResourceduring the window still answersResourceNotFoundException.Change
StorageError::TableBeingDeleted, worded once in the engine's message table (Attempt to change a resource which is still in use: Table is being deleted: <name>) and mapped toResourceInUseExceptionby theDeleteTable,UpdateTableand TTL error mappers.DeleteTableon PostgreSQL and SQLite selects the row whatever its status and answers aDELETINGrow withTableBeingDeleted; MongoDB'sDeleteTablehad no status filter at all, so a second call re-ran the collection drops and answered 200; Cassandra'sDeleteTablenever marks the row, so its check covers the abandoned-restore sweep's window only.DELETINGrow as found, soPutItem,GetItem,Query,Scan(and the batch and transaction calls, which the service was not measured for) are served until the table is gone. A statement or transaction that reaches a data table the control plane has dropped (the window between the data drop and the catalog row's removal) answersResourceNotFoundExceptionrather than a 500. On SQLite only a missing_ddb_<id>table counts; a missing shared table such asstream_recordsstays an internal error.QueryorScannames an index that does not resolve, the engine asks the catalog for the table again, uncached: a table that is gone isResourceNotFoundExceptionand its entry is dropped, and an existing table keeps theValidationException.TagResourceandUntagResourceon aDELETINGtable answerResourceInUseExceptionwith the sentence above, as the service does;ListTagsOfResourceis served. The tag handlers read the status withdescribe_table, since the key info carries none.DELETINGwith the epoch as its transition time, which noDeleteTablemark carries, and the data plane reads that row as theCREATINGtarget it was: not found, as before this change.DELETINGmark run in a task of their own that the request awaits, so a client timeout does not cancel them, and the control-plane worker finishes a delete whose document is stillDELETING60 s after its mark (a step that failed, a process that died). Without this, the refusal of a secondDeleteTableleft such a tableDELETINGfor good. The task and a control-plane pass can run the steps at once, so the last step removes the document and the tags in one transaction, and the tags only when it removed this table id'sDELETINGdocument: a table created into the name has the same ARN.DeleteTablecalls get one winner: SQLite's status lookup moved inside the write transaction and theDELETINGmark is fenced on the table id; MongoDB's mark is anupdate_oneconditioned on table id and status, and its finaldelete_oneis keyed on both. PostgreSQL already held the rowFOR UPDATE; a test pins it.Per backend
DeleteTable,UpdateTableResourceInUseException, the sentence aboveResourceNotFoundException)ResourceNotFoundExceptionon a cold cache. The collections normally drop inside theDeleteTablecall milliseconds after the mark; a delete that waits on a lock or stops staysDELETINGuntil it is finished, and the table is not served meanwhileDeleteTablenever writesDELETING)On PostgreSQL the row is
DELETINGwheneverDeleteTableleaves the drop to the control plane: withcontrol_plane_delay_secondsof 1 or more, and at any delay when another session holds the data table; #504 marks it on the immediate path too. On SQLite it isDELETINGon the asynchronousDeleteTablepath (control_plane_delay_secondsof 1 or more; the default is 0.25).Tests
CREATINGtable keeping its answer; on PostgreSQL and SQLite for the four data-plane requests during the window, for a put, a get, aTransactGetItemsand aTransactWriteItemsafter the data table is dropped under aDELETINGrow, and for ten rounds of two concurrentDeleteTablecalls (also on MongoDB).DeleteTablewhose caller gives up while a transaction holds the data collection finishes once the transaction ends; the control plane finishes a document leftDELETINGwith a past transition time or none, and leaves one whose time is ahead. A second run of a delete's steps after the name is created again leaves the new table and its tag in place, and a run whose last step meets another run's uncommitted removal of the document returns success and leaves the document to that run. These tests skip withoutEXTENDDB_TEST_MONGODB_URI, fail whenEXTENDDB_TEST_MONGODB_REQUIREDis set without it, and run in the MongoDB workflow's replica set job._ddb_tables and for the shared tables.tests/python/test_deleting_window_control_plane.py(dual-target, PostgreSQL): holds the data table locked from a second connection and runs the control-plane rows,TagResource,UntagResourceandListTagsOfResourcein the window (the last lists the tag set beforeDeleteTable), the data-plane rows, andQueryandScanon an index after a read in the window and the table's removal; then confirms the window write is gone with the table. The tests run at the CI job's delay of 0.05.create_and_cleanup_tablewaits for the table to go when its teardownDeleteTableanswersTable is being deleted.42P01/no such tableerror; the SQLite and MongoDB fences removed each fail the pair test on round 0 with two winners; the MongoDB task and control-plane pass removed fail their two tests; the epoch claim reverted fails the restore-sweep test; the engine's index re-check and tag refusal reverted fail their wire tests; the old SQLite matcher fails the shared-table test; the old tag step restored fails the second-run test, with the new table's tag gone; without the write-conflict arm the meeting-run test fails with an internalWriteConflicterror..github/workflows/integration.yml: the PostgreSQL storage-level step runs every test target with--tests, sodeleting_windowruns there with no list entry; the comment above the step names the DELETING window tests..github/workflows/integration-mongodb.yml: a step that runsdeleting_window_testsagainst the replica set.Measured, not changed here
UpdateTimeToLiveduring the window: service 200; ExtendDBResourceInUseException.DescribeTimeToLive: serviceCannot describe time to live while table is in DELETING state: Current table state is DELETING; ExtendDB 200. Filed separately.DeleteTableon Cassandra both complete (the backend never writesDELETING); filed separately.UpdateItem,DeleteItem,BatchWriteItem,BatchGetItem,TransactWriteItems,TransactGetItemsandExportTableToPointInTimeduring the window are served on PostgreSQL and SQLite like the four measured requests; not measured on the service.Rebase on #401
On PostgreSQL #401 already answers
TableNotFoundfor a statement that reaches a dropped data table, as this branch did. The rebased branch keeps #401's three helpers indata/data_engine.rs(table_gone,read_target_gone,table_gone_among) and drops this branch'sdropped_table_as_not_foundandtransaction_dropped_table_as_not_found. #401's helpers stay because its transaction helper names the table whose data table the error names (this branch's named the first table in the request) and its read helper answers a Query or Scan whose index table is gone withIndexNotFound. Thedb_errorcalls this branch puts on the commits and single statements input_item.rs,delete_item.rsandupdate_item.rsstay, next to #401'sfix_up_stale_writeandfix_up_autocommitcalls.data_table_missingmatches theSQLSTATE 42P01prefix thatdb_errorkeeps, and the relation name in the message when the prefix is absent.Rebase on #395
#395 made
DeleteTableof a table that is alreadyDELETINGreturn the table's description on PostgreSQL and SQLite. The service refuses that request. On 2026-10-30 a secondDeleteTableanswered 400ResourceInUseException"Attempt to change a resource which is still in use: Table is being deleted: " in 12 of 12 runs, on empty tables and on tables of 300 items, from at once to 3 s after the first, whileDescribeTablereportedDELETING. That agrees with the 2026-10-25 runs above. The rebased branch keeps #395's select, which fetches aDELETINGrow, and answers that row withTableBeingDeleted; this branch's second lookup without the status filter is dropped, since the select now finds the row. These tests of #395 change with it:delete_table_of_a_deleting_table_returns_its_descriptionincrates/storage-postgres/tests/table_snapshot.rsbecomesdelete_table_of_a_deleting_table_is_refused_as_being_deleted,an_open_held_with_deletes_queued_completesexpects one description and eightTableBeingDeletedon the separate data database, andtest_delete_of_a_deleting_table_returns_its_descriptionintests/test_table_operations.pybecomestest_delete_of_a_deleting_table_is_refused_as_being_deleted. The troubleshooting entry for a table stuck inDELETINGsays the same.