Repository navigation
fix(server): name the supplied ARN in a denial - #509
Merged
Merged
Conversation
…ount's resource The authorization layer canonicalizes every supplied resource ARN to this deployment's partition, service, region and account so a Deny on the real resource matches however the caller spelled it. The denial message then named that canonical ARN. DynamoDB names the ARN as the caller supplied it, partition, service and region included, and answers a resource in another account with AccessDeniedException "Access is denied" for every principal (measured 2026-10-20 with TagResource, admin and no-policy callers). Each authorization target now carries both spellings: the canonical ARN the policies are matched against and the ARN as written, which the denial names. A tag operation whose ResourceArn names a table or stream in another account is refused with "Access is denied" before the IAM decision, the answer the engine already gave an authorized caller. Fixes #447
…ble ARNs Tag operations now refuse a resource ARN in another account before the shape check. The service answers "Access is denied" for another account's index ARN and backup ARN on TagResource, where the same ARN in this account gets the shape ValidationException; the branch answered the ValidationException for both. A denial names the field as written only when it parses as a DynamoDB resource ARN. An index Query or Scan whose TableName is an index ARN or a malformed resource named that string with "/index/<name>" appended, a resource the caller never sent; it now names the canonical index ARN.
LeeroyHannigan
requested review from
amrith,
c33howard,
jcshepherd,
pdf-amzn,
robinnsc and
yesyayen
as code owners
October 24, 2026 14:20
hitesh96db
reviewed
Nov 3, 2026
| && r.account != account_id | ||
| { | ||
| return Err(DynamoDbError::AccessDeniedException( | ||
| "Access is denied".to_owned(), |
There was a problem hiding this comment.
The behavior in DynamoDB is slightly different depending on if the account in the supplied ARN actually exists or not.
Account exists
(26-11-02 16:15:26) <254> [~]
7cf34dd7ebe8 % aws dynamodb tag-resource
--region us-east-1
--resource-arn arn:aws:dynamodb:us-east-1:395104918382:table/table-with-tags
--tags Key=Color,Value=Green
An error occurred (AccessDeniedException) when calling the TagResource operation: User: arn:aws:sts::124355654114:assumed-role/Admin/hts-Isengard is not authorized to perform: dynamodb:TagResource on resource: arn:aws:dynamodb:us-east-1:395104918382:table/table-with-tags because no resource-based policy allows the dynamodb:TagResource action
In case of a valid account, a resource-based policy must exist for the cross-account to work, so this is a valid use-case.
Random account that doesn't exist (999999999999)
7cf34dd7ebe8 % aws dynamodb tag-resource
--region us-east-1
--resource-arn arn:aws:dynamodb:us-east-1:999999999999:table/table-with-tags
--tags Key=Color,Value=Green
An error occurred (AccessDeniedException) when calling the TagResource operation: Access is denied.
Account simply doesn't exist ^
Until we plan to support cross-account access, we should re-name this to something more useful like "Cross-account access is not supported", and this would deviate from DDB's behavior but it seems like we'll have to do that anyways since we're not checking for account existence during AuthZ either (something to consider when/if we add cross-account support).
Collaborator
Author
There was a problem hiding this comment.
Changed in 1e0f4f2: the refusal now gives the service's answer for each operation, and it depends on whether the account exists.
I measured it on 2026-11-03 in us-east-1 with an admin (dynamodb:* on *) and a role with no policy, against an account that exists and against 999999999999. Both callers got the same answers:
- Tag operations,
DescribeTable,UpdateTable,BatchWriteItem,TransactGetItems,TransactWriteItems,DescribeStream,GetShardIterator: for an account that exists,AccessDeniedException"User: is not authorized to perform: dynamodb: on resource: because no resource-based policy allows the dynamodb: action". For 999999999999, "Access is denied". GetItem,PutItem,UpdateItem,Query,Scan,BatchGetItem: the same denial when the account exists, and "The caller is not authorized to access the specified resource." when it does not.CreateTable,DescribeTimeToLive,UpdateTimeToLive,DescribeContinuousBackups,UpdateContinuousBackups,CreateBackup,ListBackups,DescribeBackup,RestoreTableFromBackup,ExportTableToPointInTime:ValidationException"This action is only supported by accounts that match the resource owner’s account." when the account exists, "Access is denied" when it does not.RestoreTableToPointInTimegets theValidationExceptionin both cases.- A batch or transaction with tables in more than one account:
ValidationException"Invalid request! All tables in the request must be from a single account."
I kept the service's text instead of "Cross-account access is not supported". A client written against DynamoDB gets the code and message it already handles, and the denial says what is missing: a resource-based policy in the other account. ExtendDB has no resource-based policies (the differences page lists them as not implemented), so the request is always refused, before the IAM decision and with the same answer for every principal.
On account existence: the refusal looks the account up in this deployment's catalog. An account the catalog holds gets the answers for an account that exists, and any other account gets the answers for 999999999999. The differences page has a new row with these answers.
Tests: tests/test_authz_cross_account.py sends 26 of these requests (all but UpdateTable, UpdateTimeToLive and UpdateContinuousBackups, which the unit tests cover) as an admin and as a caller with no policy, against an account the catalog holds and one it does not, plus the batch and transaction cases across accounts. It passes on SQLite and PostgreSQL, and 5 of its 7 tests fail with the refusal removed. The unit tests in authz_resource/cross_account/tests.rs check the answer for each operation.
hitesh96db
reviewed
Nov 3, 2026
| pub(crate) fn new(action: &str, resource: String) -> Self { | ||
| Self { | ||
| action: action.to_owned(), | ||
| supplied: resource.clone(), |
There was a problem hiding this comment.
I'd prefer parsing and validating the supplied input before AuthZ and actually just using that as the "resource" for authz. Any reason not to just do that before AuthZ?
Collaborator
Author
There was a problem hiding this comment.
Validating before the IAM decision: the service checks some fields before it and some after. For TagResource with a table ARN in another partition, service or region (aws-cn, s3, us-west-2), an admin gets ValidationException "Invalid TableArn: Invalid ResourceArn provided as input ". A caller with no policy gets AccessDeniedException naming the ARN as sent (measured 2026-10-20 and 2026-10-21, in the PR body). So the IAM decision comes first for those fields. Checking them earlier here would give the caller with no policy a ValidationException the service does not give. The checks the service makes first already run first: the ResourceArn shape check for an index or backup ARN, and the StreamArn, BackupArn and export TableArn shape checks.
Using the supplied ARN as the resource: done for the tag operations in e47395f. The service matches their policies against the ResourceArn as written. On 2026-11-03 a role allowed dynamodb:* on the canonical table ARN only could list the tags through the canonical ARN and was denied ListTagsOfResource on the aws-cn, s3 and us-west-2 spellings, with "no identity-based policy allows". ExtendDB now matches the tag operations' policies against the ARN as written and still reads resource tags under the canonical ARN. The engine tag handler refuses the respelled ARN after the IAM decision, so an allowed caller still gets the same ValidationException as on the service.
The other ARN fields keep the canonical match, because the engine resolves them to this deployment's resource: a StreamArn, an export TableArn or a restore SourceTableArn with another partition, service or region, and a BackupArn in another region. If the policies matched the ARN as written there, a Deny on the stream would not match the same stream spelled with another region, and the engine would serve it.
A table ARN in TableName with another partition, service or region is refused by the service with a ValidationException before the IAM decision. ExtendDB authorizes it as this deployment's table and the engine then refuses it. That predates this PR, so I opened #794 for it. #451 covers the same gap for StreamArn.
Tests: test_tag_operations_match_the_arn_as_written in tests/test_authz_cross_account.py and the policy_resource() assertions in supplied_resource_tests.rs. Both fail when the tag operations match the canonical ARN.
hitesh96db
reviewed
Nov 3, 2026
| ) && let Some(arn) = input.get("ResourceArn").and_then(Value::as_str) | ||
| && !arn.is_empty() | ||
| { | ||
| authz_resource::check_resource_account(arn, account_id)?; |
There was a problem hiding this comment.
We should have this check and a good useful error message for all operations since we don't support cross-account access right now.
Collaborator
Author
There was a problem hiding this comment.
Done in 1e0f4f2. The refusal runs before the IAM decision for every operation that names a resource by ARN. It reads ResourceArn on the tag operations, StreamArn on DescribeStream and GetShardIterator, BackupArn (trimmed, as the engine trims it) on DescribeBackup, DeleteBackup and RestoreTableFromBackup, TableArn on ExportTableToPointInTime, SourceTableArn on RestoreTableToPointInTime, every table in a BatchGetItem, BatchWriteItem, TransactGetItems or TransactWriteItems request, and TableName on every other operation. A Query or Scan on an index names the index ARN, as the service does. The answers per operation are the measured ones in my reply on the account check above.
Fields the service reads as a name are not checked, because the service does not refuse them as another account's (measured 2026-11-03): ListStreams with an ARN in TableName gets "TableName must be at least 3 characters long and at most 255 characters long", a RestoreTableToPointInTime SourceTableName gets the [a-zA-Z0-9_.-]+ pattern error for the admin, and the caller with no policy is denied dynamodb:Scan on the target table, and the ImportTable table name is authorized as table/<value as written> in the caller's account. 597c1dc makes ImportTable authorize that resource, so its policies match the name the engine uses.
DeleteTable, DeleteItem and DeleteBackup were not sent to the service with another account's ARN. They take the answers of UpdateTable, UpdateItem and DescribeBackup.
Tests: tests/test_authz_cross_account.py sends 26 requests, covering the tag operations, the item operations, a Query on an index, batches and transactions, both stream operations, the backup operations, export, both restores, CreateTable, time to live and continuous backups. It runs them as an admin and as a caller with no policy, against an account the catalog holds and one it does not, and passes on SQLite and PostgreSQL. The unit tests in authz_resource/cross_account/tests.rs check which field each operation reads.
Developer mode skips authorization, and this refusal with it.
…ervice does A request that names a resource in another account by ARN was refused with "Access is denied" before the IAM decision only on the tag operations. Every other operation authorized the request against the same resource in the caller's account, so a caller with no policy got the identity-based denial and an allowed caller reached the engine. DynamoDB answers such a request before the IAM decision, and gives a caller allowed dynamodb:* on * and a caller with no policy the same answer. The answer depends on the operation and on whether the account exists. For an account that exists it is the AccessDeniedException that names the ARN as written and says no resource-based policy allows the action, except on CreateTable and the time-to-live, continuous-backup, backup, export and restore operations, which answer a ValidationException saying only the resource owner's account may call them. For an account that does not exist it is "The caller is not authorized to access the specified resource." on GetItem, PutItem, UpdateItem, Query, Scan and BatchGetItem, and "Access is denied" on the rest. A batch or transaction with tables in more than one account is a ValidationException. An account that exists gets the ARN shape checks first; one that does not is refused before them. The new authz_resource::cross_account module finds the first resource a request names in another account and builds that answer. ExtendDB has no resource-based policies, so such a request is always refused. An account exists here when this deployment's catalog holds it. Fields the service reads as a name (the ListStreams TableName, a restore SourceTableName or TargetTableName, the ImportTable table name) are not checked. check_resource_account, which covered the tag operations only, is removed. Measured 2026-11-03 in us-east-1 for an admin and a no-policy caller, with an account that exists and with 999999999999, which does not.
…rn as written TagResource, UntagResource and ListTagsOfResource matched their policies against the ResourceArn rewritten to this deployment's partition, service and region. A caller allowed only on the canonical table ARN was therefore authorized for the aws-cn, s3 and us-west-2 spellings of it, and the engine tag handler then refused them with "Invalid TableArn". DynamoDB matches the ARN as written: measured 2026-11-03 with ListTagsOfResource for a caller allowed dynamodb:* on the canonical table ARN only, each respelling is an AccessDeniedException that names the ARN as written and says no identity-based policy allows the action. AuthzTarget gains match_supplied, set on the tag operations only, and check_authorization matches the policies against policy_resource(): the ARN as written when match_supplied is set, the canonical ARN otherwise. Resource tags are still read under the canonical ARN. The engine tag handler refuses another partition, service or region of a table or stream ARN after the IAM decision, so no tag is read or written for a resource other than the one the policies matched. The other ARN fields keep the canonical match. The engine resolves a StreamArn, an export TableArn or a restore SourceTableArn with another partition, service or region, and a BackupArn in another region, to this deployment's resource, so a Deny written against that resource has to match the respelled ARN. The differences page now says that a table ARN in TableName with another partition, service or region is authorized as this deployment's table, where the service refuses it with a ValidationException before the IAM decision for every principal (measured 2026-11-03).
ImportTable's TableCreationParameters.TableName names the table to create. When the value was a table ARN, authorization parsed it and matched the policies against that table in the caller's account, while the engine takes the value as written for the new table's name. DynamoDB reads the field as a name even when it is an ARN. Measured 2026-11-03 with another account's table ARN as the name, for an admin and a no-policy caller: the denial names arn:aws:dynamodb:<region>:<caller account>:table/<value as written>. ImportTable now authorizes that resource, so the policies and the engine see the same name.
The comments on the cross-account refusal and on the wire test said ExtendDB places the ARN shape checks as the service does, citing a StreamArn in another partition. It does not for that case: its StreamArn check covers neither the partition nor the region (#451), so such a StreamArn in an account the catalog holds gets the denial, where the service answers ValidationException "Invalid StreamArn". A table ARN in TableName with another partition, service or region, or naming an index or backup, gets the cross-account answer, where the service answers a ValidationException (#794). The module and function comments, the wire test's comment and the row in docs/differences-from-dynamodb.md now say which answers match and which do not. No behaviour changes.
Collaborator
Author
|
@hitesh96db thanks for the review. All three comments are addressed; replies are inline. Merged main twice (eb40e4e and e3b0f96). The first merge conflicted in the unit tests of New commits:
Measurements, live DynamoDB in us-east-1 on 2026-11-03, with an admin and a role with no policy, against an account that exists and against 999999999999: the answer for each operation family, the order of the ARN shape checks, the batch and transaction case across accounts, the fields read as names, and the tag operations' policy match for a role allowed on the canonical table ARN only. The tables are in the PR description. Tests:
A table ARN in |
This was referenced Nov 3, 2026
…am ARN test_describe_stream_invalid_arn sends a stream ARN in account 000000000000, which no deployment's catalog holds, and accepted only ResourceNotFoundException or ValidationException. This branch refuses a resource in another account before the IAM decision, so the test failed on every backend with AccessDeniedException. The service answers DescribeStream for an account that does not exist with AccessDeniedException "Access is denied" (measured 2026-11-03), and the test now expects that answer.
hitesh96db
reviewed
Nov 3, 2026
| | Table ARN as `TableName` | Accepted; the request acts on that table | Authorized against that table, then the request fails with `ResourceNotFoundException` | | ||
| | AccessDenied message for an ARN outside this deployment (caller with no policy) | Names the ARN as supplied; an ARN in another account gets `Access is denied` | Names the ARN rewritten to this deployment's partition, service, region and account; an ARN in another account gets the full `is not authorized to perform` message. The decision is the same. | | ||
| | AccessDenied message for an ARN outside this deployment (caller with no policy) | Names the ARN as supplied. A table ARN in `TableName` with another partition, service or region never reaches the IAM decision: it is `ValidationException` ("Invalid AWS partition", "Invalid vendor code", "Invalid AWS region") for every principal (measured) | Names the ARN as supplied. A table ARN in `TableName` with another partition, service or region is authorized as this deployment's table, and a caller with no policy is denied naming the ARN as supplied | | ||
| | A resource ARN in another account | Refused with the same answer for a caller allowed `dynamodb:*` on `*` and a caller with no policy. When the account exists: `AccessDeniedException` "User: <principal> is not authorized to perform: dynamodb:<action> on resource: <ARN as supplied> because no resource-based policy allows the dynamodb:<action> action", except `CreateTable`, the time-to-live, continuous-backup, backup, export and restore operations, which answer `ValidationException` "This action is only supported by accounts that match the resource owner’s account.". When it does not: `AccessDeniedException` "The caller is not authorized to access the specified resource." on `GetItem`, `PutItem`, `UpdateItem`, `Query`, `Scan` and `BatchGetItem`, the same `ValidationException` on `RestoreTableToPointInTime`, and `AccessDeniedException` "Access is denied" on the rest. A batch or transaction with tables in more than one account is `ValidationException` "Invalid request! All tables in the request must be from a single account." (measured) | The same answers, before the IAM decision, where an account exists when this deployment's catalog holds it. `DeleteTable`, `DeleteItem` and `DeleteBackup` take the answers of `UpdateTable`, `UpdateItem` and `DescribeBackup` (not measured). A table ARN in `TableName` that also has another partition, service or region, or that names an index or backup, gets these answers too. The service answers it with a `ValidationException` (the partition, vendor or region text in the row above, or "Valid ARN format is ..." for an index or backup ARN): on `GetItem` whether or not the account exists, and on `DescribeTable` when it exists (measured; the other operations were not sent). A `StreamArn` with another partition or region, in an account this deployment's catalog holds, gets the denial above, where the service answers `ValidationException` "Invalid StreamArn" (measured) | |
There was a problem hiding this comment.
what does (measured) mean here? Does it mean it's tested? Do we need to really mention that in the docs?
Collaborator
Author
There was a problem hiding this comment.
It means that we sent that request to DynamoDB and recorded the answer. The page uses the note in about 20 rows, starting with #307 and #384, but it does not say what the note means. b301f02 adds one sentence under the introduction that says it. If you prefer to drop the note, I would do that for the whole page in a separate PR. Most of the uses are outside this change.
The page uses (measured) or (measured on <service or backend>) in about 20 rows and does not say what the note means. One sentence under the introduction now says it.
LeeroyHannigan
commented
Nov 3, 2026
| //! | Operations | Account exists | Account does not exist | | ||
| //! |---|---|---| | ||
| //! | the tag operations, `DescribeTable`, `UpdateTable`, `BatchWriteItem`, `TransactGetItems`, `TransactWriteItems`, `DescribeStream`, `GetShardIterator` | [`resource_policy_denial`] | "Access is denied" | | ||
| //! | `GetItem`, `PutItem`, `UpdateItem`, `Query`, `Scan`, `BatchGetItem` | [`resource_policy_denial`] | "The caller is not authorized to access the specified resource." | |
Collaborator
Author
There was a problem hiding this comment.
Thanks for digging into this. ExtendDB answers each operation with the message the service returns today, so a client sees the same message from ExtendDB as from DynamoDB. The table above records those answers. If the service's answer for an operation changes, we measure again and change that row, the table and the tests together. One message for every data-plane operation now would make BatchWriteItem, TransactGetItems and TransactWriteItems differ from the service.
robinnsc
added a commit
that referenced
this pull request
Nov 4, 2026
Shared traits (crates/storage). `BackupEngine` is replaced by two traits every backend implements: `BackupCatalog`, the catalog index of backups and restore jobs the engine binds (one row per backup addressed by `BackupKey` (account, backup id, ARN), one restore job record per target addressed by `RestoreTargetKey` (account, name, table id), the listings, `start_backup_delete` under the row's lock, `remove_restore_target`, `list_dead_restore_jobs`, `fail_restore_job`, `check_maintenance_claim`, `catalog_id`) with its `JobClaim` (generation, heartbeat, conditional `set_status`, `read`), and `LegacyBackupReader`, the frozen in-catalog format's create, describe, list, delete, restore-source and restore, whose `create` and `restore` now take the job's `JobPermit` and whose `abandoned_creating_backups` reports a dead CREATING row. `BackupRow` derives the API's source-table shapes from `source_table_details_json` (reading only the table id and ARN, so a later build's required member does not break a listing); `BackupStatus` maps DELETING to DELETED and hides DELETED and FAILED. The in-use, not-found and LimitExceeded texts every backend answers are one implementation in `backup_catalog` (`limit_exceeded_message(JobBound)`, the text naming the bound met: `[backup] max_concurrent_jobs`, the `[storage.sqlite] pool_size` snapshot-slot bound, PostgreSQL's connection slots), as is `JOB_HEARTBEAT_INTERVAL` (15 s), the period every backend's jobs heartbeat at. `StorageError::AlreadyExists` is the claim's refusal; `MigrationId::BackupCatalogIndex = 6` is the migration. Config. A `[backup]` section (`deny_unknown_fields`, every key with a default): `data_file_target_bytes`, `heartbeat_interval_secs`, `stale_job_after_secs`, `failed_row_retention_secs`, `reconcile_interval_secs`, `max_concurrent_jobs`, `spool_path`, with `stale_job_after_secs` at least three times the period the jobs heartbeat at (`max(heartbeat_interval_secs, JOB_HEARTBEAT_INTERVAL)`, 45 s at the least), `max_concurrent_jobs` at most 1024, and the floors checked at load; `init` writes the commented block and the sample config carries it. `ServerComponentsOptions.max_concurrent_jobs` lets the PostgreSQL factory raise both pools to `2 * jobs + 10` with a warning. Engine. `BackupRuntime` holds the job slots and the configured store (none on this head); a request over `max_concurrent_jobs` is `LimitExceededException` at once. Describe, List and Delete read the catalog index first and fall back to the legacy reader; List merges `list_account_backup_rows` with the reader's list under the existing paging, filtering and cursor rules, and `TableName` accepts a table ARN as CreateBackup does. CreateBackup runs the request order of the design (ACTIVE probe; the region and the name fit a backup ARN, each refusal naming its cause: `server.region` outside `[a-z0-9-]`, then a name over 255 bytes or made only of dots, checked before the ARN is formed so the limit is named whatever the length; vector refusal; permit last; deprecation warning; the legacy create); restore checks the target name through `describe_table` after the ARN (a table of the name `TableAlreadyExistsException`, a CREATING restore target `TableInUseException` naming its source backup), then the backup, then takes the permit last and hands it to the backend; the backend's fenced name check under the registration stays. Delete over a row runs the in-use check before the other-store check, as a restore does, and builds the description before the DELETING write; a row whose recorded source table this build cannot read is left out of ListBackups (logged) rather than failing the listing. Both spellings of a backup ARN (as written and re-spelled in this region) parse into `BackupArn` where the store-prefix rules allow; a legacy backup of a name they refuse (dot-only, over 255 bytes) skips the catalog index and describes, restores and deletes through the legacy reader as on main. Reconciler (crates/engine/src/backup/reconciler.rs). A cycle at startup and every `reconcile_interval_secs`, spawned by `serve` beside the backend workers and drained with them: checks the maintenance claim, fails each dead legacy restore (`heartbeat_at` older than `stale_job_after_secs`) and removes its CREATING target (a target the removal finds no longer CREATING is left to its owner and reported as such), removes the target of a FAILED record whose removal did not land once the record is that old, with no second FAILED write (the FAILED-target duty of design 4(e), on every backend; on Cassandra, where the removal is a claim to DELETING and then a drop, a target left DELETING with its registration is listed on the same rule and its removal resumed from the drop), logs a dead manifest restore once per record and leaves it to a later release, deletes each abandoned in-catalog CREATING backup through the reader, and records the worker metrics (`WorkerLastSuccess`, `WorkerCycleLatency`, `WorkerErrorCount` under `BackupReconciler`). It replaces the per-backend sweeps: PostgreSQL's control-plane pass, SQLite's startup sweeps, MongoDB's startup and worker sweep, Cassandra's 15 s poller. PostgreSQL. Migration 6 (`0006_backup_catalog_index.sql`, breaking, convergent, one transaction): `backup_index` (`backup_arn COLLATE "C"`, so the listings' `(created_at, backup_arn)` order is byte order whatever the database's collation), the twelve job columns on `table_restores` with the backfill of existing registrations (DONE with `completed_at = creation_date_time`, or RUNNING with the epoch heartbeat and generation 0 where the table is CREATING), `backup_maintenance`, `settings.catalog_id`. The claim is an advisory lock on a dedicated connection (`ClaimSession`) that the backup job's heartbeats and status writes travel through (the legacy restore heartbeats through it and writes DONE and FAILED on pooled connections, each conditioned on RUNNING and the generation); the reconciler's dead rule is the heartbeat alone, the lock being the claim's exclusion, as on the other backends. The legacy restore registers RUNNING with its generation, marks DONE in the transaction that flips the table ACTIVE, and FAILED before it removes a failed target. The restore slot semaphores (per account and per server) and `sweep_abandoned_restores` are removed. `RESTORE_LOCK_SEED` and `abort_restore` are `pub` and `#[doc(hidden)]` so the tests bind them. SQLite. Migration 6 as a code body (`m0006_backup_catalog_index.rs`) with the same objects; the twelve columns are nullable (ADD COLUMN on a populated table) and the application writes every one. The claim is an in-process slot (`JobSlots`, a drop guard), since one process serves a file; a restart ages every RUNNING record to the epoch so the first cycle removes an interrupted restore, and a CREATING `backups` row whose slot no task holds is reported dead whatever `stale_after` says (`backups` has no heartbeat column); the reader's `delete` removes such a row whole, with no DELETED tombstone. The restore heartbeats every `JOB_HEARTBEAT_INTERVAL` (15 s) under the engine write lock, so a beat queues behind every writer holding or waiting for that lock. Conditional writes never have an unknown outcome here. `backup_server_slot_count` stays beside `max_concurrent_jobs`; its slots are shared by backups and table snapshots (#395), the formula is stated once on `SqliteConfig::pool_size`, and the legacy create tries a slot without waiting, as a snapshot open does, and answers the one `LimitExceededException` text (`limit_exceeded_message`), naming the `[storage.sqlite] pool_size` snapshot-slot bound, for both. MongoDB. `backup_index` and `backup_maintenance` collections and a `restore_job` subdocument on a restore target's `tables` document, created by migration 6, a code body on the numbered runner (`m0006_backup_catalog_index.rs`), which also backfills the registrations and writes the catalog id; every request it sends is safe to send again (an existing collection accepted by its error code, the backfill limited to registrations without a job record, the id an upsert with `$setOnInsert`). The claim is the row, born owned by an upsert with `heartbeat_at: "$$NOW"`; the lock for DeleteBackup and a restore's registration is a session transaction that increments `lock_seq` on the backup's document, which replaces the `restore_in_progress`/`delete_in_progress` claim markers and the sweep that cleared them. Both in-use checks read at the transaction read concern (majority-committed data) while DescribeTable reads at `local`, so a RestoreTableFromBackup or DeleteBackup issued within the majority-commit lag after DescribeTable first shows the target ACTIVE can still answer BackupInUseException and succeeds when retried; the differences document's concurrent-restores row states it with the measured lag. The legacy DeleteBackup writes DELETING before the drop and DELETED after; describe and list hide DELETING. A restore request that outlives the 60 s registration bound answers InternalServerError while its task goes on; the target it leaves is stated in the differences document's restore row. The restore slot semaphores and the startup sweep are removed. The factory registers and reads the catalog's status (`register_and_check`) before it builds the engine or starts #473's startup key repair, so a server that refuses a catalog without migration 6 has written nothing; a live test pins that refusal on scratch databases holding a document the repair would rewrite. The protocol model (`backup_protocol_model.rs`) is rewritten for the new steps. Cassandra. Migration 6, a code body on the numbered runner (`m0006_backup_catalog_index.rs`): it executes `migrations/catalog/0006_backup_catalog_index.cql` as the runner executes a file body (the lease checked before each statement, schema agreement after each DDL statement), then backfills the registrations with lightweight transactions conditioned on the job status being unset and inserts the catalog id `IF NOT EXISTS`, so a run that fails part way leaves migration 6 unrecorded and the next `migrate --yes` runs the body again. The file adds the index columns (`store_kind`, `store_root`, `store_id`, `backup_id`, `owner_generation`, `heartbeat_at`, `completed_at`, `failure_reason`, `source_table_details_json`) on the reused `backups_by_arn`, the status-keyed `backup_index_by_status`, the `restore_*` job columns on `tables`, and `backup_maintenance`, every statement an `ALTER TABLE ... ADD IF NOT EXISTS` or `CREATE TABLE IF NOT EXISTS`, which the runner qualifies with the keyspace; the backfill and the id name the keyspace in their own statements, and no code or test issues `USE`. Code bodies get the lease check and the file execution on `CassandraStores` (`check`, `progress`, `run_file`, which moves there from the runner). An index row is a `backups_by_arn` row with `store_kind` set, and the two readers skip each other's rows. Claims are lightweight transactions `IF owner_generation = ?`; the legacy create heartbeats its CREATING row and polls its claim once per 64-item bucket, and `abandoned_creating_backups` fences a stale one to DELETING under a fresh `owner_generation` before reporting it, scanning DELETING rows first so a row it fenced is reported once. A job stops after three heartbeat errors in a row, as after a heartbeat that does not apply, and touches neither its row nor its data again. An orphan `backup_index_by_status` entry left by a crash between the row's write and the batch is skipped by every reader; the sweep that removes it is `backup sync`'s. `poll_abandoned_restores` and its worker are removed. The crate gains a `test-hooks` feature (as MongoDB's), compiled only for the `direct_integration` binary, which CI runs with `--features test-hooks`. Migration. One change, one number: migration 6, `backup_catalog_index`, on every backend, recorded in each store's ledger (migration 5 is #335's `drop_continuous_backups`). It is breaking (stop pre-upgrade servers first; the previous release refuses a catalog carrying it, exit 4) and convergent; a restore in flight at the stop is a RUNNING record whose owner is dead, failed and removed on the first reconciler pass. Rebased onto main after #335 and #681 merged: the migration moved from id 5 to 6, Cassandra's V006 file and MongoDB's versioned step became migration 6 code bodies on #681's runner, the `0.0.5` catalog version and this PR's own serve-time version checks went (the runner's status check refuses the catalog on every backend), and the PITR code #335 removed is gone from the legacy reader. Moved onto main at e8ca7df: #416's hand-written default for `ServerComponentsOptions` gains `max_concurrent_jobs: 0`, and the differences document's backup rows keep #509's text beside this PR's. Docs. The upgrade manual (migration 6 and its rollback, with the per-backend objects and the Cassandra and MongoDB rerun sentence, and the SQLite startup sentence: the server ages every RUNNING restore record and the first reconciler cycle fails an in-catalog restore and removes its target), the admin guide (`[backup]`, the job bound in place of the per-account and half-pool restore slots, the pooler note), the deployment guide (the job bound in place of the restore slots; the connection formulas gain `+ max_concurrent_jobs`), the high-level design and getting-started connection formulas, the limits document's concurrent-restores row, the differences document's restore rows (the MongoDB registration bound among them), the admin guide's `region` row, the differences document's `BackupArn` and backup creation rows, the limits document's `BackupName`/`TableName` row and the upgrade manual's `server.region` paragraph (CreateBackup refused under a region outside `[a-z0-9-]`; a name no backup ARN can carry refused with the texts), `CONTRIBUTING.md`'s migration example renumbered to 7, the sample config, `docs/schema/postgres.txt` (regenerated for the collation), `sqlite.txt`, `cassandra.txt` and `mongodb.txt` regenerated, and the backup/restore design's PR 6 section and section 4 (the migration number as implemented, the interface block with `list_dead_restore_jobs`, `fail_restore_job` and `abandoned_creating_backups`, revision log items 26 and 27). Tests. Counts on this tree; added and removed are test functions against main. The MongoDB and Cassandra counts are listed on this tree (`--list`); their suites ran on the top of the stack. extenddb-storage 215 (13 added; 216 beside a crate that enables its `test-hooks` feature, as `cargo test --workspace` does), extenddb-config 14 (6 added), extenddb-engine 255 (25 added: the target name before the backup and the permit, catalog describe, list, delete and restore paths, permit order, the ARN rules, reconciler cycle, duties, metrics and worker). extenddb-storage-postgres: lib 75, tests/backup_restore 37 (27 kept, 1 rewritten, 2 removed, 9 added), every other binary green (fresh 6, released 12, golden 7, crash 9, guards 7, runs 7, owners 3, serving 5, tag_blobs 1, vector_control_plane 42, doc_version_literals 3, table_snapshot 81 with its restore record written as a DONE legacy job, ...). extenddb-storage-sqlite 248 (17 added and 4 removed, among them a target removed only under its full key and migration 6's golden test; 1 ignored benchmark). extenddb-storage-mongodb: lib 165 plus 46 live tests against a replica set (among them a catalog without migration 6 refused with no repair started), migrations_fresh 8, guards 7, holders 10, locks 10, released 5 (migration 6's golden test added), interpreter_selftests 18, pushdown_parity 20; 29 added and 27 removed with the slots, the sweep and the claim markers. extenddb-storage-cassandra: lib 128 (4 added, 1 removed), migrations 21, migrations_stopped 8 (migration 6's golden test added), direct_integration 205 (13 added, 1 rewritten, 1 removed; three of the added tests build with `--features test-hooks`), ttl_integration 26 passed and main's one failure below. The released-catalog fixture of tests/test_resource_tag_migration.py strips migration 6's objects before it clears the ledger, since the check after migration 1 refuses a `table_restores` carrying the job columns. Through the binary on every backend: a catalog at main's migration 5 is refused by `serve` (exit 3), migrated by `migrate --yes`, then served, and main's binary refuses it (exit 4); a fresh install's schema equals `docs/schema/<backend>.txt`. The MongoDB workflow's jobs pass the CI way (pytest 1636 passed, 50 skipped; comprehensive 502; `tests/rust` 514; the catalog checks), and so do the Cassandra workflow's jobs (`tests/rust` 514, comprehensive 502, the catalog-lock, older-binary, two-node and authorizer checks), apart from the pytest job's parallel leg on the review host, whose 30 s request timeouts on account creation main's binary meets the same way there and whose files pass run serially. Wire fidelity (tests/test_backup_restore_fidelity.py, 36 passed and 1 skipped) ran inside those pytest jobs on MongoDB and Cassandra; the PostgreSQL and SQLite runs (64 and 63 passed) are from the tree before its rebases. `cargo fmt`, `clippy -D warnings` on 1.97, `check --all-targets` on 1.88. Not changed here. The backup store (`[backup.store]`, the store identity file) and the store-backed CreateBackup writer, worker and `backup` commands; the manifest restore job; every `backup_index` row is "of another store" for Delete and Restore until a store is configured. The Cassandra `test_ttl_enable_rejects_asynchronous_gsi` failure is main's: the test and `metadata_engine.rs` are as on main (the test matches "asynchronously propagated GSIs" and the refusal says "global secondary index"). `heartbeat_interval_secs` reaches no job yet: every in-catalog job heartbeats at `JOB_HEARTBEAT_INTERVAL`, which the stale floor is computed from. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
8 tasks done
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
Shared traits (crates/storage). `BackupEngine` is replaced by two traits every backend implements: `BackupCatalog`, the catalog index of backups and restore jobs the engine binds (one row per backup addressed by `BackupKey` (account, backup id, ARN), one restore job record per target addressed by `RestoreTargetKey` (account, name, table id), the listings, `start_backup_delete` under the row's lock, `remove_restore_target`, `list_dead_restore_jobs`, `fail_restore_job`, `check_maintenance_claim`, `catalog_id`) with its `JobClaim` (generation, heartbeat, conditional `set_status`, `read`), and `LegacyBackupReader`, the frozen in-catalog format's create, describe, list, delete, restore-source and restore, whose `create` and `restore` now take the job's `JobPermit` and whose `abandoned_creating_backups` reports a dead CREATING row. `BackupRow` derives the API's source-table shapes from `source_table_details_json` (reading only the table id and ARN, so a later build's required member does not break a listing); `BackupStatus` maps DELETING to DELETED and hides DELETED and FAILED. The in-use, not-found and LimitExceeded texts every backend answers are one implementation in `backup_catalog` (`limit_exceeded_message(JobBound)`, the text naming the bound met: `[backup] max_concurrent_jobs`, the `[storage.sqlite] pool_size` snapshot-slot bound, PostgreSQL's connection slots), as is `JOB_HEARTBEAT_INTERVAL` (15 s), the period every backend's jobs heartbeat at. `StorageError::AlreadyExists` is the claim's refusal; `MigrationId::BackupCatalogIndex = 6` is the migration. Config. A `[backup]` section (`deny_unknown_fields`, every key with a default): `data_file_target_bytes`, `heartbeat_interval_secs`, `stale_job_after_secs`, `failed_row_retention_secs`, `reconcile_interval_secs`, `max_concurrent_jobs`, `spool_path`, with `stale_job_after_secs` at least three times the period the jobs heartbeat at (`max(heartbeat_interval_secs, JOB_HEARTBEAT_INTERVAL)`, 45 s at the least), `max_concurrent_jobs` at most 1024, and the floors checked at load; `init` writes the commented block and the sample config carries it. `ServerComponentsOptions.max_concurrent_jobs` lets the PostgreSQL factory raise both pools to `2 * jobs + 10` with a warning. Engine. `BackupRuntime` holds the job slots and the configured store (none on this head); a request over `max_concurrent_jobs` is `LimitExceededException` at once. Describe, List and Delete read the catalog index first and fall back to the legacy reader; List merges `list_account_backup_rows` with the reader's list under the existing paging, filtering and cursor rules, and `TableName` accepts a table ARN as CreateBackup does. CreateBackup runs the request order of the design (ACTIVE probe; the region and the name fit a backup ARN, each refusal naming its cause: `server.region` outside `[a-z0-9-]`, then a name over 255 bytes or made only of dots, checked before the ARN is formed so the limit is named whatever the length; vector refusal; permit last; deprecation warning; the legacy create); restore checks the target name through `describe_table` after the ARN (a table of the name `TableAlreadyExistsException`, a CREATING restore target `TableInUseException` naming its source backup), then the backup, then takes the permit last and hands it to the backend; the backend's fenced name check under the registration stays. Delete over a row runs the in-use check before the other-store check, as a restore does, and builds the description before the DELETING write; a row whose recorded source table this build cannot read is left out of ListBackups (logged) rather than failing the listing. Both spellings of a backup ARN (as written and re-spelled in this region) parse into `BackupArn` where the store-prefix rules allow; a legacy backup of a name they refuse (dot-only, over 255 bytes) skips the catalog index and describes, restores and deletes through the legacy reader as on main. Reconciler (crates/engine/src/backup/reconciler.rs). A cycle at startup and every `reconcile_interval_secs`, spawned by `serve` beside the backend workers and drained with them: checks the maintenance claim, fails each dead legacy restore (`heartbeat_at` older than `stale_job_after_secs`) and removes its CREATING target (a target the removal finds no longer CREATING is left to its owner and reported as such), removes the target of a FAILED record whose removal did not land once the record is that old, with no second FAILED write (the FAILED-target duty of design 4(e), on every backend; on Cassandra, where the removal is a claim to DELETING and then a drop, a target left DELETING with its registration is listed on the same rule and its removal resumed from the drop), logs a dead manifest restore once per record and leaves it to a later release, deletes each abandoned in-catalog CREATING backup through the reader, and records the worker metrics (`WorkerLastSuccess`, `WorkerCycleLatency`, `WorkerErrorCount` under `BackupReconciler`). It replaces the per-backend sweeps: PostgreSQL's control-plane pass, SQLite's startup sweeps, MongoDB's startup and worker sweep, Cassandra's 15 s poller. PostgreSQL. Migration 6 (`0006_backup_catalog_index.sql`, breaking, convergent, one transaction): `backup_index` (`backup_arn COLLATE "C"`, so the listings' `(created_at, backup_arn)` order is byte order whatever the database's collation), the twelve job columns on `table_restores` with the backfill of existing registrations (DONE with `completed_at = creation_date_time`, or RUNNING with the epoch heartbeat and generation 0 where the table is CREATING), `backup_maintenance`, `settings.catalog_id`. The claim is an advisory lock on a dedicated connection (`ClaimSession`) that the backup job's heartbeats and status writes travel through (the legacy restore heartbeats through it and writes DONE and FAILED on pooled connections, each conditioned on RUNNING and the generation); the reconciler's dead rule is the heartbeat alone, the lock being the claim's exclusion, as on the other backends. The legacy restore registers RUNNING with its generation, marks DONE in the transaction that flips the table ACTIVE, and FAILED before it removes a failed target. The restore slot semaphores (per account and per server) and `sweep_abandoned_restores` are removed. `RESTORE_LOCK_SEED` and `abort_restore` are `pub` and `#[doc(hidden)]` so the tests bind them. SQLite. Migration 6 as a code body (`m0006_backup_catalog_index.rs`) with the same objects; the twelve columns are nullable (ADD COLUMN on a populated table) and the application writes every one. The claim is an in-process slot (`JobSlots`, a drop guard), since one process serves a file; a restart ages every RUNNING record to the epoch so the first cycle removes an interrupted restore, and a CREATING `backups` row whose slot no task holds is reported dead whatever `stale_after` says (`backups` has no heartbeat column); the reader's `delete` removes such a row whole, with no DELETED tombstone. The restore heartbeats every `JOB_HEARTBEAT_INTERVAL` (15 s) under the engine write lock, so a beat queues behind every writer holding or waiting for that lock. Conditional writes never have an unknown outcome here. `backup_server_slot_count` stays beside `max_concurrent_jobs`; its slots are shared by backups and table snapshots (#395), the formula is stated once on `SqliteConfig::pool_size`, and the legacy create tries a slot without waiting, as a snapshot open does, and answers the one `LimitExceededException` text (`limit_exceeded_message`), naming the `[storage.sqlite] pool_size` snapshot-slot bound, for both. MongoDB. `backup_index` and `backup_maintenance` collections and a `restore_job` subdocument on a restore target's `tables` document, created by migration 6, a code body on the numbered runner (`m0006_backup_catalog_index.rs`), which also backfills the registrations and writes the catalog id; every request it sends is safe to send again (an existing collection accepted by its error code, the backfill limited to registrations without a job record, the id an upsert with `$setOnInsert`). The claim is the row, born owned by an upsert with `heartbeat_at: "$$NOW"`; the lock for DeleteBackup and a restore's registration is a session transaction that increments `lock_seq` on the backup's document, which replaces the `restore_in_progress`/`delete_in_progress` claim markers and the sweep that cleared them. Both in-use checks read at the transaction read concern (majority-committed data) while DescribeTable reads at `local`, so a RestoreTableFromBackup or DeleteBackup issued within the majority-commit lag after DescribeTable first shows the target ACTIVE can still answer BackupInUseException and succeeds when retried; the differences document's concurrent-restores row states it with the measured lag. The legacy DeleteBackup writes DELETING before the drop and DELETED after; describe and list hide DELETING. A restore request that outlives the 60 s registration bound answers InternalServerError while its task goes on; the target it leaves is stated in the differences document's restore row. The restore slot semaphores and the startup sweep are removed. The factory registers and reads the catalog's status (`register_and_check`) before it builds the engine or starts #473's startup key repair, so a server that refuses a catalog without migration 6 has written nothing; a live test pins that refusal on scratch databases holding a document the repair would rewrite. The protocol model (`backup_protocol_model.rs`) is rewritten for the new steps. Cassandra. Migration 6, a code body on the numbered runner (`m0006_backup_catalog_index.rs`): it executes `migrations/catalog/0006_backup_catalog_index.cql` as the runner executes a file body (the lease checked before each statement, schema agreement after each DDL statement), then backfills the registrations with lightweight transactions conditioned on the job status being unset and inserts the catalog id `IF NOT EXISTS`, so a run that fails part way leaves migration 6 unrecorded and the next `migrate --yes` runs the body again. The file adds the index columns (`store_kind`, `store_root`, `store_id`, `backup_id`, `owner_generation`, `heartbeat_at`, `completed_at`, `failure_reason`, `source_table_details_json`) on the reused `backups_by_arn`, the status-keyed `backup_index_by_status`, the `restore_*` job columns on `tables`, and `backup_maintenance`, every statement an `ALTER TABLE ... ADD IF NOT EXISTS` or `CREATE TABLE IF NOT EXISTS`, which the runner qualifies with the keyspace; the backfill and the id name the keyspace in their own statements, and no code or test issues `USE`. Code bodies get the lease check and the file execution on `CassandraStores` (`check`, `progress`, `run_file`, which moves there from the runner). An index row is a `backups_by_arn` row with `store_kind` set, and the two readers skip each other's rows. Claims are lightweight transactions `IF owner_generation = ?`; the legacy create heartbeats its CREATING row and polls its claim once per 64-item bucket, and `abandoned_creating_backups` fences a stale one to DELETING under a fresh `owner_generation` before reporting it, scanning DELETING rows first so a row it fenced is reported once. A job stops after three heartbeat errors in a row, as after a heartbeat that does not apply, and touches neither its row nor its data again. An orphan `backup_index_by_status` entry left by a crash between the row's write and the batch is skipped by every reader; the sweep that removes it is `backup sync`'s. `poll_abandoned_restores` and its worker are removed. The crate gains a `test-hooks` feature (as MongoDB's), compiled only for the `direct_integration` binary, which CI runs with `--features test-hooks`. Migration. One change, one number: migration 6, `backup_catalog_index`, on every backend, recorded in each store's ledger (migration 5 is servers first; the previous release refuses a catalog carrying it, exit 4) and convergent; a restore in flight at the stop is a RUNNING record whose owner is dead, failed and removed on the first reconciler pass. Rebased onto main after #335 and #681 merged: the migration moved from id 5 to 6, Cassandra's V006 file and MongoDB's versioned step became migration 6 code bodies on #681's runner, the `0.0.5` catalog version and this PR's own serve-time version checks went (the runner's status check refuses the catalog on every backend), and the PITR code #335 removed is gone from the legacy reader. Moved onto main at e8ca7df: #416's hand-written default for `ServerComponentsOptions` gains `max_concurrent_jobs: 0`, and the differences document's backup rows keep #509's text beside this PR's. Docs. The upgrade manual (migration 6 and its rollback, with the per-backend objects and the Cassandra and MongoDB rerun sentence, and the SQLite startup sentence: the server ages every RUNNING restore record and the first reconciler cycle fails an in-catalog restore and removes its target), the admin guide (`[backup]`, the job bound in place of the per-account and half-pool restore slots, the pooler note), the deployment guide (the job bound in place of the restore slots; the connection formulas gain `+ max_concurrent_jobs`), the high-level design and getting-started connection formulas, the limits document's concurrent-restores row, the differences document's restore rows (the MongoDB registration bound among them), the admin guide's `region` row, the differences document's `BackupArn` and backup creation rows, the limits document's `BackupName`/`TableName` row and the upgrade manual's `server.region` paragraph (CreateBackup refused under a region outside `[a-z0-9-]`; a name no backup ARN can carry refused with the texts), `CONTRIBUTING.md`'s migration example renumbered to 7, the sample config, `docs/schema/postgres.txt` (regenerated for the collation), `sqlite.txt`, `cassandra.txt` and `mongodb.txt` regenerated, and the backup/restore design's PR 6 section and section 4 (the migration number as implemented, the interface block with `list_dead_restore_jobs`, `fail_restore_job` and `abandoned_creating_backups`, revision log items 26 and 27). Tests. Counts on this tree; added and removed are test functions against main. The MongoDB and Cassandra counts are listed on this tree (`--list`); their suites ran on the top of the stack. extenddb-storage 215 (13 added; 216 beside a crate that enables its `test-hooks` feature, as `cargo test --workspace` does), extenddb-config 14 (6 added), extenddb-engine 255 (25 added: the target name before the backup and the permit, catalog describe, list, delete and restore paths, permit order, the ARN rules, reconciler cycle, duties, metrics and worker). extenddb-storage-postgres: lib 75, tests/backup_restore 37 (27 kept, 1 rewritten, 2 removed, 9 added), every other binary green (fresh 6, released 12, golden 7, crash 9, guards 7, runs 7, owners 3, serving 5, tag_blobs 1, vector_control_plane 42, doc_version_literals 3, table_snapshot 81 with its restore record written as a DONE legacy job, ...). extenddb-storage-sqlite 248 (17 added and 4 removed, among them a target removed only under its full key and migration 6's golden test; 1 ignored benchmark). extenddb-storage-mongodb: lib 165 plus 46 live tests against a replica set (among them a catalog without migration 6 refused with no repair started), migrations_fresh 8, guards 7, holders 10, locks 10, released 5 (migration 6's golden test added), interpreter_selftests 18, pushdown_parity 20; 29 added and 27 removed with the slots, the sweep and the claim markers. extenddb-storage-cassandra: lib 128 (4 added, 1 removed), migrations 21, migrations_stopped 8 (migration 6's golden test added), direct_integration 205 (13 added, 1 rewritten, 1 removed; three of the added tests build with `--features test-hooks`), ttl_integration 26 passed and main's one failure below. The released-catalog fixture of tests/test_resource_tag_migration.py strips migration 6's objects before it clears the ledger, since the check after migration 1 refuses a `table_restores` carrying the job columns. Through the binary on every backend: a catalog at main's migration 5 is refused by `serve` (exit 3), migrated by `migrate --yes`, then served, and main's binary refuses it (exit 4); a fresh install's schema equals `docs/schema/<backend>.txt`. The MongoDB workflow's jobs pass the CI way (pytest 1636 passed, 50 skipped; comprehensive 502; `tests/rust` 514; the catalog checks), and so do the Cassandra workflow's jobs (`tests/rust` 514, comprehensive 502, the catalog-lock, older-binary, two-node and authorizer checks), apart from the pytest job's parallel leg on the review host, whose 30 s request timeouts on account creation main's binary meets the same way there and whose files pass run serially. Wire fidelity (tests/test_backup_restore_fidelity.py, 36 passed and 1 skipped) ran inside those pytest jobs on MongoDB and Cassandra; the PostgreSQL and SQLite runs (64 and 63 passed) are from the tree before its rebases. `cargo fmt`, `clippy -D warnings` on 1.97, `check --all-targets` on 1.88. Not changed here. The backup store (`[backup.store]`, the store identity file) and the store-backed CreateBackup writer, worker and `backup` commands; the manifest restore job; every `backup_index` row is "of another store" for Delete and Restore until a store is configured. The Cassandra `test_ttl_enable_rejects_asynchronous_gsi` failure is main's: the test and `metadata_engine.rs` are as on main (the test matches "asynchronously propagated GSIs" and the refusal says "global secondary index"). `heartbeat_interval_secs` reaches no job yet: every in-catalog job heartbeats at `JOB_HEARTBEAT_INTERVAL`, which the stale floor is computed from. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
Shared traits (crates/storage). `BackupEngine` is replaced by two traits every backend implements: `BackupCatalog`, the catalog index of backups and restore jobs the engine binds (one row per backup addressed by `BackupKey` (account, backup id, ARN), one restore job record per target addressed by `RestoreTargetKey` (account, name, table id), the listings, `start_backup_delete` under the row's lock, `remove_restore_target`, `list_dead_restore_jobs`, `fail_restore_job`, `check_maintenance_claim`, `catalog_id`) with its `JobClaim` (generation, heartbeat, conditional `set_status`, `read`), and `LegacyBackupReader`, the frozen in-catalog format's create, describe, list, delete, restore-source and restore, whose `create` and `restore` now take the job's `JobPermit` and whose `abandoned_creating_backups` reports a dead CREATING row. `BackupRow` derives the API's source-table shapes from `source_table_details_json` (reading only the table id and ARN, so a later build's required member does not break a listing); `BackupStatus` maps DELETING to DELETED and hides DELETED and FAILED. The in-use, not-found and LimitExceeded texts every backend answers are one implementation in `backup_catalog` (`limit_exceeded_message(JobBound)`, the text naming the bound met: `[backup] max_concurrent_jobs`, the `[storage.sqlite] pool_size` snapshot-slot bound, PostgreSQL's connection slots), as is `JOB_HEARTBEAT_INTERVAL` (15 s), the period every backend's jobs heartbeat at. `StorageError::AlreadyExists` is the claim's refusal; `MigrationId::BackupCatalogIndex = 6` is the migration. Config. A `[backup]` section (`deny_unknown_fields`, every key with a default): `data_file_target_bytes`, `heartbeat_interval_secs`, `stale_job_after_secs`, `failed_row_retention_secs`, `reconcile_interval_secs`, `max_concurrent_jobs`, `spool_path`, with `stale_job_after_secs` at least three times the period the jobs heartbeat at (`max(heartbeat_interval_secs, JOB_HEARTBEAT_INTERVAL)`, 45 s at the least), `max_concurrent_jobs` at most 1024, and the floors checked at load; `init` writes the commented block and the sample config carries it. `ServerComponentsOptions.max_concurrent_jobs` lets the PostgreSQL factory raise both pools to `2 * jobs + 10` with a warning. Engine. `BackupRuntime` holds the job slots and the configured store (none on this head); a request over `max_concurrent_jobs` is `LimitExceededException` at once. Describe, List and Delete read the catalog index first and fall back to the legacy reader; List merges `list_account_backup_rows` with the reader's list under the existing paging, filtering and cursor rules, and `TableName` accepts a table ARN as CreateBackup does. CreateBackup runs the request order of the design (ACTIVE probe; the region and the name fit a backup ARN, each refusal naming its cause: `server.region` outside `[a-z0-9-]`, then a name over 255 bytes or made only of dots, checked before the ARN is formed so the limit is named whatever the length; vector refusal; permit last; deprecation warning; the legacy create); restore checks the target name through `describe_table` after the ARN (a table of the name `TableAlreadyExistsException`, a CREATING restore target `TableInUseException` naming its source backup), then the backup, then takes the permit last and hands it to the backend; the backend's fenced name check under the registration stays. Delete over a row runs the in-use check before the other-store check, as a restore does, and builds the description before the DELETING write; a row whose recorded source table this build cannot read is left out of ListBackups (logged) rather than failing the listing. Both spellings of a backup ARN (as written and re-spelled in this region) parse into `BackupArn` where the store-prefix rules allow; a legacy backup of a name they refuse (dot-only, over 255 bytes) skips the catalog index and describes, restores and deletes through the legacy reader as on main. Reconciler (crates/engine/src/backup/reconciler.rs). A cycle at startup and every `reconcile_interval_secs`, spawned by `serve` beside the backend workers and drained with them: checks the maintenance claim, fails each dead legacy restore (`heartbeat_at` older than `stale_job_after_secs`) and removes its CREATING target (a target the removal finds no longer CREATING is left to its owner and reported as such), removes the target of a FAILED record whose removal did not land once the record is that old, with no second FAILED write (the FAILED-target duty of design 4(e), on every backend; on Cassandra, where the removal is a claim to DELETING and then a drop, a target left DELETING with its registration is listed on the same rule and its removal resumed from the drop), logs a dead manifest restore once per record and leaves it to a later release, deletes each abandoned in-catalog CREATING backup through the reader, and records the worker metrics (`WorkerLastSuccess`, `WorkerCycleLatency`, `WorkerErrorCount` under `BackupReconciler`). It replaces the per-backend sweeps: PostgreSQL's control-plane pass, SQLite's startup sweeps, MongoDB's startup and worker sweep, Cassandra's 15 s poller. PostgreSQL. Migration 6 (`0006_backup_catalog_index.sql`, breaking, convergent, one transaction): `backup_index` (`backup_arn COLLATE "C"`, so the listings' `(created_at, backup_arn)` order is byte order whatever the database's collation), the twelve job columns on `table_restores` with the backfill of existing registrations (DONE with `completed_at = creation_date_time`, or RUNNING with the epoch heartbeat and generation 0 where the table is CREATING), `backup_maintenance`, `settings.catalog_id`. The claim is an advisory lock on a dedicated connection (`ClaimSession`) that the backup job's heartbeats and status writes travel through (the legacy restore heartbeats through it and writes DONE and FAILED on pooled connections, each conditioned on RUNNING and the generation); the reconciler's dead rule is the heartbeat alone, the lock being the claim's exclusion, as on the other backends. The legacy restore registers RUNNING with its generation, marks DONE in the transaction that flips the table ACTIVE, and FAILED before it removes a failed target. The restore slot semaphores (per account and per server) and `sweep_abandoned_restores` are removed. `RESTORE_LOCK_SEED` and `abort_restore` are `pub` and `#[doc(hidden)]` so the tests bind them. SQLite. Migration 6 as a code body (`m0006_backup_catalog_index.rs`) with the same objects; the twelve columns are nullable (ADD COLUMN on a populated table) and the application writes every one. The claim is an in-process slot (`JobSlots`, a drop guard), since one process serves a file; a restart ages every RUNNING record to the epoch so the first cycle removes an interrupted restore, and a CREATING `backups` row whose slot no task holds is reported dead whatever `stale_after` says (`backups` has no heartbeat column); the reader's `delete` removes such a row whole, with no DELETED tombstone. The restore heartbeats every `JOB_HEARTBEAT_INTERVAL` (15 s) under the engine write lock, so a beat queues behind every writer holding or waiting for that lock. Conditional writes never have an unknown outcome here. `backup_server_slot_count` stays beside `max_concurrent_jobs`; its slots are shared by backups and table snapshots (#395), the formula is stated once on `SqliteConfig::pool_size`, and the legacy create tries a slot without waiting, as a snapshot open does, and answers the one `LimitExceededException` text (`limit_exceeded_message`), naming the `[storage.sqlite] pool_size` snapshot-slot bound, for both. MongoDB. `backup_index` and `backup_maintenance` collections and a `restore_job` subdocument on a restore target's `tables` document, created by migration 6, a code body on the numbered runner (`m0006_backup_catalog_index.rs`), which also backfills the registrations and writes the catalog id; every request it sends is safe to send again (an existing collection accepted by its error code, the backfill limited to registrations without a job record, the id an upsert with `$setOnInsert`). The claim is the row, born owned by an upsert with `heartbeat_at: "$$NOW"`; the lock for DeleteBackup and a restore's registration is a session transaction that increments `lock_seq` on the backup's document, which replaces the `restore_in_progress`/`delete_in_progress` claim markers and the sweep that cleared them. Both in-use checks read at the transaction read concern (majority-committed data) while DescribeTable reads at `local`, so a RestoreTableFromBackup or DeleteBackup issued within the majority-commit lag after DescribeTable first shows the target ACTIVE can still answer BackupInUseException and succeeds when retried; the differences document's concurrent-restores row states it with the measured lag. The legacy DeleteBackup writes DELETING before the drop and DELETED after; describe and list hide DELETING. A restore request that outlives the 60 s registration bound answers InternalServerError while its task goes on; the target it leaves is stated in the differences document's restore row. The restore slot semaphores and the startup sweep are removed. The factory registers and reads the catalog's status (`register_and_check`) before it builds the engine or starts #473's startup key repair, so a server that refuses a catalog without migration 6 has written nothing; a live test pins that refusal on scratch databases holding a document the repair would rewrite. The protocol model (`backup_protocol_model.rs`) is rewritten for the new steps. Cassandra. Migration 6, a code body on the numbered runner (`m0006_backup_catalog_index.rs`): it executes `migrations/catalog/0006_backup_catalog_index.cql` as the runner executes a file body (the lease checked before each statement, schema agreement after each DDL statement), then backfills the registrations with lightweight transactions conditioned on the job status being unset and inserts the catalog id `IF NOT EXISTS`, so a run that fails part way leaves migration 6 unrecorded and the next `migrate --yes` runs the body again. The file adds the index columns (`store_kind`, `store_root`, `store_id`, `backup_id`, `owner_generation`, `heartbeat_at`, `completed_at`, `failure_reason`, `source_table_details_json`) on the reused `backups_by_arn`, the status-keyed `backup_index_by_status`, the `restore_*` job columns on `tables`, and `backup_maintenance`, every statement an `ALTER TABLE ... ADD IF NOT EXISTS` or `CREATE TABLE IF NOT EXISTS`, which the runner qualifies with the keyspace; the backfill and the id name the keyspace in their own statements, and no code or test issues `USE`. Code bodies get the lease check and the file execution on `CassandraStores` (`check`, `progress`, `run_file`, which moves there from the runner). An index row is a `backups_by_arn` row with `store_kind` set, and the two readers skip each other's rows. Claims are lightweight transactions `IF owner_generation = ?`; the legacy create heartbeats its CREATING row and polls its claim once per 64-item bucket, and `abandoned_creating_backups` fences a stale one to DELETING under a fresh `owner_generation` before reporting it, scanning DELETING rows first so a row it fenced is reported once. A job stops after three heartbeat errors in a row, as after a heartbeat that does not apply, and touches neither its row nor its data again. An orphan `backup_index_by_status` entry left by a crash between the row's write and the batch is skipped by every reader; the sweep that removes it is `backup sync`'s. `poll_abandoned_restores` and its worker are removed. The crate gains a `test-hooks` feature (as MongoDB's), compiled only for the `direct_integration` binary, which CI runs with `--features test-hooks`. Migration. One change, one number: migration 6, `backup_catalog_index`, on every backend, recorded in each store's ledger (migration 5 is servers first; the previous release refuses a catalog carrying it, exit 4) and convergent; a restore in flight at the stop is a RUNNING record whose owner is dead, failed and removed on the first reconciler pass. Rebased onto main after #335 and #681 merged: the migration moved from id 5 to 6, Cassandra's V006 file and MongoDB's versioned step became migration 6 code bodies on #681's runner, the `0.0.5` catalog version and this PR's own serve-time version checks went (the runner's status check refuses the catalog on every backend), and the PITR code #335 removed is gone from the legacy reader. Moved onto main at e8ca7df: #416's hand-written default for `ServerComponentsOptions` gains `max_concurrent_jobs: 0`, and the differences document's backup rows keep #509's text beside this PR's. Docs. The upgrade manual (migration 6 and its rollback, with the per-backend objects and the Cassandra and MongoDB rerun sentence, and the SQLite startup sentence: the server ages every RUNNING restore record and the first reconciler cycle fails an in-catalog restore and removes its target), the admin guide (`[backup]`, the job bound in place of the per-account and half-pool restore slots, the pooler note), the deployment guide (the job bound in place of the restore slots; the connection formulas gain `+ max_concurrent_jobs`), the high-level design and getting-started connection formulas, the limits document's concurrent-restores row, the differences document's restore rows (the MongoDB registration bound among them), the admin guide's `region` row, the differences document's `BackupArn` and backup creation rows, the limits document's `BackupName`/`TableName` row and the upgrade manual's `server.region` paragraph (CreateBackup refused under a region outside `[a-z0-9-]`; a name no backup ARN can carry refused with the texts), `CONTRIBUTING.md`'s migration example renumbered to 7, the sample config, `docs/schema/postgres.txt` (regenerated for the collation), `sqlite.txt`, `cassandra.txt` and `mongodb.txt` regenerated, and the backup/restore design's PR 6 section and section 4 (the migration number as implemented, the interface block with `list_dead_restore_jobs`, `fail_restore_job` and `abandoned_creating_backups`, revision log items 26 and 27). Tests. Counts on this tree; added and removed are test functions against main. The MongoDB and Cassandra counts are listed on this tree (`--list`); their suites ran on the top of the stack. extenddb-storage 215 (13 added; 216 beside a crate that enables its `test-hooks` feature, as `cargo test --workspace` does), extenddb-config 14 (6 added), extenddb-engine 255 (25 added: the target name before the backup and the permit, catalog describe, list, delete and restore paths, permit order, the ARN rules, reconciler cycle, duties, metrics and worker). extenddb-storage-postgres: lib 75, tests/backup_restore 37 (27 kept, 1 rewritten, 2 removed, 9 added), every other binary green (fresh 6, released 12, golden 7, crash 9, guards 7, runs 7, owners 3, serving 5, tag_blobs 1, vector_control_plane 42, doc_version_literals 3, table_snapshot 81 with its restore record written as a DONE legacy job, ...). extenddb-storage-sqlite 248 (17 added and 4 removed, among them a target removed only under its full key and migration 6's golden test; 1 ignored benchmark). extenddb-storage-mongodb: lib 165 plus 46 live tests against a replica set (among them a catalog without migration 6 refused with no repair started), migrations_fresh 8, guards 7, holders 10, locks 10, released 5 (migration 6's golden test added), interpreter_selftests 18, pushdown_parity 20; 29 added and 27 removed with the slots, the sweep and the claim markers. extenddb-storage-cassandra: lib 128 (4 added, 1 removed), migrations 21, migrations_stopped 8 (migration 6's golden test added), direct_integration 205 (13 added, 1 rewritten, 1 removed; three of the added tests build with `--features test-hooks`), ttl_integration 26 passed and main's one failure below. The released-catalog fixture of tests/test_resource_tag_migration.py strips migration 6's objects before it clears the ledger, since the check after migration 1 refuses a `table_restores` carrying the job columns. Through the binary on every backend: a catalog at main's migration 5 is refused by `serve` (exit 3), migrated by `migrate --yes`, then served, and main's binary refuses it (exit 4); a fresh install's schema equals `docs/schema/<backend>.txt`. The MongoDB workflow's jobs pass the CI way (pytest 1636 passed, 50 skipped; comprehensive 502; `tests/rust` 514; the catalog checks), and so do the Cassandra workflow's jobs (`tests/rust` 514, comprehensive 502, the catalog-lock, older-binary, two-node and authorizer checks), apart from the pytest job's parallel leg on the review host, whose 30 s request timeouts on account creation main's binary meets the same way there and whose files pass run serially. Wire fidelity (tests/test_backup_restore_fidelity.py, 36 passed and 1 skipped) ran inside those pytest jobs on MongoDB and Cassandra; the PostgreSQL and SQLite runs (64 and 63 passed) are from the tree before its rebases. `cargo fmt`, `clippy -D warnings` on 1.97, `check --all-targets` on 1.88. Not changed here. The backup store (`[backup.store]`, the store identity file) and the store-backed CreateBackup writer, worker and `backup` commands; the manifest restore job; every `backup_index` row is "of another store" for Delete and Restore until a store is configured. The Cassandra `test_ttl_enable_rejects_asynchronous_gsi` failure is main's: the test and `metadata_engine.rs` are as on main (the test matches "asynchronously propagated GSIs" and the refusal says "global secondary index"). `heartbeat_interval_secs` reaches no job yet: every in-catalog job heartbeats at `JOB_HEARTBEAT_INTERVAL`, which the stale floor is computed from. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A caller with no policy who names a resource by an ARN in another partition, service or region gets an
AccessDeniedExceptionwhose resource is the ARN rewritten to this deployment. DynamoDB names the ARN as the caller wrote it. This change keeps the caller's spelling for the message.A request that names a resource in another account by ARN is now refused before the IAM decision with the answer DynamoDB gives, which depends on the operation and on whether the account exists. Before, only the tag operations refused it, always with
Access is denied, and every other operation authorized the same resource in the caller's account.The tag operations now match their policies against the
ResourceArnas written, as DynamoDB does, andImportTableauthorizes its table name as written.Fixes #447
What changed
crates/server/src/authz_resource.rs:authz_targetsreturnsAuthzTarget { action, resource, supplied, match_supplied }.resourceis the canonical ARN for this deployment. Resource tags are read under it, and the policies are matched against it except on the tag operations.suppliedis what a denial names. It is the request field as written when the field parses as a DynamoDB resource ARN (ResourceArn,StreamArn, the exportTableArn,SourceTableArn, a table ARN inTableName, aRequestItemskey or aTransactItemstable; aBackupArnis trimmed first, as the engine trims it). Otherwise it is the canonical ARN: for a bare name, and for an index Query or Scan whoseTableNameis not a table ARN.match_suppliedis set onTagResource,UntagResourceandListTagsOfResource, whose policies are matched againstsupplied(policy_resource()). The engine tag handler refuses another partition, service or region after the IAM decision, so no tag is read or written for a resource other than the one the policies matched. The other ARN fields keep the canonical match because the engine resolves them to this deployment's resource, and a Deny on that resource has to match the respelled ARN.ImportTableauthorizestable/<TableCreationParameters.TableName as written>in the caller's account.crates/server/src/authz_resource/cross_account.rs(new):refusefinds the first resource a request names by ARN in another account and returns the service's answer for the operation. An account exists when this deployment's catalog holds it. An account that exists gets the ARN shape checks first; one that does not is refused before them.check_resource_account, which covered the tag operations only, is removed. ExtendDB's shape checks do not yet cover aStreamArn's partition or region (DescribeStream and GetShardIterator accept a StreamArn with another partition, region or service spelling #451) or a table ARN inTableName(A table ARN in TableName with another partition, service or region is not refused as the service refuses it #794); see Not changed here.crates/server/src/authorization.rs:check_authorizationtakes anAuthzTarget; the policy evaluation usestarget.policy_resource(), the resource tag lookup usestarget.resource, anddeny_messagenamestarget.supplied.crates/server/src/request_helpers.rs:authorize_requestcallscross_account::refusefor every operation before the IAM decision, then the existingResourceArnshape check; everycheck_authorizationcall site builds a target.docs/differences-from-dynamodb.md: a new row for a resource ARN in another account, and the denial message, backupResourceArn,BackupArnandCreateBackuprows updated.Service measurements
Live DynamoDB, scratch account
<scratch-acct>, us-east-1, tablebb-arn-main. The TagResource rows were measured three times on 2026-10-20 and 2026-10-21, with identical answers.arn:aws-cn:dynamodb:us-east-1:<acct>:table/bb-arn-mainInvalid TableArn: Invalid ResourceArn provided as input <arn>User: <principal> is not authorized to perform: dynamodb:TagResource on resource: arn:aws-cn:dynamodb:us-east-1:<acct>:table/bb-arn-main because no identity-based policy allows the dynamodb:TagResource actionarn:aws:s3:us-east-1:<acct>:table/bb-arn-mainarn:aws:s3:us-east-1:<acct>:table/bb-arn-mainarn:aws:dynamodb:us-west-2:<acct>:table/bb-arn-mainarn:aws:dynamodb:us-west-2:<acct>:table/bb-arn-mainarn:aws:dynamodb:us-east-1:111111111111:table/bb-arn-mainAccess is deniedAccess is deniedarn:aws:dynamodb:us-east-1:<acct>:table/bb-rv-arn1-nopeMeasured 2026-10-24, admin only:
Access is deniedin every caseAccess is denied; the same ARNs in this account get the shape ValidationExceptionAccess is deniedGetItem with
TableNameset to the canonical table ARN, no-policy caller (2026-10-20): denial namingarn:aws:dynamodb:us-east-1:<acct>:table/bb-arn-main.Measured 2026-11-03 with an admin and a role with no policy, naming a table ARN in an account that exists and in
999999999999, which does not. Both callers got the same answers, and the other-account rows above match the answers for an account that does not exist.User: <principal> is not authorized to perform: dynamodb:<action> on resource: <ARN as sent> because no resource-based policy allows the dynamodb:<action> actionAccess is deniedThe caller is not authorized to access the specified resource.This action is only supported by accounts that match the resource owner’s account.Access is deniedInvalid request! All tables in the request must be from a single account.Also measured 2026-11-03:
aws iam simulate-custom-policyon 2026-10-24, for a user and an assumed-role caller, gave implicitDeny for an Allow on the canonical table ARN and no match for a Deny on it, with theaws-cn,s3,us-west-2and other-account spellings.ResourceArn, and aStreamArnin another partition or region, get the shape ValidationException when the account exists andAccess is deniedwhen it does not.<table ARN as sent>/index/<index>. A TransactWriteItems member names its own action (UpdateItem,ConditionCheckItem).ListStreamsTableName,RestoreTableToPointInTimeSourceTableNameand theImportTabletable name are read as names:ListStreamsgets its length ValidationException for both callers,SourceTableNamegets its pattern ValidationException for the admin (the caller with no policy is denieddynamodb:Scanon the target table), andImportTableis denied onarn:aws:dynamodb:us-east-1:<caller account>:table/<value as written>for both callers.dynamodb:*on the canonical table ARN only lists the tags through the canonical ARN and is deniedListTagsOfResourceon theaws-cn,s3andus-west-2spellings, withno identity-based policy allows.DescribeTableandGetItemwith those three spellings inTableNameget ValidationException (Invalid AWS partition,Invalid vendor code,Invalid AWS region) for the admin and for the role allowed on the canonical ARN.DeleteTable,DeleteItemandDeleteBackupwere not sent with another account's ARN; they take the answers ofUpdateTable,UpdateItemandDescribeBackup.Tests
tests/test_authz_resource_arn.py:test_tag_denial_names_supplied_arn_noperm: the denial for the aws-cn, s3, us-west-2 and nonexistent ARNs carrieson resource: <arn as sent> because no identity-based policy allows. Fails on main (the message carries the rewritten ARN). The nonexistent row passes on main too, because that ARN is already canonical.test_tag_cross_account_is_access_denied_for_every_principal: a no-policy caller and an admin both getAccess is deniedfor a table ARN in an account the deployment does not hold. Fails on main for the no-policy caller.test_tag_cross_account_index_and_backup_arns_are_access_denied: an admin getsAccess is deniedfor the index and backup ARNs of an account the deployment does not hold. Fails on main.test_get_item_table_arn_in_tablename_denial_names_table_arn: pins the measured GetItem message. Passes on main as well (the canonical ARN is both spellings).tests/test_authz_cross_account.py:test_other_account_answer_depends_on_operation_and_account, as an admin and as a no-policy caller: 26 requests (the tag operations, the item operations, Query on an index, BatchGetItem, BatchWriteItem, both transactions, both stream operations, the backup operations, export, both restores, CreateTable, DescribeTimeToLive, DescribeContinuousBackups) against an account the catalog holds and one it does not.test_tables_in_more_than_one_account_are_refused, both callers.test_shape_checks_run_before_a_known_accounts_denial.test_tag_operations_match_the_arn_as_written.test_import_table_name_is_read_as_a_name.With the refusal removed, 5 of the 7 fail. With the tag operations matched against the canonical ARN, the shape checks run for an account the catalog does not hold and
ImportTableauthorized on the parsed ARN, the three matching tests fail and the other four pass.Unit tests:
cross_account::tests(8 tests),supplied_resource_tests(5 tests) andauthorization::resource_tag_tests::denial_names_the_supplied_resource. The server unit tests pass (266).Backends: the change is in the server's authorization layer; the account lookup uses each backend's catalog store. Both wire files pass on SQLite and PostgreSQL (54 tests each). On SQLite the other wire files that name other accounts also pass:
test_cross_account_isolation.py,test_auth_integration.py,test_auth_permissions.py,test_backup_restore_fidelity.py,test_idempotency_account_scope.py,test_import_export.py,test_authz_arn_conditions.pyandtest_auth_error_fidelity.py(225 tests). MongoDB and Cassandra were not run.Interactions with open PRs
fix/stream-arn-account-checkconflicts with this branch incrates/server/src/request_helpers.rs(the checks before the IAM decision) and in seven places incrates/server/src/authz_resource.rs. The resolution keepscross_account::refuseahead of fix(server): refuse stream ARNs from another account or region #466's checks, since it answers another account's stream ARN as the service does, and keeps fix(server): refuse stream ARNs from another account or region #466's region check for this account's stream ARNs. fix(server): refuse stream ARNs from another account or region #466 readst[0].1fromauthz_targetsinauthz_resource.rsandauthz_resource/stream_scope.rs; whichever PR merges second changes those tot[0].resource.clone(). The merged tree was not built.fix/table-arn-as-nameresolves a table ARN inTableNameand treats the ARN's account as the caller's. With this change another account's ARN is refused before it reaches that resolution. A trial merge with this branch conflicts incrates/server/src/request_helpers.rs.Not changed here
TableNamewith another partition, service or region is authorized as this deployment's table, and an allowed caller then getsResourceNotFoundException(item operations) or the table-name pattern ValidationException (DescribeTable). The service refuses it with a ValidationException before the IAM decision. That predates this PR and is filed as A table ARN in TableName with another partition, service or region is not refused as the service refuses it #794; DescribeStream and GetShardIterator accept a StreamArn with another partition, region or service spelling #451 covers the same gap forStreamArn.StreamArnwith another partition or region, in an account the catalog holds, gets the resource-based-policy denial; the service answersValidationException"Invalid StreamArn". ATableNameholding another account's table ARN with another partition, service or region, or another account's index or backup ARN, gets the cross-account answer; the service answers aValidationExceptiononGetItemfor either account and onDescribeTablewhen the account exists. ExtendDB does not check those fields' shape before the IAM decision in the caller's account either (DescribeStream and GetShardIterator accept a StreamArn with another partition, region or service spelling #451, A table ARN in TableName with another partition, service or region is not refused as the service refuses it #794).TableNameon the item operations is not resolved to the table (A table ARN passed as TableName returns ResourceNotFoundException #442).GetItemwith the canonical ARN; they go through the sameAuthzTargetpath.