[LeeroyHannigan]

What

This pull request adds the four bulk operations to ExtendDB: StartBulkPutItems, DescribeBulkOperation, ListBulkOperations and CancelBulkOperation. A bulk put reads newline-delimited DynamoDB JSON from the objects under an input bucket and prefix. It writes each item into an existing table with PutItem semantics, in the background, and reports the rows that fail under an output bucket and prefix.

The server configuration selects one of two stores for those buckets with [bulk] store. The S3 store reads and writes buckets in AWS S3 or in an S3-compatible service, with the server's own AWS credentials. The filesystem store maps each bucket to a directory in the caller's account subtree beneath one root on the server.

The SQLite and PostgreSQL backends run bulk operations. On the MongoDB and Cassandra backends the four operations answer ValidationException.

The branch is six commits on main at 1a68484c4, one for each concern, and a merge of main at d35d8601a.

Why

A user loads many items into a table that already serves traffic. They write the items to a bucket as newline-delimited DynamoDB JSON and start a bulk put. Then they poll DescribeBulkOperation until the operation ends, and read the error report for the rows that failed. Code written for these operations also handles ResourceInUseException when another operation runs on the table.

On main the four operations answer UnknownOperationException, so that code cannot run against ExtendDB. With this change, that code gets DynamoDB's request and response shapes, statuses, error texts and report layout. The list under Differences from DynamoDB gives the exceptions.

Commits

1. feat(core): add the bulk operation types, ARNs and limits adds the request and response types of the four operations, the bulk [limits] keys and the bulk metric names. The server and the engine share its ARN parsing.
2. feat(s3,config): add the S3 client crate and the [bulk] configuration adds crates/s3 (extenddb-s3), which opens one store's S3 clients with the standard AWS credential chain, and the [bulk] section that selects the store.
3. feat(storage): add the bulk operation store and the batched put adds migration 8 (bulk_operations) on SQLite and PostgreSQL, the BulkOperationStore trait and its two SQL implementations, and the batched put with its row outcome ledger. The put path that PutItem and the bulk runner share moves into one module in this commit.
4. feat(engine,server): add StartBulkPutItems and the bulk operation APIs adds the four handlers, the runner and its two workers, and the filesystem and S3 stores. The authorization of the operations and of their S3 actions is in this commit, with the server code that starts and stops the bulk workers. The engine and the server are one commit because the server builds the engine's OperationContext, which gains the bulk fields.
5. test: add the bulk put suite and the S3 bulk store job adds the Python suite in tests/bulk_put, the --bulk-store option of devtools/run-tests, and a CI job against an S3-compatible service.
6. docs: document bulk put items adds a usage guide chapter, admin guide sections, security model entries, a differences page section, troubleshooting entries and an upgrade manual section.

cargo check --workspace --all-targets passes on each commit with Rust 1.97.0. The S3 and configuration commit carries the whole Cargo.lock. On that commit and on the storage commit, cargo prunes the lock entries that only later manifests use, so --locked fails on those two. The storage commit has three dead-code warnings for items that the engine and server commit uses, so cargo clippy -- -D warnings fails on it alone.

The merge of main at d35d8601a follows the six commits. It resolves the conflicts with #672, #466, #875, #847 and #700, and its message lists each resolution. Two commits follow the merge. One sets EXTENDDB_TEST_BACKUP_STORE in the S3 bulk suite job, which devtools/run-tests requires since #700. The other lists each parse-level cause in the usage guide with its report message.

feat(core): add the bulk operation types, ARNs and limits

What it adds

crates/core now has the request and response types of StartBulkPutItems, DescribeBulkOperation, ListBulkOperations and CancelBulkOperation, with the member names of the API model. An enum value outside its set fails deserialization with a ValidationException, as in the existing types.

The arn module parses a bulk operation ARN, and resolves a TableName written as a table ARN to the bare table name. The server's authorization layer and the engine's handlers call the same functions, so the server evaluates a policy against the resource that the handler serves.

LimitsConfig has 17 new bulk keys, each with a serde default. The new MetricName variants are BulkOperationsStarted, BulkOperationsFinished and BulkPutItemsWritten. The new QuerySource variants, BulkClaim and BulkRetention, name the two bulk workers. No request reaches this code before the engine and server commit.

Decisions to review

• The ARN checks run in the order that the service applies them (measured). The shape comes first, then the account and the region, then the partition, the service and the operation id. An ARN of another account gets the answer for another account even when it carries a malformed id. As a result, a caller learns nothing about the ids of another account.
• A TableName that is a well-formed table ARN of another account answers AccessDeniedException, Access is denied, before the server reads any policy. A table ARN of the caller's account in another region answers ValidationException with 1 validation error detected: Invalid AWS region in '{arn}'. Both are the texts that the service answers (measured).
• Two limit defaults follow the service's quotas: 50,000 input objects for each operation and 5 GiB for each object. The other defaults are ExtendDB's own. An account may have 50 non-terminal operations, and an operation may read 10 GiB of input. A ClientToken may be up to 1,024 bytes long, and a terminal record stays for 30 days. A process runs 4 operations at once, at most 2 for one account, under a 60-second lease. A runner checkpoints every 25 rows and writes up to 25 rows in each storage transaction.
• bulk_put_write_units_per_second, the ceiling on bulk writes in one process, defaults to 80,000 write units a second. The default equals per_account_max_wcu, the per-account write ceiling of the request path.

Differences from DynamoDB

• The operation limits are [limits] keys, where DynamoDB has service quotas.
• bulk_operation_retention_days sets how long a terminal operation stays describable.

feat(s3,config): add the S3 client crate and the [bulk] configuration

What it adds

crates/s3 (extenddb-s3) opens one store's S3 clients from three settings: region, endpoint_url and force_path_style. The clients sign with the standard AWS credential chain. The crate resolves the chain once at startup and logs the outcome under the store's name. Temporary keys that the SDK cannot renew get a warning. Without an endpoint_url, the crate also sends one STS GetCallerIdentity request at startup. The request runs in a task of its own with a 5-second bound, and the crate logs the identity that the credentials sign as.

The clients follow the region redirects that AWS S3 sends. They send a call that fails with a transient fault again after 1 second. The wait doubles up to 30 seconds, and no new try starts later than 120 seconds after the first one.

crates/config reads [bulk]. store is filesystem, the default, or s3. [bulk.filesystem] path is the root of the filesystem store. [bulk.s3] has region (default [server] region), endpoint_url, force_path_style and allow_insecure_endpoint. The console settings page lists the bulk store settings and the bulk limits.

The workspace manifest adds the crate and enables credentials-login on aws-config, so the credential chain resolves a profile that aws login wrote. It names aws-sdk-sts 1.97.0 and aws-credential-types 1.2.11 directly, at the versions already in Cargo.lock. For the engine's row parser, it enables the unbounded_depth feature of serde_json. The minimum Rust version stays at 1.88, and aws-config 1.8.13 and aws-sdk-s3 1.122.0 stay at the versions that main pins. Cargo.lock gains 27 packages and changes the version of none, and the commit updates the four license notice files.

Decisions to review

• Credentials come only from the standard AWS credential chain. [bulk.s3] refuses the six credential key names, such as access_key_id and aws_secret_access_key, with a message that names the chain. The configuration file carries no secret.
• A plain http endpoint_url whose host is not a loopback form (127.x.y.z, ::1 or localhost) refuses startup unless allow_insecure_endpoint = true. With that key set, the server accepts the endpoint and logs a warning. The backup store applies this rule, with these loopback forms.
• The clients send their requests to AWS S3 or to endpoint_url alone. They ignore an endpoint from the environment or from an AWS profile, such as an AWS_ENDPOINT_URL that points DynamoDB clients at ExtendDB.
• The identity check never delays startup. It logs one line: the identity, a refusal by STS, a server clock more than 15 minutes off, or a check that could not complete. It runs only against AWS S3, because an S3-compatible service has no STS.

Differences from DynamoDB

This commit dispatches nothing. The first entry under Differences from DynamoDB comes from the store that [bulk] store selects. DynamoDB reads and writes S3 as the caller, and the S3 store signs with the server's credentials.

feat(storage): add the bulk operation store and the batched put

What it adds

Migration 8, bulk_operations, adds five tables through the migration ledger. On PostgreSQL the catalog database gets bulk_operations, bulk_operation_input_files and bulk_operation_report_records. The data database gets bulk_row_outcomes and bulk_write_owners, beside the items that they record. SQLite gets all five in its one database, and MongoDB and Cassandra record the migration as not applicable. Every statement is CREATE ... IF NOT EXISTS, so a second run changes nothing. docs/schema/postgres.txt and docs/schema/sqlite.txt gain the new objects and lose none. #740 also adds a migration 8, restore_progress. Of the two pull requests, the one that merges second takes 9.

crates/storage adds the BulkOperationStore trait, which StorageEngine now requires. Its methods are the storage steps of an operation: start, read, list, cancel, claim, checkpoint, read report records in pages, finish and purge. The backends implement only these steps. The engine implements the lifecycle around them, with the claim policy, batching, resume, rendering and the terminal transitions. A contract suite behind the test-util feature runs the same cases on SQLite and PostgreSQL.

DataEngine gains put_items_batch, a required method. It applies a batch of rows in one transaction. In that transaction it records each marked row's outcome in bulk_row_outcomes, with the size of the item that the row replaced. DataEngine::put_item takes a new marker: Option<&BulkRowMarker> argument, and every existing caller passes None. MongoDB and Cassandra refuse the bulk methods with ValidationException, Bulk operations are not supported on this storage backend.

The put path moves to crates/engine/src/put_item/payload.rs. handle_put_item and the bulk runner both call its resolve_put_condition and validate_put_payload, and handle_put_item writes through its apply_put. validate_put_item in crates/core has no caller left, and the commit deletes it. The engine's Cargo.toml is in this commit because the put helper's tests need the SQLite engine as a dev-dependency. The manifest adds the dependencies that the engine and server commit uses: ruzstd, md-5, flate2, rustix on unix, extenddb-s3 and aws-sdk-s3.

Decisions to review

• A run applies each row once. A batch records every row that it applies, plain or conditional, in bulk_row_outcomes in its own data transaction. A resumed run reads the rows recorded past its checkpoint and accounts for them from the ledger. It does not write them again, so a client write that came after the row keeps its value. The ledger is in the data database, beside the items, because the catalog and the data tables can be separate databases on PostgreSQL. No transaction then covers both.
• Each claim records itself in bulk_write_owners before its first batch. A batch whose marker names another owner writes nothing and answers StorageError::Transient. A runner that lost its lease thus cannot commit a late batch.
• On PostgreSQL and SQLite, put_item and put_items_batch call one in-transaction row helper. For that reason this commit restructures the single-item put path of both backends.
• A unique partial index on (account_id, table_id) over IN_PROGRESS and CANCELLING allows one non-terminal operation for each table. The start transaction answers a second start at once.
• The token index stores the SHA-256 hash of each ClientToken, for each account, so the token itself never enters an index. A token replays its operation for eight hours from the operation's start.
• bulk_operations has no foreign key to tables. A run ends FAILED when DeleteTable deletes its table, and the operation stays describable until retention purges it.
• bulk_operation_report_records.record is TEXT on PostgreSQL, where jsonb would refuse a DynamoDB string that contains U+0000.
• On PostgreSQL the start transaction waits at most 200 ms at a time for the account's advisory lock. It takes the table row with FOR SHARE NOWAIT. The server never accepts both a Start and an UpdateTable that creates an index on the same table. Leases use the database clock (clock_timestamp()), and bulk store transactions run under a 10-second idle_in_transaction_session_timeout. The backend retries a batch that PostgreSQL aborts as a deadlock victim, after a jittered wait.
• On SQLite the start and the index creation read each other's rows inside BEGIN IMMEDIATE transactions, so the database write lock orders them.
• PutItem changes order. resolve_put_condition runs the nesting-depth check of the expression values before the table lookup, as DeleteItem does. A request with an over-deep expression value now gets that ValidationException when its table is missing, where main answers ResourceNotFoundException. When its item is also invalid, it gets the nesting-depth error. On main that request gets the item's error.

Differences from DynamoDB

• ExtendDB always refuses a second start on a table with a running operation at once, with ResourceInUseException. DynamoDB can also accept it and fail it later.
• The MongoDB and Cassandra backends refuse the four operations.

feat(engine,server): add StartBulkPutItems and the bulk operation APIs

What it adds

The engine (crates/engine/src/bulk_put) dispatches the four operations.

• StartBulkPutItems checks a request in seven steps. The steps are body deserialization, field constraints, feature support, bulk store, table lookup, input listing and start transaction. A request that is wrong at two steps gets the earlier step's error. Start lists every input object before it answers, so the response carries TotalInputBytes.
• A retry with the same ClientToken and the same parameters returns the existing operation. A retry with other parameters gets IdempotentParameterMismatchException.
• DescribeBulkOperation and CancelBulkOperation check the operation ARN in layers and answer an ARN of another account or region without a storage read. ListBulkOperations pages newest first, 10 operations to a page by default and at most 25.
• A claim loop in each server claims runnable operations every second. It runs up to bulk_operation_worker_concurrency operations in the process and bulk_operation_worker_per_account for one account. A retention worker purges terminal operations older than bulk_operation_retention_days, at startup and then daily, 1,000 operations at a time.
• The runner reads the input objects one at a time, in the byte order of their keys. It checks each row with the validation and condition handling of PutItem, through the shared put path. It writes up to bulk_put_write_batch_rows rows in each storage transaction. After every bulk_put_batch_size rows and at the end of each file, it checkpoints its cursor, counters, usage and report records. Each checkpoint renews the runner's lease.
• A row with an item fault fails alone, with the PutItem text in its report record, and the load continues. A line that the parser cannot read abandons the rest of its file.
• The runner writes the report records to gzip objects once, when the operation ends, with the manifest. Each requested kind gets its own sequence of objects, rolled at 200 MiB of compressed output.
• Two token buckets pace the writes, in 1 KB write units: one for the process, from bulk_put_write_units_per_second, and one for each operation, from its MaxWriteRequestUnits.
• UpdateTable refuses a new global secondary index while an operation on the table is IN_PROGRESS or CANCELLING, with the service's text. Start refuses a table with an index that is not ACTIVE.

The two stores:

• The filesystem store reads and writes {root}/{account id}/{S3Bucket}/.... It opens every input file and report directory by a walk from a handle on the account directory, with openat and O_NOFOLLOW. The walk never follows a symbolic link out of the account subtree. It needs unix, so on other targets Start refuses the filesystem store. Report files get mode 0600 and report directories 0700.
• The S3 store lists the input at Start with ListObjectsV2, 1,000 keys to a page. It reads each object with ranged GETs of 8 MiB. Each GET carries If-Match with the ETag from the listing, so a read takes bytes only from the object listed at Start. The store writes each report object as a multipart upload with the requested server-side encryption.
• The S3 store sends an S3BucketOwner other than the caller's ExtendDB account id as x-amz-expected-bucket-owner, and S3 decides. Each S3 call that a Start makes ends within 10 seconds.
• The S3 store maps S3 errors to the service's FailureCode values, with S3's message and request ids. A code about the server's own credentials, account or clock, such as ExpiredToken, answers InternalServerError, and the server logs it.

The server (crates/server) and the CLI (crates/app):

• Start needs two actions: dynamodb:StartBulkOperation on arn:aws:dynamodb:<region>:<account-id>:bulk-operation/*, then dynamodb:BulkPutItems on the table ARN. The server evaluates Describe and Cancel on the operation ARN, and List on the bulk-operation/* wildcard.
• A bulk operation reads and writes through three S3 actions. The server evaluates them for the caller against the caller's ExtendDB policies, on both stores. It checks s3:ListBucket at Start, and s3:GetObject before the runner opens each input object and at each checkpoint. It checks s3:PutObject at the first record of each report kind, before each report object and before the manifest. Developer mode evaluates no S3 action.
• The server resolves [bulk] once at startup. Without the section, bulk operations are off, except in developer mode, which uses a filesystem store at ./.extenddb/buckets. The server creates the filesystem store's account directories at startup, when the management API creates an account, and at Start. It warns when the S3 store's credentials are an access key that it also accepts from its clients.
• extenddb init writes the [bulk] examples and the bulk [limits] keys as comments.
• BulkOperationsStarted, BulkOperationsFinished and BulkPutItemsWritten count the starts, the terminal statuses and the items written.

Decisions to review

• The four operations stay out of the cross-account pre-check of main (crates/server/src/authz_resource/cross_account.rs). The bulk request path answers an ARN of another account itself, before the server reads any policy. Start answers such a TableName with Access is denied, and Describe and Cancel answer such an operation ARN with the IAM denial text, both as measured.
• The bulk workers run in the worker set of main and stop with it. When the server stops its workers, the claim loop aborts a run still running after one second. The run keeps its lease until the lease lapses. A later claim resumes it from its last checkpoint, with its applied rows in the ledger.
• A transient storage or store fault leaves the operation to resume from its last checkpoint. After ten claims in a row find the checkpoint where the claim before them found it, the operation ends FAILED, or CANCELLED when a cancel is pending.
• Start records the caller on the operation. The later S3 checks read the caller's identity policies, group policies and permissions boundary when they run, and use the session data recorded at Start. A policy change takes effect at the next check, and a session that expires during a load does not fail it.
• On the S3 store, the caller's ExtendDB policies and the buckets that the server's credentials reach are the only separation between ExtendDB accounts. A caller whose policies allow the S3 actions on * reaches every bucket that those credentials reach. The security model tells operators to grant each account's callers that account's buckets only.
• Start lists the whole input prefix before it answers. A long listing on a slow endpoint can outlast a client's read timeout.
• A daemonized serve keeps the directory it started in as its working directory, where main changes to /. A relative path in the configuration then resolves as it does with --foreground.
• The log filter keeps the aws_config target at warn at every level, since the SDK's profile loader logs the access key id at info.
• A row's condition gets the attribute name size check that PutItem applies to ExpressionAttributeNames and ExpressionAttributeValues, before the runner resolves the condition.

Differences from DynamoDB

Most entries under Differences from DynamoDB come from this commit. The exceptions are the operation limits and the retention period, which the core commit sets. The storage commit adds the other two: the synchronous refusal of a second start and the refusal on MongoDB and Cassandra.

test: add the bulk put suite and the S3 bulk store job

What it adds

tests/bulk_put sends the four operations to a running server as SigV4-signed requests. It covers validation, the lifecycle, idempotency, listing, the store rules, authorization and the S3 actions that an operation checks for its caller. The suite writes its inputs to, and reads its reports from, the store that EXTENDDB_TEST_BULK_STORE names. devtools/run-tests sets that variable from its new --bulk-store option. devtools/run-tests-bulk-store.sh writes the matching [bulk] section. On MongoDB and Cassandra the suite checks that the server refuses the bulk requests, then skips.

.github/workflows/integration-s3.yml starts an S3-compatible service in a container, pinned by digest. It runs the engine's bulk store contract tests and the S3 smoke test against that service. Then it runs the suite on a PostgreSQL server whose [bulk] store is s3. integration.yml calls the workflow as a job, and its integration gate requires that job.

The provisioned test user may call s3:ListBucket, s3:GetObject and s3:PutObject. A CLI lifecycle test checks that a daemonized serve resolves relative bulk, import and export paths against its start directory. devtools/run-tests no longer fails the CLI lifecycle stage when --filter selects none of its tests.

Decisions to review

• The suite signs its own requests with botocore's SigV4 signer, as tests/test_import_export.py does.
• On MongoDB and Cassandra the suite asserts the refusal before it skips, so a backend that answers in another way fails the suite.
• devtools/run-tests-bulk-store.sh accepts only an S3 endpoint on localhost or 127.0.0.1, from EXTENDDB_TEST_S3_ENDPOINT, with the keys from EXTENDDB_TEST_S3_ACCESS_KEY_ID and EXTENDDB_TEST_S3_SECRET_ACCESS_KEY.

Differences from DynamoDB

This commit adds no difference. The suite checks the behaviour that the other commits add.

docs: document bulk put items

What it adds

• docs/manuals/03-usage-guide.md gains a Bulk Put Items chapter. It covers request members, the input on each store, the four operations, the status machine and failure messages. Its other sections cover counters, the error report, row errors, one operation per table, idempotency and configuration.
• docs/manuals/05-admin-guide.md documents [bulk], the S3 store's AWS credentials and the permissions they need, and the startup log lines. It gives the rule for plain http endpoints, the bulk [limits] keys and report file reclaim. A new Bulk Operations chapter covers the operator's cancel and the faults during a load.
• docs/manuals/10-security-model.md lists the five DynamoDB actions with their resources, and the three S3 actions with their condition keys. It also says what a policy variable in an S3 Resource matches on each store.
• docs/differences-from-dynamodb.md gets a Bulk Operations table, and docs/troubleshooting.md has new entries for the bulk error messages and the startup lines.
• docs/manuals/07-upgrade-manual.md lists migration 8 and covers backup and restore during a bulk operation. The guides that print the catalog migration show migration 8.
• CONTRIBUTING.md moves its worked example to migration 9, because stream_pending and bulk_operations hold 7 and 8.
• README.md lists the four operations, and extenddb.sample.toml has the [bulk] examples and the bulk limits. The design documents cover the bulk catalog tables, a batch's index checks and the bulk metrics.
• REQ-DEPLOY-006 in docs/design/01-requirements.md raises the container image limit from 50 MB to 60 MB.

Decisions to review

• The usage guide documents the S3 store and the filesystem store in one chapter, and says where the two behave differently.
• The upgrade manual tells operators to cancel the bulk operations, or let them finish, before a backup that they intend to restore from. The catalog and the data database keep one operation's state as a matched pair, and a restore must bring both back to the same point.

Differences from DynamoDB

This commit adds no difference. The differences page lists the differences that the other commits add.

Differences from DynamoDB

docs/differences-from-dynamodb.md carries this list in its Bulk Operations table.

• Input and report location: DynamoDB reads and writes S3 buckets as the caller. ExtendDB uses the store that [bulk] store selects. The S3 store reads and writes AWS S3 or an S3-compatible service with the server's own AWS credentials. The filesystem store maps S3Bucket to a directory in the caller's account subtree beneath one root. On both stores the report records and the manifest keep the arn:aws:s3::: spelling in inputSourceArn.
• Input failures decided at Start: Start detects four faults before the operation reads any row. They are a missing input bucket, an input S3BucketOwner that the bucket does not match, a denied s3:ListBucket and an unusable output KMS key. DynamoDB answers IN_PROGRESS and ends the operation FAILED seconds later (measured). ExtendDB creates the operation already FAILED, so the Start response carries the code, an EndTime equal to StartTime, no byte members and no manifest. At max_bulk_failed_starts_per_hour such starts in an hour for one account (default 100), Start answers ThrottlingException and records nothing. On a table with an operation running, such a Start answers ResourceInUseException and creates nothing, as DynamoDB does.
• An output location that cannot be used: neither checks it at Start. At the first report record, both hold a missing bucket, an owner mismatch or a refused s3:PutObject and write every good row. Then both end FAILED with no report and no manifest (measured). DynamoDB also writes report objects during the run, and a write that fails there stops the operation at once (measured). ExtendDB writes its reports only when the operation ends, so it can write more rows than DynamoDB before it fails. DynamoDB's answers for two cases here are not measured. They are a refused s3:PutObject at the manifest write and the count of rows whose condition failed under a held fault.
• S3BucketOwner: DynamoDB fails an owner other than the bucket's with S3AccessDenied, Access Denied (measured). The filesystem store fails any account other than the caller's in the same way, because the caller's account owns every bucket directory there. The S3 store sends an owner other than the caller's ExtendDB account id to S3 as x-amz-expected-bucket-owner, and S3 decides.
• The caller's S3 permissions: DynamoDB performs s3:ListBucket, s3:GetObject and s3:PutObject as the caller, so identity, bucket and KMS key policies apply (measured). ExtendDB evaluates the three actions against the caller's ExtendDB identity policies. It has no bucket policies and evaluates no kms: action. Its denial carries no request-id tail, because no request reached S3. On the S3 store, S3 then evaluates the server's own request, and a denial that S3 returns names the principal that the server signs as.
• The FailureMessage of an S3 fault: the S3 store adds the two tails that DynamoDB adds (measured), with S3's request ids. (Service: Amazon S3; Status Code: ...; Proxy: null) follows a fault of the listing, the input reads or the manifest. (Service: S3, Status Code: ...) (SDK Attempt Count: 1) follows a fault of a report write. The filesystem store gives the message without a tail.
• A second start on a table with a running operation: DynamoDB rejects it with ResourceInUseException or accepts it and fails it later with that FailureCode. ExtendDB always rejects it at once.
• TotalInputBytes in the Start response: DynamoDB omits it until it has the total. ExtendDB lists the input before Start returns, so the member is present at once.
• MaxWriteRequestUnits and PerformanceTier: ExtendDB paces an operation's writes at MaxWriteRequestUnits 1 KB write units a second, under the server's bulk_put_write_units_per_second. It checks, stores and returns PerformanceTier, which does not change the rate. ExtendDB refuses ACCELERATED on a table with a vector index, as DynamoDB does.
• InputFormat ION: the API model has it, and ExtendDB refuses it with a ValidationException that names the value.
• S3SseAlgorithm and S3SseKmsKeyId: the S3 store applies them to every report object and the manifest. The filesystem store stores and echoes them, and they have no effect on its files.
• AWS KMS keys on the input objects, the report objects and the table: on the S3 store, S3 uses the server's KMS permissions. ExtendDB evaluates no kms: action for the caller. An input object under a key that the server may not use fails with S3AccessDenied, and one under a disabled key with KmsDisabledException. A disabled or asymmetric report key fails the operation at Start with KmsDisabledException or KmsInvalidKeyUsageException. A KMS fault's FailureMessage carries S3's text, where DynamoDB's carries the text of its own KMS call. The filesystem store has no KMS. On both stores ExtendDB records the table's encryption settings only, so no table key fault refuses Start. DynamoDB refuses Start on a table whose key is disabled or pending deletion.
• Report files for each kind: DynamoDB writes numbered files in one partition directory for each input hash partition. It rolls a file once its compressed bytes pass 200 MiB. ExtendDB writes one partition directory, rolls at the same 200 MiB, and applies its record and byte caps to the whole report.
• Report objects left incomplete: on the S3 store each report object is a multipart upload until its write completes at the end of the operation. A server that stops during those writes, or an abort that fails, leaves an incomplete upload in the output bucket. The upload stays until a lifecycle rule aborts it.
• The order of the input objects: in two measured runs DynamoDB read the second object, by key order, in full before it finished the first. ExtendDB reads one object at a time in the byte order of the keys. As a result, a row in a later object overwrites a row with the same key in an earlier one.
• Operation limits: ExtendDB has [limits] keys, such as max_bulk_input_bytes, max_bulk_input_files and max_bulk_item_count, where DynamoDB has service quotas.
• Operation record retention: bulk_operation_retention_days defaults to 30. A purged operation answers ResourceNotFoundException, and its report files stay.
• The MongoDB and Cassandra backends: the four operations answer ValidationException, Bulk operations are not supported on this storage backend.
• U+0000 in an attribute name or a string value: DynamoDB stores it. ExtendDB fails the row with an item-level record and the load continues, because PostgreSQL's jsonb cannot store the character. ExtendDB stores binary values with zero bytes.
• The manifest entry of a file that the runner is reading when a cancel arrives: DynamoDB reports NOT_AVAILABLE with no range. ExtendDB reports PARTIALLY_PROCESSED with the range of rows that it counted, because the runner checkpoints at row boundaries.

Follow-ups

1. The S3 store has its own client crate, crates/s3. Moving it onto crates/backup-store waits until the backup stack on the merged feat(backup): catalog index, BackupCatalog, JobClaim, legacy reader [backup/restore 6] #671 is complete. feat(backup): [backup.store], the store identity file, backup purge [backup/restore 7] #672 and feat(backup): CreateBackup job, worker and reconciler duties [backup/restore 8] #700 are in main, and feat(backup): RestoreTableFromBackup job and cross-backend matrix [backup/restore 9] #740 is open.
2. bulk_operations.input_files always holds [], because the input files are rows of bulk_operation_input_files. Migration 8 is not released, so the column can leave its CREATE TABLE before v1.0 freezes the migration.
3. The record text for a row whose attribute name is over 65,535 bytes follows the text of PutItem. The service's text for that row is not measured.

Testing done

The Rust suites:

• crates/core: wire names, enum sets, ARN layers and description shapes.
• crates/s3: the clients over the SDK's replay HTTP client. The tests cover endpoints, region redirects, retries and deadlines, the identity check and its log lines, and the temporary-keys warning.
• crates/config: [bulk] sections, refused credential keys, the rule for plain http endpoints and environment overrides.
• crates/storage: the bulk operation store contract suite, with the same cases on SQLite and on PostgreSQL. It covers start and idempotency, the table guard and caps, claims and leases, checkpoints and report pages. The other cases cover finish, purge, the ledger, the write owner, index creation during a run, lock waits and operations with many input files.
• crates/engine: the handlers through dispatch, and the runner end to end on SQLite and, with EXTENDDB_TEST_PG_CONNECTION_STRING set, on PostgreSQL. The runner tests cover crash and resume, lease takeover, the ledger, row and file faults, cancellation, reports, the manifest and pacing. The S3 store tests use the SDK's replay HTTP client and a local stub listener. The bulk store contract suite runs each case on the filesystem store and, with EXTENDDB_TEST_S3_ENDPOINT set, on the S3 store.
• crates/server: authorization of the five DynamoDB actions and the three S3 actions with their condition keys and policy variables, cross-account cases, and [bulk] resolution at startup.

The Python suite in tests/bulk_put runs against a server on either store, through devtools/run-tests --extenddb --pytest --bulk-store filesystem or --bulk-store s3, with EXTENDDB_TEST_BACKUP_STORE set as devtools/run-tests requires since #700. The S3 job runs devtools/run-tests --extenddb --pytest --bulk-store s3 --parallel --filter bulk_put against an S3-compatible service.

The behaviours that the docs mark as measured match recorded requests to DynamoDB and their responses. The measurements cover these areas:

• request checks, their order and their error texts, ARN layers, and the Describe and List shapes
• row faults that fail a row, and those that abandon the rest of a file, with their texts
• bounds on the expression maps
• report keys, record shapes, the roll at 200 MiB and the manifest
• usage charges on tables with and without indexes
• terminal statuses for missing buckets, owner mismatches, quota failures, cancels and deleted tables
• UpdateTable during a run, and a stream enabled during a load
• ClientToken replay, and AWS KMS keys on the inputs, the reports and the table

The S3 store follows these behaviours of AWS S3, each measured:

• S3 keeps keys with // and ./ segments as written.
• S3 puts the requested encryption on report objects and on the manifest.
• S3 refuses a request whose expected bucket owner, from S3BucketOwner, does not own the bucket.
• S3 redirects a request for a bucket in another region, and the store follows the redirect to the bucket.
• S3 answers 412 to a GET whose If-Match no longer matches.

Measurements of AWS STS gave the error codes that the identity check reads. The cases were an unknown key, a wrong secret, an expired session token and a clock more than 15 minutes off.

Runs on the merge of main at d35d8601a:

• cargo test --workspace --exclude extenddb-backup-store in the CI job for Rust 1.88: 4,880 passed, 0 failed, 71 ignored.
• cargo test -p extenddb-storage-postgres --tests in CI: 505 passed, 2 ignored.
• Against local database servers: cargo test -p extenddb-engine --lib with its PostgreSQL cases passed 1,255 tests, and the MongoDB and Cassandra migration tests passed.
• cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings and cargo +1.88.0 check --workspace --all-targets are clean, and the license notices are current.
• A fresh install's schema dump on PostgreSQL and on SQLite equals docs/schema.

The integration jobs need #895, which raises their time limits. Until #895 merges, run-integration, run-integration-sqlite and mongodb-rust can stop at their 40-minute limit.

Checklist

• I have read CONTRIBUTING.md
• All tests pass (cargo test --workspace)
• Code is formatted (cargo fmt --check)
• Clippy is clean (cargo clippy -- -W clippy::pedantic)
• I have added or updated tests for new functionality
• I have updated documentation if behavior changed
• Breaking changes are noted below (if any)
• If this changes the wire protocol, Storage trait, auth model, on-disk
  format, or public CLI surface, an RFC has been accepted or is linked
  below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: none for bulk put items. Migration 8 follows docs/adr/0023-catalog-migrations.md.

Breaking changes

• A catalog without migration 8 needs extenddb migrate --yes before a server of this build starts on it, because the server refuses a catalog with pending migrations.
• PutItem runs the nesting-depth check of the expression values before the table lookup. Only a request that combines an over-deep expression value with a missing table or an invalid item gets a different error.
• A daemonized serve keeps the directory it started in as its working directory, where main changes to /. A relative path in the configuration or the environment, such as an [import] or [export] root or run_dir, now resolves against that directory.
• The Rust API inside the workspace: DataEngine::put_item takes a trailing marker argument, DataEngine::put_items_batch is a required method, StorageEngine requires BulkOperationStore, and extenddb_core::validation::validate_put_item is gone.

---

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.