Skip to content

fix(expressions): validate in the service's order, with its messages - #399

Open
LeeroyHannigan wants to merge 17 commits into
mainfrom
fix/transact-validation-order
Open

LeeroyHannigan wants to merge 17 commits into
mainfrom
fix/transact-validation-order

Conversation

@LeeroyHannigan

@LeeroyHannigan LeeroyHannigan commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

What

Issue #108 lists 106 requests whose response differs from Amazon DynamoDB. This change makes ExtendDB validate expressions and transactions in the service's order and return the service's messages for the groups below. Every expectation was measured against the service (us-east-1, 2026-10-14), and every new test file was run against the service before it was run against ExtendDB.

Semantic validation of expressions. After the grammar pass, the service walks the parsed expression once, in expression order, checking each node after its operands, then checks overlapping paths, then unused names and values. New validate_update_semantics and validate_condition_semantics in crates/core/src/expression/semantics/ (one file per rule family) follow that walk. The parsers now do grammar only, keep function names as written, and record section order for the walk.

  • UpdateExpression: + and - take numbers, list_append takes lists, ADD takes a number or a set, DELETE takes a set. A placeholder of the wrong type is refused with the service's type vocabulary (S, N, L for operators and functions; STRING, NUMBER, Binary, BOOL, NULL, LIST, MAP for ADD and DELETE). Function names are case-sensitive: IF_NOT_EXISTS(x, :v) and foo(x) are Invalid function name, a condition function is not allowed in an update expression, list_append is refused as an operand of a function or of +, both functions take two operands, and the first operand of if_not_exists is a document path. A repeated section is reported after that section's own actions, so SET a = :v SET b = :q reports :q. Before this change IF_NOT_EXISTS(x, :v) was accepted and written, and DELETE s :string was accepted when s did not exist. SET a = :string + :n was already refused when it was evaluated (An operand in the update expression has an incorrect data type); it is now refused before the table is read, with the service's text.
  • ConditionExpression and FilterExpression: the six condition functions are matched as written (ATTRIBUTE_EXISTS(pk) was accepted and evaluated); if_not_exists and list_append are not allowed in a condition expression; a boolean function in operand position and size in condition position are not allowed to be used this way; operand counts are fixed per function; attribute_exists and attribute_not_exists take a document path; begins_with takes a string or binary and attribute_type a string naming a type, both checked at request time with the operation's prefix.
  • Every unused name or value is listed in one set, sorted, so the message is the same on every backend. It named one key chosen by hash order, which differed between backends and processes.

Legacy conditions. ScanFilter, QueryFilter and KeyConditions bypass the expression parser, so they get their own request-time check, ahead of the desugaring and the table lookup, with the rules measured against the service on 2026-10-23 for every ComparisonOperator and AttributeValue type: the operator name (Invalid comparison operator), the value count (Invalid number of argument(s) for the <op> ComparisonOperator), the first value's type (ComparisonOperator <op> is not valid for <type> AttributeValue type), that every value has the first value's type, and that a BETWEEN lower bound is not above its upper bound, each with the One or more parameter values were invalid: prefix. A range key operator the service does not index (NE, NULL, NOT_NULL, CONTAINS, NOT_CONTAINS, IN) in KeyConditions is refused with Attempted conditional constraint is not an indexable operation. Before this change a ScanFilter with BEGINS_WITH and a number returned 200 on an empty table.

Request stage on the read APIs. Query and Scan check the FilterExpression, then the KeyConditionExpression (with its Invalid KeyConditionExpression: prefix and ahead of the key schema match, so an undefined #n is reported as such), then the ProjectionExpression, then unused names and values, all before the table is looked up. An undefined :q in a filter returned 200 on an empty table. Limit and Segment checks precede the expressions on Scan. A KeyConditionExpression function name is answered by name: contains, attribute_exists, attribute_not_exists and attribute_type give Invalid operator used in KeyConditionExpression with no function name, size gives KeyConditionExpressions cannot contain nested operations, and any other name, Begins_With included, gives Invalid function name; function: <name>. The first undefined #name or :value in the text is the one named, names and values alike. A Limit under 1 on Query and Scan gives 1 validation error detected: Value at 'Limit' failed to satisfy constraint: Member must have value greater than or equal to 1, ahead of the table lookup and every expression. On Scan, the range checks on Segment and TotalSegments come first, and the checks that the two go together and that Segment is below TotalSegments come after the expressions and the unused names and values and before the table lookup.

Single-item order. UpdateItem checks the whole ConditionExpression before the UpdateExpression. PutItem, UpdateItem, and DeleteItem wrap every expression-stage error in 1 validation error detected: and omit the : ConditionExpression is null suffix; the read, batch, and transaction APIs return the messages bare. The expression size: N suffix follows the operation (present on a Scan FilterExpression and a transaction ConditionExpression, absent on a Query FilterExpression and the single-item ConditionExpression). Eleven assertions in three existing dual-target test files asserted the unprefixed text and fail against the service (test_expression_error_prefix.py 3, test_expression_param_usage.py 6, test_expression_size_limit.py 2); they are corrected, and one assertion is added to the third file. Legacy AttributeUpdates with an ADD of a value that is not a number or a set, or a DELETE with a value that is not a set, are refused with 1 validation error detected: One or more parameter values were invalid: ADD action is not supported for the type S (and DELETE action with value is not supported for the type N), ahead of the table lookup and key validation. A ConditionCheck with an empty ConditionExpression gives Invalid ConditionExpression: The expression can not be empty;.

Transactions. TransactGetItems and TransactWriteItems validate in phases, each over every item before the next. TransactWriteItems checks, one item after another and before any table is looked up, the nesting of the item's ExpressionAttributeValues (32 wrappers pass, 33 do not), its expressions (UpdateExpression before ConditionExpression), the unused names and values, and for a Put the attribute name size, the 400 KB item size and the item nesting. Then it looks the tables up, checks the 4 MB total, and makes one pass over the items in order that stops at the first failing item: a key that does not fit the schema, then a key attribute named in an Update (both a cancellation with ValidationError on that item and None on every other item), then an empty key value (a top-level ValidationException). Duplicate targets come next, and last an Update that stores a value nested as deep as an item may not hold, which is a ValidationError reason on every such item. TransactGetItems checks the projections, looks the tables up, checks the key schema (first offending item only) and then duplicate keys. An undefined reference is the request error the service returns rather than a per-item cancellation reason, and a key schema mismatch leaves the other items' conditions unevaluated. TransactWriteItems returns "ItemCollectionMetrics": {} when SIZE is requested and no table has a local secondary index, and the ConsumedCapacity entries of both transaction APIs are ordered by table creation time, oldest first, as measured on the service (section below).

Key-shaped messages. GetItem, DeleteItem, UpdateItem, BatchGetItem, and BatchWriteItem (Put and Delete) refuse a key with a missing, mistyped, or extra attribute with The provided key element does not match the schema. PutItem keeps Missing the key pk in the item and Type mismatch for key. BatchWriteItem returns "ItemCollectionMetrics": {} under SIZE with no LSI.

Table ARNs in TableName (eight #108 rows) stay unsupported by decision: an ExtendDB table ARN names ExtendDB's account and region, so a table that streams or replicates to Amazon DynamoDB would answer to two ARNs. docs/differences-from-dynamodb.md records this. Open PR #200 takes the opposite position and needs reconciling before either merges.

This is one PR with three pieces that can be read on their own. The legacy condition validator (legacy_filter.rs, legacy_filter_tests.rs, and the 79 wire cases in tests/test_expr_operand_type_and_size.py) is in aef6e1b, 475465a and 90fdc32. The ConsumedCapacity order is in 68dcd0a, 217b5f9 and b7ba331: the creation_date_time field on TableKeyInfo, the column read in the SQLite, PostgreSQL and Cassandra storage crates, the MongoDB catalog write that keeps milliseconds, and tests/test_transact_capacity_order.py. The decision to keep table ARNs in TableName unsupported is one row in docs/differences-from-dynamodb.md, added in 29229c2 with the semantic pass. Commits ac76c66 and 964f3c3 only move code: the semantic pass split by rule family, and the update parser tests into their own file. The six commits from af5f681 to cbb04f0 change the transaction phases, the Query key condition checks, four request stage texts, the Scan parameter order, and add tests for each.

Why

Closes part of #108. A client that maps on the service's error codes and messages, or that reads CancellationReasons, ItemCollectionMetrics, or ConsumedCapacity, saw different responses from ExtendDB on these requests, and request shapes the service refuses were accepted: a function name in the wrong case, a DELETE of a string on an absent attribute, a repeated update section, overlapping update paths, a nested list_append, and an if_not_exists whose first operand is not a path.

Of the 106 rows in #108, 83 returned the service's status, error type, and message (30 on main) when measured on this branch on 2026-10-14, before the legacy condition work above. The 24 that remained are in these groups:

  • the parser's syntax wording (Syntax error; token: X, near: Y; 14 rows);
  • the ResourceNotFound message text (3 rows);
  • the table ARN rows (5), unsupported by decision;
  • the service's rendering of an over-limit list value in the 1 validation error detected message (2 rows: 101 transaction items, 26 batch writes).

The 4 MB transaction message also remains: the service's text carries its own serialized payload size.

Testing done

  • tests/test_expression_semantics.py (new, 159 cases): operand types, function names, the walk order, the unused set, condition function rules, Query and Scan request-stage checks, key-shaped messages, BatchWriteItem ItemCollectionMetrics. tests/test_transact_validation_order.py (new, 31 cases): the transaction phases, first-offender cancellation, item size and 4 MB positions. tests/test_update_expression_rules.py (new, 37 cases): repeated sections, operands, overlap, the single-item prefix. Every expected response was captured from the service; all three files pass against the service and on SQLite, PostgreSQL, and MongoDB. 164 of the 186 cases in the first two fail on main.
  • The expression, validation, transaction, batch, query, and scan suites pass on all three backends (767 on SQLite and PostgreSQL, 751 on MongoDB with the four test_transact_absent_item_isolation cases of fix(postgres): lock missing items that TransactWriteItems checks or deletes #387 unchanged). The remaining full-suite failures on a dev-mode SQLite server (metrics, import and export, gsi async, misc operations) are identical on main and need infrastructure the dev server lacks.
  • Unit tests for the semantic pass in semantics/tests.rs (10) pin the walk order and the per-node rules inside one expression. The orders between expressions and stages (condition before update, the Query stages, the transaction phases) are pinned by the wire tests. At 4f4fc21: cargo test --workspace 2,021 passed on stable; on 1.88.0, 1,784 passed with the Cassandra crate excluded, because the local Cassandra node was held by another workload (that crate passed on stable in the same run); cargo fmt --check and cargo clippy --all-targets -- -D warnings on 1.97.0 clean.
  • tests/test_transact_capacity_order.py (new, 6 cases) and the legacy condition cases in tests/test_expr_operand_type_and_size.py and tests/test_expression_param_usage.py: measured on the service on 2026-10-23. The eight test files this PR adds or changes pass on MongoDB at 4f4fc21 (391 passed), including the ordering test with tables created within one second of each other.
  • Latency of the changed paths on the dev-mode SQLite server, this branch against main, one client, two runs (p50): TransactWriteItems with 100 Puts 23.8 to 23.9 ms against 24.0 to 24.1; UpdateItem with 50 SET actions 1.48 to 1.50 ms against 1.47; PutItem with a 20-clause condition 1.17 to 1.19 ms against 1.18; an UpdateItem with an invalid expression 0.88 ms against 0.88 to 0.89.
  • tests/test_transact_write_phase_order.py (new, 16 cases), tests/test_request_stage_checks.py (new, 30 cases) and tests/test_transact_value_nesting.py (new, 6 cases) pin the transaction check order, the key condition, Limit, Segment, legacy AttributeUpdates and empty expression texts, and the value nesting thresholds. Every expectation was measured on the service on 2026-10-27, two samples each, and all 52 cases pass against the service and against a SQLite dev-mode server. 861 tests pass on the dev-mode server across the transaction, expression, query, scan, item, batch and update files. At 95e90f8, cargo test -p extenddb-core -p extenddb-engine passes (530 and 187), cargo fmt --check and cargo clippy --all-targets -- -D warnings on 1.97.0 are clean for core, engine and storage-sqlite, and cargo check on 1.88.0 passes for core and engine. The workspace totals quoted in the next bullet were not re-run.

ConsumedCapacity order per backend

TransactGetItems and TransactWriteItems return one ConsumedCapacity entry per table, ordered by the table's creation time, oldest first. Two tables with the same stored creation time are listed in name order.

The precision of the stored creation time differs per backend, so the point at which two creates tie differs with it:

  • SQLite: about 0.24 microseconds. The catalog text carries nanosecond digits, and the value the order uses is an f64 of epoch seconds.
  • PostgreSQL: microseconds, from a TIMESTAMPTZ column.
  • MongoDB: milliseconds. The catalog write used to drop the sub-second part, so two tables created inside one second tied and the array came back in name order. A catalog row written before that fix keeps whole seconds and still ties.
  • Cassandra: milliseconds, from a CQL timestamp column.

Measured on SQLite, PostgreSQL and MongoDB. Cassandra reads the same catalog column, and the six cases of tests/test_transact_capacity_order.py pass on it in the Cassandra integration job at 4f4fc21.

Known differences

These are differences from the service on the legacy condition paths that this PR does not change, measured against the live service on 2026-10-23 and filed as #477. Since #429, main answers items 5 and 7 with the service's message for a table's own key attributes; the other six are present on main.

  1. Two legacy condition entries that are both invalid: the service names the second entry in map order, ExtendDB names the first.
  2. Query with a QueryFilter on a primary key attribute: the service refuses it, ExtendDB returns 200.
  3. ScanFilter EQ with an N AttributeValue whose value is not a number: the service reports the numeric conversion, ExtendDB returns 200.
  4. ScanFilter BEGINS_WITH with an N AttributeValue whose value is not a number: the service reports the numeric conversion first, ExtendDB reports the operator against type message.
  5. KeyConditions on the sort key with a value whose type is valid but is not the schema type: the service refuses it with the schema type message, ExtendDB answered HTTP 500 before fix(query): classify and order key conditions on a multi-attribute GSI the way the service does #429.
  6. KeyConditions EQ or BEGINS_WITH with an empty string: the service refuses it, ExtendDB returns 200.
  7. KeyConditions EQ on an S partition key with an N value: the service refuses it with the schema type message, ExtendDB returned 200 before fix(query): classify and order key conditions on a multi-attribute GSI the way the service does #429.
  8. BatchGetItem with ReturnConsumedCapacity TOTAL where every key is absent: the service returns one entry per table, ExtendDB returns an empty list.

Measured against the live service on 2026-10-27 and present on main:

  1. An empty ConditionExpression on a transaction Put, Delete or Update, and on PutItem, UpdateItem or DeleteItem: the service answers Invalid ConditionExpression: The expression can not be empty;, ExtendDB treats it as no condition.
  2. An attribute name over the limit: the service answers Attribute name is too large, must be less than 65536 bytes, ExtendDB words it differently. The boundary agrees.
  3. A legacy ADD of an empty list: the service accepts it, ExtendDB refuses it.
  4. Scan Segment and TotalSegments range checks: the service answers Value at 'TotalSegments' failed to satisfy constraint: ... and Value at 'Segment' ..., ExtendDB uses another wording, and its text for a Segment at or above TotalSegments ends with the two values.
  5. A Scan with ExpressionAttributeValues and no expression that uses them: the service answers ExpressionAttributeValues can only be specified when using expressions, ExtendDB adds : FilterExpression is null.
  6. An Update that stores a value nested as deep as an item may not hold is reported with the other items' condition results on the service (ValidationError next to ConditionalCheckFailed), and with None for them on ExtendDB.

Rebase on main

Rebased onto main at f49c5f8 on 31 October. The branch keeps its 17 commits in the same order; the merge commit f9355f5 is gone, and the first commit carries what that merge had resolved (table_key_info_uncached in transact_get_items.rs, the imports in transact_write_items.rs, and the table ARN row of the differences document). The commit ids in the What section are the ones on this head. The ids in Testing done and in the ConsumedCapacity section (4f4fc21, 95e90f8) name the heads those runs were made at, before the rebase.

The main PRs this branch was resolved against, and what changed in its code because of them:

git range-diff 9d132022..17b805da f49c5f82..1454cc93: the first commit differs by the former merge's resolutions and by #429's lines in the handle_query code it moves and above the function it removes, and the fifth by the stream_ownership.rs line above. The other fifteen carry the same diff.

At 1454cc9: cargo fmt --all --check and cargo clippy --locked --workspace --all-targets -- -D warnings on 1.97.0 clean; cargo check --locked --workspace on 1.88.0 passes; cargo test --locked --lib passes for core (726), engine (192) and server (237), #429's 20 unit tests in key_condition.rs and leading_keys among them. The eleven test files this PR adds or changes, plus tests/test_fgac_leading_keys.py and #429's tests/test_gsi_multi_range_query.py, pass on a SQLite server and on a PostgreSQL server built from this head: 537 cases passed on each, and one case of tests/test_gsi_multi_range_query.py skipped, the one that runs only on a backend that refuses a condition on a later RANGE attribute. Against Cassandra 4.1 with password authentication, as the Cassandra integration job runs them, extenddb-storage-cassandra's unit tests (75) and direct storage tests (181) pass.

CI passed all 35 checks on 1454cc932.

Checklist

  • I have read CONTRIBUTING.md
  • All tests pass (cargo test --workspace)
  • Code is formatted (cargo fmt --check)
  • Clippy is clean (cargo clippy -- -W clippy::pedantic)
  • I have added or updated tests for new functionality
  • I have updated documentation if behavior changed
  • Breaking changes are noted below (if any)
  • If this changes the wire protocol, Storage trait, auth model, on-disk format, or public CLI surface, an RFC has been accepted or is linked below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: n/a. The change alters wire-visible error text and response members toward the live service, which CONTRIBUTING lists under the RFC trigger; it is a parity fix measured against the service rather than a protocol proposal. A maintainer's call on whether an ADR is wanted.

Breaking changes

Requests the service refuses and ExtendDB accepted are now refused: function names in the wrong case in an UpdateExpression or a condition, a DELETE of a non-set value on an absent attribute, a repeated update section, overlapping update paths, a nested list_append, an if_not_exists whose first operand is not a path, and an undefined :value or #name in a Query or Scan filter (previously a 200 when no row was evaluated). Requests that evaluation already refused are now refused before the table is read, with the service's text: + or - on a non-numeric placeholder, ADD of a wrong type, and list_append of a non-list. A Scan with Segment and no TotalSegments now reports an expression error ahead of the missing parameter, as the service does. Error messages for these groups change to the service's text. Key mismatch messages on the key-shaped APIs change to The provided key element does not match the schema. ItemCollectionMetrics is present as {} on TransactWriteItems and BatchWriteItem under SIZE with no LSI. The ConsumedCapacity array of both transaction APIs is ordered by table creation time. ScanFilter, QueryFilter and KeyConditions conditions the service refuses are refused at request time with the service's messages.


By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

Comment thread crates/core/src/expression/parser.rs Outdated
// itself; whether that function may stand alone, whether its name exists,
// and how many operands it takes are semantic questions answered after
// the grammar pass (see `validate_condition_semantics`), so a syntax error
// further along still wins, as on Amazon DynamoDB.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does "wins" mean in this context?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Wins" meant precedence between two errors in one expression, and the comment did not say so. The grammar pass in parser.rs reports every syntax error first; the semantic checks (function name, whether a function may stand alone, operand count) run afterwards in validate_condition_semantics (crates/core/src/expression/semantics/condition_rules.rs:31), over the whole parsed tree. So for foo(pk) AND the response is the syntax error at the end of the text, not the unknown function name that appears first.

The service has the same precedence. Measured 2026-10-23 on PutItem: foo(pk) AND, ATTRIBUTE_EXISTS(pk) AND, begins_with(pk, :n) AND (with :n a number) and attribute_exists(#x) AND (with #x undefined) each answer Invalid ConditionExpression: Syntax error; token: "<EOF>", near: "AND"; foo(pk) alone answers Invalid function name; function: foo. The four cases are pinned in TestConditionFunctions.test_a_syntax_error_beats_the_semantic_errors in tests/test_expression_semantics.py (343c2fb).

I rewrote the comment to say this without the word (crates/core/src/expression/parser.rs:136-144, and the same wording in update_parser/mod.rs:279-286), in a8d83c6.

Comment thread crates/core/src/expression/parser.rs Outdated

let mut args = Vec::new();
if *pos < tokens.len() && tokens[*pos] != Token::RParen {
// `f()` is a syntax error on the service, reported at the `)`.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A syntax error because there are no arguments? Please clarify.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. The service's grammar needs at least one operand between the parentheses, so attribute_exists() and foo() are refused at the ) as a syntax error, Syntax error; token: ")", near: "()", before the function name or the operand count is looked at. Measured 2026-10-23 for both spellings on PutItem (ConditionExpression); the UpdateItem case SET x = foo() was measured 2026-10-14. One operand too few is a different error: begins_with(pk) parses and the semantic pass reports Incorrect number of operands for operator or function; operator or function: begins_with, number of operands: 1 (also measured 2026-10-23).

The check is at crates/core/src/expression/parser.rs:241-251 (and update_parser/mod.rs for SET operands); the wire cases are foo(), attribute_exists() and begins_with(pk) in TestConditionFunctions in tests/test_expression_semantics.py. I rewrote the comment to say this in a8d83c6.

Comment thread crates/engine/src/capacity_helpers.rs Outdated
/// Per-table capacity totals in the order the tables first appear in the
/// request.
///
/// Amazon DynamoDB's own order of the `ConsumedCapacity` entries of a

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't tell if this is saying ExtendDB's behavior (in terms of sorting) is now equivalent to DynamoDB's, or if ExtendDB is providing firmer ordering guarantees than DynamoDB (and why).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is now the service's order. DynamoDB documents no order for the ConsumedCapacity entries of a transaction, so I measured it on 2026-10-23 with three tables created on 2026-10-22, 2026-10-23 11:01 and 2026-10-23 14:45 UTC, with names that sort opposite to the creation order. For the request orders [newest, middle, oldest], [middle, newest, oldest] and [oldest, newest, middle], TransactGetItems and TransactWriteItems both returned the tables oldest first. The order follows neither the request nor the names, it follows the tables' creation time.

PerTableCapacity (now crates/engine/src/per_table_capacity.rs) orders the entries by creation time, oldest first, with name order breaking a tie. TableKeyInfo carries the creation time from the catalog row the handlers already read for the key schema, so there is no extra catalog read. Before this PR the handlers aggregated into a HashMap and the order changed between processes and backends.

Two tables tie when their stored creation times are equal, and the precision of that value differs per backend: nanoseconds on SQLite, microseconds on PostgreSQL, milliseconds on MongoDB and Cassandra. MongoDB was writing whole seconds, so three tables created inside one second got the same value and came back in name order; that write now keeps the millisecond part. A catalog row written by an earlier build still holds whole seconds. tests/test_transact_capacity_order.py creates three tables whose names sort opposite to their creation order and asserts the order on both operations; it runs against the service and against ExtendDB, and it passes on SQLite, PostgreSQL and MongoDB. docs/differences-from-dynamodb.md has the row under Capacity and Throttling, with the precision per backend.

};

// Validate begins_with operand types upfront (before any rows are read).
if let Some(ref f) = filter {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where are these checks happening now?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The expression form moved them into the semantic pass. The legacy QueryFilter path lost them, which was a regression in this PR; it is fixed in 343c2fb and 4132394.

FilterExpression: the begins_with and ordering operand checks run inside the semantic pass, at request time, before the table lookup. The call is parse_condition_checked at crates/engine/src/query.rs:170, which runs validate_condition_semantics (crates/core/src/expression/semantics/condition_rules.rs:31); the begins_with operand type is at condition_rules.rs:131 and the ordering operand types at condition_rules.rs:54 and 60. The checks are in the walk because the service reports them in expression order with the other semantic errors (#x = :v AND begins_with(pk, :n) with an undefined #x reports the name, begins_with(pk, :n) AND #x = :v reports the operand type; both in TestConditionFunctions). Covered by tests/test_expr_operand_type_and_size.py and tests/python/test_begins_with_type_check.py.

QueryFilter (legacy): desugar_filter bypasses the parser, so the removal left that path with no request-time check; an empty table returned 200 and a non-empty one failed on the first row evaluated. The service checks each legacy condition at request time, before the table lookup, with its own messages. I measured the full operator by type grid on 2026-10-23: One or more parameter values were invalid: ComparisonOperator BEGINS_WITH is not valid for N AttributeValue type and the same text for every other refused pair (EQ and NE take every type; BEGINS_WITH takes S and B; CONTAINS and NOT_CONTAINS take S, N, B, BOOL and NULL; the ordering operators, BETWEEN and IN take S, N and B), plus the value count (Invalid number of argument(s) for the <op> ComparisonOperator), mixed types (AttributeValues inside AttributeValueList must be of same type), an inverted BETWEEN range and an unknown operator name. validate_legacy_condition in crates/engine/src/legacy_filter.rs now runs those checks on each QueryFilter, ScanFilter and KeyConditions entry before it is desugared, with the service's text; the Invalid FilterExpression: wording is gone. The grid is pinned in the engine's unit tests (legacy_filter_tests.rs) and one row per message on the wire in TestLegacyFilterOperandType in tests/test_expr_operand_type_and_size.py.

Comment thread crates/engine/src/scan.rs
}

// Validate begins_with operand types upfront (before any rows are scanned).
if let Some(ref f) = filter {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar question to query.rs: where are these checks happening now?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The answer is the same as for Query. FilterExpression: inside the semantic pass through parse_condition_checked at crates/engine/src/scan.rs:197, before the projection, the unused check and the table lookup (scan.rs:300); the begins_with and ordering operand checks are at crates/core/src/expression/semantics/condition_rules.rs:131, 54 and 60.

ScanFilter (legacy): the check had been dropped. 343c2fb and 4132394 restore it in desugar_filter (validate_legacy_condition in crates/engine/src/legacy_filter.rs) with the service's messages, measured 2026-10-23 on an empty table for every operator and type: BEGINS_WITH with a number answers One or more parameter values were invalid: ComparisonOperator BEGINS_WITH is not valid for N AttributeValue type, LT with a boolean names BOOL, BETWEEN with a list then a number names L, and the value count, mixed types, inverted BETWEEN range and unknown operator get their own texts. The ScanFilter rows are in TestLegacyFilterOperandType in tests/test_expr_operand_type_and_size.py.

@jcshepherd jcshepherd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some of the agent-generated comments are read pleasantly but actually are ambiguous which makes it hard to determine whether the code is even solving a real problem. I flagged a couple. Also some operand validation code in query.rs and scan.rs was deleted and it's not obvious that it is still being done somewhere (assuming it was needed).

Comment thread crates/core/src/expression/resolver.rs Outdated
///
/// Returns `DynamoDbError::ValidationException` naming the first undefined
/// reference, names before values, in the order the expressions are given.
pub fn validate_refs_defined(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unused.

jcshepherd
jcshepherd previously approved these changes Oct 26, 2026

@jcshepherd jcshepherd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dead code in resolver.rs: otherwise okay.

…ts messages

Issue #108 lists 106 requests whose response differs from Amazon DynamoDB.
Every expectation in this change was measured against the service
(us-east-1, 2026-10-14); the probes and captured responses are in the
issue's notes, and every new test file was run against the service before
it was run here.

Semantic validation of expressions. After the grammar pass, the service
walks the parsed expression once, in expression order, checking each node
after its operands, then checks overlapping paths, then unused names and
values. A new `validate_update_semantics` and `validate_condition_semantics`
in the core crate follow that walk; the parsers now do grammar only, keep
function names as written, and record section order for the walk.

- UpdateExpression: `+` and `-` take numbers, `list_append` takes lists,
  `ADD` takes a number or a set, `DELETE` takes a set; a placeholder of the
  wrong type is refused with the service's type vocabulary (`S`, `N`, `L` for
  operators and functions; `STRING`, `NUMBER`, `Binary`, `BOOL`, `NULL`,
  `LIST`, `MAP` for ADD and DELETE). Function names are case-sensitive:
  `IF_NOT_EXISTS(x, :v)` and `foo(x)` are `Invalid function name`, a
  condition function is `not allowed in an update expression`,
  `list_append` is refused as an operand of a function or of `+`, both
  functions take two operands, the first operand of `if_not_exists` is a
  document path. A repeated section is reported after that section's own
  actions (`SET a = :v SET b = :q` reports `:q`). Previously `IF_NOT_EXISTS`,
  `DELETE s :string`, and `SET a = :string + :n` were accepted and written.
- ConditionExpression and FilterExpression: the six condition functions are
  matched as written (`ATTRIBUTE_EXISTS(pk)` was accepted and evaluated);
  `if_not_exists` and `list_append` are `not allowed in a condition
  expression`; a boolean function in operand position and `size` in
  condition position are `not allowed to be used this way`; operand counts
  are fixed per function; `attribute_exists` and `attribute_not_exists` take
  a document path; `begins_with` takes a string or binary and
  `attribute_type` a string naming a type, both checked at request time with
  the operation's prefix.
- Every unused name or value is listed in one set, sorted, so the message is
  the same on every backend; it named one key chosen by hash order.

Request stage on the read APIs. Query and Scan check the FilterExpression,
then the KeyConditionExpression (with its `Invalid KeyConditionExpression:`
prefix, and ahead of the key schema match, so an undefined `#n` is reported
as such), then the ProjectionExpression, then unused names and values, all
before the table is looked up. An undefined `:q` in a filter returned 200 on
an empty table. Limit and Segment checks precede the expressions on Scan.

Single-item order. UpdateItem checks the whole ConditionExpression before
the UpdateExpression. PutItem, UpdateItem, and DeleteItem wrap every
expression-stage error in `1 validation error detected: ` and omit the
`: ConditionExpression is null` suffix; the read, batch, and transaction
APIs return the messages bare. The ` expression size: N` suffix follows the
operation. Three existing dual-target tests asserted the unprefixed text and
fail against the service; they are corrected.

Transactions. TransactGetItems and TransactWriteItems validate in phases,
each over every item before the next: per item, expressions (UpdateExpression
before ConditionExpression), unused names and values, then the item's 400 KB
size; then table lookup; then the 4 MB total; then key schema, as a
cancellation with ValidationError on the first offending item only and None
on every other (it flagged every offender); then duplicates. An undefined
reference is the request error the service returns rather than a per-item
cancellation reason, and a key schema mismatch leaves other items'
conditions unevaluated. TransactWriteItems returns `"ItemCollectionMetrics":
{}` when SIZE is requested and no table has a local secondary index, and its
ConsumedCapacity entries are in first-appearance order rather than
per-process hash order.

Key-shaped messages. GetItem, DeleteItem, UpdateItem, BatchGetItem, and
BatchWriteItem (Put and Delete) refuse a key with a missing, mistyped, or
extra attribute with `The provided key element does not match the schema`;
PutItem keeps `Missing the key pk in the item` and `Type mismatch for key`.
BatchWriteItem returns `"ItemCollectionMetrics": {}` under SIZE with no LSI.

Table ARNs in `TableName` (eight #108 rows) stay unsupported by decision: an
ExtendDB table ARN names ExtendDB's account and region, so a table that
streams or replicates to Amazon DynamoDB would answer to two ARNs.
docs/differences-from-dynamodb.md records this.

Remaining #108 gaps after this change, all stated in the PR: the parser's
syntax wording (`Syntax error; token: X, near: Y`), the ResourceNotFound
message text, the 4 MB message (the service's carries its own payload size),
the service's rendering of over-limit list values, and the table ARN rows.

Testing: tests/test_transact_validation_order.py (31 cases),
tests/test_update_expression_rules.py (37), and
tests/test_expression_semantics.py (155) pass against the service and on
SQLite, PostgreSQL, and MongoDB; 164 of the 186 cases in the first and third
fail on main. The expression, validation, transaction, batch, query, and
scan suites pass on all three backends (767 on SQLite and PostgreSQL, 751 on
MongoDB with the four #387 cases unchanged). Of the 106 #108 rows, 83 now
return the service's status, error type, and message (30 on main). cargo
test --workspace 1,500 passed; fmt; clippy -D warnings on 1.97.0; cargo
+1.88.0 check.

Closes part of #108.
… three comments

The move of the FilterExpression operand-type checks into the semantic
pass dropped them for the legacy QueryFilter and ScanFilter parameters,
which bypass the parser. A ScanFilter with BEGINS_WITH and a number, or
LT and a boolean, was accepted at request time and refused only when a
row was evaluated, so an empty table returned 200. The desugared legacy
filter now runs the same begins_with and ordering operand checks as
before, with the FilterExpression prefix, before any row is read. Wire
tests cover both parameters on an empty table.

Comments reworded after review: the parsers state that grammar errors
are reported before semantic errors and why the semantic checks are
deferred; the empty operand list comment names the measured cases; the
PerTableCapacity doc states that first-appearance order is a firmer
guarantee than the service gives, with the measured orders.
…'s message

A ScanFilter or QueryFilter condition whose AttributeValueList holds a value of
a type its ComparisonOperator does not take is refused at request time with the
service's text, `One or more parameter values were invalid: ComparisonOperator
<op> is not valid for <type> AttributeValue type`, naming the first such value.
Measured against Amazon DynamoDB on 2026-10-23 on an empty table: BEGINS_WITH
with N, LT and GT with BOOL, and BETWEEN with L then N, which names the L. The
check runs in desugar_filter on each condition before it is desugared; the
`Invalid FilterExpression:` wording that the previous commit restored is gone.
attribute_type_code is exported from extenddb-core for the type tag.

Pin the condition-expression order for a syntax error after a semantic error:
`foo(pk) AND`, `ATTRIBUTE_EXISTS(pk) AND`, `begins_with(pk, :n) AND` and
`attribute_exists(#x) AND` each answer the syntax error on the service
(measured 2026-10-23), and the test asserts that no semantic message is
returned.

Document the order of ConsumedCapacity entries in transaction responses. With
two tables the service returned the same table first for both request orders
(measured 2026-10-23), so the order does not follow the request; ExtendDB keeps
first appearance in the request on every backend. No wire test asserts a
multi-table order.
… and messages

Measured the full ComparisonOperator by AttributeValue type grid against
Amazon DynamoDB on 2026-10-23 for ScanFilter, QueryFilter and KeyConditions,
with the value counts, mixed-type lists, inverted BETWEEN ranges and unknown
operator names. validate_legacy_condition in legacy_filter.rs now applies the
measured rules to every legacy condition before it is desugared and before the
table is looked up: the operator name (`Invalid comparison operator`), the
value count (`Invalid number of argument(s) for the <op> ComparisonOperator`),
the first value's type (EQ and NE take every type; BEGINS_WITH takes S and B;
CONTAINS and NOT_CONTAINS take S, N, B, BOOL and NULL; LE, LT, GE, GT, BETWEEN
and IN take S, N and B), that every value has the first value's type
(`AttributeValues inside AttributeValueList must be of same type`), and that a
BETWEEN lower bound is not above its upper bound. Each message carries the
`One or more parameter values were invalid: ` prefix. KeyConditions get the
same checks ahead of the table lookup, and a range key operator the service
does not index (NE, NULL, NOT_NULL, CONTAINS, NOT_CONTAINS, IN) is refused
with `Attempted conditional constraint is not an indexable operation`. The
engine's own count messages for these parameters are gone.

compare_orderable is exported from extenddb-core for the range check. The
grid is pinned in legacy_filter_tests.rs and one wire row per message in
TestLegacyFilterOperandType.

The ConsumedCapacity order note now cites the three-table measurement: 16
transactions in eight request orders all listed the tables in name order.
Amazon DynamoDB documents no order for the ConsumedCapacity entries of
TransactGetItems and TransactWriteItems. Measured on 2026-10-23 with three
single-hash-key tables whose names sort opposite to their creation order,
in three request orders: both operations listed the tables by creation
time, oldest first, whatever order the request named them in.

TableKeyInfo now carries the table's creation time. Every backend reads it
from the catalog row or document it already fetches for the key schema
(SQLite and PostgreSQL creation_date_time, MongoDB creation_date_time,
Cassandra created_at), so no request makes another catalog read.

PerTableCapacity moves from capacity_helpers.rs to per_table_capacity.rs,
takes the TableKeyInfo of each operation, and lists tables oldest first,
with name order breaking a tie on the same creation time. Both transaction
handlers use it, including the idempotent replay path of
TransactWriteItems, which now has a key_info accessor on PreparedOp.

tests/test_transact_capacity_order.py creates three tables whose names
sort opposite to their creation order, waiting for ACTIVE between them,
and asserts the order for both operations at TOTAL and INDEXES over the
three measured request orders; it runs against the service and against
ExtendDB. The row in docs/differences-from-dynamodb.md states the measured
order and ExtendDB's tie rule.
create_table wrote the catalog creation time as whole seconds, so two
tables created inside the same second carried the same value. The
transaction ConsumedCapacity order sorts tables by that value and falls
back to name order on a tie, which the service does not do, and
tests/test_transact_capacity_order.py failed 5 of 6 on this backend.

The write now keeps the millisecond part, which is the precision
bson::DateTime stores. Every reader already converts through
timestamp_millis, so DescribeTable, ListTables, the backup and restore
paths and the transaction key-info fetcher return the sub-second value,
the same as the PostgreSQL backend. Catalog rows written by earlier
builds keep whole seconds and can still tie.
The ConsumedCapacity order row said two tables created in the same
millisecond are listed in name order. The precision of the stored
creation time differs per backend, so the tie does too: nanoseconds on
SQLite, microseconds on PostgreSQL, milliseconds on MongoDB and
Cassandra, and whole seconds for a MongoDB catalog row written before
the precision fix. The row now says which, and names the backends the
order was measured on.

The service-side measurement no longer claims the three tables had a
single hash key, which the recorded run does not state. Same wording in
the test docstring and the PerTableCapacity doc comment.

The empty-operand-list comment in the condition parser now cites the
2026-10-23 measurement as well as the 2026-10-14 one.
semantics.rs had grown to 819 lines. It is now a module directory with
one file per rule family: update_rules.rs for the update-expression
rules, condition_rules.rs for the condition-expression rules, refs.rs
for the name and value usage checks both passes share, errors.rs for
the message builders, and tests.rs for the tests. mod.rs keeps the
module documentation, the two function-name tables and the two public
entry points.

No behaviour change. The functions moved unedited; the only edits are
the per-file imports, pub(super) on the helpers the siblings call, and
the test module's imports in place of its use super::*. The nine
semantic-pass tests keep their names and pass.
update_parser.rs had grown from 571 lines to 630. It is now a module
directory: mod.rs holds the parser at 417 lines and tests.rs holds the
21 tests at 215 lines.

No behaviour change. The test module moved unedited, keeping its
use super::*, and the tests keep their names. One comment inside it
lost an em dash.
The update-parser ADD/DELETE doc comment and the scan Segment check
comment carried an em dash. Both are now two clauses.
TransactWriteItems looked the tables up one item at a time and ran four
checks inside that loop, so the first item's defect hid a missing table on a
later item and the answer changed with the item order. The service reports
them in two places, measured with the items in each order.

Before any table is looked up, in the pass that also checks the expressions:
the nesting of every ExpressionAttributeValues value of the item, used or
not, on every kind of item (worded "ExpressionAttributeValues contains
invalid value: ... for key :v"), and for a Put the attribute name size, then
the item size, then the item nesting.

After every table is looked up and after the 4 MB total, in one pass over
the items in order: a key that does not fit the schema, then a key attribute
named in an Update (a ValidationError cancellation reason on that item, not
a top-level error), then an empty key value (a top-level
ValidationException). The first failing item decides. The empty value rule
is left out of the key shape check this pass uses, so a mistyped range key
is reported ahead of an empty hash key.

tests/test_transact_write_phase_order.py pins each placement with the items
in both orders, and the order of the checks inside one item.
check_key_condition_refs collected the references of a parsed
KeyConditionExpression into sets, sorted them, and named the smallest name,
then the smallest value. With two undefined references the message named an
alphabetical pick, and any name beat any value. The service names the first
reference in the text, names and values alike: "#b = :p AND #a > :q" names
#b, "#a = :p AND #b > :q" names #a, "pk = :q AND #n = :v" names :q, and
"pk = :z AND sk > :y" names :z.

parse_key_condition_expr now takes the request's expression maps and walks
the tokens in order once the grammar pass succeeds, so the order is the
text's. The separate check function and its call in Query are gone.

validate_refs_defined had no caller and documented an order it did not
implement (it sorted both lists); it is removed with its re-export.

tests/test_request_stage_checks.py pins the five expressions above on the
wire.
Four paths gave a text or an order the service does not.

A KeyConditionExpression clause with an identifier followed by "(" always
got "Invalid function name". The service answers by name. contains,
attribute_exists, attribute_not_exists and attribute_type get "Invalid
operator used in KeyConditionExpression" with no function name, on the sort
key and on the hash key. size gets "KeyConditionExpressions cannot contain
nested operations". Any other name, begins_with in another case included,
gets "Invalid function name; function: <name>". The text comes from
key_condition_function_refusal, next to the function name lists.

Scan answered "Value '0' at 'limit' ..." for a Limit under 1. The service
answers "Value at 'Limit' failed to satisfy constraint: Member must have value
greater than or equal to 1" for Scan as it does for Query, on an existing and
on a missing table, ahead of every expression error. Query already did.

An ADD with a string, map or boolean value, and a DELETE with a value that is
not a set, sent as legacy AttributeUpdates, went through the UpdateExpression
semantic pass and came back as "Invalid UpdateExpression: Incorrect operand
type ..." for a request that carried no expression. The service answers "1
validation error detected: One or more parameter values were invalid: ADD
action is not supported for the type S" and "... DELETE action with value is
not supported for the type N", ahead of the table lookup and key validation.
attribute_updates.rs checks the value types before the desugaring, in
attribute name order.

An empty ConditionExpression on a ConditionCheck was dropped as "no
condition" and then refused with a text the service does not use. The service
answers "Invalid ConditionExpression: The expression can not be empty;". Put,
Delete and Update still treat an empty string as no condition.

tests/test_request_stage_checks.py pins each of them on the wire, and unit
tests cover the function name texts and the value type texts.
Four comments described code that had moved or never worked that way.

scan.rs said Segment, Limit and Select are checked after the expressions.
Limit and the Segment range checks run first.

transact_get_items.rs said a ValidationError reason per offending item. Only
the first offending item carries one; every other item reports None.

storage-sqlite ddl.rs said the creation time column holds milliseconds,
while per_table_capacity.rs and the differences table said nanoseconds on
SQLite. CreateTable writes RFC 3339 with up to nanosecond digits, and the
value the transaction handlers order by is an f64 of epoch seconds, which
resolves about 0.24 microseconds. All three now say that.

The comments in transact_write_items.rs that called an empty or oversize key
an up-front top-level error went with the code they sat on in the commit that
moved the key checks.

validate_unused_attributes sorts the unused names and values so the message
is the same on every backend and process, and no test failed without the
sort. The new unit test builds fresh maps twenty times, each iterating in a
fresh random order, and expects {:a, :m, :z} and {#a, #m, #z}.
The first commit of this series checked every ExpressionAttributeValues
value of a transaction item against the item's nesting limit. That refused a
ConditionCheck value 32 wrappers deep, which the service accepts (an existing
test pins it). The service accepts 32 wrappers in an ExpressionAttributeValues
value and refuses 33, as a top-level ValidationException worded
"ExpressionAttributeValues contains invalid value: ... for key :v".
validate_expression_value_nesting_depth applies that threshold.

A value an Update stores with SET becomes an attribute of the item, so 32
wrappers pass that check and then break the item limit. The service answers a
TransactionCanceledException with a ValidationError reason "Nesting Levels
have exceeded supported limits" on every such item, after the lookup, the 4 MB
total, the key pass and the duplicate check. stored_value_nesting_reasons
reports it there. Before, the same value was refused as a top-level
ValidationException inside the table lookup loop, so a missing table on a later
item was hidden. The service also reports the other items' condition results
next to this reason; ExtendDB reports None for them, because no condition has
been evaluated at that point.

key_pass_failure now returns the whole list of reasons, so the handler no
longer builds it.

tests/test_transact_value_nesting.py pins the 31, 32 and 33 wrapper boundaries
for each place a value can sit and the position of the reason among the other
checks.
…essions

Scan checked that Segment and TotalSegments go together, and that Segment is
below TotalSegments, ahead of every expression check. On 2026-10-27 the
service answered an undefined filter value, an unused value and an undefined
projection name ahead of those three mismatches, and a missing table after
them. Only the range checks on Segment, TotalSegments and Limit come first.

scan.rs keeps the range checks where they were and moves the three pairing
checks between the unused names and values check and the table lookup. Their
texts are unchanged.

The Segment assertion in test_parameter_checks_come_before_the_expressions
expected the TotalSegments message ahead of an undefined filter value, which
fails against the service; it now expects the filter error. TestScanSegmentChecks
in tests/test_request_stage_checks.py pins the messages, the position after
the filter, the unused values and the projection, the position ahead of the
table lookup, and that Limit still comes first.
… reason cover

The Scan comment said Select stays after the existence check; only the Select
checks that need the index do, and validate_select_projection runs ahead of the
table lookup. The transact_write_helpers header cited a 500-line limit that both
files are over; the limit is 600. The doc comment on stored_value_nesting_reasons
read as if ExtendDB reported the other items' condition results beside the
nesting reason as the service does; it reports None for them, and the
differences document now carries the row (measured 2026-10-27).
@LeeroyHannigan
LeeroyHannigan force-pushed the fix/transact-validation-order branch from 17b805d to 1454cc9 Compare October 31, 2026 17:50
LeeroyHannigan added a commit that referenced this pull request Oct 31, 2026
…e service does

Legacy QueryFilter, ScanFilter and KeyConditions values are now held to the
service's rules that need the key schema, and to its number check, on every
backend (issue #477, stacked on #399):

- a legacy N value that is not a number answers the number message before
  the operator, count, type and range checks on that entry, inside NS, L
  and M too;
- a KeyConditions value whose type is not the key's AttributeDefinitions
  type answers "Condition parameter type does not match schema type" (the
  range key answered 500 before, the hash key 200);
- a non-indexable operator on any KeyConditions entry, the hash key
  included, is refused at request time;
- an empty string on a key through KeyConditions is refused with the shared
  key message, hash key first, after the schema checks;
- a QueryFilter entry on a key of the queried table or index is refused
  after every other Query check;
- BatchGetItem charges each absent key the minimum read and lists every
  requested table in ConsumedCapacity, in table name order.

Which entry the service names when several legacy entries are invalid is
not a function of the request; the differences document records it.
@navareamol navareamol added the v1 Needed for the 1.0 release label Nov 2, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

v1 Needed for the 1.0 release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants