Skip to content

fix(query): refuse a key condition with no HASH equality the way the service does - #479

Open
LeeroyHannigan wants to merge 1 commit into
mainfrom
fix/427-lone-non-equality
Open

LeeroyHannigan wants to merge 1 commit into
mainfrom
fix/427-lone-non-equality

Conversation

@LeeroyHannigan

@LeeroyHannigan LeeroyHannigan commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #427.

Summary

A Query whose KeyConditionExpression has no equality clause on the HASH attribute was refused with a parser sentence of ExtendDB's own. The service chooses the sentence from the key schema, and this change does the same.

Measured on the service on 2026-10-23 (every row repeated, identical answers):

Shape Service sentence
Single HASH attribute, a clause names it (pk > :x, pk BETWEEN :x AND :y, begins_with(pk, :x), pk > :x AND sk = :y, sk = :x AND pk > :y, zz = :x AND pk > :y, GSI forms) Query key condition not supported
Single HASH attribute, no clause names it (sk > :x, zz > :x, sk > :x AND zz > :y, zz = :x AND yy = :y, GSI forms) Query condition missed key schema element: <HASH>
Two HASH attributes, or one HASH with several RANGE attributes, any operator, any attribute, including an N value on the HASH attribute HASH key attributes h1, h2 must have equality conditions specified in the query
Two conditions on one attribute Invalid KeyConditionExpression: KeyConditionExpressions must only contain one condition per key
Undefined #name or :value in the key condition (the rows include pk = :x with no values) Invalid KeyConditionExpression: An expression attribute name used in the document path is not defined; attribute name: #n (value form likewise), reported before any of the above

Ordering, measured: unused values, filter syntax, projection syntax, the Select conflicts and undefined key references come before this refusal; an ExclusiveStartKey fault and an oversized key value come after it.

Changes

  • crates/core/src/expression/key_condition/no_hash_equality.rs: KeyCondition::lacks_hash_equality and refuse_without_hash_equality, choosing the sentence from the key schema, and refuse_two_conditions_on_one_attribute and operands_naming, which the engine calls before its type check.
  • crates/core/src/expression/key_condition/references.rs: KeyCondition::validate_references, the undefined-reference refusal with the prefix.
  • The parser records one or two clauses with no equality instead of refusing them, so the engine can choose the sentence. When only the second of two such clauses is a begins_with, the parser keeps that clause whole as the sort key condition, which the engine's begins_with check reads.
  • handle_query skips fix(query): classify and order key conditions on a multi-attribute GSI the way the service does #429's classification for a key condition with no equality clause, and on a schema with one HASH attribute for an equality on another attribute paired with a non-equality on the HASH attribute (sk = :x AND pk > :y). Every other key condition goes through fix(query): classify and order key conditions on a multi-attribute GSI the way the service does #429's classification as on main. After validate_select_projection it calls refuse_without_hash_equality, which checks the references of every key condition and refuses the ones the classification skipped: two conditions on one attribute first, then, on a schema with one HASH attribute, fix(query): classify and order key conditions on a multi-attribute GSI the way the service does #429's type and begins_with checks on the clause that names the HASH attribute, then the key schema's wording.

Tests

  • Unit tests in no_hash_equality/tests.rs, references.rs and crates/engine/src/query/no_hash_equality_tests.rs.
  • tests/python/test_query_lone_non_equality.py, 83 rows. 67 of them passed against the service as a file; the other 16 repeat requests measured on 23 October (427-8c, row 6, and rows 5a and 5b with the N value) and on 1 November (the ten rows of TestValueTypes, two of them on the file's single-idx in place of the measured index, and the two begins_with rows on a table with HASH pk (N) and RANGE sk). All 83 pass on SQLite and PostgreSQL 16 builds of this head's tree, and a SQLite build of main at f49c5f82 fails 76 of them. Four rows assert a prefix or a suffix of the sentence, for the filter, projection and Select errors ExtendDB words differently.
  • Regression: the files and counts under "Rebase on main".

Rebase on main

Rebased on main at f49c5f82 (#429) on 31 October. #429 changed crates/engine/src/query.rs, and #429 and #511 (1c0523c9) both changed crates/core/src/expression/key_condition.rs and crates/server/src/leading_keys/tests.rs. Only query.rs conflicted. key_condition.rs merged without a conflict.

In handle_query, #429 replaced the partition key block with resolve_key_schema_with, which classifies the clauses and runs the key value type check, then refuse_begins_with_on_number. That block now skips a key condition with no equality clause, and, on a schema with one HASH attribute, an equality on another attribute paired with a non-equality on the HASH attribute (sk = :x AND pk > :y). Every other key condition goes through it as on main, including zz = :x AND yy = :y on the table and r = :x on the index with HASH h1, h2. A condition the block skips reaches the engine's refuse_without_hash_equality, a new function in query.rs called after validate_select_projection. It checks the references first and refuses two conditions on one attribute next. On a schema with one HASH attribute, when a clause names the HASH attribute of the key schema queried, it then runs #429's type check and begins_with check on that clause, against that attribute's definition. Then it refuses with the key schema's wording. On an index with several key attributes no value is checked, so h > :n with an N value answers the HASH sentence.

The service gave that order on 1 November, two samples each, on a table with HASH pk and RANGE sk (strings), on a table with HASH pk (N) and RANGE sk (S), and on an index with HASH gsi1_pk and RANGE SK of a table with HASH PK. :x and :y are strings, :n and :m numbers.

Request Service sentence
pk > :x AND pk < :n, pk > :n AND pk < :m Invalid KeyConditionExpression: KeyConditionExpressions must only contain one condition per key
pk > :n AND sk > :y, sk = :x AND pk > :n, pk BETWEEN :n AND :m One or more parameter values were invalid: Condition parameter type does not match schema type
pk > :x AND sk > :n Query key condition not supported
sk > :n, zz > :n Query condition missed key schema element: pk
On the index, PK > :n and SK > :n Query condition missed key schema element: gsi1_pk
On the table with HASH pk (N), begins_with(pk, :n) AND sk > :x and sk > :x AND begins_with(pk, :n) Invalid KeyConditionExpression: Incorrect operand type for operator or function; operator or function: begins_with, operand type: N

The type check runs in that function, after the unused value and filter checks, because of rows 5a and 5b (measured on 23 October and again on 31 October): pk > :n with an N value on the S key and an unused :y answers the unused value sentence, and with FilterExpression="nope(" it answers the filter syntax error. With no other fault, pk > :n answers the type sentence (row 427-8c).

#511 binds dynamodb:LeadingKeys to the operand of a single non-equality clause on the partition key. Two tests changed with it:

  • query_kce_without_a_partition_key_equality_leaves_the_key_undetermined asserted that no key is bound for pk > :v, begins_with(pk, :v), pk > :v AND sk > :s and sk = :v AND pk > :s. It is now query_kce_without_a_partition_key_equality_binds_only_a_partition_operand and asserts fix(auth): bind LeadingKeys for a refused Query's partition operand #511's values. For sk = :v AND pk > :s that is :s, the operand of the clause on pk. The parser records :v as the partition key value of that condition, and the test asserts that :v is never bound. For sk > :v and sk > :v AND zz > :s, where the parser copies :v into an empty partition key condition, no key is bound.
  • fix(auth): bind LeadingKeys for a refused Query's partition operand #511's the_parser_still_rejects_every_single_clause_shape_whose_operand_is_now_bound asserted that the parser refuses pk > :v, pk < :v, pk <= :v, pk >= :v and begins_with(pk, :v). The parser now records them, so the test is the_engine_still_refuses_every_shape_whose_operand_is_now_bound. It asserts that refuse_without_hash_equality in core answers Query key condition not supported for each, pk > :v AND sk = :s and pk > :v AND sk < :s included.

crates/engine/src/query/no_hash_equality_tests.rs is new. It calls the engine's refuse_without_hash_equality with attribute definitions for each row of the table above, row 427-8c, row 6 (begins_with(pk, :n) on an N key) and h > :n on the multi-attribute index. no_hash_equality.rs gains refuse_two_conditions_on_one_attribute and operands_naming, which that function calls.

tests/python/test_query_lone_non_equality.py has 83 rows. The 16 new ones are 427-8c, row 6, rows 5a and 5b with :n an N value, the ten rows of TestValueTypes for the table above, whose index rows run on the file's single-idx, and the two begins_with rows, which run on a table with HASH pk (N) and RANGE sk (S). The file's docstring had left 427-8c and row 6 for this merge.

The checks below ran on af51f014 (tree 0ded8b6b). cargo fmt --check, workspace clippy with -D warnings, cargo +1.97.0 check --workspace --all-targets and cargo +1.88.0 check --workspace pass. The --lib tests pass: core 738 (1 ignored), engine 191, server 238.

On SQLite and PostgreSQL 16 servers built from af51f014, with the same counts on each:

  • tests/python/test_query_lone_non_equality.py: 83 passed.
  • tests/test_gsi_multi_range_query.py: 63 passed, 1 skipped.
  • tests/test_fgac_leading_keys.py: 31 passed.
  • tests/test_query_scan.py, tests/test_multipart_gsi.py and tests/test_read_paging.py: 141 passed.
  • tests/python/test_query_scan.py: 27 passed.

On a SQLite server built from main at f49c5f82, the wire file fails 76 of its 83 rows. The seven that pass are the two legacy KeyConditions rows, four rows with an equality clause that main already answers with the service's sentence, and sk = :x AND pk > :n, which #429's classification answers with the type sentence on main.

CI passed all 35 checks on af51f0148.

Related behaviour

  • <>, NOT, OR, IN and contains in a key condition: the service says Invalid KeyConditionExpression: Invalid operator used in KeyConditionExpression; ExtendDB's text differs.
  • pk = :x AND z > :x (second clause on a non-key attribute): Query condition missed key schema element: sk on the service, 200 here.
  • Select=ALL_PROJECTED_ATTRIBUTES without an index: ExtendDB's text lacks the 1 validation error detected: prefix.
  • An undefined :value in a FilterExpression is refused up front by the service and only noticed at evaluation here.
  • begins_with(sk, :x) AND begins_with(pk, :n) on a table with HASH pk (N): the service answers the begins_with operand type sentence, as it does with the two clauses the other way round (measured 2026-11-01). This branch answers Query key condition not supported: of two begins_with clauses the parser keeps the first whole, and the engine reads the operator of that clause only.

They are in #489.

Overlap with other open PRs

#484 refuses a second condition on one key attribute (KeyConditionExpressions must only contain one condition per key) when it parses the key condition, for every key schema, which covers pk > :x AND pk < :y. It merges with this branch without a conflict. With both merged, handle_query refuses that shape at the parse, before the copy of the refusal in no_hash_equality.rs is reached.

…service does

Closes #427.

A Query whose KeyConditionExpression has no equality clause on the HASH
attribute was refused with a parser sentence of ExtendDB's own. The service
chooses the sentence from the key schema. Measured on 2026-10-23, every row
repeated with identical answers:

- Single HASH attribute, any clause names it (so the clause on it is a
  non-equality): "Query key condition not supported". Rows: pk > :x,
  pk BETWEEN :x AND :y, begins_with(pk, :x), pk > :x AND sk = :y,
  sk = :x AND pk > :y, zz = :x AND pk > :y, and the GSI forms.
- Single HASH attribute, no clause names it: "Query condition missed key
  schema element: <HASH>". Rows: sk > :x, zz > :x, sk > :x AND zz > :y,
  zz = :x AND yy = :y, and the GSI forms naming the index HASH attribute.
- Several key attributes (two HASH attributes, or one HASH with several
  RANGE attributes): "HASH key attributes h1, h2 must have equality
  conditions specified in the query", naming every HASH attribute, for
  every operator and for an N value on the HASH attribute.
- Two conditions on one attribute: "Invalid KeyConditionExpression:
  KeyConditionExpressions must only contain one condition per key".
- An undefined #name or :value in the key condition is refused first, with
  the "Invalid KeyConditionExpression: " prefix. The rows include pk = :x
  with no values.
- Single HASH attribute, a value of the wrong type: pk > :n with an N value
  on an S key answers "One or more parameter values were invalid: Condition
  parameter type does not match schema type", and begins_with(pk, :n) on an
  N key answers "Invalid KeyConditionExpression: Incorrect operand type for
  operator or function; operator or function: begins_with, operand type: N".
- Legacy KeyConditions is desugared with every reference defined and an
  equality clause, or refused there; no change.

Measured on 2026-11-01, two samples each, on a single HASH attribute
schema: two conditions on one attribute answer the one condition per key
sentence whatever their value types (pk > :x AND pk < :n). A wrong-typed
value is reported only in the clause on the HASH attribute:
pk > :n AND sk > :y, sk = :x AND pk > :n and pk BETWEEN :n AND :m answer
the type sentence, and pk > :x AND sk > :n answers "Query key condition not
supported". With no clause on the HASH attribute no value is checked:
sk > :n answers "Query condition missed key schema element: pk", and on an
index with HASH gsi1_pk and RANGE SK, PK > :n (PK the table's HASH
attribute) and SK > :n answer "Query condition missed key schema element:
gsi1_pk". On a table with HASH pk (N) and RANGE sk (S),
begins_with(pk, :n) AND sk > :x and sk > :x AND begins_with(pk, :n) answer
the begins_with operand type sentence, as begins_with(pk, :n) alone does.

Ordering, measured: unused values, filter syntax, projection syntax, the
Select conflicts and undefined key references are reported before this
refusal; an ExclusiveStartKey fault and an oversized key value after it.
An unused value and an invalid FilterExpression are also reported before
the type sentence for pk > :n (measured again on 2026-10-31).

The refusal is in crates/core/src/expression/key_condition/no_hash_equality.rs,
with KeyCondition::lacks_hash_equality, refuse_two_conditions_on_one_attribute,
operands_naming and refuse_without_hash_equality, and
KeyCondition::validate_references is in key_condition/references.rs. The
parser now records one or two clauses with no equality instead of refusing
them, so the engine can choose the sentence. When only the second of two
such clauses is a begins_with, the parser keeps that clause whole as the
sort key condition, which the begins_with check reads.

handle_query skips the classification from #429 for a key condition with no
equality clause, and on a single HASH attribute schema for an equality on
another attribute paired with a non-equality on the HASH attribute
(sk = :x AND pk > :y). Every other key condition goes through that
classification as before. After validate_select_projection, handle_query
calls the engine's refuse_without_hash_equality. It checks the references
of every key condition and refuses the ones the classification skipped. It
refuses two conditions on one attribute first. Then, on a single HASH
attribute schema and only when a clause names the HASH attribute, it runs
the type check and the begins_with check from #429 on that clause, against
that attribute's definition. Then it refuses with the key schema's wording.
On a multi-attribute key schema no value is checked, because the service
answers h > :n with the HASH sentence.

LeadingKeys keeps the binding from #511.
query_kce_without_a_partition_key_equality_binds_only_a_partition_operand
asserts the values it binds for the shapes the parser now records, and
the_engine_still_refuses_every_shape_whose_operand_is_now_bound asserts that
the core refusal answers "Query key condition not supported" for each shape
it binds.

Tests: unit tests in no_hash_equality/tests.rs, references.rs and
crates/engine/src/query/no_hash_equality_tests.rs. The wire file
tests/python/test_query_lone_non_equality.py has 83 rows. 67 of them passed
against the service as a file; the other 16 repeat requests measured on
2026-10-23 (427-8c, row 6, and rows 5a and 5b with the N value) and on
2026-11-01 (the ten rows of TestValueTypes, two of them on the file's
single-idx in place of the measured index, and the two begins_with rows on
a table with HASH pk (N) and RANGE sk). All 83 pass on SQLite and
PostgreSQL 16 builds of this commit, and a SQLite build of main at f49c5f8
fails 76 of them.

Related behaviour, measured: "<>", NOT, OR, IN and contains in
a key condition get "Invalid operator used in KeyConditionExpression" on the
service; pk = :x AND z > :x gets "missed key schema element: sk" on the
service and 200 here; the Select ALL_PROJECTED_ATTRIBUTES text lacks the
"1 validation error detected: " prefix here. A key condition with an
equality clause and a HASH attribute without one (sk = :x on a table,
g1 = :x on an index with HASH g1, g2) is refused by the key schema checks
handle_query runs before it parses the FilterExpression. The service
reports an unused value or an invalid FilterExpression first (measured
2026-10-31).
@LeeroyHannigan
LeeroyHannigan force-pushed the fix/427-lone-non-equality branch from 9fc4ba3 to af51f01 Compare November 1, 2026 11:27

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Query: a lone non-equality condition on the partition key gets the parser's message, not the service's

1 participant