Repository navigation
fix(query): refuse a key condition with no HASH equality the way the service does - #479
Open
LeeroyHannigan wants to merge 1 commit into
Open
LeeroyHannigan wants to merge 1 commit into
LeeroyHannigan wants to merge 1 commit into
Conversation
LeeroyHannigan
requested review from
amrith,
c33howard,
jcshepherd,
pdf-amzn,
robinnsc and
yesyayen
as code owners
October 24, 2026 00:39
This was referenced Oct 24, 2026
fix(query): classify and order key conditions on a multi-attribute GSI the way the service does
#429
Merged
LeeroyHannigan
force-pushed
the
fix/427-lone-non-equality
branch
from
November 1, 2026 10:22
f15581d to
9fc4ba3
Compare
…service does Closes #427. A Query whose KeyConditionExpression has no equality clause on the HASH attribute was refused with a parser sentence of ExtendDB's own. The service chooses the sentence from the key schema. Measured on 2026-10-23, every row repeated with identical answers: - Single HASH attribute, any clause names it (so the clause on it is a non-equality): "Query key condition not supported". Rows: pk > :x, pk BETWEEN :x AND :y, begins_with(pk, :x), pk > :x AND sk = :y, sk = :x AND pk > :y, zz = :x AND pk > :y, and the GSI forms. - Single HASH attribute, no clause names it: "Query condition missed key schema element: <HASH>". Rows: sk > :x, zz > :x, sk > :x AND zz > :y, zz = :x AND yy = :y, and the GSI forms naming the index HASH attribute. - Several key attributes (two HASH attributes, or one HASH with several RANGE attributes): "HASH key attributes h1, h2 must have equality conditions specified in the query", naming every HASH attribute, for every operator and for an N value on the HASH attribute. - Two conditions on one attribute: "Invalid KeyConditionExpression: KeyConditionExpressions must only contain one condition per key". - An undefined #name or :value in the key condition is refused first, with the "Invalid KeyConditionExpression: " prefix. The rows include pk = :x with no values. - Single HASH attribute, a value of the wrong type: pk > :n with an N value on an S key answers "One or more parameter values were invalid: Condition parameter type does not match schema type", and begins_with(pk, :n) on an N key answers "Invalid KeyConditionExpression: Incorrect operand type for operator or function; operator or function: begins_with, operand type: N". - Legacy KeyConditions is desugared with every reference defined and an equality clause, or refused there; no change. Measured on 2026-11-01, two samples each, on a single HASH attribute schema: two conditions on one attribute answer the one condition per key sentence whatever their value types (pk > :x AND pk < :n). A wrong-typed value is reported only in the clause on the HASH attribute: pk > :n AND sk > :y, sk = :x AND pk > :n and pk BETWEEN :n AND :m answer the type sentence, and pk > :x AND sk > :n answers "Query key condition not supported". With no clause on the HASH attribute no value is checked: sk > :n answers "Query condition missed key schema element: pk", and on an index with HASH gsi1_pk and RANGE SK, PK > :n (PK the table's HASH attribute) and SK > :n answer "Query condition missed key schema element: gsi1_pk". On a table with HASH pk (N) and RANGE sk (S), begins_with(pk, :n) AND sk > :x and sk > :x AND begins_with(pk, :n) answer the begins_with operand type sentence, as begins_with(pk, :n) alone does. Ordering, measured: unused values, filter syntax, projection syntax, the Select conflicts and undefined key references are reported before this refusal; an ExclusiveStartKey fault and an oversized key value after it. An unused value and an invalid FilterExpression are also reported before the type sentence for pk > :n (measured again on 2026-10-31). The refusal is in crates/core/src/expression/key_condition/no_hash_equality.rs, with KeyCondition::lacks_hash_equality, refuse_two_conditions_on_one_attribute, operands_naming and refuse_without_hash_equality, and KeyCondition::validate_references is in key_condition/references.rs. The parser now records one or two clauses with no equality instead of refusing them, so the engine can choose the sentence. When only the second of two such clauses is a begins_with, the parser keeps that clause whole as the sort key condition, which the begins_with check reads. handle_query skips the classification from #429 for a key condition with no equality clause, and on a single HASH attribute schema for an equality on another attribute paired with a non-equality on the HASH attribute (sk = :x AND pk > :y). Every other key condition goes through that classification as before. After validate_select_projection, handle_query calls the engine's refuse_without_hash_equality. It checks the references of every key condition and refuses the ones the classification skipped. It refuses two conditions on one attribute first. Then, on a single HASH attribute schema and only when a clause names the HASH attribute, it runs the type check and the begins_with check from #429 on that clause, against that attribute's definition. Then it refuses with the key schema's wording. On a multi-attribute key schema no value is checked, because the service answers h > :n with the HASH sentence. LeadingKeys keeps the binding from #511. query_kce_without_a_partition_key_equality_binds_only_a_partition_operand asserts the values it binds for the shapes the parser now records, and the_engine_still_refuses_every_shape_whose_operand_is_now_bound asserts that the core refusal answers "Query key condition not supported" for each shape it binds. Tests: unit tests in no_hash_equality/tests.rs, references.rs and crates/engine/src/query/no_hash_equality_tests.rs. The wire file tests/python/test_query_lone_non_equality.py has 83 rows. 67 of them passed against the service as a file; the other 16 repeat requests measured on 2026-10-23 (427-8c, row 6, and rows 5a and 5b with the N value) and on 2026-11-01 (the ten rows of TestValueTypes, two of them on the file's single-idx in place of the measured index, and the two begins_with rows on a table with HASH pk (N) and RANGE sk). All 83 pass on SQLite and PostgreSQL 16 builds of this commit, and a SQLite build of main at f49c5f8 fails 76 of them. Related behaviour, measured: "<>", NOT, OR, IN and contains in a key condition get "Invalid operator used in KeyConditionExpression" on the service; pk = :x AND z > :x gets "missed key schema element: sk" on the service and 200 here; the Select ALL_PROJECTED_ATTRIBUTES text lacks the "1 validation error detected: " prefix here. A key condition with an equality clause and a HASH attribute without one (sk = :x on a table, g1 = :x on an index with HASH g1, g2) is refused by the key schema checks handle_query runs before it parses the FilterExpression. The service reports an unused value or an invalid FilterExpression first (measured 2026-10-31).
LeeroyHannigan
force-pushed
the
fix/427-lone-non-equality
branch
from
November 1, 2026 11:27
9fc4ba3 to
af51f01
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #427.
Summary
A Query whose KeyConditionExpression has no equality clause on the HASH attribute was refused with a parser sentence of ExtendDB's own. The service chooses the sentence from the key schema, and this change does the same.
Measured on the service on 2026-10-23 (every row repeated, identical answers):
pk > :x,pk BETWEEN :x AND :y,begins_with(pk, :x),pk > :x AND sk = :y,sk = :x AND pk > :y,zz = :x AND pk > :y, GSI forms)Query key condition not supportedsk > :x,zz > :x,sk > :x AND zz > :y,zz = :x AND yy = :y, GSI forms)Query condition missed key schema element: <HASH>HASH key attributes h1, h2 must have equality conditions specified in the queryInvalid KeyConditionExpression: KeyConditionExpressions must only contain one condition per key#nameor:valuein the key condition (the rows includepk = :xwith no values)Invalid KeyConditionExpression: An expression attribute name used in the document path is not defined; attribute name: #n(value form likewise), reported before any of the aboveOrdering, measured: unused values, filter syntax, projection syntax, the Select conflicts and undefined key references come before this refusal; an ExclusiveStartKey fault and an oversized key value come after it.
Changes
crates/core/src/expression/key_condition/no_hash_equality.rs:KeyCondition::lacks_hash_equalityandrefuse_without_hash_equality, choosing the sentence from the key schema, andrefuse_two_conditions_on_one_attributeandoperands_naming, which the engine calls before its type check.crates/core/src/expression/key_condition/references.rs:KeyCondition::validate_references, the undefined-reference refusal with the prefix.begins_with, the parser keeps that clause whole as the sort key condition, which the engine'sbegins_withcheck reads.handle_queryskips fix(query): classify and order key conditions on a multi-attribute GSI the way the service does #429's classification for a key condition with no equality clause, and on a schema with one HASH attribute for an equality on another attribute paired with a non-equality on the HASH attribute (sk = :x AND pk > :y). Every other key condition goes through fix(query): classify and order key conditions on a multi-attribute GSI the way the service does #429's classification as on main. Aftervalidate_select_projectionit callsrefuse_without_hash_equality, which checks the references of every key condition and refuses the ones the classification skipped: two conditions on one attribute first, then, on a schema with one HASH attribute, fix(query): classify and order key conditions on a multi-attribute GSI the way the service does #429's type andbegins_withchecks on the clause that names the HASH attribute, then the key schema's wording.Tests
no_hash_equality/tests.rs,references.rsandcrates/engine/src/query/no_hash_equality_tests.rs.tests/python/test_query_lone_non_equality.py, 83 rows. 67 of them passed against the service as a file; the other 16 repeat requests measured on 23 October (427-8c, row 6, and rows 5a and 5b with the N value) and on 1 November (the ten rows ofTestValueTypes, two of them on the file'ssingle-idxin place of the measured index, and the twobegins_withrows on a table with HASHpk(N) and RANGEsk). All 83 pass on SQLite and PostgreSQL 16 builds of this head's tree, and a SQLite build of main atf49c5f82fails 76 of them. Four rows assert a prefix or a suffix of the sentence, for the filter, projection andSelecterrors ExtendDB words differently.Rebase on main
Rebased on main at
f49c5f82(#429) on 31 October. #429 changedcrates/engine/src/query.rs, and #429 and #511 (1c0523c9) both changedcrates/core/src/expression/key_condition.rsandcrates/server/src/leading_keys/tests.rs. Onlyquery.rsconflicted.key_condition.rsmerged without a conflict.In
handle_query, #429 replaced the partition key block withresolve_key_schema_with, which classifies the clauses and runs the key value type check, thenrefuse_begins_with_on_number. That block now skips a key condition with no equality clause, and, on a schema with one HASH attribute, an equality on another attribute paired with a non-equality on the HASH attribute (sk = :x AND pk > :y). Every other key condition goes through it as on main, includingzz = :x AND yy = :yon the table andr = :xon the index with HASHh1,h2. A condition the block skips reaches the engine'srefuse_without_hash_equality, a new function inquery.rscalled aftervalidate_select_projection. It checks the references first and refuses two conditions on one attribute next. On a schema with one HASH attribute, when a clause names the HASH attribute of the key schema queried, it then runs #429's type check andbegins_withcheck on that clause, against that attribute's definition. Then it refuses with the key schema's wording. On an index with several key attributes no value is checked, soh > :nwith an N value answers the HASH sentence.The service gave that order on 1 November, two samples each, on a table with HASH
pkand RANGEsk(strings), on a table with HASHpk(N) and RANGEsk(S), and on an index with HASHgsi1_pkand RANGESKof a table with HASHPK.:xand:yare strings,:nand:mnumbers.pk > :x AND pk < :n,pk > :n AND pk < :mInvalid KeyConditionExpression: KeyConditionExpressions must only contain one condition per keypk > :n AND sk > :y,sk = :x AND pk > :n,pk BETWEEN :n AND :mOne or more parameter values were invalid: Condition parameter type does not match schema typepk > :x AND sk > :nQuery key condition not supportedsk > :n,zz > :nQuery condition missed key schema element: pkPK > :nandSK > :nQuery condition missed key schema element: gsi1_pkpk(N),begins_with(pk, :n) AND sk > :xandsk > :x AND begins_with(pk, :n)Invalid KeyConditionExpression: Incorrect operand type for operator or function; operator or function: begins_with, operand type: NThe type check runs in that function, after the unused value and filter checks, because of rows 5a and 5b (measured on 23 October and again on 31 October):
pk > :nwith an N value on the S key and an unused:yanswers the unused value sentence, and withFilterExpression="nope("it answers the filter syntax error. With no other fault,pk > :nanswers the type sentence (row 427-8c).#511 binds
dynamodb:LeadingKeysto the operand of a single non-equality clause on the partition key. Two tests changed with it:query_kce_without_a_partition_key_equality_leaves_the_key_undeterminedasserted that no key is bound forpk > :v,begins_with(pk, :v),pk > :v AND sk > :sandsk = :v AND pk > :s. It is nowquery_kce_without_a_partition_key_equality_binds_only_a_partition_operandand asserts fix(auth): bind LeadingKeys for a refused Query's partition operand #511's values. Forsk = :v AND pk > :sthat is:s, the operand of the clause onpk. The parser records:vas the partition key value of that condition, and the test asserts that:vis never bound. Forsk > :vandsk > :v AND zz > :s, where the parser copies:vinto an empty partition key condition, no key is bound.the_parser_still_rejects_every_single_clause_shape_whose_operand_is_now_boundasserted that the parser refusespk > :v,pk < :v,pk <= :v,pk >= :vandbegins_with(pk, :v). The parser now records them, so the test isthe_engine_still_refuses_every_shape_whose_operand_is_now_bound. It asserts thatrefuse_without_hash_equalityin core answersQuery key condition not supportedfor each,pk > :v AND sk = :sandpk > :v AND sk < :sincluded.crates/engine/src/query/no_hash_equality_tests.rsis new. It calls the engine'srefuse_without_hash_equalitywith attribute definitions for each row of the table above, row 427-8c, row 6 (begins_with(pk, :n)on an N key) andh > :non the multi-attribute index.no_hash_equality.rsgainsrefuse_two_conditions_on_one_attributeandoperands_naming, which that function calls.tests/python/test_query_lone_non_equality.pyhas 83 rows. The 16 new ones are 427-8c, row 6, rows 5a and 5b with:nan N value, the ten rows ofTestValueTypesfor the table above, whose index rows run on the file'ssingle-idx, and the twobegins_withrows, which run on a table with HASHpk(N) and RANGEsk(S). The file's docstring had left 427-8c and row 6 for this merge.The checks below ran on
af51f014(tree0ded8b6b).cargo fmt --check, workspace clippy with-D warnings,cargo +1.97.0 check --workspace --all-targetsandcargo +1.88.0 check --workspacepass. The--libtests pass: core 738 (1 ignored), engine 191, server 238.On SQLite and PostgreSQL 16 servers built from
af51f014, with the same counts on each:tests/python/test_query_lone_non_equality.py: 83 passed.tests/test_gsi_multi_range_query.py: 63 passed, 1 skipped.tests/test_fgac_leading_keys.py: 31 passed.tests/test_query_scan.py,tests/test_multipart_gsi.pyandtests/test_read_paging.py: 141 passed.tests/python/test_query_scan.py: 27 passed.On a SQLite server built from main at
f49c5f82, the wire file fails 76 of its 83 rows. The seven that pass are the two legacyKeyConditionsrows, four rows with an equality clause that main already answers with the service's sentence, andsk = :x AND pk > :n, which #429's classification answers with the type sentence on main.CI passed all 35 checks on
af51f0148.Related behaviour
<>,NOT,OR,INandcontainsin a key condition: the service saysInvalid KeyConditionExpression: Invalid operator used in KeyConditionExpression; ExtendDB's text differs.pk = :x AND z > :x(second clause on a non-key attribute):Query condition missed key schema element: skon the service, 200 here.Select=ALL_PROJECTED_ATTRIBUTESwithout an index: ExtendDB's text lacks the1 validation error detected:prefix.:valuein a FilterExpression is refused up front by the service and only noticed at evaluation here.begins_with(sk, :x) AND begins_with(pk, :n)on a table with HASHpk(N): the service answers thebegins_withoperand type sentence, as it does with the two clauses the other way round (measured 2026-11-01). This branch answersQuery key condition not supported: of twobegins_withclauses the parser keeps the first whole, and the engine reads the operator of that clause only.They are in #489.
sk = :xon the table,g1 = :xon an index with HASHg1,g2): the service reports an unused value or an invalidFilterExpressionbefore the missing key sentence or the HASH sentence; ExtendDB reports that sentence first, from the key schema checkshandle_queryruns before it parses theFilterExpression. Filed as Query: the missing key refusal comes before the filter syntax and unused value checks when the key condition has an equality #712.#namein a key condition with an equality clause: the service answers#p = :xwith noExpressionAttributeNameswithInvalid KeyConditionExpression: An expression attribute name used in the document path is not defined; attribute name: #p(measured 2026-11-01). By the code, the key schema checks run beforerefuse_without_hash_equalitychecks the references, so ExtendDB answersQuery condition missed key schema element: pk, on this branch and on main. Added to Query: the missing key refusal comes before the filter syntax and unused value checks when the key condition has an equality #712.Overlap with other open PRs
#484 refuses a second condition on one key attribute (
KeyConditionExpressions must only contain one condition per key) when it parses the key condition, for every key schema, which coverspk > :x AND pk < :y. It merges with this branch without a conflict. With both merged,handle_queryrefuses that shape at the parse, before the copy of the refusal inno_hash_equality.rsis reached.