stack-encrypt: a query's refusals are written separately for Rust callers and the Go binding, and have drifted once — give both one query preparation #1279

Background

Stack Encrypt (packages/stack-encrypt) is CipherStash's Rust encryption engine. Besides encrypting values, it derives terms: deterministic, one-way values that let a database search encrypted data. To search, a caller asks a query: they name an operator (op::Eq for equality, op::Lt / op::Between for ordering, op::Matches for text match) and a value. The engine picks the index that answers that operator (an HMAC for equality, ORE or OPE for ordering, a Bloom filter for match) and returns the query term to send to the database. ADR-0009 sets out this model.

There are two ways into a query:

• The typed path. A Rust program writes cipher.query(op::Eq, "bob@example.com").on(&plan). The types decide at compile time which index answers. Each index says what it accepts through the op::Supports trait.
• The binding path. Other languages reach the engine through stack_encrypt::dynamic. Today the only one is the Go SDK, which runs the engine as a WebAssembly guest. Here the operator arrives as a name ("eq", "between", "matches") and the value as data. The entry points are dynamic::check_query (refuse a bad query without any key), dynamic::query_term (derive the term), and dynamic::verify (after decrypting, drop rows a Bloom filter matched by chance).

ADR-0007 says a binding enters through a plan and never through a second executor: the binding path must behave exactly as the typed path does.

Problem

Before a query term exists, both paths refuse some queries:

1. match options the index cannot use (for example an n-gram length of zero);
2. match text the index cannot normalize, or text that cuts no token (its filter would match every row);
3. equality text the index cannot normalize (a code point unassigned in Unicode 16);
4. a between whose low bound is above its high bound.

These rules are not written once. Verified on next at 5887218c:

• Match text is mostly shared now. The typed admit, the binding's check_input and the binding's term derivation all call QueryTokens::of_query. But the binding's term derivation still calls it a second time, behind a Use::Query flag. So the write path carries a flag it never needs.
• The inverted range is written twice. The typed path compares the two plaintexts with >. The binding has its own comparator, integer_order, over the two frames (sign and 128-bit word).
• Equality normalization is refused at different times. The binding refuses it in check_input, before any keyset is resolved. The typed path's Supports<Eq>::check refuses nothing, so a typed query only fails when the term is derived, after the keyset is resolved.
• Supports has three overlapping methods: check, verifier and admit, where admit is check then verifier. The match impl's verifier swallows a failure with unwrap_or_default and relies on admit having run first. The query chain calls check before resolving the keyset and admit after, so a match query is tokenized twice.

What happens when a copy drifts. It already has. The last commit of #1215 (914c5168) fixed a panic: a plan built by hand with an n-gram length of zero passed check_query and verify, and crashed inside the tokenizer. The term derivation already refused those options; the binding's copy of the check did not. The shared of_query helper came from that fix. The range and equality rules have no shared helper yet.

Why nothing catches it. Agreement between the paths is held by doc comments ("the typed path's rule") and parity tests. Those tests compare terms, not refusals, over a handful of queries. No test feeds every refusal through both paths.

Proposal

Solution

Give every refusal a query can make before its term exists one home: a crate-private module beneath the operators. Both paths call it. The typed path keeps everything that only types can express, such as the match case check. The binding path calls the same functions with the options it read from the plan. A refusal is then written once, and check_query refuses exactly what query_term and the typed chain refuse.

Implementation decisions

1. New crate-private module in op (working name op::prepare). It has one function per index, and each takes its options as a value, not as a type parameter:
   • match: validate the options, normalize and tokenize the text, refuse empty text, and return the QueryTokens. This absorbs QueryTokens::of_query or wraps it.
   • range: refuse a low bound above the high bound. It is generic over PartialOrd, so the typed path passes its plaintexts.
   • equality: refuse text the index cannot normalize, before any keyset.
   • Everything else: nothing to refuse, nothing to keep.
     The rules about a query live in op. sem keeps the tokenizer and the term primitives.
2. Supports gets one method in place of check, verifier and admit: prepare(&self, input: &In) -> Result<Self::Verifier, Error>. A verifier can then only exist if the check passed, and unwrap_or_default goes away. Supports is new in this release (stack-encrypt 0.3.0 is unpublished; crates.io has 0.2.0), so this costs nothing outside the crate. The only implementation outside the crate is a trybuild UI test, which is updated.
3. The built-in Supports impls call op::prepare. Equality gains the normalization refusal, so a typed equality query is refused before the keyset is resolved, as the binding already is. The match impl keeps its case check (a Matches<C> whose case disagrees with the index's downcase) in front of the shared call. The binding has no case parameter, so it has nothing to refuse there.
4. The query chain calls prepare once. It runs before the keyset is resolved, and the verifier is kept for when the term is derived. The second call and the second tokenization go.
5. The binding calls the same functions. check_input becomes a dispatch from IndexSpec and the input to op::prepare. Its range arm keeps only the unpacking of the two integer bounds from their frames (any width, up to 128 bits) and passes them to the shared range function. verify takes its tokens from the shared match function.
6. The Use::Query branch in the binding's term derivation is removed. The derivation then only derives terms. Every caller that derives a query term has already prepared the input; the implementer confirms this for the term-by-index path used by se_term as well.
7. Error values do not change. Each refusal keeps its current TermError / PlanError variant and its stable diagnostic code, because the Go host matches on those codes.
8. Changelog. The unreleased op::Supports entries in stack-encrypt's CHANGELOG are edited in place to describe prepare. No breaking-change line is added, because no released version had check / verifier / admit.

User stories

1. As a Rust developer querying through the typed chain, I want an invalid query refused before any keyset is resolved, so that a bad query never costs a ZeroKMS round trip.
2. As a Rust developer, I want an equality query on text the index cannot normalize refused before the keyset is resolved, so that the typed path fails as early as the Go SDK does.
3. As a Rust developer, I want a between with its bounds the wrong way round refused with TermError::InvertedRange, so that I find my bug instead of getting an empty result.
4. As a Rust developer, I want a match query on text that cuts no token refused with TermError::EmptyTermText, so that I never send a filter that matches every row.
5. As a Rust developer with a hand-built plan, I want match options the index cannot use refused with TermError::InvalidOptions, so that a bad plan returns an error instead of panicking.
6. As a Rust developer, I want a case-sensitive Matches against a case-folding index still refused with PlanError::MatchCase, so that the compile-time and await-time guarantees of ADR-0009 hold.
7. As a Rust developer, I want my match query tokenized once, so that a query costs no more work than it needs.
8. As a Go developer using generated query methods, I want check_query to refuse exactly what query_term refuses, so that a query that passes validation cannot fail or panic when it runs.
9. As a Go developer, I want verify to refuse exactly what query_term refuses, so that verification behaves like the query it checks.
10. As a Go developer, I want every refusal to keep its current diagnostic code, so that my error handling and the Go SDK's Diagnostic checks keep working.
11. As a Go developer querying an integer field of any width up to 128 bits, I want an inverted range refused the same way the Rust typed path refuses it, so that both SDKs agree on what a valid range is.
12. As a maintainer of the binding path, I want no copy of a typed query rule in the binding, so that ADR-0007's "never a second executor" holds by construction.
13. As a maintainer adding a new refusal, I want to write it in one function, so that both paths get it at once.
14. As a maintainer adding a new index, I want one place to say what a query on it refuses and keeps, so that I do not have to find three call sites.
15. As a maintainer of an index that implements Supports, I want a single prepare method, so that I cannot return a verifier for an input I would have refused.
16. As a maintainer of the write path, I want term derivation to carry no query-only flag, so that storing a value cannot accidentally take a query rule.
17. As a reviewer, I want one table test that runs every refusal through both paths, so that drift between them fails CI instead of reaching a customer.
18. As a reviewer, I want the doc comments that say "the typed path's rule" replaced by calls to that rule, so that prose no longer has to keep code in step.
19. As a security reviewer, I want the shared module to keep handling plaintext the way today's code does (tokens in Zeroizing, no plaintext in errors), so that the refactor adds no leak.
20. As a Go SDK user, I want query term bytes unchanged by this refactor, so that rows written before it still match.

Testing decisions

• A good test here drives a public entry point and checks the error variant or the term bytes. It does not check which internal function ran.
• The highest seam is the public interface of each path: the typed query chain (cipher.query(op, value).on(&plan)) and the binding's dynamic::check_query, dynamic::query_term and dynamic::verify. These were agreed in the design discussion.
• One table test pins agreement. Each row is one refusal (invalid options, unnormalizable match text, empty match text, unnormalizable equality text, inverted range for several integer widths). Each row runs through the typed chain, check_query, query_term, and verify where it applies, and asserts the same TermError / PlanError variant from each. The table test needs no live keys: refusals happen before any key is requested. It uses the fake data key source the integration tests already share, and its load-counting variant lets each typed row also assert that no key was loaded.
• Interface tests on op::prepare, one per refusal and one per verifier it returns. This is an internal seam, private to the crate and used by its own tests.
• Remove tests that pin a single refusal on one path only, once the table covers that refusal. Keep tests that pin anything else (for example term bytes or wire forms).
• Term bytes must not move. The existing query_operators.json fixture (also read by the Go tests) must pass unchanged. So must the value_kinds.json fixture.
• Prior art: the refusal tests in the op integration tests (typed path) and in the dynamic::query unit tests (binding path); the trybuild UI tests for the Supports signature; the guest's native tests and leak scenarios for error paths that carry plaintext.
• Mutation testing: stack-encrypt is gated by cargo-mutants --in-diff in CI. Every surviving mutant in the new module needs a test, not an exclusion (see AGENTS.md, "Mutation testing").

Out of scope

• One operator catalog shared by the typed traits, OpSpec and the Go test fake (enginetest). This is a separate deepening, which can follow this one.
• Match sizing (target_limit) re-parsing term bytes with its own query_bits, and the missing expected_tokens prior for plans written as data.
• Typing the binding's control messages instead of parsing an untyped FfiValue tree (stack-encrypt: dynamic control messages (plan, options, query results) are parsed from an untyped FfiValue tree — give each its own type, and stop eql-bindings forcing dynamic #1252).
• Go SDK changes. No Go code should need to change, because the diagnostic codes and term bytes stay the same.
• Adding a case parameter to the binding's matches operator.
• New refusals. This moves the existing ones; it does not add any.

Relationship to other work

• Tracking: implement ADR-0009, operator-typed query terms for Stack Encrypt #1178 is the tracking issue for ADR-0009; this tidies the query half it built.
• feat(stack-encrypt): query a plan field by operator through the guest, with verify and target_limit as exports #1215 (merged into next) added the binding path, and its last commit fixed the drift described above.
• feat(stack-encrypt)!: hold a dynamic plaintext as its frame, not a value tree #1245 and feat(stack-encrypt,golang): give every Go integer width, dates, timestamps, decimals and 128-bit integers their own kinds #1246 (merged into next) moved query inputs to frames and added integer widths up to 128 bits. That is why the binding's range comparator works on sign and word.
• stack-encrypt: dynamic control messages (plan, options, query results) are parsed from an untyped FfiValue tree — give each its own type, and stop eql-bindings forcing dynamic #1252 types the binding's control messages. It touches the same module, but not the refusal rules.

Further notes

• The work targets next, the release-candidate branch for stack-encrypt 0.3.0 (chore: release candidate for EQL v4 and Stack Encrypt Rust and Go support #1229). Supports must change before 0.3.0 is published, because after that, changing it is a breaking change.
• Read packages/stack-encrypt/CONTEXT.md for the domain terms (query, query term, operator, index, plan, binding), and ADR-0007 and ADR-0009 before starting.