Skip to content

feat(eql): give EQL v4 range query domains and eql_v4.between - #1243

Draft
coderdan wants to merge 4 commits into
feat/eql-v4-equality-typesfrom
feat/eql-v4-range-domains
Draft

feat(eql): give EQL v4 range query domains and eql_v4.between#1243
coderdan wants to merge 4 commits into
feat/eql-v4-equality-typesfrom
feat/eql-v4-range-domains

Conversation

@coderdan

@coderdan coderdan commented Oct 17, 2026

Copy link
Copy Markdown
Contributor

Summary

EQL v4 can now take a BETWEEN query as one operand. EQL is the SQL library we install into a customer's PostgreSQL database to store and search encrypted values. EQL v4 is the copy of it that Stack Encrypt (our Rust engine) writes into.

ADR-0009 decision 7 makes a range query one term carrying both bounds, so a caller can't pair a low bound from one column with a high bound from another. Until now there was nothing in SQL to bind that term to. This PR adds a range query domain for each ordered family and ordering scheme, and an inlinable eql_v4.between(column, range) that uses the column's order-term index.

Stacked on #1241.

Changes

SQL (EQL v4 only)

  • Range query domains: eql_v4.query_<family>_range_ore and eql_v4.query_<family>_range_ope, one per (family, ordering term), as decision 8 says. That's 18 in total, over 9 families.
    • Wire shape: {"v":4,"i":{…},"lo":<bound>,"hi":<bound>}. Each bound is the value the single-bound domain stores under ob (ORE, an array of hex blocks) or op (OPE, a hex string).
    • The CHECK requires v, i, lo and hi, forbids c, and checks each bound's shape.
  • eql_v4.between(column, range), inclusive at both ends. It is LANGUAGE sql, IMMUTABLE, not STRICT and has no SET clause, so the planner inlines it and uses the functional index on the order term.
    • There are 29 overloads, one per stored ordered domain: _ord_ore and _search_ore take the ORE range, while _ord, _ord_ope and _search take the OPE range.
    • eql_v4.lo(r) and eql_v4.hi(r) read a bound with the term's own constructor.
  • The range parameter must be cast to its exact range domain ($1::eql_v4.query_integer_range_ore). An uncast $1, a jsonb parameter, or a range of another family or term fails with 42725 "function … is not unique". The single-bound operators work differently: they accept an untyped $1. The function reference, the SQL comments and both READMEs say this.
  • between has to be written schema-qualified, because BETWEEN is an SQL keyword.

Keeping v3 unchanged

  • One source tree renders both EQL versions. A new file directive, -- ONLY EQL VERSION: 4, confines the range files to v4:
    • eql-codegen select <3|4> filters the files for each version's build.
    • The renderer refuses to render a confined file as the other version.
    • The installer-completeness and Doxygen scripts honour the directive.
  • The v3 installer and both uninstallers are byte-identical to the base branch. No line of the v4 installer is lost; adding the range files only shifts where a few existing blocks sit in topological order.
  • docs:validate:documented-sql now also validates the v4-only files.

Bindings

  • Rust: 18 …RangeOreQuery and …RangeOpeQuery structs in eql_bindings::v4, listed by v4::all_range_query(). They don't derive Stack Encrypt's traits, because no ordered type is producible yet.
  • Go: 18 []byte types in encrypt/eql/eql_gen.go, with Value, Scan and JSON methods. They are not in the Types table.

Docs and release notes

  • New "Range queries in EQL v4" section in docs/reference/eql-functions.md.
  • packages/eql/AGENTS.md, the eql-bindings and Go READMEs and CHANGELOGs, and .changeset/eql-v4-bindings.md. The changeset says the v4 bundle reaches users as the eql-v4-install.sql attached to each Go release, not through npm or the crate.

Verification

  • mise run build, with the v3 installer and uninstallers compared byte for byte against the base branch: identical.
  • mise run test:encryption:postgres against a disposable postgres:17: text_eq 1, v4_types 3 and v4_range 6 tests passed. The v4_range tests cover:
    • Integer ORE and both OPE column domains: rows inside, outside and exactly at both bounds; across zero; a one-value range; an empty range; the full i32 range; an inverted range; a NULL range.
    • Text ORE and OPE.
    • EXPLAIN: an index scan with both bounds in the Index Cond, including for text.
    • The domain CHECKs.
    • A catalog-driven smoke test over all 9 ordered families and all 29 between overloads.
    • The exact-cast rule, pinned with prepared statements.
  • Inlining: when one between overload was made STRICT by hand, the EXPLAIN test failed, so the test does catch lost inlining. The reviewer also confirmed an Index Scan under a generic plan with a real $1.
  • mise run test:crates, test:encryption, test:codegen, codegen:parity, types:check and typescript:check all passed.
  • The docs validators passed: required tags, 100% coverage, documented-sql (286 files, including the 18 range files) and doc anchors.
  • The other EQL gates passed: installer-complete, symbol order, self-contained v3, v4 isolation, schemas parity, public identifiers, and the matrix inventory and coverage. eql_v4.lints() returns no rows.
  • pnpm test:scripts: 1362 passed.
  • Go: all guests rebuilt, go vet ./... and go test ./... -count=1 passed, and go generate ./... leaves no diff. mise run wasm:guest:test: 71 of 71.
  • Not run: the SQLx suite (test:sqlx). It needs CipherStash credentials to generate fixtures through ZeroKMS. The database tests live in packages/eql/tests/encryption instead, which installs the v4 bundle, whereas SQLx installs v3.

Where the test order terms come from. The ordered EQL types aren't producible yet (#1236, #1132). So the tests take order terms from cipherstash-client's OreIndexer and OpeIndexer over the fake key source's index key, the same pattern as ste_vec_selectors.rs. Stack Encrypt is untouched, and nothing in this PR derives order terms.

Related

Closes #1237

Review notes

  • I had an independent review run on the first commit. It found nothing blocking; its accuracy, coverage and validator findings are the three follow-up commits.
  • Follow-ups, deliberately not in this PR:
    • Range fields in the Go Types table and the Rust Target table, so stashgen can map an ordered type to its range type. This waits until Between is producible.
    • Stack Encrypt building the {v, i, lo, hi} envelope for a range on an EQL target. query_term returns Error::Internal for that case today. Needed once the ordered types are producible.
    • ADR-0009 says a range's ORE bounds will be "left ciphertexts" once block ORE lands. But EQL's compare_ore_block_256_term reads the right half from the query side, so query bounds need full ciphertexts. I've noted this on stashgen refuses TextOrdOre and TextSearchOre — EQL stores block ORE terms and the engine derives CLLW ORE #1132, where it needs settling before block ORE is built.

https://claude.ai/code/session_01BZhkvKLpNtKuzj3HCfMXoL

@coderdan
feat(eql): give EQL v4 range query domains and eql_v4.between
9288443 
ADR-0009 decision 7 makes a Between query one term holding both bounds,
and asks that it stay one operand in SQL. EQL v4 had no domain to carry
it and no function to apply it, so a Between term had nothing to bind to
but two comparisons, which loses the one-term guarantee (#1237).

Every ordered family now has one range query domain per order term,
beside its single-bound ones (decision 8): eql_v4.query_integer_range_ore
beside query_integer_ord_ore, query_integer_range_ope beside
query_integer_ord_ope, and the same for smallint, bigint, date,
timestamp, numeric, text, real and double. A range is
{"v": 4, "i": ..., "lo": <term>, "hi": <term>}: each bound is the value
the single-bound domain holds under ob or op, which is Stack Encrypt's
RangeQuery { lo, hi } with nothing added. The CHECK requires both bounds
in their term's shape and forbids c.

eql_v4.between(column, range) has an overload for every ordered column
domain against the range of the term it stores, and is
ord_term(a) >= lo(r) AND ord_term(a) <= hi(r): LANGUAGE sql, IMMUTABLE,
not STRICT, no SET clause, so PostgreSQL inlines it and a btree on the
column's order term serves it as one range scan. lo and hi read a bound
with the term's own constructor, as a single-bound term is read.

All of it is generated from the catalog (Term::range_suffix,
DomainFamily::range_terms / range_columns, RANGE_BOUND_KEYS) by a new
eql-codegen range module, as are the bindings: a v4 struct per range
domain (IntegerRangeOreQuery, ..., v4::all_range_query()) and a Go type
of the same name in encrypt/eql, so a range binds as one parameter.

EQL v3 does not change. One source tree renders both versions, so the
range files carry a new directive, `-- ONLY EQL VERSION: 4`. The build
selects each version's files with `eql-codegen select` before rendering,
`render` refuses to render a confined file as the other version, the
installer completeness gate filters its disk side with its own grep, the
symbol-order checker now runs on the v4 order too, and the Doxygen
filter reads a confined file from its version's rendered tree. The v3
installer and both uninstallers are byte-identical to the base; the v4
installer only gains lines.

The ordered EQL types are not producible yet (#1132, #1236), so the
PostgreSQL tests (tests/encryption/tests/v4_range.rs, run by
test:encryption:postgres) take their order terms from cipherstash-client's
ORE and OPE indexers over FakeDataKeySource's index key, and the rest of
each value from a real Stack Encrypt Integer or TextEq. They cover rows
inside, outside and at both bounds for ORE and OPE over integers and
text, inverted and NULL ranges, a range of the wrong term, the CHECKs,
and an EXPLAIN showing the order-term index serving between with both
bounds in its index condition.

Claude-Session: https://claude.ai/code/session_01BZhkvKLpNtKuzj3HCfMXoL
@coderdan
docs(eql): say a range must be cast to its exact domain, and what its…
2967004 
… envelope is

The docs said a range of the wrong term fails "when the query is planned".
It fails when the statement is parsed, with 42725 ("function
eql_v4.between(...) is not unique"), and so do an uncast $1, a jsonb
parameter and a range of the wrong family: PostgreSQL resolves a
function's domain arguments as their base type, jsonb, which every
between overload takes. The single-bound operators resolve an untyped
$1; between does not. The function reference, the generated SQL doc
comments, the eql-bindings README and changelog, the Go README and
changelog and the changeset now say the range parameter must be cast to
its exact range domain. The 42725 behaviour stays; there is no
catch-all overload.

They also said a range is Stack Encrypt's RangeQuery { lo, hi } "with
nothing added". It is not: RangeQuery has no serialized form,
Operand::Range carries raw bytes, the wire adds v and i, and query_term
refuses a range on an EQL target today. Every description now says each
bound is the single-bound domain's ob (ORE) or op (OPE) value, in a
{v, i, lo, hi} envelope, which the engine will need to build when the
ordered types become producible.

The changeset no longer implies the npm package or the crate ships the
v4 SQL: the v4 bundle reaches users as the eql-v4-install.sql attached
to each Go release.

Claude-Session: https://claude.ai/code/session_01BZhkvKLpNtKuzj3HCfMXoL
@coderdan
test(eql): smoke-test between over every ordered family, and pin the …
55b95db 
…exact cast

postgres_between_smoke_test_over_every_ordered_family loops over the
catalog: for each of the nine ordered families it stores five ascending
values in every column domain that carries an ordering term, and checks
that a range from the second value to the fourth finds the rows at both
bounds and the one inside, and neither row outside, through every one of
the 29 between overloads (the _search and _search_ore ones included).
For text it also checks that a btree on the order term serves between,
for ORE on _ord_ore and for OPE on _search. Every family gets its terms
from cipherstash-client's indexers, as the other range tests do; c, hm
and bf are real Stack Encrypt values sealed from the value's text, since
between never reads them and most of these families have no producible
type.

postgres_between_needs_the_exact_range_domain prepares between with an
uncast $1, a jsonb parameter, a range of another family and one of
another term, and pins 42725 for each, beside the exact cast and the
single-bound operator, which both prepare.

The cipherstash-client comment in the test crate's manifest now names
v4_range.rs, and chrono and rust_decimal (already in the lockfile) name
the date, timestamp and numeric plaintexts.

Claude-Session: https://claude.ai/code/session_01BZhkvKLpNtKuzj3HCfMXoL
@coderdan
build(eql): validate the v4-only range files in docs:validate:documen…
308ee9a 
…ted-sql

The task ran psql over build/render/eql_v3/src only, so the 18 files
confined to EQL v4 (the range domains and between) were never validated:
they are not rendered into the v3 tree. It now also validates every file
of build/render/eql_v4/src with no counterpart in the v3 tree, and fails
if it finds none. The other v4 files are the v3 ones under another
schema name and are not run twice. The AGENTS paragraph on confined
files says so.

Claude-Session: https://claude.ai/code/session_01BZhkvKLpNtKuzj3HCfMXoL
@changeset-bot

changeset-bot Bot commented Oct 17, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 308ee9a

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes changesets to release 12 packages
Name Type
@cipherstash/eql Minor
stash Patch
@cipherstash/stack-prisma Patch
@cipherstash/basic-example Patch
@cipherstash/e2e Patch
@cipherstash/prisma-example Patch
@cipherstash/stack-drizzle Patch
@cipherstash/stack-supabase Patch
@cipherstash/stack Patch
@cipherstash/wizard Patch
@cipherstash/bench Patch
@cipherstash/test-kit Patch

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@coderabbitai

coderabbitai Bot commented Oct 17, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@coderdan