Repository navigation
Conversation
…oducer Stack Encrypt can produce one EQL type because its ciphertext, equality and order encodings disagree about which kinds exist, and the mapping from a kind to a Rust type lives in stack-encrypt (Scalar, dynamic::Value) instead of vitaminc. EQL also cannot tell a Stack Encrypt column from a cipherstash-client one, so a query from one producer against the other's column matches nothing, silently. ADR-0002 records the decisions from the value-encodings RFC review: the new kinds and their frozen tags, one canonical form per kind for terms, term derivation moving into vitaminc-prf and a new vitaminc-ore, and EQL v4 as the v3 SQL emitted under a second name with "v": 4 envelopes, so Postgres keeps the two producers apart. The plan-builder plan defined "EQL v4" as a name for a Stack Encrypt payload in the v3 envelope and eql_v3 domains; it now points at the ADR.
…m stack Starting vitaminc-ore under ADR-0002 surfaced four problems: order encodings, scheme traits and term derivation spread over three repositories; scheme crates that accept raw, hand-encoded bytes; vitaminc-prf reaching orderable-bytes through aead-value's canonical module; and one canonical form shared by equality and order, which rules out leakage-tuned domains such as an exact timestamp for equality with day resolution for order. The plan settles the replacement design: vitaminc-ore owns the order encodings and an OrderScheme trait with a sealed input, and ore-rs and cllw-ore implement it; each term crate applies a semantic baseline and per-domain transforms named in its labels; PRF has no ordering dependency; orderable-bytes is frozen. Exhaustive matching over Value is left open, with both approaches and their shared prerequisite laid out. It includes current and target dependency diagrams and the steps, and ADR-0002 now points to it. Claude-Session: https://claude.ai/code/session_012zHhtTEo968WazExKxA5oT
The ADR said Postgres refuses to compare an eql_v4 query term with an eql_v3 column at plan time. It does not. EQL's domains are over jsonb, and with no operator defined for the exact pair Postgres resolves `=` as the built-in jsonb = jsonb, so the query returns no rows silently, as a mismatched pair within v3 already does. Verified on Postgres 17 with both bundles from #1141 installed in one database. The decision stands: separate names still reject a wrong-version insert, make the function form ambiguous, and give typed clients distinct types. Consequences now says where the protection stops, and Deferred names the cross-version blocker operators that would close the gap (#1160).
The ADR deferred to a plan whose own banner disclaims being a specification, so an implementer had no authoritative text for the canonical forms, the order-scheme layering or the release order. The amended decisions now live in the ADR body, with exhaustive handling of Value recorded as the one open decision. Review fixes: pin the timestamp microsecond rounding to Postgres's (round half to even, checked on 17.5) and the text fold's step order; let unassigned code points pass through instead of refusing writes; forbid lossy order transforms on _ord domains that answer = with the order term; choose a binding's kind from the declared EQL target, with a kind-to-family table, and take a Go date in the value's own location; copy the exact leaf byte layouts from tags.rs; say why float equality is now acceptable and that text order terms change; keep cllw-ore's Bool impl; reintroduce --eql-version deliberately on install and migration; list the eql_v3_-prefix detectors, installer schema lists and single-bundle release tooling v4 needs; and name the owner of the deferred cross-version blockers.
…ic.=) as a loud path ADR-0002's host-language table claimed the seven non-EQL integer kinds and Bytes go through Stack Encrypt's own targets, but no target names them yet. Say so, and say how Go's narrow integers reach a target today. The Deferred section listed typed clients and the function form as the only things that catch a cross-producer query, while Consequences also names OPERATOR(public.=). List all three.
ADR-0002 separates the two producers of EQL payloads by Postgres type: cipherstash-client's terms stay in eql_v3, Stack Encrypt's move to eql_v4, and both come from one SQL source. This is the first step of #1141: the source stops naming its schemas and payload version literally. The hand-written SQL under src/v3, the version template, the search_path pin script and the uninstaller now write {{prefix}} where they wrote eql_v3, and {{eql_version}} where domain CHECKs compared VALUE->>'v' to '3'. eql-codegen emits the same placeholders: its schema, typname-prefix and version constants are placeholder strings, and its unit tests render output as v3 before asserting. The Rust bindings keep the literal eql_v3_ prefix because they describe v3 payloads. A new `eql-codegen render <3|4> <out> <file>...` substitutes them. It refuses an unknown or unterminated placeholder and any literal eql_v3 or eql_v4, since a literal name would render the same for both versions and let the v4 bundle reach into v3. build.sh renders the ordered surface into build/render/eql_v3/ and runs the symbol-order and installer-completeness gates on the rendered files. The docs tooling reads rendered SQL too. The committed generated surface is regenerated in the next commit, so codegen:parity fails at this one alone. Refs #1141
Output of `cargo run -p eql-codegen` after the previous commit; no hand edits. `mise run codegen:parity` reproduces it. Only v3 is rendered so far, and the installer and uninstaller built from this commit are byte-identical to the ones built before the placeholders. Refs #1141
Gate 1 of verify_installer_complete.sh compared the order against the rendered tree, which build.sh writes from that same order, so a dropped file could no longer fail it. It now lists the source tree and maps the paths into the rendered namespace (a truncated order fails again). The Doxygen filter read through a process substitution, whose exit status is lost, so an unreadable input passed docs_v3_grep.sh with empty output; it is a plain pipe under pipefail now. eql-codegen render refuses an absolute or .. path instead of writing over the source. build/ is anchored so tasks/build/ is not ignored, the codegen templates are build cache sources, a moved doc comment is back on its function, and the schema, JSON and pin-script comments name the rendered version (v3 output is byte-identical).
- `eql-codegen render` refuses an empty file list instead of exiting 0
having written nothing.
- The literal-name guard no longer matches a longer version such as
`eql_v30`, and its error says to write the placeholder, comments
included.
- `list-schemas [3|4]` prints rendered names, and the schema parity gate
renders schema.sql with the build's renderer before comparing.
- The Doxygen filter refuses any `{{` left after its substitution.
- verify_installer_complete.sh takes its rendered root from the order
file's directory by default.
- The render tests clean up their scratch directories on failure.
- pin_search_path_v3.sql's header says it is rendered, not appended
verbatim. This is the only change to the v3 installer, and it is a
comment.
…endering it The filter substituted the placeholders with its own sed, a second renderer without eql-codegen's checks. It now reads a src/ or tasks/ file's rendered copy from build/render/eql_v3/, which every caller already builds, and refuses a file with no rendered copy. Other files, such as the filter's own test fixtures, are read as given. The Doxygen XML is byte-identical (297 files). docs_v3_grep.sh counts the hidden ste_vec_contains aliases in the rendered file too, so it no longer depends on the placeholder's spelling, and now depends on build. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB
… traits on v4 only Stack Encrypt's output is not byte-for-byte compatible with cipherstash-client's, which writes the eql_v3 domains, so an eql_v3 struct deriving EncryptFrom or DecryptInto claims a compatibility that does not exist. ADR-0002 gives Stack Encrypt its own name, eql_v4. eql-codegen now writes the scalar domains a second time into eql_bindings::v4: eql_v4_* names, a SchemaVersion pinned to 4, a schema id under eql/v4, and an inventory. encryption_gap refuses every v3 domain, so TextEq and TextEqQuery derive their traits only in v4, and a codegen test asserts no v3 struct derives either. The Stack Encrypt target table moves to v4/targets.rs and names eql_v4 domains; the Go package encrypt/eql is regenerated from it. v4 has no SteVec bindings, payload enums, TypeScript export or JSON Schema yet. No published eql-bindings carried the stack-encrypt feature, so nothing released loses a trait. The encryption test crate's PostgreSQL test now installs the v4 bundle; the Go guest and the stack-encrypt fixtures name the v4 domains. Refs #1142
ADR-0002 installs one SQL source under two names, so a column's domain says which engine wrote its terms. The build now renders the placeholder source a second time as eql_v4 and writes release/cipherstash-encrypt-v4.sql and its uninstaller. The v3 artefacts are unchanged, byte for byte. verify_no_v3_in_v4.sh fails the build if the v4 bundle names eql_v3 or checks for payload version 3: a stray v3 reference would install cleanly and then compare v4 data with v3 terms. The symbol-order check stays on v3 only, since v4 is the same files in the same order with every name renamed. v3_v4_coexistence_tests installs both bundles into one database and pins what the separate names enforce: a wrong-version payload fails the domain CHECK, and the cross-version function form is ambiguous. It also pins the gap the corrected ADR records: a plain `=` across versions falls back to jsonb = jsonb and returns no rows, silently, until cross-version blocker operators exist (#1160). Refs #1141
The guest's eql-feature round-trip test still asserted "v": 3 after Stack Encrypt moved to writing EQL v4, failing the WASI/Go CI job. Also update two comments that named the v3 envelope and the eql_v3.query_<name> operand.
…dles verify_no_v3_in_v4.sh piped grep into head under pipefail. When the matches outgrew the pipe buffer, grep died of SIGPIPE, the condition read false, and a wholesale eql_v3 leak passed while a single stray reference failed. It now captures the matches before printing them, and a new self-test (test:v4_isolation, run in CI) feeds it a 20,000-line leak. With v3 and v4 in one database, each lints() reported 3,840 errors about the other version's blocker operators: encrypted_domain_types took every jsonb domain in public, while blockers are recognised only in the lint's own schemas. The public arm is now limited to the lint's own prefix, and the coexistence test asserts both lints are clean with both installed. The coexistence tests also pin OPERATOR(public.=) between versions (ambiguous) and the eq_term text comparison (runs, refuses nothing), and build.sh and mise run clean remove the v4 artefacts too.
…he version boundary The inventory boxes PhantomData<T> handles, and their DomainType impl did not forward schema_id, so every v4 entry reported an /eql/v3/ $id. A type-level schema_id_base_static() now carries each type's base through the handle, and the generated v4 types override it. That surfaced in a new v4 parity test, the counterpart of catalog_parity.rs: v4::all() is the scalar catalog under eql_v4 names in catalog order, each query twin lives in eql_v4, term keys match the catalog, and each generation's TextEq refuses the other's version, both directly and through the inventory. The encryption suite refuses a real v4 payload whose version alone was set back to 3. types:generate snapshots and restores src/v4 as well as src/v3, types:check diffs it, and the pre-commit hook triggers on it. The v4 module doc no longer links a feature-gated module, and the Go READMEs name the eql_v4 column and query operand the guest now writes.
- The gate accepts a payload-version read only in the form `->> 'v' = '4'`, so IN (...), swapped operands, casts and jsonb comparisons are refused without being listed. A grep fault ends the gate with 2 rather than reading as no match. - The self-test asserts the exact exit status for every case: 0 clean, 1 leak, 2 bad input. A crashed gate no longer passes the rejection cases. - test:installer_complete checks the v4 installer as well as v3. - lints() matches its own public domains with starts_with: in LIKE, the underscores inside the prefix are wildcards.
v3 and v4 each had a hand-written SchemaVersion that differed only in the constant and the schema name. They are now one type, SchemaVersion<const V: u16 = 3>: v3::SchemaVersion and v4::SchemaVersion are SchemaVersion<3> and SchemaVersion<4>, and plain SchemaVersion still means v3. The TypeScript binding is written by hand, since ts-rs's derive has no const generics, and bindings/v3/SchemaVersion.ts is byte-identical. Also: - The changeset says what moved to v4 was never released. - The Go README says how to install EQL v4 before using an eql_v4 column. - The targets test exempts only the SteVec rows from the compiled-type check, and asserts they are not producible. - packages/eql/AGENTS.md: types:check covers src/v4, and v4 shares v3's term types. - A private intra-doc link in encryption::targets no longer fails rustdoc -D warnings with the stack-encrypt feature. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB
…atalog operator_across_versions_returns_no_rows read the planner's choice from EXPLAIN text, whose deparse can change between Postgres majors. It now stores the comparison as a view and reads the operator it bound: an EQL operator from pg_depend, and the built-in jsonb = jsonb, which is pinned and has no dependency row, from the view's stored query tree. A control checks that a same-version comparison binds the EQL operator, so an empty list is a finding rather than a probe that sees nothing. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB
The hook never ran in cipherstash/stack, and would have matched nothing if it had: - `mise run install-hooks` set core.hooksPath to `tasks/githooks`. Git resolves a relative hooksPath against the root of the working tree, which here is the monorepo root, so git looked for `tasks/githooks` there and ran no hook. It now writes the path from the root, through `git rev-parse --show-prefix`. - Git runs a hook from the root of the working tree, so the staged paths carried a `packages/eql/` prefix that the hook's patterns, anchored at `crates/` and `src/`, never matched. The hook now lists only the staged paths under this package, relative to it. - `mise run` ran from the monorepo root, where this package's tasks are not defined. The hook now works from the package root. The prefix is computed from paths rather than with `cd` and `git diff --relative`, because git exports GIT_DIR to a hook, and with GIT_DIR set git takes the current directory as the top of the working tree. Checked with `git hook run pre-commit` from the monorepo root: a staged `src/v3/schema.sql` runs `test:schemas:parity`, and an unrelated staged file runs nothing. Running the hook from the package directory still works. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB
…text types A Go partner needs a pre-release of EQL v4 with Stack Encrypt before vitaminc 0.6 settles ordering. These are the v4 types that need no ordering term and whose value kind is already in vitaminc 0.5's value enum: Text, Integer, IntegerEq, Bigint, BigintEq, Boolean, Real and Double, beside TextEq. vitaminc 0.5 implements Encrypt for String but not for i32, i64, bool or the floats, and stack-encrypt deliberately has no combinator that re-encodes a plaintext. So eql_bindings::encryption::plaintext adds Int32, Int64, Bool, Float32 and Float64: each seals vitaminc's tagged leaf, the bytes FfiValue seals for the same value, opens only its own tag, and the integers derive the equality term of the integer inside. A value written through these types and one the Go SDK writes as a plain field open each other, and IntegerEq's term is a plain i32's. The derive reasons now have one source, encryption_gap: ordering terms wait for vitaminc-ore, float equality for vitaminc 0.6's canonical float, smallint/date/timestamp/numeric for their value kinds. Json stays unproducible: a JSON value seals as a tree of leaves and EQL's c holds one, and how a document becomes one leaf is a wire decision not made here. The encryption test crate checks each type through both entry points, against the value enum's leaf and the plain integer's term, and in PostgreSQL on an eql_v4 install: equality queries on IntegerEq and BigintEq return the right rows and every stored value opens. Refs #1130, #1133
Text became producible in this PR, so it is now the live storage-only type the no-query rule describes, and TextOrdOre's refusal names vitaminc-ore rather than CLLW. Both unit tests still asserted the old table and failed under --all-features.
Their equality terms came from vitaminc 0.5's per-width PRF domains (i32-le/v1, i64-le/v1). vitaminc main replaces those with one integer-orderable/v1 domain for every width, so shipping them now would either break the promise that IntegerEq matches a Go int32 equality field or leave every stored hm unmatchable after the upgrade. Both leave ENCRYPTION_DOMAINS with a reason that names the domain, the generated dispatch and the Go table mark them unproducible, and the PRF impls on Int32 and Int64 are removed so nothing derives a term under the old domains. The storage-only types, TextEq and the plaintexts stay. v4_types.rs now covers what the review found missing: a leaf the value enum sealed opens as each typed plaintext, every producible row goes through all three by-name entry points (a swapped type or plaintext in the generator fails), Float64 is checked against the value enum, and the Postgres test reads every column back. The width test that pinned the per-width design is gone. The guest's producibility assertions and the eql-bindings unit tests use the final producible set, and the stashgen README lists each producible type with the Go field type it takes.
… bits, check widths - plaintext.rs: Float32 and Float64 compare by bit pattern, so -0.0 is not 0.0 and a NaN equals itself, the equality a round trip preserves. Unit tests pin each leaf's tag, width and little-endian order through the same reader Decrypt uses, and refuse another tag or width. - The generated plaintext doc line shows without the stack-encrypt feature again; only its link to crate::encryption is feature-gated. - stashgen refuses a field of the right kind and the wrong width (int64 into Integer, float64 into Real) itself, naming both widths, as its README says. - enginetest.Static lists the seven types the engine produces, with their widths. - The eql-bindings README no longer says TextEq is the only producible type. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB
…ch terms TextMatch needs no ordering term, and the engine already derives match terms. Under ADR-0002 the terms only have to be ones EQL's SQL can compare, not cipherstash-client's bytes: both the stored value and the query come from Stack Encrypt, and cipherstash-client never writes an eql_v4 column. EQL's bloom filter is a smallint[] of set bit positions, and @@ tests containment. The new BloomFilter adapter writes the engine's positions under its default match configuration (3-grams, lower-cased, k = 3, a 256-bit filter), so every position fits a smallint, and refuses one that does not rather than wrapping it. It names the match term MatchTerm, the spelling that compiles against both the tree and the crates.io stack-encrypt 0.2.0 the published crate builds against. Proven in PostgreSQL on an eql_v4 install: @@ returns every row for a shared substring, one row for a unique one and none for a miss. Text too short to make a 3-gram is refused rather than stored with a filter no query could match. Refs #1131
…age against crates.io The adapter's turbofish named four type parameters for target::matching, and stack-encrypt 0.2.0 on crates.io, which eql-bindings publishes against, takes three, so cargo publish --dry-run --all-features failed. Naming the Encryption type instead lets inference fill the rest under either version. The match configuration was Stack Encrypt's default (k = 3, 256 bits), taken from another crate and recorded nowhere. A 256-bit filter saturates on a few hundred characters, so @@ returned most of a table, and a change to that default would have left stored filters disagreeing with new queries. eql-bindings now pins its own EqlMatch (3-grams, lower-cased, k = 6, 2048 bits); a unit test pins the values and an encryption test pins the filter positions for one query, so a change has to be deliberate. Also: TextMatchQuery's generated docs say match query terms, the Postgres test pins the reverse operand order (query @@ stored) as the known SQL gap it is, the Go guest lists TextMatch as producible, and the READMEs and changeset say that short text cannot be stored yet. Storing an empty filter for it needs a stack-encrypt release first: 0.2.0 refuses token-less text inside the term derivation, before anything here can see it.
…he test engine TextMatch - The eql-bindings README, the stashgen README and the changeset say that only `stored @@ query` matches until #1171 is fixed, and that a TextMatchQuery for text shorter than three characters is refused too. - The PostgreSQL test's reversed-order assertion names #1171, and the filter test says its negative case is a property of one key source and configuration, not of Bloom filters. - enginetest.Static produces TextMatch; the refusal case for an EQL type the engine cannot produce now names IntegerEq; the users golden case stores Name as TextMatch, so generation of its field methods is exercised. - encryption_gap drops its Bloom branch, which no catalog domain reached once text_match became producible. - The generated TextMatchQuery doc keeps the plaintext line without the stack-encrypt feature. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB
…eSQL stashgen needed no change: it maps int32, int64, bool, float32, float64 and string to their kinds and accepts any type the engine's se_targets table marks producible, so every type the previous commits made producible already generates typed Encrypt and Query methods. A Profile fixture stores one field in each new type. The hermetic test seals it through generated code over the deterministic EQL guest, checks every value is a v4 envelope with the type's keys, and opens it back. A cross-language fixture, v4_queries.json, holds the IntegerEq, BigintEq and TextMatch query operands the Rust dispatch derives under the shared test index key; the Go fields derive them byte for byte. Checked by hand against PostgreSQL: rows sealed by this Go code, loaded into an eql_v4 install, answer =, = and @@ with exactly the expected ids. The Go module has no PostgreSQL driver, and adding one for a test would change every consumer's dependency graph, so that check is not committed. The encrypt and stashgen READMEs name eql_v4 and list the producible types with the Go field type each takes. Refs #1144
… withdrawn integer equality types Nothing in the repository put a value the generated Go code sealed into an eql_v4 domain or operator; the PostgreSQL check was manual. The Go test now writes v4_rows.json, the Profile rows the generated code seals over the deterministic EQL guest. On every later run it checks that file against today's output, envelopes and match terms byte for byte and ciphertexts by opening them (their AEAD nonce is random). The EQL suite opens those rows with stack-kms's DeterministicSource over the same seed, and its PostgreSQL test, which test:encryption:postgres runs in CI, inserts them into eql_v4_* columns and queries name @@ with the Go query operand from v4_queries.json. With IntegerEq and BigintEq held back, Profile drops Age and stores Balance as Bigint, and v4_queries.json carries TextMatch only. The one stored-versus-query check had used Age, so it is replaced by a TextMatch check that a contained needle's filter bits are a subset of the stored filter and an absent one's are not. Every stored field's i is checked against its table and column. The Go workflow now runs when the fixtures or EQL's lockfile change, and biome leaves the generated rows file alone.
…re stable - TestV4RowsMatchTheCommittedFixture writes the rows it seals to EQL_V4_ROWS_OUT, and the Rust suite opens the file EQL_V4_ROWS names. The Go job runs both, so the ciphertexts the Go SDK seals in that run are opened by native Rust, not only the committed ones. - EQL_UPDATE_FIXTURES rewrites v4_rows.json only when something besides the ciphertexts moved, so a regeneration that changes nothing leaves no diff of fresh nonces. - The Go and Rust TextMatch checks say that their negative cases hold for one seed and configuration, the PostgreSQL expectation folds case as `@@` does, and a needle shorter than three characters is asserted to be refused. - tests-golang.yml runs on any change under packages/eql/tests/encryption. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB
The encrypt README and package doc opened with the index= grammar (Ciphertext plus term byte columns, Fields.Email.Equality), while the stashgen README and the EQL v4 alpha setup guide use encrypt_into= and Fields.Email.Query. A reader starting from go doc wrote tags the guide does not describe, for a storage model it does not cover. Both quickstarts now declare encrypt_into fields, build the query with Query, and store each field as one eql_v4 column, with the install step and the query cast. The index= form stays documented and is named as the alternative; the example program still uses it, and the README now says so rather than calling it the eight steps.
…the Go SDK can query by operator without a tokenizer of its own The encrypt guest derives a query term from a plan, a field, an operator name and its input, verifies a page of decrypted values in one call, and sizes a probabilistic query's fetch from its term and the table's statistics; the last two use no key and run with no cipher. se_targets entries carry the operators each EQL type answers, and the Go reader requires them. Cipher.QueryTerm, Cipher.Verify and Cipher.TargetLimit wrap the exports for generated code, and the Go tests hold them to the Rust query fixture through the deterministic guest. Part of #1188
ORE and OPE still order text as its raw UTF-8, while equality and match terms derive from its Unicode 16 NFC form. On a column with both an equality and an order index, two forms of `café` are one value to `=` and two to an ordering. `ore_of`'s rustdoc and the CHANGELOG now record this as a known gap, to be closed with block ORE (#1140).
…oposed until ADR-0009 is accepted The data grammar gains the operator and its input, the term comes back with its query domain, verify and target_limit become engine exports, and the catalog reports each EQL type's operators and their fidelity. The amendment rests on ADR-0009, which is proposed, and takes effect when it is accepted. Part of #1188
…n reaches only through admit The mutants gate found Match's verifier and QueryTokens::of untested: the chain takes the query's tokens through admit, which the match index overrides, so replacing either with an empty default changed no test. The new unit test builds a query from the verifier and checks it verifies a row holding the tokens and refuses one that does not.
…r yet, and pin it Review follow-ups. On top of next the EQL v4 catalog produces Integer, Bigint, Text, TextEq, TextMatch, Boolean, Real and Double, so the ADR-0007 amendment's "today TextEq answers eq and no other type is produced", and its "both wait for the EQL v4 types", were wrong. TextMatch is produced and has a query twin but answers no operator, because the catalog does not carry the match options `verify` cuts tokens by; the amendment and the `dynamic::query` module docs now say so, and say the gate is that catalog work. Nothing held that in place: the catalog tests checked TextEq and the types not produced, so a produced type starting to answer `matches` would have passed. The guest and Go tests now require every type but TextEq to answer nothing, and a guest test refuses `matches` and `eq` on a TextMatch field as not answered. The module docs no longer call the dynamic ordering rule the typed path's default: the typed default is ORE, and a field declaring only OPE is answered through it here, which the typed path spells `.via(Ope)`. The `Reason` docs count five inputs, not four.
… a cipher, wipe both verify buffers, and name the field on bad stats Copilot review. An `eq` query on text with a code point unassigned in Unicode 16 passed `check_query`, which validated ranges and match text but not equality normalization, so `se_query_term` consulted the cipher (and could load a keyset) before derivation refused it; a guest with no cipher reported STATUS_STATE where the refusal is a term failure. `check_input` now runs the equality normalization the derivation runs, keylessly; a test in the engine and one at the guest boundary pin both the check and the op. `se_verify` took the query and the values buffers one after the other with `?`, so a bad query pointer returned before the page of decrypted values was taken and wiped. Both are taken before either result is used. (The Go host wipes every staged buffer after the call regardless.) Statistics are parsed with no field to hand, so a malformed `stats` in `se_target_limit` was refused naming no field; it now names the query's, as every other query refusal does.
The fixture is written by serde_json's pretty printer when STACK_ENCRYPT_UPDATE_FIXTURES is set, and the test compares the file with what the engine derives, so Biome's JSON formatting would make every regeneration a diff. It joins record_lowering.json and the other generated fixtures in biome.json's excludes.
…tokens FieldPlan::new is public and does not check the MatchOptions of a match output, so a plan built by hand can carry an n-gram length of zero. The term derivation validates them, but check_query and query_term reach the tokenizer first through check_input, and verify calls it directly, so such a plan panicked in the tokenizer rather than returning InvalidOptions. query_tokens now validates the options first.
…nswers, so a generator holds no copy of the table dynamic::field_operators lists the operators a field of a data plan answers, each with the index that answers it: for a field declared with indexes, what OpSpec::answered_by finds under its kind; for a field that names an EQL type, what the type's catalog entry lists. query_term reads the same table, so what a generator is told a field answers is what a query on it finds. operator_to_value writes one entry in the catalog's wire form, which TargetDescriptor::to_value now shares. Part of #1189
- The catalog reader and se_query_term's reader share one parser each for an operator's name, fidelity and index. A name is one of eq, lt, lte, gt, gte, between and matches, the set the engine's OpSpec::parse accepts. The catalog took any non-empty name, so "like" or "Eq" parsed and failed later as unknown_operator from Cipher.QueryTerm, and se_query_term took any operator and any index, which the catalog refused. Both now refuse the same values, and a test feeds every name the engine writes through both. - The query fixture test checks that query_operators.json was recorded under queryPlan, so a fixture regenerated under another plan fails by name rather than as mismatched term bytes.
…n, with verify and target_limit An EQL type with a match index answered no probabilistic operator: the catalog named its match index but not the options the filter is built under, and `verify` has to cut the query's tokens exactly as the stored filter was cut. So `TextMatch`, which is produced and has a twin, answered nothing, and a Go program had no way to query it once the generated `Query` method goes. `TargetDescriptor` now carries the options of the type's match index (`with_match_options`, `match_options`), off the wire because a host has no use for them. A type that has them answers `matches` as well as its exact operators, through its match index under those options: `query_term` derives the type's query twin, after the same refusals an index field's `matches` makes (options, normalization, text that cuts no token), and `verify` and `target_limit` run under the type's options, `target_limit` reading the number of positions from the twin's `bf`. A type with a match index and no options still answers no probabilistic operator. `eql-bindings` gives the options for each type with a match index (`Target::match_options`, `EqlMatch`'s), and the Go guest's resolver hands them to the engine with the descriptor, so `TextMatch` answers `matches` through `TextMatchQuery` and still refuses `eq`. The guest and Go catalog tests, the guest's TextMatch test and ADR-0007's 2026-10-16 amendment say so.
…ers, with Verify and TargetLimit on a matches query stashgen asks the engine which operators each sealed field answers, through a new guest export, se_operators, and writes one method for each: Eq, Lt, Lte, Gt, Gte, Between and Matches, each taking the field's own Go type. A field has no method for an operator it does not answer, so such a query does not compile. The generator holds no copy of which index answers which operator; the static test engine is held to the guest by a test that generates the same declarations through both and compares the files. Eq returns an EqualityTerm (or, on an encrypt_into field, the EQL type's query value), an ordering an OreTerm or OpeTerm, and Between an encrypt.Range of the two bounds' terms. Matches returns an encrypt.MatchQuery, whose answer can hold false positives, so it carries Verify, one guest call for a page of decrypted values, and TargetLimit. Every query type implements driver.Valuer. MatchQuery keeps the text it was derived from, because Verify cuts its tokens again in the engine; it never prints it. On an encrypt_into field whose EQL type has a match index (TextMatch), Matches returns an encrypt.EQLMatchQuery: the type's query value to bind (eql.TextMatchQuery, cast to its query domain) with the same Verify and TargetLimit, which run under the type's match options. se_query, Cipher.Query and gensupport's per-index term methods are gone: eq on a target field is its twin, and a query names an operator. se_term stays, now behind the unexported Cipher.derive, because a record call's term failure does not say which field it came from and the host asks se_term again to name it. The order tests read the terms Encrypt stores, since a query method reaches only the index that answers its operator. The EQL v4 tests query Profile's TextMatch field through Matches and verify the decrypted names. The query fixture's declaration is testusers.QueryProfile, since testusers.Profile is the EQL v4 profile. Part of #1189
…SQL, through database/sql and pgx The test stores generated rows in separate columns, finds one by its equality term, and runs a matches query whose filter containment PostgreSQL tests with @>, then decrypts the rows it returned and verifies them: the verified rows are the Rust query fixture's. It uses the deterministic guest's keys and skips unless DATABASE_URL is set; the Linux Go job now starts PostgreSQL 17 and sets it. pgx is a test-only import. The declaration is testusers.QueryProfile, the query fixture's. Part of #1189
…decrypt-and-verify step, and say se_query is gone The README and the package documentation describe one method per operator, what each returns, why a matches query carries Verify and TargetLimit and an exact one neither, and that a MatchQuery holds its text because Go cannot wipe a string. The ADR-0007 amendment records se_operators, the removal of se_query, and the one reason se_term stays. Part of #1189
…ition
The mutants gate found twin_query_bits' strictly-ascending check untested: no case repeated a position, so < and <= read the same. The twin refusal list now holds {"bf":[5,5]}.
coderdan
removed this pull request from stack #1230
October 17, 2026 05:57
docs(adr): ADR-0009, operators select the index and a query term is not the stored term
…-query feat(stack-encrypt)!: a query names an operator, not an index
…rify feat(stack-encrypt): verify a match query's rows after decrypt
…imit feat(stack-encrypt): size a match query's fetch with target_limit
…tch-empty-filter feat(stack-encrypt): store short TextMatch text as an empty filter, refuse only short queries
…-operators feat(stack-encrypt): ordering operators Lt, Lte, Gt, Gte and Between, and remove query_index
…rators feat(stack-encrypt): JSON operators Contains, ContainedBy and AtPath
…-adr0001 feat(stack-encrypt)!: move the stack crates, the guests and the Go module onto vitaminc ADR 0001
…through-vitaminc feat(stack-encrypt)!: derive terms through vitaminc-normalize and walk dynamic values as Plaintext
…-exports feat(stack-encrypt): query a plan field by operator through the guest, with verify and target_limit as exports
- Generated files name gensupport.GeneratedVersion2. The layout changed (Field.Term and Range in place of Equality, Match, Ore, Ope, Query and JSON), so a file from an older stashgen now stops at its version constant instead of on a missing method inside the file. - MatchQuery and EQLMatchQuery hold their query text, cipher, plan and field in one unexported value behind a pointer. fmt prints a nested pointer as an address, so the text no longer prints through %p (which skips Format) or through a struct that holds the query in an unexported field. The two types share Verify, TargetLimit, printing and the zero-value refusal instead of repeating them. - The zero MatchQuery's Value returns ErrEncoding, as the zero EQLMatchQuery's now does too, rather than binding NULL and matching no row without an error. - stashgen refuses a match index on an index-only field: the query's rows are checked by decrypting them, and such a field stores no ciphertext. - stashgen notices, on stderr and in the file, an index no generated method queries: an ore or ope index on text, bytes or a float, or the second of two order indexes on an integer. The terms are still stored. - A nil cipher is ErrEncoding from generated code as from the library; the duplicate checks in gensupport.Matches and EQLMatches are gone. - stashgen.Engine's Check returns the operators each field answers, and Operators is gone, so the guest engine lowers a declaration once and asks se_targets once per engine. - The PostgreSQL test closes its rows, so a failure mid-loop no longer blocks the cleanup that closes the connection. - The EQL v4 test checks Verify against a 3-gram oracle with case folded, over a page that includes an upper-cased needle and a needle's 3-grams out of order. - pgx moves to v5.11.0 and golang.org/x/crypto to v0.58.0, and go.mod marks pgx as test-only. - A Go module CHANGELOG records this PR's breaking changes, the Engine interface among them.
…ualify the no-request claim The package doc and README showed users.Fields.Email.Matches against the quick start's User, whose Email is encrypt_into=TextEq and has only Eq; the example now says it uses separate byte columns as example/ declares them. Both also said a query sends no request: a query derives its term locally, but the first use of a keyset, by a query or anything else, still loads it from ZeroKMS. The README's Queries table gains the TextMatch row, and the text says when stashgen notices an index no method queries and why a match index on an index-only field is refused.
The module has shipped only an alpha, so the generated-file layout change in this PR keeps gensupport.GeneratedVersion1 rather than declaring a version 2: a file from the alpha's stashgen fails on its missing methods instead, and the CHANGELOG says to run stashgen again.
feat(golang)!: generate one query method per operator, with Verify and TargetLimit on a matches query
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This is the integration branch for a release candidate of EQL v4 and of Stack Encrypt's Rust and Go support. EQL is the SQL that lets PostgreSQL store and search encrypted values; EQL v4 is a second copy of it for values written by Stack Encrypt, the Rust encryption engine behind the Go SDK. The open pull requests for both are merged here first, so they can be tested together before any of them reaches
main.The release version is not decided yet.
Why a branch, not main
Some of this work depends on vitaminc and ore.rs code that is not yet on crates.io, so it pins those crates to a git commit.
stack-encryptandeql-bindingsare published crates, andcargo publishdrops those git pins, so nothing on this branch can be released until vitaminc 0.6 is published (cipherstash/vitaminc#389, which waits on cipherstash/vitaminc#463). Once it is, the git pins become crates.io versions and this branch merges tomainas a normal release.What lands here, in order
feat/aead-remove-legacy-traitsto one commit on vitamincmain, and takes vitaminc's breaking changes since then.Rules for this branch
rev, neverbranch. A branch is deleted when its stack merges.packages/eqlandpackages/stack-encrypt/fuzz. Two commits mean two copies of vitaminc's traits, and the code stops compiling.Changes
Verification
nextwas created frommainat 2a70a47. Nothing has been built yet, because nothing has changed.Related