json_tree: a subtree read sends a token for the whole top-level field because member tokens mix in the sibling ordinal — make tokens derivable and drop gates #1360

Background

EQL (Encrypt Query Language) is the SQL library we install into a customer's PostgreSQL database. EQL v4 has a new column type, public.eql_v4_json_tree, that stores an encrypted JSON document as one entry per JSON node, so the database can answer JSON queries without the encryption key. Stack Encrypt, our Rust encryption engine (packages/stack-encrypt), writes these documents and reads the answers back. None of this is released yet.

To read part of a document (a "subtree read", for example the value at $.profile.address.city), the client sends the database a token: a 16-byte secret derived from the path. The database uses it to find the node in each row and walk the entries beneath it. A token is a capability over the structure under its node (which nodes exist and how they nest, not their values), in every row of the column, and it never expires. So the less a read's token covers, the better.

Each node's token is a hash of its parent's token. Today an object member's token also mixes in its ordinal: its place among its siblings, in an order that depends on which other keys the object has in that row (child_token and shuffled_ordinals in packages/stack-encrypt/src/json/derive.rs).

Problem

The client does not know which sibling keys a row has, so it cannot work out an object member's ordinal, and so it cannot derive the token of any node below a top-level field (except through array positions). Instead it sends the deepest token it can derive plus a list of "gates", and the database walks down from there (subtree_key_for in derive.rs, eql_v4.json_tree_subtree_at in packages/eql/src/v3/json_tree/functions.sql). Step by step, for a read of $.profile.address.city:

1. The client derives the token for $.profile, the top-level field, and two gates (the selectors of $.profile.address and $.profile.address.city).
2. The database receives $.profile's token.
3. From then on, for as long as the column exists, the database can map the full structure of $.profile in every row, although the caller only asked for one leaf.

Two more problems come from the same cause:

• A read returns a node's descendants but never the node itself, because the database finds a node through its parent's token. So an empty object, an empty array, a plain value and a missing field all come back as "no rows", and the client cannot tell them apart (json_tree: a subtree read of {}, [], a scalar or a missing field all return no rows, so the client cannot tell them apart #1356).
• The fix in progress for that (fix(eql): return the node itself from every json_tree subtree read #1359) sends an array's token to read one element. A read of $.tags[1] then hands the database the structure of every element of $.tags in every row: a wider leak than before.

Who is affected: every customer who reads part of a json_tree column. Nothing catches it, because every answer is correct; the cost is what the database learns. The leakage page (packages/eql/docs/reference/json-tree-leakage.md) states the current behaviour.

Proposal

Agreed with the owner on 2026-10-24. The full write-up, with formulas, the SQL, the client checks, the leakage and open questions, is the "Design change (2026-10-24)" section of docs/plans/2026-10-17-json-tree-index.md, in the draft pull request linked below.

1. Make an object member's token independent of its siblings. Derive it from the parent's token and the member's own selector (a keyed hash of its path), without the ordinal. Array elements keep their index. Ordinals stay, but only for listing a node's children. Then the client can derive the token of any path without a wildcard from the path alone, and a token always covers exactly the node asked for.
2. Add a "self tag" to every entry: an 8-byte hash of the entry's own token, salted per document, so the database can find a node from its own token. A read then returns the node itself first, then what is beneath it, so {}, [], a value and "missing" are four different answers, as PostgreSQL's own jsonb gives. It costs about 10% more storage, to be measured.
3. Expand wildcards on the client. For $.items[*].price, send one token per index up to a bound (64 by default, configurable), plus one "probe" token for the element just past the bound. If the probe finds something, fail loudly naming the bound; never return a partial answer. Remove the gates and json_tree_subtree_at; the read becomes json_tree_subtree(doc, tokens, budget).
4. Keep the client's checks from fix(stack-encrypt): check that a json_tree read returns the entry asked for #1358, adapted to the new read (the write-up recommends keeping the check that a subtree's rows are consecutive, since the self tag is not authenticated).
5. Regenerate every fixture, pinned test vector and benchmark table: the stored format changes, and nothing stored breaks because it is unreleased.
6. Measure in cipherstash/benches (branch feat/json-tree-benches): stored size, read latency against today's walk at bounds of 8, 64 and 256, probe cost and ingest time.

Interim: none needed. Nothing is released, and #1358 keeps reads correct in the meantime.

Relationship to other work

• json_tree: a subtree read of {}, [], a scalar or a missing field all return no rows, so the client cannot tell them apart #1356: the problem this solves as a side effect (the four answers).
• fix(eql): return the node itself from every json_tree subtree read #1359: its mechanism (head found by selector, last array position sent as a gate from the array's token) is superseded by this. Its behaviour (the four answers, reads opening to "no value" where the row has no node) and its oracle test carry over. To be closed once the implementation pull request is open.
• fix(stack-encrypt): check that a json_tree read returns the entry asked for #1358 / json_tree: a value read with -> or a subtree walk is not checked to be the path asked for #1355: the client checks this adapts. This work is stacked on fix(stack-encrypt): check that a json_tree read returns the entry asked for #1358.
• json_tree: the stored entry order reveals each row's shape and the alphabetical order of sibling keys #1357: the stored-order leak. Not changed by this, still a separate decision.
• test: hold json_tree containment to jsonb with proptest, and restructure the JSON index tests #1345: should merge before fix(stack-encrypt): check that a json_tree read returns the entry asked for #1358, which it conflicts with.