Repository navigation
json_tree: the stored entry order reveals each row's shape and the alphabetical order of sibling keys #1357
Description
Background
EQL (Encrypt Query Language) is the SQL library we install into a customer's PostgreSQL database. EQL v4's public.eql_v4_json_tree column type (#1250, #1288, #1325, #1329, #1336) stores an encrypted JSON document as a list sv of entries, one per JSON node, plus a containment projection q (one pair of keyed hashes per node, nested by array element). Every value and key name is encrypted. Each entry's path selector s is a keyed hash of the node's path, the same in every row of the column.
To keep a database that holds a token for one subtree from ranking key names it guesses, an object member's position under its parent (its ordinal) is a keyed shuffle of its siblings' keys (design decision 11 in docs/plans/2026-10-17-json-tree-index.md).
Problem
sv and q store the nodes in depth-first order with the root first, and an object's members in the sorted order of their key names (serde_json's Map without preserve_order). The order is visible to anyone holding the table, with no key and no query. Comparing rows, since selectors are the same in every row, an observer learns:
- each row's exact shape: two rows with the same sequence of selectors have the same tree;
- every node's parent, and the node count of every subtree;
- which nested entries belong to which array element;
- the alphabetical order of sibling key names (that the key behind one selector sorts before the key behind another), which narrows guesses at what each selector names and partly undoes the keyed member ordinals.
The leakage page (packages/eql/docs/reference/json-tree-leakage.md, "the order of the entries in sv") states this, and the design record lists it as an open decision ("Open for the owner: the stored entry order", Phase 4 notes, review of #1336). No option changes it today.
Also worth knowing: a build that turns on serde_json's preserve_order feature anywhere in the dependency graph would store the application's own field order instead, which leaks more.
Proposal
Not scheduled yet; to be done in its own pull request after the extracted-entry fix (#1355) and the empty-container fix (#1356).
- Investigate a per-document keyed permutation of the stored order: entries stored in an order derived from the document's key, with
j(the position each ciphertext is bound to) the permuted position, so the order says nothing about the tree. - Rebuild documents without reading the tree from the order: the rebuild would need each entry's parent explicitly (it is inside the ciphertext or derivable on the client), and the subtree walks would sort by depth-first position after decryption rather than by
j. - Do the same for the pairs of
q. Note thatqstill has to nest by array element for containment to stay exact, so element grouping stays visible whatever the order ofsv. - Measure on real-world documents how much the permutation actually hides once
q's nesting is taken into account, and what it costs (walk cost, document size, rebuild cost), before deciding. - Pin
serde_json'spreserve_orderoff, or make the stored order independent of it, either way.
Activity
coderdan commentedon Oct 25, 2026
Owner decision 2026-10-25: deferred until the stored order is reconsidered as a whole, together with two things found during the review of #1363:
svin document order. A keyed permutation of the stored entries breaks that as written; the order-independent replacement would check the authenticated positions and child counts instead of adjacency.Mapiterates: UTF-8 key order by default, insertion order if any crate in the build enablespreserve_order. Reads do not depend on it (member ordinals are a keyed rank, tokens never involve order, the client rebuilds objects by label), but the leakage differs, and a writer-chosen permutation would make the stored order independent of the serializer. The leakage page states the current behaviour.https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
coderdan commentedon Oct 25, 2026
A spike for this is up as #1371 (draft, not for merging as it stands). Two findings worth recording here:
jsonb @>over the projection, which reads it as sets, and every read finds entries by tag and returns the stored slot. So a random stored order needs no SQL change at all: the spike shuffles the entries (root kept at slot 0) and the projection, and the PostgreSQL oracle passes 12 of 14 tests under it, the two failures being assertions on slot numbers.The write-up with the measurements, the comparison with a selector-keyed order, and what remains visible (the selector multiset, the element grouping of
q, the heads, the lengths) goes to the owner separately.https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
fix(stack-encrypt): clear the spike's CI failures and pin the tag hel…
docs(eql): stop promising document order from the json_tree walks