Skip to content

json_tree: the stored entry order reveals each row's shape and the alphabetical order of sibling keys #1357

Closed

Description

@coderdan

Background

EQL (Encrypt Query Language) is the SQL library we install into a customer's PostgreSQL database. EQL v4's public.eql_v4_json_tree column type (#1250, #1288, #1325, #1329, #1336) stores an encrypted JSON document as a list sv of entries, one per JSON node, plus a containment projection q (one pair of keyed hashes per node, nested by array element). Every value and key name is encrypted. Each entry's path selector s is a keyed hash of the node's path, the same in every row of the column.

To keep a database that holds a token for one subtree from ranking key names it guesses, an object member's position under its parent (its ordinal) is a keyed shuffle of its siblings' keys (design decision 11 in docs/plans/2026-10-17-json-tree-index.md).

Problem

sv and q store the nodes in depth-first order with the root first, and an object's members in the sorted order of their key names (serde_json's Map without preserve_order). The order is visible to anyone holding the table, with no key and no query. Comparing rows, since selectors are the same in every row, an observer learns:

  • each row's exact shape: two rows with the same sequence of selectors have the same tree;
  • every node's parent, and the node count of every subtree;
  • which nested entries belong to which array element;
  • the alphabetical order of sibling key names (that the key behind one selector sorts before the key behind another), which narrows guesses at what each selector names and partly undoes the keyed member ordinals.

The leakage page (packages/eql/docs/reference/json-tree-leakage.md, "the order of the entries in sv") states this, and the design record lists it as an open decision ("Open for the owner: the stored entry order", Phase 4 notes, review of #1336). No option changes it today.

Also worth knowing: a build that turns on serde_json's preserve_order feature anywhere in the dependency graph would store the application's own field order instead, which leaks more.

Proposal

Not scheduled yet; to be done in its own pull request after the extracted-entry fix (#1355) and the empty-container fix (#1356).

  1. Investigate a per-document keyed permutation of the stored order: entries stored in an order derived from the document's key, with j (the position each ciphertext is bound to) the permuted position, so the order says nothing about the tree.
  2. Rebuild documents without reading the tree from the order: the rebuild would need each entry's parent explicitly (it is inside the ciphertext or derivable on the client), and the subtree walks would sort by depth-first position after decryption rather than by j.
  3. Do the same for the pairs of q. Note that q still has to nest by array element for containment to stay exact, so element grouping stays visible whatever the order of sv.
  4. Measure on real-world documents how much the permutation actually hides once q's nesting is taken into account, and what it costs (walk cost, document size, rebuild cost), before deciding.
  5. Pin serde_json's preserve_order off, or make the stored order independent of it, either way.

Activity

  1. coderdan commented on Oct 25, 2026

    @coderdan
    ContributorAuthor

    Owner decision 2026-10-25: deferred until the stored order is reconsidered as a whole, together with two things found during the review of #1363:

    https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

  2. coderdan commented on Oct 25, 2026

    @coderdan
    ContributorAuthor

    A spike for this is up as #1371 (draft, not for merging as it stands). Two findings worth recording here:

    • The database never needs to generate or invert the permutation. Containment is jsonb @> over the projection, which reads it as sets, and every read finds entries by tag and returns the stored slot. So a random stored order needs no SQL change at all: the spike shuffles the entries (root kept at slot 0) and the projection, and the PostgreSQL oracle passes 12 of 14 tests under it, the two failures being assertions on slot numbers.
    • What the order-based integrity check loses, a cascade of the children's AEAD tags into each parent's AAD gives back, order-free: the client places a read's rows by the chain's tags and opens each node under the digest of its children's tags. With that, the oracle passes 14 of 14 and every hostile-row test still refuses.

    The write-up with the measurements, the comparison with a selector-keyed order, and what remains visible (the selector multiset, the element grouping of q, the heads, the lengths) goes to the owner separately.

    https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

  3. self-assigned this
    on Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

SDKenhancementNew feature or requestneeds-triageMaintainer needs to evaluate the issue

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions