diff --git a/docs/plans/2026-10-17-json-tree-index.md b/docs/plans/2026-10-17-json-tree-index.md index 9d963b8d6..75d8c6f0b 100644 --- a/docs/plans/2026-10-17-json-tree-index.md +++ b/docs/plans/2026-10-17-json-tree-index.md @@ -124,6 +124,8 @@ Arrays. `jsonb_array_length(doc, t)` is the count of children of `t`; `jsonb_arr > **2026-10-24:** as written, a positional token names the element's children, not the element: a scalar at `$.a[2]` has no children and its selector is every sibling's, so `subtree(doc, t($.a[2]))` could neither return nor identify it (review of #1250). The implementation reaches a position from its parent: the client sends the parent's token and a position gate `#2:` (`JsonSubtreeKey::gates()`), `json_tree_subtree_at` admits the entry at ordinal 2 only if it carries the element selector, and returns that entry itself at depth 0 with its descendants below it. See the Phase 3b and 3c notes below (#1329). +> **2026-10-24 (design change, not yet built):** an object member's token drops its ordinal, every entry gains a self tag `x`, and the client expands wildcards itself, so it derives the token of any wildcard-free path and gates go away. See "Design change (2026-10-24)" at the end. + #### The sealed root: no root token, the document at the root (decided 2026-10-18) Two properties make a token worse than a selector. It is a capability over structure rather than a presence bit, and it is deterministic per column, so a server that has seen it once can re-walk every row for as long as the column lives. The chain as written above hands the server the **root** token in two legitimate queries: a wildcard extraction anchored at the root (`$[*].x`) and `jsonb_array_elements` on a root array. `t($)` unlocks the shape of every document in the column, once, forever. Any shallow token is nearly as broad. @@ -241,6 +243,8 @@ With the sealed-root option off, the root entry's `c` is the root's label and ki > **2026-10-22:** as built, a stored document is `{"v", "i", "h", "sv": [{s, c, p?, op?}], "q"}`, with no `k`; a containment needle is a projection, `{"sv", "el", "alt"?}`, with no `grp`; and a subtree request is a token and one gate per step below it. The size estimates below predate both; the measured sizes are under Phase 3b. +> **2026-10-24 (design change, not yet built):** an entry becomes `{s, c, x, p?, op?}`, with the eight-byte self tag `x` on every entry, and a subtree request becomes a list of tokens with no gates. See "Design change (2026-10-24)" at the end. + **Size.** Per 16-byte field, hex costs 32 characters plus about six of framing. Measured against the envelope RFC's fixture document (`{"hello":"world-1","nested":{"deep":"constant"},"number":1}`, 5 nodes): | | v3 today | proposed | @@ -284,6 +288,8 @@ On a flat five-node document the proposal is about 8% larger: the two chain fiel > **2026-10-23 (review of #1336):** the "proposed" column of the shape-equality and parent-and-child rows is wrong for the format as built, and was wrong for the design as written. Neither the tags nor their salt are what decides it: the entries of `sv` (and the pairs of `q`) are stored in depth-first pre-order, an object's members in sorted key order, under selectors that are the same in every row. A row's sequence of selectors therefore fixes its shape exactly, the parent of every node is inferable by comparing rows (and is exact for the top-level fields under the sealed root, whose entries are the ones with no `p`), every array's length is the count of its element selector, and the relative order of sibling key names is visible across rows. `packages/eql/docs/reference/json-tree-leakage.md` states the format as built; the decision this leaves open is recorded under Phase 4. +> **2026-10-24 (design change, not yet built):** the "subtree extraction" and "broadest token" rows change: a read sends the tokens of exactly the nodes asked for, never a top-level field's or an array's unless that is what is read. See "Design change (2026-10-24)" at the end. + ### 7. Performance - **Encrypt:** per node two PRFs (`s`, `vs`), three hashes (`t`, `p`, mask), one small AEAD. v3 does two MACs and one AEAD over the subtree bytes. Fewer bytes encrypted at every depth above the leaves. @@ -373,7 +379,7 @@ The index, the SQL and the live oracle landed in the pull request stacked on thi - *Nonces are random, not derived from the selector and ordinal.* Every entry seals through the engine's one AEAD path (`vitaminc-encrypt`'s AES-256-GCM with its own random 96-bit nonce, carried in `c`), so there is one leaf cipher in the crate and no second AES-GCM call site. Uniqueness under one document key is then probabilistic (`N² / 2^97` at `N` entries) rather than structural. The entry's AAD still binds its ordinal `j`, so a moved or reordered entry fails to open as the design requires; the cost is twelve bytes per entry, which Phase 3 should count. - *The ordinal is `o`, not `i`*, because `i` is the envelope's identifier, which `->` grafts onto an extracted entry. An extracted entry also carries `j`, its ordinal in the document, which its AAD is bound to, and a walk adds `d`, its depth below the start. (2026-10-22: since Phase 3b `o` is not stored at all.) -- *The subtree walk returns a node's descendants and not the node itself.* The walk addresses nodes by their parent, and the design's wire has no field that names a node by its own token. The client reads the node's kind from its children's labels; a node with no descendants, an empty container or a scalar, yields no rows, and `->` on its selector returns the node. A self tag per node would close this at sixteen bytes an entry and is left to Phase 4. (2026-10-22: a gated walk, `json_tree_subtree_at`, now returns the node its gates reach at `d = 0`, with no self tag; the ungated `json_tree_subtree` still returns descendants only.) +- *The subtree walk returns a node's descendants and not the node itself.* The walk addresses nodes by their parent, and the design's wire has no field that names a node by its own token. The client reads the node's kind from its children's labels; a node with no descendants, an empty container or a scalar, yields no rows, and `->` on its selector returns the node. A self tag per node would close this at sixteen bytes an entry and is left to Phase 4. (2026-10-22: a gated walk, `json_tree_subtree_at`, now returns the node its gates reach at `d = 0`, with no self tag; the ungated `json_tree_subtree` still returns descendants only.) (2026-10-24: an eight-byte self tag on every entry is now the design; see "Design change (2026-10-24)" at the end.) - *Containment is not `STRICT` in SQL.* PostgreSQL inlines a `STRICT` SQL function only when it can prove the body is NULL for a NULL argument, and the `OR` that adds the needle's `alt` probe is not provably NULL; un-inlined, the call was an opaque filter and the GIN index never engaged. `eql_v4.json_tree_contains` and the `@>` / `<@` wrappers over it are declared without `STRICT` and return NULL for a NULL side on their own. - *`<@` cannot match a stored root scalar inside a needle root array*, since a stored document carries no `alt` pair. The oracle records the case; `@>` is exact. - *Value selectors compare strings byte for byte*, with no Unicode normalization, because `jsonb @>` does, and the oracle is agreement with `jsonb @>`. The scalar equality index normalizes to NFC; a JSON value does not. Numbers canonicalize by value, so `2` and `2.0` are one value, as `jsonb` compares them. @@ -399,7 +405,7 @@ Still open from the measurements, filed as a follow-up: `->` scans `sv` on every A review of the stored form after the Phase 3 measurements found three fields nothing read and two the walk did not need, and replaced the chain step. The wire is now `{"v", "i", "h", "sv": [{s, c, p?, op?}], "q"}`. - **`vs`, `n` and `g` are gone from the entries.** Once `q` was stored (Phase 3), no SQL function read an entry's value selector or group tags: containment reads `q` alone, and the engine read them only to rebuild `q` on deserialize. The engine now takes the stored `q` as the projection. That is no change in trust: `q` and the entry fields were equally unauthenticated (only `c` is, through its AAD), and a tampered `q` changes which rows a containment query returns and nothing else. On the nested benchmark document (213 entries on average) the values of those three fields were 15% of `sv`. -- **Tokens chain by ordinal.** `t(child) = H("t" ‖ t(parent) ‖ LE32(o))` replaces the label-derived step. The server derives every hit's token from its parent's as it walks, so `w` (the wrapped token) is not stored. The cost is that the client cannot derive an object member's token, because an ordinal is a rank among sibling keys it does not know. It derives a chain head's token and those of the array positions that directly follow the head, and sends one gate per further step (`JsonSubtreeKey::gates()`: the step's selector, or `#n` for a position); `json_tree_subtree_at(doc, token, gates, budget)` descends through them, and `json_tree_children` and `json_tree_child_count` take the same gates. A wildcard step's selector is every element's, which is the old `subtree_at` gate generalised to every depth. +- **Tokens chain by ordinal.** `t(child) = H("t" ‖ t(parent) ‖ LE32(o))` replaces the label-derived step. The server derives every hit's token from its parent's as it walks, so `w` (the wrapped token) is not stored. The cost is that the client cannot derive an object member's token, because an ordinal is a rank among sibling keys it does not know. It derives a chain head's token and those of the array positions that directly follow the head, and sends one gate per further step (`JsonSubtreeKey::gates()`: the step's selector, or `#n` for a position); `json_tree_subtree_at(doc, token, gates, budget)` descends through them, and `json_tree_children` and `json_tree_child_count` take the same gates. A wildcard step's selector is every element's, which is the old `subtree_at` gate generalised to every depth. (2026-10-24: to be reversed for object members, whose tokens drop the ordinal, and with them the gates; see "Design change (2026-10-24)" at the end.) - **The walk probes to the first miss.** The ordinals under a node are dense from zero (positions for arrays, a permutation for objects), so the walk tries `0, 1, 2, …` and stops at the first miss. `o` is not stored and `json_tree_fanout` is gone. A subtree of `N` nodes costs about `2N` probes (`N` hits, and a miss for every node visited), each a SHA-256 and a map lookup, plus a SHA-256 per hit for its token, where it cost `F` probes per visited node, `F` being the widest node in the whole document; on the nested corpus a wide `items` array taxed every object walk in the same document. - **Depth is reported from the reached node.** A gated walk returns the node its gates reach with `d = 0` and its descendants beneath it, so the client can tell "the value at the path" (one reached node) from "the children of a node" (no `d = 0` row) without the path. A wildcard reaches several nodes and opens as a list. The ungated `json_tree_subtree(doc, token)` is now `json_tree_subtree_at` with no gates, and returns the token's node's descendants from `d = 1` as before. - **Leakage.** At rest, the same as Phase 3 less what the stored ordinals told: their multiset gave, for every `k`, how many nodes in the chain have more than `k` children. `p` is still salted by the document `iv`. Under a token, the same: a token still unlocks the shape of that subtree in every row. The `w` mask's binding to `c` (a swapped ciphertext broke the walk on the server) is gone; the swap still fails when the client opens the entry, because `c` is bound to `s` and `j` in its AAD. @@ -513,3 +519,128 @@ Two of the three items that gate a release are closed here; the third, the final 9. **Native `@>` over the element-nested projection, built from tags or from a nested `sv`.** Both give identical leakage; the first keeps the flat `sv` every other function reads, the second drops two fields from the wire and changes `->` to a `$.**` lookup. Proposal: projection from tags first, measure both in Phase 2. 10. **Position-keyed parent tags on by default.** They remove sibling linkability at rest for one small integer per entry and `W × F` hashes per walk. Proposal: on, unless Phase 3 shows the walk cost matters on large documents. 11. **Key the label step of the token chain.** As written, a child's token is an unkeyed hash of its parent's token and its label, so whoever holds a token can test guesses of every key name beneath it (see "What a token is" under the token chain). Deriving the step under the index key, `PRF(json/token ‖ column, t(parent) ‖ 0x01 ‖ key)`, closes that at no cost to the server, which unwraps child tokens rather than deriving them. The same holds for anything else the client derives from a label under a token. Proposal: key it. + +## Design change (2026-10-24): derivable tokens, the self tag, and wildcard expansion on the client + +**Status:** agreed with the owner on 2026-10-24 and not yet implemented. The implementation is stacked on this write-up (see "Plan" below), and the leakage page (`packages/eql/docs/reference/json-tree-leakage.md`) is revised here to describe the format after the change, so neither should reach `next` without the other. Tracked in the issue named in the pull request that adds this section. + +### Why + +Since Phase 3b a child's token is `t(child) = H16("t" ‖ t(parent) ‖ s(child) ‖ LE32(o))`, where `H16` is the first sixteen bytes of SHA-256, `s(child)` is the child's path selector and `o` is its **ordinal**, its place among its siblings. An array element's ordinal is its index. An object member's is its rank among its siblings: the members are sorted by `PRF(json/rank ‖ column, PAE(t(parent), key))` (`rank` and `shuffled_ordinals` in `packages/stack-encrypt/src/json/derive.rs`, the design's decision 11), so a member's ordinal depends on which other keys the object holds in that row. The client does not know a row's sibling keys, so it cannot derive the token of any node below a top-level field except through array positions (`subtree_key_for`). Everything else this document has had to add to the read path follows from that one fact: + +- **Gates exist only for it.** From the first key step below a head, the client stops deriving tokens and sends one gate per remaining step, which the server walks through (`json_tree_subtree_at`). A read of `$.profile.address.city` therefore sends `$.profile`'s token: a capability over that whole top-level field, in every row of the column, for as long as the column lives, to read one leaf. +- **It is why a read could not return the node it names.** A walk finds a node by its parent's token, and a chain head has none, so the ungated walk returned a node's descendants and never the node: `{}`, `[]`, a scalar and a missing field all came back as no rows (#1356). A read of `$.tags[1]` derived the element's own token and so had the same problem. +- **The patch for that widens the leak.** PR #1359 returns the node by sending a head's selector beside its token, and turns a path's last array position into a gate from the array's token (`#n:`). A read of `$.tags[1]` then sends `$.tags`'s token, which walks every element of that array in every row. + +Three changes remove all of it: a member's token stops depending on its ordinal, every entry carries a tag derived from its own token, and the client expands wildcards into concrete paths itself. + +### 1. Ordinal-free member tokens + +``` +t(member) = H16("t" ‖ t(parent) ‖ s(member)) object member: no ordinal +t(element) = H16("t" ‖ t(array) ‖ s(element) ‖ LE32(i)) array element: i is its index +p(child) = H8("p" ‖ iv ‖ t(parent) ‖ LE32(o)) unchanged; o is the stored ordinal +``` + +`H8` is the first eight bytes of SHA-256, as Phase 3d made the parent tag. Object keys are unique, so a member's selector alone separates it from its siblings, and the ordinal adds nothing to its token but the dependency on the other keys. Every element of an array shares one selector (the wildcard path's), so an element keeps its index. Since an element's ordinal already is its index, an element's token is unchanged. Chain heads and the root derive as today: under the sealed root a head's token is `PRF(json/token ‖ column, path)`, and with the root open the root's token is that PRF of `$` and a top-level field's follows the member rule from it. + +**Ordinals stay, for enumeration only.** The parent tag `p` keeps the ordinal, and an object member's ordinal is still its keyed rank. Enumerating a node's children is still a probe of `o = 0, 1, 2, …` until the first miss, and the keyed rank still keeps a token's holder from ranking guessed key names against the ordinals the probe reveals. Only the token stops depending on it. + +**The consequence is the point of the change: the client derives the token of any wildcard-free path from the path alone.** Head, then one hash per step, the step's selector from the path and, for a position, the index from the path. A token sent therefore always covers exactly the node asked for, and the gates have no reason to exist. + +**An element and a member still never share a token.** The Phase 3b review (#1329) bound the child's selector into the step so that the token of `$.a[0]` could not be, in a row whose `$.a` is an object, the token of whichever member ranks first. That property holds under the new rule twice over: an element's selector is the wildcard path's and a member's names its key, so the hashed inputs differ; and the inputs have different lengths (33 bytes for a member, 37 for an element), so they differ even if two selectors ever coincided. A member and an element cannot sit under the same parent in one row, and across rows a token for `$.a[0]` finds nothing in a row whose `$.a` is an object, because no entry there carries that token's self tag (next section). + +**The server cannot tell which rule a hit takes, and the self tag tells it.** As it walks, the server derives each hit's token from its parent's to go one level further. Today that is one rule for every hit. Under the new rules it needs to know whether the hit is a member or an element, and nothing stored says so: the kind is inside `c`, and a selector is a keyed hash that does not say whether its path ends in a key or a wildcard. So the walk derives the member candidate `H16("t" ‖ t ‖ s)`, keeps it if the hit's stored self tag `x` is `H8("x" ‖ iv ‖ candidate)`, and otherwise derives the element candidate with the hit's ordinal and checks that. A hit whose `x` matches neither is returned but not descended into; the client's check refuses it. That is up to four hashes per hit where the walk spends one today, on top of the two probes per node; the benchmarks below measure it. + +**Key names stay out of reach.** A member's token now hashes its selector where it hashed its ordinal. The selector is a PRF under the index key, so whoever holds a parent's token still cannot test a guessed key name: they would need the guess's selector, which takes the index key. Nothing about decision 11 moves. + +**The AAD does not change.** An entry's AAD (`entry_aad`) binds the keyset, the column's AAD, the data key's tag, the entry's selector `s` and its position `j` in `sv`. It does not bind the sibling ordinal `o`, the token, `p` or the new `x`, and it need not: the change moves only fields the server derives and compares, none of which is authenticated today. + +### 2. A self tag on every entry + +``` +x(node) = H8("x" ‖ iv ‖ t(node)) stored on every entry: the root, the heads and every node below +``` + +The self tag lets the server find an entry from its own token, where `p` finds it only from its parent's. It is salted by the document's `iv` like `p`, so at rest it is random bytes that link nothing within a row or across rows until a query supplies a token. It is on every entry, so its presence marks nothing: unlike `p`, which the heads lack, no entry is singled out by carrying it. + +**The sealed root carries one.** The root's token is derivable from `$` alone, with the root sealed or open, so the root entry gets `x = H8("x" ‖ iv ‖ PRF(json/token ‖ column, "$"))`. Today a sealed root has no token at all (the walk gives it none, and `json_subtree_key` refuses `$` as `TermError::JsonSealedRoot`). Under the sealed root nothing chains from `t($)`: the heads derive from their own paths and carry no `p`. So the root's token finds the root entry and nothing beneath it, which is the document's ciphertext the budget fallback already returns, and a read of `$` becomes an ordinary read that opens the whole document. The client still does not send `t($)` unless `$` is what is read. With the root open, `t($)` remains what it is today, the whole column's shape, and a read of `$` sends it. + +**What a read returns.** The node at depth 0, found by its `x`, then everything beneath it, enumerated under the node's own token. So `{}`, `[]`, a scalar and an absent node come back as an empty object's entry, an empty array's entry, the scalar's entry and no rows: the four distinct answers `jsonb #>` gives, which is what #1356 asks for and the semantics #1359 implements. An entry's ciphertext already carries its kind and child count, so the client tells `{}` from `[]` from the entry alone. + +**Collisions.** Eight bytes, like `p`. A probe with a token that names no node in the row matches some stored `x` with probability about `E / 2^64` for a row of `E` entries, per token. A false hit is an authentic entry of the row at the wrong place, which the client's check refuses (section 4), so a collision costs a refused read, not a wrong answer. + +**Size.** Eight bytes is twelve characters of padded base64, plus `,"x":""` framing: about 19 characters per entry, against roughly 200 per entry today, so about 10% more stored characters. Random bytes do not compress, so the on-disk share should be about the same. To be measured (section 7). + +### 3. Wildcard expansion on the client, and no gates + +A path with a wildcard (`$.items[*].price`) names no one node, so it has no token. Today the server resolves the wildcard by walking from the deepest derivable ancestor through a selector gate. After the change the client resolves it: it expands each wildcard step to the indices `0 … N−1` and sends one token per concrete path, plus one **probe** token for index `N` of each array it expands. For `$.items[*].price` that is `t($.items[0].price)` … `t($.items[N−1].price)` and the probe `t($.items[N])`. The probe is the element itself, not the element's `price`, because an element past the bound need not have one. + +- **The bound.** `N` defaults to 64, in the spirit of the walk's default budget, and is set by the caller where the read is derived. In a row where a probe hits, an array at that level is longer than `N`, and the read fails loudly with an error that names the bound (and not the path, whose key names can be customer data). It never returns a partial answer. The caller can raise `N`, or read the array itself: a read of `$.items` sends the array's own token, which is honest, because the whole array is then what was asked for. +- **Nested wildcards multiply.** Two wildcard levels send `N²` paths, `N` probes at the inner level and one at the outer: `N² + N + 1` tokens, 4,161 at the default. Each token costs the server one hash per row it scans, so a nested wildcard over many rows wants a smaller `N`. +- **A wildcard head becomes readable.** Under the sealed root, `$[*].x` on a root array was refused, because a wildcard head names no one node. Each `$[i]` is a head with its own token, so it now expands like any other wildcard. + +**The SQL.** `json_tree_subtree_at` and its gates are removed. The read is + +``` +eql_v4.json_tree_subtree(doc, tokens text[], budget integer DEFAULT 64) +``` + +For each token in turn, it looks the entry up by `x` (one map of the document's entries keyed by `x`, built once per call beside the existing map keyed by `p`), returns it at depth 0, then returns its descendants by ordinal probing under the node's own token, deriving each hit's token as section 1 describes. Rows come out in token order, and within a token in document order. A token with no entry yields no row. `json_tree_children(doc, token)` and `json_tree_child_count(doc, token)` lose their gates and take any derivable node's token: they enumerate one level below it, which needs no child token and so no rule choice. + +**The budget.** The walk counts rows across all the tokens of one call, the depth-0 rows included. With the root sealed and the count past the budget, it returns the root entry alone at depth 0, as today. The client opens the root to the whole document and evaluates the same concrete paths and the same probes over it. A probe that finds an element there is the same truncation error the walked read raises, so one row gives the same answer under the budget and over it, which is the property the Phase 3b review (#1329) held the fallback to. The client could answer a wildcard in full from the document, but then the answer would depend on the budget; see the open questions. With the root open there is no fallback, as today. + +**The walk's bound needs rewording.** "The bound" in `functions.sql` argues that no two rows share a token because a hit's token is a hash of its probe, which includes the ordinal. A member's token no longer includes it: two entries forged under one parent with the same `s` and the member self tag would derive one token, and the walk would visit the subtree beneath it twice. The walk's other limit, `LIMIT` the entry count plus one, still caps the rows, and the client refuses a repeated position (`j`) as it does today. The comment has to say so. + +### 4. The client's check after the change + +PR #1358 made the client open a walk's rows only through the key it derived and refuse any row that does not fit (`Error::JsonWrongEntry`). It adapts: + +- **Each depth-0 row** must carry `x = H8("x" ‖ iv ‖ t)` for one of the tokens sent, and its authenticated selector and decrypted label must agree with that token's path: the path's selector, and the key or index its last step names. +- **Each deeper row** must sit at the next consecutive position `j` within its subtree, and its decrypted label must agree with its selector, as #1358 checks today; and it must carry an `x` (and a `p`) that matches the child token the client recomputes from its parent's token, the row's selector and its label (the key for a member; the index, from the label, for an element). +- The child counts in each container's ciphertext still catch a dropped row, and a repeated position is still refused. + +`x` and `p` are not authenticated. The database holds every token a read sends and every row's `iv`, so it can write any `x` or `p` it likes. Matching them catches a row corrupted by someone who never saw the token, and a server that walks wrongly by mistake, but not a dishonest database. What binds a row to its place against the database is what the AAD covers: the selector, the position `j`, and the label and child count inside `c`. The check that each subtree's rows are consecutive in `sv` is on `j`, so it is one of those. Nothing replaces it: the `x` check is a consistency check, and the bound fields with the consecutive-position rule are the security check, as the leakage page says. Open question 1 shows what the rule refuses that nothing else does. + +Still allowed by design (ADR 0010, `packages/stack-encrypt/docs/adr/0010-what-a-stored-value-authenticates.md`): an entry swapped in from another row under the same column context, returned with that row's header, opens. Row provenance is the database's word. + +### 5. Wire + +A stored entry becomes `{s, c, x, p?, op?}`: `x` on every entry, `p` below the heads as now. The domain CHECK on `public.eql_v4_json_tree_entry` and the `JsonTreeEntry` newtypes in `packages/eql/crates/eql-bindings/src/v4/json_tree.rs` gain `x` as padded base64 of eight bytes. Nothing else in `h`, `sv` or `q` changes. Counted informally, this is the format's sixth stored form, revision 5 to 6; no field records that number (see the open questions). EQL v4's `json_tree` is unreleased, so nothing stored breaks: every fixture, every pinned vector and every benchmark table is regenerated. + +### 6. Leakage after the change + +`packages/eql/docs/reference/json-tree-leakage.md` carries the full statement; in short: + +- **A subtree read sends exactly the tokens of the nodes asked for.** A read of `$.profile.address.city` sends that leaf's token and nothing that covers `$.profile`. No capability over a top-level field or an array leaves the client unless that field or array is itself what is read. +- **A wildcard read sends `N + 1` tokens for each array it expands**, `N² + N + 1` in all for two levels. The database learns which of them exist in each row, so, up to `N`, which elements hold the rest of the path (each array's length, where the path ends at the element, which the stored entry order already shows, #1357), whether each array is longer than `N`, and the structure beneath each node found. It learns nothing about the elements' sibling fields. One exception: a probe that hits is a token for the element at index `N`, and the server enumerates that element's subtree in each row where it exists, although the read then fails. The exception is in the token, not in what the SQL does with it: whoever holds an element's token can derive that element's subtree in any row, now or in rows written later, so it holds for every variant of the read that sends the element's token (open question 3). +- **Unchanged:** a token is still column-wide and still does not expire; the `->` extraction path and containment send and reveal what they do today; at rest, `x` adds nothing beyond what `p` already shows; and the stored-order disclosure (#1357) is untouched by this change and stays a separate decision. + +### 7. Plan + +1. **This write-up**, as a draft pull request against PR #1358's branch. +2. **The implementation**, stacked on this branch, so the code and the revised leakage page land together. Stack Encrypt: the member rule and `x` in `derive.rs` (`child_token`, the walk, `seal`), concrete-path expansion in `path.rs`, and `JsonSubtreeKey` carrying the expanded tokens, which of them are probes, and the bound. SQL: `json_tree_subtree(doc, tokens, budget)`, the `x` map, the member-or-element choice, `json_tree_children` and `json_tree_child_count` from a token, the rewritten bound comment, and `json_tree_subtree_at` and the gates removed. `eql-bindings`: `x` on `JsonTreeEntry`, the CHECK, and `JsonTreeSubtree::from_key`. The Go guest has no `json_tree` yet and is untouched. Tests: + - pinned derivation vectors for a member token, an element token, a head token, the root token and `x`; + - a property test (proptest) that `t($.a.b)` is the same in rows whose `$.a` has different sets of sibling keys, which is the central property, and that element tokens differ by index; + - a property test, and the PostgreSQL oracle, that every wildcard-free read equals `jsonb #>` for all four answers (`{}`, `[]`, a scalar, absent), at a head, under array positions and deeper, with the root sealed and open, at budget 0 and budget 1; + - wildcard reads equal `jsonb_path_query` when every array is at most `N` long, and fail loudly with the bound when one is longer, nested wildcards included; + - the sealed-root fallback gives the same answers, the truncation error included; + - hostile PostgreSQL tests: a forged `x`, a row from another node, a moved subtree, a truncated walk, and a swapped entry from another row, which must still open (ADR 0010); + - `x` present on every entry, the root and the heads included; + - a stored-size regression test with a tolerance; + - `cargo mutants --in-diff` on both crates. +3. **Measure**, by a separate agent in `cipherstash/benches`, branch `feat/json-tree-benches` (crate `json-tree-bench/`, which builds Stack Encrypt and `eql-bindings` from the stack branch named in its `Cargo.toml`), against the local PostgreSQL on port 5400, database `jsonbench`, at 10,000 and 100,000 rows. A wire change means dropping the `json_tree` tables, reinstalling the v4 bundle built from the implementation branch, and re-ingesting. To report: the stored-size delta per corpus, in characters and `pg_column_size`; read latency of the token-list read against today's walk at `N` = 8, 64 and 256; the cost of the probes and of the member-or-element choice; and ingest time. The `check:v4:json_tree` correctness task calls the gated walk and has to move to the token list first. +4. **Close #1359 as superseded** once the implementation pull request is open. Its mechanism (a head found by selector, the last position sent as a gate from the array's token) is replaced. Its semantics carry over: the four answers, `JsonSubtree` opening as `Option` to `None` where the row has no node, and the oracle test `postgres_every_subtree_read_answers_as_jsonb_does`, rewritten for the token list. + +**Merge order.** #1345 (ready) rewrites `packages/stack-encrypt/tests/json_index.rs`, which #1358 also touches, and should merge first; #1358 then rebases, and this branch and the implementation follow it. + +### Open questions + +1. **The consecutive-position check stays; `x` does not replace it.** `x` and `p` are unauthenticated, and the database can compute both from the tokens it was sent and the row's `iv`, while `j` is bound by the AAD. So matching them is a consistency check, and the check that each subtree's rows are consecutive in `sv` remains part of the security check. It is what refuses a node spliced in from under another element of the same array. A read of `$.items[0]` returns the element at depth 0 and then its members. Answered with the element's own entry followed by `$.items[1].price` in place of `$.items[0].price`, with `x` rewritten to the self tag the client recomputes for `$.items[0].price`, it passes every other check: every element shares the wildcard's selector, so the selector is right; the label is `price`; and the element's child count is met. Only the position refuses it, because a member must sit at the position after the row before it. The rule does not reach a node that is itself the depth-0 row: a read of `$.items[1].price` is one row with nothing before it to fix its position, so `$.items[0].price` with its `x` rewritten opens in its place, as the leakage page says. Proposal: keep the consecutive check and add the `x` check beside it. Checking `p` adds nothing over `x`, and for a member it needs every sibling's key to recompute the rank, so it could be dropped. +2. **The server needs the self tag to tell a member from an element** (section 1). The two rules leave the walk unable to choose which one derives a hit's token. Section 1 resolves it by testing the member candidate against `x`, at up to four hashes per hit instead of one. The alternative, a stored kind bit, would reveal at rest which entries are array elements. Confirm the choice, and the cost once measured. +3. **A probe that hits unlocks one element the caller did not ask for.** A probe is the token of the element at index `N`, so where that element exists the server enumerates its subtree, and the leakage page has to carve out an exception to "exactly the nodes asked for". The capability is in the token, not in what the SQL does with it. Whoever holds `t($.items[N])` can find that element in any row by its self tag, probe for its children's parent tags, and derive every descendant's token by the public derivations of sections 1 and 2, whatever the function returns, and can keep doing so in other rows and in rows written later. So a separate `probes text[]` argument answered with existence only would not remove the exception: it holds for every variant that sends the element's token. Only a different derivation would. For example, the client could send `H16("e" ‖ t)` in place of the token, with each entry storing a tag the server can match from that value (an existence tag beside `x`, or `x` itself derived from it), so that the server learns whether the element exists in each row but cannot derive its token, its parent-tag probes or anything beneath it. That is a change to the stored format and a future option, not this design. It would also tell the server which values are probes, which it can already pick out: a probe is the last token of its level, and the token count gives the bound away. Proposal: send probes as ordinary tokens and keep the exception. +4. **The fallback's answer to a long array.** Section 3 has the fallback apply the same bound, so one row's answer does not depend on the budget. The client could instead resolve the wildcard in full from the document it holds. The two answer differently for a row with an array longer than `N` under the budget versus over it. Decide which. +5. **No field records the wire revision.** The header's `JsonHeader::FORMAT_VERSION` is still `1` and the document's `v` is `4` through all five earlier stored forms. `packages/stack-encrypt/src/json/mod.rs` says a changed derivation is a new version, never an edit in place, but the chain step is a plain SHA-256 with the domain byte `"t"`, not a PAE domain label that can be bumped. Since nothing is released, the choice is between editing in place (as the earlier forms did) and bumping `FORMAT_VERSION` to `2` so that a reader refuses an old document instead of misreading it. +6. **A read of `$` under the sealed root becomes legal.** Today `json_subtree_key` refuses it (`TermError::JsonSealedRoot`), and the root has no token. With `x` on the root, `$` returns the root entry, which is what the fallback already returns. Confirm that `$` should become a read rather than stay refused. +7. **`json_tree_child_count` of an absent node.** Today it is `0` for a token that names no node. `jsonb_array_length` of a missing path is NULL, and with `x` the function can tell an absent node from an empty one. The reasoning of #1356 suggests NULL. +8. **The leakage page describes the format after the change.** It otherwise describes the format as built, so this branch must not reach `next` ahead of the implementation. Stacking the implementation on this branch is the proposed way to make sure. diff --git a/packages/eql/docs/reference/json-tree-leakage.md b/packages/eql/docs/reference/json-tree-leakage.md index f70901e7b..97b38af2c 100644 --- a/packages/eql/docs/reference/json-tree-leakage.md +++ b/packages/eql/docs/reference/json-tree-leakage.md @@ -17,14 +17,15 @@ Someone who holds the column and no key, such as a database administrator, a bac | field | what it reveals | |---|---| | `s`, the path selector on every node | **Which paths the row has, and how many nodes sit at each.** A selector is a keyed hash of the path, the same in every row of the column, so two rows with a selector in common hold the same path. Key names are inside the hash and not readable. Array elements share their array's selector, so `$.tags[0]` and `$.tags[1]` look alike, but each element is its own entry: the number of entries carrying `$.tags[*]`'s selector is the length of `$.tags` (the total over every array at that path, when the path runs through another array). | -| the order of the entries in `sv` | **The row's shape.** Entries are stored in depth-first order, the root first, each node followed by everything beneath it, and an object's members in the sorted order of their key names. So a row's sequence of selectors fixes its shape exactly: two rows with the same sequence have the same tree with the same key names, and two rows that differ show where. Which selector's path sits under which is not written down, but it is the same in every row, so an observer comparing rows learns it, and with it each row's parent and child relations and the node count of every subtree. Siblings stored in sorted key order also show, across rows, **the alphabetical order of sibling key names** (for example that the key behind one selector sorts before the key behind another), which narrows guesses at what each selector names. | -| `q`, the containment projection: one `{s, vs}` pair per node, nested by array element, in the same order | **Whether two rows hold the same value at the same path**, through the value selector `vs`, a keyed hash of the path and the value (for an object or an array, of its kind alone). So at each path the database sees which rows agree and how often each value occurs: a frequency histogram of every path, and whether one row's document contains another's. Not the value itself, and not whether two different paths hold the same value. **Which nodes sit inside the same array element**, through the nesting, for every element that is an object or an array. That grouping is what makes containment exact for arrays of objects, and it is the one thing this format reveals that v3 did not. | +| the order of the entries in `sv` | **The row's shape.** Entries are stored in depth-first order, the root first, each node followed by everything beneath it, and an object's members in the byte order of their key names' UTF-8 encodings (the order of serde_json's default map; a build that turns on serde_json's `preserve_order` feature stores them in insertion order instead, which shows the application's field order). So a row's sequence of selectors fixes its shape exactly: two rows with the same sequence have the same tree with the same key names, and two rows that differ show where. Which selector's path sits under which is not written down, but it is the same in every row, so an observer comparing rows learns it, and with it each row's parent and child relations and the node count of every subtree. Siblings stored in key order also show, across rows, **the byte order of sibling key names** (for example that the key behind one selector sorts before the key behind another; byte order, not a locale's alphabetical order, so `Z` sorts before `a`), which narrows guesses at what each selector names. | +| `q`, the containment projection: one `{s, vs}` pair per node, nested by array element, in the same order | **Whether two rows hold the same value at the same path**, through the value selector `vs`, a keyed hash of the path and the value (for an object or an array, of its kind alone). So at each path the database sees which rows agree and how often each value occurs: a frequency histogram of every path, and whether one row's document contains another's. Not the value itself, and not whether two different paths hold the same value (the ordering terms below do show that, at the paths the index orders). **Which nodes sit inside the same array element**, through the nesting, for every element that is an object or an array. That grouping is what makes containment exact for arrays of objects, and it is the one thing this format reveals that v3 did not. | | `p`, the parent tag below the chain's heads | Nothing beyond the entry order on its own. It is salted by the document's `iv`, so the same position in two rows is unrelated bytes, and it does not say which entry is the parent. Which entries carry no `p` does show: with the sealed root on, those are the root's direct members, and since everything beneath a member follows it, **the node count of each top-level field's subtree** can be read off a single row exactly. With the sealed root off, only the root lacks one. | +| `x`, the self tag on every entry | Nothing on its own. It is a hash of the entry's own token salted by the document's `iv`, so it is unrelated bytes in every row and links no entry to another, and since every entry carries one, its presence marks none. Given a token, it is how the database finds that token's node in any row. | | each ciphertext's length, and the number of entries | **The node count**, and for each node the length of its key name (for an object member) plus, for a scalar, the length of its value, exactly, less the AEAD's fixed overhead. A string's length in bytes and a number's length in digits are therefore visible per node. With the sealed root on, the root entry's length is the document's length as JSON. | -| `op`, the ordering term, only on number and string leaves at the paths the index orders | **The order of that leaf's value among every row that holds one, and which rows hold equal values.** The term is order-preserving encryption (CLLW OPE) under one key per column, so leaves at two ordered paths compare with each other too. It is deterministic and not a random value: two values whose encodings share leading bytes (for a string, its first characters) get terms that share leading bytes, so the term also shows how long that common prefix is. A term's presence at an ordered path also says the leaf is a number or a string. By default no path is ordered; see the options below. | +| `op`, the ordering term, only on number and string leaves at the paths the index orders | **The order of that leaf's value among every row that holds one, and which rows hold equal values.** The term is order-preserving encryption (CLLW OPE) under one key per column, so leaves at two ordered paths compare with each other too, and two ordered paths holding equal values carry equal terms. Strings are ordered and compared after their case and accents are folded, so `"Abc"`, `"abc"` and `"ábc"` get the same term: the equality it shows is of folded values. It is deterministic and not a random value: two values whose encodings share leading bytes (for a string, its first characters) get terms that share leading bytes, so the term also shows how long that common prefix is. **Its length is the leaf's length**: one byte per bit of the value's order encoding plus one, so 65 bytes for every number and `8 × (n + 1) + 1` bytes for a string whose folded UTF-8 form is `n` bytes. A term's presence at an ordered path also says the leaf is a number or a string. By default no path is ordered; see the options below. | | `h`, the header | The keyset the data key was minted under, the data key's identifier (unique to the row), and whether the root is sealed. `v` and `i` are the version and the table and column identifier every EQL value carries. | -Not revealed at rest: any value's or key name's text, and whether two different paths hold the same value. Revealed at rest, by the entry order, although no field stores it: the shape of every row, the parent of every node, every array's length, and the sorted order of sibling key names. +Not revealed at rest: any value's or key name's text, and, except through the ordering terms at the paths the index orders, whether two different paths hold the same value. Revealed at rest, by the entry order, although no field stores it: the shape of every row, the parent of every node, every array's length, and the byte order of sibling key names. ## What a query shows @@ -33,18 +34,21 @@ A query hands the database a term derived from the plaintext it asks about, and - **Containment (`@>`, `<@`), and equality at a path, which is sent as a containment needle.** The needle's own projection: the selectors of its paths, the value selectors of its nodes, and its element grouping. So the database learns which paths and which (path, value) pairs you asked about, and which rows hold them. - **Path extraction (`->`, `->>`, `json_tree_path_query`, `json_tree_path_exists`).** The selector of the path asked. A position in the path reads as the wildcard, so a position is not sent. The entry returned is a ciphertext. - **Ordering at a path (`<`, `<=`, `>`, `>=`).** The path's selector and the bound's own ordering term, which compares with the stored terms exactly as they compare with each other: the database learns the bound's place in the order of the stored values at every ordered path, and whether it equals one. `ORDER BY ord_term(…)` sends no term and shows nothing the stored terms do not. -- **A subtree read (`json_tree_subtree`, `json_tree_subtree_at`, `json_tree_children`, `json_tree_child_count`).** A **token**, and one gate per step below it. The token is not the token of the node you asked for: it is the token of the deepest node on the path the client can derive, which is the top-level field the path starts with or, while the path goes on through array positions, the element they reach. Reading `$.profile.address.city` sends `$.profile`'s token. A gate is the step's selector or, for an array position, `#n:`, with the position `n` in the clear. A token is a capability over the structure of its node's subtree in *every row of the column*: whoever holds it can find each node under that field in any row, see how the nodes nest, link each one to its parent, and derive the tokens of the nodes beneath. It opens no value. Tokens are column-wide and do not expire, so a token sent once is useful to the database for as long as the column lives. Send a token to the database in a query and nowhere else. With the sealed root on (the default) the broadest token a query can send covers one top-level field's subtree, and no token covers a whole document. With it off, every subtree read whose path begins with a key sends the root's token, which covers every row's whole document. +- **A subtree read (`json_tree_subtree`, `json_tree_children`, `json_tree_child_count`).** A **token** for exactly the node the path names, which the client derives from the path alone. Reading `$.profile.address.city` sends that leaf's token and nothing that covers `$.profile`. No token over a top-level field or an array leaves the client unless that field or array is itself what is read. + - **A wildcard** (`$.items[*].price`) is expanded by the client into one token per concrete path, up to a bound `N` (64 by default, set by the caller), plus a **probe** token for the element at index `N` of every array a wildcard expands. A probe that finds an element means the array is longer than the bound, and the read fails rather than return part of the answer. So the database learns which of the nodes exist in each row, which shows, up to `N`, which elements hold the rest of the path (each array's length, where the path ends at the element; the entry order already shows it), whether each array is longer than `N`, and the structure beneath each node it finds. It learns nothing about the elements' sibling fields. In a row where a probe finds an element, the database also walks that element's subtree, although the read then fails. That exception is in the token, not in what the SQL function does with it: a probe is the element's own token, so the database can find that element in any row, enumerate beneath it and derive its descendants' tokens, in rows written later too, whatever the function returns. It holds for any read that sends the element's token. Two wildcard levels send `N² + N + 1` tokens, `N + 1` of them probes, one at index `N` of each array the read expands: the `N` inner arrays and the outer one. + - **What a token is.** A capability over the structure of its node's subtree in *every row of the column*: whoever holds it can find that node in any row (through its self tag `x`), see how the nodes beneath it nest, link each one to its parent, and derive the tokens of the nodes beneath. It opens no value. Tokens are column-wide and do not expire, so a token sent once is useful to the database for as long as the column lives. Send a token to the database in a query and nowhere else. + - **The root.** With the sealed root on (the default) no token covers a whole document: the root's token finds the root entry, whose ciphertext is the document, and nothing beneath it. With it off, a read of `$` itself sends the root's token, which covers every row's whole document; a read of anything below `$` does not. ## What is authenticated -Only the ciphertext `c`. Opening an entry verifies that it was sealed under this keyset, for this column, under the data key its row's header names (a fresh one for every row), with the selector `s` and the position `j` it is opened with; a ciphertext moved into another row or column, or opened under another selector or position, fails. Nothing in it names a row, so an entry that travels with its own row's header still opens; see below. When the client opens a whole document it takes each entry's position from its place in `sv`, so a moved, swapped or re-labelled entry fails to open. The selectors, the projection, the parent tags, the ordering terms, the header's sealed-root flag, and which entries a row holds are not authenticated, by the same rule every EQL stored value follows (ADR 0010 in `packages/stack-encrypt/docs/adr`). +Only the ciphertext `c`. Opening an entry verifies that it was sealed under this keyset, for this column, under the data key its row's header names (a fresh one for every row), with the selector `s` and the position `j` it is opened with; a ciphertext moved into another row or column, or opened under another selector or position, fails. Nothing in it names a row, so an entry that travels with its own row's header still opens; see below. When the client opens a whole document it takes each entry's position from its place in `sv`, so a moved, swapped or re-labelled entry fails to open. The selectors, the projection, the parent tags, the self tags, the ordering terms, the header's sealed-root flag, and which entries a row holds are not authenticated, by the same rule every EQL stored value follows (ADR 0010 in `packages/stack-encrypt/docs/adr`). The consequence to design around: tampering with those fields changes **which rows a query returns, where a walk stops, and which entries of a row an extraction or a walk hands back**. An entry the database returns from `->` or a walk carries its own `s` and `j`, and the header it carries names the data key, so a decrypted value is always one that was sealed in this column. The client also checks that it is the entry it asked for, from what it derived for the query, and refuses anything else (`JsonWrongEntry`): - **`->`.** The entry must carry the selector the client derived for the path. Another path's entry, such as the row's `$.bonus` returned for `$.salary`, is refused, and so is the sealed root returned for a field (it opens to the whole document); the root is accepted only when `$` itself was asked for. -- **A walk.** A node a gated walk reaches must carry the path's selector, and the position the path ends in, if it ends in one. Every other row must carry the selector of its parent's path extended by its own key (or the wildcard, for an array element), which the client derives from the decrypted key, and a subtree's rows must follow each other in the document with no gap, through to the last row the node's child count calls for. A row of another path, a reordered row, a row whose parent is not among the rows, and a row left out of a node's subtree are refused. +- **A subtree read.** The node each token names, returned first, must carry that path's selector, the key or position the path ends in, and the self tag of one of the tokens sent. Every other row must carry the selector of its parent's path extended by its own key (or the wildcard, for an array element), which the client derives from the decrypted key, and a subtree's rows must follow each other in the document with no gap. A row of another path, a dropped or reordered row, and a row whose parent is not among the rows are refused. The self tags are not authenticated, so matching them catches a row corrupted by someone who never held the token, not a dishonest database; the selector, the position and the label are what bind a row to its place. -What the checks do not cover: **which row an entry comes from**. Nothing in an entry names its row, so another row's `$.salary`, returned with that row's header, opens; that is the rule every EQL stored value follows (ADR 0010). And **a position before the last step of a path**: every element of an array shares its selector, so a walk for `$.orders[1].items` can be answered with `$.orders[0].items` of the same row. And, for a walk with no gates (`json_tree_subtree`, whose path is a top-level field, or the root when the root is not sealed, followed only by array positions), **the node the walk starts from**, which it does not return: its children are checked against each other, so a gap between two of them and an array that does not start at element `0` are refused, but children left out after the last one returned, and members left out anywhere, are not (a member's place is a keyed rank the client does not hold), and for a path that ends in a position (`$.orders[3]`) the children can be another element's, since every element's children share their selectors. Both close when the walk returns the node it starts from, whose child count bounds its children (#1356). A query's result is not integrity-protected as a whole: each decrypted value is genuine and sealed at the path asked for, read as a selector reads it, but which row it came from, and which of the same-selector nodes above, is the database's word. +What the checks do not cover: **which row an entry comes from**. Nothing in an entry names its row, so another row's `$.salary`, returned with that row's header, opens; that is the rule every EQL stored value follows (ADR 0010). And **a position before the last step of a path**: every element of an array shares its selector, so a read of `$.orders[1].items` can be answered with `$.orders[0].items` of the same row. A query's result is not integrity-protected as a whole: each decrypted value is genuine and at the path asked for, but which row it came from is the database's word. ## Compared with EQL v3 (`public.eql_v3_json_search`) @@ -60,11 +64,11 @@ What the checks do not cover: **which row an entry comes from**. Nothing in an e | which leaves sit in the same array element | no | yes, through the projection's nesting: the one new disclosure, required for exact containment, bounded to array elements | | equal values in sibling elements | ciphertexts equal | ciphertexts differ | | ordering terms | on scalar leaves, with no per-path choice | only at the paths the index orders; none by default | -| a subtree read | reveals the selector asked | reveals a token for the top-level field the path starts with, and with it that field's shape in every row for the life of the column | +| a subtree read | reveals the selector asked | reveals a token for exactly the node asked for (one per concrete path, and one probe for each array a wildcard expands), and with it that node's shape in every row for the life of the column | ## The two options that change the picture -- **`sealed_root`** (default on). The root entry's ciphertext is the whole document, so a full read is one decryption and the broadest token a query can send is one top-level field's. The costs are that the root entry's length reveals the document's size, the row is larger, and the root's direct members are marked by their missing `p`, which puts each top-level field's node count in plain view. Off, there is no document-sized ciphertext, but every subtree read whose path begins with a key sends the root's token, the whole column's shape. +- **`sealed_root`** (default on). The root entry's ciphertext is the whole document, so a full read is one decryption and no token a query sends covers a whole document: the root's token finds the root entry and nothing beneath it. The costs are that the root entry's length reveals the document's size, the row is larger, and the root's direct members are marked by their missing `p`, which puts each top-level field's node count in plain view. Off, there is no document-sized ciphertext, but a read of `$` itself sends the root's token, the whole column's shape. - **`ordered`** (default none). Each path listed carries an ordering term and takes on the `op` row above; a path not listed cannot be range-queried or sorted, and a query that tries is refused by the client before any key loads. Order only the paths your queries sort or range over. One disclosure above is a property of the stored format that no option changes today: the depth-first entry order, with members in sorted key order. It is an open decision recorded in the design document (`docs/plans/2026-10-17-json-tree-index.md` at the repository root, Phase 4 notes) and in #1357.