test(eql): hold the planner to the json_tree indexes on every supported PostgreSQL, and document what the tree reveals

.changeset/eql-json-tree-planner-and-leakage.md

@@ -0,0 +1,5 @@
+---
+"@cipherstash/eql": patch
+---
+
+EQL v4 `json_tree` (unreleased): the reference docs gain "What an encrypted JSON tree reveals" (`docs/reference/json-tree-leakage.md`), one page on what a stored row shows the database at rest (including the row's shape, which the stored depth-first entry order gives away), what each query type hands it, what is and is not authenticated, and the side-by-side with v3 `ste_vec`. The encryption test suite now holds the planner to the GIN and btree indexes with every setting at its default, on inline and TOASTed documents, for one-shot and prepared statements, and CI runs that suite on every supported PostgreSQL version.

.github/workflows/test-eql.yml

@@ -100,7 +100,8 @@ on:
       - "packages/eql/docker/**"
       - "packages/eql/README.md"
       - "packages/eql/SUPABASE.md"
-      # The `rust-crates` job's encryption steps compile these by PATH
+      # The `rust-crates` and `rust-encryption-postgres` jobs' encryption
+      # steps compile these by PATH
       # (`eql-bindings`' `stack-encrypt` feature): stack-encrypt, its derive
       # and stack-kms through `path =`, and stack-auth and stack-profile
       # through `workspace = true` entries the root `Cargo.toml` resolves to
@@ -449,6 +450,70 @@ jobs:
         run: |
           mise run test:sqlx:partition
 
+  # The Rust encryption suite against a live database, once per supported
+  # PostgreSQL version: the JSON tree family's oracle (containment over the
+  # encrypted column against native `jsonb @>` over the plaintexts), its
+  # walks and orderings, and the plan-shape test that holds the planner to
+  # the GIN and btree indexes with every setting at its default. Plan choice
+  # is a per-version property of the planner, which is why this is a matrix
+  # and not a step of `rust-crates`. PR and manual dispatch run PG 17 alone;
+  # a push or the schedule runs the full set, like the SQLx jobs (see
+  # `setup`). Listed in `ci-required`'s `needs`, like every job here: the
+  # step it replaced sat inside `rust-crates`, which is.
+  rust-encryption-postgres:
+    name: "Rust encryption with PostgreSQL ${{ matrix.postgres-version }}"
+    needs: [changes, setup]
+    # Exclusion, not an allowlist — see the note on `build-archive`.
+    if: >-
+      github.event_name != 'pull_request'
+      || needs.changes.outputs.relevant == 'true'
+    runs-on: blacksmith-16vcpu-ubuntu-2204
+    strategy:
+      fail-fast: false
+      matrix:
+        postgres-version: ${{ fromJSON(needs.setup.outputs.pg-versions) }}
+    services:
+      encryption-postgres:
+        image: postgres:${{ matrix.postgres-version }}
+        env:
+          POSTGRES_PASSWORD: postgres
+        # Container port only: Docker picks the host port, and the test step
+        # reads it from `job.services`, so the number is written once.
+        ports:
+          - 5432
+        options: >-
+          --health-cmd pg_isready
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
+        with:
+          persist-credentials: false
+      - uses: jdx/mise-action@1648a7812b9aeae629881980618f079932869151 # v4
+        with:
+          version: 2026.4.0
+          install: true
+          cache: true
+          working_directory: packages/eql
+          # Must name runs-on's image; see scripts/__tests__/workflow-mise-cache-image.test.mjs
+          cache_key_prefix: mise-v1-ubuntu-2204
+      # Restores what `rust-crates` saves (the stack-encrypt feature tree)
+      # and never saves itself: a matrix of fast jobs racing one key would
+      # overwrite it with whichever finished first.
+      - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
+        with:
+          workspaces: packages/eql
+          shared-key: rust-crates
+          save-if: false
+      # The task takes the connection from EQL_TEST_DATABASE_URL alone, and
+      # the host port in it comes from the service, so the two cannot
+      # disagree.
+      - name: Test Rust encryption with PostgreSQL ${{ matrix.postgres-version }}
+        env:
+          EQL_TEST_DATABASE_URL: "host=localhost port=${{ job.services.encryption-postgres.ports['5432'] }} user=postgres password=postgres dbname=postgres"
+        run: mise run test:encryption:postgres
+
   # docs:validate + Clean-DB v3 install smoke. Both are version-relevant, so
   # they follow the event's PG set (PG17 on PR; 14-17 in the queue). Moved out
   # of the old per-version test job so they run ONCE per version, not per shard.
@@ -548,20 +613,6 @@ jobs:
       github.event_name != 'pull_request'
       || needs.changes.outputs.relevant == 'true'
     runs-on: blacksmith-16vcpu-ubuntu-2204
-    services:
-      encryption-postgres:
-        image: postgres:17
-        env:
-          POSTGRES_PASSWORD: postgres
-        # Container port only: Docker picks the host port, and the test step
-        # reads it from `job.services`, so the number is written once.
-        ports:
-          - 5432
-        options: >-
-          --health-cmd pg_isready
-          --health-interval 10s
-          --health-timeout 5s
-          --health-retries 5
     steps:
       - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
         with:
@@ -603,20 +654,14 @@ jobs:
       - name: Check the timing harness's Cargo.lock
         run: cargo metadata --locked --format-version 1 --manifest-path tests/timing/ore_block_v2/ctgen/Cargo.toml > /dev/null
 
-      # The three encryption checks are mise tasks rather than raw cargo here,
-      # so `eql-suite-ci.test.mjs` counts them among the cargo tasks a root
+      # The encryption checks are mise tasks rather than raw cargo here, so
+      # `eql-suite-ci.test.mjs` counts them among the cargo tasks a root
       # workflow must reach, and `eql-workflow-filters.test.mjs` derives the
-      # paths they read. The database task takes the connection from
-      # EQL_TEST_DATABASE_URL alone, and the host port in it comes from the
-      # service, so the two cannot disagree.
+      # paths they read. The database half runs in `rust-encryption-postgres`
+      # below, once per supported PostgreSQL version.
       - name: Test Rust text equality
         run: mise run test:encryption
 
-      - name: Test Rust text equality with PostgreSQL
-        env:
-          EQL_TEST_DATABASE_URL: "host=localhost port=${{ job.services.encryption-postgres.ports['5432'] }} user=postgres password=postgres dbname=postgres"
-        run: mise run test:encryption:postgres
-
       - name: Compile Rust text equality for WASI without HTTP
         run: mise run check:encryption:wasi
 
@@ -1011,8 +1056,9 @@ jobs:
   ci-required:
     name: "ci-required"
     needs: [changes, setup, build-archive, test, validate, schema, rust-crates,
-            codegen, self-contained-v3, matrix-coverage, splinter, docs-static,
-            known-failures, doc-anchors, e2e]
+            rust-encryption-postgres, codegen, self-contained-v3,
+            matrix-coverage, splinter, docs-static, known-failures,
+            doc-anchors, e2e]
     if: always()
     runs-on: blacksmith-16vcpu-ubuntu-2204
     # Same opt-out as `setup`, and this is the worse of the two to get wrong:

docs/plans/2026-10-17-json-tree-index.md

@@ -282,6 +282,8 @@ On a flat five-node document the proposal is about 8% larger: the two chain fiel
 
 > **2026-10-22:** parent tags are position-keyed (Phase 1), so siblings do not share a `p` at rest. Since Phase 3b there is no `g`: which leaves sit in the same array element is visible at rest through the nesting of the stored projection `q` instead.
 
+> **2026-10-23 (review of #1336):** the "proposed" column of the shape-equality and parent-and-child rows is wrong for the format as built, and was wrong for the design as written. Neither the tags nor their salt are what decides it: the entries of `sv` (and the pairs of `q`) are stored in depth-first pre-order, an object's members in sorted key order, under selectors that are the same in every row. A row's sequence of selectors therefore fixes its shape exactly, the parent of every node is inferable by comparing rows (and is exact for the top-level fields under the sealed root, whose entries are the ones with no `p`), every array's length is the count of its element selector, and the relative order of sibling key names is visible across rows. `packages/eql/docs/reference/json-tree-leakage.md` states the format as built; the decision this leaves open is recorded under Phase 4.
+
 ### 7. Performance
 
 - **Encrypt:** per node two PRFs (`s`, `vs`), three hashes (`t`, `p`, mask), one small AEAD. v3 does two MACs and one AEAD over the subtree bytes. Fewer bytes encrypted at every depth above the leaves.
@@ -487,6 +489,15 @@ The rows were taken from the Phase 3c tables (one ordered path a corpus), re-enc
 
 The nested document is now about 14 times its plaintext on disk (18 at Phase 3c, 40 at Phase 3). The byte saving exceeds the character saving: pglz compressed hex better than it compresses base64 and base85, but not by enough to matter. The benchmark corpora are not re-ingested and no query is re-timed in this note; the walk, containment and ordering paths are covered by the oracle suite, which passed against the bundle built from the tree. Of the two levers left from the Phase 3 list, one is closed and one is open. **The derived nonce is rejected (2026-10-23).** It would save the twelve random bytes in every `c`, about 7% of a nested document, but it needs vitaminc to expose a caller-chosen nonce (a nonce strategy that reads the leaf's AAD, and a nonce-less leaf wire form), and a caller-chosen nonce is exactly what vitaminc's AEAD layer refuses to offer: its misuse resistance rests on never letting a caller repeat a (key, nonce) pair, and the safety argument here ("every seal mints a fresh data key") is an invariant of stack-encrypt that vitaminc could not check. Seven percent does not buy weakening that promise; the random nonce stays. Unpadded base64 (two characters a tag, about 3%) remains open and untaken.
 
+## Implementation notes (2026-10-23, Phase 4: the planner held to its indexes, and the leakage page)
+
+Two of the three items that gate a release are closed here; the third, the final re-bench, follows them.
+
+- **Planner fragility (gate 2).** `eql_v4.json_tree_query` is plpgsql with `COST 1000` so that a bitmap scan over the GIN index beats a sequential scan even when the documents are TOASTed and the heap looks like a few pages. That was observed in the benches, at 10,000 and 100,000 rows, on one PostgreSQL version, and nothing held it. `postgres_the_planner_picks_the_indexes_without_help` in the encryption suite now does: on a flat table (1,000 four-node documents, every row inline) and a nested table (300 documents of about 220 nodes, every row TOASTed, both regimes pinned by `pg_column_size`), with every planner setting at its default and after `ANALYZE`, containment plans as a bitmap scan over the GIN index and a range at an ordered path as a scan of the functional btree on `ord_term(doc -> selector)`. A control sets the function's cost to 1 for one transaction and shows the planner taking the sequential scan on the nested table, so the assertion is known to be able to fail. The suite runs in `test-eql.yml` in a new `rust-encryption-postgres` job, a matrix over the supported PostgreSQL versions (17 on a pull request, 14 to 17 on a push, like the SQLx jobs); it was a step of the single-version `rust-crates` job. Measured locally on 14, 15, 16 and 17 before the commit.
+- **Leakage (gate 3).** `packages/eql/docs/reference/json-tree-leakage.md` is the one page: what a stored row shows at rest, field by field; what each query type hands the database; what is authenticated; the side-by-side with v3 from section 6 above, brought up to date (no `g`, position-keyed `p`, ordering opt-in) and corrected where section 6 was wrong (the entry order; see the note under its table); and the two options that change the picture. It says what a token is and that it is column-wide and does not expire, which is the sentence a customer most needs before sending one.
+- **Open for the owner: the stored entry order (2026-10-23, review of #1336).** `sv` and `q` hold the nodes in the walk's order, depth-first, an object's members in the sorted order of their keys (serde_json's `Map` without `preserve_order`; a build that unifies that feature on would store insertion order instead, which leaks the application's field order). Across rows this reveals each row's shape, its parent relation, every array's length and the alphabetical order of sibling key names, which partly undoes what the keyed rank (decision 11) hides from a token holder. The leakage page says so. Closing it is a format change, not a documentation one: a stored order that does not follow the tree (a document-keyed permutation, say, with `j` the permuted position), a rebuild that does not read the tree from the order, and the same for the pairs of `q`, whose nesting would still group array elements. Not taken here.
+- **Open for the owner: an extracted entry opens under the selector the server returns with it.** `JsonNode` and `JsonSubtree` open each entry under the `s` and `j` that arrive beside it, and under the data key the header that arrives with it names. The entry's AAD (`entry_aad` in `packages/stack-encrypt/src/json/derive.rs`) binds the keyset, the column's context, the data key's tag, the selector and the ordinal, and nothing that identifies a row. So a decrypted value is always one sealed in that column, but not necessarily the one at the path asked for, nor one from the row the result puts it in: `->` on `$.salary` can be answered with the row's `$.bonus`, with the sealed root (which opens to the whole document), or with another row's `$.salary` returned together with that row's header. The last is by design: two values sealed under the same context are interchangeable between rows (ADR 0010 in `packages/stack-encrypt/docs/adr`), and row provenance is the database's word. The leakage page states this under "What is authenticated". Checking the returned `s` against the selector the client derived would close the path substitution for `->`, though not the row one; a walk's rows would need each child's selector checked against its parent's path.
+
 ## Decisions to take before Phase 1
 
 > **2026-10-22:** these are the open questions as they stood before Phase 1; the implementation notes above record how each was taken. Since then, decision 4's group tags exist only as the nesting of the stored projection `q`, decision 9 was settled by storing the projection (Phase 3), and decision 10's ordinal is not stored (Phase 3b).

packages/eql/docs/README.md

@@ -14,6 +14,7 @@ This directory contains the documentation for the Encrypt Query Language (EQL).
 - [Writing fast queries against EQL columns](reference/query-performance.md) - Performance overview (points to Database Indexes)
 - [Adding a Scalar Encrypted-Domain Type](reference/adding-a-scalar-encrypted-domain-type.md) - How the `eql_v3.<T>` domain families are generated
 - [EQL with JSON and JSONB](reference/json-support.md)
+- [What an encrypted JSON tree reveals](reference/json-tree-leakage.md) - What `public.eql_v4_json_tree` (EQL v4) shows the database at rest and at query time, and what it does not
 - [EQL payload / wire format](../crates/eql-bindings/README.md) - Canonical wire types for the encrypted payload (envelope `v`/`i`/`c` and the `hm`/`op`/`ob`/`bf` index terms)
 - [Client-side index configuration](https://cipherstash.com/docs/stack/cipherstash/encryption/schema) - Configuring searchable encryption in CipherStash Stack / CipherStash Proxy