diff --git a/.claude/board/INTEGRATION_PLANS.md b/.claude/board/INTEGRATION_PLANS.md index 185dfb18d..10c167fe4 100644 --- a/.claude/board/INTEGRATION_PLANS.md +++ b/.claude/board/INTEGRATION_PLANS.md @@ -1,3 +1,13 @@ +## 2026-10-31 — D-DYNBAG: DR{dynamic bag}, holographic bags as polyfill, the PostgreSQL endgame → `.claude/plans/d-dynbag-holographic-bags-v1.md` + +**Status:** PROPOSED, plan only. **Board:** [STATUS_BOARD](STATUS_BOARD.md) D-DYNBAG. DuckDB's selection vector made explicit: a Quack selection becomes a mask plus one pointer table (`dt64::dr::LocalMap`) and runs on a local tile whose width follows the survivor count (DR64 / DR256 / multi-tile). Relations on a tile are read in a semiring per operation (B for presence, ℕ for bag multiplicity; K-relations), one address with several readings. The bag-op polyfill names each relational operation once with one realization per semiring. ℕ-compose via `ndarray::simd::matmul_i8_to_i32` is the candidate home for the many-to-many join (falsifier D-DYNBAG-1). Endgame: a PostgreSQL-compatible surface (`sql-quack-probe`, modelled on `lance-graph-cypher-quack`) scored against DuckDB and PostgreSQL in ns, losses reported. The membrane stays a reference note on EXO-50. +**Confidence:** plan; word existence verified in code, no new measurement. + +## 2026-10-31 — D-SOC-FENCE: one place per concern, checked in CI → `.claude/plans/d-soc-fence-v1.md` + +**Status:** PROPOSED, plan only. **Board:** [STATUS_BOARD](STATUS_BOARD.md) D-SOC-FENCE. A mapping file (`.claude/soc/one-place.toml`) names each concern's sole owner, its basis, the symbols only the owner names and what others reach it through. A `syn`-based symbol fence plus a `cargo metadata` dependency fence report every use outside it. A two-way ratchet baseline holds known violations. Report-only first; blocking once the baseline is committed. Triggered by PR #1884's direct `ndarray::simd` fold calls bypassing Quack and mask-risc. +**Confidence:** plan; blast radius counted by `git grep` as candidates, not read. + ## 2026-10-30 — D-XLOG-NATIVE: XLOG plan vocabulary over the ndarray::simd polyfill → `.claude/plans/d-xlog-native-v1.md` **Status:** PROPOSED, docs only. **Board:** [entry](entries/2026-10-30-d-xlog-native-crosswalk.md), [STATUS_BOARD](STATUS_BOARD.md) D-XLOG-NATIVE. Crosswalk of XLOG's compiled-plan nodes onto the `ndarray::simd` polyfill that `lance-graph-onnx` aliases, in the D-ONNX-NATIVE categories (ALIAS / INLINE / NATIVE / REFUSE), name-checked, not tested; shared operations are the same native symbols as ONNX's. Decides that a `lance-graph-xlog` crate (ndarray only, like `lance-graph-onnx`; no `xlog-*` dependency, no plan type) waits until selection rows are tested and the many-to-many join has a native home. diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 1a91e32a3..934091086 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -12,17 +12,46 @@ | D-MIST-NATIVE-3 | axes past 64 (worlds, propositions, rules) | Queued | refused today | | D-MIST-NATIVE-4 | intake from compiled HLL | Queued | upstream provenance defects (`mistaber-rs#3`, `#4`) | +## D-DYNBAG — DR{dynamic bag}: selection vector made explicit, holographic bags as polyfill, PostgreSQL endgame (2026-10-31) + +Plan: `.claude/plans/d-dynbag-holographic-bags-v1.md`. Plan only. + +| D-id | Deliverable | Status | Gate | +|---|---|---|---| +| D-DYNBAG-0 | PR #1884 layering fix: XLOG intake as lanes, support + bag count via Quack → mask-risc, fold oracles, cost columns | Queued | operator go-ahead | +| D-DYNBAG-1 | ℕ-compose probe: `matmul_i8_to_i32` over two DR64² tiles for `p(X, Z)`; B projection = `relcomp::compose` | Queued | D-DYNBAG-0 | +| D-DYNBAG-2 | semiring parameter K (B, ℕ) in relalg `Rel` + Quack surface; typed refusal per (op, K) | Queued | D-DYNBAG-1 | +| D-DYNBAG-3 | missing ℕ words backend-first (i64 masked select, min, truncated minus, compare-to-mask, masked sum; maybe bit-packed product) | Queued | D-DYNBAG-2 | +| D-DYNBAG-4 | DR{dynamic}: width from survivor count, multi-tile map, cross-tile fold/compose | Queued | DT64 `dr` owner | +| D-DYNBAG-5 | `sql-quack-probe` modelled on `lance-graph-cypher-quack` | Queued | parser decision | +| D-DYNBAG-6 | scoreboard vs DuckDB / PostgreSQL, ns units, losses reported | Queued | D-DYNBAG-5 | +| D-DYNBAG-7 | second Datalog frontend (Scallop) through the same path | Queued | D-DYNBAG-2 | + +## D-SOC-FENCE — one place per concern, checked in CI (2026-10-31) + +Plan: `.claude/plans/d-soc-fence-v1.md`. Plan only; nothing runs yet. + +| D-id | Deliverable | Status | Gate | +|---|---|---|---| +| D-SOC-FENCE-0 | census of §1 candidates, read and classified; mapping file + baseline drafts | Queued | §5 decisions | +| D-SOC-FENCE-1 | `syn`-based symbol fence + dependency fence, fixture falsifiers, report-only workflow | Queued | D-SOC-FENCE-0 | +| D-SOC-FENCE-2 | mapping file + baseline committed; CI blocks new violations and stale entries | Queued | D-SOC-FENCE-1 | +| D-SOC-FENCE-3 | per-crate burn-down through each concern's `reach_via` | Queued | D-SOC-FENCE-2, owning sessions | +| D-SOC-FENCE-4 | `/v3-audit` item 1 and the lgj G11 allowlist read from the mapping file | Queued | D-SOC-FENCE-2 | + ## D-XLOG-NATIVE — XLOG plan vocabulary over the ndarray::simd polyfill, the algebra under lance-graph-onnx (2026-10-30) **Status:** PROPOSED / DOCS ONLY. [Plan](../plans/d-xlog-native-v1.md); [board entry](entries/2026-10-30-d-xlog-native-crosswalk.md). No crate yet. | D-id | Deliverable | Status | Gate | |---|---|---|---| -| D-XLOG-NATIVE-0 | crosswalk of XLOG plan nodes onto `ndarray::simd` (ALIAS / INLINE / NATIVE / REFUSE), shared symbols with lance-graph-onnx, name-checked | In PR | docs review | +| D-XLOG-NATIVE-0 | crosswalk of XLOG plan nodes onto `ndarray::simd` (ALIAS / INLINE / NATIVE / REFUSE), shared symbols with lance-graph-onnx, name-checked | Shipped (#1866) | docs | | D-XLOG-NATIVE-1 | test the selection rows (eq/ne_u32_to_mask, i32 comparisons, mask_and/or/andnot) against the probe's two oracles, signatures checked | Queued | N2 | | D-XLOG-NATIVE-2 | test masked_group_count_u32 keyed by one table's column against the RIR walk, with the out-of-universe refusal | Queued | N2, N3 | | D-XLOG-NATIVE-3 | ask the ndarray / mask-risc owners for a native home for the many-to-many join (fixture: Gate B) | Queued | owner decision | | D-XLOG-NATIVE-4 | create `crates/lance-graph-xlog` (ndarray only, no `xlog-*` dep, no plan type) once plan §4 holds | Queued | N1–N5 | +| D-XLOG-NATIVE-6 | probe executes on `ndarray::simd` (`src/native.rs`): support via `eq_u32_to_mask` / `mask_scatter_or_u32` / `mask_and`, bag count via `masked_group_count_u32` ×2 + `checked_binary_i64`; DR `LocalMap` up to 256; register path kept as differential (plan Amendment B) | In PR | lab probe | +| D-XLOG-NATIVE-5 | bag count bound through the production bind store (`lance-graph-dt64` + `dt64::dr`), polyfill words by position; Epi5 reading of XLOG results pinned (plan Amendment A) | In PR | lab execution only | ## D-LITHO — the photolithography contract (2026-10-30) diff --git a/.claude/board/entries/2026-10-31-d-xlog-native-5-bindstore-epi5.md b/.claude/board/entries/2026-10-31-d-xlog-native-5-bindstore-epi5.md new file mode 100644 index 000000000..883edc094 --- /dev/null +++ b/.claude/board/entries/2026-10-31-d-xlog-native-5-bindstore-epi5.md @@ -0,0 +1,12 @@ +# D-XLOG-NATIVE-5 — XLOG bag count bound through the production bind store; Epi5 reading of XLOG results + +**Date 2026-10-31 · Status IN PR · Plan `.claude/plans/d-xlog-native-v1.md` Amendment A** + +- `crates/xlog-relcomp-probe/tests/dto_binding.rs` routes, seals and admits through `lance-graph-dt64` (`BindStore`, `dr::FrameStore`, `MethodRef`). The count is three steps over two polyfill words, named by position in one lab polyfill lens: mask-risc's keyed count twice, then `checked_binary_i64(Mul)`. Refusals: stale frame (`MapChanged`), retracted position (`Unrouted`). Execution stays in the lab register, which refuses the admitted Bag population (`BagNeedsMultiplicity`). +- `tests/epi5_reading.rs` covers XLOG results written as `CausalEdge64` bits 59..63: + - a base fact is `Direct × Open`; a composed edge with Y retained is `IndirectKnown × Open`; + - the reading is refused when undeclared or under `V1Legacy` provenance; + - an unstamped composed edge reads as `Direct`. +- Plan Amendment A: a binding names mask-risc's op vocabulary (addenda 8 and 9); the `ndarray::simd` column is what an executor would call. + +**OPEN:** method identity (`MethodSignature` / `MethodRef` / `MethodKey`, EXO-50); frame catalogue extent (D8); production classids and the XLOG Epi5 declaration (OGAR); many-to-many join home (D-XLOG-NATIVE-3). diff --git a/.claude/board/entries/2026-10-31-d-xlog-native-6-ndarray-simd-execution.md b/.claude/board/entries/2026-10-31-d-xlog-native-6-ndarray-simd-execution.md new file mode 100644 index 000000000..ebd8d4ef9 --- /dev/null +++ b/.claude/board/entries/2026-10-31-d-xlog-native-6-ndarray-simd-execution.md @@ -0,0 +1,11 @@ +# D-XLOG-NATIVE-6 — the XLOG probe executes on the ndarray::simd polyfill + +**Date 2026-10-31 · Status IN PR · Plan `.claude/plans/d-xlog-native-v1.md` Amendment B** + +`crates/xlog-relcomp-probe/src/native.rs` answers the admitted two-atom chain: +- support: `eq_u32_to_mask` for bound ends, `mask_scatter_or_u32` onto Y, then `mask_and`; +- bag count: `masked_group_count_u32` for each side, then `checked_binary_i64(Mul)` (ndarray #407). + +The Y axis is DR's `LocalMap`, so the path answers 200 Y where the register's 64-slot tile refuses. `admit_count` admits XLOG's `count` rule. The register path remains as the differential, with every mask compared in Gate B. CubeCL was not needed. + +**OPEN:** a many-to-many join keeping both ends, and chains past two atoms (still NATIVE); the overflow refusal has no reachable fixture. diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index 6bdd9df51..dd2adb759 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,10 +25,12 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -520 entries, 2026-08-06 .. 2026-10-31. +523 entries, 2026-08-06 .. 2026-10-31. | date | entry id | finding | file | |---|---|---|---| +| 2026-10-31 | `D-XLOG-NATIVE-6` | | [2026-10-31-d-xlog-native-6-ndarray-simd-execution.md](2026-10-31-d-xlog-native-6-ndarray-simd-execution.md) | +| 2026-10-31 | `D-XLOG-NATIVE-5` | | [2026-10-31-d-xlog-native-5-bindstore-epi5.md](2026-10-31-d-xlog-native-5-bindstore-epi5.md) | | 2026-10-31 | `D-SAG-6-2-JOIN-IDENT` | | [2026-10-31-d-sag-6-2-span-join-identity.md](2026-10-31-d-sag-6-2-span-join-identity.md) | | 2026-10-31 | `D-SAG-6-2-LAB` | | [2026-10-31-d-sag-6-2-lab-projection.md](2026-10-31-d-sag-6-2-lab-projection.md) | | 2026-10-31 | `D-SAG-6-1-CHECKLIST` | | [2026-10-31-d-sag-6-1-checklist.md](2026-10-31-d-sag-6-1-checklist.md) | diff --git a/.claude/plans/d-dynbag-holographic-bags-v1.md b/.claude/plans/d-dynbag-holographic-bags-v1.md new file mode 100644 index 000000000..4c850722b --- /dev/null +++ b/.claude/plans/d-dynbag-holographic-bags-v1.md @@ -0,0 +1,442 @@ +# D-DYNBAG — DR{dynamic bag}: DuckDB's selection vector made explicit, holographic bags as a polyfill, and the PostgreSQL endgame + +**Status:** PROPOSED, plan only. No code, crate, ABI or test lands with this plan. Each claim below is labelled: +- **VERIFIED-IN-CODE**, with its location; +- **MEASURED**, with its command; +- **WORKING-MODEL** (the design this plan proposes); +- **HYPOTHESIS** (unproven; carries a falsifier); +- **OPEN** (undecided; carries an owner). + +**Date:** 2026-10-31. **Parent work:** D-XLOG-DR64 (#1857), D-XLOG-NATIVE (#1866, #1884), D-DT64-6 Amendments A.5 addenda 7–12 (#1869, #1871, #1883), D-SOC-FENCE (#1884, plan only), the ONNX classid correction (#1870). + +**Directives this plan carries (operator, 2026-10-31):** +1. Inside lance-graph and ndarray, a tuple is a description only, the way `WHERE (x, y)` is in Quack. +2. Use the DuckDB / Quack selection idea to make a selection global → local, and run bag operations over DR256 / DR64. +3. "DR{dynamic bag}" is that, plus treating DR as a holographic representation, plus a bag-op polyfill. +4. The endgame is to beat PostgreSQL with Quack + holographic bags + the polyfill. A related, earlier endgame statement: beat Iceberg at its own game. +5. The external Quack surface may need an Abacus-pattern membrane. That idea lives on EXO-50 as a reference only and is not designed here (§12). + +--- + +## 1. Thesis + +A query is a **description**: a selection, a join body, a projection, an aggregate. It is stated over global identities and run over local positions. + +**Running a query (WORKING-MODEL):** +1. **Select.** The selection is evaluated as masks. +2. **Relabel.** Its survivors get dense local positions through one pointer table. That is DuckDB's selection vector, made explicit as DR's `LocalMap`. +3. **Tile.** The local positions form a tile whose width is chosen from the survivor count (DR64, DR256 or several tiles): **DR{dynamic}**. +4. **Read in a semiring.** Every relation on that tile is read in a semiring chosen per operation: Boolean for presence and `DISTINCT`, natural numbers for bag multiplicity, later provenance or probability. The address stays one; the values have several readings. That is the **holographic** reading. +5. **Name the operation.** Each operation (select, union, intersect, except, project, join, distinct, compose) is one name with one implementation per semiring. That is the **bag-op polyfill**, in the sense that `ndarray::simd` is a polyfill: one name, the realization chosen elsewhere. + +No row is ever built after intake. + +**The endgame:** a PostgreSQL-compatible surface (SQL in, rows out at the edge only) that answers with that machinery. It must match PostgreSQL's semantics and beat its executor on the queries where folds over addresses beat tuple-at-a-time execution. + +--- + +## 2. Vocabulary (precise, used consistently below) + +| Term | Meaning here | Never means | +|---|---|---| +| **tuple** | a *description*: named positions over lanes, read at a local position (lance-graph P0, "a tuple is a zero-copy fold template, never a value") | an owned row, `Vec<(u32, u32)>`, a struct per row | +| **intake** | the one named place where external row-shaped input becomes lanes (columns). Same status as `RowStore::importRows` in lance-graph-java: visible, named, counted | a hidden conversion inside an operator | +| **global id** | a value's identity outside the tile (an XLOG constant, a SQL key, a `NodeGuid`) | a position | +| **local position** | the dense index a survivor gets in its tile, `0..W` | an identity | +| **pointer table** | the local → global table of one tile; the only thing a selection materializes. VERIFIED-IN-CODE: `lance_graph_dt64::dr::LocalMap` (`globals[local]`, plus a sorted global → local index; lookups counted) | a copy of the rows | +| **mask** | one bit per row or per local position, in the `ndarray::simd` word convention (bit `i % 64` of word `i / 64`, tail bits zero) | an index list | +| **selection vector** | DuckDB's term: the indices of the rows that survived a filter. Here it has two forms: the **mask** (bit form, for folds) and the **pointer table** (index form, for relabelling) | a materialized subset of rows | +| **DR64² tile** | one binary relation over two 64-wide local axes: `[u64; 64]`, one bit per cell (row = left key, column = right key). VERIFIED-IN-CODE: `dr_math_binding_probe::relcomp::Rows` | a list of pairs | +| **DR256²** | the same at width 256 | — | +| **DR64³** | the path space of a two-atom chain `(x, y, z)` across two DR64² tiles sharing the y axis. Addressed, never filled: densely it would be 64³ bits = 32 KiB (VERIFIED-IN-CODE: `Extent::dense_bytes`, addendum 10) | a stored cube | +| **DR{dynamic}** | a tile whose width is chosen from the survivor count of its selection (§4) | a fixed 64 | +| **witness** | one full body binding of a rule or join (e.g. `(x, y, z)` for `r(X,Y), s(Y,Z)`) | a stored tuple | +| **support** | the set of answers with at least one witness (Boolean semiring) | a count | +| **multiplicity** | the number of witnesses per answer (natural-number semiring) | a certification or a truth value (§9.4) | +| **semiring K** | the algebra a relation's cells take values in (§5) | a data type | +| **holographic** | one DR address, read in several semirings at once: a bit plane for B, a count lane for ℕ, later annotation lanes | a superposition or bundle (no VSA is involved) | +| **bag-op polyfill** | the set of relational operation names, each with one realization per semiring, chosen by the description's K (§6) | a new evaluator | +| **membrane** | the outer boundary where row-shaped external formats meet descriptions (§12, reference only) | an inner layer | + +--- + +## 3. DuckDB's selection vector, made explicit + +### 3.1 What DuckDB does (background) + +- **Vectorized execution.** DuckDB runs over `DataChunk`s of up to 2048 rows (`STANDARD_VECTOR_SIZE`). +- **Filters build selection vectors.** A filter does not copy the surviving rows. It writes a `SelectionVector`: an array of row indices into the chunk. +- **Downstream reads through it.** Operators read the chunk through that selection, so a filter costs one index array and no row movement. +- **Joins and aggregates copy.** They read through the selection, then build hash tables or write output chunks, and that is where copies happen. + +### 3.2 The same idea, explicit, in this stack (WORKING-MODEL) + +| Step | DuckDB | Here | Owner | +|---|---|---|---| +| filter | predicate → `SelectionVector` | Quack `Filter` → mask-risc `Pred` / `MaskOp` → **mask** | Quack describes, mask-risc evaluates | +| survivors as indices | the selection vector itself | **pointer table**: `LocalMap::prepare(width, globals)` over the survivors' global ids | DT64 `dr` | +| downstream operator | reads the chunk through the selection | reads lanes at local positions; every bag operation runs on the **local tile** | mask-risc, `ndarray::simd` | +| join | hash table build and probe (copies keys and payload) | DR²: a tile per relation; DR³: path space through shared axes. Joins are folds or compose words over tiles (§6, §7) | polyfill | +| aggregate | hash aggregate (a table of groups) | `GroupReduce` over local keys: an O(groups) sink, no hash table | mask-risc | +| output | materialized chunks | rows exist only at the edge, through the pointer table, by name | intake / membrane | + +### 3.3 What is explicit that DuckDB keeps implicit + +1. **Both forms of the selection are first class.** The mask serves folds; the pointer table serves relabelling. DuckDB has only the index form and rebuilds masks when it needs them. +2. **The global → local relabelling is a named object with a cost.** `LocalMap` counts its lookups (VERIFIED-IN-CODE: `LocalMap::lookups`), so a test can show a projection never calls it. +3. **The tile width is a decision, not a constant.** DuckDB's 2048 is a vector size; here the width follows the survivor count (§4). +4. **The relabelling is versioned.** A frame records which coordinate assignment it was sealed under: `LocalMap::digest` → `FrameRevisions::coordinates` (VERIFIED-IN-CODE, `dt64::dr`). A stale tile is refused (`FrameRefusal::CoordinatesChanged`), never silently reused. + +### 3.4 Rules + +- The pointer table is the only thing a selection materializes. Lanes are read in place. +- A mask never crosses a program boundary as an O(N) intermediate when an O(groups) sink answers the question (the same rule Quack's `lower_group_by_semantic` states: "No O(N) kept-mask crosses a program boundary", VERIFIED-IN-CODE). +- A global id the pointer table does not hold is refused (`MapRefusal::Unmapped`), never dropped. + +--- + +## 4. DR{dynamic}: the tile width chosen from the selection + +### 4.1 Width rule (WORKING-MODEL) + +| Survivors on the axis | Tile | Basis | +|---|---|---| +| ≤ 64 | one DR64 tile | `Width::W64` (VERIFIED-IN-CODE, `dt64::dr`) | +| 65 – 256 | one DR256 tile | `Width::W256` | +| > 256 | ⌈n / 256⌉ DR256 tiles, under a thin-provisioned reservation | `Axis { tile, tiles }`, `Extent::DR256_GLOBAL_X64` reserves 256 × 256 = 65,536 identity positions; only routed tiles are claimed (`BindStore::claimed`, addendum 11) | + +- **Choosing 64 vs 256 is a cost decision.** A DR64 register is 512 B, a DR256 one 8 KiB. A tile that fits in 64 must not pay for 256. +- **Which width wins for a given density is MEASURED, not assumed** (§10). + +### 4.2 What exists and what is missing + +| Piece | Status | +|---|---| +| widths 64 and 256; one `LocalMap` per tile | VERIFIED-IN-CODE (`dt64::dr::{Width, LocalMap}`; `LocalMap::prepare` refuses more ids than the tile holds) | +| reserving many tiles, claiming few | VERIFIED-IN-CODE (`Axis`, `Extent`, `Claim`, addendum 11) | +| **a global → (tile, local) map across tiles** | **OPEN, gap.** `LocalMap` maps into one tile. A selection of 7,620 survivors (Jina, addendum 11) needs a tile index plus a local position, and the cross-tile operations of §6 need to address both. Owner: DT64 `dr`. | +| choosing the width from the survivor count | **gap**, small: a function from `n` to `Axis` | +| cross-tile folds and compose | **gap** (§7.4) | + +### 4.3 Dynamic over time + +- **A frame pins the tile layout.** When the population moves, the frame is advanced and re-sealed, and the old tile layout stays replayable at its route version. +- **No in-place resize.** A resize is a new frame, never a mutation of a sealed one (D5 immutability, addendum 12's frame witness). + +--- + +## 5. The holographic reading: one address, several semirings + +### 5.1 K-relations (background) + +- **The formalism.** Green, Karvounarakis and Tannen, "Provenance Semirings" (PODS 2007): a relation is a function from tuples to a commutative semiring K. +- **The operators are fixed by K.** + - union adds; + - join multiplies; + - projection sums over the projected-away columns; + - selection multiplies by 0 or 1. +- **The semiring decides what is answered:** + - K = Boolean gives set semantics; + - K = ℕ gives bag semantics, PostgreSQL's default; + - K = polynomials gives provenance; + - K = [0, 1] gives probabilities. +- **The same formalism runs through probabilistic Datalog.** Scallop (provenance-semiring Datalog) is built on it; XLOG's README reports accuracy parity with Scallop on MNIST-addition. + +### 5.2 The readings of one DR address + +| K | Value at a cell | Carrier | Holds | Status | +|---|---|---|---|---| +| **B** (∨, ∧) | 0 or 1 | bit plane: a DR64² / DR256² register, masks | presence, `DISTINCT`, support | VERIFIED-IN-CODE (masks everywhere; the DR64 register in `relcomp`) | +| **ℕ** (+, ×) | multiplicity | **count lane**: one integer per local position (1-D), or a count tile (2-D, §7) | bag semantics, `COUNT(*)`, XLOG's `count` | 1-D: VERIFIED-IN-CODE (`GroupReduce{Count}` sinks; the probe's bag count). 2-D: HYPOTHESIS (§7) | +| ℕ[X] (provenance) | which witnesses | annotation lane | why an answer holds | OPEN; not in scope until ℕ is proven | +| [0, 1] (probability) | weight | annotation lane | ProbLog / Scallop style | OPEN; XLOG's probabilistic programs stay refused (D-XLOG-DR64 Gate G) | + +**Projection between readings (WORKING-MODEL):** +- **ℕ → B** is `count > 0`: a compare-to-mask. +- **B → ℕ** is 0/1, through the signed-mask idiom. +- **The holographic property:** for every operation, the ℕ result projected to B equals the B result. That is the first falsifier of everything in §6 and §7. + +### 5.3 What "holographic" is not + +- **Not VSA.** No superposition, no bundling, no `Vsa16kF32`. The readings are separate lanes at one address, not a bundle. +- **Not a CE64 field.** A multiplicity is never written into CausalEdge64's frequency/confidence bits or its Epi5 bits (§9.4). + +--- + +## 6. The bag-op polyfill + +**One operation name, one realization per semiring.** The description carries K; the realization is chosen by K, never by the caller. Every row marks its word VERIFIED-IN-CODE (a mask-risc or `ndarray::simd` word that exists) or names the gap. + +| Operation | Over B | Word (B) | Over ℕ | Word (ℕ) | +|---|---|---|---|---| +| selection `σ` | mask ∧ predicate | mask-risc `Pred`, `MaskOp::And` | count × 0/1 | the B mask applied to the count lane; **gap**: a masked-select over an i64 lane (`blend` is i32) | +| union | ∨ | `mask_or` | + (UNION ALL) | `checked_binary_i64(I64Op::Add)` (VERIFIED-IN-CODE, `ndarray::simd`, #407) | +| intersect | ∧ | `mask_and` | min | **gap**: elementwise i64 min (mask-risc has `GroupFold::MinI32` only as a group fold) | +| except | ∧ ¬ | `mask_andnot` | truncated subtraction `max(a − b, 0)` | **gap** | +| projection onto a key | scatter ∨ | `Terminal::ScatterOrU32` | scatter + | `Terminal::GroupReduce { key: Lane, fold: Count }` (VERIFIED-IN-CODE). **Correction:** `Terminal::ScatterCountU32` is `COUNT(DISTINCT …)`, held and not a lowering target; it is NOT the bag projection (an earlier answer said it was) | +| join on y, projected to y | forward ∧ backward | `ScatterOrU32` per side + `mask_and` | deg_R(y) × deg_S(y) | `GroupReduce{Count}` per side + `checked_binary_i64(Mul)` (VERIFIED-IN-CODE, the probe's `bag_count`, Gate B differential) | +| **join keeping both ends (x, z)** | Boolean matrix product | lab only: `relcomp::compose` | integer matrix product | **HYPOTHESIS:** `ndarray::simd::matmul_i8_to_i32` (§7) | +| distinct | identity | — | `count > 0` | compare-to-mask (`*_to_mask` family over i64: **gap**; i32 exists) | +| count of a population | popcount | `popcount_batch_u64`, `Terminal::Count` | sum of the lane | `masked_sum_i32` (i32) / **gap** for i64 | +| group aggregate | — | — | `GroupReduce { Count / SumSymI32 / MinI32 / MaxI32 }` | VERIFIED-IN-CODE (mask-risc) | + +**Rules (WORKING-MODEL):** +- **Choosing a realization is the polyfill's job, as in `ndarray::simd`.** A caller names an operation and a K; it never names a kernel. +- **Unsupported (operation, K) pairs are typed refusals**, never a fallback to B. Falling back to B is exactly the support-for-bag error the XLOG fixture exposed (support 1, true count 4). +- **New words land backend-first:** in mask-risc or `ndarray::simd`, owned there, never inline in a frontend (D-SOC-FENCE `fold-evaluator`). + +--- + +## 7. ℕ-compose: the many-to-many join as integer matrix multiplication + +### 7.1 The mathematics (WORKING-MODEL) + +- **Inputs.** R ⊆ X × Y and S ⊆ Y × Z on local tiles. As 0/1 matrices, A is |X| × |Y| and B is |Y| × |Z|. +- **The product counts witnesses.** (A · B)[x][z] = Σ_y A[x][y] · B[y][z] is exactly the number of y with r(x, y) ∧ s(y, z): the ℕ answer of `p(X, Z) :- r(X, Y), s(Y, Z)`, both ends kept. +- **B is its projection.** The Boolean answer (`relcomp::compose`) is `(A · B) > 0`. + +### 7.2 The word that may already exist + +- VERIFIED-IN-CODE: `ndarray::simd` re-exports `matmul_i8_to_i32` (`src/simd.rs` lines 845 and 863, from `hpc::amx_matmul`) and `gemm_u8_i8`. +- `matmul_i8_to_i32(lhs: ArrayView2, rhs: ArrayView2, out: ArrayViewMut2)` takes an AMX `TDPBUSD` path when `amx_available() && m % 16 == 0 && n % 16 == 0 && k % 64 == 0`, and is otherwise a bit-identical scalar i32 reference. + +### 7.3 Exactness bounds + +| Case | Bound | Exact? | +|---|---|---| +| one hop, 0/1 inputs, tile width W | each cell ≤ W (64 or 256) | yes, fits i32 | +| one hop on a DR64 tile | m = n = k = 64: meets the AMX alignment | yes | +| a chain of hops (counts feed the next product) | inputs exceed i8 after one hop | **no.** Needs a wider integer product or an overflow refusal per hop. Typed refusal until a word exists | +| bags with input multiplicities > 1 (base relations are sets, §9.2, so only derived inputs) | as for chains | refused, as above | + +### 7.4 Zero-copy and tiles + +- **The current signature copies.** `matmul_i8_to_i32` packs its inputs (`pack_contig`) and allocates `c = vec![0i32; m * n]` per call (VERIFIED-IN-CODE). +- **A DR64² tile is bit-packed**, so feeding it in means unpacking bits to i8: a copy per tile, not per row. +- **Status: HYPOTHESIS, to measure.** Whether that per-tile cost is acceptable, or whether a bit-packed i1 × i1 → i32 word is needed (popcount of row ∧ column, the classic bit-matrix product), is decided by measurement. If a new word is needed, it lands in `ndarray::simd` (backend-first), never here. +- **Cross-tile compose is a gap.** Tile (i, k) × tile (k, j), summed over k, as in blocked GEMM, needs the multi-tile map of §4.2. + +### 7.5 First falsifier (D-DYNBAG-1) + +- **Equality:** on the XLOG multiplicity fixture and the 300-case Gate B differential, `matmul_i8_to_i32` over two DR64² tiles equals scalar enumeration of witnesses per (x, z) (oracle 1) and a bag walk of XLOG's RIR (oracle 2). +- **Projection:** `count > 0` equals `relcomp::compose`, bit for bit (the holographic property, §5.2). +- **Disable runs:** + - swapping the operands fails the test; + - reading the result as B before counting fails the test. + +--- + +## 8. Layering: one place per concern + +Consistent with D-SOC-FENCE §2. Nothing in this plan adds a layer. + +| Layer | Owner | Role in this plan | Never | +|---|---|---|---| +| frontend | XLOG compiler, SQL parser, Cypher parser (`lance-graph-cypher-quack`), Mistaber | produces the plan / AST | evaluates | +| intake | one named function per frontend | external rows → lanes, once; base relations as sets | hides inside an operator | +| describe | `lance-graph-relalg` (`Rel` IR, `join_body::is_alpha_acyclic`, proven rewrites, lowering into one mask-risc program per fold) | the tuple description, gaining a semiring parameter K | evaluates | +| surface | `lance-graph-quack` (`Filter`, `GroupBy`, `Filter::sql_where`, `Semijoin`) | what a caller writes, DuckDB-shaped | evaluates ("may build a Program and must never evaluate one") | +| address | `lance_graph_dt64::dr` (`LocalMap`, `Extent`, `FrameStore`) | global → local, tile width, frame witness | computes | +| bind | `lance-graph-dt64` (`BindStore`) | which method a class's step names (polyfill words by position) | computes | +| evaluate | `lance-graph-mask-risc` | the one fold evaluator | is bypassed | +| compute | `ndarray::simd` | the kernels, including matmul for ℕ-compose | is called from a frontend directly for folds | +| edge | membrane (§12) | rows out | lives inside | + +--- + +## 9. Frontends + +### 9.1 XLOG (first; already in the lab) + +- **Today (VERIFIED-IN-CODE, `crates/xlog-relcomp-probe`, PR #1884):** the admitted two-atom chain's support and bag count run in `src/native.rs` by calling `ndarray::simd` directly. That is a second evaluator, D-SOC-FENCE's first named violation. Facts land as hand-rolled `Vec<(u32, u32)>`. +- **Target (WORKING-MODEL):** + - intake XLOG facts into lanes, as sets (XLOG's own `load_grouped_facts_into_store` skips repeated rows, VERIFIED-IN-CODE at the pinned revision); + - express the chain and `admit_count` as relalg/Quack descriptions with K = B or ℕ, and evaluate through mask-risc; + - ℕ-compose (§7) for `p(X, Z)`; + - the register path stays as the B differential; + - the oracles compare by folds, not expanded rows. +- **Effort estimate:** about 1.5–2 days in the probe. Blast radius: the probe only; no workspace crate changes. + +### 9.2 Base relations are sets + +- The line drawn on PR #1884 for the XLOG case (VERIFIED-IN-CODE, `facts()` dedup, test `a_repeated_base_fact_is_not_a_second_witness`). +- Generalized: + +| Case | Rule | +|---|---| +| the same tuple twice in one base relation | one tuple: the relation is a set at intake | +| the same values in two relations | two facts, never merged | +| two different witnesses of one answer | counted: this is ℕ multiplicity | +| one fact in two representations (lanes and a DR² bit plane) | one fact: the bit plane is a reading, never added | +| provenance of a duplicate (two sources asserting one fact) | not a count; belongs to the witness/evidence path (Epi5 contract), in its own lane if ever needed | + +- **SQL is the exception.** A SQL base table is a bag (no implicit `DISTINCT`), so SQL intake keeps multiplicity where XLOG intake drops it. The frontend states which; the polyfill never guesses. + +### 9.3 SQL / PostgreSQL + +**Template (VERIFIED-IN-CODE):** `lance-graph-cypher-quack`. Its own description: "Cypher over tabular inputs lowered to lance-graph-quack … a demand classifier choosing the carrier, and typed refusals for every other shape. No DataFusion on its call path." + +**Proposed (WORKING-MODEL):** `sql-quack-probe`, a lab crate with the same shape: +- an upstream SQL parser, read-only; +- a demand classifier: Distinct, Bag or count; +- lowering to relalg/Quack with K chosen from SQL semantics; +- typed refusals. + +| SQL | K | Lowering | Status | +|---|---|---|---| +| `SELECT DISTINCT y FROM r JOIN s ON r.y = s.y` | B | §6 join row (B) | words exist | +| `SELECT y FROM r JOIN s …` | ℕ | count lane, rows emitted at the edge with multiplicity | count exists; edge emission is the membrane | +| `SELECT y, COUNT(*) … GROUP BY y` | ℕ | deg × deg | exists (probe) | +| `SELECT r.x, s.z … JOIN …` | ℕ | ℕ-compose | HYPOTHESIS (§7) | +| `WHERE` with NULL | B, three-valued | `Filter::sql_where` | VERIFIED-IN-CODE (Quack) | +| `UNION ALL` / `UNION` | ℕ / B | + / ∨ | words exist (i64 add, `mask_or`) | +| `INTERSECT ALL` / `EXCEPT ALL` | ℕ | min / truncated minus | gaps (§6) | +| `LEFT` / `FULL JOIN` | — | — | refused (needs NULL-extension in the description) | +| `ORDER BY`, `LIMIT` | — | — | **OPEN:** lance-graph's rule is that order belongs to coordinates, not decoded rows. Either a key-ordered projection serves it, or it is a named materialization at the edge | +| `WITH RECURSIVE` | — | semi-naive deltas | OPEN (D-XLOG-DR64-2 shape) | +| `SUM/MIN/MAX` on i32 | ℕ / group folds | `GroupReduce` | exists | +| `SUM` on i64 / numeric, floats | — | — | gaps / refused | + +**Constraints:** +- **Not DataFusion.** It is in its grace period (2026-09-05). `lance-graph-callcenter` stays as the existing Supabase-shaped membrane and is not extended for this. +- **The SQL parser is a dependency decision.** The AdaWorldAPI-fork rule applies; whether a fork exists is OPEN (§13). +- **Reference oracles.** Scalar enumeration, plus a real PostgreSQL if one can run in the lab, and DuckDB (which reads the same data and is the shape Quack follows). + +### 9.4 Epi5 and presence: what bags never touch + +From PR #1884, `tests/epi5_reading.rs`: +- **Multiplicity is not epistemic.** A bag count is never written into CE64 frequency/confidence (bits 24..39) or Epi5 (bits 59..63). +- **If a result is ever written as a CE64 edge:** a base fact is `Direct × Open`, a composed edge with its intermediate retained is `IndirectKnown × Open`, and a join certifies nothing. +- **Absent is not zero.** `raw5 == 0` reads as `Direct × Open`. "Absent" is the class's `FieldMask` presence bit, never a sentinel inside a lane. The same holds for count lanes: an absent cell is not a cell with count 0 unless the description says the population is total. + +### 9.5 Cypher, Mistaber, ONNX + +- **Cypher.** `lance-graph-cypher-quack` exists (experimental, routed nowhere). Its pattern matches are DR² / DR³ bodies, so ℕ-compose serves path counts. +- **Mistaber.** Its Kripke holdings are a least fixpoint over `relcomp::compose` (D-MIST-NATIVE). It is blocked on a production compose, the same gap as §7. +- **ONNX.** Elementwise (`lance-graph-onnx` aliases `ndarray::simd`), so it is not a bag consumer. It shares the polyfill's B words. + +--- + +## 10. The endgame: beat PostgreSQL (and keep Iceberg on the scoreboard) + +### 10.1 What PostgreSQL does, and what we put against it + +| PostgreSQL | How it works | Here | Expected outcome (HYPOTHESIS until §10.3 measures it) | +|---|---|---|---| +| heap tuples, row-oriented pages | a row is read whole | lanes; a query reads only the lanes it names | win on narrow selections over wide tables | +| volcano executor (`next()` per tuple), JIT for expressions | per-tuple dispatch | folds over 64-row words, one evaluator | win on scans, filters and aggregates | +| hash join, merge join, nested loop | build a hash table and probe; copies keys and payload | many-to-one: gather, semijoin. Many-to-many: ℕ-compose on tiles | win when tiles are dense enough for matmul; **may lose** on very sparse many-to-many (to measure) | +| HashAggregate / GroupAggregate | hash table of groups | `GroupReduce`, an O(groups) sink | win | +| bag semantics by default | a row per witness | ℕ count lane; rows only at the edge | win when the answer is a count; even when rows are demanded, emission is at the edge only | +| three-valued NULL logic | per-tuple | `Filter::sql_where`, validity planes | parity required | +| B-tree / BRIN indexes | separate structures | the address prefix (classid / HHTL) and DR coordinates are the index | win on point and prefix lookups; to measure | +| MVCC snapshots | xmin/xmax per tuple | Lance versions, plus DT64 route versions | parity required (snapshot reads) | +| concurrent writers, transactions | row locks, WAL | one writer per mailbox, Lance commits | **different model; state it, don't hide it.** Not a win claimed here | +| extensions, ecosystem, wire protocol | mature | membrane (§12) | **PostgreSQL wins today** | + +### 10.2 Iceberg (the table-format scoreboard) + +- **Pruning.** Iceberg prunes by manifest statistics (min/max per file), which only exclude files that cannot match. The DR address prunes exactly. +- **Deletes.** Iceberg writes delete files; here a delete is a mask (`mask_andnot`). +- **Filtered counts.** Iceberg answers `COUNT(*)` from metadata only without a filter; here filtered and grouped counts are folds. +- **Interop.** Iceberg wins on engine neutrality. The membrane (§12) is the only answer to that. + +### 10.3 Scoreboard protocol (gates, not claims) + +- **Units.** Per operation in ns; batches at most in µs. Cold costs (parse, compile, intake) are reported in µs and are not gates. A millisecond result is a failed gate, never a number to report (lance-graph P0). +- **Referees.** DuckDB on the same data (it also reads Iceberg). A real PostgreSQL where the lab can run one; otherwise stated as missing, never approximated. +- **Query classes:** + + | Class | Example | + |---|---| + | selective filter | point and range | + | filtered + grouped count | — | + | two-table bag join count | XLOG/SQL fixture | + | many-to-many join keeping both ends | ℕ-compose | + | `UNION ALL` / `DISTINCT` | — | + | delete then count | — | + | snapshot read at an older version | — | + +- **Loss cases are mandatory.** Every class runs a dense and a sparse variant. A wide-row, full-scan case is included where PostgreSQL is expected to do well. Results are reported even when they go against us. +- **Correctness first.** Every timed query asserts its answer against the referee before its time counts (`query-stage-profiler` rule). + +--- + +## 11. Waves, D-ids and gates + +| Wave | D-id | Work | Gate (falsifier) | Depends on | Tier | +|---|---|---|---|---|---| +| 0 | D-DYNBAG-0 | Fix PR #1884's layering: XLOG intake as lanes (sets), support and bag count through Quack → mask-risc, oracles by folds, cost probe gains native and Quack columns | all existing Gate B / litho tests green; register differential unchanged; disable runs red; `native.rs` no longer calls `ndarray::simd` fold words | operator go-ahead on #1884 | Opus orchestrates; Sonnet implements from spec | +| 1 | D-DYNBAG-1 | ℕ-compose probe: `matmul_i8_to_i32` over two DR64² tiles for `p(X, Z)` | §7.5: equality with both oracles, B projection equals `relcomp::compose`, disable runs red; timed vs `relcomp::compose` in ns | D-DYNBAG-0 | Sonnet; Opus reviews | +| 2 | D-DYNBAG-2 | Semiring parameter in relalg's `Rel` (B, ℕ) and Quack's surface; typed refusal for unsupported (op, K) | B/ℕ projection property holds for every admitted op; an unsupported pair is refused, never a fallback to B | D-DYNBAG-1 | relalg / Quack owners | +| 3 | D-DYNBAG-3 | Missing ℕ words, backend-first: masked select over i64, elementwise min, truncated minus, i64 compare-to-mask, i64 masked sum; and, if §7.4 measures it necessary, a bit-packed i1 × i1 → i32 product | each word with can-fire and can-stay-silent tests in its owner crate | D-DYNBAG-2 | mask-risc / ndarray owners | +| 4 | D-DYNBAG-4 | DR{dynamic}: width from survivor count, multi-tile global → (tile, local) map, cross-tile fold and compose | answers identical across widths 64 / 256 / multi-tile on the same data; frame refusal on a moved layout | D-DYNBAG-1, DT64 owner | DT64 `dr` owner | +| 5 | D-DYNBAG-5 | `sql-quack-probe` (lab), modelled on `lance-graph-cypher-quack`: the §9.3 table, supported rows plus typed refusals | each supported query equals scalar enumeration and the referee; each refused shape names its refusal | D-DYNBAG-2, §13 parser decision | Sonnet; Opus reviews | +| 6 | D-DYNBAG-6 | Scoreboard (§10.3) against DuckDB and, where possible, PostgreSQL | every class has dense and sparse results, correctness asserted, ns units; losses reported | D-DYNBAG-5 | Opus | +| 7 | D-DYNBAG-7 | Second Datalog frontend through the same path (Scallop: provenance semirings), as the test that the polyfill is frontend-neutral | Scallop programs in the admitted shape answer identically to Scallop | D-DYNBAG-2 | later | + +**Ordering rules:** +- Nothing past Wave 1 starts before the ℕ-compose falsifier has run. It decides whether the many-to-many join has a home. +- Wave 3 words land in their owner crates, never in a probe. + +--- + +## 12. The membrane (reference only) + +On EXO-50, posted 2026-10-31, as an idea to keep in view and not designed here: +- **The question.** Should the external Quack surface get an Abacus-pattern membrane, amortizing at the outer boundary the way `lance-graph-callcenter` does for its Supabase-shaped API? +- **Its open half.** Whether the conversion cost is per call or per frame. + +This plan only notes where it would sit: the "edge" row of §8, and the PostgreSQL wire protocol and row emission of §10.1. It does not design it. + +--- + +## 13. Open decisions + +| # | Decision | Owner | Blocks | +|---|---|---|---| +| 1 | Go-ahead to fix PR #1884's layering (D-DYNBAG-0) | operator | everything | +| 2 | Is `matmul_i8_to_i32`'s per-tile unpack and allocation acceptable, or is a bit-packed word needed? | ndarray owner, after D-DYNBAG-1 measures | D-DYNBAG-3 | +| 3 | Where K lives in relalg's IR: on the relation, on the operation, or both | relalg owner | D-DYNBAG-2 | +| 4 | The multi-tile map's shape (tile index × local position) and its frame witness | DT64 `dr` owner | D-DYNBAG-4 | +| 5 | SQL parser: AdaWorldAPI fork or none | operator | D-DYNBAG-5 | +| 6 | `ORDER BY` / `LIMIT`: key-ordered projection or named edge materialization | operator | D-DYNBAG-5 | +| 7 | Method identity (`MethodSignature` / `MethodRef` / `MethodKey`) | EXO-50 | binding the polyfill words to classes in production | +| 8 | Whether a real PostgreSQL can run as a referee in the lab | operator | D-DYNBAG-6 | + +--- + +## 14. Risks + +| Risk | Mitigation | +|---|---| +| A polyfill layer that starts evaluating (a fourth evaluator) | D-SOC-FENCE `fold-evaluator` and `query-surface` concerns; description layers may build programs, never run them | +| Falling back to B when ℕ is unsupported (support-for-bag) | typed refusal per (op, K); the XLOG fixture as a standing test | +| ℕ-compose loses on sparse tiles | the scoreboard runs sparse variants; a sparse many-to-many may keep a gather-based path; decided by measurement | +| Count overflow in chains | per-hop overflow refusal; i32 bound stated per tile width | +| Claiming a PostgreSQL win from a toy fixture | §10.3: referee, correctness first, dense and sparse, loss cases reported | +| Copy creep at intake (rows kept "for convenience") | one named intake per frontend; D-SOC-FENCE row for hand-rolled tuples | +| Dynamic width turning into in-place resize | resize is a new frame (§4.3) | + +--- + +## 15. What this plan refuses + +- No row engine, no volcano iterator, no hash table for joins or groups. +- No DataFusion on the new path. +- No hand-rolled tuple after intake, in production, probes or oracles. +- No direct `ndarray::simd` fold call from a frontend. +- No multiplicity written into CE64 truth or Epi5 bits. +- No VSA in the holographic reading. +- No PostgreSQL win claimed without the §10.3 protocol. + +--- + +## 16. Cross-references + +- `.claude/plans/d-xlog-native-v1.md`: XLOG crosswalk, Amendments A (binding, Epi5) and B (execution). +- `.claude/plans/d-soc-fence-v1.md`: one place per concern, checked in CI. +- `.claude/plans/d-dt64-6-routing-bindstore-behavior-v1.md`: A.5 addenda 7–12 (polyfill lens, extent, thin provisioning, the `dr` module). +- `crates/xlog-relcomp-probe` (PR #1884): native path, binding, Epi5 tests. +- `crates/lance-graph-cypher-quack`: the SQL probe's template. +- EXO-50: the membrane reference note (2026-10-31). diff --git a/.claude/plans/d-soc-fence-v1.md b/.claude/plans/d-soc-fence-v1.md new file mode 100644 index 000000000..979b99902 --- /dev/null +++ b/.claude/plans/d-soc-fence-v1.md @@ -0,0 +1,119 @@ +# D-SOC-FENCE — one place per concern, checked in CI + +**Status:** PROPOSED, plan only. No tool, workflow, mapping file or code change lands with this plan. + +**Date:** 2026-10-31. **Trigger:** PR #1884's `xlog-relcomp-probe/src/native.rs` counts by calling `ndarray::simd` group and scatter functions directly. That bypasses `lance-graph-quack` (the query surface) and `lance-graph-mask-risc` (the one fold evaluator). D-DT64-6 A.5 addenda 8 and 12 had already ruled that folds name mask-risc's vocabulary and belong to Quack. Nothing mechanical caught it. + +**Rule.** Each concern has exactly one owning place. Code elsewhere reaches that concern only through its owner. The record of owners and exceptions is one mapping file. A CI check reads that file and reports every use outside it as a separation-of-concerns (SoC) violation. Known violations are listed in a baseline that only ever shrinks. + +## 1. Blast radius (why plan-only) + +Counted on 2026-10-31 at `84f80eb` with `git grep`. **These are candidate counts, not findings**: a search finds candidates and only reading them decides. + +| What was searched | Candidate files, by crate | +|---|---| +| `ndarray::simd` masking, group, scatter, gather or popcount functions imported outside mask-risc | lance-graph-planner 5, dir-rmap-probe 5, math-polyfill-probe 4, drf-focus-probe 4, crepe-plan-probe 4, cognitive-shader-driver 4, soa-view-probe 3, lance-graph-onnx 3, tract-harvest-probe 2, lance-graph-benches 2, d-diamond-1-probe 2, r2il-mask-abi-probe 1, lance-graph-quack 1, lance-graph-contract 1, jc 1 | +| `ndarray::hpc::` imported | thinking-engine 32, bgz-tensor 13, lance-graph-planner 10, lance-graph 4, and 7 other crates with fewer than 4 | +| `core::arch` / `std::arch` | 8 crates. Some hits are feature detection (`is_x86_feature_detected!`), not intrinsics | +| CE64 bits 59..63 written by shift outside `causal-edge` | cognitive-shader-driver examples (2 files) | +| `mask_risc::execute*` called from Quack | 3 sites, all after a `#[cfg(test)]` at line 2510 (test-only, to confirm) | + +Two false-positive classes are already visible: test-only code, and feature detection. A check that fails on the first run would block unrelated PRs across about 25 crates. So the check starts in report-only mode (Wave 1), and blocks only new violations once the baseline is committed (Wave 2). + +## 2. The mapping file: `.claude/soc/one-place.toml` + +The mapping file is the only place where an owner is written down. Docs, cards and CLAUDE.md files link to it rather than restating it. + +```toml +[[concern]] +id = "fold-evaluator" +statement = "Masked folds (keyed count/sum/min/max, scatter, gather, popcount terminals) are evaluated by mask-risc" +owner = { crate = "lance-graph-mask-risc", paths = ["src/exec.rs"] } +basis = ["D-DT64-6 A.5 addendum 8", "lance-graph-mask-risc Cargo description"] +symbols = ["ndarray::simd::masked_group_*", "ndarray::simd::mask_scatter_*", "ndarray::simd::mask_gather_*"] +reach_via = ["lance_graph_quack::*", "lance_graph_mask_risc::{Program, execute, execute_into}"] + +[[concern.sanctioned]] +crate = "lance-graph-onnx" +scope = "elementwise aliases only (mask_and/or/not, *_to_mask)" +basis = "D-ONNX-NATIVE" +``` + +Every concern carries five things: +- `statement`: the concern in one sentence; +- `owner`: crate and paths; +- `basis`: the ruling or contract it comes from; +- `symbols`: what only the owner may name; +- `reach_via`: what everyone else uses instead. + +A sanctioned exception carries a named crate, a scope and a basis. It is never a blanket exemption. + +Each use the check finds is classified as one of: + +| class | meaning | +|---|---| +| `OWNER` | inside the owner's paths | +| `SANCTIONED` | a named exception with its scope and basis | +| `TEST` | in `#[cfg(test)]`, `tests/` or `benches/`. Allowed by default; a concern can opt out | +| `TRACKED` | a known violation listed in the baseline, with an owner and a D-id | +| `VIOLATION` | anything else; fails CI from Wave 2 | + +### Concerns to confirm in Wave 0 + +Each row below is a proposal. Wave 0 confirms or corrects it by reading the cited basis and the owner's code. No row enters the file on this plan's say-so. + +| id | owner (proposed) | everyone else goes through | basis | +|---|---|---|---| +| `simd-facade` | `ndarray::simd` (the AdaWorldAPI fork) | `ndarray::simd::*`, never `ndarray::hpc::*`, `core::arch` intrinsics or a second SIMD crate | lance-graph-java and tesseract-rs `simd-savant`; mask-risc Cargo description ("T1 facade"). In lance-graph, which `hpc` modules are re-exported by `simd` is not yet established. Wave 0 decides whether `hpc` imports are violations or sanctioned | +| `fold-evaluator` | `lance-graph-mask-risc` | Quack, or a mask-risc `Program` | addendum 8; Quack crate docs | +| `query-surface` | `lance-graph-quack` builds Programs and never evaluates them | — | Quack crate docs ("may build a Program and must never evaluate one") | +| `selection-description` | `lance-graph-relalg` | — | addendum 8 ("relalg describes") | +| `dr-addressing` | `lance_graph_dt64::dr` (`FrameStore`, `Guards`, `Focus`, `Scope`, `LocalMap`, `ClassAxis`) | `dt64::dr` | addendum 12. The `dr-math-binding-probe` copies are `TRACKED` duplicates | +| `bind-store` | `lance-graph-dt64` (`BindStore`, `MethodBinding`, the route log) | `BindStore` | D-DT64-6 D4/D5 | +| `method-identity` | **OPEN** (`MethodRef` vs `MethodSignature` vs `MethodKey`, EXO-50) | — | listed with no owner; not enforced until decided | +| `classid-split` | `ogar_codebook::{classid_concept, classid_canon, …}`, `dt64::view_class` | those functions; no bit math on a composed classid | `/v3-audit` item 1 | +| `epi5-field` | `CausalEdge64::{epistemic_raw5, with_epistemic_raw5}`, `Epi5Declarations::project_state5` | those; no shifts or masks on bits 59..63 elsewhere | `epistemic_state5.rs` module docs | +| `presence` | the class's `FieldMask` / `WideFieldMask` | presence bits; never a sentinel value inside a field | `class_view.rs` C2 | +| `tuple-template` | relalg (describes) and Quack (surface); the pointer table is `dt64::dr::LocalMap`; one named intake per frontend | descriptions over lanes; no owned row (`Vec<(..)>`, row structs) after intake, in production, probes or oracles | lance-graph CLAUDE.md P0 ("a tuple is a zero-copy fold template"); D-DYNBAG §2, §8 | + +`/v3-audit` keeps its own checks. Its item 1 moves into the mapping file, so the rule is written in one place, and the command reads it from there. + +## 3. The check: `soc-fence` + +There are two levels, and neither relies on a regex for its verdict. + +1. **Dependency fence (exact).** `cargo metadata` runs over the root workspace and over every standalone probe workspace (each `crates/*/Cargo.toml` with its own `[workspace]`). It lists which crates depend directly on each owner's underlying dependency (for example `ndarray`). A crate may depend on `ndarray` directly only if it is an owner or sanctioned for at least one concern that needs it. +2. **Symbol fence (parsed).** A small Rust tool built on `syn` parses every `.rs` file. + - It resolves `use` trees (groups, renames, globs, `self`) and fully qualified paths in expressions and macros, and records `#[cfg(test)]` scope. + - It reports `(concern, crate, file, line, symbol, class)`. + - It is `syn` rather than `grep` because the measured failures here are grep failures: G11 (lance-graph-java) missed fully qualified imports, and CLAUDE.md records `impl lance_graph_contract::hotplug::CapabilityAuthority for …` evading `impl.*CapabilityAuthority`. + - Precedent for a Rust tool under `.claude/tools/`: `preflight_drift.rs`. + +**Baseline** (`.claude/soc/known-violations.toml`). One entry per `TRACKED` use: `(concern, crate, file, symbol, owner, d_id, reason)`. It is a ratchet in both directions. A new violation fails CI, and so does a baseline entry that no longer matches anything. Fixing a violation therefore means deleting its line in the same PR, and the list cannot outlive the code (as with `known_integration_call_breaches.json` in ghidra-mcp). + +**Workflow** (`.github/workflows/soc-fence.yml`). Runs on changes to `crates/**`, `.claude/soc/**` and the tool itself. In Wave 1 it reports and always passes. From Wave 2 it fails on `VIOLATION` and on a stale baseline entry. + +**Its own falsifiers** (in the tool's tests). A fixture crate holds one violation per concern, written in three spellings: a plain `use`, a grouped `use`, and a fully qualified path in an expression. Each must be reported (the can-fire half). A clean fixture holds owner, sanctioned and test-only uses, and must report nothing (the can-stay-silent half). Removing any concern from the mapping file must turn its fixture test red. Feature detection (`is_x86_feature_detected!`) is in the clean fixture. + +## 4. Waves + +| Wave | D-id | Work | Gate | Tier | +|---|---|---|---|---| +| 0 | D-SOC-FENCE-0 | Census. Read every candidate in §1 and classify it OWNER / SANCTIONED / TEST / VIOLATION. Confirm or correct each concern in §2 against its basis. Write the mapping file draft and the baseline draft | every baseline entry cites the file and line it read; each concern's basis is quoted, not paraphrased | Haiku `harvest-scout` / `quote-harvester` for candidates and quotes; Sonnet classifies; Opus rules on owner disputes | +| 1 | D-SOC-FENCE-1 | The `syn` tool, its fixture falsifiers, the workflow in report-only mode | fixture can-fire and can-stay-silent both green; a disable run per concern red | Sonnet from a written spec; Opus reviews | +| 2 | D-SOC-FENCE-2 | Commit the mapping file and baseline; the workflow blocks new violations and stale entries | run on main: zero `VIOLATION`, baseline matches exactly | Opus | +| 3 | D-SOC-FENCE-3 | Burn down the baseline, one PR per crate, each deleting its own baseline lines. Each move routes the crate through its concern's `reach_via` (for example XLOG's counting through Quack → mask-risc) | each PR: the crate's tests green; its baseline lines gone | owning sessions | +| 4 | D-SOC-FENCE-4 | Fold `/v3-audit` item 1 and the lance-graph-java G11 allowlist into the mapping file, so each fence has one written source | `/v3-audit` and the G11 test both read the file; their old inline lists deleted | Opus | + +Nothing in Waves 0–2 changes production code. Wave 3 is the only wave that touches crates, and it splits by owner. + +## 5. Decisions needed before Wave 0 + +1. **Which concerns are in.** The ten rows of §2 are proposals; `method-identity` stays unenforced until EXO-50 decides. +2. **Whether `ndarray::hpc::*` imports are violations** in lance-graph (thinking-engine 32, bgz-tensor 13, planner 10). The lance-graph-java and tesseract-rs rule says never; whether lance-graph follows it has not been ruled. +3. **Whether lab probes are in scope.** Proposed: yes, as `TRACKED` entries, so a probe's shortcut is visible rather than exempt. +4. **Tool language.** Proposed: Rust with `syn`, to resolve paths. A Python regex tool would repeat the measured G11 failure. + +## 6. The first known violation + +PR #1884's `crates/xlog-relcomp-probe/src/native.rs` (concern `fold-evaluator`). It is to be fixed in that PR, by routing the counts through Quack → mask-risc and keeping only `checked_binary_i64` direct, rather than entered in the baseline. diff --git a/.claude/plans/d-xlog-native-v1.md b/.claude/plans/d-xlog-native-v1.md index 3f3041307..2d2088f09 100644 --- a/.claude/plans/d-xlog-native-v1.md +++ b/.claude/plans/d-xlog-native-v1.md @@ -1,6 +1,6 @@ # D-XLOG-NATIVE — XLOG plan vocabulary over the ndarray::simd polyfill (the algebra under lance-graph-onnx) -**Status:** PROPOSED / DOCS ONLY. This adds no crate, code, ABI or test. It is the crosswalk that decides whether a `lance-graph-xlog` crate, wired like `lance-graph-onnx`, is worth creating. +**Status:** PROPOSED. The crosswalk (§2) is docs only; it decides whether a `lance-graph-xlog` crate, wired like `lance-graph-onnx`, is worth creating. Amendment A (2026-10-31) records what moved since: the binding now names mask-risc's op vocabulary, the probe binds through the production bind store, and XLOG results have a stated Epi5 reading. **Date:** 2026-10-30. **Parent work:** D-XLOG-DR64-0 (#1857), D-LITHO-1 (#1861, #1864; D-LITHO is now folded into D-DT64-6 Amendment A). **Precedent:** [D-ONNX-NATIVE](d-onnx-native-v1.md). @@ -66,7 +66,7 @@ Status of every row: **name-checked, not tested.** Each native target is a funct | `Count`, keyed by a column of one table | `masked_group_count_u32` | INLINE | `GroupReduce { Lane, Count }` | keys past the group universe are dropped natively; the adapter refuses them instead | | `Count` keyed through a many-to-one key | `masked_group_count_u32_via` | INLINE | `GroupReduce { Via, Count }` | | | `Count` with no key | `popcount_batch_u64` | ALIAS | `Terminal::Count` | | -| `Count` over a many-to-many join (the D-LITHO-1 multiplicity fixture) | none | NATIVE | none | deg_R(y) × deg_S(y): both factors are `masked_group_count_u32` results, but their per-key product is no native function. The probe's `BagNeedsMultiplicity` refusal stands | +| `Count` over a many-to-many join (the D-LITHO-1 multiplicity fixture) | `masked_group_count_u32` twice, then `checked_binary_i64(I64Op::Mul)` | NATIVE | `GroupReduce { Lane, Count }` twice | deg_R(y) × deg_S(y). The binding exists (Amendment A); no executor runs it, and the probe's `BagNeedsMultiplicity` refusal stands. The multiply word is the ndarray #407 re-export, not checked from here | | `Sum` on an `I32` column | `masked_group_sum_i32` / `masked_sum_i32` | INLINE | `GroupSumI32` / `MaskedSumI32` | same row bound as the mask-risc terminal; beyond it, refuse. `masked_sum_i32` is the symbol ONNX's `ReduceSum(Where(..))` composition uses | | `Min` / `Max` on an `I32` column | `masked_group_min_i32` / `masked_group_max_i32` | INLINE | `GroupReduce { MinI32/MaxI32 }` | an empty group must be reported as absent, not as the seed value | | `LogSumExp` | none | REFUSE | — | a float, probabilistic aggregate | @@ -107,6 +107,7 @@ Status of every row: **name-checked, not tested.** Each native target is a funct - [ ] D-XLOG-NATIVE-2: in the probe, test `masked_group_count_u32` keyed by one table's column against the RIR walk, including the out-of-universe refusal. - [ ] D-XLOG-NATIVE-3: ask the ndarray and mask-risc owners for a native home for the many-to-many join; the probe's Gate B is the fixture. - [ ] D-XLOG-NATIVE-4: create the crate once §4 holds. +- [x] D-XLOG-NATIVE-5: bind the bag count through the production bind store (`lance-graph-dt64` and its `dr` module), and state the Epi5 reading of XLOG results (Amendment A). ## 6. Gates @@ -115,3 +116,50 @@ Status of every row: **name-checked, not tested.** Each native target is a funct - **N3, typed refusal.** Every REFUSE and NATIVE row is a typed refusal in the adapter, never a silent fallback. - **N4, no dependency leak.** `cargo tree -p lance-graph-xlog` shows `ndarray` as the only normal dependency, and no `xlog-*` or arrow 60. - **N5, one algebra.** Every operation both facades name is the same native symbol in `lance-graph-onnx` and `lance-graph-xlog`. + +## Amendment A (2026-10-31): the binding's vocabulary, the production bind store, Epi5 + +**A.1 What a binding names.** D-DT64-6 A.5 addenda 8 and 9 (operator) settle the catalogue-id question: there are no ids. The polyfill is one lens classid, its words are that lens's fields labelled through its `ClassView`, and a word is a field position (`AlgebraRef { lens, word }`). For folds the word is mask-risc's op vocabulary (`Pred`, `MaskOp`, `Terminal`); `ndarray::simd` is named only where mask-risc has no word, such as `checked_binary_i64`. So in §2 the **mask-risc column is what a binding names**, and the `ndarray::simd` column is what an executor would call. The ALIAS / INLINE categories and gates N1–N5 are about the crate's execution symbols and are unchanged. + +**A.2 The probe binds through production.** `crates/xlog-relcomp-probe/tests/dto_binding.rs` now routes the answer positions with `BindStore::route`, seals frames with `lance_graph_dt64::dr::FrameStore`, labels fields through `Resolved::field_view`, and admits the count by `MethodRef` (`Population::Bag`). Its steps name two polyfill words: mask-risc's keyed count twice, then the checked i64 multiply. Execution stays in the lab: `bind_dr64` and `relcomp` are not in `dt64::dr` (addendum 12), so the execution test carries the admitted population into the lab register, which refuses it (`BagNeedsMultiplicity`). `tests/litho.rs` keeps the lab frame store because `relcomp::Chain` only accepts that one. + +**A.3 Epi5 (CE64 bits 59..63).** If an XLOG result is ever written as a `CausalEdge64`: + +| result | `EpistemicState5` | raw5 | +|---|---|---| +| base fact (`r(1, 2)`) | `Direct × Open` | 0 | +| composed edge (`r ; s`, X to Z through a retained Y) | `IndirectKnown × Open` | 1 | + +- A join certifies nothing (`Open`). +- If an answer keeps no witness of Y, its topology is `IndirectUnknown`. That case is stated, not tested. +- A bag count is a value field of the binding, never a certification and never frequency/confidence. +- `raw5 == 0` is `Direct × Open`, so an unstamped composed edge silently claims a direct relation. +- The reading is opaque until the XLOG lens has an `Epi5Declarations` entry. That is an OGAR declaration, including which rail it sits on; the probe uses a lab placeholder. + +`tests/epi5_reading.rs` pins these rules. + +**A.4 Still open.** Method identity: `MethodSignature` vs `MethodRef` vs Tract's `MethodKey` (EXO-50). Whether a frame records the store's catalogue extent (addendum 7, D8 owner). The XLOG and polyfill lenses' production classids and the XLOG Epi5 declaration (OGAR). A native home for the many-to-many join (D-XLOG-NATIVE-3). + +## Amendment B (2026-10-31): execution on the `ndarray::simd` polyfill + +Operator: refactor XLOG onto the `ndarray::simd` polyfill (CubeCL as an intermediary only in the worst case). It was not needed: every function below is on the `ndarray::simd` facade, including `checked_binary_i64` (ndarray #407, merged and read in the checkout). + +`crates/xlog-relcomp-probe/src/native.rs` is now the probe's execution path for the admitted two-atom chain. Each step is one function: + +| step | `ndarray::simd` | +|---|---| +| a bound end (`X == x`, `Z == z`) | `eq_u32_to_mask` | +| each side projected onto Y | `mask_scatter_or_u32` (the demanded result, never an intermediate) | +| support = forward ∧ backward | `mask_and` | +| rows per Y on each side | `masked_group_count_u32` | +| derivations per Y | `checked_binary_i64(I64Op::Mul)`; an overflow is refused | + +- The Y domain is DR's `lance_graph_dt64::dr::LocalMap`, at 64 or 256 wide. The native path is therefore not limited to one 64-slot tile; a 200-Y fixture is answered that the register refuses. +- `admit_count` admits XLOG's compiled `count` rule (`Count` keyed by the answer column, over a join `admit` accepts) and refuses any other aggregate (`UnsupportedAggregate`). +- The DR64 register path (`propagate`) stays as the differential. Every Gate B answer asserts that the native forward, backward and answer masks equal the register's. +- The overflow refusal is not exercised by a test: per-side counts are bounded by the row count, so no fixture reaches `i64`. + +Effect on §2: +- The two-atom many-to-many chain projected onto Y (support and bag count) now has a native home. Its rows move from NATIVE to INLINE: they are tested against both oracles. +- Still NATIVE: a many-to-many join whose answer keeps both ends (X, Z), and chains of more than two atoms. +- §4 condition 2 holds for the projected chain only. diff --git a/crates/xlog-relcomp-probe/Cargo.toml b/crates/xlog-relcomp-probe/Cargo.toml index 0ec0ace3d..e4c3c6bb2 100644 --- a/crates/xlog-relcomp-probe/Cargo.toml +++ b/crates/xlog-relcomp-probe/Cargo.toml @@ -17,6 +17,11 @@ description = "Lowers a restricted, XLOG-compiled RIR (BrainyBlaze/xlog, CPU-onl # The existing native relation register, binding guards and compose kernel. # Consumed read-only through its public API. dr-math-binding-probe = { path = "../dr-math-binding-probe" } +# The execution path: the ndarray::simd polyfill, on the coordinate +# lance-graph-onnx and lance-graph-mask-risc use. +ndarray = { git = "https://github.com/AdaWorldAPI/ndarray", branch = "master", default-features = false, features = ["std"] } +# DR's global -> local map for the answer axis (lance_graph_dt64::dr::LocalMap). +lance-graph-dt64 = { path = "../lance-graph-dt64" } # XLOG's CUDA-free compiler crates, pinned to one upstream commit # (MIT OR Apache-2.0; notices in the upstream repository). xlog-core = { git = "https://github.com/BrainyBlaze/xlog", rev = "bff6da663308190148c5ce58a715d86525b62e67" } @@ -27,6 +32,8 @@ xlog-logic = { git = "https://github.com/BrainyBlaze/xlog", rev = "bff6da6633081 # The shared contract types a DTO binding is stated in (NodeGuid, ClassView, # WideFieldMask). Same path crate dr-math-binding-probe already uses. lance-graph-contract = { path = "../lance-graph-contract" } +# CausalEdge64, for the Epi5 reading of XLOG results (tests/epi5_reading.rs). +causal-edge = { path = "../causal-edge" } [patch."https://github.com/AdaWorldAPI/lance-graph"] lance-graph-contract = { path = "../lance-graph-contract" } diff --git a/crates/xlog-relcomp-probe/README.md b/crates/xlog-relcomp-probe/README.md index 2d4afd733..97bb01e4d 100644 --- a/crates/xlog-relcomp-probe/README.md +++ b/crates/xlog-relcomp-probe/README.md @@ -296,21 +296,49 @@ two things nothing defines yet: - a per-Y product: for a two-atom chain the count is deg_R(y) · deg_S(y) (XLOG's own factorized count multiplies trie-range lengths the same way). -**The same operation as a DTO binding** (`tests/dto_binding.rs`, EXO-50 -message 19). DR routes first: lab ids map to local slots under a sealed frame. -Then DT64 binds meaning: - -- identities: `NodeGuid` references (V3 tail, lab classid `0x7F01 << 16`, a - stand-in, not a minted concept), decoding back to the Y ids 2, 3, 7; -- property mappings: R and S as fields 0 and 1 of a `ClassView`, selected by - `WideFieldMask::from_positions(&[0, 1])`; -- method: `(owner_class 0x7F01, method_ordinal 1)` with `Population::Bag`, - named by reference only; +**The same operation through the production bind store** +(`tests/dto_binding.rs`, `lance-graph-dt64`). DR routes first: the answer +positions (the Y ids 2, 3, 7) are routed to the XLOG lens in the bind store's +route log, under a sealed `dt64::dr::FrameStore`. Then DT64 binds meaning: + +- identities: `NodeGuid` references (V3 tail, lab lens classid `0x7F01_0001`, + a stand-in, not a minted concept); +- property mappings: R and S as fields 0 and 1 of the lens, labelled by its + `ClassView` through `Resolved::field_view`; +- method: a `MethodBinding` of `Population::Bag`, admitted by `MethodRef`. + Its three steps name polyfill words by position: mask-risc's keyed count + (`Terminal::GroupReduce`) for R and for S, then + `ndarray::simd::checked_binary_i64(Mul)`. The polyfill is one lab lens + (`0x7F0A_0001`); no word has a classid; - expected values: {2: 4, 3: 1, 7: 1}, keyed by identity. -The binding refuses a stale route (`MapChanged`), and executing the bound Bag -method is refused by `bind_dr64` (`BagNeedsMultiplicity`): that is the failing -half. No DT64 bind store exists (D-DT64-6 is PROPOSED); the binding is a -test-local value and persists nothing. Disable runs: dropping the route check -turns `dt64_does_not_bind_against_a_stale_route` red; declaring the method -Distinct turns the method and refusal tests red. +The binding refuses a stale frame (`MapChanged`) and a retracted position +(`Unrouted`; the old route version still resolves). Execution has no +production home: the admitted population is carried into the lab register, +whose `bind_dr64` refuses it (`BagNeedsMultiplicity`). That is the failing +half. `tests/litho.rs` keeps the lab frame store, because `relcomp::Chain` +accepts only that one. + +**Epi5** (`tests/epi5_reading.rs`). An XLOG result written as a +`CausalEdge64` carries `Direct x Open` for a base fact and +`IndirectKnown x Open` for a composed edge whose Y is retained. A join +certifies nothing, and a bag count is never a certification or a truth value. +`raw5 == 0` reads as `Direct x Open`, so an unstamped composed edge would +claim a direct relation. Without a declaration for the XLOG lens, projection +refuses (`UndeclaredClass`). + +## Execution on the `ndarray::simd` polyfill (`src/native.rs`) + +The admitted chain is answered with `ndarray::simd` functions, one per step: + +- a bound end: `eq_u32_to_mask`; +- each side projected onto Y: `mask_scatter_or_u32`; +- the support: `mask_and`; +- the bag count: `masked_group_count_u32` for each side, then `checked_binary_i64(Mul)`. + +The Y axis is DR's `lance_graph_dt64::dr::LocalMap` (64 or 256 wide). `admit_count` admits XLOG's `count` rule. The DR64 register path is kept as the differential: `tests/gate_b.rs` asserts that every native mask equals the register's. `tests/litho.rs` checks the native counts against both oracles. Disable runs, each turning tests red: + +- dropping the AND: 5 failed; +- adding instead of multiplying: 2 failed; +- ignoring a bound: 5 + 1 failed; +- admitting any aggregate: 1 failed. diff --git a/crates/xlog-relcomp-probe/src/lib.rs b/crates/xlog-relcomp-probe/src/lib.rs index 1e2251553..681329955 100644 --- a/crates/xlog-relcomp-probe/src/lib.rs +++ b/crates/xlog-relcomp-probe/src/lib.rs @@ -38,6 +38,8 @@ use std::collections::BTreeMap; +pub mod native; + use dr_math_binding_probe::relcomp::{ComposeRefusal, Rows, Work, compose}; use dr_math_binding_probe::{ Axis, BoundDr64, Focus, FrameStore, Guards, MethodSignature, Operand, Refusal, Scope, bind_dr64, @@ -87,6 +89,8 @@ pub enum LowerRefusal { /// A `ChainJoin` whose own fields and whose `fallback` plan describe /// different queries. ChainJoinDisagrees, + /// A `GroupBy` other than one `Count` keyed by the answer column. + UnsupportedAggregate, } /// The admitted rule body, in RIR terms. @@ -285,7 +289,13 @@ pub fn facts(program: &Program, predicate: &str) -> Result, Lowe _ => None, }; match (id(&rule.head.terms[0]), id(&rule.head.terms[1])) { - (Some(a), Some(b)) => out.push((a, b)), + // A base relation is a set: XLOG's fact loader skips a row it + // has already seen, so a repeated fact is not a second witness. + (Some(a), Some(b)) => { + if !out.contains(&(a, b)) { + out.push((a, b)); + } + } _ => { return Err(LowerRefusal::FactTerm { predicate: predicate.to_string(), diff --git a/crates/xlog-relcomp-probe/src/native.rs b/crates/xlog-relcomp-probe/src/native.rs new file mode 100644 index 000000000..b1ff8f791 --- /dev/null +++ b/crates/xlog-relcomp-probe/src/native.rs @@ -0,0 +1,229 @@ +//! The admitted chain query and its bag count, computed on the +//! `ndarray::simd` polyfill. +//! +//! This is the execution path of the probe. The DR64 register path +//! (`propagate`, over `dr-math-binding-probe::relcomp`) stays as a +//! differential cross-check: it answers support only, and it is limited to +//! one 64-slot tile. +//! +//! Every step is one `ndarray::simd` function, with no loop over rows of our +//! own: +//! +//! | step | function | +//! |---|---| +//! | a bound end (`X == x`, `Z == z`) | `eq_u32_to_mask` | +//! | the other relation's ends projected onto Y | `mask_scatter_or_u32` (result only) | +//! | forward ∧ backward | `mask_and` | +//! | rows per Y, on each side | `masked_group_count_u32` | +//! | derivations per Y = left count × right count | `checked_binary_i64(Mul)` | +//! +//! The Y domain is DR's: a `lance_graph_dt64::dr::LocalMap` (global id → +//! local position), prepared by the caller and never derived here. A Y id the +//! map does not hold is refused. A product past `i64` is refused, never +//! wrapped. +//! +//! The only Rust loops here relabel inputs and outputs. They build the key +//! columns from the tuples (one local lookup per row) and decode the answer +//! bits back to ids. Neither is a computation over the population. + +use lance_graph_dt64::dr::{LocalMap, MapRefusal}; +use ndarray::simd::{ + I64ArithError, I64Op, checked_binary_i64, eq_u32_to_mask, mask_and, mask_scatter_or_u32, + masked_group_count_u32, +}; +use xlog_core::AggOp; +use xlog_ir::{ProjectExpr, RirNode}; + +use crate::{ChainQuery, LowerRefusal, admit}; + +/// Why the native path did not answer. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum NativeRefusal { + /// A Y id outside DR's local map. + Domain(MapRefusal), + /// A per-Y derivation count past `i64`. + Count(I64ArithError), +} + +/// The masks over the Y domain, one bit per local position. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Support { + /// Y with a left partner (under the left bound, if any). + pub forward: Vec, + /// Y with a right partner (under the right bound, if any). + pub backward: Vec, + /// `forward & backward`: the answer. + pub answer: Vec, +} + +/// One relation's columns: the free end and the Y end as local positions. +struct Side { + free: Vec, + y_local: Vec, +} + +fn side( + tuples: &[(u32, u32)], + y_of: fn(&(u32, u32)) -> u32, + free_of: fn(&(u32, u32)) -> u32, + mid: &LocalMap, +) -> Result { + let y_local = tuples + .iter() + .map(|t| mid.local(u64::from(y_of(t))).map(u32::from)) + .collect::>() + .map_err(NativeRefusal::Domain)?; + Ok(Side { + free: tuples.iter().map(free_of).collect(), + y_local, + }) +} + +/// The selected rows of one side: all of them, or those whose free end +/// equals the bound. +fn rows(side: &Side, bound: Option) -> Vec { + let n = side.free.len(); + let mut words = vec![0u64; n.div_ceil(64)]; + match bound { + Some(v) => eq_u32_to_mask(&side.free, v, &mut words), + None => { + for (i, w) in words.iter_mut().enumerate() { + let live = (n - i * 64).min(64); + *w = if live == 64 { + u64::MAX + } else { + (1u64 << live) - 1 + }; + } + } + } + words +} + +fn sides( + left: &[(u32, u32)], + right: &[(u32, u32)], + mid: &LocalMap, +) -> Result<(Side, Side), NativeRefusal> { + // r(X, Y): Y is column 1, X free. s(Y, Z): Y is column 0, Z free. + Ok(( + side(left, |t| t.1, |t| t.0, mid)?, + side(right, |t| t.0, |t| t.1, mid)?, + )) +} + +/// The support of `q(Y) :- r(X, Y), s(Y, Z)` (with optional bounds on X and +/// Z) over DR's Y map. +pub fn support( + q: &ChainQuery, + left: &[(u32, u32)], + right: &[(u32, u32)], + mid: &LocalMap, +) -> Result { + let (l, r) = sides(left, right, mid)?; + let n = mid.len(); + let words = n.div_ceil(64); + let mut forward = vec![0u64; words]; + let mut backward = vec![0u64; words]; + mask_scatter_or_u32(&rows(&l, q.left_bound), &l.y_local, &mut forward, n); + mask_scatter_or_u32(&rows(&r, q.right_bound), &r.y_local, &mut backward, n); + let mut answer = vec![0u64; words]; + mask_and(&forward, &backward, &mut answer); + Ok(Support { + forward, + backward, + answer, + }) +} + +/// Per-Y derivation counts of the same body under bag semantics: one per +/// `(x, y, z)` witness, as XLOG's `count` counts body bindings. Returned for +/// every local position; a Y with no derivation has 0. +pub fn bag_count( + q: &ChainQuery, + left: &[(u32, u32)], + right: &[(u32, u32)], + mid: &LocalMap, +) -> Result, NativeRefusal> { + let (l, r) = sides(left, right, mid)?; + let n = mid.len(); + let mut per_left = vec![0i64; n]; + let mut per_right = vec![0i64; n]; + masked_group_count_u32(&rows(&l, q.left_bound), &l.y_local, &mut per_left); + masked_group_count_u32(&rows(&r, q.right_bound), &r.y_local, &mut per_right); + let mut out = vec![0i64; n]; + checked_binary_i64(I64Op::Mul, &per_left, &per_right, &mut out) + .map_err(NativeRefusal::Count)?; + Ok(out) +} + +/// Global ids of the set bits of `mask`, in local order. +pub fn ids(mid: &LocalMap, mask: &[u64]) -> Vec { + (0..mid.len()) + .filter(|&i| mask[i / 64] >> (i % 64) & 1 == 1) + .filter_map(|i| mid.global(i as u8)) + .collect() +} + +/// `(global id, count)` for every Y with at least one derivation, in local +/// order. +pub fn nonzero(mid: &LocalMap, counts: &[i64]) -> Vec<(u64, i64)> { + counts + .iter() + .enumerate() + .filter(|(_, c)| **c != 0) + .filter_map(|(i, c)| mid.global(i as u8).map(|g| (g, *c))) + .collect() +} + +/// Admit XLOG's compiled `count` rule over the chain: +/// `qc(Y, count(W)) :- r(X, Y), s(Y, Z)`, which XLOG compiles to +/// `Project[0, 1](GroupBy{key 0, Count of 1}(Project[Y, W](Join)))`. +/// The join must be the one [`admit`] accepts. Any other aggregate, key or +/// projection is refused. +pub fn admit_count(body: &RirNode) -> Result { + let RirNode::Project { input, columns } = body else { + return Err(LowerRefusal::UnsupportedNode(name(body))); + }; + if !matches!( + columns.as_slice(), + [ProjectExpr::Column(0), ProjectExpr::Column(1)] + ) { + return Err(LowerRefusal::UnsupportedProjection); + } + let RirNode::GroupBy { + input, + key_cols, + aggs, + } = input.as_ref() + else { + return Err(LowerRefusal::UnsupportedNode(name(input))); + }; + if key_cols.as_slice() != [0] || aggs.as_slice() != [(1, AggOp::Count)] { + return Err(LowerRefusal::UnsupportedAggregate); + } + let RirNode::Project { input, columns } = input.as_ref() else { + return Err(LowerRefusal::UnsupportedNode(name(input))); + }; + // `count(W)` over an inner join counts rows: no column is null, so which + // column W is does not matter. Y must be a join column `admit` accepts. + let [ProjectExpr::Column(y), ProjectExpr::Column(_)] = columns.as_slice() else { + return Err(LowerRefusal::UnsupportedProjection); + }; + admit(&RirNode::Project { + input: input.clone(), + columns: vec![ProjectExpr::Column(*y)], + }) +} + +fn name(n: &RirNode) -> &'static str { + match n { + RirNode::Project { .. } => "Project", + RirNode::GroupBy { .. } => "GroupBy", + RirNode::Join { .. } => "Join", + RirNode::ChainJoin { .. } => "ChainJoin", + RirNode::Scan { .. } => "Scan", + RirNode::Filter { .. } => "Filter", + _ => "other", + } +} diff --git a/crates/xlog-relcomp-probe/tests/dto_binding.rs b/crates/xlog-relcomp-probe/tests/dto_binding.rs index 5a5bc152a..bbaaef9cc 100644 --- a/crates/xlog-relcomp-probe/tests/dto_binding.rs +++ b/crates/xlog-relcomp-probe/tests/dto_binding.rs @@ -1,30 +1,34 @@ -//! The bag-multiplicity operation stated as a DTO binding over coordinates DR -//! has already routed (EXO-50 message 19): which identities, which property -//! mappings, which method by reference, and the expected values. +//! The bag-multiplicity operation bound through the production bind store +//! (`lance-graph-dt64`): which identities, which property mappings, which +//! method by reference, and the expected values. //! //! The stages stay separate and in order: //! -//! 1. **DR routes.** Lab ids map to local slots (`AddressMap`) under a sealed -//! frame (`FrameStore`). Dimensions are DR's; nothing here chooses one. -//! 2. **DT64 binds meaning.** Identities are `NodeGuid`s; the relations R and -//! S are property mappings named by a `ClassView` field set and selected by -//! a `WideFieldMask`; the counting method is named by -//! `(owner_class, method_ordinal)` with `Population::Bag`, never embedded. -//! 3. **Execution** binds the registers. For a Bag method it refuses today +//! 1. **DR routes.** Lab ids are DR positions routed to the XLOG lens classid +//! in the bind store's route log, under a sealed `dt64::dr::FrameStore`. +//! 2. **DT64 binds meaning.** The lens's fields (r, s and the three counts) +//! are labelled by its `ClassView` and selected by a `WideFieldMask`. The +//! counting method is a `MethodBinding` of `Population::Bag`, reached by a +//! `MethodRef` and checked by `admit`. Its steps name polyfill words by +//! position (mask-risc's keyed count twice, then a checked i64 multiply); +//! no word has a classid. +//! 3. **Execution** has no production home. The lab register +//! (`dr-math-binding-probe::bind_dr64`) refuses a Bag method //! (`Refusal::BagNeedsMultiplicity`): that is the failing half. //! -//! No DT64 bind store exists yet (D-DT64-6 is PROPOSED). `Binding` below is a -//! test-local value showing what XLOG would hand one; it is not a store, it -//! persists nothing and it copies no business data (identities and field -//! positions only). +//! The classids are lab values (`0x7F01_0001` for the XLOG lens, +//! `0x7F0A_0001` for the polyfill lens). Production classids are OGAR mints. -use dr_math_binding_probe::{ - Axis, AxisKind, Focus, FrameRevisions, FrameStore, Guards, MethodSignature, Operand, - Population, Refusal, Scope, bind_dr64, -}; +use dr_math_binding_probe as lab; use lance_graph_contract::canonical_node::{NodeGuid, TailVariant}; -use lance_graph_contract::class_view::{ClassId, ClassView, FieldMask, WideFieldMask}; +use lance_graph_contract::class_view::{ClassId, ClassView, WideFieldMask}; use lance_graph_contract::ontology::{DisplayTemplate, FieldRef}; +use lance_graph_contract::set_rules::{Population, Truth}; +use lance_graph_dt64::dr::{Focus, FrameRefusal, FrameRevisions, FrameStore, Guards, Scope}; +use lance_graph_dt64::{ + AlgebraRef, Arg, BindStore, ClassSchema, Effect, Extent, MethodBinding, MethodId, MethodRef, + Refusal, RouteVersion, SchemaRev, SigRev, ValueDemand, view_class, +}; use xlog_logic::parse_program; use xlog_relcomp_probe::{AddressMap, SLOTS, facts, register}; @@ -37,49 +41,64 @@ q(Y) :- r(X, Y), s(Y, Z). ?- q(Y). "; -/// Lab stand-ins. A real binding takes these from the OGAR codebook through -/// hotplug; they are not minted concepts. -const LAB_CLASS: ClassId = 0x7F01; -const LAB_CLASSID: u32 = (LAB_CLASS as u32) << 16; -const COUNT_METHOD: u16 = 1; +/// The XLOG lens (lab). Its concept half is the `ClassView` class. +const XLOG: u32 = 0x7F01_0001; +const XLOG_VIEW: ClassId = 0x7F01; +/// The polyfill vocabulary is one lens (lab); its words are its fields. +const POLYFILL: u32 = 0x7F0A_0001; +const POLYFILL_VIEW: ClassId = 0x7F0A; -const X: Axis = Axis { - class: 10, - kind: AxisKind::Identity, +/// The lens's fields: the relations R and S, then the two keyed counts and +/// their product. +const XLOG_FIELDS: [&str; 5] = ["r", "s", "cnt_r", "cnt_s", "count"]; +/// The polyfill words used here, by position. mask-risc folds (the keyed +/// count); `ndarray::simd` computes what mask-risc lacks (the i64 multiply). +const WORDS: [&str; 2] = [ + "lance_graph_mask_risc::Terminal::GroupReduce(GroupKey::Lane, GroupFold::Count)", + "ndarray::simd::checked_binary_i64(I64Op::Mul)", +]; +const ARITY: [u8; 2] = [2, 2]; +const GROUP_COUNT: AlgebraRef = AlgebraRef { + lens: POLYFILL, + word: 0, }; -const Y: Axis = Axis { - class: 11, - kind: AxisKind::Property, -}; -const Z: Axis = Axis { - class: 12, - kind: AxisKind::Community, +const MUL_I64: AlgebraRef = AlgebraRef { + lens: POLYFILL, + word: 1, }; +const COUNT: MethodId = MethodId(1); + const FOCUS: Focus = Focus { revision: 1 }; const SCOPE: Scope = Scope { id: 2 }; -/// The class's property mappings: field 0 is R, field 1 is S. -struct LabClassView { - fields: Vec, +/// Both lenses' fields, labelled late. Nothing here is stored in the bind +/// store. +struct Views { + xlog: Vec, + polyfill: Vec, } -impl LabClassView { +impl Views { fn new() -> Self { + let refs = |labels: &[&str], ns: &str| { + labels + .iter() + .map(|l| FieldRef::new(format!("lab:{ns}/{l}"), (*l).to_string())) + .collect() + }; Self { - fields: vec![ - FieldRef::new("lab:xlog/r", "r"), - FieldRef::new("lab:xlog/s", "s"), - ], + xlog: refs(&XLOG_FIELDS, "xlog"), + polyfill: refs(&WORDS, "polyfill"), } } } -impl ClassView for LabClassView { +impl ClassView for Views { fn fields(&self, class: ClassId) -> &[FieldRef] { - if class == LAB_CLASS { - &self.fields - } else { - &[] + match class { + XLOG_VIEW => &self.xlog, + POLYFILL_VIEW => &self.polyfill, + _ => &[], } } fn template(&self, _class: ClassId) -> DisplayTemplate { @@ -90,122 +109,198 @@ impl ClassView for LabClassView { } } -/// A lab id on one axis, as a `NodeGuid` reference (V3 tail; the axis class -/// sits in `family`, the lab id in `identity`). -fn guid(axis: Axis, id: u32) -> NodeGuid { - NodeGuid::mint_for( - TailVariant::V3, - LAB_CLASSID, - 0, - 0, - 0, - 0, - u32::from(axis.class), - id, - ) +/// `q(Y) :- r(X,Y), s(Y,Z)` counted per Y: R keyed by its y column (1), S by +/// its y column (0), then the product. Bag: one count per derivation. +fn xlog_count() -> MethodBinding { + MethodBinding { + id: COUNT, + sig_rev: SigRev(1), + name: "count q(Y)".to_string(), + on: WideFieldMask::from_positions(&[0, 1]), + when: None, + reads: WideFieldMask::from_positions(&[0, 1]), + effects: vec![ + Effect { + target: 2, + algebra: GROUP_COUNT, + args: vec![Arg::Field(0), Arg::Const(1)], + }, + Effect { + target: 3, + algebra: GROUP_COUNT, + args: vec![Arg::Field(1), Arg::Const(0)], + }, + Effect { + target: 4, + algebra: MUL_I64, + args: vec![Arg::Field(2), Arg::Field(3)], + }, + ], + population: Population::Bag, + truth: Truth::TwoValued, + demand: ValueDemand::Values, + } +} + +fn bind_store() -> BindStore { + let mut s = BindStore::new(); + s.declare_algebra(POLYFILL, SchemaRev(1), Extent::DR64_CUBE, ARITY.to_vec()) + .unwrap(); + let schema = ClassSchema { + schema_rev: SchemaRev(1), + field_count: XLOG_FIELDS.len(), + extent: Extent::DR64_CUBE, + }; + s.register(XLOG, schema, vec![xlog_count()]).unwrap(); + s +} + +/// A lab id on the answer axis, as a `NodeGuid` reference (V3 tail). +fn guid(id: u32) -> NodeGuid { + NodeGuid::mint_for(TailVariant::V3, XLOG, 0, 0, 0, 0, 11, id) +} + +type Tuples = Vec<(u32, u32)>; + +fn fixture() -> (Tuples, Tuples) { + let p = parse_program(MULT).unwrap(); + (facts(&p, "r").unwrap(), facts(&p, "s").unwrap()) +} + +/// The answer axis: every Y the join can reach, in id order. +fn answer_ids(r: &[(u32, u32)], s: &[(u32, u32)]) -> Vec { + let mut ys: Vec = r + .iter() + .map(|t| t.1) + .chain(s.iter().map(|t| t.0)) + .map(|y| u16::try_from(y).unwrap()) + .collect(); + ys.sort_unstable(); + ys.dedup(); + ys } -/// DR's output: local slots under a sealed frame. +/// DR's output: the answer positions routed to the XLOG lens at one route +/// version, under a sealed frame. struct Routed { - xs: AddressMap, - ys: AddressMap, - zs: AddressMap, + positions: Vec, + version: RouteVersion, guards: Guards, } -fn route(store: &FrameStore, r: &[(u32, u32)], s: &[(u32, u32)]) -> Routed { +fn route(store: &mut BindStore, frames: &FrameStore, positions: Vec) -> Routed { + let mut version = store.current_version(); + for &p in &positions { + version = store.route(p, XLOG).unwrap(); + } Routed { - xs: AddressMap::new(X, r.iter().map(|t| t.0)).unwrap(), - ys: AddressMap::new(Y, r.iter().map(|t| t.1).chain(s.iter().map(|t| t.0))).unwrap(), - zs: AddressMap::new(Z, s.iter().map(|t| t.1)).unwrap(), + positions, + version, guards: Guards { - frame: store.seal(), + frame: frames.seal(), focus: FOCUS, scope: SCOPE, }, } } -/// What DT64 records for the operation: identities by reference, the -/// selected property mappings, the method by reference, and the expected -/// values. It is only built against a current route (`bind` checks the frame). -struct Binding { +/// What DT64 hands back for the operation: identities by reference, the +/// selected property mappings, the admitted method, the expected values. +struct Binding<'s> { answer_identities: Vec, - class: ClassId, selected: WideFieldMask, - method: MethodSignature, + method: &'s MethodBinding, expected: Vec<(NodeGuid, u64)>, } #[derive(Debug, PartialEq, Eq)] enum BindRefusal { - /// DT64 never binds against a route that is no longer current. - Route(Refusal), + /// DT64 never binds against a frame that is no longer current. + Frame(FrameRefusal), + /// The bind store refused a lookup. + Store(Refusal), } -fn bind(store: &FrameStore, routed: &Routed) -> Result { - store +fn bind<'s>( + store: &'s BindStore, + frames: &FrameStore, + views: &Views, + routed: &Routed, +) -> Result, BindRefusal> { + frames .check(&routed.guards.frame) - .map_err(BindRefusal::Route)?; - let id_of = |slot: usize| routed.ys.ids(1u64 << slot)[0]; + .map_err(BindRefusal::Frame)?; + for &p in &routed.positions { + let r = store + .resolve_at(views, routed.version, p, SchemaRev(1)) + .map_err(BindRefusal::Store)?; + assert_eq!(r.class, XLOG); + } + let method = store + .admit(&MethodRef::of(XLOG, &xlog_count())) + .map_err(BindRefusal::Store)?; Ok(Binding { - answer_identities: (0..routed.ys.len()).map(|i| guid(Y, id_of(i))).collect(), - class: LAB_CLASS, + answer_identities: routed + .positions + .iter() + .map(|&p| guid(u32::from(p))) + .collect(), selected: WideFieldMask::from_positions(&[0, 1]), - method: MethodSignature { - population: Population::Bag, - ..MethodSignature::presence(LAB_CLASS, COUNT_METHOD, Y, Y) - }, + method, // Per-Y derivation counts; see tests/litho.rs for the two oracles. - expected: vec![(guid(Y, 2), 4), (guid(Y, 3), 1), (guid(Y, 7), 1)], + expected: vec![(guid(2), 4), (guid(3), 1), (guid(7), 1)], }) } -type Tuples = Vec<(u32, u32)>; - -fn fixture() -> (Tuples, Tuples) { - let p = parse_program(MULT).unwrap(); - (facts(&p, "r").unwrap(), facts(&p, "s").unwrap()) +fn revisions(relation_map: u64) -> FrameRevisions { + FrameRevisions { + source: 1, + relation_map, + coordinates: 1, + } } #[test] -fn the_binding_names_identities_property_mappings_and_a_method() { +fn the_binding_resolves_through_the_bind_store() { let (r, s) = fixture(); - let store = FrameStore::new(FrameRevisions { - source: 1, - relation_map: 1, - coordinates: 1, - }); - let routed = route(&store, &r, &s); - let b = bind(&store, &routed).unwrap(); + let views = Views::new(); + let mut store = bind_store(); + let frames = FrameStore::new(revisions(1)); + let routed = route(&mut store, &frames, answer_ids(&r, &s)); + let b = bind(&store, &frames, &views, &routed).unwrap(); - // Identities: references that decode back to the lab ids, in slot order. + // Identities: references that decode back to the lab ids, in id order. let ids: Vec<(u32, u16)> = b .answer_identities .iter() .map(|g| (g.classid(), g.identity_v2())) .collect(); - assert_eq!( - ids, - vec![(LAB_CLASSID, 2), (LAB_CLASSID, 3), (LAB_CLASSID, 7)] - ); + assert_eq!(ids, vec![(XLOG, 2), (XLOG, 3), (XLOG, 7)]); - // Property mappings: R and S, resolved through the ClassView, not stored. - let view = LabClassView::new(); - assert!(b.selected.has(0) && b.selected.has(1) && !b.selected.has(2)); - let rows: Vec<&str> = view - .render_rows(b.class, FieldMask::from_positions(&[0, 1])) - .into_iter() - .map(|r| r.label) + // Property mappings: R and S, labelled by the lens's ClassView. + let resolved = store.resolve(&views, XLOG, SchemaRev(1)).unwrap(); + let rows: Vec<(u8, &str)> = resolved + .field_view(&b.selected) + .map(|r| (r.position, r.label)) .collect(); - assert_eq!(rows, vec!["r", "s"]); + assert_eq!(rows, vec![(0, "r"), (1, "s")]); + assert_eq!(view_class(XLOG), XLOG_VIEW); - // Method: a reference, Bag population; nothing about how it counts. + // Method: admitted by reference; Bag; two words of the polyfill lens. + assert_eq!(b.method.population, Population::Bag); assert_eq!( - (b.method.owner_class, b.method.method_ordinal), - (LAB_CLASS, COUNT_METHOD) + store.words_used(XLOG, POLYFILL), + Ok(WideFieldMask::from_positions(&[0, 1])) ); - assert_eq!(b.method.population, Population::Bag); + assert_eq!(store.users_of(GROUP_COUNT), vec![(XLOG, COUNT)]); + let words = store + .resolve_algebra(&views, POLYFILL, SchemaRev(1)) + .unwrap(); + let labels: Vec<&str> = words + .field_view(&WideFieldMask::from_positions(&[0, 1])) + .map(|r| r.label) + .collect(); + assert_eq!(labels, WORDS.to_vec()); // Expected values, keyed by identity. assert_eq!(b.expected.len(), b.answer_identities.len()); @@ -218,50 +313,108 @@ fn the_binding_names_identities_property_mappings_and_a_method() { } #[test] -fn dt64_does_not_bind_against_a_stale_route() { +fn dt64_does_not_bind_against_a_stale_frame() { let (r, s) = fixture(); - let mut store = FrameStore::new(FrameRevisions { - source: 1, - relation_map: 1, - coordinates: 1, - }); - let routed = route(&store, &r, &s); - store.advance(FrameRevisions { - source: 1, - relation_map: 2, - coordinates: 1, - }); + let views = Views::new(); + let mut store = bind_store(); + let mut frames = FrameStore::new(revisions(1)); + let routed = route(&mut store, &frames, answer_ids(&r, &s)); + frames.advance(revisions(2)); assert_eq!( - bind(&store, &routed).err(), - Some(BindRefusal::Route(Refusal::MapChanged)) + bind(&store, &frames, &views, &routed).err(), + Some(BindRefusal::Frame(FrameRefusal::MapChanged)) ); } -/// The failing half: executing the bound Bag method is refused today, so the -/// expected counts cannot be produced from the register. +/// Routing is versioned; behaviour is not. A retracted answer position is +/// unrouted from the new version on, and the old version still resolves. #[test] -fn executing_the_bound_bag_method_is_refused_today() { +fn a_retracted_position_is_unrouted_and_the_old_version_replays() { let (r, s) = fixture(); - let store = FrameStore::new(FrameRevisions { + let views = Views::new(); + let mut store = bind_store(); + let frames = FrameStore::new(revisions(1)); + let mut routed = route(&mut store, &frames, answer_ids(&r, &s)); + let old = routed.version; + let new = store.retract(7); + routed.version = new; + assert_eq!( + bind(&store, &frames, &views, &routed).err(), + Some(BindRefusal::Store(Refusal::Unrouted { + position: 7, + version: new + })) + ); + assert_eq!( + store + .resolve_at(&views, old, 7, SchemaRev(1)) + .map(|r| r.class), + Ok(XLOG) + ); +} + +/// The failing half: production has no executor, and the lab register +/// refuses the admitted Bag method, so the expected counts cannot be produced +/// from the register yet. +#[test] +fn executing_the_admitted_bag_method_is_refused_in_the_lab_register() { + let (r, s) = fixture(); + let views = Views::new(); + let mut store = bind_store(); + let frames = FrameStore::new(revisions(1)); + let routed = route(&mut store, &frames, answer_ids(&r, &s)); + let b = bind(&store, &frames, &views, &routed).unwrap(); + + // The lab register has its own frame witness and signature type; the + // admitted binding's population is carried across, never re-chosen. + let population = match b.method.population { + Population::Bag => lab::Population::Bag, + Population::Distinct => lab::Population::Distinct, + }; + let (x, y, z) = ( + lab::Axis { + class: 10, + kind: lab::AxisKind::Identity, + }, + lab::Axis { + class: 11, + kind: lab::AxisKind::Property, + }, + lab::Axis { + class: 12, + kind: lab::AxisKind::Community, + }, + ); + let lab_frames = lab::FrameStore::new(lab::FrameRevisions { source: 1, relation_map: 1, coordinates: 1, }); - let routed = route(&store, &r, &s); - let b = bind(&store, &routed).unwrap(); - let rr = register(&r, &routed.xs, &routed.ys).unwrap(); - let _ss = register(&s, &routed.ys, &routed.zs).unwrap(); - let e = bind_dr64( - &store, - b.method, - b.method, - &routed.guards, - (FOCUS, SCOPE), - Operand { + let xs = AddressMap::new(x, r.iter().map(|t| t.0)).unwrap(); + let ys = AddressMap::new(y, r.iter().map(|t| t.1).chain(s.iter().map(|t| t.0))).unwrap(); + let zs = AddressMap::new(z, s.iter().map(|t| t.1)).unwrap(); + let rr = register(&r, &xs, &ys).unwrap(); + let _ss = register(&s, &ys, &zs).unwrap(); + let sig = lab::MethodSignature { + population, + ..lab::MethodSignature::presence(XLOG_VIEW, COUNT.0 as u16, y, y) + }; + let guards = lab::Guards { + frame: lab_frames.seal(), + focus: lab::Focus { revision: 1 }, + scope: lab::Scope { id: 2 }, + }; + let e = lab::bind_dr64( + &lab_frames, + sig, + sig, + &guards, + (guards.focus, guards.scope), + lab::Operand { rows: &rr, validity: None, }, ); - assert_eq!(e.err(), Some(Refusal::BagNeedsMultiplicity)); + assert_eq!(e.err(), Some(lab::Refusal::BagNeedsMultiplicity)); assert_eq!(rr.len(), SLOTS); } diff --git a/crates/xlog-relcomp-probe/tests/epi5_reading.rs b/crates/xlog-relcomp-probe/tests/epi5_reading.rs new file mode 100644 index 000000000..a8db12d01 --- /dev/null +++ b/crates/xlog-relcomp-probe/tests/epi5_reading.rs @@ -0,0 +1,109 @@ +//! How an XLOG result reads in `CausalEdge64` bits 59..63, the +//! `EpistemicState5` field (`Topology2 × Certification3`). +//! +//! The rule this pins, for any XLOG result that is ever written as a CE64 +//! edge: +//! +//! - A base fact (`r(1, 2)`) is `Direct × Open`. +//! - A composed edge (`r ; s`, X to Z through Y) is `IndirectKnown × Open`: +//! the register keeps the intermediate Y, so it is known. +//! - A join certifies nothing (`Open`): sharing a key is not an association +//! in a declared population, let alone a cause. +//! - A bag count is a value in the binding (`tests/dto_binding.rs`), never a +//! certification and never frequency/confidence. +//! +//! `raw5 == 0` is `Direct × Open`, not "unknown", so a composed edge left +//! unstamped silently claims a direct relation. The code is also opaque +//! without a declaration: the XLOG lens has none in production (an OGAR mint +//! decision), and projection refuses until it has one. + +use causal_edge::edge::CausalEdge64; +use lance_graph_contract::band_reading::EdgeProvenance; +use lance_graph_contract::class_view::ClassId; +use lance_graph_contract::epistemic_state5::{ + Certification3, Epi5Declarations, Epi5Gen, Epi5ReadError, Epi5Reading, EpistemicState5, + Topology2, fact, +}; +use lance_graph_contract::rail_geometry::RailAxis; + +/// The XLOG lens's view class (lab, as in `tests/dto_binding.rs`). +const XLOG_VIEW: ClassId = 0x7F01; +/// Lab placeholder: which rail an XLOG edge's reading is declared on is part +/// of the OGAR declaration this probe does not make. +const RAIL: RailAxis = RailAxis::Taxonomy; + +const BASE: EpistemicState5 = + EpistemicState5::new(Epi5Gen::V1, Topology2::Direct, Certification3::Open); +const COMPOSED: EpistemicState5 = + EpistemicState5::new(Epi5Gen::V1, Topology2::IndirectKnown, Certification3::Open); + +/// S=0x56, P=0x34, O=0x12, frequency 0xCD, confidence 0xAB; upper half zero. +const EDGE: CausalEdge64 = CausalEdge64(0x0000_00AB_CD12_3456); +const LOW_40: u64 = (1 << 40) - 1; + +fn declared() -> Epi5Declarations { + let mut d = Epi5Declarations::new(); + d.declare(XLOG_VIEW, RAIL, Epi5Reading::default()); + d +} + +fn read(d: &Epi5Declarations, e: CausalEdge64) -> Result { + d.project_state5( + XLOG_VIEW, + RAIL, + Epi5Gen::V1, + e.epistemic_raw5(), + EdgeProvenance::V2Stamped, + ) +} + +#[test] +fn an_xlog_edge_has_no_reading_until_its_lens_is_declared() { + let e = EDGE.with_epistemic_raw5(COMPOSED.raw()); + assert_eq!( + read(&Epi5Declarations::new(), e), + Err(Epi5ReadError::UndeclaredClass(XLOG_VIEW)) + ); +} + +#[test] +fn base_facts_and_composed_edges_carry_different_topologies() { + let d = declared(); + let base = EDGE.with_epistemic_raw5(BASE.raw()); + let composed = EDGE.with_epistemic_raw5(COMPOSED.raw()); + assert_eq!((BASE.raw(), COMPOSED.raw()), (0, 1)); + assert_eq!(read(&d, base), Ok(BASE)); + assert_eq!(read(&d, composed), Ok(COMPOSED)); + assert!(COMPOSED.asserts(fact::INTERMEDIATE_KNOWN)); + assert!( + !COMPOSED.asserts(fact::ASSOCIATED), + "a join certifies nothing" + ); + // Only bits 59..63 moved: palette indices and F/C are untouched. + assert_eq!(composed.0 & LOW_40, EDGE.0 & LOW_40); + assert_eq!(composed.0 & !(0b1_1111 << 59), EDGE.0); +} + +/// The pitfall: a composed edge written without its state reads as a direct +/// one. +#[test] +fn an_unstamped_composed_edge_reads_as_direct() { + let unstamped = read(&declared(), EDGE).unwrap(); + assert_eq!(unstamped, BASE); + assert_ne!(unstamped, COMPOSED); +} + +#[test] +fn legacy_provenance_is_refused() { + let e = EDGE.with_epistemic_raw5(COMPOSED.raw()); + assert_eq!( + declared().project_state5( + XLOG_VIEW, + RAIL, + Epi5Gen::V1, + e.epistemic_raw5(), + EdgeProvenance::V1Legacy, + ), + Err(Epi5ReadError::UnknownProvenance(EdgeProvenance::V1Legacy)) + ); +} diff --git a/crates/xlog-relcomp-probe/tests/gate_b.rs b/crates/xlog-relcomp-probe/tests/gate_b.rs index 097b4d87d..f39b225fe 100644 --- a/crates/xlog-relcomp-probe/tests/gate_b.rs +++ b/crates/xlog-relcomp-probe/tests/gate_b.rs @@ -16,10 +16,12 @@ use dr_math_binding_probe::relcomp::ComposeRefusal; use dr_math_binding_probe::{ Axis, AxisKind, Focus, FrameRevisions, FrameStore, Guards, MethodSignature, Refusal, Scope, }; +use lance_graph_dt64::dr::{LocalMap, MapRefusal, Width}; use xlog_core::RelId; use xlog_ir::{CompareOp, ConstValue, Expr, JoinType, ProjectExpr, RirNode}; use xlog_logic::ast::Program; use xlog_logic::{Compiler, MagicSetStatus, parse_program, rewrite_magic_sets}; +use xlog_relcomp_probe::native::{self, NativeRefusal}; use xlog_relcomp_probe::{ AddressMap, ChainQuery, DomainRefusal, EvalRefusal, LowerRefusal, SLOTS, admit, admit_program, facts, propagate, register, @@ -265,9 +267,29 @@ fn answer(p: &Prepared) -> Result<(BTreeSet, xlog_relcomp_probe::Propagatio z, &p.ys, )?; + // The native path is the execution path; the register is its + // differential. Every mask must agree, not only the answer. + let mid = local(&p.ys); + let n = native::support(&p.q, &p.r, &p.s, &mid).expect("native support"); + let ids = |m: &[u64]| -> Vec { + native::ids(&mid, m) + .into_iter() + .map(|g| u32::try_from(g).unwrap()) + .collect() + }; + assert_eq!(ids(&n.forward), p.ys.ids(prop.forward), "forward"); + assert_eq!(ids(&n.backward), p.ys.ids(prop.backward), "backward"); + assert_eq!(ids(&n.answer), p.ys.ids(prop.answer), "answer"); Ok((p.ys.ids(prop.answer).into_iter().collect(), prop)) } +/// DR's local map for the Y axis, in the same ascending order the register's +/// `AddressMap` assigns. +fn local(ys: &AddressMap) -> LocalMap { + let globals: Vec = ys.ids(ys.valid()).into_iter().map(u64::from).collect(); + LocalMap::prepare(Width::W256, &globals).unwrap() +} + #[test] fn xlog_compiles_the_fixture_to_the_admitted_shape() { let c = compile(FIXTURE); @@ -628,3 +650,61 @@ fn domain_boundary_63_64_65_and_tail_bits() { assert_eq!(p.forward, 1); assert_eq!(p.backward, 1); } + +// ── the native path past one 64-slot tile ───────────────────────────────── + +/// The register holds one 64-slot tile; the native path answers over DR's +/// 256-wide map with the same functions, and still agrees with scalar +/// enumeration. +#[test] +fn the_native_path_answers_past_one_tile() { + let r: Vec<(u32, u32)> = (0..200).map(|y| (y % 3, y * 5)).collect(); + let s: Vec<(u32, u32)> = (0..200) + .filter(|y| y % 7 != 0) + .map(|y| (y * 5, y % 4)) + .collect(); + let ys: BTreeSet = r.iter().map(|t| t.1).chain(s.iter().map(|t| t.0)).collect(); + assert_eq!( + AddressMap::new(Y, ys.iter().copied()), + Err(DomainRefusal::Overflow { needed: 200 }) + ); + let globals: Vec = ys.iter().map(|&y| u64::from(y)).collect(); + let mid = LocalMap::prepare(Width::W256, &globals).unwrap(); + for (xb, zb) in [ + (None, None), + (Some(1), None), + (None, Some(2)), + (Some(2), Some(3)), + ] { + let q = ChainQuery { + left: RelId(0), + left_bound: xb, + right: RelId(1), + right_bound: zb, + }; + let got: BTreeSet = + native::ids(&mid, &native::support(&q, &r, &s, &mid).unwrap().answer) + .into_iter() + .map(|g| u32::try_from(g).unwrap()) + .collect(); + let want = scalar(&r, &s, xb, zb); + assert!(!want.is_empty(), "the case selects something"); + assert_eq!(got, want, "bounds {xb:?} {zb:?}"); + } +} + +/// A Y outside DR's map is refused, never dropped. +#[test] +fn the_native_path_refuses_an_unmapped_y() { + let mid = LocalMap::prepare(Width::W64, &[2, 3]).unwrap(); + let q = ChainQuery { + left: RelId(0), + left_bound: None, + right: RelId(1), + right_bound: None, + }; + assert_eq!( + native::support(&q, &[(1, 2), (1, 9)], &[(2, 4)], &mid).err(), + Some(NativeRefusal::Domain(MapRefusal::Unmapped)) + ); +} diff --git a/crates/xlog-relcomp-probe/tests/litho.rs b/crates/xlog-relcomp-probe/tests/litho.rs index bd6062e5c..701cf3137 100644 --- a/crates/xlog-relcomp-probe/tests/litho.rs +++ b/crates/xlog-relcomp-probe/tests/litho.rs @@ -20,10 +20,12 @@ use dr_math_binding_probe::{ Axis, AxisKind, Focus, FrameRevisions, FrameStore, Guards, MethodSignature, Operand, Population, Refusal, Scope, bind_dr64, }; +use lance_graph_dt64::dr::{LocalMap, Width}; use xlog_core::{AggOp, RelId}; use xlog_ir::{CompareOp, ConstValue, Expr, JoinType, ProjectExpr, RirNode}; use xlog_logic::ast::Program; use xlog_logic::{Compiler, parse_program}; +use xlog_relcomp_probe::native::{admit_count, bag_count, nonzero}; use xlog_relcomp_probe::{ AddressMap, EvalRefusal, LowerRefusal, SLOTS, admit, admit_program, facts, propagate, register, }; @@ -518,3 +520,112 @@ fn the_register_path_carries_bag_counts() { .collect(); assert_eq!(read_out, expected()); } + +// ── Part 3: the bag count on the ndarray::simd polyfill ──────────────────── + +fn y_map(r: &[(u32, u32)], s: &[(u32, u32)]) -> LocalMap { + let ys: BTreeSet = r + .iter() + .map(|t| u64::from(t.1)) + .chain(s.iter().map(|t| u64::from(t.0))) + .collect(); + LocalMap::prepare(Width::W64, &ys.into_iter().collect::>()).unwrap() +} + +/// XLOG's own `count` rule is admitted on the native path and answered with +/// two keyed counts and a checked multiply; both oracles agree. +#[test] +fn the_native_path_carries_bag_counts() { + let c = compile(MULT); + let r = facts(&c.program, "r").unwrap(); + let s = facts(&c.program, "s").unwrap(); + let q = admit_count(rule(&c, "qc")).expect("the count plan is admitted"); + assert_eq!(q, admit(rule(&c, "q")).unwrap(), "same body as q"); + let mid = y_map(&r, &s); + let got: BTreeMap = nonzero(&mid, &bag_count(&q, &r, &s, &mid).unwrap()) + .into_iter() + .map(|(y, n)| (u32::try_from(y).unwrap(), u64::try_from(n).unwrap())) + .collect(); + assert_eq!(got, expected()); + assert_eq!(got, scalar_bag(&r, &s)); + assert_eq!( + got, + counts(walk(rule(&c, "qc"), &edb(&c), false), 0, Some(1)) + ); +} + +/// A bound end restricts both factors before the multiply. +#[test] +fn the_native_bag_count_respects_a_bound_end() { + let src = MULT.replace( + "qc(Y, count(X)) :- r(X, Y), s(Y, Z).", + "qc(Y, count(X)) :- r(X, Y), s(Y, 4).", + ); + let c = compile(&src); + let r = facts(&c.program, "r").unwrap(); + let s = facts(&c.program, "s").unwrap(); + let q = admit_count(rule(&c, "qc")).unwrap(); + assert_eq!(q.right_bound, Some(4)); + let mid = y_map(&r, &s); + let got: BTreeMap = nonzero(&mid, &bag_count(&q, &r, &s, &mid).unwrap()) + .into_iter() + .map(|(y, n)| (u32::try_from(y).unwrap(), u64::try_from(n).unwrap())) + .collect(); + // Y = 2: two X times one Z (4); Y = 3: one times one; Y = 7 has no Z = 4. + assert_eq!(got, BTreeMap::from([(2, 2), (3, 1)])); + assert_eq!( + got, + counts(walk(rule(&c, "qc"), &edb(&c), false), 0, Some(1)) + ); +} + +/// Only `Count` keyed by the answer column is admitted. +#[test] +fn the_native_path_refuses_other_aggregates() { + let c = compile(MULT); + let RirNode::Project { input, columns } = rule(&c, "qc").clone() else { + panic!("expected Project") + }; + let RirNode::GroupBy { + input: inner, + key_cols, + .. + } = *input + else { + panic!("expected GroupBy") + }; + let sum = RirNode::Project { + input: Box::new(RirNode::GroupBy { + input: inner, + key_cols, + aggs: vec![(1, AggOp::Sum)], + }), + columns, + }; + assert_eq!(admit_count(&sum), Err(LowerRefusal::UnsupportedAggregate)); + assert_eq!( + admit_count(rule(&c, "q")), + Err(LowerRefusal::UnsupportedNode("ChainJoin")), + "the support rule is not a count" + ); +} + +/// A repeated base fact is one tuple, as in XLOG's fact loader: the count +/// does not change when `r(1, 2)` and `s(2, 4)` are declared twice. +#[test] +fn a_repeated_base_fact_is_not_a_second_witness() { + let src = MULT + .replace("r(1, 2).", "r(1, 2). r(1, 2).") + .replace("s(2, 4).", "s(2, 4). s(2, 4)."); + let c = compile(&src); + let r = facts(&c.program, "r").unwrap(); + let s = facts(&c.program, "s").unwrap(); + assert_eq!((r.len(), s.len()), (4, 4), "duplicates collapse"); + let q = admit_count(rule(&c, "qc")).unwrap(); + let mid = y_map(&r, &s); + let got: BTreeMap = nonzero(&mid, &bag_count(&q, &r, &s, &mid).unwrap()) + .into_iter() + .map(|(y, n)| (u32::try_from(y).unwrap(), u64::try_from(n).unwrap())) + .collect(); + assert_eq!(got, expected()); +}