diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 920a7808b..73319cc23 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -38,7 +38,7 @@ Plan: `.claude/plans/d-dynbag-holographic-bags-v1.md`. Plan only. | D-id | Deliverable | Status | Gate | |---|---|---|---| | D-DYNBAG-0 | PR #1884 layering fix: XLOG intake as lanes, support + bag count via Quack → mask-risc, fold oracles, cost columns | Shipped (#1906): gates 1–7 (plan §17.7); cost columns not done | operator go-ahead (given 2026-11-01) | -| D-DYNBAG-1 | ℕ-compose probe: `matmul_i8_to_i32` over two DR64² tiles for `p(X, Z)`; B projection = `relcomp::compose` | Queued | D-DYNBAG-0 | +| D-DYNBAG-1 | ℕ-compose probe: `matmul_i8_to_i32` over two DR64² tiles for `p(X, Z)`; B projection = `relcomp::compose` | In PR: exact vs both oracles (300 cases), projection = compose; MEASURED ~21 µs/tile on the AVX-512 VNNI tier (no AMX) vs 50 ns–2.4 µs for B (plan §7.6) | D-DYNBAG-0 | | D-DYNBAG-2 | semiring parameter K (B, ℕ) in relalg `Rel` + Quack surface; typed refusal per (op, K) | Queued | D-DYNBAG-1 | | D-DYNBAG-3 | missing ℕ words backend-first (i64 masked select, min, truncated minus, compare-to-mask, masked sum; maybe bit-packed product) | Queued | D-DYNBAG-2 | | D-DYNBAG-4 | DR{dynamic}: width from survivor count, multi-tile map, cross-tile fold/compose | Queued | DT64 `dr` owner | diff --git a/.claude/board/entries/2026-11-01-dynbag-1-n-compose.md b/.claude/board/entries/2026-11-01-dynbag-1-n-compose.md new file mode 100644 index 000000000..86de9b8d0 --- /dev/null +++ b/.claude/board/entries/2026-11-01-dynbag-1-n-compose.md @@ -0,0 +1,23 @@ +# D-DYNBAG-1: ℕ-compose for `p(X, Z)`, exact; `matmul_i8_to_i32` is the wrong carrier on this host as called + +**Date 2026-11-01 · ID D-DYNBAG-1 · Status IN PR · Plan `.claude/plans/d-dynbag-holographic-bags-v1.md` §7.6 · Crate `crates/xlog-relcomp-probe` (`src/ncompose.rs`, `tests/litho.rs` part 4, `examples/ncompose_cost.rs`)** + +- `p(X, Z) :- r(X, Y), s(Y, Z)` is answered as `matmul_i8_to_i32` over two DR64² 0/1 tiles. It equals both oracles (scalar witnesses per (x, z); XLOG's RIR bag walk) on the fixture and on 300 random cases. Its `> 0` reading equals `relcomp::compose`. +- Disable runs red: operands swapped (caught by the differential only); B read before counting. +- MEASURED, AVX-512 VNNI tier (Xeon model 0x55, no AMX; the cost example prints the tier): + + | case | `compose` (B) | `matmul_i8_to_i32` (ℕ) | + |---|---|---| + | sparse | 50 ns | about 21 µs per tile | + | dense | 2.4 µs | about 21 µs per tile | + +HYPOTHESIS: the 21 µs is per-call pack/alloc overhead, not arithmetic. + +A first version said "no VNNI, scalar tier" from a mis-grepped flag (`avx512_vnni`); this entry is corrected. + +**OPEN:** +- A bit-packed i1 × i1 → i32 word (popcount of R[x] ∧ Sᵀ[z]) in `ndarray::simd`, the D-DYNBAG-3 candidate. +- The AMX tier: not on this host; never certify it from a shared VM (ndarray AMX_GOTCHAS 14). +- A non-allocating `matmul_i8_to_i32` entry point. +- Bounds and chains longer than two atoms (refused). +- Cross-tile compose (MISSING-7). diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index efac24535..71bf1c47c 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,12 +25,13 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -539 entries, 2026-08-06 .. 2026-11-01. +540 entries, 2026-08-06 .. 2026-11-01. | date | entry id | finding | file | |---|---|---|---| | 2026-11-01 | `D-DT64-7-5` | | [2026-11-01-maptable-versioned-local-map.md](2026-11-01-maptable-versioned-local-map.md) | | 2026-11-01 | `D-XLOG-NATIVE-4` | | [2026-11-01-lance-graph-xlog-crate.md](2026-11-01-lance-graph-xlog-crate.md) | +| 2026-11-01 | `D-DYNBAG-1` | | [2026-11-01-dynbag-1-n-compose.md](2026-11-01-dynbag-1-n-compose.md) | | 2026-11-01 | `D-DYNBAG-0` | | [2026-11-01-dynbag-0-xlog-through-quack-mask-risc.md](2026-11-01-dynbag-0-xlog-through-quack-mask-risc.md) | | 2026-10-31 | `x2-dt64-admission-policy` | | [2026-10-31-x2-dt64-admission-policy.md](2026-10-31-x2-dt64-admission-policy.md) | | 2026-10-31 | `r1-r2il-binding-reading` | | [2026-10-31-r1-r2il-binding-reading.md](2026-10-31-r1-r2il-binding-reading.md) | diff --git a/.claude/plans/d-dynbag-holographic-bags-v1.md b/.claude/plans/d-dynbag-holographic-bags-v1.md index d128c34a2..47812804e 100644 --- a/.claude/plans/d-dynbag-holographic-bags-v1.md +++ b/.claude/plans/d-dynbag-holographic-bags-v1.md @@ -612,6 +612,29 @@ Evidence: four read-only harvests at `d3c2a29`/`f3b2cfe`, each claim cited file: §17 stands. One addition: the binding that selects the executed program is DT64's `MethodBinding` (its `effects` name the polyfill words), and the selection mechanism follows Tract's precedent (resolve once, dispatch through the resolved entry, frame-checked per run) without copying Tract's registry. If X1 needs a type in DT64, D-DYNBAG-0 stops and asks rather than adding it in the probe. +### 7.6 Result (2026-11-01, branch `ccr-85acaff0-oszq40-dynbag1`) + +- **Equality: met.** `xlog-relcomp-probe` `ncompose::pair_counts` (two DR64² i8 tiles, `matmul_i8_to_i32`) equals scalar enumeration per (x, z) and a bag walk of XLOG's RIR for `p`, on the multiplicity fixture and on a 300-case differential (ids 0..6 per axis; ℕ ≠ B in over 100 cases). +- **Projection: met.** `count > 0` equals `relcomp::compose` on the DR64 register, as (x, z) id sets, on every case. +- **Admission.** `ncompose::admit_pair` takes the join in both orders XLOG's optimizer emits, and a `ChainJoin` only when its own relations match its fallback. Bounds are refused. `q(Y)` is still never admitted as `p(X, Z)`. +- **Disable runs: red.** Operands swapped: the differential fails; the hand fixture alone passes, so the differential is what catches it. Result read as B before counting: both tests fail. +- **Cost (MEASURED, `examples/ncompose_cost.rs`; host: Intel Xeon model 0x55 (Skylake-SP / Cascade Lake class), 4 vCPU VM, AVX-512 with VNNI, no AMX. The product ran on its AVX-512 VNNI `VPDPBUSD` tier, as the example now prints):** + + | case | `compose` (B) | `matmul_i8_to_i32` (ℕ, tiles prepared) | `pair_counts` (ℕ, from tuples) | + |---|---|---|---| + | sparse, 4 × 4 tuples | 50 ns | 20,849 ns | 21,420 ns | + | dense 64 × 64 | 2,402 ns | 20,829 ns | 158,637 ns | + +- **Answer to open decision 2 (on this host):** through `matmul_i8_to_i32`'s current signature, a 64² tile costs about 21 µs whatever the density. It is exact, but it is the wrong carrier for sparse tiles here: 400× slower than B on sparse data, 9× on dense. + - **HYPOTHESIS, not profiled:** the 21 µs is mostly per-call overhead (`pack_contig` of both inputs, the u8 shift copy, a fresh output `Vec`, the bias subtraction, the copy back; §7.4), not arithmetic. 64³ multiply-adds is about 4,096 `VPDPBUSD` instructions. + - **Untested:** the AMX tier, a non-allocating entry point, and a bit-packed i1 × i1 → i32 word (popcount of R[x] ∧ Sᵀ[z], which skips empty rows). Per §7.4 either lands in `ndarray::simd`, backend-first; the bit-packed word is the D-DYNBAG-3 candidate. + - Building the tiles from tuples (about 17 ns per tuple lookup, dense) also dominates end-to-end time. +- **Correction (same day):** the first version of this result said "no VNNI, scalar tier". That came from grepping `/proc/cpuinfo` for `avx512vnni`, which the kernel spells `avx512_vnni`. The numbers were unchanged on re-run. +- **AMX (ndarray `.claude/AMX_GOTCHAS.md`):** + - This host has no AMX: model 0x55 is pre-Sapphire Rapids, so `amx_available() == false` is genuine (Gotcha 4 does not apply). + - Any later AMX measurement must not be certified from a shared VM (Gotcha 14: tile state is silently corrupted under host contention). + - Its parity tests must also run under deliberate CPU contention, and must not count a skipped tile test as a pass (Gotcha 9). + --- ## 19. Polyfills are composed, not hardcoded (2026-10-31, operator decision before coding) diff --git a/crates/xlog-relcomp-probe/examples/ncompose_cost.rs b/crates/xlog-relcomp-probe/examples/ncompose_cost.rs new file mode 100644 index 000000000..6744fb7ef --- /dev/null +++ b/crates/xlog-relcomp-probe/examples/ncompose_cost.rs @@ -0,0 +1,125 @@ +//! D-DYNBAG-1 cost probe: ℕ-compose (`matmul_i8_to_i32` over DR64² tiles) +//! against the Boolean `relcomp::compose` on the DR64 register, same data. +//! Hot paths only, reported in ns per call (median of 201 runs). +//! +//! cargo run --release --example ncompose_cost --manifest-path crates/xlog-relcomp-probe/Cargo.toml +use std::hint::black_box; +use std::time::Instant; + +use dr_math_binding_probe::relcomp::{Work, compose}; +use dr_math_binding_probe::{Axis, AxisKind}; +use lance_graph_dt64::dr::{LocalMap, Width}; +use ndarray::Array2; +use ndarray::simd::{amx_available, matmul_i8_to_i32}; +use xlog_relcomp_probe::ncompose::{PairQuery, pair_counts}; +use xlog_relcomp_probe::{AddressMap, register}; + +const AX: [Axis; 3] = [ + Axis { + class: 10, + kind: AxisKind::Identity, + }, + Axis { + class: 11, + kind: AxisKind::Property, + }, + Axis { + class: 12, + kind: AxisKind::Community, + }, +]; + +type Tuples = Vec<(u32, u32)>; + +fn median_ns(mut f: impl FnMut()) -> u128 { + let mut t: Vec = (0..201) + .map(|_| { + let s = Instant::now(); + f(); + s.elapsed().as_nanos() + }) + .collect(); + t.sort_unstable(); + t[100] +} + +fn local(ids: impl Iterator) -> LocalMap { + let mut v: Vec = ids.map(u64::from).collect(); + v.sort_unstable(); + v.dedup(); + LocalMap::prepare(Width::W64, &v).unwrap() +} + +fn main() { + // The tier `matmul_i8_to_i32` dispatches to, in its own order: + // AMX, then AVX-512 VNNI, then AVX-VNNI, then the scalar reference. + let tier = if amx_available() { + "AMX TDPBUSD" + } else if std::is_x86_feature_detected!("avx512vnni") { + "AVX-512 VNNI VPDPBUSD" + } else if std::is_x86_feature_detected!("avxvnni") { + "AVX-VNNI" + } else { + "scalar reference" + }; + println!("amx_available: {} | matmul tier: {tier}", amx_available()); + let full: Vec<(u32, u32)> = (0..64).flat_map(|a| (0..64).map(move |b| (a, b))).collect(); + let cases: [(&str, Tuples, Tuples); 2] = [ + ( + "sparse", + vec![(1, 2), (5, 2), (1, 3), (9, 7)], + vec![(2, 4), (2, 6), (3, 4), (7, 8)], + ), + ("dense 64x64", full.clone(), full), + ]; + let q = PairQuery { + left: xlog_core::RelId(0), + right: xlog_core::RelId(1), + }; + for (name, r, s) in cases { + // Boolean: the register composition. + let xs = AddressMap::new(AX[0], r.iter().map(|t| t.0)).unwrap(); + let ys = + AddressMap::new(AX[1], r.iter().map(|t| t.1).chain(s.iter().map(|t| t.0))).unwrap(); + let zs = AddressMap::new(AX[2], s.iter().map(|t| t.1)).unwrap(); + let (rr, ss) = ( + register(&r, &xs, &ys).unwrap(), + register(&s, &ys, &zs).unwrap(), + ); + let mut work = Work::default(); + let b = median_ns(|| { + black_box(compose(black_box(&rr), black_box(&ss), &mut work)); + }); + // ℕ: the product alone, on prepared i8 tiles, output reused. + let (lx, ly, lz) = ( + local(r.iter().map(|t| t.0)), + local(r.iter().map(|t| t.1).chain(s.iter().map(|t| t.0))), + local(s.iter().map(|t| t.1)), + ); + let mut a = Array2::::zeros((64, 64)); + let mut bm = Array2::::zeros((64, 64)); + for &(x, y) in &r { + a[( + lx.local(x.into()).unwrap() as usize, + ly.local(y.into()).unwrap() as usize, + )] = 1; + } + for &(y, z) in &s { + bm[( + ly.local(y.into()).unwrap() as usize, + lz.local(z.into()).unwrap() as usize, + )] = 1; + } + let mut out = Array2::::zeros((64, 64)); + let n = median_ns(|| { + matmul_i8_to_i32(black_box(a.view()), black_box(bm.view()), out.view_mut()).unwrap(); + }); + // ℕ end to end: tiles built from the tuples, then the product. + let e = median_ns(|| { + black_box(pair_counts(&q, &r, &s, &lx, &ly, &lz).unwrap()); + }); + println!( + "{name}: compose (B) {b} ns | matmul (N, tiles prepared) {n} ns | pair_counts (N, from tuples) {e} ns" + ); + } +} diff --git a/crates/xlog-relcomp-probe/src/lib.rs b/crates/xlog-relcomp-probe/src/lib.rs index 9723fe92a..1ebf1255d 100644 --- a/crates/xlog-relcomp-probe/src/lib.rs +++ b/crates/xlog-relcomp-probe/src/lib.rs @@ -39,6 +39,7 @@ use std::collections::BTreeMap; pub mod native; +pub mod ncompose; pub mod recursive; use dr_math_binding_probe::relcomp::{ComposeRefusal, Rows, Work, compose}; diff --git a/crates/xlog-relcomp-probe/src/ncompose.rs b/crates/xlog-relcomp-probe/src/ncompose.rs new file mode 100644 index 000000000..50a205008 --- /dev/null +++ b/crates/xlog-relcomp-probe/src/ncompose.rs @@ -0,0 +1,163 @@ +//! ℕ-compose: `p(X, Z) :- r(X, Y), s(Y, Z)` with both ends kept, as an +//! integer matrix product over DR tiles (D-DYNBAG-1, plan +//! `d-dynbag-holographic-bags-v1.md` §7). +//! +//! R and S are sets of tuples (§9.2). On local tiles they are 0/1 matrices +//! A (|X| × |Y|) and B (|Y| × |Z|). Then `(A · B)[x][z]` is the number of `y` +//! with `r(x, y) ∧ s(y, z)`: the ℕ answer, one per witness. Its projection +//! `> 0` is the Boolean composition (`relcomp::compose`). +//! +//! The product is `ndarray::simd::matmul_i8_to_i32` (AMX `TDPBUSD`, AVX-512 +//! VNNI, or the scalar reference, bit-identical). One hop over 0/1 inputs is +//! exact: a cell is at most the Y tile width. A chain of hops (counts feeding +//! the next product) is not admitted here (§7.3). +//! +//! Lab only. The axes are DR's `LocalMap`s, one per X, Y, Z, chosen by the +//! caller; an id a map does not hold is refused. Bounds on X or Z are refused +//! by [`admit_pair`]; they are a later step. + +use lance_graph_dt64::dr::{LocalMap, MapRefusal}; +use ndarray::Array2; +use ndarray::simd::matmul_i8_to_i32; +use xlog_core::RelId; +use xlog_ir::{JoinType, ProjectExpr, RirNode}; + +use crate::{LowerRefusal, node_name, scan_with_bound}; + +/// `p(X, Z) :- r(X, Y), s(Y, Z)`, admitted. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct PairQuery { + /// `r(X, Y)`. + pub left: RelId, + /// `s(Y, Z)`. + pub right: RelId, +} + +/// Why the product did not answer. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum ComposeRefusal { + /// An id outside the axis's DR map. + Domain(MapRefusal), + /// The product itself refused (shape). + Matmul(String), +} + +/// Admit XLOG's compiled body for `p(X, Z) :- r(X, Y), s(Y, Z)`: an inner +/// join on Y keeping X and Z, in either join order. Bounds, other +/// projections and other joins are refused. +pub fn admit_pair(body: &RirNode) -> Result { + // XLOG's optimizer promotes the chain to `ChainJoin` (in either join + // order, chosen from statistics) and keeps the plain plan as `fallback`. + // Admit it only when its own relations name the same query as its + // fallback; a disagreement is refused, never resolved by picking one. + if let RirNode::ChainJoin { + left, + right, + left_key, + right_key, + output_columns, + fallback, + } = body + { + // The chain's columns are left ++ right, as for `Join` below. + let (r_side, s_side, xz) = match (*left_key, *right_key) { + (1, 0) => (left, right, (0, 3)), + (0, 1) => (right, left, (2, 1)), + _ => { + return Err(LowerRefusal::NeedsInverse { + left_keys: vec![*left_key], + right_keys: vec![*right_key], + }); + } + }; + // The chain must answer the same query as its fallback on its own: + // its projection and its inputs are checked, not only its relations. + match output_columns.as_slice() { + [ProjectExpr::Column(x), ProjectExpr::Column(z)] if (*x, *z) == xz => {} + _ => return Err(LowerRefusal::UnsupportedProjection), + } + let (l, lb) = scan_with_bound(r_side, 0)?; + let (r, rb) = scan_with_bound(s_side, 1)?; + if lb.is_some() || rb.is_some() { + return Err(LowerRefusal::UnsupportedFilter); + } + let own = PairQuery { left: l, right: r }; + return match admit_pair(fallback) { + Ok(fb) if fb == own => Ok(own), + Ok(_) => Err(LowerRefusal::ChainJoinDisagrees), + Err(e) => Err(e), + }; + } + let RirNode::Project { input, columns } = body else { + return Err(LowerRefusal::UnsupportedNode(node_name(body))); + }; + let RirNode::Join { + left, + right, + left_keys, + right_keys, + join_type, + } = input.as_ref() + else { + return Err(LowerRefusal::UnsupportedNode(node_name(input))); + }; + if *join_type != JoinType::Inner { + return Err(LowerRefusal::UnsupportedJoin(*join_type)); + } + // Join output columns are left ++ right: + // r(X,Y) ⋈ s(Y,Z): keys [1]/[0], columns [X,Y,Y,Z]: X at 0, Z at 3; + // s(Y,Z) ⋈ r(X,Y): keys [0]/[1], columns [Y,Z,X,Y]: X at 2, Z at 1. + let (r_side, s_side, xz) = match (left_keys.as_slice(), right_keys.as_slice()) { + ([1], [0]) => (left, right, (0, 3)), + ([0], [1]) => (right, left, (2, 1)), + _ => { + return Err(LowerRefusal::NeedsInverse { + left_keys: left_keys.clone(), + right_keys: right_keys.clone(), + }); + } + }; + match columns.as_slice() { + [ProjectExpr::Column(x), ProjectExpr::Column(z)] if (*x, *z) == xz => {} + _ => return Err(LowerRefusal::UnsupportedProjection), + } + let (l, lb) = scan_with_bound(r_side, 0)?; + let (r, rb) = scan_with_bound(s_side, 1)?; + if lb.is_some() || rb.is_some() { + return Err(LowerRefusal::UnsupportedFilter); + } + Ok(PairQuery { left: l, right: r }) +} + +/// A relation's 0/1 matrix over two DR axes. A repeated tuple sets the same +/// cell: base relations are sets. +fn matrix( + tuples: &[(u32, u32)], + rows: &LocalMap, + cols: &LocalMap, +) -> Result, ComposeRefusal> { + let mut m = Array2::::zeros((rows.width().positions(), cols.width().positions())); + for &(a, b) in tuples { + let i = rows.local(u64::from(a)).map_err(ComposeRefusal::Domain)?; + let j = cols.local(u64::from(b)).map_err(ComposeRefusal::Domain)?; + m[(usize::from(i), usize::from(j))] = 1; + } + Ok(m) +} + +/// Witnesses per (x, z), indexed by local position on the X and Z tiles. +pub fn pair_counts( + _q: &PairQuery, + left: &[(u32, u32)], + right: &[(u32, u32)], + xs: &LocalMap, + ys: &LocalMap, + zs: &LocalMap, +) -> Result, ComposeRefusal> { + let a = matrix(left, xs, ys)?; + let b = matrix(right, ys, zs)?; + let mut out = Array2::::zeros((a.nrows(), b.ncols())); + matmul_i8_to_i32(a.view(), b.view(), out.view_mut()) + .map_err(|e| ComposeRefusal::Matmul(format!("{e:?}")))?; + Ok(out) +} diff --git a/crates/xlog-relcomp-probe/tests/litho.rs b/crates/xlog-relcomp-probe/tests/litho.rs index 701cf3137..46a64c813 100644 --- a/crates/xlog-relcomp-probe/tests/litho.rs +++ b/crates/xlog-relcomp-probe/tests/litho.rs @@ -629,3 +629,205 @@ fn a_repeated_base_fact_is_not_a_second_witness() { .collect(); assert_eq!(got, expected()); } + +// ── Part 4: ℕ-compose, `p(X, Z)` with both ends kept (D-DYNBAG-1) ────────── + +use xlog_relcomp_probe::ncompose::{ComposeRefusal, admit_pair, pair_counts}; + +/// The multiplicity fixture with the rule that keeps both ends. +fn pairs_src(facts_src: &str) -> String { + format!( + "pred r(u32, u32).\npred s(u32, u32).\npred p(u32, u32).\n{facts_src}\np(X, Z) :- r(X, Y), s(Y, Z).\n?- p(X, Z).\n" + ) +} + +const MULT_FACTS: &str = "r(1, 2). r(5, 2). r(1, 3). r(9, 7).\ns(2, 4). s(2, 6). s(3, 4). s(7, 8)."; + +fn axis(ids: impl Iterator) -> LocalMap { + let ids: BTreeSet = ids.map(u64::from).collect(); + LocalMap::prepare(Width::W64, &ids.into_iter().collect::>()).unwrap() +} + +/// The ℕ answer of `p(X, Z)` through two DR64² tiles, keyed by ids. +fn n_compose(r: &[(u32, u32)], s: &[(u32, u32)], p: &RirNode) -> BTreeMap<(u32, u32), u64> { + let q = admit_pair(p).expect("p(X, Z) is admitted"); + let xs = axis(r.iter().map(|t| t.0)); + let ys = axis(r.iter().map(|t| t.1).chain(s.iter().map(|t| t.0))); + let zs = axis(s.iter().map(|t| t.1)); + let c = pair_counts(&q, r, s, &xs, &ys, &zs).unwrap(); + let mut out = BTreeMap::new(); + for ((x, z), n) in c.indexed_iter().filter(|(_, n)| **n != 0) { + let x = u32::try_from(xs.global(x as u8).unwrap()).unwrap(); + let z = u32::try_from(zs.global(z as u8).unwrap()).unwrap(); + out.insert((x, z), u64::try_from(*n).unwrap()); + } + out +} + +/// Oracle 1: every (x, y, z) witness, per (x, z). +fn scalar_pairs(r: &[(u32, u32)], s: &[(u32, u32)]) -> BTreeMap<(u32, u32), u64> { + let mut out = BTreeMap::new(); + for &(x, y) in r { + for &(y2, z) in s { + if y == y2 { + *out.entry((x, z)).or_insert(0) += 1; + } + } + } + out +} + +/// Oracle 2: XLOG's own plan for `p`, walked with bag semantics, counted per +/// output row. +fn walked_pairs(c: &Compiled, p: &RirNode) -> BTreeMap<(u32, u32), u64> { + let mut out = BTreeMap::new(); + for t in walk(p, &edb(c), false) { + let k = (u32::try_from(t[0]).unwrap(), u32::try_from(t[1]).unwrap()); + *out.entry(k).or_insert(0) += 1; + } + out +} + +/// B: the Boolean composition on the DR64 register, as (x, z) ids. +fn register_pairs(r: &[(u32, u32)], s: &[(u32, u32)]) -> BTreeSet<(u32, u32)> { + let xs = AddressMap::new(X, r.iter().map(|t| t.0)).unwrap(); + let ys = AddressMap::new(Y, r.iter().map(|t| t.1).chain(s.iter().map(|t| t.0))).unwrap(); + let zs = AddressMap::new(Z, s.iter().map(|t| t.1)).unwrap(); + let (rr, ss) = ( + register(r, &xs, &ys).unwrap(), + register(s, &ys, &zs).unwrap(), + ); + let rz = compose(&rr, &ss, &mut Work::default()); + let mut out = BTreeSet::new(); + for x in xs.ids(xs.valid()) { + let row = rz[xs.slot(x).unwrap()] & zs.valid(); + for z in zs.ids(row) { + out.insert((x, z)); + } + } + out +} + +#[test] +fn n_compose_counts_witnesses_per_x_z() { + let c = compile(&pairs_src(MULT_FACTS)); + let (r, s) = ( + facts(&c.program, "r").unwrap(), + facts(&c.program, "s").unwrap(), + ); + let p = rule(&c, "p"); + let got = n_compose(&r, &s, p); + let by_hand = BTreeMap::from([ + ((1, 4), 2), + ((1, 6), 1), + ((5, 4), 1), + ((5, 6), 1), + ((9, 8), 1), + ]); + assert_eq!(got, by_hand); + assert_eq!(got, scalar_pairs(&r, &s), "oracle 1"); + assert_eq!(got, walked_pairs(&c, p), "oracle 2"); + // B is the projection of ℕ, bit for bit. + let support: BTreeSet<(u32, u32)> = got.keys().copied().collect(); + assert_eq!(support, register_pairs(&r, &s)); +} + +/// The 300-case differential: random sets over ids 0..6 on every axis. +#[test] +fn n_compose_differential_against_both_oracles() { + let mut seed: u64 = 0x9E37_79B9_7F4A_7C15; + let mut next = |m: u64| { + seed = seed.wrapping_mul(6_364_136_223_846_793_005).wrapping_add(1); + (seed >> 33) % m + }; + let mut nonempty = 0; + for _ in 0..300 { + let mut f = String::new(); + // X, Y and Z over 0..6: an (x, z) pair often meets through more + // than one y, so ℕ differs from B in most cases. + for _ in 0..(1 + next(12)) { + f.push_str(&format!("r({}, {}). ", next(6), next(6))); + } + for _ in 0..(1 + next(12)) { + f.push_str(&format!("s({}, {}). ", next(6), next(6))); + } + let c = compile(&pairs_src(&f)); + let (r, s) = ( + facts(&c.program, "r").unwrap(), + facts(&c.program, "s").unwrap(), + ); + let p = rule(&c, "p"); + let got = n_compose(&r, &s, p); + assert_eq!(got, scalar_pairs(&r, &s), "{f}"); + assert_eq!(got, walked_pairs(&c, p), "{f}"); + let support: BTreeSet<(u32, u32)> = got.keys().copied().collect(); + assert_eq!(support, register_pairs(&r, &s), "{f}"); + nonempty += usize::from(got.values().any(|n| *n > 1)); + } + assert!( + nonempty > 30, + "multiplicity > 1 occurred in {nonempty} cases" + ); +} + +#[test] +fn more_ids_than_the_tile_is_refused() { + let r: Vec<(u32, u32)> = (0..65).map(|x| (x, 1)).collect(); + let s = vec![(1, 2)]; + let c = compile(&pairs_src(MULT_FACTS)); + let q = admit_pair(rule(&c, "p")).unwrap(); + let xs = LocalMap::prepare(Width::W64, &(0..64).collect::>()).unwrap(); + let ys = axis([1].into_iter()); + let zs = axis([2].into_iter()); + assert!(matches!( + pair_counts(&q, &r, &s, &xs, &ys, &zs), + Err(ComposeRefusal::Domain(_)) + )); +} + +#[test] +fn q_y_is_still_not_admitted_as_p_x_z() { + let c = compile(&pairs_src(MULT_FACTS)); + assert!(admit(rule(&c, "p")).is_err()); + let mult = compile(MULT); + assert!( + admit_pair(rule(&mult, "q")).is_err(), + "q(Y) is not a pair rule" + ); +} + +/// A `ChainJoin` is admitted only when its own projection and inputs say what +/// the fallback says: a reordered projection or a filter on a chain input is +/// refused, not answered as the fallback's query. +#[test] +fn a_chain_join_that_disagrees_with_its_fallback_is_refused() { + let c = compile(&pairs_src(MULT_FACTS)); + let body = rule(&c, "p").clone(); + let RirNode::ChainJoin { .. } = &body else { + panic!("the compiled pair rule is a ChainJoin: {body:?}"); + }; + assert!(admit_pair(&body).is_ok()); + + let mut swapped = body.clone(); + if let RirNode::ChainJoin { output_columns, .. } = &mut swapped { + output_columns.reverse(); + } + assert_eq!( + admit_pair(&swapped), + Err(LowerRefusal::UnsupportedProjection) + ); + + let mut filtered = body.clone(); + if let RirNode::ChainJoin { left, .. } = &mut filtered { + let input = std::mem::replace(left.as_mut(), RirNode::Unit); + **left = RirNode::Filter { + input: Box::new(input), + predicate: Expr::Compare { + left: Box::new(Expr::Column(0)), + op: CompareOp::Eq, + right: Box::new(Expr::Const(ConstValue::U32(1))), + }, + }; + } + assert_eq!(admit_pair(&filtered), Err(LowerRefusal::UnsupportedFilter)); +}