diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 920a7808b..a21d2bab2 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -71,6 +71,7 @@ Plan: `.claude/plans/d-soc-fence-v1.md`. Plan only; nothing runs yet. | D-XLOG-NATIVE-4 | create `crates/lance-graph-xlog` (shape per Amendment C/D: Quack + mask-risc + DT64, no `xlog-*` or arrow dep, no plan type) | Shipped (#1907) | N1, N4, N5 met; N2 scalar oracle in-crate + both oracles in the probe; N3 typed refusals | | D-XLOG-NATIVE-6 | probe executes on `ndarray::simd` (`src/native.rs`): support via `eq_u32_to_mask` / `mask_scatter_or_u32` / `mask_and`, bag count via `masked_group_count_u32` ×2 + `checked_binary_i64`; DR `LocalMap` up to 256; register path kept as differential (plan Amendment B) | Shipped (#1884) | lab probe; layering fix is D-DYNBAG-0 | | D-XLOG-NATIVE-5 | bag count bound through the production bind store (`lance-graph-dt64` + `dt64::dr`), polyfill words by position; Epi5 reading of XLOG results pinned (plan Amendment A) | Shipped (#1884) | lab execution only | +| D-XLOG-NATIVE-7 | resident hypercube context (`lance_graph_xlog::cube`): one intake for every relation, `Copy` view handles, terminals only run programs; XLOG plans executed on it in the XLOG fork (`xlog-cube`) | In PR (#1911; xlog#2): parity + 300-case differential; 12/230 integration rules admitted | recursion, derived inputs | ## D-LITHO — the photolithography contract (2026-10-30) diff --git a/.claude/board/entries/2026-11-01-xlog-resident-hypercube-context.md b/.claude/board/entries/2026-11-01-xlog-resident-hypercube-context.md new file mode 100644 index 000000000..bcfca12dd --- /dev/null +++ b/.claude/board/entries/2026-11-01-xlog-resident-hypercube-context.md @@ -0,0 +1,66 @@ +# D-XLOG-NATIVE-7: one resident hypercube as XLOG's execution context (probe) + +**Date 2026-11-01 · ID D-XLOG-NATIVE-7 · Status PROBE · Plan `.claude/plans/d-xlog-native-v1.md` (Amendment F, pending beside E in #1910) · Crate `crates/lance-graph-xlog` (`cube`)** + +DECISION (direction, from the session): the experiment replaces the machine +under XLOG's CUDA-facing API with one resident context, rather than speeding up +operators one relation at a time. Observable results of admitted classical +XLOG queries must match; the internal model is free. + +What `lance_graph_xlog::cube` does (VERIFIED-IN-CODE, `src/cube.rs`): + +- `Cube::new` looks up every id of every relation, both columns, once in one + tile's `LocalMap` under a `MapTable` frame. Nothing is looked up again. +- `scan` / `bind` / `join` return `View`s: `Copy` handles into the cube's node + list. Making a view runs no program and builds no row. +- Terminals: `support` (two `ScatterOrU32` + `mask_and`), `bag_count` (two + `GroupReduce Count` + checked product), `materialize_rows` (the one named + place tuples are built). Each re-checks the frame. +- Any relation joins any other on either column from the same lanes: lanes + are stored per relation, not per join orientation. + +TEST-PINNED (`tests/cube.rs`, 6 tests): + +- parity with the per-call `support` / `bag_count` under five bound shapes; +- all 36 (relation pair × column pair) joins over three relations match an + enumeration oracle, and `LocalMap::lookups()` is unchanged after intake; +- 50 composed joins run 0 programs; `support` then runs exactly 2, builds 0 rows; +- a bound outside the map short-circuits its side (1 program, not 2); +- 300-case randomised differential (ids 0..12, either orientation, optional + bounds): bag count, support and rows; 150+ non-empty; +- stale frame, foreign view, bound on the join column, non-join terminal refused. + +Disable runs, all red: bound ignored in programs; support on the wrong column; +support skips the frame check; `materialize_rows` ignores the bound. + +XLOG side (MEASURED by reading `xlog-runtime` / `xlog-cuda`): + +- `CudaBuffer` and `CudaKernelProvider` are concrete structs, not traits. + `CudaBuffer` fields are `pub(crate)`; callers use 16 public methods. +- `xlog-runtime` calls 79 of the provider's 397 public functions; ~25 are + telemetry/resets, ~30 are WCOJ/clique variants. +- 12 `provider.device()` sites: 1 in the core dispatcher (join-index cache + key), 2 `synchronize`, 9 in `epistemic_workspace` / `resident` (outside the + 13 RIR kinds). +- `xlog-cuda/build.rs` panics without `nvcc` unless `XLOG_RUSTDOC_NO_CUDA=1` + (empty kernel data). With it, `cargo check -p xlog-runtime` finishes on a + host with no CUDA toolkit (MEASURED 2026-11-01, 3 warnings, 0 errors), so a + device-less build of the XLOG fork is reachable. + +Comparison with Amendment E (caller-level polyfill): + +- E keeps XLOG's API and moves its callers onto `lance-graph-xlog` one + operator at a time; every call still owns its inputs. +- F gives the CUDA-facing handles a shared context: a `CudaBuffer` would be a + `View` plus the cube it belongs to, so a join between two XLOG buffers needs + no new lanes, no index (XLOG's `JoinIndexKey` cache becomes redundant) and no + device. +- Neither is promoted. F is better for the 13 RIR kinds only if the dispatcher + can keep buffers as views between nodes; a node that downloads rows ends the + advantage at that node. + +**OPEN:** recursion (semi-naive delta over versioned masks), union and +difference as views; more than one tile; n-ary relations; a `CudaBuffer::View` variant and a +device-less provider in the XLOG fork; per-node cost vs the per-call path. +Reasoning surfaces (NARS revision, thought layers, CausalEdge64) are not in this +probe and need their own contracts. diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index efac24535..1fb5f35a1 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,10 +25,11 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -539 entries, 2026-08-06 .. 2026-11-01. +540 entries, 2026-08-06 .. 2026-11-01. | date | entry id | finding | file | |---|---|---|---| +| 2026-11-01 | `D-XLOG-NATIVE-7` | | [2026-11-01-xlog-resident-hypercube-context.md](2026-11-01-xlog-resident-hypercube-context.md) | | 2026-11-01 | `D-DT64-7-5` | | [2026-11-01-maptable-versioned-local-map.md](2026-11-01-maptable-versioned-local-map.md) | | 2026-11-01 | `D-XLOG-NATIVE-4` | | [2026-11-01-lance-graph-xlog-crate.md](2026-11-01-lance-graph-xlog-crate.md) | | 2026-11-01 | `D-DYNBAG-0` | | [2026-11-01-dynbag-0-xlog-through-quack-mask-risc.md](2026-11-01-dynbag-0-xlog-through-quack-mask-risc.md) | diff --git a/crates/lance-graph-xlog/src/cube.rs b/crates/lance-graph-xlog/src/cube.rs new file mode 100644 index 000000000..7689a941c --- /dev/null +++ b/crates/lance-graph-xlog/src/cube.rs @@ -0,0 +1,402 @@ +//! The resident hypercube: one execution context for every relation, and +//! views into it (plan `d-xlog-native-v1.md` Amendment F). +//! +//! [`support`](crate::support) and [`bag_count`](crate::bag_count) build their +//! lanes per call, oriented for one join. A [`Cube`] builds them once, for +//! every relation and both columns, against one shared coordinate map: each +//! id in either column of any relation is looked up once, at construction, +//! and stored as its local position in the tile. Afterwards any relation can +//! join any other on either column without new lanes, new lookups or a new +//! index. +//! +//! The operations a caller composes ([`Cube::scan`], [`Cube::bind`], +//! [`Cube::join`]) return [`View`]s. A view is a `Copy` handle into the +//! cube's node list; making one touches no rows and runs nothing. Only the +//! terminals run: +//! +//! | terminal | answer | how | +//! |---|---|---| +//! | [`Cube::support`] | which join values have a partner on both sides | two `ScatterOrU32` programs, `and` | +//! | [`Cube::bag_count`] | derivations per join value | two `GroupReduce Count` programs, checked `i64` product | +//! | [`Cube::materialize_rows`] | the joined tuples | the one named place rows are built | +//! +//! Every terminal re-checks the cube's `MapTable` frame, so a view over a +//! stale coordinate assignment is refused. [`CubeStats`] counts programs run +//! and rows materialized, so a test can show that composing views runs +//! nothing. +//! +//! Scope: one tile (at most 256 ids across all relations), binary relations, +//! a join of two sides, at most one bound per side and only on the side's +//! free column. Recursion, union and difference are not here. + +use std::cell::Cell; +use std::sync::atomic::{AtomicU64, Ordering}; + +use lance_graph_dt64::dr::{Frame, LocalMap, MapRead, MapRefusal, MapTable, TileId}; +use lance_graph_mask_risc::{LaneRef, Out}; +use lance_graph_quack::{Agg, Cmp, Col, Filter, GroupAddr, GroupAgg, Mask, Query}; +use ndarray::simd::{checked_binary_i64, mask_and, I64Op}; + +use crate::{all_live, run_lanes, NativeRefusal}; + +/// A relation's index in the order given to [`Cube::new`]. +pub type RelId = u32; + +/// A handle into a [`Cube`]. Copying it copies no rows. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct View { + cube: u64, + node: u32, +} + +#[derive(Clone, Copy, Debug)] +enum Node { + Scan(RelId), + /// `col == value`. `None`: the value is not in the map, so no row + /// matches. + Bind { + rel: RelId, + col: u8, + value: Option, + }, + /// `left.lcol == right.rcol`. + Join { + left: View, + lcol: u8, + right: View, + rcol: u8, + }, +} + +/// Why a view was not built or a terminal did not run. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum CubeRefusal { + /// A view from another cube. + ForeignView, + /// No relation with this id. + UnknownRel(RelId), + /// A column other than 0 or 1. + NoSuchColumn(u8), + /// `bind` takes a scan; `join` takes scans or binds. + NotASide, + /// The terminal takes a join. + NotAJoin, + /// A bound on the join column (not a free column). + BoundOnJoinColumn, + /// An id in a relation outside the shared map. + Domain(MapRefusal), + /// The map is not readable under the cube's frame. + Map(MapRead), + /// The computation refused. + Native(NativeRefusal), +} + +/// What the cube has done since it was built. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct CubeStats { + /// mask-risc programs run. + pub programs: u64, + /// Tuples built by [`Cube::materialize_rows`]. + pub rows_materialized: u64, +} + +/// One relation's lanes: both columns as local positions in the shared map. +struct Rel { + n: usize, + cols: [Vec; 2], + live: Vec, +} + +/// One side of a join, resolved from a view. +#[derive(Clone, Copy)] +struct Side { + rel: RelId, + /// `(free column, value)`; `Some(None)`: bound to an unmapped id. + bound: Option<(u8, Option)>, +} + +/// A side with the column it joins on. +type JoinSide = (Side, u8); + +static NEXT_CUBE: AtomicU64 = AtomicU64::new(1); + +/// The resident hypercube. See the module docs. +pub struct Cube { + id: u64, + frame: Frame, + tile: TileId, + n: usize, + rels: Vec, + nodes: Vec, + stats: Cell, +} + +impl Cube { + /// Build the cube: every id of every relation looked up once in the + /// tile's map, read under `frame`. + pub fn new( + table: &MapTable, + frame: &Frame, + tile: TileId, + relations: &[&[(u32, u32)]], + ) -> Result { + let map = table.map(frame, tile).map_err(CubeRefusal::Map)?; + let rels = relations + .iter() + .map(|tuples| rel(tuples, map)) + .collect::>()?; + Ok(Self { + id: NEXT_CUBE.fetch_add(1, Ordering::Relaxed), + frame: frame.clone(), + tile, + n: map.len(), + rels, + nodes: Vec::new(), + stats: Cell::new(CubeStats::default()), + }) + } + + pub fn stats(&self) -> CubeStats { + self.stats.get() + } + + fn push(&mut self, node: Node) -> View { + self.nodes.push(node); + View { + cube: self.id, + node: (self.nodes.len() - 1) as u32, + } + } + + fn node(&self, v: View) -> Result { + if v.cube != self.id { + return Err(CubeRefusal::ForeignView); + } + Ok(self.nodes[v.node as usize]) + } + + /// Every row of a relation. + pub fn scan(&mut self, rel: RelId) -> Result { + if rel as usize >= self.rels.len() { + return Err(CubeRefusal::UnknownRel(rel)); + } + Ok(self.push(Node::Scan(rel))) + } + + /// The rows of a scan whose column `col` holds `value`. One map lookup + /// for `value`, under the cube's frame. + pub fn bind( + &mut self, + table: &MapTable, + scan: View, + col: u8, + value: u32, + ) -> Result { + let Node::Scan(rel) = self.node(scan)? else { + return Err(CubeRefusal::NotASide); + }; + check_col(col)?; + let map = self.map(table)?; + let value = map.local(u64::from(value)).ok(); + Ok(self.push(Node::Bind { rel, col, value })) + } + + /// `left.lcol == right.rcol`, where each side is a scan or a bind. + pub fn join( + &mut self, + left: View, + lcol: u8, + right: View, + rcol: u8, + ) -> Result { + check_col(lcol)?; + check_col(rcol)?; + self.side(left, lcol)?; + self.side(right, rcol)?; + Ok(self.push(Node::Join { + left, + lcol, + right, + rcol, + })) + } + + fn side(&self, v: View, join_col: u8) -> Result { + match self.node(v)? { + Node::Scan(rel) => Ok(Side { rel, bound: None }), + Node::Bind { rel, col, value } if col != join_col => Ok(Side { + rel, + bound: Some((col, value)), + }), + Node::Bind { .. } => Err(CubeRefusal::BoundOnJoinColumn), + Node::Join { .. } => Err(CubeRefusal::NotASide), + } + } + + fn sides(&self, v: View) -> Result<(JoinSide, JoinSide), CubeRefusal> { + let Node::Join { + left, + lcol, + right, + rcol, + } = self.node(v)? + else { + return Err(CubeRefusal::NotAJoin); + }; + Ok(( + (self.side(left, lcol)?, lcol), + (self.side(right, rcol)?, rcol), + )) + } + + fn map<'t>(&self, table: &'t MapTable) -> Result<&'t LocalMap, CubeRefusal> { + table.map(&self.frame, self.tile).map_err(CubeRefusal::Map) + } + + fn run(&self, side: Side, query: &Query, out: Out<'_>) -> Result<(), CubeRefusal> { + let r = &self.rels[side.rel as usize]; + let lanes = [LaneRef::U32(&r.cols[0]), LaneRef::U32(&r.cols[1])]; + let mut s = self.stats.get(); + s.programs += 1; + self.stats.set(s); + run_lanes(r.n, &r.live, &lanes, query, out).map_err(CubeRefusal::Native) + } + + /// The filter of a side, or `None` when it is bound to an unmapped id and + /// no row can match (nothing runs). + fn filter(side: Side) -> Option { + match side.bound { + None => Some(Filter::plane(Mask(0))), + Some((_, None)) => None, + Some((col, Some(v))) => { + Some(Filter::cmp(Col(u16::from(col)), Cmp::EqU32(u32::from(v)))) + } + } + } + + fn reached(&self, side: Side, join_col: u8) -> Result, CubeRefusal> { + let mut words = vec![0u64; self.n.div_ceil(64)]; + // An empty domain has nothing to reach; mask-risc refuses an empty sink. + if self.n == 0 { + return Ok(words); + } + if let Some(filter) = Self::filter(side) { + let q = Query { + filter, + agg: Agg::ScatterOrU32 { + fk: Col(u16::from(join_col)), + out_rows: self.n as u32, + }, + }; + self.run(side, &q, Out::Mask(&mut words))?; + } + Ok(words) + } + + fn degrees(&self, side: Side, join_col: u8) -> Result, CubeRefusal> { + let mut counts = vec![0i64; self.n]; + // An empty domain has nothing to count; mask-risc refuses an empty sink. + if self.n == 0 { + return Ok(counts); + } + if let Some(filter) = Self::filter(side) { + let q = Query { + filter, + agg: Agg::GroupReduce { + key: GroupAddr::Local(Col(u16::from(join_col))), + agg: GroupAgg::Count, + }, + }; + self.run(side, &q, Out::I64(&mut counts))?; + } + Ok(counts) + } + + /// The join values with a partner on both sides: one bit per local + /// position of the shared map. + pub fn support(&self, table: &MapTable, join: View) -> Result, CubeRefusal> { + let ((l, lc), (r, rc)) = self.sides(join)?; + self.map(table)?; + let a = self.reached(l, lc)?; + let b = self.reached(r, rc)?; + let mut out = vec![0u64; a.len()]; + mask_and(&a, &b, &mut out); + Ok(out) + } + + /// Derivations per join value: one per pair of matching rows, as XLOG's + /// `count` counts body bindings. A count past `i64` is refused. + pub fn bag_count(&self, table: &MapTable, join: View) -> Result, CubeRefusal> { + let ((l, lc), (r, rc)) = self.sides(join)?; + self.map(table)?; + let a = self.degrees(l, lc)?; + let b = self.degrees(r, rc)?; + let mut out = vec![0i64; self.n]; + checked_binary_i64(I64Op::Mul, &a, &b, &mut out) + .map_err(|e| CubeRefusal::Native(NativeRefusal::Count(e)))?; + Ok(out) + } + + /// The joined tuples `(left free, join value, right free)` as global ids: + /// the one place this module builds rows, for a caller that downloads the + /// relation. Restricted to the join's support first, so rows with no + /// partner are never visited twice. + pub fn materialize_rows( + &self, + table: &MapTable, + join: View, + ) -> Result, CubeRefusal> { + let ((l, lc), (r, rc)) = self.sides(join)?; + let map = self.map(table)?; + let keep = &self.support(table, join)?; + let rows = |side: Side, jc: u8| { + let rel = &self.rels[side.rel as usize]; + let fc = 1 - jc as usize; + let (j, f) = (&rel.cols[jc as usize], &rel.cols[fc]); + let bound = side.bound.map(|(_, v)| v); + (0..j.len()) + .filter(move |&i| keep[j[i] as usize / 64] >> (j[i] % 64) & 1 == 1) + .filter(move |&i| bound.is_none_or(|v| v == Some(f[i] as u8))) + .map(move |i| (j[i], f[i])) + }; + let mut out = Vec::new(); + for (jl, fl) in rows(l, lc) { + for (jr, fr) in rows(r, rc) { + if jl == jr { + let g = |x: u32| map.global(x as u8).expect("local position from intake"); + out.push((g(fl), g(jl), g(fr))); + } + } + } + let mut s = self.stats.get(); + s.rows_materialized += out.len() as u64; + self.stats.set(s); + Ok(out) + } +} + +fn check_col(col: u8) -> Result<(), CubeRefusal> { + if col > 1 { + Err(CubeRefusal::NoSuchColumn(col)) + } else { + Ok(()) + } +} + +fn rel(tuples: &[(u32, u32)], map: &LocalMap) -> Result { + let mut cols = [ + Vec::with_capacity(tuples.len()), + Vec::with_capacity(tuples.len()), + ]; + for &(a, b) in tuples { + for (c, id) in [(0, a), (1, b)] { + let l = map.local(u64::from(id)).map_err(CubeRefusal::Domain)?; + cols[c].push(u32::from(l)); + } + } + Ok(Rel { + n: tuples.len(), + live: all_live(tuples.len()), + cols, + }) +} diff --git a/crates/lance-graph-xlog/src/lib.rs b/crates/lance-graph-xlog/src/lib.rs index c11b4276d..4eff2a4f1 100644 --- a/crates/lance-graph-xlog/src/lib.rs +++ b/crates/lance-graph-xlog/src/lib.rs @@ -48,6 +48,8 @@ use lance_graph_quack::{ pub use ndarray::simd::mask_and as and; use ndarray::simd::{checked_binary_i64, mask_and, I64ArithError, I64Op}; +pub mod cube; + /// The bounds of the admitted chain: `X == left`, `Z == right` when set. #[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] pub struct Bounds { @@ -96,6 +98,20 @@ struct Side { live: Vec, } +/// The live plane of `n` rows: every row set. +pub(crate) fn all_live(n: usize) -> Vec { + (0..n.div_ceil(64)) + .map(|i| { + let rows = (n - i * 64).min(64); + if rows == 64 { + u64::MAX + } else { + (1u64 << rows) - 1 + } + }) + .collect() +} + fn side( tuples: &[(u32, u32)], y_of: fn(&(u32, u32)) -> u32, @@ -107,17 +123,7 @@ fn side( .map(|t| mid.local(u64::from(y_of(t))).map(u32::from)) .collect::>() .map_err(NativeRefusal::Domain)?; - let n = tuples.len(); - let live = (0..n.div_ceil(64)) - .map(|i| { - let rows = (n - i * 64).min(64); - if rows == 64 { - u64::MAX - } else { - (1u64 << rows) - 1 - } - }) - .collect(); + let live = all_live(tuples.len()); Ok(Side { free: tuples.iter().map(free_of).collect(), y_local, @@ -148,13 +154,24 @@ fn selected(bound: Option) -> Filter { /// Describe, lower and run one query over one side. fn run(side: &Side, query: &Query, out: Out<'_>) -> Result<(), NativeRefusal> { - let n = side.free.len(); - let masks: [&[u64]; 1] = [&side.live]; let lanes = [LaneRef::U32(&side.free), LaneRef::U32(&side.y_local)]; + run_lanes(side.free.len(), &side.live, &lanes, query, out) +} + +/// Lower and run one query over `n` rows: one live plane (mask 0) and the +/// given lanes. +pub(crate) fn run_lanes( + n: usize, + live: &[u64], + lanes: &[LaneRef<'_>], + query: &Query, + out: Out<'_>, +) -> Result<(), NativeRefusal> { + let masks: [&[u64]; 1] = [live]; let planes = Planes { n_rows: n, masks: &masks, - lanes: &lanes, + lanes, }; let program = lower(query).map_err(NativeRefusal::Lower)?; let mut scratch = Scratch::for_program(&program, n).map_err(NativeRefusal::Exec)?; diff --git a/crates/lance-graph-xlog/tests/cube.rs b/crates/lance-graph-xlog/tests/cube.rs new file mode 100644 index 000000000..64507cc11 --- /dev/null +++ b/crates/lance-graph-xlog/tests/cube.rs @@ -0,0 +1,278 @@ +//! The resident hypercube (plan `d-xlog-native-v1.md` Amendment F) against a +//! scalar oracle and against the per-call free functions, plus the gates that +//! make it a context rather than a faster operator: composing views runs +//! nothing, any pair of relations joins on any columns without a new lookup, +//! and a stale frame is refused. + +use std::collections::BTreeMap; + +use lance_graph_dt64::dr::{LocalMap, MapRead, MapTable, Width}; +use lance_graph_xlog::cube::{Cube, CubeRefusal, View}; +use lance_graph_xlog::{bag_count, ids, nonzero, support, Bounds}; + +const TILE: u32 = 7; + +/// r(X, Y), s(Y, Z), t(Z, X): repeated rows, an id with no partner. +const R: [(u32, u32); 6] = [(1, 2), (5, 2), (1, 3), (9, 7), (4, 8), (1, 2)]; +const S: [(u32, u32); 5] = [(2, 4), (2, 6), (3, 4), (7, 8), (9, 1)]; +const T: [(u32, u32); 4] = [(4, 1), (6, 5), (8, 9), (4, 4)]; + +fn table_over(rels: &[&[(u32, u32)]]) -> MapTable { + let mut all: Vec = rels + .iter() + .flat_map(|r| r.iter().flat_map(|&(a, b)| [u64::from(a), u64::from(b)])) + .collect(); + all.sort_unstable(); + all.dedup(); + let mut t = MapTable::new(1, 1); + t.advance( + 1, + 1, + vec![(TILE, Some(LocalMap::prepare(Width::W256, &all).unwrap()))], + ); + t +} + +/// Every pair of matching rows, enumerated. +fn oracle( + l: &[(u32, u32)], + lc: u8, + lb: Option, + r: &[(u32, u32)], + rc: u8, + rb: Option, +) -> Vec<(u64, u64, u64)> { + let col = |t: (u32, u32), c: u8| if c == 0 { t.0 } else { t.1 }; + let mut out = Vec::new(); + for &a in l { + for &b in r { + let (ja, fa, jb, fb) = (col(a, lc), col(a, 1 - lc), col(b, rc), col(b, 1 - rc)); + if ja == jb && lb.is_none_or(|v| v == fa) && rb.is_none_or(|v| v == fb) { + out.push((u64::from(fa), u64::from(ja), u64::from(fb))); + } + } + } + out.sort_unstable(); + out +} + +fn counts(rows: &[(u64, u64, u64)]) -> BTreeMap { + let mut m = BTreeMap::new(); + for &(_, y, _) in rows { + *m.entry(y).or_insert(0) += 1; + } + m +} + +/// A side of a join: a scan, or a bind on its free column. +fn side(cube: &mut Cube, table: &MapTable, rel: u32, join_col: u8, bound: Option) -> View { + let scan = cube.scan(rel).unwrap(); + match bound { + None => scan, + Some(v) => cube.bind(table, scan, 1 - join_col, v).unwrap(), + } +} + +#[test] +fn the_cube_answers_what_the_free_functions_answer() { + let rels: [&[(u32, u32)]; 2] = [&R, &S]; + let table = table_over(&rels); + let mut cube = Cube::new(&table, &table.seal(), TILE, &rels).unwrap(); + let map = table.map(&table.seal(), TILE).unwrap(); + + let mut ys: Vec = R + .iter() + .map(|t| u64::from(t.1)) + .chain(S.iter().map(|t| u64::from(t.0))) + .collect(); + ys.sort_unstable(); + ys.dedup(); + let y_map = LocalMap::prepare(Width::W64, &ys).unwrap(); + + for (left, right) in [ + (None, None), + (Some(1), None), + (None, Some(4)), + (Some(1), Some(6)), + (Some(99), None), + ] { + let l = side(&mut cube, &table, 0, 1, left); + let r = side(&mut cube, &table, 1, 0, right); + let j = cube.join(l, 1, r, 0).unwrap(); + let q = Bounds { left, right }; + + let free = support(q, &R, &S, &y_map).unwrap(); + assert_eq!( + ids(map, &cube.support(&table, j).unwrap()), + ids(&y_map, &free.answer), + "{q:?}" + ); + let free = bag_count(q, &R, &S, &y_map).unwrap(); + assert_eq!( + nonzero(map, &cube.bag_count(&table, j).unwrap()), + nonzero(&y_map, &free), + "{q:?}" + ); + } +} + +#[test] +fn any_pair_joins_on_any_columns_without_a_new_lookup() { + let rels: [&[(u32, u32)]; 3] = [&R, &S, &T]; + let table = table_over(&rels); + let mut cube = Cube::new(&table, &table.seal(), TILE, &rels).unwrap(); + let map = table.map(&table.seal(), TILE).unwrap(); + let after_intake = map.lookups(); + assert_eq!(after_intake, 2 * (R.len() + S.len() + T.len()) as u64); + + let mut checked = 0; + for a in 0..3u32 { + for b in 0..3u32 { + for lc in 0..2u8 { + for rc in 0..2u8 { + let l = cube.scan(a).unwrap(); + let r = cube.scan(b).unwrap(); + let j = cube.join(l, lc, r, rc).unwrap(); + let want = oracle(rels[a as usize], lc, None, rels[b as usize], rc, None); + let got = cube.bag_count(&table, j).unwrap(); + let got: BTreeMap = nonzero(map, &got).into_iter().collect(); + assert_eq!(got, counts(&want), "rel {a}.{lc} = rel {b}.{rc}"); + let mut rows = cube.materialize_rows(&table, j).unwrap(); + rows.sort_unstable(); + assert_eq!(rows, want, "rel {a}.{lc} = rel {b}.{rc}"); + checked += 1; + } + } + } + } + assert_eq!(checked, 36); + // 36 joins in every orientation: no id was looked up again. + assert_eq!(map.lookups(), after_intake); +} + +#[test] +fn composing_views_runs_nothing() { + let rels: [&[(u32, u32)]; 3] = [&R, &S, &T]; + let table = table_over(&rels); + let mut cube = Cube::new(&table, &table.seal(), TILE, &rels).unwrap(); + let mut last = None; + for i in 0..50u32 { + let l = side(&mut cube, &table, i % 3, 1, Some(1)); + let r = cube.scan((i + 1) % 3).unwrap(); + last = Some(cube.join(l, 1, r, 0).unwrap()); + } + assert_eq!(cube.stats().programs, 0); + assert_eq!(cube.stats().rows_materialized, 0); + + cube.support(&table, last.unwrap()).unwrap(); + assert_eq!(cube.stats().programs, 2, "a support is two programs"); + assert_eq!( + cube.stats().rows_materialized, + 0, + "a support builds no rows" + ); + cube.bag_count(&table, last.unwrap()).unwrap(); + assert_eq!(cube.stats().programs, 4); + assert_eq!(cube.stats().rows_materialized, 0); +} + +#[test] +fn a_bound_outside_the_map_runs_nothing_on_its_side() { + let rels: [&[(u32, u32)]; 2] = [&R, &S]; + let table = table_over(&rels); + let mut cube = Cube::new(&table, &table.seal(), TILE, &rels).unwrap(); + let l = side(&mut cube, &table, 0, 1, Some(1000)); + let r = cube.scan(1).unwrap(); + let j = cube.join(l, 1, r, 0).unwrap(); + assert!(cube.support(&table, j).unwrap().iter().all(|w| *w == 0)); + assert_eq!(cube.stats().programs, 1, "only the unbound side ran"); +} + +/// Randomised: two relations over ids 0..12, every orientation, bound or not. +#[test] +fn differential_against_enumeration() { + let mut seed = 0x9E37_79B9_7F4A_7C15u64; + let mut next = |m: u32| { + seed ^= seed << 13; + seed ^= seed >> 7; + seed ^= seed << 17; + (seed % u64::from(m)) as u32 + }; + let mut nonempty = 0; + for _ in 0..300 { + let a: Vec<(u32, u32)> = (0..next(20) + 1).map(|_| (next(12), next(12))).collect(); + let b: Vec<(u32, u32)> = (0..next(20) + 1).map(|_| (next(12), next(12))).collect(); + let rels: [&[(u32, u32)]; 2] = [&a, &b]; + let table = table_over(&rels); + let mut cube = Cube::new(&table, &table.seal(), TILE, &rels).unwrap(); + let map = table.map(&table.seal(), TILE).unwrap(); + let (lc, rc) = (next(2) as u8, next(2) as u8); + let lb = (next(3) == 0).then(|| next(12)); + let rb = (next(3) == 0).then(|| next(12)); + let l = side(&mut cube, &table, 0, lc, lb); + let r = side(&mut cube, &table, 1, rc, rb); + let j = cube.join(l, lc, r, rc).unwrap(); + let want = oracle(&a, lc, lb, &b, rc, rb); + nonempty += usize::from(!want.is_empty()); + let got: BTreeMap = nonzero(map, &cube.bag_count(&table, j).unwrap()) + .into_iter() + .collect(); + assert_eq!(got, counts(&want)); + let sup: Vec = counts(&want).into_keys().collect(); + assert_eq!( + ids(map, &cube.support(&table, j).unwrap()) + .into_iter() + .collect::>(), + sup.into_iter().collect() + ); + let mut rows = cube.materialize_rows(&table, j).unwrap(); + rows.sort_unstable(); + assert_eq!(rows, want); + } + assert!(nonempty > 150, "differential mostly empty: {nonempty}"); +} + +#[test] +fn a_stale_frame_a_foreign_view_and_a_bound_join_column_are_refused() { + let rels: [&[(u32, u32)]; 2] = [&R, &S]; + let mut table = table_over(&rels); + let mut cube = Cube::new(&table, &table.seal(), TILE, &rels).unwrap(); + let mut other = Cube::new(&table, &table.seal(), TILE, &rels).unwrap(); + let l = cube.scan(0).unwrap(); + let r = cube.scan(1).unwrap(); + let j = cube.join(l, 1, r, 0).unwrap(); + + let foreign = other.scan(0).unwrap(); + assert_eq!(cube.join(foreign, 1, r, 0), Err(CubeRefusal::ForeignView)); + let bound = cube.bind(&table, l, 1, 2).unwrap(); + assert_eq!( + cube.join(bound, 1, r, 0), + Err(CubeRefusal::BoundOnJoinColumn) + ); + assert_eq!(cube.support(&table, l), Err(CubeRefusal::NotAJoin)); + + table.advance(1, 1, vec![]); + assert!(matches!( + cube.support(&table, j), + Err(CubeRefusal::Map(MapRead::Stale(_))) + )); + assert!(matches!( + cube.bag_count(&table, j), + Err(CubeRefusal::Map(MapRead::Stale(_))) + )); +} + +/// Two empty relations: an empty map, and every terminal answers empty +/// instead of refusing (mask-risc rejects a zero-length `I64` sink). +#[test] +fn an_empty_domain_answers_empty() { + let rels: [&[(u32, u32)]; 2] = [&[], &[]]; + let table = table_over(&rels); + let mut cube = Cube::new(&table, &table.seal(), TILE, &rels).unwrap(); + let l = cube.scan(0).unwrap(); + let r = cube.scan(1).unwrap(); + let j = cube.join(l, 1, r, 0).unwrap(); + assert_eq!(cube.bag_count(&table, j), Ok(vec![])); + assert_eq!(cube.support(&table, j), Ok(vec![])); + assert_eq!(cube.materialize_rows(&table, j), Ok(vec![])); +}