diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index ebf5636f0..b45a128f9 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -55,7 +55,7 @@ | D-DT64-6-3 | register OGIT verbs as DT64 predicates in 64-wide blocks | Queued | D-DT64-6-2 green | | D-DT64-6-4 | consumer read-only access via HotPlug activation | Queued | D-DT64-6-3 green | | D-DT64-6-5 | OGIT becomes cross-check oracle once a domain has attested predicates | Queued | one attested domain | -| D-DT64-6-BS | `crates/lance-graph-dt64`: the bind store (workspace member, contract-only dep). Closes D-DRDT §4 gaps 1–4 (MethodId + SigRev, K3 guard as `SetExpr` + `Logic`, ordered effects, one `resolve`); D5 immutable classid, derived classid + versioned route log; D-DRDT §5 falsifier | In PR (#1869) | 19 tests green, clippy/fmt clean; disable runs (Amendment A.5) | +| D-DT64-6-BS | `crates/lance-graph-dt64`: the bind store (workspace member, contract-only dep). Closes D-DRDT §4 gaps 1–4 (MethodId + SigRev, K3 guard as `SetExpr` + `Logic`, ordered effects, one `resolve`); D5 immutable classid, derived classid + versioned route log; D-DRDT §5 falsifier | **Shipped** #1869 | 19 tests green, clippy/fmt clean; disable runs (Amendment A.5) | ## D-DT64 — DTO Tensor 64: versioned semantic predicate adapter and DR projections (2026-10-26) @@ -90,7 +90,7 @@ | **D-ONNX-INTAKE-3** | S3 one complete static subgraph on existing native methods | Tract arm RUN as lab (D-TRACT-HARVEST-0): tiny graph exact, Jina RMSNorm slice bounded | scalar oracle + ORT; no Burn/Tract in the native call path; unsupported nodes refuse | | **D-ONNX-INTAKE-4** | S4 numeric/quantization parity for real model blockers | Queued (conditional; not authorized) | IntegerExact / bounded FP / SourceParity / CertifiedMath kept distinct | | **D-ONNX-INTAKE-5** | S5 whole-model forward pass and cost closure | Goal, not authorized | outputs vs ORT under declared tolerance; every cost stage reported | -| **D-TRACT-HARVEST-0** | ONNX graph as a DR#3 graph (nodes Identity, values Property, dep = do_ ∘ read_by) resolved through a DT64 ON×WHEN×DO registry (op signature → classid → native DO); Tract cold facts; ORT (optimizer off) oracle; `crates/tract-harvest-probe` | Lab probe implemented (this PR, redesigned to the graph); plan [d-tract-harvest-0-native-algebra-bridge-v1.md](../plans/d-tract-harvest-0-native-algebra-bridge-v1.md) §0 | 26 tests, 13 disable runs red; full model 4,146/7,620 nodes WHEN-true with inference-stage facts (§0.1), 0 refused for missing facts, >64 positions untiled, first unregistered groups Unsqueeze/Shape/Gather/Concat/MatMul | +| **D-TRACT-HARVEST-0** | ONNX graph as a DR#3 graph (nodes Identity, values Property, dep = do_ ∘ read_by) resolved through a DT64 ON×WHEN×DO registry (one ONNX lens classid; each op signature a field under it labelling a classid-free algebra address → native DO, #1870); Tract cold facts; ORT (optimizer off) oracle; `crates/tract-harvest-probe` | Lab probe implemented (this PR, redesigned to the graph); plan [d-tract-harvest-0-native-algebra-bridge-v1.md](../plans/d-tract-harvest-0-native-algebra-bridge-v1.md) §0 | 26 tests, 13 disable runs red; full model 4,146/7,620 nodes WHEN-true with inference-stage facts (§0.1), 0 refused for missing facts, >64 positions untiled, first unregistered groups Unsqueeze/Shape/Gather/Concat/MatMul | | **D-TRACT-SHAPE-1** | Cold shape-subgraph folding (Const/Shape/Symbolic/Data classes; fold through the node's own WHEN/DO; stale-input refusal), shape-op registrations (Shape, Gather i64, Unsqueeze, Concat, Reshape, Cast i64, Erf via `vserf`, scalar-Y Where via `blend_i32_scalar`), producer-checked Tract fact admission | **Shipped** #1853 + follow-up In PR (lab probe); plan [d-tract-harvest-0-native-algebra-bridge-v1.md](../plans/d-tract-harvest-0-native-algebra-bridge-v1.md) §0.2–§0.4 | #1853 merged (40 tests, 9 disable runs red; coverage 4,146/0/3,474 → 6,404/56/1,160). Follow-up **In PR**: registry v3 (int64 compares/Where, ConstantOfShape, Expand), optional-output producer test; coverage 6,581/114/925. Checked int64 follow-up **In PR** (plan §0.4): registry v4, int64 Add/Sub/Mul/Div/Range on ndarray #401; `i64_arith` folds and matches ORT; Jina 6,871/1/748, all shape-dependent Reshapes fold; remaining shape-only blockers are the rotary-embedding chain, And, one broadcast LessOrEqual. Rotary follow-up **In PR** (plan §0.5): registry v5 (Cast i64→f32/bool, MatMul f32, Transpose, Cos/Sin, And, any-broadcast compares), pinned external inv_freq; Jina rotary fold matches ORT ≤ 1 ULP; Jina 7,274/0/346, 0 shape-only nodes blocked (#1858 merged). Bind-store consumer follow-up **In PR** (plan §0.6): `bind::route` / `bind::resolve` split the registry into DR64 route and DT64 bind store; equal to `lookup` on all tiny fixtures; Jina 7,274 routed+resolved / 346 unrouted (#1865 merged). **Correction In PR** (plan §0.7): one classid is the ONNX codebook lens; op signatures are items `MethodKey { lens, ordinal }` (registry v6). **Field view** #1867 merged (plan §0.8); ONNX is one classid (`ONNX_LENS`), per-op ids purged from the plan text and from D-DT64-6 A.2 (In PR): registry v7 splits lens-agnostic algebra addresses (`methods`) from the lens's labelled fields (`regs`) and its `WideFieldMask` (`exposed`) | ## D-TFLP — TFLite models on the native polyfill: int8 codes as Palette256 (2026-10-25) diff --git a/.claude/plans/d-dt64-6-routing-bindstore-behavior-v1.md b/.claude/plans/d-dt64-6-routing-bindstore-behavior-v1.md index 3b6dcadc4..a9c64f655 100644 --- a/.claude/plans/d-dt64-6-routing-bindstore-behavior-v1.md +++ b/.claude/plans/d-dt64-6-routing-bindstore-behavior-v1.md @@ -291,3 +291,49 @@ The physical lane kind (u32 code vs i32, which picks XLOG's comparator) is the S - `MethodBinding::footprint()` is ON ∪ reads ∪ writes, and `Resolved::methods_touching(mask)` lists the methods a meaning mask reaches. A risk or RBAC mask thereby applies to behaviour, not only to fields. Open, for the frame owner (D8): registering a class or declaring an algebra entry moves no revision a frame can check; only routes are versioned. Classes and entries are immutable and append-only, so nothing a frame read can change. Whether a frame should still record the store's catalogue extent (for example a count of bindings) is undecided. + +**A.5 addendum 7 (2026-10-30, after #1869 and #1870 merged).** #1870 settles addendum 1's open question on tract-harvest-probe's per-op classids: there is one ONNX lens classid (`ONNX_LENS`), and each op signature is a field under it labelling a classid-free algebra address. That is the shape this crate implements: the lens is a registered class, its fields come from its `ClassView`, and each method's effects reference `AlgebraEntry`s by `AlgebraRef`. Addendum 1's `EffectOp::Call` text is superseded by addendum 3. Still open: whether tract's `MethodKey { lens, ordinal }` and this crate's `MethodRef { class, method }` are one identity (EXO-50); the frame extent question above. + +**A.5 addendum 8 (2026-10-30, operator: "ever heard of polyfill?").** The question "who declares the catalogue entries and assigns their ids" had an answer all along: the polyfill. In `ndarray::simd` a consumer names a facade word and nothing else. Nobody declares the word to the consumer and nobody assigns it a number, and which backend realizes it is decided per build. So: +- `AlgebraId` is removed. `AlgebraRef` is the word's facade path; there is no second, numeric spelling. +- The per-entry `SigRev` and `AlgebraRevisionMismatch` are removed. A changed realization keeps the word, and a different operation is a different word. +- `AlgebraEntry { word, inputs }` only mirrors the word's arity so `ArityMismatch` can be checked. The polyfill decides what the word is; the store mirrors it, and a word mirrored with two arities is refused (`AlgebraImmutable`). + +Which polyfill (operator: "check lance-graph-relalg and lance-graph-mask-risc, that's the polyfill"): `lance-graph-relalg` describes a selection and lowers it, `lance-graph-mask-risc`'s op vocabulary (`Pred`, `MaskOp`, `Terminal`) is the one polyfill above `ndarray::simd` that every fold lowers to, and `ndarray::simd` computes. So a keyed count names mask-risc's `Terminal::GroupReduce { key: GroupKey::Lane, fold: GroupFold::Count }`, not the `ndarray::simd` word it delegates to. mask-risc has no elementwise i64 arithmetic, so XLOG's multiply names an `ndarray::simd` word. That word was reachable only as `ndarray::hpc::checked_i64::checked_binary_i64`; a consumer never names `hpc`, so ndarray #407 re-exports it through `ndarray::simd`. Disable runs (on commit `d140ab4c8`): removing the `AlgebraImmutable` arm or the `UnknownAlgebra` check each leaves exactly one test red (18 passed, 1 failed). + +**A.5 addendum 9 (2026-10-30, operator: "your DTO mapping is one classid one meaning surface, same as ClassView WideFieldMask; did you use the pattern").** No: addendum 8 kept the polyfill as a string-keyed catalogue beside `ClassView`, a second meaning surface. Now the polyfill vocabulary is one lens classid, read the way every class is read: +- Its words are the fields of that lens, and their labels (the mask-risc op or `ndarray::simd` word) come from its `ClassView`. +- A word is a field position, `AlgebraRef { lens, word }`, and a set of words is a `WideFieldMask`. +- `declare_algebra(lens, schema_rev, arity)` binds the lens with one operand count per word. It is immutable like a class (`AlgebraImmutable`). +- `resolve_algebra` reads the lens through its `ClassView`, with the same schema-revision and field-basis checks as `resolve` (one shared `check_basis`). `field_view` gives each word's position and label. +- `words_used(class, lens)` gives a consumer's words as a mask over the lens. + +No word gets a classid. This is the ONNX shape of #1870, applied to the polyfill. Disable runs on `9d2fba457`, each turning tests red: + +| guard removed | failing tests | +|---|---| +| the `words_used` lens filter | 1 | +| the `AlgebraImmutable` arm | 1 | +| the arity position | 5 | +| the shared basis check | 4 | + +Which classid the polyfill lens carries in production is OGAR's mint (D6), not this crate's; the test uses a lab value. + +**A.5 addendum 10 (2026-10-30, operator: "make sure that consumers like ONNX are not limited to 64 but instead can choose the other modes like DR256² × 64").** A.1 already gives the binding this job: it names DR64 or DR256 and states its cost in those units. Until now the store only capped a class at 256 fields. `ClassSchema` now carries an `Extent` the consumer chooses: +- The extent is Identity × Property × Predicate, each axis `Width::W64` or `Width::W256`. `Extent::DR64_CUBE` and `Extent::DR256_SQ_X64` are named, and any other combination is allowed. +- Registration refuses fields past the property axis (`FieldsExceedExtent`) and methods past the predicate axis (`MethodsExceedExtent`). +- The extent is part of the immutable binding (D5). +- `dense_bytes` states what a dense Boolean volume of the extent would cost if anyone materialized one: DR64³ is 32 KiB and DR256² × 64 is 512 KiB. It is a cost statement and allocates nothing (§7). +- The polyfill lens declares its extent too. + +The test registers an ONNX graph with 200 values: DR64³ refuses it, naming the axis, and DR256² × 64 binds it. A graph with 70 nodes needs a 256-wide predicate axis. + +Disable runs on the extent commit: + +| guard removed | failing tests | +|---|---| +| the property-axis check | 1 | +| the predicate-axis check | 1 | +| W256 read as 64 | 2 | + +Still capped at 64: `tract-harvest-probe`'s `dr3::TILE = 64`, a single DR64 tile built from `relcomp::Rows = [u64; 64]`. That is the tract session's lab crate. Its full model (4,146 of 7,620 nodes WHEN-true) already reports ">64 positions untiled". Lifting it is that session's change; the binding side no longer forces 64. diff --git a/crates/lance-graph-contract/src/morton8x8.rs b/crates/lance-graph-contract/src/morton8x8.rs index 7f63c8c20..cd21f1bf6 100644 --- a/crates/lance-graph-contract/src/morton8x8.rs +++ b/crates/lance-graph-contract/src/morton8x8.rs @@ -295,7 +295,11 @@ mod tests { for start in (0..=u16::MAX as u32).step_by(run as usize) { let first = Morton8x8::from_code(start as u16); let (x0, y0) = (u32::from(first.x()), u32::from(first.y())); - assert_eq!((x0 % side, y0 % side), (0, 0), "j={j} run {start}: corner not aligned"); + assert_eq!( + (x0 % side, y0 % side), + (0, 0), + "j={j} run {start}: corner not aligned" + ); let mut seen = vec![false; run as usize]; for c in start..start + run { let m = Morton8x8::from_code(c as u16); diff --git a/crates/lance-graph-dt64/src/lib.rs b/crates/lance-graph-dt64/src/lib.rs index 7ce4f4b70..e37916c87 100644 --- a/crates/lance-graph-dt64/src/lib.rs +++ b/crates/lance-graph-dt64/src/lib.rs @@ -56,12 +56,20 @@ //! //! # Agnostic algebra, switched by the classid //! -//! Addresses and algebra methods are agnostic. The algebra catalogue -//! ([`AlgebraEntry`], [`BindStore::declare_algebra`]) is classid-free global -//! behaviour (D-DT64-6 D6). A classid switches labels (its [`ClassView`]) and -//! polyfill (its bindings' [`Effect`]s, each an [`AlgebraRef`] with ordered -//! operands) over them. Two consumers reuse one entry by referring to the -//! same [`AlgebraId`]; [`BindStore::users_of`] lists them. +//! Addresses and algebra methods are agnostic. The algebra is the polyfill +//! (D-DT64-6 D4), classid-free global behaviour (D6): `lance-graph-relalg` +//! describes, `lance-graph-mask-risc`'s op vocabulary folds, `ndarray::simd` +//! computes. The polyfill vocabulary is one meaning surface like any other: one +//! lens classid, its words are the fields of that lens, their labels come +//! from its [`ClassView`], and a word is a field position ([`AlgebraRef`]), so +//! a set of words is a [`WideFieldMask`]. No word gets a classid of its own. +//! The store binds the lens ([`BindStore::declare_algebra`]: schema revision +//! and each word's operand count) so a binding can be checked; it decides +//! nothing about what a word does. A consumer classid switches labels (its +//! [`ClassView`]) and polyfill (its bindings' [`Effect`]s, each a word with +//! ordered operands) over them. Two consumers reuse a word by naming its +//! position; [`BindStore::users_of`] lists them, and +//! [`BindStore::words_used`] gives a class's words as a mask over the lens. //! //! [`ActionDef::object_class`]: lance_graph_contract::action::ActionDef::object_class //! [`ClassMethods::classid`]: lance_graph_contract::codegen_manifest::ClassMethods::classid @@ -95,45 +103,26 @@ pub struct SchemaRev(pub u32); #[derive(Debug, Clone, Copy, Default, PartialEq, Eq, PartialOrd, Ord, Hash)] pub struct RouteVersion(pub u64); -/// A classid-free algebra method: global behaviour (D-DT64-6 D6, "global -/// behaviour needs no classid and lives in lance-graph"). Addresses and -/// algebra methods are agnostic; a classid's binding is what switches labels -/// and polyfill over them. +/// A polyfill word: a field position under the polyfill's lens classid. The +/// word's name (a mask-risc op, an `ndarray::simd` facade word) is that +/// field's label in the lens's [`ClassView`], resolved late like any other +/// label; the store never resolves or runs the word. #[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)] -pub struct AlgebraId(pub u32); - -/// One entry of the algebra catalogue: the polyfill (D4: over mask-risc / -/// ndarray) named by reference, how many operands it takes, and its signature -/// revision. Immutable once declared; versioned in the store directly (D6). -/// The store never resolves or runs the polyfill. -#[derive(Debug, Clone, PartialEq, Eq, Hash)] -pub struct AlgebraEntry { - pub id: AlgebraId, - pub sig_rev: SigRev, - pub polyfill: String, - pub inputs: u8, -} - -/// What a class binds an effect to: an algebra entry at the signature -/// revision it was bound against. -#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] pub struct AlgebraRef { - pub id: AlgebraId, - pub sig_rev: SigRev, + pub lens: u32, + pub word: u8, } -impl AlgebraRef { - /// The faithful reference to `entry`. - pub fn of(entry: &AlgebraEntry) -> AlgebraRef { - AlgebraRef { - id: entry.id, - sig_rev: entry.sig_rev, - } - } +/// A bound polyfill lens: its schema, as for any class, and the operand count +/// of each word (one per field position). +#[derive(Debug, Clone, PartialEq, Eq)] +struct AlgebraLens { + schema: ClassSchema, + arity: Vec, } -/// One DO step: which field position of the class it writes, the algebra -/// entry that writes it, and what it reads. A method's effects are an ORDERED +/// One DO step: which field position of the class it writes, the polyfill +/// word that writes it, and what it reads. A method's effects are an ORDERED /// list. #[derive(Debug, Clone, PartialEq, Eq, Hash)] pub struct Effect { @@ -239,13 +228,70 @@ impl MethodBinding { } } -/// A class's schema as bound: the schema revision and the field count the -/// methods were bound against. The field basis itself is the [`ClassView`] -/// class [`view_class`] of the classid, never declared here. +/// Positions per axis of a DR focus (D-DRF-1 §1): DR64 or DR256. +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub enum Width { + W64, + W256, +} + +impl Width { + pub const fn positions(self) -> usize { + match self { + Width::W64 => 64, + Width::W256 => 256, + } + } +} + +/// The logical extent a class binds over, chosen by the consumer per axis: +/// Identity × Property × Predicate. D-DT64-6 A.1: the DT binding names DR64 +/// or DR256 and states its cost in those units. Logical only: nothing is +/// allocated here, and no cube is assumed to exist (§7). +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +pub struct Extent { + pub identity: Width, + pub property: Width, + /// The class's predicates: its methods (ON × WHEN × DO registrations). + pub predicate: Width, +} + +impl Extent { + /// DR64³: 64 × 64 × 64. + pub const DR64_CUBE: Extent = Extent { + identity: Width::W64, + property: Width::W64, + predicate: Width::W64, + }; + /// DR256² × 64: a 256 × 256 Identity × Property tile under 64 predicates. + pub const DR256_SQ_X64: Extent = Extent { + identity: Width::W256, + property: Width::W256, + predicate: Width::W64, + }; + + /// Logical cells. + pub const fn cells(self) -> usize { + self.identity.positions() * self.property.positions() * self.predicate.positions() + } + + /// What a dense Boolean volume of this extent would cost, in bytes, if + /// anyone materialized it (DR64³ is 32 KiB, D-DRF-1 §1). A statement of + /// cost, not an allocation. + pub const fn dense_bytes(self) -> usize { + self.cells() / 8 + } +} + +/// A class's schema as bound: the schema revision, the field count the +/// methods were bound against, and the extent the consumer chose. The field +/// basis itself is the [`ClassView`] class [`view_class`] of the classid, +/// never declared here. #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] pub struct ClassSchema { pub schema_rev: SchemaRev, pub field_count: usize, + pub extent: Extent, } /// The [`ClassView`] class a classid is read through: its concept half, by @@ -307,6 +353,18 @@ pub enum Refusal { }, /// The declared field count exceeds [`MAX_FIELDS`]. BasisTooWide { class: u32, field_count: usize }, + /// More fields than the extent's Property axis holds. + FieldsExceedExtent { + class: u32, + field_count: usize, + property: Width, + }, + /// More methods than the extent's Predicate axis holds. + MethodsExceedExtent { + class: u32, + methods: usize, + predicate: Width, + }, /// Asked at another schema revision than the one bound. SchemaRevisionMismatch { class: u32, @@ -334,25 +392,17 @@ pub enum Refusal { GuardAdded { class: u32, method: MethodId }, /// The reference carries another guard than the bound one. GuardMismatch { class: u32, method: MethodId }, - /// An effect names an algebra entry the catalogue does not hold. + /// An effect names a lens that is not bound, or a position outside it. UnknownAlgebra { class: u32, method: MethodId, - algebra: AlgebraId, - }, - /// An effect was bound against another revision of the entry. - AlgebraRevisionMismatch { - class: u32, - method: MethodId, - algebra: AlgebraId, - bound: SigRev, - asked: SigRev, + algebra: AlgebraRef, }, /// An effect passes another number of operands than the entry takes. ArityMismatch { class: u32, method: MethodId, - algebra: AlgebraId, + algebra: AlgebraRef, expected: u8, got: usize, }, @@ -364,8 +414,8 @@ pub enum Refusal { effect: usize, position: u8, }, - /// The algebra entry is already declared with other content. - AlgebraImmutable(AlgebraId), + /// The polyfill lens is already bound with other content. + AlgebraImmutable(u32), /// No classid was routed to the position at that version. Unrouted { position: u16, @@ -512,8 +562,8 @@ impl<'s, 'v> Resolved<'s, 'v> { #[derive(Debug, Clone, Default)] pub struct BindStore { classes: BTreeMap, - /// The classid-free algebra catalogue (D6). - algebra: BTreeMap, + /// The bound polyfill lenses, by lens classid (D6). + algebra: BTreeMap, /// Per position, the route events in version order: `(version, class)`, /// `None` for a retract. routes: BTreeMap)>>, @@ -564,27 +614,18 @@ impl BindStore { validate(class, &schema, &methods)?; for m in &methods { for e in &m.effects { - let (method, algebra) = (m.id, e.algebra.id); - let entry = self.algebra.get(&algebra).ok_or(Refusal::UnknownAlgebra { + let (method, algebra) = (m.id, e.algebra); + let inputs = self.arity(algebra).ok_or(Refusal::UnknownAlgebra { class, method, algebra, })?; - if entry.sig_rev != e.algebra.sig_rev { - return Err(Refusal::AlgebraRevisionMismatch { - class, - method, - algebra, - bound: entry.sig_rev, - asked: e.algebra.sig_rev, - }); - } - if usize::from(entry.inputs) != e.args.len() { + if usize::from(inputs) != e.args.len() { return Err(Refusal::ArityMismatch { class, method, algebra, - expected: entry.inputs, + expected: inputs, got: e.args.len(), }); } @@ -605,31 +646,86 @@ impl BindStore { } } - /// Declare a classid-free algebra entry. Idempotent for an identical - /// entry; any other content under a declared id is refused. - pub fn declare_algebra(&mut self, entry: AlgebraEntry) -> Result<(), Refusal> { - match self.algebra.get(&entry.id) { - Some(bound) if *bound == entry => Ok(()), - Some(_) => Err(Refusal::AlgebraImmutable(entry.id)), + /// Bind a polyfill lens: one classid whose fields are the polyfill's + /// words, `arity[p]` being the operand count of the word at position `p`. + /// Idempotent for an identical binding; any other is refused (D5). + pub fn declare_algebra( + &mut self, + lens: u32, + schema_rev: SchemaRev, + extent: Extent, + arity: Vec, + ) -> Result<(), Refusal> { + let schema = ClassSchema { + schema_rev, + field_count: arity.len(), + extent, + }; + validate(lens, &schema, &[])?; + let bound = AlgebraLens { schema, arity }; + match self.algebra.get(&lens) { + Some(b) if *b == bound => Ok(()), + Some(_) => Err(Refusal::AlgebraImmutable(lens)), None => { - self.algebra.insert(entry.id, entry); + self.algebra.insert(lens, bound); Ok(()) } } } - /// The algebra entry `id`, if declared. - pub fn algebra(&self, id: AlgebraId) -> Option<&AlgebraEntry> { - self.algebra.get(&id) + /// The operand count of `word`, if its lens is bound and the position is + /// inside it. + pub fn arity(&self, word: AlgebraRef) -> Option { + self.algebra + .get(&word.lens)? + .arity + .get(usize::from(word.word)) + .copied() + } + + /// The polyfill lens read through its [`ClassView`], exactly as a class + /// is: [`Resolved::field_view`] gives each word's position and label. It + /// has no methods; its words are its fields. + pub fn resolve_algebra<'s, 'v, V: ClassView + ?Sized>( + &'s self, + view: &'v V, + lens: u32, + schema_rev: SchemaRev, + ) -> Result, Refusal> { + let bound = self.algebra.get(&lens).ok_or(Refusal::UnknownClass(lens))?; + check_basis(view, lens, &bound.schema, schema_rev).map(|fields| Resolved { + class: lens, + schema: &bound.schema, + fields, + methods: &[], + derived_from: None, + }) + } + + /// The words of `lens` that `class`'s methods use, as a mask over the + /// lens. + pub fn words_used(&self, class: u32, lens: u32) -> Result { + let entry = self + .classes + .get(&class) + .ok_or(Refusal::UnknownClass(class))?; + let words: Vec = entry + .methods + .iter() + .flat_map(|m| &m.effects) + .filter(|e| e.algebra.lens == lens) + .map(|e| e.algebra.word) + .collect(); + Ok(WideFieldMask::from_positions(&words)) } - /// Every `(classid, method)` whose DO uses algebra entry `id`: which - /// classids switch over it. - pub fn users_of(&self, id: AlgebraId) -> Vec<(u32, MethodId)> { + /// Every `(classid, method)` whose DO uses `word`: which classids switch + /// over it. + pub fn users_of(&self, word: AlgebraRef) -> Vec<(u32, MethodId)> { let mut out = Vec::new(); for (&class, entry) in &self.classes { for m in &entry.methods { - if m.effects.iter().any(|e| e.algebra.id == id) { + if m.effects.iter().any(|e| e.algebra == word) { out.push((class, m.id)); } } @@ -698,21 +794,7 @@ impl BindStore { .classes .get(&class) .ok_or(Refusal::UnknownClass(class))?; - if entry.schema.schema_rev != schema_rev { - return Err(Refusal::SchemaRevisionMismatch { - class, - bound: entry.schema.schema_rev, - asked: schema_rev, - }); - } - let fields = view.fields(view_class(class)); - if fields.len() != entry.schema.field_count { - return Err(Refusal::FieldBasisChanged { - class, - bound: entry.schema.field_count, - view: fields.len(), - }); - } + let fields = check_basis(view, class, &entry.schema, schema_rev)?; Ok(Resolved { class, schema: &entry.schema, @@ -769,9 +851,36 @@ impl BindStore { } } -/// Registration checks: the basis fits a [`WideFieldMask`], MethodIds are -/// unique in the class, and every field position a binding names is inside -/// the declared basis. +/// The lens's fields from the [`ClassView`], refused when the schema revision +/// or the field basis is not the bound one. +fn check_basis<'v, V: ClassView + ?Sized>( + view: &'v V, + class: u32, + schema: &ClassSchema, + schema_rev: SchemaRev, +) -> Result<&'v [FieldRef], Refusal> { + if schema.schema_rev != schema_rev { + return Err(Refusal::SchemaRevisionMismatch { + class, + bound: schema.schema_rev, + asked: schema_rev, + }); + } + let fields = view.fields(view_class(class)); + if fields.len() != schema.field_count { + return Err(Refusal::FieldBasisChanged { + class, + bound: schema.field_count, + view: fields.len(), + }); + } + Ok(fields) +} + +/// Registration checks: the basis fits a [`WideFieldMask`] and the chosen +/// extent, the methods fit the extent's predicate axis, MethodIds are unique +/// in the class, and every field position a binding names is inside the +/// declared basis. fn validate(class: u32, schema: &ClassSchema, methods: &[MethodBinding]) -> Result<(), Refusal> { if schema.field_count > MAX_FIELDS { return Err(Refusal::BasisTooWide { @@ -779,6 +888,21 @@ fn validate(class: u32, schema: &ClassSchema, methods: &[MethodBinding]) -> Resu field_count: schema.field_count, }); } + let extent = schema.extent; + if schema.field_count > extent.property.positions() { + return Err(Refusal::FieldsExceedExtent { + class, + field_count: schema.field_count, + property: extent.property, + }); + } + if methods.len() > extent.predicate.positions() { + return Err(Refusal::MethodsExceedExtent { + class, + methods: methods.len(), + predicate: extent.predicate, + }); + } for (i, m) in methods.iter().enumerate() { if methods[..i].iter().any(|p| p.id == m.id) { return Err(Refusal::DuplicateMethodId { diff --git a/crates/lance-graph-dt64/tests/bind_store.rs b/crates/lance-graph-dt64/tests/bind_store.rs index 42b94e66e..9b231c8c0 100644 --- a/crates/lance-graph-dt64/tests/bind_store.rs +++ b/crates/lance-graph-dt64/tests/bind_store.rs @@ -6,8 +6,8 @@ use lance_graph_contract::ontology::{DisplayTemplate, FieldRef}; use lance_graph_contract::set_algebra::{Cmp, Logic, SetExpr}; use lance_graph_contract::set_rules::{Population, Truth}; use lance_graph_dt64::{ - view_class, AlgebraEntry, AlgebraId, AlgebraRef, Arg, BindStore, ClassSchema, Effect, Guard, - MethodBinding, MethodId, MethodRef, Refusal, RouteVersion, SchemaRev, SigRev, ValueDemand, + view_class, AlgebraRef, Arg, BindStore, ClassSchema, Effect, Extent, Guard, MethodBinding, + MethodId, MethodRef, Refusal, RouteVersion, SchemaRev, SigRev, ValueDemand, Width, }; /// A ClassView with one class of `n` fields; `n` can be changed to model a @@ -26,6 +26,16 @@ impl View { .collect(), } } + + fn with_labels(class: ClassId, labels: &[&str]) -> View { + View { + class, + fields: labels + .iter() + .map(|l| FieldRef::new(format!("urn:{l}"), (*l).to_string())) + .collect(), + } + } } impl ClassView for View { @@ -49,63 +59,58 @@ const ORDER_V2: u32 = 0x0A1E_0002; /// The lens ORDER is read through: its concept half (`view_class`). const VIEW_CLASS: ClassId = 0x0A1E; +/// A schema in the DR64³ extent. fn schema(rev: u32, fields: usize) -> ClassSchema { + schema_in(rev, fields, Extent::DR64_CUBE) +} + +fn schema_in(rev: u32, fields: usize, extent: Extent) -> ClassSchema { ClassSchema { schema_rev: SchemaRev(rev), field_count: fields, + extent, } } -/// The classid-free algebra catalogue the tests bind against (lab ids). -const ASSIGN: AlgebraId = AlgebraId(1); -const ACCUMULATE: AlgebraId = AlgebraId(2); -const GROUP_COUNT: AlgebraId = AlgebraId(10); -const MUL_I64: AlgebraId = AlgebraId(11); - -fn entry(id: AlgebraId, polyfill: &str, inputs: u8) -> AlgebraEntry { - AlgebraEntry { - id, - sig_rev: SigRev(1), - polyfill: polyfill.to_string(), - inputs, +/// The polyfill vocabulary is one lens classid (lab), its words the fields +/// of that lens. No word has a classid of its own. +const POLYFILL: u32 = 0x7F0A_0001; +/// The words' labels, by position: what the polyfill lens's ClassView holds. +/// relalg describes, mask-risc folds, `ndarray::simd` computes; the two +/// `lab::` words stand in for any nullary step in the ordering tests. +const WORDS: [&str; 4] = [ + "lab::assign", + "lab::accumulate", + // mask-risc's keyed count, COUNT(*) GROUP BY key: (relation, key column) + "lance_graph_mask_risc::Terminal::GroupReduce(GroupKey::Lane, GroupFold::Count)", + // ndarray #407: checked i64 multiply on the ndarray::simd facade + "ndarray::simd::checked_binary_i64(I64Op::Mul)", +]; +const ARITY: [u8; 4] = [0, 0, 2, 2]; + +const fn word(word: u8) -> AlgebraRef { + AlgebraRef { + lens: POLYFILL, + word, } } +const ASSIGN: AlgebraRef = word(0); +const ACCUMULATE: AlgebraRef = word(1); +const GROUP_COUNT: AlgebraRef = word(2); +const MUL_I64: AlgebraRef = word(3); -/// A store holding only the algebra catalogue: no classid anywhere. +/// A store holding only the polyfill lens: one classid, no consumer. fn catalogue() -> BindStore { let mut s = BindStore::new(); - for e in [ - entry(ASSIGN, "assign", 0), - entry(ACCUMULATE, "accumulate", 0), - // (relation field, key column) - entry(GROUP_COUNT, "ndarray::simd::masked_group_count_u32", 2), - entry( - MUL_I64, - "ndarray::hpc::checked_i64::checked_binary_i64(Mul)", - 2, - ), - ] { - s.declare_algebra(e).unwrap(); - } + s.declare_algebra(POLYFILL, SchemaRev(1), Extent::DR64_CUBE, ARITY.to_vec()) + .unwrap(); s } -fn uses(id: AlgebraId) -> AlgebraRef { - AlgebraRef { - id, - sig_rev: SigRev(1), - } -} - -fn effect(target: u8, algebra: &str) -> Effect { - let id = match algebra { - "assign" => ASSIGN, - "accumulate" => ACCUMULATE, - other => panic!("no lab entry {other}"), - }; +fn effect(target: u8, algebra: AlgebraRef) -> Effect { Effect { target, - algebra: uses(id), + algebra, args: vec![], } } @@ -132,10 +137,7 @@ fn store_with_order() -> BindStore { s.register( ORDER, schema(1, 4), - vec![method( - 10, - vec![effect(2, "assign"), effect(3, "accumulate")], - )], + vec![method(10, vec![effect(2, ASSIGN), effect(3, ACCUMULATE)])], ) .unwrap(); s @@ -173,16 +175,10 @@ fn an_unregistered_class_is_unknown_not_empty() { #[test] fn class_behaviour_is_immutable() { let mut s = store_with_order(); - let same = vec![method( - 10, - vec![effect(2, "assign"), effect(3, "accumulate")], - )]; + let same = vec![method(10, vec![effect(2, ASSIGN), effect(3, ACCUMULATE)])]; assert_eq!(s.register(ORDER, schema(1, 4), same), Ok(())); - let reordered = vec![method( - 10, - vec![effect(3, "accumulate"), effect(2, "assign")], - )]; + let reordered = vec![method(10, vec![effect(3, ACCUMULATE), effect(2, ASSIGN)])]; assert_eq!( s.register(ORDER, schema(1, 4), reordered), Err(Refusal::BehaviourImmutable(ORDER)) @@ -206,10 +202,7 @@ fn override_is_a_derived_class_and_a_reroute_and_replay_reads_the_old_route() { ORDER, ORDER_V2, schema(1, 4), - vec![method( - 11, - vec![effect(3, "accumulate"), effect(2, "assign")], - )], + vec![method(11, vec![effect(3, ACCUMULATE), effect(2, ASSIGN)])], ) .unwrap(); let v2 = s.route(7, ORDER_V2).unwrap(); @@ -278,8 +271,8 @@ fn routes_and_derivations_need_bound_classes() { #[test] fn same_read_write_sets_different_effect_order_stay_distinct_and_a_dropped_guard_refuses() { let mut s = catalogue(); - let a = method(20, vec![effect(2, "assign"), effect(2, "accumulate")]); - let b = method(21, vec![effect(2, "accumulate"), effect(2, "assign")]); + let a = method(20, vec![effect(2, ASSIGN), effect(2, ACCUMULATE)]); + let b = method(21, vec![effect(2, ACCUMULATE), effect(2, ASSIGN)]); // The negative mold: equal unordered read and write sets. assert_eq!(a.reads, b.reads); assert_eq!(a.writes(), b.writes()); @@ -300,8 +293,8 @@ fn same_read_write_sets_different_effect_order_stay_distinct_and_a_dropped_guard // Each faithful reference admits its own binding with its own order. let ra = MethodRef::of(ORDER, &a); let rb = MethodRef::of(ORDER, &b); - assert_eq!(s.admit(&ra).unwrap().effects[0].algebra.id, ASSIGN); - assert_eq!(s.admit(&rb).unwrap().effects[0].algebra.id, ACCUMULATE); + assert_eq!(s.admit(&ra).unwrap().effects[0].algebra, ASSIGN); + assert_eq!(s.admit(&rb).unwrap().effects[0].algebra, ACCUMULATE); // A route that dropped the guard witness refuses. let dropped = MethodRef { @@ -349,7 +342,7 @@ fn a_guard_added_to_an_unguarded_method_refuses() { let mut s = catalogue(); let free = MethodBinding { when: None, - ..method(30, vec![effect(2, "assign")]) + ..method(30, vec![effect(2, ASSIGN)]) }; s.register(ORDER, schema(1, 4), vec![free.clone()]).unwrap(); assert!(s.admit(&MethodRef::of(ORDER, &free)).is_ok()); @@ -426,7 +419,7 @@ fn registration_refuses_ambiguity_and_out_of_basis_positions() { s.register( ORDER, schema(1, 4), - vec![method(1, vec![]), method(1, vec![effect(2, "assign")])] + vec![method(1, vec![]), method(1, vec![effect(2, ASSIGN)])] ), Err(Refusal::DuplicateMethodId { class: ORDER, @@ -437,7 +430,7 @@ fn registration_refuses_ambiguity_and_out_of_basis_positions() { s.register( ORDER, schema(1, 4), - vec![method(1, vec![effect(4, "assign")])] + vec![method(1, vec![effect(4, ASSIGN)])] ), Err(Refusal::FieldOutOfBasis { class: ORDER, @@ -470,8 +463,8 @@ fn registration_refuses_ambiguity_and_out_of_basis_positions() { assert_eq!( s.register( ORDER, - schema(1, 256), - vec![method(1, vec![effect(255, "assign")])] + schema_in(1, 256, Extent::DR256_SQ_X64), + vec![method(1, vec![effect(255, ASSIGN)])] ), Ok(()) ); @@ -543,18 +536,18 @@ fn xlog_count() -> MethodBinding { // R keyed by its y column (1) Effect { target: 2, - algebra: uses(GROUP_COUNT), + algebra: GROUP_COUNT, args: vec![Arg::Field(0), Arg::Const(1)], }, // S keyed by its y column (0) Effect { target: 3, - algebra: uses(GROUP_COUNT), + algebra: GROUP_COUNT, args: vec![Arg::Field(1), Arg::Const(0)], }, Effect { target: 4, - algebra: uses(MUL_I64), + algebra: MUL_I64, args: vec![Arg::Field(2), Arg::Field(3)], }, ], @@ -570,7 +563,7 @@ fn onnx_mul() -> MethodBinding { name: "Mul(int64)".to_string(), effects: vec![Effect { target: 2, - algebra: uses(MUL_I64), + algebra: MUL_I64, args: vec![Arg::Field(0), Arg::Field(1)], }], population: Population::Distinct, @@ -584,17 +577,39 @@ fn consumers_switch_over_shared_classid_free_algebra() { s.register(XLOG, schema(1, 5), vec![xlog_count()]).unwrap(); s.register(ONNX, schema(1, 3), vec![onnx_mul()]).unwrap(); - // One algebra entry, reused by both classids; nothing minted for it. + // One word, reused by both classids by position; nothing minted for it. assert_eq!( s.users_of(MUL_I64), vec![(XLOG, MethodId(1)), (ONNX, MethodId(1))] ); assert_eq!(s.users_of(GROUP_COUNT), vec![(XLOG, MethodId(1))]); assert!(s.users_of(ASSIGN).is_empty()); - assert_eq!(s.class_count(), 2, "only the consumers carry classids"); assert_eq!( - s.algebra(MUL_I64).unwrap().polyfill, - "ndarray::hpc::checked_i64::checked_binary_i64(Mul)" + s.class_count(), + 2, + "the consumers; the polyfill is one lens" + ); + assert_eq!(s.arity(MUL_I64), Some(2)); + // Which words each consumer uses, as a mask over the polyfill lens. + assert_eq!( + s.words_used(XLOG, POLYFILL), + Ok(WideFieldMask::from_positions(&[2, 3])) + ); + assert_eq!( + s.words_used(ONNX, POLYFILL), + Ok(WideFieldMask::from_positions(&[3])) + ); + // Another lens's mask is empty: the mask is over one lens only. + s.declare_algebra( + 0x7F0B_0001, + SchemaRev(1), + Extent::DR64_CUBE, + vec![2, 2, 2, 2], + ) + .unwrap(); + assert_eq!( + s.words_used(XLOG, 0x7F0B_0001), + Ok(WideFieldMask::from_positions(&[])) ); // The classid switches the labels: each reads its own lens. @@ -608,19 +623,38 @@ fn consumers_switch_over_shared_classid_free_algebra() { ); } -/// An algebra entry is immutable once declared (D5 applied to D6). +/// The polyfill lens is read like any class: one classid, one ClassView, the +/// words as fields, a set of words as a WideFieldMask. Its binding is +/// immutable like a class's. #[test] -fn an_algebra_entry_is_immutable() { +fn the_polyfill_is_one_lens_read_through_its_classview() { let mut s = catalogue(); - let same = entry( - MUL_I64, - "ndarray::hpc::checked_i64::checked_binary_i64(Mul)", - 2, + let view = View::with_labels(view_class(POLYFILL), &WORDS); + let lens = s.resolve_algebra(&view, POLYFILL, SchemaRev(1)).unwrap(); + let rows: Vec<(u8, &str)> = lens + .field_view(&WideFieldMask::from_positions(&[2, 3])) + .map(|r| (r.position, r.label)) + .collect(); + assert_eq!(rows, vec![(2, WORDS[2]), (3, WORDS[3])]); + assert_eq!(lens.methods.len(), 0, "its words are fields, not methods"); + // A ClassView whose basis moved under the lens is refused, as for a class. + let moved = View::with_labels(view_class(POLYFILL), &[WORDS[0]]); + assert_eq!( + s.resolve_algebra(&moved, POLYFILL, SchemaRev(1)) + .unwrap_err(), + Refusal::FieldBasisChanged { + class: POLYFILL, + bound: 4, + view: 1 + } ); - assert_eq!(s.declare_algebra(same), Ok(())); assert_eq!( - s.declare_algebra(entry(MUL_I64, "wrapping_mul", 2)), - Err(Refusal::AlgebraImmutable(MUL_I64)) + s.declare_algebra(POLYFILL, SchemaRev(1), Extent::DR64_CUBE, ARITY.to_vec()), + Ok(()) + ); + assert_eq!( + s.declare_algebra(POLYFILL, SchemaRev(1), Extent::DR64_CUBE, vec![0, 0, 2, 3]), + Err(Refusal::AlgebraImmutable(POLYFILL)) ); } @@ -661,21 +695,11 @@ fn effect_operands_and_entries_are_checked() { } ); assert_eq!( - refused(&mut s, &|e| e.algebra = uses(AlgebraId(99))), + refused(&mut s, &|e| e.algebra = word(4)), Refusal::UnknownAlgebra { class: XLOG, method: MethodId(1), - algebra: AlgebraId(99) - } - ); - assert_eq!( - refused(&mut s, &|e| e.algebra.sig_rev = SigRev(0)), - Refusal::AlgebraRevisionMismatch { - class: XLOG, - method: MethodId(1), - algebra: MUL_I64, - bound: SigRev(1), - asked: SigRev(0) + algebra: word(4) } ); assert_eq!(s.class_count(), 0, "no refused class was bound"); @@ -808,3 +832,61 @@ fn a_risk_mask_reaches_the_methods_that_touch_it() { 0 ); } + +/// A consumer chooses its extent; nothing caps it at 64. An ONNX graph with +/// 200 values and 70 nodes does not fit DR64³ and is refused there with the +/// axis named; DR256² × 64 holds the 200 values, and a 256-wide predicate +/// axis holds the 70 methods. The cost is stated, not allocated. +#[test] +fn a_consumer_chooses_its_extent_and_is_not_capped_at_64() { + assert_eq!(Extent::DR64_CUBE.dense_bytes(), 32 * 1024); + assert_eq!(Extent::DR256_SQ_X64.dense_bytes(), 512 * 1024); + + let node = |id: u32| MethodBinding { + id: MethodId(id), + when: None, + on: WideFieldMask::from_positions(&[]), + reads: WideFieldMask::from_positions(&[]), + ..method(id, vec![]) + }; + let mut s = catalogue(); + assert_eq!( + s.register(ONNX, schema(1, 200), vec![node(1)]), + Err(Refusal::FieldsExceedExtent { + class: ONNX, + field_count: 200, + property: Width::W64 + }) + ); + assert_eq!( + s.register(ONNX, schema_in(1, 200, Extent::DR256_SQ_X64), vec![node(1)]), + Ok(()), + "200 values fit the 256-wide property axis" + ); + + let nodes: Vec = (1..=70).map(node).collect(); + assert_eq!( + s.register( + ONNX_BIG, + schema_in(1, 200, Extent::DR256_SQ_X64), + nodes.clone() + ), + Err(Refusal::MethodsExceedExtent { + class: ONNX_BIG, + methods: 70, + predicate: Width::W64 + }) + ); + let wide = Extent { + predicate: Width::W256, + ..Extent::DR256_SQ_X64 + }; + assert_eq!(s.register(ONNX_BIG, schema_in(1, 200, wide), nodes), Ok(())); + // The extent is part of the binding, so it cannot be changed later. + assert_eq!( + s.register(ONNX, schema_in(1, 200, wide), vec![node(1)]), + Err(Refusal::BehaviourImmutable(ONNX)) + ); +} + +const ONNX_BIG: u32 = 0x7F03_0002;