Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .claude/board/LATEST_STATE.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
## 2026-11-02 — main after #1924 (merged `78f9996af`)

- Codegraph refresh splits into a fenced prepare / commit (crate-private); the F-LAND / F-DISCARD lab tests pin it, in debug and release.
- Board: the D-TGL lifecycle contract, the G8 bag contract, production gates G9–G12 + EXO-148 (open), and the G10 × G11 lab spec D-TGL-PUB.

## 2026-11-02 — main after #1922 and #1923 (merged `2e19a5adc`)

- `lance-graph-codegraph` opens stores of any size on a multi-tile xlog cube.
Expand Down
14 changes: 14 additions & 0 deletions .claude/board/PR_ARC_INVENTORY.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,17 @@
## 2026-11-02 — lance-graph PR #1924 (merged `78f9996af`, branch `claude/clone-repositories-71a5sw`) — F-LAND / F-DISCARD lab, D-TGL lifecycle contract, G10 × G11 lab spec

- **Added:**
- Codegraph `prepare_refresh` / `commit_refresh` with a generation fence (`RefreshRefusal::Stale`), crate-private.
- Lab tests F-LAND (second branch recomputed against the current W0 == clean open, call sites included) and F-DISCARD.
- The post-merge record for #1923.
- **Locked:** a preparation made against an older graph generation is refused before any change. Verified in release, where the stale change would otherwise be accepted silently.
- **Docs:**
- `thought-git-lifecycle-contract-v1.md` (D-TGL: GitHub-analogue lifecycle, G8 bag contract, gates G9–G12 + EXO-148);
- `g10-g11-atomic-publication-lab-v1.md` (D-TGL-PUB: boundaries B1–B5, falsifiers L1–L6).
- **Deferred:** the lab `WalSink`, the L1–L6 test, all production gates.
- **Confidence:** 7 lib tests in debug and release; 2 disables red in debug, 1 in release.
- **Stranded:** `check_route` (two commits after the merge point), carried by the next PR.

## 2026-11-02 — lance-graph PR #1923 (merged `2e19a5adc`, branch `claude/clone-repositories-71a5sw`) — D-CODEGRAPH-0 wide closure, atomic refresh, versioned DR index (D-DYNBAG-4)

- **Added:**
Expand Down
2 changes: 1 addition & 1 deletion .claude/board/STATUS_BOARD.md
Original file line number Diff line number Diff line change
Expand Up @@ -249,7 +249,7 @@ Plan: [d-codegraph-0-resident-codegraph-v1.md](../plans/d-codegraph-0-resident-c
| **D-TGL-G10** | atomic publication: prepare all, durable commit first, then infallible in-memory apply from the land record, replay after crash | OPEN (production); named gap: no prepare/validate route API on `BindStore` | a `Fenced` leaves no published partial state |
| **D-TGL-G11** | branch-local routing; no branch-minted DR position as durable routing key | OPEN (production) | a branch route leaves `BindStore::current_version()` unchanged until land |
| **D-TGL-G12** | identities apart: W-slot, generation, ThoughtCommitId, DurableRef, RouteVersion, Lance version, source SHA (provenance only) | OPEN (production) | joins only through receipt / CommitWitness / MetaKey / land record |
| **D-TGL-PUB** | G10 × G11 lab: prepared land (Codegraph + DT64 route intent), fenced durable commit through a test `WalSink`, crash recovery from the durable record, branch-local routing | SPEC: plan `g10-g11-atomic-publication-lab-v1.md`; boundaries B1–B5 located; needs `BindStore::check_route` (read-only, missing) | L1–L6, each with a disable; L4 shows route re-application ticks `RouteVersion` without a watermark |
| **D-TGL-PUB** | G10 × G11 lab: prepared land (Codegraph + DT64 route intent), fenced durable commit through a test `WalSink`, crash recovery from the durable record, branch-local routing | SPEC: plan `g10-g11-atomic-publication-lab-v1.md`; boundaries B1–B5 located. Step 1 SHIPPED (lab-ready): `BindStore::check_route` (read-only; `route` runs the same `RouteLog::check`), 3 tests, 3 disables red. Step 2 SHIPPED (lab): `LabWal` in `lance-graph-codegraph/src/lab_wal.rs` (test-only `WalSink`: fence, one version per commit, reconcile, `HashConflict`, payload-free scan, image read, injected I/O / Ambiguous faults), 4 tests, 4 disables red; planner is a dev-dependency, one ndarray via `[patch]`. Step 3 (L1–L6) open | L1–L6, each with a disable; L4 shows route re-application ticks `RouteVersion` without a watermark |
| **D-TGL-EXO148** | `DurableRef` + never-reused `ThoughtCommitId` before any durable land, replay or PR record | OPEN (production, EXO-148) | reuse test (D-RST-1) |
| **D-DT64-AUTH** | DT64 / F### as the one method authority: census of production words with a two-way exception ratchet; parallel identities (`MethodSignature`, tract `Registry`, XLOG `Word`, ogar-loco `FnIndex`) stay as listed `LAB` adapters | QUEUED: plan `d-dt64-method-authority-census-v1.md` (operator decision 2026-11-02); egglog pinned to one commit in `egglog-r2il-probe`, `dr-math-binding-probe`, `xlog-relcomp-probe` | undeclared production word → red; stale exception → red; current tree green |
| **D-CODEGRAPH-0-D** | local stdio MCP over C | QUEUED | only after B and C gates |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,3 +28,37 @@
- **The smallest common falsifier set is L1–L6.** Each has its own disable. The decisive one is L4: without the watermark, a second recovery ticks `RouteVersion` while the Codegraph stays unchanged.

OPEN: `check_route`, the lab `WalSink`, and the L1–L6 test are three separate reviewable steps (plan §4). None is implemented.

## Step 1 done: `BindStore::check_route` (2026-11-02)
- **Code:**
- `BindStore::check_route(&self, position, class) -> Result<(), Refusal>` applies `route`'s two refusals and writes nothing.
- `route` runs the same check (`route_identity`, then `RouteLog::check`) before it pushes, so the two cannot disagree.
- **Tests** (`crates/lance-graph-dt64/tests/bind_store.rs`):
- agreement with `route` on 5 cases: 2 accepted, 1 `UnknownClass`, 2 `OutsideReservation` at the 255/256 boundary and beyond;
- writes nothing: version, `routed_at` and `claimed` unchanged; the twin `route` does move them;
- a checked intent stays valid as the store grows: other classes bound, other routes and retracts, then applied without a refusal.
- **Disables (each red on the agreement test):** always accept; skip the reservation; skip the class lookup.
- "Writes nothing" holds by the type (`&self`), so it has no disable. Its twin shows the comparison can fail.
- **Gates:** dt64 suites green (25 + 29); clippy and fmt clean. dt64 already runs in `rust-test.yml`.

## Step 2 done: the lab `WalSink` (2026-11-02)
- **Code:** `LabWal` in `crates/lance-graph-codegraph/src/lab_wal.rs`, `#[cfg(test)]` only. It is an in-memory `WalSink` that follows the production contract:
- `Fenced` against the head, with nothing written;
- one version per commit;
- reconciliation of a re-submitted `(cycle, batch_hash)`;
- `HashConflict` for another batch of a durable cycle;
- a payload-free `scan_sealed`.
- **Lab seams:**
- injected faults: I/O before publish, and `Ambiguous` after publish (B3);
- `image(cycle)`, the durable payload read that recovery must use.
- **Dependencies:**
- `lance-graph-planner`, `lance-graph-contract` and `tokio` (rt) are dev-dependencies only; `cargo tree -e normal` shows no planner.
- The crate's own workspace patches the ndarray git coordinate to the sibling checkout, as the root workspace does. `cargo tree -i ndarray` shows one package; `-d` shows no duplicate.
- CI's `test` job already checks out ndarray as a sibling.
- **Tests:**
- stale base fenced, nothing written;
- a commit publishes one version, and the payload lives only in the image;
- a re-submit reconciles, a different batch conflicts;
- a fault before publish and one after publish leave different durable states.
- **Disables (each red on its own test):** drop the fence; leak the payload through `scan`; skip the reconcile lookup; make `Ambiguous` publish nothing.
- **Gates:** codegraph suites green (12 lib tests); clippy and fmt clean.
Original file line number Diff line number Diff line change
Expand Up @@ -36,3 +36,10 @@
- The Codegraph has no `BindStore`, so F-DISCARD's "global `RouteVersion` unchanged" holds by construction. DT64 routing staging is open (G10/G11: no prepare/validate route API on `BindStore`).
- `PreparedRefresh` / `commit_refresh` are crate-private. Nothing publishes, persists or routes.
- Lab results do not satisfy G9–G12 or EXO-148.

## Correction (codex P2 on #1924, fixed in #1925)
- **Defect:** a read between `prepare_refresh` and `commit_refresh` (`impact` and `closure` take `&self`) could fill a closure cache the preparation had not seen. That cache was built from the pre-commit edges. `commit_refresh` kept it, because the prepared plan said "no cache", and the generation fence does not move on a read. `impact` then answered from the old graph: `m::x` was missing as a caller of `m::b`.
- **Fix:** when the batch changes edges, `commit_refresh` drops every closure cache it has no prepared refresh for.
- **Test:** `lifecycle::a_closure_cached_between_prepare_and_commit_is_not_served_stale`, which compares the result with a clean open.
- **Disable:** dropping the new arm turns it red in debug and release.
- **Why the lab tests missed it:** they never read between prepare and commit. Interleaved reads are part of the lifecycle (a PR is read while it waits), so the fixture now covers that case.
4 changes: 2 additions & 2 deletions .claude/plans/g10-g11-atomic-publication-lab-v1.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,10 +78,10 @@ The in-memory publication is one infallible step (Codegraph `commit_refresh` + t

## 4. What has to exist before the lab runs (each its own reviewable step)

1. `BindStore::check_route` (read-only; the two refusals of `route`, no write). Its own falsifiers:
1. **DONE (2026-11-02):** `BindStore::check_route` (read-only; the two refusals of `route`, no write; `route` runs the same check). Its own falsifiers:
- it agrees with `route` on accept and refuse for every case;
- it leaves `current_version()` unchanged.
2. A lab-only test `WalSink` beside the Codegraph test (dev-dependency on `lance-graph-planner`; test `WalSink` impls exist in ten places, for example `lance-graph-planner/examples/blw_fusion.rs` `MemWal`). No production edge from Codegraph to the planner.
2. **DONE (2026-11-02):** a lab-only test `WalSink` (`LabWal`, `crates/lance-graph-codegraph/src/lab_wal.rs`) beside the Codegraph test (dev-dependency on `lance-graph-planner`; test `WalSink` impls exist in ten places, for example `lance-graph-planner/examples/blw_fusion.rs` `MemWal`). No production edge from Codegraph to the planner.
3. The lab test itself, in-crate, because `prepare_refresh` / `commit_refresh` are crate-private.

Out of scope: `LanceCycleWriter` (needs the 512-byte ABI and real Lance), `ThoughtCommitId`/`DurableRef` (EXO-148), MergeQueue, review gates.
15 changes: 15 additions & 0 deletions crates/lance-graph-codegraph/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,18 @@ lance-graph-dt64 = { path = "../lance-graph-dt64" }
lance-graph-quack = { path = "../lance-graph-quack" }
lance-graph-mask-risc = { path = "../lance-graph-mask-risc" }
ndarray = { git = "https://github.com/AdaWorldAPI/ndarray", branch = "master", default-features = false, features = ["std"] }

[dev-dependencies]
# Lab only (D-TGL-PUB): the durable writer contract (`WalSink`, `CommitOutcome`,
# `CommitError`, `DetachedCycleBatch`) for the G10 x G11 publication lab. Never
# a production edge from the codegraph to the planner.
lance-graph-planner = { path = "../lance-graph-planner" }
lance-graph-contract = { path = "../lance-graph-contract" }
tokio = { version = "1", features = ["rt"] }

# One ndarray (CARGO-COMPUTE-SUBSTRATE rule 9): the planner names the sibling
# checkout by path, this crate and its xlog/mask-risc deps by the git
# coordinate. Point the coordinate at the same checkout, as the lance-graph
# root workspace does, so the graph holds one ndarray package.
[patch."https://github.com/AdaWorldAPI/ndarray"]
ndarray = { path = "../../../ndarray" }
53 changes: 50 additions & 3 deletions crates/lance-graph-codegraph/src/graph.rs
Original file line number Diff line number Diff line change
Expand Up @@ -838,14 +838,27 @@ impl Codegraph {
}
None => self.cube.frame().clone(),
};
let changed = !staged.is_empty();
self.cube.commit(&self.table, &frame, staged);
for (slot, cell) in [(0usize, &mut self.rconf), (1, &mut self.rposs)] {
if let Some(next) = closures[slot].take() {
cell.take();
if let Some((rows, st)) = next {
match closures[slot].take() {
// Prepared against the cache that was there: replace it.
Some(Some((rows, st))) => {
cell.take();
stats.rows_recomputed += st.rows_recomputed;
let _ = cell.set((rows, st));
}
// The cache was there but could not be refreshed: drop it.
Some(None) => {
cell.take();
}
// No cache at preparation. A read between prepare and commit
// may have filled one (`closure` / `impact` take `&self`) from
// the pre-commit edges; the edges change now, so it is stale.
None if changed => {
cell.take();
}
None => {}
}
}

Expand Down Expand Up @@ -1439,6 +1452,40 @@ mod lifecycle {
);
}

/// Codex P2 on #1924. A closure cache filled by a read between prepare and
/// commit (`impact` takes `&self`) was built from the pre-commit edges.
/// Red if commit keeps it (disable: drop the `None if changed` arm in
/// `commit_refresh`): `impact` would then answer from the old graph.
#[test]
fn a_closure_cached_between_prepare_and_commit_is_not_served_stale() {
let w0 = store("w0", &m_v1(), &n_v1());
let branch_a = store("a", &m_v2(), &n_v1());
let mut g = Codegraph::open(&w0).expect("open W0");
assert!(g.rconf.get().is_none() && g.rposs.get().is_none());
let a = g
.prepare_refresh(&branch_a, "m")
.expect("A")
.expect("A changes");
let b = g.pos("m::b").expect("m::b");
let before = g.impact(b).expect("impact");
// Anti-vacuity: the read did fill a cache the preparation did not see.
assert!(g.rconf.get().is_some() || g.rposs.get().is_some());
g.commit_refresh(a).expect("commit");
let after = g.impact(b).expect("impact");
let x = g.pos("m::x").expect("m::x");
let has = |m: &[u64]| {
m.get(x as usize / 64)
.is_some_and(|w| w >> (x % 64) & 1 == 1)
};
assert!(!has(&before.confirmed), "m::x did not exist before");
assert!(
has(&after.confirmed),
"impact of m::b must include its new caller m::x"
);
let mut clean = Codegraph::open(&branch_a).expect("clean");
assert_eq!(answers(&mut g), answers(&mut clean));
}

/// F-DISCARD. A prepared branch that is dropped leaves W0 exactly as it
/// was: interner, positions, `last_owner`, lanes, digests, the map
/// table's version and history, the cube's frame and logs, masks,
Expand Down
Loading
Loading