From fa943a6c0bd7c392db9070af6bbc7ef1d784ed1b Mon Sep 17 00:00:00 2001 From: James Ross Date: Sun, 9 Aug 2026 20:31:09 -0700 Subject: [PATCH 1/7] docs: record Echo Keep CAS feasibility exploration --- .../echo-keep-physical-cas-interop-plan.md | 1029 +++++++++++++++++ 1 file changed, 1029 insertions(+) create mode 100644 docs/plans/echo-keep-physical-cas-interop-plan.md diff --git a/docs/plans/echo-keep-physical-cas-interop-plan.md b/docs/plans/echo-keep-physical-cas-interop-plan.md new file mode 100644 index 00000000..a558f308 --- /dev/null +++ b/docs/plans/echo-keep-physical-cas-interop-plan.md @@ -0,0 +1,1029 @@ + + + +# Echo × Keep Physical CAS Interop and Identity Corroboration Exploration (No ADR) + +## TL;DR + +| Question | Exploratory answer | +| ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| What are we evaluating? | Whether Keep can become Echo's physical retained-content engine for CAS-addressed materialized readings, WSC import, reconstruction, range reads, and retention workflows. | +| What is the central hypothesis? | The same exact bytes can be independently admitted under Echo's content-only hash law and Keep's versioned logical-identity law, producing an explicit correspondence witness without pretending the two identities are equal. | +| What remains Echo-owned? | Semantic coordinates, causal truth, WSC content hashes, authorization, replay policy, materialization intents, and observations. | +| What may become Keep-owned? | Keep's independent logical byte identity, chunking, layouts, physical publication, restart recovery, retained closure, range reads, and eventually compaction or garbage collection. | +| What does the adapter own? | Proof that one exact byte stream satisfied both identity systems, plus translation between Echo content claims and exact Keep reconstruction coordinates. | +| What is the first artifact? | An exploratory `EchoKeepLogicalBindingV1` for Echo hash + length ↔ Keep `BlobId`, plus `EchoKeepRealizationBindingV1` for one exact Keep `LayoutId`. | +| Are Echo `BlobHash` and Keep `BlobId` interchangeable? | No. They assert different propositions and must never be cast, substituted, or silently normalized into one identity. | +| Can one logical binding have multiple Keep layouts? | Yes. Different verified `LayoutId` values may lawfully realize the same Keep `BlobId`. A different verified Keep `BlobId` for the same exact Echo bytes is an obstruction. | +| What is the first implementation posture? | A standalone Rust 1.96 interop spike using Keep's non-durable `ReferenceStore` as the executable oracle, followed by a narrow fallible WSC streaming port. | +| What is the likely production replacement target? | Echo's `DiskTier`, not Echo's content-only hash namespace or all of `echo-cas`. | +| When is an ADR required? | Before freezing a persisted binding format, changing WSC wire identity, adopting a production Keep backend, changing durable Keep read/publication contracts, or declaring a permanent ownership boundary. | + +## Repository posture + +This is an exploratory design artifact. It records a hypothesis and a sequence +of executable tests; it does not establish a durable architectural boundary. + +No ADR is introduced by this branch. If this document is carried in +`docs/plans/` for branch-local work, it must remain explicitly non-normative and +must be removed, superseded, or converted into the appropriate durable decision +record before the underlying identity, format, public API, durability, recovery, +or ownership boundary is adopted. + +### Source-reality audit findings (2026-08-04) + +Confirmed from the current `flyingrobots/echo` and `flyingrobots/keep` source: + +- **Echo CAS remains full-materializing by design.** + - `crate::echo_cas::BlobStore` still exposes a sync `get` path. + - `MemoryTier` returns buffered bytes from memory as `Arc<[u8]>`. + - `DiskTier` still reads complete blobs and re-validates full hash before + returning a complete byte buffer. +- **WSC boundary is still `Option>` today.** + - `crates/warp-core/src/wsc/store.rs` continues to require full-bytes + materialization for `cas_blob_bytes`. + - `Warp CLI` still uses an unavailable CAS stand-in for this path (`UnavailableCasStore`), + so any streaming replacement must not assume immediate CLI parity. +- **Keep split is real and relevant.** + - The `ReferenceStore` is a strong non-durable executable oracle. + - Keep retains a separate durable catalog/recovery stack (`Store`, `State`, + writer authority, retained closures, replay states) and cannot be treated as + one replacement for current `echo-cas` behavior. +- **Toolchain mismatch remains unsolved.** + - `echo-keep` currently resolves to a 1.96+ Rust profile (`flyingrobots/keep` + requirement), while the working echo branch is pinned to 1.90. +- **Viability status is unchanged:** + - The plan remains credible for exploration and spike proving. + - It is **not yet production-accurate** because streaming WSC CAS APIs and + Keep durable logical-read integration are both still design-level commitments + without code lockstep in this repo. + +## Objective + +Decide whether Keep can replace or host the physical storage responsibilities +currently associated with `echo-cas` for CAS-addressed materialized readings, +WSC import, reconstruction, range reads, and retention workflows, without +breaking Echo's invariants for causal truth, replay safety, deterministic +identity, or semantic authority. + +The primary exploration is **independent identity corroboration**: + +> One exact byte stream is independently admitted under Echo's content-only +> BLAKE3 identity and Keep's versioned logical `BlobId`; the adapter retains a +> witness that both systems observed the same bytes without claiming that their +> identities are equal. + +This changes the purpose of the integration. The goal is not CAS API parity and +not identity replacement. The goal is a lawful division of responsibility: + +- Echo explains **why the bytes matter**. +- Keep proves **which exact bytes and reconstruction plan exist**. +- The adapter witnesses **that both identity systems admitted the same + material**. + +## Primary architecture + +```mermaid +flowchart TD + S[Echo semantic coordinate
why the bytes matter] + C[Echo content claim
BlobHash + byte length] + L[Logical corroboration
Echo bytes ↔ Keep BlobId] + R[Keep realization
BlobId + LayoutId / RetentionAnchor] + P[Keep physical evidence
publication + recovery + retained closure] + O[Echo causal observation
materialization effect observed] + + S --> C + C --> L + L --> R + R --> P + P --> O +``` + +The evidence chain is intentionally layered: + +1. An Echo semantic coordinate names the question the bytes answer. +2. An Echo content claim names the raw bytes under Echo's content-only law. +3. A logical corroboration witness binds that claim to a Keep `BlobId` obtained + from the same exact bytes. +4. A realization witness names one exact Keep `LayoutId` capable of + reconstructing those bytes. +5. Keep publication and retention evidence proves the physical realization is + available under Keep's own laws. +6. Echo records the authorized external effect and its verified observation in + causal history. + +No lower layer acquires the authority of the layer above it. + +## Corroboration ladder + +The integration should expose progressively stronger evidence postures instead +of one vague `stored` or `verified` boolean. + +| Level | Required evidence | Claim established | Explicit nonclaim | +| ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | ------------------------------------------------------------------ | +| C0 — dual-coordinate candidate | Echo content hash and exact length computed while Keep ingests the same complete source stream, plus the candidate Keep `BlobId` and `LayoutId` | Both identity calculations observed one complete source stream during the operation | Does not prove that retained Keep state can reconstruct the bytes | +| C1 — retained reconstruction corroborated | C0 plus complete Keep reconstruction through the named layout, a matching `ReconstructionReceipt`, and an independently recomputed Echo hash and length | The named Keep realization reconstructs exact bytes satisfying both identity laws | Does not prove process survival or durable publication | +| C2 — durable physical corroboration | C1 plus exact Keep publication, pinned-generation, and retained-closure evidence | A durable Keep state supports the same verified logical correspondence | Does not grant Echo causal authority | +| C3 — causally observed materialization | C2 plus a committed Echo materialization observation matching a previously committed intent | Echo has causally witnessed the verified physical materialization | Does not make Keep publication history part of Echo causal history | + +The exploratory binding type represents C1, not merely C0. Phase C must produce +C2 evidence. Phase D must reconcile C2 into C3 through Echo's existing outbox. +No API may accept a weaker level where a stronger level is required. + +## Current Echo state (as of branch `echo-keep`) + +- `echo-cas` owns a content-only CAS identity: + - `BlobHash = BLAKE3(bytes)` with no domain prefix; + - identical bytes intentionally share one hash regardless of semantic role; + - semantic meaning is carried separately by `SemanticBlobCoordinate` and + retained descriptors. +- The public `BlobStore` interface is synchronous and materializing: + - `get` returns `Option>`; + - the source comments already identify this return shape as unsuitable for + future disk and cold streaming tiers; + - `MemoryTier` is an infallible in-process implementation. +- `DiskTier` is fallible and filesystem-backed: + - it stores bytes by content-only hash; + - `get` reads the entire blob into memory and re-verifies the hash; + - it provides ordinary process-restart persistence; + - it does not currently expose a complete publication, synchronization, + crash-classification, recovery, or persistent-retention protocol comparable + to Keep's durable store; + - its pin set is process-local. +- The WSC CAS bridge remains full-materialization oriented: + - `WscCasBlobStorePort::cas_blob_bytes` returns `Option>`; + - CAS-addressed WSC import loads full bytes, recomputes Echo's content hash, + checks length, and returns the full allocation; + - `warp-cli` currently supplies an unavailable CAS implementation for these + paths. +- Echo already has the correct causal side-effect protocol: + - `MaterializationIntentRecord` records authorized external work; + - `MaterializationObservationRecord` records verified completion; + - outbox recovery distinguishes already observed, matching existing + artifact, missing artifact, digest mismatch, metadata mismatch, and missing + retained material. + +### Consequence + +Echo already has the semantic identity and causal protocol that must survive the +integration. Its weak point is the physical blob interface and implementation, +not the meaning of its content hashes. + +## Current Keep state (as of the attached worktree) + +Keep exposes two materially different layers that must not be conflated. + +### Reference adapter + +`ReferenceStore` is a bounded, deterministic, in-memory executable model: + +- ingestion uses a fixed read buffer and bounded chunk-detector state; +- staging is invisible until explicit `commit`; +- publication returns `PublishedBlob { BlobId, LayoutId }`; +- reconstruction authenticates every selected chunk, replays the registered + storage profile, verifies the complete `BlobId`, and writes to a caller-owned + `Write`; +- exact range reads load only overlapping chunks and return a deliberately + narrower receipt; +- process death loses all committed state; +- stored chunk material may grow with admitted logical content, so the adapter + is not an O(1)-total-memory backend merely because its detector and read + buffers are bounded. + +### Durable store and retention machinery + +Keep also has substantial durable infrastructure beyond `ReferenceStore`: + +- immutable segments and exact record formats; +- catalog generations and publication heads; +- platform admission and a store-wide writer authority; +- synchronized stage, immutable-pool, catalog, and `HEAD` publication; +- restart-loaded catalog snapshots; +- typed recovery inventory, stage classification, completion, discard, + resumption, and next-head finalization; +- process-death crash-matrix evidence; +- version-2 retention anchors, roots, manifests, heads, closure verification, + transition planning, and publication vocabulary. + +The remaining gap is not "make Keep durable from scratch." It is to expose a +high-level, streamable, retained-logical-blob capability that can satisfy Echo's +consumer boundary without importing Echo semantics into Keep core. + +### Keep proof scopes + +- A `BlobId` is a versioned logical identity for exact finite bytes. Parsing a + coordinate does not prove that matching content exists. +- A `PublishedBlob` identifies a visible `BlobId` and one exact `LayoutId` in + the reference adapter. +- A `ReconstructionReceipt` proves complete authenticated reconstruction under + one layout. +- A `RangeReadReceipt` proves only the requested bytes came from authenticated + overlapping chunks under an admitted layout. It does not prove the complete + blob, unrequested chunks, or storage-profile boundaries. +- A `RetentionAnchor` combines `BlobId` and `LayoutId`, but the anchor alone + does not prove closure or byte availability; retained closure must be verified + separately. + +### Toolchain consequence + +Echo is pinned to Rust 1.90 while Keep declares Rust 1.96. A Keep dependency +cannot simply be added to Echo's ordinary workspace without first making an +explicit toolchain decision. The exploration should therefore begin in a +standalone Rust 1.96 interop workspace or dedicated higher-toolchain CI lane. + +## Core invariants + +### 1. Identity laws remain distinct + +Echo's `BlobHash` and Keep's `BlobId` are not aliases and do not have a lawful +cast between them. This exploration binds Echo's current content-hash law to +Keep `BlobId` version 1 specifically. + +- Echo identifies `BLAKE3(bytes)` directly. +- Keep identifies exact finite logical bytes under a versioned, + length-committing identity law. + +Both use BLAKE3 internally, but they assert different propositions. + +### 2. Corroboration requires the same exact bytes + +A binding may be admitted only after one exact byte stream has been observed by +both identity calculations and the Keep realization has passed complete +reconstruction verification. + +A parsed `BlobId`, a filename, a catalog entry, a `LayoutId`, or an existing +mapping is insufficient by itself. + +### 3. Logical identity and physical realization remain separate + +The logical corroboration is: + +```text +Echo content hash + exact byte length ↔ Keep BlobId +``` + +A Keep `LayoutId` is one physical/logical reconstruction realization of that +Keep `BlobId`. + +Multiple independently verified layouts may lawfully realize the same logical +binding. The adapter must not treat alternate valid layouts as an identity +collision. + +### 4. Semantic authority remains in Echo + +Keep must not infer or own: + +- semantic coordinates; +- causal authority; +- WSC basis meaning; +- admission policy; +- replay decisions; +- materialization authorization; +- application retention policy. + +Keep reports only the physical evidence its own contracts support. + +### 5. Physical claims remain in Keep + +Echo must not pretend that a content hash, semantic coordinate, or causal fact +proves physical presence, durability, retained closure, or recoverability. + +Those claims require Keep evidence. + +### 6. Proof scope is represented explicitly + +Whole-blob, range, publication, retained-closure, and causal-observation claims +must remain distinct. A narrow receipt cannot be promoted into a stronger claim +by convention or naming. + +### 7. Failure is fail-closed and typed + +Absence, I/O failure, corruption, identity disagreement, layout disagreement, +unsupported platform, resource refusal, stale publication state, and missing +retained closure must remain distinguishable. + +### 8. Causal side effects use Echo's existing outbox + +The integration must reuse Echo's materialization intent, idempotency, +observation, and recovery posture. It must not invent an independent causal +write protocol for CAS effects. + +### 9. WSC identity does not change during exploration + +WSC content hashes remain Echo content-only hashes. Keep coordinates are +corroborating and realization evidence, not replacement wire identities. + +## Exploratory evidence artifacts + +The initial spike should separate logical corroboration from physical +realization with two in-memory typed values: + +```rust +pub struct EchoKeepLogicalBindingV1 { + pub echo_content_hash: [u8; 32], + pub byte_len: u64, + pub keep_blob_id: keep::BlobId, +} + +pub struct EchoKeepRealizationBindingV1 { + pub logical: EchoKeepLogicalBindingV1, + pub keep_layout_id: keep::LayoutId, +} +``` + +These are conceptual test types, not yet frozen public APIs or persisted +formats. + +### Interpretation + +- `EchoKeepLogicalBindingV1` is the same-byte corroboration claim. +- `EchoKeepRealizationBindingV1` names one exact Keep layout that realizes the + corroborated logical bytes. +- `logical.keep_blob_id + keep_layout_id` can be viewed as the Keep-side + `RetentionAnchor` coordinate. +- One logical binding may have multiple independently verified realization + bindings. +- Echo semantic coordinates remain above the logical binding and may point to + the same content through different semantic roles. +- Keep publication and retained-closure evidence remain below the realization + binding and must be attached separately when stronger physical claims are + required. + +### Admission rules + +A logical binding and one realization binding are admitted only when all of the +following hold: + +1. Echo's raw content hash and exact length were computed from the complete + source stream. +2. Keep staged and committed that same source stream and returned the named + `BlobId` and `LayoutId`. +3. Keep completely reconstructed the exact named layout. +4. The reconstructed bytes independently reproduced the expected Echo content + hash and length. +5. The Keep reconstruction receipt named the same `BlobId` and `LayoutId` as + the candidate realization binding. + +### Conflict rules + +After complete verification: + +- same Echo hash and length under the same declared Keep identity version, + different Keep `BlobId` → **identity corroboration obstruction**; +- same Keep `BlobId`, different Echo hash or length → **identity corroboration + obstruction**; +- same logical corroboration, different valid `LayoutId` → **lawful alternate + realization**; +- same `LayoutId`, different Keep `BlobId` → **layout identity obstruction**; +- parsed coordinates without verified bytes → **unproven candidate**, not a + binding. + +### Deferred format decision + +This branch must not freeze: + +- canonical binding bytes; +- a binding digest domain; +- a persistent mapping database; +- a WSC field; +- a Keep durable record; +- a public Rust ABI. + +Those choices affect identity and compatibility and therefore require a durable +decision record before adoption. + +## Gap matrix + +| Gap | Severity | Required closure | +| ----------------------------- | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | +| Identity correspondence | Critical | Prove same-byte corroboration without collapsing Echo `BlobHash` into Keep `BlobId`. | +| Interface shape | Critical | Replace `Option>` at the WSC consumer boundary with a fallible bounded streaming or write-through capability. | +| Binding lookup | Critical | Resolve Echo content claims to exact Keep logical and layout coordinates through verified bindings. | +| Durable logical reads | Critical for production | Expose caller-owned-output reconstruction from Keep's durable store without retaining every selected segment in one snapshot allocation. | +| Causal publication | Critical for production | Bind Keep publication evidence to Echo's materialization intent and observation protocol. | +| Retention ownership | Critical for production | Keep owns physical liveness and closure; Echo owns semantic retention intent and causal meaning. | +| Proof-scope separation | Critical | Prevent range, presence, publication, and closure receipts from impersonating complete identity proof. | +| Error semantics | High | Distinguish absence, I/O, corruption, mismatch, stale state, unsupported platform, and resource refusal. | +| Downstream WSC recovery shape | High | Removing the raw CAS `Vec` does not by itself bound memory if recovery retains every decoded WAL payload. | +| Toolchain | High | Resolve Echo Rust 1.90 versus Keep Rust 1.96 before workspace integration. | +| Platform posture | High | Keep's current production filesystem claims are limited to its admitted Linux ext4 profile. | +| Retention persistence | Medium during spike | Do not confuse Echo's process-local pins or Keep reference-store presence with durable retained closure. | + +## Feasibility assessment + +Keep is a strong candidate to become Echo's physical retained-content engine, +but it is not a drop-in `BlobStore` implementation and should not replace +Echo's content-only identity namespace. + +The safe target architecture is: + +```text +Echo semantic coordinates and content claims + │ + ▼ +Echo × Keep corroboration adapter + │ + ▼ +Keep logical identity, layouts, publication, recovery, and retention +``` + +The likely production migration is therefore: + +- preserve Echo `BlobHash` and WSC content hashes; +- preserve Echo semantic coordinates and causal records; +- replace or retire `DiskTier` behind a new fallible physical-content port; +- retain `MemoryTier` as a useful in-process implementation; +- use Keep as the serious physical implementation only after durable logical + read, binding persistence, recovery, and retention evidence are closed. + +## Proposed staged integration architecture + +## Phase A — independent identity corroboration + +### Goal + +Prove that one exact stream can yield stable Echo and Keep identities and that a +specific Keep layout reconstructs bytes matching the Echo content claim. + +### Implementation posture + +- Create a standalone Rust 1.96 interop workspace or dedicated test package. +- Depend on the Echo branch and Keep worktree through local paths. +- Keep the exploratory binding type private to the spike. +- Use Keep's `ReferenceStore` as the executable oracle, not as a production + backend. + +### Ingest choreography + +1. Wrap the source in a reader that updates: + - an Echo raw BLAKE3 accumulator; + - checked exact byte-length accounting. +2. Pass that reader into Keep staging. +3. Obtain the candidate Keep `BlobId` and `LayoutId` from staged or published + work. +4. Commit the Keep reference-store stage. +5. Reconstruct the exact returned layout into a sink that independently + recomputes Echo's raw hash and length. +6. Compare the source Echo claim, reconstructed Echo claim, `PublishedBlob`, and + `ReconstructionReceipt`. +7. Admit `EchoKeepLogicalBindingV1` and one `EchoKeepRealizationBindingV1` only after all values agree. + +### Required golden cases + +- empty bytes; +- small text; +- deterministic binary ramp; +- exact chunk-boundary and boundary-plus-one sizes; +- large deterministic virtual input; +- nearby edited states with expected chunk reuse; +- arbitrary source read partitioning; +- short reads and interrupted reads; +- short writes and interrupted writes during reconstruction. + +### Required negative cases + +- incorrect expected Echo hash; +- incorrect byte length; +- parsed but unverified Keep `BlobId`; +- wrong Keep `BlobId` paired with a valid layout; +- wrong `LayoutId` paired with a valid `BlobId`; +- missing chunk; +- corrupt chunk; +- corrupt layout; +- profile-boundary mismatch; +- full reconstruction that produces a different Echo content hash; +- conflicting logical binding for the same fully verified Echo content claim. + +### Acceptance claim + +```text +echo.keep.same-bytes-binding/v1 +keep.echo.identity-agreement/v1 +``` + +For every admitted case, one exact byte stream produces a stable Echo content +claim, a stable Keep `BlobId` v1, and at least one exact Keep `LayoutId`; +complete reconstruction through that layout reproduces the Echo content claim +exactly. + +The first name is the adapter-facing capability. The second is the existing +Keep Golden File Worldline destination. The Keep capability must not be marked +complete merely because an in-process adapter test passes; closure requires a +reviewed cross-repository conformance corpus or equivalent independently +reproducible evidence. + +### Explicit nonclaims + +- no process survival; +- no durable binding format; +- no production retention; +- no WSC API change; +- no O(1) total process memory claim for `ReferenceStore`; +- no replacement of `echo-cas`. + +## Phase B — WSC read-path streamability + +### Goal + +Remove the mandatory whole-blob return value from CAS-addressed WSC validation +while preserving Echo content-hash and length verification. + +### Boundary + +Introduce a narrow, fallible WSC consumer port, conceptually: + +```rust +pub trait WscCasReadPort { + fn copy_verified( + &self, + claim: WscCasClaim, + output: &mut dyn std::io::Write, + ) -> Result; +} +``` + +The final API may instead expose a reader-like value, but the semantics must be +identical: + +- normal absence is distinct from failure; +- I/O is fallible; +- the complete Echo content hash is verified; +- exact length is verified; +- no success receipt exists before EOF and final verification; +- no admitted WSC semantic result escapes from provisional bytes; +- the implementation does not require a full adapter-owned `Vec`. + +### Implementations + +1. Adapter over Echo `MemoryTier`. +2. Adapter over Echo `DiskTier`. +3. Experimental Keep `ReferenceStore` adapter resolving through an admitted + `EchoKeepRealizationBindingV1`. + +### Order of conversion + +1. Convert CAS-addressed retained-material availability checks first. They load, + verify, and discard bytes and therefore provide the cleanest streaming + witness. +2. Convert CAS-addressed WAL-segment validation next. +3. Preserve byte-slice convenience APIs as wrappers around the streaming path, + not the reverse. + +### Verification rule + +A Keep reconstruction receipt is necessary but not sufficient for WSC import. +The WSC boundary must still verify Echo's raw content hash and exact length. + +### Memory claim + +Phase B may claim: + +> WSC validation no longer requires a second adapter-owned whole-segment input +> allocation. + +It must not yet claim: + +> CAS-addressed WSC import is constant-memory regardless of output evidence. + +Current WAL recovery structures retain decoded frame payloads. True bounded +end-to-end import requires a later incremental recovery fold or compact evidence +shape. + +## Phase C — durable Keep logical-read capability + +### Goal + +Expose a generic Keep-owned capability that reconstructs or range-reads an exact +retained logical blob from a pinned durable store generation into a caller-owned +writer. + +### Keep ownership rule + +The capability must be named and designed in Keep's vocabulary. Keep core must +remain independent of Echo, WSC, semantic coordinates, causal authority, and +application policy. + +### Required durable capability + +Given an admitted store snapshot and exact `RetentionAnchor`: + +- verify the anchor's layout and complete retained closure as required by the + requested proof scope; +- locate exact immutable records; +- authenticate selected chunks and layout evidence; +- reconstruct or range-read into caller-owned output; +- return a precise receipt; +- avoid loading all selected segment bytes into one durable-snapshot allocation; +- preserve pinned-generation semantics while later generations publish. + +### Platform posture + +The first production experiment is limited to Keep's admitted writable, +non-casefolded Linux ext4 profile. Unsupported platforms must return a typed +refusal rather than silently degrading durability or verification claims. + +### Toolchain posture + +Choose one of the following explicitly before code enters Echo's ordinary +workspace: + +- keep the adapter in a Rust 1.96 boundary workspace; +- add a separate higher-toolchain CI package; +- deliberately upgrade Echo through its own change; +- lower Keep's Rust requirement only if Keep independently supports and tests + that contract. + +Toolchain drift must not be smuggled in as an incidental dependency change. + +### Acceptance claims + +```text +echo.keep.durable-reconstruct/v1 +echo.keep.durable-range/v1 +echo.keep.restart-read/v1 +``` + +## Phase D — causal publication and retained evidence + +### Goal + +Bind Keep publication and retention evidence to Echo's existing materialization +outbox without creating hidden side effects or a second causal protocol. + +### Choreography + +```mermaid +sequenceDiagram + participant E as Echo + participant W as Echo WAL / Outbox + participant A as Echo × Keep Adapter + participant K as Keep + + E->>W: Commit MaterializationIntentRecord + W-->>A: Authorized effect + idempotency token + A->>K: Stage exact source bytes + K-->>A: Candidate BlobId + LayoutId + A->>K: Commit / publish + K-->>A: Publication receipt + A->>K: Reconstruct exact layout + K-->>A: Reconstruction receipt + bytes to verifier + A->>A: Verify Echo hash + length; admit binding + A-->>E: Existing artifact and binding evidence + E->>W: Commit MaterializationObservationRecord +``` + +### Causal rules + +1. Echo commits authorization before the external Keep effect. +2. Keep staging remains invisible until its explicit commit or durable + publication point. +3. The adapter admits no binding before complete same-byte verification. +4. Echo commits an observation only after Keep publication and binding evidence + agree with the authorized effect. +5. Recovery may discover matching Keep evidence after publication but before + Echo observation and classify it as an existing matching artifact. +6. Mismatch, unavailable retained closure, unsupported platform, or ambiguous + binding obstructs replay. +7. Reads, chunk lookups, dedupe hits, and cache behavior are not themselves + causal events. + +### Binding persistence requirement + +Production recovery cannot rely on a digest of binding bytes that are no longer +recoverable. Before Phase D is adopted, the canonical binding evidence must be +retained through a governed carrier, such as: + +- an Echo WAL payload; +- an Echo retained evidence artifact; +- a separately governed adapter record referenced from the outbox; +- another explicitly admitted durable format. + +Selecting that carrier and encoding is an ADR trigger. + +### Crash matrix + +Exercise at least: + +```text +before Echo intent commit +after Echo intent / before Keep stage +during Keep stage +after Keep stage / before Keep publication +after Keep publication / before binding verification +after binding verification / before Echo observation +after Echo observation commit +retry after every boundary +matching existing publication +conflicting existing publication +missing retained closure +conflicting logical binding +unsupported platform +``` + +### Acceptance claims + +```text +echo.keep.outbox-reconciliation/v1 +echo.keep.binding-recovery/v1 +echo.keep.retention-closure/v1 +``` + +## Phase E — production replacement decision + +### Goal + +Decide whether a Keep-backed physical tier should replace Echo `DiskTier` for a +specific production posture. + +### Replacement target + +The candidate target is Echo's filesystem blob implementation, not: + +- Echo `BlobHash`; +- WSC content hashes; +- `SemanticBlobCoordinate`; +- Echo causal anchors; +- the materialization outbox; +- all in-memory `echo-cas` use. + +### Required parity and improvement gates + +- identical Echo content hashes for all fixtures; +- identical WSC import and replay outcomes; +- no semantic-coordinate drift; +- no causal-evidence drift; +- precise absence and failure classification; +- deterministic binding and layout selection; +- stable retry outcomes; +- restart recovery from every admitted crash state; +- retained-closure verification; +- bounded transient memory relative to the declared operation; +- explicit supported-platform posture; +- migration plan for existing `DiskTier` bytes and semantic descriptors; +- no wire-format change unless separately authorized. + +### ADR gate + +A production decision must record: + +- Echo and Keep identity ownership; +- canonical binding shape and encoding; +- binding persistence and recovery owner; +- full-blob versus range proof scopes; +- retention ownership; +- outbox metadata binding; +- toolchain policy; +- platform support; +- durable read and publication contracts; +- error taxonomy; +- migration and rollback; +- compatibility and release evidence. + +## Replay-safe write semantics + +The integration must distinguish three different claims: + +```text +Authorized + Echo committed an intent permitting the external effect. + +Physically published + Keep returned exact publication evidence under its durability contract. + +Causally observed + Echo committed an observation matching the authorized effect and retained + evidence. +``` + +A Keep publication is not automatically an Echo causal fact. An Echo intent is +not proof that physical bytes exist. The adapter's binding is the bridge, not a +shortcut around either system's law. + +## Concurrency model + +### Write authority + +Use one writer per Keep store during all integration phases unless Keep adopts a +stronger deliberate concurrency model. + +Do not introduce one-writer-per-blob locks as an adapter invention. Keep catalog +and retention publication serialize store-level generations and therefore +require store-level authority. + +### Reader posture + +Many readers are allowed when each read is pinned to an immutable admitted +snapshot or generation. + +A reader that begins under generation `N` must not silently switch to generation +`N+1` during the operation. + +### Binding multiplicity + +The binding index must model: + +```text +one Echo logical content claim + ↔ one verified Keep BlobId + ↔ one or more verified Keep LayoutIds +``` + +Selection of a layout must be explicit or deterministic. Keep's convenience +rule of selecting the lowest canonical committed `LayoutId` is acceptable for a +read API, but evidence must name the exact layout actually used. + +### Concurrency tests + +- a second writer receives a typed refusal; +- concurrent retries do not publish duplicate conflicting state; +- an exact publication retry returns the same synchronized outcome; +- a pinned reader completes against its original generation while a successor + publishes; +- reader completion order does not alter bytes, bindings, or receipts; +- alternate valid layouts do not create a false logical-identity conflict; +- conflicting verified logical identities obstruct deterministically. + +## Streaming and memory test harness + +Create an interop harness that tests source stream → Keep → reconstructed stream +→ Echo verifier with explicit resource accounting. + +### Inputs + +- deterministic virtual pseudo-random data with O(1) source state; +- exact empty, tiny, boundary, and very large inputs; +- configurable short-read and interrupted-read schedules; +- deterministic nearby edits; +- corruption injection for chunks, layouts, catalogs, bindings, and metadata. + +### Sinks + +- counting writer; +- bounded writer; +- short writer; +- interrupted writer; +- Echo raw-hash and exact-length verifier; +- optional incremental WAL decoder for Phase B. + +### Metrics + +- bytes requested and emitted; +- maximum read chunk; +- maximum write chunk; +- allocation count; +- maximum single allocation; +- transient high-water above retained result state; +- adapter-owned retained bytes; +- Keep-owned retained bytes; +- duplicate raw full-buffer allocations; +- selected chunk count for range reads. + +OS RSS may be collected as benchmark information but must not be the sole +deterministic correctness gate. + +### Memory claims by layer + +- Keep detector and reader scratch state may be bounded independently of blob + size. +- `ReferenceStore` retained chunk material grows with admitted content and must + be accounted separately. +- WSC streaming may remove a duplicate raw input buffer while decoded recovery + evidence still grows with frame payload count. +- A true end-to-end bounded import claim requires a compact incremental recovery + fold. + +## Proof-scope tests + +The test suite must make the following substitutions impossible: + +- `RangeReadReceipt` used as complete-blob proof; +- `PublishedBlob` used as durability proof; +- `RetentionAnchor` used as closure proof; +- parsed `BlobId` used as content-presence proof; +- Echo semantic coordinate used as physical-presence proof; +- Keep `BlobId` used as Echo WSC content-hash proof; +- Echo content hash used as Keep `BlobId` proof; +- materialization intent used as completion proof; +- Keep publication used as Echo causal observation. + +## Capability ledger + +| Capability | Phase | Claim | +| ------------------------------------ | ----- | ------------------------------------------------------------------------------------------------------------------------------------- | +| `echo.keep.same-bytes-binding/v1` | A | Complete source and reconstruction verification admit one Echo ↔ Keep logical binding and exact layout realization. | +| `keep.echo.identity-agreement/v1` | A | Cross-repository conformance evidence closes Keep's declared Echo identity-boundary milestone without equating the two identity laws. | +| `echo.keep.reference-reconstruct/v1` | A | Keep reference reconstruction reproduces the Echo content claim exactly. | +| `echo.keep.reference-range/v1` | A/B | Exact ranges are authenticated under their deliberately narrow proof scope. | +| `echo.keep.wsc-stream-validation/v1` | B | WSC validation no longer requires a CAS port returning a full `Vec`. | +| `echo.keep.durable-reconstruct/v1` | C | A pinned durable Keep generation reconstructs an exact retained blob to caller-owned output. | +| `echo.keep.durable-range/v1` | C | A pinned durable Keep generation reads an exact authenticated range without whole-blob materialization. | +| `echo.keep.restart-read/v1` | C | Restart recovers a lawful generation capable of the same verified read. | +| `echo.keep.outbox-reconciliation/v1` | D | Echo outbox recovery reconciles matching, missing, or conflicting Keep publication evidence. | +| `echo.keep.binding-recovery/v1` | D | The exact identity binding remains recoverable after process death. | +| `echo.keep.retention-closure/v1` | D | Required Keep anchors have verified retained closure under the selected generation. | +| `echo.keep.platform-posture/v1` | C/D | Supported and unsupported filesystem postures are explicit and typed. | +| `echo.keep.disk-tier-replacement/v1` | E | Keep can replace Echo `DiskTier` for the declared production posture without semantic or replay drift. | + +## Open questions + +### Where should the binding live? + +Deferred. The branch may use an in-memory test type. Production persistence is +an ADR decision. + +### Does the Echo semantic coordinate belong inside the logical binding? + +No. The logical binding proves same-byte correspondence. Semantic coordinates +belong in an evidence envelope above it so multiple meanings may lawfully refer +to the same content. + +### Can one Echo content claim map to multiple Keep layouts? + +Yes, provided every layout independently names and reconstructs the same Keep +`BlobId` and Echo content claim. The binding model must preserve this +one-to-many realization relationship. + +### Should WSC migrate before the general `echo-cas` API? + +Yes. WSC is the demonstrated consumer with the currently broken +`Option>` contract. Generalize only after another consumer proves the +same capability boundary. + +### Should the first adapter return `Read` or write into a caller-owned sink? + +Either can work. A sink-oriented port makes final verification and provisional +output discipline explicit and avoids committing prematurely to reader +lifetimes. The chosen API must remain fallible and must not expose success before +complete Echo verification. + +### Is one writer per blob sufficient? + +No. Use one writer per Keep store because catalog and retention publication are +store-generation transitions. + +### Does Keep need more durability work before the spike? + +Not for Phase A or B. `ReferenceStore` is the correct oracle there. Production +requires a streamable durable logical-read surface and completion of the exact +retention and recovery posture selected by the integration. + +### Does removing `Vec` make WSC import bounded-memory? + +Not by itself. It removes one mandatory raw materialization. The retained WAL +recovery result still requires separate redesign for a true end-to-end bound. + +## Decision for this exploration branch + +No ADR is introduced for this branch. + +The immediate work is: + +1. Build the standalone Rust 1.96 identity-corroboration spike. +2. Define the private exploratory logical and realization binding types. +3. Add same-source and reconstructed-source dual-verification tests. +4. Prove lawful alternate-layout handling and conflicting-logical-identity + obstruction. +5. Introduce a narrow fallible WSC streaming port for retained-material + validation. +6. Implement Echo memory and disk adapters through that port. +7. Implement the Keep `ReferenceStore` adapter using admitted bindings. +8. Add full reconstruction and exact range paths with explicit proof scopes. +9. Measure allocation and transient-memory behavior without overstating total + process bounds. +10. Only after those claims pass, design the durable Keep logical-read and + binding-persistence boundary. + +The branch must not claim that Keep replaces `echo-cas`, that Echo and Keep use +the same identity, or that production durability and retention are complete. + +## Final architectural hypothesis + +The durable destination, if the evidence supports it, is: + +```text +Echo semantic coordinate ++ Echo content-only identity ++ exact byte length + │ + ▼ +independently verified Echo ↔ Keep logical binding + │ + ▼ +one or more exact Keep realizations +BlobId + LayoutId + │ + ▼ +Keep publication, recovery, and retained-closure evidence + │ + ▼ +Echo materialization observation and causal replay posture +``` + +This is stronger than replacing one CAS implementation with another. + +It gives Echo and Keep independent identity laws, independent evidence, and a +lawful correspondence between them: + +> Echo can prove why the artifact matters. Keep can prove which exact bytes and +> reconstruction plan exist. The adapter can prove that both systems observed +> the same material. From 60956c55e817a8ce62f2a9280901e2c2490d13a0 Mon Sep 17 00:00:00 2001 From: James Ross Date: Sun, 9 Aug 2026 20:39:08 -0700 Subject: [PATCH 2/7] docs: adopt Echo Keep physical content boundary --- docs/README.md | 1 + .../echo-keep-physical-content-boundary.md | 300 +++++ .../echo-keep-physical-cas-interop-plan.md | 1175 ++++------------- 3 files changed, 548 insertions(+), 928 deletions(-) create mode 100644 docs/architecture/echo-keep-physical-content-boundary.md diff --git a/docs/README.md b/docs/README.md index 388aa1a9..6d24de6c 100644 --- a/docs/README.md +++ b/docs/README.md @@ -20,6 +20,7 @@ causal history. Git history is the archive; GitHub owns live work and status. - [There Is No Graph](architecture/there-is-no-graph.md) - [Continuum transport](architecture/continuum-transport.md) - [Echo optics adapter notes](architecture/echo-optics-adapter-notes.md) +- [Echo × Keep physical content boundary](architecture/echo-keep-physical-content-boundary.md) ## Case Studies diff --git a/docs/architecture/echo-keep-physical-content-boundary.md b/docs/architecture/echo-keep-physical-content-boundary.md new file mode 100644 index 00000000..0cc974d1 --- /dev/null +++ b/docs/architecture/echo-keep-physical-content-boundary.md @@ -0,0 +1,300 @@ + + + +# Echo × Keep Physical Content Boundary + +- **Status:** Accepted for experimental conformance; production adoption is + not accepted. +- **Decision date:** 2026-08-09 +- **Implementation posture:** No Echo physical-content port or Keep adapter is + implemented on this branch. + +## Decision + +> Keep establishes physical content truth. Echo decides what that truth means +> causally. + +Keep may become an experimental physical backend for Echo. It must not replace +`echo-cas` until the identity bridge, consumer port, durable read aperture, +cross-store crash protocol, migration, and rollback gates have executable +evidence. + +Echo owns the substitution boundary: + +```text +echo-core + │ + ▼ +Echo-owned physical-content port + ├──────────────▶ existing echo-cas adapter + │ + └──────────────▶ echo-keep adapter ─────────▶ Keep +``` + +Keep must not depend on Echo, expose Echo concepts, or adopt the current weak +`BlobStore` contract as its foundational public API. Keep-specific types remain +inside the adapter and a physical-evidence envelope; they do not cross ordinary +Echo semantic APIs. + +The [integration and migration plan](../plans/echo-keep-physical-cas-interop-plan.md) +owns change-local sequencing and evidence gates. This page owns the boundary +regardless of which implementation phase is active. + +## Division of responsibility + +| Echo owns | Keep owns | Adapter owns | +| --------------------------------------------- | ------------------------------------ | ------------------------------------------------------ | +| Echo content identity and WSC hashes | Versioned logical `BlobId` | Witnessed Echo-ID to Keep-ID binding | +| Semantic coordinates and application meaning | `LayoutId` and physical realizations | Translation between Echo requests and Keep coordinates | +| Causal basis, law, and observer aperture | Chunking, segments, and catalogs | Quarantined output and receipt validation | +| Materialization intent and observation | Physical publication and recovery | Physical-evidence envelope for Echo | +| Retry, admissibility, and history publication | Retention generations and closure | Cross-store operation reconciliation | + +Keep receipts never grant Echo authority. Echo observations never prove Keep +presence, retention, or durability without matching Keep evidence. + +```text +Keep receipt + "These exact bytes were authenticated from this physical evidence." + +Echo observation + "Under this causal basis and law, this physical result matters this way." +``` + +One must not silently become the other. + +## Echo-owned consumer shape + +The first substitution boundary is a fallible, complete-object, +caller-output-oriented port. The following pseudostructure names semantic +obligations, not a frozen Rust ABI: + +```rust +fn reconstruct( + &self, + view: &PhysicalContentView, + target: EchoContentId, + output: &mut dyn std::io::Write, +) -> Result; +``` + +`PhysicalContentView` is an Echo-owned opaque capability. A Keep adapter binds +it to one pinned Keep generation or immutable view. An `echo-cas` adapter may +support a weaker posture initially, but it must report unsupported evidence +rather than manufacture generation or absence claims. + +The initial port includes: + +- complete-object reconstruction; +- expected staged ingestion; +- explicit physical publication; +- authenticated success receipts; +- evidenced refusal only where the backend can prove it; +- operational failures that carry no content claim; +- quarantined output semantics. + +The initial port excludes: + +- range reads; +- an optional range argument; +- consumer-controlled Keep layouts unless required by a real consumer; +- generalized compaction controls; +- a broad retention-management trait; +- implicit backend fallback. + +Echo may retain a bounded materializing helper implemented over this port. The +helper is not Keep's foundational contract and must require an explicit byte +limit. + +## Output visibility + +An ordinary `Write` sink can fail after accepting a prefix. A failed Keep call +therefore does not prove that the sink is untouched. + +The Echo adapter enforces: + +> No complete receipt, no application-visible bytes. + +It reconstructs into quarantine and reveals or promotes the result only after +the complete receipt and Echo identity both verify: + +```text +backend reconstruction + │ + ▼ +unpublished temporary artifact + │ + ├── failure ──▶ discard; reveal nothing + │ + └── receipt ──▶ verify Echo identity; promote +``` + +Quarantine may be a bounded memory buffer for small content or a temporary +artifact for larger content. The rule must not imply unbounded whole-object +memory allocation. + +## Decisions and failures + +Content truth and operational success are separate result planes: + +```rust +enum ReconstructionDecision { + Authenticated(ReconstructionReceipt), + Refused(ReconstructionRefusalReceipt), +} + +enum ReconstructionOperationError { + InputOutput, + ResourceLimit, + Cancellation, + CapabilityUnavailable, +} +``` + +These names are conceptual and not a frozen ABI. + +An authenticated refusal must bind enough evidence to support its proposition. +Absence is evidenced only when the pinned view is complete enough to prove +non-membership. A timeout, unreadable catalog, exhausted resource limit, +cancellation, or unavailable capability teaches Echo nothing about content +truth. + +Echo may admit an evidenced refusal as an observation only under an Echo law +that explicitly accepts that refusal class and its physical aperture. A Rust +error alone is not a witnessed refusal. + +## Identity bridge + +Echo `BlobHash` and Keep `BlobId` are distinct typed identities. They must not +be cast, substituted, or equated because both currently use 32-byte BLAKE3 +digests. + +The adapter establishes their relation by applying both identity laws to one +exact source stream and then verifying reconstruction: + +```text + ┌── Echo identity law ──▶ EchoContentId +one exact stream ───┤ + └── Keep identity law ──▶ Keep BlobId +``` + +The resulting logical binding retains both identities and the exact logical +length. A separate realization binding names one or more lawful Keep layouts. +The persisted carrier, canonical bytes, and digest domain remain undecided. + +Keep's `stage_expected` can verify an expected Keep `BlobId`; it cannot by +itself prove an Echo-to-Keep identity relation. The adapter owns the second +identity calculation and the binding witness. + +## Semantic observation and physical evidence + +Ordinary Echo semantics and Keep realization provenance remain separate: + +```rust +struct ContentObservation { + content_id: EchoContentId, + logical_length: ByteLength, + proof_scope: CompleteObject, +} + +struct PhysicalEvidence { + keep_blob_id: KeepBlobId, + keep_generation: KeepGenerationId, + keep_layout_id: KeepLayoutId, + receipt_version: ReceiptVersion, +} +``` + +These are conceptual ownership examples, not frozen records. + +Different lawful Keep layouts for the same exact bytes produce the same +ordinary Echo content observation and different physical provenance. Echo must +not hash `LayoutId` into ordinary semantic state merely because a receipt +contains it. + +If no exact layout was requested, the Keep adapter may select any admitted +realization under a deterministic policy. If an exact layout was requested, it +must use that layout or refuse; it may not fall back to another layout. + +## Physical observer aperture + +Each Echo execution frame or work unit should hold one physical-content view: + +```text +Echo execution frame + ├── causal basis and graph view + └── PhysicalContentView + └── pinned Keep generation capability +``` + +Every physical read in the frame resolves through that same view. The adapter +must retain the underlying evidence for the view's lifetime and must not +silently advance it. + +New content produced during a frame remains staged or appears through an +explicit Echo-owned overlay. It becomes part of a later physical view only +after the causal and physical publication protocol completes. + +This prevents mixed-generation reads, mutable-`HEAD` coupling, and receipts +whose physical evidence aperture is unclear. + +## Cross-store invariant + +Keep publication and Echo WAL publication are separate durable transitions. +Their integration requires an explicit operation identity, provisional +physical retention, and recovery reconciliation. + +The governing invariant is: + +> Orphaned physical content is acceptable. A committed Echo reference to +> unavailable content is not. + +The migration plan owns the crash matrix. The read port must not conceal the +cross-store state machine or convert “write Keep, then write Echo” into an +implicit protocol. + +## Fallback + +During an explicitly declared migration posture, Echo may consult the old +backend after a Keep miss only when policy records the fallback, re-verifies +the bytes, and schedules or performs an explicit backfill. + +Once Keep is authoritative, a Keep refusal is not permission to ask the old +store silently. Silent fallback would conceal the absence, corruption, and +retention failures the boundary exists to expose. + +## Current source posture + +Echo's current [`BlobStore`](../../crates/echo-cas/src/lib.rs) is synchronous +and materializing. Its `get` path collapses ordinary absence into `Option` and +returns a complete `Arc<[u8]>`. [`DiskTier`](../../crates/echo-cas/src/disk.rs) +already needs a separate fallible API, demonstrating that the existing trait +is too weak for durable physical evidence. + +The current [WSC CAS port](../../crates/warp-core/src/wsc/store.rs) returns +`Option>` and likewise collapses failure posture while requiring full +materialization. + +[ADR 0020](../adr/0020-retained-reading-storage-and-proof-boundary.md) already +requires byte identity, semantic reading identity, and proof identity to remain +distinct. This boundary extends that law to the Keep integration without +changing WSC wire identity. + +## Production adoption gate + +A production Keep backend, persisted binding format, changed WSC identity, or +permanent `echo-cas` replacement requires a separate accepted ADR after the +conformance and crash evidence exists. That decision must govern: + +- exact Echo and Keep identity preimages; +- binding carrier, encoding, versioning, and recovery; +- physical-view lifetime and retention; +- success and refusal receipt formats; +- output quarantine and promotion; +- cross-store crash recovery; +- toolchain and supported platforms; +- backfill, rollback, and fallback removal; +- compatibility and release posture. + +Until then, Keep is an experimental Echo backend, not Echo's sole durable +content authority. diff --git a/docs/plans/echo-keep-physical-cas-interop-plan.md b/docs/plans/echo-keep-physical-cas-interop-plan.md index a558f308..4749f296 100644 --- a/docs/plans/echo-keep-physical-cas-interop-plan.md +++ b/docs/plans/echo-keep-physical-cas-interop-plan.md @@ -1,1029 +1,348 @@ -# Echo × Keep Physical CAS Interop and Identity Corroboration Exploration (No ADR) - -## TL;DR - -| Question | Exploratory answer | -| ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| What are we evaluating? | Whether Keep can become Echo's physical retained-content engine for CAS-addressed materialized readings, WSC import, reconstruction, range reads, and retention workflows. | -| What is the central hypothesis? | The same exact bytes can be independently admitted under Echo's content-only hash law and Keep's versioned logical-identity law, producing an explicit correspondence witness without pretending the two identities are equal. | -| What remains Echo-owned? | Semantic coordinates, causal truth, WSC content hashes, authorization, replay policy, materialization intents, and observations. | -| What may become Keep-owned? | Keep's independent logical byte identity, chunking, layouts, physical publication, restart recovery, retained closure, range reads, and eventually compaction or garbage collection. | -| What does the adapter own? | Proof that one exact byte stream satisfied both identity systems, plus translation between Echo content claims and exact Keep reconstruction coordinates. | -| What is the first artifact? | An exploratory `EchoKeepLogicalBindingV1` for Echo hash + length ↔ Keep `BlobId`, plus `EchoKeepRealizationBindingV1` for one exact Keep `LayoutId`. | -| Are Echo `BlobHash` and Keep `BlobId` interchangeable? | No. They assert different propositions and must never be cast, substituted, or silently normalized into one identity. | -| Can one logical binding have multiple Keep layouts? | Yes. Different verified `LayoutId` values may lawfully realize the same Keep `BlobId`. A different verified Keep `BlobId` for the same exact Echo bytes is an obstruction. | -| What is the first implementation posture? | A standalone Rust 1.96 interop spike using Keep's non-durable `ReferenceStore` as the executable oracle, followed by a narrow fallible WSC streaming port. | -| What is the likely production replacement target? | Echo's `DiskTier`, not Echo's content-only hash namespace or all of `echo-cas`. | -| When is an ADR required? | Before freezing a persisted binding format, changing WSC wire identity, adopting a production Keep backend, changing durable Keep read/publication contracts, or declaring a permanent ownership boundary. | - -## Repository posture - -This is an exploratory design artifact. It records a hypothesis and a sequence -of executable tests; it does not establish a durable architectural boundary. - -No ADR is introduced by this branch. If this document is carried in -`docs/plans/` for branch-local work, it must remain explicitly non-normative and -must be removed, superseded, or converted into the appropriate durable decision -record before the underlying identity, format, public API, durability, recovery, -or ownership boundary is adopted. - -### Source-reality audit findings (2026-08-04) - -Confirmed from the current `flyingrobots/echo` and `flyingrobots/keep` source: - -- **Echo CAS remains full-materializing by design.** - - `crate::echo_cas::BlobStore` still exposes a sync `get` path. - - `MemoryTier` returns buffered bytes from memory as `Arc<[u8]>`. - - `DiskTier` still reads complete blobs and re-validates full hash before - returning a complete byte buffer. -- **WSC boundary is still `Option>` today.** - - `crates/warp-core/src/wsc/store.rs` continues to require full-bytes - materialization for `cas_blob_bytes`. - - `Warp CLI` still uses an unavailable CAS stand-in for this path (`UnavailableCasStore`), - so any streaming replacement must not assume immediate CLI parity. -- **Keep split is real and relevant.** - - The `ReferenceStore` is a strong non-durable executable oracle. - - Keep retains a separate durable catalog/recovery stack (`Store`, `State`, - writer authority, retained closures, replay states) and cannot be treated as - one replacement for current `echo-cas` behavior. -- **Toolchain mismatch remains unsolved.** - - `echo-keep` currently resolves to a 1.96+ Rust profile (`flyingrobots/keep` - requirement), while the working echo branch is pinned to 1.90. -- **Viability status is unchanged:** - - The plan remains credible for exploration and spike proving. - - It is **not yet production-accurate** because streaming WSC CAS APIs and - Keep durable logical-read integration are both still design-level commitments - without code lockstep in this repo. - -## Objective - -Decide whether Keep can replace or host the physical storage responsibilities -currently associated with `echo-cas` for CAS-addressed materialized readings, -WSC import, reconstruction, range reads, and retention workflows, without -breaking Echo's invariants for causal truth, replay safety, deterministic -identity, or semantic authority. - -The primary exploration is **independent identity corroboration**: - -> One exact byte stream is independently admitted under Echo's content-only -> BLAKE3 identity and Keep's versioned logical `BlobId`; the adapter retains a -> witness that both systems observed the same bytes without claiming that their -> identities are equal. - -This changes the purpose of the integration. The goal is not CAS API parity and -not identity replacement. The goal is a lawful division of responsibility: - -- Echo explains **why the bytes matter**. -- Keep proves **which exact bytes and reconstruction plan exist**. -- The adapter witnesses **that both identity systems admitted the same - material**. - -## Primary architecture - -```mermaid -flowchart TD - S[Echo semantic coordinate
why the bytes matter] - C[Echo content claim
BlobHash + byte length] - L[Logical corroboration
Echo bytes ↔ Keep BlobId] - R[Keep realization
BlobId + LayoutId / RetentionAnchor] - P[Keep physical evidence
publication + recovery + retained closure] - O[Echo causal observation
materialization effect observed] - - S --> C - C --> L - L --> R - R --> P - P --> O -``` - -The evidence chain is intentionally layered: - -1. An Echo semantic coordinate names the question the bytes answer. -2. An Echo content claim names the raw bytes under Echo's content-only law. -3. A logical corroboration witness binds that claim to a Keep `BlobId` obtained - from the same exact bytes. -4. A realization witness names one exact Keep `LayoutId` capable of - reconstructing those bytes. -5. Keep publication and retention evidence proves the physical realization is - available under Keep's own laws. -6. Echo records the authorized external effect and its verified observation in - causal history. - -No lower layer acquires the authority of the layer above it. - -## Corroboration ladder - -The integration should expose progressively stronger evidence postures instead -of one vague `stored` or `verified` boolean. - -| Level | Required evidence | Claim established | Explicit nonclaim | -| ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | ------------------------------------------------------------------ | -| C0 — dual-coordinate candidate | Echo content hash and exact length computed while Keep ingests the same complete source stream, plus the candidate Keep `BlobId` and `LayoutId` | Both identity calculations observed one complete source stream during the operation | Does not prove that retained Keep state can reconstruct the bytes | -| C1 — retained reconstruction corroborated | C0 plus complete Keep reconstruction through the named layout, a matching `ReconstructionReceipt`, and an independently recomputed Echo hash and length | The named Keep realization reconstructs exact bytes satisfying both identity laws | Does not prove process survival or durable publication | -| C2 — durable physical corroboration | C1 plus exact Keep publication, pinned-generation, and retained-closure evidence | A durable Keep state supports the same verified logical correspondence | Does not grant Echo causal authority | -| C3 — causally observed materialization | C2 plus a committed Echo materialization observation matching a previously committed intent | Echo has causally witnessed the verified physical materialization | Does not make Keep publication history part of Echo causal history | - -The exploratory binding type represents C1, not merely C0. Phase C must produce -C2 evidence. Phase D must reconcile C2 into C3 through Echo's existing outbox. -No API may accept a weaker level where a stronger level is required. - -## Current Echo state (as of branch `echo-keep`) - -- `echo-cas` owns a content-only CAS identity: - - `BlobHash = BLAKE3(bytes)` with no domain prefix; - - identical bytes intentionally share one hash regardless of semantic role; - - semantic meaning is carried separately by `SemanticBlobCoordinate` and - retained descriptors. -- The public `BlobStore` interface is synchronous and materializing: - - `get` returns `Option>`; - - the source comments already identify this return shape as unsuitable for - future disk and cold streaming tiers; - - `MemoryTier` is an infallible in-process implementation. -- `DiskTier` is fallible and filesystem-backed: - - it stores bytes by content-only hash; - - `get` reads the entire blob into memory and re-verifies the hash; - - it provides ordinary process-restart persistence; - - it does not currently expose a complete publication, synchronization, - crash-classification, recovery, or persistent-retention protocol comparable - to Keep's durable store; - - its pin set is process-local. -- The WSC CAS bridge remains full-materialization oriented: - - `WscCasBlobStorePort::cas_blob_bytes` returns `Option>`; - - CAS-addressed WSC import loads full bytes, recomputes Echo's content hash, - checks length, and returns the full allocation; - - `warp-cli` currently supplies an unavailable CAS implementation for these - paths. -- Echo already has the correct causal side-effect protocol: - - `MaterializationIntentRecord` records authorized external work; - - `MaterializationObservationRecord` records verified completion; - - outbox recovery distinguishes already observed, matching existing - artifact, missing artifact, digest mismatch, metadata mismatch, and missing - retained material. - -### Consequence - -Echo already has the semantic identity and causal protocol that must survive the -integration. Its weak point is the physical blob interface and implementation, -not the meaning of its content hashes. - -## Current Keep state (as of the attached worktree) - -Keep exposes two materially different layers that must not be conflated. - -### Reference adapter - -`ReferenceStore` is a bounded, deterministic, in-memory executable model: - -- ingestion uses a fixed read buffer and bounded chunk-detector state; -- staging is invisible until explicit `commit`; -- publication returns `PublishedBlob { BlobId, LayoutId }`; -- reconstruction authenticates every selected chunk, replays the registered - storage profile, verifies the complete `BlobId`, and writes to a caller-owned - `Write`; -- exact range reads load only overlapping chunks and return a deliberately - narrower receipt; -- process death loses all committed state; -- stored chunk material may grow with admitted logical content, so the adapter - is not an O(1)-total-memory backend merely because its detector and read - buffers are bounded. - -### Durable store and retention machinery - -Keep also has substantial durable infrastructure beyond `ReferenceStore`: - -- immutable segments and exact record formats; -- catalog generations and publication heads; -- platform admission and a store-wide writer authority; -- synchronized stage, immutable-pool, catalog, and `HEAD` publication; -- restart-loaded catalog snapshots; -- typed recovery inventory, stage classification, completion, discard, - resumption, and next-head finalization; -- process-death crash-matrix evidence; -- version-2 retention anchors, roots, manifests, heads, closure verification, - transition planning, and publication vocabulary. - -The remaining gap is not "make Keep durable from scratch." It is to expose a -high-level, streamable, retained-logical-blob capability that can satisfy Echo's -consumer boundary without importing Echo semantics into Keep core. - -### Keep proof scopes - -- A `BlobId` is a versioned logical identity for exact finite bytes. Parsing a - coordinate does not prove that matching content exists. -- A `PublishedBlob` identifies a visible `BlobId` and one exact `LayoutId` in - the reference adapter. -- A `ReconstructionReceipt` proves complete authenticated reconstruction under - one layout. -- A `RangeReadReceipt` proves only the requested bytes came from authenticated - overlapping chunks under an admitted layout. It does not prove the complete - blob, unrequested chunks, or storage-profile boundaries. -- A `RetentionAnchor` combines `BlobId` and `LayoutId`, but the anchor alone - does not prove closure or byte availability; retained closure must be verified - separately. - -### Toolchain consequence - -Echo is pinned to Rust 1.90 while Keep declares Rust 1.96. A Keep dependency -cannot simply be added to Echo's ordinary workspace without first making an -explicit toolchain decision. The exploration should therefore begin in a -standalone Rust 1.96 interop workspace or dedicated higher-toolchain CI lane. - -## Core invariants - -### 1. Identity laws remain distinct - -Echo's `BlobHash` and Keep's `BlobId` are not aliases and do not have a lawful -cast between them. This exploration binds Echo's current content-hash law to -Keep `BlobId` version 1 specifically. - -- Echo identifies `BLAKE3(bytes)` directly. -- Keep identifies exact finite logical bytes under a versioned, - length-committing identity law. - -Both use BLAKE3 internally, but they assert different propositions. - -### 2. Corroboration requires the same exact bytes - -A binding may be admitted only after one exact byte stream has been observed by -both identity calculations and the Keep realization has passed complete -reconstruction verification. - -A parsed `BlobId`, a filename, a catalog entry, a `LayoutId`, or an existing -mapping is insufficient by itself. - -### 3. Logical identity and physical realization remain separate - -The logical corroboration is: - -```text -Echo content hash + exact byte length ↔ Keep BlobId -``` +# Echo × Keep Physical Content Integration Plan -A Keep `LayoutId` is one physical/logical reconstruction realization of that -Keep `BlobId`. +- **Status:** Change-local implementation and evidence plan. +- **Decision:** Architecture approved; production replacement not approved. +- **Historical source:** The original feasibility exploration is preserved in + Git commit `fa943a6c0`. +- **Boundary owner:** + [Echo × Keep physical content boundary](../architecture/echo-keep-physical-content-boundary.md) +- **Keep contract:** `flyingrobots/keep` document + `docs/invariants/authenticated-reconstruction/README.md` -Multiple independently verified layouts may lawfully realize the same logical -binding. The adapter must not treat alternate valid layouts as an identity -collision. +This plan owns sequencing, conformance evidence, and cutover gates. It does not +own the durable architecture boundary, live priority, or release status. The +Draft pull request and GitHub work items own current progress. -### 4. Semantic authority remains in Echo +## Verdict -Keep must not infer or own: +Keep is ready to become an experimental Echo backend. It is not ready to +become Echo's sole durable content authority, and `echo-cas` must not be +removed yet. -- semantic coordinates; -- causal authority; -- WSC basis meaning; -- admission policy; -- replay decisions; -- materialization authorization; -- application retention policy. +The next milestone is not “make Keep implement `BlobStore`.” It is: -Keep reports only the physical evidence its own contracts support. +> Give Echo a receipt-bearing physical-content boundary, wrap `echo-cas` +> behind it, then prove Keep conforms. -### 5. Physical claims remain in Keep +## Scope -Echo must not pretend that a content hash, semantic coordinate, or causal fact -proves physical presence, durability, retained closure, or recoverability. +The first integration proves: -Those claims require Keep evidence. +- complete-object reconstruction; +- expected staged ingestion; +- explicit publication; +- same-source Echo and Keep identity corroboration; +- complete reconstructed-source corroboration; +- quarantined output visibility; +- authenticated success versus evidenced refusal versus operational failure; +- backend-neutral behavior across existing `echo-cas` and Keep's + non-durable `ReferenceStore`. -### 6. Proof scope is represented explicitly +The first integration does not claim: -Whole-blob, range, publication, retained-closure, and causal-observation claims -must remain distinct. A narrow receipt cannot be promoted into a stronger claim -by convention or naming. +- durable Keep authority; +- range-read support at the Echo port; +- a persisted identity-binding format; +- end-to-end bounded WSC recovery memory; +- production retention or crash recovery; +- `DiskTier` replacement; +- `echo-cas` removal. -### 7. Failure is fail-closed and typed +## Milestone 1 — identity-bridge decision -Absence, I/O failure, corruption, identity disagreement, layout disagreement, -unsupported platform, resource refusal, stale publication state, and missing -retained closure must remain distinguishable. +Before adapter code, record the exact identity relation: -### 8. Causal side effects use Echo's existing outbox +| Question | Required evidence | +| ---------------------------------------------------- | ---------------------------------------------------- | +| What fields form Echo's current content identity? | Exact typed structure and canonical input bytes | +| What algorithm and domain does Echo hash? | Source anchor and golden fixtures | +| Does Echo identity include logical length? | Explicit yes or no | +| Can an Echo ID become a Keep `BlobId` without bytes? | Proof or explicit no | +| Is the conversion one-to-one? | Argument over exact preimages plus mutation fixtures | +| Which identity remains in Echo WAL and WSC? | Explicit ownership decision | +| What happens when versions or algorithms differ? | Typed refusal; no implicit conversion | +| Are both identities retained during migration? | Explicit carrier posture | -The integration must reuse Echo's materialization intent, idempotency, -observation, and recovery posture. It must not invent an independent causal -write protocol for CAS effects. +The expected initial answer is that both identities are retained and their +relation is established from the same exact bytes. No 32-byte digest cast is +lawful. -### 9. WSC identity does not change during exploration - -WSC content hashes remain Echo content-only hashes. Keep coordinates are -corroborating and realization evidence, not replacement wire identities. - -## Exploratory evidence artifacts - -The initial spike should separate logical corroboration from physical -realization with two in-memory typed values: +The exploratory logical binding is private and noncanonical: ```rust -pub struct EchoKeepLogicalBindingV1 { - pub echo_content_hash: [u8; 32], - pub byte_len: u64, - pub keep_blob_id: keep::BlobId, -} - -pub struct EchoKeepRealizationBindingV1 { - pub logical: EchoKeepLogicalBindingV1, - pub keep_layout_id: keep::LayoutId, +struct EchoKeepLogicalBindingV1 { + echo_content_id: EchoContentId, + keep_blob_id: KeepBlobId, + logical_length: ByteLength, + binding_contract: BindingContractVersion, } ``` -These are conceptual test types, not yet frozen public APIs or persisted -formats. - -### Interpretation - -- `EchoKeepLogicalBindingV1` is the same-byte corroboration claim. -- `EchoKeepRealizationBindingV1` names one exact Keep layout that realizes the - corroborated logical bytes. -- `logical.keep_blob_id + keep_layout_id` can be viewed as the Keep-side - `RetentionAnchor` coordinate. -- One logical binding may have multiple independently verified realization - bindings. -- Echo semantic coordinates remain above the logical binding and may point to - the same content through different semantic roles. -- Keep publication and retained-closure evidence remain below the realization - binding and must be attached separately when stronger physical claims are - required. - -### Admission rules - -A logical binding and one realization binding are admitted only when all of the -following hold: - -1. Echo's raw content hash and exact length were computed from the complete - source stream. -2. Keep staged and committed that same source stream and returned the named - `BlobId` and `LayoutId`. -3. Keep completely reconstructed the exact named layout. -4. The reconstructed bytes independently reproduced the expected Echo content - hash and length. -5. The Keep reconstruction receipt named the same `BlobId` and `LayoutId` as - the candidate realization binding. - -### Conflict rules - -After complete verification: - -- same Echo hash and length under the same declared Keep identity version, - different Keep `BlobId` → **identity corroboration obstruction**; -- same Keep `BlobId`, different Echo hash or length → **identity corroboration - obstruction**; -- same logical corroboration, different valid `LayoutId` → **lawful alternate - realization**; -- same `LayoutId`, different Keep `BlobId` → **layout identity obstruction**; -- parsed coordinates without verified bytes → **unproven candidate**, not a - binding. - -### Deferred format decision - -This branch must not freeze: - -- canonical binding bytes; -- a binding digest domain; -- a persistent mapping database; -- a WSC field; -- a Keep durable record; -- a public Rust ABI. - -Those choices affect identity and compatibility and therefore require a durable -decision record before adoption. - -## Gap matrix - -| Gap | Severity | Required closure | -| ----------------------------- | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | -| Identity correspondence | Critical | Prove same-byte corroboration without collapsing Echo `BlobHash` into Keep `BlobId`. | -| Interface shape | Critical | Replace `Option>` at the WSC consumer boundary with a fallible bounded streaming or write-through capability. | -| Binding lookup | Critical | Resolve Echo content claims to exact Keep logical and layout coordinates through verified bindings. | -| Durable logical reads | Critical for production | Expose caller-owned-output reconstruction from Keep's durable store without retaining every selected segment in one snapshot allocation. | -| Causal publication | Critical for production | Bind Keep publication evidence to Echo's materialization intent and observation protocol. | -| Retention ownership | Critical for production | Keep owns physical liveness and closure; Echo owns semantic retention intent and causal meaning. | -| Proof-scope separation | Critical | Prevent range, presence, publication, and closure receipts from impersonating complete identity proof. | -| Error semantics | High | Distinguish absence, I/O, corruption, mismatch, stale state, unsupported platform, and resource refusal. | -| Downstream WSC recovery shape | High | Removing the raw CAS `Vec` does not by itself bound memory if recovery retains every decoded WAL payload. | -| Toolchain | High | Resolve Echo Rust 1.90 versus Keep Rust 1.96 before workspace integration. | -| Platform posture | High | Keep's current production filesystem claims are limited to its admitted Linux ext4 profile. | -| Retention persistence | Medium during spike | Do not confuse Echo's process-local pins or Keep reference-store presence with durable retained closure. | - -## Feasibility assessment - -Keep is a strong candidate to become Echo's physical retained-content engine, -but it is not a drop-in `BlobStore` implementation and should not replace -Echo's content-only identity namespace. - -The safe target architecture is: +One logical binding may have multiple realization bindings: -```text -Echo semantic coordinates and content claims - │ - ▼ -Echo × Keep corroboration adapter - │ - ▼ -Keep logical identity, layouts, publication, recovery, and retention +```rust +struct EchoKeepRealizationBindingV1 { + logical: EchoKeepLogicalBindingV1, + keep_layout_id: KeepLayoutId, +} ``` -The likely production migration is therefore: - -- preserve Echo `BlobHash` and WSC content hashes; -- preserve Echo semantic coordinates and causal records; -- replace or retire `DiskTier` behind a new fallible physical-content port; -- retain `MemoryTier` as a useful in-process implementation; -- use Keep as the serious physical implementation only after durable logical - read, binding persistence, recovery, and retention evidence are closed. - -## Proposed staged integration architecture - -## Phase A — independent identity corroboration - -### Goal +These pseudostructures are obligation checklists, not persisted formats or +public APIs. -Prove that one exact stream can yield stable Echo and Keep identities and that a -specific Keep layout reconstructs bytes matching the Echo content claim. +### Identity witnesses -### Implementation posture +Run both identity calculations over one source stream, then reconstruct +through Keep and independently recalculate Echo identity and exact length. -- Create a standalone Rust 1.96 interop workspace or dedicated test package. -- Depend on the Echo branch and Keep worktree through local paths. -- Keep the exploratory binding type private to the spike. -- Use Keep's `ReferenceStore` as the executable oracle, not as a production - backend. - -### Ingest choreography - -1. Wrap the source in a reader that updates: - - an Echo raw BLAKE3 accumulator; - - checked exact byte-length accounting. -2. Pass that reader into Keep staging. -3. Obtain the candidate Keep `BlobId` and `LayoutId` from staged or published - work. -4. Commit the Keep reference-store stage. -5. Reconstruct the exact returned layout into a sink that independently - recomputes Echo's raw hash and length. -6. Compare the source Echo claim, reconstructed Echo claim, `PublishedBlob`, and - `ReconstructionReceipt`. -7. Admit `EchoKeepLogicalBindingV1` and one `EchoKeepRealizationBindingV1` only after all values agree. - -### Required golden cases +Required cases: - empty bytes; - small text; - deterministic binary ramp; -- exact chunk-boundary and boundary-plus-one sizes; +- chunk boundary and boundary-plus-one sizes; - large deterministic virtual input; -- nearby edited states with expected chunk reuse; -- arbitrary source read partitioning; -- short reads and interrupted reads; -- short writes and interrupted writes during reconstruction. - -### Required negative cases - -- incorrect expected Echo hash; -- incorrect byte length; -- parsed but unverified Keep `BlobId`; -- wrong Keep `BlobId` paired with a valid layout; -- wrong `LayoutId` paired with a valid `BlobId`; -- missing chunk; -- corrupt chunk; -- corrupt layout; -- profile-boundary mismatch; -- full reconstruction that produces a different Echo content hash; -- conflicting logical binding for the same fully verified Echo content claim. - -### Acceptance claim - -```text -echo.keep.same-bytes-binding/v1 -keep.echo.identity-agreement/v1 -``` - -For every admitted case, one exact byte stream produces a stable Echo content -claim, a stable Keep `BlobId` v1, and at least one exact Keep `LayoutId`; -complete reconstruction through that layout reproduces the Echo content claim -exactly. - -The first name is the adapter-facing capability. The second is the existing -Keep Golden File Worldline destination. The Keep capability must not be marked -complete merely because an in-process adapter test passes; closure requires a -reviewed cross-repository conformance corpus or equivalent independently -reproducible evidence. - -### Explicit nonclaims - -- no process survival; -- no durable binding format; -- no production retention; -- no WSC API change; -- no O(1) total process memory claim for `ReferenceStore`; -- no replacement of `echo-cas`. - -## Phase B — WSC read-path streamability - -### Goal - -Remove the mandatory whole-blob return value from CAS-addressed WSC validation -while preserving Echo content-hash and length verification. - -### Boundary - -Introduce a narrow, fallible WSC consumer port, conceptually: - -```rust -pub trait WscCasReadPort { - fn copy_verified( - &self, - claim: WscCasClaim, - output: &mut dyn std::io::Write, - ) -> Result; -} -``` - -The final API may instead expose a reader-like value, but the semantics must be -identical: - -- normal absence is distinct from failure; -- I/O is fallible; -- the complete Echo content hash is verified; -- exact length is verified; -- no success receipt exists before EOF and final verification; -- no admitted WSC semantic result escapes from provisional bytes; -- the implementation does not require a full adapter-owned `Vec`. - -### Implementations - -1. Adapter over Echo `MemoryTier`. -2. Adapter over Echo `DiskTier`. -3. Experimental Keep `ReferenceStore` adapter resolving through an admitted - `EchoKeepRealizationBindingV1`. - -### Order of conversion - -1. Convert CAS-addressed retained-material availability checks first. They load, - verify, and discard bytes and therefore provide the cleanest streaming - witness. -2. Convert CAS-addressed WAL-segment validation next. -3. Preserve byte-slice convenience APIs as wrappers around the streaming path, - not the reverse. - -### Verification rule - -A Keep reconstruction receipt is necessary but not sufficient for WSC import. -The WSC boundary must still verify Echo's raw content hash and exact length. - -### Memory claim - -Phase B may claim: - -> WSC validation no longer requires a second adapter-owned whole-segment input -> allocation. +- arbitrary short and interrupted reads; +- nearby edits with deterministic chunk reuse; +- mismatched Echo identity; +- mismatched length; +- mismatched Keep `BlobId`; +- wrong or corrupt layout; +- missing or corrupt chunk. -It must not yet claim: +Acceptance requires both routes. Same-source computation alone does not prove +retained reconstruction. Reconstructed-source computation is route-independent +evidence, not an independent BLAKE3 implementation. -> CAS-addressed WSC import is constant-memory regardless of output evidence. +## Milestone 2 — Echo physical-content port -Current WAL recovery structures retain decoded frame payloads. True bounded -end-to-end import requires a later incremental recovery fold or compact evidence -shape. +Define the Echo-owned port from the +[boundary contract](../architecture/echo-keep-physical-content-boundary.md). +Keep types must not appear in its ordinary consumer surface. -## Phase C — durable Keep logical-read capability +First implement the existing Echo backends: -### Goal +1. `MemoryTier` adapter; +2. `DiskTier` adapter; +3. any WSC retained-content consumer needed by the conformance fixture. -Expose a generic Keep-owned capability that reconstructs or range-reads an exact -retained logical blob from a pinned durable store generation into a caller-owned -writer. +This establishes that the port is consumer-shaped, not reverse-engineered +from Keep. -### Keep ownership rule +The adapters must expose their actual evidence posture. A current backend that +cannot prove pinned-view absence must return unavailable evidence rather than +convert `None` into an authenticated absence receipt. -The capability must be named and designed in Keep's vocabulary. Keep core must -remain independent of Echo, WSC, semantic coordinates, causal authority, and -application policy. +### Output witness -### Required durable capability +Inject a sink failure after a deterministic prefix and prove: -Given an admitted store snapshot and exact `RetentionAnchor`: +- no application-visible content is promoted; +- no Echo content observation is emitted; +- quarantined bytes are discarded or remain explicitly unpublished; +- operational failure does not become a refusal receipt. -- verify the anchor's layout and complete retained closure as required by the - requested proof scope; -- locate exact immutable records; -- authenticate selected chunks and layout evidence; -- reconstruct or range-read into caller-owned output; -- return a precise receipt; -- avoid loading all selected segment bytes into one durable-snapshot allocation; -- preserve pinned-generation semantics while later generations publish. +## Milestone 3 — backend-neutral conformance -### Platform posture +Add a single conformance suite that runs against the existing Echo adapters and +the experimental Keep `ReferenceStore` adapter. -The first production experiment is limited to Keep's admitted writable, -non-casefolded Linux ext4 profile. Unsupported platforms must return a typed -refusal rather than silently degrading durability or verification claims. +Mandatory laws: -### Toolchain posture +| Case | Required result | +| ------------------------------------------- | -------------------------------------------------------- | +| Same bytes through different Keep layouts | Same Echo observation; different physical provenance | +| Missing content in a complete view | Evidenced absence, not generic `None` | +| Missing content in an incomplete view | No content claim | +| Corrupt content | Integrity refusal; never fallback bytes | +| Sink failure after a prefix | No Echo observation; no promoted output | +| Range receipt where whole proof is required | Rejected by type or adapter | +| Expected identity mismatch during ingestion | No publication | +| Re-layout between independent reads | Logical observation stable; physical evidence may change | +| Exact `LayoutId` requested but unavailable | Refusal; no alternate-layout fallback | +| Operational timeout or resource refusal | No authenticated absence claim | -Choose one of the following explicitly before code enters Echo's ordinary -workspace: +The Keep adapter lives in Echo or an interop crate above both projects. Keep +must never index by Echo hash or import Echo semantics. -- keep the adapter in a Rust 1.96 boundary workspace; -- add a separate higher-toolchain CI package; -- deliberately upgrade Echo through its own change; -- lower Keep's Rust requirement only if Keep independently supports and tests - that contract. +The first Keep backend uses `ReferenceStore` only. Its process-memory state is +not durable evidence. -Toolchain drift must not be smuggled in as an incidental dependency change. +## Milestone 4 — durable Keep consumer contract -### Acceptance claims +Before durable integration, Keep must expose a consolidated generic capability +with this semantic shape: ```text -echo.keep.durable-reconstruct/v1 -echo.keep.durable-range/v1 -echo.keep.restart-read/v1 +admitted immutable Keep view ++ target BlobId ++ optional exact LayoutId ++ proof scope ++ caller-owned output +──────────────────────────── +receipt or evidenced refusal + or operational failure ``` -## Phase D — causal publication and retained evidence - -### Goal - -Bind Keep publication and retention evidence to Echo's existing materialization -outbox without creating hidden side effects or a second causal protocol. - -### Choreography - -```mermaid -sequenceDiagram - participant E as Echo - participant W as Echo WAL / Outbox - participant A as Echo × Keep Adapter - participant K as Keep - - E->>W: Commit MaterializationIntentRecord - W-->>A: Authorized effect + idempotency token - A->>K: Stage exact source bytes - K-->>A: Candidate BlobId + LayoutId - A->>K: Commit / publish - K-->>A: Publication receipt - A->>K: Reconstruct exact layout - K-->>A: Reconstruction receipt + bytes to verifier - A->>A: Verify Echo hash + length; admit binding - A-->>E: Existing artifact and binding evidence - E->>W: Commit MaterializationObservationRecord -``` +The durable operation must: -### Causal rules +- pin one immutable generation or catalog view; +- retain all required evidence for the read lifetime; +- verify retained closure required by the proof scope; +- stream exact logical bytes without one adapter-owned whole-blob allocation; +- name the generation in its receipt; +- distinguish evidenced refusal from operation failure; +- state that unsuccessful ordinary output may contain an untrusted prefix. -1. Echo commits authorization before the external Keep effect. -2. Keep staging remains invisible until its explicit commit or durable - publication point. -3. The adapter admits no binding before complete same-byte verification. -4. Echo commits an observation only after Keep publication and binding evidence - agree with the authorized effect. -5. Recovery may discover matching Keep evidence after publication but before - Echo observation and classify it as an existing matching artifact. -6. Mismatch, unavailable retained closure, unsupported platform, or ambiguous - binding obstructs replay. -7. Reads, chunk lookups, dedupe hits, and cache behavior are not themselves - causal events. +Keep owns this API in Keep vocabulary. Echo does not supply WSC, causal, +semantic, or outbox concepts to Keep core. -### Binding persistence requirement +Production experiments remain limited to Keep's explicitly admitted platform +profile. Unsupported platforms return typed posture rather than degraded +durability claims. -Production recovery cannot rely on a digest of binding bytes that are no longer -recoverable. Before Phase D is adopted, the canonical binding evidence must be -retained through a governed carrier, such as: +Echo currently uses Rust 1.90 and Keep requires Rust 1.96. The durable adapter +must make that boundary explicit through a toolchain upgrade, separate package +or CI lane, or independently supported Keep MSRV change. It must not arrive as +an incidental dependency update. -- an Echo WAL payload; -- an Echo retained evidence artifact; -- a separately governed adapter record referenced from the outbox; -- another explicitly admitted durable format. +## Milestone 5 — cross-store publication protocol -Selecting that carrier and encoding is an ADR trigger. +Keep publication and Echo WAL publication require a durable operation identity +and a recovery state machine. -### Crash matrix +The governing invariant is: -Exercise at least: +> Orphaned physical content is acceptable. A committed Echo reference to +> unavailable content is not. -```text -before Echo intent commit -after Echo intent / before Keep stage -during Keep stage -after Keep stage / before Keep publication -after Keep publication / before binding verification -after binding verification / before Echo observation -after Echo observation commit -retry after every boundary -matching existing publication -conflicting existing publication -missing retained closure -conflicting logical binding -unsupported platform -``` - -### Acceptance claims +Proposed choreography: ```text -echo.keep.outbox-reconciliation/v1 -echo.keep.binding-recovery/v1 -echo.keep.retention-closure/v1 +1. Echo assigns operation identity O. +2. Adapter establishes expected Echo and Keep identities. +3. Keep stages and verifies the exact content. +4. Keep durably publishes under a provisional anchor or lease for O. +5. Echo commits the causal reference or observation for O. +6. Keep finalizes long-term retention for O. +7. Recovery records or derives the reconciled completion posture. ``` -## Phase E — production replacement decision - -### Goal - -Decide whether a Keep-backed physical tier should replace Echo `DiskTier` for a -specific production posture. - -### Replacement target +### Crash matrix -The candidate target is Echo's filesystem blob implementation, not: +| Crash point | Lawful recovery | +| ------------------------------------------------------ | --------------------------------------------------------- | +| Before Keep publication | Discard or resume staging | +| After Keep publication, before Echo WAL | Preserve as provisional orphan; eventually collect | +| After Echo WAL, before final retention | Recover through provisional anchor and finalize | +| After retention finalization | Complete | +| Echo committed but Keep evidence missing or unprovable | Integrity obstruction; never silent fallback | +| Keep published but Echo state unreadable | Preserve provisional evidence until Echo recovery decides | -- Echo `BlobHash`; -- WSC content hashes; -- `SemanticBlobCoordinate`; -- Echo causal anchors; -- the materialization outbox; -- all in-memory `echo-cas` use. +Exercise before, during, and after every physical synchronization and Echo WAL +commit boundary. Retry must be idempotent by operation identity. -### Required parity and improvement gates +The binding carrier and canonical encoding are intentionally deferred until +this protocol identifies what recovery must retain. Freezing either requires a +production ADR. -- identical Echo content hashes for all fixtures; -- identical WSC import and replay outcomes; -- no semantic-coordinate drift; -- no causal-evidence drift; -- precise absence and failure classification; -- deterministic binding and layout selection; -- stable retry outcomes; -- restart recovery from every admitted crash state; -- retained-closure verification; -- bounded transient memory relative to the declared operation; -- explicit supported-platform posture; -- migration plan for existing `DiskTier` bytes and semantic descriptors; -- no wire-format change unless separately authorized. +## Milestone 6 — shadow, backfill, and cutover decision -### ADR gate +Migration retains both Echo and Keep identities. -A production decision must record: +Backfill procedure: -- Echo and Keep identity ownership; -- canonical binding shape and encoding; -- binding persistence and recovery owner; -- full-blob versus range proof scopes; -- retention ownership; -- outbox metadata binding; -- toolchain policy; -- platform support; -- durable read and publication contracts; -- error taxonomy; -- migration and rollback; -- compatibility and release evidence. +1. Read existing content from the declared source backend. +2. Calculate Echo identity and exact length. +3. Stage expected Keep identity from the same bytes. +4. Reconstruct from Keep into quarantine. +5. Recalculate Echo identity and length. +6. Admit the binding only after all values agree. +7. Record backend provenance and migration outcome. -## Replay-safe write semantics +Shadow comparison must never silently repair the authoritative result. A +difference is an obstruction with retained evidence. -The integration must distinguish three different claims: +During an explicit migration posture, policy may permit recorded fallback: ```text -Authorized - Echo committed an intent permitting the external effect. - -Physically published - Keep returned exact publication evidence under its durability contract. - -Causally observed - Echo committed an observation matching the authorized effect and retained - evidence. +Try Keep under migration policy. +If evidence is unavailable, consult echo-cas explicitly. +Record fallback and backend provenance. +Verify bytes and backfill Keep. ``` -A Keep publication is not automatically an Echo causal fact. An Echo intent is -not proof that physical bytes exist. The adapter's binding is the bridge, not a -shortcut around either system's law. - -## Concurrency model - -### Write authority - -Use one writer per Keep store during all integration phases unless Keep adopts a -stronger deliberate concurrency model. - -Do not introduce one-writer-per-blob locks as an adapter invention. Keep catalog -and retention publication serialize store-level generations and therefore -require store-level authority. - -### Reader posture - -Many readers are allowed when each read is pinned to an immutable admitted -snapshot or generation. - -A reader that begins under generation `N` must not silently switch to generation -`N+1` during the operation. - -### Binding multiplicity - -The binding index must model: +After Keep becomes authoritative: ```text -one Echo logical content claim - ↔ one verified Keep BlobId - ↔ one or more verified Keep LayoutIds +Keep refusal + ≠ permission to consult echo-cas silently ``` -Selection of a layout must be explicit or deterministic. Keep's convenience -rule of selecting the lowest canonical committed `LayoutId` is acceptable for a -read API, but evidence must name the exact layout actually used. - -### Concurrency tests - -- a second writer receives a typed refusal; -- concurrent retries do not publish duplicate conflicting state; -- an exact publication retry returns the same synchronized outcome; -- a pinned reader completes against its original generation while a successor - publishes; -- reader completion order does not alter bytes, bindings, or receipts; -- alternate valid layouts do not create a false logical-identity conflict; -- conflicting verified logical identities obstruct deterministically. - -## Streaming and memory test harness - -Create an interop harness that tests source stream → Keep → reconstructed stream -→ Echo verifier with explicit resource accounting. - -### Inputs - -- deterministic virtual pseudo-random data with O(1) source state; -- exact empty, tiny, boundary, and very large inputs; -- configurable short-read and interrupted-read schedules; -- deterministic nearby edits; -- corruption injection for chunks, layouts, catalogs, bindings, and metadata. - -### Sinks - -- counting writer; -- bounded writer; -- short writer; -- interrupted writer; -- Echo raw-hash and exact-length verifier; -- optional incremental WAL decoder for Phase B. - -### Metrics - -- bytes requested and emitted; -- maximum read chunk; -- maximum write chunk; -- allocation count; -- maximum single allocation; -- transient high-water above retained result state; -- adapter-owned retained bytes; -- Keep-owned retained bytes; -- duplicate raw full-buffer allocations; -- selected chunk count for range reads. - -OS RSS may be collected as benchmark information but must not be the sole -deterministic correctness gate. +### Production decision gates -### Memory claims by layer - -- Keep detector and reader scratch state may be bounded independently of blob - size. -- `ReferenceStore` retained chunk material grows with admitted content and must - be accounted separately. -- WSC streaming may remove a duplicate raw input buffer while decoded recovery - evidence still grows with frame payload count. -- A true end-to-end bounded import claim requires a compact incremental recovery - fold. - -## Proof-scope tests - -The test suite must make the following substitutions impossible: - -- `RangeReadReceipt` used as complete-blob proof; -- `PublishedBlob` used as durability proof; -- `RetentionAnchor` used as closure proof; -- parsed `BlobId` used as content-presence proof; -- Echo semantic coordinate used as physical-presence proof; -- Keep `BlobId` used as Echo WSC content-hash proof; -- Echo content hash used as Keep `BlobId` proof; -- materialization intent used as completion proof; -- Keep publication used as Echo causal observation. - -## Capability ledger - -| Capability | Phase | Claim | -| ------------------------------------ | ----- | ------------------------------------------------------------------------------------------------------------------------------------- | -| `echo.keep.same-bytes-binding/v1` | A | Complete source and reconstruction verification admit one Echo ↔ Keep logical binding and exact layout realization. | -| `keep.echo.identity-agreement/v1` | A | Cross-repository conformance evidence closes Keep's declared Echo identity-boundary milestone without equating the two identity laws. | -| `echo.keep.reference-reconstruct/v1` | A | Keep reference reconstruction reproduces the Echo content claim exactly. | -| `echo.keep.reference-range/v1` | A/B | Exact ranges are authenticated under their deliberately narrow proof scope. | -| `echo.keep.wsc-stream-validation/v1` | B | WSC validation no longer requires a CAS port returning a full `Vec`. | -| `echo.keep.durable-reconstruct/v1` | C | A pinned durable Keep generation reconstructs an exact retained blob to caller-owned output. | -| `echo.keep.durable-range/v1` | C | A pinned durable Keep generation reads an exact authenticated range without whole-blob materialization. | -| `echo.keep.restart-read/v1` | C | Restart recovers a lawful generation capable of the same verified read. | -| `echo.keep.outbox-reconciliation/v1` | D | Echo outbox recovery reconciles matching, missing, or conflicting Keep publication evidence. | -| `echo.keep.binding-recovery/v1` | D | The exact identity binding remains recoverable after process death. | -| `echo.keep.retention-closure/v1` | D | Required Keep anchors have verified retained closure under the selected generation. | -| `echo.keep.platform-posture/v1` | C/D | Supported and unsupported filesystem postures are explicit and typed. | -| `echo.keep.disk-tier-replacement/v1` | E | Keep can replace Echo `DiskTier` for the declared production posture without semantic or replay drift. | - -## Open questions - -### Where should the binding live? - -Deferred. The branch may use an in-memory test type. Production persistence is -an ADR decision. - -### Does the Echo semantic coordinate belong inside the logical binding? - -No. The logical binding proves same-byte correspondence. Semantic coordinates -belong in an evidence envelope above it so multiple meanings may lawfully refer -to the same content. - -### Can one Echo content claim map to multiple Keep layouts? - -Yes, provided every layout independently names and reconstructs the same Keep -`BlobId` and Echo content claim. The binding model must preserve this -one-to-many realization relationship. - -### Should WSC migrate before the general `echo-cas` API? - -Yes. WSC is the demonstrated consumer with the currently broken -`Option>` contract. Generalize only after another consumer proves the -same capability boundary. - -### Should the first adapter return `Read` or write into a caller-owned sink? - -Either can work. A sink-oriented port makes final verification and provisional -output discipline explicit and avoids committing prematurely to reader -lifetimes. The chosen API must remain fallible and must not expose success before -complete Echo verification. - -### Is one writer per blob sufficient? - -No. Use one writer per Keep store because catalog and retention publication are -store-generation transitions. - -### Does Keep need more durability work before the spike? - -Not for Phase A or B. `ReferenceStore` is the correct oracle there. Production -requires a streamable durable logical-read surface and completion of the exact -retention and recovery posture selected by the integration. - -### Does removing `Vec` make WSC import bounded-memory? +- identical Echo identity for every fixture; +- backend-neutral conformance suite green; +- complete output-quarantine evidence; +- pinned-generation durable reads; +- crash recovery at every cross-store boundary; +- binding recovery after process death; +- retained-closure verification; +- deterministic layout-selection policy; +- bounded transient-memory measurements; +- explicit platform and toolchain posture; +- existing-content migration and rollback rehearsal; +- no semantic, WSC, or causal-evidence drift; +- no silent fallback. -Not by itself. It removes one mandatory raw materialization. The retained WAL -recovery result still requires separate redesign for a true end-to-end bound. +Only then may a production ADR decide whether Keep replaces `DiskTier` for a +specific supported posture. That decision does not replace Echo content +identity, WSC hashes, semantic coordinates, causal anchors, materialization +intent or observation, or in-memory CAS uses. -## Decision for this exploration branch +## Receipt durability questions -No ADR is introduced for this branch. +Every stored Echo reference to a Keep receipt must declare which posture it +expects: -The immediate work is: +1. ephemeral statement about one completed operation; +2. locator for replayable retained evidence; +3. portable self-contained proof. -1. Build the standalone Rust 1.96 identity-corroboration spike. -2. Define the private exploratory logical and realization binding types. -3. Add same-source and reconstructed-source dual-verification tests. -4. Prove lawful alternate-layout handling and conflicting-logical-identity - obstruction. -5. Introduce a narrow fallible WSC streaming port for retained-material - validation. -6. Implement Echo memory and disk adapters through that port. -7. Implement the Keep `ReferenceStore` adapter using admitted bindings. -8. Add full reconstruction and exact range paths with explicit proof scopes. -9. Measure allocation and transient-memory behavior without overstating total - process bounds. -10. Only after those claims pass, design the durable Keep logical-read and - binding-persistence boundary. +Current Keep reconstruction receipts are primarily posture 1. They can support +posture 2 only when their generation and supporting evidence remain retained. +They are not automatically posture 3. -The branch must not claim that Keep replaces `echo-cas`, that Echo and Keep use -the same identity, or that production durability and retention are complete. +The adapter must not persist a receipt as durable causal evidence until its +supporting-evidence retention and revalidation contract is explicit. -## Final architectural hypothesis +## Validation commands -The durable destination, if the evidence supports it, is: +Documentation changes on this branch must pass: -```text -Echo semantic coordinate -+ Echo content-only identity -+ exact byte length - │ - ▼ -independently verified Echo ↔ Keep logical binding - │ - ▼ -one or more exact Keep realizations -BlobId + LayoutId - │ - ▼ -Keep publication, recovery, and retained-closure evidence - │ - ▼ -Echo materialization observation and causal replay posture +```bash +cargo xtask docs-lint +tests/docs/test_adr_namespace.sh +git diff --check ``` -This is stronger than replacing one CAS implementation with another. +Implementation milestones add the narrow executable witnesses described above +and the directly relevant workspace checks. A green documentation plan is not +implementation evidence. + +## Stop conditions -It gives Echo and Keep independent identity laws, independent evidence, and a -lawful correspondence between them: +Stop and require a separate decision before: -> Echo can prove why the artifact matters. Keep can prove which exact bytes and -> reconstruction plan exist. The adapter can prove that both systems observed -> the same material. +- freezing public port types; +- freezing persisted binding bytes or a digest domain; +- changing Echo or WSC content identity; +- introducing a Keep dependency into Echo's ordinary Rust 1.90 workspace; +- claiming authenticated absence from an incomplete view; +- treating a Keep receipt as an Echo observation; +- treating a range receipt as complete-object proof; +- removing or silently bypassing `echo-cas`; +- declaring Keep the sole durable content authority. From 19f428bce2ee6b490db8f1da8596cb2cd5efb0d9 Mon Sep 17 00:00:00 2001 From: James Ross Date: Sun, 9 Aug 2026 20:52:13 -0700 Subject: [PATCH 3/7] docs: make physical evidence contracts executable --- .../echo-keep-physical-content-boundary.md | 58 +++++++++++++++---- .../echo-keep-physical-cas-interop-plan.md | 38 ++++++++++-- 2 files changed, 81 insertions(+), 15 deletions(-) diff --git a/docs/architecture/echo-keep-physical-content-boundary.md b/docs/architecture/echo-keep-physical-content-boundary.md index 0cc974d1..a51385be 100644 --- a/docs/architecture/echo-keep-physical-content-boundary.md +++ b/docs/architecture/echo-keep-physical-content-boundary.md @@ -74,7 +74,7 @@ fn reconstruct( &self, view: &PhysicalContentView, target: EchoContentId, - output: &mut dyn std::io::Write, + destination: &mut dyn TransactionalContentDestination, ) -> Result; ``` @@ -83,6 +83,12 @@ it to one pinned Keep generation or immutable view. An `echo-cas` adapter may support a weaker posture initially, but it must report unsupported evidence rather than manufacture generation or absence claims. +`TransactionalContentDestination` is likewise an Echo-owned semantic +capability, not an ordinary `Write` alias or a frozen ABI. It supplies private +staging plus an atomic commit operation. A destination that cannot guarantee +all-or-nothing visibility reports `CapabilityUnavailable` before +reconstruction. + The initial port includes: - complete-object reconstruction; @@ -108,15 +114,30 @@ limit. ## Output visibility -An ordinary `Write` sink can fail after accepting a prefix. A failed Keep call -therefore does not prove that the sink is untouched. +An ordinary `Write` sink can fail after accepting a prefix. Keep may therefore +leave an untrusted prefix in the writer supplied by its adapter. That writer is +always the private staging writer of the transactional destination, never an +application-visible sink. The Echo adapter enforces: > No complete receipt, no application-visible bytes. -It reconstructs into quarantine and reveals or promotes the result only after -the complete receipt and Echo identity both verify: +The adapter follows one visibility protocol: + +1. Begin a destination transaction and obtain its private staging writer. +2. Reconstruct through the backend into that writer. +3. On any failure, abort; a staged prefix may remain but is never visible. +4. Seal the staging artifact and verify the complete receipt, Echo identity, + and exact length. +5. Atomically commit the sealed artifact or return an operational failure. +6. Emit an Echo content observation only after commit succeeds. + +Commit changes the destination from its prior state directly to the complete +sealed artifact. A failed commit leaves the prior state visible and the new +artifact unpublished. Implementations may use an atomic same-filesystem rename +or return a sealed immutable handle; copying into an arbitrary live writer is +not a conforming commit. ```text backend reconstruction @@ -131,7 +152,8 @@ unpublished temporary artifact Quarantine may be a bounded memory buffer for small content or a temporary artifact for larger content. The rule must not imply unbounded whole-object -memory allocation. +memory allocation. Crash cleanup may collect unpublished staging artifacts, +but recovery must never infer publication from their existence. ## Decisions and failures @@ -154,10 +176,26 @@ enum ReconstructionOperationError { These names are conceptual and not a frozen ABI. An authenticated refusal must bind enough evidence to support its proposition. -Absence is evidenced only when the pinned view is complete enough to prove -non-membership. A timeout, unreadable catalog, exhausted resource limit, -cancellation, or unavailable capability teaches Echo nothing about content -truth. +Absence is evidenced only for a known Keep `BlobId` binding and one pinned view +whose named membership-index contract certifies that it covers every published +logical `BlobId` admitted by that view. No Echo-to-Keep binding means +`CapabilityUnavailable`, not physical absence. + +A complete-view absence receipt binds: + +- the Echo identity, Keep `BlobId`, and exact pinned-view identifier; +- the versioned completeness predicate and authenticated view-root commitment; +- a target-bound non-membership witness under that root; and +- the retention guard or anchor that kept the root and witness closure valid + through verification. + +The backend may realize the witness with an authenticated ordered index, a +Merkle non-membership proof, or a complete canonical scan. The adapter must +verify the format-specific witness, completeness predicate, target, and view +root before constructing the backend-neutral refusal. Missing or unreadable +root records, index pages, retention evidence, or witness material are +operational failures. A timeout, exhausted resource limit, cancellation, or +unavailable capability likewise teaches Echo nothing about content truth. Echo may admit an evidenced refusal as an observation only under an Echo law that explicitly accepts that refusal class and its physical aperture. A Rust diff --git a/docs/plans/echo-keep-physical-cas-interop-plan.md b/docs/plans/echo-keep-physical-cas-interop-plan.md index 4749f296..ff25878a 100644 --- a/docs/plans/echo-keep-physical-cas-interop-plan.md +++ b/docs/plans/echo-keep-physical-cas-interop-plan.md @@ -9,8 +9,9 @@ Git commit `fa943a6c0`. - **Boundary owner:** [Echo × Keep physical content boundary](../architecture/echo-keep-physical-content-boundary.md) -- **Keep contract:** `flyingrobots/keep` document - `docs/invariants/authenticated-reconstruction/README.md` +- **Keep contract:** External `flyingrobots/keep` document + [`docs/invariants/authenticated-reconstruction/README.md` at revision + `2b87899853b61d2f616f98a33b3d45657af3f621`](https://github.com/flyingrobots/keep/blob/2b87899853b61d2f616f98a33b3d45657af3f621/docs/invariants/authenticated-reconstruction/README.md) This plan owns sequencing, conformance evidence, and cutover gates. It does not own the durable architecture boundary, live priority, or release status. The @@ -142,9 +143,17 @@ Inject a sink failure after a deterministic prefix and prove: - no application-visible content is promoted; - no Echo content observation is emitted; -- quarantined bytes are discarded or remain explicitly unpublished; +- the prefix exists only in the destination's private staging artifact; +- abort leaves the prior destination state visible and the staged artifact + explicitly unpublished; - operational failure does not become a refusal receipt. +Inject commit failure after successful reconstruction and identity +corroboration. Prove that the sealed artifact remains unpublished, the prior +destination state remains visible, and no Echo observation is emitted. A +destination without atomic commit support must fail with +`CapabilityUnavailable` before reconstruction begins. + ## Milestone 3 — backend-neutral conformance Add a single conformance suite that runs against the existing Echo adapters and @@ -165,6 +174,19 @@ Mandatory laws: | Exact `LayoutId` requested but unavailable | Refusal; no alternate-layout fallback | | Operational timeout or resource refusal | No authenticated absence claim | +The complete-view absence cases must use the boundary's single witness rule. +The fixture supplies a known Echo-to-Keep binding, pinned view identifier, +versioned completeness predicate, authenticated view-root commitment, +target-bound non-membership witness, and retention guard for the witness +closure. It must prove: + +- a valid witness under the matching complete view yields evidenced absence; +- an incomplete view yields no content claim; +- a witness for another target, root, or generation is rejected; +- a missing root, index page, witness node, or retention guard is an + operational failure; and +- a present target can never be admitted through a forged absence witness. + The Keep adapter lives in Echo or an interop crate above both projects. Keep must never index by Echo hash or import Echo semantics. @@ -192,10 +214,16 @@ The durable operation must: - pin one immutable generation or catalog view; - retain all required evidence for the read lifetime; - verify retained closure required by the proof scope; -- stream exact logical bytes without one adapter-owned whole-blob allocation; +- stream exact logical bytes into the adapter-owned private staging writer + without one adapter-owned whole-blob memory allocation; - name the generation in its receipt; - distinguish evidenced refusal from operation failure; -- state that unsuccessful ordinary output may contain an untrusted prefix. +- state that unsuccessful ordinary output may contain an untrusted prefix that + remains quarantined and cannot become application-visible. + +Echo, not Keep, owns sealing, Echo identity verification, and the atomic +destination commit. A durable Keep receipt does not publish the staged artifact +or authorize an Echo observation. Keep owns this API in Keep vocabulary. Echo does not supply WSC, causal, semantic, or outbox concepts to Keep core. From cd9ca6fc875bbceb9f757703123addd993c68269 Mon Sep 17 00:00:00 2001 From: James Ross Date: Sun, 9 Aug 2026 21:04:01 -0700 Subject: [PATCH 4/7] docs: align Keep publication with Echo authority --- .../echo-keep-physical-content-boundary.md | 21 +- .../echo-keep-physical-cas-interop-plan.md | 381 +----------------- 2 files changed, 25 insertions(+), 377 deletions(-) diff --git a/docs/architecture/echo-keep-physical-content-boundary.md b/docs/architecture/echo-keep-physical-content-boundary.md index a51385be..a36df0aa 100644 --- a/docs/architecture/echo-keep-physical-content-boundary.md +++ b/docs/architecture/echo-keep-physical-content-boundary.md @@ -36,9 +36,9 @@ Keep must not depend on Echo, expose Echo concepts, or adopt the current weak inside the adapter and a physical-evidence envelope; they do not cross ordinary Echo semantic APIs. -The [integration and migration plan](../plans/echo-keep-physical-cas-interop-plan.md) -owns change-local sequencing and evidence gates. This page owns the boundary -regardless of which implementation phase is active. +[Echo issue #722](https://github.com/flyingrobots/echo/issues/722) owns +change-local integration sequencing and evidence gates. This page owns the +boundary regardless of which implementation phase is active. ## Division of responsibility @@ -287,7 +287,20 @@ The governing invariant is: > Orphaned physical content is acceptable. A committed Echo reference to > unavailable content is not. -The migration plan owns the crash matrix. The read port must not conceal the +Echo must commit the authorized physical-publication request and claim before +Keep performs the publication effect. Keep then publishes under a durable +reconciliation anchor keyed by that operation identity, and Echo records the +result as a settlement or observation before any execution resumes on it. This +is the request-before-effect and settlement-before-resumption law from +[ADR 0026](../adr/0026-durable-external-action-settlement.md). + +The reconciliation anchor is not a time-expiring lease. Keep must retain it +until an explicit recovery transition proves that long-term retention is +durable or that Echo durably settled abandonment before publication. Echo +unavailability cannot make the anchor expire. Finalization makes long-term +retention durable before releasing the provisional anchor. + +Issue #722 owns the executable crash matrix. The read port must not conceal the cross-store state machine or convert “write Keep, then write Echo” into an implicit protocol. diff --git a/docs/plans/echo-keep-physical-cas-interop-plan.md b/docs/plans/echo-keep-physical-cas-interop-plan.md index ff25878a..874f5743 100644 --- a/docs/plans/echo-keep-physical-cas-interop-plan.md +++ b/docs/plans/echo-keep-physical-cas-interop-plan.md @@ -1,376 +1,11 @@ -# Echo × Keep Physical Content Integration Plan - -- **Status:** Change-local implementation and evidence plan. -- **Decision:** Architecture approved; production replacement not approved. -- **Historical source:** The original feasibility exploration is preserved in - Git commit `fa943a6c0`. -- **Boundary owner:** - [Echo × Keep physical content boundary](../architecture/echo-keep-physical-content-boundary.md) -- **Keep contract:** External `flyingrobots/keep` document - [`docs/invariants/authenticated-reconstruction/README.md` at revision - `2b87899853b61d2f616f98a33b3d45657af3f621`](https://github.com/flyingrobots/keep/blob/2b87899853b61d2f616f98a33b3d45657af3f621/docs/invariants/authenticated-reconstruction/README.md) - -This plan owns sequencing, conformance evidence, and cutover gates. It does not -own the durable architecture boundary, live priority, or release status. The -Draft pull request and GitHub work items own current progress. - -## Verdict - -Keep is ready to become an experimental Echo backend. It is not ready to -become Echo's sole durable content authority, and `echo-cas` must not be -removed yet. - -The next milestone is not “make Keep implement `BlobStore`.” It is: - -> Give Echo a receipt-bearing physical-content boundary, wrap `echo-cas` -> behind it, then prove Keep conforms. - -## Scope - -The first integration proves: - -- complete-object reconstruction; -- expected staged ingestion; -- explicit publication; -- same-source Echo and Keep identity corroboration; -- complete reconstructed-source corroboration; -- quarantined output visibility; -- authenticated success versus evidenced refusal versus operational failure; -- backend-neutral behavior across existing `echo-cas` and Keep's - non-durable `ReferenceStore`. - -The first integration does not claim: - -- durable Keep authority; -- range-read support at the Echo port; -- a persisted identity-binding format; -- end-to-end bounded WSC recovery memory; -- production retention or crash recovery; -- `DiskTier` replacement; -- `echo-cas` removal. - -## Milestone 1 — identity-bridge decision - -Before adapter code, record the exact identity relation: - -| Question | Required evidence | -| ---------------------------------------------------- | ---------------------------------------------------- | -| What fields form Echo's current content identity? | Exact typed structure and canonical input bytes | -| What algorithm and domain does Echo hash? | Source anchor and golden fixtures | -| Does Echo identity include logical length? | Explicit yes or no | -| Can an Echo ID become a Keep `BlobId` without bytes? | Proof or explicit no | -| Is the conversion one-to-one? | Argument over exact preimages plus mutation fixtures | -| Which identity remains in Echo WAL and WSC? | Explicit ownership decision | -| What happens when versions or algorithms differ? | Typed refusal; no implicit conversion | -| Are both identities retained during migration? | Explicit carrier posture | - -The expected initial answer is that both identities are retained and their -relation is established from the same exact bytes. No 32-byte digest cast is -lawful. - -The exploratory logical binding is private and noncanonical: - -```rust -struct EchoKeepLogicalBindingV1 { - echo_content_id: EchoContentId, - keep_blob_id: KeepBlobId, - logical_length: ByteLength, - binding_contract: BindingContractVersion, -} -``` - -One logical binding may have multiple realization bindings: - -```rust -struct EchoKeepRealizationBindingV1 { - logical: EchoKeepLogicalBindingV1, - keep_layout_id: KeepLayoutId, -} -``` - -These pseudostructures are obligation checklists, not persisted formats or -public APIs. - -### Identity witnesses - -Run both identity calculations over one source stream, then reconstruct -through Keep and independently recalculate Echo identity and exact length. - -Required cases: - -- empty bytes; -- small text; -- deterministic binary ramp; -- chunk boundary and boundary-plus-one sizes; -- large deterministic virtual input; -- arbitrary short and interrupted reads; -- nearby edits with deterministic chunk reuse; -- mismatched Echo identity; -- mismatched length; -- mismatched Keep `BlobId`; -- wrong or corrupt layout; -- missing or corrupt chunk. - -Acceptance requires both routes. Same-source computation alone does not prove -retained reconstruction. Reconstructed-source computation is route-independent -evidence, not an independent BLAKE3 implementation. - -## Milestone 2 — Echo physical-content port - -Define the Echo-owned port from the -[boundary contract](../architecture/echo-keep-physical-content-boundary.md). -Keep types must not appear in its ordinary consumer surface. - -First implement the existing Echo backends: - -1. `MemoryTier` adapter; -2. `DiskTier` adapter; -3. any WSC retained-content consumer needed by the conformance fixture. - -This establishes that the port is consumer-shaped, not reverse-engineered -from Keep. - -The adapters must expose their actual evidence posture. A current backend that -cannot prove pinned-view absence must return unavailable evidence rather than -convert `None` into an authenticated absence receipt. - -### Output witness - -Inject a sink failure after a deterministic prefix and prove: - -- no application-visible content is promoted; -- no Echo content observation is emitted; -- the prefix exists only in the destination's private staging artifact; -- abort leaves the prior destination state visible and the staged artifact - explicitly unpublished; -- operational failure does not become a refusal receipt. - -Inject commit failure after successful reconstruction and identity -corroboration. Prove that the sealed artifact remains unpublished, the prior -destination state remains visible, and no Echo observation is emitted. A -destination without atomic commit support must fail with -`CapabilityUnavailable` before reconstruction begins. - -## Milestone 3 — backend-neutral conformance - -Add a single conformance suite that runs against the existing Echo adapters and -the experimental Keep `ReferenceStore` adapter. - -Mandatory laws: - -| Case | Required result | -| ------------------------------------------- | -------------------------------------------------------- | -| Same bytes through different Keep layouts | Same Echo observation; different physical provenance | -| Missing content in a complete view | Evidenced absence, not generic `None` | -| Missing content in an incomplete view | No content claim | -| Corrupt content | Integrity refusal; never fallback bytes | -| Sink failure after a prefix | No Echo observation; no promoted output | -| Range receipt where whole proof is required | Rejected by type or adapter | -| Expected identity mismatch during ingestion | No publication | -| Re-layout between independent reads | Logical observation stable; physical evidence may change | -| Exact `LayoutId` requested but unavailable | Refusal; no alternate-layout fallback | -| Operational timeout or resource refusal | No authenticated absence claim | - -The complete-view absence cases must use the boundary's single witness rule. -The fixture supplies a known Echo-to-Keep binding, pinned view identifier, -versioned completeness predicate, authenticated view-root commitment, -target-bound non-membership witness, and retention guard for the witness -closure. It must prove: - -- a valid witness under the matching complete view yields evidenced absence; -- an incomplete view yields no content claim; -- a witness for another target, root, or generation is rejected; -- a missing root, index page, witness node, or retention guard is an - operational failure; and -- a present target can never be admitted through a forged absence witness. - -The Keep adapter lives in Echo or an interop crate above both projects. Keep -must never index by Echo hash or import Echo semantics. - -The first Keep backend uses `ReferenceStore` only. Its process-memory state is -not durable evidence. - -## Milestone 4 — durable Keep consumer contract - -Before durable integration, Keep must expose a consolidated generic capability -with this semantic shape: - -```text -admitted immutable Keep view -+ target BlobId -+ optional exact LayoutId -+ proof scope -+ caller-owned output -──────────────────────────── -receipt or evidenced refusal - or operational failure -``` - -The durable operation must: - -- pin one immutable generation or catalog view; -- retain all required evidence for the read lifetime; -- verify retained closure required by the proof scope; -- stream exact logical bytes into the adapter-owned private staging writer - without one adapter-owned whole-blob memory allocation; -- name the generation in its receipt; -- distinguish evidenced refusal from operation failure; -- state that unsuccessful ordinary output may contain an untrusted prefix that - remains quarantined and cannot become application-visible. - -Echo, not Keep, owns sealing, Echo identity verification, and the atomic -destination commit. A durable Keep receipt does not publish the staged artifact -or authorize an Echo observation. - -Keep owns this API in Keep vocabulary. Echo does not supply WSC, causal, -semantic, or outbox concepts to Keep core. - -Production experiments remain limited to Keep's explicitly admitted platform -profile. Unsupported platforms return typed posture rather than degraded -durability claims. - -Echo currently uses Rust 1.90 and Keep requires Rust 1.96. The durable adapter -must make that boundary explicit through a toolchain upgrade, separate package -or CI lane, or independently supported Keep MSRV change. It must not arrive as -an incidental dependency update. - -## Milestone 5 — cross-store publication protocol - -Keep publication and Echo WAL publication require a durable operation identity -and a recovery state machine. - -The governing invariant is: - -> Orphaned physical content is acceptable. A committed Echo reference to -> unavailable content is not. - -Proposed choreography: - -```text -1. Echo assigns operation identity O. -2. Adapter establishes expected Echo and Keep identities. -3. Keep stages and verifies the exact content. -4. Keep durably publishes under a provisional anchor or lease for O. -5. Echo commits the causal reference or observation for O. -6. Keep finalizes long-term retention for O. -7. Recovery records or derives the reconciled completion posture. -``` - -### Crash matrix - -| Crash point | Lawful recovery | -| ------------------------------------------------------ | --------------------------------------------------------- | -| Before Keep publication | Discard or resume staging | -| After Keep publication, before Echo WAL | Preserve as provisional orphan; eventually collect | -| After Echo WAL, before final retention | Recover through provisional anchor and finalize | -| After retention finalization | Complete | -| Echo committed but Keep evidence missing or unprovable | Integrity obstruction; never silent fallback | -| Keep published but Echo state unreadable | Preserve provisional evidence until Echo recovery decides | - -Exercise before, during, and after every physical synchronization and Echo WAL -commit boundary. Retry must be idempotent by operation identity. - -The binding carrier and canonical encoding are intentionally deferred until -this protocol identifies what recovery must retain. Freezing either requires a -production ADR. - -## Milestone 6 — shadow, backfill, and cutover decision - -Migration retains both Echo and Keep identities. - -Backfill procedure: - -1. Read existing content from the declared source backend. -2. Calculate Echo identity and exact length. -3. Stage expected Keep identity from the same bytes. -4. Reconstruct from Keep into quarantine. -5. Recalculate Echo identity and length. -6. Admit the binding only after all values agree. -7. Record backend provenance and migration outcome. - -Shadow comparison must never silently repair the authoritative result. A -difference is an obstruction with retained evidence. - -During an explicit migration posture, policy may permit recorded fallback: - -```text -Try Keep under migration policy. -If evidence is unavailable, consult echo-cas explicitly. -Record fallback and backend provenance. -Verify bytes and backfill Keep. -``` - -After Keep becomes authoritative: - -```text -Keep refusal - ≠ permission to consult echo-cas silently -``` - -### Production decision gates - -- identical Echo identity for every fixture; -- backend-neutral conformance suite green; -- complete output-quarantine evidence; -- pinned-generation durable reads; -- crash recovery at every cross-store boundary; -- binding recovery after process death; -- retained-closure verification; -- deterministic layout-selection policy; -- bounded transient-memory measurements; -- explicit platform and toolchain posture; -- existing-content migration and rollback rehearsal; -- no semantic, WSC, or causal-evidence drift; -- no silent fallback. - -Only then may a production ADR decide whether Keep replaces `DiskTier` for a -specific supported posture. That decision does not replace Echo content -identity, WSC hashes, semantic coordinates, causal anchors, materialization -intent or observation, or in-memory CAS uses. - -## Receipt durability questions - -Every stored Echo reference to a Keep receipt must declare which posture it -expects: - -1. ephemeral statement about one completed operation; -2. locator for replayable retained evidence; -3. portable self-contained proof. - -Current Keep reconstruction receipts are primarily posture 1. They can support -posture 2 only when their generation and supporting evidence remain retained. -They are not automatically posture 3. - -The adapter must not persist a receipt as durable causal evidence until its -supporting-evidence retention and revalidation contract is explicit. - -## Validation commands - -Documentation changes on this branch must pass: - -```bash -cargo xtask docs-lint -tests/docs/test_adr_namespace.sh -git diff --check -``` - -Implementation milestones add the narrow executable witnesses described above -and the directly relevant workspace checks. A green documentation plan is not -implementation evidence. - -## Stop conditions - -Stop and require a separate decision before: - -- freezing public port types; -- freezing persisted binding bytes or a digest domain; -- changing Echo or WSC content identity; -- introducing a Keep dependency into Echo's ordinary Rust 1.90 workspace; -- claiming authenticated absence from an incomplete view; -- treating a Keep receipt as an Echo observation; -- treating a range receipt as complete-object proof; -- removing or silently bypassing `echo-cas`; -- declaring Keep the sole durable content authority. +# Echo × Keep Integration Tracker + +This path does not own an implementation plan or work status. +[Echo issue #722](https://github.com/flyingrobots/echo/issues/722) owns the live +integration sequence, dependencies, evidence gates, and cutover work. The +[physical-content boundary](../architecture/echo-keep-physical-content-boundary.md) +owns current architectural truth. Git commit `fa943a6c0` preserves the original +feasibility exploration as historical source material. From b1f095ae47008b46b1268dd600650e91e3761ae1 Mon Sep 17 00:00:00 2001 From: James Ross Date: Sun, 9 Aug 2026 23:17:52 -0700 Subject: [PATCH 5/7] docs: replace ADR queue with semantic decisions --- .github/workflows/ci.yml | 4 +- AGENTS.md | 18 +- docs/DOCUMENTATION_STANDARDS.md | 205 ++++++++++++++++++ docs/README.md | 24 +- docs/adr/0012-repository-knowledge-model.md | 9 +- docs/adr/README.md | 35 +-- .../echo-keep-physical-content-boundary.md | 9 +- docs/invariants/FIXED-TIMESTEP.md | 5 +- docs/topics/README.md | 11 +- tests/docs/test_adr_namespace.sh | 179 --------------- tests/docs/test_documentation_model.sh | 127 +++++++++++ tests/docs/test_knowledge_model.sh | 1 + 12 files changed, 412 insertions(+), 215 deletions(-) create mode 100644 docs/DOCUMENTATION_STANDARDS.md delete mode 100755 tests/docs/test_adr_namespace.sh create mode 100755 tests/docs/test_documentation_model.sh diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 57dbe01f..6b371580 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -40,8 +40,8 @@ jobs: run: bash tests/docs/test_retired_component_refs.sh - name: repository knowledge model regression run: bash tests/docs/test_knowledge_model.sh - - name: ADR namespace regression - run: bash tests/docs/test_adr_namespace.sh + - name: Documentation model regression + run: bash tests/docs/test_documentation_model.sh - name: generated rule truth regression run: bash tests/docs/test_generated_rule_truth.sh - name: determinism policy gate regression diff --git a/AGENTS.md b/AGENTS.md index e0892b8b..f47daf32 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -68,7 +68,9 @@ Repository knowledge has one owner for each kind of truth: - **Current architectural truth**: `docs/architecture/`, `docs/spec/`, `docs/invariants/`, and `docs/topics/`. -- **Durable architectural decisions**: accepted ADRs in `docs/adr/`. +- **Documentation policy and decision relationships**: + `docs/DOCUMENTATION_STANDARDS.md`. +- **Historical numbered decisions**: the closed archive in `docs/adr/`. - **Live work, priority, dependencies, and status**: GitHub Issues, Projects, pull requests, and review threads. - **Shipped externally meaningful behavior**: `CHANGELOG.md`. @@ -77,18 +79,20 @@ Repository knowledge has one owner for each kind of truth: Use `README.md`, `GUIDE.md`, and `docs/README.md` as entrances. Do not recreate cycles, retrospectives, a checked-in backlog, a checked-in status ledger, or a post-hoc design document. Change-local design and test plans may live in the -issue or pull request. Write an ADR only when a decision changes a durable -architectural boundary or invariant. +issue or pull request. Record a durable decision in the named current document +that owns its concept, and state supersession, refinement, dependency, and +related-decision edges explicitly. Do not allocate a new numbered ADR. When recovering context, read the relevant canonical topic/spec/invariant and -ADR, then inspect the current GitHub issue or pull request, `git log -n 5`, and -`git status`. +architecture document, follow any explicit links into the historical ADR +archive, then inspect the current GitHub issue or pull request, `git log -n 5`, +and `git status`. ## Work Loop ```text scope claim --> decide whether an ADR is required +-> identify the canonical documentation owner -> name the test plan and executable witness -> RED -> GREEN @@ -139,7 +143,7 @@ changes, compare the branch's actual behavior and artifacts with the current documentation. Revisit the relevant entrances and canonical owners: `README.md`, `GUIDE.md`, `docs/README.md`, `docs/architecture/`, `docs/spec/`, `docs/invariants/`, `docs/topics/`, and the evidence anchors of any applicable -accepted ADRs. +historical ADRs. Search specifically for stale current-state claims such as “not implemented,” “fixture-only,” old ownership or authority boundaries, obsolete version or diff --git a/docs/DOCUMENTATION_STANDARDS.md b/docs/DOCUMENTATION_STANDARDS.md new file mode 100644 index 00000000..b0ed01c2 --- /dev/null +++ b/docs/DOCUMENTATION_STANDARDS.md @@ -0,0 +1,205 @@ + + + +# Documentation Standards + +**Status:** Current project policy for new and substantially changed +documentation. +**Supersedes:** [ADR 0012: Repository Knowledge Model After Method](adr/0012-repository-knowledge-model.md) + +Echo documentation is part of the engineering contract. Its job is to make +the current system, its governing decisions, and the evidence for those claims +findable without reconstructing a private chronology. + +This policy adapts the structure of Colorful Language's +[documentation standard](https://github.com/flyingrobots/colorful-language/blob/09dc60a9f23834f8511bcb3067cb1ac4393fae8a/docs/DOCUMENTATION_STANDARDS.md) +to Echo's existing knowledge model. Echo keeps its established split among +architecture, specifications, invariants, topics, and executable witnesses. It +does not adopt Colorful's roadmap, goalpost, or topic test-plan machinery. + +## Core rules + +1. Give every durable claim one canonical owner. +2. Give every page one primary reader job. +3. Name pages after concepts, contracts, or reader tasks, not sequence slots. +4. State relationships explicitly; directory order and filename order carry no + architectural meaning. +5. Keep current truth, planned work, historical reasoning, and executable + evidence distinguishable. +6. Link strong claims to code, tests, specifications, invariants, or retained + historical decisions. + +## Corpus map + +| Location | Job | +| ---------------------------------------------------------- | ------------------------------------------------------------------- | +| `README.md`, `GUIDE.md` | Project entrances and supported routes into deeper documentation. | +| `docs/README.md` | Documentation spine and routing index. | +| `docs/topics/` | Living explanations of durable Echo concepts and boundaries. | +| `docs/architecture/` | Current cross-subsystem architecture and accepted boundary designs. | +| `docs/spec/` | Normative protocols, encodings, and conformance contracts. | +| `docs/invariants/` | Compact laws that implementations must preserve. | +| `docs/determinism/` | Determinism policy, hazards, and evidence contracts. | +| `docs/adr/` | Closed archive of numbered historical decision records. | +| GitHub Issues, Projects, pull requests, and review threads | Live work, plans, priority, blockers, and status. | +| `CHANGELOG.md` | Externally meaningful behavior that shipped. | +| Git history | Removed material and the exact evolution of checked-in documents. | + +Do not create a second current reference merely because another directory or +document type is convenient. Link to the canonical owner and add only the +reader-specific context the new page needs. + +## Page jobs + +A page should primarily help its reader do one of these jobs: + +- learn a concept or boundary; +- perform a supported task; +- look up an exact contract; +- understand a decision and its tradeoffs; +- troubleshoot an observable failure; +- change the implementation and verify the result. + +A page may link across jobs, but it should not become a tutorial, reference, +roadmap, architecture guide, and historical diary at once. + +### Topics + +A topic describes the current conceptual model. It states the boundary, +invariants, ownership, and evidence anchors needed to understand that concept. +Update it in the same change that changes the boundary. + +Topics do not own live implementation queues. A limitation may be stated as a +current fact; the work to change it belongs in GitHub. + +### Architecture + +An architecture page explains current cross-subsystem structure or an accepted +boundary design. When an accepted design is not implemented, the page must say +so prominently and link to the GitHub owner for implementation state. It must +not describe planned behavior as existing runtime behavior. + +### Specifications and invariants + +Specifications define conformance. Invariants state compact laws. They are not +explanatory essays or implementation plans. Examples that form part of a +contract should be executable or backed by exact fixtures when practical. + +## Durable decisions + +A durable decision changes a long-lived boundary, identity, format, invariant, +authority split, compatibility promise, or recovery law. Record the decision +where a future reader will look for the concept: + +- update the owning topic for a conceptual boundary; +- update the owning architecture page for a cross-subsystem boundary; +- update the owning specification or invariant for a normative contract; +- add a semantically named `rationale.md` beside a larger concept when the + tradeoffs would otherwise overwhelm its current reference. + +Do not allocate a number merely to prove that a decision happened. The numbered +ADR sequence in `docs/adr/` is a closed historical archive. Existing ADRs remain +valuable evidence and may be linked, refined, or superseded, but new durable +decisions use semantic names in their owning current-documentation area. + +### Relationship contract + +When a decision relates materially to another decision, include the applicable +relationship near the top of the owning document: + +- **Supersedes:** the named older decision no longer governs the stated scope. +- **Superseded by:** the named newer decision now governs the stated scope. +- **Refines:** this decision adds precision without replacing the older one. +- **Depends on:** this decision requires another decision or contract to hold. +- **Related:** the documents illuminate the same boundary but neither governs + the other. + +Use descriptive links, not bare identifiers. A supersession must be recorded in +both directions so readers entering through either document can follow it. +Absence of a relationship line means no such relationship is claimed; it does +not mean “whatever has the larger number wins.” + +## Current truth, plans, and history + +Living references describe current implementation truth or clearly labeled +accepted contracts. GitHub owns change-local plans and status. Git history owns +the exact old text. + +Do not check in backlogs, cycle packets, retrospectives, review transcripts, +status ledgers, or roadmap checklists. A short checked-in redirect may remain +when an old stable path must route readers to its current owner. + +Historical reasoning must not masquerade as current behavior. Mark retained +historical documents clearly and link to the current owner that supersedes or +refines them. + +## Evidence and citations + +Strong claims should point to the smallest durable witness that establishes +them: + +- source or public API for ownership and shape; +- tests, fixtures, or golden vectors for behavior; +- a specification or invariant for normative law; +- an accepted architecture page for an unimplemented boundary contract; +- a historical decision for retained reasoning. + +Source links support an explanation; they do not replace one. Prefer +repository-relative links for checked-in sources. Use stable external +permalinks when the exact outside revision matters. + +Never claim that a command, test, visual inspection, review, or runtime path was +verified when it was not run or observed. + +## Examples and safety + +Examples must use supported behavior and enough context to interpret them. +Separate copyable commands from expected output. Do not put shell prompts in a +copyable command block. + +Put warnings before destructive, privileged, costly, or irreversible commands. +State the scope and consequence, and provide a safer check or recovery route +when one exists. + +## Writing and structure + +- Lead with the result, decision, warning, or essential condition. +- Prefer exact Echo terms and define unfamiliar ones at first use. +- Use active voice when it clarifies ownership. +- Use prose for causality and tradeoffs, lists for parallel facts, and tables + for genuinely two-dimensional comparisons. +- Use descriptive link text rather than “here” or a bare path. +- Treat length and style metrics as editorial signals, not universal merge + gates. + +## Maintenance loop + +For a meaningful change: + +1. Identify the canonical owner of the affected claim. +2. Name the smallest executable witness when behavior changes. +3. Update design rationale only when the tradeoff needs durable explanation. +4. Implement and validate the change. +5. Update the current owner after the behavior or accepted contract changes. +6. Add explicit decision relationships when governance changed. +7. Update `docs/README.md` when a durable route was added or moved. +8. Keep live follow-up work in GitHub. + +## Review checklist + +Before calling a documentation change done, verify that: + +- the page has one primary reader job; +- the durable claim has one canonical owner; +- current behavior and accepted-but-unimplemented design are distinguishable; +- plans and status remain in GitHub; +- durable decision relationships are explicit and bidirectional when they + supersede; +- strong claims have appropriate evidence anchors; +- new durable pages are linked from `docs/README.md`; +- internal links resolve; +- Markdown and whitespace checks pass. + +The objective is not uniform paperwork. The objective is a corpus in which a +reader can find what governs a concept, why it governs, what it replaced, and +what proves it without decoding a global number line. diff --git a/docs/README.md b/docs/README.md index 6d24de6c..196a6ae6 100644 --- a/docs/README.md +++ b/docs/README.md @@ -8,6 +8,7 @@ causal history. Git history is the archive; GitHub owns live work and status. ## Start Here +- [Documentation standards](DOCUMENTATION_STANDARDS.md) - [Architecture outline](architecture/outline.md) - [Application contract hosting](architecture/application-contract-hosting.md) - [Local contract host quickstart](quickstart-local-contract-host.md) @@ -41,10 +42,15 @@ causal history. Git history is the archive; GitHub owns live work and status. - [Runtime constellation](topics/RuntimeConstellation.md) - [Security posture and threat models](topics/security/README.md) -## Durable Decisions +## Decision History -- [ADR map](adr/README.md) -- [Repository knowledge model](adr/0012-repository-knowledge-model.md) +Current decisions are reached through the semantic architecture, topic, and +contract sections above and below. This section routes to the retained numbered +archive; it is not an allocation queue or a substitute for explicit +relationships. + +- [Legacy numbered decision archive](adr/README.md) +- [Superseded repository knowledge model](adr/0012-repository-knowledge-model.md) - [Echo/Continuum authority boundary](adr/0013-echo-continuum-authority-boundary.md) - [Generated rule authorship and footprints](adr/0014-generated-rule-authorship-and-footprints.md) - [Registry, provider, and host boundary](adr/0015-registry-provider-host-boundary.md) @@ -80,7 +86,11 @@ causal history. Git history is the archive; GitHub owns live work and status. ## Knowledge Ownership Current architecture belongs in architecture documents, specifications, -invariants, and living topics. Accepted durable decisions belong in ADRs. -Externally meaningful shipped behavior belongs in `CHANGELOG.md`. Live design, -priority, dependencies, review state, and follow-up work belong in GitHub -Issues, Projects, pull requests, and review threads. +invariants, and living topics. Durable decisions update the named document that +owns their concept and state relationships explicitly. Numbered ADRs are a +closed historical archive, not the current decision queue. Externally +meaningful shipped behavior belongs in `CHANGELOG.md`. Live design, priority, +dependencies, review state, and follow-up work belong in GitHub Issues, +Projects, pull requests, and review threads. The +[documentation standards](DOCUMENTATION_STANDARDS.md) define the complete +contract. diff --git a/docs/adr/0012-repository-knowledge-model.md b/docs/adr/0012-repository-knowledge-model.md index 19f43f5d..9ad0d4ff 100644 --- a/docs/adr/0012-repository-knowledge-model.md +++ b/docs/adr/0012-repository-knowledge-model.md @@ -3,9 +3,10 @@ # ADR 0012: Repository Knowledge Model After Method -- **Status:** Accepted +- **Status:** Superseded - **Date:** 2026-07-13 - **Decision owner:** James Ross +- **Superseded by:** [Documentation Standards](../DOCUMENTATION_STANDARDS.md) ## Context @@ -20,6 +21,12 @@ evidence-first engineering discipline. ## Decision +This section preserves the repository knowledge model accepted in July 2026. +The current policy is the semantically named +[Documentation Standards](../DOCUMENTATION_STANDARDS.md), which retains the +separation of current truth, motion, evidence, and history while replacing the +numbered ADR queue with concept-owned decisions and explicit relationships. + Repository knowledge has five non-overlapping homes: 1. **GitHub issues, pull requests, and projects own motion.** Priority, current diff --git a/docs/adr/README.md b/docs/adr/README.md index a971974c..633089b3 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -1,23 +1,32 @@ -# Architecture Decision Records +# Legacy Architecture Decision Archive -Architecture Decision Records preserve decisions whose consequences outlive a -single issue or pull request. They explain why a boundary exists; they do not -track work, progress, priority, or release readiness. +These numbered records preserve Echo decisions accepted before the repository +adopted concept-owned documentation. They remain useful historical evidence, +but their numbers are locators only: sequence does not express dependency, +supersession, refinement, importance, or current authority. -## Contract +The archive closed after record 0026 on 2026-08-09. New durable decisions update +the semantically named topic, architecture document, specification, invariant, +or colocated rationale that owns the concept. The +[documentation standards](../DOCUMENTATION_STANDARDS.md) define that current +contract. -- Use a four-digit sequence and a short noun phrase: - `0002-example-boundary.md`. -- Give every new record a status: `Proposed`, `Accepted`, - `Partially superseded`, `Superseded`, or `Rejected`. -- Do not rewrite an accepted decision to conceal history. Add a new record and - mark the old record superseded. -- Record alternatives and consequences, not an implementation diary. +## Archive contract + +- Do not allocate another numbered record. +- Preserve accepted historical text; add a clear status or relationship note + when current documentation supersedes or refines it. +- Record supersession in both the historical record and its new semantic owner. +- Follow explicit descriptive links; never infer a relationship from numbers. - Keep work state in GitHub issues and pull requests. +Some older records predate the relationship contract and name only a status. +When one is materially revisited, add descriptive forward and reverse links +without rewriting its original reasoning. + ## Index | ADR | Status | Decision | @@ -33,7 +42,7 @@ track work, progress, priority, or release readiness. | [0009](ADR-0009-Inter-Worldline-Communication.md) | Superseded | Historical frontier-relative state-patch transport | | [0010](ADR-0010-observational-seek-and-administrative-rewind.md) | Accepted | Observational seek and administrative rewind | | [0011](ADR-0011-explicit-observation-contract.md) | Partially superseded | Explicit observation mechanics; ADR 0021 governs public optics | -| [0012](0012-repository-knowledge-model.md) | Accepted | Repository knowledge model after Method | +| [0012](0012-repository-knowledge-model.md) | Superseded | Historical repository knowledge model after Method | | [0013](0013-echo-continuum-authority-boundary.md) | Accepted | Echo and Continuum authority boundary | | [0014](0014-generated-rule-authorship-and-footprints.md) | Accepted | Generated rule authorship and footprint honesty | | [0015](0015-registry-provider-host-boundary.md) | Partially superseded | Compiler, registry, provider, and host responsibilities | diff --git a/docs/architecture/echo-keep-physical-content-boundary.md b/docs/architecture/echo-keep-physical-content-boundary.md index a36df0aa..bf545a35 100644 --- a/docs/architecture/echo-keep-physical-content-boundary.md +++ b/docs/architecture/echo-keep-physical-content-boundary.md @@ -8,6 +8,9 @@ - **Decision date:** 2026-08-09 - **Implementation posture:** No Echo physical-content port or Keep adapter is implemented on this branch. +- **Refines:** [Retained reading storage and proof boundary](../adr/0020-retained-reading-storage-and-proof-boundary.md) +- **Depends on:** [Durable external-action settlement](../adr/0026-durable-external-action-settlement.md) +- **Related:** [Keep authenticated reconstruction contract](https://github.com/flyingrobots/keep/blob/3bf7b9179db41e90620e6d1875c2d40222a2330b/docs/architecture/authenticated-reconstruction-contract.md) ## Decision @@ -334,8 +337,10 @@ changing WSC wire identity. ## Production adoption gate A production Keep backend, persisted binding format, changed WSC identity, or -permanent `echo-cas` replacement requires a separate accepted ADR after the -conformance and crash evidence exists. That decision must govern: +permanent `echo-cas` replacement requires a separately reviewable accepted +change to Echo's named physical-content documentation after the conformance and +crash evidence exists. That change must refine or supersede this experimental +boundary explicitly and govern: - exact Echo and Keep identity preimages; - binding carrier, encoding, versioning, and recovery; diff --git a/docs/invariants/FIXED-TIMESTEP.md b/docs/invariants/FIXED-TIMESTEP.md index f5fcc51b..5fe7a390 100644 --- a/docs/invariants/FIXED-TIMESTEP.md +++ b/docs/invariants/FIXED-TIMESTEP.md @@ -226,8 +226,9 @@ as uniform integers with no per-tick metadata. - Replay is structurally sound without recording per-tick time deltas. The quantum is a worldline parameter, not a per-entry field. - No variable-dt plumbing needs to exist in the codebase. If a future - use case demands variable dt, it requires a new ADR to relax this invariant - with explicit constraints. + use case demands variable dt, a separately reviewable durable decision must + update or supersede this invariant with explicit constraints and + relationships. ## Cross-references diff --git a/docs/topics/README.md b/docs/topics/README.md index 04493eed..bd7ceb35 100644 --- a/docs/topics/README.md +++ b/docs/topics/README.md @@ -5,8 +5,15 @@ Topics describe Echo's current architectural boundaries. They are neither historical design packets nor work trackers. Update a topic in the same change -that changes its boundary, and anchor strong claims to code, tests, specs, or an -ADR. +that changes its boundary, and anchor strong claims to code, tests, +specifications, invariants, accepted architecture, or retained historical +decisions. + +Topics are organized by named concepts, not a global chronology. When one +topic supersedes, refines, depends on, or materially relates to another +decision, state that relationship with descriptive links near the top of the +owning page. See the [documentation standards](../DOCUMENTATION_STANDARDS.md) +for the page and relationship contracts. ## Index diff --git a/tests/docs/test_adr_namespace.sh b/tests/docs/test_adr_namespace.sh deleted file mode 100755 index 4253337f..00000000 --- a/tests/docs/test_adr_namespace.sh +++ /dev/null @@ -1,179 +0,0 @@ -#!/usr/bin/env bash -# SPDX-License-Identifier: Apache-2.0 -# © James Ross Ω FLYING•ROBOTS - -set -euo pipefail - -repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" -cd "$repo_root" - -readonly legacy_adrs=( - "ADR-0001-warp-two-plane-skeleton-and-attachments.md" - "ADR-0002-warp-instances-descended-attachments.md" - "ADR-0003-Materialization-Bus.md" - "ADR-0004-No-Global-State.md" - "ADR-0005-Physics.md" - "ADR-0006-Ban-Non-Determinism.md" - "ADR-0007-BOAW-Storage.md" - "ADR-0008-Worldline-Runtime-Model.md" - "ADR-0009-Inter-Worldline-Communication.md" - "ADR-0010-observational-seek-and-administrative-rewind.md" - "ADR-0011-explicit-observation-contract.md" -) - -readonly current_adrs=( - "0012-repository-knowledge-model.md" - "0013-echo-continuum-authority-boundary.md" - "0014-generated-rule-authorship-and-footprints.md" - "0015-registry-provider-host-boundary.md" - "0016-continuum-transport-identity.md" - "0017-universal-little-endian-codec.md" - "0018-sessions-causal-posture-and-authority.md" - "0019-bunny-owns-reusable-geometry.md" - "0020-retained-reading-storage-and-proof-boundary.md" - "0021-public-optic-observation-boundary.md" - "0022-application-requested-causal-anchor-admission.md" - "0023-admitted-executable-operation-packages.md" - "0024-anchored-node-creation-from-absence.md" - "0025-scheduler-owned-executable-operation-actions.md" - "0026-durable-external-action-settlement.md" -) - -readonly current_adr_last=26 - -readonly superseded_legacy_adrs=( - "ADR-0003-Materialization-Bus.md" - "ADR-0005-Physics.md" - "ADR-0006-Ban-Non-Determinism.md" - "ADR-0009-Inter-Worldline-Communication.md" -) - -readonly partially_superseded_legacy_adrs=( - "ADR-0007-BOAW-Storage.md" - "ADR-0011-explicit-observation-contract.md" -) - -readonly collided_paths=( - "docs/adr/0001-repository-knowledge-model.md" - "docs/adr/0002-echo-continuum-authority-boundary.md" - "docs/adr/0003-generated-rule-authorship-and-footprints.md" - "docs/adr/0004-registry-provider-host-boundary.md" - "docs/adr/0005-continuum-transport-identity.md" - "docs/adr/0006-universal-little-endian-codec.md" - "docs/adr/0007-sessions-causal-posture-and-authority.md" - "docs/adr/0008-bunny-owns-reusable-geometry.md" -) - -tmp_dir="$(mktemp -d)" -trap 'rm -rf "$tmp_dir"' EXIT - -failures=0 -fail() { - echo "adr-namespace: $*" >&2 - failures=$((failures + 1)) -} - -for basename in "${legacy_adrs[@]}" "${current_adrs[@]}"; do - path="docs/adr/${basename}" - [[ -f "$path" ]] || fail "missing canonical record ${path}" -done - -for basename in "${superseded_legacy_adrs[@]}"; do - path="docs/adr/${basename}" - if ! grep -Eq -- '^- \*\*Status:\*\* Superseded$|^- Status: Superseded$' "${path}"; then - fail "superseded legacy ADR lacks superseded status: ${path}" - fi - if ! grep -F -- "(${basename})" docs/adr/README.md | grep -Fq -- '| Superseded'; then - fail "ADR index does not mark ${basename} superseded" - fi -done - -for basename in "${partially_superseded_legacy_adrs[@]}"; do - path="docs/adr/${basename}" - if ! grep -Eq -- '^- \*\*Status:\*\* Partially superseded$|^- Status: Partially superseded$' "${path}"; then - fail "partially superseded legacy ADR lacks partial status: ${path}" - fi - if ! grep -F -- "(${basename})" docs/adr/README.md | grep -Fq -- '| Partially superseded'; then - fail "ADR index does not mark ${basename} partially superseded" - fi -done - -for path in "${collided_paths[@]}"; do - if [[ -e "$path" ]]; then - fail "collided ADR path still exists: ${path}" - fi - - if git grep -Fq -- "$path" -- . \ - ':(exclude)tests/docs/test_adr_namespace.sh'; then - fail "live reference still uses collided ADR path: ${path}" - fi -done - -while IFS= read -r path; do - basename="${path##*/}" - case "$basename" in - ADR-[0-9][0-9][0-9][0-9]-*.md) - id="${basename#ADR-}" - id="${id%%-*}" - ;; - [0-9][0-9][0-9][0-9]-*.md) - id="${basename%%-*}" - ;; - *) - fail "non-canonical ADR filename: ${path}" - continue - ;; - esac - printf '%s|%s\n' "$id" "$path" >>"${tmp_dir}/records" -done < <(find docs/adr -maxdepth 1 -type f -name '*.md' ! -name README.md | sort) - -if [[ -s "${tmp_dir}/records" ]]; then - cut -d'|' -f1 "${tmp_dir}/records" | sort >"${tmp_dir}/actual-ids" - duplicate_ids="$(uniq -d "${tmp_dir}/actual-ids")" - [[ -z "$duplicate_ids" ]] || fail "duplicate ADR IDs: ${duplicate_ids//$'\n'/, }" -else - : >"${tmp_dir}/actual-ids" - fail "no canonical ADR records found" -fi - -for ((number = 1; number <= current_adr_last; number += 1)); do - printf '%04d\n' "$number" -done >"${tmp_dir}/expected-ids" - -if ! diff -u "${tmp_dir}/expected-ids" "${tmp_dir}/actual-ids" >"${tmp_dir}/id-diff"; then - cat "${tmp_dir}/id-diff" >&2 - fail "ADR IDs must be the unique contiguous range 0001 through $(printf '%04d' "$current_adr_last")" -fi - -for basename in "${legacy_adrs[@]}" "${current_adrs[@]}"; do - count="$(grep -Foc -- "(${basename})" docs/adr/README.md || true)" - [[ "$count" == "1" ]] || fail "README must link ${basename} exactly once (found ${count})" -done - -while IFS= read -r link; do - [[ -f "docs/adr/${link}" ]] || fail "README link does not resolve: docs/adr/${link}" -done < <(perl -ne 'while (/\]\(([^)#]+\.md)(?:#[^)]*)?\)/g) { print "$1\n" }' docs/adr/README.md) - -for basename in "${current_adrs[@]}"; do - id="${basename%%-*}" - path="docs/adr/${basename}" - if [[ -f "$path" ]] && ! grep -Eq -- "^# ADR ${id}:" "$path"; then - fail "H1 number does not match filename: ${path}" - fi -done - -for basename in "${legacy_adrs[@]}"; do - id="${basename#ADR-}" - id="${id%%-*}" - path="docs/adr/${basename}" - if [[ -f "$path" ]] && ! grep -Eq -- "^# ADR-${id}:" "$path"; then - fail "H1 number does not match filename: ${path}" - fi -done - -if ((failures > 0)); then - echo "adr-namespace: ${failures} violation(s)" >&2 - exit 1 -fi - -echo "adr-namespace: canonical IDs 0001 through $(printf '%04d' "$current_adr_last") are unique and indexed" diff --git a/tests/docs/test_documentation_model.sh b/tests/docs/test_documentation_model.sh new file mode 100755 index 00000000..f278e867 --- /dev/null +++ b/tests/docs/test_documentation_model.sh @@ -0,0 +1,127 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: Apache-2.0 +# © James Ross Ω FLYING•ROBOTS + +set -euo pipefail + +repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +cd "$repo_root" + +# This is an immutable archive inventory, not an allocator. Array order carries +# no architectural meaning; current relationships live in named documents. +readonly archived_records=( + "ADR-0001-warp-two-plane-skeleton-and-attachments.md" + "ADR-0002-warp-instances-descended-attachments.md" + "ADR-0003-Materialization-Bus.md" + "ADR-0004-No-Global-State.md" + "ADR-0005-Physics.md" + "ADR-0006-Ban-Non-Determinism.md" + "ADR-0007-BOAW-Storage.md" + "ADR-0008-Worldline-Runtime-Model.md" + "ADR-0009-Inter-Worldline-Communication.md" + "ADR-0010-observational-seek-and-administrative-rewind.md" + "ADR-0011-explicit-observation-contract.md" + "0012-repository-knowledge-model.md" + "0013-echo-continuum-authority-boundary.md" + "0014-generated-rule-authorship-and-footprints.md" + "0015-registry-provider-host-boundary.md" + "0016-continuum-transport-identity.md" + "0017-universal-little-endian-codec.md" + "0018-sessions-causal-posture-and-authority.md" + "0019-bunny-owns-reusable-geometry.md" + "0020-retained-reading-storage-and-proof-boundary.md" + "0021-public-optic-observation-boundary.md" + "0022-application-requested-causal-anchor-admission.md" + "0023-admitted-executable-operation-packages.md" + "0024-anchored-node-creation-from-absence.md" + "0025-scheduler-owned-executable-operation-actions.md" + "0026-durable-external-action-settlement.md" +) + +failures=0 +fail() { + echo "documentation-model: $*" >&2 + failures=$((failures + 1)) +} + +for required_path in \ + docs/DOCUMENTATION_STANDARDS.md \ + docs/README.md \ + docs/topics/README.md \ + docs/adr/README.md; do + [[ -f "$required_path" ]] || fail "missing policy owner ${required_path}" +done + +if ! grep -Fq -- 'closed historical archive' docs/DOCUMENTATION_STANDARDS.md; then + fail "documentation policy does not close the numbered ADR archive" +fi + +if ! grep -Fq -- 'Do not allocate a new numbered ADR.' AGENTS.md; then + fail "agent policy still permits numbered ADR allocation" +fi + +if grep -ERiq --exclude-dir=adr \ + '(requires?|create|write|add|allocate) (a |an )?(new |separate )?ADR' \ + docs; then + fail "current documentation still routes a durable decision into a new ADR" +fi + +tmp_dir="$(mktemp -d)" +trap 'rm -rf "$tmp_dir"' EXIT + +printf '%s\n' "${archived_records[@]}" | sort >"${tmp_dir}/expected" +find docs/adr -maxdepth 1 -type f -name '*.md' ! -name README.md \ + -exec basename {} \; | sort >"${tmp_dir}/actual" + +if ! diff -u "${tmp_dir}/expected" "${tmp_dir}/actual" >"${tmp_dir}/archive-diff"; then + cat "${tmp_dir}/archive-diff" >&2 + fail "numbered ADR archive differs from its closed inventory" +fi + +for basename in "${archived_records[@]}"; do + count="$(grep -Foc -- "(${basename})" docs/adr/README.md || true)" + [[ "$count" == "1" ]] || \ + fail "archive index must link ${basename} exactly once (found ${count})" + + case "$basename" in + ADR-[0-9][0-9][0-9][0-9]-*.md) + id="${basename#ADR-}" + id="${id%%-*}" + expected_heading="# ADR-${id}:" + ;; + [0-9][0-9][0-9][0-9]-*.md) + id="${basename%%-*}" + expected_heading="# ADR ${id}:" + ;; + *) + fail "unexpected historical filename shape: ${basename}" + continue + ;; + esac + + if ! grep -Fq -- "$expected_heading" "docs/adr/${basename}"; then + fail "historical heading does not match filename: docs/adr/${basename}" + fi +done + +while IFS= read -r link; do + [[ -f "docs/adr/${link}" ]] || \ + fail "archive index link does not resolve: docs/adr/${link}" +done < <(perl -ne 'while (/\]\(([^)#]+\.md)(?:#[^)]*)?\)/g) { print "$1\n" }' docs/adr/README.md) + +if ! grep -Eq -- '^- \*\*Status:\*\* Superseded$' \ + docs/adr/0012-repository-knowledge-model.md; then + fail "historical repository knowledge model is not marked superseded" +fi + +if ! grep -F -- '(0012-repository-knowledge-model.md)' docs/adr/README.md | \ + grep -Fq -- '| Superseded'; then + fail "archive index does not mark repository knowledge model superseded" +fi + +if ((failures > 0)); then + echo "documentation-model: ${failures} violation(s)" >&2 + exit 1 +fi + +echo "documentation-model: semantic policy current; numbered ADR archive closed and indexed" diff --git a/tests/docs/test_knowledge_model.sh b/tests/docs/test_knowledge_model.sh index 02afc3eb..029b5273 100644 --- a/tests/docs/test_knowledge_model.sh +++ b/tests/docs/test_knowledge_model.sh @@ -8,6 +8,7 @@ repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" cd "${repo_root}" readonly required_docs=( + "docs/DOCUMENTATION_STANDARDS.md" "docs/adr/README.md" "docs/adr/0012-repository-knowledge-model.md" "docs/adr/0013-echo-continuum-authority-boundary.md" From 80d4e9ae25acb2506ff4ec0489251b8c92e5bc40 Mon Sep 17 00:00:00 2001 From: James Ross Date: Mon, 10 Aug 2026 18:47:03 -0700 Subject: [PATCH 6/7] docs: bind Echo-Keep publication evidence --- docs/adr/0012-repository-knowledge-model.md | 3 +++ .../echo-keep-physical-content-boundary.md | 22 +++++++++++-------- 2 files changed, 16 insertions(+), 9 deletions(-) diff --git a/docs/adr/0012-repository-knowledge-model.md b/docs/adr/0012-repository-knowledge-model.md index 9ad0d4ff..9e87f2af 100644 --- a/docs/adr/0012-repository-knowledge-model.md +++ b/docs/adr/0012-repository-knowledge-model.md @@ -27,6 +27,9 @@ The current policy is the semantically named separation of current truth, motion, evidence, and history while replacing the numbered ADR queue with concept-owned decisions and explicit relationships. +The five-home model below is retained historical text from July 2026. It does +not govern current ownership; the Documentation Standards policy does. + Repository knowledge has five non-overlapping homes: 1. **GitHub issues, pull requests, and projects own motion.** Priority, current diff --git a/docs/architecture/echo-keep-physical-content-boundary.md b/docs/architecture/echo-keep-physical-content-boundary.md index bf545a35..0ab8507f 100644 --- a/docs/architecture/echo-keep-physical-content-boundary.md +++ b/docs/architecture/echo-keep-physical-content-boundary.md @@ -131,8 +131,9 @@ The adapter follows one visibility protocol: 1. Begin a destination transaction and obtain its private staging writer. 2. Reconstruct through the backend into that writer. 3. On any failure, abort; a staged prefix may remain but is never visible. -4. Seal the staging artifact and verify the complete receipt, Echo identity, - and exact length. +4. Seal the staging artifact and verify that the complete receipt binds the + requested target, pinned view, Keep identity, Echo identity, and exact + logical length. 5. Atomically commit the sealed artifact or return an operational failure. 6. Emit an Echo content observation only after commit succeeds. @@ -150,7 +151,7 @@ unpublished temporary artifact │ ├── failure ──▶ discard; reveal nothing │ - └── receipt ──▶ verify Echo identity; promote + └── receipt ──▶ verify target, view, identities, and length; promote ``` Quarantine may be a bounded memory buffer for small content or a temporary @@ -290,12 +291,15 @@ The governing invariant is: > Orphaned physical content is acceptable. A committed Echo reference to > unavailable content is not. -Echo must commit the authorized physical-publication request and claim before -Keep performs the publication effect. Keep then publishes under a durable -reconciliation anchor keyed by that operation identity, and Echo records the -result as a settlement or observation before any execution resumes on it. This -is the request-before-effect and settlement-before-resumption law from -[ADR 0026](../adr/0026-durable-external-action-settlement.md). +Echo must commit a non-authoritative physical-publication intent and operation +claim before Keep performs the publication effect. That intent and claim are +WAL coordination records: they cannot be read as an Echo content reference or +content observation. Keep then publishes under a durable reconciliation anchor +keyed by that operation identity. After durable reconciliation, Echo creates +the authoritative content reference and records the result as a settlement or +observation before any execution resumes on it. This is the +request-before-effect and settlement-before-resumption law from [ADR +0026](../adr/0026-durable-external-action-settlement.md). The reconciliation anchor is not a time-expiring lease. Keep must retain it until an explicit recovery transition proves that long-term retention is From 01ac62a531835b18a008249f34beee090b377347 Mon Sep 17 00:00:00 2001 From: James Ross Date: Mon, 10 Aug 2026 18:55:04 -0700 Subject: [PATCH 7/7] docs: close obsolete ADR routes --- ADVANCED_GUIDE.md | 3 ++- ARCHITECTURE.md | 3 ++- CONTRIBUTING.md | 10 +++++---- GUIDE.md | 4 +++- README.md | 7 ++++--- ...ined-reading-storage-and-proof-boundary.md | 1 + tests/docs/test_documentation_model.sh | 21 ++++++++++++++++++- 7 files changed, 38 insertions(+), 11 deletions(-) diff --git a/ADVANCED_GUIDE.md b/ADVANCED_GUIDE.md index 85c31b5f..601515fa 100644 --- a/ADVANCED_GUIDE.md +++ b/ADVANCED_GUIDE.md @@ -15,7 +15,8 @@ Use [GUIDE.md](GUIDE.md) for operator orientation and the - [Strictly Deterministic Math](docs/determinism/SPEC_DETERMINISTIC_MATH.md) owns the normative numeric policy; - [specifications](docs/spec/), [invariants](docs/invariants/), and - [ADRs](docs/adr/) own executable contracts and durable decisions. + [topics](docs/topics/) own executable contracts and concept-owned durable + decisions under the [Documentation Standards](docs/DOCUMENTATION_STANDARDS.md). Do not recover the former graph-substrate, stale spec-path, generated-client, or fabricated trig-oracle guidance from this page. Git history is the archive, diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 991404a0..63601eb6 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -20,7 +20,8 @@ owns the boundary you are changing: [Runtime Constellation](docs/topics/RuntimeConstellation.md), and [WARP Optics](docs/topics/WarpOptics.md) — living cross-module doctrine; - [specifications](docs/spec/), [invariants](docs/invariants/), and - [ADRs](docs/adr/) — executable contracts and durable decisions. + [topics](docs/topics/) — executable contracts and concept-owned durable + decisions under the [Documentation Standards](docs/DOCUMENTATION_STANDARDS.md). The current north star is: diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 5432b9e0..ab10e573 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -48,8 +48,9 @@ prioritize: - Keep `main` pristine. Create feature branches like `echo/` or `timeline/`. - Before starting work, ensure `git status` is clean. If not, resolve or coordinate with the human operator. - Keep change-local design, test plans, review state, and follow-up work in the - GitHub issue or pull request. Record only durable architectural decisions as - ADRs in `docs/adr/`. + GitHub issue or pull request. Record durable architectural decisions in the + semantically named current document that owns the concept, following + `docs/DOCUMENTATION_STANDARDS.md`. ## Testing Expectations @@ -73,8 +74,9 @@ prioritize: ## Documentation & Telemetry - Update relevant docs in `docs/` whenever behavior or architecture changes. -- Record durable architectural decisions in ADRs (`docs/adr/`). Keep - change-local rationale in the issue or pull request. +- Record durable architectural decisions with their owning concept under + `docs/DOCUMENTATION_STANDARDS.md`. Keep change-local rationale in the issue or + pull request; treat `docs/adr/` as a closed historical archive. ## Submitting Changes diff --git a/GUIDE.md b/GUIDE.md index 824bbe8d..67959a45 100644 --- a/GUIDE.md +++ b/GUIDE.md @@ -62,7 +62,9 @@ If you need a comprehensive spec, use the [docs/README.md](./docs/README.md) map If you need current architectural truth, use the [architecture](./docs/architecture/), [specification](./docs/spec/), [invariant](./docs/invariants/), and [topic](./docs/topics/) maps. Durable -decisions live in [ADRs](./docs/adr/); live work and status live in GitHub. +decisions live with their owning concept under the +[Documentation Standards](./docs/DOCUMENTATION_STANDARDS.md); the numbered +[ADR directory](./docs/adr/) is historical. Live work and status live in GitHub. If you are just starting, use the [README.md](./README.md) and the orientation tracks above. diff --git a/README.md b/README.md index 978a156f..76ad2658 100644 --- a/README.md +++ b/README.md @@ -601,9 +601,10 @@ the external Hello Echo application. Compiler-produced application packages use the generic `run-edict-operation` witness command. Live work, priorities, and status are maintained in GitHub Issues, Projects, -pull requests, and review threads. Architectural decisions live in -[`docs/adr/`](docs/adr/), while current doctrine lives in -[`docs/topics/`](docs/topics/). +pull requests, and review threads. Durable decisions live in the semantically +named current document that owns their concept, under the +[`Documentation Standards`](docs/DOCUMENTATION_STANDARDS.md). The numbered +[`docs/adr/`](docs/adr/) directory is a closed historical archive. ## Benchmarks And Gates diff --git a/docs/adr/0020-retained-reading-storage-and-proof-boundary.md b/docs/adr/0020-retained-reading-storage-and-proof-boundary.md index 23e957dc..2dcd7343 100644 --- a/docs/adr/0020-retained-reading-storage-and-proof-boundary.md +++ b/docs/adr/0020-retained-reading-storage-and-proof-boundary.md @@ -5,6 +5,7 @@ - **Status:** Accepted - **Date:** 2026-07-13 +- **Refined by:** [Echo–Keep Physical Content Boundary](../architecture/echo-keep-physical-content-boundary.md) ## Context diff --git a/tests/docs/test_documentation_model.sh b/tests/docs/test_documentation_model.sh index f278e867..1f801d75 100755 --- a/tests/docs/test_documentation_model.sh +++ b/tests/docs/test_documentation_model.sh @@ -60,9 +60,28 @@ if ! grep -Fq -- 'Do not allocate a new numbered ADR.' AGENTS.md; then fail "agent policy still permits numbered ADR allocation" fi +readonly root_guidance=( + README.md + GUIDE.md + CONTRIBUTING.md + ARCHITECTURE.md + ADVANCED_GUIDE.md +) + +if grep -Eiq \ + 'Architectural decisions live in|decisions live in \[ADRs|Record (only )?durable architectural decisions (as|in) ADRs|\[ADRs\].*durable decisions' \ + "${root_guidance[@]}"; then + fail "root guidance still treats the numbered ADR archive as current ownership" +fi + +readonly current_guidance=( + "${root_guidance[@]}" + docs +) + if grep -ERiq --exclude-dir=adr \ '(requires?|create|write|add|allocate) (a |an )?(new |separate )?ADR' \ - docs; then + "${current_guidance[@]}"; then fail "current documentation still routes a durable decision into a new ADR" fi