Add crash-tested segment-store recovery

.github/workflows/ci.yml

@@ -33,6 +33,12 @@ jobs:
       - name: Check Golden File Worldline corpus
         run: cargo xtask golden-file-worldline-check
 
+      - name: Run process-death crash matrix (debug)
+        run: cargo xtask durability-crash-matrix
+
+      - name: Run process-death crash matrix (optimized)
+        run: cargo run --quiet --release --locked --package xtask -- durability-crash-matrix
+
       - name: Check protocol conformance corpora
         run: cargo xtask conformance-check
 

CHANGELOG.md

@@ -10,13 +10,112 @@ after its public API and format compatibility policies are established.
 
 ### Changed
 
+- Repository crash-matrix execution now terminates isolated writer process
+  groups at all 105 canonical before/during/after coordinates, retains open
+  writer and stage authority until termination, executes production
+  initialization, segment-writing, catalog-publication, and recovery-discard
+  protocols through fault-injecting port decorators, and verifies exact Golden
+  File Worldline namespaces, bytes, hard links, released locks, recovery
+  classifications, immutable artifacts, and published visible state after
+  restart. CI runs the complete matrix in debug and optimized profiles.
+- Production filesystem initialization now admits only one documented
+  writable, non-casefolded Linux ext4 profile, independently applies it to
+  every existing protocol directory, requires each child to share the root's
+  device and mount identity, refuses ambiguous or foreign root namespaces
+  before mutation, completes the canonical directory shape idempotently,
+  retains writer authority, and returns only after synchronizing the root.
+- Published filesystem stores can now reacquire writer authority without
+  mutation through a typed platform-admission boundary that requires the exact
+  initialized root shape plus a regular `HEAD`.
+- Writer-lock acquisition now reopens `writer.lock` after kernel locking and
+  refuses when the resolved entry no longer has the locked device and inode.
+- Writer authority now also retains an advisory lock on the pinned store-root
+  inode, so replacing `writer.lock` cannot split live cooperative authority.
+- Recovery inventory now counts the root and three protocol directories before
+  retaining names, enforces the configurable protocol-bounded entry ceiling,
+  stops namespace counting at the first globally excessive entry, refuses
+  count drift and duplicates exactly, and returns deterministic
+  namespace-and-raw-byte ordering through a read-only storage port.
+- Filesystem recovery inventory now pins the root and all three protocol
+  directories without following links, verifies child-directory identity
+  before and after bounded scanning, and preserves raw Linux entry-name bytes
+  without mutating protocol state.
+- Fixed recovery stages can now be fingerprinted relative to the pinned
+  recovery inventory capability. Observation admits only regular files, never
+  follows links, streams under the name-selected bound, and refuses entry
+  replacement or length drift without mutating protocol state.
+- Complete caller-supplied segment-stage bytes now classify as a validated
+  reusable prefix, a complete admitted immutable segment, or an exact
+  truncation only while every available fixed-framing byte remains canonical.
+  Proven partial-framing corruption, complete-looking corruption, duplicate
+  identities, and caller-policy excess remain typed refusals.
+- Storage-independent reusable-segment recovery now plans only from an exact
+  reusable assessment within the selected resource policy, consumes reopening
+  authority, re-admits the materialized prefix against saved evidence, rebuilds
+  digest and duplicate-identity state, and returns the ordinary append-only
+  stage without rewriting admitted bytes.
+- Filesystem reusable-segment recovery now retains pinned root, namespace, and
+  writer-lock authority in the returned stage; reopens `current.seg` read-write
+  without following links or truncation; bounds, materializes, and re-admits
+  its exact prefix; recomputes exact stage evidence immediately before handoff;
+  revalidates the final entry and append position; and refuses missing,
+  changed, linked, replaced, or namespace-drifted evidence before writing.
+- Complete caller-supplied catalog and candidate-head stages now distinguish
+  exact fixed-header, declared-body, or fixed-width truncation from canonical
+  bytes only while every available fixed-framing byte remains canonical.
+  Proven partial-framing corruption, complete-looking corruption, and
+  oversized stages remain typed refusals without claiming transitive catalog
+  reachability.
+- Read-only recovery assessment now admits materialized stage bytes only when
+  their canonical-name stage, exact length, and recomputed versioned
+  fingerprint equal prior observation evidence, then dispatches through the
+  stage-selected semantic classifier.
+- Explicit truncated-stage recovery now plans only from an exact semantic
+  truncation, retains its evidence and reason, refuses changed evidence before
+  mutation, and returns a discard receipt only after the name-selected parent
+  directory is synchronized. An already absent stage remains an idempotent
+  input and still requires synchronization.
+- Filesystem truncated-stage discard now admits the platform, retains the root
+  and `writer.lock` locks, pins every protocol namespace, reopens stage bytes
+  without following links, refuses replacement or fingerprint drift before
+  unlink, and synchronizes the typed `staging` or root parent before returning
+  a receipt.
+- Explicit complete-stage recovery now plans only from exact complete segment
+  or catalog assessments, owns bounded stage evidence and immutable-pool
+  coordinates, re-synchronizes an exact present stage before linking, verifies
+  existing pool entries, synchronizes the selected pool before exact stage
+  removal, and returns a valid-orphan receipt only after staging
+  synchronization. It never creates or finalizes a publication head.
+- Filesystem complete-stage recovery now retains pinned root and writer
+  authority, revalidates exact stage evidence at synchronization and link
+  boundaries, uses no-clobber immutable-pool links, never follows stage or pool
+  links, preserves conflicting or replaced entries, verifies exact pool bytes,
+  and accepts stage/pool, reappeared-stage, and completed pool-only retries.
+- Storage-independent next-head recovery now binds a complete `head.next`
+  assessment to its exact transitive catalog snapshot, admits only generation
+  one over an uninitialized root or the exact successor of an expected current
+  snapshot, synchronizes a ready candidate before replacement, distinguishes
+  ready from already-finalized retries, and returns a receipt only after root
+  synchronization.
+- Filesystem next-head recovery now retains pinned root and writer authority,
+  reconstructs complete current and candidate views under exact namespace and
+  stage evidence, synchronizes and reverifies the candidate before atomic
+  replacement, refuses reappeared candidates on retry, and returns only after
+  root synchronization.
+- Store initialization now exposes one storage-port state machine that admits
+  the platform before mutation, opens and locks `writer.lock`, admits the three
+  protocol directories in order, synchronizes the root, and preserves the
+  exact failed phase without executing later transitions.
+- Repository crash-matrix tooling now exposes one typed, ordered vocabulary for
+  `KEEP-CRASH-001` through `KEEP-CRASH-035`. Each identifier is bound to its
+  segment, catalog, head, recovery-discard, or initialization sequence, and
+  only record append admits an occurrence counter.
 - Catalog decoding now verifies the catalog checksum and physical digest before
   interpreting entry semantics. Corrupt identity-bearing bytes therefore fail
   at the integrity boundary instead of producing a semantic entry error.
-- Filesystem catalog publisher construction now consumes an unforgeable
-  `FilesystemPlatformAdmission`. No public producer exists until crash-tested
-  initialization can establish the platform contract in issue #17; acquiring
-  `FilesystemWriterLock` alone no longer authorizes production construction.
+- Filesystem catalog publisher construction consumes an unforgeable
+  `FilesystemPlatformAdmission`; acquiring `FilesystemWriterLock` alone does
+  not authorize production construction.
 - Filesystem segment selection now consumes sealed stages through the publisher
   that created them. Process-local publisher authority prevents an unrelated
   metadata-equivalent `ClosedSegment` from authorizing retained

Cargo.toml

@@ -12,10 +12,15 @@ keywords = ["cas", "storage", "content-addressed", "deduplication"]
 categories = ["data-structures", "filesystem"]
 publish = false
 
+[features]
+default = []
+repository-tasks = []
+
 [dependencies]
 blake3 = { version = "=1.8.5", default-features = false, features = ["pure", "std"] }
 cap-fs-ext = { version = "=4.0.2", default-features = false, features = ["std"] }
 cap-std = { version = "=4.0.2", default-features = false }
+rustix = { version = "=1.1.4", default-features = false, features = ["fs", "std"] }
 
 [dev-dependencies]
 allocation-counter = { version = "=0.8.1", default-features = false }

README.md

@@ -37,8 +37,15 @@ verification. A platform-admitted `FilesystemCatalogPublisher` exclusively
 creates the fixed `current.seg` stage without truncating existing evidence,
 and the `FilesystemSegmentStage` lifetime keeps that writer authority borrowed
 until the writable stage closes. Publisher construction consumes an
-unforgeable `FilesystemPlatformAdmission`; no public producer exists until
-issue #17 implements crash-tested initialization and platform admission.
+unforgeable `FilesystemPlatformAdmission`. On Linux, its public initializer
+admits only one writable, non-casefolded ext4 store profile, requires every
+existing protocol directory to share the root's filesystem and mount identity,
+refuses unknown, aliased, or foreign namespace entries before mutation, creates
+or verifies the canonical `writer.lock`, `staging`, `segments`, and `catalogs`
+shape, and returns only after root synchronization with the writer lock
+retained. After publication, `FilesystemPlatformAdmission::reopen` reacquires
+the existing writer lock without mutation and requires that exact initialized
+shape plus a regular `HEAD` before returning new publisher authority.
 
 `FilesystemCatalogPublisher` retains one kernel-managed writer lock and pinned
 root, staging, segment-pool, and catalog-pool capabilities for the complete
@@ -56,16 +63,69 @@ logical reads.
 
 The reference CAS is executable evidence for M2 storage laws, not a durable
 backend. Its committed state is process memory; process death loses it all.
-The durable boundary does not yet initialize, platform-admit, or recover a
-store root. Acquiring `FilesystemWriterLock` alone cannot construct a
-filesystem publisher. Issue #17 must admit the exact existing `writer.lock`,
-`staging`, `segments`, and `catalogs` namespace before it can return
-`FilesystemPlatformAdmission`. Leftover `head.next`, staged recovery evidence,
-unknown namespace entries, and ambiguous crash states remain explicit recovery
-work. An absent `HEAD` is admitted for first publication only when both
-immutable pools are empty. Retention, complete namespace verification,
-compaction, and garbage collection remain planned. Presence in the reference
-CAS does not claim retention, crash recovery, or durability.
+The durable boundary can initialize or reopen and platform-admit a store only
+under the documented Linux ext4 contract. Acquiring `FilesystemWriterLock`
+alone cannot construct a filesystem publisher. Ambiguous crash states remain
+explicit recovery work. An absent `HEAD` is admitted for first publication
+only when both immutable pools are empty. The public storage-independent
+recovery inventory counts all four protocol namespaces before retaining names,
+applies a configurable ceiling no greater than 2,097,152 entries, and returns
+duplicate-free deterministic raw name order.
+`FilesystemRecoveryInventoryReader` implements that contract with pinned,
+no-follow namespace capabilities and pre/post identity verification on the
+admitted Linux ext4 profile. Its bounded stage-fingerprint operation opens
+fixed stages relative to those capabilities, refuses links and nonregular
+files, and verifies entry identity and length after reading. Complete
+caller-supplied segment-stage bytes can be classified as a reusable prefix,
+complete admitted segment, or exact truncation only while every available
+fixed-framing byte remains canonical. Catalog and next-head stages apply the
+same prefix rule before distinguishing truncation from complete canonical
+bytes.
+Materialized bytes enter read-only semantic assessment only after their stage,
+length, and recomputed fingerprint match prior observation evidence.
+An exact reusable segment assessment can authorize storage-independent
+continuation: the executor consumes writer authority, re-admits the complete
+bounded prefix, rebuilds digest and duplicate-identity state, and returns the
+ordinary append-only stage without rewriting admitted bytes.
+`FilesystemRecoverySegmentResumer` implements that contract with pinned
+namespaces and writer authority, no-follow read-write reopening, exact bounded
+materialization, final streamed fingerprint plus entry and namespace
+revalidation, and an append position equal to the admitted prefix length.
+
+Exact truncation assessments can authorize durable, evidence-bound discard.
+Complete segment and catalog assessments can authorize verified immutable-pool
+completion through `FilesystemRecoveryStageCompleter`; its receipt proves a
+valid orphan, not reachability. A complete `head.next` and its transitive
+`CatalogSnapshot` can authorize storage-independent finalization only when the
+candidate is generation one over an uninitialized root or the exact successor
+of the expected current snapshot. The executor distinguishes first
+finalization from an already-finalized retry and returns only after root
+synchronization. `FilesystemRecoveryNextHeadFinalizer` retains pinned writer
+authority, reconstructs the complete current and candidate views without
+following links, verifies namespace and stage identity, synchronizes and
+reverifies the exact candidate, atomically replaces `HEAD`, and synchronizes
+the root. An already-finalized retry requires `head.next` to be absent.
+The repository-owned process-death matrix executes all 105
+`KEEP-CRASH-001`–`KEEP-CRASH-035` before/during/after coordinates in isolated
+process groups. Crash children execute the production initialization,
+segment-writing, catalog-publication, and recovery-discard protocols through
+fault-injecting port decorators; they do not synthesize the target namespace.
+Restart verification compares the exact Golden File Worldline namespace and
+bytes, checks hard-link identity and writer-lock release, runs the production
+recovery classifiers and immutable-artifact admission, and reconstructs the
+exact published generation and visible one-zero chunk when `HEAD` exists. This
+matrix proves application process-death behavior; it does not simulate host
+power loss. Retention, compaction, and garbage collection remain planned.
+Presence in the reference CAS does not claim retention, crash recovery, or
+durability.
+
+Run the complete debug-profile matrix:
+
+```bash
+cargo xtask durability-crash-matrix
+```
+
+CI also runs the command through an optimized `xtask` build.
 
 ```rust
 use keep::BlobId;

conformance/segment-store/v1/README.md

@@ -105,6 +105,9 @@ complete validation. A `truncated-tail-or-reusable-stage` result, for example,
 means the observed prefix decides between those two typed outcomes; it does
 not authorize truncation.
 
-The table is a planned oracle for issue #17. Until a production recovery
-adapter and crash harness exist, its evidence status is specification, not
-implementation.
+The issue #17 harness executes 105 canonical process-death cases from this
+table. It terminates an isolated writer process group, compares
+the exact restarted namespace and bytes against an independent expected-state
+model, and exercises the production recovery classifiers and Golden File
+Worldline restart loader. The harness proves application process-death
+behavior; host power loss remains outside its claim.

docs/dependencies/cap-std-and-cap-fs-ext-4.0.2.md

@@ -10,8 +10,9 @@
 
 Keep admits the exactly pinned `cap-std` 4.0.2 and `cap-fs-ext` 4.0.2 packages
 for the library's segment-store filesystem adapter and behind the `xtask`
-crate's `repository-tasks` feature. Rustix 1.1.4 remains admitted only behind
-that `xtask` feature.
+crate's `repository-tasks` feature. The library also admits Rustix 1.1.4 for
+safe Linux filesystem-profile inspection and no-symlink root opening; `xtask`
+uses the same exact version behind `repository-tasks`.
 
 `cap-std::fs::Dir` pins the admitted repository or corpus directory and opens
 entries relative to that capability. `cap-fs-ext` supplies no-follow and
@@ -25,8 +26,11 @@ other ambiguous filesystem state before reading source or protocol bytes.
 The capability packages are present in Keep's published library graph and
 production filesystem behavior. No dependency-owned type crosses Keep's public
 API or enters content identities or durable formats. The segment-store writer
-lock retains capability and file handles behind `FilesystemWriterLock`; its
-public acquisition boundary accepts only `std::path::Path`.
+lock retains the root capability plus root-lock and writer-lock file handles
+behind `FilesystemWriterLock`; its public acquisition boundary accepts only
+`std::path::Path`. The production initializer uses Rustix's safe `openat2`,
+`fstatfs`, `fstatvfs`, and ext4 inode flag APIs to admit only the documented
+writable, non-casefolded Linux ext4 profile.
 
 The bounded subprocess adapter uses Rustix's safe filesystem API to mark child
 stdin nonblocking before deadline-bounded input transfer. It uses Rustix's safe
@@ -66,11 +70,12 @@ work, and require unsafe code that Keep otherwise forbids.
 ## Features and resolved graph
 
 All three direct dependencies disable default features. Keep enables only
-`cap-fs-ext`'s `std` feature and Rustix's `fs`, `process`, and `std` features;
-`cap-std` has no enabled feature. The library's capability dependencies are
-unconditional because the production segment-store adapter requires them.
-The `xtask` declarations remain optional and are activated solely by
-`repository-tasks`; Rustix is not a direct library dependency.
+`cap-fs-ext`'s `std` feature and Rustix's `fs` and `std` features in the
+library; `cap-std` has no enabled feature. The library's filesystem
+dependencies are unconditional because the production segment-store adapter
+requires them. The `xtask` declarations remain optional and are activated
+solely by `repository-tasks`; that feature additionally enables Rustix's
+`process` feature.
 
 The locked non-Windows graph introduced for this boundary is:
 
@@ -117,9 +122,9 @@ dependencies.
 
 ## Failure and recovery boundaries
 
-An open, metadata, read, writer-lock acquisition, descriptor-duplication,
-descriptor-flag, child-directory setup, child-spawn, stdin-write,
-output-collection, deadline, or cleanup failure is a typed refusal.
+An open, metadata, platform-profile, read, writer-lock acquisition,
+descriptor-duplication, descriptor-flag, child-directory setup, child-spawn,
+stdin-write, output-collection, deadline, or cleanup failure is a typed refusal.
 Repository tasks never repair, rewrite, or substitute repository data.
 Repository-task handles exist only for one verification process and carry no
 durability or recovery semantics. `FilesystemWriterLock` retains the pinned