Skip to content

Validate C++ disaster recovery traces with Lean - #8282

Draft
Amaury Chamayou (achamayou) wants to merge 6 commits into
achamayou-solid-tribblefrom
achamayou-fluffy-parakeet
Draft

Validate C++ disaster recovery traces with Lean#8282
Amaury Chamayou (achamayou) wants to merge 6 commits into
achamayou-solid-tribblefrom
achamayou-fluffy-parakeet

Conversation

@achamayou

Copy link
Copy Markdown
Member

Summary

Part 4/4 of the replacement stack for #8241. This PR validates real C++ recovery-decision-protocol executions against the permanent Lean model introduced below it.

Stack:

This PR depends on draft deletion PR #8280 and must remain gated until the lower layers are ready.

Approach

  • Defers the production host restart until the JOINING state transaction globally commits in both default and trace-enabled builds. In trace-enabled builds, the committed join_restart observation is emitted before restart is requested. This is a commit-before-restart guarantee, not an exactly-once guarantee.
  • Adds the OFF-by-default CCF_RECOVERY_TRACE option, a commit-aware trace table, versioned RDP_TRACE records, per-node sequences, message IDs, causal IDs, exact ordered retry-send batches, accepted receives, committed timeouts, and one-shot effects.
  • Adds an isolated package at lean/disaster-recovery-trace/. It depends locally on ../disaster-recovery and imports only DisasterRecovery.Protocol.Model; no trace modules or targets are added to the canonical package.
  • Defines and documents the strict ccf.recovery_decision_protocol.trace/1 NDJSON contract. The validator rejects the shortest incompatible prefix and checks parser/replay behavior without treating synthetic tests as implementation evidence.
  • Extracts records from each node, validates identity, sequence, message IDs, and causal edges, deterministically topologically orders them, writes *.recovery.ndjson, and invokes the isolated validator.
  • Wires quorum, failover, and repeated-timeout recovery scenarios into the Milan and Genoa SNP jobs. Those jobs build with tracing enabled and upload captured NDJSON even on failure.

Evidence and isolation

Real captured C++ SNP traces are the implementation conformance evidence. Small in-memory Lean and Python tests cover rejection, extraction, and ordering only.

lean/disaster-recovery/ remains byte-for-byte unchanged. This PR does not recreate lean/disaster-recovery-migration/ or tla/disaster-recovery/, and contains no migration, Rust, or Stateright code.

Validation

  • Built node_frontend_test with CCF_RECOVERY_TRACE both OFF and ON.
  • Ran node_frontend_test through each build's tests.sh wrapper.
  • Ran the focused Python extraction/ordering suite (8 tests).
  • Ran lake exe cache get, lake build, warnings-as-errors/no-sorry checks, and lake exe trace-checks for the isolated package from a shorter temporary checkout. The generated Copilot worktree path is long enough to expose a local Lake output-directory issue; the same committed package builds successfully at the shorter path used by normal CI checkouts.
  • Ran scripts/ci-checks.sh -f successfully.

Gates

Keep this PR in draft while #8280 remains gated. The coordinator will add GitHub preview stack metadata after all four PRs pass preflight.

Request host restart only from the committed JOINING state hook so aborted recovery transactions cannot trigger a restart.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Gate versioned RDP_TRACE records behind CCF_RECOVERY_TRACE and publish receive, timeout, effect, and retry-send events only at their required commit boundaries.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replay strict version 1 recovery traces against the canonical model through a local package dependency, with focused rejection tests, no-sorry checks, documentation, and a shallow workflow.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Validate trace identity, per-node sequences, message IDs, and causal edges before producing a deterministic NDJSON linearization for the isolated Lean validator.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Build the isolated validator in Milan and Genoa trace-enabled jobs, validate quorum, failover, and repeated-timeout recoveries, and retain generated NDJSON artifacts on failure.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Exercise the focused Python extraction and causal-ordering suite in the dedicated trace workflow.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant