Skip to content

scry#215: the MC/DC evidence and the shipped artifact now share a compiler - #242

Merged
avrabe merged 1 commit into
mainfrom
feat215-evidence-substrate-rebased
Nov 17, 2026
Merged

avrabe merged 1 commit into
mainfrom
feat215-evidence-substrate-rebased

Conversation

@avrabe

@avrabe avrabe commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Supersedes #236: the same commit rebased onto main 75fd030. #236 conflicts with main and its session ended on 2026-11-08, so this is a new branch rather than a push onto #236's.

Two changes in the rebase:

What it does (full rationale in the commit message):

  • crates/scry-mcdc/rust-toolchain.toml pins the MC/DC subject to rustc 1.91.1. That is the compiler recorded in v3.3.0's signed scry-3.3.0-wasm32-wasip2.wasm. The compiler skew alone was measured at 13 proved conditions (395 → 382).
  • tools/check-evidence-substrate.py runs in the required MC/DC (witness) job. It asserts the toolchain file against MODULE.bazel and the active rustc against the file. The witness-version skew (rev 6ff96e75 = v0.29.0 in the gate, v0.30.0 for released evidence) stays open, as a self-cleaning allowlist entry owned by The MC/DC coverage evidence measures a different binary than the one we ship — different compiler, profile, and target #215.
  • The FEAT-072 release delta copies the toolchain file into the previous-tag worktree, so both sides build under the same compiler.

Re-verified on the rebased tree:

Check Result
check-evidence-substrate.py --self-test OK
check-evidence-substrate.py exit 0, 1 known-open skew owned by #215, nothing stale
check-gate-coverage.py 13 crates in both gates
check-required-checks.py --against-file PASS
check_selfhosted_fork_guard.py 9 jobs guarded
claim-check.py 7/7
rivet validate PASS (one prose-mention warning on FEAT-103, same kind as existing artifacts)
commit trailer guard PASS

Not re-run here: the 9-mutation suite from #236's commit message, and the full MC/DC gate under 1.91.1. The MC/DC job in this PR's CI covers the second.

Refs #215. Related, still open: #235 (witness version), #237 (unsigned evidence asset).

🤖 Generated with Claude Code

https://claude.ai/code/session_01KEjmjcVaTucsMZy5Fubas7

…piler

The MC/DC gate, the released mcdc-evidence tarball, the Pages dashboard
and the FEAT-072 release delta all measure scry_mcdc.wasm — a wasip1 core
module built by Cargo — while the thing we ship and sign is scry.wasm, a
wasip2 component built by Bazel. The wasip1/DWARF choice is deliberate
and documented. Two other skews were not.

BOTH MEASURED ON SHIPPED ARTIFACTS, because a build config is a
hypothesis about an artifact and not evidence about it.

(1) COMPILER. v3.3.0's signed scry-3.3.0-wasm32-wasip2.wasm records

    $ wasm-tools metadata show scry-3.3.0-wasm32-wasip2.wasm
      name         scry_analyzer_component_wasm_lib_release_wasm_base.wasm
      processed-by rustc [1.91.1 (ed61e7d7e 2025-11-07)]

confirming MODULE.bazel. The MC/DC subject had NO rust-toolchain.toml
anywhere in the repo and was built by whatever @stable resolved to that
day, so the skew widened on its own and nothing recorded which compiler
produced any given number.

(2) TOOL. v3.3.0's evidence manifest records witness_version "0.30.0",
while the CI gate installs rev 6ff96e75 — which git ls-remote resolves to
v0.29.0^{}, NOT v0.30.0. The floors were proved by one tool version and
the released evidence produced by another. Upstream is at v0.32.0.

#215'S PROPOSED FIX IS NOT POSSIBLE AS WRITTEN, and the measurement is
why: wasmtime 48.0.8 declares rust-version = 1.95.0, so the workspace
cannot be pinned to 1.91.1. It IS possible for the MC/DC subject alone,
because crates/scry-mcdc is its OWN workspace, depends only on
scry-sai-core, has zero wasmtime in its graph, and mcdc-gate.sh does
`cd "$(dirname "$0")"` before building — so a toolchain file there
governs exactly that build and nothing else.

THE SKEW WAS WORTH 13 CONDITIONS. Measured with witness held constant at
the gate's own pinned rev, changing ONLY the compiler:

    rustc 1.96.1 -> 1.91.1
    decisions_total      1626 -> 1677   (+51)
    conditions_proved     395 ->  382   (-13)
    decisions_full_mcdc    18 ->   16   (-2)

Both clear the floors (155 / 3) with margin, so the gate stays green. But
the evidence was measurably compiler-dependent while one side floated.

THE FILE IS NOT THE COMPILER. Comparing rust-toolchain.toml against
MODULE.bazel is a claim about what WILL compile — the same
config-as-evidence error this change exists to fix, one level up. rustup
precedence lets RUSTUP_TOOLCHAIN or a directory override beat a toolchain
file, and a green gate under the wrong rustc looks identical to a green
one. So the guard step also asserts the ACTIVE rustc where the build
happens and prints it, which puts the evidence's substrate in the CI log.
Negative-controlled, and the hazard is real rather than theoretical:
RUSTUP_TOOLCHAIN=1.98.1 DOES beat the file (measured: rustc 1.98.1) and
the assertion catches it.

A CONSEQUENCE THE MEASUREMENT EXPOSED: the FEAT-072 release delta builds
the previous tag in a git worktree, which predates the toolchain file —
so the old side would compile under @stable and the new side pinned, and
a ~51-decision compiler artifact would be reported as source change,
which is the one thing that delta exists to isolate. The delta step now
copies the toolchain file into the worktree first, and says so in its
status line.

TWO MORE REFERENCES WERE FOUND BY WRITING THE CHECK. The witness rev also
appears bare in two actions/cache keys:

    key: witness-bins-${{ runner.os }}-6ff96e75a1b495aa165f1ee91abda3049fffe628

Bump an install rev and leave a cache key stale, and the cache serves
binaries the install step no longer names — the gate would report
coverage from a witness it does not pin. 4 references found by the first
version of the check, 6 by this one.

WIRED INTO A REQUIRED JOB, which needed a design decision. A check that
is intentionally red cannot be required, and one that is not wired is
gate-potency's defined-but-not-enforced hole. So EXPECTED_SKEWS reports a
known-open skew instead of failing it — and the allowlist is
SELF-CLEANING: an entry whose skew is no longer present FAILS and names
itself, so it cannot outlive the defect. Losing the subject entirely is
never allowlistable, even by its own key. The step sits in
`MC/DC (witness)`, no continue-on-error, no job-level if.

MUTATION-CHECKED ON THE FINAL FILE: 9 mutations (M1-M9), 9 killed,
unmutated control green. An earlier run of 7 did not cover the bare-SHA
scan, which was added afterwards; this is the full suite re-run on what
is committed.

THREE survived a first pass, and every one was a gap in the TABLE rather
than a defect in the code — the same error three times:

  * the SHA-format regex was never load-bearing, because every tag case
    also carried two DIFFERENT refs, so len(distinct) > 1 blocked them
    regardless. Fixed with a case where every ref is an IDENTICAL tag.
  * the lost-subject mutation was never tested by its own allowlist key;
    the case allowlisted `witness-version`, not `no-refs`.
  * dropping the bare-SHA scan's `witness`-line guard survived because no
    unrelated 40-hex string exists in the workflows TODAY. scry#232 is
    pinning every GitHub Action to a commit SHA, so an unguarded scan
    would have started failing CI for no reason the day it merged. Fixed
    by running the PARSER against synthetic workflow text, which none of
    the other cases did — they all feed `verdict` pre-built tuples.

Each case exercised the OVERLAP of two defences rather than the gap
between them, which is also what survived the first attempt at scry#208.

CORRECTION CARRIED IN THE ARTIFACT: the first draft of this change called
the mcdc-evidence tarball "signed". IT IS NOT. Measured on v3.3.0 — of 20
primary assets, 18 are listed in a cosign-signed SHA256SUMS.txt and 18
also carry individual .sig/.pem pairs; scry-v3.3.0-mcdc-evidence.tar.gz
is in NEITHER and is the ONLY primary asset with no integrity protection.
The error came from filtering the asset list on `mcdc|evidence`, seeing
compliance-evidence.tar.gz return with its .pem and .sig, and carrying
the signing across to the neighbouring bundle. scry#215's own body says
"the evidence bundle we sign and ship", so it was already in circulation;
it also reached witness#360 and has been corrected there.

That makes it a REQUIREMENT VIOLATION, not a gap: REQ-010 — the
requirement this traces to — asks for the evidence "surfaced as a SIGNED
truth-table report". It is surfaced and it is not signed. Filed
separately rather than fixed here; adding an asset to the signing set is
a release-path change and this branch already touches release.yml once.

THE WITNESS SKEW IS LEFT OPEN DELIBERATELY. The v0.30.0 side is
internally coherent — the cargo install --tag and the compliance@v0.30.0
action match each other — so converging either changes the tool that
proved the floors or alters an unsimulated release path. Tracked by the
allowlist, which cannot rot.

VERIFIED (all local; no CI has run on this SHA):
  rustc --version in crates/scry-mcdc -> 1.91.1 (ed61e7d7e 2025-11-07),
    byte-identical to the shipped component's processed-by string
  mcdc-gate.sh with witness 0.29.0 -> exit 0, 382 >= 155, 16 >= 3
  check-evidence-substrate.py --self-test -> OK (13 cases + a parser
    probe, both outcomes exercised)
  check-evidence-substrate.py -> exit 0, 1 known-open skew, nothing stale
  check-gate-coverage.py -> OK, 13 crates in both gates (ci.yml edited)
  ci.yml and release.yml both still parse as YAML
  rivet validate -> PASS

Refs: FEAT-103

Rebased onto main 75fd030 from #236 (de09aa9), whose session ended on
2026-11-08. Two changes in the rebase: the ci.yml conflict keeps main's
SHA-pinned rust-cache, and the artifact is renumbered FEAT-102 ->
FEAT-103, because #240 took FEAT-102 first. Re-run on the rebased tree:
check-evidence-substrate --self-test OK and check exit 0 (1 known-open
skew, owned by #215); gate-coverage, required-checks, fork-guard and
claim-check all pass; rivet validate PASS.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KkNzkNYzPh7366DkNijeNc
Claude-Session: https://claude.ai/code/session_01KEjmjcVaTucsMZy5Fubas7
@github-actions

Copy link
Copy Markdown

📐 rivet artifact delta

PR: #242 Base SHA: 75fd0308

Validation

head — `rivet validate` result
  SR-11 (sw-req, status: accepted) — missing: sw-integration-verification, unit-verification
  SR-12 (sw-req, status: accepted) — missing: sw-integration-verification, unit-verification
  SR-13 (sw-req, status: accepted) — missing: sw-integration-verification, unit-verification
  SR-2 (sw-req, status: accepted) — missing: sw-integration-verification, unit-verification
  SR-3 (sw-req, status: accepted) — missing: sw-integration-verification, unit-verification
  SR-4 (sw-req, status: accepted) — missing: sw-integration-verification, unit-verification
  SR-5 (sw-req, status: accepted) — missing: sw-integration-verification, unit-verification
  SR-6 (sw-req, status: accepted) — missing: sw-integration-verification, unit-verification
  SR-7 (sw-req, status: accepted) — missing: sw-integration-verification, unit-verification
  SR-8 (sw-req, status: accepted) — missing: sw-integration-verification, unit-verification
  SR-9 (sw-req, status: accepted) — missing: sw-integration-verification, unit-verification
  SYS-1 (system-req, status: accepted) — missing: sys-integration-verification
  SYS-2 (system-req, status: accepted) — missing: sys-integration-verification
  SYS-3 (system-req, status: accepted) — missing: sys-integration-verification
  SYS-4 (system-req, status: accepted) — missing: sys-integration-verification
  SYS-5 (system-req, status: accepted) — missing: sys-integration-verification
  → run `rivet validate --explain SR-1` to see which link type and source types satisfy a gap

Result: PASS (200 warnings)
Schemas: common@0.4.0 (embedded), dev@0.6.0 (embedded), research@0.1.0 (embedded), research-ext@0.1.0 (on-disk), dev-ext@0.1.0 (on-disk), safety-case@0.1.0 (embedded), aspice@0.3.0 (embedded)
base — `rivet validate` result (for comparison)
  SR-11 (sw-req, status: accepted) — missing: sw-integration-verification, unit-verification
  SR-12 (sw-req, status: accepted) — missing: sw-integration-verification, unit-verification
  SR-13 (sw-req, status: accepted) — missing: sw-integration-verification, unit-verification
  SR-2 (sw-req, status: accepted) — missing: sw-integration-verification, unit-verification
  SR-3 (sw-req, status: accepted) — missing: sw-integration-verification, unit-verification
  SR-4 (sw-req, status: accepted) — missing: sw-integration-verification, unit-verification
  SR-5 (sw-req, status: accepted) — missing: sw-integration-verification, unit-verification
  SR-6 (sw-req, status: accepted) — missing: sw-integration-verification, unit-verification
  SR-7 (sw-req, status: accepted) — missing: sw-integration-verification, unit-verification
  SR-8 (sw-req, status: accepted) — missing: sw-integration-verification, unit-verification
  SR-9 (sw-req, status: accepted) — missing: sw-integration-verification, unit-verification
  SYS-1 (system-req, status: accepted) — missing: sys-integration-verification
  SYS-2 (system-req, status: accepted) — missing: sys-integration-verification
  SYS-3 (system-req, status: accepted) — missing: sys-integration-verification
  SYS-4 (system-req, status: accepted) — missing: sys-integration-verification
  SYS-5 (system-req, status: accepted) — missing: sys-integration-verification
  → run `rivet validate --explain SR-1` to see which link type and source types satisfy a gap

Result: PASS (199 warnings)
Schemas: common@0.4.0 (embedded), dev@0.6.0 (embedded), research@0.1.0 (embedded), research-ext@0.1.0 (on-disk), dev-ext@0.1.0 (on-disk), safety-case@0.1.0 (embedded), aspice@0.3.0 (embedded)

Artifact stats

base head
Total artifacts 273 274
full stats — head
Artifact summary:
  academic-reference               24
  competitive-analysis             11
  design-decision                  23
  feature                         103
  market-finding                    7
  requirement                      21
  safety-context                    3
  safety-goal                       5
  safety-justification              4
  safety-solution                   6
  safety-strategy                   1
  stakeholder-req                   3
  sw-req                           13
  sw-verification                  13
  sys-verification                  5
  system-req                        5
  technology-evaluation            12
  verification                     15
  TOTAL                           274

Orphan artifacts (no links): 12
  CA-001
  CA-002
  CA-003
  CA-004
  CA-005
  CA-006
  CA-007
  CA-008
  CA-009
  CA-010
  CA-011
  FEAT-078

Diagnostics: 0 error(s), 200 warning(s), 11 info(s)

Diff (base → head)

+ FEAT-103  v3.4 — The MC/DC evidence and the shipped artifact share a compiler, and the skew that remains is tracked (scry#215)

1 added, 0 removed, 0 modified, 273 unchanged

~ NEW    WARN: [FEAT-103] prose mentions 'FEAT-072' but no typed link to it; add a link in `links:` or remove the mention
0 new errors, 0 resolved errors, 1 new warnings, 0 resolved warnings

AADL model — head

spar/scry.aadl: OK

Posted by the rivet-delta workflow. Informational only — does not gate the PR.

@avrabe
avrabe merged commit 03682ae into main Nov 17, 2026
13 checks passed
@avrabe
avrabe deleted the feat215-evidence-substrate-rebased branch November 17, 2026 19:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant