Repository navigation
scry#215: the MC/DC evidence and the shipped artifact now share a compiler - #242
Merged
Merged
Conversation
…piler
The MC/DC gate, the released mcdc-evidence tarball, the Pages dashboard
and the FEAT-072 release delta all measure scry_mcdc.wasm — a wasip1 core
module built by Cargo — while the thing we ship and sign is scry.wasm, a
wasip2 component built by Bazel. The wasip1/DWARF choice is deliberate
and documented. Two other skews were not.
BOTH MEASURED ON SHIPPED ARTIFACTS, because a build config is a
hypothesis about an artifact and not evidence about it.
(1) COMPILER. v3.3.0's signed scry-3.3.0-wasm32-wasip2.wasm records
$ wasm-tools metadata show scry-3.3.0-wasm32-wasip2.wasm
name scry_analyzer_component_wasm_lib_release_wasm_base.wasm
processed-by rustc [1.91.1 (ed61e7d7e 2025-11-07)]
confirming MODULE.bazel. The MC/DC subject had NO rust-toolchain.toml
anywhere in the repo and was built by whatever @stable resolved to that
day, so the skew widened on its own and nothing recorded which compiler
produced any given number.
(2) TOOL. v3.3.0's evidence manifest records witness_version "0.30.0",
while the CI gate installs rev 6ff96e75 — which git ls-remote resolves to
v0.29.0^{}, NOT v0.30.0. The floors were proved by one tool version and
the released evidence produced by another. Upstream is at v0.32.0.
#215'S PROPOSED FIX IS NOT POSSIBLE AS WRITTEN, and the measurement is
why: wasmtime 48.0.8 declares rust-version = 1.95.0, so the workspace
cannot be pinned to 1.91.1. It IS possible for the MC/DC subject alone,
because crates/scry-mcdc is its OWN workspace, depends only on
scry-sai-core, has zero wasmtime in its graph, and mcdc-gate.sh does
`cd "$(dirname "$0")"` before building — so a toolchain file there
governs exactly that build and nothing else.
THE SKEW WAS WORTH 13 CONDITIONS. Measured with witness held constant at
the gate's own pinned rev, changing ONLY the compiler:
rustc 1.96.1 -> 1.91.1
decisions_total 1626 -> 1677 (+51)
conditions_proved 395 -> 382 (-13)
decisions_full_mcdc 18 -> 16 (-2)
Both clear the floors (155 / 3) with margin, so the gate stays green. But
the evidence was measurably compiler-dependent while one side floated.
THE FILE IS NOT THE COMPILER. Comparing rust-toolchain.toml against
MODULE.bazel is a claim about what WILL compile — the same
config-as-evidence error this change exists to fix, one level up. rustup
precedence lets RUSTUP_TOOLCHAIN or a directory override beat a toolchain
file, and a green gate under the wrong rustc looks identical to a green
one. So the guard step also asserts the ACTIVE rustc where the build
happens and prints it, which puts the evidence's substrate in the CI log.
Negative-controlled, and the hazard is real rather than theoretical:
RUSTUP_TOOLCHAIN=1.98.1 DOES beat the file (measured: rustc 1.98.1) and
the assertion catches it.
A CONSEQUENCE THE MEASUREMENT EXPOSED: the FEAT-072 release delta builds
the previous tag in a git worktree, which predates the toolchain file —
so the old side would compile under @stable and the new side pinned, and
a ~51-decision compiler artifact would be reported as source change,
which is the one thing that delta exists to isolate. The delta step now
copies the toolchain file into the worktree first, and says so in its
status line.
TWO MORE REFERENCES WERE FOUND BY WRITING THE CHECK. The witness rev also
appears bare in two actions/cache keys:
key: witness-bins-${{ runner.os }}-6ff96e75a1b495aa165f1ee91abda3049fffe628
Bump an install rev and leave a cache key stale, and the cache serves
binaries the install step no longer names — the gate would report
coverage from a witness it does not pin. 4 references found by the first
version of the check, 6 by this one.
WIRED INTO A REQUIRED JOB, which needed a design decision. A check that
is intentionally red cannot be required, and one that is not wired is
gate-potency's defined-but-not-enforced hole. So EXPECTED_SKEWS reports a
known-open skew instead of failing it — and the allowlist is
SELF-CLEANING: an entry whose skew is no longer present FAILS and names
itself, so it cannot outlive the defect. Losing the subject entirely is
never allowlistable, even by its own key. The step sits in
`MC/DC (witness)`, no continue-on-error, no job-level if.
MUTATION-CHECKED ON THE FINAL FILE: 9 mutations (M1-M9), 9 killed,
unmutated control green. An earlier run of 7 did not cover the bare-SHA
scan, which was added afterwards; this is the full suite re-run on what
is committed.
THREE survived a first pass, and every one was a gap in the TABLE rather
than a defect in the code — the same error three times:
* the SHA-format regex was never load-bearing, because every tag case
also carried two DIFFERENT refs, so len(distinct) > 1 blocked them
regardless. Fixed with a case where every ref is an IDENTICAL tag.
* the lost-subject mutation was never tested by its own allowlist key;
the case allowlisted `witness-version`, not `no-refs`.
* dropping the bare-SHA scan's `witness`-line guard survived because no
unrelated 40-hex string exists in the workflows TODAY. scry#232 is
pinning every GitHub Action to a commit SHA, so an unguarded scan
would have started failing CI for no reason the day it merged. Fixed
by running the PARSER against synthetic workflow text, which none of
the other cases did — they all feed `verdict` pre-built tuples.
Each case exercised the OVERLAP of two defences rather than the gap
between them, which is also what survived the first attempt at scry#208.
CORRECTION CARRIED IN THE ARTIFACT: the first draft of this change called
the mcdc-evidence tarball "signed". IT IS NOT. Measured on v3.3.0 — of 20
primary assets, 18 are listed in a cosign-signed SHA256SUMS.txt and 18
also carry individual .sig/.pem pairs; scry-v3.3.0-mcdc-evidence.tar.gz
is in NEITHER and is the ONLY primary asset with no integrity protection.
The error came from filtering the asset list on `mcdc|evidence`, seeing
compliance-evidence.tar.gz return with its .pem and .sig, and carrying
the signing across to the neighbouring bundle. scry#215's own body says
"the evidence bundle we sign and ship", so it was already in circulation;
it also reached witness#360 and has been corrected there.
That makes it a REQUIREMENT VIOLATION, not a gap: REQ-010 — the
requirement this traces to — asks for the evidence "surfaced as a SIGNED
truth-table report". It is surfaced and it is not signed. Filed
separately rather than fixed here; adding an asset to the signing set is
a release-path change and this branch already touches release.yml once.
THE WITNESS SKEW IS LEFT OPEN DELIBERATELY. The v0.30.0 side is
internally coherent — the cargo install --tag and the compliance@v0.30.0
action match each other — so converging either changes the tool that
proved the floors or alters an unsimulated release path. Tracked by the
allowlist, which cannot rot.
VERIFIED (all local; no CI has run on this SHA):
rustc --version in crates/scry-mcdc -> 1.91.1 (ed61e7d7e 2025-11-07),
byte-identical to the shipped component's processed-by string
mcdc-gate.sh with witness 0.29.0 -> exit 0, 382 >= 155, 16 >= 3
check-evidence-substrate.py --self-test -> OK (13 cases + a parser
probe, both outcomes exercised)
check-evidence-substrate.py -> exit 0, 1 known-open skew, nothing stale
check-gate-coverage.py -> OK, 13 crates in both gates (ci.yml edited)
ci.yml and release.yml both still parse as YAML
rivet validate -> PASS
Refs: FEAT-103
Rebased onto main 75fd030 from #236 (de09aa9), whose session ended on
2026-11-08. Two changes in the rebase: the ci.yml conflict keeps main's
SHA-pinned rust-cache, and the artifact is renumbered FEAT-102 ->
FEAT-103, because #240 took FEAT-102 first. Re-run on the rebased tree:
check-evidence-substrate --self-test OK and check exit 0 (1 known-open
skew, owned by #215); gate-coverage, required-checks, fork-guard and
claim-check all pass; rivet validate PASS.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KkNzkNYzPh7366DkNijeNc
Claude-Session: https://claude.ai/code/session_01KEjmjcVaTucsMZy5Fubas7
📐 rivet artifact deltaPR: #242 Base SHA: Validationhead — `rivet validate` resultbase — `rivet validate` result (for comparison)Artifact stats
full stats — headDiff (base → head)AADL model — headPosted by the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #236: the same commit rebased onto main
75fd030. #236 conflicts with main and its session ended on 2026-11-08, so this is a new branch rather than a push onto #236's.Two changes in the rebase:
ci.ymlconflict: main pinnedSwatinem/rust-cacheto a SHA (ci: pin all GitHub Actions to commit SHAs (not tags) #232) where scry#215: the MC/DC evidence and the shipped artifact now share a compiler #236 added the substrate guard step; both are kept.Refs: FEAT-103.What it does (full rationale in the commit message):
crates/scry-mcdc/rust-toolchain.tomlpins the MC/DC subject to rustc 1.91.1. That is the compiler recorded in v3.3.0's signedscry-3.3.0-wasm32-wasip2.wasm. The compiler skew alone was measured at 13 proved conditions (395 → 382).tools/check-evidence-substrate.pyruns in the requiredMC/DC (witness)job. It asserts the toolchain file against MODULE.bazel and the activerustcagainst the file. The witness-version skew (rev 6ff96e75 = v0.29.0 in the gate, v0.30.0 for released evidence) stays open, as a self-cleaning allowlist entry owned by The MC/DC coverage evidence measures a different binary than the one we ship — different compiler, profile, and target #215.Re-verified on the rebased tree:
check-evidence-substrate.py --self-testcheck-evidence-substrate.pycheck-gate-coverage.pycheck-required-checks.py --against-filecheck_selfhosted_fork_guard.pyclaim-check.pyrivet validateNot re-run here: the 9-mutation suite from #236's commit message, and the full MC/DC gate under 1.91.1. The MC/DC job in this PR's CI covers the second.
Refs #215. Related, still open: #235 (witness version), #237 (unsigned evidence asset).
🤖 Generated with Claude Code
https://claude.ai/code/session_01KEjmjcVaTucsMZy5Fubas7