Skip to content

Make the proof and mutation gates say what they prove - #371

Open
jserv wants to merge 6 commits into
mainfrom
verify
Open

Make the proof and mutation gates say what they prove#371
jserv wants to merge 6 commits into
mainfrom
verify

Conversation

@jserv

@jserv jserv commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Three commits, each standing on its own.

Emit verify profiles from the recipe's variables

The frama-c MCP server can run the proofs make verify runs, but only if it loads the same program. Every field that decides that had to be restated by hand, and a hand-written function set is the drift a proof gate cannot afford: it produces a confident answer about a program the build system does not prove.

print-verify-profiles emits that JSON from the variables the verify-<name> recipe consumes. Include directories and force-includes move into shared variables so neither list is spelled twice; they were duplicated at first, under a comment claiming they could not drift.

FRAMAC_ISYSTEM_DIRS is used unquoted rather than through patsubst: its value carries a shell substitution containing a space, so a patsubst over it splits mid-command. That mangling still proves green on any target whose sources need no libc, which is most of them.

Contract the two proved functions that had none

elf_place_segment and futex_uaddr_is_aligned sat in their targets' -wp-fct lists with no ACSL contract, so WP proved only that their bodies commit no runtime error. Nothing there can reject a broken source.

elf_place_segment now states the property its caller depends on: an accepted placement never overlaps the runtime infra reserve, which holds the page-table pool, the shim code and the EL1-only shim data.

Both raise the obligation count, so both floors move with them. MIN_GOALS is a tripwire against an emptied body or a dropped contract, which prove 0 of 0, and it only means that while it sits at the target's baseline. Left alone it would have tolerated verify-elf losing 15 obligations and verify-futexdeadline losing 2, silently.

Make the mutation gate say what it actually proves

A function proved by two targets could only be mutated in the one whose VERIFY_<T>_SRC named its file. hex_nibble lives in src/utils.h and is proved by both verify-elf and verify-rsp precisely because assuming it once was a past mistake, and neither could mutate it. The runner now shadows a mutant header through MUTANT_INCDIR.

The staged path mirrors the original's path under src/ and MUTANT_INCDIR is the staging root, not the copy's parent. Deriving it as the parent got src/utils.h right by luck and every nested header wrong: "proved/netlink.h" fell through -Isrc to the real header, so the run proved unmutated code while reporting a mutation nobody caught. A #error probe now confirms each shadow is reached; the unmutated baseline structurally cannot, since it stages a byte-identical copy.

Mutation runs pass -wp-cache none, because WP stores a timeout as a verdict and replays it, which would be a catch with no prover run behind it.

One thing this makes visible

The gate reported "121 caught", which read as 121 proofs demonstrably rejecting their broken sources. It does not mean that. All 121 are caught by exhausting the prover, not by refutation - [Unknown]/[Failed] never appear in this tree, and raising the budget eightfold to 240s on a quiet host changes nothing.

They still count, and the reasoning is recorded rather than assumed: what makes exhaustion evidence is the baseline proving every goal while each mutant exhausts on the one its own function owns. A merely-hard goal would exhaust in the baseline too, and a failing baseline is fatal rather than scored.


Summary by cubic

Makes the Frama-C proof gate and the mutation gate prove what they claim: verify profiles are emitted from the same variables the recipes consume, proved functions gain the contracts their callers rely on, included headers can finally be mutated, and mutation verdicts count only goals the mutated function owns.

Proof gate

  • make print-verify-profiles emits the frama-c-mcp JSON from the same variables verify-<name> consumes, so the server can no longer prove a different program; include dirs and force-includes moved into shared FRAMAC_INCLUDE_DIRS / FRAMAC_FORCE_INCLUDES instead of being spelled twice.
  • elf_place_segment now states an accepted placement never overlaps the runtime infra reserve, and futex_uaddr_is_aligned states its result in the bit domain; elf_place_segment's assigns clause was dropped after proving only against log_impl's WP-generated frame — check-wp-result.py now reports that frame, matching both wordings Frama-C uses for a missing spec — so VERIFY_ELF_MIN_GOALS went 156→172 with the new contract and back to 159 once the false frame fell, and VERIFY_FUTEXDEADLINE_MIN_GOALS went 128→130.
  • check-proof-targets now runs on make check, and BUILD_FLAVOR_GOALS skips print-% goals so that scanner's sub-make cannot wipe a sanitizer build's objects.

Mutation gate

  • Header mutants are staged under MUTANT_INCDIR (the staging root, not the copy's parent) so they shadow the real files, and a #error probe confirms each shadow is reached — run serially, since a concurrent sibling could spoil its exit-status answer — because the byte-identical baseline cannot detect a fall-through.
  • Mutation runs pass -wp-cache none because WP otherwise replays a stored timeout as a catch; baselines that fail only while the pool is busy are retried serially, and their targets' mutations then run serially too.
  • Verdicts report refutation and resource exhaustion separately — every catch here is [Timeout] — and MUTANT_ESCALATE=<n>, off by default, re-runs resource verdicts at a larger budget, reporting MISSED for a mutation that merely turned an easy goal into a slow one. Every open goal must belong to the mutated function, decided by the longest matching name in the target's own proved set and checked by a self-test, with caller obligations naming the callee qualifying for both; coverage is counted by function and by (target, function) pair.

Written for commit b6ff88c. Summary will update on new commits.

Review in cubic

cubic-dev-ai[bot]

This comment was marked as resolved.

cubic-dev-ai[bot]

This comment was marked as resolved.

cubic-dev-ai[bot]

This comment was marked as resolved.

cubic-dev-ai[bot]

This comment was marked as resolved.

cubic-dev-ai[bot]

This comment was marked as resolved.

cubic-dev-ai[bot]

This comment was marked as resolved.

cubic-dev-ai[bot]

This comment was marked as resolved.

The frama-c MCP server can run the proofs make verify runs, but only if
it loads the same program. Every field that decides that (sources,
functions, memory model, machdep, include paths, force-includes,
defines, provers, timeout) had to be restated by hand on the server
side, and a hand-written function set is the drift a proof gate cannot
afford: it produces a confident answer about a program the build system
does not prove.

print-verify-profiles emits that JSON from the variables the
verify-<name> recipe consumes, so the two cannot disagree. The include
directories and force-includes move into FRAMAC_INCLUDE_DIRS and
FRAMAC_FORCE_INCLUDES, named once and turned into -I and -include flags
by FRAMAC_CPP_ARGS, because spelling either list twice is the drift this
arrangement exists to prevent. They were duplicated at first, under a
comment claiming they could not.

FRAMAC_ISYSTEM_DIRS is used unquoted rather than through patsubst. Its
value carries a shell substitution containing a space, so a patsubst
over it splits mid-command and emits
"-isystem $(frama-c -isystem -print-share-path)/libc". That mangling
still proves green on any target whose sources need no libc, which is
most of them.

nostdinc and the isystem path are emitted rather than omitted. Without
them the real macOS headers win over Frama-C's modeled libc and some
files parse as a different program: src/syscall/sys.c is one, whose
rusage _Static_assert only holds against the modeled header.

BUILD_FLAVOR_GOALS also skips print-% goals. check-proof-targets.py
shells out to make print-verify-targets on every make check, and a
sub-make evaluates the build-flavor guard while it reads the makefiles.
Without the skip, running that scanner beside a sanitizer build wipes
the build's objects from under it.
Frama-C words a missing spec two ways, and only one of them was being
read. "Neither code nor explicit exits and terminates" leaves a
hand-written assigns in force and completes the rest; "Neither code nor
specification" generates the frame as well. The second is the stronger
assumption and was the one the pattern did not match, so the "assumed
(spec generated, body not analyzed)" line named the weaker case and hid
the stronger one.

log_impl arrives through that second wording. Every proof of a function
that logs therefore rested on an invented assigns \nothing, and no
report anywhere said so; verify-elf disclosed memcpy and pread while the
frame it had actually assumed went unnamed.

Across the 21 targets log_impl is the only name this surfaces. The other
four already matched, so nothing that was reported stops being reported.
elf_place_segment and futex_uaddr_is_aligned sat in their targets'
-wp-fct lists with no ACSL contract, so WP proved only that their bodies
commit no runtime error. Nothing there can reject a broken source: widen
an alignment mask or move a bounds comparison and every obligation still
discharges. A proof that cannot reject a broken source proves nothing,
and these two had been passing that way for a while.

elf_place_segment now states the property its caller depends on: an
accepted placement never overlaps the runtime infra reserve. That
reserve holds the page-table pool, the shim code and the EL1-only shim
data, so a segment the loader accepts on top of it is a guest-controlled
write into elfuse's own control structures.

No assigns clause, and that is the honest answer rather than an
omission. The function logs on both rejection paths, log.c locks a mutex
and writes stderr, and WP checks a frame against log_impl's generated
spec, which is assigns \nothing. A frame stated here would discharge
against that assumption and still be false, which serves a caller worse
than no frame at all. It is also the only place the question arises: of
the six contracted functions in elf.c only this one logs.

Its \separated precondition discharges and is kept, because stating the
non-aliasing beats leaving it implicit in the caveat model's
assumptions. \valid on the two out-parameters is deliberately absent and
that was measured, not assumed: with both stated, the matching
obligations in elf_check_placement time out at the 30s budget on a host
at 0.3 runnable threads per CPU with the WP cache defeated, so it is the
model and not the machine. Caveat already assumes a formal pointer
parameter is valid, which is why the body proves without them.

futex_uaddr_is_aligned states its result in the bit domain rather than
as uaddr % 4 == 0. The readable form does not discharge: bridging modulo
and bitmask on a 64-bit value is what exhausts the prover, not the
property.

Both floors move with the obligations they generate: elf 156 to 159,
futexdeadline 128 to 130. MIN_GOALS is a tripwire against an emptied
body or a dropped contract, which prove 0 of 0, and it only means that
while it sits at the target's baseline.
Three things it could not do, and one it claimed that was not true.

A function proved by two targets could only be mutated in the one whose
VERIFY_<T>_SRC named its file. hex_nibble lives in src/utils.h and is
proved by verify-elf and verify-rsp precisely because assuming it once
was a past mistake, and neither could mutate it. The runner now stages a
mutant header below its own directory and hands that directory to
MUTANT_INCDIR, where it precedes src/ in the include search. What a
target may mutate widens from its source to its source plus the headers
in VERIFY_<T>_SCAN, which is already its declared input closure.

The staged path mirrors the original's path under src/ and MUTANT_INCDIR
is the staging root, not the copy's parent. Deriving it as the parent
got src/utils.h right by luck and every nested header wrong:
"proved/netlink.h" resolved to <parent>/proved/netlink.h, missed, fell
through -Isrc to the real header, and the run proved unmutated code
while reporting a mutation nobody caught.

The unmutated baseline cannot catch that, though a comment here claimed
it could. The baseline stages a copy identical to the file it shadows,
so whether the preprocessor opens the shadow or falls through, the
program proved is the same and the run passes either way. A probe
staging a copy that carries #error does catch it, and it reads the log
rather than make's stdout, since the recipe redirects Frama-C there and
a non-zero exit alone is equally what a broken override gives.

The probe runs serially, unlike everything else here. It answers a
yes/no question from make's exit status, and that status is the one
thing a concurrent sibling can spoil: alongside the baseline pool it
reported "the target proved anyway" for a staged header whose first line
is #error, while the identical invocation by hand exited non-zero with
the probe named twice in its log. A control that can report the opposite
of the truth is worse than no control.

A mutation is scored caught when a goal comes back open, and open has
two causes: the prover reached a conclusion the mutant cannot satisfy,
or it ran out of budget. Only the first is refutation. Both are now
reported apart, and both still count, because measurement says the first
never happens here: across every mutation log the tag is [Timeout],
never [Unknown] or [Failed], and raising the budget eightfold to 240s on
a host at 0.3 to 0.6 runnable threads per CPU left all four
futexdeadline mutations exhausting exactly as at 30s. Refusing to count
exhaustion would not make the gate stricter, it would make caught
unreachable. What separates a broken contract from a hard one is the
baseline: it proves every goal, and each mutant exhausts on the one its
own function owns.

Mutation runs pass -wp-cache none. WP's cache defaults to update and
stores a timeout as a verdict just like a conclusion, so a replayed
timeout would be a catch with no prover run behind it. make verify keeps
its cache, which is what makes a re-prove cheap.
A mutation counted as caught when the target stopped proving, and
nothing checked which goal stopped proving. For 120 of the 121 entries
that was safe by construction: FCT_ARG narrows the run to the mutated
function, so any open goal is one of its own. The narrowing is dropped
for a mutation that edits a contract, because a contract moves its
callers' goals too and the caller is named nowhere in the entry. There
the target proves everything, and an unrelated goal getting slower
scored exactly like the proof rejecting a broken source.

Every open goal must now belong to the mutated function, not merely one
of them: a run that broke its own goal and also slowed an unrelated one
would otherwise pass with the stray invisible.

Deciding ownership from the goal name is the whole of it, and the name
gives less to work with than it looks. WP composes it as
<model>_<function>_<clause>, so substring containment hands a shorter
name every goal of a longer one, and this tree proves three such pairs:
timespec_valid beside timespec_valid_capped, gva_chunk_clamp beside
gva_chunk_clamp_args_ok, gva_leaf_target beside gva_leaf_target_args_ok.
Matching the name as a whole component before its clause fixes the
prefix direction and not the tail direction, where bar would own
foo_bar's goals. There is nothing fixed to anchor the front against
either, since the prefix is the memory model's spelling: bytes_ for
netlinkwalk, typed_ for most, typed_caveat_ for elf.

So the owner is the longest of the target's own proved functions that
names a component, which is the only one that can be the function whose
clause the goal states. Both collision directions fall out of that, and
proved_functions() already has the set.

A call obligation is the exception, and it needs both halves. WP names
those <caller>_call_<callee>_<clause> and both sides own it: it is the
caller's obligation, and it exists because of the callee's contract.
Longest-candidate therefore runs over the caller side alone, because the
callee is routinely the longer name. nl_put_attr calls
netlink_attr_extent and verify-netlinkwalk proves both, so measuring
across the whole goal gave the caller's own obligation to its callee and
reported ELSEWHERE for a mutation the proof does reject.

The self-test carries the shapes that got this wrong. The rule is pure
string reasoning, it decides caught against ELSEWHERE, and a table run
costs prover time while exercising only the shapes that happen to occur.
Nine cases cost nothing and pin the ones that do not.

verify-mutants unchanged: timespec 9, gva 11, elf 12, sockaddr 4,
netlinkwalk 3.
Every catch this gate produces is an exhausted prover, never a
refutation. That is measured and it is not going to change: across every
mutation log the tag is [Timeout], and raising the budget eightfold left
all four futexdeadline mutations exhausting exactly as at 30s. Refusing
to count exhaustion would not make the gate stricter, it would make
"caught" unreachable.

What it does leave is a residual gap the gate stated and could not test.
A mutation that turns an easy true goal into a hard true one exhausts at
the short budget and would discharge at a long one, and no verdict tag
tells it from a rejection. Such a mutation is one the proof does not
reject, and it was scoring as caught.

--escalate SECONDS re-runs each resource verdict at a larger budget and
reports MISSED for any mutation that then proves. It escalates whichever
run produced the verdict, narrowed or full, since repeating the other
one would escalate a run that had nothing to say.

Reading that second run is one function shared with the first, not a
second reading beside it. A longer budget changes how long the prover
had and nothing else, so an infra marker or a goal owned by another
function means at 240s what it meant at 30s; reading the escalated run
for refutation alone would let a crashed prover, an unparsable mutant, a
broken override and a stray goal all come back RESOURCE, which passes.
The escalated run may still say what the first could not, which is why
it runs: MISSED when the mutant now proves, and caught with the budget
named when more of it turns exhaustion into refutation.

Eight cases pin those readings, written to discriminate rather than to
pass. A crashed run prints whatever it reached before dying, so a case
pairing an infra marker with nothing else cannot tell a lost infra check
from the fallback that returns INFRA anyway. Each carries a resource or
refutation tag alongside the thing that must outrank it, and removing
the infra check, the ownership check or the resource reading fails the
self-test.

Off by default. It costs the escalated budget on precisely the goal that
already ran out of the short one, once per mutation, and every mutation
in the table is a resource verdict.

So the default summary now says which budget it ran at and states
plainly that it did not measure this, rather than leaving a reader to
infer it from a count. The eightfold measurement above is no longer a
remembered experiment either:

  make verify-mutants MUTANT_TARGET=futexdeadline MUTANT_ESCALATE=240
  4 mutations, 4 caught (4 resource verdicts)
    4 caught by exhausting the prover rather than by refutation, at 240s
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant