Conversation
The frama-c MCP server can run the proofs make verify runs, but only if it loads the same program. Every field that decides that (sources, functions, memory model, machdep, include paths, force-includes, defines, provers, timeout) had to be restated by hand on the server side, and a hand-written function set is the drift a proof gate cannot afford: it produces a confident answer about a program the build system does not prove. print-verify-profiles emits that JSON from the variables the verify-<name> recipe consumes, so the two cannot disagree. The include directories and force-includes move into FRAMAC_INCLUDE_DIRS and FRAMAC_FORCE_INCLUDES, named once and turned into -I and -include flags by FRAMAC_CPP_ARGS, because spelling either list twice is the drift this arrangement exists to prevent. They were duplicated at first, under a comment claiming they could not. FRAMAC_ISYSTEM_DIRS is used unquoted rather than through patsubst. Its value carries a shell substitution containing a space, so a patsubst over it splits mid-command and emits "-isystem $(frama-c -isystem -print-share-path)/libc". That mangling still proves green on any target whose sources need no libc, which is most of them. nostdinc and the isystem path are emitted rather than omitted. Without them the real macOS headers win over Frama-C's modeled libc and some files parse as a different program: src/syscall/sys.c is one, whose rusage _Static_assert only holds against the modeled header. BUILD_FLAVOR_GOALS also skips print-% goals. check-proof-targets.py shells out to make print-verify-targets on every make check, and a sub-make evaluates the build-flavor guard while it reads the makefiles. Without the skip, running that scanner beside a sanitizer build wipes the build's objects from under it.
Frama-C words a missing spec two ways, and only one of them was being read. "Neither code nor explicit exits and terminates" leaves a hand-written assigns in force and completes the rest; "Neither code nor specification" generates the frame as well. The second is the stronger assumption and was the one the pattern did not match, so the "assumed (spec generated, body not analyzed)" line named the weaker case and hid the stronger one. log_impl arrives through that second wording. Every proof of a function that logs therefore rested on an invented assigns \nothing, and no report anywhere said so; verify-elf disclosed memcpy and pread while the frame it had actually assumed went unnamed. Across the 21 targets log_impl is the only name this surfaces. The other four already matched, so nothing that was reported stops being reported.
elf_place_segment and futex_uaddr_is_aligned sat in their targets' -wp-fct lists with no ACSL contract, so WP proved only that their bodies commit no runtime error. Nothing there can reject a broken source: widen an alignment mask or move a bounds comparison and every obligation still discharges. A proof that cannot reject a broken source proves nothing, and these two had been passing that way for a while. elf_place_segment now states the property its caller depends on: an accepted placement never overlaps the runtime infra reserve. That reserve holds the page-table pool, the shim code and the EL1-only shim data, so a segment the loader accepts on top of it is a guest-controlled write into elfuse's own control structures. No assigns clause, and that is the honest answer rather than an omission. The function logs on both rejection paths, log.c locks a mutex and writes stderr, and WP checks a frame against log_impl's generated spec, which is assigns \nothing. A frame stated here would discharge against that assumption and still be false, which serves a caller worse than no frame at all. It is also the only place the question arises: of the six contracted functions in elf.c only this one logs. Its \separated precondition discharges and is kept, because stating the non-aliasing beats leaving it implicit in the caveat model's assumptions. \valid on the two out-parameters is deliberately absent and that was measured, not assumed: with both stated, the matching obligations in elf_check_placement time out at the 30s budget on a host at 0.3 runnable threads per CPU with the WP cache defeated, so it is the model and not the machine. Caveat already assumes a formal pointer parameter is valid, which is why the body proves without them. futex_uaddr_is_aligned states its result in the bit domain rather than as uaddr % 4 == 0. The readable form does not discharge: bridging modulo and bitmask on a 64-bit value is what exhausts the prover, not the property. Both floors move with the obligations they generate: elf 156 to 159, futexdeadline 128 to 130. MIN_GOALS is a tripwire against an emptied body or a dropped contract, which prove 0 of 0, and it only means that while it sits at the target's baseline.
Three things it could not do, and one it claimed that was not true. A function proved by two targets could only be mutated in the one whose VERIFY_<T>_SRC named its file. hex_nibble lives in src/utils.h and is proved by verify-elf and verify-rsp precisely because assuming it once was a past mistake, and neither could mutate it. The runner now stages a mutant header below its own directory and hands that directory to MUTANT_INCDIR, where it precedes src/ in the include search. What a target may mutate widens from its source to its source plus the headers in VERIFY_<T>_SCAN, which is already its declared input closure. The staged path mirrors the original's path under src/ and MUTANT_INCDIR is the staging root, not the copy's parent. Deriving it as the parent got src/utils.h right by luck and every nested header wrong: "proved/netlink.h" resolved to <parent>/proved/netlink.h, missed, fell through -Isrc to the real header, and the run proved unmutated code while reporting a mutation nobody caught. The unmutated baseline cannot catch that, though a comment here claimed it could. The baseline stages a copy identical to the file it shadows, so whether the preprocessor opens the shadow or falls through, the program proved is the same and the run passes either way. A probe staging a copy that carries #error does catch it, and it reads the log rather than make's stdout, since the recipe redirects Frama-C there and a non-zero exit alone is equally what a broken override gives. The probe runs serially, unlike everything else here. It answers a yes/no question from make's exit status, and that status is the one thing a concurrent sibling can spoil: alongside the baseline pool it reported "the target proved anyway" for a staged header whose first line is #error, while the identical invocation by hand exited non-zero with the probe named twice in its log. A control that can report the opposite of the truth is worse than no control. A mutation is scored caught when a goal comes back open, and open has two causes: the prover reached a conclusion the mutant cannot satisfy, or it ran out of budget. Only the first is refutation. Both are now reported apart, and both still count, because measurement says the first never happens here: across every mutation log the tag is [Timeout], never [Unknown] or [Failed], and raising the budget eightfold to 240s on a host at 0.3 to 0.6 runnable threads per CPU left all four futexdeadline mutations exhausting exactly as at 30s. Refusing to count exhaustion would not make the gate stricter, it would make caught unreachable. What separates a broken contract from a hard one is the baseline: it proves every goal, and each mutant exhausts on the one its own function owns. Mutation runs pass -wp-cache none. WP's cache defaults to update and stores a timeout as a verdict just like a conclusion, so a replayed timeout would be a catch with no prover run behind it. make verify keeps its cache, which is what makes a re-prove cheap.
A mutation counted as caught when the target stopped proving, and nothing checked which goal stopped proving. For 120 of the 121 entries that was safe by construction: FCT_ARG narrows the run to the mutated function, so any open goal is one of its own. The narrowing is dropped for a mutation that edits a contract, because a contract moves its callers' goals too and the caller is named nowhere in the entry. There the target proves everything, and an unrelated goal getting slower scored exactly like the proof rejecting a broken source. Every open goal must now belong to the mutated function, not merely one of them: a run that broke its own goal and also slowed an unrelated one would otherwise pass with the stray invisible. Deciding ownership from the goal name is the whole of it, and the name gives less to work with than it looks. WP composes it as <model>_<function>_<clause>, so substring containment hands a shorter name every goal of a longer one, and this tree proves three such pairs: timespec_valid beside timespec_valid_capped, gva_chunk_clamp beside gva_chunk_clamp_args_ok, gva_leaf_target beside gva_leaf_target_args_ok. Matching the name as a whole component before its clause fixes the prefix direction and not the tail direction, where bar would own foo_bar's goals. There is nothing fixed to anchor the front against either, since the prefix is the memory model's spelling: bytes_ for netlinkwalk, typed_ for most, typed_caveat_ for elf. So the owner is the longest of the target's own proved functions that names a component, which is the only one that can be the function whose clause the goal states. Both collision directions fall out of that, and proved_functions() already has the set. A call obligation is the exception, and it needs both halves. WP names those <caller>_call_<callee>_<clause> and both sides own it: it is the caller's obligation, and it exists because of the callee's contract. Longest-candidate therefore runs over the caller side alone, because the callee is routinely the longer name. nl_put_attr calls netlink_attr_extent and verify-netlinkwalk proves both, so measuring across the whole goal gave the caller's own obligation to its callee and reported ELSEWHERE for a mutation the proof does reject. The self-test carries the shapes that got this wrong. The rule is pure string reasoning, it decides caught against ELSEWHERE, and a table run costs prover time while exercising only the shapes that happen to occur. Nine cases cost nothing and pin the ones that do not. verify-mutants unchanged: timespec 9, gva 11, elf 12, sockaddr 4, netlinkwalk 3.
Every catch this gate produces is an exhausted prover, never a
refutation. That is measured and it is not going to change: across every
mutation log the tag is [Timeout], and raising the budget eightfold left
all four futexdeadline mutations exhausting exactly as at 30s. Refusing
to count exhaustion would not make the gate stricter, it would make
"caught" unreachable.
What it does leave is a residual gap the gate stated and could not test.
A mutation that turns an easy true goal into a hard true one exhausts at
the short budget and would discharge at a long one, and no verdict tag
tells it from a rejection. Such a mutation is one the proof does not
reject, and it was scoring as caught.
--escalate SECONDS re-runs each resource verdict at a larger budget and
reports MISSED for any mutation that then proves. It escalates whichever
run produced the verdict, narrowed or full, since repeating the other
one would escalate a run that had nothing to say.
Reading that second run is one function shared with the first, not a
second reading beside it. A longer budget changes how long the prover
had and nothing else, so an infra marker or a goal owned by another
function means at 240s what it meant at 30s; reading the escalated run
for refutation alone would let a crashed prover, an unparsable mutant, a
broken override and a stray goal all come back RESOURCE, which passes.
The escalated run may still say what the first could not, which is why
it runs: MISSED when the mutant now proves, and caught with the budget
named when more of it turns exhaustion into refutation.
Eight cases pin those readings, written to discriminate rather than to
pass. A crashed run prints whatever it reached before dying, so a case
pairing an infra marker with nothing else cannot tell a lost infra check
from the fallback that returns INFRA anyway. Each carries a resource or
refutation tag alongside the thing that must outrank it, and removing
the infra check, the ownership check or the resource reading fails the
self-test.
Off by default. It costs the escalated budget on precisely the goal that
already ran out of the short one, once per mutation, and every mutation
in the table is a resource verdict.
So the default summary now says which budget it ran at and states
plainly that it did not measure this, rather than leaving a reader to
infer it from a count. The eightfold measurement above is no longer a
remembered experiment either:
make verify-mutants MUTANT_TARGET=futexdeadline MUTANT_ESCALATE=240
4 mutations, 4 caught (4 resource verdicts)
4 caught by exhausting the prover rather than by refutation, at 240s
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three commits, each standing on its own.
Emit verify profiles from the recipe's variables
The frama-c MCP server can run the proofs
make verifyruns, but only if it loads the same program. Every field that decides that had to be restated by hand, and a hand-written function set is the drift a proof gate cannot afford: it produces a confident answer about a program the build system does not prove.print-verify-profilesemits that JSON from the variables theverify-<name>recipe consumes. Include directories and force-includes move into shared variables so neither list is spelled twice; they were duplicated at first, under a comment claiming they could not drift.FRAMAC_ISYSTEM_DIRSis used unquoted rather than throughpatsubst: its value carries a shell substitution containing a space, so apatsubstover it splits mid-command. That mangling still proves green on any target whose sources need no libc, which is most of them.Contract the two proved functions that had none
elf_place_segmentandfutex_uaddr_is_alignedsat in their targets'-wp-fctlists with no ACSL contract, so WP proved only that their bodies commit no runtime error. Nothing there can reject a broken source.elf_place_segmentnow states the property its caller depends on: an accepted placement never overlaps the runtime infra reserve, which holds the page-table pool, the shim code and the EL1-only shim data.Both raise the obligation count, so both floors move with them.
MIN_GOALSis a tripwire against an emptied body or a dropped contract, which prove 0 of 0, and it only means that while it sits at the target's baseline. Left alone it would have toleratedverify-elflosing 15 obligations andverify-futexdeadlinelosing 2, silently.Make the mutation gate say what it actually proves
A function proved by two targets could only be mutated in the one whose
VERIFY_<T>_SRCnamed its file.hex_nibblelives insrc/utils.hand is proved by bothverify-elfandverify-rspprecisely because assuming it once was a past mistake, and neither could mutate it. The runner now shadows a mutant header throughMUTANT_INCDIR.The staged path mirrors the original's path under
src/andMUTANT_INCDIRis the staging root, not the copy's parent. Deriving it as the parent gotsrc/utils.hright by luck and every nested header wrong:"proved/netlink.h"fell through-Isrcto the real header, so the run proved unmutated code while reporting a mutation nobody caught. A#errorprobe now confirms each shadow is reached; the unmutated baseline structurally cannot, since it stages a byte-identical copy.Mutation runs pass
-wp-cache none, because WP stores a timeout as a verdict and replays it, which would be a catch with no prover run behind it.One thing this makes visible
The gate reported "121 caught", which read as 121 proofs demonstrably rejecting their broken sources. It does not mean that. All 121 are caught by exhausting the prover, not by refutation -
[Unknown]/[Failed]never appear in this tree, and raising the budget eightfold to 240s on a quiet host changes nothing.They still count, and the reasoning is recorded rather than assumed: what makes exhaustion evidence is the baseline proving every goal while each mutant exhausts on the one its own function owns. A merely-hard goal would exhaust in the baseline too, and a failing baseline is fatal rather than scored.
Summary by cubic
Makes the Frama-C proof gate and the mutation gate prove what they claim: verify profiles are emitted from the same variables the recipes consume, proved functions gain the contracts their callers rely on, included headers can finally be mutated, and mutation verdicts count only goals the mutated function owns.
Proof gate
make print-verify-profilesemits the frama-c-mcp JSON from the same variablesverify-<name>consumes, so the server can no longer prove a different program; include dirs and force-includes moved into sharedFRAMAC_INCLUDE_DIRS/FRAMAC_FORCE_INCLUDESinstead of being spelled twice.elf_place_segmentnow states an accepted placement never overlaps the runtime infra reserve, andfutex_uaddr_is_alignedstates its result in the bit domain;elf_place_segment'sassignsclause was dropped after proving only againstlog_impl's WP-generated frame —check-wp-result.pynow reports that frame, matching both wordings Frama-C uses for a missing spec — soVERIFY_ELF_MIN_GOALSwent 156→172 with the new contract and back to 159 once the false frame fell, andVERIFY_FUTEXDEADLINE_MIN_GOALSwent 128→130.check-proof-targetsnow runs onmake check, andBUILD_FLAVOR_GOALSskipsprint-%goals so that scanner's sub-make cannot wipe a sanitizer build's objects.Mutation gate
MUTANT_INCDIR(the staging root, not the copy's parent) so they shadow the real files, and a#errorprobe confirms each shadow is reached — run serially, since a concurrent sibling could spoil its exit-status answer — because the byte-identical baseline cannot detect a fall-through.-wp-cache nonebecause WP otherwise replays a stored timeout as a catch; baselines that fail only while the pool is busy are retried serially, and their targets' mutations then run serially too.[Timeout]— andMUTANT_ESCALATE=<n>, off by default, re-runs resource verdicts at a larger budget, reporting MISSED for a mutation that merely turned an easy goal into a slow one. Every open goal must belong to the mutated function, decided by the longest matching name in the target's own proved set and checked by a self-test, with caller obligations naming the callee qualifying for both; coverage is counted by function and by (target, function) pair.Written for commit b6ff88c. Summary will update on new commits.