tools: ask every compiler whether it reports what it finds - #3451
Merged
Conversation
Closes #3450 Refs #3448 One instrument had been censoring its own totals -- clang's default `-ferror-limit=20`, which made every C error count published for this repository a floor. The obvious next question is whether the others do the same, and nobody had asked. Planting 50 known errors per language: clang (default) 20 TRUNCATES clang (-ferror-limit=0) 50 complete rustc 50 complete zig 50 complete iverilog 50 complete (TWO diagnostic lines per error) yosys 1 ABORTS on the first error yosys is not capped, it stops. An error COUNT from it is 0 or 1 and means nothing; only its pass/fail is a measurement. THE FIXTURES WERE THE HARD PART, and two of the five were wrong in ways that read as findings about the compiler rather than about me: * the Zig fixture named its functions `f16` and `f32`, which SHADOW ZIG PRIMITIVES, so it failed with 2 errors of a different kind before reaching the undefined names -- indistinguishable from a cap; * the Verilog fixture used implicitly declared identifiers, which yosys treats as a WARNING, so it reported zero -- indistinguishable from silence. `--self-check` therefore requires each fixture to produce the expected KIND of diagnostic before its count is believed. WHAT IS NOT AFFECTED, measured rather than hoped: * acceptance counts. Clang's exit code is 1 with and without the limit on a failing file, and 0 on a clean one, so a count of FILES that compile cannot be censored by a per-file cap. Every `cc accepts N` figure stands. * the repository's gates. The only tool invoking `cc` is check_duplicate_agreement.py, and it reads `returncode`. * the Rust totals, since rustc reports completely. What was wrong is exactly the C error totals in my own reports: 3849 was 15188, 3825 was 15133, and after #3449 it is 15126. The tool DEGRADES rather than lying. A runner without zig or yosys checks the compilers it has and NAMES what it skipped; only a run with nothing available exits 2. That matters because the CI runner carries neither, and a gate that cannot run must say so rather than fail forever. Four controls: a planted truncation reddens it, a one-compiler PATH passes while naming four skips, an empty PATH exits 2, and a fixture that stops producing its error fails the self-check. Wired into Spec Guards, under a second. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
gHashTag
enabled auto-merge (squash)
September 8, 2026 02:13
Contributor
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3450 · Refs #3448
One instrument had been censoring its own totals — clang's default
-ferror-limit=20, which made every C error count published for thisrepository a floor. The obvious next question is whether the others do the
same, and nobody had asked.
Measured — 50 known errors planted per language
clang(default)clang -ferror-limit=0rustczig build-objiverilogyosysyosysis not capped, it stops. An error count from it is 0 or 1 and meansnothing; only its pass/fail is a measurement.
Two of the five fixtures were wrong, and both read as findings
f16andf32— which shadow Zigprimitives — so it failed with 2 errors of an entirely different kind
before reaching the undefined names. Indistinguishable from a cap.
treats as a warning, so it reported zero. Indistinguishable from silence.
Three of five instruments were misread on the first attempt and none of them
was at fault.
--self-checknow requires each fixture to produce the expectedkind of diagnostic before its count is believed.
What is not affected — measured, not hoped
ccischeck_duplicate_agreement.py, and it readsreturncoderustcreports completelyWhat was wrong is exactly the C error totals in my own reports:
It degrades rather than lying
The CI runner carries neither zig nor yosys, and a gate that cannot run must
say so rather than fail forever. So a runner checks the compilers it has and
names what it skipped; only a run with nothing available exits 2.
NOT CHECKED (4 of 6)Wired into
Spec Guards, under a second.