Repository navigation
Audit the generated C and C++ harness - #743
Merged
Merged
Conversation
The harnesses are helpers inside a template string, so no tool reads them as source. A C helper with a 4096-slot stack queue went unseen until an input of 5000 nodes crashed it, and the tests that found it each had to be written for that one helper. Each prelude is now compiled with warnings as errors. About 25 function judges per language, enough to show every parameter type, return type and special field once, are built with a stub candidate under AddressSanitizer and UBSan. C runs inputs past the sizes its helpers assume: 4097 items, and 100000 for trees and lists. C++ stops at 129, since g++ takes tens of seconds over a longer literal and its helpers grow vectors. A leak is excused only when an input builder allocated it, so a helper added later is checked without being listed, and a case that reports an error fails the run. Each fixed-capacity array and each judge known to fail is listed with the failure it must show, and the audit fails when one is fixed, so the entry leaves with the fix. It is its own portable gate rather than a browser test: the browser shards are cut by file size, and the slowest one takes about twice as long as the portable lane. On CI a missing compiler or sanitizer fails the gate, since a run that skips every audit still exits 0. The audit costs about 200 CPU seconds. scripts/dump-harness.mjs prints a prelude or one function judge's program to read or compile by hand. Change-Id: I588b70ae54a266e2450816ffa95947ff56cfe841
jserv
force-pushed
the
c-harness-audit
branch
from
November 16, 2026 22:35
9dfccea to
cb9bd25
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem and behavior
The C and C++ harnesses the page compiles around a candidate are helpers inside a template string in
web/compiler-explorer.js, so nothing reads them as source. No compiler warns on them and no sanitizer runs them past the cases a test was written for. A C helper that kept every node of a tree in a 4096-slot stack array went unseen until a 5000-node input crashed it, and the tests that found it were each written for that one helper.Approach
A new gate,
c-cpp-harness-audit, treats each prelude and the program built for each judge as the code they are. Each prelude is compiled with warnings as errors. About 25 function judges per language, enough to show every parameter type, return type and special field once, are built with a stub candidate under AddressSanitizer and UBSan. C runs inputs past the sizes its helpers assume: 4097 items, and 100000 for trees and lists. C++ stops at 129, because g++ takes tens of seconds over a longer literal and its helpers grow vectors. A leak is excused only when an input builder allocated it, so a helper added later is checked without being listed, and a case that reports an error fails the run. A scan lists every fixed-capacity array in a prelude, and a new one fails until it is listed with the bound that makes it safe.The audit finds the stack queue above, in
findTreeNode, which this change does not fix. It is listed as a known failure with the failure it must show, so the gate passes onmaintoday and fails once the helper is fixed, which tells whoever fixes it to delete the two entries.It is its own portable gate and not a browser test, because the browser shards are cut by file size and this is CPU-heavy for its size. On CI a missing compiler or sanitizer fails the gate, since a run that skips every audit still exits 0.
scripts/dump-harness.mjsprints a prelude or one function judge's program to read or compile by hand.Validation
node --test tests/harness/harness-audit.test.js: 68 tests pass. Pinned to four cores withtaskset -c 0-3it takes about 65 seconds and 200 CPU seconds. Building every distinct shape instead took 576 CPU seconds, and one C++ program alone costs about 5.findTreeNodefix from the open pull request for it (the known-failure entry fails and says to delete it). With a broken g++ andCI=truethe run exits 1, and withoutCIthe C audit still runs../scripts/test.sh portablepasses, and./scripts/indent.sh --check,eslint .andscripts/check-skill-refs.pyare clean.main: the audit took 33 to 36 seconds,gates (portable)338 to 375 seconds against 302, and the slowest browser shard 511 to 549 seconds against 573. The portable job grew by 36 to 73 seconds and stays well under the slowest shard, so the slowest lane did not get longer.Class judges, which call a class the candidate defines, and Java are not audited.