Skip to content

Audit the generated C and C++ harness - #743

Merged
jserv merged 1 commit into
mainfrom
c-harness-audit
Nov 16, 2026
Merged

jserv merged 1 commit into
mainfrom
c-harness-audit

Conversation

@jserv

@jserv jserv commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Problem and behavior

The C and C++ harnesses the page compiles around a candidate are helpers inside a template string in web/compiler-explorer.js, so nothing reads them as source. No compiler warns on them and no sanitizer runs them past the cases a test was written for. A C helper that kept every node of a tree in a 4096-slot stack array went unseen until a 5000-node input crashed it, and the tests that found it were each written for that one helper.

Approach

A new gate, c-cpp-harness-audit, treats each prelude and the program built for each judge as the code they are. Each prelude is compiled with warnings as errors. About 25 function judges per language, enough to show every parameter type, return type and special field once, are built with a stub candidate under AddressSanitizer and UBSan. C runs inputs past the sizes its helpers assume: 4097 items, and 100000 for trees and lists. C++ stops at 129, because g++ takes tens of seconds over a longer literal and its helpers grow vectors. A leak is excused only when an input builder allocated it, so a helper added later is checked without being listed, and a case that reports an error fails the run. A scan lists every fixed-capacity array in a prelude, and a new one fails until it is listed with the bound that makes it safe.

The audit finds the stack queue above, in findTreeNode, which this change does not fix. It is listed as a known failure with the failure it must show, so the gate passes on main today and fails once the helper is fixed, which tells whoever fixes it to delete the two entries.

It is its own portable gate and not a browser test, because the browser shards are cut by file size and this is CPU-heavy for its size. On CI a missing compiler or sanitizer fails the gate, since a run that skips every audit still exits 0. scripts/dump-harness.mjs prints a prelude or one function judge's program to read or compile by hand.

Validation

  • node --test tests/harness/harness-audit.test.js: 68 tests pass. Pinned to four cores with taskset -c 0-3 it takes about 65 seconds and 200 CPU seconds. Building every distinct shape instead took 576 CPU seconds, and one C++ program alone costs about 5.
  • Each of these makes it fail, and I ran each: a C helper that leaks its own buffer (5 judges fail), a new 8192-slot array in a C helper (the scan fails), a C++ helper that leaks (3 fail), a C++ helper that throws (28 fail), and the findTreeNode fix from the open pull request for it (the known-failure entry fails and says to delete it). With a broken g++ and CI=true the run exits 1, and without CI the C audit still runs.
  • ./scripts/test.sh portable passes, and ./scripts/indent.sh --check, eslint . and scripts/check-skill-refs.py are clean.
  • CI time, over the three full runs of this change against the last run on main: the audit took 33 to 36 seconds, gates (portable) 338 to 375 seconds against 302, and the slowest browser shard 511 to 549 seconds against 573. The portable job grew by 36 to 73 seconds and stays well under the slowest shard, so the slowest lane did not get longer.

Class judges, which call a class the candidate defines, and Java are not audited.

cubic-dev-ai[bot]

This comment was marked as resolved.

The harnesses are helpers inside a template string, so no tool reads
them as source. A C helper with a 4096-slot stack queue went unseen
until an input of 5000 nodes crashed it, and the tests that found it
each had to be written for that one helper.

Each prelude is now compiled with warnings as errors. About 25 function
judges per language, enough to show every parameter type, return type
and special field once, are built with a stub candidate under
AddressSanitizer and UBSan. C runs inputs past the sizes its helpers
assume: 4097 items, and 100000 for trees and lists. C++ stops at 129,
since g++ takes tens of seconds over a longer literal and its helpers
grow vectors. A leak is excused only when an input builder allocated
it, so a helper added later is checked without being listed, and a case
that reports an error fails the run. Each fixed-capacity array and each
judge known to fail is listed with the failure it must show, and the
audit fails when one is fixed, so the entry leaves with the fix.

It is its own portable gate rather than a browser test: the browser
shards are cut by file size, and the slowest one takes about twice as
long as the portable lane. On CI a missing compiler or sanitizer fails
the gate, since a run that skips every audit still exits 0. The audit
costs about 200 CPU seconds.

scripts/dump-harness.mjs prints a prelude or one function judge's
program to read or compile by hand.

Change-Id: I588b70ae54a266e2450816ffa95947ff56cfe841
@jserv
jserv merged commit 21c45b1 into main Nov 16, 2026
21 checks passed
@jserv
jserv deleted the c-harness-audit branch November 16, 2026 22:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant