Skip to content

conformance: the tally gets its own channel, which the spec cannot write - #959

Merged
tannevaled merged 2 commits into
mainfrom
conformance-result-channel
Nov 20, 2026
Merged

tannevaled merged 2 commits into
mainfrom
conformance-result-channel

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

The judge read its verdict out of the stream the subject writes to

The ratchet's per-file pass count came from the subject's own stdout. The shim reported there, and run.sh fished the line back out:

res="$(printf '%s\n' "$out" | grep '^RBGO_RESULT' | head -1)"
p="$(printf '%s' "$res" | sed -E 's/.*pass=([0-9]+).*/\1/')"

The shim prints at exit, so anything the spec printed was first and head -1 took it. A file that failed every example while printing RBGO_RESULT pass=9999 fail=0 error=0 skip=0 was recorded as OK <path> 9999, and UPDATE_BASELINE=1 banked that number — measured 2026-10-31 on #843.

No adversary is needed. A spec for puts, for $stdout or for String#% can emit the prefix by accident; a SPEC_SHA bump pulls in whatever upstream has written since; and run.sh's 2>&1 meant a warning text was enough.

The tally gets its own channel

run.sh names a path per spec file in RBGO_RESULT_FILE. The shim reads that name once, before any spec code runs, and deletes it from ENV in the same breath — so the subject can neither print the tally nor learn where it goes. The file holds the pass count alone, as decimal digits and nothing else, so reading it is cat: the grep, the head and the sed are gone, and with them the last place where a not-quite-matching line produced a non-numeric count.

channel content reader
RBGO_RESULT_FILE the pass count, digits only run.sh, for the ratchet
stdout — unchanged the four-field tally, then RBGO_DETAIL a human, and cmd/specdelta

cmd/specdelta keeps the stream, because the four fields and the detail lines are only there. Its own guard against a forgery — refuse two result lines in one run — was already in place, which is why this was a defect of the ratchet alone. Giving specdelta the same out-of-band channel would mean widening the file's format, and is not part of this.

That ENV.delete returns the value and leaves nil behind was measured against MRI 4.0.7 with cmd/oracle: both engines agreed byte for byte.

⭐ A side effect worth naming: run_one no longer holds the subject's whole output in a shell variable. The sweep's own watcher comment lists core/string/multiply_spec.rb at 6.2 GB of allocation — that output was being captured in order to be thrown away.

Three gates, because reading cannot tell the fix from the defect

Both versions mention RBGO_RESULT, both print a tally, and the whole difference is which stream the judge believes. So two of the three run the real binary over the real shim, and the forgery is actually printed by a real spec.

gate what it pins
TestASpecCannotDictateItsOwnRecordedPassCount the recorded count is 1 — the true count, which differs from the forged 9999 and from zero, so a fix that recorded 0 for everything would not pass. Plus: the spec saw nil for ENV["RBGO_RESULT_FILE"], and the stdout line still reads pass=1 fail=1 error=0 skip=0 for specdelta. Its control is that the forgery was printed — were it merely absent, every other assertion would hold for the wrong reason
TestAHandRunStillGetsItsTallyOnStdout the other side of the boundary: name no file and the tally must still appear, and no file may be written. The stdout line is not a leftover for someone tidying up after this
TestTheRatchetReadsTheTallyFromItsOwnFile the shell half, which no Go test above can reach: a revert to grep piped into head -1 there would restore the defect while the shim's file channel simply went unread

⛔ The first version of that third gate refused the fix's own comment — the line in run_one that names the extraction it replaced. The rule was too broad by exactly one case, and banning a string anywhere in the file would forbid ever documenting the defect. The verdict is now a pure function, shellCodeHas, that skips full-line comments, with a witness on each side of its boundary (TestShellCodeHasSeesCodeAndNotComments, 6 rows: the extraction itself, indented code, two comments naming it, a trailing comment treated as code, and a clean file).

Verified by mutation, not by reading: with ENV.delete reverted to ENV[…], the behavioural test goes red and names the leaked path.

⛔ What was NOT measured here, and what measures it

The full ratchet sweep was not run locally. The corpus was gone from /tmp/rubyspec-corpus — which is #857 happening in the wild — and the 1-minute load was 7.47 against a 5-minute 22.79, where a sweep with a 25 s per-file timeout turns load into FILEFAIL.

The rubyspec-ratchet lane runs run.sh on every pull request and judges file by file, so a broken run_one comes back as every file FILEFAIL. That lane, on this branch, is the witness — not a green from main, which does not contain this change.

⛔ Correcting six README citations turned up something worse

The 26-line insertion moved every line-number citation into spec_helper.rb. Only three of the six are inside TestReadmeStubbedHelpersMatchTheShim's population — and those three were right. The three outside it had already drifted:

  • README.md cited ruby_exe/ruby_cmd at :882-883; they were at :904-905 before this change.
  • the conformance README cited ComplainMatcher at :204-249; the class ended at :236.

Measured, not computed from the +26 shift: run.sh:106→:118, output_to_fd :280→:306, ruby_exe :904→:930, ruby_cmd :905→:931, ComplainMatcher :204-249→:230-262.

The existing test's own comment names spec_helper.rb:882-883 and says a citation must never be mistaken for a claim — the blind spot is documented at the exact spot that created it. Widening the gate is a separate change and is filed, not smuggled in here.

Both READMEs now describe the two channels and say which reader each is for.

Dispositions

Closes #843
Refs #857, #941, #940, #942

🤖 Generated with Claude Code

tannevaled and others added 2 commits November 20, 2026 08:29
The ratchet read its per-file pass count out of the SUBJECT'S OWN STDOUT.
The shim reported there, and run.sh fished the line back out with

    res="$(printf '%s\n' "$out" | grep '^RBGO_RESULT' | head -1)"
    p="$(printf '%s' "$res" | sed -E 's/.*pass=([0-9]+).*/\1/')"

The shim prints at exit, so anything the spec printed was FIRST and
`head -1` took it. A file that failed every example while printing
`RBGO_RESULT pass=9999 fail=0 error=0 skip=0` was recorded as
`OK <path> 9999`, and UPDATE_BASELINE=1 banked that number. Measured
2026-10-31 (#843).

That needs no adversary. A spec for `puts`, for `$stdout` or for
`String#%` can emit the prefix by accident, a SPEC_SHA bump pulls in
whatever upstream has written since, and run.sh's `2>&1` meant a
WARNING text was enough.

So run.sh now names a path per spec file in RBGO_RESULT_FILE, and the
shim reads that name ONCE, before any spec code runs, and DELETES it
from ENV in the same breath. The subject can then neither print the
tally nor learn where it goes. (That `ENV.delete` returns the value and
leaves nil behind was measured against MRI 4.0.7 with cmd/oracle: both
engines agreed byte for byte.) The file holds the pass count alone, as
decimal digits and nothing else, so reading it is `cat` -- the grep,
the head and the sed are gone, and with them the last place where a
not-quite-matching line produced a non-numeric count.

The stdout line is UNCHANGED, for a hand-run and for cmd/specdelta,
which needs the four-field tally and the RBGO_DETAIL lines that the
file does not carry. specdelta's own guard against a forgery -- refuse
TWO result lines in one run -- was already there, and is why this was
a defect of the ratchet alone.

⭐ A side effect worth naming: run_one no longer holds the subject's
whole output in a shell variable. The sweep's own watcher comment lists
core/string/multiply_spec.rb at 6.2 GB of allocation; that output was
being captured to be thrown away.

Three gates land with it, and the first two run the real binary over the
real shim because READING cannot tell this fix from the defect -- both
files mention RBGO_RESULT, both print a tally, and the whole difference
is which stream the judge believes:

  - TestASpecCannotDictateItsOwnRecordedPassCount actually prints the
    forgery from a real spec. It asserts the recorded count is 1 (the
    true count, which differs from the forged 9999 AND from zero, so a
    fix that recorded 0 for everything would not pass), that the spec
    saw nil for ENV["RBGO_RESULT_FILE"], and that the shim's own stdout
    line still reads pass=1 fail=1 error=0 skip=0 for specdelta. Its
    control is that the forgery WAS printed: were it merely absent,
    every other assertion would hold for the wrong reason.
  - TestAHandRunStillGetsItsTallyOnStdout is the other side of the
    boundary: name no file and the tally must still appear on stdout,
    and no file may be written. The stdout line is not a leftover for
    someone tidying up after this change.
  - TestTheRatchetReadsTheTallyFromItsOwnFile gates the shell half,
    which no Go test above can reach: a revert to `grep | head -1`
    there would restore the defect while the shim's file channel simply
    went unread.

⛔ The first version of that third gate refused the fix's OWN COMMENT --
the line in run_one that names the `grep … | head -1` it replaced. The
rule was too broad by exactly one case, and banning a string anywhere
in the file would forbid ever DOCUMENTING the defect. The verdict is now
a pure function, shellCodeHas, which skips full-line comments and has a
witness on EACH side of its boundary (TestShellCodeHasSeesCodeAndNotComments,
6 rows: the extraction itself, indented code, two comments naming it, a
trailing comment treated as code, and a clean file).

Verified by mutation, not by reading: with ENV.delete reverted to
ENV[...], the behavioural test goes red and NAMES the leaked path.

⛔ The full ratchet sweep was NOT run locally. The corpus was gone from
/tmp/rubyspec-corpus -- which is #857 happening in the wild -- and the
1-minute load was 7.47 against a 5-minute 22.79, where a sweep with a
25 s per-file timeout would turn load into FILEFAIL. The rubyspec-ratchet
lane runs run.sh on every pull request and judges file by file, so a
broken run_one comes back as every file FILEFAIL; that lane, on this
branch, is the witness.

Six README line-number citations into spec_helper.rb moved with the
26-line insertion, and correcting them turned up something worse. Only
three are inside TestReadmeStubbedHelpersMatchTheShim's population, and
those three were right. The three OUTSIDE it had ALREADY DRIFTED:
README.md cited ruby_exe/ruby_cmd at :882-883 when they were at :904-905
before this change, and the conformance README cited ComplainMatcher at
:204-249 when the class ended at :236. Measured, not computed from the
+26 shift: run.sh:106 -> :118, output_to_fd :280 -> :306, ruby_exe :904
-> :930, ruby_cmd :905 -> :931, ComplainMatcher :204-249 -> :230-262.
The blind spot is filed separately rather than widened here.

Both READMEs now describe the two channels and say which reader each
one is for.

Closes #843
Refs #857, #941

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… start

The Windows lane went red on both behavioural gates, and the message was
"the probe never printed its forged tally, so this run proves nothing"
with NOTHING after the colon.

⭐ That is the CONTROL doing its job. The test refused to report a pass
from a run that had proved nothing, instead of asserting on an empty
string and going green for the wrong reason.

Two defects, and the second is the one worth naming.

1. `go build -o <path>` writes EXACTLY the path given, so on Windows the
   test produced an extensionless file that CreateProcess will not
   start. socket_test.go has carried `bin += ".exe"` for this reason;
   this test did not.

2. ⛔ runProbe wrote `out, _ := cmd.CombinedOutput()` under a comment
   justifying only a non-zero EXIT STATUS -- "a spec with a failing
   example is a normal outcome for this harness". That is true, and it
   is not the same thing as a binary that never ran. Both arrive as a
   discarded error and an empty `out`, and empty output makes every
   assertion downstream fail for a reason that is not the one printed.

   "Could not ask" and "answered differently" need opposite responses --
   the rule cmd/oracle's own documentation states, and which this helper
   broke while testing a channel whose whole point is that an absence
   must not read as a measurement.

   An `*exec.ExitError` is now tolerated and anything else is fatal, so
   the lane will say "could not run <path>: ..." rather than blaming the
   probe for a silence it did not cause.

Nothing else moved: the shim, run.sh and the three gates' assertions are
unchanged. On this commit's parent the ratchet lane -- which runs the
real sweep through the changed run.sh and judges file by file -- was
already SUCCESS, along with 24 of the 26 checks.

Refs #843

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit 5d0c805 into main Nov 20, 2026
26 checks passed
@tannevaled
tannevaled deleted the conformance-result-channel branch November 20, 2026 10:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The conformance harness reads its tally out of the spec's own stdout: a spec file can dictate its recorded pass count

1 participant