Repository navigation
conformance: the tally gets its own channel, which the spec cannot write - #959
Merged
Merged
Conversation
The ratchet read its per-file pass count out of the SUBJECT'S OWN STDOUT.
The shim reported there, and run.sh fished the line back out with
res="$(printf '%s\n' "$out" | grep '^RBGO_RESULT' | head -1)"
p="$(printf '%s' "$res" | sed -E 's/.*pass=([0-9]+).*/\1/')"
The shim prints at exit, so anything the spec printed was FIRST and
`head -1` took it. A file that failed every example while printing
`RBGO_RESULT pass=9999 fail=0 error=0 skip=0` was recorded as
`OK <path> 9999`, and UPDATE_BASELINE=1 banked that number. Measured
2026-10-31 (#843).
That needs no adversary. A spec for `puts`, for `$stdout` or for
`String#%` can emit the prefix by accident, a SPEC_SHA bump pulls in
whatever upstream has written since, and run.sh's `2>&1` meant a
WARNING text was enough.
So run.sh now names a path per spec file in RBGO_RESULT_FILE, and the
shim reads that name ONCE, before any spec code runs, and DELETES it
from ENV in the same breath. The subject can then neither print the
tally nor learn where it goes. (That `ENV.delete` returns the value and
leaves nil behind was measured against MRI 4.0.7 with cmd/oracle: both
engines agreed byte for byte.) The file holds the pass count alone, as
decimal digits and nothing else, so reading it is `cat` -- the grep,
the head and the sed are gone, and with them the last place where a
not-quite-matching line produced a non-numeric count.
The stdout line is UNCHANGED, for a hand-run and for cmd/specdelta,
which needs the four-field tally and the RBGO_DETAIL lines that the
file does not carry. specdelta's own guard against a forgery -- refuse
TWO result lines in one run -- was already there, and is why this was
a defect of the ratchet alone.
⭐ A side effect worth naming: run_one no longer holds the subject's
whole output in a shell variable. The sweep's own watcher comment lists
core/string/multiply_spec.rb at 6.2 GB of allocation; that output was
being captured to be thrown away.
Three gates land with it, and the first two run the real binary over the
real shim because READING cannot tell this fix from the defect -- both
files mention RBGO_RESULT, both print a tally, and the whole difference
is which stream the judge believes:
- TestASpecCannotDictateItsOwnRecordedPassCount actually prints the
forgery from a real spec. It asserts the recorded count is 1 (the
true count, which differs from the forged 9999 AND from zero, so a
fix that recorded 0 for everything would not pass), that the spec
saw nil for ENV["RBGO_RESULT_FILE"], and that the shim's own stdout
line still reads pass=1 fail=1 error=0 skip=0 for specdelta. Its
control is that the forgery WAS printed: were it merely absent,
every other assertion would hold for the wrong reason.
- TestAHandRunStillGetsItsTallyOnStdout is the other side of the
boundary: name no file and the tally must still appear on stdout,
and no file may be written. The stdout line is not a leftover for
someone tidying up after this change.
- TestTheRatchetReadsTheTallyFromItsOwnFile gates the shell half,
which no Go test above can reach: a revert to `grep | head -1`
there would restore the defect while the shim's file channel simply
went unread.
⛔ The first version of that third gate refused the fix's OWN COMMENT --
the line in run_one that names the `grep … | head -1` it replaced. The
rule was too broad by exactly one case, and banning a string anywhere
in the file would forbid ever DOCUMENTING the defect. The verdict is now
a pure function, shellCodeHas, which skips full-line comments and has a
witness on EACH side of its boundary (TestShellCodeHasSeesCodeAndNotComments,
6 rows: the extraction itself, indented code, two comments naming it, a
trailing comment treated as code, and a clean file).
Verified by mutation, not by reading: with ENV.delete reverted to
ENV[...], the behavioural test goes red and NAMES the leaked path.
⛔ The full ratchet sweep was NOT run locally. The corpus was gone from
/tmp/rubyspec-corpus -- which is #857 happening in the wild -- and the
1-minute load was 7.47 against a 5-minute 22.79, where a sweep with a
25 s per-file timeout would turn load into FILEFAIL. The rubyspec-ratchet
lane runs run.sh on every pull request and judges file by file, so a
broken run_one comes back as every file FILEFAIL; that lane, on this
branch, is the witness.
Six README line-number citations into spec_helper.rb moved with the
26-line insertion, and correcting them turned up something worse. Only
three are inside TestReadmeStubbedHelpersMatchTheShim's population, and
those three were right. The three OUTSIDE it had ALREADY DRIFTED:
README.md cited ruby_exe/ruby_cmd at :882-883 when they were at :904-905
before this change, and the conformance README cited ComplainMatcher at
:204-249 when the class ended at :236. Measured, not computed from the
+26 shift: run.sh:106 -> :118, output_to_fd :280 -> :306, ruby_exe :904
-> :930, ruby_cmd :905 -> :931, ComplainMatcher :204-249 -> :230-262.
The blind spot is filed separately rather than widened here.
Both READMEs now describe the two channels and say which reader each
one is for.
Closes #843
Refs #857, #941
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… start The Windows lane went red on both behavioural gates, and the message was "the probe never printed its forged tally, so this run proves nothing" with NOTHING after the colon. ⭐ That is the CONTROL doing its job. The test refused to report a pass from a run that had proved nothing, instead of asserting on an empty string and going green for the wrong reason. Two defects, and the second is the one worth naming. 1. `go build -o <path>` writes EXACTLY the path given, so on Windows the test produced an extensionless file that CreateProcess will not start. socket_test.go has carried `bin += ".exe"` for this reason; this test did not. 2. ⛔ runProbe wrote `out, _ := cmd.CombinedOutput()` under a comment justifying only a non-zero EXIT STATUS -- "a spec with a failing example is a normal outcome for this harness". That is true, and it is not the same thing as a binary that never ran. Both arrive as a discarded error and an empty `out`, and empty output makes every assertion downstream fail for a reason that is not the one printed. "Could not ask" and "answered differently" need opposite responses -- the rule cmd/oracle's own documentation states, and which this helper broke while testing a channel whose whole point is that an absence must not read as a measurement. An `*exec.ExitError` is now tolerated and anything else is fatal, so the lane will say "could not run <path>: ..." rather than blaming the probe for a silence it did not cause. Nothing else moved: the shim, run.sh and the three gates' assertions are unchanged. On this commit's parent the ratchet lane -- which runs the real sweep through the changed run.sh and judges file by file -- was already SUCCESS, along with 24 of the 26 checks. Refs #843 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The judge read its verdict out of the stream the subject writes to
The ratchet's per-file pass count came from the subject's own stdout. The shim reported there, and
run.shfished the line back out:The shim prints at exit, so anything the spec printed was first and
head -1took it. A file that failed every example while printingRBGO_RESULT pass=9999 fail=0 error=0 skip=0was recorded asOK <path> 9999, andUPDATE_BASELINE=1banked that number — measured 2026-10-31 on #843.No adversary is needed. A spec for
puts, for$stdoutor forString#%can emit the prefix by accident; aSPEC_SHAbump pulls in whatever upstream has written since; andrun.sh's2>&1meant a warning text was enough.The tally gets its own channel
run.shnames a path per spec file inRBGO_RESULT_FILE. The shim reads that name once, before any spec code runs, and deletes it fromENVin the same breath — so the subject can neither print the tally nor learn where it goes. The file holds the pass count alone, as decimal digits and nothing else, so reading it iscat: thegrep, theheadand thesedare gone, and with them the last place where a not-quite-matching line produced a non-numeric count.RBGO_RESULT_FILErun.sh, for the ratchetRBGO_DETAILcmd/specdeltacmd/specdeltakeeps the stream, because the four fields and the detail lines are only there. Its own guard against a forgery — refuse two result lines in one run — was already in place, which is why this was a defect of the ratchet alone. Giving specdelta the same out-of-band channel would mean widening the file's format, and is not part of this.That
ENV.deletereturns the value and leavesnilbehind was measured against MRI 4.0.7 withcmd/oracle: both engines agreed byte for byte.⭐ A side effect worth naming:
run_oneno longer holds the subject's whole output in a shell variable. The sweep's own watcher comment listscore/string/multiply_spec.rbat 6.2 GB of allocation — that output was being captured in order to be thrown away.Three gates, because reading cannot tell the fix from the defect
Both versions mention
RBGO_RESULT, both print a tally, and the whole difference is which stream the judge believes. So two of the three run the real binary over the real shim, and the forgery is actually printed by a real spec.TestASpecCannotDictateItsOwnRecordedPassCountnilforENV["RBGO_RESULT_FILE"], and the stdout line still readspass=1 fail=1 error=0 skip=0for specdelta. Its control is that the forgery was printed — were it merely absent, every other assertion would hold for the wrong reasonTestAHandRunStillGetsItsTallyOnStdoutTestTheRatchetReadsTheTallyFromItsOwnFilegreppiped intohead -1there would restore the defect while the shim's file channel simply went unread⛔ The first version of that third gate refused the fix's own comment — the line in
run_onethat names the extraction it replaced. The rule was too broad by exactly one case, and banning a string anywhere in the file would forbid ever documenting the defect. The verdict is now a pure function,shellCodeHas, that skips full-line comments, with a witness on each side of its boundary (TestShellCodeHasSeesCodeAndNotComments, 6 rows: the extraction itself, indented code, two comments naming it, a trailing comment treated as code, and a clean file).Verified by mutation, not by reading: with
ENV.deletereverted toENV[…], the behavioural test goes red and names the leaked path.⛔ What was NOT measured here, and what measures it
The full ratchet sweep was not run locally. The corpus was gone from
/tmp/rubyspec-corpus— which is #857 happening in the wild — and the 1-minute load was 7.47 against a 5-minute 22.79, where a sweep with a 25 s per-file timeout turns load intoFILEFAIL.The
rubyspec-ratchetlane runsrun.shon every pull request and judges file by file, so a brokenrun_onecomes back as every fileFILEFAIL. That lane, on this branch, is the witness — not a green frommain, which does not contain this change.⛔ Correcting six README citations turned up something worse
The 26-line insertion moved every line-number citation into
spec_helper.rb. Only three of the six are insideTestReadmeStubbedHelpersMatchTheShim's population — and those three were right. The three outside it had already drifted:README.mdcitedruby_exe/ruby_cmdat:882-883; they were at:904-905before this change.ComplainMatcherat:204-249; the class ended at:236.Measured, not computed from the +26 shift:
run.sh:106→:118,output_to_fd :280→:306,ruby_exe :904→:930,ruby_cmd :905→:931,ComplainMatcher :204-249→:230-262.The existing test's own comment names
spec_helper.rb:882-883and says a citation must never be mistaken for a claim — the blind spot is documented at the exact spot that created it. Widening the gate is a separate change and is filed, not smuggled in here.Both READMEs now describe the two channels and say which reader each is for.
Dispositions
Closes #843
Refs #857, #941, #940, #942
/tmpwhen the sweep was wanted.🤖 Generated with Claude Code