Skip to content

specdelta: a hang published as an absence, and a report no tool would read - #915

Merged
tannevaled merged 2 commits into
mainfrom
specdelta-kill-not-absence
Nov 10, 2026
Merged

tannevaled merged 2 commits into
mainfrom
specdelta-kill-not-absence

Conversation

@tannevaled

@tannevaled tannevaled commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

run() ended with

return parse(out.String())

which builds a fresh result, so the killed it had set two lines above was dropped on the floor. Every row specdelta has ever printed carried killed == false. Three consequences, each worse than the last:

  1. The KILLED on timeout note was unreachable dead code.

  2. A hang was published as an absence. A killed engine almost never prints a RESULT line, so ran is false too — and the note switch tested DID NOT RUN first. A file the clock stopped was reported as DID NOT RUN on the treatment. Those are opposite findings: one says the timeout is too short, or the engine hangs, the other says this is unimplemented. They call for opposite work.

  3. The documented control for a whole report could only answer 0. "Grep the report for KILLED; if the count is 0 the table is sound" is how a sweep is judged trustworthy, and that grep was answering by construction.

Point 3 is not hypothetical. Measured yesterday: three sweeps of the ruby/spec corpus against MRI were declared sound on exactly that grep — including a library/ chunk that ran 3h45 with one spec file observed blocked for 4m46s while consuming no CPU. The report for that chunk contains no KILLED anywhere, and it could not have.

The fix

killed is carried across the parse, and tested before the DID NOT RUN arms. The switch moved into outcomeNote() so the ordering is a test rather than a comment.

Controls, in both directions

test what it pins
TestATimeoutIsCarriedOutOfRun /bin/sleep as an engine guaranteed to outlive the timeout. Against the unfixed line it fails with "came back with killed == false"; against the fix it passes in 0.16 s — and the time is the witness that the timeout is what ended it.
TestAnEngineThatFinishesIsNotMarkedKilled the other direction: a note that fires on every row is as useless as one that never fires.
TestAKillIsReportedBeforeAnAbsence all seven outcomes, with a killed-and-silent result as the realistic case — that is what a real timeout looks like, and it is the one the old order got wrong.

The first was run against the unfixed code before being run against the fix, so this is a break check that broke.

A second commit, found while checking the first

The re-measurement that this fix made possible produced a disagreement between two of my own readings: cmd/specroll, reading a library/ report in Go, counted 47 net-ftp rows with 39 of them unloadable, while grep over the same file found no mention of net-ftp at all.

grep was wrong:

$ file parity-lib-b.txt
parity-lib-b.txt: RAGE Package Format (RPF)

ruby/spec prints raw binary on purpose — zlib round-trips, net-http bodies, String#unpack — and the shim's RBGO_DETAIL line quotes the example's description and message verbatim, so NUL bytes reached the report. grep skips a file it decides is binary: not "0 matches", but no output and exit 1. Three separate checks of that report were read as clean on that silence — "is there a KILLED row?", "which files did not load?", "does it mention net-ftp?" — and all three had never looked inside it.

printable() strips control bytes from an engine's output before it is parsed, keeping tab and newline because the format is built on them. run.sh has done exactly this since it was written, one layer further out (tr -d '\000'); this is the same guard where the Go half can rely on it. Its tests pin that the text either side of a stripped byte survives — dropping the whole description would lose the finding instead of the byte — and that the separators do not get stripped with the rest.

⛔ Consequence for existing reports

Any report produced before this commit cannot distinguish the two causes, and every DID NOT RUN on the treatment row in one may be a timeout rather than a missing feature.

🤖 Generated with Claude Code

tannevaled and others added 2 commits November 10, 2026 03:29
…not fire

run() ended with

    return parse(out.String())

which builds a FRESH result, so the `killed` it had set two lines above
was dropped on the floor. Every row specdelta has ever printed carried
killed == false. Three consequences, each worse than the last:

1. The "KILLED on timeout" note was unreachable dead code.

2. A killed engine almost never prints a RESULT line, so `ran` is false
   too -- and the note switch tested DID NOT RUN first. A file the CLOCK
   stopped was therefore reported as "DID NOT RUN on the treatment": a
   hang published as an absence. Those are opposite findings. One says
   the timeout is too short or the engine hangs; the other says the
   feature is unimplemented, and they call for opposite work.

3. The documented control for a whole report -- grep it for KILLED, and
   trust the table when the count is 0 -- answered 0 BY CONSTRUCTION.
   Measured 2026-11-09: three sweeps of the ruby/spec corpus were
   declared sound on exactly that grep, including one of library/ that
   took 3h45 with a spec file observed blocked for 4m46s.

The fix carries `killed` across the parse and tests it BEFORE the
DID-NOT-RUN arms, in outcomeNote(), extracted so the ordering is a test
and not a comment.

Controls, in both directions:

  - TestATimeoutIsCarriedOutOfRun uses /bin/sleep as an engine guaranteed
    to outlive the timeout. Run against the unfixed line it fails with
    "came back with killed == false"; against the fix it passes in 0.16s,
    and the time is the witness that the timeout is what ended it.
  - TestAnEngineThatFinishesIsNotMarkedKilled is the other direction: a
    note that fires on every row is as useless as one that never fires.
  - TestAKillIsReportedBeforeAnAbsence pins all seven outcomes, with a
    killed-and-silent result as the realistic case -- that is what a real
    timeout looks like, and it is the one the old order got wrong.

⛔ Reports produced before this commit cannot distinguish the two causes.
Any "DID NOT RUN on the treatment" row in them may be a timeout.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ruby/spec files print raw binary on purpose -- zlib round-trips, net-http
bodies, String#unpack -- and the shim's RBGO_DETAIL line quotes the
example's description and message verbatim. Those bytes went straight
into the report.

⛔ MEASURED 2026-11-10. One library/ sweep produced a report holding NUL
bytes, and `file` called it "RAGE Package Format (RPF)". Every grep over
it then returned NOTHING AT ALL -- not "0 matches", but no output and
exit 1, because grep skips a file it decides is binary. Three separate
checks of that report were read as clean on that silence:

  "is there a KILLED row?"            -> no output, read as none
  "which files did not load?"         -> no output, read as none
  "does it mention net-ftp at all?"   -> no output, while cmd/specroll
                                         reading the same file in Go
                                         counted 47 net-ftp rows, 39 of
                                         them unloadable

The Go reader was right and the pipeline was wrong, which is the whole
argument for putting a verdict in a program rather than in a search.

printable() strips control bytes from an engine's output before it is
parsed, keeping tab and newline because the format is built on them.
run.sh has done exactly this since it was written, one layer further out
(`tr -d '\000'`); this is the same guard where the Go half can rely on
it.

Two tests: a dirty RBGO_DETAIL line must parse with the surrounding
result intact AND with the text either side of the stripped byte still
there (dropping the whole description would lose the finding instead of
the byte), and the format's own separators must survive.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tannevaled
tannevaled force-pushed the specdelta-kill-not-absence branch from faa69aa to 73838c3 Compare November 10, 2026 02:47
@tannevaled tannevaled changed the title specdelta: a hang was published as an absence, and the control could not fire specdelta: a hang published as an absence, and a report no tool would read Nov 10, 2026
@tannevaled
tannevaled merged commit 76e3376 into main Nov 10, 2026
26 checks passed
@tannevaled
tannevaled deleted the specdelta-kill-not-absence branch November 10, 2026 03:56
tannevaled added a commit that referenced this pull request Nov 10, 2026
The README has always said `library/`, `optional/`, `command_line/` and
`security/` are "not run at all" and that the figure is therefore not a
percentage of Ruby. True, and it left the reader with no idea whether the
unmeasured half was a rounding error or the bulk of the work.

It has now been measured once, against MRI 4.0.7 through the same shim:
970 files differ, MRI passes 3,676 examples in them and rbgo 1,335, and
152 files produce no result on rbgo. Three readings matter more than the
total, and all three change what someone would do next:

  - 78 of the unswept files CANNOT be measured under this shim at all.
    Every `command_line/` file is built on `ruby_exe`, and `optional/capi`
    needs a compiled C extension: MRI itself scores 0 in them. That part
    is harness work, not interpreter work.
  - The largest single debt is a HANG, not a missing feature: Net::FTP is
    442 examples behind and 39 of its 56 files are killed by the clock on
    rbgo and none on MRI (#916).
  - The rest is concentrated in eight modules, and the outright absences
    (Coverage 53, StringScanner's byte methods 27, CGI escapes 19,
    WeakRef 9) are small beside them.

Marked as one dated run and NOT a gate, in the same words the table above
it uses: nothing re-measures it, so it is a map of where the work is and
not a figure to quote.

The conformance README gains the two things #915 and #914 added and
nothing documented:

  - `KILLED on timeout` and why it is reported BEFORE `DID NOT RUN` --
    with the note that reports written before that commit cannot tell a
    hang from an absence, and that the timeout should be picked for the
    question (60x the file's work to detect a regression; a SHORT budget
    to tell a hang from a gap -- net-ftp at 300s was still running after
    77 minutes, at 20s it answered in 13).
  - `cmd/specroll`, its reconciliation against each report's own footer,
    and why the invariant differs between a full and a -quiet report.
  - `printable()`: a report holding a spec's raw binary was called "RAGE
    Package Format (RPF)" by `file`, and grep then skipped it silently.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant