From c7637ac489e205a4a984ae823961b7b049dbed1f Mon Sep 17 00:00:00 2001 From: tannevaled Date: Tue, 10 Nov 2026 15:07:28 +0100 Subject: [PATCH] docs: what the unswept 42% of ruby/spec holds, and cmd/specroll The README has always said `library/`, `optional/`, `command_line/` and `security/` are "not run at all" and that the figure is therefore not a percentage of Ruby. True, and it left the reader with no idea whether the unmeasured half was a rounding error or the bulk of the work. It has now been measured once, against MRI 4.0.7 through the same shim: 970 files differ, MRI passes 3,676 examples in them and rbgo 1,335, and 152 files produce no result on rbgo. Three readings matter more than the total, and all three change what someone would do next: - 78 of the unswept files CANNOT be measured under this shim at all. Every `command_line/` file is built on `ruby_exe`, and `optional/capi` needs a compiled C extension: MRI itself scores 0 in them. That part is harness work, not interpreter work. - The largest single debt is a HANG, not a missing feature: Net::FTP is 442 examples behind and 39 of its 56 files are killed by the clock on rbgo and none on MRI (#916). - The rest is concentrated in eight modules, and the outright absences (Coverage 53, StringScanner's byte methods 27, CGI escapes 19, WeakRef 9) are small beside them. Marked as one dated run and NOT a gate, in the same words the table above it uses: nothing re-measures it, so it is a map of where the work is and not a figure to quote. The conformance README gains the two things #915 and #914 added and nothing documented: - `KILLED on timeout` and why it is reported BEFORE `DID NOT RUN` -- with the note that reports written before that commit cannot tell a hang from an absence, and that the timeout should be picked for the question (60x the file's work to detect a regression; a SHORT budget to tell a hang from a gap -- net-ftp at 300s was still running after 77 minutes, at 20s it answered in 13). - `cmd/specroll`, its reconciliation against each report's own footer, and why the invariant differs between a full and a -quiet report. - `printable()`: a report holding a spec's raw binary was called "RAGE Package Format (RPF)" by `file`, and grep then skipped it silently. Co-Authored-By: Claude Opus 5 --- README.md | 31 ++++++++++++ scripts/conformance/rubyspec/README.md | 69 ++++++++++++++++++++++++++ 2 files changed, 100 insertions(+) diff --git a/README.md b/README.md index 6cd7f788..732fedf4 100644 --- a/README.md +++ b/README.md @@ -196,6 +196,37 @@ there is no honest way to turn it into a percentage of the language: - **An "example" is not a feature.** Examples are not weighted, so a heavily specified method contributes far more than a rarely used one. +**What the unswept 42 % holds — measured once, 2026-11-10.** The trees above are +not gated, but they have now been measured a single time, against MRI 4.0.7 +through the same shim, so the gap is a number rather than a guess +(`cmd/specdelta` + `cmd/specroll`, corpus at the pinned `SPEC_SHA`, `c604cce1`): + +| | | +| --- | --- | +| spec files in the corpus | 3,814 — of which 2,206 are swept and 1,608 are not | +| files where the two engines differ | 970 | +| examples MRI passes in them / rbgo passes | 3,676 / 1,335 | +| the gap | 2,341 | +| files rbgo produces no result for | 152 — **75 of them hang**, the rest are genuine absences | + +Three readings matter more than the total: + +- **78 of the unswept files cannot be measured at all under this shim.** All 32 + `command_line/` files are built on `ruby_exe`, and the 46 `optional/capi/` + files need a compiled C extension: **MRI itself scores 0** in them. That part + of the corpus needs harness work, not interpreter work. +- **The largest single debt is a hang, not a missing feature.** `Net::FTP` alone + is 442 examples behind, and 39 of its 56 spec files have to be killed by the + clock on rbgo and none on MRI ([#916](https://github.com/go-embedded-ruby/ruby/issues/916)). +- **The rest is concentrated:** `matrix` 233, `bigdecimal` 180, `stringscanner` + 154, `date`+`datetime` 179, `zlib` 130, `stringio` 94, `openssl` 93. The + outright absences are small by comparison — `Coverage` 53, `StringScanner`'s + newer byte methods 27, four `CGI` escape files 19, `WeakRef` 9. + +⛔ This is **one dated run, and not a gate**: nothing re-measures it, so treat it +as a map of where the work is rather than as a figure to quote. The swept +`language/` + `core/` number above is the one CI enforces. + **What CI guarantees is per file, not a total.** `BASELINE` records the passing count of each of the 2,206 spec files, and the run fails if any file passes fewer than its record or stops loading. There is no margin and no single number diff --git a/scripts/conformance/rubyspec/README.md b/scripts/conformance/rubyspec/README.md index 838295cf..740a2776 100644 --- a/scripts/conformance/rubyspec/README.md +++ b/scripts/conformance/rubyspec/README.md @@ -509,6 +509,75 @@ unreadable result line, two result lines in one run, a missing or malformed population** — a mistyped `-dir` would otherwise print `0 files, no regressions` and exit 0. +### ⛔ A hang is not an absence, and the report says which + +A row whose engine was stopped by `-timeout` reads + +```console +library/net-ftp/put_spec.rb 39 0 -39 KILLED on timeout, the treatment +``` + +and the kill is reported **before** `DID NOT RUN`. The order is the whole point: +a killed engine prints no `RBGO_RESULT` line either, so `ran` is false and the +`DID NOT RUN` arms would otherwise claim every killed row — publishing a file +the **clock** stopped as a file the engine could not **load**. Those call for +opposite work: *the timeout is too short, or it hangs* against *this is +unimplemented*. + +It did exactly that until 2026-11-10 (#915): `run()` returned a freshly parsed +result, so `killed` was false on every row ever printed and this note was dead +code. The documented way to judge a whole report — *grep it for `KILLED`; trust +the table when the count is 0* — could only ever answer 0. The first sweep after +the fix turned **75 files** from "unimplemented" into "hangs" (`Net::FTP` 39, +`Net::HTTP` 27, `Socket` 8, `Timeout` 1 — issue #916). + +⛔ **Reports written before that commit cannot tell the two apart.** + +**Pick the timeout for the question.** For *did this regress*, give a file +roughly 60× its usual work. For *does it hang or is it missing*, a SHORT budget +is the better instrument: 20 s is already ~100× a normal spec file, anything +past it is a hang by any definition, and the answer arrives in minutes. The +first re-measurement of `net-ftp` at 300 s was still running after 77 minutes +with nothing printed; at 20 s it answered in 13. + +### Control bytes never reach the report + +ruby/spec prints raw binary on purpose — zlib round-trips, `Net::HTTP` bodies, +`String#unpack` — and the shim quotes an example's description verbatim into +`RBGO_DETAIL`. Those bytes used to land in the report, and `file` then called it +*"RAGE Package Format (RPF)"*: **grep skips a file it decides is binary**, so +every search over it returned no output and exit 1, which reads exactly like +"no matches". Three checks of one `library/` report were passed on that silence. +`printable()` strips control bytes before parsing, keeping tab and newline +because the format is built on them — the same guard `run.sh` has always had one +layer out (`tr -d '\000'`). + +## Rolling a report up to the module — `cmd/specroll` + +specdelta answers per **file**. *"What is left to reach parity with MRI?"* is a +question about **modules**, and one run of `library/` is 497 rows with 189 +moved — a shape no one reads by eye, and summarising it by eye is a count done +by eye. + +```console +$ go run ./scripts/conformance/rubyspec/cmd/specroll -depth 2 -top 5 parity-*.txt +module files MRI rbgo gap unloadable (examples) +library/net-ftp 47 448 6 442 39 (388) +library/net-http 124 354 14 340 30 (75) +library/matrix 80 334 101 233 +library/bigdecimal 41 298 118 180 2 (8) +library/stringscanner 33 215 61 154 3 (27) +TOTAL 970 3676 1335 2341 152 (620) +``` + +It **checks its arithmetic against each report's own footer** and refuses when +they disagree — a rollup is exactly where a dropped row is invisible, since the +table still looks tidy and the only symptom is a total nobody re-derives. The +invariant differs by report shape: a full report must match on all three +numbers, a `-quiet` one only on the **delta**, since a file it omits contributes +0 to it. It also keeps *behind* and *absent* in separate columns, and says so +when a `-quiet` input makes the two count columns cover only the moved files. + ## Caveats The shim is not full MSpec. What is genuinely stubbed raises `SpecSkip`, so