Repository navigation
Description
Functional parity with MRI: the ordered plan, and what each phase is actually blocked on
This is a roadmap, not a wish. Every figure in it is either a measurement with a
date or marked as unmeasured, and the ordering is derived from the measurements
rather than from how interesting the work looks.
⛔ First, a definition of done, because "parity" is otherwise unfalsifiable
- Every ruby/spec tree that can be measured under the shim is swept and
gated per file, not justlanguage/+core/. - For every spec file, rbgo's passing count ≥ MRI's, or the shortfall is a
named open issue. - No silent divergence. Where rbgo cannot do something, it raises. A
plausible wrong answer is worse than an absence, because an absence is
honest and a wrong answer is relied upon. - The trees that cannot be measured are made measurable or declared out of
scope in writing — never left as a silent zero.
Criterion 3 is the one that reorders this list. Criterion 4 is the one that
today's library/socket discovery (#937) exists to prevent repeating.
The ordering principle
By harm per unit of work, where harm ranks:
a hang (nothing downstream is measurable through it) > a silent wrong
answer (the embedder believes a guarantee that does not hold) > a
measurement defect (every number below is read through it) > absent
surface (honest, merely incomplete).
What I verified today, with cmd/oracle against MRI 4.0.7
Five probes, run as the same program under both engines. Three of them
corrected the text of their own issue, and one refuted it outright — which is
why this plan is built on probes rather than on issue titles.
| issue | its claim | measured 2026-11-15 |
|---|---|---|
#803 Timeout.timeout never times out |
silent, unprotected | CONFIRMED — and for CPU-bound blocks too, not only sleeps |
| #768 no call-depth limit, the host dies | fatal Go stack overflow | ⭐ REFUTED — rescuable SystemStackError, byte-identical to MRI. Candidate for closing |
#909 Enumerable#to_set silent |
no deprecation warning | CONFIRMED — MRI writes 206 bytes to stderr, rbgo 0 |
#774 Kernel#fork runs in-process |
the child mutates the parent | CONFIRMED — x=99 vs MRI's x=1; and Process.fork is absent while Kernel#fork exists |
#817 Object.constants is empty |
0 vs MRI's 120 | CONFIRMED but INVERTED — rbgo answers 1620, MRI 120. Over-reporting, not emptiness |
⚠ My first #774 probe asked Process.respond_to?(:fork), got false, and took
an "absent" branch without ever calling fork — a negative that came from
the wrong question. Worth repeating because it is the cheapest mistake to make
here: a probe that does not execute the thing cannot report on it.
Phase 1 — liveness. Nothing downstream is measurable through a hang.
| item | state |
|---|---|
#940 Thread.new never returns for a block that neither blocks nor finishes |
filed today, root cause located |
#933 Thread#kill cannot interrupt a native socket call — 12 ioBlock sites, 8 hanging files |
open; #935 landed the primitive on 1 of 13 |
#895 TestUDPSocketLoopback can hang |
open; candidate consequence of #940 |
#940 is the lead item of the whole plan. The safepoint machinery is complete —
serviceSafepoint is called from 19 sites — and the interpreter's four jump
sites call none of them, so the one door a CPU-bound thread can reach is the one
that was never wired. It is the CPU half of #803 and of #933; neither can be
finished before it.
⛔ Blocked on a measurement window, not on design. The change is in the
interpreter's hottest loop, and this repository's rule is that performance is
part of every change. This machine's 1-minute load has been 36–90 all morning,
where a throughput figure cannot tell one design from another. Two designs to
measure against each other: a counter, versus a flag set by a timer (MRI's
rb_thread_schedule).
Phase 2 — measurement integrity. Every number below is read through these.
| item | what it costs us |
|---|---|
| #843 the harness reads its tally from the spec's own stdout | a spec file can dictate its recorded pass count |
| #837 two platform sources the comment says are one | on the ubuntu lane 48 linux examples never run and 23 darwin ones do |
#865 a before(:all)/after(:all) failure |
counted in no tally at all |
| #937's unexplained 103 absent examples in one file | a third of addrinfo/initialize_spec.rb produces no result, cause unknown |
#857 the corpus cache defaults to /tmp |
a reboot empties it |
This phase adds zero parity and makes parity measurable. Today's history
argues for it: library/socket read as a flat zero on both sides until #936,
and the same sweep that now reports −1566 reported "+217 in rbgo's favour"
twenty minutes earlier.
Phase 3 — silent wrong answers (criterion 3)
| item | the fix's shape |
|---|---|
#774 Kernel#fork |
⛔ this row was wrong and is corrected — see below |
#803 Timeout.timeout |
after #940 |
#817 Object.constants |
1620 → MRI's curated 120; both receivers return one table in both engines, so this is one defect, not two |
#909 to_set deprecation |
one warning |
#676 rbgo -e 'exit 3' exits 0 / #773 Run() returns nil |
an exit status that never reaches the process |
#686 IO.read accepts a positional Hash MRI rejects |
~22 examples counted with no judge |
| #643 the VM closes no IO at shutdown | MRI closes every one |
| #768 | ✅ closed as refuted, with the probe, 2026-11-15 |
⛔ The #774 row above was wrong within the hour, and the correction is the
useful part of this plan. I had written: raise NotImplementedError, because
MRI does that where fork is unavailable, so it is faithful and an honest absence
beats a silent wrong answer. Then I read the consumer.
The in-process emulation is load-bearing for the headline puppet apply
path: spawn.go:20 states that Puppet::Util::Execution.execute is built on it,
spawn.go:203 rests the STDOUT.reopen(pipe) on it, io.go:184 and
io_descriptors.go:645 handle descriptors specifically around a forked block,
and socket_dialerr_test.go:101 asserts that a child's exit is
runForkBlock returning its status. Raising would trade a shipped, advertised
capability for a conformance criterion — a maintainer's decision, not something
to slip in under "parity".
The three ways out, and what each costs, are written up on #774. My reading is
that making the emulation opt-in is the only one that satisfies criterion 3
without removing a feature, but it is a scope decision and I am not taking it
unilaterally.
The general lesson, since this plan will be read as instructions: a parity
criterion is not a licence to remove a capability. Read what depends on a
behaviour before prescribing its replacement — criterion 3 says "raise rather
than answer wrongly", and it cannot mean "raise rather than serve a consumer who
needs the answer".
Phase 4 — the measured surface, best ratio first
| item | examples |
|---|---|
#669 parser: do…end in an arrow-lambda body after a paren-less call |
160 from one defect — the best ratio in the tracker |
#937 library/socket |
−1560: Addrinfo#initialize −151, BasicSocket#recvmsg(+_nonblock) −100, the address/option accessors −188, the _nonblock family −79, AncillaryData#initialize −40 |
matrix |
233 |
bigdecimal |
180 (+ #825 #sqrt, promised on the docs site) |
date + datetime |
179 |
stringscanner |
154 (+ 27 newer byte methods, absent) |
zlib |
130 |
stringio |
94 |
openssl |
93 |
| small, named | Coverage 53, four CGI escape files 19, WeakRef 9, #670 Tempfile 8, #633 32 Errno constants on darwin |
#937's first slice is not the 19 predicate methods it first looked like:
measured, 40 of the 48 non-passing examples in addrinfo/initialize_spec.rb are
three missing Socket::SOCK_* constants (15) and one service-name port
path (25). The 103 absent examples in Phase 2 must be explained first, or a fix
aimed at the visible 48 leaves the larger half untouched.
Phase 5 — what no amount of interpreter work can move
78 files, where MRI itself scores 0 under this shim: 32 command_line/
specs built on ruby_exe (a subprocess), and 46 optional/capi specs needing a
compiled C extension — which CGO=0 makes a declared non-goal rather than a
gap.
This phase is a decision to write down, per criterion 4: either the harness
grows a real ruby_exe, or these are excluded from the denominator in writing.
Leaving them as a silent zero is what criterion 4 exists to forbid.
What the gap total is, honestly
2,341 across the 970 files where the two engines differ (2026-11-10, MRI
4.0.7) — a floor, not a total, because library/socket contributed zero to
it on both sides. Socket alone adds ≈1,560 (2026-11-14, MRI 4.0.6).
⛔ The two are deliberately not summed here: different runs, different MRI
patch releases, and a sum across two measurements is how two errors cancel. One
re-measurement under a quiet machine replaces both numbers with one; until then
the floor is the honest claim.
Refs #940, #933, #895, #843, #837, #865, #857, #937, #774, #803, #817, #909, #676, #773, #686, #643, #768, #669, #825, #670, #633
🤖 Generated with Claude Code
Activity
tannevaled commentedon Nov 15, 2026
Phase 2 gains a concrete item, and it is a pattern this repository already has twice
Phase 2 is "measurement integrity". The entry it was missing is the one that
would have prevented the four corrections of #938: the conformance figures are
retyped into prose, and nothing can tell when the prose and the record
disagree. Seven README drift gates stayed green through all four, because none
of them reads a figure from a dated run.
⛔ I had designed a weaker gate for this and was about to build it in
go-fleettools: markers in the prose (measured=,code=) checked againstgit logover the declared paths. Two readings stopped it before the first lineof code, and the second one matters more than the first.
A sibling org proves the stronger shape
go-scikit/refbenchv0.3.0 does not compare a published table to a record —it generates it.
refbench table <perf.json>rendersperf-table.md;refbench check <perf.json>refuses a table the record did not produce.Generated prose cannot drift at all, where a comparison always leaves the
question "with what tolerance".
And this repository already has that pattern, twice
Which is the better citation, because it needs no new dependency:
TestBenchmarkedModulesSayWhereTheirCodeLives(bench_backing_test.go)bench/modules/BACKING.mdgo.mod+bench/modules/*.rbprelude_frozen_test.gointernal/vm/prelude_frozen_gen.goThe first one already carries the two details that make such a gate honest, and
they should be copied rather than re-invented:
-update-backing), so the record is rewritten by acommand and never by hand;
if len(scripts) == 0 { t.Fatal("no benchmark scripts found; this test would pass by measuring nothing") }. A gate whosesubject went missing must fail, not pass.
What applying it to the conformance table takes
Every piece exists:
specdeltawrites per-file reports; a sweep isparity-*.txtspecroll -depth 2already renders exactly the module table the README carriesscripts/conformance/rubyspec/BASELINEis committed and gated, one row per spec fileSo: commit the sweep's report as
BASELINEis committed, generate the README'smodule table from it with
specroll, and add the drift test with its ownpopulation guard.
⛔ What it does NOT fix, said here so nobody assumes otherwise
It makes prose-versus-record drift impossible. It does not make the
record fresh: a committed report is a dated measurement, and a code change that
would move it leaves the record and the generated prose stale together.
That is a strictly smaller problem — one record to re-run instead of three
surfaces to re-read — and it is the honest limit. The generated rows should
therefore carry the corpus SHA and the MRI patch release per row, which is also
what today's prose does inconsistently: #938 had to label the published gap a
floor precisely because two figures from different runs (MRI 4.0.7 and
4.0.6) must not be summed, and provenance in a paragraph is provenance nobody
re-reads.
⚠ Filed as a comment here rather than as its own issue on purpose:
priorartexited 2 on this subject — "repo tracker NOT READ … stopped at the budget
reserve: 905 left, 1000 reserved" — and an unread tracker is not an absence of
prior art. A peer also pointed at #913/#765 as this same shape already built
twice, which the two gates above appear to be. When the budget allows a real
read, this belongs in its own issue if it is not already one.
🤖 Generated with Claude Code
tannevaled commentedon Nov 15, 2026
Correction to this plan: Phase 2 is NOT load-independent, and I claimed it was
The plan says Phase 1's safepoint work is "blocked on a measurement window, not
on design", and implies Phase 2 (measurement integrity) is free to proceed. I
tried to proceed with the generated-conformance-table gate from the comment
above, and it is blocked by the same thing.
A generated table needs a seed record, and the seed has to be a sweep anyone
would trust. Committing a stale one is worse than committing none: the
README's current table is already labelled a floor (#938) and
library/socketalone understates it by ~1560 (#937), so generating today's prose from the
2026-11-10 report would enshrine a figure known to be wrong in a known
direction — and make it look derived.
A trustworthy sweep is ~48 minutes with a per-file timeout, which is exactly the
measurement a loaded host cannot give: a timeout cannot tell a hang from a slow
file. So Phase 2's first item sits behind the same window as Phase 1's.
⛔ And a near-miss worth writing down, because it would have read as success
Looking for the report behind the README's table, I replayed a 2026-11-10 report
on disk:
against what the README publishes:
1335 appears in both — as the
gapin the report and as therbgoscore inthe README. Different quantities, same digits. Anyone grepping for the
published figure to identify its source report would have found it and stopped.
It is not the source: that file is a
library/-only-quietreport covering 339moved files, where the README's table covers 970 across four trees.
⭐
specrollbehaved correctly on the other candidate, which is the part worthkeeping: given a file that was not a specdelta report at all it exited 2 with
"the report format and this parser have drifted apart" rather than parsing
something and returning a plausible table. A tool that refuses is what let the
first comparison be trusted.
What this changes in the ordering
Nothing about the phases, but one honest line about the dependency: Phases 1
and 2 share a prerequisite that is neither design nor code — one quiet host,
once, long enough for a 48-minute sweep and an 8-round alternating benchmark.
Until then both phases can be written and neither can be finished, and the
plan should not read as though Phase 2 were available meanwhile.
🤖 Generated with Claude Code
README: a cross-reference to a section that was never written, and a …
docs: "front-end essentially complete" is true of the corpora it meas…
vm: Object.constants stops reporting the qualified index keys (1620 -…
vm: flush every open file at interpreter teardown, as MRI does (#948)
conformance: the tally gets its own channel, which the spec cannot wr…