Skip to content

Commit f12d0e5

Browse files
tannevaledclaude
andcommitted
README: measured ruby/spec conformance, and the claims that had drifted
The ruby/spec figure said 15,400+, which was the FLOOR bumped on 2026-08-25 (#451). Measured on main (68cb53a) against the pinned corpus today: 21,901 passing, four runs 21845/21903/21901/21902, floor 21,740. Also corrected against the code and .github/workflows/ci.yml: * the 100% coverage claim ("across all six 64-bit targets and three OSes"): the gate runs on ubuntu + macOS only, per-PR arch lanes are amd64/arm64 without coverage, the exotic four are nightly QEMU + scheduled hardware * the bound-module count, 181 -> 178 modules across 176 orgs, with the seven bound outside the go-ruby-* convention named * Set (in-tree over object.Hash, not go-composites/set), BigDecimal (go-ruby-bigdecimal, not go-composites/bigfloat) and Date (go-ruby-date, not go-composites/date) * the phase badge: plan-rbgo.md marks Phase 6 "substantially landed", not done Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent d498ddc commit f12d0e5

1 file changed

Lines changed: 102 additions & 17 deletions

File tree

‎README.md‎

Lines changed: 102 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
[![Docs](https://img.shields.io/badge/docs-mkdocs--material-9B1C2E)](https://go-embedded-ruby.github.io/docs/)
66
[![License](https://img.shields.io/badge/license-BSD--3--Clause-blue)](LICENSE)
77
[![Go](https://img.shields.io/badge/go-1.26.4%2B-00ADD8)](https://go.dev/dl/)
8-
[![Phase](https://img.shields.io/badge/phases-0--7%20done%20%C2%B7%208%20active-1a7f37)](https://go-embedded-ruby.github.io/docs/roadmap/)
8+
[![Phase](https://img.shields.io/badge/phases-6%20%2B%208%20in%20progress-1a7f37)](https://go-embedded-ruby.github.io/docs/roadmap/)
99

1010
**A pure-Go implementation of Ruby — one static binary, full dynamism, zero cgo.**
1111

@@ -37,7 +37,9 @@ reusable, tested and 6-arch by any Go program — no interpreter required.
3737

3838
The `go-ruby-*` family is a growing set of **standalone pure-Go modules — all
3939
CI-green, 100% coverage, 6-arch** — each its own org. The `go.mod` currently
40-
**binds 181 of them into rbgo** as native modules:
40+
**binds 178 such modules across 176 orgs** into rbgo as native modules (count:
41+
`grep -oE 'github\.com/go-ruby-[a-z0-9-]*/[a-z0-9-]*' go.mod | sort -u | wc -l`;
42+
`go-ruby-widgets` contributes three):
4143

4244
| Library | Role | Org · landing |
4345
| --- | --- | --- |
@@ -73,11 +75,17 @@ The full bound family (alphabetical, all native modules in `go.mod`):
7375
`railties`, `rake`, `ransack`, `rdoc`, `redis`, `regexp`, `reline`, `resolv`,
7476
`resque`, `rexml`, `roda`, `rolify`, `rouge`, `rqrcode`, `rspec`, `rss`,
7577
`rubocop`, `rubygems`, `saml`, `sass`, `scanf`, `securerandom`,
76-
`semantic-puppet`, `sequel`, `set`, `shellwords`, `shrine`, `sidekiq`,
78+
`semantic-puppet`, `sequel`, `shellwords`, `shrine`, `sidekiq`,
7779
`simplecov`, `sinatra`, `slim`, `sodium`, `sqlite3`, `strscan`, `thor`,
7880
`timecop`, `toml`, `tsort`, `typhoeus`, `tzinfo`, `unicode-normalize`, `uri`,
79-
`vcr`, `warden`, `webauthn`, `webmock`, `webrick`, `widgets`, `xslt`, `yaml`,
80-
`zeitwerk`, `zlib` — each at `github.com/go-ruby-<name>/<name>`.
81+
`vcr`, `warden`, `webauthn`, `webmock`, `webrick`, `widgets`, `yaml`,
82+
`zeitwerk`, `zlib`, plus `activeldap`, `ldap` and `fast-gettext-locale` — each at
83+
`github.com/go-ruby-<name>/<name>`.
84+
85+
Seven more are bound from **outside** the `go-ruby-*` naming convention, at
86+
`github.com/go-<name>/<name>`: `commonmark`, `kramdown`, `liquid`, `mustache`,
87+
`nokogiri`, `rouge` and `xslt`. (`set` is not bound at all — see *Set* under
88+
*Supported today*: it is implemented in-tree.)
8189

8290
Beyond the `go-ruby-*` family, the scientific / container stack binds the
8391
pure-Go [go-ndarray](https://github.com/go-ndarray/ndarray),
@@ -99,13 +107,70 @@ GUI toolkit), `require "tui"` (terminal-cell toolkit) and `require "mvvm"`
99107

100108
## Status
101109

102-
### Conformance & performance (campaign final, 2026-06-27)
110+
### Runtime conformance — ruby/spec (measured 2026-09-23)
111+
112+
Since July the work has been **runtime** conformance: making the VM behave as
113+
Ruby does, measured against [ruby/spec](https://github.com/ruby/spec)'s
114+
`language/` and `core/` suites. Measured on `main` (68cb53a) against the pinned
115+
corpus at `SPEC_SHA=87b1631992bd00cf0c4934474766d54dad088191`, darwin/arm64,
116+
`scripts/conformance/rubyspec/run.sh`:
117+
118+
| | |
119+
| --- | --- |
120+
| **passing examples** | **21,901** (four consecutive runs: 21 845 · 21 903 · 21 901 · 21 902) |
121+
| fail / error | 1 349 / 794 |
122+
| skipped | 473 |
123+
| pass rate of examples that ran | **91.1 %** (21 901 of 24 044) |
124+
| spec files | 2 191 of 2 206 produce a result; 15 produce none |
125+
| **CI floor** (`FLOOR`) | **21 740** |
126+
127+
**The floor is not the score.** `FLOOR` says *"no run may come in below this"* —
128+
it is a shrink-only ratchet, raised deliberately in its own PR after a win. The
129+
measured total normally sits **above** it; today the gap is ~160 examples. Quoting
130+
the floor as the conformance figure understates it, and quoting the measured total
131+
as a guarantee overstates it.
132+
133+
**The run-to-run spread is a known bug, not noise in the method.** The 58-example
134+
range above is one file, `core/module/autoload_spec.rb`, which crashes on roughly
135+
one run in four popping a frame ([#615](https://github.com/go-embedded-ruby/ruby/issues/615));
136+
when it does, the whole file's examples are lost. Take the *low* run as the
137+
guaranteed figure.
138+
139+
> **Not every gain is VM conformance.** Two of the recent jumps came from fixing
140+
> the **measurement**, not the interpreter, and it is worth keeping them apart:
141+
>
142+
> - **The mspec shim** ([#624](https://github.com/go-embedded-ruby/ruby/pull/624),
143+
> refs [#621](https://github.com/go-embedded-ruby/ruby/issues/621)) ran each
144+
> example under `instance_eval`, so a helper defined in an outer `describe` was
145+
> unreachable from a nested one. `core/string/valid_encoding/utf_8_spec.rb`
146+
> scored **0 of 28 — and 0 of 28 under MRI 4.0.5 too**, through the same shim.
147+
> Fixing the shim moved the corpus 20 905 → 20 936, of which **28 are
148+
> attributable**.
149+
> - **The parser upgrade to v0.2.0**
150+
> ([#625](https://github.com/go-embedded-ruby/ruby/pull/625)) made **13
151+
> `language/*_spec.rb` files parseable that had never parsed at all**. They had
152+
> been contributing zero to *both* columns — not failing, invisible. `language/`
153+
> went 1 443 → 1 776 passing (+333), while `fail+error` rose 283 → 436, because
154+
> those files brought their own failures with them.
155+
>
156+
> Both are gains in what the measurement can **see**, not in what the VM can
157+
> **do**. The rest of the climb — the floor went from **6 000** when the ratchet
158+
> landed on 2026-08-03 ([#263](https://github.com/go-embedded-ruby/ruby/pull/263))
159+
> to **21 740** today, across 26 conformance waves — is the VM.
160+
161+
There is no honest denominator for "percent of Ruby". 91.1 % is the share of the
162+
examples *this shim actually ran*; the shim is not mspec, it stubs some matchers
163+
(so their examples are **skipped**, not passed), and 473 skips and 15 unreadable
164+
files sit outside the ratio entirely.
165+
166+
### Front-end conformance & performance (campaign final, 2026-06-27)
103167

104168
A major conformance + performance campaign measured rbgo's pure-Go **front-end
105169
(parse + compile)** against the MRI 4.0.5 oracle on the two largest real-world
106170
Ruby codebases and a suite of popular libraries. All figures are
107171
**front-end acceptance**, not whole-application execution (see the honesty note
108-
below).
172+
below). **These figures date from 2026-06-27 and have not been re-measured
173+
since**; reproduce them with `scripts/conformance/heavyweight/sweep.sh`.
109174

110175
- **Parse: 99.96 % of Rails + Puppet** (5577 / 5579 `.rb` files). The only 2
111176
misses are intentional syntax-error fixtures **MRI itself rejects** — i.e.
@@ -368,20 +433,22 @@ JRuby**):
368433
with **no cgo / no FFTW**, returning `Complex` spectra.
369434
- **Set:** Ruby's `Set` — `new`/`[]`, `add`(`<<`)/`add?`/`delete`/`merge`/`clear`,
370435
`include?`/`member?`/`size`/`length`/`count`/`empty?`, `subset?`/`superset?`,
371-
`union`(`|`)/`intersection`(`&`)/`difference`(`-`), `each`/`to_a`/`to_set` —
372-
binding the pure-Go
373-
[go-composites/set](https://github.com/go-composites/set) library.
436+
`union`(`|`)/`intersection`(`&`)/`difference`(`-`), `each`/`to_a`/`to_set`.
437+
Implemented **in-tree** (`internal/vm/set.go` + the prelude) as a shell over
438+
rbgo's own `object.Hash`, exactly as MRI's `set.rb` is Hash-backed, so members
439+
key by the same `#hash`/`#eql?` semantics Hash keys use.
374440
- **Time:** Ruby's `Time` — `now`/`at`/`parse`, arithmetic (`+`/`-`/`<=>`),
375441
`strftime`/`strptime`, `year`/`month`/`mday`/`hour`/`min`/`sec`/`wday`, weekday
376442
predicates (`monday?`…`sunday?`), `utc`/`getutc`/`zone`, `to_i`/`to_f` —
377443
binding [go-composites/time](https://github.com/go-composites/time).
378444
- **Date:** Ruby's `Date` — `new`/`parse`, `+`/`-`/`<<`/`>>` and
379445
`next_day`/`prev_day`/`next_month`/`prev_month`,
380446
`year`/`month`/`mday`/`wday`/`yday`/`cwday`, `leap?`, comparisons — binding
381-
[go-composites/date](https://github.com/go-composites/date).
447+
[go-ruby-date](https://github.com/go-ruby-date/date).
382448
- **BigDecimal:** arbitrary-precision decimal — `+ - * / **`,
383449
`sqrt`/`abs`/`ceil`/`floor`/`round`/`pow`, `to_f`/`to_i`, `zero?`, comparisons
384-
— binding [go-composites/bigfloat](https://github.com/go-composites/bigfloat).
450+
— binding
451+
[go-ruby-bigdecimal](https://github.com/go-ruby-bigdecimal/bigdecimal).
385452
- **Bag:** a multiset / counter (element → multiplicity) — `add`(`<<`)/`delete`,
386453
`count`/`size`/`distinct`/`most_common`, `union`/`difference`/`intersection`,
387454
`include?`/`each`/`to_a` — binding
@@ -472,8 +539,16 @@ throughput tracks the underlying driver rather than the interpreter:
472539
the page's DOM/Canvas through the built-in `JS` module
473540
(`JS.document`/`JS.log`/`JS::Ref#call`/`JS.raf`).
474541

475-
**100% coverage** is enforced in CI across all six 64-bit targets (amd64, arm64,
476-
riscv64, loong64, ppc64le, s390x) and three OSes. Phase 8 (conformance and
542+
**100% coverage** is enforced in CI on the two **POSIX** lanes — ubuntu and
543+
macOS. The full `-race` suite also runs on **windows**, but the coverage gate is
544+
not applied there: a few POSIX-only paths (opening `/dev/null` as a character
545+
device) are unreachable, so its measured coverage is structurally below 100 %.
546+
Per-PR architecture cover is the two **native** lanes, `amd64` and `arm64`, which
547+
run `go test ./...` (not the coverage gate); the four exotic 64-bit targets
548+
(`riscv64`/`loong64`/`ppc64le`/`s390x`) are validated **off** the per-PR path —
549+
nightly under QEMU and on scheduled real hardware. A `wasip1/wasm` lane builds and
550+
runs the interpreter under wazero. `.github/workflows/ci.yml` is the authority.
551+
Phase 8 (conformance and
477552
representation/perf tuning) is well advanced: the conformance campaign above
478553
brought the front-end to ~100 % parse / 99.82 % parse+compile on real-world Ruby,
479554
and on the performance side small-integer interning and capture-tracked
@@ -613,13 +688,23 @@ runs the ruby/spec **language + core** suites through rbgo under a minimal
613688
MSpec-compatible shim and sums the passing examples; a per-PR CI lane fails if
614689
that total drops below the frozen floor in
615690
[`FLOOR`](scripts/conformance/rubyspec/FLOOR), so language + core conformance is
616-
measured on every change and can only go up (**15,400+** passing examples and
617-
climbing). It complements the differential oracle below: the ratchet is the
618-
absolute floor, the oracle catches divergences the specs don't cover.
691+
measured on every change and can only go up. It complements the differential
692+
oracle below: the ratchet is the absolute floor, the oracle catches divergences
693+
the specs don't cover.
694+
695+
**The floor and the measurement are two different numbers.** `FLOOR` is
696+
**21 740** — the level CI refuses to fall below. A run on `main` today measures
697+
**21 901** passing (low run of four: 21 845), i.e. normally ~160 above the floor;
698+
the floor is raised to meet it in a deliberate PR after each wave. See
699+
*Runtime conformance* above for the full breakdown, the run-to-run spread
700+
([#615](https://github.com/go-embedded-ruby/ruby/issues/615)) and what the ratio
701+
does and does not mean.
619702

620703
```bash
621704
scripts/conformance/rubyspec/run.sh # measure vs the floor
622705
UPDATE_FLOOR=1 scripts/conformance/rubyspec/run.sh # print the new floor after a win
706+
# point it at an existing corpus instead of cloning:
707+
SPECDIR=/path/to/ruby-spec CACHE=/path/to/ruby-spec scripts/conformance/rubyspec/run.sh
623708
```
624709

625710
### CI layering

0 commit comments

Comments
 (0)