Skip to content

docs: four published claims the last twelve pull requests falsified - #938

Merged
tannevaled merged 1 commit into
mainfrom
docs-measured-reality
Nov 15, 2026
Merged

tannevaled merged 1 commit into
mainfrom
docs-measured-reality

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

docs: four published claims the last twelve pull requests falsified

Every figure here was already measured and merged. None of it needed a new run —
what it needed was someone to ask which published sentences the merges had made
false, which is a different question from "what did we fix", and nobody asks it
on the way out of a merge.

The four

1. library/socket 8 killed → 7, and the headline 75 → 9 is 75 → 8.
#935 measured it in so many words: "nine files killed by the clock across
library/socket + library/timeout before, eight after"
. The README kept the
old row for a day.

2. A figure withdrawn rather than adjusted. The README reported the remaining
hangs as "275 passing against MRI's 371". That control scored 7 on those
190 files, because library/socket/spec_helper.rb calls MSpec.enable_feature
and the conformance shim defined no MSpec (#936). A number counted against a
broken control is not a smaller number — it is not a number. It is removed, with
the reason in its place.

3. The corpus map was compiled while its largest module read zero. The
"the rest is concentrated: matrix 233, bigdecimal 180, stringscanner 154…" list
was written while library/socket contributed a flat zero on both sides.
Repaired, socket is MRI 1790 / rbgo 224 — about −1560, larger than every
module on that list put together (#937), and it is unimplemented surface rather
than hangs.

⛔ The two totals are deliberately not summed. They come from different runs
against different MRI patch releases (4.0.7 and 4.0.6), and a sum across two
measurements is how two errors cancel. The published gap is marked a floor
instead, which is a weaker claim that is actually true.

4. docs/SHARED-ENGINES.md told readers the opposite of the outcome. Its
net-ftp section still concluded "#916 is not closed, because the measurement
did not move … still 39 killed"
, and that net-ftp was not available for
extraction. True on 2026-11-13; the inverse of what happened. #927 released the
GVL on the data connection — dialData, acceptData, sendData,
retrlines, retrbinary — and #928 added Net::FTP.default_passive, which was
what had kept ruby/spec's active-mode fixture from ever reaching that path.
Re-measured: 0 killed, −128.

That section now also records why the inference was wrong, because the wrong
inference is the reusable part: "the fix did not move the measurement" was read
as "the fix is incomplete"
, when the mechanism was simply wired at only some of
its doors. The extraction blocker is lifted, and the −128 the new home inherits
is named — the same rule this document already applies to digest and
securerandom.

Why no gate caught any of this

Seven README drift gates run in this repository's root package, and all seven are
green before and after this change: the conformance total against BASELINE,
the Ruby version against the runtime, the Go floor, the numeric surface, the wasm
lane, and the shim line numbers the prose cites.

Not one of them looks at a figure in a dated prose table, and that is not an
oversight — there is nothing to compare such a figure against, because the run
that produced it is gone. The honest response is the one already in the README:
the dated map says of itself that it is "one dated run, and not a gate", and the
only number CI enforces is per-file.

The gates did catch what they can catch. When a 22-line insertion moved two
README-cited spec_helper.rb line numbers, every lane went red for a change that
touched one Ruby file.

Checks

  • gotest -run '<the seven README gates>' . — all seven pass, each alternative
    matched (gotest fails and names any that match nothing).
  • No Go files are touched.
  • The landing and the docs site carry the same falsified claims and are not
    fixed here: one surface per pull request. Both are in flight separately —
    layouts/_default/list.html on the landing states the inverted headline
    ("the largest single debt is a hang rather than a missing feature"), and
    content/en/conformance.md on the docs site repeats the net-ftp figures.

Refs #916, #933, #935, #936, #937, #927, #928

(#916 and #933 stay open: eight files still hang. #927, #928, #935 and #936 are
the merged pull requests whose outcomes these documents had not caught up with.
#937 stays open and is the one this change makes visible.)

🤖 Generated with Claude Code

Every figure here was already measured and merged; none of it needed a new run.
What it needed was someone to ask which published sentences the merges had made
false, which is a different question from "what did we fix".

## The four

1. `library/socket` **8 killed -> 7**, and the headline **75 -> 9** is **75 -> 8**.
   #935 measured it in the sentence "nine files killed by the clock across
   `library/socket` + `library/timeout` before, eight after".

2. A withdrawn figure. The README said the remaining hangs were "275 passing
   against MRI's 371". That control scored **7** on those 190 files, because
   `library/socket/spec_helper.rb` calls `MSpec.enable_feature` and the shim
   defined no `MSpec` (#936). A number counted against a broken control is not a
   smaller number, it is not a number; it is removed rather than adjusted.

3. The corpus map's "the rest is concentrated: matrix 233, bigdecimal 180, ..."
   list was compiled while `library/socket` contributed a flat **zero on both
   sides**. Repaired, socket is MRI 1790 / rbgo 224 -- about **-1560**, larger
   than every module on that list put together (#937). The two totals are
   deliberately **not summed**: different runs, different MRI patch releases
   (4.0.7 and 4.0.6), and a sum across two measurements is how two errors cancel.
   The published gap is marked a **floor** instead.

4. `docs/SHARED-ENGINES.md` still concluded, at the top of its `net-ftp` section,
   that **"#916 is not closed, because the measurement did not move ... still 39
   killed"**, and told the reader `net-ftp` was not available for extraction.
   That was true on 2026-11-13 and is the opposite of the outcome: #927 released
   the GVL on the **data** connection and #928 added `Net::FTP.default_passive`,
   and net-ftp is **0 killed, -128**. The section now records why the inference
   was wrong -- "the fix did not move the measurement" was read as "the fix is
   incomplete" when the mechanism was wired at only some of its doors -- and
   lifts the extraction blocker while naming the -128 the new home inherits.

## Why a doc gate did not catch any of them

Seven README drift gates run in this repository's root package and all seven are
green both before and after this change: they check the conformance total
against `BASELINE`, the Ruby version against the runtime, the Go floor, the
numeric surface, the wasm lane, and the shim line numbers the prose cites. Not
one of them looks at a figure in a dated prose table, because there is nothing
to compare such a figure against -- the run that produced it is gone.

That is the honest limit, and it is stated in the README rather than papered
over: the dated map is explicitly "not a gate", and the only number CI enforces
is per-file. The gates caught the thing they can catch -- when a 22-line
insertion moved two cited line numbers, every lane went red.

Refs #916, #933, #935, #936, #937

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant