Skip to content

ruby/spec: raise the floor, 14200 -> 14900 - #419

Merged
tannevaled merged 1 commit into
mainfrom
chore/raise-the-floor
Aug 21, 2026
Merged

tannevaled merged 1 commit into
mainfrom
chore/raise-the-floor

Conversation

@tannevaled

@tannevaled tannevaled commented Aug 21, 2026 •

Copy link
Copy Markdown
Contributor

The floor had not moved while the total climbed under it:

passing examples floor
17 Aug 14303 14200
18 Aug 14567, then 14625 14200
now 14949 – 14963 14200

A floor 750 below the measurement tolerates a regression of 750 examples in silence, which is the one thing a ratchet exists not to do.

Not pinned to the measurement, and I was wrong to propose that first

The first version of this PR set the floor to the exact total, arguing the total was deterministic because two runs of one commit both said 14878. Two more runs of the same tree then said 14949 and 14963 — fourteen apart. Two samples agreeing is not determinism, and that PR's own ratchet run then OOMed, which is what held it.

14900 is 49 below the lowest of those — absorbing a jitter observed at 14 — and 700 above where the floor sat.

Why the lane can be trusted to retake this measurement now

Five memory blowups are fixed (#415, #416, #417, #418, #421; #412 is the watcher that found them). The difference is not "it went green", it is what the watcher reads during a run:

lowest free memory
the runs that died, for a fortnight 214 MB
the first green run, one fix in 3005 MB
now, all five in 13236 MB of 15

The sweep no longer approaches the runner's memory at all.

If it ever reds by about 150

That is a heavy spec file failing to load, not a conformance regression — read failed to load in the log first. It is 74 now, and was 79 before this week; five more files load than did.

🤖 Generated with Claude Code

@tannevaled tannevaled changed the title ruby/spec: raise the floor to what is now measured, 14200 -> 14903 [HELD] ruby/spec: raise the floor to what is now measured, 14200 -> 14903 Aug 21, 2026
@tannevaled

Copy link
Copy Markdown
Contributor Author

Holding this. Its premise is wrong.

I argued the total was safe to pin tight because two runs of the same commit both reported 14878 to the example. That was two green samples, not determinism — the lane still OOMs, and this PR own ratchet run proved it: memory 13396 MB -> 214 MB, runner gone, same signature as before.

There is a fifth blowup, the largest of them, which I catalogued in the first sweep and never diagnosed: language/singleton_class_spec.rb, 61.3 GB on current main. Minimal reproduction, six lines:

o = Object.new
class << o
  CONST = self
end
class << o
  x = CONST     # 14 GB. ruby: instant
end

It is the constant read alone — not the comparison the spec makes, and not self (p self.class in the same body is 75 MB). Constant resolution inside a reopened singleton class body walks something unbounded.

A floor is only worth pinning to a measurement the lane can retake. This waits for that fix.

The floor had not moved while the total climbed under it: 14303 on
17 August, 14567 and 14625 on the 18th, and 14949-14963 now that the
five memory blowups are fixed and five more spec files load. A floor
750 below the measurement tolerates a regression of 750 examples in
silence, which is the one thing a ratchet exists not to do.

Not pinned to the measurement, and I was wrong to propose that the
first time. I argued the total was deterministic because two runs of one
commit both said 14878. Two more runs of the same tree then said 14949
and 14963 — fourteen apart. Two samples agreeing is not determinism.

14900 is 49 below the lowest of those, which absorbs a jitter observed
at 14, and 700 above where the floor sat.

What would drop it much further is a heavy spec file failing to load,
worth about 150 examples. That is now unlikely rather than routine: with
the blowups fixed, the sweep's lowest free memory is 13.2 GB of 15, where
a fortnight of runs died at 214 MB. If this lane ever reds by roughly
150, read "failed to load" in the log before looking for a conformance
regression — it is 74 now, and was 79 before this week.
@tannevaled
tannevaled force-pushed the chore/raise-the-floor branch from 5d0ecf5 to 3c09a0f Compare August 21, 2026 12:33
@tannevaled tannevaled changed the title [HELD] ruby/spec: raise the floor to what is now measured, 14200 -> 14903 ruby/spec: raise the floor, 14200 -> 14900 Aug 21, 2026
@tannevaled
tannevaled merged commit 92d189a into main Aug 21, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant