Repository navigation
Enumerator::Lazy: pull an Enumerator source rather than read it whole - #418
Merged
Merged
Conversation
lazySource walked a Range by counter and an Array by index, and
materialised anything else — "so it must be finite", as its own comment
said. An Enumerator.new { |y| loop { … } } is not finite, so lazy was
defeated at its source: s.lazy.first(3) allocated 8 GB and killed a CI
runner. It is the fourth and last of the reasons the rubyspec-ratchet
lane was red.
It is pulled now, through the fiber behind Enumerator#next — machinery
that already existed and already handled an endless generator: s.next
three times costs 79 MB. Only the lazy path was not using it.
s.lazy.first(3) before 7.4 GB after 79 MB
s.lazy.take(3).force before 8.0 GB after 79 MB
s.lazy.take(3).flat_map{}.force before 12.7 GB after 80 MB
core/enumerator/lazy/flat_map_spec.rb goes from 48.2 GB to 78 MB.
The pull runs on a copy of the enumerator, carrying its definition and
none of its external-iteration state. That is what makes a terminal
operation start from the beginning every time, and leaves the caller's
own #next cursor where it was — both checked against ruby 3, which
agrees line for line, including next/lazy/next interleaved.
This was referenced Aug 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
lazySourcewalked aRangeby counter and anArrayby index, and materialised anything else — "so it must be finite", as its own comment said.An
Enumerator.new { |y| loop { … } }is not finite. So lazy was defeated at its own source:s.lazy.first(3)allocated 8 GB and killed a CI runner. This is the fourth and last of the reasonsrubyspec-ratchetwas red (#412 is the watcher that found them; #415, #416 and #417 are the other three).It is pulled now
Through the fiber behind
Enumerator#next— machinery that already existed and already handled an endless generator:s.nextthree times costs 79 MB. Only the lazy path was not using it.s.lazy.first(3)s.lazy.take(3).forces.lazy.take(3).flat_map{}.forcecore/enumerator/lazy/flat_map_spec.rbgoes from 48.2 GB to 78 MB, and now reports a result instead of dying.Why the pull runs on a copy
The copy carries the enumerator's definition and none of its external-iteration state. That is what makes a terminal operation start from the beginning every time, and leaves the caller's own
#nextcursor where it was.Both were checked against ruby 3, which agrees line for line — including the interleaved case that would catch a shared cursor:
A pipeline that stops early leaves that fiber suspended, exactly as an abandoned
#nextdoes. That is the existing behaviour of external iteration here, not something this adds, and the comment says so.Full
internal/vmsuite green.🤖 Generated with Claude Code