Skip to content

Enumerator::Lazy: pull an Enumerator source rather than read it whole - #418

Merged
tannevaled merged 1 commit into
mainfrom
fix/lazy-over-a-generator
Aug 21, 2026
Merged

tannevaled merged 1 commit into
mainfrom
fix/lazy-over-a-generator

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

lazySource walked a Range by counter and an Array by index, and materialised anything else — "so it must be finite", as its own comment said.

An Enumerator.new { |y| loop { … } } is not finite. So lazy was defeated at its own source: s.lazy.first(3) allocated 8 GB and killed a CI runner. This is the fourth and last of the reasons rubyspec-ratchet was red (#412 is the watcher that found them; #415, #416 and #417 are the other three).

It is pulled now

Through the fiber behind Enumerator#next — machinery that already existed and already handled an endless generator: s.next three times costs 79 MB. Only the lazy path was not using it.

before after
s.lazy.first(3) 7.4 GB 79 MB
s.lazy.take(3).force 8.0 GB 79 MB
s.lazy.take(3).flat_map{}.force 12.7 GB 80 MB

core/enumerator/lazy/flat_map_spec.rb goes from 48.2 GB to 78 MB, and now reports a result instead of dying.

Why the pull runs on a copy

The copy carries the enumerator's definition and none of its external-iteration state. That is what makes a terminal operation start from the beginning every time, and leaves the caller's own #next cursor where it was.

Both were checked against ruby 3, which agrees line for line — including the interleaved case that would catch a shared cursor:

t = Enumerator.new { |y| y << 1; y << 2; y << 3 }
t.next        # 1
t.lazy.first(2)  # [1, 2]   ← from the beginning
t.next        # 2          ← the cursor did not move

A pipeline that stops early leaves that fiber suspended, exactly as an abandoned #next does. That is the existing behaviour of external iteration here, not something this adds, and the comment says so.

Full internal/vm suite green.

🤖 Generated with Claude Code

lazySource walked a Range by counter and an Array by index, and
materialised anything else — "so it must be finite", as its own comment
said. An Enumerator.new { |y| loop { … } } is not finite, so lazy was
defeated at its source: s.lazy.first(3) allocated 8 GB and killed a CI
runner. It is the fourth and last of the reasons the rubyspec-ratchet
lane was red.

It is pulled now, through the fiber behind Enumerator#next — machinery
that already existed and already handled an endless generator: s.next
three times costs 79 MB. Only the lazy path was not using it.

  s.lazy.first(3)                     before 7.4 GB   after 79 MB
  s.lazy.take(3).force                before 8.0 GB   after 79 MB
  s.lazy.take(3).flat_map{}.force     before 12.7 GB  after 80 MB

core/enumerator/lazy/flat_map_spec.rb goes from 48.2 GB to 78 MB.

The pull runs on a copy of the enumerator, carrying its definition and
none of its external-iteration state. That is what makes a terminal
operation start from the beginning every time, and leaves the caller's
own #next cursor where it was — both checked against ruby 3, which
agrees line for line, including next/lazy/next interleaved.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant