Reduce memory allocations for RI and POT generators

[skatkov]

Each documented method currently retains an array of RDoc::Parser::RubyColorizer::ColoredToken objects and their fragmented source strings.

This data is unnecessary for:

• RI, whose serialized method representation excludes token_stream
• POT, which extracts translatable documentation comments
• Coverage reports, which generate no method-source pages

Large generated SDKs can contain millions of tokens. Avoiding those allocations substantially reduces parsing time and memory without changing generated output.

Implementation

Generators can declare whether they require method source:

class RDoc::Generator::RI
  def self.store_method_source? = false
end

class RDoc::Generator::POT
  def self.store_method_source? = false
end

The Ruby parser checks that capability before syntax highlighting a method body.

Generators without the capability default to true, preserving compatibility with existing third-party generators.

This also allows third-party generators (like rdoc-markdown) to skip this costly process if they don't have need for method.token_stream or method.markup_code.

If store_method_source? was disabled, but token_stream was still accessed, an error will be raised.

Benchmark

Here is a code that was used for benchmark this fix:
https://github.com/skatkov/rdoc-store-method-source-benchmark

RDoc 8.0.0 and google-api-client 0.53.0 on Ruby 4.0.6:

Variant	Parse RSS	Peak RSS	Colored tokens	Live heap slots	Time
Baseline	1,069.4 MiB	1,508.4 MiB	3,900,930	10,031,484	61.72s
Optimized	463.6 MiB	730.0 MiB	0	2,229,446	51.62s

• Parse RSS reduction: 56.6%
• Peak RSS reduction: 51.6%
• Generated RI output: identical

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[tompng]

We need this respond_to? for custom generators 👍
But can we define store_method_source? = true for all existing generators in this repository?

[tompng]

RDoc::RubyGemsHook#generate parses files once and then generates both formats from the same store, swapping the generator afterwards (simplified):

class RDoc::RubyGemsHook
  def generate
    @rdoc.store = RDoc::Store.new(parse_options)
    @rdoc.parse_files parse_options.files

    document 'ri',    options, @ri_dir   if @generate_ri   and (@force or not File.exist? @ri_dir)
    document 'aliki', options, @rdoc_dir if @generate_rdoc and (@force or not File.exist? @rdoc_dir)
  end
end

parse_options takes its generator from the gem's spec.rdoc_options. So if a gem specifies --format=ri there, parsing skips method source tokens, and the subsequent aliki generation raises:

method source for Foo#bar was not stored; set store_method_source? to true

Reproduction: https://gist.github.com/tompng/5fe25eaeb6cb53164a3c49374d330914

Since the generator can be swapped after parsing like this, deriving the decision from options.generator at parse time is not reliable. I think store_method_source should be an explicit option that RubyGemsHook sets before parsing, e.g.:

parse_options.store_method_source = Generator::Aliki.store_method_source? || Generator::RI.store_method_source?
@rdoc.parse_files parse_options.files

(strictly, only the generators that will actually run need to be OR-ed)

def store_method_source?
  # return true or false if it's explicitly set
  return @store_method_source unless store_method_source.nil?
  # fallback path (@coveragage_report, @generator.store_method_source?)`
end

[skatkov]

Good catch!

I didn't know this was even possible. I'll take a closer look

[skatkov]

I'm closing this PR, suggested approach is not workable.

I'm leaning toward changing from eager code highlight approach to lazy one and working on a prototype in this PR:
#1792