Skip to content

Regexp linear_time?/quote/\u encoding + dup named_captures; Lazy#each value packing - #577

Merged
tannevaled merged 3 commits into
mainfrom
wave15/rxenum
Sep 6, 2026
Merged

tannevaled merged 3 commits into
mainfrom
wave15/rxenum

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

What

MRI-4.0.6 conformance for core/regexp and core/enumerator, edited only in
internal/vm/regexp.go and internal/vm/lazy.go (plus new _test.go).

Regexp

  • Regexp.linear_time?(re_or_str[, flags]) — new singleton method. A pattern
    matches in linear time unless it uses a back-reference (\1…\9, \k<name>)
    or a subexpression call (\g<name>), which is exactly the go-ruby-regexp
    engine's own memoisation gate (HasBackref / HasCall). A String argument is
    compiled (RegexpError on a bad pattern); a Regexp argument ignores flags and
    warns flags ignored. Ref: MRI re.c rb_reg_s_linear_time_p.
  • Regexp#named_captures — a repeated capture name now collects every group
    index it owns instead of keeping only the last, e.g.
    /(?<is>.)(?<pat>.)(?<is>.)/.named_captures == {"is"=>[1,3], "pat"=>[2]}.
    Ref: re.c named-capture map.
  • Regexp#encoding / #fixed_encoding? — a \u escape resolving to a
    non-ASCII code point now fixes the pattern to UTF-8 even when the source bytes
    are all ASCII (/\u{9879}/.encoding == UTF-8, fixed_encoding? == true).
    Ref: re.c unescape_unicode → rb_utf8_encoding.
  • Regexp.quote / .escape — the result now carries MRI's encoding tag:
    US-ASCII when the operand is all ASCII, otherwise the operand's own encoding
    (BINARY for an ASCII-8BIT input). Ref: re.c rb_reg_quote.

Enumerator / Lazy

  • Enumerator::Lazy#each value packing — a bare lazy now forwards a
    multi-value source yield to the block by ordinary arity ({ |x| } sees the
    first value, { |*a| } the whole tuple), while a lazy that has run any op
    packs a multi-value yield into a single Array element, matching MRI's
    rb_enum_values_pack. The pipeline already threaded the raw arguments
    alongside the gathered value; the terminal #each was dropping them.

Measured deltas (ruby/spec corpus)

  • core/regexp: pass 168 → 185 (+17), fail 72 → 67, error 18 → 6
    (linear_time +12, quote +2, named_captures +1, encoding +1, fixed_encoding +1).
  • core/enumerator: each_spec pass 22 → 24 (+2) (the lazy each fix);
    full core/enumerator/lazy suite re-measured with no regressions.

Out of reach from this file set (noted, not attempted)

The u / s / e / n regexp literal modifiers are dropped by the
go-ruby-parser lexer (lexer.go: only i/m/x/o are recorded) before
they ever reach regexp.go, so the options_spec FIXEDENCODING/NOENCODING
cases, the //nixm inspect ordering, == for /x/u vs /x/n, and the
u/s/e/n rows of compile/new/encoding/fixed_encoding cannot be
fixed here — they need a change in the parser dependency, which is outside this
agent's file set.

Tests

Whitebox _test.go (package vm) cover every new branch — including the
malformed-\u and trailing-backslash paths a compilable Regexp can never reach,
and all three arms of the lazy value-packing rule. go build ./...,
go test ./internal/vm/, gofmt, and the wasip1/wasm build are all green;
100% coverage on the new/changed code in regexp.go and lazy.go. Every
expectation was checked byte-for-byte against MRI 4.0.5.

🤖 Generated with Claude Code

tannevaled and others added 3 commits September 6, 2026 14:14
…t build)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reachable core/regexp conformance wins (parser-blocked u/s/e/n literal
modifiers left aside — the go-ruby-parser lexer drops them before they
reach this package):

- Regexp.linear_time?(re_or_str[, flags]): new singleton method. Linear
  unless the pattern uses a back-reference (\1…\9, \k<>) or a
  subexpression call (\g<>), which is exactly the go-ruby-regexp engine's
  own memoisation gate (HasBackref/HasCall). A String arg is compiled
  (RegexpError on a bad pattern); a Regexp arg ignores flags and warns
  "flags ignored" (ref: MRI rb_reg_s_linear_time_p; re.c). +12 (was all
  errors: undefined method).

- Regexp#named_captures: accumulate every group index a repeated name
  owns instead of overwriting — /(?<is>.)(?<pat>.)(?<is>.)/.named_captures
  == {"is"=>[1,3],"pat"=>[2]} (ref: re.c named_captures). +1.

- Regexp#encoding / #fixed_encoding?: a \u escape resolving to a
  non-ASCII code point fixes the pattern to UTF-8 even when the source
  bytes are all ASCII (/\u{9879}/.encoding == UTF-8, fixed_encoding? ==
  true), matching re.c's unescape_unicode → rb_utf8_encoding. +2
  (encoding_spec, fixed_encoding_spec).

- Regexp.quote / .escape: tag the result encoding as MRI's rb_reg_quote
  does — US-ASCII when the operand is all ASCII, otherwise the operand's
  own encoding (BINARY for an ASCII-8BIT input). +2 (quote_spec 7/7).

Whitebox tests cover every new branch, including the malformed-\u and
trailing-backslash paths a compilable Regexp can never reach. Every
expectation checked byte-for-byte against MRI 4.0.5.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Enumerator::Lazy#each mishandled a source that yields several values at
once. The lazy pipeline already threads the raw arguments of a multi-value
yield (`multi`) alongside the gathered value, but the terminal #each sink
dropped them, so it always passed the packed Array as a single block
argument.

MRI's rule (rb_enum_values_pack, confirmed against ruby 4.0.5):
- a bare lazy (no ops) forwards the yielded values to the block by
  ordinary arity, so `{ |x| }` sees the first value and `{ |*a| }` the
  whole tuple;
- once any op has run, a multi-value yield is packed into one Array
  element, which the block then receives as its single value.

lazyRun's sink now carries `multi`, and #each spreads it only for a bare
lazy (len(ops)==0); an op-bearing pipeline keeps the packed value. Fixes
core/enumerator/each_spec (lazy variant) and the enumerable_value_packing
shared examples across the lazy select/reject/drop/take/uniq/compact
suites, with no regressions.

core/enumerator/each_spec: pass 22 -> 24.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit a4dcc51 into main Sep 6, 2026
7 checks passed
@tannevaled
tannevaled deleted the wave15/rxenum branch September 6, 2026 13:34
tannevaled added a commit that referenced this pull request Sep 6, 2026
…578)

Wave-15 merged three disjoint-file, bibliography-first conformance PRs:
- #576 core/io: fd-based IO constructors (IO.open/new/for_fd + mode/encoding/
  binmode/autoclose resolution per io.c) — core/io +107.
- #575 core/kernel: Kernel#caller_locations (unblocks the raise_spec collection
  cascade), caller start/length/range args, raise #exception protocol + empty
  bare-raise message per MRI — core/kernel +29.
- #577 core/regexp+enumerator: Regexp.linear_time?, quote/escape result
  encoding, \u-escape fixed_encoding, named_captures dup names, Lazy value
  packing — core/regexp +17, enumerator +2.

Measured total on merged main: 19179 passing (files loaded 2171, +7 that
previously failed to load). Local measure at TIMEOUT=25; CI runs TIMEOUT=60 so
its total is >= this floor.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant