Skip to content

MatchData: support \u escapes and duplicate / non-ASCII capture names (MRI 4.0.6) - #565

Merged
tannevaled merged 2 commits into
mainfrom
feat/agent-matchdata
Sep 5, 2026
Merged

tannevaled merged 2 commits into
mainfrom
feat/agent-matchdata

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

MatchData: advance MRI 4.0.6 conformance (core/matchdata residuals)

The core/matchdata cluster carried ~27 ERRORs from a shared root cause that was
not in the MatchData methods at all: the regexp engine rejects three pieces of
valid Ruby (Onigmo) regexp syntax at compile time, so every example that
built such a pattern crashed with RegexpError before any MatchData method ran.

The three engine limitations, all handled now inside internal/vm/regexp.go:

  • \uHHHH / \u{…} escapes — translated to the literal characters they
    denote (an ASCII metacharacter is emitted backslash-escaped so it stays
    literal; an invalid escape is left for the engine, which raises RegexpError
    exactly as MRI does). Unblocked offset, byteoffset.
  • Duplicate capture names — Ruby lets several groups share a name. Each named
    group is rewritten to a unique ASCII synthetic name; name resolution then
    applies Ruby's rule (the highest-indexed group with that name that participated
    in the match). Unblocked named_captures, names, element_reference, and the
    begin/end/bytebegin/byteend name variants.
  • Non-ASCII capture names (e.g. (?<æ>…)) — same rewrite path, keyed on any
    non-ASCII byte in a name; \k<name> / \g<name> references (both <> and ''
    delimiters) are rewritten to match.

The rewrite is gated: it engages only when a source actually has duplicate or
non-ASCII names, and \u translation short-circuits when the source has no \u.
For every other pattern the engine source is byte-identical to before and the
name map stays nil, so ordinary patterns are entirely unaffected.

Result (core/matchdata)

183 pass, 0 error, 2 fail (was ~2 fail + 27 error). Every fixed example was
checked byte-for-byte against ruby 4.0.6.

Two residual failures are out of scope (they need changes outside
regexp.go): dup needs generic instance-variable storage on native-backed
objects (object_model.go/builtins.go), and one element_reference example is
an EUC-JP encoding-tracking gap.

Tests / checks

  • New internal/vm/matchdata_residuals_test.go (package vm_test) covers every
    new branch; all changed/added regexp.go functions are at 100.0% coverage.
  • go test ./internal/vm/ green, go test ./... green, GOOS=js GOARCH=wasm
    build OK, gofmt -l clean.

🤖 Generated with Claude Code

tannevaled and others added 2 commits September 5, 2026 21:18
…names

The regexp engine rejects Ruby's \uHHHH / \u{...} escapes and cannot compile
patterns whose capture groups share a name or carry a non-ASCII name — both
valid in Ruby (Onigmo). This crashed ~27 core/matchdata examples (begin, end,
offset, byte* variants, named_captures, names, element_reference) at compile
time before any MatchData method ran.

compileRegexp now translates \u escapes to the literal characters they denote
and, only when a source has duplicate or non-ASCII capture names, rewrites each
named group to a unique ASCII synthetic name (fixing up \k/\g references),
recording per original name the synthetic names in source order. Name
resolution goes through MatchData#indexOfName, which applies Ruby's shared-name
rule: the highest-indexed group with that name that participated in the match.
Ordinary patterns are untouched (nil nameMap, engine handles names directly).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…riter

Cover the new \u-escape translation and duplicate/non-ASCII capture-name
handling with black-box vm_test cases (every new branch exercised).

Also: reject an empty \u{} list (Ruby's "invalid Unicode list"), drop the
unreachable synthetic-name lookup guard in indexOfName, and simplify the
character-class scan in rewriteNamedGroups (Ruby forbids an empty class, so a
valid pattern never opens a class with ']').

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit e0520c3 into main Sep 5, 2026
7 checks passed
@tannevaled
tannevaled deleted the feat/agent-matchdata branch September 5, 2026 20:44
tannevaled added a commit that referenced this pull request Sep 5, 2026
Locks in #565 (MatchData / regexp-engine \u escapes + duplicate/non-ASCII
capture names), #566 (Method/UnboundMethod) and #567 (Array + pack). Measured
18767 passing on the pinned corpus, up from 18650.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant