Repository navigation
feat(lexer): record regexp encoding flags n/u/e/s - #23
Merged
Merged
Conversation
The regexp trailing-flag loop previously collected only i/m/x/o and
silently discarded the encoding modifiers n/u/e/s, which blocked rbgo
regexp encoding conformance. MRI records an ARG_ENCODING for each of
these (rb_char_to_option_kcode in re.c): n=ASCII-8BIT, u=UTF-8,
e=EUC-JP, s=Windows-31J. Append them to Flags in source order for both
the /…/ (lexRegexp) and %r{…} (lexPercentRXS) forms; any other trailing
lowercase letter is still consumed harmlessly.
Update the Flags doc comments on token.Token and ast.RegexpLit, and
replace the old TestRegexpEncodingFlagsStillDropped with
TestRegexpEncodingFlagsRecorded (plus a %r variant).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The regexp trailing-flag loop recorded only
i/m/x/oand silently discarded the encoding modifiersn/u/e/s. That dropped information is exactly what rbgo needs to build aRegexpwith the right encoding, and it has blocked regexp encoding conformance across two waves.This records
n/u/e/sintoFlagsin source order, for both regexp forms:/…/flags—lexRegexp%r{…}flags—lexPercentRXSAny other trailing lowercase letter is still consumed harmlessly (unchanged behavior).
MRI reference
MRI records an
ARG_ENCODINGfor each encoding letter (rb_char_to_option_kcodeinre.c, driven fromregx_optionsinparse.y):nARG_ENCODING_NONEuARG_ENCODING_FIXEDeARG_ENCODING_FIXEDsARG_ENCODING_FIXEDi/m/xremain the onig option bits andoremains "interpolate once"; these are unchanged.Changes
lexer/lexer.go: appendn/u/e/sin both trailing-flag loops (lexRegexpandlexPercentRXS); doc comment updated.token/token.go,ast/ast.go:Flagsdoc comments now note the encoding flags and source-order semantics.percent_rxs_test.go:TestRegexpEncodingFlagsStillDropped→TestRegexpEncodingFlagsRecorded(asserts/x/n→n,/x/u→u,/x/e→e,/x/s→s, and order for/a/mn,/a/un); plusTestPercentRegexpEncodingFlagsRecordedlocking the%r{…}path.Verification
GOWORK=off go build ./...,go vet ./..., andgo test ./...all green; the four touched files aregofmt-clean.🤖 Generated with Claude Code