Repository navigation
fix(lexer): a backslash escapes the next character in a %w/%W/%i/%I word list - #25
Merged
Merged
Conversation
In a word list a backslash escapes the character after it, so that character
is an ordinary character of the current word: it never separates two words and
never opens or closes the list. The lexer was escape-blind, which cost three
distinct defects.
- An escaped space split the word. `%w[a\ b c]` lexed as three words
("a\", "b", "c") where MRI has two ("a b", "c"). ruby/spec's Dir fixtures
contain a directory literally named `special/test\ +()[]{}` written inside
a %w list, so every Dir spec that enumerates it saw a phantom `+()[]{}`
entry.
- An escaped delimiter ended the list. `%w[a\]b]` was a parse error, and
`%w[a\[b]` an "unterminated %-array literal".
- An escaped backslash was doubled: `%w[a\\b]` yielded `a\\b`, not `a\b`.
MRI's rule (parse.y, tokadd_string) follows from the backslash branch being
tested before the word-splitting and term/paren branches. In a
non-interpolating %w/%i the backslash is KEPT literally except before
whitespace, a delimiter, or another backslash, where it escapes a literal
character instead — so `%w[a\tb]` is the four characters a \ t b and not a
tab. In an interpolating %W/%I the full double-quote escape set applies, with
one exception: MRI's QWORDS test precedes its EXPAND one, so a backslash
before a newline is a literal newline rather than a line continuation.
Splitting moves from the parser (strings.Fields, which cannot see an escape)
to the lexer, which still knows the delimiters; the WORDS/SYMBOLS token gains
a Words field and keeps its raw body in Lit. %I words are now unescaped too,
so `%I[a\ b]` is :"a b" rather than the four characters a \ space b.
Every rule above was checked against the installed MRI (ruby 4.0.5) rather
than read off parse.y alone: 62 word-list sources were run through both `ruby`
and this parser and compared as raw bytes, all 62 identical. The three further
cases in that set are interpolating and fold to a StrInterp node instead of a
constant, and were checked by hand.
Four tables covering the non-interpolating (%w, %i) and interpolating (%W, %I) forms: escaped space, escaped delimiter for all six delimiter pairs, escaped backslash, escaped tab and newline, the backslashes %w keeps verbatim because it has no escape set, the double-quote escape set %W applies instead, and the unescaped whitespace and bracket nesting that must keep behaving as before. Every expected value is the value the installed MRI (ruby 4.0.5) prints for the same source; each was run through `ruby -e 'p <src>'` and compared as raw bytes, so a failure here is a divergence from Ruby rather than from taste. The MRI output is quoted beside each case. Also covers the scanner's new end-of-input branch: a list ending on a trailing backslash, or on an escaped terminator, is unterminated rather than accepted.
tannevaled
force-pushed
the
wave23/wsplit
branch
from
September 11, 2026 05:32
04be301 to
997e65a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The defect
In a
%w/%W/%i/%Iword list, a backslash escapes the character after it,so that character belongs to the current word: it never separates two words and
never opens or closes the list. The lexer was escape-blind, which cost three
distinct defects:
%w[a\ b c]["a b", "c"]["a\\", "b", "c"]— split, and the backslash kept%w[a\]b]["a]b"]%w[a\[b]["a[b"]unterminated %-array literal%w[a\\b]["a\\b"]["a\\\\b"]— the backslash doubled%I[a\tb][:"a\tb"][:"a\\tb"]— escapes never resolvedThe first row is the one that blocks specs. ruby/spec's
Dirfixtures contain adirectory literally named
special/test\ +()[]{}, written inside a%w[...]list. Splitting it fabricated a phantom
+()[]{}entry, which then poisonedDir.entries,Dir.children,Dir.each,Dir.foreach,Dir.readand theglob specs — roughly 15-20
core/dirspecs across three conformance waves.The rule
From MRI's
parse.y(tokadd_string), and it all follows from the backslashbranch being tested before the word-splitting and term/paren branches:
%w/%i(str_sword:QWORDS, noEXPAND, noESCAPE) keep thebackslash literally, except before whitespace, a delimiter, or another
backslash — there it escapes a literal character. So
%w[a\tb]is the fourcharacters
a \ t band not a tab, while%w[a\\b]is the threecharacters
a \ b.%W/%I(str_dword:QWORDS|EXPAND) apply the full double-quoteescape set, with one exception: MRI tests
STR_FUNC_QWORDSbeforeSTR_FUNC_EXPAND, so a backslash before a newline is a literal newline, notthe line continuation it would be inside a string.
The change
Word splitting moves from the parser to the lexer.
strings.Fieldscannot tella separating space from an escaped literal one, and the parser no longer knows
the delimiters needed to resolve
\]. TheWORDS/SYMBOLStoken gains aWordsfield and keeps its raw body inLit.lexPercentArray— a backslash consumes the next byte, so an escapeddelimiter neither nests nor terminates; ending on one is unterminated.
splitWordList(new) — splits and unescapes a%w/%ibody per the rule above.splitPercentWords— no longer splits on an escaped space; rewrites anescaped newline so the string decoder yields the literal newline MRI does.
unescapeInterpWord(new) — a%Iword with no interpolation now has itsescapes resolved instead of becoming a symbol of the raw text.
Verification
The rules were confirmed against the installed MRI (ruby 4.0.5) rather than read
off
parse.yalone. 65 word-list sources were run through bothrubyand thisparser and compared as raw bytes, so nothing is lost to formatting:
62 identical. The other three are interpolating cases that legitimately fold
to a
StrInterp/Callnode instead of a constant, and were checked by hand —including
%W[a\ b#{1}c], where the escaped space lands in the literal prefix"a b"beside a live interpolation.go build ./...,go vet ./...,go test ./...— all green.gofmt -lclean for every file touched.(
%w[a[b]c],%w(a(b)c), all six delimiter pairs).No existing test pinned the old splitting, so none had to be changed.
🤖 Generated with Claude Code