Skip to content

fix(lexer): a backslash escapes the next character in a %w/%W/%i/%I word list - #25

Merged
tannevaled merged 2 commits into
mainfrom
wave23/wsplit
Sep 11, 2026
Merged

tannevaled merged 2 commits into
mainfrom
wave23/wsplit

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

The defect

In a %w/%W/%i/%I word list, a backslash escapes the character after it,
so that character belongs to the current word: it never separates two words and
never opens or closes the list. The lexer was escape-blind, which cost three
distinct defects:

source MRI before this PR
%w[a\ b c] ["a b", "c"] ["a\\", "b", "c"] — split, and the backslash kept
%w[a\]b] ["a]b"] parse error
%w[a\[b] ["a[b"] unterminated %-array literal
%w[a\\b] ["a\\b"] ["a\\\\b"] — the backslash doubled
%I[a\tb] [:"a\tb"] [:"a\\tb"] — escapes never resolved

The first row is the one that blocks specs. ruby/spec's Dir fixtures contain a
directory literally named special/test\ +()[]{}, written inside a %w[...]
list. Splitting it fabricated a phantom +()[]{} entry, which then poisoned
Dir.entries, Dir.children, Dir.each, Dir.foreach, Dir.read and the
glob specs — roughly 15-20 core/dir specs across three conformance waves.

The rule

From MRI's parse.y (tokadd_string), and it all follows from the backslash
branch being tested before the word-splitting and term/paren branches:

  • %w/%i (str_sword: QWORDS, no EXPAND, no ESCAPE) keep the
    backslash literally, except before whitespace, a delimiter, or another
    backslash — there it escapes a literal character. So %w[a\tb] is the four
    characters a \ t b and not a tab, while %w[a\\b] is the three
    characters a \ b.
  • %W/%I (str_dword: QWORDS|EXPAND) apply the full double-quote
    escape set, with one exception: MRI tests STR_FUNC_QWORDS before
    STR_FUNC_EXPAND, so a backslash before a newline is a literal newline, not
    the line continuation it would be inside a string.

The change

Word splitting moves from the parser to the lexer. strings.Fields cannot tell
a separating space from an escaped literal one, and the parser no longer knows
the delimiters needed to resolve \]. The WORDS/SYMBOLS token gains a
Words field and keeps its raw body in Lit.

  • lexPercentArray — a backslash consumes the next byte, so an escaped
    delimiter neither nests nor terminates; ending on one is unterminated.
  • splitWordList (new) — splits and unescapes a %w/%i body per the rule above.
  • splitPercentWords — no longer splits on an escaped space; rewrites an
    escaped newline so the string decoder yields the literal newline MRI does.
  • unescapeInterpWord (new) — a %I word with no interpolation now has its
    escapes resolved instead of becoming a symbol of the raw text.

Verification

The rules were confirmed against the installed MRI (ruby 4.0.5) rather than read
off parse.y alone. 65 word-list sources were run through both ruby and this
parser and compared as raw bytes, so nothing is lost to formatting:
62 identical. The other three are interpolating cases that legitimately fold
to a StrInterp/Call node instead of a constant, and were checked by hand —
including %W[a\ b#{1}c], where the escaped space lands in the literal prefix
"a b" beside a live interpolation.

  • go build ./..., go vet ./..., go test ./... — all green.
  • The CI coverage gate was run locally: 100.0%, every function at 100%.
  • gofmt -l clean for every file touched.
  • Bracket nesting and terminator handling are unchanged and covered
    (%w[a[b]c], %w(a(b)c), all six delimiter pairs).

No existing test pinned the old splitting, so none had to be changed.

🤖 Generated with Claude Code

In a word list a backslash escapes the character after it, so that character
is an ordinary character of the current word: it never separates two words and
never opens or closes the list. The lexer was escape-blind, which cost three
distinct defects.

  - An escaped space split the word. `%w[a\ b c]` lexed as three words
    ("a\", "b", "c") where MRI has two ("a b", "c"). ruby/spec's Dir fixtures
    contain a directory literally named `special/test\ +()[]{}` written inside
    a %w list, so every Dir spec that enumerates it saw a phantom `+()[]{}`
    entry.
  - An escaped delimiter ended the list. `%w[a\]b]` was a parse error, and
    `%w[a\[b]` an "unterminated %-array literal".
  - An escaped backslash was doubled: `%w[a\\b]` yielded `a\\b`, not `a\b`.

MRI's rule (parse.y, tokadd_string) follows from the backslash branch being
tested before the word-splitting and term/paren branches. In a
non-interpolating %w/%i the backslash is KEPT literally except before
whitespace, a delimiter, or another backslash, where it escapes a literal
character instead — so `%w[a\tb]` is the four characters a \ t b and not a
tab. In an interpolating %W/%I the full double-quote escape set applies, with
one exception: MRI's QWORDS test precedes its EXPAND one, so a backslash
before a newline is a literal newline rather than a line continuation.

Splitting moves from the parser (strings.Fields, which cannot see an escape)
to the lexer, which still knows the delimiters; the WORDS/SYMBOLS token gains
a Words field and keeps its raw body in Lit. %I words are now unescaped too,
so `%I[a\ b]` is :"a b" rather than the four characters a \ space b.

Every rule above was checked against the installed MRI (ruby 4.0.5) rather
than read off parse.y alone: 62 word-list sources were run through both `ruby`
and this parser and compared as raw bytes, all 62 identical. The three further
cases in that set are interpolating and fold to a StrInterp node instead of a
constant, and were checked by hand.
Four tables covering the non-interpolating (%w, %i) and interpolating (%W, %I)
forms: escaped space, escaped delimiter for all six delimiter pairs, escaped
backslash, escaped tab and newline, the backslashes %w keeps verbatim because
it has no escape set, the double-quote escape set %W applies instead, and the
unescaped whitespace and bracket nesting that must keep behaving as before.

Every expected value is the value the installed MRI (ruby 4.0.5) prints for
the same source; each was run through `ruby -e 'p <src>'` and compared as raw
bytes, so a failure here is a divergence from Ruby rather than from taste. The
MRI output is quoted beside each case.

Also covers the scanner's new end-of-input branch: a list ending on a trailing
backslash, or on an escaped terminator, is unterminated rather than accepted.
@tannevaled
tannevaled merged commit d16dba2 into main Sep 11, 2026
11 checks passed
@tannevaled
tannevaled deleted the wave23/wsplit branch September 11, 2026 10:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant