Skip to content

parser: keyword parameters in a paren-less lambda list, the r/i-suffix panic, and seven more one-liner gaps - #39

Merged
tannevaled merged 3 commits into
mainfrom
wt47-lam-kwargs
Sep 27, 2026
Merged

tannevaled merged 3 commits into
mainfrom
wt47-lam-kwargs

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

Eight refusals and one over-acceptance family, each traced to a named production in MRI's grammar, measured against origin/main (v0.7.0, 8afa48e).

What the "DIED" actually was

core/proc/parameters_spec.rb scored 0 of 27 in the census and the run was reported as having died. It had not. On origin/main, rbgo core/proc/parameters_spec.rb prints exactly one line and exits 1:

parse error at line 65: expected {, got "x" (LABEL)

No panic, no hang — the parser refusing input, which is the least alarming of the three possibilities the brief named. parser.Parse recovers every panic by contract, so even the -3r defect below surfaced as internal parser error: …, never as a crash. "DIED" is the sweep harness classifying a file that emits no RBGO_RESULT line, and a file that cannot be loaded emits none.

It is also not the same defect as -3r: core/proc/parameters_spec.rb contains no rational or imaginary literal at all (grep '[0-9][ri]\b' → nothing). The two were independent.

The productions

# shape production (parse.y v3_4_0 unless noted)
1 -> x: 1 { x } f_larglist: f_args (5216) — the SAME f_args the parenthesised form takes; f_args: args_tail (6473); args_tail: f_kwarg(f_kw) … (6371); f_kw: f_label arg_value | f_label (6611); f_label: tLABEL (6591). Also checked against 4.0's f_largs: f_args-list(largs_tail, none) with its tail-only-args(tail) alternative, which says the same.
2 -3r, -3i, -3ri negate_lit sets minus on the OUTERMOST node (NODE_RATIONAL and NODE_IMAGINARY each carry their own); arg: tUMINUS_NUM simple_numeric (6256).
3 0x10r, 0b10i, 0o7r, 0d9i number_literal_suffix (9044), called with NUM_SUFFIX_ALL on the hex (9839), binary (9863), decimal (9892) and octal (9920) paths alike.
4 def M::name(v), def self::m defs_head: k_def singleton dot_or_colon def_name (3405); dot_or_colon: '.' | tCOLON2 (6839); singleton: var_ref | '(' expr rparen (6711); var_ref: user_variable | keyword_variable; fname: ident_or_const | tFID | op | reswords (3755).
5 when ⏎ value; in ⏎; case ⏎ case_body: k_when case_args then (6191) has no terms between k_when and case_args; then: term | keyword_then | term keyword_then (4890); k_case terms? (6136) makes the terminator after a subject-less case optional. MRI has no grammar rule for the newline at all — each keyword sets a value-expecting lexer state and parser_yylex's '\n' case never emits a terminator in one.
6 -> (**nil), { |**nil| } block_args_tail: f_any_kwrest opt_f_block_arg (5006); f_any_kwrest: f_kwrest | f_no_kwarg; f_no_kwarg: p_kwnorest: kwrest_mark keyword_nil.
7 BEGIN { }, END { } top_stmt: keyword_BEGIN begin_block (3020); begin_block: '{' top_compstmt '}' (3029); stmt_or_begin (3093) is where "BEGIN is permitted only at toplevel" lives; stmt: k_END allow_exits '{' compstmt '}' (3193).
8 x.%(1), 6.0./(2) MRI's method-name states are EXPR_FNAME and EXPR_DOT, and neither belongs to EXPR_BEG_ANY — the set parser_yylex tests before calling parse_percent or opening a regexp.
9 def (1).m, def __FILE__.m the literal check inside singleton itself (6724-6744), over NODE_STR/DSTR/XSTR/DXSTR/REGX/DREGX/SYM/DSYM/LINE/FILE/ENCODING/INTEGER/FLOAT/RATIONAL/IMAGINARY/LIST/ZLIST.

Two mechanisms were extended rather than duplicated, per the standing rule that a rule restated at several call sites drifts:

  • atMethodName() replaces the prevType != token.DEF test that each of the four %-literal openers and the regexp opener carried separately. That form named def but not a dot, which is defect 8 — one predicate now carries EXPR_FNAME and EXPR_DOT for all five.
  • numberLiteralSuffix() is one helper for the decimal and every radix path, so the two rules in MRI's function that a naive reading misses are stated once: an r after an i is masked off (2ri yes, 2ir no), and a letter / _ / non-ASCII byte ending the scan rejects the whole suffix and rewinds. That second rule is why p 2rescue nil is (p 2) rescue nil and prints 2.
  • checkSingletonRecv() is called from both def receiver paths.

The gate: the enumerated table

135 shapes, each judged by both ruby 4.0.5 front ends (ruby -c and ruby --parser=parse.y -c), with every refusal's message kept and classified rather than counted by exit status.

shapes enumerated                      : 135
judged by BOTH MRI parsers (they agree):  125
MRI parsers DISAGREE, asserted NEITHER :   10

X  REFUSED -> accepted : 60
Y  accepted -> REFUSED : 16
Z  MRI accepts and we refuse  (MUST BE 0) : 0
W  MRI refuses and we accept              : 0

All 16 Y are over-acceptances MRI also refuses: def M::Bar::baz (the receiver is a single var_ref, never a constant path — we used to read it as def M with a ::Bar::baz body, defining the wrong method silently), def obj&.m (dot_or_colon has no tANDDOT alternative), and the 14 literal receivers.

The 10 disagreements asserted neither: a[foo bar do end], f(foo bar do end), and eight not-in-expr shapes (a = not x, f(not x), p not x, !not x, def f; return not x; end, x = not not y, a[not x], x = not y and z) — prism accepts, parse.y refuses, and this parser sides with parse.y.

The mandated controls all hold, base and branch, under both MRI parsers: refused — while begin; y do end; end; end, while y do end; end, [foo bar], if x end, while x end, case x; when 1 end, p(x rescue y); accepted — a[foo bar], f(foo bar, 1), x = (y do end), [defined? foo], {k: defined? foo}, p((x rescue y)).

The floor: two sweeps

population base (8afa48e) branch gained lost
MRI stdlib — rubylibdir, 728 .rb 725 / 728 728 / 728 3 0
ruby/spec language/+core/ — 2206 *_spec.rb 2205 / 2206 2206 / 2206 1 0
ruby/spec whole tree — 4413 .rb 4407 / 4413 4411 / 4413 4 0

SPEC_SHA 87b1631; the tree also holds 3815 *_spec.rb in total, of which 2206 sit under language/+core/.

Gained: rbconfig.rb, mkmf.rb (both def M::name), rubygems/request.rb (when ⏎ value), core/proc/parameters_spec.rb (the 27-example file), language/fixtures/begin_file.rb (BEGIN), shared/kernel/fixtures/END.rb (END), library/bigdecimal/shared/modulo.rb (6543.21.%(137)).

The 2 files still refused in the whole tree are command_line/fixtures/bad_syntax.rb and core/exception/fixtures/syntax_error.rb — fixtures that exist to be invalid; ruby -c exits 1 on both under both parsers, so refusing them is correct and they must stay refused.

Sweep integrity. Output is indexed by input index from a file, so a dropped placeholder cannot silently produce an empty key column. Asserted per sweep: driver exit 0; rows == inputs; distinct paths == inputs; distinct indices == inputs; zero empty key columns; more than one verdict value; and a floor on the count itself.

Two of the three populations now parse completely, which makes "more than one verdict value" unsatisfiable from the real inputs alone — a saturated population and a broken probe look identical. Each sweep therefore appends three control files: two that must come back REFUSED (def f(, x = = 1) and one that must come back OK (p 1). Every run reports controls=REFUSED/REFUSED/OK, so the assertion has something to bite on at 100%.

Regression caught by the floor, and fixed

Making in continue across a newline turned

def in
  in_loc.slice
end

into def in in_loc.slice, and prism/node.rb — one of the 728 stdlib files — stopped parsing. A reserved word in a method-name position is a name, not the operator it usually spells: MRI's states for that are EXPR_FNAME and EXPR_DOT, where a keyword yields EXPR_ENDFN and nothing value-expecting. The lexer now records whether each token stood in such a position (prevFname, from the same atMethodName predicate) and will not continue a line from one. That also fixes the pre-existing def if / def while / def and cases of the same shape.

Coverage, Go 1.27.1

Measured with Go 1.27.1, not the host 1.26.4, and each cover -func resolved in the tree that produced its profile.

  • parser package 100.0% → 100.0%, with zero uncovered blocks in parser.go.
  • lexer package 44.1% → 49.9%.
  • the below-100 LIST gains nothing and loses radixDigit (the new lexer test is the first to reach lexRadixInt, 0.0% on main).
  • the FULL function set gains five functions and loses none — nothing orphaned.

Two new below-100 entries appeared on the first pass and were resolved by asking whether each was reachable first:

  • def (foo)m — the refusal when something other than . or :: follows a parenthesised receiver. Reachable; it is now in the refuse table (MRI refuses it too).
  • negateLiteral's default arm — not reachable from any Ruby source, since parseUnary calls the helper only after parsePrimary consumed an INT or FLOAT token. It is kept rather than deleted, because deleting it means restoring the trailing n.(*ast.FloatLit) assertion that panicked on every -3r; its job is to keep the never-panics contract for a node kind that should not arrive. A white-box internal test witnesses that job, using the same seam parseHook uses in robustness_internal_test.go.

gofmt -l . under Go 1.27.1 reports only paren_singleton_def_test.go, which was already non-empty on origin/main and is untouched here.

Every new test was A/B-ed with a mutation that still compiles — 12 of them, one per fix — and each made its own test fail and pass again on revert. The first attempt at one of those mutations silently hit the wrong call site (s/// without /g patched the def parameter guard instead of the lambda one) and the test kept passing; that is what the A/B is for.

Premises this measurement contradicted

  1. The "DIED" was a plain parse error. One line, exit 1, no panic and no hang — and unrelated to the rational defect, since that file holds no r/i literal.
  2. A panic here does not take the host process. Parse recovers by contract and returns internal parser error: …; the crash risk was real for the AST but never for the caller.
  3. The leading then on its own line was already fixed on v0.7.0. if x⏎then⏎1⏎end and case x⏎when 1⏎then 2⏎end both parse on 8afa48e. ripper/filter.rb and ripper/lexer.rb are not among main's three stdlib failures.
  4. -> () { … rescue … } was already fixed too, in all three spellings (-> () {}, -> {}, -> (a) {}).
  5. -> x: 1 { x } is [[:key, :x]], not [[:keyreq, :x]]. The keyreq in the brief belongs to -> x: { x }, the spelling with no default — two different rows of core/proc/parameters_spec.rb.
  6. {k: not x} is not one of the not-in-expr disagreements. Both 4.0.5 parsers refuse it; they agree. Eight disagreeing not shapes were found, not nine.
  7. [yield 1] is not a semantic refusal. Written inside the method that makes yield legal — def f; [yield 1]; end — both MRI parsers still refuse it: it is the [foo bar] rule, a command call in an arg position. The other five shapes on that list (yield, break, next, top-level return, super) do become legal inside their contexts, and all five parse here.
  8. The bibliography and the oracle are different grammars. parse.y v3_4_0 is what this repo cites; ruby 4.0.5 is what judges it. singleton is one production where they differ: v3_4_0 applies its literal check only to the '(' expr rparen alternative, while 4.0 restructured the rule into value_expr(singleton_expr) and applies it to the bare var_ref spelling too. def __FILE__.m parses under v3_4_0 and is refused by 4.0.5 under both front ends. This branch follows 4.0.5 and says so in the code. 4.0 also renamed the lambda's f_args to f_largs, which is why both grammars were checked for gap 1.

Not fixed

  • def (:"a#{b}").m stays accepted where MRI refuses it. This parser desugars an interpolated symbol to "a#{b}".to_sym, an *ast.Call indistinguishable from a real .to_sym call — which MRI does accept as a receiver. Refusing on that shape would refuse a legal program; it needs a distinct AST node for a dsym.
  • The 27 examples of core/proc/parameters_spec.rb are now reachable but the pass count is unmeasured: that needs an rbgo build, go-embedded-ruby/ruby is a different repository, and its local checkout is on another session's feature branch.

🤖 Generated with Claude Code

tannevaled and others added 3 commits September 27, 2026 10:45
…one-liner gaps

Seven refusals, each traced to one production in MRI's grammar.

1. `-> x: 1 { x }` — a paren-less lambda parameter list beginning with a keyword
   parameter. `f_larglist: f_args` (parse.y v3_4_0 5216) is the SAME `f_args` the
   parenthesised form takes, and `f_args: args_tail` (6473) makes a bare
   `args_tail` a whole list; `args_tail` starts with `f_kwarg(f_kw)` (6371),
   `f_kw: f_label arg_value | f_label` (6611), `f_label: tLABEL` (6591). The
   guard that decided whether a paren-less list was present named IDENT, `*`,
   `**` and `&` but not tLABEL, so the list was invisible and the `{` that
   followed was read as the body opener.

2. `-3r` / `-3i` / `-3ri` — negateLiteral ended in an unchecked
   `n.(*ast.FloatLit)`, and applyNumSuffix wraps the literal in RationalLit /
   ImaginaryLit. Now a full switch with a failing default. Negating the innermost
   numeric matches MRI's `negate_lit`, which flags the outermost node.

3. `0x10r`, `0b10i`, `0o7r`, `0d9i` were unlexed: lexRadixInt read no suffix.
   The suffix scan is now one helper shared by the decimal and radix paths,
   implementing MRI's number_literal_suffix (9044) — including the two rules that
   were missing entirely: an `r` after an `i` is masked off (`2ri` yes, `2ir` no),
   and a letter or `_` ending the scan rejects the WHOLE suffix, which is why
   `2rescue nil` is `2` with a modifier rescue.

4. `def M::name(v)`, `def self::m` — `dot_or_colon: '.' | tCOLON2` (6839) in
   `defs_head` (3405), so `::` is the same separator as `.`. Also `def nil.m` /
   `def true.m` / `def false.m` (`var_ref: keyword_variable`, 6711). Reaches
   rbconfig.rb and mkmf.rb. `def M::Bar::baz` correctly becomes a REFUSAL: the
   receiver is a single `var_ref`, never a constant path, and we used to accept it
   by mis-reading it as `def M` with a `::Bar::baz` body.

5. `when`, `in` and `case` now continue across a newline, as `if`/`while`/`and`
   already did. MRI has no rule for it: each sets a value-expecting lexer state
   and a newline there is discarded, so `case_body: k_when case_args then` (6191)
   never sees a terminator it does not allow. Reaches rubygems/request.rb:125.
   `k_case terms?` (6136) also makes `case when x then ... end` legal with no
   terminator at all.

6. `-> (**nil) { }` / `{ |**nil| }` — `block_args_tail: f_any_kwrest ...` (5006),
   `f_any_kwrest: f_kwrest | f_no_kwarg`, `f_no_kwarg: kwrest_mark keyword_nil`.

7. `BEGIN { }` / `END { }` (3020/3029 and 3193), with `BEGIN` refused anywhere but
   a `top_stmts` list — the whole purpose of MRI's `stmt_or_begin` (3093) — and
   still allowed inside another BEGIN, because `begin_block` reads a
   `top_compstmt`. BEGIN hoists to the front of the program as MRI's
   `eval_tree_begin` does; END desugars to `at_exit`, the replacement MRI's own
   warning names, measured equivalent for local visibility on ruby 4.0.5.

8. `x.%(1)` and `6.0./(2)` — the four literal openers were each guarded by
   `prevType != token.DEF`, which named `def` but not a dot, so `%` after `.`
   opened a %Q string and `/` opened a regexp. One `atMethodName()` predicate now
   carries the rule (MRI's EXPR_FNAME|EXPR_DOT, neither in EXPR_BEG_ANY).

Two premises this measurement contradicted: the leading `then` on its own line
and `-> () { ... rescue ... }` are ALREADY parsed on v0.7.0, and the "DIED" on
core/proc/parameters_spec.rb is a plain one-line parse error with exit 1 — no
panic, no hang, and no rational literal anywhere in that file.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rom continuing its line, and add the tests

Two corrections and the test tables for the batch.

The regression the sweep caught: making `in` continue across a newline turned
`def in` + a body on the next line into `def in in_loc.slice`, and
prism/node.rb — one of the 728 stdlib files — stopped parsing. A reserved word in
a method-NAME position is a name, not the operator it usually spells: MRI's states
for that are EXPR_FNAME (after `def`) and EXPR_DOT (after a dot), where a keyword
sets EXPR_ENDFN and nothing value-expecting. The lexer now records whether each
token stood in such a position (prevFname, from the same atMethodName predicate)
and refuses to continue the line from one, which also fixes the pre-existing
`def if` / `def while` / `def and` cases of the same shape.

The over-acceptance the shape table caught: `def (1).m`, `def ("s").m`,
`def (:sym).m`, `def ([]).m`, `def (/re/).m`, `def (2r).m`, `def (`x`).m` and
`def __FILE__.m` are all "can't define singleton method for literals" in MRI,
over a fixed node list in the `singleton` production (parse.y v3_4_0 6724-6744).
14 shapes; one helper, called from both receiver paths so they cannot drift.

Ruby 4.0 MOVED that check, which is worth recording: v3_4_0 applies it only to
the `'(' expr rparen` alternative, while 4.0.5 restructured `singleton` into
`value_expr(singleton_expr)` and applies it to the bare `var_ref` spelling too.
`def __FILE__.m` is the shape that distinguishes the two grammars — legal under
v3_4_0, refused by ruby 4.0.5 under both its front ends. The oracle is 4.0.5.

One shape is knowingly left accepted: `def (:"a#{b}").m`. This parser desugars an
interpolated symbol to `"a#{b}".to_sym`, an *ast.Call indistinguishable from a
real `.to_sym` call, which MRI accepts as a receiver — refusing on that shape
would refuse a legal program.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Coverage, measured with Go 1.27.1 (1.26 disagrees about basic-block counts, so a
1.26 figure cannot witness this gate).

The parser package went 100.0% -> 99.9% with two uncovered blocks, both new:

  - `def (foo)m` — the refusal when something other than `.` or `::` follows a
    parenthesised receiver. Reachable, and now in the refuse table (MRI refuses it
    too, under both parsers).

  - negateLiteral's default arm. NOT reachable from any Ruby source: parseUnary
    calls the helper only after parsePrimary consumed an INT or FLOAT token. It is
    kept rather than deleted, because deleting it means restoring the trailing
    `n.(*ast.FloatLit)` assertion that panicked on every `-3r`; its job is to keep
    the never-panics contract for a node kind that does not belong there. A
    white-box internal test witnesses that job, the same seam parseHook uses in
    robustness_internal_test.go.

The lexer package's own tests never reached lexRadixInt (0.0% on main), so the two
new predicates there — atMethodName and numberLiteralSuffix — landed below 100 as
well. lexer/method_name_and_num_suffix_test.go covers both, including the two
rules a "consume r then consume i" reading gets wrong, and as a side effect
radixDigit LEAVES the below-100 list.

Result against origin/main, Go 1.27.1: parser 100.0% -> 100.0%, lexer 44.1% ->
49.9%; the below-100 LIST gains nothing and loses radixDigit; the FULL function
set gains five functions and loses none, so nothing was orphaned.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit 4ecc81d into main Sep 27, 2026
11 checks passed
@tannevaled
tannevaled deleted the wt47-lam-kwargs branch September 27, 2026 09:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant