feat(newline): 'terminator' mode, a NEWLINE at every line break - #76
Open
theoephraim wants to merge 1 commit into
Open
feat(newline): 'terminator' mode, a NEWLINE at every line break#76theoephraim wants to merge 1 commit into
theoephraim wants to merge 1 commit into
Conversation
newline mode was built for dotenv-style grammars, but its NEWLINE is a statement SEPARATOR: one token between two content-bearing lines, placed at the start of the later line, with blank and comment-only lines collapsed and nothing before the first or after the last line. A grammar whose AST keeps blank lines and comments as lines (env-spec: File = many(Line), Line = Stmt NEWLINE? | NEWLINE) cannot be written on that, which is why env-spec sits on indent mode today with INDENT/DEDENT filler in every rule and an indent-stack tree-sitter scanner it never uses. NewlineConfig.mode: 'terminator' emits one NEWLINE at EVERY block-context line break, placed at the break: blank lines, comment-only lines, leading and trailing breaks included; breaks inside flow delimiters stay suspended; a final line with no break has none. Positions are exact per line, so a blank-line node maps to its own line. This is also the shape the derived tree-sitter scanner already has (stateless, one NEWLINE per break where the grammar permits one), so the two agree more closely than in separator mode. 'separator' stays the default and is byte-identical. The emitted engine inherits the mode through the createLexer fallback. The portable TS/Go/Rust newline lexers on the portable-newline-mode branch will need the same three emission points once that lands. Gate: test/newline-mode.ts sections 7 to 9 (lexer offsets, parser, the emitted engine inheriting the mode, and the derived tree-sitter parser agreeing on newline nodes for blank, leading, and trailing breaks).
theoephraim
force-pushed
the
feat/newline-terminator-mode
branch
from
September 8, 2026 04:51
41eef25 to
09cebfd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
newline.mode: 'terminator': a NEWLINE at every line breaknewlinemode was built for dotenv-style grammars, but its NEWLINE is a statement separator: one token between two content-bearing lines, placed at the start of the later line, with blank and comment-only lines collapsed and nothing before the first or after the last line. A grammar whose AST keeps blank lines and comments as lines cannot be written on that. env-spec is one (File = many(Line),Line = Stmt NEWLINE? | NEWLINE), which is why it sits onindentmode today withmany(alt(INDENT, DEDENT))filler in every rule and an indent-stack tree-sitter scanner it never uses.NewlineConfig.mode: 'terminator'emits one NEWLINE at every block-context line break, placed at the break (zero-width, before the\n/\r\n): blank lines, comment-only lines, leading and trailing breaks included; breaks inside flow delimiters stay suspended; a final line with no break has none. Positions are exact per line, so a blank-line node maps to its own line. It is also the shape the derived tree-sitter scanner already has (stateless, one NEWLINE per break where the grammar permits one), so parser and scanner agree more closely than in separator mode.'separator'stays the default and is byte-identical.Three emission points in
gen-lexer.ts: a content line's break, a blank line, a whitespace-only (tab) line; the line-start boundary emission is skipped in terminator mode. The emitted engine inherits the mode through the createLexer fallback. The portable TS/Go/Rust newline lexers onportable-newline-modewill need the same three points once that branch lands; happy to follow up there.Gate:
test/newline-mode.tssections 7 to 9. Section 7 pins the lexer (offsets for every case above, CRLF, flow suspension, no INDENT/DEDENT, separator mode unchanged) and the parser (a blank line is its own leaf at its own offset). Section 8 emits the grammar withjsTargetand checks the emitted engine parses identically to the interpreter, since the mode reaches it through the baked createLexer config. Section 9 runs the tree-sitter CLI when present and checks the derived parser produces onenewlinenode per break for blank, leading, and trailing breaks, and none inside flow.One thing section 9 turned up, outside this change:
Line = Stmt NEWLINE? | NEWLINE(the natural shape for a grammar that keeps blank lines) is LR-ambiguous after a statement, sotree-sitter generatereports an unresolved conflict for it. The PEG engine resolves it greedily; tree-sitter needs a declared conflict or aprecon the optional. The section therefore checks the scanner claim onProgram = many(NEWLINE | Stmt). Emitting that resolution fromgen-treesitter(a trailingopt(X)whose X also starts a sibling alternative) looks like a small follow-up; happy to take it.npm run check: 50/50.