Skip to content

feat(newline): 'terminator' mode, a NEWLINE at every line break - #76

Open
theoephraim wants to merge 1 commit into
johnsoncodehk:masterfrom
dmno-dev:feat/newline-terminator-mode
Open

feat(newline): 'terminator' mode, a NEWLINE at every line break#76
theoephraim wants to merge 1 commit into
johnsoncodehk:masterfrom
dmno-dev:feat/newline-terminator-mode

Conversation

@theoephraim

@theoephraim theoephraim commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

newline.mode: 'terminator': a NEWLINE at every line break

newline mode was built for dotenv-style grammars, but its NEWLINE is a statement separator: one token between two content-bearing lines, placed at the start of the later line, with blank and comment-only lines collapsed and nothing before the first or after the last line. A grammar whose AST keeps blank lines and comments as lines cannot be written on that. env-spec is one (File = many(Line), Line = Stmt NEWLINE? | NEWLINE), which is why it sits on indent mode today with many(alt(INDENT, DEDENT)) filler in every rule and an indent-stack tree-sitter scanner it never uses.

NewlineConfig.mode: 'terminator' emits one NEWLINE at every block-context line break, placed at the break (zero-width, before the \n / \r\n): blank lines, comment-only lines, leading and trailing breaks included; breaks inside flow delimiters stay suspended; a final line with no break has none. Positions are exact per line, so a blank-line node maps to its own line. It is also the shape the derived tree-sitter scanner already has (stateless, one NEWLINE per break where the grammar permits one), so parser and scanner agree more closely than in separator mode. 'separator' stays the default and is byte-identical.

Three emission points in gen-lexer.ts: a content line's break, a blank line, a whitespace-only (tab) line; the line-start boundary emission is skipped in terminator mode. The emitted engine inherits the mode through the createLexer fallback. The portable TS/Go/Rust newline lexers on portable-newline-mode will need the same three points once that branch lands; happy to follow up there.

Gate: test/newline-mode.ts sections 7 to 9. Section 7 pins the lexer (offsets for every case above, CRLF, flow suspension, no INDENT/DEDENT, separator mode unchanged) and the parser (a blank line is its own leaf at its own offset). Section 8 emits the grammar with jsTarget and checks the emitted engine parses identically to the interpreter, since the mode reaches it through the baked createLexer config. Section 9 runs the tree-sitter CLI when present and checks the derived parser produces one newline node per break for blank, leading, and trailing breaks, and none inside flow.

One thing section 9 turned up, outside this change: Line = Stmt NEWLINE? | NEWLINE (the natural shape for a grammar that keeps blank lines) is LR-ambiguous after a statement, so tree-sitter generate reports an unresolved conflict for it. The PEG engine resolves it greedily; tree-sitter needs a declared conflict or a prec on the optional. The section therefore checks the scanner claim on Program = many(NEWLINE | Stmt). Emitting that resolution from gen-treesitter (a trailing opt(X) whose X also starts a sibling alternative) looks like a small follow-up; happy to take it.

npm run check: 50/50.

newline mode was built for dotenv-style grammars, but its NEWLINE is a
statement SEPARATOR: one token between two content-bearing lines, placed
at the start of the later line, with blank and comment-only lines
collapsed and nothing before the first or after the last line. A grammar
whose AST keeps blank lines and comments as lines (env-spec: File =
many(Line), Line = Stmt NEWLINE? | NEWLINE) cannot be written on that,
which is why env-spec sits on indent mode today with INDENT/DEDENT
filler in every rule and an indent-stack tree-sitter scanner it never
uses.

NewlineConfig.mode: 'terminator' emits one NEWLINE at EVERY block-context
line break, placed at the break: blank lines, comment-only lines,
leading and trailing breaks included; breaks inside flow delimiters stay
suspended; a final line with no break has none. Positions are exact per
line, so a blank-line node maps to its own line. This is also the shape
the derived tree-sitter scanner already has (stateless, one NEWLINE per
break where the grammar permits one), so the two agree more closely than
in separator mode. 'separator' stays the default and is byte-identical.

The emitted engine inherits the mode through the createLexer fallback.
The portable TS/Go/Rust newline lexers on the portable-newline-mode
branch will need the same three emission points once that lands.

Gate: test/newline-mode.ts sections 7 to 9 (lexer offsets, parser, the emitted
engine inheriting the mode, and the derived tree-sitter parser agreeing on
newline nodes for blank, leading, and trailing breaks).
@theoephraim
theoephraim force-pushed the feat/newline-terminator-mode branch from 41eef25 to 09cebfd Compare September 8, 2026 04:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant