Skip to content

feat(emit): standalone output for createLexer-fallback grammars - #75

Open
theoephraim wants to merge 1 commit into
johnsoncodehk:masterfrom
dmno-dev:feat/emit-standalone-lexer
Open

feat(emit): standalone output for createLexer-fallback grammars#75
theoephraim wants to merge 1 commit into
johnsoncodehk:masterfrom
dmno-dev:feat/emit-standalone-lexer

Conversation

@theoephraim

@theoephraim theoephraim commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Standalone emitted output for createLexer-fallback grammars

A grammar whose lexing is one of the data-driven state machines (indent, newline, markup) has no specialized lexer to embed (emit-lexer.ts returns null), so its jsTarget output reaches the shared createLexer runtime at load time. Until now that was always an import of this repo's src/gen-lexer.ts by absolute path, resolved at emit time. Right for the in-repo gates; wrong for anyone shipping the emitted file, which only loads on the machine that emitted it. env-spec is an indent grammar, so this is the one thing standing between us and committing an emitted parser the way we used to commit the peggy output.

emitParser(grammar, target, opts) gains opts.lexerRuntime:

  • 'import' (default): the absolute-path import, unchanged.
  • 'inline': src/gen-lexer.ts, the two helpers it uses (token-pattern.ts, grammar-utils.ts) and types.ts copied verbatim into the module at emit time, inside one function scope so only createLexer escapes. The output is standalone: no import, and it type-checks alone. The sources stay the single copy, so an engine fix reaches inlined consumers on their next emit exactly like the import path.
  • { import: spec }: a caller-owned specifier written verbatim (a package entry, a relative path the caller controls).

Self-contained grammars ignore the option. Target.emitParser takes the optional third argument; the portable targets do not read it.

Gate: test/emit-standalone.ts (registered in emit-parity). For YAML and HTML it proves the inline module has no import statement, loads from a directory outside the repo, parses tree-and-errors identically to the default emitted engine (and to the interpreter on clean parses), and passes tsc --strict without --allowImportingTsExtensions; that { import } writes exactly the given specifier with the rest of the output byte-identical to the default, and that a self-contained grammar (TypeScript) ignores the option with byte-identical output. README gets one paragraph in the emitter section.

Noticed while writing the gate, unrelated to this change: the emitted HTML engine rejects <ul><li>one<li>two</ul> (implicit </li>) while the interpreter accepts it. Happy to file it separately.

npm run check: 50/50.

A grammar whose lexing is one of the data-driven state machines (indent,
newline, markup) has no specialized lexer to embed, so its jsTarget
output reaches the shared createLexer runtime at load time. Until now
that was always an import of this repo's src/gen-lexer.ts by ABSOLUTE
path: right for the in-repo gates, wrong for anyone shipping the emitted
file, which only loaded on the machine that emitted it.

emitParser(grammar, target, opts) gains opts.lexerRuntime:
  'import' (default)  the absolute-path import, unchanged
  'inline'            gen-lexer.ts plus the two helpers it uses and the
                      type declarations copied verbatim into the module,
                      scoped so only createLexer escapes; the output is
                      standalone (no import) and type-checks alone
  { import: spec }    a caller-owned specifier written verbatim

Gate: test/emit-standalone.ts proves for YAML and HTML that the inline
module loads from a directory outside the repo, parses tree-and-errors
identically to the default emitted engine (and to the interpreter on
clean parses), and passes tsc --strict without
--allowImportingTsExtensions.
@theoephraim
theoephraim force-pushed the feat/emit-standalone-lexer branch from 11e9ddf to 5e8bd44 Compare September 8, 2026 04:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant