feat(emit): standalone output for createLexer-fallback grammars - #75
Open
theoephraim wants to merge 1 commit into
Open
feat(emit): standalone output for createLexer-fallback grammars#75theoephraim wants to merge 1 commit into
theoephraim wants to merge 1 commit into
Conversation
A grammar whose lexing is one of the data-driven state machines (indent,
newline, markup) has no specialized lexer to embed, so its jsTarget
output reaches the shared createLexer runtime at load time. Until now
that was always an import of this repo's src/gen-lexer.ts by ABSOLUTE
path: right for the in-repo gates, wrong for anyone shipping the emitted
file, which only loaded on the machine that emitted it.
emitParser(grammar, target, opts) gains opts.lexerRuntime:
'import' (default) the absolute-path import, unchanged
'inline' gen-lexer.ts plus the two helpers it uses and the
type declarations copied verbatim into the module,
scoped so only createLexer escapes; the output is
standalone (no import) and type-checks alone
{ import: spec } a caller-owned specifier written verbatim
Gate: test/emit-standalone.ts proves for YAML and HTML that the inline
module loads from a directory outside the repo, parses tree-and-errors
identically to the default emitted engine (and to the interpreter on
clean parses), and passes tsc --strict without
--allowImportingTsExtensions.
theoephraim
force-pushed
the
feat/emit-standalone-lexer
branch
from
September 8, 2026 04:51
11e9ddf to
5e8bd44
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Standalone emitted output for createLexer-fallback grammars
A grammar whose lexing is one of the data-driven state machines (
indent,newline,markup) has no specialized lexer to embed (emit-lexer.tsreturnsnull), so itsjsTargetoutput reaches the sharedcreateLexerruntime at load time. Until now that was always animportof this repo'ssrc/gen-lexer.tsby absolute path, resolved at emit time. Right for the in-repo gates; wrong for anyone shipping the emitted file, which only loads on the machine that emitted it. env-spec is an indent grammar, so this is the one thing standing between us and committing an emitted parser the way we used to commit the peggy output.emitParser(grammar, target, opts)gainsopts.lexerRuntime:'import'(default): the absolute-path import, unchanged.'inline':src/gen-lexer.ts, the two helpers it uses (token-pattern.ts,grammar-utils.ts) andtypes.tscopied verbatim into the module at emit time, inside one function scope so onlycreateLexerescapes. The output is standalone: no import, and it type-checks alone. The sources stay the single copy, so an engine fix reaches inlined consumers on their next emit exactly like the import path.{ import: spec }: a caller-owned specifier written verbatim (a package entry, a relative path the caller controls).Self-contained grammars ignore the option.
Target.emitParsertakes the optional third argument; the portable targets do not read it.Gate:
test/emit-standalone.ts(registered inemit-parity). For YAML and HTML it proves the inline module has no import statement, loads from a directory outside the repo, parses tree-and-errors identically to the default emitted engine (and to the interpreter on clean parses), and passestsc --strictwithout--allowImportingTsExtensions; that{ import }writes exactly the given specifier with the rest of the output byte-identical to the default, and that a self-contained grammar (TypeScript) ignores the option with byte-identical output. README gets one paragraph in the emitter section.Noticed while writing the gate, unrelated to this change: the emitted HTML engine rejects
<ul><li>one<li>two</ul>(implicit</li>) while the interpreter accepts it. Happy to file it separately.npm run check: 50/50.