Repository navigation
feat!: parse by offset and report where a failure happened - #2
Merged
Merged
Conversation
Parsers took a sliced suffix of the input, so nothing in the system knew
where it was in the original source. Position was not being withheld, it
did not exist: `position` returned the remaining length, and 50 of the 52
failure sites passed no message because there was nothing useful to say.
A `Parser<T>` now reads from an offset and never slices:
type Parser<T> = (input: string, index?: number) => Result<T>;
The offset defaults to 0, so composing the built-in parsers is unchanged
and `myParser('input')` still works.
Results also carry the furthest offset any branch reached and what was
expected there, threaded by the new `merge`. That is what makes the
message point at the mistake rather than at wherever the parse stopped:
before : Not all input consumed: ",b=x,c=3"
offsets: 1:4 expected end of input, found ','
now : 1:7 expected digit, found 'x'
1 | a=1,b=x,c=3
| ^
`separatedBy` succeeds at 1:4 with one pair, so without the trace the
error describes the wrong character.
Falling out of the offsets:
- `parse` returns a failure with line, column and expectations instead of
throwing; `run` throws a `ParseError` carrying the same.
- `grammar` names left-recursive rules instead of overflowing the stack.
- `memoize` keys on an offset, so entries are cheap and are dropped when
a different input is parsed.
- `regex` matches with a sticky pattern, anchored by construction; a
leading `^` is redundant and stripped.
- `position` reports the offset from the start, which can be paired with
`locate` to give a parsed node a source span.
Simple loops got faster (many(char) over 200k: 4.7ms -> 3.3ms) since
nothing allocates a slice per step. Choice-heavy loops pay for the trace
(many(choice) over 20k: 7.3ms -> 10.8ms).
BREAKING CHANGE: `Success.remaining` is replaced by `Success.index`, and
`Failure.error` by `index`/`furthest`/`expected`. `success(value,
remaining)` becomes `success(value, index)` and `failure(message)`
becomes `failure(index, ...expected)`. `run` throws a `ParseError`
rather than a generic `Error`, `match`'s failure branch receives
`(index, expected)`, and `position` returns an offset rather than the
remaining length. Composing the built-in parsers needs no change; see
"Migrating from 0.2.x" in the README.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The furthest-failure trace rode on every Result and was threaded by
`merge`. That is what Parsec and Parsimmon do and it works, but it put
two bookkeeping fields on `Success` and allocated on every step that
backtracked — which is every `choice`, every `separatedBy`, every
sequence that retries:
separatedBy(digits, ',') 20k 5.71ms -> 13.29ms (vs master)
kv pairs 10k 5.56ms -> 9.94ms
many(choice x3) 20k 8.51ms -> 10.92ms
It also showed up in every documented example, including successes,
where an `expected` on a completed parse reads as a contradiction:
fold(digit, 0, (acc, d) => acc + d)('123')
// { ok: true, value: 6, index: 3, furthest: 3, expected: ['digit'] }
The trace now lives in a mutable context, created per `run`/`parse` and
passed as a third argument. It only ever moves forward, so backtracking
cannot lose it, and it is written only on the failure path:
type Parser<T> = (input, index?, ctx?) => Result<T>;
type Success<T> = { ok: true; value: T; index: number };
type Failure = { ok: false; index: number; expected: readonly string[] };
Messages are unchanged — same offsets, same expectation sets — and the
combinators go back to their plain form. Against master:
many(char) 200k 4.63ms -> 2.89ms
many(regex) 200k 10.23ms -> 7.97ms
separatedBy 20k 5.71ms -> 5.26ms
kv pairs 10k 5.56ms -> 5.25ms
many(choice x3) 20k 8.51ms -> 8.53ms
bracketed list 10k 5.94ms -> 2.27ms
A combinator that fails to pass `ctx` on still parses correctly and only
produces a worse message, which no ordinary test would catch, so
`core/propagation.test.ts` checks every parser call site in
combinators/, primitives/ and terminals/ and picks up new files on its
own. It immediately found `not` leaking the failure it exists to
swallow: `not` and `label` now isolate their inner context, and
`memoize` keeps each entry's trace so a cache hit still contributes.
The README generator takes an export's name from its declaration rather
than from the file name, since one file may now document several
exports, and renders {@link} tags as code.
BREAKING CHANGE: `failure` takes the parse context first, as
`failure(ctx, index, ...expected)`. `Success` no longer carries
`furthest`/`expected` and `Failure` no longer carries `furthest`.
`merge` is gone — `context` and `record` replace it for hand-written
parsers. Composing the built-in parsers needs no change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each entry point was built separately, so the core types and
constructors were inlined into four of the five bundles. Importing from
more than one entry — which is how the library is meant to be used, and
what the quick start itself does — shipped that code several times over.
One build with five entries emits it once as a shared chunk:
one entry: terminals 2264 B -> 2924 B
quickstart: core+comb+term+prim 16967 B -> 14797 B
total shipped 17357 B -> 15187 B
A single entry point costs a little more unbundled, since a chunk is an
all-or-nothing unit. Anyone who bundles comes out ahead, because the
duplicate copies are what a tree-shaker could never remove:
app importing char + many 1232 B -> 850 B
Type resolution is unchanged, and the chunks are picked up by the
existing `files: ["dist"]`.
The core types block still described the pre-0.3.0 shape — a parser taking a sliced suffix and a failure carrying an optional message — so it contradicted the code it is meant to explain. It now documents the offset and the parse context, including the one rule that is easy to get wrong and impossible to notice: a parser you call must be passed `ctx`, or the parse still succeeds and only the error message quietly suffers. Also corrects three names that no longer matched: `npm run coverage` and `npm run watch` are `test:coverage` and `build:watch`, and `node` was described as an alias for an `ast` export that does not exist.
Coveralls was failing the PR: the new error machinery had no tests of
its own, and it was reached only incidentally by tests aimed at other
things. `format`, `parse` and the two branches in `diagnose` that pick
the context over the result — the ones that make an error point at the
furthest offset rather than where parsing stopped — were the gaps that
mattered, since that behaviour is the reason the release exists.
Adds `format.test.ts` and `parse.test.ts`, and covers the left-recursion
guard in `grammar`, the committed branch of `label` that keeps the inner
error, and `take`'s singular/plural wording.
Also drops an unreachable fallback in `format`: `line - 1` is the number
of newlines before the offset and `split('\n')` always yields one more
entry than that, so the index is never out of range.
Statements, branches, functions and lines are all at 100%.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Parsers took a sliced suffix of the input, so nothing in the system knew where it was in the original source. Position wasn't being withheld — it didn't exist.
positionreturned the remaining length, and 50 of the 52 failure sites passed no message, because there was nothing useful to say.What changed
A
Parser<T>now reads from an offset and never slices:Both new arguments are optional, so
myParser('input')still works —test/quickstart.test.tspassed through the entire refactor untouched.ctxis a per-parse context holding the furthest offset any branch reached and what was wanted there. It only moves forward, so backtracking can't lose it, and it's written only on the failure path. That's what makes the message point at the real mistake:separatedBysucceeds at1:4with one pair, so without it the error would read1:4 expected end of input— technically true, and pointing at the wrong character.choiceunions the expectations of every alternative tried at an offset:Falling out of the offsets
parsereturns a failure with line, column and expectations instead of throwing;runthrows aParseErrorcarrying the same.grammarnames left-recursive rules instead of overflowing the stack.memoizekeys on an offset, so entries are cheap and are dropped when a different input is parsed.regexmatches with a sticky pattern, anchored by construction; a leading^is redundant and stripped.positionreports the offset from the start, so it pairs withlocateto give a parsed node a source span.Performance
Median ms, invoked as
rundoes so tracking cost is included:many(char)200kmany(regex)200kseparatedBy(digits)20kmany(choice ×3)20kFaster or tied everywhere. Nothing allocates a slice per step any more.
The first draft (8f4c8aa's parent) carried the trace on every
Resultand threaded it with amerge— what Parsec and Parsimmon do. It worked, but it allocated on every step that backtracked (separatedBy20k went to 13.29ms, 2.3× master) and put two bookkeeping fields onSuccess, so every documented example grew a trace — including successes, where anexpectedon a completed parse reads as a contradiction. Moving it to a context fixed both.Packaging
Each entry point was built separately, inlining core into four of the five bundles. Importing from more than one entry — how the library is meant to be used, and what the quick start does — shipped that code several times over. One build with five entries emits it once:
char+manyA single unbundled entry costs +660 B, since a chunk is all-or-nothing. Anyone who bundles comes out ahead: the duplicate copies were exactly what a tree-shaker could never remove.
The one sharp edge
A combinator that fails to pass
ctxon still parses correctly and only produces a worse message — nothing an ordinary test would catch.src/core/propagation.test.tsscans every parser call site incombinators/,primitives/andterminals/and picks up new files on its own.It earned its keep immediately: it caught
notleaking the failure it exists to swallow.notandlabelnow isolate their inner context, andmemoizekeeps each entry's trace so a cache hit still contributes to the message.Breaking changes
Success.remaining→Success.index;Failure.error→index/expected.success(value, remaining)→success(value, index),failure(message)→failure(ctx, index, ...expected).runthrows aParseErrorrather than a genericError,match's failure branch receives(index, expected), andpositionreturns an offset rather than the remaining length.Composing the built-in parsers needs no change. See Migrating from 0.2.x in the README for the full table.
Verification
258 test files, 1324 tests, no type errors, lint clean, five bundles build.attwunchanged — 5 green onnode16 (from ESM)andbundler, same expected CJS reds for an ESM-only package.Checked against the published artifact rather than just
src: imported all five entries fromdist/and ran a real parse — messages correct,instanceof ParseErrorworks across the chunk boundary. Every error example in the README is executed and matches verbatim.Version bumped to
0.3.0.🤖 Generated with Claude Code