Skip to content

feat!: parse by offset and report where a failure happened - #2

Merged
sovrin merged 5 commits into
masterfrom
sovrin/offset-errors
Sep 19, 2026
Merged

sovrin merged 5 commits into
masterfrom
sovrin/offset-errors

Conversation

@sovrin

@sovrin sovrin commented Sep 19, 2026

Copy link
Copy Markdown
Owner

Parsers took a sliced suffix of the input, so nothing in the system knew where it was in the original source. Position wasn't being withheld — it didn't exist. position returned the remaining length, and 50 of the 52 failure sites passed no message, because there was nothing useful to say.

before : Error: Not all input consumed: ",b=x,c=3"

after  : ParseError: 1:7 expected digit, found 'x'
           1 | a=1,b=x,c=3
             |       ^

What changed

A Parser<T> now reads from an offset and never slices:

type Parser<T> = (input: string, index?: number, ctx?: Context) => Result<T>;
type Success<T> = { ok: true; value: T; index: number };
type Failure = { ok: false; index: number; expected: readonly string[] };

Both new arguments are optional, so myParser('input') still works — test/quickstart.test.ts passed through the entire refactor untouched.

ctx is a per-parse context holding the furthest offset any branch reached and what was wanted there. It only moves forward, so backtracking can't lose it, and it's written only on the failure path. That's what makes the message point at the real mistake: separatedBy succeeds at 1:4 with one pair, so without it the error would read 1:4 expected end of input — technically true, and pointing at the wrong character.

choice unions the expectations of every alternative tried at an offset:

1:1 expected 'const', 'let' or 'var', found 'l'
  1 | lot x
    | ^

Falling out of the offsets

  • parse returns a failure with line, column and expectations instead of throwing; run throws a ParseError carrying the same.
  • grammar names left-recursive rules instead of overflowing the stack.
  • memoize keys on an offset, so entries are cheap and are dropped when a different input is parsed.
  • regex matches with a sticky pattern, anchored by construction; a leading ^ is redundant and stripped.
  • position reports the offset from the start, so it pairs with locate to give a parsed node a source span.

Performance

Median ms, invoked as run does so tracking cost is included:

benchmark master this branch
many(char) 200k 4.63 2.89
many(regex) 200k 10.23 7.97
separatedBy(digits) 20k 5.71 5.26
kv pairs 10k 5.56 5.25
many(choice ×3) 20k 8.51 8.53
bracketed list 10k 5.94 2.27

Faster or tied everywhere. Nothing allocates a slice per step any more.

The first draft (8f4c8aa's parent) carried the trace on every Result and threaded it with a merge — what Parsec and Parsimmon do. It worked, but it allocated on every step that backtracked (separatedBy 20k went to 13.29ms, 2.3× master) and put two bookkeeping fields on Success, so every documented example grew a trace — including successes, where an expected on a completed parse reads as a contradiction. Moving it to a context fixed both.

Packaging

Each entry point was built separately, inlining core into four of the five bundles. Importing from more than one entry — how the library is meant to be used, and what the quick start does — shipped that code several times over. One build with five entries emits it once:

before after
quickstart's four entries 16 967 B 14 797 B
total shipped 17 357 B 15 187 B
app importing char + many 1 232 B 850 B

A single unbundled entry costs +660 B, since a chunk is all-or-nothing. Anyone who bundles comes out ahead: the duplicate copies were exactly what a tree-shaker could never remove.

The one sharp edge

A combinator that fails to pass ctx on still parses correctly and only produces a worse message — nothing an ordinary test would catch. src/core/propagation.test.ts scans every parser call site in combinators/, primitives/ and terminals/ and picks up new files on its own.

It earned its keep immediately: it caught not leaking the failure it exists to swallow. not and label now isolate their inner context, and memoize keeps each entry's trace so a cache hit still contributes to the message.

Breaking changes

Success.remaining → Success.index; Failure.error → index/expected. success(value, remaining) → success(value, index), failure(message) → failure(ctx, index, ...expected). run throws a ParseError rather than a generic Error, match's failure branch receives (index, expected), and position returns an offset rather than the remaining length.

Composing the built-in parsers needs no change. See Migrating from 0.2.x in the README for the full table.

Verification

258 test files, 1324 tests, no type errors, lint clean, five bundles build. attw unchanged — 5 green on node16 (from ESM) and bundler, same expected CJS reds for an ESM-only package.

Checked against the published artifact rather than just src: imported all five entries from dist/ and ran a real parse — messages correct, instanceof ParseError works across the chunk boundary. Every error example in the README is executed and matches verbatim.

Version bumped to 0.3.0.

🤖 Generated with Claude Code

sovrin and others added 2 commits September 6, 2026 23:40
Parsers took a sliced suffix of the input, so nothing in the system knew
where it was in the original source. Position was not being withheld, it
did not exist: `position` returned the remaining length, and 50 of the 52
failure sites passed no message because there was nothing useful to say.

A `Parser<T>` now reads from an offset and never slices:

    type Parser<T> = (input: string, index?: number) => Result<T>;

The offset defaults to 0, so composing the built-in parsers is unchanged
and `myParser('input')` still works.

Results also carry the furthest offset any branch reached and what was
expected there, threaded by the new `merge`. That is what makes the
message point at the mistake rather than at wherever the parse stopped:

    before : Not all input consumed: ",b=x,c=3"
    offsets: 1:4 expected end of input, found ','
    now    : 1:7 expected digit, found 'x'
               1 | a=1,b=x,c=3
                 |       ^

`separatedBy` succeeds at 1:4 with one pair, so without the trace the
error describes the wrong character.

Falling out of the offsets:

- `parse` returns a failure with line, column and expectations instead of
  throwing; `run` throws a `ParseError` carrying the same.
- `grammar` names left-recursive rules instead of overflowing the stack.
- `memoize` keys on an offset, so entries are cheap and are dropped when
  a different input is parsed.
- `regex` matches with a sticky pattern, anchored by construction; a
  leading `^` is redundant and stripped.
- `position` reports the offset from the start, which can be paired with
  `locate` to give a parsed node a source span.

Simple loops got faster (many(char) over 200k: 4.7ms -> 3.3ms) since
nothing allocates a slice per step. Choice-heavy loops pay for the trace
(many(choice) over 20k: 7.3ms -> 10.8ms).

BREAKING CHANGE: `Success.remaining` is replaced by `Success.index`, and
`Failure.error` by `index`/`furthest`/`expected`. `success(value,
remaining)` becomes `success(value, index)` and `failure(message)`
becomes `failure(index, ...expected)`. `run` throws a `ParseError`
rather than a generic `Error`, `match`'s failure branch receives
`(index, expected)`, and `position` returns an offset rather than the
remaining length. Composing the built-in parsers needs no change; see
"Migrating from 0.2.x" in the README.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The furthest-failure trace rode on every Result and was threaded by
`merge`. That is what Parsec and Parsimmon do and it works, but it put
two bookkeeping fields on `Success` and allocated on every step that
backtracked — which is every `choice`, every `separatedBy`, every
sequence that retries:

    separatedBy(digits, ',') 20k   5.71ms -> 13.29ms   (vs master)
    kv pairs 10k                   5.56ms ->  9.94ms
    many(choice x3) 20k            8.51ms -> 10.92ms

It also showed up in every documented example, including successes,
where an `expected` on a completed parse reads as a contradiction:

    fold(digit, 0, (acc, d) => acc + d)('123')
    // { ok: true, value: 6, index: 3, furthest: 3, expected: ['digit'] }

The trace now lives in a mutable context, created per `run`/`parse` and
passed as a third argument. It only ever moves forward, so backtracking
cannot lose it, and it is written only on the failure path:

    type Parser<T> = (input, index?, ctx?) => Result<T>;
    type Success<T> = { ok: true; value: T; index: number };
    type Failure = { ok: false; index: number; expected: readonly string[] };

Messages are unchanged — same offsets, same expectation sets — and the
combinators go back to their plain form. Against master:

    many(char) 200k          4.63ms -> 2.89ms
    many(regex) 200k        10.23ms -> 7.97ms
    separatedBy 20k          5.71ms -> 5.26ms
    kv pairs 10k             5.56ms -> 5.25ms
    many(choice x3) 20k      8.51ms -> 8.53ms
    bracketed list 10k       5.94ms -> 2.27ms

A combinator that fails to pass `ctx` on still parses correctly and only
produces a worse message, which no ordinary test would catch, so
`core/propagation.test.ts` checks every parser call site in
combinators/, primitives/ and terminals/ and picks up new files on its
own. It immediately found `not` leaking the failure it exists to
swallow: `not` and `label` now isolate their inner context, and
`memoize` keeps each entry's trace so a cache hit still contributes.

The README generator takes an export's name from its declaration rather
than from the file name, since one file may now document several
exports, and renders {@link} tags as code.

BREAKING CHANGE: `failure` takes the parse context first, as
`failure(ctx, index, ...expected)`. `Success` no longer carries
`furthest`/`expected` and `Failure` no longer carries `furthest`.
`merge` is gone — `context` and `record` replace it for hand-written
parsers. Composing the built-in parsers needs no change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@sovrin
sovrin merged commit a88208b into master Sep 19, 2026
4 checks passed
sovrin and others added 3 commits September 19, 2026 21:00
Each entry point was built separately, so the core types and
constructors were inlined into four of the five bundles. Importing from
more than one entry — which is how the library is meant to be used, and
what the quick start itself does — shipped that code several times over.

One build with five entries emits it once as a shared chunk:

    one entry: terminals              2264 B ->  2924 B
    quickstart: core+comb+term+prim  16967 B -> 14797 B
    total shipped                    17357 B -> 15187 B

A single entry point costs a little more unbundled, since a chunk is an
all-or-nothing unit. Anyone who bundles comes out ahead, because the
duplicate copies are what a tree-shaker could never remove:

    app importing char + many        1232 B ->   850 B

Type resolution is unchanged, and the chunks are picked up by the
existing `files: ["dist"]`.
The core types block still described the pre-0.3.0 shape — a parser
taking a sliced suffix and a failure carrying an optional message — so
it contradicted the code it is meant to explain. It now documents the
offset and the parse context, including the one rule that is easy to get
wrong and impossible to notice: a parser you call must be passed `ctx`,
or the parse still succeeds and only the error message quietly suffers.

Also corrects three names that no longer matched: `npm run coverage` and
`npm run watch` are `test:coverage` and `build:watch`, and `node` was
described as an alias for an `ast` export that does not exist.
Coveralls was failing the PR: the new error machinery had no tests of
its own, and it was reached only incidentally by tests aimed at other
things. `format`, `parse` and the two branches in `diagnose` that pick
the context over the result — the ones that make an error point at the
furthest offset rather than where parsing stopped — were the gaps that
mattered, since that behaviour is the reason the release exists.

Adds `format.test.ts` and `parse.test.ts`, and covers the left-recursion
guard in `grammar`, the committed branch of `label` that keeps the inner
error, and `take`'s singular/plural wording.

Also drops an unreachable fallback in `format`: `line - 1` is the number
of newlines before the offset and `split('\n')` always yields one more
entry than that, so the index is never out of range.

Statements, branches, functions and lines are all at 100%.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant