Skip to content

fix(extraction): vendor tree-sitter runtime 0.25.10 + #5997 (error-recovery livelock) - #18

Merged
inth3shadows merged 3 commits into
mainfrom
fix/tree-sitter-runtime-5997
Nov 6, 2026
Merged

inth3shadows merged 3 commits into
mainfrom
fix/tree-sitter-runtime-5997

Conversation

@inth3shadows

@inth3shadows inth3shadows commented Nov 6, 2026 •

Copy link
Copy Markdown
Owner

Closes #19.

Fixes a parser livelock: a .tsx file containing <a {{b:>c:d(e f) (tree-sitter-typescript upstream colbymchenry#323, fork inth3shadows/tree-sitter-typescript#16) timed out its parse worker three times (3 x 30s) and was then dropped from the index.

Root cause: tree-sitter's error recovery (tree-sitter/tree-sitter#5997), not the grammar. The fix shipped only in 0.27.1, and web-tree-sitter 0.27 rejects wasm with the legacy dylink section. Every tree-sitter-wasms 0.1.x grammar and the vendored Dart use that section, so the straight bump failed 193 tests. Instead, this vendors 0.25.10's runtime built with the 3-line patch.

before (npm 0.25.10) 0.27.1 bump this PR
PoC project (2 files) 92s, poc.tsx dropped 1.46s 1.43s, both indexed
full suite 1 failed (busiest-symbol, pre-existing) 193 failed 0 failed, 10549 passed

Changes:

  • src/extraction/runtime/tree-sitter.wasm (sha256 d48b82be…), loaded through Parser.init({ locateFile }) in initGrammars(), which throws if the file is missing.
  • copy-assets copies it to dist/extraction/runtime/, and check-ui-build fails a build without it (the bundle copies all of dist).
  • web-tree-sitter is pinned exactly at 0.25.10.
  • __tests__/tsx-error-recovery-hang.test.ts: two hang inputs × tsx/typescript must parse within 2s, plus a glue-version guard.
  • docs/grammars/web-tree-sitter-runtime.{md,patch}: provenance, rebuild steps, exit plan.

Verified:

  • Reproducibility: the same build without the patch is byte-identical to npm's 0.25.10 tree-sitter.wasm (f38dcc4b…).
  • Regression test with npm's unpatched runtime swapped in: × tsx 2007ms … did not finish within 2000ms. With the vendored runtime: 3 passed.
  • tsc --noEmit exit 0; npm run build exit 0 (runtime ok).
  • npm audit: 7 advisories, identical on main (vite/vitest/rollup/svelte dev tooling).

Round 1 review fixes (8d405ac):

Exit plan: rebuild the legacy-dylink grammars, move both paths to tree-sitter 0.27.1+, and delete the vendored runtime and crate.

…covery livelock)

A .tsx file holding `<a {{b:>c:d(e f)` timed out its parse worker 3 x 30s and was
dropped (fork issue inth3shadows/tree-sitter-typescript#16, upstream colbymchenry#323). The loop
is in tree-sitter's error recovery (tree-sitter/tree-sitter#5997), fixed only in
0.27.1 — which rejects legacy-dylink grammars (all tree-sitter-wasms 0.1.x + dart):
193 test failures. So: 0.25.10 built with the 3-line patch, loaded via locateFile.

- src/extraction/runtime/tree-sitter.wasm (sha256 d48b82be); unpatched rebuild ==
  npm 0.25.10 tree-sitter.wasm byte-for-byte (f38dcc4b)
- initGrammars() throws if the runtime is missing; copy-assets + check-ui-build ship it
- web-tree-sitter pinned 0.25.10 exact; test fails if the glue version moves
- PoC project: 92s, 1 file dropped -> 1.43s, 2 files indexed
…review round 1)

F1 kernel: codegraph-kernel/vendor/tree-sitter = crates.io tree-sitter 0.25.10 (its
parser.c == git v0.25.10) + the #5997 patch, via [patch.crates-io]. Kernel hang test
(child process, 10s): unpatched crate -> tsx ETIMEDOUT; patched -> 5/5 pass.
F2 dev scripts (ts-differential, add-lang/dump-ast, add-lang/check-grammar,
gen-go-stdlib-methods) and grammar-wasm-bytes test init through the vendored runtime
(scripts/tree-sitter-runtime.mjs); dump-ast on the PoC: exit 124 -> 0.
F5 check-ui-build verifies the dist runtime's sha256 (npm wasm swapped in -> exit 1).
F7 drop the hang test's redundant wall-clock assertion.
F8 CHANGELOG/doc: both paths covered.

Suite with the kernel built: 11099 passed, 0 failed. Kernel tests: 312 passed.
ts-differential: exit 0, 0 new / 0 worse.
- release gate runs tsx-error-recovery-hang.test.ts with CODEGRAPH_KERNEL_EXPECT=1;
  the test fails when EXPECT is set and no kernel is staged
- sha256 pin on the vendored parser.c (runs without a kernel build; fails with the
  patch reverted)
- kernel child asserts `defer` (the parse went through error recovery), not just done
- tree-sitter = "=0.25.10" (the [patch] replaces exactly this version)
- codegraph-kernel/vendor/** linguist-vendored; CHANGELOG entry trimmed to one sentence

Release gate set (kernel built): 19 files, 319 passed.
@inth3shadows
inth3shadows merged commit 36117d3 into main Nov 6, 2026
1 check passed
inth3shadows added a commit that referenced this pull request Nov 6, 2026
Fork PR inth3shadows/tree-sitter-typescript#21:
- S1 `in(): T {}` after a block-bodied arrow field; S2 `for (var k\n in!o)` /
  `in<T>x` stay for-in (member-`in` ASI now needs the new never-produced
  `_member_end_marker`, valid only where a class/object-type member ends)
- S3 `unique<NBSP>symbol` (extras whitespace); S4 `unique // the symbol` stays a name

Docs: commit + hashes; S6 (no `let x: unique`), S9 limits listed, S10 unverifiable
counts removed; table growth noted (states +18%, wasm +15%, parse speed unchanged).
Fixtures for S1-S4 (.ts, + .tsx where valid). Differential re-baselined: divergent
set unchanged (3 + 5 gap), local 56. Main (PR #18 runtime fix) merged in.

Kernel built: shapes/member-asi/variance/kernel parity/hang 154 passed; full suite
11104 passed, 1 failed (busiest-symbol, pre-existing: fails on em.36's index too).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native kernel: tree-sitter 0.25 error-recovery livelock on TSX (#5997) still hits kernel-routed parses

1 participant