Repository navigation
fix(extraction): vendor tree-sitter runtime 0.25.10 + #5997 (error-recovery livelock) - #18
Merged
Merged
Conversation
…covery livelock)
A .tsx file holding `<a {{b:>c:d(e f)` timed out its parse worker 3 x 30s and was
dropped (fork issue inth3shadows/tree-sitter-typescript#16, upstream colbymchenry#323). The loop
is in tree-sitter's error recovery (tree-sitter/tree-sitter#5997), fixed only in
0.27.1 — which rejects legacy-dylink grammars (all tree-sitter-wasms 0.1.x + dart):
193 test failures. So: 0.25.10 built with the 3-line patch, loaded via locateFile.
- src/extraction/runtime/tree-sitter.wasm (sha256 d48b82be); unpatched rebuild ==
npm 0.25.10 tree-sitter.wasm byte-for-byte (f38dcc4b)
- initGrammars() throws if the runtime is missing; copy-assets + check-ui-build ship it
- web-tree-sitter pinned 0.25.10 exact; test fails if the glue version moves
- PoC project: 92s, 1 file dropped -> 1.43s, 2 files indexed
…review round 1) F1 kernel: codegraph-kernel/vendor/tree-sitter = crates.io tree-sitter 0.25.10 (its parser.c == git v0.25.10) + the #5997 patch, via [patch.crates-io]. Kernel hang test (child process, 10s): unpatched crate -> tsx ETIMEDOUT; patched -> 5/5 pass. F2 dev scripts (ts-differential, add-lang/dump-ast, add-lang/check-grammar, gen-go-stdlib-methods) and grammar-wasm-bytes test init through the vendored runtime (scripts/tree-sitter-runtime.mjs); dump-ast on the PoC: exit 124 -> 0. F5 check-ui-build verifies the dist runtime's sha256 (npm wasm swapped in -> exit 1). F7 drop the hang test's redundant wall-clock assertion. F8 CHANGELOG/doc: both paths covered. Suite with the kernel built: 11099 passed, 0 failed. Kernel tests: 312 passed. ts-differential: exit 0, 0 new / 0 worse.
- release gate runs tsx-error-recovery-hang.test.ts with CODEGRAPH_KERNEL_EXPECT=1; the test fails when EXPECT is set and no kernel is staged - sha256 pin on the vendored parser.c (runs without a kernel build; fails with the patch reverted) - kernel child asserts `defer` (the parse went through error recovery), not just done - tree-sitter = "=0.25.10" (the [patch] replaces exactly this version) - codegraph-kernel/vendor/** linguist-vendored; CHANGELOG entry trimmed to one sentence Release gate set (kernel built): 19 files, 319 passed.
inth3shadows
added a commit
that referenced
this pull request
Nov 6, 2026
Fork PR inth3shadows/tree-sitter-typescript#21: - S1 `in(): T {}` after a block-bodied arrow field; S2 `for (var k\n in!o)` / `in<T>x` stay for-in (member-`in` ASI now needs the new never-produced `_member_end_marker`, valid only where a class/object-type member ends) - S3 `unique<NBSP>symbol` (extras whitespace); S4 `unique // the symbol` stays a name Docs: commit + hashes; S6 (no `let x: unique`), S9 limits listed, S10 unverifiable counts removed; table growth noted (states +18%, wasm +15%, parse speed unchanged). Fixtures for S1-S4 (.ts, + .tsx where valid). Differential re-baselined: divergent set unchanged (3 + 5 gap), local 56. Main (PR #18 runtime fix) merged in. Kernel built: shapes/member-asi/variance/kernel parity/hang 154 passed; full suite 11104 passed, 1 failed (busiest-symbol, pre-existing: fails on em.36's index too).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #19.
Fixes a parser livelock: a
.tsxfile containing<a {{b:>c:d(e f)(tree-sitter-typescript upstream colbymchenry#323, fork inth3shadows/tree-sitter-typescript#16) timed out its parse worker three times (3 x 30s) and was then dropped from the index.Root cause: tree-sitter's error recovery (tree-sitter/tree-sitter#5997), not the grammar. The fix shipped only in 0.27.1, and web-tree-sitter 0.27 rejects wasm with the legacy
dylinksection. Every tree-sitter-wasms 0.1.x grammar and the vendored Dart use that section, so the straight bump failed 193 tests. Instead, this vendors 0.25.10's runtime built with the 3-line patch.Changes:
src/extraction/runtime/tree-sitter.wasm(sha256d48b82be…), loaded throughParser.init({ locateFile })ininitGrammars(), which throws if the file is missing.copy-assetscopies it todist/extraction/runtime/, andcheck-ui-buildfails a build without it (the bundle copies all ofdist).0.25.10.__tests__/tsx-error-recovery-hang.test.ts: two hang inputs × tsx/typescript must parse within 2s, plus a glue-version guard.docs/grammars/web-tree-sitter-runtime.{md,patch}: provenance, rebuild steps, exit plan.Verified:
tree-sitter.wasm(f38dcc4b…).× tsx 2007ms … did not finish within 2000ms. With the vendored runtime: 3 passed.tsc --noEmitexit 0;npm run buildexit 0 (runtime ok).npm audit: 7 advisories, identical on main (vite/vitest/rollup/svelte dev tooling).Round 1 review fixes (8d405ac):
codegraph-kernel/vendor/tree-sitteris the crates.io 0.25.10 crate, whoseparser.cis identical to the git tag's, with the same patch, wired in through[patch.crates-io]. The kernel hang test runsextractFilein a child with a 10s timeout: the unpatched crate gives tsxETIMEDOUT; the patched one passes 5/5.ts-differential,add-lang/dump-ast,add-lang/check-grammar,gen-go-stdlib-methods) andgrammar-wasm-bytes.test.tsnow init through the vendored runtime (scripts/tree-sitter-runtime.mjs).dump-aston the PoC: exit 124 -> 0.check-ui-buildchecks the dist runtime's sha256: npm's wasm swapped in -> exit 1.Exit plan: rebuild the legacy-
dylinkgrammars, move both paths to tree-sitter 0.27.1+, and delete the vendored runtime and crate.