Skip to content

fix(haskell): stop escaped char literals at the closing quote (#2439) - #2547

Open
Zhuoxi2000 wants to merge 6 commits into
DeusData:mainfrom
Zhuoxi2000:fix-haskell-escaped-char-swallow
Open

Zhuoxi2000 wants to merge 6 commits into
DeusData:mainfrom
Zhuoxi2000:fix-haskell-escaped-char-swallow

Conversation

@Zhuoxi2000

@Zhuoxi2000 Zhuoxi2000 commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes #2439

When a Haskell definition contains an escaped char literal ('\n') and the next definition has a primed name (g'), the first definition swallows the second. With the repro from #2439, f spans lines 3-5 and g' gets no node. The parse also reports has_error even though the module is valid. The same bug has a one-line form: ['\n','\t'] lexes as a single char token and reports no error.

Cause. The scanner is not the culprit. The fault is in the grammar's internal lexer: grammar/literal.js defines char: choice(/'[^']'/, /'\\[^ ]*'/). In the escape branch, [^ ]* also matches newlines and quotes, and tree-sitter keeps the longest match, so the token runs to the last ' before the next space. In the repro that is the prime in g'. The regex is the same at our pin 7fa19f195803 and on upstream master, so there is no upstream fix to pull.

Change (reworked per review, following the pascal model from #2211). haskell becomes a self-maintained fork of the pin we already vendor, tree-sitter/tree-sitter-haskell @ 7fa19f195803 (v0.23.1):

  • tools/tree-sitter-haskell/ holds the pinned grammar source (grammar.js + grammar/), the one patch patches/escaped-char-literal.patch (applied in grammar/literal.js), package.json / tree-sitter.json, the regenerated src/, and test/corpus/escaped_char_literal.txt (3 cases, tree-sitter test passes).
  • The patched token is /'\\('|[^'\s]+)'/: a backslash, then either a quote ('\'') or a run of non-quote, non-whitespace characters, then the closing quote. '\\', '\x41', '\SOH', '\^A', '\1234', MagicHash '\n'# lex as before, and upstream's th.txt corpus cases (TH quotes 'foo, ''Bar) still pass. Node types are unchanged (node-types.json is identical to the unpatched regeneration).
  • internal/cbm/vendored/grammars/haskell/parser.c is regenerated with tree-sitter-cli 0.25.10 (ABI 15, as before). scanner.c, unicode.h and tree_sitter/*.h are untouched. The new parser.c has 0 non-ASCII bytes and the same state/symbol/token/field/production counts (12862/537/156/84/560).
  • Registered like pascal: MANIFEST.md (counts line, fork note, provenance row, fork table row), scripts/audit-license-provenance.py FORKS map, THIRD_PARTY.md, and scripts/vendored-checksums.txt via scripts/security-vendored.sh --update (only the MANIFEST.md and haskell/parser.c lines change). LICENSE stays upstream's MIT, byte-identical.
  • tests/test_test_impact.c maps tools/tree-sitter-haskell/src/parser.c as repo-data, like pascal's.

Fidelity. Regenerating the UNPATCHED grammar with 0.25.10 reproduces the previously vendored parser.c byte-for-byte (apart from the trailing newline), so the parser.c diff is the char lexer and nothing else. The patched parser passes all 724 cases of upstream's corpus (same as the unpatched regeneration) and gives a tree identical to the unpatched one on upstream's examples/Basic.hs.

One behaviour difference from the earlier hand patch, for the better: an escaped char followed directly by a stray ' ('\n'', not valid Haskell) is now reported as an ERROR node instead of being absorbed into the token.

Checklist

  • Every commit is signed off (git commit -s). scripts/check-dco.sh origin/main..HEAD reports OK for both commits.
  • Tests pass locally, with one caveat: the ASan+UBSan runner from make -f Makefile.cbm test hangs on this machine during ASan shadow-memory init, before main() runs (macOS 26.5, Apple clang 17), unrelated to this change. An unsanitized runner was used instead, make -f Makefile.cbm -j4 SANITIZE= BUILD_DIR=build/nosan build/nosan/test-runner, and the sanitized run is left to CI. Results with the fix:
    • extraction: 488 passed
    • pipeline: 412 passed
    • language: 250 passed
    • lang_contract: 43 passed
    • node_creation_probe: 80 passed
    • grammar_regression: 1 passed
    • grammar_labels: 2 passed
    • grammar_imports: 1 passed
    • security: 38 passed
    • parse_coverage: 61 passed
    • test_impact: 89 passed
    • scripts/security-vendored.sh: Vendored integrity check passed (1083 files)
    • tests/test_vendored_integrity_contract.sh: OK
    • python3 scripts/audit-license-provenance.py: PASSED, grammars/haskell: IDENTICAL
    • npx tree-sitter-cli@0.25.10 test in tools/tree-sitter-haskell/: 3/3
  • Lint: lint-ci runs cppcheck and clang-format over LINT_SRCS/LINT_HDRS only, and this PR changes none of those files (the C changes are the vendored parser.c and two test files), so the cppcheck and clang-format results on main are unchanged; lint-no-suppress and scripts/lint-memory-core.py are unaffected. clang-format was not run locally this time (not installed here): the tests/test_extraction.c hunks are byte-identical to the first revision of this PR, and tests/test_test_impact.c gains one line mirroring its neighbour. clang-tidy was not run; it is the local pre-commit linter and not part of lint-ci.
  • New behavior is covered by a test (reproduce-first). Both tests are in tests/test_extraction.c, unchanged from the first version of this PR, and the first commit adds only these tests:

Test evidence

Before (test commit only, on current main f05c221):

$ ./build/nosan/test-runner extraction
  haskell_escaped_char_does_not_swallow_primed_def         FAIL tests/test_extraction.c:1912: (int)f->end_line == 5, expected 3 == 3
  haskell_escaped_char_list_lexes_each_literal             FAIL tests/test_extraction.c:1961: "'\n','\t','\'','\\','\x41','\SOH','a' " != "'\n' '\t' '\'' '\\' '\x41' '\SOH' 'a' "
  486 passed, 2 failed

After (this PR):

$ ./build/nosan/test-runner extraction
  haskell_escaped_char_does_not_swallow_primed_def       PASS
  haskell_escaped_char_list_lexes_each_literal           PASS
  488 passed

AI assistance: this change was drafted with an AI coding assistant (Claude) and verified locally with the tests above.

@Zhuoxi2000
Zhuoxi2000 requested a review from DeusData as a code owner October 4, 2026 19:46
@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown

Thanks for opening this — it has been seen, and it is queued.

This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence.

Current review status: working through a backlog. 0.9.1-rc.1 is out, so the release freeze that held reviews is over — but it left a large queue of open pull requests behind it, and we are reading through them oldest-first. The background is in discussion #1144.

What that means for this PR, concretely:

  • It will not be closed for inactivity. No stale bot touches pull requests here.
  • It may still sit a while before a human reads it. That is on us, not on you.
  • Older PRs are read first, so a recent one is not being skipped — it is behind a queue.

Things that will genuinely speed it up whenever review does happen:

  • Keep it rebased on main — the tree is moving quickly right now, and a conflicting branch cannot be reviewed as the diff you intended.
  • Get CI green, or say which failures you believe are pre-existing.
  • Keep the change to one claim. Bundled features and refactors get split before they get merged, which costs you a round trip.
  • Every commit needs a sign-off (git commit -s) — CI enforces DCO.

If this fixes a bug, a reproduction we can run is worth more than a description of the symptom.

Thanks for contributing, and sorry in advance for the wait.

@icecold009 icecold009 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI-assisted review using OpenAI Codex and TypeSafe Jev (jev-1.13.0), on behalf of @icecold009. This is independent community feedback, not a maintainer decision.

I reviewed all four changed files and traced the generated lexer transitions for whitespace/newline stopping, escaped apostrophe handling, and the optional # suffix. I found no actionable defect in the patch. Both added regression cases ran in the hosted extraction suite (410 passed, 0 failed) on this head.

The current required PR / ci-ok check is failing because the Windows CLANG64 shard fails daemon_conflict_log_windows_concurrent_appends_are_not_dropped (tests/test_daemon_version.c:686). That failing test is outside the changed files; I could not establish a link to this Haskell parser patch. Please confirm it reproduces on the base or rerun before merge.

@Zhuoxi2000

Copy link
Copy Markdown
Contributor Author

Drafted with an AI coding assistant (Claude).

Thanks @icecold009 for going through the lexer changes.

On the red ci-ok: the only failure is daemon_conflict_log_windows_concurrent_appends_are_not_dropped (tests/test_daemon_version.c:686) in the Windows CLANG64 1/2 shard. This PR only touches the vendored Haskell grammar, its checksums and tests/test_extraction.c, so it doesn't reach the daemon conflict log. The same shard passed on the eight most recent PR runs I checked from 10-05 (#514 NestJS DI, the three Scala PRs, the Swift and TS ones, among others), so it looks like an intermittent failure in that concurrency test rather than something this change triggers. I can't re-run jobs on this repo; could a maintainer re-run the failed shard when convenient?

@DeusData DeusData added bug Something isn't working parsing/quality Graph extraction bugs, false positives, missing edges labels Oct 19, 2026
@DeusData

Copy link
Copy Markdown
Owner

Thanks for the detailed Haskell lexer explanation, and thanks to @icecold009 for the community review. I have read the Windows failure/rerun request; this sweep has not rerun CI or established its cause. The hand-patched generated lexer versus grammar regeneration choice also remains for maintainer review.

@DeusData

Copy link
Copy Markdown
Owner

Thank you, Edson, for the clear root cause and for the reproduce-first tests, which we want to keep exactly as they are - tracking #2439 down to the char-literal token instead of the scanner was the hard part. For token-level grammar fixes we keep a regenerable fork of the grammar (as we do for javascript/tsx and pascal), so that a future re-vendor does not have to re-derive hand-edited lexer states. So we would like to take you up on your offer: narrow the char regex in literal.js (for example to /'\('|[^'\s]+)'/) and regenerate parser.c with tree-sitter-cli 0.25.10. If you would rather not set up the fork wiring yourself, just say so and we are happy to do that part on our side and credit you. Thanks again for a careful fix.

…eusData#2439)

The vendored grammar lexes an escaped char literal with /'\\[^ ]*'/,
whose [^ ]* also crosses newlines and quotes. With only newlines between
'\n' and a primed name (g'), the char token runs through g', so f spans
g' and g' gets no node. On one line, ['\n','\t'] lexes as a single char
token with no parse error. Add regression tests for both forms.

Signed-off-by: Edson <zhuoxi2000@gmail.com>
…DeusData#2439)

An escaped char literal followed by a primed definition swallowed that
definition: with

    f = '\n'

    g' = 2

the `char` token ran from '\n through g', so f spanned three lines, g'
got no node and the valid module reported has_error. On one line
['\n','\t'] lexed as a single char token, silently.

Root cause is upstream tree-sitter-haskell (the pin we vendor): the
escape branch of the `char` token is /'\\[^ ]*'/, and [^ ]* also
matches newlines and quotes, so the lexer's longest match runs on to the
last ' before the next space. The external scanner is not involved. The
regex is unchanged on upstream master, so there is no fix to pull.

Fix: haskell becomes a self-maintained fork of the pin we already vendor
(tree-sitter/tree-sitter-haskell @ 7fa19f195803, v0.23.1), following the
javascript/tsx and pascal model. One patch
(tools/tree-sitter-haskell/patches/escaped-char-literal.patch) narrows
that one regex in grammar/literal.js to /'\\('|[^'\s]+)'/: a backslash,
then either a quote ('\'') or a run of non-quote, non-whitespace
characters, then the closing quote. '\\', '\x41', '\SOH', '\^A', '\1234'
and MagicHash '\n'# lex as before; a stray ' after a literal is now an
ERROR node instead of being absorbed. Node types are unchanged.
Regenerated with tree-sitter-cli 0.25.10 (ABI 15, as before; scanner.c,
unicode.h and tree_sitter/*.h untouched); parser.c has 0 non-ASCII bytes
and the same state/symbol/token/field/production counts.

Fidelity: regenerating the UNPATCHED grammar with 0.25.10 reproduces the
previously vendored parser.c byte-for-byte (apart from the trailing
newline). The patched parser passes all 724 cases of upstream's corpus
and gives a tree identical to the unpatched one on upstream's
examples/Basic.hs.

Registered like pascal: MANIFEST.md fork row + note, audit FORKS map,
THIRD_PARTY.md, vendored checksums. LICENSE stays upstream's MIT,
byte-identical.

Tests: the two reproduce-first extraction tests from the previous commit
are unchanged (the DeusData#2439 repro, and the one-line list of seven escapes);
the fork carries test/corpus/escaped_char_literal.txt; test_impact maps
the fork's parser.c as repo-data like pascal's.

Signed-off-by: Edson <zhuoxi2000@gmail.com>
@Zhuoxi2000
Zhuoxi2000 force-pushed the fix-haskell-escaped-char-swallow branch from 340bb58 to af8bac8 Compare October 26, 2026 01:33
@Zhuoxi2000

Copy link
Copy Markdown
Contributor Author

Done, reworked as a regenerable fork following the pascal model (#2211):

  • tools/tree-sitter-haskell/ carries the pinned grammar source (grammar.js + grammar/, @ 7fa19f195803), patches/escaped-char-literal.patch (the one hunk in grammar/literal.js), package.json / tree-sitter.json, the regenerated src/, and test/corpus/escaped_char_literal.txt (3 cases).
  • The regex is the one proposed: /'\\('|[^'\s]+)'/. Regenerated with tree-sitter-cli 0.25.10, ABI 15 as before; scanner.c, unicode.h and tree_sitter/*.h are untouched, and the vendored parser.c has 0 non-ASCII bytes and the same state/symbol/token counts (12862/537/156).
  • Fidelity: regenerating the unpatched grammar with 0.25.10 reproduces the previously vendored parser.c byte-for-byte (apart from the trailing newline), so the parser.c diff is only the char lexer. The patched parser passes all 724 cases of upstream's corpus and gives an identical tree on upstream's examples/Basic.hs. One behaviour difference from the hand patch, for the better: a stray ' right after a literal ('\n'') is now an ERROR node instead of being absorbed.
  • Registered like pascal: MANIFEST note + fork rows, audit FORKS map, THIRD_PARTY.md, vendored checksums (security-vendored.sh --update; the integrity contract and audit-license-provenance.py pass).
  • The two reproduce-first tests are unchanged; the branch is rebased on current main (it had drifted after fix(haskell): name infix-form definitions after the operator #2457), so it is 2 commits: tests, then the fork.

One thing for your call: following the pascal layout, tools/tree-sitter-haskell/src/parser.c duplicates the 19 MB vendored file. If you would rather not carry that copy in tools/, say so and we'll drop src/parser.c there (keeping grammar.json / node-types.json) — the vendored one is the one that ships either way.

Review map: of the 40 files, 33 are the new tools/tree-sitter-haskell/ (pinned upstream source plus the regenerated src/). The hand-reviewable diff is the grammar/literal.js hunk, the char lexer states in the vendored parser.c (+844/−852), and the MANIFEST / THIRD_PARTY / audit FORKS / checksums / test_test_impact.c lines.

PR description updated to match.

@DeusData

Copy link
Copy Markdown
Owner

Thank you, Edson - this is exactly the rework we hoped for, and it came back within hours. We have been through it end to end: the patch reproduces your literal.js from the pinned upstream file, src/grammar.json differs from upstream only in that one regex, node-types.json is unchanged, and the regenerated parser.c only renumbers lexer states inside ts_lex, with every table count unchanged and no non-ASCII bytes. One maintainer sign-off on the fork registration is still pending on our side; nothing is needed from you in the meantime.

On your question about tools/tree-sitter-haskell/src/parser.c: please keep it. It is byte-identical to the vendored copy, so the repository stores it only once, and it matches the layout of the pascal fork.

The fork registration put the haskell fork row in the "Self-maintained
forks" table and also replaced the original row in the "Vendored from
verified upstream" table with the same text, so haskell was listed
twice and the upstream table carried 140 rows against the 139 the
summary line declares. pascal, the model for this fork, is listed only
in the forks table; drop the copy from the upstream table so both
tables agree with the summary counts (139 upstream, 18 self-maintained).

The license audit is unaffected: haskell is routed through the FORKS
map, and its manifest parser never matched the fork-format row.

MANIFEST.md is itself covered by scripts/vendored-checksums.txt, so
re-hash it with scripts/security-vendored.sh --update; that MANIFEST.md
line is the only one that changes. The vendored haskell parser.c is not
touched.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Brings the branch up to origin/main 9f278b7. One conflict, in
scripts/vendored-checksums.txt: both sides re-hashed the MANIFEST.md
line (main for its objectscript scanner patches, this branch for the
haskell fork). Resolved by taking main's file and re-running
scripts/security-vendored.sh --update on the merged tree; against
origin/main the only lines that differ are MANIFEST.md (the merged
file) and haskell/parser.c (this branch's regenerated parser). Every
other file merged cleanly and carries exactly this branch's change.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
@DeusData

DeusData commented Nov 1, 2026

Copy link
Copy Markdown
Owner

Thank you, Edson - this is a model rework. Tracking #2439 down to the escape branch of the char token instead of the scanner was the hard part, your two reproduce-first tests pin both forms of the bug exactly, and the fork is laid out just the way we keep pascal and javascript/tsx: the pinned source, one small patch, a regenerated parser.c that differs only in the char lexer, and the registration in the audit FORKS map, THIRD_PARTY.md and the checksums. We are taking it as is, including your FORKS row in scripts/audit-license-provenance.py.

We pushed two small commits on top of your branch so it can merge:

  1. fix(grammar): drop the duplicate haskell MANIFEST row and re-hash it. Your fork row went into the "Self-maintained forks" table, and the same row had also replaced haskell's old entry in the "Vendored from verified upstream" table, so haskell was listed twice and that table had 140 rows against the 139 in the summary line. pascal is listed only in the forks table, so we removed the copy from the upstream table. MANIFEST.md is itself checksummed, so its line in scripts/vendored-checksums.txt was regenerated with scripts/security-vendored.sh --update (the only line that changed). parser.c is untouched.

  2. Merge origin/main. main had moved on and had also re-hashed MANIFEST.md, so scripts/vendored-checksums.txt conflicted. We took main's file and re-ran scripts/security-vendored.sh --update on the merged tree; compared with main, the only lines that differ are MANIFEST.md and your regenerated haskell/parser.c. Every other file carries exactly your change.

We checked the result locally: both of your tests fail with main's grammar and pass with yours, the extraction, language, lang_contract, grammar and test_impact suites pass, and the vendored integrity check passes.

Since your branch now has these two new commits, please pull before pushing again. We will merge as soon as the required checks are green. Thanks again for a careful, well-documented fix.

Zhuoxi2000 and others added 2 commits November 10, 2026 21:21
Resolve the vendored-registry conflicts with the vbnet grammar (DeusData#1179):
keep both the haskell fork and vbnet entries in THIRD_PARTY.md and
MANIFEST.md, recount the manifest totals (163 = 139 upstream + 19
self-maintained + 5 registry-disagreement; ABI line unchanged), and
refresh the MANIFEST.md checksum.

Signed-off-by: Edson <zhuoxi2000@gmail.com>
Brings the branch up to main 1ec2f26. Conflicts were the grammar
bookkeeping only:

- THIRD_PARTY.md: main's thirteen self-maintained forks (sql, vba added)
  plus haskell make fourteen; the haskell paragraph sits between pascal
  and main's new sql paragraph; main's tree-sitter-vba section is kept.
- MANIFEST.md summary: main's 164 grammars with haskell moved from
  vendored-from-upstream to self-maintained: 138 + 21 + 5 = 164,
  recounted from the tree (164 grammar dirs, 164 parser.c, ABI
  9/78/77). haskell stays only in the self-maintained forks table.
- scripts/vendored-checksums.txt: regenerated with
  scripts/security-vendored.sh --update.

scripts/audit-license-provenance.py merged cleanly: the haskell FORKS
row sits alphabetically between dotenv and javascript, next to main's
sql and vba rows.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
@DeusData

Copy link
Copy Markdown
Owner

Thank you again, Edson, and thanks for merging main yourself on 11-11 - that kept this small.

main moved on in the meantime: it vendored the VBA grammar (#2969) and turned the SQL grammar into a self-maintained fork, both registered in the same places as your haskell fork, plus a jsonnet scanner fix and other work outside the grammars. So we merged current main into your branch once more, in one signed-off merge commit; your own commits are untouched. Only the grammar bookkeeping conflicted:

  1. THIRD_PARTY.md: main's thirteen self-maintained forks plus your haskell make fourteen. Your haskell paragraph sits between pascal and main's new sql paragraph, and main's tree-sitter-vba section is kept.
  2. MANIFEST.md summary line: main's 164 grammars, with haskell moved from vendored-from-upstream to self-maintained, gives 138 + 21 + 5 = 164. We recounted it from the tree (164 grammar directories, 164 parser.c files, ABI 9/78/77, so the ABI line is unchanged). haskell is still listed only in the self-maintained forks table.
  3. scripts/vendored-checksums.txt was regenerated with scripts/security-vendored.sh --update; compared with main, only the MANIFEST.md line and your haskell/parser.c line differ.

scripts/audit-license-provenance.py merged without a conflict: your FORKS row sits alphabetically between dotenv and javascript, next to main's new sql and vba rows. Every other file carries exactly your change.

CI is the check for this merge. If you push to this branch again, please pull first so the merge commit is kept. We will merge as soon as the required checks are green. Thanks again for a careful, well-documented fix!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working parsing/quality Graph extraction bugs, false positives, missing edges

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Haskell: an escaped char literal followed by a primed name swallows the next definition (vendored tree-sitter-haskell)

3 participants