Repository navigation
fix(haskell): stop escaped char literals at the closing quote (#2439) - #2547
Zhuoxi2000 wants to merge 6 commits into
Conversation
|
Thanks for opening this — it has been seen, and it is queued. This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence. Current review status: working through a backlog. What that means for this PR, concretely:
Things that will genuinely speed it up whenever review does happen:
If this fixes a bug, a reproduction we can run is worth more than a description of the symptom. Thanks for contributing, and sorry in advance for the wait. |
icecold009
left a comment
There was a problem hiding this comment.
AI-assisted review using OpenAI Codex and TypeSafe Jev (jev-1.13.0), on behalf of @icecold009. This is independent community feedback, not a maintainer decision.
I reviewed all four changed files and traced the generated lexer transitions for whitespace/newline stopping, escaped apostrophe handling, and the optional # suffix. I found no actionable defect in the patch. Both added regression cases ran in the hosted extraction suite (410 passed, 0 failed) on this head.
The current required PR / ci-ok check is failing because the Windows CLANG64 shard fails daemon_conflict_log_windows_concurrent_appends_are_not_dropped (tests/test_daemon_version.c:686). That failing test is outside the changed files; I could not establish a link to this Haskell parser patch. Please confirm it reproduces on the base or rerun before merge.
|
Drafted with an AI coding assistant (Claude). Thanks @icecold009 for going through the lexer changes. On the red |
|
Thanks for the detailed Haskell lexer explanation, and thanks to @icecold009 for the community review. I have read the Windows failure/rerun request; this sweep has not rerun CI or established its cause. The hand-patched generated lexer versus grammar regeneration choice also remains for maintainer review. |
|
Thank you, Edson, for the clear root cause and for the reproduce-first tests, which we want to keep exactly as they are - tracking #2439 down to the char-literal token instead of the scanner was the hard part. For token-level grammar fixes we keep a regenerable fork of the grammar (as we do for javascript/tsx and pascal), so that a future re-vendor does not have to re-derive hand-edited lexer states. So we would like to take you up on your offer: narrow the char regex in literal.js (for example to /'\('|[^'\s]+)'/) and regenerate parser.c with tree-sitter-cli 0.25.10. If you would rather not set up the fork wiring yourself, just say so and we are happy to do that part on our side and credit you. Thanks again for a careful fix. |
…eusData#2439) The vendored grammar lexes an escaped char literal with /'\\[^ ]*'/, whose [^ ]* also crosses newlines and quotes. With only newlines between '\n' and a primed name (g'), the char token runs through g', so f spans g' and g' gets no node. On one line, ['\n','\t'] lexes as a single char token with no parse error. Add regression tests for both forms. Signed-off-by: Edson <zhuoxi2000@gmail.com>
…DeusData#2439) An escaped char literal followed by a primed definition swallowed that definition: with f = '\n' g' = 2 the `char` token ran from '\n through g', so f spanned three lines, g' got no node and the valid module reported has_error. On one line ['\n','\t'] lexed as a single char token, silently. Root cause is upstream tree-sitter-haskell (the pin we vendor): the escape branch of the `char` token is /'\\[^ ]*'/, and [^ ]* also matches newlines and quotes, so the lexer's longest match runs on to the last ' before the next space. The external scanner is not involved. The regex is unchanged on upstream master, so there is no fix to pull. Fix: haskell becomes a self-maintained fork of the pin we already vendor (tree-sitter/tree-sitter-haskell @ 7fa19f195803, v0.23.1), following the javascript/tsx and pascal model. One patch (tools/tree-sitter-haskell/patches/escaped-char-literal.patch) narrows that one regex in grammar/literal.js to /'\\('|[^'\s]+)'/: a backslash, then either a quote ('\'') or a run of non-quote, non-whitespace characters, then the closing quote. '\\', '\x41', '\SOH', '\^A', '\1234' and MagicHash '\n'# lex as before; a stray ' after a literal is now an ERROR node instead of being absorbed. Node types are unchanged. Regenerated with tree-sitter-cli 0.25.10 (ABI 15, as before; scanner.c, unicode.h and tree_sitter/*.h untouched); parser.c has 0 non-ASCII bytes and the same state/symbol/token/field/production counts. Fidelity: regenerating the UNPATCHED grammar with 0.25.10 reproduces the previously vendored parser.c byte-for-byte (apart from the trailing newline). The patched parser passes all 724 cases of upstream's corpus and gives a tree identical to the unpatched one on upstream's examples/Basic.hs. Registered like pascal: MANIFEST.md fork row + note, audit FORKS map, THIRD_PARTY.md, vendored checksums. LICENSE stays upstream's MIT, byte-identical. Tests: the two reproduce-first extraction tests from the previous commit are unchanged (the DeusData#2439 repro, and the one-line list of seven escapes); the fork carries test/corpus/escaped_char_literal.txt; test_impact maps the fork's parser.c as repo-data like pascal's. Signed-off-by: Edson <zhuoxi2000@gmail.com>
340bb58 to
af8bac8
Compare
|
Done, reworked as a regenerable fork following the pascal model (#2211):
One thing for your call: following the pascal layout, Review map: of the 40 files, 33 are the new PR description updated to match. |
|
Thank you, Edson - this is exactly the rework we hoped for, and it came back within hours. We have been through it end to end: the patch reproduces your literal.js from the pinned upstream file, src/grammar.json differs from upstream only in that one regex, node-types.json is unchanged, and the regenerated parser.c only renumbers lexer states inside ts_lex, with every table count unchanged and no non-ASCII bytes. One maintainer sign-off on the fork registration is still pending on our side; nothing is needed from you in the meantime. On your question about tools/tree-sitter-haskell/src/parser.c: please keep it. It is byte-identical to the vendored copy, so the repository stores it only once, and it matches the layout of the pascal fork. |
The fork registration put the haskell fork row in the "Self-maintained forks" table and also replaced the original row in the "Vendored from verified upstream" table with the same text, so haskell was listed twice and the upstream table carried 140 rows against the 139 the summary line declares. pascal, the model for this fork, is listed only in the forks table; drop the copy from the upstream table so both tables agree with the summary counts (139 upstream, 18 self-maintained). The license audit is unaffected: haskell is routed through the FORKS map, and its manifest parser never matched the fork-format row. MANIFEST.md is itself covered by scripts/vendored-checksums.txt, so re-hash it with scripts/security-vendored.sh --update; that MANIFEST.md line is the only one that changes. The vendored haskell parser.c is not touched. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Brings the branch up to origin/main 9f278b7. One conflict, in scripts/vendored-checksums.txt: both sides re-hashed the MANIFEST.md line (main for its objectscript scanner patches, this branch for the haskell fork). Resolved by taking main's file and re-running scripts/security-vendored.sh --update on the merged tree; against origin/main the only lines that differ are MANIFEST.md (the merged file) and haskell/parser.c (this branch's regenerated parser). Every other file merged cleanly and carries exactly this branch's change. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
Thank you, Edson - this is a model rework. Tracking #2439 down to the escape branch of the We pushed two small commits on top of your branch so it can merge:
We checked the result locally: both of your tests fail with main's grammar and pass with yours, the extraction, language, lang_contract, grammar and test_impact suites pass, and the vendored integrity check passes. Since your branch now has these two new commits, please pull before pushing again. We will merge as soon as the required checks are green. Thanks again for a careful, well-documented fix. |
Resolve the vendored-registry conflicts with the vbnet grammar (DeusData#1179): keep both the haskell fork and vbnet entries in THIRD_PARTY.md and MANIFEST.md, recount the manifest totals (163 = 139 upstream + 19 self-maintained + 5 registry-disagreement; ABI line unchanged), and refresh the MANIFEST.md checksum. Signed-off-by: Edson <zhuoxi2000@gmail.com>
Brings the branch up to main 1ec2f26. Conflicts were the grammar bookkeeping only: - THIRD_PARTY.md: main's thirteen self-maintained forks (sql, vba added) plus haskell make fourteen; the haskell paragraph sits between pascal and main's new sql paragraph; main's tree-sitter-vba section is kept. - MANIFEST.md summary: main's 164 grammars with haskell moved from vendored-from-upstream to self-maintained: 138 + 21 + 5 = 164, recounted from the tree (164 grammar dirs, 164 parser.c, ABI 9/78/77). haskell stays only in the self-maintained forks table. - scripts/vendored-checksums.txt: regenerated with scripts/security-vendored.sh --update. scripts/audit-license-provenance.py merged cleanly: the haskell FORKS row sits alphabetically between dotenv and javascript, next to main's sql and vba rows. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
Thank you again, Edson, and thanks for merging main yourself on 11-11 - that kept this small. main moved on in the meantime: it vendored the VBA grammar (#2969) and turned the SQL grammar into a self-maintained fork, both registered in the same places as your haskell fork, plus a jsonnet scanner fix and other work outside the grammars. So we merged current main into your branch once more, in one signed-off merge commit; your own commits are untouched. Only the grammar bookkeeping conflicted:
scripts/audit-license-provenance.py merged without a conflict: your FORKS row sits alphabetically between dotenv and javascript, next to main's new sql and vba rows. Every other file carries exactly your change. CI is the check for this merge. If you push to this branch again, please pull first so the merge commit is kept. We will merge as soon as the required checks are green. Thanks again for a careful, well-documented fix! |
What does this PR do?
Fixes #2439
When a Haskell definition contains an escaped char literal (
'\n') and the next definition has a primed name (g'), the first definition swallows the second. With the repro from #2439,fspans lines 3-5 andg'gets no node. The parse also reportshas_erroreven though the module is valid. The same bug has a one-line form:['\n','\t']lexes as a singlechartoken and reports no error.Cause. The scanner is not the culprit. The fault is in the grammar's internal lexer:
grammar/literal.jsdefineschar: choice(/'[^']'/, /'\\[^ ]*'/). In the escape branch,[^ ]*also matches newlines and quotes, and tree-sitter keeps the longest match, so the token runs to the last'before the next space. In the repro that is the prime ing'. The regex is the same at our pin7fa19f195803and on upstream master, so there is no upstream fix to pull.Change (reworked per review, following the pascal model from #2211).
haskellbecomes a self-maintained fork of the pin we already vendor, tree-sitter/tree-sitter-haskell @7fa19f195803(v0.23.1):tools/tree-sitter-haskell/holds the pinned grammar source (grammar.js+grammar/), the one patchpatches/escaped-char-literal.patch(applied ingrammar/literal.js),package.json/tree-sitter.json, the regeneratedsrc/, andtest/corpus/escaped_char_literal.txt(3 cases,tree-sitter testpasses)./'\\('|[^'\s]+)'/: a backslash, then either a quote ('\'') or a run of non-quote, non-whitespace characters, then the closing quote.'\\','\x41','\SOH','\^A','\1234', MagicHash'\n'#lex as before, and upstream'sth.txtcorpus cases (TH quotes'foo,''Bar) still pass. Node types are unchanged (node-types.jsonis identical to the unpatched regeneration).internal/cbm/vendored/grammars/haskell/parser.cis regenerated withtree-sitter-cli 0.25.10(ABI 15, as before).scanner.c,unicode.handtree_sitter/*.hare untouched. The newparser.chas 0 non-ASCII bytes and the same state/symbol/token/field/production counts (12862/537/156/84/560).scripts/audit-license-provenance.pyFORKSmap,THIRD_PARTY.md, andscripts/vendored-checksums.txtviascripts/security-vendored.sh --update(only the MANIFEST.md andhaskell/parser.clines change). LICENSE stays upstream's MIT, byte-identical.tests/test_test_impact.cmapstools/tree-sitter-haskell/src/parser.casrepo-data, like pascal's.Fidelity. Regenerating the UNPATCHED grammar with 0.25.10 reproduces the previously vendored
parser.cbyte-for-byte (apart from the trailing newline), so theparser.cdiff is thecharlexer and nothing else. The patched parser passes all 724 cases of upstream's corpus (same as the unpatched regeneration) and gives a tree identical to the unpatched one on upstream'sexamples/Basic.hs.One behaviour difference from the earlier hand patch, for the better: an escaped char followed directly by a stray
'('\n'', not valid Haskell) is now reported as an ERROR node instead of being absorbed into the token.Checklist
git commit -s).scripts/check-dco.sh origin/main..HEADreports OK for both commits.make -f Makefile.cbm testhangs on this machine during ASan shadow-memory init, beforemain()runs (macOS 26.5, Apple clang 17), unrelated to this change. An unsanitized runner was used instead,make -f Makefile.cbm -j4 SANITIZE= BUILD_DIR=build/nosan build/nosan/test-runner, and the sanitized run is left to CI. Results with the fix:extraction: 488 passedpipeline: 412 passedlanguage: 250 passedlang_contract: 43 passednode_creation_probe: 80 passedgrammar_regression: 1 passedgrammar_labels: 2 passedgrammar_imports: 1 passedsecurity: 38 passedparse_coverage: 61 passedtest_impact: 89 passedscripts/security-vendored.sh:Vendored integrity check passed(1083 files)tests/test_vendored_integrity_contract.sh: OKpython3 scripts/audit-license-provenance.py: PASSED,grammars/haskell: IDENTICALnpx tree-sitter-cli@0.25.10 testintools/tree-sitter-haskell/: 3/3lint-ciruns cppcheck and clang-format overLINT_SRCS/LINT_HDRSonly, and this PR changes none of those files (the C changes are the vendoredparser.cand two test files), so the cppcheck and clang-format results onmainare unchanged;lint-no-suppressandscripts/lint-memory-core.pyare unaffected. clang-format was not run locally this time (not installed here): thetests/test_extraction.chunks are byte-identical to the first revision of this PR, andtests/test_test_impact.cgains one line mirroring its neighbour. clang-tidy was not run; it is the local pre-commit linter and not part oflint-ci.tests/test_extraction.c, unchanged from the first version of this PR, and the first commit adds only these tests:haskell_escaped_char_does_not_swallow_primed_defis the Haskell: an escaped char literal followed by a primed name swallows the next definition (vendored tree-sitter-haskell) #2439 repro. It assertsfends on line 3,g'andhexist, and there is no parse error.haskell_escaped_char_list_lexes_each_literalchecks the one-line form. It asserts that['\n','\t','\'','\\','\x41','\SOH','a']gives seven separatechartokens with no parse error.Test evidence
Before (test commit only, on current
mainf05c221):After (this PR):
AI assistance: this change was drafted with an AI coding assistant (Claude) and verified locally with the tests above.