Repository navigation
Conversation
…eusData#1546) tree-sitter-c and the grammars built on it (cpp, cuda, objc, glsl, hlsl, ispc, slang) read a backslash line splice inside a string literal as an escape_sequence, `\` plus one character. In a CRLF file that character is the CR, the LF then ends the literal, and the rest of the function, often the rest of the file, falls into one ERROR region. In the issue's file the two functions after the literal are missing and Middle spans to the end of the file. Before parsing those languages, rewrite each `\` CR LF as space `\` LF in extract_parse_bytes, next to the PHP inline-HTML mask (DeusData#2000). The rewrite keeps the length and the line break, so offsets and line numbers still refer to the file on disk, and the buffer is copied only when a splice is found. Signed-off-by: Wei-Lun Li <b84330808@gmail.com>
|
Thanks for opening this — it has been seen, and it is queued. This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence. Current review status: working through a backlog. What that means for this PR, concretely:
Things that will genuinely speed it up whenever review does happen:
If this fixes a bug, a reproduction we can run is worth more than a description of the symptom. Thanks for contributing, and sorry in advance for the wait. |
…eusData#1546) The per-file extract now parses C-family files with each backslash CR LF rewritten as space backslash LF, but the cross-file C/C++ LSP pass parses the file a second time from the original bytes: the parallel driver from its retained copy, the sequential driver from a fresh read. Both end in c_lsp_cross_with_registry_run or c_lsp_cross_run, which parsed those bytes as they came, so after a CRLF string splice the rest of the file was one ERROR region again for that pass, and a call only that pass resolves was lost. Export the mask as cbm_c_mask_crlf_splices (declared in cbm.h next to the PHP inline-HTML mask) and apply it at the top of both cross bodies, before the test-owner check, the way cbm_run_php_lsp_cross_with_bindings does for DeusData#2000. The owner digest then sees the same bytes the extract hashed. The copy comes from the call's arena and is made only when a splice is found. A 66-file C++ repo (parallel driver) with the reporter's string splice, followed by a call through a struct declared in another header: LF 208 edges; CRLF 207 before (after_one -> ops.run, lsp_field_access, missing) and 208 after, byte-identical to LF. The 3-file version (sequential driver): 28, 27, 28, also identical to LF. redis 4f20cb48: the LF graph (40,941 nodes, 141,136 edges) and the CRLF graph (40,940 nodes, 141,144 edges) are unchanged. Tests: a pipeline test through both drivers with the shared C/C++ registry (the parallel harness can now opt into it, like the C# one), and a c_lsp test through the per-file, shared-registry and batch entries, each with an LF control. Both fail without the two calls; removing either call alone fails the test that covers its body. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
Hi @b84330808, thank you very much for this, and welcome to the project! This is a really careful first contribution. You tracked #1546 down to the actual grammar rule instead of treating it as a Windows quirk. You put the fix right next to the #2000 mask, where the next reader will look for it, and kept offsets and line numbers pointing at the file on disk. You also used the reporter's file byte for byte as the test. Spelling out exactly how you ran the tests (the reduced build, the four unrelated failures, the revert run on your fork) made this much easier to review. Thank you. Here is what we checked on your commit (macOS arm64, PR head 5830c55, merge base b62a2ff):
We pushed one small follow-up commit to your branch: 8efb059. The review turned up one gap. Per-file extraction now parses the rewritten bytes, but the cross-file C/C++ LSP pass parses each file a second time from the original bytes: the parallel driver from the copy it keeps after extraction, the sequential driver from a fresh read. Both end in What the follow-up does:
The numbers:
A note, no change needed. Because If you push anything else to this branch, please pull first ( Next, CI runs on the updated branch. Once it is green, a maintainer will make the merge decision. Thanks again for the thorough work! |
|
(Written with Claude Code.) Thanks a lot for the thorough review and for the follow-up. Good catch on the cross-file LSP pass re-reading the original bytes. Noted on the stored-text trade-off, and I'll pull before any further push. |
What does this PR do?
Fixes #1546.
The cause is in the grammar, so it isn't Windows-specific. tree-sitter-c's
escape_sequenceis\followed by one character (/[^xuU]/). For a line splice inside a string literal in a CRLF file, that character is the CR. The LF then ends the literal, and the rest of the function, often the rest of the file, ends up in one ERROR region. cpp, cuda, objc, glsl, hlsl, ispc and slang share the rule. Upstream tree-sitter-c master still has it.On current main (b62a2ff),
cbm_extract_fileon @noohi's CRLFrepro.cpp(sha256 57d03ea7…) returns this:TrailerOne and TrailerTwo are missing. The LF copy gives all four functions. When the file also has a
#define, the preprocessed second pass gets some defs back. Even then, the variable before the next function spans too many lines and a call shows up at module level. The third test below covers that case.The fix rewrites each
\CR LF to space\LF before parsing these languages. It lives inextract_parse_bytes, next to the PHP inline-HTML mask from #2000. The length and the line break stay in place, so offsets and line numbers still point into the file on disk. The next line is byte-identical. The buffer is copied only when a splice is found. Splices in macros and//comments already parsed with CRLF and still do.Tests, in
tests/test_extraction.c:extract_cpp_crlf_string_splice_keeps_later_functions_issue1546: the reporter's file byte for byte. It checks all four functions and their lines.extract_c_family_crlf_string_splice_parses_clean_issue1546: the same check for C, C++, CUDA, ObjC, GLSL, HLSL, ISPC and Slang.extract_c_crlf_splices_in_macro_comment_and_string_issue1546: a macro, a comment and a string splice in one CRLF file. It goes through the preprocessed second pass.How I ran them: this Windows machine has no make or zlib, so I couldn't run
make -f Makefile.cbm test. I built a reduced test binary with zig cc 0.16 (clang 20). It has the foundation and extraction sources, lsp_all, the tree-sitter runtime and the eight C-family grammars, with the other grammars stubbed out. In it I ran the 87 C-family tests of the extraction suite plus the three new ones:The 4 failures in both runs come from grammars I left out of the reduced build: three objectscript_macro_* tests and the cmd.json case of
extract_variants_only_in_c_preprocessor_languages_c1.cbm.cand the test file compile clean with-Wall -Wextra -Werror. The added hunks have no clang-format 20.1.8 drift.I also ran it on Linux (ubuntu-24.04 GitHub Actions on my fork):
make -f Makefile.cbm test-focused TEST_SUITES=extractiongives 542 passed, andscripts/test.shends withAll tests passed.cbm.creverted to main with the new tests kept: all three new tests fail, e.g.extract_cpp_crlf_string_splice_keeps_later_functions_issue1546 FAIL tests/test_extraction.c:442: ASSERT(!(r->parse_incomplete)).I haven't run
lint-ci(cppcheck, clang-tidy).I used Claude Code to write this change.
Checklist
git commit -s) — required, CI rejectsunsigned commits (DCO, see CONTRIBUTING.md)
make -f Makefile.cbm test)make -f Makefile.cbm lint-ci)