Skip to content

fix(extract): read a CRLF line splice in C-family strings as LF does (#1546) - #2983

Open
b84330808 wants to merge 2 commits into
DeusData:mainfrom
b84330808:fix-crlf-line-splice
Open

b84330808 wants to merge 2 commits into
DeusData:mainfrom
b84330808:fix-crlf-line-splice

Conversation

@b84330808

@b84330808 b84330808 commented Nov 14, 2026 •

Copy link
Copy Markdown

What does this PR do?

Fixes #1546.

The cause is in the grammar, so it isn't Windows-specific. tree-sitter-c's escape_sequence is \ followed by one character (/[^xuU]/). For a line splice inside a string literal in a CRLF file, that character is the CR. The LF then ends the literal, and the rest of the function, often the rest of the file, ends up in one ERROR region. cpp, cuda, objc, glsl, hlsl, ispc and slang share the rule. Upstream tree-sitter-c master still has it.

On current main (b62a2ff), cbm_extract_file on @noohi's CRLF repro.cpp (sha256 57d03ea7…) returns this:

has_error=0 parse_incomplete=1 defs=3
  Function Head         lines 1-4
  Function Middle       lines 6-20

TrailerOne and TrailerTwo are missing. The LF copy gives all four functions. When the file also has a #define, the preprocessed second pass gets some defs back. Even then, the variable before the next function spans too many lines and a call shows up at module level. The third test below covers that case.

The fix rewrites each \ CR LF to space \ LF before parsing these languages. It lives in extract_parse_bytes, next to the PHP inline-HTML mask from #2000. The length and the line break stay in place, so offsets and line numbers still point into the file on disk. The next line is byte-identical. The buffer is copied only when a splice is found. Splices in macros and // comments already parsed with CRLF and still do.

Tests, in tests/test_extraction.c:

  • extract_cpp_crlf_string_splice_keeps_later_functions_issue1546: the reporter's file byte for byte. It checks all four functions and their lines.
  • extract_c_family_crlf_string_splice_parses_clean_issue1546: the same check for C, C++, CUDA, ObjC, GLSL, HLSL, ISPC and Slang.
  • extract_c_crlf_splices_in_macro_comment_and_string_issue1546: a macro, a comment and a string splice in one CRLF file. It goes through the preprocessed second pass.

How I ran them: this Windows machine has no make or zlib, so I couldn't run make -f Makefile.cbm test. I built a reduced test binary with zig cc 0.16 (clang 20). It has the foundation and extraction sources, lsp_all, the tree-sitter runtime and the eight C-family grammars, with the other grammars stubbed out. In it I ran the 87 C-family tests of the extraction suite plus the three new ones:

passed failed
main b62a2ff 80 7 (the 3 new tests + 4 below)
this branch 83 4

The 4 failures in both runs come from grammars I left out of the reduced build: three objectscript_macro_* tests and the cmd.json case of extract_variants_only_in_c_preprocessor_languages_c1. cbm.c and the test file compile clean with -Wall -Wextra -Werror. The added hunks have no clang-format 20.1.8 drift.

I also ran it on Linux (ubuntu-24.04 GitHub Actions on my fork):

  • this branch: make -f Makefile.cbm test-focused TEST_SUITES=extraction gives 542 passed, and scripts/test.sh ends with All tests passed.
  • cbm.c reverted to main with the new tests kept: all three new tests fail, e.g. extract_cpp_crlf_string_splice_keeps_later_functions_issue1546 FAIL tests/test_extraction.c:442: ASSERT(!(r->parse_incomplete)).

I haven't run lint-ci (cppcheck, clang-tidy).

I used Claude Code to write this change.

Checklist

  • Every commit is signed off (git commit -s) — required, CI rejects
    unsigned commits (DCO, see CONTRIBUTING.md)
  • Tests pass locally (make -f Makefile.cbm test)
  • Lint passes (make -f Makefile.cbm lint-ci)
  • New behavior is covered by a test (reproduce-first for bug fixes)

…eusData#1546)

tree-sitter-c and the grammars built on it (cpp, cuda, objc, glsl, hlsl,
ispc, slang) read a backslash line splice inside a string literal as an
escape_sequence, `\` plus one character. In a CRLF file that character is
the CR, the LF then ends the literal, and the rest of the function, often
the rest of the file, falls into one ERROR region. In the issue's file the
two functions after the literal are missing and Middle spans to the end of
the file.

Before parsing those languages, rewrite each `\` CR LF as space `\` LF in
extract_parse_bytes, next to the PHP inline-HTML mask (DeusData#2000). The rewrite
keeps the length and the line break, so offsets and line numbers still
refer to the file on disk, and the buffer is copied only when a splice is
found.

Signed-off-by: Wei-Lun Li <b84330808@gmail.com>
@b84330808
b84330808 requested a review from DeusData as a code owner November 14, 2026 04:55
@github-actions

Copy link
Copy Markdown

Thanks for opening this — it has been seen, and it is queued.

This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence.

Current review status: working through a backlog. 0.9.1-rc.1 is out, so the release freeze that held reviews is over — but it left a large queue of open pull requests behind it, and we are reading through them oldest-first. The background is in discussion #1144.

What that means for this PR, concretely:

  • It will not be closed for inactivity. No stale bot touches pull requests here.
  • It may still sit a while before a human reads it. That is on us, not on you.
  • Older PRs are read first, so a recent one is not being skipped — it is behind a queue.

Things that will genuinely speed it up whenever review does happen:

  • Keep it rebased on main — the tree is moving quickly right now, and a conflicting branch cannot be reviewed as the diff you intended.
  • Get CI green, or say which failures you believe are pre-existing.
  • Keep the change to one claim. Bundled features and refactors get split before they get merged, which costs you a round trip.
  • Every commit needs a sign-off (git commit -s) — CI enforces DCO.

If this fixes a bug, a reproduction we can run is worth more than a description of the symptom.

Thanks for contributing, and sorry in advance for the wait.

…eusData#1546)

The per-file extract now parses C-family files with each backslash CR LF
rewritten as space backslash LF, but the cross-file C/C++ LSP pass parses
the file a second time from the original bytes: the parallel driver from
its retained copy, the sequential driver from a fresh read. Both end in
c_lsp_cross_with_registry_run or c_lsp_cross_run, which parsed those
bytes as they came, so after a CRLF string splice the rest of the file
was one ERROR region again for that pass, and a call only that pass
resolves was lost.

Export the mask as cbm_c_mask_crlf_splices (declared in cbm.h next to
the PHP inline-HTML mask) and apply it at the top of both cross bodies,
before the test-owner check, the way cbm_run_php_lsp_cross_with_bindings
does for DeusData#2000. The owner digest then sees the same bytes the extract
hashed. The copy comes from the call's arena and is made only when a
splice is found.

A 66-file C++ repo (parallel driver) with the reporter's string splice,
followed by a call through a struct declared in another header: LF 208
edges; CRLF 207 before (after_one -> ops.run, lsp_field_access, missing)
and 208 after, byte-identical to LF. The 3-file version (sequential
driver): 28, 27, 28, also identical to LF. redis 4f20cb48: the LF graph
(40,941 nodes, 141,136 edges) and the CRLF graph (40,940 nodes, 141,144
edges) are unchanged.

Tests: a pipeline test through both drivers with the shared C/C++
registry (the parallel harness can now opt into it, like the C# one), and
a c_lsp test through the per-file, shared-registry and batch entries,
each with an LF control. Both fail without the two calls; removing either
call alone fails the test that covers its body.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
@DeusData

Copy link
Copy Markdown
Owner

Hi @b84330808, thank you very much for this, and welcome to the project! This is a really careful first contribution. You tracked #1546 down to the actual grammar rule instead of treating it as a Windows quirk. You put the fix right next to the #2000 mask, where the next reader will look for it, and kept offsets and line numbers pointing at the file on disk. You also used the reporter's file byte for byte as the test. Spelling out exactly how you ran the tests (the reduced build, the four unrelated failures, the revert run on your fork) made this much easier to review. Thank you.

Here is what we checked on your commit (macOS arm64, PR head 5830c55, merge base b62a2ff):

  • New tests, RED and GREEN. With cbm.c reset to the merge base, exactly your three new tests fail, each for the right reason. With your cbm.c the extraction suite passes (542 tests) under ASan/UBSan, and so does the complexity suite (10 tests).
  • All eight grammars. With the CRLF string splice, every one of C, C++, CUDA, ObjC, GLSL, HLSL, ISPC and Slang is parse-partial on main and clean with this PR. C# is correctly left alone.
  • Real code. We used redis at 4f20cb48 (1,768 indexed files). With LF line endings, main and this PR produce byte-identical graphs (40,941 nodes, 141,136 edges). On a copy with every .c/.h file converted to CRLF, the PR clears the parse-partial flag on src/aof.c (the one string splice inside a function), and that function's structure fields go back to their LF values.
  • Edge cases, LF and CRLF. "a\\ followed by CRLF, a splice as the last three bytes of the file, a backslash-CR without LF, splices in comments and macro continuations, a splice inside an identifier, a C++ raw string, three splices in a row, and a string-literal call argument all behave as expected.

We pushed one small follow-up commit to your branch: 8efb059.

The review turned up one gap. Per-file extraction now parses the rewritten bytes, but the cross-file C/C++ LSP pass parses each file a second time from the original bytes: the parallel driver from the copy it keeps after extraction, the sequential driver from a fresh read. Both end in c_lsp_cross_with_registry_run and c_lsp_cross_run (internal/cbm/lsp/c_lsp.c), so after a CRLF string splice the rest of the file was one ERROR region again for that pass, and a call that only the cross-file pass can resolve went missing. The change was small and you had allowed maintainer edits, so we made it ourselves rather than ask you for another round. Your commit is untouched; ours sits on top of it.

What the follow-up does:

  1. Your c_mask_crlf_splices becomes the public cbm_c_mask_crlf_splices, declared in cbm.h next to cbm_php_mask_inline_html. c_splice_lang stays where you put it, in extract_parse_bytes.
  2. Both C/C++ cross bodies call it first, before the test-owner check, the same way cbm_run_php_lsp_cross_with_bindings re-applies the PHP: symbols declared after an inline-HTML block never reach the graph, and the whole tail of a valid file is reported parse_partial (0.10.8) #2000 mask. The owner digest check then sees the same bytes the extraction hashed. The copy comes from the call's arena and is only made when a splice is found.
  3. Two tests, each with an LF control: a pipeline test in tests/test_parallel.c that runs both drivers (the parallel test harness can now use the shared C/C++ registry, as it already could for C#), and a test in tests/test_c_lsp.c that runs the per-file, shared-registry and batch entry points. Both fail without the two new calls, and removing either call on its own fails the test that covers it.

The numbers:

  • A 66-file C++ repo (parallel driver) with your reporter's Middle() followed by a call through a struct declared in another header: 208 edges with LF. With CRLF it had 207 before the follow-up (the CALLS edge after_one -> ops.run, lsp_field_access, was missing) and has 208 after, byte-identical to the LF graph. A 3-file version (sequential driver): 28 with LF, 27 before, 28 after, also identical to LF.
  • redis 4f20cb48: the LF graph and the CRLF graph are both byte-identical before and after the follow-up.
  • Suites under ASan/UBSan: extraction 542, c_lsp 778, parallel 92, pipeline 491, complexity 10, test_conventions 15 and lsp_resolution_probe 89 tests, all passing. clang-format, cppcheck and the memory-core lint are clean on the changed code, and the branch still merges cleanly into current main.

A note, no change needed. Because extract_parse_bytes replaces source for the whole extraction, stored text that spans a splice carries the rewritten bytes. On the CRLF redis copy, 12 macro signatures and 4 docstrings show space-backslash-LF where the file has backslash-CR-LF. Offsets and line numbers are unaffected, and code snippets are read from disk, so we are fine with this trade-off. We mention it only so it is a known effect.

If you push anything else to this branch, please pull first (git pull on fix-crlf-line-splice), so your next push builds on our commit instead of replacing it.

Next, CI runs on the updated branch. Once it is green, a maintainer will make the merge decision. Thanks again for the thorough work!

@b84330808

Copy link
Copy Markdown
Author

(Written with Claude Code.) Thanks a lot for the thorough review and for the follow-up. Good catch on the cross-file LSP pass re-reading the original bytes. Noted on the stored-text trade-off, and I'll pull before any further push.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

C/C++: a CRLF file with a backslash-continued string literal swallows the rest of the file, and every later function is missing from the graph

2 participants