fix(c): a function that writes a macro before a block still uses the macro

CHANGELOG.md

@@ -271,7 +271,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 - In C++, a function is now named by its declarator, never by something written in the clauses after its parameters. nlohmann/json's constructor `basic_json(CompatibleType&& val) noexcept(noexcept(JSONSerializer<U>::to_json(…)))` was missing from the index, replaced by a method `JSONSerializer::to_json` that doesn't exist and that other `JSONSerializer<T>::to_json(…)` calls linked to; a method declared with `throw(std::bad_alloc)` was indexed as `std::bad_alloc`, and a function with a trailing `requires std::integral<T>` as `std::integral<T>`. Re-index C++ projects after upgrading.
 - In C and C++, a header or source file named like a test, such as protocolbuffers/protobuf's `conformance_test.h` and `test_runner.h`, now counts as part of the code that `#include`s it, directly or through other headers. Before, calls from that code into it were dropped, so protobuf's conformance suites' `suite_.ReportFailure(…)` and `RunValidInputTest(…)` calls, and its unit tests' calls to the shared `TestUtil` helpers, linked to nothing. A test file that only tests include is still kept out of reach of the rest of your code. Re-index C and C++ projects after upgrading.
 - In C++, a class, struct or method now keeps the namespaces and the class it is declared in when the parser misreads something earlier in the file, such as an unknown macro in front of a member or in a class header. Before, everything after the misread code could lose them, or land inside a class it isn't in: in protocolbuffers/protobuf, `FieldDescriptor` was indexed without `google::protobuf::`, and in RocksDB, the `Opts` struct declared inside a cache table class was indexed outside it, so finding a class's base classes and the methods called on its objects fell back to guessing by name. Methods that had come loose from their class are its members again, and classes the misread code had hidden, like leveldb's POSIX file and environment classes, are now indexed. Re-index C++ projects after upgrading.
-- In C, an `#if` group whose branches aren't whole statements, like an `else if (…) { … }` arm kept behind `#ifdef`, an `if (…)` whose body comes after the `#endif`, or a function's first line that differs per build, no longer turns into functions named `if` that the rest of the function, or of the file, is filed under. Such a group is now read with all its branches in a row when that is still C, and as one build of the file when it isn't, so functions that were missing or nested inside another, like ST's `LL_SetFlashLatency` or tre's `tre_parse`, are indexed with their calls. A block macro written in capitals, like `ATOMIC_BLOCK(…) { … }`, is now read like a lowercase iterator macro. Re-index C projects after upgrading. Thanks @danusha2345.
+- In C, an `#if` group whose branches aren't whole statements, like an `else if (…) { … }` arm kept behind `#ifdef`, an `if (…)` whose body comes after the `#endif`, or a function's first line that differs per build, no longer turns into functions named `if` that the rest of the function, or of the file, is filed under. Such a group is now read with all its branches in a row when that is still C, and as one build of the file when it isn't, so functions that were missing or nested inside another, like ST's `LL_SetFlashLatency` or tre's `tre_parse`, are indexed with their calls. A block macro written in capitals, like `ATOMIC_BLOCK(…) { … }`, is now read like a lowercase iterator macro, and the function that writes it still counts as using the macro. Re-index C projects after upgrading. Thanks @danusha2345.
 - In C++, a call on a variable, parameter or member declared as a standard-library or other outside type with a lowercase name, like `std::string`, `std::vector`, `std::set` or `absl::flat_hash_set`, no longer links to a project method that only shares the method's name, even one of a project class in another namespace with the type's name, like your own `util::flat_hash_set`. A variable declared outside the calling function, like another function's or a file's `std::string text`, no longer links its calls to such a class either. In google/leveldb, `saved_key_.clear()` on a `std::string saved_key_;` reached `Slice::clear`, and in protocolbuffers/protobuf, `proto.append(…)` on a `std::string` reached an unrelated `LeftoverBuffer::append`. The same goes for `->` through a raw pointer like `std::string* out`, while `->` through a smart pointer, iterator or optional still links to the type it holds. Comments also no longer pass for a variable's declaration, so more calls now link to the method of the type the variable is really declared with. A call on a member reached through other members, like facebook/rocksdb's `options.cf_paths.emplace_back(…)` on a `std::vector`, `props->rep.db_id.size()` on a `std::string` or `bbto.filter_policy.reset(…)` on a `std::shared_ptr`, no longer links to a project method of that name either: those had reached `autovector::emplace_back`, `BlobContents::size` and `FSWritableFilePtr::reset`. The same goes for a member of an outside type like `absl::Status`. Re-index C++ projects after upgrading.
 - In C++, a class written with attribute macros the parser can't read now keeps its members. That covers several macros between `class` and the class name, like protocolbuffers/protobuf's generated `class PROTOBUF_EXPORT PROTOBUF_FUTURE_ADD_EARLY_WARN_UNUSED Any final`, a macro with arguments there, like RocksDB's `struct ALIGN_AS(64U) HandleImpl`, or one before a template specialization, and macros on a member: in front of it (`PROTOBUF_FUTURE_ADD_EARLY_NODISCARD`), between a pointer and its name (`PROTOBUF_NONNULL`), after its parameters or its name (`ABSL_ATTRIBUTE_LIFETIME_BOUND`, google/leveldb's `LOCKS_EXCLUDED(mu_)` and `GUARDED_BY(mu_)`), or alone on a line above a comment ({fmt}'s `FMT_BEGIN_EXPORT`). Before, protobuf's generated message classes were missing and their members, like `GetDescriptor`, `GetReflection` and `unknown_fields`, were indexed as functions outside any class, and leveldb's annotated methods were indexed under the annotation's name, so calls to them linked to nothing. Re-index C++ projects after upgrading.
 - In C++, a call on a range-based `for` loop's variable, like `suite->SetVerbose(verbose)` inside `for (ConformanceTestSuite *suite : suites)`, now links to the method of the type the loop declares. Before, the loop's declaration was never read, so the call was matched by the variable's name alone: in RocksDB, `log->file()` on a `log::Writer` went to `BlobLogWriter::file`, and a loop variable named like a member of its class was taken for that member. The variable counts only inside its loop. Re-index C++ projects after upgrading.

__tests__/c-statement-macro-names.test.ts

@@ -0,0 +1,226 @@
+/**
+ * A C macro used as a statement, with a block or the next statement where its
+ * `;` would be (`TEST_CONTEXT("…") {` in redis, `XXH_ASSERT(c)` in xxhash), is
+ * blanked before parsing: the parser reads a call followed by a block as an
+ * error, or as a K&R function. The blank took the macro's name with it, so the
+ * function no longer used the macro, and a change to the macro missed it.
+ *
+ * In a statement of a function body the name stays, as `name;`. Anywhere else a
+ * `;` would be the error the blank avoids, so the whole call still goes.
+ */
+import { describe, it, expect, beforeAll } from 'vitest';
+import { extractFromSource } from '../src/extraction';
+import { initGrammars, loadGrammarsForLanguages, getParser } from '../src/extraction/grammars';
+import { blankCStatementMacroCalls } from '../src/extraction/languages/c-cpp';
+import { EXTRACTORS } from '../src/extraction/languages';
+
+/** The macros a function references, by the value references extraction writes. */
+function macrosUsedBy(source: string, fn: string): string[] {
+  const result = extractFromSource('t.c', source, 'c');
+  const name = (id: string): string | undefined => result.nodes.find((n) => n.id === id)?.name;
+  const holder = result.nodes.find((n) => n.name === fn);
+  expect(holder, fn).toBeDefined();
+  return result.edges
+    .filter((e) => e.kind === 'references' && e.source === holder!.id)
+    .map((e) => name(e.target)!)
+    .sort();
+}
+
+describe('a C macro used as a statement', () => {
+  beforeAll(async () => {
+    await initGrammars();
+    await loadGrammarsForLanguages(['c']);
+  });
+
+  it('is still used by a function that writes it before a block', () => {
+    const source = [
+      '#define TEST_CONTEXT(context) printf("%s", context);',
+      'int use(const char *s);',
+      'int mstrTest(void) {',
+      '    TEST_CONTEXT("Create simple short mstr") {',
+      '        use("foo");',
+      '    }',
+      '    return 0;',
+      '}',
+      '',
+    ].join('\n');
+    expect(macrosUsedBy(source, 'mstrTest')).toEqual(['TEST_CONTEXT']);
+    const result = extractFromSource('t.c', source, 'c');
+    expect(result.unresolvedReferences.filter((r) => r.referenceKind === 'calls').map((r) => r.referenceName)).toContain('use');
+  });
+
+  it('is still used when the next statement stands where its `;` would be', () => {
+    const source = [
+      '#define XXH_ASSERT(c) ((void)0)',
+      'int h(int len) {',
+      '    int a = len;',
+      '    XXH_ASSERT(a != 0)',
+      '    {   int b = a;',
+      '        return b;',
+      '    }',
+      '}',
+      '',
+    ].join('\n');
+    expect(macrosUsedBy(source, 'h')).toEqual(['XXH_ASSERT']);
+  });
+
+  it('is still used in a run of them, each without its `;`', () => {
+    const source = [
+      '#define check_len(c) ((void)0)',
+      'int h(int len) {',
+      '    check_len(len != 0)',
+      '    check_len(len != 1)',
+      '    {',
+      '        return len;',
+      '    }',
+      '}',
+      '',
+    ].join('\n');
+    expect(blankCStatementMacroCalls(source).match(/check_len;/g)).toHaveLength(2);
+    expect(macrosUsedBy(source, 'h')).toEqual(['check_len']);
+  });
+
+  it('is still used over lines, after a label and in a nested block', () => {
+    const source = [
+      '#define EACH(a, b) for (;;)',
+      'void f(int x, void *head) {',
+      '    EACH(x,',
+      '         head) {',
+      '        x--;',
+      '    }',
+      '}',
+      'void g(int x) {',
+      '    if (x) {',
+      '        EACH(x, 0) {',
+      '            x--;',
+      '        }',
+      '    } else',
+      '        EACH(x, 1) {',
+      '            x++;',
+      '        }',
+      '    do',
+      '        EACH(x, 2) {',
+      '            x++;',
+      '        }',
+      '    while (x);',
+      'out:',
+      '    EACH(x, 3) {',
+      '        x++;',
+      '    }',
+      '}',
+      '',
+    ].join('\n');
+    expect(macrosUsedBy(source, 'f')).toEqual(['EACH']);
+    expect(macrosUsedBy(source, 'g')).toEqual(['EACH']);
+    const output = blankCStatementMacroCalls(source);
+    expect(output.length).toBe(source.length);
+    expect(output.match(/EACH;/g)).toHaveLength(3);
+    const tree = getParser('c')!.parse(EXTRACTORS.c!.preParse!(source))!;
+    expect(tree.rootNode.hasError).toBe(false);
+  });
+
+  it('is still used in the block of an `else`, a `do`, a `case` and a statement expression', () => {
+    const source = [
+      '#define EACH(a) for (;;)',
+      'int g(int x) {',
+      '    if (x) {',
+      '        x--;',
+      '    } else {',
+      '        EACH(x) {',
+      '            x--;',
+      '        }',
+      '    }',
+      '    do {',
+      '        EACH(x) {',
+      '            x--;',
+      '        }',
+      '    } while (x);',
+      '    switch (x) {',
+      '    case 1:',
+      '        EACH(x) {',
+      '            x--;',
+      '        }',
+      '        break;',
+      '    }',
+      '    return ({',
+      '        EACH(x) {',
+      '            x--;',
+      '        }',
+      '        x;',
+      '    });',
+      '}',
+      '',
+    ].join('\n');
+    expect(blankCStatementMacroCalls(source).match(/EACH;/g)).toHaveLength(4);
+    expect(getParser('c')!.parse(EXTRACTORS.c!.preParse!(source))!.rootNode.hasError).toBe(false);
+  });
+
+  it('goes whole as the body of a control statement, whose `else` or `while` comes after it', () => {
+    const source = [
+      'void g(int x) {',
+      '    if (x)',
+      '        EACH(x, 0) {',
+      '            x--;',
+      '        }',
+      '    else',
+      '        EACH(x, 1) {',
+      '            x++;',
+      '        }',
+      '    do',
+      '        EACH(x, 2) {',
+      '            x++;',
+      '        }',
+      '    while (x);',
+      '}',
+      '',
+    ].join('\n');
+    expect(blankCStatementMacroCalls(source)).not.toContain('EACH');
+    expect(getParser('c')!.parse(EXTRACTORS.c!.preParse!(source))!.rootNode.hasError).toBe(false);
+  });
+
+  it('goes whole where no statement can stand', () => {
+    const parses = (source: string): void => {
+      const preParsed = EXTRACTORS.c!.preParse!(source);
+      expect(preParsed).not.toMatch(/\b(?:entry|members|values|inner|tail);/);
+      expect(getParser('c')!.parse(preParsed)!.rootNode.hasError).toBe(false);
+    };
+    // In an initializer.
+    parses(['static const int table[] = {', '    entry(a)', '    entry(b)', '    last', '};', ''].join('\n'));
+    // In a struct's or an enum's body.
+    parses(['struct s {', '    members(a)', '    int x;', '};', ''].join('\n'));
+    parses(['enum e {', '    values(a)', '    last', '};', ''].join('\n'));
+    // In a function, in an initializer again.
+    parses(['void f(void) {', '    int t[] = {', '        entry(a)', '        entry(b)', '        last', '    };', '    (void)t;', '}', ''].join('\n'));
+    // In the arguments of another macro.
+    parses(['void f(void) {', '    OUTER(', '        inner(a)', '        inner(b)', '        last', '    );', '}', ''].join('\n'));
+    // In an expression over lines.
+    parses(['int f(int a, int b) {', '    return a +', '        tail(a)', '        b;', '}', ''].join('\n'));
+  });
+
+  it('keeps every line and column, and a `\\r\\n` line end', () => {
+    const source = 'void f(void) {\r\n    EACH(1) {\r\n    }\r\n    ASSERT(2)\r\n    g();\r\n}\r\n';
+    const output = blankCStatementMacroCalls(source);
+    expect(output).toBe('void f(void) {\r\n    EACH;   {\r\n    }\r\n    ASSERT;  \r\n    g();\r\n}\r\n');
+  });
+
+  it('reads a branch of `#if` that leaves a brace open as no statement, without losing the rest of the file', () => {
+    const source = [
+      'void f(int a) {',
+      '#ifdef X',
+      '    if (a) {',
+      '#else',
+      '    if (!a) {',
+      '#endif',
+      '        a--;',
+      '    }',
+      '}',
+      'void g(int a) {',
+      '    EACH(a) {',
+      '        a--;',
+      '    }',
+      '}',
+      '',
+    ].join('\n');
+    expect(blankCStatementMacroCalls(source)).toContain('    EACH;   {');
+  });
+});

__tests__/cpp-raw-string-preparse-1505.test.ts

@@ -101,13 +101,13 @@ value UPARAM(ref) UE_DEPRECATED(
     { name: 'line-leading', blank: blankCppAnnotationMacroCalls, head: '', macro: 'ANNOTATE', tail: '\nint helper_after() { return 1; }\n' },
     { name: 'inline', blank: blankCppInlineAnnotationMacros, head: 'using Alias ', macro: 'UE_DEPRECATED', tail: ' = int;\n' },
     { name: 'C parameterized', blank: blankCParameterizedAnnotationMacros, head: 'static void ', macro: '__section', tail: ' helper_after(void) {}\n' },
-    { name: 'C iterator', blank: blankCStatementMacroCalls, head: 'void iterate() {\n  ', macro: 'for_each_item', tail: ' {\n    visit();\n  }\n}\n' },
-  ])('balances a genuine $name macro containing a raw-string argument', ({ blank, head, macro, tail }) => {
+    { name: 'C iterator', blank: blankCStatementMacroCalls, head: 'void iterate() {\n  ', macro: 'for_each_item', tail: ' {\n    visit();\n  }\n}\n', kept: 'for_each_item;' },
+  ])('balances a genuine $name macro containing a raw-string argument', ({ blank, head, macro, tail, kept = '' }) => {
     const annotation = `${macro}(R"TAG(" ) unbalanced ( " \\
 UE_DEPRECATED(
 )TAG")`;
     expect(blank(head + annotation + tail))
-      .toBe(head + annotation.replace(/[^\r\n]/g, ' ') + tail);
+      .toBe(head + kept + annotation.slice(kept.length).replace(/[^\r\n]/g, ' ') + tail);
   });
 
   it('balances a C declaration macro with a raw-string argument', () => {

__tests__/extraction.test.ts

@@ -14238,7 +14238,8 @@ describe('C/C++ kernel-port preParse blanks (R7a)', () => {
     ].join('\n');
     const out = blankCStatementMacroCalls(src);
     expect(out.length).toBe(src.length);
-    expect(out).not.toContain('list_for_each_entry');
+    expect(out).toContain('list_for_each_entry;');
+    expect(out).not.toContain('head, member');
     expect(out).toContain('use(pos);');
     // A real call statement ends with `;` — untouched.
     expect(out).toContain('use(pos);');
@@ -14445,7 +14446,7 @@ describe('C/C++ kernel-port preParse blanks (R7a)', () => {
     ].join('\n');
     const out = blankCStatementMacroCalls(src);
     expect(out.length).toBe(src.length);
-    expect(out).not.toContain('hlist_for_each_entry_rcu');
+    expect(out).toContain('hlist_for_each_entry_rcu;');
     expect(out).not.toContain('lockdep_is_held');
     expect(out).toContain('use(p);');
     // A wrapped REAL call ends in `;` — untouched.