Skip to content

chore(vendor): vendor the VB6/VBA tree-sitter grammar - #2969

Merged
DeusData merged 3 commits into
mainfrom
feat/vendor-vba-grammar
Nov 14, 2026
Merged

DeusData merged 3 commits into
mainfrom
feat/vendor-vba-grammar

Conversation

@DeusData

Copy link
Copy Markdown
Owner

What this vendors

A Visual Basic for Applications / Visual Basic 6 tree-sitter grammar, as a
self-maintained fork in tools/tree-sitter-vba/, generated into
internal/cbm/vendored/grammars/vba/:

  • Grammar: harumiWeb/tree-sitter-vba
    @ 83c5c3c3c7210468fb6745b585b5fb6532aa2376 (tag v0.14.5, upstream
    HEAD), its vba dialect. MIT, (c) 2026 harumiWeb.
  • No local patch. Upstream does not track its generated parser (its
    ADR 0002), so the fork carries upstream's grammar source in upstream's own
    layout and the parser is generated here with tree-sitter-cli 0.25.10
    (ABI 15), the generator every fork in this repository uses.

This is the precondition for VB6/VBA extraction (#2564, which we port onto
this grammar after #1179 merges). It registers no language.

Why this pin (v0.14.5, not v0.13.0)

#2564 carried v0.13.0 (63b2f8d0). Both pins were regenerated with the same
CLI and run over the same three corpora (below):

v0.13.0 v0.14.5
procedures outside ERROR nodes 34,573 34,573
VBCCR files without ERROR 98 114
expression_statement nodes 1,276 56

The procedure count is the same, but v0.13.0 misreads statements without
leaving an ERROR node: a bare call such as Err.Raise a, b, c after a label
becomes an expression statement and loses its callee. That is a wrong CALLS
edge nobody would see in a parse-error count. Upstream fixed these in
v0.14.0 to v0.14.3, and fixed Case obj Is Nothing (an ERROR in v0.13.0) in
v0.14.1.

Why the vba dialect (not upstream's vb6 one)

Since v0.14.0 upstream generates a second, vb6 dialect from the same core.
Measured with the tree-sitter CLI on two pinned MIT VB6 code bases and
upstream's VBA examples (vba / vb6):

corpus files files without ERROR procedures outside ERROR
kellyethridge/VBCorLib @ 100acbbae0c0 (pure VB6) 655 643 / 653 14,084 / 14,084
Kr00l/VBCCR @ 4b86c5c2c075 (dual-target VB6, 70 .ctl) 161 114 / 61 12,385 / 1,552
upstream examples/ (VBA) 481 481 / 413 8,104 / 6,573

Real VB6 libraries keep a VBA7 branch next to the VB6 one
(#If VBA7 Then ... Declare PtrSafe ... LongPtr ... #Else ... #End If).
A tree-sitter grammar parses both branches, and the vb6 dialect rejects the
VBA7 forms; its error recovery then covers 83 of the 161 VBCCR files almost
entirely. In no file of any corpus does the vba dialect see fewer procedures
than vb6. The vb6 dialect is cleaner only on VB6 component reference lines
(see Known gaps), and its parser.c is 75.8 MB, above the 50 MB line.

Provenance, by hash

  • tools/tree-sitter-vba/common/define-grammar.js is upstream's file
    byte-identical (git blob 5e1f7d758ce1).
  • tools/tree-sitter-vba/vba/grammar.js without its 40-line provenance
    header is upstream's vba/grammar.js (blob 9247dbbfe166).
  • parser.c is generated with tree-sitter-cli 0.25.10 using upstream's own
    command (generate --abi 15 --output vba/src vba/grammar.js). Two
    independent generations are byte-identical (SHA-256 bd2a7ecde688...), and
    identical to a generation from upstream's own layout and tree-sitter.json.
  • Cross-checks against upstream's own artifacts: the generated
    node-types.json is upstream's checked-in vba/src/node-types.json
    byte-identical (blob 500e63561011), and the parse-table counts (16,029
    states, 417 symbols, 180 tokens, 65 fields, 362 productions) equal those of
    upstream's 0.26.9-generated bindings/go/parser.c.
  • LICENSE is upstream's MIT text byte-identical (blob efec104b6dc4).
  • Nothing else from upstream was run or vendored: no npm or Go binding, no
    script, workflow, Wasm/playground tooling, agent instruction file or test
    fixture. Generation ran with network access denied and writes confined
    to a scratch directory.

Generated-parser review

48,349,333 bytes, 1,119,783 lines, ABI 15. 0 non-ASCII bytes and 0 control
bytes other than LF; full tokenization leaves 0 unmatched characters; one
comment (the generator stamp); 490 string literals, none containing a space;
729 hex literals, every one a lexer code-point comparison against
lookahead; no call outside the tree-sitter macros. No external scanner
(EXTERNAL_TOKEN_COUNT 0, no .external_scanner in the language struct).
tree_sitter/*.h are byte-identical to copies already vendored here.

License

MIT, (c) 2026 harumiWeb. The shared grammar core's vb6 layer came upstream
through harumiWeb/tree-sitter-vba#56 under the same license. Upstream's
THIRD_PARTY_LICENSE.md covers only its examples/third_party test
fixtures (16 MIT projects and one CC BY 3.0); none of them is carried here.
No copyleft code is involved.

Sizes

parser.c is 48,349,333 bytes, under the 50 MB line, so no LFS question
arises. The copy in tools/tree-sitter-vba/vba/src/ and the vendored copy
are the same git blob.

Known gaps (for the extraction PR)

On real VB6 the vba dialect leaves three localized error shapes:

  • a .frm/.ctl component reference line,
    Object = "{GUID}#1.2#0"; "COMDLG32.OCX": a 16-character ERROR on that
    line;
  • a designer resource offset ending in a hex letter,
    ToolboxBitmap = "X.ctx":003B: a 1-character ERROR;
  • a procedure whose header is duplicated across #If VBA7 / #Else with an
    Attribute Name.VB_Description line inside each branch: error recovery
    runs to the end of the file (2 of the 161 VBCCR files).

All three are recorded in internal/cbm/vendored/grammars/MANIFEST.md.

What it enables next

VB6/VBA extraction through #2564, which we port onto this grammar with
maintainer commits after #1179 merges (Tokarzewski's commits stay). This PR
registers no language: internal/cbm/grammar_vba.c compiles and links
tree_sitter_vba, but no CBMLanguage, file extension or name alias routes
to it yet. The extraction PR adds them under the vb6 and vba names;
vb and visualbasic mean VB.NET.

The README and Chocolatey counts move from 163 to 164 languages, because
the language-count contract counts vendored grammar directories.

Verification

  • tests/test_parse_coverage.c: a dual-target VB6 class module (the IDE
    class header, a Declare in both #If VBA7 branches, the LongPtr shim
    enum, Implements, an Event, Property Get/Let/Set, a label, bare calls)
    parses through tree_sitter_vba() with no ERROR node, every procedure is
    a declaration node, and every bare call keeps its callee; a CRLF VB6 form
    parses clean; broken source still reports an ERROR. With the v0.13.0
    parser swapped in, the class case fails (65 passed, 1 failed); the vb6
    dialect also errors on it (at PtrSafe).
  • scripts/security-vendored.sh (1,101 files, no dangerous calls),
    tests/test_vendored_integrity_contract.sh,
    tests/test_language_count_contract.sh (164) and
    scripts/audit-grammar-security.sh pass. The notice bundle carries the
    vba LICENSE, and the SBOM generator counts 164 grammars. The license
    provenance audit (which routes vba to upstream through its FORKS map)
    runs in the hosted security workflow.
  • Narrow suites under ASan/UBSan: parse_coverage (66 passed), plus language,
    lang_contract, registry, repro_language_registry, grammar_regression,
    grammar_labels and extraction (972 passed), with no sanitizer report.

Refs #721, #2564

Vendors harumiWeb/tree-sitter-vba (MIT, (c) 2026 harumiWeb) at
83c5c3c3c721 (tag v0.14.5, upstream HEAD), its vba dialect, as a
self-maintained fork in tools/tree-sitter-vba/. This is the precondition
for VB6/VBA extraction (#2564, ported after #1179); it registers no
language yet.

Why generated here: upstream does not track its generated parser (its
ADR 0002), so the fork carries upstream's grammar source in upstream's
layout with no local patch, and parser.c is generated in this repository
with tree-sitter-cli 0.25.10 (ABI 15), the generator every fork here
uses. Upstream's own copy is generated by 0.26.9.

Why this pin: v0.13.0 (the pin #2564 carried) exposes the same procedures
but misreads statements without an ERROR node. On the three corpora
below it yields 1,276 expression_statement nodes where v0.14.5 yields 56
(`Err.Raise a, b, c` after a label loses its callee), and it leaves an
ERROR at `Case obj Is Nothing`.

Why the vba dialect rather than upstream's vb6 one: measured with the
CLI on kellyethridge/VBCorLib @ 100acbbae0c0 (655 VB6 files),
Kr00l/VBCCR @ 4b86c5c2c075 (161 VB6 files, 70 .ctl) and upstream's 481
VBA examples, procedures outside ERROR nodes are 14,084/14,084,
12,385/1,552 and 8,104/6,573 (vba/vb6). vb6 rejects the PtrSafe/LongPtr
forms dual-target VB6 keeps inside #If VBA7 blocks and then loses whole
files; vba never sees fewer procedures in any file. vb6's parser.c is
also 75.8 MB against 48.3 MB.

Provenance by hash: common/define-grammar.js is upstream's blob
5e1f7d7; vba/grammar.js minus its 40-line provenance header is
upstream's blob 9247dbbfe166. Two independent generations are
byte-identical, and identical to a generation from upstream's own layout
and tree-sitter.json. The generated node-types.json is upstream's
checked-in file byte-identical (500e635), and the parse-table counts
equal upstream's own generated parser. Review: 0 non-ASCII bytes, 0
unmatched characters on full tokenization, one comment (the generator
stamp), no string literal with a space, no external scanner, no call
outside the tree-sitter macros. No upstream script, package hook or
binary was run or vendored.

- internal/cbm/vendored/grammars/vba/: the generated parser.c, the
  tree_sitter/ headers (byte-identical to copies already vendored) and
  upstream's MIT LICENSE byte-identical.
- internal/cbm/grammar_vba.c: compiles and links tree_sitter_vba. No
  CBMLanguage, extension or alias routes to it yet; extraction adds them
  under the vb6/vba names (vb and visualbasic mean VB.NET).
- MANIFEST.md (fork note with the measurements and known gaps, forks
  table, 164 grammars, ABI recount), THIRD_PARTY.md (fork list and a vba
  section), scripts/vendored-checksums.txt, and the FORKS entry in
  scripts/audit-license-provenance.py.
- README.md and the Chocolatey nuspec say 164 languages: the
  language-count contract defines the count as the vendored grammar
  directories.
- tests/test_parse_coverage.c: a dual-target VB6 class module (IDE class
  header, Declare in both #If VBA7 branches, Property Get/Let/Set,
  Implements, events, a label, bare calls) parses with no ERROR node and
  every bare call keeps its callee; a CRLF VB6 form parses clean; broken
  source is still flagged. The class case fails on v0.13.0 and on the vb6
  dialect.

Refs #721, #2564

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
@DeusData DeusData mentioned this pull request Nov 12, 2026
4 tasks done
Brings in #2957 (sql became a self-maintained fork), #2959 and #2962.
Both sides added a self-maintained grammar fork, so every conflict keeps
both additions:

- MANIFEST.md summary: 164 grammars = 139 vendored-from-upstream + 20
  self-maintained (sql moved over on main, vba added here) + 5
  registry-disagreement; the ABI tally stays 9/78/77 (sql stayed ABI 15).
  The notes and the forks table carry both the sql and the vba rows.
- THIRD_PARTY.md: thirteen further self-maintained forks including sql and
  vba; main's sql paragraph, then this branch's tree-sitter-vba section.
- tests/test_parse_coverage.c: both test blocks and both sets of
  RUN_TEST registrations.
- scripts/vendored-checksums.txt: regenerated with
  security-vendored.sh --update; against main only the MANIFEST.md line
  and the five vba files differ.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Brings in #2974 (incl. the jsonnet scanner patch) and #2970. One conflict:
scripts/vendored-checksums.txt, where #2974 and the vba vendoring both
rewrite the MANIFEST.md line. Resolved by taking main's file and
regenerating it with scripts/security-vendored.sh --update; against main it
differs only in the MANIFEST.md line and the five vba lines, and the
integrity verify passes.

THIRD_PARTY.md, MANIFEST.md and tests/test_parse_coverage.c merged cleanly;
each equals main plus this branch's own change (identical patch-ids), and
test_parse_coverage.c has 76 TEST and 76 RUN_TEST entries (71 at the merge
base, +2 from main, +3 from this branch). The MANIFEST summary (164
grammars) and ABI line (9/78/77) match a recount from the tree.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
@DeusData
DeusData merged commit ea6ad87 into main Nov 14, 2026
42 checks passed
DeusData added a commit to ankitsusheel/codebase-memory-mcp that referenced this pull request Nov 14, 2026
Brings the branch up to main 08ab4a0 (DeusData#2966, DeusData#2969, DeusData#2970, DeusData#2977,
DeusData#2978, DeusData#2981).

scripts/vendored-checksums.txt: main (DeusData#2969) vendored the VB6/VBA
grammar while this branch edits MANIFEST.md. Regenerated with
scripts/security-vendored.sh --update (1101 files); only the
MANIFEST.md line differs from main.

Everything else merged without conflict and was checked by hand:
- MANIFEST.md keeps main's vba, sql and jsonnet text; only this
  branch's vbnet registration sentence and custom-handling row differ.
- The language count stays 164 (vendored grammar directories); the
  language-count contract passes. VB.NET adds a language name to the
  README list, not a grammar.
- DeusData#2977's receiver gate is an allowlist (gate_scope); VB.NET is not on
  it, so it stays GATE_OFF, and receiver_chain_admits is unchanged.
- The VB.NET case-insensitive retry still hooks
  cbm_registry_resolve_lang, cbm_registry_resolve_ref and
  resolve_call_lang; the Field-call USAGE policy and the enum-member
  gate are intact.
- DeusData#2981's linear-growth rewrites in extract_defs.c touch no VB.NET
  function; find_class_body still decides VB.NET first.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
DeusData added a commit that referenced this pull request Nov 14, 2026
Main brought in #2969 (vendored VB6/VBA grammar), #2981 (linear-growth
rewrites in extraction and eight LSP passes, a node for C# operators, C
increments recorded as writes), #2977 (receiver gate in the registry),
#2978 (possible override targets in trace_path), #2966 (CONTRIBUTING test
reference) and #2985 (freebsd-vm action bump).

Two files conflicted:

- internal/cbm/vendored/grammars/MANIFEST.md: the summary header. Took
  main's counts (164 grammars, 20 first-party, ABI 9x/78x/77x), which a
  recount of the merged tree confirms (164 grammar directories, LANGUAGE_VERSION
  13/14/15 = 9/78/77). Kept this branch's 2026-11-13 ObjectScript recount
  line above main's 2026-11-12 vba line; main had folded the 2026-11-11 sql
  line into its vba line, so that line goes. Main's vba prose and table rows
  and this branch's ObjectScript v1.11.0 rows are both present.
- scripts/vendored-checksums.txt: regenerated with
  scripts/security-vendored.sh --update. Only the MANIFEST.md hash moved;
  the result differs from this branch only by main's vba entries and from
  main only by the ObjectScript entries.

Every other file merged cleanly. For each file both sides changed
(THIRD_PARTY.md, extract_calls.c, extract_defs.c, extract_usages.c,
helpers.c, lang_specs.c, tests/test_parse_coverage.c) the merged file
differs from each parent by exactly the other parent's change.
#2981 rewrote objectscript_routine_preceding_tag onto a memoized sibling
search with the same probe; this branch does not touch that function.
ObjectScript stays outside #2977's receiver gate allowlist.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant