Skip to content

chore(vendor): vendor the VB.NET tree-sitter grammar - #2948

Merged
DeusData merged 2 commits into
mainfrom
feat/vendor-vbnet-grammar
Nov 10, 2026
Merged

DeusData merged 2 commits into
mainfrom
feat/vendor-vbnet-grammar

Conversation

@DeusData

Copy link
Copy Markdown
Owner

What this vendors

A Visual Basic .NET tree-sitter grammar, as a self-maintained fork in
tools/tree-sitter-vbnet/, generated into
internal/cbm/vendored/grammars/vbnet/:

  • Grammar: govindbanura/tree-sitter-vbnet
    @ 538b7087bf80e86004531b392fe1186379c0a2b5 (tag v0.1-beta, upstream
    HEAD). MIT, (c) 2025 Govind Banura.
  • Patch: CodeGraph's real-world VB.NET patch,
    docs/grammars/tree-sitter-vbnet.patch from
    colbymchenry/codegraph
    @ fe7542b6f6a4939b89964affeea74f78a1c78b96 (the latest revision of that
    file, 2026-10-06). MIT, (c) 2026 Colby Mchenry.

Both licenses ship next to the code, byte-identical to their upstreams:
LICENSE (tree-sitter-vbnet) and LICENSE-codegraph. The external scanner
and the patched rules are CodeGraph's work, so its notice has to travel with
the binary too. Both texts land in the release notice bundle.

Why this grammar

It is the licensed VB.NET grammar. CodeAnt-AI/tree-sitter-vb-dotnet, the
grammar #1179 was built on, carries no license, so nobody can vendor it.

Why the patch

Upstream checks in only grammar.js (no generated src/, no scanner), and
it does not parse the code VB.NET projects are made of. On the same sample:

grammar ERROR nodes
upstream @ 538b708, unpatched 8, starting at the Single literals of the WinForms Designer part (8.25!, New SizeF(6.0!, 13.0!)) and at the body-less MustOverride member
+ patch @ e871c49a (the revision #2885 carried) 1: Sub NewItem() lexes as a constructor plus a stray Item
+ patch @ fe7542b6 (this PR) 0

The earlier revision also mis-parses silently: Public SharedCache As Integer
becomes a Shared field named Cache, and Public Dimension becomes Dim +
ension. The pinned revision keeps names that begin with a keyword whole.

Provenance, by hash

  • The patch file is byte-identical to CodeGraph's (git blob 37918d81).
  • Upstream grammar.js at the pin is the patch's pre-image (21c8523);
    tools/tree-sitter-vbnet/grammar.js without its 41-line provenance header
    is the patch's post-image (c9c1d85), and CodeGraph's scanner.c is
    8334fdb, exactly as the patch's index lines record (our one local
    scanner patch, below, then takes it to b3473d1).
  • parser.c is generated here with tree-sitter-cli 0.25.10 (ABI 15, the
    generator the other forks use). Two independent generations are
    byte-identical (SHA-256 d0c29e9a7741...). Nothing else from upstream was
    run or vendored (it also checks in a Windows parser.obj and a build.py).
  • Generated-parser review: 0 non-ASCII bytes, full tokenization with 0
    unmatched characters, no string literal containing a space, one comment
    (the generator stamp), no call outside the tree-sitter macros and the
    scanner hooks. The scanner is stateless (no allocation, no I/O, every loop
    stops at end of input) and compiles with 0 warnings under
    -Wall -Wextra -Wpedantic.

One fix of ours on top (second commit)

Measuring parse cost turned up a quadratic in CodeGraph's scanner. The
parser offers its XML-literal scanner a < wherever a declaration or member
starts, where valid VB only ever has an attribute (<Serializable()> _).
The scanner took <Serializable for an XML element and read the rest of
the file looking for a closing tag before giving up, once per attribute:
a 1.35 MB synthetic file of attribute-decorated classes took 21 s to parse.
The scan now stops at the first character well-formed XML cannot contain
(( in a tag, a < or & in element text that starts no tag or
reference). Parse trees are byte-identical before and after on all 25
sample and synthetic files tried, XML literals of every shape included,
and that file now parses in 0.08 s. The change is a separate patch file
(tools/tree-sitter-vbnet/patches/xml-literal-tag-syntax.patch) with a row
in the manifest's local-patch table and a complexity guard
(complexity_vbnet_attribute_lexing_is_linear, lexer-bytes ratio 3.99
before, 2.0 after). One residual is recorded: an attribute without an
argument list (<Fact>) is a well-formed XML start tag, so its scan can
still run to a non-XML character or the end of the file.

What it enables next

VB.NET extraction through #1179, which we port onto this grammar with #2885
folded in (with credit). This PR registers no language:
internal/cbm/grammar_vbnet.c compiles and links tree_sitter_vbnet, but
no CBMLanguage, file extension or name alias routes to it yet. The
extraction PR adds those, with vb and visualbasic meaning VB.NET (VB6
keeps its own name), and case-insensitive name resolution.

Known gaps the extraction PR has to handle (from CodeGraph's notes, reproduced
here): a GoTo label at column 0 leaves a localized ERROR, and a file whose
last line has no newline ends in a MISSING node.

The README and Chocolatey counts move from 162 to 163 languages, because
the language-count contract counts vendored grammar directories.

Verification

  • tests/test_parse_coverage.c: real VB.NET (a WinForms Designer partial
    class, Handles, Imports, Namespace, Inherits/Implements, MustOverride,
    #Region, Sub New, Property, keyword-prefixed names) parses through
    tree_sitter_vbnet() with no ERROR node, and a broken file still reports
    one. The first case fails on unpatched upstream and on the earlier patch
    revision.
  • scripts/security-vendored.sh, tests/test_vendored_integrity_contract.sh,
    tests/test_language_count_contract.sh (fails at 162, passes at 163) and
    scripts/audit-grammar-security.sh pass. The provenance audit's own vbnet
    path returns IDENTICAL against upstream's license. The SBOM generator and
    the notice bundle pick up the new grammar and both license texts.
  • complexity_vbnet_attribute_lexing_is_linear: lexer bytes 14.6 M -> 58.3 M
    (ratio 3.99, red) on the first commit, 102 K -> 205 K (ratio 2.00) with the
    fix.
  • Control: helm @ 9b88efd6 indexes to a byte-identical graph on main and on
    both commits (canonical hash eac947e1527d35c8, 12,575 nodes, 39,922
    edges); the binary links tree_sitter_vbnet, and nothing routes to it.
  • Narrow suites under ASan/UBSan (parse_coverage, extraction, language,
    lang_contract, registry, repro_language_registry, grammar_regression,
    grammar_labels, complexity): 1,037 passed, no sanitizer report.

Refs #1178, #1179, #2885

Vendors govindbanura/tree-sitter-vbnet (MIT, (c) 2025 Govind Banura) at
538b7087bf80 (tag v0.1-beta, upstream HEAD) with CodeGraph's real-world
VB.NET patch (colbymchenry/codegraph docs/grammars/tree-sitter-vbnet.patch
at fe7542b6f6a4, MIT, (c) 2026 Colby Mchenry), as a self-maintained fork
in tools/tree-sitter-vbnet/. This is the precondition for VB.NET
extraction (#1179, folding in #2885); it registers no language yet.

Why this grammar: it is the licensed one. CodeAnt-AI/tree-sitter-vb-dotnet,
which #1179 was built on, carries no license and cannot be vendored.

Why the patch: upstream checks in grammar.js only and does not parse the
code VB.NET projects are made of. Unpatched, the Single literals in every
WinForms .Designer.vb (New SizeF(6.0!, 13.0!)) and body-less MustOverride
members are ERROR nodes. The 2026-10-06 revision of the patch is pinned
because it also keeps names that begin with a keyword whole: the earlier
revision (the one #2885 carried) turns `Sub NewItem()` into an ERROR and
`Public SharedCache As Integer` into a Shared field named Cache.

Provenance by hash: the patch file is byte-identical to CodeGraph's (blob
37918d8); upstream grammar.js at the pin is the patch's pre-image
(21c8523), and tools/tree-sitter-vbnet/grammar.js minus its 41-line
provenance header is the post-image (c9c1d85), as is scanner.c (8334fdb).
parser.c is generated here with tree-sitter-cli 0.25.10 (ABI 15); two
independent generations are byte-identical. No upstream script, package
hook or binary was run or vendored.

- internal/cbm/vendored/grammars/vbnet/: the generated parser.c,
  CodeGraph's external scanner.c (two stateless tokens), the tree_sitter/
  headers, upstream's MIT LICENSE byte-identical, and CodeGraph's MIT text
  as LICENSE-codegraph: the scanner and the patched rules are CodeGraph's
  work, so its notice has to ship with the binary too.
- internal/cbm/grammar_vbnet.c: compiles and links tree_sitter_vbnet. No
  CBMLanguage, extension or alias routes to it yet; extraction adds them,
  with `vb` and `visualbasic` meaning VB.NET (VB6 keeps its own name).
- MANIFEST.md (fork note, forks table, 163 grammars, ABI recount),
  THIRD_PARTY.md (fork list and a vbnet section; the stale "160
  pre-generated parsers" now reads 163), scripts/vendored-checksums.txt,
  and the FORKS entry in scripts/audit-license-provenance.py.
- README.md and the Chocolatey nuspec say 163 languages: the
  language-count contract defines the count as the vendored grammar
  directories.
- tests/test_parse_coverage.c: real VB.NET (a WinForms Designer partial
  class, Handles, Inherits/Implements, MustOverride, #Region, Property,
  keyword-prefixed names) parses with no ERROR node. The case fails on
  unpatched upstream (8 ERROR nodes) and on the earlier patch revision (1).

Refs #1178, #1179, #2885

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The vendored vbnet scanner lexes a whole VB XML literal as one external
token, and the parser offers it a '<' wherever a declaration or member
starts. There, in valid VB, '<' opens an attribute block
(`<Serializable()> _`), never an XML literal. The scanner nevertheless
took `<Serializable` for an element start and read the rest of the file
looking for a closing tag before giving up: O(rest of file) per
attribute, O(n^2) per file. Generated proxies and designer code put an
attribute on every member: 2,500 attributes in a 77 KB file advanced
38.5 M characters, and a 1.35 MB synthetic file took 21 s to parse.

The scan now fails at the first character well-formed XML cannot
contain: inside a tag, outside a quoted value, anything other than name
characters, blanks, '=', quotes, '/>', '>' and '<%='; in element text, a
'<' that starts no tag, closing tag, comment, PI or embedded expression,
and an '&' that starts no reference. No valid XML literal lexes
differently: parse trees are byte-identical before and after on all 25
sample and synthetic files tried, XML literals of every shape included.
The 1.35 MB file now parses in 0.08 s.

Residual, recorded in MANIFEST.md: an attribute without an argument list
(`<Fact>`) is a well-formed start tag, so its scan still runs until a
character that cannot be XML, or the end of the file.

The change is kept as tools/tree-sitter-vbnet/patches/
xml-literal-tag-syntax.patch (CodeGraph's scanner 8334fdb -> b3473d1)
and in MANIFEST.md's local-patch table. The new
complexity_vbnet_attribute_lexing_is_linear counts the bytes the lexer
pulls through a chunked TSInput for k and 2k attribute-decorated
classes: ratio 3.99 before the fix, 2.0 after.

Refs #1178, #1179

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
@DeusData
DeusData merged commit 6d34ca1 into main Nov 10, 2026
43 checks passed
DeusData added a commit to ankitsusheel/codebase-memory-mcp that referenced this pull request Nov 12, 2026
Point the VB.NET language spec at the node kinds of the vendored
govindbanura/tree-sitter-vbnet grammar (DeusData#2948): Sub and Function,
MustOverride, Declare and interface members are functions; Class,
Module, Structure, Interface and Enum declarations are types; `F()`
(invocation_expression), `F(x)` (array_access_expression) and `New T`
(object_creation_expression) are call nodes, since VB.NET writes calls
and indexing alike.

The names `vb`, `visualbasic`, `vbnet` and `vb.net` select VB.NET in
the user config (VB6 keeps its own name). *Test.vb and *Tests.vb are
test files, VB.NET joins C# and F# in the .NET family for
cross-language resolution, and the vendored grammar MANIFEST records
the registration.

Co-authored-by: Ankit Susheel <ankitsusheel@gmail.com>
Co-authored-by: jorge blanco <jorge.luis.blanco@hotmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant