Repository navigation
Feat/721 vb6 grammar r2 - #2564
Tokarzewski wants to merge 5 commits into
Conversation
Vendors tree-sitter-vba @ 63b2f8d0d65c (v0.13.0, MIT, ABI 15, lexer-only, no external scanner) as CBM_LANG_VB6 and wires it end to end: - discover: .bas/.ctl/.dsr/.pag map directly; .cls (Apex/ObjectScript) and .frm (FORM) keep their table owners and the DeusData#2074 content sniffers now route VB6 modules to CBM_LANG_VB6 instead of CBM_LANG_COUNT. - lang_specs: Sub/Function/Property Get|Let|Set/Declare/Event as callables; Type -> Class with typed Fields (As clause unwrapped); Enum -> Enum with members; module-level Dim/Const -> one Variable per declarator; call_statement/call_expression/raise_event_statement callees; Implements as import; VB6 keyword set; source_file module parents. - release surfaces: MANIFEST (163 grammars, 75x ABI-15, vba row + note on the locally generated parser.c), THIRD_PARTY, vendored checksums, new-languages.json, and the 162 -> 163 language-count claims on all seven contract surfaces. - tests: extension/sniff mapping, grammar regression + label golden, calls contract, extraction block, and the four repro registries (capability ledger, call-node manifest, invariant breadth, argument matrix B). Class-module synthesis (.cls/.frm as a Class with Methods) is a separate follow-up change. Refs DeusData#721. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Tokarzewski <bartlomiej.tokarzewski@gmail.com>
The classify content-rule table added upstream still expected the CBM_LANG_COUNT stub for VB6 .cls/.frm; the grammar now routes them. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Tokarzewski <bartlomiej.tokarzewski@gmail.com>
A VB6 .cls/.frm/.ctl/.dsr/.pag file is one COM class, but tree-sitter-vba has no class node, so the file's procedures surfaced as free Functions and the calls inside them were attributed to the Module. - helpers.c: cbm_vb6_is_class_module_path / cbm_vb6_file_class_name / cbm_vb6_file_class_qn resolve the class from `Attribute VB_Name` (file stem fallback). One helper feeds both extractors so the QNs cannot drift; it returns NULL for .bas standard modules and for every other language. - extract_defs.c: cbm_extract_definitions synthesises the Class def (bases from the designer `Begin VB.Form` type and every `Implements`) and runs the def walk with enclosing_class_qn set to it; extract_func_def promotes the file's procedures to Methods (QN class.name, parent_class set so DEFINES_METHOD links). - extract_unified.c: the scope-frame walker's baseline enclosing_class_qn starts as the file-class QN instead of NULL. push_scope saves and pop_expired_scopes restores that baseline exactly like any other frame value, so compute_func_qn yields class.name for every procedure and each in-body call sources to the Method, never the Module. No extra WalkState field is needed. Stacks on the VB6 grammar commit that introduces CBM_LANG_VB6. Refs DeusData#721. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Tokarzewski <bartlomiej.tokarzewski@gmail.com>
- grammar_regression: Widget.cls yields exactly one Class (base IShape), two Methods whose parent_class is the class QN, and no Function; the call inside Go carries enclosing_func_qn class.Go. ModA.bas yields no Class and no Method. - lang_contract: a Widget.cls CALL_CASES row, plus a dedicated contract that indexes the file through the production pipeline and checks the CALLS edge into Helper is sourced by the Method Go and by nothing else (a Module-sourced edge would surface the module's name as the caller). - extraction: vb6_implements_is_an_import (from the grammar commit) now expects Go in Widget.cls to be a Method, not a Function -- the class synthesis this series adds makes every .cls procedure a Method. Refs DeusData#721. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Tokarzewski <bartlomiej.tokarzewski@gmail.com>
|
Thanks for opening this — it has been seen, and it is queued. This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence. Current review status: working through a backlog. What that means for this PR, concretely:
Things that will genuinely speed it up whenever review does happen:
If this fixes a bug, a reproduction we can run is worth more than a description of the symptom. Thanks for contributing, and sorry in advance for the wait. |
No code change. test-windows-guards failed on tests/windows/test_daemon_stability.py (section_start_status_port), a Windows daemon-port test this branch does not touch; re-triggering CI. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Tokarzewski <bartlomiej.tokarzewski@gmail.com>
|
Thanks for consolidating the VB6/VBA work and clearly documenting the generated-parser provenance and size. That exception, the delivery shape, and the class-module naming choices need an explicit maintainer decision. I have recorded them for review; please hold off on further splitting or regeneration until that direction is settled. |
|
Hi @Tokarzewski, thank you for your patience, and thank you again for this work. Consolidating the grammar and the class-module synthesis into one branch, documenting exactly how parser.c was generated, and laying out the open design points made this decision much easier for us. Here is what we decided and why:
You do not need to do anything. Please hold off on pushing to this branch while we work on it; if you want to add something later, please pull first, since our commits will sit on top of yours. We will post here again when the vendoring PR is up. |
|
As promised, the vendoring PR is up: #2969. It vendors harumiWeb/tree-sitter-vba at v0.14.5, using its |
Vendors harumiWeb/tree-sitter-vba (MIT, (c) 2026 harumiWeb) at 83c5c3c3c721 (tag v0.14.5, upstream HEAD), its vba dialect, as a self-maintained fork in tools/tree-sitter-vba/. This is the precondition for VB6/VBA extraction (DeusData#2564, ported after DeusData#1179); it registers no language yet. Why generated here: upstream does not track its generated parser (its ADR 0002), so the fork carries upstream's grammar source in upstream's layout with no local patch, and parser.c is generated in this repository with tree-sitter-cli 0.25.10 (ABI 15), the generator every fork here uses. Upstream's own copy is generated by 0.26.9. Why this pin: v0.13.0 (the pin DeusData#2564 carried) exposes the same procedures but misreads statements without an ERROR node. On the three corpora below it yields 1,276 expression_statement nodes where v0.14.5 yields 56 (`Err.Raise a, b, c` after a label loses its callee), and it leaves an ERROR at `Case obj Is Nothing`. Why the vba dialect rather than upstream's vb6 one: measured with the CLI on kellyethridge/VBCorLib @ 100acbbae0c0 (655 VB6 files), Kr00l/VBCCR @ 4b86c5c2c075 (161 VB6 files, 70 .ctl) and upstream's 481 VBA examples, procedures outside ERROR nodes are 14,084/14,084, 12,385/1,552 and 8,104/6,573 (vba/vb6). vb6 rejects the PtrSafe/LongPtr forms dual-target VB6 keeps inside #If VBA7 blocks and then loses whole files; vba never sees fewer procedures in any file. vb6's parser.c is also 75.8 MB against 48.3 MB. Provenance by hash: common/define-grammar.js is upstream's blob 5e1f7d7; vba/grammar.js minus its 40-line provenance header is upstream's blob 9247dbbfe166. Two independent generations are byte-identical, and identical to a generation from upstream's own layout and tree-sitter.json. The generated node-types.json is upstream's checked-in file byte-identical (500e635), and the parse-table counts equal upstream's own generated parser. Review: 0 non-ASCII bytes, 0 unmatched characters on full tokenization, one comment (the generator stamp), no string literal with a space, no external scanner, no call outside the tree-sitter macros. No upstream script, package hook or binary was run or vendored. - internal/cbm/vendored/grammars/vba/: the generated parser.c, the tree_sitter/ headers (byte-identical to copies already vendored) and upstream's MIT LICENSE byte-identical. - internal/cbm/grammar_vba.c: compiles and links tree_sitter_vba. No CBMLanguage, extension or alias routes to it yet; extraction adds them under the vb6/vba names (vb and visualbasic mean VB.NET). - MANIFEST.md (fork note with the measurements and known gaps, forks table, 164 grammars, ABI recount), THIRD_PARTY.md (fork list and a vba section), scripts/vendored-checksums.txt, and the FORKS entry in scripts/audit-license-provenance.py. - README.md and the Chocolatey nuspec say 164 languages: the language-count contract defines the count as the vendored grammar directories. - tests/test_parse_coverage.c: a dual-target VB6 class module (IDE class header, Declare in both #If VBA7 branches, Property Get/Let/Set, Implements, events, a label, bare calls) parses with no ERROR node and every bare call keeps its callee; a CRLF VB6 form parses clean; broken source is still flagged. The class case fails on v0.13.0 and on the vb6 dialect. Refs DeusData#721, DeusData#2564 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
What does this PR do?
Adds Visual Basic 6 / VBA as a language (
CBM_LANG_VB6) and models VB6 class modules asClass+Methodnodes. This is part two of #721, following the merged short-term disambiguation in #2074. It supersedes my earlier #2086 (grammar) and #2087 (class modules), which I withdrew before review reached a decision; this is the same work rebased onto currentmainas a single branch. Happy to split it back into the two PRs, or to take only the grammar, whichever is easier to review.Refs #721.
Two commits are the grammar and wiring, two are the class-module extraction and its tests:
feat(lang): add Visual Basic 6 / VBA grammar (harumiWeb/tree-sitter-vba)test(language): VB6 content rows now classify as CBM_LANG_VB6feat(extract): VB6 class modules as Class + Methodstest(extract): VB6 class-module synthesis and Method call attributionGrammar
harumiWeb/tree-sitter-vba @
63b2f8d0d65c(v0.13.0, MIT, ABI 15, lexer-only, no external scanner). It parses the module export headers natively (VERSION,Begin ... End,BeginProperty ... EndProperty, GUID blocks,Attributelines), so the.frmdesigner block needs no preprocessing step. VB6-specific statements are covered (Declare,Implements,Event/RaiseEvent,WithEvents,Property Get|Let|Set)..cls/.frmare routed by the #2074 content sniffers (now returningCBM_LANG_VB6instead of the unsupported marker);.bas/.ctl/.dsr/.pagmap directly by extension.Alternatives I rejected: arrmee-wt/tree-sitter-vba (GPL-3.0, outside
scripts/license-policy.json), joannefan/tree-sitter-vb6 (no licence file, no generated parser), CodeAnt-AI/tree-sitter-vb-dotnet (VB.NET, which is #1073).Provenance caveat, please decide. Upstream gitignores
src/parser.c, so the vendoredparser.c(about 46 MB) was generated from the pinnedgrammar.jswithtree-sitter-cli 0.26.9(tree-sitter generate --abi 15, the project's ownpnpm generate). Byte-identity with an upstream artefact therefore cannot be claimed.MANIFEST.mdrecords the exact regeneration command and marks the rowNOT-IN-REGISTRIES (see note). The size comes from per-keyword case-insensitive expansion. This was the open question on the earlier PRs; I would rather you decide it explicitly than have me assume it is acceptable.Extraction
Sub/Function/Property Get|Let|Set/Declare/Eventbecome callables;Type ... End Typebecomes a Class with typed Fields;Enumbecomes an Enum with members; module-levelDim/Constbecome Variables;Implementsbecomes an import.call_statement,call_expressionandraise_event_statement.y = Foo(1)cannot be told apart from array indexing in VB6 syntax, so an unresolved callee simply produces no edge..cls/.frm/.ctl/.dsr/.pag): oneClassper file, named fromAttribute VB_Name = "..."(falling back to the file stem), with base classes from the designer block type and everyImplements. Procedures becomeMethods withDEFINES_METHODedges;.basstandard modules keepFunctions.cbm_extract_unified(), and both the definition and call sides derive the class QN from a single helper (cbm_vb6_file_class_qn), so they cannot drift. The class name comes fromAttribute VB_Name, not from the File node QN, so the File-QN change in v0.11.0 does not affect it.Open design points
Property Get/Let/Settriplets share one QN and collapse to oneMethodnode (consistent on both sides).Declare ... LibandEventdeclarations inside a.clsbecome Methods of that class.Dim/Constin a.clsstay module-level Variables rather than class Fields (possible follow-up).Not in this PR
.vbs(no header markers),.vbp/.vbgproject files, VB.NET (#1073).Known grammar gaps
Lines are reported as
parse_partialand the files still index. Measured when I first wrote this, not re-measured on the rebase: a call with a leading omitted argument (.Add , x),Print #f, ..., graphicsLine (x, y)-(x2, y2),_-continued comments, and the.frmheaderObject = "{GUID}..."lines.What a new grammar touches here
Enum,
lang_specs.crow, language table,userconfig.cnames (vb6,vba,vb,visualbasic), tests, the repro registries, the seven language-count surfaces (162 to 163) withtests/test_language_count_contract.sh,MANIFEST.md(row, ABI tally regenerated from the tree: 9/78/76, re-vendor note),THIRD_PARTY.md,scripts/vendored-checksums.txt,scripts/new-languages.json.Checklist
git commit -s)language grammar_regression grammar_labels lang_contract extraction): 709 passed, 0 failed, includingcontract_vb6_class_module_calls_source_from_methodtest-runner: 8542 passed, 6 failed, 88 skipped. The same 6 fail on an unmodifiedmainbuild on this machine (4 intest_cli.c, install/update ACL errors; 2 intest_test_impact_git.c), so they are not caused by this change. Theincrementalsuite was skipped (fixture clone needs network)bash tests/test_language_count_contract.sh: 163 grammars, 7 surfacesscripts/vendored-checksums.txtmatch the committed blobsclang-format-20 --dry-run --Werrorclean on every touchedsrc/andinternal/cbmsource and header file; cppcheck not run locally, relying on CI