Repository navigation
Conversation
Re-vendors objectscript_udl and objectscript_routine from intersystems/tree-sitter-objectscript a7ffcdf2de8e to tag v1.11.0 (25cf0f67a380, upstream HEAD); both stay ABI 15. The refreshed scanner lexes Language = python method bodies with Python's string, triple-quote and #-comment rules, so a brace inside a Python string, comment or docstring no longer runs the body past its closing brace and drops the methods that follow it (#1261). Both parser.c files are upstream's committed bytes, and regenerating them from upstream's src/grammar.json and tree-sitter.json with tree-sitter-cli 0.27.1 (upstream's locked version) reproduces them byte-for-byte. The three local patches recorded in MANIFEST.md are re-applied unchanged: the objectscript_common.h include repoint, the uint8_t -> int widening of two loop counters, and the empty-state memcpy guard; upstream v1.11.0 still carries the original form of all three. Updates the MANIFEST.md note, pin rows and ABI recount, THIRD_PARTY.md and scripts/vendored-checksums.txt, and adds two parse-coverage tests: Python bodies with braces in strings, comments and docstrings parse completely, and an unclosed Python body is still reported as partial. Refs #1261, #2368 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…action Ports the first-party extraction changes of #2368 by Thomas Dyar onto the v1.11.0 grammars: the extractor and test hunks of his re-vendor commit (1a6b77c) and of his routine-call commit (2e84d7e). From upstream v1.9.15 on, the grammar hides `method_call` inside `Do`/`Job` (the receiver and `oref_method` sit directly under `do_parameter`/`job_argument`), inlines `core_trigger`/`external_trigger` into `trigger`, replaces `storage_body` with `external_method_body_content`, and no longer always wraps a routine call target in `line_ref`: - cbm_objectscript_is_instance_call() and cbm_is_call_site() feed the `Do obj.M()` shape to call and usage extraction (UDL only, as before); - objectscript_routine_callee_container() takes the callee of a routine_tag_call / extrinsic_function from `line_ref`, `method_name` (`Do accept`) or `routine_ref` (`$$^Other`), so every call form keeps its callee text; - a Trigger body without a wrapper is everything after trigger_name (extract_ident_tokens_from() skips the subtrees before it); - Storage XML properties fall back to external_method_body_content. Tests (his): the #1261 repro (six Language = python body shapes, each followed by two methods whose CALLS must survive), the eight routine call forms, and Storage XML properties. Not ported: his vendored grammar files, his zero-length memcpy guard (main already carries the same guard from #2800) and his MANIFEST.md, THIRD_PARTY.md and vendored-checksums.txt lines; the vendor commit before this one replaces them. The six touched files carry exactly his changes (identical patch-ids). Refs #1261, #2368 Co-authored-by: Thomas Dyar <tdyar@intersystems.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…mmar The v1.9.19 UDL grammar hides method_call: `Do obj.M(x)` now parses as do_parameter(lvn, oref_method). The registration could never match, so repro_primary_call_metadata_names_exist_in_grammars failed. Instance calls are still found by cbm_is_call_site through cbm_objectscript_is_instance_call, which already recognizes do_parameter/job_argument with an oref_method child. Remove the dead kind from objectscript_udl_call_types and its ledger row (historical 226 -> 225, direct/primary counts down by one), and point the UDL method-call behavior case at do_parameter, the node that now carries the call. Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
Brings in #2974 and #2970. One conflict: scripts/vendored-checksums.txt, where #2974 (jsonnet scanner patch) and the vendor commit both rewrite the MANIFEST.md line. Resolved by taking main's file and regenerating it with scripts/security-vendored.sh --update; the result differs from main only in the MANIFEST.md line and the eight refreshed ObjectScript files, and the integrity verify passes. MANIFEST.md, THIRD_PARTY.md and tests/test_parse_coverage.c merged cleanly with both sides kept. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Main brought in #2969 (vendored VB6/VBA grammar), #2981 (linear-growth rewrites in extraction and eight LSP passes, a node for C# operators, C increments recorded as writes), #2977 (receiver gate in the registry), #2978 (possible override targets in trace_path), #2966 (CONTRIBUTING test reference) and #2985 (freebsd-vm action bump). Two files conflicted: - internal/cbm/vendored/grammars/MANIFEST.md: the summary header. Took main's counts (164 grammars, 20 first-party, ABI 9x/78x/77x), which a recount of the merged tree confirms (164 grammar directories, LANGUAGE_VERSION 13/14/15 = 9/78/77). Kept this branch's 2026-11-13 ObjectScript recount line above main's 2026-11-12 vba line; main had folded the 2026-11-11 sql line into its vba line, so that line goes. Main's vba prose and table rows and this branch's ObjectScript v1.11.0 rows are both present. - scripts/vendored-checksums.txt: regenerated with scripts/security-vendored.sh --update. Only the MANIFEST.md hash moved; the result differs from this branch only by main's vba entries and from main only by the ObjectScript entries. Every other file merged cleanly. For each file both sides changed (THIRD_PARTY.md, extract_calls.c, extract_defs.c, extract_usages.c, helpers.c, lang_specs.c, tests/test_parse_coverage.c) the merged file differs from each parent by exactly the other parent's change. #2981 rewrote objectscript_routine_preceding_tag onto a memoized sibling search with the same probe; this branch does not touch that function. ObjectScript stays outside #2977's receiver gate allowlist. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
Merged current main into this branch again so it is mergeable (merge commit 0bde6de, second parent 1ec2f26). What main brought in since the last merge: #2969 (vendored VB6/VBA grammar), #2981 (linear-growth rewrites in extraction and the LSP passes, C# operator nodes, C increments as writes), #2977 (registry receiver gate), #2978 (possible override targets in trace_path), #2966 (CONTRIBUTING fix) and #2985 (freebsd-vm bump). Conflicts and how they were resolved:
Everything else merged cleanly, and for each file both sides touched the merged file differs from each parent by exactly the other side's change. ObjectScript is outside #2977's receiver gate, and #2981's rewrite of the routine preceding-tag lookup keeps the same probe. As noted earlier, CI is the first build of the ported extraction changes from #2368. It is also the first run of main's new |
What this lands
Fixes #1261, and lands #2368 by Thomas Dyar (@isc-tdyar) together with a
newer, re-verified grammar. Thank you, Thomas: the #1261 analysis, the
documentation of the hidden wrapper rules, and the extraction work are
yours, and this PR carries them with your authorship.
Commits, in order:
objectscript_udl(.cls) andobjectscript_routine(
.mac/.inc/.int/.rtn), fromintersystems/tree-sitter-objectscript
a7ffcdf2de8e(2026-06-15) to tagv1.11.0, full commit25cf0f67a3804727e250076a0926aa71c46fffab(2026-10-22, upstreammainHEAD). MIT, (c) 2025 InterSystems Corporation. Both stay ABI 15.
Co-authored-by: Thomas Dyar: the extractor and test hunks of hisre-vendor commit and of his routine-call commit from fix(objectscript): re-vendor grammar for Language = python bodies #2368. That covers
the
Do obj.M()instance-call site detection, the routine calleecontainer (
line_ref/method_name/routine_ref), the Trigger andStorage body fallbacks, and his tests: the Language = python method body scanner drops all following methods in the class #1261 repro, the eight routine
call forms and Storage XML properties. The six touched files carry exactly
his changes (identical patch-ids against fix(objectscript): re-vendor grammar for Language = python bodies #2368). Not ported: his vendored
files, his zero-length
memcpyguard (main already has the same guardfrom fix: guard empty ObjectScript scanner state #2800), and his MANIFEST, THIRD_PARTY and checksum lines, all
replaced by commit 1.
fix(lang_specs): drop ObjectScript UDL method_call, hidden by the grammar, cherry-picked unchanged with his authorship and sign-off.main: one conflict inscripts/vendored-checksums.txt(with fix: two causes of run-to-run graph variation and three silent content losses #2974's jsonnet scanner patch),resolved by regenerating the file with
scripts/security-vendored.sh.GitHub cannot render #2368's diff because of the two generated parsers, so
the grammar is re-vendored and re-verified here, at a newer pin.
Build note: the extraction half (commits 2 and 3) was not built locally;
this PR's CI is its first build. Every local proof below (suites, corpus
numbers, regeneration) covers the vendor half only, run with main's
extractor.
Why refresh (#1261)
At
a7ffcdfthe scanner lexed aLanguage = pythonmethod body bybalancing braces only. A brace inside a Python string, comment or docstring
ran the body past its closing
}, and the methods after it were lost. Onmain the loss is silent: the new parse-coverage test below finds no
parse_partialflag on that file, only the missing methods. Upstream lexesthese bodies with Python's string, triple-quote and
#-comment rules sincev1.9.15.
Why the halves land together
The refresh changes node shapes that main's ObjectScript extraction depends
on. From v1.9.15 upstream hides
method_callinsideDo/Job(the receiverand
oref_methodsit directly underdo_parameter/job_argument), inlinescore_trigger/external_triggerintotrigger, replacesstorage_bodywith
external_method_body_content, and wraps a routineDo label/$$^routinetarget inmethod_name/routine_refinstead of alwaysline_ref. With main's extractor and this grammar alone:Do obj.M()instance calls lose theirCALLS edge (
objectscript_udl_calls_typed_new,_typed_param,_typed_property,objectscript_data_flows_instance_method_args,iris_export_xml_calls_extracted) and Trigger bodies lose their tokens(
objectscript_udl_trigger_body_quit,_trigger_body_tokens);ownership cases in
repro_call_scope_usagesandrepro_call_node_behaviors, and themethod_callrow inrepro_language_registry(the node no longer exists in the grammar).These are exactly the shapes the extraction commits handle (commits 2 and
3), and #2368's own tests pin them; with the
48a06b6grammar of #2368 thesame failures occur, so they come from shape changes both pins share.
Why v1.11.0
Four candidates were staged the same way (upstream files plus our three
local patches) and compared with one parse harness:
48a06b6401af(the pinin #2368, package version 1.9.20),
v1.10.1(942eedc37afe),v1.10.2(
a1a5d7ae8752) andv1.11.0(25cf0f67a380)..clsfiles with an ERROR (of 429).cls)Corpora: intersystems/ipm @
571e04a19aba(MIT; 429.cls, 22 routinefiles), intersystems/Samples-ObjectScript @
8e63f5e7a522andintersystems-community/webterminal @
91b54afa0ef4(both MIT).following methods in four; every candidate parses all six cleanly.
*Referenceclasses andModuleName.cls), and no file loses a methoddefinition against main or against 48a06b6, in any corpus.
refresh hides are the same set at 48a06b6 and at v1.11.0, so the
extractor changes in fix(objectscript): re-vendor grammar for Language = python bodies #2368, written against 48a06b6, target exactly the
shapes v1.11.0 produces.
identical on these corpora.
Provenance, by hash
udl/src->objectscript_udl/,objectscript_routine/src->objectscript_routine/,common/scanner.h-> each directory's
objectscript_common.h, rootLICENSE-> eachLICENSE.parser.cfiles and alltree_sitter/*.hare upstream's files at25cf0f67a380, byte-identical.package-lock.json). Regenerating them here from upstream's committedsrc/grammar.jsonandtree-sitter.jsonwith the 0.27.1 release binary(SHA-256 matches the release's published digest), in a no-network,
write-confined sandbox, reproduces both byte-for-byte (SHA-256
0c0c651b4a42...udl,d496de950b08...routine), and bothnode-types.json. No upstream JavaScript was evaluated.LICENSEis byte-identical to the previous pin.MANIFEST.mdare re-applied exactly asmain carries them; upstream v1.11.0 still has all three original forms:
the
#include "../../common/scanner.h"repoint toobjectscript_common.h,the
uint8_t->intwidening of two loop counters inobjectscript_common.h, and theif (length > 0)guard before thedeserialize
memcpy(fix: guard empty ObjectScript scanner state #2800). The same staging procedure applied toa7ffcdfreproduces main's current files byte-for-byte.workflows or test fixtures.
Generated-parser and scanner review
parser.c(udl / routine): 49,392,087 / 48,934,992 bytes; 0 non-ASCII and0 control bytes other than LF; full tokenization leaves 0 unmatched
characters; one comment (the generator stamp); no string literal contains
a space; no call outside the tree-sitter macros and the external-scanner
entry points.
scanner.cx2,objectscript_common.h;about 620 added lines) is lexer logic over the
TSLexerAPI andwctype:fixed-size local arrays with bounds checks, no I/O, no environment or
process access, no function pointers or mutable statics; the debug logger
stays under
#if 0. The UDL scanner's serialized state stays about 520bytes, well under tree-sitter's 1024-byte buffer.
tree_sitter/array.hgains tree-sitter's C++-only_array__cast, whichis the identity in C;
parser.handalloc.hare unchanged.Sizes
parser.cis 49,392,087 bytes (udl) and 48,934,992 bytes (routine), bothunder the 50 MB line, so no LFS question arises. Headroom is small: upstream
grew each file by about 10 MB since
a7ffcdf.License
MIT, (c) 2025 InterSystems Corporation; the vendored
LICENSEisbyte-identical to the previous pin and to upstream
main. No copyleft codeis involved.
THIRD_PARTY.mdis updated; the SBOM generator counts grammardirectories and its aggregate MIT entry is unchanged (163 grammars).
Graph effect on a real corpus (vendor half only)
intersystems/ipm @
571e04a19aba, indexed before and after the grammarrefresh, both times with main's extractor (the extraction commits were not
built locally, so they are not in these numbers):
.clsfiles flaggedparse_partial/parse_unusableOf the 89 CALLS edges present only before, 16 were attributed to a file's
module node because the methods around them had been lost, and now come from
the method itself; 67 are
..GetModuleDir(...)calls from test classesthat inherit
Test.PM.Integration.Base, which become unresolved onceSync.cls(an unrelated class with its ownGetModuleDir, previously lostto a parse error) is extracted. That last effect is a resolution question,
not a parse one, and is tracked separately.
(177 of ipm's 429
.clsfiles declareClass %IPM...; main's.clsclassifier routes a
%-prefixed class name to Apex, so those files areunaffected by this change either way. Also tracked separately.)
Tests
tests/test_parse_coverage.c:objectscript_python_body_braces_parse_completely_issue1261(Pythonbodies with a brace in a string, a
#comment and a docstring, followedby an ObjectScript method: all four methods extracted, no parse error) and
objectscript_unclosed_python_body_remains_partial_issue1261(a Pythonbody that never closes is still reported). With the
a7ffcdfgrammar thefirst fails (
NoteandAftermissing, and no partial flag); withv1.11.0 both pass.
objectscript_udl_python_body_braces_do_not_swallow_class(thesix Language = python method body scanner drops all following methods in the class #1261 shapes, ported in commit 2): run against main's extractor, four
shapes fail on
a7ffcdfand all six pass on v1.11.0.objectscript_routine_call_forms_callee_textandobjectscript_udl_storage_docstringpin the routine call forms andStorage XML properties under the new shapes; with main's extractor and
v1.11.0 they fail, which is what commit 2 fixes.
Verification
Local, vendor half only (this grammar with main's extractor), narrow suites
under ASan/UBSan:
parse_coverage(73 passed),grammar_regression,grammar_labels,language,lang_contract,registry,repro_call_argument_matrix_b,repro_call_node_manifest,repro_lexical_binding_precision,pipeline_objectscript_timeoutandpipeline_semantic_manifest_repropass,and the bug-repro board's
repro_grammar_miscObjectScript batteries pass(18 of 18).
extraction,repro_call_scope_usages,repro_call_node_behaviorsandrepro_language_registryfail with thevendor commit alone, exactly as listed under "Why the halves land together";
the extraction commits address each of them.
Extraction half (commits 2 and 3): not built locally; this PR's CI is its
first build. Checked without compiling: every hunk applies to this tree in
commit order with no fuzz; the six files of commit 2 have the same patch-ids
as #2368's changes to them, and commit 3 has the same patch-id and message
as #2368's commit; the touched production files are clang-format clean; the
added lines contain no
NOLINT, raw allocation orfopen. The hunks werereviewed line by line.
Vendored integrity, on the final tree:
scripts/security-vendored.sh(
--updatechanges onlyMANIFEST.mdand the eight refreshed ObjectScriptfiles against main; verify passes),
tests/test_vendored_integrity_contract.sh,scripts/audit-grammar-security.sh,tests/test_language_count_contract.sh(163, unchanged) and
scripts/check-dco.sh origin/main..HEADpass. Thelicense provenance audit runs in the hosted security workflow.
Fixes #1261
Lands #2368 (closed by hand once this merges).