Skip to content

fix(objectscript): refresh the grammars to v1.11.0 and land #2368's extraction changes (#1261) - #2979

Open
DeusData wants to merge 5 commits into
mainfrom
fix/objectscript-v1.11.0-1261
Open

DeusData wants to merge 5 commits into
mainfrom
fix/objectscript-v1.11.0-1261

Conversation

@DeusData

Copy link
Copy Markdown
Owner

What this lands

Fixes #1261, and lands #2368 by Thomas Dyar (@isc-tdyar) together with a
newer, re-verified grammar. Thank you, Thomas: the #1261 analysis, the
documentation of the hidden wrapper rules, and the extraction work are
yours, and this PR carries them with your authorship.

Commits, in order:

  1. Vendor commit (ours): the two InterSystems ObjectScript grammars,
    objectscript_udl (.cls) and objectscript_routine
    (.mac/.inc/.int/.rtn), from
    intersystems/tree-sitter-objectscript
    a7ffcdf2de8e (2026-06-15) to tag v1.11.0, full commit
    25cf0f67a3804727e250076a0926aa71c46fffab (2026-10-22, upstream main
    HEAD). MIT, (c) 2025 InterSystems Corporation. Both stay ABI 15.
  2. His extraction changes, ported onto this grammar as one commit with
    Co-authored-by: Thomas Dyar: the extractor and test hunks of his
    re-vendor commit and of his routine-call commit from fix(objectscript): re-vendor grammar for Language = python bodies #2368. That covers
    the Do obj.M() instance-call site detection, the routine callee
    container (line_ref / method_name / routine_ref), the Trigger and
    Storage body fallbacks, and his tests: the Language = python method body scanner drops all following methods in the class #1261 repro, the eight routine
    call forms and Storage XML properties. The six touched files carry exactly
    his changes (identical patch-ids against fix(objectscript): re-vendor grammar for Language = python bodies #2368). Not ported: his vendored
    files, his zero-length memcpy guard (main already has the same guard
    from fix: guard empty ObjectScript scanner state #2800), and his MANIFEST, THIRD_PARTY and checksum lines, all
    replaced by commit 1.
  3. His fix(lang_specs): drop ObjectScript UDL method_call, hidden by the grammar, cherry-picked unchanged with his authorship and sign-off.
  4. A merge of current main: one conflict in
    scripts/vendored-checksums.txt (with fix: two causes of run-to-run graph variation and three silent content losses #2974's jsonnet scanner patch),
    resolved by regenerating the file with scripts/security-vendored.sh.

GitHub cannot render #2368's diff because of the two generated parsers, so
the grammar is re-vendored and re-verified here, at a newer pin.

Build note: the extraction half (commits 2 and 3) was not built locally;
this PR's CI is its first build. Every local proof below (suites, corpus
numbers, regeneration) covers the vendor half only, run with main's
extractor.

Why refresh (#1261)

At a7ffcdf the scanner lexed a Language = python method body by
balancing braces only. A brace inside a Python string, comment or docstring
ran the body past its closing }, and the methods after it were lost. On
main the loss is silent: the new parse-coverage test below finds no
parse_partial flag on that file, only the missing methods. Upstream lexes
these bodies with Python's string, triple-quote and #-comment rules since
v1.9.15.

Why the halves land together

The refresh changes node shapes that main's ObjectScript extraction depends
on. From v1.9.15 upstream hides method_call inside Do/Job (the receiver
and oref_method sit directly under do_parameter/job_argument), inlines
core_trigger/external_trigger into trigger, replaces storage_body
with external_method_body_content, and wraps a routine Do label /
$$^routine target in method_name / routine_ref instead of always
line_ref. With main's extractor and this grammar alone:

  • 7 existing extraction tests fail: Do obj.M() instance calls lose their
    CALLS edge (objectscript_udl_calls_typed_new, _typed_param,
    _typed_property, objectscript_data_flows_instance_method_args,
    iris_export_xml_calls_extracted) and Trigger bodies lose their tokens
    (objectscript_udl_trigger_body_quit, _trigger_body_tokens);
  • 5 call-node repro cases fail: the typed-receiver and routine tag-call
    ownership cases in repro_call_scope_usages and
    repro_call_node_behaviors, and the method_call row in
    repro_language_registry (the node no longer exists in the grammar).

These are exactly the shapes the extraction commits handle (commits 2 and
3), and #2368's own tests pin them; with the 48a06b6 grammar of #2368 the
same failures occur, so they come from shape changes both pins share.

Why v1.11.0

Four candidates were staged the same way (upstream files plus our three
local patches) and compared with one parse harness: 48a06b6401af (the pin
in #2368, package version 1.9.20), v1.10.1 (942eedc37afe), v1.10.2
(a1a5d7ae8752) and v1.11.0 (25cf0f67a380).

a7ffcdf (main) 48a06b6 v1.10.1 v1.10.2 v1.11.0
#1261 shapes parsing clean (of 6) 1 6 6 6 6
ipm .cls files with an ERROR (of 429) 138 7 3 3 3
ipm bytes under a top-level ERROR 353,816 1,662 1,323 1,323 1,323
ipm method definitions outside ERROR 1,979 2,153 2,153 2,153 2,153
Samples-ObjectScript + webterminal files with an ERROR (of 35 .cls) 10 6 6 6 6

Corpora: intersystems/ipm @ 571e04a19aba (MIT; 429 .cls, 22 routine
files), intersystems/Samples-ObjectScript @ 8e63f5e7a522 and
intersystems-community/webterminal @ 91b54afa0ef4 (both MIT).

  • On the Language = python method body scanner drops all following methods in the class #1261 shapes, main's pin errors on five of six and loses the two
    following methods in four; every candidate parses all six cleanly.
  • v1.11.0 clears four ipm files that 48a06b6 still fails (three Storage
    *Reference classes and ModuleName.cls), and no file loses a method
    definition against main or against 48a06b6, in any corpus.
  • The node kinds that main's ObjectScript extraction names and that the
    refresh hides are the same set at 48a06b6 and at v1.11.0, so the
    extractor changes in fix(objectscript): re-vendor grammar for Language = python bodies #2368, written against 48a06b6, target exactly the
    shapes v1.11.0 produces.
  • v1.11.0 is upstream's HEAD and newest tag; v1.10.2 and v1.11.0 are
    identical on these corpora.

Provenance, by hash

  • Upstream layout maps as before: udl/src -> objectscript_udl/,
    objectscript_routine/src -> objectscript_routine/, common/scanner.h
    -> each directory's objectscript_common.h, root LICENSE -> each
    LICENSE.
  • Both parser.c files and all tree_sitter/*.h are upstream's files at
    25cf0f67a380, byte-identical.
  • Upstream generated both parsers with tree-sitter-cli 0.27.1 (its
    package-lock.json). Regenerating them here from upstream's committed
    src/grammar.json and tree-sitter.json with the 0.27.1 release binary
    (SHA-256 matches the release's published digest), in a no-network,
    write-confined sandbox, reproduces both byte-for-byte (SHA-256
    0c0c651b4a42... udl, d496de950b08... routine), and both
    node-types.json. No upstream JavaScript was evaluated.
  • LICENSE is byte-identical to the previous pin.
  • The three local patches recorded in MANIFEST.md are re-applied exactly as
    main carries them; upstream v1.11.0 still has all three original forms:
    the #include "../../common/scanner.h" repoint to objectscript_common.h,
    the uint8_t -> int widening of two loop counters in
    objectscript_common.h, and the if (length > 0) guard before the
    deserialize memcpy (fix: guard empty ObjectScript scanner state #2800). The same staging procedure applied to
    a7ffcdf reproduces main's current files byte-for-byte.
  • Nothing else from upstream is vendored: no bindings, queries, scripts,
    workflows or test fixtures.

Generated-parser and scanner review

  • parser.c (udl / routine): 49,392,087 / 48,934,992 bytes; 0 non-ASCII and
    0 control bytes other than LF; full tokenization leaves 0 unmatched
    characters; one comment (the generator stamp); no string literal contains
    a space; no call outside the tree-sitter macros and the external-scanner
    entry points.
  • The hand-written scanner delta (scanner.c x2, objectscript_common.h;
    about 620 added lines) is lexer logic over the TSLexer API and wctype:
    fixed-size local arrays with bounds checks, no I/O, no environment or
    process access, no function pointers or mutable statics; the debug logger
    stays under #if 0. The UDL scanner's serialized state stays about 520
    bytes, well under tree-sitter's 1024-byte buffer.
  • tree_sitter/array.h gains tree-sitter's C++-only _array__cast, which
    is the identity in C; parser.h and alloc.h are unchanged.

Sizes

parser.c is 49,392,087 bytes (udl) and 48,934,992 bytes (routine), both
under the 50 MB line, so no LFS question arises. Headroom is small: upstream
grew each file by about 10 MB since a7ffcdf.

License

MIT, (c) 2025 InterSystems Corporation; the vendored LICENSE is
byte-identical to the previous pin and to upstream main. No copyleft code
is involved. THIRD_PARTY.md is updated; the SBOM generator counts grammar
directories and its aggregate MIT entry is unchanged (163 grammars).

Graph effect on a real corpus (vendor half only)

intersystems/ipm @ 571e04a19aba, indexed before and after the grammar
refresh, both times with main's extractor (the extraction commits were not
built locally, so they are not in these numbers):

before after
UDL .cls files flagged parse_partial / parse_unusable 89 / 4 0 / 0
Method nodes 974 1,141
CALLS edges 1,759 1,837
DEFINES_METHOD edges 969 1,136

Of the 89 CALLS edges present only before, 16 were attributed to a file's
module node because the methods around them had been lost, and now come from
the method itself; 67 are ..GetModuleDir(...) calls from test classes
that inherit Test.PM.Integration.Base, which become unresolved once
Sync.cls (an unrelated class with its own GetModuleDir, previously lost
to a parse error) is extracted. That last effect is a resolution question,
not a parse one, and is tracked separately.

(177 of ipm's 429 .cls files declare Class %IPM...; main's .cls
classifier routes a %-prefixed class name to Apex, so those files are
unaffected by this change either way. Also tracked separately.)

Tests

Verification

Local, vendor half only (this grammar with main's extractor), narrow suites
under ASan/UBSan:
parse_coverage (73 passed), grammar_regression, grammar_labels,
language, lang_contract, registry, repro_call_argument_matrix_b,
repro_call_node_manifest, repro_lexical_binding_precision,
pipeline_objectscript_timeout and pipeline_semantic_manifest_repro pass,
and the bug-repro board's repro_grammar_misc ObjectScript batteries pass
(18 of 18). extraction, repro_call_scope_usages,
repro_call_node_behaviors and repro_language_registry fail with the
vendor commit alone, exactly as listed under "Why the halves land together";
the extraction commits address each of them.

Extraction half (commits 2 and 3): not built locally; this PR's CI is its
first build.
Checked without compiling: every hunk applies to this tree in
commit order with no fuzz; the six files of commit 2 have the same patch-ids
as #2368's changes to them, and commit 3 has the same patch-id and message
as #2368's commit; the touched production files are clang-format clean; the
added lines contain no NOLINT, raw allocation or fopen. The hunks were
reviewed line by line.

Vendored integrity, on the final tree: scripts/security-vendored.sh
(--update changes only MANIFEST.md and the eight refreshed ObjectScript
files against main; verify passes), tests/test_vendored_integrity_contract.sh,
scripts/audit-grammar-security.sh, tests/test_language_count_contract.sh
(163, unchanged) and scripts/check-dco.sh origin/main..HEAD pass. The
license provenance audit runs in the hosted security workflow.

Fixes #1261
Lands #2368 (closed by hand once this merges).

DeusData and others added 4 commits November 14, 2026 00:18
Re-vendors objectscript_udl and objectscript_routine from
intersystems/tree-sitter-objectscript a7ffcdf2de8e to tag v1.11.0
(25cf0f67a380, upstream HEAD); both stay ABI 15. The refreshed scanner
lexes Language = python method bodies with Python's string, triple-quote
and #-comment rules, so a brace inside a Python string, comment or
docstring no longer runs the body past its closing brace and drops the
methods that follow it (#1261).

Both parser.c files are upstream's committed bytes, and regenerating them
from upstream's src/grammar.json and tree-sitter.json with tree-sitter-cli
0.27.1 (upstream's locked version) reproduces them byte-for-byte. The three
local patches recorded in MANIFEST.md are re-applied unchanged: the
objectscript_common.h include repoint, the uint8_t -> int widening of two
loop counters, and the empty-state memcpy guard; upstream v1.11.0 still
carries the original form of all three.

Updates the MANIFEST.md note, pin rows and ABI recount, THIRD_PARTY.md and
scripts/vendored-checksums.txt, and adds two parse-coverage tests: Python
bodies with braces in strings, comments and docstrings parse completely,
and an unclosed Python body is still reported as partial.

Refs #1261, #2368

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…action

Ports the first-party extraction changes of #2368 by Thomas Dyar onto the
v1.11.0 grammars: the extractor and test hunks of his re-vendor commit
(1a6b77c) and of his routine-call commit (2e84d7e). From upstream v1.9.15
on, the grammar hides `method_call` inside `Do`/`Job` (the receiver and
`oref_method` sit directly under `do_parameter`/`job_argument`), inlines
`core_trigger`/`external_trigger` into `trigger`, replaces `storage_body`
with `external_method_body_content`, and no longer always wraps a routine
call target in `line_ref`:

- cbm_objectscript_is_instance_call() and cbm_is_call_site() feed the
  `Do obj.M()` shape to call and usage extraction (UDL only, as before);
- objectscript_routine_callee_container() takes the callee of a
  routine_tag_call / extrinsic_function from `line_ref`, `method_name`
  (`Do accept`) or `routine_ref` (`$$^Other`), so every call form keeps
  its callee text;
- a Trigger body without a wrapper is everything after trigger_name
  (extract_ident_tokens_from() skips the subtrees before it);
- Storage XML properties fall back to external_method_body_content.

Tests (his): the #1261 repro (six Language = python body shapes, each
followed by two methods whose CALLS must survive), the eight routine call
forms, and Storage XML properties.

Not ported: his vendored grammar files, his zero-length memcpy guard (main
already carries the same guard from #2800) and his MANIFEST.md,
THIRD_PARTY.md and vendored-checksums.txt lines; the vendor commit before
this one replaces them. The six touched files carry exactly his changes
(identical patch-ids).

Refs #1261, #2368

Co-authored-by: Thomas Dyar <tdyar@intersystems.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…mmar

The v1.9.19 UDL grammar hides method_call: `Do obj.M(x)` now parses as
do_parameter(lvn, oref_method). The registration could never match, so
repro_primary_call_metadata_names_exist_in_grammars failed. Instance
calls are still found by cbm_is_call_site through
cbm_objectscript_is_instance_call, which already recognizes
do_parameter/job_argument with an oref_method child.

Remove the dead kind from objectscript_udl_call_types and its ledger row
(historical 226 -> 225, direct/primary counts down by one), and point the
UDL method-call behavior case at do_parameter, the node that now carries
the call.

Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
Brings in #2974 and #2970. One conflict: scripts/vendored-checksums.txt,
where #2974 (jsonnet scanner patch) and the vendor commit both rewrite the
MANIFEST.md line. Resolved by taking main's file and regenerating it with
scripts/security-vendored.sh --update; the result differs from main only in
the MANIFEST.md line and the eight refreshed ObjectScript files, and the
integrity verify passes. MANIFEST.md, THIRD_PARTY.md and
tests/test_parse_coverage.c merged cleanly with both sides kept.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Main brought in #2969 (vendored VB6/VBA grammar), #2981 (linear-growth
rewrites in extraction and eight LSP passes, a node for C# operators, C
increments recorded as writes), #2977 (receiver gate in the registry),
#2978 (possible override targets in trace_path), #2966 (CONTRIBUTING test
reference) and #2985 (freebsd-vm action bump).

Two files conflicted:

- internal/cbm/vendored/grammars/MANIFEST.md: the summary header. Took
  main's counts (164 grammars, 20 first-party, ABI 9x/78x/77x), which a
  recount of the merged tree confirms (164 grammar directories, LANGUAGE_VERSION
  13/14/15 = 9/78/77). Kept this branch's 2026-11-13 ObjectScript recount
  line above main's 2026-11-12 vba line; main had folded the 2026-11-11 sql
  line into its vba line, so that line goes. Main's vba prose and table rows
  and this branch's ObjectScript v1.11.0 rows are both present.
- scripts/vendored-checksums.txt: regenerated with
  scripts/security-vendored.sh --update. Only the MANIFEST.md hash moved;
  the result differs from this branch only by main's vba entries and from
  main only by the ObjectScript entries.

Every other file merged cleanly. For each file both sides changed
(THIRD_PARTY.md, extract_calls.c, extract_defs.c, extract_usages.c,
helpers.c, lang_specs.c, tests/test_parse_coverage.c) the merged file
differs from each parent by exactly the other parent's change.
#2981 rewrote objectscript_routine_preceding_tag onto a memoized sibling
search with the same probe; this branch does not touch that function.
ObjectScript stays outside #2977's receiver gate allowlist.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
@DeusData

Copy link
Copy Markdown
Owner Author

Merged current main into this branch again so it is mergeable (merge commit 0bde6de, second parent 1ec2f26).

What main brought in since the last merge: #2969 (vendored VB6/VBA grammar), #2981 (linear-growth rewrites in extraction and the LSP passes, C# operator nodes, C increments as writes), #2977 (registry receiver gate), #2978 (possible override targets in trace_path), #2966 (CONTRIBUTING fix) and #2985 (freebsd-vm bump).

Conflicts and how they were resolved:

  • internal/cbm/vendored/grammars/MANIFEST.md, summary header: main's counts (164 grammars, ABI 9x/78x/77x), confirmed by a recount of the merged tree, plus this branch's ObjectScript v1.11.0 recount line. Both the vba rows and the ObjectScript v1.11.0 rows are kept.
  • scripts/vendored-checksums.txt: regenerated with scripts/security-vendored.sh --update; only the MANIFEST.md hash changed.

Everything else merged cleanly, and for each file both sides touched the merged file differs from each parent by exactly the other side's change. ObjectScript is outside #2977's receiver gate, and #2981's rewrite of the routine preceding-tag lookup keeps the same probe.

As noted earlier, CI is the first build of the ported extraction changes from #2368. It is also the first run of main's new complexity_flat_preceding_label_is_linear test against the v1.11.0 routine grammar.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Language = python method body scanner drops all following methods in the class

2 participants