Skip to content

fix(objectscript): re-vendor grammar for Language = python bodies - #2368

Open
isc-tdyar wants to merge 3 commits into
DeusData:mainfrom
isc-tdyar:fix/objectscript-python-body
Open

isc-tdyar wants to merge 3 commits into
DeusData:mainfrom
isc-tdyar:fix/objectscript-python-body

Conversation

@isc-tdyar

Copy link
Copy Markdown
Contributor

Fixes #1261

Re-vendors objectscript_udl and objectscript_routine from
intersystems/tree-sitter-objectscript a7ffcdf → 48a06b6 (v1.9.19). The
new scanner lexes Language = python bodies with Python string, triple-quote
and #-comment rules. A { inside a Python string no longer runs the body
past its closing brace and swallows the rest of the class.

Vendoring

  • Files are byte-identical to upstream except for the two documented local
    patches, both re-applied: the objectscript_common.h include repoint and
    the uint8_t → int loop-counter widening. tree_sitter/array.h changes
    come from upstream's own tree-sitter bump (a C++-only cast).
  • ABI stays 15. MANIFEST (note and table rows), THIRD_PARTY and
    scripts/vendored-checksums.txt are updated. Only the 8 re-vendored files
    change checksums.

Correction to my July note

The body node was not renamed. It is still external_method_body_content.
What did change is that v1.9.15+ hides several wrapper rules, which broke
7 existing extraction tests:

  • method_call in Do / Job: the receiver and oref_method now sit
    directly under do_parameter / job_argument. The new
    cbm_objectscript_is_instance_call() and cbm_is_call_site() feed that
    shape to call and usage extraction (UDL only, as before).
  • core_trigger / external_trigger: inlined into trigger. The body is now
    everything after trigger_name.
  • storage_body: replaced by external_method_body_content, handled with a
    fallback lookup.

The old node kinds are still accepted. The expr_atom, glvn and
member_name lookups already fall back to direct children, so they need no
change.

Tests

  • objectscript_udl_python_body_braces_do_not_swallow_class is the Language = python method body scanner drops all following methods in the class #1261
    repro. It covers six body shapes, and 4 of them fail on a7ffcdf. For each
    shape it asserts that the following methods exist, that the Python body
    ends before them, and that their CALLS survive.
  • objectscript_udl_storage_docstring covers Storage XML properties. This
    was previously untested, and the test fails without the new fallback.

scripts/lint.sh --ci passes. Extraction and pipeline suites pass. In the
full suite, the only failures are the same environmental
daemon/version-cohort/VS Code-install tests that fail on main on this
machine.

Happy to split the re-vendor into its own commit, or drop it and keep only
the extractor changes if you'd rather vendor it yourself.

@isc-tdyar
isc-tdyar requested a review from DeusData as a code owner September 26, 2026 14:23

@DeusData DeusData left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, @isc-tdyar, and thanks for correcting the July note so clearly. Documenting the hidden wrapper rules (method_call in Do/Job, core_trigger/external_trigger, storage_body) saved us real digging, and the #1261 repro with six body shapes is exactly the kind of test we like.

The diff is too large for our automated review tooling, so we verified the vendored files by hand against a fresh clone of intersystems/tree-sitter-objectscript at 48a06b6401af. Every file is byte-identical to upstream except the two patches you declared:

  • the include repoint in both scanner.c;
  • the uint8_t → int loop-counter widening in both objectscript_common.h.

The LICENSE files are unchanged, all 14 ObjectScript entries in vendored-checksums.txt match, and the generated files contain no non-ASCII bytes. The extractor changes are ObjectScript-only, and the MANIFEST and THIRD_PARTY updates are complete.

One tiny cosmetic note, not blocking: upstream has no v1.9.19 tag, and package.json/tree-sitter.json at that commit say 1.9.20. The SHA pin is what matters, and it's correct.

Approved. It merges once CI is green on this head. Thank you again!

@github-actions

Copy link
Copy Markdown

Thanks for opening this — it has been seen, and it is queued.

This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence.

Current review status: working through a backlog. 0.9.1-rc.1 is out, so the release freeze that held reviews is over — but it left a large queue of open pull requests behind it, and we are reading through them oldest-first. The background is in discussion #1144.

What that means for this PR, concretely:

  • It will not be closed for inactivity. No stale bot touches pull requests here.
  • It may still sit a while before a human reads it. That is on us, not on you.
  • Older PRs are read first, so a recent one is not being skipped — it is behind a queue.

Things that will genuinely speed it up whenever review does happen:

  • Keep it rebased on main — the tree is moving quickly right now, and a conflicting branch cannot be reviewed as the diff you intended.
  • Get CI green, or say which failures you believe are pre-existing.
  • Keep the change to one claim. Bundled features and refactors get split before they get merged, which costs you a round trip.
  • Every commit needs a sign-off (git commit -s) — CI enforces DCO.

If this fixes a bug, a reproduction we can run is worth more than a description of the symptom.

Thanks for contributing, and sorry in advance for the wait.

@DeusData

Copy link
Copy Markdown
Owner

@isc-tdyar, one small thing surfaced now that CI has reached this PR. security / security-static fails at "Layer 8: Vendored dependency integrity", and it's only a stale checksum.

scripts/vendored-checksums.txt also pins internal/cbm/vendored/grammars/MANIFEST.md, and that line still has the old hash:

MISMATCH: internal/cbm/vendored/grammars/MANIFEST.md
  expected: a66a04ed735fdb3d4a90ee4353b8f0490dc0360da5965b0d13ec5d6766f146fa
  actual:   d8a1c02093f1512d278f1841f10a6694bb30c1fa035393365e90e1bf30f2ef0a

The eight grammar files you updated all match, just as our upstream byte-diff found. Running scripts/security-vendored.sh --update and committing the result should change exactly that one line. The check exists so that any edit to vendored content, the manifest included, is a deliberate, reviewed change. Your manifest edit is exactly that, so this just records it.

We're also looking at two sanitizer jobs that failed on this run (test-diag, test-lsan-macos) and will tell you if either is related to the change. Thank you!

@DeusData DeusData left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@isc-tdyar, a follow-up now that CI has reached this PR. It found one real problem beyond the checksum, so this isn't mergeable yet despite the earlier approval. Sorry for the back-and-forth, and thank you for bearing with it.

1. A call-scope regression in routines (the important one). repro_call_scope_objectscript_routine_tag_exact_owner passes on main but fails on this head in the sanitizer lane:

[call-scope] case=objectscript_routine_scope invariant=exact_call_owner_and_target expected=1 actual=0
[call-scope] case=objectscript_routine_scope invariant=callee_not_usage       expected=0 actual=1
[call-scope] case=objectscript_routine_scope invariant=one_call               expected=1 actual=0
FAIL tests/repro/repro_call_scope_usages.c:651: failures == 3, expected 0

With the new grammar, a tag call inside a routine no longer produces its CALLS edge and is recorded as a usage instead. The routine scanner changed shape in this re-vendor (COMPILED_HEADER and lex_compiled_header are gone), so the call probably now arrives under a node kind the routine extractor doesn't treat as a call site yet, much like the method_call / core_trigger wrappers you already handled for UDL. Could you look at what the routine tree now produces for that fixture and extend the extractor the same way? We keep graph quality ahead of grammar freshness, so we can't land a re-vendor that drops existing call edges.

2. A UBSan report in the scanners (small, but please guard it). Both objectscript_routine/scanner.c:84 and objectscript_udl/scanner.c:177 call memcpy(payload, buffer, length). Tree-sitter calls deserialize(NULL, 0) to reset, and memcpy with a NULL source is undefined behaviour even when the length is 0, so the sanitizer lane reports it. It's upstream code, and main has the same pattern, but it now shows up on this head's path. A one-line local patch, if (length) memcpy(payload, buffer, length);, recorded in MANIFEST.md next to the existing include and loop-counter patches, would settle it (or upstream it to intersystems/tree-sitter-objectscript).

3. The stale MANIFEST.md checksum from our earlier note, via scripts/security-vendored.sh --update.

The other red shards look like the same failure repeated across platforms. We'll re-check them once the fix is in. Thanks again. The byte-identical re-vendor and the wrapper work are solid, and this is the last stretch.

@isc-tdyar
isc-tdyar force-pushed the fix/objectscript-python-body branch from a59d53a to f277c5e Compare September 28, 2026 15:10
@isc-tdyar

Copy link
Copy Markdown
Contributor Author

Thanks. All three are addressed in the new push, rebased onto current
main, plus two more failures of the same kind that you didn't list.

I should have caught these locally. My full run had about 20 failures, and I
wrote them all off as environmental without checking each one. Three of
them were real: the routine repro you flagged, and two UDL ones covered
below. This time I checked every failure. The other 16 are daemon, version-cohort,
subprocess-timeout, and CLI-install tests that depend on this machine's
sandbox. None of them touch ObjectScript.

Routine call scope. The routine grammar changed shape in v1.9.19.
routine_tag_call and extrinsic_function no longer always wrap their
target in line_ref:

Form Old child New child
Do accept(x) line_ref(identifier) method_name(identifier)
Do accept line_ref(identifier) method_name(identifier)
$$^Other(1) line_ref(routine_name) routine_ref(routine_name)
accept^Other etc. line_ref(...) line_ref(...) (unchanged)

Here identifier is objectscript_identifier. The node text is identical
in each case. A new helper,
objectscript_routine_callee_container, checks line_ref, then
method_name, then routine_ref. Both the callee-name path and
objectscript_callee_expr use it, so the callee occurrence is consumed
again and accept no longer leaks as a usage. objectscript_callee_leaf
already handled method_name. routine_ref has no identifier leaf, the same
as the old line_ref(routine_name).

I added objectscript_routine_call_forms_callee_text to pin the callee text
of all eight call forms (Do label/label+args/label^routine/^routine/
label+offset^routine, and $$ label/label^routine/^routine). It failed on
the previous head: 1 of 3 accept calls was extracted. The repro test
passes now.

UDL method_call. The UDL grammar now hides method_call, so
Do target.accept(x) parses as do_parameter(lvn, oref_method). The
registration could never match, which failed
repro_primary_call_metadata_names_exist_in_grammars and the AST-kind
check in repro_call_node_behavior_objectscript_udl_method_call. Extraction
was unaffected: cbm_is_call_site already finds these calls through
cbm_objectscript_is_instance_call. I dropped method_call from
objectscript_udl_call_types, removed its ledger row (226 to 225, with the
direct and primary counts down by one), and pointed the behavior case at
do_parameter. The suites the ledger header names all pass. This is a
separate commit in case you'd rather handle the ledger differently.

Scanner memcpy. Both deserializers now use
if (length) memcpy(payload, buffer, length);, recorded as two rows in
the MANIFEST.md local-patch table next to the rescript entry. The
ObjectScript vendoring note also points to it.

Checksums. Regenerated with scripts/security-vendored.sh --update.
Three lines changed: MANIFEST.md and the two scanner.c files.

@DeusData

Copy link
Copy Markdown
Owner

Thank you, @isc-tdyar. This is a thorough round, and every point checks out:

  • objectscript_routine_callee_container restores the routine tag-call edge. The sanitizer lane that caught the regression is green again (8218 passed, 0 failed).
  • The if (length) guards are in both scanners, their MANIFEST local-patch rows are recorded, and the checksums match.
  • objectscript_routine_call_forms_callee_text pins all eight call forms.
  • Dropping the method_call row fits the ledger's rule that every registered call node owns a row. Thanks, too, for catching the two UDL failures we had missed.

One conflict is on us. After your rebase we merged a ReScript scanner patch (#2352) that edits the same MANIFEST table and the MANIFEST checksum line. The fix is to keep both sets of rows and re-run scripts/security-vendored.sh --update to regenerate the checksum. We're happy to do that on your branch if you prefer. Just say which you'd like. Once it's resolved and the fresh CI (including the macOS legs) is green, we'll approve and merge.

@isc-tdyar
isc-tdyar force-pushed the fix/objectscript-python-body branch 2 times, most recently from 50717be to cc71a20 Compare September 29, 2026 07:19
@isc-tdyar

Copy link
Copy Markdown
Contributor Author

Thanks. I've resolved it on my side: rebased onto current main, kept both
the #2352 ReScript row and the two ObjectScript rows in the MANIFEST table,
and regenerated the checksums with scripts/security-vendored.sh --update.

main moved again while that was in flight, so there was a second conflict
in extract_body_ident_tokens (extract_defs.c). I kept the new heap-spill
stack and its contract comment, and re-applied the from_byte parameter the
Trigger path uses. The integrity check, lint, and the extraction suite pass
locally. Fresh CI is running now.

@DeusData DeusData added bug Something isn't working parsing/quality Graph extraction bugs, false positives, missing edges dependencies Pull requests that update a dependency file labels Oct 19, 2026
@DeusData

Copy link
Copy Markdown
Owner

Thanks for resolving the manifest and extraction conflicts and reporting the checks. That addresses the latest request in the thread. The updated head still needs the maintainer recheck before landing; no further contributor changes are requested here.

Re-vendor objectscript_udl and objectscript_routine from
intersystems/tree-sitter-objectscript a7ffcdf -> 48a06b6 (v1.9.19). The
new scanner lexes Language = python method bodies with Python string,
triple-quote and '#'-comment rules, so a '{' inside a string or comment no
longer runs the body past its closing brace and swallows the rest of the
class.

Vendored files stay byte-identical to upstream apart from the two
documented local patches (objectscript_common.h include repoint and the
uint8_t -> int loop-counter widening), both re-applied. ABI stays 15.

v1.9.15+ also hides several wrapper nodes, which the extractor relied on:

- `Do obj.M()` / `Job obj.M()`: the method_call wrapper is gone and its
  receiver + oref_method now sit directly under do_parameter /
  job_argument. cbm_objectscript_is_instance_call() recognises both shapes
  and cbm_is_call_site() feeds it to call and usage extraction.
- Trigger: core_trigger / external_trigger are hidden; the body is now
  everything after trigger_name.
- Storage: storage_body is replaced by external_method_body_content.

The old node kinds are still accepted. expr_atom / glvn / member_name
lookups already fell back to their direct children and need no change.

Tests: objectscript_udl_python_body_braces_do_not_swallow_class (the
objectscript_udl_storage_docstring (Storage XML properties; previously
untested, fails without the new fallback).

Fixes DeusData#1261

Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
The v1.9.19 routine grammar no longer wraps every routine_tag_call and
extrinsic_function target in line_ref: a bare label (`Do accept`) is now
a direct method_name and a bare routine (`$$^Other`) a direct
routine_ref. Look up all three containers so each call form keeps its
previous callee text, and add an extraction test covering every form.

Guard the zero-length memcpy in both ObjectScript scanner deserializers
and record the patch in MANIFEST.md; refresh vendored checksums.

Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
…mmar

The v1.9.19 UDL grammar hides method_call: `Do obj.M(x)` now parses as
do_parameter(lvn, oref_method). The registration could never match, so
repro_primary_call_metadata_names_exist_in_grammars failed. Instance
calls are still found by cbm_is_call_site through
cbm_objectscript_is_instance_call, which already recognizes
do_parameter/job_argument with an oref_method child.

Remove the dead kind from objectscript_udl_call_types and its ledger row
(historical 226 -> 225, direct/primary counts down by one), and point the
UDL method-call behavior case at do_parameter, the node that now carries
the call.

Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
@DeusData

Copy link
Copy Markdown
Owner

Thank you for the ObjectScript grammar refresh and for resolving the manifest and extraction conflicts. The revised head is waiting on our recheck, with no further contributor request. Sorry for the delay—I’ll follow up soon.

@DeusData

Copy link
Copy Markdown
Owner

Thank you so much, @isc-tdyar, for your patience, and sorry this waited on us for so long. Your part of this PR has been complete since September: the #1261 repro with six body shapes, the routine callee-container fix covering all eight call forms, the method_call cleanup, and the clear write-up of which wrapper rules the newer grammar hides.

Here is how it lands, in #2979. The re-vendor here changes about 3.6 million lines of generated parser code, more than GitHub can render as a diff, and our practice for vendored code is to re-vendor from upstream ourselves and verify every file against a fresh clone (you kindly offered exactly that). Upstream has since released v1.11.0, which parses your #1261 shapes and clears most of the remaining errors on a real codebase (intersystems/ipm: files with a parse error went from 138 to 3), so #2979 vendors v1.11.0 with the same three local patches.

It turned out that the refreshed grammar and your extraction changes have to land together: with the new grammar alone, Do obj.M() and routine Do label() calls lose their edges, which is exactly what your changes fix. So #2979 carries both, in this order:

  1. our re-vendor at v1.11.0;
  2. your extraction changes from this PR, exactly as you wrote them (the same patch), with you as co-author on the commit;
  3. your fix(lang_specs): drop ObjectScript UDL method_call commit, cherry-picked unchanged under your name and sign-off.

Your scanner memcpy guard and the MANIFEST rows are not carried, because main already had the same guard from #2800. Once #2979 is merged we will close this PR with a link to the merge commit, and it closes #1261. Thank you again for the care you put into this one!

@isc-tdyar

Copy link
Copy Markdown
Contributor Author

Thanks for your collaboration and accepting my meager contributions!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working dependencies Pull requests that update a dependency file parsing/quality Graph extraction bugs, false positives, missing edges

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Language = python method body scanner drops all following methods in the class

2 participants