Repository navigation
Conversation
DeusData
left a comment
There was a problem hiding this comment.
Thank you, @isc-tdyar, and thanks for correcting the July note so clearly. Documenting the hidden wrapper rules (method_call in Do/Job, core_trigger/external_trigger, storage_body) saved us real digging, and the #1261 repro with six body shapes is exactly the kind of test we like.
The diff is too large for our automated review tooling, so we verified the vendored files by hand against a fresh clone of intersystems/tree-sitter-objectscript at 48a06b6401af. Every file is byte-identical to upstream except the two patches you declared:
- the include repoint in both
scanner.c; - the
uint8_t→intloop-counter widening in bothobjectscript_common.h.
The LICENSE files are unchanged, all 14 ObjectScript entries in vendored-checksums.txt match, and the generated files contain no non-ASCII bytes. The extractor changes are ObjectScript-only, and the MANIFEST and THIRD_PARTY updates are complete.
One tiny cosmetic note, not blocking: upstream has no v1.9.19 tag, and package.json/tree-sitter.json at that commit say 1.9.20. The SHA pin is what matters, and it's correct.
Approved. It merges once CI is green on this head. Thank you again!
|
Thanks for opening this — it has been seen, and it is queued. This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence. Current review status: working through a backlog. What that means for this PR, concretely:
Things that will genuinely speed it up whenever review does happen:
If this fixes a bug, a reproduction we can run is worth more than a description of the symptom. Thanks for contributing, and sorry in advance for the wait. |
|
@isc-tdyar, one small thing surfaced now that CI has reached this PR.
The eight grammar files you updated all match, just as our upstream byte-diff found. Running We're also looking at two sanitizer jobs that failed on this run ( |
DeusData
left a comment
There was a problem hiding this comment.
@isc-tdyar, a follow-up now that CI has reached this PR. It found one real problem beyond the checksum, so this isn't mergeable yet despite the earlier approval. Sorry for the back-and-forth, and thank you for bearing with it.
1. A call-scope regression in routines (the important one). repro_call_scope_objectscript_routine_tag_exact_owner passes on main but fails on this head in the sanitizer lane:
[call-scope] case=objectscript_routine_scope invariant=exact_call_owner_and_target expected=1 actual=0
[call-scope] case=objectscript_routine_scope invariant=callee_not_usage expected=0 actual=1
[call-scope] case=objectscript_routine_scope invariant=one_call expected=1 actual=0
FAIL tests/repro/repro_call_scope_usages.c:651: failures == 3, expected 0
With the new grammar, a tag call inside a routine no longer produces its CALLS edge and is recorded as a usage instead. The routine scanner changed shape in this re-vendor (COMPILED_HEADER and lex_compiled_header are gone), so the call probably now arrives under a node kind the routine extractor doesn't treat as a call site yet, much like the method_call / core_trigger wrappers you already handled for UDL. Could you look at what the routine tree now produces for that fixture and extend the extractor the same way? We keep graph quality ahead of grammar freshness, so we can't land a re-vendor that drops existing call edges.
2. A UBSan report in the scanners (small, but please guard it). Both objectscript_routine/scanner.c:84 and objectscript_udl/scanner.c:177 call memcpy(payload, buffer, length). Tree-sitter calls deserialize(NULL, 0) to reset, and memcpy with a NULL source is undefined behaviour even when the length is 0, so the sanitizer lane reports it. It's upstream code, and main has the same pattern, but it now shows up on this head's path. A one-line local patch, if (length) memcpy(payload, buffer, length);, recorded in MANIFEST.md next to the existing include and loop-counter patches, would settle it (or upstream it to intersystems/tree-sitter-objectscript).
3. The stale MANIFEST.md checksum from our earlier note, via scripts/security-vendored.sh --update.
The other red shards look like the same failure repeated across platforms. We'll re-check them once the fix is in. Thanks again. The byte-identical re-vendor and the wrapper work are solid, and this is the last stretch.
a59d53a to
f277c5e
Compare
|
Thanks. All three are addressed in the new push, rebased onto current I should have caught these locally. My full run had about 20 failures, and I Routine call scope. The routine grammar changed shape in v1.9.19.
Here I added UDL Scanner Checksums. Regenerated with |
|
Thank you, @isc-tdyar. This is a thorough round, and every point checks out:
One conflict is on us. After your rebase we merged a ReScript scanner patch (#2352) that edits the same MANIFEST table and the MANIFEST checksum line. The fix is to keep both sets of rows and re-run |
50717be to
cc71a20
Compare
|
Thanks. I've resolved it on my side: rebased onto current
|
|
Thanks for resolving the manifest and extraction conflicts and reporting the checks. That addresses the latest request in the thread. The updated head still needs the maintainer recheck before landing; no further contributor changes are requested here. |
Re-vendor objectscript_udl and objectscript_routine from
intersystems/tree-sitter-objectscript a7ffcdf -> 48a06b6 (v1.9.19). The
new scanner lexes Language = python method bodies with Python string,
triple-quote and '#'-comment rules, so a '{' inside a string or comment no
longer runs the body past its closing brace and swallows the rest of the
class.
Vendored files stay byte-identical to upstream apart from the two
documented local patches (objectscript_common.h include repoint and the
uint8_t -> int loop-counter widening), both re-applied. ABI stays 15.
v1.9.15+ also hides several wrapper nodes, which the extractor relied on:
- `Do obj.M()` / `Job obj.M()`: the method_call wrapper is gone and its
receiver + oref_method now sit directly under do_parameter /
job_argument. cbm_objectscript_is_instance_call() recognises both shapes
and cbm_is_call_site() feeds it to call and usage extraction.
- Trigger: core_trigger / external_trigger are hidden; the body is now
everything after trigger_name.
- Storage: storage_body is replaced by external_method_body_content.
The old node kinds are still accepted. expr_atom / glvn / member_name
lookups already fell back to their direct children and need no change.
Tests: objectscript_udl_python_body_braces_do_not_swallow_class (the
objectscript_udl_storage_docstring (Storage XML properties; previously
untested, fails without the new fallback).
Fixes DeusData#1261
Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
The v1.9.19 routine grammar no longer wraps every routine_tag_call and extrinsic_function target in line_ref: a bare label (`Do accept`) is now a direct method_name and a bare routine (`$$^Other`) a direct routine_ref. Look up all three containers so each call form keeps its previous callee text, and add an extraction test covering every form. Guard the zero-length memcpy in both ObjectScript scanner deserializers and record the patch in MANIFEST.md; refresh vendored checksums. Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
…mmar The v1.9.19 UDL grammar hides method_call: `Do obj.M(x)` now parses as do_parameter(lvn, oref_method). The registration could never match, so repro_primary_call_metadata_names_exist_in_grammars failed. Instance calls are still found by cbm_is_call_site through cbm_objectscript_is_instance_call, which already recognizes do_parameter/job_argument with an oref_method child. Remove the dead kind from objectscript_udl_call_types and its ledger row (historical 226 -> 225, direct/primary counts down by one), and point the UDL method-call behavior case at do_parameter, the node that now carries the call. Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
|
Thank you for the ObjectScript grammar refresh and for resolving the manifest and extraction conflicts. The revised head is waiting on our recheck, with no further contributor request. Sorry for the delay—I’ll follow up soon. |
cc71a20 to
5cad9a7
Compare
|
Thank you so much, @isc-tdyar, for your patience, and sorry this waited on us for so long. Your part of this PR has been complete since September: the #1261 repro with six body shapes, the routine callee-container fix covering all eight call forms, the method_call cleanup, and the clear write-up of which wrapper rules the newer grammar hides. Here is how it lands, in #2979. The re-vendor here changes about 3.6 million lines of generated parser code, more than GitHub can render as a diff, and our practice for vendored code is to re-vendor from upstream ourselves and verify every file against a fresh clone (you kindly offered exactly that). Upstream has since released v1.11.0, which parses your #1261 shapes and clears most of the remaining errors on a real codebase (intersystems/ipm: files with a parse error went from 138 to 3), so #2979 vendors v1.11.0 with the same three local patches. It turned out that the refreshed grammar and your extraction changes have to land together: with the new grammar alone,
Your scanner memcpy guard and the MANIFEST rows are not carried, because main already had the same guard from #2800. Once #2979 is merged we will close this PR with a link to the merge commit, and it closes #1261. Thank you again for the care you put into this one! |
|
Thanks for your collaboration and accepting my meager contributions! |
Fixes #1261
Re-vendors
objectscript_udlandobjectscript_routinefromintersystems/tree-sitter-objectscript
a7ffcdf→48a06b6(v1.9.19). Thenew scanner lexes
Language = pythonbodies with Python string, triple-quoteand
#-comment rules. A{inside a Python string no longer runs the bodypast its closing brace and swallows the rest of the class.
Vendoring
patches, both re-applied: the
objectscript_common.hinclude repoint andthe
uint8_t→intloop-counter widening.tree_sitter/array.hchangescome from upstream's own tree-sitter bump (a C++-only cast).
scripts/vendored-checksums.txtare updated. Only the 8 re-vendored fileschange checksums.
Correction to my July note
The body node was not renamed. It is still
external_method_body_content.What did change is that v1.9.15+ hides several wrapper rules, which broke
7 existing extraction tests:
method_callinDo/Job: the receiver andoref_methodnow sitdirectly under
do_parameter/job_argument. The newcbm_objectscript_is_instance_call()andcbm_is_call_site()feed thatshape to call and usage extraction (UDL only, as before).
core_trigger/external_trigger: inlined intotrigger. The body is noweverything after
trigger_name.storage_body: replaced byexternal_method_body_content, handled with afallback lookup.
The old node kinds are still accepted. The
expr_atom,glvnandmember_namelookups already fall back to direct children, so they need nochange.
Tests
objectscript_udl_python_body_braces_do_not_swallow_classis the Language = python method body scanner drops all following methods in the class #1261repro. It covers six body shapes, and 4 of them fail on
a7ffcdf. For eachshape it asserts that the following methods exist, that the Python body
ends before them, and that their CALLS survive.
objectscript_udl_storage_docstringcovers Storage XML properties. Thiswas previously untested, and the test fails without the new fallback.
scripts/lint.sh --cipasses. Extraction and pipeline suites pass. In thefull suite, the only failures are the same environmental
daemon/version-cohort/VS Code-install tests that fail on
mainon thismachine.
Happy to split the re-vendor into its own commit, or drop it and keep only
the extractor changes if you'd rather vendor it yourself.