Repository navigation
fix(python): bind function-local imports in the scope of their def (#2285) - #2976
DryadeCore wants to merge 3 commits into
Conversation
…eusData#2285) An import inside a def body was never recorded (parse_python_imports walks the module's direct statements only) and py_process_statement had no case for an import statement. Inside the def, `Store` fell through to the module scope: `store = Store(); store.load_local()` stayed unresolved and the weak-member guard refused a name guess, so the call got no CALLS edge. When the module bound the same name, the call resolved against the module-level class instead: a wrong edge. - Extraction records every import inside a def body, at any depth, flagged CBMImport.function_scope. A local name a module-level import binds as well stays unrecorded: per-name consumers cannot tell the two bindings apart, so the def's import then binds UNKNOWN (fail closed). - cbm_import_binds_runtime_name() is false for a flagged import: no file-wide CALLS import map binds it. The LSP maps already include it. - py_process_statement replays an import statement where it stands, into the current scope, matching the import entries afresh per statement. - python_receiver_is_exempt ignores flagged imports, so a def's import does not exempt a same-named receiver elsewhere in the file. Tests: extraction flags (def, try in a method, nested def, shadowed alias, module-level if); pipeline sequential + parallel: module-level control, import in the def, import in a try block, an unrelated class with the same method name, a sibling def without the import, and a def whose import shadows a module-level name. Co-Authored-By: Dammerzone <dammerzone@users.noreply.github.com> Signed-off-by: Dryade <contact@dryade.ai>
|
Thanks for opening this — it has been seen, and it is queued. This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence. Current review status: working through a backlog. What that means for this PR, concretely:
Things that will genuinely speed it up whenever review does happen:
If this fixes a bug, a reproduction we can run is worth more than a description of the symptom. Thanks for contributing, and sorry in advance for the wait. |
…n the scope of their def Conflict in internal/cbm/cbm.h cbm_import_binds_runtime_name, kept both: - lev: a re-export binds no local name (cbm_import_binds_local). - DeusData#2976: an import inside a def (function_scope) binds no module-wide name; the Python LSP binds it where it stands. The rule is now binds_local && !type_checking_only && !function_scope. Signed-off-by: chidev <chicagowebdev@gmail.com>
DeusData
left a comment
There was a problem hiding this comment.
Thank you for this, and for the care that went into it: the root-cause write-up, the module-level controls, the parallel twin of the pipeline test and the reproduce-first evidence made it a pleasure to review. The core idea is right. Binding a def's import where the statement stands is exactly what #2285 needs, and we confirmed your three new tests fail on main for the right reason and pass on your branch merged with current main.
We also indexed two real projects before and after the change (django and pytest 8.3.5, sequential and parallel), and it does a lot of good: 93 existing calls move from a name guess to an exact LSP binding, 47 new LSP edges appear, 18 edges that pointed at the wrong target now point at the right one (for example reverse inside URLNode.render no longer binds ListMixin.reverse, and the postgresql backend tests now bind postgresql.base.DatabaseWrapper instead of the GIS MySQL one), and wrong edges such as selenium's Select -> django.forms.widgets.Select disappear. Graphs of non-Python projects are byte-identical.
The same runs show a few regressions in exactly the case the PR targets, and we would like these fixed before merging:
-
Function-local imports of project code lose their name-based fallback. On django 27 correct CALLS edges disappear (for example
AdminSite.get_urls -> django.urls.conf.re_path,check_url_namespaces_unique -> get_resolver,apply_limit_choices_to_to_formfield -> Q / Exists / OuterRef,OGRGeometry.geos -> GEOSGeometry), and on pytest 7 (getfuncargnames -> outcomes.fail,Parser.parse -> try_argcomplete, ...). These are calls the LSP cannot type (a decorated function, a functools.partial, a re-exported name), so on main they resolved through unique_name / suffix_match. The cause iscbm_suppress_external_import_shadowin src/pipeline/registry.c: it reads "the import map has no key for this name" as evidence that the import is external, andcbm_import_binds_runtime_name()now keeps every function-local import out of that map. Skipping those imports in the guard's loop, right after the local_name match:if (imp->function_scope) { continue; /* a def's own import never enters the file-wide import map */ }restores all 34 edges in our runs; 18 of the 22 wrong edges your branch removes stay removed (they are caught by
cbm_python_import_binding_contradicts, which judges an import by its materialized IMPORTS edge) and the other 4 go back to main's behaviour. Minimal repro:pkg/__init__.py (empty) pkg/outcomes.py def _with_exception(t): def decorate(func): func.Exception = t return func return decorate @_with_exception(RuntimeError) def fail(reason=""): raise RuntimeError(reason) pkg/compat.py def getfuncargnames(function): try: return function.__code__.co_varnames except AttributeError: from pkg.outcomes import fail fail("no code object")main: getfuncargnames -> pkg.outcomes.fail (unique_name); this branch: no edge.
-
A re-exported name imported a second time binds UNKNOWN in the def. Once the module (or an earlier def) has replayed the import,
py_canonical_named_importhas rewritten that entry's QN in place to the re-export target, so the def's identical statement no longer matches it and the name binds UNKNOWN. On django this turnsFormsModelTestCase.test_formfield_initial -> forms.fields.CharField.__init__(an LSP edge on main) into a suffix_match guess todb.models.fields.CharField. Repro:pkg/__init__.py, pkg/models/__init__.py (empty) pkg/forms/fields.py class CharField: def __init__(self, max_length): self.max_length = max_length def clean(self, value): return value pkg/models/fields.py (the same class CharField) pkg/forms/__init__.py from pkg.forms.fields import CharField pkg/use.py from pkg.forms import CharField def reimported_in_def(): from pkg.forms import CharField field = CharField(20) return field.clean(2) pkg/use3.py def first_def(): from pkg.forms import CharField return CharField(40).clean(4) def second_def(): from pkg.forms import CharField return CharField(50).clean(5)main: reimported_in_def -> CharField.init and -> CharField.clean (LSP). This branch: a single suffix_match edge to the class. In use3.py, first_def resolves but second_def gets nothing. Importing from the defining module (
from pkg.forms.fields import CharField) works in both places, so it is the in-place rewrite. Matching each statement against an unmodified copy of the entry's import path, and canonicalizing only the bound value, should fix both shapes; please add both to the pipeline test. -
Two pytest edges through a dotted import inside a method disappear:
pytestPDB._init_pdbandPytestPdbWrapper.do_continuedoimport _pytest.configand then call_pytest.config.create_terminal_writer(...). main has lsp_module_attr edges for both; this branch has none. We could not reduce it to a small fixture yet. It may be related to how the replayed unaliased dotted import binds the root_pytest. Could you take a look? -
Sequential and parallel indexing now disagree on base classes.
build_import_mapin src/pipeline/pass_semantic.c (the sequential INHERITS / DECORATES map) takes every recorded import, while the parallel resolver's map goes throughcbm_import_binds_runtime_name(). On django the nestedCustomDatabaseWrapper(DatabaseWrapper)classes in the postgresql backend tests inherit frompostgresql.base.DatabaseWrapperin a sequential index and from the GIS MySQL wrapper in a parallel one. Please skip function_scope imports in that map too (its fast path and its IMPORTS-edge fallback), so both drivers agree. Resolving nested-class bases through a def's import can then be a follow-up for both drivers together.
Smaller notes, not blocking:
- In the
shadoweddef of your test, a bare constructor callOther()still binds the module-levelother.Otherthrough the file-wide import map (main does the same through the LSP), so "fail closed" currently holds for member calls only. An extra assertion would document that. - The new IMPORTS edges for function-local imports are welcome: 307 of 316 on django and 66 of 72 on pytest name an in-project module, and "who imports X" then also counts lazy imports. The few wrong ones come from an existing problem that module-level imports already show on main: a bare absolute import such as
import warningsbinds to a same-named project module like_pytest/warnings.py, although Python 3 resolves it to the standard library. That one is not yours to fix, so we will add a commit for it on top of your branch in this PR once your round is in. py_process_statementallocates a fresh array per visited import statement; one per-context scratch buffer cleared per statement would avoid that. Minor.
Thanks again. This is a valuable fix, and we are happy to help with any of the above. Once the changes are in, we will re-run the same before/after comparison.
…g and driver parity (DeusData#2285 review) - External-import guard: cbm_suppress_external_import_shadow skips a def's own import. It never enters the file-wide import map, so its missing key is no evidence of an external binding; a call the LSP cannot type (a decorated function) keeps its name-based edge again. - Re-exported names: py_lsp keeps the module path each import row was added with (import_source_qns) and matches every import statement against it. An earlier replay canonicalizes import_module_qns to the re-export target, so the same statement in a later def no longer matched and bound UNKNOWN. - Driver parity: the sequential INHERITS / DECORATES import map (pass_semantic build_import_map) skips function_scope imports like the parallel driver's map, so a class nested in a def gets the same base in both drivers. Its IMPORTS-edge fallback runs only for a file with no extracted import, so no def's import reaches it. - py_process_statement reuses one per-context consumed buffer per import statement instead of an arena allocation each time. Tests: pipeline twins for the decorated-function fallback, both re-export shapes and the dotted import inside a method (edge kept), a sequential/parallel INHERITS parity test, an assertion documenting that a bare constructor call in the shadowing def still binds the module-level class, and a registry unit row. Co-Authored-By: Dammerzone <dammerzone@users.noreply.github.com> Signed-off-by: Dryade <contact@dryade.ai>
|
Thanks for the careful review and the before/after runs. Points 1, 2 and 4 are fixed in ab6aa4c and point 3 is diagnosed but not fixed (details below), on top of a merge of current 1. Name fallback lost (decorated functions). 2. Re-exported names bound UNKNOWN in a later def. 3. 4. Sequential / parallel parity. Non-blocking notes:
Verification, locally on linux/arm64 (ASan + UBSan build): the new tests for points 1, 2 and 4 fail on the previous head and pass now; suites Edit: corrected point 3 and the verification paragraph; my first version of this comment overstated both. |
What does this PR do?
A class imported inside a function body never typed anything in that function, so a
method call on an instance of it got no CALLS edge. Found while indexing a Python
project. This is the first gap of #2285 (function-local imports), for the receiver-type
case.
Minimal repro
On main (a3e9c91)
load_topandload_kwhave their caller;load_localandload_local_tryhave none. A function-localfrom m import ffollowed by a baref()already resolves (through the name-based fallback); the gap is receiver typing:
store = Store()whereStoreis imported inside the def, thenstore.method().Root cause
parse_python_imports(internal/cbm/extract_imports.c) records only the module'sdirect import statements (plus a module-level
if TYPE_CHECKING:block since Python cross-file receiver inference loses typed instance fields #1277).An import inside a def body is never recorded, so it never reaches the LSP import map.
py_process_statement(internal/cbm/lsp/py_lsp.c) has no case for an importstatement: only the module-level replay in
py_lsp_process_filebinds imports. Insidea function the name
Storefalls through to the module scope, where it is unbound,so
Store()is UNKNOWN,store.load_localis unresolved, and the weak-member guardthen (correctly) refuses a name guess. When the module scope does bind the same name,
the function's own import is ignored and the call resolves against the module-level
class: a wrong edge.
Fix
Same shape as the TYPE_CHECKING fix of #1277:
nested defs, class bodies inside the def), flagged
CBMImport.function_scope. A localname that a module-level import binds as well stays unrecorded: per-name consumers
(IMPORTS edge local names, the import maps) cannot tell two bindings of one name
apart, so the function's import then binds UNKNOWN in its def (fail closed).
cbm_import_binds_runtime_name()is false for a flagged import, so no file-wide CALLSimport map binds it. The Python LSP map already takes
include_type_onlymaps, and theparallel dispatch builds its own LSP map for a file with such an import.
py_process_statementreplays an import statement where it stands, into the currentscope, with the existing
py_replay_import_statement. Each statement matches theimport entries afresh (several defs may import one name). An unmatched import binds
UNKNOWN, which also stops a function-local import from being resolved through a
same-named module-level binding.
python_receiver_is_exemptignores flagged imports, so a function's import does notexempt a same-named receiver elsewhere in the file from the weak-member guard.
Graph effect besides the CALLS edges: a function-local import now gets its IMPORTS edge
from the file (the edge carries the local name as before), which #2285 also lists as
missing. Not in this PR: nested definitions and the package-map path (the other two gaps
of #2285).
Tests
tests/test_extraction.cpython_function_local_imports_flagged_issue2285: flags atdef, try-in-method and nested-def depth; module-level import unflagged; a def's alias
of a module-bound name unrecorded; a module-level
ifimport still unrecorded.tests/test_pipeline.cpython_function_local_import_issue2285(+_parallel, 50padding files to take the parallel path), asserting CALLS edges:
use_top -> Store.load_top= 1 (LSP)use_local -> Store.load_local= 1 (LSP)use_local_try -> Store.load_local_try= 1 (LSP)use_local -> Other.load_local= 0no_local_import -> Store.load_local= 0from other import Other:module_level -> Other.load_local= 1from store import Store as Other:shadowed -> Other.load_local= 0Results (linux arm64, gcc 13.3, base a3e9c91)
Fixture above,
query_graphper method, release build (scripts/build.sh):scripts/test.sh --suites extraction,extraction_imports,extraction_inheritance,py_lsp,py_lsp_stress,pipeline(ASan + UBSan):
1242 passed, exit 0, including the three new tests.Reproduce-first: with the extraction and LSP changes reverted and the tests kept:
Not run: the full default
scripts/test.shleg, the TSan leg, cppcheck, clang-tidy.Checklist
git commit -s)make -f Makefile.cbm test) -- see results above for what was runmake -f Makefile.cbm lint-ci) -- clang-format 20 clean on the changedhunks; cppcheck not run
Refs #2285
Written by an AI coding agent (Claude Code) operating this account on behalf of its owner, who authorised this submission and is accountable for it.
🤖 Generated with Claude Code