Skip to content

fix(python): bind function-local imports in the scope of their def (#2285) - #2976

Open
DryadeCore wants to merge 3 commits into
DeusData:mainfrom
DryadeCore:fix/python-local-import-receiver-type
Open

DryadeCore wants to merge 3 commits into
DeusData:mainfrom
DryadeCore:fix/python-local-import-receiver-type

Conversation

@DryadeCore

Copy link
Copy Markdown

What does this PR do?

A class imported inside a function body never typed anything in that function, so a
method call on an instance of it got no CALLS edge. Found while indexing a Python
project. This is the first gap of #2285 (function-local imports), for the receiver-type
case.

Minimal repro

pkg/__init__.py    (empty)
pkg/store.py       class Store: def load_top / load_local / load_local_try / load_kw
pkg/a_top.py       from pkg.store import Store          (module level)
                   def use_top(): store = Store(); return store.load_top(1)
pkg/b_local.py     def use_local():
                       from pkg.store import Store      (inside the def)
                       store = Store(); return store.load_local(1)
pkg/c_local_try.py def use_local_try():
                       try:
                           from pkg.store import Store  (inside a try block of the def)
                           store = Store(); items = store.load_local_try(1)
                       except Exception: return None
                       return items
pkg/d_kw.py        module-level import, call inside try with a keyword arg
codebase-memory-mcp cli index_repository --repo-path repro --name fixture
codebase-memory-mcp cli query_graph --project fixture \
  --query "MATCH (a)-[:CALLS]->(b) WHERE b.name = 'load_local' RETURN a.name, b.qualified_name"

On main (a3e9c91) load_top and load_kw have their caller; load_local and
load_local_try have none. A function-local from m import f followed by a bare f()
already resolves (through the name-based fallback); the gap is receiver typing:
store = Store() where Store is imported inside the def, then store.method().

Root cause

  • parse_python_imports (internal/cbm/extract_imports.c) records only the module's
    direct import statements (plus a module-level if TYPE_CHECKING: block since Python cross-file receiver inference loses typed instance fields #1277).
    An import inside a def body is never recorded, so it never reaches the LSP import map.
  • py_process_statement (internal/cbm/lsp/py_lsp.c) has no case for an import
    statement: only the module-level replay in py_lsp_process_file binds imports. Inside
    a function the name Store falls through to the module scope, where it is unbound,
    so Store() is UNKNOWN, store.load_local is unresolved, and the weak-member guard
    then (correctly) refuses a name guess. When the module scope does bind the same name,
    the function's own import is ignored and the call resolves against the module-level
    class: a wrong edge.

Fix

Same shape as the TYPE_CHECKING fix of #1277:

  • Extraction records every import inside a def body, at any depth (try/if/with,
    nested defs, class bodies inside the def), flagged CBMImport.function_scope. A local
    name that a module-level import binds as well stays unrecorded: per-name consumers
    (IMPORTS edge local names, the import maps) cannot tell two bindings of one name
    apart, so the function's import then binds UNKNOWN in its def (fail closed).
  • cbm_import_binds_runtime_name() is false for a flagged import, so no file-wide CALLS
    import map binds it. The Python LSP map already takes include_type_only maps, and the
    parallel dispatch builds its own LSP map for a file with such an import.
  • py_process_statement replays an import statement where it stands, into the current
    scope, with the existing py_replay_import_statement. Each statement matches the
    import entries afresh (several defs may import one name). An unmatched import binds
    UNKNOWN, which also stops a function-local import from being resolved through a
    same-named module-level binding.
  • python_receiver_is_exempt ignores flagged imports, so a function's import does not
    exempt a same-named receiver elsewhere in the file from the weak-member guard.

Graph effect besides the CALLS edges: a function-local import now gets its IMPORTS edge
from the file (the edge carries the local name as before), which #2285 also lists as
missing. Not in this PR: nested definitions and the package-map path (the other two gaps
of #2285).

Tests

  • tests/test_extraction.c python_function_local_imports_flagged_issue2285: flags at
    def, try-in-method and nested-def depth; module-level import unflagged; a def's alias
    of a module-bound name unrecorded; a module-level if import still unrecorded.
  • tests/test_pipeline.c python_function_local_import_issue2285 (+ _parallel, 50
    padding files to take the parallel path), asserting CALLS edges:
    • control, module-level import: use_top -> Store.load_top = 1 (LSP)
    • import inside the def: use_local -> Store.load_local = 1 (LSP)
    • import inside a try block of the def: use_local_try -> Store.load_local_try = 1 (LSP)
    • negative, unrelated class with the same method name: use_local -> Other.load_local = 0
    • a sibling def without the import: no_local_import -> Store.load_local = 0
    • control, module-level from other import Other: module_level -> Other.load_local = 1
    • a def that shadows it with from store import Store as Other: shadowed -> Other.load_local = 0

Results (linux arm64, gcc 13.3, base a3e9c91)

Fixture above, query_graph per method, release build (scripts/build.sh):

main  load_local:     "rows":[]
main  load_local_try: "rows":[]
fix   load_local:     "rows":[["use_local","fixture.pkg.store.Store.load_local"]]
fix   load_local_try: "rows":[["use_local_try","fixture.pkg.store.Store.load_local_try"]]
      (load_top and load_kw unchanged: one caller each, before and after)

scripts/test.sh --suites extraction,extraction_imports,extraction_inheritance,py_lsp,py_lsp_stress,pipeline
(ASan + UBSan): 1242 passed, exit 0, including the three new tests.

Reproduce-first: with the extraction and LSP changes reverted and the tests kept:

python_function_local_imports_flagged_issue2285  FAIL tests/test_extraction.c:2023: store is NULL
python_function_local_import_issue2285           FAIL tests/test_pipeline.c:21804: local_ok == 0, expected 1
python_function_local_import_issue2285_parallel  FAIL tests/test_pipeline.c:21804: local_ok == 0, expected 1

Not run: the full default scripts/test.sh leg, the TSan leg, cppcheck, clang-tidy.

Checklist

  • Every commit is signed off (git commit -s)
  • Tests pass locally (make -f Makefile.cbm test) -- see results above for what was run
  • Lint passes (make -f Makefile.cbm lint-ci) -- clang-format 20 clean on the changed
    hunks; cppcheck not run
  • New behavior is covered by a test (reproduce-first for bug fixes)

Refs #2285

Written by an AI coding agent (Claude Code) operating this account on behalf of its owner, who authorised this submission and is accountable for it.

🤖 Generated with Claude Code

…eusData#2285)

An import inside a def body was never recorded (parse_python_imports walks
the module's direct statements only) and py_process_statement had no case
for an import statement. Inside the def, `Store` fell through to the module
scope: `store = Store(); store.load_local()` stayed unresolved and the
weak-member guard refused a name guess, so the call got no CALLS edge. When
the module bound the same name, the call resolved against the module-level
class instead: a wrong edge.

- Extraction records every import inside a def body, at any depth,
  flagged CBMImport.function_scope. A local name a module-level import
  binds as well stays unrecorded: per-name consumers cannot tell the two
  bindings apart, so the def's import then binds UNKNOWN (fail closed).
- cbm_import_binds_runtime_name() is false for a flagged import: no
  file-wide CALLS import map binds it. The LSP maps already include it.
- py_process_statement replays an import statement where it stands, into
  the current scope, matching the import entries afresh per statement.
- python_receiver_is_exempt ignores flagged imports, so a def's import does
  not exempt a same-named receiver elsewhere in the file.

Tests: extraction flags (def, try in a method, nested def, shadowed alias,
module-level if); pipeline sequential + parallel: module-level control,
import in the def, import in a try block, an unrelated class with the same
method name, a sibling def without the import, and a def whose import
shadows a module-level name.

Co-Authored-By: Dammerzone <dammerzone@users.noreply.github.com>
Signed-off-by: Dryade <contact@dryade.ai>
@DryadeCore
DryadeCore requested a review from DeusData as a code owner November 13, 2026 17:53
@github-actions

Copy link
Copy Markdown

Thanks for opening this — it has been seen, and it is queued.

This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence.

Current review status: working through a backlog. 0.9.1-rc.1 is out, so the release freeze that held reviews is over — but it left a large queue of open pull requests behind it, and we are reading through them oldest-first. The background is in discussion #1144.

What that means for this PR, concretely:

  • It will not be closed for inactivity. No stale bot touches pull requests here.
  • It may still sit a while before a human reads it. That is on us, not on you.
  • Older PRs are read first, so a recent one is not being skipped — it is behind a queue.

Things that will genuinely speed it up whenever review does happen:

  • Keep it rebased on main — the tree is moving quickly right now, and a conflicting branch cannot be reviewed as the diff you intended.
  • Get CI green, or say which failures you believe are pre-existing.
  • Keep the change to one claim. Bundled features and refactors get split before they get merged, which costs you a round trip.
  • Every commit needs a sign-off (git commit -s) — CI enforces DCO.

If this fixes a bug, a reproduction we can run is worth more than a description of the symptom.

Thanks for contributing, and sorry in advance for the wait.

chidev added a commit to lev-os/codebase-memory-mcp that referenced this pull request Nov 14, 2026
…n the scope of their def

Conflict in internal/cbm/cbm.h cbm_import_binds_runtime_name, kept both:
- lev: a re-export binds no local name (cbm_import_binds_local).
- DeusData#2976: an import inside a def (function_scope) binds no module-wide
  name; the Python LSP binds it where it stands.
The rule is now binds_local && !type_checking_only && !function_scope.

Signed-off-by: chidev <chicagowebdev@gmail.com>

@DeusData DeusData left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this, and for the care that went into it: the root-cause write-up, the module-level controls, the parallel twin of the pipeline test and the reproduce-first evidence made it a pleasure to review. The core idea is right. Binding a def's import where the statement stands is exactly what #2285 needs, and we confirmed your three new tests fail on main for the right reason and pass on your branch merged with current main.

We also indexed two real projects before and after the change (django and pytest 8.3.5, sequential and parallel), and it does a lot of good: 93 existing calls move from a name guess to an exact LSP binding, 47 new LSP edges appear, 18 edges that pointed at the wrong target now point at the right one (for example reverse inside URLNode.render no longer binds ListMixin.reverse, and the postgresql backend tests now bind postgresql.base.DatabaseWrapper instead of the GIS MySQL one), and wrong edges such as selenium's Select -> django.forms.widgets.Select disappear. Graphs of non-Python projects are byte-identical.

The same runs show a few regressions in exactly the case the PR targets, and we would like these fixed before merging:

  1. Function-local imports of project code lose their name-based fallback. On django 27 correct CALLS edges disappear (for example AdminSite.get_urls -> django.urls.conf.re_path, check_url_namespaces_unique -> get_resolver, apply_limit_choices_to_to_formfield -> Q / Exists / OuterRef, OGRGeometry.geos -> GEOSGeometry), and on pytest 7 (getfuncargnames -> outcomes.fail, Parser.parse -> try_argcomplete, ...). These are calls the LSP cannot type (a decorated function, a functools.partial, a re-exported name), so on main they resolved through unique_name / suffix_match. The cause is cbm_suppress_external_import_shadow in src/pipeline/registry.c: it reads "the import map has no key for this name" as evidence that the import is external, and cbm_import_binds_runtime_name() now keeps every function-local import out of that map. Skipping those imports in the guard's loop, right after the local_name match:

     if (imp->function_scope) {
         continue; /* a def's own import never enters the file-wide import map */
     }
    

    restores all 34 edges in our runs; 18 of the 22 wrong edges your branch removes stay removed (they are caught by cbm_python_import_binding_contradicts, which judges an import by its materialized IMPORTS edge) and the other 4 go back to main's behaviour. Minimal repro:

    pkg/__init__.py   (empty)
    pkg/outcomes.py   def _with_exception(t):
                          def decorate(func):
                              func.Exception = t
                              return func
                          return decorate
    
                      @_with_exception(RuntimeError)
                      def fail(reason=""):
                          raise RuntimeError(reason)
    pkg/compat.py     def getfuncargnames(function):
                          try:
                              return function.__code__.co_varnames
                          except AttributeError:
                              from pkg.outcomes import fail
                              fail("no code object")
    

    main: getfuncargnames -> pkg.outcomes.fail (unique_name); this branch: no edge.

  2. A re-exported name imported a second time binds UNKNOWN in the def. Once the module (or an earlier def) has replayed the import, py_canonical_named_import has rewritten that entry's QN in place to the re-export target, so the def's identical statement no longer matches it and the name binds UNKNOWN. On django this turns FormsModelTestCase.test_formfield_initial -> forms.fields.CharField.__init__ (an LSP edge on main) into a suffix_match guess to db.models.fields.CharField. Repro:

    pkg/__init__.py, pkg/models/__init__.py   (empty)
    pkg/forms/fields.py    class CharField:
                               def __init__(self, max_length):
                                   self.max_length = max_length
                               def clean(self, value):
                                   return value
    pkg/models/fields.py   (the same class CharField)
    pkg/forms/__init__.py  from pkg.forms.fields import CharField
    pkg/use.py             from pkg.forms import CharField
    
                           def reimported_in_def():
                               from pkg.forms import CharField
                               field = CharField(20)
                               return field.clean(2)
    pkg/use3.py            def first_def():
                               from pkg.forms import CharField
                               return CharField(40).clean(4)
    
                           def second_def():
                               from pkg.forms import CharField
                               return CharField(50).clean(5)
    

    main: reimported_in_def -> CharField.init and -> CharField.clean (LSP). This branch: a single suffix_match edge to the class. In use3.py, first_def resolves but second_def gets nothing. Importing from the defining module (from pkg.forms.fields import CharField) works in both places, so it is the in-place rewrite. Matching each statement against an unmodified copy of the entry's import path, and canonicalizing only the bound value, should fix both shapes; please add both to the pipeline test.

  3. Two pytest edges through a dotted import inside a method disappear: pytestPDB._init_pdb and PytestPdbWrapper.do_continue do import _pytest.config and then call _pytest.config.create_terminal_writer(...). main has lsp_module_attr edges for both; this branch has none. We could not reduce it to a small fixture yet. It may be related to how the replayed unaliased dotted import binds the root _pytest. Could you take a look?

  4. Sequential and parallel indexing now disagree on base classes. build_import_map in src/pipeline/pass_semantic.c (the sequential INHERITS / DECORATES map) takes every recorded import, while the parallel resolver's map goes through cbm_import_binds_runtime_name(). On django the nested CustomDatabaseWrapper(DatabaseWrapper) classes in the postgresql backend tests inherit from postgresql.base.DatabaseWrapper in a sequential index and from the GIS MySQL wrapper in a parallel one. Please skip function_scope imports in that map too (its fast path and its IMPORTS-edge fallback), so both drivers agree. Resolving nested-class bases through a def's import can then be a follow-up for both drivers together.

Smaller notes, not blocking:

  • In the shadowed def of your test, a bare constructor call Other() still binds the module-level other.Other through the file-wide import map (main does the same through the LSP), so "fail closed" currently holds for member calls only. An extra assertion would document that.
  • The new IMPORTS edges for function-local imports are welcome: 307 of 316 on django and 66 of 72 on pytest name an in-project module, and "who imports X" then also counts lazy imports. The few wrong ones come from an existing problem that module-level imports already show on main: a bare absolute import such as import warnings binds to a same-named project module like _pytest/warnings.py, although Python 3 resolves it to the standard library. That one is not yours to fix, so we will add a commit for it on top of your branch in this PR once your round is in.
  • py_process_statement allocates a fresh array per visited import statement; one per-context scratch buffer cleared per statement would avoid that. Minor.

Thanks again. This is a valuable fix, and we are happy to help with any of the above. Once the changes are in, we will re-run the same before/after comparison.

Dryade and others added 2 commits November 14, 2026 12:49
…g and driver parity (DeusData#2285 review)

- External-import guard: cbm_suppress_external_import_shadow skips a def's
  own import. It never enters the file-wide import map, so its missing key
  is no evidence of an external binding; a call the LSP cannot type (a
  decorated function) keeps its name-based edge again.
- Re-exported names: py_lsp keeps the module path each import row was
  added with (import_source_qns) and matches every import statement
  against it. An earlier replay canonicalizes import_module_qns to the
  re-export target, so the same statement in a later def no longer
  matched and bound UNKNOWN.
- Driver parity: the sequential INHERITS / DECORATES import map
  (pass_semantic build_import_map) skips function_scope imports like the
  parallel driver's map, so a class nested in a def gets the same base in
  both drivers. Its IMPORTS-edge fallback runs only for a file with no
  extracted import, so no def's import reaches it.
- py_process_statement reuses one per-context consumed buffer per import
  statement instead of an arena allocation each time.

Tests: pipeline twins for the decorated-function fallback, both
re-export shapes and the dotted import inside a method (edge kept), a
sequential/parallel INHERITS parity test, an assertion documenting that
a bare constructor call in the shadowing def still binds the module-level
class, and a registry unit row.

Co-Authored-By: Dammerzone <dammerzone@users.noreply.github.com>
Signed-off-by: Dryade <contact@dryade.ai>
@DryadeCore

DryadeCore commented Nov 14, 2026 •

Copy link
Copy Markdown
Author

Thanks for the careful review and the before/after runs. Points 1, 2 and 4 are fixed in ab6aa4c and point 3 is diagnosed but not fixed (details below), on top of a merge of current main.

1. Name fallback lost (decorated functions). cbm_suppress_external_import_shadow now skips a def's own import. It never enters the file-wide import map, so its missing key is no evidence of an external binding, and a call the LSP cannot type keeps its name-based edge. Tests: python_def_import_name_fallback_issue2285 (+ _parallel), plus a registry unit row.

2. Re-exported names bound UNKNOWN in a later def. py_lsp now keeps the module path each import row was added with (import_source_qns) and matches every import statement against that, since a replay canonicalizes import_module_qns to the re-export target. Tests: python_def_import_reexport_issue2285 (+ _parallel), both re-export shapes.

3. import _pytest.config edges: reproduced, root cause found, not fixed here. On pytest 8.3.5 the two create_terminal_writer edges are still absent on this branch. The edges on main exist by accident: they disappear on main when debugging.py's import unittest is aliased, and come back with any project file named unittest.py. import unittest binds to the project's _pytest/unittest.py (the stdlib-vs-project issue you mention), and py_bind_dotted_prefixes then binds _pytest as a side effect. On the branch the def's own import _pytest.config binds _pytest in the def correctly, but the LSP cannot type an unaliased dotted import; module-level import p.config gets no LSP edge on main either. python_def_dotted_import_issue2285 (+ _parallel) only pins that the call keeps exactly one edge, which holds on main and on the branch. I would propose typing unaliased dotted imports as a follow-up PR; tell me if you prefer it in this one.

4. Sequential / parallel parity. build_import_map in pass_semantic.c skips function_scope imports like the parallel driver's map, so a class nested in a def gets the same base in both drivers. Test: python_def_import_nested_base_driver_parity_issue2285.

Non-blocking notes:

  • Added the assertion documenting that a bare constructor call in the shadowing def still binds the module-level class.
  • py_process_statement reuses one per-context buffer, cleared per statement, instead of allocating per import statement.
  • The wrong IMPORTS targets from bare absolute imports are the existing module-level behaviour you describe; I left that out of this PR.

Verification, locally on linux/arm64 (ASan + UBSan build): the new tests for points 1, 2 and 4 fail on the previous head and pass now; suites extraction extraction_imports extraction_inheritance py_lsp py_lsp_stress pipeline registry give 1380 passed, 0 failed. On pytest 8.3.5, getfuncargnames -> fail and Parser.parse -> try_argcomplete are back. Not run: the full default test leg, TSan, cppcheck, clang-tidy, and django (points 2 and 4 are checked there only through the fixtures).

Edit: corrected point 3 and the verification paragraph; my first version of this comment overstated both.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants