Title: feat(extract): share process/thread pools for AST, resolution, and cache by albertbu · Pull Request #3289 · Graphify-Labs/graphify

albertbu · 2026-09-02T12:24:01Z

sice review: albertbu/graphify@albertb/detect-git-enum...albertb/extract-pools

Stacked on albertb/detect-git-enum (PR1). Review the compare link above, or the single commit 1c879f0 — the GitHub diff vs v8 includes PR1.

Summary

Route AST extract, JS/Python symbol resolution, and cache probes through graphify.parallel so worker counts follow GRAPHIFY_MAX_WORKERS / --max-workers (Windows still capped at 61).
Cache probes use a thread pool; resolution uses a process pool and falls back to sequential on small jobs or BrokenProcessPool.
Stat-index load/flush is lock-guarded for concurrent probes, and _flush_stat_index still refuses to mkdir a cache root that vanished mid-run (atexit stat-index flush recreates the analyzed directory if it was deleted mid-run (worktree race leaves husk dirs) #2974).

Test plan

pytest tests/test_parallel.py tests/test_extract.py tests/test_stat_index_husk.py -q
graphify extract <code-corpus> --code-only matches a sequential run (GRAPHIFY_MAX_WORKERS=1) on node/edge identity

Use git ls-files, Google repo manifests, and nested worktrees so ignored trees are pruned at directory granularity instead of a Python walk. Co-authored-by: Cursor <cursoragent@cursor.com>

Gating the out-of-root check on islink() admitted regular files reached by following a symlink directory, including graphify-out/memory/ which skips ignore pruning. Restore the v8 resolve()-under-root admit guard and prune those dirs during the memory walk too. Co-authored-by: Cursor <cursoragent@cursor.com>

Route CPU-heavy extract and JS/Python fact collection through graphify.parallel so worker counts follow GRAPHIFY_MAX_WORKERS. Co-authored-by: Cursor <cursoragent@cursor.com>

graphify-labs

Graphify reviewed this change.

Worth a look — the grounded gate found no coupling regressions or blocking issues, but 5 advisory finding(s) below merit a look before merge.

Formal verification. 1 change(s) tested, no difference found (not proven).

Graphify review — findings

Adds gitignore-aware tree enumeration to detect(): it walks large trees via git ls-files, Google repo project lists, and nested worktrees/submodules so gitignored directories are skipped. Parallelizes AST extraction, JS/Python symbol resolution, and cache probes over shared graphify.parallel process/thread pools tunable with GRAPHIFY_MAX_WORKERS. Guards the shared stat-index cache behind a re-entrant lock and publishes _stat_index_root last so concurrent probes can't race into a half-loaded index or produce a torn write on flush.

Worth a look

_stat_index_dirty written without global declaration inside lock block — graphify/cache.py · Escalate · medium
- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Double-close in flush error path can close an unrelated reused fd — graphify/cache.py:397 · Escalate · medium
- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Initial cache lookup still reads shared index without the lock — graphify/cache.py:584 · Escalate · medium
- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Git fast path includes deleted tracked files — graphify/detect.py:1246 · Escalate · medium
- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
detect() skips nested git checkouts when scan root is itself a git repo — graphify/detect.py:1549 · Escalate · medium
- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 3422 functions depend on the 1669 functions this change touches.

Health — this change adds coupling hotspots:

new: extract() — 527 callers, 43 callees
new: _rebuild_code() — 113 callers, 50 callees
new: detect() — 119 callers, 17 callees
new: save_semantic_cache() — 58 callers, 9 callees
new: _extract_generic() — 18 callers, 25 callees
new: save_manifest() — 40 callers, 11 callees
new: extract_files_direct() — 17 callers, 20 callees
new: extract_js() — 85 callers, 4 callees
…and 74 more — each is listed as a finding

Verification — 3422 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

verification_scope: 3217 function(s) in the blast radius were not formally verified this run

Formal verification

Could not verify: Could not verify cached\_word\_count.

The verifier did not have enough to check cached\_word\_count, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `path` is annotated `Path` — outside the synthesizable primitive/collection set

Could not verify: Could not verify \_ensure\_stat\_index.

The verifier did not have enough to check \_ensure\_stat\_index, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set

Could not verify: Could not verify file\_hash.

The verifier did not have enough to check file\_hash, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `path` is annotated `Path` — outside the synthesizable primitive/collection set

No difference found (not proven): No behavior difference found in \_flush\_stat\_index (not a proof).

The verifier ran both versions of \_flush\_stat\_index on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.

Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.

Note: An input the sampler did not try could still differ.

Could not verify: Could not verify detect.

The verifier did not have enough to check detect, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set

Could not verify: Could not verify detect\_incremental.

The verifier did not have enough to check detect\_incremental, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set

Could not verify: Could not verify \_git\_tracked\_path\_keys.

The verifier did not have enough to check \_git\_tracked\_path\_keys, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set

Could not verify: Could not verify \_is\_noise\_dir.

The verifier did not have enough to check \_is\_noise\_dir, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `parent` is annotated `'Path | None'` — outside the synthesizable primitive/collection set

Could not verify: Could not verify extract.

The verifier did not have enough to check extract, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `cache_root` is annotated `Path | None` — outside the synthesizable primitive/collection set

Could not verify: Could not verify \_extract\_parallel.

The verifier did not have enough to check \_extract\_parallel, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set

Could not verify: Could not verify \_extract\_sequential.

The verifier did not have enough to check \_extract\_sequential, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set

Could not verify: Could not verify \_augment\_symbol\_resolution\_edges.

The verifier did not have enough to check \_augment\_symbol\_resolution\_edges, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set

Could not verify: Could not verify \_collect\_js\_symbol\_resolution\_facts.

The verifier did not have enough to check \_collect\_js\_symbol\_resolution\_facts, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `facts` is annotated `_SymbolResolutionFacts` — outside the synthesizable primitive/collection set

Could not verify: Could not verify \_collect\_python\_symbol\_resolution\_facts.

The verifier did not have enough to check \_collect\_python\_symbol\_resolution\_facts, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set

· 11 grounded finding(s) anchored inline below; 71 more finding(s) on lines outside this diff (see the check run).

graphify-labs · 2026-09-02T12:42:15Z

+    return name[dot:].lower() in _CORPUS_EXT_LOWER
+
+
 def _is_noise_dir(part: str, parent: "Path | None" = None) -> bool:


⚠️ Health regression — _is_noise_dir()

10 callers depend on it (afferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

graphify-labs · 2026-09-02T12:42:15Z

+    return _path_is_under(worktree, root) or _path_is_under(root, worktree)
+
+
+def _git_enumerate_named_worktrees(


⚠️ Health regression — _git_enumerate_named_worktrees()

fans out to 7 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

graphify-labs · 2026-09-02T12:42:15Z

+    )
+
+
+def _git_enumerate_files(


⚠️ Health regression — _git_enumerate_files()

fans out to 8 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

graphify-labs · 2026-09-02T12:42:15Z



-def detect(root: Path, *, follow_symlinks: bool | None = None, google_workspace: bool | None = None, extra_excludes: list[str] | None = None, cache_root: Path | None = None, gitignore: bool = True) -> dict:
+def detect(root: Path, *, follow_symlinks: bool | None = None, google_workspace: bool | None = None, extra_excludes: list[str] | None = None, cache_root: Path | None = None, gitignore: bool = True, code_only: bool = False) -> dict:


⚠️ Health regression — detect()

fans out to 17 callees (efferent coupling); 119 callers depend on it (afferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

graphify-labs · 2026-09-02T12:42:15Z

+    vprint(f"scanning {root} ...")
+    heartbeat = _Heartbeat("scanning")
+
+    def _walk_from(


⚠️ Health regression — _walk_from()

fans out to 8 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

graphify-labs · 2026-09-02T12:42:15Z

+    return per_file, uncached_work
+
+
 def _extract_parallel(


⚠️ Health regression — _extract_parallel()

high coupling complexity (Ca·Ce = 20).

Grounded coupling-delta finding (deterministic), not an LLM guess.

graphify-labs · 2026-09-02T12:42:15Z

+_PARALLEL_THRESHOLD = PARALLEL_THRESHOLD


 def extract(


⚠️ Health regression — extract()

fans out to 43 callees (efferent coupling); 527 callers depend on it (afferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

graphify-labs · 2026-09-02T12:42:15Z

-    ]
-    if not js_paths:
-        return
+def _collect_js_facts_for_path(path: Path) -> _SymbolResolutionFacts:


⚠️ Health regression — _collect_js_facts_for_path()

fans out to 26 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

graphify-labs · 2026-09-02T12:42:15Z

        return _read_text(function_node, source)
    return None

+def _collect_python_facts_for_path(path: Path, root: Path) -> _SymbolResolutionFacts:


⚠️ Health regression — _collect_python_facts_for_path()

fans out to 11 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

graphify-labs · 2026-09-02T12:42:15Z

+    return [fn(item) for item in batch]
+
+
+def map_in_process_pool(


⚠️ Health regression — map_in_process_pool()

8 callers depend on it (afferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

albertbu and others added 3 commits September 2, 2026 14:48

feat(detect): enumerate large trees via git, repo, and submodules

6805542

Use git ls-files, Google repo manifests, and nested worktrees so ignored trees are pruned at directory granularity instead of a Python walk. Co-authored-by: Cursor <cursoragent@cursor.com>

feat(extract): share process/thread pools for AST, resolution, and cache

66cfc96

Route CPU-heavy extract and JS/Python fact collection through graphify.parallel so worker counts follow GRAPHIFY_MAX_WORKERS. Co-authored-by: Cursor <cursoragent@cursor.com>

albertbu force-pushed the albertb/extract-pools branch from 1c879f0 to 66cfc96 Compare September 2, 2026 12:32

graphify-labs Bot reviewed Sep 2, 2026

View reviewed changes

Navigation Menu

Uh oh!

Uh oh!

Title: feat(extract): share process/thread pools for AST, resolution, and cache - #3289

Title: feat(extract): share process/thread pools for AST, resolution, and cache#3289
albertbu wants to merge 3 commits into
Graphify-Labs:v8from
albertbu:albertb/extract-pools

albertbu commented Sep 2, 2026

Uh oh!

graphify-labs Bot left a comment •

edited

Loading

Uh oh!

graphify-labs Bot Sep 2, 2026

Uh oh!

graphify-labs Bot Sep 2, 2026

Uh oh!

graphify-labs Bot Sep 2, 2026

Uh oh!

graphify-labs Bot Sep 2, 2026

Uh oh!

graphify-labs Bot Sep 2, 2026

Uh oh!

graphify-labs Bot Sep 2, 2026

Uh oh!

graphify-labs Bot Sep 2, 2026

Uh oh!

graphify-labs Bot Sep 2, 2026

Uh oh!

graphify-labs Bot Sep 2, 2026

Uh oh!

graphify-labs Bot Sep 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		return name[dot:].lower() in _CORPUS_EXT_LOWER


		def _is_noise_dir(part: str, parent: "Path \| None" = None) -> bool:

		return _path_is_under(worktree, root) or _path_is_under(root, worktree)


		def _git_enumerate_named_worktrees(



		def detect(root: Path, *, follow_symlinks: bool \| None = None, google_workspace: bool \| None = None, extra_excludes: list[str] \| None = None, cache_root: Path \| None = None, gitignore: bool = True) -> dict:
		def detect(root: Path, *, follow_symlinks: bool \| None = None, google_workspace: bool \| None = None, extra_excludes: list[str] \| None = None, cache_root: Path \| None = None, gitignore: bool = True, code_only: bool = False) -> dict:

Uh oh!

Uh oh!

Conversation

albertbu commented Sep 2, 2026

Summary

Test plan

Uh oh!

graphify-labs Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Graphify review — findings

Impact & health

Graphify review

Gate & verification

graphify gate

Formal verification

Uh oh!

graphify-labs Bot Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

graphify-labs Bot Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

graphify-labs Bot Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

graphify-labs Bot Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

graphify-labs Bot Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

graphify-labs Bot Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

graphify-labs Bot Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

graphify-labs Bot Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

graphify-labs Bot Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

graphify-labs Bot Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

graphify-labs Bot left a comment •

edited

Loading