fix(python): discover interpreters to find all site-packages#165
Merged
ashishkurmi merged 2 commits intoJul 16, 2026
Merged
Conversation
…gnize The disk-based Python scanner gathered site-packages from a fixed allow-list of globs, so packages installed under any interpreter not on that list were invisible — most visibly Apple's Command Line Tools python (/Library/Developer/CommandLineTools/.../Python3.framework), whose site-packages no glob matched. After an agent upgraded from the pre-1.13 command-based scan to the 1.13 disk scan, packages there stopped being reported and showed as "No Longer Available" in the dashboard while still installed. Discover packages by on-disk structure instead of exact paths, without executing any interpreter (running python — even `python -c` — risks the macOS install-prompt / TCC behavior the disk scan exists to avoid): - Hand the existing dist-info recognizer (ScanRoots) a bounded set of Python install-tree roots (frameworks incl. CLT and Xcode, Homebrew, system, and version managers pyenv/asdf/uv/rye/conda/pipx, plus user-site) and let it recognize *.dist-info / *.egg-info metadata anywhere beneath them, so a package is found regardless of the interpreter's internal layout or version. - Roots are symlink-resolved, deduped, and nested roots dropped so no subtree is walked twice. Global discovery is independent of search_dirs. - Raise the METADATA read cap from 1 MiB to 32 MiB so a package whose METADATA embeds a large README is no longer silently dropped. - Log the scanned global install roots (count at info, full paths at debug) so field logs show exactly where the agent looked. Still TCC-safe (the walk uses the existing tcc.Skipper). SCAN_COVERAGE.md is updated to document the model and what is/isn't covered by default.
swarit-stepsecurity
force-pushed
the
swarit/fix/wt/py-interpreter-discovery
branch
from
July 16, 2026 10:29
0811fe6 to
0e1aa7e
Compare
There was a problem hiding this comment.
Pull request overview
This PR aims to prevent regressions in disk-based Python package scanning by expanding the set of on-disk roots searched for *.dist-info/METADATA / *.egg-info/PKG-INFO, and by increasing the per-metadata-file read cap to avoid dropping packages with large embedded descriptions.
Changes:
- Switch global disk scan to use a new
GlobalPythonRoots(...)that unions the prior static roots with newly discovered “install tree” roots. - Add filesystem-based discovery of Python install trees (pyenv/asdf/uv/conda/etc, plus common system/framework locations).
- Increase the Python package metadata read cap from 1 MiB to 32 MiB.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
SCAN_COVERAGE.md |
Updates Python scan documentation to describe disk-based scanning and coverage/limitations. |
internal/detector/pythonscan.go |
Uses GlobalPythonRoots for the enterprise disk-based global package scan. |
internal/detector/pythonenv.go |
Introduces install-tree globbing, root normalization/deduping, and the GlobalPythonRoots union+logging. |
internal/detector/pythonenv_test.go |
Adds unit tests for path dedupe and subsumed-root dropping helpers. |
internal/detector/pythondist.go |
Raises METADATA/PKG-INFO max read size and switches to GlobalPythonRoots. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+35
to
+55
| if home, err := os.UserHomeDir(); err == nil && home != "" { | ||
| j := func(p ...string) string { return filepath.Join(append([]string{home}, p...)...) } | ||
| pats = append(pats, | ||
| // Version managers / user installs (one tree per version/env). | ||
| j(".pyenv", "versions", "*"), | ||
| j(".asdf", "installs", "python", "*"), | ||
| j(".rye", "py", "*"), | ||
| j(".local", "lib", "python*"), | ||
| j(".local", "share", "uv", "python", "*"), | ||
| j(".local", "share", "pipx", "venvs", "*"), | ||
| // conda / mamba base + named envs. | ||
| j("miniconda3"), j("miniconda3", "envs", "*"), | ||
| j("anaconda3"), j("anaconda3", "envs", "*"), | ||
| j("miniforge3"), j("miniforge3", "envs", "*"), | ||
| j(".conda", "envs", "*"), | ||
| // User-site under ~/Library (macOS). The whole-home skipper skips | ||
| // ~/Library wholesale, so this specific tree is passed as its own | ||
| // explicit root (ShouldSkip treats an explicit walk root as opt-in). | ||
| j("Library", "Python", "*"), | ||
| ) | ||
| } |
Comment on lines
+179
to
+180
| - **TCC-protected user directories** — `~/Documents`, `~/Desktop`, `~/Downloads`, and `~/Library` are skipped to avoid macOS permission prompts. Projects kept there are missed unless `include_tcc_protected: true` is set **and** the agent has Full Disk Access (see [macos-tcc-permissions.md](docs/macos-tcc-permissions.md)). | ||
| - **Locations outside `$HOME`** — e.g. `/opt`, `/srv`, `/data`, `/Users/Shared`, or a separate repos volume. Add them via `search_dirs`. |
Comment on lines
+157
to
+158
| Python scanning detects installed packages and project virtual environments. Enable with `--enable-python-scan`. **No Python interpreter or package manager is executed** — packages are read from on-disk install metadata (`*.dist-info/METADATA` and `*.egg-info/PKG-INFO`, per PEP 376/627), so scanning never triggers a macOS install prompt. (The pre-1.13 command-based path is still available via `--legacy-python-scan` / `use_legacy_python_scan`.) | ||
|
|
Address review on PR step-security#165: - Thread the executor through Python install-root discovery and resolve the home directory via executor.ResolveHome instead of os.UserHomeDir, so a root-run launchd scan anchors the version-manager trees (pyenv/asdf/uv/ conda/pipx) on the logged-in user's home rather than /var/root, where the user's installs actually live. Filesystem access (Glob/EvalSymlinks/Stat) and GOOS also go through the executor, keeping the logic mockable. - SCAN_COVERAGE.md: clarify that the whole-home walk skips ~/Library while the global user-site ~/Library/Python/* is still scanned as its own explicit root, so global user-site packages remain covered.
ashishkurmi
approved these changes
Jul 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The disk-based Python scanner gathered site-packages from a fixed allow-list of globs (PythonGlobalRoots), so packages installed under any interpreter not on that list were never scanned. The clearest case is Apple's Command Line Tools python
(/Library/Developer/CommandLineTools/.../Python3.framework), whose site-packages no glob matched. After the agent upgraded from the pre-1.13 command-based scan (which ran whatever pip/conda/uv was on PATH) to the 1.13 disk scan, packages there stopped being reported and showed as "No Longer Available" in the dashboard while still installed.
Discover interpreters directly instead: enumerate python executables on PATH, in version-manager trees (pyenv/asdf/uv/conda), and at common locations, then ask each one via a small
python -c(site.getsitepackagesAlso raise the METADATA read cap from 1 MiB to 32 MiB so an installed package whose METADATA embeds a large README is no longer silently dropped from the scan.
What does this PR do?
Type of change
Testing
./stepsecurity-dev-machine-guard --verbose./stepsecurity-dev-machine-guard --json | python3 -m json.toolmake lintmake testRelated Issues