Skip to content

Package inventory: NuGet, Maven, Go, crates.io, RubyGems, Pub, Hex, SwiftURL, CRAN; more npm and PyPI sources (#535–#543) - #544

Merged
neamul602 merged 2 commits into
devfrom
feat/535-543-package-ecosystems
Nov 19, 2026
Merged

neamul602 merged 2 commits into
devfrom
feat/535-543-package-ecosystems

Conversation

@neamul602

@neamul602 neamul602 commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Package inventory for every ecosystem the server accepts since BrainStation-23/Compliance-OS#1047, plus more npm and PyPI sources. Tracker: BrainStation-23/Compliance-OS#1048.

Closes #535, closes #536, closes #537, closes #538, closes #539, closes #540, closes #541, closes #542, closes #543.

Contract

  • Settings.Ecosystems (agent_get_package_settings → ecosystems). The agent sends only what is listed; a missing list means npm, PyPI and Packagist, so an older server never sees a row it would reject (a rejected row stops a full snapshot from retiring packages). Dropping unlisted rows doesn't make the snapshot incomplete. Roots of unaccepted ecosystems aren't read at all.
  • New scope cache for download caches. Ecosystem names and normalisation as in docs/agent-ingestion.md ("Packages (#798)"); Go's standard library goes as stdlib without the go prefix.

Per issue

Issue Project (project) Installed (global) Cache (cache) Walk
#535 NuGet obj/project.assets.json, packages.lock.json, packages.config .dotnet/tools/.store .nuget/packages skip bin, packages beside *.csproj/*.sln; probe obj only for project.assets.json
#536 Maven gradle.lockfile, buildscript-gradle.lockfile, pom.xml (direct; literal or same-file ${prop}) – .m2/repository (+ settings.xml localRepository), Gradle files-2.1 skip target / build beside pom.xml / build.gradle*
#537 Go go.mod (replace applied, local dropped) + go.sum (downloaded, de-duplicated) go/bin via debug/buildinfo (never executed), with stdlib go/pkg/mod/cache/download (case-unescaped) –
#538 crates.io Cargo.lock (crates.io sources only) .cargo/.crates2.json / .crates.toml .cargo/registry/cache/<crates.io index> skip target beside Cargo.toml
#539 RubyGems Gemfile.lock / gems.locked (GEM specs, platform suffix dropped) gemspec file names in specifications/ (+ default/): user, rbenv, rvm, asdf, mise, system, Homebrew, RubyInstaller – –
#540 Pub / Hex / SwiftURL / CRAN pubspec.lock (pub.dev hosted), mix.lock (public :hex), Package.resolved v1–v3, renv.lock (CRAN / Posit mirror) pub global packages; R libraries (DESCRIPTION, Repository: CRAN) Pub hosted cache skip build beside pubspec.yaml, deps/_build beside mix.exs, Pods beside Podfile, renv/library
#541 npm bun.lock (JSONC); bun.lockb counted and logged Bun global, pnpm global (links followed only inside its own node_modules), asdf, mise – –
#542 PyPI pdm.lock, pylock.toml / pylock.*.toml pipx, uv tools and Pythons (%APPDATA%\uv\data and the older %APPDATA%\uv), conda base and envs, Microsoft Store Python, asdf, mise – –
#543 venvs .venv / venv / env with a pyvenv.cfg, only when the folder has no Python lockfile (or only an unpinned requirements*.txt); location is the venv – – nothing below the venv

Notes for review

  • All readers only read files: no child process, no network. Every file read is capped; directory listings now count against a per-run budget (maxDirsPerRun, 500k), since caches are read by listing version folders. A run over budget is incomplete and goes as a delta, as before. Resuming a cache walk across runs (as the project scan does) isn't implemented: the caches are listed folder by folder without reading files, which fits well inside the 2-minute global run on any cache we've seen. If the lab pass shows otherwise, that's a follow-up.
  • Links and junctions are still never followed, except pnpm's global links, and only when they resolve inside that node_modules.
  • The [[package]] tables of Cargo.lock, pdm.lock and pylock.toml go through a small shared reader (toml.go) that skips multi-line arrays, so array elements are never taken for keys. No YAML, TOML or Elixir library.
  • requirements*.txt now also drops per-line options on the same line (name==1.0 --hash=…), which were previously skipped.

Tests

  • Fixtures, table tests and a fuzz target for every new parser (ecosystems_test.go, lockfile_fuzz_test.go); each fuzz target ran locally. One crasher found by fuzzing go.mod is fixed and kept in testdata/fuzz.
  • collect_ecosystems_test.go: every new global/cache root against a temp tree, including a real Go binary (the test binary), a pnpm symlinked layout with a link that escapes, and the legacy-server gate.
  • walk_ecosystems_test.go: the three Report virtualenv packages for Python projects without a lockfile #543 cases, go.mod + go.sum, obj probing, the beside-skips, renv/library.
  • syncer_test.go: nothing outside the server's list is sent, and the snapshot stays full.
  • make test quality-check format-check verify-cross pass locally.

Not done here: the real-device checks in each issue's acceptance (packages on the Packages tab with a toolchain installed, an OSV match on a known vulnerable version).

🤖 Generated with Claude Code


Devin Review

…x, SwiftURL and CRAN packages; more npm and PyPI sources

Package inventory gains the ecosystems the server accepts since
Compliance-OS#1047, and more sources for npm and PyPI:

- #535 NuGet: obj/project.assets.json, packages.lock.json, packages.config;
  .NET global tools; the NuGet cache (scope cache).
- #536 Maven: gradle.lockfile, pom.xml direct dependencies; ~/.m2 (and a
  settings.xml localRepository) and the Gradle module cache.
- #537 Go: go.mod + go.sum; go install binaries via debug/buildinfo
  (with stdlib); the module download cache.
- #538 crates.io: Cargo.lock, cargo install, the registry cache.
- #539 RubyGems: Gemfile.lock, installed gems from gemspec file names.
- #540 Pub, Hex, SwiftURL, CRAN: pubspec.lock, pub global and cache;
  mix.lock; Package.resolved v1-v3; renv.lock and R libraries.
- #541 npm: bun.lock, Bun and pnpm global installs, asdf and mise.
- #542 PyPI: pipx, uv, conda, Microsoft Store Python, asdf and mise;
  pdm.lock and pylock.toml.
- #543 virtualenv packages for Python projects without a lockfile.

The agent sends an ecosystem only when agent_get_package_settings lists
it in `ecosystems`; without the list it keeps to npm, PyPI and Packagist,
so an older server never rejects a snapshot. Roots of ecosystems the
server doesn't accept aren't read. Directory listings now count against
a per-run budget, since the caches are read by listing version folders.

Closes #535, closes #536, closes #537, closes #538, closes #539,
closes #540, closes #541, closes #542, closes #543.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@neamul602 neamul602 self-assigned this Nov 19, 2026
@coderabbitai

coderabbitai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI (base), Organization UI (inherited)
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 43a8dc59-68b5-4101-a54f-cfb0aa13b952

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 9 potential issues.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

Devin Review

Comment thread internal/service/packages/walk.go Resolved
Comment thread internal/service/packages/walk.go Resolved
Comment thread internal/service/packages/pub.go Resolved
Comment thread internal/service/packages/rubygems.go Resolved
Comment thread internal/service/packages/cran.go Resolved
Comment thread internal/service/packages/walk.go Resolved
Comment thread internal/service/packages/cargo.go Resolved
- A venv is skipped only when every requirements*.txt in the folder is
  fully pinned, and a file that includes another (-r) doesn't count as
  pinned.
- A venv's site-packages is read only when no folder on the way is a link.
- mix.lock: the repository is read from the end of the entry, so a
  private dependency no longer hides a public package.
- bun.lock: comments are dropped before trailing commas, so a comment
  between a trailing comma and "}" no longer breaks the file.
- Gemspec names with digit-led segments (iso-639-0.3.8) split at the
  first segment that looks like a release.
- The Cargo cache reads only crates.io's own git index folder, not every
  GitHub-hosted registry.
- Fuzz target for the R DESCRIPTION parser.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

@neamul602
neamul602 merged commit 6852072 into dev Nov 19, 2026
24 checks passed
@neamul602
neamul602 deleted the feat/535-543-package-ecosystems branch November 19, 2026 06:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant