Repository navigation
Package inventory: NuGet, Maven, Go, crates.io, RubyGems, Pub, Hex, SwiftURL, CRAN; more npm and PyPI sources (#535–#543) - #544
Conversation
…x, SwiftURL and CRAN packages; more npm and PyPI sources Package inventory gains the ecosystems the server accepts since Compliance-OS#1047, and more sources for npm and PyPI: - #535 NuGet: obj/project.assets.json, packages.lock.json, packages.config; .NET global tools; the NuGet cache (scope cache). - #536 Maven: gradle.lockfile, pom.xml direct dependencies; ~/.m2 (and a settings.xml localRepository) and the Gradle module cache. - #537 Go: go.mod + go.sum; go install binaries via debug/buildinfo (with stdlib); the module download cache. - #538 crates.io: Cargo.lock, cargo install, the registry cache. - #539 RubyGems: Gemfile.lock, installed gems from gemspec file names. - #540 Pub, Hex, SwiftURL, CRAN: pubspec.lock, pub global and cache; mix.lock; Package.resolved v1-v3; renv.lock and R libraries. - #541 npm: bun.lock, Bun and pnpm global installs, asdf and mise. - #542 PyPI: pipx, uv, conda, Microsoft Store Python, asdf and mise; pdm.lock and pylock.toml. - #543 virtualenv packages for Python projects without a lockfile. The agent sends an ecosystem only when agent_get_package_settings lists it in `ecosystems`; without the list it keeps to npm, PyPI and Packagist, so an older server never rejects a snapshot. Roots of ecosystems the server doesn't accept aren't read. Directory listings now count against a per-run budget, since the caches are read by listing version folders. Closes #535, closes #536, closes #537, closes #538, closes #539, closes #540, closes #541, closes #542, closes #543. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
There was a problem hiding this comment.
Devin Review found 9 potential issues.
1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
- A venv is skipped only when every requirements*.txt in the folder is fully pinned, and a file that includes another (-r) doesn't count as pinned. - A venv's site-packages is read only when no folder on the way is a link. - mix.lock: the repository is read from the end of the entry, so a private dependency no longer hides a public package. - bun.lock: comments are dropped before trailing commas, so a comment between a trailing comma and "}" no longer breaks the file. - Gemspec names with digit-led segments (iso-639-0.3.8) split at the first segment that looks like a release. - The Cargo cache reads only crates.io's own git index folder, not every GitHub-hosted registry. - Fuzz target for the R DESCRIPTION parser. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|



Package inventory for every ecosystem the server accepts since BrainStation-23/Compliance-OS#1047, plus more npm and PyPI sources. Tracker: BrainStation-23/Compliance-OS#1048.
Closes #535, closes #536, closes #537, closes #538, closes #539, closes #540, closes #541, closes #542, closes #543.
Contract
Settings.Ecosystems(agent_get_package_settings→ecosystems). The agent sends only what is listed; a missing list means npm, PyPI and Packagist, so an older server never sees a row it would reject (a rejected row stops a full snapshot from retiring packages). Dropping unlisted rows doesn't make the snapshot incomplete. Roots of unaccepted ecosystems aren't read at all.cachefor download caches. Ecosystem names and normalisation as indocs/agent-ingestion.md("Packages (#798)"); Go's standard library goes asstdlibwithout thegoprefix.Per issue
project)global)cache)obj/project.assets.json,packages.lock.json,packages.config.dotnet/tools/.store.nuget/packagesbin,packagesbeside*.csproj/*.sln; probeobjonly forproject.assets.jsongradle.lockfile,buildscript-gradle.lockfile,pom.xml(direct; literal or same-file${prop}).m2/repository(+settings.xmllocalRepository), Gradlefiles-2.1target/buildbesidepom.xml/build.gradle*go.mod(replace applied, local dropped) +go.sum(downloaded, de-duplicated)go/binviadebug/buildinfo(never executed), withstdlibgo/pkg/mod/cache/download(case-unescaped)Cargo.lock(crates.io sources only).cargo/.crates2.json/.crates.toml.cargo/registry/cache/<crates.io index>targetbesideCargo.tomlGemfile.lock/gems.locked(GEMspecs, platform suffix dropped)specifications/(+default/): user, rbenv, rvm, asdf, mise, system, Homebrew, RubyInstallerpubspec.lock(pub.dev hosted),mix.lock(public:hex),Package.resolvedv1–v3,renv.lock(CRAN / Posit mirror)DESCRIPTION,Repository: CRAN)buildbesidepubspec.yaml,deps/_buildbesidemix.exs,PodsbesidePodfile,renv/librarybun.lock(JSONC);bun.lockbcounted and loggednode_modules), asdf, misepdm.lock,pylock.toml/pylock.*.toml%APPDATA%\uv\dataand the older%APPDATA%\uv), conda base and envs, Microsoft Store Python, asdf, mise.venv/venv/envwith apyvenv.cfg, only when the folder has no Python lockfile (or only an unpinnedrequirements*.txt); location is the venvNotes for review
maxDirsPerRun, 500k), since caches are read by listing version folders. A run over budget is incomplete and goes as a delta, as before. Resuming a cache walk across runs (as the project scan does) isn't implemented: the caches are listed folder by folder without reading files, which fits well inside the 2-minute global run on any cache we've seen. If the lab pass shows otherwise, that's a follow-up.node_modules.[[package]]tables of Cargo.lock, pdm.lock and pylock.toml go through a small shared reader (toml.go) that skips multi-line arrays, so array elements are never taken for keys. No YAML, TOML or Elixir library.requirements*.txtnow also drops per-line options on the same line (name==1.0 --hash=…), which were previously skipped.Tests
ecosystems_test.go,lockfile_fuzz_test.go); each fuzz target ran locally. One crasher found by fuzzinggo.modis fixed and kept intestdata/fuzz.collect_ecosystems_test.go: every new global/cache root against a temp tree, including a real Go binary (the test binary), a pnpm symlinked layout with a link that escapes, and the legacy-server gate.walk_ecosystems_test.go: the three Report virtualenv packages for Python projects without a lockfile #543 cases, go.mod + go.sum,objprobing, the beside-skips,renv/library.syncer_test.go: nothing outside the server's list is sent, and the snapshot stays full.make test quality-check format-check verify-crosspass locally.Not done here: the real-device checks in each issue's acceptance (packages on the Packages tab with a toolchain installed, an OSV match on a known vulnerable version).
🤖 Generated with Claude Code