getdeps: accept fedora in install-system-deps --distro and fill rpm gaps by michel-slm · Pull Request #488 · facebook/CacheLib

michel-slm · 2026-09-15T15:25:58Z

Fedora is already detected by getdeps_platform.py and mapped to the rpm package manager, but install-system-deps --distro only accepted ubuntu and centos_stream, so the Fedora package list could not be produced from another host.

Also let the remaining cachelib deps that Fedora packages be satisfied from the system with --allow-system-packages instead of being built from source. libaio and numa already had rpm entries scoped to CentOS Stream (and Alma/Rocky for libaio); those selectors now include Fedora. The other two gain new [rpms.distro=fedora] sections. Package names and versions were checked against the Fedora 44 repositories:

manifest	Fedora package	Fedora	manifest
benchmark	google-benchmark-devel	1.9.5	1.8.0
libaio	libaio-devel	0.3.111	0.3.113
liboqs	liboqs-devel	0.12.0	0.12.0
numa	numactl-devel	2.0.19	2.0.19

magic_enum is deliberately not mapped: Fedora ships 0.9.5, which installs <magic_enum.hpp> at the include root, while cachelib includes the <magic_enum/magic_enum.hpp> layout introduced in 0.9.6.

Tested on Fedora 44 (linux-fedora-44 host type): all changed manifests pass validate-manifest, install-system-deps --recursive --dry-run --distro fedora cachelib lists the four packages above, and a full build --allow-system-packages cachelib succeeds with fizz linking the system liboqs and cachebench linking the system libaio and libnuma.

meta-codesync · 2026-09-15T15:29:16Z

@michel-slm has imported this pull request. If you are a Meta employee, you can view this in D120159819.

Fedora is already detected by getdeps_platform.py and mapped to the rpm package manager, but `install-system-deps --distro` only accepted ubuntu and centos_stream, so the Fedora package list could not be produced from another host. Also let the remaining cachelib deps that Fedora packages be satisfied from the system with --allow-system-packages instead of being built from source. libaio and numa already had rpm entries scoped to CentOS Stream (and Alma/Rocky for libaio); those selectors now include Fedora. The other two gain new [rpms.distro=fedora] sections. Package names and versions were checked against the Fedora 44 repositories: | manifest | Fedora package | Fedora | manifest | |-----------|------------------------|---------|----------| | benchmark | google-benchmark-devel | 1.9.5 | 1.8.0 | | libaio | libaio-devel | 0.3.111 | 0.3.113 | | liboqs | liboqs-devel | 0.12.0 | 0.12.0 | | numa | numactl-devel | 2.0.19 | 2.0.19 | magic_enum is deliberately not mapped: Fedora ships 0.9.5, which installs <magic_enum.hpp> at the include root, while cachelib includes the <magic_enum/magic_enum.hpp> layout introduced in 0.9.6. Tested on Fedora 44 (linux-fedora-44 host type): all changed manifests pass validate-manifest, `install-system-deps --recursive --dry-run --distro fedora cachelib` lists the four packages above, and a full `build --allow-system-packages cachelib` succeeds with fizz linking the system liboqs and cachebench linking the system libaio and libnuma. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Michel Lind <salimma@fedoraproject.org>

facebook-github-tools · 2026-09-15T17:07:56Z

@michel-slm has updated the pull request. You must reimport the pull request before landing.

`getdeps.py vendor --output-dir DIR project` fetches every third-party dependency of the project and copies its source tree to DIR/<name>, skipping deps satisfied by system packages and the project itself. It also writes DIR/getdeps-vendor.txt listing each vendored project with its fetcher hash (the pinned git revision or the archive sha256). This is the getdeps analogue of `cargo vendor` / `go mod vendor`: distro packagers can ship the result as a vendor tarball next to the project source, as Fedora's vendored-dependency guidelines require, and a follow-up change will let `build` consume such a directory offline. Extracted trees rather than the original archives and clones are vendored so the result is self-contained, reviewable and license-scannable; .git directories are dropped and symlinks are followed for the same reason (subproject fetchers symlink into the scratch dir, which does not exist on an offline builder). Tested on Fedora 44: `getdeps.py --allow-system-packages vendor --no-tests --output-dir /var/tmp/cachelib-vendor cachelib` produced eleven trees (about 330 MB, fbthrift being 216 MB of that) plus getdeps-vendor.txt, with no .git directories or symlinks left behind. Once the Fedora rpm mappings from facebook#488 land that drops to the seven projects Fedora does not package. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Michel Lind <salimma@fedoraproject.org>

`getdeps.py --vendor-dir DIR build project` takes every third-party dependency from DIR/<project>, as populated by `getdeps.py vendor`, instead of cloning or downloading it. The lookup sits in the manifest's fetcher selection after the system-package check, so a dependency that --allow-system-packages resolves to an installed package still wins, and anything not found in DIR raises an error naming the missing project rather than falling back to the network. That single check is the offline guarantee: no GitFetcher or ArchiveFetcher is ever constructed. The vendored tree is wrapped in the existing LocalDirFetcher, the same mechanism --src-dir uses, so no fetcher code changes. Its hash is fixed and it always reports the sources as changed, which means repeated builds against a vendor dir reconfigure their dependencies each time; that is acceptable for the one-shot distro builds this is meant for. --free-up-disk only removed build trees whose sources live under the scratch dir, a guard meant to protect a user's own --src-dir checkout. Vendored sources are ours to clean up after, so the guard now also accepts the vendor dir (stored realpath'd so the prefix comparison matches LocalDirFetcher's realpath'd source paths). Known limitation: patchfiles are applied with `git apply` from the enclosing git top-level, so a vendor dir placed inside another git checkout would mis-apply patches for the (few) manifests that carry them. A distro build directory is not a git checkout, so this does not affect the intended use; making the patch step independent of the surrounding repository is left for a follow-up. Tested on Fedora 44 (aarch64, 4 cores) on a tree that also carried facebook#488, facebook#489 and facebook#490, against the seven-project vendor dir produced by the previous commit (magic_enum, sparsemap, folly, fizz, wangle, mvfst, fbthrift; 276 MB): unshare -rn python3 build/fbcode_builder/getdeps.py \ --allow-system-packages --vendor-dir /var/tmp/cachelib-vendor \ --scratch-path /var/tmp/getdeps-offline-scratch --num-jobs 2 \ --extra-cmake-defines '{"CMAKE_POLICY_VERSION_MINIMUM":"3.5"}' \ build --free-up-disk --no-tests --src-dir=. cachelib `unshare -rn` puts the build in its own network namespace. Inspected from outside while it ran, the build process was in net:[4026532485] versus the shell's net:[4026531833]; `nsenter -n ip -brief link` inside it showed only `lo` DOWN, `getent hosts github.com` failed (exit 2), and `ss -tunap` listed no sockets. The build finished in 72 minutes with exit 0, all eight projects installed, no "Download with" or "Cloning" line in the log, and cachebench linking the system glog, liboqs, libaio and libnuma. Requesting a project that is neither vendored nor allowed from system packages fails with the new error, and `show-source-dir --recursive` resolves every vendored project into the vendor dir. The --free-up-disk fix was verified separately: an offline sparsemap build with the flag left no build tree behind, where the 72-minute run (made before the fix) had left 11 GB. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Michel Lind <salimma@fedoraproject.org>

glog >= 0.7 requires consumers to define GLOG_USE_GLOG_EXPORT, which its exported glog::glog target carries as an INTERFACE_COMPILE_DEFINITION. Our FindGlog module constructs its own imported glog::glog target from find_library/find_path results and never sets that definition, so on a distro shipping glog 0.7 (Fedora 44 has 0.7.1) every cachelib translation unit fails with: /usr/include/glog/logging.h:1228:30: error: expected primary-expression before 'public' folly/io/Cursor.h:834:5: error: 'LogMessageFatal' is not a member of 'google' Try find_package(glog CONFIG) first and only fall back to the manual search when no config file is installed. getdeps' from-source glog also ships a config file, so the fallback now only matters for glog builds installed without CMake support. The module's documented outputs are still honoured on the config path: GLOG_INCLUDE_DIR(S) is taken from the target's interface include directories and GLOG_LIBRARIES is set to glog::glog itself, so a consumer that links through the variable rather than the target still inherits the GLOG_USE_GLOG_EXPORT definition. In-tree callers all use the target directly. Tested on Fedora 44 in a single `getdeps.py --allow-system-packages build --no-tests cachelib` run together with the Fedora rpm mappings from facebook#488 and the manifest change that honours --no-tests for cachelib (facebook#490); the build succeeds against glog-devel 0.7.1 with glog_DIR=/usr/lib64/cmake/glog in the CMake cache. A probe project using only ${GLOG_LIBRARIES}/${GLOG_INCLUDE_DIRS} compiles and links against glog 0.7.1 on the config path, and the fallback path (with CMAKE_DISABLE_FIND_PACKAGE_glog=TRUE) still yields GLOG_LIBRARIES=/usr/lib64/libglog.so and GLOG_INCLUDE_DIRS=/usr/include. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Michel Lind <salimma@fedoraproject.org>

Summary: cachelib's CMakeLists defaults BUILD_TESTS to ON and the manifest never overrode it, so `getdeps.py build --no-tests cachelib` still compiled every test binary. Wire the getdeps test context to BUILD_TESTS the same way the folly and fizz manifests do. With tests off the cachelib build on Fedora 44 drops from 497 to 110 ninja steps, which matters for memory- and CPU-constrained distro builders. Tested on Fedora 44 in a single `getdeps.py --allow-system-packages build --no-tests cachelib` run together with the Fedora rpm mappings from facebook/CacheLib#488 and the FindGlog config-mode fix; the resulting CMake cache has BUILD_TESTS:BOOL=OFF and the build succeeds. X-link: facebook/CacheLib#490 Reviewed By: likewhatevs Differential Revision: D120177373 Pulled By: michel-slm fbshipit-source-id: 8b079e9878b945f778c882f84dbb4433660b181d

Summary: cachelib's CMakeLists defaults BUILD_TESTS to ON and the manifest never overrode it, so `getdeps.py build --no-tests cachelib` still compiled every test binary. Wire the getdeps test context to BUILD_TESTS the same way the folly and fizz manifests do. With tests off the cachelib build on Fedora 44 drops from 497 to 110 ninja steps, which matters for memory- and CPU-constrained distro builders. Tested on Fedora 44 in a single `getdeps.py --allow-system-packages build --no-tests cachelib` run together with the Fedora rpm mappings from #488 and the FindGlog config-mode fix; the resulting CMake cache has BUILD_TESTS:BOOL=OFF and the build succeeds. Pull Request resolved: #490 Test Plan: ## Verification of both directions of the `test` context Environment: Fedora 44, getdeps with `--allow-system-packages`, on a tree that also carried #488, #489 and the vendoring branch; deps already built, so only cachelib's own configure step ran each time. ### `--no-tests` passed → `BUILD_TESTS=OFF` From the full offline build earlier today (`build --free-up-disk --no-tests --src-dir=. cachelib`, exit 0): ``` $ grep BUILD_TESTS build/cachelib/CMakeCache.txt BUILD_TESTS:BOOL=OFF ``` cachelib compiled in 221 ninja steps and no `*-test-*` executables were produced. ### `--no-tests` omitted → `BUILD_TESTS=ON` Same scratch dir, same options minus `--no-tests`, with a no-op CMake target so that only the configure step runs: ``` python3 build/fbcode_builder/getdeps.py --allow-system-packages ... \ build --no-deps --cmake-target help --src-dir=. cachelib ``` getdeps passed the define from the new `[cmake.defines.test=on]` section on the cmake command line: ``` $ grep -oE 'DBUILD_TESTS=[A-Z]+' build.log | sort | uniq -c 1 DBUILD_TESTS=ON ``` and the cache flipped accordingly: ``` $ grep BUILD_TESTS build/cachelib/CMakeCache.txt BUILD_TESTS:BOOL=ON ``` After that configure, ninja knows about 212 `*-test-*` executable targets, and a dry run lists 276 remaining steps (the test binaries) on top of the already-built non-test objects, matching the 497-step total seen before this change. ### CI implication The GitHub Actions build step invokes `getdeps.py build ... cachelib` without `--no-tests`, so it lands in the `test=on` branch and keeps building the tests that the subsequent `getdeps.py test` step runs (783 passing on today's main). Only the `--no-tests` path changes behaviour, from silently building everything to actually skipping the tests. Reviewed By: likewhatevs Differential Revision: D120177373 Pulled By: michel-slm fbshipit-source-id: 8b079e9878b945f778c882f84dbb4433660b181d

Summary: cachelib's CMakeLists defaults BUILD_TESTS to ON and the manifest never overrode it, so `getdeps.py build --no-tests cachelib` still compiled every test binary. Wire the getdeps test context to BUILD_TESTS the same way the folly and fizz manifests do. With tests off the cachelib build on Fedora 44 drops from 497 to 110 ninja steps, which matters for memory- and CPU-constrained distro builders. Tested on Fedora 44 in a single `getdeps.py --allow-system-packages build --no-tests cachelib` run together with the Fedora rpm mappings from facebook/CacheLib#488 and the FindGlog config-mode fix; the resulting CMake cache has BUILD_TESTS:BOOL=OFF and the build succeeds. X-link: facebook/CacheLib#490 Reviewed By: likewhatevs Differential Revision: D120177373 Pulled By: michel-slm fbshipit-source-id: 8b079e9878b945f778c882f84dbb4433660b181d

Summary: Fedora is already detected by getdeps_platform.py and mapped to the rpm package manager, but `install-system-deps --distro` only accepted ubuntu and centos_stream, so the Fedora package list could not be produced from another host. Also let the remaining cachelib deps that Fedora packages be satisfied from the system with --allow-system-packages instead of being built from source. libaio and numa already had rpm entries scoped to CentOS Stream (and Alma/Rocky for libaio); those selectors now include Fedora. The other two gain new [rpms.distro=fedora] sections. Package names and versions were checked against the Fedora 44 repositories: | manifest | Fedora package | Fedora | manifest | |-----------|------------------------|---------|----------| | benchmark | google-benchmark-devel | 1.9.5 | 1.8.0 | | libaio | libaio-devel | 0.3.111 | 0.3.113 | | liboqs | liboqs-devel | 0.12.0 | 0.12.0 | | numa | numactl-devel | 2.0.19 | 2.0.19 | magic_enum is deliberately not mapped: Fedora ships 0.9.5, which installs `<magic_enum.hpp>` at the include root, while cachelib includes the `<magic_enum/magic_enum.hpp>` layout introduced in 0.9.6. Tested on Fedora 44 (linux-fedora-44 host type): all changed manifests pass validate-manifest, `install-system-deps --recursive --dry-run --distro fedora cachelib` lists the four packages above, and a full `build --allow-system-packages cachelib` succeeds with fizz linking the system liboqs and cachebench linking the system libaio and libnuma. X-link: facebook/CacheLib#488 Reviewed By: likewhatevs Differential Revision: D120159819 Pulled By: michel-slm fbshipit-source-id: 8fcee0d3f90447209f259df7ba8a0a41aae4a18c

Summary: cachelib's CMakeLists defaults BUILD_TESTS to ON and the manifest never overrode it, so `getdeps.py build --no-tests cachelib` still compiled every test binary. Wire the getdeps test context to BUILD_TESTS the same way the folly and fizz manifests do. With tests off the cachelib build on Fedora 44 drops from 497 to 110 ninja steps, which matters for memory- and CPU-constrained distro builders. Tested on Fedora 44 in a single `getdeps.py --allow-system-packages build --no-tests cachelib` run together with the Fedora rpm mappings from facebook/CacheLib#488 and the FindGlog config-mode fix; the resulting CMake cache has BUILD_TESTS:BOOL=OFF and the build succeeds. X-link: facebook/CacheLib#490 Reviewed By: likewhatevs Differential Revision: D120177373 Pulled By: michel-slm fbshipit-source-id: 8b079e9878b945f778c882f84dbb4433660b181d

Summary: Fedora is already detected by getdeps_platform.py and mapped to the rpm package manager, but `install-system-deps --distro` only accepted ubuntu and centos_stream, so the Fedora package list could not be produced from another host. Also let the remaining cachelib deps that Fedora packages be satisfied from the system with --allow-system-packages instead of being built from source. libaio and numa already had rpm entries scoped to CentOS Stream (and Alma/Rocky for libaio); those selectors now include Fedora. The other two gain new [rpms.distro=fedora] sections. Package names and versions were checked against the Fedora 44 repositories: | manifest | Fedora package | Fedora | manifest | |-----------|------------------------|---------|----------| | benchmark | google-benchmark-devel | 1.9.5 | 1.8.0 | | libaio | libaio-devel | 0.3.111 | 0.3.113 | | liboqs | liboqs-devel | 0.12.0 | 0.12.0 | | numa | numactl-devel | 2.0.19 | 2.0.19 | magic_enum is deliberately not mapped: Fedora ships 0.9.5, which installs `<magic_enum.hpp>` at the include root, while cachelib includes the `<magic_enum/magic_enum.hpp>` layout introduced in 0.9.6. Tested on Fedora 44 (linux-fedora-44 host type): all changed manifests pass validate-manifest, `install-system-deps --recursive --dry-run --distro fedora cachelib` lists the four packages above, and a full `build --allow-system-packages cachelib` succeeds with fizz linking the system liboqs and cachebench linking the system libaio and libnuma. X-link: facebook/CacheLib#488 Reviewed By: likewhatevs Differential Revision: D120159819 Pulled By: michel-slm fbshipit-source-id: 8fcee0d3f90447209f259df7ba8a0a41aae4a18c

Summary: Three commits, best read in order. Together they give getdeps the equivalent of `cargo vendor` plus `--offline`, which is what Fedora's vendored-dependency guidelines need for packaging cachelib without maintaining separate folly/fizz/wangle/mvfst/fbthrift packages in lockstep. ## getdeps: add a vendor subcommand `getdeps.py vendor --output-dir DIR project` fetches every third-party dependency of the project and copies its source tree to `DIR/<name>`, skipping deps satisfied by system packages and the project itself. It also writes DIR/getdeps-vendor.txt listing each vendored project with its fetcher hash (the pinned git revision or the archive sha256). This is the getdeps analogue of `cargo vendor` / `go mod vendor`: distro packagers can ship the result as a vendor tarball next to the project source, as Fedora's vendored-dependency guidelines require, and a follow-up change will let `build` consume such a directory offline. Extracted trees rather than the original archives and clones are vendored so the result is self-contained, reviewable and license-scannable; .git directories are dropped and symlinks are followed for the same reason (subproject fetchers symlink into the scratch dir, which does not exist on an offline builder). Tested on Fedora 44: `getdeps.py --allow-system-packages vendor --no-tests --output-dir /var/tmp/cachelib-vendor cachelib` produced eleven trees (about 330 MB, fbthrift being 216 MB of that) plus getdeps-vendor.txt, with no .git directories or symlinks left behind. Once the Fedora rpm mappings from facebook/CacheLib#488 land that drops to the seven projects Fedora does not package. ## getdeps: add --vendor-dir to build from vendored sources offline `getdeps.py --vendor-dir DIR build project` takes every third-party dependency from `DIR/<project>`, as populated by `getdeps.py vendor`, instead of cloning or downloading it. The lookup sits in the manifest's fetcher selection after the system-package check, so a dependency that --allow-system-packages resolves to an installed package still wins, and anything not found in DIR raises an error naming the missing project rather than falling back to the network. That single check is the offline guarantee: no GitFetcher or ArchiveFetcher is ever constructed. The vendored tree is wrapped in the existing LocalDirFetcher, the same mechanism --src-dir uses, so no fetcher code changes. Its hash is fixed and it always reports the sources as changed, which means repeated builds against a vendor dir reconfigure their dependencies each time; that is acceptable for the one-shot distro builds this is meant for. --free-up-disk only removed build trees whose sources live under the scratch dir, a guard meant to protect a user's own --src-dir checkout. Vendored sources are ours to clean up after, so the guard now also accepts the vendor dir (stored realpath'd so the prefix comparison matches LocalDirFetcher's realpath'd source paths). Known limitation: patchfiles are applied with `git apply` from the enclosing git top-level, so a vendor dir placed inside another git checkout would mis-apply patches for the (few) manifests that carry them. A distro build directory is not a git checkout, so this does not affect the intended use; making the patch step independent of the surrounding repository is left for a follow-up. Tested on Fedora 44 (aarch64, 4 cores) on a tree that also carried facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490, against the seven-project vendor dir produced by the previous commit (magic_enum, sparsemap, folly, fizz, wangle, mvfst, fbthrift; 276 MB): ``` unshare -rn python3 build/fbcode_builder/getdeps.py \ --allow-system-packages --vendor-dir /var/tmp/cachelib-vendor \ --scratch-path /var/tmp/getdeps-offline-scratch --num-jobs 2 \ --extra-cmake-defines '{"CMAKE_POLICY_VERSION_MINIMUM":"3.5"}' \ build --free-up-disk --no-tests --src-dir=. cachelib ``` `unshare -rn` puts the build in its own network namespace. Inspected from outside while it ran, the build process was in net:[4026532485] versus the shell's net:[4026531833]; `nsenter -n ip -brief link` inside it showed only `lo` DOWN, `getent hosts github.com` failed (exit 2), and `ss -tunap` listed no sockets. The build finished in 72 minutes with exit 0, all eight projects installed, no "Download with" or "Cloning" line in the log, and cachebench linking the system glog, liboqs, libaio and libnuma. Requesting a project that is neither vendored nor allowed from system packages fails with the new error, and `show-source-dir --recursive` resolves every vendored project into the vendor dir. The --free-up-disk fix was verified separately: an offline sparsemap build with the flag left no build tree behind, where the 72-minute run (made before the fix) had left 11 GB. ## getdeps: add tests for vendor and --vendor-dir `getdeps/test/vendor_test.py`, in the style of the existing builder tests (MagicMock loader and build options, real `ManifestParser` objects), 5 tests: `vendor` copies a non-system dependency's tree, skips a dependency that resolves to a `SystemPackageFetcher`, skips the project itself, drops `.git`, materialises symlinks and writes `getdeps-vendor.txt`; `vendor` replaces a stale tree in the output dir; `--vendor-dir` resolves a download-URL manifest to a `LocalDirFetcher` on `<vendor-dir>/<name>`; a project missing from the vendor dir fails naming the project rather than constructing a network fetcher; and without `--vendor-dir` the normal `ArchiveFetcher` is still chosen. Run from `build/fbcode_builder` with `python3 -m unittest getdeps.test.vendor_test`. Verified together with facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490 applied; each of those stands alone and none of them is required for this change to apply. 🤖 Generated with [Claude Code](https://claude.com/claude-code) X-link: facebook/CacheLib#491 Reviewed By: aleivag, likewhatevs Differential Revision: D120204081 Pulled By: michel-slm fbshipit-source-id: f877310e93bcddbf6ff6f81f72e2738e23ecdfa5

Summary: Three commits, best read in order. Together they give getdeps the equivalent of `cargo vendor` plus `--offline`, which is what Fedora's vendored-dependency guidelines need for packaging cachelib without maintaining separate folly/fizz/wangle/mvfst/fbthrift packages in lockstep. ## getdeps: add a vendor subcommand `getdeps.py vendor --output-dir DIR project` fetches every third-party dependency of the project and copies its source tree to `DIR/<name>`, skipping deps satisfied by system packages and the project itself. It also writes DIR/getdeps-vendor.txt listing each vendored project with its fetcher hash (the pinned git revision or the archive sha256). This is the getdeps analogue of `cargo vendor` / `go mod vendor`: distro packagers can ship the result as a vendor tarball next to the project source, as Fedora's vendored-dependency guidelines require, and a follow-up change will let `build` consume such a directory offline. Extracted trees rather than the original archives and clones are vendored so the result is self-contained, reviewable and license-scannable; .git directories are dropped and symlinks are followed for the same reason (subproject fetchers symlink into the scratch dir, which does not exist on an offline builder). Tested on Fedora 44: `getdeps.py --allow-system-packages vendor --no-tests --output-dir /var/tmp/cachelib-vendor cachelib` produced eleven trees (about 330 MB, fbthrift being 216 MB of that) plus getdeps-vendor.txt, with no .git directories or symlinks left behind. Once the Fedora rpm mappings from #488 land that drops to the seven projects Fedora does not package. ## getdeps: add --vendor-dir to build from vendored sources offline `getdeps.py --vendor-dir DIR build project` takes every third-party dependency from `DIR/<project>`, as populated by `getdeps.py vendor`, instead of cloning or downloading it. The lookup sits in the manifest's fetcher selection after the system-package check, so a dependency that --allow-system-packages resolves to an installed package still wins, and anything not found in DIR raises an error naming the missing project rather than falling back to the network. That single check is the offline guarantee: no GitFetcher or ArchiveFetcher is ever constructed. The vendored tree is wrapped in the existing LocalDirFetcher, the same mechanism --src-dir uses, so no fetcher code changes. Its hash is fixed and it always reports the sources as changed, which means repeated builds against a vendor dir reconfigure their dependencies each time; that is acceptable for the one-shot distro builds this is meant for. --free-up-disk only removed build trees whose sources live under the scratch dir, a guard meant to protect a user's own --src-dir checkout. Vendored sources are ours to clean up after, so the guard now also accepts the vendor dir (stored realpath'd so the prefix comparison matches LocalDirFetcher's realpath'd source paths). Known limitation: patchfiles are applied with `git apply` from the enclosing git top-level, so a vendor dir placed inside another git checkout would mis-apply patches for the (few) manifests that carry them. A distro build directory is not a git checkout, so this does not affect the intended use; making the patch step independent of the surrounding repository is left for a follow-up. Tested on Fedora 44 (aarch64, 4 cores) on a tree that also carried #488, #489 and #490, against the seven-project vendor dir produced by the previous commit (magic_enum, sparsemap, folly, fizz, wangle, mvfst, fbthrift; 276 MB): ``` unshare -rn python3 build/fbcode_builder/getdeps.py \ --allow-system-packages --vendor-dir /var/tmp/cachelib-vendor \ --scratch-path /var/tmp/getdeps-offline-scratch --num-jobs 2 \ --extra-cmake-defines '{"CMAKE_POLICY_VERSION_MINIMUM":"3.5"}' \ build --free-up-disk --no-tests --src-dir=. cachelib ``` `unshare -rn` puts the build in its own network namespace. Inspected from outside while it ran, the build process was in net:[4026532485] versus the shell's net:[4026531833]; `nsenter -n ip -brief link` inside it showed only `lo` DOWN, `getent hosts github.com` failed (exit 2), and `ss -tunap` listed no sockets. The build finished in 72 minutes with exit 0, all eight projects installed, no "Download with" or "Cloning" line in the log, and cachebench linking the system glog, liboqs, libaio and libnuma. Requesting a project that is neither vendored nor allowed from system packages fails with the new error, and `show-source-dir --recursive` resolves every vendored project into the vendor dir. The --free-up-disk fix was verified separately: an offline sparsemap build with the flag left no build tree behind, where the 72-minute run (made before the fix) had left 11 GB. ## getdeps: add tests for vendor and --vendor-dir `getdeps/test/vendor_test.py`, in the style of the existing builder tests (MagicMock loader and build options, real `ManifestParser` objects), 5 tests: `vendor` copies a non-system dependency's tree, skips a dependency that resolves to a `SystemPackageFetcher`, skips the project itself, drops `.git`, materialises symlinks and writes `getdeps-vendor.txt`; `vendor` replaces a stale tree in the output dir; `--vendor-dir` resolves a download-URL manifest to a `LocalDirFetcher` on `<vendor-dir>/<name>`; a project missing from the vendor dir fails naming the project rather than constructing a network fetcher; and without `--vendor-dir` the normal `ArchiveFetcher` is still chosen. Run from `build/fbcode_builder` with `python3 -m unittest getdeps.test.vendor_test`. Verified together with #488, #489 and #490 applied; each of those stands alone and none of them is required for this change to apply. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Pull Request resolved: #491 Test Plan: Imported from GitHub, without a `Test Plan:` line. Evidence: vendor_dir realpath storage (addresses "Missing verification of realpath storage for vendor_dir") - Added VendorDirRealpathTest to getdeps/test/vendor_test.py: test_vendor_dir_stored_as_realpath verifies BuildOptions stores a symlinked vendor_dir input as os.path.realpath (with a sanity assert the aliased input genuinely differs, so the test would fail if realpath were not applied), plus test_vendor_dir_none_stays_none. - Command: buck test fbcode//opensource/fbcode_builder/getdeps/test:test -> Pass 105, Fail 0. - Test run: https://www.internalfb.com/intern/testinfra/testrun/17732923734731577 - OSS layout: python3 -m unittest getdeps.test.vendor_test -> Ran 9 tests, OK. Reviewed By: aleivag, likewhatevs Differential Revision: D120204081 Pulled By: michel-slm fbshipit-source-id: f877310e93bcddbf6ff6f81f72e2738e23ecdfa5

Summary: Three commits, best read in order. Together they give getdeps the equivalent of `cargo vendor` plus `--offline`, which is what Fedora's vendored-dependency guidelines need for packaging cachelib without maintaining separate folly/fizz/wangle/mvfst/fbthrift packages in lockstep. ## getdeps: add a vendor subcommand `getdeps.py vendor --output-dir DIR project` fetches every third-party dependency of the project and copies its source tree to `DIR/<name>`, skipping deps satisfied by system packages and the project itself. It also writes DIR/getdeps-vendor.txt listing each vendored project with its fetcher hash (the pinned git revision or the archive sha256). This is the getdeps analogue of `cargo vendor` / `go mod vendor`: distro packagers can ship the result as a vendor tarball next to the project source, as Fedora's vendored-dependency guidelines require, and a follow-up change will let `build` consume such a directory offline. Extracted trees rather than the original archives and clones are vendored so the result is self-contained, reviewable and license-scannable; .git directories are dropped and symlinks are followed for the same reason (subproject fetchers symlink into the scratch dir, which does not exist on an offline builder). Tested on Fedora 44: `getdeps.py --allow-system-packages vendor --no-tests --output-dir /var/tmp/cachelib-vendor cachelib` produced eleven trees (about 330 MB, fbthrift being 216 MB of that) plus getdeps-vendor.txt, with no .git directories or symlinks left behind. Once the Fedora rpm mappings from facebook/CacheLib#488 land that drops to the seven projects Fedora does not package. ## getdeps: add --vendor-dir to build from vendored sources offline `getdeps.py --vendor-dir DIR build project` takes every third-party dependency from `DIR/<project>`, as populated by `getdeps.py vendor`, instead of cloning or downloading it. The lookup sits in the manifest's fetcher selection after the system-package check, so a dependency that --allow-system-packages resolves to an installed package still wins, and anything not found in DIR raises an error naming the missing project rather than falling back to the network. That single check is the offline guarantee: no GitFetcher or ArchiveFetcher is ever constructed. The vendored tree is wrapped in the existing LocalDirFetcher, the same mechanism --src-dir uses, so no fetcher code changes. Its hash is fixed and it always reports the sources as changed, which means repeated builds against a vendor dir reconfigure their dependencies each time; that is acceptable for the one-shot distro builds this is meant for. --free-up-disk only removed build trees whose sources live under the scratch dir, a guard meant to protect a user's own --src-dir checkout. Vendored sources are ours to clean up after, so the guard now also accepts the vendor dir (stored realpath'd so the prefix comparison matches LocalDirFetcher's realpath'd source paths). Known limitation: patchfiles are applied with `git apply` from the enclosing git top-level, so a vendor dir placed inside another git checkout would mis-apply patches for the (few) manifests that carry them. A distro build directory is not a git checkout, so this does not affect the intended use; making the patch step independent of the surrounding repository is left for a follow-up. Tested on Fedora 44 (aarch64, 4 cores) on a tree that also carried facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490, against the seven-project vendor dir produced by the previous commit (magic_enum, sparsemap, folly, fizz, wangle, mvfst, fbthrift; 276 MB): ``` unshare -rn python3 build/fbcode_builder/getdeps.py \ --allow-system-packages --vendor-dir /var/tmp/cachelib-vendor \ --scratch-path /var/tmp/getdeps-offline-scratch --num-jobs 2 \ --extra-cmake-defines '{"CMAKE_POLICY_VERSION_MINIMUM":"3.5"}' \ build --free-up-disk --no-tests --src-dir=. cachelib ``` `unshare -rn` puts the build in its own network namespace. Inspected from outside while it ran, the build process was in net:[4026532485] versus the shell's net:[4026531833]; `nsenter -n ip -brief link` inside it showed only `lo` DOWN, `getent hosts github.com` failed (exit 2), and `ss -tunap` listed no sockets. The build finished in 72 minutes with exit 0, all eight projects installed, no "Download with" or "Cloning" line in the log, and cachebench linking the system glog, liboqs, libaio and libnuma. Requesting a project that is neither vendored nor allowed from system packages fails with the new error, and `show-source-dir --recursive` resolves every vendored project into the vendor dir. The --free-up-disk fix was verified separately: an offline sparsemap build with the flag left no build tree behind, where the 72-minute run (made before the fix) had left 11 GB. ## getdeps: add tests for vendor and --vendor-dir `getdeps/test/vendor_test.py`, in the style of the existing builder tests (MagicMock loader and build options, real `ManifestParser` objects), 5 tests: `vendor` copies a non-system dependency's tree, skips a dependency that resolves to a `SystemPackageFetcher`, skips the project itself, drops `.git`, materialises symlinks and writes `getdeps-vendor.txt`; `vendor` replaces a stale tree in the output dir; `--vendor-dir` resolves a download-URL manifest to a `LocalDirFetcher` on `<vendor-dir>/<name>`; a project missing from the vendor dir fails naming the project rather than constructing a network fetcher; and without `--vendor-dir` the normal `ArchiveFetcher` is still chosen. Run from `build/fbcode_builder` with `python3 -m unittest getdeps.test.vendor_test`. Verified together with facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490 applied; each of those stands alone and none of them is required for this change to apply. 🤖 Generated with [Claude Code](https://claude.com/claude-code) X-link: facebook/CacheLib#491 Reviewed By: aleivag, likewhatevs Differential Revision: D120204081 Pulled By: michel-slm fbshipit-source-id: f877310e93bcddbf6ff6f81f72e2738e23ecdfa5

meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Sep 15, 2026

michel-slm force-pushed the getdeps-fedora-distro branch from 90f623a to 6d81391 Compare September 15, 2026 15:27

michel-slm force-pushed the getdeps-fedora-distro branch from 6d81391 to fb84175 Compare September 15, 2026 17:07

This was referenced Sep 15, 2026

cmake: prefer glog's own CMake config over the FindGlog module #489

Closed

getdeps: honour --no-tests for cachelib #490

Closed

getdeps: vendor subcommand and --vendor-dir for offline builds #491

Closed

Navigation Menu

Uh oh!

getdeps: accept fedora in install-system-deps --distro and fill rpm gaps - #488

getdeps: accept fedora in install-system-deps --distro and fill rpm gaps#488
michel-slm wants to merge 1 commit into
facebook:mainfrom
michel-slm:getdeps-fedora-distro

michel-slm commented Sep 15, 2026 •

edited

Loading

Uh oh!

meta-codesync Bot commented Sep 15, 2026

Uh oh!

facebook-github-tools Bot commented Sep 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

michel-slm commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

meta-codesync Bot commented Sep 15, 2026

Uh oh!

facebook-github-tools Bot commented Sep 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

michel-slm commented Sep 15, 2026 •

edited

Loading