STAC-25540: fetch private-index wheels, then drop the credential - #364
Conversation
The vsphere suite is the only one that needs a credential: it pins
vsphere-automation-sdk==1.82.0, an unmodified upstream VMware wheel that
VMware withdrew from public PyPI, so it resolves only from our GitLab
Package Registry. Public PyPI serves a 0.0.1 placeholder squatting the
name, which installs cleanly and fails much later.
setup_artifact_registry.sh left the registry password in a 0600 ~/.netrc
for the remainder of the job, so the suite's tests, its tox environment
and its whole dependency tree could read it. Test code is PR-authored,
which made the credential effectively readable by whoever opened the pull
request (STAC-25463 review, P1).
That was contained by putting the job behind the private-package-index
GitHub Environment, releasing the credential only after a human approved
each run. It worked, but SHARED_PATHS covers the CI files, so it fired on
roughly one commit in six and blocked authors on their own pull requests.
Replace it with a narrower fix. fetch_private_wheels.sh writes the netrc,
downloads one hardcoded package set into a wheelhouse under RUNNER_TEMP,
deletes the netrc, and rewrites ~/.pip/pip.conf to resolve that wheelhouse
via find-links. Everything afterwards runs with no credential on disk and
no authenticated index configured, and the environment gate is gone.
Details that matter:
* The package set is hardcoded, not read from the tree: while the
credential is on disk, a pull request must not be able to redirect pip.
* --only-binary=:all:, because downloading an sdist executes its
setup.py and would hand upstream code a readable ~/.netrc.
* A system interpreter, never the toolchain virtualenv, whose pip is
built by repository code. The path is resolved rather than hardcoded
-- bci/python:3.13 ships /usr/bin/python3.13 and no /usr/bin/python3,
and `python3` on PATH is a /usr/local/bin shim -- and the result is
then rejected if it resolves inside GITHUB_WORKSPACE, which is what
makes "system" a guarantee rather than an assumption.
* An EXIT trap revokes the credential on the failure paths too, and the
script then asserts the netrc is gone and the wheel actually arrived.
Rejected: a separate credentialed job publishing the wheels with
upload-artifact. Workflow artifacts on a public repository are anonymously
downloadable, so that would have amounted to publicly redistributing a
vendor artifact VMware withdrew, under no SPDX licence.
Residual risk, recorded in the workflow: a pull request that edits the
workflow itself can still reach the secret, since pull_request runs the
PR's own copy. That holds for every secret in every repository and is
contained by review, not CI. What changed is that ordinary test code and
its dependencies no longer see it.
setup_artifact_registry.sh is untouched; .gitlab-ci.yml still uses it.
Verified: shellcheck and actionlint clean, Zizmor clean (no findings).
Verified inside the pinned BCI image that the interpreter resolves to
/usr/bin/python3.13 and that a workspace-resident interpreter is refused.
Verified that a missing argument, missing credentials, a workspace
interpreter and a failed download all exit non-zero with no netrc left
behind and the password in no file or log; and that find-links in
~/.pip/pip.conf is honoured with no command-line flags, which is what tox
relies on via passenv HOME.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
bf43e63 to
e6d160f
Compare
Verified end-to-end in CIRun 31009062454 is green, including and then, in the suite itself, with no credential present: One finding worth recording. The wheelhouse came to 23 wheels, and the private closure is larger than the single pinned package — Also note the first attempt failed, which is worth keeping visible: That failure also exercised the error path for real: the |
cd1e292
into
STAC-25463-integrations-github-ci
What
Removes the
private-package-indexGitHub Environment gate and replaces it with a narrower fix for the same finding: the credential is now written, used for one fixed download, and destroyed before any repository code runs.Closes STAC-25540. Stacked on #362 — review and merge that first; this targets its branch and will retarget automatically.
Why
vsphereis the only suite needing a credential. It pinsvsphere-automation-sdk==1.82.0, an unmodified upstream VMware wheel that VMware withdrew from public PyPI. We self-host it in the GitLab Package Registry only because that org was private. Public PyPI serves a0.0.1placeholder squatting the name, which installs cleanly and fails much later.setup_artifact_registry.shleft the password in a0600 ~/.netrcfor the rest of the job, so the suite's tests, its tox environment and its entire dependency tree could read it — and that code is PR-authored. That was @LouisLotter's P1 in #362.The environment gate answered it by requiring a human to release each run. It worked, but
SHARED_PATHScovers the CI files, so it fired on roughly one commit in six (28 of 176 in the last year) and blocked authors on their own PRs.prevent_self_reviewhas since been disabled as a stopgap; this removes the need for the gate at all.How
.setup-scripts/fetch_private_wheels.shwrites the netrc, downloads a fixed package set into a wheelhouse underRUNNER_TEMP, deletes the netrc, and rewrites~/.pip/pip.confto resolve that wheelhouse throughfind-links. Everything after it runs with no credential on disk and no authenticated index configured.Four details are load-bearing:
--only-binary=:all:— downloading an sdist executes itssetup.py, which would hand upstream code a process with a readable~/.netrc.EXITtrap revokes the credential on failure paths too; the script then asserts the netrc is gone and the wheel actually arrived, rather than silently falling through to the PyPI placeholder.Rejected alternative
A separate credentialed job publishing the wheels via
upload-artifact. Workflow artifacts on a public repository are anonymously downloadable, so that would have amounted to publicly redistributing a vendor artifact VMware withdrew, under no SPDX licence. Re-hosting it publicly is a licensing question, not an engineering one.Residual risk, stated plainly
A PR that edits this workflow can still reach the secret, because
pull_requestruns the PR's own copy of the workflow. That is true of every secret in every repository and is contained by review and branch protection, not by CI. What changed is that ordinary test code and its dependency tree no longer see it, and that needed no workflow edit at all.Fork PRs still skip
vsphere— they get no Actions secrets on a public repo, so they could not fetch the wheel regardless. The fork guard stays.Validation
shellcheckclean on the new script andselect-checks.sh;actionlintclean; Zizmor clean, no findings.find-linksin~/.pip/pip.confis honoured with no command-line flags, and that the same install fails without it — this is the mechanismvsphere/tox.inidepends on viapassenv HOME.setup_artifact_registry.shis untouched —.gitlab-ci.ymlstill references it in three places.After merge
The
private-package-indexenvironment can be deleted from repo settings; nothing will reference it.