From 666cef9debd1a444683350fd97ee854a9b894ecc Mon Sep 17 00:00:00 2001 From: Louis Parkin Date: Mon, 3 Aug 2026 09:47:32 +0200 Subject: [PATCH 01/11] STAC-25463 Add GitHub Actions check-test workflow (integrations CI migration phase 1) Ports the pure-Python half of .gitlab-ci.yml to GitHub Actions as the first phase of moving stackstate-agent-integrations off GitLab (STAC-25142). What moved: * The `test_` job family -> a single `check-tests` matrix. * The per-job `changes:` rules -> .github/scripts/select-checks.sh, which reproduces them with `git diff` against the merge base. Done in plain git because StackVista enforces a strict third-party action allowlist. * The `checksdev validate *` commands that rode inside test_stackstate_checks_base -> their own job, so a metadata failure reads as its own PR check. Scoped out, with reasons recorded in the workflow header: * The five Docker-dependent suites (splunk x4 + stackstate_checks_dev) are phase 2; both public ARC runners ship a DinD sidecar, so this is about exercising that path, not provisioning it. * test_postgres is dead config - it tests a `postgres` check that does not exist in this repository. * print_env is not ported at any phase: a bare `printenv` on a PUBLIC repo publishes every CI credential in scope to a world-readable log. * There is no Windows lane to port. `.gitlab-ci.yml` defines a `.windows_env` anchor that no job has ever referenced, and Windows is not a supported target. * Publishing stays on GitLab until integrations gets its own publishing role (pulumi-infra section 7.4), so the GitLab pipeline remains live. Phase 1 needs no new secret provisioning: REGISTRY_HOST/USER/PASSWORD are org-level visibility=all (STAC-25350) and so already reach this PUBLIC repo. The image reference composes vars.REGISTRY_HOST + /docker rather than using vars.REGISTRY_DOCKER_URL, which is private-visibility and unreadable here. Validated: actionlint clean (with a new .github/actionlint.yaml declaring the self-hosted labels), zizmor clean, shellcheck clean, and the selector exercised against real git history for the push, single-check, multi-check, shared-library and no-match cases. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/actionlint.yaml | 9 ++ .github/scripts/select-checks.sh | 128 +++++++++++++++++ .github/workflows/checks-tests.yml | 214 +++++++++++++++++++++++++++++ 3 files changed, 351 insertions(+) create mode 100644 .github/actionlint.yaml create mode 100755 .github/scripts/select-checks.sh create mode 100644 .github/workflows/checks-tests.yml diff --git a/.github/actionlint.yaml b/.github/actionlint.yaml new file mode 100644 index 00000000..898078e3 --- /dev/null +++ b/.github/actionlint.yaml @@ -0,0 +1,9 @@ +# Self-hosted runner labels available to this repository, so actionlint does not +# report them as unknown. These are ARC scale sets in the `public` runner group +# (StackVista/argocd-apps, cluster_definitions/tooling-main/apps/github-runner-*); +# the `-public` tier is the one a PUBLIC repository is allowed to schedule on. +self-hosted-runner: + labels: + - docker-public + - xlarge-public + - arm64-xlarge-public diff --git a/.github/scripts/select-checks.sh b/.github/scripts/select-checks.sh new file mode 100755 index 00000000..e0683e8f --- /dev/null +++ b/.github/scripts/select-checks.sh @@ -0,0 +1,128 @@ +#!/usr/bin/env bash +# +# Selects which integration check suites the test matrix should run, reproducing +# the `changes:` rules that gated each `test_` job in .gitlab-ci.yml +# (GitLab -> GitHub migration, STAC-25463). +# +# GitLab evaluated a per-job `changes:` list; GitHub has no job-level path filter, +# so the equivalent is computed once here and fanned out as a matrix. This is done +# in plain git rather than a path-filter action: StackVista enforces a strict +# third-party action allowlist, and `git diff` against the merge base is exactly +# what the GitLab rule meant. +# +# Selection rules, ported from .gitlab-ci.yml: +# * A change to a shared library, the setup scripts, or this CI wiring runs +# EVERY suite (GitLab: the `base_changes` anchor). +# * Otherwise only the suites whose own directory changed run. +# * GitLab's `splunk_base_build_rule` -- a change to splunk_base also runs the +# other three splunk suites, which import its test helpers -- is not ported +# here because no splunk suite runs yet. It lands with them in phase 2. +# * push / workflow_dispatch run everything (GitLab: `master_branch`, +# `release_branch`). +# +# Writes `checks=` to $GITHUB_OUTPUT for `fromJson()` in a matrix. + +set -euo pipefail + +# Suites currently running on GitHub Actions. Phase 1 is the 15 pure-Python +# suites: they need no Docker daemon, so they are the low-risk half of the port. +# +# Deliberately NOT here yet (phase 2, needs the DinD story exercised first): +# splunk_base, splunk_health, splunk_metric, splunk_topology +# -- each drives a real Splunk container via docker-compose. +# stackstate_checks_dev +# -- its tests exercise the toolkit's own Docker helpers. +# Both public ARC runners provide a DinD sidecar, so this is a matter of proving +# it rather than provisioning anything. +# +# Deliberately dropped, not pending: +# postgres -- .gitlab-ci.yml carried a `test_postgres` job for a check that does +# not exist in this repository. It is dead config, not a gap. +CHECKS=( + agent_integration_sample + agent_v2_integration_sample + agent_v2_integration_stateful_sample + agent_v2_integration_transactional_sample + dynatrace_base + dynatrace_health + dynatrace_topology + kubelet + openmetrics + servicenow + stackstate_checks_base + static_health + static_topology + vsphere + zabbix +) + +# A change anywhere here invalidates every suite: the base classes and the test +# helpers are imported by all of them, and the setup scripts build the venv the +# suites run in. +SHARED_PATHS=( + stackstate_checks_base/ + stackstate_checks_dev/ + stackstate_checks_tests_helper/ + .setup-scripts/ + .github/workflows/checks-tests.yml + .github/scripts/select-checks.sh +) + +emit() { + local -a selected=("$@") + local json + if [ "${#selected[@]}" -eq 0 ]; then + json="[]" + else + json=$(printf '%s\n' "${selected[@]}" | sort -u | jq -R . | jq -c -s .) + fi + echo "checks=${json}" >>"${GITHUB_OUTPUT}" + echo "Selected suites: ${json}" +} + +# Anything that is not a pull request is a full run. On the release branch the +# whole matrix is the point (it is what Cerberus reports on), and a manual +# dispatch is an explicit request for everything. +if [ "${EVENT_NAME}" != "pull_request" ]; then + echo "Event '${EVENT_NAME}' is not a pull request: running every suite." + emit "${CHECKS[@]}" + exit 0 +fi + +# Diffing against the merge base keeps a stale base branch from dragging +# unrelated commits into the change set. +MERGE_BASE=$(git merge-base "origin/${BASE_REF}" HEAD) +mapfile -t CHANGED < <(git diff --name-only "${MERGE_BASE}" HEAD) + +echo "Changed files (${#CHANGED[@]}) against ${BASE_REF} @ ${MERGE_BASE}:" +printf ' %s\n' "${CHANGED[@]}" + +matches_prefix() { + local file=$1 prefix + shift + for prefix in "$@"; do + case "${file}" in + "${prefix}"*) return 0 ;; + esac + done + return 1 +} + +for file in "${CHANGED[@]}"; do + if matches_prefix "${file}" "${SHARED_PATHS[@]}"; then + echo "'${file}' is shared CI or library code: running every suite." + emit "${CHECKS[@]}" + exit 0 + fi +done + +SELECTED=() +for file in "${CHANGED[@]}"; do + for check in "${CHECKS[@]}"; do + if [ "${file#"${check}"/}" != "${file}" ]; then + SELECTED+=("${check}") + fi + done +done + +emit "${SELECTED[@]+"${SELECTED[@]}"}" diff --git a/.github/workflows/checks-tests.yml b/.github/workflows/checks-tests.yml new file mode 100644 index 00000000..d40905a8 --- /dev/null +++ b/.github/workflows/checks-tests.yml @@ -0,0 +1,214 @@ +name: Check tests + +# Ported from .gitlab-ci.yml as part of the GitLab -> GitHub migration +# (STAC-25142 / STAC-25463), phase 1: the pure-Python check suites. +# +# WHAT MOVED +# linux_deps + the `test_` job family -> the `check-tests` matrix below. +# The per-job `changes:` rules -> .github/scripts/select-checks.sh. +# The validate suite that rode along inside `test_stackstate_checks_base` +# -> its own `validate` job, so a metadata +# failure is legible as its own PR check +# instead of hiding inside a test job. +# +# WHAT IS DELIBERATELY NOT HERE +# splunk_{base,health,metric,topology} and stackstate_checks_dev +# The only five suites that need a Docker daemon (the four splunk suites +# drive a real Splunk container via docker-compose; checks_dev tests the +# toolkit's own Docker helpers). Both public ARC runners ship a DinD +# sidecar, so this is about exercising that path rather than provisioning +# anything -- phase 2. +# test_postgres +# Dead config: .gitlab-ci.yml tests a `postgres` check that does not exist +# in this repository. Dropped, not pending. +# print_env +# A bare `printenv`. This repository is PUBLIC, so that job publishes every +# CI credential in scope to a world-readable log. Not ported at any phase; +# `secrets: inherit` is likewise never used here. +# The Windows lane +# There is none to port. `.gitlab-ci.yml` defines a `.windows_env` anchor +# but no job has ever referenced it, and Windows is not a supported target, +# so the orphaned .setup-scripts/conda_env.ps1 + windows_*.cmd helpers can +# be retired with the GitLab pipeline (STAC-25464). +# publish-checks-dev / the runner-image `docker` job +# Publishing needs write credentials this repo does not hold; pulumi-infra +# schedules integrations' publishing role for its section 7.4. Until then +# releases keep running from GitLab, so the GitLab pipeline stays live. +# A Cerberus failure notification +# Unlike stackstate-agent, this pipeline has never had one -- there is no +# notify job in .gitlab-ci.yml and no .cerberus directory -- so adding it +# would be new capability, not a port. It also needs CERBERUS_LAMBDA_URL, +# which is a private-visibility org secret and so unreadable from this +# PUBLIC repo without a pulumi-infra grant. Tracked as a follow-up. +# +# CREDENTIALS +# Pulling the runner image needs vars.REGISTRY_HOST / vars.REGISTRY_USER and +# secrets.REGISTRY_PASSWORD, which are org-level visibility=all (STAC-25350) +# and therefore already reach this PUBLIC repo -- phase 1 needs no new secret +# provisioning. Note it composes vars.REGISTRY_HOST + /docker rather than using +# vars.REGISTRY_DOCKER_URL: that variable is private-visibility and so is NOT +# readable here. Org secrets are never exposed to fork PRs, hence the same-repo +# guard on every job that pulls the image. + +on: + pull_request: + # Mirrors GitLab's `master_branch` rule, which hardcoded the release branch the + # same way: the full matrix runs there regardless of what a given commit + # touched, so the branch always has a complete verdict. + push: + branches: + - stackstate-7.78.2 + workflow_dispatch: + +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +defaults: + run: + shell: bash + +env: + # Tag of the prebuilt test image. Kept in lockstep with RUNNER_IMAGE_TAG in + # .gitlab-ci.yml until the GitLab pipeline is retired; the image is still built + # and published by that pipeline's `docker` job. + RUNNER_IMAGE_TAG: 20260625-py313 + +jobs: + select-checks: + name: Select check suites to run + # Not strictly credential-bound, but there is nothing to select for a fork PR + # whose downstream jobs cannot pull the image anyway. + if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository + runs-on: docker-public + timeout-minutes: 10 + outputs: + checks: ${{ steps.select.outputs.checks }} + runner_image: ${{ steps.image.outputs.ref }} + steps: + - name: Check out repository + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + with: + # Full history: the selector diffs against the merge base with the base + # branch, which a shallow clone cannot resolve. + fetch-depth: 0 + persist-credentials: false + + - name: Resolve the runner image reference + id: image + env: + REGISTRY_HOST: ${{ vars.REGISTRY_HOST }} + run: | + set -euo pipefail + # Same image the GitLab pipeline runs on, via the same read-only proxy: + # GitLab's ${REGISTRY_DOCKER_URL} is registry.tooling.stackstate.io/docker. + echo "ref=${REGISTRY_HOST}/docker/stackstate/stackstate-agent-integrations-runner:${RUNNER_IMAGE_TAG}" >>"${GITHUB_OUTPUT}" + + - name: Select check suites + id: select + env: + EVENT_NAME: ${{ github.event_name }} + BASE_REF: ${{ github.base_ref }} + run: | + set -euo pipefail + # The ARC work volume is owned by the runner uid; mark it safe so the + # selector's git calls are not rejected as "dubious ownership". + git config --global --add safe.directory '*' + bash .github/scripts/select-checks.sh + + validate: + name: Check metadata validation (checksdev validate) + # Ported from the `checksdev validate *` commands that opened + # test_stackstate_checks_base. Cheap, repo-wide, and independent of the + # matrix, so it runs on every change rather than per suite. + if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository + needs: select-checks + runs-on: docker-public + timeout-minutes: 30 + container: + # Tag-pinned rather than digest-pinned: the tag is produced by the GitLab + # `docker` job and updated in lockstep with .gitlab-ci.yml, so pinning a + # digest here would silently drift from the pipeline that builds it. + image: ${{ needs.select-checks.outputs.runner_image }} # zizmor: ignore[unpinned-images] + credentials: + username: ${{ vars.REGISTRY_USER }} + password: ${{ secrets.REGISTRY_PASSWORD }} + steps: + - name: Check out repository + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + with: + persist-credentials: false + + - name: Build the toolchain virtualenv + run: | + set -eo pipefail + git config --global --add safe.directory '*' + # Creates venv/ and installs checksdev; the GitLab `linux_deps` job did + # this once and shipped venv/ as an artifact. Here each job builds its + # own: the matrix legs run in parallel, so repeating it costs runner + # time but no wall-clock, and it avoids relocating a venv through the + # artifact store. Worth revisiting with real timings, the way the + # agent's cache image was justified (STAC-25429). + source .setup-scripts/setup_env.sh + + - name: checksdev validate + run: | + set -eo pipefail + source venv/bin/activate + checksdev validate config + checksdev validate dep + checksdev validate manifest --include-extras + checksdev validate metadata + checksdev validate service-checks + + check-tests: + name: Check tests (${{ matrix.check }}) + if: >- + ${{ (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository) + && needs.select-checks.outputs.checks != '[]' }} + needs: select-checks + runs-on: docker-public + timeout-minutes: 45 + strategy: + # One suite's failure should not mask the state of the other fourteen. + fail-fast: false + matrix: + check: ${{ fromJson(needs.select-checks.outputs.checks) }} + container: + image: ${{ needs.select-checks.outputs.runner_image }} # zizmor: ignore[unpinned-images] + credentials: + username: ${{ vars.REGISTRY_USER }} + password: ${{ secrets.REGISTRY_PASSWORD }} + steps: + - name: Check out repository + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + with: + persist-credentials: false + + - name: Build the toolchain virtualenv + run: | + set -eo pipefail + git config --global --add safe.directory '*' + source .setup-scripts/setup_env.sh + + - name: checksdev test ${{ matrix.check }} + env: + CHECK: ${{ matrix.check }} + run: | + set -eo pipefail + source venv/bin/activate + checksdev test --cov "${CHECK}" + + - name: checksdev benchmarks ${{ matrix.check }} + env: + CHECK: ${{ matrix.check }} + # Non-blocking, matching GitLab's `|| true`: benchmarks are reported for + # information and have never gated a merge. + continue-on-error: true + run: | + set -eo pipefail + source venv/bin/activate + checksdev test "${CHECK}" --bench From 1e5640c9da10464b7da349ccabc9f9fb61cdb739 Mon Sep 17 00:00:00 2001 From: Louis Parkin Date: Mon, 3 Aug 2026 15:44:33 +0200 Subject: [PATCH 02/11] STAC-25463 Re-trigger CI now that the public runner group grant is applied pulumi-infra #262 is merged and applied (~ 1 updated), so docker-public can now pick up jobs from this repository. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> From 7f95e232c4e498050277ba5f1711dd5e654379f8 Mon Sep 17 00:00:00 2001 From: Louis Parkin Date: Mon, 3 Aug 2026 16:25:58 +0200 Subject: [PATCH 03/11] STAC-25463: configure the private PyPI index before running check tests The GitLab `.linux_test` anchor ran .setup-scripts/setup_artifact_registry.sh immediately before `checksdev test`, and the port dropped it. Fourteen of the fifteen phase-1 suites resolve everything from public PyPI and did not notice, but the vsphere suite pins vsphere-automation-sdk==1.82.0, which public PyPI does not carry (only a 0.0.1 placeholder), so it failed with: ERROR: Could not find a version that satisfies the requirement vsphere-automation-sdk==1.82.0 (from versions: 0.0.1) Restores the step in the same position as GitLab had it. The script writes ~/.pip/pip.conf and a 0600 ~/.netrc, so credentials never appear in the index URL. The step is guarded on the credentials being present rather than run unconditionally: only one suite actually needs the private index, so an absent credential should not fail the fourteen that do not. When it is missing the job emits a warning annotation naming the consequence, and the affected suite still fails visibly on its own unresolvable pin. Requires the pull-only grant in StackVista/pulumi-infra#263. setup_artifactory_docker.sh stays out; it is a registry docker login that only the deferred Docker suites need, and it comes across with them in phase 2. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/checks-tests.yml | 47 ++++++++++++++++++++++++++---- 1 file changed, 41 insertions(+), 6 deletions(-) diff --git a/.github/workflows/checks-tests.yml b/.github/workflows/checks-tests.yml index d40905a8..17ee86c6 100644 --- a/.github/workflows/checks-tests.yml +++ b/.github/workflows/checks-tests.yml @@ -17,7 +17,9 @@ name: Check tests # drive a real Splunk container via docker-compose; checks_dev tests the # toolkit's own Docker helpers). Both public ARC runners ship a DinD # sidecar, so this is about exercising that path rather than provisioning -# anything -- phase 2. +# anything -- phase 2. That phase also brings across +# .setup-scripts/setup_artifactory_docker.sh (the registry docker login) +# and COMPOSE_HTTP_TIMEOUT, which only those suites need. # test_postgres # Dead config: .gitlab-ci.yml tests a `postgres` check that does not exist # in this repository. Dropped, not pending. @@ -44,11 +46,19 @@ name: Check tests # CREDENTIALS # Pulling the runner image needs vars.REGISTRY_HOST / vars.REGISTRY_USER and # secrets.REGISTRY_PASSWORD, which are org-level visibility=all (STAC-25350) -# and therefore already reach this PUBLIC repo -- phase 1 needs no new secret -# provisioning. Note it composes vars.REGISTRY_HOST + /docker rather than using -# vars.REGISTRY_DOCKER_URL: that variable is private-visibility and so is NOT -# readable here. Org secrets are never exposed to fork PRs, hence the same-repo -# guard on every job that pulls the image. +# and therefore already reach this PUBLIC repo. Note it composes +# vars.REGISTRY_HOST + /docker rather than using vars.REGISTRY_DOCKER_URL: that +# variable is private-visibility and so is NOT readable here. Org secrets are +# never exposed to fork PRs, hence the same-repo guard on every job that pulls +# the image. +# +# Resolving check requirements additionally needs the private PyPI index, for +# the handful of pins that are not on public PyPI. That is +# vars.GITLAB_PACKAGE_REGISTRY_PYPI_SIMPLE_URL and +# secrets.GITLAB_PACKAGE_REGISTRY_USER, granted to this repo in pulumi-infra +# (StackVista/pulumi-infra#263), alongside the already-org-wide +# secrets.GITLAB_PACKAGE_REGISTRY_READONLY_PASSWORD. Pull-only: this repo's +# *publishing* role is still deferred, per the note above. on: pull_request: @@ -194,6 +204,31 @@ jobs: git config --global --add safe.directory '*' source .setup-scripts/setup_env.sh + - name: Configure the private PyPI index (GitLab Package Registry) + env: + GITLAB_PACKAGE_REGISTRY_PYPI_SIMPLE_URL: ${{ vars.GITLAB_PACKAGE_REGISTRY_PYPI_SIMPLE_URL }} + GITLAB_PACKAGE_REGISTRY_USER: ${{ secrets.GITLAB_PACKAGE_REGISTRY_USER }} + GITLAB_PACKAGE_REGISTRY_READONLY_PASSWORD: ${{ secrets.GITLAB_PACKAGE_REGISTRY_READONLY_PASSWORD }} + # A few checks pin wheels that are not on public PyPI and are served from the + # central registry instead (currently only vsphere, which pins + # vsphere-automation-sdk). This mirrors what the GitLab `.linux_test` anchor did + # immediately before `checksdev test`. The script writes ~/.pip/pip.conf and a + # 0600 ~/.netrc, keeping credentials out of the index URL. + # + # Guarded rather than unconditional: only one suite actually needs the private + # index, so a missing credential should not fail the fourteen that resolve + # everything from public PyPI. When it is missing we say so loudly, and the + # affected suite still fails visibly on its own unresolvable pin. + run: | + set -eo pipefail + if [ -z "${GITLAB_PACKAGE_REGISTRY_PYPI_SIMPLE_URL}" ] \ + || [ -z "${GITLAB_PACKAGE_REGISTRY_USER}" ] \ + || [ -z "${GITLAB_PACKAGE_REGISTRY_READONLY_PASSWORD}" ]; then + echo "::warning title=Private PyPI index not configured::GitLab Package Registry credentials are not available to this job. Checks that pin wheels absent from public PyPI (currently 'vsphere') will fail to resolve their requirements." + exit 0 + fi + .setup-scripts/setup_artifact_registry.sh + - name: checksdev test ${{ matrix.check }} env: CHECK: ${{ matrix.check }} From 29b6c54d1ae38ae193bd27eabf41e15013f25558 Mon Sep 17 00:00:00 2001 From: Louis Parkin Date: Tue, 4 Aug 2026 09:06:10 +0200 Subject: [PATCH 04/11] STAC-25463: re-trigger CI now that the PyPI pull credentials are provisioned pulumi-infra#263 merged and applied (+ 2 created), so GITLAB_PACKAGE_REGISTRY_PYPI_SIMPLE_URL and GITLAB_PACKAGE_REGISTRY_USER now resolve in this repo. Expecting the vsphere suite to resolve its vsphere-automation-sdk pin and the matrix to go fully green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> From 7800da092a111c96a81c3d787d2c17ba97303b3a Mon Sep 17 00:00:00 2001 From: Louis Parkin Date: Tue, 4 Aug 2026 09:33:24 +0200 Subject: [PATCH 05/11] STAC-25463: defer the vsphere suite; its SDK is not served by any index vsphere pins vsphere-automation-sdk==1.82.0. Public PyPI serves only a 0.0.1 placeholder, and with the private index now provisioned and configured the result is unchanged: pip still reports "from versions: 0.0.1". The index itself is fine. setup_artifact_registry.sh runs and writes its config, pip reports no authentication error, and the same credentials and project are used by stackstate-agent, whose DEB build resolves against them. So the registry is reachable and simply does not carry this package. requirements.in says the wheel was to be built and published to artifactory.tooling.stackstate.io, which has since been retired in favour of the GitLab package registry. Whether the wheel was migrated, and to which project, needs GitLab API access to settle, so it is tracked separately rather than left failing in this PR. Phase 1 therefore ships the fourteen suites that resolve entirely from public PyPI. The registry step stays: it is correct, proven to work, and the deferred Docker suites will need it. Also drops a stale reference to Cerberus in the selector, which this pipeline deliberately does not wire up. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/scripts/select-checks.sh | 22 +++++++++++++++++----- .github/workflows/checks-tests.yml | 29 ++++++++++++++++++----------- 2 files changed, 35 insertions(+), 16 deletions(-) diff --git a/.github/scripts/select-checks.sh b/.github/scripts/select-checks.sh index e0683e8f..c77a73cc 100755 --- a/.github/scripts/select-checks.sh +++ b/.github/scripts/select-checks.sh @@ -24,8 +24,9 @@ set -euo pipefail -# Suites currently running on GitHub Actions. Phase 1 is the 15 pure-Python -# suites: they need no Docker daemon, so they are the low-risk half of the port. +# Suites currently running on GitHub Actions. Phase 1 is the 14 pure-Python +# suites that resolve entirely from public PyPI, so they need neither a Docker +# daemon nor the private package index. # # Deliberately NOT here yet (phase 2, needs the DinD story exercised first): # splunk_base, splunk_health, splunk_metric, splunk_topology @@ -35,6 +36,17 @@ set -euo pipefail # Both public ARC runners provide a DinD sidecar, so this is a matter of proving # it rather than provisioning anything. # +# vsphere +# -- blocked on sourcing vsphere-automation-sdk==1.82.0, which public PyPI +# does not carry (it serves only a 0.0.1 placeholder). The private index +# is wired up correctly and authenticates -- pip reports no 401 and the +# same credentials work for stackstate-agent -- but the registry does not +# serve this package either, so pip still sees only 0.0.1. requirements.in +# says the wheel was to be built and published to +# artifactory.tooling.stackstate.io, which has since been retired, so +# where it lives now is an open question that needs GitLab API access to +# settle. Tracked separately rather than left failing here. +# # Deliberately dropped, not pending: # postgres -- .gitlab-ci.yml carried a `test_postgres` job for a check that does # not exist in this repository. It is dead config, not a gap. @@ -52,7 +64,6 @@ CHECKS=( stackstate_checks_base static_health static_topology - vsphere zabbix ) @@ -81,8 +92,9 @@ emit() { } # Anything that is not a pull request is a full run. On the release branch the -# whole matrix is the point (it is what Cerberus reports on), and a manual -# dispatch is an explicit request for everything. +# whole matrix is the point -- the branch should always carry a complete verdict, +# regardless of what a given commit touched -- and a manual dispatch is an +# explicit request for everything. if [ "${EVENT_NAME}" != "pull_request" ]; then echo "Event '${EVENT_NAME}' is not a pull request: running every suite." emit "${CHECKS[@]}" diff --git a/.github/workflows/checks-tests.yml b/.github/workflows/checks-tests.yml index 17ee86c6..7475d6e4 100644 --- a/.github/workflows/checks-tests.yml +++ b/.github/workflows/checks-tests.yml @@ -23,6 +23,14 @@ name: Check tests # test_postgres # Dead config: .gitlab-ci.yml tests a `postgres` check that does not exist # in this repository. Dropped, not pending. +# vsphere +# Blocked upstream, not by this port. It pins vsphere-automation-sdk==1.82.0, +# which public PyPI does not serve (only a 0.0.1 placeholder) and which the +# private index does not serve either -- pip reports no auth error and the +# same credentials work for stackstate-agent, so the index is reachable and +# simply does not carry it. requirements.in says the wheel was to be built and +# published to the since-retired artifactory.tooling.stackstate.io, so where it +# lives now needs settling before this suite can run anywhere. # print_env # A bare `printenv`. This repository is PUBLIC, so that job publishes every # CI credential in scope to a world-readable log. Not ported at any phase; @@ -183,7 +191,7 @@ jobs: runs-on: docker-public timeout-minutes: 45 strategy: - # One suite's failure should not mask the state of the other fourteen. + # One suite's failure should not mask the state of the others. fail-fast: false matrix: check: ${{ fromJson(needs.select-checks.outputs.checks) }} @@ -209,22 +217,21 @@ jobs: GITLAB_PACKAGE_REGISTRY_PYPI_SIMPLE_URL: ${{ vars.GITLAB_PACKAGE_REGISTRY_PYPI_SIMPLE_URL }} GITLAB_PACKAGE_REGISTRY_USER: ${{ secrets.GITLAB_PACKAGE_REGISTRY_USER }} GITLAB_PACKAGE_REGISTRY_READONLY_PASSWORD: ${{ secrets.GITLAB_PACKAGE_REGISTRY_READONLY_PASSWORD }} - # A few checks pin wheels that are not on public PyPI and are served from the - # central registry instead (currently only vsphere, which pins - # vsphere-automation-sdk). This mirrors what the GitLab `.linux_test` anchor did - # immediately before `checksdev test`. The script writes ~/.pip/pip.conf and a - # 0600 ~/.netrc, keeping credentials out of the index URL. + # Mirrors what the GitLab `.linux_test` anchor did immediately before + # `checksdev test`. The script writes ~/.pip/pip.conf and a 0600 ~/.netrc, + # keeping credentials out of the index URL. # - # Guarded rather than unconditional: only one suite actually needs the private - # index, so a missing credential should not fail the fourteen that resolve - # everything from public PyPI. When it is missing we say so loudly, and the - # affected suite still fails visibly on its own unresolvable pin. + # No phase-1 suite depends on this today -- all fourteen resolve from public + # PyPI -- but the deferred Docker suites will, and keeping it here means the + # index is configured wherever `checksdev test` runs. Guarded rather than + # unconditional so an absent credential cannot fail suites that do not use it; + # when it is missing we say so loudly rather than failing obscurely later. run: | set -eo pipefail if [ -z "${GITLAB_PACKAGE_REGISTRY_PYPI_SIMPLE_URL}" ] \ || [ -z "${GITLAB_PACKAGE_REGISTRY_USER}" ] \ || [ -z "${GITLAB_PACKAGE_REGISTRY_READONLY_PASSWORD}" ]; then - echo "::warning title=Private PyPI index not configured::GitLab Package Registry credentials are not available to this job. Checks that pin wheels absent from public PyPI (currently 'vsphere') will fail to resolve their requirements." + echo "::warning title=Private PyPI index not configured::GitLab Package Registry credentials are not available to this job. Any check pinning a wheel that public PyPI does not serve will fail to resolve its requirements." exit 0 fi .setup-scripts/setup_artifact_registry.sh From ecad1da9554c390027f52685be826eb30131096c Mon Sep 17 00:00:00 2001 From: Louis Parkin Date: Tue, 4 Aug 2026 11:49:12 +0200 Subject: [PATCH 06/11] STAC-25463 Run the vsphere suite: pass HOME through tox so pip sees the private index The suite was deferred on the theory that vsphere-automation-sdk==1.82.0 was unavailable. It is not: the wheel is published in the central registry (GitLab project 71271774) at exactly the pinned version, alongside its vapi/nsx dependencies. The real cause is that pip never contacted the index at all. tox drops every variable a testenv does not name in `passenv`, HOME among them, and pip then resolves `~` from the passwd database instead of the environment. A container job on GitHub Actions runs with HOME=/github/home, so setup_artifact_registry.sh wrote /github/home/.pip/pip.conf while pip inside tox read /root/.pip/pip.conf -- absent -- and quietly resolved from public PyPI alone, which serves only a 0.0.1 placeholder for that name. That is also why the failure carried no 401: there was no request to authenticate. And why GitLab is unaffected: its container runs as root with HOME=/root, so the passwd fallback lands on the same path the script wrote. Adding HOME to the vsphere testenv's passenv is enough. Re-enables the suite, taking phase 1 from 14 to 15, and records the trap next to both the setup step and the passenv entry so the deferred Docker suites do not rediscover it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/scripts/select-checks.sh | 17 +++-------------- .github/workflows/checks-tests.yml | 26 +++++++++++++------------- vsphere/tox.ini | 8 ++++++++ 3 files changed, 24 insertions(+), 27 deletions(-) diff --git a/.github/scripts/select-checks.sh b/.github/scripts/select-checks.sh index c77a73cc..870b4b61 100755 --- a/.github/scripts/select-checks.sh +++ b/.github/scripts/select-checks.sh @@ -24,9 +24,8 @@ set -euo pipefail -# Suites currently running on GitHub Actions. Phase 1 is the 14 pure-Python -# suites that resolve entirely from public PyPI, so they need neither a Docker -# daemon nor the private package index. +# Suites currently running on GitHub Actions. Phase 1 is the 15 suites that need +# no Docker daemon. # # Deliberately NOT here yet (phase 2, needs the DinD story exercised first): # splunk_base, splunk_health, splunk_metric, splunk_topology @@ -36,17 +35,6 @@ set -euo pipefail # Both public ARC runners provide a DinD sidecar, so this is a matter of proving # it rather than provisioning anything. # -# vsphere -# -- blocked on sourcing vsphere-automation-sdk==1.82.0, which public PyPI -# does not carry (it serves only a 0.0.1 placeholder). The private index -# is wired up correctly and authenticates -- pip reports no 401 and the -# same credentials work for stackstate-agent -- but the registry does not -# serve this package either, so pip still sees only 0.0.1. requirements.in -# says the wheel was to be built and published to -# artifactory.tooling.stackstate.io, which has since been retired, so -# where it lives now is an open question that needs GitLab API access to -# settle. Tracked separately rather than left failing here. -# # Deliberately dropped, not pending: # postgres -- .gitlab-ci.yml carried a `test_postgres` job for a check that does # not exist in this repository. It is dead config, not a gap. @@ -64,6 +52,7 @@ CHECKS=( stackstate_checks_base static_health static_topology + vsphere zabbix ) diff --git a/.github/workflows/checks-tests.yml b/.github/workflows/checks-tests.yml index 7475d6e4..70138097 100644 --- a/.github/workflows/checks-tests.yml +++ b/.github/workflows/checks-tests.yml @@ -23,14 +23,6 @@ name: Check tests # test_postgres # Dead config: .gitlab-ci.yml tests a `postgres` check that does not exist # in this repository. Dropped, not pending. -# vsphere -# Blocked upstream, not by this port. It pins vsphere-automation-sdk==1.82.0, -# which public PyPI does not serve (only a 0.0.1 placeholder) and which the -# private index does not serve either -- pip reports no auth error and the -# same credentials work for stackstate-agent, so the index is reachable and -# simply does not carry it. requirements.in says the wheel was to be built and -# published to the since-retired artifactory.tooling.stackstate.io, so where it -# lives now needs settling before this suite can run anywhere. # print_env # A bare `printenv`. This repository is PUBLIC, so that job publishes every # CI credential in scope to a world-readable log. Not ported at any phase; @@ -221,11 +213,19 @@ jobs: # `checksdev test`. The script writes ~/.pip/pip.conf and a 0600 ~/.netrc, # keeping credentials out of the index URL. # - # No phase-1 suite depends on this today -- all fourteen resolve from public - # PyPI -- but the deferred Docker suites will, and keeping it here means the - # index is configured wherever `checksdev test` runs. Guarded rather than - # unconditional so an absent credential cannot fail suites that do not use it; - # when it is missing we say so loudly rather than failing obscurely later. + # `vsphere` is the one phase-1 suite that needs this; the deferred Docker + # suites will too. Guarded rather than unconditional so an absent credential + # cannot fail the suites that do not use it; when it is missing we say so + # loudly rather than failing obscurely later. + # + # NOTE: writing these under $HOME is only half the job. tox drops every + # variable absent from a testenv's `passenv`, HOME included, and pip then + # resolves `~` from the passwd database rather than the environment. Because + # a container job here runs with HOME=/github/home instead of the account + # default, a testenv that installs from this index must list HOME in its + # passenv or it will silently read a pip.conf that was never written and fall + # back to public PyPI alone -- with no auth error to show for it. See + # vsphere/tox.ini. run: | set -eo pipefail if [ -z "${GITLAB_PACKAGE_REGISTRY_PYPI_SIMPLE_URL}" ] \ diff --git a/vsphere/tox.ini b/vsphere/tox.ini index 3b0e5544..a4b86bec 100644 --- a/vsphere/tox.ini +++ b/vsphere/tox.ini @@ -14,9 +14,17 @@ deps = setuptools<78 -e../stackstate_checks_base[deps] -rrequirements-dev.txt +; vsphere-automation-sdk is pinned to a build that only exists in the private +; GitLab package index, so `pip install -r requirements.in` below has to see the +; ~/.pip/pip.conf and ~/.netrc that .setup-scripts/setup_artifact_registry.sh +; writes. tox drops every variable not listed in passenv, and pip then resolves +; `~` from the passwd database instead of the environment -- which silently points +; at the wrong home whenever HOME is not the account's default, as in a GitHub +; Actions container job where HOME is /github/home. Hence HOME below. passenv = DOCKER* COMPOSE* + HOME commands = pip install -r requirements.in pip install -r requirements-dev.txt From bea6ddf8bbf2561192fc6bf5c7f893b41f9657b3 Mon Sep 17 00:00:00 2001 From: Louis Parkin Date: Tue, 4 Aug 2026 13:42:32 +0200 Subject: [PATCH 07/11] STAC-25463 Address review: BCI image, credential-free PR runs, stable gate, tag coverage Four findings from review, in order. Tag coverage. GitLab's .base_integration_rules includes the release_branch anchor (if: $CI_COMMIT_TAG), so every test job also ran on tag pipelines. The workflow declared only push.branches and silently lost that. Adds a tags filter using '**' rather than '*', which would skip any tag name containing a slash. The selector already treats every non-pull_request event as a full run, so no change was needed there. BCI base image, which also removes the credentials. The GitLab runner image is FROM python:3.13.14-bookworm, against the mandatory SUSE BCI policy. That image exists mainly to carry Docker CE and docker-compose, which only the phase-2 Splunk and checks_dev suites use -- phase 1 needs a Python toolchain and nothing else. Switches the container to a digest-pinned registry.suse.com/bci/python, adding gcc, python313-devel and libffi-devel, which the image does not ship and which cython and pyyaml==6.0.1 need to compile from source. Because that registry is public, the jobs no longer need vars.REGISTRY_USER or secrets.REGISTRY_PASSWORD. This is the substantive part of the credential finding: every job here runs PR-authored workflow, setup and test code on a PUBLIC repository, so a secret reachable from that code is reachable by anyone who can open a branch. It also unblocks Dependabot, which receives no Actions secrets and so could never have pulled the private image. What remains is the read-only private PyPI pull, scoped to the one step that needs it and guarded so its absence warns rather than fails. The fork boundary stays, now justified by running unreviewed code on self-hosted runners rather than by secret exposure. BCI ships 3.13.13 against the 3.13.14 the agent embeds. Patch releases are bugfix-only, and the full matrix including vsphere against the private index was verified green on 3.13.13 before making the switch. Stable required check. The only test statuses were dynamically named matrix legs in a job that disappears when the selector returns an empty array, which branch protection cannot depend on. Adds a ci-success job that always runs and aggregates the results, treating skipped as legitimate but failure and cancelled as fatal, plus the requested Zizmor job. ci-success deliberately checks out nothing so it remains safe on fork PRs, where it is the only way a required check can report at all. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/checks-tests.yml | 200 +++++++++++++++++++++++------ 1 file changed, 164 insertions(+), 36 deletions(-) diff --git a/.github/workflows/checks-tests.yml b/.github/workflows/checks-tests.yml index 70138097..c45ec075 100644 --- a/.github/workflows/checks-tests.yml +++ b/.github/workflows/checks-tests.yml @@ -19,7 +19,10 @@ name: Check tests # sidecar, so this is about exercising that path rather than provisioning # anything -- phase 2. That phase also brings across # .setup-scripts/setup_artifactory_docker.sh (the registry docker login) -# and COMPOSE_HTTP_TIMEOUT, which only those suites need. +# and COMPOSE_HTTP_TIMEOUT, which only those suites need. It will also need +# a docker client, which the BCI Python image used here does not carry -- +# either a BCI image with docker added, or a DinD service container, rather +# than a return to the private python:3.13.14-bookworm runner image. # test_postgres # Dead config: .gitlab-ci.yml tests a `postgres` check that does not exist # in this repository. Dropped, not pending. @@ -44,21 +47,29 @@ name: Check tests # PUBLIC repo without a pulumi-infra grant. Tracked as a follow-up. # # CREDENTIALS -# Pulling the runner image needs vars.REGISTRY_HOST / vars.REGISTRY_USER and -# secrets.REGISTRY_PASSWORD, which are org-level visibility=all (STAC-25350) -# and therefore already reach this PUBLIC repo. Note it composes -# vars.REGISTRY_HOST + /docker rather than using vars.REGISTRY_DOCKER_URL: that -# variable is private-visibility and so is NOT readable here. Org secrets are -# never exposed to fork PRs, hence the same-repo guard on every job that pulls -# the image. +# The container image is SUSE BCI from registry.suse.com, which is public, so +# these jobs need no registry credentials at all. That is deliberate: this is a +# PUBLIC repository, and every job here executes PR-authored workflow, setup and +# test code. Any secret exposed to that code is exposed to whoever can open a +# branch. The earlier design pulled a private runner image with +# vars.REGISTRY_USER / secrets.REGISTRY_PASSWORD; dropping it removes the +# registry password from the PR path entirely and, as a side effect, lets +# Dependabot PRs run -- they receive no Actions secrets, so the image pull +# could never have succeeded for them. # -# Resolving check requirements additionally needs the private PyPI index, for -# the handful of pins that are not on public PyPI. That is +# One credential remains, and only on the step that needs it: the read-only +# pull from the private PyPI index, for pins that public PyPI does not serve +# (currently vsphere-automation-sdk). That is # vars.GITLAB_PACKAGE_REGISTRY_PYPI_SIMPLE_URL and # secrets.GITLAB_PACKAGE_REGISTRY_USER, granted to this repo in pulumi-infra # (StackVista/pulumi-infra#263), alongside the already-org-wide -# secrets.GITLAB_PACKAGE_REGISTRY_READONLY_PASSWORD. Pull-only: this repo's -# *publishing* role is still deferred, per the note above. +# secrets.GITLAB_PACKAGE_REGISTRY_READONLY_PASSWORD. It is pull-only and +# least-privilege by construction; this repo's *publishing* role is still +# deferred, per the note above. The step is guarded so its absence degrades to +# a warning rather than failing suites that do not use the index. +# +# Fork PRs are still excluded from the credentialed jobs. Secret exposure is no +# longer the reason -- running unreviewed code on self-hosted runners is. on: pull_request: @@ -68,6 +79,13 @@ on: push: branches: - stackstate-7.78.2 + # GitLab's `release_branch` anchor (`if: $CI_COMMIT_TAG`) put every test job + # in .base_integration_rules on tag pipelines too, so releases carry the same + # verdict as the branch they cut from. Tags here are `-` + # (7.78.2-4), but the rule was any-tag and this stays faithful to it: `**` + # matches tag names containing `/`, which a bare `*` would silently skip. + tags: + - '**' workflow_dispatch: permissions: @@ -82,22 +100,40 @@ defaults: shell: bash env: - # Tag of the prebuilt test image. Kept in lockstep with RUNNER_IMAGE_TAG in - # .gitlab-ci.yml until the GitLab pipeline is retired; the image is still built - # and published by that pipeline's `docker` job. - RUNNER_IMAGE_TAG: 20260625-py313 + # SUSE BCI Python, per the mandatory BCI base-image policy. Digest-pinned so a + # run is reproducible and so Zizmor's unpinned-images audit passes; the comment + # carries the human-readable version the digest resolves to. + # + # This deliberately replaces the GitLab pipeline's + # stackstate-agent-integrations-runner image (FROM python:3.13.14-bookworm). + # That image exists mainly to carry Docker CE and docker-compose, which only the + # phase-2 Splunk / checks_dev suites use; the phase-1 suites need a Python + # toolchain and nothing else. Dropping it also drops the registry credentials + # these jobs used to need -- see CREDENTIALS below. + # + # BCI publishes 3.13.13, one patch behind the 3.13.14 the agent embeds and the + # GitLab image pinned. CPython patch releases are bugfix-only, and the full + # phase-1 matrix (including vsphere against the private index) was verified + # green on 3.13.13 before this switch. Worth realigning when BCI ships .14. + BCI_PYTHON_IMAGE: registry.suse.com/bci/python:3.13@sha256:7d36dd3ba6596fb690e31d956952059fd010604ad6309f06462c02c4c9c01461 # 3.13.13 + + # Packages the BCI image does not ship but the toolchain build needs: cython and + # pyyaml==6.0.1 have no cp313 wheels and are compiled from source. + BCI_BUILD_PACKAGES: gcc python313-devel libffi-devel jobs: select-checks: name: Select check suites to run - # Not strictly credential-bound, but there is nothing to select for a fork PR - # whose downstream jobs cannot pull the image anyway. + # Fork PRs never run: this is a PUBLIC repository on self-hosted runners, and + # a fork branch is unreviewed code. The jobs no longer carry registry + # credentials, but running arbitrary PR code on our own runners is a separate + # concern from secret exposure, so the boundary stays. if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository runs-on: docker-public timeout-minutes: 10 outputs: checks: ${{ steps.select.outputs.checks }} - runner_image: ${{ steps.image.outputs.ref }} + image: ${{ steps.image.outputs.ref }} steps: - name: Check out repository uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 @@ -107,15 +143,14 @@ jobs: fetch-depth: 0 persist-credentials: false - - name: Resolve the runner image reference + - name: Resolve the container image reference id: image - env: - REGISTRY_HOST: ${{ vars.REGISTRY_HOST }} + # `container.image` cannot read the `env` context, so the pin defined once + # at workflow level is republished here as an output the container jobs can + # reference. Keeps a single source of truth for the digest. run: | set -euo pipefail - # Same image the GitLab pipeline runs on, via the same read-only proxy: - # GitLab's ${REGISTRY_DOCKER_URL} is registry.tooling.stackstate.io/docker. - echo "ref=${REGISTRY_HOST}/docker/stackstate/stackstate-agent-integrations-runner:${RUNNER_IMAGE_TAG}" >>"${GITHUB_OUTPUT}" + echo "ref=${BCI_PYTHON_IMAGE}" >>"${GITHUB_OUTPUT}" - name: Select check suites id: select @@ -139,19 +174,22 @@ jobs: runs-on: docker-public timeout-minutes: 30 container: - # Tag-pinned rather than digest-pinned: the tag is produced by the GitLab - # `docker` job and updated in lockstep with .gitlab-ci.yml, so pinning a - # digest here would silently drift from the pipeline that builds it. - image: ${{ needs.select-checks.outputs.runner_image }} # zizmor: ignore[unpinned-images] - credentials: - username: ${{ vars.REGISTRY_USER }} - password: ${{ secrets.REGISTRY_PASSWORD }} + # Digest-pinned at workflow level (BCI_PYTHON_IMAGE); the ignore is only + # because Zizmor cannot follow the pin through a job output. + image: ${{ needs.select-checks.outputs.image }} # zizmor: ignore[unpinned-images] steps: - name: Check out repository uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 with: persist-credentials: false + - name: Install toolchain build dependencies + run: | + set -eo pipefail + zypper --non-interactive --gpg-auto-import-keys refresh + # shellcheck disable=SC2086 # deliberately word-split into package args + zypper --non-interactive install ${BCI_BUILD_PACKAGES} + - name: Build the toolchain virtualenv run: | set -eo pipefail @@ -188,16 +226,22 @@ jobs: matrix: check: ${{ fromJson(needs.select-checks.outputs.checks) }} container: - image: ${{ needs.select-checks.outputs.runner_image }} # zizmor: ignore[unpinned-images] - credentials: - username: ${{ vars.REGISTRY_USER }} - password: ${{ secrets.REGISTRY_PASSWORD }} + # Digest-pinned at workflow level (BCI_PYTHON_IMAGE); the ignore is only + # because Zizmor cannot follow the pin through a job output. + image: ${{ needs.select-checks.outputs.image }} # zizmor: ignore[unpinned-images] steps: - name: Check out repository uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 with: persist-credentials: false + - name: Install toolchain build dependencies + run: | + set -eo pipefail + zypper --non-interactive --gpg-auto-import-keys refresh + # shellcheck disable=SC2086 # deliberately word-split into package args + zypper --non-interactive install ${BCI_BUILD_PACKAGES} + - name: Build the toolchain virtualenv run: | set -eo pipefail @@ -254,3 +298,87 @@ jobs: set -eo pipefail source venv/bin/activate checksdev test "${CHECK}" --bench + + workflow-security: + name: Workflow security scan (Zizmor) + # Same fork boundary as the rest: this repository is PUBLIC and the runners are + # self-hosted, so unreviewed fork code is not checked out here even for a + # read-only audit. Depends on select-checks only to reuse the pinned image. + if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository + needs: select-checks + runs-on: docker-public + timeout-minutes: 15 + container: + # Digest-pinned at workflow level (BCI_PYTHON_IMAGE); the ignore is only + # because Zizmor cannot follow the pin through a job output. + image: ${{ needs.select-checks.outputs.image }} # zizmor: ignore[unpinned-images] + permissions: + contents: read + steps: + - name: Check out repository + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + with: + persist-credentials: false + + - name: Install Zizmor + env: + ZIZMOR_VERSION: 1.28.0 + run: | + set -eo pipefail + python3.13 -m venv /tmp/zizmor-venv + /tmp/zizmor-venv/bin/pip install --quiet "zizmor==${ZIZMOR_VERSION}" + + - name: Zizmor audit + run: | + set -eo pipefail + # Offline: the online audits need a GitHub token, and handing this job a + # token to scan PR-authored workflows is the very shape Zizmor exists to + # catch. Persona `regular` keeps it to findings worth blocking on. + /tmp/zizmor-venv/bin/zizmor \ + --persona regular \ + --collect=workflows,actions \ + . + + ci-success: + name: CI success + # The single stable status for branch protection. Every other status here is + # either dynamically named (the matrix legs are `Check tests ()`) or + # conditional (check-tests is skipped entirely when the selector returns an + # empty array, and everything is skipped for fork PRs). A required check must + # exist on every run, so this job always runs and decides the verdict itself. + # That includes fork PRs, where everything upstream is skipped: without a + # status here a fork PR could never satisfy branch protection. It checks out + # nothing and runs no repository code, so it stays safe to run in that context. + if: always() + needs: + - select-checks + - validate + - check-tests + - workflow-security + runs-on: docker-public + timeout-minutes: 5 + steps: + - name: Evaluate upstream job results + env: + # `skipped` is a legitimate outcome and must not be conflated with + # success: check-tests skips when no suite was selected, and the whole + # pipeline skips for fork PRs. Only `failure` and `cancelled` fail here. + RESULTS: >- + select-checks=${{ needs.select-checks.result }} + validate=${{ needs.validate.result }} + check-tests=${{ needs.check-tests.result }} + workflow-security=${{ needs.workflow-security.result }} + run: | + set -euo pipefail + failed=0 + for entry in ${RESULTS}; do + echo " ${entry}" + case "${entry#*=}" in + failure | cancelled) failed=1 ;; + esac + done + if [ "${failed}" -ne 0 ]; then + echo "::error title=CI failed::One or more required jobs did not succeed (see above)." + exit 1 + fi + echo "All required jobs succeeded or were legitimately skipped." From 863b1cd7330e2cbb81f8ebc34ccfb7551ac6eabe Mon Sep 17 00:00:00 2001 From: Louis Parkin Date: Wed, 5 Aug 2026 09:46:45 +0200 Subject: [PATCH 08/11] STAC-25463 Isolate private-index credentials and harden the CI gate Addresses the second review on #362. Credential exposure (P1). setup_artifact_registry.sh writes the GitLab Package Registry password to ~/.netrc, which the suite's own PR-authored test code could then read. Scoping the env block to one step never limited that. The suites needing the private index are now split into check-tests-private-index, behind the private-package-index GitHub Environment with dev-team as required reviewers and prevent_self_review enabled, so the credential is released only after someone other than the author approves that run. The other 14 suites no longer reference any secret. select-checks.sh partitions the selection into `checks` and `private_checks` to drive the split. Aggregate gate (P1). ci-success treated every non-failure as success, so a run where everything skipped reported green and could satisfy branch protection with no CI executed. It now requires select-checks, validate and workflow-security to succeed outright, and permits a matrix skip only when the selector chose nothing -- plus, for the private-index matrix only, when the PR is from a fork and could not have had the credential. Runners (P2). Everything moves to GitHub-hosted runners. The phase-1 suites are pure Python and need no Docker daemon, so docker-public bought no capability while forcing fork PRs to be excluded -- which is what made the all-skipped green verdict reachable. Forks now run the full credential-free matrix, and the direct BCI pull no longer crosses the self-hosted NAT. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/scripts/select-checks.sh | 68 +++++++- .github/workflows/checks-tests.yml | 255 ++++++++++++++++++++++------- 2 files changed, 255 insertions(+), 68 deletions(-) diff --git a/.github/scripts/select-checks.sh b/.github/scripts/select-checks.sh index 870b4b61..36dcee05 100755 --- a/.github/scripts/select-checks.sh +++ b/.github/scripts/select-checks.sh @@ -20,7 +20,16 @@ # * push / workflow_dispatch run everything (GitLab: `master_branch`, # `release_branch`). # -# Writes `checks=` to $GITHUB_OUTPUT for `fromJson()` in a matrix. +# Writes two arrays to $GITHUB_OUTPUT for `fromJson()` in a matrix: +# checks -- suites that need no credentials +# private_checks -- suites that install from the private GitLab PyPI index +# +# The split is a security boundary, not a convenience. The credential-free suites +# run on GitHub-hosted runners with no secrets in scope at all, so a fork PR can +# run them safely. The private-index suites need a registry password written to +# ~/.netrc, which any test code executing afterwards can read, so they run behind +# a protected GitHub Environment that requires a human approval first (STAC-25463 +# review). Keeping them in one matrix would hand that credential to every suite. set -euo pipefail @@ -56,6 +65,18 @@ CHECKS=( zabbix ) +# Suites whose requirements resolve only against the private GitLab PyPI index. +# `vsphere` pins vsphere-automation-sdk, which VMware never published to public +# PyPI (the name is squatted there by an unrelated 0.0.1 placeholder), so it is +# mirrored into the StackVista package registry and needs authentication. +# +# Everything not listed here is credential-free and must stay that way: adding a +# suite to this list moves it behind a manual approval gate, and removing the +# need for the private index is always the better fix. +PRIVATE_INDEX_CHECKS=( + vsphere +) + # A change anywhere here invalidates every suite: the base classes and the test # helpers are imported by all of them, and the setup scripts build the venv the # suites run in. @@ -68,16 +89,45 @@ SHARED_PATHS=( .github/scripts/select-checks.sh ) -emit() { - local -a selected=("$@") - local json - if [ "${#selected[@]}" -eq 0 ]; then - json="[]" +to_json() { + if [ "$#" -eq 0 ]; then + echo "[]" else - json=$(printf '%s\n' "${selected[@]}" | sort -u | jq -R . | jq -c -s .) + printf '%s\n' "$@" | sort -u | jq -R . | jq -c -s . fi - echo "checks=${json}" >>"${GITHUB_OUTPUT}" - echo "Selected suites: ${json}" +} + +is_private_index() { + local candidate=$1 check + for check in "${PRIVATE_INDEX_CHECKS[@]}"; do + [ "${candidate}" = "${check}" ] && return 0 + done + return 1 +} + +emit() { + local -a selected=("$@") + local -a public=() private=() + local check + for check in ${selected[@]+"${selected[@]}"}; do + if is_private_index "${check}"; then + private+=("${check}") + else + public+=("${check}") + fi + done + + local public_json private_json + public_json=$(to_json ${public[@]+"${public[@]}"}) + private_json=$(to_json ${private[@]+"${private[@]}"}) + + { + echo "checks=${public_json}" + echo "private_checks=${private_json}" + } >>"${GITHUB_OUTPUT}" + + echo "Selected credential-free suites: ${public_json}" + echo "Selected private-index suites: ${private_json}" } # Anything that is not a pull request is a full run. On the release branch the diff --git a/.github/workflows/checks-tests.yml b/.github/workflows/checks-tests.yml index c45ec075..2ebc2dfe 100644 --- a/.github/workflows/checks-tests.yml +++ b/.github/workflows/checks-tests.yml @@ -57,19 +57,40 @@ name: Check tests # Dependabot PRs run -- they receive no Actions secrets, so the image pull # could never have succeeded for them. # -# One credential remains, and only on the step that needs it: the read-only -# pull from the private PyPI index, for pins that public PyPI does not serve -# (currently vsphere-automation-sdk). That is -# vars.GITLAB_PACKAGE_REGISTRY_PYPI_SIMPLE_URL and +# One credential remains: the read-only pull from the private PyPI index, for +# pins that public PyPI does not serve (currently vsphere-automation-sdk). That +# is vars.GITLAB_PACKAGE_REGISTRY_PYPI_SIMPLE_URL and # secrets.GITLAB_PACKAGE_REGISTRY_USER, granted to this repo in pulumi-infra # (StackVista/pulumi-infra#263), alongside the already-org-wide # secrets.GITLAB_PACKAGE_REGISTRY_READONLY_PASSWORD. It is pull-only and # least-privilege by construction; this repo's *publishing* role is still -# deferred, per the note above. The step is guarded so its absence degrades to -# a warning rather than failing suites that do not use the index. +# deferred, per the note above. # -# Fork PRs are still excluded from the credentialed jobs. Secret exposure is no -# longer the reason -- running unreviewed code on self-hosted runners is. +# Scoping it to a single step is not enough, because the script writes it to +# ~/.netrc and the suite's own test code runs afterwards in the same job and can +# read it. So the suites that need it are split into their own job, +# `check-tests-private-index`, behind the `private-package-index` GitHub +# Environment. Required reviewers there mean the credential is released only +# after a human approves that specific run. The remaining suites never see it. +# +# Residual, and deliberately left alone here: the underlying secrets are still +# repo- and org-level, so a pull request that edits this workflow could add a +# new reference outside the gated job. That change is visible in the diff and is +# what review and the Zizmor audit are for. Making them environment-only secrets +# would remove even that path, but the org-level password is shared with other +# repos (stackstate-agent among them), so it needs its own change. +# +# RUNNERS +# Everything runs on GitHub-hosted runners. The suites are pure-Python and need +# no Docker daemon, so the self-hosted docker-public pool bought nothing while +# costing real isolation: fork PRs had to be excluded from it, which in turn +# meant a fork could never produce a CI verdict. On hosted runners forks run the +# full credential-free matrix, and only the private-index job keeps a fork guard +# -- forks receive no secrets in any case. This also removes the question of +# pulling upstream images across the self-hosted NAT: the BCI reference is +# direct, from a public registry, on infrastructure that is meant to reach it. +# The phase-2 Docker suites will need a runner with a daemon; that decision +# belongs with them, not here. on: pull_request: @@ -124,15 +145,16 @@ env: jobs: select-checks: name: Select check suites to run - # Fork PRs never run: this is a PUBLIC repository on self-hosted runners, and - # a fork branch is unreviewed code. The jobs no longer carry registry - # credentials, but running arbitrary PR code on our own runners is a separate - # concern from secret exposure, so the boundary stays. - if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository - runs-on: docker-public + # Runs for forks too. Every job in this workflow now runs on GitHub-hosted + # runners with no secrets in scope, except the private-index suites, which are + # isolated behind a protected environment below. There is therefore nothing a + # fork branch can reach here, and blocking forks outright would leave them + # unable to satisfy branch protection at all (STAC-25463 review). + runs-on: ubuntu-latest timeout-minutes: 10 outputs: checks: ${{ steps.select.outputs.checks }} + private_checks: ${{ steps.select.outputs.private_checks }} image: ${{ steps.image.outputs.ref }} steps: - name: Check out repository @@ -168,10 +190,10 @@ jobs: name: Check metadata validation (checksdev validate) # Ported from the `checksdev validate *` commands that opened # test_stackstate_checks_base. Cheap, repo-wide, and independent of the - # matrix, so it runs on every change rather than per suite. - if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository + # matrix, so it runs on every change rather than per suite. Credential-free, + # so it runs for fork PRs too. needs: select-checks - runs-on: docker-public + runs-on: ubuntu-latest timeout-minutes: 30 container: # Digest-pinned at workflow level (BCI_PYTHON_IMAGE); the ignore is only @@ -214,11 +236,12 @@ jobs: check-tests: name: Check tests (${{ matrix.check }}) - if: >- - ${{ (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository) - && needs.select-checks.outputs.checks != '[]' }} + # No fork guard: these suites carry no credentials and run on GitHub-hosted + # runners, so a fork branch has nothing to reach. Skipped only when the + # selector legitimately picked no credential-free suite. + if: needs.select-checks.outputs.checks != '[]' needs: select-checks - runs-on: docker-public + runs-on: ubuntu-latest timeout-minutes: 45 strategy: # One suite's failure should not mask the state of the others. @@ -248,6 +271,75 @@ jobs: git config --global --add safe.directory '*' source .setup-scripts/setup_env.sh + - name: checksdev test ${{ matrix.check }} + env: + CHECK: ${{ matrix.check }} + run: | + set -eo pipefail + source venv/bin/activate + checksdev test --cov "${CHECK}" + + - name: checksdev benchmarks ${{ matrix.check }} + env: + CHECK: ${{ matrix.check }} + # Non-blocking, matching GitLab's `|| true`: benchmarks are reported for + # information and have never gated a merge. + continue-on-error: true + run: | + set -eo pipefail + source venv/bin/activate + checksdev test "${CHECK}" --bench + + check-tests-private-index: + name: Check tests, private index (${{ matrix.check }}) + # Isolated from `check-tests` because this is the only job that handles a + # credential. `setup_artifact_registry.sh` writes the GitLab Package Registry + # password into a 0600 ~/.netrc, and everything that runs afterwards -- the + # suite's own tests, its tox environment, its transitive dependencies -- can + # read that file. Test code is PR-authored, so the credential is effectively + # exposed to whoever opens the pull request. + # + # Two things contain that (STAC-25463 review): + # * `environment:` puts the job behind a protected GitHub Environment with + # required reviewers, so the credential is only released after a human has + # approved this specific run, having seen the diff. + # * The fork guard stays here, and only here. A fork PR skips this job and + # still gets a full credential-free verdict from the other suites, so + # nothing silently reports green without tests. + if: >- + ${{ (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository) + && needs.select-checks.outputs.private_checks != '[]' }} + needs: select-checks + runs-on: ubuntu-latest + timeout-minutes: 45 + environment: private-package-index + strategy: + fail-fast: false + matrix: + check: ${{ fromJson(needs.select-checks.outputs.private_checks) }} + container: + # Digest-pinned at workflow level (BCI_PYTHON_IMAGE); the ignore is only + # because Zizmor cannot follow the pin through a job output. + image: ${{ needs.select-checks.outputs.image }} # zizmor: ignore[unpinned-images] + steps: + - name: Check out repository + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + with: + persist-credentials: false + + - name: Install toolchain build dependencies + run: | + set -eo pipefail + zypper --non-interactive --gpg-auto-import-keys refresh + # shellcheck disable=SC2086 # deliberately word-split into package args + zypper --non-interactive install ${BCI_BUILD_PACKAGES} + + - name: Build the toolchain virtualenv + run: | + set -eo pipefail + git config --global --add safe.directory '*' + source .setup-scripts/setup_env.sh + - name: Configure the private PyPI index (GitLab Package Registry) env: GITLAB_PACKAGE_REGISTRY_PYPI_SIMPLE_URL: ${{ vars.GITLAB_PACKAGE_REGISTRY_PYPI_SIMPLE_URL }} @@ -257,10 +349,10 @@ jobs: # `checksdev test`. The script writes ~/.pip/pip.conf and a 0600 ~/.netrc, # keeping credentials out of the index URL. # - # `vsphere` is the one phase-1 suite that needs this; the deferred Docker - # suites will too. Guarded rather than unconditional so an absent credential - # cannot fail the suites that do not use it; when it is missing we say so - # loudly rather than failing obscurely later. + # Unguarded, unlike the earlier revision: this job only runs for suites + # that cannot resolve without the private index, so a missing credential + # is a hard configuration error and must fail loudly rather than warn and + # let pip fall through to public PyPI. # # NOTE: writing these under $HOME is only half the job. tox drops every # variable absent from a testenv's `passenv`, HOME included, and pip then @@ -275,8 +367,8 @@ jobs: if [ -z "${GITLAB_PACKAGE_REGISTRY_PYPI_SIMPLE_URL}" ] \ || [ -z "${GITLAB_PACKAGE_REGISTRY_USER}" ] \ || [ -z "${GITLAB_PACKAGE_REGISTRY_READONLY_PASSWORD}" ]; then - echo "::warning title=Private PyPI index not configured::GitLab Package Registry credentials are not available to this job. Any check pinning a wheel that public PyPI does not serve will fail to resolve its requirements." - exit 0 + echo "::error title=Private PyPI index not configured::This suite installs from the GitLab Package Registry, but its credentials are not available to this job." + exit 1 fi .setup-scripts/setup_artifact_registry.sh @@ -291,8 +383,7 @@ jobs: - name: checksdev benchmarks ${{ matrix.check }} env: CHECK: ${{ matrix.check }} - # Non-blocking, matching GitLab's `|| true`: benchmarks are reported for - # information and have never gated a merge. + # Non-blocking, matching GitLab's `|| true`. continue-on-error: true run: | set -eo pipefail @@ -301,12 +392,11 @@ jobs: workflow-security: name: Workflow security scan (Zizmor) - # Same fork boundary as the rest: this repository is PUBLIC and the runners are - # self-hosted, so unreviewed fork code is not checked out here even for a - # read-only audit. Depends on select-checks only to reuse the pinned image. - if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository + # Credential-free and read-only, and it runs on GitHub-hosted infrastructure, + # so fork PRs are audited too. Depends on select-checks only to reuse the + # pinned image. needs: select-checks - runs-on: docker-public + runs-on: ubuntu-latest timeout-minutes: 15 container: # Digest-pinned at workflow level (BCI_PYTHON_IMAGE); the ignore is only @@ -343,42 +433,89 @@ jobs: name: CI success # The single stable status for branch protection. Every other status here is # either dynamically named (the matrix legs are `Check tests ()`) or - # conditional (check-tests is skipped entirely when the selector returns an - # empty array, and everything is skipped for fork PRs). A required check must - # exist on every run, so this job always runs and decides the verdict itself. - # That includes fork PRs, where everything upstream is skipped: without a - # status here a fork PR could never satisfy branch protection. It checks out - # nothing and runs no repository code, so it stays safe to run in that context. + # conditional, so this job always runs and decides the verdict itself. + # + # It must distinguish a legitimate skip from an absent pipeline. An earlier + # revision treated every non-failure as success, which meant a run where + # everything skipped still reported green and could satisfy branch protection + # without executing any CI at all (STAC-25463 review). The rules below are + # therefore explicit about which skips are allowed and why. if: always() needs: - select-checks - validate - check-tests + - check-tests-private-index - workflow-security - runs-on: docker-public + runs-on: ubuntu-latest timeout-minutes: 5 steps: - name: Evaluate upstream job results env: - # `skipped` is a legitimate outcome and must not be conflated with - # success: check-tests skips when no suite was selected, and the whole - # pipeline skips for fork PRs. Only `failure` and `cancelled` fail here. - RESULTS: >- - select-checks=${{ needs.select-checks.result }} - validate=${{ needs.validate.result }} - check-tests=${{ needs.check-tests.result }} - workflow-security=${{ needs.workflow-security.result }} + SELECT_CHECKS: ${{ needs.select-checks.result }} + VALIDATE: ${{ needs.validate.result }} + WORKFLOW_SECURITY: ${{ needs.workflow-security.result }} + CHECK_TESTS: ${{ needs.check-tests.result }} + CHECK_TESTS_PRIVATE: ${{ needs.check-tests-private-index.result }} + SELECTED_CHECKS: ${{ needs.select-checks.outputs.checks }} + SELECTED_PRIVATE_CHECKS: ${{ needs.select-checks.outputs.private_checks }} + IS_FORK: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != github.repository }} run: | set -euo pipefail - failed=0 - for entry in ${RESULTS}; do - echo " ${entry}" - case "${entry#*=}" in - failure | cancelled) failed=1 ;; - esac - done - if [ "${failed}" -ne 0 ]; then - echo "::error title=CI failed::One or more required jobs did not succeed (see above)." + status=0 + + # These three run on every event, including fork PRs. They have no + # legitimate skip: if one did not run, the pipeline did not run. + require_success() { + local name=$1 result=$2 + printf ' %-24s %s\n' "${name}" "${result}" + if [ "${result}" != "success" ]; then + echo "::error title=Required job did not succeed::${name} reported '${result}'." + status=1 + fi + } + + require_success "select-checks" "${SELECT_CHECKS}" + require_success "validate" "${VALIDATE}" + require_success "workflow-security" "${WORKFLOW_SECURITY}" + + # The credential-free matrix may only skip when the selector chose + # nothing. A skip with suites selected means they never ran. + printf ' %-24s %s (selected: %s)\n' "check-tests" "${CHECK_TESTS}" "${SELECTED_CHECKS}" + case "${CHECK_TESTS}" in + success) ;; + skipped) + if [ "${SELECTED_CHECKS}" != "[]" ]; then + echo "::error title=Selected suites never ran::check-tests was skipped while ${SELECTED_CHECKS} was selected." + status=1 + fi + ;; + *) + echo "::error title=Check tests did not succeed::check-tests reported '${CHECK_TESTS}'." + status=1 + ;; + esac + + # The private-index matrix has one extra legitimate skip: fork PRs never + # receive the credential, by design. They still get a full verdict from + # the credential-free suites above, so this cannot hide an empty run. + printf ' %-24s %s (selected: %s, fork: %s)\n' \ + "check-tests-private" "${CHECK_TESTS_PRIVATE}" "${SELECTED_PRIVATE_CHECKS}" "${IS_FORK}" + case "${CHECK_TESTS_PRIVATE}" in + success) ;; + skipped) + if [ "${SELECTED_PRIVATE_CHECKS}" != "[]" ] && [ "${IS_FORK}" != "true" ]; then + echo "::error title=Selected suites never ran::check-tests-private-index was skipped while ${SELECTED_PRIVATE_CHECKS} was selected." + status=1 + fi + ;; + *) + echo "::error title=Private-index tests did not succeed::check-tests-private-index reported '${CHECK_TESTS_PRIVATE}'." + status=1 + ;; + esac + + if [ "${status}" -ne 0 ]; then exit 1 fi - echo "All required jobs succeeded or were legitimately skipped." + echo "All required jobs succeeded; every skip was legitimate." From b1715b40cc53983caab06024c957d05a3ea9a717 Mon Sep 17 00:00:00 2001 From: Louis Parkin Date: Wed, 5 Aug 2026 10:36:59 +0200 Subject: [PATCH 09/11] STAC-25463: correct deferred-scope notes and reference follow-up tickets The header claimed "releases keep running from GitLab, so the GitLab pipeline stays live". That is no longer true: the GitLab project (stackvista/agent/stackstate-agent-integrations, id 13169244) was archived on 2026-07-20, so publish-checks-dev cannot run there either. Publishing a new stackstate_checks_dev is currently impossible on any platform, and the five unported suites have no CI at all. Say so plainly rather than leaving a reviewer with a false premise. Also drop the stale claim that phase 2 rides on an ARC DinD sidecar -- every job now runs on ubuntu-latest, which already provides a Docker daemon, so phase 2 only needs a docker client in the job image. Deferred work now carries ticket references instead of vague "phase 2" / "a follow-up" pointers: STAC-25531 the five Docker-daemon suites STAC-25532 publish-checks-dev STAC-25533 Cerberus failure notification Comments only; no behaviour change. actionlint, shellcheck and zizmor clean, selector smoke-tested. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/scripts/select-checks.sh | 10 ++++++---- .github/workflows/checks-tests.yml | 29 +++++++++++++++++------------ 2 files changed, 23 insertions(+), 16 deletions(-) diff --git a/.github/scripts/select-checks.sh b/.github/scripts/select-checks.sh index 36dcee05..13924722 100755 --- a/.github/scripts/select-checks.sh +++ b/.github/scripts/select-checks.sh @@ -16,7 +16,8 @@ # * Otherwise only the suites whose own directory changed run. # * GitLab's `splunk_base_build_rule` -- a change to splunk_base also runs the # other three splunk suites, which import its test helpers -- is not ported -# here because no splunk suite runs yet. It lands with them in phase 2. +# here because no splunk suite runs yet. It lands with them in phase 2 +# (STAC-25531). # * push / workflow_dispatch run everything (GitLab: `master_branch`, # `release_branch`). # @@ -36,13 +37,14 @@ set -euo pipefail # Suites currently running on GitHub Actions. Phase 1 is the 15 suites that need # no Docker daemon. # -# Deliberately NOT here yet (phase 2, needs the DinD story exercised first): +# Deliberately NOT here yet (phase 2, STAC-25531 -- needs a docker client in the +# job image): # splunk_base, splunk_health, splunk_metric, splunk_topology # -- each drives a real Splunk container via docker-compose. # stackstate_checks_dev # -- its tests exercise the toolkit's own Docker helpers. -# Both public ARC runners provide a DinD sidecar, so this is a matter of proving -# it rather than provisioning anything. +# ubuntu-latest already provides a working Docker daemon, so this is a matter of +# giving the job a docker client rather than provisioning a runner. # # Deliberately dropped, not pending: # postgres -- .gitlab-ci.yml carried a `test_postgres` job for a check that does diff --git a/.github/workflows/checks-tests.yml b/.github/workflows/checks-tests.yml index 2ebc2dfe..5cab099f 100644 --- a/.github/workflows/checks-tests.yml +++ b/.github/workflows/checks-tests.yml @@ -12,17 +12,19 @@ name: Check tests # instead of hiding inside a test job. # # WHAT IS DELIBERATELY NOT HERE -# splunk_{base,health,metric,topology} and stackstate_checks_dev +# splunk_{base,health,metric,topology} and stackstate_checks_dev (STAC-25531) # The only five suites that need a Docker daemon (the four splunk suites # drive a real Splunk container via docker-compose; checks_dev tests the -# toolkit's own Docker helpers). Both public ARC runners ship a DinD -# sidecar, so this is about exercising that path rather than provisioning -# anything -- phase 2. That phase also brings across +# toolkit's own Docker helpers). ubuntu-latest ships a working Docker +# daemon, so phase 2 does not need to provision anything -- but it does +# need a docker client inside the job, which the BCI Python image used +# here does not carry: either a BCI image with docker added, or a service +# container, rather than a return to the private +# python:3.13.14-bookworm runner image. Phase 2 also brings across # .setup-scripts/setup_artifactory_docker.sh (the registry docker login) -# and COMPOSE_HTTP_TIMEOUT, which only those suites need. It will also need -# a docker client, which the BCI Python image used here does not carry -- -# either a BCI image with docker added, or a DinD service container, rather -# than a return to the private python:3.13.14-bookworm runner image. +# and COMPOSE_HTTP_TIMEOUT, which only those suites need, plus GitLab's +# splunk_base_build_rule (a splunk_base change must also run the other +# three, which import its test helpers). # test_postgres # Dead config: .gitlab-ci.yml tests a `postgres` check that does not exist # in this repository. Dropped, not pending. @@ -35,16 +37,19 @@ name: Check tests # but no job has ever referenced it, and Windows is not a supported target, # so the orphaned .setup-scripts/conda_env.ps1 + windows_*.cmd helpers can # be retired with the GitLab pipeline (STAC-25464). -# publish-checks-dev / the runner-image `docker` job +# publish-checks-dev / the runner-image `docker` job (STAC-25532) # Publishing needs write credentials this repo does not hold; pulumi-infra -# schedules integrations' publishing role for its section 7.4. Until then -# releases keep running from GitLab, so the GitLab pipeline stays live. +# schedules integrations' publishing role for its section 7.4. Note that +# the GitLab project was archived on 2026-07-20, so this job can no longer +# run there either -- publishing a new stackstate_checks_dev is currently +# not possible on any platform, and the target registry needs deciding +# (GitLab package registry vs CodeArtifact, cf. STAC-25407). # A Cerberus failure notification # Unlike stackstate-agent, this pipeline has never had one -- there is no # notify job in .gitlab-ci.yml and no .cerberus directory -- so adding it # would be new capability, not a port. It also needs CERBERUS_LAMBDA_URL, # which is a private-visibility org secret and so unreadable from this -# PUBLIC repo without a pulumi-infra grant. Tracked as a follow-up. +# PUBLIC repo without a pulumi-infra grant. Tracked as STAC-25533. # # CREDENTIALS # The container image is SUSE BCI from registry.suse.com, which is public, so From e6d160f1ecf0e7735accdca35cf2607d7c6721bc Mon Sep 17 00:00:00 2001 From: Louis Parkin Date: Wed, 5 Aug 2026 14:56:21 +0200 Subject: [PATCH 10/11] STAC-25540: fetch private-index wheels, then drop the credential The vsphere suite is the only one that needs a credential: it pins vsphere-automation-sdk==1.82.0, an unmodified upstream VMware wheel that VMware withdrew from public PyPI, so it resolves only from our GitLab Package Registry. Public PyPI serves a 0.0.1 placeholder squatting the name, which installs cleanly and fails much later. setup_artifact_registry.sh left the registry password in a 0600 ~/.netrc for the remainder of the job, so the suite's tests, its tox environment and its whole dependency tree could read it. Test code is PR-authored, which made the credential effectively readable by whoever opened the pull request (STAC-25463 review, P1). That was contained by putting the job behind the private-package-index GitHub Environment, releasing the credential only after a human approved each run. It worked, but SHARED_PATHS covers the CI files, so it fired on roughly one commit in six and blocked authors on their own pull requests. Replace it with a narrower fix. fetch_private_wheels.sh writes the netrc, downloads one hardcoded package set into a wheelhouse under RUNNER_TEMP, deletes the netrc, and rewrites ~/.pip/pip.conf to resolve that wheelhouse via find-links. Everything afterwards runs with no credential on disk and no authenticated index configured, and the environment gate is gone. Details that matter: * The package set is hardcoded, not read from the tree: while the credential is on disk, a pull request must not be able to redirect pip. * --only-binary=:all:, because downloading an sdist executes its setup.py and would hand upstream code a readable ~/.netrc. * A system interpreter, never the toolchain virtualenv, whose pip is built by repository code. The path is resolved rather than hardcoded -- bci/python:3.13 ships /usr/bin/python3.13 and no /usr/bin/python3, and `python3` on PATH is a /usr/local/bin shim -- and the result is then rejected if it resolves inside GITHUB_WORKSPACE, which is what makes "system" a guarantee rather than an assumption. * An EXIT trap revokes the credential on the failure paths too, and the script then asserts the netrc is gone and the wheel actually arrived. Rejected: a separate credentialed job publishing the wheels with upload-artifact. Workflow artifacts on a public repository are anonymously downloadable, so that would have amounted to publicly redistributing a vendor artifact VMware withdrew, under no SPDX licence. Residual risk, recorded in the workflow: a pull request that edits the workflow itself can still reach the secret, since pull_request runs the PR's own copy. That holds for every secret in every repository and is contained by review, not CI. What changed is that ordinary test code and its dependencies no longer see it. setup_artifact_registry.sh is untouched; .gitlab-ci.yml still uses it. Verified: shellcheck and actionlint clean, Zizmor clean (no findings). Verified inside the pinned BCI image that the interpreter resolves to /usr/bin/python3.13 and that a workspace-resident interpreter is refused. Verified that a missing argument, missing credentials, a workspace interpreter and a failed download all exit non-zero with no netrc left behind and the password in no file or log; and that find-links in ~/.pip/pip.conf is honoured with no command-line flags, which is what tox relies on via passenv HOME. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/scripts/select-checks.sh | 9 +- .github/workflows/checks-tests.yml | 97 +++++++++-------- .setup-scripts/fetch_private_wheels.sh | 142 +++++++++++++++++++++++++ vsphere/tox.ini | 18 ++-- 4 files changed, 212 insertions(+), 54 deletions(-) create mode 100755 .setup-scripts/fetch_private_wheels.sh diff --git a/.github/scripts/select-checks.sh b/.github/scripts/select-checks.sh index 13924722..1b844442 100755 --- a/.github/scripts/select-checks.sh +++ b/.github/scripts/select-checks.sh @@ -27,10 +27,11 @@ # # The split is a security boundary, not a convenience. The credential-free suites # run on GitHub-hosted runners with no secrets in scope at all, so a fork PR can -# run them safely. The private-index suites need a registry password written to -# ~/.netrc, which any test code executing afterwards can read, so they run behind -# a protected GitHub Environment that requires a human approval first (STAC-25463 -# review). Keeping them in one matrix would hand that credential to every suite. +# run them safely. The private-index suites need a registry password, and any test +# code executing alongside it could read it, so they run in a separate job that +# downloads the private packages into a local wheelhouse and destroys the +# credential before the suite starts (STAC-25463 review, STAC-25540). Keeping them +# in one matrix would hand that credential to every suite. set -euo pipefail diff --git a/.github/workflows/checks-tests.yml b/.github/workflows/checks-tests.yml index 5cab099f..5a48c84f 100644 --- a/.github/workflows/checks-tests.yml +++ b/.github/workflows/checks-tests.yml @@ -71,16 +71,22 @@ name: Check tests # least-privilege by construction; this repo's *publishing* role is still # deferred, per the note above. # -# Scoping it to a single step is not enough, because the script writes it to -# ~/.netrc and the suite's own test code runs afterwards in the same job and can -# read it. So the suites that need it are split into their own job, -# `check-tests-private-index`, behind the `private-package-index` GitHub -# Environment. Required reviewers there mean the credential is released only -# after a human approves that specific run. The remaining suites never see it. +# Scoping it to a single step was not enough on its own, because the old setup +# script wrote it to ~/.netrc and the suite's own test code runs afterwards in +# the same job and could read it. So the suites that need it are split into +# their own job, `check-tests-private-index`, and that job now downloads the +# private packages into a local wheelhouse and deletes the credential before any +# repository code runs (STAC-25540). The remaining suites never see it. +# +# This replaced the `private-package-index` GitHub Environment, whose required +# reviewers released the credential only after a human approved each run. It +# worked, but SHARED_PATHS covers the CI files, so it fired on roughly one +# commit in six and blocked authors on their own pull requests for no gain the +# wheelhouse does not also provide. # # Residual, and deliberately left alone here: the underlying secrets are still # repo- and org-level, so a pull request that edits this workflow could add a -# new reference outside the gated job. That change is visible in the diff and is +# new reference outside that step. That change is visible in the diff and is # what review and the Zizmor audit are for. Making them environment-only secrets # would remove even that path, but the org-level password is shared with other # repos (stackstate-agent among them), so it needs its own change. @@ -298,26 +304,35 @@ jobs: check-tests-private-index: name: Check tests, private index (${{ matrix.check }}) # Isolated from `check-tests` because this is the only job that handles a - # credential. `setup_artifact_registry.sh` writes the GitLab Package Registry - # password into a 0600 ~/.netrc, and everything that runs afterwards -- the - # suite's own tests, its tox environment, its transitive dependencies -- can - # read that file. Test code is PR-authored, so the credential is effectively - # exposed to whoever opens the pull request. + # credential at all: vsphere pins a package that resolves solely from the + # private GitLab Package Registry. + # + # The credential is confined to a single step (STAC-25540). It is written, + # used for one fixed pip download, and destroyed before any repository code + # runs; the suite then installs from the resulting local wheelhouse with + # nothing to authenticate against. That replaces the protected GitHub + # Environment this job used to sit behind, which answered the same STAC-25463 + # review finding by requiring a human to release every run -- and, because + # SHARED_PATHS covers the CI files, fired on roughly one commit in six and + # blocked authors on their own pull requests. + # + # The fork guard stays here, and only here. A fork PR gets no Actions secrets + # on a public repository, so it could not fetch the wheel in any case; it + # skips this job and still gets a full credential-free verdict from the other + # suites, so nothing silently reports green without tests. # - # Two things contain that (STAC-25463 review): - # * `environment:` puts the job behind a protected GitHub Environment with - # required reviewers, so the credential is only released after a human has - # approved this specific run, having seen the diff. - # * The fork guard stays here, and only here. A fork PR skips this job and - # still gets a full credential-free verdict from the other suites, so - # nothing silently reports green without tests. + # Residual risk, stated plainly: a pull request that edits this workflow can + # still reach the secret, because `pull_request` runs the PR's own copy of the + # workflow. That is true of every secret in every repository and is contained + # by review and branch protection, not by CI. What changed is that the + # credential is no longer readable by ordinary test code and its dependency + # tree, which needed no workflow edit at all. if: >- ${{ (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository) && needs.select-checks.outputs.private_checks != '[]' }} needs: select-checks runs-on: ubuntu-latest timeout-minutes: 45 - environment: private-package-index strategy: fail-fast: false matrix: @@ -345,37 +360,33 @@ jobs: git config --global --add safe.directory '*' source .setup-scripts/setup_env.sh - - name: Configure the private PyPI index (GitLab Package Registry) + - name: Fetch private-index wheels and revoke the credential env: GITLAB_PACKAGE_REGISTRY_PYPI_SIMPLE_URL: ${{ vars.GITLAB_PACKAGE_REGISTRY_PYPI_SIMPLE_URL }} GITLAB_PACKAGE_REGISTRY_USER: ${{ secrets.GITLAB_PACKAGE_REGISTRY_USER }} GITLAB_PACKAGE_REGISTRY_READONLY_PASSWORD: ${{ secrets.GITLAB_PACKAGE_REGISTRY_READONLY_PASSWORD }} - # Mirrors what the GitLab `.linux_test` anchor did immediately before - # `checksdev test`. The script writes ~/.pip/pip.conf and a 0600 ~/.netrc, - # keeping credentials out of the index URL. + # The only step in this workflow with a secret in scope. The script writes + # ~/.netrc, downloads one fixed package set, deletes the netrc, and leaves + # ~/.pip/pip.conf pointing at a local wheelhouse. Everything after it -- + # checksdev, tox, the suite's tests and their dependency tree -- runs with + # no credential on disk and no authenticated index configured. # - # Unguarded, unlike the earlier revision: this job only runs for suites - # that cannot resolve without the private index, so a missing credential - # is a hard configuration error and must fail loudly rather than warn and - # let pip fall through to public PyPI. + # It replaces setup_artifact_registry.sh here, which left the netrc in + # place for the rest of the job (STAC-25463 review P1, STAC-25540). That + # script is untouched and still serves the GitLab pipeline definitions. # - # NOTE: writing these under $HOME is only half the job. tox drops every - # variable absent from a testenv's `passenv`, HOME included, and pip then - # resolves `~` from the passwd database rather than the environment. Because - # a container job here runs with HOME=/github/home instead of the account - # default, a testenv that installs from this index must list HOME in its - # passenv or it will silently read a pip.conf that was never written and fall - # back to public PyPI alone -- with no auth error to show for it. See - # vsphere/tox.ini. + # The wheelhouse lives in RUNNER_TEMP rather than the workspace so it + # cannot be mistaken for repository content or swept into a build. + # + # NOTE: pip.conf is read from $HOME, so tox must still pass HOME into the + # testenv. tox drops every variable absent from `passenv`, and pip then + # resolves `~` from the passwd database rather than the environment -- + # which points at the wrong home in a container job, where HOME is + # /github/home. Without it the suite silently falls back to public PyPI + # and installs the 0.0.1 placeholder. See vsphere/tox.ini. run: | set -eo pipefail - if [ -z "${GITLAB_PACKAGE_REGISTRY_PYPI_SIMPLE_URL}" ] \ - || [ -z "${GITLAB_PACKAGE_REGISTRY_USER}" ] \ - || [ -z "${GITLAB_PACKAGE_REGISTRY_READONLY_PASSWORD}" ]; then - echo "::error title=Private PyPI index not configured::This suite installs from the GitLab Package Registry, but its credentials are not available to this job." - exit 1 - fi - .setup-scripts/setup_artifact_registry.sh + .setup-scripts/fetch_private_wheels.sh "${RUNNER_TEMP}/private-wheels" - name: checksdev test ${{ matrix.check }} env: diff --git a/.setup-scripts/fetch_private_wheels.sh b/.setup-scripts/fetch_private_wheels.sh new file mode 100755 index 00000000..f62319df --- /dev/null +++ b/.setup-scripts/fetch_private_wheels.sh @@ -0,0 +1,142 @@ +#!/usr/bin/env bash +# Makes the packages that exist only in the private GitLab Package Registry +# available to a local wheelhouse, and destroys the credential before returning. +# +# Why this exists (STAC-25540): vsphere pins vsphere-automation-sdk==1.82.0, an +# unmodified upstream VMware wheel that VMware withdrew from public PyPI. We +# self-host it in the GitLab Package Registry only because that org was private; +# public PyPI now serves a 0.0.1 placeholder squatting the name. +# +# The predecessor, setup_artifact_registry.sh, left a 0600 ~/.netrc in place for +# the remainder of the job, so every later step -- the tox environment, the +# suite's own tests, their transitive dependencies -- could read the password. +# Test code is PR-authored, which made that credential effectively readable by +# whoever opened the pull request (STAC-25463 review, P1). +# +# Here the credential exists only for the duration of one pip invocation whose +# package set is fixed below, and pip is then pointed at the resulting wheelhouse +# so the rest of the job resolves offline with nothing to authenticate against. +set -euo pipefail + +WHEELHOUSE_ARG="${1:-}" +if [ -z "${WHEELHOUSE_ARG}" ]; then + echo "usage: ${0##*/} " >&2 + exit 2 +fi + +# Absolute: pip.conf's find-links is resolved against the working directory of +# whichever process reads it, and tox runs pip from the suite directory. +mkdir -p "${WHEELHOUSE_ARG}" +WHEELHOUSE="$(cd "${WHEELHOUSE_ARG}" && pwd)" + +# Hardcoded on purpose, and deliberately not read from the working tree. While +# the credential is on disk, a pull request must not be able to redirect pip at a +# package of its choosing. +PRIVATE_REQUIREMENTS=( + "vsphere-automation-sdk==1.82.0" +) + +for var in GITLAB_PACKAGE_REGISTRY_PYPI_SIMPLE_URL GITLAB_PACKAGE_REGISTRY_USER GITLAB_PACKAGE_REGISTRY_READONLY_PASSWORD; do + if [ -z "${!var:-}" ]; then + echo "::error title=Private PyPI index not configured::${var} is not available to this job, but this suite cannot resolve without the private index." + exit 1 + fi +done + +NETRC="${HOME}/.netrc" +PIP_CONF_DIR="${HOME}/.pip" + +revoke_credential() { + rm -f "${NETRC}" +} +# Covers the error paths too: a failed download must not leave the password on a +# disk that PR-authored test code goes on to run against. +trap revoke_credential EXIT + +# Hostname only; the simple URL carries a path after the first '/'. +NETRC_HOST="${GITLAB_PACKAGE_REGISTRY_PYPI_SIMPLE_URL%%/*}" + +umask 077 +cat > "${NETRC}" </dev/null || true)"; do + if [ -n "${candidate}" ] && [ -x "${candidate}" ]; then + PYTHON="${candidate}" + break + fi +done +if [ -z "${PYTHON}" ]; then + echo "::error title=No system interpreter::Could not locate a python3 to download with." + exit 1 +fi +if [ -n "${GITHUB_WORKSPACE:-}" ]; then + PYTHON_DIR="$(cd "$(dirname "${PYTHON}")" && pwd)" + case "${PYTHON_DIR}/" in + "${GITHUB_WORKSPACE%/}/"*) + echo "::error title=Refusing a workspace interpreter::Resolved python3 at ${PYTHON}, which is inside the checkout and therefore PR-controlled." + exit 1 + ;; + esac +fi + +echo "→ Downloading private-index packages into ${WHEELHOUSE}" +printf ' %s\n' "${PRIVATE_REQUIREMENTS[@]}" +echo " using ${PYTHON}" + +# --only-binary=:all: matters as much as the interpreter choice. Downloading an +# sdist executes its setup.py, so allowing one would hand arbitrary upstream code +# a process with the registry password readable at ~/.netrc. +"${PYTHON}" -m pip download \ + --disable-pip-version-check \ + --no-cache-dir \ + --only-binary=:all: \ + --extra-index-url "https://${GITLAB_PACKAGE_REGISTRY_PYPI_SIMPLE_URL}" \ + --dest "${WHEELHOUSE}" \ + "${PRIVATE_REQUIREMENTS[@]}" + +revoke_credential +trap - EXIT + +if [ -f "${NETRC}" ]; then + echo "::error title=Credential not revoked::${NETRC} still exists after download; refusing to continue." + exit 1 +fi + +# A silent miss here would fall through to public PyPI and install the 0.0.1 +# placeholder, which fails much later and far less legibly. +if ! find "${WHEELHOUSE}" -maxdepth 1 -iname 'vsphere_automation_sdk-*.whl' | grep -q .; then + echo "::error title=Private wheel missing::vsphere-automation-sdk was not downloaded into ${WHEELHOUSE}." + exit 1 +fi + +# Replaces the extra-index-url that setup_artifact_registry.sh used to write. +# Nothing after this point authenticates anywhere: the private packages resolve +# from the local wheelhouse, and everything else still comes from public PyPI. +mkdir -p "${PIP_CONF_DIR}" +cat > "${PIP_CONF_DIR}/pip.conf" < Date: Wed, 5 Aug 2026 16:12:17 +0200 Subject: [PATCH 11/11] STAC-25540: keep the private-index credential off the pull request path The previous revision confined the GitLab package registry credential to a single step and claimed that reaching it would require editing the workflow. That claim was wrong. A `pull_request` run executes the pull request's own copy of the workflow *and* of every script it calls, so a pull request could rewrite fetch_private_wheels.sh, reorder the steps, or add one of its own. Confining a secret inside pull-request-controlled code is hardening, not a boundary. So the private-index suites no longer run on pull requests at all. They run on push, tag and workflow_dispatch events, whose contents are reviewed before they reach the release branch, and where the secret is therefore not exposed to unreviewed code. * select-checks.sh moves private-index suites to a new `deferred_private_checks` output on pull requests, so the matrix is empty rather than gated, and the reason is reported. * check-tests-private-index is conditioned on the event, replacing the fork guard that is now redundant. * fetch_private_wheels.sh refuses to run on a pull_request event, so the rule survives an edit to the workflow condition. * ci-success reports the deferral, so a green pull request never implies vsphere was covered when it was not. The cost is that vsphere is verified on the release branch rather than on the pull request that changes it. That is accepted knowingly: the suite changes a handful of times a year, and the alternatives cost more than they return. The durable fix is to stop needing the private index -- VMware now publishes this SDK to public PyPI under renamed packages -- which deletes the credential, the job and this gap together. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/scripts/select-checks.sh | 54 ++++++++--- .github/workflows/checks-tests.yml | 127 +++++++++++++++---------- .setup-scripts/fetch_private_wheels.sh | 30 ++++-- 3 files changed, 139 insertions(+), 72 deletions(-) diff --git a/.github/scripts/select-checks.sh b/.github/scripts/select-checks.sh index 1b844442..6efeed1a 100755 --- a/.github/scripts/select-checks.sh +++ b/.github/scripts/select-checks.sh @@ -21,17 +21,25 @@ # * push / workflow_dispatch run everything (GitLab: `master_branch`, # `release_branch`). # -# Writes two arrays to $GITHUB_OUTPUT for `fromJson()` in a matrix: -# checks -- suites that need no credentials -# private_checks -- suites that install from the private GitLab PyPI index +# Writes three arrays to $GITHUB_OUTPUT for `fromJson()` in a matrix: +# checks -- suites that need no credentials +# private_checks -- suites that install from the private GitLab PyPI +# index, and are cleared to run on this event +# deferred_private_checks -- private-index suites withheld from this event +# (always empty outside pull requests) # # The split is a security boundary, not a convenience. The credential-free suites -# run on GitHub-hosted runners with no secrets in scope at all, so a fork PR can -# run them safely. The private-index suites need a registry password, and any test -# code executing alongside it could read it, so they run in a separate job that -# downloads the private packages into a local wheelhouse and destroys the -# credential before the suite starts (STAC-25463 review, STAC-25540). Keeping them -# in one matrix would hand that credential to every suite. +# run with no secrets in scope at all. The private-index suites need a registry +# password, so they are kept in a separate job -- and, on pull requests, are not +# run at all (STAC-25540, second review pass). +# +# That last part is the whole point, so it is worth stating plainly: a +# `pull_request` run executes the pull request's own copy of the workflow and of +# every script it calls. Hardening the job cannot keep a determined pull request +# away from a secret the run is holding -- it can always edit the thing that holds +# it. The only run that cannot leak the credential is a run that never receives +# it, so these suites are deferred to push, tag and workflow_dispatch events, +# whose contents are reviewed before they reach the release branch. set -euo pipefail @@ -74,8 +82,12 @@ CHECKS=( # mirrored into the StackVista package registry and needs authentication. # # Everything not listed here is credential-free and must stay that way: adding a -# suite to this list moves it behind a manual approval gate, and removing the -# need for the private index is always the better fix. +# suite to this list stops it running on pull requests altogether, and removing +# the need for the private index is always the better fix. For vsphere that fix +# looks reachable -- VMware now publishes the SDK to public PyPI under renamed +# packages (vmware-vapi-runtime, vmware-vapi-common-client, pyvmomi) and ships +# the NSX/VMC wheels from its own public index -- so this list should shrink to +# nothing once the pin is modernised. PRIVATE_INDEX_CHECKS=( vsphere ) @@ -110,7 +122,7 @@ is_private_index() { emit() { local -a selected=("$@") - local -a public=() private=() + local -a public=() private=() deferred=() local check for check in ${selected[@]+"${selected[@]}"}; do if is_private_index "${check}"; then @@ -120,17 +132,33 @@ emit() { fi done - local public_json private_json + # Pull requests do not run the private-index suites at all (STAC-25540, second + # review pass). See the security-boundary note at the top of this file: a + # `pull_request` run executes the pull request's own copy of the workflow and + # scripts, so the credential can only be protected by withholding it. These + # suites run on the release branch instead, where the code has been reviewed. + if [ "${EVENT_NAME}" = "pull_request" ] && [ "${#private[@]}" -gt 0 ]; then + deferred=("${private[@]}") + private=() + fi + + local public_json private_json deferred_json public_json=$(to_json ${public[@]+"${public[@]}"}) private_json=$(to_json ${private[@]+"${private[@]}"}) + deferred_json=$(to_json ${deferred[@]+"${deferred[@]}"}) { echo "checks=${public_json}" echo "private_checks=${private_json}" + echo "deferred_private_checks=${deferred_json}" } >>"${GITHUB_OUTPUT}" echo "Selected credential-free suites: ${public_json}" echo "Selected private-index suites: ${private_json}" + if [ "${deferred_json}" != "[]" ]; then + echo "Deferred private-index suites: ${deferred_json}" + echo "::notice title=Private-index suites do not run on pull requests::${deferred_json} resolve only against the private package registry. Pull requests are deliberately given no credential to reach it, so these suites run on ${BASE_REF:-the release branch} after merge." + fi } # Anything that is not a pull request is a full run. On the release branch the diff --git a/.github/workflows/checks-tests.yml b/.github/workflows/checks-tests.yml index 5a48c84f..a2495c5d 100644 --- a/.github/workflows/checks-tests.yml +++ b/.github/workflows/checks-tests.yml @@ -71,33 +71,46 @@ name: Check tests # least-privilege by construction; this repo's *publishing* role is still # deferred, per the note above. # -# Scoping it to a single step was not enough on its own, because the old setup -# script wrote it to ~/.netrc and the suite's own test code runs afterwards in -# the same job and could read it. So the suites that need it are split into -# their own job, `check-tests-private-index`, and that job now downloads the -# private packages into a local wheelhouse and deletes the credential before any -# repository code runs (STAC-25540). The remaining suites never see it. +# No pull request ever receives it. `pull_request` runs execute the pull +# request's own copy of this workflow and of every script it calls, so a run +# that holds a secret can always be made to disclose it -- by editing the +# fetching script, reordering steps, or adding one. A repository secret and +# pull-request-controlled code cannot be arranged into a boundary. The suites +# that need this credential therefore do not run on pull requests at all; they +# run on push, tag and workflow_dispatch events, whose contents are reviewed +# before reaching the release branch. See `check-tests-private-index`. # -# This replaced the `private-package-index` GitHub Environment, whose required -# reviewers released the credential only after a human approved each run. It -# worked, but SHARED_PATHS covers the CI files, so it fired on roughly one -# commit in six and blocked authors on their own pull requests for no gain the -# wheelhouse does not also provide. +# Within those runs the credential is still confined to a single step +# (STAC-25540): the script writes ~/.netrc, downloads one fixed package set into +# a local wheelhouse, deletes the netrc, and points pip at the wheelhouse, so +# the suite and its dependency tree install with nothing to authenticate +# against. The predecessor left the netrc readable for the rest of the job. That +# is defence in depth, not the boundary -- the boundary is the event condition +# above. # -# Residual, and deliberately left alone here: the underlying secrets are still -# repo- and org-level, so a pull request that edits this workflow could add a -# new reference outside that step. That change is visible in the diff and is -# what review and the Zizmor audit are for. Making them environment-only secrets -# would remove even that path, but the org-level password is shared with other -# repos (stackstate-agent among them), so it needs its own change. +# Two earlier answers to the same review finding are recorded here so they are +# not re-proposed. A `private-package-index` GitHub Environment with required +# reviewers did gate the credential, but SHARED_PATHS covers the CI files, so it +# fired on roughly one commit in six and blocked authors on their own pull +# requests -- while only ever constraining people who already have write access. +# Prefetching the wheelhouse on a trusted event and passing it to pull requests +# through the Actions cache also works, but any pull request can read a cache, +# and a pull request can only restore one from its base branch. +# +# The durable fix is to stop needing the index: VMware now publishes this SDK to +# public PyPI under renamed packages (vmware-vapi-runtime, +# vmware-vapi-common-client, pyvmomi) and serves the NSX/VMC wheels from its own +# public index, so modernising the pin removes the credential, this job and the +# pull-request coverage gap in one change. # # RUNNERS # Everything runs on GitHub-hosted runners. The suites are pure-Python and need # no Docker daemon, so the self-hosted docker-public pool bought nothing while # costing real isolation: fork PRs had to be excluded from it, which in turn -# meant a fork could never produce a CI verdict. On hosted runners forks run the -# full credential-free matrix, and only the private-index job keeps a fork guard -# -- forks receive no secrets in any case. This also removes the question of +# meant a fork could never produce a CI verdict. On hosted runners forks run +# exactly the same matrix as any other pull request -- the private-index job is +# off the pull-request path entirely, so no fork-specific guard is needed for +# it any more. This also removes the question of # pulling upstream images across the self-hosted NAT: the BCI reference is # direct, from a public registry, on infrastructure that is meant to reach it. # The phase-2 Docker suites will need a runner with a daemon; that decision @@ -156,16 +169,17 @@ env: jobs: select-checks: name: Select check suites to run - # Runs for forks too. Every job in this workflow now runs on GitHub-hosted - # runners with no secrets in scope, except the private-index suites, which are - # isolated behind a protected environment below. There is therefore nothing a - # fork branch can reach here, and blocking forks outright would leave them - # unable to satisfy branch protection at all (STAC-25463 review). + # Runs for forks too. Every job that runs on a pull request does so with no + # secrets in scope at all -- the one job that uses a credential does not run + # on pull requests (see `check-tests-private-index`). There is therefore + # nothing a fork branch can reach here, and blocking forks outright would + # leave them unable to satisfy branch protection at all (STAC-25463 review). runs-on: ubuntu-latest timeout-minutes: 10 outputs: checks: ${{ steps.select.outputs.checks }} private_checks: ${{ steps.select.outputs.private_checks }} + deferred_private_checks: ${{ steps.select.outputs.deferred_private_checks }} image: ${{ steps.image.outputs.ref }} steps: - name: Check out repository @@ -307,28 +321,33 @@ jobs: # credential at all: vsphere pins a package that resolves solely from the # private GitLab Package Registry. # - # The credential is confined to a single step (STAC-25540). It is written, - # used for one fixed pip download, and destroyed before any repository code - # runs; the suite then installs from the resulting local wheelhouse with - # nothing to authenticate against. That replaces the protected GitHub - # Environment this job used to sit behind, which answered the same STAC-25463 - # review finding by requiring a human to release every run -- and, because - # SHARED_PATHS covers the CI files, fired on roughly one commit in six and - # blocked authors on their own pull requests. + # This job does not run on pull requests (STAC-25540, second review pass). + # + # An earlier revision ran it on pull requests with the credential confined to + # a single step, and claimed that reaching it would require editing this + # workflow. That claim was wrong, and the review was right to call it: a + # `pull_request` run executes the pull request's own copy of the workflow AND + # of every script it calls, so a pull request could rewrite the fetch script, + # reorder these steps, or simply add a step of its own. Repository secrets + # plus pull-request-controlled code do not make a security boundary, however + # carefully the code in between is written. # - # The fork guard stays here, and only here. A fork PR gets no Actions secrets - # on a public repository, so it could not fetch the wheel in any case; it - # skips this job and still gets a full credential-free verdict from the other - # suites, so nothing silently reports green without tests. + # Confining the credential to one step is still worth doing and is still done + # -- it keeps the password away from the suite's dependency tree, which needed + # no malice at all to read it -- but it is hardening, not a boundary. The + # boundary is this condition: the run simply never receives the secret. # - # Residual risk, stated plainly: a pull request that edits this workflow can - # still reach the secret, because `pull_request` runs the PR's own copy of the - # workflow. That is true of every secret in every repository and is contained - # by review and branch protection, not by CI. What changed is that the - # credential is no longer readable by ordinary test code and its dependency - # tree, which needed no workflow edit at all. + # The cost is that vsphere is verified on the release branch rather than on + # the pull request that changes it. That is a real gap, accepted knowingly: + # the suite's own directory changes a handful of times a year, and the + # alternatives all cost more than they return right now. Sharing a prefetched + # wheelhouse through the Actions cache would work, but a pull request run can + # only restore caches from its base branch, and any pull request can read + # them. The durable fix is to stop needing the private index at all: VMware + # now publishes this SDK to public PyPI under renamed packages, so modernising + # the pin deletes this job, its credential and this trade-off together. if: >- - ${{ (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository) + ${{ github.event_name != 'pull_request' && needs.select-checks.outputs.private_checks != '[]' }} needs: select-checks runs-on: ubuntu-latest @@ -475,7 +494,7 @@ jobs: CHECK_TESTS_PRIVATE: ${{ needs.check-tests-private-index.result }} SELECTED_CHECKS: ${{ needs.select-checks.outputs.checks }} SELECTED_PRIVATE_CHECKS: ${{ needs.select-checks.outputs.private_checks }} - IS_FORK: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != github.repository }} + DEFERRED_PRIVATE_CHECKS: ${{ needs.select-checks.outputs.deferred_private_checks }} run: | set -euo pipefail status=0 @@ -512,15 +531,17 @@ jobs: ;; esac - # The private-index matrix has one extra legitimate skip: fork PRs never - # receive the credential, by design. They still get a full verdict from - # the credential-free suites above, so this cannot hide an empty run. - printf ' %-24s %s (selected: %s, fork: %s)\n' \ - "check-tests-private" "${CHECK_TESTS_PRIVATE}" "${SELECTED_PRIVATE_CHECKS}" "${IS_FORK}" + # The private-index matrix legitimately skips whenever the selector + # chose nothing for this event. On pull requests that is always: those + # suites are deferred rather than selected, because the run holds no + # credential to reach the private index with. The deferral is reported + # so a green pull request never quietly implies vsphere was covered. + printf ' %-24s %s (selected: %s, deferred: %s)\n' \ + "check-tests-private" "${CHECK_TESTS_PRIVATE}" "${SELECTED_PRIVATE_CHECKS}" "${DEFERRED_PRIVATE_CHECKS}" case "${CHECK_TESTS_PRIVATE}" in success) ;; skipped) - if [ "${SELECTED_PRIVATE_CHECKS}" != "[]" ] && [ "${IS_FORK}" != "true" ]; then + if [ "${SELECTED_PRIVATE_CHECKS}" != "[]" ]; then echo "::error title=Selected suites never ran::check-tests-private-index was skipped while ${SELECTED_PRIVATE_CHECKS} was selected." status=1 fi @@ -531,6 +552,10 @@ jobs: ;; esac + if [ "${DEFERRED_PRIVATE_CHECKS}" != "[]" ]; then + echo "::notice title=Not covered by this run::${DEFERRED_PRIVATE_CHECKS} need the private package registry and do not run on pull requests. They run on the release branch after merge." + fi + if [ "${status}" -ne 0 ]; then exit 1 fi diff --git a/.setup-scripts/fetch_private_wheels.sh b/.setup-scripts/fetch_private_wheels.sh index f62319df..f138ce66 100755 --- a/.setup-scripts/fetch_private_wheels.sh +++ b/.setup-scripts/fetch_private_wheels.sh @@ -7,17 +7,31 @@ # self-host it in the GitLab Package Registry only because that org was private; # public PyPI now serves a 0.0.1 placeholder squatting the name. # -# The predecessor, setup_artifact_registry.sh, left a 0600 ~/.netrc in place for -# the remainder of the job, so every later step -- the tox environment, the -# suite's own tests, their transitive dependencies -- could read the password. -# Test code is PR-authored, which made that credential effectively readable by -# whoever opened the pull request (STAC-25463 review, P1). +# This script only ever runs on events whose contents have been reviewed -- push, +# tag and workflow_dispatch. It does NOT run on pull requests, and the guard below +# enforces that independently of the workflow, because a pull request can edit the +# workflow as freely as it can edit this file. That is the actual protection for +# the credential; everything else here is defence in depth (STAC-25540, second +# review pass). # -# Here the credential exists only for the duration of one pip invocation whose -# package set is fixed below, and pip is then pointed at the resulting wheelhouse -# so the rest of the job resolves offline with nothing to authenticate against. +# The defence in depth still matters. The predecessor, setup_artifact_registry.sh, +# left a 0600 ~/.netrc in place for the remainder of the job, so every later step +# -- the tox environment, the suite's own tests, their transitive dependencies -- +# could read the password. That needed no malice from anyone. Here the credential +# exists only for the duration of one pip invocation whose package set is fixed +# below, and pip is then pointed at the resulting wheelhouse so the rest of the +# job resolves offline with nothing to authenticate against. set -euo pipefail +# A pull request must never reach the registry password, and must not be able to +# arrange for this script to fetch it one. The workflow already declines to run +# the job on pull requests; this is the same rule stated where it cannot be +# removed by editing a YAML condition. +if [ "${GITHUB_EVENT_NAME:-}" = "pull_request" ]; then + echo "::error title=Refusing to fetch on a pull request::${0##*/} handles the private registry credential and must not run on pull_request events; the private-index suites run on the release branch instead." + exit 1 +fi + WHEELHOUSE_ARG="${1:-}" if [ -z "${WHEELHOUSE_ARG}" ]; then echo "usage: ${0##*/} " >&2