From 5c91ebec92c119dd2ac25bcd73f1ad4724c4fd44 Mon Sep 17 00:00:00 2001 From: Louis Parkin Date: Thu, 6 Aug 2026 11:47:05 +0200 Subject: [PATCH 1/6] STAC-25533: report release-branch CI failures to Slack via Cerberus This pipeline has never had a failure notification. Its retired .gitlab-ci.yml had no notify job and no .cerberus directory, so a broken release branch has always been silent here. STAC-25510 is what that costs: stackstate-process-agent's image publishing broke on 2026-07-23 and nobody noticed for 12 days. Adds a reusable cerberus-notify.yml, structurally identical to the one stackstate-process-agent gained in STAC-25519, and calls it as a terminal job from checks-tests.yml. `action: notify`, never `block`. Blocking locks the branch, additionally requires the Cerberus GitHub App to be installed here, and mutates the branch protection pulumi-infra now owns (STAC-25522) -- a later pulumi apply would silently unlock the branch again. Push events only, which given the existing `push:` trigger means the release branch and release tags. Pull request failures already have an owner watching them, and on a public repo anyone can open a PR that fails on purpose, so notifying on them would hand the CI channel a spam vector. The condition is keyed on `github.event_name` rather than a hardcoded ref so the next release-branch bump only has to touch the `push:` trigger, not two places that must agree. Hung off ci-success rather than the individual jobs so there is one funnel for all failures instead of a notify job per pipeline job. Two departures from the process-agent original: - The header records this as new capability rather than a port. - COMMIT_TITLE falls back to the sha. Tag pushes carry no `head_commit`, and this workflow runs on them; process-agent's copy is push-to-master only and never hits that case. Both secrets are declared `required: false` and guarded at runtime: a caller passing `${{ secrets.X }}` for a secret the repo does not hold yields an empty string, which GitHub rejects against a required secret and fails the call before the guard can warn. Missing configuration warns and exits 0 rather than adding a second red job to an already-failed run. The repo-level secret grant is StackVista/pulumi-infra#277 -- the org copies are visibility=private and this repo is public. Until that merges this job warns and passes. Validated: actionlint (with shellcheck) and Zizmor 1.28.0 clean; payload construction dry-run produces valid JSON for both a normal commit and the empty-head_commit tag case. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/cerberus-notify.yml | 122 ++++++++++++++++++++++++++ .github/workflows/checks-tests.yml | 32 +++++-- 2 files changed, 149 insertions(+), 5 deletions(-) create mode 100644 .github/workflows/cerberus-notify.yml diff --git a/.github/workflows/cerberus-notify.yml b/.github/workflows/cerberus-notify.yml new file mode 100644 index 00000000..2b08e387 --- /dev/null +++ b/.github/workflows/cerberus-notify.yml @@ -0,0 +1,122 @@ +name: Cerberus notify + +# New capability, not a port (STAC-25142 / STAC-25533). Unlike stackstate-agent +# and stackstate-process-agent, this repo's retired .gitlab-ci.yml had no notify +# job and no .cerberus directory, so a failed release-branch pipeline has always +# been silent here. STAC-25510 is what that costs: process-agent's image +# publishing broke on 2026-07-23 and went unnoticed for 12 days. +# +# Structure and calling convention follow +# stackstate-process-agent/.github/workflows/cerberus-notify.yml, which in turn +# follows `cerberus-block-on-master-fail` in StackVista/stackstate. Cerberus is +# the internal notify/block Lambda (source: https://github.com/StackVista/cerberus). +# `platform: github` makes it build GitHub pipeline/commit URLs rather than +# GitLab ones. +# +# `action: notify`, never `action: block`. Policy for migrated repos is notify by +# default. Blocking locks the branch (`lock_branch`), additionally requires the +# Cerberus GitHub App to be installed here, and mutates branch protection that +# pulumi-infra owns (STAC-25522) out from under it -- a subsequent pulumi apply +# would silently unlock the branch again. +# +# Prerequisites: CERBERUS_LAMBDA_URL and CERBERUS_API_TOKEN must both reach this +# repo as REPO-level secrets. The org-level copies are visibility=private, which +# excludes this PUBLIC repo. pulumi-infra provisions the pair together +# (github/repoVariables/resources.yaml, StackVista/pulumi-infra#277). If either +# is missing, this workflow warns and exits 0 rather than adding a second red job +# to an already-failed run -- the annotation is the signal. +# +# The bearer token is not optional going forward. StackVista/cerberus#4 +# (STAC-24889) adds `Authorization: Bearer ` verification to every +# non-Slack request; before it, the endpoint was entirely unauthenticated. +# Sending the header is forward-compatible -- the currently deployed Lambda +# ignores unknown headers -- so this works either side of that deploy. Without +# it, the first failure after cerberus#4 ships would get a 401 and no Slack +# message. +# +# The Slack channel is deliberately not sent. Cerberus resolves it as +# `util.GetOrDefault(req.Context, "channel", s.Channel)`, and GetOrDefault treats +# an empty or whitespace value as absent, so omitting `channel` falls back to the +# Lambda's own SLACK_CHANNEL. + +on: + workflow_call: + inputs: + suite: + description: Suite label shown in the Slack message, e.g. checks. + required: true + type: string + secrets: + # `required: false`. A caller passing `${{ secrets.X }}` for a secret the + # repo does not hold yields an empty string, which GitHub rejects as "not + # provided" against a required secret and fails the call before the run + # step's guard can warn -- the failure mode this workflow exists to avoid. + CERBERUS_LAMBDA_URL: + required: false + CERBERUS_API_TOKEN: + required: false + +# Nothing here reads the repository; the payload is built entirely from the +# github context. +permissions: {} + +jobs: + notify: + name: Notify Slack via Cerberus + runs-on: ubuntu-24.04 + timeout-minutes: 5 + steps: + - name: Post the failure to Cerberus + env: + CERBERUS_LAMBDA_URL: ${{ secrets.CERBERUS_LAMBDA_URL }} + CERBERUS_API_TOKEN: ${{ secrets.CERBERUS_API_TOKEN }} + REPOSITORY: ${{ github.repository }} + BRANCH: ${{ github.ref_name }} + PIPELINE: ${{ github.run_id }} + COMMIT_SHA: ${{ github.sha }} + # Empty on tag pushes, which carry no head_commit. COMMIT_TITLE below + # falls back to the sha so the Slack message is never blank. + COMMIT_MESSAGE: ${{ github.event.head_commit.message }} + SUITE: ${{ inputs.suite }} + run: | + set -euo pipefail + + if [ -z "${CERBERUS_LAMBDA_URL}" ] || [ -z "${CERBERUS_API_TOKEN}" ]; then + echo "::warning title=Cerberus not configured::CERBERUS_LAMBDA_URL and/or CERBERUS_API_TOKEN is not visible to this repo, so the ${SUITE} failure was not reported to Slack. Both are provisioned as repo-level secrets by pulumi-infra (STAC-25533)." + exit 0 + fi + + COMMIT_TITLE=$(printf '%s' "${COMMIT_MESSAGE}" | head -n1) + if [ -z "${COMMIT_TITLE}" ]; then + COMMIT_TITLE="${COMMIT_SHA}" + fi + + # Not --verbose: it echoes request headers, and the Authorization + # header carries the shared token. GitHub would mask it, but not + # emitting it is better than relying on masking. + curl --fail --silent --show-error \ + -X POST "${CERBERUS_LAMBDA_URL}" \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer ${CERBERUS_API_TOKEN}" \ + -d "$(jq -n \ + --arg repo "${REPOSITORY}" \ + --arg branch "${BRANCH}" \ + --arg pipeline "${PIPELINE}" \ + --arg sha "${COMMIT_SHA}" \ + --arg title "${COMMIT_TITLE}" \ + --arg suite "${SUITE}" \ + '{ + action: "notify", + context: { + platform: "github", + "project.id": $repo, + "project.slug": $repo, + "project.name": "StackState Agent Integrations", + branch: $branch, + pipeline: $pipeline, + "commit.sha": $sha, + "commit.title": $title, + suite: $suite + } + }' + )" diff --git a/.github/workflows/checks-tests.yml b/.github/workflows/checks-tests.yml index a2495c5d..1691d3f3 100644 --- a/.github/workflows/checks-tests.yml +++ b/.github/workflows/checks-tests.yml @@ -45,11 +45,8 @@ name: Check tests # not possible on any platform, and the target registry needs deciding # (GitLab package registry vs CodeArtifact, cf. STAC-25407). # A Cerberus failure notification -# Unlike stackstate-agent, this pipeline has never had one -- there is no -# notify job in .gitlab-ci.yml and no .cerberus directory -- so adding it -# would be new capability, not a port. It also needs CERBERUS_LAMBDA_URL, -# which is a private-visibility org secret and so unreadable from this -# PUBLIC repo without a pulumi-infra grant. Tracked as STAC-25533. +# Delivered in STAC-25533; see the `cerberus-notify` job at the end of this +# file and .github/workflows/cerberus-notify.yml. # # CREDENTIALS # The container image is SUSE BCI from registry.suse.com, which is public, so @@ -560,3 +557,28 @@ jobs: exit 1 fi echo "All required jobs succeeded; every skip was legitimate." + + cerberus-notify: + name: Report failure to Slack (Cerberus) + # Terminal job. ci-success already aggregates every other job, so hanging the + # notification off it gives one funnel for all failures rather than a notify + # job per pipeline job. + # + # Push events only, which given the `push:` trigger above means the release + # branch and release tags. Deliberately not pull requests: those failures + # already have an owner watching them, and a public repo would let anyone + # open a PR that fails on purpose to spam the CI channel. Post-merge and tag + # failures are the ones with nobody watching -- exactly the gap that let + # STAC-25510 sit unnoticed for 12 days. + # + # Keyed on `github.event_name` rather than a hardcoded ref so the next + # release-branch bump only has to update the `push:` trigger above, not two + # places that must agree. + needs: ci-success + if: failure() && github.event_name == 'push' + uses: ./.github/workflows/cerberus-notify.yml + with: + suite: checks + secrets: + CERBERUS_LAMBDA_URL: ${{ secrets.CERBERUS_LAMBDA_URL }} + CERBERUS_API_TOKEN: ${{ secrets.CERBERUS_API_TOKEN }} From 246a3f15386bc329ffa2c74c26ec49c98a3b2bd7 Mon Sep 17 00:00:00 2001 From: Louis Parkin Date: Thu, 6 Aug 2026 15:33:01 +0200 Subject: [PATCH 2/6] STAC-25531: run the five Docker-daemon test suites in GitHub Actions Phase 2 of the GitLab->GitHub CI migration. The four splunk_* suites and stackstate_checks_dev need a live Docker daemon, so phase 1 left them out entirely -- they currently run nowhere. They cannot run inside the BCI job container. get_docker_hostname() resolves to `localhost` unless DOCKER_HOST is set, and compose publishes its ports on the Docker host, so the test process must share the daemon's network namespace. GitLab avoided this with a docker:dind service whose hostname resolved identically from both sides; on GitHub the equivalent is to run directly on the runner. Hence a separate `check-tests-docker` job with no container and actions/setup-python, rather than extending the existing matrix. The fixtures pulled their images through registry.tooling.stackstate.io, which answers 401 from the public internet -- a credential a public repo's pull_request jobs must never hold. All five paths were /docker/ proxy-cache entries, so they are repointed at the unambiguous Docker Hub originals via ${SPLUNK_IMAGE}/${VAULT_IMAGE} with public defaults, leaving self-hosted and local users able to override back to the proxy. The selector gains a DOCKER_CHECKS set and a third output. splunk_base is a build dependency of the other three splunk suites, so a change to it fans out to all four; the reverse does not hold. setup_artifactory_docker.sh is deliberately not ported -- it existed only to log Docker in to that proxy. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/scripts/select-checks.sh | 74 +++++++++-- .github/workflows/checks-tests.yml | 122 +++++++++++++++--- splunk_base/tests/compose/docker-compose.yaml | 8 +- .../tests/compose/docker-compose.yaml | 10 +- .../tests/compose/docker-compose.yaml | 8 +- .../tests/compose/docker-compose.yaml | 8 +- .../tests/docker/test_default.yaml | 6 +- 7 files changed, 202 insertions(+), 34 deletions(-) diff --git a/.github/scripts/select-checks.sh b/.github/scripts/select-checks.sh index 6efeed1a..92860958 100755 --- a/.github/scripts/select-checks.sh +++ b/.github/scripts/select-checks.sh @@ -43,17 +43,7 @@ set -euo pipefail -# Suites currently running on GitHub Actions. Phase 1 is the 15 suites that need -# no Docker daemon. -# -# Deliberately NOT here yet (phase 2, STAC-25531 -- needs a docker client in the -# job image): -# splunk_base, splunk_health, splunk_metric, splunk_topology -# -- each drives a real Splunk container via docker-compose. -# stackstate_checks_dev -# -- its tests exercise the toolkit's own Docker helpers. -# ubuntu-latest already provides a working Docker daemon, so this is a matter of -# giving the job a docker client rather than provisioning a runner. +# Suites currently running on GitHub Actions. # # Deliberately dropped, not pending: # postgres -- .gitlab-ci.yml carried a `test_postgres` job for a check that does @@ -69,13 +59,47 @@ CHECKS=( kubelet openmetrics servicenow + splunk_base + splunk_health + splunk_metric + splunk_topology stackstate_checks_base + stackstate_checks_dev static_health static_topology vsphere zabbix ) +# Suites that need a real Docker daemon: the four splunk suites drive a Splunk +# container through docker-compose, and stackstate_checks_dev tests the toolkit's +# own Docker helpers (STAC-25531). +# +# These run as their own matrix directly on the runner, not inside the BCI +# container the other suites use. That is not a preference -- the tests resolve +# their target host through `get_docker_hostname()`, which reads DOCKER_HOST and +# falls back to `localhost`. Compose publishes its ports on the Docker host, so +# `localhost` is correct only when the test process shares a network namespace +# with the daemon. Inside a job container it would resolve to the container +# itself and every connection would be refused. GitLab avoided this by pointing +# DOCKER_HOST at a `docker:dind` service, whose hostname then resolved for both. +DOCKER_CHECKS=( + splunk_base + splunk_health + splunk_metric + splunk_topology + stackstate_checks_dev +) + +# splunk_health, splunk_metric and splunk_topology all build on splunk_base, so a +# change there has to run all four. Ported from the `splunk_base_build_rule` +# anchor in .gitlab-ci.yml, which added the same fan-out to every splunk job. +SPLUNK_DEPENDENTS=( + splunk_health + splunk_metric + splunk_topology +) + # Suites whose requirements resolve only against the private GitLab PyPI index. # `vsphere` pins vsphere-automation-sdk, which VMware never published to public # PyPI (the name is squatted there by an unrelated 0.0.1 placeholder), so it is @@ -120,13 +144,23 @@ is_private_index() { return 1 } +is_docker() { + local candidate=$1 check + for check in "${DOCKER_CHECKS[@]}"; do + [ "${candidate}" = "${check}" ] && return 0 + done + return 1 +} + emit() { local -a selected=("$@") - local -a public=() private=() deferred=() + local -a public=() docker=() private=() deferred=() local check for check in ${selected[@]+"${selected[@]}"}; do if is_private_index "${check}"; then private+=("${check}") + elif is_docker "${check}"; then + docker+=("${check}") else public+=("${check}") fi @@ -142,18 +176,21 @@ emit() { private=() fi - local public_json private_json deferred_json + local public_json docker_json private_json deferred_json public_json=$(to_json ${public[@]+"${public[@]}"}) + docker_json=$(to_json ${docker[@]+"${docker[@]}"}) private_json=$(to_json ${private[@]+"${private[@]}"}) deferred_json=$(to_json ${deferred[@]+"${deferred[@]}"}) { echo "checks=${public_json}" + echo "docker_checks=${docker_json}" echo "private_checks=${private_json}" echo "deferred_private_checks=${deferred_json}" } >>"${GITHUB_OUTPUT}" echo "Selected credential-free suites: ${public_json}" + echo "Selected docker-daemon suites: ${docker_json}" echo "Selected private-index suites: ${private_json}" if [ "${deferred_json}" != "[]" ]; then echo "Deferred private-index suites: ${deferred_json}" @@ -207,4 +244,15 @@ for file in "${CHANGED[@]}"; do done done +# splunk_base is a library for the other three splunk suites, so pull them in +# whenever it changes. Ported from `splunk_base_build_rule` in .gitlab-ci.yml. +# `emit` sorts and de-duplicates, so adding them unconditionally is safe. +for check in ${SELECTED[@]+"${SELECTED[@]}"}; do + if [ "${check}" = "splunk_base" ]; then + echo "'splunk_base' changed: also running ${SPLUNK_DEPENDENTS[*]}." + SELECTED+=("${SPLUNK_DEPENDENTS[@]}") + break + fi +done + emit "${SELECTED[@]+"${SELECTED[@]}"}" diff --git a/.github/workflows/checks-tests.yml b/.github/workflows/checks-tests.yml index 1691d3f3..39640da2 100644 --- a/.github/workflows/checks-tests.yml +++ b/.github/workflows/checks-tests.yml @@ -1,10 +1,24 @@ name: Check tests # Ported from .gitlab-ci.yml as part of the GitLab -> GitHub migration -# (STAC-25142 / STAC-25463), phase 1: the pure-Python check suites. +# (STAC-25142), phase 1 (STAC-25463) and phase 2 (STAC-25531). # # WHAT MOVED -# linux_deps + the `test_` job family -> the `check-tests` matrix below. +# linux_deps + the `test_` job family -> the `check-tests` matrix below, +# plus `check-tests-docker` for the five +# suites needing a Docker daemon (STAC-25531) +# and `check-tests-private-index` for vsphere. +# splunk_base_build_rule -> SPLUNK_DEPENDENTS in select-checks.sh. +# .linux_splunk_test's docker pull + COMPOSE_HTTP_TIMEOUT +# -> steps on `check-tests-docker`. +# setup_artifactory_docker.sh -> not ported. It logged docker in to the +# SUSE Private Registry so compose could +# pull Splunk and Vault through the proxy. +# Those fixtures now name the public +# Docker Hub images directly, so there is +# no login to perform -- and a public +# repo's PR jobs must not hold registry +# credentials anyway. # The per-job `changes:` rules -> .github/scripts/select-checks.sh. # The validate suite that rode along inside `test_stackstate_checks_base` # -> its own `validate` job, so a metadata @@ -12,19 +26,6 @@ name: Check tests # instead of hiding inside a test job. # # WHAT IS DELIBERATELY NOT HERE -# splunk_{base,health,metric,topology} and stackstate_checks_dev (STAC-25531) -# The only five suites that need a Docker daemon (the four splunk suites -# drive a real Splunk container via docker-compose; checks_dev tests the -# toolkit's own Docker helpers). ubuntu-latest ships a working Docker -# daemon, so phase 2 does not need to provision anything -- but it does -# need a docker client inside the job, which the BCI Python image used -# here does not carry: either a BCI image with docker added, or a service -# container, rather than a return to the private -# python:3.13.14-bookworm runner image. Phase 2 also brings across -# .setup-scripts/setup_artifactory_docker.sh (the registry docker login) -# and COMPOSE_HTTP_TIMEOUT, which only those suites need, plus GitLab's -# splunk_base_build_rule (a splunk_base change must also run the other -# three, which import its test helpers). # test_postgres # Dead config: .gitlab-ci.yml tests a `postgres` check that does not exist # in this repository. Dropped, not pending. @@ -175,6 +176,7 @@ jobs: timeout-minutes: 10 outputs: checks: ${{ steps.select.outputs.checks }} + docker_checks: ${{ steps.select.outputs.docker_checks }} private_checks: ${{ steps.select.outputs.private_checks }} deferred_private_checks: ${{ steps.select.outputs.deferred_private_checks }} image: ${{ steps.image.outputs.ref }} @@ -461,6 +463,76 @@ jobs: --collect=workflows,actions \ . + check-tests-docker: + name: Check tests, docker (${{ matrix.check }}) + # The four splunk suites and stackstate_checks_dev, which need a real Docker + # daemon (STAC-25531). Credential-free like check-tests above, so no fork + # guard: the test containers now come from public Docker Hub rather than the + # authenticated SUSE Private Registry proxy the GitLab pipeline used. + # + # Runs directly on the runner rather than in the BCI container the other + # suites use, and that is forced by the tests, not chosen. They resolve their + # target through `get_docker_hostname()`, which reads DOCKER_HOST and falls + # back to `localhost`; compose publishes its ports on the Docker host, so + # `localhost` only reaches them when the test process shares a network + # namespace with the daemon. From inside a job container it would resolve to + # the container itself and every connection would be refused. GitLab dodged + # this with a `docker:dind` service whose hostname resolved from both sides; + # on a GitHub-hosted runner the daemon is simply already there. + # + # The consequence is that these suites use the runner's Python rather than the + # BCI image's. Same 3.13 minor, pinned below. + if: needs.select-checks.outputs.docker_checks != '[]' + needs: select-checks + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + check: ${{ fromJson(needs.select-checks.outputs.docker_checks) }} + steps: + - name: Check out repository + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + with: + persist-credentials: false + + - name: Set up Python + uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0 + with: + # Tracks BCI_PYTHON_IMAGE's 3.13 series so both matrices test the same + # minor. setup-python resolves the latest available patch. + python-version: '3.13' + + - name: Build the toolchain virtualenv + run: | + set -eo pipefail + git config --global --add safe.directory '*' + source .setup-scripts/setup_env.sh + + - name: Pre-pull the Splunk image + # Ported from .linux_splunk_test in .gitlab-ci.yml, whose comment reads + # "Pull splunk to aovid pulling during compose, which breaks in python". + # The Splunk image is large and the compose client's HTTP timeout can + # expire mid-pull; pulling first leaves compose only having to start it. + if: startsWith(matrix.check, 'splunk_') + env: + SPLUNK_IMAGE: splunk/splunk:latest + run: | + set -eo pipefail + docker pull "${SPLUNK_IMAGE}" + + - name: checksdev test ${{ matrix.check }} + env: + CHECK: ${{ matrix.check }} + # Ported verbatim from .linux_test / .linux_splunk_test in + # .gitlab-ci.yml: Splunk is slow to come up and the default 60s + # compose timeout expires before it accepts connections. + COMPOSE_HTTP_TIMEOUT: '300' + run: | + set -eo pipefail + source venv/bin/activate + checksdev test --cov "${CHECK}" + ci-success: name: CI success # The single stable status for branch protection. Every other status here is @@ -477,6 +549,7 @@ jobs: - select-checks - validate - check-tests + - check-tests-docker - check-tests-private-index - workflow-security runs-on: ubuntu-latest @@ -488,8 +561,10 @@ jobs: VALIDATE: ${{ needs.validate.result }} WORKFLOW_SECURITY: ${{ needs.workflow-security.result }} CHECK_TESTS: ${{ needs.check-tests.result }} + CHECK_TESTS_DOCKER: ${{ needs.check-tests-docker.result }} CHECK_TESTS_PRIVATE: ${{ needs.check-tests-private-index.result }} SELECTED_CHECKS: ${{ needs.select-checks.outputs.checks }} + SELECTED_DOCKER_CHECKS: ${{ needs.select-checks.outputs.docker_checks }} SELECTED_PRIVATE_CHECKS: ${{ needs.select-checks.outputs.private_checks }} DEFERRED_PRIVATE_CHECKS: ${{ needs.select-checks.outputs.deferred_private_checks }} run: | @@ -528,6 +603,23 @@ jobs: ;; esac + # Same rule as check-tests: these carry no credentials and run on every + # event, so the only legitimate skip is an empty selection. + printf ' %-24s %s (selected: %s)\n' "check-tests-docker" "${CHECK_TESTS_DOCKER}" "${SELECTED_DOCKER_CHECKS}" + case "${CHECK_TESTS_DOCKER}" in + success) ;; + skipped) + if [ "${SELECTED_DOCKER_CHECKS}" != "[]" ]; then + echo "::error title=Selected suites never ran::check-tests-docker was skipped while ${SELECTED_DOCKER_CHECKS} was selected." + status=1 + fi + ;; + *) + echo "::error title=Docker tests did not succeed::check-tests-docker reported '${CHECK_TESTS_DOCKER}'." + status=1 + ;; + esac + # The private-index matrix legitimately skips whenever the selector # chose nothing for this event. On pull requests that is always: those # suites are deferred rather than selected, because the run holds no diff --git a/splunk_base/tests/compose/docker-compose.yaml b/splunk_base/tests/compose/docker-compose.yaml index e6c4fc09..4d1d7e0c 100644 --- a/splunk_base/tests/compose/docker-compose.yaml +++ b/splunk_base/tests/compose/docker-compose.yaml @@ -1,7 +1,13 @@ version: '3' services: splunk: - image: "registry.tooling.stackstate.io/docker/splunk/splunk:latest" + # Docker Hub's official splunk/splunk. The previous + # registry.tooling.stackstate.io/docker/... reference was the SUSE Private + # Registry's Docker Hub proxy-cache of this same image; the proxy needs + # credentials, which this public repo's CI must not hold, so the default + # pulls direct. Override SPLUNK_IMAGE to go back through the proxy on a + # self-hosted runner. + image: "${SPLUNK_IMAGE:-splunk/splunk:latest}" environment: SPLUNK_START_ARGS: --accept-license SPLUNK_PASSWORD: admin12345 diff --git a/splunk_health/tests/compose/docker-compose.yaml b/splunk_health/tests/compose/docker-compose.yaml index 810b56e9..0ec399c4 100644 --- a/splunk_health/tests/compose/docker-compose.yaml +++ b/splunk_health/tests/compose/docker-compose.yaml @@ -1,8 +1,14 @@ version: '3' services: splunk: - # this image is defined in gitlab.com/stackvista/stackstate-installation/plugin-products/splunk - image: "registry.tooling.stackstate.io/docker/splunk/splunk:latest" + # Docker Hub's official splunk/splunk. The previous + # registry.tooling.stackstate.io/docker/... reference was the SUSE Private + # Registry's Docker Hub proxy-cache of this same image, not a StackState + # build -- the adjacent comment claiming otherwise was wrong. The proxy needs + # credentials, which this public repo's CI must not hold, so the default + # pulls direct. Override SPLUNK_IMAGE to go back through the proxy on a + # self-hosted runner. + image: "${SPLUNK_IMAGE:-splunk/splunk:latest}" environment: SPLUNK_START_ARGS: --accept-license SPLUNK_PASSWORD: admin12345 diff --git a/splunk_metric/tests/compose/docker-compose.yaml b/splunk_metric/tests/compose/docker-compose.yaml index e6c4fc09..4d1d7e0c 100644 --- a/splunk_metric/tests/compose/docker-compose.yaml +++ b/splunk_metric/tests/compose/docker-compose.yaml @@ -1,7 +1,13 @@ version: '3' services: splunk: - image: "registry.tooling.stackstate.io/docker/splunk/splunk:latest" + # Docker Hub's official splunk/splunk. The previous + # registry.tooling.stackstate.io/docker/... reference was the SUSE Private + # Registry's Docker Hub proxy-cache of this same image; the proxy needs + # credentials, which this public repo's CI must not hold, so the default + # pulls direct. Override SPLUNK_IMAGE to go back through the proxy on a + # self-hosted runner. + image: "${SPLUNK_IMAGE:-splunk/splunk:latest}" environment: SPLUNK_START_ARGS: --accept-license SPLUNK_PASSWORD: admin12345 diff --git a/splunk_topology/tests/compose/docker-compose.yaml b/splunk_topology/tests/compose/docker-compose.yaml index e6c4fc09..4d1d7e0c 100644 --- a/splunk_topology/tests/compose/docker-compose.yaml +++ b/splunk_topology/tests/compose/docker-compose.yaml @@ -1,7 +1,13 @@ version: '3' services: splunk: - image: "registry.tooling.stackstate.io/docker/splunk/splunk:latest" + # Docker Hub's official splunk/splunk. The previous + # registry.tooling.stackstate.io/docker/... reference was the SUSE Private + # Registry's Docker Hub proxy-cache of this same image; the proxy needs + # credentials, which this public repo's CI must not hold, so the default + # pulls direct. Override SPLUNK_IMAGE to go back through the proxy on a + # self-hosted runner. + image: "${SPLUNK_IMAGE:-splunk/splunk:latest}" environment: SPLUNK_START_ARGS: --accept-license SPLUNK_PASSWORD: admin12345 diff --git a/stackstate_checks_dev/tests/docker/test_default.yaml b/stackstate_checks_dev/tests/docker/test_default.yaml index 83239633..be58fe67 100644 --- a/stackstate_checks_dev/tests/docker/test_default.yaml +++ b/stackstate_checks_dev/tests/docker/test_default.yaml @@ -3,7 +3,11 @@ version: '3' services: vault: - image: registry.tooling.stackstate.io/docker/hashicorp/vault:latest + # Docker Hub's official hashicorp/vault. Was pulled through the SUSE + # Private Registry proxy-cache, which needs credentials this public repo's + # CI must not hold. Override VAULT_IMAGE to use the proxy on a self-hosted + # runner. + image: ${VAULT_IMAGE:-hashicorp/vault:latest} container_name: checksdev_vault cap_add: - IPC_LOCK From d0976661f5e5749dcd2033bc4adf4c0cc83d933f Mon Sep 17 00:00:00 2001 From: Louis Parkin Date: Thu, 6 Aug 2026 15:49:11 +0200 Subject: [PATCH 3/6] STAC-25531: resolve Docker Compose v2 instead of the EOL v1 binary The five Docker suites failed on their first GitHub run with `FileNotFoundError: 'docker-compose'`. Compose v1 -- the standalone Python binary -- reached end of life in July 2023 and is no longer present on the hosted runner images; they ship v2 as the `docker compose` subcommand. The GitLab runner image still carried v1, which is why this never surfaced there. Add `compose_command()`, which prefers `docker compose` and falls back to `docker-compose`, so developers still on an older toolchain keep working, and route all four call sites through it. `DOCKER_COMPOSE_COMMAND` overrides the detection. The result is cached; container-name log checks are untouched, only the compose-file path changes. The two test modules shelled out to `docker-compose` directly to set up and tear down their own fixtures, so they go through the resolver too. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../stackstate_checks/dev/conditions.py | 3 +- .../stackstate_checks/dev/docker.py | 36 +++++++++++++++++-- .../tests/test_conditions.py | 7 ++-- stackstate_checks_dev/tests/test_docker.py | 10 +++--- 4 files changed, 44 insertions(+), 12 deletions(-) diff --git a/stackstate_checks_dev/stackstate_checks/dev/conditions.py b/stackstate_checks_dev/stackstate_checks/dev/conditions.py index fb9cd02a..a67349f3 100644 --- a/stackstate_checks_dev/stackstate_checks/dev/conditions.py +++ b/stackstate_checks_dev/stackstate_checks/dev/conditions.py @@ -165,7 +165,8 @@ def __call__(self): class CheckDockerLogs(CheckCommandOutput): def __init__(self, identifier, patterns, matches=1, stdout=True, stderr=True, attempts=60, wait=1): if file_exists(identifier): - command = ['docker-compose', '-f', identifier, 'logs'] + from .docker import compose_command + command = compose_command() + ['-f', identifier, 'logs'] else: command = ['docker', 'logs', identifier] diff --git a/stackstate_checks_dev/stackstate_checks/dev/docker.py b/stackstate_checks_dev/stackstate_checks/dev/docker.py index 291de4f4..ccee90c4 100644 --- a/stackstate_checks_dev/stackstate_checks/dev/docker.py +++ b/stackstate_checks_dev/stackstate_checks/dev/docker.py @@ -2,6 +2,7 @@ # All rights reserved # Licensed under a 3-clause BSD style license (see LICENSE) import os +import shutil from contextlib import contextmanager from six import string_types @@ -25,6 +26,35 @@ def get_docker_hostname(): return urlparse(os.getenv('DOCKER_HOST', '')).hostname or 'localhost' +_COMPOSE_COMMAND = None + + +def compose_command(): + """Return the Docker Compose invocation available on this machine. + + Compose v1 (the standalone ``docker-compose`` binary) reached end of life in + July 2023 and is absent from current CI runner images, which ship v2 as the + ``docker compose`` subcommand instead. Prefer v2 and fall back to v1 so + developers still on an older toolchain keep working. + + Set ``DOCKER_COMPOSE_COMMAND`` to override the detection. + """ + global _COMPOSE_COMMAND + + override = os.getenv('DOCKER_COMPOSE_COMMAND', '').strip() + if override: + return override.split() + + if _COMPOSE_COMMAND is None: + _COMPOSE_COMMAND = ['docker-compose'] + if shutil.which('docker'): + result = run_command(['docker', 'compose', 'version'], capture=True) + if result.code == 0: + _COMPOSE_COMMAND = ['docker', 'compose'] + + return list(_COMPOSE_COMMAND) + + def get_container_ip(container_id_or_name): """Get a Docker container's IP address from its id or name.""" command = [ @@ -39,7 +69,7 @@ def get_container_ip(container_id_or_name): def compose_file_active(compose_file): - command = ['docker-compose', '-f', compose_file, 'ps'] + command = compose_command() + ['-f', compose_file, 'ps'] lines = run_command(command, capture='out', check=True).stdout.splitlines() for i, line in enumerate(lines, 1): @@ -204,7 +234,7 @@ def __init__(self, compose_file, build=False, service_name=None): self.compose_file = compose_file self.build = build self.service_name = service_name - self.command = ['docker-compose', '-f', self.compose_file, 'up', '-d'] + self.command = compose_command() + ['-f', self.compose_file, 'up', '-d'] if self.build: self.command.append('--build') @@ -220,7 +250,7 @@ class ComposeFileDown(LazyFunction): def __init__(self, compose_file, check=True): self.compose_file = compose_file self.check = check - self.command = ['docker-compose', '-f', self.compose_file, 'down'] + self.command = compose_command() + ['-f', self.compose_file, 'down'] def __call__(self): return run_command(self.command, check=self.check) diff --git a/stackstate_checks_dev/tests/test_conditions.py b/stackstate_checks_dev/tests/test_conditions.py index ddb19bb8..0a254455 100644 --- a/stackstate_checks_dev/tests/test_conditions.py +++ b/stackstate_checks_dev/tests/test_conditions.py @@ -9,6 +9,7 @@ from stackstate_checks.dev.conditions import ( CheckCommandOutput, CheckDockerLogs, CheckEndpoints, WaitFor ) +from stackstate_checks.dev.docker import compose_command from stackstate_checks.dev.errors import RetryError from stackstate_checks.dev.subprocess import run_command @@ -82,7 +83,7 @@ class TestCheckDockerLogs: def test_no_matches(self): compose_file = os.path.join(DOCKER_DIR, 'test_default.yaml') - run_command(['docker-compose', '-f', compose_file, 'down']) + run_command(compose_command() + ['-f', compose_file, 'down']) check_docker_logs = CheckDockerLogs(compose_file, 'Vault server started', attempts=1) with pytest.raises(RetryError): @@ -93,10 +94,10 @@ def test_matches(self): check_docker_logs = CheckDockerLogs(compose_file, 'Vault server started') try: - run_command(['docker-compose', '-f', compose_file, 'up', '-d'], check=True) + run_command(compose_command() + ['-f', compose_file, 'up', '-d'], check=True) check_docker_logs() finally: - run_command(['docker-compose', '-f', compose_file, 'down'], capture=True) + run_command(compose_command() + ['-f', compose_file, 'down'], capture=True) class TestCheckEndpoints: diff --git a/stackstate_checks_dev/tests/test_docker.py b/stackstate_checks_dev/tests/test_docker.py index c53dc3f2..5d3093a1 100644 --- a/stackstate_checks_dev/tests/test_docker.py +++ b/stackstate_checks_dev/tests/test_docker.py @@ -5,7 +5,7 @@ import pytest -from stackstate_checks.dev.docker import compose_file_active, docker_run +from stackstate_checks.dev.docker import compose_command, compose_file_active, docker_run from stackstate_checks.dev.subprocess import run_command pytestmark = [pytest.mark.docker] @@ -16,7 +16,7 @@ class TestComposeFileActive: def test_down(self): compose_file = os.path.join(DOCKER_DIR, 'test_default.yaml') - run_command(['docker-compose', '-f', compose_file, 'down'], capture=True) + run_command(compose_command() + ['-f', compose_file, 'down'], capture=True) assert compose_file_active(compose_file) is False @@ -24,10 +24,10 @@ def test_up(self): compose_file = os.path.join(DOCKER_DIR, 'test_default.yaml') try: - run_command(['docker-compose', '-f', compose_file, 'up', '-d'], check=True) + run_command(compose_command() + ['-f', compose_file, 'up', '-d'], check=True) assert compose_file_active(compose_file) is True finally: - run_command(['docker-compose', '-f', compose_file, 'down'], capture=True) + run_command(compose_command() + ['-f', compose_file, 'down'], capture=True) class TestDockerRun: @@ -39,4 +39,4 @@ def test_compose_file(self): assert compose_file_active(compose_file) is True assert compose_file_active(compose_file) is False finally: - run_command(['docker-compose', '-f', compose_file, 'down'], capture=True) + run_command(compose_command() + ['-f', compose_file, 'down'], capture=True) From 788e46609932124c85e6c812a11d4e6e59a4424d Mon Sep 17 00:00:00 2001 From: Louis Parkin Date: Thu, 6 Aug 2026 15:55:44 +0200 Subject: [PATCH 4/6] STAC-25531: detect an active compose project in a v2-compatible way compose_file_active() parsed the human-readable `ps` table, scanning for the row of dashes that Compose v1 printed between the header and the service rows. Compose v2 does not emit that separator, so the function returned False unconditionally and the two remaining stackstate_checks_dev tests failed even once the command itself resolved correctly. Use `ps -q` instead: one container id per running service, empty when the project is down. That output is stable across both generations, so it needs no version branching. Verified against a live daemon -- False before up, True after up, False after down. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../stackstate_checks/dev/docker.py | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/stackstate_checks_dev/stackstate_checks/dev/docker.py b/stackstate_checks_dev/stackstate_checks/dev/docker.py index ccee90c4..7f61f3bf 100644 --- a/stackstate_checks_dev/stackstate_checks/dev/docker.py +++ b/stackstate_checks_dev/stackstate_checks/dev/docker.py @@ -69,14 +69,12 @@ def get_container_ip(container_id_or_name): def compose_file_active(compose_file): - command = compose_command() + ['-f', compose_file, 'ps'] - lines = run_command(command, capture='out', check=True).stdout.splitlines() - - for i, line in enumerate(lines, 1): - if set(line.strip()) == {'-'}: - return len(lines[i:]) >= 1 - - return False + # `ps -q` prints one container id per running service and nothing at all when + # the project is down. Unlike the human-readable `ps` table -- whose v1 layout + # had a `-----` separator row that v2 does not emit -- this output is stable + # across both Compose generations. + command = compose_command() + ['-f', compose_file, 'ps', '-q'] + return bool(run_command(command, capture='out', check=True).stdout.strip()) @contextmanager From beba732c6a307703cec78c67ff56578868aeb67c Mon Sep 17 00:00:00 2001 From: Louis Parkin Date: Thu, 6 Aug 2026 16:31:21 +0200 Subject: [PATCH 5/6] STAC-25544: move vsphere to the publicly-published VMware SDK vsphere pinned `vsphere-automation-sdk==1.82.0`, which VMware never published to public PyPI -- the name is squatted there by an unrelated 0.0.1 placeholder -- so it was mirrored into the private GitLab package registry and pulled with a credential. That credential could not be held safely. A `pull_request` run executes the pull request's own copy of the workflow and of every script it calls, so a run holding a secret can always be made to disclose it. The only sound answer was to withhold it, which meant vsphere did not run on pull requests at all and was verified only after merge -- a real coverage gap, accepted at the time because the alternatives cost more. VMware publishes the same SDK to public PyPI under renamed packages, actively maintained: pyvmomi, vmware-vcenter, vmware-vapi-runtime and vmware-vapi-common-client, all at 9.1.0.0. Between them they satisfy every import the check makes -- pyVmomi/pyVim, vmware.vapi.* and com.vmware.* alike. So the pin, the credential, the separate job and the coverage gap all go together: - requirements.in names the four public packages - fetch_private_wheels.sh is deleted - PRIVATE_INDEX_CHECKS, is_private_index() and the private/deferred selector outputs are gone; the selector now emits `checks` and `docker_checks` - check-tests-private-index is removed and vsphere joins the ordinary matrix, so it runs on pull requests like every other suite - ci-success loses the private-index arm and the deferral notice - vsphere/tox.ini no longer documents a wheelhouse that does not exist No job in this workflow now has a secret in scope, which is a stronger and much simpler property to maintain than any arrangement that held one. Verified: the four pins resolve from public PyPI alone, and the full vsphere suite passes against them -- 58 passed. actionlint, shellcheck and Zizmor clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/scripts/select-checks.sh | 86 ++------ .github/workflows/checks-tests.yml | 259 ++++++------------------- .setup-scripts/fetch_private_wheels.sh | 156 --------------- vsphere/requirements.in | 10 +- vsphere/tox.ini | 15 +- 5 files changed, 83 insertions(+), 443 deletions(-) delete mode 100755 .setup-scripts/fetch_private_wheels.sh diff --git a/.github/scripts/select-checks.sh b/.github/scripts/select-checks.sh index 92860958..34662b5a 100755 --- a/.github/scripts/select-checks.sh +++ b/.github/scripts/select-checks.sh @@ -15,31 +15,24 @@ # EVERY suite (GitLab: the `base_changes` anchor). # * Otherwise only the suites whose own directory changed run. # * GitLab's `splunk_base_build_rule` -- a change to splunk_base also runs the -# other three splunk suites, which import its test helpers -- is not ported -# here because no splunk suite runs yet. It lands with them in phase 2 -# (STAC-25531). +# other three splunk suites, which import its test helpers. # * push / workflow_dispatch run everything (GitLab: `master_branch`, # `release_branch`). # -# Writes three arrays to $GITHUB_OUTPUT for `fromJson()` in a matrix: -# checks -- suites that need no credentials -# private_checks -- suites that install from the private GitLab PyPI -# index, and are cleared to run on this event -# deferred_private_checks -- private-index suites withheld from this event -# (always empty outside pull requests) +# Writes two arrays to $GITHUB_OUTPUT for `fromJson()` in a matrix: +# checks -- suites that run in the shared BCI container +# docker_checks -- suites that need a live Docker daemon and so run directly +# on the runner (STAC-25531) # -# The split is a security boundary, not a convenience. The credential-free suites -# run with no secrets in scope at all. The private-index suites need a registry -# password, so they are kept in a separate job -- and, on pull requests, are not -# run at all (STAC-25540, second review pass). -# -# That last part is the whole point, so it is worth stating plainly: a -# `pull_request` run executes the pull request's own copy of the workflow and of -# every script it calls. Hardening the job cannot keep a determined pull request -# away from a secret the run is holding -- it can always edit the thing that holds -# it. The only run that cannot leak the credential is a run that never receives -# it, so these suites are deferred to push, tag and workflow_dispatch events, -# whose contents are reviewed before they reach the release branch. +# Every suite here is credential-free, and that is worth keeping. Until +# STAC-25544 `vsphere` resolved only against a private package registry, which +# meant withholding the credential from pull requests and therefore not running +# the suite on them at all -- a real coverage gap, because a `pull_request` run +# executes the pull request's own copy of the workflow and of every script it +# calls, so a run holding a secret cannot be hardened against the pull request +# that edits it. Modernising the VMware pin onto public PyPI removed the secret +# and with it the gap. If a suite ever appears to need a registry credential +# again, removing that need is the fix; splitting the matrix is not. set -euo pipefail @@ -100,22 +93,6 @@ SPLUNK_DEPENDENTS=( splunk_topology ) -# Suites whose requirements resolve only against the private GitLab PyPI index. -# `vsphere` pins vsphere-automation-sdk, which VMware never published to public -# PyPI (the name is squatted there by an unrelated 0.0.1 placeholder), so it is -# mirrored into the StackVista package registry and needs authentication. -# -# Everything not listed here is credential-free and must stay that way: adding a -# suite to this list stops it running on pull requests altogether, and removing -# the need for the private index is always the better fix. For vsphere that fix -# looks reachable -- VMware now publishes the SDK to public PyPI under renamed -# packages (vmware-vapi-runtime, vmware-vapi-common-client, pyvmomi) and ships -# the NSX/VMC wheels from its own public index -- so this list should shrink to -# nothing once the pin is modernised. -PRIVATE_INDEX_CHECKS=( - vsphere -) - # A change anywhere here invalidates every suite: the base classes and the test # helpers are imported by all of them, and the setup scripts build the venv the # suites run in. @@ -136,14 +113,6 @@ to_json() { fi } -is_private_index() { - local candidate=$1 check - for check in "${PRIVATE_INDEX_CHECKS[@]}"; do - [ "${candidate}" = "${check}" ] && return 0 - done - return 1 -} - is_docker() { local candidate=$1 check for check in "${DOCKER_CHECKS[@]}"; do @@ -154,48 +123,27 @@ is_docker() { emit() { local -a selected=("$@") - local -a public=() docker=() private=() deferred=() + local -a public=() docker=() local check for check in ${selected[@]+"${selected[@]}"}; do - if is_private_index "${check}"; then - private+=("${check}") - elif is_docker "${check}"; then + if is_docker "${check}"; then docker+=("${check}") else public+=("${check}") fi done - # Pull requests do not run the private-index suites at all (STAC-25540, second - # review pass). See the security-boundary note at the top of this file: a - # `pull_request` run executes the pull request's own copy of the workflow and - # scripts, so the credential can only be protected by withholding it. These - # suites run on the release branch instead, where the code has been reviewed. - if [ "${EVENT_NAME}" = "pull_request" ] && [ "${#private[@]}" -gt 0 ]; then - deferred=("${private[@]}") - private=() - fi - - local public_json docker_json private_json deferred_json + local public_json docker_json public_json=$(to_json ${public[@]+"${public[@]}"}) docker_json=$(to_json ${docker[@]+"${docker[@]}"}) - private_json=$(to_json ${private[@]+"${private[@]}"}) - deferred_json=$(to_json ${deferred[@]+"${deferred[@]}"}) { echo "checks=${public_json}" echo "docker_checks=${docker_json}" - echo "private_checks=${private_json}" - echo "deferred_private_checks=${deferred_json}" } >>"${GITHUB_OUTPUT}" echo "Selected credential-free suites: ${public_json}" echo "Selected docker-daemon suites: ${docker_json}" - echo "Selected private-index suites: ${private_json}" - if [ "${deferred_json}" != "[]" ]; then - echo "Deferred private-index suites: ${deferred_json}" - echo "::notice title=Private-index suites do not run on pull requests::${deferred_json} resolve only against the private package registry. Pull requests are deliberately given no credential to reach it, so these suites run on ${BASE_REF:-the release branch} after merge." - fi } # Anything that is not a pull request is a full run. On the release branch the diff --git a/.github/workflows/checks-tests.yml b/.github/workflows/checks-tests.yml index 39640da2..70feab23 100644 --- a/.github/workflows/checks-tests.yml +++ b/.github/workflows/checks-tests.yml @@ -1,16 +1,21 @@ name: Check tests # Ported from .gitlab-ci.yml as part of the GitLab -> GitHub migration -# (STAC-25142), phase 1 (STAC-25463) and phase 2 (STAC-25531). +# (STAC-25142), phase 1 (STAC-25463), phase 2 (STAC-25531) and the private-index +# removal (STAC-25544). # # WHAT MOVED # linux_deps + the `test_` job family -> the `check-tests` matrix below, # plus `check-tests-docker` for the five -# suites needing a Docker daemon (STAC-25531) -# and `check-tests-private-index` for vsphere. +# suites needing a Docker daemon (STAC-25531). # splunk_base_build_rule -> SPLUNK_DEPENDENTS in select-checks.sh. # .linux_splunk_test's docker pull + COMPOSE_HTTP_TIMEOUT # -> steps on `check-tests-docker`. +# setup_artifact_registry.sh -> not ported. It configured pip against the +# private GitLab PyPI index for vsphere. +# STAC-25544 moved that pin to the packages +# VMware publishes on public PyPI, so there +# is no private index left to configure. # setup_artifactory_docker.sh -> not ported. It logged docker in to the # SUSE Private Registry so compose could # pull Splunk and Vault through the proxy. @@ -50,69 +55,56 @@ name: Check tests # file and .github/workflows/cerberus-notify.yml. # # CREDENTIALS -# The container image is SUSE BCI from registry.suse.com, which is public, so -# these jobs need no registry credentials at all. That is deliberate: this is a -# PUBLIC repository, and every job here executes PR-authored workflow, setup and -# test code. Any secret exposed to that code is exposed to whoever can open a -# branch. The earlier design pulled a private runner image with -# vars.REGISTRY_USER / secrets.REGISTRY_PASSWORD; dropping it removes the -# registry password from the PR path entirely and, as a side effect, lets -# Dependabot PRs run -- they receive no Actions secrets, so the image pull -# could never have succeeded for them. +# There are none, and that is the design. This is a PUBLIC repository and every +# job here executes PR-authored workflow, setup and test code, so any secret in +# scope is a secret available to whoever can open a branch. A `pull_request` run +# executes the pull request's own copy of this workflow and of every script it +# calls, which means a run holding a secret can always be made to disclose it -- +# by editing the script that fetches it, reordering steps, or adding one. A +# repository secret and pull-request-controlled code do not compose into a +# security boundary, however carefully the code in between is written. The only +# run that cannot leak a credential is a run that never receives one. # -# One credential remains: the read-only pull from the private PyPI index, for -# pins that public PyPI does not serve (currently vsphere-automation-sdk). That -# is vars.GITLAB_PACKAGE_REGISTRY_PYPI_SIMPLE_URL and -# secrets.GITLAB_PACKAGE_REGISTRY_USER, granted to this repo in pulumi-infra -# (StackVista/pulumi-infra#263), alongside the already-org-wide -# secrets.GITLAB_PACKAGE_REGISTRY_READONLY_PASSWORD. It is pull-only and -# least-privilege by construction; this repo's *publishing* role is still -# deferred, per the note above. +# Two credentials were removed to get here. The private runner image needed +# vars.REGISTRY_USER / secrets.REGISTRY_PASSWORD; switching to public SUSE BCI +# dropped it, and as a side effect let Dependabot PRs run at all, since they +# receive no Actions secrets and so could never have pulled that image. # -# No pull request ever receives it. `pull_request` runs execute the pull -# request's own copy of this workflow and of every script it calls, so a run -# that holds a secret can always be made to disclose it -- by editing the -# fetching script, reordering steps, or adding one. A repository secret and -# pull-request-controlled code cannot be arranged into a boundary. The suites -# that need this credential therefore do not run on pull requests at all; they -# run on push, tag and workflow_dispatch events, whose contents are reviewed -# before reaching the release branch. See `check-tests-private-index`. +# The second was the read-only pull from the private GitLab PyPI index, which +# existed solely because vsphere pinned `vsphere-automation-sdk` -- a package +# VMware never published to public PyPI, where the name is squatted by an +# unrelated 0.0.1 placeholder. Withholding that credential from pull requests +# was the only sound way to hold it, which meant vsphere was verified after +# merge rather than on the pull request that changed it. STAC-25544 removed the +# need instead: VMware publishes the same SDK to public PyPI under renamed +# packages (pyvmomi, vmware-vcenter, vmware-vapi-runtime, +# vmware-vapi-common-client), so the pin, the credential, the separate job and +# the coverage gap went together. vsphere is now an ordinary suite in the main +# matrix and runs on pull requests like every other one. # -# Within those runs the credential is still confined to a single step -# (STAC-25540): the script writes ~/.netrc, downloads one fixed package set into -# a local wheelhouse, deletes the netrc, and points pip at the wheelhouse, so -# the suite and its dependency tree install with nothing to authenticate -# against. The predecessor left the netrc readable for the rest of the job. That -# is defence in depth, not the boundary -- the boundary is the event condition -# above. -# -# Two earlier answers to the same review finding are recorded here so they are -# not re-proposed. A `private-package-index` GitHub Environment with required +# Three earlier answers to that same review finding are recorded so they are not +# re-proposed. A `private-package-index` GitHub Environment with required # reviewers did gate the credential, but SHARED_PATHS covers the CI files, so it # fired on roughly one commit in six and blocked authors on their own pull # requests -- while only ever constraining people who already have write access. -# Prefetching the wheelhouse on a trusted event and passing it to pull requests +# Prefetching a wheelhouse on a trusted event and handing it to pull requests # through the Actions cache also works, but any pull request can read a cache, -# and a pull request can only restore one from its base branch. +# and can only restore one from its base branch. Confining the credential to a +# single step was worth doing and was done, but it is hardening, not a boundary. # -# The durable fix is to stop needing the index: VMware now publishes this SDK to -# public PyPI under renamed packages (vmware-vapi-runtime, -# vmware-vapi-common-client, pyvmomi) and serves the NSX/VMC wheels from its own -# public index, so modernising the pin removes the credential, this job and the -# pull-request coverage gap in one change. +# If a dependency ever appears to need a private index again, removing that need +# is the fix. Reintroducing a credential to this workflow is not. # # RUNNERS -# Everything runs on GitHub-hosted runners. The suites are pure-Python and need -# no Docker daemon, so the self-hosted docker-public pool bought nothing while -# costing real isolation: fork PRs had to be excluded from it, which in turn -# meant a fork could never produce a CI verdict. On hosted runners forks run -# exactly the same matrix as any other pull request -- the private-index job is -# off the pull-request path entirely, so no fork-specific guard is needed for -# it any more. This also removes the question of -# pulling upstream images across the self-hosted NAT: the BCI reference is -# direct, from a public registry, on infrastructure that is meant to reach it. -# The phase-2 Docker suites will need a runner with a daemon; that decision -# belongs with them, not here. +# Everything runs on GitHub-hosted runners. The self-hosted docker-public pool +# bought nothing while costing real isolation: fork PRs had to be excluded from +# it, which in turn meant a fork could never produce a CI verdict. On hosted +# runners forks run exactly the same matrix as any other pull request, and no +# job holds a secret, so no fork-specific guard is needed anywhere. This also +# removes the question of pulling upstream images across the self-hosted NAT: +# the BCI reference is direct, from a public registry, on infrastructure meant +# to reach it. The Docker suites need a daemon rather than a container, so they +# run directly on the hosted runner -- see `check-tests-docker`. on: pull_request: @@ -156,8 +148,8 @@ env: # # BCI publishes 3.13.13, one patch behind the 3.13.14 the agent embeds and the # GitLab image pinned. CPython patch releases are bugfix-only, and the full - # phase-1 matrix (including vsphere against the private index) was verified - # green on 3.13.13 before this switch. Worth realigning when BCI ships .14. + # phase-1 matrix was verified green on 3.13.13 before this switch. Worth + # realigning when BCI ships .14. BCI_PYTHON_IMAGE: registry.suse.com/bci/python:3.13@sha256:7d36dd3ba6596fb690e31d956952059fd010604ad6309f06462c02c4c9c01461 # 3.13.13 # Packages the BCI image does not ship but the toolchain build needs: cython and @@ -167,18 +159,14 @@ env: jobs: select-checks: name: Select check suites to run - # Runs for forks too. Every job that runs on a pull request does so with no - # secrets in scope at all -- the one job that uses a credential does not run - # on pull requests (see `check-tests-private-index`). There is therefore - # nothing a fork branch can reach here, and blocking forks outright would + # Runs for forks too. No job in this workflow has a secret in scope, so there + # is nothing a fork branch can reach here, and blocking forks outright would # leave them unable to satisfy branch protection at all (STAC-25463 review). runs-on: ubuntu-latest timeout-minutes: 10 outputs: checks: ${{ steps.select.outputs.checks }} docker_checks: ${{ steps.select.outputs.docker_checks }} - private_checks: ${{ steps.select.outputs.private_checks }} - deferred_private_checks: ${{ steps.select.outputs.deferred_private_checks }} image: ${{ steps.image.outputs.ref }} steps: - name: Check out repository @@ -314,116 +302,6 @@ jobs: source venv/bin/activate checksdev test "${CHECK}" --bench - check-tests-private-index: - name: Check tests, private index (${{ matrix.check }}) - # Isolated from `check-tests` because this is the only job that handles a - # credential at all: vsphere pins a package that resolves solely from the - # private GitLab Package Registry. - # - # This job does not run on pull requests (STAC-25540, second review pass). - # - # An earlier revision ran it on pull requests with the credential confined to - # a single step, and claimed that reaching it would require editing this - # workflow. That claim was wrong, and the review was right to call it: a - # `pull_request` run executes the pull request's own copy of the workflow AND - # of every script it calls, so a pull request could rewrite the fetch script, - # reorder these steps, or simply add a step of its own. Repository secrets - # plus pull-request-controlled code do not make a security boundary, however - # carefully the code in between is written. - # - # Confining the credential to one step is still worth doing and is still done - # -- it keeps the password away from the suite's dependency tree, which needed - # no malice at all to read it -- but it is hardening, not a boundary. The - # boundary is this condition: the run simply never receives the secret. - # - # The cost is that vsphere is verified on the release branch rather than on - # the pull request that changes it. That is a real gap, accepted knowingly: - # the suite's own directory changes a handful of times a year, and the - # alternatives all cost more than they return right now. Sharing a prefetched - # wheelhouse through the Actions cache would work, but a pull request run can - # only restore caches from its base branch, and any pull request can read - # them. The durable fix is to stop needing the private index at all: VMware - # now publishes this SDK to public PyPI under renamed packages, so modernising - # the pin deletes this job, its credential and this trade-off together. - if: >- - ${{ github.event_name != 'pull_request' - && needs.select-checks.outputs.private_checks != '[]' }} - needs: select-checks - runs-on: ubuntu-latest - timeout-minutes: 45 - strategy: - fail-fast: false - matrix: - check: ${{ fromJson(needs.select-checks.outputs.private_checks) }} - container: - # Digest-pinned at workflow level (BCI_PYTHON_IMAGE); the ignore is only - # because Zizmor cannot follow the pin through a job output. - image: ${{ needs.select-checks.outputs.image }} # zizmor: ignore[unpinned-images] - steps: - - name: Check out repository - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 - with: - persist-credentials: false - - - name: Install toolchain build dependencies - run: | - set -eo pipefail - zypper --non-interactive --gpg-auto-import-keys refresh - # shellcheck disable=SC2086 # deliberately word-split into package args - zypper --non-interactive install ${BCI_BUILD_PACKAGES} - - - name: Build the toolchain virtualenv - run: | - set -eo pipefail - git config --global --add safe.directory '*' - source .setup-scripts/setup_env.sh - - - name: Fetch private-index wheels and revoke the credential - env: - GITLAB_PACKAGE_REGISTRY_PYPI_SIMPLE_URL: ${{ vars.GITLAB_PACKAGE_REGISTRY_PYPI_SIMPLE_URL }} - GITLAB_PACKAGE_REGISTRY_USER: ${{ secrets.GITLAB_PACKAGE_REGISTRY_USER }} - GITLAB_PACKAGE_REGISTRY_READONLY_PASSWORD: ${{ secrets.GITLAB_PACKAGE_REGISTRY_READONLY_PASSWORD }} - # The only step in this workflow with a secret in scope. The script writes - # ~/.netrc, downloads one fixed package set, deletes the netrc, and leaves - # ~/.pip/pip.conf pointing at a local wheelhouse. Everything after it -- - # checksdev, tox, the suite's tests and their dependency tree -- runs with - # no credential on disk and no authenticated index configured. - # - # It replaces setup_artifact_registry.sh here, which left the netrc in - # place for the rest of the job (STAC-25463 review P1, STAC-25540). That - # script is untouched and still serves the GitLab pipeline definitions. - # - # The wheelhouse lives in RUNNER_TEMP rather than the workspace so it - # cannot be mistaken for repository content or swept into a build. - # - # NOTE: pip.conf is read from $HOME, so tox must still pass HOME into the - # testenv. tox drops every variable absent from `passenv`, and pip then - # resolves `~` from the passwd database rather than the environment -- - # which points at the wrong home in a container job, where HOME is - # /github/home. Without it the suite silently falls back to public PyPI - # and installs the 0.0.1 placeholder. See vsphere/tox.ini. - run: | - set -eo pipefail - .setup-scripts/fetch_private_wheels.sh "${RUNNER_TEMP}/private-wheels" - - - name: checksdev test ${{ matrix.check }} - env: - CHECK: ${{ matrix.check }} - run: | - set -eo pipefail - source venv/bin/activate - checksdev test --cov "${CHECK}" - - - name: checksdev benchmarks ${{ matrix.check }} - env: - CHECK: ${{ matrix.check }} - # Non-blocking, matching GitLab's `|| true`. - continue-on-error: true - run: | - set -eo pipefail - source venv/bin/activate - checksdev test "${CHECK}" --bench - workflow-security: name: Workflow security scan (Zizmor) # Credential-free and read-only, and it runs on GitHub-hosted infrastructure, @@ -550,7 +428,6 @@ jobs: - validate - check-tests - check-tests-docker - - check-tests-private-index - workflow-security runs-on: ubuntu-latest timeout-minutes: 5 @@ -562,11 +439,8 @@ jobs: WORKFLOW_SECURITY: ${{ needs.workflow-security.result }} CHECK_TESTS: ${{ needs.check-tests.result }} CHECK_TESTS_DOCKER: ${{ needs.check-tests-docker.result }} - CHECK_TESTS_PRIVATE: ${{ needs.check-tests-private-index.result }} SELECTED_CHECKS: ${{ needs.select-checks.outputs.checks }} SELECTED_DOCKER_CHECKS: ${{ needs.select-checks.outputs.docker_checks }} - SELECTED_PRIVATE_CHECKS: ${{ needs.select-checks.outputs.private_checks }} - DEFERRED_PRIVATE_CHECKS: ${{ needs.select-checks.outputs.deferred_private_checks }} run: | set -euo pipefail status=0 @@ -603,8 +477,8 @@ jobs: ;; esac - # Same rule as check-tests: these carry no credentials and run on every - # event, so the only legitimate skip is an empty selection. + # Same rule as check-tests: no job in this workflow carries a + # credential, so the only legitimate skip is an empty selection. printf ' %-24s %s (selected: %s)\n' "check-tests-docker" "${CHECK_TESTS_DOCKER}" "${SELECTED_DOCKER_CHECKS}" case "${CHECK_TESTS_DOCKER}" in success) ;; @@ -620,31 +494,6 @@ jobs: ;; esac - # The private-index matrix legitimately skips whenever the selector - # chose nothing for this event. On pull requests that is always: those - # suites are deferred rather than selected, because the run holds no - # credential to reach the private index with. The deferral is reported - # so a green pull request never quietly implies vsphere was covered. - printf ' %-24s %s (selected: %s, deferred: %s)\n' \ - "check-tests-private" "${CHECK_TESTS_PRIVATE}" "${SELECTED_PRIVATE_CHECKS}" "${DEFERRED_PRIVATE_CHECKS}" - case "${CHECK_TESTS_PRIVATE}" in - success) ;; - skipped) - if [ "${SELECTED_PRIVATE_CHECKS}" != "[]" ]; then - echo "::error title=Selected suites never ran::check-tests-private-index was skipped while ${SELECTED_PRIVATE_CHECKS} was selected." - status=1 - fi - ;; - *) - echo "::error title=Private-index tests did not succeed::check-tests-private-index reported '${CHECK_TESTS_PRIVATE}'." - status=1 - ;; - esac - - if [ "${DEFERRED_PRIVATE_CHECKS}" != "[]" ]; then - echo "::notice title=Not covered by this run::${DEFERRED_PRIVATE_CHECKS} need the private package registry and do not run on pull requests. They run on the release branch after merge." - fi - if [ "${status}" -ne 0 ]; then exit 1 fi diff --git a/.setup-scripts/fetch_private_wheels.sh b/.setup-scripts/fetch_private_wheels.sh deleted file mode 100755 index f138ce66..00000000 --- a/.setup-scripts/fetch_private_wheels.sh +++ /dev/null @@ -1,156 +0,0 @@ -#!/usr/bin/env bash -# Makes the packages that exist only in the private GitLab Package Registry -# available to a local wheelhouse, and destroys the credential before returning. -# -# Why this exists (STAC-25540): vsphere pins vsphere-automation-sdk==1.82.0, an -# unmodified upstream VMware wheel that VMware withdrew from public PyPI. We -# self-host it in the GitLab Package Registry only because that org was private; -# public PyPI now serves a 0.0.1 placeholder squatting the name. -# -# This script only ever runs on events whose contents have been reviewed -- push, -# tag and workflow_dispatch. It does NOT run on pull requests, and the guard below -# enforces that independently of the workflow, because a pull request can edit the -# workflow as freely as it can edit this file. That is the actual protection for -# the credential; everything else here is defence in depth (STAC-25540, second -# review pass). -# -# The defence in depth still matters. The predecessor, setup_artifact_registry.sh, -# left a 0600 ~/.netrc in place for the remainder of the job, so every later step -# -- the tox environment, the suite's own tests, their transitive dependencies -- -# could read the password. That needed no malice from anyone. Here the credential -# exists only for the duration of one pip invocation whose package set is fixed -# below, and pip is then pointed at the resulting wheelhouse so the rest of the -# job resolves offline with nothing to authenticate against. -set -euo pipefail - -# A pull request must never reach the registry password, and must not be able to -# arrange for this script to fetch it one. The workflow already declines to run -# the job on pull requests; this is the same rule stated where it cannot be -# removed by editing a YAML condition. -if [ "${GITHUB_EVENT_NAME:-}" = "pull_request" ]; then - echo "::error title=Refusing to fetch on a pull request::${0##*/} handles the private registry credential and must not run on pull_request events; the private-index suites run on the release branch instead." - exit 1 -fi - -WHEELHOUSE_ARG="${1:-}" -if [ -z "${WHEELHOUSE_ARG}" ]; then - echo "usage: ${0##*/} " >&2 - exit 2 -fi - -# Absolute: pip.conf's find-links is resolved against the working directory of -# whichever process reads it, and tox runs pip from the suite directory. -mkdir -p "${WHEELHOUSE_ARG}" -WHEELHOUSE="$(cd "${WHEELHOUSE_ARG}" && pwd)" - -# Hardcoded on purpose, and deliberately not read from the working tree. While -# the credential is on disk, a pull request must not be able to redirect pip at a -# package of its choosing. -PRIVATE_REQUIREMENTS=( - "vsphere-automation-sdk==1.82.0" -) - -for var in GITLAB_PACKAGE_REGISTRY_PYPI_SIMPLE_URL GITLAB_PACKAGE_REGISTRY_USER GITLAB_PACKAGE_REGISTRY_READONLY_PASSWORD; do - if [ -z "${!var:-}" ]; then - echo "::error title=Private PyPI index not configured::${var} is not available to this job, but this suite cannot resolve without the private index." - exit 1 - fi -done - -NETRC="${HOME}/.netrc" -PIP_CONF_DIR="${HOME}/.pip" - -revoke_credential() { - rm -f "${NETRC}" -} -# Covers the error paths too: a failed download must not leave the password on a -# disk that PR-authored test code goes on to run against. -trap revoke_credential EXIT - -# Hostname only; the simple URL carries a path after the first '/'. -NETRC_HOST="${GITLAB_PACKAGE_REGISTRY_PYPI_SIMPLE_URL%%/*}" - -umask 077 -cat > "${NETRC}" </dev/null || true)"; do - if [ -n "${candidate}" ] && [ -x "${candidate}" ]; then - PYTHON="${candidate}" - break - fi -done -if [ -z "${PYTHON}" ]; then - echo "::error title=No system interpreter::Could not locate a python3 to download with." - exit 1 -fi -if [ -n "${GITHUB_WORKSPACE:-}" ]; then - PYTHON_DIR="$(cd "$(dirname "${PYTHON}")" && pwd)" - case "${PYTHON_DIR}/" in - "${GITHUB_WORKSPACE%/}/"*) - echo "::error title=Refusing a workspace interpreter::Resolved python3 at ${PYTHON}, which is inside the checkout and therefore PR-controlled." - exit 1 - ;; - esac -fi - -echo "→ Downloading private-index packages into ${WHEELHOUSE}" -printf ' %s\n' "${PRIVATE_REQUIREMENTS[@]}" -echo " using ${PYTHON}" - -# --only-binary=:all: matters as much as the interpreter choice. Downloading an -# sdist executes its setup.py, so allowing one would hand arbitrary upstream code -# a process with the registry password readable at ~/.netrc. -"${PYTHON}" -m pip download \ - --disable-pip-version-check \ - --no-cache-dir \ - --only-binary=:all: \ - --extra-index-url "https://${GITLAB_PACKAGE_REGISTRY_PYPI_SIMPLE_URL}" \ - --dest "${WHEELHOUSE}" \ - "${PRIVATE_REQUIREMENTS[@]}" - -revoke_credential -trap - EXIT - -if [ -f "${NETRC}" ]; then - echo "::error title=Credential not revoked::${NETRC} still exists after download; refusing to continue." - exit 1 -fi - -# A silent miss here would fall through to public PyPI and install the 0.0.1 -# placeholder, which fails much later and far less legibly. -if ! find "${WHEELHOUSE}" -maxdepth 1 -iname 'vsphere_automation_sdk-*.whl' | grep -q .; then - echo "::error title=Private wheel missing::vsphere-automation-sdk was not downloaded into ${WHEELHOUSE}." - exit 1 -fi - -# Replaces the extra-index-url that setup_artifact_registry.sh used to write. -# Nothing after this point authenticates anywhere: the private packages resolve -# from the local wheelhouse, and everything else still comes from public PyPI. -mkdir -p "${PIP_CONF_DIR}" -cat > "${PIP_CONF_DIR}/pip.conf" < Date: Thu, 6 Aug 2026 16:39:08 +0200 Subject: [PATCH 6/6] STAC-25544: update the embedded agent requirements to the renamed SDK `checksdev validate dep` cross-checks every pin in a check's requirements.in against agent_requirements.in, the manifest of what the agent embeds. That file still listed vsphere-automation-sdk, so the four replacement packages were reported missing from the embedded environment. Replace the single old entry with the four public ones. Verified locally: `checksdev validate dep` exits 0, as do config, manifest, metadata and service-checks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../stackstate_checks/base/data/agent_requirements.in | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/stackstate_checks_base/stackstate_checks/base/data/agent_requirements.in b/stackstate_checks_base/stackstate_checks/base/data/agent_requirements.in index 8953ca40..3f8ed080 100644 --- a/stackstate_checks_base/stackstate_checks/base/data/agent_requirements.in +++ b/stackstate_checks_base/stackstate_checks/base/data/agent_requirements.in @@ -28,5 +28,8 @@ simplejson==3.20.1 six==1.16.0 urllib3==2.7.0 uuid==1.30 -vsphere-automation-sdk==1.82.0 +pyvmomi==9.1.0.0 +vmware-vapi-common-client==9.1.0.0 +vmware-vapi-runtime==9.1.0.0 +vmware-vcenter==9.1.0.0 zeep==3.4.0