STAC-25533: report release-branch CI failures to Slack via Cerberus - #366
Merged
Conversation
This pipeline has never had a failure notification. Its retired .gitlab-ci.yml
had no notify job and no .cerberus directory, so a broken release branch has
always been silent here. STAC-25510 is what that costs: stackstate-process-agent's
image publishing broke on 2026-07-23 and nobody noticed for 12 days.
Adds a reusable cerberus-notify.yml, structurally identical to the one
stackstate-process-agent gained in STAC-25519, and calls it as a terminal job
from checks-tests.yml.
`action: notify`, never `block`. Blocking locks the branch, additionally
requires the Cerberus GitHub App to be installed here, and mutates the branch
protection pulumi-infra now owns (STAC-25522) -- a later pulumi apply would
silently unlock the branch again.
Push events only, which given the existing `push:` trigger means the release
branch and release tags. Pull request failures already have an owner watching
them, and on a public repo anyone can open a PR that fails on purpose, so
notifying on them would hand the CI channel a spam vector. The condition is
keyed on `github.event_name` rather than a hardcoded ref so the next
release-branch bump only has to touch the `push:` trigger, not two places that
must agree.
Hung off ci-success rather than the individual jobs so there is one funnel for
all failures instead of a notify job per pipeline job.
Two departures from the process-agent original:
- The header records this as new capability rather than a port.
- COMMIT_TITLE falls back to the sha. Tag pushes carry no `head_commit`, and
this workflow runs on them; process-agent's copy is push-to-master only and
never hits that case.
Both secrets are declared `required: false` and guarded at runtime: a caller
passing `${{ secrets.X }}` for a secret the repo does not hold yields an empty
string, which GitHub rejects against a required secret and fails the call
before the guard can warn. Missing configuration warns and exits 0 rather than
adding a second red job to an already-failed run.
The repo-level secret grant is StackVista/pulumi-infra#277 -- the org copies
are visibility=private and this repo is public. Until that merges this job
warns and passes.
Validated: actionlint (with shellcheck) and Zizmor 1.28.0 clean; payload
construction dry-run produces valid JSON for both a normal commit and the
empty-head_commit tag case.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Phase 2 of the GitLab->GitHub CI migration. The four splunk_* suites and
stackstate_checks_dev need a live Docker daemon, so phase 1 left them out
entirely -- they currently run nowhere.
They cannot run inside the BCI job container. get_docker_hostname() resolves
to `localhost` unless DOCKER_HOST is set, and compose publishes its ports on
the Docker host, so the test process must share the daemon's network
namespace. GitLab avoided this with a docker:dind service whose hostname
resolved identically from both sides; on GitHub the equivalent is to run
directly on the runner. Hence a separate `check-tests-docker` job with no
container and actions/setup-python, rather than extending the existing matrix.
The fixtures pulled their images through registry.tooling.stackstate.io,
which answers 401 from the public internet -- a credential a public repo's
pull_request jobs must never hold. All five paths were /docker/ proxy-cache
entries, so they are repointed at the unambiguous Docker Hub originals via
${SPLUNK_IMAGE}/${VAULT_IMAGE} with public defaults, leaving self-hosted and
local users able to override back to the proxy.
The selector gains a DOCKER_CHECKS set and a third output. splunk_base is a
build dependency of the other three splunk suites, so a change to it fans out
to all four; the reverse does not hold.
setup_artifactory_docker.sh is deliberately not ported -- it existed only to
log Docker in to that proxy.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The five Docker suites failed on their first GitHub run with `FileNotFoundError: 'docker-compose'`. Compose v1 -- the standalone Python binary -- reached end of life in July 2023 and is no longer present on the hosted runner images; they ship v2 as the `docker compose` subcommand. The GitLab runner image still carried v1, which is why this never surfaced there. Add `compose_command()`, which prefers `docker compose` and falls back to `docker-compose`, so developers still on an older toolchain keep working, and route all four call sites through it. `DOCKER_COMPOSE_COMMAND` overrides the detection. The result is cached; container-name log checks are untouched, only the compose-file path changes. The two test modules shelled out to `docker-compose` directly to set up and tear down their own fixtures, so they go through the resolver too. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
compose_file_active() parsed the human-readable `ps` table, scanning for the row of dashes that Compose v1 printed between the header and the service rows. Compose v2 does not emit that separator, so the function returned False unconditionally and the two remaining stackstate_checks_dev tests failed even once the command itself resolved correctly. Use `ps -q` instead: one container id per running service, empty when the project is down. That output is stable across both generations, so it needs no version branching. Verified against a live daemon -- False before up, True after up, False after down. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
vsphere pinned `vsphere-automation-sdk==1.82.0`, which VMware never published
to public PyPI -- the name is squatted there by an unrelated 0.0.1 placeholder
-- so it was mirrored into the private GitLab package registry and pulled with
a credential.
That credential could not be held safely. A `pull_request` run executes the
pull request's own copy of the workflow and of every script it calls, so a run
holding a secret can always be made to disclose it. The only sound answer was
to withhold it, which meant vsphere did not run on pull requests at all and
was verified only after merge -- a real coverage gap, accepted at the time
because the alternatives cost more.
VMware publishes the same SDK to public PyPI under renamed packages, actively
maintained: pyvmomi, vmware-vcenter, vmware-vapi-runtime and
vmware-vapi-common-client, all at 9.1.0.0. Between them they satisfy every
import the check makes -- pyVmomi/pyVim, vmware.vapi.* and com.vmware.* alike.
So the pin, the credential, the separate job and the coverage gap all go
together:
- requirements.in names the four public packages
- fetch_private_wheels.sh is deleted
- PRIVATE_INDEX_CHECKS, is_private_index() and the private/deferred selector
outputs are gone; the selector now emits `checks` and `docker_checks`
- check-tests-private-index is removed and vsphere joins the ordinary matrix,
so it runs on pull requests like every other suite
- ci-success loses the private-index arm and the deferral notice
- vsphere/tox.ini no longer documents a wheelhouse that does not exist
No job in this workflow now has a secret in scope, which is a stronger and much
simpler property to maintain than any arrangement that held one.
Verified: the four pins resolve from public PyPI alone, and the full vsphere
suite passes against them -- 58 passed. actionlint, shellcheck and Zizmor clean.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
`checksdev validate dep` cross-checks every pin in a check's requirements.in against agent_requirements.in, the manifest of what the agent embeds. That file still listed vsphere-automation-sdk, so the four replacement packages were reported missing from the embedded environment. Replace the single old entry with the four public ones. Verified locally: `checksdev validate dep` exits 0, as do config, manifest, metadata and service-checks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
STAC-25544: move vsphere to the publicly-published VMware SDK
STAC-25531: run the five Docker-daemon test suites in GitHub Actions
LouisLotter
reviewed
Aug 6, 2026
LouisLotter
approved these changes
Aug 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pipeline has never had a failure notification — the retired
.gitlab-ci.ymlhad no notify job and no.cerberusdirectory, so a broken release branch is silent. STAC-25510 is what that costs: process-agent's image publishing broke and nobody noticed for 12 days.Adds a reusable
cerberus-notify.yml(structurally identical to the one process-agent gained in STAC-25519) and calls it as a terminal job offci-success.Worth a look:
push:trigger that means the release branch and release tags. PR failures already have an owner watching, and on a public repo anyone can open a deliberately-failing PR — notifying on those hands the CI channel a spam vector. Keyed ongithub.event_namerather than a hardcoded ref so the next release-branch bump touches one place, not two.action: notify, neverblock. Blocking locks the branch, needs the Cerberus App installed here, and mutates the branch protection pulumi-infra now owns (STAC-25522).Depends on StackVista/pulumi-infra#277 for the repo-level secrets (the org copies are
visibility=private; this repo is public). Until that merges the job warns and exits 0 rather than adding a second red job to an already-failed run.Validated: actionlint + shellcheck and Zizmor 1.28.0 clean; payload dry-run produces valid JSON for both a normal commit and the empty-
head_committag case.https://stackstate.atlassian.net/browse/STAC-25533