Skip to content

STAC-25519 Report failed master builds to Slack via Cerberus - #250

Merged
LouisParkin merged 1 commit into
masterfrom
STAC-25519-cerberus-notify
Aug 4, 2026
Merged

STAC-25519 Report failed master builds to Slack via Cerberus#250
LouisParkin merged 1 commit into
masterfrom
STAC-25519-cerberus-notify

Conversation

@LouisParkin

Copy link
Copy Markdown

Restores the notify-on-master-fail job we lost in the GitLab → GitHub migration. Right now a red master on this repo is completely silent.

That is not hypothetical. Image publishing to Quay broke on 23 July and went unnoticed for 12 days (STAC-25510) — master was failing the whole time and nothing told us.

Part of the GitLab CI parity work under STAC-25142 / STAC-25519.

What this does

Adds a reusable cerberus-notify.yml and calls it from ci.yml as a terminal job:

cerberus-notify:
  needs: ci-success
  if: failure() && github.ref == 'refs/heads/master' && github.event_name == 'push'

It hangs off ci-success, which already aggregates every other job, so all failures funnel through one notification instead of one per job. Master pushes only — PRs stay quiet.

Follows the cerberus-block-on-master-fail convention from StackVista/stackstate, sending platform: github so Cerberus builds GitHub pipeline/commit URLs rather than GitLab ones. channel is deliberately omitted: Cerberus resolves it as util.GetOrDefault(req.Context, "channel", s.Channel), so leaving it out uses the Lambda's own default (per @fzhdanovSLACK_CHANNEL_ID is no longer needed caller-side).

Two decisions worth reviewing

action: notify, not block. Policy for migrated repos is notify by default. Blocking would lock master via lock_branch on every failure, additionally requires the Cerberus GitHub App to be installed on this repo, and mutates Pulumi-managed branch protection — a pulumi-infra apply during a block window silently unlocks it.

It ships before the secret exists, on purpose. CERBERUS_LAMBDA_URL is an org secret with visibility=private, and this repo is public, so it does not reach us. Widening org visibility is not an option — the Cerberus endpoint is unauthenticated, so the URL is the capability. It needs to arrive as a repo-level secret from pulumi-infra, which is tracked separately and blocked on the same missing stack config key as stackstate-agent.

Until that lands the workflow emits a warning annotation and exits 0, rather than adding a second red job to a run that is already failing. The secret is declared required: false for a specific reason: passing ${{ secrets.X }} for a secret the repo lacks yields an empty string, and GitHub rejects that against a required secret, failing the call before the guard can run — exactly the failure mode this is meant to avoid. So merging this now is safe, and it starts working the moment the secret appears, with no further change here.

Validation

  • actionlint type-checks the reusable-workflow call. I confirmed the check is real with a negative control — renaming suitesuitte correctly produced input "suitte" is not defined. Clean on the new files; the two remaining findings are pre-existing in other jobs.
  • zizmor --collect=workflows,actions,dependabot . → no findings.
  • Payload exercised against a stubbed endpoint with a multi-line commit message containing "quotes", $(whoami) and backticks. The message goes through env: rather than being interpolated into the script, so it is JSON-escaped, not executed, and only the first line is used as the title. Also verified: missing secret → warn + exit 0 with no call made; empty commit message → no set -u trip.

Not covered here

The other two parity gaps have their own tickets and both need a decision before any code: STAC-25520 (S3 binary publishing) and STAC-25521 (beest verification trigger).

The GitLab pipeline had a `notify-on-master-fail` job that posted to the
team's CI channel through Cerberus. The GitHub migration dropped it, so a
red master became completely silent.

That gap has already cost us: image publishing to Quay broke on 2026-07-23
and nobody noticed for 12 days (STAC-25510) because there was nothing to
tell us.

Adds a reusable cerberus-notify.yml and hangs it off ci-success as a
terminal job, so every failure funnels through one notification rather than
one per job. Sends `platform: github` so Cerberus builds GitHub pipeline and
commit URLs, and omits `channel` so the Lambda's own default applies.

`action: notify`, not `block`: policy for migrated repos is notify by
default, and blocking would lock master on every failure, need the Cerberus
GitHub App installed here, and fight Pulumi over branch protection.

CERBERUS_LAMBDA_URL does not reach this repo yet. The org-level secret is
visibility=private and this repo is public, and widening it is not an option
because the endpoint is unauthenticated -- the URL is the entire capability.
It needs to arrive as a repo-level secret from pulumi-infra. Until then the
workflow emits a warning annotation and exits 0 rather than adding a second
red job to an already failed run.

Verified locally: actionlint type-checks the reusable call (confirmed with a
negative control), zizmor reports no findings, and the payload was exercised
against a stubbed endpoint with a multi-line commit message containing
quotes and shell metacharacters -- it is passed via env, not interpolated,
so it is escaped rather than executed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@LouisParkin
LouisParkin requested a review from craffit August 4, 2026 11:40
@LouisParkin
LouisParkin merged commit 2e70bb5 into master Aug 4, 2026
8 checks passed
@LouisParkin
LouisParkin deleted the STAC-25519-cerberus-notify branch August 4, 2026 13:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants