STAC-25142 Add GitHub Actions lint + unit-test workflow (agent CI migration phase 1) - #444
STAC-25142 Add GitHub Actions lint + unit-test workflow (agent CI migration phase 1)#444LouisParkin wants to merge 40 commits into
Conversation
3cd1a54 to
7308812
Compare
76349d9 to
c2e9da2
Compare
…ration phase 1) Ports filename_linting + unbranded/branded unit_tests from .gitlab-ci-agent.yml to GitHub Actions, tracking the active stackstate-7.78.2 pipeline. - filename-linting on `docker` (cheap gate; only needs `git ls-files`). - unbranded/branded unit tests on `xlarge` inside the datadog_build image (tag 7af9194f), pulled via the read-only quay proxy. - 7.78.2 deltas vs 7.71.2: `--major-version` dropped from agent.build/inv test (removed upstream in DD 7.78), and branded runs scripts/verify_branding_literals.sh after fix_branding.sh. Test-execution policy (architect directive — minimize redundant runs): each suite runs once. filename-lint on every PR; the two heavy ~2h suites on pull_request, deduped via concurrency cancel-in-progress. They move to merge_group (run once at land) at phase 6; master/version-branch pushes then build+publish without re-running tests. Security: minimal contents: read, SHA-pinned checkout, persist-credentials false, no PATs, registry-proxy secret only via container.credentials, fork PRs skipped. Zizmor clean; actionlint clean apart from self-hosted labels. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…; conda needs bash)
…ds the 3.x version tag
…on phase 2) Port build_binaries and build_cluster_agent from .gitlab-ci-agent.yml to .github/workflows/build-binaries.yml: build the branded (StackState) production agent (+ rtloader) and cluster-agent in the datadog_build container on public xlarge runners, once per change (pull_request), deduped via concurrency. Carry over the GitLab pipeline speedups: self-contained jobs with no shared deps prerequisite (the Go module cache is untransportable between jobs and shared vendoring breaks the go.mod<->vendor reconcile), version.txt generated in-job from git, and a runner-local persistent-cache follow-up noted for the real warm-cache win. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…e image Speeds up the migrated stackstate-agent GitHub Actions build/test jobs by replacing each job's cold `go clean -modcache` + `inv deps` (go mod download + tidy) reconcile with a warm GOMODCACHE delivered as a prebuilt CI image, keyed by a content hash of the module graph. Mirrors the StackGraph ci-metadata / build-ci-image pattern. - .github/scripts/agent-godeps-cache-metadata.sh: content-addressed image tag from every go.mod/go.sum + go.work + modules.yml plus the Dockerfile/script themselves, so any graph or mechanism change rotates the tag. - .github/docker/godeps-cache/Dockerfile: FROM datadog_build, bakes external modules into GOMODCACHE via a per-module `go mod download` loop. - .github/workflows/godeps-cache.yml: reusable workflow (tag lookup + build only when missing); pushes to quay.io/stackstate (private), pulled by consumers through the registry.tooling quay proxy with the existing REGISTRY_* creds. - build-binaries.yml / lint-and-unit-tests.yml: add the godeps-cache job, repoint the 4 heavy jobs to the cache image, drop the per-job modcache reset + inv deps (keep `go work sync` / `go work vendor` against the warm cache). Zizmor clean (narrow unpinned-images ignore on the content-addressed container refs, which are effectively digest-pinned but computed per run). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
c2e9da2 to
c07d0c8
Compare
RUN ran under dash so the read -d '' loop never iterated; -mod=mod is rejected in workspace mode.
STAC-25429 Warm the Go module cache with a prebuilt, content-addressed CI image
STAC-25459 Wire inv check-mod-tidy into CI, gated on the godeps cache hash
… STAC-25142-agent-omnibus-deb
STAC-25142 Add omnibus DEB build and container image workflows
…ncel STAC-25494 stagger godeps-cache builds instead of gating on concurrency
STAC-25466 tidy the Go workspace manifests
…warns instead of failing
GitLab fans .gitlab-ci-agent.yml out twice from .gitlab-ci.yml (agent-x86 / agent-arm) with per-arch variables. Only the amd64 half had been ported. The arm64-xlarge-public ARC scale set landing on argocd-apps main (STAC-25492) removes the blocker, so the fan-out becomes an `arch` matrix here. Make the Go dependency cache image arch-aware. The metadata script hashed only the module graph plus BASE_IMAGE_TAG, and both datadog_build images share tag 7af9194f -- so the two arches would have computed the SAME cache tag and one arch's image would have satisfied the other's existence check. Fold the base image NAME into the hash and the arch into the tag, making that collision impossible by construction. godeps-cache.yml takes a required `arch` input, selects datadog_build_linux_x64 vs datadog_build_linux_arm64, and builds each arch NATIVELY on a runner of that architecture -- QEMU/binfmt would make the Dockerfile's module extraction unusably slow. The two arches are independent images rather than a multi-arch manifest: consumers reference them from `container.image`, which resolves against the runner's own architecture anyway. The metadata job stays on amd64, since hashing files and calling `docker manifest inspect` are arch-independent. build-binaries.yml and build-deb.yml matrix over arch, with `fail-fast: false` so an arm64 regression cannot abort the amd64 leg that has been green for weeks. Matrix legs upload artifacts concurrently and actions/upload-artifact v4 rejects a second upload to an existing name, so the cluster-agent and DEB artifacts are now per-arch. Cache-image stagger (STAC-25494) keeps one designated builder per arch: lint-and-unit-tests owns amd64, build-binaries owns arm64, everyone else waits. Both arches run on pull requests, matching GitLab, where agent-x86 and agent-arm shared the same commit-or-MR rule. Of the GitLab per-arch variables only ARCH carries over; see the build-deb.yml header for why PACKAGE_ARCH / DD_PKG_ARCH / STS_PKG_ARCH are not needed and why DOCKER_ARCH (aarch64, not arm64) matters only to the publishing lane. Lint and unit tests deliberately gain no arm64 lane: each suite runs once on the path to master. Validated with actionlint (only the expected unknown-self-hosted-label notes), zizmor (no findings) and shellcheck (clean). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
STAC-25461 report release-branch build failures to Slack via Cerberus
…into the arm64 lane
Merging STAC-25461 (PR #449) into this branch produced no textual conflict but a broken workflow: the new `cerberus-notify` job in build-binaries.yml and build-deb.yml declares `needs: godeps-cache`, and this branch splits that job into godeps-cache-amd64 / godeps-cache-arm64. Both files failed actionlint with "needs job godeps-cache which does not exist in this workflow". List both per-arch jobs instead. That also preserves the intent recorded in the STAC-25461 comment -- `needs` lists every job, not just the leaves, so an early cache failure that skips its dependents is still reported to Slack. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
STAC-25498 Add the arm64 build lane to the GitHub Actions pipeline
The Cerberus Lambda already knows which channel to post to, so passing one
from the workflow is redundant. `notify.go` resolves the channel as:
Channel: util.GetOrDefault(req.Context, "channel", s.Channel)
and `util.GetOrDefault` (internal/util/context.go:27) treats an empty or
whitespace value as absent, falling back to the Lambda's own SLACK_CHANNEL
environment variable. Since this repo does not hold the secret, the previous
code was sending `channel: ""` on every call and hitting that fallback
anyway -- the field never did anything.
Dropping it halves what pulumi-infra has to provision for this repo
(StackVista/pulumi-infra#258 now asks for CERBERUS_LAMBDA_URL alone), which
matters because org-level secrets default to visibility=private and are
therefore invisible to this PUBLIC repo.
The `channel` key is removed from the payload entirely rather than sent
empty; GetOrDefault makes the two equivalent, and omitting it states the
intent. A comment records how to reintroduce it if agent failures should
ever go somewhere other than the shared CI channel.
Verified: actionlint clean (only the pre-existing self-hosted runner-label
warnings), zizmor clean, and the jq payload still emits valid JSON in the
shape Cerberus expects.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
STAC-25461 Drop SLACK_CI_REPORT_CHANNEL from the Cerberus notify path
Added commit: Cerberus bearer token (STAC-25536)Heads-up for anyone who already reviewed — one commit added on top: StackVista/cerberus#4 (STAC-24889) adds authentication to the Cerberus Lambda. The endpoint was previously completely unauthenticated — anyone who knew the URL could lock/unlock a protected branch or post as the Cerberus Slack bot. Every non-Slack request must now carry That PR is still open, so Sending the header now is forward-compatible (the deployed Lambda ignores unknown headers), so this is correct either side of that deploy and needs no coordination. Fixing it here rather than as a follow-up PR because this branch hasn't merged yet. Changes
Validation
The identical fix shipped for |
Cerberus now requires CI callers to present a shared bearer token. Send it from the notify script. Suspend xtrace around the curl and drop --verbose: the script runs under set -exuo pipefail, so either would echo the token into the job log.
StackVista/cerberus#4 (STAC-24889) adds authentication to the Cerberus Lambda: every non-Slack request must carry Authorization: Bearer <ssm:/cerberus/api-token> compared constant-time in internal/auth/auth.go verifyBearer. Before it, the endpoint was completely unauthenticated -- anyone who knew the URL could lock or unlock a protected branch or post as the Cerberus Slack bot. That PR is still open, so the workflow added in STAC-25461 would work as written today. The moment cerberus#4 deploys, the unauthenticated POST gets a 401, curl --fail turns the notify job red, and no Slack message is sent -- exactly the silent-failure condition the job exists to remove. Sending the header is forward-compatible: the deployed Lambda ignores unknown headers, so this is correct either side of that deploy and needs no coordination. CERBERUS_API_TOKEN already exists as a repo-level secret alongside CERBERUS_LAMBDA_URL, so no pulumi-infra change is needed. It is declared `required: false` for the same reason as the URL -- passing `${{ secrets.X }}` for a secret the repo lacks yields an empty string, which GitHub rejects against a *required* secret and fails the call before the guard can warn. The guard now covers both, so a missing secret still warns and exits 0 rather than adding a second red job to an already-failed run. Also drops `curl --verbose`, which echoes request headers -- the Authorization header now carries the shared token. GitHub would mask it, but not emitting it beats relying on masking. The same fix shipped for stackstate-process-agent as STAC-25529. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ed1f889 to
689909a
Compare
STAC-24889: authenticate Cerberus calls
What
Phase 1 of the
stackstate-agentGitLab → GitHub CI migration (STAC-25142): ports the lint + unit-test jobs from.gitlab-ci-agent.ymlto a GitHub Actions workflow,.github/workflows/lint-and-unit-tests.yml.filename-lintingdockerfilename_linting— cheap gate; only needsgit ls-files, so the GitLab job'sinv deps/go work vendorboilerplate is droppedunbranded-unit-testsxlargeunbranded_unit_tests— fullinv agent.build --race+inv test, DataDog brandingbranded-unit-testsxlargebranded_unit_tests— same +./fix_branding.sh, StackState brandingAll three run inside
container: <proxy>/quay/stackstate/datadog_build_linux_x64:0cd01a13— the sameBUILD_IMAGEthe GitLab pipeline uses (produced bydatadog-agent-buildimages, pulled via the read-only quay proxy). No new runner class is needed:xlarge(6 CPU / 40Gi, DinD) maps to GitLabsts-k8s-xl-runner.Test-execution policy (architect directive — minimize redundant runs)
Each suite runs once on the path to master:
filename-lintingon every PR (fast).pull_request, deduped viaconcurrency: cancel-in-progress.merge_group(run once at land) and drop frompull_request; master/version-branch pushes then build + publish without re-running tests.Open question raised for architects (biggest compute lever, tracked in the migration scope doc): unbranded and branded are nearly the same ~2h job twice — do we need both on every path, or branded-as-gate + unbranded on the queue only?
Prerequisite for green checks
This is a public repo, so the read-only registry-proxy credentials must be available to it for the
container:image pull:vars.REGISTRY_HOST,vars.REGISTRY_USER,secrets.REGISTRY_PASSWORD. If the first run fails on the image pull, these need scoping to this repo (org secret scope, or a repo-level read robot as done fordatadog-agent-buildimages). Org secrets are never exposed to fork PRs, so fork PRs skip these jobs by design.Security
contents: readonly, SHA-pinnedactions/checkout,persist-credentials: false, no PATs, registry secret only viacontainer.credentials. Zizmor: clean. actionlint: clean apart from expected self-hosted-runner label warnings (docker,xlarge).Validation
container:-on-xlarge(manual DinD) path for the first time — that's the main thing this PR validates.Jira: https://stackstate.atlassian.net/browse/STAC-25142
🤖 Generated with Claude Code