STAC-25531: run the five Docker-daemon test suites in GitHub Actions - #367
Merged
Merged
Conversation
Phase 2 of the GitLab->GitHub CI migration. The four splunk_* suites and
stackstate_checks_dev need a live Docker daemon, so phase 1 left them out
entirely -- they currently run nowhere.
They cannot run inside the BCI job container. get_docker_hostname() resolves
to `localhost` unless DOCKER_HOST is set, and compose publishes its ports on
the Docker host, so the test process must share the daemon's network
namespace. GitLab avoided this with a docker:dind service whose hostname
resolved identically from both sides; on GitHub the equivalent is to run
directly on the runner. Hence a separate `check-tests-docker` job with no
container and actions/setup-python, rather than extending the existing matrix.
The fixtures pulled their images through registry.tooling.stackstate.io,
which answers 401 from the public internet -- a credential a public repo's
pull_request jobs must never hold. All five paths were /docker/ proxy-cache
entries, so they are repointed at the unambiguous Docker Hub originals via
${SPLUNK_IMAGE}/${VAULT_IMAGE} with public defaults, leaving self-hosted and
local users able to override back to the proxy.
The selector gains a DOCKER_CHECKS set and a third output. splunk_base is a
build dependency of the other three splunk suites, so a change to it fans out
to all four; the reverse does not hold.
setup_artifactory_docker.sh is deliberately not ported -- it existed only to
log Docker in to that proxy.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The five Docker suites failed on their first GitHub run with `FileNotFoundError: 'docker-compose'`. Compose v1 -- the standalone Python binary -- reached end of life in July 2023 and is no longer present on the hosted runner images; they ship v2 as the `docker compose` subcommand. The GitLab runner image still carried v1, which is why this never surfaced there. Add `compose_command()`, which prefers `docker compose` and falls back to `docker-compose`, so developers still on an older toolchain keep working, and route all four call sites through it. `DOCKER_COMPOSE_COMMAND` overrides the detection. The result is cached; container-name log checks are untouched, only the compose-file path changes. The two test modules shelled out to `docker-compose` directly to set up and tear down their own fixtures, so they go through the resolver too. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
compose_file_active() parsed the human-readable `ps` table, scanning for the row of dashes that Compose v1 printed between the header and the service rows. Compose v2 does not emit that separator, so the function returned False unconditionally and the two remaining stackstate_checks_dev tests failed even once the command itself resolved correctly. Use `ps -q` instead: one container id per running service, empty when the project is down. That output is stable across both generations, so it needs no version branching. Verified against a live daemon -- False before up, True after up, False after down. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
Author
|
Green — all five suites pass on run 31108281630. Splunk comes up well inside the timeout on a hosted runner, and runner Python 3.13 behaves the same as the BCI 3.13.13 the matrix uses. Getting there took two fixes the first run exposed, both in the toolkit rather than the workflow:
|
vsphere pinned `vsphere-automation-sdk==1.82.0`, which VMware never published
to public PyPI -- the name is squatted there by an unrelated 0.0.1 placeholder
-- so it was mirrored into the private GitLab package registry and pulled with
a credential.
That credential could not be held safely. A `pull_request` run executes the
pull request's own copy of the workflow and of every script it calls, so a run
holding a secret can always be made to disclose it. The only sound answer was
to withhold it, which meant vsphere did not run on pull requests at all and
was verified only after merge -- a real coverage gap, accepted at the time
because the alternatives cost more.
VMware publishes the same SDK to public PyPI under renamed packages, actively
maintained: pyvmomi, vmware-vcenter, vmware-vapi-runtime and
vmware-vapi-common-client, all at 9.1.0.0. Between them they satisfy every
import the check makes -- pyVmomi/pyVim, vmware.vapi.* and com.vmware.* alike.
So the pin, the credential, the separate job and the coverage gap all go
together:
- requirements.in names the four public packages
- fetch_private_wheels.sh is deleted
- PRIVATE_INDEX_CHECKS, is_private_index() and the private/deferred selector
outputs are gone; the selector now emits `checks` and `docker_checks`
- check-tests-private-index is removed and vsphere joins the ordinary matrix,
so it runs on pull requests like every other suite
- ci-success loses the private-index arm and the deferral notice
- vsphere/tox.ini no longer documents a wheelhouse that does not exist
No job in this workflow now has a secret in scope, which is a stronger and much
simpler property to maintain than any arrangement that held one.
Verified: the four pins resolve from public PyPI alone, and the full vsphere
suite passes against them -- 58 passed. actionlint, shellcheck and Zizmor clean.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
`checksdev validate dep` cross-checks every pin in a check's requirements.in against agent_requirements.in, the manifest of what the agent embeds. That file still listed vsphere-automation-sdk, so the four replacement packages were reported missing from the embedded environment. Replace the single old entry with the four public ones. Verified locally: `checksdev validate dep` exits 0, as do config, manifest, metadata and service-checks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
STAC-25544: move vsphere to the publicly-published VMware SDK
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 2 of STAC-25142. The four
splunk_*suites andstackstate_checks_devneed a live Docker daemon, so phase 1 left them out — they currently run nowhere.Why a separate job, not the existing matrix.
get_docker_hostname()returnslocalhostunlessDOCKER_HOSTis set, and compose publishes ports on the Docker host. The test process therefore has to share the daemon's network namespace, which rules out the BCI job container. GitLab hid this behind adocker:dindservice whose hostname resolved the same from both sides. So:check-tests-dockerruns directly on the runner withactions/setup-python.Registry change. All five fixtures pulled through
registry.tooling.stackstate.io, which answers401publicly — a credential this public repo'spull_requestjobs must not hold. Every path was a/docker/proxy-cache entry, so they now point at the Docker Hub originals via${SPLUNK_IMAGE}/${VAULT_IMAGE}, defaulted publicly and overridable back to the proxy.splunk_baseis a build dependency of the other three splunk suites, so the selector fans a change to it out to all four.Stacked on #366 — retarget to
stackstate-7.78.2once that merges.Validated: actionlint + shellcheck + Zizmor clean; selector routing exercised in a scratch worktree for push,
splunk_base,splunk_metric, andkubeletchanges.