Skip to content

ci: disable the testcontainers reaper in go-unit-tests - #2554

Merged
EItanya merged 1 commit into
kagent-dev:mainfrom
jjamroga:ci-disable-ryuk
Aug 25, 2026
Merged

ci: disable the testcontainers reaper in go-unit-tests#2554
EItanya merged 1 commit into
kagent-dev:mainfrom
jjamroga:ci-disable-ryuk

Conversation

@jjamroga

Copy link
Copy Markdown
Collaborator

Problem

go-unit-tests intermittently fails with two Postgres-backed packages timing out before any container starts:

migrate_test.go:255:         failed to start postgres container: ... reaper: from container "1c88b27d": wait for reaper 1c88b27d: context deadline exceeded
migration_000007_test.go:32: startTestDB: start container: ...     reaper: from container "1c88b27d": wait for reaper 1c88b27d: context deadline exceeded

FAIL github.com/kagent-dev/kagent/go/core/pkg/cli/db/migrate  61.298s
FAIL github.com/kagent-dev/kagent/go/core/pkg/migrations       114.543s

Neither is an assertion failure — both are 60s timeouts waiting on the testcontainers Ryuk reaper, so no Postgres container is ever created and the tests fail without evaluating anything.

Cause

go test ./... runs each package as its own process, so every process using internal/dbtest contends for one session-scoped Ryuk reaper. The logs show the race directly — earlier packages obtain that exact container, then later ones wait 60s for the same ID and give up:

19:14:39 ⏳ Waiting for Reaper "1c88b27d" to be ready
19:14:40 🔥 Reaper obtained from Docker for this test session 1c88b27d
19:15:00 ⏳ Waiting for Reaper "1c88b27d" to be ready      <- times out

Observed on unrelated PRs and on main:

Fix

Set TESTCONTAINERS_RYUK_DISABLED=true on the step.

Ryuk exists to reap containers leaked by a test run that dies without cleaning up. GitHub runners are ephemeral and discarded after the job, so that cleanup has no value here, and dbtest already terminates its containers via t.Cleanup. Disabling the reaper removes the contended singleton without giving up anything CI relies on.

Scoped to go-unit-tests: it is the only job running testcontainers-backed tests. The e2e job runs just core/test/e2e against a Kind cluster.

🤖 Generated with Claude Code

go-unit-tests intermittently fails with two Postgres-backed packages timing
out before any container starts:

  reaper: from container "1c88b27d": wait for reaper 1c88b27d: context
  deadline exceeded

  FAIL github.com/kagent-dev/kagent/go/core/pkg/cli/db/migrate
  FAIL github.com/kagent-dev/kagent/go/core/pkg/migrations

`go test ./...` runs each package as its own process, so the processes that
use internal/dbtest all contend for one session-scoped Ryuk reaper. In the
failing runs earlier packages log "Reaper obtained" for that exact container
and later ones then wait 60s for the same ID and give up, so no Postgres
container is ever created and the tests fail without evaluating anything.

Ryuk exists to reap containers leaked by a test run that dies without
cleaning up. GitHub runners are ephemeral and discarded after the job, so
that cleanup has no value here, and dbtest already terminates its containers
via t.Cleanup. Disabling the reaper removes the contended singleton without
giving up anything CI relies on.

Signed-off-by: Jonathan Jamroga <jjamroga@gmail.com>
@EItanya
EItanya enabled auto-merge (squash) August 25, 2026 13:20
@EItanya
EItanya merged commit 6dbe028 into kagent-dev:main Aug 25, 2026
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants