ci: disable the testcontainers reaper in go-unit-tests - #2554
Merged
Conversation
go-unit-tests intermittently fails with two Postgres-backed packages timing out before any container starts: reaper: from container "1c88b27d": wait for reaper 1c88b27d: context deadline exceeded FAIL github.com/kagent-dev/kagent/go/core/pkg/cli/db/migrate FAIL github.com/kagent-dev/kagent/go/core/pkg/migrations `go test ./...` runs each package as its own process, so the processes that use internal/dbtest all contend for one session-scoped Ryuk reaper. In the failing runs earlier packages log "Reaper obtained" for that exact container and later ones then wait 60s for the same ID and give up, so no Postgres container is ever created and the tests fail without evaluating anything. Ryuk exists to reap containers leaked by a test run that dies without cleaning up. GitHub runners are ephemeral and discarded after the job, so that cleanup has no value here, and dbtest already terminates its containers via t.Cleanup. Disabling the reaper removes the contended singleton without giving up anything CI relies on. Signed-off-by: Jonathan Jamroga <jjamroga@gmail.com>
EItanya
approved these changes
Aug 25, 2026
EItanya
enabled auto-merge (squash)
August 25, 2026 13:20
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
go-unit-testsintermittently fails with two Postgres-backed packages timing out before any container starts:Neither is an assertion failure — both are 60s timeouts waiting on the testcontainers Ryuk reaper, so no Postgres container is ever created and the tests fail without evaluating anything.
Cause
go test ./...runs each package as its own process, so every process usinginternal/dbtestcontends for one session-scoped Ryuk reaper. The logs show the race directly — earlier packages obtain that exact container, then later ones wait 60s for the same ID and give up:Observed on unrelated PRs and on
main:1c88b27d)main, reaper69c2d478) — same two tests, same errorFix
Set
TESTCONTAINERS_RYUK_DISABLED=trueon the step.Ryuk exists to reap containers leaked by a test run that dies without cleaning up. GitHub runners are ephemeral and discarded after the job, so that cleanup has no value here, and
dbtestalready terminates its containers viat.Cleanup. Disabling the reaper removes the contended singleton without giving up anything CI relies on.Scoped to
go-unit-tests: it is the only job running testcontainers-backed tests. Thee2ejob runs justcore/test/e2eagainst a Kind cluster.🤖 Generated with Claude Code