Skip to content

Run tests right after each image build - #1576

Merged
rosbo merged 1 commit into
mainfrom
parallel-test-after-build
Oct 30, 2026
Merged

rosbo merged 1 commit into
mainfrom
parallel-test-after-build

Conversation

@rosbo

@rosbo rosbo commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

Previously, tests waited in a separate Test stage until all image builds (CPU, GPU, TPU) had finished. Since the GPU build is much slower, the CPU tests sat idle for a long time.

Now each branch runs its tests and diff in parallel right after its own build:

  • CPU: Test CPU Image, Test kaggle-environments (uses the CPU image) and Diff CPU Image
  • GPU: Test on T4x2 and Diff GPU Image
  • TPU: Diff TPU VM Image only, unchanged (TPU images are tested outside this pipeline, see b/336842777)

Declarative pipelines don't allow nested parallel blocks inside a parallel branch, so each branch uses a scripted parallel inside a single Test/Diff … Image stage. Timeouts and retry(2) are unchanged. Tests now run on the same agent that built the image, at the same time as the diff.

Label CPU/GPU Staging Images still waits for all branches to finish.

Tests and diffs now run in parallel immediately after their image is built,
instead of waiting for all builds to finish. Test kaggle-environments runs
alongside the CPU image test since it uses the CPU image.
@rosbo
rosbo requested a review from calderjo October 30, 2026 17:45
@rosbo
rosbo merged commit 6a20e87 into main Oct 30, 2026
9 checks passed
@rosbo
rosbo deleted the parallel-test-after-build branch October 30, 2026 18:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants