Skip to content

ci: split the workspace test job and cache the PR Docker build - #495

Merged
samgbafa merged 3 commits into
mainfrom
ci/split-test-and-docker-cache
Nov 1, 2026
Merged

samgbafa merged 3 commits into
mainfrom
ci/split-test-and-docker-cache

Conversation

@samgbafa

@samgbafa samgbafa commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

What

CI-only (.github/workflows/rust.yml, .github/workflows/docker.yml; no Dockerfile, release.yml, release-plz.yml, tag/deploy/publish, logship, write-probe or personal-node change). Approved by Sam as "the free split" (audit: /var/tmp/ci-audit-2026-11-01/REPORT.md, F5).

  1. test is a 4-leg matrix with the same postgres:16-alpine service, env, iptables check, failure dump and step timeouts in every leg. cargo test --workspace is partitioned by cargo target kind, keeping --workspace so feature unification and build artifacts are identical:

    leg runs
    unit cargo test --workspace --lib
    integration-a-o PG-regression guard; cargo test --workspace --bins --test '[a-o]*'; cargo test --workspace --doc
    integration-rest cargo test --workspace --test '[!a-o]*' (exact glob complement: a new test file lands in exactly one leg)
    duckdb the two existing --features duckdb commands, unchanged

    A test-gate job keeps the old check name and fails unless the whole matrix succeeded.

  2. PR-only Docker mount cache. The Dockerfile's RUN --mount=type=cache dirs (cargo registry, /app/target) start empty on a hosted runner and are not part of the exported layer cache, so the cargo-chef cook layer cached nothing and every PR push recompiled all deps plus the app in release. Two new steps, both if: github.event_name == 'pull_request': actions/cache + reproducible-containers/buildkit-cache-dance (pinned to the v3.3.2 commit SHA) restore/inject/extract those mounts, keyed per commit. Image, target, Dockerfile, cache-from/cache-to, and every push/tag/release/dispatch path are untouched.

Job graph, old vs new (ci workflow; docker.yml and others unchanged)

old: threshold | build | test (workspace, PostgreSQL-backed) ~26.6 min | clippy | fmt | capabilities | m1-realdata | lockfiles | at-rest harness | w5 lockfile
new: threshold | build | clippy | fmt | capabilities | m1-realdata | lockfiles | at-rest harness | w5 lockfile
     test (..., unit) ~9.4 | test (..., integration-a-o) ~8.3 | test (..., integration-rest) ~7.4 | test (..., duckdb) ~9.4 (warm)
     └─ test (workspace, PostgreSQL-backed)   <- gate, same name as before, ~3 s

Renames: none lost. test (workspace, PostgreSQL-backed) now is the gate job; new checks test (workspace, PostgreSQL-backed, unit | integration-a-o | integration-rest | duckdb). Job id test is now the matrix; test-gate is new.

Identical test coverage

Old single job (main push run 41059168544, same source tree as this PR's base d7f88e23) vs the four new legs (PR run 41267197637), compare2.py, multiset of executed test <name> ... <status> lines plus per-binary Running/Doc-tests/test result lines:

executed test lines   before/after: 3548 3548 | workspace 2366/2366, duckdb-core 1160/1160, duckdb-node 22/22
status counts         before/after: ok 3533, ignored 15 both
test binaries run     before/after: 82 82   (82 "test result" lines both)
only before: 0   only after: 0   result-line multiset diff: none   -> IDENTICAL

(The log merges stderr Running lines with stdout test lines out of order, so a per-binary attribution script shows false mismatches in the old single-job log too; the compare above does not depend on attribution.) Compare scripts/output: /var/tmp/tcnode-ci-sIYe/data/compare2.py, compare-output.txt.

Numbers (minutes)

BEFORE, 20 most recent all-green pull_request head SHAs (gh API, base main; PR-to-green = first run created to last workflow update across ci + Docker + Release):

median p90
test (workspace, PostgreSQL-backed) job 26.6 28.6
Docker build job 26.5 28.3
PR-to-green 28.6 32.5

AFTER, real runs on this PR. Measuring runs add a one-line comment to tinycloud-core/src/lib.rs (reverted; the final head diff is the two YAML files only) so Docker really recompiles; a source-identical head hits the existing layer cache from main and builds in about 4 min.

run head ci (longest test leg / ci wall) Docker build job PR-to-green
1 seed, no source change ac10fb5 duckdb leg cold 33.4 (cold DuckDB build, no cache) / 33.4 0.3 (layers all hit) 33.4
2 source change, mount cache cold 0f62606 unit 7.5, duckdb 10.6 (PR cache from run 1) / 10.7 28.0 (step 25.2, +extract/save 2.4) 29.0
3 source change, mount cache warm 6266b4c unit 9.4, duckdb 9.4 / 10.4 16.7 (step 12.4 vs 25.2 cold) 17.5
final head, attempt 1 3378e36 default legs slowed by a 10 min cache restore (unit 20.6) / 20.7 3.7 20.7
final head, attempt 2 (rerun) 3378e36 unit 9.1, rest 7.9, a-o 7.3, duckdb 22.1 (its PR-scoped cache was evicted: repo cache is at the 10 GB cap and main was writing release buildkit blobs) / 22.2 (attempt 1) 22.2

Honest reading: with warm caches, the test side drops from 26.6 to about 9.4 min and Docker from 26.5 to 16.7 min on a warm same-PR push, so PR-to-green goes about 28.6 -> 17.5. The audit's 9.2 ceiling is not reached: the release compile of the app is about 12 min even with deps cached, plus about 3 min of inject/extract. The first push of any PR is still a cold Docker build (about 28 min), because a PR's mount cache is private to that PR; seeding it from main would need a non-publishing job and is not done here. Runs 1, 3 and the final attempts are not a clean steady state: see risks.

Risks / not measured

  • Repo cache cap: 10.4 of 10 GB used before this PR. The duckdb leg adds a second large rust-cache key (rust-1.97.1-test-duckdb, cold until the first main push after merge saves it) and each PR push saves one mount-cache entry. If it is evicted, that leg is cold (about 22 min, same as the old job). The default legs keep the existing rust-1.97.1-test-default-duckdb key, warm on day one. I deleted this PR's own caches after measuring. A cap increase or cleanup is Sam's call.
  • Cache saves stay main-only for rust-cache (the temporary PR save used for measuring run 1 is removed from the final head).
  • Reviewer ran against logs from the TEMP measuring head; the final workflow differs only by that save-if.

Actions cost

Public repo: standard runners are $0 net. Runner-minutes per ci run go up: legs sum to about 34.4 min warm (565+498+442+561 s) vs 26.6 min median before, +7.8 min, plus the 3 s gate; one more PG service start and cache restore per leg. Docker adds about 3.3 min inject/extract/restore per PR push but saves about 11 min compile on warm pushes. At the $0.006/min list price the extra is about $0.05 per ci run gross, $0 billed.

Independent review

Spawned one reviewer agent (not me): VERDICT: APPROVE, no required changes. Full verdict: /var/tmp/tcnode-ci-sIYe/review-verdict.md (host path, agent host). Covers coverage/exactly-once, services/features, gate, no release/deploy path or Dockerfile change (docker steps cannot run on push/tag/release/dispatch), supply chain and cache risks. The three scripts that parse docker.yml (test-docker-tags.py, test-deploy-recovery.py, check-deployment-policy-probes.mjs) pass.

Merge order

Frozen order stands: 1.25.14 release, then #491 -> #489 -> #490 -> #492 -> #493 -> #494 (TC-2092); this merges after them and will be rebased when asked. Change is two workflow files only. Not merged.

@samgbafa
samgbafa force-pushed the ci/split-test-and-docker-cache branch from 6266b4c to 3378e36 Compare November 1, 2026 03:45
@samgbafa
samgbafa marked this pull request as ready for review November 1, 2026 04:32
@samgbafa
samgbafa merged commit b67ebe0 into main Nov 1, 2026
27 checks passed
@samgbafa samgbafa mentioned this pull request Nov 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant