The shared Linux toolchain image for pace, morph, Lightweight, and
formula-cpp.
It includes GCC 14, Clang 22, CMake, Ninja, pace-cc, and pace-kv. It is
published to GHCR by .github/workflows/publish.yml. Consumers use the moving
latest tag, which every successful master build advances.
Consumers reference ghcr.io/lastrada-software/cpp-ci:latest and never a
pinned digest. Read the usage guide before migrating a workflow.
In particular, every CMake configure must enable the cache with
-DUSE_COMPILER_CACHE=ON and set four pace-cc environment variables
(PACE_ADDR, PACE_TOKEN, PACE_SOURCE_DIR, PACE_BINARY_DIR). An
authenticated pace-kv alone does not distinguish cache readers from writers.
publish.yml runs in four stages:
- resolve pins an immutable Ubuntu base-image digest and the exact
pacemastercommit, and reads the warm-up groups fromwarm-targets.json. - build builds the toolchain image with no seed, smoke-tests it, and pushes
it as
<sha>-base. Pull requests stop here. - warm runs one job per group in parallel, inside the base image. Each job
starts from the previously published seed for its group, so only translation
units whose inputs changed are recompiled. Each job pushes its result as a
candidate seed image,
cpp-ci-seed-pace-<group>:<sha>. A store that came out byte-identical to the previous seed re-tags the previous seed image, so its layer is not uploaded again. - publish smoke-tests the candidate seeds with the base image's
pace-kv, adds them to<sha>-baseas one layer per group, and pushes<sha>directly from buildkit, so layers the registry already holds are skipped. Onmasterit also moveslatestand each group's seed to the new candidates.
If any warm-up group fails, publish does not run, and latest stays on the
previous image with its previous seeds. Consumers pick up each new image
automatically, so no changes are needed in their repositories.
The warmer builds each configured project's presets with the compile cache
enabled, into single-file pace-kv stores. Each store is one group of presets,
and the stores are baked into the image as /opt/pace/seed-cache/<project>/<group>.cow.
A consumer copies the store for the group it builds.
Status: pace is the first project onboarded. The other three are not yet
onboarded. The blockers are tracked upstream:
Lightweight: a test compile definition embeds an absolute build-tree path, so cache entries are not shared across checkouts. See Lightweight#693.formula-cpp: no cache integration. See formula-cpp#36.morph: not yet measured.
warm-targets.json lists the groups. pace has three:
| Group | Presets |
|---|---|
gcc |
gcc debug, gcc release, gate-gcc-release |
clang |
clang debug, clang release |
clang-extra |
gate-clang-debug, clang tracy |
sanitizers |
clang asan-ubsan, clang tsan |
Each preset carries configure_args, which must match the flags pace's
build.yml configures that preset with: gcc-release and clang-release as
the linux job does, clang-debug as clang-asan-ubsan does, and clang-tsan
as clang-tsan does. A flag that changes a compile definition changes the cache
key of every translation unit it reaches (PACE_ENABLE_TLS adds a PUBLIC define
to the Pace library, so it reaches all of them), and a seed warmed without it
never hits.
Coverage presets are excluded, because pace-cc binds coverage objects to the
checkout root and they never hit from another checkout. Windows presets are out
of scope.
scripts/warm-project.sh runs inside the base image:
- starts a local
pace-kvon a single-file store, with an on-disk cap ofWARM_MAX_DISK(default4g), which evicts least-recently-used entries; - if the previous seed exists, starts from a copy of it. A store the new
pace-kvrefuses (for example after a storage format change) is discarded, and the group starts empty; - for each preset, configures with
-DUSE_COMPILER_CACHE=ON -DENABLE_TIDY=OFFplus the preset'sconfigure_args, and builds it. clang-tidy is off because it re-runs on every file, even on a cache hit, and adds nothing to the store. It doesn't change the compile command, so the seeded entries still hit for consumers that build with tidy on; - fails fast: a failing configure or build stops the group, and a configure that disables the cache fails it too;
- fails if
pace-ccrecords no statistics, so an uncached warm-up is never published; - stops
pace-kvwith SIGTERM, so batched writes are flushed, and reopens the store once.
The project checkout sits at the same path inside the container as on the host,
and that path is pace CI's own checkout path (PACE_CI_WORKSPACE,
/home/runner/work/pace/pace), with CPM's source cache inside it at
.cache/CPM as pace CI keeps it. Some compiles depend on these paths in ways
pace-cc cannot map away: pace's generated tests/CheckoutPaths.hpp spells the
checkout path, and CMake names the object files of zstd, lz4 and catch2's main by
an absolute path when CPM's cache is outside the source tree. Warmed at any other
path, those 43 gcc-release units miss on every CI run.
- Parallel groups, one store each.
pace-kvcannot merge stores, so each group job writes its own store. Groups run in parallel to cut wall-clock time. - Incremental reuse. Each group starts from the previously published seed, so only changed translation units are recompiled. The disk cap bounds the reused store.
- Promotion on
masteronly. The candidate seeds become the starting point for the next warm-up only when the image that contains them is published. Branch runs never movelatest, so their seeds cannot become a starting point. - Build only, no tests. The seed depends on what compiled, not on which
tests passed, and
ctestadds no cache entries. It used to run for the gcc and clang release presets, as warnings only, and took about 7.5 minutes per group even when every compile hit (run 42597225082), which made it the critical path of a warm run. A build failure still fails the warm-up. - Groups stay under the job limit. The clang group ran for 5 h 30 min in
run 42095583072 and was cancelled by the 330-minute
timeout-minutes, which skippedpublish. Its gate and tracy presets moved toclang-extra. - Warm-up runs inside
publish.yml, not on a separate schedule. The seed is produced by the samepace-cc,pace-kv, and compilers it is baked beside. Cache keys include the compiler banner, andpace-kvrefuses stores written with a different storage format. - Credentials. The warm-up uses a local
pace-kv, so it needs noPACE_TOKEN.paceis a public repository, so no checkout token is needed. - Runner. The warm-up runs on
ubuntu-24.04unless theWARM_RUNNERrepository or organization variable selects another runner. A hosted runner may not have the time or disk for a full warm-up, so a larger runner may be needed.
These come from building on the same machine, against a local pace-kv store.
| Build | Compiles | Hits | Notes |
|---|---|---|---|
pace gcc-release, cold, checkout A |
1,173 | 0 | 541 s build |
pace gcc-release, checkout B, same store |
1,173 | 1,149 (98.0%) | 52 s build; 23 misses not investigated |
pace gcc-release, starting from the previous seed |
1,173 | 1,172 (100%) | incremental reuse |
Lightweight gcc-release, checkout B, same store |
243 | 104 (43%) | cause: test compile definition with absolute path |
Lightweight after that define is made relative |
228 | 226 (99.1%) | counterfactual |
formula-cpp gcc-release, checkout B, launchers set by hand |
415 | 339 (81.7%) | no cache integration; prefix-map rules not applied |
The store size for pace has not been measured yet, so the 4g cap is a
placeholder. It should be set from the first real run.
- The 23
pacemisses are unexplained. A verbose run would show which translation units miss. - Warm-up time on hosted runners is unknown. Nine presets, two of them sanitizer builds, may exceed the time or disk of a standard runner. The groups shorten the wall-clock time, not the total work.
- A configure can fail intermittently on the cache exchange. Once, a configure
reported
fetch exchange failedand disabled the cache. It did not recur in the verbose re-run. The script now fails the group when the cache is disabled, so this cannot publish an uncached seed. - Compiler banner drift. If the
apt.llvm.orgclang 22 point release changes between daily builds, the cache is re-keyed every day. The base build pins the compiler version, so the banner should be compared between runs. - Seed size in every pull. The seeds are part of the image, so every job that pulls the image also pulls them. Reuse keeps the stores bounded by the disk cap, not by the number of runs.