Instruments cache is never saved when valgrind is built from source (e.g. ubuntu-26.04) #248

Summary

On GitHub-hosted ubuntu-26.04 runners the CodSpeed instruments cache is never saved, so the valgrind build is repeated on every single run.

The root cause looks like it is in the runner rather than in this action (see "Root cause" below). I'm filing it here first because this is where the misleading warning shows up — happy to move it to CodSpeedHQ/codspeed if you prefer.

Environment

• CodSpeedHQ/action@v5, codspeed runner v5.4.2
• runs-on: ubuntu-26.04, mode: instrumentation
• Example workflow (the benchmark job): https://github.com/trim21/bencode-py/blob/master/.github/workflows/tests.yaml

What happens

CodSpeedHQ/valgrind-codspeed only publishes packages for ubuntu-22.04 and ubuntu-24.04 (latest release 3.26.0-0codspeed8), so on 26.04 the runner falls back to a source build:

##[group]Running the benchmarks
##[warning]CodSpeed does not publish a valgrind package for ubuntu 26.04
Building valgrind-codspeed from source, this can take a few minutes

The instruments cache is configured as usual, but it misses on every run:

  key: codspeed-instruments-instrumentation-ubuntu-26.04-X64-7.0.0-1012-azure-5.4.2
  restore-keys: codspeed-instruments-instrumentation-ubuntu-26.04-X64-7.0.0-1012-azure-
Cache not found for input keys: codspeed-instruments-instrumentation-ubuntu-26.04-X64-7.0.0-1012-azure-5.4.2, codspeed-instruments-instrumentation-ubuntu-26.04-X64-7.0.0-1012-azure-

and at the end of the job the Cache CodSpeed instruments post step reports that there is nothing to save, so the cache is never populated:

##[start-action display=Cache CodSpeed instruments;id=...]
Post job cleanup.
[warning]Path Validation Error: Path(s) specified in the action for caching do(es) not exist, hence no cache is being saved.

There is no Cache saved with key: codspeed-instruments-... line anywhere in the logs, i.e. this is not a "first run misses, later runs hit" situation.

Observed on three consecutive runs:

• https://github.com/trim21/bencode-py/actions/runs/38151647297/job/114507418793
• https://github.com/trim21/bencode-py/actions/runs/38150422237/job/114503833317
• https://github.com/trim21/bencode-py/actions/runs/38149322379/job/114500884089

The Run benchmarks step takes ~148 s per run, dominated by that build. On images that do have a prebuilt package the cache works — see e.g. developmentseed/async-tiff#254, which logs Cache hit for restore-key: codspeed-instruments-simulation-Linux-X64-6.14.0-1017-azure-4.10.6.

Root cause (in the runner)

The instruments cache is the feature from CodSpeedHQ/codspeed#121, implemented in CodSpeedHQ/codspeed#129 and #140. In the runner:

• src/executor/helpers/apt.rs, install_cached(): restores from the cache directory, runs the install closure, then saves the files of the installed packages (from dpkg -L). Add caching for tools installed by the runner codspeed#129 describes it as "limited to apt based system".
• src/executor/valgrind/setup.rs, install_valgrind_from_package(): the published-package path, and the only caller that passes setup_cache_dir to apt::install_cached(...).
• src/executor/valgrind/setup.rs, try_install_from_source(): the path taken when no package is published. It calls build_from_source::build_and_install(), which never receives setup_cache_dir, so it neither restores from nor saves to the cache.
• src/executor/valgrind/build_from_source.rs: clones the sources into a TempDir, runs autogen.sh / configure / make, installs with sudo make install, then drops the temporary directory. Nothing is ever written to the cache directory.

That is why the directory ~/.cache/codspeed-action never exists on a distribution without a published package, and why this action's cache step can only warn and save nothing.

For the record, the if [ "true" = "true" ] && [ -n "~/.cache/codspeed-action" ] seen in the logs is just the interpolated form of the cache-instruments and instruments-cache-dir inputs of this action (defaults true and ~/.cache/codspeed-action), so that part is fine — I initially suspected it, it isn't the problem.

Suggested fixes

1. Runner: make the source build participate in the setup cache, e.g. install into the cache directory (make install DESTDIR=...) and on the next run restore it and skip the build when it matches the expected version/triplet.
2. Runner or action: when the instruments cache is not going to be populated, don't configure it (or disable it), so runs don't end with a warning that reads like a user misconfiguration.
3. valgrind-codspeed: publish packages for ubuntu-26.04, or document which distributions have one.

I can test a fix on our repository (the benchmark job) if that helps.