CodSpeed benchmark results appear to have some variance between runs, which can make small performance changes harder to distinguish from noise.
One possible source of this is CPU-dependent glibc optimizations. GitHub Actions runners can be backed by different CPU types even when using the same runner image, and glibc uses runtime CPU feature detection to select optimized implementations. CodSpeed has documented this as a source of benchmark variance, including differences caused by glibc’s malloc implementation. (CodSpeed)
We should investigate setting GLIBC_TUNABLES for the CodSpeed benchmark environment to disable CPU features that can cause glibc to select different implementations depending on the underlying runner CPU. In particular, it may be worth investigating whether disabling the x86-64-v4 feature set provides a sufficiently consistent baseline.
cc: @oech3
CodSpeed benchmark results appear to have some variance between runs, which can make small performance changes harder to distinguish from noise.
One possible source of this is CPU-dependent glibc optimizations. GitHub Actions runners can be backed by different CPU types even when using the same runner image, and glibc uses runtime CPU feature detection to select optimized implementations. CodSpeed has documented this as a source of benchmark variance, including differences caused by glibc’s malloc implementation. (CodSpeed)
We should investigate setting
GLIBC_TUNABLESfor the CodSpeed benchmark environment to disable CPU features that can cause glibc to select different implementations depending on the underlying runner CPU. In particular, it may be worth investigating whether disabling thex86-64-v4feature set provides a sufficiently consistent baseline.cc: @oech3