Skip to content

Reduce benchmark variance from CodSpeed environment #14872

Description

@xtqqczze

CodSpeed benchmark results appear to have some variance between runs, which can make small performance changes harder to distinguish from noise.

One possible source of this is CPU-dependent glibc optimizations. GitHub Actions runners can be backed by different CPU types even when using the same runner image, and glibc uses runtime CPU feature detection to select optimized implementations. CodSpeed has documented this as a source of benchmark variance, including differences caused by glibc’s malloc implementation. (⁠CodSpeed)

We should investigate setting GLIBC_TUNABLES for the CodSpeed benchmark environment to disable CPU features that can cause glibc to select different implementations depending on the underlying runner CPU. In particular, it may be worth investigating whether disabling the x86-64-v4 feature set provides a sufficiently consistent baseline.

cc: @oech3

Activity

  1. oech3 commented on Sep 26, 2026

    @oech3
    Contributor

    It would be better to ask upstream(CodSpeed) to disable v4 by default instead of setting it at here.

  2. changed the title [-]Reduce CodSpeed benchmark variance[/-] [+]Reduce benchmark variance from CodSpeed environment[/+] on Sep 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions