runner-benchmarks

Transparent, reproducible benchmarks comparing GitHub-hosted runners with BuildPulse runners, on real open-source workloads. Every number we publish links to a run in this repo's Actions tab.

Method

• Same jobs, two runners: every workload runs identically on ubuntu-latest (GitHub-hosted, 4 vCPU) and bp-ubuntu-latest-x64-4x (BuildPulse, 4 vCPU / 16 GB).

• Real workloads, pinned versions (no synthetic loops):

  • node-build: vuejs/core at a pinned tag — install, build, unit tests
  • go-build: gohugoio/hugo at a pinned tag — full compile
  • docker-build: hugo's own multi-stage Dockerfile at the pinned tag, no cache

• Queue time is measured from the GitHub API (created_at → started_at per job), not self-reported.

• Medians over repeated runs. Single runs lie; we dispatch batches and report median and p95. The harvest script is in scripts/.

• Hardware receipts (hardware.yml): sysbench CPU/memory and fio disk numbers from the exact machines the workloads run on, same commands both sides.

• Cache, cold vs warm (docker-cache.yml): the warm build runs on a FRESH machine, so the speedup is the cache infrastructure, not a lingering daemon. GitHub gets its own recommended remote cache (buildx gha backend, run-scoped); BuildPulse uses the docker-builder pool's persistent layer cache on local NVMe.

CircleCI and Semaphore

BuildPulse machines also take CircleCI jobs (through a CircleCI self-hosted machine runner) and Semaphore jobs (through a Semaphore self-hosted agent). The same three workloads, pinned the same way, run on the vendor's own 4 vCPU machine and on a BuildPulse x64-4x joined through the vendor's runner.

• Configs: .circleci/config.yml and .semaphore/semaphore.yml, both generated by scripts/gen-vendor-configs.py.
• Same toolchain everywhere: Node 20.18.1 and Go 1.23.4 are installed from the same tarballs on every machine, so only the machine differs.
• 10 reps per machine, chained (rep k waits for rep k-1), medians reported.
• Job time is the vendor's own job duration, the number its bill is based on.
• Raw results, every rep of every job: results/circleci-2026-10-13.json and results/semaphore-2026-10-12.json. These runs were made from our own CircleCI and Semaphore projects with the configs here. To rerun them, connect a fork to CircleCI or Semaphore, create a resource class or agent type in BuildPulse, and set its name in the generator.

CircleCI, 4 vCPU (2026-10-13), median job time

Workload	CircleCI large	CircleCI large.gen2	BuildPulse x64-4x
Docker image build (hugo, no cache)	229.3 s	175.4 s	129.2 s
Node install, build, tests (vuejs/core)	80.9 s	57.4 s	47.8 s
Go compile (hugo)	60.3 s	40.0 s	35.5 s
CPU	Intel Xeon Platinum 8375C	AMD EPYC 9R14	AMD EPYC 9R45

Semaphore, 4 vCPU (2026-10-12), median job time

Workload	Semaphore f1-standard-4	BuildPulse x64-4x
Docker image build (hugo, no cache)	138.5 s	133.5 s
Node install, build, tests (vuejs/core)	51.5 s	45.5 s
Go compile (hugo)	34 s	35.5 s

Prices and what they mean per build are in the write-ups: CircleCI and Semaphore.

October 2026 suite

Three heavier workloads that look more like real CI. Each runs five times per runner, all five at once, on ubuntu-latest (GitHub-hosted, 4 vCPU / 16 GB) and BuildPulse 4 vCPU / 16 GB:

• rust.yml: aome510/spotify-player at a pinned commit. cargo fmt --check, cargo test and cargo clippy with a fixed feature set.
• typescript.yml: the opencode monorepo at a pinned commit. bun install into an empty cache, then bun turbo typecheck --concurrency=4.
• docker-build.yml: PostHog's production image at a pinned commit, built with buildx and --no-cache. BuildPulse side: bp-docker-builder-x64-4x.

A job's time is the sum of its steps named phase: ..., as GitHub records them. Setup that only one image needs (freeing disk, node-gyp) is not timed. scripts/medians.py <run id> prints the median per runner for any run.

Anyone can fork this and reproduce. If you get different numbers, open an issue.