Skip to content

W0: go-scikit/refbench, the shared reference-benchmark harness #27

Open

Description

@tannevaled

Tracking issue for wave W0 of docs/performance-campaign.md: the shared reference-benchmark harness go-scikit/refbench.

Why

go run ./tools/perfinventory over the 14 stack repositories (#26) finds:

| **total** | | 2081 | 234 | 131 | 53 | 1663 | 15 |

That is 2081 public functions: 234 are timed against their reference by Go code, 131 are paired by outside scripts, 53 are timed alone, and 1663 are not timed. The harnesses that exist differ in statistic (best of 5, median of 7, mean against timeit best of 5), in load gating, and in core count. Five repositories drive their comparison from shell or Python scripts.

Scope (design section 3)

  • Create go-scikit/refbench with ghnew -public.
  • Case registry in Go: the measured function as a value, sizes, the input formula, the reference call, the parity class (exact / tolerance / none), and the expected allocations.
  • Python worker through uv run --no-project --python <pinned> --with-requirements refs.txt, with pinned versions and hashes. It reports the NumPy, SciPy and BLAS versions.
  • C worker as a subprocess speaking the same JSON-lines protocol, so the library has no cgo.
  • Runner: calibrated loops, the same statistic on both sides, interleaved and alternated rounds, and the load read before every round. Use go-fleettools/loadavg with wait or refuse at 8. Follow abbench's rule that a failed child stops the run.
  • Outputs: docs/perf-table.md (generated) and docs/perf.json (versioned schema). docs/perf.md stays hand-written.
  • CI mode: allocation counts equal to the JSON, the table regenerated from the JSON, and every public function registered or exempted with a reason.
  • refbench issues: open or update one issue per slower row and close it when a valid run is no longer slower. Read go-fleettools ghissue first.
  • Pin one NumPy version stack-wide. COVERAGE.md says 2.5.3; floats and stats measure 2.5.4.

Then W1: go-dsp/signal, go-simd/floats (bench/numpycmp retired), go-geometry/predicates (go-geometry/predicates#4).

🤖 Generated with Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions