Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# The golden files are compared byte for byte: no CRLF conversion on Windows.
testdata/golden/* -text
6 changes: 4 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,10 @@ jobs:
COVERPKG=$(go list ./... | paste -sd, -)
go test -race -coverpkg="$COVERPKG" -coverprofile=cover.out ./...
go tool cover -func=cover.out | tail -n 40
go tool cover -func=cover.out | tail -1 | grep -q '100.0%' || {
echo "::error::coverage below 100%"; exit 1; }
# go tool cover's total is rounded to one decimal (99.95 reads 100.0);
# covblocks lists every block the profile never entered (exit 1 if
# any, exit 2 on an empty or unreadable profile).
go run github.com/go-fleettools/fleettools/cmd/covblocks@v0.104.0 cover.out

# int is 32 bits here: v0.1.0 did not compile on 386 and arm.
arch-386:
Expand Down
147 changes: 147 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,144 @@ committed sources (`-O2 -ffp-contract=off`), so the library never needs cgo. A
repository that already has a cgo module may keep it and time it through the
same runner (go-geometry/predicates `bench/` does).

## Several references per case (schema 2)

SciPy is not the only bar: Julia, Boost.Math and the Fortran libraries are
references too. `Case.Ref` stays the primary reference (existing tables and
sentences stay true); `Case.Refs` adds more, timed in the same rounds
([design](docs/design/multiref.md); the maintainer decisions of
[#15](https://github.com/go-scikit/refbench/issues/15) override it):

```go
c.Refs = []refbench.Ref{
{ID: "julia:Bessels.besselk", License: "MIT", Timer: julia.Timer("kv", "bessels_kv"), Eval: julia.Evaler("kv", "bessels_kv", out, nil)},
{ID: "boost:cyl_bessel_k", License: "BSL-1.0", Timer: boost.Timer("kv", "kv")},
}
c.RefEval = scipyEval // the primary reference's values on the accuracy points
c.Accuracy = &refbench.Accuracy{Go: goEval, Score: ulpAgainstMpmath, Budget: 4}
```

- **Balanced rounds.** With *k* sides (Go, the primary, the extra ones) the
rounds follow a Williams design (a balanced Latin square, `k` orders for even
*k*, `2k` for odd): every side takes every position equally often **and every
ordered pair of sides is adjacent equally often**, so no side always follows
the same other one (a plain rotation balances positions but not that
carryover). `Rounds` is rounded up to a whole period. With two sides it is
exactly the alternation above. A round above the gate is discarded for all
sides.
- **One verdict per reference**, the rule above, plus a headline against the
**fastest side** (`best`, the primary included). **A gap against any faster
reference is a defect, even when that reference is less accurate**: `refbench
issues` drafts one issue per case when Go is slower than any side, titled after
the fastest of them, with every reference, its license, version, ratio and max
ulp in the body.
- **Accuracy in the same row.** `Accuracy.Score` is the repository's own scorer
(special: ulp against the mpmath `hi+lo` fixtures); Go, the primary
(`Case.RefEval`) and every `Ref.Eval` are scored on the same points, once,
ungated, before the rounds. `AccResult` carries `MaxULP` (+Inf, written
`"inf"`, when a finite value was expected and NaN came back), `MaxULPFin`,
`Disagree` (the number of such special-value disagreements) and `Coverage`, so
a reference that refuses half the domain does not look accurate. Labels:
`ahead`, `faster-less-accurate` (an accuracy defect), `behind`, `tradeoff`
(slower than a reference that is less accurate than Go's stated `Budget`: still
a perf defect). **The labels are judged on `MaxULPFin`**, the worst error over
the finite results: one special-value disagreement must not decide between
`behind` and `tradeoff`; it is reported (`MaxULP`, `Disagree`), not judged.
- **Not enforced yet: the SciPy column check.** Decision of #15: the first
multi-reference run must reproduce the fixtures' SciPy column bit for bit,
which proves the inputs reach the workers unaltered. Step 1 has the helper
(`FirstDiff`, tested on its own) but **no gate calls it**: the Python worker
has no `L` request yet, so nothing can run SciPy on the file the other workers
read. This is deferred to step 2 (Python `L`, then a gate with a planted
negative control). What step 1 does enforce is the SHA-256 of every argument
file, for the workers that have `L`.
- **Schema.** A repository with no `Refs` and no `Accuracy` keeps writing schema 1,
byte for byte (a golden test pins it). One with `Refs`, or with `Accuracy`
(`go_accuracy`, `primary_accuracy`, `go_budget` are schema 2 fields; one
predicate decides what is written and what a schema 1 file may carry), writes
**schema 2**, which an older
refbench refuses (`perf.json: schema 2, this build reads 1`): bump refbench in
the same PR. `Round` carries a map and a slice, so it is no longer comparable
with `==` (use `reflect.DeepEqual`). The table gains a second section,
"Against other references".
- **CI stays cheap.** `rbtest` starts no runtime: besides allocations, table and
coverage it checks that every `Ref.ID` (`<kind>:<name>`, unique) has an entry in
its row, that every reference has a license, that none is GPL unless its kind
is a black-box kind (`r`, `gsl`), and that the table regenerates from the JSON.
- **Calibrate** discards one untimed first call: a reference that compiles on its
first call (Julia) would otherwise last longer than `MinTime` at `iters = 1`
and settle on single-call samples.

### The rbw1 worker protocol

One request line, one reply line, plain ASCII, at most 4096 bytes (a longer line
is refused, never buffered), for Julia, R, C++ and Fortran (Python keeps JSON):

```
L <key> <path> <cols> <rows> load float64 little-endian column-major -> ok <sha256 of the bytes read>
W <key> <fn> one untimed call (JIT warm-up) -> ok
T <key> <fn> <iters> time iters calls inside the worker -> ns <float>
E <key> <fn> <outpath> evaluate once, write rows float64 -> ok <n_nonfinite> <n_raised>
V versions and observed threading -> v k=v k=v ...
```

`<fn>` is a name in a fixed table in the worker (`KERNELS` in Julia,
`rbw1_kernels` in the harness): nothing read from the pipe is evaluated. Paths
must lie in the run directory (`DataDir`: `os.UserCacheDir()/refbench/<repo>/<commit>`,
never `/tmp`), sizes are bounded (256 MiB). `WriteF64` writes an argument file and
returns its SHA-256, `RBW1.Load` compares it with the worker's and refuses a
mismatch naming both digests, and **a key whose last `Load` did not verify
(a mismatch, or a reload that failed after an earlier success) cannot be warmed,
timed or evaluated**. `RBW1.Timer` runs one
untimed `W` before the first timed call. `CheckSingleThread` refuses a worker
whose `V` reply does not show `threads=1` (and every `*_threads` at 1): the
environment variables are asked for, what the worker observed is checked.
Julia reports its real thread, GC-thread and BLAS-thread counts. The compiled
harness reports `threads_source`: `omp` (`omp_get_max_threads()`, when built with
OpenMP), `proc` (the process's thread count from `/proc/self/status`, Linux) or
`assumed` (the harness starts no thread; **a linked library that starts its own is
not detected there**: the check is unverified for compiled workers on those
hosts, and the record says so). A Julia kernel that raises on every point is a
broken kernel and is refused, not reported as an all-NaN column.
`ParseRBW1Reply` has a fuzz target and a committed corpus.

**Gap:** `worker.jl` needs Julia and runs in no CI lane (CI installs no runtime).
`REFBENCH_JULIA=1 go test -run TestJuliaWorkerReal .` runs it for real (pkgx,
pinned Julia, kernels that need no package) on a host that has them; it passed
on the reference host before this release.

### Julia references

`refbench.StartJulia(ctx, refbench.JuliaConfig{...})` runs the julialang.org
binary through **pkgx** (never brew), pinned (`JuliaConfig.Version`, refused
unless the worker reports exactly it):

```
pkgx +julialang.org@1.13.3 -- julia --startup-file=no --history-file=no -t 1 --project=<dir> -e <worker> <kernels.jl>
```

with `JULIA_NUM_THREADS=1`, `JULIA_NUM_GC_THREADS=1`, `JULIA_PKG_OFFLINE=true` and
`JULIA_DEPOT_PATH` set. The project (`Project.toml`, `Manifest.toml` with exact
versions and git-tree-sha1 content hashes) is committed in the consumer
repository; the depot is instantiated and precompiled once, outside any gated
run. `kernels.jl` defines `KERNELS = Dict("name" => scalar_function, ...)`; the
timed body is `out .= f.(args...)` in a function barrier on typed
`Vector{Float64}`s, with no `@fastmath`. The versions (Julia, LLVM, every
package of the Manifest with its tree hash) come back under `julia/...` keys for
`Options.RefVersions`, so `-perf.resume` refuses a different set.

### Compiled references (C, C++, Fortran, Boost.Math)

`workers/compiled/rbw1.cpp` (embedded as `CompiledHarnessSource`) is the harness
of a small stdin/stdout worker; a program adds a kernel table
(`rbw1_kernels`) and the kernels, in C++ or in Fortran (`bind(C)`); see
`workers/compiled/examples`. `BuildWorker(ctx, cc, out, refbench.ReleaseFlags,
sources)` compiles it (`-O2`, the compiler's own FP contraction, no
`-march=native`: the reference's normal release build; a different flag set is a
different `Ref.ID`) and returns the SHA-256 of every source and the flags for
`ref_versions`. R is not wired yet; when it comes it is `Rscript` only, GPL code is
never linked or read.

## CI mode: allocations, table, denominator

No Python, no timing, on every pull request. `rbtest` is the `go test` glue,
Expand Down Expand Up @@ -148,6 +286,8 @@ module) declares the parent of the running binary's module path, or the module
itself: a binary of one repository run from another's `bench/` directory refuses,
naming both module paths. `refbench.RunCI` is what the default mode calls:

0. with `Case.Refs`: every reference has an entry in its row, a license, and no
GPL outside a black-box kind (see above);
1. every case's allocations per call must equal `docs/perf.json`'s (an increase
is a regression; a decrease must be recorded);
2. `docs/perf-table.md` must be what `docs/perf.json` generates (no hand edits);
Expand All @@ -173,6 +313,13 @@ and the commit (`<case>.title.txt` and `<case>.md` with `--out`, for
`gh issue create --title ... --body-file`). A deferred report, or one with no
valid row, drafts nothing; a partial one drafts from its valid rows.

## Not yet (multi-reference, step 1)

- Restricting a run to some kinds (`-perf.refs`) and `case@refID` in the
deferred list; the Python `L` request (the NumPy `setup` stays).
- The R worker; a shared home for the per-library kernels (Boost, AMOS,
QUADPACK, MINPACK); `refbench prepare`.

## Not in v0.1.0

- Closing or commenting on an issue whose row is no longer slower.
Expand Down
164 changes: 164 additions & 0 deletions a_guards_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,164 @@
package refbench

import (
"context"
"os/exec"
"path/filepath"
"strings"
"testing"
)

// This file runs first (its name sorts before the others). Each test reaches a
// guard that protects a nil or an index, and asserts the REFUSAL. A mutant that
// removes the guard then fails an assertion here, instead of the suite going
// red only because the code after the missing guard crashes (a guard held by a
// crash is not held). Where the mutant necessarily crashes inside the call, the
// call is wrapped by mustNotPanic, which turns the crash into a failure that
// names the missing guard.

func mustNotPanic(t *testing.T, what string, f func()) {
t.Helper()
defer func() {
if r := recover(); r != nil {
t.Errorf("%s panicked: guard missing (%v)", what, r)
}
}()
f()
}

func TestGuardReplyShapes(t *testing.T) {
for _, in := range []string{"ns", "ns 1 2"} {
mustNotPanic(t, "ParseRBW1Reply("+in+")", func() {
if _, err := ParseRBW1Reply([]byte(in)); err == nil {
t.Errorf("%q accepted", in)
}
})
}
dir := t.TempDir()
path, rows, sha, err := WriteF64(dir, "x", []float64{1})
if err != nil {
t.Fatal(err)
}
reply := "ok"
w := fake(t, dir, []string{"f"}, func(string) string { return reply })
mustNotPanic(t, "Load with an 'ok' reply that has no digest", func() {
if err := w.Load("x", path, 1, rows, sha); err == nil {
t.Error("a load reply without a digest accepted")
}
})
w.rows["x"] = 1
mustNotPanic(t, "Eval with an 'ok' reply that has no counts", func() {
if _, _, err := w.Eval("x", "f", filepath.Join(dir, "o")); err == nil {
t.Error("an eval reply without counts accepted")
}
})
}

func TestGuardFiles(t *testing.T) {
dir := t.TempDir()
mustNotPanic(t, "WriteF64 with no column", func() {
if _, _, _, err := WriteF64(dir, "a"); err == nil {
t.Error("no column accepted")
}
})
mustNotPanic(t, "WriteF64 with an empty column", func() {
if _, _, _, err := WriteF64(dir, "a", []float64{}); err == nil {
t.Error("an empty column accepted")
}
})
mustNotPanic(t, "ReadF64 of a missing file", func() {
if _, err := ReadF64(filepath.Join(dir, "missing.f64"), 1); err == nil {
t.Error("a missing file read")
}
})
}

func TestGuardStarts(t *testing.T) {
// A process that CAN start: without the refusal of the bad configuration it
// would be started and the worker would be nil.
helperRBW1(t, "ok")
mustNotPanic(t, "StartRBW1 with a bad run directory", func() {
if _, err := StartRBW1(context.Background(), RBW1Config{Dir: "relative", Funcs: []string{"f"}}, nil, "x"); err == nil {
t.Error("a relative run directory accepted")
}
})
execCommand = func(ctx context.Context, _ string, _ ...string) *exec.Cmd {
return exec.CommandContext(ctx, "/nonexistent/refbench-guard")
}
mustNotPanic(t, "StartJulia when the worker cannot start", func() {
_, _, err := StartJulia(context.Background(), JuliaConfig{Pkgx: "pkgx", Version: "1.13.3", Project: "/p", Kernels: "/k.jl", Depot: "/d", Dir: t.TempDir(), Funcs: []string{"f"}})
if err == nil {
t.Error("a worker that cannot start accepted")
}
})
}

func TestGuardLabelsAndRefIDs(t *testing.T) {
acc := &AccResult{MaxULPFin: 1, Coverage: 1}
mustNotPanic(t, "Label without Go accuracy", func() {
if got := Label(Faster, nil, acc, 0); got != "" {
t.Errorf("label = %q", got)
}
})
mustNotPanic(t, "Label without reference accuracy", func() {
if got := Label(Faster, acc, nil, 0); got != "" {
t.Errorf("label = %q", got)
}
})
mustNotPanic(t, "Label without a verdict", func() {
if got := Label("", acc, acc, 0); got != "" {
t.Errorf("label = %q", got)
}
})
a := []RefRow{{ID: "a:x"}, {ID: "b:y"}}
mustNotPanic(t, "sameRefIDs of lists of different lengths", func() {
if sameRefIDs(a, a[:1]) || sameRefIDs(a[:1], a) {
t.Error("lists of different lengths are the same")
}
})
}

func TestGuardRunnerNils(t *testing.T) {
var log []string
c := multiCase(&log)
c.Ref = nil
mustNotPanic(t, "Run of a case without a primary timer", func() {
if _, err := Run(context.Background(), []Case{c}, Options{}); err == nil || !strings.Contains(err.Error(), "no reference timer") {
t.Errorf("err = %v", err)
}
})
c = multiCase(&log)
c.Accuracy, c.RefEval = nil, nil
row := c.row(0)
mustNotPanic(t, "scoreAccuracy without Accuracy", func() {
if err := scoreAccuracy(c, &row); err != nil || row.GoAccuracy != nil {
t.Errorf("%v %+v", err, row)
}
})
c = multiCase(&log)
c.RefEval, c.Refs[0].Eval = nil, nil
row = c.row(0)
mustNotPanic(t, "scoreAccuracy of sides without Eval", func() {
if err := scoreAccuracy(c, &row); err != nil || row.PrimaryAccuracy != nil || row.Refs[0].Accuracy != nil || row.GoAccuracy == nil {
t.Errorf("%v %+v", err, row)
}
})
}

func TestGuardNilCells(t *testing.T) {
mustNotPanic(t, "ulpCell(nil)", func() {
if ulpCell(nil) != "—" {
t.Error("ulpCell(nil)")
}
})
mustNotPanic(t, "covCell(nil)", func() {
if covCell(nil) != "—" {
t.Error("covCell(nil)")
}
})
mustNotPanic(t, "accCell(nil)", func() {
if accCell(nil) != "no accuracy" {
t.Error("accCell(nil)")
}
})
}
21 changes: 21 additions & 0 deletions case.go
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,16 @@ type Case struct {
// Ref times the reference body, inside the reference's process. Nil for a
// case only used by the allocation check.
Ref Timer
// Refs are additional references (Julia, Boost, Fortran, R, ...), timed in
// the same rounds as Go and Ref. May be empty: a case with no Refs writes
// the schema 1 files it always wrote.
Refs []Ref
// RefEval evaluates the primary reference on the accuracy points; nil gives
// it no accuracy column. Used only with Accuracy.
RefEval EvalFunc
// Accuracy, when set, scores Go, the primary reference and every Ref that
// has an Eval on the same points, once, before the timing rounds.
Accuracy *Accuracy
}

// Validate checks a registry: unique names, the required fields, a known parity.
Expand All @@ -66,6 +76,9 @@ func Validate(cases []Case) error {
case c.Parity != ParityExact && c.Parity != ParityTolerance && c.Parity != ParityNone:
return fmt.Errorf("case %q: unknown parity %q", c.Name, c.Parity)
}
if err := validateRefs(c); err != nil {
return err
}
seen[c.Name] = true
}
return nil
Expand All @@ -75,7 +88,15 @@ var errNoTime = errors.New("a calibration run took no measurable time")

// Calibrate returns the iteration count for which one run of t lasts at least
// minDur: it doubles from 1, then scales by the last measurement.
//
// One untimed call comes first and its result is discarded. A reference that
// compiles on its first call (Julia) would otherwise last longer than minDur at
// iters = 1, and Calibrate would settle on single-call samples. The call costs
// one iteration and is harmless for any other reference.
func Calibrate(t Timer, minDur time.Duration) (int, error) {
if _, err := t(1); err != nil {
return 0, err
}
// 1<<40 on 64-bit targets; half of MaxInt where int is 32 bits (a constant
// larger than int does not compile there).
const maxIters = min(1<<40, math.MaxInt>>1)
Expand Down
Loading
Loading