Skip to content

Several references per case: Case.Refs, schema 2, rbw1 workers, Julia (step 1 of #15) - #16

Merged
tannevaled merged 12 commits into
mainfrom
feat/multiref
Nov 13, 2026
Merged

tannevaled merged 12 commits into
mainfrom
feat/multiref

Conversation

@tannevaled

@tannevaled tannevaled commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Refs #15

Step 1 of several references per case. The "Decisions" section of #15 is binding; docs/design/multiref.md is the full design, with an override note at its top where the decisions differ.

What this adds

  • Case.Refs []Ref (ID <kind>:<name>, license, Timer, Eval), Case.RefEval and Case.Accuracy. The primary reference stays Case.Ref.
  • Rounds rotate through every side (once per position per k rounds; two sides stay today's alternation); Rounds is rounded up to a multiple of k when extra references exist. A round above the gate is discarded for every side.
  • One verdict per reference, a headline against the fastest side (primary included), accuracy-aware labels (ahead, faster-less-accurate, behind, tradeoff). refbench issues drafts one issue per case when Go is slower than ANY side, even a less accurate one, with every reference, license, ratio and max ulp in the body.
  • Schema 2 when Refs is used; schema 1 stays byte for byte (golden files recorded from the code before this change: testdata/golden). Strict schema 2 reader, fuzzed, corpus committed. The table gains an "Against other references" section.
  • CI (rbtest) stays runtime-free; it additionally checks reference entries, licenses (no GPL outside kinds r, gsl) and the table.
  • rbw1 plain-text worker protocol (L, W, T, E, V): fixed function table, 4 KiB line bound (oversized line is refused, tested under a memory limit and from a child process under GOMEMLIMIT), paths inside the run directory, float64 inputs verified by SHA-256 (negative control: an altered input is refused naming both digests), CheckSingleThread against what the worker reports, strict reply parser with a fuzz target and corpus.
  • Calibrate discards one untimed first call (a JIT would otherwise make it choose iters = 1).
  • Julia worker (workers/julia/worker.jl, julialang.org binary through pkgx, pinned and checked against the reported version; Julia, LLVM and every Manifest package with its git-tree-sha1 come back under julia/... keys for ref_versions).
  • Compiled-worker skeleton (workers/compiled/rbw1.cpp, C++ harness with SHA-256, kernel table, C++ and Fortran examples), BuildWorker records source digests and flags; the harness is built and exercised end to end in the tests when a C++ compiler exists.
  • README updated.

Notes for review

  • Round is no longer comparable with == (it carries a map and a slice): two tests moved to reflect.DeepEqual. A consumer comparing rounds would break at compile time.
  • pkgx 2.11 rejects +julialang.org@=1.13.3 ("invalid semver"); @1.13.3 is used and the reported version is enforced.
  • Review fixes: one predicate decides schema 1 vs 2 (accuracy without references round-trips); a key whose last Load did not verify cannot be timed; rounds follow a Williams design (every ordered pair of sides equally often); labels are judged on MaxULPFin with special-value disagreements counted separately; the compiled harness reports where its thread count comes from (omp, proc, or assumed, which is not verified); worker.jl bounds each factor before multiplying and TestJuliaWorkerReal (REFBENCH_JULIA=1) runs it for real, since no CI lane has Julia.
  • Not enforced in step 1: the Several references per case: Julia, Boost, Fortran and R alongside SciPy, with accuracy per reference #15 decision that the first multi-reference run reproduces the fixtures' SciPy column bit for bit. FirstDiff exists but no gate calls it (the Python worker has no L yet); it is deferred to step 2 with a planted negative control, and the README says so.
  • Round 2: ahead is blocked when Go has Disagree > 0 or lower coverage than the reference (the 10%-NaN probe is a test); boundary tests for the request and 256 MiB bounds, the Load dimensions, WriteF64/ReadF64, the Eval reply counts and out-path, ns at 2^63, Budget at 0, version pairs at 256 and the printable-ASCII edges; Close reports the process's exit status.
  • mutsweep on rbw1.go, multiref.go, runner.go, workers.go and table.go (the three summary lines are copied from the runs):
    • mutsweep -mutate boundary: 29 mutations: 29 caught, 0 caught by PANIC, 0 not mutants, 0 survived, 0 hung
    • default mode: 95 mutations: 68 caught, 0 caught by PANIC, 27 not mutants, 0 survived, 0 hung
    • both run on the final head (cb70778). Every nil or index guard is now reached first by a_guards_test.go, which asserts the refusal and wraps the call so that a missing guard fails by assertion, not by a crash.
  • CI coverage gate is now covblocks (it names every uncovered block).
  • Not in this step: -perf.refs and case@refID in the deferred list, the Python L request, the R worker, refbench prepare, a shared home for the per-library kernels. Nothing is wired into go-numerics.
  • 100.0% coverage holds (including -race); gofmt, vet and GOARCH=386 vet are clean.

🤖 Generated with Claude Code

tannevaled and others added 12 commits November 13, 2026 14:59
Case.Refs []Ref (ID, license, Timer, Eval) with Case.Accuracy: rounds rotate
every side through every position, one verdict per reference, a headline
against the fastest side, accuracy-aware labels, one issue per case when any
reference is faster. A repository with no Refs keeps writing schema 1 byte for
byte (golden files recorded before the change). Calibrate discards one
untimed first call so that a JIT does not choose iters = 1. Adds the design as
docs/design/multiref.md.

Refs #15

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The rbw1 plain-text protocol (L, W, T, E, V), strict reply parser with a fuzz
target and committed corpus, 4 KiB line bound (refused, tested under a memory
limit), fixed function table, paths inside the run directory, float64 argument
files verified by SHA-256 before any timing, single-thread checked against what
the worker reports. A Julia worker (julialang.org binary through pkgx, pinned;
version and Manifest tree hashes recorded under julia/ keys) and a C++/Fortran
harness with examples; BuildWorker records source digests and flags.

Refs #15

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
README section on several references per case (rounds, verdicts, accuracy,
schema 2, rbw1, Julia, compiled workers) and what step 1 leaves out. pkgx 2.11
rejects the "@=" exact form: the worker is started with +julialang.org@<version>
and checked against what it reports.

Refs #15

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Windows' clock is too coarse for the spin-based verdict tests, and a CRLF
checkout changed the golden bytes. Verdict, label and issue tests now run on a
hand-built report; .gitattributes marks testdata/golden as -text.

Refs #15

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The compiled-worker test is skipped on Windows, which left the default
SingleThreadEnv branch of StartRBW1 uncovered there. Cover it with the fake
process; 100% coverage now holds with that test skipped.

Refs #15

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ed total

Refs #15

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
covblocks (now the CI gate) named rbtest.go:78-88 on every OS and, on Windows,
the rbw1 Load/Eval/ask refusals that the mixed-separator test paths never
reached.

Refs #15

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…axULPFin labels

- Schema: one predicate (Row.usesMultiRef) decides what Finish writes and what
  a schema 1 file may carry; a case with Accuracy and no Refs now round-trips.
- A key whose last Load did not verify cannot be warmed, timed or evaluated.
- Rounds follow a Williams design: every ordered pair of sides is adjacent
  equally often (a rotation kept first-order carryover).
- Labels are judged on MaxULPFin; special-value disagreements are counted
  separately (AccResult.Disagree).
- Compiled harness reports where its thread count comes from.
- worker.jl: bound each factor before the product; a kernel that raises on
  every point is refused; TestJuliaWorkerReal (REFBENCH_JULIA=1) runs it.
- README and design: the SciPy-column check is stated as deferred to step 2.
- Tests for the request and 256 MiB boundaries and the Eval out-path.

Refs #15

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- Label: a Go with Disagree > 0 or lower coverage than the reference is never
  "ahead" (faster-less-accurate); the probe is a test.
- Boundary tests: request line, 256 MiB, Load dimensions (no product, no
  division), WriteF64/ReadF64, Eval reply counts and out-path, ns at 2^63,
  Budget at 0, version pairs at 256, printable ASCII edges.
- Close reports the process's exit status; the fake worker's reply pipe is
  buffered so a client defect fails an assertion instead of hanging.
- mutsweep boundary: 29 mutations, 29 caught, 0 survived, 0 hung.

Refs #15

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…defect

TestRunVerdictAndInvalid asserted ratio > 0 and failed on Windows once (a
coarse clock, ratio 0). It checks the ratio structurally now.

Refs #15

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Refs #15

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit d61a56c into main Nov 13, 2026
5 checks passed
@tannevaled
tannevaled deleted the feat/multiref branch November 13, 2026 19:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant