Repository navigation
Several references per case: Case.Refs, schema 2, rbw1 workers, Julia (step 1 of #15) - #16
Merged
Merged
Conversation
Case.Refs []Ref (ID, license, Timer, Eval) with Case.Accuracy: rounds rotate every side through every position, one verdict per reference, a headline against the fastest side, accuracy-aware labels, one issue per case when any reference is faster. A repository with no Refs keeps writing schema 1 byte for byte (golden files recorded before the change). Calibrate discards one untimed first call so that a JIT does not choose iters = 1. Adds the design as docs/design/multiref.md. Refs #15 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The rbw1 plain-text protocol (L, W, T, E, V), strict reply parser with a fuzz target and committed corpus, 4 KiB line bound (refused, tested under a memory limit), fixed function table, paths inside the run directory, float64 argument files verified by SHA-256 before any timing, single-thread checked against what the worker reports. A Julia worker (julialang.org binary through pkgx, pinned; version and Manifest tree hashes recorded under julia/ keys) and a C++/Fortran harness with examples; BuildWorker records source digests and flags. Refs #15 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
README section on several references per case (rounds, verdicts, accuracy, schema 2, rbw1, Julia, compiled workers) and what step 1 leaves out. pkgx 2.11 rejects the "@=" exact form: the worker is started with +julialang.org@<version> and checked against what it reports. Refs #15 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Windows' clock is too coarse for the spin-based verdict tests, and a CRLF checkout changed the golden bytes. Verdict, label and issue tests now run on a hand-built report; .gitattributes marks testdata/golden as -text. Refs #15 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The compiled-worker test is skipped on Windows, which left the default SingleThreadEnv branch of StartRBW1 uncovered there. Cover it with the fake process; 100% coverage now holds with that test skipped. Refs #15 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ed total Refs #15 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
covblocks (now the CI gate) named rbtest.go:78-88 on every OS and, on Windows, the rbw1 Load/Eval/ask refusals that the mixed-separator test paths never reached. Refs #15 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…axULPFin labels - Schema: one predicate (Row.usesMultiRef) decides what Finish writes and what a schema 1 file may carry; a case with Accuracy and no Refs now round-trips. - A key whose last Load did not verify cannot be warmed, timed or evaluated. - Rounds follow a Williams design: every ordered pair of sides is adjacent equally often (a rotation kept first-order carryover). - Labels are judged on MaxULPFin; special-value disagreements are counted separately (AccResult.Disagree). - Compiled harness reports where its thread count comes from. - worker.jl: bound each factor before the product; a kernel that raises on every point is refused; TestJuliaWorkerReal (REFBENCH_JULIA=1) runs it. - README and design: the SciPy-column check is stated as deferred to step 2. - Tests for the request and 256 MiB boundaries and the Eval out-path. Refs #15 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- Label: a Go with Disagree > 0 or lower coverage than the reference is never "ahead" (faster-less-accurate); the probe is a test. - Boundary tests: request line, 256 MiB, Load dimensions (no product, no division), WriteF64/ReadF64, Eval reply counts and out-path, ns at 2^63, Budget at 0, version pairs at 256, printable ASCII edges. - Close reports the process's exit status; the fake worker's reply pipe is buffered so a client defect fails an assertion instead of hanging. - mutsweep boundary: 29 mutations, 29 caught, 0 survived, 0 hung. Refs #15 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…defect TestRunVerdictAndInvalid asserted ratio > 0 and failed on Windows once (a coarse clock, ratio 0). It checks the ratio structurally now. Refs #15 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Refs #15 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #15
Step 1 of several references per case. The "Decisions" section of #15 is binding;
docs/design/multiref.mdis the full design, with an override note at its top where the decisions differ.What this adds
Case.Refs []Ref(ID<kind>:<name>, license, Timer, Eval),Case.RefEvalandCase.Accuracy. The primary reference staysCase.Ref.Roundsis rounded up to a multiple of k when extra references exist. A round above the gate is discarded for every side.ahead,faster-less-accurate,behind,tradeoff).refbench issuesdrafts one issue per case when Go is slower than ANY side, even a less accurate one, with every reference, license, ratio and max ulp in the body.Refsis used; schema 1 stays byte for byte (golden files recorded from the code before this change:testdata/golden). Strict schema 2 reader, fuzzed, corpus committed. The table gains an "Against other references" section.rbtest) stays runtime-free; it additionally checks reference entries, licenses (no GPL outside kindsr,gsl) and the table.L,W,T,E,V): fixed function table, 4 KiB line bound (oversized line is refused, tested under a memory limit and from a child process under GOMEMLIMIT), paths inside the run directory, float64 inputs verified by SHA-256 (negative control: an altered input is refused naming both digests),CheckSingleThreadagainst what the worker reports, strict reply parser with a fuzz target and corpus.Calibratediscards one untimed first call (a JIT would otherwise make it choose iters = 1).workers/julia/worker.jl, julialang.org binary through pkgx, pinned and checked against the reported version; Julia, LLVM and every Manifest package with its git-tree-sha1 come back underjulia/...keys forref_versions).workers/compiled/rbw1.cpp, C++ harness with SHA-256, kernel table, C++ and Fortran examples),BuildWorkerrecords source digests and flags; the harness is built and exercised end to end in the tests when a C++ compiler exists.Notes for review
Roundis no longer comparable with==(it carries a map and a slice): two tests moved toreflect.DeepEqual. A consumer comparing rounds would break at compile time.+julialang.org@=1.13.3("invalid semver");@1.13.3is used and the reported version is enforced.Loaddid not verify cannot be timed; rounds follow a Williams design (every ordered pair of sides equally often); labels are judged onMaxULPFinwith special-value disagreements counted separately; the compiled harness reports where its thread count comes from (omp,proc, orassumed, which is not verified);worker.jlbounds each factor before multiplying andTestJuliaWorkerReal(REFBENCH_JULIA=1) runs it for real, since no CI lane has Julia.FirstDiffexists but no gate calls it (the Python worker has noLyet); it is deferred to step 2 with a planted negative control, and the README says so.aheadis blocked when Go hasDisagree > 0or lower coverage than the reference (the 10%-NaN probe is a test); boundary tests for the request and 256 MiB bounds, theLoaddimensions,WriteF64/ReadF64, theEvalreply counts and out-path,nsat 2^63,Budgetat 0, version pairs at 256 and the printable-ASCII edges;Closereports the process's exit status.mutsweep -mutate boundary:29 mutations: 29 caught, 0 caught by PANIC, 0 not mutants, 0 survived, 0 hung95 mutations: 68 caught, 0 caught by PANIC, 27 not mutants, 0 survived, 0 hunga_guards_test.go, which asserts the refusal and wraps the call so that a missing guard fails by assertion, not by a crash.-perf.refsandcase@refIDin the deferred list, the PythonLrequest, the R worker,refbench prepare, a shared home for the per-library kernels. Nothing is wired into go-numerics.-race); gofmt, vet and GOARCH=386 vet are clean.🤖 Generated with Claude Code