Repository navigation
harness: aqpbm-core harness module (engine traits, protocol, fit, records, conformance kit) - #256
Conversation
The engine traits, records and every module of the harness core, with todo!() bodies; ThreadCpuTime in aqpbm-core with its unit test. The implementation follows once the tests are approved.
82 tests from the approved S1 plan: machine key, protocol, fit with leave-one-setting-out, accuracy curves, data and data statistics, the fake engine and conformance kit, run directory, gate and sweep, and the dependency rule. 31 are moved or adapted from ASAPPlanner #1332. They compile against the todo!() skeleton and fail there, except the dependency rule, which already holds.
The engine traits, measurement protocol (idle check, pinning, warm-up, median of an odd run count), machine key, relative-weighted fit with leave-one-setting-out, accuracy curves, data chunks and statistics, run records and directory, gate and sweep arithmetic, the fake-clock engine and the conformance kit. aqpbm-core's ThreadCpuTime reads CLOCK_THREAD_CPUTIME_ID; aqpbm-datagen gains TimeSeriesCursor so a reader can own the generated series and draw samples lazily. All 82 tests pass except the two ignored timing tests (F7, P15).
- H1: a Processes engine is measured on its server pids. ProcessesCpuClock sums utime + stime under a ProcFs (thread CPU 0); run_arm and conformance::check take MeasurementClocks and build the engine's clock from cpu_accounting(). E10 now runs a fake server under a fake /proc. - M1: the calling-thread check repeats each step until 20 ms of CPU, then applies the 10% rule. - M2: answers are keyed by query id outside the labels; QUERY_LABEL deleted. - M3: measure_sizes runs on the protocol, takes a core and a LoadProbe, and returns a Result. - M4: one signed relative error per held-out point. - M5: samples share their series labels (Arc); run_arm documents the memory of its installs; consecutive jobs share one idle sample. - L1-L8 and nits: Option instead of NaN in records, NaN-cost gate failure and zero-duration refusal, private RunDir path and create_new result files, staleness message, FakeClock::ticks deleted, owning into_samples iterator, ConstantNotIdentified doc, relative units tolerance, renames.
a1c6fe8 to
e979cd3
Compare
Re-pinning the main thread, the process-CPU median, a zombie server pid, server CPU at ns resolution, a nonexistent core refused before the idle sample, and answers freed outside the timed window.
- Pinning reads the process's allowed cores once, before any pin, so the main thread can be re-pinned; a core outside the set is refused before the idle sample. - run_job takes the EngineClock and medians the CPU it measures the engine by (process CPU for Processes engines); MedianClock deleted; measure_sizes moves to protocol.rs and takes the EngineClock too. - Processes engines read their own cgroup v2 cpu.stat usage_usec, else their threads' schedstat (ns); a zombie pid is ProcessExited; the wall clock is read innermost around the call. - run_arm frees every run's answers after the job, outside the timed window. - NodeTerms and InstalledArm::node_terms, ProfilingMethod and MeasurementClocks::os deleted; ranking_agreement takes SelectedAndExactCosts rows; data_statistics groups by label Arc and keeps a running value range. - docs/aqpbm-harness.md, README link, into_samples in the datagen doc; test comments no longer carry PR numbers; generic machine-key example.
|
Is it possible / appropriate to put |
Yes, that's another way to place it. I can change. |
The aqpbm-harness crate becomes the harness module of aqpbm-core
(aqpbm_core::harness::{engine, protocol, machine, rundir, records, fit,
accuracy, data, data_stats, gate, sweep, fake, conformance, arm_run}).
Its tests move to aqpbm-core/tests/harness_*.rs, the dependency rule
now checks aqpbm-core with default features off, and the design doc
becomes docs/aqpbm-core-harness.md. aqpbm-core gains only thiserror and
serde's rc feature.
harness::accuracy becomes harness::error_quantiles, so it no longer reads like core's accuracy comparators; the crate doc points to the harness; the last aqpbm_harness temp-file prefix is renamed.
|
Done: the harness now lives in |
Summary
aqpbm-core,aqpbm_core::harness: an engine-neutral measurement harness, with its 82 tests. It holds the engine traits, the measurement protocol, the machine key, model fitting with leave-one-setting-out, accuracy curves, data statistics, run records, gate and sweep arithmetic, a fake-clock engine and the conformance kit.aqpbm-corealso gains a per-thread CPU clock, andaqpbm-datagengains an owning sample iterator so that a reader can own the generated series.cpu_accounting()names: the calling thread for an in-process engine, the server pids' own CPU for an external one (their own cgroup v2cpu.stat, else their threads'schedstat, at µs/ns resolution). Engines never report their own time.Moved into aqpbm-core per review
The harness was its own crate,
aqpbm-harness, up toa4daa63. Per review it now lives inaqpbm-coreas theharnessmodule (commitb8555e2, a pure move plus path fixes: no behavior change, no test assertion change).aqpbm-harnessaqpbm_core::harnessaqpbm-harness/src/<module>.rsaqpbm-core/src/harness/<module>.rs(lib.rs→harness/mod.rs)aqpbm-harness/tests/<name>.rsaqpbm-core/tests/harness_<name>.rs(accuracy.rs→harness_error_quantiles.rs)aqpbm_harness::accuracyaqpbm_core::harness::error_quantilesaqpbm-core/tests/dependencies.rs(cargo tree -p aqpbm-core --no-default-features)aqpbm-core/tests/dependencies.rs(cargo tree -p aqpbm-core --no-default-features)docs/aqpbm-core-harness.mddocs/aqpbm-core-harness.md, linked fromdocs/aqpbm-core.mdandREADME.mdCrate layout after the move:
default-features = falseaqpbm-datagenaqpbm-coremeasure(),GroundTruthcomparators, per-sketch JSONL records, clocks (metrics), and nowharnessaqpbm-datagen, anyhow, bitflags, chrono, rand, rand_xoshiro, serde (+rc), serde_json, thiserror, libc. No sketch library, polars, clap or planner crate.No concept is kept twice: the harness reads core's clocks (
metrics::ThreadCpuTime,metrics::CpuTimeSampler) instead of its own. Two names look alike but hold different things, so both stay:harness::recordsholds per-arm engine run records whilebenchmark_resultholds per-sketch JSONL records;harness::error_quantiles(namedaccuracybefore the move) turns per-trial errors into(δ, error)curves over merge counts, whileaccuracy::*comparators produce the per-trial errors againstGroundTruth.Review guide
main, rebased after datagen: one sample grid and one stream seed for every generator #255 was squash-merged; the branch no longer carries datagen: one sample grid and one stream seed for every generator #255's commits.d1ac008→8f22bc0: the crate skeleton andThreadCpuTime(P16);a537a11→5366103: the 82 tests, approved before the implementation;a1c6fe8→19cba8a: the implementation;e979cd3: the independent-review fixes (table below). Test changes in this commit are listed under "Test changes after approval".8d3b78d: failing tests for the final-review fixes (written against the old interface; 6 fail one979cd3, see Verification);a4daa63: the final-review fixes ("Final review fixes" below);b8555e2: the harness moves intoaqpbm-core("Moved into aqpbm-core per review" above);1785074: independent-review fixes of the move:harness::accuracy→harness::error_quantiles(so it is not read as core'saccuracycomparators), the crate doc points toharness, one leftover temp-file prefix renamed.protocol.rs(MeasurementClocks,ProcessesCpuClock,Protocol::run_job,measure_sizes),arm_run.rs(run_arm),fit.rs(fit_constants) andconformance.rs(check). The rest is small and pure.records.rs,gate.rs,sweep.rsand the owninginto_samplesiterator inaqpbm-datagen. The iterator body moved into a private cursor;samples()andinto_samples()both use it.Review fixes
Processesengines were timed on the harness processProcessesCpuClocksumsutime + stimeof the server pids under aProcFs(thread CPU 0).run_armandconformance::checktakeMeasurementClocksand build the engine's clock fromcpu_accounting()viaclock_for; the caller's clock is used only forCallingThread. A missing pid fails before any install; a pid that disappears mid-run fails the job (CpuClock::take_read_error)./procwhose stat file advances; samples are exactly 10,150 ns (evaluate) and 10,050 ns (ingest) while the harness clock stays 0. Fails on the old wiring (process CPU 0). New: conformance on the fake server, and a missing pid →EngineFailed { step: "cpu accounting" }.queryanswer label could overwrite a data labelAnswers(BTreeMap<query id, BTreeMap<SeriesLabels, f64>>), serialized as{query id: [[labels, value], ...]};QUERY_LABELdeleted.query="x"label keeps it.measure_sizesduplicated the protocol, panicked, did not composeProtocol::run_job_accounted: takes a core and aLoadProbe, the closure returnsResult<u64, EngineError>, returnsResult<Vec<FitPoint>, ProtocolError>. ACalibratableplugs in as `relative_errorsummed signed errorspoint_relative_errors: Option<Vec<f64>>, one signed error per held-out point.Sample.labels: Arc<SeriesLabels>, shared by every sample of a series (serderc).run_armdoc: (warm-up + N)× one arm's memory (6× by default), and steps run with cold caches./proc/statcounts 10 ms ticks, so 10 ticks resolve the 10% limit; at 50 ms one stray tick reads 20%.JobRecord.loadavg: [Option<f64>; 2](unreadable →None,unknowninstatus.log);LeaveOneSettingOut.rms_log_error: Option<f64>.None, JSON round-trips;ArmResultwith unreadable loadavg round-trips.GateFailure::EngineCostNotANumber;ArmRunError::NoDatafor a workload whose data last 0 ms.RunDir.pathprivate withpath(); result files openedcreate_new.FakeClock::ticksdeleted.CalibrationRunandCalibratedTargetkept: the design names them as the harness-ownedcalibrate-run.jsonrecord, written by S2'scalibrate.GeneratedTimeSeries::into_samples(self) -> OwnedTimeSeriesSamples; the cursor is private.into_samplesequalssamples()in plain and counter mode.ConstantNotIdentifieddocArmResult.engine_cost_per_second→measured_engine_cost_per_second;MeasuredUsage.memory_bytes→held_bytes;run_jobdocuments that the thread stays pinned.FakeEngine::installsalready documented ("Arms installed so far"); kept.Final review fixes
NodeTermshad no reader and overlapsCalibratable::charged_unitsNodeTermsandInstalledArm::node_termsdeleted (see "Moved to S2").pin_current_threadread the main thread's (already pinned) mask: the second core was refusedOnceLock) from the calling thread before the harness pins any thread.tests/pin_main_thread.rs(harness = false, runs on the main thread):run_jobon core a then core b. Failed one979cd3:core 1 is outside the allowed cores 0.run_jobalways took the median over thread CPU (wrong forProcesses)run_jobtakes&EngineClock; the median is overEngineClock::engine_cpu_ns(thread CPU forCallingThread, process CPU forProcesses).MedianClockdeleted; one public entry (run_next_jobis the crate-internal variant that reuses the previous after-sample).measure_sizestakes&EngineClocktoo.e979cd3(median 70, threadSome(0)).tests/arm_run_answer_drops.rs: a counting global allocator sees no answer freed between the wall-clock reads around a call. Failed one979cd3: 8 frees in the window./proc/<pid>/statticks); zombies counted as runningProcessesCpuClockreads the pids' own cgroup v2cpu.statusage_usecwhen all pids share one group that is not the harness's and not the root; otherwise the sum of/proc/<pid>/task/*/schedstat(ns). Each reading checks every pid: missing →MissingProcess, stateZ/X→ newProcessExited. The wall clock is read last before and first after the call.e979cd3: 0); cgroupusage_usec 2500→ 2,500,000 ns, and schedstat when the group is the harness's or split; zombie pid →ProcessExitedat build and on a later reading (failed one979cd3).run_jobchecks the core is in the allowed set before the idle sample.CoreNotAllowed, probe read 0 times (failed one979cd3: 1).run_armholds warm-up + N installs (6× memory)run_armdoc states the memory and why.data_statisticscloned labels per sample and kept every value (optional)Arc::as_ptrof the labels (one clone per seriesArc); values keep a running range and sign.MeasurementClocks::os()deleted;ProfilingMethodandNodeProfile.methoddeleted;RunDir::write_datakept with a new test.PredictedCost,CalibrationRun,CalibratedTargetkept (see "Kept although unread").a_data_record_is_filed_under_data_by_metric.measure_sizesinfit.rsprotocol.rs;fit.rsis pure.calibration_stalenessstays inrecords.rs(optional, declined: it compares the records'EngineDescriptions).ranking_agreementtook an unnamed 4-tupleSelectedAndExactCosts { predicted_selected, predicted_exact, measured_selected, measured_exact }.docs/aqpbm-core-harness.md(components, protocol, calibration, code interface) linked fromREADME.md;docs/aqpbm-datagen.mddescribesinto_samples(); "Moved from ASAPPlanner #1332" → "Moved from the ASAPPlanner test…" in the test comments; thenoise_bandTODO no longer names a stage;CostTarget.constantdoc names the engine's constant andcharged_units;machine_keydoc example is generic (no CPU vendor/model).Moved to S2
NodeTerms/InstalledArm::node_terms: per node, the units an engine's cost model charges per constant, for fitting constants from node profiles. Dropped from this PR because nothing reads it yet. S2 adds it back with its validation, and with a remainder fieldoperations_without_constant: f64(the work the cost model charges without a named constant, as in ASAPPlanner'sNodeCostTerms). It should be reconciled withCalibratable::charged_unitsthere.Kept although unread
PredictedCost.engine_cost_per_second(inArmResult.predicted): read by the S2 sweep, which ranks arms by predicted and measured cost (ranking_agreement).CalibrationRun,CalibratedTarget: thecalibrate-run.jsonrecord written by S2'scalibrate.run_arm,measure_sizes,conformance::check,RunDir::{create, write_run, write_data, write_arm, append_status},calibration_staleness,gate_verdict,regret,ranking_agreement,noise_band.Test changes after approval
Approved tests changed only where a review fix changes an interface they use:
tests/fit.rsF6fit_recovers_fake_clock_kernel, F7fit_recovers_spin_kernel(M3):measure_sizesgainscore(0) and a probe (an always-idleIdleProbein F6,OsLoadProbein F7), the kernel closure returnsOk(units), and the call gains.unwrap(). Expected points and tolerances unchanged. The probe andOkwere needed beyond.unwrap(): the review asked for a core, aLoadProbeand a closure that composes withprepare_cost_target.tests/fit.rsF8, F9, F10 and the outlier fold test (M4, L1):relative_error→point_relative_errors(one point per setting, so the helper asserts length 1 and takes it);rms_log_error→rms_log_error.unwrap(). Expected values unchanged.tests/fake.rsE7 (M2): counts answered series withanswers.iter().count()instead ofanswers.0.len(); E7–E9 build the run's clocks withMeasurementClocks::with_calling_thread(H1); thechunkhelper wraps labels with.into()(M5).tests/fake.rsE10 (H1): rewritten as described above; it now checks exact values instead of>= 10_000.tests/conformance.rs(H1):checktakesMeasurementClocks::with_calling_thread(&clock).tests/rundir.rs,tests/protocol.rs(L3, L1, nit):run.path→run.path();engine_cost_per_second→measured_engine_cost_per_second; thejobhelper wraps load averages inSome.tests/data.rs,tests/data_stats.rs(M5): sample helpers wrap labels with.into(); G1's hash iteratessample.labels.iter().Final review fixes (interface changes only; expected values unchanged):
tests/protocol.rspinning_runs_the_job_on_the_requested_core,idle_check_refuses_a_busy_core_and_never_runs_the_job,a_core_that_turns_busy_during_the_job_marks_it,warm_up_runs_are_discarded,an_even_or_zero_run_count_is_refusedandtests/fake.rsstep_costs_come_from_the_harness_clock(fix 3):run_job(.., &clock, ..)→run_job(.., &EngineClock::CallingThread(&clock), ..).tests/protocol.rsprocess_cpu_from_a_proc_pid_stat_line→process_state_from_a_proc_pid_stat_line(same line, now checks stateS);process_accounting_sums_every_pid_and_names_a_missing_onewrites threadschedstatfiles instead ofutime(still 3 s, still names pid 999) (fix 5).tests/fit.rsF6, F7 (fixes 3, 11):measure_sizesimported fromprotocol, clock wrapped inEngineClock::CallingThread.tests/fake.rsE9/E10,tests/conformance.rs(fix 5):proc_fs: server.proc_fs→process_cpu_files: server.cpu_files;ProcFs { root, ticks_per_second }→ProcessCpuFiles { proc_root, cgroup_root }; E10's doc saysschedstat; E9 no longer checksProfilingMethod(fix 10).tests/sweep.rsX5 (fix 12): rows areSelectedAndExactCosts.New tests (10 + 8): conformance
a_processes_engine_is_checked_on_its_server_pids; fakeconsecutive_jobs_share_one_idle_sample,a_series_label_named_query_is_kept,a_workload_without_data_is_refused; fitheld_out_errors_of_opposite_sign_do_not_cancel,no_scored_fold_has_no_rms_log_error; gatea_cost_that_is_not_a_number_fails_the_gate; rundira_result_file_is_never_overwritten,a_calibration_from_a_dirty_build_names_the_recorded_build; datageninto_samples_yields_the_samples_by_series_index.Behavior: before → after
Before, no shared harness existed. After, this is the whole flow, and each line is covered by a test:
Interfaces
All of
aqpbm_core::harnessis new. The engine traits and records:The protocol and one arm run:
Machine, fitting and accuracy:
Data, records, gate, sweep, fake and conformance:
Outside the new crate:
Design scope
aqpbm_core::harness(module)thiserrorand serde'srcfeature toaqpbm-core; with default features off, core has no sketch library, polars, clap or planner crate (Z1 checks this).ThreadCpuTime(aqpbm-core)CpuTimeSampler.OwnedTimeSeriesSamples(aqpbm-datagen)Answerskeyed by query id,format_seriesProcessCpuFiles,ProcessesCpuClockschedstat.ProtocolError::ProcessExitedEngineClock::engine_cpu_nsrun_job,run_arm,measure_sizes).SelectedAndExactCosts(sweep.rs)docs/aqpbm-core-harness.mddocs/aqpbm-core.md.tests/pin_main_thread.rs,tests/arm_run_answer_drops.rsMeasurementClocks,EngineClockCpuAccountingto the clock it is measured on, forrun_arm, conformance and S2's subcommands.FakeServerProcessespath end to end without a real server.OsLoadProbe/procform ofLoadProbethat every real run uses.workload_chunksConformanceFailure::EngineFailedAlgorithm changes
run_job): refuse an even or zero run count; refuse a core outside the allowed set (read once before any pin); check that the core is idle; pin; run the warm-up; timemeasured_runscalls on the given clock; check idle again, and mark the job if the core is now busy. The CPU clocks are read before the wall clock ahead of the call and after it behind the call. The median is the middle sample by thread CPU, or by process CPU for aProcessesengine (EngineClock::engine_cpu_ns). Withinrun_armandmeasure_sizes, a job's after-sample is the next job's before-sample.MeasurementClocks::clock_for(cpu_accounting())picks the caller's clock forCallingThread, and builds aProcessesCpuClockover the server pids forProcesses, reading every pid once so a missing or exited server fails before any install. The clock reads the pids' cgroup v2cpu.statusage_usecwhen every pid is in one group that is neither the root nor the harness's own; otherwise the sum of their threads'schedstat(a thread that exits during a step takes its CPU with it).run_arm: it installs warm-up + N arms up front. Each step (one refresh chunk, or one evaluation) is one protocol job, and its k-th call goes to install k, so every measured ingest is a first ingest. An evaluation runs after the chunk that ends at or after its time. Every call's answers are moved out of the timed call and freed after the job; answers and node profiles come from the last install.measured_engine_cost_per_secondcomes fromInstalledArm::priceover the summed medians per second of data. A workload whose data last 0 ms is refused.fit_constants: Householder QR on the design matrix, with each row weighted by 1/measured and each column normalized. A zero or collinear column isConstantNotIdentifiedand set to 0. A negative constant is clamped to 0 and the fit is rerun without it. The points are sorted first, so the result does not depend on their order (F13). A residual above 10% is flagged.leave_one_setting_out: one fold per setting, with a signed relative error per held-out point. A fold is unscored when the held-out setting needs a constant the fold could not identify. The RMS log error is taken over scored points only,Nonewhen there are none.measure_sizes: one protocol job per size; the point takes the units of the median run.error_quantile: rankn - floor(δ n), withδ nsnapped to an integer within 1e-9. The test case: 0.57 × 100 picks rank 43, whereceil((1 - 0.57) * 100)gives 44.CallingThreadengine, ingest (over successive chunks) and evaluate are each repeated until 20 ms of CPU (at most 100,000 calls), and fail if process CPU exceeds thread CPU by more than 10%;Processesengine's pids must be readable.scannode chargedns_per_callper evaluation, plus one node per query chargedns_per_seriesper answered series. With aFakeServerthe charges go to the server's stat file instead of the harness clock. Its window lookup is a binary search over time-ordered samples.Decisions and open questions
Decided (user):
ingestandevaluatereturn aStepReportwith no engine timing.Calibratable::charged_unitsis added.noise_bandcarries a TODO pointing at S2, and P10 checks only the value.Made here (flag if you disagree):
Core check, then idle check, then pinning. A nonexistent core is refused before it is sampled (final fix 6); a busy core is refused without touching affinity. P4 (core 3 busy) now needs core 3 to exist.
The allowed set is the process's affinity, read once (
OnceLock) from the calling thread before the harness pins any thread (final fix 2). A thread pinned earlier, the main thread included, does not narrow it; a latertasksetof the process is not seen.The machine key keeps the architecture's underscore (
x86_64), as M1's expected key requires. Everything else in the key is[a-z0-9-], andRunDir::createaccepts[a-z0-9_-]for the key.Answers are keyed by query id outside the labels (review M2), so two queries over one metric never collide and no label is reserved.
measure_sizesreturns aResult(review M3) and runs each size as one protocol job.The off-thread check measures at least 20 ms per step (review M1), so the plain 10% rule holds with
OsCpuClockin a process with other threads. The fake's fault (9 ms against 1 ms per call) is far outside it.installs: Cell<u32>is a public field ofFakeEngine. Tests build the fake with struct-update syntax, which needs every field public.The idle-sampling interval stays 100 ms; samples are shared between consecutive jobs instead (review M5). Rationale in the fixes table.
CalibrationRunandCalibratedTargetstay (review L5): the design names them as the harness-ownedcalibrate-run.jsonrecord; S2'scalibratewrites them.Cgroup only when it is the engine's own (final fix 5): one group for every pid, not the root, not the harness's. Other processes in that group (a container's helpers) are counted as the engine's.
Open: none.
Verification
On
b8555e2(full workspace, clean target; after1785074,cargo test -p aqpbm-core210 passed, 2 ignored, and workspace clippy and fmt pass again):cargo test --workspaceaqpbm-coreharness tests 96 passed, 2 ignored (F7, P15: timing), includingdependenciesandharness_pin_main_threadcargo clippy --workspace --all-targets -- -D warningscargo fmt --all -- --checkcargo tree -p aqpbm-core --no-default-features -e normalEarlier, on
a4daa63(before the move):cargo test -p aqpbm-harnessdependenciesandpin_main_threadcargo test -p aqpbm-core --lib metrics::timecargo test -p aqpbm-datagen -p aqpbm-usecasecargo fmt --all --checkcargo clippy -p aqpbm-harness -p aqpbm-core -p aqpbm-datagen --all-targets -- -D warnings8d3b78done979cd3core 1 is outside the allowed cores 0; median 70 with threadSome(0); zombie readOk; schedstat 0 instead of 5,000; nonexistent core sampled once; 8 answer frees inside timed callsThe 82 approved tests: 81 in
aqpbm-core/tests/harness_*.rsplus P16 inaqpbm-core. The review fixes add 9 harness tests and 1 datagen test (listed above).tests/machine.rstests/protocol.rs+aqpbm-coretime.rstests/fit.rstests/accuracy.rstests/data.rstests/data_stats.rstests/conformance.rstests/fake.rstests/rundir.rstests/gate.rs,tests/sweep.rstests/dependencies.rsThe 31 tests moved or adapted from ASAPPlanner #1332 (each test's doc comment names its source):
machine_key_from_cpu_cores_memory(machine_and_tables.rs)machine_key_from_cpu_cores_memorymachine_key_ignores_hostnamemachine_key_ignores_memory_sizemachine_key_without_edac_is_unknown_memoryAQPBM_MEMORY_TYPEmachine_key_memory_type_overridemachine_key_mixed_dimm_typesmachine_key_slug_charsetfit_recovers_exact_line(cost_fit.rs)fit_constants(["ns_per_unit", "ns_per_invocation"], …)fit_is_relative_weightedfit_five_sizes_tightensfit_negative_intercept_is_clampedNegativeConstantClamped { constant: "ns_per_invocation" }fit_flags_nonlinearfit_recovers_fake_clock_kernelmeasure_sizeson the harnessFakeClockfit_recovers_spin_kernelOsCpuClock; still ignored (timing)quantile_of_errors_nearest_rank(measured_accuracy.rs)Resultworst_over_merge_counts_is_recordedaccuracy_table_is_monotoneaccuracy_curve_is_monotone_and_the_worst_of_its_merge_countsdatagen_seeded_bit_identical(end_to_end.rs)a_data_record_regenerates_bit_identical_samplesdata/*.datagen.jsonfacts_reads_jsonl_samples(source_facts.rs)samples_jsonl_reads_and_names_the_bad_linedata::read_samples_jsonlfacts_series_countstatistics_series_countdata_statistics(samples, label_sets)facts_sample_interval_median_gapfacts_distinct_per_labelfacts_label_set_from_workload_byfacts_column_magnitudes_exclude_zerovalue_magnitudefacts_sign_suggestion_sign_bitfacts_counter_resetfacts_single_sample_seriesfacts_datagen_round_tripstatistics_of_generated_dataDataRecordfacts_seeded_bit_identical(end_to_end.rs)statistics_json_is_byte_identical_across_runsgate_rule_pure(gate.rs)gate_output_files(gate.rs)run_dir_is_machine_key_utc_subcommand_label