Skip to content

harness: aqpbm-core harness module (engine traits, protocol, fit, records, conformance kit) - #256

Merged
GordonYuanyc merged 8 commits into
mainfrom
harness/core-tests
Nov 17, 2026
Merged

GordonYuanyc merged 8 commits into
mainfrom
harness/core-tests

Conversation

@zzylol

@zzylol zzylol commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Problem: cost calibration, decision sweeps, the showcase gate, node profiling and external-engine comparisons each run on their own local harness. Each has its own timing loop, its own machine handling and its own copy of the data formula, so their numbers cannot be compared. Example: one plan run twice as two arms differs by about 11%, and nothing records that band or calls such differences ties.
  • Change: a new module of aqpbm-core, aqpbm_core::harness: an engine-neutral measurement harness, with its 82 tests. It holds the engine traits, the measurement protocol, the machine key, model fitting with leave-one-setting-out, accuracy curves, data statistics, run records, gate and sweep arithmetic, a fake-clock engine and the conformance kit. aqpbm-core also gains a per-thread CPU clock, and aqpbm-datagen gains an owning sample iterator so that a reader can own the generated series.
  • Effect: any engine (the Planner's executor, ClickHouse, VictoriaMetrics, Prometheus or a fake) implements one trait. The harness then measures it under one protocol: an idle check, a pinned core, warm-up, and the median of an odd number of runs. The harness times every step itself, on the clock the engine's cpu_accounting() names: the calling thread for an in-process engine, the server pids' own CPU for an external one (their own cgroup v2 cpu.stat, else their threads' schedstat, at µs/ns resolution). Engines never report their own time.

Moved into aqpbm-core per review

The harness was its own crate, aqpbm-harness, up to a4daa63. Per review it now lives in aqpbm-core as the harness module (commit b8555e2, a pure move plus path fixes: no behavior change, no test assertion change).

Before After
crate aqpbm-harness module aqpbm_core::harness
aqpbm-harness/src/<module>.rs aqpbm-core/src/harness/<module>.rs (lib.rs → harness/mod.rs)
aqpbm-harness/tests/<name>.rs aqpbm-core/tests/harness_<name>.rs (accuracy.rs → harness_error_quantiles.rs)
aqpbm_harness::accuracy aqpbm_core::harness::error_quantiles
aqpbm-core/tests/dependencies.rs (cargo tree -p aqpbm-core --no-default-features) aqpbm-core/tests/dependencies.rs (cargo tree -p aqpbm-core --no-default-features)
docs/aqpbm-core-harness.md docs/aqpbm-core-harness.md, linked from docs/aqpbm-core.md and README.md

Crate layout after the move:

Crate Holds Normal dependencies with default-features = false
aqpbm-datagen the one sample generator serde, rand
aqpbm-core measure(), GroundTruth comparators, per-sketch JSONL records, clocks (metrics), and now harness aqpbm-datagen, anyhow, bitflags, chrono, rand, rand_xoshiro, serde (+rc), serde_json, thiserror, libc. No sketch library, polars, clap or planner crate.
 # Cargo.toml (workspace)
-    "aqpbm-harness",
-aqpbm-harness = { path = "aqpbm-harness" }
 # aqpbm-core/Cargo.toml
-serde.workspace = true
+serde = { workspace = true, features = ["rc"] }
+thiserror.workspace = true
+[[test]]
+name = "harness_pin_main_thread"
+harness = false
 // aqpbm-core/src/lib.rs
+pub mod harness;
 // every path
-aqpbm_harness::engine::EngineUnderTest
+aqpbm_core::harness::engine::EngineUnderTest

No concept is kept twice: the harness reads core's clocks (metrics::ThreadCpuTime, metrics::CpuTimeSampler) instead of its own. Two names look alike but hold different things, so both stay: harness::records holds per-arm engine run records while benchmark_result holds per-sketch JSONL records; harness::error_quantiles (named accuracy before the move) turns per-trial errors into (δ, error) curves over merge counts, while accuracy::* comparators produce the per-trial errors against GroundTruth.

Review guide

  • Base: main, rebased after datagen: one sample grid and one stream seed for every generator #255 was squash-merged; the branch no longer carries datagen: one sample grid and one stream seed for every generator #255's commits.
  • Commits:
    1. d1ac008 → 8f22bc0: the crate skeleton and ThreadCpuTime (P16);
    2. a537a11 → 5366103: the 82 tests, approved before the implementation;
    3. a1c6fe8 → 19cba8a: the implementation;
    4. e979cd3: the independent-review fixes (table below). Test changes in this commit are listed under "Test changes after approval".
    5. 8d3b78d: failing tests for the final-review fixes (written against the old interface; 6 fail on e979cd3, see Verification);
    6. a4daa63: the final-review fixes ("Final review fixes" below);
    7. b8555e2: the harness moves into aqpbm-core ("Moved into aqpbm-core per review" above);
    8. 1785074: independent-review fixes of the move: harness::accuracy → harness::error_quantiles (so it is not read as core's accuracy comparators), the crate doc points to harness, one leftover temp-file prefix renamed.
  • Read first: protocol.rs (MeasurementClocks, ProcessesCpuClock, Protocol::run_job, measure_sizes), arm_run.rs (run_arm), fit.rs (fit_constants) and conformance.rs (check). The rest is small and pure.
  • Mechanical: records.rs, gate.rs, sweep.rs and the owning into_samples iterator in aqpbm-datagen. The iterator body moved into a private cursor; samples() and into_samples() both use it.

Review fixes

Finding Fix Test
H1 Processes engines were timed on the harness process ProcessesCpuClock sums utime + stime of the server pids under a ProcFs (thread CPU 0). run_arm and conformance::check take MeasurementClocks and build the engine's clock from cpu_accounting() via clock_for; the caller's clock is used only for CallingThread. A missing pid fails before any install; a pid that disappears mid-run fails the job (CpuClock::take_read_error). E10 extended: a fake server (pid 4242) under a fake /proc whose stat file advances; samples are exactly 10,150 ns (evaluate) and 10,050 ns (ingest) while the harness clock stays 0. Fails on the old wiring (process CPU 0). New: conformance on the fake server, and a missing pid → EngineFailed { step: "cpu accounting" }.
M1 off-thread margin of 10% + 1 ms flaky under real clocks Each step of the calling-thread check repeats (ingest over successive chunks of a long probe, evaluate at the last end) until 20 ms of thread or process CPU, at most 100,000 calls; then the 10% rule, with no absolute margin. The failure reports CPU per call. E5 unchanged (9.0 ms vs 1.0 ms per call).
M2 the query answer label could overwrite a data label Answers(BTreeMap<query id, BTreeMap<SeriesLabels, f64>>), serialized as {query id: [[labels, value], ...]}; QUERY_LABEL deleted. New: a series with its own query="x" label keeps it.
M3 measure_sizes duplicated the protocol, panicked, did not compose Built on Protocol::run_job_accounted: takes a core and a LoadProbe, the closure returns Result<u64, EngineError>, returns Result<Vec<FitPoint>, ProtocolError>. A Calibratable plugs in as ` size
M4 relative_error summed signed errors point_relative_errors: Option<Vec<f64>>, one signed error per held-out point. New: +10% and −10% in one setting are both reported.
M5 labels cloned per sample Sample.labels: Arc<SeriesLabels>, shared by every sample of a series (serde rc). G1–G5, D1–D10
M5 warm-up + N installs run_arm doc: (warm-up + N)× one arm's memory (6× by default), and steps run with cold caches. —
M5 200 ms idle sampling per job Consecutive jobs share one sample: a job's after-sample is the next job's before-sample (nothing but the harness runs on the core between them). One 100 ms interval per job instead of two. 100 ms is kept: /proc/stat counts 10 ms ticks, so 10 ticks resolve the 10% limit; at 50 ms one stray tick reads 20%. New: 12 jobs sample the core 13 times.
L1 NaN in JSON JobRecord.loadavg: [Option<f64>; 2] (unreadable → None, unknown in status.log); LeaveOneSettingOut.rms_log_error: Option<f64>. New: no scored fold → None, JSON round-trips; ArmResult with unreadable loadavg round-trips.
L2 NaN/infinite cost passed the gate; zero duration GateFailure::EngineCostNotANumber; ArmRunError::NoData for a workload whose data last 0 ms. New: both.
L3 run-directory protection bypassable RunDir.path private with path(); result files opened create_new. New: a second write of one arm fails, the first file is kept.
L4 staleness message blamed the current engine Names which side is dirty: recorded, current, or both. New: recorded-only dirty.
L5 dead code FakeClock::ticks deleted. CalibrationRun and CalibratedTarget kept: the design names them as the harness-owned calibrate-run.json record, written by S2's calibrate. —
L6 cursor accepted any series GeneratedTimeSeries::into_samples(self) -> OwnedTimeSeriesSamples; the cursor is private. New datagen unit test: into_samples equals samples() in plain and counter mode.
L7 ConstantNotIdentified doc Now names all three causes: no non-zero term, collinear column, fewer points than constants. —
L8 exact float compare of units Relative tolerance 1e-9. —
nit ArmResult.engine_cost_per_second → measured_engine_cost_per_second; MeasuredUsage.memory_bytes → held_bytes; run_job documents that the thread stays pinned. FakeEngine::installs already documented ("Arms installed so far"); kept. —

Final review fixes

# Finding Status Fix Test
1 NodeTerms had no reader and overlaps Calibratable::charged_units done NodeTerms and InstalledArm::node_terms deleted (see "Moved to S2"). —
2 pin_current_thread read the main thread's (already pinned) mask: the second core was refused done The allowed cores are read once (OnceLock) from the calling thread before the harness pins any thread. New tests/pin_main_thread.rs (harness = false, runs on the main thread): run_job on core a then core b. Failed on e979cd3: core 1 is outside the allowed cores 0.
3 Public run_job always took the median over thread CPU (wrong for Processes) done run_job takes &EngineClock; the median is over EngineClock::engine_cpu_ns (thread CPU for CallingThread, process CPU for Processes). MedianClock deleted; one public entry (run_next_job is the crate-internal variant that reuses the previous after-sample). measure_sizes takes &EngineClock too. New: process costs 90, 30, 70, 50, 20 → median 50, no thread CPU. Failed on e979cd3 (median 70, thread Some(0)).
4 Answers of all but the last run were freed inside the timed call done Every call's answers move into a vector reserved before the job; all but the last are freed after the job. New tests/arm_run_answer_drops.rs: a counting global allocator sees no answer freed between the wall-clock reads around a call. Failed on e979cd3: 8 frees in the window.
5 Process CPU at 10 ms resolution (/proc/<pid>/stat ticks); zombies counted as running done ProcessesCpuClock reads the pids' own cgroup v2 cpu.stat usage_usec when all pids share one group that is not the harness's and not the root; otherwise the sum of /proc/<pid>/task/*/schedstat (ns). Each reading checks every pid: missing → MissingProcess, state Z/X → new ProcessExited. The wall clock is read last before and first after the call. New: schedstat sum 2,000 + 3,000 = 5,000 ns (failed on e979cd3: 0); cgroup usage_usec 2500 → 2,500,000 ns, and schedstat when the group is the harness's or split; zombie pid → ProcessExited at build and on a later reading (failed on e979cd3).
6 A nonexistent core was sampled by the idle check first done run_job checks the core is in the allowed set before the idle sample. New: core 100000 → CoreNotAllowed, probe read 0 times (failed on e979cd3: 1).
7 run_arm holds warm-up + N installs (6× memory) done (documented); sequential installs declined Sequential installs would make one protocol job span whole runs instead of one step, so the idle check and median would no longer be per step. The run_arm doc states the memory and why. —
8 100 ms idle sample per job (optional) declined Sampling every k jobs would leave k − 1 jobs without a before-sample; jobs already share one sample between neighbours. —
9 data_statistics cloned labels per sample and kept every value (optional) done Series are found by Arc::as_ptr of the labels (one clone per series Arc); values keep a running range and sign. D1–D10 unchanged.
10 Dead code done MeasurementClocks::os() deleted; ProfilingMethod and NodeProfile.method deleted; RunDir::write_data kept with a new test. PredictedCost, CalibrationRun, CalibratedTarget kept (see "Kept although unread"). New: a_data_record_is_filed_under_data_by_metric.
11 measure_sizes in fit.rs done Moved to protocol.rs; fit.rs is pure. calibration_staleness stays in records.rs (optional, declined: it compares the records' EngineDescriptions). F6/F7 import path only.
12 ranking_agreement took an unnamed 4-tuple done SelectedAndExactCosts { predicted_selected, predicted_exact, measured_selected, measured_exact }. X5 builds rows with it.
13 Docs done New docs/aqpbm-core-harness.md (components, protocol, calibration, code interface) linked from README.md; docs/aqpbm-datagen.md describes into_samples(); "Moved from ASAPPlanner #1332" → "Moved from the ASAPPlanner test …" in the test comments; the noise_band TODO no longer names a stage; CostTarget.constant doc names the engine's constant and charged_units; machine_key doc example is generic (no CPU vendor/model). —

Moved to S2

  • NodeTerms / InstalledArm::node_terms: per node, the units an engine's cost model charges per constant, for fitting constants from node profiles. Dropped from this PR because nothing reads it yet. S2 adds it back with its validation, and with a remainder field operations_without_constant: f64 (the work the cost model charges without a named constant, as in ASAPPlanner's NodeCostTerms). It should be reconciled with Calibratable::charged_units there.

Kept although unread

  • PredictedCost.engine_cost_per_second (in ArmResult.predicted): read by the S2 sweep, which ranks arms by predicted and measured cost (ranking_agreement).
  • CalibrationRun, CalibratedTarget: the calibrate-run.json record written by S2's calibrate.
  • Entry points S2 calls: run_arm, measure_sizes, conformance::check, RunDir::{create, write_run, write_data, write_arm, append_status}, calibration_staleness, gate_verdict, regret, ranking_agreement, noise_band.

Test changes after approval

Approved tests changed only where a review fix changes an interface they use:

  • tests/fit.rs F6 fit_recovers_fake_clock_kernel, F7 fit_recovers_spin_kernel (M3): measure_sizes gains core (0) and a probe (an always-idle IdleProbe in F6, OsLoadProbe in F7), the kernel closure returns Ok(units), and the call gains .unwrap(). Expected points and tolerances unchanged. The probe and Ok were needed beyond .unwrap(): the review asked for a core, a LoadProbe and a closure that composes with prepare_cost_target.
  • tests/fit.rs F8, F9, F10 and the outlier fold test (M4, L1): relative_error → point_relative_errors (one point per setting, so the helper asserts length 1 and takes it); rms_log_error → rms_log_error.unwrap(). Expected values unchanged.
  • tests/fake.rs E7 (M2): counts answered series with answers.iter().count() instead of answers.0.len(); E7–E9 build the run's clocks with MeasurementClocks::with_calling_thread (H1); the chunk helper wraps labels with .into() (M5).
  • tests/fake.rs E10 (H1): rewritten as described above; it now checks exact values instead of >= 10_000.
  • tests/conformance.rs (H1): check takes MeasurementClocks::with_calling_thread(&clock).
  • tests/rundir.rs, tests/protocol.rs (L3, L1, nit): run.path → run.path(); engine_cost_per_second → measured_engine_cost_per_second; the job helper wraps load averages in Some.
  • tests/data.rs, tests/data_stats.rs (M5): sample helpers wrap labels with .into(); G1's hash iterates sample.labels.iter().

Final review fixes (interface changes only; expected values unchanged):

  • tests/protocol.rs pinning_runs_the_job_on_the_requested_core, idle_check_refuses_a_busy_core_and_never_runs_the_job, a_core_that_turns_busy_during_the_job_marks_it, warm_up_runs_are_discarded, an_even_or_zero_run_count_is_refused and tests/fake.rs step_costs_come_from_the_harness_clock (fix 3): run_job(.., &clock, ..) → run_job(.., &EngineClock::CallingThread(&clock), ..).
  • tests/protocol.rs process_cpu_from_a_proc_pid_stat_line → process_state_from_a_proc_pid_stat_line (same line, now checks state S); process_accounting_sums_every_pid_and_names_a_missing_one writes thread schedstat files instead of utime (still 3 s, still names pid 999) (fix 5).
  • tests/fit.rs F6, F7 (fixes 3, 11): measure_sizes imported from protocol, clock wrapped in EngineClock::CallingThread.
  • tests/fake.rs E9/E10, tests/conformance.rs (fix 5): proc_fs: server.proc_fs → process_cpu_files: server.cpu_files; ProcFs { root, ticks_per_second } → ProcessCpuFiles { proc_root, cgroup_root }; E10's doc says schedstat; E9 no longer checks ProfilingMethod (fix 10).
  • tests/sweep.rs X5 (fix 12): rows are SelectedAndExactCosts.
  • 31 test doc comments: PR number dropped (fix 13).

New tests (10 + 8): conformance a_processes_engine_is_checked_on_its_server_pids; fake consecutive_jobs_share_one_idle_sample, a_series_label_named_query_is_kept, a_workload_without_data_is_refused; fit held_out_errors_of_opposite_sign_do_not_cancel, no_scored_fold_has_no_rms_log_error; gate a_cost_that_is_not_a_number_fails_the_gate; rundir a_result_file_is_never_overwritten, a_calibration_from_a_dirty_build_names_the_recorded_build; datagen into_samples_yields_the_samples_by_series_index.

Behavior: before → after

Before, no shared harness existed. After, this is the whole flow, and each line is covered by a test:

// The fake engine: 50 ns per ingested sample, 150 ns per answered series, 10 µs per call, on a fake clock.
let engine = FakeEngine::default();
// Probe workloads: `probe` sampled every 10 s from 0; workload 0 is sum_over_time(probe[10s]),
// workload 1 is count_over_time(probe[10s]); data seed 7; each with PromQL and SQL texts.
let workload = &conformance::probe_workloads()[0];
// Every adapter runs this kit: deterministic answers, left-open windows, calibration units,
// work on the calling thread. Every failure is listed, in check order.
// The clocks: the caller's for a CallingThread engine, /proc/<pid>/stat for a Processes engine.
let clocks = MeasurementClocks::with_calling_thread(&engine.clock);
conformance::check(&mut engine.clone(), &clocks)?;                         // E1: Ok, 4 checks passed
// One arm under the protocol (warm-up 1, 5 runs): 6 installs, each step one job, the k-th call on install k.
let result = arm_run::run_arm(&engine, workload, &arm, &schedule, &Protocol::default(), 0, &mut probe, &clocks)?;
// The harness clock measured it: ingest of 1,000 samples = 60,000 ns, evaluate of 4 series = 10,600 ns (E7).
// The same seed gives a byte-identical ArmResult JSON; seed 8 changes the answers, not the costs (E8).
let run_dir = RunDir::create(root, &machine.machine_key(), utc, "sweep", "sketch-benefit")?;
// -> <root>/runs/x86_64-intel-xeon-e5-2683-v3-56c-registered-ddr4/20261116T201500Z-sweep-sketch-benefit/ (R1)
run_dir.write_arm(&result)?;                                               // results/fake/<workload>__exact.json (R5)

Interfaces

All of aqpbm_core::harness is new. The engine traits and records:

+// aqpbm-core/src/harness/engine.rs
+pub trait EngineUnderTest {
+    fn description(&self) -> EngineDescription;
+    fn cpu_accounting(&self) -> CpuAccounting;
+    fn install(&self, workload: &WorkloadSpec, arm: &ArmSpec) -> Result<Box<dyn InstalledArm>, EngineError>;
+    fn calibration(&mut self) -> Option<&mut dyn Calibratable> { None }
+}
+pub struct EngineDescription { pub id: String, pub version: String, pub version_dirty: bool, pub build_profile: String }
+pub enum CpuAccounting { CallingThread, Processes(Vec<u32>) }
+pub struct ArmSpec { pub name: String, pub raw_data_retained: bool, pub engine_options: serde_json::Value }
+pub struct WorkloadSpec { pub name: String, pub queries: Vec<WorkloadQuery>, pub data: Vec<DataRecord> }
+pub struct WorkloadQuery { pub id: String, pub texts: Vec<QueryText>, pub window_ms: u64, pub evaluation_interval_ms: u64 }
+pub struct QueryText { pub language: QueryLanguage, pub text: String }
+pub enum QueryLanguage { PromQl, Sql }
+pub trait InstalledArm {
+    fn plan_summary(&self) -> PlanSummary;
+    fn predicted(&self) -> Option<PredictedCost> { None }
+    fn ingest(&mut self, chunk: &SampleChunk) -> Result<StepReport, EngineError>;
+    fn evaluate(&mut self, at_ms: i64) -> Result<(Answers, StepReport), EngineError>;
+    fn footprint(&self) -> Result<Footprint, EngineError>;
+    fn node_profiles(&mut self) -> Option<Vec<NodeProfile>> { None }
+    fn price(&self, usage: &MeasuredUsage) -> Option<f64> { None }
+}
+/// Counters an engine keeps itself; never its own time.
+pub struct StepReport { pub scanned_bytes: Option<u64>, pub engine_reported_cpu_ns: Option<u64> }
+/// What the harness measured for one step.
+pub struct StepCost { pub thread_cpu_ns: Option<u64>, pub process_cpu_ns: u64, pub wall_ns: u64, pub report: StepReport }
+/// Per query id, one value per output series; serialized as {query id: [[labels, value], ...]}.
+pub struct Answers(pub BTreeMap<String, BTreeMap<SeriesLabels, f64>>);
+impl Answers {
+    pub fn iter(&self) -> impl Iterator<Item = (&str, &SeriesLabels, f64)>;
+    pub fn get(&self, query: &str, series: &SeriesLabels) -> Option<f64>;
+}
+/// `probe{instance="instance_0"}`.
+pub fn format_series(labels: &SeriesLabels) -> String;
+pub struct PlanSummary { pub text: String, pub node_kinds: Vec<String>, pub stored_outputs: Vec<String> }
+pub struct PredictedCost { pub engine_cost_per_second: f64 }
+pub struct Footprint { pub view_bytes: u64, pub raw_retained_bytes: u64, pub peak_bytes: Option<u64> }
+/// held_bytes = Footprint::view_bytes + Footprint::raw_retained_bytes.
+pub struct MeasuredUsage { pub ingest_cpu_ns_per_second: f64, pub query_cpu_ns_per_second: f64, pub held_bytes: u64 }
+pub struct NodeProfile { pub node: String, pub cpu_ns_per_batch: Vec<u64> }
+pub trait Calibratable {
+    fn cost_targets(&self) -> Vec<CostTarget>;
+    fn prepare_cost_target(&mut self, target: &CostTarget, size: u64)
+        -> Result<Box<dyn FnMut() -> Result<u64, EngineError>>, EngineError>;
+    /// The units the engine's own cost model charges `target.constant` at `size`.
+    fn charged_units(&self, target: &CostTarget, size: u64) -> f64;
+}
+pub struct CostTarget { pub key: serde_json::Value, pub constant: String, pub unit: String, pub sizes: Vec<u64> }
+pub struct EngineError(pub String);

The protocol and one arm run:

+// aqpbm-core/src/harness/protocol.rs
+pub trait CpuClock {
+    fn thread_cpu_ns(&self) -> u64; fn process_cpu_ns(&self) -> u64; fn wall_ns(&self) -> u64;
+    /// The first failed reading since the last call (a server that exited); checked after every measured run.
+    fn take_read_error(&self) -> Option<ProtocolError> { None }
+}
+/// ThreadCpuTime for the thread, CpuTimeSampler (getrusage) for the process, Instant for wall time.
+pub struct OsCpuClock;
+/// The /proc root and the cgroup v2 mount; `os()` is /proc and /sys/fs/cgroup.
+pub struct ProcessCpuFiles { pub proc_root: PathBuf, pub cgroup_root: PathBuf }
+impl ProcessCpuFiles { pub fn os() -> Self; }
+/// Process CPU = the pids' own cgroup cpu.stat usage_usec, else their threads' schedstat (ns);
+/// thread CPU = 0; wall = Instant. A missing pid or a zombie fails the reading.
+pub struct ProcessesCpuClock { /* private */ }
+impl ProcessesCpuClock { pub fn new(pids: Vec<u32>, files: ProcessCpuFiles) -> Result<Self, ProtocolError>; }
+/// The caller's clock for CallingThread engines; the files the harness reads for Processes engines.
+pub struct MeasurementClocks<'a> { pub calling_thread: &'a dyn CpuClock, pub process_cpu_files: ProcessCpuFiles }
+impl<'a> MeasurementClocks<'a> {
+    pub fn with_calling_thread(calling_thread: &'a dyn CpuClock) -> Self;
+    pub fn clock_for(&self, accounting: &CpuAccounting) -> Result<EngineClock<'a>, ProtocolError>;
+}
+pub enum EngineClock<'a> { CallingThread(&'a dyn CpuClock), Processes(ProcessesCpuClock) }
+impl EngineClock<'_> {
+    pub fn as_clock(&self) -> &dyn CpuClock;
+    /// Thread CPU for CallingThread, process CPU for Processes: what medians are taken over.
+    pub fn engine_cpu_ns(&self, cost: &StepCost) -> u64;
+}
+/// Refuses a core outside the allowed set read once before any pin.
+pub fn pin_current_thread(core: usize) -> Result<(), ProtocolError>;
+pub fn core_busy_fraction(core: usize, before: &str, after: &str) -> Result<f64, ProtocolError>;
+pub fn process_state_from_stat(stat: &str) -> Result<char, ProtocolError>;
+pub fn processes_cpu_ns(proc_root: &Path, pids: &[u32]) -> Result<u64, ProtocolError>;
+pub trait LoadProbe { fn core_busy_fraction(&mut self, core: usize) -> Result<f64, ProtocolError>; fn loadavg(&mut self) -> f64; }
+/// Two /proc/stat reads `sample_interval` apart (default 100 ms), and /proc/loadavg.
+pub struct OsLoadProbe { pub sample_interval: Duration }
+pub struct Protocol { pub warmup_runs: u32, pub measured_runs: u32, pub max_core_busy_fraction: f64 }
+impl Protocol {
+    /// Core check, idle check, pin, warm up, run `measured_runs` times, idle check again;
+    /// the median is over `clock.engine_cpu_ns`; the thread stays pinned.
+    pub fn run_job(&self, core: usize, probe: &mut dyn LoadProbe, clock: &EngineClock,
+        job: &mut dyn FnMut() -> Result<StepReport, EngineError>) -> Result<JobRecord, ProtocolError>;
+    pub fn check_run_count(&self) -> Result<(), ProtocolError>;
+}
+pub struct JobRecord { pub core: usize, pub busy_before: f64, pub busy_after: f64, pub loadavg: [Option<f64>; 2],
+    pub marks: Vec<JobMark>, pub samples: Vec<StepCost>, pub median: StepCost }
+pub enum JobMark { BusyAfter { busy_fraction: f64 } }
+pub fn median(values: &[f64]) -> Result<f64, ProtocolError>;
+pub fn noise_band(arm_a: &[f64], arm_b: &[f64]) -> Result<f64, ProtocolError>;
+pub enum Comparison { Cheaper, Tie, Costlier }
+pub fn compare(candidate: f64, reference: f64, band: f64) -> Comparison;
+pub fn measure_sizes(protocol: &Protocol, core: usize, probe: &mut dyn LoadProbe, clock: &EngineClock,
+    sizes: &[u64], run: &mut dyn FnMut(u64) -> Result<u64, EngineError>) -> Result<Vec<FitPoint>, ProtocolError>;
+pub enum ProtocolError { CoreNotAllowed { .. }, CoreBusy { .. }, IdleUnmeasurable { .. }, RunCountNotOdd(u32),
+    NonPositiveMedian { .. }, Parse { .. }, MissingProcess { .. }, ProcessExited { pid: u32, state: char },
+    Pin { .. }, Job(EngineError) }
+
+// aqpbm-core/src/harness/arm_run.rs
+pub struct ArmSchedule { pub refresh_ms: u64, pub evaluate_at_ms: Vec<i64> }
+pub fn run_arm(engine: &dyn EngineUnderTest, workload: &WorkloadSpec, arm: &ArmSpec, schedule: &ArmSchedule,
+    protocol: &Protocol, core: usize, probe: &mut dyn LoadProbe, clocks: &MeasurementClocks) -> Result<ArmResult, ArmRunError>;
+pub enum ArmRunError { Engine(..), Protocol(..), Data(..), NoData { workload: String } }

Machine, fitting and accuracy:

+// aqpbm-core/src/harness/machine.rs
+pub const UNKNOWN_MEMORY_TYPE: &str = "unknown-memory";
+pub const MEMORY_TYPE_ENV: &str = "AQPBM_MEMORY_TYPE";
+pub struct MachineDescription { pub cpu_architecture: String, pub cpu_model: String, pub logical_cores: u32,
+    pub memory_type: String, pub memory_kib: u64, pub os: String }
+impl MachineDescription {
+    pub fn from_proc(cpu_architecture: &str, cpuinfo: &str, meminfo: &str, dimm_memory_types: &[&str],
+        memory_type_override: Option<&str>, os: &str) -> Result<Self, MachineError>;
+    pub fn detect(memory_type_override: Option<&str>) -> Result<Self, MachineError>;
+    /// `x86_64-intel-xeon-e5-2683-v3-56c-registered-ddr4`; never the hostname or memory size.
+    pub fn machine_key(&self) -> String;
+    pub fn warning(&self) -> Option<String>;
+}
+
+// aqpbm-core/src/harness/fit.rs
+pub struct FitPoint { pub setting: String, pub terms: Vec<f64>, pub measured: f64 }
+pub struct ConstantsFit { pub constants: Vec<(String, f64)>, pub flags: Vec<FitFlag> }
+pub enum FitFlag { NegativeConstantClamped { .. }, ResidualAboveTolerance { .. }, ConstantNotIdentified { .. } }
+pub const RESIDUAL_TOLERANCE: f64 = 0.10;
+pub fn fit_constants(names: &[&str], points: &[FitPoint]) -> Result<ConstantsFit, FitError>;
+/// One signed (predicted - measured) / measured per held-out point.
+pub struct HeldOutSetting { pub setting: String, pub fit: ConstantsFit, pub point_relative_errors: Option<Vec<f64>> }
+pub struct LeaveOneSettingOut { pub held_out: Vec<HeldOutSetting>, pub rms_log_error: Option<f64>, pub settings_scored: usize }
+pub fn leave_one_setting_out(names: &[&str], points: &[FitPoint]) -> Result<LeaveOneSettingOut, FitError>;
+
+// aqpbm-core/src/harness/error_quantiles.rs
+pub fn error_quantile(errors: &[f64], failure_probability: f64) -> Result<f64, AccuracyError>;
+pub struct MergeCountErrors { pub key: serde_json::Value, pub merge_count: u32, pub trials: u32, pub error_at: Vec<(f64, f64)> }
+pub struct AccuracyCurve { pub key: serde_json::Value, pub trials: u32, pub error_at: Vec<(f64, f64)> }
+pub fn worst_over_merge_counts(results: &[MergeCountErrors]) -> Vec<AccuracyCurve>;

Data, records, gate, sweep, fake and conformance:

+// aqpbm-core/src/harness/data.rs
+pub type SeriesLabels = BTreeMap<String, String>;
+/// `labels` is shared by every sample of one series.
+pub struct Sample { pub labels: Arc<SeriesLabels>, pub ts_ms: i64, pub value: f64 }
+pub fn read_samples_jsonl(text: &str) -> Result<Vec<Sample>, SampleReadError>;
+pub struct DataRecord { pub metric: String, pub description: TimeSeriesDescription }
+pub struct SampleChunk { pub end_ms: i64, pub samples: Vec<Sample> }
+pub fn refresh_chunks(record: &DataRecord, refresh_ms: u64)
+    -> Result<impl Iterator<Item = Result<SampleChunk, DataError>>, DataError>;
+/// The refresh chunks of several metrics merged by end.
+pub fn workload_chunks(records: &[DataRecord], refresh_ms: u64)
+    -> Result<impl Iterator<Item = Result<SampleChunk, DataError>>, DataError>;
+
+// aqpbm-core/src/harness/data_stats.rs
+pub struct DataStatistics { pub series_count: u64, pub sample_interval_ms: Option<u64>,
+    pub distinct_values_per_label: BTreeMap<String, u64>, pub distinct_values_per_label_set: Vec<(Vec<String>, u64)>,
+    pub value_magnitude: Option<ObservedMagnitude>, pub value_sign: ObservedSign }
+pub fn data_statistics(samples: &[Sample], label_sets: &[Vec<String>]) -> BTreeMap<String, DataStatistics>;
+
+// aqpbm-core/src/harness/records.rs, rundir.rs
+pub struct RunRecord { pub args: Vec<String>, pub harness_version: String, pub machine: MachineDescription,
+    pub engines: Vec<EngineDescription>, pub protocol: Protocol }
+pub struct ArmResult { pub engine: String, pub workload: String, pub arm: String, pub plan: PlanSummary,
+    pub predicted: Option<PredictedCost>, pub ingest: Vec<JobRecord>, pub evaluate: Vec<JobRecord>,
+    pub answers: Vec<(i64, Answers)>, pub footprint: Footprint, pub measured_engine_cost_per_second: Option<f64>,
+    pub node_profiles: Option<Vec<NodeProfile>> }
+pub struct CalibrationRun { pub engine: EngineDescription, pub machine: MachineDescription,
+    pub protocol: Protocol, pub targets: Vec<CalibratedTarget> }
+pub fn calibration_staleness(recorded: &EngineDescription, current: &EngineDescription) -> Option<String>;
+pub struct RunDir { /* private path */ }
+impl RunDir {
+    pub fn path(&self) -> &Path;
+    pub fn create(root: &Path, machine_key: &str, utc: DateTime<Utc>, subcommand: &str, label: &str) -> Result<Self, RunDirError>;
+    pub fn write_run(&self, run: &RunRecord) -> Result<(), RunDirError>;
+    pub fn write_data(&self, data: &DataRecord) -> Result<PathBuf, RunDirError>;
+    /// Every result file is created new; an existing one is refused.
+    pub fn write_arm(&self, result: &ArmResult) -> Result<PathBuf, RunDirError>;
+    pub fn append_status(&self, job: &str, record: &JobRecord) -> Result<(), RunDirError>;
+}
+
+// aqpbm-core/src/harness/gate.rs, sweep.rs
+pub const GATE_MAX_COST_RATIO: f64 = 1.05;
+pub struct GateRow { pub workload: String, pub setting: String,
+    pub exact_engine_cost_per_second: Option<f64>, pub selected_engine_cost_per_second: Result<Option<f64>, String>,
+    pub exact_query_cpu_ns: Option<u64>, pub selected_query_cpu_ns: Option<u64>,
+    pub measured_error: f64, pub target_error: f64 }
+impl GateRow { pub fn raw_cpu_ratio(&self) -> Option<f64>; }
+pub enum GateFailure { CostAboveExact { .. }, ErrorAboveTarget { .. }, ExactArmMissing { .. }, SelectedArmFailed { .. },
+    EngineCostMissing { .. }, ErrorNotANumber { .. }, EngineCostNotANumber { .. }, MissingRow { .. }, ExtraRow { .. } }
+pub fn gate_verdict(expected: &[(String, String)], rows: &[GateRow], cost_gate: bool) -> GateVerdict;
+pub fn regret(arms: &[(String, f64)], selected: &str) -> Result<f64, SweepError>;
+pub struct SelectedAndExactCosts { pub predicted_selected: f64, pub predicted_exact: f64,
+    pub measured_selected: f64, pub measured_exact: f64 }
+pub fn ranking_agreement(rows: &[SelectedAndExactCosts], band: f64) -> f64;
+
+// aqpbm-core/src/harness/fake.rs, conformance.rs
+pub struct FakeClock(pub Rc<Cell<FakeTicks>>);
+pub struct FakeEngine { pub clock: FakeClock, pub ns_per_sample: u64, pub ns_per_series: u64, pub ns_per_call: u64,
+    pub engine_reported_cpu_ns: Option<u64>, pub cpu_accounting: CpuAccounting, pub faults: Vec<FakeFault>,
+    pub installs: Cell<u32>, pub server: Option<FakeServer> }
+/// A fake external server: its work is written to its thread's schedstat under a fake /proc, not charged to the harness clock.
+pub struct FakeServer { pub pid: u32, pub cpu_files: ProcessCpuFiles, /* private CPU */ }
+impl FakeServer { pub fn start(root: &Path, pid: u32) -> io::Result<Self>; pub fn cpu_ns(&self) -> u64; }
+pub enum FakeFault { NondeterministicAnswers, CountsWindowStart, CalibrationUnitsDouble, WorksOnAnotherThread }
+pub enum ConformanceCheck { DeterministicAnswers, LeftOpenWindow, CalibrationUnits, WorkOnCallingThread }
+pub enum ConformanceFailure { NondeterministicAnswers { .. }, WindowNotLeftOpen { .. }, UnitsMismatch { .. },
+    EngineFailed { check, step, message }, WorkOffCallingThread { .. } }
+pub fn check(engine: &mut dyn EngineUnderTest, clocks: &MeasurementClocks) -> Result<ConformanceReport, Vec<ConformanceFailure>>;
+pub fn probe_workloads() -> Vec<WorkloadSpec>;

Outside the new crate:

 // aqpbm-core/src/metrics/time.rs
+/// CPU time of the calling thread, from CLOCK_THREAD_CPUTIME_ID; 0 off unix.
+pub struct ThreadCpuTime;
+impl ThreadCpuTime { pub fn now_ns() -> u64; }

 // aqpbm-datagen/src/timeseries.rs
 impl GeneratedTimeSeries {
     pub fn samples(&self) -> TimeSeriesSamples<'_>;
+    /// The samples of `samples()` from an iterator that owns the series, as (series index, timestamp ms, value).
+    pub fn into_samples(self) -> OwnedTimeSeriesSamples;
 }
+pub struct OwnedTimeSeriesSamples { /* private */ }
+impl Iterator for OwnedTimeSeriesSamples { type Item = Result<(usize, i64, f64), DataGenError>; }

Design scope

New type or file Scope Why it is general
aqpbm_core::harness (module) measurement harness Any engine is measured through one trait. It adds only thiserror and serde's rc feature to aqpbm-core; with default features off, core has no sketch library, polars, clap or planner crate (Z1 checks this).
ThreadCpuTime (aqpbm-core) measurement primitive A per-thread CPU clock, for any in-process benchmark. It sits next to the process-wide CpuTimeSampler.
OwnedTimeSeriesSamples (aqpbm-datagen) data generation Lets any reader own the generated series and draw samples lazily, here the refresh-chunk iterator. The old iterator body moved into a private cursor that both iterators use; no code is duplicated, and a cursor can no longer be read against the wrong series.
Answers keyed by query id, format_series engine records Every multi-query workload needs answer keys that do not collide, without reserving a label name; every failure message prints a series.
ProcessCpuFiles, ProcessesCpuClock protocol Any external server measured by its own CPU: its cgroup v2 group, else its threads' schedstat.
ProtocolError::ProcessExited protocol Any server that exits (a zombie) during a run fails the job by name.
EngineClock::engine_cpu_ns protocol The one rule for which CPU an engine is measured by, used by every median (run_job, run_arm, measure_sizes).
SelectedAndExactCosts (sweep.rs) sweep One setting's predicted and measured costs of the selected and the exact arm; named fields instead of a 4-tuple.
docs/aqpbm-core-harness.md docs The harness module's design doc, linked from docs/aqpbm-core.md.
tests/pin_main_thread.rs, tests/arm_run_answer_drops.rs tests Own binaries: one must run on the main thread, the other installs a global allocator.
MeasurementClocks, EngineClock protocol The one place that maps an engine's CpuAccounting to the clock it is measured on, for run_arm, conformance and S2's subcommands.
FakeServer fake engine Exercises the Processes path end to end without a real server.
OsLoadProbe protocol The /proc form of LoadProbe that every real run uses.
workload_chunks data A workload with several metrics ingests them on one refresh grid.
ConformanceFailure::EngineFailed conformance An adapter that errors on a probe step fails by name instead of panicking.

Algorithm changes

  • Protocol (run_job): refuse an even or zero run count; refuse a core outside the allowed set (read once before any pin); check that the core is idle; pin; run the warm-up; time measured_runs calls on the given clock; check idle again, and mark the job if the core is now busy. The CPU clocks are read before the wall clock ahead of the call and after it behind the call. The median is the middle sample by thread CPU, or by process CPU for a Processes engine (EngineClock::engine_cpu_ns). Within run_arm and measure_sizes, a job's after-sample is the next job's before-sample.
  • Clocks: MeasurementClocks::clock_for(cpu_accounting()) picks the caller's clock for CallingThread, and builds a ProcessesCpuClock over the server pids for Processes, reading every pid once so a missing or exited server fails before any install. The clock reads the pids' cgroup v2 cpu.stat usage_usec when every pid is in one group that is neither the root nor the harness's own; otherwise the sum of their threads' schedstat (a thread that exits during a step takes its CPU with it).
  • run_arm: it installs warm-up + N arms up front. Each step (one refresh chunk, or one evaluation) is one protocol job, and its k-th call goes to install k, so every measured ingest is a first ingest. An evaluation runs after the chunk that ends at or after its time. Every call's answers are moved out of the timed call and freed after the job; answers and node profiles come from the last install. measured_engine_cost_per_second comes from InstalledArm::price over the summed medians per second of data. A workload whose data last 0 ms is refused.
  • fit_constants: Householder QR on the design matrix, with each row weighted by 1/measured and each column normalized. A zero or collinear column is ConstantNotIdentified and set to 0. A negative constant is clamped to 0 and the fit is rerun without it. The points are sorted first, so the result does not depend on their order (F13). A residual above 10% is flagged.
  • leave_one_setting_out: one fold per setting, with a signed relative error per held-out point. A fold is unscored when the held-out setting needs a constant the fold could not identify. The RMS log error is taken over scored points only, None when there are none.
  • measure_sizes: one protocol job per size; the point takes the units of the median run.
  • error_quantile: rank n - floor(δ n), with δ n snapped to an integer within 1e-9. The test case: 0.57 × 100 picks rank 43, where ceil((1 - 0.57) * 100) gives 44.
  • Conformance:
    • each probe runs on two fresh installs, and the first differing answer per workload is reported;
    • a count at t = 20,000 ms over samples at 10,000 and 20,000 ms must be 1;
    • each calibration closure is called once per target size, and its units must match the charged units within 1e-9 relative;
    • for a CallingThread engine, ingest (over successive chunks) and evaluate are each repeated until 20 ms of CPU (at most 100,000 calls), and fail if process CPU exceeds thread CPU by more than 10%;
    • a Processes engine's pids must be readable.
  • Fake engine: one shared scan node charged ns_per_call per evaluation, plus one node per query charged ns_per_series per answered series. With a FakeServer the charges go to the server's stat file instead of the harness clock. Its window lookup is a binary search over time-ordered samples.

Decisions and open questions

Decided (user):

  • The engine traits live here.
  • ingest and evaluate return a StepReport with no engine timing.
  • Calibratable::charged_units is added.
  • The run count must be odd (default 5).
  • An existing run directory is refused (R3).
  • The noise-band field goes into the protocol record in S2, not here. noise_band carries a TODO pointing at S2, and P10 checks only the value.

Made here (flag if you disagree):

  1. Core check, then idle check, then pinning. A nonexistent core is refused before it is sampled (final fix 6); a busy core is refused without touching affinity. P4 (core 3 busy) now needs core 3 to exist.

  2. The allowed set is the process's affinity, read once (OnceLock) from the calling thread before the harness pins any thread (final fix 2). A thread pinned earlier, the main thread included, does not narrow it; a later taskset of the process is not seen.

  3. The machine key keeps the architecture's underscore (x86_64), as M1's expected key requires. Everything else in the key is [a-z0-9-], and RunDir::create accepts [a-z0-9_-] for the key.

  4. Answers are keyed by query id outside the labels (review M2), so two queries over one metric never collide and no label is reserved.

  5. measure_sizes returns a Result (review M3) and runs each size as one protocol job.

  6. The off-thread check measures at least 20 ms per step (review M1), so the plain 10% rule holds with OsCpuClock in a process with other threads. The fake's fault (9 ms against 1 ms per call) is far outside it.

  7. installs: Cell<u32> is a public field of FakeEngine. Tests build the fake with struct-update syntax, which needs every field public.

  8. The idle-sampling interval stays 100 ms; samples are shared between consecutive jobs instead (review M5). Rationale in the fixes table.

  9. CalibrationRun and CalibratedTarget stay (review L5): the design names them as the harness-owned calibrate-run.json record; S2's calibrate writes them.

  10. Cgroup only when it is the engine's own (final fix 5): one group for every pid, not the root, not the harness's. Other processes in that group (a container's helpers) are counted as the engine's.

Open: none.

Verification

On b8555e2 (full workspace, clean target; after 1785074, cargo test -p aqpbm-core 210 passed, 2 ignored, and workspace clippy and fmt pass again):

Command Result
cargo test --workspace all pass; aqpbm-core harness tests 96 passed, 2 ignored (F7, P15: timing), including dependencies and harness_pin_main_thread
cargo clippy --workspace --all-targets -- -D warnings pass
cargo fmt --all -- --check pass
cargo tree -p aqpbm-core --no-default-features -e normal no polars, sketch library, clap, sketch-bench or planner crate

Earlier, on a4daa63 (before the move):

Command Result
cargo test -p aqpbm-harness 96 passed, 0 failed, 2 ignored (F7, P15: timing, not run), including Z1 dependencies and pin_main_thread
cargo test -p aqpbm-core --lib metrics::time 3 passed, including P16
cargo test -p aqpbm-datagen -p aqpbm-usecase datagen 76 passed + 1 doctest; usecase 7 passed
cargo fmt --all --check pass
cargo clippy -p aqpbm-harness -p aqpbm-core -p aqpbm-datagen --all-targets -- -D warnings pass
The 6 tests of 8d3b78d on e979cd3 all fail: re-pin core 1 is outside the allowed cores 0; median 70 with thread Some(0); zombie read Ok; schedstat 0 instead of 5,000; nonexistent core sampled once; 8 answer frees inside timed calls

The 82 approved tests: 81 in aqpbm-core/tests/harness_*.rs plus P16 in aqpbm-core. The review fixes add 9 harness tests and 1 datagen test (listed above).

File Tests Ids
tests/machine.rs 9 M1–M9
tests/protocol.rs + aqpbm-core time.rs 15 + 1 P1–P15, P16
tests/fit.rs 13 F1–F13
tests/accuracy.rs 6 A1–A6
tests/data.rs 5 G1–G5
tests/data_stats.rs 10 D1–D10
tests/conformance.rs 6 E1–E6
tests/fake.rs 4 E7–E10
tests/rundir.rs 7 R1–R7
tests/gate.rs, tests/sweep.rs 3 + 2 X1–X3, X4–X5
tests/dependencies.rs 1 Z1

The 31 tests moved or adapted from ASAPPlanner #1332 (each test's doc comment names its source):

ASAPPlanner test (file) Here Adaptation
machine_key_from_cpu_cores_memory (machine_and_tables.rs) M1 machine_key_from_cpu_cores_memory path only
machine_key_ignores_hostname M2 path only
machine_key_ignores_memory_size M3 path only
machine_key_without_edac_is_unknown_memory M4 warning names AQPBM_MEMORY_TYPE
machine_key_memory_type_override M5 path only
machine_key_mixed_dimm_types M6 path only
machine_key_slug_charset M7 path only
fit_recovers_exact_line (cost_fit.rs) F1 fit_constants(["ns_per_unit", "ns_per_invocation"], …)
fit_is_relative_weighted F2 as F1
fit_five_sizes_tightens F3 as F1
fit_negative_intercept_is_clamped F4 NegativeConstantClamped { constant: "ns_per_invocation" }
fit_flags_nonlinear F5 as F1
fit_recovers_fake_clock_kernel F6 measure_sizes on the harness FakeClock
fit_recovers_spin_kernel F7 OsCpuClock; still ignored (timing)
quantile_of_errors_nearest_rank (measured_accuracy.rs) A1 returns Result
worst_over_merge_counts_is_recorded A3 opaque JSON key
accuracy_table_is_monotone A5 accuracy_curve_is_monotone_and_the_worst_of_its_merge_counts seeded synthetic errors
datagen_seeded_bit_identical (end_to_end.rs) G1 a_data_record_regenerates_bit_identical_samples through data/*.datagen.json
facts_reads_jsonl_samples (source_facts.rs) G5 samples_jsonl_reads_and_names_the_bad_line data::read_samples_jsonl
facts_series_count D1 statistics_series_count data_statistics(samples, label_sets)
facts_sample_interval_median_gap D2 same
facts_distinct_per_label D3 same
facts_label_set_from_workload_by D4 label sets passed in
facts_column_magnitudes_exclude_zero D5 value_magnitude
facts_sign_suggestion_sign_bit D6 observed sign only
facts_counter_reset D7 same
facts_single_sample_series D8 same
facts_datagen_round_trip D9 statistics_of_generated_data through DataRecord
facts_seeded_bit_identical (end_to_end.rs) D10 statistics_json_is_byte_identical_across_runs statistics only
gate_rule_pure (gate.rs) X1 engine cost decides; raw CPU reported
gate_output_files (gate.rs) R1 run_dir_is_machine_key_utc_subcommand_label new run directory layout

@zzylol zzylol added the enhancement New feature or request label Nov 16, 2026
@zzylol zzylol changed the title harness: aqpbm-harness core skeleton and its failing tests harness: aqpbm-harness core (engine traits, protocol, fit, records, conformance kit) Nov 16, 2026
@zzylol
zzylol marked this pull request as ready for review November 16, 2026 22:21
@zzylol
zzylol changed the base branch from datagen/one-sample-grid to main November 16, 2026 23:03
The engine traits, records and every module of the harness core, with
todo!() bodies; ThreadCpuTime in aqpbm-core with its unit test. The
implementation follows once the tests are approved.
82 tests from the approved S1 plan: machine key, protocol, fit with
leave-one-setting-out, accuracy curves, data and data statistics, the
fake engine and conformance kit, run directory, gate and sweep, and the
dependency rule. 31 are moved or adapted from ASAPPlanner #1332. They
compile against the todo!() skeleton and fail there, except the
dependency rule, which already holds.
The engine traits, measurement protocol (idle check, pinning, warm-up,
median of an odd run count), machine key, relative-weighted fit with
leave-one-setting-out, accuracy curves, data chunks and statistics, run
records and directory, gate and sweep arithmetic, the fake-clock engine
and the conformance kit. aqpbm-core's ThreadCpuTime reads
CLOCK_THREAD_CPUTIME_ID; aqpbm-datagen gains TimeSeriesCursor so a reader
can own the generated series and draw samples lazily.

All 82 tests pass except the two ignored timing tests (F7, P15).
- H1: a Processes engine is measured on its server pids. ProcessesCpuClock
  sums utime + stime under a ProcFs (thread CPU 0); run_arm and
  conformance::check take MeasurementClocks and build the engine's clock
  from cpu_accounting(). E10 now runs a fake server under a fake /proc.
- M1: the calling-thread check repeats each step until 20 ms of CPU, then
  applies the 10% rule.
- M2: answers are keyed by query id outside the labels; QUERY_LABEL deleted.
- M3: measure_sizes runs on the protocol, takes a core and a LoadProbe,
  and returns a Result.
- M4: one signed relative error per held-out point.
- M5: samples share their series labels (Arc); run_arm documents the
  memory of its installs; consecutive jobs share one idle sample.
- L1-L8 and nits: Option instead of NaN in records, NaN-cost gate failure
  and zero-duration refusal, private RunDir path and create_new result
  files, staleness message, FakeClock::ticks deleted, owning into_samples
  iterator, ConstantNotIdentified doc, relative units tolerance, renames.
Re-pinning the main thread, the process-CPU median, a zombie server pid,
server CPU at ns resolution, a nonexistent core refused before the idle
sample, and answers freed outside the timed window.
- Pinning reads the process's allowed cores once, before any pin, so the
  main thread can be re-pinned; a core outside the set is refused before
  the idle sample.
- run_job takes the EngineClock and medians the CPU it measures the engine
  by (process CPU for Processes engines); MedianClock deleted;
  measure_sizes moves to protocol.rs and takes the EngineClock too.
- Processes engines read their own cgroup v2 cpu.stat usage_usec, else
  their threads' schedstat (ns); a zombie pid is ProcessExited; the wall
  clock is read innermost around the call.
- run_arm frees every run's answers after the job, outside the timed window.
- NodeTerms and InstalledArm::node_terms, ProfilingMethod and
  MeasurementClocks::os deleted; ranking_agreement takes
  SelectedAndExactCosts rows; data_statistics groups by label Arc and keeps
  a running value range.
- docs/aqpbm-harness.md, README link, into_samples in the datagen doc; test
  comments no longer carry PR numbers; generic machine-key example.
@zzylol
zzylol requested a review from GordonYuanyc November 17, 2026 06:40
@GordonYuanyc

Copy link
Copy Markdown
Collaborator

Is it possible / appropriate to put aqpbm-harness into aqpbm-core, as aqpbm-harness seems to be some shared functionalities across different cases/crates/etc.?

@zzylol

zzylol commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Is it possible / appropriate to put aqpbm-harness into aqpbm-core, as aqpbm-harness seems to be some shared functionalities across different cases/crates/etc.?

Yes, that's another way to place it. I can change.

The aqpbm-harness crate becomes the harness module of aqpbm-core
(aqpbm_core::harness::{engine, protocol, machine, rundir, records, fit,
accuracy, data, data_stats, gate, sweep, fake, conformance, arm_run}).
Its tests move to aqpbm-core/tests/harness_*.rs, the dependency rule
now checks aqpbm-core with default features off, and the design doc
becomes docs/aqpbm-core-harness.md. aqpbm-core gains only thiserror and
serde's rc feature.
@zzylol zzylol changed the title harness: aqpbm-harness core (engine traits, protocol, fit, records, conformance kit) harness: aqpbm-core harness module (engine traits, protocol, fit, records, conformance kit) Nov 17, 2026
harness::accuracy becomes harness::error_quantiles, so it no longer
reads like core's accuracy comparators; the crate doc points to the
harness; the last aqpbm_harness temp-file prefix is renamed.
@zzylol

zzylol commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Done: the harness now lives in aqpbm-core as the harness module (aqpbm_core::harness, moved in b8555e2, review fixes in 1785074), and the aqpbm-harness crate is gone. Heavy dependencies stay behind core's optional features, so aqpbm-core with default-features = false still has no sketch library, polars or clap (checked by aqpbm-core/tests/dependencies.rs), which keeps it light for downstream users.

@GordonYuanyc GordonYuanyc left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@GordonYuanyc
GordonYuanyc merged commit 12e0f31 into main Nov 17, 2026
1 of 2 checks passed
@GordonYuanyc
GordonYuanyc deleted the harness/core-tests branch November 17, 2026 20:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants