Skip to content

feat(devtools, executor, planner): the in-process ASAP lane of the engine comparison (asap_run bench-external --lane asap) - #1410

Merged
zzylol merged 7 commits into
mainfrom
ap2-asap-lane
Nov 17, 2026
Merged

zzylol merged 7 commits into
mainfrom
ap2-asap-lane

Conversation

@zzylol

@zzylol zzylol commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Problem: an engine comparison runs one selected plan on several engines over the same data, but the Planner's own executor could not be one of them: nothing installed a plan file and ran it as an engine under test of the sketch-bench harness.
  • Change: asap_devtools::asap_lane::AsapEngine installs a plan file written by asap_planner --plan-file as given and runs it with the Planner's executor on the calling thread. asap_run bench-external --lane asap --run-dir R runs it over a harness run directory. A plan the executor cannot compile becomes a not executable: cell, and the other lanes still run that plan.
  • Effect: sum_over_time(m[5m]) every 10 s, planned as a chain of 10 s tumbling windows built at ingestion time, runs over 2 series × 360 s. It gives 36 ingest steps and 7 evaluations, each answered exactly (e.g. {s="s_0"} 1525.73 at 300 s). Footprint: 21,120 bytes of stored views and 4,232 bytes of raw samples.

Review guide

  • Base: ap1-plan-file (feat: plan file for every selected plan; the Planner's executor check only reported (--plan-file) #1402, --plan-file), stacked on it. Commits: the failing tests first, then the lane, then the independent review's fixes, then the executor-only example switched to absent_over_time, then the harness taken from aqpbm-core, then the lane tests adapted to feat: plan file for every selected plan; the Planner's executor check only reported (--plan-file) #1402's plan file (written for every selected plan; the Planner's executor check only reported).
  • Read first: crates/devtools/src/asap_lane/installed_plan.rs (install, ingest, evaluate). Then crates/executor/src/sources/memory.rs (raw tables bound for appending).
  • Careful review: the refresh-window arithmetic (window_ends, and the empty windows before the data start), and the raw lookback in InstalledPlan::install.
  • Mechanical: InstallablePlan::compile_query_run / compile_ingestion_refresh return the DAGs the compile check already built. PhysicalASAPDAG::raw_data_schemas is the schema union that the planner tests' stand-in runtime had, moved out so the lane and the stand-in share it.
  • Pins: aqpbm-core with default-features = false (the harness is aqpbm_core::harness; its normal dependencies are aqpbm-datagen, anyhow, bitflags, chrono, libc, rand, rand_xoshiro, serde, serde_json, thiserror: no sketch library, polars, hdrhistogram or CLI crate) and, as a dev-dependency, aqpbm-datagen, both pinned to sketch-bench 7c0d121192beaf3b68b1b1ec36aab8aafc90d0ae, the head of its bench-external PR. sketch-bench never depends on the planner.

Behavior: before → after

The user writes a planning workload (sum_over_time(m[5m]) every 10 s, exact, m ingesting at 1,000 rows/s from 1,000 series sampled every second) and a harness run directory with workloads/w.json (the same query, 2 series, one sample per second for 360 s):

asap_planner --workload planning.json --run-until stage3 --plan-file R/plans/ingesting/w.plan.json
asap_run bench-external --lane asap --run-dir R --warmup 0 --runs 1
  • Before: there is no asap_run. The plan file can be translated for other engines, but the Planner's executor cannot run it as a lane.

  • After: asap_run prints w ingesting Ran and writes results/asap/w__ingesting.cell.json and results/asap/w__ingesting.json. The arm result holds:

    • stored_outputs ["n1 refreshed AtIngestion, kept 300000 ms"];
    • footprint: 21,120 view bytes and 4,232 raw bytes;
    • answers at 300,000, 310,000, …, 360,000 ms per series, e.g. {s="s_0"}: 1525.733751382602;
    • CPU per ingest and per evaluation step, as the harness measures it.

    For a plan the executor cannot compile (absent_over_time(m[10s])), asap_planner --plan-file still writes the plan file and prints note: the Planner's executor cannot run this plan: .... On the ASAP lane, the cell is NotRun { error: "not executable: the executor cannot compile the plan's query run: ..." }, from the lane's own compile at install.

This serves the engine comparison: the ASAP lane runs exactly the DAG that the other lanes translate.

Interfaces

crates/devtools/src/asap_lane/mod.rs

+/// The Planner's executor as an engine under test of the sketch-bench harness: installs the
+/// plan file named by `arm.engine_options["plan_file"]`; version from `ASAP_PLANNER_COMMIT`
+/// (and `ASAP_PLANNER_DIRTY=1`) at build time.
+pub struct AsapEngine;
+impl EngineUnderTest for AsapEngine {
+    /// `asap`, the build's commit and profile.
+    fn description(&self) -> EngineDescription;
+    /// `CallingThread`: every ingestion and query run runs on the calling thread.
+    fn cpu_accounting(&self) -> CpuAccounting;
+    /// Reads and validates the plan file, matches `query_texts[i]` to one of the texts (any
+    /// language) of `workload.queries[i]`, compiles it. Errors (bad input):
+    /// `plan file <path>: ...` (missing, not a plan, other queries),
+    /// `not executable: ...` (the executor cannot compile it),
+    /// `unsupported by the ASAP lane: ...` (a view refreshed on first read, a group with no
+    /// bounded window, a SQL table source).
+    fn install(&self, workload: &WorkloadSpec, arm: &ArmSpec)
+        -> Result<Box<dyn InstalledArm>, EngineError>;
+}

crates/devtools/src/bin/asap_run.rs

+# Runs every (workload, variant) cell of a run directory on AsapEngine (aqpbm_core::harness::bench_external::run_lane,
+# idle probe and CPU clock as approxbench's lanes). Exit 2: usage error; 1: the lane cannot run.
+asap_run bench-external --lane asap --run-dir <dir> [--variant <v>]... [--core <n>] [--warmup <n>] [--runs <n>] [--max-core-busy <f>]

crates/executor/src/sources/memory.rs

 impl InMemoryDataStore {
     /// Bind `batches`, all in `schema`, as the rows of `source`: every read returns every row.
     pub fn with_raw_data(self, source: Source, schema: SchemaRef, batches: Vec<Batch>) -> Result<Self, Error>;
+    /// Bind `source` for appending its rows as they arrive, each batch kept until the newest
+    /// end is `lookback_ms` past its end (`None`: forever). Bad input: already bound.
+    pub fn with_appended_raw_data(self, source: Source, schema: SchemaRef, lookback_ms: Option<u64>) -> Result<Self, Error>;
+    /// Append `batches` covering `(start, end]`, then drop the batches past the lookback; a read
+    /// returns the batches overlapping its range. Bad input: not bound for appending, another
+    /// schema, empty range or not after the last end.
+    pub fn append_raw_data(&mut self, source: &Source, time_range_ms: (i64, i64), batches: Vec<Batch>) -> Result<(), Error>;
+    /// Bytes of the raw rows held.
+    pub fn raw_data_bytes(&self) -> usize;
+    /// Bytes of the materialized views' stored rows.
+    pub fn materialized_view_bytes(&self) -> usize;
     /// Append `batches` as the rows of `view` covering `time_range_ms`.
     pub fn append_materialized_view(&mut self, view: &MaterializedView, time_range_ms: (i64, i64), batches: Vec<Batch>) -> Result<(), Error>;
 }

crates/planner/src/installable/mod.rs

 impl InstallablePlan {
+    /// The executor's query run, as a runtime runs it at each evaluation; `Err` is
+    /// `QueryRunNotExecutable` (unsatisfiable request) or why the views cannot be derived.
+    pub fn compile_query_run(&self) -> Result<CompiledPhysicalDAG, InstallablePlanError>;
+    /// The executor's refresh of one ingestion refresh group (its views as roots), as a runtime
+    /// runs it once per window; `Err` is `MaterializedViewNotExecutable` or why the views
+    /// cannot be derived.
+    pub fn compile_ingestion_refresh(&self, group: &IngestionRefreshGroup) -> Result<CompiledPhysicalDAG, InstallablePlanError>;
 }

crates/types/src/plan/materialized_view/raw_data.rs, mod.rs

 impl PhysicalASAPDAG {
+    /// Each raw-data source the DAG's `Scan`s read, with the union of their columns by name:
+    /// the raw table a runtime binds, e.g. `m` as `(ts, value, $promql_series_identity, job)`.
+    pub fn raw_data_schemas(&self) -> Vec<(Source, Schema)>;
 }
 impl RawDataRetention {
-    /// (test-support only)
+    /// The source; now public, since a runtime decides what raw data to keep from it.
     pub fn source(&self) -> &Source;
-    /// (test-support only)
+    /// The longest `retention_ms` of a raw-data `Scan` of it; `None` when one reads it unbounded.
     pub fn lookback_ms(&self) -> Option<u64>;
 }

Usage (end to end; it runs as asap_lane::the_asap_lane_runs_a_planned_sum_over_time_and_answers_exactly and asap_run_runs_the_asap_lane_over_a_run_dir):

// The plan file: asap_planner plans the workload (sum_over_time(m[5m]) every 10 s, exact,
// m ingesting) and writes the selected InstallablePlan (exit 0; a stderr note when the
// Planner's executor cannot run it).
//   asap_planner --workload planning.json --run-until stage3 --plan-file R/plans/ingesting/w.plan.json
// The run directory R also holds workloads/w.json (the harness WorkloadSpec: query "sum" with the
// same text, 2 series of m, one sample per second for 360 s) and plans/ingesting/variant.json.
let run = RunDir::open(Path::new("R"))?;
// One lane process: the harness installs the plan file per run (engine options
// {"plan_file": ".../w.plan.json"}), ingests 36 chunks of 10 s and evaluates at 300..360 s.
let cells = run_lane(&run, &AsapEngine, None, LaneSetup { protocol: &protocol, core, probe, clocks: &clocks, args })?;
// AsapEngine::install: from_json + validate; query text 0 matches workload query "sum"; compiles the
// query run (TimeRange 300s -> SummaryMerge -> FinalizeExactAccumulator over view n1) and the
// refresh of group {n1} (SummaryTumblingWindowChain, width 10 s, phase 0); binds m kept 310 s.
// ingest: appends each chunk's rows of m for (end - 10 s, end], then refreshes n1's window ending there.
// evaluate(T): one query run at T; root 0's rows become answers["sum"][{s="s_0"}] = 1525.73 (T = 300 s).
assert_eq!(cells[0].outcome, CellOutcome::Ran);

Algorithm changes

The ASAP lane runtime (new; installed_plan.rs):

  • Problem: there was no runtime in-tree, outside tests, that maintains a plan's views as samples arrive.
  • Algorithm:
    1. At install, compile the query run and each ingestion refresh group once.
    2. Bind each time series the plan scans (raw_data_schemas). It is kept back to the larger of its query-time lookback (raw_data_retention) and the longest refresh width plus one chunk.
    3. Refresh empty the windows before the data start that a view keeps.
    4. On each chunk (a, b]: route samples once per series (a cached table and ordinal per label-map address) into one dictionary-encoded batch per series table, and append it.
    5. Then, for each group and each window end e ∈ (a, b] with e + phase ≡ 0 (mod width), run one ingestion run and append each view's output for (e − width, e].
    6. On each evaluation: run one query run, root i → query i.
  • Correctness:
    • Answers equal a hand computation from the regenerated samples at every scheduled time, for an ingestion-time chain and for raw scans at query time, with 4 interleaved installs.
    • The conformance kit passes (determinism, left-open windows, work on the calling thread).
    • The executor's tiling check refuses a missing window instead of answering wrong.
  • Complexity:
    • Ingest is O(samples) plus a dictionary rebuild per table when a series first appears, plus one ingestion run per window.
    • Each run instantiates the compiled DAG, which is O(nodes).
    • Raw reads return only the batches overlapping the requested range.
  • Assumptions and limits:
    • Evaluations are on the evaluation interval's grid at phase 0, and chunks are one interval each, as ArmSchedule::for_workload makes them.
    • Not supported: views refreshed on first read, groups with no bounded window, and SQL table sources. Each is refused at install as unsupported by the ASAP lane:.

Decisions and open questions

  • Needs reviewer decision: output series labels. The lane reports the executor's PromQL output labels, so sum_over_time(m[5m]) answers under {s="s_0"}, without __name__. The harness's Answers doc example keeps __name__. A report that joins lanes needs one convention. Recommendation: PromQL's (the Prometheus ground truth drops it), and the harness doc follows.
  • Decided:
    • A plan text matches a workload query when it equals any of that query's texts, in any language: suites may carry MetricsQL texts.
    • The lane reuses the executor's InMemoryDataStore, extended with raw tables bound for appending, rather than implementing its own DataStore. The alternative would have duplicated the store's projection and view tiling.
    • The lane compiles the plan at install: a plan the executor cannot compile is a not executable: cell, not a missing plan. asap_planner writes the plan file either way, so the other lanes still get it.

Verification

Run on a build host, at the PR head.

Scope Result Newly failing vs base Notes
nextest -p asap-devtools --test devtools asap_lane 5 passed — new
nextest -p asap-executor 337 passed, 2 skipped — includes the new raw-table test
nextest -p asap-planner --lib installable 8 passed —
nextest -p asap-planner --test planner 63 passed, 1 failed no summary_sharing::identical_ungrouped_queries_share_their_producers fails identically at the base commit
nextest -p asap-types 280 passed —
nextest -p asap-devtools --test devtools -E 'test(/^asap_lane::|^asap_planner::/)' (after the rebase onto #1402's new head) 58 passed — lane + CLI tests
cargo clippy -p asap-devtools -p asap-executor -p asap-planner -p asap-types --all-targets -- -D warnings clean —
cargo fmt --check clean —

New tests:

  • conformance_kit_passes_on_the_asap_engine: the harness's conformance kit on probe plans the planner library made.
  • the_asap_lane_runs_a_planned_sum_over_time_and_answers_exactly: an ingestion-time chain and raw query-time scans answer exactly at every scheduled time, with 4 installs interleaved, and the footprint and plan summary are filled in.
  • a_plan_the_executor_cannot_compile_is_a_not_executable_cell: for absent_over_time, asap_planner --plan-file exits 0, writes the file and prints the executor note; the lane gives NotRun starting not executable: .
  • a_plan_for_other_queries_is_refused_naming_the_query: these are refused: no plan file, a missing file, not a plan, another text, a count mismatch. A plan text that appears as a query's second text matches.
  • asap_run_runs_the_asap_lane_over_a_run_dir: the binary writes a Ran cell, and an unknown lane exits 2.
  • appended_raw_data_is_read_by_range_and_dropped_after_its_lookback: the store reads by overlap, drops at the lookback boundary, is append-only, and keeps the whole-bound tables unchanged.

End-to-end CLI evidence: see Behavior. That run used the build's debug binaries over the demo run directory.

Follow-ups

  • predicted() from Stage 3's report. The plan file does not carry it.
  • Views refreshed on first read, and unbounded refresh groups. Both are refused today.
  • Per-operator timing (node_profiles).
  • The output-label convention above.

@zzylol zzylol added dev-tools Developer tools, diagnostics, and demos enhancement New feature or request testing Corpus, regression, differential, or benchmark tests labels Nov 17, 2026
@zzylol
zzylol force-pushed the ap2-asap-lane branch 2 times, most recently from 26cddec to f865c26 Compare November 17, 2026 21:10
AsapEngine installs a plan file written by asap_planner --plan-file and
runs it with the Planner's executor under the sketch-bench harness:
the harness's conformance kit, sum_over_time answered exactly from an
ingestion-time chain and from raw samples at query time, a plan the
executor cannot compile as a 'not executable' cell, install refusals,
and asap_run bench-external --lane asap. The executor's in-memory store
gains appended, time-ranged raw data dropped after its lookback.
AsapEngine (asap_devtools::asap_lane) installs a plan file written by
asap_planner --plan-file and runs it with the Planner's executor as an
engine under test of the sketch-bench harness; asap_run bench-external
--lane asap runs it over a run directory. A plan the executor cannot
compile is an install error starting 'not executable:'.

The lane is a deployment runtime over the executor's public API: the
executor's InMemoryDataStore gains raw tables bound for appending, read
by the batches overlapping the requested range and dropped after their
lookback, and byte accessors; InstallablePlan exposes the compiled query
run and ingestion refreshes its compile check already built; the
RawDataRetention accessors are public, since a runtime reads them.
One raw-table schema rule (PhysicalASAPDAG::raw_data_schemas) for the
lane and the planner tests' stand-in runtime; samples routed once per
series, with dictionary texts cached per table; views' windows before
the data start refreshed empty at install; two answer rows of one
series are an error; no empty raw appends; simpler raw binding in the
in-memory store. Pins sketch-bench at the rebased head (harness
unchanged).
last_over_time now compiles on the executor; absent_over_time is still
planned but not compiled.
sketch-bench moved its harness into aqpbm-core (aqpbm_core::harness);
pin aqpbm-core without default features (no sketch library, polars or
CLI crate) at the bench-external head.
…no longer gates

asap_planner --plan-file now writes every selected plan; a plan the
Planner's executor cannot run comes with a stderr note. The lane tests write
plan files without --executor-check, and the absent_over_time case checks
the note, then the lane's not executable cell.
The check compares thread and process CPU; other tests running in the same process added process CPU the lane did not spend, so it failed whenever it ran beside them.
@zzylol
zzylol changed the base branch from ap1-plan-file to main November 17, 2026 21:25
@zzylol
zzylol merged commit 1abbeab into main Nov 17, 2026
2 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dev-tools Developer tools, diagnostics, and demos enhancement New feature or request testing Corpus, regression, differential, or benchmark tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant