Skip to content

engines-external: read an ASAPPlanner plan file and translate it to ClickHouse views - #260

Draft
zzylol wants to merge 16 commits into
mainfrom
engines-external/translator
Draft

zzylol wants to merge 16 commits into
mainfrom
engines-external/translator

Conversation

@zzylol

@zzylol zzylol commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Problem: sketch-bench cannot run an ASAPPlanner plan on ClickHouse. Example: the planner picks stored 1-minute windows for sum by (job) (sum_over_time(latency[1h])), evaluated every 60 s. Nothing turns that plan into ClickHouse materialized views and a query, so the ClickHouse baseline cannot run the same plan as the ASAP executor (Translate an InstallablePlan into each engine's deployment form #253, Engine capability profiles with conformance tests #252).
  • Change: a new crate, aqpbm-engines-external.
    • It reads an ASAPPlanner plan file (InstallablePlan, wire version 15) with a small local decoder.
    • It translates the plan into ClickHouse tables, materialized views (AggregatingMergeTree, -State/-Merge) and one SQL query per query root.
    • The translator is the capability check: a node it has no rule for is refused, with the node id and a reason.
    • It commits four ASAPPlanner deployment-capabilities files (ClickHouse and VictoriaMetrics, each equivalent-only and with-substitutes), and a README table with each entry's class and evidence.
  • Effect:
    • asap_planner --deployment-capabilities capabilities/clickhouse.equivalent-only.json plans only what ClickHouse can do, and this crate turns every such plan of the public workloads into SQL with the same answers. All 206 public PromQL workloads were planned with each ClickHouse file: all 412 plans translate, and no equivalent-only plan has a substitution.
    • Sharing built at ingestion is kept; query-time sharing that is lost is reported.
    • The one substitute so far, stored counter windows, is recorded with its guarantee difference.

Review guide

  • Base: harness: aqpbm-core harness module (engine traits, protocol, fit, records, conformance kit) #256 (harness/core-tests). Draft.
  • Commits:
    1. edb2270: the failing tests;
    2. 1276b94: the fixes from the independent review of the tests;
    3. 4b5df37: the implementation;
    4. 6529c5a: the fixes from the independent review of the implementation;
    5. edacc69: the failing tests for the second independent review (below);
    6. 7b6710a: its fixes;
    7. ebd5be9: the fixes from the independent review of those fixes (last table);
    8. 8a8dda4: cites ASAPPlanner main bc725bda8, where #1374 is merged; every committed fixture regenerated there is byte-identical.
  • Read first: src/clickhouse.rs: translate, refuse_unsupported_fields, Translator::relation (the pattern dispatch), Translator::stored_read (the counter segment query), Translator::binary; src/plan_file.rs: PlanFile::from_value, operator_of.
  • Mechanical: the capability JSON files (one format), tests/fixtures/plans/** and tests/fixtures/workloads/** (planner output and its inputs, never hand-edited; see REV).

Second review: findings and fixes

Finding Fix Test
HIGH Stored rate/increase was wrong under parallel reads (1.7× at 16 threads): deltaSumTimestampMerge of non-adjacent window states counts the increase between them twice Each stored row is read as one time-ordered segment (finalizeAggregation); a series' segments are joined in time order with lagInFrame (Σ in-segment increase + Σ max(0, first_k − last_(k−1)), deltaSumTimestamp's own rule); overlapping segments are an error; new s_last_value (argMax) state stored_rate_is_right_under_parallel_reads (2,000 counters, 60 inserts, 1/16/40 threads, merged, partly merged), stored_rate_differs_from_promql_by_the_first_value_after_each_reset, overlapping_stored_segments_are_an_error, golden f2
MEDIUM The equivalent-only file admitted plans the translator refused (top_k, grouped summaries incl. the showcase q4c shape, regrouping merges, instant selectors) New rules: grouped query-time summaries (over raw samples or another relation), regrouping merges, grouped stored chains, top_k (LIMIT k BY), instant selectors (argMax), Project, arithmetic BinaryOp; raw data kept at ingestion time is read like a query-time scan public_plans_translate (29 planner-written public plans), unit SQL tests, ClickHouse answers for each rule, public_translations_run
MEDIUM An unaligned evaluation time gave a silently wrong answer Every stored read checks (SELECT throwIf({t} % w != 0, …)) = 0 (a scalar subquery, so it runs even when no row matches) unaligned_evaluation_time_is_an_error, goldens
MEDIUM Retention was dropped retention_ms decoded; ClickHouseDeployment.table_retention per table (the planner's raw-data retention rule for sources); ttl_statements() kept out of script(): a TTL deletes by wall-clock time, so it applies only under wall-clock data (under replay the stored bytes include rows past retention) retention_is_recorded_and_ttl_is_separate, ttl_statements_apply_and_null_values_are_refused
Predicate semantics Only "semantics": "promql" label equality is LabelEquals only_promql_label_equality_is_label_equals
Aggregate.filters is one optional filter per measure measure_filters: Vec<Option<Value>>; [null] is no filter aggregate_filters_are_one_optional_predicate_per_measure
Series identity type String (one value per series), labels LowCardinality(String) goldens
Source sort key for label filters ORDER BY (<filtered labels>, $promql_series_identity, ts) + minmax index on ts a_label_filter_leads_the_source_sorting_key, golden f4
NULL values were stored as 0 INSERT_SETTINGS (input_format_null_as_default = 0, insert_null_as_default = 0) ttl_statements_apply_and_null_values_are_refused
Binary pin README cites 26.8.39.13 amd64compat, git hash d5bbb64d, its sha256; X10 checks version() and the git hash X10
Dead checks, translator state, names Predicate columns checked in the first pass; the exact kind resolved once; the query step travels with each relation; SummaryTumblingWindowChain, FinalizeExactAccumulator, s_positive_delta_sum —
README permalinks, JSON format Line-range permalinks at the binary's commit; all four files pretty-printed in the README's order capability_files_are_pretty_printed

Review of the fixes (applied in ebd5be9)

Finding Fix Test
A per-series BinaryOp between two metrics matched nothing (the series identity names the metric) Refused per_series_binary_operation_across_metrics_is_refused
A metric only stored chains read kept every raw row ENGINE = Null source (its views still see every insert) a_metric_only_chains_read_keeps_no_rows, goldens f1, f2, f5
A Project could take a non-value column as the value, or drop a label Refused projections_keep_every_label_and_one_value
A raw-data Scan read outside a time range got a TTL No retention, as in the planner —
Names, docs, test-helper duplication, README cost and rounding notes Renamed (unfinalized_kind, whole_number_argument, PlanNode::is_raw_data_scan); helpers in tests/common —

ASAPPlanner #1374 moved to 7834fed4d (its excluded_candidates dropped counter_semantics, merged_across_windows and sketch_parameters, and added sketch_parameter_range). The four files use none of those fields and stay valid; the six named fixtures regenerate byte-identical at that head.

Checked on ClickHouse

All #[ignore] tests (AQPBM_CLICKHOUSE_LOCAL) pass on ClickHouse 26.8.39.13: the goldens with hand-computed answers (X10), and every case in clickhouse_run.rs:

Case Data Answer
stored rate, parallel reads 2,000 counters v = s · (1 + k mod 7), 60 inserts rate = 1 + k mod 7 per counter, summed per job, at 1/16/40 threads and merged/partly merged
stored vs query-time rate with resets x (reset at an insert boundary), y (at a window boundary), z (partial drop + reset), w (one sample per insert) stored 298/299, 416/299, 247/299, 1; query-time 1, 417/299, 642/299, 1
q4c grouped sum a = 1.0 + 0.5, b = 2.0 per second a 90, b 120
top_k rates 1–5 and a NaN series k = 2 → i5, i4; k = 5 → i1..i5
instant selector (5 ms lookback) samples at t − 10, t − 5, t − 3, t − 1, t ms a 3, b 4 (t − 5 excluded)
avg as sum / count (mixed-03) constant series 1, 3, 5 r1/s1 2, r2/s1 5; an overflowing sum is an error

Manual check

At #1374 head 7834fed4d, the 206 public PromQL workloads were planned with each file (out-of-repo driver calling installable_plan_of): every file is accepted; ClickHouse equivalent-only and with-substitutes give a plan for all 206; VictoriaMetrics equivalent-only gives no plan for 80 (ExcludedByDeployment, counter_resets_not_handled), as its README says; for sum by (job) (rate(requests_total[5m])) the ClickHouse equivalent-only file selects the query-time rate.

Design scope

New item Scope Why it is general
TableRetention, ClickHouseDeployment::{table_retention, ttl_statements} ClickHouse deployment any plan's retained tables; TTL kept apart because it depends on the run's clock, not the plan
INSERT_SETTINGS ClickHouse ingest contract every insert into a source table
PlanNode::{retention_ms, timestamp_columns, is_value_column, is_raw_data_scan} plan-file decoding the planner's node fields and its raw-data-scan rule; a value column is any non-label, non-timestamp output
PlanOperator::{Project, BinaryOp}, measure_filters plan-file decoding planner operators and fields, decoded as on the wire
Relation.{step_ms, unfinalized_kind, source_tables} (private) translator every relation carries its evaluation step, whether it is an unfinalized query-time state, and the tables it reads

Interfaces

  pub mod clickhouse {
+     pub const INSERT_SETTINGS: &str;
      pub struct ClickHouseDeployment {
          pub tables: Vec<String>,
          pub materialized_views: Vec<String>,
          pub queries: Vec<String>,
          pub query_time_steps_ms: Vec<u64>,
+         pub table_retention: Vec<TableRetention>,
          pub substitutions: Vec<Substitution>,
          pub sharing_lost: Vec<SharingLost>,
      }
+     pub struct TableRetention { pub table: String, pub row_time_sql: String, pub retention_ms: u64 }
      impl ClickHouseDeployment {
          pub fn script(&self) -> String;
+         pub fn ttl_statements(&self) -> Vec<String>;
      }
      pub fn translate(plan: &PlanFile) -> Result<ClickHouseDeployment, TranslationError>;
  }
  pub mod plan_file {
      pub struct PlanNode {
          pub id: NodeId,
          pub operator: PlanOperator,
          pub timing: Timing,
          pub lifetime: Lifetime,
+         pub retention_ms: Option<u64>,
          pub output_columns: Vec<String>,
          pub label_columns: Vec<String>,
+         pub timestamp_columns: Vec<String>,
      }
+     impl PlanNode { pub fn is_value_column(&self, index: usize) -> bool; pub fn is_raw_data_scan(&self) -> bool; }
      pub enum PlanOperator {
-         WindowChain { child, family, width_ms, reduction, input, filter },
+         SummaryTumblingWindowChain { child, family, width_ms, reduction, input, filter },
-         Finalize { child },
+         FinalizeExactAccumulator { child },
-         Aggregate { child, measures, reduction, filters: Vec<Value>, having },
+         Aggregate { child, measures, reduction, measure_filters: Vec<Option<Value>>, having },
+         Project { child: NodeId, expressions: Vec<Value> },
+         BinaryOp { lhs: NodeId, rhs: NodeId, operator: Value, return_bool: bool },
          ...
      }
  }

Verification

  • cargo test -p aqpbm-engines-external: 40 pass (10 ClickHouse cases ignored without a server); cargo clippy -p aqpbm-engines-external --all-targets -- -D warnings: clean; cargo fmt --check: clean.
  • #[ignore] tests with AQPBM_CLICKHOUSE_LOCAL="<clickhouse 26.8.39.13> local": clickhouse_run 10 of 10, X10 pass.
  • Full sweep (out of repo): the 412 ClickHouse plans of the 206 public PromQL workloads all translate; 0 substitutions in the 206 equivalent-only plans.

The engine traits, records and every module of the harness core, with
todo!() bodies; ThreadCpuTime in aqpbm-core with its unit test. The
implementation follows once the tests are approved.
82 tests from the approved S1 plan: machine key, protocol, fit with
leave-one-setting-out, accuracy curves, data and data statistics, the
fake engine and conformance kit, run directory, gate and sweep, and the
dependency rule. 31 are moved or adapted from ASAPPlanner #1332. They
compile against the todo!() skeleton and fail there, except the
dependency rule, which already holds.
The engine traits, measurement protocol (idle check, pinning, warm-up,
median of an odd run count), machine key, relative-weighted fit with
leave-one-setting-out, accuracy curves, data chunks and statistics, run
records and directory, gate and sweep arithmetic, the fake-clock engine
and the conformance kit. aqpbm-core's ThreadCpuTime reads
CLOCK_THREAD_CPUTIME_ID; aqpbm-datagen gains TimeSeriesCursor so a reader
can own the generated series and draw samples lazily.

All 82 tests pass except the two ignored timing tests (F7, P15).
- H1: a Processes engine is measured on its server pids. ProcessesCpuClock
  sums utime + stime under a ProcFs (thread CPU 0); run_arm and
  conformance::check take MeasurementClocks and build the engine's clock
  from cpu_accounting(). E10 now runs a fake server under a fake /proc.
- M1: the calling-thread check repeats each step until 20 ms of CPU, then
  applies the 10% rule.
- M2: answers are keyed by query id outside the labels; QUERY_LABEL deleted.
- M3: measure_sizes runs on the protocol, takes a core and a LoadProbe,
  and returns a Result.
- M4: one signed relative error per held-out point.
- M5: samples share their series labels (Arc); run_arm documents the
  memory of its installs; consecutive jobs share one idle sample.
- L1-L8 and nits: Option instead of NaN in records, NaN-cost gate failure
  and zero-duration refusal, private RunDir path and create_new result
  files, staleness message, FakeClock::ticks deleted, owning into_samples
  iterator, ConstantNotIdentified doc, relative units tolerance, renames.
Re-pinning the main thread, the process-CPU median, a zombie server pid,
server CPU at ns resolution, a nonexistent core refused before the idle
sample, and answers freed outside the timed window.
@zzylol zzylol added the enhancement New feature or request label Nov 17, 2026
@zzylol
zzylol requested a review from GordonYuanyc November 17, 2026 06:26
- Pinning reads the process's allowed cores once, before any pin, so the
  main thread can be re-pinned; a core outside the set is refused before
  the idle sample.
- run_job takes the EngineClock and medians the CPU it measures the engine
  by (process CPU for Processes engines); MedianClock deleted;
  measure_sizes moves to protocol.rs and takes the EngineClock too.
- Processes engines read their own cgroup v2 cpu.stat usage_usec, else
  their threads' schedstat (ns); a zombie pid is ProcessExited; the wall
  clock is read innermost around the call.
- run_arm frees every run's answers after the job, outside the timed window.
- NodeTerms and InstalledArm::node_terms, ProfilingMethod and
  MeasurementClocks::os deleted; ranking_agreement takes
  SelectedAndExactCosts rows; data_statistics groups by label Arc and keeps
  a running value range.
- docs/aqpbm-harness.md, README link, into_samples in the datagen doc; test
  comments no longer carry PR numbers; generic machine-key example.
The aqpbm-harness crate becomes the harness module of aqpbm-core
(aqpbm_core::harness::{engine, protocol, machine, rundir, records, fit,
accuracy, data, data_stats, gate, sweep, fake, conformance, arm_run}).
Its tests move to aqpbm-core/tests/harness_*.rs, the dependency rule
now checks aqpbm-core with default features off, and the design doc
becomes docs/aqpbm-core-harness.md. aqpbm-core gains only thiserror and
serde's rc feature.
@zzylol
zzylol force-pushed the engines-external/translator branch from 8a8dda4 to 4032055 Compare November 17, 2026 19:50
harness::accuracy becomes harness::error_quantiles, so it no longer
reads like core's accuracy comparators; the crate doc points to the
harness; the last aqpbm_harness temp-file prefix is renamed.
…kHouse translator

New crate `aqpbm-engines-external` (skeleton only) with its tests:

- tests/plan_file.rs (R1-R6): reading an ASAPPlanner InstallablePlan
  file, wire version 15: node fields, label-equality predicates, other
  wire versions refused, unknown operators kept, standalone scalar
  queries refused, roots and readers.
- tests/clickhouse_translate.rs (X1-X10): tables, materialized views and
  one query per root against hand-written goldens; the stored-rate
  substitute recorded; query-time sharing reported; every refused field
  named at the node that carries it; X10 (ignored) runs the goldens on
  ClickHouse 26.8.34.13 and checks hand-computed answers.
- tests/capability_files.rs (K1): the four deployment-capabilities files
  equal the hand-computed values.

Fixtures are plan files written by the planner for public synthetic
workloads (tests/fixtures/plans/REV); edge cases change one field of a
loaded fixture in test code.
- X6 also checks root 13, the query-time rate, against F3's query.
- X10's counter data now has three series, one per branch of Prometheus
  extrapolatedRate: a reset, a start gap above 1.1 average gaps, and the
  zero-point clamp; each rate is computed by hand in the test. X10 also
  checks the ClickHouse version first.
- R3 nulls the DAG of the other-version file: the version is read first.
- Mutations use planner wire literals ({"Boolean": true}, {"Float64": 1.0}).
- X4 checks that a label value with a quote and a backslash is escaped.
- X2-X5 assert no substitutions / lost sharing where none are expected.
- REV records how the fixtures were made.
…lickHouse

- plan_file.rs: decoder for InstallablePlan wire version 15 over JSON
  values (no ASAPPlanner dependency). The version is checked first; every
  field that can change an answer is decoded; unknown operators decode to
  Other; standalone scalar queries are refused.
- clickhouse.rs: translate() refuses, in node id order, every operator
  and field it has no rule for, then translates each root: stored exact
  Sum/Count/Min/Max/Rate/Increase chains into AggregatingMergeTree tables
  and materialized views, query-time builds and Aggregates into SQL over
  the raw table, with right-closed windows and Prometheus extrapolatedRate.
  The stored counter chain is recorded as a substitution; query-time nodes
  shared by several readers are reported as lost sharing.
- capabilities/: the four deployment-capabilities files and the README
  with each entry's class and evidence.
Correctness:
- A stored chain's view keeps its Scan's label predicates (WHERE).
- A stored read refuses a range that is not a whole number of windows,
  and every query reports the step its evaluation time must be a
  multiple of (query_time_steps_ms: the window width it reads, 1 if none).
- The decoder checks every id: children, edges and roots name nodes, and
  a node reads only earlier nodes (no cycle); roots match query texts;
  versions beyond u32, overflowing or sub-millisecond ranges are refused.
- Labels are the utf8 columns of the output schema, not names; a Reduce
  column must be a label of its input.
- Query-time results kept across evaluations are refused (the ClickHouse
  files say they are not retained).
- Measures over a column, unknown measure arguments and quantile levels
  outside [0, 1] are refused; a grouped SummaryAgg is refused at its node.
- The ClickHouse files list no sketch until the translator builds sketch
  states; the README says so and states the time-order requirement of
  deltaSumTimestamp states.
- Label columns are LowCardinality(String).

Simplicity: one StoredChain per chain (BTreeMap), one ExactKind table,
one free unsupported(), RawSamples instead of a Raw relation variant;
unused fields deleted (wire_version, retention_ms, parameters, the
estimate query, the table reference).

ClickHouse version: the runs use 26.8.39.13, the 26.8 LTS build for CPUs
without AVX2; the goldens pass on it (X10).
Tests first for the independent review of the translator:

- Stored rate under parallel reads and partial merges, counter resets inside an
  insert, at an insert boundary and at a window boundary, single-sample
  segments, and overlapping segments (ClickHouse, ignored without a server).
- Every committed public plan translates, and an equivalent-only one has no
  substitution; the plans are planner-written from public synthetic workloads
  at ASAPPlanner #1374 head 7834fed4d (the named fixtures are unchanged).
- Grouped query-time summaries (the showcase q4c shape), regrouping merges,
  top_k, instant selectors, projections and arithmetic binary operators, with
  hand-computed answers on ClickHouse.
- An unaligned evaluation time is an error; retention is recorded with TTL
  statements kept apart; a NULL value is refused on insert.
- Predicate semantics, per-measure filters, the series identity type, the
  source sorting key, the binary's git hash, and one JSON format for the
  capability files.
- Stored rate/increase: each stored row is read as one time-ordered segment
  (finalizeAggregation) and a series' segments are joined in time order, so
  the answer no longer depends on how a parallel read merges
  deltaSumTimestamp states (it was 1.7x too high at 16 threads). The step
  into a segment follows deltaSumTimestamp's own rule; overlapping segments
  are an error. A new argMax state keeps each segment's last value.
- New rules, so that every equivalent-only plan of the public workloads
  translates: query-time summaries grouped or over another relation (the
  showcase q4c shape), regrouping merges, grouped stored chains, top_k,
  instant selectors, Project and arithmetic BinaryOp. A checked division
  with a value that is not finite is an error.
- A stored read fails at an evaluation time that is not a window end.
- Retention is decoded and recorded per table; TTL statements are offered
  apart from the script, for wall-clock data only.
- PromQL-semantics label equality only; per-measure filters; the series
  identity is a String; the source table is sorted by its filtered labels,
  the series identity and ts, with a minmax index on ts; inserts refuse NULL
  values; names follow the planner's; the query step travels with each
  relation; one JSON format for the capability files; the README cites
  the running binary (26.8.39.13 amd64compat, git hash d5bbb64d) with
  permalinks.
- A per-series binary operation between two metrics is refused: the series
  identity names the metric, so a join on it would match nothing where
  PromQL matches the series without their names.
- A metric that only stored chains read is a Null table: its views still
  see every insert, and no raw rows are kept.
- A projection must keep every label and take the value column.
- A raw-data Scan read outside a time range or window chain keeps every
  row (no TTL), as in the planner's raw-data retention.
- Names: Relation.unfinalized_kind, whole_number_argument,
  PlanNode::is_raw_data_scan; the retention_ms doc follows the planner.
- README: floating-point summation and the cost of the stored counter read.
- Test helpers folded into the common module.
…d capability files

ASAPPlanner #1374 is merged into main as bc725bda8. Every committed plan
fixture regenerated there is byte-identical, and the four capability files
load there unchanged.
@zzylol
zzylol force-pushed the engines-external/translator branch from 4032055 to 2ed3d47 Compare November 17, 2026 19:51
Base automatically changed from harness/core-tests to main November 17, 2026 20:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant