Skip to content

engines-external: ClickHouse adapter, the clickhouse lane, and conformance on a real server - #263

Draft
zzylol wants to merge 6 commits into
bench-external/run-slicefrom
engines-external/clickhouse-adapter
Draft

zzylol wants to merge 6 commits into
bench-external/run-slicefrom
engines-external/clickhouse-adapter

Conversation

@zzylol

@zzylol zzylol commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Problem: bench-external has no ClickHouse lane, so a planner plan cannot run on a ClickHouse server. Example: the showcase sum by (job) (sum_over_time(latency[1m])), every 10 s over 1,000 series, planned with capabilities/clickhouse.equivalent-only.json. engines-external: read an ASAPPlanner plan file and translate it to ClickHouse views #260 can translate that plan to SQL, but nothing installs the SQL on a server, feeds it the harness's data, or reads the server's CPU and memory.
  • Change:
    • ClickHouseEngine, an EngineUnderTest that talks to a ClickHouse server over HTTP. Each install translates its plan file into a fresh database.
    • The clickhouse lane: approxbench bench-external run --lane clickhouse --clickhouse-url URL --cgroup-agent-url URL.
    • scripts/extbench-up.sh, which starts the server: a checked binary in a pinned container on given cores, with the cgroup agent beside it.
    • One conformance case per row of the capability README, run against a real server.
  • Effect: the probe workloads run on the clickhouse lane and match the fake lane exactly in the report (0 missing points, 0 extra points, error 0). Every README row's conformance case passes on ClickHouse 26.8.39.13.

Review guide

  • Base: harness: bench-external lanes over one run directory, report, and cgroup v2 server CPU and memory #261 (bench-external/run-slice). Draft.
  • Commits:
    1. c7a36bb: the failing tests (the module, the lane and the script did not exist yet);
    2. f9b5ae8: the implementation;
    3. 61853a1: the fixes from the independent implementation review.
  • Read first: aqpbm-engines-external/src/clickhouse_server.rs: ClickHouseEngine::install, ClickHouseArm::ingest/evaluate, parse_answer, Http::run. Then scripts/extbench-up.sh.
  • Mechanical: the lane wiring in aqpbm-cli/src/bench_external_cmd.rs; the SourceTable and stored_tables outputs of the translator; the two planner-written probe plans (tests/fixtures/plans/REV).

How the adapter runs an arm

Step What it sends
install CREATE DATABASE bench_<pid>_<n>, then the translated tables and views with database=<db>
ingest(chunk) per source table, one INSERT … SETTINGS input_format_null_as_default = 0, insert_null_as_default = 0 SELECT … FROM input(…) FORMAT TabSeparated with the chunk's rows in chunk order
evaluate(t) each query root's SQL with param_t=t, max_threads=<n> (1 by default), FORMAT TabSeparatedWithNames
footprint system.parts bytes of the view tables and of the source tables
drop DROP DATABASE … SYNC
  • Every request carries wait_end_of_query=1, so a mid-answer exception (e.g. the throwIf on an evaluation time that is not a window end) is an error, and max_execution_time.
  • Query ids: plan root i answers workload query i. The plan's text must equal one of that query's texts, in any language.
  • Series: a sample's series identity is its label map as sorted compact JSON. A per-series answer is keyed by that map (__name__ kept, as the harness keys answers); an aggregated answer by its label columns, empty ones left out (as PromQL leaves out an empty label).
  • Refused at install:
    • a plan the translator refuses;
    • a stored window step that does not divide the evaluation interval and the data start;
    • a label named ts or value;
    • databases left by an earlier run.
  • CPU:
    • The server's own cgroup is read through the agent (CpuAccounting::Cgroup).
    • The server's summary header has no CPU field. Checked on 26.8.39.13, it holds read/written rows and bytes, elapsed_ns and memory_usage. So engine_reported_cpu_ns is None.
    • scanned_bytes is the summed read_bytes.

Conformance on a real server

The server was started with scripts/extbench-up.sh (cores 2 and 22, one physical core, NUMA node 0; agent on another core). The tests ran from the driver host over the private network: cargo test -p aqpbm-engines-external --test clickhouse_conformance -- --ignored --test-threads 1, 13 of 13 passed.

README row Case Measured
stored Sum / Count / Min / Max stored_sum_count_min_max a 8100 / 7200 / 1.5 / 3, b 10800 / 3600 / 2 / 4, as computed by hand
stored Rate / Increase (b) stored_rate_is_promql_minus_the_first_value_after_each_reset, three installs fed in turn, max_threads 1 and 16 x 298/299, y 416/299, z 247/299, w 1 in every install
query-time Rate / Increase query_time_rate_is_promql x 1, y 417/299, z 642/299, w 1
query-time grouped sums; Aggregate grouped_sums_quantiles_and_aggregates q4c a 90, b 120; q1 59.41; q3 81.15; F4 600
instant selector instant_selector a 3, b 4 (the sample at t − 5 ms excluded)
top_k top_k_ties_and_nan k = 2: i5, i7 (tie by labels); k = 6: every series but the NaN one
regrouping, Project, BinaryOp average_as_sum_over_count_and_checked_division r1/s1 2, r2/s1 5; an overflowing sum is an error
IRate; kept query-time results (c) class_c_rows_are_refused both refused at install
raw retained; retention query_time_rate_is_promql, storage_and_retention query-time: raw 20,595 B, views 0; stored-only: raw 0 (Null source), views 2,423 B; the TTL statements apply
window boundary the_harness_conformance_kit_passes DeterministicAnswers, LeftOpenWindow
binary pin the_server_is_the_pinned_binary 26.8.39.13, git hash d5bbb64d

Other measurements:

  • Lane vs fake (the_clickhouse_lane_matches_the_fake_lane, warm-up 1, 3 runs): both probe workloads Ran, with 0 missing points, 0 extra points and error 0. Query CPU was about 6–8 ms per evaluation on the server's cgroup, against 0.009 ms for the in-process fake.
  • Suite size (suite_sized_workload): 1,000 series, 1 s samples, 10 min from 1,700,000,000,000 ms; 60 inserts of 10,000 rows and 55 evaluations in 10.3 s wall time, using 3.3 s of server cgroup CPU. The answers equal the per-job sums computed in the test, and the server was never throttled.
  • Idle server: 0.38 s of CPU per 10 s, about 3.8% of one core, from the server's own background threads. It shows in the whole-arm server CPU and in the idle check of the server's cores.

Start script

  • Config: without a main config file the server runs on its embedded configuration and does not read a mounted config.d. Checked: display_name and the logger level did not change. Settings are therefore --key=value overrides.
  • Ports: the embedded configuration also defines the native, MySQL, PostgreSQL and interserver ports, and a port of 0 binds a random one. They get the four ports after the HTTP port, on the same private address.
  • Refusals:
    • a LISTEN_HOST that is not private or loopback (0.0.0.0 and public addresses; tested);
    • a binary whose sha256 differs;
    • a data dir it did not create (marker file; tested);
    • a missing container cgroup;
    • an agent that does not answer.
  • Agent: it runs outside the container, so its reads are never charged to the server. AGENT_CORES pins it.
  • versions.txt records the binary sha256, the version and git hash, the base image digest, the cpuset and the cores' sibling list, the CPU model, the cgroup and the kernel.

Interfaces

+pub mod clickhouse_server {
+    pub struct ClickHouseServer { pub url: String, pub cgroup_agent_url: String, pub max_threads: u32, pub statement_timeout: Duration }
+    pub struct ClickHouseEngine;  // impl EngineUnderTest
+    impl ClickHouseEngine { pub fn new(server: ClickHouseServer) -> Result<ClickHouseEngine, EngineError> }
+    pub fn series_identity(labels: &SeriesLabels) -> String
+}
 pub mod clickhouse {
+    pub const SERIES_IDENTITY: &str;  // was private
     pub struct ClickHouseDeployment {
         pub tables: Vec<String>,
+        pub sources: Vec<SourceTable>,
+        pub stored_tables: Vec<String>,
         ...
     }
+    pub struct SourceTable { pub metric: String, pub table: String, pub label_columns: Vec<String> }
 }
-pub const LANES: [&str; 1] = ["fake"];
+pub const LANES: [&str; 2] = ["fake", "clickhouse"];
+approxbench bench-external run --lane clickhouse --clickhouse-url URL --cgroup-agent-url URL [--clickhouse-max-threads N] [--clickhouse-statement-timeout-s S]
+scripts/extbench-up.sh clickhouse up|down <data-dir> <cores> <numa> <http-port> <agent-port> [<name>]; check-listen-host <address>

Design scope

New item Scope Why it is general
aqpbm-engines-external/src/clickhouse_server.rs engine adapter: a ClickHouse server under test any plan the translator accepts, any workload; no workload or host names
SourceTable, stored_tables translator output what any adapter needs to load data and size views
scripts/extbench-up.sh measurement setup for a server in its own cgroup one engine now; its checks (private address, pinned binary, cgroup, agent) apply to the next engine

Not in this PR

  • A per-query CPU counter from the server: it would need a system.query_log read after the arm, and the harness has no after-arm hook yet.
  • Sketch rows: the translator builds no sketch states yet.
  • The first comparison run (showcase-1m, ClickHouse vs ASAP).

Verification

  • cargo test -p aqpbm-engines-external -p aqpbm-cli: all pass. cargo clippy -p aqpbm-engines-external -p aqpbm-cli --all-targets -- -D warnings: clean. cargo fmt --all --check: clean.
  • Conformance: 13 of 13 on ClickHouse 26.8.39.13, as above.

@zzylol zzylol added the enhancement New feature or request label Nov 17, 2026
@zzylol
zzylol requested a review from GordonYuanyc November 17, 2026 16:44
@zzylol
zzylol force-pushed the bench-external/run-slice branch from af7fa57 to 671d399 Compare November 17, 2026 16:46
@zzylol
zzylol force-pushed the engines-external/clickhouse-adapter branch 3 times, most recently from ef51665 to d483622 Compare November 17, 2026 18:37
- The adapter against an HTTP stub: install creates a database per install
  and the translated tables in it; query ids map by index and text (any
  language); stored steps must divide the evaluation interval and the data
  start; ingest sends one insert per source table in chunk order, with the
  series identity as sorted JSON and TSV escaping; evaluate parses series and
  values; exception headers, error statuses and a silent server are errors;
  footprint, description, CPU accounting, leftover databases, and SYNC drops.
- The clickhouse lane needs its server and cgroup agent.
- scripts/extbench-up.sh parses and refuses public listen addresses.
- Conformance against a real server (ignored without one): one case per
  capability README row with hand-computed answers, the harness conformance
  kit, a suite-sized workload, and the lane against the fake lane.
- Planner-written probe plans for the harness kit.
- ClickHouseEngine (EngineUnderTest over HTTP): each install translates its
  plan file into a fresh database; one insert per source table per chunk;
  every query root at t with param_t, max_threads and wait_end_of_query;
  answers keyed by the workload query ids, series by the decoded series
  identity or by the label columns; footprint from system.parts; CPU from
  the server's cgroup through the agent (the server reports no per-query
  CPU in its summary header, so the engine counter is None).
- The translator also returns each metric's source table and label columns
  and the stored tables.
- approxbench bench-external run --lane clickhouse --clickhouse-url URL
  --cgroup-agent-url URL [--clickhouse-max-threads N].
- scripts/extbench-up.sh starts the server in a pinned container from a
  checked binary, with settings as command-line overrides (without a main
  config file the server reads no config.d), on a private address only,
  with the cgroup agent beside it and versions.txt.
… adapter

- Ingest makes one pass over the chunk, with each source's insert statement
  built once at install; a chunk with a sample at or before 0 inserts
  nothing; label names are quoted as the translator quotes them; TSV
  escapes \r and reads \r, \b and \f back.
- Install numbers its database from one counter for the whole process, so
  two engines in one process never share a database name.
- The fake lane refuses the clickhouse flags.
- extbench-up.sh: kills only its own agent, checks the agent answers, pins
  the agent with AGENT_CORES, resolves relative paths, gives the server's
  other protocols fixed ports on the private address (a port of 0 binds a
  random one), and never wipes a data dir it did not make (tested).
- A failed install drops its database (tested); the README maps each
  capability row to its conformance case.
… is a bare selector

PromQL's output labels: a function or aggregation drops the metric name,
and only a bare selector keeps it. The report joins lanes by exact label
maps, so the ClickHouse lane must key answers that way.
The ClickHouse lane keys answers by PromQL's output labels: a root that
applies a function or aggregation drops the metric name, and a root that is
only an instant selector keeps it.
@zzylol
zzylol force-pushed the bench-external/run-slice branch from 31a0040 to 7c0d121 Compare November 17, 2026 20:13
@zzylol
zzylol force-pushed the engines-external/clickhouse-adapter branch from d483622 to c9e5042 Compare November 17, 2026 20:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant