Skip to content

engines-external, harness, cli: run translated plans on a real VictoriaMetrics server with a paced wall-clock feeder - #265

Draft
zzylol wants to merge 10 commits into
engines-external/victoriametrics-translatorfrom
engines-external/victoriametrics-adapter
Draft

zzylol wants to merge 10 commits into
engines-external/victoriametrics-translatorfrom
engines-external/victoriametrics-adapter

Conversation

@zzylol

@zzylol zzylol commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Problem: translated plans (previous PR) had nothing to run them on a real VictoriaMetrics server. Stream aggregation runs on wall-clock time, so the harness's replayed data would be bucketed wrongly.
  • Change: VictoriaMetricsEngine (an EngineUnderTest) installs a translated plan on a single-node v1.154.1 server, feeds data in real time with wall-clock timestamps, waits for the stream flushes, and answers each query over HTTP. The cgroup agent can hold the server's -streamAggr.config; approxbench bench-external run --lane victoriametrics; scripts/extbench-up-victoriametrics.sh starts the pinned image.
  • Effect: on a real server, every conformance case passes (stored Sum/Count/Min/Max per series, grouped, grouped by no label and two windows in one install: 3 evaluations × 7 queries equal to the exact windows; 12 query-time queries incl. rate/increase by VictoriaMetrics' formula: 3 × 12 equal; a point is visible at its own time; a late chunk fails), and a lane run of max_over_time(latency[5m]) every 10 s over 1,000 series answers 7 evaluations × 1,000 series with 0 error against VictoriaMetrics' raw evaluation of the same text.

Review guide

Behavior: before → after

User runs, from the driver host, with the server started by the script on the server host:

approxbench bench-external run --run-dir R --lane victoriametrics \
  --victoriametrics-url http://<server>:18428 --cgroup-agent-url http://<server>:19101 --warmup 0 --runs 1

over a run directory whose plan is max_over_time(latency[5m]) every 10 s, 1,000 series at 1 Hz, planned with victoriametrics.equivalent-only.json (a per-series 10 s Max window chain).

  • Before: unknown lane 'victoriametrics'.
  • After: maxf5m victoriametrics.equivalent-only Ran. The cell installs one per-series stream aggregation (latency:10s_max, no by) and reads it with max_over_time(latency:10s_max[300s] offset -4500ms). It ingests 360,000 samples over 6 minutes of wall time and evaluates 7 times. All 7,000 answer points equal VictoriaMetrics' raw evaluation of max_over_time(latency[300s]) at the same wall times (max relative error 0.0). Absolute numbers from the server's cgroup (one physical core, 2 hyperthreads):
measure value
ingest CPU 18,594 ns per sample (6,694 ms in total; whole-server CPU over each 10 s step, including flushes, merges and the adapter's /metrics reads)
query CPU 36.2 ms per evaluation (20.5–72.4 ms)
whole-arm server CPU 7,292 ms
server memory before / after 82.9 / 97.4 MiB (memory.current)
data growth 3.27 MB (vm_data_size_bytes)

approxbench bench-external report joins it (q 36.220 ms, i 18594.3 ns/sample [cgroup]).

Interfaces

src/victoriametrics_server.rs

+/// How to reach the server.
+pub struct VictoriaMetricsServer {
+    /// The HTTP interface, e.g. `http://10.0.0.1:8428`.
+    pub url: String,
+    /// The cgroup agent beside the server, started with `--server-config` on the server's `-streamAggr.config`.
+    pub cgroup_agent_url: String,
+    /// How long one request may take, e.g. 600 s.
+    pub request_timeout: Duration,
+}
+/// A VictoriaMetrics server as an engine under test; one arm at a time.
+pub struct VictoriaMetricsEngine { /* private */ }
+impl VictoriaMetricsEngine {
+    /// Connects to the server and reads its version (`vm_app_version`).
+    pub fn new(server: VictoriaMetricsServer) -> Result<VictoriaMetricsEngine, EngineError>;
+}
+impl EngineUnderTest for VictoriaMetricsEngine { /* description, cpu_accounting = Cgroup(Agent), install */ }
+/// Data time → wall time: the smallest multiple of `step_ms` that puts the data start's step boundary ≥ 2 s after `now_ms`.
+pub fn wall_offset_ms(now_ms: i64, data_start_ms: i64, step_ms: u64) -> i64;
+/// How early a slice is sent: max(1 s, the narrowest window's read shift (w − 1 s)/2).
+pub fn lead_ms(aggregations: &[StreamAggregation]) -> i64;
+/// The feeder's slice length (1 s).
+pub const SLICE_MS: i64;
+/// One slice of a chunk and its send time.
+pub struct Slice<'a> { pub end_ms: i64, pub send_at_wall_ms: i64, pub samples: Vec<&'a Sample> }
+pub fn slices(chunk: &SampleChunk, offset_ms: i64, lead_ms: i64) -> Vec<Slice<'_>>;
+/// `/api/v1/import` JSON lines, one per series.
+pub fn import_body(samples: &[&Sample], offset_ms: i64) -> String;
+/// What the adapter reads from `/metrics`; stream counters of one aggregation name only.
+pub struct ServerMetrics {
+    pub app_version: Option<String>,
+    pub config_reloads: u64,
+    pub config_reload_errors: u64,
+    pub running_positions: BTreeSet<u32>,
+    pub flushes_by_position: BTreeMap<u32, u64>,
+    pub too_old_samples: u64,
+    pub data_size_bytes: u64,
+}
+impl ServerMetrics { pub fn parse(text: &str, aggregation_name: &str) -> ServerMetrics; }
+/// One query answer as harness series: labels without `__name__`, plus `__name__ = metric_name` when given.
+pub fn parse_answer(body: &str, metric_name: Option<&str>) -> Result<BTreeMap<SeriesLabels, f64>, String>;
+/// The `__name__` a root's answers keep: the metric for a bare selector (instant `TimeRange` root), none otherwise, as PromQL and the other lanes key answers.
+pub fn answer_metric_name(plan: &PlanFile, root: NodeId, query: &MetricsQlQuery) -> Option<String>;
+/// How long after the previous arm's newest sample this arm's data must start: 0 without counters; 10 min + the longest range for a plan with rate/increase/irate states.
+pub fn previous_arm_gap_ms(plan: &PlanFile) -> i64;

src/victoriametrics.rs

 impl VictoriaMetricsDeployment {
     pub fn stream_aggregation_config(&self) -> String;
+    /// The same config with every entry named `name` (one name per install).
+    pub fn stream_aggregation_config_named(&self, name: &str) -> String;
     pub fn recording_rules(&self) -> String;
 }

aqpbm-core/src/harness/cgroup.rs

+/// The name under which the agent serves, and replaces on `PUT`, its server config file.
+pub const SERVER_CONFIG: &str = "server-config";
+/// The largest server config the agent accepts (1 MiB).
+pub const MAX_SERVER_CONFIG_BYTES: usize;
+/// Replaces the server config the agent holds, e.g. a new `-streamAggr.config`.
+pub fn put_server_config(agent_url: &str, text: &str) -> Result<(), CgroupError>;
-pub fn serve_agent(listener: TcpListener, dir: &Path, proc_stat: &Path) -> std::io::Result<()>;
+/// With `server_config`, also `GET`/`PUT /server-config` (atomic replace; 404 otherwise).
+pub fn serve_agent(listener: TcpListener, dir: &Path, proc_stat: &Path, server_config: Option<&Path>) -> std::io::Result<()>;

CLI: approxbench bench-external run --lane victoriametrics --victoriametrics-url U --cgroup-agent-url A (refused unless --warmup 0 --runs 1); approxbench cgroup-agent --server-config <path>.

Usage: the server host runs LISTEN_HOST=<private address> APPROXBENCH=… scripts/extbench-up-victoriametrics.sh up <data-dir> <cores> <numa> 18428 19101; the driver runs the lane over a run directory. Per cell, install translates the plan, PUTs the named config to the agent, reloads, waits until the server runs the named aggregators, computes the wall offset and sleeps to the data start; each ingest paces 1 s slices to /api/v1/import, waits to the chunk end and for every tick's flush, and force-flushes; each evaluate runs the MetricsQL queries at t + offset with nocache=1.

Algorithm

  • Wall offset: a multiple of every window width and the evaluation interval, so window ends stay window ends; the data's step boundary lands on a tick after the config load (stream aggregation drops the window in progress at load and samples older than the current window).
  • Pacing: 1 s slices sent lead = max(1 s, h_min) early (h = (w − 1 s)/2): every sample lands in its own window (lead < w), and samples do not lag the wall clock, so the server does not delay flushes (flushAfterMsec). A slice reaching the server < 100 ms before its window end fails the step.
  • Flush wait (end of ingest): each aggregation must have flushed once per tick in (previous chunk end, chunk end] (counted from the data start for the first chunk), within h of the chunk end; the server's too_old counter for this install's aggregations must not grow. Stored reads use [W] offset -h, which is right while flushes lag less than h.
  • Isolation between arms: one arm at a time; each install names its aggregators (reloads always apply; counters are per install); an arm that reads counters starts its data ≥ 10 min + its longest range after the previous arm's newest sample (MetricsQL's look-back for the sample before a range); other arms start right away. Answers keep __name__ only for a bare selector, as PromQL and the other lanes key them.
  • Complexity: a run takes its data's duration in wall time; per chunk, one import per 1 s slice, a few /metrics reads.
  • Limits: one arm per engine (the lane needs --warmup 0 --runs 1); the server is shared across cells, so the footprint is the data growth since install.

Design scope

  • src/victoriametrics_server.rs: the VictoriaMetrics engine adapter (engine-adapter scope, sibling of the ClickHouse adapter). VictoriaMetricsServer/VictoriaMetricsEngine mirror the ClickHouse names; ServerMetrics, Slice are what the adapter reads and sends.
  • Agent server config (cgroup.rs): the agent is the one process on the server host the harness talks to; holding the server's one config file there keeps the server unexposed. Only the configured file can be written.
  • scripts/extbench-up-victoriametrics.sh: a separate file so it does not collide with the ClickHouse adapter's extbench-up.sh; they merge into one script when both land.

Decisions and open questions

  • Needs reviewer decision: None.
  • Decided (2026-11-17): one arm at a time (rejected: one series namespace per install via an injected label, which changes the queries); lead max(1 s, h) with the flush wait at the end of ingest (flush CPU is ingest CPU; an evaluation is only queries); config through the agent (rejected: a config URL served by the driver, which the server needs at start); nocache=1 and -search.disableCache (VictoriaMetrics' instant-rollup cache otherwise answers windows ≥ 3 h from cached partials).

Verification

Scope Result Notes
cargo test -p aqpbm-engines-external (build host) all passed; ClickHouse-binary and real-server tests ignored
tests/victoriametrics_server.rs (new, 9) passed wall offset (incl. an unaligned data start), lead below every width, 1 s slices sent a lead early, import body, /metrics parsing per aggregation name, answers keyed like the harness, one arm at a time with a named config and unload on drop, plan checks before anything loads, a query-time arm ingests then evaluates (fake server)
tests/victoriametrics_translate.rs passed + named config, + 7-minute window refused
tests/victoriametrics_conformance.rs (new, #[ignore]) on a real v1.154.1 server (pinned digest), run from a separate driver host 4 passed stored windows: 3 evaluations × 7 queries exact (Sum/Count/Min/Max per series without by, by: [job], Reduce([]) Sum and Max, two windows 10 s and 20 s in one install, samples on window ends, a series whose later windows are empty); query-time rows: 3 × 12 exact (sum by of sum_over_time, quantile_over_time, avg by of quantiles, count, max, avg_over_time, instant selector, topk, 0/0 dropped, rate and increase by VictoriaMetrics' formula with a full reset and a partial drop < prev/8); latency offset 0; a late chunk fails with a pacing error
aqpbm-core harness tests (tests/harness_cgroup.rs) passed + agent PUT/GET /server-config: atomic replace, 413 over 1 MiB, read-only files stay unwritable, 404 without a config path
aqpbm-cli --test bench_external_cli passed + lane victoriametrics needs its URLs and --warmup 0 --runs 1
lane run (CLI, real server) 1 cell Ran, 0 error on 7,000 points see Behavior
cargo fmt --check, cargo clippy --all-targets (3 crates) clean

Found on the way and fixed here: an earlier conformance run failed because VictoriaMetrics' increase read the previous run's last sample of the same series (it looks back 5 minutes plus the larger of the range and the 5-minute step); arms now start 10 minutes plus their longest range after the previous arm's newest sample, and the conformance data carries a per-run label.

Follow-ups

  • vmalert -replay cross-check of the stream outputs against recording_rules().
  • histogram_bucket for KLL / DDSketch, exact query-time substitutes for HLL / KMV / top-k, stored rate / increase.
  • An idle-server CPU baseline (the adapter's /metrics polling and VictoriaMetrics' background work land in ingest CPU); a driver/server clock-skew check (both hosts NTP-synchronized here); restarting the server per cell; hand-written VictoriaMetrics views for the showcase suite.
  • Shared helpers with the ClickHouse adapter (query_ids_of, operator names, plan_summary) move to the crate root when both adapters land.

@zzylol zzylol added the enhancement New feature or request label Nov 17, 2026
@zzylol
zzylol requested a review from GordonYuanyc November 17, 2026 17:52
… adapter, the agent's server config and the victoriametrics lane

Unit tests for the wall offset, the paced slices and lead, the import body,
the server metrics and answers, and the install path against a fake server;
the agent's PUT/GET of one server config file; the lane's URL and one-run
checks; and #[ignore] conformance cases per README row against a real
v1.154.1 server.
…iaMetrics server with a paced wall-clock feeder

VictoriaMetricsEngine installs one arm at a time: the translated
stream-aggregation config goes to the server through the cgroup agent
(PUT /server-config) and is reloaded; data time maps to wall time by an
offset that keeps window ends; each chunk is sent in 1 s slices a lead early
(max(1 s, the narrowest read shift)); the ingest of a chunk ending at a
window end waits for the flush (failing on lag or late samples) and force
flushes, so evaluations run only the queries (nocache=1). The agent can hold
one server config file; the lane victoriametrics needs its URLs and
--warmup 0 --runs 1; scripts/extbench-up-victoriametrics.sh starts the
pinned v1.154.1 image with the stream-aggregation flags and the agent.
…w fixes

A query-time arm ingests then evaluates; each install's config names its
aggregations, and a dropped arm unloads its config; server metrics are read
per aggregation name; the named config of the translator.
…iaMetrics adapter

Every ingest waits for its chunk end, also without stream aggregation; each
install names its aggregations (reloads always apply, a refused config is an
error, flush and too-old counts are per install); every tick inside a chunk
must flush, within the read shift; a slice must reach the server 100 ms
before its window ends; an arm's data starts 10 minutes plus its longest
range after the previous arm's newest sample, beyond MetricsQL's look-back
for the sample before a range; label values are parsed by name. The agent
creates its temporary file afresh (never following a link) and answers 500
with the cause when it cannot write. Conformance data carries a per-run
label so reruns never share series.
…arm's samples to age out

A window rollup reads only its window, which holds only the arm's own data,
so the 10-minute-plus-range wait applies only to plans with rate, increase
or irate states. A suite of query-time arms no longer idles 11 minutes per
cell.
…e selector, as PromQL and the other lanes key them
…re selector

PromQL drops the metric name in every function and aggregation, and the
other lanes key their answers so; the report joins answers by these keys.
@zzylol
zzylol force-pushed the engines-external/victoriametrics-translator branch from 6938a09 to 8bd0e8c Compare November 17, 2026 20:19
@zzylol
zzylol force-pushed the engines-external/victoriametrics-adapter branch from e2580f6 to 5c7a511 Compare November 17, 2026 20:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant