Skip to content

engines-external: translate plan files to VictoriaMetrics stream aggregation and MetricsQL - #264

Draft
zzylol wants to merge 4 commits into
bench-external/run-slicefrom
engines-external/victoriametrics-translator
Draft

zzylol wants to merge 4 commits into
bench-external/run-slicefrom
engines-external/victoriametrics-translator

Conversation

@zzylol

@zzylol zzylol commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Problem: the VictoriaMetrics lane needs ASAPPlanner plan files turned into what VictoriaMetrics runs: stream aggregation for stored windows and MetricsQL for queries. Until now only ClickHouse had a translator.
  • Change: victoriametrics::translate(&PlanFile) produces a -streamAggr.config, a vmalert recording-rule form of the same outputs, and one MetricsQL query per root. A node it has no rule for is refused (UnsupportedShape); answer changes are recorded as substitutions.
  • Effect: max_over_time(latency{job="j3"}[5m]) every 60 s (planned with victoriametrics.with-substitutes.json) becomes one per-series stream aggregation match: 'latency{job="j3"}', interval: 60s, outputs: [max] (no by) and the query max_over_time(latency:60s_max[300s] offset -29500ms). Of the full public sweep (206 PromQL workloads × 2 VictoriaMetrics files = 332 plans), 326 translate; the 6 refusals are with-substitutes plans with 1 s windows, which VictoriaMetrics cannot store separably; no equivalent-only plan has a substitution.

Review guide

  • Stack: base bench-external/run-slice (harness: bench-external lanes over one run directory, report, and cgroup v2 server CPU and memory #261, which contains engines-external: read an ASAPPlanner plan file and translate it to ClickHouse views #260; the harness now lives in aqpbm_core::harness, which this PR does not use). The adapter, feeder and conformance PR stacks on this one.
  • Read first: aqpbm-engines-external/src/victoriametrics.rs (module docs, translate, Translator::stored_read, add_stream_output, refuse_unsupported_fields), then capabilities/README.md § VictoriaMetrics (rows and the run contract).
  • Careful review: the stored-read shift h = (w − 1 s)/2, the sum_samples and count_samples pairing, the refusal rules.
  • Mechanical: SharingLost::of_plan moved from clickhouse.rs to lib.rs (no behavior change); planner-written fixtures under tests/fixtures/plans/public/*.victoriametrics.* and four workloads; the snapshot tests/golden/victoriametrics/public_plans.txt.

Behavior: before → after

Workload (public, grid/sot__1h__1hz__60s): sum by (job) (sum_over_time(latency[1h])) every 60 s, planned with victoriametrics.equivalent-only.json. The planner picks a per-series 60 s Sum window chain read over 1 h.

  • Before: no VictoriaMetrics translation; the lane had nothing to install.
  • After:
    - match: 'latency'
      interval: 60s
      outputs: [sum_samples, count_samples]
      enable_windows: true
      ignore_first_intervals: 0
      flush_on_shutdown: false
    sum by (job) (sum_over_time(latency:60s_sum_samples[3600s] offset -29500ms) and count_over_time(latency:60s_count_samples[3600s] offset -29500ms))
    
    time_step_ms = 60000 (evaluate at window ends), per_series_of = None.

Why the read looks like this (measured on a v1.154.1 server and checked in the source at 64c2e75fe7dc):

  • single-node VictoriaMetrics stores a stream output at its flush time in whole seconds (app/vminsert/common/streamaggr.go pushAggregateSeries), not at the window end: the window (T0, T0+10 s] was stored at T0 + 9 s. Reading [W] offset -h with h = (w − 1 s)/2 counts each window ending in (t − W, t] exactly once while the flush lag is below h;
  • sum_samples also writes 0 for an empty window while its series is not yet stale, where the planner has no row; count_samples skips empty windows, so the Sum read keeps only series count_samples has in the range.

Serves the goal-4 comparison (ASAPPlanner plans on VictoriaMetrics stream aggregation, same plan on every data plane).

Interfaces

src/victoriametrics.rs

+/// What a plan becomes on VictoriaMetrics.
+pub struct VictoriaMetricsDeployment {
+    /// One stream aggregation per (selector, interval, grouping), in order of first read.
+    pub stream_aggregations: Vec<StreamAggregation>,
+    /// One query per query root, in root order.
+    pub queries: Vec<MetricsQlQuery>,
+    pub substitutions: Vec<Substitution>,
+    pub sharing_lost: Vec<SharingLost>,
+}
+/// One entry of the stream-aggregation config, e.g. `latency{job="j3"}` every 60 s, per series, output `max`.
+pub struct StreamAggregation {
+    pub metric: String,
+    pub selector: String,
+    pub interval_ms: u64,
+    /// `None`: per series (no `by`, no `without`). `Some(labels)`: `by: [labels]`, sorted, never empty.
+    pub by: Option<Vec<String>>,
+    pub outputs: Vec<StreamOutput>,
+}
+/// A stream-aggregation output this translator uses.
+pub enum StreamOutput { SumSamples, CountSamples, Min, Max }
+/// One MetricsQL instant query, sent with `nocache=1`.
+pub struct MetricsQlQuery {
+    pub expr: String,
+    /// The evaluation time must be a multiple of this (the widths of the stream outputs read; 1 when none).
+    pub time_step_ms: u64,
+    /// `Some(metric)` when each answer series is one input series of `metric`.
+    pub per_series_of: Option<String>,
+}
+impl StreamOutput { pub fn name(self) -> &'static str; }
+impl StreamAggregation {
+    /// `<metric>:<interval>[_by_<labels>]_<output>`, as VictoriaMetrics names it.
+    pub fn output_metric(&self, output: StreamOutput) -> String;
+}
+impl VictoriaMetricsDeployment {
+    /// The `-streamAggr.config` YAML; `[]` when there is none.
+    pub fn stream_aggregation_config(&self) -> String;
+    /// The same outputs as vmalert recording rules (one group per interval, `eval_alignment: true`, `eval_delay: 0s`): the cross-check form, which needs no real time.
+    pub fn recording_rules(&self) -> String;
+}
+/// Translates `plan`; the first node with no rule is refused.
+pub fn translate(plan: &PlanFile) -> Result<VictoriaMetricsDeployment, TranslationError>;

src/lib.rs

+pub mod victoriametrics;
 impl SharingLost {
+    /// The query-time nodes of `plan` that two or more readers share (moved from `clickhouse.rs`).
+    pub fn of_plan(plan: &PlanFile) -> Vec<SharingLost>;
 }

capabilities/victoriametrics.with-substitutes.json: supported_summaries is now [] (each sketch joins with its rule, as in the ClickHouse files), state_not_mergeable_across_windows is [Rate, Increase] (no stored counter rule yet), and IRate is excluded in both forms (the same two entries as ClickHouse).

Usage. asap_planner --deployment-capabilities capabilities/victoriametrics.equivalent-only.json --plan-file p.json on the workload above, then victoriametrics::translate(&PlanFile::from_json(..)?): the adapter (next PR) writes stream_aggregation_config() as the server's -streamAggr.config, reloads it, feeds wall-clock data, and at each window end t runs each queries[i].expr at t with nocache=1; the answer series are keyed per per_series_of.

Algorithm

  • Rules (README § VictoriaMetrics): stored exact Sum/Count/Min/Max chains → stream outputs (per series: no by/without; grouped: by; grouped by no label: the per-series form plus the outer aggregate); query-time SummaryAgg/Aggregate over a raw range → *_over_time, quantile_over_time, rate/increase; aggregates over other patterns → sum/count/min/max/avg/quantile/topk by (…); instant selector → last_over_time(sel[lookback]); Project keeps labels by name; arithmetic one-to-one BinaryOp.
  • Refused: stored Rate/Increase; widths not whole seconds, below 2 s, or not dividing 86,400 s (stream aggregation aligns to Go's zero time); ranges that are not whole windows; filters, HAVING, without, sketches, other predicates, vector matching, bool, checked relative division, label renames, label names MetricsQL cannot write, a grouped aggregate straight over raw samples, two chains writing one output name.
  • Substitutions: query-time rate/increase (no extrapolation; VictoriaMetrics' prevValue and partial-reset rules); every division (VictoriaMetrics drops NaN series; a checked division answers as VictoriaMetrics divides).
  • Complexity: linear in nodes × the depth of each root's pattern (each root is translated by recursion; shared query-time nodes are inlined per reader and reported as SharingLost).
  • Limits: the capability file cannot say "no windows under 2 s", so such plans are refused at translation (6 of 332 in the sweep).

Design scope

  • src/victoriametrics.rs: the VictoriaMetrics translator, sibling of clickhouse.rs (engine translation scope). VictoriaMetricsDeployment, StreamAggregation, StreamOutput, MetricsQlQuery name what the engine installs and runs; none is tied to one workload.
  • SharingLost::of_plan (lib.rs): plan-level, shared by both translators.

Decisions and open questions

  • Needs reviewer decision: None.
  • Decided (2026-11-17):
    • Per-series stored chains omit by and without, so aggregateOnlyByTime keeps every input label (one output per input series); Reduce([]) = per-series form + outer aggregate (rejected: by: [__name__], which would store one series but writes the same output name as the per-series form).
    • Read shift h = (w − 1 s)/2 (rejected: no shift, which breaks once the flush lags; w/2, which fails for 2 s windows).
    • sum_samples read with and count_over_time(count_samples) (rejected: accepting the empty-window 0).
    • Stored Rate/Increase not offered: the increase output adds the full new value on any drop, unlike MetricsQL increase (prev/8 partial resets).
    • Citations at the v1.154.1 tag commit 64c2e75fe7dc.

Verification

Scope Result Notes
cargo test -p aqpbm-engines-external (build host) all passed; ClickHouse-binary tests ignored as before unchanged ClickHouse tests green after the SharingLost move
tests/victoriametrics_translate.rs (new, 23 tests) passed hand-written goldens (stored Sum/Max/Count, query-time shapes), per-series without by, by, Reduce([]), refusals on their nodes, widths, recording rules, output merging and name collisions, escaping and quoting, rate/division substitutions, mixed-workload shapes, a reviewed snapshot of all 25 public VictoriaMetrics plans
tests/public_plans.rs passed 29 ClickHouse + 25 VictoriaMetrics planner-written plans; mixed-08/09 with-substitutes listed as expected 1 s-window refusals
tests/capability_files.rs passed new with-substitutes literal
full public sweep (332 VictoriaMetrics plans, local) 326 translate, 6 refused (1 s windows, all with-substitutes), 0 equivalent-only substitutions
cargo fmt --check, cargo clippy -p aqpbm-engines-external --all-targets clean

Real-server behavior used above (stored output time, empty-window zero, negative offset, label keeping of rollups) was measured on a v1.154.1 server; the conformance cases come with the adapter PR.

Follow-ups

  • Adapter, paced wall-clock feeder and conformance on a real server (next PR).
  • vmalert -replay cross-check using recording_rules(); histogram_bucket for KLL/DDSketch; query-time exact substitutes for HLL/KMV/top-k; stored rate/increase.

Goldens written by hand before the translator; planner-written VictoriaMetrics
plans of public synthetic workloads (showcase shapes plus a cover of every node
shape of the full sweep); the new with-substitutes capability literal.
…egation and MetricsQL

Stored exact Sum/Count/Min/Max chains become stream-aggregation outputs (per
series with no by/without, grouped with by, grouped by no label as the
per-series form plus the outer aggregate), read with a half-window shift
because single-node VictoriaMetrics stores an output at its flush time;
query-time rollups, aggregates, instant selectors, projections and arithmetic
become MetricsQL. Query-time rate/increase and checked divisions are recorded
substitutions; widths that are not whole seconds, below 2 s or not dividing a
day are refused. The with-substitutes capability file drops sketches and
stored counter chains until their rules exist; the README cites v1.154.1.
… translator

Output-name collisions are checked on the final names (by [a_b] vs by [a, b]);
by labels are YAML-quoted; label values escape tab and carriage return;
matchers are sorted so equal selectors compare equal; every division records
that VictoriaMetrics drops NaN series (and the checked-division difference);
the README adds the no-result-cache contract (instant rollups over 3 h use
cached partials otherwise) and the stale-marker note; a reviewed snapshot of
every public plan's config and queries guards the translations.
@zzylol
zzylol force-pushed the engines-external/victoriametrics-translator branch from 6938a09 to 8bd0e8c Compare November 17, 2026 20:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant