Repository navigation
Conversation
zzylol
force-pushed
the
bench-external/run-slice
branch
from
November 17, 2026 20:13
31a0040 to
7c0d121
Compare
Goldens written by hand before the translator; planner-written VictoriaMetrics plans of public synthetic workloads (showcase shapes plus a cover of every node shape of the full sweep); the new with-substitutes capability literal.
…egation and MetricsQL Stored exact Sum/Count/Min/Max chains become stream-aggregation outputs (per series with no by/without, grouped with by, grouped by no label as the per-series form plus the outer aggregate), read with a half-window shift because single-node VictoriaMetrics stores an output at its flush time; query-time rollups, aggregates, instant selectors, projections and arithmetic become MetricsQL. Query-time rate/increase and checked divisions are recorded substitutions; widths that are not whole seconds, below 2 s or not dividing a day are refused. The with-substitutes capability file drops sketches and stored counter chains until their rules exist; the README cites v1.154.1.
… translator Output-name collisions are checked on the final names (by [a_b] vs by [a, b]); by labels are YAML-quoted; label values escape tab and carriage return; matchers are sorted so equal selectors compare equal; every division records that VictoriaMetrics drops NaN series (and the checked-division difference); the README adds the no-result-cache contract (instant rollups over 3 h use cached partials otherwise) and the stale-marker note; a reviewed snapshot of every public plan's config and queries guards the translations.
zzylol
force-pushed
the
engines-external/victoriametrics-translator
branch
from
November 17, 2026 20:19
6938a09 to
8bd0e8c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
victoriametrics::translate(&PlanFile)produces a-streamAggr.config, a vmalert recording-rule form of the same outputs, and one MetricsQL query per root. A node it has no rule for is refused (UnsupportedShape); answer changes are recorded as substitutions.max_over_time(latency{job="j3"}[5m])every 60 s (planned withvictoriametrics.with-substitutes.json) becomes one per-series stream aggregationmatch: 'latency{job="j3"}',interval: 60s,outputs: [max](noby) and the querymax_over_time(latency:60s_max[300s] offset -29500ms). Of the full public sweep (206 PromQL workloads × 2 VictoriaMetrics files = 332 plans), 326 translate; the 6 refusals are with-substitutes plans with 1 s windows, which VictoriaMetrics cannot store separably; no equivalent-only plan has a substitution.Review guide
bench-external/run-slice(harness: bench-external lanes over one run directory, report, and cgroup v2 server CPU and memory #261, which contains engines-external: read an ASAPPlanner plan file and translate it to ClickHouse views #260; the harness now lives inaqpbm_core::harness, which this PR does not use). The adapter, feeder and conformance PR stacks on this one.aqpbm-engines-external/src/victoriametrics.rs(module docs,translate,Translator::stored_read,add_stream_output,refuse_unsupported_fields), thencapabilities/README.md§ VictoriaMetrics (rows and the run contract).h = (w − 1 s)/2, thesum_samples and count_samplespairing, the refusal rules.SharingLost::of_planmoved fromclickhouse.rstolib.rs(no behavior change); planner-written fixtures undertests/fixtures/plans/public/*.victoriametrics.*and four workloads; the snapshottests/golden/victoriametrics/public_plans.txt.Behavior: before → after
Workload (public,
grid/sot__1h__1hz__60s):sum by (job) (sum_over_time(latency[1h]))every 60 s, planned withvictoriametrics.equivalent-only.json. The planner picks a per-series 60 s Sum window chain read over 1 h.time_step_ms = 60000(evaluate at window ends),per_series_of = None.Why the read looks like this (measured on a v1.154.1 server and checked in the source at
64c2e75fe7dc):app/vminsert/common/streamaggr.gopushAggregateSeries), not at the window end: the window(T0, T0+10 s]was stored atT0 + 9 s. Reading[W] offset -hwithh = (w − 1 s)/2counts each window ending in(t − W, t]exactly once while the flush lag is belowh;sum_samplesalso writes 0 for an empty window while its series is not yet stale, where the planner has no row;count_samplesskips empty windows, so the Sum read keeps only seriescount_sampleshas in the range.Serves the goal-4 comparison (ASAPPlanner plans on VictoriaMetrics stream aggregation, same plan on every data plane).
Interfaces
src/victoriametrics.rssrc/lib.rscapabilities/victoriametrics.with-substitutes.json:supported_summariesis now[](each sketch joins with its rule, as in the ClickHouse files),state_not_mergeable_across_windowsis[Rate, Increase](no stored counter rule yet), and IRate is excluded in both forms (the same two entries as ClickHouse).Usage.
asap_planner --deployment-capabilities capabilities/victoriametrics.equivalent-only.json --plan-file p.jsonon the workload above, thenvictoriametrics::translate(&PlanFile::from_json(..)?): the adapter (next PR) writesstream_aggregation_config()as the server's-streamAggr.config, reloads it, feeds wall-clock data, and at each window endtruns eachqueries[i].exprattwithnocache=1; the answer series are keyed perper_series_of.Algorithm
by/without; grouped:by; grouped by no label: the per-series form plus the outer aggregate); query-timeSummaryAgg/Aggregateover a raw range →*_over_time,quantile_over_time,rate/increase; aggregates over other patterns →sum/count/min/max/avg/quantile/topk by (…); instant selector →last_over_time(sel[lookback]);Projectkeeps labels by name; arithmetic one-to-oneBinaryOp.without, sketches, other predicates, vector matching, bool, checked relative division, label renames, label names MetricsQL cannot write, a grouped aggregate straight over raw samples, two chains writing one output name.rate/increase(no extrapolation; VictoriaMetrics' prevValue and partial-reset rules); every division (VictoriaMetrics drops NaN series; a checked division answers as VictoriaMetrics divides).SharingLost).Design scope
src/victoriametrics.rs: the VictoriaMetrics translator, sibling ofclickhouse.rs(engine translation scope).VictoriaMetricsDeployment,StreamAggregation,StreamOutput,MetricsQlQueryname what the engine installs and runs; none is tied to one workload.SharingLost::of_plan(lib.rs): plan-level, shared by both translators.Decisions and open questions
byandwithout, soaggregateOnlyByTimekeeps every input label (one output per input series);Reduce([])= per-series form + outer aggregate (rejected:by: [__name__], which would store one series but writes the same output name as the per-series form).h = (w − 1 s)/2(rejected: no shift, which breaks once the flush lags;w/2, which fails for 2 s windows).sum_samplesread withand count_over_time(count_samples)(rejected: accepting the empty-window 0).increaseoutput adds the full new value on any drop, unlike MetricsQLincrease(prev/8 partial resets).64c2e75fe7dc.Verification
cargo test -p aqpbm-engines-external(build host)SharingLostmovetests/victoriametrics_translate.rs(new, 23 tests)by,by,Reduce([]), refusals on their nodes, widths, recording rules, output merging and name collisions, escaping and quoting, rate/division substitutions, mixed-workload shapes, a reviewed snapshot of all 25 public VictoriaMetrics planstests/public_plans.rstests/capability_files.rscargo fmt --check,cargo clippy -p aqpbm-engines-external --all-targetsReal-server behavior used above (stored output time, empty-window zero, negative offset, label keeping of rollups) was measured on a v1.154.1 server; the conformance cases come with the adapter PR.
Follow-ups
vmalert -replaycross-check usingrecording_rules();histogram_bucketfor KLL/DDSketch; query-time exact substitutes for HLL/KMV/top-k; stored rate/increase.