Repository navigation
engines-external, harness, cli: run translated plans on a real VictoriaMetrics server with a paced wall-clock feeder - #265
Draft
zzylol wants to merge 10 commits into
Conversation
… adapter, the agent's server config and the victoriametrics lane Unit tests for the wall offset, the paced slices and lead, the import body, the server metrics and answers, and the install path against a fake server; the agent's PUT/GET of one server config file; the lane's URL and one-run checks; and #[ignore] conformance cases per README row against a real v1.154.1 server.
…iaMetrics server with a paced wall-clock feeder VictoriaMetricsEngine installs one arm at a time: the translated stream-aggregation config goes to the server through the cgroup agent (PUT /server-config) and is reloaded; data time maps to wall time by an offset that keeps window ends; each chunk is sent in 1 s slices a lead early (max(1 s, the narrowest read shift)); the ingest of a chunk ending at a window end waits for the flush (failing on lag or late samples) and force flushes, so evaluations run only the queries (nocache=1). The agent can hold one server config file; the lane victoriametrics needs its URLs and --warmup 0 --runs 1; scripts/extbench-up-victoriametrics.sh starts the pinned v1.154.1 image with the stream-aggregation flags and the agent.
…w fixes A query-time arm ingests then evaluates; each install's config names its aggregations, and a dropped arm unloads its config; server metrics are read per aggregation name; the named config of the translator.
…iaMetrics adapter Every ingest waits for its chunk end, also without stream aggregation; each install names its aggregations (reloads always apply, a refused config is an error, flush and too-old counts are per install); every tick inside a chunk must flush, within the read shift; a slice must reach the server 100 ms before its window ends; an arm's data starts 10 minutes plus its longest range after the previous arm's newest sample, beyond MetricsQL's look-back for the sample before a range; label values are parsed by name. The agent creates its temporary file afresh (never following a link) and answers 500 with the cause when it cannot write. Conformance data carries a per-run label so reruns never share series.
…t a whole number of them)
…r the previous arm's samples to age out
…arm's samples to age out A window rollup reads only its window, which holds only the arm's own data, so the 10-minute-plus-range wait applies only to plans with rate, increase or irate states. A suite of query-time arms no longer idles 11 minutes per cell.
…e selector, as PromQL and the other lanes key them
…re selector PromQL drops the metric name in every function and aggregation, and the other lanes key their answers so; the report joins answers by these keys.
…aqpbm_core::harness
zzylol
force-pushed
the
engines-external/victoriametrics-translator
branch
from
November 17, 2026 20:19
6938a09 to
8bd0e8c
Compare
zzylol
force-pushed
the
engines-external/victoriametrics-adapter
branch
from
November 17, 2026 20:19
e2580f6 to
5c7a511
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
VictoriaMetricsEngine(anEngineUnderTest) installs a translated plan on a single-node v1.154.1 server, feeds data in real time with wall-clock timestamps, waits for the stream flushes, and answers each query over HTTP. The cgroup agent can hold the server's-streamAggr.config;approxbench bench-external run --lane victoriametrics;scripts/extbench-up-victoriametrics.shstarts the pinned image.max_over_time(latency[5m])every 10 s over 1,000 series answers 7 evaluations × 1,000 series with 0 error against VictoriaMetrics' raw evaluation of the same text.Review guide
engines-external/victoriametrics-translator(engines-external: translate plan files to VictoriaMetrics stream aggregation and MetricsQL #264) → harness: bench-external lanes over one run directory, report, and cgroup v2 server CPU and memory #261 → engines-external: read an ASAPPlanner plan file and translate it to ClickHouse views #260. The harness isaqpbm_core::harness(moved from theaqpbm-harnesscrate); this PR's agent change is inaqpbm-core/src/harness/cgroup.rs.aqpbm-engines-external/src/victoriametrics_server.rsmodule docs, theninstall,ingest,wait_for_flushes,load_config,wall_offset_ms,lead_ms.runFlusher/Push; the agent'sPUT /server-config(aqpbm-core/src/harness/cgroup.rs).stream_aggregation_config_named.Behavior: before → after
User runs, from the driver host, with the server started by the script on the server host:
over a run directory whose plan is
max_over_time(latency[5m])every 10 s, 1,000 series at 1 Hz, planned withvictoriametrics.equivalent-only.json(a per-series 10 s Max window chain).unknown lane 'victoriametrics'.maxf5m victoriametrics.equivalent-only Ran. The cell installs one per-series stream aggregation (latency:10s_max, noby) and reads it withmax_over_time(latency:10s_max[300s] offset -4500ms). It ingests 360,000 samples over 6 minutes of wall time and evaluates 7 times. All 7,000 answer points equal VictoriaMetrics' raw evaluation ofmax_over_time(latency[300s])at the same wall times (max relative error 0.0). Absolute numbers from the server's cgroup (one physical core, 2 hyperthreads):/metricsreads)memory.current)vm_data_size_bytes)approxbench bench-external reportjoins it (q 36.220 ms, i 18594.3 ns/sample [cgroup]).Interfaces
src/victoriametrics_server.rssrc/victoriametrics.rsimpl VictoriaMetricsDeployment { pub fn stream_aggregation_config(&self) -> String; + /// The same config with every entry named `name` (one name per install). + pub fn stream_aggregation_config_named(&self, name: &str) -> String; pub fn recording_rules(&self) -> String; }aqpbm-core/src/harness/cgroup.rsCLI:
approxbench bench-external run --lane victoriametrics --victoriametrics-url U --cgroup-agent-url A(refused unless--warmup 0 --runs 1);approxbench cgroup-agent --server-config <path>.Usage: the server host runs
LISTEN_HOST=<private address> APPROXBENCH=… scripts/extbench-up-victoriametrics.sh up <data-dir> <cores> <numa> 18428 19101; the driver runs the lane over a run directory. Per cell,installtranslates the plan, PUTs the named config to the agent, reloads, waits until the server runs the named aggregators, computes the wall offset and sleeps to the data start; eachingestpaces 1 s slices to/api/v1/import, waits to the chunk end and for every tick's flush, and force-flushes; eachevaluateruns the MetricsQL queries att + offsetwithnocache=1.Algorithm
lead = max(1 s, h_min)early (h = (w − 1 s)/2): every sample lands in its own window (lead < w), and samples do not lag the wall clock, so the server does not delay flushes (flushAfterMsec). A slice reaching the server < 100 ms before its window end fails the step.(previous chunk end, chunk end](counted from the data start for the first chunk), withinhof the chunk end; the server'stoo_oldcounter for this install's aggregations must not grow. Stored reads use[W] offset -h, which is right while flushes lag less thanh.__name__only for a bare selector, as PromQL and the other lanes key them./metricsreads.--warmup 0 --runs 1); the server is shared across cells, so the footprint is the data growth since install.Design scope
src/victoriametrics_server.rs: the VictoriaMetrics engine adapter (engine-adapter scope, sibling of the ClickHouse adapter).VictoriaMetricsServer/VictoriaMetricsEnginemirror the ClickHouse names;ServerMetrics,Sliceare what the adapter reads and sends.cgroup.rs): the agent is the one process on the server host the harness talks to; holding the server's one config file there keeps the server unexposed. Only the configured file can be written.scripts/extbench-up-victoriametrics.sh: a separate file so it does not collide with the ClickHouse adapter'sextbench-up.sh; they merge into one script when both land.Decisions and open questions
max(1 s, h)with the flush wait at the end of ingest (flush CPU is ingest CPU; an evaluation is only queries); config through the agent (rejected: a config URL served by the driver, which the server needs at start);nocache=1and-search.disableCache(VictoriaMetrics' instant-rollup cache otherwise answers windows ≥ 3 h from cached partials).Verification
cargo test -p aqpbm-engines-external(build host)tests/victoriametrics_server.rs(new, 9)/metricsparsing per aggregation name, answers keyed like the harness, one arm at a time with a named config and unload on drop, plan checks before anything loads, a query-time arm ingests then evaluates (fake server)tests/victoriametrics_translate.rstests/victoriametrics_conformance.rs(new,#[ignore]) on a real v1.154.1 server (pinned digest), run from a separate driver hostby,by: [job],Reduce([])Sum and Max, two windows 10 s and 20 s in one install, samples on window ends, a series whose later windows are empty); query-time rows: 3 × 12 exact (sum by of sum_over_time, quantile_over_time, avg by of quantiles, count, max, avg_over_time, instant selector, topk, 0/0 dropped, rate and increase by VictoriaMetrics' formula with a full reset and a partial drop < prev/8); latency offset 0; a late chunk fails with a pacing erroraqpbm-coreharness tests (tests/harness_cgroup.rs)PUT/GET /server-config: atomic replace, 413 over 1 MiB, read-only files stay unwritable, 404 without a config pathaqpbm-cli --test bench_external_clivictoriametricsneeds its URLs and--warmup 0 --runs 1Ran, 0 error on 7,000 pointscargo fmt --check,cargo clippy --all-targets(3 crates)Found on the way and fixed here: an earlier conformance run failed because VictoriaMetrics'
increaseread the previous run's last sample of the same series (it looks back 5 minutes plus the larger of the range and the 5-minute step); arms now start 10 minutes plus their longest range after the previous arm's newest sample, and the conformance data carries a per-run label.Follow-ups
vmalert -replaycross-check of the stream outputs againstrecording_rules().histogram_bucketfor KLL / DDSketch, exact query-time substitutes for HLL / KMV / top-k, stored rate / increase./metricspolling and VictoriaMetrics' background work land in ingest CPU); a driver/server clock-skew check (both hosts NTP-synchronized here); restarting the server per cell; hand-written VictoriaMetrics views for the showcase suite.query_ids_of, operator names,plan_summary) move to the crate root when both adapters land.