Skip to content

phase 3: VictoriaMetrics vs ASAPQuery-backend on the five use cases - #245

Merged
GordonYuanyc merged 10 commits into
mainfrom
phase3
Nov 11, 2026
Merged

GordonYuanyc merged 10 commits into
mainfrom
phase3

Conversation

@GordonYuanyc

Copy link
Copy Markdown
Collaborator

Summary

Phase 3 runs phase 2's five use cases, queries and data on two time series systems:

  • VictoriaMetrics, which keeps every raw sample;
  • ASAPQuery-backend standalone: planned with raw_data_retained: false, with its data plane started with --no-forward, so it keeps sketches only.

Both systems get the same samples and the same instant queries, and are scored with phase 2's rules.

  • aqpbm-systemeval (new crate, binary systemeval):
    • It writes a spec's samples to a system by Prometheus Remote Write, in time order.
    • At each evaluation time it makes the samples written so far visible, then asks the queries due at that time. ASAPQuery-backend gets a watermark; VictoriaMetrics gets /internal/force_flush.
    • Loading everything first and querying afterwards does not work, because ASAPQuery-backend evicts panes older than the plan's retention.
    • Two write modes:
      • bulk: as fast as each system accepts;
      • replay: --speed N times the data's own rate.
  • planeval: answer scoring has moved to score.rs so that systemeval can share it.
  • scripts/run_use_case_on_systems.py: one run covers one use case, every arm that keeps no raw samples, and both systems. Per arm it:
    • plans the whole workload with ASAPQuery-backend's control_plane, and marks the queries it cannot plan as not supported by ASAPQuery-backend;
    • starts each system on a fresh port and data directory;
    • records insert rate, write and visibility time, peak and final resident memory, disk, CPU time, and query latency;
    • scores VictoriaMetrics answers against planeval's in-process exact plan;
    • uses VictoriaMetrics answers as the exact arm and ASAPQuery-backend answers as the approximate arm for the rules. Where VictoriaMetrics cannot answer a query (entropy_over_time), the in-process exact plan stands in.
  • Grafana:
    • Five new "Phase 3 · N · ..." dashboards, which read phase3/runs/ under the served data directory.
    • The runner writes to target/grafana-data/phase3 by default.
    • The phase 2 dashboards are unchanged.

Results (noraw, max error 0.05, bulk)

Use case Queries ASAPQuery-backend cannot plan Rule agreement
2 Autoscaling (synthetic, real) none 0.95–1.00
4 Canary (synthetic, real, real native) none Flagger, Argo Rollouts, CloudWatch 1.00; ours 0.71–0.79; Keptn 0.58–0.76
1 DDoS (synthetic, CAIDA 2019, MAWI 2019) distinct_5s, packets_5s; entropy_10s unless guarantees are ignored low on CAIDA and MAWI, see known limits
3 Video 4 of the queries; every rule needs one of them none evaluated
5 Top clients (synthetic, real, NASA) top10_clients_1m; on CAIDA and MAWI both top-k queries hit the executor memory limit 0.86–1.00
  • ASAPQuery-backend usually ingests 3–5x faster and stays at tens of MB resident; VictoriaMetrics starts at one to two hundred MB.
    • The exception is use case 5 on real data, where ASAPQuery-backend peaks at 650–860 MB.
  • Replay at 60x (use case 2): both systems keep up, with at most about 0.15 s lag.
  • Wherever VictoriaMetrics answered, its answers equal the in-process exact plan.

Known limits

  • Planner versions differ. ASAPQuery-backend main pins ASAPPlanner 05fb36e, older than planeval's b822553. At 05fb36e, a query whose window equals its interval cannot be planned without raw data.
  • Use cases 1 and 5 lose samples in ASAPQuery-backend.
    • These workloads write one sample per event, so a series often has several samples with the same millisecond timestamp.
    • ASAPQuery-backend follows the Prometheus data model and keeps only the first of them. VictoriaMetrics keeps all of them.
    • Use case 1 loses most of its samples this way, for example 10.6M of 11.2M on CAIDA 2019.
    • ASAPQuery-backend's dropped-sample counter misses a drop when another definition already buffered that sample.
  • Fixed executor memory limit. ASAPQuery-backend's executor has a fixed 64 MiB limit that its data plane does not expose.
  • Plain policy only. The bound and rerun policies from phase 2 have not been carried over.

Test plan

  • cargo build --release and cargo test --release --workspace
  • python3 -m unittest discover -p "test_*.py" in scripts/ (233 tests)
  • All five use cases, synthetic and real, on both systems; 71 runs loaded into Grafana, and every phase 3 dashboard panel returns rows

🤖 Generated with Claude Code

GordonYuanyc and others added 10 commits November 10, 2026 19:02
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Write and ask its queries at each evaluation time

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… planned to keep no raw samples

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oard

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nt plan below its root

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…its costs; failed evaluations have no latency

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s each inside a path; wider columns

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…shboards as on main

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@GordonYuanyc
GordonYuanyc merged commit 13c0069 into main Nov 11, 2026
2 checks passed
@GordonYuanyc
GordonYuanyc deleted the phase3 branch November 11, 2026 05:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant