Skip to content

feat: execute ASAPPlanner main's InstallablePlan through asap-executor - #852

Draft
zzylol wants to merge 1 commit into
mainfrom
sync/planner-main-1760a69f
Draft

zzylol wants to merge 1 commit into
mainfrom
sync/planner-main-1760a69f

Conversation

@zzylol

@zzylol zzylol commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Supersedes #845. #845 bumps the ASAPPlanner pin only and does not compile: the Planner's plan file, executor API and planning inputs changed under it (66 compile errors). This PR pins ASAPPlanner 1760a69f and runs the plan through the Planner's executor. Planner main is now b511e4f8, which differs only by a CI fix.

What changes for a user

  • control_plane writes plan.json (wire version 15) and, with --report report.json, Stage 3's SelectedPlanReport (cost, predicted usage per node and per query). --plan-selection-settings is replaced by --cost-inputs. --measured-operation-costs and --ignore-accuracy-guarantees are kept.
  • data_plane keeps the raw samples it receives by Remote Write. Planned queries no longer read raw samples from Prometheus. New flags: --run-memory-limit-bytes and --view-retention-margin-ms.
  • /api/v1/status/asap reports ingestion counters (samples, and windows refreshed, failed or skipped) in place of precompute and query_run.
  • The backend declares query_time_results_retained: false, so the Planner selects no view refreshed on first read. The data plane does not refresh such views yet.

How the data plane runs a plan

Step Planner API used
Read and check the plan InstallablePlan::from_json, validate, materialized_views, ingestion_refresh_groups
Store one DataStore: raw samples per scanned metric as (ts, value, $promql_series_identity, labels...); view rows in the executor's InMemoryDataStore
Ingestion per refresh group (W, phase p), one execute_ingestion run per window (e - W, e] once the watermark reaches e; rows appended with the range they cover
Query one compiled query run per root (compile with the views it reads, Scope::Query), rows turned into a PromQL vector

The plan's retention counts back from the newest evaluation, but ingestion refreshes windows ahead of the queries it answers. Views and query-time raw samples are therefore kept --view-retention-margin-ms (default 60 s) past their retention. The end-to-end run found this: without the margin, a query at a time 20 s behind ingestion found its oldest windows already dropped.

Code deleted, replaced by the Planner

Backend code Replaced by
sds.rs (definitions, keys, build order) materialized_views, ingestion_refresh_groups
precompute.rs pane builds ingestion runs per refresh window
raw_sample_window.rs DataStore::read_raw_data with the executor's Scan/TimeRange
per_sample.rs (per-sample state updates) none yet: views are built from the window's raw samples
query.rs raw fetches from Prometheus, QueryRun panes query runs over the data plane's own store

Their tests are deleted with them (per_sample, precompute, query, query_run, sds, answers).

Tests

  • New planner_executor_parity: 14 public workloads (examples/, plus ASAPPlanner's random-mixed-seed-862 01–06 and four sketch-plan snapshot workloads copied unchanged into data_plane/tests/workloads/), each planned with raw data kept and not kept. The data plane runs over HTTP with --no-forward; the Planner executor runs in process on the same plan and samples. Result: 28 plans, 85 of 85 queries equal, all with rows.
  • New a_query_behind_the_newest_window_is_still_answered.
  • Adapted to the Planner's current inputs: fixed_interval becomes fixed_interval_at, and values_non_negative becomes column_signs. Two expectations changed with the Planner's selection:
    • The signed-sample DDSketch test now needs ignore_accuracy_guarantees, because there is no guaranteed plan without raw data. It is bounded by the plan's own α (0.1).
    • The joint-quantile test no longer asserts one shared sketch. The Planner now builds two per-series DDSketch chains (their Scans differ only by a region column); answers are still within α.
  • cargo fmt --check, cargo clippy --workspace --all-targets --locked -D warnings and cargo test --workspace --locked pass.
  • End to end: Prometheus 3.5 scraping itself every second sends Remote Write to data_plane --no-forward. Four planned queries (rate, increase, max_over_time, sum(sum_over_time)) all run from tumbling-window views refreshed at ingestion. After about 2 h, at three evaluation times, every answer equals Prometheus's own answer exactly (56 + 56 + 1 + 1 series). RSS stayed at 19 MB.

Not done yet

  • Views refreshed on first read (query-time AcrossEvaluations).
  • Range queries (query_range is forwarded or refused, as before).
  • One engine thread runs ingestion and queries in turn, and views are built from each window's raw samples rather than updated per sample.
  • A planned query is answered only once the windows it reads are refreshed, so a query at "now" waits for --allowed-lateness-ms.

🤖 Generated with Claude Code

Pins ASAPPlanner 1760a69f (asap-planner, asap-executor). The plan file is
now wire version 15: a PhysicalASAPDAG with each view's and query-time
Scan's retention_ms, and the query texts.

control_plane: plan_installable returns (InstallablePlan,
SelectedPlanReport); deployment inputs are DeploymentCapabilities and
CostInputs; --cost-inputs, --report; load_plan checks the plan compiles.
The backend declares query_time_results_retained: false (no view
refreshed on first read yet).

data_plane: the stored-summary, precompute, per-sample and raw-window
modules are replaced by the executor. A time-indexed store keeps raw
samples (Remote Write) and view rows (InMemoryDataStore); ingestion runs
one ingestion run per refresh group window; each query is one query run
of its root. Views are kept a margin past their retention so queries
behind ingestion are answered. Raw samples are no longer read from
Prometheus.

Tests: planner_executor_parity compares the data plane over HTTP with the
Planner executor in process on 14 public workloads, raw data kept or not.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol zzylol added enhancement New feature or request area: control plane Planning, configuration, routing, and control-plane orchestration area: data plane Runtime ingestion, storage, and query-serving data paths integration: ASAPPlanner ASAPPlanner API, IR, candidate, and selected-plan integration test: end-to-end Process-boundary, differential, conformance, and full-stack tests labels Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: control plane Planning, configuration, routing, and control-plane orchestration area: data plane Runtime ingestion, storage, and query-serving data paths enhancement New feature or request integration: ASAPPlanner ASAPPlanner API, IR, candidate, and selected-plan integration test: end-to-end Process-boundary, differential, conformance, and full-stack tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant