Repository navigation
Conversation
Pins ASAPPlanner 1760a69f (asap-planner, asap-executor). The plan file is now wire version 15: a PhysicalASAPDAG with each view's and query-time Scan's retention_ms, and the query texts. control_plane: plan_installable returns (InstallablePlan, SelectedPlanReport); deployment inputs are DeploymentCapabilities and CostInputs; --cost-inputs, --report; load_plan checks the plan compiles. The backend declares query_time_results_retained: false (no view refreshed on first read yet). data_plane: the stored-summary, precompute, per-sample and raw-window modules are replaced by the executor. A time-indexed store keeps raw samples (Remote Write) and view rows (InMemoryDataStore); ingestion runs one ingestion run per refresh group window; each query is one query run of its root. Views are kept a margin past their retention so queries behind ingestion are answered. Raw samples are no longer read from Prometheus. Tests: planner_executor_parity compares the data plane over HTTP with the Planner executor in process on 14 public workloads, raw data kept or not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #845. #845 bumps the ASAPPlanner pin only and does not compile: the Planner's plan file, executor API and planning inputs changed under it (66 compile errors). This PR pins ASAPPlanner
1760a69fand runs the plan through the Planner's executor. Planner main is nowb511e4f8, which differs only by a CI fix.What changes for a user
control_planewritesplan.json(wire version 15) and, with--report report.json, Stage 3'sSelectedPlanReport(cost, predicted usage per node and per query).--plan-selection-settingsis replaced by--cost-inputs.--measured-operation-costsand--ignore-accuracy-guaranteesare kept.data_planekeeps the raw samples it receives by Remote Write. Planned queries no longer read raw samples from Prometheus. New flags:--run-memory-limit-bytesand--view-retention-margin-ms./api/v1/status/asapreportsingestioncounters (samples, and windows refreshed, failed or skipped) in place ofprecomputeandquery_run.query_time_results_retained: false, so the Planner selects no view refreshed on first read. The data plane does not refresh such views yet.How the data plane runs a plan
InstallablePlan::from_json,validate,materialized_views,ingestion_refresh_groupsDataStore: raw samples per scanned metric as(ts, value, $promql_series_identity, labels...); view rows in the executor'sInMemoryDataStoreW, phasep), oneexecute_ingestionrun per window(e - W, e]once the watermark reachese; rows appended with the range they covercompilewith the views it reads,Scope::Query), rows turned into a PromQL vectorThe plan's retention counts back from the newest evaluation, but ingestion refreshes windows ahead of the queries it answers. Views and query-time raw samples are therefore kept
--view-retention-margin-ms(default 60 s) past their retention. The end-to-end run found this: without the margin, a query at a time 20 s behind ingestion found its oldest windows already dropped.Code deleted, replaced by the Planner
sds.rs(definitions, keys, build order)materialized_views,ingestion_refresh_groupsprecompute.rspane buildsraw_sample_window.rsDataStore::read_raw_datawith the executor'sScan/TimeRangeper_sample.rs(per-sample state updates)query.rsraw fetches from Prometheus,QueryRunpanesTheir tests are deleted with them (
per_sample,precompute,query,query_run,sds,answers).Tests
planner_executor_parity: 14 public workloads (examples/, plus ASAPPlanner'srandom-mixed-seed-86201–06 and four sketch-plan snapshot workloads copied unchanged intodata_plane/tests/workloads/), each planned with raw data kept and not kept. The data plane runs over HTTP with--no-forward; the Planner executor runs in process on the same plan and samples. Result: 28 plans, 85 of 85 queries equal, all with rows.a_query_behind_the_newest_window_is_still_answered.fixed_intervalbecomesfixed_interval_at, andvalues_non_negativebecomescolumn_signs. Two expectations changed with the Planner's selection:ignore_accuracy_guarantees, because there is no guaranteed plan without raw data. It is bounded by the plan's own α (0.1).Scans differ only by aregioncolumn); answers are still within α.cargo fmt --check,cargo clippy --workspace --all-targets --locked -D warningsandcargo test --workspace --lockedpass.data_plane --no-forward. Four planned queries (rate,increase,max_over_time,sum(sum_over_time)) all run from tumbling-window views refreshed at ingestion. After about 2 h, at three evaluation times, every answer equals Prometheus's own answer exactly (56 + 56 + 1 + 1 series). RSS stayed at 19 MB.Not done yet
AcrossEvaluations).query_rangeis forwarded or refused, as before).--allowed-lateness-ms.🤖 Generated with Claude Code