Repository navigation
Description
Status: agreed direction (Milind, Zeying), 2026-10-27. Plan revised 2026-11-10: port the backend one component at a time onto the current planner interface instead of copying it (see "Revision"). Revised again 2026-11-17: steps filed as sub-issues with dependencies; PR-stack conventions; runtime plan install; open questions resolved.
Context
- ASAPPlanner turns PromQL/SQL/MetricsQL workloads into one
PhysicalASAPDAGacross all queries. It also ownsasap-executor(crates/executor): the physical operator library, sketch kernels overasap_sketchlib, DAG compiler and runtime. - ASAPQuery-backend was rebuilt in refactor!: plan with ASAPPlanner main's stage pipeline and drop the old execution stack ASAPQuery-backend#818 through 822 (Oct 18–21) as a thin (~4k LOC) deployment of that executor. It answers planned instant PromQL queries over Prometheus Remote Write and forwards the rest. Known scale bugs: Query runs fail with "execution memory limit exceeded" above ~2M raw samples per window ASAPQuery-backend#848, A pane never closes when its series stops, so every query over that window falls back ASAPQuery-backend#772, Series interner grows without bound under series churn ASAPQuery-backend#842. Removed features: Capabilities removed with the old backend that are not replaced yet (#818) ASAPQuery-backend#823. The repo is frozen.
- ASAPQuery works end to end on many workloads and has the surrounding scaffolding: Prometheus/ClickHouse/Elastic query adapters with fallback, range queries, Remote Write/VictoriaMetrics/JSON ingest,
promql-compliance,benchmarks/,asap-tools(CloudLab). Its plan contract (streaming_config/inference_configfromasap-planner-rs) is incompatible with ASAPPlanner's DAG.
Revision (2026-11-10)
- The backend no longer compiles against ASAPPlanner
main. Bumping its pin from05fb36e4(174 commits behind) gives 31 errors. The types its data plane is built on (stored_outputs,StoredOutputBuild,RawSampleWindow,InputContract,Scope::revision) were removed. Copying it as-is would import code written against an interface that no longer exists. Its design, tests and hard-won logic (watermark/barrier, per-sample fast path, Prometheus raw fetch, series interner, admission) are still worth porting. - The planner/runtime interface is now narrow:
- Plan file:
InstallablePlan { dag, query_texts }(wire version 11). - Derived from the DAG:
materialized_views(): what to keep, refreshedAtIngestionorOnFirstRead.ingestion_refresh_groups(): views refreshed together, per window width and phase.raw_data_retention().
- The runtime implements one trait,
asap_executor::sources::DataStore:read_raw_data(source, request)read_materialized_view(view, request): exact tiling; missing state is an error, never empty.
crates/planner/tests/runtime_install.rscontains an in-memory reference runtime on this interface.
- Plan file:
- ASAPQuery splits into a shell and a core.
- The shell has narrow, plan-agnostic seams, and we keep it:
IngestSource→DecodedSample;QueryRequestAdapter/QueryResponseAdapter;FallbackClient;- the engine entry
handle_query(str, t) -> Result<Option<(labels, QueryResult)>, QueryExecutionError>.
- The core is threaded by
AggregationConfig/aggregation_idfrom routing through workers, accumulators,StoreandSimpleEngine. It has no 1:1 counterpart in a shared multi-query DAG, so it is replaced by growing the new core under the shell's seams, one component per step.
- The shell has narrow, plan-agnostic seams, and we keep it:
Decision
Build the new core inside ASAPQuery behind --plan plan.json, porting backend code one component at a time onto the current interface. Run it side by side with the legacy engine, reach parity on live e2e tests, switch over, then delete the legacy core. Sunset the ASAPQuery-backend repo. asap-executor stays in ASAPPlanner as a pinned git dependency.
Target architecture
| Component | Responsibility | Comes from |
|---|---|---|
| Physical operator library | All computation: operators, sketch build/merge/readout | asap-executor (ASAPPlanner). Replaces ASAPQuery's accumulators, window merging and native query DAG |
| SDS | View catalog: which views the plan keeps, which windows are stored and ready, retention/eviction, view → storage key | New, on materialized_views(); ideas from backend sds.rs |
| Storage engine | Holds view states and their metadata by key | New; ASAPQuery's Store designs inform later persistence/spill |
| Precompute engine | Turns ingested samples into view states: one ingestion run per refresh group and window once the watermark passes; watermarks | Port of backend precompute.rs, per_sample.rs, series interner; fed by ASAPQuery's ingest adapters. Multi-worker later |
| Query engine | Matches a query to a plan root, runs it through DataStore, converts Arrow output to QueryResult; adapters and fallback unchanged |
ASAPQuery's servers/adapters/fallback + new PlanEngine behind the existing handle_query contract; port of backend prometheus.rs raw fetch |
Code lives in one new dag module in asap-query-engine (sds, storage, precompute, query). A plan subcommand wraps plan_installable. No new crates.
Plan
Each step is a sub-issue and a stack of small PRs; it makes a defined class of plans run. The harness shows each step as scoreboard rows moving from "refused" to "answered". Owner: Milind.
| Step | Issue | Blocked by |
|---|---|---|
S0a: x-asap-execution header + error-contract fixes (legacy only) |
#921 | — |
| S0b: CI runs compliance + feature-gated tests on every PR | #922 | — |
| S0c: harness: workload generator, scoreboard, watermark endpoint, perf capture | #923 | S0a |
S1: planner deps, plan subcommand, --plan load/validate, install() |
#924 | — |
S2: PlanEngine over raw data only |
#925 | S0c, S1, ProjectASAP/ASAPPlanner#1279 (harness arm only) |
S3: SDS + storage, serve read_materialized_view |
#926 | S1 |
| S4: precompute engine; then archive ASAPQuery-backend | #927 | S3, S0c |
| S4b: install a plan while running | #928 | S4 |
S5: OnFirstRead views |
#929 | S2, S3 |
| S6: range queries (per-step instant) | #930 | S2 |
S7a: sql-compliance suite |
#931 | S0c, S1 |
| S7: SQL serving (ClickHouse, Elastic SQL); query-time SQL refused | #932 | S4, S7a |
| S8: parity → flip default → delete legacy in slices | #933 | S2, S4b, S5, S6, S7 |
| Query → plan matching by pattern/capability | #934 | S2 |
- Start in parallel: S0a, S0b, S1. S2 and S3 are independent of each other.
- Critical path: S1 → S3 → S4 → S7 → S8.
- If ASAPPlanner#1279 slips, S2 lands up to wiring using hand-written raw-only plans.
PR stack
- Plain stacked branches, each based on the one below.
- One role per PR, in this order: Interface (types/signatures, bodies return refused) → Contract tests (
#[ignore]d, they are the spec) → Implementation (one function/component, removes its ignores) → Wiring (intohandle_query/ingest/CLI) → Harness (new suite or arm). - Every PR compiles and passes tests; only behavior is incomplete. Unfinished paths return "refused: not implemented", never
todo!(). Lower PRs can merge while upper ones are in review;git bisectworks. - PR description: role; what it deliberately doesn't do yet; command to verify; scoreboard diff (Wiring/Harness).
- Size: ~400 changed lines or fewer. Ported backend code is exempt but goes in its own PR, diffable against the original file.
Example, S2: (1) Interface: PlanEngine skeleton, --plan returns refused → (2) Contract tests: matching, raw fetch, Arrow → QueryResult → (3) exact-text matching → (4) Prometheus raw DataStore (port of backend prometheus.rs) → (5) Arrow → QueryResult → (6) Wiring + promql-compliance arm.
Rejected alternatives:
- Translate
plan.jsoninto legacy configs. Legacy configs can't express shared views or multi-level DAGs, and the translator would be throwaway. - Swap legacy accumulators for executor kernels. That is work on code being deleted, and it doesn't move the plan contract.
Testing
Goal: harnesses that expose correctness and performance bugs for the whole repo, PromQL and SQL, covering plan → install → live data plane on real data.
- Tiers.
cargo test: fast, in-process; each component's own tests, incl. ported backend tests.- Docker compliance suites in CI:
promql-compliance, newsql-compliance,benchmarks/. - CloudLab via
asap-toolsfor scale.
- One workload definition. The suite YAML is the source of truth; both engines are planned from it.
- One engine per process. Legacy and
--planrun as separate containers next to the reference (Prometheus or ClickHouse). - Who answered. Both engines set
x-asap-execution: plan | forwarded | refusedplus a reason; tests run with forwarding off. A forwarded answer never counts as a pass. - Scoreboard. Per query and engine: answered-correct, answered-wrong, or not answered (reason). Correct means within the plan's accuracy guarantee for that query, against the reference. The legacy-vs-
--plandiff is diagnostic only (differentasap_sketchlibversions). - Deterministic time. Seed data in the past. Start the engine with
--stream-start-ms= seed start. Send the watermark barrier after seeding instead of sleeping. Off-grid evaluation times are reported as "not answered: off grid". - Performance. Ingest throughput, query latency, CPU and RSS, measured from outside the process. Reported as trends (JSON artifact per run); never blocks PRs.
- Parity = switch-over criterion. For every suite,
--plananswers at least what legacy answers, within guarantee. Perf is reported, not gated.
Decisions log
| Topic | Decision |
|---|---|
| Executor location | Stays in ASAPPlanner, pinned by git rev |
| Planner pin | Bumped manually every few weeks, one PR that re-runs the harness; plan-format changes are announced to ASAPQuery's owner |
| Backend code | Ported per component onto the current interface, not copied |
| Migration style | Side by side, parity, then switch over |
| ASAPQuery-backend | Frozen; archived once its logic is ported (step 4) |
| SDS vs storage | Two components: SDS metadata; storage holds states + their metadata |
| Precompute windows | Tumbling only; sliding windows and late-data forwarding re-added later if needed |
| Query-time plans | PromQL: supported (steps 2, 5). SQL: refused with an error |
| Query → plan matching | Exact text for now; follow-up issue for pattern/capability matching |
| Plan install | One install(plan); startup = install into empty engine. Runtime install of a finished plan.json (S4b): shared views keep state, new views refused until warm, dropped views freed, atomic swap. Engine never runs the planner |
| Legacy replanning loop | Dropped (LocalPlannerClient, config hot-swap) |
| Elastic | Survives via Elastic SQL (planner SQL frontend). QueryDSL removed |
| RQE route | Kept until parity; revisit then, removed soon after |
| Storage form | Live in-memory states first; serialized ASAPv1 bytes with persistence later |
| CI | Compliance suites run on every PR; perf reported, never gating |
| PR workflow | Plain stacked branches, one role per PR, every PR green (see "PR stack") |
| Owner | Milind, all steps |
| Planner reference runtime as a third oracle | Later |
| MILP (ProjectASAP/sketch-bench#143 / #790) | Stays for now; moves into ASAPPlanner later |
Open questions
- Target dates for PromQL parity (S6) and SQL (S7).
Follow-ups
- DeploymentCapabilities: allow forbidding ingestion-time materialization ASAPPlanner#1279: capability to forbid ingestion-time materialization (needed by S2's harness arm).
References
- ASAPPlanner:
crates/executor/README.md,crates/executor/src/sources/data_store.rs,crates/planner/src/installable.rs,crates/types/src/ir/materialized_view.rs,crates/planner/tests/runtime_install.rs,docs/design_docs/physical-planning-and-deployment.md - ASAPQuery-backend: README; refactor!: plan with ASAPPlanner main's stage pipeline and drop the old execution stack ASAPQuery-backend#818 through 822 (rebuild), Capabilities removed with the old backend that are not replaced yet (#818) ASAPQuery-backend#823 (removed capabilities)
- ASAPQuery:
promql-compliance/ARCHITECTURE.md,docs/729-native-query-dag-todos.md,asap-tools/experiments/experiment_run_e2e.py - sketch-bench
phase2-ver3:aqpbm-planeval/(backend vs Prometheus harness)
Activity
plansubcommand,install()#924