Skip to content

Migrate ASAPQuery-backend's DAG data plane into ASAPQuery and run ASAPPlanner plans end to end #918

Open
0 of 14 issues completed

Description

@milindsrivastava1997

Status: agreed direction (Milind, Zeying), 2026-10-27. Plan revised 2026-11-10: port the backend one component at a time onto the current planner interface instead of copying it (see "Revision"). Revised again 2026-11-17: steps filed as sub-issues with dependencies; PR-stack conventions; runtime plan install; open questions resolved.

Context

Revision (2026-11-10)

  • The backend no longer compiles against ASAPPlanner main. Bumping its pin from 05fb36e4 (174 commits behind) gives 31 errors. The types its data plane is built on (stored_outputs, StoredOutputBuild, RawSampleWindow, InputContract, Scope::revision) were removed. Copying it as-is would import code written against an interface that no longer exists. Its design, tests and hard-won logic (watermark/barrier, per-sample fast path, Prometheus raw fetch, series interner, admission) are still worth porting.
  • The planner/runtime interface is now narrow:
    • Plan file: InstallablePlan { dag, query_texts } (wire version 11).
    • Derived from the DAG:
      • materialized_views(): what to keep, refreshed AtIngestion or OnFirstRead.
      • ingestion_refresh_groups(): views refreshed together, per window width and phase.
      • raw_data_retention().
    • The runtime implements one trait, asap_executor::sources::DataStore:
      • read_raw_data(source, request)
      • read_materialized_view(view, request): exact tiling; missing state is an error, never empty.
    • crates/planner/tests/runtime_install.rs contains an in-memory reference runtime on this interface.
  • ASAPQuery splits into a shell and a core.
    • The shell has narrow, plan-agnostic seams, and we keep it:
      • IngestSource → DecodedSample;
      • QueryRequestAdapter/QueryResponseAdapter;
      • FallbackClient;
      • the engine entry handle_query(str, t) -> Result<Option<(labels, QueryResult)>, QueryExecutionError>.
    • The core is threaded by AggregationConfig/aggregation_id from routing through workers, accumulators, Store and SimpleEngine. It has no 1:1 counterpart in a shared multi-query DAG, so it is replaced by growing the new core under the shell's seams, one component per step.

Decision

Build the new core inside ASAPQuery behind --plan plan.json, porting backend code one component at a time onto the current interface. Run it side by side with the legacy engine, reach parity on live e2e tests, switch over, then delete the legacy core. Sunset the ASAPQuery-backend repo. asap-executor stays in ASAPPlanner as a pinned git dependency.

Target architecture

Component Responsibility Comes from
Physical operator library All computation: operators, sketch build/merge/readout asap-executor (ASAPPlanner). Replaces ASAPQuery's accumulators, window merging and native query DAG
SDS View catalog: which views the plan keeps, which windows are stored and ready, retention/eviction, view → storage key New, on materialized_views(); ideas from backend sds.rs
Storage engine Holds view states and their metadata by key New; ASAPQuery's Store designs inform later persistence/spill
Precompute engine Turns ingested samples into view states: one ingestion run per refresh group and window once the watermark passes; watermarks Port of backend precompute.rs, per_sample.rs, series interner; fed by ASAPQuery's ingest adapters. Multi-worker later
Query engine Matches a query to a plan root, runs it through DataStore, converts Arrow output to QueryResult; adapters and fallback unchanged ASAPQuery's servers/adapters/fallback + new PlanEngine behind the existing handle_query contract; port of backend prometheus.rs raw fetch

Code lives in one new dag module in asap-query-engine (sds, storage, precompute, query). A plan subcommand wraps plan_installable. No new crates.

Plan

Each step is a sub-issue and a stack of small PRs; it makes a defined class of plans run. The harness shows each step as scoreboard rows moving from "refused" to "answered". Owner: Milind.

Step Issue Blocked by
S0a: x-asap-execution header + error-contract fixes (legacy only) #921 —
S0b: CI runs compliance + feature-gated tests on every PR #922 —
S0c: harness: workload generator, scoreboard, watermark endpoint, perf capture #923 S0a
S1: planner deps, plan subcommand, --plan load/validate, install() #924 —
S2: PlanEngine over raw data only #925 S0c, S1, ProjectASAP/ASAPPlanner#1279 (harness arm only)
S3: SDS + storage, serve read_materialized_view #926 S1
S4: precompute engine; then archive ASAPQuery-backend #927 S3, S0c
S4b: install a plan while running #928 S4
S5: OnFirstRead views #929 S2, S3
S6: range queries (per-step instant) #930 S2
S7a: sql-compliance suite #931 S0c, S1
S7: SQL serving (ClickHouse, Elastic SQL); query-time SQL refused #932 S4, S7a
S8: parity → flip default → delete legacy in slices #933 S2, S4b, S5, S6, S7
Query → plan matching by pattern/capability #934 S2
  • Start in parallel: S0a, S0b, S1. S2 and S3 are independent of each other.
  • Critical path: S1 → S3 → S4 → S7 → S8.
  • If ASAPPlanner#1279 slips, S2 lands up to wiring using hand-written raw-only plans.

PR stack

  • Plain stacked branches, each based on the one below.
  • One role per PR, in this order: Interface (types/signatures, bodies return refused) → Contract tests (#[ignore]d, they are the spec) → Implementation (one function/component, removes its ignores) → Wiring (into handle_query/ingest/CLI) → Harness (new suite or arm).
  • Every PR compiles and passes tests; only behavior is incomplete. Unfinished paths return "refused: not implemented", never todo!(). Lower PRs can merge while upper ones are in review; git bisect works.
  • PR description: role; what it deliberately doesn't do yet; command to verify; scoreboard diff (Wiring/Harness).
  • Size: ~400 changed lines or fewer. Ported backend code is exempt but goes in its own PR, diffable against the original file.

Example, S2: (1) Interface: PlanEngine skeleton, --plan returns refused → (2) Contract tests: matching, raw fetch, Arrow → QueryResult → (3) exact-text matching → (4) Prometheus raw DataStore (port of backend prometheus.rs) → (5) Arrow → QueryResult → (6) Wiring + promql-compliance arm.

Rejected alternatives:

  • Translate plan.json into legacy configs. Legacy configs can't express shared views or multi-level DAGs, and the translator would be throwaway.
  • Swap legacy accumulators for executor kernels. That is work on code being deleted, and it doesn't move the plan contract.

Testing

Goal: harnesses that expose correctness and performance bugs for the whole repo, PromQL and SQL, covering plan → install → live data plane on real data.

  • Tiers.
    • cargo test: fast, in-process; each component's own tests, incl. ported backend tests.
    • Docker compliance suites in CI: promql-compliance, new sql-compliance, benchmarks/.
    • CloudLab via asap-tools for scale.
  • One workload definition. The suite YAML is the source of truth; both engines are planned from it.
  • One engine per process. Legacy and --plan run as separate containers next to the reference (Prometheus or ClickHouse).
  • Who answered. Both engines set x-asap-execution: plan | forwarded | refused plus a reason; tests run with forwarding off. A forwarded answer never counts as a pass.
  • Scoreboard. Per query and engine: answered-correct, answered-wrong, or not answered (reason). Correct means within the plan's accuracy guarantee for that query, against the reference. The legacy-vs---plan diff is diagnostic only (different asap_sketchlib versions).
  • Deterministic time. Seed data in the past. Start the engine with --stream-start-ms = seed start. Send the watermark barrier after seeding instead of sleeping. Off-grid evaluation times are reported as "not answered: off grid".
  • Performance. Ingest throughput, query latency, CPU and RSS, measured from outside the process. Reported as trends (JSON artifact per run); never blocks PRs.
  • Parity = switch-over criterion. For every suite, --plan answers at least what legacy answers, within guarantee. Perf is reported, not gated.

Decisions log

Topic Decision
Executor location Stays in ASAPPlanner, pinned by git rev
Planner pin Bumped manually every few weeks, one PR that re-runs the harness; plan-format changes are announced to ASAPQuery's owner
Backend code Ported per component onto the current interface, not copied
Migration style Side by side, parity, then switch over
ASAPQuery-backend Frozen; archived once its logic is ported (step 4)
SDS vs storage Two components: SDS metadata; storage holds states + their metadata
Precompute windows Tumbling only; sliding windows and late-data forwarding re-added later if needed
Query-time plans PromQL: supported (steps 2, 5). SQL: refused with an error
Query → plan matching Exact text for now; follow-up issue for pattern/capability matching
Plan install One install(plan); startup = install into empty engine. Runtime install of a finished plan.json (S4b): shared views keep state, new views refused until warm, dropped views freed, atomic swap. Engine never runs the planner
Legacy replanning loop Dropped (LocalPlannerClient, config hot-swap)
Elastic Survives via Elastic SQL (planner SQL frontend). QueryDSL removed
RQE route Kept until parity; revisit then, removed soon after
Storage form Live in-memory states first; serialized ASAPv1 bytes with persistence later
CI Compliance suites run on every PR; perf reported, never gating
PR workflow Plain stacked branches, one role per PR, every PR green (see "PR stack")
Owner Milind, all steps
Planner reference runtime as a third oracle Later
MILP (ProjectASAP/sketch-bench#143 / #790) Stays for now; moves into ASAPPlanner later

Open questions

  • Target dates for PromQL parity (S6) and SQL (S7).

Follow-ups

References

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions