Repository navigation
Description
Example
The MVP deployment plans this PromQL query (workload tools/dag-viewer/workloads/sketch-heavy/frequency-moments-univmon-l2__r4-10hz-6h-every60s-releps0.05-d0.05.json from #1388):
l2_over_time(dns_query_client[6h])
with 4 gauge series at 10 Hz (40 rows/s), evaluated every 60 s, relative error 0.05 at δ 0.05.
Stage 3 selects an ingestion-time UnivMon tumbling-window chain (1-minute windows, 360 of them merged per evaluation). Its update counts each sample value once per series:
"SummaryTumblingWindowChain": {
"child": 0,
"filter": null,
"input": { "item": { "Column": "SampleValue" }, "weight": { "Constant": 1.0 } },
"reduction": "PerEntity",
"summary_type": { "Sketch": "UnivMon" },
"tumbling_window_width_ms": 60000
}Today: the plan cannot be installed. Building the installable plan fails with
the executor cannot refresh materialized view 1 from its inputs: invalid DAG: per-entity update requires a projected value column
The snapshot crates/devtools/tests/sketch_plan_snapshots/sketch-heavy-univmon-l2-mvp.json in #1388 records this under materialized_views_error. A mixed dashboard workload in the same set (sketch-heavy-edge-dashboard-mvp.json: top-k, total rate, L2 and distinct over two metrics) fails the same way at its UnivMon chain (refreshed with views [1, 15]), so the whole plan cannot run.
Expected: the executor builds one UnivMon state per series and tumbling window, counting each sample value once, and merges 360 windows at query time, as the plan says.
Current behavior
The executor compiles a SummaryTumblingWindowChain with reduction: PerEntity only when the update's weight is a column (crates/executor/src/physical_planner/mod.rs, the PlannerReduction::PerEntity arm of the tumbling-window chain, around line 657):
PlannerReduction::PerEntity => {
let SummaryInputExpr::Column(value) = &update.weight else {
return Err(invalid("per-entity update requires a projected value column"));
};
...
Operator::per_group_summary_build(windowed.clone(), summary_type.clone(), node.parameters.clone(), value, Some(time), groups)?
}A UnivMon value stream is a unit-frequency update: the counted value is in item and the weight is the constant 1.0. The executor already accepts that shape elsewhere: summary_agg_build (same file, around line 1462) recognizes it with capability::unit_frequency_column(update) for the unkeyed frequency families (UnivMon, HLL) and builds over the item column, but only for Reduce reductions. The per-entity arm never asks unit_frequency_column, so the chain is rejected.
Expected behavior and invariant
Any update shape the executor accepts for a Reduce summary build it also accepts for a PerEntity chain: for a unit-frequency update (item a column, weight the constant 1) of an unkeyed frequency family, the per-entity build reads the item column as its value and groups by every other column except the time column. The planner keeps selecting this plan. Executor limits must not restrict the planner's design, so the fix is in the executor and Stage 2/3 do not change.
Minimal reproduction
cargo run --release -p asap-devtools --bin asap_planner -- \
--workload tools/dag-viewer/workloads/sketch-heavy/frequency-moments-univmon-l2__r4-10hz-6h-every60s-releps0.05-d0.05.json \
--run-until stage3 --format json \
--deployment-capabilities crates/devtools/tests/sketch_plan_snapshots/deployments/mvp.capabilities.json \
--cost-inputs tools/dag-viewer/cost-inputs/aws-fargate-us-east-1.jsonThe JSON output carries the error above. Or run the sketch-heavy-univmon-l2-mvp entry of sketch_plans_match_snapshots (#1388).
Impact
No plan whose UnivMon value-stream chain is built at ingestion time per series can run on the executor, so the UnivMon family of the sketch-heavy workload set (and any dashboard mixing it with other queries) cannot be measured end to end. Planning itself is correct.
Regression test
An executor test that compiles and runs a SummaryTumblingWindowChain with reduction: PerEntity, summary_type: Sketch(UnivMon) and update { item: Column(SampleValue), weight: Constant(1.0) } over two series of known values, and checks the merged L2 per series against the exact L2 within UnivMon's error. It fails today with "per-entity update requires a projected value column". After the fix, the two #1388 snapshots lose materialized_views_error and record their materialized views.
End-to-end verification
Re-run sketch_plans_match_snapshots (release) with UPDATE_SNAPSHOTS=1 and check that sketch-heavy-univmon-l2-mvp and sketch-heavy-edge-dashboard-mvp now list their views; then run the UnivMon workload on the executor with generated data and compare l2_over_time with the exact answer.
Related
- test(devtools): sketch-heavy workload set, five family snapshots, and a rollup test that selects a KLL #1388: the sketch-heavy workload set and the two snapshots that record this.
Activity