Repository navigation
Description
Example
Workload y16 has two PromQL queries, each evaluated every 60 s at phase 15 s. Raw data is not kept, and there are 1,000 gauge series of m:
sum_over_time(m[5m])
sum_over_time(m[5m] offset 1m)
Stage 3 selects node 5, FinalizeExactAccumulator, with lifetime OneEvaluation(Memory). The executor computes it to the end once per query run and holds its output until its consumer reads it (#1290).
Planner charge (plan_cost detail) |
Executor holds (12 series, review harness) | |
|---|---|---|
| y16 node 5, Finalize | holds 32000 bytes for 1,000 accumulators: 32 B/row, so 384 B at 12 rows |
1,464 B (122 B/row) |
n14 node 3, KLL SummaryMerge |
holds 1256000 bytes for 1,000 states: 1,256 B/row, so 15,072 B at 12 rows |
35,608 B |
n14 node 7, KLL SummaryMerge |
1,256 B/row, so 15,072 B at 12 rows | 60,184 B |
- Today: the planner charges 2.4 to 8 times less memory than the executor holds for the same rows. The other workloads measured show the same gap: b17 and g10 aggregates are charged 24 B but hold 200 B, and n02's
TimeRangeis charged 1,680 B but holds 9,320 B. - Expected: the bytes the planner charges for a held
OneEvaluationoutput match, within a small factor, the bytes the executor reserves for it.
How this was measured: the Stage 3 review harness ran each workload at 12 series per metric, with a temporary trace (not committed) that printed output_bytes (Batch::memory_bytes) for every batch a held producer reserves. The planner figures are its plan_cost per-node details for the workload's 1,000 series, scaled to 12 rows.
Why
- The planner (
crates/cost/src/node/usage/mod.rs, theheld_bytesthat produces theholds {bytes} bytesdetail) charges each output row's state or value bytes: 32 B for an exact accumulator, 24 B for one aggregate value, and the KLL state size. - The executor charges each batch's Arrow buffers to the run (
runtime::Producer, throughPhysicalOperator::output_bytes). That includes the label or series-identity columns (aMapcolumn per series), the timestamp column, validity bitmaps and buffer capacity. - A one-row aggregate's batch costs about 200 B whatever its value size.
Since #1290, a query-time OneEvaluation output is held whole until its last consumer reads it. That makes this the memory the run actually holds.
Proposal
- Make the planner's
OneEvaluationmemory charge count the output's label columns and buffer layout, not only state or value bytes. That means per-row bytes for each output column (labels or series identity, timestamp, value or state), plus a fixed per-batch overhead. - Check the result against the executor: a test that runs a plan's held output and compares
RunContext::peak_bytes(or the held batches'memory_bytes) with the planner's charge. - The open alternative is to leave the per-row state charge and add a measured per-row label overhead per metric in
CostInputs.
Scope and acceptance
- In scope: the working-memory bytes the planner charges for query-time
OneEvaluation(and query-timeAcrossEvaluations) outputs. - Out of scope: stored view sizes, and CPU cost.
- Acceptance: for y16 node 5 and n14 nodes 3 and 7, the planner's held bytes at 12 series are within 1.5 times the executor's measured held bytes (1,464 B; 35,608 B; 60,184 B). A test fails today for y16 node 5 (384 B charged vs 1,464 B held).
Status and related
- Found while verifying Executor deadlocks when two query roots that share a producer are drained one after another #1290 (PR fix(executor): run each node by its plan Lifetime, so roots drain in any order #1300). On 2026-11-14 it was decided to file this mismatch separately.
- The planner charges the memory for only about 0.001–0.02 ms per evaluation in these workloads, so its effect on plan cost is small today.
Activity
zzylol commentedon Nov 16, 2026
The series identity has two prices on adjacent edges
Since #1363, a summary state's row is charged its own copy of the series identity at 64 bytes (
state_row_bytes_beside_state, constantSERIES_IDENTITY_BYTES). Every other operator output edge still prices it at 16 bytes, the raw-row price (column_bytesthroughrow_bytes).Example:
sum by (job) (sum_over_time(m[5m]))over 1,000 series with stored per-series Sum windows carryingjob. Each stored state row is charged 144 bytes: 56 of state, 64 for the series identity, 16 forjoband 8 for the window time. TheFinalizeExactAccumulatorright above it outputs the same series identity column, but charges 16 bytes for it.The edges that still charge 16 bytes for the identity:
FinalizeExactAccumulatorandSummaryEstimateoutputs;Aggregate(hash aggregate) outputs;TimeRange,Filter,Projectand the other pass-through operators;Scan's output batches (its scanned bytes), andBinaryOpand join outputs.The held
OneEvaluationoutput charge in this issue (held_bytes) inherits 64 bytes only for summary nodes, so y16's Finalize and n02'sTimeRangeare still charged 16.Retained raw rows should keep the 16-byte price, since the store shares one copy of each series' labels. That covers a materialized
ScanorTimeShift(raw_rows_materialized_at_ingestion) and the raw data a query-timeScankeeps on disk.Measured effect of pricing every operator output edge at 64 bytes (identity only; raw retention left at 16), as a plan survey against #1363's head:
So the change needs its own measurement against the executor's batch bytes. That measurement is this issue's acceptance test. #1363 shares the column price (
column_bytes) and the constant, and leaves the edges above to this issue.