Skip to content

Planner charges less memory for a held OneEvaluation output than the executor holds #1301

Open

Description

@zzylol

Example

Workload y16 has two PromQL queries, each evaluated every 60 s at phase 15 s. Raw data is not kept, and there are 1,000 gauge series of m:

sum_over_time(m[5m])
sum_over_time(m[5m] offset 1m)

Stage 3 selects node 5, FinalizeExactAccumulator, with lifetime OneEvaluation(Memory). The executor computes it to the end once per query run and holds its output until its consumer reads it (#1290).

Planner charge (plan_cost detail) Executor holds (12 series, review harness)
y16 node 5, Finalize holds 32000 bytes for 1,000 accumulators: 32 B/row, so 384 B at 12 rows 1,464 B (122 B/row)
n14 node 3, KLL SummaryMerge holds 1256000 bytes for 1,000 states: 1,256 B/row, so 15,072 B at 12 rows 35,608 B
n14 node 7, KLL SummaryMerge 1,256 B/row, so 15,072 B at 12 rows 60,184 B
  • Today: the planner charges 2.4 to 8 times less memory than the executor holds for the same rows. The other workloads measured show the same gap: b17 and g10 aggregates are charged 24 B but hold 200 B, and n02's TimeRange is charged 1,680 B but holds 9,320 B.
  • Expected: the bytes the planner charges for a held OneEvaluation output match, within a small factor, the bytes the executor reserves for it.

How this was measured: the Stage 3 review harness ran each workload at 12 series per metric, with a temporary trace (not committed) that printed output_bytes (Batch::memory_bytes) for every batch a held producer reserves. The planner figures are its plan_cost per-node details for the workload's 1,000 series, scaled to 12 rows.

Why

  • The planner (crates/cost/src/node/usage/mod.rs, the held_bytes that produces the holds {bytes} bytes detail) charges each output row's state or value bytes: 32 B for an exact accumulator, 24 B for one aggregate value, and the KLL state size.
  • The executor charges each batch's Arrow buffers to the run (runtime::Producer, through PhysicalOperator::output_bytes). That includes the label or series-identity columns (a Map column per series), the timestamp column, validity bitmaps and buffer capacity.
  • A one-row aggregate's batch costs about 200 B whatever its value size.

Since #1290, a query-time OneEvaluation output is held whole until its last consumer reads it. That makes this the memory the run actually holds.

Proposal

  • Make the planner's OneEvaluation memory charge count the output's label columns and buffer layout, not only state or value bytes. That means per-row bytes for each output column (labels or series identity, timestamp, value or state), plus a fixed per-batch overhead.
  • Check the result against the executor: a test that runs a plan's held output and compares RunContext::peak_bytes (or the held batches' memory_bytes) with the planner's charge.
  • The open alternative is to leave the per-row state charge and add a measured per-row label overhead per metric in CostInputs.

Scope and acceptance

  • In scope: the working-memory bytes the planner charges for query-time OneEvaluation (and query-time AcrossEvaluations) outputs.
  • Out of scope: stored view sizes, and CPU cost.
  • Acceptance: for y16 node 5 and n14 nodes 3 and 7, the planner's held bytes at 12 series are within 1.5 times the executor's measured held bytes (1,464 B; 35,608 B; 60,184 B). A test fails today for y16 node 5 (384 B charged vs 1,464 B held).

Status and related

Activity

  1. added
    bugSomething isn't working
    cost-modelCost formulas, statistics, ranking, and selection
    on Nov 14, 2026
  2. zzylol commented on Nov 16, 2026

    @zzylol
    ContributorAuthor

    The series identity has two prices on adjacent edges

    Since #1363, a summary state's row is charged its own copy of the series identity at 64 bytes (state_row_bytes_beside_state, constant SERIES_IDENTITY_BYTES). Every other operator output edge still prices it at 16 bytes, the raw-row price (column_bytes through row_bytes).

    Example: sum by (job) (sum_over_time(m[5m])) over 1,000 series with stored per-series Sum windows carrying job. Each stored state row is charged 144 bytes: 56 of state, 64 for the series identity, 16 for job and 8 for the window time. The FinalizeExactAccumulator right above it outputs the same series identity column, but charges 16 bytes for it.

    The edges that still charge 16 bytes for the identity:

    • FinalizeExactAccumulator and SummaryEstimate outputs;
    • Aggregate (hash aggregate) outputs;
    • TimeRange, Filter, Project and the other pass-through operators;
    • a query-time Scan's output batches (its scanned bytes), and BinaryOp and join outputs.

    The held OneEvaluation output charge in this issue (held_bytes) inherits 64 bytes only for summary nodes, so y16's Finalize and n02's TimeRange are still charged 16.

    Retained raw rows should keep the 16-byte price, since the store shares one copy of each series' labels. That covers a materialized Scan or TimeShift (raw_rows_materialized_at_ingestion) and the raw data a query-time Scan keeps on disk.

    Measured effect of pricing every operator output edge at 64 bytes (identity only; raw retention left at 16), as a plan survey against #1363's head:

    So the change needs its own measurement against the executor's batch bytes. That measurement is this issue's acceptance test. #1363 shares the column price (column_bytes) and the constant, and leaves the edges above to this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingcost-modelCost formulas, statistics, ranking, and selection

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions