Skip to content

[Performance] No candidate filters a finished grouped result on a group key; filtered queries re-aggregate filtered states instead #891

Open

Description

@zzylol

Example

An observability recording rule and a pod-specific drill-down read the same CPU counter and 5-minute window:

sum by (cluster, namespace, pod, container) (rate(container_cpu_usage_seconds_total[5m]))
sum by (pod) (rate(container_cpu_usage_seconds_total{namespace="production",pod="checkout-123"}[5m]))

Both queries get valid plans, but no candidate reuses the broad query's rate state for the filtered query. Removing the namespace/pod predicates enables selected sharing of the scan, time range, rate build, and rate finalization.

Tested on origin/main at c3be284c97c1f7ecdc490d78ddecf3e5f78c2007 on 2026-10-22. This is a concrete reproducer related to #664's filter/coverage-selection work, rather than a report of incorrect CPU values.

User or operational impact

A recording rule maintains CPU data across pods, then an SRE agent asks about one pod. The planner cannot exploit the overlap in this tested workload, so it selects independent rate computations. With raw retention disabled, it still maintains two independent 10-pane chains (20 stored outputs).

Metric and target

Target: enumerate at least one legal candidate that reuses per-series rate state and applies the pod/namespace selection before the filtered query's final aggregation. Stage 3 should compare that candidate with independent computation; it need not always select sharing.

No measured latency or memory improvement is claimed. The immediate acceptance metric is candidate coverage and successful execution of the shared candidate.

Representative workload and environment

Synthetic planning facts: 1,000 counter series; samples every 15 seconds; ingestion at 1,000/15 samples per second; both queries repeat every 30 seconds, are predictable, and require exact results.

Save this exact tested workload as cpu.json:

{
  "query_workload": {
    "language": "promql",
    "repeating_queries": [
      {
        "query": "sum by (cluster, namespace, pod, container) (rate(container_cpu_usage_seconds_total[5m]))",
        "demand": {
          "fixed_interval": 30000
        },
        "requirements": {
          "accuracy": {
            "max_error": 0.0
          }
        },
        "predictability": {
          "predictable": {
            "known_at": null
          }
        }
      },
      {
        "query": "sum by (pod) (rate(container_cpu_usage_seconds_total{namespace=\"production\",pod=\"checkout-123\"}[5m]))",
        "demand": {
          "fixed_interval": 30000
        },
        "requirements": {
          "accuracy": {
            "max_error": 0.0
          }
        },
        "predictability": {
          "predictable": {
            "known_at": null
          }
        }
      }
    ]
  },
  "data_workload": {
    "arrival": {
      "ingesting": {
        "rows_per_second": 66.66666666666667
      }
    },
    "sources": {
      "container_cpu_usage_seconds_total": {
        "sample_interval": 15000,
        "series_count": 1000,
        "metric_type": "counter"
      }
    }
  }
}
cargo run -p asap-devtools --bin stage_pipeline -- \
  --workload cpu.json --enforce-accuracy-guarantees --out cpu.default.json
cargo run -p asap-devtools --bin stage_pipeline -- \
  --workload cpu.json --enforce-accuracy-guarantees --exhaustive \
  --max-combinations 10000 --max-displayed 100 --out cpu.exhaustive.json

Control: replace only the second query with sum by (pod) (rate(container_cpu_usage_seconds_total[5m])) and rerun. This was tested and selects shared rate state.

Current baseline

  • Default search separates the filtered workload into two components and selects P1+P1 without cross-query sharing.
  • Exhaustive search enumerates 4 logical / 9 physical candidates and selects independent P1; none of its logical candidates offers cross-query sharing. This is not only a default-search component split.
  • Unfiltered control selects P8-m1, sharing scan, time range, SummaryAgg rate build, and FinalizeExactAccumulator.
  • With --deployment-capabilities '{"raw_data_retained":false}', the original filtered workload selects two independent ingestion-time rate chains, not one shared chain.

Correctness constraints

Retain namespace, pod, cluster, container, and full per-series identity as needed. Apply the filtered consumer's label selection without restricting the broad consumer. Handle counter resets per original series before summing; do not merge independent counters and then compute their rate. Refuse reuse when retained labels or state coverage cannot answer the requested filter.

Measurement method and acceptable variance

Use the fixed workload above with exact accuracy checks. Inspect all exhaustive logical candidates for reuse and compare default/exhaustive selection costs. Candidate coverage should be deterministic; runtime speed is not an acceptance threshold in this report.

Add end-to-end fixtures with multiple pods/namespaces and counter resets. Require the shared candidate to agree with independent evaluation for both queries, including a nonexistent pod filter. These numerical checks remain follow-up work; the current evidence covers planning and executor compilation only.

Related: #664 (filter push-down and coverage selection), #580 (sharing/coverage parity).

Activity

  1. added
    enhancementNew feature or request
    promqlPromQL front-end lowering (L1→L2)
    on Oct 22, 2026
  2. zzylol commented on Oct 23, 2026

    @zzylol
    ContributorAuthor

    Deferred until the memo search (#879, #888) lands. This sharing/reuse rule will be added as a memo transformation rather than to the legacy search.

  3. changed the title [-][Performance] Filtered pod CPU query cannot reuse broader per-series rate state[/-] [+][Performance] Filtered pod CPU query reads the grouped rate by filtering states, not the finished result[/+] on Nov 16, 2026
  4. zzylol commented on Nov 16, 2026

    @zzylol
    ContributorAuthor

    Remaining gap (on the #1207 stack)

    Example. The planner fuzz case for this issue (crates/devtools/tests/planner_fuzz.rs, issue_891_filtered_rate_reads_the_grouped_rate): W = sum by (cluster, namespace, pod, container) (rate(container_cpu_usage_seconds_total[5m])), plus a query that filters that result on a group key.

    cost per second
    W alone 0.008500
    W plus the filtered query, as the planner plans it 0.008600 (+0.000100)
    W plus the filtered query, as the witness plans it 0.008535 (+0.000035)
    gap, planner minus witness 0.000065
    fuzz tolerance (1% of W) 0.000086

    What changed. The filtered query now reuses the grouped rate, so the relation holds and the test passes. But the planner's plan adds 0.000100/s where the witness adds 0.000035/s, and the 0.000065/s gap is under the tolerance by only 0.000021/s.

    Cause. The two plans read the grouped rate at different points:

    • The planner's plan filters the grouped sum's states, then runs SummaryMerge and FinalizeExactAccumulator for the filtered query.
    • The witness filters W's finished result, a Filter over rows W already computes. That costs about a third as much.

    No candidate filters the finished result on a group key.

    Fix direction. Add a candidate that applies a group-key filter to the finished grouped result when the filter's columns are all group keys, so Stage 3 can choose it. The test asserts related_cost_per_second - witness.cost_per_second <= tolerance_per_second. Once the candidate exists, tighten that assertion to equality within rounding.

  5. zzylol commented on Nov 16, 2026

    @zzylol
    ContributorAuthor

    Measured gap under the refit costs (stage-printer port, #1239 head d744bd7ae)

    With the measured operator costs now on main, the gap is no longer inside the fuzz tolerance. Both planner_fuzz cases fail, and both show the same missing candidate.

    Example 1 (this issue, exact rate). W = sum by (cluster, namespace, pod, container) (rate(container_cpu_usage_seconds_total[5m])); q' = the same query with {cluster="cluster1"}.

    plan for q' cost added to W, per second
    planner filter the grouped rate's states, then Aggregate sum again +0.0925 (0.514207 − 0.421706)
    witness Filter over W's finished result +0.00004 (0.421744 − 0.421706)
    fuzz tolerance (1% of the related plan) 0.0051

    Example 2 (grouped variant of #902, KLL). W = p90 per city over 5 min; q' = the same with AND city = 'city1' GROUP BY city.

    plan for q' cost added to W, per second
    planner Filter over the merged KLL states, then a second SummaryMerge and SummaryEstimate +0.080 (1.969967 − 1.889801)
    witness Filter over W's finished estimate +0.00001 (1.889813 − 1.889801)
    fuzz tolerance 0.0197

    Why this is a missing candidate, not a costing mismatch. The MILP objective equals plan_cost for the selected plan, and the witness plan prices at the lower cost under the same cost model. So Stage 3 would pick the witness if the memo offered it. It does not: the planner has no candidate that applies a group-key filter to the finished grouped result. It only filters the states below the final aggregation, then re-merges or re-aggregates them.

    Fix (unchanged). Add a candidate that applies a filter to the finished grouped result when every filtered column is a group key (and the filter keeps the input's guarantee, which the witness already checks: key_filter_keeps_input_guarantee: true). Then flip both tests to assert the planner's added cost equals the witness's within rounding.

    Retitled to name the general gap; the pod CPU workload above stays the first example.

  6. changed the title [-][Performance] Filtered pod CPU query reads the grouped rate by filtering states, not the finished result[/-] [+][Performance] No candidate filters a finished grouped result on a group key; filtered queries re-aggregate filtered states instead[/+] on Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    CSEenhancementNew feature or requestpromqlPromQL front-end lowering (L1→L2)

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions