Skip to content

Rollup over chain windows maps keys off by one (window_end), so it is never offered #1267

Open

Description

@zzylol

Problem, by example

A chain of 1-minute KLL states grouped by (job, host) over m, and a rollup by job over its last 5 windows:

C = SummaryTumblingWindowChain[1m, KLL, by (job, host)](TimeRange…(Scan m))
    schema: (window_end, job, host, state)        -- columns 0, 1, 2, 3
R = SummaryMerge[group_by = [1]](TimeRange(5m, C))   -- regroup by column 1 = job

R should cover what SummaryAgg[KLL, by (job)](TimeRange(5m, Scan m)) covers, so a quantile by (job) (…) query could read the chain. Today R is always rejected (NotAGroupKey), so the reuse is missed. The answer is never wrong: the rollup simply is not offered. Derived from the code at 73785fa (head of #1230), not from a run.

Why it happens

A chain's state schema puts window_end first: (window_end, keys…, state) (chain_schema in crates/types/src/ir/operator/asap.rs L772-L797, insert(0, Field::new("window_end", …)) at L791). A TimeRange over the chain keeps that schema. A plain SummaryAgg state is (keys…, state).

Both places that map a rollup key (a column of the input state) to the build's input column index the build's group keys by the state column directly:

  • regrouped in crates/types/src/stages/logical/aggregate_coverage.rs (L332-L353): Reduction::Reduce(by) … => by.keys().get(key), and Reduction::PerEntity => Some(key).
  • rolled_up_build in crates/types/src/ir/operator/asap.rs (L716-L722): the same mapping.

For the example, key 1 (job) maps to by.keys()[1] (host); regrouped then compares the state column's name (job) with the input column's name (host), finds them different and returns NotAGroupKey. The last key maps past the end of by.keys() and returns None. A per-entity chain is off by one the same way (the chain drops the build's time column and inserts window_end at 0). So over chain windows the name check in regrouped always rejects. rolled_up_build has no name check; it is reached only for rollups already in the memo (FinalizeExactAccumulator and top-k schema derivation, L341 and L1008), which regrouped keeps out today, so it is not wrong today but would be once regrouped is fixed alone.

No test covers a rollup over chain windows.

Proposal

Map a state column to its build's group key by position among the state's key columns, not by raw column index: skip a time-indexed state's window_end (the state schema's time_index when the build is a chain). Put the mapping in one helper used by both regrouped and rolled_up_build, so they cannot diverge again.

Scope and acceptance

In scope: the key mapping in regrouped and rolled_up_build, and tests.

Acceptance (fails today):

  • In crates/types/tests/memo_aggregate_coverage.rs: R above gets a coverage whose definition equals that of SummaryAgg[KLL, by (job)](TimeRange(5m, …)), and the same for a per-entity chain regrouped by one label.
  • A rollup by a key that is not a group key of the chain (e.g. window_end) is still rejected.
  • FinalizeExactAccumulator over R derives the by (job) schema.

Status and related work

Found by a semantic review of #1230 (PR 2 of #1207). Decided 2026-11-10: #1230 keeps only the core data-structure, interface and definition changes; case fixes go to follow-up issues and later PRs.

Activity

  1. added
    bugSomething isn't working
    rollupFine-to-coarse aggregation and summary reuse
    subpopulationsGrouped, hierarchical, or multi-subpopulation summaries
    windowWindow functions, frames, panes, or temporal semantics
    on Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingrollupFine-to-coarse aggregation and summary reusesubpopulationsGrouped, hierarchical, or multi-subpopulation summarieswindowWindow functions, frames, panes, or temporal semantics

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions