Skip to content

[Refactor] Executor time walks (counter_window, node_below_shifts, below_time_shifts) should use the shared time_window walks #1292

Open

Description

@zzylol

Example

rate(m[5m] offset 1m) reads m through TimeShift(1m) -> TimeRange(5m). The executor's physical planner works out that window three times, with three walks:

Walk Where What it does
counter_window crates/executor/src/physical_planner/time_window.rs (about line 250) walks down from the counter evaluation to each TimeRange, sums the TimeShift offsets right below it by hand, and returns (offset, lookback) as a bare (i64, i64): here (60_000, 300_000)
node_below_shifts same file (about line 211) follows plan-node TimeShifts down to the first other node; None at an @ shift
below_time_shifts same file (about line 233) the same skip over the restored IR nodes, through a resolver

The shared walks in asap_types::plan::time_window already compute this window: windows_below gives TimeWindow { end_offset_ms: 60_000, width_ms: 300_000 }, and end_offset_below sums the shifts below a node. The cost model, Stage 3, the installer and the rest of the executor (window_read_above, window_below, time_range_window) use them. These three do not.

Concrete maintenance problem

A change to how a window is read (for example, how an @ anchor or a negative offset is treated) has to be made in each copy, and the copies already differ:

  • counter_window ignores an @ shift (it adds no offset and keeps walking); windows_below and end_offset_below return None for one.
  • At a chain reached without passing a TimeRange, counter_window keeps walking below the chain (and finds no range over raw rows); windows_below stops there and reports the chain's tumbling window.
  • node_below_shifts and below_time_shifts are the same rule over two node types.

Behavior that must remain unchanged

  • A counter evaluation still gets the union of the back-to-back windows it reads, and still errs ("ambiguous logical windows") when they are not back to back.
  • time_window_operator, the chain build (stale markers over a time series) and the per-entity summary build still find the Scan or chain below the shifts.
  • Every executor test that runs a shifted or offset window gives the same answers.

Before and after structure

  • counter_window becomes windows_below plus the back-to-back check, and returns a TimeWindow instead of (i64, i64). Where its answer must differ from windows_below (a chain reached without a range, an @ shift), that difference is decided once and named in the shared walk's interface instead of in a copy.
  • node_below_shifts and below_time_shifts become one walk next to end_offset_below (which already stops at the first node that is not a shift), generic over the node type with closures, as the other shared walks are.
  • materialized_view/raw_data.rs check_raw_window_below also re-walks the shifts below a range, but it needs its own error reasons; it can stay.

Why the proposed structure is minimally complex

It deletes code: the three walks become calls to the walks that already exist. No new type is needed, TimeWindow already exists.

Existing unit and end-to-end protection

The executor's offset and counter tests, crates/integration-tests (time_range, the PromQL execution tests) and the shared walks' own tests in asap_types::plan::time_window.

Compatibility or migration impact

None outside the executor: the functions are pub(super) / pub(crate).

Activity

  1. added
    windowWindow functions, frames, panes, or temporal semantics
    on Nov 13, 2026
  2. zzylol commented on Nov 14, 2026

    @zzylol
    ContributorAuthor

    Approved design: one fix for #1291 and #1292

    This comment is posted on both #1291 and #1292. They are fixed together, in one PR that closes both, after #1300 merges. No code is written until then.

    Problem, by example

    Take f01's plan for sum_over_time(m[5m] offset 1h), evaluated every 60 s, with no raw data kept:

    0 Scan(m)                              ingestion time
    1 SummaryTumblingWindowChain(60 s)     ingestion time, stored (view 1), retention 0
    2 TimeShift(1h)                        ingestion time, stored (view 2), retention 3,900,000
    3 TimeRange(5m) -> SummaryMerge -> FinalizeExactAccumulator   query time
    

    Measured on #1300's head (48819b44a), with a hand-built plan of this shape and with the review harness on the forced plans f01–f03 and g01–g04/g13:

    Runtime Refresh of view 2 Query at T Answer
    View by view (view 1 is a stored slot under view 2) refused: "materialized view 1 is read unbounded". With the refusal removed, slot 1 is read over (e, e], an empty request, and view 2 stores 0 rows. refused, or 0 rows
    Both views refreshed in one run (as check_compiles compiles them) compiles view 2 is read over (T − 60 s, T] (from window_below) instead of (T − 65m, T − 60m], and TimeRange(5m) then drops every row [], 0 of 12 series

    What should happen: the refresh at e reads the chain's output keyed in (e − 60 s, e], and the query at T reads view 2's outputs keyed in (T − 65m, T − 60m]. The result is Σ m over (T − 65m, T − 60m], which is equal to the exact answer.

    Why it happens

    1. Time semantics are implemented in several places, and they disagree.
    2. Width 0 has no stated meaning.
      • The planner reads the chain under a stored shift through {0, 0}, meaning "the newest output", and gives the chain retention 0. That relies on an unwritten rule: the store always keeps the newest output.
      • The executor turns {0, 0} into the empty request (e, e].
    3. The failure is silent.
      • InMemoryDataStore::read_materialized_view returns no rows, and no error, for (e, e].
      • check_compiles accepts the plan.

    Proposal (approved)

    1. Every time window is over the time keys of stored outputs.

    • Keys. A chain's output is keyed by its window end. Any other stored output is keyed by the end of the interval it summarizes.
    • Reads. A read {o, w} at evaluation time t means the outputs whose key lies in (t − o − w, t − o].
    • Width 0 is no longer allowed. "The newest output" is {o, output_interval}, which for a chain of width c is {o, c}.
    • Cost.
      • Chains are unchanged: div_ceil(c, c) is 1 retained window, the same as with retention 0.
      • A non-chain stored result that is read only at its current time keeps one more output interval. That is the correct cost: today a reader at an offset o is given retention o, and the store, which drops outputs keyed at or before newest − retention, drops the output keyed at exactly t − o.

    2. The planner computes each reader's window once and writes it into the plan.

    pub struct PhysicalASAPDAGEdge {
        pub producer: PhysicalASAPNodeId,
        pub consumer: PhysicalASAPNodeId,
        pub input_index: usize,
        /// Which of the producer's time-indexed outputs this consumer reads, relative to the
        /// consumer's evaluation time t: time keys in (t - ends_before_evaluation_ms - length_ms,
        /// t - ends_before_evaluation_ms]. None = every output (or the producer is not time-indexed).
        pub read_window: Option<TimeWindow>,
    }

    Example, with the plan above:

    Edge read_window Meaning
    chain → stored offset 1h shift {0, 60_000} at refresh e, the chain output keyed in (e − 60 s, e]
    shift → TimeRange(5m) {3_600_000, 300_000} at query T, the shift outputs keyed in (T − 65m, T − 60m]

    3. Fail loudly.

    Scope and acceptance

    In scope (one PR):

    Acceptance:

    • The f01-shaped executor test answers the direct sums in both runtimes (view by view, and both views in one run). Today it is refused, or answers [].
    • Forced f01–f03 and g01–g04/g13 match exact on all 12 series. They cover:
      • offset < range: 90 s on [5m];
      • offset = range: 5m on [5m];
      • offset > range: 1h on [5m], and 2m on [1m];
      • phases 15, 20 and 30 s;
      • other readers of the same chain.
    • A read whose window is not covered by the stored outputs errors, and does not return 0 rows.
    • check_compiles refuses an edge whose read_window reaches past its producer's retention.
    • An execution-vs-exact test of a stored offset shift runs in CI.

    Status and related work

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    refactorwindowWindow functions, frames, panes, or temporal semantics

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions