Skip to content

depends_on_evaluation_time: SQL table scans and @ end() under @ T are marked independent; TimeRange over @ T is marked dependent #1264

Closed

Description

@zzylol

Problem, by example

GroupProperties::depends_on_evaluation_time decides whether a stored result can serve every evaluation. When it is false, PhysicalCandidate::delivers_on returns true for every execution context, so one stored copy is reused forever. The rule gets three cases wrong. Each example below is derived from the code at 73785fa (head of #1230), not from a planner run.

(a) A SQL table scan is marked independent of the evaluation time.

-- evaluated every minute
SELECT count(*) FROM metrics

The SQL frontend lowers this to Aggregate(count, Scan metrics), with no TimeRange and no now(). depends_on_evaluation_time is false for the Scan and the Aggregate, so a result stored once delivers on every evaluation grid, and the query returns the first minute's count forever. Rows keep arriving into metrics, so the count changes between evaluations.

Today Expected
Scan metrics does not depend depends (data arrives)
count(*) over it stored once, reused on every grid recomputed or refreshed per evaluation

(b) @ <timestamp> hides an inner @ end() / @ start().

max_over_time((rate(x[5m] @ end()))[10m:1m] @ 1700000000)

The PromQL frontend lifts the inner @ end() above rate(x[5m]) as TimeShift { at: End }, and puts the outer @ 1700000000 as TimeShift { at: Timestamp } above the subquery. The TimeShift { at: Timestamp } arm returns false without looking at its input, so the whole subquery is marked independent. In Prometheus the inner @ end() takes precedence: it resolves to the request's end, which is the evaluation time for an instant query. The inner rate therefore changes with the evaluation time.

(c) A missed optimization: @ <timestamp> below a TimeRange.

sum_over_time(x[5m] @ 1700000000)

The result is the same at every evaluation. But the frontend keeps @ T below the TimeRange (TimeRange(5m, TimeShift(@ T, Scan x))), and the TimeRange arm returns true unconditionally, so the group is marked dependent and is never stored once for all evaluations. The same holds for an instant selector x @ 1700000000 (TimeRange(Instant, TimeShift(@ T, Scan x))).

Why it happens

depends_on_evaluation_time in crates/types/src/stages/logical/group.rs (L131-L159):

  • Operator::NonASAP(NonASAPOp::TimeRange { .. }) => return true (L139) — true regardless of whether the input is pinned by @ T (case c).
  • TimeShift { at: Some(AtModifier::Timestamp(_)), .. } => return false (L140-L147) — returns before the input groups are read (case b).
  • Every other operator depends only on now() / time() in its scalar expressions or on its input groups, so a Scan with neither is always false (case a). The test table in crates/types/tests/memo_group_properties.rs (L387, G1 | Scan latency | no) encodes this.

The consumer is PhysicalCandidate::delivers_on in crates/types/src/stages/physical/candidate.rs (L58-L63): a non-chain AcrossEvaluations result whose group does not depend on the evaluation time delivers on every context.

The frontend shapes come from selector and anchored in crates/frontend-promql/src/promql.rs (L1172, L1193: @ start()/@ end() are lifted above the subtree, @ <ts> stays inside it below the TimeRange) and the subquery arm (L230-L247: the subquery's own @/offset become a TimeShift above PromqlSubquery).

PromQL time-series scans are not affected by (a) today: every selector is lowered under a TimeRange (range or instant lookback), which already marks it dependent.

Proposal

  1. Scans of arriving data depend on the evaluation time. A Scan of a source whose data arrives (the data workload's arrival; an unknown arrival is treated as arriving) depends on it, unless a fixed closed interval bounds it: an enclosing @ T, or a literal ts BETWEEN a AND b (both bounds literal). Open question: whether a fixed interval should still be treated as dependent when late data can arrive inside it; the conservative answer is yes unless the source declares no late arrivals.
  2. @ T pins only its own time, not inner anchors. TimeShift { at: Timestamp } is false only when no input group reads @ start()/@ end(), now() or time(). One way: carry two bits per group, "reads the evaluation-time anchor through a TimeRange" (cleared by @ T) and "reads the request start/end or the wall clock" (never cleared by @ T).
  3. A TimeRange over a subtree pinned by @ T does not depend. With the two bits of (2), a TimeRange sets the first bit only when its input is not pinned.

Scope and acceptance

In scope: the rule in group.rs, its test table, and delivers_on if the property splits.

Acceptance (each fails today):

  • SELECT count(*) FROM metrics over an arriving source: the Scan and the Aggregate groups depend on the evaluation time, and a stored result of them does not deliver on a 60 s grid beyond its own times.
  • max_over_time((rate(x[5m] @ end()))[10m:1m] @ 1700000000): the top group depends on the evaluation time.
  • sum_over_time(x[5m] @ 1700000000) and x @ 1700000000: the top group does not depend on it.

Status and related work

Found by a semantic review of #1230 (PR 2 of #1207). Decided 2026-11-10: #1230 keeps only the core data-structure, interface and definition changes; case fixes like this one go to follow-up issues and later PRs. Related: #1251 (evaluate @ end()/@ start() subtrees once per request).

Activity

  1. added
    bugSomething isn't working
    irPre-ASAP or post-ASAP intermediate representation
    windowWindow functions, frames, panes, or temporal semantics
    on Nov 10, 2026
  2. zzylol commented on Nov 10, 2026

    @zzylol
    ContributorAuthor

    Scheduling: this is fixed before #1207 PR 7 (#1237, execution contexts / AcrossEvaluations), because storing a result across evaluations relies on depends_on_evaluation_time being right; a Scan of a source that receives data must not be reused as time-independent.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingirPre-ASAP or post-ASAP intermediate representationwindowWindow functions, frames, panes, or temporal semantics

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions