Skip to content

Non-finite values across the executor should follow PromQL #1311

Open

Description

@zzylol

Example

rate(requests_total[5m]) over a counter that has one NaN sample. The NaN is an ordinary value written by an exporter, not a stale marker:

requests_total{instance="a"}  t=0s: 10   t=15s: NaN   t=30s: 14   t=45s: 20

A second case. sum by (job) (q(data) / on(instance, job, host, region) q(neg)) divides by 0 when neg's quantile is 0. PromQL then gives ±Inf or NaN, and sum adds them up. #1309 makes the exact Sum, Count, Min and Max accumulators accept those values. Every other path below still refuses them or handles them differently.

Stale markers are not part of this issue. A Prometheus stale marker is the NaN with bits 0x7ff0000000000002 (STALE_MARKER, operators/series_window.rs:18). It marks a series as gone and is not a sample, so it must be dropped before accumulation and never counted, summed or sketched. The executor drops it today:

Any change here must keep that: only non-stale NaN and ±Inf reach the paths below.

Current behaviour per path (at 4a37273cb)

Path Code Non-finite input today PromQL
Default AccumulatorUpdater::validate_single_input (KLL, HLL, KMV, Hydra-KLL, UnivMon updaters; exact accumulators before #1309) factory/mod.rs:77-83, :92-94 error accumulator input must be finite quantile/count over NaN or ±Inf is defined (for example quantile_over_time sorts NaN and ±Inf)
Exact Sum/Count/Min/Max, updater and state-column paths factory/mod.rs:92, state_column_updater/mod.rs:318 error (Count in the state-column path already counts NaN) fixed in #1309: IEEE sum, NaN-skipping min/max, count counts it
Exact Rate/Increase (counters), both paths same lines; summary_kernels/increase.rs has no NaN handling error defined (see the example)
DDSketch factory/mod.rs:228-240 error (outside its domain) a sketch cannot hold NaN or ±Inf; the planner or executor needs a rule (for example, count them beside the sketch)
Count-Min / Count Sketch weights factory/mod.rs:357 error — (weights are counts)
HydraCms weights hydra_cms.rs:207 error —
Distinct-count items (HLL, KMV) factory/mod.rs:65-67 error count_values/distinct over NaN are defined
UnivMon samples univmon.rs:107-109 error defined
Weighted-frequency identities weighted_frequency.rs:52-54 error —
Columnar frequency aggregates (L2, entropy) operators/aggregate/columnar.rs:337 error frequency aggregate requires a finite floating identity defined
Summary-update expression Expression::FiniteFloat64 expressions/mod.rs:332-334 error summary update must be finite (only a test constructs it today) —
Ingestion-time aligned arithmetic (BinaryOp at ingestion time, physical_planner/mod.rs:892-931) operators/aligned_binary.rs:145-146, 166-174 error on a non-finite input or result x / 0 is ±Inf, 0 / 0 NaN
Checked divisions (checked_relative_division, checked_finite_division) expressions/arithmetic.rs:36-46 error by design (the planner asks for it) —
Query-time arithmetic and temporal aggregates expressions/arithmetic.rs:2, operators/aggregate/temporal.rs:136-147 IEEE results kept; min/max skip NaN matches

Why

Each numeric kernel validates its input with finite_input by default. Ingestion-time paths fail the stream rather than drop an update, because a dropped update corrupts maintained state (aligned_binary's doc). Nothing yet says what PromQL result each kernel should keep for a non-finite value, so each kernel refuses it.

Proposal

Per path, either follow PromQL or make the refusal a planner-visible rule:

  1. Counters: feed NaN and ±Inf through IncreaseAccumulator with Prometheus's comparisons. A NaN is never a reset, and a NaN or ±Inf endpoint propagates.

  2. Aligned arithmetic: keep IEEE results (±Inf, NaN) instead of failing. A non-finite result is then an input to the accumulator below, and the rules in this list decide what that accumulator does with it.

  3. Sketches (DDSketch, KLL, HLL, KMV, UnivMon): a sketch cannot represent NaN or ±Inf. Two options:

    • (a) keep the refusal, and have the planner not choose a sketch where the input can be non-finite (a division whose denominator can be 0);
    • (b) count non-finite values beside the sketch, so a quantile can still return ±Inf or NaN as PromQL does.

    Option (a) restricts the planner, so (b) is preferred unless it is too costly.

  4. Frequency identities and weights: a NaN identity can be one item (as count_values treats it). Weights stay finite (they are counts).

  5. FiniteFloat64: remove it if no plan constructs it, or apply the same rule as the kernel it feeds.

Scope and acceptance

  • In scope: every path in the table except checked divisions, which refuse non-finite values by design.
  • Out of scope: stale markers. They are dropped before accumulation, and a regression test must keep that: a stale marker inside a window is not counted by count_over_time and not summed by sum_over_time.
  • Acceptance: for each path, a unit test feeds NaN and ±Inf and gets the PromQL result, or a planner rule keeps such a plan from being chosen. Both examples above execute against the exact reference: rate over a NaN counter sample, and a sum over q / 0 at ingestion time.

Status and related work

Activity

  1. added
    bugSomething isn't working
    promqlPromQL front-end lowering (L1→L2)
    accuracyAccuracy requirements, guarantees, and propagation
    on Nov 14, 2026
  2. zzylol commented on Nov 15, 2026

    @zzylol
    ContributorAuthor

    Add avg_over_time

    avg_over_time still fails on non-finite samples when no raw data is retained. Pass 1 rewrites it as an exact sum ÷ an exact count with checked_finite_division: true (crates/stage1-pass1-per-computation-candidates/src/candidates/measure_output.rs). Its comment says the original query runs when an operand or the result is not finite, but the executor's checked division fails instead.

    Example, from the #1309 review harness with NaN and ±Inf samples:

    avg_over_time(data[5m])
    

    In the same runs, sum_over_time, max_over_time, min_over_time, count_over_time and sum by (job) (max_over_time(...)) match hand-computed PromQL results. rate, increase, and sum/avg of rate over a non-positive counter still fail at ingestion, as the table above says.

    Proposal: add a row for the avg_over_time rewrite (sum ÷ count with checked_finite_division). Either the rewrite's division keeps IEEE results when an operand is non-finite (NaN ÷ n is NaN, +Inf ÷ n is +Inf), or the planner does not choose the rewrite where the input can be non-finite. A count is never 0 for a window that has samples.

  3. zzylol commented on Nov 17, 2026

    @zzylol
    ContributorAuthor

    irate now has an exact state too (ScalarState::LastTwoSamples, PR #1241), and it refuses non-finite samples the same way, so this issue covers rate, increase and irate.

    Example, from a run of the executor against Prometheus 3.16.0 (promtool test rules) on #1241's branch. Series x{instance="e_nan"} has samples 1, 2, NaN, 4, 5, NaN, 6 at 200 s, 210 s, …, 260 s (the NaNs are ordinary values, not stale markers). At 300 s:

    Query Prometheus 3.16.0 Executor (exact plan)
    rate(x[1m]) 0.08 SummaryAgg: accumulator input must be finite
    increase(x[1m]) 4.8 same error
    irate(x[1m]) NaN (6 − NaN) same error

    One NaN or ±Inf sample in any series fails the whole query. A series x{instance="f_inf"} with an Inf sample answers +Inf in Prometheus for all three and fails the same way here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    accuracyAccuracy requirements, guarantees, and propagationbugSomething isn't workingpromqlPromQL front-end lowering (L1→L2)

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions