Repository navigation
Description
Example
rate(requests_total[5m]) over a counter that has one NaN sample. The NaN is an ordinary value written by an exporter, not a stale marker:
requests_total{instance="a"} t=0s: 10 t=15s: NaN t=30s: 14 t=45s: 20
- Today: the exact
Rateaccumulator refuses the NaN and fails the stream withaccumulator input must be finite. The unit testexact_accumulators_take_non_finite_values_as_promql_doesin fix(accuracy): relative error through negation, abs, set operators and relabels; readouts compose only through error-carrying inputs; exact accumulators take non-finite values and reject stale markers; finalize sign of counters #1309 asserts this refusal forRateandIncreaseover[1.0, NaN]. - Expected: PromQL semantics. Prometheus's
extrapolatedRatetakeslast − firstplus counter-reset corrections over the float samples. A NaN compares false, so a NaN in the middle is never treated as a reset, and a NaN first or last sample makes the result NaN. The exact result for this series has to be pinned by a test against Prometheus's evaluation; the executor must not fail the query.
A second case. sum by (job) (q(data) / on(instance, job, host, region) q(neg)) divides by 0 when neg's quantile is 0. PromQL then gives ±Inf or NaN, and sum adds them up. #1309 makes the exact Sum, Count, Min and Max accumulators accept those values. Every other path below still refuses them or handles them differently.
Stale markers are not part of this issue. A Prometheus stale marker is the NaN with bits 0x7ff0000000000002 (STALE_MARKER, operators/series_window.rs:18). It marks a series as gone and is not a sample, so it must be dropped before accumulation and never counted, summed or sketched. The executor drops it today:
- in range windows (series_window.rs:297, :303);
- in instant snapshots (current_series.rs:98);
- in
TimeRange(time_range.rs:210); - in tumbling-window chains over a time-series scan (physical_planner/mod.rs:483).
Any change here must keep that: only non-stale NaN and ±Inf reach the paths below.
Current behaviour per path (at 4a37273cb)
| Path | Code | Non-finite input today | PromQL |
|---|---|---|---|
Default AccumulatorUpdater::validate_single_input (KLL, HLL, KMV, Hydra-KLL, UnivMon updaters; exact accumulators before #1309) |
factory/mod.rs:77-83, :92-94 | error accumulator input must be finite |
quantile/count over NaN or ±Inf is defined (for example quantile_over_time sorts NaN and ±Inf) |
| Exact Sum/Count/Min/Max, updater and state-column paths | factory/mod.rs:92, state_column_updater/mod.rs:318 | error (Count in the state-column path already counts NaN) | fixed in #1309: IEEE sum, NaN-skipping min/max, count counts it |
| Exact Rate/Increase (counters), both paths | same lines; summary_kernels/increase.rs has no NaN handling | error | defined (see the example) |
| DDSketch | factory/mod.rs:228-240 | error (outside its domain) | a sketch cannot hold NaN or ±Inf; the planner or executor needs a rule (for example, count them beside the sketch) |
| Count-Min / Count Sketch weights | factory/mod.rs:357 | error | — (weights are counts) |
| HydraCms weights | hydra_cms.rs:207 | error | — |
| Distinct-count items (HLL, KMV) | factory/mod.rs:65-67 | error | count_values/distinct over NaN are defined |
| UnivMon samples | univmon.rs:107-109 | error | defined |
| Weighted-frequency identities | weighted_frequency.rs:52-54 | error | — |
| Columnar frequency aggregates (L2, entropy) | operators/aggregate/columnar.rs:337 | error frequency aggregate requires a finite floating identity |
defined |
Summary-update expression Expression::FiniteFloat64 |
expressions/mod.rs:332-334 | error summary update must be finite (only a test constructs it today) |
— |
Ingestion-time aligned arithmetic (BinaryOp at ingestion time, physical_planner/mod.rs:892-931) |
operators/aligned_binary.rs:145-146, 166-174 | error on a non-finite input or result | x / 0 is ±Inf, 0 / 0 NaN |
Checked divisions (checked_relative_division, checked_finite_division) |
expressions/arithmetic.rs:36-46 | error by design (the planner asks for it) | — |
| Query-time arithmetic and temporal aggregates | expressions/arithmetic.rs:2, operators/aggregate/temporal.rs:136-147 | IEEE results kept; min/max skip NaN | matches |
Why
Each numeric kernel validates its input with finite_input by default. Ingestion-time paths fail the stream rather than drop an update, because a dropped update corrupts maintained state (aligned_binary's doc). Nothing yet says what PromQL result each kernel should keep for a non-finite value, so each kernel refuses it.
Proposal
Per path, either follow PromQL or make the refusal a planner-visible rule:
-
Counters: feed NaN and ±Inf through
IncreaseAccumulatorwith Prometheus's comparisons. A NaN is never a reset, and a NaN or ±Inf endpoint propagates. -
Aligned arithmetic: keep IEEE results (±Inf, NaN) instead of failing. A non-finite result is then an input to the accumulator below, and the rules in this list decide what that accumulator does with it.
-
Sketches (DDSketch, KLL, HLL, KMV, UnivMon): a sketch cannot represent NaN or ±Inf. Two options:
- (a) keep the refusal, and have the planner not choose a sketch where the input can be non-finite (a division whose denominator can be 0);
- (b) count non-finite values beside the sketch, so a quantile can still return ±Inf or NaN as PromQL does.
Option (a) restricts the planner, so (b) is preferred unless it is too costly.
-
Frequency identities and weights: a NaN identity can be one item (as
count_valuestreats it). Weights stay finite (they are counts). -
FiniteFloat64: remove it if no plan constructs it, or apply the same rule as the kernel it feeds.
Scope and acceptance
- In scope: every path in the table except checked divisions, which refuse non-finite values by design.
- Out of scope: stale markers. They are dropped before accumulation, and a regression test must keep that: a stale marker inside a window is not counted by
count_over_timeand not summed bysum_over_time. - Acceptance: for each path, a unit test feeds NaN and ±Inf and gets the PromQL result, or a planner rule keeps such a plan from being chosen. Both examples above execute against the exact reference: rate over a NaN counter sample, and a sum over
q / 0at ingestion time.
Status and related work
- Follow-up of Relative error through negation/abs/unless/label_replace; non-finite exact-sum input; finalize sign of rate accumulators #1307 I1, decided 2026-11-14. fix(accuracy): relative error through negation, abs, set operators and relabels; readouts compose only through error-carrying inputs; exact accumulators take non-finite values and reject stale markers; finalize sign of counters #1309 makes exact Sum, Count, Min and Max follow PromQL; everything else is left to this issue.
- Related: fix(accuracy): relative error through negation, abs, set operators and relabels; readouts compose only through error-carrying inputs; exact accumulators take non-finite values and reject stale markers; finalize sign of counters #1309.
Activity
zzylol commentedon Nov 15, 2026
Add
avg_over_timeavg_over_timestill fails on non-finite samples when no raw data is retained. Pass 1 rewrites it as an exact sum ÷ an exact count withchecked_finite_division: true(crates/stage1-pass1-per-computation-candidates/src/candidates/measure_output.rs). Its comment says the original query runs when an operand or the result is not finite, but the executor's checked division fails instead.Example, from the #1309 review harness with NaN and ±Inf samples:
{"raw_data_retained": false}. One series' window holds a NaN, or both +Inf and -Inf.sum_over_timeis NaN, so the average is NaN). A window holding only +Inf gives +Inf.accumulator input must be finite.SeriesBinary failed: invalid DAG: unsupported binary operation or invalid checked-division domain(crates/executor/src/expressions/arithmetic.rs: a checked division refuses non-finite operands and quotients).In the same runs,
sum_over_time,max_over_time,min_over_time,count_over_timeandsum by (job) (max_over_time(...))match hand-computed PromQL results.rate,increase, andsum/avgofrateover a non-positive counter still fail at ingestion, as the table above says.Proposal: add a row for the
avg_over_timerewrite (sum ÷ count withchecked_finite_division). Either the rewrite's division keeps IEEE results when an operand is non-finite (NaN ÷ n is NaN, +Inf ÷ n is +Inf), or the planner does not choose the rewrite where the input can be non-finite. A count is never 0 for a window that has samples.zzylol commentedon Nov 17, 2026
iratenow has an exact state too (ScalarState::LastTwoSamples, PR #1241), and it refuses non-finite samples the same way, so this issue coversrate,increaseandirate.Example, from a run of the executor against Prometheus 3.16.0 (
promtool test rules) on #1241's branch. Seriesx{instance="e_nan"}has samples 1, 2, NaN, 4, 5, NaN, 6 at 200 s, 210 s, …, 260 s (the NaNs are ordinary values, not stale markers). At 300 s:rate(x[1m])SummaryAgg: accumulator input must be finiteincrease(x[1m])irate(x[1m])One NaN or ±Inf sample in any series fails the whole query. A series
x{instance="f_inf"}with anInfsample answers+Infin Prometheus for all three and fails the same way here.