Repository navigation
Description
Example
With #1302, avg by (job) (quantile_over_time(0.9, data[300s])) over non-negative data plans a DDSketch chain when raw data is not retained. Negating the quantile does not:
{"query_workload": {"language": "promql", "repeating_queries": [{
"query": "sum by (job) (-quantile_over_time(0.9, data[300s]))",
"demand": {"fixed_interval_at": {"interval": 1000, "evaluation_phase": 0}},
"requirements": {"accuracy": {"max_error": 0.05, "failure_probability": 0.3, "accuracy_scope": "whole_result"}},
"predictability": {"predictable": {"known_at": null}}}]},
"data_workload": {"arrival": {"ingesting": {"rows_per_second": 10000.0}},
"sources": {"data": {"series_count": 1000, "metric_type": "gauge", "sample_interval": 100,
"column_signs": {"value": "non_negative"}}}}}| Deployment | Today (#1302 head) | Expected |
|---|---|---|
{"raw_data_retained": false} |
NoValidCandidate: the selection MILP is infeasible |
a DDSketch plan, as for the un-negated query |
| default (raw retained) | exact sum over raw data (2.68e-7/s in the review harness) |
a DDSketch plan |
The input of sum is known to be NonPositive (Stage 0 column_signs), so the #1302 sign rule would accept relative error there. The edge is cut one step lower, at the negation.
This issue collects three findings from the #1302 review. They are independent; only the first one blocks plans.
L1. Negation, abs, unless and label_replace keep relative error but have no relative contract
Current behavior. For non-negative data, neg and mixed (undeclared), the review planned these queries without raw data. All are NoValidCandidate; with raw data they plan exact:
| Query | Why the value's relative error is kept |
|---|---|
avg by (job) (-quantile_over_time(0.9, data[5m])) |
` |
avg by (job) (abs(quantile_over_time(0.9, mixed[5m]))) |
` |
avg by (job) (quantile_over_time(0.9, data[5m]) unless quantile_over_time(0.9, neg[5m])) |
unless keeps or drops whole left-hand series by labels; kept values are copied |
avg by (job) (label_replace(quantile_over_time(0.9, data[5m]), "x", "$1", "instance", "(.*)")) |
the value is copied; only a label changes |
Cause.
-xandabs(x): in aProject,keeps_error(crates/accuracy/src/error_contract/mod.rs:257) acceptsone_lipschitzitems only in absolute error (line 263). In relative error it accepts onlyscaled_by_a_constant(line 264). Negation andabsare 1-Lipschitz and keep relative error. High confidence.unless: a setBinaryOpmatches none oferror_contract'sBinaryOparms (onlyArithmeticis handled, line 116), so it falls to_ => Noneand has no contract in any metric. High confidence from the code.label_replace: the PromQL frontend lowers it to a per-series relabel that keeps the value (crates/frontend-promql/src/promql.rs:45). The review found that the relative edge is cut there. I have not traced which operator or check cuts it. Medium confidence.
Proposal.
- Accept negation and
absof a column in relative error too, with β = 1. - Give
unless(andand, which keeps left-hand values) β = 1 on the left input in every metric. The right input is read only for its labels, so it gets β = 0. - Find where the relabel is cut, and give it β = 1 in every metric.
Acceptance. The example above plans a DDSketch chain without raw data. A unit test in error_contract/tests.rs covers each operator.
I1. An exact sum accumulator rejects a non-finite input
Example. sum by (job) (quantile_over_time(0.9, data[5m]) / on(instance, job, host, region) quantile_over_time(0.9, neg[5m])), with data non-negative and neg non-positive, and raw data not retained. With #1302 it plans DDSketch readouts with an ingestion-time exact Sum accumulator chain (2.59e-9/s). With near-zero data the divisor can be 0, and execution fails:
stream: node 13 (SummaryAgg) failed: operator failed: accumulator input must be finite
The exact reference evaluation returns NaN for that group instead.
Cause. finite_input (crates/executor/src/summary_kernels/factory/mod.rs:77-83) rejects non-finite values for numeric accumulators. This is an executor limitation, not the error contract: the same query with raw data retained executes. High confidence.
Proposal. Let the exact Sum (and Count, Min, Max) accumulators accept ±Inf and NaN with PromQL semantics: a sum that sees NaN is NaN, and a sum that sees ±Inf is ±Inf or NaN. Do not fail the stream.
Acceptance. An executor test feeds an exact sum accumulator a NaN and an Inf and gets the PromQL result. The query above executes with near-zero data and matches the reference.
I3. Finalizing an increase or rate accumulator takes the weight's sign
Example. rate(neg[5m]) over a neg declared non_positive. measure_sign gives the logical Aggregate{Rate} the sign Unknown, which is correct: a counter-reset-aware delta over non-positive values can have either sign. The physical FinalizeExactAccumulator over an ExactAggregate(Rate) state gives its output the sign of the update weight, NonPositive.
Cause. FinalizeExactAccumulator uses readout_columns(child, output, |state, _| if counts { NonNegative } else { state }) (crates/types/src/stages/logical/group/column_properties.rs:466-478). Every non-count accumulator kind passes the state's recorded sign through, including Increase, Rate and IRate. measure_sign (column_properties.rs:615) gives those kinds a sign only for a NonNegative input. High confidence from the code.
Impact. None today. GroupProperties are derived once from a group's first expression, and the finalize output is exact, so the #1302 rule never reads this sign. It would matter if a group were ever created from a finalize, or if its sign were read for an approximate input.
Proposal. Derive the finalize's sign per accumulator kind with the same rule as measure_sign: Sum, Min and Max keep the state's sign; Count is non-negative; Increase, Rate and IRate are non-negative only for a non-negative state, else Unknown.
Acceptance. A memo_group_properties test: a finalize over an ExactAggregate(Rate) state of a non-positive source is Unknown.
Status and related work
- Found in the review of fix(accuracy): relative error through sum and avg of values of one sign #1302 (the sign rule for relative error through
sum/avg), part of Tracking: implement the optimization formulation (memo candidates, properties, edges y_{u,c,i}, MILP) as named code per stage #1207. - L1 and I3 predate fix(accuracy): relative error through sum and avg of values of one sign #1302. I1 is reachable only since fix(accuracy): relative error through sum and avg of values of one sign #1302, which first gives such queries a sketch plan.
- Evidence comes from the review harness: Stage 0 sign dumps, and planning and execution against the exact reference with uniform, skewed and near-zero data.
Activity
fix(accuracy): relative error through sum and avg of values of one si…
fix(accuracy): relative error through negation, abs, set operators an…
fix(accuracy): relative error through negation, abs, set operators an…