Repository navigation
Description
Problem, by example
A deployment that keeps no raw samples runs a repeating rate, and declares that samples may arrive up to 5 s late:
{"query_workload": {"language": "promql", "repeating_queries": [{
"query": "sum(rate(requests_total[1m]))",
"demand": {"fixed_interval_at": {"interval": 60000, "evaluation_phase": 0}},
"requirements": {},
"predictability": {"predictable": {"known_at": null}}}]},
"data_workload": {"arrival": {"ingesting": {"rows_per_second": 1000.0, "max_out_of_order_ms": 5000}},
"sources": {"requests_total": {"series_count": 1000, "sample_interval": 1000, "metric_type": "counter"}}}}with DeploymentCapabilities { raw_data_retained: false, .. }.
The planner selects an ingestion-time rate state, the same plan as without max_out_of_order_ms (checked by runtime_install::the_report_names_the_data_facts_the_plan_assumed). Because the bound is declared, the plan output carries no in-order-arrival warning: the planner takes the declared bound as handled.
Now one series receives, in this arrival order:
| arrival order | ts (ms) |
value |
|---|---|---|
| 1 | 10,000 | 100 |
| 2 | 12,000 | 120 |
| 3 | 11,000 | 110 |
The third sample is 1 s late, inside the declared 5 s bound.
| today (from reading the executor code; not run) | expected | |
|---|---|---|
ingestion-time rate build |
StateColumnUpdater::update_value returns the operator error "counter samples must have strictly increasing timestamps within each group", so the build fails |
the sample at 11,000 is folded in timestamp order, between 10,000 and 12,000 |
| answer | none (ingestion fails) | the same answer as if the samples had arrived in order |
The same happens to SQL rows: with max_out_of_order_ms declared, Stage 2 also offers ingestion-time counter states over SQL tables, and a row inside the bound fails the build the same way.
Why it happens
The planner and the executor read max_out_of_order_ms differently.
- Planner: Stage 2 offers an ingestion-time
rate,increaseoriratestate when its rows are ordered. A declaredmax_out_of_order_mscounts as ordered, because a reorder buffer is meant to restore the order within that bound (ingestion_row_orderincrates/stage2-physical-optimization/src/candidate.rs). Without the bound, only PromQL samples are taken to arrive in order (an assumed data fact, which the plan output names as a warning). - Executor: nothing reads
max_out_of_order_ms.crates/executorhas no reorder buffer, and the bound is not part of the installable plan. A counter state checks each sample against the group's previous one:StateColumnUpdater::update_value(crates/executor/src/summary_kernels/state_column_updater/mod.rs, around line 303) rejects a sample whose timestamp is not after the previous one, with an operator error;IncreaseAccumulator::update(crates/executor/src/summary_kernels/increase.rs, around line 76), the counter kernel underneath, drops such a sample instead.
So a late sample inside the declared bound either fails ingestion or is lost, although the plan was chosen on the premise that the bound is handled.
Proposal
Extend the executor; do not restrict the planner's offer.
- Carry the declared bound to the executor: the installable plan (or the runtime's ingestion config built from the data workload) says, for each ingestion-time counter state, how late a row may arrive.
- Add a reorder buffer in front of each ingestion-time counter state build: per series (group), hold rows until the watermark, the newest timestamp seen minus
max_out_of_order_ms, passes them, then fold them in timestamp order. A tumbling window closes only once the watermark passes its end. - A row later than the bound stays an input error, as today.
Open: whether the buffer sits per state build or once per source in the ingestion path, shared by every counter state over that source.
Scope and acceptance
In scope: the executor's ingestion path for rate, increase and irate states (SummaryAgg and SummaryTumblingWindowChain at ingestion time), for PromQL samples and SQL rows, and carrying the bound into what the executor installs.
Out of scope: lateness relative to the evaluation time, the finality of stored results over absolute intervals (#1298).
Acceptance: an executor test that plans the workload above, feeds the three samples in the arrival order shown, and gets the same answer as with in-order arrival. A sample 6 s late is still rejected as an input error. It fails today at the build.
Status and related work
- Found in review of feat(types, cost, planner): warn about each data fact a plan assumes because the workload does not declare it (DDSketch range, in-order arrival) #1365, which warns when the plan assumes in-order arrival because
max_out_of_order_msis undeclared. With the bound declared there is no warning, which is right for the planner but relies on this executor gap being closed. - Related: Declare how late rows may arrive, so a stored result over an absolute interval is final at E + allowed lateness #1298 (declaring lateness relative to the evaluation time); rate(c[5m]) / rate(c[10m]) plans but fails at execution: counter samples must have strictly increasing timestamps #1341 (the same executor check fires when one shared chain is read by two windows).
Activity