Skip to content

Executor ignores max_out_of_order_ms: a late counter sample inside the declared bound fails the ingestion-time rate build #1367

Open

Description

@zzylol

Problem, by example

A deployment that keeps no raw samples runs a repeating rate, and declares that samples may arrive up to 5 s late:

{"query_workload": {"language": "promql", "repeating_queries": [{
   "query": "sum(rate(requests_total[1m]))",
   "demand": {"fixed_interval_at": {"interval": 60000, "evaluation_phase": 0}},
   "requirements": {},
   "predictability": {"predictable": {"known_at": null}}}]},
 "data_workload": {"arrival": {"ingesting": {"rows_per_second": 1000.0, "max_out_of_order_ms": 5000}},
   "sources": {"requests_total": {"series_count": 1000, "sample_interval": 1000, "metric_type": "counter"}}}}

with DeploymentCapabilities { raw_data_retained: false, .. }.

The planner selects an ingestion-time rate state, the same plan as without max_out_of_order_ms (checked by runtime_install::the_report_names_the_data_facts_the_plan_assumed). Because the bound is declared, the plan output carries no in-order-arrival warning: the planner takes the declared bound as handled.

Now one series receives, in this arrival order:

arrival order ts (ms) value
1 10,000 100
2 12,000 120
3 11,000 110

The third sample is 1 s late, inside the declared 5 s bound.

today (from reading the executor code; not run) expected
ingestion-time rate build StateColumnUpdater::update_value returns the operator error "counter samples must have strictly increasing timestamps within each group", so the build fails the sample at 11,000 is folded in timestamp order, between 10,000 and 12,000
answer none (ingestion fails) the same answer as if the samples had arrived in order

The same happens to SQL rows: with max_out_of_order_ms declared, Stage 2 also offers ingestion-time counter states over SQL tables, and a row inside the bound fails the build the same way.

Why it happens

The planner and the executor read max_out_of_order_ms differently.

  • Planner: Stage 2 offers an ingestion-time rate, increase or irate state when its rows are ordered. A declared max_out_of_order_ms counts as ordered, because a reorder buffer is meant to restore the order within that bound (ingestion_row_order in crates/stage2-physical-optimization/src/candidate.rs). Without the bound, only PromQL samples are taken to arrive in order (an assumed data fact, which the plan output names as a warning).
  • Executor: nothing reads max_out_of_order_ms. crates/executor has no reorder buffer, and the bound is not part of the installable plan. A counter state checks each sample against the group's previous one:
    • StateColumnUpdater::update_value (crates/executor/src/summary_kernels/state_column_updater/mod.rs, around line 303) rejects a sample whose timestamp is not after the previous one, with an operator error;
    • IncreaseAccumulator::update (crates/executor/src/summary_kernels/increase.rs, around line 76), the counter kernel underneath, drops such a sample instead.

So a late sample inside the declared bound either fails ingestion or is lost, although the plan was chosen on the premise that the bound is handled.

Proposal

Extend the executor; do not restrict the planner's offer.

  1. Carry the declared bound to the executor: the installable plan (or the runtime's ingestion config built from the data workload) says, for each ingestion-time counter state, how late a row may arrive.
  2. Add a reorder buffer in front of each ingestion-time counter state build: per series (group), hold rows until the watermark, the newest timestamp seen minus max_out_of_order_ms, passes them, then fold them in timestamp order. A tumbling window closes only once the watermark passes its end.
  3. A row later than the bound stays an input error, as today.

Open: whether the buffer sits per state build or once per source in the ingestion path, shared by every counter state over that source.

Scope and acceptance

In scope: the executor's ingestion path for rate, increase and irate states (SummaryAgg and SummaryTumblingWindowChain at ingestion time), for PromQL samples and SQL rows, and carrying the bound into what the executor installs.

Out of scope: lateness relative to the evaluation time, the finality of stored results over absolute intervals (#1298).

Acceptance: an executor test that plans the workload above, feeds the three samples in the arrival order shown, and gets the same answer as with in-order arrival. A sample 6 s late is still rejected as an input error. It fails today at the build.

Status and related work

Activity

  1. added
    bugSomething isn't working
    promqlPromQL front-end lowering (L1→L2)
    sqlSQL front-end lowering (DataFusion → L2)
    on Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpromqlPromQL front-end lowering (L1→L2)sqlSQL front-end lowering (DataFusion → L2)

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions