Repository navigation
Description
Example
Workload y16: two PromQL queries, both every 60 s (evaluation phase 15 s), with raw data not kept (raw_data_retained: false). There are 1,000 series of m at 1 sample/s.
sum_over_time(m[5m])
sum_over_time(m[5m] offset 1m)
The planner shares one materialized tumbling-window chain (node 1, exact Sum, 15 s windows) between both queries. At query time, node 1 has two readers: TimeRange(300s) for the first query (root 8), and TimeShift(60s) → TimeRange(300s) for the second (root 5).
The caller drains the root streams one after another:
let outputs = instance.execute(compiled.roots(), context)?; // roots [8, 5]
for mut output in outputs {
while let Some(batch) = output.next().await { /* collect */ }
}- Today: the loop never finishes. The process sits at 0% CPU, waiting on root 8.
- Expected: both roots return their answers, or
executereturns a clear error instead of hanging.
If the same streams are drained concurrently with futures::future::join_all, both complete, and the answers match the exact reference.
Workloads n02 (two SQL count(*) ... GROUP BY src queries over 5 and 15 minutes) and n14 (two PromQL quantile_over_time queries over 10 and 5 minutes, raw data not kept) hang the same way. All three were found with the Stage 3 review harness (#1236), and all three complete with join_all.
Why
A node read by several consumers runs as one shared Producer (crates/executor/src/runtime/mod.rs). Each consumer reads it through its own Input.
- The producer keeps a queue of batches. A batch stays in the queue until every reader has read it (
Producer::trimdrops only up to the slowest reader's position). Input::poll_nextreturnsPoll::Pendingwithout polling the upstream stream once the queue holdsmax_buffered_batchesbatches. The default is 8 (Limits::defaultincrates/executor/src/runtime/context.rs). It registers the reader's waker inwaiters.- Those wakers are woken only from
trim, which runs when some reader advances or is dropped. - When the caller drains root 8 to the end before polling root 5, root 5's reader never advances. The queue fills to 8, root 8's reader returns
Pending, and nothing will ever wake it.
execute (crates/executor/src/plan/mod.rs) returns one Input stream per root and does not say that the roots must be polled together. Draining them in order is the natural way to use a Vec of streams.
Proposal
Options:
- (a) Document it. State on
executethat all root streams must be polled concurrently. This is cheap, but the next caller who drains in order still hangs silently. - (b) Make the hang impossible. For example:
- Return one combined stream that yields each batch tagged with its root, so the runtime polls all roots itself.
- Let a lagging reader's batches spill, or make the queue unbounded for readers that have not started, bounded by
max_bytesinstead ofmax_buffered_batches. - Detect the case and return an error. When a reader would block on a full queue and the slowest reader belongs to a root the caller is not polling, fail with an error that names the shared node and says to poll the roots concurrently.
Recommendation: (b). A silent hang is the worst outcome, so at minimum detect it and return a clear error (b3). Then decide between a combined stream (b1) and spilling (b2) for full support.
Scope and acceptance
In scope: the runtime's backpressure between readers of a shared producer when the roots are drained one after another.
Acceptance: an executor test with two roots sharing one producer that emits more than max_buffered_batches batches. Draining the roots in order either completes with the same answers as join_all or returns the new error within a bounded time. Today the test hangs.
Related
Found during the functional review of the Stage 3 baseline (#1236, PR 6 of #1207). This is not a planner or Stage 3 bug: the plans are correct, and they answer correctly when the roots are polled concurrently.
Activity
zzylol commentedon Nov 13, 2026
A one-query variant: two branches of one query reading the same stored chain
Workload
b17hangs the same way, but it has only one query, so one root. Draining that single root withjoin_alldoes not help.raw_data_retained: false);mat one sample per second, 2,000 rows/s arriving.The selected plan has one stored ingestion-time chain, node 1. Both sides of the division read it at query time: one directly through
TimeRange(5m), the other throughTimeShift(5m)and thenTimeRange(5m).What the Stage 3 review harness saw (#1236):
RVSTEP done ids=[1]).execute ids=[11]never returns, and the harness's 900 s timeout kills it.9ec3847fc.plan_cost.b17_binop_offsets_noraw.jsonandb17_binop_offsets_noraw.caps.json, from the round-4 review's workload set.The likely cause is the shared
Producerbackpressure described above. The two readers of node 1 are inside one root, so if the operator that combines them (theBinaryOp, or the twosums below it) polls one input to the end before the other, the second reader never advances and the queue fills. That is a guess from the code; I have not traced it in the runtime.If it holds, the acceptance test above needs a one-root case as well: one root whose two branches read one shared producer that emits more than
max_buffered_batchesbatches. It would rule out option (a): a caller cannot fix this by polling roots concurrently.fix(executor): run each node by its plan Lifetime, so roots drain in …
fix(executor): run each node by its plan Lifetime, so roots drain in …
fix(executor): run each node by its plan Lifetime, so roots drain in …