Skip to content

Executor deadlocks when two query roots that share a producer are drained one after another #1290

Closed

Description

@zzylol

Example

Workload y16: two PromQL queries, both every 60 s (evaluation phase 15 s), with raw data not kept (raw_data_retained: false). There are 1,000 series of m at 1 sample/s.

sum_over_time(m[5m])
sum_over_time(m[5m] offset 1m)

The planner shares one materialized tumbling-window chain (node 1, exact Sum, 15 s windows) between both queries. At query time, node 1 has two readers: TimeRange(300s) for the first query (root 8), and TimeShift(60s) → TimeRange(300s) for the second (root 5).

The caller drains the root streams one after another:

let outputs = instance.execute(compiled.roots(), context)?; // roots [8, 5]
for mut output in outputs {
    while let Some(batch) = output.next().await { /* collect */ }
}
  • Today: the loop never finishes. The process sits at 0% CPU, waiting on root 8.
  • Expected: both roots return their answers, or execute returns a clear error instead of hanging.

If the same streams are drained concurrently with futures::future::join_all, both complete, and the answers match the exact reference.

Workloads n02 (two SQL count(*) ... GROUP BY src queries over 5 and 15 minutes) and n14 (two PromQL quantile_over_time queries over 10 and 5 minutes, raw data not kept) hang the same way. All three were found with the Stage 3 review harness (#1236), and all three complete with join_all.

Why

A node read by several consumers runs as one shared Producer (crates/executor/src/runtime/mod.rs). Each consumer reads it through its own Input.

  1. The producer keeps a queue of batches. A batch stays in the queue until every reader has read it (Producer::trim drops only up to the slowest reader's position).
  2. Input::poll_next returns Poll::Pending without polling the upstream stream once the queue holds max_buffered_batches batches. The default is 8 (Limits::default in crates/executor/src/runtime/context.rs). It registers the reader's waker in waiters.
  3. Those wakers are woken only from trim, which runs when some reader advances or is dropped.
  4. When the caller drains root 8 to the end before polling root 5, root 5's reader never advances. The queue fills to 8, root 8's reader returns Pending, and nothing will ever wake it.

execute (crates/executor/src/plan/mod.rs) returns one Input stream per root and does not say that the roots must be polled together. Draining them in order is the natural way to use a Vec of streams.

Proposal

Options:

  • (a) Document it. State on execute that all root streams must be polled concurrently. This is cheap, but the next caller who drains in order still hangs silently.
  • (b) Make the hang impossible. For example:
    1. Return one combined stream that yields each batch tagged with its root, so the runtime polls all roots itself.
    2. Let a lagging reader's batches spill, or make the queue unbounded for readers that have not started, bounded by max_bytes instead of max_buffered_batches.
    3. Detect the case and return an error. When a reader would block on a full queue and the slowest reader belongs to a root the caller is not polling, fail with an error that names the shared node and says to poll the roots concurrently.

Recommendation: (b). A silent hang is the worst outcome, so at minimum detect it and return a clear error (b3). Then decide between a combined stream (b1) and spilling (b2) for full support.

Scope and acceptance

In scope: the runtime's backpressure between readers of a shared producer when the roots are drained one after another.

Acceptance: an executor test with two roots sharing one producer that emits more than max_buffered_batches batches. Draining the roots in order either completes with the same answers as join_all or returns the new error within a bounded time. Today the test hangs.

Related

Found during the functional review of the Stage 3 baseline (#1236, PR 6 of #1207). This is not a planner or Stage 3 bug: the plans are correct, and they answer correctly when the roots are polled concurrently.

Activity

  1. zzylol commented on Nov 13, 2026

    @zzylol
    ContributorAuthor

    A one-query variant: two branches of one query reading the same stored chain

    Workload b17 hangs the same way, but it has only one query, so one root. Draining that single root with join_all does not help.

    sum(sum_over_time(m[5m] offset 5m)) / sum(sum_over_time(m[5m]))
    
    • every 60 s, evaluation phase 0;
    • raw data not kept (raw_data_retained: false);
    • 1,000 series of gauge m at one sample per second, 2,000 rows/s arriving.

    The selected plan has one stored ingestion-time chain, node 1. Both sides of the division read it at query time: one directly through TimeRange(5m), the other through TimeShift(5m) and then TimeRange(5m).

    What the Stage 3 review harness saw (#1236):

    The likely cause is the shared Producer backpressure described above. The two readers of node 1 are inside one root, so if the operator that combines them (the BinaryOp, or the two sums below it) polls one input to the end before the other, the second reader never advances and the queue fills. That is a guess from the code; I have not traced it in the runtime.

    If it holds, the acceptance test above needs a one-root case as well: one root whose two branches read one shared producer that emits more than max_buffered_batches batches. It would rule out option (a): a caller cannot fix this by polling roots concurrently.

  2. added 3 commits that reference this issue on Nov 14, 2026
    84b6848
    fcce276
    93f223a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions