diff --git a/crates/executor/README.md b/crates/executor/README.md index de199a07f..b9e8075e3 100644 --- a/crates/executor/README.md +++ b/crates/executor/README.md @@ -6,12 +6,50 @@ a storage implementation nor DataFusion. Batches are Arrow columns (`values::Batch`); DataFusion informed the design, it is not the execution framework. -`plan::PhysicalDAG` binds typed operator inputs to node IDs. Each execution starts -one producer per reachable node, shares output batches among its consumers, and -bounds buffering. Dropping one consumer does not cancel other consumers. A -`RunContext` carries query or ingestion scope, cancellation and byte accounting. -Executions use the caller's worker and worker-local streams, with no internal -thread pool. Poll multiple root streams concurrently when they share inputs. +`plan::PhysicalDAG` binds typed operator inputs to node IDs, each node with the +plan's `Lifetime`. A `RunContext` carries query or ingestion scope, +cancellation and byte accounting. Executions use the caller's worker and +worker-local streams, with no internal thread pool. + +A node runs once per evaluation time in a run (an `@` anchor evaluates the +nodes below it at another time; a leaf read per consumer runs once per +consumer), as its `Lifetime` and the run's scope say: + +- Query time, a leaf with `None` or `AcrossEvaluations` (rows a store or + the operator keeps): `None` (a raw data scan the plan does not + materialize, or any other leaf such as a constant, which holds its own + rows) or `AcrossEvaluations` (a materialized view slot, or raw data kept + at ingestion). Each consumer reads it on its own, and + nothing is held. Stage 3 never gives a query-time `None` result more than + one consumer; a view slot, or a leaf in a hand-built DAG, may have + several. For example, an `AcrossEvaluations` scan read by two queries is + two store reads. +- Query time, `OneEvaluation` (a computed node or a leaf) with more than + one consumer: the node runs once, to the end, before any consumer reads + it, and every consumer reads its finished output independently, as the + planner charges one run whose output is held. Likewise a computed + `AcrossEvaluations` node with more than one consumer, which only a + hand-built DAG has at query time (a compiled plan reads it through its + view slot). For example, a `OneEvaluation` scan read by three queries + is one store read. The output is charged to the run's byte budget until + every consumer has read it; over the budget the run fails with + `MemoryLimit` (nothing spills). With one consumer the node streams to it, + since no sibling can wait on that consumer. +- Query time, `None` computed node: streamed to its one consumer. More than + one consumer is a planner bug; `execute` returns an error naming the node + before any operator starts. +- Ingestion time: every node streams, whatever its lifetime. A node with + several consumers (one per-row node feeding several chain builds or views) + runs once and each batch reaches every consumer's own queue. The queues + are bounded only by the run's byte budget: a consumer that lags holds what + the others have read. `execute_ingestion` polls every root itself and + returns one future that resolves to each root's batches, for the caller to + write. + +`execute` runs a query and returns one stream per root; the caller may drain +them in any order. It rejects an ingestion scope, which `execute_ingestion` +runs, and `execute_ingestion` rejects a query scope. Dropping one consumer +does not cancel other consumers. `operators::Operator` implements native batch sources, scalar values, projection, filtering, grouped exact aggregation, semi-join, grouped Sort and @@ -28,6 +66,7 @@ use asap_executor::{ plan::PhysicalDAG, runtime::{Limits, RunContext, Scope}, }; +use asap_executor::planner::stages::Lifetime; use asap_executor::planner::ir::schema::DataType; use futures::{executor::block_on, StreamExt}; @@ -36,8 +75,8 @@ let negate = Operator::project(source.schema(), vec![ ("value".into(), Expression::Negate(Box::new(Expression::Column(0)))), ])?; let mut plan = PhysicalDAG::default(); -plan.add(0, vec![], source)?; -plan.add(1, vec![0], negate)?; +plan.add(0, vec![], source, Lifetime::None)?; +plan.add(1, vec![0], negate, Lifetime::None)?; let run = RunContext::new( Scope::Query { evaluation_time_ms: 1000 }, Limits::default(), @@ -76,7 +115,7 @@ See [the design](../../docs/design_docs/physical-planning-and-deployment.md). ## Module boundaries - `plan`: immutable DAG, operator interface, schemas and execution properties. -- `runtime`: per-run streams, shared producers, memory reservations and cancellation. +- `runtime`: per-run streams, how each node runs by its `Lifetime`, memory reservations and cancellation. - `expressions`: scalar evaluation; typed builders and the Planner expression adapter. - `operators`: projection, filter, joins, aggregate/window, sort, limit and summary implementations. - `sources`: the `DataStore` data interface, `Scan` and the in-memory store. diff --git a/crates/executor/src/operators/aggregate/temporal.rs b/crates/executor/src/operators/aggregate/temporal.rs index 12771677e..02df9ec8e 100644 --- a/crates/executor/src/operators/aggregate/temporal.rs +++ b/crates/executor/src/operators/aggregate/temporal.rs @@ -326,6 +326,7 @@ mod tests { values::Batch, }; use futures::StreamExt; + use planner_types::stages::Lifetime; use planner_types::ir::schema::{ DataType, Field as SummaryField, FieldDataType as SummaryFamilyType, Schema, @@ -383,12 +384,21 @@ mod tests { 0, vec![], Operator::source(schema.clone(), vec![batch]).unwrap(), + Lifetime::None, ) .unwrap(); - dag.add(1, vec![0], operator).unwrap(); + dag.add(1, vec![0], operator, Lifetime::None).unwrap(); let context = RunContext::new(scope.clone(), Limits::default()).unwrap(); - let mut output = dag.execute(&[1], context).unwrap().remove(0); - let result = futures::executor::block_on(output.next()).unwrap().unwrap(); + let result = if matches!(scope, Scope::Ingestion { .. }) { + let run = dag.execute_ingestion(&[1], context).unwrap(); + futures::executor::block_on(run) + .unwrap() + .remove(0) + .remove(0) + } else { + let mut output = dag.execute(&[1], context).unwrap().remove(0); + futures::executor::block_on(output.next()).unwrap().unwrap() + }; assert_eq!( format!("{:?}", result.to_rows()[0][0]), format!("{expected:?}") diff --git a/crates/executor/src/operators/series_labels.rs b/crates/executor/src/operators/series_labels.rs index 2bce411a4..1fee5822e 100644 --- a/crates/executor/src/operators/series_labels.rs +++ b/crates/executor/src/operators/series_labels.rs @@ -1108,6 +1108,7 @@ mod columnar_tests { use crate::plan::PhysicalDAG; use crate::runtime::{Limits, Scope}; use planner_types::ir::schema::Field; + use planner_types::stages::Lifetime; /// The row-at-a-time rewrite this operator ran before (the reference). fn reference( @@ -1248,9 +1249,10 @@ mod columnar_tests { ], ) .unwrap(), + Lifetime::None, ) .unwrap(); - dag.add(1, vec![0], operator).unwrap(); + dag.add(1, vec![0], operator, Lifetime::None).unwrap(); let context = RunContext::new( Scope::Query { evaluation_time_ms: 0, diff --git a/crates/executor/src/physical_planner/compiled.rs b/crates/executor/src/physical_planner/compiled.rs index 7d074d659..83ef73c0d 100644 --- a/crates/executor/src/physical_planner/compiled.rs +++ b/crates/executor/src/physical_planner/compiled.rs @@ -2,6 +2,7 @@ use super::*; use crate::sources::{DataStore, Scan}; use planner_types::plan::materialized_view::MaterializedView; +use planner_types::stages::Lifetime; #[derive(Clone)] #[cfg_attr(any(test, feature = "test-support"), derive(serde::Serialize))] @@ -52,26 +53,36 @@ pub(super) struct RawDataSlot { pub struct CompiledPhysicalDAG { nodes: BTreeMap, roots: Vec, + /// The plan lifetime of each compiled Planner node, which the runtime + /// runs it by (see `NodeRun` in the runtime). A helper node the lowering + /// adds is not in it and has `Lifetime::None`. Example: node 1, a stored view, + /// maps to `AcrossEvaluations(Memory)`, and node 3, a query-time output + /// two queries read, to `OneEvaluation(Memory)`. + lifetimes: BTreeMap, } impl CompiledPhysicalDAG { /// Assemble already-lowered operators; leaves read no data store (for - /// example [`Operator::source`] holds its rows). + /// example [`Operator::source`] holds its rows). Every node runs as + /// `Lifetime::None`, so at query time a computed node may have at most + /// one consumer; for a shared one, build a [`PhysicalDAG`] with + /// `add(…, lifetime)`. pub fn from_operators( operators: BTreeMap, Operator)>, roots: Vec, ) -> Result { - let mut result = Self::new(roots); + let mut result = Self::new(roots, BTreeMap::new()); for (id, (inputs, operator)) in operators { result.add(id, inputs, operator)?; } result.validate()?; Ok(result) } - pub(super) fn new(roots: Vec) -> Self { + pub(super) fn new(roots: Vec, lifetimes: BTreeMap) -> Self { Self { nodes: BTreeMap::new(), roots, + lifetimes, } } pub(super) fn add_materialized_view( @@ -157,13 +168,14 @@ impl CompiledPhysicalDAG { fn unbound(&self) -> Result, Error> { let mut dag = PhysicalDAG::default(); for (&id, node) in &self.nodes { + let lifetime = self.lifetime(id); match node { Node::MaterializedView(MaterializedViewSlot { schema, .. }) | Node::RawData(RawDataSlot { schema, .. }) => { - dag.add(id, vec![], UnboundInput(schema.clone()))? + dag.add(id, vec![], UnboundInput(schema.clone()), lifetime)? } Node::Operator { inputs, operator } => { - dag.add(id, inputs.clone(), operator.clone())? + dag.add(id, inputs.clone(), operator.clone(), lifetime)? } } } @@ -180,6 +192,7 @@ impl CompiledPhysicalDAG { ) -> Result, Error> { let mut dag = PhysicalDAG::default(); for (&id, node) in &self.nodes { + let lifetime = self.lifetime(id); match node { Node::MaterializedView(slot) => { let scan = Scan::materialized_view( @@ -188,7 +201,7 @@ impl CompiledPhysicalDAG { slot.schema.clone(), slot.read_window, ); - dag.add(id, vec![], scan)?; + dag.add(id, vec![], scan, lifetime)?; } Node::RawData(slot) => { let scan = Scan::raw_data( @@ -198,16 +211,19 @@ impl CompiledPhysicalDAG { slot.predicates.clone(), slot.read_window, ); - dag.add(id, vec![], scan)?; + dag.add(id, vec![], scan, lifetime)?; } Node::Operator { inputs, operator } => { - dag.add(id, inputs.clone(), operator.clone())?; + dag.add(id, inputs.clone(), operator.clone(), lifetime)? } } } dag.validate(&self.roots)?; Ok(dag) } + fn lifetime(&self, id: NodeId) -> Lifetime { + self.lifetimes.get(&id).copied().unwrap_or_default() + } } /// Stands in for an input slot when validating or deriving properties. struct UnboundInput(SchemaRef); diff --git a/crates/executor/src/physical_planner/mod.rs b/crates/executor/src/physical_planner/mod.rs index 0123a59da..ffd9d1b6a 100644 --- a/crates/executor/src/physical_planner/mod.rs +++ b/crates/executor/src/physical_planner/mod.rs @@ -183,7 +183,11 @@ fn compile_internal( consumers: &consumers, roots: &compile_roots, }; - let mut physical_dag = CompiledPhysicalDAG::new(roots.to_vec()); + let lifetimes = ordered + .iter() + .map(|&id| (id, nodes[&id].lifetime)) + .collect(); + let mut physical_dag = CompiledPhysicalDAG::new(roots.to_vec(), lifetimes); for id in ordered { let node = nodes[&id]; let auxiliary = helper_id(id, 0)?; diff --git a/crates/executor/src/physical_planner/test_support.rs b/crates/executor/src/physical_planner/test_support.rs index fb288a546..149794748 100644 --- a/crates/executor/src/physical_planner/test_support.rs +++ b/crates/executor/src/physical_planner/test_support.rs @@ -37,7 +37,10 @@ pub fn set_build_parameters<'a>( /// The materialized views at `nodes`, in that order: each node that is /// not a view is marked materialized at ingestion time, and it and /// every node below it run at ingestion time (their outputs and -/// outgoing edges); every view without a retention or coverage is +/// outgoing edges). A `OneEvaluation` node moved there becomes +/// `Lifetime::None`, as ingestion has no evaluation to hold it for (the +/// DAG's validation rejects `OneEvaluation` at ingestion): for example a scan five tumbling windows share streams to all five. Every +/// view without a retention or coverage is /// retained for ever with a coverage of its own. For tests that bind /// states at those nodes, as a deployment binds its views. pub fn views_at( @@ -87,6 +90,9 @@ pub fn views_at( } if at_ingestion.contains(&id) { node.output_state.timing = ExecutionTiming::IngestionTime; + if matches!(node.lifetime, Lifetime::OneEvaluation(_)) { + node.lifetime = Lifetime::None; + } } } for edge in &mut dag.edges { diff --git a/crates/executor/src/plan/mod.rs b/crates/executor/src/plan/mod.rs index 1b87df0af..71884a244 100644 --- a/crates/executor/src/plan/mod.rs +++ b/crates/executor/src/plan/mod.rs @@ -1,8 +1,9 @@ //! Immutable physical DAG, operator contracts and pre-execution validation. use crate::{ - runtime::{Input, OutputStream, RunContext}, + runtime::{IngestionOutputs, Input, OutputStream, RunContext}, Error, }; +use planner_types::stages::Lifetime; use std::{ collections::{BTreeMap, BTreeSet}, fmt::Debug, @@ -47,6 +48,10 @@ pub trait PhysicalOperator { pub(crate) struct Node<'a, V, S> { pub(crate) inputs: Vec, pub(crate) operator: Box + 'a>, + /// How long its output is kept, from the plan: what the runtime runs it + /// by (see the crate README). Example: `OneEvaluation`, computed once + /// and held for its consumers at query time. + pub(crate) lifetime: Lifetime, } pub struct PhysicalDAG<'a, V, S> { pub(crate) nodes: BTreeMap>, @@ -59,24 +64,35 @@ impl Default for PhysicalDAG<'_, V, S> { } } impl<'a, V: 'a, S: Clone + PartialEq + Debug + 'a> PhysicalDAG<'a, V, S> { + /// Add node `id`, reading `inputs`, run by its plan `lifetime`. pub fn add( &mut self, id: NodeId, inputs: Vec, operator: impl PhysicalOperator + 'a, + lifetime: Lifetime, ) -> Result<(), Error> { - self.add_boxed(id, inputs, Box::new(operator)) + self.add_boxed(id, inputs, Box::new(operator), lifetime) } + /// [`Self::add`] with an already boxed operator. pub fn add_boxed( &mut self, id: NodeId, inputs: Vec, operator: Box + 'a>, + lifetime: Lifetime, ) -> Result<(), Error> { if self.nodes.contains_key(&id) { return Err(Error::Invalid(format!("duplicate node {id}"))); } - self.nodes.insert(id, Node { inputs, operator }); + self.nodes.insert( + id, + Node { + inputs, + operator, + lifetime, + }, + ); Ok(()) } pub fn validate(&self, roots: &[NodeId]) -> Result<(), Error> { @@ -153,6 +169,11 @@ impl<'a, V: 'a, S: Clone + PartialEq + Debug + 'a> PhysicalDAG<'a, V, S> { .map(|(id, (_, properties))| (id, properties)) .collect()) } + /// A query run: one output stream per root, which the caller may drain + /// in any order. Rejects an ingestion scope (run it with + /// [`Self::execute_ingestion`]), and a computed node with + /// `Lifetime::None` and more than one consumer at one evaluation time (a + /// planner bug), before any operator starts. pub fn execute<'r>( &'r self, roots: &[NodeId], @@ -163,4 +184,17 @@ impl<'a, V: 'a, S: Clone + PartialEq + Debug + 'a> PhysicalDAG<'a, V, S> { { crate::runtime::execute(self, roots, context) } + /// An ingestion run: one future that resolves to every root's batches, + /// in `roots` order, for the caller to write. Rejects a query scope (run + /// it with [`Self::execute`]). + pub fn execute_ingestion<'r>( + &'r self, + roots: &[NodeId], + context: RunContext, + ) -> Result, Error>> + 'r, Error> + where + 'a: 'r, + { + crate::runtime::execute_ingestion(self, roots, context) + } } diff --git a/crates/executor/src/runtime/context.rs b/crates/executor/src/runtime/context.rs index e20b7307c..95e361db6 100644 --- a/crates/executor/src/runtime/context.rs +++ b/crates/executor/src/runtime/context.rs @@ -17,13 +17,11 @@ pub enum Scope { } #[derive(Clone, Debug)] pub struct Limits { - pub max_buffered_batches: usize, pub max_bytes: usize, } impl Default for Limits { fn default() -> Self { Self { - max_buffered_batches: 8, max_bytes: 64 * 1024 * 1024, } } @@ -43,7 +41,7 @@ pub struct RunContext { } impl RunContext { pub fn new(scope: Scope, limits: Limits) -> Result { - if limits.max_buffered_batches == 0 || limits.max_bytes == 0 { + if limits.max_bytes == 0 { return Err(Error::Invalid("execution limits must be positive".into())); } if matches!(&scope, Scope::Ingestion { window_start_ms, window_end_ms, .. } if window_start_ms > window_end_ms) diff --git a/crates/executor/src/runtime/mod.rs b/crates/executor/src/runtime/mod.rs index 89039cffc..163c89a37 100644 --- a/crates/executor/src/runtime/mod.rs +++ b/crates/executor/src/runtime/mod.rs @@ -1,21 +1,24 @@ -//! Per-run producer sharing, streams, backpressure and resource ownership. +//! Per-run producers run by each node's `Lifetime`, streams and resource ownership. use crate::{ - plan::{NodeId, PhysicalDAG}, + plan::{Node, NodeId, PhysicalDAG}, Error, }; -use futures::{stream::LocalBoxStream, Stream}; +use futures::{stream::LocalBoxStream, Stream, TryStreamExt}; +use planner_types::stages::Lifetime; use std::{ cell::RefCell, collections::{BTreeMap, VecDeque}, fmt::Debug, pin::Pin, - rc::Rc, + rc::{Rc, Weak}, sync::Arc, task::{Context, Poll, Waker}, }; mod context; pub use context::{Limits, Reservation, RunContext, Scope}; pub type OutputStream<'a, V> = LocalBoxStream<'a, Result>; +/// Every root's batches of an ingestion run, in the order of its roots. +pub type IngestionOutputs = Vec>>; /// An output owns its memory reservation even after it leaves the DAG's queue. pub struct SharedValue { value: Arc, @@ -41,117 +44,295 @@ impl SharedValue { } } +/// A query-time run: one output stream per root. Rejects an ingestion +/// scope, and a computed node with [`Lifetime::None`] and more than one +/// consumer (a planner bug), before any operator starts. pub(crate) fn execute<'r, V: 'r, S: Clone + PartialEq + Debug + 'r>( dag: &'r PhysicalDAG<'_, V, S>, roots: &[NodeId], context: RunContext, ) -> Result>, Error> { - if context.is_cancelled() { - return Err(Error::Cancelled); + if context.evaluation_time_ms().is_none() { + return Err(Error::Invalid( + "`execute` takes a query scope; run an ingestion scope with `execute_ingestion`".into(), + )); } - dag.validate(roots)?; - let mut pending = roots.to_vec(); - let mut visited = std::collections::BTreeSet::new(); - while let Some(id) = pending.pop() { - if visited.insert(id) { - let node = &dag.nodes[&id]; - node.operator.validate_context(&context)?; - pending.extend(node.inputs.iter().copied()); + start(dag, roots, context) +} + +/// An ingestion run: every root's batches, the outputs the caller writes. +/// The runtime polls the roots together, so a node feeding several of them +/// is computed once and each batch reaches every consumer. Rejects a query +/// scope. +pub(crate) fn execute_ingestion<'r, V: 'r, S: Clone + PartialEq + Debug + 'r>( + dag: &'r PhysicalDAG<'_, V, S>, + roots: &[NodeId], + context: RunContext, +) -> Result, Error>> + 'r, Error> { + if context.evaluation_time_ms().is_some() { + return Err(Error::Invalid( + "`execute_ingestion` takes an ingestion scope; run a query scope with `execute`".into(), + )); + } + let outputs = start(dag, roots, context)?; + Ok(futures::future::try_join_all( + outputs.into_iter().map(TryStreamExt::try_collect), + )) +} + +/// A node runs once per time it is evaluated at: an `@` anchor above it +/// evaluates it at another time than the run's. +type ProducerKey = (NodeId, Option); + +/// How a node runs in a run, from the run's scope, its plan `Lifetime`, +/// whether it is a leaf and how many consumers it has at its evaluation +/// time. This is the one statement of the rule; the crate README describes +/// it for callers. +#[derive(Clone, Copy, PartialEq, Eq)] +enum NodeRun { + /// Query time, a leaf with `Lifetime::None` (a raw data slot, or any + /// other leaf such as a constant, whose rows the operator keeps) or + /// `AcrossEvaluations` (a materialized view slot, or raw data kept at + /// ingestion, whose rows a store keeps). Each consumer reads it on its own and nothing is held. + /// Stage 3 never gives a query-time `None` result more than one + /// consumer; a view slot, or a leaf in a hand-built DAG, may have + /// several. Example: an `AcrossEvaluations` scan read by two queries is + /// two store reads. + ReadPerConsumer, + /// Query time, a `OneEvaluation` node (computed or a leaf) with more + /// than one consumer: run to the end once before any consumer reads it, + /// then read by each consumer, as the planner charges one run whose + /// output is held. Likewise a computed `AcrossEvaluations` node with + /// more than one consumer, which only a hand-built DAG has at query time + /// (a compiled plan reads it through its view slot). Example: a + /// `OneEvaluation` scan read by three queries is one store read whose + /// batches every query reads. + Held, + /// Every other node: one run whose batches reach each consumer as they + /// come. At query time it has one consumer (a computed `None` node with + /// several is a planner bug, which `start` rejects). + Streamed, +} +fn run_as(node: &Node<'_, V, S>, query: bool, consumers: usize) -> NodeRun { + match (query, node.lifetime, node.inputs.is_empty()) { + (true, Lifetime::None | Lifetime::AcrossEvaluations(_), true) => NodeRun::ReadPerConsumer, + (true, Lifetime::OneEvaluation(_) | Lifetime::AcrossEvaluations(_), _) if consumers > 1 => { + NodeRun::Held } + _ => NodeRun::Streamed, + } +} + +/// A (node, evaluation time) a run reaches. Example: a scan read by both +/// sides of a division at time T has `consumers: 2`, and `input_context` is +/// T's context (another time's under an `@` anchor). +struct ReachedNode { + consumers: usize, + /// The run context its inputs run in. + input_context: RunContext, +} +type Reached = BTreeMap; + +/// Visit `id` once per evaluation time, before any operator starts: +/// validate its run context, count its consumers and record its inputs' +/// context. +fn reach( + dag: &PhysicalDAG<'_, V, S>, + id: NodeId, + context: &RunContext, + reached: &mut Reached, +) -> Result<(), Error> { + let key = (id, context.evaluation_time_ms()); + if let Some(reached) = reached.get_mut(&key) { + reached.consumers += 1; + return Ok(()); + } + let node = &dag.nodes[&id]; + node.operator.validate_context(context)?; + let input_context = node + .operator + .input_context(context) + .map_err(|source| Error::AtNode { + node: id, + operation: node.operator.name().into(), + source: Box::new(source), + })?; + for &child in &node.inputs { + reach(dag, child, &input_context, reached)?; } - // A node runs once per time it is evaluated at: an `@` anchor above it - // evaluates it at another time than the run's. - type ProducerKey = (NodeId, Option); - fn build<'r, V: 'r, S: 'r>( - dag: &'r PhysicalDAG<'_, V, S>, - id: NodeId, - context: &RunContext, - states: &mut BTreeMap>>>, - ) -> Result>>, Error> { - let key = (id, context.evaluation_time_ms()); + reached.insert( + key, + ReachedNode { + consumers: 1, + input_context, + }, + ); + Ok(()) +} + +fn build<'r, V: 'r, S: 'r>( + dag: &'r PhysicalDAG<'_, V, S>, + id: NodeId, + context: &RunContext, + reached: &Reached, + states: &mut BTreeMap>>>, + run_producers: &RunProducers<'r, V>, +) -> Result>>, Error> { + let key = (id, context.evaluation_time_ms()); + let node = &dag.nodes[&id]; + let reached_node = &reached[&key]; + let run = run_as(node, key.1.is_some(), reached_node.consumers); + if run != NodeRun::ReadPerConsumer { if let Some(state) = states.get(&key) { return Ok(Rc::clone(state)); } - let node = &dag.nodes[&id]; - let input_context = - node.operator - .input_context(context) - .map_err(|source| Error::AtNode { - node: id, - operation: node.operator.name().into(), - source: Box::new(source), - })?; - let mut inputs = Vec::new(); - for &child in &node.inputs { - inputs.push(Input::subscribe(build(dag, child, &input_context, states)?)); - } - let stream = node - .operator - .start(inputs, context.clone()) - .map_err(|source| Error::AtNode { - node: id, - operation: node.operator.name().into(), - source: Box::new(source), - })?; - let op = node.operator.as_ref(); - let state = Rc::new(RefCell::new(Producer { - stream: Some(stream), + } + let input_context = &reached_node.input_context; + let mut inputs = Vec::new(); + for &child in &node.inputs { + inputs.push(Input::subscribe(build( + dag, + child, + input_context, + reached, + states, + run_producers, + )?)); + } + let stream = node + .operator + .start(inputs, context.clone()) + .map_err(|source| Error::AtNode { node: id, operation: node.operator.name().into(), - size: Box::new(move |value| op.output_bytes(value)), - context: context.clone(), - queue: VecDeque::new(), - base: 0, - next_reader: 0, - batches_polled: 0, - readers: BTreeMap::new(), - waiters: BTreeMap::new(), - finished: false, - failure: None, - })); + source: Box::new(source), + })?; + let op = node.operator.as_ref(); + let state = Rc::new(RefCell::new(Producer { + stream: Some(stream), + node: id, + operation: node.operator.name().into(), + size: Box::new(move |value| op.output_bytes(value)), + context: context.clone(), + hold_until_finished: run == NodeRun::Held, + consumers: match run { + NodeRun::ReadPerConsumer => 1, + NodeRun::Held | NodeRun::Streamed => reached_node.consumers, + }, + run_producers: Rc::clone(run_producers), + unread: BTreeMap::new(), + next_reader: 0, + batches_polled: 0, + waiters: BTreeMap::new(), + finished: false, + failure: None, + })); + run_producers.borrow_mut().push(Rc::downgrade(&state)); + if run != NodeRun::ReadPerConsumer { states.insert(key, Rc::clone(&state)); - Ok(state) + } + Ok(state) +} + +fn start<'r, V: 'r, S: Clone + PartialEq + Debug + 'r>( + dag: &'r PhysicalDAG<'_, V, S>, + roots: &[NodeId], + context: RunContext, +) -> Result>, Error> { + if context.is_cancelled() { + return Err(Error::Cancelled); + } + dag.validate(roots)?; + let mut reached = Reached::new(); + for &root in roots { + reach(dag, root, &context, &mut reached)?; + } + for (&(id, time), ReachedNode { consumers, .. }) in &reached { + let node = &dag.nodes[&id]; + if let (Some(time), Lifetime::None, false, 2..) = + (time, node.lifetime, node.inputs.is_empty(), *consumers) + { + return Err(Error::Invalid(format!( + "node {id} ({}) at evaluation time {time} has {consumers} consumers but \ + Lifetime::None; a shared query-time output must be OneEvaluation (planner bug)", + node.operator.name() + ))); + } } let mut states = BTreeMap::new(); + let run_producers = RunProducers::default(); roots .iter() - .map(|&id| build(dag, id, &context, &mut states).map(Input::subscribe)) + .map(|&id| { + build(dag, id, &context, &reached, &mut states, &run_producers).map(Input::subscribe) + }) .collect() } +/// Every producer of one run, so a cancellation seen by one releases the +/// batches the others hold. +type RunProducers<'a, V> = Rc>>>>>; + +/// One run of a node, read by its consumers in the run (one, for a leaf +/// read per consumer). struct Producer<'a, V> { node: NodeId, operation: String, stream: Option>, size: Box usize + 'a>, context: RunContext, - queue: VecDeque>, - base: u64, + /// At query time, a [`NodeRun::Held`] node is computed to the end before + /// any consumer reads it, and its batches stay charged to the run until + /// every consumer has read them. + hold_until_finished: bool, + /// How many consumers subscribe to it in the run; its stream is dropped + /// early only once all have subscribed and all have left. + consumers: usize, + /// The run's producers, this one included. + run_producers: RunProducers<'a, V>, + /// The batches each reader has not read yet, by reader. + unread: BTreeMap>>, next_reader: u64, batches_polled: usize, - readers: BTreeMap, + /// The readers waiting for the next batch or the end. waiters: BTreeMap, finished: bool, failure: Option, } impl Producer<'_, V> { - fn trim(&mut self) { - let minimum = self - .readers - .values() - .copied() - .min() - .unwrap_or(self.base + self.queue.len() as u64); - while self.base < minimum { - self.queue.pop_front(); - self.base += 1; - } + fn wake_all(&mut self) { for (_, waker) in std::mem::take(&mut self.waiters) { waker.wake(); } - if self.readers.is_empty() { - self.stream = None; - self.queue.clear(); + } + /// The end of the node's output, seen while polling its stream, which + /// is dropped. + fn finish(&mut self, failure: Option) { + self.end(failure); + self.stream = None; + } + /// Stop the run: every consumer gets `Cancelled` next, and the batches + /// not yet read are released. The stream is kept, and dropped with the + /// node's last input: the producer that saw the cancellation may be + /// borrowed up the poll stack, and an input inside this stream, of + /// another consumer of it, would borrow it again on drop. So state an + /// operator keeps inside its stream (for example an accumulator's + /// reserved bytes) is freed only when the caller drops the run's + /// inputs, as callers do after `Cancelled`. + fn cancel(&mut self) { + if self.failure != Some(Error::Cancelled) { + self.end(Some(Error::Cancelled)); + } + } + /// Every consumer gets `failure`, or the end, next. A failed held output + /// is not read in part: every consumer gets only the error. A cancelled + /// output, held or not, releases every batch not yet read. + fn end(&mut self, failure: Option) { + if (failure.is_some() && self.hold_until_finished) || failure == Some(Error::Cancelled) { + self.unread.values_mut().for_each(VecDeque::clear); } + self.failure = failure; + self.finished = true; + self.wake_all(); } } pub struct Input<'a, V> { @@ -165,8 +346,7 @@ impl<'a, V> Input<'a, V> { let mut state = producer.borrow_mut(); let id = state.next_reader; state.next_reader += 1; - let base = state.base; - state.readers.insert(id, base); + state.unread.insert(id, VecDeque::new()); id }; Self { @@ -179,9 +359,13 @@ impl<'a, V> Input<'a, V> { impl Drop for Input<'_, V> { fn drop(&mut self) { let mut state = self.producer.borrow_mut(); - state.readers.remove(&self.reader); + state.unread.remove(&self.reader); state.waiters.remove(&self.reader); - state.trim(); + // A consumer may drop its input in `start`, before a later consumer + // subscribes; the stream is kept for that one. + if state.unread.is_empty() && state.next_reader as usize >= state.consumers { + state.stream = None; + } } } impl Stream for Input<'_, V> { @@ -193,84 +377,82 @@ impl Stream for Input<'_, V> { } let mut state = this.producer.borrow_mut(); state.context.register(cx.waker()); - if state.context.is_cancelled() { - state.failure = Some(Error::Cancelled); - state.finished = true; - state.stream = None; - state.queue.clear(); - } - let position = state.readers[&this.reader]; - let index = (position - state.base) as usize; - if let Some(value) = state.queue.get(index).cloned() { - state.readers.insert(this.reader, position + 1); - state.trim(); - return Poll::Ready(Some(Ok(value))); - } - if state.finished { - this.done = true; - state.readers.remove(&this.reader); - let failure = state.failure.clone(); - state.trim(); - return Poll::Ready(failure.map(Err)); - } - state.waiters.insert(this.reader, cx.waker().clone()); - if state.queue.len() >= state.context.control.limits.max_buffered_batches { - return Poll::Pending; - } - // Always-ready sources must still give cancellation and other roots a turn. - if state.batches_polled >= 32 { - state.batches_polled = 0; - cx.waker().wake_by_ref(); - return Poll::Pending; - } - let polled = state - .stream - .as_mut() - .expect("unfinished producer") - .as_mut() - .poll_next(cx); - if matches!(&polled, Poll::Ready(Some(Ok(_)))) { - state.batches_polled += 1; - } - match polled { - Poll::Pending => Poll::Pending, - Poll::Ready(Some(Ok(value))) => match state.context.reserve((state.size)(&value)) { - Ok(reservation) => { - let value = SharedValue { - value: Arc::new(value), - _reservation: Rc::new(reservation), - }; - state.queue.push_back(value.clone()); - state.readers.insert(this.reader, position + 1); - state.trim(); - Poll::Ready(Some(Ok(value))) + loop { + if state.context.is_cancelled() && state.failure != Some(Error::Cancelled) { + state.cancel(); + // The others; a producer being polled up the stack sees the + // cancellation itself. + let run_producers = Rc::clone(&state.run_producers); + for producer in run_producers.borrow().iter().filter_map(Weak::upgrade) { + if let Ok(mut producer) = producer.try_borrow_mut() { + producer.cancel(); + } } - Err(error) => { - state.failure = Some(error.clone()); - state.finished = true; - state.stream = None; - this.done = true; - state.readers.remove(&this.reader); - state.trim(); - Poll::Ready(Some(Err(error))) + } + if state.finished || !state.hold_until_finished { + let unread = state + .unread + .get_mut(&this.reader) + .expect("subscribed reader"); + if let Some(value) = unread.pop_front() { + return Poll::Ready(Some(Ok(value))); } - }, - Poll::Ready(result) => { - let error = result.and_then(Result::err).map(|source| match source { - Error::AtNode { .. } | Error::Cancelled | Error::MemoryLimit => source, - source => Error::AtNode { - node: state.node, - operation: state.operation.clone(), - source: Box::new(source), - }, - }); - state.failure = error.clone(); - state.finished = true; - state.stream = None; + } + if state.finished { this.done = true; - state.readers.remove(&this.reader); - state.trim(); - Poll::Ready(error.map(Err)) + state.unread.remove(&this.reader); + return Poll::Ready(state.failure.clone().map(Err)); + } + // Always-ready sources must still give cancellation and other roots a turn. + if state.batches_polled >= 32 { + state.batches_polled = 0; + cx.waker().wake_by_ref(); + return Poll::Pending; + } + let polled = state + .stream + .as_mut() + .expect("unfinished producer") + .as_mut() + .poll_next(cx); + match polled { + Poll::Pending => { + // Any reader's progress wakes this one: the upstream + // wakes only the reader that polled it last. + state.waiters.insert(this.reader, cx.waker().clone()); + return Poll::Pending; + } + Poll::Ready(Some(Ok(value))) => { + state.batches_polled += 1; + match state.context.reserve((state.size)(&value)) { + Ok(reservation) => { + let value = SharedValue { + value: Arc::new(value), + _reservation: Rc::new(reservation), + }; + for unread in state.unread.values_mut() { + unread.push_back(value.clone()); + } + // A held output is read only once it has + // finished, which wakes every reader. + if !state.hold_until_finished { + state.wake_all(); + } + } + Err(error) => state.finish(Some(error)), + } + } + Poll::Ready(result) => { + let error = result.and_then(Result::err).map(|source| match source { + Error::AtNode { .. } | Error::Cancelled | Error::MemoryLimit => source, + source => Error::AtNode { + node: state.node, + operation: state.operation.clone(), + source: Box::new(source), + }, + }); + state.finish(error); + } } } } diff --git a/crates/executor/src/runtime/tests.rs b/crates/executor/src/runtime/tests.rs index 20816e952..7f40d940f 100644 --- a/crates/executor/src/runtime/tests.rs +++ b/crates/executor/src/runtime/tests.rs @@ -1,6 +1,7 @@ use super::*; use crate::plan::PhysicalOperator; use futures::{executor::block_on, stream, StreamExt}; +use planner_types::stages::StorageMedium; use std::cell::Cell; struct Source { @@ -61,15 +62,13 @@ impl PhysicalOperator for Identity { .boxed_local()) } } -fn context() -> RunContext { +/// A query run at time 100 with a byte budget of `max_bytes`. +fn query_context(max_bytes: usize) -> RunContext { RunContext::new( Scope::Query { evaluation_time_ms: 100, }, - Limits { - max_buffered_batches: 1, - max_bytes: 1024, - }, + Limits { max_bytes }, ) .unwrap() } @@ -88,80 +87,115 @@ fn source(fail: bool) -> (Source, Rc>, Rc>) { ) } -// A shared producer runs once, and the slow reader bounds producer progress. +const ONE_EVALUATION: Lifetime = Lifetime::OneEvaluation(StorageMedium::Memory); + +// At query time each consumer of a not-materialized store read (a +// `Lifetime::None` leaf) reads it on its own, so one consumer never waits for +// another. #[test] -fn shared_source_backpressure_and_reader_drop() { +fn not_materialized_leaf_is_read_per_consumer() { let (source, starts, polls) = source(false); let mut dag = PhysicalDAG::default(); - dag.add(0, vec![], source).unwrap(); - let context = context(); + dag.add(0, vec![], source, Lifetime::None).unwrap(); + let context = query_context(1024); let mut readers = dag.execute(&[0, 0], context.clone()).unwrap(); - let mut slow = readers.pop().unwrap(); - let mut fast = readers.pop().unwrap(); - assert_eq!(starts.get(), 1); - let first = block_on(fast.next()).unwrap().unwrap(); - assert_eq!(*first, 0); - let mut cx = Context::from_waker(futures::task::noop_waker_ref()); - assert!(Pin::new(&mut fast).poll_next(&mut cx).is_pending()); - assert_eq!(polls.get(), 1); - let same = block_on(slow.next()).unwrap().unwrap(); - assert!(Arc::ptr_eq(&first.value, &same.value)); - drop(same); + let second = readers.pop().unwrap(); + let mut first = readers.pop().unwrap(); + assert_eq!(starts.get(), 2); + assert_eq!(*block_on(first.next()).unwrap().unwrap(), 0); + drop(second); + for expected in 1..4 { + assert_eq!(*block_on(first.next()).unwrap().unwrap(), expected); + } + assert!(block_on(first.next()).is_none()); + assert_eq!(polls.get(), 4); drop(first); assert_eq!(context.retained_bytes(), 0); - assert_eq!(*block_on(fast.next()).unwrap().unwrap(), 1); - drop(slow); - assert_eq!(*block_on(fast.next()).unwrap().unwrap(), 2); - assert_eq!(*block_on(fast.next()).unwrap().unwrap(), 3); - assert!(block_on(fast.next()).is_none()); - assert_eq!(polls.get(), 4); - drop(fast); +} + +// A leaf whose rows the store keeps across evaluations (an +// `AcrossEvaluations` raw data scan) is read from the store by each +// consumer on its own: nothing is held in the run. Example: two queries over +// one retained scan are two store reads, and the first query's first batch +// is the only one charged while it reads it. +#[test] +fn stored_leaf_is_read_per_consumer() { + let (source, starts, polls) = source(false); + let mut dag = PhysicalDAG::default(); + dag.add( + 0, + vec![], + source, + Lifetime::AcrossEvaluations(StorageMedium::Memory), + ) + .unwrap(); + let context = query_context(1024); + let mut readers = dag.execute(&[0, 0], context.clone()).unwrap(); + let second = readers.pop().unwrap(); + let mut first = readers.pop().unwrap(); + let batch = block_on(first.next()).unwrap().unwrap(); + assert_eq!(*batch, 0); + assert_eq!(context.retained_bytes(), 8); + drop(batch); + assert_eq!( + values(block_on(first.collect::>())), + vec![Ok(1), Ok(2), Ok(3)] + ); + assert_eq!( + values(block_on(second.collect::>())), + vec![Ok(0), Ok(1), Ok(2), Ok(3)] + ); + assert_eq!((starts.get(), polls.get()), (2, 8)); assert_eq!(context.retained_bytes(), 0); } -// Independent branches consume a common node concurrently without duplicate work. +// A materialized node runs once per run, to the end, and every consumer +// reads its finished output, charged to the run until all have read it. #[test] -fn diamond_and_run_isolation() { +fn materialized_node_runs_once_and_is_held() { let (source, starts, polls) = source(false); let mut dag = PhysicalDAG::default(); - dag.add(0, vec![], source).unwrap(); - dag.add(1, vec![0], Identity).unwrap(); - dag.add(2, vec![0], Identity).unwrap(); + dag.add(0, vec![], source, Lifetime::None).unwrap(); + dag.add(1, vec![0], Identity, ONE_EVALUATION).unwrap(); + dag.add(2, vec![1], Identity, Lifetime::None).unwrap(); + dag.add(3, vec![1], Identity, Lifetime::None).unwrap(); for _ in 0..2 { - let mut outputs = dag.execute(&[1, 2], context()).unwrap(); - let a = outputs.pop().unwrap(); - let b = outputs.pop().unwrap(); - let (a, b) = - block_on(async { futures::join!(a.collect::>(), b.collect::>()) }); - assert_eq!( - a.iter().map(|v| **v.as_ref().unwrap()).collect::>(), - vec![0, 1, 2, 3] - ); - assert_eq!( - b.iter().map(|v| **v.as_ref().unwrap()).collect::>(), - vec![0, 1, 2, 3] - ); + let run = query_context(1024); + let outputs = dag.execute(&[2, 3], run.clone()).unwrap(); + for output in outputs { + assert_eq!( + values(block_on(output.collect::>())), + vec![Ok(0), Ok(1), Ok(2), Ok(3)] + ); + } + // The held output, and the first root's batches collected while the + // second root has not read it yet. + assert_eq!(run.peak_bytes(), 2 * 4 * 8); + assert_eq!(run.retained_bytes(), 0); } assert_eq!(starts.get(), 2); assert_eq!(polls.get(), 8); } -// Failure reaches every subscriber; cancellation stops further producer work. +// Failure reaches every consumer, and a held node that fails partway is not +// read in part: each consumer gets only the error, so none reports success +// after its first batch. Cancellation stops further producer work. #[test] fn broadcast_error_and_cancel() { let (source, _, polls) = source(true); let mut dag = PhysicalDAG::default(); - dag.add(0, vec![], source).unwrap(); - let mut outputs = dag.execute(&[0, 0], context()).unwrap(); + dag.add(0, vec![], source, Lifetime::None).unwrap(); + dag.add(1, vec![0], Identity, ONE_EVALUATION).unwrap(); + let mut outputs = dag.execute(&[1, 1], query_context(1024)).unwrap(); let a = outputs.pop().unwrap(); let b = outputs.pop().unwrap(); let (a, b) = block_on(async { futures::join!(a.collect::>(), b.collect::>()) }); for values in [a, b] { - assert_eq!(values.len(), 2); - assert!(matches!(values[1], Err(Error::AtNode { node: 0, .. }))); + assert_eq!(values.len(), 1); + assert!(matches!(values[0], Err(Error::AtNode { node: 0, .. }))); } assert_eq!(polls.get(), 2); - let run = context(); + let run = query_context(1024); let mut output = dag.execute(&[0], run.clone()).unwrap().remove(0); run.cancel(); assert!(matches!( @@ -172,22 +206,13 @@ fn broadcast_error_and_cancel() { assert_eq!(polls.get(), 2); } -// Retaining a consumer output retains its budget lease after queue eviction. +// Retaining a consumer output retains its budget lease after it is read. #[test] fn retained_outputs_count_against_budget() { let (source, _, _) = source(false); let mut dag = PhysicalDAG::default(); - dag.add(0, vec![], source).unwrap(); - let run = RunContext::new( - Scope::Query { - evaluation_time_ms: 0, - }, - Limits { - max_buffered_batches: 1, - max_bytes: 8, - }, - ) - .unwrap(); + dag.add(0, vec![], source, Lifetime::None).unwrap(); + let run = query_context(8); let mut input = dag.execute(&[0], run.clone()).unwrap().remove(0); let held = block_on(input.next()).unwrap().unwrap(); assert_eq!(run.retained_bytes(), 8); @@ -206,16 +231,16 @@ fn retained_outputs_count_against_budget() { fn invalid_dags_do_not_start_sources() { let (source, starts, _) = source(false); let mut dag = PhysicalDAG::default(); - dag.add(0, vec![], source).unwrap(); - dag.add(1, vec![2], Identity).unwrap(); - dag.add(2, vec![1], Identity).unwrap(); - assert!(dag.execute(&[0, 1], context()).is_err()); + dag.add(0, vec![], source, Lifetime::None).unwrap(); + dag.add(1, vec![2], Identity, Lifetime::None).unwrap(); + dag.add(2, vec![1], Identity, Lifetime::None).unwrap(); + assert!(dag.execute(&[0, 1], query_context(1024)).is_err()); assert_eq!(starts.get(), 0); let mut missing = PhysicalDAG::default(); - missing.add(1, vec![9], Identity).unwrap(); + missing.add(1, vec![9], Identity, Lifetime::None).unwrap(); assert!(missing.validate(&[1]).is_err()); let mut arity = PhysicalDAG::default(); - arity.add(1, vec![], Identity).unwrap(); + arity.add(1, vec![], Identity, Lifetime::None).unwrap(); assert!(arity.validate(&[1]).is_err()); } @@ -225,8 +250,8 @@ fn ready_sources_cooperate_with_cancellation() { let (mut source, _, polls) = source(false); source.end = 10_000; let mut dag = PhysicalDAG::default(); - dag.add(0, vec![], source).unwrap(); - let context = context(); + dag.add(0, vec![], source, Lifetime::None).unwrap(); + let context = query_context(1024); let mut input = dag.execute(&[0], context.clone()).unwrap().remove(0); block_on(async { let drain = async { @@ -252,9 +277,423 @@ fn ready_sources_cooperate_with_cancellation() { fn depth_limit_covers_shared_paths() { let (source, _, _) = source(false); let mut dag = PhysicalDAG::default(); - dag.add(0, vec![], source).unwrap(); + dag.add(0, vec![], source, Lifetime::None).unwrap(); for id in 1..129 { - dag.add(id, vec![id - 1], Identity).unwrap(); + dag.add(id, vec![id - 1], Identity, Lifetime::None).unwrap(); } assert!(dag.validate(&(0..129).collect::>()).is_err()); } + +// Reads its left input to the end before its right, as `SeriesBinary` does. +struct LeftThenRight; +impl PhysicalOperator for LeftThenRight { + fn name(&self) -> &str { + "LeftThenRight" + } + fn input_schemas(&self) -> Vec<()> { + vec![(), ()] + } + fn output_schema(&self) {} + fn output_bytes(&self, _: &u64) -> usize { + 8 + } + fn start<'a>( + &'a self, + inputs: Vec>, + _: RunContext, + ) -> Result, Error> { + Ok(stream::iter(inputs) + .flatten() + .map(|value| value.map(|v| *v)) + .boxed_local()) + } +} + +// Reads its input one minute before the run's evaluation time, like a +// `TimeShift` with an `@` anchor. +struct MinuteEarlier; +impl PhysicalOperator for MinuteEarlier { + fn name(&self) -> &str { + "MinuteEarlier" + } + fn input_context(&self, context: &RunContext) -> Result { + Ok(context.at_evaluation_time(context.evaluation_time_ms().unwrap() - 60_000)) + } + fn input_schemas(&self) -> Vec<()> { + vec![()] + } + fn output_schema(&self) {} + fn output_bytes(&self, _: &u64) -> usize { + 8 + } + fn start<'a>( + &'a self, + mut inputs: Vec>, + _: RunContext, + ) -> Result, Error> { + Ok(inputs + .remove(0) + .map(|value| value.map(|v| *v)) + .boxed_local()) + } +} + +/// Runs `run` on its own thread and fails the test if it does not return in +/// time, so a deadlocked run fails instead of hanging the test binary. +fn within_timeout(run: impl FnOnce() -> T + Send + 'static) -> T { + let (sender, receiver) = std::sync::mpsc::channel(); + std::thread::spawn(move || sender.send(run()).unwrap()); + receiver + .recv_timeout(std::time::Duration::from_secs(10)) + .expect("the run did not finish") +} + +/// More batches than any queue bound the runtime ever had (8 by default). +const MANY_BATCHES: u64 = 20; +/// Node 0, with plan lifetime `leaf`, emits `MANY_BATCHES` batches; nodes 1 +/// and 2 each read it. +fn two_readers_of_one_leaf( + dag: &mut PhysicalDAG<'static, u64, ()>, + leaf: Lifetime, +) -> Rc> { + let (mut source, starts, _) = source(false); + source.end = MANY_BATCHES; + dag.add(0, vec![], source, leaf).unwrap(); + dag.add(1, vec![0], Identity, Lifetime::None).unwrap(); + dag.add(2, vec![0], Identity, Lifetime::None).unwrap(); + starts +} +fn ingestion_context(max_bytes: usize) -> RunContext { + RunContext::new( + Scope::Ingestion { + window_start_ms: 0, + window_end_ms: 10, + }, + Limits { max_bytes }, + ) + .unwrap() +} +fn values(results: Vec, Error>>) -> Vec> { + results.into_iter().map(|r| r.map(|v| *v)).collect() +} +fn all_batches() -> Vec> { + (0..MANY_BATCHES).map(Ok).collect() +} + +// Two roots reading one leaf: draining them one after another gives the +// same answers as polling them together. A leaf whose rows the store keeps +// (`None` or `AcrossEvaluations`) is read once per consumer (2 reads per +// run); a `OneEvaluation` leaf is read once and held for both (1 read per +// run). +#[test] +fn roots_reading_one_leaf_drain_in_order() { + for (leaf, reads_per_run) in [ + (Lifetime::None, 2), + (Lifetime::AcrossEvaluations(StorageMedium::Memory), 2), + (ONE_EVALUATION, 1), + ] { + let reads = within_timeout(move || { + let mut dag = PhysicalDAG::default(); + let starts = two_readers_of_one_leaf(&mut dag, leaf); + let outputs = dag.execute(&[1, 2], query_context(1024)).unwrap(); + let joined: Vec<_> = block_on(futures::future::join_all( + outputs.into_iter().map(|output| output.collect::>()), + )) + .into_iter() + .map(values) + .collect(); + assert_eq!(joined, vec![all_batches(), all_batches()]); + let run = query_context(1024); + let outputs = dag.execute(&[1, 2], run.clone()).unwrap(); + let drained: Vec<_> = outputs + .into_iter() + .map(|output| values(block_on(output.collect::>()))) + .collect(); + assert_eq!(drained, joined); + assert_eq!(run.retained_bytes(), 0); + starts.get() + }); + assert_eq!(reads, 2 * reads_per_run, "{leaf:?}"); + } +} + +// One root with two branches over one leaf, combined by an operator that +// reads its left branch to the end first. +#[test] +fn one_root_with_two_branches_over_one_leaf() { + for (leaf, reads) in [(Lifetime::None, 2), (ONE_EVALUATION, 1)] { + let starts = within_timeout(move || { + let mut dag = PhysicalDAG::default(); + let starts = two_readers_of_one_leaf(&mut dag, leaf); + dag.add(3, vec![1, 2], LeftThenRight, Lifetime::None) + .unwrap(); + let output = dag.execute(&[3], query_context(1024)).unwrap().remove(0); + let both: Vec<_> = all_batches().into_iter().chain(all_batches()).collect(); + assert_eq!(values(block_on(output.collect::>())), both); + starts.get() + }); + assert_eq!(starts, reads, "{leaf:?}"); + } +} + +// A materialized node with one consumer streams to it: no sibling can wait +// on that consumer, so nothing is held. Its first batch arrives after one +// source batch, not after the whole output. +#[test] +fn one_evaluation_node_with_one_consumer_streams() { + let (source, _, polls) = source(false); + let mut dag = PhysicalDAG::default(); + dag.add(0, vec![], source, ONE_EVALUATION).unwrap(); + dag.add(1, vec![0], Identity, ONE_EVALUATION).unwrap(); + let run = query_context(1024); + let mut output = dag.execute(&[1], run.clone()).unwrap().remove(0); + assert_eq!(*block_on(output.next()).unwrap().unwrap(), 0); + assert_eq!(polls.get(), 1); + assert_eq!(run.peak_bytes(), 8); +} + +// At ingestion time one read of a source feeds several chain builds: the +// runtime polls every root itself and each batch reaches every consumer. +#[test] +fn ingestion_fan_out_reads_the_source_once() { + within_timeout(|| { + let mut dag = PhysicalDAG::default(); + let starts = two_readers_of_one_leaf(&mut dag, Lifetime::None); + dag.add(3, vec![0], Identity, Lifetime::None).unwrap(); + dag.add(4, vec![1, 2], LeftThenRight, Lifetime::None) + .unwrap(); + let run = ingestion_context(1024); + let outputs = block_on(dag.execute_ingestion(&[3, 4], run.clone()).unwrap()).unwrap(); + let outputs: Vec>> = outputs + .into_iter() + .map(|batches| batches.into_iter().map(|v| Ok(*v)).collect()) + .collect(); + let both: Vec<_> = all_batches().into_iter().chain(all_batches()).collect(); + assert_eq!(outputs, vec![all_batches(), both]); + assert_eq!(starts.get(), 1); + assert_eq!(run.retained_bytes(), 0); + }); +} + +// Each entry point runs one scope and names the other for the other scope. +#[test] +fn execute_rejects_ingestion_scope_and_vice_versa() { + let mut dag = PhysicalDAG::default(); + two_readers_of_one_leaf(&mut dag, Lifetime::None); + let Err(error) = dag.execute(&[1], ingestion_context(1024)) else { + panic!("`execute` ran an ingestion scope"); + }; + assert_eq!( + error, + Error::Invalid( + "`execute` takes a query scope; run an ingestion scope with `execute_ingestion`".into() + ) + ); + let Err(error) = dag.execute_ingestion(&[1], query_context(1024)) else { + panic!("`execute_ingestion` ran a query scope"); + }; + assert_eq!( + error, + Error::Invalid( + "`execute_ingestion` takes an ingestion scope; run a query scope with `execute`".into() + ) + ); +} + +// A computed node that is not materialized has exactly one consumer at +// query time; two mean a planner bug, reported before any operator starts. +#[test] +fn query_time_fan_out_of_a_not_materialized_node_is_an_error() { + let (source, starts, _) = source(false); + let mut dag = PhysicalDAG::default(); + dag.add(0, vec![], source, Lifetime::None).unwrap(); + dag.add(1, vec![0], Identity, Lifetime::None).unwrap(); + dag.add(2, vec![1], Identity, Lifetime::None).unwrap(); + dag.add(3, vec![1], Identity, Lifetime::None).unwrap(); + let Err(error) = dag.execute(&[2, 3], query_context(1024)) else { + panic!("a not-materialized node with two consumers ran"); + }; + assert_eq!( + error, + Error::Invalid( + "node 1 (Identity) at evaluation time 100 has 2 consumers but Lifetime::None; \ + a shared query-time output must be OneEvaluation (planner bug)" + .into() + ) + ); + assert_eq!(starts.get(), 0); + // Read at two evaluation times, it runs once per time, each with one consumer. + dag.add(4, vec![1], MinuteEarlier, Lifetime::None).unwrap(); + let outputs = dag.execute(&[2, 4], query_context(1024)).unwrap(); + for output in outputs { + assert_eq!(values(block_on(output.collect::>())).len(), 4); + } +} + +// A one-evaluation output over the run's byte budget fails with +// `MemoryLimit`; it is not spilled. +#[test] +fn one_evaluation_output_over_budget_is_a_memory_limit() { + within_timeout(|| { + let mut dag = PhysicalDAG::default(); + let (mut source, _, _) = source(false); + source.end = MANY_BATCHES; + dag.add(0, vec![], source, Lifetime::None).unwrap(); + dag.add(1, vec![0], Identity, ONE_EVALUATION).unwrap(); + let run = query_context(5 * 8); + let output = dag.execute(&[1], run.clone()).unwrap().remove(0); + let read = values(block_on(output.collect::>())); + assert_eq!(read.last(), Some(&Err(Error::MemoryLimit))); + assert_eq!(run.retained_bytes(), 0); + }); +} + +// Ignores its input: drops it in `start` and emits nothing. +struct DropsInput; +impl PhysicalOperator for DropsInput { + fn name(&self) -> &str { + "DropsInput" + } + fn input_schemas(&self) -> Vec<()> { + vec![()] + } + fn output_schema(&self) {} + fn output_bytes(&self, _: &u64) -> usize { + 8 + } + fn start<'a>( + &'a self, + _: Vec>, + _: RunContext, + ) -> Result, Error> { + Ok(stream::empty().boxed_local()) + } +} + +// A consumer that drops its input in `start`, before the node's other +// consumer subscribes, leaves the node running for that one. +#[test] +fn an_input_dropped_at_start_leaves_the_node_to_later_consumers() { + let (source, _, _) = source(false); + let mut dag = PhysicalDAG::default(); + dag.add(0, vec![], source, Lifetime::None).unwrap(); + dag.add(1, vec![0], Identity, ONE_EVALUATION).unwrap(); + dag.add(2, vec![1], DropsInput, Lifetime::None).unwrap(); + dag.add(3, vec![1], Identity, Lifetime::None).unwrap(); + let run = query_context(1024); + let outputs = dag.execute(&[2, 3], run.clone()).unwrap(); + let drained: Vec<_> = outputs + .into_iter() + .map(|output| values(block_on(output.collect::>()))) + .collect(); + assert_eq!(drained, vec![vec![], vec![Ok(0), Ok(1), Ok(2), Ok(3)]]); + assert_eq!(run.retained_bytes(), 0); +} + +// A cancellation seen by one root releases what a held node keeps for a +// root nobody polls, and every root then ends with `Cancelled`. +#[test] +fn cancellation_releases_held_batches_of_unpolled_roots() { + let (mut source, _, _) = source(false); + source.end = MANY_BATCHES; + let mut dag = PhysicalDAG::default(); + dag.add(0, vec![], source, Lifetime::None).unwrap(); + dag.add(1, vec![0], Identity, ONE_EVALUATION).unwrap(); + dag.add(2, vec![1], Identity, Lifetime::None).unwrap(); + dag.add(3, vec![1], Identity, Lifetime::None).unwrap(); + let run = query_context(1024); + let mut outputs = dag.execute(&[2, 3], run.clone()).unwrap(); + let first = block_on(outputs[0].next()).unwrap().unwrap(); + assert_eq!(*first, 0); + drop(first); + assert!(run.retained_bytes() > 0); + run.cancel(); + assert!(matches!( + block_on(outputs[0].next()), + Some(Err(Error::Cancelled)) + )); + assert_eq!(run.retained_bytes(), 0); + assert!(matches!( + block_on(outputs[1].next()), + Some(Err(Error::Cancelled)) + )); +} + +/// Passes its input through and cancels the run from inside its own poll +/// once it has passed `after` batches, as an operator that hits a run-wide +/// stop condition would. +struct CancelsAfter { + after: usize, +} +impl PhysicalOperator for CancelsAfter { + fn name(&self) -> &str { + "CancelsAfter" + } + fn input_schemas(&self) -> Vec<()> { + vec![()] + } + fn output_schema(&self) {} + fn output_bytes(&self, _: &u64) -> usize { + 8 + } + fn start<'a>( + &'a self, + mut inputs: Vec>, + context: RunContext, + ) -> Result, Error> { + let mut passed = 0; + Ok(inputs + .remove(0) + .map(move |value| { + passed += 1; + if passed == self.after { + context.cancel(); + } + value.map(|value| *value) + }) + .boxed_local()) + } +} +/// A source, node 1 (`CancelsAfter` 3 batches, with lifetime `shared`) and +/// two `Identity` roots 2 and 3 over node 1. +fn cancelled_inside_a_shared_node(shared: Lifetime) -> PhysicalDAG<'static, u64, ()> { + let (mut source, _, _) = source(false); + source.end = MANY_BATCHES; + let mut dag = PhysicalDAG::default(); + dag.add(0, vec![], source, Lifetime::None).unwrap(); + dag.add(1, vec![0], CancelsAfter { after: 3 }, shared) + .unwrap(); + dag.add(2, vec![1], Identity, Lifetime::None).unwrap(); + dag.add(3, vec![1], Identity, Lifetime::None).unwrap(); + dag +} + +// An operator that cancels the run inside its own poll, while another +// consumer of its node is not being polled, ends every root with +// `Cancelled`: stopping the other producers does not drop their streams, +// whose inputs would borrow the producer being polled. +#[test] +fn cancel_inside_a_held_node_ends_every_root() { + let dag = cancelled_inside_a_shared_node(ONE_EVALUATION); + let run = query_context(1 << 20); + let outputs = dag.execute(&[2, 3], run.clone()).unwrap(); + let drained: Vec<_> = outputs + .into_iter() + .map(|output| values(block_on(output.collect::>()))) + .collect(); + assert_eq!( + drained, + vec![vec![Err(Error::Cancelled)], vec![Err(Error::Cancelled)]] + ); + assert_eq!(run.retained_bytes(), 0); +} + +// The same at ingestion time, over a streamed fan-out node. +#[test] +fn cancel_inside_an_ingestion_fan_out_ends_the_run() { + let dag = cancelled_inside_a_shared_node(Lifetime::None); + let run = ingestion_context(1 << 20); + let outputs = block_on(dag.execute_ingestion(&[2, 3], run.clone()).unwrap()); + assert!(matches!(outputs, Err(Error::Cancelled))); + assert_eq!(run.retained_bytes(), 0); +} diff --git a/crates/executor/tests/aggregate_memory_limit.rs b/crates/executor/tests/aggregate_memory_limit.rs index ac4a75085..d3ece60a5 100644 --- a/crates/executor/tests/aggregate_memory_limit.rs +++ b/crates/executor/tests/aggregate_memory_limit.rs @@ -9,6 +9,7 @@ use asap_executor::runtime::{Limits, RunContext, Scope}; use asap_executor::values::{Batch, BatchColumn}; use asap_executor::Error; use futures::{executor::block_on, StreamExt}; +use planner_types::stages::Lifetime; use std::alloc::{GlobalAlloc, Layout, System}; use std::sync::atomic::{AtomicUsize, Ordering}; use std::sync::{Arc, Mutex}; @@ -87,12 +88,18 @@ fn run_aggregate( }) .collect(); let mut dag = PhysicalDAG::default(); - dag.add(0, vec![], Operator::source(input.clone(), batches).unwrap()) - .unwrap(); + dag.add( + 0, + vec![], + Operator::source(input.clone(), batches).unwrap(), + Lifetime::None, + ) + .unwrap(); dag.add( 1, vec![0], Operator::aggregate(input, vec![0], measures).unwrap(), + Lifetime::None, ) .unwrap(); let context = RunContext::new( @@ -101,7 +108,6 @@ fn run_aggregate( }, Limits { max_bytes: limit_bytes, - ..Limits::default() }, ) .unwrap(); diff --git a/crates/executor/tests/blocking_resources.rs b/crates/executor/tests/blocking_resources.rs index b7f7409c8..b783608f9 100644 --- a/crates/executor/tests/blocking_resources.rs +++ b/crates/executor/tests/blocking_resources.rs @@ -12,6 +12,7 @@ use planner_types::ir::scalar::ScalarValue; use planner_types::ir::schema::{DataType, Field, FieldDataType}; use planner_types::ir::Predicate; use planner_types::ir::ScalarExpr; +use planner_types::stages::Lifetime; use std::sync::Arc; fn schema(width: usize) -> SchemaRef { @@ -33,10 +34,7 @@ fn context(max_bytes: usize) -> RunContext { Scope::Query { evaluation_time_ms: 0, }, - Limits { - max_bytes, - ..Limits::default() - }, + Limits { max_bytes }, ) .unwrap() } @@ -50,6 +48,7 @@ fn source(n: usize) -> PhysicalDAG<'static, Batch, SchemaRef> { vec![Batch::try_new(schema(1), vec![vec![Value::Int64(1)]; n]).unwrap()], ) .unwrap(), + Lifetime::None, ) .unwrap(); dag @@ -159,8 +158,13 @@ fn sort_with_fetch_holds_groups_times_fetch_rows() { }) .collect(); let mut dag = PhysicalDAG::default(); - dag.add(0, vec![], Operator::source(input.clone(), batches).unwrap()) - .unwrap(); + dag.add( + 0, + vec![], + Operator::source(input.clone(), batches).unwrap(), + Lifetime::None, + ) + .unwrap(); let key = SortKey { column: 1, descending: true, @@ -170,6 +174,7 @@ fn sort_with_fetch_holds_groups_times_fetch_rows() { 1, vec![0], Operator::sort(input.clone(), vec![key], vec![0], fetch).unwrap(), + Lifetime::None, ) .unwrap(); let context = context(1024 * 1024); @@ -222,8 +227,13 @@ fn cooperative_sort_preserves_ties_across_chunks() { ) .unwrap(); let mut dag = PhysicalDAG::default(); - dag.add(0, vec![], Operator::source(schema(2), vec![batch]).unwrap()) - .unwrap(); + dag.add( + 0, + vec![], + Operator::source(schema(2), vec![batch]).unwrap(), + Lifetime::None, + ) + .unwrap(); dag.add( 1, vec![0], @@ -238,6 +248,7 @@ fn cooperative_sort_preserves_ties_across_chunks() { None, ) .unwrap(), + Lifetime::None, ) .unwrap(); let mut output = dag @@ -289,6 +300,7 @@ fn weighted_summary_build_yields_within_a_batch() { 0, vec![], Operator::source(input.clone(), vec![batch]).unwrap(), + Lifetime::None, ) .unwrap(); let family = FieldDataType::Sketch(SketchAlgorithm::CmsWithHeap); @@ -338,6 +350,7 @@ fn frequency_dictionary_enforces_memory_budget() { .unwrap()], ) .unwrap(), + Lifetime::None, ) .unwrap(); for (reduction, succeeds) in [ diff --git a/crates/executor/tests/cheap_measures_memory_limit.rs b/crates/executor/tests/cheap_measures_memory_limit.rs index 873286fef..997d96c7a 100644 --- a/crates/executor/tests/cheap_measures_memory_limit.rs +++ b/crates/executor/tests/cheap_measures_memory_limit.rs @@ -10,6 +10,7 @@ use asap_executor::runtime::{Limits, RunContext, Scope}; use asap_executor::values::{Batch, BatchColumn}; use asap_executor::Error; use futures::{executor::block_on, StreamExt}; +use planner_types::stages::Lifetime; use std::alloc::{GlobalAlloc, Layout, System}; use std::sync::atomic::{AtomicUsize, Ordering}; use std::sync::Arc; @@ -47,22 +48,21 @@ fn aggregate( 0, vec![], Operator::source(input.clone(), vec![batch]).unwrap(), + Lifetime::None, ) .unwrap(); dag.add( 1, vec![0], Operator::aggregate(input, groups, measures).unwrap(), + Lifetime::None, ) .unwrap(); let context = RunContext::new( Scope::Query { evaluation_time_ms: 0, }, - Limits { - max_bytes, - ..Limits::default() - }, + Limits { max_bytes }, ) .unwrap(); let baseline = ALLOCATED.load(Ordering::Relaxed); diff --git a/crates/executor/tests/common/mod.rs b/crates/executor/tests/common/mod.rs index a3713995e..ababbefbc 100644 --- a/crates/executor/tests/common/mod.rs +++ b/crates/executor/tests/common/mod.rs @@ -1,5 +1,8 @@ #![allow(dead_code)] +use asap_executor::runtime::{RunContext, Scope}; +use asap_executor::values::{Batch, SchemaRef}; +use futures::TryStreamExt; use planner_types::ir::{OperatorNode, QueryRoot}; use std::rc::Rc; @@ -51,3 +54,29 @@ fn run_all_stages( ) .expect("Pass 1 accepts the query") } + +/// Every root's batches in one run of `dag`, in `roots` order: through +/// `execute` at query time (each root drained in turn) and +/// `execute_ingestion` at ingestion time. +pub fn run_roots( + dag: &asap_executor::plan::PhysicalDAG<'_, Batch, SchemaRef>, + roots: &[u64], + context: RunContext, +) -> Result>, asap_executor::Error> { + futures::executor::block_on(async { + let outputs = match context.scope { + Scope::Ingestion { .. } => dag.execute_ingestion(roots, context)?.await?, + Scope::Query { .. } => { + let mut outputs = Vec::new(); + for output in dag.execute(roots, context)? { + outputs.push(output.try_collect::>().await?); + } + outputs + } + }; + Ok(outputs + .into_iter() + .map(|batches| batches.iter().map(|batch| (**batch).clone()).collect()) + .collect()) + }) +} diff --git a/crates/executor/tests/current_series_heap.rs b/crates/executor/tests/current_series_heap.rs index 139941149..f3bc40193 100644 --- a/crates/executor/tests/current_series_heap.rs +++ b/crates/executor/tests/current_series_heap.rs @@ -258,7 +258,6 @@ fn current_series_observes_resource_limits() { }, Limits { max_bytes: if cancelled { 1 << 20 } else { 1 }, - ..Limits::default() }, ) .unwrap(); diff --git a/crates/executor/tests/current_timestamp.rs b/crates/executor/tests/current_timestamp.rs index a55052c4b..947fba376 100644 --- a/crates/executor/tests/current_timestamp.rs +++ b/crates/executor/tests/current_timestamp.rs @@ -8,7 +8,7 @@ use asap_executor::{ sources::InMemoryDataStore, values::{Batch, Value}, }; -use futures::{executor::block_on, StreamExt}; + use planner_types::ir::operator::Source; use planner_types::ir::properties::ExecutionDataState; use planner_types::ir::scalar::{ArithmeticOpKind, CompareOpKind, ExprSemantics, ScalarValue}; @@ -110,22 +110,17 @@ fn kept_users(scope: Scope) -> Result, String> { .instantiate(Arc::new(data_store())) .unwrap(); let context = RunContext::new(scope, Limits::default()).unwrap(); - block_on(async { - let mut output = bound - .execute(&[0], context) - .map_err(|e| e.to_string())? - .remove(0); - let mut users = Vec::new(); - while let Some(batch) = output.next().await { - for row in batch.map_err(|e| e.to_string())?.to_rows() { - let Value::Int64(user) = row[1] else { - panic!("user {:?}", row[1]) - }; - users.push(user); - } - } - Ok(users) - }) + let batches = crate::common::run_roots(&bound, &[0], context) + .map_err(|e| e.to_string())? + .remove(0); + let mut users = Vec::new(); + for row in batches.iter().flat_map(Batch::to_rows) { + let Value::Int64(user) = row[1] else { + panic!("user {:?}", row[1]) + }; + users.push(user); + } + Ok(users) } #[test] diff --git a/crates/executor/tests/exact_state_columns.rs b/crates/executor/tests/exact_state_columns.rs index f8cae953a..caf10dbe9 100644 --- a/crates/executor/tests/exact_state_columns.rs +++ b/crates/executor/tests/exact_state_columns.rs @@ -9,6 +9,7 @@ use asap_executor::runtime::{Limits, RunContext, Scope}; use asap_executor::summary_kernels::exact::ExactEvaluation; use asap_executor::values::{Batch, SchemaRef, Value}; use futures::{executor::block_on, StreamExt}; +use planner_types::stages::Lifetime; use std::sync::Arc; fn field(name: &str, dtype: DataType) -> Field { @@ -66,9 +67,14 @@ fn context() -> RunContext { fn run(operator: Operator, inputs: Vec) -> Vec { let mut dag = PhysicalDAG::default(); let schema = inputs[0].schema().clone(); - dag.add(0, vec![], Operator::source(schema, inputs).unwrap()) - .unwrap(); - dag.add(1, vec![0], operator).unwrap(); + dag.add( + 0, + vec![], + Operator::source(schema, inputs).unwrap(), + Lifetime::None, + ) + .unwrap(); + dag.add(1, vec![0], operator, Lifetime::None).unwrap(); block_on(async { let mut output = dag.execute(&[1], context()).unwrap().remove(0); let mut batches = Vec::new(); diff --git a/crates/executor/tests/grouped_limit_memory_limit.rs b/crates/executor/tests/grouped_limit_memory_limit.rs index e1b03188a..4d9068176 100644 --- a/crates/executor/tests/grouped_limit_memory_limit.rs +++ b/crates/executor/tests/grouped_limit_memory_limit.rs @@ -9,6 +9,7 @@ use asap_executor::runtime::{Limits, RunContext, Scope}; use asap_executor::values::{Batch, BatchColumn}; use asap_executor::Error; use futures::{executor::block_on, StreamExt}; +use planner_types::stages::Lifetime; use std::alloc::{GlobalAlloc, Layout, System}; use std::sync::atomic::{AtomicUsize, Ordering}; use std::sync::Arc; @@ -60,17 +61,22 @@ fn many_groups_in_one_batch_fail_near_the_memory_limit() { 0, vec![], Operator::source(input.clone(), vec![batch]).unwrap(), + Lifetime::None, + ) + .unwrap(); + dag.add( + 1, + vec![0], + Operator::limit(input, 1, 0, vec![0]).unwrap(), + Lifetime::None, ) .unwrap(); - dag.add(1, vec![0], Operator::limit(input, 1, 0, vec![0]).unwrap()) - .unwrap(); let context = RunContext::new( Scope::Query { evaluation_time_ms: 0, }, Limits { max_bytes: LIMIT_BYTES, - ..Limits::default() }, ) .unwrap(); diff --git a/crates/executor/tests/keyed_summary_row_weight.rs b/crates/executor/tests/keyed_summary_row_weight.rs index cdf195bb8..fe1873d52 100644 --- a/crates/executor/tests/keyed_summary_row_weight.rs +++ b/crates/executor/tests/keyed_summary_row_weight.rs @@ -91,10 +91,15 @@ fn top_items(build: Operator, input: SchemaRef, batch: Batch) -> Vec<(String, St ]); let evaluation = Operator::keyed_evaluation(build.schema(), 1, 8, output).unwrap(); let mut dag = PhysicalDAG::default(); - dag.add(0, vec![], Operator::source(input, vec![batch]).unwrap()) - .unwrap(); - dag.add(1, vec![0], build).unwrap(); - dag.add(2, vec![1], evaluation).unwrap(); + dag.add( + 0, + vec![], + Operator::source(input, vec![batch]).unwrap(), + Lifetime::None, + ) + .unwrap(); + dag.add(1, vec![0], build, Lifetime::None).unwrap(); + dag.add(2, vec![1], evaluation, Lifetime::None).unwrap(); let context = RunContext::new( Scope::Query { evaluation_time_ms: 0, @@ -290,10 +295,11 @@ fn int64_weight_column_equals_float64_column() { vec![batch(&int64, Some(Value::Int64((1 << 53) + 1)))], ) .unwrap(), + Lifetime::None, ) .unwrap(); - dag.add(1, vec![0], build).unwrap(); - dag.add(2, vec![1], evaluation).unwrap(); + dag.add(1, vec![0], build, Lifetime::None).unwrap(); + dag.add(2, vec![1], evaluation, Lifetime::None).unwrap(); let context = RunContext::new( Scope::Query { evaluation_time_ms: 0, @@ -332,10 +338,16 @@ fn null_weight_adds_nothing() { 0, vec![], Operator::source(nullable.clone(), vec![nulls.clone()]).unwrap(), + Lifetime::None, + ) + .unwrap(); + dag.add( + 1, + vec![0], + keyed_build(&nullable, RowWeight::Column(2)), + Lifetime::None, ) .unwrap(); - dag.add(1, vec![0], keyed_build(&nullable, RowWeight::Column(2))) - .unwrap(); let context = RunContext::new( Scope::Query { evaluation_time_ms: 0, diff --git a/crates/executor/tests/physical_dag.rs b/crates/executor/tests/physical_dag.rs index 98c67cc59..5dbdc04e1 100644 --- a/crates/executor/tests/physical_dag.rs +++ b/crates/executor/tests/physical_dag.rs @@ -17,6 +17,7 @@ use planner_types::plan::physical_asap_dag::{ EdgeRole, GroupingEdgeCompatibility, PhysicalASAPDAG, PhysicalASAPDAGEdge, PhysicalASAPDAGNode, PhysicalASAPOperatorPayload, WindowEdgeCompatibility, }; +use planner_types::stages::{Lifetime, StorageMedium}; use planner_types::summary_type::{ExactKind, ExactParams}; use std::sync::Arc; fn schema(fields: &[(&str, DataType, bool)]) -> SchemaRef { @@ -72,23 +73,12 @@ fn with_table( } fn run(dag: &PhysicalDAG<'_, Batch, SchemaRef>, root: u64, scope: Scope) -> Vec> { - let context = RunContext::new( - scope, - Limits { - max_buffered_batches: 1, - ..Limits::default() - }, - ) - .unwrap(); - block_on(async { - let mut stream = dag.execute(&[root], context.clone()).unwrap().remove(0); - let mut rows = vec![]; - while let Some(batch) = stream.next().await { - rows.extend(batch.unwrap().to_rows().iter().cloned()); - } - assert_eq!(context.retained_bytes(), 0); - rows - }) + let context = RunContext::new(scope, Limits::default()).unwrap(); + let batches = crate::common::run_roots(dag, &[root], context.clone()) + .unwrap() + .remove(0); + assert_eq!(context.retained_bytes(), 0); + batches.iter().flat_map(Batch::to_rows).collect() } fn query() -> Scope { Scope::Query { @@ -134,6 +124,7 @@ fn grouped_sort_limit_across_batches() { 0, vec![], Operator::source(schema.clone(), batches).unwrap(), + Lifetime::None, ) .unwrap(); dag.add( @@ -150,10 +141,16 @@ fn grouped_sort_limit_across_batches() { None, ) .unwrap(), + Lifetime::None, + ) + .unwrap(); + dag.add( + 2, + vec![1], + Operator::limit(schema, 1, 1, vec![0]).unwrap(), + Lifetime::None, ) .unwrap(); - dag.add(2, vec![1], Operator::limit(schema, 1, 1, vec![0]).unwrap()) - .unwrap(); assert_eq!(floats(&run(&dag, 2, query()), 1), vec![5., 4.]); } @@ -169,15 +166,32 @@ fn summary_construction_merge_and_evaluation_at_both_phases() { Operator::per_group_summary_build(schema.clone(), family, None, 0, None, vec![]).unwrap(); let state = build.schema(); let mut dag = PhysicalDAG::default(); - dag.add(0, vec![], Operator::source(schema, batches).unwrap()) - .unwrap(); - dag.add(1, vec![0], build).unwrap(); - dag.add(2, vec![1, 1], Operator::union(state.clone(), 2).unwrap()) - .unwrap(); + dag.add( + 0, + vec![], + Operator::source(schema, batches).unwrap(), + Lifetime::None, + ) + .unwrap(); + dag.add( + 1, + vec![0], + build, + Lifetime::OneEvaluation(StorageMedium::Memory), + ) + .unwrap(); + dag.add( + 2, + vec![1, 1], + Operator::union(state.clone(), 2).unwrap(), + Lifetime::None, + ) + .unwrap(); dag.add( 3, vec![2], Operator::summary_merge(state.clone(), 0, vec![]).unwrap(), + Lifetime::None, ) .unwrap(); dag.add( @@ -195,6 +209,7 @@ fn summary_construction_merge_and_evaluation_at_both_phases() { ), ) .unwrap(), + Lifetime::None, ) .unwrap(); for scope in [ @@ -232,13 +247,15 @@ fn diamond_semijoin_preserves_left_values_and_multiplicity() { 0, vec![], Operator::source(schema.clone(), batches).unwrap(), + Lifetime::None, ) .unwrap(); - dag.add(1, vec![0], filter).unwrap(); + dag.add(1, vec![0], filter, Lifetime::None).unwrap(); dag.add( 2, vec![0, 1], Operator::semi_join(schema.clone(), schema, vec![(0, 0)]).unwrap(), + Lifetime::None, ) .unwrap(); let rows = run(&dag, 2, query()); @@ -270,19 +287,26 @@ fn exact_integer_and_empty_extrema() { .unwrap()], ) .unwrap(), + Lifetime::None, ) .unwrap(); - dag.add(1, vec![0], aggregate).unwrap(); + dag.add(1, vec![0], aggregate, Lifetime::None).unwrap(); assert!(matches!(run(&dag,1,query())[0][0],Value::Int64(v) if v==value+2)); let mut empty = PhysicalDAG::default(); empty - .add(0, vec![], Operator::source(schema.clone(), vec![]).unwrap()) + .add( + 0, + vec![], + Operator::source(schema.clone(), vec![]).unwrap(), + Lifetime::None, + ) .unwrap(); empty .add( 1, vec![0], Operator::aggregate(schema, vec![], vec![("min".into(), Reduction::Min(0))]).unwrap(), + Lifetime::None, ) .unwrap(); assert!(matches!(run(&empty, 1, query())[0][0], Value::Null)); @@ -302,9 +326,9 @@ fn scalar_negation_and_vector_conversion() { .unwrap(); let convert = Operator::vector_to_scalar(project.schema(), 0).unwrap(); let mut dag = PhysicalDAG::default(); - dag.add(0, vec![], scalar).unwrap(); - dag.add(1, vec![0], project).unwrap(); - dag.add(2, vec![1], convert).unwrap(); + dag.add(0, vec![], scalar, Lifetime::None).unwrap(); + dag.add(1, vec![0], project, Lifetime::None).unwrap(); + dag.add(2, vec![1], convert, Lifetime::None).unwrap(); assert_eq!(floats(&run(&dag, 2, query()), 0), vec![-7.]); let scalar = Operator::scalar(Value::Float64(f64::NAN), DataType::Float64).unwrap(); let predicate = Expression::Equal( @@ -313,8 +337,8 @@ fn scalar_negation_and_vector_conversion() { ); let filter = Operator::filter(scalar.schema(), predicate).unwrap(); let mut dag = PhysicalDAG::default(); - dag.add(0, vec![], scalar).unwrap(); - dag.add(1, vec![0], filter).unwrap(); + dag.add(0, vec![], scalar, Lifetime::None).unwrap(); + dag.add(1, vec![0], filter, Lifetime::None).unwrap(); assert!(run(&dag, 1, query()).is_empty()); } @@ -381,9 +405,10 @@ fn kll_raw_partial_and_precomputed_are_native_dags() { 0, vec![], Operator::source(input.clone(), vec![batch]).unwrap(), + Lifetime::None, ) .unwrap(); - dag.add(1, vec![0], build.clone()).unwrap(); + dag.add(1, vec![0], build.clone(), Lifetime::None).unwrap(); run( &dag, 1, @@ -407,6 +432,7 @@ fn kll_raw_partial_and_precomputed_are_native_dags() { vec![Batch::try_new(state.clone(), rows).unwrap()], ) .unwrap(), + Lifetime::None, ) .unwrap(); states.push(0); @@ -426,21 +452,24 @@ fn kll_raw_partial_and_precomputed_are_native_dags() { .unwrap()], ) .unwrap(), + Lifetime::None, ) .unwrap(); - dag.add(2, vec![1], build.clone()).unwrap(); + dag.add(2, vec![1], build.clone(), Lifetime::None).unwrap(); states.push(2); } dag.add( 3, states.clone(), Operator::union(state.clone(), states.len()).unwrap(), + Lifetime::None, ) .unwrap(); dag.add( 4, vec![3], Operator::summary_merge(state.clone(), 0, vec![]).unwrap(), + Lifetime::None, ) .unwrap(); dag.add( @@ -454,6 +483,7 @@ fn kll_raw_partial_and_precomputed_are_native_dags() { ), ) .unwrap(), + Lifetime::None, ) .unwrap(); floats(&run(&dag, 5, query()), 0)[0] @@ -496,6 +526,7 @@ fn exact_state_and_family_validation() { vec![Batch::try_new(schema.clone(), vec![vec![value]]).unwrap()], ) .unwrap(), + Lifetime::None, ) .unwrap(); dag.add( @@ -513,6 +544,7 @@ fn exact_state_and_family_validation() { ), ) .unwrap(), + Lifetime::None, ) .unwrap(); assert_eq!(floats(&run(&dag, 1, query()), 0), vec![7.]); @@ -640,10 +672,15 @@ fn empty_exact_count_is_an_integer_state_evaluation() { ) .unwrap(); let mut dag = PhysicalDAG::default(); - dag.add(0, vec![], Operator::source(input, vec![]).unwrap()) - .unwrap(); - dag.add(1, vec![0], build).unwrap(); - dag.add(2, vec![1], read).unwrap(); + dag.add( + 0, + vec![], + Operator::source(input, vec![]).unwrap(), + Lifetime::None, + ) + .unwrap(); + dag.add(1, vec![0], build, Lifetime::None).unwrap(); + dag.add(2, vec![1], read, Lifetime::None).unwrap(); assert!(matches!(run(&dag, 2, query())[0][0], Value::Int64(0))); } @@ -664,6 +701,7 @@ fn extrema_preserve_numeric_values_in_the_presence_of_nan() { .unwrap()], ) .unwrap(), + Lifetime::None, ) .unwrap(); dag.add( @@ -678,6 +716,7 @@ fn extrema_preserve_numeric_values_in_the_presence_of_nan() { ], ) .unwrap(), + Lifetime::None, ) .unwrap(); let rows = run(&dag, 1, query()); @@ -866,10 +905,11 @@ fn planner_expressions_preserve_collection_and_nullable_types() { .unwrap()], ) .unwrap(), + Lifetime::None, ) .unwrap(); let projected = project.schema(); - dag.add(1, vec![0], project).unwrap(); + dag.add(1, vec![0], project, Lifetime::None).unwrap(); let predicate = ScalarExpr::Compare { semantics: planner_types::ir::ExprSemantics::Sql, left: Box::new(ScalarExpr::Column(0)), @@ -884,6 +924,7 @@ fn planner_expressions_preserve_collection_and_nullable_types() { Expression::planner(CompiledExpression::compile(&predicate, &projected).unwrap()), ) .unwrap(), + Lifetime::None, ) .unwrap(); let rows = run(&dag, 2, query()); @@ -953,6 +994,7 @@ fn native_relational_join_kinds_preserve_unmatched_rows() { vec![Batch::try_new(input.clone(), rows).unwrap()], ) .unwrap(), + Lifetime::None, ) .unwrap(); } @@ -967,6 +1009,7 @@ fn native_relational_join_kinds_preserve_unmatched_rows() { output, ) .unwrap(), + Lifetime::None, ) .unwrap(); assert_eq!(run(&dag, 2, query()).len(), count, "{kind:?}"); @@ -1058,11 +1101,12 @@ fn assert_weighted_rate_topk(count_sketch: bool) { 0, vec![], Operator::source(raw.clone(), vec![Batch::try_new(raw, rows).unwrap()]).unwrap(), + Lifetime::None, ) .unwrap(); - dag.add(1, vec![0], rates).unwrap(); - dag.add(2, vec![1], build).unwrap(); - dag.add(3, vec![2], evaluation).unwrap(); + dag.add(1, vec![0], rates, Lifetime::None).unwrap(); + dag.add(2, vec![1], build, Lifetime::None).unwrap(); + dag.add(3, vec![2], evaluation, Lifetime::None).unwrap(); dag.add( 4, vec![3], @@ -1077,10 +1121,16 @@ fn assert_weighted_rate_topk(count_sketch: bool) { None, ) .unwrap(), + Lifetime::None, + ) + .unwrap(); + dag.add( + 5, + vec![4], + Operator::limit(output, 2, 0, vec![0]).unwrap(), + Lifetime::None, ) .unwrap(); - dag.add(5, vec![4], Operator::limit(output, 2, 0, vec![0]).unwrap()) - .unwrap(); for scope in [ query(), Scope::Ingestion { diff --git a/crates/executor/tests/physical_semantics.rs b/crates/executor/tests/physical_semantics.rs index c13d0ab70..f6958dc53 100644 --- a/crates/executor/tests/physical_semantics.rs +++ b/crates/executor/tests/physical_semantics.rs @@ -16,6 +16,7 @@ use planner_types::ir::NonASAPOp; use planner_types::ir::Predicate; use planner_types::ir::ScalarExpr; use planner_types::plan::physical_asap_dag::{PhysicalASAPDAGNode, PhysicalASAPOperatorPayload}; +use planner_types::stages::{Lifetime, StorageMedium}; use std::sync::Arc; fn schema(fields: &[(&str, DataType, bool)]) -> SchemaRef { @@ -38,10 +39,7 @@ fn context() -> RunContext { Scope::Query { evaluation_time_ms: 0, }, - Limits { - max_buffered_batches: 1, - ..Limits::default() - }, + Limits::default(), ) .unwrap() } @@ -64,9 +62,14 @@ fn unary(input: SchemaRef, batches: Vec>>, op: Operator) -> Vec]) -> Vec>> { @@ -108,10 +111,11 @@ fn join(left: Vec, right: Vec, kind: JoinKind, keyed: bool) -> Vec id, vec![], Operator::source(input.clone(), batches).unwrap(), + Lifetime::None, ) .unwrap(); } - dag.add(2, vec![0, 1], op).unwrap(); + dag.add(2, vec![0, 1], op, Lifetime::None).unwrap(); collect(&dag, 2) } @@ -258,18 +262,25 @@ fn sort_with_fetch_over_ascending_input_equals_sort_then_limit() { .into_iter() .map(|rows| Batch::try_new(input.clone(), rows).unwrap()) .collect(); - dag.add(0, vec![], Operator::source(input.clone(), source).unwrap()) - .unwrap(); + dag.add( + 0, + vec![], + Operator::source(input.clone(), source).unwrap(), + Lifetime::None, + ) + .unwrap(); dag.add( 1, vec![0], Operator::sort(input.clone(), key.clone(), groups.clone(), None).unwrap(), + Lifetime::None, ) .unwrap(); dag.add( 2, vec![1], Operator::limit(input.clone(), fetch, 0, groups.clone()).unwrap(), + Lifetime::None, ) .unwrap(); assert_eq!( @@ -359,18 +370,25 @@ fn sort_with_fetch_equals_sort_then_limit() { .into_iter() .map(|rows| Batch::try_new(input.clone(), rows).unwrap()) .collect(); - dag.add(0, vec![], Operator::source(input.clone(), source).unwrap()) - .unwrap(); + dag.add( + 0, + vec![], + Operator::source(input.clone(), source).unwrap(), + Lifetime::None, + ) + .unwrap(); dag.add( 1, vec![0], Operator::sort(input.clone(), key.clone(), groups.clone(), None).unwrap(), + Lifetime::None, ) .unwrap(); dag.add( 2, vec![1], Operator::limit(input.clone(), fetch, 0, groups.clone()).unwrap(), + Lifetime::None, ) .unwrap(); let limited = collect(&dag, 2); @@ -392,11 +410,17 @@ fn try_chain( ops: Vec, ) -> Result>, String> { let mut dag = PhysicalDAG::default(); - dag.add(0, vec![], Operator::source(input.clone(), batches).unwrap()) - .unwrap(); + dag.add( + 0, + vec![], + Operator::source(input.clone(), batches).unwrap(), + Lifetime::None, + ) + .unwrap(); let root = ops.len() as u64; for (index, op) in ops.into_iter().enumerate() { - dag.add(index as u64 + 1, vec![index as u64], op).unwrap(); + dag.add(index as u64 + 1, vec![index as u64], op, Lifetime::None) + .unwrap(); } block_on(async { let mut stream = dag @@ -587,13 +611,19 @@ fn identities_equal_but_for_an_excluded_label_aggregate_in_one_group() { .into_iter() .map(|rows| Batch::try_new(input.clone(), rows).unwrap()) .collect(); - dag.add(0, vec![], Operator::source(input.clone(), source).unwrap()) - .unwrap(); - dag.add(1, vec![0], project).unwrap(); + dag.add( + 0, + vec![], + Operator::source(input.clone(), source).unwrap(), + Lifetime::None, + ) + .unwrap(); + dag.add(1, vec![0], project, Lifetime::None).unwrap(); dag.add( 2, vec![1], Operator::aggregate(projected, vec![0], vec![("sum".into(), Reduction::Sum(1))]).unwrap(), + Lifetime::None, ) .unwrap(); let mut sums = collect(&dag, 2) @@ -836,47 +866,59 @@ fn planner_comparisons_handle_nan_without_execution_errors() { } } -// A bounded LIMIT branch must unsubscribe so another branch can drain the producer. +// At ingestion time a LIMIT consumer of a fan-out node drops its read once it +// has its rows: the node's later batches are not queued for it, and the +// sibling reading the same node still gets every batch. #[test] -fn limit_branch_finishes_without_blocking_shared_sibling() { +fn limit_consumer_of_an_ingestion_fan_out_drops_its_read() { let input = schema(&[("v", DataType::Int64, false)]); let mut dag = PhysicalDAG::default(); - let batches = (0..100) + let batches: Vec<_> = (0..100) .map(|v| Batch::try_new(input.clone(), vec![vec![Value::Int64(v)]]).unwrap()) .collect(); - dag.add(0, vec![], Operator::source(input.clone(), batches).unwrap()) - .unwrap(); + let batch_bytes = batches[0].memory_bytes(); + dag.add( + 0, + vec![], + Operator::source(input.clone(), batches).unwrap(), + Lifetime::None, + ) + .unwrap(); dag.add( 1, vec![0], Operator::limit(input.clone(), 1, 0, vec![]).unwrap(), + Lifetime::None, ) .unwrap(); - dag.add(2, vec![0, 1], Operator::union(input, 2).unwrap()) - .unwrap(); - // Bound polls as well as rows so a backpressure regression cannot hang the suite. - use futures::{task::noop_waker_ref, Stream}; - use std::{ - pin::Pin, - task::{Context, Poll}, - }; - let run = context(); - let mut stream = dag.execute(&[2], run.clone()).unwrap().remove(0); - let mut cx = Context::from_waker(noop_waker_ref()); - let mut count = 0; - for _ in 0..2000 { - match Pin::new(&mut stream).poll_next(&mut cx) { - Poll::Ready(Some(batch)) => count += batch.unwrap().to_rows().len(), - Poll::Ready(None) => { - assert_eq!(count, 101); - drop(stream); - assert_eq!(run.retained_bytes(), 0); - return; - } - Poll::Pending => {} - } - } - panic!("shared LIMIT/Union failed to make progress"); + dag.add( + 2, + vec![0, 1], + Operator::union(input, 2).unwrap(), + Lifetime::None, + ) + .unwrap(); + let run = RunContext::new( + Scope::Ingestion { + window_start_ms: 0, + window_end_ms: 1, + }, + Limits::default(), + ) + .unwrap(); + let outputs = block_on(dag.execute_ingestion(&[2], run.clone()).unwrap()).unwrap(); + let rows: usize = outputs[0].iter().map(|batch| batch.num_rows()).sum(); + assert_eq!(rows, 101); + // The run holds the 101 output batches it returns and no more: had the + // LIMIT kept its read, its queue would hold the 99 source batches it + // never reads. + assert!( + run.peak_bytes() <= (101 + 2) * batch_bytes, + "{}", + run.peak_bytes() + ); + drop(outputs); + assert_eq!(run.retained_bytes(), 0); } // Mixed numeric comparisons must not round Int64 values through f64 before comparing. @@ -977,6 +1019,7 @@ fn kll_partial_merge_and_multiple_evaluations_preserve_population() { vec![Batch::try_new(input.clone(), rows).unwrap()], ) .unwrap(), + Lifetime::None, ) .unwrap(); dag.add( @@ -991,18 +1034,25 @@ fn kll_partial_merge_and_multiple_evaluations_preserve_population() { vec![], ) .unwrap(), + Lifetime::OneEvaluation(StorageMedium::Memory), ) .unwrap(); } let state = Operator::per_group_summary_build(input, family, Some(kll_512()), 0, None, vec![]) .unwrap() .schema(); - dag.add(6, vec![3, 4], Operator::union(state.clone(), 2).unwrap()) - .unwrap(); + dag.add( + 6, + vec![3, 4], + Operator::union(state.clone(), 2).unwrap(), + Lifetime::None, + ) + .unwrap(); dag.add( 7, vec![6], Operator::summary_merge(state.clone(), 0, vec![]).unwrap(), + Lifetime::OneEvaluation(StorageMedium::Memory), ) .unwrap(); let mut roots = vec![]; @@ -1020,6 +1070,7 @@ fn kll_partial_merge_and_multiple_evaluations_preserve_population() { ), ) .unwrap(), + Lifetime::None, ) .unwrap(); roots.push(id); @@ -1097,12 +1148,14 @@ fn compacted_kll_tumbling_windows_merge_within_the_calibrated_rank_error() { vec![Batch::try_new(input.clone(), rows).unwrap()], ) .unwrap(), + Lifetime::None, ) .unwrap(); dag.add( TUMBLING_WINDOW_COUNT + tumbling_window, vec![tumbling_window], build(), + Lifetime::None, ) .unwrap(); } @@ -1111,12 +1164,14 @@ fn compacted_kll_tumbling_windows_merge_within_the_calibrated_rank_error() { union, (TUMBLING_WINDOW_COUNT..2 * TUMBLING_WINDOW_COUNT).collect(), Operator::union(state.clone(), TUMBLING_WINDOW_COUNT as usize).unwrap(), + Lifetime::None, ) .unwrap(); dag.add( union + 1, vec![union], Operator::summary_merge(state.clone(), 0, vec![]).unwrap(), + Lifetime::None, ) .unwrap(); for (i, q) in [0.01, 0.1, 0.25, 0.5, 0.75, 0.9, 0.99] @@ -1135,6 +1190,7 @@ fn compacted_kll_tumbling_windows_merge_within_the_calibrated_rank_error() { }), ) .unwrap(), + Lifetime::None, ) .unwrap(); let Value::Float64(estimate) = collect(&dag, id)[0][0] else { @@ -1193,16 +1249,23 @@ fn counter_tumbling_windows_merge_in_any_order() { id, vec![], Operator::source(input.clone(), vec![batch]).unwrap(), + Lifetime::None, ) .unwrap(); - dag.add(id + 4, vec![id], build()).unwrap(); + dag.add(id + 4, vec![id], build(), Lifetime::None).unwrap(); } - dag.add(8, vec![4, 5, 6], Operator::union(state.clone(), 3).unwrap()) - .unwrap(); + dag.add( + 8, + vec![4, 5, 6], + Operator::union(state.clone(), 3).unwrap(), + Lifetime::None, + ) + .unwrap(); dag.add( 9, vec![8], Operator::summary_merge(state, 0, vec![]).unwrap(), + Lifetime::None, ) .unwrap(); assert_eq!(increase(&dag, 9), increase(&dag, 7)); @@ -1215,16 +1278,18 @@ fn zero_column_output_obeys_memory_limit() { let input = schema(&[]); let batch = Batch::try_new(input.clone(), vec![vec![]; 200]).unwrap(); let mut dag = PhysicalDAG::default(); - dag.add(0, vec![], Operator::source(input, vec![batch]).unwrap()) - .unwrap(); + dag.add( + 0, + vec![], + Operator::source(input, vec![batch]).unwrap(), + Lifetime::None, + ) + .unwrap(); let run = RunContext::new( Scope::Query { evaluation_time_ms: 0, }, - Limits { - max_bytes: 1024, - ..Limits::default() - }, + Limits { max_bytes: 1024 }, ) .unwrap(); let mut stream = dag.execute(&[0], run.clone()).unwrap().remove(0); @@ -1257,9 +1322,14 @@ fn empty_exact_summary_extrema_agree_with_ordinary_aggregation() { .unwrap(); let state = build.schema(); let mut dag = PhysicalDAG::default(); - dag.add(0, vec![], Operator::source(input.clone(), vec![]).unwrap()) - .unwrap(); - dag.add(1, vec![0], build).unwrap(); + dag.add( + 0, + vec![], + Operator::source(input.clone(), vec![]).unwrap(), + Lifetime::None, + ) + .unwrap(); + dag.add(1, vec![0], build, Lifetime::None).unwrap(); dag.add( 2, vec![1], @@ -1275,6 +1345,7 @@ fn empty_exact_summary_extrema_agree_with_ordinary_aggregation() { ), ) .unwrap(), + Lifetime::None, ) .unwrap(); let rows = collect(&dag, 2); @@ -1491,10 +1562,11 @@ fn filtered_summary_build_keeps_empty_groups() { vec![Batch::try_new(input.clone(), rows.clone()).unwrap()], ) .unwrap(), + Lifetime::None, ) .unwrap(); - dag.add(1, vec![0], build).unwrap(); - dag.add(2, vec![1], read).unwrap(); + dag.add(1, vec![0], build, Lifetime::None).unwrap(); + dag.add(2, vec![1], read, Lifetime::None).unwrap(); assert_eq!(printed(&collect(&dag, 2)), expected); } } @@ -1711,9 +1783,10 @@ fn exact_frequency_rejects_non_finite_identities() { vec![Batch::try_new(input, vec![vec![Value::Float64(f64::INFINITY)]]).unwrap()], ) .unwrap(), + Lifetime::None, ) .unwrap(); - dag.add(1, vec![0], operator).unwrap(); + dag.add(1, vec![0], operator, Lifetime::None).unwrap(); let run = context(); let result = block_on(async { let mut stream = dag.execute(&[1], run.clone()).unwrap().remove(0); diff --git a/crates/executor/tests/precompute_population.rs b/crates/executor/tests/precompute_population.rs index 7d908bb4e..d92c4f31e 100644 --- a/crates/executor/tests/precompute_population.rs +++ b/crates/executor/tests/precompute_population.rs @@ -5,7 +5,7 @@ use asap_executor::{ runtime::{Limits, RunContext, Scope}, values::{Batch, Value}, }; -use futures::{executor::block_on, StreamExt}; + use planner_types::ir::operator::{GroupKeys, Reduction}; use planner_types::ir::properties::*; use planner_types::ir::scalar::ColumnRef; @@ -217,14 +217,8 @@ fn native_run( // run's end. store.append_materialized_view(view, (2000 - 60_000, 2000), vec![input])?; let physical_dag = program.instantiate(Arc::new(store))?; - block_on(async { - let mut rows = Vec::new(); - let mut stream = physical_dag.execute(program.roots(), context)?.remove(0); - while let Some(batch) = stream.next().await { - rows.extend(batch?.to_rows().iter().cloned()); - } - Ok(rows) - }) + let batches = crate::common::run_roots(&physical_dag, program.roots(), context)?.remove(0); + Ok(batches.iter().flat_map(Batch::to_rows).collect()) } fn ingestion_context(limits: Limits) -> RunContext { RunContext::new( @@ -356,10 +350,7 @@ fn precompute_dag_enforces_cancellation_and_budget() { &program, family, vec![sum_state(1.)], - ingestion_context(Limits { - max_bytes: 1, - ..Limits::default() - }), + ingestion_context(Limits { max_bytes: 1 }), ) .unwrap_err(); assert!(format!("{error:?}").contains("MemoryLimit")); diff --git a/crates/executor/tests/promql_binary.rs b/crates/executor/tests/promql_binary.rs index 825f9f953..0643bd71f 100644 --- a/crates/executor/tests/promql_binary.rs +++ b/crates/executor/tests/promql_binary.rs @@ -248,7 +248,6 @@ fn binary_obeys_memory_and_cancellation() { }, Limits { max_bytes: if cancel { 10000 } else { 1 }, - ..Limits::default() }, ) .unwrap(); diff --git a/crates/executor/tests/raw_scan.rs b/crates/executor/tests/raw_scan.rs index 28408ed3a..192d63371 100644 --- a/crates/executor/tests/raw_scan.rs +++ b/crates/executor/tests/raw_scan.rs @@ -191,14 +191,12 @@ fn raw_scan_to_sort_limit_at_both_phases() { ) .unwrap() }; - let rows = block_on(async { - let mut output = bound.execute(&[2], ctx.clone()).unwrap().remove(0); - let mut rows = vec![]; - while let Some(batch) = output.next().await { - rows.extend(batch.unwrap().to_rows().iter().cloned()); - } - rows - }); + let rows: Vec<_> = crate::common::run_roots(&bound, &[2], ctx.clone()) + .unwrap() + .remove(0) + .iter() + .flat_map(Batch::to_rows) + .collect(); assert!( matches!(rows.as_slice(), [a,b] if matches!(a.as_slice(), [Value::Int64(9)]) && matches!(b.as_slice(), [Value::Int64(3)])) ); @@ -235,7 +233,8 @@ impl DataStore for CountingStore { no_data(view) } } -// Binding and cancellation do not perform I/O; fan-out opens one cursor per run. +// Binding and cancellation do not perform I/O; at query time each consumer +// of a raw data slot opens its own cursor. #[test] fn lazy_open_shared_producer_and_cancellation() { let (scan, schema, _) = fixture(); @@ -260,7 +259,7 @@ fn lazy_open_shared_producer_and_cancellation() { assert!(all.iter().flatten().all(Result::is_ok)); }); } - assert_eq!(opened.load(Ordering::SeqCst), 2); + assert_eq!(opened.load(Ordering::SeqCst), 4); } // Unsupported predicates fail before opening any cursor; a store without // the source and a failing reader fail the run, not as an empty result. @@ -340,10 +339,7 @@ fn schema_drift_and_memory_limits_fail_the_scan() { Scope::Query { evaluation_time_ms: 0, }, - Limits { - max_bytes: 1, - max_buffered_batches: 1, - }, + Limits { max_bytes: 1 }, ) .unwrap(); block_on(async { @@ -991,12 +987,7 @@ fn a_materialized_view_is_requested_for_the_window_it_summarizes() { Limits::default(), ) .unwrap(); - block_on(async { - let mut output = physical_dag.execute(&[4], ingestion).unwrap().remove(0); - while let Some(batch) = output.next().await { - batch.unwrap(); - } - }); + crate::common::run_roots(&physical_dag, &[4], ingestion).unwrap(); assert_eq!( store.time_ranges_ms.lock().unwrap()[1], (window_end_ms - 5 * MINUTE_MS, window_end_ms) diff --git a/crates/executor/tests/series_window_memory_limit.rs b/crates/executor/tests/series_window_memory_limit.rs index 07ca7d29f..31b58cfca 100644 --- a/crates/executor/tests/series_window_memory_limit.rs +++ b/crates/executor/tests/series_window_memory_limit.rs @@ -8,6 +8,7 @@ use asap_executor::runtime::{Limits, RunContext, Scope}; use asap_executor::values::{Batch, Value}; use asap_executor::Error; use futures::{executor::block_on, StreamExt}; +use planner_types::stages::Lifetime; use std::alloc::{GlobalAlloc, Layout, System}; use std::sync::atomic::{AtomicUsize, Ordering}; use std::sync::Arc; @@ -76,16 +77,20 @@ fn a_long_subquery_over_many_series_fails_near_the_memory_limit() { ) .unwrap(); let mut dag = PhysicalDAG::default(); - dag.add(0, vec![], Operator::source(input, vec![batch]).unwrap()) - .unwrap(); - dag.add(1, vec![0], window).unwrap(); + dag.add( + 0, + vec![], + Operator::source(input, vec![batch]).unwrap(), + Lifetime::None, + ) + .unwrap(); + dag.add(1, vec![0], window, Lifetime::None).unwrap(); let context = RunContext::new( Scope::Query { evaluation_time_ms: 3_600_000, }, Limits { max_bytes: LIMIT_BYTES, - ..Limits::default() }, ) .unwrap(); diff --git a/crates/executor/tests/state_column_updater.rs b/crates/executor/tests/state_column_updater.rs index bcfd31660..71a3f4c7d 100644 --- a/crates/executor/tests/state_column_updater.rs +++ b/crates/executor/tests/state_column_updater.rs @@ -19,6 +19,7 @@ use asap_executor::summary_kernels::{ use asap_executor::values::{Batch, SchemaRef, Value}; use asap_executor::{AggregateCore, StateColumnUpdater}; use futures::{executor::block_on, StreamExt}; +use planner_types::stages::Lifetime; use std::collections::BTreeMap; use std::num::NonZeroU32; use std::sync::Arc; @@ -228,9 +229,10 @@ fn batch_build(build: Operator) -> BTreeMap { 0, vec![], Operator::source(input_schema(), vec![input_batch()]).unwrap(), + Lifetime::None, ) .unwrap(); - dag.add(1, vec![0], build).unwrap(); + dag.add(1, vec![0], build, Lifetime::None).unwrap(); let context = RunContext::new( Scope::Query { evaluation_time_ms: 10_000, diff --git a/crates/executor/tests/stored_tumbling_window_memory_limit.rs b/crates/executor/tests/stored_tumbling_window_memory_limit.rs index 49886383b..ab3ea4b89 100644 --- a/crates/executor/tests/stored_tumbling_window_memory_limit.rs +++ b/crates/executor/tests/stored_tumbling_window_memory_limit.rs @@ -11,6 +11,7 @@ use asap_executor::runtime::{Limits, RunContext, Scope}; use asap_executor::values::{Batch, BatchColumn}; use asap_executor::{Error, StateColumnUpdater}; use futures::{executor::block_on, StreamExt}; +use planner_types::stages::Lifetime; use std::sync::Arc; const SERIES: u32 = 1_000; @@ -80,22 +81,21 @@ fn merge_stored_tumbling_windows(max_bytes: usize) -> Result { 0, vec![], Operator::source(schema.clone(), vec![tumbling_window; TUMBLING_WINDOW_COUNT]).unwrap(), + Lifetime::None, ) .unwrap(); dag.add( 1, vec![0], Operator::summary_merge(schema, 1, vec![0]).unwrap(), + Lifetime::None, ) .unwrap(); let context = RunContext::new( Scope::Query { evaluation_time_ms: 0, }, - Limits { - max_bytes, - ..Limits::default() - }, + Limits { max_bytes }, ) .unwrap(); block_on(async { @@ -141,22 +141,21 @@ fn compacting_a_stored_tumbling_window_does_not_charge_its_shared_sketches() { 0, vec![], Operator::source(schema.clone(), vec![tumbling_window]).unwrap(), + Lifetime::None, ) .unwrap(); dag.add( 1, vec![0], Operator::limit(schema, kept, 0, vec![]).unwrap(), + Lifetime::None, ) .unwrap(); let context = RunContext::new( Scope::Query { evaluation_time_ms: 0, }, - Limits { - max_bytes: 8 << 20, - ..Limits::default() - }, + Limits { max_bytes: 8 << 20 }, ) .unwrap(); let rows = block_on(async { @@ -184,6 +183,7 @@ fn sorting_then_limiting_stored_tumbling_windows_does_not_charge_their_sketches( 0, vec![], Operator::source(schema.clone(), vec![first, second]).unwrap(), + Lifetime::None, ) .unwrap(); let key = SortKey { @@ -195,18 +195,21 @@ fn sorting_then_limiting_stored_tumbling_windows_does_not_charge_their_sketches( 1, vec![0], Operator::sort(schema.clone(), vec![key], vec![], None).unwrap(), + Lifetime::None, + ) + .unwrap(); + dag.add( + 2, + vec![1], + Operator::limit(schema, 980, 0, vec![]).unwrap(), + Lifetime::None, ) .unwrap(); - dag.add(2, vec![1], Operator::limit(schema, 980, 0, vec![]).unwrap()) - .unwrap(); let context = RunContext::new( Scope::Query { evaluation_time_ms: 0, }, - Limits { - max_bytes: 8 << 20, - ..Limits::default() - }, + Limits { max_bytes: 8 << 20 }, ) .unwrap(); let rows = block_on(async { @@ -231,22 +234,21 @@ fn a_merge_passing_stored_states_through_does_not_charge_them() { 0, vec![], Operator::source(schema.clone(), vec![tumbling_window]).unwrap(), + Lifetime::None, ) .unwrap(); dag.add( 1, vec![0], Operator::summary_merge(schema, 1, vec![0]).unwrap(), + Lifetime::None, ) .unwrap(); let context = RunContext::new( Scope::Query { evaluation_time_ms: 0, }, - Limits { - max_bytes: 8 << 20, - ..Limits::default() - }, + Limits { max_bytes: 8 << 20 }, ) .unwrap(); let rows = block_on(async { @@ -410,7 +412,6 @@ fn a_materialized_view_providers_states_are_not_charged_to_the_run() { }, Limits { max_bytes: 32 << 20, - ..Limits::default() }, ) .unwrap(); diff --git a/crates/executor/tests/summary_build_memory_limit.rs b/crates/executor/tests/summary_build_memory_limit.rs index ea30af325..19d3b4ac5 100644 --- a/crates/executor/tests/summary_build_memory_limit.rs +++ b/crates/executor/tests/summary_build_memory_limit.rs @@ -9,6 +9,7 @@ use asap_executor::runtime::{Limits, RunContext, Scope}; use asap_executor::values::{Batch, Value}; use asap_executor::Error; use futures::{executor::block_on, StreamExt}; +use planner_types::stages::Lifetime; use std::alloc::{GlobalAlloc, Layout, System}; use std::sync::atomic::{AtomicUsize, Ordering}; use std::sync::Arc; @@ -61,6 +62,7 @@ fn one_large_batch_fails_near_the_memory_limit() { 0, vec![], Operator::source(input.clone(), vec![batch]).unwrap(), + Lifetime::None, ) .unwrap(); dag.add( @@ -75,6 +77,7 @@ fn one_large_batch_fails_near_the_memory_limit() { vec![0], ) .unwrap(), + Lifetime::None, ) .unwrap(); let context = RunContext::new( @@ -83,7 +86,6 @@ fn one_large_batch_fails_near_the_memory_limit() { }, Limits { max_bytes: LIMIT_BYTES, - ..Limits::default() }, ) .unwrap(); diff --git a/crates/executor/tests/text_extremum_memory.rs b/crates/executor/tests/text_extremum_memory.rs index 0cc056fa4..221919d64 100644 --- a/crates/executor/tests/text_extremum_memory.rs +++ b/crates/executor/tests/text_extremum_memory.rs @@ -9,6 +9,7 @@ use asap_executor::runtime::{Limits, RunContext, Scope}; use asap_executor::values::{Batch, BatchColumn}; use asap_executor::Error; use futures::{executor::block_on, StreamExt}; +use planner_types::stages::Lifetime; use std::sync::Arc; /// Run `measures` over `batch`, grouped by `groups`, under `max_bytes`; @@ -25,22 +26,21 @@ fn aggregate( 0, vec![], Operator::source(input.clone(), vec![batch]).unwrap(), + Lifetime::None, ) .unwrap(); dag.add( 1, vec![0], Operator::aggregate(input, groups, measures).unwrap(), + Lifetime::None, ) .unwrap(); let context = RunContext::new( Scope::Query { evaluation_time_ms: 0, }, - Limits { - max_bytes, - ..Limits::default() - }, + Limits { max_bytes }, ) .unwrap(); block_on(async { diff --git a/crates/executor/tests/time_series_window.rs b/crates/executor/tests/time_series_window.rs index ef24d850c..4772139fb 100644 --- a/crates/executor/tests/time_series_window.rs +++ b/crates/executor/tests/time_series_window.rs @@ -6,7 +6,7 @@ use asap_executor::physical_planner::{compile, promql_rows}; use asap_executor::runtime::{Limits, RunContext, Scope}; use asap_executor::sources::InMemoryDataStore; use asap_executor::values::{Batch, Value}; -use futures::{executor::block_on, StreamExt}; + use planner_types::ir::operator::{Source, TimeShift}; use planner_types::ir::scalar::{CompareOpKind, ScalarValue}; use planner_types::ir::schema::{DataType, Field, Schema, PROMQL_SERIES_IDENTITY}; @@ -136,19 +136,17 @@ fn points_of(dag: &PhysicalASAPDAG, store: InMemoryDataStore, scope: Scope) -> V .instantiate(Arc::new(store)) .unwrap(); let context = RunContext::new(scope, Limits::default()).unwrap(); - block_on(async { - let mut output = bound.execute(&roots, context).unwrap().remove(0); - let mut points = Vec::new(); - while let Some(batch) = output.next().await { - for row in batch.unwrap().to_rows() { - match (&row[1], &row[2]) { - (Value::Timestamp(ts), Value::Float64(v)) => points.push((*ts, *v)), - other => panic!("{other:?}"), - } - } + let mut points = Vec::new(); + let batches = crate::common::run_roots(&bound, &roots, context) + .unwrap() + .remove(0); + for row in batches.iter().flat_map(Batch::to_rows) { + match (&row[1], &row[2]) { + (Value::Timestamp(ts), Value::Float64(v)) => points.push((*ts, *v)), + other => panic!("{other:?}"), } - points - }) + } + points } fn at(evaluation_time_ms: i64) -> Scope { diff --git a/crates/executor/tests/tumbling_window_chain.rs b/crates/executor/tests/tumbling_window_chain.rs index e6c3950de..9a9bee44d 100644 --- a/crates/executor/tests/tumbling_window_chain.rs +++ b/crates/executor/tests/tumbling_window_chain.rs @@ -14,6 +14,7 @@ use planner_types::ir::scalar::ColumnRef; use planner_types::ir::schema::{DataType, Field, FieldDataType, Schema}; use planner_types::ir::{ASAPOp, NonASAPOp, Operator, OperatorNode, QueryRoot, TimeRangeKind}; use planner_types::plan::physical_asap_dag::{PhysicalASAPDAG, PhysicalASAPNodeId}; +use planner_types::stages::Lifetime; use planner_types::summary_type::{ExactKind, ExactParams, SummaryUpdate}; use std::collections::BTreeMap; use std::rc::Rc; @@ -143,19 +144,11 @@ fn run( roots: &[u64], context: RunContext, ) -> Vec>> { - // The roots share producers, so they are read together. - block_on(futures::future::join_all( - dag.execute(roots, context) - .unwrap() - .into_iter() - .map(|mut output| async move { - let mut rows = Vec::new(); - while let Some(batch) = output.next().await { - rows.extend(batch.unwrap().to_rows()); - } - rows - }), - )) + crate::common::run_roots(dag, roots, context) + .unwrap() + .into_iter() + .map(|batches| batches.iter().flat_map(Batch::to_rows).collect()) + .collect() } /// The id of the node of `dag` that is `operator`'s node in `nodes`. @@ -265,13 +258,7 @@ fn a_chain_built_at_ingestion_answers_queries_through_its_time_ranges() { window_end += WIDTH_MS; } let evaluation_time_ms = 4 * HOUR_MS; - let context = RunContext::new( - Scope::Query { evaluation_time_ms }, - Limits { - ..Limits::default() - }, - ) - .unwrap(); + let context = RunContext::new(Scope::Query { evaluation_time_ms }, Limits::default()).unwrap(); let answers = { let query = query.instantiate(store.clone()).unwrap(); run(&query, &[now as u64, shifted as u64], context) @@ -405,13 +392,14 @@ fn a_range_window_stamps_each_row_with_its_tumbling_windows_end() { 0, vec![], ExecutorOperator::source(schema.clone(), vec![batch]).unwrap(), + Lifetime::None, ) .unwrap(); let window = ExecutorOperator::time_range(schema, 1, 0, 3 * MINUTE_MS, None, None) .unwrap() .with_tumbling_window_ends(MINUTE_MS) .unwrap(); - dag.add(1, vec![0], window).unwrap(); + dag.add(1, vec![0], window, Lifetime::None).unwrap(); let context = RunContext::new( Scope::Query { evaluation_time_ms: 10 * MINUTE_MS, diff --git a/crates/executor/tests/univmon_execution.rs b/crates/executor/tests/univmon_execution.rs index c6fa4aff3..1866ac810 100644 --- a/crates/executor/tests/univmon_execution.rs +++ b/crates/executor/tests/univmon_execution.rs @@ -16,6 +16,7 @@ use planner_types::ir::schema::{DataType, Schema}; use planner_types::ir::schema::{Field, FieldDataType}; use planner_types::ir::ASAPOp; use planner_types::plan::physical_asap_dag::{PhysicalASAPDAGNode, PhysicalASAPOperatorPayload}; +use planner_types::stages::{Lifetime, StorageMedium}; use planner_types::summary_type::{ SketchAlgorithm, SketchStatistic, SummaryInputExpr, SummaryUpdate, }; @@ -60,8 +61,14 @@ fn three_statistics(batch: Batch, build: Operator) -> Result, Error> { 0, vec![], Operator::source(batch.schema().clone(), vec![batch])?, + Lifetime::None, + )?; + dag.add( + 1, + vec![0], + build, + Lifetime::OneEvaluation(StorageMedium::Memory), )?; - dag.add(1, vec![0], build)?; for (id, statistic) in [ (2, SketchStatistic::Cardinality), (3, SketchStatistic::FrequencyL2), @@ -71,6 +78,7 @@ fn three_statistics(batch: Batch, build: Operator) -> Result, Error> { id, vec![1], Operator::evaluation(state.clone(), 0, SummaryEvaluation::Sketch(statistic))?, + Lifetime::None, )?; } let outputs = block_on(futures::future::join_all( diff --git a/crates/integration-tests/tests/kll_tumbling_window_execution.rs b/crates/integration-tests/tests/kll_tumbling_window_execution.rs index 714b69c8c..f4a27a1db 100644 --- a/crates/integration-tests/tests/kll_tumbling_window_execution.rs +++ b/crates/integration-tests/tests/kll_tumbling_window_execution.rs @@ -18,6 +18,7 @@ use asap_types::ir::{ASAPOp, NonASAPOp, OperatorNode, QueryRoot, TimeRangeKind}; use asap_types::plan::materialized_view::MaterializedView; use asap_types::plan::physical_asap_dag::PhysicalASAPDAG; use asap_types::plan::physical_asap_dag::ViewCoverage; +use asap_types::stages::{Lifetime, StorageMedium}; use asap_types::summary_type::{SketchAlgorithm, SketchStatistic, SummaryUpdate}; use futures::{executor::block_on, StreamExt}; use std::{ @@ -346,6 +347,7 @@ fn five_tumbling_windows_roundtrip_and_shared_merge_runs_once() { 0, vec![], Operator::source(schema.clone(), vec![restored]).unwrap(), + Lifetime::None, ) .unwrap(); dag.add( @@ -355,12 +357,23 @@ fn five_tumbling_windows_roundtrip_and_shared_merge_runs_once() { operator: merge.clone(), starts: starts.clone(), }, + Lifetime::OneEvaluation(StorageMedium::Memory), + ) + .unwrap(); + dag.add( + 2, + vec![1], + evaluation(merge.output_schema(), 0.5), + Lifetime::None, + ) + .unwrap(); + dag.add( + 3, + vec![1], + evaluation(merge.output_schema(), 0.99), + Lifetime::None, ) .unwrap(); - dag.add(2, vec![1], evaluation(merge.output_schema(), 0.5)) - .unwrap(); - dag.add(3, vec![1], evaluation(merge.output_schema(), 0.99)) - .unwrap(); let outputs = block_on(futures::future::join_all( dag.execute( &[2, 3], diff --git a/crates/integration-tests/tests/physical_common/mod.rs b/crates/integration-tests/tests/physical_common/mod.rs index 0f8b39835..db20bb72f 100644 --- a/crates/integration-tests/tests/physical_common/mod.rs +++ b/crates/integration-tests/tests/physical_common/mod.rs @@ -4,7 +4,7 @@ use asap_executor::{ sources::InMemoryDataStore, values::Batch, }; -use futures::{executor::block_on, StreamExt}; +use futures::{executor::block_on, StreamExt, TryStreamExt}; use std::sync::Arc; /// Run `plan` against `store` and collect each root's batches. @@ -15,22 +15,22 @@ pub fn execute( scope: Scope, ) -> Vec> { let dag = plan.instantiate(Arc::new(store)).unwrap(); - block_on(async { - let streams = dag - .execute( - plan.roots(), - RunContext::new(scope, Limits::default()).unwrap(), - ) - .unwrap(); - futures::future::join_all(streams.into_iter().map(|mut stream| async move { - let mut batches = Vec::new(); - while let Some(batch) = stream.next().await { - batches.push((*batch.unwrap()).clone()); - } - batches - })) - .await - }) + let context = RunContext::new(scope, Limits::default()).unwrap(); + let outputs = match context.scope { + Scope::Ingestion { .. } => { + block_on(dag.execute_ingestion(plan.roots(), context).unwrap()).unwrap() + } + Scope::Query { .. } => dag + .execute(plan.roots(), context) + .unwrap() + .into_iter() + .map(|stream| block_on(stream.try_collect::>()).unwrap()) + .collect(), + }; + outputs + .into_iter() + .map(|batches| batches.iter().map(|batch| (**batch).clone()).collect()) + .collect() } /// Execute a relational DAG at query time against an in-memory connector diff --git a/crates/planner/src/tutorials/_05_executing_a_plan.md b/crates/planner/src/tutorials/_05_executing_a_plan.md index b9230caba..c80d785b1 100644 --- a/crates/planner/src/tutorials/_05_executing_a_plan.md +++ b/crates/planner/src/tutorials/_05_executing_a_plan.md @@ -13,8 +13,10 @@ building a deployment runtime on top of the planner. and its roots into a `CompiledPhysicalDAG`, opening no reader. - Instantiate: `CompiledPhysicalDAG::instantiate` binds every raw data and materialized view slot to a `DataStore`, here an `InMemoryDataStore`. -- Execute: `PhysicalDAG::execute` with a `RunContext` whose `Scope` is - `Ingestion` (to build views) or `Query` (to answer a query at an - evaluation time). -- Reading the results: the returned batches, one per root, and checking them - against the exact answer. +- Execute: `PhysicalDAG::execute_ingestion` with a `Scope::Ingestion` + context (to build views), or `PhysicalDAG::execute` with a + `Scope::Query` context (to answer a query at an evaluation time). Each + rejects the other's scope. +- Reading the results: `execute_ingestion` returns every root's batches for + the caller to write; `execute` returns one stream per root, drained in any + order. Check them against the exact answer. diff --git a/crates/planner/tests/columnar_batches.rs b/crates/planner/tests/columnar_batches.rs index 2598a088f..6451a6272 100644 --- a/crates/planner/tests/columnar_batches.rs +++ b/crates/planner/tests/columnar_batches.rs @@ -21,7 +21,6 @@ use asap_planner::{ plan_installable, DeploymentCapabilities, FrontendInput, PlanningModels, PlanningWorkload, UserInput, }; -use futures::StreamExt; use crate::time_series; use time_series::{series_identity, RawDataSample}; @@ -163,13 +162,12 @@ async fn a_tumbling_window_is_built_from_arrow_columns_with_one_label_entry_per_ .unwrap(); // The built tumbling window: one row per series, its KLL in a state column. let tumbling_window = build - .execute(compiled.roots(), context) + .execute_ingestion(compiled.roots(), context) .unwrap() - .remove(0) - .next() .await .unwrap() - .unwrap(); + .remove(0) + .remove(0); let state_column = tumbling_window .schema() .fields diff --git a/crates/planner/tests/sql_table_scans.rs b/crates/planner/tests/sql_table_scans.rs index 77d9656cd..309a83ace 100644 --- a/crates/planner/tests/sql_table_scans.rs +++ b/crates/planner/tests/sql_table_scans.rs @@ -23,7 +23,7 @@ use asap_types::ir::operator::Source; use asap_types::ir::schema::{DataType, Field, Schema}; use asap_types::ir::{ASAPOp, NonASAPOp}; use asap_types::plan::physical_asap_dag::PhysicalASAPOperatorPayload; -use futures::{executor::block_on, StreamExt}; +use futures::{executor::block_on, TryStreamExt}; const MINUTE_MS: i64 = 60_000; /// The evaluation time of every run. @@ -205,18 +205,25 @@ fn run_bound( } } let instance = compiled.instantiate(Arc::new(store)).unwrap(); - block_on(async { - let context = RunContext::new(scope, Limits::default()).unwrap(); - let outputs = instance.execute(compiled.roots(), context).unwrap(); - futures::future::join_all(outputs.into_iter().map(|mut output| async move { - let mut batches = Vec::new(); - while let Some(batch) = output.next().await { - batches.push((*batch.unwrap()).clone()); - } - batches - })) - .await - }) + let context = RunContext::new(scope, Limits::default()).unwrap(); + let outputs = match context.scope { + Scope::Ingestion { .. } => block_on( + instance + .execute_ingestion(compiled.roots(), context) + .unwrap(), + ) + .unwrap(), + Scope::Query { .. } => instance + .execute(compiled.roots(), context) + .unwrap() + .into_iter() + .map(|output| block_on(output.try_collect::>()).unwrap()) + .collect(), + }; + outputs + .into_iter() + .map(|batches| batches.iter().map(|batch| (**batch).clone()).collect()) + .collect() } /// The data plane of `plan` at [`EVALUATION_MS`] over the table rows diff --git a/crates/types/src/plan/physical_asap_dag.rs b/crates/types/src/plan/physical_asap_dag.rs index afc2ae94b..b263241c5 100644 --- a/crates/types/src/plan/physical_asap_dag.rs +++ b/crates/types/src/plan/physical_asap_dag.rs @@ -27,7 +27,7 @@ use crate::ir::properties::execution_data_state::{ExecutionDataState, ExecutionT use crate::ir::properties::selection::SelectionBox; use crate::ir::schema::Schema; use crate::stages::PhysicalCandidateId; -use crate::stages::{Lifetime, PhysicalPropertiesError}; +use crate::stages::{Lifetime, PhysicalPropertiesError, StorageMedium}; use crate::summary_type::SketchParams; /// A node's operator, with each child replaced by its node id. @@ -673,10 +673,16 @@ impl PhysicalASAPDAG { } } -/// Test-only: `roots` exported as one DAG with every node at query time and -/// not materialized, and no θ (every `parameters` is `None`), with each -/// exported node's operator node by node id. Executor tests hand-build DAGs -/// this way; planning builds the plan in Stage 3. +/// Test-only: `roots` exported as one DAG with every node at query time, and +/// no θ (every `parameters` is `None`), with each exported node's operator +/// node by node id. A node with more than one consumer (edges and root +/// positions), a leaf included, is held for one evaluation +/// (`OneEvaluation(Memory)`), as Stage 3 selects for a shared query-time +/// output; every other node is not materialized. Example: a scan read by two +/// queries is `OneEvaluation`, read once and held for both. Consumers are +/// counted per node, not per evaluation time, so a node read at two times +/// under an `@` anchor is conservatively held too. Executor tests hand-build +/// DAGs this way; planning builds the plan in Stage 3. #[cfg(any(test, feature = "test-support"))] pub(crate) fn physical_asap_dag_at_query_time_with_node_ids( roots: &[crate::ir::QueryRoot], @@ -716,10 +722,22 @@ pub(crate) fn physical_asap_dag_at_query_time_with_node_ids( view_coverage: None, }) .collect(); - Ok(( - PhysicalASAPDAG::new(nodes, roots, scalar_roots)?, - nodes_by_id, - )) + let mut dag = PhysicalASAPDAG::new(nodes, roots, scalar_roots)?; + let mut consumers = vec![0usize; dag.nodes.len()]; + for producer in dag + .edges + .iter() + .map(|edge| edge.producer) + .chain(dag.roots.iter().copied()) + { + consumers[producer] += 1; + } + for node in &mut dag.nodes { + if consumers[node.id] > 1 { + node.lifetime = Lifetime::OneEvaluation(StorageMedium::Memory); + } + } + Ok((dag, nodes_by_id)) } fn grouping_compatibility( diff --git a/crates/types/src/test_support/forwarders.rs b/crates/types/src/test_support/forwarders.rs index 34213563b..e3092249c 100644 --- a/crates/types/src/test_support/forwarders.rs +++ b/crates/types/src/test_support/forwarders.rs @@ -33,8 +33,10 @@ pub fn matches_node(input: &CurrentSeriesInput, node: &Rc) -> bool } /// `physical_asap_dag::physical_asap_dag_at_query_time_with_node_ids`: `roots` -/// exported with every node at query time, not materialized and without θ, -/// with each exported node's operator node by node id. +/// exported as one DAG with every node at query time and without θ, with +/// each exported node's operator node by node id. A node with more than one +/// consumer, a leaf included, is `OneEvaluation`, as Stage 3 selects for a +/// shared query-time output; every other node is not materialized. pub fn physical_asap_dag_at_query_time_with_node_ids( roots: &[crate::ir::QueryRoot], ) -> Result< diff --git a/docs/design_docs/physical-planning-and-deployment.md b/docs/design_docs/physical-planning-and-deployment.md index 25f9580fc..507a73930 100644 --- a/docs/design_docs/physical-planning-and-deployment.md +++ b/docs/design_docs/physical-planning-and-deployment.md @@ -517,10 +517,15 @@ An **input slot** is a position in a compiled DAG where data is read at run time, together with that position's schema, which is the DAG node's `output_schema`. There are two kinds: -| Slot | Where | Holds | Read by | -| --- | --- | --- | --- | -| **Raw data slot** | every `Scan` leaf | the `Scan`'s `Source`, schema, compiled predicates, and the window its consumers read (from the `TimeRange`, `TimeShift` and chain nodes above it) | `Source` | -| **Materialized view slot** | each view the DAG reads | the `MaterializedView`, its schema, and the rows a read selects: a chain's, the windows the `TimeRange`s above it select; any other view's, the window its state summarizes (the union of the windows of every `TimeRange` below it) | `MaterializedView` | +| Slot | Where | Holds | Lifetime | Read by | +| --- | --- | --- | --- | --- | +| **Raw data slot** | every `Scan` leaf | the `Scan`'s `Source`, schema, compiled predicates, and the window its consumers read (from the `TimeRange`, `TimeShift` and chain nodes above it) | the `Scan` node's | `Source` | +| **Materialized view slot** | each view the DAG reads | the `MaterializedView`, its schema, and the rows a read selects: a chain's, the windows the `TimeRange`s above it select; any other view's, the window its state summarizes (the union of the windows of every `TimeRange` below it) | the view node's, `AcrossEvaluations` | `MaterializedView` | + +The compiled DAG keeps each Planner node's `Lifetime` in one map, and the +runtime runs every node, slots included, by it: how a slot is read at +query time (by each consumer, or once and held) is the read rule in +[the executor README](../../crates/executor/README.md). ```rust pub fn compile(dag: &PhysicalASAPDAG, views: &[MaterializedView], roots: &[NodeId])