Skip to content
57 changes: 48 additions & 9 deletions crates/executor/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,50 @@ a storage implementation nor DataFusion. Batches are Arrow columns
(`values::Batch`); DataFusion informed the design, it is not the execution
framework.

`plan::PhysicalDAG` binds typed operator inputs to node IDs. Each execution starts
one producer per reachable node, shares output batches among its consumers, and
bounds buffering. Dropping one consumer does not cancel other consumers. A
`RunContext` carries query or ingestion scope, cancellation and byte accounting.
Executions use the caller's worker and worker-local streams, with no internal
thread pool. Poll multiple root streams concurrently when they share inputs.
`plan::PhysicalDAG` binds typed operator inputs to node IDs, each node with the
plan's `Lifetime`. A `RunContext` carries query or ingestion scope,
cancellation and byte accounting. Executions use the caller's worker and
worker-local streams, with no internal thread pool.

A node runs once per evaluation time in a run (an `@` anchor evaluates the
nodes below it at another time; a leaf read per consumer runs once per
consumer), as its `Lifetime` and the run's scope say:

- Query time, a leaf with `None` or `AcrossEvaluations` (rows a store or
the operator keeps): `None` (a raw data scan the plan does not
materialize, or any other leaf such as a constant, which holds its own
rows) or `AcrossEvaluations` (a materialized view slot, or raw data kept
at ingestion). Each consumer reads it on its own, and
nothing is held. Stage 3 never gives a query-time `None` result more than
one consumer; a view slot, or a leaf in a hand-built DAG, may have
several. For example, an `AcrossEvaluations` scan read by two queries is
two store reads.
- Query time, `OneEvaluation` (a computed node or a leaf) with more than
one consumer: the node runs once, to the end, before any consumer reads
it, and every consumer reads its finished output independently, as the
planner charges one run whose output is held. Likewise a computed
`AcrossEvaluations` node with more than one consumer, which only a
hand-built DAG has at query time (a compiled plan reads it through its
view slot). For example, a `OneEvaluation` scan read by three queries
is one store read. The output is charged to the run's byte budget until
every consumer has read it; over the budget the run fails with
`MemoryLimit` (nothing spills). With one consumer the node streams to it,
since no sibling can wait on that consumer.
- Query time, `None` computed node: streamed to its one consumer. More than
one consumer is a planner bug; `execute` returns an error naming the node
before any operator starts.
- Ingestion time: every node streams, whatever its lifetime. A node with
several consumers (one per-row node feeding several chain builds or views)
runs once and each batch reaches every consumer's own queue. The queues
are bounded only by the run's byte budget: a consumer that lags holds what
the others have read. `execute_ingestion` polls every root itself and
returns one future that resolves to each root's batches, for the caller to
write.

`execute` runs a query and returns one stream per root; the caller may drain
them in any order. It rejects an ingestion scope, which `execute_ingestion`
runs, and `execute_ingestion` rejects a query scope. Dropping one consumer
does not cancel other consumers.

`operators::Operator` implements native batch sources, scalar values,
projection, filtering, grouped exact aggregation, semi-join, grouped Sort and
Expand All @@ -28,6 +66,7 @@ use asap_executor::{
plan::PhysicalDAG,
runtime::{Limits, RunContext, Scope},
};
use asap_executor::planner::stages::Lifetime;
use asap_executor::planner::ir::schema::DataType;
use futures::{executor::block_on, StreamExt};

Expand All @@ -36,8 +75,8 @@ let negate = Operator::project(source.schema(), vec![
("value".into(), Expression::Negate(Box::new(Expression::Column(0)))),
])?;
let mut plan = PhysicalDAG::default();
plan.add(0, vec![], source)?;
plan.add(1, vec![0], negate)?;
plan.add(0, vec![], source, Lifetime::None)?;
plan.add(1, vec![0], negate, Lifetime::None)?;
let run = RunContext::new(
Scope::Query { evaluation_time_ms: 1000 },
Limits::default(),
Expand Down Expand Up @@ -76,7 +115,7 @@ See [the design](../../docs/design_docs/physical-planning-and-deployment.md).
## Module boundaries

- `plan`: immutable DAG, operator interface, schemas and execution properties.
- `runtime`: per-run streams, shared producers, memory reservations and cancellation.
- `runtime`: per-run streams, how each node runs by its `Lifetime`, memory reservations and cancellation.
- `expressions`: scalar evaluation; typed builders and the Planner expression adapter.
- `operators`: projection, filter, joins, aggregate/window, sort, limit and summary implementations.
- `sources`: the `DataStore` data interface, `Scan` and the in-memory store.
Expand Down
16 changes: 13 additions & 3 deletions crates/executor/src/operators/aggregate/temporal.rs
Original file line number Diff line number Diff line change
Expand Up @@ -326,6 +326,7 @@ mod tests {
values::Batch,
};
use futures::StreamExt;
use planner_types::stages::Lifetime;

use planner_types::ir::schema::{
DataType, Field as SummaryField, FieldDataType as SummaryFamilyType, Schema,
Expand Down Expand Up @@ -383,12 +384,21 @@ mod tests {
0,
vec![],
Operator::source(schema.clone(), vec![batch]).unwrap(),
Lifetime::None,
)
.unwrap();
dag.add(1, vec![0], operator).unwrap();
dag.add(1, vec![0], operator, Lifetime::None).unwrap();
let context = RunContext::new(scope.clone(), Limits::default()).unwrap();
let mut output = dag.execute(&[1], context).unwrap().remove(0);
let result = futures::executor::block_on(output.next()).unwrap().unwrap();
let result = if matches!(scope, Scope::Ingestion { .. }) {
let run = dag.execute_ingestion(&[1], context).unwrap();
futures::executor::block_on(run)
.unwrap()
.remove(0)
.remove(0)
} else {
let mut output = dag.execute(&[1], context).unwrap().remove(0);
futures::executor::block_on(output.next()).unwrap().unwrap()
};
assert_eq!(
format!("{:?}", result.to_rows()[0][0]),
format!("{expected:?}")
Expand Down
4 changes: 3 additions & 1 deletion crates/executor/src/operators/series_labels.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1108,6 +1108,7 @@ mod columnar_tests {
use crate::plan::PhysicalDAG;
use crate::runtime::{Limits, Scope};
use planner_types::ir::schema::Field;
use planner_types::stages::Lifetime;

/// The row-at-a-time rewrite this operator ran before (the reference).
fn reference(
Expand Down Expand Up @@ -1248,9 +1249,10 @@ mod columnar_tests {
],
)
.unwrap(),
Lifetime::None,
)
.unwrap();
dag.add(1, vec![0], operator).unwrap();
dag.add(1, vec![0], operator, Lifetime::None).unwrap();
let context = RunContext::new(
Scope::Query {
evaluation_time_ms: 0,
Expand Down
32 changes: 24 additions & 8 deletions crates/executor/src/physical_planner/compiled.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
use super::*;
use crate::sources::{DataStore, Scan};
use planner_types::plan::materialized_view::MaterializedView;
use planner_types::stages::Lifetime;

#[derive(Clone)]
#[cfg_attr(any(test, feature = "test-support"), derive(serde::Serialize))]
Expand Down Expand Up @@ -52,26 +53,36 @@ pub(super) struct RawDataSlot {
pub struct CompiledPhysicalDAG {
nodes: BTreeMap<NodeId, Node>,
roots: Vec<NodeId>,
/// The plan lifetime of each compiled Planner node, which the runtime
/// runs it by (see `NodeRun` in the runtime). A helper node the lowering
/// adds is not in it and has `Lifetime::None`. Example: node 1, a stored view,
/// maps to `AcrossEvaluations(Memory)`, and node 3, a query-time output
/// two queries read, to `OneEvaluation(Memory)`.
lifetimes: BTreeMap<NodeId, Lifetime>,
}

impl CompiledPhysicalDAG {
/// Assemble already-lowered operators; leaves read no data store (for
/// example [`Operator::source`] holds its rows).
/// example [`Operator::source`] holds its rows). Every node runs as
/// `Lifetime::None`, so at query time a computed node may have at most
/// one consumer; for a shared one, build a [`PhysicalDAG`] with
/// `add(…, lifetime)`.
pub fn from_operators(
operators: BTreeMap<NodeId, (Vec<NodeId>, Operator)>,
roots: Vec<NodeId>,
) -> Result<Self, Error> {
let mut result = Self::new(roots);
let mut result = Self::new(roots, BTreeMap::new());
for (id, (inputs, operator)) in operators {
result.add(id, inputs, operator)?;
}
result.validate()?;
Ok(result)
}
pub(super) fn new(roots: Vec<NodeId>) -> Self {
pub(super) fn new(roots: Vec<NodeId>, lifetimes: BTreeMap<NodeId, Lifetime>) -> Self {
Self {
nodes: BTreeMap::new(),
roots,
lifetimes,
}
}
pub(super) fn add_materialized_view(
Expand Down Expand Up @@ -157,13 +168,14 @@ impl CompiledPhysicalDAG {
fn unbound(&self) -> Result<PhysicalDAG<'_, Batch, SchemaRef>, Error> {
let mut dag = PhysicalDAG::default();
for (&id, node) in &self.nodes {
let lifetime = self.lifetime(id);
match node {
Node::MaterializedView(MaterializedViewSlot { schema, .. })
| Node::RawData(RawDataSlot { schema, .. }) => {
dag.add(id, vec![], UnboundInput(schema.clone()))?
dag.add(id, vec![], UnboundInput(schema.clone()), lifetime)?
}
Node::Operator { inputs, operator } => {
dag.add(id, inputs.clone(), operator.clone())?
dag.add(id, inputs.clone(), operator.clone(), lifetime)?
}
}
}
Expand All @@ -180,6 +192,7 @@ impl CompiledPhysicalDAG {
) -> Result<PhysicalDAG<'static, Batch, SchemaRef>, Error> {
let mut dag = PhysicalDAG::default();
for (&id, node) in &self.nodes {
let lifetime = self.lifetime(id);
match node {
Node::MaterializedView(slot) => {
let scan = Scan::materialized_view(
Expand All @@ -188,7 +201,7 @@ impl CompiledPhysicalDAG {
slot.schema.clone(),
slot.read_window,
);
dag.add(id, vec![], scan)?;
dag.add(id, vec![], scan, lifetime)?;
}
Node::RawData(slot) => {
let scan = Scan::raw_data(
Expand All @@ -198,16 +211,19 @@ impl CompiledPhysicalDAG {
slot.predicates.clone(),
slot.read_window,
);
dag.add(id, vec![], scan)?;
dag.add(id, vec![], scan, lifetime)?;
}
Node::Operator { inputs, operator } => {
dag.add(id, inputs.clone(), operator.clone())?;
dag.add(id, inputs.clone(), operator.clone(), lifetime)?
}
}
}
dag.validate(&self.roots)?;
Ok(dag)
}
fn lifetime(&self, id: NodeId) -> Lifetime {
self.lifetimes.get(&id).copied().unwrap_or_default()
}
}
/// Stands in for an input slot when validating or deriving properties.
struct UnboundInput(SchemaRef);
Expand Down
6 changes: 5 additions & 1 deletion crates/executor/src/physical_planner/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -183,7 +183,11 @@ fn compile_internal(
consumers: &consumers,
roots: &compile_roots,
};
let mut physical_dag = CompiledPhysicalDAG::new(roots.to_vec());
let lifetimes = ordered
.iter()
.map(|&id| (id, nodes[&id].lifetime))
.collect();
let mut physical_dag = CompiledPhysicalDAG::new(roots.to_vec(), lifetimes);
for id in ordered {
let node = nodes[&id];
let auxiliary = helper_id(id, 0)?;
Expand Down
8 changes: 7 additions & 1 deletion crates/executor/src/physical_planner/test_support.rs
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,10 @@ pub fn set_build_parameters<'a>(
/// The materialized views at `nodes`, in that order: each node that is
/// not a view is marked materialized at ingestion time, and it and
/// every node below it run at ingestion time (their outputs and
/// outgoing edges); every view without a retention or coverage is
/// outgoing edges). A `OneEvaluation` node moved there becomes
/// `Lifetime::None`, as ingestion has no evaluation to hold it for (the
/// DAG's validation rejects `OneEvaluation` at ingestion): for example a scan five tumbling windows share streams to all five. Every
/// view without a retention or coverage is
/// retained for ever with a coverage of its own. For tests that bind
/// states at those nodes, as a deployment binds its views.
pub fn views_at(
Expand Down Expand Up @@ -87,6 +90,9 @@ pub fn views_at(
}
if at_ingestion.contains(&id) {
node.output_state.timing = ExecutionTiming::IngestionTime;
if matches!(node.lifetime, Lifetime::OneEvaluation(_)) {
node.lifetime = Lifetime::None;
}
}
}
for edge in &mut dag.edges {
Expand Down
40 changes: 37 additions & 3 deletions crates/executor/src/plan/mod.rs
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
//! Immutable physical DAG, operator contracts and pre-execution validation.
use crate::{
runtime::{Input, OutputStream, RunContext},
runtime::{IngestionOutputs, Input, OutputStream, RunContext},
Error,
};
use planner_types::stages::Lifetime;
use std::{
collections::{BTreeMap, BTreeSet},
fmt::Debug,
Expand Down Expand Up @@ -47,6 +48,10 @@ pub trait PhysicalOperator<V, S> {
pub(crate) struct Node<'a, V, S> {
pub(crate) inputs: Vec<NodeId>,
pub(crate) operator: Box<dyn PhysicalOperator<V, S> + 'a>,
/// How long its output is kept, from the plan: what the runtime runs it
/// by (see the crate README). Example: `OneEvaluation`, computed once
/// and held for its consumers at query time.
pub(crate) lifetime: Lifetime,
}
pub struct PhysicalDAG<'a, V, S> {
pub(crate) nodes: BTreeMap<NodeId, Node<'a, V, S>>,
Expand All @@ -59,24 +64,35 @@ impl<V, S> Default for PhysicalDAG<'_, V, S> {
}
}
impl<'a, V: 'a, S: Clone + PartialEq + Debug + 'a> PhysicalDAG<'a, V, S> {
/// Add node `id`, reading `inputs`, run by its plan `lifetime`.
pub fn add(
&mut self,
id: NodeId,
inputs: Vec<NodeId>,
operator: impl PhysicalOperator<V, S> + 'a,
lifetime: Lifetime,
) -> Result<(), Error> {
self.add_boxed(id, inputs, Box::new(operator))
self.add_boxed(id, inputs, Box::new(operator), lifetime)
}
/// [`Self::add`] with an already boxed operator.
pub fn add_boxed(
&mut self,
id: NodeId,
inputs: Vec<NodeId>,
operator: Box<dyn PhysicalOperator<V, S> + 'a>,
lifetime: Lifetime,
) -> Result<(), Error> {
if self.nodes.contains_key(&id) {
return Err(Error::Invalid(format!("duplicate node {id}")));
}
self.nodes.insert(id, Node { inputs, operator });
self.nodes.insert(
id,
Node {
inputs,
operator,
lifetime,
},
);
Ok(())
}
pub fn validate(&self, roots: &[NodeId]) -> Result<(), Error> {
Expand Down Expand Up @@ -153,6 +169,11 @@ impl<'a, V: 'a, S: Clone + PartialEq + Debug + 'a> PhysicalDAG<'a, V, S> {
.map(|(id, (_, properties))| (id, properties))
.collect())
}
/// A query run: one output stream per root, which the caller may drain
/// in any order. Rejects an ingestion scope (run it with
/// [`Self::execute_ingestion`]), and a computed node with
/// `Lifetime::None` and more than one consumer at one evaluation time (a
/// planner bug), before any operator starts.
pub fn execute<'r>(
&'r self,
roots: &[NodeId],
Expand All @@ -163,4 +184,17 @@ impl<'a, V: 'a, S: Clone + PartialEq + Debug + 'a> PhysicalDAG<'a, V, S> {
{
crate::runtime::execute(self, roots, context)
}
/// An ingestion run: one future that resolves to every root's batches,
/// in `roots` order, for the caller to write. Rejects a query scope (run
/// it with [`Self::execute`]).
pub fn execute_ingestion<'r>(
&'r self,
roots: &[NodeId],
context: RunContext,
) -> Result<impl std::future::Future<Output = Result<IngestionOutputs<V>, Error>> + 'r, Error>
where
'a: 'r,
{
crate::runtime::execute_ingestion(self, roots, context)
}
}
4 changes: 1 addition & 3 deletions crates/executor/src/runtime/context.rs
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,11 @@ pub enum Scope {
}
#[derive(Clone, Debug)]
pub struct Limits {
pub max_buffered_batches: usize,
pub max_bytes: usize,
}
impl Default for Limits {
fn default() -> Self {
Self {
max_buffered_batches: 8,
max_bytes: 64 * 1024 * 1024,
}
}
Expand All @@ -43,7 +41,7 @@ pub struct RunContext {
}
impl RunContext {
pub fn new(scope: Scope, limits: Limits) -> Result<Self, Error> {
if limits.max_buffered_batches == 0 || limits.max_bytes == 0 {
if limits.max_bytes == 0 {
return Err(Error::Invalid("execution limits must be positive".into()));
}
if matches!(&scope, Scope::Ingestion { window_start_ms, window_end_ms, .. } if window_start_ms > window_end_ms)
Expand Down
Loading
Loading