From 8fe46ae2926889d61c0e167ae4cab0e809a181c9 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 18:09:17 +0000 Subject: [PATCH 01/67] docs(concepts): memo groups, properties, requirements and the property decision tree Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6br9PL4SDeVS1XpezKJ --- docs/design_docs/concepts/README.md | 1 + .../memo-groups-properties-requirements.md | 307 ++++++++++++++++++ 2 files changed, 308 insertions(+) create mode 100644 docs/design_docs/concepts/memo-groups-properties-requirements.md diff --git a/docs/design_docs/concepts/README.md b/docs/design_docs/concepts/README.md index 8510e12e9..c204e363a 100644 --- a/docs/design_docs/concepts/README.md +++ b/docs/design_docs/concepts/README.md @@ -7,3 +7,4 @@ These pages introduce the vocabulary and representations used throughout the rep - [Mapping concepts](asap-aware-mapping.md) - [Pre-ASAP IR](pre-asap-ir.md) - [Post-ASAP IR](post-asap-ir.md) +- [Memo groups, properties and requirements](memo-groups-properties-requirements.md) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md new file mode 100644 index 000000000..74d5b56a8 --- /dev/null +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -0,0 +1,307 @@ +# Memo groups, properties and requirements + +This page defines the memo's main data structures: groups, candidates, properties and requirements. It also lists every property and requirement, and gives the decision tree Stage 1 and Stage 2 walk to enumerate candidates. The symbols follow the planner formulation: +- a candidate is $v = (e_v, p_v)$; +- its logical properties are $p^{\log}$ and its physical properties are $p^{\mathrm{phys}}$; +- a consumer input's requirements are $r^{\log}_{c,i}$ and $r^{\mathrm{phys}}_{c,i}$; +- whether a producer satisfies them is $\alpha^{\log}$ and $\alpha^{\mathrm{phys}}$. + +## Example + +Query A: `quantile_over_time(0.95, latency[1m])` with `max_error 0.01`. Query B: `quantile_over_time(0.99, latency[1m]) / quantile_over_time(0.50, latency[1m])` with the same target. + +``` +G1 Scan latency[1m] + producers: v1 = Scan (QueryTime, None) +G2 KLL state over G1 one group, whatever the size + producers: v2 = SummaryAgg[KLL](G1) θ = {k = 200} ε = 0.008 δ = 0.01 + v3 = SummaryAgg[KLL](G1) θ = {k = 345} ε = 0.005 δ = 0.01 + (each also expanded by Stage 2 into its (τ, ℓ, σ) alternatives) +G3 p95 of G2 producers: SummaryEstimate[q=0.95](G2) +G4 p99 of G2 producers: SummaryEstimate[q=0.99](G2) +G5 p50 of G2 producers: SummaryEstimate[q=0.50](G2) +G6 G4 / G5 producers: Divide(G4, G5) input requirement: error_cap on each input +Query roots: A → G3, B → G6 +``` + +The ε values are illustrative. + +- G2 is one group: the KLL state over `latency[1m]`. Its sizes are candidates of G2, not separate groups. +- G3, G4 and G5 are consumers of G2. They are not stored in G2; each is a group with its own producers whose expression reads G2. +- A meets 0.01 with either size. B adds its two operands' errors (0.016 with `k = 200`, 0.010 with `k = 345`) and needs `k = 345` for both, so a plan where all three read v3 builds one KLL state. + +## 1. Properties and requirements + +### Properties + +A property is a value a candidate has. Every property is one dimension of a product: a struct with one field per dimension, never a flat enum that lists the combinations. Invalid combinations are rejected by the struct's checked constructor. + +| Dimension | Values | Logical or physical | Fixed by | +|---|---|---|---| +| Which sketch to use (representation) | `Exact` (exact accumulator kinds: Sum, Count, Min, Max, Increase, Rate, IRate), or `Summary { family }` with family in KLL, DDSketch, CMS, CountSketch, CMS with heap, CountSketch with heap, UnivMon, HLL, KMV, the Hydra variants | logical | the operator $\operatorname{op}_v$ | +| Sketch parameters $\theta$ | per family, from a predefined list: KLL `k`; DDSketch `alpha`; CMS / CountSketch `width × depth`; CMS / CountSketch with heap `width × depth × heap_size`; UnivMon `heap_size × rows × cols × layers`; HLL `precision`; KMV `k` | logical | enumerated | +| Error bound $(\varepsilon_v, \delta_v)$ | precomputed per $\theta$, in the family's accuracy metric (below); exact is $(0, 0)$ | logical | derived from $\theta$, not chosen | +| Window configuration $\omega$ | none; or tumbling windows of width $w$ and count $n$, with $w$ from the gcd closure of the readers' ranges and steps. Later: sliding windows by merge or by subtraction | logical | enumerated | +| Timing $\tau$ | `IngestionTime`, `QueryTime` | physical | enumerated | +| Lifetime $\ell$ | `None`, `OneEvaluation(Medium)`, `AcrossEvaluations(Medium)`, with `Medium` in `Memory`, `Disk` | physical | enumerated | +| Schedule $\sigma$ | only when $\ell \ne$ None. `Periodic { interval, phase }` for `OneEvaluation`; `TumblingWindowGrid { phase }` for `AcrossEvaluations` | physical | enumerated from the readers' schedules | + +A candidate is materialized when $\ell \ne$ None. This is derived, not stored. + +Invalid combinations, rejected by `new`: + +| Combination | Why | +|---|---| +| (IngestionTime, OneEvaluation) | there is no evaluation at ingestion time | +| $\ell$ = None with a schedule | where an unmaterialized result runs is its reader's execution context, not a property | +| $\ell \ne$ None without a schedule | a materialized result must say on which times or grid it is delivered | +| OneEvaluation with a grid, AcrossEvaluations with a periodic schedule | each lifetime has one kind of schedule | +| `Exact` with a non-zero error bound | exact is $(0, 0)$ | + +Valid physical states: (QueryTime, None), (IngestionTime, None), (QueryTime, OneEvaluation(m)), (QueryTime, AcrossEvaluations(m)), (IngestionTime, AcrossEvaluations(m)), for each medium $m$. + +### Requirements + +A requirement is a constraint that a consumer, a query or the deployment puts on producers. One requirement can constrain several property dimensions, and one dimension can be constrained by several requirements. + +| Requirement | Who states it | Values | Constrains | Checked where | +|---|---|---|---|---| +| Accuracy | each query, at its root | metric: absolute (in the value's units, or times the stream's L1 or L2 norm), relative, rank; bound $\varepsilon_q$; failure probability $\delta_q$; scope: per value, whole result, or groups above a share of the records | representation and $\theta$ of every approximate candidate the query reads, through their $(\varepsilon_v, \delta_v)$ | Stage 3, row (K): the composed error at the root. Not per edge, because a per-input bound would fix each input's share of the budget | +| Validity cap `error_cap` | a non-linear consumer (ratio, product), per input | $\varepsilon$ cap | $\theta$ of the producer | per edge, $\alpha^{\log}$: row (K)'s linear bound holds only below the cap | +| Summary family | a consumer that reads summary state, per input | a family, e.g. KLL for a quantile readout | representation | per edge, $\alpha^{\log}$ | +| Window | a consumer that reads windows, per input | the reader's range and step | window configuration: the width must divide both | per edge, $\alpha^{\log}$ | +| Capability of ingestion-time compute | the deployment and the data workload | the source has an ingestion pipeline; the executor supports the family; query-time results may be kept across evaluations | timing; representation; (QueryTime, AcrossEvaluations) | candidate generation: such candidates are not offered | +| Reader timing and evaluation | a consumer, per input | when it reads (ingestion or query time) and from which evaluation (same or later) | timing and lifetime of the producer | per edge, $\alpha^{\mathrm{phys}}$ | +| Reader schedule | a periodic consumer, per input | its interval and phase $(I_c, \varphi_c)$ | schedule of the producer | per edge, $\alpha^{\mathrm{phys}}$ | +| Latency, resource budgets | the workload | per-query latency bound; memory and CPU budgets | the whole plan | Stage 3, rows (L) and (B) | + +Accuracy metrics that may be added later, when a primitive needs them: top-k recall or precision, and a false-positive rate for membership summaries. + +How requirements map to property dimensions: + +``` +Accuracy (query) ──► representation, θ (through ε_v, δ_v; row (K)) +error_cap (edge) ──► θ +Summary family (edge) ──► representation +Window (edge) ──► window configuration +Ingestion capability (deploy)──► timing, representation, lifetime +Reader timing/evaluation ──► timing, lifetime +Reader schedule ──► schedule +Latency, budgets (workload) ──► every dimension, through cost +``` + +## 2. Decision tree of properties + +Each memo group's candidates are the leaves of a tree. Every level decides one property dimension, and each edge out of a node is one value of that dimension. A path from the root to a leaf is one candidate. + +```mermaid +flowchart TD + G["Group G
(what is computed)"] --> R1["Exact"] + G --> R2["Summary: KLL"] + G --> R3["Summary: DDSketch"] + G --> R4["…other families
(prune: executor support,
consumer's family requirement)"] + + R2 --> T1["θ: k = 200
ε = 0.008"] + R2 --> T2["θ: k = 345
ε = 0.005"] + R2 --> T3["θ: k = …
(prune: ε above every
reading query's target)"] + + T2 --> W1["ω: no windows"] + T2 --> W2["ω: tumbling, width w
(prune: w must divide
some reader's range and step)"] + + W2 --> P1["τ = QueryTime"] + W2 --> P2["τ = IngestionTime
(prune: no ingestion pipeline)"] + + P1 --> L1["ℓ = None"] + P1 --> L2["ℓ = OneEvaluation(m)"] + P1 --> L3["ℓ = AcrossEvaluations(m)
(prune: deployment keeps
no query-time results)"] + P2 --> L4["ℓ = None"] + P2 --> L5["ℓ = AcrossEvaluations(m)"] + P2 -. invalid .-> L6["ℓ = OneEvaluation"] + + L2 --> S1["σ: Periodic (I, φ)
from the gcd closure
of the readers' schedules"] + L3 --> S2["σ: grid phase φ mod w"] + L5 --> S3["σ: grid phase 0"] +``` + +The levels, top to bottom: + +| Level | Dimension | Stage | Values offered | Pruned when | +|---|---|---|---|---| +| 1 | representation (operator) | Stage 1, Pass 1 and Pass 2 rules | each way to compute the group | the executor cannot build the family; no consumer's family requirement accepts it | +| 2 | $\theta$ | Stage 1, Pass 1 | the family's predefined list | $\varepsilon_v$ alone exceeds every reading query's target | +| 3 | window configuration $\omega$ | Stage 1, Pass 2 | tumbling widths from the gcd closure of readers' ranges and steps | no reader's range and step is a multiple of the width | +| 4 | timing $\tau$ | Stage 2 | QueryTime; IngestionTime | the source has no ingestion pipeline | +| 5 | lifetime $\ell$ | Stage 2 | None, OneEvaluation(m), AcrossEvaluations(m) | invalid with $\tau$; the deployment keeps no query-time results across evaluations | +| 6 | schedule $\sigma$ | Stage 2 | the gcd closure $\Sigma$ of the readers' schedules | only offered when $\ell \ne$ None | + +What the tree does not decide: +- **Which producer each consumer input reads.** That is the edge choice $y_{u,c,i}$ in Stage 3. Two consumers of one group may read different leaves, e.g. A reads `k = 200` and B reads `k = 345`. +- **Where an unmaterialized candidate runs.** A candidate with $\ell$ = None runs inside each reader's execution context; Stage 3 chooses it. +- **How much a materialized candidate retains.** The amount $h_v$ depends on the readers' longest range; Stage 3 chooses it. + +### Order of decisions + +**The order does not affect correctness.** The set of leaves is the product of the dimensions' value sets, minus the invalid combinations and the pruned values. Every prune rule depends only on the value it removes and on values already decided above it, so any order that decides a pruned dimension after the dimensions its prune rule reads yields the same set of leaves. + +**The order affects the efficiency of enumeration.** The number of nodes visited is $\sum_{d} \prod_{d' \le d} |\text{values kept at } d'|$, so a dimension that prunes much should come early, and a dimension whose values multiply without pruning should come late. Four considerations set the order above: + +1. **Logical before physical.** Logical dimensions (representation, $\theta$, $\omega$) change the result's values or error; physical dimensions ($\tau$, $\ell$, $\sigma$) change only when and how long it is computed. Deciding logical dimensions first lets Stage 1 hash-cons groups and check $\alpha^{\log}$ on edges once, before Stage 2 multiplies each logical candidate by its physical alternatives. +2. **Strong prunes early.** The family is pruned by the executor and by consumers' family requirements, and $\theta$ by the accuracy target, before the window and physical levels multiply them. +3. **Dependent dimensions adjacent.** $\ell$ is constrained by $\tau$ (no OneEvaluation at ingestion), and $\sigma$ exists only when $\ell \ne$ None and its kind follows $\ell$. Deciding them in the order $\tau \to \ell \to \sigma$ means each invalid combination is pruned at the first level where it can be detected. +4. **Deployment-wide prunes as early as they can go.** If the deployment has no ingestion pipeline, the IngestionTime subtree is pruned once per candidate. Moving $\tau$ above $\omega$ would prune it once per group instead; this is an alternative order with the same leaves. + +Alternative orders worth comparing: +- **$\tau$ first.** In a deployment where most sources have no ingestion pipeline, deciding timing at the group level prunes half the tree before $\theta$ and $\omega$ multiply it. +- **$\theta$ last among logical dimensions.** $\theta$ has the most values. Deciding $\omega$ before $\theta$ visits fewer nodes when many widths are pruned by readers' ranges. +- **Requirement-driven (top-down) vs producer-driven (bottom-up).** Top-down enumeration starts from each consumer's requirement and offers only the values it accepts; bottom-up offers every value and lets $\alpha$ filter edges. Top-down visits fewer nodes but must union the requirements of all consumers of a group, so that sharing between consumers is not lost. + +## 3. Modeling a memo group by its producers + +### Producers, not consumers + +A memo group stores its **producers**: the candidates that compute it. It does not store its consumers. A consumer of $G$ is a candidate of another group whose expression reads $G$; the producer–consumer edges $(u, c, i)$ connect them. + +| | Group stores its producers (chosen) | Group stores each consumer's own producer | +|---|---|---| +| Group identity | what is computed: operator and input groups, without properties | what is computed **for one reader** | +| Two readers of the same state | one group, two edges | two copies, joined only by an extra "shared build" candidate | +| A rule adds a new way to compute | one new producer in the group; every reader sees it | the rule must copy every reader that could use it | +| Where requirements live | on the consumer's input, $r_{c,i}$ | on the producer, so a producer is fixed to one reader | +| Accuracy | checked at each query root over the chosen edges | fixed per producer as a share of one query's budget | + +Groups found by hash-consing are therefore shared automatically. Two queries that read the same state read one group, and Stage 3 decides through edges whether they read the same producer. + +Consumers of a group are derived, not stored: `consumers(G)` is the set of candidates whose expression has $G$ as an input. + +### Memo group data structure + +```rust +/// The memo: every group, and each query's root over groups. +pub struct Memo { + pub groups: Vec, + pub query_roots: Vec>, +} + +/// G: one logical result, or one state family, over specified input data. +/// Groups are hash-consed on `identity`, so an identical sub-expression of two +/// queries is one group. +pub struct MemoGroup { + pub id: MemoGroupId, + /// What is computed, without any property: the operator kind, its input + /// groups, and the arguments that change the result's meaning (family, + /// reduction, grouping, filter, query window). Sketch parameters, error + /// bounds and window layout are not part of it. + pub identity: GroupIdentity, + /// The producers of this group: every way to compute it. + pub candidates: Vec, +} +``` + +### Property definition + +```rust +/// p_v^log: one field per dimension. `new` rejects `Exact` with a non-zero +/// error bound. +pub struct LogicalProperties { + representation: Representation, // which sketch, and θ + error_bound: ErrorBound, // (ε_v, δ_v), precomputed per θ + window_layout: Option, // ω_v +} +pub enum Representation { + Exact, + Summary { family: SummaryFamily, parameters: SummaryParameters }, +} +pub struct ErrorBound { metric: AccuracyMetric, epsilon: f64, delta: f64 } +pub enum WindowLayout { Tumbling { width_ms: u64, count: u32 } } // sliding variants later + +/// p_v^phys: one field per dimension. `new` rejects the invalid combinations. +pub struct PhysicalProperties { + timing: ExecutionTiming, // τ + lifetime: Lifetime, // ℓ + schedule: Option, // σ, only when ℓ ≠ None +} +pub enum ExecutionTiming { IngestionTime, QueryTime } +pub enum Lifetime { None, OneEvaluation(Medium), AcrossEvaluations(Medium) } +pub enum Medium { Memory, Disk } +pub enum Schedule { + Periodic { interval_ms: u64, phase_ms: u64 }, + TumblingWindowGrid { phase_ms: u64 }, +} +``` + +### Property requirement definition + +```rust +/// r^log_{c,i}: one optional constraint per dimension; `None` accepts any value. +pub struct LogicalRequirement { + pub family: Option, + pub window: Option, // the reader's range and step + pub error_cap: Option, // validity of row (K)'s linear bound +} + +/// r^phys_{c,i}: one field per dimension. `new` rejects (IngestionTime, Later). +pub struct PhysicalRequirement { + reader_timing: ExecutionTiming, + reads_from: Evaluation, // Same or Later + reader_schedule: Option, // (I_c, φ_c) of a periodic reader +} + +/// α^log and α^phys: one check per dimension, joined by "and". +pub fn satisfies(producer: &LogicalProperties, requirement: &LogicalRequirement) -> bool; +pub fn satisfies_physical(producer: &PhysicalProperties, requirement: &PhysicalRequirement) -> bool; + +/// The accuracy target stays at the query root; Stage 3 checks it (row (K)). +pub struct QueryRequirement { pub accuracy: AccuracyTarget } +``` + +### Candidate data structure definition + +```rust +/// v = (e_v, p_v^log): one producer of a memo group, from Stage 1. +pub struct CandidateForMemoGroup { + pub expression: CandidateExpression, // e_v + pub logical_properties: LogicalProperties, // p_v^log + /// r^log_{c,i}: one per input group of `expression`. + pub input_requirements: Vec, +} + +/// e_v = op_v(G_{v,1}, …, G_{v,n}): one operator with its arguments, whose +/// children are groups. No private nodes. +pub struct CandidateExpression { pub operator: Operator } + +/// One physical alternative of a logical candidate, from Stage 2. +pub struct PhysicalCandidate { + pub logical: (MemoGroupId, usize), // which logical candidate + pub properties: PhysicalProperties, // p_v^phys + pub input_requirements: Vec, // r^phys_{c,i} +} + +/// (u, c, i): producer u may serve input i of consumer c, when α^log and α^phys hold. +pub struct Edge { pub producer: CandidateId, pub consumer: CandidateId, pub input: usize } +``` + +### Extensibility + +**Add a value to a property dimension**, e.g. a new summary family or `Medium::Disk`: +1. Add the enum variant. +2. Every exhaustive `match` on the dimension stops compiling until it handles the new value. This is why matches on properties name every value and never use a `_` arm. +3. For a family: add its predefined $\theta$ list, its error-bound function, its cost, and the readouts it answers. +4. Add any new invalid combinations to `new`. + +**Add a property dimension**, e.g. a compression codec or a partitioning: +1. Decide whether it is logical (it changes the result's values or error) or physical (it changes only when, where or how long the result is computed). Add a field to `LogicalProperties` or `PhysicalProperties`. +2. If consumers can constrain it, add an optional field to the matching requirement struct and one check to `satisfies` or `satisfies_physical`. +3. Add its invalid combinations with the other dimensions to `new`. +4. Add a level to the decision tree, with its prune rule, and place it by the considerations in "Order of decisions". +5. Add its cost terms. + +**Add a requirement:** +1. If it constrains one edge (a consumer input), add an optional field to `LogicalRequirement` or `PhysicalRequirement`, set it where the consumer is created, and check it in `satisfies`. +2. If it constrains a whole query or the whole plan (accuracy, latency, budgets), add it to the query's or workload's requirements and a row to Stage 3. Do not split it into per-edge requirements: that fixes each input's share in advance. +3. If it comes from the deployment, prune the values it rules out during candidate generation. + +**Add a rule:** +1. A rule matches over groups and their candidates' expressions, never over a reader's private operators. +2. It adds candidates $(e, p^{\log})$ to an existing group, or adds a new group with its candidates. It never copies a reader: every existing consumer of the group sees the new candidate through edges. +3. A candidate whose result has the same meaning as an existing group is added to that group, so hash-consing keeps sharing. +4. Each new candidate states its input requirements. From 8d59b7d7c987a24b378d9e46373c1655c33b7d89 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 19:01:51 +0000 Subject: [PATCH 02/67] docs(concepts): memo group as expression, input requirements and candidates over existing IR Co-Authored-By: Claude Opus 5.5 --- .../memo-groups-properties-requirements.md | 228 +++++++++++------- 1 file changed, 136 insertions(+), 92 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 74d5b56a8..37623e34c 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -1,7 +1,7 @@ # Memo groups, properties and requirements This page defines the memo's main data structures: groups, candidates, properties and requirements. It also lists every property and requirement, and gives the decision tree Stage 1 and Stage 2 walk to enumerate candidates. The symbols follow the planner formulation: -- a candidate is $v = (e_v, p_v)$; +- a group has an expression $e_G$ over its producer groups, and a candidate $v$ of it has properties $p_v$; - its logical properties are $p^{\log}$ and its physical properties are $p^{\mathrm{phys}}$; - a consumer input's requirements are $r^{\log}_{c,i}$ and $r^{\mathrm{phys}}_{c,i}$; - whether a producer satisfies them is $\alpha^{\log}$ and $\alpha^{\mathrm{phys}}$. @@ -30,7 +30,7 @@ The ε values are illustrative. - G3, G4 and G5 are consumers of G2. They are not stored in G2; each is a group with its own producers whose expression reads G2. - A meets 0.01 with either size. B adds its two operands' errors (0.016 with `k = 200`, 0.010 with `k = 345`) and needs `k = 345` for both, so a plan where all three read v3 builds one KLL state. -## 1. Properties and requirements +## 1. Properties and requirements over properties ### Properties @@ -38,9 +38,9 @@ A property is a value a candidate has. Every property is one dimension of a prod | Dimension | Values | Logical or physical | Fixed by | |---|---|---|---| -| Which sketch to use (representation) | `Exact` (exact accumulator kinds: Sum, Count, Min, Max, Increase, Rate, IRate), or `Summary { family }` with family in KLL, DDSketch, CMS, CountSketch, CMS with heap, CountSketch with heap, UnivMon, HLL, KMV, the Hydra variants | logical | the operator $\operatorname{op}_v$ | -| Sketch parameters $\theta$ | per family, from a predefined list: KLL `k`; DDSketch `alpha`; CMS / CountSketch `width × depth`; CMS / CountSketch with heap `width × depth × heap_size`; UnivMon `heap_size × rows × cols × layers`; HLL `precision`; KMV `k` | logical | enumerated | -| Error bound $(\varepsilon_v, \delta_v)$ | precomputed per $\theta$, in the family's accuracy metric (below); exact is $(0, 0)$ | logical | derived from $\theta$, not chosen | +| Which family to use (operator) | `Exact` (exact accumulator kinds: Sum, Count, Min, Max, Increase, Rate, IRate), or `Summary { family }` with family in KLL, DDSketch, CMS, CountSketch, CMS with heap, CountSketch with heap, UnivMon, HLL, KMV, the Hydra variants | logical | enumerated | +| Parameter configuration $\theta$ | per family, from a predefined list: KLL `k`; DDSketch `alpha`; CMS / CountSketch `width × depth`; CMS / CountSketch with heap `width × depth × heap_size`; UnivMon `heap_size × rows × cols × layers`; HLL `precision`; KMV `k` | logical | enumerated | +| Error bound $(\varepsilon_v, \delta_v)$ | computed on demand from $\theta$ and the statistic, in the family's accuracy metric (below); exact is $(0, 0)$ | logical | derived from $\theta$, not stored or chosen | | Window configuration $\omega$ | none; or tumbling windows of width $w$ and count $n$, with $w$ from the gcd closure of the readers' ranges and steps. Later: sliding windows by merge or by subtraction | logical | enumerated | | Timing $\tau$ | `IngestionTime`, `QueryTime` | physical | enumerated | | Lifetime $\ell$ | `None`, `OneEvaluation(Medium)`, `AcrossEvaluations(Medium)`, with `Medium` in `Memory`, `Disk` | physical | enumerated | @@ -56,21 +56,20 @@ Invalid combinations, rejected by `new`: | $\ell$ = None with a schedule | where an unmaterialized result runs is its reader's execution context, not a property | | $\ell \ne$ None without a schedule | a materialized result must say on which times or grid it is delivered | | OneEvaluation with a grid, AcrossEvaluations with a periodic schedule | each lifetime has one kind of schedule | -| `Exact` with a non-zero error bound | exact is $(0, 0)$ | Valid physical states: (QueryTime, None), (IngestionTime, None), (QueryTime, OneEvaluation(m)), (QueryTime, AcrossEvaluations(m)), (IngestionTime, AcrossEvaluations(m)), for each medium $m$. -### Requirements +### Requirements over properties A requirement is a constraint that a consumer, a query or the deployment puts on producers. One requirement can constrain several property dimensions, and one dimension can be constrained by several requirements. | Requirement | Who states it | Values | Constrains | Checked where | |---|---|---|---|---| -| Accuracy | each query, at its root | metric: absolute (in the value's units, or times the stream's L1 or L2 norm), relative, rank; bound $\varepsilon_q$; failure probability $\delta_q$; scope: per value, whole result, or groups above a share of the records | representation and $\theta$ of every approximate candidate the query reads, through their $(\varepsilon_v, \delta_v)$ | Stage 3, row (K): the composed error at the root. Not per edge, because a per-input bound would fix each input's share of the budget | +| Accuracy | each query, at its root | metric: absolute (in the value's units, or times the stream's L1 or L2 norm), relative, rank; bound $\varepsilon_q$; failure probability $\delta_q$; scope: per value, whole result, or groups above a share of the records | operator family and $\theta$ of every approximate candidate the query reads, through their $(\varepsilon_v, \delta_v)$ | Stage 3, row (K): the composed error at the root. Not per edge, because a per-input bound would fix each input's share of the budget | | Validity cap `error_cap` | a non-linear consumer (ratio, product), per input | $\varepsilon$ cap | $\theta$ of the producer | per edge, $\alpha^{\log}$: row (K)'s linear bound holds only below the cap | -| Summary family | a consumer that reads summary state, per input | a family, e.g. KLL for a quantile readout | representation | per edge, $\alpha^{\log}$ | +| Summary family | a consumer that reads summary state, per input | a family, e.g. KLL for a quantile readout | operator family | per edge, $\alpha^{\log}$ | | Window | a consumer that reads windows, per input | the reader's range and step | window configuration: the width must divide both | per edge, $\alpha^{\log}$ | -| Capability of ingestion-time compute | the deployment and the data workload | the source has an ingestion pipeline; the executor supports the family; query-time results may be kept across evaluations | timing; representation; (QueryTime, AcrossEvaluations) | candidate generation: such candidates are not offered | +| Capability of ingestion-time compute | the deployment and the data workload | the source has an ingestion pipeline; the executor supports the family; query-time results may be kept across evaluations | timing; operator family; (QueryTime, AcrossEvaluations) | candidate generation: such candidates are not offered | | Reader timing and evaluation | a consumer, per input | when it reads (ingestion or query time) and from which evaluation (same or later) | timing and lifetime of the producer | per edge, $\alpha^{\mathrm{phys}}$ | | Reader schedule | a periodic consumer, per input | its interval and phase $(I_c, \varphi_c)$ | schedule of the producer | per edge, $\alpha^{\mathrm{phys}}$ | | Latency, resource budgets | the workload | per-query latency bound; memory and CPU budgets | the whole plan | Stage 3, rows (L) and (B) | @@ -80,11 +79,11 @@ Accuracy metrics that may be added later, when a primitive needs them: top-k rec How requirements map to property dimensions: ``` -Accuracy (query) ──► representation, θ (through ε_v, δ_v; row (K)) +Accuracy (query) ──► operator family, θ (through ε_v, δ_v; row (K)) error_cap (edge) ──► θ -Summary family (edge) ──► representation +Summary family (edge) ──► operator family Window (edge) ──► window configuration -Ingestion capability (deploy)──► timing, representation, lifetime +Ingestion capability (deploy)──► timing, operator family, lifetime Reader timing/evaluation ──► timing, lifetime Reader schedule ──► schedule Latency, budgets (workload) ──► every dimension, through cost @@ -127,7 +126,7 @@ The levels, top to bottom: | Level | Dimension | Stage | Values offered | Pruned when | |---|---|---|---|---| -| 1 | representation (operator) | Stage 1, Pass 1 and Pass 2 rules | each way to compute the group | the executor cannot build the family; no consumer's family requirement accepts it | +| 1 | family (operator) | Stage 1, Pass 1 and Pass 2 rules | each way to compute the group | the executor cannot build the family; no consumer's family requirement accepts it | | 2 | $\theta$ | Stage 1, Pass 1 | the family's predefined list | $\varepsilon_v$ alone exceeds every reading query's target | | 3 | window configuration $\omega$ | Stage 1, Pass 2 | tumbling widths from the gcd closure of readers' ranges and steps | no reader's range and step is a multiple of the width | | 4 | timing $\tau$ | Stage 2 | QueryTime; IngestionTime | the source has no ingestion pipeline | @@ -145,7 +144,7 @@ What the tree does not decide: **The order affects the efficiency of enumeration.** The number of nodes visited is $\sum_{d} \prod_{d' \le d} |\text{values kept at } d'|$, so a dimension that prunes much should come early, and a dimension whose values multiply without pruning should come late. Four considerations set the order above: -1. **Logical before physical.** Logical dimensions (representation, $\theta$, $\omega$) change the result's values or error; physical dimensions ($\tau$, $\ell$, $\sigma$) change only when and how long it is computed. Deciding logical dimensions first lets Stage 1 hash-cons groups and check $\alpha^{\log}$ on edges once, before Stage 2 multiplies each logical candidate by its physical alternatives. +1. **Logical before physical.** Logical dimensions (operator family, $\theta$, $\omega$) change the result's values or error; physical dimensions ($\tau$, $\ell$, $\sigma$) change only when and how long it is computed. Deciding logical dimensions first lets Stage 1 hash-cons groups and check $\alpha^{\log}$ on edges once, before Stage 2 multiplies each logical candidate by its physical alternatives. 2. **Strong prunes early.** The family is pruned by the executor and by consumers' family requirements, and $\theta$ by the accuracy target, before the window and physical levels multiply them. 3. **Dependent dimensions adjacent.** $\ell$ is constrained by $\tau$ (no OneEvaluation at ingestion), and $\sigma$ exists only when $\ell \ne$ None and its kind follows $\ell$. Deciding them in the order $\tau \to \ell \to \sigma$ means each invalid combination is pruned at the first level where it can be detected. 4. **Deployment-wide prunes as early as they can go.** If the deployment has no ingestion pipeline, the IngestionTime subtree is pruned once per candidate. Moving $\tau$ above $\omega$ would prune it once per group instead; this is an alternative order with the same leaves. @@ -155,26 +154,31 @@ Alternative orders worth comparing: - **$\theta$ last among logical dimensions.** $\theta$ has the most values. Deciding $\omega$ before $\theta$ visits fewer nodes when many widths are pruned by readers' ranges. - **Requirement-driven (top-down) vs producer-driven (bottom-up).** Top-down enumeration starts from each consumer's requirement and offers only the values it accepts; bottom-up offers every value and lets $\alpha$ filter edges. Top-down visits fewer nodes but must union the requirements of all consumers of a group, so that sharing between consumers is not lost. -## 3. Modeling a memo group by its producers +## 3. Modeling a memo group -### Producers, not consumers +A memo group stores its **producers**: all candidate ways to compute its result, before Stage 3 selects any. A consumer of $G$ is a candidate of another group whose expression reads $G$; the producer–consumer edges $(u, c, i)$ connect them. Here, `consumers(G)` is derived from those input references rather than stored in the group. -A memo group stores its **producers**: the candidates that compute it. It does not store its consumers. A consumer of $G$ is a candidate of another group whose expression reads $G$; the producer–consumer edges $(u, c, i)$ connect them. +> Side note: [CMU-DB's optd-original](https://github.com/cmu-db/optd-original) stores the list of children as input producers to a memo group; while storing consumers vs storing producers in the memo group structure to form the DAG are equivalent in terms of correctness and efficiency. -| | Group stores its producers (chosen) | Group stores each consumer's own producer | -|---|---|---| -| Group identity | what is computed: operator and input groups, without properties | what is computed **for one reader** | -| Two readers of the same state | one group, two edges | two copies, joined only by an extra "shared build" candidate | -| A rule adds a new way to compute | one new producer in the group; every reader sees it | the rule must copy every reader that could use it | -| Where requirements live | on the consumer's input, $r_{c,i}$ | on the producer, so a producer is fixed to one reader | -| Accuracy | checked at each query root over the chosen edges | fixed per producer as a share of one query's budget | +Candidate enumeration efficiency depends on: + +- **Property and requirement fields:** which dimensions are represented, their legal values, which values are derived, and which constraints permit pruning or sharing (Section 1). +- **Decision-tree order:** when each dimension is decided and its requirements are checked, so invalid combinations can be pruned before later dimensions multiply them (Section 2). + +Groups found by hash-consing expose the same producer alternatives to all their readers. Stage 3 decides whether readers use the same producer candidate or different ones. **Reusing a computation alternative in the memo is distinct from reusing its computed result at runtime. Selecting the same candidate for multiple readers does not by itself mean computing it only once.** -Groups found by hash-consing are therefore shared automatically. Two queries that read the same state read one group, and Stage 3 decides through edges whether they read the same producer. +- **Materialized candidate** ($\ell \ne$ None): its result is retained. Readers connected to the same materialized producer can reuse that result instead of computing it separately, subject to its lifetime and schedule. +- **Unmaterialized candidate** ($\ell =$ None): no independent shared result is retained. The computation runs as part of each reader's execution, so multiple readers selecting the same candidate may each execute it separately. -Consumers of a group are derived, not stored: `consumers(G)` is the set of candidates whose expression has $G$ as an input. +For example, if p50 and p99 select the same KLL candidate, a shared materialized producer builds and retains one KLL state for both readouts within the applicable lifetime and schedule. With an unmaterialized candidate, the p50 execution can build its own KLL state and the p99 execution another, even though both builds use the same candidate configuration. ### Memo group data structure +A memo group holds three things: +- **Query semantics:** its expression, and the producer groups the expression reads. +- **Eligibility:** for each input, the requirement a producer must meet. +- **Candidates:** the ways to compute it. + ```rust /// The memo: every group, and each query's root over groups. pub struct Memo { @@ -182,39 +186,66 @@ pub struct Memo { pub query_roots: Vec>, } -/// G: one logical result, or one state family, over specified input data. -/// Groups are hash-consed on `identity`, so an identical sub-expression of two -/// queries is one group. pub struct MemoGroup { - pub id: MemoGroupId, - /// What is computed, without any property: the operator kind, its input - /// groups, and the arguments that change the result's meaning (family, - /// reduction, grouping, filter, query window). Sketch parameters, error - /// bounds and window layout are not part of it. - pub identity: GroupIdentity, - /// The producers of this group: every way to compute it. - pub candidates: Vec, + // Query semantics. + /// The logical operator, from the existing IR. Its children are the + /// producer groups G_i it reads, in order (`expression.children()`), which + /// fix the input data and schema. + pub expression: Operator, + // Eligibility. + /// r_i: one requirement per producer group, by input index i. + pub input_requirements: Vec, + // Candidates. + /// The ways to compute this group; x_v in the MILP. + pub candidates: Vec, +} +``` + +The expression is the existing IR's [`Operator`](../../../crates/types/src/ir/operator/node.rs), with `C = MemoGroupId`. It wraps [`NonASAPOp`](../../../crates/types/src/ir/operator/non_asap.rs) or [`ASAPOp`](../../../crates/types/src/ir/operator/asap.rs), and each variant carries its own arguments and its inputs. For example, `Scan` carries the source, predicates and schema, and `Filter` carries a predicate and its child. No new identity type is needed. + +Hash-consing finds or inserts a group by its expression: the operator, its arguments and its input group ids. For example, the KLL state over one source and query window is one group for `k = 200` and `k = 345`; these are two candidates of that group. A different source, filter or query window gives a different group. Properties are not part of the expression: the family, sketch parameters, window layout, timing, lifetime and schedule belong to candidates. A requested query window belongs to the expression; the internal window layout used to compute it does not. + +`MemoGroupId` is the handle of an interned group. Looking up an existing expression returns that group's id instead of adding a second group. + +### Candidate data structure definition + +```rust +/// v: one way to compute its group. The group's expression says what is +/// computed; the candidate's properties say how. +pub struct Candidate { + /// One value per property dimension. + pub properties: Properties, +} +``` + +Stage 1 decides the logical dimensions of each candidate, and Stage 2 expands each one into its physical alternatives: + +```rust +pub struct Properties { + pub logical: LogicalProperties, // Stage 1 + pub physical: PhysicalProperties, // Stage 2 } + +/// (u, c, i): producer u may serve input i of consumer c when u's properties +/// satisfy c's group's input_requirements[i]. +pub struct Edge { pub producer: CandidateId, pub consumer: CandidateId, pub input: usize } ``` ### Property definition +A property is a struct with one field per dimension (P1, P2, …), each its own enum: + ```rust -/// p_v^log: one field per dimension. `new` rejects `Exact` with a non-zero -/// error bound. +/// p_v^log. pub struct LogicalProperties { - representation: Representation, // which sketch, and θ - error_bound: ErrorBound, // (ε_v, δ_v), precomputed per θ - window_layout: Option, // ω_v -} -pub enum Representation { - Exact, - Summary { family: SummaryFamily, parameters: SummaryParameters }, + family: Family, // which sketch, or exact + parameter_configuration: Option, // θ; None for exact + window_layout: Option, // ω_v } -pub struct ErrorBound { metric: AccuracyMetric, epsilon: f64, delta: f64 } +pub enum Family { Exact(ExactKind), Sketch(SketchAlgorithm) } pub enum WindowLayout { Tumbling { width_ms: u64, count: u32 } } // sliding variants later -/// p_v^phys: one field per dimension. `new` rejects the invalid combinations. +/// p_v^phys. `new` rejects the invalid combinations. pub struct PhysicalProperties { timing: ExecutionTiming, // τ lifetime: Lifetime, // ℓ @@ -227,59 +258,72 @@ pub enum Schedule { Periodic { interval_ms: u64, phase_ms: u64 }, TumblingWindowGrid { phase_ms: u64 }, } + +// Reuse the existing family-specific conversions in asap_accuracy. +// Parameter configuration → error bound (including metric and failure probability). +// `family` describes the exact value or sketch, including its parameters; +// `query` identifies the statistic read from it. +pub(crate) fn local_guarantee( + family: &FieldDataType, + query: &SketchStatistic, +) -> Option; + +// Error target → parameter configuration for the chosen family and intent. +pub(crate) fn size_params( + kind: SketchAlgorithm, + intent: &AggIntent, + eps: f64, + delta: f64, +) -> SketchParams; ``` +`new` checks that `parameter_configuration` belongs to `family`, as [`SketchKind::new`](../../../crates/types/src/ir/schema/sketch.rs) does today, and reuses the existing `SketchParams` type. + +`error_bound` is derived when needed, not stored in `LogicalProperties`. Reuse [`estimators::local_guarantee`](../../../crates/accuracy/src/estimators/mod.rs) for parameters → guarantee and [`estimators::size_params`](../../../crates/accuracy/src/estimators/mod.rs) for target → parameters. These existing crate-private functions dispatch to the family's accuracy and sizing implementations; the declarations above document those interfaces, not new implementations. [`sizing::default_size_params`](../../../crates/accuracy/src/sizing.rs) already delegates to `size_params`. If another crate needs direct access, expose the existing implementation through the accuracy crate rather than duplicate its formulas. + +Deriving a guarantee also needs the statistic being read: one family's parameters can give different guarantees for different statistics. Exact values yield an exact guarantee; an unsupported statistic or missing error model yields `None`. Target → parameters is a sizing operation with discrete values and family limits, not an exact mathematical inverse; validate the derived guarantee against the target rather than assume the target is achievable. + ### Property requirement definition +A property requirement is also a struct with one field per dimension (R1, R2, …), each its own enum. `None` accepts any value. + ```rust -/// r^log_{c,i}: one optional constraint per dimension; `None` accepts any value. -pub struct LogicalRequirement { +/// r_i: what input i's producer must satisfy. +pub struct PropertyRequirement { + pub accuracy: Option, pub family: Option, - pub window: Option, // the reader's range and step - pub error_cap: Option, // validity of row (K)'s linear bound + pub window: Option, // the reader's range and step + pub reader_timing: Option, + pub reads_from: Option, // Same or Later + pub reader_schedule: Option, // (I_c, φ_c) of a periodic reader } -/// r^phys_{c,i}: one field per dimension. `new` rejects (IngestionTime, Later). -pub struct PhysicalRequirement { - reader_timing: ExecutionTiming, - reads_from: Evaluation, // Same or Later - reader_schedule: Option, // (I_c, φ_c) of a periodic reader -} - -/// α^log and α^phys: one check per dimension, joined by "and". -pub fn satisfies(producer: &LogicalProperties, requirement: &LogicalRequirement) -> bool; -pub fn satisfies_physical(producer: &PhysicalProperties, requirement: &PhysicalRequirement) -> bool; - -/// The accuracy target stays at the query root; Stage 3 checks it (row (K)). -pub struct QueryRequirement { pub accuracy: AccuracyTarget } +/// The set of legal properties for an input: legal_properties(r) = f(r). +pub fn satisfies(producer: &Properties, requirement: &PropertyRequirement) -> bool; ``` -### Candidate data structure definition +The set of legal properties is a function of the requirement, $\text{legal}(r) = f(r)$. One requirement can be written at different levels. For example, the same accuracy requirement can be stated two ways: -```rust -/// v = (e_v, p_v^log): one producer of a memo group, from Stage 1. -pub struct CandidateForMemoGroup { - pub expression: CandidateExpression, // e_v - pub logical_properties: LogicalProperties, // p_v^log - /// r^log_{c,i}: one per input group of `expression`. - pub input_requirements: Vec, -} - -/// e_v = op_v(G_{v,1}, …, G_{v,n}): one operator with its arguments, whose -/// children are groups. No private nodes. -pub struct CandidateExpression { pub operator: Operator } +| Requirement | Mapped to properties | Producer candidates enumerated | +|---|---|---| +| Accuracy = 95% rank accuracy | every family with rank error, and every configuration of it that meets 95% rank accuracy | (KLL, `k = 100`), (KLL, `k = 200`), … | +| Accuracy = KLL's `k ≥ 100` | KLL with `k ≥ 100` | (KLL, `k = 100`), (KLL, `k = 200`), … | -/// One physical alternative of a logical candidate, from Stage 2. -pub struct PhysicalCandidate { - pub logical: (MemoGroupId, usize), // which logical candidate - pub properties: PhysicalProperties, // p_v^phys - pub input_requirements: Vec, // r^phys_{c,i} +```rust +pub enum AccuracyRequirement { + /// Stated on the result: a metric, bound and failure probability. The + /// legal properties are derived through `size_params` / `local_guarantee`. + Target(AccuracyTarget), + /// Stated on the properties directly, e.g. KLL with k ≥ 100. + Parameters { family: SketchAlgorithm, at_least: SketchParams }, } - -/// (u, c, i): producer u may serve input i of consumer c, when α^log and α^phys hold. -pub struct Edge { pub producer: CandidateId, pub consumer: CandidateId, pub input: usize } ``` +What goes into a property requirement affects only the efficiency of enumeration, not correctness: +- **Some requirements already hold by the query semantics.** The expression and its producer groups already fix the input data and its fields, so no requirement needs to restate them. +- **It depends on how rules generate the candidate space.** A rule that only generates legal candidates needs no requirement for that dimension; a rule that generates every value needs the requirement to filter them. +- **A requirement prunes; it does not decide the plan.** A tighter requirement removes candidates earlier. Each query's composed error is still checked at its root in Stage 3 (row (K)), so a requirement that is too loose costs enumeration time, not a wrong plan. + ### Extensibility **Add a value to a property dimension**, e.g. a new summary family or `Medium::Disk`: @@ -289,19 +333,19 @@ pub struct Edge { pub producer: CandidateId, pub consumer: CandidateId, pub inpu 4. Add any new invalid combinations to `new`. **Add a property dimension**, e.g. a compression codec or a partitioning: -1. Decide whether it is logical (it changes the result's values or error) or physical (it changes only when, where or how long the result is computed). Add a field to `LogicalProperties` or `PhysicalProperties`. -2. If consumers can constrain it, add an optional field to the matching requirement struct and one check to `satisfies` or `satisfies_physical`. +1. Decide whether it is logical (it changes the result's values or error) or physical (it changes only when, where or how long the result is computed). Add a field (a new enum) to `LogicalProperties` or `PhysicalProperties`. +2. If consumers can constrain it, add an optional field to `PropertyRequirement` and one check to `satisfies`. 3. Add its invalid combinations with the other dimensions to `new`. 4. Add a level to the decision tree, with its prune rule, and place it by the considerations in "Order of decisions". 5. Add its cost terms. **Add a requirement:** -1. If it constrains one edge (a consumer input), add an optional field to `LogicalRequirement` or `PhysicalRequirement`, set it where the consumer is created, and check it in `satisfies`. +1. If it constrains one edge (a consumer input), add an optional field to `PropertyRequirement`, set it where the consumer's group is created, and check it in `satisfies`. 2. If it constrains a whole query or the whole plan (accuracy, latency, budgets), add it to the query's or workload's requirements and a row to Stage 3. Do not split it into per-edge requirements: that fixes each input's share in advance. 3. If it comes from the deployment, prune the values it rules out during candidate generation. **Add a rule:** 1. A rule matches over groups and their candidates' expressions, never over a reader's private operators. -2. It adds candidates $(e, p^{\log})$ to an existing group, or adds a new group with its candidates. It never copies a reader: every existing consumer of the group sees the new candidate through edges. +2. It adds candidates to an existing group, or adds a new group with its expression, input requirements and candidates. It never copies a reader: every existing consumer of the group sees the new candidate through edges. 3. A candidate whose result has the same meaning as an existing group is added to that group, so hash-consing keeps sharing. -4. Each new candidate states its input requirements. +4. Each new group states its input requirements. From 32bb7cccc9273e20f5a80f22289d6e0ba68ddfb1 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 19:08:22 +0000 Subject: [PATCH 03/67] docs(concepts): align memo group, candidate, property and requirement definitions with the code and #1200 Co-Authored-By: Claude Opus 5.5 --- .../memo-groups-properties-requirements.md | 275 +++++++++++------- 1 file changed, 165 insertions(+), 110 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 37623e34c..6fb8cd85d 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -1,7 +1,7 @@ # Memo groups, properties and requirements This page defines the memo's main data structures: groups, candidates, properties and requirements. It also lists every property and requirement, and gives the decision tree Stage 1 and Stage 2 walk to enumerate candidates. The symbols follow the planner formulation: -- a group has an expression $e_G$ over its producer groups, and a candidate $v$ of it has properties $p_v$; +- a candidate is $v = (e_v, p_v)$; - its logical properties are $p^{\log}$ and its physical properties are $p^{\mathrm{phys}}$; - a consumer input's requirements are $r^{\log}_{c,i}$ and $r^{\mathrm{phys}}_{c,i}$; - whether a producer satisfies them is $\alpha^{\log}$ and $\alpha^{\mathrm{phys}}$. @@ -36,19 +36,19 @@ The ε values are illustrative. A property is a value a candidate has. Every property is one dimension of a product: a struct with one field per dimension, never a flat enum that lists the combinations. Invalid combinations are rejected by the struct's checked constructor. -| Dimension | Values | Logical or physical | Fixed by | -|---|---|---|---| -| Which family to use (operator) | `Exact` (exact accumulator kinds: Sum, Count, Min, Max, Increase, Rate, IRate), or `Summary { family }` with family in KLL, DDSketch, CMS, CountSketch, CMS with heap, CountSketch with heap, UnivMon, HLL, KMV, the Hydra variants | logical | enumerated | -| Parameter configuration $\theta$ | per family, from a predefined list: KLL `k`; DDSketch `alpha`; CMS / CountSketch `width × depth`; CMS / CountSketch with heap `width × depth × heap_size`; UnivMon `heap_size × rows × cols × layers`; HLL `precision`; KMV `k` | logical | enumerated | -| Error bound $(\varepsilon_v, \delta_v)$ | computed on demand from $\theta$ and the statistic, in the family's accuracy metric (below); exact is $(0, 0)$ | logical | derived from $\theta$, not stored or chosen | -| Window configuration $\omega$ | none; or tumbling windows of width $w$ and count $n$, with $w$ from the gcd closure of the readers' ranges and steps. Later: sliding windows by merge or by subtraction | logical | enumerated | -| Timing $\tau$ | `IngestionTime`, `QueryTime` | physical | enumerated | -| Lifetime $\ell$ | `None`, `OneEvaluation(Medium)`, `AcrossEvaluations(Medium)`, with `Medium` in `Memory`, `Disk` | physical | enumerated | -| Schedule $\sigma$ | only when $\ell \ne$ None. `Periodic { interval, phase }` for `OneEvaluation`; `TumblingWindowGrid { phase }` for `AcrossEvaluations` | physical | enumerated from the readers' schedules | +| Dimension | Values | Logical or physical | Fixed by | Code | +|---|---|---|---|---| +| Which family to use (operator) | `Exact` (exact accumulator kinds: Sum, Count, Min, Max, Increase, Rate, IRate), or `Summary { family }` with family in KLL, DDSketch, CMS, CountSketch, CMS with heap, CountSketch with heap, UnivMon, HLL, KMV, the Hydra variants | logical | enumerated | `FieldDataType::{ExactAggregate, Sketch}` in the candidate's operator ([`schema/mod.rs`](../../../crates/types/src/ir/schema/mod.rs)); `SketchAlgorithm` | +| Parameter configuration $\theta$ | per family, from a predefined list: KLL `k`; DDSketch `alpha`; CMS / CountSketch `width × depth`; CMS / CountSketch with heap `width × depth × heap_size`; UnivMon `heap_size × rows × cols × layers`; HLL `precision`; KMV `k` | logical | enumerated | `SketchParams` inside `SketchKind` ([`sketch.rs`](../../../crates/types/src/ir/schema/sketch.rs)), sized by `size_params` per `accuracy_budget_parts` | +| Error bound $(\varepsilon_v, \delta_v)$ | computed on demand from $\theta$ and the statistic, in the family's accuracy metric (below); exact is $(0, 0)$ | logical | derived from $\theta$, not stored or chosen | `local_guarantee` → `AccuracyGuarantee` ([`estimators`](../../../crates/accuracy/src/estimators/mod.rs)) | +| Window configuration $\omega$ | none; or tumbling windows of width $w$ and count $n$, with $w$ from the gcd closure of the readers' ranges and steps. Later: sliding windows by merge or by subtraction | logical | enumerated | `SummaryTumblingWindowChain` arguments, added by Pass 2 ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs)) | +| Timing $\tau$ | `IngestionTime`, `QueryTime` | physical | enumerated | `ExecutionTiming` ([`execution_data_state.rs`](../../../crates/types/src/ir/properties/execution_data_state.rs)); `PhysicalCandidate::timing` | +| Lifetime $\ell$ | `None`, `OneEvaluation(Medium)`, `AcrossEvaluations(Medium)`, with `Medium` in `Memory`, `Disk` | physical | enumerated | `Materialization` on main; `Lifetime`, `Medium` in #1214 ([`physical/`](../../../crates/types/src/physical)) | +| Schedule $\sigma$ | only when $\ell \ne$ None. `Periodic { interval, phase }` for `OneEvaluation`; `TumblingWindowGrid { phase }` for `AcrossEvaluations` | physical | enumerated from the readers' schedules | not a property: schedule groups `z[m, G]` in the MILP ([`cost_terms.rs`](../../../crates/plan-selection/src/milp/cost_terms.rs)); a property in #1200 PR 2 | A candidate is materialized when $\ell \ne$ None. This is derived, not stored. -Invalid combinations, rejected by `new`: +Invalid combinations, rejected by `PhysicalProperties::new` (the list in #1200; #1214 implements the first, the others come with the schedule in #1200 PR 2): | Combination | Why | |---|---| @@ -56,6 +56,8 @@ Invalid combinations, rejected by `new`: | $\ell$ = None with a schedule | where an unmaterialized result runs is its reader's execution context, not a property | | $\ell \ne$ None without a schedule | a materialized result must say on which times or grid it is delivered | | OneEvaluation with a grid, AcrossEvaluations with a periodic schedule | each lifetime has one kind of schedule | +| a phase at or above the grid width or interval, or interval 0 | the phase is taken modulo the width or interval | +| IngestionTime with a grid phase ≠ 0 | ingestion-time grids are anchored at the epoch today: the plan carries no origin | Valid physical states: (QueryTime, None), (IngestionTime, None), (QueryTime, OneEvaluation(m)), (QueryTime, AcrossEvaluations(m)), (IngestionTime, AcrossEvaluations(m)), for each medium $m$. @@ -124,14 +126,14 @@ flowchart TD The levels, top to bottom: -| Level | Dimension | Stage | Values offered | Pruned when | -|---|---|---|---|---| -| 1 | family (operator) | Stage 1, Pass 1 and Pass 2 rules | each way to compute the group | the executor cannot build the family; no consumer's family requirement accepts it | -| 2 | $\theta$ | Stage 1, Pass 1 | the family's predefined list | $\varepsilon_v$ alone exceeds every reading query's target | -| 3 | window configuration $\omega$ | Stage 1, Pass 2 | tumbling widths from the gcd closure of readers' ranges and steps | no reader's range and step is a multiple of the width | -| 4 | timing $\tau$ | Stage 2 | QueryTime; IngestionTime | the source has no ingestion pipeline | -| 5 | lifetime $\ell$ | Stage 2 | None, OneEvaluation(m), AcrossEvaluations(m) | invalid with $\tau$; the deployment keeps no query-time results across evaluations | -| 6 | schedule $\sigma$ | Stage 2 | the gcd closure $\Sigma$ of the readers' schedules | only offered when $\ell \ne$ None | +| Level | Dimension | Stage | Values offered | Pruned when | Code | +|---|---|---|---|---|---| +| 1 | family (operator) | Stage 1, Pass 1 and Pass 2 rules | each way to compute the group | the executor cannot build the family; no consumer's family requirement accepts it | Pass 1 ([`per_computation_candidates.rs`](../../../crates/logical-asap-optimization/src/pass1/per_computation_candidates.rs)); Pass 2 rules ([`rules/`](../../../crates/logical-asap-optimization/src/pass2/memo/rules)) | +| 2 | $\theta$ | Stage 1, Pass 1 | the family's predefined list | $\varepsilon_v$ alone exceeds every reading query's target | Pass 1 sizing per `accuracy_budget_parts` ([`memo_population.rs`](../../../crates/logical-asap-optimization/src/pass1/memo_population.rs)); a predefined list in #1203 PR 2 | +| 3 | window configuration $\omega$ | Stage 1, Pass 2 | tumbling widths from the gcd closure of readers' ranges and steps | no reader's range and step is a multiple of the width | Pass 2 ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs), [`state_groups.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/state_groups.rs)) | +| 4 | timing $\tau$ | Stage 2 | QueryTime; IngestionTime | the source has no ingestion pipeline | Stage 2 `enumerate_candidate_configurations` ([`configurations.rs`](../../../crates/physical-asap-optimization/src/catalog/configurations.rs)) | +| 5 | lifetime $\ell$ | Stage 2 | None, OneEvaluation(m), AcrossEvaluations(m) | invalid with $\tau$; the deployment keeps no query-time results across evaluations | same function, with `DeploymentCapabilities::query_time_results_retained` | +| 6 | schedule $\sigma$ | Stage 2 | the gcd closure $\Sigma$ of the readers' schedules | only offered when $\ell \ne$ None | Stage 3 MILP today; Stage 2 in #1200 PR 2 | What the tree does not decide: - **Which producer each consumer input reads.** That is the edge choice $y_{u,c,i}$ in Stage 3. Two consumers of one group may read different leaves, e.g. A reads `k = 200` and B reads `k = 345`. @@ -179,129 +181,190 @@ A memo group holds three things: - **Eligibility:** for each input, the requirement a producer must meet. - **Candidates:** the ways to compute it. +In the code, these three parts are stored in two places: the group itself ([`MemoGroup`](../../../crates/types/src/stages/memo.rs)) holds the query semantics, and the memo ([`Memo`](../../../crates/types/src/stages/candidates.rs)) holds each group's candidates by group index. Eligibility is not stored yet. + ```rust -/// The memo: every group, and each query's root over groups. +// crates/types/src/stages/memo.rs +pub struct MemoGroup { + // Query semantics. + /// Its logical node as the frontends see it; its children are other + /// groups' nodes. + pub logical_node: Rc, + /// Its child groups (the producer groups G_i), in the node's child order. + pub child_groups: Vec, + // Derived, kept for traversal. + pub parents: Vec, + pub queries: Vec, +} + +// crates/types/src/stages/candidates.rs pub struct Memo { pub groups: Vec, pub query_roots: Vec>, -} - -pub struct MemoGroup { - // Query semantics. - /// The logical operator, from the existing IR. Its children are the - /// producer groups G_i it reads, in order (`expression.children()`), which - /// fix the input data and schema. - pub expression: Operator, - // Eligibility. - /// r_i: one requirement per producer group, by input index i. - pub input_requirements: Vec, - // Candidates. - /// The ways to compute this group; x_v in the MILP. - pub candidates: Vec, + // Candidates: one list per group, by group index; x_v in the MILP. + pub candidates: Vec>, } ``` -The expression is the existing IR's [`Operator`](../../../crates/types/src/ir/operator/node.rs), with `C = MemoGroupId`. It wraps [`NonASAPOp`](../../../crates/types/src/ir/operator/non_asap.rs) or [`ASAPOp`](../../../crates/types/src/ir/operator/asap.rs), and each variant carries its own arguments and its inputs. For example, `Scan` carries the source, predicates and schema, and `Filter` carries a predicate and its child. No new identity type is needed. +| Part | Code | Target | +|---|---|---| +| Expression | `MemoGroup::logical_node`, an [`OperatorNode`](../../../crates/types/src/ir/operator/node.rs) whose [`Operator`](../../../crates/types/src/ir/operator/node.rs) is a `NonASAPOp` or `ASAPOp` variant with its own arguments | unchanged: the existing IR, no new identity type | +| Producer groups $G_i$ | `MemoGroup::child_groups` | unchanged | +| Eligibility $r_{G,i}$ | not stored: each candidate's private node chain fixes its producer | one logical requirement (#1203) and one physical requirement (#1200) per consumer input; see "Property requirement definition" | +| Candidates | `Memo::candidates[g]` | unchanged place; each candidate restructured (#1203) | +| Consumers | `MemoGroup::parents` (derived) | unchanged: derived from the producer groups | -Hash-consing finds or inserts a group by its expression: the operator, its arguments and its input group ids. For example, the KLL state over one source and query window is one group for `k = 200` and `k = 345`; these are two candidates of that group. A different source, filter or query window gives a different group. Properties are not part of the expression: the family, sketch parameters, window layout, timing, lifetime and schedule belong to candidates. A requested query window belongs to the expression; the internal window layout used to compute it does not. +Hash-consing finds or inserts a group by its logical node: the operator, its arguments and its child groups. Stage 0 does this today (`hash_consed` in [`stage0.rs`](../../../crates/logical-asap-optimization/src/stage0.rs)); #1203 makes Pass 1 look up existing groups the same way. A different source, filter or query window gives a different group. A requested query window belongs to the logical node; the internal window layout used to compute it does not. -`MemoGroupId` is the handle of an interned group. Looking up an existing expression returns that group's id instead of adding a second group. +The group's logical node is the frontend's operator, e.g. `quantile_over_time(0.95, latency[1m])`. It names no family and no sketch parameters: those are chosen by the candidates. A candidate's own operator (below) is an `ASAPOp` such as `SummaryAgg`, whose `FieldDataType::Sketch(SketchKind, …)` carries the family and $\theta$. Today the KLL state is not its own group: it is a private node inside each reader's candidate. #1203 makes every intermediate result its own group, so the KLL state over `latency[1m]` becomes a group, and `k = 200` and `k = 345` become two candidates of it. Its identity drops the sketch parameters, $\delta$ and the tumbling-window layout (#1203, decision 1). ### Candidate data structure definition +A candidate is one way to compute its group: an operator over producer groups, plus its properties. + ```rust -/// v: one way to compute its group. The group's expression says what is -/// computed; the candidate's properties say how. -pub struct Candidate { - /// One value per property dimension. - pub properties: Properties, +// crates/types/src/stages/candidates.rs (code today) +pub struct CandidateForMemoGroup { + pub fused_groups: Vec, + /// The implementation's operator, and its private nodes down to + /// `input_groups`. Family and θ are inside it (FieldDataType). + pub root_node: Rc, + pub input_groups: Vec, + pub origin: OriginOfCandidateForMemoGroup, + pub accuracy_budget_parts: usize, } ``` -Stage 1 decides the logical dimensions of each candidate, and Stage 2 expands each one into its physical alternatives: - ```rust -pub struct Properties { - pub logical: LogicalProperties, // Stage 1 - pub physical: PhysicalProperties, // Stage 2 +// Target (#1203): e_v and p_v^log as separate fields, one operator, no private nodes. +pub struct CandidateForMemoGroup { + pub expression: CandidateExpression, // e_v + pub logical_properties: LogicalProperties, // p_v^log + pub input_requirements: Vec, // r^log_{c,i}, one per input group + pub origin: OriginOfCandidateForMemoGroup, + pub fused_groups: Vec, } +pub struct CandidateExpression { pub operator: Operator } +``` -/// (u, c, i): producer u may serve input i of consumer c when u's properties -/// satisfy c's group's input_requirements[i]. -pub struct Edge { pub producer: CandidateId, pub consumer: CandidateId, pub input: usize } +Stage 2 expands each operator node of a logical candidate into physical candidates, one per valid physical property value: + +```rust +// crates/types/src/stages/physical/memo_candidates.rs, after #1200 PR 1 (#1214) +pub struct PhysicalCandidate { + pub operator_node: OperatorNodeOfCandidateForMemoGroupId, + pub properties: PhysicalProperties, // p_v^phys; was timing + materialization + /// Per input, the candidates that may produce it (filtered by α^phys). + pub producer_candidates_per_input: Vec, + pub may_produce_query_output: bool, + pub may_keep_across_evaluations_per_sole_reader_query: Vec, +} ``` +| Formulation | Code today | Target | +|---|---|---| +| $e_v$ | `root_node` + `input_groups` (may hold private nodes) | `expression: Operator` (#1203) | +| $p^{\log}_v$ | inside `root_node`'s `FieldDataType`; sized per `accuracy_budget_parts` | `logical_properties` (#1203) | +| $p^{\mathrm{phys}}_v$ | `PhysicalCandidate::{timing, materialization}` on main | `PhysicalCandidate::properties` (#1200 PR 1, #1214) | +| edges $(u, c, i)$ | `producer_candidates_per_input` (sets, no edge ids) | `LogicalEdge`, `PhysicalEdge` (#1201) | + ### Property definition -A property is a struct with one field per dimension (P1, P2, …), each its own enum: +Each property is a struct with one field per dimension, each its own enum. + +**Logical properties** (#1203 target; today inside `root_node`): ```rust -/// p_v^log. pub struct LogicalProperties { - family: Family, // which sketch, or exact - parameter_configuration: Option, // θ; None for exact - window_layout: Option, // ω_v + representation: Representation, // family and θ + window_layout: Option, // ω_v +} +/// Reuses the code's checked types: `SketchKind::new` already rejects +/// parameters of another algorithm. +pub enum Representation { + Exact(ExactKind), // crates/types/src/ir/schema/sketch.rs + Summary(SketchKind), // SketchKind { algorithm, params } } -pub enum Family { Exact(ExactKind), Sketch(SketchAlgorithm) } -pub enum WindowLayout { Tumbling { width_ms: u64, count: u32 } } // sliding variants later +pub enum WindowLayout { Tumbling { width_ms: u64, count: u32 } } // sliding variants: #1206 +``` + +Today these values are the `FieldDataType` of the candidate's summary operator: `ExactAggregate(ExactKind, ExactParams)` or `Sketch(SketchKind, GroupingStrategy)` ([`schema/mod.rs`](../../../crates/types/src/ir/schema/mod.rs)). The tumbling-window layout is the `SummaryTumblingWindowChain` operator's arguments. + +`error_bound` is derived when needed, not stored in `LogicalProperties`. Reuse [`estimators::local_guarantee`](../../../crates/accuracy/src/estimators/mod.rs) for parameters → guarantee and [`estimators::size_params`](../../../crates/accuracy/src/estimators/mod.rs) for target → parameters. These existing crate-private functions dispatch to the family's accuracy and sizing implementations. [`sizing::default_size_params`](../../../crates/accuracy/src/sizing.rs) already delegates to `size_params`. If another crate needs direct access, expose the existing implementation through the accuracy crate rather than duplicate its formulas. (#1203 currently proposes storing `error_bound`; this page derives it instead.) + +```rust +// crates/accuracy/src/estimators/mod.rs (code today) +pub(crate) fn local_guarantee(family: &FieldDataType, query: &SketchStatistic) -> Option; +pub(crate) fn size_params(kind: SketchAlgorithm, intent: &AggIntent, eps: f64, delta: f64) -> SketchParams; +``` + +Deriving a guarantee also needs the statistic being read: one family's parameters can give different guarantees for different statistics. Exact values yield an exact guarantee; an unsupported statistic or missing error model yields `None`. Target → parameters is a sizing operation with discrete values and family limits, not an exact mathematical inverse; validate the derived guarantee against the target rather than assume the target is achievable. -/// p_v^phys. `new` rejects the invalid combinations. +**Physical properties** (#1200). PR 1 (#1214) adds timing and lifetime; PR 2 adds the schedule. + +```rust +// crates/types/src/ir/properties/execution_data_state.rs (code today) +pub enum ExecutionTiming { IngestionTime, QueryTime } + +// crates/types/src/physical/properties.rs (#1214) +pub enum Medium { Memory, Disk } +pub enum Lifetime { None, OneEvaluation(Medium), AcrossEvaluations(Medium) } pub struct PhysicalProperties { timing: ExecutionTiming, // τ lifetime: Lifetime, // ℓ - schedule: Option, // σ, only when ℓ ≠ None + // schedule: Option, // σ, #1200 PR 2 } -pub enum ExecutionTiming { IngestionTime, QueryTime } -pub enum Lifetime { None, OneEvaluation(Medium), AcrossEvaluations(Medium) } -pub enum Medium { Memory, Disk } -pub enum Schedule { - Periodic { interval_ms: u64, phase_ms: u64 }, - TumblingWindowGrid { phase_ms: u64 }, +impl PhysicalProperties { + pub fn new(timing: ExecutionTiming, lifetime: Lifetime) -> Result; + pub fn timing(self) -> ExecutionTiming; + pub fn lifetime(self) -> Lifetime; + pub fn is_materialized(self) -> bool; // ℓ ≠ None } -// Reuse the existing family-specific conversions in asap_accuracy. -// Parameter configuration → error bound (including metric and failure probability). -// `family` describes the exact value or sketch, including its parameters; -// `query` identifies the statistic read from it. -pub(crate) fn local_guarantee( - family: &FieldDataType, - query: &SketchStatistic, -) -> Option; - -// Error target → parameter configuration for the chosen family and intent. -pub(crate) fn size_params( - kind: SketchAlgorithm, - intent: &AggIntent, - eps: f64, - delta: f64, -) -> SketchParams; +// #1200 PR 2 +pub enum Schedule { + Periodic { interval_ms: u64, phase_ms: u64 }, // OneEvaluation + TumblingWindowGrid { phase_ms: u64 }, // AcrossEvaluations +} ``` -`new` checks that `parameter_configuration` belongs to `family`, as [`SketchKind::new`](../../../crates/types/src/ir/schema/sketch.rs) does today, and reuses the existing `SketchParams` type. - -`error_bound` is derived when needed, not stored in `LogicalProperties`. Reuse [`estimators::local_guarantee`](../../../crates/accuracy/src/estimators/mod.rs) for parameters → guarantee and [`estimators::size_params`](../../../crates/accuracy/src/estimators/mod.rs) for target → parameters. These existing crate-private functions dispatch to the family's accuracy and sizing implementations; the declarations above document those interfaces, not new implementations. [`sizing::default_size_params`](../../../crates/accuracy/src/sizing.rs) already delegates to `size_params`. If another crate needs direct access, expose the existing implementation through the accuracy crate rather than duplicate its formulas. - -Deriving a guarantee also needs the statistic being read: one family's parameters can give different guarantees for different statistics. Exact values yield an exact guarantee; an unsupported statistic or missing error model yields `None`. Target → parameters is a sizing operation with discrete values and family limits, not an exact mathematical inverse; validate the derived guarantee against the target rather than assume the target is achievable. +On main, timing and lifetime are still folded into one enum, [`Materialization`](../../../crates/types/src/physical/materialization.rs) (`NotMaterialized`, `MatAtQuery`, `MatAtQueryKeptAcrossEvaluations`, `MatAtIngestion`), next to `PhysicalCandidate::timing`. #1214 deletes it. The schedule is chosen in the MILP today (schedule groups `z[m, G]`, [`cost_terms.rs`](../../../crates/plan-selection/src/milp/cost_terms.rs)); #1200 PR 2 makes it a property. ### Property requirement definition -A property requirement is also a struct with one field per dimension (R1, R2, …), each its own enum. `None` accepts any value. +A requirement is stated per consumer input $(c, i)$, in two parts that follow the two property parts. Each is a struct with one field per dimension. ```rust -/// r_i: what input i's producer must satisfy. -pub struct PropertyRequirement { - pub accuracy: Option, +/// r^log_{c,i} (#1203): one optional constraint per dimension; None accepts any value. +pub struct LogicalRequirement { pub family: Option, - pub window: Option, // the reader's range and step - pub reader_timing: Option, - pub reads_from: Option, // Same or Later - pub reader_schedule: Option, // (I_c, φ_c) of a periodic reader + pub window: Option, // the reader's range and step + pub error_cap: Option, // validity of row (K)'s linear bound } - -/// The set of legal properties for an input: legal_properties(r) = f(r). -pub fn satisfies(producer: &Properties, requirement: &PropertyRequirement) -> bool; +/// α^log +pub fn satisfies(producer: &LogicalProperties, requirement: &LogicalRequirement) -> bool; + +/// r^phys_{c,i} (#1200): when the consumer reads the input, and on which times. +/// `new` rejects (IngestionTime, Later): ingestion has no evaluations. +pub struct PhysicalRequirement { + reader_timing: ExecutionTiming, + reads_from: Evaluation, // which evaluation's output the reader reads + reader_schedule: Option, // (I_c, φ_c); None for a non-periodic reader +} +pub enum Evaluation { Same, Later } +/// α^phys +pub fn satisfies_physical(producer: PhysicalProperties, requirement: PhysicalRequirement) -> bool; ``` +| Requirement | Code today | Target | +|---|---|---| +| $r^{\log}_{c,i}$ | not represented | `LogicalRequirement` (#1203) | +| $\alpha^{\log}$ | not computed | `satisfies` (#1203) | +| $r^{\mathrm{phys}}_{c,i}$ | the consumer's timing | `PhysicalRequirement` (#1200 PR 2) | +| $\alpha^{\mathrm{phys}}$ | [`compatible_timing`](../../../crates/physical-asap-optimization/src/catalog/configurations.rs) on main; `PhysicalProperties::readable_at(reader_timing)` in #1214 | `satisfies_physical` (#1200 PR 2) | +| accuracy | [`AccuracyTarget`](../../../crates/types/src/ir/properties/accuracy.rs) at the query root, split into equal shares by `accuracy_budget_parts` | `AccuracyTarget` at the query root, checked by row (K) (#1204) | +| deployment capability | [`DeploymentCapabilities`](../../../crates/logical-asap-optimization/src/deployment_capabilities.rs): `supported_summaries`, `raw_data_retained`, `query_time_results_retained` | unchanged; prunes during generation | + The set of legal properties is a function of the requirement, $\text{legal}(r) = f(r)$. One requirement can be written at different levels. For example, the same accuracy requirement can be stated two ways: | Requirement | Mapped to properties | Producer candidates enumerated | @@ -309,15 +372,7 @@ The set of legal properties is a function of the requirement, $\text{legal}(r) = | Accuracy = 95% rank accuracy | every family with rank error, and every configuration of it that meets 95% rank accuracy | (KLL, `k = 100`), (KLL, `k = 200`), … | | Accuracy = KLL's `k ≥ 100` | KLL with `k ≥ 100` | (KLL, `k = 100`), (KLL, `k = 200`), … | -```rust -pub enum AccuracyRequirement { - /// Stated on the result: a metric, bound and failure probability. The - /// legal properties are derived through `size_params` / `local_guarantee`. - Target(AccuracyTarget), - /// Stated on the properties directly, e.g. KLL with k ≥ 100. - Parameters { family: SketchAlgorithm, at_least: SketchParams }, -} -``` +In the code, the first form is the query's `AccuracyTarget`; Pass 1 maps it to parameters with `size_params`. The second form is what a sized candidate's `SketchParams` fixes today. #1203 keeps the first form at the query root and adds only `error_cap` per edge. What goes into a property requirement affects only the efficiency of enumeration, not correctness: - **Some requirements already hold by the query semantics.** The expression and its producer groups already fix the input data and its fields, so no requirement needs to restate them. @@ -334,18 +389,18 @@ What goes into a property requirement affects only the efficiency of enumeration **Add a property dimension**, e.g. a compression codec or a partitioning: 1. Decide whether it is logical (it changes the result's values or error) or physical (it changes only when, where or how long the result is computed). Add a field (a new enum) to `LogicalProperties` or `PhysicalProperties`. -2. If consumers can constrain it, add an optional field to `PropertyRequirement` and one check to `satisfies`. +2. If consumers can constrain it, add an optional field to `LogicalRequirement` or `PhysicalRequirement` and one check to `satisfies` or `satisfies_physical`. 3. Add its invalid combinations with the other dimensions to `new`. 4. Add a level to the decision tree, with its prune rule, and place it by the considerations in "Order of decisions". 5. Add its cost terms. **Add a requirement:** -1. If it constrains one edge (a consumer input), add an optional field to `PropertyRequirement`, set it where the consumer's group is created, and check it in `satisfies`. +1. If it constrains one edge (a consumer input), add a field to `LogicalRequirement` or `PhysicalRequirement`, set it where the consumer is created, and check it in `satisfies` or `satisfies_physical`. 2. If it constrains a whole query or the whole plan (accuracy, latency, budgets), add it to the query's or workload's requirements and a row to Stage 3. Do not split it into per-edge requirements: that fixes each input's share in advance. 3. If it comes from the deployment, prune the values it rules out during candidate generation. **Add a rule:** 1. A rule matches over groups and their candidates' expressions, never over a reader's private operators. -2. It adds candidates to an existing group, or adds a new group with its expression, input requirements and candidates. It never copies a reader: every existing consumer of the group sees the new candidate through edges. +2. It adds candidates to an existing group, or adds a new group with its candidates. It never copies a reader: every existing consumer of the group sees the new candidate through edges. 3. A candidate whose result has the same meaning as an existing group is added to that group, so hash-consing keeps sharing. -4. Each new group states its input requirements. +4. Each new candidate states its input requirements. From 67dd6122c0ae9290a455779af072a2ac0a6b5622 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 19:11:14 +0000 Subject: [PATCH 04/67] docs(concepts): data structures and interfaces per stage, from the #1207 formulation Co-Authored-By: Claude Opus 5.5 --- .../memo-groups-properties-requirements.md | 229 ++++++++++++++++++ 1 file changed, 229 insertions(+) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 6fb8cd85d..120fc501c 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -404,3 +404,232 @@ What goes into a property requirement affects only the efficiency of enumeration 2. It adds candidates to an existing group, or adds a new group with its candidates. It never copies a reader: every existing consumer of the group sees the new candidate through edges. 3. A candidate whose result has the same meaning as an existing group is added to that group, so hash-consing keeps sharing. 4. Each new candidate states its input requirements. + +## 4. Data structures and interfaces per stage + +This section gives the target interface of each stage after the formulation refactor (#1207). Each stage's boundary type carries the notation that stage defines, and no stage reads a notation a later stage defines: logical stages carry no physical properties. + +| Stage | Input | Output (boundary type) | Notation it defines | Sub-issue | +|---|---|---|---|---| +| Stage 0: frontends, hash-consing | queries | `DagOfLogicalMemoGroups` | $G$, query roots | — | +| Stage 1, Pass 1 | `DagOfLogicalMemoGroups` | `Memo` | $v = (e_v, p^{\log}_v)$, $r^{\log}_{c,i}$, $\theta$ list | #1203 | +| Stage 1, Pass 2 | `Memo` | `LogicalASAPMemoCandidates` | rule-added candidates; logical edges with $\alpha^{\log}$ | #1203, #1201 | +| Stage 2 | `LogicalASAPMemoCandidates` | `PhysicalASAPMemoCandidates` | $p^{\mathrm{phys}}_v = (\tau, \ell, \sigma)$, $r^{\mathrm{phys}}_{c,i}$, physical edges $\mathcal E$ with $\alpha^{\mathrm{phys}}$ | #1200, #1201 | +| Stage 3 | `PhysicalASAPMemoCandidates` | `DagOfSelectedCandidates` | contexts $\kappa$; cost $w_v$ (`asap-cost`); $x$, $y$, $z$, $t$, $h$; rows (A)–(K), (R), (L), (B) | #1204 | +| Realizer, export | `DagOfSelectedCandidates` | `PhysicalASAPDAG` | — | #1200 (export fields) | + +### Stage 0: frontends and hash-consing + +No change. One group per distinct logical sub-expression; the queries' roots point at groups. + +```rust +// crates/types/src/stages/memo.rs +pub struct MemoGroupId(pub u32); + +/// G: one logical result or one state family over specified input data. +pub struct MemoGroup { + pub logical_node: Rc, // expression; no physical fields (#1200) + pub child_groups: Vec, // producer groups G_i + pub parents: Vec, // derived: consumers' groups + pub queries: Vec, // derived: queries that reach it +} + +pub struct DagOfLogicalMemoGroups { + pub groups: Vec, + pub query_roots: Vec>, +} +``` + +### Stage 1, Pass 1: candidates of each group + +Pass 1 adds every way to compute each group as candidates. Every intermediate result is its own group (no private nodes), and a summary family's sizes are candidates of one state group. + +```rust +// crates/types/src/stages/candidates.rs +pub struct CandidateForMemoGroupId { pub group: MemoGroupId, pub index: usize } + +pub struct Memo { + pub groups: Vec, + pub query_roots: Vec>, + pub candidates: Vec>, // by group +} +impl Memo { + /// Hash-consing: the group with this expression, added if new. + pub fn find_or_add_group(&mut self, expression: Operator) -> MemoGroupId; + pub fn add_candidate(&mut self, group: MemoGroupId, candidate: CandidateForMemoGroup) -> CandidateForMemoGroupId; + pub fn candidate(&self, id: CandidateForMemoGroupId) -> &CandidateForMemoGroup; +} + +/// v = (e_v, p_v^log) +pub struct CandidateForMemoGroup { + pub expression: CandidateExpression, // e_v + pub logical_properties: LogicalProperties, // p_v^log + pub input_requirements: Vec, // r^log_{c,i}, one per input group + pub fused_groups: Vec, +} +/// e_v = op_v(G_{v,1}, …, G_{v,n}): one operator, its children are groups. +pub struct CandidateExpression { pub operator: Operator } + +pub struct LogicalProperties { + representation: Representation, // family and θ, from the predefined list + window_layout: Option, // ω_v +} +impl LogicalProperties { + pub fn new(representation: Representation, window_layout: Option) -> Result; + /// (ε_v, δ_v) for `statistic`, derived through `local_guarantee`. + pub fn error_bound(&self, statistic: &SketchStatistic) -> Option; +} +pub enum Representation { Exact(ExactKind), Summary(SketchKind) } +pub enum WindowLayout { Tumbling { width_ms: u64, count: u32 } } + +pub struct LogicalRequirement { + pub family: Option, + pub window: Option, // reader's range and step: a width must divide both + pub error_cap: Option, // validity of row (K)'s linear bound +} +/// α^log_{u,c,i} +pub fn satisfies(producer: &LogicalProperties, requirement: &LogicalRequirement) -> bool; +``` + +Deleted: `root_node` with private nodes, `input_groups` (the expression's children), `accuracy_budget_parts`, the `SharedBuild` and `SummaryResult` origins. + +### Stage 1, Pass 2: rules and logical edges + +Rules add candidates to existing groups, or add groups. A rule never copies a reader. After the last rule, Pass 2 builds the logical edges once. + +```rust +// crates/logical-asap-optimization/src/pass2 +/// A rule: reads the memo, adds candidates or groups. +fn apply(memo: &mut Memo) -> Result<(), Pass2RuleError>; + +// crates/types/src/stages/logical_memo_candidates.rs +/// (u, c, i) over logical candidates: u is a candidate of G_{c,i}. +pub struct LogicalEdge { + pub producer: CandidateForMemoGroupId, + pub consumer: CandidateForMemoGroupId, + pub input: usize, + pub satisfies_logical_requirement: bool, // α^log +} +pub struct LogicalASAPMemoCandidates { + pub memo: Memo, + pub edges: Vec, // grouped by (consumer, input) +} +``` + +### Stage 2: physical candidates and edges + +Stage 2 expands each logical candidate into one physical candidate per valid $(\tau, \ell, \sigma)$, gives each input its physical requirement, and turns each logical edge into the physical edges between the two candidates' physical alternatives. + +```rust +// crates/types/src/physical/properties.rs +pub enum ExecutionTiming { IngestionTime, QueryTime } +pub enum Medium { Memory, Disk } +pub enum Lifetime { None, OneEvaluation(Medium), AcrossEvaluations(Medium) } +pub enum Schedule { + Periodic { interval_ms: u64, phase_ms: u64 }, // OneEvaluation + TumblingWindowGrid { phase_ms: u64 }, // AcrossEvaluations +} +/// p_v^phys = (τ, ℓ, σ); `new` rejects the invalid combinations (Section 1). +pub struct PhysicalProperties { timing: ExecutionTiming, lifetime: Lifetime, schedule: Option } +impl PhysicalProperties { + pub fn new(timing: ExecutionTiming, lifetime: Lifetime, schedule: Option) -> Result; + pub fn timing(&self) -> ExecutionTiming; + pub fn lifetime(&self) -> Lifetime; + pub fn schedule(&self) -> Option<&Schedule>; + pub fn is_materialized(&self) -> bool; // ℓ ≠ None +} + +/// r^phys_{c,i}; `new` rejects (IngestionTime, Later). +pub struct PhysicalRequirement { + reader_timing: ExecutionTiming, + reads_from: Evaluation, + reader_schedule: Option, // (I_c, φ_c) +} +pub enum Evaluation { Same, Later } +/// α^phys_{u,c,i} +pub fn satisfies_physical(producer: &PhysicalProperties, requirement: &PhysicalRequirement) -> bool; + +// crates/types/src/stages/physical +pub struct PhysicalCandidateId(pub usize); +pub struct PhysicalEdgeId(pub usize); + +/// One physical alternative of a logical candidate. +pub struct PhysicalCandidate { + pub logical: CandidateForMemoGroupId, // e_v and p_v^log + pub properties: PhysicalProperties, // p_v^phys + pub input_requirements: Vec, // r^phys_{c,i} +} +/// (u, c, i) over physical candidates. +pub struct PhysicalEdge { + pub producer: PhysicalCandidateId, + pub consumer: PhysicalCandidateId, + pub input: usize, + pub satisfies_logical_requirement: bool, // α^log + pub satisfies_physical_requirement: bool, // α^phys +} +pub struct PhysicalASAPMemoCandidates { + pub logical: LogicalASAPMemoCandidates, + pub candidates: Vec, // C + pub query_roots: Vec>, + pub pruned_candidates: Vec>, + // edges stored per (consumer, input); see `edges()` +} +impl PhysicalASAPMemoCandidates { + pub fn edges(&self) -> impl Iterator; // E + pub fn admissible_edges_into(&self, consumer: PhysicalCandidateId, input: usize) -> &[PhysicalEdgeId]; // P_{c,i} + pub fn admissible_edges_out_of(&self, producer: PhysicalCandidateId) -> &[PhysicalEdgeId]; +} +``` + +Deleted: `Materialization`, `PhysicalCandidate::{timing, materialization, operator_node}`, `producer_candidates_per_input` as the interface, `compatible_timing`, `ReaderConstraint::{IncompatibleOutputs, AllowedOutputs}` (now $\alpha^{\mathrm{phys}}$ through $\sigma$), `may_keep_across_evaluations_per_sole_reader_query`. + +### Stage 3: the MILP + +Stage 3 reads the physical candidates and edges and selects the DAG. Variables and rows are named after the formulation. + +```rust +// crates/plan-selection/src/milp/model.rs +/// κ: where an ℓ = None candidate runs. Only those candidates are indexed by it. +/// `ContextId` indexes the workload's contexts; the id types inside are illustrative. +pub struct ContextId(pub usize); +pub enum Context { + QueryEvaluation(QueryId), // κ_q + IngestionPipeline(RefreshWindowId), // I_r + Materialized(PhysicalCandidateId), // κ_m: inside a materialized m's computation +} + +struct SelectionVariables { + x_materialized: Vec, // x_v, ℓ ≠ None + x_in_context: HashMap<(ContextId, PhysicalCandidateId), Variable>, // x_{κ,v}, ℓ = None + y: HashMap<(Option, PhysicalEdgeId), Variable>, // y_{u,c,i} on admissible edges + z: Vec, // z_{q,v} + t: Vec, // t_v, cyclic components only + h: Vec, // h_v, ℓ ≠ None + latency: LatencyVariables, // (L) +} +fn input_satisfaction_rows(..); // (A) Σ_u y_{u,c,i} = x_c +fn producer_selection_rows(..); // (B) y_{u,c,i} ≤ x_u +fn query_coverage_rows(..); // (C), (D) +fn acyclicity_rows(..); // (E) +fn utilization_rows(..); // (F) +fn reader_constraint_rows(..); // (J) +fn accuracy_rows(..); // (K) Σ λ_{q,u} ε_u a_{q,u} ≤ ε_q, Σ δ_u a_{q,u} ≤ δ_q +fn retention_rows(..); // (R) h_v ≥ need_{c,i} · y_{v,c,i} +fn latency_rows(..); // (L) +fn resource_budget_rows(..); // (B) budgets +// Objective: min Σ w_v x_v + Σ ρ_κ w_v x_{κ,v} + Σ π_v h_v + +// crates/types/src/stages/physical/graph.rs (unchanged) +pub struct SelectedEdge { pub producer: SelectedNodeId, pub consumer: SelectedNodeId, pub input: usize } +pub struct DagOfSelectedCandidates { + pub candidate_of_node: Vec, + pub edges: Vec, // decoded from y = 1 + pub node_of_output_reference: Vec, +} +``` + +(G) $y \le \alpha^{\log}$ and (H) $y \le \alpha^{\mathrm{phys}}$ are not rows: $y$ exists only on admissible edges. Deleted: `ResultSlot`, `slot_of`, the `e` binaries, schedule groups `z[m, G]`, `N[q]`, `A[q, parts]`, and `p[ctx, c]` for materialized candidates. + +### Realizer and export + +The realizer builds the plan from `DagOfSelectedCandidates`; each node's physical properties come from its `PhysicalCandidate`. The export writes timing and lifetime (and later the schedule) per node of `PhysicalASAPDAG`, changed together with the backend. From 31ec62b99d71a1c871d506a4bd3cdf0db2b715da Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 19:20:20 +0000 Subject: [PATCH 05/67] docs(concepts): keep the per-stage interfaces as section 3, drop the earlier modeling section Co-Authored-By: Claude Opus 5.5 --- .../memo-groups-properties-requirements.md | 250 +----------------- 1 file changed, 1 insertion(+), 249 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 120fc501c..7769b3b10 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -156,256 +156,8 @@ Alternative orders worth comparing: - **$\theta$ last among logical dimensions.** $\theta$ has the most values. Deciding $\omega$ before $\theta$ visits fewer nodes when many widths are pruned by readers' ranges. - **Requirement-driven (top-down) vs producer-driven (bottom-up).** Top-down enumeration starts from each consumer's requirement and offers only the values it accepts; bottom-up offers every value and lets $\alpha$ filter edges. Top-down visits fewer nodes but must union the requirements of all consumers of a group, so that sharing between consumers is not lost. -## 3. Modeling a memo group -A memo group stores its **producers**: all candidate ways to compute its result, before Stage 3 selects any. A consumer of $G$ is a candidate of another group whose expression reads $G$; the producer–consumer edges $(u, c, i)$ connect them. Here, `consumers(G)` is derived from those input references rather than stored in the group. - -> Side note: [CMU-DB's optd-original](https://github.com/cmu-db/optd-original) stores the list of children as input producers to a memo group; while storing consumers vs storing producers in the memo group structure to form the DAG are equivalent in terms of correctness and efficiency. - -Candidate enumeration efficiency depends on: - -- **Property and requirement fields:** which dimensions are represented, their legal values, which values are derived, and which constraints permit pruning or sharing (Section 1). -- **Decision-tree order:** when each dimension is decided and its requirements are checked, so invalid combinations can be pruned before later dimensions multiply them (Section 2). - -Groups found by hash-consing expose the same producer alternatives to all their readers. Stage 3 decides whether readers use the same producer candidate or different ones. **Reusing a computation alternative in the memo is distinct from reusing its computed result at runtime. Selecting the same candidate for multiple readers does not by itself mean computing it only once.** - -- **Materialized candidate** ($\ell \ne$ None): its result is retained. Readers connected to the same materialized producer can reuse that result instead of computing it separately, subject to its lifetime and schedule. -- **Unmaterialized candidate** ($\ell =$ None): no independent shared result is retained. The computation runs as part of each reader's execution, so multiple readers selecting the same candidate may each execute it separately. - -For example, if p50 and p99 select the same KLL candidate, a shared materialized producer builds and retains one KLL state for both readouts within the applicable lifetime and schedule. With an unmaterialized candidate, the p50 execution can build its own KLL state and the p99 execution another, even though both builds use the same candidate configuration. - -### Memo group data structure - -A memo group holds three things: -- **Query semantics:** its expression, and the producer groups the expression reads. -- **Eligibility:** for each input, the requirement a producer must meet. -- **Candidates:** the ways to compute it. - -In the code, these three parts are stored in two places: the group itself ([`MemoGroup`](../../../crates/types/src/stages/memo.rs)) holds the query semantics, and the memo ([`Memo`](../../../crates/types/src/stages/candidates.rs)) holds each group's candidates by group index. Eligibility is not stored yet. - -```rust -// crates/types/src/stages/memo.rs -pub struct MemoGroup { - // Query semantics. - /// Its logical node as the frontends see it; its children are other - /// groups' nodes. - pub logical_node: Rc, - /// Its child groups (the producer groups G_i), in the node's child order. - pub child_groups: Vec, - // Derived, kept for traversal. - pub parents: Vec, - pub queries: Vec, -} - -// crates/types/src/stages/candidates.rs -pub struct Memo { - pub groups: Vec, - pub query_roots: Vec>, - // Candidates: one list per group, by group index; x_v in the MILP. - pub candidates: Vec>, -} -``` - -| Part | Code | Target | -|---|---|---| -| Expression | `MemoGroup::logical_node`, an [`OperatorNode`](../../../crates/types/src/ir/operator/node.rs) whose [`Operator`](../../../crates/types/src/ir/operator/node.rs) is a `NonASAPOp` or `ASAPOp` variant with its own arguments | unchanged: the existing IR, no new identity type | -| Producer groups $G_i$ | `MemoGroup::child_groups` | unchanged | -| Eligibility $r_{G,i}$ | not stored: each candidate's private node chain fixes its producer | one logical requirement (#1203) and one physical requirement (#1200) per consumer input; see "Property requirement definition" | -| Candidates | `Memo::candidates[g]` | unchanged place; each candidate restructured (#1203) | -| Consumers | `MemoGroup::parents` (derived) | unchanged: derived from the producer groups | - -Hash-consing finds or inserts a group by its logical node: the operator, its arguments and its child groups. Stage 0 does this today (`hash_consed` in [`stage0.rs`](../../../crates/logical-asap-optimization/src/stage0.rs)); #1203 makes Pass 1 look up existing groups the same way. A different source, filter or query window gives a different group. A requested query window belongs to the logical node; the internal window layout used to compute it does not. - -The group's logical node is the frontend's operator, e.g. `quantile_over_time(0.95, latency[1m])`. It names no family and no sketch parameters: those are chosen by the candidates. A candidate's own operator (below) is an `ASAPOp` such as `SummaryAgg`, whose `FieldDataType::Sketch(SketchKind, …)` carries the family and $\theta$. Today the KLL state is not its own group: it is a private node inside each reader's candidate. #1203 makes every intermediate result its own group, so the KLL state over `latency[1m]` becomes a group, and `k = 200` and `k = 345` become two candidates of it. Its identity drops the sketch parameters, $\delta$ and the tumbling-window layout (#1203, decision 1). - -### Candidate data structure definition - -A candidate is one way to compute its group: an operator over producer groups, plus its properties. - -```rust -// crates/types/src/stages/candidates.rs (code today) -pub struct CandidateForMemoGroup { - pub fused_groups: Vec, - /// The implementation's operator, and its private nodes down to - /// `input_groups`. Family and θ are inside it (FieldDataType). - pub root_node: Rc, - pub input_groups: Vec, - pub origin: OriginOfCandidateForMemoGroup, - pub accuracy_budget_parts: usize, -} -``` - -```rust -// Target (#1203): e_v and p_v^log as separate fields, one operator, no private nodes. -pub struct CandidateForMemoGroup { - pub expression: CandidateExpression, // e_v - pub logical_properties: LogicalProperties, // p_v^log - pub input_requirements: Vec, // r^log_{c,i}, one per input group - pub origin: OriginOfCandidateForMemoGroup, - pub fused_groups: Vec, -} -pub struct CandidateExpression { pub operator: Operator } -``` - -Stage 2 expands each operator node of a logical candidate into physical candidates, one per valid physical property value: - -```rust -// crates/types/src/stages/physical/memo_candidates.rs, after #1200 PR 1 (#1214) -pub struct PhysicalCandidate { - pub operator_node: OperatorNodeOfCandidateForMemoGroupId, - pub properties: PhysicalProperties, // p_v^phys; was timing + materialization - /// Per input, the candidates that may produce it (filtered by α^phys). - pub producer_candidates_per_input: Vec, - pub may_produce_query_output: bool, - pub may_keep_across_evaluations_per_sole_reader_query: Vec, -} -``` - -| Formulation | Code today | Target | -|---|---|---| -| $e_v$ | `root_node` + `input_groups` (may hold private nodes) | `expression: Operator` (#1203) | -| $p^{\log}_v$ | inside `root_node`'s `FieldDataType`; sized per `accuracy_budget_parts` | `logical_properties` (#1203) | -| $p^{\mathrm{phys}}_v$ | `PhysicalCandidate::{timing, materialization}` on main | `PhysicalCandidate::properties` (#1200 PR 1, #1214) | -| edges $(u, c, i)$ | `producer_candidates_per_input` (sets, no edge ids) | `LogicalEdge`, `PhysicalEdge` (#1201) | - -### Property definition - -Each property is a struct with one field per dimension, each its own enum. - -**Logical properties** (#1203 target; today inside `root_node`): - -```rust -pub struct LogicalProperties { - representation: Representation, // family and θ - window_layout: Option, // ω_v -} -/// Reuses the code's checked types: `SketchKind::new` already rejects -/// parameters of another algorithm. -pub enum Representation { - Exact(ExactKind), // crates/types/src/ir/schema/sketch.rs - Summary(SketchKind), // SketchKind { algorithm, params } -} -pub enum WindowLayout { Tumbling { width_ms: u64, count: u32 } } // sliding variants: #1206 -``` - -Today these values are the `FieldDataType` of the candidate's summary operator: `ExactAggregate(ExactKind, ExactParams)` or `Sketch(SketchKind, GroupingStrategy)` ([`schema/mod.rs`](../../../crates/types/src/ir/schema/mod.rs)). The tumbling-window layout is the `SummaryTumblingWindowChain` operator's arguments. - -`error_bound` is derived when needed, not stored in `LogicalProperties`. Reuse [`estimators::local_guarantee`](../../../crates/accuracy/src/estimators/mod.rs) for parameters → guarantee and [`estimators::size_params`](../../../crates/accuracy/src/estimators/mod.rs) for target → parameters. These existing crate-private functions dispatch to the family's accuracy and sizing implementations. [`sizing::default_size_params`](../../../crates/accuracy/src/sizing.rs) already delegates to `size_params`. If another crate needs direct access, expose the existing implementation through the accuracy crate rather than duplicate its formulas. (#1203 currently proposes storing `error_bound`; this page derives it instead.) - -```rust -// crates/accuracy/src/estimators/mod.rs (code today) -pub(crate) fn local_guarantee(family: &FieldDataType, query: &SketchStatistic) -> Option; -pub(crate) fn size_params(kind: SketchAlgorithm, intent: &AggIntent, eps: f64, delta: f64) -> SketchParams; -``` - -Deriving a guarantee also needs the statistic being read: one family's parameters can give different guarantees for different statistics. Exact values yield an exact guarantee; an unsupported statistic or missing error model yields `None`. Target → parameters is a sizing operation with discrete values and family limits, not an exact mathematical inverse; validate the derived guarantee against the target rather than assume the target is achievable. - -**Physical properties** (#1200). PR 1 (#1214) adds timing and lifetime; PR 2 adds the schedule. - -```rust -// crates/types/src/ir/properties/execution_data_state.rs (code today) -pub enum ExecutionTiming { IngestionTime, QueryTime } - -// crates/types/src/physical/properties.rs (#1214) -pub enum Medium { Memory, Disk } -pub enum Lifetime { None, OneEvaluation(Medium), AcrossEvaluations(Medium) } -pub struct PhysicalProperties { - timing: ExecutionTiming, // τ - lifetime: Lifetime, // ℓ - // schedule: Option, // σ, #1200 PR 2 -} -impl PhysicalProperties { - pub fn new(timing: ExecutionTiming, lifetime: Lifetime) -> Result; - pub fn timing(self) -> ExecutionTiming; - pub fn lifetime(self) -> Lifetime; - pub fn is_materialized(self) -> bool; // ℓ ≠ None -} - -// #1200 PR 2 -pub enum Schedule { - Periodic { interval_ms: u64, phase_ms: u64 }, // OneEvaluation - TumblingWindowGrid { phase_ms: u64 }, // AcrossEvaluations -} -``` - -On main, timing and lifetime are still folded into one enum, [`Materialization`](../../../crates/types/src/physical/materialization.rs) (`NotMaterialized`, `MatAtQuery`, `MatAtQueryKeptAcrossEvaluations`, `MatAtIngestion`), next to `PhysicalCandidate::timing`. #1214 deletes it. The schedule is chosen in the MILP today (schedule groups `z[m, G]`, [`cost_terms.rs`](../../../crates/plan-selection/src/milp/cost_terms.rs)); #1200 PR 2 makes it a property. - -### Property requirement definition - -A requirement is stated per consumer input $(c, i)$, in two parts that follow the two property parts. Each is a struct with one field per dimension. - -```rust -/// r^log_{c,i} (#1203): one optional constraint per dimension; None accepts any value. -pub struct LogicalRequirement { - pub family: Option, - pub window: Option, // the reader's range and step - pub error_cap: Option, // validity of row (K)'s linear bound -} -/// α^log -pub fn satisfies(producer: &LogicalProperties, requirement: &LogicalRequirement) -> bool; - -/// r^phys_{c,i} (#1200): when the consumer reads the input, and on which times. -/// `new` rejects (IngestionTime, Later): ingestion has no evaluations. -pub struct PhysicalRequirement { - reader_timing: ExecutionTiming, - reads_from: Evaluation, // which evaluation's output the reader reads - reader_schedule: Option, // (I_c, φ_c); None for a non-periodic reader -} -pub enum Evaluation { Same, Later } -/// α^phys -pub fn satisfies_physical(producer: PhysicalProperties, requirement: PhysicalRequirement) -> bool; -``` - -| Requirement | Code today | Target | -|---|---|---| -| $r^{\log}_{c,i}$ | not represented | `LogicalRequirement` (#1203) | -| $\alpha^{\log}$ | not computed | `satisfies` (#1203) | -| $r^{\mathrm{phys}}_{c,i}$ | the consumer's timing | `PhysicalRequirement` (#1200 PR 2) | -| $\alpha^{\mathrm{phys}}$ | [`compatible_timing`](../../../crates/physical-asap-optimization/src/catalog/configurations.rs) on main; `PhysicalProperties::readable_at(reader_timing)` in #1214 | `satisfies_physical` (#1200 PR 2) | -| accuracy | [`AccuracyTarget`](../../../crates/types/src/ir/properties/accuracy.rs) at the query root, split into equal shares by `accuracy_budget_parts` | `AccuracyTarget` at the query root, checked by row (K) (#1204) | -| deployment capability | [`DeploymentCapabilities`](../../../crates/logical-asap-optimization/src/deployment_capabilities.rs): `supported_summaries`, `raw_data_retained`, `query_time_results_retained` | unchanged; prunes during generation | - -The set of legal properties is a function of the requirement, $\text{legal}(r) = f(r)$. One requirement can be written at different levels. For example, the same accuracy requirement can be stated two ways: - -| Requirement | Mapped to properties | Producer candidates enumerated | -|---|---|---| -| Accuracy = 95% rank accuracy | every family with rank error, and every configuration of it that meets 95% rank accuracy | (KLL, `k = 100`), (KLL, `k = 200`), … | -| Accuracy = KLL's `k ≥ 100` | KLL with `k ≥ 100` | (KLL, `k = 100`), (KLL, `k = 200`), … | - -In the code, the first form is the query's `AccuracyTarget`; Pass 1 maps it to parameters with `size_params`. The second form is what a sized candidate's `SketchParams` fixes today. #1203 keeps the first form at the query root and adds only `error_cap` per edge. - -What goes into a property requirement affects only the efficiency of enumeration, not correctness: -- **Some requirements already hold by the query semantics.** The expression and its producer groups already fix the input data and its fields, so no requirement needs to restate them. -- **It depends on how rules generate the candidate space.** A rule that only generates legal candidates needs no requirement for that dimension; a rule that generates every value needs the requirement to filter them. -- **A requirement prunes; it does not decide the plan.** A tighter requirement removes candidates earlier. Each query's composed error is still checked at its root in Stage 3 (row (K)), so a requirement that is too loose costs enumeration time, not a wrong plan. - -### Extensibility - -**Add a value to a property dimension**, e.g. a new summary family or `Medium::Disk`: -1. Add the enum variant. -2. Every exhaustive `match` on the dimension stops compiling until it handles the new value. This is why matches on properties name every value and never use a `_` arm. -3. For a family: add its predefined $\theta$ list, its error-bound function, its cost, and the readouts it answers. -4. Add any new invalid combinations to `new`. - -**Add a property dimension**, e.g. a compression codec or a partitioning: -1. Decide whether it is logical (it changes the result's values or error) or physical (it changes only when, where or how long the result is computed). Add a field (a new enum) to `LogicalProperties` or `PhysicalProperties`. -2. If consumers can constrain it, add an optional field to `LogicalRequirement` or `PhysicalRequirement` and one check to `satisfies` or `satisfies_physical`. -3. Add its invalid combinations with the other dimensions to `new`. -4. Add a level to the decision tree, with its prune rule, and place it by the considerations in "Order of decisions". -5. Add its cost terms. - -**Add a requirement:** -1. If it constrains one edge (a consumer input), add a field to `LogicalRequirement` or `PhysicalRequirement`, set it where the consumer is created, and check it in `satisfies` or `satisfies_physical`. -2. If it constrains a whole query or the whole plan (accuracy, latency, budgets), add it to the query's or workload's requirements and a row to Stage 3. Do not split it into per-edge requirements: that fixes each input's share in advance. -3. If it comes from the deployment, prune the values it rules out during candidate generation. - -**Add a rule:** -1. A rule matches over groups and their candidates' expressions, never over a reader's private operators. -2. It adds candidates to an existing group, or adds a new group with its candidates. It never copies a reader: every existing consumer of the group sees the new candidate through edges. -3. A candidate whose result has the same meaning as an existing group is added to that group, so hash-consing keeps sharing. -4. Each new candidate states its input requirements. - -## 4. Data structures and interfaces per stage +## 3. Data structures and interfaces per stage This section gives the target interface of each stage after the formulation refactor (#1207). Each stage's boundary type carries the notation that stage defines, and no stage reads a notation a later stage defines: logical stages carry no physical properties. From a99e9ff30b13795417c44a05212595ac2da3b063 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 20:14:57 +0000 Subject: [PATCH 06/67] docs(concepts): sections 1-2 in the formulation's terms; section 4 rules per pass and stage Co-Authored-By: Claude Opus 5.5 --- .../memo-groups-properties-requirements.md | 263 ++++++++++++------ 1 file changed, 176 insertions(+), 87 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 7769b3b10..602dd1d22 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -1,110 +1,138 @@ # Memo groups, properties and requirements -This page defines the memo's main data structures: groups, candidates, properties and requirements. It also lists every property and requirement, and gives the decision tree Stage 1 and Stage 2 walk to enumerate candidates. The symbols follow the planner formulation: -- a candidate is $v = (e_v, p_v)$; -- its logical properties are $p^{\log}$ and its physical properties are $p^{\mathrm{phys}}$; -- a consumer input's requirements are $r^{\log}_{c,i}$ and $r^{\mathrm{phys}}_{c,i}$; -- whether a producer satisfies them is $\alpha^{\log}$ and $\alpha^{\mathrm{phys}}$. +This page defines the memo's data structures in the terms of the optimization problem formulation: workloads, memo groups, candidates, properties and property requirements. It also gives the decision tree Stage 1 and Stage 2 walk to enumerate candidates, and each stage's interfaces (section 3). Where tracking issue #1207 extends the formulation, the extension is marked **(#1207)**. + +## Notation + +| Notation | Definition | +|---|---| +| $\mathcal W = \{(q, \mathcal T_q, A_q) \mid q \in \mathcal Q\}$ | workload over an evaluation horizon $H$: each query expression $q$, its evaluation schedule $\mathcal T_q \subseteq [0, H)$, and its accuracy target $A_q$ | +| $\mathcal Q$ | set of queries, each with one final output | +| $G$ | memo group: a logical result or an intermediate state family over specified input data, independent of parameter configurations | +| $G_q$ | memo group of query $q$'s final result | +| $v = (e_v, p_v)$ | candidate of a memo group: an expression with its properties | +| $e_v = \operatorname{op}_v(G_{v,1}, \dots, G_{v,n_v})$ | expression: operator $\operatorname{op}_v$ with its operator arguments (e.g. the summary family), over input memo groups $G_{v,i}$ | +| $p_v = (p^{\log}_v, p^{\mathrm{phys}}_v)$ | properties: logical and physical | +| $p^{\log}_v = \boldsymbol\theta_v$ | summary algorithm parameters, e.g. sketch size; **(#1207)** also the window layout $\omega_v$ | +| $p^{\mathrm{phys}}_v = (\tau_v, \ell_v)$ | execution timing and lifetime; **(#1207)** also the schedule $\sigma_v$ when $\ell_v \ne$ None | +| $(\varepsilon_v, \delta_v)$ | error bound and failure probability of $\boldsymbol\theta_v$ for the statistic $\operatorname{op}_v$ reads; derived, not a property | +| $c \in G_c$, $u \in G_u$ | consumer candidate and its group; producer candidate and its group | +| $\mathcal I(c)$ | input indices of consumer $c$ | +| $G_{c,i}$ | memo group required by input $i$ of consumer $c$ | +| $r_{c,i} = (r^{\log}_{c,i}, r^{\mathrm{phys}}_{c,i})$ | property requirement of input $i$ of consumer $c$ | +| $p(u) \succeq r_{c,i}$ | producer $u$'s properties satisfy the requirement: $p^{\log}(u) \succeq r^{\log}_{c,i} \land p^{\mathrm{phys}}(u) \succeq r^{\mathrm{phys}}_{c,i}$ | +| $\alpha^{\log}_{u,c,i}$, $\alpha^{\mathrm{phys}}_{u,c,i}$ | fixed indicators of $p^{\log}_u \succeq r^{\log}_{c,i}$ and $p^{\mathrm{phys}}_u \succeq r^{\mathrm{phys}}_{c,i}$ | +| $\mathcal C$ | physical candidates, each with a fixed parameter configuration, timing and lifetime | +| $P_{c,i}$ | physical candidates in the required input group, $G_{c,i} \cap \mathcal C$, before checking requirements | +| $\mathcal E$ | potential dependencies $\{(u, c, i) \mid c \in \mathcal C,\ i \in \mathcal I(c),\ u \in P_{c,i}\}$ | +| $x_v$, $y_{u,c,i}$, $z_{q,v}$, $t_v$ | MILP variables: candidate selected; producer $u$ supplies input $i$ of $c$; $v \in G_q$ supplies $q$'s result; topological rank | +| (A)–(I) | MILP rows: input satisfaction, producer selection and sharing, query coverage, output selection, acyclicity, utilization, logical requirements ($y \le \alpha^{\log}$), physical requirements ($y \le \alpha^{\mathrm{phys}}$), domains | +| $w_v$ | fixed cost: $C_{\text{read}}(v) + C_{\text{compute}}(v)$ plus $C^{\text{one}}_{\text{retain}}(v)$ or $C^{\text{across}}_{\text{retain}}(v)$ by $\ell_v$ | +| $\Pi$, $V_\Pi$ | selected physical ASAP DAG and its node set; $\operatorname{Cost}(\Pi) = \sum_{v \in \mathcal C} w_v x_v$ | +| **(#1207)** $\kappa$ | execution context of a candidate with $\ell_v$ = None: a query evaluation $\kappa_q$, an ingestion pipeline $I_r$, or a materialized candidate's computation $\kappa_m$ | +| **(#1207)** (K), (R), (L), budgets | accuracy at each query root, $\sum_u \lambda_{q,u}\varepsilon_u a_{q,u} \le \varepsilon_q$ and $\sum_u \delta_u a_{q,u} \le \delta_q$; retention $h_v$; latency; resource budgets | ## Example -Query A: `quantile_over_time(0.95, latency[1m])` with `max_error 0.01`. Query B: `quantile_over_time(0.99, latency[1m]) / quantile_over_time(0.50, latency[1m])` with the same target. +Workload: query A, `quantile_over_time(0.95, latency[1m])`, and query B, `quantile_over_time(0.99, latency[1m]) / quantile_over_time(0.50, latency[1m])`. Both are evaluated every 10 s ($\mathcal T_A = \mathcal T_B = 10\,\text{s}\cdot\mathbb Z$) with accuracy target $A_q$: rank error 0.01. ``` G1 Scan latency[1m] - producers: v1 = Scan (QueryTime, None) -G2 KLL state over G1 one group, whatever the size - producers: v2 = SummaryAgg[KLL](G1) θ = {k = 200} ε = 0.008 δ = 0.01 - v3 = SummaryAgg[KLL](G1) θ = {k = 345} ε = 0.005 δ = 0.01 - (each also expanded by Stage 2 into its (τ, ℓ, σ) alternatives) -G3 p95 of G2 producers: SummaryEstimate[q=0.95](G2) -G4 p99 of G2 producers: SummaryEstimate[q=0.99](G2) -G5 p50 of G2 producers: SummaryEstimate[q=0.50](G2) -G6 G4 / G5 producers: Divide(G4, G5) input requirement: error_cap on each input -Query roots: A → G3, B → G6 + candidate: v1 = Scan p^phys = (QueryTime, None) +G2 KLL state over G1 one group, whatever θ + candidates: v2 = SummaryAgg[KLL](G1) θ = {k = 200} ε = 0.008 δ = 0.01 + v3 = SummaryAgg[KLL](G1) θ = {k = 345} ε = 0.005 δ = 0.01 + (Stage 2 expands each into its (τ, ℓ) alternatives) +G3 = G_A SummaryEstimate[q=0.95](G2) +G4 SummaryEstimate[q=0.99](G2) +G5 SummaryEstimate[q=0.50](G2) +G6 = G_B Divide(G4, G5) r^log on each input: error cap (#1207) ``` The ε values are illustrative. -- G2 is one group: the KLL state over `latency[1m]`. Its sizes are candidates of G2, not separate groups. -- G3, G4 and G5 are consumers of G2. They are not stored in G2; each is a group with its own producers whose expression reads G2. -- A meets 0.01 with either size. B adds its two operands' errors (0.016 with `k = 200`, 0.010 with `k = 345`) and needs `k = 345` for both, so a plan where all three read v3 builds one KLL state. +- G2 is one memo group: the KLL state over `latency[1m]`. Its parameter configurations are candidates of G2, not separate groups. The family KLL is an argument of $\operatorname{op}_v$. +- G3, G4 and G5 are consumers of G2: their candidates' expressions read G2 as $G_{c,1}$. G2 does not store them. +- A meets 0.01 with either θ. B adds its two operands' errors (0.016 with `k = 200`, 0.010 with `k = 345`) and needs `k = 345` for both, so a plan where G3's, G4's and G5's candidates all read v3 builds one KLL state. -## 1. Properties and requirements over properties +## 1. Properties and property requirements ### Properties -A property is a value a candidate has. Every property is one dimension of a product: a struct with one field per dimension, never a flat enum that lists the combinations. Invalid combinations are rejected by the struct's checked constructor. +A property is a value a candidate has. Properties are products of dimensions: one field per dimension, never a flat enum that lists the combinations. Invalid combinations are rejected by the checked constructor. -| Dimension | Values | Logical or physical | Fixed by | Code | -|---|---|---|---|---| -| Which family to use (operator) | `Exact` (exact accumulator kinds: Sum, Count, Min, Max, Increase, Rate, IRate), or `Summary { family }` with family in KLL, DDSketch, CMS, CountSketch, CMS with heap, CountSketch with heap, UnivMon, HLL, KMV, the Hydra variants | logical | enumerated | `FieldDataType::{ExactAggregate, Sketch}` in the candidate's operator ([`schema/mod.rs`](../../../crates/types/src/ir/schema/mod.rs)); `SketchAlgorithm` | -| Parameter configuration $\theta$ | per family, from a predefined list: KLL `k`; DDSketch `alpha`; CMS / CountSketch `width × depth`; CMS / CountSketch with heap `width × depth × heap_size`; UnivMon `heap_size × rows × cols × layers`; HLL `precision`; KMV `k` | logical | enumerated | `SketchParams` inside `SketchKind` ([`sketch.rs`](../../../crates/types/src/ir/schema/sketch.rs)), sized by `size_params` per `accuracy_budget_parts` | -| Error bound $(\varepsilon_v, \delta_v)$ | computed on demand from $\theta$ and the statistic, in the family's accuracy metric (below); exact is $(0, 0)$ | logical | derived from $\theta$, not stored or chosen | `local_guarantee` → `AccuracyGuarantee` ([`estimators`](../../../crates/accuracy/src/estimators/mod.rs)) | -| Window configuration $\omega$ | none; or tumbling windows of width $w$ and count $n$, with $w$ from the gcd closure of the readers' ranges and steps. Later: sliding windows by merge or by subtraction | logical | enumerated | `SummaryTumblingWindowChain` arguments, added by Pass 2 ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs)) | -| Timing $\tau$ | `IngestionTime`, `QueryTime` | physical | enumerated | `ExecutionTiming` ([`execution_data_state.rs`](../../../crates/types/src/ir/properties/execution_data_state.rs)); `PhysicalCandidate::timing` | -| Lifetime $\ell$ | `None`, `OneEvaluation(Medium)`, `AcrossEvaluations(Medium)`, with `Medium` in `Memory`, `Disk` | physical | enumerated | `Materialization` on main; `Lifetime`, `Medium` in #1214 ([`physical/`](../../../crates/types/src/physical)) | -| Schedule $\sigma$ | only when $\ell \ne$ None. `Periodic { interval, phase }` for `OneEvaluation`; `TumblingWindowGrid { phase }` for `AcrossEvaluations` | physical | enumerated from the readers' schedules | not a property: schedule groups `z[m, G]` in the MILP ([`cost_terms.rs`](../../../crates/plan-selection/src/milp/cost_terms.rs)); a property in #1200 PR 2 | +What is computed is the expression $e_v$, not a property: the operator, its arguments (including the summary family), and its input groups. A memo group's candidates differ in their properties, and, when rules add other ways to compute the same result, in their expressions. -A candidate is materialized when $\ell \ne$ None. This is derived, not stored. +| Dimension | Values | Part of | Code | +|---|---|---|---| +| Parameter configuration $\boldsymbol\theta_v$ | per family, from a predefined list: KLL `k`; DDSketch `alpha`; CMS / CountSketch `width × depth`; CMS / CountSketch with heap `width × depth × heap_size`; UnivMon `heap_size × rows × cols × layers`; HLL `precision`; KMV `k`. None for an exact operator | $p^{\log}_v$ | `SketchParams` inside `SketchKind` ([`sketch.rs`](../../../crates/types/src/ir/schema/sketch.rs)), sized by `size_params` per `accuracy_budget_parts` today | +| **(#1207)** Window layout $\omega_v$ | none; or tumbling windows of width $w$ and count $n$, with $w$ from the gcd closure of the readers' ranges and steps. Later: sliding windows by merge or by subtraction | $p^{\log}_v$ | `SummaryTumblingWindowChain` arguments, added by Pass 2 ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs)) | +| Execution timing $\tau_v$ | IngestionTime, QueryTime | $p^{\mathrm{phys}}_v$ | `ExecutionTiming` ([`execution_data_state.rs`](../../../crates/types/src/ir/properties/execution_data_state.rs)) | +| Lifetime $\ell_v$ | None, OneEvaluation, AcrossEvaluations; **(#1207)** a materialized lifetime carries its medium, Memory or Disk | $p^{\mathrm{phys}}_v$ | `Materialization` on main; `Lifetime`, `Medium` in #1214 ([`physical/`](../../../crates/types/src/physical)) | +| **(#1207)** Schedule $\sigma_v$ | only when $\ell_v \ne$ None: Periodic (interval, phase) for OneEvaluation; tumbling-window grid phase for AcrossEvaluations | $p^{\mathrm{phys}}_v$ | schedule groups `z[m, G]` in the MILP today ([`cost_terms.rs`](../../../crates/plan-selection/src/milp/cost_terms.rs)); a property in #1200 PR 2 | -Invalid combinations, rejected by `PhysicalProperties::new` (the list in #1200; #1214 implements the first, the others come with the schedule in #1200 PR 2): +Lifetime None means the output is discarded after its reader's execution completes. OneEvaluation retains the result until all readers within one evaluation finish. AcrossEvaluations retains it throughout the evaluation period, so later evaluations reuse it. A candidate is materialized when $\ell_v \ne$ None; this is derived, not stored. + +The error bound $(\varepsilon_v, \delta_v)$ is not a dimension: it is a function of $\boldsymbol\theta_v$ and the statistic $\operatorname{op}_v$ reads, derived when needed (`local_guarantee` in [`estimators`](../../../crates/accuracy/src/estimators/mod.rs)). An exact candidate has $(0, 0)$. + +Invalid combinations of $p^{\mathrm{phys}}$, rejected by `PhysicalProperties::new` (#1200; #1214 implements the first, the others come with $\sigma$ in #1200 PR 2): | Combination | Why | |---|---| | (IngestionTime, OneEvaluation) | there is no evaluation at ingestion time | -| $\ell$ = None with a schedule | where an unmaterialized result runs is its reader's execution context, not a property | +| $\ell$ = None with a schedule | where an unmaterialized result runs is its reader's execution context $\kappa$, not a property | | $\ell \ne$ None without a schedule | a materialized result must say on which times or grid it is delivered | | OneEvaluation with a grid, AcrossEvaluations with a periodic schedule | each lifetime has one kind of schedule | | a phase at or above the grid width or interval, or interval 0 | the phase is taken modulo the width or interval | | IngestionTime with a grid phase ≠ 0 | ingestion-time grids are anchored at the epoch today: the plan carries no origin | -Valid physical states: (QueryTime, None), (IngestionTime, None), (QueryTime, OneEvaluation(m)), (QueryTime, AcrossEvaluations(m)), (IngestionTime, AcrossEvaluations(m)), for each medium $m$. +Valid $(\tau, \ell)$: (QueryTime, None), (IngestionTime, None), (QueryTime, OneEvaluation), (QueryTime, AcrossEvaluations), (IngestionTime, AcrossEvaluations), each materialized one for each medium. + +### Property requirements -### Requirements over properties +Each input $i$ of consumer $c$ names a producer group $G_{c,i}$ and a property requirement $r_{c,i} = (r^{\log}_{c,i}, r^{\mathrm{phys}}_{c,i})$. Logical satisfaction checks configuration requirements; physical satisfaction checks whether the producer's timing and lifetime jointly support the required access. For a monotone size parameter, $\succeq$ reduces to numerical dominance, e.g. $k_u \ge k_{\text{required}}$ for KLL. -A requirement is a constraint that a consumer, a query or the deployment puts on producers. One requirement can constrain several property dimensions, and one dimension can be constrained by several requirements. +The workload's accuracy target $A_q$ is not an input requirement **(#1207)**: a per-input bound would fix each input's share of the query's budget. Row (K) bounds each query's composed error at its root instead. What stays in $r^{\log}$ is what depends only on the consumer. -| Requirement | Who states it | Values | Constrains | Checked where | +| Requirement | Stated by | Values | Constrains | Checked by | |---|---|---|---|---| -| Accuracy | each query, at its root | metric: absolute (in the value's units, or times the stream's L1 or L2 norm), relative, rank; bound $\varepsilon_q$; failure probability $\delta_q$; scope: per value, whole result, or groups above a share of the records | operator family and $\theta$ of every approximate candidate the query reads, through their $(\varepsilon_v, \delta_v)$ | Stage 3, row (K): the composed error at the root. Not per edge, because a per-input bound would fix each input's share of the budget | -| Validity cap `error_cap` | a non-linear consumer (ratio, product), per input | $\varepsilon$ cap | $\theta$ of the producer | per edge, $\alpha^{\log}$: row (K)'s linear bound holds only below the cap | -| Summary family | a consumer that reads summary state, per input | a family, e.g. KLL for a quantile readout | operator family | per edge, $\alpha^{\log}$ | -| Window | a consumer that reads windows, per input | the reader's range and step | window configuration: the width must divide both | per edge, $\alpha^{\log}$ | -| Capability of ingestion-time compute | the deployment and the data workload | the source has an ingestion pipeline; the executor supports the family; query-time results may be kept across evaluations | timing; operator family; (QueryTime, AcrossEvaluations) | candidate generation: such candidates are not offered | -| Reader timing and evaluation | a consumer, per input | when it reads (ingestion or query time) and from which evaluation (same or later) | timing and lifetime of the producer | per edge, $\alpha^{\mathrm{phys}}$ | -| Reader schedule | a periodic consumer, per input | its interval and phase $(I_c, \varphi_c)$ | schedule of the producer | per edge, $\alpha^{\mathrm{phys}}$ | -| Latency, resource budgets | the workload | per-query latency bound; memory and CPU budgets | the whole plan | Stage 3, rows (L) and (B) | +| Accuracy target $A_q$ | the workload, per query | metric: absolute (in the value's units, or times the stream's L1 or L2 norm), relative, rank; bound $\varepsilon_q$; failure probability $\delta_q$; scope: per value, whole result, or groups above a share of the records | $\boldsymbol\theta_u$ of every approximate candidate $q$ depends on, through $(\varepsilon_u, \delta_u)$ | row (K) **(#1207)** | +| Error cap, in $r^{\log}_{c,i}$ | a non-linear consumer (ratio, product), per input | a cap on $\varepsilon_u$; for a monotone size parameter, a minimum size $k_u \ge k_{\text{required}}$ | $\boldsymbol\theta_u$ | $\alpha^{\log}$, row (G): row (K)'s linear bound holds only below the cap | +| Window, in $r^{\log}_{c,i}$ **(#1207)** | a consumer that reads windows, per input | the reader's range and step | $\omega_u$: the width must divide both | $\alpha^{\log}$, row (G) | +| Summary family, in $r^{\log}_{c,i}$ | a consumer that reads summary state | a family | usually already fixed by $G_{c,i}$, whose $\operatorname{op}$ names the family | $\alpha^{\log}$, row (G) | +| Reader timing and evaluation, in $r^{\mathrm{phys}}_{c,i}$ | the consumer, per input | when it reads (ingestion or query time) and from which evaluation (same or later) | $\tau_u$, $\ell_u$ | $\alpha^{\mathrm{phys}}$, row (H) | +| Reader schedule, in $r^{\mathrm{phys}}_{c,i}$ **(#1207)** | a periodic consumer, from $\mathcal T_q$ | its interval and phase $(I_c, \varphi_c)$ | $\sigma_u$ | $\alpha^{\mathrm{phys}}$, row (H) | +| Deployment capability | the deployment and data | the source has an ingestion pipeline; the executor supports the family; query-time results may be kept across evaluations | $\tau$, $\operatorname{op}$, (QueryTime, AcrossEvaluations) | candidate generation: such candidates are not in $\mathcal C$ | +| Latency, resource budgets **(#1207)** | the workload | per-query latency bound; memory and CPU budgets | the whole plan | rows (L) and budgets | Accuracy metrics that may be added later, when a primitive needs them: top-k recall or precision, and a false-positive rate for membership summaries. -How requirements map to property dimensions: +How requirements map to properties: ``` -Accuracy (query) ──► operator family, θ (through ε_v, δ_v; row (K)) -error_cap (edge) ──► θ -Summary family (edge) ──► operator family -Window (edge) ──► window configuration -Ingestion capability (deploy)──► timing, operator family, lifetime -Reader timing/evaluation ──► timing, lifetime -Reader schedule ──► schedule -Latency, budgets (workload) ──► every dimension, through cost +A_q (workload) ──► θ_u of every approximate input (through ε_u, δ_u; row (K)) +error cap r^log ──► θ_u +window r^log ──► ω_u +reader timing r^phys ──► τ_u, ℓ_u +reader schedule r^phys ──► σ_u +deployment ──► τ, op, ℓ (not generated) +latency, budgets ──► every dimension, through w_v ``` ## 2. Decision tree of properties -Each memo group's candidates are the leaves of a tree. Every level decides one property dimension, and each edge out of a node is one value of that dimension. A path from the root to a leaf is one candidate. +Each memo group's candidates are the leaves of a tree. The first level chooses the expression $e_v$; every later level decides one property dimension. A path from the root to a leaf is one physical candidate in $\mathcal C$. ```mermaid flowchart TD - G["Group G
(what is computed)"] --> R1["Exact"] - G --> R2["Summary: KLL"] - G --> R3["Summary: DDSketch"] - G --> R4["…other families
(prune: executor support,
consumer's family requirement)"] + G["Memo group G
(logical result)"] --> R1["e_v: exact operator"] + G --> R2["e_v: SummaryAgg[KLL]"] + G --> R3["e_v: SummaryAgg[DDSketch]"] + G --> R4["e_v: …other operators
(prune: executor support)"] R2 --> T1["θ: k = 200
ε = 0.008"] R2 --> T2["θ: k = 345
ε = 0.005"] - R2 --> T3["θ: k = …
(prune: ε above every
reading query's target)"] + R2 --> T3["θ: k = …
(prune: ε above every
reading query's ε_q)"] T2 --> W1["ω: no windows"] T2 --> W2["ω: tumbling, width w
(prune: w must divide
some reader's range and step)"] @@ -113,49 +141,48 @@ flowchart TD W2 --> P2["τ = IngestionTime
(prune: no ingestion pipeline)"] P1 --> L1["ℓ = None"] - P1 --> L2["ℓ = OneEvaluation(m)"] - P1 --> L3["ℓ = AcrossEvaluations(m)
(prune: deployment keeps
no query-time results)"] + P1 --> L2["ℓ = OneEvaluation"] + P1 --> L3["ℓ = AcrossEvaluations
(prune: deployment keeps
no query-time results)"] P2 --> L4["ℓ = None"] - P2 --> L5["ℓ = AcrossEvaluations(m)"] + P2 --> L5["ℓ = AcrossEvaluations"] P2 -. invalid .-> L6["ℓ = OneEvaluation"] - L2 --> S1["σ: Periodic (I, φ)
from the gcd closure
of the readers' schedules"] + L2 --> S1["σ: Periodic (I, φ)
from the gcd closure
of the readers' 𝒯_q"] L3 --> S2["σ: grid phase φ mod w"] L5 --> S3["σ: grid phase 0"] ``` The levels, top to bottom: -| Level | Dimension | Stage | Values offered | Pruned when | Code | +| Level | Decides | Stage | Values offered | Pruned when | Code | |---|---|---|---|---|---| -| 1 | family (operator) | Stage 1, Pass 1 and Pass 2 rules | each way to compute the group | the executor cannot build the family; no consumer's family requirement accepts it | Pass 1 ([`per_computation_candidates.rs`](../../../crates/logical-asap-optimization/src/pass1/per_computation_candidates.rs)); Pass 2 rules ([`rules/`](../../../crates/logical-asap-optimization/src/pass2/memo/rules)) | -| 2 | $\theta$ | Stage 1, Pass 1 | the family's predefined list | $\varepsilon_v$ alone exceeds every reading query's target | Pass 1 sizing per `accuracy_budget_parts` ([`memo_population.rs`](../../../crates/logical-asap-optimization/src/pass1/memo_population.rs)); a predefined list in #1203 PR 2 | -| 3 | window configuration $\omega$ | Stage 1, Pass 2 | tumbling widths from the gcd closure of readers' ranges and steps | no reader's range and step is a multiple of the width | Pass 2 ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs), [`state_groups.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/state_groups.rs)) | -| 4 | timing $\tau$ | Stage 2 | QueryTime; IngestionTime | the source has no ingestion pipeline | Stage 2 `enumerate_candidate_configurations` ([`configurations.rs`](../../../crates/physical-asap-optimization/src/catalog/configurations.rs)) | -| 5 | lifetime $\ell$ | Stage 2 | None, OneEvaluation(m), AcrossEvaluations(m) | invalid with $\tau$; the deployment keeps no query-time results across evaluations | same function, with `DeploymentCapabilities::query_time_results_retained` | -| 6 | schedule $\sigma$ | Stage 2 | the gcd closure $\Sigma$ of the readers' schedules | only offered when $\ell \ne$ None | Stage 3 MILP today; Stage 2 in #1200 PR 2 | - -What the tree does not decide: -- **Which producer each consumer input reads.** That is the edge choice $y_{u,c,i}$ in Stage 3. Two consumers of one group may read different leaves, e.g. A reads `k = 200` and B reads `k = 345`. -- **Where an unmaterialized candidate runs.** A candidate with $\ell$ = None runs inside each reader's execution context; Stage 3 chooses it. -- **How much a materialized candidate retains.** The amount $h_v$ depends on the readers' longest range; Stage 3 chooses it. +| 1 | expression $e_v$ ($\operatorname{op}_v$ with its family) | Stage 1, Pass 1 and Pass 2 rules | each way to compute the group | the executor cannot build the family | Pass 1 ([`per_computation_candidates.rs`](../../../crates/logical-asap-optimization/src/pass1/per_computation_candidates.rs)); Pass 2 rules ([`rules/`](../../../crates/logical-asap-optimization/src/pass2/memo/rules)) | +| 2 | $\boldsymbol\theta_v$ | Stage 1, Pass 1 | the family's predefined list | $\varepsilon_v$ alone exceeds every reading query's $\varepsilon_q$ | Pass 1 sizing per `accuracy_budget_parts` ([`memo_population.rs`](../../../crates/logical-asap-optimization/src/pass1/memo_population.rs)); a predefined list in #1203 PR 2 | +| 3 | $\omega_v$ **(#1207)** | Stage 1, Pass 2 | tumbling widths from the gcd closure of readers' ranges and steps | no reader's range and step is a multiple of the width | Pass 2 ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs), [`state_groups.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/state_groups.rs)) | +| 4 | $\tau_v$ | Stage 2 | QueryTime; IngestionTime | the source has no ingestion pipeline | `enumerate_candidate_configurations` ([`configurations.rs`](../../../crates/physical-asap-optimization/src/catalog/configurations.rs)) | +| 5 | $\ell_v$ | Stage 2 | None, OneEvaluation, AcrossEvaluations | invalid with $\tau_v$; the deployment keeps no query-time results across evaluations | same function, with `DeploymentCapabilities::query_time_results_retained` | +| 6 | $\sigma_v$ **(#1207)** | Stage 2 | the gcd closure $\Sigma$ of the readers' schedules $\mathcal T_q$ | only offered when $\ell_v \ne$ None | Stage 3 MILP today; Stage 2 in #1200 PR 2 | + +What the tree does not decide, because it is a choice of the MILP over $\mathcal C$ and $\mathcal E$: +- **Which producer each consumer input reads:** $y_{u,c,i}$. Two consumers of one group may read different leaves, e.g. one reads `k = 200` and another `k = 345`. +- **Where an unmaterialized candidate runs (#1207):** a candidate with $\ell_v$ = None runs inside each reading context $\kappa$. +- **How much a materialized candidate retains (#1207):** $h_v$, from the selected readers' longest range, row (R). ### Order of decisions -**The order does not affect correctness.** The set of leaves is the product of the dimensions' value sets, minus the invalid combinations and the pruned values. Every prune rule depends only on the value it removes and on values already decided above it, so any order that decides a pruned dimension after the dimensions its prune rule reads yields the same set of leaves. +**The order does not affect correctness.** The leaves are the product of the dimensions' value sets, minus the invalid combinations and the pruned values. Every prune rule depends only on the value it removes and on values decided above it, so any order that decides a pruned dimension after the dimensions its prune rule reads yields the same $\mathcal C$. **The order affects the efficiency of enumeration.** The number of nodes visited is $\sum_{d} \prod_{d' \le d} |\text{values kept at } d'|$, so a dimension that prunes much should come early, and a dimension whose values multiply without pruning should come late. Four considerations set the order above: -1. **Logical before physical.** Logical dimensions (operator family, $\theta$, $\omega$) change the result's values or error; physical dimensions ($\tau$, $\ell$, $\sigma$) change only when and how long it is computed. Deciding logical dimensions first lets Stage 1 hash-cons groups and check $\alpha^{\log}$ on edges once, before Stage 2 multiplies each logical candidate by its physical alternatives. -2. **Strong prunes early.** The family is pruned by the executor and by consumers' family requirements, and $\theta$ by the accuracy target, before the window and physical levels multiply them. -3. **Dependent dimensions adjacent.** $\ell$ is constrained by $\tau$ (no OneEvaluation at ingestion), and $\sigma$ exists only when $\ell \ne$ None and its kind follows $\ell$. Deciding them in the order $\tau \to \ell \to \sigma$ means each invalid combination is pruned at the first level where it can be detected. -4. **Deployment-wide prunes as early as they can go.** If the deployment has no ingestion pipeline, the IngestionTime subtree is pruned once per candidate. Moving $\tau$ above $\omega$ would prune it once per group instead; this is an alternative order with the same leaves. +1. **Logical before physical.** $e_v$, $\boldsymbol\theta_v$ and $\omega_v$ change the result's values or error; $\tau_v$, $\ell_v$ and $\sigma_v$ change only when and how long it is computed and kept. Deciding $p^{\log}$ first lets Stage 1 hash-cons groups and compute $\alpha^{\log}$ once, before Stage 2 multiplies each logical candidate by its $p^{\mathrm{phys}}$ alternatives. +2. **Strong prunes early.** The operator is pruned by the executor, and $\boldsymbol\theta_v$ by the accuracy targets, before the window and physical levels multiply them. +3. **Dependent dimensions adjacent.** $\ell_v$ is constrained by $\tau_v$, and $\sigma_v$ exists only when $\ell_v \ne$ None and its kind follows $\ell_v$. Deciding $\tau \to \ell \to \sigma$ prunes each invalid combination at the first level where it can be detected. +4. **Deployment-wide prunes as early as they can go.** Without an ingestion pipeline, the IngestionTime subtree is pruned once per candidate. Moving $\tau$ above $\omega$ would prune it once per group instead; this is an alternative order with the same leaves. Alternative orders worth comparing: -- **$\tau$ first.** In a deployment where most sources have no ingestion pipeline, deciding timing at the group level prunes half the tree before $\theta$ and $\omega$ multiply it. -- **$\theta$ last among logical dimensions.** $\theta$ has the most values. Deciding $\omega$ before $\theta$ visits fewer nodes when many widths are pruned by readers' ranges. -- **Requirement-driven (top-down) vs producer-driven (bottom-up).** Top-down enumeration starts from each consumer's requirement and offers only the values it accepts; bottom-up offers every value and lets $\alpha$ filter edges. Top-down visits fewer nodes but must union the requirements of all consumers of a group, so that sharing between consumers is not lost. - +- **$\tau$ first.** Where most sources have no ingestion pipeline, deciding timing at the group level prunes half the tree before $\boldsymbol\theta$ and $\omega$ multiply it. +- **$\boldsymbol\theta$ last among logical dimensions.** $\boldsymbol\theta$ has the most values. Deciding $\omega$ before $\boldsymbol\theta$ visits fewer nodes when many widths are pruned by readers' ranges. +- **Requirement-driven (top-down) vs producer-driven (bottom-up).** Top-down enumeration starts from each consumer's $r_{c,i}$ and offers only the values it accepts; bottom-up offers every value and lets $\alpha$ filter $\mathcal E$. Top-down visits fewer nodes but must union the requirements of all consumers of a group, so that sharing between consumers is not lost. ## 3. Data structures and interfaces per stage @@ -167,7 +194,7 @@ This section gives the target interface of each stage after the formulation refa | Stage 1, Pass 1 | `DagOfLogicalMemoGroups` | `Memo` | $v = (e_v, p^{\log}_v)$, $r^{\log}_{c,i}$, $\theta$ list | #1203 | | Stage 1, Pass 2 | `Memo` | `LogicalASAPMemoCandidates` | rule-added candidates; logical edges with $\alpha^{\log}$ | #1203, #1201 | | Stage 2 | `LogicalASAPMemoCandidates` | `PhysicalASAPMemoCandidates` | $p^{\mathrm{phys}}_v = (\tau, \ell, \sigma)$, $r^{\mathrm{phys}}_{c,i}$, physical edges $\mathcal E$ with $\alpha^{\mathrm{phys}}$ | #1200, #1201 | -| Stage 3 | `PhysicalASAPMemoCandidates` | `DagOfSelectedCandidates` | contexts $\kappa$; cost $w_v$ (`asap-cost`); $x$, $y$, $z$, $t$, $h$; rows (A)–(K), (R), (L), (B) | #1204 | +| Stage 3 | `PhysicalASAPMemoCandidates` | `DagOfSelectedCandidates` | contexts $\kappa$; cost $w_v$ (`asap-cost`); $x$, $y$, $z$, $t$, $h$; rows (A)–(K), (R), (L), budgets | #1204 | | Realizer, export | `DagOfSelectedCandidates` | `PhysicalASAPDAG` | — | #1200 (export fields) | ### Stage 0: frontends and hash-consing @@ -368,7 +395,7 @@ fn reader_constraint_rows(..); // (J) fn accuracy_rows(..); // (K) Σ λ_{q,u} ε_u a_{q,u} ≤ ε_q, Σ δ_u a_{q,u} ≤ δ_q fn retention_rows(..); // (R) h_v ≥ need_{c,i} · y_{v,c,i} fn latency_rows(..); // (L) -fn resource_budget_rows(..); // (B) budgets +fn resource_budget_rows(..); // budgets (#1207) // Objective: min Σ w_v x_v + Σ ρ_κ w_v x_{κ,v} + Σ π_v h_v // crates/types/src/stages/physical/graph.rs (unchanged) @@ -385,3 +412,65 @@ pub struct DagOfSelectedCandidates { ### Realizer and export The realizer builds the plan from `DagOfSelectedCandidates`; each node's physical properties come from its `PhysicalCandidate`. The export writes timing and lifetime (and later the schedule) per node of `PhysicalASAPDAG`, changed together with the backend. + +## 4. Rules: how each pass and stage adds candidates + +A rule reads the memo and adds to it only through the interfaces of section 3: +- `Memo::find_or_add_group(expression)` for a group, so an expression that already exists returns its group (hash-consing); +- `Memo::add_candidate(group, CandidateForMemoGroup { expression, logical_properties, input_requirements, fused_groups })` for a candidate $v = (e_v, p^{\log}_v)$ with its $r^{\log}_{v,i}$. + +A rule never copies a reader and never edits an existing candidate: every consumer of a group sees a new candidate of that group through $\mathcal E$. Rules match over groups and their candidates' expressions, never over a reader's private operators. The order of the rules below is the order the code runs them; it affects which groups exist when a later rule matches, not which plans are valid. + +### Stage 0: hash-consing + +| Rule | Code | Matches | Adds | +|---|---|---|---| +| Hash-consing | `hash_consed` ([`stage0.rs`](../../../crates/logical-asap-optimization/src/stage0.rs)) | every operator node of every query root | one `MemoGroup` per distinct expression; `query_roots` point at $G_q$. No candidates. | + +### Stage 1, Pass 1: ways to compute each group + +Pass 1 adds, for each group a query computes, every way to compute it. Each intermediate result of a way to compute is its own group (`find_or_add_group`), so `Project ← SummaryEstimate ← SummaryAgg` is three groups, each with a one-operator candidate. + +| Way to compute | Code | Adds | +|---|---|---| +| Exact | [`per_computation_candidates.rs`](../../../crates/logical-asap-optimization/src/pass1/per_computation_candidates.rs), [`memo_population.rs`](../../../crates/logical-asap-optimization/src/pass1/memo_population.rs) | a candidate with the exact operator; $\boldsymbol\theta_v$ none, $(\varepsilon_v, \delta_v) = (0, 0)$ | +| Exact accumulator | same | a state group `ExactAggregate(kind)` over the input group, one candidate; a `FinalizeExactAccumulator` candidate in the result's group reading it | +| Summary, per family the executor supports | same | a state group `SummaryAgg[family]` over the input group, with **one candidate per $\boldsymbol\theta$ in the family's predefined list**, dropping a $\boldsymbol\theta$ whose $\varepsilon_v$ alone exceeds every reading query's $\varepsilon_q$; a `SummaryEstimate[statistic]` candidate in the result's group reading the state group, with $r^{\log}$ = error cap where the consumer is non-linear | +| Fused build (#1203 2c) | same | a build that computes several targets at once: a candidate of the outer target's state group, with `fused_groups` naming the others | + +Example: for A and B, Pass 1 adds G2 = `SummaryAgg[KLL](G1)` with candidates `k = 200` and `k = 345`, and G3, G4, G5 = `SummaryEstimate[q](G2)`. A and B's three estimates read one state group because their expressions over G1 hash-cons to one group. + +Today, Pass 1 instead sizes one $\boldsymbol\theta$ per `accuracy_budget_parts` and keeps `SummaryEstimate ← SummaryAgg` as private nodes inside one candidate. #1203 2a makes the intermediate results groups; #1203 PR 2 replaces sizing by the predefined list. + +### Stage 1, Pass 2: rules over groups + +| # | Rule | Code | Matches | Adds | Status | +|---|---|---|---|---|---| +| 1 | Local window composition (tumbling windows) | `add_local_window_composition` ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs)) | a summary state group read by window readers whose ranges and steps share a gcd width $w$ | a candidate of the **same state group** with $\omega_v$ = tumbling $(w, n)$, its $r^{\log}$ window requirement on each reader input | today it copies readers per layout (`TumblingWindows` origin); #1160 makes it a candidate of the state group | +| 2 | Scan widening | `add_scan_label_candidates` ([`scan_labels.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/scan_labels.rs)) | several scan groups of one metric with different label sets | a group for the widened scan (`find_or_add_group`), and in each original scan group a candidate `Project(widened)` | #1203 2a-5 (#1222) | +| 3 | Summary inputs | `factor_summary_inputs` ([`summary_inputs.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/summary_inputs.rs)) | a value projection or filter below a summary build | that projection or filter as its own group; the build's candidate reads it | #1203 2a-2 (#1218); after 2a, Pass 1 already makes it a group | +| 4 | Top-k reuse | `add_topk_candidates` ([`topk.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/topk.rs)) | top-k groups over one ranking with different `k` | a candidate of each top-k group reading the largest-`k` group, with an exact-input requirement | #1203 2a-3 (#1220) | +| 5 | Finer aggregate | `add_finer_aggregate_candidates` ([`finer_aggregate.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/finer_aggregate.rs)) | a SQL aggregate whose grouping is coarser than another aggregate group's | a candidate of the coarse group: filter, then merge the finer group's rows with each measure's rollup | #1203 2a-3 (#1220) | +| 6 | Rollup | `add_rollup_candidates` ([`rollup.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/rollup.rs)) | a coarse aggregate or summary group and a finer group whose state rolls up | a candidate of the coarse group reading the finer group (exact or sketch merge) | #1203 2a-4 (#1221) | +| 7 | Window rollup | `add_window_rollup_candidates` (same) | a longer-range group whose windows a shorter-range group's state covers | a candidate of the longer-range group merging the shorter one's windows | #1203 2a-4 (#1221) | +| 8 | Summary results | `factor_summary_results` ([`summary_results.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/summary_results.rs)) | a summary readout inside a composite candidate | the readout as its own group | #1203 2a-2 (#1218) makes the readout a group in Pass 1; the rule is then deleted | +| 9 | Shared configurations | `add_state_configuration_candidates` ([`shared_configuration_readers.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/shared_configuration_readers.rs)) | readers that could share a summary at a size or window horizon that covers them | reader copies at the shared size (`SharedBuild` origin) | deleted in #1203 2b: the state group already holds every $\boldsymbol\theta$, and $y$ chooses per reader | +| 10 | Shared summary state groups | `add_shared_summary_state_groups` ([`state_groups.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/state_groups.rs)) | summary builds directly over input groups | per-size concrete state groups with reader copies | deleted in #1203 2b and #1160: one state group per state family | +| — | Logical edges (#1201) | after the last rule | every candidate $c$ and input $i$ | one `LogicalEdge { producer, consumer, input, satisfies_logical_requirement }` per $u \in G_{c,i}$, with $\alpha^{\log}_{u,c,i}$ = `satisfies(p^log_u, r^log_{c,i})`; output `LogicalASAPMemoCandidates { memo, edges }` | #1201 | + +After #1203 2b, every Pass 2 rule has the same shape: match groups, then `find_or_add_group` for any new intermediate result and `add_candidate` for the new way to compute an existing group. A rule adds no candidate a group already has (same expression and $p^{\log}$). + +### Stage 2: physical expansion and edges + +Stage 2 adds no logical candidates. It expands each logical candidate into physical candidates, then builds $\mathcal E$. + +| Step | Code | Reads | Adds | +|---|---|---|---| +| Physical expansion | `enumerate_candidate_configurations` ([`configurations.rs`](../../../crates/physical-asap-optimization/src/catalog/configurations.rs)) | each `CandidateForMemoGroup`, deployment capabilities, readers' $\mathcal T_q$ | one `PhysicalCandidate { logical, properties, input_requirements }` per valid $p^{\mathrm{phys}} = (\tau, \ell, \sigma)$ that the deployment allows (section 2, levels 4–6), with one `PhysicalRequirement` per input: the reader's timing, the evaluation it reads from, and its schedule | +| Physical edges (#1201) | after expansion | each `LogicalEdge` | one `PhysicalEdge { producer, consumer, input, satisfies_logical_requirement, satisfies_physical_requirement }` per pair of the two logical candidates' physical candidates, with $\alpha^{\mathrm{phys}}$ = `satisfies_physical(p^phys_u, r^phys_{c,i})`; the admissible ones ($\alpha^{\log} = \alpha^{\mathrm{phys}} = 1$) through `admissible_edges_into` / `admissible_edges_out_of` | + +Example: v3 (`k = 345`) becomes, among others, (QueryTime, None), (QueryTime, OneEvaluation, Periodic(10 s, 0)) and (IngestionTime, AcrossEvaluations, grid phase 0). G3's estimate at query time has an admissible edge to each of them: same timing, or a query-time reader of an ingestion-time result kept across evaluations. + +### Stage 3: no rules + +Stage 3 adds no candidates and no edges. It reads $\mathcal C$ and the admissible part of $\mathcal E$, creates the MILP variables of section 3 over them, and decodes $\{x = 1\}$ and $\{y = 1\}$ into `DagOfSelectedCandidates`. From 3fb403972dcedbecfa656fcd35797a3be1ecd8df Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 20:29:23 +0000 Subject: [PATCH 07/67] docs(concepts): error propagation function per consumer; validity and pruning caps Co-Authored-By: Claude Opus 5.5 --- .../memo-groups-properties-requirements.md | 51 ++++++++++++++++--- 1 file changed, 45 insertions(+), 6 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 602dd1d22..d29e6a1d4 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -30,6 +30,9 @@ This page defines the memo's data structures in the terms of the optimization pr | $w_v$ | fixed cost: $C_{\text{read}}(v) + C_{\text{compute}}(v)$ plus $C^{\text{one}}_{\text{retain}}(v)$ or $C^{\text{across}}_{\text{retain}}(v)$ by $\ell_v$ | | $\Pi$, $V_\Pi$ | selected physical ASAP DAG and its node set; $\operatorname{Cost}(\Pi) = \sum_{v \in \mathcal C} w_v x_v$ | | **(#1207)** $\kappa$ | execution context of a candidate with $\ell_v$ = None: a query evaluation $\kappa_q$, an ingestion pipeline $I_r$, or a materialized candidate's computation $\kappa_m$ | +| $g_c$ | error propagation function of consumer $c$: $\varepsilon_c \le g_c(\varepsilon_{c,1}, \dots, \varepsilon_{c,n})$ | +| $\lambda_{q,c,i}$ | slope of the composed propagation functions from input $i$ of $c$ to query $q$'s result | +| $\text{cap}_{c,i}$ | error cap of input $i$ of $c$: the smaller of $g_c$'s validity cap and the pruning cap | | **(#1207)** (K), (R), (L), budgets | accuracy at each query root, $\sum_u \lambda_{q,u}\varepsilon_u a_{q,u} \le \varepsilon_q$ and $\sum_u \delta_u a_{q,u} \le \delta_q$; retention $h_v$; latency; resource budgets | ## Example @@ -46,7 +49,7 @@ G2 KLL state over G1 one group, whatever θ G3 = G_A SummaryEstimate[q=0.95](G2) G4 SummaryEstimate[q=0.99](G2) G5 SummaryEstimate[q=0.50](G2) -G6 = G_B Divide(G4, G5) r^log on each input: error cap (#1207) +G6 = G_B Divide(G4, G5) r^log on each input: cap = min(validity cap of /, ε_B / λ) ``` The ε values are illustrative. @@ -94,10 +97,39 @@ Each input $i$ of consumer $c$ names a producer group $G_{c,i}$ and a property r The workload's accuracy target $A_q$ is not an input requirement **(#1207)**: a per-input bound would fix each input's share of the query's budget. Row (K) bounds each query's composed error at its root instead. What stays in $r^{\log}$ is what depends only on the consumer. +#### Error propagation and the error cap + +Each consumer $c$ has an error propagation function $g_c$ that bounds its output error by its input errors: + +$$\varepsilon_c \le g_c(\varepsilon_{c,1}, \dots, \varepsilon_{c,n}).$$ + +Composing them from each query's root down gives the query's error, and the accuracy target $A_q$ is $g_q(\dots) \le \varepsilon_q$ at the root. + +| Consumer | $g_c$ | Linear | +|---|---|---| +| single readout, `Project`, filter | $\varepsilon_{c,1}$ | yes | +| `sum by` over $n$ series (absolute error) | $n\,\varepsilon_{c,1}$ | yes | +| $X + Y$, $X - Y$ (absolute error) | $\varepsilon_X + \varepsilon_Y$ | yes | +| $X / Y$ (relative error) | $\dfrac{\varepsilon_X + \varepsilon_Y}{1 - \varepsilon_Y}$ | no | + +Row (K) is the composed $g$ made linear: $\lambda_{q,c,i}$ is its slope from input $i$ of $c$ to $q$'s result, so the weights sit on $(q, c, i)$. Failure probabilities add by a union bound and need no $g$. + +From $g_c$ come two caps on an input's error, both checked per edge as part of $r^{\log}_{c,i}$: + +1. **Validity cap: correctness, non-linear consumers only.** A non-linear $g_c$ is bounded by its linear form only in a region, e.g. $\varepsilon_Y \le$ cap for a ratio, where $\frac{\varepsilon_X+\varepsilon_Y}{1-\varepsilon_Y} \le \frac{1}{1-\text{cap}}(\varepsilon_X+\varepsilon_Y)$, so $\lambda = \frac{1}{1-\text{cap}}$ (1.11 for cap 0.1). Without the cap, row (K) could accept a plan whose true error is unbounded. A linear $g_c$ is exact everywhere and has no validity cap. +2. **Pruning cap: efficiency only, every consumer, linear ones included.** An input whose error alone breaks every target it could serve is in no feasible plan: $\lambda_{q,c,i}\,\varepsilon_u > \varepsilon_q$ for every query $q$ that may reach $c$. Removing such edges removes no feasible plan, since row (K) rejects them anyway; it keeps them out of $\mathcal E$ and the MILP. It is the per-edge form of Pass 1's "drop a $\boldsymbol\theta$ whose $\varepsilon$ alone exceeds every reading query's target", and prunes more because it uses each reader's own $\lambda$. + +$$\text{cap}_{c,i} = \min\Bigl(\text{validity cap of } g_c,\ \max_{q \text{ that may reach } c} \frac{\varepsilon_q}{\lambda_{q,c,i}}\Bigr), \qquad \alpha^{\log}_{u,c,i} = 0 \text{ when } \varepsilon_u > \text{cap}_{c,i}.$$ + +For a monotone size parameter, $\varepsilon_u \le \text{cap}_{c,i}$ is a minimum size, e.g. $k_u \ge k_{\text{required}}$ for KLL. + +**What stays at the root.** A per-input cap cannot replace row (K): with several inputs, $\varepsilon_X + \varepsilon_Y \le \varepsilon_q$ couples them, and one cap per input would fix each input's share in advance (the rejected budget shares). The caps only prune; row (K) decides. + + | Requirement | Stated by | Values | Constrains | Checked by | |---|---|---|---|---| | Accuracy target $A_q$ | the workload, per query | metric: absolute (in the value's units, or times the stream's L1 or L2 norm), relative, rank; bound $\varepsilon_q$; failure probability $\delta_q$; scope: per value, whole result, or groups above a share of the records | $\boldsymbol\theta_u$ of every approximate candidate $q$ depends on, through $(\varepsilon_u, \delta_u)$ | row (K) **(#1207)** | -| Error cap, in $r^{\log}_{c,i}$ | a non-linear consumer (ratio, product), per input | a cap on $\varepsilon_u$; for a monotone size parameter, a minimum size $k_u \ge k_{\text{required}}$ | $\boldsymbol\theta_u$ | $\alpha^{\log}$, row (G): row (K)'s linear bound holds only below the cap | +| Error cap $\text{cap}_{c,i}$, in $r^{\log}_{c,i}$ | every consumer, per input, from $g_c$ and the targets of the queries that may reach it | the smaller of the validity cap (non-linear $g_c$ only) and the pruning cap; for a monotone size parameter, a minimum size $k_u \ge k_{\text{required}}$ | $\boldsymbol\theta_u$ | $\alpha^{\log}$, row (G): validity cap for correctness of (K), pruning cap for efficiency | | Window, in $r^{\log}_{c,i}$ **(#1207)** | a consumer that reads windows, per input | the reader's range and step | $\omega_u$: the width must divide both | $\alpha^{\log}$, row (G) | | Summary family, in $r^{\log}_{c,i}$ | a consumer that reads summary state | a family | usually already fixed by $G_{c,i}$, whose $\operatorname{op}$ names the family | $\alpha^{\log}$, row (G) | | Reader timing and evaluation, in $r^{\mathrm{phys}}_{c,i}$ | the consumer, per input | when it reads (ingestion or query time) and from which evaluation (same or later) | $\tau_u$, $\ell_u$ | $\alpha^{\mathrm{phys}}$, row (H) | @@ -111,7 +143,7 @@ How requirements map to properties: ``` A_q (workload) ──► θ_u of every approximate input (through ε_u, δ_u; row (K)) -error cap r^log ──► θ_u +error cap r^log ──► θ_u (validity cap from g_c; pruning cap from ε_q / λ_{q,c,i}) window r^log ──► ω_u reader timing r^phys ──► τ_u, ℓ_u reader schedule r^phys ──► σ_u @@ -264,12 +296,19 @@ pub enum WindowLayout { Tumbling { width_ms: u64, count: u32 } } pub struct LogicalRequirement { pub family: Option, pub window: Option, // reader's range and step: a width must divide both - pub error_cap: Option, // validity of row (K)'s linear bound + pub error_cap: Option, // cap_{c,i}: min(validity cap of g_c, pruning cap) } /// α^log_{u,c,i} pub fn satisfies(producer: &LogicalProperties, requirement: &LogicalRequirement) -> bool; + +/// g_c linearized: λ per input and the validity cap (None when g_c is linear). +/// New abstraction, not yet in #1207: pending approval. +pub struct ErrorPropagation { pub slope_per_input: Vec, pub validity_cap: Option } +pub fn error_propagation(consumer: &Operator) -> ErrorPropagation; ``` +The validity cap is known when the consumer is created. The pruning cap needs $\lambda_{q,c,i}$ and the queries that may reach $c$, so it is filled in when Pass 2 builds the logical edges. + Deleted: `root_node` with private nodes, `input_groups` (the expression's children), `accuracy_budget_parts`, the `SharedBuild` and `SummaryResult` origins. ### Stage 1, Pass 2: rules and logical edges @@ -435,7 +474,7 @@ Pass 1 adds, for each group a query computes, every way to compute it. Each inte |---|---|---| | Exact | [`per_computation_candidates.rs`](../../../crates/logical-asap-optimization/src/pass1/per_computation_candidates.rs), [`memo_population.rs`](../../../crates/logical-asap-optimization/src/pass1/memo_population.rs) | a candidate with the exact operator; $\boldsymbol\theta_v$ none, $(\varepsilon_v, \delta_v) = (0, 0)$ | | Exact accumulator | same | a state group `ExactAggregate(kind)` over the input group, one candidate; a `FinalizeExactAccumulator` candidate in the result's group reading it | -| Summary, per family the executor supports | same | a state group `SummaryAgg[family]` over the input group, with **one candidate per $\boldsymbol\theta$ in the family's predefined list**, dropping a $\boldsymbol\theta$ whose $\varepsilon_v$ alone exceeds every reading query's $\varepsilon_q$; a `SummaryEstimate[statistic]` candidate in the result's group reading the state group, with $r^{\log}$ = error cap where the consumer is non-linear | +| Summary, per family the executor supports | same | a state group `SummaryAgg[family]` over the input group, with **one candidate per $\boldsymbol\theta$ in the family's predefined list**, dropping a $\boldsymbol\theta$ whose $\varepsilon_v$ alone exceeds every reading query's $\varepsilon_q$; a `SummaryEstimate[statistic]` candidate in the result's group reading the state group. Each consumer's `error_cap` starts as its $g_c$'s validity cap (none for a linear $g_c$) | | Fused build (#1203 2c) | same | a build that computes several targets at once: a candidate of the outer target's state group, with `fused_groups` naming the others | Example: for A and B, Pass 1 adds G2 = `SummaryAgg[KLL](G1)` with candidates `k = 200` and `k = 345`, and G3, G4, G5 = `SummaryEstimate[q](G2)`. A and B's three estimates read one state group because their expressions over G1 hash-cons to one group. @@ -456,7 +495,7 @@ Today, Pass 1 instead sizes one $\boldsymbol\theta$ per `accuracy_budget_parts` | 8 | Summary results | `factor_summary_results` ([`summary_results.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/summary_results.rs)) | a summary readout inside a composite candidate | the readout as its own group | #1203 2a-2 (#1218) makes the readout a group in Pass 1; the rule is then deleted | | 9 | Shared configurations | `add_state_configuration_candidates` ([`shared_configuration_readers.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/shared_configuration_readers.rs)) | readers that could share a summary at a size or window horizon that covers them | reader copies at the shared size (`SharedBuild` origin) | deleted in #1203 2b: the state group already holds every $\boldsymbol\theta$, and $y$ chooses per reader | | 10 | Shared summary state groups | `add_shared_summary_state_groups` ([`state_groups.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/state_groups.rs)) | summary builds directly over input groups | per-size concrete state groups with reader copies | deleted in #1203 2b and #1160: one state group per state family | -| — | Logical edges (#1201) | after the last rule | every candidate $c$ and input $i$ | one `LogicalEdge { producer, consumer, input, satisfies_logical_requirement }` per $u \in G_{c,i}$, with $\alpha^{\log}_{u,c,i}$ = `satisfies(p^log_u, r^log_{c,i})`; output `LogicalASAPMemoCandidates { memo, edges }` | #1201 | +| — | Logical edges (#1201) | after the last rule | every candidate $c$ and input $i$ | one `LogicalEdge { producer, consumer, input, satisfies_logical_requirement }` per $u \in G_{c,i}$, after tightening each `error_cap` with the pruning cap $\max_q \varepsilon_q / \lambda_{q,c,i}$; $\alpha^{\log}_{u,c,i}$ = `satisfies(p^log_u, r^log_{c,i})`; output `LogicalASAPMemoCandidates { memo, edges }` | #1201 | After #1203 2b, every Pass 2 rule has the same shape: match groups, then `find_or_add_group` for any new intermediate result and `add_candidate` for the new way to compute an existing group. A rule adds no candidate a group already has (same expression and $p^{\log}$). From 730be23bbfa0b1cb76270deb58947c538b19bd1d Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 20:31:18 +0000 Subject: [PATCH 08/67] docs(concepts): ErrorPropagation approved Co-Authored-By: Claude Opus 5.5 --- .../design_docs/concepts/memo-groups-properties-requirements.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index d29e6a1d4..04d7f1e8a 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -302,7 +302,7 @@ pub struct LogicalRequirement { pub fn satisfies(producer: &LogicalProperties, requirement: &LogicalRequirement) -> bool; /// g_c linearized: λ per input and the validity cap (None when g_c is linear). -/// New abstraction, not yet in #1207: pending approval. +/// Approved 2026-11-05. pub struct ErrorPropagation { pub slope_per_input: Vec, pub validity_cap: Option } pub fn error_propagation(consumer: &Operator) -> ErrorPropagation; ``` From d6aea8a5cb1bf7eb2b6ca3a9fceead878397c4f3 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 20:33:54 +0000 Subject: [PATCH 09/67] =?UTF-8?q?docs(concepts):=20=CE=BB=20is=20a=20propa?= =?UTF-8?q?gation=20factor=20read=20from=20the=20query=20expression?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- .../memo-groups-properties-requirements.md | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 04d7f1e8a..9d4b5d34e 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -31,7 +31,7 @@ This page defines the memo's data structures in the terms of the optimization pr | $\Pi$, $V_\Pi$ | selected physical ASAP DAG and its node set; $\operatorname{Cost}(\Pi) = \sum_{v \in \mathcal C} w_v x_v$ | | **(#1207)** $\kappa$ | execution context of a candidate with $\ell_v$ = None: a query evaluation $\kappa_q$, an ingestion pipeline $I_r$, or a materialized candidate's computation $\kappa_m$ | | $g_c$ | error propagation function of consumer $c$: $\varepsilon_c \le g_c(\varepsilon_{c,1}, \dots, \varepsilon_{c,n})$ | -| $\lambda_{q,c,i}$ | slope of the composed propagation functions from input $i$ of $c$ to query $q$'s result | +| $\lambda_{q,c,i}$ | propagation factor of input $i$ of $c$ to query $q$'s result, read off $q$'s expression | | $\text{cap}_{c,i}$ | error cap of input $i$ of $c$: the smaller of $g_c$'s validity cap and the pruning cap | | **(#1207)** (K), (R), (L), budgets | accuracy at each query root, $\sum_u \lambda_{q,u}\varepsilon_u a_{q,u} \le \varepsilon_q$ and $\sum_u \delta_u a_{q,u} \le \delta_q$; retention $h_v$; latency; resource budgets | @@ -112,11 +112,15 @@ Composing them from each query's root down gives the query's error, and the accu | $X + Y$, $X - Y$ (absolute error) | $\varepsilon_X + \varepsilon_Y$ | yes | | $X / Y$ (relative error) | $\dfrac{\varepsilon_X + \varepsilon_Y}{1 - \varepsilon_Y}$ | no | -Row (K) is the composed $g$ made linear: $\lambda_{q,c,i}$ is its slope from input $i$ of $c$ to $q$'s result, so the weights sit on $(q, c, i)$. Failure probabilities add by a union bound and need no $g$. +Each $g_c$ is a sum of its inputs' errors, each times a constant **propagation factor** $\lambda_{c,i}$: $g_c = \sum_i \lambda_{c,i}\,\varepsilon_{c,i}$ (for a non-linear $g_c$, within its validity cap, below). The factors come from the query expression itself: each operator and its arguments, e.g. `sum by` over $n$ series has factor $n$. No measurement or optimization is involved. A query's factor for input $i$ of $c$ multiplies the factors along the path in $q$'s expression from that input to $q$'s root, and adds over paths when the input reaches the root more than once: + +$$\lambda_{q,c,i} = \sum_{\text{paths } c = c_0 \to c_1 \to \dots \to c_m = \text{root of } q} \lambda_{c_0,i}\,\lambda_{c_1,i_1} \cdots \lambda_{c_m,i_m}.$$ + +Row (K) is $\sum \lambda_{q,c,i}\,\varepsilon_u\,y \le \varepsilon_q$ with these factors, so the weights sit on $(q, c, i)$. Two queries reading one group through different expressions have different factors. Failure probabilities add by a union bound and need no factor. From $g_c$ come two caps on an input's error, both checked per edge as part of $r^{\log}_{c,i}$: -1. **Validity cap: correctness, non-linear consumers only.** A non-linear $g_c$ is bounded by its linear form only in a region, e.g. $\varepsilon_Y \le$ cap for a ratio, where $\frac{\varepsilon_X+\varepsilon_Y}{1-\varepsilon_Y} \le \frac{1}{1-\text{cap}}(\varepsilon_X+\varepsilon_Y)$, so $\lambda = \frac{1}{1-\text{cap}}$ (1.11 for cap 0.1). Without the cap, row (K) could accept a plan whose true error is unbounded. A linear $g_c$ is exact everywhere and has no validity cap. +1. **Validity cap: correctness, non-linear consumers only.** A non-linear $g_c$ is bounded by its linear form only in a region, e.g. $\varepsilon_Y \le$ cap for a ratio, where $\frac{\varepsilon_X+\varepsilon_Y}{1-\varepsilon_Y} \le \frac{1}{1-\text{cap}}(\varepsilon_X+\varepsilon_Y)$, so both factors are $\frac{1}{1-\text{cap}}$ (1.11 for cap 0.1). Without the cap, row (K) could accept a plan whose true error is unbounded. A linear $g_c$ is exact everywhere and has no validity cap. 2. **Pruning cap: efficiency only, every consumer, linear ones included.** An input whose error alone breaks every target it could serve is in no feasible plan: $\lambda_{q,c,i}\,\varepsilon_u > \varepsilon_q$ for every query $q$ that may reach $c$. Removing such edges removes no feasible plan, since row (K) rejects them anyway; it keeps them out of $\mathcal E$ and the MILP. It is the per-edge form of Pass 1's "drop a $\boldsymbol\theta$ whose $\varepsilon$ alone exceeds every reading query's target", and prunes more because it uses each reader's own $\lambda$. $$\text{cap}_{c,i} = \min\Bigl(\text{validity cap of } g_c,\ \max_{q \text{ that may reach } c} \frac{\varepsilon_q}{\lambda_{q,c,i}}\Bigr), \qquad \alpha^{\log}_{u,c,i} = 0 \text{ when } \varepsilon_u > \text{cap}_{c,i}.$$ @@ -301,9 +305,10 @@ pub struct LogicalRequirement { /// α^log_{u,c,i} pub fn satisfies(producer: &LogicalProperties, requirement: &LogicalRequirement) -> bool; -/// g_c linearized: λ per input and the validity cap (None when g_c is linear). +/// g_c as propagation factors λ_{c,i}, read from the consumer's operator and +/// arguments, and the validity cap (None when g_c is linear). /// Approved 2026-11-05. -pub struct ErrorPropagation { pub slope_per_input: Vec, pub validity_cap: Option } +pub struct ErrorPropagation { pub factor_per_input: Vec, pub validity_cap: Option } pub fn error_propagation(consumer: &Operator) -> ErrorPropagation; ``` From 8b9c74d75e2d6851c3f3bc7e08d300e2cfd8f302 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 20:40:09 +0000 Subject: [PATCH 10/67] docs(concepts): keep the candidate's origin, listed as not in the formulation --- .../concepts/memo-groups-properties-requirements.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 9d4b5d34e..5843e188c 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -281,6 +281,9 @@ pub struct CandidateForMemoGroup { pub logical_properties: LogicalProperties, // p_v^log pub input_requirements: Vec, // r^log_{c,i}, one per input group pub fused_groups: Vec, + /// Which rule added it. Not in the formulation: kept for `stage_pipeline --text` + /// and the viewer (approved 2026-11-05). + pub origin: OriginOfCandidateForMemoGroup, } /// e_v = op_v(G_{v,1}, …, G_{v,n}): one operator, its children are groups. pub struct CandidateExpression { pub operator: Operator } @@ -316,6 +319,8 @@ The validity cap is known when the consumer is created. The pruning cap needs $\ Deleted: `root_node` with private nodes, `input_groups` (the expression's children), `accuracy_budget_parts`, the `SharedBuild` and `SummaryResult` origins. +Concepts not in the formulation: `origin: OriginOfCandidateForMemoGroup` (Pass1, SummaryInput, TumblingWindows, Rollup, TopKReuse, WidenedScans) records which rule added a candidate; nothing in planning reads it. + ### Stage 1, Pass 2: rules and logical edges Rules add candidates to existing groups, or add groups. A rule never copies a reader. After the last rule, Pass 2 builds the logical edges once. From a1c1ecdf2ec412aef3c69eabdefe2d64776949a3 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 20:55:52 +0000 Subject: [PATCH 11/67] docs(concepts): consumer and query/context instead of reader; PhysicalRequirement.consumer_{timing,schedule} Co-Authored-By: Claude Opus 5.5 --- .../memo-groups-properties-requirements.md | 66 ++++++++++--------- 1 file changed, 35 insertions(+), 31 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 5843e188c..fff471115 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -16,7 +16,7 @@ This page defines the memo's data structures in the terms of the optimization pr | $p^{\log}_v = \boldsymbol\theta_v$ | summary algorithm parameters, e.g. sketch size; **(#1207)** also the window layout $\omega_v$ | | $p^{\mathrm{phys}}_v = (\tau_v, \ell_v)$ | execution timing and lifetime; **(#1207)** also the schedule $\sigma_v$ when $\ell_v \ne$ None | | $(\varepsilon_v, \delta_v)$ | error bound and failure probability of $\boldsymbol\theta_v$ for the statistic $\operatorname{op}_v$ reads; derived, not a property | -| $c \in G_c$, $u \in G_u$ | consumer candidate and its group; producer candidate and its group | +| $c \in G_c$, $u \in G_u$ | consumer candidate and its group; producer candidate and its group. This page says *consumer* for $c$ and *query* $q$ (or its execution context $\kappa$) for who needs a result on a schedule; it does not use "reader" | | $\mathcal I(c)$ | input indices of consumer $c$ | | $G_{c,i}$ | memo group required by input $i$ of consumer $c$ | | $r_{c,i} = (r^{\log}_{c,i}, r^{\mathrm{phys}}_{c,i})$ | property requirement of input $i$ of consumer $c$ | @@ -69,12 +69,12 @@ What is computed is the expression $e_v$, not a property: the operator, its argu | Dimension | Values | Part of | Code | |---|---|---|---| | Parameter configuration $\boldsymbol\theta_v$ | per family, from a predefined list: KLL `k`; DDSketch `alpha`; CMS / CountSketch `width × depth`; CMS / CountSketch with heap `width × depth × heap_size`; UnivMon `heap_size × rows × cols × layers`; HLL `precision`; KMV `k`. None for an exact operator | $p^{\log}_v$ | `SketchParams` inside `SketchKind` ([`sketch.rs`](../../../crates/types/src/ir/schema/sketch.rs)), sized by `size_params` per `accuracy_budget_parts` today | -| **(#1207)** Window layout $\omega_v$ | none; or tumbling windows of width $w$ and count $n$, with $w$ from the gcd closure of the readers' ranges and steps. Later: sliding windows by merge or by subtraction | $p^{\log}_v$ | `SummaryTumblingWindowChain` arguments, added by Pass 2 ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs)) | +| **(#1207)** Window layout $\omega_v$ | none; or tumbling windows of width $w$ and count $n$, with $w$ from the gcd closure of the ranges and steps of the queries that reach the group. Later: sliding windows by merge or by subtraction | $p^{\log}_v$ | `SummaryTumblingWindowChain` arguments, added by Pass 2 ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs)) | | Execution timing $\tau_v$ | IngestionTime, QueryTime | $p^{\mathrm{phys}}_v$ | `ExecutionTiming` ([`execution_data_state.rs`](../../../crates/types/src/ir/properties/execution_data_state.rs)) | | Lifetime $\ell_v$ | None, OneEvaluation, AcrossEvaluations; **(#1207)** a materialized lifetime carries its medium, Memory or Disk | $p^{\mathrm{phys}}_v$ | `Materialization` on main; `Lifetime`, `Medium` in #1214 ([`physical/`](../../../crates/types/src/physical)) | | **(#1207)** Schedule $\sigma_v$ | only when $\ell_v \ne$ None: Periodic (interval, phase) for OneEvaluation; tumbling-window grid phase for AcrossEvaluations | $p^{\mathrm{phys}}_v$ | schedule groups `z[m, G]` in the MILP today ([`cost_terms.rs`](../../../crates/plan-selection/src/milp/cost_terms.rs)); a property in #1200 PR 2 | -Lifetime None means the output is discarded after its reader's execution completes. OneEvaluation retains the result until all readers within one evaluation finish. AcrossEvaluations retains it throughout the evaluation period, so later evaluations reuse it. A candidate is materialized when $\ell_v \ne$ None; this is derived, not stored. +Lifetime None means the output is discarded after its consumer's execution completes. OneEvaluation retains the result until all its consumers within one evaluation finish. AcrossEvaluations retains it throughout the evaluation period, so later evaluations reuse it. A candidate is materialized when $\ell_v \ne$ None; this is derived, not stored. The error bound $(\varepsilon_v, \delta_v)$ is not a dimension: it is a function of $\boldsymbol\theta_v$ and the statistic $\operatorname{op}_v$ reads, derived when needed (`local_guarantee` in [`estimators`](../../../crates/accuracy/src/estimators/mod.rs)). An exact candidate has $(0, 0)$. @@ -83,7 +83,7 @@ Invalid combinations of $p^{\mathrm{phys}}$, rejected by `PhysicalProperties::ne | Combination | Why | |---|---| | (IngestionTime, OneEvaluation) | there is no evaluation at ingestion time | -| $\ell$ = None with a schedule | where an unmaterialized result runs is its reader's execution context $\kappa$, not a property | +| $\ell$ = None with a schedule | where an unmaterialized result runs is its consumer's execution context $\kappa$, not a property | | $\ell \ne$ None without a schedule | a materialized result must say on which times or grid it is delivered | | OneEvaluation with a grid, AcrossEvaluations with a periodic schedule | each lifetime has one kind of schedule | | a phase at or above the grid width or interval, or interval 0 | the phase is taken modulo the width or interval | @@ -121,7 +121,7 @@ Row (K) is $\sum \lambda_{q,c,i}\,\varepsilon_u\,y \le \varepsilon_q$ with these From $g_c$ come two caps on an input's error, both checked per edge as part of $r^{\log}_{c,i}$: 1. **Validity cap: correctness, non-linear consumers only.** A non-linear $g_c$ is bounded by its linear form only in a region, e.g. $\varepsilon_Y \le$ cap for a ratio, where $\frac{\varepsilon_X+\varepsilon_Y}{1-\varepsilon_Y} \le \frac{1}{1-\text{cap}}(\varepsilon_X+\varepsilon_Y)$, so both factors are $\frac{1}{1-\text{cap}}$ (1.11 for cap 0.1). Without the cap, row (K) could accept a plan whose true error is unbounded. A linear $g_c$ is exact everywhere and has no validity cap. -2. **Pruning cap: efficiency only, every consumer, linear ones included.** An input whose error alone breaks every target it could serve is in no feasible plan: $\lambda_{q,c,i}\,\varepsilon_u > \varepsilon_q$ for every query $q$ that may reach $c$. Removing such edges removes no feasible plan, since row (K) rejects them anyway; it keeps them out of $\mathcal E$ and the MILP. It is the per-edge form of Pass 1's "drop a $\boldsymbol\theta$ whose $\varepsilon$ alone exceeds every reading query's target", and prunes more because it uses each reader's own $\lambda$. +2. **Pruning cap: efficiency only, every consumer, linear ones included.** An input whose error alone breaks every target it could serve is in no feasible plan: $\lambda_{q,c,i}\,\varepsilon_u > \varepsilon_q$ for every query $q$ that may reach $c$. Removing such edges removes no feasible plan, since row (K) rejects them anyway; it keeps them out of $\mathcal E$ and the MILP. It is the per-edge form of Pass 1's "drop a $\boldsymbol\theta$ whose $\varepsilon$ alone exceeds every reading query's target", and prunes more because it uses each consumer's own $\lambda$. $$\text{cap}_{c,i} = \min\Bigl(\text{validity cap of } g_c,\ \max_{q \text{ that may reach } c} \frac{\varepsilon_q}{\lambda_{q,c,i}}\Bigr), \qquad \alpha^{\log}_{u,c,i} = 0 \text{ when } \varepsilon_u > \text{cap}_{c,i}.$$ @@ -134,10 +134,10 @@ For a monotone size parameter, $\varepsilon_u \le \text{cap}_{c,i}$ is a minimum |---|---|---|---|---| | Accuracy target $A_q$ | the workload, per query | metric: absolute (in the value's units, or times the stream's L1 or L2 norm), relative, rank; bound $\varepsilon_q$; failure probability $\delta_q$; scope: per value, whole result, or groups above a share of the records | $\boldsymbol\theta_u$ of every approximate candidate $q$ depends on, through $(\varepsilon_u, \delta_u)$ | row (K) **(#1207)** | | Error cap $\text{cap}_{c,i}$, in $r^{\log}_{c,i}$ | every consumer, per input, from $g_c$ and the targets of the queries that may reach it | the smaller of the validity cap (non-linear $g_c$ only) and the pruning cap; for a monotone size parameter, a minimum size $k_u \ge k_{\text{required}}$ | $\boldsymbol\theta_u$ | $\alpha^{\log}$, row (G): validity cap for correctness of (K), pruning cap for efficiency | -| Window, in $r^{\log}_{c,i}$ **(#1207)** | a consumer that reads windows, per input | the reader's range and step | $\omega_u$: the width must divide both | $\alpha^{\log}$, row (G) | +| Window, in $r^{\log}_{c,i}$ **(#1207)** | a consumer that reads windows, per input | the range and step of the queries that reach the consumer | $\omega_u$: the width must divide both | $\alpha^{\log}$, row (G) | | Summary family, in $r^{\log}_{c,i}$ | a consumer that reads summary state | a family | usually already fixed by $G_{c,i}$, whose $\operatorname{op}$ names the family | $\alpha^{\log}$, row (G) | -| Reader timing and evaluation, in $r^{\mathrm{phys}}_{c,i}$ | the consumer, per input | when it reads (ingestion or query time) and from which evaluation (same or later) | $\tau_u$, $\ell_u$ | $\alpha^{\mathrm{phys}}$, row (H) | -| Reader schedule, in $r^{\mathrm{phys}}_{c,i}$ **(#1207)** | a periodic consumer, from $\mathcal T_q$ | its interval and phase $(I_c, \varphi_c)$ | $\sigma_u$ | $\alpha^{\mathrm{phys}}$, row (H) | +| Consumer timing and evaluation, in $r^{\mathrm{phys}}_{c,i}$ | the consumer, per input | when it reads (ingestion or query time) and from which evaluation (same or later) | $\tau_u$, $\ell_u$ | $\alpha^{\mathrm{phys}}$, row (H) | +| Consumer schedule, in $r^{\mathrm{phys}}_{c,i}$ **(#1207)** | the consumer's execution context $\kappa$, from the $\mathcal T_q$ of its query | the context's interval and phase $(I_\kappa, \varphi_\kappa)$; the producer must deliver on them, $T_\kappa \subseteq T_{\sigma_u}$ | $\sigma_u$ | $\alpha^{\mathrm{phys}}$, row (H) | | Deployment capability | the deployment and data | the source has an ingestion pipeline; the executor supports the family; query-time results may be kept across evaluations | $\tau$, $\operatorname{op}$, (QueryTime, AcrossEvaluations) | candidate generation: such candidates are not in $\mathcal C$ | | Latency, resource budgets **(#1207)** | the workload | per-query latency bound; memory and CPU budgets | the whole plan | rows (L) and budgets | @@ -149,8 +149,8 @@ How requirements map to properties: A_q (workload) ──► θ_u of every approximate input (through ε_u, δ_u; row (K)) error cap r^log ──► θ_u (validity cap from g_c; pruning cap from ε_q / λ_{q,c,i}) window r^log ──► ω_u -reader timing r^phys ──► τ_u, ℓ_u -reader schedule r^phys ──► σ_u +consumer timing r^phys ──► τ_u, ℓ_u +consumer schedule r^phys ──► σ_u deployment ──► τ, op, ℓ (not generated) latency, budgets ──► every dimension, through w_v ``` @@ -168,10 +168,10 @@ flowchart TD R2 --> T1["θ: k = 200
ε = 0.008"] R2 --> T2["θ: k = 345
ε = 0.005"] - R2 --> T3["θ: k = …
(prune: ε above every
reading query's ε_q)"] + R2 --> T3["θ: k = …
(prune: ε above the ε_q of
every query reaching G)"] T2 --> W1["ω: no windows"] - T2 --> W2["ω: tumbling, width w
(prune: w must divide
some reader's range and step)"] + T2 --> W2["ω: tumbling, width w
(prune: w must divide the range and
step of some query reaching G)"] W2 --> P1["τ = QueryTime"] W2 --> P2["τ = IngestionTime
(prune: no ingestion pipeline)"] @@ -183,7 +183,7 @@ flowchart TD P2 --> L5["ℓ = AcrossEvaluations"] P2 -. invalid .-> L6["ℓ = OneEvaluation"] - L2 --> S1["σ: Periodic (I, φ)
from the gcd closure
of the readers' 𝒯_q"] + L2 --> S1["σ: Periodic (I, φ)
from the gcd closure
of the queries' 𝒯_q"] L3 --> S2["σ: grid phase φ mod w"] L5 --> S3["σ: grid phase 0"] ``` @@ -193,16 +193,16 @@ The levels, top to bottom: | Level | Decides | Stage | Values offered | Pruned when | Code | |---|---|---|---|---|---| | 1 | expression $e_v$ ($\operatorname{op}_v$ with its family) | Stage 1, Pass 1 and Pass 2 rules | each way to compute the group | the executor cannot build the family | Pass 1 ([`per_computation_candidates.rs`](../../../crates/logical-asap-optimization/src/pass1/per_computation_candidates.rs)); Pass 2 rules ([`rules/`](../../../crates/logical-asap-optimization/src/pass2/memo/rules)) | -| 2 | $\boldsymbol\theta_v$ | Stage 1, Pass 1 | the family's predefined list | $\varepsilon_v$ alone exceeds every reading query's $\varepsilon_q$ | Pass 1 sizing per `accuracy_budget_parts` ([`memo_population.rs`](../../../crates/logical-asap-optimization/src/pass1/memo_population.rs)); a predefined list in #1203 PR 2 | -| 3 | $\omega_v$ **(#1207)** | Stage 1, Pass 2 | tumbling widths from the gcd closure of readers' ranges and steps | no reader's range and step is a multiple of the width | Pass 2 ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs), [`state_groups.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/state_groups.rs)) | +| 2 | $\boldsymbol\theta_v$ | Stage 1, Pass 1 | the family's predefined list | $\varepsilon_v$ alone exceeds the $\varepsilon_q$ of every query that reaches the group | Pass 1 sizing per `accuracy_budget_parts` ([`memo_population.rs`](../../../crates/logical-asap-optimization/src/pass1/memo_population.rs)); a predefined list in #1203 PR 2 | +| 3 | $\omega_v$ **(#1207)** | Stage 1, Pass 2 | tumbling widths from the gcd closure of the ranges and steps of the queries reaching the group | no such query's range and step is a multiple of the width | Pass 2 ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs), [`state_groups.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/state_groups.rs)) | | 4 | $\tau_v$ | Stage 2 | QueryTime; IngestionTime | the source has no ingestion pipeline | `enumerate_candidate_configurations` ([`configurations.rs`](../../../crates/physical-asap-optimization/src/catalog/configurations.rs)) | | 5 | $\ell_v$ | Stage 2 | None, OneEvaluation, AcrossEvaluations | invalid with $\tau_v$; the deployment keeps no query-time results across evaluations | same function, with `DeploymentCapabilities::query_time_results_retained` | -| 6 | $\sigma_v$ **(#1207)** | Stage 2 | the gcd closure $\Sigma$ of the readers' schedules $\mathcal T_q$ | only offered when $\ell_v \ne$ None | Stage 3 MILP today; Stage 2 in #1200 PR 2 | +| 6 | $\sigma_v$ **(#1207)** | Stage 2 | the gcd closure $\Sigma$ of the schedules $\mathcal T_q$ of the queries reaching the group | only offered when $\ell_v \ne$ None | Stage 3 MILP today; Stage 2 in #1200 PR 2 | What the tree does not decide, because it is a choice of the MILP over $\mathcal C$ and $\mathcal E$: - **Which producer each consumer input reads:** $y_{u,c,i}$. Two consumers of one group may read different leaves, e.g. one reads `k = 200` and another `k = 345`. - **Where an unmaterialized candidate runs (#1207):** a candidate with $\ell_v$ = None runs inside each reading context $\kappa$. -- **How much a materialized candidate retains (#1207):** $h_v$, from the selected readers' longest range, row (R). +- **How much a materialized candidate retains (#1207):** $h_v$, from the longest range among the queries its selected consumers serve, row (R). ### Order of decisions @@ -217,7 +217,7 @@ What the tree does not decide, because it is a choice of the MILP over $\mathcal Alternative orders worth comparing: - **$\tau$ first.** Where most sources have no ingestion pipeline, deciding timing at the group level prunes half the tree before $\boldsymbol\theta$ and $\omega$ multiply it. -- **$\boldsymbol\theta$ last among logical dimensions.** $\boldsymbol\theta$ has the most values. Deciding $\omega$ before $\boldsymbol\theta$ visits fewer nodes when many widths are pruned by readers' ranges. +- **$\boldsymbol\theta$ last among logical dimensions.** $\boldsymbol\theta$ has the most values. Deciding $\omega$ before $\boldsymbol\theta$ visits fewer nodes when many widths are pruned by the queries' ranges. - **Requirement-driven (top-down) vs producer-driven (bottom-up).** Top-down enumeration starts from each consumer's $r_{c,i}$ and offers only the values it accepts; bottom-up offers every value and lets $\alpha$ filter $\mathcal E$. Top-down visits fewer nodes but must union the requirements of all consumers of a group, so that sharing between consumers is not lost. ## 3. Data structures and interfaces per stage @@ -302,7 +302,7 @@ pub enum WindowLayout { Tumbling { width_ms: u64, count: u32 } } pub struct LogicalRequirement { pub family: Option, - pub window: Option, // reader's range and step: a width must divide both + pub window: Option, // range and step of the queries reaching the consumer: a width must divide both pub error_cap: Option, // cap_{c,i}: min(validity cap of g_c, pruning cap) } /// α^log_{u,c,i} @@ -323,7 +323,7 @@ Concepts not in the formulation: `origin: OriginOfCandidateForMemoGroup` (Pass1, ### Stage 1, Pass 2: rules and logical edges -Rules add candidates to existing groups, or add groups. A rule never copies a reader. After the last rule, Pass 2 builds the logical edges once. +Rules add candidates to existing groups, or add groups. A rule never copies a consumer. After the last rule, Pass 2 builds the logical edges once. ```rust // crates/logical-asap-optimization/src/pass2 @@ -369,10 +369,14 @@ impl PhysicalProperties { /// r^phys_{c,i}; `new` rejects (IngestionTime, Later). pub struct PhysicalRequirement { - reader_timing: ExecutionTiming, + consumer_timing: ExecutionTiming, reads_from: Evaluation, - reader_schedule: Option, // (I_c, φ_c) + /// (I_κ, φ_κ) of the consumer's execution context, from its query's T_q; + /// None for a non-periodic query. + consumer_schedule: Option, } +/// A periodic evaluation schedule φ + I·ℤ. +pub struct EvaluationSchedule { interval_ms: u64, phase_ms: u64 } pub enum Evaluation { Same, Later } /// α^phys_{u,c,i} pub fn satisfies_physical(producer: &PhysicalProperties, requirement: &PhysicalRequirement) -> bool; @@ -409,7 +413,7 @@ impl PhysicalASAPMemoCandidates { } ``` -Deleted: `Materialization`, `PhysicalCandidate::{timing, materialization, operator_node}`, `producer_candidates_per_input` as the interface, `compatible_timing`, `ReaderConstraint::{IncompatibleOutputs, AllowedOutputs}` (now $\alpha^{\mathrm{phys}}$ through $\sigma$), `may_keep_across_evaluations_per_sole_reader_query`. +Deleted: `Materialization`, `PhysicalCandidate::{timing, materialization, operator_node}`, `producer_candidates_per_input` as the interface, `compatible_timing`, the reader constraints `ReaderConstraint::{IncompatibleOutputs, AllowedOutputs}` (now $\alpha^{\mathrm{phys}}$ through $\sigma$), `may_keep_across_evaluations_per_sole_reader_query`. ### Stage 3: the MILP @@ -440,7 +444,7 @@ fn producer_selection_rows(..); // (B) y_{u,c,i} ≤ x_u fn query_coverage_rows(..); // (C), (D) fn acyclicity_rows(..); // (E) fn utilization_rows(..); // (F) -fn reader_constraint_rows(..); // (J) +fn consumer_constraint_rows(..); // (J): MinimumQueryReaders, HighestSharedQueryMaterialization over reach fn accuracy_rows(..); // (K) Σ λ_{q,u} ε_u a_{q,u} ≤ ε_q, Σ δ_u a_{q,u} ≤ δ_q fn retention_rows(..); // (R) h_v ≥ need_{c,i} · y_{v,c,i} fn latency_rows(..); // (L) @@ -468,7 +472,7 @@ A rule reads the memo and adds to it only through the interfaces of section 3: - `Memo::find_or_add_group(expression)` for a group, so an expression that already exists returns its group (hash-consing); - `Memo::add_candidate(group, CandidateForMemoGroup { expression, logical_properties, input_requirements, fused_groups })` for a candidate $v = (e_v, p^{\log}_v)$ with its $r^{\log}_{v,i}$. -A rule never copies a reader and never edits an existing candidate: every consumer of a group sees a new candidate of that group through $\mathcal E$. Rules match over groups and their candidates' expressions, never over a reader's private operators. The order of the rules below is the order the code runs them; it affects which groups exist when a later rule matches, not which plans are valid. +A rule never copies a consumer and never edits an existing candidate: every consumer of a group sees a new candidate of that group through $\mathcal E$. Rules match over groups and their candidates' expressions, never over a consumer's private operators. The order of the rules below is the order the code runs them; it affects which groups exist when a later rule matches, not which plans are valid. ### Stage 0: hash-consing @@ -484,7 +488,7 @@ Pass 1 adds, for each group a query computes, every way to compute it. Each inte |---|---|---| | Exact | [`per_computation_candidates.rs`](../../../crates/logical-asap-optimization/src/pass1/per_computation_candidates.rs), [`memo_population.rs`](../../../crates/logical-asap-optimization/src/pass1/memo_population.rs) | a candidate with the exact operator; $\boldsymbol\theta_v$ none, $(\varepsilon_v, \delta_v) = (0, 0)$ | | Exact accumulator | same | a state group `ExactAggregate(kind)` over the input group, one candidate; a `FinalizeExactAccumulator` candidate in the result's group reading it | -| Summary, per family the executor supports | same | a state group `SummaryAgg[family]` over the input group, with **one candidate per $\boldsymbol\theta$ in the family's predefined list**, dropping a $\boldsymbol\theta$ whose $\varepsilon_v$ alone exceeds every reading query's $\varepsilon_q$; a `SummaryEstimate[statistic]` candidate in the result's group reading the state group. Each consumer's `error_cap` starts as its $g_c$'s validity cap (none for a linear $g_c$) | +| Summary, per family the executor supports | same | a state group `SummaryAgg[family]` over the input group, with **one candidate per $\boldsymbol\theta$ in the family's predefined list**, dropping a $\boldsymbol\theta$ whose $\varepsilon_v$ alone exceeds the $\varepsilon_q$ of every query that reaches the group; a `SummaryEstimate[statistic]` candidate in the result's group reading the state group. Each consumer's `error_cap` starts as its $g_c$'s validity cap (none for a linear $g_c$) | | Fused build (#1203 2c) | same | a build that computes several targets at once: a candidate of the outer target's state group, with `fused_groups` naming the others | Example: for A and B, Pass 1 adds G2 = `SummaryAgg[KLL](G1)` with candidates `k = 200` and `k = 345`, and G3, G4, G5 = `SummaryEstimate[q](G2)`. A and B's three estimates read one state group because their expressions over G1 hash-cons to one group. @@ -495,7 +499,7 @@ Today, Pass 1 instead sizes one $\boldsymbol\theta$ per `accuracy_budget_parts` | # | Rule | Code | Matches | Adds | Status | |---|---|---|---|---|---| -| 1 | Local window composition (tumbling windows) | `add_local_window_composition` ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs)) | a summary state group read by window readers whose ranges and steps share a gcd width $w$ | a candidate of the **same state group** with $\omega_v$ = tumbling $(w, n)$, its $r^{\log}$ window requirement on each reader input | today it copies readers per layout (`TumblingWindows` origin); #1160 makes it a candidate of the state group | +| 1 | Local window composition (tumbling windows) | `add_local_window_composition` ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs)) | a summary state group whose window consumers serve queries whose ranges and steps share a gcd width $w$ | a candidate of the **same state group** with $\omega_v$ = tumbling $(w, n)$, its $r^{\log}$ window requirement on each consumer input | today it copies consumers per layout (`TumblingWindows` origin); #1160 makes it a candidate of the state group | | 2 | Scan widening | `add_scan_label_candidates` ([`scan_labels.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/scan_labels.rs)) | several scan groups of one metric with different label sets | a group for the widened scan (`find_or_add_group`), and in each original scan group a candidate `Project(widened)` | #1203 2a-5 (#1222) | | 3 | Summary inputs | `factor_summary_inputs` ([`summary_inputs.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/summary_inputs.rs)) | a value projection or filter below a summary build | that projection or filter as its own group; the build's candidate reads it | #1203 2a-2 (#1218); after 2a, Pass 1 already makes it a group | | 4 | Top-k reuse | `add_topk_candidates` ([`topk.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/topk.rs)) | top-k groups over one ranking with different `k` | a candidate of each top-k group reading the largest-`k` group, with an exact-input requirement | #1203 2a-3 (#1220) | @@ -503,8 +507,8 @@ Today, Pass 1 instead sizes one $\boldsymbol\theta$ per `accuracy_budget_parts` | 6 | Rollup | `add_rollup_candidates` ([`rollup.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/rollup.rs)) | a coarse aggregate or summary group and a finer group whose state rolls up | a candidate of the coarse group reading the finer group (exact or sketch merge) | #1203 2a-4 (#1221) | | 7 | Window rollup | `add_window_rollup_candidates` (same) | a longer-range group whose windows a shorter-range group's state covers | a candidate of the longer-range group merging the shorter one's windows | #1203 2a-4 (#1221) | | 8 | Summary results | `factor_summary_results` ([`summary_results.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/summary_results.rs)) | a summary readout inside a composite candidate | the readout as its own group | #1203 2a-2 (#1218) makes the readout a group in Pass 1; the rule is then deleted | -| 9 | Shared configurations | `add_state_configuration_candidates` ([`shared_configuration_readers.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/shared_configuration_readers.rs)) | readers that could share a summary at a size or window horizon that covers them | reader copies at the shared size (`SharedBuild` origin) | deleted in #1203 2b: the state group already holds every $\boldsymbol\theta$, and $y$ chooses per reader | -| 10 | Shared summary state groups | `add_shared_summary_state_groups` ([`state_groups.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/state_groups.rs)) | summary builds directly over input groups | per-size concrete state groups with reader copies | deleted in #1203 2b and #1160: one state group per state family | +| 9 | Shared configurations | `add_state_configuration_candidates` ([`shared_configuration_readers.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/shared_configuration_readers.rs)) | consumers that could share a summary at a size or window horizon that covers them | consumer copies at the shared size (`SharedBuild` origin) | deleted in #1203 2b: the state group already holds every $\boldsymbol\theta$, and $y$ chooses per consumer input | +| 10 | Shared summary state groups | `add_shared_summary_state_groups` ([`state_groups.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/state_groups.rs)) | summary builds directly over input groups | per-size concrete state groups with consumer copies | deleted in #1203 2b and #1160: one state group per state family | | — | Logical edges (#1201) | after the last rule | every candidate $c$ and input $i$ | one `LogicalEdge { producer, consumer, input, satisfies_logical_requirement }` per $u \in G_{c,i}$, after tightening each `error_cap` with the pruning cap $\max_q \varepsilon_q / \lambda_{q,c,i}$; $\alpha^{\log}_{u,c,i}$ = `satisfies(p^log_u, r^log_{c,i})`; output `LogicalASAPMemoCandidates { memo, edges }` | #1201 | After #1203 2b, every Pass 2 rule has the same shape: match groups, then `find_or_add_group` for any new intermediate result and `add_candidate` for the new way to compute an existing group. A rule adds no candidate a group already has (same expression and $p^{\log}$). @@ -515,10 +519,10 @@ Stage 2 adds no logical candidates. It expands each logical candidate into physi | Step | Code | Reads | Adds | |---|---|---|---| -| Physical expansion | `enumerate_candidate_configurations` ([`configurations.rs`](../../../crates/physical-asap-optimization/src/catalog/configurations.rs)) | each `CandidateForMemoGroup`, deployment capabilities, readers' $\mathcal T_q$ | one `PhysicalCandidate { logical, properties, input_requirements }` per valid $p^{\mathrm{phys}} = (\tau, \ell, \sigma)$ that the deployment allows (section 2, levels 4–6), with one `PhysicalRequirement` per input: the reader's timing, the evaluation it reads from, and its schedule | +| Physical expansion | `enumerate_candidate_configurations` ([`configurations.rs`](../../../crates/physical-asap-optimization/src/catalog/configurations.rs)) | each `CandidateForMemoGroup`, deployment capabilities, the queries' $\mathcal T_q$ | one `PhysicalCandidate { logical, properties, input_requirements }` per valid $p^{\mathrm{phys}} = (\tau, \ell, \sigma)$ that the deployment allows (section 2, levels 4–6), with one `PhysicalRequirement` per input: the consumer's timing, the evaluation it reads from, and its context's schedule | | Physical edges (#1201) | after expansion | each `LogicalEdge` | one `PhysicalEdge { producer, consumer, input, satisfies_logical_requirement, satisfies_physical_requirement }` per pair of the two logical candidates' physical candidates, with $\alpha^{\mathrm{phys}}$ = `satisfies_physical(p^phys_u, r^phys_{c,i})`; the admissible ones ($\alpha^{\log} = \alpha^{\mathrm{phys}} = 1$) through `admissible_edges_into` / `admissible_edges_out_of` | -Example: v3 (`k = 345`) becomes, among others, (QueryTime, None), (QueryTime, OneEvaluation, Periodic(10 s, 0)) and (IngestionTime, AcrossEvaluations, grid phase 0). G3's estimate at query time has an admissible edge to each of them: same timing, or a query-time reader of an ingestion-time result kept across evaluations. +Example: v3 (`k = 345`) becomes, among others, (QueryTime, None), (QueryTime, OneEvaluation, Periodic(10 s, 0)) and (IngestionTime, AcrossEvaluations, grid phase 0). G3's estimate at query time has an admissible edge to each of them: same timing, or a query-time consumer of an ingestion-time result kept across evaluations. ### Stage 3: no rules From 3e8facac1b8c7472d0d4f87ea539e35ca26b3e0a Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 21:27:57 +0000 Subject: [PATCH 12/67] docs(concepts): section 5, printing every logical DAG a memo group represents Co-Authored-By: Claude Opus 5.5 --- .../memo-groups-properties-requirements.md | 125 ++++++++++++++++++ 1 file changed, 125 insertions(+) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index fff471115..8855040e4 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -527,3 +527,128 @@ Example: v3 (`k = 345`) becomes, among others, (QueryTime, None), (QueryTime, On ### Stage 3: no rules Stage 3 adds no candidates and no edges. It reads $\mathcal C$ and the admissible part of $\mathcal E$, creates the MILP variables of section 3 over them, and decodes $\{x = 1\}$ and $\{y = 1\}$ into `DagOfSelectedCandidates`. + +## 5. Printing every logical DAG a memo group represents + +A memo group stores its candidates and, through their expressions, the groups they read. It represents a set of logical DAGs without listing them. This section shows how to list them, using only the section 3 interfaces. Planning never does this: Stage 3's MILP chooses one DAG without listing the others. The listing is for inspection and for checking the MILP on small workloads. + +### What one logical DAG is + +A logical DAG rooted at group $G$ is a set of logical candidates and edges $(u, c, i)$ such that: +1. exactly one candidate of $G$ is in it (the root); +2. every candidate $c$ in it has, for each input $i$, exactly one edge $(u, c, i)$, with $u$ a candidate of $G_{c,i}$ and $\alpha^{\log}_{u,c,i} = 1$ (rows (A) and (G)); +3. every candidate in it is the root or the producer of some edge (row (F)); +4. the edges form no cycle (row (E)). + +A candidate is one node however many edges read it: two consumers that choose the same producer share it, which is what $x_u \in \{0, 1\}$ says. Two consumers of one group may choose different candidates of it. The listing is therefore a choice of one producer per consumer input $(c, i)$, not one candidate per group. + +### Pseudocode + +```rust +/// One logical DAG: its nodes and, for each node's inputs, the chosen producer. +struct LogicalDag { + root: CandidateForMemoGroupId, + /// For each node c in the DAG, one producer per input i, in input order. + producers: BTreeMap>, +} + +/// Every logical DAG rooted at `root_group`, each exactly once. +fn logical_dags(memo: &Memo, root_group: MemoGroupId) -> Vec { + let mut all = Vec::new(); + for root in candidates_of(memo, root_group) { + let dag = LogicalDag { root, producers: BTreeMap::new() }; + expand(memo, dag, VecDeque::from([root]), &mut all); + } + all +} + +/// Choose producers for the inputs of every node in `pending`, in FIFO order. +/// Each DAG is reached by exactly one sequence of choices, so none repeats. +fn expand(memo: &Memo, dag: LogicalDag, mut pending: VecDeque, all: &mut Vec) { + let Some(consumer) = pending.pop_front() else { + all.push(dag); // every input of every node has a producer + return; + }; + let candidate = memo.candidate(consumer); + let inputs: Vec = candidate.expression.operator.children().copied().collect(); + // Every way to choose one producer per input of `consumer`: the product over i + // of the admissible producers in G_{c,i}. + for choice in cartesian_product(inputs.iter().enumerate().map(|(i, &group)| { + candidates_of(memo, group) + .filter(|&u| satisfies(&memo.candidate(u).logical_properties, // α^log = 1 + &candidate.input_requirements[i])) + .collect::>() + })) { + // Row (E): a producer that already reaches `consumer` would close a cycle. + if choice.iter().any(|&u| reaches(&dag, u, consumer)) { + continue; + } + let mut next = dag.clone(); + next.producers.insert(consumer, choice.clone()); + let mut next_pending = pending.clone(); + for &u in &choice { + // A producer already in the DAG is shared, not expanded again. + if !next.producers.contains_key(&u) && !next_pending.contains(&u) && u != next.root { + next_pending.push_back(u); + } + } + expand(memo, next, next_pending, all); + } +} + +fn candidates_of(memo: &Memo, group: MemoGroupId) -> impl Iterator + '_ { + (0..memo.candidates[group.0 as usize].len()).map(move |index| CandidateForMemoGroupId { group, index }) +} + +/// Whether `from` reaches `to` through the producers chosen so far. +fn reaches(dag: &LogicalDag, from: CandidateForMemoGroupId, to: CandidateForMemoGroupId) -> bool; + +/// Print one DAG as an indented tree; a node read a second time is printed as a +/// reference to its first occurrence, so sharing is visible. +fn print(memo: &Memo, dag: &LogicalDag) { + fn node(memo: &Memo, dag: &LogicalDag, c: CandidateForMemoGroupId, depth: usize, seen: &mut HashSet) { + let candidate = memo.candidate(c); + if !seen.insert(c) { + println!("{}↑ G{}[{}] (shared)", " ".repeat(depth), c.group.0, c.index); + return; + } + println!("{}G{}[{}] {} {:?}", " ".repeat(depth), c.group.0, c.index, + candidate.expression.operator.name(), candidate.logical_properties); + for &u in dag.producers.get(&c).into_iter().flatten() { + node(memo, dag, u, depth + 1, seen); + } + } + node(memo, dag, dag.root, 0, &mut HashSet::new()); +} +``` + +For a workload, list from every query root at once: start `pending` with one root candidate per $G_q$ (rows (C) and (D)), taking the product over queries of their root candidates. A candidate two queries reach is one node, as above. + +### Example + +For the example's query A (root G3) and query B (root G6), with no error caps applied: + +``` +A: 2 DAGs B: 4 DAGs (G4 and G5 choose independently) +G3[0] SummaryEstimate q=0.95 G6[0] Divide + G2[0] SummaryAgg KLL θ={k=200} G4[0] SummaryEstimate q=0.99 + G1[0] Scan G2[0] SummaryAgg KLL θ={k=200} +G3[0] SummaryEstimate q=0.95 G1[0] Scan + G2[1] SummaryAgg KLL θ={k=345} G5[0] SummaryEstimate q=0.50 + G1[0] Scan ↑ G2[0] (shared) + … and (k=200, k=345), (k=345, k=200), (k=345, k=345) +``` + +Listed from both roots at once there are $2 \times 4 = 8$ DAGs. In the one where all three estimates choose `k = 345`, `G2[1]` is a single node read by three consumers, so its cost is counted once. Row (K) later accepts only the DAGs that meet each query's target: for B, only (k=345, k=345). Error caps (section 1) remove edges before the listing, so a capped producer never appears. + +### Size + +Without sharing, the number of DAGs rooted at $G$ satisfies + +$$N(G) = \sum_{v \in G} \prod_{i} \sum_{u \in G_{v,i},\ \alpha^{\log}_{u,v,i} = 1} N(u), \qquad N(u) = \prod_i \sum_{w} N(w) \text{ for } u\text{'s own inputs},$$ + +which grows as a product over the DAG's depth. Sharing only identifies DAGs that choose the same candidate twice; it does not reduce the count much. This is why only small workloads are listed, and why Stage 3 chooses with the MILP instead. + +### Physical DAGs + +The same procedure lists physical DAGs from Stage 2's output. Use `PhysicalCandidate`s instead of logical candidates, and `admissible_edges_into(c, i)` (which already applies $\alpha^{\log}$ and $\alpha^{\mathrm{phys}}$) instead of `candidates_of` with `satisfies`. A physical candidate with $\ell = $ None that two consumers in different contexts $\kappa$ choose runs once per context; in the listing it is still one node, and the contexts are a Stage 3 choice (#1207). From 86d7ace5bc6a176d6a60f611de36988b1687484d Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 21:28:12 +0000 Subject: [PATCH 13/67] docs(concepts): section 5 size bound; reaches covers self-reads Co-Authored-By: Claude Opus 5.5 --- .../concepts/memo-groups-properties-requirements.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 8855040e4..fe138037a 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -600,7 +600,8 @@ fn candidates_of(memo: &Memo, group: MemoGroupId) -> impl Iterator bool; /// Print one DAG as an indented tree; a node read a second time is printed as a @@ -643,11 +644,11 @@ Listed from both roots at once there are $2 \times 4 = 8$ DAGs. In the one where ### Size -Without sharing, the number of DAGs rooted at $G$ satisfies +Counting each consumer input's choice independently gives an upper bound on the number of DAGs rooted at $G$: -$$N(G) = \sum_{v \in G} \prod_{i} \sum_{u \in G_{v,i},\ \alpha^{\log}_{u,v,i} = 1} N(u), \qquad N(u) = \prod_i \sum_{w} N(w) \text{ for } u\text{'s own inputs},$$ +$$N(G) = \sum_{v \in G} N(v), \qquad N(v) = \prod_{i} \sum_{u \in P_{v,i},\ \alpha^{\log}_{u,v,i} = 1} N(u),$$ -which grows as a product over the DAG's depth. Sharing only identifies DAGs that choose the same candidate twice; it does not reduce the count much. This is why only small workloads are listed, and why Stage 3 chooses with the MILP instead. +with $N(v) = 1$ for a candidate with no inputs. It is an upper bound because a shared producer is expanded once in a DAG, where the formula expands it once per consumer, and because cyclic choices are dropped. $N$ multiplies along the depth of the memo, which is why only small workloads are listed, and why Stage 3 chooses with the MILP instead. ### Physical DAGs From 7bfb7f4ddedcb5aa57f58b618394285c5cd9ddea Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 22:12:52 +0000 Subject: [PATCH 14/67] docs(concepts): logical properties and requirement mirror the physical interfaces; satisfies_logical Co-Authored-By: Claude Opus 5.5 --- .../memo-groups-properties-requirements.md | 63 ++++++++++++++----- 1 file changed, 47 insertions(+), 16 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index fe138037a..33c5df6f2 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -288,34 +288,57 @@ pub struct CandidateForMemoGroup { /// e_v = op_v(G_{v,1}, …, G_{v,n}): one operator, its children are groups. pub struct CandidateExpression { pub operator: Operator } +/// p_v^log = (θ_v, ω_v); `new` rejects invalid combinations (parameters of +/// another family, a window layout on an exact operator that cannot be windowed). pub struct LogicalProperties { representation: Representation, // family and θ, from the predefined list window_layout: Option, // ω_v } impl LogicalProperties { pub fn new(representation: Representation, window_layout: Option) -> Result; - /// (ε_v, δ_v) for `statistic`, derived through `local_guarantee`. - pub fn error_bound(&self, statistic: &SketchStatistic) -> Option; + pub fn representation(&self) -> &Representation; + pub fn window_layout(&self) -> Option<&WindowLayout>; } pub enum Representation { Exact(ExactKind), Summary(SketchKind) } pub enum WindowLayout { Tumbling { width_ms: u64, count: u32 } } +/// r^log_{c,i}: one optional constraint per dimension; None accepts any value. +/// `new` rejects invalid combinations (a cap that is not positive and finite, +/// a cap without the statistic it applies to, a window with zero range or step). pub struct LogicalRequirement { - pub family: Option, - pub window: Option, // range and step of the queries reaching the consumer: a width must divide both - pub error_cap: Option, // cap_{c,i}: min(validity cap of g_c, pruning cap) + family: Option, + /// Range and step of the queries reaching the consumer: a width must divide both. + window: Option, + /// The statistic the consumer reads from this input; the producer's ε is + /// derived for it. + statistic: Option, + /// cap_{c,i} = min(validity cap of g_c, pruning cap max_q ε_q / λ_{q,c,i}). + error_cap: Option, +} +impl LogicalRequirement { + pub fn new(family: Option, window: Option, + statistic: Option, error_cap: Option) -> Result; + pub fn family(&self) -> Option; + pub fn window(&self) -> Option<&WindowRequirement>; + pub fn statistic(&self) -> Option<&SketchStatistic>; + pub fn error_cap(&self) -> Option; + /// g_c of a consumer: the propagation factor λ_{c,i} per input, read from the + /// operator and its arguments, and the validity cap (None when g_c is linear). + /// Sets each input's first `error_cap`; Stage 3 composes the factors into λ_{q,c,i}. + pub fn error_propagation(consumer: &Operator) -> ErrorPropagation; } -/// α^log_{u,c,i} -pub fn satisfies(producer: &LogicalProperties, requirement: &LogicalRequirement) -> bool; - -/// g_c as propagation factors λ_{c,i}, read from the consumer's operator and -/// arguments, and the validity cap (None when g_c is linear). /// Approved 2026-11-05. pub struct ErrorPropagation { pub factor_per_input: Vec, pub validity_cap: Option } -pub fn error_propagation(consumer: &Operator) -> ErrorPropagation; + +/// α^log_{u,c,i}: one check per dimension, joined by "and". The error check +/// derives the producer's (ε_u, δ_u) for `requirement.statistic()` through +/// `estimators::local_guarantee` and compares ε_u with `error_cap`. +pub fn satisfies_logical(producer: &LogicalProperties, requirement: &LogicalRequirement) -> bool; ``` -The validity cap is known when the consumer is created. The pruning cap needs $\lambda_{q,c,i}$ and the queries that may reach $c$, so it is filled in when Pass 2 builds the logical edges. +The logical and physical sides have the same shape: a properties struct and a requirement struct, each with private fields, a checked `new`, one accessor per dimension, and one `satisfies_*` function. Invalid values of both logical structs are reported through one error type, `InvalidLogicalProperties`, as `InvalidPhysicalProperties` does for the physical side. + +The error bound is not a method of `LogicalProperties`: it depends on the statistic the consumer reads, so `satisfies_logical` derives it from the requirement's statistic. The validity cap is known when the consumer is created (`LogicalRequirement::error_propagation`). The pruning cap needs $\lambda_{q,c,i}$ and the queries that may reach $c$, so Pass 2 builds each input's final requirement with `LogicalRequirement::new` when it builds the logical edges. Deleted: `root_node` with private nodes, `input_groups` (the expression's children), `accuracy_budget_parts`, the `SharedBuild` and `SummaryResult` origins. @@ -367,7 +390,8 @@ impl PhysicalProperties { pub fn is_materialized(&self) -> bool; // ℓ ≠ None } -/// r^phys_{c,i}; `new` rejects (IngestionTime, Later). +/// r^phys_{c,i}; `new` rejects (IngestionTime, Later), reported through +/// `InvalidPhysicalProperties` (one error type per side). pub struct PhysicalRequirement { consumer_timing: ExecutionTiming, reads_from: Evaluation, @@ -375,6 +399,13 @@ pub struct PhysicalRequirement { /// None for a non-periodic query. consumer_schedule: Option, } +impl PhysicalRequirement { + pub fn new(consumer_timing: ExecutionTiming, reads_from: Evaluation, + consumer_schedule: Option) -> Result; + pub fn consumer_timing(&self) -> ExecutionTiming; + pub fn reads_from(&self) -> Evaluation; + pub fn consumer_schedule(&self) -> Option<&EvaluationSchedule>; +} /// A periodic evaluation schedule φ + I·ℤ. pub struct EvaluationSchedule { interval_ms: u64, phase_ms: u64 } pub enum Evaluation { Same, Later } @@ -509,7 +540,7 @@ Today, Pass 1 instead sizes one $\boldsymbol\theta$ per `accuracy_budget_parts` | 8 | Summary results | `factor_summary_results` ([`summary_results.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/summary_results.rs)) | a summary readout inside a composite candidate | the readout as its own group | #1203 2a-2 (#1218) makes the readout a group in Pass 1; the rule is then deleted | | 9 | Shared configurations | `add_state_configuration_candidates` ([`shared_configuration_readers.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/shared_configuration_readers.rs)) | consumers that could share a summary at a size or window horizon that covers them | consumer copies at the shared size (`SharedBuild` origin) | deleted in #1203 2b: the state group already holds every $\boldsymbol\theta$, and $y$ chooses per consumer input | | 10 | Shared summary state groups | `add_shared_summary_state_groups` ([`state_groups.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/state_groups.rs)) | summary builds directly over input groups | per-size concrete state groups with consumer copies | deleted in #1203 2b and #1160: one state group per state family | -| — | Logical edges (#1201) | after the last rule | every candidate $c$ and input $i$ | one `LogicalEdge { producer, consumer, input, satisfies_logical_requirement }` per $u \in G_{c,i}$, after tightening each `error_cap` with the pruning cap $\max_q \varepsilon_q / \lambda_{q,c,i}$; $\alpha^{\log}_{u,c,i}$ = `satisfies(p^log_u, r^log_{c,i})`; output `LogicalASAPMemoCandidates { memo, edges }` | #1201 | +| — | Logical edges (#1201) | after the last rule | every candidate $c$ and input $i$ | one `LogicalEdge { producer, consumer, input, satisfies_logical_requirement }` per $u \in G_{c,i}$, after tightening each `error_cap` with the pruning cap $\max_q \varepsilon_q / \lambda_{q,c,i}$; $\alpha^{\log}_{u,c,i}$ = `satisfies_logical(p^log_u, r^log_{c,i})`; output `LogicalASAPMemoCandidates { memo, edges }` | #1201 | After #1203 2b, every Pass 2 rule has the same shape: match groups, then `find_or_add_group` for any new intermediate result and `add_candidate` for the new way to compute an existing group. A rule adds no candidate a group already has (same expression and $p^{\log}$). @@ -575,7 +606,7 @@ fn expand(memo: &Memo, dag: LogicalDag, mut pending: VecDeque>() })) { @@ -652,4 +683,4 @@ with $N(v) = 1$ for a candidate with no inputs. It is an upper bound because a s ### Physical DAGs -The same procedure lists physical DAGs from Stage 2's output. Use `PhysicalCandidate`s instead of logical candidates, and `admissible_edges_into(c, i)` (which already applies $\alpha^{\log}$ and $\alpha^{\mathrm{phys}}$) instead of `candidates_of` with `satisfies`. A physical candidate with $\ell = $ None that two consumers in different contexts $\kappa$ choose runs once per context; in the listing it is still one node, and the contexts are a Stage 3 choice (#1207). +The same procedure lists physical DAGs from Stage 2's output. Use `PhysicalCandidate`s instead of logical candidates, and `admissible_edges_into(c, i)` (which already applies $\alpha^{\log}$ and $\alpha^{\mathrm{phys}}$) instead of `candidates_of` with `satisfies_logical`. A physical candidate with $\ell = $ None that two consumers in different contexts $\kappa$ choose runs once per context; in the listing it is still one node, and the contexts are a Stage 3 choice (#1207). From 155489104bd0b044bb35a3dc1aaf76e78a4048e7 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 22:24:42 +0000 Subject: [PATCH 15/67] docs(concepts): drop fields other things fix; LogicalProperties {parameters, window_layout}, LogicalRequirement {parameter_floor}, r^phys = f(p^phys_c); at-a-glance tables Co-Authored-By: Claude Opus 5.5 --- .../memo-groups-properties-requirements.md | 174 ++++++++++-------- 1 file changed, 99 insertions(+), 75 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 33c5df6f2..3031eea50 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -19,7 +19,7 @@ This page defines the memo's data structures in the terms of the optimization pr | $c \in G_c$, $u \in G_u$ | consumer candidate and its group; producer candidate and its group. This page says *consumer* for $c$ and *query* $q$ (or its execution context $\kappa$) for who needs a result on a schedule; it does not use "reader" | | $\mathcal I(c)$ | input indices of consumer $c$ | | $G_{c,i}$ | memo group required by input $i$ of consumer $c$ | -| $r_{c,i} = (r^{\log}_{c,i}, r^{\mathrm{phys}}_{c,i})$ | property requirement of input $i$ of consumer $c$ | +| $r_{c,i} = (r^{\log}_{c,i}, r^{\mathrm{phys}}_{c,i})$ | property requirement of input $i$ of consumer $c$; $r^{\mathrm{phys}}_{c,i}$ is a function of $p^{\mathrm{phys}}_c$ | | $p(u) \succeq r_{c,i}$ | producer $u$'s properties satisfy the requirement: $p^{\log}(u) \succeq r^{\log}_{c,i} \land p^{\mathrm{phys}}(u) \succeq r^{\mathrm{phys}}_{c,i}$ | | $\alpha^{\log}_{u,c,i}$, $\alpha^{\mathrm{phys}}_{u,c,i}$ | fixed indicators of $p^{\log}_u \succeq r^{\log}_{c,i}$ and $p^{\mathrm{phys}}_u \succeq r^{\mathrm{phys}}_{c,i}$ | | $\mathcal C$ | physical candidates, each with a fixed parameter configuration, timing and lifetime | @@ -32,7 +32,7 @@ This page defines the memo's data structures in the terms of the optimization pr | **(#1207)** $\kappa$ | execution context of a candidate with $\ell_v$ = None: a query evaluation $\kappa_q$, an ingestion pipeline $I_r$, or a materialized candidate's computation $\kappa_m$ | | $g_c$ | error propagation function of consumer $c$: $\varepsilon_c \le g_c(\varepsilon_{c,1}, \dots, \varepsilon_{c,n})$ | | $\lambda_{q,c,i}$ | propagation factor of input $i$ of $c$ to query $q$'s result, read off $q$'s expression | -| $\text{cap}_{c,i}$ | error cap of input $i$ of $c$: the smaller of $g_c$'s validity cap and the pruning cap | +| $\text{cap}_{c,i}$, $\boldsymbol\theta^{\min}_{c,i}$ | error cap of input $i$ of $c$ (the smaller of $g_c$'s validity cap and the pruning cap), and the parameter floor $r^{\log}_{c,i}$ stores: the smallest listed $\boldsymbol\theta$ whose $\varepsilon$ is at most the cap | | **(#1207)** (K), (R), (L), budgets | accuracy at each query root, $\sum_u \lambda_{q,u}\varepsilon_u a_{q,u} \le \varepsilon_q$ and $\sum_u \delta_u a_{q,u} \le \delta_q$; retention $h_v$; latency; resource budgets | ## Example @@ -69,7 +69,7 @@ What is computed is the expression $e_v$, not a property: the operator, its argu | Dimension | Values | Part of | Code | |---|---|---|---| | Parameter configuration $\boldsymbol\theta_v$ | per family, from a predefined list: KLL `k`; DDSketch `alpha`; CMS / CountSketch `width × depth`; CMS / CountSketch with heap `width × depth × heap_size`; UnivMon `heap_size × rows × cols × layers`; HLL `precision`; KMV `k`. None for an exact operator | $p^{\log}_v$ | `SketchParams` inside `SketchKind` ([`sketch.rs`](../../../crates/types/src/ir/schema/sketch.rs)), sized by `size_params` per `accuracy_budget_parts` today | -| **(#1207)** Window layout $\omega_v$ | none; or tumbling windows of width $w$ and count $n$, with $w$ from the gcd closure of the ranges and steps of the queries that reach the group. Later: sliding windows by merge or by subtraction | $p^{\log}_v$ | `SummaryTumblingWindowChain` arguments, added by Pass 2 ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs)) | +| **(#1207)** Window layout $\omega_v$ | none; or tumbling windows of width $w$, from the gcd closure of the ranges and steps of the queries that reach the group. The window count is not a property: it is the group's query range divided by $w$, and how many windows to retain is $h_v$ (Stage 3). Later: sliding windows by merge or by subtraction | $p^{\log}_v$ | `SummaryTumblingWindowChain` arguments, added by Pass 2 ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs)) | | Execution timing $\tau_v$ | IngestionTime, QueryTime | $p^{\mathrm{phys}}_v$ | `ExecutionTiming` ([`execution_data_state.rs`](../../../crates/types/src/ir/properties/execution_data_state.rs)) | | Lifetime $\ell_v$ | None, OneEvaluation, AcrossEvaluations; **(#1207)** a materialized lifetime carries its medium, Memory or Disk | $p^{\mathrm{phys}}_v$ | `Materialization` on main; `Lifetime`, `Medium` in #1214 ([`physical/`](../../../crates/types/src/physical)) | | **(#1207)** Schedule $\sigma_v$ | only when $\ell_v \ne$ None: Periodic (interval, phase) for OneEvaluation; tumbling-window grid phase for AcrossEvaluations | $p^{\mathrm{phys}}_v$ | schedule groups `z[m, G]` in the MILP today ([`cost_terms.rs`](../../../crates/plan-selection/src/milp/cost_terms.rs)); a property in #1200 PR 2 | @@ -93,7 +93,7 @@ Valid $(\tau, \ell)$: (QueryTime, None), (IngestionTime, None), (QueryTime, OneE ### Property requirements -Each input $i$ of consumer $c$ names a producer group $G_{c,i}$ and a property requirement $r_{c,i} = (r^{\log}_{c,i}, r^{\mathrm{phys}}_{c,i})$. Logical satisfaction checks configuration requirements; physical satisfaction checks whether the producer's timing and lifetime jointly support the required access. For a monotone size parameter, $\succeq$ reduces to numerical dominance, e.g. $k_u \ge k_{\text{required}}$ for KLL. +Each input $i$ of consumer $c$ names a producer group $G_{c,i}$ and a property requirement $r_{c,i} = (r^{\log}_{c,i}, r^{\mathrm{phys}}_{c,i})$. A requirement holds only what neither $G_{c,i}$'s identity nor the consumer's own properties already fix (decided 2026-11-05): the family and the query window are in $G_{c,i}$'s identity, and $r^{\mathrm{phys}}_{c,i}$ is a function of the consumer's own $p^{\mathrm{phys}}_c$, not a stored value. Logical satisfaction checks configuration requirements; physical satisfaction checks whether the producer's timing and lifetime jointly support the required access. For a monotone size parameter, $\succeq$ reduces to numerical dominance, e.g. $k_u \ge k_{\text{required}}$ for KLL. The workload's accuracy target $A_q$ is not an input requirement **(#1207)**: a per-input bound would fix each input's share of the query's budget. Row (K) bounds each query's composed error at its root instead. What stays in $r^{\log}$ is what depends only on the consumer. @@ -133,11 +133,9 @@ For a monotone size parameter, $\varepsilon_u \le \text{cap}_{c,i}$ is a minimum | Requirement | Stated by | Values | Constrains | Checked by | |---|---|---|---|---| | Accuracy target $A_q$ | the workload, per query | metric: absolute (in the value's units, or times the stream's L1 or L2 norm), relative, rank; bound $\varepsilon_q$; failure probability $\delta_q$; scope: per value, whole result, or groups above a share of the records | $\boldsymbol\theta_u$ of every approximate candidate $q$ depends on, through $(\varepsilon_u, \delta_u)$ | row (K) **(#1207)** | -| Error cap $\text{cap}_{c,i}$, in $r^{\log}_{c,i}$ | every consumer, per input, from $g_c$ and the targets of the queries that may reach it | the smaller of the validity cap (non-linear $g_c$ only) and the pruning cap; for a monotone size parameter, a minimum size $k_u \ge k_{\text{required}}$ | $\boldsymbol\theta_u$ | $\alpha^{\log}$, row (G): validity cap for correctness of (K), pruning cap for efficiency | -| Window, in $r^{\log}_{c,i}$ **(#1207)** | a consumer that reads windows, per input | the range and step of the queries that reach the consumer | $\omega_u$: the width must divide both | $\alpha^{\log}$, row (G) | -| Summary family, in $r^{\log}_{c,i}$ | a consumer that reads summary state | a family | usually already fixed by $G_{c,i}$, whose $\operatorname{op}$ names the family | $\alpha^{\log}$, row (G) | -| Consumer timing and evaluation, in $r^{\mathrm{phys}}_{c,i}$ | the consumer, per input | when it reads (ingestion or query time) and from which evaluation (same or later) | $\tau_u$, $\ell_u$ | $\alpha^{\mathrm{phys}}$, row (H) | -| Consumer schedule, in $r^{\mathrm{phys}}_{c,i}$ **(#1207)** | the consumer's execution context $\kappa$, from the $\mathcal T_q$ of its query | the context's interval and phase $(I_\kappa, \varphi_\kappa)$; the producer must deliver on them, $T_\kappa \subseteq T_{\sigma_u}$ | $\sigma_u$ | $\alpha^{\mathrm{phys}}$, row (H) | +| Parameter floor $\boldsymbol\theta^{\min}_{c,i}$, the only field of $r^{\log}_{c,i}$ | every consumer, per input, from $g_c$ and the targets of the queries that may reach it | the smallest $\boldsymbol\theta$ in the family's predefined list whose $\varepsilon$, for the statistic $c$ reads, is at most $\text{cap}_{c,i}$ = min(validity cap, pruning cap); e.g. $k_u \ge k_{\text{required}}$ | $\boldsymbol\theta_u$ | $\alpha^{\log}$, row (G): $\boldsymbol\theta_u \succeq \boldsymbol\theta^{\min}_{c,i}$ | +| Consumer timing, $r^{\mathrm{phys}}_{c,i}$ from $\tau_c$ | the consumer's own $p^{\mathrm{phys}}_c$ | when $c$ runs: ingestion or query time | $\tau_u$, $\ell_u$ | $\alpha^{\mathrm{phys}}$, row (H) | +| Consumer schedule, $r^{\mathrm{phys}}_{c,i}$ from $\sigma_c$ or $\kappa$ **(#1207)** | a materialized consumer's own $\sigma_c$; for $\ell_c$ = None, the times $T_\kappa$ of each context $\kappa$ it runs in (from its query's $\mathcal T_q$) | interval and phase | $\sigma_u$: the producer delivers on those times | $\alpha^{\mathrm{phys}}$, row (H): in Stage 2 for a materialized consumer; per context in Stage 3 for $\ell_c$ = None | | Deployment capability | the deployment and data | the source has an ingestion pipeline; the executor supports the family; query-time results may be kept across evaluations | $\tau$, $\operatorname{op}$, (QueryTime, AcrossEvaluations) | candidate generation: such candidates are not in $\mathcal C$ | | Latency, resource budgets **(#1207)** | the workload | per-query latency bound; memory and CPU budgets | the whole plan | rows (L) and budgets | @@ -146,15 +144,55 @@ Accuracy metrics that may be added later, when a primitive needs them: top-k rec How requirements map to properties: ``` -A_q (workload) ──► θ_u of every approximate input (through ε_u, δ_u; row (K)) -error cap r^log ──► θ_u (validity cap from g_c; pruning cap from ε_q / λ_{q,c,i}) -window r^log ──► ω_u -consumer timing r^phys ──► τ_u, ℓ_u -consumer schedule r^phys ──► σ_u -deployment ──► τ, op, ℓ (not generated) -latency, budgets ──► every dimension, through w_v +A_q (workload) ──► θ_u of every approximate input (through ε_u, δ_u; row (K)) +parameter floor θ^min r^log ──► θ_u (from cap = min(validity cap of g_c, pruning cap ε_q / λ_{q,c,i})) +τ_c r^phys = f(p^phys_c) ──► τ_u, ℓ_u +σ_c or T_κ r^phys = f(p^phys_c) ──► σ_u +G_{c,i}'s identity (family, query window) ──► op_u, ω_u (by construction: no requirement) +deployment ──► τ, op, ℓ (not generated) +latency, budgets ──► every dimension, through w_v ``` +### Properties and requirements at a glance (decided 2026-11-05) + +A field is kept only when it is a real choice: anything the expression, $G_{c,i}$'s identity or the consumer's own properties already fix is not repeated. + +| Struct | Field | Values | Symbol | `new` rejects | +|---|---|---|---|---| +| `LogicalProperties` ($p^{\log}_v$) | `parameters` | `Option`: θ from the family's predefined list; None for an exact operator | $\boldsymbol\theta_v$ | parameters of a family other than the operator's; parameters on an exact operator | +| | `window_layout` | `None`, or `Tumbling { width_ms }` (later: sliding by merge or subtraction) | $\omega_v$ | a window on an operator that cannot be windowed; width 0 | +| `LogicalRequirement` ($r^{\log}_{c,i}$) | `parameter_floor` | `Option`: the smallest θ in the list whose $\varepsilon$ ≤ $\text{cap}_{c,i}$ | $\boldsymbol\theta^{\min}_{c,i}$ | a floor not in the family's predefined list | +| `PhysicalProperties` ($p^{\mathrm{phys}}_v$) | `timing` | IngestionTime, QueryTime | $\tau_v$ | (IngestionTime, OneEvaluation) | +| | `lifetime` | None, OneEvaluation(Medium), AcrossEvaluations(Medium); Medium ∈ {Memory, Disk} | $\ell_v$ | | +| | `schedule` | None; Periodic { interval, phase }; TumblingWindowGrid { phase } | $\sigma_v$ | the schedule combinations listed above | +| ($r^{\mathrm{phys}}_{c,i}$) | none | a function of the consumer's $p^{\mathrm{phys}}_c$: its $\tau_c$, and its $\sigma_c$ or each context's $T_\kappa$ | | | + +Not fields, because something else already fixes them: the summary family ($\operatorname{op}_v$'s argument, part of the group identity); the window count (query range ÷ width; retention is $h_v$); a family or window requirement on an input ($G_{c,i}$'s identity, and the schedule check); the statistic a consumer reads (its own operator, used once to compute $\boldsymbol\theta^{\min}$); which evaluation produced the output (reuse across evaluations is the producer's $\ell$, a cost choice). + +**α^log** = `satisfies_logical(p^log_u, r^log_{c,i})`: $\boldsymbol\theta_u \succeq \boldsymbol\theta^{\min}_{c,i}$, compared per family on the parameters that set $\varepsilon$ (KLL `k` ≥; CMS / CountSketch `width` ≥; DDSketch `alpha` ≤; HLL `precision` ≥; KMV `k` ≥; UnivMon and the heap variants on each sizing parameter). No floor, or an exact producer, always satisfies. + +**α^phys** = `satisfies_physical(p^phys_u, p^phys_c)`, timing and lifetime: + +| Consumer $\tau_c$ ↓ / producer $(\tau_u, \ell_u)$ → | (QT, None) | (QT, OneEval) | (QT, AcrossEval) | (IT, None) | (IT, AcrossEval) | +|---|---|---|---|---|---| +| QueryTime | ✅ | ✅ | ✅ | ❌ an unmaterialized ingestion-time output is gone by query time | ✅ | +| IngestionTime | ❌ | ❌ | ❌ | ✅ | ✅ | + +and schedule, for a producer with $\ell_u \ne$ None, against the consumer's times (its own $\sigma_c$ in Stage 2 when $\ell_c \ne$ None; each context's $T_\kappa$ in Stage 3 when $\ell_c$ = None): + +| Producer $\sigma_u$ | Delivers on consumer times $(I, \varphi)$ when | +|---|---| +| none ($\ell_u$ = None) | always: it runs inside the consumer's context, chosen by Stage 3 ($x_{\kappa,v}$) | +| Periodic $(I_\sigma, \varphi_\sigma)$ | $I_\sigma \mid I$ and $\varphi \equiv \varphi_\sigma \pmod{I_\sigma}$; a non-periodic consumer is never served | +| TumblingWindowGrid $\varphi_g$, width $w$ | $w \mid I$ and $\varphi \equiv \varphi_g \pmod w$; a non-periodic consumer is never served | + +| Struct | Set by | When | +|---|---|---| +| `LogicalProperties` | Pass 1 (one candidate per θ in the list); Pass 2 rules (window layouts) | when the candidate is created | +| `LogicalRequirement` | the consumer: floor from the validity cap of `LogicalRequirement::error_propagation(consumer)` | when the consumer is created; Pass 2 tightens it with the pruning cap when it builds the logical edges | +| `PhysicalProperties` | Stage 2 physical expansion (every valid $(\tau, \ell, \sigma)$) | Stage 2 | +| $r^{\mathrm{phys}}_{c,i}$ | derived from $p^{\mathrm{phys}}_c$ | checked in Stage 2 (timing, lifetime; schedule of a materialized consumer) and Stage 3 (schedule per context) | + ## 2. Decision tree of properties Each memo group's candidates are the leaves of a tree. The first level chooses the expression $e_v$; every later level decides one property dimension. A path from the root to a leaf is one physical candidate in $\mathcal C$. @@ -288,57 +326,51 @@ pub struct CandidateForMemoGroup { /// e_v = op_v(G_{v,1}, …, G_{v,n}): one operator, its children are groups. pub struct CandidateExpression { pub operator: Operator } -/// p_v^log = (θ_v, ω_v); `new` rejects invalid combinations (parameters of -/// another family, a window layout on an exact operator that cannot be windowed). +/// p_v^log = (θ_v, ω_v). The family is the operator's argument, not a property. +/// `new` rejects parameters of another family (or on an exact operator) and a +/// window on an operator that cannot be windowed. pub struct LogicalProperties { - representation: Representation, // family and θ, from the predefined list - window_layout: Option, // ω_v + parameters: Option, // θ_v, from the family's predefined list; None for exact + window_layout: Option, // ω_v } impl LogicalProperties { - pub fn new(representation: Representation, window_layout: Option) -> Result; - pub fn representation(&self) -> &Representation; + pub fn new(expression: &CandidateExpression, parameters: Option, + window_layout: Option) -> Result; + pub fn parameters(&self) -> Option<&SketchParams>; pub fn window_layout(&self) -> Option<&WindowLayout>; } -pub enum Representation { Exact(ExactKind), Summary(SketchKind) } -pub enum WindowLayout { Tumbling { width_ms: u64, count: u32 } } +/// The window count is the group's query range ÷ width; retention is h_v (Stage 3). +pub enum WindowLayout { Tumbling { width_ms: u64 } } + +/// The family's predefined θ list (#1207 step 6), each with its error bound +/// derived through `estimators::local_guarantee` for a given statistic. +pub fn parameter_list(family: SketchAlgorithm) -> &'static [SketchParams]; -/// r^log_{c,i}: one optional constraint per dimension; None accepts any value. -/// `new` rejects invalid combinations (a cap that is not positive and finite, -/// a cap without the statistic it applies to, a window with zero range or step). +/// r^log_{c,i}: only what G_{c,i}'s identity does not fix. `new` rejects a floor +/// that is not in the family's predefined list. pub struct LogicalRequirement { - family: Option, - /// Range and step of the queries reaching the consumer: a width must divide both. - window: Option, - /// The statistic the consumer reads from this input; the producer's ε is - /// derived for it. - statistic: Option, - /// cap_{c,i} = min(validity cap of g_c, pruning cap max_q ε_q / λ_{q,c,i}). - error_cap: Option, + /// θ^min_{c,i}: the smallest θ in the list whose ε, for the statistic the + /// consumer reads, is at most cap_{c,i} = min(validity cap of g_c, pruning cap). + parameter_floor: Option, } impl LogicalRequirement { - pub fn new(family: Option, window: Option, - statistic: Option, error_cap: Option) -> Result; - pub fn family(&self) -> Option; - pub fn window(&self) -> Option<&WindowRequirement>; - pub fn statistic(&self) -> Option<&SketchStatistic>; - pub fn error_cap(&self) -> Option; - /// g_c of a consumer: the propagation factor λ_{c,i} per input, read from the - /// operator and its arguments, and the validity cap (None when g_c is linear). - /// Sets each input's first `error_cap`; Stage 3 composes the factors into λ_{q,c,i}. + pub fn new(parameter_floor: Option) -> Result; + pub fn parameter_floor(&self) -> Option<&SketchParams>; + /// g_c: λ_{c,i} per input, read from the operator and its arguments, and the + /// validity cap (None when g_c is linear). Sets each input's first floor; + /// Stage 3 composes the factors into λ_{q,c,i}. pub fn error_propagation(consumer: &Operator) -> ErrorPropagation; } /// Approved 2026-11-05. pub struct ErrorPropagation { pub factor_per_input: Vec, pub validity_cap: Option } -/// α^log_{u,c,i}: one check per dimension, joined by "and". The error check -/// derives the producer's (ε_u, δ_u) for `requirement.statistic()` through -/// `estimators::local_guarantee` and compares ε_u with `error_cap`. +/// α^log_{u,c,i}: θ_u ⪰ θ^min_{c,i}, compared per family on the parameters that set ε. pub fn satisfies_logical(producer: &LogicalProperties, requirement: &LogicalRequirement) -> bool; ``` -The logical and physical sides have the same shape: a properties struct and a requirement struct, each with private fields, a checked `new`, one accessor per dimension, and one `satisfies_*` function. Invalid values of both logical structs are reported through one error type, `InvalidLogicalProperties`, as `InvalidPhysicalProperties` does for the physical side. +The logical and physical properties have the same shape: private fields, a checked `new`, one accessor per dimension, one error type per side (`InvalidLogicalProperties`, `InvalidPhysicalProperties`), and one `satisfies_*` function. Only the logical side has a stored requirement; the physical requirement is a function of the consumer's own physical properties. -The error bound is not a method of `LogicalProperties`: it depends on the statistic the consumer reads, so `satisfies_logical` derives it from the requirement's statistic. The validity cap is known when the consumer is created (`LogicalRequirement::error_propagation`). The pruning cap needs $\lambda_{q,c,i}$ and the queries that may reach $c$, so Pass 2 builds each input's final requirement with `LogicalRequirement::new` when it builds the logical edges. +The error bound is not stored: it depends on the statistic the consumer reads, which is the consumer's own operator argument. It is used once, when the consumer's floor is computed: the smallest θ in `parameter_list(family)` whose `local_guarantee` for that statistic has ε ≤ cap. The validity cap is known when the consumer is created (`LogicalRequirement::error_propagation`). The pruning cap needs $\lambda_{q,c,i}$ and the queries that may reach $c$, so Pass 2 builds each input's final requirement with `LogicalRequirement::new` when it builds the logical edges. Deleted: `root_node` with private nodes, `input_groups` (the expression's children), `accuracy_budget_parts`, the `SharedBuild` and `SummaryResult` origins. @@ -390,27 +422,18 @@ impl PhysicalProperties { pub fn is_materialized(&self) -> bool; // ℓ ≠ None } -/// r^phys_{c,i}; `new` rejects (IngestionTime, Later), reported through -/// `InvalidPhysicalProperties` (one error type per side). -pub struct PhysicalRequirement { - consumer_timing: ExecutionTiming, - reads_from: Evaluation, - /// (I_κ, φ_κ) of the consumer's execution context, from its query's T_q; - /// None for a non-periodic query. - consumer_schedule: Option, -} -impl PhysicalRequirement { - pub fn new(consumer_timing: ExecutionTiming, reads_from: Evaluation, - consumer_schedule: Option) -> Result; - pub fn consumer_timing(&self) -> ExecutionTiming; - pub fn reads_from(&self) -> Evaluation; - pub fn consumer_schedule(&self) -> Option<&EvaluationSchedule>; -} -/// A periodic evaluation schedule φ + I·ℤ. +/// A periodic schedule φ + I·ℤ: a materialized consumer's own times, or a +/// context's times T_κ (Stage 3). pub struct EvaluationSchedule { interval_ms: u64, phase_ms: u64 } -pub enum Evaluation { Same, Later } -/// α^phys_{u,c,i} -pub fn satisfies_physical(producer: &PhysicalProperties, requirement: &PhysicalRequirement) -> bool; +impl PhysicalProperties { + /// Whether this producer delivers on `times` (the schedule table in section 1); + /// always true when ℓ = None. + pub fn delivers_on(&self, times: &EvaluationSchedule) -> bool; +} +/// α^phys_{u,c,i} = r^phys_{c,i} as a function of the consumer's own p^phys_c: +/// timing and lifetime, and, for a materialized consumer, `delivers_on` its σ_c. +/// For a consumer with ℓ = None, Stage 3 checks `delivers_on(T_κ)` per context. +pub fn satisfies_physical(producer: &PhysicalProperties, consumer: &PhysicalProperties) -> bool; // crates/types/src/stages/physical pub struct PhysicalCandidateId(pub usize); @@ -420,7 +443,6 @@ pub struct PhysicalEdgeId(pub usize); pub struct PhysicalCandidate { pub logical: CandidateForMemoGroupId, // e_v and p_v^log pub properties: PhysicalProperties, // p_v^phys - pub input_requirements: Vec, // r^phys_{c,i} } /// (u, c, i) over physical candidates. pub struct PhysicalEdge { @@ -460,6 +482,8 @@ pub enum Context { IngestionPipeline(RefreshWindowId), // I_r Materialized(PhysicalCandidateId), // κ_m: inside a materialized m's computation } +/// T_κ: when the context runs; y_{κ,u,c,i} exists only if `p^phys_u.delivers_on(T_κ)`. +fn context_times(context: &Context) -> Option; struct SelectionVariables { x_materialized: Vec, // x_v, ℓ ≠ None @@ -519,7 +543,7 @@ Pass 1 adds, for each group a query computes, every way to compute it. Each inte |---|---|---| | Exact | [`per_computation_candidates.rs`](../../../crates/logical-asap-optimization/src/pass1/per_computation_candidates.rs), [`memo_population.rs`](../../../crates/logical-asap-optimization/src/pass1/memo_population.rs) | a candidate with the exact operator; $\boldsymbol\theta_v$ none, $(\varepsilon_v, \delta_v) = (0, 0)$ | | Exact accumulator | same | a state group `ExactAggregate(kind)` over the input group, one candidate; a `FinalizeExactAccumulator` candidate in the result's group reading it | -| Summary, per family the executor supports | same | a state group `SummaryAgg[family]` over the input group, with **one candidate per $\boldsymbol\theta$ in the family's predefined list**, dropping a $\boldsymbol\theta$ whose $\varepsilon_v$ alone exceeds the $\varepsilon_q$ of every query that reaches the group; a `SummaryEstimate[statistic]` candidate in the result's group reading the state group. Each consumer's `error_cap` starts as its $g_c$'s validity cap (none for a linear $g_c$) | +| Summary, per family the executor supports | same | a state group `SummaryAgg[family]` over the input group, with **one candidate per $\boldsymbol\theta$ in the family's predefined list**, dropping a $\boldsymbol\theta$ whose $\varepsilon_v$ alone exceeds the $\varepsilon_q$ of every query that reaches the group; a `SummaryEstimate[statistic]` candidate in the result's group reading the state group. Each consumer's `parameter_floor` starts from its $g_c$'s validity cap (none for a linear $g_c$) | | Fused build (#1203 2c) | same | a build that computes several targets at once: a candidate of the outer target's state group, with `fused_groups` naming the others | Example: for A and B, Pass 1 adds G2 = `SummaryAgg[KLL](G1)` with candidates `k = 200` and `k = 345`, and G3, G4, G5 = `SummaryEstimate[q](G2)`. A and B's three estimates read one state group because their expressions over G1 hash-cons to one group. @@ -530,7 +554,7 @@ Today, Pass 1 instead sizes one $\boldsymbol\theta$ per `accuracy_budget_parts` | # | Rule | Code | Matches | Adds | Status | |---|---|---|---|---|---| -| 1 | Local window composition (tumbling windows) | `add_local_window_composition` ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs)) | a summary state group whose window consumers serve queries whose ranges and steps share a gcd width $w$ | a candidate of the **same state group** with $\omega_v$ = tumbling $(w, n)$, its $r^{\log}$ window requirement on each consumer input | today it copies consumers per layout (`TumblingWindows` origin); #1160 makes it a candidate of the state group | +| 1 | Local window composition (tumbling windows) | `add_local_window_composition` ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs)) | a summary state group whose window consumers serve queries whose ranges and steps share a gcd width $w$ | a candidate of the **same state group** with $\omega_v$ = tumbling width $w$; only widths dividing the group's query range are offered, and the step and phase are checked by the schedule rule ($\alpha^{\mathrm{phys}}$) | today it copies consumers per layout (`TumblingWindows` origin); #1160 makes it a candidate of the state group | | 2 | Scan widening | `add_scan_label_candidates` ([`scan_labels.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/scan_labels.rs)) | several scan groups of one metric with different label sets | a group for the widened scan (`find_or_add_group`), and in each original scan group a candidate `Project(widened)` | #1203 2a-5 (#1222) | | 3 | Summary inputs | `factor_summary_inputs` ([`summary_inputs.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/summary_inputs.rs)) | a value projection or filter below a summary build | that projection or filter as its own group; the build's candidate reads it | #1203 2a-2 (#1218); after 2a, Pass 1 already makes it a group | | 4 | Top-k reuse | `add_topk_candidates` ([`topk.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/topk.rs)) | top-k groups over one ranking with different `k` | a candidate of each top-k group reading the largest-`k` group, with an exact-input requirement | #1203 2a-3 (#1220) | @@ -540,7 +564,7 @@ Today, Pass 1 instead sizes one $\boldsymbol\theta$ per `accuracy_budget_parts` | 8 | Summary results | `factor_summary_results` ([`summary_results.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/summary_results.rs)) | a summary readout inside a composite candidate | the readout as its own group | #1203 2a-2 (#1218) makes the readout a group in Pass 1; the rule is then deleted | | 9 | Shared configurations | `add_state_configuration_candidates` ([`shared_configuration_readers.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/shared_configuration_readers.rs)) | consumers that could share a summary at a size or window horizon that covers them | consumer copies at the shared size (`SharedBuild` origin) | deleted in #1203 2b: the state group already holds every $\boldsymbol\theta$, and $y$ chooses per consumer input | | 10 | Shared summary state groups | `add_shared_summary_state_groups` ([`state_groups.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/state_groups.rs)) | summary builds directly over input groups | per-size concrete state groups with consumer copies | deleted in #1203 2b and #1160: one state group per state family | -| — | Logical edges (#1201) | after the last rule | every candidate $c$ and input $i$ | one `LogicalEdge { producer, consumer, input, satisfies_logical_requirement }` per $u \in G_{c,i}$, after tightening each `error_cap` with the pruning cap $\max_q \varepsilon_q / \lambda_{q,c,i}$; $\alpha^{\log}_{u,c,i}$ = `satisfies_logical(p^log_u, r^log_{c,i})`; output `LogicalASAPMemoCandidates { memo, edges }` | #1201 | +| — | Logical edges (#1201) | after the last rule | every candidate $c$ and input $i$ | one `LogicalEdge { producer, consumer, input, satisfies_logical_requirement }` per $u \in G_{c,i}$, after tightening each `parameter_floor` with the pruning cap $\max_q \varepsilon_q / \lambda_{q,c,i}$; $\alpha^{\log}_{u,c,i}$ = `satisfies_logical(p^log_u, r^log_{c,i})`; output `LogicalASAPMemoCandidates { memo, edges }` | #1201 | After #1203 2b, every Pass 2 rule has the same shape: match groups, then `find_or_add_group` for any new intermediate result and `add_candidate` for the new way to compute an existing group. A rule adds no candidate a group already has (same expression and $p^{\log}$). @@ -550,8 +574,8 @@ Stage 2 adds no logical candidates. It expands each logical candidate into physi | Step | Code | Reads | Adds | |---|---|---|---| -| Physical expansion | `enumerate_candidate_configurations` ([`configurations.rs`](../../../crates/physical-asap-optimization/src/catalog/configurations.rs)) | each `CandidateForMemoGroup`, deployment capabilities, the queries' $\mathcal T_q$ | one `PhysicalCandidate { logical, properties, input_requirements }` per valid $p^{\mathrm{phys}} = (\tau, \ell, \sigma)$ that the deployment allows (section 2, levels 4–6), with one `PhysicalRequirement` per input: the consumer's timing, the evaluation it reads from, and its context's schedule | -| Physical edges (#1201) | after expansion | each `LogicalEdge` | one `PhysicalEdge { producer, consumer, input, satisfies_logical_requirement, satisfies_physical_requirement }` per pair of the two logical candidates' physical candidates, with $\alpha^{\mathrm{phys}}$ = `satisfies_physical(p^phys_u, r^phys_{c,i})`; the admissible ones ($\alpha^{\log} = \alpha^{\mathrm{phys}} = 1$) through `admissible_edges_into` / `admissible_edges_out_of` | +| Physical expansion | `enumerate_candidate_configurations` ([`configurations.rs`](../../../crates/physical-asap-optimization/src/catalog/configurations.rs)) | each `CandidateForMemoGroup`, deployment capabilities, the queries' $\mathcal T_q$ | one `PhysicalCandidate { logical, properties, input_requirements }` per valid $p^{\mathrm{phys}} = (\tau, \ell, \sigma)$ that the deployment allows (section 2, levels 4–6), | +| Physical edges (#1201) | after expansion | each `LogicalEdge` | one `PhysicalEdge { producer, consumer, input, satisfies_logical_requirement, satisfies_physical_requirement }` per pair of the two logical candidates' physical candidates, with $\alpha^{\mathrm{phys}}$ = `satisfies_physical(p^phys_u, p^phys_c)`; the admissible ones ($\alpha^{\log} = \alpha^{\mathrm{phys}} = 1$) through `admissible_edges_into` / `admissible_edges_out_of` | Example: v3 (`k = 345`) becomes, among others, (QueryTime, None), (QueryTime, OneEvaluation, Periodic(10 s, 0)) and (IngestionTime, AcrossEvaluations, grid phase 0). G3's estimate at query time has an admissible edge to each of them: same timing, or a query-time consumer of an ingestion-time result kept across evaluations. From 095269268a8a2f635beadb225fe10cd6ad1bef15 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 22:28:58 +0000 Subject: [PATCH 16/67] docs(concepts): parameter_list approved Co-Authored-By: Claude Opus 5.5 --- .../design_docs/concepts/memo-groups-properties-requirements.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 3031eea50..faed1f308 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -342,7 +342,7 @@ impl LogicalProperties { /// The window count is the group's query range ÷ width; retention is h_v (Stage 3). pub enum WindowLayout { Tumbling { width_ms: u64 } } -/// The family's predefined θ list (#1207 step 6), each with its error bound +/// Approved 2026-11-05. The family's predefined θ list, each with its error bound /// derived through `estimators::local_guarantee` for a given statistic. pub fn parameter_list(family: SketchAlgorithm) -> &'static [SketchParams]; From 4e7c752de1696d286b485df3f86f1385def51bbc Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 22:29:57 +0000 Subject: [PATCH 17/67] docs(concepts): context_times -> Context::evaluation_schedule Co-Authored-By: Claude Opus 5.5 --- .../concepts/memo-groups-properties-requirements.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index faed1f308..952ee15fc 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -482,8 +482,11 @@ pub enum Context { IngestionPipeline(RefreshWindowId), // I_r Materialized(PhysicalCandidateId), // κ_m: inside a materialized m's computation } -/// T_κ: when the context runs; y_{κ,u,c,i} exists only if `p^phys_u.delivers_on(T_κ)`. -fn context_times(context: &Context) -> Option; +/// T_κ: when the context runs; y_{κ,u,c,i} exists only if `p^phys_u.delivers_on(κ.evaluation_schedule())`. +impl Context { + /// The context's evaluation schedule T_κ (None for a non-periodic query). + pub fn evaluation_schedule(&self) -> Option; +} struct SelectionVariables { x_materialized: Vec, // x_v, ℓ ≠ None From 2990b0536097feabbce93f00f63238bf547def4e Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 02:50:43 +0000 Subject: [PATCH 18/67] docs(concepts): constraints, not rows (constraint (K), *_constraints functions) Co-Authored-By: Claude Opus 5.5 --- .../memo-groups-properties-requirements.md | 64 +++++++++---------- 1 file changed, 32 insertions(+), 32 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 952ee15fc..e002301cc 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -26,7 +26,7 @@ This page defines the memo's data structures in the terms of the optimization pr | $P_{c,i}$ | physical candidates in the required input group, $G_{c,i} \cap \mathcal C$, before checking requirements | | $\mathcal E$ | potential dependencies $\{(u, c, i) \mid c \in \mathcal C,\ i \in \mathcal I(c),\ u \in P_{c,i}\}$ | | $x_v$, $y_{u,c,i}$, $z_{q,v}$, $t_v$ | MILP variables: candidate selected; producer $u$ supplies input $i$ of $c$; $v \in G_q$ supplies $q$'s result; topological rank | -| (A)–(I) | MILP rows: input satisfaction, producer selection and sharing, query coverage, output selection, acyclicity, utilization, logical requirements ($y \le \alpha^{\log}$), physical requirements ($y \le \alpha^{\mathrm{phys}}$), domains | +| (A)–(I) | MILP constraints: input satisfaction, producer selection and sharing, query coverage, output selection, acyclicity, utilization, logical requirements ($y \le \alpha^{\log}$), physical requirements ($y \le \alpha^{\mathrm{phys}}$), domains | | $w_v$ | fixed cost: $C_{\text{read}}(v) + C_{\text{compute}}(v)$ plus $C^{\text{one}}_{\text{retain}}(v)$ or $C^{\text{across}}_{\text{retain}}(v)$ by $\ell_v$ | | $\Pi$, $V_\Pi$ | selected physical ASAP DAG and its node set; $\operatorname{Cost}(\Pi) = \sum_{v \in \mathcal C} w_v x_v$ | | **(#1207)** $\kappa$ | execution context of a candidate with $\ell_v$ = None: a query evaluation $\kappa_q$, an ingestion pipeline $I_r$, or a materialized candidate's computation $\kappa_m$ | @@ -95,7 +95,7 @@ Valid $(\tau, \ell)$: (QueryTime, None), (IngestionTime, None), (QueryTime, OneE Each input $i$ of consumer $c$ names a producer group $G_{c,i}$ and a property requirement $r_{c,i} = (r^{\log}_{c,i}, r^{\mathrm{phys}}_{c,i})$. A requirement holds only what neither $G_{c,i}$'s identity nor the consumer's own properties already fix (decided 2026-11-05): the family and the query window are in $G_{c,i}$'s identity, and $r^{\mathrm{phys}}_{c,i}$ is a function of the consumer's own $p^{\mathrm{phys}}_c$, not a stored value. Logical satisfaction checks configuration requirements; physical satisfaction checks whether the producer's timing and lifetime jointly support the required access. For a monotone size parameter, $\succeq$ reduces to numerical dominance, e.g. $k_u \ge k_{\text{required}}$ for KLL. -The workload's accuracy target $A_q$ is not an input requirement **(#1207)**: a per-input bound would fix each input's share of the query's budget. Row (K) bounds each query's composed error at its root instead. What stays in $r^{\log}$ is what depends only on the consumer. +The workload's accuracy target $A_q$ is not an input requirement **(#1207)**: a per-input bound would fix each input's share of the query's budget. Constraint (K) bounds each query's composed error at its root instead. What stays in $r^{\log}$ is what depends only on the consumer. #### Error propagation and the error cap @@ -116,35 +116,35 @@ Each $g_c$ is a sum of its inputs' errors, each times a constant **propagation f $$\lambda_{q,c,i} = \sum_{\text{paths } c = c_0 \to c_1 \to \dots \to c_m = \text{root of } q} \lambda_{c_0,i}\,\lambda_{c_1,i_1} \cdots \lambda_{c_m,i_m}.$$ -Row (K) is $\sum \lambda_{q,c,i}\,\varepsilon_u\,y \le \varepsilon_q$ with these factors, so the weights sit on $(q, c, i)$. Two queries reading one group through different expressions have different factors. Failure probabilities add by a union bound and need no factor. +Constraint (K) is $\sum \lambda_{q,c,i}\,\varepsilon_u\,y \le \varepsilon_q$ with these factors, so the weights sit on $(q, c, i)$. Two queries reading one group through different expressions have different factors. Failure probabilities add by a union bound and need no factor. From $g_c$ come two caps on an input's error, both checked per edge as part of $r^{\log}_{c,i}$: -1. **Validity cap: correctness, non-linear consumers only.** A non-linear $g_c$ is bounded by its linear form only in a region, e.g. $\varepsilon_Y \le$ cap for a ratio, where $\frac{\varepsilon_X+\varepsilon_Y}{1-\varepsilon_Y} \le \frac{1}{1-\text{cap}}(\varepsilon_X+\varepsilon_Y)$, so both factors are $\frac{1}{1-\text{cap}}$ (1.11 for cap 0.1). Without the cap, row (K) could accept a plan whose true error is unbounded. A linear $g_c$ is exact everywhere and has no validity cap. -2. **Pruning cap: efficiency only, every consumer, linear ones included.** An input whose error alone breaks every target it could serve is in no feasible plan: $\lambda_{q,c,i}\,\varepsilon_u > \varepsilon_q$ for every query $q$ that may reach $c$. Removing such edges removes no feasible plan, since row (K) rejects them anyway; it keeps them out of $\mathcal E$ and the MILP. It is the per-edge form of Pass 1's "drop a $\boldsymbol\theta$ whose $\varepsilon$ alone exceeds every reading query's target", and prunes more because it uses each consumer's own $\lambda$. +1. **Validity cap: correctness, non-linear consumers only.** A non-linear $g_c$ is bounded by its linear form only in a region, e.g. $\varepsilon_Y \le$ cap for a ratio, where $\frac{\varepsilon_X+\varepsilon_Y}{1-\varepsilon_Y} \le \frac{1}{1-\text{cap}}(\varepsilon_X+\varepsilon_Y)$, so both factors are $\frac{1}{1-\text{cap}}$ (1.11 for cap 0.1). Without the cap, constraint (K) could accept a plan whose true error is unbounded. A linear $g_c$ is exact everywhere and has no validity cap. +2. **Pruning cap: efficiency only, every consumer, linear ones included.** An input whose error alone breaks every target it could serve is in no feasible plan: $\lambda_{q,c,i}\,\varepsilon_u > \varepsilon_q$ for every query $q$ that may reach $c$. Removing such edges removes no feasible plan, since constraint (K) rejects them anyway; it keeps them out of $\mathcal E$ and the MILP. It is the per-edge form of Pass 1's "drop a $\boldsymbol\theta$ whose $\varepsilon$ alone exceeds every reading query's target", and prunes more because it uses each consumer's own $\lambda$. $$\text{cap}_{c,i} = \min\Bigl(\text{validity cap of } g_c,\ \max_{q \text{ that may reach } c} \frac{\varepsilon_q}{\lambda_{q,c,i}}\Bigr), \qquad \alpha^{\log}_{u,c,i} = 0 \text{ when } \varepsilon_u > \text{cap}_{c,i}.$$ For a monotone size parameter, $\varepsilon_u \le \text{cap}_{c,i}$ is a minimum size, e.g. $k_u \ge k_{\text{required}}$ for KLL. -**What stays at the root.** A per-input cap cannot replace row (K): with several inputs, $\varepsilon_X + \varepsilon_Y \le \varepsilon_q$ couples them, and one cap per input would fix each input's share in advance (the rejected budget shares). The caps only prune; row (K) decides. +**What stays at the root.** A per-input cap cannot replace constraint (K): with several inputs, $\varepsilon_X + \varepsilon_Y \le \varepsilon_q$ couples them, and one cap per input would fix each input's share in advance (the rejected budget shares). The caps only prune; constraint (K) decides. | Requirement | Stated by | Values | Constrains | Checked by | |---|---|---|---|---| -| Accuracy target $A_q$ | the workload, per query | metric: absolute (in the value's units, or times the stream's L1 or L2 norm), relative, rank; bound $\varepsilon_q$; failure probability $\delta_q$; scope: per value, whole result, or groups above a share of the records | $\boldsymbol\theta_u$ of every approximate candidate $q$ depends on, through $(\varepsilon_u, \delta_u)$ | row (K) **(#1207)** | -| Parameter floor $\boldsymbol\theta^{\min}_{c,i}$, the only field of $r^{\log}_{c,i}$ | every consumer, per input, from $g_c$ and the targets of the queries that may reach it | the smallest $\boldsymbol\theta$ in the family's predefined list whose $\varepsilon$, for the statistic $c$ reads, is at most $\text{cap}_{c,i}$ = min(validity cap, pruning cap); e.g. $k_u \ge k_{\text{required}}$ | $\boldsymbol\theta_u$ | $\alpha^{\log}$, row (G): $\boldsymbol\theta_u \succeq \boldsymbol\theta^{\min}_{c,i}$ | -| Consumer timing, $r^{\mathrm{phys}}_{c,i}$ from $\tau_c$ | the consumer's own $p^{\mathrm{phys}}_c$ | when $c$ runs: ingestion or query time | $\tau_u$, $\ell_u$ | $\alpha^{\mathrm{phys}}$, row (H) | -| Consumer schedule, $r^{\mathrm{phys}}_{c,i}$ from $\sigma_c$ or $\kappa$ **(#1207)** | a materialized consumer's own $\sigma_c$; for $\ell_c$ = None, the times $T_\kappa$ of each context $\kappa$ it runs in (from its query's $\mathcal T_q$) | interval and phase | $\sigma_u$: the producer delivers on those times | $\alpha^{\mathrm{phys}}$, row (H): in Stage 2 for a materialized consumer; per context in Stage 3 for $\ell_c$ = None | +| Accuracy target $A_q$ | the workload, per query | metric: absolute (in the value's units, or times the stream's L1 or L2 norm), relative, rank; bound $\varepsilon_q$; failure probability $\delta_q$; scope: per value, whole result, or groups above a share of the records | $\boldsymbol\theta_u$ of every approximate candidate $q$ depends on, through $(\varepsilon_u, \delta_u)$ | constraint (K) **(#1207)** | +| Parameter floor $\boldsymbol\theta^{\min}_{c,i}$, the only field of $r^{\log}_{c,i}$ | every consumer, per input, from $g_c$ and the targets of the queries that may reach it | the smallest $\boldsymbol\theta$ in the family's predefined list whose $\varepsilon$, for the statistic $c$ reads, is at most $\text{cap}_{c,i}$ = min(validity cap, pruning cap); e.g. $k_u \ge k_{\text{required}}$ | $\boldsymbol\theta_u$ | $\alpha^{\log}$, constraint (G): $\boldsymbol\theta_u \succeq \boldsymbol\theta^{\min}_{c,i}$ | +| Consumer timing, $r^{\mathrm{phys}}_{c,i}$ from $\tau_c$ | the consumer's own $p^{\mathrm{phys}}_c$ | when $c$ runs: ingestion or query time | $\tau_u$, $\ell_u$ | $\alpha^{\mathrm{phys}}$, constraint (H) | +| Consumer schedule, $r^{\mathrm{phys}}_{c,i}$ from $\sigma_c$ or $\kappa$ **(#1207)** | a materialized consumer's own $\sigma_c$; for $\ell_c$ = None, the times $T_\kappa$ of each context $\kappa$ it runs in (from its query's $\mathcal T_q$) | interval and phase | $\sigma_u$: the producer delivers on those times | $\alpha^{\mathrm{phys}}$, constraint (H): in Stage 2 for a materialized consumer; per context in Stage 3 for $\ell_c$ = None | | Deployment capability | the deployment and data | the source has an ingestion pipeline; the executor supports the family; query-time results may be kept across evaluations | $\tau$, $\operatorname{op}$, (QueryTime, AcrossEvaluations) | candidate generation: such candidates are not in $\mathcal C$ | -| Latency, resource budgets **(#1207)** | the workload | per-query latency bound; memory and CPU budgets | the whole plan | rows (L) and budgets | +| Latency, resource budgets **(#1207)** | the workload | per-query latency bound; memory and CPU budgets | the whole plan | constraints (L) and budgets | Accuracy metrics that may be added later, when a primitive needs them: top-k recall or precision, and a false-positive rate for membership summaries. How requirements map to properties: ``` -A_q (workload) ──► θ_u of every approximate input (through ε_u, δ_u; row (K)) +A_q (workload) ──► θ_u of every approximate input (through ε_u, δ_u; constraint (K)) parameter floor θ^min r^log ──► θ_u (from cap = min(validity cap of g_c, pruning cap ε_q / λ_{q,c,i})) τ_c r^phys = f(p^phys_c) ──► τ_u, ℓ_u σ_c or T_κ r^phys = f(p^phys_c) ──► σ_u @@ -240,7 +240,7 @@ The levels, top to bottom: What the tree does not decide, because it is a choice of the MILP over $\mathcal C$ and $\mathcal E$: - **Which producer each consumer input reads:** $y_{u,c,i}$. Two consumers of one group may read different leaves, e.g. one reads `k = 200` and another `k = 345`. - **Where an unmaterialized candidate runs (#1207):** a candidate with $\ell_v$ = None runs inside each reading context $\kappa$. -- **How much a materialized candidate retains (#1207):** $h_v$, from the longest range among the queries its selected consumers serve, row (R). +- **How much a materialized candidate retains (#1207):** $h_v$, from the longest range among the queries its selected consumers serve, constraint (R). ### Order of decisions @@ -268,7 +268,7 @@ This section gives the target interface of each stage after the formulation refa | Stage 1, Pass 1 | `DagOfLogicalMemoGroups` | `Memo` | $v = (e_v, p^{\log}_v)$, $r^{\log}_{c,i}$, $\theta$ list | #1203 | | Stage 1, Pass 2 | `Memo` | `LogicalASAPMemoCandidates` | rule-added candidates; logical edges with $\alpha^{\log}$ | #1203, #1201 | | Stage 2 | `LogicalASAPMemoCandidates` | `PhysicalASAPMemoCandidates` | $p^{\mathrm{phys}}_v = (\tau, \ell, \sigma)$, $r^{\mathrm{phys}}_{c,i}$, physical edges $\mathcal E$ with $\alpha^{\mathrm{phys}}$ | #1200, #1201 | -| Stage 3 | `PhysicalASAPMemoCandidates` | `DagOfSelectedCandidates` | contexts $\kappa$; cost $w_v$ (`asap-cost`); $x$, $y$, $z$, $t$, $h$; rows (A)–(K), (R), (L), budgets | #1204 | +| Stage 3 | `PhysicalASAPMemoCandidates` | `DagOfSelectedCandidates` | contexts $\kappa$; cost $w_v$ (`asap-cost`); $x$, $y$, $z$, $t$, $h$; constraints (A)–(K), (R), (L), budgets | #1204 | | Realizer, export | `DagOfSelectedCandidates` | `PhysicalASAPDAG` | — | #1200 (export fields) | ### Stage 0: frontends and hash-consing @@ -470,7 +470,7 @@ Deleted: `Materialization`, `PhysicalCandidate::{timing, materialization, operat ### Stage 3: the MILP -Stage 3 reads the physical candidates and edges and selects the DAG. Variables and rows are named after the formulation. +Stage 3 reads the physical candidates and edges and selects the DAG. Variables and constraints are named after the formulation. ```rust // crates/plan-selection/src/milp/model.rs @@ -497,16 +497,16 @@ struct SelectionVariables { h: Vec, // h_v, ℓ ≠ None latency: LatencyVariables, // (L) } -fn input_satisfaction_rows(..); // (A) Σ_u y_{u,c,i} = x_c -fn producer_selection_rows(..); // (B) y_{u,c,i} ≤ x_u -fn query_coverage_rows(..); // (C), (D) -fn acyclicity_rows(..); // (E) -fn utilization_rows(..); // (F) -fn consumer_constraint_rows(..); // (J): MinimumQueryReaders, HighestSharedQueryMaterialization over reach -fn accuracy_rows(..); // (K) Σ λ_{q,u} ε_u a_{q,u} ≤ ε_q, Σ δ_u a_{q,u} ≤ δ_q -fn retention_rows(..); // (R) h_v ≥ need_{c,i} · y_{v,c,i} -fn latency_rows(..); // (L) -fn resource_budget_rows(..); // budgets (#1207) +fn input_satisfaction_constraints(..); // (A) Σ_u y_{u,c,i} = x_c +fn producer_selection_constraints(..); // (B) y_{u,c,i} ≤ x_u +fn query_coverage_constraints(..); // (C), (D) +fn acyclicity_constraints(..); // (E) +fn utilization_constraints(..); // (F) +fn consumer_constraint_constraints(..); // (J): MinimumQueryReaders, HighestSharedQueryMaterialization over reach +fn accuracy_constraints(..); // (K) Σ λ_{q,u} ε_u a_{q,u} ≤ ε_q, Σ δ_u a_{q,u} ≤ δ_q +fn retention_constraints(..); // (R) h_v ≥ need_{c,i} · y_{v,c,i} +fn latency_constraints(..); // (L) +fn resource_budget_constraints(..); // budgets (#1207) // Objective: min Σ w_v x_v + Σ ρ_κ w_v x_{κ,v} + Σ π_v h_v // crates/types/src/stages/physical/graph.rs (unchanged) @@ -518,7 +518,7 @@ pub struct DagOfSelectedCandidates { } ``` -(G) $y \le \alpha^{\log}$ and (H) $y \le \alpha^{\mathrm{phys}}$ are not rows: $y$ exists only on admissible edges. Deleted: `ResultSlot`, `slot_of`, the `e` binaries, schedule groups `z[m, G]`, `N[q]`, `A[q, parts]`, and `p[ctx, c]` for materialized candidates. +(G) $y \le \alpha^{\log}$ and (H) $y \le \alpha^{\mathrm{phys}}$ are not constraints in the model: $y$ exists only on admissible edges. Deleted: `ResultSlot`, `slot_of`, the `e` binaries, schedule groups `z[m, G]`, `N[q]`, `A[q, parts]`, and `p[ctx, c]` for materialized candidates. ### Realizer and export @@ -594,9 +594,9 @@ A memo group stores its candidates and, through their expressions, the groups th A logical DAG rooted at group $G$ is a set of logical candidates and edges $(u, c, i)$ such that: 1. exactly one candidate of $G$ is in it (the root); -2. every candidate $c$ in it has, for each input $i$, exactly one edge $(u, c, i)$, with $u$ a candidate of $G_{c,i}$ and $\alpha^{\log}_{u,c,i} = 1$ (rows (A) and (G)); -3. every candidate in it is the root or the producer of some edge (row (F)); -4. the edges form no cycle (row (E)). +2. every candidate $c$ in it has, for each input $i$, exactly one edge $(u, c, i)$, with $u$ a candidate of $G_{c,i}$ and $\alpha^{\log}_{u,c,i} = 1$ (constraints (A) and (G)); +3. every candidate in it is the root or the producer of some edge (constraint (F)); +4. the edges form no cycle (constraint (E)). A candidate is one node however many edges read it: two consumers that choose the same producer share it, which is what $x_u \in \{0, 1\}$ says. Two consumers of one group may choose different candidates of it. The listing is therefore a choice of one producer per consumer input $(c, i)$, not one candidate per group. @@ -637,7 +637,7 @@ fn expand(memo: &Memo, dag: LogicalDag, mut pending: VecDeque>() })) { - // Row (E): a producer that already reaches `consumer` would close a cycle. + // Constraint (E): a producer that already reaches `consumer` would close a cycle. if choice.iter().any(|&u| reaches(&dag, u, consumer)) { continue; } @@ -681,7 +681,7 @@ fn print(memo: &Memo, dag: &LogicalDag) { } ``` -For a workload, list from every query root at once: start `pending` with one root candidate per $G_q$ (rows (C) and (D)), taking the product over queries of their root candidates. A candidate two queries reach is one node, as above. +For a workload, list from every query root at once: start `pending` with one root candidate per $G_q$ (constraints (C) and (D)), taking the product over queries of their root candidates. A candidate two queries reach is one node, as above. ### Example @@ -698,7 +698,7 @@ G3[0] SummaryEstimate q=0.95 G1[0] Scan … and (k=200, k=345), (k=345, k=200), (k=345, k=345) ``` -Listed from both roots at once there are $2 \times 4 = 8$ DAGs. In the one where all three estimates choose `k = 345`, `G2[1]` is a single node read by three consumers, so its cost is counted once. Row (K) later accepts only the DAGs that meet each query's target: for B, only (k=345, k=345). Error caps (section 1) remove edges before the listing, so a capped producer never appears. +Listed from both roots at once there are $2 \times 4 = 8$ DAGs. In the one where all three estimates choose `k = 345`, `G2[1]` is a single node read by three consumers, so its cost is counted once. Constraint (K) later accepts only the DAGs that meet each query's target: for B, only (k=345, k=345). Error caps (section 1) remove edges before the listing, so a capped producer never appears. ### Size From 294066a8187d787c09c936d5795b694e2f9b0634 Mon Sep 17 00:00:00 2001 From: Zeying Zhu <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 22:04:01 -0500 Subject: [PATCH 19/67] Update memo-groups-properties-requirements.md --- .../memo-groups-properties-requirements.md | 960 ++++++++++-------- 1 file changed, 518 insertions(+), 442 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index e002301cc..ca3c51934 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -1,290 +1,179 @@ # Memo groups, properties and requirements -This page defines the memo's data structures in the terms of the optimization problem formulation: workloads, memo groups, candidates, properties and property requirements. It also gives the decision tree Stage 1 and Stage 2 walk to enumerate candidates, and each stage's interfaces (section 3). Where tracking issue #1207 extends the formulation, the extension is marked **(#1207)**. +This document explains how the optimizer represents alternatives, checks whether they can be composed, and selects an executable plan. It follows the supplied **Optimization Problem Formulation** for the baseline model and labels additional design work **Extension (#1207)**. -## Notation +The Rust blocks are interface sketches, not standalone compilable implementations. Repository paths and issue references are retained from the supplied draft; their current implementation status has not been independently checked. -| Notation | Definition | -|---|---| -| $\mathcal W = \{(q, \mathcal T_q, A_q) \mid q \in \mathcal Q\}$ | workload over an evaluation horizon $H$: each query expression $q$, its evaluation schedule $\mathcal T_q \subseteq [0, H)$, and its accuracy target $A_q$ | -| $\mathcal Q$ | set of queries, each with one final output | -| $G$ | memo group: a logical result or an intermediate state family over specified input data, independent of parameter configurations | -| $G_q$ | memo group of query $q$'s final result | -| $v = (e_v, p_v)$ | candidate of a memo group: an expression with its properties | -| $e_v = \operatorname{op}_v(G_{v,1}, \dots, G_{v,n_v})$ | expression: operator $\operatorname{op}_v$ with its operator arguments (e.g. the summary family), over input memo groups $G_{v,i}$ | -| $p_v = (p^{\log}_v, p^{\mathrm{phys}}_v)$ | properties: logical and physical | -| $p^{\log}_v = \boldsymbol\theta_v$ | summary algorithm parameters, e.g. sketch size; **(#1207)** also the window layout $\omega_v$ | -| $p^{\mathrm{phys}}_v = (\tau_v, \ell_v)$ | execution timing and lifetime; **(#1207)** also the schedule $\sigma_v$ when $\ell_v \ne$ None | -| $(\varepsilon_v, \delta_v)$ | error bound and failure probability of $\boldsymbol\theta_v$ for the statistic $\operatorname{op}_v$ reads; derived, not a property | -| $c \in G_c$, $u \in G_u$ | consumer candidate and its group; producer candidate and its group. This page says *consumer* for $c$ and *query* $q$ (or its execution context $\kappa$) for who needs a result on a schedule; it does not use "reader" | -| $\mathcal I(c)$ | input indices of consumer $c$ | -| $G_{c,i}$ | memo group required by input $i$ of consumer $c$ | -| $r_{c,i} = (r^{\log}_{c,i}, r^{\mathrm{phys}}_{c,i})$ | property requirement of input $i$ of consumer $c$; $r^{\mathrm{phys}}_{c,i}$ is a function of $p^{\mathrm{phys}}_c$ | -| $p(u) \succeq r_{c,i}$ | producer $u$'s properties satisfy the requirement: $p^{\log}(u) \succeq r^{\log}_{c,i} \land p^{\mathrm{phys}}(u) \succeq r^{\mathrm{phys}}_{c,i}$ | -| $\alpha^{\log}_{u,c,i}$, $\alpha^{\mathrm{phys}}_{u,c,i}$ | fixed indicators of $p^{\log}_u \succeq r^{\log}_{c,i}$ and $p^{\mathrm{phys}}_u \succeq r^{\mathrm{phys}}_{c,i}$ | -| $\mathcal C$ | physical candidates, each with a fixed parameter configuration, timing and lifetime | -| $P_{c,i}$ | physical candidates in the required input group, $G_{c,i} \cap \mathcal C$, before checking requirements | -| $\mathcal E$ | potential dependencies $\{(u, c, i) \mid c \in \mathcal C,\ i \in \mathcal I(c),\ u \in P_{c,i}\}$ | -| $x_v$, $y_{u,c,i}$, $z_{q,v}$, $t_v$ | MILP variables: candidate selected; producer $u$ supplies input $i$ of $c$; $v \in G_q$ supplies $q$'s result; topological rank | -| (A)–(I) | MILP constraints: input satisfaction, producer selection and sharing, query coverage, output selection, acyclicity, utilization, logical requirements ($y \le \alpha^{\log}$), physical requirements ($y \le \alpha^{\mathrm{phys}}$), domains | -| $w_v$ | fixed cost: $C_{\text{read}}(v) + C_{\text{compute}}(v)$ plus $C^{\text{one}}_{\text{retain}}(v)$ or $C^{\text{across}}_{\text{retain}}(v)$ by $\ell_v$ | -| $\Pi$, $V_\Pi$ | selected physical ASAP DAG and its node set; $\operatorname{Cost}(\Pi) = \sum_{v \in \mathcal C} w_v x_v$ | -| **(#1207)** $\kappa$ | execution context of a candidate with $\ell_v$ = None: a query evaluation $\kappa_q$, an ingestion pipeline $I_r$, or a materialized candidate's computation $\kappa_m$ | -| $g_c$ | error propagation function of consumer $c$: $\varepsilon_c \le g_c(\varepsilon_{c,1}, \dots, \varepsilon_{c,n})$ | -| $\lambda_{q,c,i}$ | propagation factor of input $i$ of $c$ to query $q$'s result, read off $q$'s expression | -| $\text{cap}_{c,i}$, $\boldsymbol\theta^{\min}_{c,i}$ | error cap of input $i$ of $c$ (the smaller of $g_c$'s validity cap and the pruning cap), and the parameter floor $r^{\log}_{c,i}$ stores: the smallest listed $\boldsymbol\theta$ whose $\varepsilon$ is at most the cap | -| **(#1207)** (K), (R), (L), budgets | accuracy at each query root, $\sum_u \lambda_{q,u}\varepsilon_u a_{q,u} \le \varepsilon_q$ and $\sum_u \delta_u a_{q,u} \le \delta_q$; retention $h_v$; latency; resource budgets | - -## Example - -Workload: query A, `quantile_over_time(0.95, latency[1m])`, and query B, `quantile_over_time(0.99, latency[1m]) / quantile_over_time(0.50, latency[1m])`. Both are evaluated every 10 s ($\mathcal T_A = \mathcal T_B = 10\,\text{s}\cdot\mathbb Z$) with accuracy target $A_q$: rank error 0.01. +## 1. Model and running example -``` -G1 Scan latency[1m] - candidate: v1 = Scan p^phys = (QueryTime, None) -G2 KLL state over G1 one group, whatever θ - candidates: v2 = SummaryAgg[KLL](G1) θ = {k = 200} ε = 0.008 δ = 0.01 - v3 = SummaryAgg[KLL](G1) θ = {k = 345} ε = 0.005 δ = 0.01 - (Stage 2 expands each into its (τ, ℓ) alternatives) -G3 = G_A SummaryEstimate[q=0.95](G2) -G4 SummaryEstimate[q=0.99](G2) -G5 SummaryEstimate[q=0.50](G2) -G6 = G_B Divide(G4, G5) r^log on each input: cap = min(validity cap of /, ε_B / λ) -``` +A **workload** contains query expressions, their evaluation schedules, and their accuracy targets. A **memo group** represents one logical result or one intermediate state family over specified input data. A **candidate** is one way to compute that result: one operator over input groups, together with its properties. -The ε values are illustrative. +Input groups leave producer choices open. Selecting a candidate does not select its entire input plan. Each consumer input chooses a producer separately, and several inputs may choose the same producer. -- G2 is one memo group: the KLL state over `latency[1m]`. Its parameter configurations are candidates of G2, not separate groups. The family KLL is an argument of $\operatorname{op}_v$. -- G3, G4 and G5 are consumers of G2: their candidates' expressions read G2 as $G_{c,1}$. G2 does not store them. -- A meets 0.01 with either θ. B adds its two operands' errors (0.016 with `k = 200`, 0.010 with `k = 345`) and needs `k = 345` for both, so a plan where G3's, G4's and G5's candidates all read v3 builds one KLL state. +The expression includes the operator's arguments, including the summary family. In this design, KLL and DDSketch states belong to separate state groups. A final-result group can offer readout candidates that use either family. This resolves the old decision tree's contradiction: it must not put KLL and DDSketch builds in one state group. -## 1. Properties and property requirements +### Running example -### Properties +Query A requests the p95 latency over one minute. Query B requests p99 divided by p50 over the same input. Both run every ten seconds. -A property is a value a candidate has. Properties are products of dimensions: one field per dimension, never a flat enum that lists the combinations. Invalid combinations are rejected by the checked constructor. +| Group | Result | Candidate expressions | +|---|---|---| +| G1 | Raw latency input | `Scan latency[1m]` | +| G2 | KLL state over G1 | `SummaryAgg[KLL](G1)` with `k = 200` or `k = 345` | +| G3 | Query A output | `SummaryEstimate[q=0.95](G2)` | +| G4 | p99 latency | `SummaryEstimate[q=0.99](G2)` | +| G5 | p50 latency | `SummaryEstimate[q=0.50](G2)` | +| G6 | Query B output | `Divide(G4, G5)` | -What is computed is the expression $e_v$, not a property: the operator, its arguments (including the summary family), and its input groups. A memo group's candidates differ in their properties, and, when rules add other ways to compute the same result, in their expressions. +G3, G4, and G5 choose a G2 producer independently. If all choose the same retained physical producer, they can reuse one computed state when their schedules and windows are compatible. An unmaterialized candidate may instead execute separately in different contexts; Section 5 explains that distinction. -| Dimension | Values | Part of | Code | -|---|---|---|---| -| Parameter configuration $\boldsymbol\theta_v$ | per family, from a predefined list: KLL `k`; DDSketch `alpha`; CMS / CountSketch `width × depth`; CMS / CountSketch with heap `width × depth × heap_size`; UnivMon `heap_size × rows × cols × layers`; HLL `precision`; KMV `k`. None for an exact operator | $p^{\log}_v$ | `SketchParams` inside `SketchKind` ([`sketch.rs`](../../../crates/types/src/ir/schema/sketch.rs)), sized by `size_params` per `accuracy_budget_parts` today | -| **(#1207)** Window layout $\omega_v$ | none; or tumbling windows of width $w$, from the gcd closure of the ranges and steps of the queries that reach the group. The window count is not a property: it is the group's query range divided by $w$, and how many windows to retain is $h_v$ (Stage 3). Later: sliding windows by merge or by subtraction | $p^{\log}_v$ | `SummaryTumblingWindowChain` arguments, added by Pass 2 ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs)) | -| Execution timing $\tau_v$ | IngestionTime, QueryTime | $p^{\mathrm{phys}}_v$ | `ExecutionTiming` ([`execution_data_state.rs`](../../../crates/types/src/ir/properties/execution_data_state.rs)) | -| Lifetime $\ell_v$ | None, OneEvaluation, AcrossEvaluations; **(#1207)** a materialized lifetime carries its medium, Memory or Disk | $p^{\mathrm{phys}}_v$ | `Materialization` on main; `Lifetime`, `Medium` in #1214 ([`physical/`](../../../crates/types/src/physical)) | -| **(#1207)** Schedule $\sigma_v$ | only when $\ell_v \ne$ None: Periodic (interval, phase) for OneEvaluation; tumbling-window grid phase for AcrossEvaluations | $p^{\mathrm{phys}}_v$ | schedule groups `z[m, G]` in the MILP today ([`cost_terms.rs`](../../../crates/plan-selection/src/milp/cost_terms.rs)); a property in #1200 PR 2 | +The old example assigned illustrative KLL rank errors of 0.008 and 0.005 and added them to obtain Query B's ratio error. That inference is invalid: rank error does not directly bound numerical quantile error or relative ratio error. The example here demonstrates representation and sharing only. Accepting Query B requires an applicable error model, including assumptions about its denominator. -Lifetime None means the output is discarded after its consumer's execution completes. OneEvaluation retains the result until all its consumers within one evaluation finish. AcrossEvaluations retains it throughout the evaluation period, so later evaluations reuse it. A candidate is materialized when $\ell_v \ne$ None; this is derived, not stored. +### Notation used by the baseline model -The error bound $(\varepsilon_v, \delta_v)$ is not a dimension: it is a function of $\boldsymbol\theta_v$ and the statistic $\operatorname{op}_v$ reads, derived when needed (`local_guarantee` in [`estimators`](../../../crates/accuracy/src/estimators/mod.rs)). An exact candidate has $(0, 0)$. +Symbols are collected here and in the sections that use equations. The prose and code use descriptive names. -Invalid combinations of $p^{\mathrm{phys}}$, rejected by `PhysicalProperties::new` (#1200; #1214 implements the first, the others come with $\sigma$ in #1200 PR 2): - -| Combination | Why | +| Symbol | Meaning | |---|---| -| (IngestionTime, OneEvaluation) | there is no evaluation at ingestion time | -| $\ell$ = None with a schedule | where an unmaterialized result runs is its consumer's execution context $\kappa$, not a property | -| $\ell \ne$ None without a schedule | a materialized result must say on which times or grid it is delivered | -| OneEvaluation with a grid, AcrossEvaluations with a periodic schedule | each lifetime has one kind of schedule | -| a phase at or above the grid width or interval, or interval 0 | the phase is taken modulo the width or interval | -| IngestionTime with a grid phase ≠ 0 | ingestion-time grids are anchored at the epoch today: the plan carries no origin | - -Valid $(\tau, \ell)$: (QueryTime, None), (IngestionTime, None), (QueryTime, OneEvaluation), (QueryTime, AcrossEvaluations), (IngestionTime, AcrossEvaluations), each materialized one for each medium. +| $\mathcal W$, $\mathcal Q$, $H$ | Workload, query set, and common evaluation horizon | +| $\mathcal T_q$, $A_q$ | Query evaluation schedule and accuracy target | +| $G$, $G_q$ | Memo group and query-output group | +| $v$, $u$, $c$, $i$ | Candidate, producer candidate, consumer candidate, and consumer input index | +| $e_v$, $p_v$ | Candidate expression and properties | +| $\boldsymbol\theta_v$, $\tau_v$, $\ell_v$ | Parameter configuration, execution timing, and result lifetime | +| $\mathcal I(c)$, $G_{c,i}$ | Consumer input indices and required group for one input | +| $r_{c,i}$ | Logical and physical requirements of one input | +| $\mathcal C$ | Physical candidate set | +| $\alpha^{\mathrm{log}}_{u,c,i}$, $\alpha^{\mathrm{phys}}_{u,c,i}$ | Fixed logical and physical compatibility indicators | -### Property requirements +The candidate and property definitions are: -Each input $i$ of consumer $c$ names a producer group $G_{c,i}$ and a property requirement $r_{c,i} = (r^{\log}_{c,i}, r^{\mathrm{phys}}_{c,i})$. A requirement holds only what neither $G_{c,i}$'s identity nor the consumer's own properties already fix (decided 2026-11-05): the family and the query window are in $G_{c,i}$'s identity, and $r^{\mathrm{phys}}_{c,i}$ is a function of the consumer's own $p^{\mathrm{phys}}_c$, not a stored value. Logical satisfaction checks configuration requirements; physical satisfaction checks whether the producer's timing and lifetime jointly support the required access. For a monotone size parameter, $\succeq$ reduces to numerical dominance, e.g. $k_u \ge k_{\text{required}}$ for KLL. +$$ +v=(e_v,p_v), \qquad e_v=\operatorname{op}_v(G_{v,1},\ldots,G_{v,n_v}). +$$ -The workload's accuracy target $A_q$ is not an input requirement **(#1207)**: a per-input bound would fix each input's share of the query's budget. Constraint (K) bounds each query's composed error at its root instead. What stays in $r^{\log}$ is what depends only on the consumer. +$$ +p_v=(p_v^{\mathrm{log}},p_v^{\mathrm{phys}}),\qquad +p_v^{\mathrm{log}}=\boldsymbol\theta_v,\qquad +p_v^{\mathrm{phys}}=(\tau_v,\ell_v). +$$ -#### Error propagation and the error cap +The supplied LaTeX uses the producer-set symbol in two different ways: the prose defines a filtered set, while its notation table defines an unfiltered set. This document follows the table and constraints (G)–(H), and gives the filtered set a separate name: -Each consumer $c$ has an error propagation function $g_c$ that bounds its output error by its input errors: +$$ +P_{c,i}=G_{c,i}\cap\mathcal C, +\qquad +P^{\mathrm{adm}}_{c,i} +=\{u\in P_{c,i}:\alpha^{\mathrm{log}}_{u,c,i}=\alpha^{\mathrm{phys}}_{u,c,i}=1\}. +$$ -$$\varepsilon_c \le g_c(\varepsilon_{c,1}, \dots, \varepsilon_{c,n}).$$ +$$ +\mathcal E=\{(u,c,i):c\in\mathcal C,\ i\in\mathcal I(c),\ u\in P_{c,i}\}. +$$ -Composing them from each query's root down gives the query's error, and the accuracy target $A_q$ is $g_q(\dots) \le \varepsilon_q$ at the root. - -| Consumer | $g_c$ | Linear | -|---|---|---| -| single readout, `Project`, filter | $\varepsilon_{c,1}$ | yes | -| `sum by` over $n$ series (absolute error) | $n\,\varepsilon_{c,1}$ | yes | -| $X + Y$, $X - Y$ (absolute error) | $\varepsilon_X + \varepsilon_Y$ | yes | -| $X / Y$ (relative error) | $\dfrac{\varepsilon_X + \varepsilon_Y}{1 - \varepsilon_Y}$ | no | +Here, group membership denotes the physical candidates belonging to that group. Property satisfaction means that both checks succeed; it is not a universal numerical ordering. -Each $g_c$ is a sum of its inputs' errors, each times a constant **propagation factor** $\lambda_{c,i}$: $g_c = \sum_i \lambda_{c,i}\,\varepsilon_{c,i}$ (for a non-linear $g_c$, within its validity cap, below). The factors come from the query expression itself: each operator and its arguments, e.g. `sum by` over $n$ series has factor $n$. No measurement or optimization is involved. A query's factor for input $i$ of $c$ multiplies the factors along the path in $q$'s expression from that input to $q$'s root, and adds over paths when the input reaches the root more than once: +## 2. Properties and input requirements -$$\lambda_{q,c,i} = \sum_{\text{paths } c = c_0 \to c_1 \to \dots \to c_m = \text{root of } q} \lambda_{c_0,i}\,\lambda_{c_1,i_1} \cdots \lambda_{c_m,i_m}.$$ +A property describes what a candidate delivers. An input requirement describes what a consumer can accept. Workload requirements such as final accuracy and latency apply to the composed plan. -Constraint (K) is $\sum \lambda_{q,c,i}\,\varepsilon_u\,y \le \varepsilon_q$ with these factors, so the weights sit on $(q, c, i)$. Two queries reading one group through different expressions have different factors. Failure probabilities add by a union bound and need no factor. +### Property dimensions -From $g_c$ come two caps on an input's error, both checked per edge as part of $r^{\log}_{c,i}$: +This is the single reference table for property dimensions. Stage descriptions below explain when these choices are made without repeating their definitions. -1. **Validity cap: correctness, non-linear consumers only.** A non-linear $g_c$ is bounded by its linear form only in a region, e.g. $\varepsilon_Y \le$ cap for a ratio, where $\frac{\varepsilon_X+\varepsilon_Y}{1-\varepsilon_Y} \le \frac{1}{1-\text{cap}}(\varepsilon_X+\varepsilon_Y)$, so both factors are $\frac{1}{1-\text{cap}}$ (1.11 for cap 0.1). Without the cap, constraint (K) could accept a plan whose true error is unbounded. A linear $g_c$ is exact everywhere and has no validity cap. -2. **Pruning cap: efficiency only, every consumer, linear ones included.** An input whose error alone breaks every target it could serve is in no feasible plan: $\lambda_{q,c,i}\,\varepsilon_u > \varepsilon_q$ for every query $q$ that may reach $c$. Removing such edges removes no feasible plan, since constraint (K) rejects them anyway; it keeps them out of $\mathcal E$ and the MILP. It is the per-edge form of Pass 1's "drop a $\boldsymbol\theta$ whose $\varepsilon$ alone exceeds every reading query's target", and prunes more because it uses each consumer's own $\lambda$. - -$$\text{cap}_{c,i} = \min\Bigl(\text{validity cap of } g_c,\ \max_{q \text{ that may reach } c} \frac{\varepsilon_q}{\lambda_{q,c,i}}\Bigr), \qquad \alpha^{\log}_{u,c,i} = 0 \text{ when } \varepsilon_u > \text{cap}_{c,i}.$$ - -For a monotone size parameter, $\varepsilon_u \le \text{cap}_{c,i}$ is a minimum size, e.g. $k_u \ge k_{\text{required}}$ for KLL. - -**What stays at the root.** A per-input cap cannot replace constraint (K): with several inputs, $\varepsilon_X + \varepsilon_Y \le \varepsilon_q$ couples them, and one cap per input would fix each input's share in advance (the rejected budget shares). The caps only prune; constraint (K) decides. - - -| Requirement | Stated by | Values | Constrains | Checked by | -|---|---|---|---|---| -| Accuracy target $A_q$ | the workload, per query | metric: absolute (in the value's units, or times the stream's L1 or L2 norm), relative, rank; bound $\varepsilon_q$; failure probability $\delta_q$; scope: per value, whole result, or groups above a share of the records | $\boldsymbol\theta_u$ of every approximate candidate $q$ depends on, through $(\varepsilon_u, \delta_u)$ | constraint (K) **(#1207)** | -| Parameter floor $\boldsymbol\theta^{\min}_{c,i}$, the only field of $r^{\log}_{c,i}$ | every consumer, per input, from $g_c$ and the targets of the queries that may reach it | the smallest $\boldsymbol\theta$ in the family's predefined list whose $\varepsilon$, for the statistic $c$ reads, is at most $\text{cap}_{c,i}$ = min(validity cap, pruning cap); e.g. $k_u \ge k_{\text{required}}$ | $\boldsymbol\theta_u$ | $\alpha^{\log}$, constraint (G): $\boldsymbol\theta_u \succeq \boldsymbol\theta^{\min}_{c,i}$ | -| Consumer timing, $r^{\mathrm{phys}}_{c,i}$ from $\tau_c$ | the consumer's own $p^{\mathrm{phys}}_c$ | when $c$ runs: ingestion or query time | $\tau_u$, $\ell_u$ | $\alpha^{\mathrm{phys}}$, constraint (H) | -| Consumer schedule, $r^{\mathrm{phys}}_{c,i}$ from $\sigma_c$ or $\kappa$ **(#1207)** | a materialized consumer's own $\sigma_c$; for $\ell_c$ = None, the times $T_\kappa$ of each context $\kappa$ it runs in (from its query's $\mathcal T_q$) | interval and phase | $\sigma_u$: the producer delivers on those times | $\alpha^{\mathrm{phys}}$, constraint (H): in Stage 2 for a materialized consumer; per context in Stage 3 for $\ell_c$ = None | -| Deployment capability | the deployment and data | the source has an ingestion pipeline; the executor supports the family; query-time results may be kept across evaluations | $\tau$, $\operatorname{op}$, (QueryTime, AcrossEvaluations) | candidate generation: such candidates are not in $\mathcal C$ | -| Latency, resource budgets **(#1207)** | the workload | per-query latency bound; memory and CPU budgets | the whole plan | constraints (L) and budgets | +| Dimension | Values and meaning | Code reference or target | +|---|---|---| +| Parameters | A predefined configuration list per family: KLL and KMV `k`; DDSketch `alpha`; HLL `precision`; CMS and CountSketch `width` and `depth`; heap variants also `heap_size`; UnivMon its sizing parameters. Exact operators have no sketch parameters. | `SketchParams` in `crates/types/src/ir/schema/sketch.rs`; predefined lists are the #1203 target | +| Window layout — Extension | No layout, or tumbling windows with a positive width. Candidate widths come from supported divisors derived from workload ranges and steps. Sliding merge/subtraction layouts remain future work. Window count is derived from range and width; retention amount is selected later. | `WindowLayout`; `pass2/memo/window_candidates.rs`; #1160 | +| Timing | `IngestionTime` or `QueryTime` | `ExecutionTiming` | +| Lifetime | `None`: discard after the consuming execution; `OneEvaluation`: retain until consumers within that evaluation finish; `AcrossEvaluations`: retain across evaluations in the specified period. A non-`None` lifetime means materialized. | `Lifetime`; #1200 / #1214 | +| Medium — Extension | `Memory` or `Disk`, carried by a materialized lifetime | `Medium`; #1214 | +| Schedule — Extension | For `OneEvaluation`, a periodic interval and phase; for windowed `AcrossEvaluations`, a tumbling-window grid phase. Unmaterialized candidates have no independent schedule. | `Schedule`; #1200 PR 2 | -Accuracy metrics that may be added later, when a primitive needs them: top-k recall or precision, and a false-positive rate for membership summaries. +Error guarantees are derived from parameters and the statistic being read. They include an error metric, a bound, and a failure probability; exact computation introduces no approximation error. The same state may support different readouts, so its guarantee must be interpreted for the relevant statistic. -How requirements map to properties: +The extended property tuples add window layout and schedule: -``` -A_q (workload) ──► θ_u of every approximate input (through ε_u, δ_u; constraint (K)) -parameter floor θ^min r^log ──► θ_u (from cap = min(validity cap of g_c, pruning cap ε_q / λ_{q,c,i})) -τ_c r^phys = f(p^phys_c) ──► τ_u, ℓ_u -σ_c or T_κ r^phys = f(p^phys_c) ──► σ_u -G_{c,i}'s identity (family, query window) ──► op_u, ω_u (by construction: no requirement) -deployment ──► τ, op, ℓ (not generated) -latency, budgets ──► every dimension, through w_v -``` +$$ +p_v^{\mathrm{log}}=(\boldsymbol\theta_v,\omega_v),\qquad +p_v^{\mathrm{phys}}=(\tau_v,\ell_v,\sigma_v). +$$ -### Properties and requirements at a glance (decided 2026-11-05) +### Logical requirements -A field is kept only when it is a real choice: anything the expression, $G_{c,i}$'s identity or the consumer's own properties already fix is not repeated. +The proposed `LogicalRequirement` stores a **parameter floor**: the weakest configuration in a family's ordered list that satisfies the applicable input error cap. Section 4 defines the cap and distinguishes validity from pruning. It does not allocate a fixed share of the query's error budget to each input. -| Struct | Field | Values | Symbol | `new` rejects | -|---|---|---|---|---| -| `LogicalProperties` ($p^{\log}_v$) | `parameters` | `Option`: θ from the family's predefined list; None for an exact operator | $\boldsymbol\theta_v$ | parameters of a family other than the operator's; parameters on an exact operator | -| | `window_layout` | `None`, or `Tumbling { width_ms }` (later: sliding by merge or subtraction) | $\omega_v$ | a window on an operator that cannot be windowed; width 0 | -| `LogicalRequirement` ($r^{\log}_{c,i}$) | `parameter_floor` | `Option`: the smallest θ in the list whose $\varepsilon$ ≤ $\text{cap}_{c,i}$ | $\boldsymbol\theta^{\min}_{c,i}$ | a floor not in the family's predefined list | -| `PhysicalProperties` ($p^{\mathrm{phys}}_v$) | `timing` | IngestionTime, QueryTime | $\tau_v$ | (IngestionTime, OneEvaluation) | -| | `lifetime` | None, OneEvaluation(Medium), AcrossEvaluations(Medium); Medium ∈ {Memory, Disk} | $\ell_v$ | | -| | `schedule` | None; Periodic { interval, phase }; TumblingWindowGrid { phase } | $\sigma_v$ | the schedule combinations listed above | -| ($r^{\mathrm{phys}}_{c,i}$) | none | a function of the consumer's $p^{\mathrm{phys}}_c$: its $\tau_c$, and its $\sigma_c$ or each context's $T_\kappa$ | | | +For KLL, a larger `k` satisfies a smaller minimum `k`. For DDSketch, a smaller `alpha` can be stronger. Multi-parameter families require the family's comparison rule, including parameters needed for failure guarantees. A single floor is valid only if the accepted configurations form an upward-closed set with a representable least element. Otherwise, compatibility must use an explicit admissible set or predicate; it must not invent a unique minimum. -Not fields, because something else already fixes them: the summary family ($\operatorname{op}_v$'s argument, part of the group identity); the window count (query range ÷ width; retention is $h_v$); a family or window requirement on an input ($G_{c,i}$'s identity, and the schedule check); the statistic a consumer reads (its own operator, used once to compute $\boldsymbol\theta^{\min}$); which evaluation produced the output (reuse across evaluations is the producer's $\ell$, a cost choice). +The required input group's identity already constrains the state family and data/window semantics. An exact producer satisfies an approximation floor only if it is also a valid producer of that group and input type. -**α^log** = `satisfies_logical(p^log_u, r^log_{c,i})`: $\boldsymbol\theta_u \succeq \boldsymbol\theta^{\min}_{c,i}$, compared per family on the parameters that set $\varepsilon$ (KLL `k` ≥; CMS / CountSketch `width` ≥; DDSketch `alpha` ≤; HLL `precision` ≥; KMV `k` ≥; UnivMon and the heap variants on each sizing parameter). No floor, or an exact producer, always satisfies. +### Physical requirements -**α^phys** = `satisfies_physical(p^phys_u, p^phys_c)`, timing and lifetime: +Physical requirements are derived from the consumer's timing and schedule, rather than stored redundantly. For an unmaterialized consumer, the schedule comes from its execution context during selection. Stage 2 checks the parts known from candidate properties; Stage 3 checks any remaining context-dependent delivery condition. -| Consumer $\tau_c$ ↓ / producer $(\tau_u, \ell_u)$ → | (QT, None) | (QT, OneEval) | (QT, AcrossEval) | (IT, None) | (IT, AcrossEval) | +| Consumer timing | QT, None | QT, OneEvaluation | QT, AcrossEvaluations | IT, None | IT, AcrossEvaluations | |---|---|---|---|---|---| -| QueryTime | ✅ | ✅ | ✅ | ❌ an unmaterialized ingestion-time output is gone by query time | ✅ | -| IngestionTime | ❌ | ❌ | ❌ | ✅ | ✅ | +| QueryTime | Allowed | Allowed | Allowed | Rejected | Allowed | +| IngestionTime | Rejected | Rejected | Rejected | Allowed | Allowed | -and schedule, for a producer with $\ell_u \ne$ None, against the consumer's times (its own $\sigma_c$ in Stage 2 when $\ell_c \ne$ None; each context's $T_\kappa$ in Stage 3 when $\ell_c$ = None): +QT means QueryTime; IT means IngestionTime. Schedule, data coverage, and retention checks still apply to allowed combinations. -| Producer $\sigma_u$ | Delivers on consumer times $(I, \varphi)$ when | -|---|---| -| none ($\ell_u$ = None) | always: it runs inside the consumer's context, chosen by Stage 3 ($x_{\kappa,v}$) | -| Periodic $(I_\sigma, \varphi_\sigma)$ | $I_\sigma \mid I$ and $\varphi \equiv \varphi_\sigma \pmod{I_\sigma}$; a non-periodic consumer is never served | -| TumblingWindowGrid $\varphi_g$, width $w$ | $w \mid I$ and $\varphi \equiv \varphi_g \pmod w$; a non-periodic consumer is never served | - -| Struct | Set by | When | -|---|---|---| -| `LogicalProperties` | Pass 1 (one candidate per θ in the list); Pass 2 rules (window layouts) | when the candidate is created | -| `LogicalRequirement` | the consumer: floor from the validity cap of `LogicalRequirement::error_propagation(consumer)` | when the consumer is created; Pass 2 tightens it with the pruning cap when it builds the logical edges | -| `PhysicalProperties` | Stage 2 physical expansion (every valid $(\tau, \ell, \sigma)$) | Stage 2 | -| $r^{\mathrm{phys}}_{c,i}$ | derived from $p^{\mathrm{phys}}_c$ | checked in Stage 2 (timing, lifetime; schedule of a materialized consumer) and Stage 3 (schedule per context) | - -## 2. Decision tree of properties - -Each memo group's candidates are the leaves of a tree. The first level chooses the expression $e_v$; every later level decides one property dimension. A path from the root to a leaf is one physical candidate in $\mathcal C$. - -```mermaid -flowchart TD - G["Memo group G
(logical result)"] --> R1["e_v: exact operator"] - G --> R2["e_v: SummaryAgg[KLL]"] - G --> R3["e_v: SummaryAgg[DDSketch]"] - G --> R4["e_v: …other operators
(prune: executor support)"] - - R2 --> T1["θ: k = 200
ε = 0.008"] - R2 --> T2["θ: k = 345
ε = 0.005"] - R2 --> T3["θ: k = …
(prune: ε above the ε_q of
every query reaching G)"] - - T2 --> W1["ω: no windows"] - T2 --> W2["ω: tumbling, width w
(prune: w must divide the range and
step of some query reaching G)"] - - W2 --> P1["τ = QueryTime"] - W2 --> P2["τ = IngestionTime
(prune: no ingestion pipeline)"] - - P1 --> L1["ℓ = None"] - P1 --> L2["ℓ = OneEvaluation"] - P1 --> L3["ℓ = AcrossEvaluations
(prune: deployment keeps
no query-time results)"] - P2 --> L4["ℓ = None"] - P2 --> L5["ℓ = AcrossEvaluations"] - P2 -. invalid .-> L6["ℓ = OneEvaluation"] - - L2 --> S1["σ: Periodic (I, φ)
from the gcd closure
of the queries' 𝒯_q"] - L3 --> S2["σ: grid phase φ mod w"] - L5 --> S3["σ: grid phase 0"] -``` +**Periodic versus grid.** A periodic schedule with interval ten seconds and phase two produces at 2, 12, 22, … seconds. A five-second tumbling grid with phase zero partitions the input into windows [0, 5), [5, 10), … . The first describes evaluation times; the second describes window boundaries. Grid alignment alone does not ensure sufficient retained history. -The levels, top to bottom: +For the proposed periodic consumer path, let the consumer interval and phase be $I$ and $\varphi$. Let the producer interval and phase be $I_s$ and $\varphi_s$, and its tumbling width and grid phase be $w$ and $\varphi_g$. -| Level | Decides | Stage | Values offered | Pruned when | Code | -|---|---|---|---|---|---| -| 1 | expression $e_v$ ($\operatorname{op}_v$ with its family) | Stage 1, Pass 1 and Pass 2 rules | each way to compute the group | the executor cannot build the family | Pass 1 ([`per_computation_candidates.rs`](../../../crates/logical-asap-optimization/src/pass1/per_computation_candidates.rs)); Pass 2 rules ([`rules/`](../../../crates/logical-asap-optimization/src/pass2/memo/rules)) | -| 2 | $\boldsymbol\theta_v$ | Stage 1, Pass 1 | the family's predefined list | $\varepsilon_v$ alone exceeds the $\varepsilon_q$ of every query that reaches the group | Pass 1 sizing per `accuracy_budget_parts` ([`memo_population.rs`](../../../crates/logical-asap-optimization/src/pass1/memo_population.rs)); a predefined list in #1203 PR 2 | -| 3 | $\omega_v$ **(#1207)** | Stage 1, Pass 2 | tumbling widths from the gcd closure of the ranges and steps of the queries reaching the group | no such query's range and step is a multiple of the width | Pass 2 ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs), [`state_groups.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/state_groups.rs)) | -| 4 | $\tau_v$ | Stage 2 | QueryTime; IngestionTime | the source has no ingestion pipeline | `enumerate_candidate_configurations` ([`configurations.rs`](../../../crates/physical-asap-optimization/src/catalog/configurations.rs)) | -| 5 | $\ell_v$ | Stage 2 | None, OneEvaluation, AcrossEvaluations | invalid with $\tau_v$; the deployment keeps no query-time results across evaluations | same function, with `DeploymentCapabilities::query_time_results_retained` | -| 6 | $\sigma_v$ **(#1207)** | Stage 2 | the gcd closure $\Sigma$ of the schedules $\mathcal T_q$ of the queries reaching the group | only offered when $\ell_v \ne$ None | Stage 3 MILP today; Stage 2 in #1200 PR 2 | +| Producer schedule | Required alignment | +|---|---| +| Periodic | $I_s\mid I$ and $\varphi\equiv\varphi_s\pmod{I_s}$ | +| Tumbling grid | $w\mid I$ and $\varphi\equiv\varphi_g\pmod w$ | +| None | Runs in the consuming context; no independent delivery schedule | -What the tree does not decide, because it is a choice of the MILP over $\mathcal C$ and $\mathcal E$: -- **Which producer each consumer input reads:** $y_{u,c,i}$. Two consumers of one group may read different leaves, e.g. one reads `k = 200` and another `k = 345`. -- **Where an unmaterialized candidate runs (#1207):** a candidate with $\ell_v$ = None runs inside each reading context $\kappa$. -- **How much a materialized candidate retains (#1207):** $h_v$, from the longest range among the queries its selected consumers serve, constraint (R). +The supplied formulation also permits a single, non-periodic evaluation. The interval/phase interface below covers only periodic consumers. Rejecting non-periodic consumers is a current scope restriction of that proposed interface, not a mathematical impossibility; supporting them needs explicit evaluation-time membership checks. -### Order of decisions +### Invalid combinations -**The order does not affect correctness.** The leaves are the product of the dimensions' value sets, minus the invalid combinations and the pruned values. Every prune rule depends only on the value it removes and on values decided above it, so any order that decides a pruned dimension after the dimensions its prune rule reads yields the same $\mathcal C$. +Checked constructors reject these combinations under the proposed schedule model: -**The order affects the efficiency of enumeration.** The number of nodes visited is $\sum_{d} \prod_{d' \le d} |\text{values kept at } d'|$, so a dimension that prunes much should come early, and a dimension whose values multiply without pruning should come late. Four considerations set the order above: +| Combination | Reason | +|---|---| +| IngestionTime with OneEvaluation | No query evaluation lifetime at ingestion time | +| None with an independent schedule | Execution follows the consuming context | +| Materialized lifetime without its required schedule | Delivery times or window grid would be unspecified | +| OneEvaluation with a grid, or windowed AcrossEvaluations with a periodic schedule | Schedule kind does not match this model's lifetime representation | +| Zero interval/width, or phase outside the canonical interval/width | Invalid schedule representation | +| Ingestion-time grid with nonzero phase | Unsupported by the draft's epoch-anchored ingestion representation | -1. **Logical before physical.** $e_v$, $\boldsymbol\theta_v$ and $\omega_v$ change the result's values or error; $\tau_v$, $\ell_v$ and $\sigma_v$ change only when and how long it is computed and kept. Deciding $p^{\log}$ first lets Stage 1 hash-cons groups and compute $\alpha^{\log}$ once, before Stage 2 multiplies each logical candidate by its $p^{\mathrm{phys}}$ alternatives. -2. **Strong prunes early.** The operator is pruned by the executor, and $\boldsymbol\theta_v$ by the accuracy targets, before the window and physical levels multiply them. -3. **Dependent dimensions adjacent.** $\ell_v$ is constrained by $\tau_v$, and $\sigma_v$ exists only when $\ell_v \ne$ None and its kind follows $\ell_v$. Deciding $\tau \to \ell \to \sigma$ prunes each invalid combination at the first level where it can be detected. -4. **Deployment-wide prunes as early as they can go.** Without an ingestion pipeline, the IngestionTime subtree is pruned once per candidate. Moving $\tau$ above $\omega$ would prune it once per group instead; this is an alternative order with the same leaves. +A tumbling grid also needs a known positive width. The `PhysicalProperties` constructor alone cannot validate a width held in logical properties; physical candidate construction must validate the pair. Non-windowed AcrossEvaluations schedules need an explicit representation before that alternative can be offered. -Alternative orders worth comparing: -- **$\tau$ first.** Where most sources have no ingestion pipeline, deciding timing at the group level prunes half the tree before $\boldsymbol\theta$ and $\omega$ multiply it. -- **$\boldsymbol\theta$ last among logical dimensions.** $\boldsymbol\theta$ has the most values. Deciding $\omega$ before $\boldsymbol\theta$ visits fewer nodes when many widths are pruned by the queries' ranges. -- **Requirement-driven (top-down) vs producer-driven (bottom-up).** Top-down enumeration starts from each consumer's $r_{c,i}$ and offers only the values it accepts; bottom-up offers every value and lets $\alpha$ filter $\mathcal E$. Top-down visits fewer nodes but must union the requirements of all consumers of a group, so that sharing between consumers is not lost. +## 3. Pipeline: interfaces and rules -## 3. Data structures and interfaces per stage +| Stage | Input | Output | Responsibility | +|---|---|---|---| +| Stage 0 | Query IR | `DagOfLogicalMemoGroups` | Hash-cons logical expressions | +| Stage 1, Pass 1 | Group DAG | `Memo` | Add initial logical candidates and intermediate groups | +| Stage 1, Pass 2 | Memo | `LogicalASAPMemoCandidates` | Add reuse alternatives and compute logical edges | +| Stage 2 | Logical memo candidates | `PhysicalASAPMemoCandidates` | Expand physical alternatives and compute compatibility | +| Stage 3 | Physical memo candidates | `DagOfSelectedCandidates` | Select implementations, dependencies, and executions | +| Realizer/export | Selected DAG | `PhysicalASAPDAG` | Build and export the executable graph | -This section gives the target interface of each stage after the formulation refactor (#1207). Each stage's boundary type carries the notation that stage defines, and no stage reads a notation a later stage defines: logical stages carry no physical properties. +Enumeration proceeds through expression, parameters, window layout, timing, lifetime, and schedule. Executor and deployment restrictions prune unsupported alternatives as soon as their prerequisites are known. Accuracy pruning is allowed only under the conditions in Section 4. Reordering dimensions preserves the candidate set only when the same alternatives and sound pruning predicates are retained. -| Stage | Input | Output (boundary type) | Notation it defines | Sub-issue | -|---|---|---|---|---| -| Stage 0: frontends, hash-consing | queries | `DagOfLogicalMemoGroups` | $G$, query roots | — | -| Stage 1, Pass 1 | `DagOfLogicalMemoGroups` | `Memo` | $v = (e_v, p^{\log}_v)$, $r^{\log}_{c,i}$, $\theta$ list | #1203 | -| Stage 1, Pass 2 | `Memo` | `LogicalASAPMemoCandidates` | rule-added candidates; logical edges with $\alpha^{\log}$ | #1203, #1201 | -| Stage 2 | `LogicalASAPMemoCandidates` | `PhysicalASAPMemoCandidates` | $p^{\mathrm{phys}}_v = (\tau, \ell, \sigma)$, $r^{\mathrm{phys}}_{c,i}$, physical edges $\mathcal E$ with $\alpha^{\mathrm{phys}}$ | #1200, #1201 | -| Stage 3 | `PhysicalASAPMemoCandidates` | `DagOfSelectedCandidates` | contexts $\kappa$; cost $w_v$ (`asap-cost`); $x$, $y$, $z$, $t$, $h$; constraints (A)–(K), (R), (L), budgets | #1204 | -| Realizer, export | `DagOfSelectedCandidates` | `PhysicalASAPDAG` | — | #1200 (export fields) | +Rules add groups through `find_or_add_group` and candidates through `add_candidate`. They match group expressions, avoid duplicating an existing expression/property pair, and do not copy consumers to make a new producer visible. After logical generation finishes, edge construction records the possible producer choices. -### Stage 0: frontends and hash-consing +### Stage 0: groups -No change. One group per distinct logical sub-expression; the queries' roots point at groups. +`hash_consed` in `crates/logical-asap-optimization/src/stage0.rs` identifies shared logical expressions and query roots. The following preserves the existing group representation described in the draft. Candidate ownership is exposed through the group in the proposed Stage 1 interface. ```rust -// crates/types/src/stages/memo.rs pub struct MemoGroupId(pub u32); -/// G: one logical result or one state family over specified input data. pub struct MemoGroup { - pub logical_node: Rc, // expression; no physical fields (#1200) - pub child_groups: Vec, // producer groups G_i - pub parents: Vec, // derived: consumers' groups - pub queries: Vec, // derived: queries that reach it + pub logical_node: Rc, + pub child_groups: Vec, + pub parents: Vec, + pub queries: Vec, + // Empty after Stage 0; populated by Stage 1. + pub candidates: Vec, } pub struct DagOfLogicalMemoGroups { @@ -293,317 +182,492 @@ pub struct DagOfLogicalMemoGroups { } ``` -### Stage 1, Pass 1: candidates of each group +`OperatorNode` recursively contains chosen child nodes and therefore describes a fixed plan fragment. `Operator` contains input group IDs and leaves their implementations open. A candidate needs its own expression because a group may offer several ways to compute the same result; it cannot reuse the group's representative `logical_node` as every candidate's expression. -Pass 1 adds every way to compute each group as candidates. Every intermediate result is its own group (no private nodes), and a summary family's sizes are candidates of one state group. +Replacing the group's representative node and child list with a single group-based expression, or removing the one-field `CandidateExpression` wrapper, remains an optional simplification from the review. Neither is required here. + +### Stage 1, Pass 1: initial candidates + +Implementation locations: `crates/logical-asap-optimization/src/pass1/per_computation_candidates.rs` and `memo_population.rs`. The #1203 target gives every intermediate result its own group; candidates contain one operator rather than private operator chains. + +| Alternative | Added representation | +|---|---| +| Exact | Exact candidate in the result group | +| Exact accumulator | Accumulator-state group plus a finalize candidate in the result group | +| Summary | One state group per supported family; one build candidate per listed parameter configuration; separate readout candidates | +| Fused build (#1203 2c) | Candidate with `fused_groups` identifying additional results; selection must enforce their availability consistently | ```rust -// crates/types/src/stages/candidates.rs -pub struct CandidateForMemoGroupId { pub group: MemoGroupId, pub index: usize } +pub struct CandidateForMemoGroupId { + pub group: MemoGroupId, + pub index: usize, +} pub struct Memo { pub groups: Vec, pub query_roots: Vec>, - pub candidates: Vec>, // by group } + impl Memo { - /// Hash-consing: the group with this expression, added if new. - pub fn find_or_add_group(&mut self, expression: Operator) -> MemoGroupId; - pub fn add_candidate(&mut self, group: MemoGroupId, candidate: CandidateForMemoGroup) -> CandidateForMemoGroupId; + pub fn find_or_add_group( + &mut self, + expression: Operator, + ) -> MemoGroupId; + + pub fn add_candidate( + &mut self, + group: MemoGroupId, + candidate: CandidateForMemoGroup, + ) -> CandidateForMemoGroupId; + pub fn candidate(&self, id: CandidateForMemoGroupId) -> &CandidateForMemoGroup; } -/// v = (e_v, p_v^log) +pub struct CandidateExpression { + pub operator: Operator, +} + pub struct CandidateForMemoGroup { - pub expression: CandidateExpression, // e_v - pub logical_properties: LogicalProperties, // p_v^log - pub input_requirements: Vec, // r^log_{c,i}, one per input group + pub expression: CandidateExpression, + pub logical_properties: LogicalProperties, + pub input_requirements: Vec, pub fused_groups: Vec, - /// Which rule added it. Not in the formulation: kept for `stage_pipeline --text` - /// and the viewer (approved 2026-11-05). + // Diagnostic provenance; not a planning decision. pub origin: OriginOfCandidateForMemoGroup, } -/// e_v = op_v(G_{v,1}, …, G_{v,n}): one operator, its children are groups. -pub struct CandidateExpression { pub operator: Operator } -/// p_v^log = (θ_v, ω_v). The family is the operator's argument, not a property. -/// `new` rejects parameters of another family (or on an exact operator) and a -/// window on an operator that cannot be windowed. pub struct LogicalProperties { - parameters: Option, // θ_v, from the family's predefined list; None for exact - window_layout: Option, // ω_v + parameters: Option, + window_layout: Option, +} + +pub enum WindowLayout { + Tumbling { width_ms: u64 }, } + impl LogicalProperties { - pub fn new(expression: &CandidateExpression, parameters: Option, - window_layout: Option) -> Result; + pub fn new( + expression: &CandidateExpression, + parameters: Option, + window_layout: Option, + ) -> Result; + pub fn parameters(&self) -> Option<&SketchParams>; pub fn window_layout(&self) -> Option<&WindowLayout>; } -/// The window count is the group's query range ÷ width; retention is h_v (Stage 3). -pub enum WindowLayout { Tumbling { width_ms: u64 } } -/// Approved 2026-11-05. The family's predefined θ list, each with its error bound -/// derived through `estimators::local_guarantee` for a given statistic. pub fn parameter_list(family: SketchAlgorithm) -> &'static [SketchParams]; -/// r^log_{c,i}: only what G_{c,i}'s identity does not fix. `new` rejects a floor -/// that is not in the family's predefined list. pub struct LogicalRequirement { - /// θ^min_{c,i}: the smallest θ in the list whose ε, for the statistic the - /// consumer reads, is at most cap_{c,i} = min(validity cap of g_c, pruning cap). parameter_floor: Option, } + impl LogicalRequirement { - pub fn new(parameter_floor: Option) -> Result; + pub fn new( + parameter_floor: Option, + ) -> Result; + pub fn parameter_floor(&self) -> Option<&SketchParams>; - /// g_c: λ_{c,i} per input, read from the operator and its arguments, and the - /// validity cap (None when g_c is linear). Sets each input's first floor; - /// Stage 3 composes the factors into λ_{q,c,i}. - pub fn error_propagation(consumer: &Operator) -> ErrorPropagation; } -/// Approved 2026-11-05. -pub struct ErrorPropagation { pub factor_per_input: Vec, pub validity_cap: Option } -/// α^log_{u,c,i}: θ_u ⪰ θ^min_{c,i}, compared per family on the parameters that set ε. -pub fn satisfies_logical(producer: &LogicalProperties, requirement: &LogicalRequirement) -> bool; +pub fn satisfies_logical( + producer: &LogicalProperties, + requirement: &LogicalRequirement, +) -> bool; ``` -The logical and physical properties have the same shape: private fields, a checked `new`, one accessor per dimension, one error type per side (`InvalidLogicalProperties`, `InvalidPhysicalProperties`), and one `satisfies_*` function. Only the logical side has a stored requirement; the physical requirement is a function of the consumer's own physical properties. +The floor representation has the applicability condition stated in Section 2. Creating requirements also needs the input family's configuration domain; a constructor without that context can validate only the information encoded in its argument. Error-model interfaces are grouped with accuracy in Section 4. -The error bound is not stored: it depends on the statistic the consumer reads, which is the consumer's own operator argument. It is used once, when the consumer's floor is computed: the smallest θ in `parameter_list(family)` whose `local_guarantee` for that statistic has ε ≤ cap. The validity cap is known when the consumer is created (`LogicalRequirement::error_propagation`). The pruning cap needs $\lambda_{q,c,i}$ and the queries that may reach $c$, so Pass 2 builds each input's final requirement with `LogicalRequirement::new` when it builds the logical edges. +Migration targets: remove private `root_node` chains, duplicate `input_groups`, and `accuracy_budget_parts`. The proposed predefined parameter lists replace sizing one candidate per budget partition. -Deleted: `root_node` with private nodes, `input_groups` (the expression's children), `accuracy_budget_parts`, the `SharedBuild` and `SummaryResult` origins. +### Stage 1, Pass 2: reuse rules and logical edges -Concepts not in the formulation: `origin: OriginOfCandidateForMemoGroup` (Pass1, SummaryInput, TumblingWindows, Rollup, TopKReuse, WidenedScans) records which rule added a candidate; nothing in planning reads it. +The following preserves the supplied rule inventory and tracking references. Paths are relative to `crates/logical-asap-optimization/src/pass2/memo/`. -### Stage 1, Pass 2: rules and logical edges - -Rules add candidates to existing groups, or add groups. A rule never copies a consumer. After the last rule, Pass 2 builds the logical edges once. +| Rule / function | File | Result or migration target | +|---|---|---| +| Local windows: `add_local_window_composition` | `window_candidates.rs` | Add layouts to the state group; #1160 removes copied window consumers | +| Scan widening: `add_scan_label_candidates` | `rules/scan_labels.rs` | Add a widened scan group and projections in the original groups; #1222 | +| Summary inputs: `factor_summary_inputs` | `rules/summary_inputs.rs` | Expose projections/filters as input groups; #1218 moves this into initial population | +| Top-k reuse: `add_topk_candidates` | `rules/topk.rs` | Read a compatible larger top-k result with an exact-input requirement; #1220 | +| Finer aggregate: `add_finer_aggregate_candidates` | `rules/finer_aggregate.rs` | Filter and roll up finer groups, using separate intermediate groups; #1220 | +| Rollup: `add_rollup_candidates` | `rules/rollup.rs` | Add compatible finer-state merges; #1221 | +| Window rollup: `add_window_rollup_candidates` | `rules/rollup.rs` | Merge shorter windows into longer-range results; #1221 | +| Summary results: `factor_summary_results` | `rules/summary_results.rs` | Removed once Pass 1 exposes readouts; #1218 | +| Shared configurations: `add_state_configuration_candidates` | `rules/shared_configuration_readers.rs` | Remove copied consumers once producer variants are selectable; #1203 2b | +| Shared state groups: `add_shared_summary_state_groups` | `state_groups.rs` | Remove per-size groups; #1203 2b / #1160 | + +After the rules, build logical edges and apply any proven-safe accuracy pruning. `input_index_of_consumer` replaces the ambiguous `input` field name throughout this document's edge interfaces. ```rust -// crates/logical-asap-optimization/src/pass2 -/// A rule: reads the memo, adds candidates or groups. fn apply(memo: &mut Memo) -> Result<(), Pass2RuleError>; -// crates/types/src/stages/logical_memo_candidates.rs -/// (u, c, i) over logical candidates: u is a candidate of G_{c,i}. pub struct LogicalEdge { pub producer: CandidateForMemoGroupId, pub consumer: CandidateForMemoGroupId, - pub input: usize, - pub satisfies_logical_requirement: bool, // α^log + pub input_index_of_consumer: usize, + pub satisfies_logical_requirement: bool, } + pub struct LogicalASAPMemoCandidates { pub memo: Memo, - pub edges: Vec, // grouped by (consumer, input) + pub edges: Vec, } ``` -### Stage 2: physical candidates and edges +### Stage 2: physical alternatives -Stage 2 expands each logical candidate into one physical candidate per valid $(\tau, \ell, \sigma)$, gives each input its physical requirement, and turns each logical edge into the physical edges between the two candidates' physical alternatives. +`enumerate_candidate_configurations` in `crates/physical-asap-optimization/src/catalog/configurations.rs` expands each logical candidate using the dimensions and checks in Section 2. It creates no new logical candidates. Each logical edge is expanded between the relevant physical alternatives. ```rust // crates/types/src/physical/properties.rs -pub enum ExecutionTiming { IngestionTime, QueryTime } -pub enum Medium { Memory, Disk } -pub enum Lifetime { None, OneEvaluation(Medium), AcrossEvaluations(Medium) } +pub enum ExecutionTiming { + IngestionTime, + QueryTime, +} + +pub enum Medium { + Memory, + Disk, +} + +pub enum Lifetime { + None, + OneEvaluation(Medium), + AcrossEvaluations(Medium), +} pub enum Schedule { - Periodic { interval_ms: u64, phase_ms: u64 }, // OneEvaluation - TumblingWindowGrid { phase_ms: u64 }, // AcrossEvaluations + Periodic { interval_ms: u64, phase_ms: u64 }, + TumblingWindowGrid { phase_ms: u64 }, +} +pub struct PhysicalProperties { + timing: ExecutionTiming, + lifetime: Lifetime, + schedule: Option, } -/// p_v^phys = (τ, ℓ, σ); `new` rejects the invalid combinations (Section 1). -pub struct PhysicalProperties { timing: ExecutionTiming, lifetime: Lifetime, schedule: Option } impl PhysicalProperties { - pub fn new(timing: ExecutionTiming, lifetime: Lifetime, schedule: Option) -> Result; + pub fn new( + timing: ExecutionTiming, + lifetime: Lifetime, + schedule: Option, + ) -> Result; pub fn timing(&self) -> ExecutionTiming; pub fn lifetime(&self) -> Lifetime; pub fn schedule(&self) -> Option<&Schedule>; - pub fn is_materialized(&self) -> bool; // ℓ ≠ None + pub fn is_materialized(&self) -> bool; } -/// A periodic schedule φ + I·ℤ: a materialized consumer's own times, or a -/// context's times T_κ (Stage 3). -pub struct EvaluationSchedule { interval_ms: u64, phase_ms: u64 } +pub struct EvaluationSchedule { + interval_ms: u64, + phase_ms: u64, +} impl PhysicalProperties { - /// Whether this producer delivers on `times` (the schedule table in section 1); - /// always true when ℓ = None. pub fn delivers_on(&self, times: &EvaluationSchedule) -> bool; } -/// α^phys_{u,c,i} = r^phys_{c,i} as a function of the consumer's own p^phys_c: -/// timing and lifetime, and, for a materialized consumer, `delivers_on` its σ_c. -/// For a consumer with ℓ = None, Stage 3 checks `delivers_on(T_κ)` per context. -pub fn satisfies_physical(producer: &PhysicalProperties, consumer: &PhysicalProperties) -> bool; - -// crates/types/src/stages/physical +pub fn satisfies_physical( + producer: &PhysicalProperties, + consumer: &PhysicalProperties, +) -> bool; pub struct PhysicalCandidateId(pub usize); pub struct PhysicalEdgeId(pub usize); -/// One physical alternative of a logical candidate. pub struct PhysicalCandidate { - pub logical: CandidateForMemoGroupId, // e_v and p_v^log - pub properties: PhysicalProperties, // p_v^phys + pub logical: CandidateForMemoGroupId, + pub properties: PhysicalProperties, } -/// (u, c, i) over physical candidates. pub struct PhysicalEdge { pub producer: PhysicalCandidateId, pub consumer: PhysicalCandidateId, - pub input: usize, - pub satisfies_logical_requirement: bool, // α^log - pub satisfies_physical_requirement: bool, // α^phys + pub input_index_of_consumer: usize, + pub satisfies_logical_requirement: bool, + pub satisfies_physical_requirement: bool, } pub struct PhysicalASAPMemoCandidates { pub logical: LogicalASAPMemoCandidates, - pub candidates: Vec, // C + pub candidates: Vec, pub query_roots: Vec>, pub pruned_candidates: Vec>, - // edges stored per (consumer, input); see `edges()` } impl PhysicalASAPMemoCandidates { - pub fn edges(&self) -> impl Iterator; // E - pub fn admissible_edges_into(&self, consumer: PhysicalCandidateId, input: usize) -> &[PhysicalEdgeId]; // P_{c,i} - pub fn admissible_edges_out_of(&self, producer: PhysicalCandidateId) -> &[PhysicalEdgeId]; + pub fn edges(&self) -> impl Iterator; + pub fn admissible_edges_into( + &self, + consumer: PhysicalCandidateId, + input_index_of_consumer: usize, + ) -> &[PhysicalEdgeId]; + pub fn admissible_edges_out_of( + &self, + producer: PhysicalCandidateId, + ) -> &[PhysicalEdgeId]; +} + +// Identifies a rejected alternative by content: it has no allocated candidate ID. +pub struct PhysicalCandidateKey { + pub logical: CandidateForMemoGroupId, + pub properties: PhysicalProperties, } ``` -Deleted: `Materialization`, `PhysicalCandidate::{timing, materialization, operator_node}`, `producer_candidates_per_input` as the interface, `compatible_timing`, the reader constraints `ReaderConstraint::{IncompatibleOutputs, AllowedOutputs}` (now $\alpha^{\mathrm{phys}}$ through $\sigma$), `may_keep_across_evaluations_per_sole_reader_query`. +`PhysicalCandidateId` indexes an emitted candidate. `PhysicalCandidateKey` describes an alternative rejected before it received an ID, allowing diagnostics to explain pruning. The admissible-edge accessor returns edge IDs, not the unfiltered producer set from Section 1. + +The `delivers_on` sketch needs access to a tumbling producer's logical window width. Supply that context in its implementation or move the check to the full physical-candidate interface; the phase-only schedule is insufficient by itself. + +Migration targets include replacing `Materialization` and duplicated physical fields with checked properties, and replacing `ReaderConstraint::{IncompatibleOutputs, AllowedOutputs}` with compatibility and schedule checks. Policy constraints involving the number of selected consuming queries are a separate concern; see Section 5. + +### Stage 3: baseline selection and realization -### Stage 3: the MILP +The baseline uses the variables and constraints in the supplied LaTeX. Stage 3 adds no candidates or potential dependencies. -Stage 3 reads the physical candidates and edges and selects the DAG. Variables and constraints are named after the formulation. +| Variable | Meaning | +|---|---| +| $x_v$ | Candidate selected | +| $y_{u,c,i}$ | Producer supplies the specified consumer input | +| $z_{q,v}$ | Candidate supplies the query output | +| $t_v$ | Topological rank | + +| Constraint | Meaning | Encoding | +|---|---|---| +| (A) | One producer per selected consumer input | $\sum_{u\in P_{c,i}}y_{u,c,i}=x_c$ | +| (B) | Chosen producers are selected | $y_{u,c,i}\le x_u$ | +| (C) | One output per query | $\sum_{v\in G_q}z_{q,v}=1$ | +| (D) | Output candidates are selected | $z_{q,v}\le x_v$ | +| (E) | Acyclic dependencies | $t_u+1\le t_c+\lvert\mathcal C\rvert(1-y_{u,c,i})$ | +| (F) | Selected candidates are used | $x_v\le\sum_{(v,c,i)\in\mathcal E}y_{v,c,i}+\sum_{q:v\in G_q}z_{q,v}$ | +| (G) | Logical compatibility | $y_{u,c,i}\le\alpha^{\mathrm{log}}_{u,c,i}$ | +| (H) | Physical compatibility | $y_{u,c,i}\le\alpha^{\mathrm{phys}}_{u,c,i}$ | +| (I) | Domains | $x_v,y_{u,c,i},z_{q,v}\in\{0,1\}$; $0\le t_v\le\lvert\mathcal C\rvert-1$ | + +Constraints are applied for every relevant candidate, input, query, or potential edge. An implementation may allocate edge variables only on admissible edges, making (G) and (H) implicit without changing the baseline feasible set. + +The baseline objective charges each selected candidate a fixed cost over the common horizon: + +$$ +\min\sum_{v\in\mathcal C}w_vx_v. +$$ + +The cost includes input reading, the candidate's own computation, and lifetime-dependent retention. It excludes producers' computation, which is charged at those producers. This objective assumes execution frequencies and retention costs are fixed before optimization and independent of the selected consumers. When that assumption does not hold, use the execution extension in Section 5; do not silently reuse the same cost coefficient. ```rust -// crates/plan-selection/src/milp/model.rs -/// κ: where an ℓ = None candidate runs. Only those candidates are indexed by it. -/// `ContextId` indexes the workload's contexts; the id types inside are illustrative. -pub struct ContextId(pub usize); -pub enum Context { - QueryEvaluation(QueryId), // κ_q - IngestionPipeline(RefreshWindowId), // I_r - Materialized(PhysicalCandidateId), // κ_m: inside a materialized m's computation +struct SelectionVariables { + x: Vec, + y: HashMap, + z: HashMap<(QueryId, PhysicalCandidateId), Variable>, + t: Vec, } -/// T_κ: when the context runs; y_{κ,u,c,i} exists only if `p^phys_u.delivers_on(κ.evaluation_schedule())`. -impl Context { - /// The context's evaluation schedule T_κ (None for a non-periodic query). - pub fn evaluation_schedule(&self) -> Option; + +// Schematic constraint-building entry points; signatures omitted. +// input_satisfaction_constraints: (A) +// producer_selection_constraints: (B) +// query_coverage_constraints: (C), (D) +// acyclicity_constraints: (E) +// utilization_constraints: (F) +// Compatibility is enforced when allocating y on admissible edges. + +pub struct SelectedEdge { + pub producer: SelectedNodeId, + pub consumer: SelectedNodeId, + pub input_index_of_consumer: usize, } -struct SelectionVariables { - x_materialized: Vec, // x_v, ℓ ≠ None - x_in_context: HashMap<(ContextId, PhysicalCandidateId), Variable>, // x_{κ,v}, ℓ = None - y: HashMap<(Option, PhysicalEdgeId), Variable>, // y_{u,c,i} on admissible edges - z: Vec, // z_{q,v} - t: Vec, // t_v, cyclic components only - h: Vec, // h_v, ℓ ≠ None - latency: LatencyVariables, // (L) -} -fn input_satisfaction_constraints(..); // (A) Σ_u y_{u,c,i} = x_c -fn producer_selection_constraints(..); // (B) y_{u,c,i} ≤ x_u -fn query_coverage_constraints(..); // (C), (D) -fn acyclicity_constraints(..); // (E) -fn utilization_constraints(..); // (F) -fn consumer_constraint_constraints(..); // (J): MinimumQueryReaders, HighestSharedQueryMaterialization over reach -fn accuracy_constraints(..); // (K) Σ λ_{q,u} ε_u a_{q,u} ≤ ε_q, Σ δ_u a_{q,u} ≤ δ_q -fn retention_constraints(..); // (R) h_v ≥ need_{c,i} · y_{v,c,i} -fn latency_constraints(..); // (L) -fn resource_budget_constraints(..); // budgets (#1207) -// Objective: min Σ w_v x_v + Σ ρ_κ w_v x_{κ,v} + Σ π_v h_v - -// crates/types/src/stages/physical/graph.rs (unchanged) -pub struct SelectedEdge { pub producer: SelectedNodeId, pub consumer: SelectedNodeId, pub input: usize } pub struct DagOfSelectedCandidates { pub candidate_of_node: Vec, - pub edges: Vec, // decoded from y = 1 + pub edges: Vec, pub node_of_output_reference: Vec, } ``` -(G) $y \le \alpha^{\log}$ and (H) $y \le \alpha^{\mathrm{phys}}$ are not constraints in the model: $y$ exists only on admissible edges. Deleted: `ResultSlot`, `slot_of`, the `e` binaries, schedule groups `z[m, G]`, `N[q]`, `A[q, parts]`, and `p[ctx, c]` for materialized candidates. +The realizer uses the selected dependencies to build the executable graph. Export carries each node's timing, lifetime, and supported schedule fields. Under the context extension, multiple selected execution nodes may refer to the same physical candidate; realization must preserve their contexts rather than merge them solely by candidate ID. -### Realizer and export +## 4. Accuracy and safe pruning — Extension (#1207) -The realizer builds the plan from `DagOfSelectedCandidates`; each node's physical properties come from its `PhysicalCandidate`. The export writes timing and lifetime (and later the schedule) per node of `PhysicalASAPDAG`, changed together with the backend. +The supplied LaTeX checks logical requirements per edge. The proposed extension additionally checks composed accuracy at each query root. This section describes the error model needed for that check; it is not an additional property dimension. -## 4. Rules: how each pass and stage adds candidates +### Local propagation -A rule reads the memo and adds to it only through the interfaces of section 3: -- `Memo::find_or_add_group(expression)` for a group, so an expression that already exists returns its group (hash-consing); -- `Memo::add_candidate(group, CandidateForMemoGroup { expression, logical_properties, input_requirements, fused_groups })` for a candidate $v = (e_v, p^{\log}_v)$ with its $r^{\log}_{v,i}$. +For a consumer, an error function bounds output error from input errors: -A rule never copies a consumer and never edits an existing candidate: every consumer of a group sees a new candidate of that group through $\mathcal E$. Rules match over groups and their candidates' expressions, never over a consumer's private operators. The order of the rules below is the order the code runs them; it affects which groups exist when a later rule matches, not which plans are valid. +$$ +\varepsilon_c\le g_c(\varepsilon_{c,1},\ldots,\varepsilon_{c,n}). +$$ -### Stage 0: hash-consing +Use **beta** for local input factors and **lambda** only for factors composed to a query root: -| Rule | Code | Matches | Adds | -|---|---|---|---| -| Hash-consing | `hash_consed` ([`stage0.rs`](../../../crates/logical-asap-optimization/src/stage0.rs)) | every operator node of every query root | one `MemoGroup` per distinct expression; `query_roots` point at $G_q$. No candidates. | +$$ +g_c(\varepsilon_{c,1},\ldots,\varepsilon_{c,n}) +\le\sum_i\beta_{c,i}\varepsilon_{c,i}. +$$ -### Stage 1, Pass 1: ways to compute each group +| Operation and metric | Applicable bound or condition | +|---|---| +| Exact pass-through | Preserves the input guarantee only when it preserves the metric's semantics | +| Sum/difference, absolute error | Sum of the operands' absolute error bounds | +| Sum over a known number of values | Sum of their bounds; a common per-value bound can be multiplied by that count | +| Ratio, relative error | Requires nonzero true denominator and denominator relative error below one; bound given below | -Pass 1 adds, for each group a query computes, every way to compute it. Each intermediate result of a way to compute is its own group (`find_or_add_group`), so `Project ← SummaryEstimate ← SummaryAgg` is three groups, each with a one-operator candidate. +A filter does not automatically preserve every accuracy guarantee. Group counts and metric conversions may need metadata or assumptions beyond the expression syntax. Propagation factors must be justified by the relevant operator/error contract. -| Way to compute | Code | Adds | -|---|---|---| -| Exact | [`per_computation_candidates.rs`](../../../crates/logical-asap-optimization/src/pass1/per_computation_candidates.rs), [`memo_population.rs`](../../../crates/logical-asap-optimization/src/pass1/memo_population.rs) | a candidate with the exact operator; $\boldsymbol\theta_v$ none, $(\varepsilon_v, \delta_v) = (0, 0)$ | -| Exact accumulator | same | a state group `ExactAggregate(kind)` over the input group, one candidate; a `FinalizeExactAccumulator` candidate in the result's group reading it | -| Summary, per family the executor supports | same | a state group `SummaryAgg[family]` over the input group, with **one candidate per $\boldsymbol\theta$ in the family's predefined list**, dropping a $\boldsymbol\theta$ whose $\varepsilon_v$ alone exceeds the $\varepsilon_q$ of every query that reaches the group; a `SummaryEstimate[statistic]` candidate in the result's group reading the state group. Each consumer's `parameter_floor` starts from its $g_c$'s validity cap (none for a linear $g_c$) | -| Fused build (#1203 2c) | same | a build that computes several targets at once: a candidate of the outer target's state group, with `fused_groups` naming the others | +For a ratio, let the operand relative-error bounds be $\varepsilon_X$ and $\varepsilon_Y$. If the numerator's relative error is defined and the denominator is nonzero, then: -Example: for A and B, Pass 1 adds G2 = `SummaryAgg[KLL](G1)` with candidates `k = 200` and `k = 345`, and G3, G4, G5 = `SummaryEstimate[q](G2)`. A and B's three estimates read one state group because their expressions over G1 hash-cons to one group. +$$ +\varepsilon_{X/Y}\le +\frac{\varepsilon_X+\varepsilon_Y}{1-\varepsilon_Y}. +$$ -Today, Pass 1 instead sizes one $\boldsymbol\theta$ per `accuracy_budget_parts` and keeps `SummaryEstimate ← SummaryAgg` as private nodes inside one candidate. #1203 2a makes the intermediate results groups; #1203 PR 2 replaces sizing by the predefined list. +Choosing a fixed validity cap $b<1$ for the denominator gives a linear upper bound within that region: -### Stage 1, Pass 2: rules over groups +$$ +\varepsilon_Y\le b +\quad\Longrightarrow\quad +\varepsilon_{X/Y}\le\frac{1}{1-b}(\varepsilon_X+\varepsilon_Y). +$$ -| # | Rule | Code | Matches | Adds | Status | -|---|---|---|---|---|---| -| 1 | Local window composition (tumbling windows) | `add_local_window_composition` ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs)) | a summary state group whose window consumers serve queries whose ranges and steps share a gcd width $w$ | a candidate of the **same state group** with $\omega_v$ = tumbling width $w$; only widths dividing the group's query range are offered, and the step and phase are checked by the schedule rule ($\alpha^{\mathrm{phys}}$) | today it copies consumers per layout (`TumblingWindows` origin); #1160 makes it a candidate of the state group | -| 2 | Scan widening | `add_scan_label_candidates` ([`scan_labels.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/scan_labels.rs)) | several scan groups of one metric with different label sets | a group for the widened scan (`find_or_add_group`), and in each original scan group a candidate `Project(widened)` | #1203 2a-5 (#1222) | -| 3 | Summary inputs | `factor_summary_inputs` ([`summary_inputs.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/summary_inputs.rs)) | a value projection or filter below a summary build | that projection or filter as its own group; the build's candidate reads it | #1203 2a-2 (#1218); after 2a, Pass 1 already makes it a group | -| 4 | Top-k reuse | `add_topk_candidates` ([`topk.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/topk.rs)) | top-k groups over one ranking with different `k` | a candidate of each top-k group reading the largest-`k` group, with an exact-input requirement | #1203 2a-3 (#1220) | -| 5 | Finer aggregate | `add_finer_aggregate_candidates` ([`finer_aggregate.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/finer_aggregate.rs)) | a SQL aggregate whose grouping is coarser than another aggregate group's | a candidate of the coarse group: filter, then merge the finer group's rows with each measure's rollup | #1203 2a-3 (#1220) | -| 6 | Rollup | `add_rollup_candidates` ([`rollup.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/rollup.rs)) | a coarse aggregate or summary group and a finer group whose state rolls up | a candidate of the coarse group reading the finer group (exact or sketch merge) | #1203 2a-4 (#1221) | -| 7 | Window rollup | `add_window_rollup_candidates` (same) | a longer-range group whose windows a shorter-range group's state covers | a candidate of the longer-range group merging the shorter one's windows | #1203 2a-4 (#1221) | -| 8 | Summary results | `factor_summary_results` ([`summary_results.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/summary_results.rs)) | a summary readout inside a composite candidate | the readout as its own group | #1203 2a-2 (#1218) makes the readout a group in Pass 1; the rule is then deleted | -| 9 | Shared configurations | `add_state_configuration_candidates` ([`shared_configuration_readers.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/shared_configuration_readers.rs)) | consumers that could share a summary at a size or window horizon that covers them | consumer copies at the shared size (`SharedBuild` origin) | deleted in #1203 2b: the state group already holds every $\boldsymbol\theta$, and $y$ chooses per consumer input | -| 10 | Shared summary state groups | `add_shared_summary_state_groups` ([`state_groups.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/state_groups.rs)) | summary builds directly over input groups | per-size concrete state groups with consumer copies | deleted in #1203 2b and #1160: one state group per state family | -| — | Logical edges (#1201) | after the last rule | every candidate $c$ and input $i$ | one `LogicalEdge { producer, consumer, input, satisfies_logical_requirement }` per $u \in G_{c,i}$, after tightening each `parameter_floor` with the pruning cap $\max_q \varepsilon_q / \lambda_{q,c,i}$; $\alpha^{\log}_{u,c,i}$ = `satisfies_logical(p^log_u, r^log_{c,i})`; output `LogicalASAPMemoCandidates { memo, edges }` | #1201 | +Both local factors are then `1 / (1 - b)`. This is a sufficient bound, not an exact equality or a claim that all feasible nonlinear plans remain represented. -After #1203 2b, every Pass 2 rule has the same shape: match groups, then `find_or_add_group` for any new intermediate result and `add_candidate` for the new way to compute an existing group. A rule adds no candidate a group already has (same expression and $p^{\log}$). +The draft's approved interface is retained here as an interface proposal: -### Stage 2: physical expansion and edges +```rust +pub struct ErrorPropagation { + pub factor_per_input: Vec, + pub validity_cap: Option, +} -Stage 2 adds no logical candidates. It expands each logical candidate into physical candidates, then builds $\mathcal E$. +impl LogicalRequirement { + pub fn error_propagation( + consumer: &Operator, + ) -> ErrorPropagation; +} +``` -| Step | Code | Reads | Adds | -|---|---|---|---| -| Physical expansion | `enumerate_candidate_configurations` ([`configurations.rs`](../../../crates/physical-asap-optimization/src/catalog/configurations.rs)) | each `CandidateForMemoGroup`, deployment capabilities, the queries' $\mathcal T_q$ | one `PhysicalCandidate { logical, properties, input_requirements }` per valid $p^{\mathrm{phys}} = (\tau, \ell, \sigma)$ that the deployment allows (section 2, levels 4–6), | -| Physical edges (#1201) | after expansion | each `LogicalEdge` | one `PhysicalEdge { producer, consumer, input, satisfies_logical_requirement, satisfies_physical_requirement }` per pair of the two logical candidates' physical candidates, with $\alpha^{\mathrm{phys}}$ = `satisfies_physical(p^phys_u, p^phys_c)`; the admissible ones ($\alpha^{\log} = \alpha^{\mathrm{phys}} = 1$) through `admissible_edges_into` / `admissible_edges_out_of` | +`factor_per_input` contains the local beta factors. A scalar `validity_cap` can represent a conservative common cap. Operator-specific caps, such as a bound only on a ratio's denominator, need either a documented input mapping or a per-input representation before implementation. Unsupported error models must be reported as unsupported rather than supplied with guessed factors. + +### Query composition and root checks + +An **error source** is one local approximation event with a stated metric and guarantee. Let $s$ index such sources in a selected, acyclic plan. Let $\varepsilon_s$ and $\delta_s$ be its bound and failure probability. Define $\lambda_{q,s}$ as the sum, over paths from that source to the query output, of products of local beta factors. Define $a_{q,s}$ as a binary indicator that query q's selected dependency graph uses that source. + +The intended root conditions are: + +$$ +\sum_s\lambda_{q,s}\varepsilon_s a_{q,s}\le\varepsilon_q, +\qquad +\sum_s\delta_s a_{q,s}\le\delta_q. +$$ + +Here, $\varepsilon_q$ and $\delta_q$ are the query target's error and failure limits. Repeated paths can amplify error and therefore contribute to lambda. A shared failure event is counted once in the union bound; distinct readout events need distinct source identities unless a joint guarantee is available. + +These equations describe the required bound. They are a linear MILP encoding only when the factors and source identities are fixed for the indexed alternatives and the reachability indicators are correctly linked to selected edges and outputs. Candidate-dependent paths, metric changes, or execution-dependent random events require a more detailed encoding. The draft does not supply that encoding, so the equations must not be described as a completed constraint implementation. + +This replaces the old unexplained `a_{q,u}` and the mixture of candidate-, edge-, and query-indexed lambda symbols. Local propagation uses beta; query propagation uses lambda; neither should be inferred from bare edge selection alone. + +### Validity caps and pruning caps + +A **validity cap** ensures that a chosen local bound applies. A **pruning cap** removes an alternative only when its nonnegative contribution alone exceeds every query target it could serve. Neither reserves an error-budget share for one input. + +For an input whose error can be expressed in the same metric, let $Q_{c,i}$ be the queries that could use it. Let $\lambda_{q,c,i}$ denote that input's query-level factor, and let $b_{c,i}$ denote its validity cap, or infinity when no cap is required. With fixed positive factors, a safe cap is: + +$$ +\operatorname{cap}_{c,i} +=\min\left(b_{c,i},\ \max_{q\in Q_{c,i}} +\frac{\varepsilon_q}{\lambda_{q,c,i}}\right). +$$ + +A zero factor gives no pruning restriction for that query. If a factor depends on the eventual plan, use a proven lower bound over all feasible downstream alternatives for this pruning test, or defer pruning. Using an arbitrary path's larger factor could discard a feasible plan. The maximum over queries protects alternatives that can serve at least one query. + +Compute the parameter floor from this cap only under the ordering condition in Section 2. Validity information is available when the consumer's error model is established; query-sensitive pruning follows logical generation and reachability analysis. Root checks still couple all contributing errors and determine feasibility. + +## 5. Execution contexts and other selection extensions — #1207 + +These extensions go beyond constraints (A)–(I) and the fixed-cost assumption in the supplied LaTeX. They are separated here so that the baseline remains understandable without an unfinished context MILP. + +### Execution identity -Example: v3 (`k = 345`) becomes, among others, (QueryTime, None), (QueryTime, OneEvaluation, Periodic(10 s, 0)) and (IngestionTime, AcrossEvaluations, grid phase 0). G3's estimate at query time has an admissible edge to each of them: same timing, or a query-time consumer of an ingestion-time result kept across evaluations. +A materialized candidate has its own delivery schedule. An unmaterialized candidate runs inside an execution that needs it. For example, a finalize candidate used by queries A and B may execute in both query contexts even though Stage 2 enumerates it only once. -### Stage 3: no rules +A **context** identifies a class of executions with a common owner and schedule across the accounting horizon. It is not necessarily one timestamp. -Stage 3 adds no candidates and no edges. It reads $\mathcal C$ and the admissible part of $\mathcal E$, creates the MILP variables of section 3 over them, and decodes $\{x = 1\}$ and $\{y = 1\}$ into `DagOfSelectedCandidates`. +```rust +pub struct ContextId(pub usize); + +pub enum Context { + QueryEvaluation(QueryId), + IngestionPipeline(RefreshWindowId), + Materialized(PhysicalCandidateId), +} + +impl Context { + // Periodic schedule when supported; None is not an implicit compatibility pass. + pub fn evaluation_schedule(&self) -> Option; +} +``` + +An unmaterialized chain inherits its consuming context. Encountering a materialized producer switches to that producer's own computation context when traversing its inputs. The context identifies where the operation runs; schedule compatibility decides whether a materialized input can supply it. + +| Extended variable | Meaning | +|---|---| +| $x_v^{\mathrm{mat}}$ | Materialized candidate selected | +| $x_{\kappa,v}$ | Unmaterialized candidate executes in context kappa | +| $y_{\kappa,u,c,i}$ | Input edge selected for a consumer execution in context kappa | +| $h_v$ | Selected retention amount for a materialized producer | + +The unindexed edge variable belongs to the baseline only. In the extension, every edge is indexed by its consumer's context, including the dedicated computation context of a materialized consumer. This avoids silently switching between two meanings of `y`. + +Activation, input satisfaction, utilization, output selection, and acyclicity must be defined for these execution instances. One binary per context/candidate assumes at most one reusable execution of that candidate within that context. If two uses require independent computations, the context or execution identity must distinguish them. + +### Cost and retention -## 5. Printing every logical DAG a memo group represents +To avoid confusing per-execution costs with the baseline's horizon-wide cost, use separate coefficient names: -A memo group stores its candidates and, through their expressions, the groups they read. It represents a set of logical DAGs without listing them. This section shows how to list them, using only the section 3 interfaces. Planning never does this: Stage 3's MILP chooses one DAG without listing the others. The listing is for inspection and for checking the MILP on small workloads. +| Symbol | Definition | +|---|---| +| $W_v^{\mathrm{mat}}$ | Fixed computation and access cost over the horizon for a materialized candidate's own schedule, excluding variable retention | +| $c_{\kappa,v}$ | Cost of one unmaterialized execution in the specified context | +| $\rho_\kappa$ | Fixed number or expected number of context executions over the common horizon | +| $h_v$ | Retention amount in a declared unit, such as retained windows | +| $\pi_v$ | Horizon-wide retention cost per unit of retained amount | +| $\operatorname{need}_{\kappa,v,c,i}$ | Retention amount producer v needs to serve this consumer input in this context | + +A proposed linear objective is: + +$$ +\min\left( +\sum_{v\text{ materialized}}W_v^{\mathrm{mat}}x_v^{\mathrm{mat}} ++\sum_{\kappa,v\text{ unmaterialized}}\rho_\kappa c_{\kappa,v}x_{\kappa,v} ++\sum_{v\text{ materialized}}\pi_v h_v +\right). +$$ + +A selected input edge imposes the corresponding retention lower bound: + +$$ +h_v\ge\operatorname{need}_{\kappa,v,c,i}\,y_{\kappa,v,c,i}. +$$ + +The retention domain, upper bounds, activation linking, and all coefficient units must be specified by the implementation. If required retention depends on selected downstream queries, that dependence needs explicit linking constraints; it is not a fixed coefficient merely because it has a name. + +### Extension status and legacy policy names + +| Earlier label | Meaning and status | +|---|---| +| (J) | Placeholder for consumer/reachability policies; absent from the supplied LaTeX and not defined by this document | +| `MinimumQueryReaders` | Legacy policy name suggesting a minimum number of distinct consuming queries. Its threshold, counting scope, and activation rule are unspecified in the supplied material; no constraint is asserted here. | +| `HighestSharedQueryMaterialization` | Legacy policy name with no supplied mathematical definition; retained as a migration question, not a defined constraint | +| (K) | Proposed root-accuracy check; Section 4 states the semantic bound and encoding conditions | +| (R) | Proposed retention constraints; lower-bound pattern above | +| (L) | Proposed per-query latency bounds; requires an execution/critical-path model | +| Resource budgets | Proposed memory and CPU limits; requires explicit accounting units and aggregation rules | + +These labels must not be presented as an implemented sequence of constraints (A)–(K). The review text contains no definitions that would justify filling the gaps by assumption. -### What one logical DAG is +## Appendix A. Enumerating DAGs for debugging -A logical DAG rooted at group $G$ is a set of logical candidates and edges $(u, c, i)$ such that: -1. exactly one candidate of $G$ is in it (the root); -2. every candidate $c$ in it has, for each input $i$, exactly one edge $(u, c, i)$, with $u$ a candidate of $G_{c,i}$ and $\alpha^{\log}_{u,c,i} = 1$ (constraints (A) and (G)); -3. every candidate in it is the root or the producer of some edge (constraint (F)); -4. the edges form no cycle (constraint (E)). +This is an inspection tool for small memos, not a planning stage. The MILP selects a plan without listing all alternatives. -A candidate is one node however many edges read it: two consumers that choose the same producer share it, which is what $x_u \in \{0, 1\}$ says. Two consumers of one group may choose different candidates of it. The listing is therefore a choice of one producer per consumer input $(c, i)$, not one candidate per group. +A logical DAG has one selected root candidate, one compatible producer per input of every reached candidate, no unused nodes, and no cycles. Producer choices are per consumer input, not per group. Two consumers can choose different candidates of the same group; choosing the same candidate shares one logical node. -### Pseudocode +The following Rust-style pseudocode preserves the draft's enumeration procedure. Helpers such as `cartesian_product` are schematic; `reaches` follows the stored consumer-to-producer dependency direction and is reflexive. This direction makes the cycle test precise. ```rust /// One logical DAG: its nodes and, for each node's inputs, the chosen producer. +#[derive(Clone)] struct LogicalDag { root: CandidateForMemoGroupId, /// For each node c in the DAG, one producer per input i, in input order. @@ -622,9 +686,14 @@ fn logical_dags(memo: &Memo, root_group: MemoGroupId) -> Vec { /// Choose producers for the inputs of every node in `pending`, in FIFO order. /// Each DAG is reached by exactly one sequence of choices, so none repeats. -fn expand(memo: &Memo, dag: LogicalDag, mut pending: VecDeque, all: &mut Vec) { +fn expand( + memo: &Memo, + dag: LogicalDag, + mut pending: VecDeque, + all: &mut Vec, +) { let Some(consumer) = pending.pop_front() else { - all.push(dag); // every input of every node has a producer + all.push(dag); // Every reached input has a producer. return; }; let candidate = memo.candidate(consumer); @@ -633,11 +702,15 @@ fn expand(memo: &Memo, dag: LogicalDag, mut pending: VecDeque>() })) { - // Constraint (E): a producer that already reaches `consumer` would close a cycle. + // In the stored consumer-to-producer direction, this would close a cycle. if choice.iter().any(|&u| reaches(&dag, u, consumer)) { continue; } @@ -654,8 +727,12 @@ fn expand(memo: &Memo, dag: LogicalDag, mut pending: VecDeque impl Iterator + '_ { - (0..memo.candidates[group.0 as usize].len()).map(move |index| CandidateForMemoGroupId { group, index }) +fn candidates_of( + memo: &Memo, + group: MemoGroupId, +) -> impl Iterator + '_ { + (0..memo.groups[group.0 as usize].candidates.len()) + .map(move |index| CandidateForMemoGroupId { group, index }) } /// Whether `from` reaches `to` through the producers chosen so far @@ -665,7 +742,13 @@ fn reaches(dag: &LogicalDag, from: CandidateForMemoGroupId, to: CandidateForMemo /// Print one DAG as an indented tree; a node read a second time is printed as a /// reference to its first occurrence, so sharing is visible. fn print(memo: &Memo, dag: &LogicalDag) { - fn node(memo: &Memo, dag: &LogicalDag, c: CandidateForMemoGroupId, depth: usize, seen: &mut HashSet) { + fn node( + memo: &Memo, + dag: &LogicalDag, + c: CandidateForMemoGroupId, + depth: usize, + seen: &mut HashSet, + ) { let candidate = memo.candidate(c); if !seen.insert(c) { println!("{}↑ G{}[{}] (shared)", " ".repeat(depth), c.group.0, c.index); @@ -681,33 +764,26 @@ fn print(memo: &Memo, dag: &LogicalDag) { } ``` -For a workload, list from every query root at once: start `pending` with one root candidate per $G_q$ (constraints (C) and (D)), taking the product over queries of their root candidates. A candidate two queries reach is one node, as above. - -### Example - -For the example's query A (root G3) and query B (root G6), with no error caps applied: +For a workload, choose one root candidate per query and initialize the pending queue with the distinct chosen roots. Deduplicate that queue before expansion so a shared root is processed once. -``` -A: 2 DAGs B: 4 DAGs (G4 and G5 choose independently) -G3[0] SummaryEstimate q=0.95 G6[0] Divide - G2[0] SummaryAgg KLL θ={k=200} G4[0] SummaryEstimate q=0.99 - G1[0] Scan G2[0] SummaryAgg KLL θ={k=200} -G3[0] SummaryEstimate q=0.95 G1[0] Scan - G2[1] SummaryAgg KLL θ={k=345} G5[0] SummaryEstimate q=0.50 - G1[0] Scan ↑ G2[0] (shared) - … and (k=200, k=345), (k=345, k=200), (k=345, k=345) -``` +Before accuracy filtering, the running example has two alternatives for A and four for B: G4 and G5 choose independently between the two G2 builds. Joint enumeration therefore has eight choices. These are structural choices only; no claim is made that they meet B's ratio-accuracy requirement. -Listed from both roots at once there are $2 \times 4 = 8$ DAGs. In the one where all three estimates choose `k = 345`, `G2[1]` is a single node read by three consumers, so its cost is counted once. Constraint (K) later accepts only the DAGs that meet each query's target: for B, only (k=345, k=345). Error caps (section 1) remove edges before the listing, so a capped producer never appears. - -### Size +| B's p99 producer | B's p50 producer | State sharing within B | +|---|---|---| +| `k = 200` | `k = 200` | One logical state candidate | +| `k = 200` | `k = 345` | Two state candidates | +| `k = 345` | `k = 200` | Two state candidates | +| `k = 345` | `k = 345` | One logical state candidate | -Counting each consumer input's choice independently gives an upper bound on the number of DAGs rooted at $G$: +The printer marks repeated candidates as references rather than expanding them twice. Physical sharing and cost additionally depend on lifetime and execution context. -$$N(G) = \sum_{v \in G} N(v), \qquad N(v) = \prod_{i} \sum_{u \in P_{v,i},\ \alpha^{\log}_{u,v,i} = 1} N(u),$$ +For an acyclic candidate-dependency space, a tree-expansion count bounds the number of DAGs: -with $N(v) = 1$ for a candidate with no inputs. It is an upper bound because a shared producer is expanded once in a DAG, where the formula expands it once per consumer, and because cyclic choices are dropped. $N$ multiplies along the depth of the memo, which is why only small workloads are listed, and why Stage 3 chooses with the MILP instead. +$$ +N(G)=\sum_{v\in G}N(v),\qquad +N(v)=\prod_i\sum_{u\in P^{\mathrm{log,adm}}_{v,i}}N(u). +$$ -### Physical DAGs +Here, $P^{\mathrm{log,adm}}_{v,i}$ is the set of logically compatible producers for that input, and a leaf has count one. The bound overcounts shared subgraphs. For a cyclic memo, this recurrence is not well-founded; use cycle-aware enumeration or an explicitly bounded unfolding instead. -The same procedure lists physical DAGs from Stage 2's output. Use `PhysicalCandidate`s instead of logical candidates, and `admissible_edges_into(c, i)` (which already applies $\alpha^{\log}$ and $\alpha^{\mathrm{phys}}$) instead of `candidates_of` with `satisfies_logical`. A physical candidate with $\ell = $ None that two consumers in different contexts $\kappa$ choose runs once per context; in the listing it is still one node, and the contexts are a Stage 3 choice (#1207). +For physical inspection, replace logical candidates and compatibility checks with physical candidates and admissible edges. Under the execution-context extension, candidate-level listing alone is not a list of executable plans: instantiate contexts and apply schedule, retention, root-accuracy, and other enabled constraints before comparing results with Stage 3. From b33c0bb6913bd5fbeada509a7ec4d8c1f154f535 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 03:34:35 +0000 Subject: [PATCH 20/67] docs(concepts): fix review issues: one Memo type, decision tree restored, grid width from the window layout, per-input validity caps, metric rule, (K) encoding, (J) definitions, (L)/budgets out of scope Co-Authored-By: Claude Opus 5.5 --- .../memo-groups-properties-requirements.md | 186 +++++++++++++----- 1 file changed, 138 insertions(+), 48 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index ca3c51934..d1fda2719 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -1,8 +1,8 @@ # Memo groups, properties and requirements -This document explains how the optimizer represents alternatives, checks whether they can be composed, and selects an executable plan. It follows the supplied **Optimization Problem Formulation** for the baseline model and labels additional design work **Extension (#1207)**. +This document explains how the optimizer represents alternatives, checks whether they can be composed, and selects an executable plan. It follows the optimization problem formulation (paper, "Optimization Problem Formulation") for the baseline model and labels additional design work **Extension (#1207)**. -The Rust blocks are interface sketches, not standalone compilable implementations. Repository paths and issue references are retained from the supplied draft; their current implementation status has not been independently checked. +The Rust blocks are interface sketches, not compilable code. Paths name today's code; issue numbers name the work that changes it. ## 1. Model and running example @@ -10,7 +10,7 @@ A **workload** contains query expressions, their evaluation schedules, and their Input groups leave producer choices open. Selecting a candidate does not select its entire input plan. Each consumer input chooses a producer separately, and several inputs may choose the same producer. -The expression includes the operator's arguments, including the summary family. In this design, KLL and DDSketch states belong to separate state groups. A final-result group can offer readout candidates that use either family. This resolves the old decision tree's contradiction: it must not put KLL and DDSketch builds in one state group. +The expression includes the operator's arguments, including the summary family. In this design, KLL and DDSketch states belong to separate state groups. A final-result group can offer readout candidates that use either family. So the decision tree's first level (Section 2) chooses the expression, family included; a state group never mixes families. ### Running example @@ -27,7 +27,7 @@ Query A requests the p95 latency over one minute. Query B requests p99 divided b G3, G4, and G5 choose a G2 producer independently. If all choose the same retained physical producer, they can reuse one computed state when their schedules and windows are compatible. An unmaterialized candidate may instead execute separately in different contexts; Section 5 explains that distinction. -The old example assigned illustrative KLL rank errors of 0.008 and 0.005 and added them to obtain Query B's ratio error. That inference is invalid: rank error does not directly bound numerical quantile error or relative ratio error. The example here demonstrates representation and sharing only. Accepting Query B requires an applicable error model, including assumptions about its denominator. +The example shows representation and sharing. Accuracy is checked by Section 4, whose metric rule matters here: KLL bounds **rank** error, and rank error does not bound a quantile's value error or the relative error of a ratio. Query B's relative ratio target therefore needs operands with a relative value-error guarantee, e.g. a DDSketch state group over G1 with `alpha` candidates, unless a declared distribution assumption converts rank error to value error. Query A, with a rank target, can use the KLL candidates directly. ### Notation used by the baseline model @@ -58,7 +58,7 @@ p_v^{\mathrm{log}}=\boldsymbol\theta_v,\qquad p_v^{\mathrm{phys}}=(\tau_v,\ell_v). $$ -The supplied LaTeX uses the producer-set symbol in two different ways: the prose defines a filtered set, while its notation table defines an unfiltered set. This document follows the table and constraints (G)–(H), and gives the filtered set a separate name: +The formulation's text defines the producer set after the requirement check, and its notation table before it. This document follows the table and constraints (G)–(H), and names the filtered set separately: $$ P_{c,i}=G_{c,i}\cap\mathcal C, @@ -128,7 +128,7 @@ For the proposed periodic consumer path, let the consumer interval and phase be | Tumbling grid | $w\mid I$ and $\varphi\equiv\varphi_g\pmod w$ | | None | Runs in the consuming context; no independent delivery schedule | -The supplied formulation also permits a single, non-periodic evaluation. The interval/phase interface below covers only periodic consumers. Rejecting non-periodic consumers is a current scope restriction of that proposed interface, not a mathematical impossibility; supporting them needs explicit evaluation-time membership checks. +The formulation also permits a single, non-periodic evaluation. The interval/phase interface below covers only periodic consumers. Rejecting non-periodic consumers is a current scope restriction of that proposed interface, not a mathematical impossibility; supporting them needs explicit evaluation-time membership checks. ### Invalid combinations @@ -141,28 +141,83 @@ Checked constructors reject these combinations under the proposed schedule model | Materialized lifetime without its required schedule | Delivery times or window grid would be unspecified | | OneEvaluation with a grid, or windowed AcrossEvaluations with a periodic schedule | Schedule kind does not match this model's lifetime representation | | Zero interval/width, or phase outside the canonical interval/width | Invalid schedule representation | -| Ingestion-time grid with nonzero phase | Unsupported by the draft's epoch-anchored ingestion representation | +| Ingestion-time grid with nonzero phase | Ingestion-time grids are anchored at the epoch: the plan carries no origin | -A tumbling grid also needs a known positive width. The `PhysicalProperties` constructor alone cannot validate a width held in logical properties; physical candidate construction must validate the pair. Non-windowed AcrossEvaluations schedules need an explicit representation before that alternative can be offered. +A tumbling grid's width is the candidate's logical window layout $\omega_v$; it is not repeated in the schedule. Building a `PhysicalCandidate` checks the pair: a grid schedule needs a tumbling layout, and its phase must be below the width. Non-windowed AcrossEvaluations schedules need their own representation before that alternative can be offered. + +### Decision tree and order of decisions + +A memo group's candidates are the leaves of a tree. The first level chooses the expression $e_v$ (operator and family); each later level decides one property dimension. A path from the root to a leaf is one physical candidate. + +```mermaid +flowchart TD + G["Memo group G
(logical result)"] --> R1["e_v: exact operator"] + G --> R2["e_v: readout of KLL state"] + G --> R3["e_v: readout of DDSketch state"] + G --> R4["e_v: other operators
(prune: executor support)"] + + R2 --> T1["θ: k = 200"] + R2 --> T2["θ: k = 345"] + R2 --> T3["θ: k = …
(prune: pruning cap)"] + + T2 --> W1["ω: no windows"] + T2 --> W2["ω: tumbling, width w
(prune: w must divide
the group's query range)"] + + W2 --> P1["τ = QueryTime"] + W2 --> P2["τ = IngestionTime
(prune: no ingestion pipeline)"] + + P1 --> L1["ℓ = None"] + P1 --> L2["ℓ = OneEvaluation"] + P1 --> L3["ℓ = AcrossEvaluations
(prune: deployment keeps
no query-time results)"] + P2 --> L4["ℓ = None"] + P2 --> L5["ℓ = AcrossEvaluations"] + P2 -. invalid .-> L6["ℓ = OneEvaluation"] + + L2 --> S1["σ: Periodic (I, φ)
from the gcd closure
of the queries' schedules"] + L3 --> S2["σ: grid phase φ mod w"] + L5 --> S3["σ: grid phase 0"] +``` + +| Level | Decides | Stage | Values offered | Pruned when | +|---|---|---|---|---| +| 1 | expression $e_v$ | Pass 1 and Pass 2 rules | each way to compute the group | the executor cannot build the family | +| 2 | $\boldsymbol\theta_v$ | Pass 1 | the family's predefined list | the pruning cap (Section 4) | +| 3 | $\omega_v$ — Extension | Pass 2 | tumbling widths from the gcd closure of the queries' ranges and steps | the width does not divide the group's query range | +| 4 | $\tau_v$ | Stage 2 | QueryTime, IngestionTime | the source has no ingestion pipeline | +| 5 | $\ell_v$ | Stage 2 | None, OneEvaluation, AcrossEvaluations | invalid with $\tau_v$; the deployment keeps no query-time results | +| 6 | $\sigma_v$ — Extension | Stage 2 | the gcd closure of the schedules of the queries reaching the group | only when $\ell_v \ne$ None | + +The tree does not decide which producer each consumer input reads ($y$), where an unmaterialized candidate runs (its context, Section 5), or how much a materialized candidate retains ($h_v$). Those are Stage 3 choices. + +**The order does not change the candidate set.** The leaves are the product of the dimensions' values minus invalid combinations and pruned values. Each prune rule reads only the value it removes and values decided above it, so any order that keeps each prune rule below the dimensions it reads yields the same leaves. + +**The order changes how many nodes enumeration visits:** $\sum_d \prod_{d' \le d} \lvert \text{values kept at } d' \rvert$. Dimensions that prune much go early; dimensions that multiply without pruning go late. The order above follows four rules: + +1. **Logical before physical.** Stage 1 hash-conses groups and computes $\alpha^{\log}$ once, before Stage 2 multiplies each logical candidate by its physical alternatives. +2. **Strong prunes early.** Executor support and the pruning cap act before windows and physical properties multiply the tree. +3. **Dependent dimensions adjacent.** $\tau \to \ell \to \sigma$ rejects each invalid combination at the first level that can detect it. +4. **Deployment-wide prunes as early as possible.** Without an ingestion pipeline the IngestionTime subtree is pruned once per candidate; moving $\tau$ above $\omega$ would prune it once per group, with the same leaves. + +Alternative orders worth measuring: $\tau$ first, when most sources lack an ingestion pipeline; $\omega$ before $\boldsymbol\theta$, when queries' ranges prune many widths; and requirement-driven (top-down) enumeration, which visits fewer nodes but must take the union of all consumers' requirements so that sharing is not lost. ## 3. Pipeline: interfaces and rules | Stage | Input | Output | Responsibility | |---|---|---|---| -| Stage 0 | Query IR | `DagOfLogicalMemoGroups` | Hash-cons logical expressions | +| Stage 0 | Query IR | `Memo` (groups without candidates) | Hash-cons logical expressions | | Stage 1, Pass 1 | Group DAG | `Memo` | Add initial logical candidates and intermediate groups | | Stage 1, Pass 2 | Memo | `LogicalASAPMemoCandidates` | Add reuse alternatives and compute logical edges | | Stage 2 | Logical memo candidates | `PhysicalASAPMemoCandidates` | Expand physical alternatives and compute compatibility | | Stage 3 | Physical memo candidates | `DagOfSelectedCandidates` | Select implementations, dependencies, and executions | | Realizer/export | Selected DAG | `PhysicalASAPDAG` | Build and export the executable graph | -Enumeration proceeds through expression, parameters, window layout, timing, lifetime, and schedule. Executor and deployment restrictions prune unsupported alternatives as soon as their prerequisites are known. Accuracy pruning is allowed only under the conditions in Section 4. Reordering dimensions preserves the candidate set only when the same alternatives and sound pruning predicates are retained. +Enumeration follows the decision tree in Section 2. Accuracy pruning uses only the caps of Section 4. Rules add groups through `find_or_add_group` and candidates through `add_candidate`. They match group expressions, avoid duplicating an existing expression/property pair, and do not copy consumers to make a new producer visible. After logical generation finishes, edge construction records the possible producer choices. ### Stage 0: groups -`hash_consed` in `crates/logical-asap-optimization/src/stage0.rs` identifies shared logical expressions and query roots. The following preserves the existing group representation described in the draft. Candidate ownership is exposed through the group in the proposed Stage 1 interface. +`hash_consed` in `crates/logical-asap-optimization/src/stage0.rs` identifies shared logical expressions and query roots. Stage 0 and Stage 1 share one type, `Memo`: Stage 0 returns it with every group's candidate list empty, and Stage 1 fills the lists. ```rust pub struct MemoGroupId(pub u32); @@ -176,7 +231,7 @@ pub struct MemoGroup { pub candidates: Vec, } -pub struct DagOfLogicalMemoGroups { +pub struct Memo { pub groups: Vec, pub query_roots: Vec>, } @@ -184,7 +239,7 @@ pub struct DagOfLogicalMemoGroups { `OperatorNode` recursively contains chosen child nodes and therefore describes a fixed plan fragment. `Operator` contains input group IDs and leaves their implementations open. A candidate needs its own expression because a group may offer several ways to compute the same result; it cannot reuse the group's representative `logical_node` as every candidate's expression. -Replacing the group's representative node and child list with a single group-based expression, or removing the one-field `CandidateExpression` wrapper, remains an optional simplification from the review. Neither is required here. +Two simplifications are possible but not adopted: storing the group as one `Operator` instead of its node and child list, and removing the one-field `CandidateExpression` wrapper. ### Stage 1, Pass 1: initial candidates @@ -203,11 +258,6 @@ pub struct CandidateForMemoGroupId { pub index: usize, } -pub struct Memo { - pub groups: Vec, - pub query_roots: Vec>, -} - impl Memo { pub fn find_or_add_group( &mut self, @@ -282,7 +332,7 @@ Migration targets: remove private `root_node` chains, duplicate `input_groups`, ### Stage 1, Pass 2: reuse rules and logical edges -The following preserves the supplied rule inventory and tracking references. Paths are relative to `crates/logical-asap-optimization/src/pass2/memo/`. +Paths are relative to `crates/logical-asap-optimization/src/pass2/memo/`. | Rule / function | File | Result or migration target | |---|---|---| @@ -297,7 +347,7 @@ The following preserves the supplied rule inventory and tracking references. Pat | Shared configurations: `add_state_configuration_candidates` | `rules/shared_configuration_readers.rs` | Remove copied consumers once producer variants are selectable; #1203 2b | | Shared state groups: `add_shared_summary_state_groups` | `state_groups.rs` | Remove per-size groups; #1203 2b / #1160 | -After the rules, build logical edges and apply any proven-safe accuracy pruning. `input_index_of_consumer` replaces the ambiguous `input` field name throughout this document's edge interfaces. +After the rules, Pass 2 builds the logical edges and applies the pruning caps of Section 4. Edges name the consumer input `input_index_of_consumer`. ```rust fn apply(memo: &mut Memo) -> Result<(), Pass2RuleError>; @@ -361,12 +411,17 @@ pub struct EvaluationSchedule { interval_ms: u64, phase_ms: u64, } -impl PhysicalProperties { - pub fn delivers_on(&self, times: &EvaluationSchedule) -> bool; +impl PhysicalCandidate { + /// Whether this producer delivers on `times` (Section 2's alignment table). + /// Reads the grid width from the candidate's logical window layout. + pub fn delivers_on(&self, memo: &Memo, times: &EvaluationSchedule) -> bool; } +/// α^phys: timing and lifetime (Section 2's table), and, for a materialized +/// consumer, that the producer delivers on the consumer's own schedule. pub fn satisfies_physical( - producer: &PhysicalProperties, - consumer: &PhysicalProperties, + memo: &Memo, + producer: &PhysicalCandidate, + consumer: &PhysicalCandidate, ) -> bool; pub struct PhysicalCandidateId(pub usize); pub struct PhysicalEdgeId(pub usize); @@ -408,15 +463,13 @@ pub struct PhysicalCandidateKey { } ``` -`PhysicalCandidateId` indexes an emitted candidate. `PhysicalCandidateKey` describes an alternative rejected before it received an ID, allowing diagnostics to explain pruning. The admissible-edge accessor returns edge IDs, not the unfiltered producer set from Section 1. - -The `delivers_on` sketch needs access to a tumbling producer's logical window width. Supply that context in its implementation or move the check to the full physical-candidate interface; the phase-only schedule is insufficient by itself. +`PhysicalCandidateId` indexes an emitted candidate. `PhysicalCandidateKey` describes an alternative rejected before it received an ID, allowing diagnostics to explain pruning. The admissible-edge accessor returns edge IDs, not the unfiltered producer set from Section 1. `delivers_on` and `satisfies_physical` take candidates rather than bare properties because a tumbling grid's width lives in the logical window layout. Migration targets include replacing `Materialization` and duplicated physical fields with checked properties, and replacing `ReaderConstraint::{IncompatibleOutputs, AllowedOutputs}` with compatibility and schedule checks. Policy constraints involving the number of selected consuming queries are a separate concern; see Section 5. ### Stage 3: baseline selection and realization -The baseline uses the variables and constraints in the supplied LaTeX. Stage 3 adds no candidates or potential dependencies. +The baseline uses the formulation's variables and constraints. Stage 3 adds no candidates or potential dependencies. | Variable | Meaning | |---|---| @@ -480,7 +533,7 @@ The realizer uses the selected dependencies to build the executable graph. Expor ## 4. Accuracy and safe pruning — Extension (#1207) -The supplied LaTeX checks logical requirements per edge. The proposed extension additionally checks composed accuracy at each query root. This section describes the error model needed for that check; it is not an additional property dimension. +The formulation checks logical requirements per edge. This extension also checks composed accuracy at each query root. This section gives the error model and its MILP encoding; error is not a property dimension. ### Local propagation @@ -504,7 +557,9 @@ $$ | Sum over a known number of values | Sum of their bounds; a common per-value bound can be multiplied by that count | | Ratio, relative error | Requires nonzero true denominator and denominator relative error below one; bound given below | -A filter does not automatically preserve every accuracy guarantee. Group counts and metric conversions may need metadata or assumptions beyond the expression syntax. Propagation factors must be justified by the relevant operator/error contract. +A filter does not automatically preserve every accuracy guarantee. Group counts may need metadata beyond the expression syntax. Each factor must follow from the operator's error contract. + +**Metric rule.** Every local guarantee states its metric: absolute (in value units, or a multiple of a stream norm), relative, or rank. A factor $\beta_{c,i}$ is defined for one input metric and one output metric. An input whose metric the consumer's error contract cannot convert gives no edge ($\alpha^{\log} = 0$); the conversion is never assumed. For example, a relative-error ratio cannot read a KLL quantile (rank error) unless a declared distribution assumption supplies the conversion. For a ratio, let the operand relative-error bounds be $\varepsilon_X$ and $\varepsilon_Y$. If the numerator's relative error is defined and the denominator is nonzero, then: @@ -523,12 +578,15 @@ $$ Both local factors are then `1 / (1 - b)`. This is a sufficient bound, not an exact equality or a claim that all feasible nonlinear plans remain represented. -The draft's approved interface is retained here as an interface proposal: +Interface (approved 2026-11-05): ```rust pub struct ErrorPropagation { + /// β_{c,i} per input, for that input's metric. pub factor_per_input: Vec, - pub validity_cap: Option, + /// b_{c,i} per input: the cap under which the linear bound holds; None + /// when the bound holds everywhere (e.g. a ratio caps only its denominator). + pub validity_cap_per_input: Vec>, } impl LogicalRequirement { @@ -538,7 +596,7 @@ impl LogicalRequirement { } ``` -`factor_per_input` contains the local beta factors. A scalar `validity_cap` can represent a conservative common cap. Operator-specific caps, such as a bound only on a ratio's denominator, need either a documented input mapping or a per-input representation before implementation. Unsupported error models must be reported as unsupported rather than supplied with guessed factors. +`factor_per_input` holds the local β factors and `validity_cap_per_input` the per-input caps, aligned by input index. An operator without an error contract is reported as unsupported; no factor is guessed. ### Query composition and root checks @@ -554,9 +612,33 @@ $$ Here, $\varepsilon_q$ and $\delta_q$ are the query target's error and failure limits. Repeated paths can amplify error and therefore contribute to lambda. A shared failure event is counted once in the union bound; distinct readout events need distinct source identities unless a joint guarantee is available. -These equations describe the required bound. They are a linear MILP encoding only when the factors and source identities are fixed for the indexed alternatives and the reachability indicators are correctly linked to selected edges and outputs. Candidate-dependent paths, metric changes, or execution-dependent random events require a more detailed encoding. The draft does not supply that encoding, so the equations must not be described as a completed constraint implementation. +**MILP encoding.** Path factors $\lambda_{q,s}$ depend on which edges are selected, so the MILP does not use them directly. It carries an error bound per selected candidate instead, which needs no path enumeration and is the same for every query reading that candidate: + +$$ +E_c \ge \varepsilon^{\mathrm{loc}}_c + \sum_{i} \beta_{c,i} F_{c,i}, +\qquad +F_{c,i} \ge E_u - M_{c,i}\,(1 - y_{u,c,i}) \quad \forall (u,c,i)\in\mathcal E, +$$ -This replaces the old unexplained `a_{q,u}` and the mixture of candidate-, edge-, and query-indexed lambda symbols. Local propagation uses beta; query propagation uses lambda; neither should be inferred from bare edge selection alone. +$$ +D_c \ge \delta^{\mathrm{loc}}_c + \sum_{i} H_{c,i}, +\qquad +H_{c,i} \ge D_u - (1 - y_{u,c,i}) \quad \forall (u,c,i)\in\mathcal E, +$$ + +$$ +E_v \le \varepsilon_q + M_q\,(1 - z_{q,v}), +\qquad +D_v \le \delta_q + (1 - z_{q,v}) +\qquad \forall q,\ v\in G_q. \tag{K} +$$ + +- $E_c, F_{c,i}, D_c, H_{c,i} \ge 0$ are continuous. $\varepsilon^{\mathrm{loc}}_c$ and $\delta^{\mathrm{loc}}_c$ are the candidate's own guarantee (zero for an exact operator); $\beta_{c,i}$ and the metric rule come from `error_propagation`. +- $M_{c,i}$ is any upper bound on $E_u$ over the producers of input $i$ (their largest feasible bound); $M_q$ likewise for the root. Exactly one $y_{u,c,i} = 1$ per selected input (constraint (A)), so $F_{c,i}$ equals the chosen producer's error bound. +- A producer reached by two paths contributes on both, which is the worst-case sum. The failure probabilities add by a union bound; a failure event reached twice is counted twice, which is conservative. +- Under the context extension (Section 5), the variables are indexed by context, $E_{\kappa,c}$, and the edges by $y_{\kappa,u,c,i}$. + +$\lambda_{q,c,i}$ is still used, but only for the pruning caps below, where a fixed lower bound suffices. ### Validity caps and pruning caps @@ -576,7 +658,7 @@ Compute the parameter floor from this cap only under the ordering condition in S ## 5. Execution contexts and other selection extensions — #1207 -These extensions go beyond constraints (A)–(I) and the fixed-cost assumption in the supplied LaTeX. They are separated here so that the baseline remains understandable without an unfinished context MILP. +These extensions go beyond constraints (A)–(I) and the formulation's fixed-cost assumption. They are separated here so that the baseline reads on its own. ### Execution identity @@ -643,19 +725,27 @@ $$ The retention domain, upper bounds, activation linking, and all coefficient units must be specified by the implementation. If required retention depends on selected downstream queries, that dependence needs explicit linking constraints; it is not a fixed coefficient merely because it has a name. -### Extension status and legacy policy names +### Consumer-count policies (J) -| Earlier label | Meaning and status | -|---|---| -| (J) | Placeholder for consumer/reachability policies; absent from the supplied LaTeX and not defined by this document | -| `MinimumQueryReaders` | Legacy policy name suggesting a minimum number of distinct consuming queries. Its threshold, counting scope, and activation rule are unspecified in the supplied material; no constraint is asserted here. | -| `HighestSharedQueryMaterialization` | Legacy policy name with no supplied mathematical definition; retained as a migration question, not a defined constraint | -| (K) | Proposed root-accuracy check; Section 4 states the semantic bound and encoding conditions | -| (R) | Proposed retention constraints; lower-bound pattern above | -| (L) | Proposed per-query latency bounds; requires an execution/critical-path model | -| Resource budgets | Proposed memory and CPU limits; requires explicit accounting units and aggregation rules | +Two policies from today's code (`ReaderConstraint` in `crates/types/src/stages/physical/graph.rs`) restrict which queries may reach a node. They need reach indicators $a_{q,v} \in \{0,1\}$: query $q$'s selected plan contains $v$. + +$$ +a_{q,v} \ge z_{q,v},\qquad +a_{q,u} \ge a_{q,c} + y_{u,c,i} - 1,\qquad +a_{q,u} \le z_{q,u} + \sum_{(u,c,i)\in\mathcal E} w_{q,u,c,i},\qquad +w_{q,u,c,i} \le a_{q,c},\ w_{q,u,c,i} \le y_{u,c,i}. +$$ + +| Policy | Meaning | Constraint (J) | +|---|---|---| +| `MinimumQueryReaders { candidate, minimum }` | a selected candidate is worth keeping only when at least `minimum` queries reach it | $\sum_q a_{q,v} \ge \text{minimum}\cdot x_v$ | +| `HighestSharedQueryMaterialization { candidate }` | a non-summary query materialization is placed where sharing stops: it is a query output, or one of its selected consumers is reached by at most one query | $\sum_q z_{q,v} + \sum_{(v,c,i)} s_{v,c,i} \ge x_v$, with $s_{v,c,i} \le y_{v,c,i}$ and $\sum_q a_{q,c} \le 1 + \lvert\mathcal Q\rvert\,(1 - s_{v,c,i})$ | + +`ReaderConstraint::{IncompatibleOutputs, AllowedOutputs}` are not kept: the schedule check of Section 2 replaces them. + +### Out of this design -These labels must not be presented as an implemented sequence of constraints (A)–(K). The review text contains no definitions that would justify filling the gaps by assumption. +Latency bounds (L) and resource budgets need an execution and critical-path model, and accounting units, that this document does not define. They are left to a later design; no interface for them is part of this one. ## Appendix A. Enumerating DAGs for debugging @@ -663,7 +753,7 @@ This is an inspection tool for small memos, not a planning stage. The MILP selec A logical DAG has one selected root candidate, one compatible producer per input of every reached candidate, no unused nodes, and no cycles. Producer choices are per consumer input, not per group. Two consumers can choose different candidates of the same group; choosing the same candidate shares one logical node. -The following Rust-style pseudocode preserves the draft's enumeration procedure. Helpers such as `cartesian_product` are schematic; `reaches` follows the stored consumer-to-producer dependency direction and is reflexive. This direction makes the cycle test precise. +Rust-style pseudocode follows. Helpers such as `cartesian_product` are schematic; `reaches` follows the stored consumer-to-producer dependency direction and is reflexive. This direction makes the cycle test precise. ```rust /// One logical DAG: its nodes and, for each node's inputs, the chosen producer. From bc00292729e1f1e05342ebd450b41d9052e7c72c Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 03:45:14 +0000 Subject: [PATCH 21/67] docs(concepts): delete both consumer-count policies; no constraint (J) Co-Authored-By: Claude Opus 5.5 --- .../memo-groups-properties-requirements.md | 20 +++---------------- 1 file changed, 3 insertions(+), 17 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index d1fda2719..b674f226d 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -465,7 +465,7 @@ pub struct PhysicalCandidateKey { `PhysicalCandidateId` indexes an emitted candidate. `PhysicalCandidateKey` describes an alternative rejected before it received an ID, allowing diagnostics to explain pruning. The admissible-edge accessor returns edge IDs, not the unfiltered producer set from Section 1. `delivers_on` and `satisfies_physical` take candidates rather than bare properties because a tumbling grid's width lives in the logical window layout. -Migration targets include replacing `Materialization` and duplicated physical fields with checked properties, and replacing `ReaderConstraint::{IncompatibleOutputs, AllowedOutputs}` with compatibility and schedule checks. Policy constraints involving the number of selected consuming queries are a separate concern; see Section 5. +Migration targets include replacing `Materialization` and duplicated physical fields with checked properties, and replacing `ReaderConstraint::{IncompatibleOutputs, AllowedOutputs}` with compatibility and schedule checks. `ReaderConstraint::{MinimumQueryReaders, HighestSharedQueryMaterialization}` are deleted too; see Section 5. ### Stage 3: baseline selection and realization @@ -725,23 +725,9 @@ $$ The retention domain, upper bounds, activation linking, and all coefficient units must be specified by the implementation. If required retention depends on selected downstream queries, that dependence needs explicit linking constraints; it is not a fixed coefficient merely because it has a name. -### Consumer-count policies (J) +### No consumer-count policies -Two policies from today's code (`ReaderConstraint` in `crates/types/src/stages/physical/graph.rs`) restrict which queries may reach a node. They need reach indicators $a_{q,v} \in \{0,1\}$: query $q$'s selected plan contains $v$. - -$$ -a_{q,v} \ge z_{q,v},\qquad -a_{q,u} \ge a_{q,c} + y_{u,c,i} - 1,\qquad -a_{q,u} \le z_{q,u} + \sum_{(u,c,i)\in\mathcal E} w_{q,u,c,i},\qquad -w_{q,u,c,i} \le a_{q,c},\ w_{q,u,c,i} \le y_{u,c,i}. -$$ - -| Policy | Meaning | Constraint (J) | -|---|---|---| -| `MinimumQueryReaders { candidate, minimum }` | a selected candidate is worth keeping only when at least `minimum` queries reach it | $\sum_q a_{q,v} \ge \text{minimum}\cdot x_v$ | -| `HighestSharedQueryMaterialization { candidate }` | a non-summary query materialization is placed where sharing stops: it is a query output, or one of its selected consumers is reached by at most one query | $\sum_q z_{q,v} + \sum_{(v,c,i)} s_{v,c,i} \ge x_v$, with $s_{v,c,i} \le y_{v,c,i}$ and $\sum_q a_{q,c} \le 1 + \lvert\mathcal Q\rvert\,(1 - s_{v,c,i})$ | - -`ReaderConstraint::{IncompatibleOutputs, AllowedOutputs}` are not kept: the schedule check of Section 2 replaces them. +Today's `ReaderConstraint` (`crates/types/src/stages/physical/graph.rs`) is deleted with all four variants (decided 2026-11-06). `IncompatibleOutputs` and `AllowedOutputs` are replaced by the schedule check of Section 2. `MinimumQueryReaders` (keep a candidate only if enough queries reach it) and `HighestSharedQueryMaterialization` (materialize where sharing stops) are left to the cost model: materializing for one query, or below the point where sharing stops, costs more and is not chosen. There is no constraint (J). ### Out of this design From 019fdb8d1c41bafcec8bc31c2dba9a032053ca18 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 03:47:11 +0000 Subject: [PATCH 22/67] docs(concepts): (K) encoding approved; worst-case counting affects bounds, not cost Co-Authored-By: Claude Opus 5.5 --- .../design_docs/concepts/memo-groups-properties-requirements.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index b674f226d..647b188ca 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -635,7 +635,7 @@ $$ - $E_c, F_{c,i}, D_c, H_{c,i} \ge 0$ are continuous. $\varepsilon^{\mathrm{loc}}_c$ and $\delta^{\mathrm{loc}}_c$ are the candidate's own guarantee (zero for an exact operator); $\beta_{c,i}$ and the metric rule come from `error_propagation`. - $M_{c,i}$ is any upper bound on $E_u$ over the producers of input $i$ (their largest feasible bound); $M_q$ likewise for the root. Exactly one $y_{u,c,i} = 1$ per selected input (constraint (A)), so $F_{c,i}$ equals the chosen producer's error bound. -- A producer reached by two paths contributes on both, which is the worst-case sum. The failure probabilities add by a union bound; a failure event reached twice is counted twice, which is conservative. +- A producer reached by two paths contributes on both, which is the worst-case sum. The failure probabilities add by a union bound; a failure event reached twice is counted twice, which is conservative. This affects only the error bounds, not cost: the objective counts each selected candidate once through $x_v$. Being conservative can only make the MILP pass over a cheaper plan that would in fact meet the target; it never accepts one that does not. Counting a shared failure event once needs failure-event identities, left for later. - Under the context extension (Section 5), the variables are indexed by context, $E_{\kappa,c}$, and the edges by $y_{\kappa,u,c,i}$. $\lambda_{q,c,i}$ is still used, but only for the pruning caps below, where a fixed lower bound suffices. From 7e86e17d89aff65c10a563d9e77e7ffdd122f022 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 03:54:06 +0000 Subject: [PATCH 23/67] docs(concepts): running example follows the metric rule (B reads a DDSketch state) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 21 ++++++++++--------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 647b188ca..003c6e75d 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -14,20 +14,21 @@ The expression includes the operator's arguments, including the summary family. ### Running example -Query A requests the p95 latency over one minute. Query B requests p99 divided by p50 over the same input. Both run every ten seconds. +Query A requests the p95 latency over one minute with a rank-error target. Query B requests p99 divided by p50 over the same input with a relative-error target. Both run every ten seconds. | Group | Result | Candidate expressions | |---|---|---| | G1 | Raw latency input | `Scan latency[1m]` | | G2 | KLL state over G1 | `SummaryAgg[KLL](G1)` with `k = 200` or `k = 345` | +| G2′ | DDSketch state over G1 | `SummaryAgg[DDSketch](G1)` with `alpha = 0.01` or `alpha = 0.02` | | G3 | Query A output | `SummaryEstimate[q=0.95](G2)` | -| G4 | p99 latency | `SummaryEstimate[q=0.99](G2)` | -| G5 | p50 latency | `SummaryEstimate[q=0.50](G2)` | +| G4 | p99 latency | `SummaryEstimate[q=0.99](G2′)` | +| G5 | p50 latency | `SummaryEstimate[q=0.50](G2′)` | | G6 | Query B output | `Divide(G4, G5)` | -G3, G4, and G5 choose a G2 producer independently. If all choose the same retained physical producer, they can reuse one computed state when their schedules and windows are compatible. An unmaterialized candidate may instead execute separately in different contexts; Section 5 explains that distinction. +KLL and DDSketch states are separate groups because the family is part of the expression. The metric rule of Section 4 decides which consumer may read which: KLL bounds **rank** error, which does not bound a quantile's value error or a ratio's relative error, so B's operands read the DDSketch state (relative value error), and A reads the KLL state. -The example shows representation and sharing. Accuracy is checked by Section 4, whose metric rule matters here: KLL bounds **rank** error, and rank error does not bound a quantile's value error or the relative error of a ratio. Query B's relative ratio target therefore needs operands with a relative value-error guarantee, e.g. a DDSketch state group over G1 with `alpha` candidates, unless a declared distribution assumption converts rank error to value error. Query A, with a rank target, can use the KLL candidates directly. +G4 and G5 choose a G2′ producer independently. If both choose the same retained physical producer, they reuse one computed state when their schedules and windows are compatible. An unmaterialized candidate may instead execute separately in different contexts; Section 5 explains that distinction. ### Notation used by the baseline model @@ -842,14 +843,14 @@ fn print(memo: &Memo, dag: &LogicalDag) { For a workload, choose one root candidate per query and initialize the pending queue with the distinct chosen roots. Deduplicate that queue before expansion so a shared root is processed once. -Before accuracy filtering, the running example has two alternatives for A and four for B: G4 and G5 choose independently between the two G2 builds. Joint enumeration therefore has eight choices. These are structural choices only; no claim is made that they meet B's ratio-accuracy requirement. +Before accuracy filtering, the running example has two alternatives for A (the two G2 builds) and four for B: G4 and G5 choose independently between the two G2′ builds. Joint enumeration therefore has eight choices. These are structural choices only; constraint (K) decides which meet each target. | B's p99 producer | B's p50 producer | State sharing within B | |---|---|---| -| `k = 200` | `k = 200` | One logical state candidate | -| `k = 200` | `k = 345` | Two state candidates | -| `k = 345` | `k = 200` | Two state candidates | -| `k = 345` | `k = 345` | One logical state candidate | +| `alpha = 0.01` | `alpha = 0.01` | One logical state candidate | +| `alpha = 0.01` | `alpha = 0.02` | Two state candidates | +| `alpha = 0.02` | `alpha = 0.01` | Two state candidates | +| `alpha = 0.02` | `alpha = 0.02` | One logical state candidate | The printer marks repeated candidates as references rather than expanding them twice. Physical sharing and cost additionally depend on lifetime and execution context. From 428aad9e58f7b28da41bfe38d55ae0ca5f6d101f Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 04:02:18 +0000 Subject: [PATCH 24/67] docs(concepts): retention need = time horizon of the served queries; latency ignored for now Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../concepts/memo-groups-properties-requirements.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 003c6e75d..e3f2d7699 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -706,7 +706,7 @@ To avoid confusing per-execution costs with the baseline's horizon-wide cost, us | $\rho_\kappa$ | Fixed number or expected number of context executions over the common horizon | | $h_v$ | Retention amount in a declared unit, such as retained windows | | $\pi_v$ | Horizon-wide retention cost per unit of retained amount | -| $\operatorname{need}_{\kappa,v,c,i}$ | Retention amount producer v needs to serve this consumer input in this context | +| $\operatorname{need}_{\kappa,v,c,i}$ | Retention amount producer v needs to serve this consumer input in this context: the time horizon the queries served by that context read, e.g. a 5m range over 10 s tumbling windows needs 30 windows. Query latency is ignored for now (decided 2026-11-06) | A proposed linear objective is: @@ -724,7 +724,7 @@ $$ h_v\ge\operatorname{need}_{\kappa,v,c,i}\,y_{\kappa,v,c,i}. $$ -The retention domain, upper bounds, activation linking, and all coefficient units must be specified by the implementation. If required retention depends on selected downstream queries, that dependence needs explicit linking constraints; it is not a fixed coefficient merely because it has a name. +The need is fixed per edge from the time horizon of the queries it serves, so constraint (R) is linear. It ignores query latency: a result is assumed read as soon as its evaluation starts, so a slow consumer that reads later does not extend the horizon. Modeling that needs the latency design, which is out of scope (Section 5). The retention unit (windows, or seconds of data) and an upper bound on $h_v$ are fixed per candidate when Stage 2 builds it. ### No consumer-count policies From 39d25f58c77b6e5fd5f659ab3c22a9c4ba634352 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 14:33:22 +0000 Subject: [PATCH 25/67] docs(concepts): CandidateForMemoGroupId.index -> index_in_group Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../concepts/memo-groups-properties-requirements.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index e3f2d7699..d902b8b5e 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -256,7 +256,8 @@ Implementation locations: `crates/logical-asap-optimization/src/pass1/per_comput ```rust pub struct CandidateForMemoGroupId { pub group: MemoGroupId, - pub index: usize, + /// Position in the group's `candidates` list. + pub index_in_group: usize, } impl Memo { @@ -809,7 +810,7 @@ fn candidates_of( group: MemoGroupId, ) -> impl Iterator + '_ { (0..memo.groups[group.0 as usize].candidates.len()) - .map(move |index| CandidateForMemoGroupId { group, index }) + .map(move |index_in_group| CandidateForMemoGroupId { group, index_in_group }) } /// Whether `from` reaches `to` through the producers chosen so far @@ -828,10 +829,10 @@ fn print(memo: &Memo, dag: &LogicalDag) { ) { let candidate = memo.candidate(c); if !seen.insert(c) { - println!("{}↑ G{}[{}] (shared)", " ".repeat(depth), c.group.0, c.index); + println!("{}↑ G{}[{}] (shared)", " ".repeat(depth), c.group.0, c.index_in_group); return; } - println!("{}G{}[{}] {} {:?}", " ".repeat(depth), c.group.0, c.index, + println!("{}G{}[{}] {} {:?}", " ".repeat(depth), c.group.0, c.index_in_group, candidate.expression.operator.name(), candidate.logical_properties); for &u in dag.producers.get(&c).into_iter().flatten() { node(memo, dag, u, depth + 1, seen); From 949b3e172d2bc21ba8352d8b5d53a397a1f987d5 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 15:07:35 +0000 Subject: [PATCH 26/67] =?UTF-8?q?docs(concepts):=20define=20schedule=20?= =?UTF-8?q?=CF=83;=20a=20grid=20window=20is=20readable=20once=20it=20ends?= =?UTF-8?q?=20(no=20refresh=20interval)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../concepts/memo-groups-properties-requirements.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index d902b8b5e..6523ddf8f 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -89,7 +89,7 @@ This is the single reference table for property dimensions. Stage descriptions b | Timing | `IngestionTime` or `QueryTime` | `ExecutionTiming` | | Lifetime | `None`: discard after the consuming execution; `OneEvaluation`: retain until consumers within that evaluation finish; `AcrossEvaluations`: retain across evaluations in the specified period. A non-`None` lifetime means materialized. | `Lifetime`; #1200 / #1214 | | Medium — Extension | `Memory` or `Disk`, carried by a materialized lifetime | `Medium`; #1214 | -| Schedule — Extension | For `OneEvaluation`, a periodic interval and phase; for windowed `AcrossEvaluations`, a tumbling-window grid phase. Unmaterialized candidates have no independent schedule. | `Schedule`; #1200 PR 2 | +| Schedule — Extension | When a materialized result can be read. For `OneEvaluation`: the periodic times it is computed (interval and phase). For windowed `AcrossEvaluations`: the phase of its tumbling-window grid; the width is the window layout's, and each window is readable as soon as it ends. Unmaterialized candidates have no independent schedule. | `Schedule`; #1200 PR 2 | Error guarantees are derived from parameters and the statistic being read. They include an error metric, a bound, and a failure probability; exact computation introduces no approximation error. The same state may support different readouts, so its guarantee must be interpreted for the relevant statistic. @@ -119,7 +119,11 @@ Physical requirements are derived from the consumer's timing and schedule, rathe QT means QueryTime; IT means IngestionTime. Schedule, data coverage, and retention checks still apply to allowed combinations. -**Periodic versus grid.** A periodic schedule with interval ten seconds and phase two produces at 2, 12, 22, … seconds. A five-second tumbling grid with phase zero partitions the input into windows [0, 5), [5, 10), … . The first describes evaluation times; the second describes window boundaries. Grid alignment alone does not ensure sufficient retained history. +**Periodic versus grid.** The schedule σ says when a materialized result can be read. +- **Periodic** (`OneEvaluation`): the result is computed at φ + I·ℤ and held until that evaluation's consumers finish. With interval 10 s and phase 2 s it exists at 2, 12, 22, … s. +- **Tumbling-window grid** (windowed `AcrossEvaluations`): the input is cut into windows [φ + k·w, φ + (k+1)·w), with w from the window layout ω. Each window is built once, when it ends, and is readable from then on (decided 2026-11-06; there is no separate refresh interval). A consumer can read a range that ends on a window boundary. With w = 10 s and phase 0, a `[1m]` query evaluated every 30 s at phase 0 merges the newest 6 windows; one evaluated at phase 15 s cannot, because its range ends inside a window. + +The grid stores only its phase: the width is ω's, so nothing is stored twice. Building each window when it ends costs the same number of window builds as building several at once later. Grid alignment alone does not ensure sufficient retained history; that is $h_v$ (Section 5). For the proposed periodic consumer path, let the consumer interval and phase be $I$ and $\varphi$. Let the producer interval and phase be $I_s$ and $\varphi_s$, and its tumbling width and grid phase be $w$ and $\varphi_g$. @@ -390,6 +394,8 @@ pub enum Lifetime { } pub enum Schedule { Periodic { interval_ms: u64, phase_ms: u64 }, + /// Phase of the window grid; the width is the window layout's (ω), and + /// each window is readable once it ends. TumblingWindowGrid { phase_ms: u64 }, } pub struct PhysicalProperties { From 12ec9fb76b7f3229651dd7008ddfbf7d4f6c2e2b Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 15:11:37 +0000 Subject: [PATCH 27/67] =?UTF-8?q?docs(concepts):=20drop=20the=20physical?= =?UTF-8?q?=20schedule=20=CF=83;=20p^phys=20=3D=20(=CF=84,=20=E2=84=93),?= =?UTF-8?q?=20grids=20anchored=20at=20phase=200?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A OneEvaluation result belongs to one evaluation context and runs on that context's schedule, chosen by Stage 3 through x_{κ,v}. Every tumbling grid is anchored at the epoch; a phase is absorbed by the width, whose candidates come from the gcd closure of ranges, steps and phases. A consumer in κ reads a windowed chain iff w | I_κ and w | φ_κ (delivers_on). The only invalid combination left is (IngestionTime, OneEvaluation). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 105 ++++++++---------- 1 file changed, 45 insertions(+), 60 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 6523ddf8f..4c737e0f7 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -85,19 +85,18 @@ This is the single reference table for property dimensions. Stage descriptions b | Dimension | Values and meaning | Code reference or target | |---|---|---| | Parameters | A predefined configuration list per family: KLL and KMV `k`; DDSketch `alpha`; HLL `precision`; CMS and CountSketch `width` and `depth`; heap variants also `heap_size`; UnivMon its sizing parameters. Exact operators have no sketch parameters. | `SketchParams` in `crates/types/src/ir/schema/sketch.rs`; predefined lists are the #1203 target | -| Window layout — Extension | No layout, or tumbling windows with a positive width. Candidate widths come from supported divisors derived from workload ranges and steps. Sliding merge/subtraction layouts remain future work. Window count is derived from range and width; retention amount is selected later. | `WindowLayout`; `pass2/memo/window_candidates.rs`; #1160 | +| Window layout — Extension | No layout, or tumbling windows with a positive width. Every grid is anchored at phase 0 (the epoch), so the width alone fixes it. Candidate widths come from the gcd closure of the queries' ranges, steps and phases. Sliding merge/subtraction layouts remain future work. Window count is derived from range and width; retention amount is selected later. | `WindowLayout`; `pass2/memo/window_candidates.rs`; #1160 | | Timing | `IngestionTime` or `QueryTime` | `ExecutionTiming` | | Lifetime | `None`: discard after the consuming execution; `OneEvaluation`: retain until consumers within that evaluation finish; `AcrossEvaluations`: retain across evaluations in the specified period. A non-`None` lifetime means materialized. | `Lifetime`; #1200 / #1214 | | Medium — Extension | `Memory` or `Disk`, carried by a materialized lifetime | `Medium`; #1214 | -| Schedule — Extension | When a materialized result can be read. For `OneEvaluation`: the periodic times it is computed (interval and phase). For windowed `AcrossEvaluations`: the phase of its tumbling-window grid; the width is the window layout's, and each window is readable as soon as it ends. Unmaterialized candidates have no independent schedule. | `Schedule`; #1200 PR 2 | Error guarantees are derived from parameters and the statistic being read. They include an error metric, a bound, and a failure probability; exact computation introduces no approximation error. The same state may support different readouts, so its guarantee must be interpreted for the relevant statistic. -The extended property tuples add window layout and schedule: +The extended logical properties add the window layout; the physical properties stay the formulation's: $$ p_v^{\mathrm{log}}=(\boldsymbol\theta_v,\omega_v),\qquad -p_v^{\mathrm{phys}}=(\tau_v,\ell_v,\sigma_v). +p_v^{\mathrm{phys}}=(\tau_v,\ell_v). $$ ### Logical requirements @@ -110,45 +109,41 @@ The required input group's identity already constrains the state family and data ### Physical requirements -Physical requirements are derived from the consumer's timing and schedule, rather than stored redundantly. For an unmaterialized consumer, the schedule comes from its execution context during selection. Stage 2 checks the parts known from candidate properties; Stage 3 checks any remaining context-dependent delivery condition. +Physical requirements are derived from the consumer's timing and execution context, rather than stored redundantly. Stage 2 checks timing against lifetime (the table below). Stage 3 checks alignment per context (below), with the schedule of the context the consumer runs in. | Consumer timing | QT, None | QT, OneEvaluation | QT, AcrossEvaluations | IT, None | IT, AcrossEvaluations | |---|---|---|---|---|---| | QueryTime | Allowed | Allowed | Allowed | Rejected | Allowed | | IngestionTime | Rejected | Rejected | Rejected | Allowed | Allowed | -QT means QueryTime; IT means IngestionTime. Schedule, data coverage, and retention checks still apply to allowed combinations. +QT means QueryTime; IT means IngestionTime. Alignment, data coverage, and retention checks still apply to allowed combinations. -**Periodic versus grid.** The schedule σ says when a materialized result can be read. -- **Periodic** (`OneEvaluation`): the result is computed at φ + I·ℤ and held until that evaluation's consumers finish. With interval 10 s and phase 2 s it exists at 2, 12, 22, … s. -- **Tumbling-window grid** (windowed `AcrossEvaluations`): the input is cut into windows [φ + k·w, φ + (k+1)·w), with w from the window layout ω. Each window is built once, when it ends, and is readable from then on (decided 2026-11-06; there is no separate refresh interval). A consumer can read a range that ends on a window boundary. With w = 10 s and phase 0, a `[1m]` query evaluated every 30 s at phase 0 merges the newest 6 windows; one evaluated at phase 15 s cannot, because its range ends inside a window. +**Alignment.** There is no physical schedule property. When a result can be read follows from its lifetime and window layout: +- **None** and **OneEvaluation**: the result is computed inside one evaluation context κ (Section 5) and read there. A OneEvaluation result belongs to one context: queries with the same schedule are evaluated together, and that evaluation's consumers share the result. When it runs is the context's schedule, and Stage 3 chooses it through $x_{\kappa,v}$, just as for ℓ = None. Sharing across differently scheduled queries through a gcd schedule is not offered: it is rarely beneficial and wastes runs. +- **Windowed AcrossEvaluations**: the input is cut into tumbling windows [k·w, (k+1)·w), with w from the window layout ω and the grid anchored at phase 0 (the epoch). Each window is built when it ends and can be read from then on. A consumer can read a range that ends on a window boundary. With w = 10 s, a `[1m]` query evaluated every 30 s at phase 0 merges the newest 6 windows; one evaluated at phase 15 s cannot, because its range ends inside a window; with that query in the workload, the gcd closure also offers w = 5 s, which both can read. -The grid stores only its phase: the width is ω's, so nothing is stored twice. Building each window when it ends costs the same number of window builds as building several at once later. Grid alignment alone does not ensure sufficient retained history; that is $h_v$ (Section 5). +A grid phase is not needed: a phase is absorbed by the width, because width candidates come from the gcd closure of the queries' ranges, steps and phases. For example, phases 0 and 5 s with w = 10 s give w = 5 s. Alignment alone does not ensure sufficient retained history; that is $h_v$ (Section 5). -For the proposed periodic consumer path, let the consumer interval and phase be $I$ and $\varphi$. Let the producer interval and phase be $I_s$ and $\varphi_s$, and its tumbling width and grid phase be $w$ and $\varphi_g$. +Let a context κ have interval $I_\kappa$ and phase $\varphi_\kappa$, and let a windowed producer have tumbling width $w$. A consumer running in κ can read the producer when: -| Producer schedule | Required alignment | +| Producer lifetime | Required alignment | |---|---| -| Periodic | $I_s\mid I$ and $\varphi\equiv\varphi_s\pmod{I_s}$ | -| Tumbling grid | $w\mid I$ and $\varphi\equiv\varphi_g\pmod w$ | -| None | Runs in the consuming context; no independent delivery schedule | +| None | Always: runs in κ | +| OneEvaluation | The result belongs to κ | +| AcrossEvaluations, tumbling width $w$ | $w\mid I_\kappa$ and $w\mid\varphi_\kappa$ | +| AcrossEvaluations, no window layout | Not offered yet: it needs its own delivery rule | The formulation also permits a single, non-periodic evaluation. The interval/phase interface below covers only periodic consumers. Rejecting non-periodic consumers is a current scope restriction of that proposed interface, not a mathematical impossibility; supporting them needs explicit evaluation-time membership checks. ### Invalid combinations -Checked constructors reject these combinations under the proposed schedule model: +The checked constructor rejects one combination: | Combination | Reason | |---|---| | IngestionTime with OneEvaluation | No query evaluation lifetime at ingestion time | -| None with an independent schedule | Execution follows the consuming context | -| Materialized lifetime without its required schedule | Delivery times or window grid would be unspecified | -| OneEvaluation with a grid, or windowed AcrossEvaluations with a periodic schedule | Schedule kind does not match this model's lifetime representation | -| Zero interval/width, or phase outside the canonical interval/width | Invalid schedule representation | -| Ingestion-time grid with nonzero phase | Ingestion-time grids are anchored at the epoch: the plan carries no origin | -A tumbling grid's width is the candidate's logical window layout $\omega_v$; it is not repeated in the schedule. Building a `PhysicalCandidate` checks the pair: a grid schedule needs a tumbling layout, and its phase must be below the width. Non-windowed AcrossEvaluations schedules need their own representation before that alternative can be offered. +A tumbling grid's width is the candidate's logical window layout $\omega_v$ and its phase is 0, so no physical field repeats or extends it. ### Decision tree and order of decisions @@ -166,7 +161,7 @@ flowchart TD R2 --> T3["θ: k = …
(prune: pruning cap)"] T2 --> W1["ω: no windows"] - T2 --> W2["ω: tumbling, width w
(prune: w must divide
the group's query range)"] + T2 --> W2["ω: tumbling, width w
from the gcd closure of ranges,
steps and phases (prune: w must
divide the group's query range)"] W2 --> P1["τ = QueryTime"] W2 --> P2["τ = IngestionTime
(prune: no ingestion pipeline)"] @@ -177,22 +172,17 @@ flowchart TD P2 --> L4["ℓ = None"] P2 --> L5["ℓ = AcrossEvaluations"] P2 -. invalid .-> L6["ℓ = OneEvaluation"] - - L2 --> S1["σ: Periodic (I, φ)
from the gcd closure
of the queries' schedules"] - L3 --> S2["σ: grid phase φ mod w"] - L5 --> S3["σ: grid phase 0"] ``` | Level | Decides | Stage | Values offered | Pruned when | |---|---|---|---|---| | 1 | expression $e_v$ | Pass 1 and Pass 2 rules | each way to compute the group | the executor cannot build the family | | 2 | $\boldsymbol\theta_v$ | Pass 1 | the family's predefined list | the pruning cap (Section 4) | -| 3 | $\omega_v$ — Extension | Pass 2 | tumbling widths from the gcd closure of the queries' ranges and steps | the width does not divide the group's query range | +| 3 | $\omega_v$ — Extension | Pass 2 | tumbling widths from the gcd closure of the queries' ranges, steps and phases | the width does not divide the group's query range | | 4 | $\tau_v$ | Stage 2 | QueryTime, IngestionTime | the source has no ingestion pipeline | | 5 | $\ell_v$ | Stage 2 | None, OneEvaluation, AcrossEvaluations | invalid with $\tau_v$; the deployment keeps no query-time results | -| 6 | $\sigma_v$ — Extension | Stage 2 | the gcd closure of the schedules of the queries reaching the group | only when $\ell_v \ne$ None | -The tree does not decide which producer each consumer input reads ($y$), where an unmaterialized candidate runs (its context, Section 5), or how much a materialized candidate retains ($h_v$). Those are Stage 3 choices. +The tree does not decide which producer each consumer input reads ($y$), in which context a candidate with ℓ = None or OneEvaluation runs (Section 5), or how much an AcrossEvaluations candidate retains ($h_v$). Those are Stage 3 choices. **The order does not change the candidate set.** The leaves are the product of the dimensions' values minus invalid combinations and pruned values. Each prune rule reads only the value it removes and values decided above it, so any order that keeps each prune rule below the dimensions it reads yields the same leaves. @@ -200,7 +190,7 @@ The tree does not decide which producer each consumer input reads ($y$), where a 1. **Logical before physical.** Stage 1 hash-conses groups and computes $\alpha^{\log}$ once, before Stage 2 multiplies each logical candidate by its physical alternatives. 2. **Strong prunes early.** Executor support and the pruning cap act before windows and physical properties multiply the tree. -3. **Dependent dimensions adjacent.** $\tau \to \ell \to \sigma$ rejects each invalid combination at the first level that can detect it. +3. **Dependent dimensions adjacent.** $\tau \to \ell$ rejects the invalid combination at the first level that can detect it. 4. **Deployment-wide prunes as early as possible.** Without an ingestion pipeline the IngestionTime subtree is pruned once per candidate; moving $\tau$ above $\omega$ would prune it once per group, with the same leaves. Alternative orders worth measuring: $\tau$ first, when most sources lack an ingestion pipeline; $\omega$ before $\boldsymbol\theta$, when queries' ranges prune many widths; and requirement-driven (top-down) enumeration, which visits fewer nodes but must take the union of all consumers' requirements so that sharing is not lost. @@ -392,40 +382,35 @@ pub enum Lifetime { OneEvaluation(Medium), AcrossEvaluations(Medium), } -pub enum Schedule { - Periodic { interval_ms: u64, phase_ms: u64 }, - /// Phase of the window grid; the width is the window layout's (ω), and - /// each window is readable once it ends. - TumblingWindowGrid { phase_ms: u64 }, -} pub struct PhysicalProperties { timing: ExecutionTiming, lifetime: Lifetime, - schedule: Option, } impl PhysicalProperties { + /// Rejects only (IngestionTime, OneEvaluation). pub fn new( timing: ExecutionTiming, lifetime: Lifetime, - schedule: Option, ) -> Result; pub fn timing(&self) -> ExecutionTiming; pub fn lifetime(&self) -> Lifetime; - pub fn schedule(&self) -> Option<&Schedule>; pub fn is_materialized(&self) -> bool; } +/// When a context runs: φ + I·ℤ. pub struct EvaluationSchedule { interval_ms: u64, phase_ms: u64, } impl PhysicalCandidate { - /// Whether this producer delivers on `times` (Section 2's alignment table). - /// Reads the grid width from the candidate's logical window layout. + /// Whether a consumer running on `times` can read this producer + /// (Section 2's alignment table): w | I and w | φ for a windowed + /// AcrossEvaluations producer, with w read from its window layout; + /// true for ℓ = None and OneEvaluation, which run in that context. pub fn delivers_on(&self, memo: &Memo, times: &EvaluationSchedule) -> bool; } -/// α^phys: timing and lifetime (Section 2's table), and, for a materialized -/// consumer, that the producer delivers on the consumer's own schedule. +/// α^phys: timing and lifetime only (Section 2's table). Alignment is +/// checked per context in Stage 3 with `delivers_on`. pub fn satisfies_physical( memo: &Memo, producer: &PhysicalCandidate, @@ -471,9 +456,9 @@ pub struct PhysicalCandidateKey { } ``` -`PhysicalCandidateId` indexes an emitted candidate. `PhysicalCandidateKey` describes an alternative rejected before it received an ID, allowing diagnostics to explain pruning. The admissible-edge accessor returns edge IDs, not the unfiltered producer set from Section 1. `delivers_on` and `satisfies_physical` take candidates rather than bare properties because a tumbling grid's width lives in the logical window layout. +`PhysicalCandidateId` indexes an emitted candidate. `PhysicalCandidateKey` describes an alternative rejected before it received an ID, allowing diagnostics to explain pruning. The admissible-edge accessor returns edge IDs, not the unfiltered producer set from Section 1. `delivers_on` takes a candidate rather than bare properties because a tumbling grid's width lives in the logical window layout. -Migration targets include replacing `Materialization` and duplicated physical fields with checked properties, and replacing `ReaderConstraint::{IncompatibleOutputs, AllowedOutputs}` with compatibility and schedule checks. `ReaderConstraint::{MinimumQueryReaders, HighestSharedQueryMaterialization}` are deleted too; see Section 5. +Migration targets include replacing `Materialization` and duplicated physical fields with checked properties, and replacing `ReaderConstraint::{IncompatibleOutputs, AllowedOutputs}` with compatibility and alignment checks. `ReaderConstraint::{MinimumQueryReaders, HighestSharedQueryMaterialization}` are deleted too; see Section 5. ### Stage 3: baseline selection and realization @@ -537,7 +522,7 @@ pub struct DagOfSelectedCandidates { } ``` -The realizer uses the selected dependencies to build the executable graph. Export carries each node's timing, lifetime, and supported schedule fields. Under the context extension, multiple selected execution nodes may refer to the same physical candidate; realization must preserve their contexts rather than merge them solely by candidate ID. +The realizer uses the selected dependencies to build the executable graph. Export carries each node's timing and lifetime. Under the context extension, multiple selected execution nodes may refer to the same physical candidate; realization must preserve their contexts rather than merge them solely by candidate ID. ## 4. Accuracy and safe pruning — Extension (#1207) @@ -670,7 +655,7 @@ These extensions go beyond constraints (A)–(I) and the formulation's fixed-cos ### Execution identity -A materialized candidate has its own delivery schedule. An unmaterialized candidate runs inside an execution that needs it. For example, a finalize candidate used by queries A and B may execute in both query contexts even though Stage 2 enumerates it only once. +An AcrossEvaluations candidate has its own computation: each window is built when it ends. A candidate with ℓ = None runs inside an execution that needs it. For example, a finalize candidate used by queries A and B may execute in both query contexts even though Stage 2 enumerates it only once. A OneEvaluation candidate also runs inside one evaluation context κ: queries with the same schedule are evaluated together, and that evaluation's consumers share the result. A **context** identifies a class of executions with a common owner and schedule across the accounting horizon. It is not necessarily one timestamp. @@ -689,16 +674,16 @@ impl Context { } ``` -An unmaterialized chain inherits its consuming context. Encountering a materialized producer switches to that producer's own computation context when traversing its inputs. The context identifies where the operation runs; schedule compatibility decides whether a materialized input can supply it. +A chain of ℓ = None and OneEvaluation candidates inherits its consuming context. Encountering an AcrossEvaluations producer switches to that producer's own computation context when traversing its inputs. The context identifies where the operation runs; `delivers_on` with the context's schedule decides whether a windowed producer can supply it. | Extended variable | Meaning | |---|---| -| $x_v^{\mathrm{mat}}$ | Materialized candidate selected | -| $x_{\kappa,v}$ | Unmaterialized candidate executes in context kappa | +| $x_v^{\mathrm{mat}}$ | AcrossEvaluations candidate selected | +| $x_{\kappa,v}$ | Candidate with ℓ = None or OneEvaluation executes in context kappa | | $y_{\kappa,u,c,i}$ | Input edge selected for a consumer execution in context kappa | -| $h_v$ | Selected retention amount for a materialized producer | +| $h_v$ | Selected retention amount for an AcrossEvaluations producer | -The unindexed edge variable belongs to the baseline only. In the extension, every edge is indexed by its consumer's context, including the dedicated computation context of a materialized consumer. This avoids silently switching between two meanings of `y`. +The unindexed edge variable belongs to the baseline only. In the extension, every edge is indexed by its consumer's context, including the dedicated computation context of an AcrossEvaluations consumer. This avoids silently switching between two meanings of `y`. Activation, input satisfaction, utilization, output selection, and acyclicity must be defined for these execution instances. One binary per context/candidate assumes at most one reusable execution of that candidate within that context. If two uses require independent computations, the context or execution identity must distinguish them. @@ -708,8 +693,8 @@ To avoid confusing per-execution costs with the baseline's horizon-wide cost, us | Symbol | Definition | |---|---| -| $W_v^{\mathrm{mat}}$ | Fixed computation and access cost over the horizon for a materialized candidate's own schedule, excluding variable retention | -| $c_{\kappa,v}$ | Cost of one unmaterialized execution in the specified context | +| $W_v^{\mathrm{mat}}$ | Fixed computation and access cost over the horizon for an AcrossEvaluations candidate, one build per window, excluding variable retention | +| $c_{\kappa,v}$ | Cost of one execution in the specified context of a candidate with ℓ = None or OneEvaluation, including a OneEvaluation result's storage within that evaluation | | $\rho_\kappa$ | Fixed number or expected number of context executions over the common horizon | | $h_v$ | Retention amount in a declared unit, such as retained windows | | $\pi_v$ | Horizon-wide retention cost per unit of retained amount | @@ -719,9 +704,9 @@ A proposed linear objective is: $$ \min\left( -\sum_{v\text{ materialized}}W_v^{\mathrm{mat}}x_v^{\mathrm{mat}} -+\sum_{\kappa,v\text{ unmaterialized}}\rho_\kappa c_{\kappa,v}x_{\kappa,v} -+\sum_{v\text{ materialized}}\pi_v h_v +\sum_{v\text{ across evaluations}}W_v^{\mathrm{mat}}x_v^{\mathrm{mat}} ++\sum_{\kappa,\ v\text{ None or OneEvaluation}}\rho_\kappa c_{\kappa,v}x_{\kappa,v} ++\sum_{v\text{ across evaluations}}\pi_v h_v \right). $$ @@ -735,7 +720,7 @@ The need is fixed per edge from the time horizon of the queries it serves, so co ### No consumer-count policies -Today's `ReaderConstraint` (`crates/types/src/stages/physical/graph.rs`) is deleted with all four variants (decided 2026-11-06). `IncompatibleOutputs` and `AllowedOutputs` are replaced by the schedule check of Section 2. `MinimumQueryReaders` (keep a candidate only if enough queries reach it) and `HighestSharedQueryMaterialization` (materialize where sharing stops) are left to the cost model: materializing for one query, or below the point where sharing stops, costs more and is not chosen. There is no constraint (J). +Today's `ReaderConstraint` (`crates/types/src/stages/physical/graph.rs`) is deleted with all four variants (decided 2026-11-06). `IncompatibleOutputs` and `AllowedOutputs` are replaced by the alignment check of Section 2. `MinimumQueryReaders` (keep a candidate only if enough queries reach it) and `HighestSharedQueryMaterialization` (materialize where sharing stops) are left to the cost model: materializing for one query, or below the point where sharing stops, costs more and is not chosen. There is no constraint (J). ### Out of this design @@ -870,4 +855,4 @@ $$ Here, $P^{\mathrm{log,adm}}_{v,i}$ is the set of logically compatible producers for that input, and a leaf has count one. The bound overcounts shared subgraphs. For a cyclic memo, this recurrence is not well-founded; use cycle-aware enumeration or an explicitly bounded unfolding instead. -For physical inspection, replace logical candidates and compatibility checks with physical candidates and admissible edges. Under the execution-context extension, candidate-level listing alone is not a list of executable plans: instantiate contexts and apply schedule, retention, root-accuracy, and other enabled constraints before comparing results with Stage 3. +For physical inspection, replace logical candidates and compatibility checks with physical candidates and admissible edges. Under the execution-context extension, candidate-level listing alone is not a list of executable plans: instantiate contexts and apply alignment, retention, root-accuracy, and other enabled constraints before comparing results with Stage 3. From 1342661b9affe1a261c16fce932de43bd480ff85 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 15:29:36 +0000 Subject: [PATCH 28/67] docs(concepts): theta listed, floors free; heap 1-2-5 steps; one evaluation context per schedule MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A sketch build's θ must be in its family's list; a requirement's floor is any value of the parameter type (the weakest θ with ε ≤ cap, or a direct need such as heap_size ≥ k). Heap variants take heap sizes from the 1-2-5 series 10 … 10 000; KMV and UnivMon distinct-count lists stop at ε = 0.01. satisfies_physical takes producer and consumer only. Context::Evaluation is keyed by the evaluation schedule. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 27 +++++++++---------- 1 file changed, 13 insertions(+), 14 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 4c737e0f7..f1efd530d 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -84,7 +84,7 @@ This is the single reference table for property dimensions. Stage descriptions b | Dimension | Values and meaning | Code reference or target | |---|---|---| -| Parameters | A predefined configuration list per family: KLL and KMV `k`; DDSketch `alpha`; HLL `precision`; CMS and CountSketch `width` and `depth`; heap variants also `heap_size`; UnivMon its sizing parameters. Exact operators have no sketch parameters. | `SketchParams` in `crates/types/src/ir/schema/sketch.rs`; predefined lists are the #1203 target | +| Parameters | A predefined configuration list per family: KLL and KMV `k`; DDSketch `alpha`; HLL `precision`; CMS and CountSketch `width` and `depth`; heap variants also `heap_size`; UnivMon its sizing parameters. The lists are today's sizing over ε = 0.1 down to 0.001; KMV's and UnivMon's distinct-count entries stop at ε = 0.01. A heap variant combines each width × depth with heap sizes from the 1-2-5 series 10, 20, 50, …, 10 000: a top-k consumer's floor is heap_size ≥ k, and the smallest listed heap that meets it is the next step up, at most 2.5× too large. A sketch build's parameters must be in its family's list; exact operators have no sketch parameters. | `SketchParams` in `crates/types/src/ir/schema/sketch.rs`; predefined lists are the #1203 target | | Window layout — Extension | No layout, or tumbling windows with a positive width. Every grid is anchored at phase 0 (the epoch), so the width alone fixes it. Candidate widths come from the gcd closure of the queries' ranges, steps and phases. Sliding merge/subtraction layouts remain future work. Window count is derived from range and width; retention amount is selected later. | `WindowLayout`; `pass2/memo/window_candidates.rs`; #1160 | | Timing | `IngestionTime` or `QueryTime` | `ExecutionTiming` | | Lifetime | `None`: discard after the consuming execution; `OneEvaluation`: retain until consumers within that evaluation finish; `AcrossEvaluations`: retain across evaluations in the specified period. A non-`None` lifetime means materialized. | `Lifetime`; #1200 / #1214 | @@ -101,7 +101,7 @@ $$ ### Logical requirements -The proposed `LogicalRequirement` stores a **parameter floor**: the weakest configuration in a family's ordered list that satisfies the applicable input error cap. Section 4 defines the cap and distinguishes validity from pruning. It does not allocate a fixed share of the query's error budget to each input. +The proposed `LogicalRequirement` stores a **parameter floor** $\boldsymbol\theta^{\min}$: the weakest parameters whose ε is at most the applicable input error cap, or a direct need such as heap_size ≥ k for a top-k. The floor is any value of the family's parameter type and need not be in the family's list; a listed producer satisfies it when $\boldsymbol\theta_u \succeq \boldsymbol\theta^{\min}$. Section 4 defines the cap and distinguishes validity from pruning. It does not allocate a fixed share of the query's error budget to each input. For KLL, a larger `k` satisfies a smaller minimum `k`. For DDSketch, a smaller `alpha` can be stronger. Multi-parameter families require the family's comparison rule, including parameters needed for failure guarantees. A single floor is valid only if the accepted configurations form an upward-closed set with a representable least element. Otherwise, compatibility must use an explicit admissible set or predicate; it must not invent a unique minimum. @@ -292,6 +292,9 @@ pub enum WindowLayout { } impl LogicalProperties { + /// Rejects parameters missing on a sketch build, present on an exact + /// operator, of another family or not in the family's list, and a window + /// of width 0 or on an operator that cannot be windowed. pub fn new( expression: &CandidateExpression, parameters: Option, @@ -309,9 +312,8 @@ pub struct LogicalRequirement { } impl LogicalRequirement { - pub fn new( - parameter_floor: Option, - ) -> Result; + /// Any floor of the family's parameter type, listed or not. + pub fn new(parameter_floor: Option) -> Self; pub fn parameter_floor(&self) -> Option<&SketchParams>; } @@ -411,11 +413,7 @@ impl PhysicalCandidate { } /// α^phys: timing and lifetime only (Section 2's table). Alignment is /// checked per context in Stage 3 with `delivers_on`. -pub fn satisfies_physical( - memo: &Memo, - producer: &PhysicalCandidate, - consumer: &PhysicalCandidate, -) -> bool; +pub fn satisfies_physical(producer: &PhysicalCandidate, consumer: &PhysicalCandidate) -> bool; pub struct PhysicalCandidateId(pub usize); pub struct PhysicalEdgeId(pub usize); @@ -655,21 +653,22 @@ These extensions go beyond constraints (A)–(I) and the formulation's fixed-cos ### Execution identity -An AcrossEvaluations candidate has its own computation: each window is built when it ends. A candidate with ℓ = None runs inside an execution that needs it. For example, a finalize candidate used by queries A and B may execute in both query contexts even though Stage 2 enumerates it only once. A OneEvaluation candidate also runs inside one evaluation context κ: queries with the same schedule are evaluated together, and that evaluation's consumers share the result. +An AcrossEvaluations candidate has its own computation: each window is built when it ends. A candidate with ℓ = None runs inside an execution that needs it. For example, a finalize candidate used by queries A (every 10 s) and C (every 15 s) may execute in both evaluation contexts even though Stage 2 enumerates it only once. A OneEvaluation candidate also runs inside one evaluation context κ, and that evaluation's consumers share the result. -A **context** identifies a class of executions with a common owner and schedule across the accounting horizon. It is not necessarily one timestamp. +A **context** identifies a class of executions with a common owner and schedule across the accounting horizon. It is not necessarily one timestamp. There is one evaluation context per evaluation schedule (interval, phase): all queries with that schedule are evaluated together and share its OneEvaluation results. Queries A and B of the running example, both every 10 s at phase 0, form one context. A non-periodic (one-off) query has no evaluation context yet: it is out of scope, as in Section 2. ```rust pub struct ContextId(pub usize); pub enum Context { - QueryEvaluation(QueryId), + Evaluation(EvaluationSchedule), IngestionPipeline(RefreshWindowId), Materialized(PhysicalCandidateId), } impl Context { - // Periodic schedule when supported; None is not an implicit compatibility pass. + // An evaluation context's schedule; None for the other contexts, which + // is not an implicit compatibility pass. pub fn evaluation_schedule(&self) -> Option; } ``` From 9f56b58ea4ef4adb5ee04b17f6658e96330353b8 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 15:37:44 +0000 Subject: [PATCH 29/67] docs(concepts): rename the interfaces as agreed with the reviewer LogicalPropertiesError, PhysicalPropertiesError, LogicalRequirements (parameter_floor_for_required_accuracy), producer_satisfies_logical_requirements, producer_satisfies_physical_requirements, StorageMedium, ExecutionContext and ExecutionContextId; PhysicalCandidate / PhysicalCandidateKey fields logical_candidate_id and physical_properties; edge fields producer_candidate_id and consumer_candidate_id; PhysicalASAPMemoCandidates fields logical_candidates and physical_candidates. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 65 ++++++++++--------- 1 file changed, 34 insertions(+), 31 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index f1efd530d..af6c6c38c 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -88,7 +88,7 @@ This is the single reference table for property dimensions. Stage descriptions b | Window layout — Extension | No layout, or tumbling windows with a positive width. Every grid is anchored at phase 0 (the epoch), so the width alone fixes it. Candidate widths come from the gcd closure of the queries' ranges, steps and phases. Sliding merge/subtraction layouts remain future work. Window count is derived from range and width; retention amount is selected later. | `WindowLayout`; `pass2/memo/window_candidates.rs`; #1160 | | Timing | `IngestionTime` or `QueryTime` | `ExecutionTiming` | | Lifetime | `None`: discard after the consuming execution; `OneEvaluation`: retain until consumers within that evaluation finish; `AcrossEvaluations`: retain across evaluations in the specified period. A non-`None` lifetime means materialized. | `Lifetime`; #1200 / #1214 | -| Medium — Extension | `Memory` or `Disk`, carried by a materialized lifetime | `Medium`; #1214 | +| Storage medium — Extension | `Memory` or `Disk`, carried by a materialized lifetime | `StorageMedium`; #1214 | Error guarantees are derived from parameters and the statistic being read. They include an error metric, a bound, and a failure probability; exact computation introduces no approximation error. The same state may support different readouts, so its guarantee must be interpreted for the relevant statistic. @@ -101,7 +101,7 @@ $$ ### Logical requirements -The proposed `LogicalRequirement` stores a **parameter floor** $\boldsymbol\theta^{\min}$: the weakest parameters whose ε is at most the applicable input error cap, or a direct need such as heap_size ≥ k for a top-k. The floor is any value of the family's parameter type and need not be in the family's list; a listed producer satisfies it when $\boldsymbol\theta_u \succeq \boldsymbol\theta^{\min}$. Section 4 defines the cap and distinguishes validity from pruning. It does not allocate a fixed share of the query's error budget to each input. +The proposed `LogicalRequirements` stores a **parameter floor** $\boldsymbol\theta^{\min}$: the weakest parameters whose ε is at most the applicable input error cap, or a direct need such as heap_size ≥ k for a top-k. The floor is any value of the family's parameter type and need not be in the family's list; a listed producer satisfies it when $\boldsymbol\theta_u \succeq \boldsymbol\theta^{\min}$. Section 4 defines the cap and distinguishes validity from pruning. It does not allocate a fixed share of the query's error budget to each input. For KLL, a larger `k` satisfies a smaller minimum `k`. For DDSketch, a smaller `alpha` can be stronger. Multi-parameter families require the family's comparison rule, including parameters needed for failure guarantees. A single floor is valid only if the accepted configurations form an upward-closed set with a representable least element. Otherwise, compatibility must use an explicit admissible set or predicate; it must not invent a unique minimum. @@ -276,7 +276,7 @@ pub struct CandidateExpression { pub struct CandidateForMemoGroup { pub expression: CandidateExpression, pub logical_properties: LogicalProperties, - pub input_requirements: Vec, + pub input_requirements: Vec, pub fused_groups: Vec, // Diagnostic provenance; not a planning decision. pub origin: OriginOfCandidateForMemoGroup, @@ -299,7 +299,7 @@ impl LogicalProperties { expression: &CandidateExpression, parameters: Option, window_layout: Option, - ) -> Result; + ) -> Result; pub fn parameters(&self) -> Option<&SketchParams>; pub fn window_layout(&self) -> Option<&WindowLayout>; @@ -307,20 +307,20 @@ impl LogicalProperties { pub fn parameter_list(family: SketchAlgorithm) -> &'static [SketchParams]; -pub struct LogicalRequirement { - parameter_floor: Option, +pub struct LogicalRequirements { + parameter_floor_for_required_accuracy: Option, } -impl LogicalRequirement { +impl LogicalRequirements { /// Any floor of the family's parameter type, listed or not. - pub fn new(parameter_floor: Option) -> Self; + pub fn new(parameter_floor_for_required_accuracy: Option) -> Self; - pub fn parameter_floor(&self) -> Option<&SketchParams>; + pub fn parameter_floor_for_required_accuracy(&self) -> Option<&SketchParams>; } -pub fn satisfies_logical( +pub fn producer_satisfies_logical_requirements( producer: &LogicalProperties, - requirement: &LogicalRequirement, + requirement: &LogicalRequirements, ) -> bool; ``` @@ -351,8 +351,8 @@ After the rules, Pass 2 builds the logical edges and applies the pruning caps of fn apply(memo: &mut Memo) -> Result<(), Pass2RuleError>; pub struct LogicalEdge { - pub producer: CandidateForMemoGroupId, - pub consumer: CandidateForMemoGroupId, + pub producer_candidate_id: CandidateForMemoGroupId, + pub consumer_candidate_id: CandidateForMemoGroupId, pub input_index_of_consumer: usize, pub satisfies_logical_requirement: bool, } @@ -374,15 +374,15 @@ pub enum ExecutionTiming { QueryTime, } -pub enum Medium { +pub enum StorageMedium { Memory, Disk, } pub enum Lifetime { None, - OneEvaluation(Medium), - AcrossEvaluations(Medium), + OneEvaluation(StorageMedium), + AcrossEvaluations(StorageMedium), } pub struct PhysicalProperties { timing: ExecutionTiming, @@ -393,7 +393,7 @@ impl PhysicalProperties { pub fn new( timing: ExecutionTiming, lifetime: Lifetime, - ) -> Result; + ) -> Result; pub fn timing(&self) -> ExecutionTiming; pub fn lifetime(&self) -> Lifetime; pub fn is_materialized(&self) -> bool; @@ -413,24 +413,27 @@ impl PhysicalCandidate { } /// α^phys: timing and lifetime only (Section 2's table). Alignment is /// checked per context in Stage 3 with `delivers_on`. -pub fn satisfies_physical(producer: &PhysicalCandidate, consumer: &PhysicalCandidate) -> bool; +pub fn producer_satisfies_physical_requirements( + producer: &PhysicalCandidate, + consumer: &PhysicalCandidate, +) -> bool; pub struct PhysicalCandidateId(pub usize); pub struct PhysicalEdgeId(pub usize); pub struct PhysicalCandidate { - pub logical: CandidateForMemoGroupId, - pub properties: PhysicalProperties, + pub logical_candidate_id: CandidateForMemoGroupId, + pub physical_properties: PhysicalProperties, } pub struct PhysicalEdge { - pub producer: PhysicalCandidateId, - pub consumer: PhysicalCandidateId, + pub producer_candidate_id: PhysicalCandidateId, + pub consumer_candidate_id: PhysicalCandidateId, pub input_index_of_consumer: usize, pub satisfies_logical_requirement: bool, pub satisfies_physical_requirement: bool, } pub struct PhysicalASAPMemoCandidates { - pub logical: LogicalASAPMemoCandidates, - pub candidates: Vec, + pub logical_candidates: LogicalASAPMemoCandidates, + pub physical_candidates: Vec, pub query_roots: Vec>, pub pruned_candidates: Vec>, } @@ -449,8 +452,8 @@ impl PhysicalASAPMemoCandidates { // Identifies a rejected alternative by content: it has no allocated candidate ID. pub struct PhysicalCandidateKey { - pub logical: CandidateForMemoGroupId, - pub properties: PhysicalProperties, + pub logical_candidate_id: CandidateForMemoGroupId, + pub physical_properties: PhysicalProperties, } ``` @@ -580,7 +583,7 @@ pub struct ErrorPropagation { pub validity_cap_per_input: Vec>, } -impl LogicalRequirement { +impl LogicalRequirements { pub fn error_propagation( consumer: &Operator, ) -> ErrorPropagation; @@ -658,15 +661,15 @@ An AcrossEvaluations candidate has its own computation: each window is built whe A **context** identifies a class of executions with a common owner and schedule across the accounting horizon. It is not necessarily one timestamp. There is one evaluation context per evaluation schedule (interval, phase): all queries with that schedule are evaluated together and share its OneEvaluation results. Queries A and B of the running example, both every 10 s at phase 0, form one context. A non-periodic (one-off) query has no evaluation context yet: it is out of scope, as in Section 2. ```rust -pub struct ContextId(pub usize); +pub struct ExecutionContextId(pub usize); -pub enum Context { +pub enum ExecutionContext { Evaluation(EvaluationSchedule), IngestionPipeline(RefreshWindowId), Materialized(PhysicalCandidateId), } -impl Context { +impl ExecutionContext { // An evaluation context's schedule; None for the other contexts, which // is not an implicit compatibility pass. pub fn evaluation_schedule(&self) -> Option; @@ -771,7 +774,7 @@ fn expand( for choice in cartesian_product(inputs.iter().enumerate().map(|(i, &group)| { candidates_of(memo, group) .filter(|&u| { - satisfies_logical( + producer_satisfies_logical_requirements( &memo.candidate(u).logical_properties, &candidate.input_requirements[i], ) From 9b82b79b7411ad2c034d955be5a5e2117b9c698e Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 15:38:29 +0000 Subject: [PATCH 30/67] docs(concepts): note the empty parameter lists of unsized families (#1228) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../design_docs/concepts/memo-groups-properties-requirements.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index af6c6c38c..d647f4bb1 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -84,7 +84,7 @@ This is the single reference table for property dimensions. Stage descriptions b | Dimension | Values and meaning | Code reference or target | |---|---|---| -| Parameters | A predefined configuration list per family: KLL and KMV `k`; DDSketch `alpha`; HLL `precision`; CMS and CountSketch `width` and `depth`; heap variants also `heap_size`; UnivMon its sizing parameters. The lists are today's sizing over ε = 0.1 down to 0.001; KMV's and UnivMon's distinct-count entries stop at ε = 0.01. A heap variant combines each width × depth with heap sizes from the 1-2-5 series 10, 20, 50, …, 10 000: a top-k consumer's floor is heap_size ≥ k, and the smallest listed heap that meets it is the next step up, at most 2.5× too large. A sketch build's parameters must be in its family's list; exact operators have no sketch parameters. | `SketchParams` in `crates/types/src/ir/schema/sketch.rs`; predefined lists are the #1203 target | +| Parameters | A predefined configuration list per family: KLL and KMV `k`; DDSketch `alpha`; HLL `precision`; CMS and CountSketch `width` and `depth`; heap variants also `heap_size`; UnivMon its sizing parameters. The lists are today's sizing over ε = 0.1 down to 0.001; KMV's and UnivMon's distinct-count entries stop at ε = 0.01. A heap variant combines each width × depth with heap sizes from the 1-2-5 series 10, 20, 50, …, 10 000: a top-k consumer's floor is heap_size ≥ k, and the smallest listed heap that meets it is the next step up, at most 2.5× too large. A sketch build's parameters must be in its family's list; exact operators have no sketch parameters. Hydra and the other families Pass 1 does not size have empty `parameter_list`s, so no build of theirs is a valid candidate until their error models exist (#1228). | `SketchParams` in `crates/types/src/ir/schema/sketch.rs`; predefined lists are the #1203 target | | Window layout — Extension | No layout, or tumbling windows with a positive width. Every grid is anchored at phase 0 (the epoch), so the width alone fixes it. Candidate widths come from the gcd closure of the queries' ranges, steps and phases. Sliding merge/subtraction layouts remain future work. Window count is derived from range and width; retention amount is selected later. | `WindowLayout`; `pass2/memo/window_candidates.rs`; #1160 | | Timing | `IngestionTime` or `QueryTime` | `ExecutionTiming` | | Lifetime | `None`: discard after the consuming execution; `OneEvaluation`: retain until consumers within that evaluation finish; `AcrossEvaluations`: retain across evaluations in the specified period. A non-`None` lifetime means materialized. | `Lifetime`; #1200 / #1214 | From 05bc1ea8d350391680881364bebffc01b8424a64 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 15:48:10 +0000 Subject: [PATCH 31/67] docs(concepts): execution contexts reuse the workload's recurrence; one-off queries supported MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An evaluation context is Evaluation { recurrence, range_evaluation }, deduplicated by equality. delivers_on takes the context: w divides the evaluation grid of FixedIntervalAt (interval, step, phase), every Scheduled time, or a OneTime query's time; never when the times are not known. ρ_κ reuses execution_demand's evaluation counting. EvaluationSchedule is deleted. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 48 +++++++++---------- 1 file changed, 22 insertions(+), 26 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index d647f4bb1..a30f8348c 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -119,21 +119,24 @@ Physical requirements are derived from the consumer's timing and execution conte QT means QueryTime; IT means IngestionTime. Alignment, data coverage, and retention checks still apply to allowed combinations. **Alignment.** There is no physical schedule property. When a result can be read follows from its lifetime and window layout: -- **None** and **OneEvaluation**: the result is computed inside one evaluation context κ (Section 5) and read there. A OneEvaluation result belongs to one context: queries with the same schedule are evaluated together, and that evaluation's consumers share the result. When it runs is the context's schedule, and Stage 3 chooses it through $x_{\kappa,v}$, just as for ℓ = None. Sharing across differently scheduled queries through a gcd schedule is not offered: it is rarely beneficial and wastes runs. +- **None** and **OneEvaluation**: the result is computed inside one evaluation context κ (Section 5) and read there. A OneEvaluation result belongs to one context: queries with equal recurrence and range evaluation are evaluated together, and that evaluation's consumers share the result. When it runs is the context's schedule, and Stage 3 chooses it through $x_{\kappa,v}$, just as for ℓ = None. Sharing across differently scheduled queries through a gcd schedule is not offered: it is rarely beneficial and wastes runs. - **Windowed AcrossEvaluations**: the input is cut into tumbling windows [k·w, (k+1)·w), with w from the window layout ω and the grid anchored at phase 0 (the epoch). Each window is built when it ends and can be read from then on. A consumer can read a range that ends on a window boundary. With w = 10 s, a `[1m]` query evaluated every 30 s at phase 0 merges the newest 6 windows; one evaluated at phase 15 s cannot, because its range ends inside a window; with that query in the workload, the gcd closure also offers w = 5 s, which both can read. A grid phase is not needed: a phase is absorbed by the width, because width candidates come from the gcd closure of the queries' ranges, steps and phases. For example, phases 0 and 5 s with w = 10 s give w = 5 s. Alignment alone does not ensure sufficient retained history; that is $h_v$ (Section 5). -Let a context κ have interval $I_\kappa$ and phase $\varphi_\kappa$, and let a windowed producer have tumbling width $w$. A consumer running in κ can read the producer when: +An evaluation context κ is described by the workload's own recurrence and range evaluation (Section 5). Let a windowed producer have tumbling width $w$. A consumer running in κ can read the producer when $w$ divides every evaluation time of κ: -| Producer lifetime | Required alignment | +| Producer lifetime, or κ's recurrence for a windowed AcrossEvaluations producer | Required alignment | |---|---| | None | Always: runs in κ | | OneEvaluation | The result belongs to κ | -| AcrossEvaluations, tumbling width $w$ | $w\mid I_\kappa$ and $w\mid\varphi_\kappa$ | +| `FixedIntervalAt { interval, evaluation_phase }`, range step $s$ | $w \mid \gcd(\text{interval}, s, \text{phase})$, the evaluation grid; $s$ counts only for a range evaluation of more than one step | +| `Scheduled(times)` | $w$ divides every time (and $s$) | +| `OneTime { execute_at: Some(t) }` | $w \mid t$ (and $s$) | +| `FixedInterval` without a phase, `OneTime { execute_at: None }`, `EstimatedRate`, `Unknown` | Never: the evaluation times are not known | | AcrossEvaluations, no window layout | Not offered yet: it needs its own delivery rule | -The formulation also permits a single, non-periodic evaluation. The interval/phase interface below covers only periodic consumers. Rejecting non-periodic consumers is a current scope restriction of that proposed interface, not a mathematical impossibility; supporting them needs explicit evaluation-time membership checks. +The evaluation grid is `QueryRecurrence::evaluation_grid_ms(range_evaluation)` in `crates/types/src/workload/mod.rs`, which the workload entries also use. One-off queries are supported: each is a context of its own recurrence, run `invocations` times. ### Invalid combinations @@ -186,7 +189,7 @@ The tree does not decide which producer each consumer input reads ($y$), in whic **The order does not change the candidate set.** The leaves are the product of the dimensions' values minus invalid combinations and pruned values. Each prune rule reads only the value it removes and values decided above it, so any order that keeps each prune rule below the dimensions it reads yields the same leaves. -**The order changes how many nodes enumeration visits:** $\sum_d \prod_{d' \le d} \lvert \text{values kept at } d' \rvert$. Dimensions that prune much go early; dimensions that multiply without pruning go late. The order above follows four rules: +**The order changes how many nodes enumeration visits:** $\sum_d \prod_{d' \le d} \lvert \text{values left at } d' \rvert$. Dimensions that prune much go early; dimensions that multiply without pruning go late. The order above follows four rules: 1. **Logical before physical.** Stage 1 hash-conses groups and computes $\alpha^{\log}$ once, before Stage 2 multiplies each logical candidate by its physical alternatives. 2. **Strong prunes early.** Executor support and the pruning cap act before windows and physical properties multiply the tree. @@ -399,17 +402,13 @@ impl PhysicalProperties { pub fn is_materialized(&self) -> bool; } -/// When a context runs: φ + I·ℤ. -pub struct EvaluationSchedule { - interval_ms: u64, - phase_ms: u64, -} impl PhysicalCandidate { - /// Whether a consumer running on `times` can read this producer - /// (Section 2's alignment table): w | I and w | φ for a windowed - /// AcrossEvaluations producer, with w read from its window layout; - /// true for ℓ = None and OneEvaluation, which run in that context. - pub fn delivers_on(&self, memo: &Memo, times: &EvaluationSchedule) -> bool; + /// Whether a consumer running in `context` can read this producer + /// (Section 2's alignment table): for a windowed AcrossEvaluations + /// producer, w (read from its window layout) divides every evaluation + /// time of the context; true for ℓ = None, and for OneEvaluation in the + /// same context. + pub fn delivers_on(&self, memo: &Memo, context: &ExecutionContext) -> bool; } /// α^phys: timing and lifetime only (Section 2's table). Alignment is /// checked per context in Stage 3 with `delivers_on`. @@ -658,25 +657,22 @@ These extensions go beyond constraints (A)–(I) and the formulation's fixed-cos An AcrossEvaluations candidate has its own computation: each window is built when it ends. A candidate with ℓ = None runs inside an execution that needs it. For example, a finalize candidate used by queries A (every 10 s) and C (every 15 s) may execute in both evaluation contexts even though Stage 2 enumerates it only once. A OneEvaluation candidate also runs inside one evaluation context κ, and that evaluation's consumers share the result. -A **context** identifies a class of executions with a common owner and schedule across the accounting horizon. It is not necessarily one timestamp. There is one evaluation context per evaluation schedule (interval, phase): all queries with that schedule are evaluated together and share its OneEvaluation results. Queries A and B of the running example, both every 10 s at phase 0, form one context. A non-periodic (one-off) query has no evaluation context yet: it is out of scope, as in Section 2. +A **context** identifies a class of executions with a common owner and schedule across the accounting horizon. It is not necessarily one timestamp. An evaluation context reuses the workload's own description of when queries run, their `QueryRecurrence` and `RangeEvaluation`: queries with equal ones are evaluated together and share its OneEvaluation results. Queries A and B of the running example, both every 10 s at phase 0, form one context; a one-off query forms the context of its `OneTime` recurrence. `QueryRecurrence` holds an `f64` (`EstimatedRate`) and cannot be hashed, so contexts are deduplicated by equality and `ExecutionContextId` indexes the list of distinct contexts. ```rust pub struct ExecutionContextId(pub usize); pub enum ExecutionContext { - Evaluation(EvaluationSchedule), + Evaluation { + recurrence: QueryRecurrence, + range_evaluation: Option, + }, IngestionPipeline(RefreshWindowId), Materialized(PhysicalCandidateId), } - -impl ExecutionContext { - // An evaluation context's schedule; None for the other contexts, which - // is not an implicit compatibility pass. - pub fn evaluation_schedule(&self) -> Option; -} ``` -A chain of ℓ = None and OneEvaluation candidates inherits its consuming context. Encountering an AcrossEvaluations producer switches to that producer's own computation context when traversing its inputs. The context identifies where the operation runs; `delivers_on` with the context's schedule decides whether a windowed producer can supply it. +A chain of ℓ = None and OneEvaluation candidates inherits its consuming context. Encountering an AcrossEvaluations producer switches to that producer's own computation context when traversing its inputs. The context identifies where the operation runs; `delivers_on` with the context decides whether a windowed producer can supply it. | Extended variable | Meaning | |---|---| @@ -697,7 +693,7 @@ To avoid confusing per-execution costs with the baseline's horizon-wide cost, us |---|---| | $W_v^{\mathrm{mat}}$ | Fixed computation and access cost over the horizon for an AcrossEvaluations candidate, one build per window, excluding variable retention | | $c_{\kappa,v}$ | Cost of one execution in the specified context of a candidate with ℓ = None or OneEvaluation, including a OneEvaluation result's storage within that evaluation | -| $\rho_\kappa$ | Fixed number or expected number of context executions over the common horizon | +| $\rho_\kappa$ | Fixed number or expected number of context executions over the common horizon, from the existing evaluation counting in `crates/cost/src/execution_demand.rs` (a one-off query's is its `invocations`) | | $h_v$ | Retention amount in a declared unit, such as retained windows | | $\pi_v$ | Horizon-wide retention cost per unit of retained amount | | $\operatorname{need}_{\kappa,v,c,i}$ | Retention amount producer v needs to serve this consumer input in this context: the time horizon the queries served by that context read, e.g. a 5m range over 10 s tumbling windows needs 30 windows. Query latency is ignored for now (decided 2026-11-06) | From 2f984cc882a83ce0db8e069ca9d099af65d93c2d Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 18:53:41 +0000 Subject: [PATCH 32/67] docs(concepts): Memo methods return Result; CandidateError find_or_add_group returns the node rules' SchemaDerivationError; add_candidate checks the candidate's expression by the same rules and requires the group's output schema and kind (CandidateError). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 21 +++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index a30f8348c..d6ece4103 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -258,20 +258,37 @@ pub struct CandidateForMemoGroupId { } impl Memo { + /// The group of `expression` (operator, arguments and input groups), + /// added if new. Fails when the expression breaks the node rules + /// (schema derivation, input validation, merge check). pub fn find_or_add_group( &mut self, expression: Operator, - ) -> MemoGroupId; + ) -> Result; + /// Adds `candidate` to `group`, or returns the id of the candidate there + /// with the same expression and logical properties. The candidate's + /// expression is checked by the same node rules as a group's, and its + /// output schema and kind must equal the group's; a rejected candidate + /// leaves the memo unchanged. pub fn add_candidate( &mut self, group: MemoGroupId, candidate: CandidateForMemoGroup, - ) -> CandidateForMemoGroupId; + ) -> Result; pub fn candidate(&self, id: CandidateForMemoGroupId) -> &CandidateForMemoGroup; } +pub enum CandidateError { + /// The candidate's expression fails the node rules (schema derivation, + /// input validation, merge check). + Schema(SchemaDerivationError), + /// Its output schema or output kind differs from the group's: it does + /// not compute the group's result. + OutputDiffersFromGroup { group: MemoGroupId }, +} + pub struct CandidateExpression { pub operator: Operator, } From 836885ce9297c98e472ad8baf30949b509e12ec9 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 19:17:19 +0000 Subject: [PATCH 33/67] docs(concepts): a memo group is its expression; private Memo groups MemoGroup holds expression, schema and result kind (no logical node or child list); Memo's groups are private behind groups(), added only through find_or_add_group (hash-consing on the expression), and Stage 0 adds each query with add_query. Appendix A reads memo.groups(). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 30 ++++++++++++++----- 1 file changed, 23 insertions(+), 7 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index d6ece4103..f68a03a07 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -215,14 +215,18 @@ Rules add groups through `find_or_add_group` and candidates through `add_candida ### Stage 0: groups -`hash_consed` in `crates/logical-asap-optimization/src/stage0.rs` identifies shared logical expressions and query roots. Stage 0 and Stage 1 share one type, `Memo`: Stage 0 returns it with every group's candidate list empty, and Stage 1 fills the lists. +`memo_of_queries` in `crates/stage0-memo` builds the groups bottom-up from the queries' nodes: each node's operator, over its children's groups, goes through `Memo::find_or_add_group`, so identical sub-expressions are one group; each query's root is then added with `Memo::add_query`. Stage 0 and Stage 1 share one type, `Memo`: Stage 0 returns it with every group's candidate list empty, and Stage 1 fills the lists. ```rust pub struct MemoGroupId(pub u32); pub struct MemoGroup { - pub logical_node: Rc, - pub child_groups: Vec, + /// What it computes; its children are the input groups. + pub expression: Operator, + /// Derived once, when the group is added. + pub schema: Schema, + /// Derived once, when the group is added. + pub result_kind: OperatorResultKind, pub parents: Vec, pub queries: Vec, // Empty after Stage 0; populated by Stage 1. @@ -230,14 +234,26 @@ pub struct MemoGroup { } pub struct Memo { - pub groups: Vec, + // Private: groups are added only through `find_or_add_group`, so the + // hash-consing index covers every group. + groups: Vec, pub query_roots: Vec>, + // Private: the groups `find_or_add_group` may return, by expression hash. + groups_by_expression_hash: HashMap>, +} + +impl Memo { + pub fn groups(&self) -> &[MemoGroup]; + /// Adds a query with this root; its groups and those below list it. + pub fn add_query(&mut self, root: QueryRoot) -> usize; } ``` -`OperatorNode` recursively contains chosen child nodes and therefore describes a fixed plan fragment. `Operator` contains input group IDs and leaves their implementations open. A candidate needs its own expression because a group may offer several ways to compute the same result; it cannot reuse the group's representative `logical_node` as every candidate's expression. +A group is its expression: one operator, with its arguments, over input groups. Hash-consing compares and hashes the expression directly, so two expressions are one group when their operators and arguments are equal and their input groups are the same. θ is not an operator argument, so builds that differ only in θ share one group. An expression whose copies may return different rows (an unordered `LIMIT`) always gets a new group. + +The group's schema and result kind come from the node rules (schema derivation, input validation, merge check) applied to its expression over its input groups. A candidate's expression is checked by the same rules and must produce the group's schema and result kind. `OperatorNode` recursively contains chosen child nodes and therefore describes a fixed plan fragment; `Operator` contains input group IDs and leaves their implementations open. A candidate needs its own expression because a group may offer several ways to compute the same result. -Two simplifications are possible but not adopted: storing the group as one `Operator` instead of its node and child list, and removing the one-field `CandidateExpression` wrapper. +Removing the one-field `CandidateExpression` wrapper is possible but not adopted. ### Stage 1, Pass 1: initial candidates @@ -815,7 +831,7 @@ fn candidates_of( memo: &Memo, group: MemoGroupId, ) -> impl Iterator + '_ { - (0..memo.groups[group.0 as usize].candidates.len()) + (0..memo.groups()[group.0 as usize].candidates.len()) .map(move |index_in_group| CandidateForMemoGroupId { group, index_in_group }) } From 7a87ff749b589b435a7a19fdd99970fd1c88450c Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 19:40:01 +0000 Subject: [PATCH 34/67] docs(concepts): GroupProperties derived from input groups; SummaryDefinition over group ids A memo group is its expression and its GroupProperties (schema, result kind, summary coverage, shareable_when_equal), derived once from its input groups' expressions and properties by the node rules, never from node trees. A group's summary coverage has a SummaryDefinition: the residual links over group ids. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 41 ++++++++++++++++--- 1 file changed, 35 insertions(+), 6 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index f68a03a07..4fea1d307 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -221,18 +221,38 @@ Rules add groups through `find_or_add_group` and candidates through `add_candida pub struct MemoGroupId(pub u32); pub struct MemoGroup { - /// What it computes; its children are the input groups. + /// What it computes: one operator with its arguments. Each `MemoGroupId` + /// in it is an input group, never this group's own id. pub expression: Operator, - /// Derived once, when the group is added. - pub schema: Schema, - /// Derived once, when the group is added. - pub result_kind: OperatorResultKind, + /// Derived once when the group is created, from its input groups' + /// `expression` and `group_properties` (optd/TiDB-style); never changes. + pub group_properties: GroupProperties, pub parents: Vec, pub queries: Vec, // Empty after Stage 0; populated by Stage 1. pub candidates: Vec, } +pub struct GroupProperties { + pub schema: Schema, + pub result_kind: OperatorResultKind, + /// What a summary-state group covers; `None` for any other group. + pub summary_coverage: Option>, + /// Whether two equal expressions return the same rows for the same + /// input rows, so they may be one group (not an unordered `LIMIT`). + pub shareable_when_equal: bool, +} + +/// The `SummaryAgg` with its lifted selection removed, written over group +/// ids: the residual links from `reads` up to the `SummaryAgg`. +pub struct SummaryDefinition { + /// The group below the residual links; `None` when the bottom link is + /// the `Scan`, rebuilt without the lifted predicates. + pub reads: Option, + /// Bottom-up, each link without its input; the last is the `SummaryAgg`. + pub links: Vec>, +} + pub struct Memo { // Private: groups are added only through `find_or_add_group`, so the // hash-consing index covers every group. @@ -249,9 +269,18 @@ impl Memo { } ``` +`SummaryCoverage` is the coverage type of today's IR with its definition as a parameter: a group's coverage has a `SummaryDefinition`, an operator node's (executor, export) its residual node tree. + A group is its expression: one operator, with its arguments, over input groups. Hash-consing compares and hashes the expression directly, so two expressions are one group when their operators and arguments are equal and their input groups are the same. θ is not an operator argument, so builds that differ only in θ share one group. An expression whose copies may return different rows (an unordered `LIMIT`) always gets a new group. -The group's schema and result kind come from the node rules (schema derivation, input validation, merge check) applied to its expression over its input groups. A candidate's expression is checked by the same rules and must produce the group's schema and result kind. `OperatorNode` recursively contains chosen child nodes and therefore describes a fixed plan fragment; `Operator` contains input group IDs and leaves their implementations open. A candidate needs its own expression because a group may offer several ways to compute the same result. +Everything that was computed on operator-node trees is computed on groups. `GroupProperties` is derived once, when the group is created, by the same rules that check operator nodes (schema derivation, input validation, merge check), reading only the input groups' `expression` and `group_properties` from the memo, never a node tree: + +- `schema` and `result_kind` from the operator and its input groups' schemas and result kinds; a scalar subquery reads the referenced group's schema. +- `summary_coverage`, for a summary state: walking down through the input groups' expressions (Filter, Project, TimeRange, TimeShift, down to the Scan), lifting their predicates into the selection; the rest is the `SummaryDefinition`. A merge's coverage comes from its inputs' coverage; a merge is valid only over inputs with the same definition and disjoint selections. +- The checks that read an input's operator (a merge of the newest windows reads a window chain; a selected build) read the input group's `expression`. +- `shareable_when_equal`: a `LIMIT` is shareable over a `Sort` with the same partitions (the input group's `expression`); a `Dedup` on every column (the input group's schema). + +A candidate's expression goes through the same derivation and must produce the group's schema and result kind. The executor and the export keep their node-tree summary coverage (`definition_id`) until Stage 3's export builds node trees from the memo. `OperatorNode` recursively contains chosen child nodes and therefore describes a fixed plan fragment; `Operator` contains input group IDs and leaves their implementations open. A candidate needs its own expression because a group may offer several ways to compute the same result. Removing the one-field `CandidateExpression` wrapper is possible but not adopted. From 4c44e01715d8690c5522d7c15865f46fe7249565 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 21:09:42 +0000 Subject: [PATCH 35/67] docs(concepts): SummaryStateDefinition, OperatorInput and MemoGroupAsInput; coverage only in the memo Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 73 +++++++++++++++---- 1 file changed, 60 insertions(+), 13 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 4fea1d307..ebdabda09 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -237,20 +237,38 @@ pub struct GroupProperties { pub schema: Schema, pub result_kind: OperatorResultKind, /// What a summary-state group covers; `None` for any other group. - pub summary_coverage: Option>, + pub summary_coverage: Option, /// Whether two equal expressions return the same rows for the same /// input rows, so they may be one group (not an unordered `LIMIT`). pub shareable_when_equal: bool, } -/// The `SummaryAgg` with its lifted selection removed, written over group -/// ids: the residual links from `reads` up to the `SummaryAgg`. -pub struct SummaryDefinition { - /// The group below the residual links; `None` when the bottom link is - /// the `Scan`, rebuilt without the lifted predicates. - pub reads: Option, - /// Bottom-up, each link without its input; the last is the `SummaryAgg`. - pub links: Vec>, +/// What a summary state computes, with every lifted selection removed. +pub struct SummaryStateDefinition { + /// The group the operators below read: the first input under the + /// SummaryAgg that is not Filter / range TimeRange / TimeShift without @ + /// / direct-column Project. + pub input_group: MemoGroupId, + /// From the SummaryAgg down, each operator not lifted into the + /// selection, with only its residual arguments; `()` marks its single + /// input (the next one, or input_group). Two states may merge only if + /// input_group and these operators are equal. + pub operators_kept_in_definition: Vec>, +} + +pub struct SummaryCoverage { + pub(crate) definition: SummaryStateDefinition, + /// Union of boxes over the output rows of the operators' input. + pub(crate) selection: Vec, +} + +impl SummaryCoverage { + pub(crate) fn derive_from_inputs( + operator: &Operator>, + ) -> Result; + /// A hash of the `SummaryStateDefinition`; equal definitions in one + /// memo get equal ids. + pub(crate) fn definition_id(&self) -> String; } pub struct Memo { @@ -269,18 +287,47 @@ impl Memo { } ``` -`SummaryCoverage` is the coverage type of today's IR with its definition as a parameter: a group's coverage has a `SummaryDefinition`, an operator node's (executor, export) its residual node tree. +The rules read an operator's inputs through one trait, so the same rules derive a front-end node's schema and a group's `GroupProperties`: + +```rust +pub(crate) trait OperatorInput: Sized { + fn schema(&self) -> &Schema; + fn result_kind(&self) -> OperatorResultKind; + /// The input's own operator, its children as inputs again. + fn operator(&self) -> Operator; + fn summary_coverage(&self) -> Option<&SummaryCoverage>; +} +// Front-end node trees; summary_coverage is None. +impl OperatorInput for Rc { … } +/// A memo group read as an input: its group_properties and expression. +pub(crate) struct MemoGroupAsInput<'m> { memo: &'m Memo, group_id: MemoGroupId } +impl OperatorInput for MemoGroupAsInput<'_> { … } + +// The existing rules, generic over the input; existing names kept. +impl Operator { + pub(crate) fn output_schema(&self) -> Result; + pub(crate) fn output_kind(&self) -> OperatorResultKind; + pub(crate) fn validate_inputs(&self) -> Result<(), SchemaDerivationError>; + pub(crate) fn is_shareable_when_equal(&self) -> bool; +} +fn derive_group_properties( + memo: &Memo, + expression: &Operator, +) -> Result; +``` + +Summary coverage exists only in the memo: a node tree has none, and `SummaryCoverage` has no type parameter. The executor reads only `definition_id` from the plan. A group is its expression: one operator, with its arguments, over input groups. Hash-consing compares and hashes the expression directly, so two expressions are one group when their operators and arguments are equal and their input groups are the same. θ is not an operator argument, so builds that differ only in θ share one group. An expression whose copies may return different rows (an unordered `LIMIT`) always gets a new group. -Everything that was computed on operator-node trees is computed on groups. `GroupProperties` is derived once, when the group is created, by the same rules that check operator nodes (schema derivation, input validation, merge check), reading only the input groups' `expression` and `group_properties` from the memo, never a node tree: +Everything that was computed on operator-node trees is computed on groups. `find_or_add_group` derives `GroupProperties` once, when the group is created, with `derive_group_properties`: the rules that check operator nodes (schema derivation, input validation, merge check), applied to the expression over `MemoGroupAsInput`s, which read only the input groups' `expression` and `group_properties`, never a node tree: - `schema` and `result_kind` from the operator and its input groups' schemas and result kinds; a scalar subquery reads the referenced group's schema. -- `summary_coverage`, for a summary state: walking down through the input groups' expressions (Filter, Project, TimeRange, TimeShift, down to the Scan), lifting their predicates into the selection; the rest is the `SummaryDefinition`. A merge's coverage comes from its inputs' coverage; a merge is valid only over inputs with the same definition and disjoint selections. +- `summary_coverage`, for a summary state: walking down from the SummaryAgg through the input groups' expressions while each is a Filter, a range TimeRange, a TimeShift without `@` or a direct-column Project, lifting their predicates into the selection. The group below them is the definition's `input_group`; a Scan is never lifted through, so its predicates stay in that group. The operators not lifted, each with its residual arguments, are `operators_kept_in_definition`. A tumbling-window chain covers what one build over its whole window does. A merge's coverage comes from its inputs' stored coverage; a merge is valid only over inputs with equal definitions and disjoint selections. - The checks that read an input's operator (a merge of the newest windows reads a window chain; a selected build) read the input group's `expression`. - `shareable_when_equal`: a `LIMIT` is shareable over a `Sort` with the same partitions (the input group's `expression`); a `Dedup` on every column (the input group's schema). -A candidate's expression goes through the same derivation and must produce the group's schema and result kind. The executor and the export keep their node-tree summary coverage (`definition_id`) until Stage 3's export builds node trees from the memo. `OperatorNode` recursively contains chosen child nodes and therefore describes a fixed plan fragment; `Operator` contains input group IDs and leaves their implementations open. A candidate needs its own expression because a group may offer several ways to compute the same result. +`add_candidate` derives a candidate's properties with the same `derive_group_properties` and requires its schema and result kind to equal the group's. `OperatorNode` recursively contains chosen child nodes and therefore describes a fixed plan fragment; `Operator` contains input group IDs and leaves their implementations open. A candidate needs its own expression because a group may offer several ways to compute the same result. Removing the one-field `CandidateExpression` wrapper is possible but not adopted. From e364e86dc3a52088a6ef194cd92539510d4e25c7 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 21:17:35 +0000 Subject: [PATCH 36/67] docs(concepts): unlifted_operators; point to the derivation examples Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index ebdabda09..4424f2a2c 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -249,11 +249,11 @@ pub struct SummaryStateDefinition { /// SummaryAgg that is not Filter / range TimeRange / TimeShift without @ /// / direct-column Project. pub input_group: MemoGroupId, - /// From the SummaryAgg down, each operator not lifted into the - /// selection, with only its residual arguments; `()` marks its single - /// input (the next one, or input_group). Two states may merge only if - /// input_group and these operators are equal. - pub operators_kept_in_definition: Vec>, + /// The operators not lifted into the selection, listed top down from + /// the SummaryAgg, each with only its residual arguments; `()` marks its + /// single input (the next one, or input_group). Two states may merge only + /// if input_group and these operators are equal. + pub unlifted_operators: Vec>, } pub struct SummaryCoverage { @@ -287,6 +287,8 @@ impl Memo { } ``` +How the definition and selection are derived, with examples: `docs/design_docs/proposals/asap-primitive-schema.md` §4.2.2. + The rules read an operator's inputs through one trait, so the same rules derive a front-end node's schema and a group's `GroupProperties`: ```rust @@ -323,7 +325,7 @@ A group is its expression: one operator, with its arguments, over input groups. Everything that was computed on operator-node trees is computed on groups. `find_or_add_group` derives `GroupProperties` once, when the group is created, with `derive_group_properties`: the rules that check operator nodes (schema derivation, input validation, merge check), applied to the expression over `MemoGroupAsInput`s, which read only the input groups' `expression` and `group_properties`, never a node tree: - `schema` and `result_kind` from the operator and its input groups' schemas and result kinds; a scalar subquery reads the referenced group's schema. -- `summary_coverage`, for a summary state: walking down from the SummaryAgg through the input groups' expressions while each is a Filter, a range TimeRange, a TimeShift without `@` or a direct-column Project, lifting their predicates into the selection. The group below them is the definition's `input_group`; a Scan is never lifted through, so its predicates stay in that group. The operators not lifted, each with its residual arguments, are `operators_kept_in_definition`. A tumbling-window chain covers what one build over its whole window does. A merge's coverage comes from its inputs' stored coverage; a merge is valid only over inputs with equal definitions and disjoint selections. +- `summary_coverage`, for a summary state: walking down from the SummaryAgg through the input groups' expressions while each is a Filter, a range TimeRange, a TimeShift without `@` or a direct-column Project, lifting their predicates into the selection. The group below them is the definition's `input_group`; a Scan is never lifted through, so its predicates stay in that group. The operators not lifted, each with its residual arguments, are `unlifted_operators`. A tumbling-window chain covers what one build over its whole window does. A merge's coverage comes from its inputs' stored coverage; a merge is valid only over inputs with equal definitions and disjoint selections. - The checks that read an input's operator (a merge of the newest windows reads a window chain; a selected build) read the input group's `expression`. - `shareable_when_equal`: a `LIMIT` is shareable over a `Sort` with the same partitions (the input group's `expression`); a `Dedup` on every column (the input group's schema). From f9d17909b356e55b8b9a1e8cbe41fd462e1837aa Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 22:46:04 +0000 Subject: [PATCH 37/67] docs(concepts): unlifted_operators name the groups their subqueries read Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../concepts/memo-groups-properties-requirements.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 4424f2a2c..4bf19d370 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -250,10 +250,12 @@ pub struct SummaryStateDefinition { /// / direct-column Project. pub input_group: MemoGroupId, /// The operators not lifted into the selection, listed top down from - /// the SummaryAgg, each with only its residual arguments; `()` marks its - /// single input (the next one, or input_group). Two states may merge only - /// if input_group and these operators are equal. - pub unlifted_operators: Vec>, + /// the SummaryAgg, each with only its residual arguments. `None` marks + /// an operator's main input (the next unlifted operator, or input_group); + /// `Some(g)` is another input group it reads, such as the group a scalar + /// subquery in a Filter reads. Two states may merge only if input_group + /// and these operators are equal. + pub unlifted_operators: Vec>>, } pub struct SummaryCoverage { From 9c810decbaf78b0ad86228c544ec88bea7d2caba Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 22:58:47 +0000 Subject: [PATCH 38/67] docs(concepts): column lineage replaces table qualifiers, with memo-group examples Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 75 +++++++++++++++++++ 1 file changed, 75 insertions(+) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 4bf19d370..f6d0eb9a5 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -335,6 +335,81 @@ Everything that was computed on operator-node trees is computed on groups. `find Removing the one-field `CandidateExpression` wrapper is possible but not adopted. +#### Column lineage + +Every output column of a group has a **lineage**: the source column its values are copied from, or none when the column is computed. It is a group property, derived once from the input groups' lineage (as optd's column-reference property is), and it replaces the table qualifier that schema fields carried. A schema field keeps only its name, type and nullability; a front end resolves a qualified name such as `f.src` with its own binding scope while it lowers the query, and the IR never needs the qualifier again. + +```rust +pub struct GroupProperties { + pub schema: Schema, // fields: name, type, nullability; no table qualifier + pub result_kind: OperatorResultKind, + /// One entry per output column, in schema order. + pub column_lineage: Vec, + pub summary_coverage: Option, + pub shareable_when_equal: bool, +} + +/// Where an output column's values come from. +pub enum ColumnLineage { + /// Copied unchanged from a column of a source (a Scan's table or metric). + SourceColumn { source: String, column: String }, + /// Computed by an operator: a measure, an expression, a summary state or readout. + Computed, +} +``` + +**Derivation, per operator** (from the input groups' `column_lineage`): + +| Operator | Output column lineage | +|---|---| +| `Scan` | each column: `SourceColumn { source, column }` | +| `Filter`, `TimeRange`, `TimeShift`, `Sort`, `Limit`, `Dedup` | the input's, unchanged | +| `Project` | a column item, renamed or not: that input column's lineage; an expression item: `Computed` | +| `Aggregate`, `SummaryAgg`, readouts | group-by keys: the input's lineage of that key; measures, states and values: `Computed` | +| `Join` | the left input's columns, then the right input's | +| `SetOp`, `Concat` | per column: the inputs' lineage if they all agree, else `Computed` | +| `BinaryOp` (vector match) | label columns: the matched side's lineage; the value: `Computed` | + +**What reads it:** +- **Summary coverage.** A selection constrains columns identified by lineage (`ColumnIdentity` is a `SourceColumn`), not by (qualifier, name). A predicate on a `Computed` column is not lifted; it stays in `unlifted_operators`. A merge's group keys must have equal lineage. +- **Candidate checks.** `add_candidate` compares a candidate's schema with its group's exactly: names, types, nullability, order. There is no qualifier to compare, so a readout renamed to the group's column name matches. + +**Example 1: a SQL aggregate and its summary candidate.** + +```sql +SELECT f.src, approx_distinct(f.dst) FROM flows f WHERE f.proto = 'tcp' GROUP BY f.src +``` + +| Group | Expression | Output columns | `column_lineage` | +|---|---|---|---| +| G1 | `Scan flows` | `src, dst, proto, ts` | `flows.src, flows.dst, flows.proto, flows.ts` | +| G2 | `Filter(proto = 'tcp')(G1)` | `src, dst, proto, ts` | same as G1 | +| G3 | `Aggregate(by src; approx_distinct(dst))(G2)` | `src, approx_distinct(dst)` | `flows.src, Computed` | +| G4 | `SummaryAgg[HLL](by src; dst)(G2)` | `src, state` | `flows.src, Computed` | +| G5 | `SummaryEstimate[cardinality](G4)` | `src, cardinality` | `flows.src, Computed` | +| — | candidate of G3: `Project(src, cardinality AS approx_distinct(dst))(G5)` | `src, approx_distinct(dst)` | `flows.src, Computed` | + +The candidate's schema equals G3's, so `add_candidate` accepts it. Before lineage, G3's key carried the qualifier `flows` and the renaming `Project` could not keep it on one column while renaming another, so this candidate was rejected. + +**Example 2: renames keep lineage.** + +```sql +SELECT source, count(*) FROM (SELECT f.src AS source FROM flows f) GROUP BY source +``` + +The inner `Project` renames `src` to `source`; its lineage stays `flows.src`. Two summary states, one grouped by `source` here and one grouped by `src` in another query, have keys of equal lineage, so a merge of the two may pass the group-key check. Compared by (qualifier, name), `source` and `flows.src` looked like different columns. + +**Example 3: the same name from two sources.** + +```sql +SELECT s.region, count(*) FROM sales s JOIN users u ON s.user = u.id GROUP BY s.region +``` + +After the join, `region` may exist on both sides. The front end resolves `s.region` to the left input's column while lowering; in the IR the two columns have lineage `sales.region` and `users.region`, so a selection on `sales.region` is never confused with one on `users.region`. + +**Example 4: PromQL label matchers.** `quantile_over_time(0.9, latency{job="api"}[1m])` and the same with `job="web"`. Each Scan's `job` column has lineage `latency.job`. The lifted matchers become selections `latency.job ∈ {api}` and `latency.job ∈ {web}`: disjoint selections on the same column, so the two KLL states may merge. + + ### Stage 1, Pass 1: initial candidates Implementation locations: `crates/logical-asap-optimization/src/pass1/per_computation_candidates.rs` and `memo_population.rs`. The #1203 target gives every intermediate result its own group; candidates contain one operator rather than private operator chains. From b4614627c91b53ac5b1b8ca2e0402188021966b3 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 23:13:23 +0000 Subject: [PATCH 39/67] docs(concepts): input_group is optional when Scan predicates lift; define fused groups Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 21 +++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index f6d0eb9a5..531ba465b 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -247,8 +247,10 @@ pub struct GroupProperties { pub struct SummaryStateDefinition { /// The group the operators below read: the first input under the /// SummaryAgg that is not Filter / range TimeRange / TimeShift without @ - /// / direct-column Project. - pub input_group: MemoGroupId, + /// / direct-column Project. `None` only when a Scan's predicates are + /// lifted; the Scan, with its residual predicates, is then the last + /// unlifted operator. + pub input_group: Option, /// The operators not lifted into the selection, listed top down from /// the SummaryAgg, each with only its residual arguments. `None` marks /// an operator's main input (the next unlifted operator, or input_group); @@ -327,7 +329,7 @@ A group is its expression: one operator, with its arguments, over input groups. Everything that was computed on operator-node trees is computed on groups. `find_or_add_group` derives `GroupProperties` once, when the group is created, with `derive_group_properties`: the rules that check operator nodes (schema derivation, input validation, merge check), applied to the expression over `MemoGroupAsInput`s, which read only the input groups' `expression` and `group_properties`, never a node tree: - `schema` and `result_kind` from the operator and its input groups' schemas and result kinds; a scalar subquery reads the referenced group's schema. -- `summary_coverage`, for a summary state: walking down from the SummaryAgg through the input groups' expressions while each is a Filter, a range TimeRange, a TimeShift without `@` or a direct-column Project, lifting their predicates into the selection. The group below them is the definition's `input_group`; a Scan is never lifted through, so its predicates stay in that group. The operators not lifted, each with its residual arguments, are `unlifted_operators`. A tumbling-window chain covers what one build over its whole window does. A merge's coverage comes from its inputs' stored coverage; a merge is valid only over inputs with equal definitions and disjoint selections. +- `summary_coverage`, for a summary state: walking down from the SummaryAgg through the input groups' expressions while each is a Filter, a range TimeRange, a TimeShift without `@` or a direct-column Project, lifting their predicates into the selection. The group below them is the definition's `input_group`. A Scan's predicates (PromQL label matchers) are lifted too; the Scan with its residual predicates is then the last unlifted operator and `input_group` is `None`. The operators not lifted, each with its residual arguments, are `unlifted_operators`. A tumbling-window chain covers what one build over its whole window does. A merge's coverage comes from its inputs' stored coverage; a merge is valid only over inputs with equal definitions and disjoint selections. - The checks that read an input's operator (a merge of the newest windows reads a window chain; a selected build) read the input group's `expression`. - `shareable_when_equal`: a `LIMIT` is shareable over a `Sort` with the same partitions (the input group's `expression`); a `Dedup` on every column (the input group's schema). @@ -419,7 +421,7 @@ Implementation locations: `crates/logical-asap-optimization/src/pass1/per_comput | Exact | Exact candidate in the result group | | Exact accumulator | Accumulator-state group plus a finalize candidate in the result group | | Summary | One state group per supported family; one build candidate per listed parameter configuration; separate readout candidates | -| Fused build (#1203 2c) | Candidate with `fused_groups` identifying additional results; selection must enforce their availability consistently | +| Fused build (#1203 2c) | Several operators of one sub-DAG pattern, in one appearance of the sub-expression, fused into one candidate operator; `fused_groups` lists the groups of the operators it replaces | ```rust pub struct CandidateForMemoGroupId { @@ -468,6 +470,10 @@ pub struct CandidateForMemoGroup { pub expression: CandidateExpression, pub logical_properties: LogicalProperties, pub input_requirements: Vec, + /// The groups of the operators this candidate fuses: a sub-DAG pattern + /// of several operators, in one appearance of the sub-expression, + /// computed by this one operator. Empty for a candidate that fuses + /// nothing. pub fused_groups: Vec, // Diagnostic provenance; not a planning decision. pub origin: OriginOfCandidateForMemoGroup, @@ -517,6 +523,13 @@ pub fn producer_satisfies_logical_requirements( The floor representation has the applicability condition stated in Section 2. Creating requirements also needs the input family's configuration domain; a constructor without that context can validate only the information encoded in its argument. Error-model interfaces are grouped with accuracy in Section 4. +**Fused groups.** A fused candidate computes, as one operator, a sub-DAG pattern of several operators in one appearance of the sub-expression; `fused_groups` lists the groups of the operators it replaces. Reusing a common sub-expression across appearances is Pass 2's reuse, not fusion. Fusion rules: +- top-k (a sort and limit over an aggregate, as one heap summary); +- sum over sum, min over min, max over max (distributive aggregates, as one aggregate); +- possibly Hydra (a summary over every sub-population at once). + +TODO: revisit each fused candidate's inputs and outputs to check that the fusing rules are correct. + Migration targets: remove private `root_node` chains, duplicate `input_groups`, and `accuracy_budget_parts`. The proposed predefined parameter lists replace sizing one candidate per budget partition. ### Stage 1, Pass 2: reuse rules and logical edges From 1baa2a1922c51f77314a65005ede725c40584122 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sat, 7 Nov 2026 03:44:14 +0000 Subject: [PATCH 40/67] docs(concepts): SourceColumn is one struct shared by lineage and summary coverage Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 20 +++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 531ba465b..ac587bcef 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -351,20 +351,32 @@ pub struct GroupProperties { pub shareable_when_equal: bool, } +/// A column of a source (a Scan's table or metric), by name. +pub struct SourceColumn { + pub source: String, + pub column: String, +} + /// Where an output column's values come from. pub enum ColumnLineage { - /// Copied unchanged from a column of a source (a Scan's table or metric). - SourceColumn { source: String, column: String }, + /// Copied unchanged from a column of a source. + SourceColumn(SourceColumn), /// Computed by an operator: a measure, an expression, a summary state or readout. Computed, } + +/// Summary coverage keys its selections by the same type. +pub struct SelectionBox { + pub columns: BTreeMap, + pub relative_time: Option<(Bound, Bound)>, +} ``` **Derivation, per operator** (from the input groups' `column_lineage`): | Operator | Output column lineage | |---|---| -| `Scan` | each column: `SourceColumn { source, column }` | +| `Scan` | each column: `SourceColumn(SourceColumn { source, column })` | | `Filter`, `TimeRange`, `TimeShift`, `Sort`, `Limit`, `Dedup` | the input's, unchanged | | `Project` | a column item, renamed or not: that input column's lineage; an expression item: `Computed` | | `Aggregate`, `SummaryAgg`, readouts | group-by keys: the input's lineage of that key; measures, states and values: `Computed` | @@ -373,7 +385,7 @@ pub enum ColumnLineage { | `BinaryOp` (vector match) | label columns: the matched side's lineage; the value: `Computed` | **What reads it:** -- **Summary coverage.** A selection constrains columns identified by lineage (`ColumnIdentity` is a `SourceColumn`), not by (qualifier, name). A predicate on a `Computed` column is not lifted; it stays in `unlifted_operators`. A merge's group keys must have equal lineage. +- **Summary coverage.** A selection constrains columns keyed by their `SourceColumn` lineage, not by (qualifier, name). A predicate on a `Computed` column is not lifted; it stays in `unlifted_operators`. A merge's group keys must have equal lineage. - **Candidate checks.** `add_candidate` compares a candidate's schema with its group's exactly: names, types, nullability, order. There is no qualifier to compare, so a readout renamed to the group's column name matches. **Example 1: a SQL aggregate and its summary candidate.** From 25b9d8eb6846ee78dc5469c59f29df7f9292f597 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sat, 7 Nov 2026 16:46:57 +0000 Subject: [PATCH 41/67] docs(concepts): chain groups, general rollup rule, dead-candidate elimination, TimeBound coverage Logical properties are parameters only; tumbling windows are range-free chain groups read through TimeRange/TimeShift and merged by SummaryMerge. Pass 2 order: rollup from the finest common group, top-k reuse, reach and floors (set_input_requirements), dead-candidate elimination with root-cause traces, chain groups, logical edges. SelectionBox keeps only columns, with TimeBound intervals on the timestamp column; aggregate_coverage covers exact Aggregate groups too. evaluation_grid_ms is None without a phase. error_propagation takes a metric (b = 0.1, constant contracts for key-only Filters and abs/clamp/negation); floor_for_cap lives in asap-accuracy. Chain retention need is max(offset + range). Worked examples throughout. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 426 ++++++++++++++---- 1 file changed, 333 insertions(+), 93 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index ac587bcef..330b55cf4 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -85,19 +85,15 @@ This is the single reference table for property dimensions. Stage descriptions b | Dimension | Values and meaning | Code reference or target | |---|---|---| | Parameters | A predefined configuration list per family: KLL and KMV `k`; DDSketch `alpha`; HLL `precision`; CMS and CountSketch `width` and `depth`; heap variants also `heap_size`; UnivMon its sizing parameters. The lists are today's sizing over ε = 0.1 down to 0.001; KMV's and UnivMon's distinct-count entries stop at ε = 0.01. A heap variant combines each width × depth with heap sizes from the 1-2-5 series 10, 20, 50, …, 10 000: a top-k consumer's floor is heap_size ≥ k, and the smallest listed heap that meets it is the next step up, at most 2.5× too large. A sketch build's parameters must be in its family's list; exact operators have no sketch parameters. Hydra and the other families Pass 1 does not size have empty `parameter_list`s, so no build of theirs is a valid candidate until their error models exist (#1228). | `SketchParams` in `crates/types/src/ir/schema/sketch.rs`; predefined lists are the #1203 target | -| Window layout — Extension | No layout, or tumbling windows with a positive width. Every grid is anchored at phase 0 (the epoch), so the width alone fixes it. Candidate widths come from the gcd closure of the queries' ranges, steps and phases. Sliding merge/subtraction layouts remain future work. Window count is derived from range and width; retention amount is selected later. | `WindowLayout`; `pass2/memo/window_candidates.rs`; #1160 | | Timing | `IngestionTime` or `QueryTime` | `ExecutionTiming` | -| Lifetime | `None`: discard after the consuming execution; `OneEvaluation`: retain until consumers within that evaluation finish; `AcrossEvaluations`: retain across evaluations in the specified period. A non-`None` lifetime means materialized. | `Lifetime`; #1200 / #1214 | -| Storage medium — Extension | `Memory` or `Disk`, carried by a materialized lifetime | `StorageMedium`; #1214 | +| Lifetime | `None`: discard after the consuming execution; `OneEvaluation`: retain until consumers within that evaluation finish; `AcrossEvaluations`: retain across evaluations in the specified period. A non-`None` lifetime means materialized. | `Lifetime`; #1200 | +| Storage medium — Extension | `Memory` or `Disk`, carried by a materialized lifetime | `StorageMedium` | Error guarantees are derived from parameters and the statistic being read. They include an error metric, a bound, and a failure probability; exact computation introduces no approximation error. The same state may support different readouts, so its guarantee must be interpreted for the relevant statistic. -The extended logical properties add the window layout; the physical properties stay the formulation's: +The properties are the formulation's, $p_v^{\mathrm{log}}=\boldsymbol\theta_v$ and $p_v^{\mathrm{phys}}=(\tau_v,\ell_v)$; in code, `LogicalProperties { parameters }` and `PhysicalProperties { timing, lifetime }`. -$$ -p_v^{\mathrm{log}}=(\boldsymbol\theta_v,\omega_v),\qquad -p_v^{\mathrm{phys}}=(\tau_v,\ell_v). -$$ +**Tumbling windows are expressions, not properties.** A tumbling-window chain is its own memo group, `SummaryTumblingWindowChain[w](X)`, with the window width $w$ as an operator argument (Section 3, "Chain groups"). Every grid is anchored at phase 0 (the epoch), so the width alone fixes it. A query over a range reads the chain through `TimeRange` and `TimeShift` groups and merges the selected windows with `SummaryMerge`. No property carries the width, and two queries with different ranges or offsets share one chain group. Sliding merge/subtraction layouts remain future work; how many windows a chain retains is $h_v$ (Section 5). ### Logical requirements @@ -118,25 +114,35 @@ Physical requirements are derived from the consumer's timing and execution conte QT means QueryTime; IT means IngestionTime. Alignment, data coverage, and retention checks still apply to allowed combinations. -**Alignment.** There is no physical schedule property. When a result can be read follows from its lifetime and window layout: +**Alignment.** There is no physical schedule property. When a result can be read follows from its lifetime and, for a chain, its window width: - **None** and **OneEvaluation**: the result is computed inside one evaluation context κ (Section 5) and read there. A OneEvaluation result belongs to one context: queries with equal recurrence and range evaluation are evaluated together, and that evaluation's consumers share the result. When it runs is the context's schedule, and Stage 3 chooses it through $x_{\kappa,v}$, just as for ℓ = None. Sharing across differently scheduled queries through a gcd schedule is not offered: it is rarely beneficial and wastes runs. -- **Windowed AcrossEvaluations**: the input is cut into tumbling windows [k·w, (k+1)·w), with w from the window layout ω and the grid anchored at phase 0 (the epoch). Each window is built when it ends and can be read from then on. A consumer can read a range that ends on a window boundary. With w = 10 s, a `[1m]` query evaluated every 30 s at phase 0 merges the newest 6 windows; one evaluated at phase 15 s cannot, because its range ends inside a window; with that query in the workload, the gcd closure also offers w = 5 s, which both can read. +- **AcrossEvaluations** is offered only to chain candidates, `SummaryTumblingWindowChain[w](X)`: the input is cut into tumbling windows [k·w, (k+1)·w), with the grid anchored at phase 0 (the epoch). Each window is built when it ends and can be read from then on. A `TimeRange` over the chain selects the windows that lie fully inside [t − range, t), so a consumer can read a range whose ends lie on window boundaries. With w = 10 s, a `[1m]` query evaluated every 30 s at phase 0 merges 6 windows; one evaluated at phase 15 s cannot, because its range ends inside a window; with that query in the workload, the gcd closure also offers w = 5 s, which both can read. Every other candidate, including the `TimeRange` and `TimeShift` groups over a chain, is computed within one evaluation. + +A grid phase is not needed: a phase is absorbed by the width, because width candidates come from the gcd closure of the queries' grids, ranges and offsets (Section 3). For example, phases 0 and 5 s with w = 10 s give w = 5 s. Alignment alone does not ensure sufficient retained history; that is $h_v$ (Section 5). + +**Why a chain needs a declared phase.** Take `sum_over_time(m[5m])` evaluated every 60 s, and a 60 s chain with windows [12:00:00, 12:01:00), [12:01:00, 12:02:00), … + +| Case | Evaluation at | Data needed | Windows | +|---|---|---|---| +| Phase 0 declared | 12:05:00 | [12:00:00, 12:05:00) | exactly the 5 windows [12:00, 12:01) … [12:04, 12:05): the merge is correct | +| No phase declared; the engine actually evaluates at second 17 | 12:05:17 | [12:00:17, 12:05:17) | [12:00, 12:01) holds only 43 s of the range and [12:05, 12:06) only 17 s: merging only full windows misses [12:00:17, 12:01:00) and [12:05:00, 12:05:17); adding the partial windows counts [12:00:00, 12:00:17) and [12:05:17, 12:06:00) too. Both answers are wrong | +| Phase 15 s declared | 12:05:15 | [12:00:15, 12:05:15) | the widths divide gcd(60 s, 5 m, 15 s) = 15 s; with w = 15 s the range is exactly 20 windows | -A grid phase is not needed: a phase is absorbed by the width, because width candidates come from the gcd closure of the queries' ranges, steps and phases. For example, phases 0 and 5 s with w = 10 s give w = 5 s. Alignment alone does not ensure sufficient retained history; that is $h_v$ (Section 5). +Prometheus staggers rule-group evaluations by a hash of the group, so a fixed interval without a declared phase does not mean phase 0. One rule decides both sides: `QueryRecurrence::evaluation_grid_ms` returns `None` for a `FixedInterval` without a phase. Pass 2 then offers that query no chain width, and `delivers_on` lets it read no chain. Workloads declare the phase with `FixedIntervalAt { interval, evaluation_phase }` to use chains. -An evaluation context κ is described by the workload's own recurrence and range evaluation (Section 5). Let a windowed producer have tumbling width $w$. A consumer running in κ can read the producer when $w$ divides every evaluation time of κ: +An evaluation context κ is described by the workload's own recurrence and range evaluation (Section 5). Let a chain producer have tumbling width $w$. A consumer running in κ can read the producer when $w$ divides every evaluation time of κ: -| Producer lifetime, or κ's recurrence for a windowed AcrossEvaluations producer | Required alignment | +| Producer lifetime, or κ's recurrence for a chain producer with ℓ = AcrossEvaluations | Required alignment | |---|---| | None | Always: runs in κ | | OneEvaluation | The result belongs to κ | | `FixedIntervalAt { interval, evaluation_phase }`, range step $s$ | $w \mid \gcd(\text{interval}, s, \text{phase})$, the evaluation grid; $s$ counts only for a range evaluation of more than one step | | `Scheduled(times)` | $w$ divides every time (and $s$) | | `OneTime { execute_at: Some(t) }` | $w \mid t$ (and $s$) | -| `FixedInterval` without a phase, `OneTime { execute_at: None }`, `EstimatedRate`, `Unknown` | Never: the evaluation times are not known | -| AcrossEvaluations, no window layout | Not offered yet: it needs its own delivery rule | +| `FixedInterval` without a phase, `OneTime { execute_at: None }`, `EstimatedRate`, `Unknown` | Never: the evaluation times are not known, and `evaluation_grid_ms` is `None` | +| AcrossEvaluations on a candidate that is not a chain | Not offered: only chain candidates are materialized across evaluations | -The evaluation grid is `QueryRecurrence::evaluation_grid_ms(range_evaluation)` in `crates/types/src/workload/mod.rs`, which the workload entries also use. One-off queries are supported: each is a context of its own recurrence, run `invocations` times. +The evaluation grid is `QueryRecurrence::evaluation_grid_ms(range_evaluation)` in `crates/types/src/workload/mod.rs`, which the workload entries, Pass 2's chain widths and `delivers_on` all use. One-off queries are supported: each is a context of its own recurrence, run `invocations` times. ### Invalid combinations @@ -146,7 +152,7 @@ The checked constructor rejects one combination: |---|---| | IngestionTime with OneEvaluation | No query evaluation lifetime at ingestion time | -A tumbling grid's width is the candidate's logical window layout $\omega_v$ and its phase is 0, so no physical field repeats or extends it. +A tumbling grid's width is an argument of the chain operator, part of the expression, and its phase is 0, so no property repeats or extends it. ### Decision tree and order of decisions @@ -157,21 +163,19 @@ flowchart TD G["Memo group G
(logical result)"] --> R1["e_v: exact operator"] G --> R2["e_v: readout of KLL state"] G --> R3["e_v: readout of DDSketch state"] + G --> R5["e_v: SummaryMerge over TimeRange
of a chain of width w
(state groups; widths from the
gcd closure of grids, ranges, offsets)"] G --> R4["e_v: other operators
(prune: executor support)"] R2 --> T1["θ: k = 200"] R2 --> T2["θ: k = 345"] R2 --> T3["θ: k = …
(prune: pruning cap)"] - T2 --> W1["ω: no windows"] - T2 --> W2["ω: tumbling, width w
from the gcd closure of ranges,
steps and phases (prune: w must
divide the group's query range)"] - - W2 --> P1["τ = QueryTime"] - W2 --> P2["τ = IngestionTime
(prune: no ingestion pipeline)"] + T2 --> P1["τ = QueryTime"] + T2 --> P2["τ = IngestionTime
(prune: no ingestion pipeline)"] P1 --> L1["ℓ = None"] P1 --> L2["ℓ = OneEvaluation"] - P1 --> L3["ℓ = AcrossEvaluations
(prune: deployment keeps
no query-time results)"] + P1 --> L3["ℓ = AcrossEvaluations
(prune: not a chain; deployment
materializes no query-time results)"] P2 --> L4["ℓ = None"] P2 --> L5["ℓ = AcrossEvaluations"] P2 -. invalid .-> L6["ℓ = OneEvaluation"] @@ -179,11 +183,10 @@ flowchart TD | Level | Decides | Stage | Values offered | Pruned when | |---|---|---|---|---| -| 1 | expression $e_v$ | Pass 1 and Pass 2 rules | each way to compute the group | the executor cannot build the family | -| 2 | $\boldsymbol\theta_v$ | Pass 1 | the family's predefined list | the pruning cap (Section 4) | -| 3 | $\omega_v$ — Extension | Pass 2 | tumbling widths from the gcd closure of the queries' ranges, steps and phases | the width does not divide the group's query range | -| 4 | $\tau_v$ | Stage 2 | QueryTime, IngestionTime | the source has no ingestion pipeline | -| 5 | $\ell_v$ | Stage 2 | None, OneEvaluation, AcrossEvaluations | invalid with $\tau_v$; the deployment keeps no query-time results | +| 1 | expression $e_v$, including a merge over a chain of width $w$ | Pass 1 and Pass 2 rules | each way to compute the group; chain widths from the gcd closure of the reaching queries' grids, ranges and offsets | the executor cannot build the family; no reaching query has a known grid | +| 2 | $\boldsymbol\theta_v$ | Pass 1 | the family's predefined list | the pruning cap (Section 4); dead-candidate elimination (Section 3) | +| 3 | $\tau_v$ | Stage 2 | QueryTime, IngestionTime | the source has no ingestion pipeline | +| 4 | $\ell_v$ | Stage 2 | None, OneEvaluation, AcrossEvaluations | invalid with $\tau_v$; AcrossEvaluations on a candidate that is not a chain; the deployment materializes no query-time results | The tree does not decide which producer each consumer input reads ($y$), in which context a candidate with ℓ = None or OneEvaluation runs (Section 5), or how much an AcrossEvaluations candidate retains ($h_v$). Those are Stage 3 choices. @@ -192,11 +195,11 @@ The tree does not decide which producer each consumer input reads ($y$), in whic **The order changes how many nodes enumeration visits:** $\sum_d \prod_{d' \le d} \lvert \text{values left at } d' \rvert$. Dimensions that prune much go early; dimensions that multiply without pruning go late. The order above follows four rules: 1. **Logical before physical.** Stage 1 hash-conses groups and computes $\alpha^{\log}$ once, before Stage 2 multiplies each logical candidate by its physical alternatives. -2. **Strong prunes early.** Executor support and the pruning cap act before windows and physical properties multiply the tree. +2. **Strong prunes early.** Executor support and the pruning cap act before physical properties multiply the tree. 3. **Dependent dimensions adjacent.** $\tau \to \ell$ rejects the invalid combination at the first level that can detect it. -4. **Deployment-wide prunes as early as possible.** Without an ingestion pipeline the IngestionTime subtree is pruned once per candidate; moving $\tau$ above $\omega$ would prune it once per group, with the same leaves. +4. **Deployment-wide prunes as early as possible.** Without an ingestion pipeline the IngestionTime subtree is pruned once per candidate; moving $\tau$ above $\boldsymbol\theta$ would prune it once per group, with the same leaves. -Alternative orders worth measuring: $\tau$ first, when most sources lack an ingestion pipeline; $\omega$ before $\boldsymbol\theta$, when queries' ranges prune many widths; and requirement-driven (top-down) enumeration, which visits fewer nodes but must take the union of all consumers' requirements so that sharing is not lost. +Alternative orders worth measuring: $\tau$ first, when most sources lack an ingestion pipeline; and requirement-driven (top-down) enumeration, which visits fewer nodes but must take the union of all consumers' requirements so that sharing is not lost. ## 3. Pipeline: interfaces and rules @@ -204,7 +207,7 @@ Alternative orders worth measuring: $\tau$ first, when most sources lack an inge |---|---|---|---| | Stage 0 | Query IR | `Memo` (groups without candidates) | Hash-cons logical expressions | | Stage 1, Pass 1 | Group DAG | `Memo` | Add initial logical candidates and intermediate groups | -| Stage 1, Pass 2 | Memo | `LogicalASAPMemoCandidates` | Add reuse alternatives and compute logical edges | +| Stage 1, Pass 2 | Memo | `LogicalASAPMemoCandidates` | Add rollup, top-k and chain alternatives, set parameter floors, eliminate dead candidates, and compute logical edges | | Stage 2 | Logical memo candidates | `PhysicalASAPMemoCandidates` | Expand physical alternatives and compute compatibility | | Stage 3 | Physical memo candidates | `DagOfSelectedCandidates` | Select implementations, dependencies, and executions | | Realizer/export | Selected DAG | `PhysicalASAPDAG` | Build and export the executable graph | @@ -236,23 +239,26 @@ pub struct MemoGroup { pub struct GroupProperties { pub schema: Schema, pub result_kind: OperatorResultKind, - /// What a summary-state group covers; `None` for any other group. - pub summary_coverage: Option, + /// What an aggregate group covers: a summary state, a chain, a merge + /// over a chain's windows, or an exact `Aggregate` result; `None` for + /// any other group. + pub aggregate_coverage: Option, /// Whether two equal expressions return the same rows for the same /// input rows, so they may be one group (not an unordered `LIMIT`). pub shareable_when_equal: bool, } -/// What a summary state computes, with every lifted selection removed. +/// What a summary state or exact aggregate computes, with every lifted +/// selection removed. pub struct SummaryStateDefinition { /// The group the operators below read: the first input under the - /// SummaryAgg that is not Filter / range TimeRange / TimeShift without @ + /// SummaryAgg (or Aggregate) that is not Filter / TimeRange / TimeShift /// / direct-column Project. `None` only when a Scan's predicates are /// lifted; the Scan, with its residual predicates, is then the last /// unlifted operator. pub input_group: Option, /// The operators not lifted into the selection, listed top down from - /// the SummaryAgg, each with only its residual arguments. `None` marks + /// the SummaryAgg (or Aggregate), each with only its residual arguments. `None` marks /// an operator's main input (the next unlifted operator, or input_group); /// `Some(g)` is another input group it reads, such as the group a scalar /// subquery in a Filter reads. Two states may merge only if input_group @@ -301,9 +307,9 @@ pub(crate) trait OperatorInput: Sized { fn result_kind(&self) -> OperatorResultKind; /// The input's own operator, its children as inputs again. fn operator(&self) -> Operator; - fn summary_coverage(&self) -> Option<&SummaryCoverage>; + fn aggregate_coverage(&self) -> Option<&SummaryCoverage>; } -// Front-end node trees; summary_coverage is None. +// Front-end node trees; aggregate_coverage is None. impl OperatorInput for Rc { … } /// A memo group read as an input: its group_properties and expression. pub(crate) struct MemoGroupAsInput<'m> { memo: &'m Memo, group_id: MemoGroupId } @@ -322,15 +328,15 @@ fn derive_group_properties( ) -> Result; ``` -Summary coverage exists only in the memo: a node tree has none, and `SummaryCoverage` has no type parameter. The executor reads only `definition_id` from the plan. +Aggregate coverage exists only in the memo: a node tree has none, and `SummaryCoverage` has no type parameter. The executor reads only `definition_id` from the plan. A group is its expression: one operator, with its arguments, over input groups. Hash-consing compares and hashes the expression directly, so two expressions are one group when their operators and arguments are equal and their input groups are the same. θ is not an operator argument, so builds that differ only in θ share one group. An expression whose copies may return different rows (an unordered `LIMIT`) always gets a new group. Everything that was computed on operator-node trees is computed on groups. `find_or_add_group` derives `GroupProperties` once, when the group is created, with `derive_group_properties`: the rules that check operator nodes (schema derivation, input validation, merge check), applied to the expression over `MemoGroupAsInput`s, which read only the input groups' `expression` and `group_properties`, never a node tree: - `schema` and `result_kind` from the operator and its input groups' schemas and result kinds; a scalar subquery reads the referenced group's schema. -- `summary_coverage`, for a summary state: walking down from the SummaryAgg through the input groups' expressions while each is a Filter, a range TimeRange, a TimeShift without `@` or a direct-column Project, lifting their predicates into the selection. The group below them is the definition's `input_group`. A Scan's predicates (PromQL label matchers) are lifted too; the Scan with its residual predicates is then the last unlifted operator and `input_group` is `None`. The operators not lifted, each with its residual arguments, are `unlifted_operators`. A tumbling-window chain covers what one build over its whole window does. A merge's coverage comes from its inputs' stored coverage; a merge is valid only over inputs with equal definitions and disjoint selections. -- The checks that read an input's operator (a merge of the newest windows reads a window chain; a selected build) read the input group's `expression`. +- `aggregate_coverage`, for a summary state or an exact `Aggregate` result: walking down from the SummaryAgg or Aggregate through the input groups' expressions while each is a Filter, a TimeRange, a TimeShift (with or without `@`) or a direct-column Project, lifting their predicates into the selection; a TimeRange or TimeShift becomes an interval on the timestamp column (below). The group below them is the definition's `input_group`. A Scan's predicates (PromQL label matchers, SQL filters, including absolute timestamp filters) are lifted too; the Scan with its residual predicates is then the last unlifted operator and `input_group` is `None`. The operators not lifted, each with its residual arguments, are `unlifted_operators`. A chain `SummaryTumblingWindowChain[w](X)` has the definition of a build over X and leaves the timestamp column unconstrained; a `TimeRange` or `TimeShift` over a chain constrains it as over raw rows. A merge's coverage comes from its inputs' stored coverage; a merge is valid only over inputs with equal definitions and disjoint selections, or over one time-indexed state. So the coverage of `SummaryMerge(TimeRange(TimeShift(Chain[w](X))))` equals that of `SummaryAgg(TimeRange(TimeShift(X)))`, and `add_candidate` accepts the merge in the original state group. +- The checks that read an input's operator (a merge over a time-indexed state reads a `TimeRange` over a chain; a selected build) read the input group's `expression`. - `shareable_when_equal`: a `LIMIT` is shareable over a `Sort` with the same partitions (the input group's `expression`); a `Dedup` on every column (the input group's schema). `add_candidate` derives a candidate's properties with the same `derive_group_properties` and requires its schema and result kind to equal the group's. `OperatorNode` recursively contains chosen child nodes and therefore describes a fixed plan fragment; `Operator` contains input group IDs and leaves their implementations open. A candidate needs its own expression because a group may offer several ways to compute the same result. @@ -347,7 +353,7 @@ pub struct GroupProperties { pub result_kind: OperatorResultKind, /// One entry per output column, in schema order. pub column_lineage: Vec, - pub summary_coverage: Option, + pub aggregate_coverage: Option, pub shareable_when_equal: bool, } @@ -365,13 +371,32 @@ pub enum ColumnLineage { Computed, } -/// Summary coverage keys its selections by the same type. +/// Aggregate coverage keys its selections by the same type. The timestamp +/// column is one of the columns; there is no separate time field. pub struct SelectionBox { pub columns: BTreeMap, - pub relative_time: Option<(Bound, Bound)>, +} + +/// An endpoint of an interval on a timestamp column. +pub enum TimeBound { + /// An absolute time, in ms since the epoch. + At(i64), + /// An offset, in ms, from the evaluation time t. + FromEvaluationTime(i64), } ``` +A repeatedly evaluated query covers different absolute times at every evaluation, so its group's coverage is written relative to the evaluation time t; an `@ T` modifier or a SQL timestamp filter is absolute. Both are intervals on the timestamp column, compared and lifted like any other column constraint. For a range evaluation over [t − D, t] (D is the query range's duration): + +| Query fragment | Constraint on `m.ts` | +|---|---| +| `m[5m]` | [t − 5m, t): both ends `FromEvaluationTime` | +| `m[5m] offset 1h` | [t − 1h − 5m, t − 1h) | +| `m[5m] @ end()` | [t − 5m, t): the same as no `@` in an instant query | +| `m[5m] @ start()` | [t − D − 5m, t − D): an offset of D | +| `m[5m] @ 1700000000` | [T − 5m, T) with T = 1700000000 s: both ends `At` | +| SQL `WHERE ts >= '2026-11-01'` | [2026-11-01, ∞): `At` | + **Derivation, per operator** (from the input groups' `column_lineage`): | Operator | Output column lineage | @@ -433,7 +458,9 @@ Implementation locations: `crates/logical-asap-optimization/src/pass1/per_comput | Exact | Exact candidate in the result group | | Exact accumulator | Accumulator-state group plus a finalize candidate in the result group | | Summary | One state group per supported family; one build candidate per listed parameter configuration; separate readout candidates | -| Fused build (#1203 2c) | Several operators of one sub-DAG pattern, in one appearance of the sub-expression, fused into one candidate operator; `fused_groups` lists the groups of the operators it replaces | + +Pass 1 candidates carry no parameter floors, not even a top-k's heap_size ≥ k: Pass 2 sets every floor once all candidates exist. +| Fused build | Several operators of one sub-DAG pattern, in one appearance of the sub-expression, fused into one candidate operator; `fused_groups` lists the groups of the operators it replaces | ```rust pub struct CandidateForMemoGroupId { @@ -463,6 +490,17 @@ impl Memo { ) -> Result; pub fn candidate(&self, id: CandidateForMemoGroupId) -> &CandidateForMemoGroup; + + /// Replaces candidate `id`'s input requirements r^log_{c,i}, one per + /// input group. The only way to change a candidate's r^log: Pass 2 sets + /// each parameter floor from the input's caps and direct needs after the + /// rules have run. Panics when the vector does not have one entry per + /// input group. + pub fn set_input_requirements( + &mut self, + id: CandidateForMemoGroupId, + input_requirements: Vec, + ); } pub enum CandidateError { @@ -481,6 +519,8 @@ pub struct CandidateExpression { pub struct CandidateForMemoGroup { pub expression: CandidateExpression, pub logical_properties: LogicalProperties, + /// r^log_{c,i}, one per input group; changed only through + /// `Memo::set_input_requirements`. pub input_requirements: Vec, /// The groups of the operators this candidate fuses: a sub-DAG pattern /// of several operators, in one appearance of the sub-expression, @@ -493,25 +533,18 @@ pub struct CandidateForMemoGroup { pub struct LogicalProperties { parameters: Option, - window_layout: Option, -} - -pub enum WindowLayout { - Tumbling { width_ms: u64 }, } impl LogicalProperties { - /// Rejects parameters missing on a sketch build, present on an exact - /// operator, of another family or not in the family's list, and a window - /// of width 0 or on an operator that cannot be windowed. + /// Rejects parameters missing on a sketch build (a chain included), + /// present on an exact operator, of another family or not in the + /// family's list. pub fn new( expression: &CandidateExpression, parameters: Option, - window_layout: Option, ) -> Result; pub fn parameters(&self) -> Option<&SketchParams>; - pub fn window_layout(&self) -> Option<&WindowLayout>; } pub fn parameter_list(family: SketchAlgorithm) -> &'static [SketchParams]; @@ -546,22 +579,222 @@ Migration targets: remove private `root_node` chains, duplicate `input_groups`, ### Stage 1, Pass 2: reuse rules and logical edges -Paths are relative to `crates/logical-asap-optimization/src/pass2/memo/`. +Pass 2 (`crates/stage1-pass2-reuse-candidates`) runs these steps in order: -| Rule / function | File | Result or migration target | +| Step | What it does | Why it is here | |---|---|---| -| Local windows: `add_local_window_composition` | `window_candidates.rs` | Add layouts to the state group; #1160 removes copied window consumers | -| Scan widening: `add_scan_label_candidates` | `rules/scan_labels.rs` | Add a widened scan group and projections in the original groups; #1222 | -| Summary inputs: `factor_summary_inputs` | `rules/summary_inputs.rs` | Expose projections/filters as input groups; #1218 moves this into initial population | -| Top-k reuse: `add_topk_candidates` | `rules/topk.rs` | Read a compatible larger top-k result with an exact-input requirement; #1220 | -| Finer aggregate: `add_finer_aggregate_candidates` | `rules/finer_aggregate.rs` | Filter and roll up finer groups, using separate intermediate groups; #1220 | -| Rollup: `add_rollup_candidates` | `rules/rollup.rs` | Add compatible finer-state merges; #1221 | -| Window rollup: `add_window_rollup_candidates` | `rules/rollup.rs` | Merge shorter windows into longer-range results; #1221 | -| Summary results: `factor_summary_results` | `rules/summary_results.rs` | Removed once Pass 1 exposes readouts; #1218 | -| Shared configurations: `add_state_configuration_candidates` | `rules/shared_configuration_readers.rs` | Remove copied consumers once producer variants are selectable; #1203 2b | -| Shared state groups: `add_shared_summary_state_groups` | `state_groups.rs` | Remove per-size groups; #1203 2b / #1160 | - -After the rules, Pass 2 builds the logical edges and applies the pruning caps of Section 4. Edges name the consumer input `input_index_of_consumer`. +| 1. Rollup from the finest common group | Adds candidates that compute a coarser group from a finer one, and synthesizes one finest common group per cluster of equal definitions | Its candidates are new consumers that later steps must see | +| 2. Top-k reuse | Reads a compatible larger top-k result with an exact-input requirement | Same reason | +| 3. Reach, caps and floors | Computes which queries reach each candidate along candidate edges, the validity and pruning caps (Section 4), and each input's parameter floor, written with `Memo::set_input_requirements` | Needs every consumer added by steps 1 and 2 | +| 4. Dead-candidate elimination | Removes candidates no query can use, transitively, and records the root cause of each | Needs the floors; keeps step 5 from building windows for dead candidates | +| 5. Chain groups | Adds tumbling-window chain groups and the merges that read them, with widths from the queries that reach each state group | Needs the reach of step 3, including rollup consumers, and only the survivors of step 4 | +| 6. Logical edges | Records every (producer, consumer, input) and whether it satisfies the input's logical requirement | Runs on the final memo | + +Rules match group expressions, add groups with `find_or_add_group` and candidates with `add_candidate`, and never copy consumers to make a new producer visible. + +#### Rollup from the finest common group + +One rule computes a coarse group $G_c$ from a finer group $G_f$. It covers coarser SQL aggregates, coarser summary states, filtered sketch rollups, rollups of finalized values, and scan widening. $G_c$ can be computed from $G_f$ when all of these hold: + +1. **Equal definition.** Both have an `aggregate_coverage` and equal `definition`s: the same computation over the same input rows, apart from the selection. +2. **Rows are a subset.** $G_c$'s selection is inside $G_f$'s. Where it is narrower, the narrowing constrains only columns that are group-by keys of $G_f$ (matched by column lineage), so a compensating `Filter` over $G_f$'s output can select those rows. A `Filter` over state rows keeps them states. +3. **Keys are coarser.** $G_c$'s keys are a subset of $G_f$'s keys, by lineage. +4. **A rollup exists.** States roll up with `SummaryMerge(group_by)`. Finalized values roll up as follows; anything else rolls up only from states: + + | Finer measure | Coarser measure | + |---|---| + | sum | sum | + | count | sum | + | min / max | min / max | + | avg, quantile, top-k, distinct count | only from states | +5. **Equal time constraints.** The constraints on the timestamp column are equal. A longer range is read from a shorter one's windows through a shared chain (below), not by this rule. + +The candidate added to $G_c$ is `Rollup(by keys of Gc)(Filter(compensation, Gf))`, without the `Filter` when the selections are equal. The compensation must use only key columns because each output row of $G_f$ has already combined all raw rows of its key: a predicate on any other column, or on a computed value, cannot separate them. + +**Finest common group.** For every cluster of at least two groups with equal definitions and equal time constraints, the rule synthesizes one group $G^*$ that every member can roll up from: +- its selection is, per column, the union of the members' constraints; +- its keys are the union of the members' keys, plus every column on which some member's selection is narrower than $G^*$'s, so each member's compensation uses only $G^*$'s keys. + +$G^*$ is added with `find_or_add_group`, so it is an existing group when one matches, and each member gets a rollup candidate from it. Only this finest group is synthesized; intermediate groupings are not enumerated. $G^*$ is no query's output: Stage 3 selects it only when it is cheaper, and it is eliminated in step 4 when nothing reads it. + +A Scan has no aggregation, so for scans every column is a key and the rollup step is empty: the finest common Scan reads the union of the members' label or column selections, and each member reads it through a compensating `Filter`/`Project`. + +**Example 1: filtered sketch rollup.** F = `quantile by (job, host)(…latency[5m])`, C = `quantile by (host)(…latency{job="api"}[5m])`. + +| Group | Expression | Coverage (definition, selection) | Keys | Candidates | +|---|---|---|---|---| +| G1 | `Scan latency` | | | | +| G2 | `TimeRange(5m, G1)` | | | | +| G3 | `SummaryAgg[KLL, by job, host](G2)` | D, `ts` ∈ [t − 5m, t) | job, host | builds | +| G4 | `Scan latency{job="api"}` | | | | +| G5 | `TimeRange(5m, G4)` | | | | +| G6 | `SummaryAgg[KLL, by host](G5)` | D, `ts` ∈ [t − 5m, t), `job` ∈ {api} | host | builds; **`SummaryMerge(by host)(G7)`** | +| G7 | `Filter(job = "api", G3)` | | | its own expression; output is states | + +The definitions are equal, the narrowing `job ∈ {api}` is on a key of G3, {host} ⊆ {job, host}, and KLL states merge. G3 is already the finest group of the cluster {G3, G6}, so $G^*$ = G3. A query `quantile by (host)(…latency{status="500"}[5m])` gets no such candidate: `status` is not a key of G3, and every state already mixes status 200 and 500 samples. + +**Example 2: rollup of a finer finalized sum.** F = `sum by (job, instance)(rate(x[1m]))`, C = `sum by (job)(rate(x[1m]))`. + +| Group | Expression | Keys | Candidates | +|---|---|---|---| +| G3 | `rate(x[1m])` per series | | | +| G4 | `Aggregate(by job, instance; sum)(G3)` | job, instance | exact; finalize of an exact accumulator state | +| G5 | `Aggregate(by job; sum)(G3)` | job | exact; finalize of an exact accumulator state; **`Aggregate(by job; sum)(G4)`** | + +Both are exact `Aggregate` groups with equal definitions and selections, so they have `aggregate_coverage` too. Stage 3 chooses by cost between reading F's finalized result (cheap when it is materialized) and merging accumulator states. + +**Example 3: finest common group that no query asks for.** Q1 = `quantile by (host)(latency[5m])`, Q2 = `quantile by (job)(latency{host="xxx"}[5m])`. + +| Group | Expression | Coverage (definition, selection) | Keys | Candidates | +|---|---|---|---|---| +| G1 | `SummaryAgg[KLL, by host](TimeRange 5m, Scan latency)` | D, all | host | builds; **`SummaryMerge(by host)(G*)`** | +| G2 | `SummaryAgg[KLL, by job](TimeRange 5m, Scan latency{host="xxx"})` | D, `host` ∈ {xxx} | job | builds; **`SummaryMerge(by job)(G3)`** | +| **G\*** | `SummaryAgg[KLL, by job, host](TimeRange 5m, Scan latency)` | D, all | job, host | builds | +| G3 | `Filter(host = "xxx", G*)` | | | its own expression | + +The hull of {all, host ∈ {xxx}} is all; the keys are {host} ∪ {job} ∪ {host}, where the last `host` is the column on which G2 is narrower. Stage 3 chooses between two KLL builds and one build of G\* with job × host states. + +#### Reach, caps and floors + +A query reaches a candidate when a path of candidate edges leads from the query's output group to it; this is the reach $a_{q,v}$, computed top down after steps 1 and 2 so that rollup consumers count. Section 4 defines the caps from it. Each input's floor is the strongest of the floor for its cap (`floor_for_cap` in `asap-accuracy`) and any direct need such as heap_size ≥ k under a top-k. `Memo::set_input_requirements` writes it; nothing else changes a candidate's r^log. + +#### Dead-candidate elimination + +A candidate that is not a query output is **dead** when it never had a consumer, or when every edge to a consumer fails: the producer is below the input's floor, the metric rule forbids the edge (Section 4), or the consumer is dead. Elimination repeats until nothing changes. Dead candidates get no logical edges and Stage 2 does not expand them; each is recorded with its root causes: + +```rust +pub struct LogicalASAPMemoCandidates { + pub memo: Memo, + pub edges: Vec, + pub pruned: Vec, +} + +/// A candidate eliminated by Pass 2. +pub struct PrunedLogicalCandidate { + pub candidate: CandidateForMemoGroupId, + /// Every cause: one per failing edge it is cut off by, so mixed causes + /// (a floor on one edge, the metric rule on another) are all listed. + pub causes: Vec, +} + +/// One path from the candidate, through consumers that are themselves +/// eliminated, to the edge that fails. One shortest path per failing edge. +pub struct PruneCause { + /// The eliminated consumers on the path, nearest first. + pub through_pruned_consumers: Vec, + pub root: RootPruneReason, +} + +pub enum RootPruneReason { + /// The producer's parameters are weaker than the consumer input's floor. + BelowAccuracyFloorRequirement { + consumer: CandidateForMemoGroupId, + input: usize, + floor: SketchParams, + }, + /// The consumer's error contract carries none of the producer's metrics. + MetricRule { + consumer: CandidateForMemoGroupId, + input: usize, + producer_metric: AccuracyMetric, + consumer_carries: Vec, + }, + /// The candidate never had a consumer and is not a query output. A + /// candidate whose consumers were eliminated is traced to their causes. + NoConsumer, +} +``` + +**Example: ratio of quantiles.** `quantile_over_time(0.99, m[5m]) / quantile_over_time(0.5, m[5m])` with relative error 0.05. + +| Group | Expression | Candidates | +|---|---|---| +| G1 | `Scan m` | | +| G2 | `TimeRange(5m, G1)` | | +| G3 | `SummaryAgg[KLL](G2)` | #0 … #6: k = 132 … 2863 | +| G3′ | `SummaryAgg[DDSketch](G2)` | #0 … #6: alpha = 0.1, 0.05, 0.02, 0.01, 0.005, 0.002, 0.001 | +| G4 | p99 | #0 exact; #1 `SummaryEstimate[q=0.99](G3)`; #2 `SummaryEstimate[q=0.99](G3′)` | +| G5 | p50 | #0 exact; #1 `SummaryEstimate[q=0.5](G3)`; #2 `SummaryEstimate[q=0.5](G3′)` | +| G6 | `Divide(G4, G5)` | #0, the query output | + +The Divide carries only relative error. Its inputs' cap is 0.05 / 1.111 = 0.045 (Section 4), so the DDSketch readouts' floor is alpha ≤ 0.02. Pass 2 records: + +``` +G4#1 SummaryEstimate[q=0.99](G3) + G4#1 → G6#0 input 0: MetricRule { producer_metric: Rank, consumer_carries: [Relative] } +G5#1 SummaryEstimate[q=0.5](G3) + G5#1 → G6#0 input 1: MetricRule { producer_metric: Rank, consumer_carries: [Relative] } +G3#0 SummaryAgg[KLL] k=132 (and likewise G3#1 … G3#6) + G3#0 → G4#1 → G6#0 input 0: MetricRule { producer_metric: Rank, consumer_carries: [Relative] } + G3#0 → G5#1 → G6#0 input 1: MetricRule { producer_metric: Rank, consumer_carries: [Relative] } +G3′#0 SummaryAgg[DDSketch] alpha=0.1 (and likewise G3′#1, alpha=0.05) + G3′#0 → G4#2 input 0: BelowAccuracyFloorRequirement { floor: alpha 0.02 } + G3′#0 → G5#2 input 0: BelowAccuracyFloorRequirement { floor: alpha 0.02 } +``` + +The KLL builds' own edges satisfy their consumers; they are dead because their only consumers are, and the trace leads through those consumers to the metric rule. The exact candidates and the DDSketch builds with alpha ≤ 0.02 survive. + +#### Chain groups + +A state group whose expression is `SummaryAgg(TimeRange(W, TimeShift*(X)))` can be computed from a chain of tumbling windows over X: + +```rust +// The chain: one row per window, (window end, keys, state): a State with a +// time index. No range and no window count: how much it retains is h_v. +SummaryTumblingWindowChain { child: X, family, input, reduction, grouping, filter, + tumbling_window_width_ms: u64 } +// Merges every input state, or regroups them; no window-selection field. +SummaryMerge { children, group_by } +``` + +`TimeRange` and `TimeShift` accept a time-indexed state: over a chain, `TimeRange(W)` at evaluation time t selects the windows lying fully inside [t − W, t), and `TimeShift(o)` moves that interval to [t − o − W, t − o). This needs w to divide W and o, and t to lie on the grid; the widths below and `delivers_on` (Section 2) ensure both. The `TimeRange` and `TimeShift` groups over a chain have only their own expression as a candidate and are computed within one evaluation; only the chain is materialized across evaluations. + +**The rule, for each width w of a state group S:** +1. Walk down from S: `SummaryAgg` → `TimeRange(W)` → zero or more `TimeShift`s → X. +2. `find_or_add_group(SummaryTumblingWindowChain[w](X))` → $G_w$. The shifts are stripped, so queries with different ranges and offsets over the same X get the same $G_w$. +3. Re-wrap the stripped shifts and then the `TimeRange` around $G_w$, in their original order, each with `find_or_add_group`. +4. `add_candidate(S, SummaryMerge(…))` over the outermost re-wrapped group. Its coverage equals S's (Section 3, Stage 0), so the candidate is accepted. + +There is one chain per width, shared by every state group over the same X and family. + +**Widths.** Each query q that reaches S (step 3) and repeats with a known evaluation grid $g_q$ (`evaluation_grid_ms` is `Some`) gives the width $\gcd(g_q, W, o_1, \ldots)$, with the offsets $o_j$ on its path; S's widths are the gcd closure of these. Queries without a known grid give none, so a group read only by such queries gets no chain. `@` modifiers: +- `@ end()` reads [t − W, t) relative to the end of the query range, as no `@` does: it is stripped and re-wrapped like a shift and adds no offset. +- `@ start()` is an offset of the query range's duration D and enters the gcd as one. +- `@ T` (absolute) contributes no widths: S gets a merge only over chains that other queries' widths created, and only for widths that divide T. Whether reusing the chain is cheaper than computing the result once and materializing it is Stage 3's choice; its retention need is in Section 5. + +**Example: three queries share one chain.** A = `sum_over_time(m[5m])`, B = `sum_over_time(m[5m] offset 1h)`, C = `sum_over_time(m[1h])`, each `FixedIntervalAt { 60 s, phase 0 }`. The widths are gcd(60 s, 5m) = gcd(60 s, 5m, 1h) = gcd(60 s, 1h) = 60 s. + +| Group | Expression | Candidates | +|---|---|---| +| G1 | `Scan m` | | +| G3a | `SummaryAgg[Sum](TimeRange(5m, G1))` | build; **`SummaryMerge(Ga)`** | +| G3b | `SummaryAgg[Sum](TimeRange(5m, TimeShift(1h, G1)))` | build; **`SummaryMerge(Gb)`** | +| G3c | `SummaryAgg[Sum](TimeRange(1h, G1))` | build; **`SummaryMerge(Gc)`** | +| **Gw** | `SummaryTumblingWindowChain[Sum, w = 60 s](G1)` | one build per θ | +| Ga | `TimeRange(5m, Gw)` | its own expression | +| Gs | `TimeShift(1h, Gw)` | its own expression | +| Gb | `TimeRange(5m, Gs)` | its own expression | +| Gc | `TimeRange(1h, Gw)` | its own expression | + +At t = 12:00, Ga selects the 5 windows [11:55, 11:56) … [11:59, 12:00), Gb the 5 windows [10:55, 10:56) … [10:59, 11:00), and Gc 60 windows. Each sample updates one chain instead of three, and the chain retains max(5m, 1h + 5m, 1h) = 65m, 65 windows (Section 5). + +**Example: the rule step by step, for B.** Before the rule: + +| Group | Expression | +|---|---| +| G1 | `Scan m` | +| G2 | `TimeShift(1h, G1)` | +| G3 | `TimeRange(5m, G2)` | +| G4 | `SummaryAgg[Sum](G3)`: B's state group, one build candidate | + +1. Walk down from G4: `SummaryAgg` reads G3; G3 is `TimeRange(W = 5m)` over G2; G2 is `TimeShift(1h)` over G1; G1 is not a shift. So W = 5m, the shifts are [`TimeShift(1h)`], X = G1. +2. `find_or_add_group(SummaryTumblingWindowChain[Sum, w = 60 s](G1))` → Gw. A and C get the same Gw. +3. `find_or_add_group(TimeShift(1h, Gw))` → Gs; `find_or_add_group(TimeRange(5m, Gs))` → Gb. +4. `add_candidate(G4, SummaryMerge(Gb))`. + +After the rule, G4 has two candidates, the build and `SummaryMerge(Gb)`; G2 and G3 stay. With widths {30 s, 60 s}, steps 2–4 run once per width, giving two chains and two merge candidates in G4; Stage 3 chooses. + +#### Logical edges ```rust fn apply(memo: &mut Memo) -> Result<(), Pass2RuleError>; @@ -572,16 +805,13 @@ pub struct LogicalEdge { pub input_index_of_consumer: usize, pub satisfies_logical_requirement: bool, } - -pub struct LogicalASAPMemoCandidates { - pub memo: Memo, - pub edges: Vec, -} ``` +An edge satisfies the logical requirement when the producer meets the input's floor and the consumer's error contract carries the producer's metric (Section 4). + ### Stage 2: physical alternatives -`enumerate_candidate_configurations` in `crates/physical-asap-optimization/src/catalog/configurations.rs` expands each logical candidate using the dimensions and checks in Section 2. It creates no new logical candidates. Each logical edge is expanded between the relevant physical alternatives. +`enumerate_candidate_configurations` in `crates/physical-asap-optimization/src/catalog/configurations.rs` expands each logical candidate that survived Pass 2 using the dimensions and checks in Section 2; it offers ℓ = AcrossEvaluations only to chain candidates, and records every other AcrossEvaluations alternative as pruned with `PruneReason::AcrossEvaluationsOnlyForChains`. It creates no new logical candidates. Each logical edge is expanded between the relevant physical alternatives. ```rust // crates/types/src/physical/properties.rs @@ -617,9 +847,9 @@ impl PhysicalProperties { impl PhysicalCandidate { /// Whether a consumer running in `context` can read this producer - /// (Section 2's alignment table): for a windowed AcrossEvaluations - /// producer, w (read from its window layout) divides every evaluation - /// time of the context; true for ℓ = None, and for OneEvaluation in the + /// (Section 2's alignment table): for a chain producer with + /// ℓ = AcrossEvaluations, w (the chain operator's width) divides every + /// evaluation time of the context (`evaluation_grid_ms`); true for ℓ = None, and for OneEvaluation in the /// same context. pub fn delivers_on(&self, memo: &Memo, context: &ExecutionContext) -> bool; } @@ -669,7 +899,7 @@ pub struct PhysicalCandidateKey { } ``` -`PhysicalCandidateId` indexes an emitted candidate. `PhysicalCandidateKey` describes an alternative rejected before it received an ID, allowing diagnostics to explain pruning. The admissible-edge accessor returns edge IDs, not the unfiltered producer set from Section 1. `delivers_on` takes a candidate rather than bare properties because a tumbling grid's width lives in the logical window layout. +`PhysicalCandidateId` indexes an emitted candidate. `PhysicalCandidateKey` describes an alternative rejected before it received an ID, allowing diagnostics to explain pruning. The admissible-edge accessor returns edge IDs, not the unfiltered producer set from Section 1. `delivers_on` takes a candidate and the memo rather than bare properties because a tumbling grid's width is an argument of the chain operator in the candidate's expression. Migration targets include replacing `Materialization` and duplicated physical fields with checked properties, and replacing `ReaderConstraint::{IncompatibleOutputs, AllowedOutputs}` with compatibility and alignment checks. `ReaderConstraint::{MinimumQueryReaders, HighestSharedQueryMaterialization}` are deleted too; see Section 5. @@ -758,12 +988,18 @@ $$ | Operation and metric | Applicable bound or condition | |---|---| -| Exact pass-through | Preserves the input guarantee only when it preserves the metric's semantics | +| Exact pass-through (column `Project`, `Sort`, `TimeRange`, `TimeShift`, `Concat`, readouts, finalize, `SummaryMerge`) | Preserves the input guarantee only when it preserves the metric's semantics; β = 1 in any metric | +| `Filter` whose predicate reads only columns copied from a source (lineage `SourceColumn`: labels and keys, which are exact) | β = 1 in any metric: it selects whole rows and never compares an approximate value | +| abs, clamp, clamp_min, clamp_max, negation | Absolute error, β = 1 (1-Lipschitz) | | Sum/difference, absolute error | Sum of the operands' absolute error bounds | | Sum over a known number of values | Sum of their bounds; a common per-value bound can be multiplied by that count | +| min / max aggregate | Absolute or relative error, β = 1 | +| avg aggregate, absolute error | β = 1 | | Ratio, relative error | Requires nonzero true denominator and denominator relative error below one; bound given below | +| Product, relative error | (1 + b) for the left operand and 1 for the right, whose relative error is capped at b: $(1+x)(1+y)-1\le(1+b)x+y$ when $y\le b$ | +| sum / avg, relative error | Holds when the values are known to be nonnegative; not offered until that sign information is an input of the error model | -A filter does not automatically preserve every accuracy guarantee. Group counts may need metadata beyond the expression syntax. Each factor must follow from the operator's error contract. +A constant contract is unconditional: it needs no row count or sign. A filter does not automatically preserve every accuracy guarantee; a `Filter` that compares an approximate value (an alert threshold such as `quantile_over_time(…) > 0.5`) has no contract. Group counts may need metadata beyond the expression syntax. Each factor must follow from the operator's error contract. **Metric rule.** Every local guarantee states its metric: absolute (in value units, or a multiple of a stream norm), relative, or rank. A factor $\beta_{c,i}$ is defined for one input metric and one output metric. An input whose metric the consumer's error contract cannot convert gives no edge ($\alpha^{\log} = 0$); the conversion is never assumed. For example, a relative-error ratio cannot read a KLL quantile (rank error) unless a declared distribution assumption supplies the conversion. @@ -774,7 +1010,7 @@ $$ \frac{\varepsilon_X+\varepsilon_Y}{1-\varepsilon_Y}. $$ -Choosing a fixed validity cap $b<1$ for the denominator gives a linear upper bound within that region: +Choosing a fixed validity cap $b<1$ for the denominator gives a linear upper bound within that region. The design fixes $b = 0.1$, for a ratio's denominator and a product's right operand alike: 0.1 is the weakest ε in every family's list, so the cap itself never excludes a listed configuration and only raises β by about 11%: $$ \varepsilon_Y\le b @@ -782,9 +1018,9 @@ $$ \varepsilon_{X/Y}\le\frac{1}{1-b}(\varepsilon_X+\varepsilon_Y). $$ -Both local factors are then `1 / (1 - b)`. This is a sufficient bound, not an exact equality or a claim that all feasible nonlinear plans remain represented. +Both local factors are then `1 / (1 - b)` = 1.111. This is a sufficient bound, not an exact equality or a claim that all feasible nonlinear plans remain represented. -Interface (approved 2026-11-05): +Interface: ```rust pub struct ErrorPropagation { @@ -796,13 +1032,17 @@ pub struct ErrorPropagation { } impl LogicalRequirements { + /// g_c's linear bound for errors in `metric`, each input's and the + /// output's. `None` when the consumer has no error contract for `metric`: + /// an input whose error is in `metric` then gives no edge (metric rule). pub fn error_propagation( consumer: &Operator, - ) -> ErrorPropagation; + metric: AccuracyMetric, + ) -> Option; } ``` -`factor_per_input` holds the local β factors and `validity_cap_per_input` the per-input caps, aligned by input index. An operator without an error contract is reported as unsupported; no factor is guessed. +`factor_per_input` holds the local β factors and `validity_cap_per_input` the per-input caps, aligned by input index. An operator without an error contract for a metric returns `None`; no factor is guessed. Every contract keeps the metric: the input's and the output's are the same. For the ratio query of Section 3, `error_propagation(Divide, Relative)` is `Some { factor_per_input: [1.111, 1.111], validity_cap_per_input: [None, Some(0.1)] }`, and the absolute and rank calls return `None`. The key-only `Filter` contract reads the input group's column lineage. ### Query composition and root checks @@ -860,7 +1100,7 @@ $$ A zero factor gives no pruning restriction for that query. If a factor depends on the eventual plan, use a proven lower bound over all feasible downstream alternatives for this pruning test, or defer pruning. Using an arbitrary path's larger factor could discard a feasible plan. The maximum over queries protects alternatives that can serve at least one query. -Compute the parameter floor from this cap only under the ordering condition in Section 2. Validity information is available when the consumer's error model is established; query-sensitive pruning follows logical generation and reachability analysis. Root checks still couple all contributing errors and determine feasibility. +Compute the parameter floor from this cap only under the ordering condition in Section 2. `floor_for_cap` in `asap-accuracy` maps a cap to the weakest listed configuration whose ε meets it, for a family and statistic; Pass 2 takes the strongest of it and any direct need such as heap_size ≥ k. For example, with relative error 0.05 at a ratio, each operand's cap is 0.05 / 1.111 = 0.045 and the denominator's is min(0.1, 0.045) = 0.045, so DDSketch's floor is alpha ≤ 0.02. Validity information is available when the consumer's error model is established; query-sensitive pruning follows logical generation and reachability analysis. Root checks still couple all contributing errors and determine feasibility. ## 5. Execution contexts and other selection extensions — #1207 @@ -868,7 +1108,7 @@ These extensions go beyond constraints (A)–(I) and the formulation's fixed-cos ### Execution identity -An AcrossEvaluations candidate has its own computation: each window is built when it ends. A candidate with ℓ = None runs inside an execution that needs it. For example, a finalize candidate used by queries A (every 10 s) and C (every 15 s) may execute in both evaluation contexts even though Stage 2 enumerates it only once. A OneEvaluation candidate also runs inside one evaluation context κ, and that evaluation's consumers share the result. +An AcrossEvaluations candidate, always a chain, has its own computation: each window is built when it ends. A candidate with ℓ = None runs inside an execution that needs it. For example, a finalize candidate used by queries A (every 10 s) and C (every 15 s) may execute in both evaluation contexts even though Stage 2 enumerates it only once. A OneEvaluation candidate also runs inside one evaluation context κ, and that evaluation's consumers share the result. A **context** identifies a class of executions with a common owner and schedule across the accounting horizon. It is not necessarily one timestamp. An evaluation context reuses the workload's own description of when queries run, their `QueryRecurrence` and `RangeEvaluation`: queries with equal ones are evaluated together and share its OneEvaluation results. Queries A and B of the running example, both every 10 s at phase 0, form one context; a one-off query forms the context of its `OneTime` recurrence. `QueryRecurrence` holds an `f64` (`EstimatedRate`) and cannot be hashed, so contexts are deduplicated by equality and `ExecutionContextId` indexes the list of distinct contexts. @@ -885,7 +1125,7 @@ pub enum ExecutionContext { } ``` -A chain of ℓ = None and OneEvaluation candidates inherits its consuming context. Encountering an AcrossEvaluations producer switches to that producer's own computation context when traversing its inputs. The context identifies where the operation runs; `delivers_on` with the context decides whether a windowed producer can supply it. +A chain of ℓ = None and OneEvaluation candidates inherits its consuming context. Encountering an AcrossEvaluations producer switches to that producer's own computation context when traversing its inputs. The context identifies where the operation runs; `delivers_on` with the context decides whether a chain producer can supply it. | Extended variable | Meaning | |---|---| @@ -904,12 +1144,12 @@ To avoid confusing per-execution costs with the baseline's horizon-wide cost, us | Symbol | Definition | |---|---| -| $W_v^{\mathrm{mat}}$ | Fixed computation and access cost over the horizon for an AcrossEvaluations candidate, one build per window, excluding variable retention | +| $W_v^{\mathrm{mat}}$ | Fixed computation and access cost over the horizon for an AcrossEvaluations candidate (a chain), one build per window, excluding variable retention | | $c_{\kappa,v}$ | Cost of one execution in the specified context of a candidate with ℓ = None or OneEvaluation, including a OneEvaluation result's storage within that evaluation | | $\rho_\kappa$ | Fixed number or expected number of context executions over the common horizon, from the existing evaluation counting in `crates/cost/src/execution_demand.rs` (a one-off query's is its `invocations`) | | $h_v$ | Retention amount in a declared unit, such as retained windows | | $\pi_v$ | Horizon-wide retention cost per unit of retained amount | -| $\operatorname{need}_{\kappa,v,c,i}$ | Retention amount producer v needs to serve this consumer input in this context: the time horizon the queries served by that context read, e.g. a 5m range over 10 s tumbling windows needs 30 windows. Query latency is ignored for now (decided 2026-11-06) | +| $\operatorname{need}_{\kappa,v,c,i}$ | Retention amount producer v needs to serve this consumer input in this context: the time horizon the queries served by that context read, e.g. a 5m range over 10 s tumbling windows needs 30 windows. Query latency is ignored for now | A proposed linear objective is: @@ -927,7 +1167,7 @@ $$ h_v\ge\operatorname{need}_{\kappa,v,c,i}\,y_{\kappa,v,c,i}. $$ -The need is fixed per edge from the time horizon of the queries it serves, so constraint (R) is linear. It ignores query latency: a result is assumed read as soon as its evaluation starts, so a slow consumer that reads later does not extend the horizon. Modeling that needs the latency design, which is out of scope (Section 5). The retention unit (windows, or seconds of data) and an upper bound on $h_v$ are fixed per candidate when Stage 2 builds it. +The need is fixed per edge from the time horizon of the queries it serves, so constraint (R) is linear. For a chain it is the reading path's offset plus range: in the example of Section 3, the edge to A's path needs 5m, B's 1h + 5m and C's 1h, so $h_v$ for the shared 60 s chain is at least 65m, 65 windows. `@ end()` needs the range; `@ start()` needs the query range's duration plus the range. An `@ T` path needs the end of the horizon − T + range; when the horizon is unbounded the need is too, and that merge candidate is not offered. It ignores query latency: a result is assumed read as soon as its evaluation starts, so a slow consumer that reads later does not extend the horizon. Modeling that needs the latency design, which is out of scope (Section 5). The retention unit (windows, or seconds of data) and an upper bound on $h_v$ are fixed per candidate when Stage 2 builds it. ### No consumer-count policies From 8066235b0179dd925bb9aee9ec6a08c980fc9ba0 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sat, 7 Nov 2026 17:00:47 +0000 Subject: [PATCH 42/67] docs(concepts): no executor-support check, ingestion-time counter and filtered builds, Stage 2 interface Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 87 +++++++++++++++---- 1 file changed, 71 insertions(+), 16 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 330b55cf4..324b084f8 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -164,7 +164,7 @@ flowchart TD G --> R2["e_v: readout of KLL state"] G --> R3["e_v: readout of DDSketch state"] G --> R5["e_v: SummaryMerge over TimeRange
of a chain of width w
(state groups; widths from the
gcd closure of grids, ranges, offsets)"] - G --> R4["e_v: other operators
(prune: executor support)"] + G --> R4["e_v: other operators"] R2 --> T1["θ: k = 200"] R2 --> T2["θ: k = 345"] @@ -175,7 +175,7 @@ flowchart TD P1 --> L1["ℓ = None"] P1 --> L2["ℓ = OneEvaluation"] - P1 --> L3["ℓ = AcrossEvaluations
(prune: not a chain; deployment
materializes no query-time results)"] + P1 --> L3["ℓ = AcrossEvaluations
(offered only to chains; prune: deployment
retains no query-time results)"] P2 --> L4["ℓ = None"] P2 --> L5["ℓ = AcrossEvaluations"] P2 -. invalid .-> L6["ℓ = OneEvaluation"] @@ -183,10 +183,12 @@ flowchart TD | Level | Decides | Stage | Values offered | Pruned when | |---|---|---|---|---| -| 1 | expression $e_v$, including a merge over a chain of width $w$ | Pass 1 and Pass 2 rules | each way to compute the group; chain widths from the gcd closure of the reaching queries' grids, ranges and offsets | the executor cannot build the family; no reaching query has a known grid | +| 1 | expression $e_v$, including a merge over a chain of width $w$ | Pass 1 and Pass 2 rules | each way to compute the group; chain widths from the gcd closure of the reaching queries' grids, ranges and offsets | no reaching query has a known grid (no chain is offered) | | 2 | $\boldsymbol\theta_v$ | Pass 1 | the family's predefined list | the pruning cap (Section 4); dead-candidate elimination (Section 3) | -| 3 | $\tau_v$ | Stage 2 | QueryTime, IngestionTime | the source has no ingestion pipeline | -| 4 | $\ell_v$ | Stage 2 | None, OneEvaluation, AcrossEvaluations | invalid with $\tau_v$; AcrossEvaluations on a candidate that is not a chain; the deployment materializes no query-time results | +| 3 | $\tau_v$ | Stage 2 | QueryTime, IngestionTime; IngestionTime for a counter state only when the source's out-of-order bound is known (Stage 2) | the source has no ingestion pipeline; QueryTime over raw rows the deployment does not retain | +| 4 | $\ell_v$ | Stage 2 | None, OneEvaluation, AcrossEvaluations; AcrossEvaluations only for chain candidates, and Disk only where the deployment offers it | invalid with $\tau_v$; the deployment retains no query-time results | + +The planner does not check whether an executor implements an operator or family: every candidate the rules produce is offered, and an executor that lacks one is extended. A value outside a dimension's declared value set (AcrossEvaluations on a candidate that is not a chain, Disk where the deployment has no disk storage) is not offered, so it is not recorded as pruned. The tree does not decide which producer each consumer input reads ($y$), in which context a candidate with ℓ = None or OneEvaluation runs (Section 5), or how much an AcrossEvaluations candidate retains ($h_v$). Those are Stage 3 choices. @@ -195,7 +197,7 @@ The tree does not decide which producer each consumer input reads ($y$), in whic **The order changes how many nodes enumeration visits:** $\sum_d \prod_{d' \le d} \lvert \text{values left at } d' \rvert$. Dimensions that prune much go early; dimensions that multiply without pruning go late. The order above follows four rules: 1. **Logical before physical.** Stage 1 hash-conses groups and computes $\alpha^{\log}$ once, before Stage 2 multiplies each logical candidate by its physical alternatives. -2. **Strong prunes early.** Executor support and the pruning cap act before physical properties multiply the tree. +2. **Strong prunes early.** The pruning cap and dead-candidate elimination act before physical properties multiply the tree. 3. **Dependent dimensions adjacent.** $\tau \to \ell$ rejects the invalid combination at the first level that can detect it. 4. **Deployment-wide prunes as early as possible.** Without an ingestion pipeline the IngestionTime subtree is pruned once per candidate; moving $\tau$ above $\boldsymbol\theta$ would prune it once per group, with the same leaves. @@ -811,7 +813,54 @@ An edge satisfies the logical requirement when the producer meets the input's fl ### Stage 2: physical alternatives -`enumerate_candidate_configurations` in `crates/physical-asap-optimization/src/catalog/configurations.rs` expands each logical candidate that survived Pass 2 using the dimensions and checks in Section 2; it offers ℓ = AcrossEvaluations only to chain candidates, and records every other AcrossEvaluations alternative as pruned with `PruneReason::AcrossEvaluationsOnlyForChains`. It creates no new logical candidates. Each logical edge is expanded between the relevant physical alternatives. +`enumerate_candidate_configurations` in `crates/physical-asap-optimization/src/catalog/configurations.rs` expands each logical candidate that survived Pass 2 using the dimensions and checks in Section 2. It offers ℓ = AcrossEvaluations only to chain candidates; the other candidates never get that value, and nothing is recorded for them. It creates no new logical candidates. Each logical edge is expanded between the relevant physical alternatives. + +Stage 2 reads the logical candidates, the data workload (per-source arrival, below) and the deployment's capabilities: + +```rust +// crates/physical-asap-optimization/src/catalog/mod.rs +pub fn generate_physical_asap_memo_candidates( + logical: &LogicalASAPMemoCandidates, + data: &DataWorkload, + capabilities: &DeploymentCapabilities, +) -> Result; +``` + +#### Data arrival + +```rust +// crates/types/src/workload/mod.rs +pub enum DataArrival { + /// Every row is already stored; no build runs at ingestion time. + AtRest, + /// Rows keep arriving while the queries run. + Ingesting { rows_per_second: f64, max_out_of_order_ms: Option }, + /// Older rows are stored and new rows keep arriving. + Mixed { rows_per_second: f64, max_out_of_order_ms: Option }, + /// Not declared: no build runs at ingestion time. + Unknown, +} +``` + +`max_out_of_order_ms` bounds how late a row may arrive relative to the newest row of its series. + +**Counter states at ingestion time.** A counter state (`rate`, `increase`, `irate`) needs each series' samples in timestamp order to detect resets. It may still run at ingestion time: the ingestion data plane keeps a per-series reorder buffer that holds `max_out_of_order_ms` of rows and releases them in order. The buffer's memory, about rows per second × out-of-order bound × bytes per row, is part of that build's cost. For `rate(c_total[5m])` with 100,000 samples/s, a 2 s bound and 16 B per row, the buffer is about 100,000 × 2 × 16 B ≈ 3.2 MB. When `max_out_of_order_ms` is `None`, a PromQL source is taken to arrive in order (bound 0, no buffer); a SQL source gets no ingestion-time counter build, because SQL rows have no ordering guarantee. + +**Filtered builds at ingestion time.** A filtered build needs no extra rule. α^phys already requires an ingestion-time build's input to be produced at ingestion time; the build reads each input row with all its columns and evaluates the filter before updating the state. For SQL `sum(x) FILTER (WHERE region = 'eu')` over `Join(events, hosts)`, the build runs at ingestion time only when the join does; it then reads each joined row, checks `region`, and adds `x` for the rows that pass. + +#### Prune reasons + +Stage 2 records every alternative it rejects, with one of these reasons; Stage 3 adds its own. + +| Reason | Rejects | +|---|---| +| `NoIngestionPipeline` | IngestionTime when the source has no ingestion pipeline | +| `RawDataNotRetained` | QueryTime over raw rows the deployment does not retain | +| `QueryTimeResultsNotRetained` | QueryTime with ℓ = OneEvaluation or AcrossEvaluations when the deployment retains no query-time results | +| `InputHasNoBuildableProducer` | a candidate for which some input has no surviving producer that satisfies α^phys | +| `ReachesNoQueryOutput` | a candidate from which no admissible path leads to a query root | + +The planner does not check executor support, so no reason names it. ```rust // crates/types/src/physical/properties.rs @@ -876,10 +925,22 @@ pub struct PhysicalEdge { pub struct PhysicalASAPMemoCandidates { pub logical_candidates: LogicalASAPMemoCandidates, pub physical_candidates: Vec, - pub query_roots: Vec>, - pub pruned_candidates: Vec>, + pub pruned_candidates: Vec>, +} +/// A rejected alternative and why; the candidate names its group +/// through its logical candidate, so no separate group field. +pub struct PrunedCandidate { + pub candidate: C, + pub reason: PruneReason, } impl PhysicalASAPMemoCandidates { + /// Query roots are read from `logical_candidates`, not copied. + pub fn new( + logical_candidates: LogicalASAPMemoCandidates, + physical_candidates: Vec, + edges: Vec, + pruned_candidates: Vec>, + ) -> Self; pub fn edges(&self) -> impl Iterator; pub fn admissible_edges_into( &self, @@ -891,15 +952,9 @@ impl PhysicalASAPMemoCandidates { producer: PhysicalCandidateId, ) -> &[PhysicalEdgeId]; } - -// Identifies a rejected alternative by content: it has no allocated candidate ID. -pub struct PhysicalCandidateKey { - pub logical_candidate_id: CandidateForMemoGroupId, - pub physical_properties: PhysicalProperties, -} ``` -`PhysicalCandidateId` indexes an emitted candidate. `PhysicalCandidateKey` describes an alternative rejected before it received an ID, allowing diagnostics to explain pruning. The admissible-edge accessor returns edge IDs, not the unfiltered producer set from Section 1. `delivers_on` takes a candidate and the memo rather than bare properties because a tumbling grid's width is an argument of the chain operator in the candidate's expression. +`PhysicalCandidateId` indexes an emitted candidate. A pruned alternative is stored as its `PhysicalCandidate` value, which has no ID, so diagnostics can explain each prune. The admissible-edge accessor returns edge IDs, not the unfiltered producer set from Section 1. `delivers_on` takes a candidate and the memo rather than bare properties because a tumbling grid's width is an argument of the chain operator in the candidate's expression. Migration targets include replacing `Materialization` and duplicated physical fields with checked properties, and replacing `ReaderConstraint::{IncompatibleOutputs, AllowedOutputs}` with compatibility and alignment checks. `ReaderConstraint::{MinimumQueryReaders, HighestSharedQueryMaterialization}` are deleted too; see Section 5. From 5d08902ac418aa32ba943e437e55dba061d248d3 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sat, 7 Nov 2026 17:01:09 +0000 Subject: [PATCH 43/67] docs(concepts): Stage 2 entry returns no Result; current crate path Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../concepts/memo-groups-properties-requirements.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 324b084f8..e7d51c814 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -813,17 +813,17 @@ An edge satisfies the logical requirement when the producer meets the input's fl ### Stage 2: physical alternatives -`enumerate_candidate_configurations` in `crates/physical-asap-optimization/src/catalog/configurations.rs` expands each logical candidate that survived Pass 2 using the dimensions and checks in Section 2. It offers ℓ = AcrossEvaluations only to chain candidates; the other candidates never get that value, and nothing is recorded for them. It creates no new logical candidates. Each logical edge is expanded between the relevant physical alternatives. +`enumerate_candidate_configurations` in `crates/stage2-physical-optimization/src/catalog/configurations.rs` expands each logical candidate that survived Pass 2 using the dimensions and checks in Section 2. It offers ℓ = AcrossEvaluations only to chain candidates; the other candidates never get that value, and nothing is recorded for them. It creates no new logical candidates. Each logical edge is expanded between the relevant physical alternatives. Stage 2 reads the logical candidates, the data workload (per-source arrival, below) and the deployment's capabilities: ```rust -// crates/physical-asap-optimization/src/catalog/mod.rs +// crates/stage2-physical-optimization/src/catalog/mod.rs pub fn generate_physical_asap_memo_candidates( logical: &LogicalASAPMemoCandidates, data: &DataWorkload, capabilities: &DeploymentCapabilities, -) -> Result; +) -> PhysicalASAPMemoCandidates; // Stage 2 cannot fail ``` #### Data arrival From 1e7bbb55231eb0ce1ae334fbaa20c0ba53c69c32 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sat, 7 Nov 2026 20:50:14 +0000 Subject: [PATCH 44/67] docs(concepts): IntervalConstraintLimit replaces TimeBound, with examples Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 44 ++++++++++++------- 1 file changed, 29 insertions(+), 15 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index e7d51c814..1992180f9 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -379,25 +379,39 @@ pub struct SelectionBox { pub columns: BTreeMap, } -/// An endpoint of an interval on a timestamp column. -pub enum TimeBound { - /// An absolute time, in ms since the epoch. - At(i64), - /// An offset, in ms, from the evaluation time t. - FromEvaluationTime(i64), +/// A constraint on one column of a selection. +pub enum Constraint { + In(Vec), + NotIn(Vec), + /// Each end is `Included`, `Excluded` or `Unbounded`. + Interval { + lower: Bound, + upper: Bound, + }, +} + +/// An end of an interval. Two limits compare only within one variant. +pub enum IntervalConstraintLimit { + /// A constant: `Float64(10.0)` for `price > 10`; `Timestamp(ms since the + /// epoch)` for `@ T` or a SQL timestamp literal. + FixedValue(ScalarValue), + /// The evaluation time t plus this many ms (negative: before t); only on + /// the timestamp column. + EvaluationTimePlusMs(i64), } ``` -A repeatedly evaluated query covers different absolute times at every evaluation, so its group's coverage is written relative to the evaluation time t; an `@ T` modifier or a SQL timestamp filter is absolute. Both are intervals on the timestamp column, compared and lifted like any other column constraint. For a range evaluation over [t − D, t] (D is the query range's duration): +A repeatedly evaluated query covers different absolute times at every evaluation, so its group's coverage is written relative to the evaluation time t (`EvaluationTimePlusMs`); an `@ T` modifier or a SQL timestamp filter is absolute (`FixedValue(Timestamp)`). Both are intervals on the timestamp column, compared and lifted like any other column constraint; a value column such as `price` uses `FixedValue` ends. For a range evaluation over [t − D, t] (D is the query range's duration): -| Query fragment | Constraint on `m.ts` | -|---|---| -| `m[5m]` | [t − 5m, t): both ends `FromEvaluationTime` | -| `m[5m] offset 1h` | [t − 1h − 5m, t − 1h) | -| `m[5m] @ end()` | [t − 5m, t): the same as no `@` in an instant query | -| `m[5m] @ start()` | [t − D − 5m, t − D): an offset of D | -| `m[5m] @ 1700000000` | [T − 5m, T) with T = 1700000000 s: both ends `At` | -| SQL `WHERE ts >= '2026-11-01'` | [2026-11-01, ∞): `At` | +| Query fragment | `lower` | `upper` | +|---|---|---| +| `m[5m]` | `Included(EvaluationTimePlusMs(-300_000))` | `Excluded(EvaluationTimePlusMs(0))` | +| `m[5m] offset 1h` | `Included(EvaluationTimePlusMs(-3_900_000))` | `Excluded(EvaluationTimePlusMs(-3_600_000))` | +| `m[5m] @ end()` | as `m[5m]`: the same as no `@` in an instant query | | +| `m[5m] @ start()` | `Included(EvaluationTimePlusMs(-D - 300_000))` | `Excluded(EvaluationTimePlusMs(-D))`: an offset of D | +| `m[5m] @ 1700000000` | `Included(FixedValue(Timestamp(1_699_999_700_000)))` | `Excluded(FixedValue(Timestamp(1_700_000_000_000)))` | +| SQL `WHERE ts >= '2026-11-01'` | `Included(FixedValue(Timestamp(..)))` | `Unbounded` | +| SQL `WHERE price > 10` (column `price`) | `Excluded(FixedValue(Float64(10.0)))` | `Unbounded` | **Derivation, per operator** (from the input groups' `column_lineage`): From 37650ffa1c4e7454012d2b017f63838b6eda637a Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sun, 8 Nov 2026 14:59:22 +0000 Subject: [PATCH 45/67] =?UTF-8?q?docs(concepts):=20ranges=20are=20(t=20?= =?UTF-8?q?=E2=88=92=20r,=20t];=20@=20end()/@=20start()=20lifted=20only=20?= =?UTF-8?q?in=20instant=20queries;=20unified=20chain=20schema;=20UnivMon?= =?UTF-8?q?=20list=20product?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 56 ++++++++++--------- 1 file changed, 31 insertions(+), 25 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 1992180f9..4eb70df60 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -84,7 +84,7 @@ This is the single reference table for property dimensions. Stage descriptions b | Dimension | Values and meaning | Code reference or target | |---|---|---| -| Parameters | A predefined configuration list per family: KLL and KMV `k`; DDSketch `alpha`; HLL `precision`; CMS and CountSketch `width` and `depth`; heap variants also `heap_size`; UnivMon its sizing parameters. The lists are today's sizing over ε = 0.1 down to 0.001; KMV's and UnivMon's distinct-count entries stop at ε = 0.01. A heap variant combines each width × depth with heap sizes from the 1-2-5 series 10, 20, 50, …, 10 000: a top-k consumer's floor is heap_size ≥ k, and the smallest listed heap that meets it is the next step up, at most 2.5× too large. A sketch build's parameters must be in its family's list; exact operators have no sketch parameters. Hydra and the other families Pass 1 does not size have empty `parameter_list`s, so no build of theirs is a valid candidate until their error models exist (#1228). | `SketchParams` in `crates/types/src/ir/schema/sketch.rs`; predefined lists are the #1203 target | +| Parameters | A predefined configuration list per family: KLL and KMV `k`; DDSketch `alpha`; HLL `precision`; CMS and CountSketch `width` and `depth`; heap variants also `heap_size`; UnivMon its sizing parameters. The lists are today's sizing over ε = 0.1 down to 0.001; KMV's and UnivMon's distinct-count entries stop at ε = 0.01. UnivMon's list is the product of its 7 L2 column counts and its 4 distinct-count heap sizes, plus the baseline, so the per-parameter maximum of an L2 floor and a distinct-count floor is dominated by a listed entry. A heap variant combines each width × depth with heap sizes from the 1-2-5 series 10, 20, 50, …, 10 000: a top-k consumer's floor is heap_size ≥ k, and the smallest listed heap that meets it is the next step up, at most 2.5× too large. A sketch build's parameters must be in its family's list; exact operators have no sketch parameters. Hydra and the other families Pass 1 does not size have empty `parameter_list`s, so no build of theirs is a valid candidate until their error models exist (#1228). | `SketchParams` in `crates/types/src/ir/schema/sketch.rs`; predefined lists are the #1203 target | | Timing | `IngestionTime` or `QueryTime` | `ExecutionTiming` | | Lifetime | `None`: discard after the consuming execution; `OneEvaluation`: retain until consumers within that evaluation finish; `AcrossEvaluations`: retain across evaluations in the specified period. A non-`None` lifetime means materialized. | `Lifetime`; #1200 | | Storage medium — Extension | `Memory` or `Disk`, carried by a materialized lifetime | `StorageMedium` | @@ -116,17 +116,17 @@ QT means QueryTime; IT means IngestionTime. Alignment, data coverage, and retent **Alignment.** There is no physical schedule property. When a result can be read follows from its lifetime and, for a chain, its window width: - **None** and **OneEvaluation**: the result is computed inside one evaluation context κ (Section 5) and read there. A OneEvaluation result belongs to one context: queries with equal recurrence and range evaluation are evaluated together, and that evaluation's consumers share the result. When it runs is the context's schedule, and Stage 3 chooses it through $x_{\kappa,v}$, just as for ℓ = None. Sharing across differently scheduled queries through a gcd schedule is not offered: it is rarely beneficial and wastes runs. -- **AcrossEvaluations** is offered only to chain candidates, `SummaryTumblingWindowChain[w](X)`: the input is cut into tumbling windows [k·w, (k+1)·w), with the grid anchored at phase 0 (the epoch). Each window is built when it ends and can be read from then on. A `TimeRange` over the chain selects the windows that lie fully inside [t − range, t), so a consumer can read a range whose ends lie on window boundaries. With w = 10 s, a `[1m]` query evaluated every 30 s at phase 0 merges 6 windows; one evaluated at phase 15 s cannot, because its range ends inside a window; with that query in the workload, the gcd closure also offers w = 5 s, which both can read. Every other candidate, including the `TimeRange` and `TimeShift` groups over a chain, is computed within one evaluation. +- **AcrossEvaluations** is offered only to chain candidates, `SummaryTumblingWindowChain[w](X)`: the input is cut into tumbling windows (k·w, (k+1)·w], with the grid anchored at phase 0 (the epoch). Windows are left-open and right-closed like a PromQL range, which `m[r]` evaluated at t reads as (t − r, t]. Each window is built when it ends and can be read from then on. A `TimeRange` over the chain selects the windows that lie fully inside (t − range, t], so a consumer can read a range whose ends lie on window boundaries. With w = 10 s, a `[1m]` query evaluated every 30 s at phase 0 merges 6 windows; one evaluated at phase 15 s cannot, because its range ends inside a window; with that query in the workload, the gcd closure also offers w = 5 s, which both can read. Every other candidate, including the `TimeRange` and `TimeShift` groups over a chain, is computed within one evaluation. A grid phase is not needed: a phase is absorbed by the width, because width candidates come from the gcd closure of the queries' grids, ranges and offsets (Section 3). For example, phases 0 and 5 s with w = 10 s give w = 5 s. Alignment alone does not ensure sufficient retained history; that is $h_v$ (Section 5). -**Why a chain needs a declared phase.** Take `sum_over_time(m[5m])` evaluated every 60 s, and a 60 s chain with windows [12:00:00, 12:01:00), [12:01:00, 12:02:00), … +**Why a chain needs a declared phase.** Take `sum_over_time(m[5m])` evaluated every 60 s, and a 60 s chain with windows (12:00:00, 12:01:00], (12:01:00, 12:02:00], … | Case | Evaluation at | Data needed | Windows | |---|---|---|---| -| Phase 0 declared | 12:05:00 | [12:00:00, 12:05:00) | exactly the 5 windows [12:00, 12:01) … [12:04, 12:05): the merge is correct | -| No phase declared; the engine actually evaluates at second 17 | 12:05:17 | [12:00:17, 12:05:17) | [12:00, 12:01) holds only 43 s of the range and [12:05, 12:06) only 17 s: merging only full windows misses [12:00:17, 12:01:00) and [12:05:00, 12:05:17); adding the partial windows counts [12:00:00, 12:00:17) and [12:05:17, 12:06:00) too. Both answers are wrong | -| Phase 15 s declared | 12:05:15 | [12:00:15, 12:05:15) | the widths divide gcd(60 s, 5 m, 15 s) = 15 s; with w = 15 s the range is exactly 20 windows | +| Phase 0 declared | 12:05:00 | (12:00:00, 12:05:00] | exactly the 5 windows (12:00, 12:01] … (12:04, 12:05]: the merge is correct | +| No phase declared; the engine actually evaluates at second 17 | 12:05:17 | (12:00:17, 12:05:17] | (12:00, 12:01] holds only 43 s of the range and (12:05, 12:06] only 17 s: merging only full windows misses (12:00:17, 12:01:00] and (12:05:00, 12:05:17]; adding the partial windows counts (12:00:00, 12:00:17] and (12:05:17, 12:06:00] too. Both answers are wrong | +| Phase 15 s declared | 12:05:15 | (12:00:15, 12:05:15] | the widths divide gcd(60 s, 5 m, 15 s) = 15 s; with w = 15 s the range is exactly 20 windows | Prometheus staggers rule-group evaluations by a hash of the group, so a fixed interval without a declared phase does not mean phase 0. One rule decides both sides: `QueryRecurrence::evaluation_grid_ms` returns `None` for a `FixedInterval` without a phase. Pass 2 then offers that query no chain width, and `delivers_on` lets it read no chain. Workloads declare the phase with `FixedIntervalAt { interval, evaluation_phase }` to use chains. @@ -142,7 +142,7 @@ An evaluation context κ is described by the workload's own recurrence and range | `FixedInterval` without a phase, `OneTime { execute_at: None }`, `EstimatedRate`, `Unknown` | Never: the evaluation times are not known, and `evaluation_grid_ms` is `None` | | AcrossEvaluations on a candidate that is not a chain | Not offered: only chain candidates are materialized across evaluations | -The evaluation grid is `QueryRecurrence::evaluation_grid_ms(range_evaluation)` in `crates/types/src/workload/mod.rs`, which the workload entries, Pass 2's chain widths and `delivers_on` all use. One-off queries are supported: each is a context of its own recurrence, run `invocations` times. +The evaluation grid is `QueryRecurrence::evaluation_grid_ms(range_evaluation)` in `crates/types/src/workload/mod.rs`, which Pass 2's chain widths and `delivers_on` both use. One-off queries are supported: each is a context of its own recurrence, run `invocations` times. ### Invalid combinations @@ -337,7 +337,7 @@ A group is its expression: one operator, with its arguments, over input groups. Everything that was computed on operator-node trees is computed on groups. `find_or_add_group` derives `GroupProperties` once, when the group is created, with `derive_group_properties`: the rules that check operator nodes (schema derivation, input validation, merge check), applied to the expression over `MemoGroupAsInput`s, which read only the input groups' `expression` and `group_properties`, never a node tree: - `schema` and `result_kind` from the operator and its input groups' schemas and result kinds; a scalar subquery reads the referenced group's schema. -- `aggregate_coverage`, for a summary state or an exact `Aggregate` result: walking down from the SummaryAgg or Aggregate through the input groups' expressions while each is a Filter, a TimeRange, a TimeShift (with or without `@`) or a direct-column Project, lifting their predicates into the selection; a TimeRange or TimeShift becomes an interval on the timestamp column (below). The group below them is the definition's `input_group`. A Scan's predicates (PromQL label matchers, SQL filters, including absolute timestamp filters) are lifted too; the Scan with its residual predicates is then the last unlifted operator and `input_group` is `None`. The operators not lifted, each with its residual arguments, are `unlifted_operators`. A chain `SummaryTumblingWindowChain[w](X)` has the definition of a build over X and leaves the timestamp column unconstrained; a `TimeRange` or `TimeShift` over a chain constrains it as over raw rows. A merge's coverage comes from its inputs' stored coverage; a merge is valid only over inputs with equal definitions and disjoint selections, or over one time-indexed state. So the coverage of `SummaryMerge(TimeRange(TimeShift(Chain[w](X))))` equals that of `SummaryAgg(TimeRange(TimeShift(X)))`, and `add_candidate` accepts the merge in the original state group. +- `aggregate_coverage`, for a summary state or an exact `Aggregate` result: walking down from the SummaryAgg or Aggregate through the input groups' expressions while each is a Filter, a TimeRange, a TimeShift (without `@` or with `@ T`; `@ end()` and `@ start()` are not lifted, see below) or a direct-column Project, lifting their predicates into the selection; a TimeRange or TimeShift becomes an interval on the timestamp column (below). The group below them is the definition's `input_group`. A Scan's predicates (PromQL label matchers, SQL filters, including absolute timestamp filters) are lifted too; the Scan with its residual predicates is then the last unlifted operator and `input_group` is `None`. The operators not lifted, each with its residual arguments, are `unlifted_operators`. A chain `SummaryTumblingWindowChain[w](X)` has the definition of a build over X and leaves the timestamp column unconstrained; a `TimeRange` or `TimeShift` over a chain constrains it as over raw rows. A merge's coverage comes from its inputs' stored coverage; a merge is valid only over inputs with equal definitions and disjoint selections, or over one time-indexed state. So the coverage of `SummaryMerge(TimeRange(TimeShift(Chain[w](X))))` equals that of `SummaryAgg(TimeRange(TimeShift(X)))`, and `add_candidate` accepts the merge in the original state group. - The checks that read an input's operator (a merge over a time-indexed state reads a `TimeRange` over a chain; a selected build) read the input group's `expression`. - `shareable_when_equal`: a `LIMIT` is shareable over a `Sort` with the same partitions (the input group's `expression`); a `Dedup` on every column (the input group's schema). @@ -401,18 +401,20 @@ pub enum IntervalConstraintLimit { } ``` -A repeatedly evaluated query covers different absolute times at every evaluation, so its group's coverage is written relative to the evaluation time t (`EvaluationTimePlusMs`); an `@ T` modifier or a SQL timestamp filter is absolute (`FixedValue(Timestamp)`). Both are intervals on the timestamp column, compared and lifted like any other column constraint; a value column such as `price` uses `FixedValue` ends. For a range evaluation over [t − D, t] (D is the query range's duration): +A repeatedly evaluated query covers different absolute times at every evaluation, so its group's coverage is written relative to the evaluation time t (`EvaluationTimePlusMs`); an `@ T` modifier or a SQL timestamp filter is absolute (`FixedValue(Timestamp)`). Both are intervals on the timestamp column, compared and lifted like any other column constraint; a value column such as `price` uses `FixedValue` ends. A PromQL range `m[r]` evaluated at t reads the samples in (t − r, t]: left-open, right-closed. | Query fragment | `lower` | `upper` | |---|---|---| -| `m[5m]` | `Included(EvaluationTimePlusMs(-300_000))` | `Excluded(EvaluationTimePlusMs(0))` | -| `m[5m] offset 1h` | `Included(EvaluationTimePlusMs(-3_900_000))` | `Excluded(EvaluationTimePlusMs(-3_600_000))` | -| `m[5m] @ end()` | as `m[5m]`: the same as no `@` in an instant query | | -| `m[5m] @ start()` | `Included(EvaluationTimePlusMs(-D - 300_000))` | `Excluded(EvaluationTimePlusMs(-D))`: an offset of D | -| `m[5m] @ 1700000000` | `Included(FixedValue(Timestamp(1_699_999_700_000)))` | `Excluded(FixedValue(Timestamp(1_700_000_000_000)))` | +| `m[5m]` | `Excluded(EvaluationTimePlusMs(-300_000))` | `Included(EvaluationTimePlusMs(0))` | +| `m[5m] offset 1h` | `Excluded(EvaluationTimePlusMs(-3_900_000))` | `Included(EvaluationTimePlusMs(-3_600_000))` | +| `m[5m] @ end()` or `@ start()`, instant query | as `m[5m]`: both equal t, and the PromQL front end lowers them as no `@` | | +| `m[5m] @ end()` or `@ start()`, range query | not lifted: the `TimeShift` stays in `unlifted_operators` | | +| `m[5m] @ 1700000000` | `Excluded(FixedValue(Timestamp(1_699_999_700_000)))` | `Included(FixedValue(Timestamp(1_700_000_000_000)))` | | SQL `WHERE ts >= '2026-11-01'` | `Included(FixedValue(Timestamp(..)))` | `Unbounded` | | SQL `WHERE price > 10` (column `price`) | `Excluded(FixedValue(Float64(10.0)))` | `Unbounded` | +A range-evaluated query evaluates its expression at every step t of [start, end]. `@ end()` and `@ start()` pin a subtree to one end of that interval, so the subtree reads the same rows at every step while a plain `m[5m]` reads different rows at every step; the two are not equal coverage, and the `@` subtree is not shared. Evaluating `@` subtrees once per request and sharing them is future work (#1251). + **Derivation, per operator** (from the input groups' `column_lineage`): | Operator | Output column lineage | @@ -421,6 +423,7 @@ A repeatedly evaluated query covers different absolute times at every evaluation | `Filter`, `TimeRange`, `TimeShift`, `Sort`, `Limit`, `Dedup` | the input's, unchanged | | `Project` | a column item, renamed or not: that input column's lineage; an expression item: `Computed` | | `Aggregate`, `SummaryAgg`, readouts | group-by keys: the input's lineage of that key; measures, states and values: `Computed` | +| `SummaryTumblingWindowChain` | `window_end` (column 0): `Computed`; keys: the input's lineage of that key; state: `Computed` | | `Join` | the left input's columns, then the right input's | | `SetOp`, `Concat` | per column: the inputs' lineage if they all agree, else `Computed` | | `BinaryOp` (vector match) | label columns: the matched side's lineage; the value: `Computed` | @@ -553,8 +556,8 @@ pub struct LogicalProperties { impl LogicalProperties { /// Rejects parameters missing on a sketch build (a chain included), - /// present on an exact operator, of another family or not in the - /// family's list. + /// present on an exact operator, or not in the family's list (which + /// also rejects another family's parameters). pub fn new( expression: &CandidateExpression, parameters: Option, @@ -641,10 +644,10 @@ A Scan has no aggregation, so for scans every column is a key and the rollup ste |---|---|---|---|---| | G1 | `Scan latency` | | | | | G2 | `TimeRange(5m, G1)` | | | | -| G3 | `SummaryAgg[KLL, by job, host](G2)` | D, `ts` ∈ [t − 5m, t) | job, host | builds | +| G3 | `SummaryAgg[KLL, by job, host](G2)` | D, `ts` ∈ (t − 5m, t] | job, host | builds | | G4 | `Scan latency{job="api"}` | | | | | G5 | `TimeRange(5m, G4)` | | | | -| G6 | `SummaryAgg[KLL, by host](G5)` | D, `ts` ∈ [t − 5m, t), `job` ∈ {api} | host | builds; **`SummaryMerge(by host)(G7)`** | +| G6 | `SummaryAgg[KLL, by host](G5)` | D, `ts` ∈ (t − 5m, t], `job` ∈ {api} | host | builds; **`SummaryMerge(by host)(G7)`** | | G7 | `Filter(job = "api", G3)` | | | its own expression; output is states | The definitions are equal, the narrowing `job ∈ {api}` is on a key of G3, {host} ⊆ {job, host}, and KLL states merge. G3 is already the finest group of the cluster {G3, G6}, so $G^*$ = G3. A query `quantile by (host)(…latency{status="500"}[5m])` gets no such candidate: `status` is not a key of G3, and every state already mixes status 200 and 500 samples. @@ -755,15 +758,18 @@ The KLL builds' own edges satisfy their consumers; they are dead because their o A state group whose expression is `SummaryAgg(TimeRange(W, TimeShift*(X)))` can be computed from a chain of tumbling windows over X: ```rust -// The chain: one row per window, (window end, keys, state): a State with a -// time index. No range and no window count: how much it retains is h_v. +// The chain: one row per window and key, (window_end, keys…, state): a State +// with a time index. Column 0, `window_end`, is the time index, computed +// (lineage `Computed`); the build's own time column is dropped; the unique +// key is (window_end, keys). No range and no window count: how much it +// retains is h_v. SummaryTumblingWindowChain { child: X, family, input, reduction, grouping, filter, tumbling_window_width_ms: u64 } // Merges every input state, or regroups them; no window-selection field. SummaryMerge { children, group_by } ``` -`TimeRange` and `TimeShift` accept a time-indexed state: over a chain, `TimeRange(W)` at evaluation time t selects the windows lying fully inside [t − W, t), and `TimeShift(o)` moves that interval to [t − o − W, t − o). This needs w to divide W and o, and t to lie on the grid; the widths below and `delivers_on` (Section 2) ensure both. The `TimeRange` and `TimeShift` groups over a chain have only their own expression as a candidate and are computed within one evaluation; only the chain is materialized across evaluations. +`TimeRange` and `TimeShift` accept a time-indexed state: over a chain, `TimeRange(W)` at evaluation time t selects the windows lying fully inside (t − W, t], and `TimeShift(o)` moves that interval to (t − o − W, t − o]. A `TimeRange` over a chain must be a range (`TimeRangeKind::Range`); an instant one is a validation error. This needs w to divide W and o, and t to lie on the grid; the widths below and `delivers_on` (Section 2) ensure both. The `TimeRange` and `TimeShift` groups over a chain have only their own expression as a candidate and are computed within one evaluation; only the chain is materialized across evaluations. **The rule, for each width w of a state group S:** 1. Walk down from S: `SummaryAgg` → `TimeRange(W)` → zero or more `TimeShift`s → X. @@ -774,8 +780,8 @@ SummaryMerge { children, group_by } There is one chain per width, shared by every state group over the same X and family. **Widths.** Each query q that reaches S (step 3) and repeats with a known evaluation grid $g_q$ (`evaluation_grid_ms` is `Some`) gives the width $\gcd(g_q, W, o_1, \ldots)$, with the offsets $o_j$ on its path; S's widths are the gcd closure of these. Queries without a known grid give none, so a group read only by such queries gets no chain. `@` modifiers: -- `@ end()` reads [t − W, t) relative to the end of the query range, as no `@` does: it is stripped and re-wrapped like a shift and adds no offset. -- `@ start()` is an offset of the query range's duration D and enters the gcd as one. +- `@ end()` and `@ start()` in an instant query equal t; the front end lowers them as no `@`, so they need nothing here. +- `@ end()` and `@ start()` in a range-evaluated query are not lifted (Section 3), so the walk stops at their `TimeShift` and the state group gets no chain. - `@ T` (absolute) contributes no widths: S gets a merge only over chains that other queries' widths created, and only for widths that divide T. Whether reusing the chain is cheaper than computing the result once and materializing it is Stage 3's choice; its retention need is in Section 5. **Example: three queries share one chain.** A = `sum_over_time(m[5m])`, B = `sum_over_time(m[5m] offset 1h)`, C = `sum_over_time(m[1h])`, each `FixedIntervalAt { 60 s, phase 0 }`. The widths are gcd(60 s, 5m) = gcd(60 s, 5m, 1h) = gcd(60 s, 1h) = 60 s. @@ -792,7 +798,7 @@ There is one chain per width, shared by every state group over the same X and fa | Gb | `TimeRange(5m, Gs)` | its own expression | | Gc | `TimeRange(1h, Gw)` | its own expression | -At t = 12:00, Ga selects the 5 windows [11:55, 11:56) … [11:59, 12:00), Gb the 5 windows [10:55, 10:56) … [10:59, 11:00), and Gc 60 windows. Each sample updates one chain instead of three, and the chain retains max(5m, 1h + 5m, 1h) = 65m, 65 windows (Section 5). +At t = 12:00, Ga selects the 5 windows (11:55, 11:56] … (11:59, 12:00], Gb the 5 windows (10:55, 10:56] … (10:59, 11:00], and Gc 60 windows. Each sample updates one chain instead of three, and the chain retains max(5m, 1h + 5m, 1h) = 65m, 65 windows (Section 5). **Example: the rule step by step, for B.** Before the rule: @@ -1236,7 +1242,7 @@ $$ h_v\ge\operatorname{need}_{\kappa,v,c,i}\,y_{\kappa,v,c,i}. $$ -The need is fixed per edge from the time horizon of the queries it serves, so constraint (R) is linear. For a chain it is the reading path's offset plus range: in the example of Section 3, the edge to A's path needs 5m, B's 1h + 5m and C's 1h, so $h_v$ for the shared 60 s chain is at least 65m, 65 windows. `@ end()` needs the range; `@ start()` needs the query range's duration plus the range. An `@ T` path needs the end of the horizon − T + range; when the horizon is unbounded the need is too, and that merge candidate is not offered. It ignores query latency: a result is assumed read as soon as its evaluation starts, so a slow consumer that reads later does not extend the horizon. Modeling that needs the latency design, which is out of scope (Section 5). The retention unit (windows, or seconds of data) and an upper bound on $h_v$ are fixed per candidate when Stage 2 builds it. +The need is fixed per edge from the time horizon of the queries it serves, so constraint (R) is linear. For a chain it is the reading path's offset plus range: in the example of Section 3, the edge to A's path needs 5m, B's 1h + 5m and C's 1h, so $h_v$ for the shared 60 s chain is at least 65m, 65 windows. `@ end()` and `@ start()` in an instant query need the range, as no `@` does; in a range query they read no chain. An `@ T` path needs the end of the horizon − T + range; when the horizon is unbounded the need is too, and that merge candidate is not offered. It ignores query latency: a result is assumed read as soon as its evaluation starts, so a slow consumer that reads later does not extend the horizon. Modeling that needs the latency design, which is out of scope (Section 5). The retention unit (windows, or seconds of data) and an upper bound on $h_v$ are fixed per candidate when Stage 2 builds it. ### No consumer-count policies From f6b4e70429ac2eaf23b3dc9400ccb8d3c99a5570 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sun, 8 Nov 2026 16:23:04 +0000 Subject: [PATCH 46/67] docs(concepts): Hydra is a rollup strategy, with a worked example Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 29 +++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 4eb70df60..8c9886fce 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -589,8 +589,9 @@ The floor representation has the applicability condition stated in Section 2. Cr **Fused groups.** A fused candidate computes, as one operator, a sub-DAG pattern of several operators in one appearance of the sub-expression; `fused_groups` lists the groups of the operators it replaces. Reusing a common sub-expression across appearances is Pass 2's reuse, not fusion. Fusion rules: - top-k (a sort and limit over an aggregate, as one heap summary); -- sum over sum, min over min, max over max (distributive aggregates, as one aggregate); -- possibly Hydra (a summary over every sub-population at once). +- sum over sum, min over min, max over max (distributive aggregates, as one aggregate). + +Hydra is not a fusion: it is a rollup strategy of the finest-common-group rule (Pass 2, Example 4). TODO: revisit each fused candidate's inputs and outputs to check that the fusing rules are correct. @@ -673,6 +674,30 @@ Both are exact `Aggregate` groups with equal definitions and selections, so they The hull of {all, host ∈ {xxx}} is all; the keys are {host} ∪ {job} ∪ {host}, where the last `host` is the column on which G2 is narrower. Stage 3 chooses between two KLL builds and one build of G\* with job × host states. +**Rollup by subpopulation readout (Hydra).** When $G^*$ is built with an algorithm that answers any subset of its keys directly (the Hydra algorithms: `HydraKll`, `HydraCms`, `HydraCountSketch`; a capability in `SummaryTypeCapability`), a member does not need a `SummaryMerge`: its **readout group** gets a candidate that reads the subpopulation straight from $G^*$'s state. A KLL rollup happens at the state level (merge the states, then read); a Hydra rollup happens at the readout level (read each subpopulation from one shared structure). This needs `SummaryEstimate` to name the keys it reads the subpopulation by (`subpopulation_keys`), which it cannot express today; both are added when Hydra is back in scope. + +**Example 4: Hydra as a rollup strategy.** Q1 = `quantile by (job)(0.99, latency[5m])`, Q2 = `quantile by (host)(0.99, latency[5m])`, Q3 = `quantile by (job, host)(0.99, latency[5m])`; 10 jobs, 1 000 hosts, 5 000 (job, host) pairs present. + +| Group | Expression | Candidates | +|---|---|---| +| G2 | `TimeRange(5m, Scan latency)` | | +| S_jh (= G\*) | `SummaryAgg[by job, host](G2)` | one `Sketch(Kll)` build per θ (one KLL per pair); one `Sketch(HydraKll)` build per θ (one shared structure) | +| S_j | `SummaryAgg[by job](G2)` | `Sketch(Kll)` builds; **`SummaryMerge(group_by job)(S_jh)`** (valid when S_jh is built with KLL) | +| S_h | `SummaryAgg[by host](G2)` | `Sketch(Kll)` builds; **`SummaryMerge(group_by host)(S_jh)`** (KLL) | +| R_jh ← Q3 | `SummaryEstimate(p99)(S_jh)` | reads S_jh | +| R_j ← Q1 | `SummaryEstimate(p99)(S_j)` | reads S_j; **`SummaryEstimate(p99, subpopulation_keys: [job])(S_jh)`** (valid when S_jh is built with Hydra) | +| R_h ← Q2 | `SummaryEstimate(p99)(S_h)` | reads S_h; **`SummaryEstimate(p99, subpopulation_keys: [host])(S_jh)`** (Hydra) | + +Stage 3 chooses among: + +| Plan | Built | States | Extra work per query | +|---|---|---|---| +| ① separate | S_jh, S_j, S_h with KLL | 5 000 + 10 + 1 000 KLLs | none | +| ② KLL and merge | S_jh with KLL; S_j, S_h merged from it | 5 000 KLLs | Q1 merges 5 000 KLLs into 10, Q2 into 1 000 | +| ③ Hydra | S_jh with HydraKll; all three read it | one structure, its size set by θ, not by the number of pairs | one subpopulation query per group | + +With many pairs and queries ③ is cheapest; with few pairs ① or ② can be. Hydra's error per subpopulation follows from its θ and enters (K) on the edges like any readout's. + #### Reach, caps and floors A query reaches a candidate when a path of candidate edges leads from the query's output group to it; this is the reach $a_{q,v}$, computed top down after steps 1 and 2 so that rollup consumers count. Section 4 defines the caps from it. Each input's floor is the strongest of the floor for its cap (`floor_for_cap` in `asap-accuracy`) and any direct need such as heap_size ≥ k under a top-k. `Memo::set_input_requirements` writes it; nothing else changes a candidate's r^log. From 97062af1b18683b116ad6b423c317a12a425dc93 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sun, 8 Nov 2026 17:06:43 +0000 Subject: [PATCH 47/67] docs(concepts): execution contexts explained with an example; grid contexts, edge costs, one consumer per None producer Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 103 +++++++++++------- 1 file changed, 64 insertions(+), 39 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 8c9886fce..156df12dc 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -1206,68 +1206,93 @@ Compute the parameter floor from this cap only under the ordering condition in S These extensions go beyond constraints (A)–(I) and the formulation's fixed-cost assumption. They are separated here so that the baseline reads on its own. -### Execution identity +### Execution contexts, by example -An AcrossEvaluations candidate, always a chain, has its own computation: each window is built when it ends. A candidate with ℓ = None runs inside an execution that needs it. For example, a finalize candidate used by queries A (every 10 s) and C (every 15 s) may execute in both evaluation contexts even though Stage 2 enumerates it only once. A OneEvaluation candidate also runs inside one evaluation context κ, and that evaluation's consumers share the result. +The cost of a plan is how often each computation runs and what one run costs. Choosing candidates does not say either: a candidate may run at every evaluation of one query, at the evaluations of several queries, as data arrives, or on its own schedule. Whether two consumers of one candidate share a run depends on whether they need it at the same times. An **execution context** κ names a class of executions that happen at the same times: within one κ, a computation runs once and every consumer in κ shares that run. -A **context** identifies a class of executions with a common owner and schedule across the accounting horizon. It is not necessarily one timestamp. An evaluation context reuses the workload's own description of when queries run, their `QueryRecurrence` and `RangeEvaluation`: queries with equal ones are evaluated together and share its OneEvaluation results. Queries A and B of the running example, both every 10 s at phase 0, form one context; a one-off query forms the context of its `OneTime` recurrence. `QueryRecurrence` holds an `f64` (`EstimatedRate`) and cannot be hashed, so contexts are deduplicated by equality and `ExecutionContextId` indexes the list of distinct contexts. +There are three kinds of context: -```rust -pub struct ExecutionContextId(pub usize); +| Context | When its executions happen | Executions per second ($\rho_\kappa$) | Example | +|---|---|---|---| +| `Evaluation { grid }` | at every time of an evaluation grid (phase + k·period), answering queries | 1000 / period (ms) | an alert every 10 s at phase 0: grid 10 s @ 0 | +| `IngestionPipeline(RefreshWindow)` | as data arrives, in batches of one refresh window `{ width_ms, phase_ms }` | 1000 / width | samples fed into the current window of a chain every 5 s | +| `Materialized(v)` | the own computation of a candidate v materialized across evaluations, on v's schedule, whoever reads it | v's (e.g. one build per window) | a query-time chain built when each window ends | +```rust pub enum ExecutionContext { - Evaluation { - recurrence: QueryRecurrence, - range_evaluation: Option, - }, - IngestionPipeline(RefreshWindowId), + Evaluation { grid: EvaluationGrid }, + IngestionPipeline(RefreshWindow), Materialized(PhysicalCandidateId), } +pub struct EvaluationGrid { pub period_ms: u64, pub phase_ms: u64 } ``` -A chain of ℓ = None and OneEvaluation candidates inherits its consuming context. Encountering an AcrossEvaluations producer switches to that producer's own computation context when traversing its inputs. The context identifies where the operation runs; `delivers_on` with the context decides whether a chain producer can supply it. +**Evaluation grids, and range queries.** An evaluation context is a grid of evaluation times, not a query. A range query is expanded into evaluations on its grid: with step 10 s, phase 0, each of its steps is an evaluation at a time of grid 10 s @ 0, so its steps and an instant query on the same grid are the same context and share runs. How many requests cover each grid time (61 for a 1 h span at 1 min steps refreshed every minute) multiplies the cost; it does not create another context. The subtree under `TimeShift { at: End }` in a range query runs on the grid of request end times, the one an instant query with the request's recurrence uses (Section 3, `@` handling). + +**Lifetimes and contexts.** A candidate with ℓ = None or OneEvaluation runs inside a context: once per execution of every context that selects it ($x_{\kappa,v}$). A candidate with ℓ = AcrossEvaluations runs once, in its own context `Materialized(v)` ($x^{\mathrm{mat}}_v$); consumers in other contexts read what it has stored, through edges whose producer must deliver on the consumer's context (`delivers_on`, Section 2). Its inputs run in its context. A ℓ = None producer has at most one consumer per context ($\sum_{c,i} y_{\kappa,u,c,i} \le x_{\kappa,u}$); consumers that share a result in one context read a OneEvaluation producer, or a materialized one. -| Extended variable | Meaning | +| Variable | Meaning | |---|---| -| $x_v^{\mathrm{mat}}$ | AcrossEvaluations candidate selected | -| $x_{\kappa,v}$ | Candidate with ℓ = None or OneEvaluation executes in context kappa | -| $y_{\kappa,u,c,i}$ | Input edge selected for a consumer execution in context kappa | -| $h_v$ | Selected retention amount for an AcrossEvaluations producer | +| $x_{\kappa,v}$ | candidate v (ℓ = None or OneEvaluation) runs in context κ | +| $x^{\mathrm{mat}}_v$ | candidate v (ℓ = AcrossEvaluations) is selected and runs in `Materialized(v)` | +| $y_{\kappa,u,c,i}$ | in context κ, input i of consumer c reads producer u | +| $z_{q,v}$ | query q's output is v | +| $h_v$ | how long a materialized v retains its rows (windows, results, raw data) | -The unindexed edge variable belongs to the baseline only. In the extension, every edge is indexed by its consumer's context, including the dedicated computation context of an AcrossEvaluations consumer. This avoids silently switching between two meanings of `y`. +**Example.** Q_A = `quantile_over_time(0.99, latency[1m])` and Q_B = `quantile_over_time(0.50, latency[1m])`, both every 10 s at phase 0; Q_C = Q_A's expression every 30 s at phase 0. -Activation, input satisfaction, utilization, output selection, and acyclicity must be defined for these execution instances. One binary per context/candidate assumes at most one reusable execution of that candidate within that context. If two uses require independent computations, the context or execution identity must distinguish them. +| Group | Expression | Candidates | +|---|---|---| +| G1 | `Scan latency` | | +| G2 | `TimeRange(1m, G1)` | | +| G3 | `SummaryAgg[KLL](G2)` | ① a build at query time; ② `SummaryMerge(TimeRange(1m, Gw))` | +| Gw | `SummaryTumblingWindowChain[KLL, w = 10 s](G1)` | a build at ingestion time, materialized across evaluations | +| G4 / G5 | p99 / p50 readouts of G3 | | -### Cost and retention +Contexts: κ10 = `Evaluation{10 s @ 0}` (Q_A, Q_B), κ30 = `Evaluation{30 s @ 0}` (Q_C), κw = `Materialized(Gw)`. -To avoid confusing per-execution costs with the baseline's horizon-wide cost, use separate coefficient names: +Plan 1, build at query time: -| Symbol | Definition | -|---|---| -| $W_v^{\mathrm{mat}}$ | Fixed computation and access cost over the horizon for an AcrossEvaluations candidate (a chain), one build per window, excluding variable retention | -| $c_{\kappa,v}$ | Cost of one execution in the specified context of a candidate with ℓ = None or OneEvaluation, including a OneEvaluation result's storage within that evaluation | -| $\rho_\kappa$ | Fixed number or expected number of context executions over the common horizon, from the existing evaluation counting in `crates/cost/src/execution_demand.rs` (a one-off query's is its `invocations`) | -| $h_v$ | Retention amount in a declared unit, such as retained windows | -| $\pi_v$ | Horizon-wide retention cost per unit of retained amount | -| $\operatorname{need}_{\kappa,v,c,i}$ | Retention amount producer v needs to serve this consumer input in this context: the time horizon the queries served by that context read, e.g. a 5m range over 10 s tumbling windows needs 30 windows. Query latency is ignored for now | +| Execution | Runs per second | Why | +|---|---|---| +| G3① in κ10 | 0.1 | read by G4 and G5 in κ10, so G3 is OneEvaluation and runs once for both | +| G3① in κ30 | 0.033 | Q_C's own run | +| G2, G1 in κ10 and κ30 | 0.1 and 0.033 | each build scans 1 minute of raw data | -A proposed linear objective is: +Plan 2, a shared chain: -$$ -\min\left( -\sum_{v\text{ across evaluations}}W_v^{\mathrm{mat}}x_v^{\mathrm{mat}} -+\sum_{\kappa,\ v\text{ None or OneEvaluation}}\rho_\kappa c_{\kappa,v}x_{\kappa,v} -+\sum_{v\text{ across evaluations}}\pi_v h_v -\right). -$$ +| Execution | Runs per second | Why | +|---|---|---| +| Gw in κw | one window per 10 s, built from the samples as they arrive | one computation, whoever reads it | +| G3② in κ10 | 0.1 | Q_A and Q_B; Gw delivers on κ10 (10 s divides 10 s) | +| G3② in κ30 | 0.033 | Q_C; Gw delivers on κ30 (10 s divides 30 s) | +| $h_{Gw}$ | 1 min, 6 windows | (R): the longest need of its selected readers | + +The MILP compares the two plans' cost per second. With much data, plan 2's single incremental build is usually cheaper; with little data, plan 1 can be. -A selected input edge imposes the corresponding retention lower bound: +### Cost and retention + +| Symbol | Definition | +|---|---| +| $W_v^{\mathrm{mat}}$ | cost per second of v's own computation in `Materialized(v)` (e.g. one build per window), excluding retention | +| $c_{\kappa,v}$ | cost of one run of v in context κ, from v's own operator and the logical size of its inputs; it does not depend on which producer v reads | +| $c_{\kappa,u,c,i}$ | cost, per run of κ, of consumer c reading input i from producer u: transferring or reading u's output, and the part of c's work that changes with u's physical form (u's θ, its number of windows). Everything that does not change when another producer of the same group is chosen is in $c_{\kappa,c}$ | +| $\rho_\kappa$ | executions of κ per second; for an evaluation grid, times the number of requests covering each grid time | +| $h_v$, $\pi_v$ | retained amount and its cost per unit per second | +| $\operatorname{need}_{\kappa,v,c,i}$ | how much of v's rows this edge reads back: for a time-indexed producer (raw data, a chain, results stored per evaluation time) the reading path's offset plus range | $$ -h_v\ge\operatorname{need}_{\kappa,v,c,i}\,y_{\kappa,v,c,i}. +\min\left( +\sum_{v}W_v^{\mathrm{mat}}x_v^{\mathrm{mat}} ++\sum_{\kappa,v}\rho_\kappa c_{\kappa,v}x_{\kappa,v} ++\sum_{\kappa,u,c,i}\rho_\kappa c_{\kappa,u,c,i}\,y_{\kappa,u,c,i} ++\sum_{v}\pi_v h_v +\right), +\qquad +h_v\ge\operatorname{need}_{\kappa,v,c,i}\,y_{\kappa,v,c,i}\ \ (\mathrm{R}). $$ -The need is fixed per edge from the time horizon of the queries it serves, so constraint (R) is linear. For a chain it is the reading path's offset plus range: in the example of Section 3, the edge to A's path needs 5m, B's 1h + 5m and C's 1h, so $h_v$ for the shared 60 s chain is at least 65m, 65 windows. `@ end()` and `@ start()` in an instant query need the range, as no `@` does; in a range query they read no chain. An `@ T` path needs the end of the horizon − T + range; when the horizon is unbounded the need is too, and that merge candidate is not offered. It ignores query latency: a result is assumed read as soon as its evaluation starts, so a slow consumer that reads later does not extend the horizon. Modeling that needs the latency design, which is out of scope (Section 5). The retention unit (windows, or seconds of data) and an upper bound on $h_v$ are fixed per candidate when Stage 2 builds it. +Every term is exact: no producer is chosen to stand for others. In the example of Section 3, the edges to A's, B's and C's paths need 5m, 1h + 5m and 1h, so the shared 60 s chain retains 65 windows. A Scan's raw data is a time-indexed producer like any other: its retention is its $h_v$ under the same (R). `@ T` needs the end of the horizon − T + range; with an unbounded horizon that candidate is not offered. Query latency is ignored: a result is taken to be read when its evaluation starts. $h_v$ has no upper bound here: it is minimized, and per-candidate bounds belong to resource budgets, outside this design. ### No consumer-count policies From 86919aec52fa2f1fca11edb2e9450cf48f84189e Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sun, 8 Nov 2026 17:17:21 +0000 Subject: [PATCH 48/67] docs(concepts): when the candidate graph can have a cycle; t only on strongly connected components Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 22 +++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 156df12dc..d423a48ad 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -1020,7 +1020,7 @@ The baseline uses the formulation's variables and constraints. Stage 3 adds no c | (B) | Chosen producers are selected | $y_{u,c,i}\le x_u$ | | (C) | One output per query | $\sum_{v\in G_q}z_{q,v}=1$ | | (D) | Output candidates are selected | $z_{q,v}\le x_v$ | -| (E) | Acyclic dependencies | $t_u+1\le t_c+\lvert\mathcal C\rvert(1-y_{u,c,i})$ | +| (E) | Acyclic dependencies, only on edges inside a strongly connected component of the admissible edges (below) | $t_u+1\le t_c+n(1-y_{u,c,i})$, $n$ = the component's size | | (F) | Selected candidates are used | $x_v\le\sum_{(v,c,i)\in\mathcal E}y_{v,c,i}+\sum_{q:v\in G_q}z_{q,v}$ | | (G) | Logical compatibility | $y_{u,c,i}\le\alpha^{\mathrm{log}}_{u,c,i}$ | | (H) | Physical compatibility | $y_{u,c,i}\le\alpha^{\mathrm{phys}}_{u,c,i}$ | @@ -1036,12 +1036,30 @@ $$ The cost includes input reading, the candidate's own computation, and lifetime-dependent retention. It excludes producers' computation, which is charged at those producers. This objective assumes execution frequencies and retention costs are fixed before optimization and independent of the selected consumers. When that assumption does not hold, use the execution extension in Section 5; do not silently reuse the same cost coefficient. +#### When the candidate graph has a cycle + +Group expressions only reference input groups, so the graph of groups has no cycle. Candidates are different: a rule may add to a group a candidate that reads another group, and when two groups can each be computed from the other, their candidates form a cycle. A plan may use either direction, but not both: if each reads the other, neither is computed. Constraint (E) rules that out. Rejecting one direction when the memo is built would lose plans: which direction is cheaper is a cost question only the MILP can answer. + +**Example.** A readout group whose column names differ from its consumer's gets a renaming `Project`: + +| Group | Candidates | +|---|---| +| Ga: `(job, p99)` | ① its own expression; ② `Project(value → p99)(Gb)` | +| Gb: `(job, value)` | ① its own expression; ② `Project(p99 → value)(Ga)` | + +Ga ① with Gb ② is a valid plan, and so is Gb ① with Ga ②; Ga ② with Gb ② is a cycle and (E) forbids it. + +Cycles can appear when two groups compute **equivalent results** that rules can derive from each other: renamings in both directions, rollups between groups with equal definitions, selections and keys, or any future rewrite that has an inverse. Today's rules add one direction only, so in practice the admissible edges have no cycle. + +$t_v$ and (E) are therefore built only for candidates inside a strongly connected component of the admissible edges, with $n$ the component's size; outside one, no edge can close a cycle. When there is no cycle, $t$ is all `None` and (E) adds no constraint. Merging groups proven equivalent into one group (as Cascades-style memos do) would remove such cycles without losing candidates; it needs an equivalence rule and is not part of this design. + ```rust struct SelectionVariables { x: Vec, y: HashMap, z: HashMap<(QueryId, PhysicalCandidateId), Variable>, - t: Vec, + /// t_v only for candidates on a cycle of admissible edges; None elsewhere. + t: Vec>, } // Schematic constraint-building entry points; signatures omitted. From 9b0cee097e64dd55c6601c67b543264957613811 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sun, 8 Nov 2026 18:59:17 +0000 Subject: [PATCH 49/67] docs(concepts): apply the 2026-11-08 decisions across the memo model Summary terminology (family, algorithm, summary type, parameters as structure only); value sign as a group property; readout required accuracy replacing parameter floors, with requirement levels for composed queries and per-metric error bounds in (K); where node, input and edge quantities live; edge costs in the Stage 3 baseline; Stage 3 builds PlanOutput { plan: PhysicalASAPDAG } with a Selection report; executor time-indexed store; Stage 2 prune reasons and ingestion / across-evaluation offers; @ end()/@ start() lifted above the anchored subtree. asap-primitive-schema terminology table follows. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 457 +++++++++++++----- 1 file changed, 327 insertions(+), 130 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index d423a48ad..152e0b835 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -6,11 +6,20 @@ The Rust blocks are interface sketches, not compilable code. Paths name today's ## 1. Model and running example -A **workload** contains query expressions, their evaluation schedules, and their accuracy targets. A **memo group** represents one logical result or one intermediate state family over specified input data. A **candidate** is one way to compute that result: one operator over input groups, together with its properties. +A **workload** contains query expressions, their evaluation schedules, and their accuracy targets. An accuracy target belongs to a query; an aggregate's measures inside the query carry none. A **memo group** represents one logical result or one intermediate summary state over specified input data. A **candidate** is one way to compute that result: one operator over input groups, together with its properties. Input groups leave producer choices open. Selecting a candidate does not select its entire input plan. Each consumer input chooses a producer separately, and several inputs may choose the same producer. -The expression includes the operator's arguments, including the summary family. In this design, KLL and DDSketch states belong to separate state groups. A final-result group can offer readout candidates that use either family. So the decision tree's first level (Section 2) chooses the expression, family included; a state group never mixes families. +**Summary terminology.** A summary state is described at three levels: + +| Term | What it is | Code | Example | +|---|---|---|---| +| Family | the kind of summary | a `FieldDataType` variant: `ExactAggregate`, `Sketch`, `SummarySample`, `Wavelet`, `StatModel` | `Sketch` | +| Algorithm | one structure within a family | `SketchAlgorithm` (`Kll`, `DDSketch`, `Cms`, `HydraKll`, …); `ExactKind`, `SamplingKind`, … for the other families | `Kll` | +| Summary type | family and algorithm together: one `FieldDataType` value | `FieldDataType::Sketch(SketchAlgorithm)` | `Sketch(Kll)` | +| Parameters θ | the structure's sizes | `SketchParams` (and the other families' parameter types), on the candidate's `LogicalProperties` | `Kll { k: 547 }` | + +The summary type is an operator argument (the `summary_type` field of `SummaryAgg` and of the chain), so it is part of the expression; θ is not. KLL and DDSketch states are therefore separate state groups, and builds that differ only in θ are candidates of one group. A final-result group can offer readout candidates over either state group. The decision tree's first level (Section 2) chooses the expression, summary type included; a state group never mixes summary types. Which statistics a summary type answers, and whether its states merge, is its row of `SummaryTypeCapability`. ### Running example @@ -19,16 +28,16 @@ Query A requests the p95 latency over one minute with a rank-error target. Query | Group | Result | Candidate expressions | |---|---|---| | G1 | Raw latency input | `Scan latency[1m]` | -| G2 | KLL state over G1 | `SummaryAgg[KLL](G1)` with `k = 200` or `k = 345` | -| G2′ | DDSketch state over G1 | `SummaryAgg[DDSketch](G1)` with `alpha = 0.01` or `alpha = 0.02` | +| G2 | KLL state over G1 | `SummaryAgg[Sketch(Kll)](G1)` with `k = 269` or `k = 547` | +| G2′ | DDSketch state over G1 | `SummaryAgg[Sketch(DDSketch)](G1)` with `alpha = 0.01` or `alpha = 0.02` | | G3 | Query A output | `SummaryEstimate[q=0.95](G2)` | | G4 | p99 latency | `SummaryEstimate[q=0.99](G2′)` | | G5 | p50 latency | `SummaryEstimate[q=0.50](G2′)` | | G6 | Query B output | `Divide(G4, G5)` | -KLL and DDSketch states are separate groups because the family is part of the expression. The metric rule of Section 4 decides which consumer may read which: KLL bounds **rank** error, which does not bound a quantile's value error or a ratio's relative error, so B's operands read the DDSketch state (relative value error), and A reads the KLL state. +KLL and DDSketch states are separate groups because the summary type is part of the expression. The metric rule of Section 4 decides which consumer may read which: KLL bounds **rank** error, which does not bound a quantile's value error or a ratio's relative error, so B's operands read the DDSketch state (relative value error), and A reads the KLL state. -G4 and G5 choose a G2′ producer independently. If both choose the same retained physical producer, they reuse one computed state when their schedules and windows are compatible. An unmaterialized candidate may instead execute separately in different contexts; Section 5 explains that distinction. +G4 and G5 choose a G2′ producer independently. If both choose the same materialized physical producer, they reuse one computed state when their schedules and windows are compatible. An unmaterialized candidate may instead execute separately in different contexts; Section 5 explains that distinction. ### Notation used by the baseline model @@ -41,9 +50,10 @@ Symbols are collected here and in the sections that use equations. The prose and | $G$, $G_q$ | Memo group and query-output group | | $v$, $u$, $c$, $i$ | Candidate, producer candidate, consumer candidate, and consumer input index | | $e_v$, $p_v$ | Candidate expression and properties | -| $\boldsymbol\theta_v$, $\tau_v$, $\ell_v$ | Parameter configuration, execution timing, and result lifetime | +| $\boldsymbol\theta_v$, $\tau_v$, $\ell_v$ | Parameters (structure only), execution timing, and result lifetime | | $\mathcal I(c)$, $G_{c,i}$ | Consumer input indices and required group for one input | -| $r_{c,i}$ | Logical and physical requirements of one input | +| $r_{c,i}$ | Logical requirement (required accuracy, readout inputs only) and physical requirement of one input | +| $n_q$ | Number of values one evaluation of query q returns | | $\mathcal C$ | Physical candidate set | | $\alpha^{\mathrm{log}}_{u,c,i}$, $\alpha^{\mathrm{phys}}_{u,c,i}$ | Fixed logical and physical compatibility indicators | @@ -84,12 +94,12 @@ This is the single reference table for property dimensions. Stage descriptions b | Dimension | Values and meaning | Code reference or target | |---|---|---| -| Parameters | A predefined configuration list per family: KLL and KMV `k`; DDSketch `alpha`; HLL `precision`; CMS and CountSketch `width` and `depth`; heap variants also `heap_size`; UnivMon its sizing parameters. The lists are today's sizing over ε = 0.1 down to 0.001; KMV's and UnivMon's distinct-count entries stop at ε = 0.01. UnivMon's list is the product of its 7 L2 column counts and its 4 distinct-count heap sizes, plus the baseline, so the per-parameter maximum of an L2 floor and a distinct-count floor is dominated by a listed entry. A heap variant combines each width × depth with heap sizes from the 1-2-5 series 10, 20, 50, …, 10 000: a top-k consumer's floor is heap_size ≥ k, and the smallest listed heap that meets it is the next step up, at most 2.5× too large. A sketch build's parameters must be in its family's list; exact operators have no sketch parameters. Hydra and the other families Pass 1 does not size have empty `parameter_list`s, so no build of theirs is a valid candidate until their error models exist (#1228). | `SketchParams` in `crates/types/src/ir/schema/sketch.rs`; predefined lists are the #1203 target | +| Parameters θ | The structure of a summary, from a predefined list per algorithm, `parameter_list(algorithm)`: KLL and KMV `k`; DDSketch `alpha`; HLL `precision`; CMS and CountSketch `width` and `depth`; heap variants also `heap_size`; UnivMon its L2 column count and distinct-count heap size. θ is structure only: no field records the error or failure probability a configuration was sized for, and equality is derived over all fields. The lists span today's sizing over ε = 0.1 down to 0.001 (KMV's and UnivMon's distinct-count entries stop at ε = 0.01). Where δ is set by a structural parameter, the list covers the δ levels with it: CMS lists each width with depths 5, 10 and 14 (δ = e^−d ≤ 1e-2, 1e-4, 1e-6). UnivMon's list has 40 entries: each of the 7 L2 column counts with each of the 4 distinct-count heap sizes, each of them alone, and the baseline. A heap variant combines each width × depth with heap sizes from the 1-2-5 series 10, 20, 50, …, 10 000, so the smallest heap that holds a top-k's k is at most 2.5× too large. A sketch build's θ must be in its algorithm's list; exact operators have none. Algorithms without an error model (the Hydra algorithms among them) have empty lists, so no build of theirs is a valid candidate until their error models exist (#1228). | `SketchParams` in `crates/types/src/ir/schema/sketch.rs`; `parameter_list` | | Timing | `IngestionTime` or `QueryTime` | `ExecutionTiming` | | Lifetime | `None`: discard after the consuming execution; `OneEvaluation`: retain until consumers within that evaluation finish; `AcrossEvaluations`: retain across evaluations in the specified period. A non-`None` lifetime means materialized. | `Lifetime`; #1200 | | Storage medium — Extension | `Memory` or `Disk`, carried by a materialized lifetime | `StorageMedium` | -Error guarantees are derived from parameters and the statistic being read. They include an error metric, a bound, and a failure probability; exact computation introduces no approximation error. The same state may support different readouts, so its guarantee must be interpreted for the relevant statistic. +An error guarantee is not a property: it depends on the summary type, θ and the statistic a readout reads, and is checked on the edge from a build to a readout (below). It has an error metric, a bound, and a failure probability; exact computation introduces no approximation error. The properties are the formulation's, $p_v^{\mathrm{log}}=\boldsymbol\theta_v$ and $p_v^{\mathrm{phys}}=(\tau_v,\ell_v)$; in code, `LogicalProperties { parameters }` and `PhysicalProperties { timing, lifetime }`. @@ -97,11 +107,56 @@ The properties are the formulation's, $p_v^{\mathrm{log}}=\boldsymbol\theta_v$ a ### Logical requirements -The proposed `LogicalRequirements` stores a **parameter floor** $\boldsymbol\theta^{\min}$: the weakest parameters whose ε is at most the applicable input error cap, or a direct need such as heap_size ≥ k for a top-k. The floor is any value of the family's parameter type and need not be in the family's list; a listed producer satisfies it when $\boldsymbol\theta_u \succeq \boldsymbol\theta^{\min}$. Section 4 defines the cap and distinguishes validity from pruning. It does not allocate a fixed share of the query's error budget to each input. +A logical requirement is the accuracy a consumer input needs from its producer. Only a readout's input (`SummaryEstimate`, `FinalizeExactAccumulator`, `EvaluatePopulation` over an approximate state) carries one: + +```rust +pub struct LogicalRequirements { + /// None on every input that is not a readout's, and on a readout's input + /// when no accuracy target reaches it. + pub required_accuracy: Option, +} + +pub struct RequiredAccuracy { + pub metric: AccuracyMetric, + /// ε_r, in the metric's units. + pub max_error: f64, + /// δ_r, per value; 1 when the target sets none. + pub failure_probability: f64, +} +``` + +**One rule for every algorithm.** A build u satisfies readout r's requirement when u's guarantee for r's statistic, evaluated at the required failure probability, is within the required error: + +$$ +\alpha^{\log}_{u,r,0}=1 +\iff +\operatorname{guarantee}(\text{summary type},\ \boldsymbol\theta_u,\ \text{statistic}_r)\big|_{\delta=\delta_r}\le\varepsilon_r\ \text{in metric}_r, +$$ + +and, for a top-k readout, `heap_size` ≥ k. An exact producer has error 0 and satisfies every requirement. Some algorithms set ε and δ with separate parameters, as CMS's width and depth do; in others one parameter sets both, as KLL's k does. The rule is the same: evaluate the guarantee at δ_r and compare it with ε_r. + +**Example: KLL, where k sets both.** A requirement of rank error 0.01. KLL's rank error is 2.296 / k^0.9723 at δ = 0.01, scaled by z(1 − δ/2) / z(0.995) below it (1.90 at δ = 1e-6): + +| k | rank error at δ = 1e-2 | rank error at δ = 1e-6 | meets (0.01, 1e-2) | meets (0.01, 1e-6) | +|---|---|---|---|---| +| 269 | 0.0100 | 0.0189 | yes | no | +| 547 | 0.0050 | 0.0095 | yes | yes | +| 1404 | 0.0020 | 0.0038 | yes | yes | + +**Example: CMS, where width sets ε and depth sets δ.** A point-count requirement of 0.01 of the stream's L1 norm. Width 272 gives ε = e / 272 = 0.0100; depth d gives δ = e^−d: + +| θ | ε (× L1) | δ | meets (0.01, 1e-2) | meets (0.01, 1e-4) | +|---|---|---|---|---| +| width 272, depth 5 | 0.0100 | 6.7e-3 | yes | no | +| width 272, depth 10 | 0.0100 | 4.5e-5 | yes | yes | -For KLL, a larger `k` satisfies a smaller minimum `k`. For DDSketch, a smaller `alpha` can be stronger. Multi-parameter families require the family's comparison rule, including parameters needed for failure guarantees. A single floor is valid only if the accepted configurations form an upward-closed set with a representable least element. Otherwise, compatibility must use an explicit admissible set or predicate; it must not invent a unique minimum. +**Where the guarantee comes from.** `guarantee` is `AccuracyEvidence::measured_guarantee(summary type, θ, statistic)` when the deployment's evidence has a measurement for that configuration, and `theoretical_guarantee` (the algorithm's error formula) otherwise. The source is decided before Pass 2 builds the logical edges, so (K) always has every readout's bound, and nothing is checked again after Stage 3. -The required input group's identity already constrains the state family and data/window semantics. An exact producer satisfies an approximation floor only if it is also a valid producer of that group and input type. +**Sign conditions.** Some guarantees hold only for values of a known sign, read from the input group's `column_sign` (Section 3, "Value sign"): DDSketch's relative error on a quantile needs a `NonNegative` input column. + +**A readout's error is its requirement.** A readout candidate's output error is (ε_r, δ_r), a node property: every producer it may read meets it. Constraint (K) composes these errors to the query root (Section 4). Which requirements a readout group gets is Pass 2's choice (Section 4, "Readout requirements"): one per requirement level, each its own readout candidate. + +The required input group's identity already constrains the summary type and data/window semantics. Inputs that are not a readout's have no requirement; their α^log is the metric rule (Section 4). ### Physical requirements @@ -115,8 +170,9 @@ Physical requirements are derived from the consumer's timing and execution conte QT means QueryTime; IT means IngestionTime. Alignment, data coverage, and retention checks still apply to allowed combinations. **Alignment.** There is no physical schedule property. When a result can be read follows from its lifetime and, for a chain, its window width: -- **None** and **OneEvaluation**: the result is computed inside one evaluation context κ (Section 5) and read there. A OneEvaluation result belongs to one context: queries with equal recurrence and range evaluation are evaluated together, and that evaluation's consumers share the result. When it runs is the context's schedule, and Stage 3 chooses it through $x_{\kappa,v}$, just as for ℓ = None. Sharing across differently scheduled queries through a gcd schedule is not offered: it is rarely beneficial and wastes runs. -- **AcrossEvaluations** is offered only to chain candidates, `SummaryTumblingWindowChain[w](X)`: the input is cut into tumbling windows (k·w, (k+1)·w], with the grid anchored at phase 0 (the epoch). Windows are left-open and right-closed like a PromQL range, which `m[r]` evaluated at t reads as (t − r, t]. Each window is built when it ends and can be read from then on. A `TimeRange` over the chain selects the windows that lie fully inside (t − range, t], so a consumer can read a range whose ends lie on window boundaries. With w = 10 s, a `[1m]` query evaluated every 30 s at phase 0 merges 6 windows; one evaluated at phase 15 s cannot, because its range ends inside a window; with that query in the workload, the gcd closure also offers w = 5 s, which both can read. Every other candidate, including the `TimeRange` and `TimeShift` groups over a chain, is computed within one evaluation. +- **None** and **OneEvaluation**: the result is computed inside one evaluation context κ (Section 5) and read there. A OneEvaluation result belongs to one context: queries on the same evaluation grid, a range query's steps included, are evaluated together, and that evaluation's consumers share the result. When it runs is the context's schedule, and Stage 3 chooses it through $x_{\kappa,v}$, just as for ℓ = None. Sharing across differently scheduled queries through a gcd schedule is not offered: it is rarely beneficial and wastes runs. +- **AcrossEvaluations** on a chain, `SummaryTumblingWindowChain[w](X)`: the input is cut into tumbling windows (k·w, (k+1)·w], with the grid anchored at phase 0 (the epoch). Windows are left-open and right-closed like a PromQL range, which `m[r]` evaluated at t reads as (t − r, t]. Each window is built when it ends and can be read from then on. A `TimeRange` over the chain selects the windows that lie fully inside (t − range, t], so a consumer can read a range whose ends lie on window boundaries. With w = 10 s, a `[1m]` query evaluated every 30 s at phase 0 merges 6 windows; one evaluated at phase 15 s cannot, because its range ends inside a window; with that query in the workload, the gcd closure also offers w = 5 s, which both can read. The `TimeRange` and `TimeShift` groups over a chain are computed within one evaluation. +- **AcrossEvaluations** on a candidate of any other group evaluated on a known grid: a time-independent result (one that does not depend on the evaluation time t, such as a result under `@ T`) is stored as is and can be read at any time; a time-dependent result is stored per evaluation time, as a time-indexed result series computed at every time of its grid. A consumer reads the value stored for its own evaluation time, or, through a `TimeShift` above it, for a shifted time; so the consumer's evaluation times must lie on the producer's grid. A grid phase is not needed: a phase is absorbed by the width, because width candidates come from the gcd closure of the queries' grids, ranges and offsets (Section 3). For example, phases 0 and 5 s with w = 10 s give w = 5 s. Alignment alone does not ensure sufficient retained history; that is $h_v$ (Section 5). @@ -130,7 +186,7 @@ A grid phase is not needed: a phase is absorbed by the width, because width cand Prometheus staggers rule-group evaluations by a hash of the group, so a fixed interval without a declared phase does not mean phase 0. One rule decides both sides: `QueryRecurrence::evaluation_grid_ms` returns `None` for a `FixedInterval` without a phase. Pass 2 then offers that query no chain width, and `delivers_on` lets it read no chain. Workloads declare the phase with `FixedIntervalAt { interval, evaluation_phase }` to use chains. -An evaluation context κ is described by the workload's own recurrence and range evaluation (Section 5). Let a chain producer have tumbling width $w$. A consumer running in κ can read the producer when $w$ divides every evaluation time of κ: +An evaluation context κ is a grid of evaluation times, derived from the workload's recurrence and range evaluation (Section 5). Let a chain producer have tumbling width $w$. A consumer running in κ can read the producer when $w$ divides every evaluation time of κ: | Producer lifetime, or κ's recurrence for a chain producer with ℓ = AcrossEvaluations | Required alignment | |---|---| @@ -140,7 +196,8 @@ An evaluation context κ is described by the workload's own recurrence and range | `Scheduled(times)` | $w$ divides every time (and $s$) | | `OneTime { execute_at: Some(t) }` | $w \mid t$ (and $s$) | | `FixedInterval` without a phase, `OneTime { execute_at: None }`, `EstimatedRate`, `Unknown` | Never: the evaluation times are not known, and `evaluation_grid_ms` is `None` | -| AcrossEvaluations on a candidate that is not a chain | Not offered: only chain candidates are materialized across evaluations | +| A time-dependent result series with ℓ = AcrossEvaluations, computed on grid $g$ | every evaluation time of κ (shifted by the reading path's `TimeShift`s) lies on $g$ | +| A time-independent result with ℓ = AcrossEvaluations | Always | The evaluation grid is `QueryRecurrence::evaluation_grid_ms(range_evaluation)` in `crates/types/src/workload/mod.rs`, which Pass 2's chain widths and `delivers_on` both use. One-off queries are supported: each is a context of its own recurrence, run `invocations` times. @@ -156,7 +213,7 @@ A tumbling grid's width is an argument of the chain operator, part of the expres ### Decision tree and order of decisions -A memo group's candidates are the leaves of a tree. The first level chooses the expression $e_v$ (operator and family); each later level decides one property dimension. A path from the root to a leaf is one physical candidate. +A memo group's candidates are the leaves of a tree. The first level chooses the expression $e_v$ (operator and summary type); each later level decides one property dimension. A path from the root to a leaf is one physical candidate. ```mermaid flowchart TD @@ -166,16 +223,16 @@ flowchart TD G --> R5["e_v: SummaryMerge over TimeRange
of a chain of width w
(state groups; widths from the
gcd closure of grids, ranges, offsets)"] G --> R4["e_v: other operators"] - R2 --> T1["θ: k = 200"] - R2 --> T2["θ: k = 345"] - R2 --> T3["θ: k = …
(prune: pruning cap)"] + R2 --> T1["θ: k = 269"] + R2 --> T2["θ: k = 547"] + R2 --> T3["θ: k = …
(prune: no readout requirement admits it)"] T2 --> P1["τ = QueryTime"] - T2 --> P2["τ = IngestionTime
(prune: no ingestion pipeline)"] + T2 --> P2["τ = IngestionTime
(time-independent candidates and their inputs;
prune: no ingestion pipeline)"] P1 --> L1["ℓ = None"] P1 --> L2["ℓ = OneEvaluation"] - P1 --> L3["ℓ = AcrossEvaluations
(offered only to chains; prune: deployment
retains no query-time results)"] + P1 --> L3["ℓ = AcrossEvaluations
(chains and grid-evaluated groups; prune:
deployment retains no query-time results)"] P2 --> L4["ℓ = None"] P2 --> L5["ℓ = AcrossEvaluations"] P2 -. invalid .-> L6["ℓ = OneEvaluation"] @@ -184,11 +241,11 @@ flowchart TD | Level | Decides | Stage | Values offered | Pruned when | |---|---|---|---|---| | 1 | expression $e_v$, including a merge over a chain of width $w$ | Pass 1 and Pass 2 rules | each way to compute the group; chain widths from the gcd closure of the reaching queries' grids, ranges and offsets | no reaching query has a known grid (no chain is offered) | -| 2 | $\boldsymbol\theta_v$ | Pass 1 | the family's predefined list | the pruning cap (Section 4); dead-candidate elimination (Section 3) | -| 3 | $\tau_v$ | Stage 2 | QueryTime, IngestionTime; IngestionTime for a counter state only when the source's out-of-order bound is known (Stage 2) | the source has no ingestion pipeline; QueryTime over raw rows the deployment does not retain | -| 4 | $\ell_v$ | Stage 2 | None, OneEvaluation, AcrossEvaluations; AcrossEvaluations only for chain candidates, and Disk only where the deployment offers it | invalid with $\tau_v$; the deployment retains no query-time results | +| 2 | $\boldsymbol\theta_v$ | Pass 1 | the algorithm's predefined list | dead-candidate elimination (Section 3): no readout requirement it could serve admits it | +| 3 | $\tau_v$ | Stage 2 | QueryTime; IngestionTime only for a candidate that is time-independent or an input of one, and for a counter state only when the source's out-of-order bound is known (Stage 2) | the source has no ingestion pipeline; QueryTime over raw rows the deployment does not retain; a Count-Min build whose weights are not proven non-negative | +| 4 | $\ell_v$ | Stage 2 | None, OneEvaluation, AcrossEvaluations; AcrossEvaluations for chains and for candidates of groups evaluated on a known grid, and Disk only where the deployment offers it | invalid with $\tau_v$; the deployment retains no query-time results | -The planner does not check whether an executor implements an operator or family: every candidate the rules produce is offered, and an executor that lacks one is extended. A value outside a dimension's declared value set (AcrossEvaluations on a candidate that is not a chain, Disk where the deployment has no disk storage) is not offered, so it is not recorded as pruned. +The planner does not check whether an executor implements an operator or summary type: every candidate the rules produce is offered, and an executor that lacks one is extended. A value outside a dimension's declared value set (IngestionTime on a time-dependent candidate, AcrossEvaluations on a candidate of a group with no known grid, Disk where the deployment has no disk storage) is not offered, so it is not recorded as pruned. The tree does not decide which producer each consumer input reads ($y$), in which context a candidate with ℓ = None or OneEvaluation runs (Section 5), or how much an AcrossEvaluations candidate retains ($h_v$). Those are Stage 3 choices. @@ -197,30 +254,52 @@ The tree does not decide which producer each consumer input reads ($y$), in whic **The order changes how many nodes enumeration visits:** $\sum_d \prod_{d' \le d} \lvert \text{values left at } d' \rvert$. Dimensions that prune much go early; dimensions that multiply without pruning go late. The order above follows four rules: 1. **Logical before physical.** Stage 1 hash-conses groups and computes $\alpha^{\log}$ once, before Stage 2 multiplies each logical candidate by its physical alternatives. -2. **Strong prunes early.** The pruning cap and dead-candidate elimination act before physical properties multiply the tree. +2. **Strong prunes early.** Dead-candidate elimination acts before physical properties multiply the tree. 3. **Dependent dimensions adjacent.** $\tau \to \ell$ rejects the invalid combination at the first level that can detect it. 4. **Deployment-wide prunes as early as possible.** Without an ingestion pipeline the IngestionTime subtree is pruned once per candidate; moving $\tau$ above $\boldsymbol\theta$ would prune it once per group, with the same leaves. Alternative orders worth measuring: $\tau$ first, when most sources lack an ingestion pipeline; and requirement-driven (top-down) enumeration, which visits fewer nodes but must take the union of all consumers' requirements so that sharing is not lost. +### Node, input or edge: where each quantity lives + +Every quantity the optimizer uses is one of three kinds, and is stored where its dependencies are: + +- a **node property** depends only on one candidate (θ, τ, ℓ, a readout's error) or one group (schema, lineage, `column_sign`); +- an **input requirement** depends only on the consumer's input (`RequiredAccuracy`; the physical requirement, derived from the consumer's timing); +- a **pair quantity** depends on both producer and consumer, so it lives on the edge (u, c, i); when it also depends on the execution context, on (edge, κ). + +Each quantity is computed once, at the earliest stage that knows everything it depends on, and later stages read it. Nothing pair-dependent is stored on a node, where it would need a representative producer or consumer. + +**Example.** A DDSketch build u with alpha = 0.01 over `latency` (declared non-negative), read by the p50 readout r of query B, whose requirement is relative error 0.01 at δ = 1e-2: + +| Quantity | Kind | Depends on | Computed at | Value | +|---|---|---|---|---| +| θ_u | node property | u | Pass 1 | alpha = 0.01 | +| `column_sign` of the build's input | group property | the input group | Stage 0 | `NonNegative` | +| `RequiredAccuracy` of r's input; r's output error | input requirement; node property | r | Pass 2 | relative, 0.01, 1e-2 | +| α^log_{u,r,0} | pair | u, r | Pass 2 | 1: DDSketch's guarantee for p50 at alpha 0.01 is relative 0.01 at δ = 0, and the input is non-negative | +| α^phys_{u,r,0} | pair | τ_u, ℓ_u, τ_r | Stage 2 | 1 for u at query time with ℓ = None and r at query time | +| `delivers_on(u, κ)` | (edge, κ) | ℓ_u, κ | Stage 3 | true: ℓ = None runs in κ | +| edge cost $c_{\kappa,u,r,0}$ | (edge, κ) | u's output size, κ | Stage 3, from `asap-cost` | reading one alpha-0.01 DDSketch per execution of κ | +| $\operatorname{need}_{\kappa,u,r,0}$ | (edge, κ) | the reading path, κ | Stage 3 | none: u is not time-indexed (for a chain producer, the path's offset plus range) | + ## 3. Pipeline: interfaces and rules | Stage | Input | Output | Responsibility | |---|---|---|---| -| Stage 0 | Query IR | `Memo` (groups without candidates) | Hash-cons logical expressions | +| Stage 0 | Query IR, each query with its accuracy target and $n_q$; the data workload's source facts | `Memo` (groups without candidates) | Hash-cons logical expressions; derive group properties | | Stage 1, Pass 1 | Group DAG | `Memo` | Add initial logical candidates and intermediate groups | -| Stage 1, Pass 2 | Memo | `LogicalASAPMemoCandidates` | Add rollup, top-k and chain alternatives, set parameter floors, eliminate dead candidates, and compute logical edges | +| Stage 1, Pass 2 | Memo | `LogicalASAPMemoCandidates` | Add rollup, top-k and chain alternatives, give readouts their requirements, eliminate dead candidates, and compute logical edges | | Stage 2 | Logical memo candidates | `PhysicalASAPMemoCandidates` | Expand physical alternatives and compute compatibility | -| Stage 3 | Physical memo candidates | `DagOfSelectedCandidates` | Select implementations, dependencies, and executions | -| Realizer/export | Selected DAG | `PhysicalASAPDAG` | Build and export the executable graph | +| Stage 3 | Physical memo candidates | `PlanOutput { plan: PhysicalASAPDAG }`, and a `Selection` report | Select implementations, dependencies, and executions, and build the plan | -Enumeration follows the decision tree in Section 2. Accuracy pruning uses only the caps of Section 4. +Enumeration follows the decision tree in Section 2. Accuracy pruning uses the readout requirements and caps of Section 4. Rules add groups through `find_or_add_group` and candidates through `add_candidate`. They match group expressions, avoid duplicating an existing expression/property pair, and do not copy consumers to make a new producer visible. After logical generation finishes, edge construction records the possible producer choices. ### Stage 0: groups -`memo_of_queries` in `crates/stage0-memo` builds the groups bottom-up from the queries' nodes: each node's operator, over its children's groups, goes through `Memo::find_or_add_group`, so identical sub-expressions are one group; each query's root is then added with `Memo::add_query`. Stage 0 and Stage 1 share one type, `Memo`: Stage 0 returns it with every group's candidate list empty, and Stage 1 fills the lists. +`memo_of_queries` in `crates/stage0-memo` builds the groups bottom-up from the queries' nodes: each node's operator, over its children's groups, goes through `Memo::find_or_add_group`, so identical sub-expressions are one group; each query's root is then added with `Memo::add_query`, with the query's accuracy target and $n_q$, the number of values one evaluation returns (Pass 2 turns a whole-result failure probability into a per-value one with it). Stage 0 and Stage 1 share one type, `Memo`: Stage 0 returns it with every group's candidate list empty, and Stage 1 fills the lists. ```rust pub struct MemoGroupId(pub u32); @@ -241,6 +320,10 @@ pub struct MemoGroup { pub struct GroupProperties { pub schema: Schema, pub result_kind: OperatorResultKind, + /// Per output column: its source column or Computed ("Column lineage"). + pub column_lineage: Vec, + /// Per output column: its value sign ("Value sign"). + pub column_sign: Vec, /// What an aggregate group covers: a summary state, a chain, a merge /// over a chain's windows, or an exact `Aggregate` result; `None` for /// any other group. @@ -337,9 +420,10 @@ A group is its expression: one operator, with its arguments, over input groups. Everything that was computed on operator-node trees is computed on groups. `find_or_add_group` derives `GroupProperties` once, when the group is created, with `derive_group_properties`: the rules that check operator nodes (schema derivation, input validation, merge check), applied to the expression over `MemoGroupAsInput`s, which read only the input groups' `expression` and `group_properties`, never a node tree: - `schema` and `result_kind` from the operator and its input groups' schemas and result kinds; a scalar subquery reads the referenced group's schema. -- `aggregate_coverage`, for a summary state or an exact `Aggregate` result: walking down from the SummaryAgg or Aggregate through the input groups' expressions while each is a Filter, a TimeRange, a TimeShift (without `@` or with `@ T`; `@ end()` and `@ start()` are not lifted, see below) or a direct-column Project, lifting their predicates into the selection; a TimeRange or TimeShift becomes an interval on the timestamp column (below). The group below them is the definition's `input_group`. A Scan's predicates (PromQL label matchers, SQL filters, including absolute timestamp filters) are lifted too; the Scan with its residual predicates is then the last unlifted operator and `input_group` is `None`. The operators not lifted, each with its residual arguments, are `unlifted_operators`. A chain `SummaryTumblingWindowChain[w](X)` has the definition of a build over X and leaves the timestamp column unconstrained; a `TimeRange` or `TimeShift` over a chain constrains it as over raw rows. A merge's coverage comes from its inputs' stored coverage; a merge is valid only over inputs with equal definitions and disjoint selections, or over one time-indexed state. So the coverage of `SummaryMerge(TimeRange(TimeShift(Chain[w](X))))` equals that of `SummaryAgg(TimeRange(TimeShift(X)))`, and `add_candidate` accepts the merge in the original state group. +- `aggregate_coverage`, for a summary state or an exact `Aggregate` result: walking down from the SummaryAgg or Aggregate through the input groups' expressions while each is a Filter, a TimeRange, a TimeShift (an offset or `@ T`; `TimeShift { at: End | Start }` is not lifted, see below) or a direct-column Project, lifting their predicates into the selection; a TimeRange or TimeShift becomes an interval on the timestamp column (below). The group below them is the definition's `input_group`. A Scan's predicates (PromQL label matchers, SQL filters, including absolute timestamp filters) are lifted too; the Scan with its residual predicates is then the last unlifted operator and `input_group` is `None`. The operators not lifted, each with its residual arguments, are `unlifted_operators`. A chain `SummaryTumblingWindowChain[w](X)` has the definition of a build over X and leaves the timestamp column unconstrained; a `TimeRange` or `TimeShift` over a chain constrains it as over raw rows. A merge's coverage comes from its inputs' stored coverage; a merge is valid only over inputs with equal definitions and disjoint selections, or over one time-indexed state. So the coverage of `SummaryMerge(TimeRange(TimeShift(Chain[w](X))))` equals that of `SummaryAgg(TimeRange(TimeShift(X)))`, and `add_candidate` accepts the merge in the original state group. - The checks that read an input's operator (a merge over a time-indexed state reads a `TimeRange` over a chain; a selected build) read the input group's `expression`. - `shareable_when_equal`: a `LIMIT` is shareable over a `Sort` with the same partitions (the input group's `expression`); a `Dedup` on every column (the input group's schema). +- `column_lineage` and `column_sign` (below), per output column. `add_candidate` derives a candidate's properties with the same `derive_group_properties` and requires its schema and result kind to equal the group's. `OperatorNode` recursively contains chosen child nodes and therefore describes a fixed plan fragment; `Operator` contains input group IDs and leaves their implementations open. A candidate needs its own expression because a group may offer several ways to compute the same result. @@ -355,6 +439,8 @@ pub struct GroupProperties { pub result_kind: OperatorResultKind, /// One entry per output column, in schema order. pub column_lineage: Vec, + /// One entry per output column, in schema order (Value sign, below). + pub column_sign: Vec, pub aggregate_coverage: Option, pub shareable_when_equal: bool, } @@ -407,13 +493,12 @@ A repeatedly evaluated query covers different absolute times at every evaluation |---|---|---| | `m[5m]` | `Excluded(EvaluationTimePlusMs(-300_000))` | `Included(EvaluationTimePlusMs(0))` | | `m[5m] offset 1h` | `Excluded(EvaluationTimePlusMs(-3_900_000))` | `Included(EvaluationTimePlusMs(-3_600_000))` | -| `m[5m] @ end()` or `@ start()`, instant query | as `m[5m]`: both equal t, and the PromQL front end lowers them as no `@` | | -| `m[5m] @ end()` or `@ start()`, range query | not lifted: the `TimeShift` stays in `unlifted_operators` | | +| `m[5m] @ end()` or `@ start()` | the subtree `m[5m]` is lifted as `m[5m]`; the `TimeShift { at: End \| Start }` above it is not lifted | | | `m[5m] @ 1700000000` | `Excluded(FixedValue(Timestamp(1_699_999_700_000)))` | `Included(FixedValue(Timestamp(1_700_000_000_000)))` | | SQL `WHERE ts >= '2026-11-01'` | `Included(FixedValue(Timestamp(..)))` | `Unbounded` | | SQL `WHERE price > 10` (column `price`) | `Excluded(FixedValue(Float64(10.0)))` | `Unbounded` | -A range-evaluated query evaluates its expression at every step t of [start, end]. `@ end()` and `@ start()` pin a subtree to one end of that interval, so the subtree reads the same rows at every step while a plain `m[5m]` reads different rows at every step; the two are not equal coverage, and the `@` subtree is not shared. Evaluating `@` subtrees once per request and sharing them is future work (#1251). +A range-evaluated query evaluates its expression at every step t of [start, end]. `@ end()` and `@ start()` pin a subtree to one end of that interval, so the subtree reads the same rows at every step. The front end lowers the subtree X without the `@`, so X is an ordinary group shared with any query over the same expression, and puts `TimeShift { at: End | Start }(X)` above it. In a range query, X runs once per request, in the evaluation context of the request's end times (for `@ start()`, end minus the span; Section 5), and its result is read at every step. In an instant query both equal t, so `TimeShift { at: End }(X)` is X. **Derivation, per operator** (from the input groups' `column_lineage`): @@ -468,6 +553,32 @@ After the join, `region` may exist on both sides. The front end resolves `s.regi **Example 4: PromQL label matchers.** `quantile_over_time(0.9, latency{job="api"}[1m])` and the same with `job="web"`. Each Scan's `job` column has lineage `latency.job`. The lifted matchers become selections `latency.job ∈ {api}` and `latency.job ∈ {web}`: disjoint selections on the same column, so the two KLL states may merge. +#### Value sign + +Some error contracts hold only for values of a known sign: a relative error of a sum or average, DDSketch's relative error on a quantile, and Count-Min's estimates, which need non-negative weights. Each output column of a group has a sign, a group property derived once from the input groups, like lineage: + +```rust +/// GroupProperties::column_sign holds one per output column. +pub enum ValueSign { NonNegative, NonPositive, Unknown } +``` + +A source's signs are declared per column in the data workload's source facts (for example, `latency`'s value is non-negative). A PromQL counter's value is non-negative without a declaration. A column without a declaration is `Unknown`. + +| Operator | Output column sign | +|---|---| +| `Scan` | the declared sign of each source column; `Unknown` when none is declared | +| `Filter`, `TimeRange`, `TimeShift`, `Sort`, `Limit`, `Dedup`, column `Project` | the input's, unchanged | +| `abs` | `NonNegative` | +| negation | flipped: `NonNegative` ↔ `NonPositive` | +| sum, avg, min, max, quantile, top-k | the input value's sign | +| count, distinct count, `rate`, `increase` | `NonNegative` | +| `SummaryAgg`, chain | the state records its input value's sign | +| `SummaryMerge` | the inputs' sign when they all agree, else `Unknown` | +| readouts | the state's recorded sign | +| any other expression | `Unknown` | + +**Example.** `topk(5, sum by (host)(gauge))` over a gauge with no declared sign: the sum's sign is `Unknown`, so Stage 2 prunes the Count-Min heap builds (`CountMinWeightsNotProvenNonNegative`) and the query uses an exact top-k. Over `rate(requests_total[1m])`, the sum is `NonNegative` and the Count-Min builds stay. + ### Stage 1, Pass 1: initial candidates Implementation locations: `crates/logical-asap-optimization/src/pass1/per_computation_candidates.rs` and `memo_population.rs`. The #1203 target gives every intermediate result its own group; candidates contain one operator rather than private operator chains. @@ -476,11 +587,11 @@ Implementation locations: `crates/logical-asap-optimization/src/pass1/per_comput |---|---| | Exact | Exact candidate in the result group | | Exact accumulator | Accumulator-state group plus a finalize candidate in the result group | -| Summary | One state group per supported family; one build candidate per listed parameter configuration; separate readout candidates | - -Pass 1 candidates carry no parameter floors, not even a top-k's heap_size ≥ k: Pass 2 sets every floor once all candidates exist. +| Summary | One state group per supported summary type; one build candidate per listed θ; separate readout candidates | | Fused build | Several operators of one sub-DAG pattern, in one appearance of the sub-expression, fused into one candidate operator; `fused_groups` lists the groups of the operators it replaces | +Pass 1 candidates carry no input requirements: Pass 2 gives readouts their requirements once all candidates exist. + ```rust pub struct CandidateForMemoGroupId { pub group: MemoGroupId, @@ -512,9 +623,8 @@ impl Memo { /// Replaces candidate `id`'s input requirements r^log_{c,i}, one per /// input group. The only way to change a candidate's r^log: Pass 2 sets - /// each parameter floor from the input's caps and direct needs after the - /// rules have run. Panics when the vector does not have one entry per - /// input group. + /// a readout's required accuracy after the rules have run. Panics when + /// the vector does not have one entry per input group. pub fn set_input_requirements( &mut self, id: CandidateForMemoGroupId, @@ -556,8 +666,8 @@ pub struct LogicalProperties { impl LogicalProperties { /// Rejects parameters missing on a sketch build (a chain included), - /// present on an exact operator, or not in the family's list (which - /// also rejects another family's parameters). + /// present on an exact operator, or not in the algorithm's list (which + /// also rejects another algorithm's parameters). pub fn new( expression: &CandidateExpression, parameters: Option, @@ -566,26 +676,25 @@ impl LogicalProperties { pub fn parameters(&self) -> Option<&SketchParams>; } -pub fn parameter_list(family: SketchAlgorithm) -> &'static [SketchParams]; +pub fn parameter_list(algorithm: SketchAlgorithm) -> &'static [SketchParams]; -pub struct LogicalRequirements { - parameter_floor_for_required_accuracy: Option, -} - -impl LogicalRequirements { - /// Any floor of the family's parameter type, listed or not. - pub fn new(parameter_floor_for_required_accuracy: Option) -> Self; - - pub fn parameter_floor_for_required_accuracy(&self) -> Option<&SketchParams>; -} +/// What a summary type answers and how its states combine; one row per +/// summary type. +pub struct SummaryTypeCapability { /* … */ } +/// α^log of one edge (Section 2): for a readout's input, guarantee(summary +/// type, θ_u, statistic) at the required δ is within the required error, and +/// a top-k's heap holds k; for any other input, the metric rule (Section 4). pub fn producer_satisfies_logical_requirements( - producer: &LogicalProperties, - requirement: &LogicalRequirements, + memo: &Memo, + producer: CandidateForMemoGroupId, + consumer: CandidateForMemoGroupId, + input_index_of_consumer: usize, + evidence: &AccuracyEvidence, ) -> bool; ``` -The floor representation has the applicability condition stated in Section 2. Creating requirements also needs the input family's configuration domain; a constructor without that context can validate only the information encoded in its argument. Error-model interfaces are grouped with accuracy in Section 4. +`LogicalRequirements` and `RequiredAccuracy` are defined in Section 2. Error-model interfaces are grouped with accuracy in Section 4. **Fused groups.** A fused candidate computes, as one operator, a sub-DAG pattern of several operators in one appearance of the sub-expression; `fused_groups` lists the groups of the operators it replaces. Reusing a common sub-expression across appearances is Pass 2's reuse, not fusion. Fusion rules: - top-k (a sort and limit over an aggregate, as one heap summary); @@ -605,8 +714,8 @@ Pass 2 (`crates/stage1-pass2-reuse-candidates`) runs these steps in order: |---|---|---| | 1. Rollup from the finest common group | Adds candidates that compute a coarser group from a finer one, and synthesizes one finest common group per cluster of equal definitions | Its candidates are new consumers that later steps must see | | 2. Top-k reuse | Reads a compatible larger top-k result with an exact-input requirement | Same reason | -| 3. Reach, caps and floors | Computes which queries reach each candidate along candidate edges, the validity and pruning caps (Section 4), and each input's parameter floor, written with `Memo::set_input_requirements` | Needs every consumer added by steps 1 and 2 | -| 4. Dead-candidate elimination | Removes candidates no query can use, transitively, and records the root cause of each | Needs the floors; keeps step 5 from building windows for dead candidates | +| 3. Reach, caps and readout requirements | Computes which queries reach each candidate along candidate edges, the validity and pruning caps (Section 4), and each readout group's requirement levels: one readout candidate per level, its `RequiredAccuracy` written with `Memo::set_input_requirements` | Needs every consumer added by steps 1 and 2 | +| 4. Dead-candidate elimination | Removes candidates no query can use, transitively, and records the root cause of each | Needs the requirements; keeps step 5 from building windows for dead candidates | | 5. Chain groups | Adds tumbling-window chain groups and the merges that read them, with widths from the queries that reach each state group | Needs the reach of step 3, including rollup consumers, and only the survivors of step 4 | | 6. Logical edges | Records every (producer, consumer, input) and whether it satisfies the input's logical requirement | Runs on the final memo | @@ -696,15 +805,15 @@ Stage 3 chooses among: | ② KLL and merge | S_jh with KLL; S_j, S_h merged from it | 5 000 KLLs | Q1 merges 5 000 KLLs into 10, Q2 into 1 000 | | ③ Hydra | S_jh with HydraKll; all three read it | one structure, its size set by θ, not by the number of pairs | one subpopulation query per group | -With many pairs and queries ③ is cheapest; with few pairs ① or ② can be. Hydra's error per subpopulation follows from its θ and enters (K) on the edges like any readout's. +With many pairs and queries ③ is cheapest; with few pairs ① or ② can be. A subpopulation readout has a required accuracy like any readout, and α^log checks Hydra's guarantee for it at θ. -#### Reach, caps and floors +#### Reach, caps and readout requirements -A query reaches a candidate when a path of candidate edges leads from the query's output group to it; this is the reach $a_{q,v}$, computed top down after steps 1 and 2 so that rollup consumers count. Section 4 defines the caps from it. Each input's floor is the strongest of the floor for its cap (`floor_for_cap` in `asap-accuracy`) and any direct need such as heap_size ≥ k under a top-k. `Memo::set_input_requirements` writes it; nothing else changes a candidate's r^log. +A query reaches a candidate when a path of candidate edges leads from the query's output group to it; this is the reach $a_{q,v}$, computed top down after steps 1 and 2 so that rollup consumers count. Section 4 defines the caps from it, and which requirement levels each readout group gets ("Readout requirements"). Pass 1's readout candidate takes the first level and `add_candidate` adds one readout candidate per further level; `Memo::set_input_requirements` writes each one's `RequiredAccuracy`, and nothing else changes a candidate's r^log. Two readout candidates with the same expression and different requirements are different candidates. #### Dead-candidate elimination -A candidate that is not a query output is **dead** when it never had a consumer, or when every edge to a consumer fails: the producer is below the input's floor, the metric rule forbids the edge (Section 4), or the consumer is dead. Elimination repeats until nothing changes. Dead candidates get no logical edges and Stage 2 does not expand them; each is recorded with its root causes: +A candidate that is not a query output is **dead** when it never had a consumer, or when every edge to a consumer fails: the producer's guarantee misses the input's required accuracy, the metric rule forbids the edge (Section 4), or the consumer is dead. Elimination repeats until nothing changes. Dead candidates get no logical edges and Stage 2 does not expand them; each is recorded with its root causes: ```rust pub struct LogicalASAPMemoCandidates { @@ -717,7 +826,8 @@ pub struct LogicalASAPMemoCandidates { pub struct PrunedLogicalCandidate { pub candidate: CandidateForMemoGroupId, /// Every cause: one per failing edge it is cut off by, so mixed causes - /// (a floor on one edge, the metric rule on another) are all listed. + /// (a missed requirement on one edge, the metric rule on another) are + /// all listed. pub causes: Vec, } @@ -730,11 +840,12 @@ pub struct PruneCause { } pub enum RootPruneReason { - /// The producer's parameters are weaker than the consumer input's floor. - BelowAccuracyFloorRequirement { + /// The producer's guarantee, at the required failure probability, + /// exceeds the consumer input's required error. + BelowRequiredAccuracy { consumer: CandidateForMemoGroupId, input: usize, - floor: SketchParams, + required: RequiredAccuracy, }, /// The consumer's error contract carries none of the producer's metrics. MetricRule { @@ -749,34 +860,35 @@ pub enum RootPruneReason { } ``` -**Example: ratio of quantiles.** `quantile_over_time(0.99, m[5m]) / quantile_over_time(0.5, m[5m])` with relative error 0.05. +**Example: ratio of quantiles.** `quantile_over_time(0.99, m[5m]) / quantile_over_time(0.5, m[5m])` with relative error 0.05, no failure probability, and `m` declared non-negative. | Group | Expression | Candidates | |---|---|---| | G1 | `Scan m` | | | G2 | `TimeRange(5m, G1)` | | -| G3 | `SummaryAgg[KLL](G2)` | #0 … #6: k = 132 … 2863 | -| G3′ | `SummaryAgg[DDSketch](G2)` | #0 … #6: alpha = 0.1, 0.05, 0.02, 0.01, 0.005, 0.002, 0.001 | -| G4 | p99 | #0 exact; #1 `SummaryEstimate[q=0.99](G3)`; #2 `SummaryEstimate[q=0.99](G3′)` | -| G5 | p50 | #0 exact; #1 `SummaryEstimate[q=0.5](G3)`; #2 `SummaryEstimate[q=0.5](G3′)` | +| G3 | `SummaryAgg[Sketch(Kll)](G2)` | #0 … #6: k = 26 … 2863 | +| G3′ | `SummaryAgg[Sketch(DDSketch)](G2)` | #0 … #6: alpha = 0.1, 0.05, 0.02, 0.01, 0.005, 0.002, 0.001 | +| G4 | p99 | #0 exact; #1 `SummaryEstimate[q=0.99](G3)`; #2–#6 `SummaryEstimate[q=0.99](G3′)` requiring relative 0.02, 0.01, 0.005, 0.002, 0.001 | +| G5 | p50 | #0 exact; #1 `SummaryEstimate[q=0.5](G3)`; #2–#6 `SummaryEstimate[q=0.5](G3′)` requiring relative 0.02 … 0.001 | | G6 | `Divide(G4, G5)` | #0, the query output | -The Divide carries only relative error. Its inputs' cap is 0.05 / 1.111 = 0.045 (Section 4), so the DDSketch readouts' floor is alpha ≤ 0.02. Pass 2 records: +The Divide carries only relative error. Its inputs' cap is 0.05 / 1.111 = 0.045 (Section 4), so the readouts of G3′ get the relative levels 0.02 … 0.001 of the error grid; 0.1 and 0.05 are looser than the cap. The KLL readouts are shown once: the metric rule removes them whatever their level. Pass 2 records: ``` G4#1 SummaryEstimate[q=0.99](G3) G4#1 → G6#0 input 0: MetricRule { producer_metric: Rank, consumer_carries: [Relative] } G5#1 SummaryEstimate[q=0.5](G3) G5#1 → G6#0 input 1: MetricRule { producer_metric: Rank, consumer_carries: [Relative] } -G3#0 SummaryAgg[KLL] k=132 (and likewise G3#1 … G3#6) +G3#0 SummaryAgg[Sketch(Kll)] k=26 (and likewise G3#1 … G3#6) G3#0 → G4#1 → G6#0 input 0: MetricRule { producer_metric: Rank, consumer_carries: [Relative] } G3#0 → G5#1 → G6#0 input 1: MetricRule { producer_metric: Rank, consumer_carries: [Relative] } -G3′#0 SummaryAgg[DDSketch] alpha=0.1 (and likewise G3′#1, alpha=0.05) - G3′#0 → G4#2 input 0: BelowAccuracyFloorRequirement { floor: alpha 0.02 } - G3′#0 → G5#2 input 0: BelowAccuracyFloorRequirement { floor: alpha 0.02 } +G3′#0 SummaryAgg[Sketch(DDSketch)] alpha=0.1 (and likewise G3′#1, alpha=0.05) + G3′#0 → G4#2 input 0: BelowRequiredAccuracy { required: relative 0.02 } + G3′#0 → G5#2 input 0: BelowRequiredAccuracy { required: relative 0.02 } + (and one cause per looser readout, G4#3 … G4#6, G5#3 … G5#6) ``` -The KLL builds' own edges satisfy their consumers; they are dead because their only consumers are, and the trace leads through those consumers to the metric rule. The exact candidates and the DDSketch builds with alpha ≤ 0.02 survive. +The KLL builds' own edges may satisfy their consumers; they are dead because their only consumers are, and the trace leads through those consumers to the metric rule. The exact candidates, every DDSketch readout, and the DDSketch builds with alpha ≤ 0.02 survive; each build connects only to the readouts whose level it meets (alpha 0.01 to the levels 0.02 and 0.01). #### Chain groups @@ -788,7 +900,7 @@ A state group whose expression is `SummaryAgg(TimeRange(W, TimeShift*(X)))` can // (lineage `Computed`); the build's own time column is dropped; the unique // key is (window_end, keys). No range and no window count: how much it // retains is h_v. -SummaryTumblingWindowChain { child: X, family, input, reduction, grouping, filter, +SummaryTumblingWindowChain { child: X, summary_type, input, reduction, filter, tumbling_window_width_ms: u64 } // Merges every input state, or regroups them; no window-selection field. SummaryMerge { children, group_by } @@ -802,11 +914,10 @@ SummaryMerge { children, group_by } 3. Re-wrap the stripped shifts and then the `TimeRange` around $G_w$, in their original order, each with `find_or_add_group`. 4. `add_candidate(S, SummaryMerge(…))` over the outermost re-wrapped group. Its coverage equals S's (Section 3, Stage 0), so the candidate is accepted. -There is one chain per width, shared by every state group over the same X and family. +There is one chain per width, shared by every state group over the same X and summary type. **Widths.** Each query q that reaches S (step 3) and repeats with a known evaluation grid $g_q$ (`evaluation_grid_ms` is `Some`) gives the width $\gcd(g_q, W, o_1, \ldots)$, with the offsets $o_j$ on its path; S's widths are the gcd closure of these. Queries without a known grid give none, so a group read only by such queries gets no chain. `@` modifiers: -- `@ end()` and `@ start()` in an instant query equal t; the front end lowers them as no `@`, so they need nothing here. -- `@ end()` and `@ start()` in a range-evaluated query are not lifted (Section 3), so the walk stops at their `TimeShift` and the state group gets no chain. +- `@ end()` and `@ start()`: the anchored subtree X is lowered without the `@` (Section 3), so a state group inside X is an ordinary state group. In a range query it is reached in the evaluation context of the request's end times, whose grid gives its widths like any query's. - `@ T` (absolute) contributes no widths: S gets a merge only over chains that other queries' widths created, and only for widths that divide T. Whether reusing the chain is cheaper than computing the result once and materializing it is Stage 3's choice; its retention need is in Section 5. **Example: three queries share one chain.** A = `sum_over_time(m[5m])`, B = `sum_over_time(m[5m] offset 1h)`, C = `sum_over_time(m[1h])`, each `FixedIntervalAt { 60 s, phase 0 }`. The widths are gcd(60 s, 5m) = gcd(60 s, 5m, 1h) = gcd(60 s, 1h) = 60 s. @@ -854,11 +965,11 @@ pub struct LogicalEdge { } ``` -An edge satisfies the logical requirement when the producer meets the input's floor and the consumer's error contract carries the producer's metric (Section 4). +An edge satisfies the logical requirement when α^log = 1 (`producer_satisfies_logical_requirements`): into a readout's input, the producer's guarantee meets the required accuracy (Section 2); into any other input, the consumer's error contract carries the producer's metric (Section 4). ### Stage 2: physical alternatives -`enumerate_candidate_configurations` in `crates/stage2-physical-optimization/src/catalog/configurations.rs` expands each logical candidate that survived Pass 2 using the dimensions and checks in Section 2. It offers ℓ = AcrossEvaluations only to chain candidates; the other candidates never get that value, and nothing is recorded for them. It creates no new logical candidates. Each logical edge is expanded between the relevant physical alternatives. +`enumerate_candidate_configurations` in `crates/stage2-physical-optimization/src/catalog/configurations.rs` expands each logical candidate that survived Pass 2 using the dimensions and checks in Section 2. It offers τ = IngestionTime only to a candidate that is time-independent (its output does not depend on the evaluation time t, as a chain's does not) or that is an input of one. It offers ℓ = AcrossEvaluations to chains and to the candidates of any group evaluated on a known grid: a time-independent result is materialized as is, a time-dependent one per evaluation time, as a time-indexed result series. Values not offered are not recorded. It creates no new logical candidates. Each logical edge is expanded between the relevant physical alternatives. Stage 2 reads the logical candidates, the data workload (per-source arrival, below) and the deployment's capabilities: @@ -904,8 +1015,9 @@ Stage 2 records every alternative it rejects, with one of these reasons; Stage 3 | `QueryTimeResultsNotRetained` | QueryTime with ℓ = OneEvaluation or AcrossEvaluations when the deployment retains no query-time results | | `InputHasNoBuildableProducer` | a candidate for which some input has no surviving producer that satisfies α^phys | | `ReachesNoQueryOutput` | a candidate from which no admissible path leads to a query root | +| `CountMinWeightsNotProvenNonNegative` | a Count-Min build (`Cms`, `CmsWithHeap`, `HydraCms`) whose weight column's `column_sign` is not `NonNegative`: its estimates are bounds only for non-negative weights | -The planner does not check executor support, so no reason names it. +These are the only reasons. The planner does not check executor support, so no reason names it. ```rust // crates/types/src/physical/properties.rs @@ -943,15 +1055,19 @@ impl PhysicalCandidate { /// Whether a consumer running in `context` can read this producer /// (Section 2's alignment table): for a chain producer with /// ℓ = AcrossEvaluations, w (the chain operator's width) divides every - /// evaluation time of the context (`evaluation_grid_ms`); true for ℓ = None, and for OneEvaluation in the - /// same context. + /// evaluation time of the context (`evaluation_grid_ms`); for a + /// materialized time-dependent result series, every evaluation time of + /// the context lies on its grid; true for ℓ = None, for OneEvaluation in + /// the same context, and for a materialized time-independent result. + /// `IngestionPipeline` and `Materialized(v)` contexts are handled the + /// same way, with their own schedules. pub fn delivers_on(&self, memo: &Memo, context: &ExecutionContext) -> bool; } /// α^phys: timing and lifetime only (Section 2's table). Alignment is /// checked per context in Stage 3 with `delivers_on`. pub fn producer_satisfies_physical_requirements( producer: &PhysicalCandidate, - consumer: &PhysicalCandidate, + consumer_timing: ExecutionTiming, ) -> bool; pub struct PhysicalCandidateId(pub usize); pub struct PhysicalEdgeId(pub usize); @@ -1003,7 +1119,7 @@ impl PhysicalASAPMemoCandidates { Migration targets include replacing `Materialization` and duplicated physical fields with checked properties, and replacing `ReaderConstraint::{IncompatibleOutputs, AllowedOutputs}` with compatibility and alignment checks. `ReaderConstraint::{MinimumQueryReaders, HighestSharedQueryMaterialization}` are deleted too; see Section 5. -### Stage 3: baseline selection and realization +### Stage 3: baseline selection and plan output The baseline uses the formulation's variables and constraints. Stage 3 adds no candidates or potential dependencies. @@ -1028,13 +1144,13 @@ The baseline uses the formulation's variables and constraints. Stage 3 adds no c Constraints are applied for every relevant candidate, input, query, or potential edge. An implementation may allocate edge variables only on admissible edges, making (G) and (H) implicit without changing the baseline feasible set. -The baseline objective charges each selected candidate a fixed cost over the common horizon: +The baseline objective charges each selected candidate its own cost and each selected edge the cost of that read, over the common horizon: $$ -\min\sum_{v\in\mathcal C}w_vx_v. +\min\sum_{v\in\mathcal C}w_vx_v+\sum_{(u,c,i)\in\mathcal E}c_{u,c,i}\,y_{u,c,i}. $$ -The cost includes input reading, the candidate's own computation, and lifetime-dependent retention. It excludes producers' computation, which is charged at those producers. This objective assumes execution frequencies and retention costs are fixed before optimization and independent of the selected consumers. When that assumption does not hold, use the execution extension in Section 5; do not silently reuse the same cost coefficient. +A cost term goes on the edge exactly when it changes if another producer of the same group is chosen: reading or transferring u's output, and the part of c's work that scales with u's form (its θ, its number of windows). Everything else is in $w_v$: v's own build or ingestion from the logical size of its inputs, its own storage $\pi_v h_v$, and its own output. So no producer has to stand for the others, and every term is exact. Section 5 indexes both terms by execution context. #### When the candidate graph has a cycle @@ -1070,24 +1186,59 @@ struct SelectionVariables { // utilization_constraints: (F) // Compatibility is enforced when allocating y on admissible edges. -pub struct SelectedEdge { - pub producer: SelectedNodeId, - pub consumer: SelectedNodeId, - pub input_index_of_consumer: usize, +``` + +#### Plan output + +Stage 3 builds the plan directly as its last step: one `PhysicalASAPDAG` node per selected execution, its inputs from the selected edges. The plan is the only plan structure, in memory and, serialized, in the plan file installed on the executor: + +```rust +pub struct PlanOutput { + pub plan: PhysicalASAPDAG, +} + +pub struct PhysicalASAPDAGNode { + // operator, inputs, timing, lifetime, … + /// How long the node's time-indexed rows are retained (h_v); set on + /// materialized nodes. + pub retention_ms: Option, + /// Set only on AcrossEvaluations nodes. + pub summary_coverage: Option, + /// The physical candidate the node was built from: in-memory provenance + /// for validation and tools, not part of the plan file; None for a plan + /// built by a custom pass. + #[serde(skip)] + pub selected_physical_candidate: Option, } -pub struct DagOfSelectedCandidates { - pub candidate_of_node: Vec, - pub edges: Vec, - pub node_of_output_reference: Vec, +/// What Stage 3 reports about its own run, beside the plan. +pub struct Selection { + pub accepted_accuracy_per_query: Vec, + pub pruned_candidates: Vec>, + pub milp_variables: usize, + pub milp_constraints: usize, + pub milp_construction_ms: u64, + pub milp_solve_ms: u64, } + +// Signature schematic. +pub fn select_plan( + physical: &PhysicalASAPMemoCandidates, + // cost inputs, … +) -> Result<(PhysicalASAPDAG, Selection), ErrorSelectingPlan>; ``` -The realizer uses the selected dependencies to build the executable graph. Export carries each node's timing and lifetime. Under the context extension, multiple selected execution nodes may refer to the same physical candidate; realization must preserve their contexts rather than merge them solely by candidate ID. +The plan's cost is `plan_cost(&plan)`; `Selection` does not repeat it. Validation checks the DAG's structure and, through `selected_physical_candidate`, that every node and input is admissible in the memo. Under the context extension, several nodes may come from one physical candidate, one per context it runs in; they are not merged by candidate. + +**Scans and windows.** A `Scan` is an ordinary node. The time window a consumer reads is the `TimeRange` and `TimeShift` nodes above it; the executor pushes their time bounds into the source read. How long raw data is retained is the Scan node's retention, $h_v$ under (R) (Section 5); a scan's read cost is on its edges. + +#### Executing the plan + +The executor keeps one time-indexed, append-only store with retention for every materialized node with a time index: raw data (a Scan), chain states, and results stored per evaluation time. Each is read only through `TimeRange` and `TimeShift` nodes, which select its rows by time, and rows older than the node's `retention_ms` are dropped. A `TimeShift { at: End | Start }` node evaluates its child once per request. Summary operators are constructed with their θ; a summary node without θ is an error. ## 4. Accuracy and safe pruning — Extension (#1207) -The formulation checks logical requirements per edge. This extension also checks composed accuracy at each query root. This section gives the error model and its MILP encoding; error is not a property dimension. +The formulation checks logical requirements per edge. This extension also checks composed accuracy at each query root. This section gives the error model, how readouts get their requirements, and the MILP encoding; error is not a property dimension. ### Local propagation @@ -1115,7 +1266,7 @@ $$ | avg aggregate, absolute error | β = 1 | | Ratio, relative error | Requires nonzero true denominator and denominator relative error below one; bound given below | | Product, relative error | (1 + b) for the left operand and 1 for the right, whose relative error is capped at b: $(1+x)(1+y)-1\le(1+b)x+y$ when $y\le b$ | -| sum / avg, relative error | Holds when the values are known to be nonnegative; not offered until that sign information is an input of the error model | +| sum / avg, relative error | β = 1 when the input column's `column_sign` is `NonNegative` or `NonPositive` (all values of one sign); no contract when it is `Unknown` | A constant contract is unconditional: it needs no row count or sign. A filter does not automatically preserve every accuracy guarantee; a `Filter` that compares an approximate value (an alert threshold such as `quantile_over_time(…) > 0.5`) has no contract. Group counts may need metadata beyond the expression syntax. Each factor must follow from the operator's error contract. @@ -1128,7 +1279,7 @@ $$ \frac{\varepsilon_X+\varepsilon_Y}{1-\varepsilon_Y}. $$ -Choosing a fixed validity cap $b<1$ for the denominator gives a linear upper bound within that region. The design fixes $b = 0.1$, for a ratio's denominator and a product's right operand alike: 0.1 is the weakest ε in every family's list, so the cap itself never excludes a listed configuration and only raises β by about 11%: +Choosing a fixed validity cap $b<1$ for the denominator gives a linear upper bound within that region. The design fixes $b = 0.1$, for a ratio's denominator and a product's right operand alike: 0.1 is the weakest ε in every algorithm's list, so the cap itself never excludes a listed configuration and only raises β by about 11%: $$ \varepsilon_Y\le b @@ -1162,9 +1313,33 @@ impl LogicalRequirements { `factor_per_input` holds the local β factors and `validity_cap_per_input` the per-input caps, aligned by input index. An operator without an error contract for a metric returns `None`; no factor is guessed. Every contract keeps the metric: the input's and the output's are the same. For the ratio query of Section 3, `error_propagation(Divide, Relative)` is `Some { factor_per_input: [1.111, 1.111], validity_cap_per_input: [None, Some(0.1)] }`, and the absolute and rank calls return `None`. The key-only `Filter` contract reads the input group's column lineage. +### Readout requirements + +Errors enter a plan only at readouts, and a readout's error is its requirement (Section 2). Pass 2 decides which requirements each readout group gets: + +- **A query with one readout** on its path gets one readout candidate, whose requirement is the query target divided by the path's factor λ (1 through pass-through operators). +- **A readout group that feeds a composed query** (several readouts combined, such as a ratio) gets one readout candidate per requirement level of the grid ε ∈ {0.1, 0.05, 0.02, 0.01, 0.005, 0.002, 0.001} × δ ∈ {1e-2, 1e-4, 1e-6}, minus the levels looser than the input's cap (below) or than the query's per-value δ. A target without a failure probability gives one δ level, δ = 1. Constraint (K) then chooses how the query's error is split among its readouts, together with the producers. +- A readout group reached by several queries gets the union of their levels. + +**Failure probability per value.** A target's δ covers the whole result unless its scope is per value. A query enters the memo with $n_q$, the number of values one evaluation returns, and Pass 2 turns a whole-result δ_q into the per-value δ_q / n_q for the readout requirements; Stage 3 needs no $n_q$. + +**Example: allocation in a ratio.** Query B = p99 / p50 with relative error 0.03, no failure probability, over `latency` declared non-negative. Each operand's cap is 0.03 / 1.111 = 0.027, so G4 (p99) and G5 (p50) each get readouts at the levels 0.02, 0.01, 0.005, 0.002 and 0.001. A DDSketch build with alpha ≤ ε meets the level ε (Section 2). (K) requires 1.111 (ε_p99 + ε_p50) ≤ 0.03: + +| p99 level | p50 level | Composed bound | Meets 0.03 | Builds it can read | +|---|---|---|---|---| +| 0.02 | 0.02 | 0.044 | no | | +| 0.02 | 0.01 | 0.033 | no | | +| 0.01 | 0.01 | 0.022 | yes | one alpha-0.01 build read by both | +| 0.02 | 0.005 | 0.028 | yes | alpha 0.02 for p99 and alpha 0.005 for p50, or one alpha-0.005 build | +| 0.005 | 0.02 | 0.028 | yes | the mirror of the row above | + +Stage 3 chooses the cheapest feasible row with its producers: usually one shared alpha-0.01 build. + +**Ignoring accuracy.** When accuracy is not enforced, the facade sets every query's target to "no accuracy target" before Pass 2: readouts get no requirement, every θ connects (a top-k's heap still holds k), and (K) has no rows. + ### Query composition and root checks -An **error source** is one local approximation event with a stated metric and guarantee. Let $s$ index such sources in a selected, acyclic plan. Let $\varepsilon_s$ and $\delta_s$ be its bound and failure probability. Define $\lambda_{q,s}$ as the sum, over paths from that source to the query output, of products of local beta factors. Define $a_{q,s}$ as a binary indicator that query q's selected dependency graph uses that source. +An **error source** is a selected readout, with its requirement's error $\varepsilon_s$ and failure probability $\delta_s$. Define $\lambda_{q,s}$ as the sum, over paths from that source to the query output, of products of local beta factors, and $a_{q,s}$ as a binary indicator that query q's selected dependency graph uses that source. The intended root conditions are: @@ -1174,39 +1349,57 @@ $$ \sum_s\delta_s a_{q,s}\le\delta_q. $$ -Here, $\varepsilon_q$ and $\delta_q$ are the query target's error and failure limits. Repeated paths can amplify error and therefore contribute to lambda. A shared failure event is counted once in the union bound; distinct readout events need distinct source identities unless a joint guarantee is available. +Here, $\varepsilon_q$ is the query target's error and $\delta_q$ its per-value failure probability. Repeated paths can amplify error and therefore contribute to lambda. -**MILP encoding.** Path factors $\lambda_{q,s}$ depend on which edges are selected, so the MILP does not use them directly. It carries an error bound per selected candidate instead, which needs no path enumeration and is the same for every query reading that candidate: +**MILP encoding.** Path factors $\lambda_{q,s}$ depend on which edges are selected, so the MILP does not use them directly. It carries an error bound per selected candidate and per metric m instead, which needs no path enumeration and is the same for every query reading that candidate: $$ -E_c \ge \varepsilon^{\mathrm{loc}}_c + \sum_{i} \beta_{c,i} F_{c,i}, +E_{c,m} \ge \varepsilon_{c}\,[m = m_c] + \sum_{i} \beta_{c,i,m} F_{c,i,m}, \qquad -F_{c,i} \ge E_u - M_{c,i}\,(1 - y_{u,c,i}) \quad \forall (u,c,i)\in\mathcal E, +F_{c,i,m} \ge E_{u,m} - M_{c,i}\,(1 - y_{u,c,i}) \quad \forall (u,c,i)\in\mathcal E, $$ $$ -D_c \ge \delta^{\mathrm{loc}}_c + \sum_{i} H_{c,i}, +D_c \ge \delta_c + \sum_{i} H_{c,i}, \qquad H_{c,i} \ge D_u - (1 - y_{u,c,i}) \quad \forall (u,c,i)\in\mathcal E, $$ $$ -E_v \le \varepsilon_q + M_q\,(1 - z_{q,v}), +E_{u,m} \le M_{c,i}\,(1 - y_{u,c,i})\quad\text{when } c\text{'s contract has no metric } m, +$$ + +$$ +E_{v,m_q} \le \varepsilon_q + M_q\,(1 - z_{q,v}), +\qquad +E_{v,m} \le M_q\,(1 - z_{q,v})\ \ (m \ne m_q), \qquad D_v \le \delta_q + (1 - z_{q,v}) \qquad \forall q,\ v\in G_q. \tag{K} $$ -- $E_c, F_{c,i}, D_c, H_{c,i} \ge 0$ are continuous. $\varepsilon^{\mathrm{loc}}_c$ and $\delta^{\mathrm{loc}}_c$ are the candidate's own guarantee (zero for an exact operator); $\beta_{c,i}$ and the metric rule come from `error_propagation`. -- $M_{c,i}$ is any upper bound on $E_u$ over the producers of input $i$ (their largest feasible bound); $M_q$ likewise for the root. Exactly one $y_{u,c,i} = 1$ per selected input (constraint (A)), so $F_{c,i}$ equals the chosen producer's error bound. -- A producer reached by two paths contributes on both, which is the worst-case sum. The failure probabilities add by a union bound; a failure event reached twice is counted twice, which is conservative. This affects only the error bounds, not cost: the objective counts each selected candidate once through $x_v$. Being conservative can only make the MILP pass over a cheaper plan that would in fact meet the target; it never accepts one that does not. Counting a shared failure event once needs failure-event identities, left for later. -- Under the context extension (Section 5), the variables are indexed by context, $E_{\kappa,c}$, and the edges by $y_{\kappa,u,c,i}$. +- $E_{c,m}, F_{c,i,m}, D_c, H_{c,i} \ge 0$ are continuous. $(\varepsilon_c, \delta_c)$ is a readout's requirement, in its metric $m_c$; it is zero for every other candidate. $\beta_{c,i,m}$ and the metric rule come from `error_propagation(consumer, m)`. +- $M_{c,i}$ is any upper bound on $E_{u,m}$ over the producers of input $i$; $M_q$ likewise for the root. Exactly one $y_{u,c,i} = 1$ per selected input (constraint (A)), so $F_{c,i,m}$ equals the chosen producer's bound. +- The third line is the metric rule on the selected path: a consumer whose contract has no metric m can read only a producer with no error in m. The root accepts error only in the target's metric. +- A producer reached by two paths contributes on both, which is the worst-case sum. The failure probabilities add by a union bound; a failure event reached twice is counted twice, which is conservative. This affects only the error bounds, not cost. Being conservative can only make the MILP pass over a cheaper plan that would in fact meet the target; it never accepts one that does not. Counting a shared failure event once needs failure-event identities, left for later. +- Under the context extension (Section 5), the variables are indexed by context, $E_{\kappa,c,m}$, and the edges by $y_{\kappa,u,c,i}$. + +**Example: why the bound is per metric.** `sort(quantile_over_time(0.99, latency[1m])) / quantile_over_time(0.5, latency[1m])` with relative error 0.03. + +| Group | Expression | Candidates | +|---|---|---| +| G4 | p99 | a readout of the KLL state, rank 0.001; readouts of the DDSketch state, relative ε | +| G7 | `Sort(G4)` | value-preserving: β = 1 in every metric | +| G5 | p50 | readouts of the DDSketch state, relative ε | +| G6 | `Divide(G7, G5)`, the query output | relative contract only | + +The Divide → Sort edge survives Pass 2: the Sort may carry relative error. With one number per candidate, choosing KLL would give $E_{G7}$ = 0.001 and a composed bound (0.001 + ε_p50) / 0.9 ≤ 0.03, accepted, which is wrong: a rank error bounds no relative error. Per metric, $E_{G7,\mathrm{rank}}$ = 0.001, and since the Divide's contract has no rank, $E_{G7,\mathrm{rank}} \le M(1 - y)$ forces that edge off when G7 reads KLL; only the DDSketch readout remains. -$\lambda_{q,c,i}$ is still used, but only for the pruning caps below, where a fixed lower bound suffices. +$\lambda_{q,c,i}$ is still used, but only for the caps below, where a fixed lower bound suffices. ### Validity caps and pruning caps -A **validity cap** ensures that a chosen local bound applies. A **pruning cap** removes an alternative only when its nonnegative contribution alone exceeds every query target it could serve. Neither reserves an error-budget share for one input. +A **validity cap** ensures that a chosen local bound applies. A **pruning cap** bounds the requirement levels a readout group gets: a level whose error alone exceeds every query target it could serve is not offered. Neither reserves an error-budget share for one input. For an input whose error can be expressed in the same metric, let $Q_{c,i}$ be the queries that could use it. Let $\lambda_{q,c,i}$ denote that input's query-level factor, and let $b_{c,i}$ denote its validity cap, or infinity when no cap is required. With fixed positive factors, a safe cap is: @@ -1216,9 +1409,9 @@ $$ \frac{\varepsilon_q}{\lambda_{q,c,i}}\right). $$ -A zero factor gives no pruning restriction for that query. If a factor depends on the eventual plan, use a proven lower bound over all feasible downstream alternatives for this pruning test, or defer pruning. Using an arbitrary path's larger factor could discard a feasible plan. The maximum over queries protects alternatives that can serve at least one query. +A zero factor gives no pruning restriction for that query. If a factor depends on the eventual plan, use a proven lower bound over all feasible downstream alternatives, or offer every level. Using an arbitrary path's larger factor could discard a feasible plan. The maximum over queries protects levels that can serve at least one query. -Compute the parameter floor from this cap only under the ordering condition in Section 2. `floor_for_cap` in `asap-accuracy` maps a cap to the weakest listed configuration whose ε meets it, for a family and statistic; Pass 2 takes the strongest of it and any direct need such as heap_size ≥ k. For example, with relative error 0.05 at a ratio, each operand's cap is 0.05 / 1.111 = 0.045 and the denominator's is min(0.1, 0.045) = 0.045, so DDSketch's floor is alpha ≤ 0.02. Validity information is available when the consumer's error model is established; query-sensitive pruning follows logical generation and reachability analysis. Root checks still couple all contributing errors and determine feasibility. +For example, with relative error 0.05 at a ratio, each operand's cap is 0.05 / 1.111 = 0.045 and the denominator's is min(0.1, 0.045) = 0.045, so the readouts get the levels 0.02 and tighter. Root checks still couple all contributing errors and determine feasibility. ## 5. Execution contexts and other selection extensions — #1207 @@ -1284,7 +1477,7 @@ Plan 2, a shared chain: | Gw in κw | one window per 10 s, built from the samples as they arrive | one computation, whoever reads it | | G3② in κ10 | 0.1 | Q_A and Q_B; Gw delivers on κ10 (10 s divides 10 s) | | G3② in κ30 | 0.033 | Q_C; Gw delivers on κ30 (10 s divides 30 s) | -| $h_{Gw}$ | 1 min, 6 windows | (R): the longest need of its selected readers | +| $h_{Gw}$ | 1 min, 6 windows | (R): the longest need of its selected consumers | The MILP compares the two plans' cost per second. With much data, plan 2's single incremental build is usually cheaper; with little data, plan 1 can be. @@ -1314,7 +1507,7 @@ Every term is exact: no producer is chosen to stand for others. In the example o ### No consumer-count policies -Today's `ReaderConstraint` (`crates/types/src/stages/physical/graph.rs`) is deleted with all four variants (decided 2026-11-06). `IncompatibleOutputs` and `AllowedOutputs` are replaced by the alignment check of Section 2. `MinimumQueryReaders` (keep a candidate only if enough queries reach it) and `HighestSharedQueryMaterialization` (materialize where sharing stops) are left to the cost model: materializing for one query, or below the point where sharing stops, costs more and is not chosen. There is no constraint (J). +Today's `ReaderConstraint` (`crates/types/src/stages/physical/graph.rs`) is deleted with all four variants. `IncompatibleOutputs` and `AllowedOutputs` are replaced by the alignment check of Section 2. `MinimumQueryReaders` (keep a candidate only if enough queries reach it) and `HighestSharedQueryMaterialization` (materialize where sharing stops) are left to the cost model: materializing for one query, or below the point where sharing stops, costs more and is not chosen. There is no constraint (J). ### Out of this design @@ -1365,12 +1558,8 @@ fn expand( // of the admissible producers in G_{c,i}. for choice in cartesian_product(inputs.iter().enumerate().map(|(i, &group)| { candidates_of(memo, group) - .filter(|&u| { - producer_satisfies_logical_requirements( - &memo.candidate(u).logical_properties, - &candidate.input_requirements[i], - ) - }) + // α^log, read from the logical edge (u, consumer, i). + .filter(|&u| satisfies_logical_requirement(u, consumer, i)) .collect::>() })) { // In the stored consumer-to-producer direction, this would close a cycle. @@ -1398,6 +1587,14 @@ fn candidates_of( .map(move |index_in_group| CandidateForMemoGroupId { group, index_in_group }) } +/// α^log of the logical edge (u, consumer, i): `satisfies_logical_requirement` +/// of `LogicalEdge`. +fn satisfies_logical_requirement( + u: CandidateForMemoGroupId, + consumer: CandidateForMemoGroupId, + i: usize, +) -> bool; + /// Whether `from` reaches `to` through the producers chosen so far /// (true when `from == to`, so a candidate never reads itself). fn reaches(dag: &LogicalDag, from: CandidateForMemoGroupId, to: CandidateForMemoGroupId) -> bool; From 271ca7db1155a0c93654dbbf52e67c680c5aaef5 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sun, 8 Nov 2026 19:51:03 +0000 Subject: [PATCH 50/67] =?UTF-8?q?docs(concepts):=20=CE=B4=20levels=201e-2.?= =?UTF-8?q?.1e-5=20(CMS=20depths=205/7/10/12),=20result-series=20grid,=20D?= =?UTF-8?q?DSketch=20sign=20either=20way?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../concepts/memo-groups-properties-requirements.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 152e0b835..8594b0ac0 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -94,7 +94,7 @@ This is the single reference table for property dimensions. Stage descriptions b | Dimension | Values and meaning | Code reference or target | |---|---|---| -| Parameters θ | The structure of a summary, from a predefined list per algorithm, `parameter_list(algorithm)`: KLL and KMV `k`; DDSketch `alpha`; HLL `precision`; CMS and CountSketch `width` and `depth`; heap variants also `heap_size`; UnivMon its L2 column count and distinct-count heap size. θ is structure only: no field records the error or failure probability a configuration was sized for, and equality is derived over all fields. The lists span today's sizing over ε = 0.1 down to 0.001 (KMV's and UnivMon's distinct-count entries stop at ε = 0.01). Where δ is set by a structural parameter, the list covers the δ levels with it: CMS lists each width with depths 5, 10 and 14 (δ = e^−d ≤ 1e-2, 1e-4, 1e-6). UnivMon's list has 40 entries: each of the 7 L2 column counts with each of the 4 distinct-count heap sizes, each of them alone, and the baseline. A heap variant combines each width × depth with heap sizes from the 1-2-5 series 10, 20, 50, …, 10 000, so the smallest heap that holds a top-k's k is at most 2.5× too large. A sketch build's θ must be in its algorithm's list; exact operators have none. Algorithms without an error model (the Hydra algorithms among them) have empty lists, so no build of theirs is a valid candidate until their error models exist (#1228). | `SketchParams` in `crates/types/src/ir/schema/sketch.rs`; `parameter_list` | +| Parameters θ | The structure of a summary, from a predefined list per algorithm, `parameter_list(algorithm)`: KLL and KMV `k`; DDSketch `alpha`; HLL `precision`; CMS and CountSketch `width` and `depth`; heap variants also `heap_size`; UnivMon its L2 column count and distinct-count heap size. θ is structure only: no field records the error or failure probability a configuration was sized for, and equality is derived over all fields. The lists span today's sizing over ε = 0.1 down to 0.001 (KMV's and UnivMon's distinct-count entries stop at ε = 0.01). Where δ is set by a structural parameter, the list covers the δ levels with it: CMS and CountSketch list each width with depths 5, 7, 10 and 12 (δ = e^−d ≤ 1e-2, 1e-3, 1e-4, 1e-5). UnivMon's list has 40 entries: each of the 7 L2 column counts with each of the 4 distinct-count heap sizes, each of them alone, and the baseline. A heap variant combines each width × depth with heap sizes from the 1-2-5 series 10, 20, 50, …, 10 000, so the smallest heap that holds a top-k's k is at most 2.5× too large. A sketch build's θ must be in its algorithm's list; exact operators have none. Algorithms without an error model (the Hydra algorithms among them) have empty lists, so no build of theirs is a valid candidate until their error models exist (#1228). | `SketchParams` in `crates/types/src/ir/schema/sketch.rs`; `parameter_list` | | Timing | `IngestionTime` or `QueryTime` | `ExecutionTiming` | | Lifetime | `None`: discard after the consuming execution; `OneEvaluation`: retain until consumers within that evaluation finish; `AcrossEvaluations`: retain across evaluations in the specified period. A non-`None` lifetime means materialized. | `Lifetime`; #1200 | | Storage medium — Extension | `Memory` or `Disk`, carried by a materialized lifetime | `StorageMedium` | @@ -152,7 +152,7 @@ and, for a top-k readout, `heap_size` ≥ k. An exact producer has error 0 and s **Where the guarantee comes from.** `guarantee` is `AccuracyEvidence::measured_guarantee(summary type, θ, statistic)` when the deployment's evidence has a measurement for that configuration, and `theoretical_guarantee` (the algorithm's error formula) otherwise. The source is decided before Pass 2 builds the logical edges, so (K) always has every readout's bound, and nothing is checked again after Stage 3. -**Sign conditions.** Some guarantees hold only for values of a known sign, read from the input group's `column_sign` (Section 3, "Value sign"): DDSketch's relative error on a quantile needs a `NonNegative` input column. +**Sign conditions.** Some guarantees hold only for values of a known sign, read from the input group's `column_sign` (Section 3, "Value sign"): DDSketch's relative error on a quantile needs an input column of one sign, `NonNegative` or `NonPositive` (negated, all-negative values are all-positive). **A readout's error is its requirement.** A readout candidate's output error is (ε_r, δ_r), a node property: every producer it may read meets it. Constraint (K) composes these errors to the query root (Section 4). Which requirements a readout group gets is Pass 2's choice (Section 4, "Readout requirements"): one per requirement level, each its own readout candidate. @@ -172,7 +172,7 @@ QT means QueryTime; IT means IngestionTime. Alignment, data coverage, and retent **Alignment.** There is no physical schedule property. When a result can be read follows from its lifetime and, for a chain, its window width: - **None** and **OneEvaluation**: the result is computed inside one evaluation context κ (Section 5) and read there. A OneEvaluation result belongs to one context: queries on the same evaluation grid, a range query's steps included, are evaluated together, and that evaluation's consumers share the result. When it runs is the context's schedule, and Stage 3 chooses it through $x_{\kappa,v}$, just as for ℓ = None. Sharing across differently scheduled queries through a gcd schedule is not offered: it is rarely beneficial and wastes runs. - **AcrossEvaluations** on a chain, `SummaryTumblingWindowChain[w](X)`: the input is cut into tumbling windows (k·w, (k+1)·w], with the grid anchored at phase 0 (the epoch). Windows are left-open and right-closed like a PromQL range, which `m[r]` evaluated at t reads as (t − r, t]. Each window is built when it ends and can be read from then on. A `TimeRange` over the chain selects the windows that lie fully inside (t − range, t], so a consumer can read a range whose ends lie on window boundaries. With w = 10 s, a `[1m]` query evaluated every 30 s at phase 0 merges 6 windows; one evaluated at phase 15 s cannot, because its range ends inside a window; with that query in the workload, the gcd closure also offers w = 5 s, which both can read. The `TimeRange` and `TimeShift` groups over a chain are computed within one evaluation. -- **AcrossEvaluations** on a candidate of any other group evaluated on a known grid: a time-independent result (one that does not depend on the evaluation time t, such as a result under `@ T`) is stored as is and can be read at any time; a time-dependent result is stored per evaluation time, as a time-indexed result series computed at every time of its grid. A consumer reads the value stored for its own evaluation time, or, through a `TimeShift` above it, for a shifted time; so the consumer's evaluation times must lie on the producer's grid. +- **AcrossEvaluations** on a candidate of any other group evaluated on a known grid: a time-independent result (one that does not depend on the evaluation time t, such as a result under `@ T`) is stored as is and can be read at any time; a time-dependent result is stored per evaluation time, as a time-indexed result series computed at every time of its grid. A consumer reads the value stored for its own evaluation time, or, through a `TimeShift` above it, for a shifted time; so the consumer's evaluation times must lie on the producer's grid. Its grid is the greatest common divisor of the grids of the queries that reach it, as for chain widths: a result read by queries every 60 s and every 120 s is computed every 60 s, and both read it. A grid phase is not needed: a phase is absorbed by the width, because width candidates come from the gcd closure of the queries' grids, ranges and offsets (Section 3). For example, phases 0 and 5 s with w = 10 s give w = 5 s. Alignment alone does not ensure sufficient retained history; that is $h_v$ (Section 5). @@ -1318,7 +1318,7 @@ impl LogicalRequirements { Errors enter a plan only at readouts, and a readout's error is its requirement (Section 2). Pass 2 decides which requirements each readout group gets: - **A query with one readout** on its path gets one readout candidate, whose requirement is the query target divided by the path's factor λ (1 through pass-through operators). -- **A readout group that feeds a composed query** (several readouts combined, such as a ratio) gets one readout candidate per requirement level of the grid ε ∈ {0.1, 0.05, 0.02, 0.01, 0.005, 0.002, 0.001} × δ ∈ {1e-2, 1e-4, 1e-6}, minus the levels looser than the input's cap (below) or than the query's per-value δ. A target without a failure probability gives one δ level, δ = 1. Constraint (K) then chooses how the query's error is split among its readouts, together with the producers. +- **A readout group that feeds a composed query** (several readouts combined, such as a ratio) gets one readout candidate per requirement level of the grid ε ∈ {0.1, 0.05, 0.02, 0.01, 0.005, 0.002, 0.001} × δ ∈ {1e-2, 1e-3, 1e-4, 1e-5}, minus the levels looser than the input's cap (below) or than the query's per-value δ. A target without a failure probability gives one δ level, δ = 1. Constraint (K) then chooses how the query's error is split among its readouts, together with the producers. - A readout group reached by several queries gets the union of their levels. **Failure probability per value.** A target's δ covers the whole result unless its scope is per value. A query enters the memo with $n_q$, the number of values one evaluation returns, and Pass 2 turns a whole-result δ_q into the per-value δ_q / n_q for the readout requirements; Stage 3 needs no $n_q$. From a33e4cc2f380001b164e1e78bece58e1490db1fa Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 9 Nov 2026 02:54:58 +0000 Subject: [PATCH 51/67] docs(concepts): guarantees at a failure probability, AccuracyEvidence trait, MemoQuery MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit theoretical_guarantee and AccuracyEvidence::measured_guarantee take the required failure probability and return the error bound at it (None when θ cannot fail that rarely); AccuracyEvidence is a trait with NoMeasuredGuarantees; a bound is compared through target_limit. The memo holds its queries as one Vec { root, accuracy_target, values_per_evaluation }, added by add_query(query). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 49 ++++++++++++++++--- 1 file changed, 43 insertions(+), 6 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 8594b0ac0..a13629400 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -150,7 +150,35 @@ and, for a top-k readout, `heap_size` ≥ k. An exact producer has error 0 and s | width 272, depth 5 | 0.0100 | 6.7e-3 | yes | no | | width 272, depth 10 | 0.0100 | 4.5e-5 | yes | yes | -**Where the guarantee comes from.** `guarantee` is `AccuracyEvidence::measured_guarantee(summary type, θ, statistic)` when the deployment's evidence has a measurement for that configuration, and `theoretical_guarantee` (the algorithm's error formula) otherwise. The source is decided before Pass 2 builds the logical edges, so (K) always has every readout's bound, and nothing is checked again after Stage 3. +**Where the guarantee comes from.** `guarantee` is `AccuracyEvidence::measured_guarantee(summary type, θ, statistic, δ_r)` when the deployment's evidence has a measurement for that configuration, and `theoretical_guarantee(summary type, θ, statistic, δ_r)` (the algorithm's error formula) otherwise. Each returns the error bound that holds at failure probability δ_r, or `None` when θ cannot fail that rarely (no edge). The bound is compared with ε_r through `target_limit(ε_r)`, which admits only floating-point noise above ε_r. The source is decided before Pass 2 builds the logical edges, so (K) always has every readout's bound, and nothing is checked again after Stage 3. + +```rust +/// The theoretical error bound of reading `statistic` from a state of +/// `summary_type` with parameters θ at `failure_probability`. +pub fn theoretical_guarantee( + summary_type: &FieldDataType, + parameters: &SketchParams, + statistic: &SketchStatistic, + failure_probability: f64, +) -> Option; + +/// The deployment's measured accuracy, read before the theoretical bound. +pub trait AccuracyEvidence { + fn measured_guarantee( + &self, + summary_type: &FieldDataType, + parameters: &SketchParams, + statistic: &SketchStatistic, + failure_probability: f64, + ) -> Option; +} +/// Evidence with no measurement (the default). +pub struct NoMeasuredGuarantees; +impl AccuracyEvidence for NoMeasuredGuarantees { … } // always None + +/// The largest evaluated error (or failure probability) that meets `limit`. +pub fn target_limit(limit: f64) -> f64; +``` **Sign conditions.** Some guarantees hold only for values of a known sign, read from the input group's `column_sign` (Section 3, "Value sign"): DDSketch's relative error on a quantile needs an input column of one sign, `NonNegative` or `NonPositive` (negated, all-negative values are all-positive). @@ -299,7 +327,7 @@ Rules add groups through `find_or_add_group` and candidates through `add_candida ### Stage 0: groups -`memo_of_queries` in `crates/stage0-memo` builds the groups bottom-up from the queries' nodes: each node's operator, over its children's groups, goes through `Memo::find_or_add_group`, so identical sub-expressions are one group; each query's root is then added with `Memo::add_query`, with the query's accuracy target and $n_q$, the number of values one evaluation returns (Pass 2 turns a whole-result failure probability into a per-value one with it). Stage 0 and Stage 1 share one type, `Memo`: Stage 0 returns it with every group's candidate list empty, and Stage 1 fills the lists. +`memo_of_queries` in `crates/stage0-memo` builds the groups bottom-up from the queries' nodes: each node's operator, over its children's groups, goes through `Memo::find_or_add_group`, so identical sub-expressions are one group; each query is then added with `Memo::add_query` as a `MemoQuery`: its root, its accuracy target and $n_q$, the number of values one evaluation returns (Pass 2 turns a whole-result failure probability into a per-value one with it). Stage 0 and Stage 1 share one type, `Memo`: Stage 0 returns it with every group's candidate list empty, and Stage 1 fills the lists. ```rust pub struct MemoGroupId(pub u32); @@ -370,15 +398,24 @@ pub struct Memo { // Private: groups are added only through `find_or_add_group`, so the // hash-consing index covers every group. groups: Vec, - pub query_roots: Vec>, + /// The queries, by index in the workload. + pub queries: Vec, // Private: the groups `find_or_add_group` may return, by expression hash. groups_by_expression_hash: HashMap>, } +pub struct MemoQuery { + pub root: QueryRoot, + /// A_q; `None` when the query has none (accuracy not enforced). + pub accuracy_target: Option, + /// n_q; `None` when the data workload does not bound it. + pub values_per_evaluation: Option, +} + impl Memo { pub fn groups(&self) -> &[MemoGroup]; - /// Adds a query with this root; its groups and those below list it. - pub fn add_query(&mut self, root: QueryRoot) -> usize; + /// Adds a query; its root's groups and those below list it. + pub fn add_query(&mut self, query: MemoQuery) -> usize; } ``` @@ -690,7 +727,7 @@ pub fn producer_satisfies_logical_requirements( producer: CandidateForMemoGroupId, consumer: CandidateForMemoGroupId, input_index_of_consumer: usize, - evidence: &AccuracyEvidence, + evidence: &dyn AccuracyEvidence, ) -> bool; ``` From 344f321c251f3a0a07d1455a79950c046f44c59e Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 9 Nov 2026 02:59:11 +0000 Subject: [PATCH 52/67] docs(concepts): Count Sketch depths, delivers_on over Stage 2's candidates, evaluation grids of groups MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Count Sketch lists depths 47, 81, 117 and 155 per width (the median-of-rows δ levels 1e-2 … 1e-5); CMS keeps 5, 7, 10, 12. delivers_on takes &PhysicalASAPMemoCandidates, so Materialized(v) reads v. A stored result that is not a chain delivers by GroupProperties::depends_on_evaluation_time and LogicalASAPMemoCandidates::evaluation_grid_of_group, with the sum_over_time(m[5m]) read every 60 s and 120 s example. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 25 +++++++++++++++---- 1 file changed, 20 insertions(+), 5 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index a13629400..782db76e3 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -94,7 +94,7 @@ This is the single reference table for property dimensions. Stage descriptions b | Dimension | Values and meaning | Code reference or target | |---|---|---| -| Parameters θ | The structure of a summary, from a predefined list per algorithm, `parameter_list(algorithm)`: KLL and KMV `k`; DDSketch `alpha`; HLL `precision`; CMS and CountSketch `width` and `depth`; heap variants also `heap_size`; UnivMon its L2 column count and distinct-count heap size. θ is structure only: no field records the error or failure probability a configuration was sized for, and equality is derived over all fields. The lists span today's sizing over ε = 0.1 down to 0.001 (KMV's and UnivMon's distinct-count entries stop at ε = 0.01). Where δ is set by a structural parameter, the list covers the δ levels with it: CMS and CountSketch list each width with depths 5, 7, 10 and 12 (δ = e^−d ≤ 1e-2, 1e-3, 1e-4, 1e-5). UnivMon's list has 40 entries: each of the 7 L2 column counts with each of the 4 distinct-count heap sizes, each of them alone, and the baseline. A heap variant combines each width × depth with heap sizes from the 1-2-5 series 10, 20, 50, …, 10 000, so the smallest heap that holds a top-k's k is at most 2.5× too large. A sketch build's θ must be in its algorithm's list; exact operators have none. Algorithms without an error model (the Hydra algorithms among them) have empty lists, so no build of theirs is a valid candidate until their error models exist (#1228). | `SketchParams` in `crates/types/src/ir/schema/sketch.rs`; `parameter_list` | +| Parameters θ | The structure of a summary, from a predefined list per algorithm, `parameter_list(algorithm)`: KLL and KMV `k`; DDSketch `alpha`; HLL `precision`; CMS and CountSketch `width` and `depth`; heap variants also `heap_size`; UnivMon its L2 column count and distinct-count heap size. θ is structure only: no field records the error or failure probability a configuration was sized for, and equality is derived over all fields. The lists span today's sizing over ε = 0.1 down to 0.001 (KMV's and UnivMon's distinct-count entries stop at ε = 0.01). Where δ is set by a structural parameter, the list covers the δ levels with it: CMS lists each width with depths 5, 7, 10 and 12 (δ = e^−d ≤ 1e-2, 1e-3, 1e-4, 1e-5). CountSketch reads the median of its rows, each wrong with probability about 1/3, so δ = P(Binomial(d, 1/3) ≥ (d+1)/2), defined for odd d: it lists each width with depths 47, 81, 117 and 155 for the same δ levels. UnivMon's list has 40 entries: each of the 7 L2 column counts with each of the 4 distinct-count heap sizes, each of them alone, and the baseline. A heap variant combines each width × depth with heap sizes from the 1-2-5 series 10, 20, 50, …, 10 000, so the smallest heap that holds a top-k's k is at most 2.5× too large. A sketch build's θ must be in its algorithm's list; exact operators have none. Algorithms without an error model (the Hydra algorithms among them) have empty lists, so no build of theirs is a valid candidate until their error models exist (#1228). | `SketchParams` in `crates/types/src/ir/schema/sketch.rs`; `parameter_list` | | Timing | `IngestionTime` or `QueryTime` | `ExecutionTiming` | | Lifetime | `None`: discard after the consuming execution; `OneEvaluation`: retain until consumers within that evaluation finish; `AcrossEvaluations`: retain across evaluations in the specified period. A non-`None` lifetime means materialized. | `Lifetime`; #1200 | | Storage medium — Extension | `Memory` or `Disk`, carried by a materialized lifetime | `StorageMedium` | @@ -352,6 +352,11 @@ pub struct GroupProperties { pub column_lineage: Vec, /// Per output column: its value sign ("Value sign"). pub column_sign: Vec, + /// Whether the result changes with the evaluation time: true under a + /// `TimeRange` relative to it (`sum_over_time(m[5m])`), false under a + /// fixed `@ T` or with no time at all. A stored result that does not + /// depend on it delivers on every context. + pub depends_on_evaluation_time: bool, /// What an aggregate group covers: a summary state, a chain, a merge /// over a chain's windows, or an exact `Aggregate` result; `None` for /// any other group. @@ -850,13 +855,17 @@ A query reaches a candidate when a path of candidate edges leads from the query' #### Dead-candidate elimination -A candidate that is not a query output is **dead** when it never had a consumer, or when every edge to a consumer fails: the producer's guarantee misses the input's required accuracy, the metric rule forbids the edge (Section 4), or the consumer is dead. Elimination repeats until nothing changes. Dead candidates get no logical edges and Stage 2 does not expand them; each is recorded with its root causes: +A candidate that is not a query output is **dead** when it never had a consumer, or when every edge to a consumer fails: the producer's guarantee misses the input's required accuracy, the metric rule forbids the edge (Section 4), or the consumer is dead. Elimination repeats until nothing changes. Dead candidates get no logical edges and Stage 2 does not expand them; each is recorded with its root causes. Pass 2 also records each group's evaluation grid, which `delivers_on` reads for a stored result that is not a chain. Example: X = `sum_over_time(m[5m])` is read by a query every 60 s and by another every 120 s, both at phase 0; X's grid is 60 s @ 0. X materialized across evaluations is stored every 60 s, so it delivers on `Evaluation { 60 s @ 0 }` and `Evaluation { 120 s @ 0 }` but not on `Evaluation { 30 s @ 0 }`. X under `@ T` does not depend on the evaluation time and delivers on every context. ```rust pub struct LogicalASAPMemoCandidates { pub memo: Memo, pub edges: Vec, pub pruned: Vec, + /// Per group: the grid its result is evaluated on, the coarsest grid + /// holding every evaluation time of the queries reaching it; `None` + /// when those times are not on one known grid. Pass 2 fills it. + pub evaluation_grid_of_group: Vec>, } /// A candidate eliminated by Pass 2. @@ -1097,8 +1106,14 @@ impl PhysicalCandidate { /// the context lies on its grid; true for ℓ = None, for OneEvaluation in /// the same context, and for a materialized time-independent result. /// `IngestionPipeline` and `Materialized(v)` contexts are handled the - /// same way, with their own schedules. - pub fn delivers_on(&self, memo: &Memo, context: &ExecutionContext) -> bool; + /// same way, with their own schedules (v's read from `candidates`). A + /// result's grid is `evaluation_grid_of_group`; whether it depends on + /// the time, `GroupProperties::depends_on_evaluation_time`. + pub fn delivers_on( + &self, + candidates: &PhysicalASAPMemoCandidates, + context: &ExecutionContext, + ) -> bool; } /// α^phys: timing and lifetime only (Section 2's table). Alignment is /// checked per context in Stage 3 with `delivers_on`. @@ -1152,7 +1167,7 @@ impl PhysicalASAPMemoCandidates { } ``` -`PhysicalCandidateId` indexes an emitted candidate. A pruned alternative is stored as its `PhysicalCandidate` value, which has no ID, so diagnostics can explain each prune. The admissible-edge accessor returns edge IDs, not the unfiltered producer set from Section 1. `delivers_on` takes a candidate and the memo rather than bare properties because a tumbling grid's width is an argument of the chain operator in the candidate's expression. +`PhysicalCandidateId` indexes an emitted candidate. A pruned alternative is stored as its `PhysicalCandidate` value, which has no ID, so diagnostics can explain each prune. The admissible-edge accessor returns edge IDs, not the unfiltered producer set from Section 1. `delivers_on` takes a candidate and Stage 2's candidates rather than bare properties because a tumbling grid's width is an argument of the chain operator in the candidate's expression, a stored result's grid is its group's `evaluation_grid_of_group`, and a `Materialized(v)` context's times are v's. Migration targets include replacing `Materialization` and duplicated physical fields with checked properties, and replacing `ReaderConstraint::{IncompatibleOutputs, AllowedOutputs}` with compatibility and alignment checks. `ReaderConstraint::{MinimumQueryReaders, HighestSharedQueryMaterialization}` are deleted too; see Section 5. From ae2551e3e7459c4db9b6f8b91c131d2c8eb268fe Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 9 Nov 2026 04:04:25 +0000 Subject: [PATCH 53/67] docs(concepts): declared column signs in the memo, value-sign rows for counts, rates, comparisons and readouts Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01WviZ6FUmrMfBTvVbBauNM6 --- .../memo-groups-properties-requirements.md | 27 ++++++++++++------- 1 file changed, 18 insertions(+), 9 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 782db76e3..0d54f30e3 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -353,9 +353,10 @@ pub struct GroupProperties { /// Per output column: its value sign ("Value sign"). pub column_sign: Vec, /// Whether the result changes with the evaluation time: true under a - /// `TimeRange` relative to it (`sum_over_time(m[5m])`), false under a - /// fixed `@ T` or with no time at all. A stored result that does not - /// depend on it delivers on every context. + /// `TimeRange` relative to it (`sum_over_time(m[5m])`) or an expression + /// reading it (`time()`, `now()`), false under a fixed `@ T` or with no + /// time at all. A stored result that does not depend on it delivers on + /// every context. pub depends_on_evaluation_time: bool, /// What an aggregate group covers: a summary state, a chain, a merge /// over a chain's windows, or an exact `Aggregate` result; `None` for @@ -407,6 +408,9 @@ pub struct Memo { pub queries: Vec, // Private: the groups `find_or_add_group` may return, by expression hash. groups_by_expression_hash: HashMap>, + // Private: the sign the data workload declares for each source column it + // declares one for; a `Scan`'s `column_sign` reads it ("Value sign"). + declared_column_signs: BTreeMap, } pub struct MemoQuery { @@ -418,6 +422,8 @@ pub struct MemoQuery { } impl Memo { + /// A memo without groups whose Scans read `declared_column_signs`. + pub fn new(declared_column_signs: BTreeMap) -> Self; pub fn groups(&self) -> &[MemoGroup]; /// Adds a query; its root's groups and those below list it. pub fn add_query(&mut self, query: MemoQuery) -> usize; @@ -604,7 +610,7 @@ Some error contracts hold only for values of a known sign: a relative error of a pub enum ValueSign { NonNegative, NonPositive, Unknown } ``` -A source's signs are declared per column in the data workload's source facts (for example, `latency`'s value is non-negative). A PromQL counter's value is non-negative without a declaration. A column without a declaration is `Unknown`. +A source's signs are declared per column in the data workload's source facts (for example, `latency`'s value is non-negative). A PromQL counter's value is non-negative without a declaration. A column without a declaration is `Unknown`. `memo_of_queries` collects the declared signs of the sources the queries scan into the memo (`Memo::new`), and `find_or_add_group` reads them for a `Scan`. | Operator | Output column sign | |---|---| @@ -612,11 +618,14 @@ A source's signs are declared per column in the data workload's source facts (fo | `Filter`, `TimeRange`, `TimeShift`, `Sort`, `Limit`, `Dedup`, column `Project` | the input's, unchanged | | `abs` | `NonNegative` | | negation | flipped: `NonNegative` ↔ `NonPositive` | -| sum, avg, min, max, quantile, top-k | the input value's sign | -| count, distinct count, `rate`, `increase` | `NonNegative` | -| `SummaryAgg`, chain | the state records its input value's sign | -| `SummaryMerge` | the inputs' sign when they all agree, else `Unknown` | -| readouts | the state's recorded sign | +| numeric literal, numeric cast | the literal's sign; the operand's sign | +| sum, avg, min, max, quantile, top-k, `first_over_time`, `last_over_time` | the input value's sign | +| count, distinct count, and the other counts and spreads (`count_values`, `changes`, `resets`, `group`, `absent`, `present_over_time`, `stddev`, `stdvar`, frequency L2 norm and entropy) | `NonNegative` | +| `rate`, `irate`, `increase` | `NonNegative` over a `NonNegative` input (a counter); else `Unknown`, since a decrease reads as a reset | +| vector comparison, `and`, `unless`; `or`; comparison with `bool` | the left side's value sign; the sides' sign when they agree; `NonNegative` (0 or 1) | +| `SummaryAgg`, chain | the state records the sign of the values it is updated with (`window_end` is `Unknown`) | +| `SummaryMerge` | the inputs' sign when they all agree, else `Unknown`; over a chain's windows, the chain's build's | +| readouts | the state's recorded sign; a distinct count, frequency L2 norm or entropy, and an exact count's finalization: `NonNegative` | | any other expression | `Unknown` | **Example.** `topk(5, sum by (host)(gauge))` over a gauge with no declared sign: the sum's sign is `Unknown`, so Stage 2 prunes the Count-Min heap builds (`CountMinWeightsNotProvenNonNegative`) and the query uses an exact top-k. Over `rate(requests_total[1m])`, the sum is `NonNegative` and the Count-Min builds stay. From b8c7056b171a72f864ab4ae4f61473579b071ce1 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 9 Nov 2026 06:05:23 +0000 Subject: [PATCH 54/67] docs(concepts): lineage over chains, window selections, @ end()/@ start() and merges; lineage and sign share one column layout Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01WviZ6FUmrMfBTvVbBauNM6 --- .../concepts/memo-groups-properties-requirements.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 0d54f30e3..9893cda22 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -553,14 +553,17 @@ A range-evaluated query evaluates its expression at every step t of [start, end] | Operator | Output column lineage | |---|---| | `Scan` | each column: `SourceColumn(SourceColumn { source, column })` | -| `Filter`, `TimeRange`, `TimeShift`, `Sort`, `Limit`, `Dedup` | the input's, unchanged | +| `Filter`, `TimeRange`, `TimeShift`, `Sort`, `Limit`, `Dedup` | the input's, unchanged; also a `TimeRange` or `TimeShift` over a chain (it selects windows) and `TimeShift { at: End \| Start }` above a selector | | `Project` | a column item, renamed or not: that input column's lineage; an expression item: `Computed` | | `Aggregate`, `SummaryAgg`, readouts | group-by keys: the input's lineage of that key; measures, states and values: `Computed` | | `SummaryTumblingWindowChain` | `window_end` (column 0): `Computed`; keys: the input's lineage of that key; state: `Computed` | +| `SummaryMerge` | per column: the inputs' lineage if they all agree, else `Computed`; over a chain's selected windows, the chain's build's (keys, then state); a regrouping merge: its keys', then the state | | `Join` | the left input's columns, then the right input's | | `SetOp`, `Concat` | per column: the inputs' lineage if they all agree, else `Computed` | | `BinaryOp` (vector match) | label columns: the matched side's lineage; the value: `Computed` | +Lineage and sign (Value sign, below) are derived together, by one per-operator column layout: an output column that copies an input column keeps its lineage and sign; a computed one is `Computed`, with the sign the operator's rule gives. + **What reads it:** - **Summary coverage.** A selection constrains columns keyed by their `SourceColumn` lineage, not by (qualifier, name). A predicate on a `Computed` column is not lifted; it stays in `unlifted_operators`. A merge's group keys must have equal lineage. - **Candidate checks.** `add_candidate` compares a candidate's schema with its group's exactly: names, types, nullability, order. There is no qualifier to compare, so a readout renamed to the group's column name matches. From e4f878703effc4ffdffe0d1984486be9fa510aeb Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 9 Nov 2026 09:25:47 +0000 Subject: [PATCH 55/67] docs(concepts): inputs of an ingestion-time materialized candidate run in its refresh window's pipeline; delivers_on in Materialized(v); constraints per execution Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01WviZ6FUmrMfBTvVbBauNM6 --- .../design_docs/concepts/memo-groups-properties-requirements.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 9893cda22..004d1dc6f 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -1504,7 +1504,7 @@ pub struct EvaluationGrid { pub period_ms: u64, pub phase_ms: u64 } **Evaluation grids, and range queries.** An evaluation context is a grid of evaluation times, not a query. A range query is expanded into evaluations on its grid: with step 10 s, phase 0, each of its steps is an evaluation at a time of grid 10 s @ 0, so its steps and an instant query on the same grid are the same context and share runs. How many requests cover each grid time (61 for a 1 h span at 1 min steps refreshed every minute) multiplies the cost; it does not create another context. The subtree under `TimeShift { at: End }` in a range query runs on the grid of request end times, the one an instant query with the request's recurrence uses (Section 3, `@` handling). -**Lifetimes and contexts.** A candidate with ℓ = None or OneEvaluation runs inside a context: once per execution of every context that selects it ($x_{\kappa,v}$). A candidate with ℓ = AcrossEvaluations runs once, in its own context `Materialized(v)` ($x^{\mathrm{mat}}_v$); consumers in other contexts read what it has stored, through edges whose producer must deliver on the consumer's context (`delivers_on`, Section 2). Its inputs run in its context. A ℓ = None producer has at most one consumer per context ($\sum_{c,i} y_{\kappa,u,c,i} \le x_{\kappa,u}$); consumers that share a result in one context read a OneEvaluation producer, or a materialized one. +**Lifetimes and contexts.** A candidate with ℓ = None or OneEvaluation runs inside a context: once per execution of every context that selects it ($x_{\kappa,v}$). A candidate with ℓ = AcrossEvaluations runs once, in its own context `Materialized(v)` ($x^{\mathrm{mat}}_v$); consumers in other contexts read what it has stored, through edges whose producer must deliver on the consumer's context (`delivers_on`, Section 2). Its inputs run in its context at query time; at ingestion time they run in the ingestion pipeline of its refresh window (its chain width, else its group's evaluation grid), so the ingestion-time builds of one window share them. `delivers_on` of a producer read in `Materialized(v)` uses v's own times: every window end of a chain v, else its group's evaluation grid. A ℓ = None producer has at most one consumer per context ($\sum_{c,i} y_{\kappa,u,c,i} \le x_{\kappa,u}$); consumers that share a result in one context read a OneEvaluation producer, or a materialized one. The constraints of Section 3 hold per execution: (A) for each execution of a consumer and each input, with $y$ indexed by the consumer's context; $t$ and (E) per execution, again only inside a strongly connected component of the selectable edges. | Variable | Meaning | |---|---| From 950b9b39470f233eab9559d51191f760061cbd34 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 9 Nov 2026 11:32:39 +0000 Subject: [PATCH 56/67] docs(concepts): RawDataNotRetained also rejects a Scan materialized across evaluations Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01WviZ6FUmrMfBTvVbBauNM6 --- .../design_docs/concepts/memo-groups-properties-requirements.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 004d1dc6f..f6beccb65 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -1069,7 +1069,7 @@ Stage 2 records every alternative it rejects, with one of these reasons; Stage 3 | Reason | Rejects | |---|---| | `NoIngestionPipeline` | IngestionTime when the source has no ingestion pipeline | -| `RawDataNotRetained` | QueryTime over raw rows the deployment does not retain | +| `RawDataNotRetained` | a Scan of raw rows the deployment does not retain, at QueryTime or AcrossEvaluations | | `QueryTimeResultsNotRetained` | QueryTime with ℓ = OneEvaluation or AcrossEvaluations when the deployment retains no query-time results | | `InputHasNoBuildableProducer` | a candidate for which some input has no surviving producer that satisfies α^phys | | `ReachesNoQueryOutput` | a candidate from which no admissible path leads to a query root | From 7f9e985632314f8eef0a7b1f2653647bb96cae7f Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 9 Nov 2026 12:00:55 +0000 Subject: [PATCH 57/67] docs(concepts): a candidate's own cost reads the logical size over the producers it may read; ingestion time reads one batch of raw data Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01WviZ6FUmrMfBTvVbBauNM6 --- .../design_docs/concepts/memo-groups-properties-requirements.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index f6beccb65..5936f1145 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -1214,7 +1214,7 @@ $$ \min\sum_{v\in\mathcal C}w_vx_v+\sum_{(u,c,i)\in\mathcal E}c_{u,c,i}\,y_{u,c,i}. $$ -A cost term goes on the edge exactly when it changes if another producer of the same group is chosen: reading or transferring u's output, and the part of c's work that scales with u's form (its θ, its number of windows). Everything else is in $w_v$: v's own build or ingestion from the logical size of its inputs, its own storage $\pi_v h_v$, and its own output. So no producer has to stand for the others, and every term is exact. Section 5 indexes both terms by execution context. +A cost term goes on the edge exactly when it changes if another producer of the same group is chosen: reading or transferring u's output, and the part of c's work that scales with u's form (its θ, its number of windows). Everything else is in $w_v$: v's own build or ingestion from the logical size of its inputs (the most any producer v may read, on an admissible edge, outputs; an ingestion-time consumer reads one batch of raw data, also from a Scan materialized at ingestion time), its own storage $\pi_v h_v$, and its own output. So no producer has to stand for the others, and every term is exact. Section 5 indexes both terms by execution context. #### When the candidate graph has a cycle From fcb8cd0fa840bfb7ad0b7d7b99ce7e894d6acd4b Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 9 Nov 2026 16:48:28 +0000 Subject: [PATCH 58/67] =?UTF-8?q?docs(concepts):=20merges=20carry=20=CE=B8?= =?UTF-8?q?,=20metric=20rule=20cuts=20edges,=20held=20memory=20per=20node,?= =?UTF-8?q?=20scan=20read=20window,=20@=20T=20without=20horizon,=20reads?= =?UTF-8?q?=5Fonly=5Ftime=5Fseries?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - A candidate passing states on (SummaryMerge, Filter/TimeRange/TimeShift over states) carries the θ of the states it reads, one per θ, and reads only producers with that θ; elimination runs again after chain groups. - The metric rule is applied per edge from the metrics each group's output accepts; a readout whose metric its output does not accept reads no state; a candidate cut off from an input is eliminated too. - ValuesPerEvaluationUnknown: a whole-result δ with n_q unknown prunes the readouts it reaches. - Stage 2 offers no (IngestionTime, None) summary build and no materialized alternative under @ T without a horizon end; QueryTimeResultsNotRetained rejects only (QueryTime, AcrossEvaluations). - Query-time memory is held only while the node computes; query latency is left for later. A query-time Scan reads the largest offset + range of its selected consumers. - GroupProperties::reads_only_time_series. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01WviZ6FUmrMfBTvVbBauNM6 --- .../memo-groups-properties-requirements.md | 95 +++++++++++++------ 1 file changed, 64 insertions(+), 31 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 5936f1145..b2537e3d2 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -94,16 +94,16 @@ This is the single reference table for property dimensions. Stage descriptions b | Dimension | Values and meaning | Code reference or target | |---|---|---| -| Parameters θ | The structure of a summary, from a predefined list per algorithm, `parameter_list(algorithm)`: KLL and KMV `k`; DDSketch `alpha`; HLL `precision`; CMS and CountSketch `width` and `depth`; heap variants also `heap_size`; UnivMon its L2 column count and distinct-count heap size. θ is structure only: no field records the error or failure probability a configuration was sized for, and equality is derived over all fields. The lists span today's sizing over ε = 0.1 down to 0.001 (KMV's and UnivMon's distinct-count entries stop at ε = 0.01). Where δ is set by a structural parameter, the list covers the δ levels with it: CMS lists each width with depths 5, 7, 10 and 12 (δ = e^−d ≤ 1e-2, 1e-3, 1e-4, 1e-5). CountSketch reads the median of its rows, each wrong with probability about 1/3, so δ = P(Binomial(d, 1/3) ≥ (d+1)/2), defined for odd d: it lists each width with depths 47, 81, 117 and 155 for the same δ levels. UnivMon's list has 40 entries: each of the 7 L2 column counts with each of the 4 distinct-count heap sizes, each of them alone, and the baseline. A heap variant combines each width × depth with heap sizes from the 1-2-5 series 10, 20, 50, …, 10 000, so the smallest heap that holds a top-k's k is at most 2.5× too large. A sketch build's θ must be in its algorithm's list; exact operators have none. Algorithms without an error model (the Hydra algorithms among them) have empty lists, so no build of theirs is a valid candidate until their error models exist (#1228). | `SketchParams` in `crates/types/src/ir/schema/sketch.rs`; `parameter_list` | +| Parameters θ | The structure of a summary, from a predefined list per algorithm, `parameter_list(algorithm)`: KLL and KMV `k`; DDSketch `alpha`; HLL `precision`; CMS and CountSketch `width` and `depth`; heap variants also `heap_size`; UnivMon its L2 column count and distinct-count heap size. θ is structure only: no field records the error or failure probability a configuration was sized for, and equality is derived over all fields. The lists span today's sizing over ε = 0.1 down to 0.001 (KMV's and UnivMon's distinct-count entries stop at ε = 0.01). Where δ is set by a structural parameter, the list covers the δ levels with it: CMS lists each width with depths 5, 7, 10 and 12 (δ = e^−d ≤ 1e-2, 1e-3, 1e-4, 1e-5). CountSketch reads the median of its rows, each wrong with probability about 1/3, so δ = P(Binomial(d, 1/3) ≥ (d+1)/2), defined for odd d: it lists each width with depths 47, 81, 117 and 155 for the same δ levels. UnivMon's list has 40 entries: each of the 7 L2 column counts with each of the 4 distinct-count heap sizes, each of them alone, and the baseline. A heap variant combines each width × depth with heap sizes from the 1-2-5 series 10, 20, 50, …, 10 000, so the smallest heap that holds a top-k's k is at most 2.5× too large. A sketch build's θ must be in its algorithm's list; exact operators have none. A candidate that passes states on without building them (`SummaryMerge`, and a `Filter`, `TimeRange` or `TimeShift` over states) carries the θ of the states it reads: Pass 2 adds one such candidate per θ among its input group's candidates, and its input edges connect only to producers with the same θ (Section 3, "Logical edges"). So every candidate of a sketch state group has a θ, and a readout's guarantee is that of its producer's θ. Algorithms without an error model (the Hydra algorithms among them) have empty lists, so no build of theirs is a valid candidate until their error models exist (#1228). | `SketchParams` in `crates/types/src/ir/schema/sketch.rs`; `parameter_list` | | Timing | `IngestionTime` or `QueryTime` | `ExecutionTiming` | | Lifetime | `None`: discard after the consuming execution; `OneEvaluation`: retain until consumers within that evaluation finish; `AcrossEvaluations`: retain across evaluations in the specified period. A non-`None` lifetime means materialized. | `Lifetime`; #1200 | | Storage medium — Extension | `Memory` or `Disk`, carried by a materialized lifetime | `StorageMedium` | -An error guarantee is not a property: it depends on the summary type, θ and the statistic a readout reads, and is checked on the edge from a build to a readout (below). It has an error metric, a bound, and a failure probability; exact computation introduces no approximation error. +An error guarantee is not a property: it depends on the summary type, θ and the statistic a readout reads, and is checked on the edge from a state candidate (a build, or a merge with the θ it carries) to a readout (below). It has an error metric, a bound, and a failure probability; exact computation introduces no approximation error. The properties are the formulation's, $p_v^{\mathrm{log}}=\boldsymbol\theta_v$ and $p_v^{\mathrm{phys}}=(\tau_v,\ell_v)$; in code, `LogicalProperties { parameters }` and `PhysicalProperties { timing, lifetime }`. -**Tumbling windows are expressions, not properties.** A tumbling-window chain is its own memo group, `SummaryTumblingWindowChain[w](X)`, with the window width $w$ as an operator argument (Section 3, "Chain groups"). Every grid is anchored at phase 0 (the epoch), so the width alone fixes it. A query over a range reads the chain through `TimeRange` and `TimeShift` groups and merges the selected windows with `SummaryMerge`. No property carries the width, and two queries with different ranges or offsets share one chain group. Sliding merge/subtraction layouts remain future work; how many windows a chain retains is $h_v$ (Section 5). +**Tumbling windows are expressions, not properties.** A tumbling-window chain is its own memo group, `SummaryTumblingWindowChain[w](X)`, with the window width $w$ as an operator argument (Section 3, "Chain groups"). Every grid is anchored at phase 0 (the epoch), so the width alone fixes it. A query over a range reads the chain through `TimeRange` and `TimeShift` groups and merges the selected windows with `SummaryMerge`, which carries the chain build's θ. No property carries the width, and two queries with different ranges or offsets share one chain group. Sliding merge/subtraction layouts remain future work; how many windows a chain retains is $h_v$ (Section 5). ### Logical requirements @@ -273,7 +273,7 @@ flowchart TD | 3 | $\tau_v$ | Stage 2 | QueryTime; IngestionTime only for a candidate that is time-independent or an input of one, and for a counter state only when the source's out-of-order bound is known (Stage 2) | the source has no ingestion pipeline; QueryTime over raw rows the deployment does not retain; a Count-Min build whose weights are not proven non-negative | | 4 | $\ell_v$ | Stage 2 | None, OneEvaluation, AcrossEvaluations; AcrossEvaluations for chains and for candidates of groups evaluated on a known grid, and Disk only where the deployment offers it | invalid with $\tau_v$; the deployment retains no query-time results | -The planner does not check whether an executor implements an operator or summary type: every candidate the rules produce is offered, and an executor that lacks one is extended. A value outside a dimension's declared value set (IngestionTime on a time-dependent candidate, AcrossEvaluations on a candidate of a group with no known grid, Disk where the deployment has no disk storage) is not offered, so it is not recorded as pruned. +The planner does not check whether an executor implements an operator or summary type: every candidate the rules produce is offered, and an executor that lacks one is extended. A value outside a dimension's declared value set (IngestionTime on a time-dependent candidate, AcrossEvaluations on a candidate of a group with no known grid, Disk where the deployment has no disk storage) is not offered, so it is not recorded as pruned. Nor is (IngestionTime, None) for a summary build (`SummaryAgg`, `SummaryTumblingWindowChain`, `MaintainPopulation`): a build fed at ingestion time keeps its state between batches, so it is materialized; the plan's validation rejects an unmaterialized one. A candidate one of whose consumers reads it under `@ T` with no known horizon end gets no materialized alternative (its retention need, the horizon end − T + range, is unbounded); its query-time alternatives stay. The tree does not decide which producer each consumer input reads ($y$), in which context a candidate with ℓ = None or OneEvaluation runs (Section 5), or how much an AcrossEvaluations candidate retains ($h_v$). Those are Stage 3 choices. @@ -365,6 +365,10 @@ pub struct GroupProperties { /// Whether two equal expressions return the same rows for the same /// input rows, so they may be one group (not an unordered `LIMIT`). pub shareable_when_equal: bool, + /// Whether every `Scan` below reads a time series (PromQL rows). A + /// PromQL top-k ranks one row per series; a counter state reads in + /// time order. + pub reads_only_time_series: bool, } /// What a summary state or exact aggregate computes, with every lifted @@ -472,6 +476,7 @@ Everything that was computed on operator-node trees is computed on groups. `find - The checks that read an input's operator (a merge over a time-indexed state reads a `TimeRange` over a chain; a selected build) read the input group's `expression`. - `shareable_when_equal`: a `LIMIT` is shareable over a `Sort` with the same partitions (the input group's `expression`); a `Dedup` on every column (the input group's schema). - `column_lineage` and `column_sign` (below), per output column. +- `reads_only_time_series`: a `Scan` of a `Source::TimeSeries` reads one; any other operator does when all its input groups do. A node tree computes the same rule through its children (`OperatorInput::reads_only_time_series`), so the rule is written once. `add_candidate` derives a candidate's properties with the same `derive_group_properties` and requires its schema and result kind to equal the group's. `OperatorNode` recursively contains chosen child nodes and therefore describes a fixed plan fragment; `Operator` contains input group IDs and leaves their implementations open. A candidate needs its own expression because a group may offer several ways to compute the same result. @@ -491,6 +496,7 @@ pub struct GroupProperties { pub column_sign: Vec, pub aggregate_coverage: Option, pub shareable_when_equal: bool, + pub reads_only_time_series: bool, } /// A column of a source (a Scan's table or metric), by name. @@ -770,7 +776,7 @@ Pass 2 (`crates/stage1-pass2-reuse-candidates`) runs these steps in order: | 2. Top-k reuse | Reads a compatible larger top-k result with an exact-input requirement | Same reason | | 3. Reach, caps and readout requirements | Computes which queries reach each candidate along candidate edges, the validity and pruning caps (Section 4), and each readout group's requirement levels: one readout candidate per level, its `RequiredAccuracy` written with `Memo::set_input_requirements` | Needs every consumer added by steps 1 and 2 | | 4. Dead-candidate elimination | Removes candidates no query can use, transitively, and records the root cause of each | Needs the requirements; keeps step 5 from building windows for dead candidates | -| 5. Chain groups | Adds tumbling-window chain groups and the merges that read them, with widths from the queries that reach each state group | Needs the reach of step 3, including rollup consumers, and only the survivors of step 4 | +| 5. Chain groups | Adds tumbling-window chain groups and the merges that read them, with widths from the queries that reach each state group, then repeats step 4 over the memo with the chains | Needs the reach of step 3, including rollup consumers, and only the survivors of step 4; the repeat prunes a merge whose θ serves no consumer | | 6. Logical edges | Records every (producer, consumer, input) and whether it satisfies the input's logical requirement | Runs on the final memo | Rules match group expressions, add groups with `find_or_add_group` and candidates with `add_candidate`, and never copy consumers to make a new producer visible. @@ -867,7 +873,7 @@ A query reaches a candidate when a path of candidate edges leads from the query' #### Dead-candidate elimination -A candidate that is not a query output is **dead** when it never had a consumer, or when every edge to a consumer fails: the producer's guarantee misses the input's required accuracy, the metric rule forbids the edge (Section 4), or the consumer is dead. Elimination repeats until nothing changes. Dead candidates get no logical edges and Stage 2 does not expand them; each is recorded with its root causes. Pass 2 also records each group's evaluation grid, which `delivers_on` reads for a stored result that is not a chain. Example: X = `sum_over_time(m[5m])` is read by a query every 60 s and by another every 120 s, both at phase 0; X's grid is 60 s @ 0. X materialized across evaluations is stored every 60 s, so it delivers on `Evaluation { 60 s @ 0 }` and `Evaluation { 120 s @ 0 }` but not on `Evaluation { 30 s @ 0 }`. X under `@ T` does not depend on the evaluation time and delivers on every context. +A candidate that is not a query output is **dead** when it never had a consumer, or when every edge to a consumer fails: the producer's guarantee misses the input's required accuracy, the metric rule cuts the edge (Section 4), or the consumer is dead. Any candidate, a query output included, is also dead when every edge on one of its inputs is cut or reads a dead producer. The metric rule cuts edges, not candidates: a candidate dies of it only through one of these two conditions. Elimination repeats until nothing changes, and runs once more after chain groups (step 5). Dead candidates get no logical edges and Stage 2 does not expand them; each is recorded with its root causes. Pass 2 also records each group's evaluation grid, which `delivers_on` reads for a stored result that is not a chain. Example: X = `sum_over_time(m[5m])` is read by a query every 60 s and by another every 120 s, both at phase 0; X's grid is 60 s @ 0. X materialized across evaluations is stored every 60 s, so it delivers on `Evaluation { 60 s @ 0 }` and `Evaluation { 120 s @ 0 }` but not on `Evaluation { 30 s @ 0 }`. X under `@ T` does not depend on the evaluation time and delivers on every context. ```rust pub struct LogicalASAPMemoCandidates { @@ -889,11 +895,12 @@ pub struct PrunedLogicalCandidate { pub causes: Vec, } -/// One path from the candidate, through consumers that are themselves +/// One path from the candidate, through candidates that are themselves /// eliminated, to the edge that fails. One shortest path per failing edge. pub struct PruneCause { - /// The eliminated consumers on the path, nearest first. - pub through_pruned_consumers: Vec, + /// The eliminated candidates on the path, nearest first: consumers + /// above, or, for a candidate cut off from an input, producers below. + pub through_pruned_candidates: Vec, pub root: RootPruneReason, } @@ -905,12 +912,28 @@ pub enum RootPruneReason { input: usize, required: RequiredAccuracy, }, - /// The consumer's error contract carries none of the producer's metrics. + /// The metric rule (Section 4): the producer is a readout whose metric + /// the consumer does not accept on that input; or, on a readout's own + /// input, the readout's metric is not one its output accepts. MetricRule { consumer: CandidateForMemoGroupId, input: usize, producer_metric: AccuracyMetric, - consumer_carries: Vec, + /// The metrics accepted on that input (for a readout's own input: + /// at its output). + accepted_metrics: Vec, + }, + /// A top-k readout's `k` exceeds the producer's heap size. + HeapSmallerThanTopK { + consumer: CandidateForMemoGroupId, + input: usize, + k: usize, + }, + /// A readout reached by query `query`, whose whole-result failure + /// probability has no per-value one: n_q is unknown. + ValuesPerEvaluationUnknown { + consumer: CandidateForMemoGroupId, + query: usize, }, /// The candidate never had a consumer and is not a query output. A /// candidate whose consumers were eliminated is traced to their causes. @@ -934,19 +957,19 @@ The Divide carries only relative error. Its inputs' cap is 0.05 / 1.111 = 0.045 ``` G4#1 SummaryEstimate[q=0.99](G3) - G4#1 → G6#0 input 0: MetricRule { producer_metric: Rank, consumer_carries: [Relative] } + G4#1 → G6#0 input 0: MetricRule { producer_metric: Rank, accepted_metrics: [Relative] } G5#1 SummaryEstimate[q=0.5](G3) - G5#1 → G6#0 input 1: MetricRule { producer_metric: Rank, consumer_carries: [Relative] } + G5#1 → G6#0 input 1: MetricRule { producer_metric: Rank, accepted_metrics: [Relative] } G3#0 SummaryAgg[Sketch(Kll)] k=26 (and likewise G3#1 … G3#6) - G3#0 → G4#1 → G6#0 input 0: MetricRule { producer_metric: Rank, consumer_carries: [Relative] } - G3#0 → G5#1 → G6#0 input 1: MetricRule { producer_metric: Rank, consumer_carries: [Relative] } + G3#0 → G4#1 input 0: MetricRule { producer_metric: Rank, accepted_metrics: [Relative] } + G3#0 → G5#1 input 0: MetricRule { producer_metric: Rank, accepted_metrics: [Relative] } G3′#0 SummaryAgg[Sketch(DDSketch)] alpha=0.1 (and likewise G3′#1, alpha=0.05) G3′#0 → G4#2 input 0: BelowRequiredAccuracy { required: relative 0.02 } G3′#0 → G5#2 input 0: BelowRequiredAccuracy { required: relative 0.02 } (and one cause per looser readout, G4#3 … G4#6, G5#3 … G5#6) ``` -The KLL builds' own edges may satisfy their consumers; they are dead because their only consumers are, and the trace leads through those consumers to the metric rule. The exact candidates, every DDSketch readout, and the DDSketch builds with alpha ≤ 0.02 survive; each build connects only to the readouts whose level it meets (alpha 0.01 to the levels 0.02 and 0.01). +The KLL builds' edges into the KLL readouts are cut too: a KLL readout's metric, rank, is not one its output accepts (only relative reaches G4 and G5), so the readout has no requirement and reads no state. The exact candidates, every DDSketch readout, and the DDSketch builds with alpha ≤ 0.02 survive; each build connects only to the readouts whose level it meets (alpha 0.01 to the levels 0.02 and 0.01). #### Chain groups @@ -964,15 +987,15 @@ SummaryTumblingWindowChain { child: X, summary_type, input, reduction, filter, SummaryMerge { children, group_by } ``` -`TimeRange` and `TimeShift` accept a time-indexed state: over a chain, `TimeRange(W)` at evaluation time t selects the windows lying fully inside (t − W, t], and `TimeShift(o)` moves that interval to (t − o − W, t − o]. A `TimeRange` over a chain must be a range (`TimeRangeKind::Range`); an instant one is a validation error. This needs w to divide W and o, and t to lie on the grid; the widths below and `delivers_on` (Section 2) ensure both. The `TimeRange` and `TimeShift` groups over a chain have only their own expression as a candidate and are computed within one evaluation; only the chain is materialized across evaluations. +`TimeRange` and `TimeShift` accept a time-indexed state: over a chain, `TimeRange(W)` at evaluation time t selects the windows lying fully inside (t − W, t], and `TimeShift(o)` moves that interval to (t − o − W, t − o]. A `TimeRange` over a chain must be a range (`TimeRangeKind::Range`); an instant one is a validation error. This needs w to divide W and o, and t to lie on the grid; the widths below and `delivers_on` (Section 2) ensure both. The `TimeRange` and `TimeShift` groups over a chain have their own expression as candidates, one per θ of the chain's builds, and are computed within one evaluation; only the chain is materialized across evaluations. **The rule, for each width w of a state group S:** 1. Walk down from S: `SummaryAgg` → `TimeRange(W)` → zero or more `TimeShift`s → X. -2. `find_or_add_group(SummaryTumblingWindowChain[w](X))` → $G_w$. The shifts are stripped, so queries with different ranges and offsets over the same X get the same $G_w$. -3. Re-wrap the stripped shifts and then the `TimeRange` around $G_w$, in their original order, each with `find_or_add_group`. -4. `add_candidate(S, SummaryMerge(…))` over the outermost re-wrapped group. Its coverage equals S's (Section 3, Stage 0), so the candidate is accepted. +2. `find_or_add_group(SummaryTumblingWindowChain[w](X))` → $G_w$, with one build per θ of S's builds that survived step 4. The shifts are stripped, so queries with different ranges and offsets over the same X get the same $G_w$. +3. Re-wrap the stripped shifts and then the `TimeRange` around $G_w$, in their original order, each with `find_or_add_group` and one candidate per θ of the group it reads. +4. `add_candidate(S, SummaryMerge[θ](…))` over the outermost re-wrapped group, one per θ of that group's candidates. Its coverage equals S's (Section 3, Stage 0), so the candidate is accepted. -There is one chain per width, shared by every state group over the same X and summary type. +There is one chain per width, shared by every state group over the same X and summary type. Steps 3 and 4 run after step 2 has run for every state group, so a chain shared by state groups whose surviving builds differ offers each of them every θ it has. Dead-candidate elimination then runs again over the memo with the chains (below), so a merge whose θ meets no consumer's requirement is pruned. **Widths.** Each query q that reaches S (step 3) and repeats with a known evaluation grid $g_q$ (`evaluation_grid_ms` is `Some`) gives the width $\gcd(g_q, W, o_1, \ldots)$, with the offsets $o_j$ on its path; S's widths are the gcd closure of these. Queries without a known grid give none, so a group read only by such queries gets no chain. `@` modifiers: - `@ end()` and `@ start()`: the anchored subtree X is lowered without the `@` (Section 3), so a state group inside X is an ordinary state group. In a range query it is reached in the evaluation context of the request's end times, whose grid gives its widths like any query's. @@ -987,10 +1010,10 @@ There is one chain per width, shared by every state group over the same X and su | G3b | `SummaryAgg[Sum](TimeRange(5m, TimeShift(1h, G1)))` | build; **`SummaryMerge(Gb)`** | | G3c | `SummaryAgg[Sum](TimeRange(1h, G1))` | build; **`SummaryMerge(Gc)`** | | **Gw** | `SummaryTumblingWindowChain[Sum, w = 60 s](G1)` | one build per θ | -| Ga | `TimeRange(5m, Gw)` | its own expression | -| Gs | `TimeShift(1h, Gw)` | its own expression | -| Gb | `TimeRange(5m, Gs)` | its own expression | -| Gc | `TimeRange(1h, Gw)` | its own expression | +| Ga | `TimeRange(5m, Gw)` | its own expression, per θ | +| Gs | `TimeShift(1h, Gw)` | its own expression, per θ | +| Gb | `TimeRange(5m, Gs)` | its own expression, per θ | +| Gc | `TimeRange(1h, Gw)` | its own expression, per θ | At t = 12:00, Ga selects the 5 windows (11:55, 11:56] … (11:59, 12:00], Gb the 5 windows (10:55, 10:56] … (10:59, 11:00], and Gc 60 windows. Each sample updates one chain instead of three, and the chain retains max(5m, 1h + 5m, 1h) = 65m, 65 windows (Section 5). @@ -1006,10 +1029,12 @@ At t = 12:00, Ga selects the 5 windows (11:55, 11:56] … (11:59, 12:00], Gb the 1. Walk down from G4: `SummaryAgg` reads G3; G3 is `TimeRange(W = 5m)` over G2; G2 is `TimeShift(1h)` over G1; G1 is not a shift. So W = 5m, the shifts are [`TimeShift(1h)`], X = G1. 2. `find_or_add_group(SummaryTumblingWindowChain[Sum, w = 60 s](G1))` → Gw. A and C get the same Gw. 3. `find_or_add_group(TimeShift(1h, Gw))` → Gs; `find_or_add_group(TimeRange(5m, Gs))` → Gb. -4. `add_candidate(G4, SummaryMerge(Gb))`. +4. `add_candidate(G4, SummaryMerge(Gb))`. (`Sum` is an exact accumulator: there is no θ, so one candidate per step.) After the rule, G4 has two candidates, the build and `SummaryMerge(Gb)`; G2 and G3 stay. With widths {30 s, 60 s}, steps 2–4 run once per width, giving two chains and two merge candidates in G4; Stage 3 chooses. +**Example: a merge carries its θ.** P = `quantile_over_time(0.9, m[5m])` with rank error 0.005 and Q = `quantile_over_time(0.9, m[10m])` with rank error 0.01, both every 60 s. Each state group keeps the KLL builds that meet its readout: P's from k = 547, Q's from k = 269. Both read one chain $G_w$ over `m` (w = 60 s), which therefore has the builds k = 269, 547, … . `TimeRange(5m, G_w)` has one candidate per θ, and P's state group gets `SummaryMerge[k=269]` and `SummaryMerge[k=547]` (and so on). The edge from `SummaryMerge[k=547]` to P's readout holds; the edge from `SummaryMerge[k=269]` fails with `BelowRequiredAccuracy { required: rank 0.005 }`, so that merge is pruned, and `TimeRange(5m, G_w)[k=269]` with it (its only consumer is). The chain build k = 269 survives: Q's merge reads it. + #### Logical edges ```rust @@ -1023,7 +1048,7 @@ pub struct LogicalEdge { } ``` -An edge satisfies the logical requirement when α^log = 1 (`producer_satisfies_logical_requirements`): into a readout's input, the producer's guarantee meets the required accuracy (Section 2); into any other input, the consumer's error contract carries the producer's metric (Section 4). +A candidate that carries a θ without building (a merge, or a `Filter`, `TimeRange` or `TimeShift` over states) has edges only from producers with the same θ; any other pair of a consumer and a candidate of its input group is an edge. An edge satisfies the logical requirement when α^log = 1 (`producer_satisfies_logical_requirements`): into a readout's input, the guarantee of the producer's θ meets the required accuracy (Section 2), and the readout's metric is one its own output accepts; into any other input, the metric rule: a readout producer's metric is one the consumer accepts on that input (Section 4). An edge from any other producer satisfies it. ### Stage 2: physical alternatives @@ -1070,7 +1095,7 @@ Stage 2 records every alternative it rejects, with one of these reasons; Stage 3 |---|---| | `NoIngestionPipeline` | IngestionTime when the source has no ingestion pipeline | | `RawDataNotRetained` | a Scan of raw rows the deployment does not retain, at QueryTime or AcrossEvaluations | -| `QueryTimeResultsNotRetained` | QueryTime with ℓ = OneEvaluation or AcrossEvaluations when the deployment retains no query-time results | +| `QueryTimeResultsNotRetained` | (QueryTime, AcrossEvaluations) when the deployment retains no query-time results; (QueryTime, OneEvaluation) is kept, since it lives within one evaluation | | `InputHasNoBuildableProducer` | a candidate for which some input has no surviving producer that satisfies α^phys | | `ReachesNoQueryOutput` | a candidate from which no admissible path leads to a query root | | `CountMinWeightsNotProvenNonNegative` | a Count-Min build (`Cms`, `CmsWithHeap`, `HydraCms`) whose weight column's `column_sign` is not `NonNegative`: its estimates are bounds only for non-negative weights | @@ -1336,6 +1361,10 @@ A constant contract is unconditional: it needs no row count or sign. A filter do **Metric rule.** Every local guarantee states its metric: absolute (in value units, or a multiple of a stream norm), relative, or rank. A factor $\beta_{c,i}$ is defined for one input metric and one output metric. An input whose metric the consumer's error contract cannot convert gives no edge ($\alpha^{\log} = 0$); the conversion is never assumed. For example, a relative-error ratio cannot read a KLL quantile (rank error) unless a declared distribution assumption supplies the conversion. +Pass 2 applies it per edge, from the metrics each group's output may be off in for the queries with a target that reach it (`asap_accuracy::accepted_metrics`): a query's root accepts its target's metric (every metric when the target names none); a candidate's input $i$ accepts the metrics $m$ its output accepts for which `error_propagation(c, m)` is `Some`; a group accepts the union over its consumers. An edge is cut when its producer is a readout whose metric its consumer's input does not accept, and a readout's own input edges are cut when its metric is not one its output accepts (so no surviving readout a targeted query reaches lacks a requirement). Only readouts introduce error, so only their edges are cut; a candidate dies only when dead-candidate elimination finds no edge left on one of its inputs. A group no query with a target reaches has no metric rule (accuracy not enforced). + +Example: `avg by (job)(q0.5 / q0.5 + q0.9)` with absolute error. The `+` carries absolute error, so the division's output accepts only absolute; the division carries only relative, so its inputs accept nothing. The edges from the DDSketch readouts (relative) into the division are cut and the exact division stays, read from its exact producers; before, the division itself was pruned because a DDSketch path existed below it. + For a ratio, let the operand relative-error bounds be $\varepsilon_X$ and $\varepsilon_Y$. If the numerator's relative error is defined and the denominator is nonzero, then: $$ @@ -1385,7 +1414,7 @@ Errors enter a plan only at readouts, and a readout's error is its requirement ( - **A readout group that feeds a composed query** (several readouts combined, such as a ratio) gets one readout candidate per requirement level of the grid ε ∈ {0.1, 0.05, 0.02, 0.01, 0.005, 0.002, 0.001} × δ ∈ {1e-2, 1e-3, 1e-4, 1e-5}, minus the levels looser than the input's cap (below) or than the query's per-value δ. A target without a failure probability gives one δ level, δ = 1. Constraint (K) then chooses how the query's error is split among its readouts, together with the producers. - A readout group reached by several queries gets the union of their levels. -**Failure probability per value.** A target's δ covers the whole result unless its scope is per value. A query enters the memo with $n_q$, the number of values one evaluation returns, and Pass 2 turns a whole-result δ_q into the per-value δ_q / n_q for the readout requirements; Stage 3 needs no $n_q$. +**Failure probability per value.** A target's δ covers the whole result unless its scope is per value. A query enters the memo with $n_q$, the number of values one evaluation returns, and Pass 2 turns a whole-result δ_q into the per-value δ_q / n_q for the readout requirements; Stage 3 needs no $n_q$. When $n_q$ is unknown no per-value δ follows: the readouts the query reaches are pruned with `RootPruneReason::ValuesPerEvaluationUnknown { consumer, query }` (no state is required to fail with probability 0). Example: a target δ = 0.01 over the whole result of a query whose number of series is not declared. **Example: allocation in a ratio.** Query B = p99 / p50 with relative error 0.03, no failure probability, over `latency` declared non-negative. Each operand's cap is 0.03 / 1.111 = 0.027, so G4 (p99) and G5 (p50) each get readouts at the levels 0.02, 0.01, 0.005, 0.002 and 0.001. A DDSketch build with alpha ≤ ε meets the level ε (Section 2). (K) requires 1.111 (ε_p99 + ε_p50) ≤ 0.03: @@ -1552,7 +1581,7 @@ The MILP compares the two plans' cost per second. With much data, plan 2's singl | $W_v^{\mathrm{mat}}$ | cost per second of v's own computation in `Materialized(v)` (e.g. one build per window), excluding retention | | $c_{\kappa,v}$ | cost of one run of v in context κ, from v's own operator and the logical size of its inputs; it does not depend on which producer v reads | | $c_{\kappa,u,c,i}$ | cost, per run of κ, of consumer c reading input i from producer u: transferring or reading u's output, and the part of c's work that changes with u's physical form (u's θ, its number of windows). Everything that does not change when another producer of the same group is chosen is in $c_{\kappa,c}$ | -| $\rho_\kappa$ | executions of κ per second; for an evaluation grid, times the number of requests covering each grid time | +| $\rho_\kappa$ | executions of κ per second; for an evaluation grid, times the number of requests covering each grid time. A candidate's run rate in κ counts only the queries of κ that reach it | | $h_v$, $\pi_v$ | retained amount and its cost per unit per second | | $\operatorname{need}_{\kappa,v,c,i}$ | how much of v's rows this edge reads back: for a time-indexed producer (raw data, a chain, results stored per evaluation time) the reading path's offset plus range | @@ -1567,7 +1596,11 @@ $$ h_v\ge\operatorname{need}_{\kappa,v,c,i}\,y_{\kappa,v,c,i}\ \ (\mathrm{R}). $$ -Every term is exact: no producer is chosen to stand for others. In the example of Section 3, the edges to A's, B's and C's paths need 5m, 1h + 5m and 1h, so the shared 60 s chain retains 65 windows. A Scan's raw data is a time-indexed producer like any other: its retention is its $h_v$ under the same (R). `@ T` needs the end of the horizon − T + range; with an unbounded horizon that candidate is not offered. Query latency is ignored: a result is taken to be read when its evaluation starts. $h_v$ has no upper bound here: it is minimized, and per-candidate bounds belong to resource budgets, outside this design. +Every term is exact: no producer is chosen to stand for others. In the example of Section 3, the edges to A's, B's and C's paths need 5m, 1h + 5m and 1h, so the shared 60 s chain retains 65 windows. A Scan's raw data is a time-indexed producer like any other: its retention is its $h_v$ under the same (R). `@ T` needs the end of the horizon − T + range; with an unbounded horizon that candidate is not offered. Query latency is left for later: a result is taken to be read when its evaluation starts. $h_v$ has no upper bound here: it is minimized, and per-candidate bounds belong to resource budgets, outside this design; a candidate with an unbounded need (a consumer whose reach has no bound) gets no $h_v$ and is not offered materialized. + +**Memory held while computing.** A query-time execution holds its output in memory only while it computes: its cost $c_{\kappa,v}$ includes bytes$_v$ × work$_v$ (s) × the memory cost per byte-second, per run. Example: a KLL state of 2 MB whose own work takes 3 ms, run 0.1 times per second, holds 2 MB × 0.003 s × 0.1/s = 600 byte·s per second. Query latency does not enter it. `plan_cost` charges the same term, so the MILP objective equals `plan_cost` of the selected plan. + +**Scan reads.** A query-time Scan keeps its retention (offset + range + range-evaluation span) but reads, per execution, the largest offset + range among its selected consumers (no span): a 1 h range query with step 1 s over an instant selector keeps 3,601,000 ms of samples and reads 1 s per step. Stage 3 and `plan_cost` share one function for it. A range query's `@ end()` / `@ start()` subtree runs once per request, in Stage 3 and in `plan_cost` alike. ### No consumer-count policies From 5d5d458c5d4756e8b957e2cf532d7551b7a0f6c5 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 9 Nov 2026 18:14:38 +0000 Subject: [PATCH 59/67] docs(concepts): scan read r_e and constraint (S) in the objective; HLL distinct count from the memo's source facts (max_distinct_per_evaluation) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01WviZ6FUmrMfBTvVbBauNM6 --- docs/design_docs/concepts/accuracy-models.md | 21 +++++++----- .../memo-groups-properties-requirements.md | 34 +++++++++++++------ 2 files changed, 37 insertions(+), 18 deletions(-) diff --git a/docs/design_docs/concepts/accuracy-models.md b/docs/design_docs/concepts/accuracy-models.md index 12004cb8b..b5630a50d 100644 --- a/docs/design_docs/concepts/accuracy-models.md +++ b/docs/design_docs/concepts/accuracy-models.md @@ -290,14 +290,19 @@ A deployment declares `DataWorkload.sources[source].max_distinct_per_evaluation` for the source the distinct count reads. It asserts an enforced maximum distinct population per evaluation, including all merged tumbling windows; the classic estimator and independent uniform bucket hashing are the model's -assumptions. Stage 1 sizes the precision from the query or allocated local -target and leaves the failure probability symbolic, as -`classic_hll_failure_probability[p=P,eps=ε](max_distinct[source])`. Pass 1 -then raises the precision when the source's declared bound needs it (for its -share of the failure probability), and Stage 3 binds the failure probability -from that bound in the normal selection checks. A bound outside the -contract's range gives a failure probability of 1; a missing bound leaves it -unknown, so the target is not met. +assumptions. The memo holds the scanned sources' facts, and +`Memo::max_distinct_per_evaluation(group)` sums the bounds of the sources +below the HLL state. The theoretical guarantee of precision p for the distinct +count at the readout's required failure probability δ is the least relative +error ε whose classic failure probability over that bound is at most δ; the +readout's edge (α^log, Pass 2) holds when that ε meets its requirement. A +bound outside the contract's range certifies no error, and a missing bound +none either: the edge fails with `BelowRequiredAccuracy`. The readout's +metric (relative) does not depend on the bound. + +Example: `COUNT(DISTINCT user_id)` within relative 0.05 at δ = 0.01. With +`max_distinct_per_evaluation` 9,636, precision 18 meets it; with 1,000,000 +(outside the contract) or no bound, no HLL precision does. The current model supports maxima from 1 to 131,072 (2^17) and precisions from 4 to 18, and certifies only configurations that remain in the linear-counting branch. diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index b2537e3d2..c7d61b7b5 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -150,16 +150,19 @@ and, for a top-k readout, `heap_size` ≥ k. An exact producer has error 0 and s | width 272, depth 5 | 0.0100 | 6.7e-3 | yes | no | | width 272, depth 10 | 0.0100 | 4.5e-5 | yes | yes | -**Where the guarantee comes from.** `guarantee` is `AccuracyEvidence::measured_guarantee(summary type, θ, statistic, δ_r)` when the deployment's evidence has a measurement for that configuration, and `theoretical_guarantee(summary type, θ, statistic, δ_r)` (the algorithm's error formula) otherwise. Each returns the error bound that holds at failure probability δ_r, or `None` when θ cannot fail that rarely (no edge). The bound is compared with ε_r through `target_limit(ε_r)`, which admits only floating-point noise above ε_r. The source is decided before Pass 2 builds the logical edges, so (K) always has every readout's bound, and nothing is checked again after Stage 3. +**Where the guarantee comes from.** `guarantee` is `AccuracyEvidence::measured_guarantee(summary type, θ, statistic, δ_r)` when the deployment's evidence has a measurement for that configuration, and `theoretical_guarantee(summary type, θ, statistic, δ_r, distinct bound)` (the algorithm's error formula) otherwise. Each returns the error bound that holds at failure probability δ_r, or `None` when θ cannot fail that rarely (no edge). The bound is compared with ε_r through `target_limit(ε_r)`, which admits only floating-point noise above ε_r. The source is decided before Pass 2 builds the logical edges, so (K) always has every readout's bound, and nothing is checked again after Stage 3. ```rust /// The theoretical error bound of reading `statistic` from a state of -/// `summary_type` with parameters θ at `failure_probability`. +/// `summary_type` with parameters θ at `failure_probability`; a distinct +/// count from classic HLL only over at most `max_distinct_per_evaluation` +/// items (`Memo::max_distinct_per_evaluation` of the state's group). pub fn theoretical_guarantee( summary_type: &FieldDataType, parameters: &SketchParams, statistic: &SketchStatistic, failure_probability: f64, + max_distinct_per_evaluation: Option, ) -> Option; /// The deployment's measured accuracy, read before the theoretical bound. @@ -412,9 +415,11 @@ pub struct Memo { pub queries: Vec, // Private: the groups `find_or_add_group` may return, by expression hash. groups_by_expression_hash: HashMap>, - // Private: the sign the data workload declares for each source column it - // declares one for; a `Scan`'s `column_sign` reads it ("Value sign"). - declared_column_signs: BTreeMap, + // Private: the data workload's facts of each source the queries scan, by + // the name the Scan reads (a counter's `value` non-negative): a `Scan`'s + // `column_sign` ("Value sign") and the distinct bound of + // `max_distinct_per_evaluation`. + source_facts: BTreeMap, } pub struct MemoQuery { @@ -426,8 +431,12 @@ pub struct MemoQuery { } impl Memo { - /// A memo without groups whose Scans read `declared_column_signs`. - pub fn new(declared_column_signs: BTreeMap) -> Self; + /// A memo without groups over the sources `source_facts` describes. + pub fn new(source_facts: BTreeMap) -> Self; + /// The most distinct items one distinct-count summary over `group`'s rows + /// counts in one evaluation: the sum of the scanned sources' declared + /// `max_distinct_per_evaluation`; `None` when one has none. + pub fn max_distinct_per_evaluation(&self, group: MemoGroupId) -> Option; pub fn groups(&self) -> &[MemoGroup]; /// Adds a query; its root's groups and those below list it. pub fn add_query(&mut self, query: MemoQuery) -> usize; @@ -619,7 +628,7 @@ Some error contracts hold only for values of a known sign: a relative error of a pub enum ValueSign { NonNegative, NonPositive, Unknown } ``` -A source's signs are declared per column in the data workload's source facts (for example, `latency`'s value is non-negative). A PromQL counter's value is non-negative without a declaration. A column without a declaration is `Unknown`. `memo_of_queries` collects the declared signs of the sources the queries scan into the memo (`Memo::new`), and `find_or_add_group` reads them for a `Scan`. +A source's signs are declared per column in the data workload's source facts (for example, `latency`'s value is non-negative). A PromQL counter's value is non-negative without a declaration. A column without a declaration is `Unknown`. `memo_of_queries` collects the facts of the sources the queries scan into the memo (`Memo::new`), and `find_or_add_group` reads their signs for a `Scan`. | Operator | Output column sign | |---|---| @@ -1584,6 +1593,8 @@ The MILP compares the two plans' cost per second. With much data, plan 2's singl | $\rho_\kappa$ | executions of κ per second; for an evaluation grid, times the number of requests covering each grid time. A candidate's run rate in κ counts only the queries of κ that reach it | | $h_v$, $\pi_v$ | retained amount and its cost per unit per second | | $\operatorname{need}_{\kappa,v,c,i}$ | how much of v's rows this edge reads back: for a time-indexed producer (raw data, a chain, results stored per evaluation time) the reading path's offset plus range | +| $r_e$, $\sigma_e$ | for an execution e of a query-time Scan: what one run reads back, and its cost per unit per second | +| $\operatorname{read}_{\kappa,u,c,i}$ | what one run of the edge's consumer reads of u: $\operatorname{need}$ without the range-evaluation span | $$ \min\left( @@ -1591,16 +1602,19 @@ $$ +\sum_{\kappa,v}\rho_\kappa c_{\kappa,v}x_{\kappa,v} +\sum_{\kappa,u,c,i}\rho_\kappa c_{\kappa,u,c,i}\,y_{\kappa,u,c,i} +\sum_{v}\pi_v h_v ++\sum_{e}\sigma_e r_e \right), \qquad -h_v\ge\operatorname{need}_{\kappa,v,c,i}\,y_{\kappa,v,c,i}\ \ (\mathrm{R}). +h_v\ge\operatorname{need}_{\kappa,v,c,i}\,y_{\kappa,v,c,i}\ \ (\mathrm{R}), +\qquad +r_e\ge\operatorname{read}_{\kappa,u,c,i}\,y_{\kappa,u,c,i}\ \ (\mathrm{S}). $$ Every term is exact: no producer is chosen to stand for others. In the example of Section 3, the edges to A's, B's and C's paths need 5m, 1h + 5m and 1h, so the shared 60 s chain retains 65 windows. A Scan's raw data is a time-indexed producer like any other: its retention is its $h_v$ under the same (R). `@ T` needs the end of the horizon − T + range; with an unbounded horizon that candidate is not offered. Query latency is left for later: a result is taken to be read when its evaluation starts. $h_v$ has no upper bound here: it is minimized, and per-candidate bounds belong to resource budgets, outside this design; a candidate with an unbounded need (a consumer whose reach has no bound) gets no $h_v$ and is not offered materialized. **Memory held while computing.** A query-time execution holds its output in memory only while it computes: its cost $c_{\kappa,v}$ includes bytes$_v$ × work$_v$ (s) × the memory cost per byte-second, per run. Example: a KLL state of 2 MB whose own work takes 3 ms, run 0.1 times per second, holds 2 MB × 0.003 s × 0.1/s = 600 byte·s per second. Query latency does not enter it. `plan_cost` charges the same term, so the MILP objective equals `plan_cost` of the selected plan. -**Scan reads.** A query-time Scan keeps its retention (offset + range + range-evaluation span) but reads, per execution, the largest offset + range among its selected consumers (no span): a 1 h range query with step 1 s over an instant selector keeps 3,601,000 ms of samples and reads 1 s per step. Stage 3 and `plan_cost` share one function for it. A range query's `@ end()` / `@ start()` subtree runs once per request, in Stage 3 and in `plan_cost` alike. +**Scan reads.** A query-time Scan keeps its retention (offset + range + range-evaluation span) but reads, per execution, the largest offset + range among its selected consumers (no span; $r_e$ under (S), its cost linear in the amount read): a 1 h range query with step 1 s over an instant selector keeps 3,601,000 ms of samples and reads 1 s per step. Stage 3 and `plan_cost` share one function for it. A range query's `@ end()` / `@ start()` subtree runs once per request, in Stage 3 and in `plan_cost` alike. ### No consumer-count policies From ae231ff95875ed612be78e0c4f67a0826606361b Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 9 Nov 2026 19:09:49 +0000 Subject: [PATCH 60/67] docs(concepts): a query-time Scan execution reads the most its context's consumers read (no read variable) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01WviZ6FUmrMfBTvVbBauNM6 --- .../concepts/memo-groups-properties-requirements.md | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index c7d61b7b5..b4e842da5 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -1593,8 +1593,6 @@ The MILP compares the two plans' cost per second. With much data, plan 2's singl | $\rho_\kappa$ | executions of κ per second; for an evaluation grid, times the number of requests covering each grid time. A candidate's run rate in κ counts only the queries of κ that reach it | | $h_v$, $\pi_v$ | retained amount and its cost per unit per second | | $\operatorname{need}_{\kappa,v,c,i}$ | how much of v's rows this edge reads back: for a time-indexed producer (raw data, a chain, results stored per evaluation time) the reading path's offset plus range | -| $r_e$, $\sigma_e$ | for an execution e of a query-time Scan: what one run reads back, and its cost per unit per second | -| $\operatorname{read}_{\kappa,u,c,i}$ | what one run of the edge's consumer reads of u: $\operatorname{need}$ without the range-evaluation span | $$ \min\left( @@ -1602,19 +1600,16 @@ $$ +\sum_{\kappa,v}\rho_\kappa c_{\kappa,v}x_{\kappa,v} +\sum_{\kappa,u,c,i}\rho_\kappa c_{\kappa,u,c,i}\,y_{\kappa,u,c,i} +\sum_{v}\pi_v h_v -+\sum_{e}\sigma_e r_e \right), \qquad -h_v\ge\operatorname{need}_{\kappa,v,c,i}\,y_{\kappa,v,c,i}\ \ (\mathrm{R}), -\qquad -r_e\ge\operatorname{read}_{\kappa,u,c,i}\,y_{\kappa,u,c,i}\ \ (\mathrm{S}). +h_v\ge\operatorname{need}_{\kappa,v,c,i}\,y_{\kappa,v,c,i}\ \ (\mathrm{R}). $$ Every term is exact: no producer is chosen to stand for others. In the example of Section 3, the edges to A's, B's and C's paths need 5m, 1h + 5m and 1h, so the shared 60 s chain retains 65 windows. A Scan's raw data is a time-indexed producer like any other: its retention is its $h_v$ under the same (R). `@ T` needs the end of the horizon − T + range; with an unbounded horizon that candidate is not offered. Query latency is left for later: a result is taken to be read when its evaluation starts. $h_v$ has no upper bound here: it is minimized, and per-candidate bounds belong to resource budgets, outside this design; a candidate with an unbounded need (a consumer whose reach has no bound) gets no $h_v$ and is not offered materialized. **Memory held while computing.** A query-time execution holds its output in memory only while it computes: its cost $c_{\kappa,v}$ includes bytes$_v$ × work$_v$ (s) × the memory cost per byte-second, per run. Example: a KLL state of 2 MB whose own work takes 3 ms, run 0.1 times per second, holds 2 MB × 0.003 s × 0.1/s = 600 byte·s per second. Query latency does not enter it. `plan_cost` charges the same term, so the MILP objective equals `plan_cost` of the selected plan. -**Scan reads.** A query-time Scan keeps its retention (offset + range + range-evaluation span) but reads, per execution, the largest offset + range among its selected consumers (no span; $r_e$ under (S), its cost linear in the amount read): a 1 h range query with step 1 s over an instant selector keeps 3,601,000 ms of samples and reads 1 s per step. Stage 3 and `plan_cost` share one function for it. A range query's `@ end()` / `@ start()` subtree runs once per request, in Stage 3 and in `plan_cost` alike. +**Scan reads.** A query-time Scan keeps its retention (offset + range + range-evaluation span) but reads, per execution, the largest offset + range among its selected consumers (no span). In the MILP an execution of the Scan in context κ is costed reading the most any consumer it may have in κ reads, a constant of the execution, so the objective equals `plan_cost` when that consumer is selected (a variable read amount, linear in the selected consumers, made the MILP of example 3a intractable): a 1 h range query with step 1 s over an instant selector keeps 3,601,000 ms of samples and reads 1 s per step. Stage 3 and `plan_cost` share one function for it. A range query's `@ end()` / `@ start()` subtree runs once per request, in Stage 3 and in `plan_cost` alike. ### No consumer-count policies From bae8a8d5ad28dcdfdad66930a08b136c7f291689 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 9 Nov 2026 20:53:17 +0000 Subject: [PATCH 61/67] =?UTF-8?q?docs(concepts):=20Pass=202=20=CE=B4=20lev?= =?UTF-8?q?els=20continue=20in=20powers=20of=20ten=20below=20the=20per-val?= =?UTF-8?q?ue=20=CE=B4;=20raw=20rows=20a=20TimeShift=20materializes=20at?= =?UTF-8?q?=20ingestion=20time=20are=20retained=20like=20a=20Scan's?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../concepts/memo-groups-properties-requirements.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index b4e842da5..8c9e5e81c 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -1332,7 +1332,7 @@ The plan's cost is `plan_cost(&plan)`; `Selection` does not repeat it. Validatio #### Executing the plan -The executor keeps one time-indexed, append-only store with retention for every materialized node with a time index: raw data (a Scan), chain states, and results stored per evaluation time. Each is read only through `TimeRange` and `TimeShift` nodes, which select its rows by time, and rows older than the node's `retention_ms` are dropped. A `TimeShift { at: End | Start }` node evaluates its child once per request. Summary operators are constructed with their θ; a summary node without θ is an error. +The executor keeps one time-indexed, append-only store with retention for every materialized node with a time index: raw data (a Scan, or a `TimeShift` by an offset over one, materialized at ingestion time), chain states, and results stored per evaluation time. Each is read only through `TimeRange` and `TimeShift` nodes, which select its rows by time, and rows older than the node's `retention_ms` are dropped. A `TimeShift { at: End | Start }` node evaluates its child once per request. Summary operators are constructed with their θ; a summary node without θ is an error. ## 4. Accuracy and safe pruning — Extension (#1207) @@ -1420,7 +1420,7 @@ impl LogicalRequirements { Errors enter a plan only at readouts, and a readout's error is its requirement (Section 2). Pass 2 decides which requirements each readout group gets: - **A query with one readout** on its path gets one readout candidate, whose requirement is the query target divided by the path's factor λ (1 through pass-through operators). -- **A readout group that feeds a composed query** (several readouts combined, such as a ratio) gets one readout candidate per requirement level of the grid ε ∈ {0.1, 0.05, 0.02, 0.01, 0.005, 0.002, 0.001} × δ ∈ {1e-2, 1e-3, 1e-4, 1e-5}, minus the levels looser than the input's cap (below) or than the query's per-value δ. A target without a failure probability gives one δ level, δ = 1. Constraint (K) then chooses how the query's error is split among its readouts, together with the producers. +- **A readout group that feeds a composed query** (several readouts combined, such as a ratio) gets one readout candidate per requirement level of the grid ε ∈ {0.1, 0.05, 0.02, 0.01, 0.005, 0.002, 0.001} × δ ∈ {1e-2, 1e-3, 1e-4, 1e-5, …}, minus the levels looser than the input's cap (below) or than the query's per-value δ. The δ levels continue in powers of ten below 1e-5 down to the first one at or below a tenth of the query's per-value δ, so up to ten readouts each find a level and their sum stays within it: `quantile_over_time(0.9, data[1h]) / quantile_over_time(0.5, data[1h])` with δ = 0.01 over 10 000 series has a per-value δ of 1e-6, and its two readouts get the levels 1e-6 and 1e-7. A target without a failure probability gives one δ level, δ = 1. Constraint (K) then chooses how the query's error is split among its readouts, together with the producers. - A readout group reached by several queries gets the union of their levels. **Failure probability per value.** A target's δ covers the whole result unless its scope is per value. A query enters the memo with $n_q$, the number of values one evaluation returns, and Pass 2 turns a whole-result δ_q into the per-value δ_q / n_q for the readout requirements; Stage 3 needs no $n_q$. When $n_q$ is unknown no per-value δ follows: the readouts the query reaches are pruned with `RootPruneReason::ValuesPerEvaluationUnknown { consumer, query }` (no state is required to fail with probability 0). Example: a target δ = 0.01 over the whole result of a query whose number of series is not declared. @@ -1605,7 +1605,7 @@ $$ h_v\ge\operatorname{need}_{\kappa,v,c,i}\,y_{\kappa,v,c,i}\ \ (\mathrm{R}). $$ -Every term is exact: no producer is chosen to stand for others. In the example of Section 3, the edges to A's, B's and C's paths need 5m, 1h + 5m and 1h, so the shared 60 s chain retains 65 windows. A Scan's raw data is a time-indexed producer like any other: its retention is its $h_v$ under the same (R). `@ T` needs the end of the horizon − T + range; with an unbounded horizon that candidate is not offered. Query latency is left for later: a result is taken to be read when its evaluation starts. $h_v$ has no upper bound here: it is minimized, and per-candidate bounds belong to resource budgets, outside this design; a candidate with an unbounded need (a consumer whose reach has no bound) gets no $h_v$ and is not offered materialized. +Every term is exact: no producer is chosen to stand for others. In the example of Section 3, the edges to A's, B's and C's paths need 5m, 1h + 5m and 1h, so the shared 60 s chain retains 65 windows. A Scan's raw data is a time-indexed producer like any other: its retention is its $h_v$ under the same (R). So are raw rows a `TimeShift` by an offset materializes at ingestion time: in o11y q21, `sum(rate(m[5m] offset 1h))` as a 1 h range query, that store retains 1 h 5 m of rows and its `TimeRange` reads 5 m of them, in the MILP and in `plan_cost` alike. `@ T` needs the end of the horizon − T + range; with an unbounded horizon that candidate is not offered. Query latency is left for later: a result is taken to be read when its evaluation starts. $h_v$ has no upper bound here: it is minimized, and per-candidate bounds belong to resource budgets, outside this design; a candidate with an unbounded need (a consumer whose reach has no bound) gets no $h_v$ and is not offered materialized. **Memory held while computing.** A query-time execution holds its output in memory only while it computes: its cost $c_{\kappa,v}$ includes bytes$_v$ × work$_v$ (s) × the memory cost per byte-second, per run. Example: a KLL state of 2 MB whose own work takes 3 ms, run 0.1 times per second, holds 2 MB × 0.003 s × 0.1/s = 600 byte·s per second. Query latency does not enter it. `plan_cost` charges the same term, so the MILP objective equals `plan_cost` of the selected plan. From 3d6ea1380a9c6dadc3f7c4d34a980238c8f76de5 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Tue, 10 Nov 2026 19:50:13 +0000 Subject: [PATCH 62/67] docs(concepts): match the memo interfaces as implemented Names and interfaces as the memo code defines them: column_signs, AggregateCoverage / AggregateDefinition / CoverageError::NoCoverage, scanned_sources with Source::is_time_series, validate_expression_over_input_groups, number_of_values_per_query_evaluation, set_candidate_input_requirements; Memo::new(data) stores the data workload's source facts. Two-scope identity: the memo compares definitions with ==, the plan's ViewCoverage::definition_id is the content id of the expanded definition. Value sign: the NonNegative definition and the arithmetic, vector(s), quantile-range and Count Sketch / UnivMon readout rows. Code blocks copy the current definitions; file paths follow the scope layout and stage crates. --- .../memo-groups-properties-requirements.md | 294 +++++++++++------- 1 file changed, 180 insertions(+), 114 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 8c9e5e81c..eb2e42270 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -183,7 +183,7 @@ impl AccuracyEvidence for NoMeasuredGuarantees { … } // always None pub fn target_limit(limit: f64) -> f64; ``` -**Sign conditions.** Some guarantees hold only for values of a known sign, read from the input group's `column_sign` (Section 3, "Value sign"): DDSketch's relative error on a quantile needs an input column of one sign, `NonNegative` or `NonPositive` (negated, all-negative values are all-positive). +**Sign conditions.** Some guarantees hold only for values of a known sign, read from the input group's `column_signs` (Section 3, "Value sign"): DDSketch's relative error on a quantile needs an input column of one sign, `NonNegative` or `NonPositive` (negated, all-negative values are all-positive). **A readout's error is its requirement.** A readout candidate's output error is (ε_r, δ_r), a node property: every producer it may read meets it. Constraint (K) composes these errors to the query root (Section 4). Which requirements a readout group gets is Pass 2's choice (Section 4, "Readout requirements"): one per requirement level, each its own readout candidate. @@ -230,7 +230,7 @@ An evaluation context κ is a grid of evaluation times, derived from the workloa | A time-dependent result series with ℓ = AcrossEvaluations, computed on grid $g$ | every evaluation time of κ (shifted by the reading path's `TimeShift`s) lies on $g$ | | A time-independent result with ℓ = AcrossEvaluations | Always | -The evaluation grid is `QueryRecurrence::evaluation_grid_ms(range_evaluation)` in `crates/types/src/workload/mod.rs`, which Pass 2's chain widths and `delivers_on` both use. One-off queries are supported: each is a context of its own recurrence, run `invocations` times. +The evaluation grid is `QueryRecurrence::evaluation_grid_ms(range_evaluation)` in `crates/types/src/workload/query.rs`, which Pass 2's chain widths and `delivers_on` both use. One-off queries are supported: each is a context of its own recurrence, run `invocations` times. ### Invalid combinations @@ -295,7 +295,7 @@ Alternative orders worth measuring: $\tau$ first, when most sources lack an inge Every quantity the optimizer uses is one of three kinds, and is stored where its dependencies are: -- a **node property** depends only on one candidate (θ, τ, ℓ, a readout's error) or one group (schema, lineage, `column_sign`); +- a **node property** depends only on one candidate (θ, τ, ℓ, a readout's error) or one group (schema, lineage, `column_signs`); - an **input requirement** depends only on the consumer's input (`RequiredAccuracy`; the physical requirement, derived from the consumer's timing); - a **pair quantity** depends on both producer and consumer, so it lives on the edge (u, c, i); when it also depends on the execution context, on (edge, κ). @@ -306,7 +306,7 @@ Each quantity is computed once, at the earliest stage that knows everything it d | Quantity | Kind | Depends on | Computed at | Value | |---|---|---|---|---| | θ_u | node property | u | Pass 1 | alpha = 0.01 | -| `column_sign` of the build's input | group property | the input group | Stage 0 | `NonNegative` | +| `column_signs` of the build's input | group property | the input group | Stage 0 | `NonNegative` | | `RequiredAccuracy` of r's input; r's output error | input requirement; node property | r | Pass 2 | relative, 0.01, 1e-2 | | α^log_{u,r,0} | pair | u, r | Pass 2 | 1: DDSketch's guarantee for p50 at alpha 0.01 is relative 0.01 at δ = 0, and the input is non-negative | | α^phys_{u,r,0} | pair | τ_u, ℓ_u, τ_r | Stage 2 | 1 for u at query time with ℓ = None and r at query time | @@ -330,135 +330,173 @@ Rules add groups through `find_or_add_group` and candidates through `add_candida ### Stage 0: groups -`memo_of_queries` in `crates/stage0-memo` builds the groups bottom-up from the queries' nodes: each node's operator, over its children's groups, goes through `Memo::find_or_add_group`, so identical sub-expressions are one group; each query is then added with `Memo::add_query` as a `MemoQuery`: its root, its accuracy target and $n_q$, the number of values one evaluation returns (Pass 2 turns a whole-result failure probability into a per-value one with it). Stage 0 and Stage 1 share one type, `Memo`: Stage 0 returns it with every group's candidate list empty, and Stage 1 fills the lists. +`memo_of_queries` in `crates/stage0-memo` starts from `Memo::new(data)`, which stores the data workload's source facts (`DataWorkload::sources`) at construction, and builds the groups bottom-up from the queries' nodes: each node's operator, over its children's groups, goes through `Memo::find_or_add_group`, so identical sub-expressions are one group; each query is then added with `Memo::add_query` as a `MemoQuery`: its root, its accuracy target and $n_q$, the number of values one evaluation returns (Pass 2 turns a whole-result failure probability into a per-value one with it). Stage 0 and Stage 1 share one type, `Memo`: Stage 0 returns it with every group's candidate list empty, and Stage 1 fills the lists. ```rust +// crates/types/src/stages/logical/group.rs pub struct MemoGroupId(pub u32); pub struct MemoGroup { - /// What it computes: one operator with its arguments. Each `MemoGroupId` - /// in it is an input group, never this group's own id. + /// What it computes: one operator, with its arguments. Each + /// `MemoGroupId` in it is an input group, never this group's own id. θ + /// is no argument of an operator, so it is not part of a group. pub expression: Operator, - /// Derived once when the group is created, from its input groups' - /// `expression` and `group_properties` (optd/TiDB-style); never changes. + /// Derived once when the group is created, from its input groups; never + /// changes. pub group_properties: GroupProperties, - pub parents: Vec, - pub queries: Vec, - // Empty after Stage 0; populated by Stage 1. + /// The groups that read it (`parent_groups()`) through their expression + /// or a candidate's: kept up to date by `find_or_add_group` and + /// `add_candidate`. + pub(super) parents: Vec, + /// The queries whose roots reach it (`reached_by_queries()`), ascending: + /// kept up to date by `add_query` and `add_candidate`. + pub(super) queries: Vec, + /// Its candidates: empty after Stage 0, filled by Stage 1. pub candidates: Vec, } +impl MemoGroup { + pub fn parent_groups(&self) -> &[MemoGroupId]; + pub fn reached_by_queries(&self) -> &[usize]; +} + pub struct GroupProperties { pub schema: Schema, pub result_kind: OperatorResultKind, - /// Per output column: its source column or Computed ("Column lineage"). - pub column_lineage: Vec, - /// Per output column: its value sign ("Value sign"). - pub column_sign: Vec, - /// Whether the result changes with the evaluation time: true under a - /// `TimeRange` relative to it (`sum_over_time(m[5m])`) or an expression - /// reading it (`time()`, `now()`), false under a fixed `@ T` or with no - /// time at all. A stored result that does not depend on it delivers on - /// every context. + /// Per output column, in schema order: the sign of its values, from the + /// input groups' and, for a `Scan`, the declared ones ("Value sign"). + pub column_signs: Vec, + /// Whether the result changes with the evaluation time: true for a + /// group under a `TimeRange` relative to the evaluation time + /// (`sum_over_time(m[5m])`) or reading it (`time()`, `now()`), false + /// for one under a fixed `@ T` or reading no time. A result that does + /// not depend on it is the same at every evaluation, so a stored copy + /// delivers on every grid. pub depends_on_evaluation_time: bool, - /// What an aggregate group covers: a summary state, a chain, a merge - /// over a chain's windows, or an exact `Aggregate` result; `None` for - /// any other group. - pub aggregate_coverage: Option, + /// What an aggregate group covers: a summary state, a chain, a + /// `TimeRange`/`TimeShift` over a chain or a merge over one, or an exact + /// `Aggregate` result; `None` for any other group. + pub aggregate_coverage: Option, /// Whether two equal expressions return the same rows for the same /// input rows, so they may be one group (not an unordered `LIMIT`). pub shareable_when_equal: bool, - /// Whether every `Scan` below reads a time series (PromQL rows). A - /// PromQL top-k ranks one row per series; a counter state reads in - /// time order. - pub reads_only_time_series: bool, + /// The sources the `Scan`s below this group read: a `Scan`'s own, else + /// the union over its input groups. + pub scanned_sources: BTreeSet, } -/// What a summary state or exact aggregate computes, with every lifted -/// selection removed. -pub struct SummaryStateDefinition { - /// The group the operators below read: the first input under the - /// SummaryAgg (or Aggregate) that is not Filter / TimeRange / TimeShift - /// / direct-column Project. `None` only when a Scan's predicates are - /// lifted; the Scan, with its residual predicates, is then the last - /// unlifted operator. +// crates/types/src/stages/logical/aggregate_coverage.rs +/// What an aggregate computes, with every lifted selection removed. +pub struct AggregateDefinition { + /// The group the unlifted operators read: the first input under the + /// `SummaryAgg` or exact `Aggregate` that is not a `Filter`, a range + /// `TimeRange`, a `TimeShift` without `@ start()`/`@ end()` or a + /// direct-column `Project`. `None` when that input is a `Scan` with a + /// liftable predicate (a PromQL label matcher): it is lifted too, and the + /// `Scan` with the rest of its predicates is the last unlifted operator, + /// so states over one source with different matchers may merge. pub input_group: Option, - /// The operators not lifted into the selection, listed top down from - /// the SummaryAgg (or Aggregate), each with only its residual arguments. `None` marks - /// an operator's main input (the next unlifted operator, or input_group); + /// The operators not lifted into the selection, listed top down from the + /// `SummaryAgg` or `Aggregate`, each with only its residual arguments. `None` marks an + /// operator's main input (the next unlifted operator, or input_group); /// `Some(g)` is another input group it reads, such as the group a scalar /// subquery in a Filter reads. Two states may merge only if input_group /// and these operators are equal. pub unlifted_operators: Vec>>, } -pub struct SummaryCoverage { - pub(crate) definition: SummaryStateDefinition, - /// Union of boxes over the output rows of the operators' input. +pub struct AggregateCoverage { + pub(crate) definition: AggregateDefinition, + /// Union of boxes over the output rows of the unlifted operators' input. pub(crate) selection: Vec, } -impl SummaryCoverage { - pub(crate) fn derive_from_inputs( - operator: &Operator>, - ) -> Result; - /// A hash of the `SummaryStateDefinition`; equal definitions in one - /// memo get equal ids. - pub(crate) fn definition_id(&self) -> String; -} +/// The coverage of what a memo group computes with `operator` over its +/// input groups; `CoverageError::NoCoverage` when it is no aggregate (or a +/// `TimeRange`/`TimeShift` over a chain whose windows are not one interval +/// on the timestamp column), and the merge errors (`EmptyMerge`, +/// `DefinitionMismatch`, `InputsOverlap`, …) for an invalid merge. +pub(super) fn aggregate_coverage( + operator: &Operator>, +) -> Result; +// crates/types/src/stages/logical/memo/mod.rs pub struct Memo { - // Private: groups are added only through `find_or_add_group`, so the - // hash-consing index covers every group. + /// The groups, by id (`groups()`); added only through + /// `find_or_add_group`. groups: Vec, - /// The queries, by index in the workload. - pub queries: Vec, - // Private: the groups `find_or_add_group` may return, by expression hash. - groups_by_expression_hash: HashMap>, - // Private: the data workload's facts of each source the queries scan, by - // the name the Scan reads (a counter's `value` non-negative): a `Scan`'s - // `column_sign` ("Value sign") and the distinct bound of - // `max_distinct_per_evaluation`. - source_facts: BTreeMap, + /// The queries, by index in the workload (`queries()`); added only + /// through `add_query`. + queries: Vec, + /// The groups `find_or_add_group` may return, by `expression_hash` of + /// their expression: every group whose sharing is legal + /// (`GroupProperties::shareable_when_equal`). + groups_by_expression_hash: HashMap>, + /// The data workload's facts per source (`DataWorkload::sources`), fixed + /// at construction: a `Scan` group's column signs read them. + pub(super) source_facts: BTreeMap, } +/// One query of the memo: its root over groups, its accuracy target and +/// $n_q$. pub struct MemoQuery { pub root: QueryRoot, - /// A_q; `None` when the query has none (accuracy not enforced). + /// $A_q$; `None` for a query without one (every query when accuracy is + /// not enforced). pub accuracy_target: Option, - /// n_q; `None` when the data workload does not bound it. - pub values_per_evaluation: Option, + /// $n_q$, the number of values one evaluation returns; `None` when the + /// data workload does not bound it. Pass 2 turns a whole-result failure + /// probability into a per-value one with it. + pub number_of_values_per_query_evaluation: Option, } impl Memo { - /// A memo without groups over the sources `source_facts` describes. - pub fn new(source_facts: BTreeMap) -> Self; - /// The most distinct items one distinct-count summary over `group`'s rows - /// counts in one evaluation: the sum of the scanned sources' declared - /// `max_distinct_per_evaluation`; `None` when one has none. - pub fn max_distinct_per_evaluation(&self, group: MemoGroupId) -> Option; + /// A memo without groups over the sources `data` describes: a `Scan` + /// group's column signs are the ones `data` declares for its source + /// (`GroupProperties::column_signs`). + pub fn new(data: &DataWorkload) -> Self; + /// The groups, by id. pub fn groups(&self) -> &[MemoGroup]; - /// Adds a query; its root's groups and those below list it. + /// The queries, by index in the workload. + pub fn queries(&self) -> &[MemoQuery]; + /// Adds `query` and returns its index: its root's groups and every + /// group below them list it (`MemoGroup::reached_by_queries`). Stage 0 + /// adds each query once. pub fn add_query(&mut self, query: MemoQuery) -> usize; } ``` +Not yet in the code: `Memo::max_distinct_per_evaluation(group)`, the most distinct items one distinct-count summary over `group`'s rows counts in one evaluation, the sum of the scanned sources' declared `max_distinct_per_evaluation` (`None` when one has none), and `GroupProperties::column_lineage` (Column lineage, below). + How the definition and selection are derived, with examples: `docs/design_docs/proposals/asap-primitive-schema.md` §4.2.2. The rules read an operator's inputs through one trait, so the same rules derive a front-end node's schema and a group's `GroupProperties`: ```rust -pub(crate) trait OperatorInput: Sized { +// crates/types/src/ir/operator/input.rs +pub trait OperatorInput: Sized + Clone { fn schema(&self) -> &Schema; fn result_kind(&self) -> OperatorResultKind; /// The input's own operator, its children as inputs again. fn operator(&self) -> Operator; - fn aggregate_coverage(&self) -> Option<&SummaryCoverage>; + /// What the input covers when it is an aggregate group in the memo; + /// `None` for any other input. + fn aggregate_coverage(&self) -> Option<&AggregateCoverage>; + /// The sources the `Scan`s at or below the input read + /// (`sources_scanned_by` of its operator): a memo group's stored + /// `GroupProperties::scanned_sources`, recomputed for a node tree. + fn scanned_sources(&self) -> BTreeSet; } -// Front-end node trees; aggregate_coverage is None. +/// A front-end node tree. Node trees carry no aggregate coverage. impl OperatorInput for Rc { … } -/// A memo group read as an input: its group_properties and expression. -pub(crate) struct MemoGroupAsInput<'m> { memo: &'m Memo, group_id: MemoGroupId } + +// crates/types/src/stages/logical/group.rs +/// A memo group read as an operator's input: its `group_properties` and +/// its `expression`, whose inputs are read the same way. +pub(super) struct MemoGroupAsInput<'m> { + pub(super) memo: &'m Memo, + pub(super) group_id: MemoGroupId, +} impl OperatorInput for MemoGroupAsInput<'_> { … } // The existing rules, generic over the input; existing names kept. @@ -468,24 +506,33 @@ impl Operator { pub(crate) fn validate_inputs(&self) -> Result<(), SchemaDerivationError>; pub(crate) fn is_shareable_when_equal(&self) -> bool; } -fn derive_group_properties( +/// Whether `operator` is a valid expression over its input groups, never a +/// node tree: its output schema derives, its inputs pass validation, and a +/// merge reads only states it may merge. Returns the output schema and, +/// for a merge, the coverage its check derived. +pub(super) fn validate_expression_over_input_groups( + operator: &Operator>, +) -> Result<(Schema, Option), SchemaDerivationError>; +/// The properties of a group computing `expression` over its input +/// groups, once it is valid over them. +pub(super) fn derive_group_properties( memo: &Memo, expression: &Operator, ) -> Result; ``` -Aggregate coverage exists only in the memo: a node tree has none, and `SummaryCoverage` has no type parameter. The executor reads only `definition_id` from the plan. +Aggregate coverage exists only in the memo: a node tree has none, and `AggregateCoverage` has no type parameter. Identity has two scopes. In the memo, coverages compare by `AggregateDefinition` `==` (it holds `MemoGroupId`s); nothing in the memo needs an id. The plan carries `ViewCoverage::definition_id` (`crates/types/src/plan/materialized_view.rs`), a content id of what the view computes: its definition with every input group expanded recursively to its expression down to the `Scan`s, with no group or node ids, canonically serialized and hashed, so it is equal across plans and processes that compute the same thing and different when they do not. A group is its expression: one operator, with its arguments, over input groups. Hash-consing compares and hashes the expression directly, so two expressions are one group when their operators and arguments are equal and their input groups are the same. θ is not an operator argument, so builds that differ only in θ share one group. An expression whose copies may return different rows (an unordered `LIMIT`) always gets a new group. -Everything that was computed on operator-node trees is computed on groups. `find_or_add_group` derives `GroupProperties` once, when the group is created, with `derive_group_properties`: the rules that check operator nodes (schema derivation, input validation, merge check), applied to the expression over `MemoGroupAsInput`s, which read only the input groups' `expression` and `group_properties`, never a node tree: +Everything that was computed on operator-node trees is computed on groups. `find_or_add_group` derives `GroupProperties` once, when the group is created, with `derive_group_properties`: the rules that check operator nodes (schema derivation, input validation, merge check: `validate_expression_over_input_groups`), applied to the expression over `MemoGroupAsInput`s, which read only the input groups' `expression` and `group_properties`, never a node tree: - `schema` and `result_kind` from the operator and its input groups' schemas and result kinds; a scalar subquery reads the referenced group's schema. - `aggregate_coverage`, for a summary state or an exact `Aggregate` result: walking down from the SummaryAgg or Aggregate through the input groups' expressions while each is a Filter, a TimeRange, a TimeShift (an offset or `@ T`; `TimeShift { at: End | Start }` is not lifted, see below) or a direct-column Project, lifting their predicates into the selection; a TimeRange or TimeShift becomes an interval on the timestamp column (below). The group below them is the definition's `input_group`. A Scan's predicates (PromQL label matchers, SQL filters, including absolute timestamp filters) are lifted too; the Scan with its residual predicates is then the last unlifted operator and `input_group` is `None`. The operators not lifted, each with its residual arguments, are `unlifted_operators`. A chain `SummaryTumblingWindowChain[w](X)` has the definition of a build over X and leaves the timestamp column unconstrained; a `TimeRange` or `TimeShift` over a chain constrains it as over raw rows. A merge's coverage comes from its inputs' stored coverage; a merge is valid only over inputs with equal definitions and disjoint selections, or over one time-indexed state. So the coverage of `SummaryMerge(TimeRange(TimeShift(Chain[w](X))))` equals that of `SummaryAgg(TimeRange(TimeShift(X)))`, and `add_candidate` accepts the merge in the original state group. - The checks that read an input's operator (a merge over a time-indexed state reads a `TimeRange` over a chain; a selected build) read the input group's `expression`. - `shareable_when_equal`: a `LIMIT` is shareable over a `Sort` with the same partitions (the input group's `expression`); a `Dedup` on every column (the input group's schema). -- `column_lineage` and `column_sign` (below), per output column. -- `reads_only_time_series`: a `Scan` of a `Source::TimeSeries` reads one; any other operator does when all its input groups do. A node tree computes the same rule through its children (`OperatorInput::reads_only_time_series`), so the rule is written once. +- `column_signs` and, once added, `column_lineage` (below), per output column. +- `scanned_sources`: a `Scan`'s own source; any other operator's is the union over its input groups'. A node tree computes the same rule through its children (`OperatorInput::scanned_sources`), so the rule is written once. A rule that needs time-series rows (a PromQL top-k ranks one row per series; a counter state reads in time order) checks that every scanned source `Source::is_time_series`. `add_candidate` derives a candidate's properties with the same `derive_group_properties` and requires its schema and result kind to equal the group's. `OperatorNode` recursively contains chosen child nodes and therefore describes a fixed plan fragment; `Operator` contains input group IDs and leaves their implementations open. A candidate needs its own expression because a group may offer several ways to compute the same result. @@ -493,7 +540,7 @@ Removing the one-field `CandidateExpression` wrapper is possible but not adopted #### Column lineage -Every output column of a group has a **lineage**: the source column its values are copied from, or none when the column is computed. It is a group property, derived once from the input groups' lineage (as optd's column-reference property is), and it replaces the table qualifier that schema fields carried. A schema field keeps only its name, type and nullability; a front end resolves a qualified name such as `f.src` with its own binding scope while it lowers the query, and the IR never needs the qualifier again. +Its key is decided in PR 2b (`SourceColumn` was removed in #1230). Every output column of a group has a **lineage**: the source column its values are copied from, or none when the column is computed. It is a group property, derived once from the input groups' lineage (as optd's column-reference property is), and it replaces the table qualifier that schema fields carried. A schema field keeps only its name, type and nullability; a front end resolves a qualified name such as `f.src` with its own binding scope while it lowers the query, and the IR never needs the qualifier again. ```rust pub struct GroupProperties { @@ -502,10 +549,10 @@ pub struct GroupProperties { /// One entry per output column, in schema order. pub column_lineage: Vec, /// One entry per output column, in schema order (Value sign, below). - pub column_sign: Vec, - pub aggregate_coverage: Option, + pub column_signs: Vec, + pub aggregate_coverage: Option, pub shareable_when_equal: bool, - pub reads_only_time_series: bool, + pub scanned_sources: BTreeSet, } /// A column of a source (a Scan's table or metric), by name. @@ -580,7 +627,7 @@ A range-evaluated query evaluates its expression at every step t of [start, end] Lineage and sign (Value sign, below) are derived together, by one per-operator column layout: an output column that copies an input column keeps its lineage and sign; a computed one is `Computed`, with the sign the operator's rule gives. **What reads it:** -- **Summary coverage.** A selection constrains columns keyed by their `SourceColumn` lineage, not by (qualifier, name). A predicate on a `Computed` column is not lifted; it stays in `unlifted_operators`. A merge's group keys must have equal lineage. +- **Aggregate coverage.** A selection constrains columns keyed by their `SourceColumn` lineage, not by (qualifier, name). A predicate on a `Computed` column is not lifted; it stays in `unlifted_operators`. A merge's group keys must have equal lineage. - **Candidate checks.** `add_candidate` compares a candidate's schema with its group's exactly: names, types, nullability, order. There is no qualifier to compare, so a readout renamed to the group's column name matches. **Example 1: a SQL aggregate and its summary candidate.** @@ -624,33 +671,46 @@ After the join, `region` may exist on both sides. The front end resolves `s.regi Some error contracts hold only for values of a known sign: a relative error of a sum or average, DDSketch's relative error on a quantile, and Count-Min's estimates, which need non-negative weights. Each output column of a group has a sign, a group property derived once from the input groups, like lineage: ```rust -/// GroupProperties::column_sign holds one per output column. -pub enum ValueSign { NonNegative, NonPositive, Unknown } +// crates/types/src/stages/logical/column_properties.rs +/// GroupProperties::column_signs holds one per output column. +pub enum ValueSign { + /// Every non-NaN value is ≥ 0. + NonNegative, + /// Every non-NaN value is ≤ 0. + NonPositive, + /// Neither is known. + Unknown, +} ``` -A source's signs are declared per column in the data workload's source facts (for example, `latency`'s value is non-negative). A PromQL counter's value is non-negative without a declaration. A column without a declaration is `Unknown`. `memo_of_queries` collects the facts of the sources the queries scan into the memo (`Memo::new`), and `find_or_add_group` reads their signs for a `Scan`. +A NaN does not break either sign: `NonNegative` means every non-NaN value is ≥ 0. + +A source's signs are declared per column in the data workload's source facts (for example, `latency`'s value is non-negative). A PromQL counter's value is non-negative without a declaration. A column without a declaration is `Unknown`. `Memo::new(data)` stores the data workload's source facts at construction, and `find_or_add_group` reads their signs for a `Scan` (a histogram's or summary's series reads its family's). | Operator | Output column sign | |---|---| | `Scan` | the declared sign of each source column; `Unknown` when none is declared | | `Filter`, `TimeRange`, `TimeShift`, `Sort`, `Limit`, `Dedup`, column `Project` | the input's, unchanged | +| `Join`; `SetOp`, `Concat`; label rewrite (`label_replace`); SQL window function | the left input's columns then the right's (a semi or anti join: the left's only); per column, the inputs' sign when they all agree, else `Unknown`; the input's, the written label `Unknown`; the input's, the window value `Unknown` | | `abs` | `NonNegative` | | negation | flipped: `NonNegative` ↔ `NonPositive` | | numeric literal, numeric cast | the literal's sign; the operand's sign | -| sum, avg, min, max, quantile, top-k, `first_over_time`, `last_over_time` | the input value's sign | -| count, distinct count, and the other counts and spreads (`count_values`, `changes`, `resets`, `group`, `absent`, `present_over_time`, `stddev`, `stdvar`, frequency L2 norm and entropy) | `NonNegative` | +| arithmetic, scalar (`x / 1000`, SQL `value * 8`) or vector (`a / b`), one rule | `+`: both `NonNegative` → `NonNegative`, both `NonPositive` → `NonPositive`; `*` and `/`: same sign → `NonNegative`, opposite → `NonPositive`; `-`, `%`, `^`, `atan2`, or an operand of `Unknown` sign → `Unknown` | +| `vector(s)` | the sign of the expression `s` (`vector(1000)`: `NonNegative`) | +| sum, avg, min, max, top-k, `first_over_time`, `last_over_time`; quantile with 0 ≤ q ≤ 1 | the input value's sign; a quantile with q outside [0, 1]: `Unknown` | +| count, distinct count, and the other counts and spreads (`count_values`, `changes`, `resets`, `group`, `absent`, `absent_over_time`, `present_over_time`, `stddev`, `stdvar`, frequency L2 norm and entropy) | `NonNegative` | | `rate`, `irate`, `increase` | `NonNegative` over a `NonNegative` input (a counter); else `Unknown`, since a decrease reads as a reset | | vector comparison, `and`, `unless`; `or`; comparison with `bool` | the left side's value sign; the sides' sign when they agree; `NonNegative` (0 or 1) | -| `SummaryAgg`, chain | the state records the sign of the values it is updated with (`window_end` is `Unknown`) | +| `SummaryAgg`, chain | the state records the sign of the values it is updated with (a constant weight: its sign; a tuple or entity identity: `Unknown`); keys keep theirs (`window_end` is `Unknown`) | | `SummaryMerge` | the inputs' sign when they all agree, else `Unknown`; over a chain's windows, the chain's build's | -| readouts | the state's recorded sign; a distinct count, frequency L2 norm or entropy, and an exact count's finalization: `NonNegative` | +| readouts | the state's recorded sign; a distinct count, frequency L2 norm or entropy, and an exact count's finalization: `NonNegative`; a quantile with q outside [0, 1]: `Unknown`; a point count or top-k of a Count Sketch or UnivMon (and their variants, `SummaryTypeCapability::estimates_may_flip_sign`): `Unknown`, since a median of signed counters can be negative over non-negative weights (Count-Min keeps the state's sign); a readout whose output does not keep the state's layout: `Unknown` | | any other expression | `Unknown` | **Example.** `topk(5, sum by (host)(gauge))` over a gauge with no declared sign: the sum's sign is `Unknown`, so Stage 2 prunes the Count-Min heap builds (`CountMinWeightsNotProvenNonNegative`) and the query uses an exact top-k. Over `rate(requests_total[1m])`, the sum is `NonNegative` and the Count-Min builds stay. ### Stage 1, Pass 1: initial candidates -Implementation locations: `crates/logical-asap-optimization/src/pass1/per_computation_candidates.rs` and `memo_population.rs`. The #1203 target gives every intermediate result its own group; candidates contain one operator rather than private operator chains. +Implementation locations: `crates/stage1-pass1-per-computation-candidates/src/lib.rs` (`pass1_memo`) and `computation_targets.rs`; the memo methods below are in `crates/types/src/stages/logical/memo/mod.rs`. The #1203 target gives every intermediate result its own group; candidates contain one operator rather than private operator chains. | Alternative | Added representation | |---|---| @@ -670,8 +730,10 @@ pub struct CandidateForMemoGroupId { impl Memo { /// The group of `expression` (operator, arguments and input groups), - /// added if new. Fails when the expression breaks the node rules - /// (schema derivation, input validation, merge check). + /// added if new. Fails, leaving the memo unchanged, when the expression + /// is not valid over its input groups + /// (`validate_expression_over_input_groups`: schema derivation, input + /// validation, merge check). pub fn find_or_add_group( &mut self, expression: Operator, @@ -679,7 +741,8 @@ impl Memo { /// Adds `candidate` to `group`, or returns the id of the candidate there /// with the same expression and logical properties. The candidate's - /// expression is checked by the same node rules as a group's, and its + /// expression is checked by `validate_expression_over_input_groups`, as + /// a group's is, and its /// output schema and kind must equal the group's; a rejected candidate /// leaves the memo unchanged. pub fn add_candidate( @@ -694,7 +757,7 @@ impl Memo { /// input group. The only way to change a candidate's r^log: Pass 2 sets /// a readout's required accuracy after the rules have run. Panics when /// the vector does not have one entry per input group. - pub fn set_input_requirements( + pub fn set_candidate_input_requirements( &mut self, id: CandidateForMemoGroupId, input_requirements: Vec, @@ -702,9 +765,10 @@ impl Memo { } pub enum CandidateError { - /// The candidate's expression fails the node rules (schema derivation, - /// input validation, merge check). - Schema(SchemaDerivationError), + /// The candidate's expression is not valid over its input groups (its + /// output schema does not derive, an input fails validation, or a merge + /// reads states it may not merge): a bug in the rule that built it. + NotValidOverInputGroups(SchemaDerivationError), /// Its output schema or output kind differs from the group's: it does /// not compute the group's result. OutputDiffersFromGroup { group: MemoGroupId }, @@ -718,7 +782,7 @@ pub struct CandidateForMemoGroup { pub expression: CandidateExpression, pub logical_properties: LogicalProperties, /// r^log_{c,i}, one per input group; changed only through - /// `Memo::set_input_requirements`. + /// `Memo::set_candidate_input_requirements`. pub input_requirements: Vec, /// The groups of the operators this candidate fuses: a sub-DAG pattern /// of several operators, in one appearance of the sub-expression, @@ -783,7 +847,7 @@ Pass 2 (`crates/stage1-pass2-reuse-candidates`) runs these steps in order: |---|---|---| | 1. Rollup from the finest common group | Adds candidates that compute a coarser group from a finer one, and synthesizes one finest common group per cluster of equal definitions | Its candidates are new consumers that later steps must see | | 2. Top-k reuse | Reads a compatible larger top-k result with an exact-input requirement | Same reason | -| 3. Reach, caps and readout requirements | Computes which queries reach each candidate along candidate edges, the validity and pruning caps (Section 4), and each readout group's requirement levels: one readout candidate per level, its `RequiredAccuracy` written with `Memo::set_input_requirements` | Needs every consumer added by steps 1 and 2 | +| 3. Reach, caps and readout requirements | Computes which queries reach each candidate along candidate edges, the validity and pruning caps (Section 4), and each readout group's requirement levels: one readout candidate per level, its `RequiredAccuracy` written with `Memo::set_candidate_input_requirements` | Needs every consumer added by steps 1 and 2 | | 4. Dead-candidate elimination | Removes candidates no query can use, transitively, and records the root cause of each | Needs the requirements; keeps step 5 from building windows for dead candidates | | 5. Chain groups | Adds tumbling-window chain groups and the merges that read them, with widths from the queries that reach each state group, then repeats step 4 over the memo with the chains | Needs the reach of step 3, including rollup consumers, and only the survivors of step 4; the repeat prunes a merge whose θ serves no consumer | | 6. Logical edges | Records every (producer, consumer, input) and whether it satisfies the input's logical requirement | Runs on the final memo | @@ -878,7 +942,7 @@ With many pairs and queries ③ is cheapest; with few pairs ① or ② can be. A #### Reach, caps and readout requirements -A query reaches a candidate when a path of candidate edges leads from the query's output group to it; this is the reach $a_{q,v}$, computed top down after steps 1 and 2 so that rollup consumers count. Section 4 defines the caps from it, and which requirement levels each readout group gets ("Readout requirements"). Pass 1's readout candidate takes the first level and `add_candidate` adds one readout candidate per further level; `Memo::set_input_requirements` writes each one's `RequiredAccuracy`, and nothing else changes a candidate's r^log. Two readout candidates with the same expression and different requirements are different candidates. +A query reaches a candidate when a path of candidate edges leads from the query's output group to it; this is the reach $a_{q,v}$, computed top down after steps 1 and 2 so that rollup consumers count. Section 4 defines the caps from it, and which requirement levels each readout group gets ("Readout requirements"). Pass 1's readout candidate takes the first level and `add_candidate` adds one readout candidate per further level; `Memo::set_candidate_input_requirements` writes each one's `RequiredAccuracy`, and nothing else changes a candidate's r^log. Two readout candidates with the same expression and different requirements are different candidates. #### Dead-candidate elimination @@ -1061,12 +1125,12 @@ A candidate that carries a θ without building (a merge, or a `Filter`, `TimeRan ### Stage 2: physical alternatives -`enumerate_candidate_configurations` in `crates/stage2-physical-optimization/src/catalog/configurations.rs` expands each logical candidate that survived Pass 2 using the dimensions and checks in Section 2. It offers τ = IngestionTime only to a candidate that is time-independent (its output does not depend on the evaluation time t, as a chain's does not) or that is an input of one. It offers ℓ = AcrossEvaluations to chains and to the candidates of any group evaluated on a known grid: a time-independent result is materialized as is, a time-dependent one per evaluation time, as a time-indexed result series. Values not offered are not recorded. It creates no new logical candidates. Each logical edge is expanded between the relevant physical alternatives. +`enumerate_candidate_configurations` in `crates/stage2-physical-optimization` expands each logical candidate that survived Pass 2 using the dimensions and checks in Section 2. It offers τ = IngestionTime only to a candidate that is time-independent (its output does not depend on the evaluation time t, as a chain's does not) or that is an input of one. It offers ℓ = AcrossEvaluations to chains and to the candidates of any group evaluated on a known grid: a time-independent result is materialized as is, a time-dependent one per evaluation time, as a time-indexed result series. Values not offered are not recorded. It creates no new logical candidates. Each logical edge is expanded between the relevant physical alternatives. Stage 2 reads the logical candidates, the data workload (per-source arrival, below) and the deployment's capabilities: ```rust -// crates/stage2-physical-optimization/src/catalog/mod.rs +// crates/stage2-physical-optimization/src/lib.rs pub fn generate_physical_asap_memo_candidates( logical: &LogicalASAPMemoCandidates, data: &DataWorkload, @@ -1077,7 +1141,7 @@ pub fn generate_physical_asap_memo_candidates( #### Data arrival ```rust -// crates/types/src/workload/mod.rs +// crates/types/src/workload/data.rs pub enum DataArrival { /// Every row is already stored; no build runs at ingestion time. AtRest, @@ -1107,12 +1171,12 @@ Stage 2 records every alternative it rejects, with one of these reasons; Stage 3 | `QueryTimeResultsNotRetained` | (QueryTime, AcrossEvaluations) when the deployment retains no query-time results; (QueryTime, OneEvaluation) is kept, since it lives within one evaluation | | `InputHasNoBuildableProducer` | a candidate for which some input has no surviving producer that satisfies α^phys | | `ReachesNoQueryOutput` | a candidate from which no admissible path leads to a query root | -| `CountMinWeightsNotProvenNonNegative` | a Count-Min build (`Cms`, `CmsWithHeap`, `HydraCms`) whose weight column's `column_sign` is not `NonNegative`: its estimates are bounds only for non-negative weights | +| `CountMinWeightsNotProvenNonNegative` | a Count-Min build (`Cms`, `CmsWithHeap`, `HydraCms`) whose weight column's `column_signs` is not `NonNegative`: its estimates are bounds only for non-negative weights | These are the only reasons. The planner does not check executor support, so no reason names it. ```rust -// crates/types/src/physical/properties.rs +// crates/types/src/ir/properties/execution_data_state.rs pub enum ExecutionTiming { IngestionTime, QueryTime, @@ -1300,8 +1364,10 @@ pub struct PhysicalASAPDAGNode { /// How long the node's time-indexed rows are retained (h_v); set on /// materialized nodes. pub retention_ms: Option, - /// Set only on AcrossEvaluations nodes. - pub summary_coverage: Option, + // A materialized view's coverage is not a node field: the plan derives + // it per view (`MaterializedView::coverage`, a `ViewCoverage`: the + // content `definition_id` and the selection) from the node's memo + // group's `aggregate_coverage`. /// The physical candidate the node was built from: in-memory provenance /// for validation and tools, not part of the plan file; None for a plan /// built by a custom pass. @@ -1364,7 +1430,7 @@ $$ | avg aggregate, absolute error | β = 1 | | Ratio, relative error | Requires nonzero true denominator and denominator relative error below one; bound given below | | Product, relative error | (1 + b) for the left operand and 1 for the right, whose relative error is capped at b: $(1+x)(1+y)-1\le(1+b)x+y$ when $y\le b$ | -| sum / avg, relative error | β = 1 when the input column's `column_sign` is `NonNegative` or `NonPositive` (all values of one sign); no contract when it is `Unknown` | +| sum / avg, relative error | β = 1 when the input column's `column_signs` is `NonNegative` or `NonPositive` (all values of one sign); no contract when it is `Unknown` | A constant contract is unconditional: it needs no row count or sign. A filter does not automatically preserve every accuracy guarantee; a `Filter` that compares an approximate value (an alert threshold such as `quantile_over_time(…) > 0.5`) has no contract. Group counts may need metadata beyond the expression syntax. Each factor must follow from the operator's error contract. @@ -1613,7 +1679,7 @@ Every term is exact: no producer is chosen to stand for others. In the example o ### No consumer-count policies -Today's `ReaderConstraint` (`crates/types/src/stages/physical/graph.rs`) is deleted with all four variants. `IncompatibleOutputs` and `AllowedOutputs` are replaced by the alignment check of Section 2. `MinimumQueryReaders` (keep a candidate only if enough queries reach it) and `HighestSharedQueryMaterialization` (materialize where sharing stops) are left to the cost model: materializing for one query, or below the point where sharing stops, costs more and is not chosen. There is no constraint (J). +`ReaderConstraint` (formerly `crates/types/src/stages/physical/graph.rs`) is deleted with all four variants. `IncompatibleOutputs` and `AllowedOutputs` are replaced by the alignment check of Section 2. `MinimumQueryReaders` (keep a candidate only if enough queries reach it) and `HighestSharedQueryMaterialization` (materialize where sharing stops) are left to the cost model: materializing for one query, or below the point where sharing stops, costs more and is not chosen. There is no constraint (J). ### Out of this design From 1ecd13dff8da890c3b010dfd73680307739f1596 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Tue, 10 Nov 2026 19:50:56 +0000 Subject: [PATCH 63/67] docs(concepts): SketchParams and parameter_list paths in the summary_type module --- .../design_docs/concepts/memo-groups-properties-requirements.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index eb2e42270..37f84a45f 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -94,7 +94,7 @@ This is the single reference table for property dimensions. Stage descriptions b | Dimension | Values and meaning | Code reference or target | |---|---|---| -| Parameters θ | The structure of a summary, from a predefined list per algorithm, `parameter_list(algorithm)`: KLL and KMV `k`; DDSketch `alpha`; HLL `precision`; CMS and CountSketch `width` and `depth`; heap variants also `heap_size`; UnivMon its L2 column count and distinct-count heap size. θ is structure only: no field records the error or failure probability a configuration was sized for, and equality is derived over all fields. The lists span today's sizing over ε = 0.1 down to 0.001 (KMV's and UnivMon's distinct-count entries stop at ε = 0.01). Where δ is set by a structural parameter, the list covers the δ levels with it: CMS lists each width with depths 5, 7, 10 and 12 (δ = e^−d ≤ 1e-2, 1e-3, 1e-4, 1e-5). CountSketch reads the median of its rows, each wrong with probability about 1/3, so δ = P(Binomial(d, 1/3) ≥ (d+1)/2), defined for odd d: it lists each width with depths 47, 81, 117 and 155 for the same δ levels. UnivMon's list has 40 entries: each of the 7 L2 column counts with each of the 4 distinct-count heap sizes, each of them alone, and the baseline. A heap variant combines each width × depth with heap sizes from the 1-2-5 series 10, 20, 50, …, 10 000, so the smallest heap that holds a top-k's k is at most 2.5× too large. A sketch build's θ must be in its algorithm's list; exact operators have none. A candidate that passes states on without building them (`SummaryMerge`, and a `Filter`, `TimeRange` or `TimeShift` over states) carries the θ of the states it reads: Pass 2 adds one such candidate per θ among its input group's candidates, and its input edges connect only to producers with the same θ (Section 3, "Logical edges"). So every candidate of a sketch state group has a θ, and a readout's guarantee is that of its producer's θ. Algorithms without an error model (the Hydra algorithms among them) have empty lists, so no build of theirs is a valid candidate until their error models exist (#1228). | `SketchParams` in `crates/types/src/ir/schema/sketch.rs`; `parameter_list` | +| Parameters θ | The structure of a summary, from a predefined list per algorithm, `parameter_list(algorithm)`: KLL and KMV `k`; DDSketch `alpha`; HLL `precision`; CMS and CountSketch `width` and `depth`; heap variants also `heap_size`; UnivMon its L2 column count and distinct-count heap size. θ is structure only: no field records the error or failure probability a configuration was sized for, and equality is derived over all fields. The lists span today's sizing over ε = 0.1 down to 0.001 (KMV's and UnivMon's distinct-count entries stop at ε = 0.01). Where δ is set by a structural parameter, the list covers the δ levels with it: CMS lists each width with depths 5, 7, 10 and 12 (δ = e^−d ≤ 1e-2, 1e-3, 1e-4, 1e-5). CountSketch reads the median of its rows, each wrong with probability about 1/3, so δ = P(Binomial(d, 1/3) ≥ (d+1)/2), defined for odd d: it lists each width with depths 47, 81, 117 and 155 for the same δ levels. UnivMon's list has 40 entries: each of the 7 L2 column counts with each of the 4 distinct-count heap sizes, each of them alone, and the baseline. A heap variant combines each width × depth with heap sizes from the 1-2-5 series 10, 20, 50, …, 10 000, so the smallest heap that holds a top-k's k is at most 2.5× too large. A sketch build's θ must be in its algorithm's list; exact operators have none. A candidate that passes states on without building them (`SummaryMerge`, and a `Filter`, `TimeRange` or `TimeShift` over states) carries the θ of the states it reads: Pass 2 adds one such candidate per θ among its input group's candidates, and its input edges connect only to producers with the same θ (Section 3, "Logical edges"). So every candidate of a sketch state group has a θ, and a readout's guarantee is that of its producer's θ. Algorithms without an error model (the Hydra algorithms among them) have empty lists, so no build of theirs is a valid candidate until their error models exist (#1228). | `SketchParams` in `crates/types/src/summary_type/parameters.rs`; `parameter_list` in `parameter_lists.rs` | | Timing | `IngestionTime` or `QueryTime` | `ExecutionTiming` | | Lifetime | `None`: discard after the consuming execution; `OneEvaluation`: retain until consumers within that evaluation finish; `AcrossEvaluations`: retain across evaluations in the specified period. A non-`None` lifetime means materialized. | `Lifetime`; #1200 | | Storage medium — Extension | `Memory` or `Disk`, carried by a materialized lifetime | `StorageMedium` | From 7fb4202e7c346d4143f415313ac40ecaeda99fbd Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Tue, 10 Nov 2026 19:51:30 +0000 Subject: [PATCH 64/67] docs(concepts): lineage key sentence without PR numbers --- .../design_docs/concepts/memo-groups-properties-requirements.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 37f84a45f..42b2a9e5d 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -540,7 +540,7 @@ Removing the one-field `CandidateExpression` wrapper is possible but not adopted #### Column lineage -Its key is decided in PR 2b (`SourceColumn` was removed in #1230). Every output column of a group has a **lineage**: the source column its values are copied from, or none when the column is computed. It is a group property, derived once from the input groups' lineage (as optd's column-reference property is), and it replaces the table qualifier that schema fields carried. A schema field keeps only its name, type and nullability; a front end resolves a qualified name such as `f.src` with its own binding scope while it lowers the query, and the IR never needs the qualifier again. +Its key is not decided yet: the `SourceColumn` type it named was removed. Every output column of a group has a **lineage**: the source column its values are copied from, or none when the column is computed. It is a group property, derived once from the input groups' lineage (as optd's column-reference property is), and it replaces the table qualifier that schema fields carried. A schema field keeps only its name, type and nullability; a front end resolves a qualified name such as `f.src` with its own binding scope while it lowers the query, and the IR never needs the qualifier again. ```rust pub struct GroupProperties { From 192359baf876f21387d0ca3c5a7fed9f4f2a62f2 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Tue, 10 Nov 2026 19:54:04 +0000 Subject: [PATCH 65/67] docs(concepts): no PR or issue numbers in the concepts docs --- .../concepts/memo-groups-properties-requirements.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 42b2a9e5d..2ba6917d9 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -1,6 +1,6 @@ # Memo groups, properties and requirements -This document explains how the optimizer represents alternatives, checks whether they can be composed, and selects an executable plan. It follows the optimization problem formulation (paper, "Optimization Problem Formulation") for the baseline model and labels additional design work **Extension (#1207)**. +This document explains how the optimizer represents alternatives, checks whether they can be composed, and selects an executable plan. It follows the optimization problem formulation (paper, "Optimization Problem Formulation") for the baseline model and labels additional design work **Extension**. The Rust blocks are interface sketches, not compilable code. Paths name today's code; issue numbers name the work that changes it. @@ -94,9 +94,9 @@ This is the single reference table for property dimensions. Stage descriptions b | Dimension | Values and meaning | Code reference or target | |---|---|---| -| Parameters θ | The structure of a summary, from a predefined list per algorithm, `parameter_list(algorithm)`: KLL and KMV `k`; DDSketch `alpha`; HLL `precision`; CMS and CountSketch `width` and `depth`; heap variants also `heap_size`; UnivMon its L2 column count and distinct-count heap size. θ is structure only: no field records the error or failure probability a configuration was sized for, and equality is derived over all fields. The lists span today's sizing over ε = 0.1 down to 0.001 (KMV's and UnivMon's distinct-count entries stop at ε = 0.01). Where δ is set by a structural parameter, the list covers the δ levels with it: CMS lists each width with depths 5, 7, 10 and 12 (δ = e^−d ≤ 1e-2, 1e-3, 1e-4, 1e-5). CountSketch reads the median of its rows, each wrong with probability about 1/3, so δ = P(Binomial(d, 1/3) ≥ (d+1)/2), defined for odd d: it lists each width with depths 47, 81, 117 and 155 for the same δ levels. UnivMon's list has 40 entries: each of the 7 L2 column counts with each of the 4 distinct-count heap sizes, each of them alone, and the baseline. A heap variant combines each width × depth with heap sizes from the 1-2-5 series 10, 20, 50, …, 10 000, so the smallest heap that holds a top-k's k is at most 2.5× too large. A sketch build's θ must be in its algorithm's list; exact operators have none. A candidate that passes states on without building them (`SummaryMerge`, and a `Filter`, `TimeRange` or `TimeShift` over states) carries the θ of the states it reads: Pass 2 adds one such candidate per θ among its input group's candidates, and its input edges connect only to producers with the same θ (Section 3, "Logical edges"). So every candidate of a sketch state group has a θ, and a readout's guarantee is that of its producer's θ. Algorithms without an error model (the Hydra algorithms among them) have empty lists, so no build of theirs is a valid candidate until their error models exist (#1228). | `SketchParams` in `crates/types/src/summary_type/parameters.rs`; `parameter_list` in `parameter_lists.rs` | +| Parameters θ | The structure of a summary, from a predefined list per algorithm, `parameter_list(algorithm)`: KLL and KMV `k`; DDSketch `alpha`; HLL `precision`; CMS and CountSketch `width` and `depth`; heap variants also `heap_size`; UnivMon its L2 column count and distinct-count heap size. θ is structure only: no field records the error or failure probability a configuration was sized for, and equality is derived over all fields. The lists span today's sizing over ε = 0.1 down to 0.001 (KMV's and UnivMon's distinct-count entries stop at ε = 0.01). Where δ is set by a structural parameter, the list covers the δ levels with it: CMS lists each width with depths 5, 7, 10 and 12 (δ = e^−d ≤ 1e-2, 1e-3, 1e-4, 1e-5). CountSketch reads the median of its rows, each wrong with probability about 1/3, so δ = P(Binomial(d, 1/3) ≥ (d+1)/2), defined for odd d: it lists each width with depths 47, 81, 117 and 155 for the same δ levels. UnivMon's list has 40 entries: each of the 7 L2 column counts with each of the 4 distinct-count heap sizes, each of them alone, and the baseline. A heap variant combines each width × depth with heap sizes from the 1-2-5 series 10, 20, 50, …, 10 000, so the smallest heap that holds a top-k's k is at most 2.5× too large. A sketch build's θ must be in its algorithm's list; exact operators have none. A candidate that passes states on without building them (`SummaryMerge`, and a `Filter`, `TimeRange` or `TimeShift` over states) carries the θ of the states it reads: Pass 2 adds one such candidate per θ among its input group's candidates, and its input edges connect only to producers with the same θ (Section 3, "Logical edges"). So every candidate of a sketch state group has a θ, and a readout's guarantee is that of its producer's θ. Algorithms without an error model (the Hydra algorithms among them) have empty lists, so no build of theirs is a valid candidate until their error models exist. | `SketchParams` in `crates/types/src/summary_type/parameters.rs`; `parameter_list` in `parameter_lists.rs` | | Timing | `IngestionTime` or `QueryTime` | `ExecutionTiming` | -| Lifetime | `None`: discard after the consuming execution; `OneEvaluation`: retain until consumers within that evaluation finish; `AcrossEvaluations`: retain across evaluations in the specified period. A non-`None` lifetime means materialized. | `Lifetime`; #1200 | +| Lifetime | `None`: discard after the consuming execution; `OneEvaluation`: retain until consumers within that evaluation finish; `AcrossEvaluations`: retain across evaluations in the specified period. A non-`None` lifetime means materialized. | `Lifetime` | | Storage medium — Extension | `Memory` or `Disk`, carried by a materialized lifetime | `StorageMedium` | An error guarantee is not a property: it depends on the summary type, θ and the statistic a readout reads, and is checked on the edge from a state candidate (a build, or a merge with the θ it carries) to a readout (below). It has an error metric, a bound, and a failure probability; exact computation introduces no approximation error. @@ -710,7 +710,7 @@ A source's signs are declared per column in the data workload's source facts (fo ### Stage 1, Pass 1: initial candidates -Implementation locations: `crates/stage1-pass1-per-computation-candidates/src/lib.rs` (`pass1_memo`) and `computation_targets.rs`; the memo methods below are in `crates/types/src/stages/logical/memo/mod.rs`. The #1203 target gives every intermediate result its own group; candidates contain one operator rather than private operator chains. +Implementation locations: `crates/stage1-pass1-per-computation-candidates/src/lib.rs` (`pass1_memo`) and `computation_targets.rs`; the memo methods below are in `crates/types/src/stages/logical/memo/mod.rs`. Every intermediate result is its own group; candidates contain one operator rather than private operator chains. | Alternative | Added representation | |---|---| @@ -1400,7 +1400,7 @@ The plan's cost is `plan_cost(&plan)`; `Selection` does not repeat it. Validatio The executor keeps one time-indexed, append-only store with retention for every materialized node with a time index: raw data (a Scan, or a `TimeShift` by an offset over one, materialized at ingestion time), chain states, and results stored per evaluation time. Each is read only through `TimeRange` and `TimeShift` nodes, which select its rows by time, and rows older than the node's `retention_ms` are dropped. A `TimeShift { at: End | Start }` node evaluates its child once per request. Summary operators are constructed with their θ; a summary node without θ is an error. -## 4. Accuracy and safe pruning — Extension (#1207) +## 4. Accuracy and safe pruning — Extension The formulation checks logical requirements per edge. This extension also checks composed accuracy at each query root. This section gives the error model, how readouts get their requirements, and the MILP encoding; error is not a property dimension. @@ -1581,7 +1581,7 @@ A zero factor gives no pruning restriction for that query. If a factor depends o For example, with relative error 0.05 at a ratio, each operand's cap is 0.05 / 1.111 = 0.045 and the denominator's is min(0.1, 0.045) = 0.045, so the readouts get the levels 0.02 and tighter. Root checks still couple all contributing errors and determine feasibility. -## 5. Execution contexts and other selection extensions — #1207 +## 5. Execution contexts and other selection extensions — Extension These extensions go beyond constraints (A)–(I) and the formulation's fixed-cost assumption. They are separated here so that the baseline reads on its own. From 7eeff73ba71b936a02c354b3ca2cd8974b8f2a6d Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Tue, 10 Nov 2026 21:16:46 +0000 Subject: [PATCH 66/67] docs(concepts): coverage in the IR, the plan's content id by expansion, sign-bit value signs Sync with the code on main: AggregateCoverage / AggregateDefinition (input) and their derivation in ir/properties/aggregate_coverage.rs, generic over OperatorInput::Reference, with map_references; ViewCoverage on PhysicalASAPDAGNode::view_coverage, built by stages::view_coverage; ValueSign as the IEEE sign bit; Memo::root_groups / queries_rooted_at; Operator::scanned_sources; point_counts_may_flip_sign. --- .../memo-groups-properties-requirements.md | 112 ++++++++++++------ 1 file changed, 73 insertions(+), 39 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 2ba6917d9..51dac1a58 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -376,7 +376,7 @@ pub struct GroupProperties { /// What an aggregate group covers: a summary state, a chain, a /// `TimeRange`/`TimeShift` over a chain or a merge over one, or an exact /// `Aggregate` result; `None` for any other group. - pub aggregate_coverage: Option, + pub aggregate_coverage: Option>, /// Whether two equal expressions return the same rows for the same /// input rows, so they may be one group (not an unordered `LIMIT`). pub shareable_when_equal: bool, @@ -385,40 +385,48 @@ pub struct GroupProperties { pub scanned_sources: BTreeSet, } -// crates/types/src/stages/logical/aggregate_coverage.rs +// crates/types/src/ir/properties/aggregate_coverage.rs /// What an aggregate computes, with every lifted selection removed. -pub struct AggregateDefinition { - /// The group the unlifted operators read: the first input under the +/// `R` is how it names an input: a memo group's id in the memo. +pub struct AggregateDefinition { + /// The input the unlifted operators read: the first input under the /// `SummaryAgg` or exact `Aggregate` that is not a `Filter`, a range /// `TimeRange`, a `TimeShift` without `@ start()`/`@ end()` or a /// direct-column `Project`. `None` when that input is a `Scan` with a /// liftable predicate (a PromQL label matcher): it is lifted too, and the /// `Scan` with the rest of its predicates is the last unlifted operator, /// so states over one source with different matchers may merge. - pub input_group: Option, + pub input: Option, /// The operators not lifted into the selection, listed top down from the /// `SummaryAgg` or `Aggregate`, each with only its residual arguments. `None` marks an - /// operator's main input (the next unlifted operator, or input_group); - /// `Some(g)` is another input group it reads, such as the group a scalar - /// subquery in a Filter reads. Two states may merge only if input_group + /// operator's main input (the next unlifted operator, or `input`); + /// `Some(r)` is another input it reads, such as the input a scalar + /// subquery in a Filter reads. Two states may merge only if `input` /// and these operators are equal. - pub unlifted_operators: Vec>>, + pub unlifted_operators: Vec>>, } -pub struct AggregateCoverage { - pub(crate) definition: AggregateDefinition, +/// What an aggregate covers, its inputs named by `R`. +pub struct AggregateCoverage { + pub(crate) definition: AggregateDefinition, /// Union of boxes over the output rows of the unlifted operators' input. pub(crate) selection: Vec, } -/// The coverage of what a memo group computes with `operator` over its -/// input groups; `CoverageError::NoCoverage` when it is no aggregate (or a -/// `TimeRange`/`TimeShift` over a chain whose windows are not one interval -/// on the timestamp column), and the merge errors (`EmptyMerge`, -/// `DefinitionMismatch`, `InputsOverlap`, …) for an invalid merge. -pub(super) fn aggregate_coverage( - operator: &Operator>, -) -> Result; +impl AggregateCoverage { + /// The same coverage with each input reference `r` replaced by `f(r)`. + pub fn map_references(self, f: impl FnMut(R) -> S) -> AggregateCoverage; +} + +impl Operator { + /// The coverage of what this operator computes over its inputs; + /// `CoverageError::NoCoverage` when it is no aggregate (or a + /// `TimeRange`/`TimeShift` over a chain whose windows are not one + /// interval on the timestamp column), and the merge errors + /// (`EmptyMerge`, `DefinitionMismatch`, `InputsOverlap`, …) for an + /// invalid merge. + pub fn aggregate_coverage(&self) -> Result, CoverageError>; +} // crates/types/src/stages/logical/memo/mod.rs pub struct Memo { @@ -463,6 +471,11 @@ impl Memo { /// group below them list it (`MemoGroup::reached_by_queries`). Stage 0 /// adds each query once. pub fn add_query(&mut self, query: MemoQuery) -> usize; + /// The groups some query's root reads, ascending: the groups of the + /// queries' roots that are a query's result or a scalar query's operand. + pub fn root_groups(&self) -> BTreeSet; + /// The queries whose root reads `group`, ascending. + pub fn queries_rooted_at(&self, group: MemoGroupId) -> Vec; } ``` @@ -479,15 +492,22 @@ pub trait OperatorInput: Sized + Clone { fn result_kind(&self) -> OperatorResultKind; /// The input's own operator, its children as inputs again. fn operator(&self) -> Operator; - /// What the input covers when it is an aggregate group in the memo; - /// `None` for any other input. - fn aggregate_coverage(&self) -> Option<&AggregateCoverage>; + /// How an aggregate's definition names this input + /// (`AggregateCoverage`): a memo group's id in the memo. + type Reference: Clone + PartialEq; + /// This input's reference; `None` for a build a rule constructs only + /// while it derives, which has none. + fn reference(&self) -> Option; + /// What the input covers when its coverage is stored (an aggregate + /// group in the memo); `None` for any other input. + fn aggregate_coverage(&self) -> Option<&AggregateCoverage>; /// The sources the `Scan`s at or below the input read - /// (`sources_scanned_by` of its operator): a memo group's stored + /// (`Operator::scanned_sources` of its operator): a memo group's stored /// `GroupProperties::scanned_sources`, recomputed for a node tree. fn scanned_sources(&self) -> BTreeSet; } -/// A front-end node tree. Node trees carry no aggregate coverage. +/// A front-end node tree (`Reference = ()`). Node trees store no aggregate +/// coverage. impl OperatorInput for Rc { … } // crates/types/src/stages/logical/group.rs @@ -497,10 +517,15 @@ pub(super) struct MemoGroupAsInput<'m> { pub(super) memo: &'m Memo, pub(super) group_id: MemoGroupId, } +/// `Reference = MemoGroupId`; its stored coverage is +/// `GroupProperties::aggregate_coverage`. impl OperatorInput for MemoGroupAsInput<'_> { … } // The existing rules, generic over the input; existing names kept. impl Operator { + /// The sources the `Scan`s at or below this operator read: a `Scan`'s + /// own, else the union over its inputs'. + pub(crate) fn scanned_sources(&self) -> BTreeSet; pub(crate) fn output_schema(&self) -> Result; pub(crate) fn output_kind(&self) -> OperatorResultKind; pub(crate) fn validate_inputs(&self) -> Result<(), SchemaDerivationError>; @@ -512,7 +537,7 @@ impl Operator { /// for a merge, the coverage its check derived. pub(super) fn validate_expression_over_input_groups( operator: &Operator>, -) -> Result<(Schema, Option), SchemaDerivationError>; +) -> Result<(Schema, Option>), SchemaDerivationError>; /// The properties of a group computing `expression` over its input /// groups, once it is valid over them. pub(super) fn derive_group_properties( @@ -521,14 +546,14 @@ pub(super) fn derive_group_properties( ) -> Result; ``` -Aggregate coverage exists only in the memo: a node tree has none, and `AggregateCoverage` has no type parameter. Identity has two scopes. In the memo, coverages compare by `AggregateDefinition` `==` (it holds `MemoGroupId`s); nothing in the memo needs an id. The plan carries `ViewCoverage::definition_id` (`crates/types/src/plan/materialized_view.rs`), a content id of what the view computes: its definition with every input group expanded recursively to its expression down to the `Scan`s, with no group or node ids, canonically serialized and hashed, so it is equal across plans and processes that compute the same thing and different when they do not. +Aggregate coverage is an IR property, derived like an output schema (`Operator::aggregate_coverage`), generic over how a definition names an input (`OperatorInput::Reference`). The memo derives it over its groups (`R` = `MemoGroupId`) and stores it per group; a node tree stores none. Identity has two scopes. In the memo, coverages compare by `AggregateDefinition` `==` (it holds `MemoGroupId`s); nothing in the memo needs an id. The plan carries `ViewCoverage::definition_id` on a materialized view's node (`PhysicalASAPDAGNode::view_coverage`, `crates/types/src/plan/physical_asap_dag.rs`), a content id of what the view computes: `stages::view_coverage(memo, group)` expands the memo's coverage (`map_references`, every input group to its expression, recursively down to the `Scan`s, with no group or node ids), serializes it canonically and hashes it, so it is equal across plans and processes that compute the same thing and different when they do not. The plan does not re-derive coverage. A group is its expression: one operator, with its arguments, over input groups. Hash-consing compares and hashes the expression directly, so two expressions are one group when their operators and arguments are equal and their input groups are the same. θ is not an operator argument, so builds that differ only in θ share one group. An expression whose copies may return different rows (an unordered `LIMIT`) always gets a new group. Everything that was computed on operator-node trees is computed on groups. `find_or_add_group` derives `GroupProperties` once, when the group is created, with `derive_group_properties`: the rules that check operator nodes (schema derivation, input validation, merge check: `validate_expression_over_input_groups`), applied to the expression over `MemoGroupAsInput`s, which read only the input groups' `expression` and `group_properties`, never a node tree: - `schema` and `result_kind` from the operator and its input groups' schemas and result kinds; a scalar subquery reads the referenced group's schema. -- `aggregate_coverage`, for a summary state or an exact `Aggregate` result: walking down from the SummaryAgg or Aggregate through the input groups' expressions while each is a Filter, a TimeRange, a TimeShift (an offset or `@ T`; `TimeShift { at: End | Start }` is not lifted, see below) or a direct-column Project, lifting their predicates into the selection; a TimeRange or TimeShift becomes an interval on the timestamp column (below). The group below them is the definition's `input_group`. A Scan's predicates (PromQL label matchers, SQL filters, including absolute timestamp filters) are lifted too; the Scan with its residual predicates is then the last unlifted operator and `input_group` is `None`. The operators not lifted, each with its residual arguments, are `unlifted_operators`. A chain `SummaryTumblingWindowChain[w](X)` has the definition of a build over X and leaves the timestamp column unconstrained; a `TimeRange` or `TimeShift` over a chain constrains it as over raw rows. A merge's coverage comes from its inputs' stored coverage; a merge is valid only over inputs with equal definitions and disjoint selections, or over one time-indexed state. So the coverage of `SummaryMerge(TimeRange(TimeShift(Chain[w](X))))` equals that of `SummaryAgg(TimeRange(TimeShift(X)))`, and `add_candidate` accepts the merge in the original state group. +- `aggregate_coverage`, for a summary state or an exact `Aggregate` result: walking down from the SummaryAgg or Aggregate through the input groups' expressions while each is a Filter, a TimeRange, a TimeShift (an offset or `@ T`; `TimeShift { at: End | Start }` is not lifted, see below) or a direct-column Project, lifting their predicates into the selection; a TimeRange or TimeShift becomes an interval on the timestamp column (below). The group below them is the definition's `input`. A Scan's predicates (PromQL label matchers, SQL filters, including absolute timestamp filters) are lifted too; the Scan with its residual predicates is then the last unlifted operator and `input` is `None`. The operators not lifted, each with its residual arguments, are `unlifted_operators`. A chain `SummaryTumblingWindowChain[w](X)` has the definition of a build over X and leaves the timestamp column unconstrained; a `TimeRange` or `TimeShift` over a chain constrains it as over raw rows. A merge's coverage comes from its inputs' stored coverage; a merge is valid only over inputs with equal definitions and disjoint selections, or over one time-indexed state. So the coverage of `SummaryMerge(TimeRange(TimeShift(Chain[w](X))))` equals that of `SummaryAgg(TimeRange(TimeShift(X)))`, and `add_candidate` accepts the merge in the original state group. - The checks that read an input's operator (a merge over a time-indexed state reads a `TimeRange` over a chain; a selected build) read the input group's `expression`. - `shareable_when_equal`: a `LIMIT` is shareable over a `Sort` with the same partitions (the input group's `expression`); a `Dedup` on every column (the input group's schema). - `column_signs` and, once added, `column_lineage` (below), per output column. @@ -550,7 +575,7 @@ pub struct GroupProperties { pub column_lineage: Vec, /// One entry per output column, in schema order (Value sign, below). pub column_signs: Vec, - pub aggregate_coverage: Option, + pub aggregate_coverage: Option>, pub shareable_when_equal: bool, pub scanned_sources: BTreeSet, } @@ -672,18 +697,23 @@ Some error contracts hold only for values of a known sign: a relative error of a ```rust // crates/types/src/stages/logical/column_properties.rs -/// GroupProperties::column_signs holds one per output column. +/// GroupProperties::column_signs holds one per output column. A sign is +/// the IEEE 754 sign bit, so `-0.0` is `NonPositive` and `+0.0` +/// `NonNegative`; with it the rules are exact: a product or quotient has +/// the XOR of its operands' sign bits, a sum of two values with one sign +/// bit keeps it, a negation flips it and `abs` clears it. pub enum ValueSign { - /// Every non-NaN value is ≥ 0. + /// Every non-NaN value has its sign bit clear (`+0.0`, positive, + /// `+inf`). NonNegative, - /// Every non-NaN value is ≤ 0. + /// Every non-NaN value has its sign bit set (`-0.0`, negative, `-inf`). NonPositive, /// Neither is known. Unknown, } ``` -A NaN does not break either sign: `NonNegative` means every non-NaN value is ≥ 0. +A NaN does not break either sign: `NonNegative` means every non-NaN value has its sign bit clear, `NonPositive` that it is set. So `x / -0` over a non-negative `x` is `NonPositive` (its result is `-inf`, `-0` or NaN). A source's signs are declared per column in the data workload's source facts (for example, `latency`'s value is non-negative). A PromQL counter's value is non-negative without a declaration. A column without a declaration is `Unknown`. `Memo::new(data)` stores the data workload's source facts at construction, and `find_or_add_group` reads their signs for a `Scan` (a histogram's or summary's series reads its family's). @@ -694,8 +724,8 @@ A source's signs are declared per column in the data workload's source facts (fo | `Join`; `SetOp`, `Concat`; label rewrite (`label_replace`); SQL window function | the left input's columns then the right's (a semi or anti join: the left's only); per column, the inputs' sign when they all agree, else `Unknown`; the input's, the written label `Unknown`; the input's, the window value `Unknown` | | `abs` | `NonNegative` | | negation | flipped: `NonNegative` ↔ `NonPositive` | -| numeric literal, numeric cast | the literal's sign; the operand's sign | -| arithmetic, scalar (`x / 1000`, SQL `value * 8`) or vector (`a / b`), one rule | `+`: both `NonNegative` → `NonNegative`, both `NonPositive` → `NonPositive`; `*` and `/`: same sign → `NonNegative`, opposite → `NonPositive`; `-`, `%`, `^`, `atan2`, or an operand of `Unknown` sign → `Unknown` | +| numeric literal, numeric cast | the literal's sign bit (`-0.0`: `NonPositive`; NaN: `Unknown`); the operand's sign | +| arithmetic, scalar (`x / 1000`, SQL `value * 8`) or vector (`a / b`), one rule | `+`: both `NonNegative` → `NonNegative`, both `NonPositive` → `NonPositive`; `*` and `/`: same sign → `NonNegative`, opposite → `NonPositive` (the result's sign bit is the XOR of the operands'); `-`, `%`, `^`, `atan2`, or an operand of `Unknown` sign → `Unknown` | | `vector(s)` | the sign of the expression `s` (`vector(1000)`: `NonNegative`) | | sum, avg, min, max, top-k, `first_over_time`, `last_over_time`; quantile with 0 ≤ q ≤ 1 | the input value's sign; a quantile with q outside [0, 1]: `Unknown` | | count, distinct count, and the other counts and spreads (`count_values`, `changes`, `resets`, `group`, `absent`, `absent_over_time`, `present_over_time`, `stddev`, `stdvar`, frequency L2 norm and entropy) | `NonNegative` | @@ -703,7 +733,7 @@ A source's signs are declared per column in the data workload's source facts (fo | vector comparison, `and`, `unless`; `or`; comparison with `bool` | the left side's value sign; the sides' sign when they agree; `NonNegative` (0 or 1) | | `SummaryAgg`, chain | the state records the sign of the values it is updated with (a constant weight: its sign; a tuple or entity identity: `Unknown`); keys keep theirs (`window_end` is `Unknown`) | | `SummaryMerge` | the inputs' sign when they all agree, else `Unknown`; over a chain's windows, the chain's build's | -| readouts | the state's recorded sign; a distinct count, frequency L2 norm or entropy, and an exact count's finalization: `NonNegative`; a quantile with q outside [0, 1]: `Unknown`; a point count or top-k of a Count Sketch or UnivMon (and their variants, `SummaryTypeCapability::estimates_may_flip_sign`): `Unknown`, since a median of signed counters can be negative over non-negative weights (Count-Min keeps the state's sign); a readout whose output does not keep the state's layout: `Unknown` | +| readouts | the state's recorded sign; a distinct count, frequency L2 norm or entropy, and an exact count's finalization: `NonNegative`; a quantile with q outside [0, 1]: `Unknown`; a point count or top-k of a Count Sketch or UnivMon (and their variants, `SummaryTypeCapability::point_counts_may_flip_sign`): `Unknown`, since a median of signed counters can be negative over non-negative weights (Count-Min keeps the state's sign); a readout whose output does not keep the state's layout: `Unknown` | | any other expression | `Unknown` | **Example.** `topk(5, sum by (host)(gauge))` over a gauge with no declared sign: the sum's sign is `Unknown`, so Stage 2 prunes the Count-Min heap builds (`CountMinWeightsNotProvenNonNegative`) and the query uses an exact top-k. Over `rate(requests_total[1m])`, the sum is `NonNegative` and the Count-Min builds stay. @@ -1364,15 +1394,19 @@ pub struct PhysicalASAPDAGNode { /// How long the node's time-indexed rows are retained (h_v); set on /// materialized nodes. pub retention_ms: Option, - // A materialized view's coverage is not a node field: the plan derives - // it per view (`MaterializedView::coverage`, a `ViewCoverage`: the - // content `definition_id` and the selection) from the node's memo - // group's `aggregate_coverage`. /// The physical candidate the node was built from: in-memory provenance /// for validation and tools, not part of the plan file; None for a plan /// built by a custom pass. #[serde(skip)] pub selected_physical_candidate: Option, + /// For a materialized view (lifetime `AcrossEvaluations`), what it + /// stores: its memo group's aggregate coverage, carried into the plan + /// as a `ViewCoverage` (the selection, and the plan-scope content id + /// of the definition, `ViewCoverage::definition_id`). Set when the + /// plan is built from the memo, where the node's group is known; `None` + /// on every other node. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub view_coverage: Option, } /// What Stage 3 reports about its own run, beside the plan. From 15d61cf63649e9e098c3d34e357f492a2f2f4eb7 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 12 Nov 2026 19:19:18 +0000 Subject: [PATCH 67/67] docs(concepts): AcrossEvaluations for a time-independent group without a grid; no AcrossEvaluations anywhere below an absolute @ T; Stage 2 function name The decision-tree row and the list of values not offered required a known grid for AcrossEvaluations, while the alignment table reads a time-independent result stored across evaluations at any time. The rule now follows the table. The @ T rule covers every group below the anchor (through any operator other than another @), not only the anchor's direct child, with an example. Stage 2's levels 3 and 4 are offered_physical_properties. --- .../concepts/memo-groups-properties-requirements.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 51dac1a58..e7e15b2f4 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -203,7 +203,7 @@ QT means QueryTime; IT means IngestionTime. Alignment, data coverage, and retent **Alignment.** There is no physical schedule property. When a result can be read follows from its lifetime and, for a chain, its window width: - **None** and **OneEvaluation**: the result is computed inside one evaluation context κ (Section 5) and read there. A OneEvaluation result belongs to one context: queries on the same evaluation grid, a range query's steps included, are evaluated together, and that evaluation's consumers share the result. When it runs is the context's schedule, and Stage 3 chooses it through $x_{\kappa,v}$, just as for ℓ = None. Sharing across differently scheduled queries through a gcd schedule is not offered: it is rarely beneficial and wastes runs. - **AcrossEvaluations** on a chain, `SummaryTumblingWindowChain[w](X)`: the input is cut into tumbling windows (k·w, (k+1)·w], with the grid anchored at phase 0 (the epoch). Windows are left-open and right-closed like a PromQL range, which `m[r]` evaluated at t reads as (t − r, t]. Each window is built when it ends and can be read from then on. A `TimeRange` over the chain selects the windows that lie fully inside (t − range, t], so a consumer can read a range whose ends lie on window boundaries. With w = 10 s, a `[1m]` query evaluated every 30 s at phase 0 merges 6 windows; one evaluated at phase 15 s cannot, because its range ends inside a window; with that query in the workload, the gcd closure also offers w = 5 s, which both can read. The `TimeRange` and `TimeShift` groups over a chain are computed within one evaluation. -- **AcrossEvaluations** on a candidate of any other group evaluated on a known grid: a time-independent result (one that does not depend on the evaluation time t, such as a result under `@ T`) is stored as is and can be read at any time; a time-dependent result is stored per evaluation time, as a time-indexed result series computed at every time of its grid. A consumer reads the value stored for its own evaluation time, or, through a `TimeShift` above it, for a shifted time; so the consumer's evaluation times must lie on the producer's grid. Its grid is the greatest common divisor of the grids of the queries that reach it, as for chain widths: a result read by queries every 60 s and every 120 s is computed every 60 s, and both read it. +- **AcrossEvaluations** on a candidate of any other group that is time-independent or evaluated on a known grid: a time-independent result (one that does not depend on the evaluation time t, such as a result under `@ T`) is stored as is and can be read at any time, so it needs no grid; a time-dependent result is stored per evaluation time, as a time-indexed result series computed at every time of its grid. A consumer reads the value stored for its own evaluation time, or, through a `TimeShift` above it, for a shifted time; so the consumer's evaluation times must lie on the producer's grid. Its grid is the greatest common divisor of the grids of the queries that reach it, as for chain widths: a result read by queries every 60 s and every 120 s is computed every 60 s, and both read it. A grid phase is not needed: a phase is absorbed by the width, because width candidates come from the gcd closure of the queries' grids, ranges and offsets (Section 3). For example, phases 0 and 5 s with w = 10 s give w = 5 s. Alignment alone does not ensure sufficient retained history; that is $h_v$ (Section 5). @@ -263,7 +263,7 @@ flowchart TD P1 --> L1["ℓ = None"] P1 --> L2["ℓ = OneEvaluation"] - P1 --> L3["ℓ = AcrossEvaluations
(chains and grid-evaluated groups; prune:
deployment retains no query-time results)"] + P1 --> L3["ℓ = AcrossEvaluations
(chains, grid-evaluated and time-independent groups; prune:
deployment retains no query-time results)"] P2 --> L4["ℓ = None"] P2 --> L5["ℓ = AcrossEvaluations"] P2 -. invalid .-> L6["ℓ = OneEvaluation"] @@ -274,9 +274,9 @@ flowchart TD | 1 | expression $e_v$, including a merge over a chain of width $w$ | Pass 1 and Pass 2 rules | each way to compute the group; chain widths from the gcd closure of the reaching queries' grids, ranges and offsets | no reaching query has a known grid (no chain is offered) | | 2 | $\boldsymbol\theta_v$ | Pass 1 | the algorithm's predefined list | dead-candidate elimination (Section 3): no readout requirement it could serve admits it | | 3 | $\tau_v$ | Stage 2 | QueryTime; IngestionTime only for a candidate that is time-independent or an input of one, and for a counter state only when the source's out-of-order bound is known (Stage 2) | the source has no ingestion pipeline; QueryTime over raw rows the deployment does not retain; a Count-Min build whose weights are not proven non-negative | -| 4 | $\ell_v$ | Stage 2 | None, OneEvaluation, AcrossEvaluations; AcrossEvaluations for chains and for candidates of groups evaluated on a known grid, and Disk only where the deployment offers it | invalid with $\tau_v$; the deployment retains no query-time results | +| 4 | $\ell_v$ | Stage 2 | None, OneEvaluation, AcrossEvaluations; AcrossEvaluations for chains and for candidates of groups that are time-independent or evaluated on a known grid, and Disk only where the deployment offers it | invalid with $\tau_v$; the deployment retains no query-time results | -The planner does not check whether an executor implements an operator or summary type: every candidate the rules produce is offered, and an executor that lacks one is extended. A value outside a dimension's declared value set (IngestionTime on a time-dependent candidate, AcrossEvaluations on a candidate of a group with no known grid, Disk where the deployment has no disk storage) is not offered, so it is not recorded as pruned. Nor is (IngestionTime, None) for a summary build (`SummaryAgg`, `SummaryTumblingWindowChain`, `MaintainPopulation`): a build fed at ingestion time keeps its state between batches, so it is materialized; the plan's validation rejects an unmaterialized one. A candidate one of whose consumers reads it under `@ T` with no known horizon end gets no materialized alternative (its retention need, the horizon end − T + range, is unbounded); its query-time alternatives stay. +The planner does not check whether an executor implements an operator or summary type: every candidate the rules produce is offered, and an executor that lacks one is extended. A value outside a dimension's declared value set (IngestionTime on a time-dependent candidate, AcrossEvaluations on a candidate of a time-dependent group with no known grid, Disk where the deployment has no disk storage) is not offered, so it is not recorded as pruned. Nor is (IngestionTime, None) for a summary build (`SummaryAgg`, `SummaryTumblingWindowChain`, `MaintainPopulation`): a build fed at ingestion time keeps its state between batches, so it is materialized; the plan's validation rejects an unmaterialized one. A candidate read under `@ T` with no known horizon end, directly or through any operators below the anchor other than another `@`, gets no alternative materialized across evaluations (its retention need, the horizon end − T + range, is unbounded): in `max_over_time(rate(c[1m])[5m:1m] @ 90)` neither the subquery nor the `rate`, its range or the scan is. A group also read on a relative path counts as read under the anchor. Its other alternatives stay. The tree does not decide which producer each consumer input reads ($y$), in which context a candidate with ℓ = None or OneEvaluation runs (Section 5), or how much an AcrossEvaluations candidate retains ($h_v$). Those are Stage 3 choices. @@ -1155,7 +1155,7 @@ A candidate that carries a θ without building (a merge, or a `Filter`, `TimeRan ### Stage 2: physical alternatives -`enumerate_candidate_configurations` in `crates/stage2-physical-optimization` expands each logical candidate that survived Pass 2 using the dimensions and checks in Section 2. It offers τ = IngestionTime only to a candidate that is time-independent (its output does not depend on the evaluation time t, as a chain's does not) or that is an input of one. It offers ℓ = AcrossEvaluations to chains and to the candidates of any group evaluated on a known grid: a time-independent result is materialized as is, a time-dependent one per evaluation time, as a time-indexed result series. Values not offered are not recorded. It creates no new logical candidates. Each logical edge is expanded between the relevant physical alternatives. +`offered_physical_properties` in `crates/stage2-physical-optimization` expands each logical candidate that survived Pass 2 using the dimensions and checks in Section 2. It offers τ = IngestionTime only to a candidate that is time-independent (its output does not depend on the evaluation time t, as a chain's does not) or that is an input of one. It offers ℓ = AcrossEvaluations to chains and to the candidates of any group that is time-independent or evaluated on a known grid, unless the group is read under an absolute `@ T`: a time-independent result is materialized as is, a time-dependent one per evaluation time, as a time-indexed result series. Values not offered are not recorded. It creates no new logical candidates. Each logical edge is expanded between the relevant physical alternatives. Stage 2 reads the logical candidates, the data workload (per-source arrival, below) and the deployment's capabilities: