Skip to content

DAG viewer / stage_pipeline: print memo candidates with properties, every producer–consumer edge with its feasibility and rejection reason, and every stage’s pruned candidates #1205

Open

Description

@zzylol

Priority: lowest (decided 2026-11-04). Done after the main path of #1207 (steps 10 and 11 there).

Example

Same queries as #1201: KLL sizes come from a predefined list, and each query's accuracy is checked at its root (row (K), #1204). Suppose the selected plan builds two KLL states instead of sharing one, and we want to know why.

  • Question. Which producers could B's inputs connect to, and why was each other one rejected? For example: "ingestion-time SummaryAgg, None: rejected, $\alpha^{\mathrm{phys}} = 0$, a query-time reader needs AcrossEvaluations". And per query: which approximate results it depends on, each with its $\varepsilon$, and their sum against the target, e.g. "B: 0.005 + 0.005 = 0.010 ≤ 0.01". A shared k=200 build would give B 0.008 + 0.008 > 0.01. These $\varepsilon$ values are illustrative; the real ones come from the $\theta$ list.
  • Today:
    • the viewer's memo panel (tools/dag-viewer/memo.js) shows each group's candidates (origin, accuracy_budget_parts) and Pass 1's and Stage 2's pruned candidates with a reason;
    • the viewer document has Stage 2's candidates with timing and materialization, but not their producers per input;
    • stage_pipeline --text (feat(devtools): stage_pipeline --text prints every planner stage's output as text #1159) prints the producer set per input, but only the admissible producers;
    • rejected producers are dropped without a reason (compatible_timing, configurations.rs:394);
    • Stage 3's pruned candidates (PruneReason::DominatedBy, UsageNotCostable) are in neither output.

This issue prints, for each memo group, its candidates with their properties, and for each consumer input every producer in the group with $\alpha^{\log}$, $\alpha^{\mathrm{phys}}$ and the reason when one is 0. It also prints every stage's pruned candidates with reasons.

Representation, rules, enumeration

  • (a) Representation. No change. The viewer and stage_pipeline print it:
    • groups;
    • candidates with $\theta$ and (timing, lifetime, schedule), and the rule that added each one (origin);
    • edges with $\alpha$ and the rejection reason.
  • (b) Rules. No change. Pruned candidates are shown with the rule or check that pruned them.
  • (c) Enumeration. No change. The chosen $x$ and $y$ are shown over the memo, so the reader sees which DAG was drawn from it.

Formulation

What to show, per the formulation:

  • per group $G$: each candidate $v = (e_v, p_v)$ with $e_v$, $\theta_v$, $\tau_v$, $\ell_v$;
  • per consumer input $(c, i)$: $G_{c,i}$, $r_{c,i} = (r^{\log}, r^{\mathrm{phys}})$, and every $u \in P_{c,i} = G_{c,i} \cap \mathcal C$ with $\alpha^{\log}{u,c,i}$, $\alpha^{\mathrm{phys}}{u,c,i}$;
  • after Stage 3: the selected $x$, $y$, $z$, and per query the terms of row (K): each $u$ it depends on with $\lambda_{q,u}\varepsilon_u$ and $\delta_u$, their sums and the targets;
  • per stage: the candidates removed before $\mathcal C$, with reasons.

Data structures and interfaces per stage

Reasons (crates/types/src/stages/):

/// Why α_{u,c,i} = 0.
pub enum EdgeRejection {
    /// α^log = 0: a non-accuracy logical need, e.g. a tumbling-window width that does not divide the reader's range.
    LogicalRequirement { required: LogicalRequirement, offered: LogicalProperties },
    /// α^phys = 0: the producer's timing or lifetime cannot serve this read.
    PhysicalRequirement { required: PhysicalRequirement, offered: PhysicalProperties },
}
impl PhysicalEdge { pub fn rejection(&self) -> Option<EdgeRejection>; }

Stage 1 (Pass 1, Pass 2):

Stage 2:

Stage 3:

stage_pipeline (crates/devtools/src/bin/stage_pipeline.rs): writes the fields above into the viewer document. --text (#1159's printer) prints them as rows under each candidate's inputs.

DAG viewer (tools/dag-viewer/memo.js):

  • Group panel: each candidate shows $\theta$, $\tau$, $\ell$.
  • Clicking a candidate: shows its inputs. Each input lists every producer in the input group: ✓ admissible, or ✗ with the rejection. A selected edge is marked.
  • Pruned section: one list per stage, Pass 1, Stage 2 and Stage 3, each with its reason.

Current vs proposed

What Today Proposed
candidate $e_v$ memo panel: label, origin, children same
$\theta_v$ inside the label (sketch parameters), plus accuracy_budget_parts its own field and row
$\tau_v$, $\ell_v$ Stage 2 candidates: timing, materialization properties: { timing, lifetime } (#1200)
$r_{c,i}$ not exported per input
$P_{c,i}$ --text prints the admissible set per input (#1159); the viewer document has none every producer, admissible or not, in both outputs
$\alpha$ and reason not recorded admissible and EdgeRejection per edge
selected $y$ SelectedEdges of the selected DAG same, also marked on the candidate's inputs
Pass 1 pruned stage1_logical_asap_optimization.pruned_candidates, with reason same
Stage 2 pruned stage2_physical_asap_optimization.pruned_candidates, with reason (stage2_pruned_candidates_json) same, with properties instead of materialization
Stage 3 pruned not exported (DominatedBy, UsageNotCostable) exported with reason and the dominating candidate

Scope and acceptance

In scope:

  • EdgeRejection;
  • the viewer document fields;
  • the memo panel's input/producer view;
  • Stage 3's pruned list.

Out of scope:

Acceptance:

  • Deletes what it replaces in the same PR, with no dead code, fallback, flag, shim or reuse of an old abstraction; the PR body lists what it deleted (Tracking: implement the optimization formulation (memo candidates, properties, edges y_{u,c,i}, MILP) as named code per stage #1207, Policy).
  • Viewer. For the example, B's Finalize inputs list every KLL size as ✓, a query-time None producer read across evaluations as ✗ with its $\alpha^{\mathrm{phys}}$ reason, and B's accuracy panel shows $0.005 + 0.005 \le 0.01$.
  • Stage 3 pruned. A dominated candidate appears under "pruned by Stage 3" with the candidate that dominates it.
  • Tests. viewer_contract and memo.test.mjs cover the new fields.

Related

Activity

  1. added
    enhancementNew feature or request
    dag-viewerDAG export and visualization tooling
    dev-toolsDeveloper tools, diagnostics, and demos
    on Nov 4, 2026
  2. zzylol commented on Nov 16, 2026

    @zzylol
    ContributorAuthor

    Not in the #1207 unit; leaving this open with its status on main 1760a69.

    Done (outside this issue's own PR, mostly by #1239 and #1277):

    Remains: the viewer-specific acceptance (B's Finalize inputs with check marks and cross reasons in the viewer, accuracy panel 0.005 + 0.005 <= 0.01, and a Stage 3 dominated candidate listed with its dominator) is not verified or tracked; viewer_contract and memo.test.mjs coverage of the new fields should be checked when this issue is picked up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dag-viewerDAG export and visualization toolingdev-toolsDeveloper tools, diagnostics, and demosenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions