Visualize execution pipelines from EXPLAIN (FORMAT INTERNAL, PIPELINES) - #140
Closed
PerFuchs wants to merge 34 commits into
Closed
Visualize execution pipelines from EXPLAIN (FORMAT INTERNAL, PIPELINES)#140PerFuchs wants to merge 34 commits into
PerFuchs wants to merge 34 commits into
Conversation
Render Hyper's merged execution units ("pipelines") on top of the operator
tree, so users can see how operators are fused into pipelines and where
pipeline breakers split them.
- Parse the `{tree, pipelines}` envelope emitted by
`EXPLAIN (FORMAT JSON, PIPELINES, ...)`, and accept kebab-case `operator-id`
/ `debug-name` in addition to the legacy camelCase spellings.
- Assign each pipeline a categorical color (left-to-right); operators shared by
several pipelines take the color of the right-most pipeline, which makes
UNION ALL and fork/share plans read naturally.
- Node coloring: tint the operator icon, a segmented under-label bar showing
every pipeline the operator belongs to, and a pipeline-colored border when
expanded.
- Edge coloring: color each edge by the pipeline(s) shared by its endpoints;
edges that carry several pipelines (e.g. above a UNION ALL target) are stroked
with a contiguous color-band gradient and a segmented start-bar. Edges with no
shared pipeline stay neutral, visualizing pipeline breakers.
- Edge thickness encodes rows flowing (statistics.output-rows / estimated-rows),
with an actual/estimated row label; new-format `statistics.*` rows are now
understood in addition to the legacy `cardinality`/`analyze.tuple-count`.
- Under ANALYZE, per-pipeline cpu-cycles drive a label-background heat overlay
(reusing the existing hotspot channel), and per-pipeline cpu-cycles/wall-time/
cost% are surfaced in the expanded node body.
- Add example plans (topology-only and ANALYZE) covering fork/share, UNION ALL,
a group-by, and a multi-join.
Co-authored-by: Cursor <cursoragent@cursor.com>
The legend was removed earlier, but its supporting data was still being built: - `assignPipelineColors` returned a `PipelineInfo[]` legend and built an unused `infoById` map; it now just assigns colors and returns void. - `TreeDescription.pipelines` / the `PipelineInfo` type are gone (nothing renders them). - `parsePipelineStatistics` and `RawPipeline.statistics` are gone; the displayed ANALYZE numbers come straight from the raw `cpuCycles`/`wallTime`. Co-authored-by: Cursor <cursoragent@cursor.com>
tree-description.ts now only talks about colors, not pipelines. TreeNode
exposes three generic color arrays:
- barsAbove: colors of a (multi-color) bar drawn above the node
- barsBelow: colors of a (multi-color) bar drawn below the node
- edgeColors: colors of the incoming edge (multi -> gradient)
replacing the pipeline-specific pipelineColor/pipelineColors/pipelineIds/
edgeColor fields.
All pipeline reasoning stays in hyper.ts, which now fills barsAbove/barsBelow
with the operator's pipeline colors and keeps pipeline membership in a local
map (no longer stored on the node).
UI: - QueryNode draws a color bar both above and below each node; the icon tint
and expanded-border coloring are dropped (identity now lives on the bars).
- The edge start-bar is removed; the incoming edge is just the (possibly
multi-color gradient) stroke.
Co-authored-by: Cursor <cursoragent@cursor.com>
Data flows leaves->root, so a node receives from its children (below) and
feeds its parent (above). hyper.ts now detects this from the tree:
- barsAbove (outgoing) = pipelines shared with the parent (all pipelines at
the root); this also colors the incoming edge.
- barsBelow (incoming) = pipelines shared with the operator children (all
pipelines at a leaf source).
A pipeline that starts or ends at a node (a pipeline breaker, or a source/sink)
is present on only one side, so breakers now read as a missing bar on each side
of the neutral edge.
Co-authored-by: Cursor <cursoragent@cursor.com>
A UNION ALL nested underneath an EXCEPT, producing a wider tree with curved edges and a multi-color edge (the unionAll->except edge carries both of the union's arm pipelines). Co-authored-by: Cursor <cursoragent@cursor.com>
…alizer)
The pipeline feature is now purely about coloring. Removed:
- per-pipeline cpu-cycles/wall-time parsing and the label-background "heat"
overlay + the pipeline stats added to the node body;
- reading rows from the new statistics.* keys for edge width/labels.
The visualizer's existing behavior is untouched: colorRelativeExecutionTime
still tints hot operators from per-operator analyze.cpu-cycles, and edge width
still comes from cardinality/analyze.tuple-count.
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
- Reorder the Tableau 20 palette so all ten saturated base hues come first and the lighter companions last, so adjacent pipelines never get near-identical shades (light-blue no longer follows blue) and typical plans (<=10 pipelines) stay fully saturated. - Order each node's below-bar (and its outgoing bar/edge) by the left-to-right position of the child that carries each pipeline, so a pipeline entering from the left child shows as the left segment. Previously the segments were ordered by global horizontal rank, which could put a colour on the wrong side. Co-authored-by: Cursor <cursoragent@cursor.com>
PipelineEdge already sets the stroke from data.colors (solid or gradient), so the style.stroke fallback was always overridden and never rendered. Co-authored-by: Cursor <cursoragent@cursor.com>
- tree-description: tighten the barsAbove/barsBelow doc comments ("in array
order"), align the edgeColors comment with the terse "incoming edge" style
used by the neighboring fields, and drop a stray blank line.
- QueryGraph: stop deriving the minimap color from barsBelow in the UI; the
minimap uses the node's existing color fields (assigned in hyper.ts).
- QueryNode.css: align the color-bar comment wording with the doc.
Co-authored-by: Cursor <cursoragent@cursor.com>
hyper.ts now assigns each operator's iconColor to its dominant (right-most) pipeline color (unless already set, e.g. the red error highlight). This colors the icon in the graph and, via the existing nodeColor/iconColor fallback, the minimap -- without the UI reaching into the bar arrays. Co-authored-by: Cursor <cursoragent@cursor.com>
convertHyperPlan now takes an optional `pipelines` argument and, when present,
projects them onto the tree by reusing the `operatorsById` map it already builds
during conversion. This removes convertHyperPlanWithPipelines, which duplicated
that map by re-walking the produced tree and reading node properties. loadHyperPlan
dispatches the `{tree, pipelines}` envelope straight to convertHyperPlan(tree, pipelines).
Co-authored-by: Cursor <cursoragent@cursor.com>
Replace the centroid "horizontal rank" with a plain pre-order DFS index (computeTreeOrder), which is easier to understand and depends only on the tree structure. A pipeline's color now comes directly from its position -- the DFS index of its deepest operator, modulo the palette -- rather than from its ordinal among the pipelines. This keeps colors stable across executions even when the engine renumbers pipeline ids, and only recolors the pipelines whose tree position actually changed. The deepest operator is used so that sibling pipelines sharing the "pipeline above" operators (e.g. a UNION ALL target) still get distinct colors. Also fold the palette assignment into the resolve/walk pass: each resolved pipeline carries its color, and the tree walk that fills the bars/edge/icon uses it directly -- dropping the separate colorOrder sort + colorById map. Co-authored-by: Cursor <cursoragent@cursor.com>
Remove computeTreeOrder (the per-node index map). The coloring pass is now a plain pre-order DFS that stamps each pipeline with the position of its deepest (last-visited) operator via a running counter -- no node->order map needed. The color is palette[deepest % len], as before. Coloring still takes a separate DFS pass from the bar assignment, on purpose: a single forward walk cannot finalize colors, because a pipeline can span two sibling subtrees (a fork/share source feeding both join inputs) and the left sibling's bar needs a color determined by an operator under the right sibling, which is visited later. Verified: the fork-source pipeline keeps one consistent color under both arms. Co-authored-by: Cursor <cursoragent@cursor.com>
Assign each pipeline its color the first time it is encountered during the pre-order walk (its top-most operator), taking the next palette color. Because a pipeline touching a node is colored no later than that node -- earlier at an ancestor, or here on first appearance -- all colors a node needs are ready by the time its bars are filled, so coloring and bar assignment happen in one DFS. This is stable across executions (colors follow the traversal, not pipeline ids), keeps a fork/share source one consistent color (it is colored once, at first sight, and reused under both arms), and yields sequential palette colors from the start. Bar segments are still ordered left-to-right by child position; the icon takes the node's last-appearing pipeline color. Tested on all four example plans (group-by, fork/share, UNION ALL, joins): distinct colors == pipeline counts, fork source consistent across arms. Co-authored-by: Cursor <cursoragent@cursor.com>
The first-appearance color is already stored on the pipeline, so `rank` was redundant: use color === "" as the "not yet seen" sentinel, rely on the stable sort for segment ties, and take the icon color from the node's right-most pipeline (last entry of the child-ordered list) rather than the last-appearing one -- which also fixes the icon to honor "right-most wins".
Replace the fork/share and UNION ALL example plans with the exact FORMAT JSON
output from hyper-db-emu/hyper-db#13438 (the current PR), and add a magic /
EXISTS semi-join example from its fixture. The envelope is unchanged
(`{tree, pipelines}` with kebab-case operator-id and pipelines carrying
id/operators/dependencies), so the parser already handles it; verified by
loading the PR's own fixture output.
- parsePipelines: ignore a non-array `pipelines` (return empty) instead of throwing, matching the loader's fall-back-and-display convention. - Drop the operator-id `.filter`: a pipeline's operator ids always resolve to a tree node, so resolve them directly. - Rename `parentIds` -> `parentPipelineIds` for clarity. - Remove a couple of redundant comments (tree-layout edge stroke, pipeline projection call).
…helpers - Rename PipelineEdge -> ColoredEdge and edge type "pipeline" -> "colored"; the UI layer is database-independent, "pipeline" is a Hyper detail. - Pass props straight to getBezierPath instead of repackaging them. - Group barsAbove/barsBelow with the node visuals and edgeColors with the other incoming-edge properties in TreeNode. - Alias EXPLAIN (FORMAT JSON) operator names (output, scan, groupBy, filter, explicitScan, tableConstruction) to the correct icons. - Move parsePipelines/assignPipelineColors above their use in convertHyperPlan.
- Revert the operator-name icon aliases: the real fix belongs in Hyper's FORMAT JSON (emit kebab-case operator names), after which the visualizer's icon config gets updated. Track as a follow-up. - Reword the assignPipelineColors comment so it is explicit that the root "output" always gets a full bar above and leaves always get a full bar below (endpoints are full; only in-between breakers are one-sided).
- Add the source SQL for the three pipeline examples (fork/share, UNION ALL, magic/EXISTS) verbatim from Hyper's pipeline_graph.test (hyper-db#13438); the committed JSON is exactly that test's expected FORMAT JSON output. - Teach plan-dumper a PIPELINES mode: EXPLAIN (FORMAT JSON, PIPELINES, EXPAND_VIEWS true, EXPRESSIONS SQL). Pipeline query files are self-contained scripts (SET/CREATE + final query) run last so their session globals do not leak into the other examples, and are Hyper-only. - Drop the ad-hoc examples that had no reproducible SQL source: the nested set-op case and the non-deterministic ANALYZE pipeline plans. - Document the #13438 build requirement in DEVELOPMENT.md.
…query - Add pipelines-materialized: the same shared-CTE query as pipelines-forkshare but with share forking disabled, so the share is materialized. The share becomes its own pipeline and the consumers read from it across a pipeline break, instead of the shared scan recurring across pipelines. - Each pipeline query file now resets the session globals it changes (back to share_forking=true, view_inlining_selectivity_threshold=0.5) and drops its temp tables, so a dump run stays isolated regardless of file order. - dump-plans.py EXPLAINs the query statement within each self-contained script (running the SET/CREATE setup and SET/DROP teardown around it). - Drop version-specific references now that this can land alongside the plan format it targets. Regenerated all pipeline examples through a hyperd built from the pipeline format branch: forkshare/unionall/magic come back byte-identical to the committed fixtures.
…_plan call Pipeline files now contribute their query to the same `plan = get_plan(...)` line as every other mode; the setup runs before it and the teardown after, instead of calling get_plan inline in the statement loop.
Detect the pipeline modes from the filename suffix (`-pipelines.sql`, `-analyze-pipelines.sql`) the same way `-steps.sql` and `-analyze.sql` are detected, and rename the query/example files accordingly.
…bars
- Switch plan-dumper to EXPLAIN (FORMAT INTERNAL, ...). It emits the same
{tree, pipelines} JSON but with the internal operator names the visualizer's
icon config already knows, so operator icons (and the crosslink edges, e.g.
explicit-scan -> share) render correctly again.
- Rewrite the pipeline examples as plain single-statement queries over the
shared t1/t2 tables, with no session settings: a selective filter keeps the
shared scan from being inlined, and fork-vs-materialize falls out of the
query shape -- independent consumers (MIN/MAX) fork the share, a self-join
(build+probe) materializes it. This removes the need for the multi-statement
setup/teardown mechanism in dump-plans.py.
- Drop the per-node endpoint bars: the root output no longer gets a bar above
and leaves no bar below; a bar is only drawn where a pipeline crosses an edge.
An explicit scan that reads a shared operator references it through a crosslink (the shared subtree is emitted inline only once, under the first reader), so the other readers had no tree child and thus no below-bar. Treat a node's crosslink target as an extra child when computing the incoming (below) bar. A reader only gets the bar when it actually shares the target's pipeline, so forked shares now show the bar under every reader while a materialized share (separate pipeline) still shows none.
When a node is expanded the below-bar sat between the icon and the properties. Move it out of the node head so it follows the property list (and still sits under the icon when collapsed). Reuse the existing magicunnesting query as the PIPELINES example instead of a second EXISTS variant.
Add margin below the bar so it does not sit on the +/- handle, and padding after the last expanded property so the bar is not flush against the properties.
Accept the kebab-case operator and expression tags emitted after Hyper PR #13883 while preserving legacy separatorless tags. Support the current statistics and row-count fields alongside their older spellings. Regenerate the Hyper example plans with hyperd built from merged commit 0457d99e65.
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Visualizes the merged execution pipelines emitted by Hyper's
EXPLAIN (FORMAT INTERNAL, PIPELINES)envelope.statistics/ row-count fields while preserving legacy plan compatibility.hyperdfrom merged Hyper commit0457d99e65.Verification:
./node_modules/.bin/eslint .../node_modules/.bin/tsc --buildfromquery-graphs/yarn workspace query-graphs-app build