feat(visualization): export the coordinate-system and domain plots as Python - #8346
Draft
kz930 wants to merge 14 commits into
Draft
feat(visualization): export the coordinate-system and domain plots as Python#8346kz930 wants to merge 14 commits into
kz930 wants to merge 14 commits into
Conversation
…ython script A workflow can be read in the editor but not taken away: there is no form of it that runs anywhere else, so a user who wants to keep a pipeline, hand it to someone without Texera, or step through it in a notebook has nothing to take. This adds the seam for one and the first few operators through it. An operator says how it reads outside the engine by implementing `StandaloneCodeGenerator`, returning a block of pandas that names its inputs and outputs as `in1df` / `out1df`. The translator walks the plan in topological order, gives every port a variable, substitutes those placeholders, and prints the leaves; `inAlldf` stands for the whole list of upstreams, which is what a variadic port like Union's needs, since any fixed count the code stated would be wrong for some workflow. An operator with no generator yet leaves a commented TODO rather than a line that looks like it works. `GET /workflow-to-python` on the compiling service returns the script for a plan it is given. Five operators implement it here — Distinct, Limit, Projection, Filter and Union — chosen to cover the shapes the translator has to handle: a single input, a config-driven one, one that renames columns, one that builds a predicate, and the variadic port. The rest of the operator set follows in later changes. `pyStringLiteral` renders a value as a Python literal with the escaping that keeps a quote or a newline in a column name from ending the literal early. The generators cannot use the runtime's decode expression, which needs an operator instance to decode through. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…re the files The standalone export claims that a generated script does what the operator does. Nothing checks it. This adds the two runners that make the claim checkable, and the file format they meet in. `OpExecHarness` runs a LogicalOp the way the engine does — compiling it to a physical plan and driving the executor — but outside a workflow, against JSONL files rather than a live upstream. `PyOpExecHarness` does the same for a Python operator, through the worker the engine uses. `StandaloneRunner` takes the other path: it asks the operator for its standalone code, wraps it in a script that binds `in1df` from the same files, and runs it. `TupleIO` is what the two meet in. A JSONL row carries values and no types, so the schema travels beside it in a sidecar; without one, a column written as INTEGER reads back as a number and the two paths disagree over a difference neither operator made. Both runners produce files, not assertions, so what to make of a difference is left to a later change. What is here is enough to run one operator both ways and see that the answers match, which is what the spec does with Distinct. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…he operator it came from The standalone export claims a generated script does what the operator does. This is what checks it, for every operator, on every configuration the operator offers. An operator is run twice. `OpExecHarness` drives it the way the engine does, compiled to a physical plan but outside a workflow, reading JSONL files rather than a live upstream; `PyOpExecHarness` does the same for a Python operator through the worker the engine uses. `StandaloneRunner` takes the other path, wrapping the operator's standalone code in a script that binds the same files. Both write files, and `Comparator` reads them back: order-insensitive by default, since the engine interleaves across workers and only the sort family promises an order. A visualization is compared as a figure rather than as a frame. What to run an operator ON is decided rather than written by hand for each. `ConfigGenerator` reads the operator's own schema — its enums, defaults, declared ranges and column pickers — and produces a base configuration plus one variant per branch the operator offers, so a switch nobody thought to try is still tried. `CanonicalFixture` is the table they run against, one column per shape an operator might ask for. `CuratedHandlers` is the escape hatch for an operator whose input cannot be derived, and `TransformVerificationRunner` decides which of the three tiers each operator takes and reports what it could not run and why. `LogicalOp.orderSensitive` and `@SampleColumn` are the two things the operators had to say for this to read them: whether row order is part of the contract, and which column a field should be pointed at when the first unused one would be a poor choice. Most of the operator set does not implement the generator yet — it arrives a family at a time — and the runner reports each of those rather than passing over it. The tier assertions for a family land with the change that gives that family its generator. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…f the one without The split this change relies on was declared but never wired. The specs carry `@IntegrationTest` and `build.sbt` reads `WCS_TEST_FILTER` to act on it, but nothing set that variable, so the filter was a no-op and the specs that fork Python ran in the job that provisions none — failing on `No module named 'pandas'` rather than on anything they were testing. The platform job now sets `skip-integration`, which excludes them. The platform-integration job sets `integration-only` and provisions what they need: Python 3.12, amber's requirements, protoc, and the generated proto bindings, which are gitignored and so have to be regenerated before a forked driver can import pyamber. Every step is guarded on the service, so the other entries in that matrix are untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The comment claimed a knob carrying a `defaultValue` is left alone. It is not: a knob holding what a fresh instance holds counts as untouched, because a config where the user typed the default is indistinguishable from one where they typed nothing. Moving it off that value is the only way the branch for a knob that IS set gets exercised. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…-coordinate-systems
… Python Eighteen visualizations that draw in a space of their own rather than over a table's columns: the polar family — polar, radar in its two forms, wind rose — the ternary pair, the three-dimensional scatter, the contour and carpet plots, the quiver field, the map, the gantt and time series over a clock, the gauge and bullet indicators, the dumbbell and filled-area comparisons, and the volcano plot. Each is compared as a figure rather than as a frame, after both paths have drawn: a plot's meaning is in the numbers behind it, and comparing the rendered picture would fail over a layout detail that carries none. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
github-actions
Bot
added
feature
dependencies
Contributor
Automated Reviewer SuggestionsBased on the
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8346 +/- ##
============================================
- Coverage 94.11% 94.09% -0.02%
- Complexity 4811 4840 +29
============================================
Files 1197 1203 +6
Lines 48813 48949 +136
Branches 5906 5930 +24
============================================
+ Hits 45939 46061 +122
+ Misses 1420 1419 -1
- Partials 1454 1469 +15
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
📊 Arrow Flight E2E bench(no arrow-flight-e2e.csv in artifact) |
The example of an operator with no generator was an operator another batch gives one to, so the assertion held only until that batch landed. A Python UDF holds whatever order these land in: its body is written by whoever drops the operator, so no generator can emit one. The word cloud assertion goes for the same reason, and the prediction op alone already covers what that test is for. The visualizations this batch adds are asserted where they route, which is the one thing the per-operator runs do not check, and a choropleth covers a column read two ways: a type rule narrows what can be shaded, an annotation names what holds the country code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Sep 2, 2026
Everything here that is not this batch's own operators belongs to apache#8327 and was carried only so the branch could compile and run its own tests before that one landed. Reviewing it twice costs more than the red build does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
github-actions
Bot
removed
dependencies
The rows these operators add to the runner and its config tests sit in files apache#8327 introduces, so they land once that does rather than as whole new files here. Each operator's own spec stays. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
carloea2
approved these changes
Sep 4, 2026
carloea2
left a comment
carloea2
left a comment
Contributor
There was a problem hiding this comment.
The coordinate and domain chart exports look good.
kz930
marked this pull request as draft
September 4, 2026 17:12
The translator no longer emits plotly into every script; it asks the operators in the plan what they need beyond pandas. All eighteen plots here draw with plotly, so all eighteen mix in `PlotlyStandaloneCode`, which the hierarchy and graph charts introduce. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
carloea2
approved these changes
Sep 4, 2026
carloea2
left a comment
carloea2
left a comment
Contributor
There was a problem hiding this comment.
I checked the new import declaration. It looks good.
Quiver Plot and Ternary Contour import `plotly.figure_factory` inside their own generated code, and use none of the three modules the mixin declares. Mixing it in would put an import in the script that the script never reads. The mixin states what a script has to import at module scope. An operator that imports what it needs inside its own code has nothing to add there. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
carloea2
approved these changes
Sep 4, 2026
carloea2
left a comment
carloea2
left a comment
Contributor
There was a problem hiding this comment.
The import cleanup looks good.
…d over Choropleth Map and Contour Plot rebound the name they were handed. Both now bind their own frame. The translator names a variable per output PORT, not per reader, so two charts drawn from one upstream are handed the same name. A chart that drops its own nulls through that name leaves the chart beside it drawing a table it never received, and a single-branch workflow never notices. Both spellings did it. `inplace=True` mutates the frame outright, and `in1df = in1df.dropna(...)` reads like a local rebinding but renames the shared variable, since the operator bodies are concatenated at module scope. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
carloea2
approved these changes
Sep 4, 2026
carloea2
left a comment
carloea2
left a comment
Contributor
There was a problem hiding this comment.
The chart code now keeps the input frame unchanged. Looks good.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
Eighteen more visualizations implement
StandaloneCodeGenerator— the ones that draw in a space of their own rather than over a table's columns: the polar family (polar, radar in its two forms, wind rose), the ternary pair, the three-dimensional scatter, the contour and carpet plots, the quiver field, the choropleth map, the gantt chart and time series over a clock, the gauge and bullet indicators, the dumbbell and filled-area comparisons, and the volcano plot.With these and #8344, every visualization Texera offers can be exported.
Each is compared as a figure rather than as a frame, after both paths have drawn. A plot's meaning is in the numbers behind it, so that is what the comparison reads; comparing the rendered picture instead would fail over a layout detail that carries no meaning.
Two behaviour fixes travel with these plots. The Filled Area Plot requires its line group once the switch that reads it is on, and the Contour Plot says what it cannot contour instead of ending the run.
Any related issues, documentation, discussions?
Part of #8325, 13 of 21; that issue lists the set in order. It needs #8327 for the trait, so it does not compile until that lands, and the rows these operators add to the verification runner follow with the harness rather than as whole new files here.
Closes #8283, closes #8076.
Closes #8419, the task this change is the whole of.
How was this PR tested?
Each operator asserts the block it emits in its own spec. Once the verification lands it is also run through the engine and through its generated script, on every configuration its schema offers, and the two answers compared; this branch is cut from main and does not carry that machinery, so those runs are not on this diff's CI.
Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)