You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A workflow can be built and read in the editor, but there is no form of it that runs anywhere else. A user who wants to keep a pipeline after the fact, hand it to a colleague who does not run Texera, or step through it in a notebook has nothing to take away.
What is missing is an export: given a plan, produce a single Python file that reads the same sources, applies the same operators in the same order, and prints its results. The operators already describe their work as Python for the engine, so the pieces exist; what is absent is a form of that description that stands on its own, without the runtime around it, and something to stitch the pieces into one script.
Proposed Solution or Design
An operator says how it reads outside the engine by implementing a StandaloneCodeGenerator trait, returning a block of pandas that names its inputs and outputs by position: in1df, in2df, out1df. A translator walks the plan in topological order, gives every output port a variable, substitutes those placeholders for the variables its upstreams were given, and prints the leaves. A variadic port, of which Union is the one example, needs the whole list of upstreams rather than a fixed count, since any count the operator states would be wrong for some workflow.
The compiling service exposes the result as an endpoint, so the editor can offer the script for a plan the user has open.
An operator that has no generator yet leaves a commented placeholder in the script rather than a line that looks like it works, so the export is useful before every operator implements it.
Each is sized to be read in one sitting. Deriving a configuration was the one that was not, at two thousand lines, so it is now two: the entry that produces one configuration, and the entry that sweeps it into the variants. The two share no member, and no line moved between them. The export comes first, then the machinery that runs an operator both ways, then the operator families that implement the trait, then the two changes that read every operator at once — those come after the families so their assertions hold as written — and the editor last. This issue stays open until the final one lands.
What the order actually requires
The list above is one order that works, but it is not a queue.
graph LR
E["PR 8327<br/>the export"]
H["PR 8356<br/>engine harness"]
FAM["operator families<br/>8340 8341 8348 8367 8368<br/>8344 → 8346 8365 8366"]
P["PR 8357<br/>python harness"]
R["run and compare<br/>8358 8359"]
T["PR 8360<br/>canonical table"]
C["configurations<br/>8361 8363"]
SW["PR 8388<br/>sweep"]
RUN["PR 8362<br/>run every operator"]
COV["PR 8364<br/>coverage report"]
UI["PR 8350<br/>the editor"]
E --> FAM
E --> R
H --> P
H --> T
P --> R
T --> C
FAM -->|"@SampleColumn"| C
R --> RUN
C --> RUN
C --> SW
RUN --> COV
SW --> COV
E -.->|endpoint only| UI
Feature Summary
A workflow can be built and read in the editor, but there is no form of it that runs anywhere else. A user who wants to keep a pipeline after the fact, hand it to a colleague who does not run Texera, or step through it in a notebook has nothing to take away.
What is missing is an export: given a plan, produce a single Python file that reads the same sources, applies the same operators in the same order, and prints its results. The operators already describe their work as Python for the engine, so the pieces exist; what is absent is a form of that description that stands on its own, without the runtime around it, and something to stitch the pieces into one script.
Proposed Solution or Design
An operator says how it reads outside the engine by implementing a
StandaloneCodeGeneratortrait, returning a block of pandas that names its inputs and outputs by position:in1df,in2df,out1df. A translator walks the plan in topological order, gives every output port a variable, substitutes those placeholders for the variables its upstreams were given, and prints the leaves. A variadic port, of which Union is the one example, needs the whole list of upstreams rather than a fixed count, since any count the operator states would be wrong for some workflow.The compiling service exposes the result as an endpoint, so the editor can offer the script for a plan the user has open.
An operator that has no generator yet leaves a commented placeholder in the script rather than a line that looks like it works, so the export is useful before every operator implements it.
The set, in order
Each is sized to be read in one sitting. Deriving a configuration was the one that was not, at two thousand lines, so it is now two: the entry that produces one configuration, and the entry that sweeps it into the variants. The two share no member, and no line moved between them. The export comes first, then the machinery that runs an operator both ways, then the operator families that implement the trait, then the two changes that read every operator at once — those come after the families so their assertions hold as written — and the editor last. This issue stays open until the final one lands.
What the order actually requires
The list above is one order that works, but it is not a queue.
Affected Area
Workflow Engine (Amber), Workflow UI