Skip to content

refactor[next]: replace workflow combinators with explicit typed pipelines - #2743

Open
egparedes wants to merge 1 commit into
otf-split-3-observabilityfrom
otf-split-4-pipeline
Open

refactor[next]: replace workflow combinators with explicit typed pipelines#2743
egparedes wants to merge 1 commit into
otf-split-3-observabilityfrom
otf-split-4-pipeline

Conversation

@egparedes

@egparedes egparedes commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

The workflow-combinator framework introduced by ADR 0011 had grown to a dozen
abstractions to express what is, in the end, function composition. Measured
against actual use, only CachedStep was a deep module; the rest were shallow
wrappers around Callable[[S], T], and the reflection loop in
NamedStepSequence.__call__ was Any-typed, defeating the static typing
ADR 0011 prized.

The named pipelines become plain frozen dataclasses with an explicit, fully
typed __call__:

  • backend.Transforms keeps its input-dependent step selection -- the match
    that used to live in step_order now lives in __call__, where the order is
    literally readable -- and the step_order method is retained only to raise,
    so a downstream override fails loudly instead of being silently ignored.
  • recipes.OTFCompileWorkflow becomes backend.CompilePipeline and spells out
    its three steps; otf.recipes and otf.toolchain are deleted.
  • Both take over emitting the stage_hook added in the previous PR. Names,
    order, count and artifacts are unchanged; the two instrumentation tests that
    assert the exact stage sequences pass unmodified, which is the proof.

Steps are now plain callables, named by the workflow.Step[S, T] alias, and
customization stays composition-time via dataclasses.replace. Deleted:
Workflow, ChainableWorkflowMixin, ReplaceEnabledWorkflowMixin,
NamedStepSequence, MultiWorkflow, StepSequence, make_step, .chain,
the three adapters in otf.toolchain, and the five adapted_*_factory
wrappers whose only job was to wrap a function into an adapter. CachedStep's
body is unchanged; it loses only the mixin bases, and with them .replace and
.chain.

Because steps no longer need to be adapter objects, the seven ffront factories
collapse to returning either the bare function or a CachedStep around it, and
the three per-step callers in decorator.py lose their wrap/unwrap dance.

What ADR 0011's decisions become: named steps with a visible order are now
dataclass fields plus an explicit __call__; statically typed composition is
checked end-to-end instead of through an Any-typed reflection loop;
customization at composition time is dataclasses.replace; and steps still
compose across backends because every existing step already satisfies
Step[S, T].

No behavior change, and -- unlike the naming PR -- no persistent cache key
rotates: fingerprints embed a class's qualified name and fields but never its
bases, and neither renamed pipeline is reachable from a persistent cache's
fingerprint graph.

Breaking, with no compatibility aliases: the deleted combinators and the
otf.recipes / otf.toolchain modules, .replace() / .chain() on the
classes that kept them via the mixins, overriding Transforms.step_order,
roundtrip.foast_to_gtir_step (now a data-only step), and
linter_factory(adapter=...) (the parameter was accepted and ignored).

@egparedes
egparedes marked this pull request as ready for review July 30, 2026 17:58
@egparedes
egparedes force-pushed the otf-split-4-pipeline branch from 55f9fcb to 706dcba Compare July 30, 2026 17:58
@egparedes
egparedes force-pushed the otf-split-4-pipeline branch from 706dcba to 605676d Compare July 31, 2026 16:15
@egparedes
egparedes force-pushed the otf-split-4-pipeline branch from 605676d to f120462 Compare August 17, 2026 15:07
@egparedes
egparedes force-pushed the otf-split-4-pipeline branch from f120462 to b70d427 Compare August 20, 2026 16:17
@egparedes
egparedes force-pushed the otf-split-4-pipeline branch from b70d427 to d670473 Compare August 20, 2026 16:52
@egparedes
egparedes force-pushed the otf-split-4-pipeline branch 2 times, most recently from eff4d3c to ded2e4d Compare August 26, 2026 11:05
@egparedes
egparedes force-pushed the otf-split-4-pipeline branch from ded2e4d to 33f9c04 Compare August 28, 2026 13:06
@egparedes
egparedes force-pushed the otf-split-4-pipeline branch from 33f9c04 to 7f833bc Compare August 28, 2026 13:09
@egparedes
egparedes force-pushed the otf-split-4-pipeline branch from 7f833bc to 03ccbeb Compare September 1, 2026 18:30
@egparedes
egparedes force-pushed the otf-split-4-pipeline branch from 03ccbeb to dc905ed Compare September 9, 2026 16:34
@egparedes
egparedes force-pushed the otf-split-4-pipeline branch from dc905ed to 308efe8 Compare September 9, 2026 16:51
…lines

The workflow-combinator framework introduced by ADR 0011 had grown to a dozen
abstractions to express what is, in the end, function composition. Measured
against actual use, only `CachedStep` was a deep module; the rest were shallow
wrappers around `Callable[[S], T]`, and the reflection loop in
`NamedStepSequence.__call__` was `Any`-typed, defeating the static typing
ADR 0011 prized.

The named pipelines become plain frozen dataclasses with an explicit, fully
typed `__call__`:

- `backend.Transforms` keeps its input-dependent step *selection* -- the `match`
  that used to live in `step_order` now lives in `__call__`, where the order is
  literally readable -- and the `step_order` method is retained only to raise,
  so a downstream override fails loudly instead of being silently ignored.
- `recipes.OTFCompileWorkflow` becomes `backend.CompilePipeline` and spells out
  its three steps; `otf.recipes` and `otf.toolchain` are deleted.
- Both take over emitting the `stage_hook` added in the previous PR. Names,
  order, count and artifacts are unchanged; the two instrumentation tests that
  assert the exact stage sequences pass unmodified, which is the proof.

Steps are now plain callables, named by the `workflow.Step[S, T]` alias, and
customization stays composition-time via `dataclasses.replace`. Deleted:
`Workflow`, `ChainableWorkflowMixin`, `ReplaceEnabledWorkflowMixin`,
`NamedStepSequence`, `MultiWorkflow`, `StepSequence`, `make_step`, `.chain`,
the three adapters in `otf.toolchain`, and the five `adapted_*_factory`
wrappers whose only job was to wrap a function into an adapter. `CachedStep`'s
body is unchanged; it loses only the mixin bases, and with them `.replace` and
`.chain`.

Because steps no longer need to be adapter objects, the seven ffront factories
collapse to returning either the bare function or a `CachedStep` around it, and
the three per-step callers in `decorator.py` lose their wrap/unwrap dance.

What ADR 0011's decisions become: named steps with a visible order are now
dataclass fields plus an explicit `__call__`; statically typed composition is
checked end-to-end instead of through an `Any`-typed reflection loop;
customization at composition time is `dataclasses.replace`; and steps still
compose across backends because every existing step already satisfies
`Step[S, T]`.

No behavior change, and -- unlike the naming PR -- no persistent cache key
rotates: fingerprints embed a class's qualified name and fields but never its
bases, and neither renamed pipeline is reachable from a persistent cache's
fingerprint graph.

Breaking, with no compatibility aliases: the deleted combinators and the
`otf.recipes` / `otf.toolchain` modules, `.replace()` / `.chain()` on the
classes that kept them via the mixins, overriding `Transforms.step_order`,
`roundtrip.foast_to_gtir_step` (now a data-only step), and
`linter_factory(adapter=...)` (the parameter was accepted and ignored).

Claude-Session: https://claude.ai/code/session_01R8zRtFMhdJ8c96XJYCXkRk
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant