refactor[next]: merge OTF definitions into stages, extract artifacts, enforce the layering

[egparedes]

Restructure the gt4py.next.otf modules so that the DSL-aware and
DSL-agnostic halves of the toolchain vocabulary live in separate modules,
without renaming any class:

• Merge otf.definitions into otf.stages: definition-stage TypeVars and
  aliases (IRDefinitionT, ArgsDefinitionT, ConcreteProgramDef,
  CompilableProgramDef) and the step contracts (TranslationStep,
  CompilationStep) now live in stages; otf.definitions is deleted.
• Extract otf.artifacts: the DSL-agnostic artifact models (ProgramSource,
  BindingSource, ExtensionSource, BuildSystemProject,
  ExecutableProgram, CompilationArtifact) plus all code-spec classes
  (otf.code_specs is deleted) and format_source (moved from
  binding.interface, breaking the interface->code_specs import edge).
• Move the ConcreteArtifact pair type from otf.toolchain to the
  DSL-neutral bottom module otf.workflow.
• Drop the step-protocol base classes from CPPCompiler, DaCeTranslator
  and DaCeCompiler; the protocols are structural, so nothing changes for
  isinstance/fingerprint purposes (fingerprints use qualified names only).
• Delete otf.stages.compilable_program_fingerprinter, a one-line alias for
  fingerprinting.strict_fingerprinter with two users, and name the
  fingerprinter directly at the gtfn and dace call sites instead. Note that
  ADR 0023 still refers to the alias by name, and describes that input
  fingerprinter as lenient where the code has always used the strict one;
  that pre-existing drift is left for a follow-up, since ADRs are
  append-only.

This severs otf.compilation from the IRs, and only that. Measured on the
module-level import graph (AST, counting only what a real interpreter runs
when the module is loaded -- the if TYPE_CHECKING: branch and function-body
imports are not edges, though a TYPE_CHECKING else: is -- and resolving
from gt4py.next import config to gt4py.next.config rather than to the
package): before,
compilation.compiler, compilation.build_systems.cmake and
compilation.build_systems.compiledb each reached 10 ffront/iterator
modules through otf.definitions -> ffront.stages; after, they reach none.
otf.runners and otf.compilation.cache were already IR-free before this
change, so no credit is due there. otf.binding is not severed: it still
reaches iterator.type_system.type_specifications through
binding.cpp_interface -> type_system.type_info, unchanged by this commit.
Fixing that means splitting is_compatible_type, whose iterator
specializations already carry a TODO at
src/gt4py/next/type_system/type_info.py:429; that is out of scope here.

otf.artifacts is IR- and frontend-free but not type-system-free: a
ProgramSource entry point is an otf.binding.interface.Function whose
parameters carry TypeSpecs, so importing otf.artifacts loads
type_system.type_specifications by construction.

tests/next_tests/unit_tests/otf_tests/test_import_boundaries.py turns the
above into a check instead of a claim. It reconstructs that same graph from
the AST and asserts the DSL-agnostic OTF modules (all of
otf.compilation.*, plus otf.workflow, otf.artifacts,
otf.binding.interface and otf.runners) reach no IR module, while pinning
otf.binding's remaining IR edge exactly, so the debt can neither grow nor
silently outlive its fix. Deferring an import into a function body is
deliberately not an edge: it is the sanctioned way to reach a higher layer
without an import-time dependency, and instrumentation.stage_dump relies
on it to pretty-print FOAST stages. It cannot be done by importing the
modules: each
of them does from gt4py.next import config, which executes the package
__init__ and pulls in the whole DSL, making every module reachable from
every other at runtime. tach is also the wrong tool here -- it models
direct edges between coarse declared modules, not transitive reachability,
and with exact = true a single gt4py.next.otf.compilation entry forces
the reverse edges (runners, gtfn, the dace workflow) to be declared too,
cascading into a repo-wide restructure of tach.toml.

Moving ConcreteArtifact rotates the two persistent translation-cache keys
(gtfn and dace) once. The strict input fingerprinter tags a dataclass with
its fully qualified name, so otf.toolchain.ConcreteArtifact and
otf.workflow.ConcreteArtifact hash differently even though the fields are
unchanged. The effect is a one-time cold rebuild, never a stale hit, and
BUILD_CACHE_VERSION_ID already rotates these keys at every release. The
later ConcreteArtifact -> ProgramWithArgs rename rotates them a second
time, so the two collapse into a single cold rebuild only if both land in
the same release.

Breaking: gt4py.next.otf.definitions and gt4py.next.otf.code_specs are
gone, and the names they held must be imported from gt4py.next.otf.stages
and gt4py.next.otf.artifacts respectively; otf.toolchain.ConcreteArtifact
moves to otf.workflow.ConcreteArtifact. No compatibility re-exports are
left behind.

[Copilot]

Pull request overview

This PR continues the gt4py.next.otf refactor by reorganizing module responsibilities: DSL-aware stage/type vocabulary is consolidated into otf.stages, DSL-agnostic artifact models and code-specs are extracted into a new otf.artifacts module, and the ConcreteArtifact carrier type is moved into the DSL-neutral otf.workflow. It updates runners, compilation/build-system code, tests, and docs to use the new import locations, and removes the old modules without compatibility re-exports.

Changes:

• Merged otf.definitions contracts/type aliases into otf.stages and deleted otf/definitions.py.
• Introduced otf.artifacts (migrating code_specs + artifact data models + format_source) and deleted otf/code_specs.py.
• Moved ConcreteArtifact to otf.workflow and updated toolchain/runners/tests/docs accordingly.

Reviewed changes

Copilot reviewed 42 out of 42 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
tests/next_tests/unit_tests/program_processor_tests/runners_tests/dace_tests/test_dace_translation.py	Update ConcreteArtifact import to otf.workflow.
tests/next_tests/unit_tests/program_processor_tests/runners_tests/dace_tests/test_dace_compilation.py	Switch OTF source models/code specs to otf.artifacts.
tests/next_tests/unit_tests/program_processor_tests/runners_tests/dace_tests/test_dace_cache_consistency.py	Switch OTF source models/code specs to otf.artifacts.
tests/next_tests/unit_tests/program_processor_tests/runners_tests/dace_tests/test_dace_bindings.py	Switch OTF source model generics/code specs to otf.artifacts.
tests/next_tests/unit_tests/program_processor_tests/runners_tests/dace_tests/test_dace_backend.py	Move CompilableProgramDef reference from definitions to stages.
tests/next_tests/unit_tests/program_processor_tests/codegens_tests/gtfn_tests/test_gtfn_module.py	Update CompilableProgramDef/code spec imports to stages+artifacts.
tests/next_tests/unit_tests/otf_tests/test_languages.py	Update code spec + ProgramSource imports to otf.artifacts.
tests/next_tests/unit_tests/otf_tests/test_compiled_program.py	Update ConcreteArtifact import to otf.workflow.
tests/next_tests/unit_tests/otf_tests/compilation_tests/build_systems_tests/test_cache_consistency.py	Update build-cache test types to otf.artifacts.
tests/next_tests/unit_tests/otf_tests/compilation_tests/build_systems_tests/conftest.py	Update test fixtures to construct otf.artifacts sources/specs.
src/gt4py/next/program_processors/runners/roundtrip.py	Update workflow typing to use stages.CompilableProgramDef and artifact types.
src/gt4py/next/program_processors/runners/gtfn.py	Switch executable/source types to otf.artifacts; inline strict fingerprinter.
src/gt4py/next/program_processors/runners/dace/workflow/translation.py	Update translation step typing + ProgramSource/code spec usage to artifacts.
src/gt4py/next/program_processors/runners/dace/workflow/factory.py	Inline strict fingerprinter and remove dependency on stages fingerprinter alias.
src/gt4py/next/program_processors/runners/dace/workflow/compilation.py	Switch extension source/compiled program types to otf.artifacts; drop nominal protocol bases.
src/gt4py/next/program_processors/runners/dace/workflow/bindings.py	Switch binding/extension source models to otf.artifacts.
src/gt4py/next/program_processors/runners/dace/workflow/backend.py	Switch CompilationArtifact/ExecutableProgram types to otf.artifacts.
src/gt4py/next/program_processors/runners/dace/program.py	Update ConcreteArtifact usage to otf.workflow.
src/gt4py/next/program_processors/codegens/gtfn/gtfn_module.py	Switch code specs + ProgramSource + format_source to otf.artifacts; use stages.TranslationStep.
src/gt4py/next/otf/workflow.py	Add ConcreteArtifact carrier type here (DSL-neutral location).
src/gt4py/next/otf/toolchain.py	Remove local ConcreteArtifact; update adapters to reference workflow.ConcreteArtifact.
src/gt4py/next/otf/stages.py	Redefine as the DSL-aware vocabulary + step contracts; remove artifact model classes.
src/gt4py/next/otf/runners.py	Switch runner contracts to otf.artifacts.CompilationArtifact.
src/gt4py/next/otf/recipes.py	Update recipe workflow generics to stages + artifacts.
src/gt4py/next/otf/definitions.py	Deleted (merged into otf.stages).
src/gt4py/next/otf/compiled_program.py	Switch compiled-program pools/futures to otf.artifacts types.
src/gt4py/next/otf/compilation/compiler.py	Switch build pipeline types/specs to otf.artifacts; drop nominal protocol base.
src/gt4py/next/otf/compilation/cache.py	Change cache folder API to accept artifacts.ExtensionSource.
src/gt4py/next/otf/compilation/build_systems/compiledb.py	Switch code spec + source/container types to otf.artifacts.
src/gt4py/next/otf/compilation/build_systems/cmake.py	Switch code spec + source/container types to otf.artifacts.
src/gt4py/next/otf/compilation_tasks.py	Update type aliases to stages.*ProgramDef and executable program type to artifacts.
src/gt4py/next/otf/code_specs.py	Deleted (moved into otf.artifacts).
src/gt4py/next/otf/binding/nanobind.py	Switch code specs + binding/extension source types; use artifacts.format_source.
src/gt4py/next/otf/binding/interface.py	Remove format_source helper (moved to otf.artifacts).
src/gt4py/next/otf/artifacts.py	New module containing code specs, source containers, artifact protocols, and format_source.
src/gt4py/next/ffront/stages.py	Update ConcreteArtifact type aliases to point to otf.workflow.
src/gt4py/next/ffront/past_to_itir.py	Move CompilableProgramDef reference to otf.stages; update examples to workflow.ConcreteArtifact.
src/gt4py/next/ffront/past_process_args.py	Update ConcreteArtifact construction to otf.workflow.
src/gt4py/next/ffront/foast_to_past.py	Update ConcreteArtifact construction to otf.workflow.
src/gt4py/next/ffront/decorator.py	Update ConcreteArtifact construction to otf.workflow.
src/gt4py/next/backend.py	Switch backend contracts to stages (DSL-aware) + artifacts (DSL-agnostic).
docs/user/next/advanced/HackTheToolchain.md	Update tutorial snippets to new module layout (workflow/stages/artifacts).

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[havogt]

The move itself checks out. I normalised the diff through the rename map
(definitions. → stages., stages.{ProgramSource,…} → artifacts., code_specs. →
artifacts., toolchain.ConcreteArtifact → workflow.ConcreteArtifact) and cancelled
matching hunks: zero non-cancelling hunks outside the moved files. No leftover
re-exports, no dangling references to the deleted modules, no import cycles anywhere in
otf.*. otf.workflow really is a DSL-neutral bottom — it reaches neither the IR, the
frontend, nor the type system — so moving ConcreteArtifact there does exactly what the
commit message says it's for.

Three comments, none of them blocking. They're all about the invariant the PR
establishes rather than the diff, since that's the part nothing currently tests.

I measured them by building the module-level import graph from the AST at
otf-split-0-trivial-deletions and at this branch, and comparing reachability into
gt4py.next.iterator, gt4py.next.ffront and gt4py.next.type_system — excluding
gt4py/next/__init__.py from traversal, since it imports the world.

---

1. The layering claim holds for otf.compilation, not for otf.binding

After this, otf.compilation, otf.runners and otf.binding no longer import any
IR/frontend module.

module	before	after
otf.compilation.compiler	IR ✓ frontend ✓	neither
otf.compilation.build_systems.cmake	IR ✓ frontend ✓	neither
otf.compilation.build_systems.compiledb	IR ✓ frontend ✓	neither
otf.runners	already neither	neither
otf.binding.nanobind	IR ✓	IR ✓
otf.binding.cpp_interface	IR ✓	IR ✓

otf.compilation is genuinely severed from the IR and the frontend, and that's the real
result here — worth stating precisely, because it's the part that's true. otf.runners
was already clean before this PR. otf.binding still reaches the IR, unchanged:

otf/binding/nanobind.py:21       from gt4py.next.otf.binding import cpp_interface, interface
otf/binding/cpp_interface.py:13  from gt4py.next.type_system import type_info as ti, ...
type_system/type_info.py:19      from gt4py.next.iterator.type_system import ...

That last edge is the one you already have a TODO on — type_info.py:429, "This function
has specializations on Iterator types, which are not part of the general / shared type
system." So otf.binding can't be clean until is_compatible_type is split, which is
clearly not this PR's job.

Suggest narrowing the sentence to otf.compilation and noting otf.binding as blocked on
that TODO — it makes the claim checkable, and it puts a second caller behind fixing it.

2. artifacts.py states an invariant it doesn't hold

src/gt4py/next/otf/artifacts.py:17:

Nothing in here knows about GT4Py IRs or the type system.

The IR half is true — artifacts has no path to the IR at all. The type-system half isn't,
structurally rather than incidentally:

artifacts.py:29                    from gt4py.next.otf.binding import interface
artifacts.ProgramSource.entry_point : interface.Function
binding/interface.py:19             Parameter.type_ : ts.TypeSpec

so import gt4py.next.otf.artifacts loads gt4py.next.type_system.type_specifications,
and every ProgramSource carries TypeSpecs in its entry point.

This is the module that defines where the DSL-agnostic line falls, so the docstring is
load-bearing: someone reading it would reasonably conclude artifacts can sink below the
type system, and it can't without moving binding.interface with it. The commit message's
narrower wording — no IR, no frontend — is the invariant that's actually true.

3. Nothing enforces the boundary, and it doesn't exist at runtime

At import time the separation isn't observable, because every one of these modules does
from gt4py.next import config, which executes the package __init__:

import gt4py.next.otf.compilation.compiler
# before: iterator=True ffront=True type_system=True  (249 gt4py modules)
# after:  iterator=True ffront=True type_system=True  (248 gt4py modules)

Which is fine — a cleaner source-level dependency graph is a good goal on its own. But it
does mean the invariant has no self-defence: it isn't visible in a profile, it doesn't
break a test when violated, and tach.toml treats gt4py.next as one opaque node, so
tach check says nothing about it either. No PR in the stack touches tach.toml.

That leaves 42 files of boundary that the next from gt4py.next.iterator import ... in
otf/compilation/ silently undoes. Worth landing the enforcement in the same PR that
creates the invariant — either tach.toml entries for the gt4py.next.otf.* submodules,
or a small test asserting the reachability set. Otherwise it's documentation, and this
stack is largely about not trusting documentation over checks.

[havogt]

Suggested change

	This module hosts the DSL-aware half of the on-the-fly compilation vocabulary
	(ex `otf.definitions`, merged here): which forms a program definition can
	take on its way from DSL source to a compilable IR program, and the typed
	contracts of the steps a compile pipeline is composed of. The DSL-agnostic
	artifact models these contracts produce live in `gt4py.next.otf.artifacts`.
	This module hosts the DSL-aware half of the on-the-fly compilation vocabulary:
	which forms a program definition can
	take on its way from DSL source to a compilable IR program, and the typed
	contracts of the steps a compile pipeline is composed of. The DSL-agnostic
	artifact models these contracts produce live in `gt4py.next.otf.artifacts`.

[havogt]

Please double-check if we are documenting history in comments/docstrings somewhere else.

[egparedes]

Applied, with the line-wrapping normalized. The history note is gone.

[egparedes]

Swept the 42 files this commit touches for ex <module> / moved from / merged here / formerly / previously / extracted from phrasing. Two real hits, both introduced by this commit, both removed:

• otf/stages.py:13 — "(ex otf.definitions, merged here)" (your suggestion above).
• otf/artifacts.py:13 — "(SourceCodeSpec and friends, ex code_specs module)".

The remaining grep hits are not history notes and are left alone: otf/compilation/cache.py:86 ("incompatibility with previously cached builds" — describes runtime behaviour), artifacts.py:187 ("source code extracted from an ExtensionSource" — describes dataflow), and two pre-existing lines in the dace runner that this commit only re-indented.

[havogt]

Not 100% sure about this: should we keep these references to other modules? There is a potential for drift.

[egparedes]

Agreed on the drift risk, but the rationale is load-bearing, so I kept it and removed the drift-prone part.

The constraint is real and I verified it: ffront/stages.py:82,96,108,121 build workflow.ConcreteArtifact[...] aliases at module scope, and otf/stages.py:23 imports ffront.stages. So putting ConcreteArtifact in otf.stages closes an ffront.stages -> otf.stages -> ffront.stages cycle. Deleting the paragraph would leave the next reader free to "tidy" it into otf.stages and rediscover that the hard way.

What I dropped is the pair of module names, which is exactly the part that rots. The docstring now states the shape of the constraint ("those parameterize it while they are being imported, so hosting it any higher would close an import cycle") and points at the mechanical check instead of at siblings:

    This is the envelope threaded through the definition-transforming half of
    the toolchain, so it is generic over DSL-frontend types. It nevertheless
    lives in this DSL-neutral bottom module rather than beside the stage
    definitions that use it: those parameterize it while they are being
    imported, so hosting it any higher would close an import cycle. The
    invariant that keeps this working -- this module reaching no IR or
    frontend module at import time -- is checked by the OTF import-boundary
    test in `tests/next_tests/unit_tests/otf_tests/`.

otf.workflow is in that test's IR_FREE_MODULES set, and I confirmed it currently reaches zero ffront/iterator modules at import time. So if the prose does drift, the test fails rather than the docstring quietly lying.

[egparedes]

All three are valid. Addressed in the amended commit (force-pushed, 5fd4ac0 -> f4abbb9); the PR description now matches the new commit message.

I rebuilt the graph independently before changing anything, with the same construction you describe: module-level statements only, if TYPE_CHECKING: blocks excluded, and from X import a resolved to X.a when that is a module. That last rule matters — resolving it to X instead makes from gt4py.next import config an edge into the package __init__, and then every otf module reaches all 94 ffront/iterator modules and the measurement says nothing. I got that wrong on the first pass and it produced exactly the uniform "94" that hides the signal.

1. Layering claim. Reproduced your numbers exactly, comparing ca5b484 against the branch tip:

module	IR/frontend reachable before	after
otf.compilation.compiler	12	0
otf.compilation.build_systems.cmake	12	0
otf.compilation.build_systems.compiledb	12	0
otf.compilation.cache	0	0
otf.runners	0	0
otf.binding.nanobind	1	1

The three that changed all went through otf.definitions -> ffront.stages. otf.runners and otf.compilation.cache were already clean, so the old claim took credit for them; otf.binding is unchanged at 1, via binding.cpp_interface -> type_system.type_info -> iterator.type_system.type_specifications. The claim is now narrowed to "this severs otf.compilation from the IRs, and only that", says explicitly that otf.runners and otf.compilation.cache were already IR-free ("no credit is due there"), and records otf.binding as still reaching the IR, blocked on the is_compatible_type TODO at src/gt4py/next/type_system/type_info.py:429 (which I confirmed is still on line 429). Splitting that function stays out of scope here.

2. artifacts.py docstring. You're right and it is structural, not incidental: artifacts.py:29 imports otf.binding.interface, ProgramSource.entry_point (artifacts.py:140) is an interface.Function, and interface.Parameter.type_ (binding/interface.py:19) is a ts.TypeSpec. The graph confirms otf.artifacts reaches type_system.type_specifications. The docstring now claims only "nothing in here knows about the GT4Py IRs or the DSL frontend" and then states the type-system dependency positively, rather than pretending it away.

3. Enforcement. Added the test, not tach.toml. I tried tach first, adding a single gt4py.next.otf.compilation module entry, and tach check produced 24 failures — not just the intended forward edges, but the reverse direction too: with exact = true, carving out one submodule makes the residual gt4py.next node unable to depend on it, so otf/runners.py, runners/gtfn.py and three dace workflow modules all break. Beyond the blast radius there is a modelling mismatch: tach checks direct declared edges, whereas this invariant is about transitive reachability (otf.compilation.cache -> otf.artifacts -> binding.interface -> ...), so expressing it in tach means declaring every intermediate as its own module and enumerating its deps. And tach counts from gt4py.next import config as a dependency on gt4py.next, i.e. it collapses precisely the distinction the check rests on.

So: tests/next_tests/unit_tests/otf_tests/test_import_boundaries.py. It rebuilds the graph from the AST, and:

• asserts the DSL-agnostic set reaches no IR module — otf.workflow, otf.artifacts, otf.binding.interface, otf.runners, and every module under otf.compilation (glob-expanded, so a new file in that package is covered without editing the test);
• pins otf.binding.{cpp_interface,nanobind}'s IR reachability to exactly {gt4py.next.iterator.type_system.type_specifications}, with the type_info.py:429 TODO cited in the constant. It fails if the debt grows, and it also fails if the set becomes empty, telling you to promote the module into the clean set — so the pin can't silently outlive its fix;
• has a guard test that fails if SRC_ROOT isn't the working tree, so it can't pass vacuously on an odd install layout.

I checked it actually bites: adding from gt4py.next.iterator import ir as itir to otf/compilation/compiler.py fails the test for compiler, cmake and compiledb, printing the shortest chain for each. 4 passed after reverting. The test's own module docstring records why it works from the AST rather than by importing.

Verification: pre-commit run --files on the four changed files (ruff, ruff-format, tach, mypy) passes; pytest tests/next_tests/unit_tests/otf_tests/test_import_boundaries.py is 4 passed.

One thing I could not do: gh pr edit --base main is rejected with Cannot change the base branch because the pull request is part of a stack. This PR still targets otf-split-0-trivial-deletions, which has since landed on main as ca5b484; the branch itself is rebased on main and contains exactly one commit, so the diff shown once the base moves will be the same one reviewed here.

[egparedes]

Two follow-ups to my previous reply, both after re-running the new check against the whole stack.

The base is now retargeted. #2736 was squash-merged, so this PR was still
targeting otf-split-0-trivial-deletions and showed up as conflicting. The
GitHub stack object refuses base changes, so I unstacked, retargeted to main,
and relinked (stack #2850). All five remaining PRs are MERGEABLE again.

The import-boundary test was wrong, and running it across the stack is what
found it. As first written it walked the AST with ast.walk, which descends
into function bodies — so it counted deferred imports as edges. It passed on
this PR but failed two commits later, on the observability PR, reporting:

gt4py.next.otf.workflow -> gt4py.next.instrumentation.stage_dump
                        -> gt4py.next.ffront.foast_pretty_printer

That is not a real violation. stage_dump's two ffront imports are inside
function bodies (stage_dump.py:123 and :193) precisely so that importing it
does not pull in the frontend. The test was contradicting its own docstring,
which claimed it counted "only the imports that a real interpreter would execute
when the module is loaded".

It now walks only what runs at module-load time: class bodies and module-level
conditionals are entered, function bodies and if TYPE_CHECKING: blocks are
not. Deferring an import into a function is the sanctioned escape hatch for
exactly this layering problem, so counting it would have forbidden the mechanism
the boundary depends on.

I re-verified it still bites rather than passing vacuously: injecting a
top-level from gt4py.next.iterator import ir into otf/compilation/compiler.py
fails the check and prints the transitive chain for all three affected modules,
while the same import inside a function body correctly does not. The check is
green at all five levels of the stack.