diff --git a/.artifacts/.gitignore b/.artifacts/.gitignore new file mode 100644 index 000000000..d6b7ef32c --- /dev/null +++ b/.artifacts/.gitignore @@ -0,0 +1,2 @@ +* +!.gitignore diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 2ad96136d..4a707998b 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -147,7 +147,7 @@ jobs: if: github.ref == 'refs/heads/main' && github.event_name != 'pull_request' uses: actions/upload-pages-artifact@v4 with: - path: site + path: .artifacts/site deploy: name: Documentation deployment · GitHub Pages diff --git a/.github/workflows/publish-to-pypi.yml b/.github/workflows/publish-to-pypi.yml index e2910e99a..bdd96ea4b 100644 --- a/.github/workflows/publish-to-pypi.yml +++ b/.github/workflows/publish-to-pypi.yml @@ -44,17 +44,17 @@ jobs: python -m pip install --upgrade pip python -m pip install build twine - name: Build source and wheel distributions - run: python -m build + run: python -m build --outdir .artifacts/dist - name: Check distribution metadata - run: python -m twine check dist/* + run: python -m twine check .artifacts/dist/* - name: Verify and install the wheel shell: bash run: | - mapfile -t wheels < <(compgen -G "dist/prik-*-py3-none-any.whl") - mapfile -t sdists < <(compgen -G "dist/prik-*.tar.gz") + mapfile -t wheels < <(compgen -G ".artifacts/dist/prik-*-py3-none-any.whl") + mapfile -t sdists < <(compgen -G ".artifacts/dist/prik-*.tar.gz") if (( ${#wheels[@]} != 1 || ${#sdists[@]} != 1 )); then echo "expected one universal wheel and one source distribution" >&2 - ls -la dist + ls -la .artifacts/dist exit 1 fi python -m venv "$RUNNER_TEMP/prik-release-check" @@ -68,7 +68,7 @@ jobs: uses: actions/upload-artifact@v4 with: name: python-package-distributions - path: dist/ + path: .artifacts/dist/ if-no-files-found: error retention-days: 7 @@ -86,6 +86,8 @@ jobs: uses: actions/download-artifact@v4 with: name: python-package-distributions - path: dist/ + path: .artifacts/dist/ - name: Publish distributions to PyPI uses: pypa/gh-action-pypi-publish@release/v1 + with: + packages-dir: .artifacts/dist/ diff --git a/.gitignore b/.gitignore index deab33b6a..70b6a2d00 100644 --- a/.gitignore +++ b/.gitignore @@ -10,13 +10,11 @@ mutants/ .ruff_cache/ .benchmarks/ htmlcov/ -site/ build/ *.pyc *.pyo *egg* -dist/* *.mod *.out diff --git a/AGENTS.md b/AGENTS.md index 42bd372c5..189b94c47 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -21,7 +21,7 @@ Do not spend context window or analysis on those files unless explicitly request When asked to change or move an API, import path, command, feature, or behavior, do not add or keep compatibility layers, aliases, shims, fallback paths, or legacy entrypoints unless explicitly requested. A requested change means the old behavior should be removed. When updating tests, remove obsolete tests that only assert removed/old implementation behavior does not exist. Do not preserve rejection or absence checks for API/features that were intentionally removed unless explicitly requested. -Before wrapper planning begins in `prik/codegen/planner.py`, the +Before wrapper planning begins in `prik/planning/planner.py`, the post-IR policy stage must have completed every semantic decision needed by wrapper generation, including object kind, ownership, transfer, destruction, mutability/writeback, nullability, output projection, release responsibility, diff --git a/CHANGELOG.md b/CHANGELOG.md index 0e661ffa8..fdf85d720 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,9 +9,75 @@ release tags add a leading `v` to the package version. ### Added +- Reorganized contributor documentation around a concise architecture guide + and one canonical page per production package, with local structures, + important objects, runnable examples, expected outputs, test owners, change + routes, and invariants. +- Consolidated cross-stage concepts and contributor workflows, retained future + and deferred designs explicitly, and removed TODO-only pages, duplicate + architecture maps, and completed migration ledgers. - Added Zenodo version and concept DOI links to the citation metadata, README, and About page. +### Changed + +- Reduced the root `prik` API to its version and normal-user build entrypoints; + parser, semantic, probe, runtime, and planning tools now use their owning + package import paths. +- Moved stage-record freezing from `prik.stage_values` to + `prik.utilities.stage_values`; the root module path was removed. +- Made `prik` an import-only package boundary by removing its direct-script + demonstration; command and stage-value examples remain available from their + owning modules. +- Expanded the contributor architecture and package guides into a complete + stage-by-stage tutorial, with every supported Python module, runnable example + result, focused test purpose, and change route recorded and checked against + the source tree. +- Moved generated documentation and distribution output under the hidden + `.artifacts/` directory in local commands and CI workflows. +- Centralized every production-file execution-example output contract in one + contributor-architecture test inventory with one named test per file. +- Renamed the central infrastructure owner to `execution_examples/` so its + responsibility is explicit in the test tree. +- Consolidated developer and maintainer material under one Contributor + Documentation tree and removed the separate maintainer documentation lane. +- Moved the bundled header-only binding runtime from the package root into + `prik.runtime.native_support`; generated builds continue to receive it under + their internal `binding_support/` include directory. +- Deferred the contributor architecture sections for the immature C input + parser and C-to-IR path while retaining the generated CPython C binding + backend documentation required by Fortran wrappers. +- Reorganized compiler and pre-parse infrastructure into `prik.compiler` and + `prik.preprocessing`, including C/Fortran preprocessing and target probes; + the former `prik.compiling`, `prik.probes`, parser-local C preprocessor, and + pipeline-local preprocessing import paths were removed. +- Replaced the public semantic-to-NumPy helper API with stage-owned semantic, + contract-runtime, and code-generation datatype catalogues, and documented the + complete internal datatype lifecycle from compiler probing to runtime + validation. +- Separated post-IR policy and wrapper planning into `prik.policy` and + `prik.planning`; code generation now renders plan-driven docstrings, and the + former maintainer import paths were removed. +- Added a top-level language-printer package for C, Fortran, and semantic + `.pyi` output, and made `pipeline.wrapper.WrapperGenerator` the single + plan-to-rendered-wrapper orchestration boundary. +- Documented the completed ownership vocabulary, lifetime-policy philosophy, + pointer-policy boundary, and maintainer change routes in one maintained + architecture reference. +- Moved exact overload selection from generated Python predicate chains to + generated C dispatchers with planned candidate IDs and direct switch-based + calls to the selected existing wrapper. +- Stopped standalone Fortran parser discovery from descending into inaccessible + procedure-internal subprograms; procedure-local callback interfaces remain + classified and discoverable. +- Made directory project parsing read and parse each discovered Fortran file + once before dependency ordering and project assembly. + +### Fixed + +- Unified source-level compile-time resolution across project and CLI parsing + so imported and host-associated kind facts also reach derived-type fields. + ## 0.2.1 — 2026-08-11 ### Added diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 716326905..63643cfdc 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -33,8 +33,8 @@ Keep the pull request easy to review: explain the problem, the solution, and how you verified it. All required GitHub checks must pass before merge. For the complete workflow, see the -[development guide](docs/developer/development-workflow.md) and -[quality-assurance guide](docs/developer/quality-assurance.md). +[contributing workflow](docs/developer/workflows/contributing.md) and +[quality-assurance guide](docs/developer/workflows/quality-assurance.md). ## License diff --git a/MANIFEST.in b/MANIFEST.in index fc4929396..e2755646a 100644 --- a/MANIFEST.in +++ b/MANIFEST.in @@ -1 +1,2 @@ include CHANGELOG.md +include .artifacts/.gitignore diff --git a/README.md b/README.md index 54ee763ce..b7fa93d0e 100644 --- a/README.md +++ b/README.md @@ -519,8 +519,8 @@ explicit build directories, depending on the command mode. ## Python API -Public entrypoints cover Fortran extension builds, parsing, semantic -conversion and `.pyi` emission: +Root entrypoints cover normal Fortran extension builds. Advanced parsing, +semantic conversion, and `.pyi` emission use their owning packages: ```python from prik import build_fortran_extension @@ -539,11 +539,9 @@ strings, focused tests, and already-preprocessed inputs. + +### Probe reports are evidence, not semantic models + +A probe report records reproducible compiler observations. It may be serialized +or cached, but it does not become semantic IR and it does not contain wrapper +policy. Source-to-IR conversion owns the interpretation of those observations. + +The Markdown datatype report lives in `prik/pipeline/type_mapping_report.py` +because it intentionally combines several stages: + +```text +probe facts -> semantic conversion -> backend NumPy projection -> Markdown +``` + +That report is documentation and inspection output. It is not an alternative +conversion path and must reuse the normal converters and backend catalogue. + +## Parsing And Semantic Normalization + +Parsers preserve native declarations rather than prematurely replacing them +with Python or NumPy types. Relevant parser facts include: + +- native base type and kind spelling; +- declaration or measured storage width; +- scalar versus array shape and rank; +- pointer, allocatable, target, optional, and value attributes; +- character kind and length syntax; +- derived-type identity and scope; +- procedure/callback signature structure; +- source coordinates and the original native spelling. + +The source-to-IR converters combine those facts with target measurements and +produce `SemanticType` plus `SemanticOrigin` and `SemanticStorageContract`. +`SemanticType.name` is the public semantic identity. `SemanticType.dtype` is +the resolved storage dtype used by later stages. They can differ when a stable +public concept has target-specific storage. + +For example, an unresolved native default integer can begin as `Int`, then +resolve to `Int32` or `Int64` after compiler measurement. The converter records +the source spelling and target provenance; it does not replace those facts with +`numpy.int32` or `numpy.int64`. + +## Semantic Scalar Catalogue + +`prik/semantics/scalar_types.py` is the single semantic vocabulary for +primitive scalar names. Its immutable `SemanticScalarSpec` records only facts +that are intrinsic to the semantic identity: + +- datatype family; +- storage width when the semantic identity fixes one; +- whether the name represents a Boolean storage contract. + +The module exposes checked helpers for scalar membership and Boolean storage +width. It does not import NumPy and contains no emitted source spelling. +Extended `Float128` and `Complex256` catalogue entries intentionally leave +`storage_bits` unresolved because supported targets can store them in 80/96/128 +or 160/192/256 bits respectively; compiler facts select the actual storage. + +Boolean names demonstrate why the semantic and NumPy layers are distinct: + +| Semantic name | Native storage contract | NumPy boundary dtype | +| --- | --- | --- | +| `Bool` | default or interoperable Boolean, normalized to 8-bit boundary storage | `numpy.bool_` | +| `Bool8` | 8 bits | `numpy.bool_` | +| `Bool16` | 16 bits | `numpy.bool_` | +| `Bool32` | 32 bits | `numpy.bool_` | +| `Bool64` | 64 bits | `numpy.bool_` | + +The binding normalizes Boolean values at the boundary, while the generated +bridge uses the compiler-resolved native logical representation. A NumPy dtype +alone therefore cannot reconstruct the original semantic Boolean contract. + +## Runtime Contract Factories + +`prik/contracts/__init__.py` owns the public names used by generated and edited +semantic `.pyi` contracts. Its private contract-factory catalogue maps a +semantic name to a real NumPy scalar factory where a portable runtime value +exists. This lets expressions such as `Float64()` create the exact scalar type +required by a generated wrapper and lets typed descriptor contracts retain a +concrete `numpy.dtype`. + +Names without a portable runtime factory remain explicit contract symbols and +raise a focused constructor error. Examples include unresolved `Int`, `UInt`, +`CEnum`, `Char`, `String`, and `Void`. The contracts package does not perform +source-to-IR conversion and its factories do not define native ABI storage. + +## Backend Primitive Scalar Catalogue + +`prik/codegen/primitive_scalar_types.py` owns two readable mappings. +`NumpyDtypeRegistry.TYPES` maps every resolved semantic dtype with a maintained +NumPy projection to its emitted expression. `PrimitiveScalarTypeRegistry.TYPES` +contains the narrower set with implemented native wrapper lowering. Each +`BackendScalarType` entry uses keyword arguments so a maintainer can audit one +row without remembering positional field order. + +The fields cover: + +| Field | Meaning | +| --- | --- | +| `semantic_name` | Resolved semantic key consumed from the wrapper plan. | +| `c_spelling` | Native binding-side storage spelling. | +| `fortran_spelling` | Generated bridge declaration spelling. | +| `python_parse_unit` | Python argument parsing unit used by the binding. | +| `numpy_type_macro` | NumPy array dtype identity checked or allocated in generated code. | +| `python_result_kind` | Result-conversion path for an ordinary procedure result. | +| `python_type_name` | Python/NumPy scalar expression shown in validation diagnostics or constructors. | +| `python_module_result_kind` | Result-conversion path for module state. | +| `cfi_type_spelling` | Descriptor element type identity for descriptor-based boundaries. | + +The catalogue contains only implemented primitive scalar lowering lanes. +Adding a semantic name to the semantic catalogue does not automatically enable +wrapper generation. Unsupported entries must continue to fail during policy or +planning rather than acquiring guessed backend spellings. + + + +## Why There Is No Universal NumPy-To-Semantic Map + +The maintained lookup direction is: + +```text +resolved semantic dtype -> stage-owned NumPy or backend facts +``` + +The reverse direction is not generally valid: + +- every Boolean storage contract projects to `numpy.bool_`; +- `numpy.longdouble`, `numpy.clongdouble`, and `numpy.uintp` vary by platform; +- source concepts such as unresolved `Int`, `CEnum`, and fixed-length native + character storage require context that a NumPy dtype does not carry; +- ownership, mutability, rank, layout, pointer association, allocation state, + and callback identity are not dtype properties. + +Runtime validation may compare an actual NumPy dtype with the exact dtype in a +completed plan. It must not use the observed dtype to infer semantic meaning or +select a different lowering path. If a future frontend accepts NumPy types as +source annotations, that frontend must own an explicitly contextual and +possibly lossy input mapping. + +## Non-Primitive Datatype Families + +### Arrays + +An array is not a separate scalar dtype. Semantic IR stores its element dtype, +rank, shape expressions, bounds provenance, layout/order, contiguity, and +pointer or allocatable attributes in `SemanticArrayContract`. Policy completes +copy/alias behavior, writeback, nullability, descriptor ownership, and result +projection. The plan then records exact validation and transfer actions. + +Generated bindings validate dtype, rank, shape, layout, alignment, +writeability, and permitted stride forms from that plan. They do not silently +cast or transpose unless policy selected an explicit copy path. + +### Characters And Strings + +Character handling combines element kind, declared or resolved length, scalar +versus array rank, and ABI byte storage. `String` is the stable semantic family, +but `numpy.str_` is only a Python-facing representation; fixed native character +storage may instead use exact byte buffers. Length and encoding constraints +must therefore survive semantic IR and policy completion. + +### Derived Types + +Derived-type identity is scoped and semantic. Generated wrappers keep native +objects opaque and use holders, accessors, and completed lifecycle policy +instead of mirroring an arbitrary native layout in Python. Field datatypes pass +through the same semantic and policy stages as ordinary variables. + +Arrays of derived types remain unsupported unless the language-support matrix +states otherwise. A primitive scalar registry entry must never be fabricated +for a derived identity. + +### Pointers And Allocatables + +Pointer and allocatable arrays combine an element semantic dtype with descriptor +kind, association/allocation state, ownership, nullability, and release +responsibility. Policy completes those decisions before planning. Runtime +handles expose descriptor-backed operations, while generated code uses the +planned element dtype for validation and descriptor metadata. + +A live zero-copy NumPy view can become stale after native deallocation, +reallocation, or pointer reassociation. Datatype matching does not solve that +lifetime boundary; see the [Policy package](../packages/policy.md). + +### Callbacks + +A callback datatype is a full prototype: argument types, result type, calling +convention, value/reference storage, rank, and mutability. It is not reducible +to a scalar function-pointer token. Semantic conversion resolves the prototype, +policy completes callback handoff and result behavior, and planning freezes the +native slots used by codegen. + +## Policy And Planning Boundaries + +Datatype facts answer questions such as “this is a rank-two `Float64` array.” +They do not answer: + +- who owns it; +- whether it is borrowed, copied, moved, or aliased; +- whether native mutation is visible or discarded; +- whether an output is hidden and projected into the Python result; +- whether storage is stack, heap, or alias; +- whether destruction or descriptor release is required; +- whether a getter or setter is exposed. + +Those decisions belong to post-IR policy completion. `WrapperPlanner` projects +the completed facts into typed transfer, result, field, module-variable, and +lifecycle plans. Backend generators dispatch from those records into named +mechanisms and must fail if a required datatype lowering is absent. + +## Failure Rules + +| Failure | Stage that should reject it | +| --- | --- | +| Compiler cannot measure a required target fact | probe service | +| Native declaration is syntactically unsupported | parser | +| Native fact cannot map to a stable semantic datatype | source-to-IR conversion | +| Datatype is known but unsafe or unsupported in its use-site context | policy completion | +| Completed datatype/policy combination has no plan representation | wrapper planner | +| Planned datatype has no backend mechanism | codegen checked dispatch | +| Runtime value has the wrong exact dtype, rank, layout, or mutability | generated binding validation | + +No stage should silently replace a failed mapping with a nearby width, host +default, NumPy coercion, or different ownership path. + +## Change Workflow And Evidence + +When adding or changing a datatype: + +1. Add parser coverage for every accepted source spelling and source location. +2. Add probe coverage when storage or kind depends on the compiler target. +3. Add or update `SemanticScalarSpec` only for stable semantic vocabulary. +4. Verify source-to-IR conversion records the resolved dtype and native + provenance. +5. Complete use-site behavior in policy and add explicit blockers for + unsupported combinations. +6. Extend wrapper-plan records only when existing transfer/result records + cannot represent the completed behavior. +7. Add one backend catalogue entry or a specialized lowering mechanism. +8. Add generated-source assertions and an end-to-end runtime case when emitted + behavior changes. +9. Update the semantic datatype reference, feature matrix, and this page when + support boundaries change. + +Primary evidence owners are: + +| Concern | Tests | +| --- | --- | +| Target measurement | `tests/fortran/data_types/probes/` | + +| Semantic scalar catalogue and conversion | `tests/fortran/data_types/semantics/`, semantic conversion tests | +| Public contract factories | semantic `.pyi` contract tests | +| Backend scalar catalogue | `tests/fortran/data_types/codegen/` | +| Generated datatype report | `tests/fortran/data_types/pipeline/test_type_mapping_report.py` | +| Runtime scalar and array behavior | feature-local end-to-end datatype and array tests | + +The report and registry tests should assert readable representative mappings, +not preserve obsolete public helpers or duplicate every internal dictionary as +an external API. diff --git a/docs/developer/contributing/contribution-guide.md b/docs/developer/contributing/contribution-guide.md deleted file mode 100644 index e457feaea..000000000 --- a/docs/developer/contributing/contribution-guide.md +++ /dev/null @@ -1,29 +0,0 @@ ---- -title: Contribution Guide -audience: contributors -prerequisites: repository checkout -related: pull-request-workflow.md, ../index.md -status: maintained -publication: reviewed ---- - -# Contribution Guide - -The root [contribution guide](../../../CONTRIBUTING.md) defines the current -submission and verification requirements. - -## Contribution license - -prik is distributed under the MIT License. Contributions are accepted under -the same MIT terms. By submitting a contribution, a contributor agrees to -license it under those terms and represents that they have the right to do so. -Contributors whose work is owned by an employer or another organization must -obtain authorization before submitting it. - -## Change workflow - -Start by identifying the public behavior and its owning stage. Update the -relevant documentation before implementation, then change the implementation -and focused tests together. The [development workflow](../development-workflow.md) -maps common changes to their required evidence, and the -[pull request workflow](pull-request-workflow.md) covers submission and review. diff --git a/docs/developer/contributing/index.md b/docs/developer/contributing/index.md deleted file mode 100644 index aeb232853..000000000 --- a/docs/developer/contributing/index.md +++ /dev/null @@ -1,25 +0,0 @@ ---- -title: Contributing -audience: contributors -prerequisites: repository checkout -related: ../index.md, ../../../CONTRIBUTING.md -status: planned-documentation -publication: draft ---- - -# Contributing - -This section will collect contribution requirements and link to detailed -developer workflows. - -## Pages - -- [Contribution guide](contribution-guide.md) -- [Pull request workflow](pull-request-workflow.md) -- [Coding standards](../coding-standards.md) -- [Review process](review-process.md) - -## TODO - -- TODO: Keep contributor-facing rules separate from repository governance. -- TODO: Keep this section synchronized with `../../../CONTRIBUTING.md`. diff --git a/docs/developer/contributing/pull-request-workflow.md b/docs/developer/contributing/pull-request-workflow.md deleted file mode 100644 index d50c08007..000000000 --- a/docs/developer/contributing/pull-request-workflow.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -title: Pull Request Workflow -audience: contributors -prerequisites: contribution guide -related: review-process.md, ../quality-assurance.md -status: planned-documentation -publication: draft ---- - -# Pull Request Workflow - -Reserved contributor page for branch preparation, tests, static analysis, -review, and merge expectations. - -## TODO - -- TODO: Document required local checks and CI gates. -- TODO: Add documentation update expectations for public behavior changes. diff --git a/docs/developer/contributing/review-process.md b/docs/developer/contributing/review-process.md deleted file mode 100644 index ebead7cb4..000000000 --- a/docs/developer/contributing/review-process.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -title: Review Process -audience: developers, contributors -prerequisites: pull request workflow -related: pull-request-workflow.md, ../testing-strategy.md -status: planned-documentation -publication: draft ---- - -# Review Process - -Reserved contributor page for review expectations, requested changes, support -evidence, and documentation completeness. - -## TODO - -- TODO: Document review criteria for code, tests, docs, and architecture. -- TODO: Link feature review to language support and roadmap updates. diff --git a/docs/developer/c-parser-reference.md b/docs/developer/deferred/c-parser.md similarity index 97% rename from docs/developer/c-parser-reference.md rename to docs/developer/deferred/c-parser.md index b0531e4de..eaa4e9169 100644 --- a/docs/developer/c-parser-reference.md +++ b/docs/developer/deferred/c-parser.md @@ -1,9 +1,9 @@ --- # PRIK_C_DOCS: title: C Parser Reference -title: Deferred Parser Reference -audience: developers -prerequisites: repository structure, parser architecture -related: adding-a-feature.md, repository-structure.md +title: Deferred C Parser Reference +audience: developers, maintainers, contributors +prerequisites: contributor architecture guide +related: ../architecture.md, ../packages/parsers.md, ../packages/semantics.md status: maintained publication: draft --- @@ -100,8 +100,7 @@ PRIK_C_DOCS_END --> - `prik.parsers.c` package - typed C parser models for partial parse reports and raw metadata - `CParser`, `parse_c_file`, and `parse_c_project` -- top-level `prik.parse_c_file` and `prik.parse_c_project` exports alongside - the `prik.parsers.c` package entrypoints +- `prik.parsers.c` package entrypoints for C parser models and operations - `CParseError` with compiler-style diagnostic formatting - explicit `prik --language c --parse` output - explicit `prik --language c --semantics` output @@ -154,7 +153,7 @@ PRIK_C_DOCS_END --> generated by the shared prik CLI - compiler-derived target ABI probing for every modeled arithmetic primitive, `size_t`, `uint32_t`, `time_t`, and opaque `FILE` handles through - `prik.probes.c_types`, with reusable memory and persistent caches + `prik.preprocessing.probes.c_types`, with reusable memory and persistent caches - C directory/file-list discovery for `.c`, `.h`, and direct `.i` inputs in explicit C mode, while leaving Fortran directory scanning unchanged - include resolution for quoted includes relative to the current file and @@ -457,21 +456,18 @@ PRIK_C_DOCS_END --> ## Public API -Implemented top-level and package entrypoints: +Implemented package entrypoints: @@ -594,15 +590,14 @@ declaration. Duplicate initialized variables, duplicate function definitions, duplicate complete tag definitions, and incompatible top-level redeclarations produce diagnostics. Local declarations inside function bodies are ignored because body contents are intentionally skipped. -`prik` exports the C file/project entrypoints in the same style as the -Fortran entrypoints. The typed C parser package remains importable directly. +The typed C parser package owns its file and project entrypoints directly. PRIK_C_DOCS_END --> Example: parse one header from Python. ## Testing Workflow @@ -1190,9 +1185,9 @@ PRIK_C_DOCS_END --> **Status:** This is a long-term architecture document, not a statement that diff --git a/docs/maintainer/design/wrapper-design-notes.md b/docs/developer/design/wrapper-open-decisions.md similarity index 99% rename from docs/maintainer/design/wrapper-design-notes.md rename to docs/developer/design/wrapper-open-decisions.md index 85d137474..14a476cbc 100644 --- a/docs/maintainer/design/wrapper-design-notes.md +++ b/docs/developer/design/wrapper-open-decisions.md @@ -1,13 +1,13 @@ --- -title: Wrapper Design Notes -audience: maintainers +title: Wrapper Open Decisions +audience: developers, maintainers, contributors prerequisites: Fortran wrapper reference, semantic IR reference -related: overall-architecture.md, ../internal-architecture/wrapper-generation-pipeline.md +related: ../architecture.md, ../packages/policy.md, ../packages/planning.md, multilanguage-runtime.md status: design publication: draft --- -# Wrapper Design Notes +# Wrapper Open Decisions Reference details live in: -- `docs/developer/fortran-parser-reference.md` +- `docs/developer/packages/parsers.md` - `docs/user/reference/fortran-wrapper.md` - `docs/user/reference/semantic-ir.md` ## Known Semantic Gaps To Track diff --git a/docs/developer/development-workflow.md b/docs/developer/development-workflow.md deleted file mode 100644 index e7c31b72a..000000000 --- a/docs/developer/development-workflow.md +++ /dev/null @@ -1,1585 +0,0 @@ ---- -title: Development Workflow -audience: developers, contributors -prerequisites: repository checkout, Python 3.10 or newer -related: index.md, quality-assurance.md -status: maintained -publication: draft ---- - -# Development Workflow - -This guide is for changing prik. It maps user-visible behavior to its owning -implementation and tests, then gives focused change and verification -workflows. - - - -## Start Here - -Install the project and QA dependencies: - -```bash -python3 -m pip install -e ".[qa]" -``` - -Run the smallest relevant test while iterating, then run the full suite: - -```bash -PYTHONPATH=. python3 -m pytest -q tests/fortran/command_line_interface/pipeline/ -PYTHONPATH=. python3 -m pytest -q -``` - -Before changing a public behavior, trace it through these layers: - - - -For example, a new CLI stage option normally requires: - -1. A focused contract test in `tests/fortran/command_line_interface/pipeline/`. -2. Dispatch or output routing in `prik/cli.py`. -3. Preprocessing tests if the option changes source loading. -4. A copy-paste command in the relevant user guide or checked example. -5. A tutorial update only when the main user workflow changes. - -## Support Evidence Rule - -Documentation must describe implemented behavior, not intended behavior. -Treat a support claim as established only when it is traceable to current -implementation plus one of these forms of evidence: - -- a focused test that proves the contract; -- a maintained fixture test that proves generated output; -- a repository command that has been run against a checked fixture; -- an explicit parser or semantic reference inventory backed by tests. - -Use these documentation roles consistently: - -| Document | Role | -| --- | --- | -| [Getting Started](../user/getting-started/index.md) | Main supported user workflow and boundaries | -| [Examples Gallery](../user/examples/index.md) | Checked commands and Python API recipes | -| [Fortran wrapper reference](../user/reference/fortran-wrapper.md) | Implemented Fortran runtime contract, mechanism, ownership, and build modes | -| [Fortran parser reference](fortran-parser-reference.md) | Developer inventory for the Fortran frontend | -| [Semantic IR reference](../user/reference/semantic-ir.md) | Accepted semantic IR and datatype contract | -| [Semantic .pyi format](../user/reference/semantic-pyi-format.md) | User-visible semantic `.pyi` syntax and roadmap | - - - -When adding a user example: - -1. Prefer a checked repository fixture or a short inline source string. -2. Run the command or snippet from the repository root. -3. Add or identify the focused test that owns the behavior. -4. State limitations next to the example when metadata is preserved but not - executed, such as `@native_call` projection metadata. - - - -### Automatically Verify Markdown Examples - -`tests/docs/test_examples.py` executes explicitly marked -`bash` CLI examples and `python` API snippets from `README.md` and Markdown -files under `docs/`. Bash examples must be `python3 -m prik` commands; the test replaces `python3` -with the active test interpreter and runs them without a shell. It rejects -shell operators, output-writing options, and options that select custom -executables or preprocessing command templates. Python snippets run with the -active test interpreter. - -Wrapper examples that need native compilation should use -`build_fortran_extension` with `TemporaryDirectory` so verification does not -leave build artifacts in the checkout. - -Mark a command that only needs to exit successfully: - -````markdown - -```bash -python3 -m prik semantics tests/fortran/source_parsing/parsing/fixtures/general/basic_subroutine.f90 -``` -```` - -Mark a command whose stdout must match the documentation exactly: - -````markdown - -```bash -python3 -m prik parse tests/fortran/source_parsing/parsing/fixtures/general/basic_subroutine.f90 -``` - - -```text -File: tests/fortran/source_parsing/parsing/fixtures/general/basic_subroutine.f90 -... -``` -```` - -Use exact checks for stable human-readable output. Use run checks for large -JSON or semantic payloads whose detailed contract is already covered by -focused tests. The same markers can precede a `python` fenced block. Do not -mark placeholder commands, snippets that modify the checkout, -environment-dependent compiler recipes, or intentionally failing diagnostic -examples. - -When a command reads a checked fixture, include its source input in the user -documentation and verify the displayed source against the fixture: - -````markdown - -```fortran -module m1 -... -end module m1 -``` -```` - -Append a target profile to an exact marker only for compiler-generated output -that is intentionally architecture-specific: - -```markdown - -``` - -Off-target checks are skipped. The matching profile must still run the command -and compare its complete output. - -Run the documentation checks directly: - -```bash -PYTHONPATH=. python3 -m pytest -q tests/docs/test_examples.py -``` - -## References - -- [Getting Started](../user/getting-started/index.md): supported end-to-end user - workflow and current boundaries. -- [Examples Gallery](../user/examples/index.md): checked CLI and Python API - recipes. -- [Fortran parser reference](fortran-parser-reference.md): Fortran frontend scope, - recursive parser organization, API/CLI behavior, diagnostics, fixture - workflow, semantic handoff, and tests. -- [Semantic `.pyi` format](../user/reference/semantic-pyi-format.md): user-visible `.pyi` - loader/printer contract and roadmap. -- [Quality assurance](quality-assurance.md): active QA commands, tool benefits, known - defects found by each tool, and scheduled triage process. - - - -## User-Facing Contract Internals - -The tutorial, examples cookbook, `.pyi` format, and semantic reference describe -CLI stages, `.pyi` syntax, datatype names, and wrapper-plan diagnostics. The developer -task is to keep those user-visible contracts stable, tested, and traceable to -implementation files. - -### Source Ownership Map - -| User-visible area | Main implementation files | Main tests | -| --- | --- | --- | -| Fortran parse output | `prik/parsers/fortran/parser.py`, `prik/parsers/fortran/models.py`, `prik/parsers/fortran/lexer.py` | `tests/fortran/source_parsing/parsing/`, `tests/fortran/source_parsing/parsing/test_fortran_fixture_suite.py`, `tests/fortran/source_parsing/parsing/test_error_handling.py` | -| CLI stage selection and output | `prik/cli.py`, `prik/parsers/fortran/cli.py` | `tests/fortran/command_line_interface/pipeline/` | -| Fortran target type probing and cache | `prik/probes/fortran_types.py` | `tests/fortran/data_types/probes/test_fortran_type_probes.py` | -| Generated target datatype mapping examples | `prik/probes/report.py` | `tests/fortran/infrastructure/types/test_mapping_report.py`, `tests/docs/test_examples.py` | -| Fortran to semantic IR | `prik/semantics/fortran2ir.py`, `prik/semantics/models.py` | `tests/fortran/semantic_ir/semantics/` | -| `.pyi` printing | `prik/codegen/printers/pyi_printer.py` | `tests/fortran/semantic_pyi_format/pipeline/`, `tests/fortran/semantic_pyi_format/pipeline/test_modern_example.py` | -| `.pyi` parsing/loading/editing | `prik/parsers/pyi/parser.py`, `prik/pipeline/pyi.py`, `prik/semantics/pyi2ir.py` | `tests/fortran/semantic_pyi_format/` | -| Semantic policy completion | `prik/semantics/policy_completion.py`, `prik/semantics/ownership.py` | `tests/fortran/infrastructure/semantics/` and feature-local `policy/` directories | -| Fortran wrapper orchestration | `prik/pipeline/build.py` | `tests/fortran/building_shared_library/end_to_end/test_source_build_modes.py`, `tests/fortran/building_shared_library/end_to_end/test_multi_source_builds.py` | -| Wrapper planning, owner-local errors, and direct lowering | `prik/codegen/plan.py`, `prik/codegen/planner.py`, `prik/codegen/generator.py` | `tests/fortran/infrastructure/codegen/`, feature-local `codegen/` stages | -| Native compilation and binding support | `prik/compiling/`, `prik/binding_support/` | `tests/fortran/building_shared_library/end_to_end/test_runtime_compatibility.py`, `tests/fortran/building_shared_library/end_to_end/test_source_build_modes.py` | -| Executable Markdown examples | `README.md`, `docs/*.md` | `tests/docs/test_examples.py` | - - - -### Wrapper Generator Class Organization - - - -Organize generators and printers using `FortranParser` in -`prik/parsers/fortran/parser.py` as the structural reference. A developer -should be able to read each class from top to bottom in the same order that -data moves through it: - -1. The class docstring states the class's responsibility and lists its method - sections. -2. Construction and public entrypoints come first. -3. Dispatched model handlers follow, grouped by feature and pipeline order. - Their names use the class's configured visitor prefix, for example - `_visit_`, `_print_`, or `_parse_`. -4. Helpers immediately follow the visitor group that owns them, or appear in - a final low-level helper section when several visitor groups share them. -5. Every method has a short contract docstring. The docstring explains the - method's purpose or invariant; it does not restate its name. - -Use the same visible section banners as `FortranParser`, for example -`Public entrypoints`, `Module visitors`, `Function visitors`, and `Shared -helpers`. Keep related visitors adjacent instead of sorting methods merely by -name. - -All model-type dispatch goes through `prik.utilities.visitor.ClassVisitor._visit` and a -matching `_` handler. Parser-model converters, semantic -lowering, `.pyi` AST visitors, bridges, bindings, and printers share that one -implementation; do not duplicate its MRO lookup in an individual class. - -An explicit table is allowed only for a genuine second dispatch dimension, -such as a completed policy action or primitive ABI datatype mapping. Such a -table must not replace model-class visitation. Do not add a second independent -visitor family, `visit_`, or scattered `isinstance` dispatch -schemes. -A method that performs ordinary work but is not a dispatch target must have a -descriptive helper name rather than a visitor-shaped name. - -Keep functionality on the class that owns its state and policy. A module-level -function is justified only when it is a deliberate public functional API or a -genuinely stateless utility shared by unrelated classes. Do not retain a -module-level function only to preserve an old internal call path. - -### `.pyi` Contract Internals - -User-visible `.pyi` syntax is first parsed to Python AST by -`prik/parsers/pyi/parser.py`, loaded from text/files by -`prik/pipeline/pyi.py`, converted to semantic IR by -`prik/semantics/pyi2ir.py`, and printed by -`prik/codegen/printers/pyi_printer.py`. The converter and printer operate on -`prik/semantics/models.py`. - -Important implementation rules: - -- `Addr(T)` and `Addr(T)` are storage contracts, not just pretty syntax. -- Array subscriptions such as `Float64[n]` are semantic array contracts. -- `Annotated[..., ORDER_F]` and `ORDER_ANY` are non-default array storage - metadata. Plain multidimensional Fortran `.pyi` arrays use `ORDER_F`; do not - print or retain that default marker in a generated contract. - `Allocatable[T[...]]` and `Pointer[T[...]]` are descriptor-handle wrappers - around the array storage contract. Output and writeback behavior is - represented by writable storage plus `Returns["name", T]` when a Python - result is projected. - -- `Final[T]` is the public constant spelling. Do not reintroduce - `Constant` as user-facing `.pyi` syntax. -- `@native_call` is projection metadata. Use it only when the Python-visible - signature intentionally differs from the native signature. -- Generated stubs should preserve behavior-changing native contracts while - staying compact; exact source intent that does not change execution can stay - in semantic IR instead of the printed `.pyi`. -- Use `SourceName("...")` only when a source identifier cannot be used as the - Python target. Do not infer source identifiers from normalized Python names. -- Binding locals derived from a Python-visible argument must use the reserved - `bound_` namespace. Generated binding sources include Python, standard-library, - optional descriptor, NumPy, and runtime headers, so their imported identifier - sets are not a stable public-name vocabulary. -- Omit `Polymorphic` only for the passed-object dummy of a type-bound procedure, - where the binding itself restores that native fact. Ordinary `class(T)` - arguments must retain it. - -When changing `.pyi` syntax: - -1. Add or update parser tests in `tests/fortran/semantic_pyi_format/parsing/`. -2. Add or update printer tests in `tests/fortran/semantic_pyi_format/pipeline/`. -3. Update fixture tests only if the public generated contract changes. -4. Update the relevant [User Guide](../user/guide/index.md) or checked - [example](../user/examples/index.md) if users need to write or read the new - syntax. -5. Update [Semantic .pyi format](../user/reference/semantic-pyi-format.md) for the full user-facing reference. -6. Update [Semantic IR reference](../user/reference/semantic-ir.md) if the underlying semantic IR contract - changes. - -### Datatype Mapping Internals - -User-visible datatype names are semantic names, not raw parser spellings. -Mapping happens during parser-to-IR conversion: - -- Fortran intrinsic/kind mapping and compiler storage-fact application live in - `prik/semantics/fortran2ir.py`. -- The shared dtype names and storage contracts live in `prik/semantics/models.py`. -- Compiler-measured mapping snapshots are generated by - `prik/probes/report.py`. - - - -When changing datatype mapping: - -1. Add focused Fortran conversion tests in - `tests/fortran/semantic_ir/semantics/`. -2. Add `.pyi` printer/loader coverage if the emitted syntax changes. -3. Update semantic fixtures only when serialized semantic IR intentionally - changes. -4. Update [Semantic IR reference](../user/reference/semantic-ir.md), plus the - relevant [User Guide](../user/guide/index.md) or checked - [example](../user/examples/index.md) when the visible user workflow or - examples change. -5. Regenerate and update the exact target mapping snapshots in - [Semantic IR reference](../user/reference/semantic-ir.md). The executable documentation test must match - the complete output of: - - - - ```bash - python3 -m prik probe --language fortran --compiler gfortran --format markdown - ``` - - - -For Fortran, keep both modern and legacy spellings in the generated report. -Legacy numeric `type*N` forms carry fixed total storage; compiler-dependent -default, kind, `DOUBLE PRECISION`, and `DOUBLE COMPLEX` forms use probe facts. - -### Error Ownership - -Diagnostics belong to the earliest stage that has enough facts to explain the -failure. Parsers report source syntax and preprocessing faults. Semantic -conversion reports facts that cannot form a valid contract. Policy completion -records every lowering decision; the wrapper planner reports an unsupported -completed policy with its owner path. Add focused tests to that owning stage, -and update the relevant user guide when a user can correct the input or -contract. - -### Parser To Wrapper Boundary - -Do not move wrapper policy into parsers. Parsers can preserve: - -- source locations; -- declaration and signature facts; -- type, pointer, array, callback, and aggregate facts; -- preprocessor provenance and diagnostics; -- unresolved references. - -Post-IR policy completion and wrapper planning decide: - -- ownership and lifetime; -- callback registration/unregistration policy; -- output-buffer projection; -- hidden pointer/size projection; -- ABI shim requirements; -- Python-visible signature adaptation. - -## Pipeline Internals - -The user-facing stages all start in `prik/cli.py`, but each stage owns a -different layer of the pipeline. - - - -### CLI And Language Resolution - -`prik/cli.py` is the shared command-line entrypoint. It is responsible for: - -- rejecting ambiguous directories and unknown suffixes without `--language`; -- building `PreprocessingConfig`; -- dispatching `parse`, `semantics`, `generate`, and `probe`; -- defaulting recognizable Fortran sources to a wrapper build when no - subcommand is selected; -- routing the default build and `generate --sources|--makefile` through - `prik/pipeline/build.py`; -- routing text, JSON, and `--out` output. - - - - - -The package-specific `prik/parsers/fortran/cli.py` remains for the Fortran parser -package entrypoint. New cross-language user behavior normally belongs in -`prik/cli.py`. - -### Preprocessing Internals - -`prik/pipeline/preprocessing.py` owns compiler-backed preprocessing and provenance. The -main value object is `PreprocessingConfig`; the main execution path is -`run_compiler_preprocessor_with_recipe(...)`. - -Important contracts: - -- The preprocessing recipe is part of the parser payload when preprocessing - happened. It records compiler, adapter, argv, include directories, defines, - undefs, standard, extra compiler args, included files, source mappings, and - diagnostics. - - - - - -### Source Loading To Semantic IR Paths - -Keep source loading, parser models, and semantic conversion separate. Semantic -converters accept parsed models; they must not hide compiler preprocessing or -source loading inside conversion helpers. - -Fortran direct Python API, no CPP/FPP macros: - -```python -from prik import parse_fortran_file -from semantics.fortran2ir import fortran_module_to_semantic_module - -parsed = parse_fortran_file(source, filename="visibility_mod.f90") -semantic = fortran_module_to_semantic_module(parsed.modules[0]) -``` - -`parse_fortran_file(...)` runs the parser's internal line preparation: -source-form detection, comment stripping, and continuation folding. It does -not expand `#define`, `#ifdef`, or other CPP/FPP directives. Raw CPP/FPP -directives are rejected with `PARSE_PREPROCESSING_REQUIRED`. - -Fortran with macros or textual configuration must be compiler-preprocessed -before parsing: - -```python -from pathlib import Path - -from prik import parse_fortran_file -from semantics.fortran2ir import fortran_file_to_semantic_modules -from prik.pipeline.preprocessing import PreprocessingConfig, preprocess_source - -path = Path("configured.F90") -preprocessed = preprocess_source( - path, - language="fortran", - config=PreprocessingConfig( - mode="compiler", - compiler="gfortran", - defines=["USE_MPI", "N=32"], - include_dirs=["include"], - ), -) - -parsed = parse_fortran_file(preprocessed.source, filename=str(path)) -modules = fortran_file_to_semantic_modules(parsed) -``` - -Choose the Fortran semantic helper from the parser model shape: - -- `fortran_module_to_semantic_module(parsed.modules[0])` for one selected - module. -- `[fortran_module_to_semantic_module(m) for m in parsed.modules]` when a file - contains multiple modules and no top-level standalone procedures matter. -- `fortran_file_to_semantic_modules(parsed, standalone_module_name=...)` when - top-level procedures should become a synthetic semantic module too. -- `fortran_project_to_semantic_modules(project)` when project-level module and - derived-type context matters. - -Fortran `parameter` values and kind expressions are not CPP macros. If the -parser leaves a Fortran compile-time expression symbolic, collect missing -values with `collect_semantic_compile_time_requirements(parsed)`, evaluate -them with the target compiler or a reusable type report, and pass -`compile_time_values=...` to the semantic converter. The shared CLI semantic -stage performs this target probing when a Fortran compiler or report is -configured; direct API callers must do it explicitly. - - - - - - - - - - - - - - - -### Semantic, `.pyi`, Wrapper-Planning, And Type-Probe Paths - - - -Input shapes are part of the contract: - -- `parse_fortran_file(source_or_path, filename=...)` accepts inline source - text. It reads from disk only when `source_or_path` names an existing file - and `filename` is omitted. Pass `filename` with inline text for diagnostic - provenance. -- `preprocess_source(path, language=..., config=...)` is path-based because it - shells out to a compiler. Feed `preprocessed.source` to the parser afterward. -- `parse_pyi_text(...)` accepts inline `.pyi` source text and returns Python - AST. `convert_pyi_to_ir(...)` converts that parsed AST to semantic IR. - `pyi_text_to_semantic_module(...)`, `pyi_file_to_semantic_module(...)`, and - `pyi_paths_to_semantic_modules(...)` combine parsing and conversion for - inline text, one file, or a file set. -- The CLI accepts source, `.pyi`, and directory paths. It does not accept - inline source text on the command line. - - - -CLI source stages: - - - -CLI `.pyi` wrapper build: - -```text -.pyi path(s) or directory - -> prik/parsers/pyi/parser.py - -> prik/pipeline/pyi.py pyi_paths_to_semantic_modules(...) - -> prik/semantics/pyi2ir.py - -> SemanticModule list - -> prik/semantics/policy_completion.py - -> complete_semantic_policies(...) - -> WrapperPlanner.build(...) -``` - -Generating `.pyi` from source is semantic conversion plus printing. In Python -API code, keep those calls visible: - -```python -from prik import emit_module_stubs, parse_fortran_file -from semantics.fortran2ir import fortran_file_to_semantic_modules - -parsed = parse_fortran_file(source, filename="api.f90") -modules = fortran_file_to_semantic_modules(parsed) -stubs = emit_module_stubs(modules) -``` - - - -Loading or editing `.pyi` is the opposite direction: - -```python -from prik import pyi_paths_to_semantic_modules - -modules = pyi_paths_to_semantic_modules("interfaces") -``` - -Use the `.pyi` helpers by input shape: - -- `parse_pyi_text(source, filename=...)` from `prik.parsers.pyi` for parser-only - AST parsing. -- `convert_pyi_to_ir(tree, module_name=..., source=...)` from `pyi2ir.py` for - AST-to-IR conversion. -- `pyi_text_to_semantic_module(source, module_name=..., filename=...)` from - `pyi_pipeline.py` for inline text. -- `pyi_file_to_semantic_module(path, module_name=...)` for one file. -- `pyi_paths_to_semantic_modules(paths_or_directory)` for a set of interfaces - that may reference each other. - -The `.pyi` pipeline uses a per-operation in-memory conversion cache. Wrapper -entry-contract discovery reuses the same converted modules when it later builds -the reconciled contract bundle, so an imported file is not parsed and converted -twice in one build. Do not make this cache process-global: semantic modules are -mutated by reconciliation, export selection, and policy completion. - - - -Compiler preprocessing flags all flow through `PreprocessingConfig`: - -| CLI flag | `PreprocessingConfig` field | Notes | -| --- | --- | --- | -| `--compiler` | `compiler` | Exact executable for direct preprocessing and automatic type probes. | -| `--preprocessor-adapter` | `adapter` | Adapter family, including `command-template`. | -| `--preprocess-template` | `command_template` | Custom command; requires `--preprocessor-adapter command-template`. | -| `-I` / `--include-dir` | `include_dirs` | Passed to compiler preprocessing and native Fortran include expansion. | -| `-D` / `--define` | `defines` | Macro definitions for compiler preprocessing. | -| `-U` / `--undef` | `undefs` | Macro undefinitions for compiler preprocessing. | -| `--std` | `std` | Passed as `-std=...`. | -| `--compiler-arg` | `compiler_args` | Raw target/sysroot/compiler options. | -| `--public-include`, `--private-include`, `--include-exposure` | include exposure fields | Controls provenance exposure, not parser grammar. | - - - - - - - - - - - -Fortran target datatype mapping and compile-time path: - -```text -Fortran source - -> parse_fortran_file(...) - -> collect_semantic_compile_time_requirements(...) - -> evaluate_fortran_type_requirements(...) - -> collect_fortran_type_storage_requirements(...) - -> evaluate_fortran_type_facts(...) - -> fortran_module_to_semantic_module(..., compile_time_values=..., type_facts=...) -``` - - - - - -### Fortran Runtime Wrapper Path - -`prik/pipeline/build.py::build_fortran_extension(...)` and -`prik/pipeline/build.py::build_pyi_extension(...)` are the public orchestration -boundaries for wrapper builds. Keep their stages explicit: - -```text -ordered source paths - -> preprocess_source(..., language="fortran") - -> parse_fortran_project(...) - -> compile-time expression and storage probes - -> fortran_project_to_semantic_modules(...) - -> merge public semantic modules - -> WrapperPlanner and WrapperCodeGenerator - -> create_shared_library(...) - -> WrapperBuildResult -``` - -The main ownership boundaries are: - -- `prik/pipeline/build.py`: source order, preprocessing/probing, semantic merge, - `.pyi` entry-contract loading, native build plan assembly, output placement, - direct-versus-Makefile mode, and artifact reporting; -- `prik/codegen/planner.py`: projection from completed semantic policy - into validated typed plans; -- `prik/codegen/generator.py`: direct bridge, binding, and source - artifact generation; -- `prik/compiling/`: compiler commands and shared-library linking; and -- `prik/binding_support/`: native binding support copied into each build. - - - -Do not move semantic ownership or projection policy into printers. Do not infer -source dependencies: multi-source source builds compile in caller order, and -the first semantic module names the merged extension. `.pyi` builds use exactly -one semantic entry contract plus a separate extension-level -`NativeBuildPlan`; they must not recover Python API facts by reparsing native -implementation sources. `--makefile` records the compiler/linker plan -without executing it; for `.pyi` builds, `prik-build.json` is written first and -`Makefile.prik` is projected from that manifest. - - - - - -Runtime verification belongs under the relevant -`tests/fortran//end_to_end/` owner. The -[`tests/fortran` index](../../tests/fortran/README.md) and permanent -[contract ledger](../../tests/fortran/CONTRACT_COVERAGE.md) map generated -behavior to compiled/imported tests. Build-mode changes should at least cover -`tests/fortran/building_shared_library/end_to_end/test_source_build_modes.py`, -`tests/fortran/building_shared_library/end_to_end/test_multi_source_builds.py`, -and the affected runtime subject test. - -### Parser Model Internals - -Parser models are source facts. They should answer "what did the source say?" -rather than "what Python wrapper should be generated?" - -Fortran: - -- `prik/parsers/fortran/parser.py` slices the file into grammar units, then parses - each unit's specification region. -- `prik/parsers/fortran/models.py` stores `FortranFile`, modules, procedures, - variables, derived types, interfaces, programs, submodules, and diagnostics. -- Execution bodies are intentionally skipped after the parser has enough - signature/source facts. - - - - - -Adding parser fields is a schema decision. Add fields only when downstream -semantic conversion, fixtures, diagnostics, or user-visible behavior need a -new fact. - -### Semantic IR Internals - -The semantic layer normalizes Fortran facts into language-neutral models from -`prik/semantics/models.py`. - - - -- `prik/semantics/fortran2ir.py` maps Fortran procedures, derived types, module - variables, kinds, shapes, storage contracts, visibility, imported references, - and compile-time values. -- `prik/codegen/printers/pyi_printer.py` emits editable user contracts. -- `prik/parsers/pyi/parser.py` parses edited contracts to Python AST. -- `prik/pipeline/pyi.py` converts edited contract text, files, and path sets. -- `prik/semantics/pyi2ir.py` converts parsed `.pyi` AST back into semantic IR. -- `prik/semantics/native_contract.py` validates immutable native scope, ABI, - placement, type, callback, and projection facts before source-free codegen. -- Named data bindings keep role-specific semantic types: `SemanticVariable` - for module variables and constants, `SemanticArgument` for callable - parameters, and `SemanticField` for Fortran derived-type components. -- `prik/semantics/policy_completion.py` completes semantic policies after - Fortran or `.pyi` conversion and before wrapper planning or lowering. - - - - - -Keep semantic IR stable where possible. If a parser change does not affect the -semantic contract, avoid changing semantic fixtures. - -### `.pyi` Projection Internals - -`@native_call` is stored as projection metadata on `SemanticFunction`. The -loader and printer currently support `Arg`, `Return`, ABI-typed literal calls -such as `Int32(1)`, `Len`, `IsPresent`, `Work`, `Pass`, and `.shape[...]` -value references. Generated Fortran contracts use it when outputs make the -Python-visible argument order differ from native order. `Pass()` preserves the -hidden passed object when a type-bound method also needs such a projection. They do not currently -implement future wrapper projection helpers such as `Addr(Arg(...))`, `As[...]`, -status-return policy, ownership conversion, or coercion execution. - -The test ownership is: - -- loader syntax and error behavior: `tests/fortran/semantic_pyi_format/parsing/`; -- printer round-trip shape: `tests/fortran/semantic_pyi_format/pipeline/`; -- policy-completion decisions: `tests/fortran/infrastructure/semantics/` and feature-local `policy/` directories; -- wrapper-plan diagnostics: `tests/fortran/infrastructure/codegen/`. - - - -When adding projection syntax, first add loader tests that prove the accepted -syntax and rejected syntax. Then add policy or wrapper-plan tests only if the -new metadata affects those layers. - -## Testing Strategy - -Use the smallest test layer that proves the behavior, then add broader -coverage only when the public contract changes. - -### Test Layers - -| Layer | Purpose | Typical files | -| --- | --- | --- | -| Focused parser tests | One construct, diagnostic, or model field | `tests/fortran/source_parsing/parsing/test_*.py` | -| Parser fixture goldens | Serialized Fortran parser contracts | `tests/fortran/source_parsing/parsing/test_fortran_fixture_suite.py` | -| Semantic tests | Fortran parser facts converted to wrapper-neutral IR | `tests/fortran/semantic_ir/semantics/` | -| Policy tests | Completed policy decisions | `tests/fortran/infrastructure/semantics/` and feature-local `policy/` directories | -| Wrapper-plan tests | Unsupported plan diagnostics and generated plan shape | `tests/fortran/infrastructure/codegen/` | -| `.pyi` tests | Editable contract loader/printer behavior | `tests/fortran/semantic_pyi_format/`, `tests/fortran/semantic_pyi_format/pipeline/` | -| CLI tests | User commands, output routing, diagnostics | `tests/fortran/command_line_interface/pipeline/`, `tests/fortran/source_preprocessing/preprocessing/` | -| Wrapper build tests | Artifact placement, direct/Makefile modes, multi-source ordering | `tests/fortran/building_shared_library/end_to_end/test_source_build_modes.py`, `tests/fortran/building_shared_library/end_to_end/test_multi_source_builds.py` | -| Wrapper runtime tests | Imported extension behavior, ownership, lifetime, and failures | Feature-local `tests/fortran/*/end_to_end/` suites indexed by `tests/fortran/README.md` | -| Property/fuzz tests | Broad parser robustness invariants | `tests/fortran/source_parsing/parsing/` and feature-local semantic property tests | - - - - - -### Choosing Tests For A Change - -- Parser-only source fact: focused parser test first; fixture golden only if - serialized output changes intentionally. -- CLI flag or output change: CLI test first; update README/user docs if the - visible command changes. -- New datatype mapping: semantic conversion test plus `.pyi` printer/loader - tests if emitted syntax changes. -- New `.pyi` syntax: loader and printer tests, plus policy or plan tests when - it changes a completed decision or lowering. -- New unsupported case: a semantic-conversion, policy, or wrapper-plan test at - the stage that detects it. -- Preprocessing behavior: preprocessing CLI tests and at least one parser path - that consumes the recipe. -- Wrapper orchestration or codegen behavior: the focused feature-local - `end_to_end/` or `codegen/` owner, including an imported runtime - assertion rather than build success alone. - -### Golden Fixture Rules - -Do not regenerate broad fixture sets to hide uncertainty. First write or run a -focused test that explains the intended behavior. Then regenerate only the -affected fixture group when the serialized contract really changed. - -Useful commands: - - - -### Coverage And CI Parity - -When investigating coverage failures, mirror the GitHub Actions coverage flow -instead of relying on a plain local run: - -```bash -COVERAGE_PROCESS_START=pyproject.toml PYTHONPATH=. coverage run -m pytest -python -m coverage combine -python -m coverage report -``` - -The `COVERAGE_PROCESS_START` environment variable matters because subprocess -CLI tests need the same coverage configuration as CI. - -## Feature Change Walkthroughs - -Use these walkthroughs when adding behavior. They are deliberately procedural: -change the smallest owned layer first, test that layer, then update downstream -contracts only when the public behavior actually changes. - - - - - - - - - - - - - - - -### Add A Fortran Parser Feature - -Example target: preserve a new declaration attribute, source fact, or argument -metadata item. - -1. Add a focused parser test in the file that owns the behavior: - `tests/fortran/source_parsing/parsing/`, - `tests/fortran/modules/parsing/test_scope_handling.py`, or - `tests/fortran/source_preprocessing/preprocessing/test_parser_boundaries.py`. -2. Implement parsing in `prik/parsers/fortran/parser.py`. Add model fields in - `prik/parsers/fortran/models.py` only if the parser output needs to expose the - new fact. -3. Add parser diagnostic coverage in `tests/fortran/source_parsing/parsing/test_error_handling.py` if - malformed source should now fail differently. -4. If project ordering, imports, or compile-time values change, update - `tests/fortran/modules/parsing/test_project_scope_models.py` or - `tests/fortran/data_types/probes/test_fortran_type_probes.py`. -5. If serialized parser JSON changes intentionally, regenerate the selected - fixture: - - ```bash - python tests/fortran/source_parsing/parsing/generate_parser_goldens.py tests/fortran/source_parsing/parsing/fixtures/general/basic_subroutine.f90 - ``` - -6. If the new fact affects semantic output, update `prik/semantics/fortran2ir.py` - and `tests/fortran/semantic_ir/semantics/`. -7. If generated `.pyi` changes, update `tests/fortran/semantic_pyi_format/pipeline/` - and the relevant fixture tests. -8. Update [Fortran parser reference](fortran-parser-reference.md), the relevant - [User Guide](../user/guide/index.md), checked - [example](../user/examples/index.md), or - [Semantic IR reference](../user/reference/semantic-ir.md) as needed. - -Focused verification: - -```bash -PYTHONPATH=. pytest -q tests/fortran/source_parsing/parsing/ -PYTHONPATH=. pytest -q tests/fortran/source_parsing/parsing/test_fortran_fixture_suite.py -PYTHONPATH=. pytest -q tests/fortran/semantic_ir/semantics/ -``` - -### Add Or Change Datatype Mapping - -Example target: map a new Fortran kind or compiler-probed storage fact. - - - -1. Add conversion coverage in `tests/fortran/semantic_ir/semantics/`. -2. Implement the mapping in `prik/semantics/fortran2ir.py`. -3. Keep the public semantic dtype names in `prik/semantics/models.py` stable unless - there is a deliberate schema decision. -4. If the emitted `.pyi` annotation changes, update - `tests/fortran/semantic_pyi_format/pipeline/` and - `tests/fortran/semantic_pyi_format/parsing/`. -5. Update the datatype tables in - [Semantic IR reference](../user/reference/semantic-ir.md), and update the - relevant [User Guide](../user/guide/index.md) or checked - [example](../user/examples/index.md) when a visible example changes. - - - - - -Focused verification: - -```bash -PYTHONPATH=. pytest -q tests/fortran/semantic_ir/semantics/ -PYTHONPATH=. pytest -q tests/fortran/semantic_pyi_format/pipeline/ tests/fortran/semantic_pyi_format/parsing/ -``` - - - -### Add `.pyi` Syntax Or Projection Behavior - -Example target: add a new `Annotated[...]` metadata item or projection helper. - -1. Add loader tests in `tests/fortran/semantic_pyi_format/parsing/`. -2. Update `prik/semantics/pyi2ir.py`. Update `prik/pipeline/pyi.py` - when loading or cross-file reconciliation changes. Update - `prik/parsers/pyi/parser.py` only when the raw Python AST parsing boundary - changes. -3. Add printer tests in `tests/fortran/semantic_pyi_format/pipeline/`. -4. Update `prik/codegen/printers/pyi_printer.py`. -5. Update semantic models in `prik/semantics/models.py` only if the IR needs a new - field or constraint. -6. Update policy completion or wrapper planning if the syntax changes a - completed decision. -7. Update [Semantic IR reference](../user/reference/semantic-ir.md), plus the - relevant [User Guide](../user/guide/index.md) or checked - [example](../user/examples/index.md) when users need the new syntax in a - workflow. - -Focused verification: - -```bash -PYTHONPATH=. pytest -q tests/fortran/semantic_pyi_format/parsing/ -PYTHONPATH=. pytest -q tests/fortran/semantic_pyi_format/pipeline/ -PYTHONPATH=. pytest -q tests/fortran/semantic_ir/semantics/ tests/fortran/infrastructure/codegen/ -``` - -### Add A Stage-Owned Error - -Example target: report a new unsupported Fortran semantic contract clearly. - - - -1. Preserve the source fact in the parser if it is not already present. -2. Raise a semantic-conversion error when no valid contract can be formed; do - not attach a deferred diagnostic payload. -3. If the source facts are valid but a selected wrapper behavior is unsafe, - express that result in completed policy and let the planner name the owner - path and reason. -4. Add a focused conversion, policy, or wrapper-plan test at that owning - stage. -5. Update the relevant user guide and [Error Handling](../user/guide/error-handling.md) - when users can correct the source or edited `.pyi` contract. - -Focused verification: - -```bash -PYTHONPATH=. pytest -q tests/fortran/semantic_ir/semantics/ -PYTHONPATH=. pytest -q tests/fortran/semantic_ir/semantics/ -PYTHONPATH=. pytest -q tests/fortran/infrastructure/codegen/ -``` - - - -### Add Or Change CLI Behavior - -Example target: add a stage option, change output routing, or improve -diagnostic formatting. - -1. Add CLI tests in `tests/fortran/command_line_interface/pipeline/` first. -2. Implement shared dispatch and output behavior in `prik/cli.py`. -3. Keep Fortran package-specific CLI behavior in `prik/parsers/fortran/cli.py`. -4. If compiler preprocessing behavior changes, update `prik/pipeline/preprocessing.py` - and preprocessing tests. -5. Update the relevant [User Guide](../user/guide/index.md) or checked - [example](../user/examples/index.md) for user-facing commands and this guide - for developer command maps. - -Focused verification: - -```bash -PYTHONPATH=. pytest -q tests/fortran/command_line_interface/pipeline/ -PYTHONPATH=. pytest -q tests/fortran/source_preprocessing/preprocessing/ -``` - -## Testing Map - -Use this map when changing one part of the project. Each section shows how to -call that part manually, which focused test file to run, and where to look for -more executable examples. Run the broader suite before merging. - -### Pre-Merge Checks - -Run the ordinary suite from the repository root before merging. Full -BLAS/LAPACK cases belong to their designated real-library lane: - -```bash -PYTHONPATH=. pytest -q -m "not real_library" \ - tests/c tests/docs tests/fortran tests/tools tests/workflows -``` - -Run the major suites individually while iterating: - -```bash -PYTHONPATH=. pytest -q tests/c -PYTHONPATH=. pytest -q -m "not real_library" tests/fortran -PYTHONPATH=. pytest -q tests/docs -PYTHONPATH=. pytest -q tests/tools -PYTHONPATH=. pytest -q tests/workflows -``` - -Maintainer-tool tests, workflow-safety tests, focused documentation smoke, one -compiled scalar-wrapper smoke test, and blocking static analysis run locally -before every push. Enable the tracked hook once in each clone: - -```bash -git config core.hooksPath .githooks -``` - -The hook runs static-analysis version validation, Ruff lint and formatting, -the codegen-complexity policy, Bandit, Vulture, the changed-code Radon policy, -the publication and user-content documentation smoke tests, one small public -CLI-to-native-call wrapper test, `tests/tools/`, and `tests/workflows/`. It -rejects the push on the first failure. GitHub Actions runs these checks again -as the shared enforcement boundary alongside the required product, complete -documentation, compiler, coverage, and real-library checks. The slower -documentation validators, verbose advisory Radon reports, and broader compiled -smoke matrix remain outside the quick local hook. - -As a project policy, do not merge pull requests unless all checks are green. - -### Fixture Maintenance - - - - - - - - - -Refresh all Fortran parser goldens: - -```bash -python tests/fortran/source_parsing/parsing/generate_parser_goldens.py -``` - -Refresh one Fortran fixture: - -```bash -python tests/fortran/source_parsing/parsing/generate_parser_goldens.py tests/fortran/source_parsing/parsing/fixtures/general/basic_subroutine.f90 -``` - -In-test Fortran parser fixture update mode: - -```bash -FORTRAN_PARSER_UPDATE_GOLDENS=1 PYTHONPATH=. pytest -q \ - tests/fortran/source_parsing/parsing/test_fortran_fixture_suite.py -``` - -Refresh semantic and `.pyi` fixtures: - -```bash -python tests/fortran/semantic_ir/semantics/generate_semantic_fixtures.py -WRAPPER_UPDATE_PYI_FIXTURES=1 python3 -m pytest -q tests/fortran/semantic_pyi_format/pipeline/test_contract_package_generation.py -``` - -When parser model output changes, include the regenerated parser goldens and a -short explanation in the PR. For `.pyi`, semantic IR, policy, or wrapper-planning behavior -changes, update the reviewed contracts under -`tests/fortran/semantic_pyi_format/pipeline/fixtures/contracts/` or -the semantic fixtures under `tests/fortran/semantic_ir/semantics/fixtures`. - - - - - - - - - - - - - - - - - - - - - -### Fortran Parser - -Manual call for one Fortran fixture: - -```bash -python -m prik parse tests/fortran/source_parsing/parsing/fixtures/general/basic_subroutine.f90 --language fortran --json -``` - -Manual Python API call: - -```python -from prik import parse_fortran_file - -parsed = parse_fortran_file( - "tests/fortran/source_parsing/parsing/fixtures/general/basic_subroutine.f90", -) -print([module.name for module in parsed.modules]) -``` - -Focused tests by concern: - -- Parser walkthrough: - `PYTHONPATH=. pytest -q tests/fortran/source_parsing/parsing/test_developer_tutorial.py` -- Procedures, declarations, derived types, and interfaces: - `PYTHONPATH=. pytest -q tests/fortran/source_parsing/parsing/` -- Scope and project behavior: - `PYTHONPATH=. pytest -q tests/fortran/modules/parsing/test_scope_handling.py tests/fortran/modules/parsing/test_project_scope_models.py` -- Preprocessing and execution-boundary behavior: - `PYTHONPATH=. pytest -q tests/fortran/source_preprocessing/preprocessing/test_parser_boundaries.py` -- Parser diagnostics: - `PYTHONPATH=. pytest -q tests/fortran/source_parsing/parsing/test_error_handling.py` -- Fixture goldens: - `PYTHONPATH=. pytest -q tests/fortran/source_parsing/parsing/test_fortran_fixture_suite.py` -- Parser error fixtures: - `PYTHONPATH=. pytest -q tests/fortran/source_parsing/parsing/test_error_fixture_suite.py` - -Regenerate one Fortran fixture: - -```bash -python tests/fortran/source_parsing/parsing/generate_parser_goldens.py tests/fortran/source_parsing/parsing/fixtures/general/basic_subroutine.f90 -``` - -Executable tutorial: `tests/fortran/source_parsing/parsing/test_developer_tutorial.py`. - -### Semantics And `.pyi` - -Manual calls: - - - -Focused tests by concern: - -- Fortran parser-to-IR conversion: - `PYTHONPATH=. pytest -q tests/fortran/semantic_ir/semantics/` -- Wrapper-plan support diagnostics: - `PYTHONPATH=. pytest -q tests/fortran/infrastructure/codegen/` -- `.pyi` printer: - `PYTHONPATH=. pytest -q tests/fortran/semantic_pyi_format/pipeline/` -- `.pyi` loader and edited stub behavior: - `PYTHONPATH=. pytest -q tests/fortran/semantic_pyi_format/parsing/` -- Semantic and `.pyi` fixtures: - `PYTHONPATH=. pytest -q tests/fortran/semantic_pyi_format/pipeline/test_contract_loading.py` - - - -Regenerate semantic and `.pyi` fixtures: - -```bash -python tests/fortran/semantic_ir/semantics/generate_semantic_fixtures.py -WRAPPER_UPDATE_PYI_FIXTURES=1 python3 -m pytest -q tests/fortran/semantic_pyi_format/pipeline/test_contract_package_generation.py -``` - -Executable examples: `tests/fortran/semantic_pyi_format/pipeline/` and -`tests/fortran/semantic_pyi_format/parsing/`. - -### CLI - -Manual calls: - - - -Focused tests: - -- Full CLI behavior: - `PYTHONPATH=. pytest -q tests/fortran/command_line_interface/pipeline/` -- Stage dispatch: - `PYTHONPATH=. pytest -q tests/fortran/command_line_interface/pipeline/ -k "parse or semantics or pyi or wrap"` -- Language and preprocessing selection: - `PYTHONPATH=. pytest -q tests/fortran/command_line_interface/pipeline/ -k "language or preprocessing"` - -Executable reference: `tests/fortran/command_line_interface/pipeline/`. diff --git a/docs/developer/feature-to-code-map.md b/docs/developer/feature-to-code-map.md index 9f7fd66b9..dc77c092e 100644 --- a/docs/developer/feature-to-code-map.md +++ b/docs/developer/feature-to-code-map.md @@ -17,35 +17,34 @@ before documentation may call the behavior supported. | Feature or behavior | Public docs | Main implementation files | Focused tests | Support evidence | | --- | --- | --- | --- | --- | -| Fortran parse output | `docs/developer/fortran-parser-reference.md` | `prik/parsers/fortran/parser.py`, `models.py`, `lexer.py`, `type_resolver.py` | `tests/fortran/source_parsing/parsing/`, `tests/fortran/source_parsing/parsing/test_fortran_fixture_suite.py` | Parser facts and diagnostics match fixtures | -| Semantic `.pyi` generation | `docs/user/reference/semantic-pyi-format.md` | `prik/codegen/printers/pyi_printer.py` | `tests/fortran/semantic_pyi_format/pipeline/`, `tests/fortran/semantic_pyi_format/pipeline/test_modern_example.py` | Printed `.pyi` round-trips or matches fixtures | +| Fortran parse output | `docs/developer/packages/parsers.md` | `prik/parsers/fortran/parser.py`, `models.py`, `lexer.py`, `type_resolver.py` | `tests/fortran/source_parsing/parsing/`, `tests/fortran/source_parsing/parsing/test_fortran_fixture_suite.py` | Parser facts and diagnostics match fixtures | +| CLI stage selection and output | `docs/user/getting-started/beginner-workflow.md`, `docs/user/reference/cli-commands.md` | `prik/cli.py`, `prik/parsers/fortran/cli.py` | `tests/fortran/command_line_interface/pipeline/`, Fortran parser CLI tests, documentation example tests | Command output and diagnostics match checked expectations | +| Compiler preprocessing | `docs/user/examples/recipes/compiler-preprocessing.md`, `docs/developer/packages/preprocessing.md`, `docs/developer/packages/parsers.md` | `prik/preprocessing/source.py`, `prik/preprocessing/fortran.py` | `tests/fortran/source_preprocessing/preprocessing/`, `tests/fortran/source_preprocessing/preprocessing/test_parser_boundaries.py` | Prepared Fortran input, dependencies, and source mappings are stable | +| Semantic `.pyi` generation | `docs/user/reference/semantic-pyi-format.md` | `prik/printers/pyi.py` | `tests/fortran/semantic_pyi_format/pipeline/`, `tests/fortran/semantic_pyi_format/pipeline/test_modern_example.py` | Printed `.pyi` round-trips or matches fixtures | | Semantic `.pyi` conversion and editing | `docs/user/reference/pyi-contracts/index.md`, `docs/user/reference/semantic-pyi-format.md`, `docs/user/examples/recipes/semantic-pyi-contracts.md` | `prik/parsers/pyi/parser.py`, `prik/pipeline/pyi.py`, `prik/semantics/pyi2ir.py`, `models.py` | `tests/fortran/semantic_pyi_format/` | Edited contracts parse to Python AST, then become semantic IR with preserved native facts | -| Semantic and wrapper-planning errors | `docs/user/guide/error-handling.md`, `docs/user/reference/diagnostic-codes.md` | `prik/semantics/fortran2ir.py`, `prik/semantics/policy_completion.py`, `prik/codegen/planner.py` | `tests/fortran/infrastructure/semantics/`, feature-local `policy/`, and feature-local `codegen/` tests | Each owning stage rejects unsupported or incomplete contracts | +| Semantic and wrapper-planning errors | `docs/user/guide/error-handling.md`, `docs/user/reference/diagnostic-codes.md` | `prik/semantics/fortran2ir.py`, `prik/policy/completion.py`, `prik/planning/planner.py` | `tests/fortran/infrastructure/semantics/`, feature-local `policy/`, and feature-local `codegen/` tests | Each owning stage rejects unsupported or incomplete contracts | | Fortran wrapper orchestration | `docs/user/reference/fortran-wrapper.md`, `docs/user/guide/building-shared-library.md` | `prik/pipeline/build.py` | `tests/fortran/building_shared_library/end_to_end/test_source_build_modes.py`, multi-source wrapper tests | Builds report artifacts and compile/link as documented | -| Completed semantic policy to wrapper artifacts | `docs/user/reference/fortran-wrapper.md` | `prik/semantics/policy_completion.py`, `prik/codegen/plan.py`, `planner.py`, `generator.py` | `tests/fortran/infrastructure/semantics/`, `tests/fortran/infrastructure/codegen/`, and feature-local policy/codegen tests | Runtime policy is explicit, the typed plan is complete, and generated artifacts compile and run | -| Native compilation and binding support | `docs/user/reference/fortran-wrapper.md`, `docs/user/guide/building-shared-library.md`, `docs/developer/build-system.md`, `docs/developer/quality-assurance.md` | `prik/compiling/`, `prik/binding_support/` | `tests/fortran/building_shared_library/end_to_end/test_runtime_compatibility.py`, build-mode tests | Generated sources compile, link, import, and clean up correctly | +| Completed semantic policy to generated wrapper | `docs/user/reference/fortran-wrapper.md` | `prik/policy/completion.py`, `prik/planning/models.py`, `prik/planning/planner.py`, `prik/codegen/docstrings.py`, `prik/pipeline/wrapper.py` | `tests/fortran/infrastructure/semantics/`, `tests/fortran/infrastructure/codegen/`, and feature-local policy/codegen tests | Runtime policy is explicit, the typed plan is complete, and the generated wrapper compiles and runs | +| Native compilation and binding support | `docs/user/reference/fortran-wrapper.md`, `docs/user/guide/building-shared-library.md`, `docs/developer/packages/compiler.md`, `docs/developer/workflows/quality-assurance.md` | `prik/compiler/`, `prik/runtime/native_support/` | `tests/fortran/building_shared_library/end_to_end/test_runtime_compatibility.py`, build-mode tests | Generated sources compile, link, import, and clean up correctly | | Source documentation structure | `docs/developer/source-map.md` | `docs/`, package README files, `tests/docs/test_reference_and_source_map.py` | documentation metadata, navigation, source-map, and example tests | Pages have metadata, audience separation, and source coverage checks | +| Semantic IR | `docs/user/reference/semantic-ir.md` | `prik/semantics/models.py`, `fortran2ir.py`, `pyi2ir.py` | `tests/fortran/semantic_ir/semantics/`, `tests/fortran/semantic_pyi_format/semantics/` | Fortran or semantic `.pyi` facts lower without losing wrapper-relevant meaning | +| Generated Fortran bridge | `docs/user/reference/fortran-wrapper.md` | `prik/codegen/fortran/bridge.py`, `prik/printers/fortran.py`, `prik/pipeline/wrapper.py` | `tests/fortran/infrastructure/codegen/`, feature-local codegen and end-to-end tests | Generated bridge compiles and preserves the native calling contract | +| Generated CPython binding | `docs/user/reference/fortran-wrapper.md` | `prik/codegen/c/binding.py`, `prik/codegen/c/python_surface.py`, `prik/printers/c.py`, `prik/pipeline/wrapper.py` | `tests/fortran/infrastructure/codegen/`, feature-local codegen and end-to-end tests | Extension imports, validates Python inputs, dispatches overloads in C, and installs the derived-class Python facade | +| Public API exports | `README.md`, `docs/user/reference/python-api.md` | `prik/__init__.py` | `tests/fortran/source_parsing/parsing/test_public_entrypoints.py` | Import paths are intentional and documented | ## First-File Rule - When the user-visible behavior changes, update the public docs in the same row before or alongside the implementation. The documentation structure test keeps @@ -55,20 +54,17 @@ this routing page tied to the source hotspots and package README files. | User workflow | Start in code | Do not mark supported until | | --- | --- | --- | -| Wrapping functions and subroutines | `prik/semantics/fortran2ir.py`, policy completion, `prik/codegen/planner.py`, bridge and binding generators | Runtime tests compile, import, call, and verify return and failure behavior | +| Wrapping functions and subroutines | `prik/semantics/fortran2ir.py`, policy completion, `prik/planning/planner.py`, bridge and binding generators | Runtime tests compile, import, call, and verify return and failure behavior | | Wrapping modules and module variables | parser module facts, semantic module conversion, naming policy, wrapper generators | Python-visible names, accessors, and unsupported module constructs are tested | | Arrays and allocatables | semantic array contracts, ownership policy, typed wrapper plans, bridge/binding array handlers | dtype, shape, rank, contiguity, mutation, returned arrays, and failure paths are tested; ordinary NumPy array actuals validate and extract their buffer directly in the C binding, descriptor handles use the planned runtime-handle path, and strided contracts carry a dense-actual role for zero-copy fast-path selection | | Pointer arguments | semantic metadata, ownership policy, bridge/binding pointer handlers | Owner, lifetime, association, and blocked cases are explicit and tested | | Optional arguments | parser optional attributes, semantic arguments, binding argument parsing | Present/absent calls and unsupported combinations are tested | | Generic interfaces | parser interface facts, semantic overload sets, `FunctionOverloadSet`, binding dispatch | Overload selection and ambiguity failures are tested at runtime | | Enumerations | parser enum facts, semantic constants/classes, codegen projection | Python-visible values and unsupported enum forms are tested | -| Packaging and distribution | `prik/pipeline/build.py`, `prik/compiling/`, future packaging integration | Build artifacts, native dependencies, and platform constraints are documented and tested | - - +| Packaging and distribution | `prik/pipeline/build.py`, `prik/compiler/`, future packaging integration | Build artifacts, native dependencies, and platform constraints are documented and tested | ## Evidence Rule diff --git a/docs/developer/fortran-parser-reference.md b/docs/developer/fortran-parser-reference.md deleted file mode 100644 index e7ec8d7f3..000000000 --- a/docs/developer/fortran-parser-reference.md +++ /dev/null @@ -1,1271 +0,0 @@ ---- -title: Fortran Parser Reference -audience: developers -prerequisites: repository structure, parser architecture -related: adding-a-fortran-construct.md, repository-structure.md -status: maintained -publication: draft ---- - -# Fortran parser reference (wrapper-focused subset) - -This document defines the currently supported parser subset, expected behavior, -and practical usage from terminal and Python. - -## 1) Supported features (comprehensive) - -### 1.1 Source forms and preprocessing - -- Free-form Fortran: `.f90`, `.f95`, `.f03`, `.f08` -- Fixed-form Fortran: `.f`, `.for`, `.ftn` -- Free/fixed comment stripping -- Continuation handling for both forms - -### 1.2 Procedure parsing - -- `subroutine` headers -- `function` headers -- Header modifiers: `pure`, `elemental`, `recursive` -- Function `result(...)` parsing (tolerant support for `results(...)`) - -### 1.3 Declaration/argument parsing - -- Intrinsic types: `integer`, `real`, `complex`, `logical`, `character` -- Kind extraction from declaration specs (`kind=...`) -- Attribute extraction: - - `intent(in|out|inout)` - - `optional` - - `value` - - `allocatable` - - `pointer` - - `target` -- Array extraction: - - `dimension(...)` - - variable-level shape syntax (`x(:)`, `x(n)`) - -### 1.4 Modules, imports, and project context - -- Module discovery -- Module variable extraction -- Shared specification-part parsing for module-like scopes (modules, - submodules, programs, and block-data units), preserving original line - numbers while skipping contained procedure bodies where they are not - wrap-relevant -- `use` extraction at module and procedure scope -- Explicit `use` symbol mappings preserve imported `source` names and local - `target` names for renamed imports -- Propagation of module-level `use` imports into contained procedures -- Folder/project parsing with dependency-aware ordering -- Cross-file kind constant resolution (e.g., kinds modules) -- Cached compile-time expression resolution for local/module parameters, - module/program variable shapes, and character lengths - -### 1.5 Derived type parsing - -- `type :: ... end type` and legacy `type name ... end type` discovery -- Parameterized derived-type headers such as `type :: buffer_type(k, n)` - and declarations such as `type(buffer_type(real64, 4))` -- Type attributes (e.g., `abstract`) -- Inheritance (`extends(parent)`) -- Field extraction including shape/pointer/allocatable -- Type-bound procedures: - - `procedure ... :: ...` bindings with attributes (e.g. `pass(self)`, `nopass`) - - `generic ... :: name => target1, target2` - -### 1.6 Parser diagnostics and wrapper planning boundary - -- Parser diagnostics report source-level parse errors and unsupported parser - constructs. -- Parser JSON remains parse-only and does not contain wrapper-plan decisions - or support diagnostics. -- Wrapper builds complete policy from semantic IR and validate the resulting - wrapper plan. Unsupported contracts report the owning plan path and - completed-policy diagnostic. - -## 2) Public API surface - -Supported public API: - -- `parse_fortran_file(source_or_path, filename=None, encoding="utf-8") -> FortranFile` -- `parse_fortran_project(files, encoding="utf-8") -> FortranProject` - -## Parser organization notes - -`prik/parsers/fortran/parser.py` is now intentionally organized into clearly labeled -sections and carries embedded implementation guidance. Start with the thin public -wrappers at the bottom, then read the class from top to bottom: - -- Regex/constants, parser-wide type aliases, private unit dataclasses, and the - compile-time resolver -- `FortranParser` internals grouped by domain: - - public parse entrypoints (`parse_file`, `parse_project`). The supported - module-level API remains the wrappers listed above. - - source-unit visitors for files, modules, submodules, programs, - procedures, interfaces, derived types, and block data - - recursive source-unit slicing (`header`, specification part, execution - part, `contains`) with original line numbers preserved on each slice - - shared declaration parsing for module variables, program/block-data - variables, procedure arguments/results, and derived-type fields - - `_helper_*` methods for scoped parsing, expression resolution, same-level - duplicate checks, and shared specification-part collection -- Thin module-level convenience wrappers that delegate to a shared parser - instance - -Parser methods carry focused docstrings, with examples where a grammar visitor -or lexical helper is easier to understand from a concrete call. - -The Fortran parser is now packaged under `prik.parsers.fortran` rather than a -top-level parser package. The package includes its CLI module, lexer, -JSON-compatible parse models, project parser, type resolver, and utility -helpers. Public callers should use the stable top-level `prik` parser exports -or `prik.parsers.fortran` package imports. - -## Implementation Inventory And Maintenance - -This file is the single maintained Fortran parser reference. It replaces the -older standalone implementation-reference document; parser feature inventory, -testing workflow, and maintenance guard policy live here. - -The implementation inventory is maintained across these surfaces: - -- `prik/parsers/fortran/parser.py` owns source slicing, declaration extraction, - diagnostics, project ordering, dependency resolution, and compile-time - expression resolution. -- `prik/parsers/fortran/models.py` owns parse-only dataclasses and JSON-compatible - parser facts. -- `prik/semantics/fortran2ir.py` owns conversion from parser facts to semantic IR, - including kind mapping, compile-time specialization, storage contracts, - projection metadata, and wrapper-planning inputs. -- `tests/fortran/source_parsing/parsing/` covers parser contracts, source-unit slicing, diagnostics, - project behavior, and fixture regressions. -- `tests/fortran/semantic_ir/semantics/` covers semantic conversion, datatype precision mapping, - wrapper planning, `.pyi` emission, and compile-time specialization. - - - -`parse_file` is the central orchestration path. It first slices the source into -direct file-level units, then each class visitor parses only its own substring -and recursively slices direct children. This is the key parser design: each -Fortran grammar unit has a header, a specification region, optional execution -region, and optional `contains` region. The differences between modules, -programs, procedures, derived types, interfaces, and block data are expressed -by small visitor decisions and grammar flags rather than separate whole-file -parsing loops. - -Nested unit boundaries and placement outside execution regions are checked even -when they are not exported as wrapper metadata. Internal procedures inside a -host procedure's `contains` block are structurally sliced, then their -declarations and bodies are skipped. Once an execution boundary is detected, -procedure bodies and standalone included execution fragments are intentionally -skipped. Procedure-local interface blocks are still visited enough to type -callback dummy arguments and to preserve interface metadata. - -### 2.1 Recursive parser sketch - -Small input: - -```fortran -module m - integer, parameter :: n = 4 -contains - subroutine scale(x) - real, intent(inout) :: x(n) - end subroutine scale -end module m -``` - -The parser handles it in this order: - -1. `parse_file` preprocesses the source and calls `_helper_slice_child_units` - at file scope. The result is one `ModuleUnit` carrying the module name, - lines, and source locations. -2. the shared `ClassVisitor._visit` dispatcher selects `_visit_ModuleUnit`. -3. `_visit_ModuleUnit` creates a module `_ParserScope`, calls - `_helper_split_unit_parts`, and sends only the module specification lines to - `_parse_specification_part`. -4. `_parse_specification_part` uses the shared declaration backend: - `_helper_parse_declaration_line` parses `integer, parameter :: n = 4`, then - `_helper_push_declaration_to_scope` appends the resulting parameter variable - to `FortranModule.variables`. -5. The module visitor recursively slices direct children from its substring. - It finds one procedure unit, `scale`, and dispatches it to - `_visit_ProcedureUnit`. -6. `_visit_ProcedureUnit` creates a procedure `_ParserScope`, splits the - procedure into header/specification/execution/contains, and visits only the - specification part. The same declaration backend parses - `real, intent(inout) :: x(n)` and pushes the metadata into the procedure - argument symbol table. - -Scope is always an explicit argument to the shared helpers. That is the reason -two modules can each define `type :: state` without conflict, while two -same-level `module m` declarations or two same-level contained procedures with -the same name are rejected by `_helper_validate_sibling_units`. - -End-name validation is strict for structural units whose names define exported -scope boundaries, such as modules, submodules, programs, interfaces, and -derived types. Procedure end-name mismatches are still tolerated while slicing -third-party sources because some accepted fixture code contains copy/paste -procedure end labels; the procedure is closed by unit kind so parsing can -continue, and duplicate procedure names are validated at the sibling scope. - -The only separate specification-line visitors are grammar-specific: -module-like units share `_parse_module_like_spec_line`, procedures use -`_parse_procedure_spec_line` for `implicit`, `external`, `import`, and -local `parameter` handling, and derived types use -`_parse_type_spec_line` for `sequence`, `private`, and type-bound -declaration rules. All three still call the same declaration parser/pusher for -actual declarations. - -Most parser organization changes are structural, but behavior, model-schema, -coverage, or fixture changes should be reflected in this reference. - -Parameter constants expose both `value` and serialized `symbolic_value` when -available. `value` is reserved for a literal/evaluated result after -compile-time folding. If an initializer cannot be evaluated safely, such as -`selected_real_kind(...)`, `value` is `None` and `symbolic_value` preserves the -original initializer for validation, debugging, downstream diagnostics, and -JSON consumers. - -Procedure-local parameters may be folded into argument shapes during procedure -finalization. Module-level and `use`-associated parameters used in procedure -argument shapes are kept symbolic in the signature (`x(n)` remains `["n"]`) -and are treated as valid scope references for policy completion. Module/program -variable shapes and parameter values can be resolved through the compile-time -resolver when enough information is available. - -## Reimplementation Guide For Another Parser - -Use the Fortran parser as the reference for any source language with nested -program units, scoped declarations, and a later semantic handoff. The details -are Fortran-specific, but the parser architecture is reusable. - -Recommended frontend responsibilities: - -- Keep one typed model layer for parse-only facts. -- Keep one parser orchestration class with thin public wrappers. -- Slice source into grammar units before parsing declarations. -- Pass scope explicitly into shared helpers rather than using global mutable - parser state for symbol resolution. -- Parse only wrapper-relevant specification facts; skip executable bodies once - they are outside the parser contract. -- Preserve source locations and original line numbers through preprocessing and - recursive slicing. -- Emit parser diagnostics for malformed source, but leave wrappability policy - to semantic policy completion. - -The Fortran data flow is: - -```text -source path or source text - -> compiler/native include preprocessing - -> FortranParser.parse_file(...) - -> source-unit slices with original line numbers - -> scoped specification parsing - -> FortranFile parser facts - -> parse_fortran_project(...) dependency ordering and namespace resolution - -> semantics.fortran2ir conversion - -> policy completion, `.pyi`, and the implemented Fortran wrapper stages -``` - -The recursive parsing pattern is: - -1. Identify direct child units at the current grammar level. -2. Split each child into header, specification part, execution part, and - `contains` part where that language construct allows them. -3. Parse declarations only from the specification part. -4. Recurse only into direct children that are legal for the current unit kind. -5. Validate sibling names and scope-local duplicate declarations. -6. Finalize procedure arguments/results after local declarations and - parameters are known. -7. Resolve cross-file or imported compile-time facts only at project or - semantic-conversion boundaries. - -When adding another parser, keep these test layers separate: - -- parser unit tests for grammar slicing and declarations; -- parser fixture tests for stable JSON/model output; -- parser error fixture tests for fatal diagnostic contracts; -- project tests for dependency ordering and cross-file resolution; -- CLI tests for frontend selection, stage dispatch, output files, and debug - behavior; -- semantic conversion tests for parser-to-IR mapping; -- `.pyi` tests for generated and edited interface round trips. - -Executable references: - -- Fortran parser walkthrough: `tests/fortran/source_parsing/parsing/test_developer_tutorial.py` -- Procedure/type parsing: `tests/fortran/source_parsing/parsing/` -- Scope and project behavior: `tests/fortran/modules/parsing/test_scope_handling.py` and - `tests/fortran/modules/parsing/test_project_scope_models.py` -- Fortran fixture workflow: `tests/fortran/source_parsing/parsing/test_fortran_fixture_suite.py` -- Shared CLI behavior: `tests/fortran/command_line_interface/pipeline/` -- Fortran semantic handoff: `tests/fortran/semantic_ir/semantics/` - -## 3) Terminal usage and expected outputs - -### 3.1 Basic CLI invocation - -```bash -python -m prik parse path/to/file.f90 -``` - -Recognizable Fortran files can omit `--language`. Directories require explicit -frontend selection: - -```bash -python -m prik parse path/to/fortran_src --language fortran -``` - -Fortran directories are recursively scanned for `.f`, `.for`, `.ftn`, `.f90`, -`.f95`, `.f03`, `.f08`. - -The Fortran frontend rejects unsupported non-Fortran syntax before -wrapper-focused parsing when it appears outside executable procedure/program -bodies, which are intentionally not represented in the extracted interface. - -The human-readable parse tree keeps scope variables compact by default as -`vars=N`. Add `--show-vars` to print the variables, or `--print-limit N` to -print only the first `N` items in each repeated section. - -### 3.2 Human-readable output example - -Input Fortran (`tests/fortran/source_parsing/parsing/fixtures/general/basic_subroutine.f90`): - - -```fortran -module m1 -contains -subroutine add1(n, x) - integer, intent(in) :: n - real(kind=8), intent(inout), dimension(n) :: x -end subroutine add1 -end module m1 -``` - -Command: - - -```bash -python -m prik parse tests/fortran/source_parsing/parsing/fixtures/general/basic_subroutine.f90 -``` - -Expected output: - - -```text -File: tests/fortran/source_parsing/parsing/fixtures/general/basic_subroutine.f90 - Modules: 1 - - module m1 (vars=0, uses=0) - Procedures: 1 - - subroutine add1(n:integer[0], x:real(8)[1]) -``` - -The same command with `--show-vars` uses the variable-expanded report path. -This fixture currently has no module variables to print, so the output remains -compact: - - -```bash -python -m prik parse tests/fortran/source_parsing/parsing/fixtures/general/basic_subroutine.f90 --show-vars -``` - - -```text -File: tests/fortran/source_parsing/parsing/fixtures/general/basic_subroutine.f90 - Modules: 1 - - module m1 (vars=0, uses=0) - Procedures: 1 - - subroutine add1(n:integer[0], x:real(8)[1]) -``` - -For large files: - -```bash -python -m prik parse path/to/file.f90 --show-vars --print-limit 50 -``` - -`--print-limit` applies independently to modules, submodules, programs, block -data units, derived types, fields, procedures, and variables when variables are -shown. Counts such as `Procedures: 80` and `Variables: 657` still show the full -totals even when only the first `N` entries are printed. - -Interpretation: - -- Parsed entities are counted per file. -- Free procedures (outside modules) are shown in top-level `Procedures`. -- Module-contained procedures are nested under each module. -- Empty sections are omitted from the human-readable report. - -More complex example: - -Input Fortran (`mixed_example.f90`): - - - -Command: - -```bash -python -m prik mixed_example.f90 -``` - -```text -File: mixed_example.f90 - Procedures: 1 - - subroutine driver(n:integer[0]) - Modules: 2 - - module math_ops (vars=1, uses=1) - Procedures: 2 - - subroutine saxpy(n:integer[0], a:real[0], x:real[1], y:real[1]) - - function dot(x:real[1], y:real[1]) - - module io_ops (vars=0, uses=0) - Procedures: 1 - - subroutine dump(v:real[1]) -``` - -### 3.3 JSON and semantic output - -Print parser JSON: - -```bash -python -m prik tests/fortran/source_parsing/parsing/fixtures/general/basic_subroutine.f90 --json -``` - -Write parser JSON: - -```bash -python -m prik parse tests/fortran/source_parsing/parsing/fixtures/general/basic_subroutine.f90 --json --out report.json -``` - -Expected JSON layout: - -- Top-level object keyed by input path -- Per-file payload with keys: - - `signatures` - - `types` - - `modules` - - `submodules` - - `programs` - - `block_data` - -When `prik parse --json` applies compiler preprocessing, the per-file payload -also contains `preprocessing_recipe`. The CLI applies compiler preprocessing -for file-based parsing; compiler linemarkers remain accepted for provenance. -The recipe records the exact compiler executable or adapter, argv, include -paths, macro flags, standard, extra compiler arguments, working directory, -include graph, source mappings, diagnostics, and optional macro metadata used -to produce the parsed stdout stream. - -Fortran CPP directives are handled by the configured compiler. Native Fortran -`include "file.inc"` statements are then expanded recursively by the -preprocessing layer before the single parser pass. Native INCLUDE is textual -insertion into the current scope; it is not a `use` import from a separately -compiled module. Include lookup is relative to the including file first, then -the configured include directories, duplicate textual inclusion is preserved, -and missing files or cycles produce `INCLUDE_NOT_FOUND` or `INCLUDE_CYCLE` -diagnostics. - -`use` import shape: - - - - - -- A renamed import such as - `use list_input, delete_input => delete_input_list` records both sides: - -```json -"uses": { - "list_input": [ - { - "source": "delete_input_list", - "target": "delete_input" - } - ] -} -``` - - - -### 3.4 Semantic and wrapper-plan output - -Parser output and semantic IR are separate stages. Run parser inspection with: - -```bash -python -m prik parse tests/fortran/source_parsing/parsing/fixtures/general/basic_subroutine.f90 -``` - -Build a wrapper with the default wrapper stage. If the completed plan cannot -lower a contract, the build reports the precise plan owner and blocker: - -```bash -python -m prik tests/fortran/source_parsing/parsing/fixtures/general/basic_subroutine.f90 -``` - -Parser JSON stays parse-only. - -Semantic IR JSON uses the same output channels, but the per-file payload is the -semantic model projection instead of raw parser output: - -```bash -python -m prik semantics tests/fortran/source_parsing/parsing/fixtures/general/basic_subroutine.f90 -``` - -Generated `.pyi` text is printed with: - -```bash -python -m prik generate --pyi tests/fortran/source_parsing/parsing/fixtures/general/basic_subroutine.f90 -``` - -### 3.5 Parse-error diagnostics and debug mode - -When parsing fails, the CLI prints a compiler-style diagnostic to `stderr` and -exits with status code `1`. By default this output is intended for end users: it -includes the source location, diagnostic code, message, source line, and caret -context, but it does **not** include a Python traceback. - -Example command: - -```bash -python -m prik tests/fortran/source_parsing/parsing/fixtures/errors/err_duplicate_argument_name.f90 -``` - -Example diagnostic shape: - -```text -tests/fortran/source_parsing/parsing/fixtures/errors/err_duplicate_argument_name.f90:1:1: error[PARSE_DUPLICATE_ARGUMENT]: Duplicate argument name 'x' in procedure 'dup'. - | -1 | subroutine dup(x, y, x) - | ^ -``` - -ANSI color is enabled by default when available; no color flag is needed for -normal use. To disable color explicitly, pass `--no-color` or set the standard -`NO_COLOR` environment variable: - -```bash -python -m prik bad.f90 --no-color -NO_COLOR=1 python -m prik bad.f90 -``` - -For parser development, use `--debug` to re-raise -`FortranParseError` and let Python print the full traceback showing where the -error was raised internally: - -```bash -python -m prik bad.f90 --debug -``` - -The same developer mode can be enabled with the environment variable -`FORTRAN_PARSER_DEBUG=1`: - -```bash -FORTRAN_PARSER_DEBUG=1 python -m prik bad.f90 -``` - -In debug mode, the traceback's final exception message also includes a -`note: parser raised at ...` line with the internal parser file, line, and -function that created the diagnostic. - -## 4) Python usage and expected outputs - -### 4.1 Parse folder namespace - -```python -from prik import parse_fortran_project -from pathlib import Path - -files = list(Path("tests/fortran/source_parsing/parsing/fixtures/general").glob("*.f90"))[:5] -project = parse_fortran_project(files) -print(len(project.files)) -print(len(project.modules)) -``` - -Expected behavior: - -- Recursively scans Fortran files. -- Resolves dependencies and module imports across files. -- Returns aggregate namespace parse output. - -### 4.2 Parse single file and convert it to semantic IR - -```python -from pathlib import Path -from prik import parse_fortran_file -from semantics.fortran2ir import fortran_file_to_semantic_modules - -p = Path("tests/fortran/source_parsing/parsing/fixtures/general/basic_subroutine.f90") -code = p.read_text() - -parsed = parse_fortran_file(code, filename=str(p)) -modules = fortran_file_to_semantic_modules(parsed, standalone_module_name=p.stem) -print("procedures", len(parsed.procedures)) -print("semantic modules", len(modules)) -``` - -Expected behavior: - -- `parsed` is a `FortranFile` aggregate model with parsed units and symbols. -- `modules` is the semantic IR projection used by `.pyi` printing and wrapper - planning. - -### 4.3 Structured argument specifications - -Compatibility fields such as `FortranArgument.shape`, `lbound`, `ubound`, and -`kind` remain serialized as strings/lists. For callers that need typed access, -argument and variable models also expose structured helpers: - -- `structured_shape` returns a `FortranShape` containing parsed dimensions. -- Slice-like dimensions such as `1:n:2` are represented as `FortranSlice`. -- Whole-expression function calls such as `lbound(x, 1)` are represented as - `FortranFunctionCall`. -- `kind_expression` and `value_expression` parse `kind` and `value` strings - using the same lightweight expression model. - -Example: - -```python -arg.shape -# ["lbound(src, 2):ubound(src, 2)"] - -dim = arg.structured_shape.dimensions[0] -dim.lower.name -# "lbound" -dim.upper.name -# "ubound" -``` - -### 4.4 Declaration-expression ownership - -The declaration parser preserves balanced Fortran 2008/2018 bound text for all -declaration owners: module variables, derived-type fields, dummy arguments, and -procedure results. Nested calls, array constructors, component references, and -colons inside nested syntax do not split an outer dimension or bound. - -Semantic conversion sends every explicit extent through the shared -`prik.utilities.declaration_expressions` layer. That layer retains the native -spelling in `source_shape` and produces the language-neutral public spelling -used by `.pyi`, including -`size(a)` to `a.size`, `size(a, dim)` to `a.shape[dim - 1]`, and `rank(a)` to -`a.ndim`. Post-IR policy then resolves public scalar and array-property -references to wrapper roles. Binding and bridge generators only render the -completed expression for their target language; they do not infer declaration -semantics. - -`lbound(a, dim)` uses the lower bound declared for that dummy axis rather than -Python's index origin. `ubound(a, dim)` combines that bound with the runtime -extent, and the shared expression layer reduces the common -`ubound-lbound+1` form to `a.shape[dim - 1]`. Direct inquiries preserve the -standard zero-extent results: lower bound one and upper bound zero. - -Parsing and preservation are intentionally broader than wrapper execution. -Valid specification expressions whose value exists only in private native -state remain available as source metadata but produce an explicit policy -blocker when no boundary role can supply them. Calls to user specification -functions also remain in the language-neutral expression. Semantic conversion -resolves each call to a local module procedure, through the declaration owner's -`USE` mappings, or to a concrete procedure interface in the same declaration -scope. It records the visible spelling, original native name, native placement, -and resolved declaration. - -A wildcard import is resolved only when file/project parsing has indexed the -named procedure in exactly one imported module; conversion does not guess from -an unavailable module export list. The `.pyi` loader reconstructs the same -identity from module functions, imports, and `@prototype` declarations. A -prototype is one signature model: annotation use makes it a callback signature, -while call use names a standalone procedure entity. Post-IR policy validates -purity, scalar-integer result, argument association, and accessibility, then -selects either a module `use` or a standalone procedure declaration backed by -the generated abstract interface. A pure prototype cannot also be a Python -callback because its generated adapter calls the Python runtime; that mixed use -is blocked before planning. The binding and bridge consume only that -completed action. Fortran 2023 vector bounds and `RANK` clauses are outside the -parser's advertised Fortran 2008/2018 language modes. - -## 5) Running tests - -Run all tests: - -```bash -PYTHONPATH=. pytest -q -``` - -Run parser-focused tests: - -```bash -python -m prik parse tests/fortran/source_parsing/parsing/fixtures/general/basic_subroutine.f90 --language fortran --json -PYTHONPATH=. pytest -q tests/fortran/source_parsing/parsing/ -PYTHONPATH=. pytest -q tests/fortran/source_parsing/parsing/test_fortran_fixture_suite.py -PYTHONPATH=. pytest -q tests/fortran/command_line_interface/pipeline/ -``` - -Focused test files by implementation area: - -- Parser walkthrough and expected developer flow: - `tests/fortran/source_parsing/parsing/test_developer_tutorial.py` -- Procedure headers, declarations, derived types, interfaces, and type-bound - procedures: - `tests/fortran/source_parsing/parsing/` -- Function header edge cases: - `tests/fortran/functions/parsing/test_function_headers.py` -- Scope handling and project namespace behavior: - `tests/fortran/modules/parsing/test_scope_handling.py` and - `tests/fortran/modules/parsing/test_project_scope_models.py` -- Preprocessing, native includes, and execution-boundary skipping: - `tests/fortran/source_preprocessing/preprocessing/test_parser_boundaries.py` -- Parser diagnostics and fatal error contracts: - `tests/fortran/source_parsing/parsing/test_error_handling.py` -- Regression contracts: - `tests/fortran/source_parsing/parsing/` -- Public entrypoints: - `tests/fortran/source_parsing/parsing/test_public_entrypoints.py` -- Parser fixture goldens: - `tests/fortran/source_parsing/parsing/test_fortran_fixture_suite.py` -- Parser error fixture goldens: - `tests/fortran/source_parsing/parsing/test_error_fixture_suite.py` -- Parser JSON shape: - `tests/fortran/source_parsing/parsing/test_json_sanity.py` -- Cached Fortran compiler/type and intrinsic-storage probing: - `tests/fortran/data_types/probes/test_fortran_type_probes.py` -- Shared CLI behavior: - `tests/fortran/command_line_interface/pipeline/` - -When adding or changing a Fortran parser feature, add a focused parser test -near the implementation concern first, then update fixture goldens only when -the serialized parser contract intentionally changes. - -Update golden JSON fixtures: - -```bash -python tests/fortran/source_parsing/parsing/generate_parser_goldens.py -``` - -Update selected fixture(s): - -```bash -python tests/fortran/source_parsing/parsing/generate_parser_goldens.py tests/fortran/source_parsing/parsing/fixtures/general/basic_subroutine.f90 -``` - -In-test auto-update mode: - -```bash -FORTRAN_PARSER_UPDATE_GOLDENS=1 PYTHONPATH=. pytest -q tests/fortran/source_parsing/parsing/test_fortran_fixture_suite.py --confcutdir=tests/ -``` - -Semantic and `.pyi` fixtures have separate generators: - -```bash -python tests/fortran/semantic_ir/semantics/generate_semantic_fixtures.py -WRAPPER_UPDATE_PYI_FIXTURES=1 python3 -m pytest -q tests/fortran/semantic_pyi_format/pipeline/test_contract_package_generation.py -``` - -## 6) Error handling - -All parse failures raise `FortranParseError`, a subclass of `ValueError`. The -exception keeps structured metadata for consumers: - -- `filename` — source path supplied to the parser, if any -- `line_number` — 1-based source line where the error was detected, if known -- `source_line` — original source text for context, if known -- `base_message` — stable error text without location/source context -- `code` — stable, explicit diagnostic category identifier; manually - constructed fallback errors use `PARSE_ERROR`, while grammar rejection uses - `PARSE_INVALID_SYNTAX` - -Diagnostic codes are for programmatic matching in tests, tools, and -documentation. The category name states the failure class directly. The shared -registry is [`diagnostic-codes.md`](../user/reference/diagnostic-codes.md). - -`str(error)` and `error.format_diagnostic(color=False)` render a -compiler-style diagnostic: - -```text -::1: error[]: - | - | - | ^ -``` - -If no filename is available, the location is rendered as ``. If a line -number or source line is unavailable, that part of the diagnostic is omitted or -shown with `?` as appropriate. Use `error.base_message` when tests or API -consumers need only the message text. - -`format_diagnostic(color=True)` adds ANSI styling. The CLI requests colored -diagnostics by default when available; pass `--no-color` or set `NO_COLOR=1` to -disable ANSI output. On Windows, ANSI console compatibility is enabled through -`colorama` when it is installed. - -For parser development, `format_diagnostic(debug=True)` appends a note with the -internal parser file, line, and function that raised the error. The CLI exposes -this through `--debug` or `FORTRAN_PARSER_DEBUG=1`; normal CLI parse errors intentionally hide Python -tracebacks. - -The sections below list each error category, the triggering condition, and the -exact `base_message` format (with `<...>` placeholders for runtime values). - -### 6.1 Unknown or unsupported type declaration - -Triggered when a declaration line cannot be matched to any known intrinsic type, -`type(...)`, or `character` variant. - -**In a procedure:** - -``` -Unknown or unsupported datatype declaration for procedure '': -``` - -Example Fortran that triggers this: - -```fortran -subroutine bad(x) - weirdtype :: x -end subroutine bad -``` - -Example error: - -``` -bad.f90:2:1: error[PARSE_UNSUPPORTED_DECLARATION]: Unknown or unsupported datatype declaration for procedure 'bad': weirdtype :: x - | -2 | weirdtype :: x - | ^ -``` - -**In a derived type:** - -``` -Unknown or unsupported datatype declaration in type '': -``` - -**In a module:** - -``` -Unknown or unsupported datatype declaration in module '': -``` - -### 6.2 Duplicate declaration - -Triggered when the same symbol is declared more than once in the same scope. - -**In a procedure (arguments and local declarations):** - -``` -Duplicate declaration of symbol '' in procedure ''. -``` - -Example: - -```fortran -subroutine dup(x) - real :: x - integer :: x -end subroutine dup -``` - -Example error: - -``` -dup.f90:3:1: error[PARSE_DUPLICATE_DECLARATION]: Duplicate declaration of symbol 'x' in procedure 'dup'. - | -3 | integer :: x - | ^ -``` - -**PARAMETER constants:** - -``` -Duplicate PARAMETER declaration of symbol '' in procedure ''. -``` - -**In a derived type:** - -``` -Duplicate field '' in derived type ''. -``` - -**In a module:** - -``` -Duplicate variable '' in module ''. -``` - -### 6.3 Duplicate procedure name - -Triggered when the same procedure name appears more than once within the same -module or global scope. -Internal procedures inside separate host `contains` blocks are scoped to their -host and do **not** conflict with each other. - -**Global scope:** - -``` -Duplicate procedure name '' in global scope. -``` - -**Module scope:** - -``` -Duplicate procedure name '' in module ''. -``` - -Example: - -```fortran -subroutine work(n) - integer, intent(in) :: n -end subroutine work - -subroutine work(n) - integer, intent(in) :: n -end subroutine work -``` - -Example error: - -``` -dup.f90:5:1: error[PARSE_DUPLICATE_PROCEDURE]: Duplicate procedure name 'work' in global scope. - | -5 | subroutine work(n) - | ^ -``` - -### 6.4 Duplicate argument name - -Triggered when a procedure's argument list contains the same name more than once. - -``` -Duplicate argument name '' in procedure ''. -``` - -Example: - -```fortran -subroutine dup(x, y, x) - integer, intent(in) :: x - real, intent(in) :: y -end subroutine dup -``` - -Example error: - -``` -dup_arg.f90:1:1: error[PARSE_DUPLICATE_ARGUMENT]: Duplicate argument name 'x' in procedure 'dup'. - | -1 | subroutine dup(x, y, x) - | ^ -``` - -### 6.5 Star-kind declarations - -Legacy `type*N` declarations, such as `real*8`, are accepted in both fixed-form -and modern-extension files. Numeric star declarations preserve their fixed -total storage width for semantic conversion. This matters most for complex -types: `complex*8` is an 8-byte `Complex64`, while modern `complex(kind=8)` is -a compiler kind and is 16 bytes on the documented `gfortran` target. -`DOUBLE PRECISION` and `DOUBLE COMPLEX` retain a compiler-dependent double-kind -expression and use the cached Fortran type probe. For `CHARACTER*N` and -`CHARACTER*(*)`, the star value is a length, not a kind or element storage -width. - -```fortran -subroutine accepted(x) - real*8 :: x -end subroutine accepted -``` - -See the [generated modern and legacy datatype mapping](../user/reference/semantic-ir.md#generated-linux-x86_64-mapping-example) -for the exact GitHub Actions target results. - -### 6.6 Source-form metadata - -The parser records source-form metadata from the filename and lexer, but does -not reject a construct solely because a `.f77` suffix was used. Grammar-region -validation still applies after preprocessing. - -### 6.7 Implicit none — undeclared argument or result - -Triggered when `implicit none` is active and an argument (or function result) -has no matching type declaration. - -**Argument:** - -``` -Argument '' in procedure '' has no type declaration (implicit none is active). -``` - -**Function result:** - -``` -Function result '' in procedure '' has no type declaration (implicit none is active). -``` - -Example: - -```fortran -subroutine foo(x, y) - implicit none - integer, intent(in) :: x -end subroutine foo -``` - -Example error: - -``` -implicit_none.f90:1:1: error[PARSE_IMPLICIT_NONE_UNDECLARED_SYMBOL]: Argument 'y' in procedure 'foo' has no type declaration (implicit none is active). - | -1 | subroutine foo(x, y) - | ^ -``` - -### 6.8 Unknown datatype for function result - -Triggered when a function result has no resolvable type after parsing (and -`implicit none` prevents implicit typing). - -``` -Unknown datatype for function result '' in procedure ''. -``` - -Example: - -```fortran -function f(x) result(res) - implicit none - real :: x -end function f -``` - -Example error: - -``` -bad.f90:1:1: error[PARSE_UNKNOWN_FUNCTION_RESULT_TYPE]: Unknown datatype for function result 'res' in procedure 'f'. - | -1 | function f(x) result(res) - | ^ -``` - -### 6.9 Unknown datatype for a module variable - -Triggered by `_validate_module_variables` when a parsed module variable still -has `base_type == "unknown"` after declaration parsing. - -``` -Unknown type for variable '' in module ''. -``` - -### 6.10 Unknown datatype for a derived type field - -Triggered by `_validate_derived_type_fields` when a field still has -`base_type == "unknown"`. - -``` -Unknown type for field '' in derived type ''. -``` - -### 6.11 PARAMETER symbol without type in `implicit none` scope - -Triggered when a legacy `PARAMETER (...)` statement names a symbol that has not -been typed and `implicit none` is in effect. - -``` -Unknown datatype for PARAMETER symbol '' in procedure ''. -``` - -Example: - -```fortran - subroutine cst(a) - implicit none - real a - parameter ( zero = 0.0e+0 ) - end -``` - -Example error: - -``` -legacy.f:4:1: error[PARSE_UNKNOWN_PARAMETER_TYPE]: Unknown datatype for PARAMETER symbol 'zero' in procedure 'cst'. - | -4 | parameter ( zero = 0.0e+0 ) - | ^ -``` - -### 6.12 Function result variable shadows an argument - -Triggered when a `result(name)` clause reuses an argument name (and the two -names are different from each other — the special case `result(f)` on a -function named `f` is allowed). - -``` -Function result variable '' in function '' shadows an argument name. -``` - -Example: - -```fortran -function f(res) result(res) - integer, intent(in) :: res -end function f -``` - -Example error: - -``` -shadow.f90:1:1: error[PARSE_RESULT_SHADOWS_ARGUMENT]: Function result variable 'res' in function 'f' shadows an argument name. - | -1 | function f(res) result(res) - | ^ -``` - -### 6.13 Failed to resolve declared argument - -An internal safety check: if a symbol was explicitly declared but its type -could not be applied (a parser regression guard), the following error is raised. - -``` -Failed to resolve declared argument '' in procedure ''. -``` - -## 7) Scope note - -This parser is intentionally wrapper-focused and not a complete Fortran front -end. Unsupported syntax should be surfaced through parser diagnostics or later -semantic policy inputs for incremental parser extension. - - -### External callback dummy declarations - -The parser accepts legacy callback-style declarations inside procedure scopes, including: - -- `external :: cb` (treated as a procedure-typed dummy) -- `real, external :: f` / `integer, external :: g` (typed external function dummies) - -Under `implicit none`, these declarations count as valid argument declarations, so callback arguments are not reported as missing datatype declarations. - -## 8) File, project, and semantic entrypoints - -Use the stable top-level API: - -- `parse_fortran_file(source_or_path, filename=None, encoding="utf-8") -> FortranFile` -- `parse_fortran_project(files, encoding="utf-8") -> FortranProject` - -Lower-level unit parsers are internal `FortranParser` methods. - -Semantic conversion lives in `prik/semantics/fortran2ir.py`. It accepts parsed `FortranFile` -(or selected `FortranModule`) structures and converts metadata into semantic IR -consumed by the `.pyi` printer and current Fortran wrapper/runtime stages. -Compiler-backed shared-CLI semantic stages resolve compiler-dependent kind -expressions, measure numeric and logical intrinsic storage with `storage_size`, -attach those facts to semantic types, and reuse memory and persistent caches. -The shared CLI applies project symbol completion even when the input contains -only one source file. That completion follows explicit renamed `use` -associations through project modules and propagates parent/ancestor -host-associated symbols into submodules before compiler-backed stages run. -This includes both a direct intrinsic rename such as `wp => real64` in a -single-file module and a re-exported chain such as `dp => rk => real64`; the -standalone compiler probe therefore receives the intrinsic expression -(`real64`) rather than a project-local alias that is out of scope in the -generated probe program. -Character declarations are excluded from storage probing: their semantic type -is `String`, while fixed or deferred element length is carried separately from -the declaration or runtime descriptor. The generated mapping report describes -the modeled eight-bit character code unit directly and does not manufacture a -compiler probe fact for character rows. For the maintained GitHub Actions -`gfortran` profile, unqualified `integer`, `real`, and `complex` map to `Int32`, -`Float32`, and `Complex64`; target-changing flags can change those mappings. -Source-driven wrapper builds add the normalized native Fortran compiler flags -to the internal probe configuration, so semantic type facts and native -implementation compilation use the same default-kind profile. The -[generated target datatype mapping](../user/reference/semantic-ir.md#generated-linux-x86_64-mapping-example) -measures and verifies those storage facts. - -The Fortran probe cache key includes the generated expression source, resolved -compiler binary identity, target flags, includes, macros, requested standard, -working directory, target-related environment, and runner. The persistent -location is `$XDG_CACHE_HOME/prik/fortran_type_probe` or -`~/.cache/prik/fortran_type_probe`; `PRIK_CACHE_DIR` changes the internal cache -root. The standalone `prik probe` command additionally exposes `--cache-dir` -and `--refresh` for explicit inspection runs. - -The standalone probe can create a reusable report containing the exact -compile-time and storage expressions needed by a source: - -```bash -python3 -m prik probe --language fortran --compiler gfortran \ - --expr='selected_real_kind(12)' \ - --expr='storage_size(real(0.0,kind=8))' \ - --out build/fortran-types.json -``` - -The report is an inspection and verification output. Semantic conversion and -wrapper builds measure the facts they need internally from their selected -compiler; the report is not a second semantic-stage input path. A missing -required expression is reported explicitly instead of falling back to an -unrelated target mapping. - -The semantic converter also supports compile-time specialization for values the -parser intentionally leaves symbolic. Use -`collect_semantic_compile_time_requirements(parsed)` to list missing parameter -or kind values, then pass a dictionary such as -`{"selected_real_kind(12)": 8}` to -`fortran_module_to_semantic_module(..., compile_time_values=...)` or -`fortran_file_to_semantic_modules(..., compile_time_values=...)`. Existing -semantic IR can be copied and specialized with -`resolve_semantic_compile_time_values(module, {"n": 64})`. diff --git a/docs/developer/index.md b/docs/developer/index.md index ba21a8e9a..a5e94c27b 100644 --- a/docs/developer/index.md +++ b/docs/developer/index.md @@ -1,37 +1,56 @@ --- -title: Developer Documentation -audience: developers, contributors +title: Contributor Documentation +audience: developers, maintainers, contributors prerequisites: repository checkout -related: development-workflow.md, source-map.md, contributing/index.md +related: architecture.md, packages/index.md, workflows/contributing.md status: maintained publication: draft --- -# Developer Documentation +# Contributor Documentation -This lane is for changing prik. It maps public behavior to implementation and -tests, then provides focused contribution workflows. +This is the single documentation area for changing, testing, governing, and +releasing PRIK. Start with the architecture guide, then open the detailed +package, workflow, concept, design, or active-roadmap page needed for the task. ## Orientation -- [Development workflow](development-workflow.md) -- [Repository structure](repository-structure.md) -- [Source map](source-map.md) -- [Feature-to-code map](feature-to-code-map.md) -- [Compiler preprocessing reference](compiler-preprocessing.md) -- [Fortran parser reference](fortran-parser-reference.md) -- [Quality assurance](quality-assurance.md) -- [Build system](build-system.md) -- [Testing strategy](testing-strategy.md) -- [Coding standards](coding-standards.md) - - - -## Change Workflows - -- [Adding a feature](adding-a-feature.md) -- [Adding a Fortran construct](adding-a-fortran-construct.md) -- [Adding a code-generation backend](adding-a-code-generation-backend.md) -- [Contributing](contributing/index.md) +- [Contributor architecture](architecture.md): shallow repository/package + structure, complete workflow, authority rules, CLI/root files, and package + routes. +- [Source package guides](packages/index.md): one detailed page per production + package, with local structure, important objects, runnable examples, tests, + change routes, and invariants. +- [Source map](source-map.md): exact file and hotspot lookup. +- [Feature-to-code map](feature-to-code-map.md): user-visible capability to + source, tests, and documentation. +- [Testing strategy](testing-strategy.md): language/feature/stage ownership and + verification selection. + +## Cross-Cutting Concepts + +- [Datatype lifecycle](concepts/datatype-lifecycle.md): compiler measurement, + semantic identity, policy, backend representation, and runtime validation. + +## Contributor Workflows + +- [Contributing](workflows/contributing.md) +- [Quality assurance](workflows/quality-assurance.md) +- [Continuous integration and delivery](workflows/ci.md) +- [Documentation architecture](workflows/documentation.md) +- [Release process](workflows/release.md) + +## Design And Planning + +- [Multilanguage runtime architecture](design/multilanguage-runtime.md) is an + explicit long-term design, not a support claim. +- [Wrapper open decisions](design/wrapper-open-decisions.md) records unresolved + or revisitable design questions. +- [Active roadmaps](roadmap/index.md) contain incomplete work only. + +## Deferred Input-Language Material + +The C parser/C-to-IR reference is retained under `deferred/` and excluded from +the published Fortran contributor workflow until that input path is mature. +This does not hide the generated CPython C binding backend used by Fortran +wrappers. diff --git a/docs/developer/packages/codegen.md b/docs/developer/packages/codegen.md new file mode 100644 index 000000000..0a8832b87 --- /dev/null +++ b/docs/developer/packages/codegen.md @@ -0,0 +1,228 @@ +--- +title: Code Generation Package +audience: developers, maintainers, contributors +prerequisites: contributor architecture guide, completed wrapper plan +related: ../architecture.md, index.md, planning.md, printers.md, pipeline.md +status: maintained +publication: draft +--- + +# Code Generation Package + +## Purpose And Boundaries + +`prik/codegen/` consumes a validated wrapper plan and produces typed C and +Fortran syntax nodes plus the planned Python facade source embedded in the +extension. It owns emitted mechanisms such as temporaries, conversions, +bridge bodies, module initialization, and class assembly. It must not complete +ownership, change wrapper support, print final native source, or compile it. + +## Local Structure + +```text +prik/codegen/ +├── __init__.py +├── nodes.py +├── primitive_scalar_types.py +├── docstrings.py +├── overloads.py +├── checks.py +├── visitor.py +├── c/ +│ ├── __init__.py +│ ├── binding.py +│ ├── python_surface.py +│ └── naming.py +└── fortran/ + ├── __init__.py + └── bridge.py +``` + +## What This Stage Receives And Produces + +```text +validated ModulePlan + -> plan-driven public docstrings + -> CBindingGenerator + PythonSurfaceEmitter + -> FortranBridgeGenerator + -> typed C/Fortran nodes and Python facade text + -> language printers +``` + +## Directory Tour + +| Module | Main entrypoints and contents | Change it when | +| --- | --- | --- | +| [`prik/codegen/__init__.py`](../../../prik/codegen/__init__.py) | Re-exports generators, selected node records, scalar lowering, and generic codegen visitor support. | The supported backend API changes. | +| [`prik/codegen/nodes.py`](../../../prik/codegen/nodes.py) | `StageRecord`-based C and Fortran node families represent source before text serialization. | Existing nodes cannot express a plan-selected native construct. | +| [`prik/codegen/primitive_scalar_types.py`](../../../prik/codegen/primitive_scalar_types.py) | `PrimitiveScalarTypeRegistry` and `NumpyDtypeRegistry` map resolved semantic scalars to C, Fortran, NumPy, CFI, and CPython spellings. | An established semantic scalar needs a backend spelling or dtype projection. | +| [`prik/codegen/docstrings.py`](../../../prik/codegen/docstrings.py) | `WrapperDocstringBuilder` renders public Python documentation from a completed plan. | Plan-derived wrapper documentation changes. | +| [`prik/codegen/overloads.py`](../../../prik/codegen/overloads.py) | `OverloadPlanQueries` answers structural questions about completed overload plans. | Shared overload-plan inspection is needed without re-deciding overload policy. | +| [`prik/codegen/checks.py`](../../../prik/codegen/checks.py) | Shared code-generation validation and complexity-check support. | A codegen invariant or its repository gate changes. | +| [`prik/codegen/visitor.py`](../../../prik/codegen/visitor.py) | `ClassVisitor` and `UnsupportedWrapperCodegenNodeError` provide backend-node dispatch and explicit unsupported-node failure. | Generic codegen visitor behavior changes. | +| [`prik/codegen/c/__init__.py`](../../../prik/codegen/c/__init__.py) | Boundary for C/CPython binding mechanics. | Establishing a deliberate C-backend import API. | +| [`prik/codegen/c/binding.py`](../../../prik/codegen/c/binding.py) | `CBindingGenerator` lowers completed binding-plan views into CPython/NumPy C nodes. | A plan-selected Python boundary, lifecycle, error, or module mechanism changes. | +| [`prik/codegen/c/naming.py`](../../../prik/codegen/c/naming.py) | Binding-local generated names that should not become global naming policy. | A C-binding private symbol convention changes. | +| [`prik/codegen/c/python_surface.py`](../../../prik/codegen/c/python_surface.py) | `PythonSurfaceContext` and `PythonSurfaceEmitter` produce planned classes, holders, and module proxies embedded in the extension. | Generated Python facade behavior changes. | +| [`prik/codegen/fortran/__init__.py`](../../../prik/codegen/fortran/__init__.py) | Boundary for Fortran bridge mechanics. | Establishing a deliberate bridge-backend import API. | +| [`prik/codegen/fortran/bridge.py`](../../../prik/codegen/fortran/bridge.py) | `FortranBridgeGenerator` lowers bridge-plan views into `bind(C)` modules, accessors, descriptors, and native calls. | A plan-selected ABI declaration, conversion, call slot, or native bridge mechanism changes. | + +Specialized emitter methods remain local because each makes the selected +mechanism auditable. Shared code must never reconstruct policy from datatype, +source `intent`, dotted shape, aliases, or local memory checks. + +## Execution Examples + +Typed nodes before printing: + +```bash +python3 prik/codegen/nodes.py +``` + +```text +C node tree: CModule -> wrap_ping -> CReturn +Fortran node tree: FortranModule -> bind_c_ping -> FortranCall +Source text rendered: False +``` + +Primitive backend representations: + +```bash +python3 prik/codegen/primitive_scalar_types.py +``` + +```text +Float64: C=double; Fortran=real(c_double); NumPy=numpy.float64 +NumPy C macro: NPY_FLOAT64 +Fresh editable node per lookup: True +``` + +Plan-driven docstrings: + +```bash +python3 prik/codegen/docstrings.py +``` + +```text +double_value(value) -> float64 + +Parameters +---------- +value : float64 + +Returns +------- +result : float64 + +Raises +------ +TypeError + If an argument has an incompatible Python type or dtype. +``` + +The Python facade: + +```bash +python3 prik/codegen/c/python_surface.py +``` + +```text +Rendered Python facade: +_prik_unset = object() + +_prik_ops_state = {} +class State: + 'Opaque native state.' + __slots__ = ('_prik_capsule', '_prik_owner', '_prik_ops', '_prik_origin') + def __new__(cls, *args, **kwargs): + 'Construction is disabled.' + raise TypeError('State objects come from native code.') +def _prik_wrap_State(capsule, owner=None, ops=None, origin='direct'): + ... +``` + +The binding and bridge files also have direct examples: + +```bash +python3 prik/codegen/c/binding.py +``` + +The complete output is 22 lines. These exact selected lines identify the plan +and the native call inside the generated binding node tree: + +```text +Native procedure: DOUBLE_VALUE +Native call slots: implicit:value +C module: binding_demo_wrapper +Header guard: BINDING_DEMO_WRAPPER_H +Header prototypes: wrap_double_value +Binding wrapper: wrap_double_value +... + CExpressionStatement(expression=CodeExpression(text='result = bind_c_double_value(bound_value)')) +... + CReturn(expression=CodeExpression(text='result_obj')) +``` + +```bash +python3 prik/codegen/fortran/bridge.py +``` + +The complete output is 17 lines. Its exact selected lines show the matching +slot and bridge call: + +```text +Native procedure: DOUBLE_VALUE +Native call slots: implicit:value +Bridge module: bind_c_bridge_demo_wrapper +... +Bridge procedure: bind_c_double_value +Binding name: bind_c_double_value +Procedure kind: function +Result: result :: real(c_double) +... + FortranAssignment(target='result', expression=CodeExpression(text='native_double_value(value)')) +Internal procedures: (none) +``` + +Together the outputs demonstrate that both backends lower one shared plan +without asking the other backend to decide policy. + +## Tests And What They Prove + +- [Codegen infrastructure](../../../tests/fortran/infrastructure/codegen/) covers nodes, generators, planning handoffs, and validation. +- [Feature-local codegen suites](../../../tests/fortran/) cover emitted mechanisms for each supported feature. +- [Direct execution inventory](../../../tests/fortran/infrastructure/execution_examples/test_execution_examples.py) fixes every direct module demonstration on this page. +- `python3 tools/check_codegen_complexity.py` protects the generator-complexity policy. + +## Change Routes + +- Add a mechanism to the narrow binding, bridge, or Python-surface emitter that + owns it. +- Add a node only when the existing syntax vocabulary cannot represent the + mechanism. +- Extend primitive lowering only for an established semantic scalar identity. +- If the change requires choosing ownership, storage, projection, setter + exposure, or support, stop and add the missing upstream policy/plan fact. + +## Invariants And Common Mistakes + +- Generators dispatch from completed plan actions; no datatype/intent fallback + may silently choose behavior. +- `WrapperDocstringBuilder` renders the plan and is not imported by planning. +- Large specialized emitters are acceptable when methods remain focused and + policy-free. + +Use one repeatable lowering sequence for every datatype family: + +1. Validate the completed object kind and action combination. +2. Binding generation lowers Python extraction or result construction. +3. Bridge generation lowers ABI declarations, representation conversion, + ordered native call slots, and native result production. +4. Function orchestration applies status handling and planned lifecycle + actions before aggregating Python results. +5. Printers serialize the formed nodes without revisiting the plan's policy. + +A new datatype should extend completed policy and one transfer/result shape, +then add one named validator and one named lowering method per affected +backend. It should not create a parallel module/function plan hierarchy or add +datatype branching to generic traversal. diff --git a/docs/developer/packages/compiler.md b/docs/developer/packages/compiler.md new file mode 100644 index 000000000..77e929746 --- /dev/null +++ b/docs/developer/packages/compiler.md @@ -0,0 +1,135 @@ +--- +title: Compiler Package +audience: developers, maintainers, contributors +prerequisites: contributor architecture guide, native compiler toolchain +related: ../architecture.md, index.md, pipeline.md, runtime.md, ../workflows/quality-assurance.md +status: maintained +publication: draft +--- + +# Compiler Package + +## Purpose And Boundaries + +`prik/compiler/` receives explicit source, object, include, library, flag, and +link inputs and turns them into native commands. It owns compiler-family +profiles, command construction and execution, and native-support installation. +It does not preprocess source, discover build order, probe datatype meaning, +or decide wrapper policy. + +## Local Structure + +```text +prik/compiler/ +├── __init__.py +├── compiler_profiles.py +├── objects.py +├── compilers.py +└── native_support.py +``` + +## What This Stage Receives And Produces + +```text +explicit ObjectFile and link inputs from prik.pipeline + -> coherent compiler-family profile + -> compile/link argv + -> recorded or executed native process + -> object file or shared extension +``` + +The selected Fortran compiler family supplies its matching C driver and +family-specific switches. The pipeline owns dependency-ready batches; the +compiler executes one request at a time. + +## Directory Tour + +| Module | Main entrypoints and contents | Change it when | +| --- | --- | --- | +| [`prik/compiler/__init__.py`](../../../prik/compiler/__init__.py) | Deliberately empty package boundary; callers use the owning modules or pipeline APIs. | Establishing a small compiler-package public import surface. | +| [`prik/compiler/compiler_profiles.py`](../../../prik/compiler/compiler_profiles.py) | Profile data and `fortran_compiler_family()` map a Fortran executable to its compatible C driver and flags. | Supporting a compiler family or changing family-specific build settings. | +| [`prik/compiler/objects.py`](../../../prik/compiler/objects.py) | `ObjectFile` is the immutable description of one source-to-object request. | A compilation input needs another explicit field or validation rule. | +| [`prik/compiler/compilers.py`](../../../prik/compiler/compilers.py) | `Compiler` builds, records, runs, and reports compile/link commands; `get_condaless_search_path()` isolates environment lookup. | Command spelling, subprocess execution, or command reporting changes. | +| [`prik/compiler/native_support.py`](../../../prik/compiler/native_support.py) | `install_native_support()` copies the bundled support payload and creates the NumPy API-version header. | The pipeline needs a different support-installation result; edit the payload itself under `runtime/native_support/`. | + +## Execution Examples + +Compiler-family selection: + +```bash +python3 prik/compiler/compiler_profiles.py +``` + +```text +Selected family: gfortran +Compiler profile: GNU +Matching C executable: gcc +Fortran module-output flag: -J +``` + +One immutable compilation request: + +```bash +python3 prik/compiler/objects.py +``` + +```text +Compile input: generated/bridge.f90 -> build/bridge.o +Language: fortran +Flags: ('-O2',) +Include directories: build/modules +``` + +Record-only command construction: + +```bash +python3 prik/compiler/compilers.py +``` + +```text +Compiler profile: GNU +Compile input: demo.c -> demo.o +Recorded without execution: True +Contains compile switch: True +Contains requested flag: True +Commands recorded: 1 +``` + +Bundled runtime installation: + +```bash +python3 prik/compiler/native_support.py +``` + +```text +Installed directory: binding_support +Binding header present: True +NumPy version header present: True +``` + +Together these outputs prove that profile selection, request construction, +native command mechanics, and support installation remain separate operations. + +## Tests And What They Prove + +- [Compiler construction tests](../../../tests/fortran/building_shared_library/compiling/) cover profile selection and compile/link argv. +- [Build pipeline tests](../../../tests/fortran/building_shared_library/pipeline/) cover compiler handoff from a build plan. +- [Source build modes](../../../tests/fortran/building_shared_library/end_to_end/test_source_build_modes.py) covers real source-build outcomes. +- [Runtime ABI compatibility](../../../tests/fortran/building_shared_library/end_to_end/test_runtime_compatibility.py) covers installed support used by a compiled extension. +- [Direct execution inventory](../../../tests/fortran/infrastructure/execution_examples/test_execution_examples.py) fixes the four demonstrations above. + +## Change Routes + +- Change driver families or flags in `compiler_profiles.py`. +- Change compile/link argv or subprocess reporting in `compilers.py`. +- Change build order, parallel scheduling, manifests, or artifact names in + `prik/pipeline/build.py`. +- Change native payload contents in `prik/runtime/native_support/`; change only + their installation here. + +## Invariants And Common Mistakes + +- Never infer ownership, dtype, Python API shape, or wrapper support here. +- Never silently mix a selected Fortran driver with an unrelated C profile. +- Each invocation receives explicit inputs; hidden project discovery belongs + upstream. diff --git a/docs/developer/packages/contracts.md b/docs/developer/packages/contracts.md new file mode 100644 index 000000000..18e55e80c --- /dev/null +++ b/docs/developer/packages/contracts.md @@ -0,0 +1,96 @@ +--- +title: Contracts Package +audience: developers, maintainers, contributors +prerequisites: contributor architecture guide, semantic .pyi format +related: index.md, parsers.md, semantics.md, ../architecture.md +status: maintained +publication: draft +--- + +# Contracts Package + +## Purpose And Boundaries + +`prik/contracts/` owns the public names written in semantic `.pyi` contracts. +Those names describe scalar types, arrays, storage, ownership requests, +projections, native calls, callbacks, and descriptor handles. The package is a +public syntax vocabulary; it does not define semantic IR, complete policy, or +generate wrappers. + +## Local Structure + +```text +prik/contracts/ +└── __init__.py +``` + +The single module is intentional. A semantic contract imports one stable +public namespace instead of depending on internal stage packages. + +## What This Stage Receives And Produces + +```text +semantic .pyi text + -> names imported from prik.contracts + -> Python AST in prik.parsers.pyi + -> contract interpretation in prik.semantics.pyi2ir + -> completed policy in prik.policy +``` + +Some primitive symbols also construct exact NumPy scalar values at runtime. +Subscriptions such as `Float64[:, :]` construct declarative contract objects; +they do not create semantic IR objects. + +## Directory Tour + +| Module | Main entrypoints and contents | Change it when | +| --- | --- | --- | +| [`prik/contracts/__init__.py`](../../../prik/contracts/__init__.py) | The complete public vocabulary: scalar and array markers, descriptor markers (`Allocatable`, `Pointer`), metadata expressions, decorators, and the small runtime constructors behind concrete scalar and descriptor contracts. | Adding, removing, or documenting public `.pyi` syntax. This one file is intentionally the stable import namespace; private `_Contract*` classes preserve annotation syntax at runtime. | + +The canonical public import path is part of the file format. Internal code may +interpret these names, but must not replace them with imports from semantics, +policy, or codegen. + +## Execution Example + +Run the real package entry file: + +```bash +python3 prik/contracts/__init__.py +``` + +```text +Float64() -> np.float64(0.0) (float64) +Float64[:, :] -> element=Float64, rank=2, shape=(slice(None, None, None), slice(None, None, None)) +``` + +The first line proves that a primitive contract scalar has exact NumPy runtime +behavior. The second proves that array subscription produces declarative rank +and shape syntax for later semantic interpretation. + +## Tests And What They Prove + +- [Contract runtime tests](../../../tests/fortran/data_types/runtime/) protect scalar and descriptor-constructor behavior. +- [Semantic `.pyi` parser tests](../../../tests/fortran/semantic_pyi_format/parsing/) protect recognition of the public vocabulary. +- [Semantic `.pyi` round-trip tests](../../../tests/fortran/semantic_pyi_format/pipeline/) protect loading and re-emission through the shared contract path. +- [Direct execution inventory](../../../tests/fortran/infrastructure/execution_examples/test_execution_examples.py) fixes the example output above. + +## Change Routes + +- Add or rename public syntax here first, then update `.pyi` parsing, + conversion, printing, user reference documentation, and focused round-trip + tests. +- Change semantic meaning in `prik/semantics/pyi2ir.py`, not in a runtime + constructor. +- Change ownership or lowering selection in policy after semantic conversion. + +## Invariants And Common Mistakes + +- Keep `prik.contracts` stable and public; do not expose internal policy models + through this namespace. +- A valid Python annotation is not automatically a supported wrapper contract. +- NumPy construction behavior must not become the semantic datatype authority. + +See the [semantic `.pyi` user reference](../../user/reference/semantic-pyi-format.md) +for the public language and the [semantics package](semantics.md) for its IR +interpretation. diff --git a/docs/developer/packages/index.md b/docs/developer/packages/index.md new file mode 100644 index 000000000..33367f5eb --- /dev/null +++ b/docs/developer/packages/index.md @@ -0,0 +1,46 @@ +--- +title: Source Package Guides +audience: developers, maintainers, contributors +prerequisites: contributor architecture guide +related: ../architecture.md, ../source-map.md, ../feature-to-code-map.md +status: maintained +publication: draft +--- + +# Source Package Guides + +These pages are the file-level companion to the +[architecture guide](../architecture.md). Read the architecture guide once for +the whole flow, then use this table to enter the owner of a change. Do not read +the guides as thirteen alternative pipelines: each describes one handoff in +the same pipeline. + +| Package | Read it when you need to change | Canonical guide | +| --- | --- | --- | +| `prik.contracts` | public semantic `.pyi` syntax | [Contracts](contracts.md) | +| `prik.compiler` | compiler profiles, command argv, or native-support installation | [Compiler](compiler.md) | +| `prik.preprocessing` | parser input, provenance, includes, or target probes | [Preprocessing](preprocessing.md) | +| `prik.parsers` | Fortran syntax facts or raw `.pyi` syntax | [Parsers](parsers.md) | +| `prik.semantics` | the shared semantic graph, types, or raw metadata | [Semantics](semantics.md) | +| `prik.policy` | completed ownership, projection, lifecycle, or support choices | [Policy](policy.md) | +| `prik.planning` | plan representation, ordering, or backend views | [Planning](planning.md) | +| `prik.codegen` | generated binding, bridge, node, or Python-facade mechanism | [Code generation](codegen.md) | +| `prik.printers` | C, Fortran, or `.pyi` text serialization | [Printers](printers.md) | +| `prik.pipeline` | wrapper, contract, report, artifact, or build orchestration | [Pipeline](pipeline.md) | +| `prik.runtime` | imported native handles or bundled native support | [Runtime](runtime.md) | +| `prik.naming` | public-name normalization or generated symbols | [Naming](naming.md) | +| `prik.utilities` | a genuinely stage-neutral helper | [Utilities](utilities.md) | + +Each guide answers the same practical questions: + +1. What does this stage receive and produce? +2. Which module owns the behavior I need to change? +3. Which classes and functions are the important entrypoints? +4. What does each direct-execution example prove? +5. Which tests protect that behavior? + +The directory tour covers every supported Python module under that package, +including package initializers and nested backend packages. The deferred +C-input frontend is intentionally excluded from the published Fortran route. +Source-tree `README.md` files remain short orientation notes and link back to +these canonical guides. diff --git a/docs/developer/packages/naming.md b/docs/developer/packages/naming.md new file mode 100644 index 000000000..804738932 --- /dev/null +++ b/docs/developer/packages/naming.md @@ -0,0 +1,82 @@ +--- +title: Naming Package +audience: developers, maintainers, contributors +prerequisites: contributor architecture guide +related: ../architecture.md, index.md, planning.md, codegen.md, ../source-map.md +status: maintained +publication: draft +--- + +# Naming Package + +## Purpose And Boundaries + +`prik/naming/` owns public and generated names whose stability and collision +rules are shared across planning and generation. It does not own semantic +policy or emitted source syntax. + +## Local Structure + +```text +prik/naming/ +├── __init__.py +├── policy.py +└── native_symbols.py +``` + +## What This Stage Receives And Produces + +```text +raw public or generated identity + occupied namespace + -> normalized public name or bounded native symbol + -> planning and code generation +``` + +## Directory Tour + +| Module | Main entrypoints and contents | Change it when | +| --- | --- | --- | +| [`prik/naming/__init__.py`](../../../prik/naming/__init__.py) | Re-exports the supported normalization and generated-symbol policy objects. | Changing the package-level naming API. | +| [`prik/naming/policy.py`](../../../prik/naming/policy.py) | `NamingPolicy`, `NormalizedPublicName`, `PublicNameRecord`, and `GeneratedSymbolRules` normalize Python names, reserve namespaces, and apply language rules. | Public-name normalization, collision handling, keyword escaping, or target language symbol rules. | +| [`prik/naming/native_symbols.py`](../../../prik/naming/native_symbols.py) | `NativeSymbolNames` retains owner identity and creates compact, deterministic compiler-safe fragments. | Bounded native-symbol spelling or hash/prefix rules. | + +## Execution Examples + +```bash +python3 prik/naming/policy.py +``` + +```text +Normalized public name: render_value +Collision-safe public name: render_value_2 +C destructor symbol: state_drop +``` + +```bash +python3 prik/naming/native_symbols.py +``` + +```text +Owner identity: geometry.point.coordinates +Stable native symbol: point_coordinate_d_c2fc5940 +Within 27-character limit: True +``` + +The first example distinguishes public namespace allocation from generated +target naming. The second preserves a readable prefix while hashing the full +owner identity under a compiler symbol limit. + +## Tests And What They Prove + +- [Naming infrastructure](../../../tests/fortran/infrastructure/naming/) covers normalization, collisions, and stable generated names. +- [Direct execution inventory](../../../tests/fortran/infrastructure/execution_examples/test_execution_examples.py) fixes the two demonstrations above. + +## Change Routes + +- Change public normalization and collision policy in `policy.py`. +- Change stable ABI fragments in `native_symbols.py` with exact-name tests. + +## Invariants And Common Mistakes + +- Never consult completed ownership or emit language syntax here. +- The same inputs must always produce the same generated symbol. diff --git a/docs/developer/packages/parsers.md b/docs/developer/packages/parsers.md new file mode 100644 index 000000000..0e191136e --- /dev/null +++ b/docs/developer/packages/parsers.md @@ -0,0 +1,180 @@ +--- +title: Parsers Package +audience: developers, maintainers, contributors +prerequisites: contributor architecture guide, prepared source +related: ../architecture.md, index.md, preprocessing.md, semantics.md, ../source-map.md +status: maintained +publication: draft +--- + +# Parsers Package + +## Purpose And Boundaries + +`prik/parsers/` owns syntax-level facts. The Fortran frontend preserves source +units, declarations, visibility, locations, and diagnostics. The semantic +`.pyi` frontend deliberately stops at a standard Python AST. A parser reports +what its input says; it does not assign a stable semantic type, choose +ownership, decide wrapper support, or emit a Python API. + +The C-input frontend is intentionally deferred from the published contributor +workflow. This guide covers the supported Fortran and semantic-`.pyi` path; +generated C binding remains documented under [code generation](codegen.md). + +## Local Structure + +```text +prik/parsers/ +├── __init__.py +├── fortran/ +│ ├── __init__.py +│ ├── __main__.py +│ ├── cli.py +│ ├── lexer.py +│ ├── models.py +│ ├── parser.py +│ ├── type_resolver.py +│ └── utils.py +└── pyi/ + ├── __init__.py + └── parser.py +``` + +## What This Stage Receives And Produces + +```text +prepared Fortran text + -> logical lines with original locations + -> Fortran parser models and diagnostics + -> Fortran-to-IR conversion + +semantic .pyi text + -> ast.Module + -> .pyi-to-IR conversion +``` + +Fortran parser models retain source spellings such as `real(kind=...)`, +`intent`, and declaration shapes. Target-dependent kind values arrive from +preprocessing probes and are resolved in semantic conversion, not here. + +## Directory Tour + +| Module | Main entrypoints and contents | Change it when | +| --- | --- | --- | +| [`prik/parsers/__init__.py`](../../../prik/parsers/__init__.py) | Declares the parser frontend namespaces. | The package-level frontend layout changes. | +| [`prik/parsers/fortran/__init__.py`](../../../prik/parsers/fortran/__init__.py) | Re-exports the supported Fortran parser API: parser functions, `FortranParser`, parser models, and `FortranParseError`. | The supported Fortran-parser import API changes. | +| [`prik/parsers/fortran/__main__.py`](../../../prik/parsers/fortran/__main__.py) | Module launcher for `python3 -m prik.parsers.fortran`; delegates to the CLI. | Module-launch behavior changes, not parser semantics. | +| [`prik/parsers/fortran/utils.py`](../../../prik/parsers/fortran/utils.py) | `detect_source_form()` and `split_csv()` are small, grammar-neutral lexical helpers. | Source-form detection or top-level comma splitting changes. | +| [`prik/parsers/fortran/lexer.py`](../../../prik/parsers/fortran/lexer.py) | `strip_comment()` and `preprocess_lines()` remove comments, fold continuations, and retain logical-line locations. | Lexical normalization or source-coordinate retention changes. | +| [`prik/parsers/fortran/models.py`](../../../prik/parsers/fortran/models.py) | Passive parser records including `FortranFile`, `FortranProject`, `FortranModule`, variables, signatures, derived types, enums, shapes, and `FortranParseError`. | A parser-level source fact or diagnostic representation changes. | +| [`prik/parsers/fortran/type_resolver.py`](../../../prik/parsers/fortran/type_resolver.py) | `extract_kind_from_type_spec()` preserves type, kind, and character syntax without measuring its meaning. | Parser-level type-spec spelling extraction changes. | +| [`prik/parsers/fortran/parser.py`](../../../prik/parsers/fortran/parser.py) | `FortranParser`, source-unit records, `parse_fortran_file()`, and `parse_fortran_project()` slice units, build models, resolve parser-level scope, and order projects. | Grammar, declaration extraction, source-unit structure, parser diagnostics, or project ordering changes. | +| [`prik/parsers/fortran/cli.py`](../../../prik/parsers/fortran/cli.py) | `main()` turns parser requests into stable human or JSON reports. | Parser CLI arguments or report presentation changes. | +| [`prik/parsers/pyi/__init__.py`](../../../prik/parsers/pyi/__init__.py) | Re-exports `parse_pyi_text()` and `parse_pyi_file()`. | The supported raw-`.pyi` parser import surface changes. | +| [`prik/parsers/pyi/parser.py`](../../../prik/parsers/pyi/parser.py) | `parse_pyi_text()` and `parse_pyi_file()` validate and return `ast.Module` without semantic interpretation. | Accepted Python syntax or raw parse diagnostics change. | + +Read `fortran/parser.py` by entrypoint, then source-unit scanning, then the +visitor that owns the construct you are changing. Do not add policy or codegen +conditions to a parser visitor: preserve the fact and let the next stage +decide whether it is supported. + +## Execution Examples + +Logical-line preparation: + +```bash +python3 prik/parsers/fortran/lexer.py +``` + +```text +Detected source form: free +line 1: subroutine shift(value,offset) +line 3: real, intent(inout) :: value +line 4: real, intent(in) :: offset +line 5: end subroutine shift +``` + +Fortran file parsing: + +```bash +python3 prik/parsers/fortran/parser.py +``` + +```text +Module: metrics +Parameter: n = 4 +Procedure: scale(values: real[1]) +``` + +Type-spec preservation: + +```bash +python3 prik/parsers/fortran/type_resolver.py +``` + +```text +integer(4) -> 4 +real(kind=selected_real_kind(15, 307)) -> selected_real_kind(15, 307) +character(len=16, kind=c_char) -> len=16, kind=c_char +``` + +Parser report formatting: + +```bash +python3 prik/parsers/fortran/cli.py +``` + +```text +File: geometry.f90 + Modules: 1 + - module geometry (vars=0, uses=0) + Procedures: 1 + - function norm(value:real[0]) -> real[0] +``` + +Raw semantic-`.pyi` parsing: + +```bash +python3 prik/parsers/pyi/parser.py +``` + +```text +Parsed AST: Module +Function node: scale +Argument annotation: Float64 +Semantic conversion performed: False +``` + +These outputs are intentionally parse-only. They show preserved source facts, +not a completed `SemanticModule`, wrapper plan, or generated source. + +## Tests And What They Prove + +- [Fortran parser tests](../../../tests/fortran/source_parsing/parsing/) cover source forms, units, declarations, diagnostics, and project ordering. +- [Fortran parser CLI tests](../../../tests/fortran/command_line_interface/pipeline/) cover parser command dispatch and report output. +- [Semantic `.pyi` parsing tests](../../../tests/fortran/semantic_pyi_format/parsing/) cover raw `.pyi` AST parsing and diagnostics. +- [Semantic IR conversion tests](../../../tests/fortran/semantic_ir/semantics/) prove the downstream Fortran-model handoff. +- [Direct execution inventory](../../../tests/fortran/infrastructure/execution_examples/test_execution_examples.py) fixes the five demonstrations above. + +## Change Routes + +- Change source form, comments, continuations, or logical locations in + `fortran/utils.py` or `fortran/lexer.py`. +- Change parser facts in `fortran/models.py`; change grammar and source-unit + construction in `fortran/parser.py`. +- Change parser report layout in `fortran/cli.py`. +- Change only raw `.pyi` AST parsing in `pyi/parser.py`; put meaning in + `semantics/pyi2ir.py`. +- If a change needs target kind values, use preprocessing probes; if it needs + ownership, projection, or support, use policy after semantic conversion. + +## Invariants And Common Mistakes + +- Preserve original source locations through lexical and structural parsing. +- Keep parser models passive and source-faithful; do not attach completed + policy to them. +- `parse_fortran_project()` only receives explicit project files; it does not + invent recursive source discovery. +- A construct that parses successfully is not automatically wrapper support. +- The `.pyi` parser returns Python AST. Contract interpretation starts only in + `semantics/pyi2ir.py`. diff --git a/docs/developer/packages/pipeline.md b/docs/developer/packages/pipeline.md new file mode 100644 index 000000000..5603b92ea --- /dev/null +++ b/docs/developer/packages/pipeline.md @@ -0,0 +1,159 @@ +--- +title: Pipeline Package +audience: developers, maintainers, contributors +prerequisites: contributor architecture guide, package guides for participating stages +related: ../architecture.md, index.md, compiler.md, planning.md, codegen.md, printers.md +status: maintained +publication: draft +--- + +# Pipeline Package + +## Purpose And Boundaries + +`prik/pipeline/` composes complete workflows across established stage +boundaries. It selects the next stage, preserves progress and timing, assigns +artifact names, writes generated payloads, coordinates compilation/linking, +and returns public results. It does not absorb parser grammar, semantic rules, +policy, backend lowering, printer formatting, or compiler command mechanics. + +## Local Structure + +```text +prik/pipeline/ +├── __init__.py +├── pyi.py +├── type_mapping_report.py +├── wrapper.py +└── build.py +``` + +## What This Stage Receives And Produces + +```text +semantic modules or source-build request + -> completed policy and WrapperPlanner + -> WrapperGenerator + -> backend node generation + -> language printers + -> GeneratedWrapper + -> build.py writes sources and creates NativeBuildPlan + -> prik.compiler compiles and links + -> WrapperBuildResult +``` + +## Directory Tour + +| Module | Main entrypoints and contents | Change it when | +| --- | --- | --- | +| [`prik/pipeline/__init__.py`](../../../prik/pipeline/__init__.py) | Package boundary for high-level workflows. | Establishing a deliberate pipeline-level import API. | +| [`prik/pipeline/pyi.py`](../../../prik/pipeline/pyi.py) | `pyi_*_to_semantic_module()` workflows and `emit_module_stubs()` load text, files, and path sets; cache one operation; reconcile external types; and emit stub packages. | `.pyi` batch loading, external-type reconciliation, per-operation cache behavior, or stub-package output changes. | +| [`prik/pipeline/type_mapping_report.py`](../../../prik/pipeline/type_mapping_report.py) | Report builders connect target probes, semantic conversion, and backend dtype projection into an auditable table. | Cross-stage datatype-report content or evidence changes. | +| [`prik/pipeline/wrapper.py`](../../../prik/pipeline/wrapper.py) | `GeneratedSource`, `GeneratedWrapper`, and `WrapperGenerator` validate/freeze a plan, invoke docstring and backend generation, print sources, name artifacts, and return one in-memory wrapper. | Plan-to-rendered-wrapper orchestration changes. | +| [`prik/pipeline/build.py`](../../../prik/pipeline/build.py) | `NativeCompilationUnit`, `NativePrebuiltArtifact`, `NativeLinkItem`, `NativeBuildPlan`, and `WrapperBuildResult` own public build APIs, output, manifests, dependency-ready compilation, linking, and extension import. | Artifact layout, native input plans, build scheduling, manifests, linking, or imported-result behavior changes. | + +## Execution Examples + +```bash +python3 prik/pipeline/pyi.py +``` + +```text +Loaded semantic module: math +Loaded contract marker: True +Functions: scale +Re-emitted module: +from prik.contracts import Float64 + +def scale( + value: Float64 +) -> Float64: ... +``` + +```bash +python3 prik/pipeline/type_mapping_report.py +``` + +```text +| `int` | signed 32-bit | `Int (Int32 storage)` | `numpy.int32` | +``` + +The exact width depends on the active target and requires a C compiler. The +columns expose native spelling, measured fact, semantic identity, and NumPy +projection rather than hiding them behind one universal datatype table. + +```bash +python3 prik/pipeline/wrapper.py +``` + +```text +Extension initializer: PyInit_generator_demo +Rendered sources: bind_c_generator_demo_wrapper.f90, generator_demo_wrapper.c, generator_demo_wrapper.h +Native support: binding_support +``` + +This result is still in memory: no file has been written or compiled. + +```bash +python3 prik/pipeline/build.py +``` + +```text +scale(3.0, 2.5) = 7.5 +``` + +The final example requires configured C and Fortran compilers. It follows the +entire public source-build path, imports the resulting extension, and calls its +generated Python API. + +## Tests And What They Prove + +- [Pipeline infrastructure](../../../tests/fortran/infrastructure/pipeline/) covers wrapper assembly and cross-stage records. +- [Semantic `.pyi` pipeline](../../../tests/fortran/semantic_pyi_format/pipeline/) covers contract loading, reconciliation, and stub emission. +- [Build pipeline](../../../tests/fortran/building_shared_library/pipeline/) covers files, manifests, and build-plan handoffs. +- [Compilation integration](../../../tests/fortran/building_shared_library/compiling/) covers native command integration. +- [End-to-end builds](../../../tests/fortran/building_shared_library/end_to_end/) covers produced extension behavior. +- [Direct execution inventory](../../../tests/fortran/infrastructure/execution_examples/test_execution_examples.py) fixes the four demonstrations above. + +## Change Routes + +- Change `.pyi` batch loading, reconciliation, or caching in `pyi.py`. +- Change cross-stage datatype reporting in `type_mapping_report.py`. +- Change plan-to-artifact orchestration in `wrapper.py`. +- Change disk output, manifests, native build requests, compilation scheduling, + linking, or imports in `build.py`. + +## Invariants And Common Mistakes + +- `WrapperGenerator` owns plan-to-rendered-wrapper orchestration, not semantic + decisions and not native compilation. +- Per-operation semantic caches must not become process-global because later + stages attach and freeze data. +- A pipeline helper delegates domain rules to their owning package. +- There is one direct generation route and no legacy retry: + + ```python + complete_semantic_policies(module) + plan = WrapperPlanner().build(module) + generated = WrapperGenerator().generate(plan) + ``` + + Unsupported completed policy fails with its exact owner before either + backend emits source. +- `.pyi` builds reuse the same backend but take API/ABI facts from one edited + entry contract plus explicit native inputs; they never reparse native source + to reconstruct the Python API. + +## Failure Ownership + +| Failure | Earliest owner | +| --- | --- | +| Compiler preprocessing or native include expansion | preprocessing | +| Required target facts cannot be measured | preprocessing probe | +| Source syntax cannot be represented | parser | +| Source facts cannot form a contract | semantic conversion | +| Lifetime, ABI, projection, or support is unsafe | policy completion | +| Completed policy is inconsistent while projected | planning | +| A supported plan lacks an emitted mechanism | binding or bridge generator | +| Native command or link plan is wrong | compiler or build pipeline | +| Imported runtime behavior is wrong | generated binding, runtime support, or upstream policy according to cause | diff --git a/docs/developer/packages/planning.md b/docs/developer/packages/planning.md new file mode 100644 index 000000000..66488f312 --- /dev/null +++ b/docs/developer/packages/planning.md @@ -0,0 +1,128 @@ +--- +title: Planning Package +audience: developers, maintainers, contributors +prerequisites: contributor architecture guide, completed policy +related: ../architecture.md, index.md, policy.md, codegen.md, ../source-map.md +status: maintained +publication: draft +--- + +# Planning Package + +## Purpose And Boundaries + +`prik/planning/` mechanically projects policy-completed semantic IR into one +backend-neutral `ModulePlan`. It joins common transfer facts with explicit +binding and bridge views, namespaces, stable native symbols, lifecycle order, +and build requirements. It may organize and validate completed decisions; it +may not reinterpret source declarations, choose policy, or render text. + +## Local Structure + +```text +prik/planning/ +├── __init__.py +├── models.py +└── planner.py +``` + +## What This Stage Receives And Produces + +```text +policy-completed SemanticModule + -> WrapperPlanner validation and projection + -> editable ModulePlan + -> freeze at WrapperGenerator boundary + -> backend node generation +``` + +## Directory Tour + +| Module | Main entrypoints and contents | Change it when | +| --- | --- | --- | +| [`prik/planning/__init__.py`](../../../prik/planning/__init__.py) | Re-exports `WrapperPlanner` and the supported plan records. | A supported planning type or import path changes. | +| [`prik/planning/models.py`](../../../prik/planning/models.py) | `ModulePlan` and typed function, argument, result, slot, lifecycle, class, overload, binding, and bridge records form the editable plan tree. | Lowering needs a new *already completed* fact represented explicitly. | +| [`prik/planning/planner.py`](../../../prik/planning/planner.py) | `WrapperPlanner` validates policy, indexes declarations, allocates names, and projects deterministic binding and bridge views; `_ClassPolicyCatalog` is a validated lookup. | A completed policy fact is projected or ordered incorrectly. | + +The private class-policy catalogue is a validated lookup, not another semantic +authority. The planner does not generate docstrings or source. + +The stable plan tree keeps orchestration at module, namespace, and function +levels and confines datatype variation to transfers, results, lifecycle +actions, and module variables: + +```text +ModulePlan + -> binding and bridge module views + -> NamespacePlan + -> FunctionPlan + -> ArgumentTransferPlan + -> ResultPlan + -> NativeCallSlotPlan + -> LifecycleActionPlan + -> ModuleVariablePlan +``` + +Each argument or result owns explicit binding and bridge views. Its native-call +slot is the same record referenced from the transfer and the function-wide ABI +ordering index, not a duplicated policy fact. Function orchestration owns call, +result, lifecycle, GIL, and status order without becoming datatype policy. + +`OverloadPlan` stores ordered candidates, exact match records, receiver +conventions, and one candidate ID per overload set. Generated dispatch chooses +an ID before making a native call, preserving first-match behavior for +overlapping optional domains without speculative calls. + +## Execution Examples + +```bash +python3 prik/planning/models.py +``` + +```text +Plan owner: demo +Python export: ping +Native procedure: PING +Native slots: 0 +``` + +```bash +python3 prik/planning/planner.py +``` + +```text +Plan owner: planner_demo +Python export: double_value +Native target: DOUBLE_VALUE +Conversion order: ('planner_demo.double_value.value',) +``` + +The model example demonstrates representation. The planner example follows +the real sequence—semantic IR, policy completion, then planning—and shows the +stable role connecting binding conversion to the native call slot. + +## Tests And What They Prove + +- [Plan model tests](../../../tests/fortran/infrastructure/codegen/test_plan.py) protect plan-record shape and freeze behavior. +- [Planner tests](../../../tests/fortran/infrastructure/codegen/test_planner.py) protect validation, projection, symbols, and order. +- [Feature-local codegen stages](../../../tests/fortran/) protect plan use for each supported feature. +- [Direct execution inventory](../../../tests/fortran/infrastructure/execution_examples/test_execution_examples.py) fixes the model and planner outputs above. + +## Change Routes + +- Add a plan field only for an already completed fact needed by lowering. +- Change projection or indexing in `planner.py`. +- Change ownership, mutability, projection, setter exposure, or support in + policy first. +- Change emitted temporaries or syntax downstream in codegen. + +## Invariants And Common Mistakes + +- Missing completed policy is an error, never a reason to infer a default. +- Binding and bridge views may share one ABI contract without hiding their + backend-specific lowering facts. +- Planning does not depend on presentation helpers such as docstring builders. +- Native slots may interleave argument, result, literal, and helper positions; + keep their function-wide order explicit. +- Lifecycle actions stay explicit because cleanup and writeback order may span + several transfers and differ on failure. diff --git a/docs/developer/packages/policy.md b/docs/developer/packages/policy.md new file mode 100644 index 000000000..376f069e7 --- /dev/null +++ b/docs/developer/packages/policy.md @@ -0,0 +1,215 @@ +--- +title: Policy Package +audience: developers, maintainers, contributors +prerequisites: contributor architecture guide, semantic IR +related: ../architecture.md, index.md, semantics.md, planning.md, runtime.md, ../../user/guide/memory-management.md +status: maintained +publication: draft +--- + +# Policy Package + +## Purpose And Boundaries + +`prik/policy/` is the final semantic authority before planning. It turns raw +semantic facts and metadata into complete immutable interoperability decisions: +public exports, object kind, owner, transfer, destruction, storage, mutability, +writeback, nullability, projection, lifecycle, descriptor operations, setter +behavior, and support blockers. + +Planning, binding, bridge, and runtime code consume these decisions. They may +validate and dispatch from them, but may not infer an alternative answer from +datatype, source `intent`, dotted-variable shape, `is_alias`, or local memory +checks. + +## Local Structure + +```text +prik/policy/ +├── __init__.py +├── models.py +├── ownership.py +├── exports.py +├── construction.py +├── completion.py +└── native_array_handles.py +``` + +## What This Stage Receives And Produces + +```text +SemanticModule + normalized raw metadata + -> export completion and semantic graph completion + -> ownership, callable, class, result, and descriptor-policy construction + -> complete_semantic_policies() + -> immutable completed policy attached to semantic IR + -> WrapperPlanner +``` + +Completion is ordered because later decisions depend on earlier facts. A +blocked decision records its owner path and reason; it is never replaced by a +downstream fallback. + +## Directory Tour + +| Module | Main entrypoints and contents | Change it when | +| --- | --- | --- | +| [`prik/policy/__init__.py`](../../../prik/policy/__init__.py) | Re-exports `complete_semantic_policies()` as the normal policy-stage entrypoint. | The supported policy import surface changes. | +| [`prik/policy/models.py`](../../../prik/policy/models.py) | Immutable records and enums for function, argument, result, slot, lifecycle, class, overload, callback, array, descriptor, status, and transformation policy. | A completed decision needs a durable backend-neutral representation. | +| [`prik/policy/ownership.py`](../../../prik/policy/ownership.py) | Ownership vocabulary, `OwnershipContext`, `OwnershipDecision`, `OwnershipPolicyResolver`, and action dispatchers resolve lifetime triples and fail-closed lowering actions. | Object kind, owner, transfer, destruction, storage, barrier, assignment, or setter selection changes. | +| [`prik/policy/exports.py`](../../../prik/policy/exports.py) | `PythonExportPolicy`, `complete_python_export_policy()`, and `completed_python_exports()` create collision-checked Python placement. | Export namespace, visibility, or collision behavior changes. | +| [`prik/policy/construction.py`](../../../prik/policy/construction.py) | Feature constructors build coherent function, result, native-slot, callback, class, overload, and module-variable policies from completed ownership decisions. | A supported feature needs different completed policy composition. | +| [`prik/policy/completion.py`](../../../prik/policy/completion.py) | `complete_semantic_policies()` runs the dependency-ordered completion pass, attaches outcomes, and validates blockers. | Completion order, cross-declaration completion, or the stage boundary changes. | +| [`prik/policy/native_array_handles.py`](../../../prik/policy/native_array_handles.py) | `NativeArrayHandlePolicy`, interop/handle/projection dispatchers, and build-requirement records complete descriptor operations and selected ABI/build requirements. | Descriptor-backed array behavior, ABI selection, allowed operations, or build headers change. | + +Start with `completion.py` to see the order, follow its call into the focused +resolver or constructor, and finish in `models.py` to confirm the durable +output. Do not begin in code generation when the question is semantic. + +## How To Read A Completed Decision + +Policy keeps related questions separate. This makes aliases, copies, views, +and cleanup auditable instead of encoding them in one overloaded `owned` +flag. + +| Question | Main vocabulary | Example answer | +| --- | --- | --- | +| What Python-facing family is this? | `ObjectKind` | `SCALAR`, `STRING`, `NUMPY_ARRAY`, `DERIVED_TYPE` | +| Who owns the represented storage? | `OwnershipOwner` | `CALLER`, `NATIVE`, `WRAPPER`, `TEMPORARY` | +| How does value or storage cross the boundary? | `TransferMode` | `BY_VALUE`, `IN_PLACE`, `COPY_RETURN`, `BORROWED_VIEW` | +| Who releases a resource? | `DestructionPolicy` | `CALLER`, `NATIVE_OWNER`, `WRAPPER_DEALLOC`, `CALL_LOCAL` | +| Where is the contract value stored? | `StorageMode` | `STACK`, `HEAP`, `ALIAS` | +| What does each boundary do? | `PythonBarrierAction`, `NativeBarrierAction`, `CodegenAction` | extract storage, pass a descriptor, copy out, construct a wrapper | +| How may native storage be assigned or exposed? | `AssignmentMode`, `SetterAction` | value copy, alias, write-through, omit setter | + +Read the lifetime triple left to right. For example, +`NATIVE + BORROWED_VIEW + NATIVE_OWNER` means Python observes live native +storage but does not own or release it. `PYTHON + COPY_RETURN + +PYTHON_REFCOUNT` means that PRIK creates an independent Python-owned result. +Only supported combinations are lowered; contradictory or unimplemented +combinations become explicit blockers. + +For every lowering-ready value, policy completion must answer all of the +following before `WrapperPlanner.build()`: + +1. Object kind and public projection. +2. Owner, transfer, destruction, and contract storage mode. +3. Python and native barrier actions, including ordered native call slots. +4. Mutability, writeback, nullability, lifecycle, release responsibility, + getter behavior, native setter assignment, and Python setter exposure. +5. Supported mechanism or an explicit blocked diagnostic. + +The binding and bridge may create local temporary variables inside a selected +implementation method, but those are emitted-code details. They are not a +license to choose a new semantic policy. + +## Execution Examples + +Completed record immutability: + +```bash +python3 prik/policy/models.py +``` + +```text +Array policy: rank=2, shape=('rows', 'columns'), order=F +Lifecycle policy: copy_out writeback via copy_in_out +Completed record mutation rejected: True +``` + +Ownership resolution: + +```bash +python3 prik/policy/ownership.py +``` + +```text +before: math.scale(value): Float64 semantic IR +after: scalar/caller/call_local; scalar_value -> pass_value +``` + +Public export completion: + +```bash +python3 prik/policy/exports.py +``` + +```text +Native semantic owner: math.SCALE_VALUE +Python export: linear_algebra.scale_value +Completed policy type: PythonExportPolicy +``` + +Feature-policy construction: + +```bash +python3 prik/policy/construction.py +``` + +```text +before: math.scale(value): Float64 semantic IR +after: direct_transfer; result=native_scalar; native=pass_value +``` + +Full ordered completion: + +```bash +python3 prik/policy/completion.py +``` + +```text +before: math.scale(value): Float64 semantic IR +after: math.scale(value): scalar_value -> pass_value +``` + +Descriptor-backed array completion: + +```bash +python3 prik/policy/native_array_handles.py +``` + +```text +Handle policy: pointer/pointer, storage=alias +Allowed operations: to_numpy, nullify +Array ABI: descriptor +Selected build header: ISO_Fortran_binding.h +``` + +The outputs move from raw semantic facts to immutable decisions. They do not +generate source; that begins only after planning. + +## Tests And What They Prove + +- [Policy infrastructure](../../../tests/fortran/infrastructure/semantics/) covers policy records, completion order, and general semantic-policy rules. +- [Native handle policy](../../../tests/fortran/infrastructure/semantics/test_native_array_handles.py) covers descriptor policy, allowed operations, and ABI requirements. +- [Feature-local policy suites](../../../tests/fortran/) cover ownership and projection decisions for the supported wrapper features. +- [Planner tests](../../../tests/fortran/infrastructure/codegen/test_planner.py) prove that planning rejects incomplete policy instead of filling it in. +- [Direct execution inventory](../../../tests/fortran/infrastructure/execution_examples/test_execution_examples.py) fixes the six demonstrations above. + +## Change Routes + +- Add reusable immutable output vocabulary in `models.py` only when it is a + semantic decision that more than one lower stage must consume. +- Change one lifetime or barrier decision in `ownership.py`; retain a blocked + result when no safe supported combination exists. +- Change Python placement in `exports.py`. +- Change the coherent composition of a supported function, class, overload, + callback, result, or module-variable policy in `construction.py`. +- Change dependency order and attachment in `completion.py`. +- Change descriptor operations, ABI, or build requirements in + `native_array_handles.py`. +- Project an already completed fact in planning; lower an already selected + mechanism in codegen. Neither is a replacement policy owner. + +## Invariants And Common Mistakes + +- Completion order stays explicit; do not replace it with an opaque pass + registry. +- Raw semantic ownership metadata is a request, not an `OwnershipDecision`. +- Hidden output projection is separate from ABI transport. +- Ordinary NumPy buffer handoff and a persistent native descriptor handoff + are distinct ABI choices. +- A valid source declaration or `.pyi` annotation is not proof of safe + wrapper support. +- If a generator guesses a decision, move that decision into policy completion + and add the focused policy test before changing lowering. diff --git a/docs/developer/packages/preprocessing.md b/docs/developer/packages/preprocessing.md new file mode 100644 index 000000000..4eaa6c678 --- /dev/null +++ b/docs/developer/packages/preprocessing.md @@ -0,0 +1,139 @@ +--- +title: Preprocessing Package +audience: developers, maintainers, contributors +prerequisites: contributor architecture guide, native project compiler flags +related: ../architecture.md, index.md, parsers.md, semantics.md, ../concepts/datatype-lifecycle.md +status: maintained +publication: draft +--- + +# Preprocessing Package + +## Purpose And Boundaries + +`prik/preprocessing/` turns original Fortran source into authoritative parser +input and measures compiler-dependent target facts needed by semantic +conversion. Compiler expansion, native Fortran includes, and executable probes +are separate mechanisms with separate results. The package does not parse +declarations, assign semantic scalar identities, choose NumPy dtypes, or +complete wrapper policy. + +## Local Structure + +```text +prik/preprocessing/ +├── __init__.py +├── source.py +├── fortran.py +└── probes/ + ├── __init__.py + └── fortran_types.py +``` + +The C preprocessing and target-probe modules remain deferred from the +published Fortran contributor workflow. + +## What This Stage Receives And Produces + +```text +original Fortran path + PreprocessingConfig + -> compiler expansion and line-marker recovery + -> native Fortran INCLUDE expansion + -> PreprocessResult(source, provenance, dependencies, recipe, diagnostics) + -> Fortran parser + +compiler identity + target flags + kind/storage requirements + -> executable target probe + -> FortranTypeProbeReport + -> Fortran-to-IR conversion +``` + +Recipes retain compiler, adapter, argv, include directories, macro flags, +included files, source mappings, and diagnostics so a build can explain or +replay its parser input. Probe cache identity includes the compiler and target +configuration; measured facts must not cross targets silently. + +## Directory Tour + +| Module | Main entrypoints and contents | Change it when | +| --- | --- | --- | +| [`prik/preprocessing/__init__.py`](../../../prik/preprocessing/__init__.py) | Re-exports the supported source-preparation records, adapters, and entrypoints, including `expand_native_fortran_includes()`. | The supported preprocessing import API changes. | +| [`prik/preprocessing/source.py`](../../../prik/preprocessing/source.py) | `PreprocessingConfig`, `PreprocessingPlan`, `PreprocessingRecipe`, `PreprocessResult`, `SourceMapping`, and `IncludedFile`; builds compiler invocations, runs them, recovers mappings, and retains diagnostics/provenance. | Compiler-preprocessor adapters, recipes, line-marker handling, source provenance, or diagnostics change. | +| [`prik/preprocessing/fortran.py`](../../../prik/preprocessing/fortran.py) | `expand_native_fortran_includes()` expands native `INCLUDE` directives recursively while preserving locations and diagnostics. | Native Fortran include discovery or expansion changes. | +| [`prik/preprocessing/probes/__init__.py`](../../../prik/preprocessing/probes/__init__.py) | Namespace marker for compiler-derived target facts. | A probe-level public import surface is deliberately introduced. | +| [`prik/preprocessing/probes/fortran_types.py`](../../../prik/preprocessing/probes/fortran_types.py) | `FortranTypeProbeRecipe` and `FortranTypeProbeReport` compile and run small target programs for kind expressions, storage widths, logical representations, and compile-time values. | Measured fact generation, validation, cache identity, or probe execution changes. | + +## Execution Examples + +Coordinated preprocessing: + +```bash +python3 prik/preprocessing/source.py +``` + +```text +Before Fortran include expansion: +module greeting +include 'constants.inc' +... +After Fortran include expansion: +module greeting +integer, parameter :: answer = 42 +... +Native includes: 1; diagnostics: 0 +``` + +Native include expansion in isolation: + +```bash +python3 prik/preprocessing/fortran.py +``` + +```text +Expanded parser input: +module geometry +integer, parameter :: dimensions = 3 +end module geometry +Native include dependencies: 1 +Generated source mappings: 5 +Diagnostics: 0 +``` + +Compiler-measured Fortran type facts: + +```bash +python3 prik/preprocessing/probes/fortran_types.py +``` + +```text +selected_int_kind(9) = 4 +``` + +The first two outputs prove that prepared source retains dependency and source +mapping facts. The probe output is a native kind value, not yet a stable +semantic scalar or NumPy dtype. The probe example requires `gfortran` or +`f95`. + +## Tests And What They Prove + +- [Fortran preprocessing](../../../tests/fortran/source_preprocessing/preprocessing/) covers adapters, recipes, mappings, dependencies, and diagnostics. +- [Parser boundary tests](../../../tests/fortran/source_preprocessing/preprocessing/test_parser_boundaries.py) prove that prepared source reaches parsing with preserved facts. +- [Fortran target probes](../../../tests/fortran/data_types/probes/test_fortran_type_probes.py) cover measured type facts and cache separation. +- [Direct execution inventory](../../../tests/fortran/infrastructure/execution_examples/test_execution_examples.py) fixes the three demonstrations above. + +## Change Routes + +- Change compiler expansion, provenance, recipes, or diagnostics in + `source.py`. +- Change native `INCLUDE` behavior in `fortran.py`. +- Change target measurement or cache identity in `probes/fortran_types.py`. +- Change parser grammar downstream; change stable scalar identity or backend + mapping in the owning semantic/codegen package. + +## Invariants And Common Mistakes + +- Preserve original source coordinates through every source transformation. +- Run native probes in temporary working directories so `.mod` and other + compiler products cannot pollute the repository. +- Do not combine textual preprocessing and target measurement into one generic + operation simply because both run before parsing. diff --git a/docs/developer/packages/printers.md b/docs/developer/packages/printers.md new file mode 100644 index 000000000..de8ced5a1 --- /dev/null +++ b/docs/developer/packages/printers.md @@ -0,0 +1,120 @@ +--- +title: Printers Package +audience: developers, maintainers, contributors +prerequisites: contributor architecture guide, formed source representations +related: ../architecture.md, index.md, codegen.md, pipeline.md, parsers.md +status: maintained +publication: draft +--- + +# Printers Package + +## Purpose And Boundaries + +`prik/printers/` is the representation-to-text boundary. C and Fortran +printers serialize backend nodes; the semantic `.pyi` printer serializes +semantic IR. Printers own formatting, escaping, indentation, declaration +order, and safe line wrapping. They do not invoke generators, choose filenames, +complete policy, or compile output. + +## Local Structure + +```text +prik/printers/ +├── __init__.py +├── c.py +├── fortran.py +└── pyi.py +``` + +## What This Stage Receives And Produces + +```text +formed C or Fortran node tree -> matching source printer -> native text +SemanticModule graph -> PyiPrinter -> editable .pyi +``` + +## Directory Tour + +| Module | Main entrypoints and contents | Change it when | +| --- | --- | --- | +| [`prik/printers/__init__.py`](../../../prik/printers/__init__.py) | Re-exports `CSourcePrinter`, `FortranSourcePrinter`, `PyiPrinter`, and `emit_module()`. | The supported printer import surface changes. | +| [`prik/printers/c.py`](../../../prik/printers/c.py) | `CSourcePrinter` serializes C translation units, headers, declarations, functions, tables, and statements. | C syntax layout, escaping, or formatting changes. | +| [`prik/printers/fortran.py`](../../../prik/printers/fortran.py) | `FortranSourcePrinter` serializes bridge modules, interfaces, declarations, procedures, and free-form wrapped statements. | Fortran source layout or line-wrapping changes. | +| [`prik/printers/pyi.py`](../../../prik/printers/pyi.py) | `PyiPrinter`, `emit_module()`, and `_PyiEmissionContext` serialize semantic modules and scope imports, aliases, namespaces, and defaults for one emission. | Editable contract spelling or emission-context behavior changes. | + +The fact that code generation calls a printer at the end of wrapper rendering +does not make printing part of codegen ownership. `pipeline/wrapper.py` +coordinates both distinct stages. + +## Execution Examples + +```bash +python3 prik/printers/c.py +``` + +```text +Rendered C binding source: +#include + +static PyObject * wrap_ping(PyObject * self) { + Py_INCREF(Py_None); + return Py_None; +} +``` + +```bash +python3 prik/printers/fortran.py +``` + +```text +Rendered Fortran bridge source: +module bind_c_printer_demo_wrapper + use iso_c_binding, only: c_double + use printer_demo, only: native_double_value => DOUBLE_VALUE + implicit none +contains + function bind_c_double_value(value) result(result) bind(c, name="DOUBLE_VALUE") + real(c_double), value :: value + real(c_double) :: result + result = native_double_value(value) + end function bind_c_double_value +end module bind_c_printer_demo_wrapper +``` + +```bash +python3 prik/printers/pyi.py +``` + +```text +Semantic module: printer_demo +from prik.contracts import Float64, bind + +@bind("DOUBLE_VALUE") +def double_value( + value: Float64 +) -> Float64: ... +``` + +The native examples prove that punctuation and layout are added to already +formed nodes. The `.pyi` example proves that required contract imports and +native identity are derived without attaching wrapper policy. + +## Tests And What They Prove + +- [Printer infrastructure](../../../tests/fortran/infrastructure/printers/) covers native syntax serialization and formatting. +- [Semantic `.pyi` round trips](../../../tests/fortran/semantic_pyi_format/) cover contract emission and re-parsing. +- [Direct execution inventory](../../../tests/fortran/infrastructure/execution_examples/test_execution_examples.py) fixes the three rendered examples above. + +## Change Routes + +- Change formatting or serialization in the matching printer. +- If information is missing from a native node, add it in generation or the + plan rather than consulting semantic IR from the printer. +- Change filenames or multi-source artifact order in the pipeline. + +## Invariants And Common Mistakes + +- Native source printers accept backend nodes, not semantic models. +- The `.pyi` printer accepts semantic IR, not wrapper plans. +- Emission contexts are per-operation and restored safely after failures. diff --git a/docs/developer/packages/runtime.md b/docs/developer/packages/runtime.md new file mode 100644 index 000000000..2326758d1 --- /dev/null +++ b/docs/developer/packages/runtime.md @@ -0,0 +1,99 @@ +--- +title: Runtime Package +audience: developers, maintainers, contributors +prerequisites: contributor architecture guide, completed native handle policy +related: ../architecture.md, index.md, policy.md, compiler.md, pipeline.md +status: maintained +publication: draft +--- + +# Runtime Package + +## Purpose And Boundaries + +`prik/runtime/` owns Python objects that remain active after importing a +generated extension and the bundled native header payload used by generated +bindings. Runtime objects validate descriptor metadata, retain owners, adapt +generated operations, and expose policy-selected NumPy views. They enforce +completed behavior; they do not decide ownership or invent missing operations. + +## Local Structure + +```text +prik/runtime/ +├── __init__.py +├── handles.py +└── native_support/ + ├── __init__.py + ├── prik_binding.h + └── LICENSE +``` + +## What This Stage Receives And Produces + +```text +generated extension operation dictionary + -> descriptor metadata validation + -> AllocatableArray or PointerArray adapter + -> policy-permitted allocate/deallocate/resize/nullify/to_numpy operations +``` + +The native-support initializer only makes the payload locatable. The compiler +installs it into a generated `binding_support/` include directory. + +## Directory Tour + +| Path | Main entrypoints and contents | Change it when | +| --- | --- | --- | +| [`prik/runtime/__init__.py`](../../../prik/runtime/__init__.py) | Package boundary for Python runtime support. | A small supported runtime import surface is deliberately introduced. | +| [`prik/runtime/handles.py`](../../../prik/runtime/handles.py) | `NativeArrayHandleBase`, `AllocatableArray`, and `PointerArray` validate generated operations, retain owners, and produce policy-permitted live NumPy views. | Handle protocol, validation, retention, descriptor conversion, or Python operation behavior changes. | +| [`prik/runtime/native_support/__init__.py`](../../../prik/runtime/native_support/__init__.py) | Locates the bundled native-support payload without creating another Python runtime API. | Payload discovery changes. | +| `runtime/native_support/prik_binding.h` | Bundled native capsule, descriptor, validation, conversion, and release support compiled into generated bindings. | A generated binding requires changed native support; also inspect `compiler/native_support.py` installation. | +| `runtime/native_support/LICENSE` | License text distributed with the native payload. | The payload licensing changes. | + +## Execution Example + +```bash +python3 prik/runtime/handles.py +``` + +```text +Runtime handle: AllocatableArray +Descriptor kind: allocatable +Initial view: [1.0, 2.0, 3.0] +Resized shape: (4,) +Generated resize received NumPy extents: True +``` + +The example supplies the same operation dictionary shape exported by a +generated extension. It proves descriptor selection, validation, operation +adaptation, and the generated NumPy extent convention. The returned NumPy +storage is live, not a detached snapshot. + +The native payload intentionally has no standalone Python example: it is +compiled only as part of a generated binding. + +## Tests And What They Prove + +- [Allocatable runtime tests](../../../tests/fortran/allocatables/runtime/) cover allocatable operations and NumPy views. +- [Pointer runtime tests](../../../tests/fortran/pointers/runtime/) cover pointer association and views. +- [Memory-management runtime tests](../../../tests/fortran/memory_management/runtime/) cover release and ownership enforcement. +- [Runtime infrastructure](../../../tests/fortran/infrastructure/runtime/) covers generated-operation protocols. +- [Compiled runtime compatibility](../../../tests/fortran/building_shared_library/end_to_end/test_runtime_compatibility.py) covers the payload in a real extension. +- [Direct execution inventory](../../../tests/fortran/infrastructure/execution_examples/test_execution_examples.py) fixes the handle demonstration above. + +## Change Routes + +- Change handle protocol, validation, retention, or adapters in `handles.py`. +- Change header implementation in `native_support/` and installation in + `prik/compiler/native_support.py`. +- Complete any new ownership, operation permission, or view policy upstream + before runtime enforcement. + +## Invariants And Common Mistakes + +- Outstanding zero-copy NumPy views cannot be revoked after native + reallocation or pointer reassociation. Callers must discard or copy them. +- Runtime must reject operations absent from completed policy rather than + guessing permission from descriptor kind. +- The native support directory is a payload, not a second pipeline stage. diff --git a/docs/developer/packages/semantics.md b/docs/developer/packages/semantics.md new file mode 100644 index 000000000..496f9dbb1 --- /dev/null +++ b/docs/developer/packages/semantics.md @@ -0,0 +1,162 @@ +--- +title: Semantics Package +audience: developers, maintainers, contributors +prerequisites: contributor architecture guide, parser package guide +related: ../architecture.md, index.md, parsers.md, policy.md, ../concepts/datatype-lifecycle.md +status: maintained +publication: draft +--- + +# Semantics Package + +## Purpose And Boundaries + +`prik/semantics/` converts Fortran parser facts or semantic `.pyi` AST into the +same language-neutral `SemanticModule` graph. It owns stable types, public and +native identities, shapes, projections, provenance, storage contracts, and raw +metadata. It does not complete ownership, select lowering actions, plan +wrappers, or emit source. + +## Local Structure + +```text +prik/semantics/ +├── __init__.py +├── models.py +├── scalar_types.py +├── fortran2ir.py +├── pyi2ir.py +├── metadata.py +├── pyi_metadata.py +├── ownership_metadata.py +├── native_array_handles.py +└── native_contract.py +``` + +The deferred C-to-IR path is intentionally excluded from the published +Fortran contributor workflow. + +## What This Stage Receives And Produces + +```text +Fortran parser models + measured target facts ─┐ + ├─> SemanticModule graph +semantic .pyi AST ─────────────────────────────┘ + -> raw ownership/native contract metadata + -> prik.policy completion +``` + +## Directory Tour + +| Module | Main entrypoints and contents | Change it when | +| --- | --- | --- | +| [`prik/semantics/__init__.py`](../../../prik/semantics/__init__.py) | Re-exports supported Fortran conversion and `.pyi` conversion entrypoints. | The supported semantic-conversion import API changes. | +| [`prik/semantics/models.py`](../../../prik/semantics/models.py) | `SemanticModule`, `SemanticFunction`, `SemanticClass`, `SemanticArgument`, `SemanticType`, storage/array contracts, and `SemanticOrigin` form the shared language-neutral graph. | A downstream consumer needs a new language-neutral fact. | +| [`prik/semantics/scalar_types.py`](../../../prik/semantics/scalar_types.py) | `SemanticScalarSpec` and the scalar catalogue give stable identities and intrinsic family/storage facts without backend spelling. | Stable scalar vocabulary or intrinsic facts change. | +| [`prik/semantics/fortran2ir.py`](../../../prik/semantics/fortran2ir.py) | `FortranToIRConverter` combines parser models and measured facts into semantic IR; public helpers handle files, modules, and projects. | A Fortran source fact needs a different semantic interpretation. | +| [`prik/semantics/pyi2ir.py`](../../../prik/semantics/pyi2ir.py) | `convert_pyi_to_ir()` interprets parsed Python AST as an editable semantic contract and reconciles external type references. | A supported `.pyi` construct needs semantic meaning. | +| [`prik/semantics/metadata.py`](../../../prik/semantics/metadata.py) | Passive keys shared by semantic owners. | A generic semantic metadata key or its canonical spelling changes. | +| [`prik/semantics/pyi_metadata.py`](../../../prik/semantics/pyi_metadata.py) | Passive keys specific to `.pyi` interpretation. | Parsed `.pyi` metadata needs a canonical key. | +| [`prik/semantics/ownership_metadata.py`](../../../prik/semantics/ownership_metadata.py) | Normalizes raw ownership and pointer requests without resolving them. | A frontend request needs preservation before policy completion. | +| [`prik/semantics/native_array_handles.py`](../../../prik/semantics/native_array_handles.py) | `NativeArrayHandleFacts` keeps descriptor, data, and element facets separate. | Semantic description of a native descriptor-backed array changes. | +| [`prik/semantics/native_contract.py`](../../../prik/semantics/native_contract.py) | `NativeContractIssue` and helpers prepare and validate source-free native placement and ABI facts. | Native contract validation or diagnostics change. | + +Combined multi-file `.pyi` loading belongs to `prik/pipeline/pyi.py`. Completed +ownership, projection, and lowering actions belong to `policy/`, never here. + +## Execution Examples + +```bash +python3 prik/semantics/models.py +``` + +```text +Semantic module: geometry +Function: scale -> native SCALE +Argument: values: Float64, rank=1, shape=('n',), order=F +Source provenance: fortran real +``` + +```bash +python3 prik/semantics/scalar_types.py +``` + +```text +Float64: family=real, storage=64 bits +Int: family=signed_integer, storage=target-dependent +Backend spelling stored here: False +``` + +```bash +python3 prik/semantics/fortran2ir.py +``` + +```text +math.scale(value): Float64 via reference storage +``` + +```bash +python3 prik/semantics/pyi2ir.py +``` + +```text +math.scale(value): Float64 -> Float64 +``` + +```bash +python3 prik/semantics/ownership_metadata.py +``` + +```text +Raw ownership request: owner=caller, transfer=in_place, destruction=caller +Pointer contract: nullable=True, lifetime=owner, reassociation=forbidden +Completed lowering action present: False +``` + +```bash +python3 prik/semantics/native_array_handles.py +``` + +```text +Descriptor kind: allocatable +Data facet: Float64, rank=2, shape=('rows', 'columns') +Element facet: Float64, rank=0 +Handle marker retained by data facet: False +``` + +```bash +python3 prik/semantics/native_contract.py +``` + +```text +Prepared origin: fortran module math +Valid contract issues: 0 +Invalid contract issue: pyi_native_type_missing at math.broken.value +``` + +These examples show stable semantic representation and raw contract facts. +None contains a completed binding or bridge action. + +## Tests And What They Prove + +- [Semantic IR conversion](../../../tests/fortran/semantic_ir/semantics/) covers Fortran-model conversion and graph shape. +- [Semantic `.pyi` behavior](../../../tests/fortran/semantic_pyi_format/) covers contract interpretation and external references. +- [Datatype semantics](../../../tests/fortran/data_types/semantics/) covers stable type and storage facts. +- [Native handle semantics](../../../tests/fortran/infrastructure/semantics/test_native_array_handles.py) covers descriptor/data/element separation. +- [Direct execution inventory](../../../tests/fortran/infrastructure/execution_examples/test_execution_examples.py) fixes the seven stage demonstrations above. + +## Change Routes + +- Change graph shape in `models.py` only when downstream contracts need a new + language-neutral fact. +- Change stable primitive vocabulary in `scalar_types.py`. +- Change frontend interpretation in the matching converter. +- Change lifetime, transfer, setter, projection, or support decisions in + policy, never in semantic conversion. + +## Invariants And Common Mistakes + +- Parser source spellings and backend dtype spellings are not semantic type + identities. +- Raw ownership metadata is not completed ownership policy. +- Preserve provenance when normalizing language-specific facts. diff --git a/docs/developer/packages/utilities.md b/docs/developer/packages/utilities.md new file mode 100644 index 000000000..16375ece1 --- /dev/null +++ b/docs/developer/packages/utilities.md @@ -0,0 +1,116 @@ +--- +title: Utilities Package +audience: developers, maintainers, contributors +prerequisites: contributor architecture guide +related: ../architecture.md, index.md, semantics.md, planning.md, codegen.md +status: maintained +publication: draft +--- + +# Utilities Package + +## Purpose And Boundaries + +`prik/utilities/` contains small mechanisms that are genuinely independent of +one compiler stage. A helper belongs here only while it avoids stage-owned +semantic policy, syntax grammar, and workflow orchestration. + +## Local Structure + +```text +prik/utilities/ +├── __init__.py +├── declaration_expressions.py +├── stage_values.py +├── strings.py +└── visitor.py +``` + +## What This Stage Receives And Produces + +```text +stage-owned caller facts + -> reusable expression, local-name, or visitor mechanism + -> requesting stage +``` + +## Directory Tour + +| Module | Main entrypoints and contents | Change it when | +| --- | --- | --- | +| [`prik/utilities/__init__.py`](../../../prik/utilities/__init__.py) | Package boundary for small stage-neutral mechanisms. | Establishing a deliberate package-level utility API. | +| [`prik/utilities/declaration_expressions.py`](../../../prik/utilities/declaration_expressions.py) | `ResolvedDeclarationExtent`, `DeclarationExpressionCall`, and `ArrayExpressionSource` translate, validate, resolve, evaluate, and render declaration extents at explicit handoffs. | An extent representation or its stage-owned translation changes. | +| [`prik/utilities/stage_values.py`](../../../prik/utilities/stage_values.py) | `StageRecord` keeps an output editable until its consumer calls `freeze()`, which recursively converts nested lists, maps, and sets into immutable values. `FrozenStageRecordError` rejects later mutation. | A cross-stage record needs an immutable consumer boundary; do not use it to make semantic policy decisions. | +| [`prik/utilities/strings.py`](../../../prik/utilities/strings.py) | Collision-safe local-name helpers allocate deterministic temporary identifiers. | Generic local name allocation changes; public name policy belongs in `naming/`. | +| [`prik/utilities/visitor.py`](../../../prik/utilities/visitor.py) | `ClassVisitor` provides exact-class dispatch with intentional MRO fallback. | Shared generic dispatch changes, not a stage's visitor methods. | + +## Execution Examples + +```bash +python3 prik/utilities/declaration_expressions.py +``` + +```text +Fortran extent: ubound(source, 1) - lbound(source, 1) + 1 +Public expression: source.shape[0] +Role-bound expression: __prik_extent_source_0 +Fortran rendering: native_source_extent_0 +Compile-time product: 6 +``` + +The expression changes representation at explicit stages. Backend rendering +uses a plan-supplied substitution and does not rediscover argument ownership. + +```bash +python3 prik/utilities/stage_values.py +``` + +```text +Editable parser output: geometry -> ['scale', 'norm'] +Frozen consumer input: geometry -> ('scale', 'norm') +Mutation rejected: ParserOutput is frozen by its consuming stage +``` + +`StageRecord` is a utility rather than a pipeline stage: the caller owns the +moment it freezes a record. Wrapper generation freezes a completed plan, +printers freeze generated syntax nodes, and build integration freezes the +generated wrapper before writing files. + +```bash +python3 prik/utilities/strings.py +``` + +```text +First available name: temporary_4 +Next counter: 5 +``` + +```bash +python3 prik/utilities/visitor.py +``` + +```text +Exact handler: literal:42 +MRO fallback: expression:Expression +``` + +## Tests And What They Prove + +- [Utility infrastructure](../../../tests/fortran/infrastructure/utilities/) covers local-name and visitor behavior. +- [Pipeline freeze-boundary tests](../../../tests/fortran/infrastructure/pipeline/test_wrapper_generator.py) cover plan and generated-node mutation rejection after consumption. +- [Declaration-expression semantics](../../../tests/fortran/arrays/semantics/test_declaration_expression_utilities.py) covers role resolution and expression rendering. +- [Direct execution inventory](../../../tests/fortran/infrastructure/execution_examples/test_execution_examples.py) fixes the four demonstrations above. + +## Change Routes + +- Keep parsing, role resolution, evaluation, and backend rendering separate in + declaration-expression code. + +## Invariants And Common Mistakes + +- Consumers define their own visitor handlers; `ClassVisitor` does not merge + frontend or backend visitor responsibilities. +- Freeze only at the consumer boundary. Freezing a record while its producing + stage is still assembling it prevents legitimate local completion. +- Move a helper out of utilities as soon as it starts selecting semantic + policy or a pipeline action. diff --git a/docs/developer/quality-assurance.md b/docs/developer/quality-assurance.md deleted file mode 100644 index 67f907243..000000000 --- a/docs/developer/quality-assurance.md +++ /dev/null @@ -1,413 +0,0 @@ ---- -title: Quality Assurance -audience: developers, contributors -prerequisites: repository checkout, QA dependencies -related: testing-strategy.md, development-workflow.md -status: maintained -publication: draft ---- - -# Quality Assurance - -Last reviewed: 2026-07-31 - -This project uses a staged Python QA stack. Fast bug-focused checks, including -the bounded parser fuzz cases, run on pull requests. Maintainers can rerun the -fuzz-marked cases manually with the deeper Hypothesis profile. - -The selected active quality stack is adopted. Future Ruff/Radon threshold -ratchets are ongoing maintenance, not unfinished rollout work. Mutation -testing and pre-commit are not part of the active stack. - -## Active Cadence - -| Cadence | Tools | -| --- | --- | -| Local pre-push | Blocking static analysis, focused documentation smoke, one compiled scalar-wrapper smoke test, maintainer-tool tests in `tests/tools/`, and workflow-safety tests in `tests/workflows/` | -| Pull request and protected-branch push | pytest, bounded property/fuzz cases, stable-seed pytest-randomly, Ruff, Bandit, Vulture, staged Radon policy, and project coverage | -| Pull request, protected-branch push, weekly, and manual | Pinned Intel IFX/ICX and LLVM Flang/Clang profile checks plus strict Fortran toolchain smoke | -| Validated pull request and main-branch push | Pinned ARM64 prik/f2py correctness and rigorous performance benchmark | -| Manual discovery | Fuzz-marked parser tests with the deeper Hypothesis fuzz profile | -| Manual triage | Full Radon reports and low-severity Bandit review | -| Annual dependency review | Dependency vulnerability audit outside the routine per-change gate | - -Active GitHub Actions checks use stable, self-contained job names. Pull requests -are coordinated by `Pull Request` in five stages: - -1. Static analysis runs first. -2. Alternate-compiler smoke testing starts after static analysis succeeds. -3. The unit-test matrix starts after compiler smoke testing succeeds; its - Ubuntu Python 3.12 entry owns the project-coverage gate instead of repeating - that suite in a separate job. -4. BLAS/LAPACK validation starts only after the complete unit-test matrix, - including its coverage entry, succeeds. -5. The same pinned ARM64 documentation performance benchmark used on `main` - runs after native-library validation, and its generated snapshot is consumed - by the strict documentation build. - -An aggregate job runs with `always()` after every stage and fails unless all -required stage results succeeded. Configure the repository ruleset with this -single required status check: - -- `Pull Request / Validation · all required checks`. - -Treat that string as ruleset API. If its workflow or job display name changes, -replace the corresponding required-status-check entry; do not retain an alias -job for the previous name. The pull-request workflow declares its jobs directly -so check names contain only the `Pull Request` workflow name and the actual job -name; it does not add reusable-workflow caller stages between them. The -purpose-specific workflows retain the same complete job names for their -independent main, release, scheduled, and manual runs. - -## Install - -Install the package plus the QA toolchain: - -```bash -python -m pip install -e ".[qa]" -python tools/check_static_analysis_versions.py -``` - -If your shell only exposes `python3`, use: - -```bash -python3 -m pip install -e ".[qa]" -python3 tools/check_static_analysis_versions.py -``` - -## Local Commands - -Fast inner loop: - -```bash -pytest -q -python -m ruff check . -python -m ruff format . -``` - -CI-shaped local coverage run: - -```bash -HYPOTHESIS_PROFILE=ci \ -COVERAGE_PROCESS_START=pyproject.toml \ -PYTHONPATH=. \ -python -m coverage run -m pytest -q --randomly-seed=1 -python -m coverage combine -python -m coverage report -``` - -For subprocess coverage investigations, mirror that command shape before -deciding a fix. A plain local coverage run can miss subprocess data. -Every Python version excludes the full real-library wrapper examples while -retaining general native-bundle coverage. The `Real Libraries` component runs -the complete BLAS, LAPACK, FFTPACK, and MINPACK examples on Python 3.12. Each -job step sources the documented `build_all.sh` entrypoint before starting -pytest. BLAS and LAPACK additionally run their CI-only full-surface audits; -FFTPACK and MINPACK run their fail-closed public-inventory tests as part of the -maintained example suites. The job therefore verifies the copyable build and -test commands for all four libraries. A pull request may use the -`ignore-real-library-wrappers` label to skip that expensive component without -disabling the ordinary Python-version matrix. - -Every pull request and push to `main` runs the canonical Python 3.12 smoke and -ordinary-suite selections through `Quality Metrics`, then combines and -publishes their coverage data. The combined coverage.py report is the blocking -project gate and must remain at or above 90%. Codecov repeats that project -target for hosted reporting. Its -patch status is informational: changed-line coverage remains visible for -review, but a tiny defensive branch cannot independently fail an otherwise -passing project report. New reachable behavior should still receive focused -tests instead of relying on that reporting policy. - -Every matrix test run also writes a path-aware JUnit report. If pytest fails, the final -workflow step reads that report and prints a compact `Failed pytest nodes` -section containing every failed test node ID, including parametrization such -as `[source]` or `[generated-pyi]`. This summary is intentionally separate from -pytest's traceback output so failed names remain easy to find at the end of a -long GitHub Actions log. If pytest exits before producing a readable report, -the final step says that no report was available instead of hiding the failure. - -Reproduce an order-dependent failure from the stable CI seed: - -```bash -pytest -q --randomly-seed= -``` - -Run the same alternate-compiler lane used by GitHub Actions: - -```bash -python3 tools/run_fortran_toolchain_lane.py --compiler=/path/to/ifx -python3 tools/run_fortran_toolchain_lane.py --compiler=/path/to/flang -``` - -Use `--plan` to inspect the two pytest commands without executing them. Every -lane first runs the compiler-profile and focused preprocessing-CLI tests, then -runs the unchanged eight-node strict `toolchain_smoke` selection. GitHub -Actions pins IFX/ICX 2026.1.1 and Flang/Clang 22.1.8 on `ubuntu-24.04`; -compiler runtime directories are exported for extension loading. These are -tested CI pins, not inferred minimum supported versions. The Intel environment -installs both `ifx_linux-64` and `dpcpp_linux-64`: the former supplies IFX, -while the latter supplies the required ICX binding compiler. - -Run property and fuzz tests: - -```bash -pytest -q -m property --hypothesis-profile=ci -HYPOTHESIS_PROFILE=fuzz pytest -q -m fuzz --hypothesis-show-statistics -``` - -Run security checks: - -```bash -python -m bandit -c pyproject.toml -r prik --severity-level medium --confidence-level medium -``` - -Run dead-code and complexity checks: - - - - - -## Tool Decisions - -### pytest And coverage.py - -**Role:** behavioral regression backbone and branch-coverage floor. - -**Evidence:** recorded full-suite baseline is `3497 passed`; combined -subprocess branch coverage is `95.34%`, above the configured `95%` gate. - -**Decision:** keep as required baseline project gates. - -### pytest-randomly - -**Role:** catches hidden test-order coupling and makes failures reproducible -with seeds. - -**Evidence:** normal CI uses `--randomly-seed=1`, so order is shuffled but -reproducible. - -**Decision:** keep stable-seed PR CI. The changing-seed scheduled job was -removed as redundant maintenance overhead. - -### Hypothesis - -**Role:** generates edge cases for parsers, AST transforms, semantic IR, and -code generation. - - - -**Decision:** keep bounded property tests in normal test coverage and longer -fuzz profiles on schedule/manual dispatch. - -### Ruff - -**Role:** fast linting and formatting for undefined names, unused imports, -suspicious patterns, modernization, simplified control flow, and high McCabe -complexity. - -**Bugs or issues found:** raw regex issues, formatting drift, and static-risk -maintenance debt. These are static-risk findings, not runtime defects. - -**Decision:** keep as a blocking gate. Line-length diagnostics remain -intentionally unselected because wrapping parser diagnostics and embedded test -sources would add noise without improving correctness. - -### Bandit - -**Role:** security scanning for subprocess, filesystem, deserialization, and -credential-like patterns. - -**Evidence:** no medium- or high-severity findings. Reviewed low-severity -findings are parser sentinel/template tokens and intentional argv-based -compiler/preprocessor subprocess calls without shell execution. - -**Decision:** keep blocking at medium confidence/severity in CI. Re-review the -full low-severity report after subprocess-boundary changes. - -### Dependency Vulnerability Review - -**Role:** dependency vulnerability scanning. - -**Evidence:** routine per-change scans were noisy and slow relative to the -dependency churn in this project. - -**Decision:** do not run dependency vulnerability scanning as a pull-request or -local per-change gate. Revisit dependencies during an annual manual review or -when adding/upgrading runtime dependencies. - -### Vulture - -**Role:** dead-code detection. - -**Bugs or issues found:** removed dead Fortran parser parameters and unused test -lambda parameters reported by CI. - -**Decision:** keep blocking in CI with narrow exclusions. - -### Radon - -**Role:** complexity and maintainability tracking. - - - -**Bugs or issues found:** Radon found maintainability hotspots. CI also exposed -that the first staged policy was too strict for unchanged legacy hotspots; the -policy was corrected. - -**Decision:** keep `tools/check_radon_policy.py` blocking and keep full Radon -reports advisory/manual. - -### GitHub Actions - -**Role:** reproducible CI and scheduled discovery. - -**Bugs or issues found:** recent remote quality runs found Ruff raw-regex -issues, Ruff formatting drift, Vulture unused test parameters, and the -too-strict Radon policy. - -**Native artifact cache:** dedicated Python 3.12 BLAS and LAPACK jobs restore -the cache used by `examples.native_library`. On a miss, the example-owned -builder compiles each implementation corpus once; PRIK and f2py reuse the same -artifact. The ordinary pytest matrix excludes that full corpus while retaining -the lighter native-bundle tests. Requested coverage runs still -collect Python 3.12 coverage data; a final coverage job combines that artifact -and uploads the XML report. - -**Failure reporting:** each pytest matrix invocation writes -`pytest-results.xml`; the final failure-only step runs -`tools/print_pytest_failures.py` so all failed node IDs appear together at the -end of the job log. - -**Decision:** keep. Review scheduled results and record actionable failures -until fixed. - -## Historical Mutation Findings - -Mutation testing was useful during rollout, but it is no longer an adopted -tool. Do not keep `mutmut` as a regular dependency, workflow, or local wrapper. -A future annual mutation audit can be run outside the normal QA stack if -needed. - -Keep the ordinary regression tests and fixes that came from it: - -- Fortran project namespace collection respecting the requested encoding; -- direct Fortran parser contracts for diagnostics, forwarding, registries, - ownership, provenance, source locations, boundaries, and loop progress. - - - -## Test Organization - -- Unit tests: keep narrow behavior tests under the owning language, feature, - and pipeline-stage directory. -- Regression tests: add focused tests next to the subsystem that failed. Mark - with `@pytest.mark.regression` when useful. -- Property tests: keep generated invariants beside the domain they exercise. -- Fuzz-like parser tests: keep bounded generators beside the owning parser - tests, mark with `@pytest.mark.fuzz`, and run with the `fuzz` Hypothesis - profile. - -Good invariants for this codebase: - -- parsing the same source twice produces the same JSON/dict representation; -- generated declarations preserve name order and source locations; -- semantic conversion is deterministic for equivalent parser models; -- Pyi emission can be parsed back into equivalent semantic IR for supported - subsets; -- malformed input raises parser-owned diagnostic exceptions, not arbitrary - exceptions. - -## Adoption Status - -Full adoption for the selected stack means: - -- fast PR gates are blocking and stable; -- fuzz-marked parser robustness tests run in the ordinary matrix and remain - available with a deeper manual profile; -- Ruff baseline ignores are removed or deliberately retained with a reason; -- Radon has a documented blocking policy for new or materially changed code. - -Current status by area: - -| Area | Status | Explanation | -| --- | --- | --- | -| Fast pull-request gates | Complete for adoption | Tests, coverage, Ruff, Bandit, Vulture, and staged Radon are wired as blocking gates. | -| Property and fuzz testing | Complete for adoption | Current parser, AST, semantic-IR, and code-generation invariants exist; future failures still need regression tests. | -| Dead-code detection | Complete for adoption | Vulture is clean and blocking; future public API additions should keep exclusions narrow. | -| Security and dependency scanning | Complete for adoption | Bandit is blocking; dependency vulnerability review is annual/manual or tied to dependency changes. | -| Complexity tracking | Complete for adoption | The staged Radon policy is blocking in CI; future hotspot decomposition can ratchet thresholds further. | - -Ongoing maintenance: - -1. Save minimized examples from actionable fuzz failures as focused regression - tests. -2. Lower Ruff/Radon complexity thresholds after hotspot refactors make that - safe. - -## Manual Fuzz Triage - -Run deeper discovery explicitly with the documented `HYPOTHESIS_PROFILE=fuzz` -command: - -1. Re-run a failing example to separate actionable failures from transient - local-environment failures. -2. Reproduce actionable failures with the logged Hypothesis profile and - save minimized examples as focused regression tests. -3. Record each actionable failure here or in the relevant issue until - the regression test and fix pass. - -## Progress Log - -| Date | Area | Result | Follow-up | -| --- | --- | --- | --- | -| 2026-05-31 | Initial stack integration | Added configuration, CI, documentation, and Hypothesis tests. | Continue staged strictness rollout. | -| 2026-05-31 | Bandit | Reviewed low-severity findings and confirmed no medium- or high-severity findings. | Re-review when command trust boundaries change. | -| 2026-05-31 | Hypothesis code generation | Added generated native-name escaping, stable synthetic-import ordering, and semantic-IR-to-Pyi parse-back invariants; fixed quoted `SourceName(...)` emission. | Keep storing minimized failures. | -| 2026-06-01 | Ruff formatting rollout | Formatted the historical Python tree and changed CI to `ruff format --check .`. | Continue complexity-policy ratchets. | -| 2026-06-01 | Radon and Ruff complexity policy | Added `tools/check_radon_policy.py`, made the staged Radon policy blocking in CI, and lowered Ruff McCabe from `50` to `45`. | Continue hotspot refactors and later threshold ratchets toward `20`. | -| 2026-06-02 | Historical mutation-derived tests | Added direct Fortran parser contracts and fixed the directory namespace encoding bug. | Keep the tests as normal regression coverage. | -| 2026-06-03 | Manual Quality workflow review | Reviewed workflow run `26832679820`: fuzz passed, changing random-order pytest passed, static analysis exposed Ruff fixes, and full-project mutation exceeded the `3h` Actions limit. | Mutation was removed from active adoption; scheduled fuzz moved to its own workflow. | -| 2026-06-03 | Quality workflow triage | Reviewed latest Quality runs; run `26856679038` for `remove mutmut` completed successfully. | No actionable scheduled or PR quality failure remains. | -| 2026-07-31 | Workflow naming and fuzz consolidation | Split the mixed workflow into purpose-named static-analysis, tests, BLAS/LAPACK, and coverage workflows; removed the stale scheduled fuzz workflow, whose pre-migration `tests/property` target no longer existed. | Keep the two fuzz-marked parser tests in the ordinary matrix and use the deeper profile manually when needed. | - - - -## References - -- Ruff configuration: https://docs.astral.sh/ruff/configuration/ -- Pytest configuration: https://docs.pytest.org/en/latest/reference/customize.html -- Coverage subprocess behavior: https://coverage.readthedocs.io/en/latest/config.html -- Codecov commit-status configuration: https://docs.codecov.com/docs/commit-status -- Hypothesis settings profiles: https://hypothesis.readthedocs.io/en/latest/tutorial/settings.html -- Vulture configuration: https://pypi.org/project/vulture/ -- Radon command line: https://radon.readthedocs.io/en/stable/commandline.html -- Bandit configuration: https://bandit.readthedocs.io/en/latest/config.html -- pytest-randomly: https://github.com/pytest-dev/pytest-randomly diff --git a/docs/developer/repository-structure.md b/docs/developer/repository-structure.md deleted file mode 100644 index 1a067301a..000000000 --- a/docs/developer/repository-structure.md +++ /dev/null @@ -1,117 +0,0 @@ ---- -title: Repository Structure -audience: contributors -prerequisites: repository checkout -related: source-map.md, feature-to-code-map.md, build-system.md, testing-strategy.md -status: maintained -publication: draft ---- - -# Repository Structure - -The repository is a Python project with native fixtures and generated wrapper -artifacts used by tests. Navigate by ownership boundary first, then by file. - -## Source Tree - -| Path | Purpose | -| --- | --- | -| `prik/` | Python package implementation. Start with [source-map.md](source-map.md) for entrypoints and [feature-to-code-map.md](feature-to-code-map.md) when starting from behavior. | -| `prik/contracts/` | Public semantic `.pyi` contract vocabulary imported directly by generated and edited contracts. | -| `prik/pipeline/` | Shared preprocessing, semantic `.pyi` loading, and high-level wrapper build orchestration. | -| `prik/probes/` | Compiler-derived target facts and target type mapping reports. | -| `prik/runtime/` | Python runtime objects used by generated extension modules. | -| `prik/types/` | Cross-layer mappings from resolved semantic types to Python ecosystem types. | -| `prik/parsers/` | Public namespace for language and semantic-contract frontends and parser models. | -| `prik/semantics/` | Semantic IR, source-to-IR conversion, `.pyi` parsing, and policy completion. | -| `prik/codegen/` | Typed wrapper plans, direct native bridge/binding lowering, and source and semantic `.pyi` printers. | -| `prik/compiling/` | Native compile objects, compiler command orchestration, native support installation, and linking. | -| `prik/binding_support/` | Bundled header-only native support copied into generated wrapper builds. | -| `prik/naming/` | Unified public-name and generated-symbol policy. | -| `prik/utilities/` | Small shared Python utilities. | -| `examples/blas/` | Complete runnable Reference BLAS correctness project and the repository's single authoritative full BLAS source set under `native/`. | -| `examples/lapack/` | Complete Reference LAPACK build and SciPy-backed float64 correctness project, with the repository's single authoritative LAPACK implementation source set under `native/`. | -| `benchmarks/` | Local prik/f2py correctness and performance comparison harness. Benchmark sources and scripts are maintained; native builds and result files are generated locally. | -| `tools/generate_performance_docs.py` | Validates paired runtime and clean-build `pyperf` results and generates the bounded public Performance snapshot and both charts. | - -The major source packages have local README files under `prik/` for -developers reading directly in the source tree. Those README files should link -back to the maintained source-navigation docs instead of old top-level docs. - -Only `prik/__init__.py`, `prik/__main__.py`, and `prik/cli.py` live directly at -the package root. Public library symbols are deliberately flattened through -`prik/__init__.py`; internal modules are imported through their owning package. -The deliberate public submodule namespaces are `prik.contracts`, whose import -path is part of semantic `.pyi` syntax, and `prik.parsers`, which groups the -language-specific frontends. Stable convenience functions remain flattened -through `prik/__init__.py`. - -## Tests - -| Path | Purpose | -| --- | --- | -| `tests/fortran//` | User-visible Fortran and semantic `.pyi` behavior, with documented features directly below the language root and stages below each feature. | -| `tests/fortran/{source_parsing,source_preprocessing,command_line_interface,semantic_ir}/` | Public cross-feature capabilities that begin from source or expose an inspection/reporting surface. | -| `tests/fortran/infrastructure/` | Internal cross-feature policy, wrapper-generation, compiler, and runtime frameworks with no honest public-capability owner. | -| `tests/fortran/building_shared_library/end_to_end/real_libraries/` | Opt-in numerical showcase tests that build actual FFTPACK and MINPACK checkouts, call their generated Python routines, and verify known results. | -| `tests/c/` | C input-language parsing, preprocessing, probe, semantic, CLI, and fixture evidence. | -| `tests/docs/` | Documentation metadata, navigation, executable examples, publication, and source-map synchronization. | -| `tests/tools/` | Maintainer commands and CI support scripts. | -| `tests/workflows/` | Exceptional checks for concrete repository-automation safety risks. | -| `examples/blas/tests/test_*.py` | User-facing real-library correctness documentation: explicit independent and PRIK/f2py differential validation for every Reference BLAS routine. | -| `examples/blas/ci/full_surface.py` | Maintainer-only complete BLAS export and smoke audit, selected explicitly by CI. | -| `examples/lapack/tests/test_*.py` | User-facing real-library correctness documentation: explicit independent and PRIK/SciPy/f2py validation for the reviewed double-precision routine inventory. | -| `examples/lapack/ci/full_surface.py` | Maintainer-only complete LAPACK export and smoke audit, selected explicitly by CI. | - - - -## Documentation - -| Path | Purpose | -| --- | --- | -| `docs/index.md` | Documentation landing page. | -| `docs/user/` | Product workflows, examples, reference, support status, and troubleshooting. | -| `docs/developer/` | Contributor-facing workflows and source navigation. | -| `docs/old_docs/` | Archived pre-reorganization material. Do not link active docs here unless explicitly discussing history. | - -## Source Navigation Contract - -Source navigation is considered maintained when these files agree: - -- [source-map.md](source-map.md): package ownership, hotspot index, and common - change routes. -- [feature-to-code-map.md](feature-to-code-map.md): user-visible features to - docs, implementation files, tests, and support evidence. -- `prik/README.md` and package README files: local entry points for developers - already browsing the source tree. -- `tests/docs/test_reference_and_source_map.py`: mechanical coverage for - the navigation pages and README links. -- `tests/docs/test_publication.py`: fail-closed website publication, lane - gating, navigation filtering, and repository-evidence link coverage. - -## Generated And Fixture Areas - -- `__prik__/` directories are wrapper build artifacts and should not be - hand-edited as source. -- `benchmarks/build/f2py/` and `benchmarks/results/` contain generated - comparison artifacts and are not repository sources. CI retains paired - result files as workflow artifacts and generates the website snapshot from - them without committing the raw files. -- Parser and `.pyi` fixture files should be regenerated with the documented - fixture commands instead of edited loosely. -- `examples/blas/native/` is maintained source, not generated test output. The - full-library and LAPACK integrations consume it directly rather than owning - another BLAS copy. -- `examples/lapack/native/` is maintained Reference LAPACK implementation - source, not generated test output. Upstream testing, timing, example, and - matrix-generator programs are outside this ownership boundary. -- `prik.egg-info/`, caches, and benchmark output are generated local artifacts, - not source ownership boundaries. - - diff --git a/docs/maintainer/roadmap/documentation-content-checklist.md b/docs/developer/roadmap/documentation-content-checklist.md similarity index 66% rename from docs/maintainer/roadmap/documentation-content-checklist.md rename to docs/developer/roadmap/documentation-content-checklist.md index 4824e037d..00b3e79d8 100644 --- a/docs/maintainer/roadmap/documentation-content-checklist.md +++ b/docs/developer/roadmap/documentation-content-checklist.md @@ -2,7 +2,7 @@ title: Documentation Content Checklist audience: maintainers prerequisites: documentation architecture -related: ../documentation-architecture.md, index.md, semantic-pyi-wrapper-checklist.md +related: ../workflows/documentation.md, index.md, semantic-pyi-wrapper-checklist.md status: active-roadmap publication: draft --- @@ -40,7 +40,7 @@ these are true: - [ ] Documentation-only changes use focused docs checks and `git diff --check`; reserve the full static-analysis suite for code, tests, build/tooling changes, or explicit pre-merge verification. -- [ ] User, Developer, and Maintainer lane entry points, `mkdocs.yml`, related +- [ ] User and Contributor area entry points, `mkdocs.yml`, related front matter, and `tests/docs/test_navigation.py` stay synchronized. @@ -83,95 +83,38 @@ more specialized pages. unsupported features, and where to report bugs. PRIK_C_DOCS_END --> -### Developer And Contributor Guides +### Contributor Architecture And Package Guides -- [ ] `docs/developer/adding-a-feature.md`: document the feature workflow - from contract docs to implementation, tests, fixtures, support matrix, and - release notes. -- [ ] `docs/developer/adding-a-fortran-construct.md`: document parser, - semantic, policy, wrapper, docs, and fixture updates for a new Fortran - construct. -- [ ] `docs/developer/adding-a-code-generation-backend.md`: document - backend acceptance criteria, ownership boundaries, generated artifacts, tests, - and support claims. -- [ ] `docs/developer/testing-strategy.md`: document test layers, focused - verification paths, fixture regeneration, documentation examples, wrapper - runtime tests, and static-analysis gates. -- [ ] `docs/developer/build-system.md`: document native compile model, - generated Makefiles, build manifests, native support files, compiler probes, - and future packaging boundaries. -- [ ] `docs/developer/coding-standards.md`: document Python style, - documentation front matter, no-compatibility-layer rule, parser/codegen - organization, public contributor rules, TODO markers, support-claim - discipline, and review expectations. -- [ ] `docs/maintainer/ci-cd.md`: document current GitHub Actions gates, - coverage policy, static-analysis policy, docs checks, and local caveats for - CI-only environment values. -- [ ] `docs/maintainer/release-process.md`: document versioning, changelog, - release verification, wheel/source distribution limits, and documentation - publication steps. -- [ ] `docs/developer/contributing/contribution-guide.md`: document setup, issue scope, - expected docs updates, tests, static checks, and pull-request checklist. -- [ ] `docs/developer/contributing/pull-request-workflow.md`: document branch workflow, - commit message policy, required evidence, review response, and CI handling. -- [ ] `docs/developer/contributing/review-process.md`: document review focus, support - claims, docs completeness, fixture quality, and blocking versus advisory - comments. -### Design And Internal Architecture +- [x] `docs/developer/architecture.md`: shallow repository/package maps, + complete wrapper workflow, stage authority, root entrypoints, change routes, + and links to canonical package owners. +- [x] `docs/developer/packages/`: one maintained guide per top-level production + package with local structure, essential objects, executable examples, expected + output, focused tests, change routes, and invariants. +- [x] `docs/developer/concepts/datatype-lifecycle.md`: cross-stage datatype + authority from target probing through semantic identity, policy, backend + representation, and runtime validation. +- [x] `docs/developer/workflows/contributing.md`: documentation-first changes, + ownership lookup, support evidence, test selection, pull requests, review, + and contribution licensing. +- [x] `docs/developer/workflows/quality-assurance.md`: active blocking/advisory + tools, exact commands, coverage parity, compiler lanes, and local limits. +- [x] `docs/developer/workflows/ci.md`: staged GitHub validation, + documentation deployment, benchmark evidence, and stable ruleset context. +- [x] `docs/developer/workflows/release.md`: package identity, trusted + publishing, artifact review, publication, and clean-environment verification. +- [x] `docs/developer/workflows/documentation.md`: documentation placement, + metadata, publication, navigation, and continuous quality. +- [x] `docs/developer/design/multilanguage-runtime.md`: explicit long-term + architecture separated from current support claims. +- [x] `docs/developer/design/wrapper-open-decisions.md`: unresolved or + revisitable design questions separated from implemented package contracts. +- [x] `docs/developer/deferred/c-parser.md`: retained but unpublished C + parser/C-to-IR material, separate from the generated CPython C backend. -- [ ] `docs/maintainer/design/overall-architecture.md`: document system components, - pipeline stages, data contracts, supported language routes, and deferred - routes. -- [ ] `docs/maintainer/design/parser-architecture.md`: document parser ownership, - preprocessing boundaries, model facts, diagnostics, and fixture strategy. -- [ ] `docs/maintainer/design/semantic-analysis.md`: document source-to-IR lowering, - `.pyi`-to-IR loading, policy completion, wrapper-planning errors, and invariants. -- [ ] `docs/maintainer/design/runtime-model.md`: document native support files, generated - wrappers, native state, callbacks, threading, and finalization. -- [ ] `docs/maintainer/design/error-propagation-model.md`: document diagnostic categories, - Python exception projection, native failure handling, cleanup, and user-facing - message shape. -- [ ] `docs/maintainer/design/memory-ownership-model.md`: finish the design page around - policy-completion ownership decisions, transfer actions, mutability, setter - exposure, and release responsibility. -- [ ] `docs/maintainer/internal-architecture/ast-design.md`: document parser AST, semantic - IR, completed wrapper plans, generated source syntax, what each layer may - store, and what must not leak across - layers. -- [ ] `docs/maintainer/internal-architecture/semantic-passes.md`: document semantic pass - ordering, completed policy decisions, planner validation, and handoff to - `ir2ast`. -- [x] `docs/maintainer/internal-architecture/wrapper-generation-pipeline.md`: maintained - explanation of the current wrapper stages, semantic-policy boundary, - pass/planner/emitter distinctions, incremental decomposition criteria, and - acceptance criteria for bridge and binding refactoring. -- [ ] `docs/maintainer/internal-architecture/type-system.md`: document scalar kinds, arrays, - characters, derived types, pointers, allocatables, callbacks, and unsupported - storage forms. -- [ ] `docs/maintainer/internal-architecture/runtime-layer.md`: document native support - installation, extension initialization, callbacks, cleanup, and shared native - state. -- [ ] `docs/maintainer/internal-architecture/ownership-tracking.md`: document ownership - facts, transfer, borrowing, alias storage, destruction, writeback, and setter - exposure. -- [ ] `docs/maintainer/internal-architecture/dependency-analysis.md`: document current - source ordering, preprocessing dependency facts, generated build plans, and - future automatic dependency discovery. -- [ ] `docs/maintainer/internal-architecture/error-handling-pipeline.md`: document - diagnostic creation, path-aware `.pyi` loader errors, wrapper-planning failures, - generated validation failures, and native runtime errors. -- [ ] `docs/maintainer/internal-architecture/symbol-tables.md`: document public naming, - generated-symbol reservation, collision policy, imports, scopes, and package - names. - - +The old TODO-only contributor pages, duplicate pipeline/source maps, completed +wrapper-plan and native-array migration ledgers, and separate internal/design +indexes were removed after their stable facts moved to these owners. ### Tutorials And Examples @@ -213,12 +156,11 @@ PRIK_C_DOCS_END --> are planned, with expected prerequisites and runtime cost. - [ ] `docs/user/examples/index.md`: split verified cookbook recipes from planned larger examples and state the evidence required for each example. -- [ ] `docs/maintainer/design/index.md`: explain which design documents are accepted - architecture and which are placeholders. -- [ ] `docs/maintainer/internal-architecture/index.md`: route maintainers to pipeline, - semantic pass, runtime, type-system, ownership, and symbol-table pages. -- [ ] `docs/developer/contributing/index.md`: route contributors to contribution, - pull-request, review, and coding-standard pages. +- [x] `docs/developer/packages/index.md`: route contributors from each production + package to its canonical guide. +- [x] `docs/developer/index.md`: distinguish implemented package references, + cross-cutting concepts, workflows, design proposals, active roadmaps, and + deferred input-language material. - [ ] Public documentation site publication gate: deploy the existing MkDocs documentation as the project website only after all of the following are true; do not create a separate marketing-content system for this milestone. @@ -237,8 +179,8 @@ PRIK_C_DOCS_END --> lowering, bridge, and binding boundaries. - [ ] Each page has been reviewed explicitly; change `publication: draft` to `publication: reviewed` only after that review. - - [ ] Each lane index is reviewed last, after the lane pages intended for its - initial publication are ready. A draft lane index keeps the complete lane + - [ ] Each area index is reviewed last, after the pages intended for its + initial publication are ready. A draft area index keeps the complete area out of production. - [ ] A local draft preview and the Pages workflow artifact have validated navigation, links, search, rendering, and the static site build before @@ -251,13 +193,13 @@ evidence. Keep them current as behavior changes, but do not treat them as the primary placeholder queue. - [x] `docs/index.md`: maintained website entry point for all reviewed - documentation lanes. -- [x] `docs/user/index.md`: maintained User documentation lane entry point. -- [x] `docs/developer/index.md`: maintained Developer documentation lane entry - point. -- [x] `docs/maintainer/README.md`: maintained Maintainer documentation entry - point, publication-gated like the User and Developer indexes. -- [x] `docs/maintainer/documentation-architecture.md`: maintained three-lane + documentation areas. +- [x] `docs/user/index.md`: maintained User documentation entry point. +- [x] `docs/developer/index.md`: maintained Contributor documentation entry + point for developers and maintainers. +- [x] `docs/developer/architecture.md`: canonical contributor architecture + orientation and folder-by-folder rollout plan. +- [x] `docs/developer/workflows/documentation.md`: maintained two-area documentation and publication contract. - [x] `docs/user/getting-started/index.md`: maintained beginner route from installation through the normal rebuild workflow. @@ -335,25 +277,29 @@ primary placeholder queue. - [x] `docs/user/examples/recipes/`: maintained recipe lane for checked command and API examples. - [x] `docs/user/language-support/feature-matrix.md`: maintained support matrix. -- [x] `docs/developer/development-workflow.md`: maintained developer workflow. +- [x] `docs/developer/workflows/contributing.md`: maintained contributor + development and review workflow. - [x] `docs/developer/source-map.md`: maintained source route map. - [x] `docs/developer/feature-to-code-map.md`: maintained feature route map. -- [x] `docs/developer/repository-structure.md`: maintained repository tree - reference. -- [x] `docs/developer/fortran-parser-reference.md`: maintained Fortran +- [x] `docs/developer/architecture.md`: maintained shallow repository/package + structure and complete stage workflow. +- [x] `docs/developer/packages/parsers.md`: maintained Fortran parser reference. -- [x] `docs/developer/quality-assurance.md`: maintained quality and QA +- [x] `docs/developer/workflows/quality-assurance.md`: maintained quality and QA policy reference. -- [x] `docs/maintainer/internal-architecture/pipeline-map.md`: maintained pipeline and - concept-ownership map. -- [x] `docs/maintainer/roadmap/semantic-pyi-wrapper-checklist.md`: active implementation +- [x] `docs/developer/packages/index.md`: maintained package ownership map and + detailed package guide index. +- [x] `docs/developer/packages/policy.md`: maintained + ownership philosophy, completed policy vocabulary, supported lifetime triples, + pointer-policy boundary, validation order, source routes, and safety boundary. +- [x] `docs/developer/roadmap/semantic-pyi-wrapper-checklist.md`: active implementation roadmap for semantic `.pyi` wrapper parity. diff --git a/docs/maintainer/roadmap/fortran-test-suite-cleanup-checklist.md b/docs/developer/roadmap/fortran-test-suite-cleanup-checklist.md similarity index 99% rename from docs/maintainer/roadmap/fortran-test-suite-cleanup-checklist.md rename to docs/developer/roadmap/fortran-test-suite-cleanup-checklist.md index ae0e74525..5a3c1afc1 100644 --- a/docs/maintainer/roadmap/fortran-test-suite-cleanup-checklist.md +++ b/docs/developer/roadmap/fortran-test-suite-cleanup-checklist.md @@ -1,8 +1,8 @@ --- title: Language-First Test Suite and Fortran Pipeline Cleanup Checklist audience: maintainers -prerequisites: testing strategy, pipeline map, current test-suite organization record -related: ../../developer/testing-strategy.md, ../../../tests/README.md, ../internal-architecture/pipeline-map.md +prerequisites: testing strategy, contributor architecture guide, current test-suite organization record +related: ../testing-strategy.md, ../../../tests/README.md, ../architecture.md status: active-roadmap publication: draft --- @@ -109,6 +109,7 @@ tests/ infrastructure/ semantics/ codegen/ + docs/ tools/ ``` @@ -144,13 +146,17 @@ behavior and public cross-feature capabilities do not. - [x] `tests/fortran/` owns tests whose native input contract is Fortran, including generated Fortran bridge and C/CPython binding behavior for that Fortran contract. + - [x] Documentation and maintainer-tool tests have named top-level owners; internal language-neutral mechanics mirror their `prik/` package under `tests/fortran/infrastructure/`. + - [x] Fortran receives the documentation-led behavioral cleanup. - [x] Old imports, forwarding fixtures, collection shims, path aliases, and compatibility fallbacks are not retained. @@ -246,9 +252,11 @@ Register a structural marker for cross-feature selection: ``` - [x] A focused `tests/fortran/` command collects no C-input test. + - [x] Feature-local fixtures live with their owner. Minimized parser regressions live under `source_parsing/parsing/`; BLAS and LAPACK live only under `examples/blas/` and `examples/lapack/`. @@ -556,6 +564,7 @@ state, datatype matrix, or public error remains covered. ### Mechanical C quarantine + ## 5. Migrate Fortran feature by feature @@ -1254,7 +1264,10 @@ before changing compiler product behavior. - [x] BLAS, LAPACK, parser-regression, and contract content exists only beneath its final owner; no SciFortran snapshot remains. - [x] Every permanent contract row resolves to final collected nodes. -- [x] Collect `tests/fortran/`, `tests/c/`, `tests/docs/`, and `tests/tools/` independently; +- [x] Collect `tests/fortran/`, `tests/docs/`, and `tests/tools/` independently; + run the local Fortran verification with `-m "not real_library"`. - [x] Run the new suites alone under the same CI-equivalent line-and-branch coverage procedure used for the baseline. @@ -1585,7 +1598,9 @@ Add time separately when: - [ ] The authoritative tree is language-first and feature-first within Fortran. + - [ ] Every Fortran test and fixture has a final Fortran owner. - [ ] Every maintained User Guide and `.pyi` feature page maps to one obvious feature directory and focused command. diff --git a/docs/developer/roadmap/index.md b/docs/developer/roadmap/index.md new file mode 100644 index 000000000..73a78750b --- /dev/null +++ b/docs/developer/roadmap/index.md @@ -0,0 +1,25 @@ +--- +title: Active Roadmaps +audience: developers, maintainers, contributors +prerequisites: contributor architecture guide, current support matrix +related: ../../user/language-support/feature-matrix.md, semantic-pyi-wrapper-checklist.md, fortran-test-suite-cleanup-checklist.md, documentation-content-checklist.md +status: active-roadmap +publication: draft +--- + +# Active Roadmaps + +Only incomplete work belongs here. Implemented behavior is documented in user +and package guides; completed migration ledgers are removed after their stable +decisions and evidence routes have moved to canonical documentation. + +## Active Work + +- [Semantic `.pyi` wrapper completion](semantic-pyi-wrapper-checklist.md) +- [Language-first test suite and remaining compiler/CI work](fortran-test-suite-cleanup-checklist.md) +- [Remaining documentation content](documentation-content-checklist.md) + +Public support status remains authoritative in the +[feature matrix](../../user/language-support/feature-matrix.md). A checked +roadmap item is evidence of completed work, not a replacement for current +architecture, tests, or user documentation. diff --git a/docs/maintainer/roadmap/semantic-pyi-wrapper-checklist.md b/docs/developer/roadmap/semantic-pyi-wrapper-checklist.md similarity index 99% rename from docs/maintainer/roadmap/semantic-pyi-wrapper-checklist.md rename to docs/developer/roadmap/semantic-pyi-wrapper-checklist.md index b2ca7faf2..0f5550c18 100644 --- a/docs/maintainer/roadmap/semantic-pyi-wrapper-checklist.md +++ b/docs/developer/roadmap/semantic-pyi-wrapper-checklist.md @@ -415,12 +415,13 @@ PRIK_C_DOCS_END --> - [x] Ownership, transfer, and destruction policy is completed after full signatures are known and before wrapper planning. The shared post-IR entrypoint is `complete_semantic_policies(...)` in - `prik/semantics/policy_completion.py`; direct ownership subpasses stay behind + `prik/policy/completion.py`; direct ownership subpasses stay behind that entrypoint. Planning and lowering consume completed policy metadata instead of recomputing policy from raw datatypes. Evidence: - `tests/semantics/policy/`, - `tests/codegen/`, - `tests/semantics/policy/`, + `tests/fortran/infrastructure/semantics/test_policy_completion.py`, + `tests/fortran/infrastructure/semantics/test_ownership.py`, + feature-local `tests/fortran/*/policy/`, + `tests/fortran/infrastructure/codegen/`, and `prik/semantics/README.md`. - [x] `.pyi` parsing and `.pyi` semantic conversion are separate stages: `prik/parsers/pyi/parser.py` parses text/files to Python AST, and @@ -428,7 +429,7 @@ PRIK_C_DOCS_END --> before semantic policy completion runs. Evidence: `tests/fortran/semantic_pyi_format/parsing/test_python_ast_contracts.py::test_pyi_parser_returns_python_ast_only`, `prik/semantics/README.md`, and - `docs/maintainer/internal-architecture/pipeline-map.md`. + `docs/developer/architecture.md` and the detailed package guides. - [x] Risky-but-explicit identity contracts document their exact behavior instead of being silently healed. Fixed-length `String[n]` `intent(inout)` identity calls may return `None` with no observable Python mutation when the @@ -550,7 +551,7 @@ PRIK_C_DOCS_END --> implemented. Remaining rank, datatype, `is_alias`, and storage checks in bridge and binding code are local emitted-code, ABI, documentation, or object-model mechanics rather than semantic policy selection. Evidence: - `prik/semantics/ownership.py`, + `prik/policy/ownership.py`, `prik/codegen/fortran/bridge.py`, `prik/codegen/c/binding.py`, `tests/semantics/policy/`, diff --git a/docs/developer/source-map.md b/docs/developer/source-map.md index 76c864557..caa6991c4 100644 --- a/docs/developer/source-map.md +++ b/docs/developer/source-map.md @@ -20,15 +20,21 @@ current Python package layout. | `prik/cli.py` | User CLI, stage selection, output routing, diagnostics, wrapper-build option validation | parser frontends, semantic conversion, wrapper planning, `prik/pipeline/build.py` | | `prik/pipeline/build.py` | End-to-end Fortran source and semantic `.pyi` extension builds | preprocessing, parser, probes, completed semantic policy, wrapper planning and generation, compilation | | `prik/__init__.py` | Public Python exports | parser public-entrypoint tests and user examples | -| `prik/semantics/ownership.py` | Central ownership, transfer, destruction, and generated-action policy | policy completion and typed wrapper planning | -| `prik/probes/fortran_types.py` | Fortran kind/storage facts and cache | semantic Fortran conversion and wrapper builds | -| `prik/probes/report.py` | Generated target datatype mapping examples | documentation example tests | +| `prik/policy/ownership.py` | Central ownership, transfer, destruction, and generated-action policy | policy completion and typed wrapper planning | +| `prik/preprocessing/source.py` | Compiler-backed Fortran source expansion, provenance, and dependency facts | parser input preparation | + +| `prik/preprocessing/fortran.py` | Native Fortran `INCLUDE` expansion and source mappings | Fortran parser input preparation | +| `prik/preprocessing/probes/fortran_types.py` | Fortran kind/storage facts and cache | semantic Fortran conversion and wrapper builds | +| `prik/pipeline/type_mapping_report.py` | Cross-stage target datatype mapping examples | semantic and codegen datatype catalogues plus documentation example tests | +| `prik/semantics/scalar_types.py` | Stable scalar names, families, and intrinsic storage facts | source-to-IR conversion and policy completion | +| `prik/codegen/primitive_scalar_types.py` | Resolved semantic-to-NumPy projection and implemented backend scalar lowering facts | mapping reports plus binding and bridge generation | | `examples/blas/` | Complete Reference BLAS correctness example, source inventory, build fixtures, and the authoritative native source set | dedicated BLAS/LAPACK workflow and full-library integration | | `examples/lapack/` | Complete Reference LAPACK source set and SciPy-exposed float64 correctness inventory | dedicated BLAS/LAPACK workflow and full-library integration | ## Common Change Routes @@ -40,31 +46,32 @@ change crosses ownership boundaries. | Change area | Open first | Public docs to update | Focused evidence | | --- | --- | --- | --- | | CLI flags, stage selection, output formatting, diagnostics | `prik/cli.py` | `docs/user/reference/cli-commands.md`, `docs/user/getting-started/beginner-workflow.md` | `tests/fortran/command_line_interface/pipeline/`, `tests/docs/test_examples.py` | -| Compiler preprocessing, include paths, macros, and target flags | `prik/pipeline/preprocessing.py` | `docs/user/examples/recipes/compiler-preprocessing.md`, `docs/developer/compiler-preprocessing.md`, `docs/developer/fortran-parser-reference.md` | `tests/fortran/source_preprocessing/preprocessing/`, `tests/fortran/source_preprocessing/preprocessing/test_parser_boundaries.py` | -| Fortran parser facts and diagnostics | `prik/parsers/fortran/parser.py` | `docs/developer/fortran-parser-reference.md`, `docs/user/examples/recipes/inspect-fortran-api.md` | `tests/fortran/source_parsing/parsing/` | -| Semantic `.pyi` parsing, conversion, printing, package generation, and round-trip behavior | `prik/parsers/pyi/parser.py`, `prik/pipeline/pyi.py`, `prik/semantics/pyi2ir.py`, `prik/codegen/printers/pyi_printer.py` | `docs/user/reference/semantic-pyi-format.md`, `docs/user/reference/pyi-contracts/index.md`, `docs/user/examples/recipes/semantic-pyi-contracts.md` | `tests/fortran/semantic_pyi_format/`, `tests/fortran/semantic_pyi_format/pipeline/test_contract_package_generation.py`, `tests/fortran/semantic_pyi_format/pipeline/test_contract_loading.py`, `tests/fortran/semantic_pyi_format/end_to_end/test_authoritative_contract_runtime.py`, `tests/fortran/semantic_pyi_format/pipeline/` | -| Wrapper-planning errors and support claims | `prik/semantics/policy_completion.py`, `prik/codegen/planner.py` | `docs/user/reference/diagnostic-codes.md`, `docs/user/language-support/feature-matrix.md` | `tests/fortran/infrastructure/semantics/`, feature-local `policy/`, and `tests/fortran/infrastructure/codegen/` | +| Compiler preprocessing, include paths, macros, and target flags | `prik/preprocessing/source.py` | `docs/user/examples/recipes/compiler-preprocessing.md`, `docs/developer/packages/preprocessing.md`, `docs/developer/packages/parsers.md` | `tests/fortran/source_preprocessing/preprocessing/`, `tests/fortran/source_preprocessing/preprocessing/test_parser_boundaries.py` | +| Datatype probing, semantic normalization, NumPy projection, and mapping reports | `prik/preprocessing/probes/fortran_types.py`, `prik/semantics/scalar_types.py`, `prik/codegen/primitive_scalar_types.py`, `prik/pipeline/type_mapping_report.py` | `docs/developer/concepts/datatype-lifecycle.md`, `docs/user/reference/semantic-ir.md` | `tests/fortran/data_types/` | +| Fortran parser facts and diagnostics | `prik/parsers/fortran/parser.py` | `docs/developer/packages/parsers.md`, `docs/user/examples/recipes/inspect-fortran-api.md` | `tests/fortran/source_parsing/parsing/` | +| Semantic `.pyi` parsing, conversion, printing, package generation, and round-trip behavior | `prik/parsers/pyi/parser.py`, `prik/pipeline/pyi.py`, `prik/semantics/pyi2ir.py`, `prik/printers/pyi.py` | `docs/user/reference/semantic-pyi-format.md`, `docs/user/reference/pyi-contracts/index.md`, `docs/user/examples/recipes/semantic-pyi-contracts.md` | `tests/fortran/semantic_pyi_format/`, `tests/fortran/semantic_pyi_format/pipeline/test_contract_package_generation.py`, `tests/fortran/semantic_pyi_format/pipeline/test_contract_loading.py`, `tests/fortran/semantic_pyi_format/end_to_end/test_authoritative_contract_runtime.py`, `tests/fortran/semantic_pyi_format/pipeline/` | +| Wrapper-planning errors and support claims | `prik/policy/completion.py`, `prik/planning/planner.py` | `docs/user/reference/diagnostic-codes.md`, `docs/user/language-support/feature-matrix.md` | `tests/fortran/infrastructure/semantics/`, feature-local `policy/`, and `tests/fortran/infrastructure/codegen/` | | Source-driven Fortran wrapper orchestration | `prik/pipeline/build.py` | `docs/user/reference/fortran-wrapper.md`, `docs/user/guide/building-shared-library.md` | `tests/fortran/building_shared_library/end_to_end/test_source_build_modes.py`, `tests/fortran/building_shared_library/end_to_end/test_multi_source_builds.py` | | Semantic `.pyi` wrapper orchestration from native artifacts | `prik/pipeline/build.py`, `prik/pipeline/pyi.py`, `prik/semantics/pyi2ir.py` | `docs/user/reference/fortran-wrapper.md`, `docs/user/reference/semantic-pyi-format.md` | `tests/fortran/building_shared_library/pipeline/test_pyi_build_modes.py`, `tests/fortran/semantic_pyi_format/end_to_end/test_authoritative_contract_runtime.py`, `tests/fortran/pyi_contracts/exports_and_modules/`, `tests/fortran/pyi_contracts/functions_and_classes/` | -| Ownership, lifetime, output projection, and unsupported wrapper policy | `prik/semantics/policy_completion.py`, `prik/semantics/ownership.py`, `prik/codegen/planner.py` | `docs/user/guide/memory-management.md`, `docs/user/reference/semantic-pyi-format.md`, `docs/user/reference/fortran-wrapper.md` | `tests/fortran/infrastructure/semantics/`, feature-local `policy/`, and `tests/fortran/infrastructure/codegen/` | -| Immediate callback policy, typed adapters, and trampolines | `prik/semantics/wrapper_policy.py`, `prik/semantics/policy_completion.py`, `prik/codegen/plan.py`, `prik/codegen/planner.py`, `prik/codegen/c/binding.py`, `prik/codegen/fortran/bridge.py` | `docs/user/guide/callbacks.md`, `docs/user/reference/semantic-pyi-format.md` | `tests/fortran/callbacks/` | -| Native compilation, binding support, and shared-library linking | `prik/pipeline/build.py`, `prik/compiling/compilers.py`, `prik/compiling/native_support.py` | `docs/user/reference/fortran-wrapper.md`, `docs/developer/build-system.md` | `tests/fortran/building_shared_library/end_to_end/test_runtime_compatibility.py`, `tests/fortran/building_shared_library/end_to_end/test_source_build_modes.py` | +| Ownership, lifetime, output projection, and unsupported wrapper policy | `prik/policy/completion.py`, `prik/policy/ownership.py`, `prik/policy/models.py`, `prik/policy/construction.py`, `prik/planning/planner.py` | `docs/user/guide/memory-management.md`, `docs/user/reference/semantic-pyi-format.md`, `docs/user/reference/fortran-wrapper.md` | `tests/fortran/infrastructure/semantics/`, feature-local `policy/`, and `tests/fortran/infrastructure/codegen/` | +| Immediate callback policy, typed adapters, and trampolines | `prik/policy/models.py`, `prik/policy/construction.py`, `prik/policy/completion.py`, `prik/planning/models.py`, `prik/planning/planner.py`, `prik/codegen/c/binding.py`, `prik/codegen/fortran/bridge.py` | `docs/user/guide/callbacks.md`, `docs/user/reference/semantic-pyi-format.md` | `tests/fortran/callbacks/` | +| Native compilation, binding support, and shared-library linking | `prik/pipeline/build.py`, `prik/compiler/compilers.py`, `prik/compiler/native_support.py` | `docs/user/reference/fortran-wrapper.md`, `docs/developer/packages/compiler.md`, `docs/developer/packages/pipeline.md` | `tests/fortran/building_shared_library/end_to_end/test_runtime_compatibility.py`, `tests/fortran/building_shared_library/end_to_end/test_source_build_modes.py` | | Public Python exports | `prik/__init__.py` | `README.md`, `docs/user/reference/python-api.md` | `tests/fortran/source_parsing/parsing/test_public_entrypoints.py` | | Reference BLAS source ownership, inventory, and numerical validation | `examples/blas/routine_inventory.py`, `examples/blas/tests/test_routine_coverage.py` | `examples/blas/README.md`, `docs/user/examples/blas-wrapper.md` | `examples/blas/tests/test_*.py`, `examples/blas/ci/full_surface.py`, dedicated real-libraries workflow | | Reference LAPACK source ownership, inventory, and numerical validation | `examples/lapack/routine_inventory.py`, `examples/lapack/tests/test_routine_coverage.py` | `examples/lapack/README.md`, `docs/user/examples/lapack-wrapper.md` | `examples/lapack/tests/test_*.py`, `examples/lapack/ci/full_surface.py`, dedicated real-libraries workflow | | FFTPACK public-module boundary, source ownership, and numerical validation | `examples/fftpack/routine_inventory.py`, `examples/fftpack/tests/test_routine_coverage.py` | `examples/fftpack/README.md`, `docs/user/examples/fftpack-wrapper.md` | `examples/fftpack/tests/test_*.py`, `tests/fortran/building_shared_library/end_to_end/real_libraries/test_fftpack_routines.py`, dedicated real-libraries workflow | | MINPACK source ownership, parameter constants, and numerical validation | `examples/minpack/routine_inventory.py`, `examples/minpack/tests/test_routine_coverage.py` | `examples/minpack/README.md`, `docs/user/examples/minpack-wrapper.md` | `examples/minpack/tests/test_*.py`, `tests/fortran/building_shared_library/end_to_end/real_libraries/test_minpack_routines.py`, dedicated real-libraries workflow | | Source navigation documentation | `docs/developer/source-map.md`, `docs/developer/feature-to-code-map.md`, package README files | `docs/developer/source-map.md` | `tests/docs/test_reference_and_source_map.py` | +| Generated Fortran bridge | `prik/codegen/fortran/bridge.py`, `prik/printers/fortran.py`, `prik/pipeline/wrapper.py` | `docs/user/reference/fortran-wrapper.md` | `tests/fortran/infrastructure/codegen/`, `tests/fortran/` generated-wrapper assertions | +| Generated CPython binding and Python-visible runtime behavior | `prik/codegen/c/binding.py`, `prik/codegen/c/python_surface.py`, `prik/codegen/c/naming.py`, `prik/printers/c.py`, `prik/pipeline/wrapper.py` | `docs/user/reference/fortran-wrapper.md`, `docs/user/reference/python-api.md` | `tests/fortran/infrastructure/codegen/`, `tests/fortran/` | ## Package Map @@ -72,23 +79,24 @@ PRIK_C_DOCS_END --> | Package | Purpose | Main files | Primary tests and docs | | --- | --- | --- | --- | | `prik/contracts/` | Public semantic `.pyi` contract vocabulary | `__init__.py` | `tests/fortran/semantic_pyi_format/`, semantic `.pyi` reference | -| `prik/pipeline/` | Source preprocessing, semantic `.pyi` loading, and wrapper build orchestration | `preprocessing.py`, `pyi.py`, `build.py` | preprocessing, `.pyi`, and wrapper build tests | -| `prik/probes/` | Compiler-derived target facts plus mapping reports | `fortran_types.py`, `report.py` | target probe and type mapping report tests | -| `prik/runtime/` | Python runtime objects consumed by generated extensions | `handles.py` | runtime handle and wrapper runtime tests | -| `prik/types/` | Semantic-to-Python ecosystem type mappings | `numpy.py` | `tests/fortran/infrastructure/types/test_numpy.py` | -| `prik/parsers/` | Public namespace for language and semantic `.pyi` frontends | child parser packages | `tests/fortran/source_parsing/parsing/`, `tests/c/parsing/`, `tests/fortran/semantic_pyi_format/parsing/` | -| `prik/parsers/fortran/` | Fortran lexer, recursive parser, models, type resolver, and parser CLI helpers | `parser.py`, `lexer.py`, `models.py`, `type_resolver.py`, `cli.py` | `tests/fortran/source_parsing/parsing/`, `docs/developer/fortran-parser-reference.md` | -| `prik/compiling/` | Native compile objects, compiler command execution, shared-library linking, and native support installation; wrapper build orchestration lives in `prik/pipeline/build.py` | `objects.py`, `compilers.py`, `compiler_profiles.py`, `native_support.py` | `tests/fortran/building_shared_library/end_to_end/test_source_build_modes.py`, `tests/fortran/building_shared_library/end_to_end/test_runtime_compatibility.py` | -| `prik/binding_support/` | Bundled header-only native binding support copied into generated wrapper builds | support header | wrapper build tests | +| `prik/compiler/` | Compiler execution, compile objects, vendor profiles, native support installation, and linking | `compilers.py`, `objects.py`, `compiler_profiles.py`, `native_support.py` | compiler and shared-library build tests | +| `prik/preprocessing/` | Compiler-backed Fortran source expansion, native includes, provenance, and target probes | `source.py`, `fortran.py`, `probes/fortran_types.py` | Fortran preprocessing and target-probe tests | +| `prik/pipeline/` | Semantic `.pyi` loading, cross-stage datatype reporting, plan-to-source wrapper generation, and native build orchestration | `pyi.py`, `type_mapping_report.py`, `wrapper.py`, `build.py` | `.pyi`, datatype report, wrapper generation, and build tests | +| `prik/runtime/` | Python runtime objects and bundled native support consumed by generated extensions | `handles.py`, `native_support/` | runtime handle, native-support, and wrapper runtime tests | +| `prik/parsers/` | Public namespace for the Fortran and semantic `.pyi` frontends | child parser packages | `tests/fortran/source_parsing/parsing/`, `tests/fortran/semantic_pyi_format/parsing/` | +| `prik/parsers/fortran/` | Fortran lexer, recursive parser, models, type resolver, and parser CLI helpers | `parser.py`, `lexer.py`, `models.py`, `type_resolver.py`, `cli.py` | `tests/fortran/source_parsing/parsing/`, `docs/developer/packages/parsers.md` | +| `prik/parsers/pyi/` | Semantic `.pyi` text/file parsing to Python AST | `parser.py` | `tests/fortran/semantic_pyi_format/parsing/`, `docs/user/reference/semantic-pyi-format.md` | +| `prik/semantics/` | Language-neutral semantic IR, scalar datatype vocabulary, Fortran-to-IR conversion, `.pyi` conversion, and raw ownership or descriptor metadata | `models.py`, `scalar_types.py`, `fortran2ir.py`, `pyi2ir.py`, `ownership_metadata.py`, `native_array_handles.py` | `tests/fortran/data_types/semantics/`, `tests/fortran/semantic_ir/semantics/`, `tests/fortran/semantic_pyi_format/semantics/` | +| `prik/policy/` | Post-IR ownership, export, wrapper-policy construction, immutable policy models, descriptor-handle policy, and ordered completion | `ownership.py`, `exports.py`, `models.py`, `native_array_handles.py`, `construction.py`, `completion.py` | infrastructure semantics and feature-local policy tests | +| `prik/planning/` | Editable backend-neutral wrapper-plan records and mechanical policy projection | `models.py`, `planner.py` | infrastructure and feature-local codegen tests | +| `prik/codegen/` | Backend datatype projection, plan-driven docstrings, and direct lowering into C and Fortran syntax nodes | `primitive_scalar_types.py`, `docstrings.py`, `nodes.py`, `c/`, `fortran/` | data-type and infrastructure codegen, feature-local codegen, and end-to-end tests | +| `prik/printers/` | Language-specific serialization of C nodes, Fortran nodes, and semantic IR | `c.py`, `fortran.py`, `pyi.py` | source-printer and semantic-contract printer tests | +| `prik/naming/` | Unified public-name and generated-symbol policy for Python, generated C, and generated Fortran targets | `policy.py`, `native_symbols.py` | naming, visibility, and wrapper runtime tests | | `prik/utilities/` | Small shared Python utilities | `strings.py`, `visitor.py` | `tests/fortran/infrastructure/utilities/` and tests that exercise callers | ## Hotspot Index @@ -102,9 +110,14 @@ update this table, the package README files, and the mechanical checks in | `prik/__init__.py` | Public Python API exports. | | `prik/cli.py` | CLI argument validation, stage selection, output routing, and wrapper-build entry. | | `prik/pipeline/build.py` | End-to-end source and `.pyi` wrapper build orchestration. | -| `prik/pipeline/preprocessing.py` | Compiler-backed source preprocessing and dependency facts. | -| `prik/probes/fortran_types.py` | Fortran kind and storage probing. | -| `prik/semantics/ownership.py` | Central ownership, transfer, destruction, and generated-action policy. | +| `prik/preprocessing/source.py` | Compiler-backed source preprocessing and dependency facts. | +| `prik/pipeline/type_mapping_report.py` | Target facts, semantic conversion, and backend NumPy projection rendered as a mapping report. | +| `prik/preprocessing/probes/fortran_types.py` | Fortran kind and storage probing. | +| `prik/semantics/scalar_types.py` | Stable primitive scalar identities, families, and intrinsic storage widths. | +| `prik/semantics/ownership_metadata.py` | Raw ownership and pointer-contract metadata keys and normalized semantic setters. | +| `prik/semantics/native_array_handles.py` | Raw semantic descriptor-handle facts attached before policy completion. | +| `prik/policy/ownership.py` | Central ownership, transfer, destruction, and generated-action policy. | +| `prik/policy/exports.py` | Completed Python namespace and export-name policy. | | `prik/parsers/fortran/parser.py` | Fortran parser project model and diagnostics. | | `prik/parsers/fortran/cli.py` | Fortran parser report formatting. | | `prik/semantics/metadata.py` | Cross-stage semantic metadata keys that survive parser, policy, printer, and lowering boundaries. | @@ -113,22 +126,31 @@ update this table, the package README files, and the mechanical checks in | `prik/parsers/pyi/parser.py` | Minimal `.pyi` text/file parsing to Python AST. | | `prik/pipeline/pyi.py` | Semantic `.pyi` text/file/path-set conversion and external-type reconciliation. | | `prik/semantics/pyi2ir.py` | Semantic `.pyi` AST conversion and validation. | -| `prik/semantics/policy_completion.py` | Post-IR semantic policy completion before wrapper planning. | -| `prik/codegen/plan.py` | Typed, policy-complete wrapper plan records. | -| `prik/codegen/planner.py` | Semantic policy to wrapper-plan conversion. | -| `prik/codegen/generator.py` | Ordered direct bridge, binding, header, and source generation. | +| `prik/policy/models.py` | Immutable backend-neutral completed wrapper-policy vocabulary. | +| `prik/policy/construction.py` | Wrapper-policy construction rules and completed-policy accessors. | +| `prik/policy/completion.py` | Ordered post-IR semantic policy completion before wrapper planning. | +| `prik/policy/native_array_handles.py` | Completed descriptor-handle policy and build requirements. | +| `prik/planning/models.py` | Typed, policy-complete wrapper plan records. | +| `prik/planning/planner.py` | Semantic policy to wrapper-plan conversion. | +| `prik/naming/native_symbols.py` | Stable generated native-symbol construction shared by planning and code generation. | +| `prik/codegen/docstrings.py` | Plan-driven Python-facing documentation generation. | +| `prik/codegen/primitive_scalar_types.py` | Primitive scalar backend and NumPy lowering catalogue. | +| `prik/pipeline/wrapper.py` | Single plan-to-rendered-wrapper orchestration and generated-wrapper result records. | | `prik/codegen/fortran/bridge.py` | Direct Fortran bridge lowering from typed plans. | | `prik/codegen/c/binding.py` | Direct Python-extension binding lowering from typed plans. | -| `prik/codegen/printers/source_printers.py` | Native binding, header, and Fortran source printing. | -| `prik/codegen/printers/pyi_printer.py` | Semantic `.pyi` printing. | -| `prik/compiling/objects.py` | Native compile object model. | -| `prik/compiling/compilers.py` | Compiler command execution and tool lookup. | -| `prik/compiling/native_support.py` | Native binding support installation for generated wrappers. | +| `prik/codegen/c/python_surface.py` | Executable derived-class facade and thin class-overload forwarding source. | +| `prik/codegen/c/naming.py` | Shared symbols referenced by generated C and the embedded Python facade. | +| `prik/printers/c.py` | C binding and header node serialization. | +| `prik/printers/fortran.py` | Fortran bridge node serialization. | +| `prik/printers/pyi.py` | Semantic IR serialization as editable `.pyi`. | +| `prik/compiler/objects.py` | Native compile object model. | +| `prik/compiler/compilers.py` | Compiler command execution and tool lookup. | +| `prik/compiler/native_support.py` | Native binding support installation for generated wrappers. | | `prik/naming/policy.py` | Public wrapper names and generated target-language symbols. | -| `prik/binding_support/` | Native binding support payload copied into generated builds. | +| `prik/runtime/native_support/` | Header-only native runtime payload copied into generated builds as `binding_support/`. | For source-driven Fortran wrappers, read in this order: - For semantic `.pyi` builds, the parser branch is replaced by: @@ -162,9 +182,9 @@ For semantic `.pyi` builds, the parser branch is replaced by: prik/parsers/pyi/parser.py -> prik/pipeline/pyi.py -> prik/semantics/pyi2ir.py - -> prik/semantics/policy_completion.py - -> prik/codegen/planner.py - -> prik/codegen/generator.py + -> prik/policy/completion.py + -> prik/planning/planner.py + -> prik/pipeline/wrapper.py ``` ```text prik/cli.py -> prik/parsers/c/parser.py - -> prik/probes/c_types.py + -> prik/preprocessing/probes/c_types.py -> prik/semantics/c2ir.py - -> prik/codegen/printers/pyi_printer.py - -> prik/semantics/policy_completion.py + -> prik/printers/pyi.py + -> prik/policy/completion.py ``` PRIK_C_DOCS_END --> @@ -196,7 +216,12 @@ The hardest source packages also have local README files: - `prik/parsers/fortran/README.md` - `prik/parsers/pyi/README.md` - `prik/semantics/README.md` -- `prik/compiling/README.md` +- `prik/policy/README.md` +- `prik/planning/README.md` +- `prik/printers/README.md` +- `prik/pipeline/README.md` +- `prik/preprocessing/README.md` +- `prik/compiler/README.md` Within Fortran, user-visible behavior is feature first and pipeline stage second: @@ -125,7 +127,10 @@ Keep fixtures beside their final behavioral owner: integration. Generate build products and temporary contracts in temporary directories. -Check in generated `.pyi` only where exact generation text, +Compiler capability probes must also run with a temporary working directory so +side products such as Fortran `.mod` files cannot escape into the repository +root merely because the primary object or executable has an explicit output +path. Check in generated `.pyi` only where exact generation text, imports, placement, or package shape is the invariant. For BLAS behavior, source `examples/blas/build_all.sh`, then run @@ -188,7 +193,7 @@ For example, semantic-policy internals use `infrastructure/semantics/test_ownership.py` and `test_policy_completion.py`; wrapper internals use `infrastructure/codegen/test_plan.py`, `test_planner.py`, and -`test_generator.py`. Other internal owners mirror `prik/compiling/`, +`tests/fortran/infrastructure/pipeline/test_wrapper_generator.py`. Other internal owners mirror `prik/compiler/`, `prik/contracts/`, `prik/pipeline/`, `prik/runtime/`, and the remaining source packages when they have real internal tests. Do not create empty mirror directories or combine multiple production owners in generic backend or policy @@ -231,3 +236,20 @@ Both use `COVERAGE_PROCESS_START=pyproject.toml`, combine subprocess data with `python3 -m coverage combine`, and retain per-file executed line and branch data. LAPACK remains CI-only unless a maintainer explicitly requests a local run. + +## Fixture Regeneration + +Regenerate broad fixture sets only after a focused test explains the intended +change. Update the narrowest affected owner: + +```bash +python3 tests/fortran/source_parsing/parsing/generate_parser_goldens.py \ + tests/fortran/source_parsing/parsing/fixtures/general/basic_subroutine.f90 +python3 tests/fortran/semantic_ir/semantics/generate_semantic_fixtures.py +WRAPPER_UPDATE_PYI_FIXTURES=1 python3 -m pytest -q \ + tests/fortran/semantic_pyi_format/pipeline/test_contract_package_generation.py +``` + +Include regenerated artifacts only when the parser, semantic IR, or public +contract representation intentionally changed. Never regenerate a broad set to +hide uncertainty or unrelated drift. diff --git a/docs/maintainer/ci-cd.md b/docs/developer/workflows/ci.md similarity index 81% rename from docs/maintainer/ci-cd.md rename to docs/developer/workflows/ci.md index cd1910d62..2f265247a 100644 --- a/docs/maintainer/ci-cd.md +++ b/docs/developer/workflows/ci.md @@ -1,18 +1,18 @@ --- -title: CI/CD -audience: maintainers +title: Continuous Integration And Delivery +audience: developers, maintainers, contributors prerequisites: testing strategy -related: ../developer/testing-strategy.md, release-process.md -status: planned-documentation +related: ../testing-strategy.md, quality-assurance.md, release.md +status: maintained publication: draft --- -# CI/CD +# Continuous Integration And Delivery GitHub Actions owns repository quality checks and the reviewed-documentation deployment. The documentation workflow builds the same filtered MkDocs site -that maintainers can preview locally, uploads the generated `site/` directory, -and deploys it through GitHub Pages. +that maintainers can preview locally, uploads the generated +`.artifacts/site/` directory, and deploys it through GitHub Pages. Workflow names identify a unique pipeline scope, and every job display name is self-contained. The `Pull Request` workflow declares the staged validation jobs @@ -20,20 +20,21 @@ directly, avoiding the extra caller and called-workflow name layers produced by nested reusable workflows. Pull requests expose one aggregate required check after those jobs complete. Required status checks must use the exact `workflow / job` context documented in the -[quality-assurance guide](../developer/quality-assurance.md). Because GitHub +[quality-assurance guide](quality-assurance.md). Because GitHub treats a renamed check as a different context, update the repository ruleset whenever either half of that name changes. -Packaging output under `build/` is generated, ignored, and must never be -committed. In addition to keeping distribution artifacts out of the source -tree, this ensures rename-aware quality gates compare the previous package -directly with its current source path instead of matching it to a duplicate -under `build/lib/`. +Generated documentation and distribution output lives under the ignored, +hidden `.artifacts/` directory and must never be committed. Packaging scratch +output and setuptools `.egg-info` metadata are directed beneath the same hidden +root. Keeping every generated copy out of the maintained source tree ensures +rename-aware quality gates compare the previous package directly with its +current source path instead of matching it to a generated duplicate. PyPI publication is deliberately separate from ordinary push and pull-request workflows. Publishing a GitHub Release triggers a build job, followed by a protected `pypi` environment job that authenticates through OpenID Connect. -See the [release process](release-process.md) for the exact trusted-publisher +See the [release process](release.md) for the exact trusted-publisher identity and approval sequence. ## Test Platforms @@ -105,7 +106,7 @@ Enable the repository once through **Settings > Pages > Build and deployment > Source > GitHub Actions**. Then open **Actions > Documentation > Run workflow**, select `main`, and run it. Later documentation changes deploy automatically after they are merged or pushed to `main`; maintainers do not build or upload -`site/` themselves. +`.artifacts/site/` themselves. Before changing a page to `publication: reviewed`, preview the production view with `python3 -m mkdocs serve`. Use @@ -113,7 +114,7 @@ with `python3 -m mkdocs serve`. Use pages with their draft warning. The lane index must also be reviewed before a page in that lane can enter the deployed artifact. -## TODO - -- TODO: Document the complete current CI quality gates and scheduled jobs. -- TODO: Link coverage troubleshooting to the maintained quality page. +Coverage troubleshooting and the exact local parity commands live in +[Quality Assurance](quality-assurance.md). Workflow-specific implementation +details remain in `.github/workflows/`; this page records the stable pipeline +contract rather than duplicating every YAML step. diff --git a/docs/developer/workflows/contributing.md b/docs/developer/workflows/contributing.md new file mode 100644 index 000000000..07818838c --- /dev/null +++ b/docs/developer/workflows/contributing.md @@ -0,0 +1,193 @@ +--- +title: Contributing Workflow +audience: developers, maintainers, contributors +prerequisites: repository checkout, Python 3.10 or newer +related: ../architecture.md, quality-assurance.md, ../testing-strategy.md, documentation.md +status: maintained +publication: reviewed +--- + +# Contributing Workflow + +This is the practical workflow for changing PRIK. The root +[`CONTRIBUTING.md`](../../../CONTRIBUTING.md) is the short public entrypoint; +this page supplies the complete contributor sequence without duplicating +package architecture. + +## Prepare The Checkout + +```bash +python3 -m pip install -e ".[qa]" +git config core.hooksPath .githooks +``` + +Create a focused branch and begin with the smallest test owner for the +behavior. Do not start with the full suite while discovering the change. + +## Change Workflow + +1. Identify the public behavior, limitation, or internal invariant. +2. Use the [architecture guide](../architecture.md), + [source map](../source-map.md), or + [feature-to-code map](../feature-to-code-map.md) to find its owner. +3. Read the owning package guide and relevant user contract before editing. +4. Update the documentation contract first when public behavior, ownership, + or limitations change. +5. Add or update focused tests at the earliest stage that proves the behavior. +6. Implement the change in the owning stage and extend downstream stages only + when their representation or mechanism genuinely changes. +7. Run focused verification, then the required static and broader checks. +8. Add a concise **Unreleased** changelog entry for visible behavior, + workflows, examples, supported features, or limitations. + +For policy-sensitive wrapper work, semantic decisions must be complete before +planning. A binding or bridge change should implement a newly selected plan +mechanism, not infer a new policy from datatype, intent, aliases, or storage. + +## Support Evidence Rule + +Documentation may claim support only when current implementation and evidence +prove it. Acceptable evidence includes: + +- a focused test for the contract; +- a maintained golden that proves exact generated representation; +- a checked repository command using a maintained fixture; or +- a compiled/imported/called runtime test for wrapper behavior. + +Parser support does not establish semantic or wrapper support. Compilation +alone does not establish runtime behavior. Unsupported cases should fail at +the earliest stage with enough facts to report a stable diagnostic. + +## Documentation Examples + +Important production files expose small public-API examples under +`if __name__ == "__main__"`; package guides document their exact commands and +outputs. Their centralized execution owner is +[`test_execution_examples.py`](../../../tests/fortran/infrastructure/execution_examples/test_execution_examples.py). + +Markdown snippets use the repository's checked markers: + +````markdown + +```bash +python3 -m prik parse path/to/example.f90 +``` + + +```text +File: path/to/example.f90 +... +``` +```` + +Use `prik-doc-test: run` when only successful execution is stable. Use +`prik-doc-source` for fixture-backed source blocks. Do not mark placeholder, +checkout-modifying, compiler-environment-dependent, or intentionally failing +commands as executable documentation. + +Run the example documentation checks with: + +```bash +python3 -m pytest -q tests/docs/test_examples.py +``` + +## Selecting Tests + +Use the [testing strategy](../testing-strategy.md) for the authoritative +placement rules. Common starting points are: + +```bash +python3 -m pytest -q tests/fortran/source_parsing/parsing/ +python3 -m pytest -q tests/fortran/semantic_ir/semantics/ +python3 -m pytest -q tests/fortran/infrastructure/semantics/ +python3 -m pytest -q tests/fortran/infrastructure/codegen/ +python3 -m pytest -q tests/fortran/command_line_interface/pipeline/ +python3 -m pytest -q tests/docs +``` + +Use a feature-local `policy/`, `codegen/`, `runtime/`, or `end_to_end/` owner +when the behavior belongs to a documented feature. Compiled tests must import +and call the generated API; build success alone is insufficient. + +## Common Change Routes + +### Add A Fortran Construct + +1. Add the smallest parser example under + `tests/fortran/source_parsing/parsing/` or the feature's parsing owner. +2. Preserve the new source fact in `prik/parsers/fortran/`; add model fields + only when downstream consumers need them. +3. Extend `prik/semantics/fortran2ir.py` and semantic tests only if the + language-neutral contract changes. +4. Complete any new ownership, projection, setter, or support decision in + `prik/policy/` before planning. +5. Extend the plan and named binding/bridge lowering mechanisms only when the + completed behavior needs a new representation. +6. Add feature-local codegen and end-to-end evidence, then update the user + guide and feature matrix. + +Regenerate only an intentionally changed Fortran parser fixture: + +```bash +python3 tests/fortran/source_parsing/parsing/generate_parser_goldens.py \ + tests/fortran/source_parsing/parsing/fixtures/general/basic_subroutine.f90 +``` + +### Add Semantic `.pyi` Syntax Or Projection + +1. Add syntax tests under `tests/fortran/semantic_pyi_format/parsing/`. +2. Change `prik/parsers/pyi/parser.py` only if raw Python AST parsing changes; + otherwise interpret the syntax in `prik/semantics/pyi2ir.py`. +3. Update `prik/printers/pyi.py` and round-trip tests for emitted syntax. +4. Update semantic models only when the IR needs a new contract fact. +5. Complete new behavior in policy, project it through planning, and add + runtime evidence when the edit affects wrappers. +6. Update the semantic `.pyi` user reference. + +### Add A Code-Generation Backend Or Mechanism + +A new backend is not accepted merely because it prints source. It must consume +the completed shared plan without importing construction rules, define its own +typed representation and printer boundary, fail closed on unsupported action +combinations, preserve shared native slots and lifecycle ordering, and provide +focused generation plus compiled/runtime evidence. Add a backend only after +the shared plan can express its requirements without backend-specific semantic +policy. + +For a mechanism inside an existing backend, start in the narrow specialized +emitter named by the package guide. Do not replace specialized methods with a +flag-driven generic emitter or move semantic decisions down to make the +mechanism easier to generate. + +### Add A Stage-Owned Error + +Report a failure at the first stage with enough facts to explain it. Syntax and +source-processing failures belong to preprocessing/parsing; invalid contracts +belong to semantic conversion; unsafe ownership, ABI, projection, or support +belongs to completed policy; inconsistent plan projection belongs to planning; +an unavailable emitted mechanism belongs to backend preflight. Assert the +stable owner path and reason at that stage rather than forcing a known failure +through native compilation. + +## Pull Request And Review + +Before opening a pull request: + +- keep the change focused and remove superseded implementation/tests/docs; +- explain the problem, stage ownership, solution, and verification; +- identify user-visible behavior and limitations; +- run the applicable focused tests and the required checks from + [Quality Assurance](quality-assurance.md); and +- ensure all required GitHub checks pass before merge. + +Review should verify dependency direction, completed-policy authority, +diagnostic ownership, focused and end-to-end evidence, generated ABI stability, +documentation consistency, and removal of obsolete paths. Reviewers should not +accept a compatibility alias for an intentionally moved internal API unless +the change explicitly requires one. + +## Contribution License + +PRIK is distributed under the MIT License. By submitting a contribution, a +contributor agrees to license it under the same terms and confirms they have +the right to do so, including any required employer authorization. diff --git a/docs/maintainer/documentation-architecture.md b/docs/developer/workflows/documentation.md similarity index 78% rename from docs/maintainer/documentation-architecture.md rename to docs/developer/workflows/documentation.md index de97b6aa3..487c2cae5 100644 --- a/docs/maintainer/documentation-architecture.md +++ b/docs/developer/workflows/documentation.md @@ -1,8 +1,8 @@ --- title: Documentation Architecture -audience: maintainers +audience: developers, maintainers, contributors prerequisites: repository checkout, documentation metadata standard -related: README.md, ../developer/testing-strategy.md, ../user/index.md +related: ../architecture.md, ../testing-strategy.md, ../../user/index.md status: maintained publication: draft --- @@ -11,23 +11,25 @@ publication: draft This page defines how prik documentation is organized and maintained. It is a repository-governance contract, not part of the product-learning material. -`mkdocs.yml` owns the complete intended navigation for the User, Developer, -and Maintainer lanes. A publication hook filters that tree so GitHub Pages -contains only pages explicitly marked as reviewed. +`mkdocs.yml` owns the complete intended navigation for the User and Contributor +areas. A publication hook filters that tree so GitHub Pages contains only +pages explicitly marked as reviewed. ## Architecture Principles -1. Active documentation has three physical lanes: `user/`, `developer/`, and - `maintainer/`. Every lane may be published after review. -2. `docs/index.md` is the website entry point. Each lane index gates its whole - lane: a draft lane index prevents every page below that lane from entering +1. Active documentation has two physical areas: `user/` and `developer/`. The + latter is presented as Contributor Documentation and serves developers, + maintainers, and future contributors from one architectural account. Both + areas may be published after review. +2. `docs/index.md` is the website entry point. Each area index gates its whole + area: a draft area index prevents every page below that area from entering the production site, even when an individual child page is marked reviewed. 3. Implemented behavior is documented as supported only when current code and tests prove it. Public user pages describe behavior and limits without exposing internal test-evidence ledgers. 4. Planned behavior is marked explicitly and never presented as an implemented user contract. -5. Maintainer policy and volatile internals do not appear in user workflows. +5. Contributor governance and volatile internals do not appear in user workflows. 6. Historical material remains under `old_docs/` and outside active navigation. 7. User-facing source-driven examples show the complete input source before the command that consumes it. Generated paths must come from an immediately @@ -49,23 +51,23 @@ contains only pages explicitly marked as reviewed. `PRIK — Python Runtime Interop Kit` identity and public description, shows the shortest checked source-to-import workflow, summarizes the product's concrete advantages, and links to real-library evidence before sending the reader into -Getting Started. Developer, Maintainer, and deeper User Guide destinations stay -available through site navigation instead of competing with that first task. +Getting Started. Contributor and deeper User Guide destinations stay available +through site navigation instead of competing with that first task. The FAQ uses natural task questions as concise routes to authoritative guides; it does not duplicate those guides. -## Audience Lanes +## Audience Areas -| Lane | Primary reader | Publication | Content | +| Area | Primary reader | Publication | Content | | --- | --- | --- | --- | | `user/` | People using prik | Documentation website after review | Getting Started, guides, performance benchmarks, tutorials, examples, public reference, support status, FAQ, troubleshooting | -| `developer/` | People changing prik | Documentation website after review | Source orientation, implementation maps, testing, coding standards, feature work, contribution workflow | -| `maintainer/` | People governing prik | Documentation website after review | Documentation policy, design decisions, internal architecture, CI administration, releases, roadmaps | +| `developer/` | Developers, maintainers, and future contributors changing or governing prik | Documentation website after review | Architecture, package guides, cross-stage concepts, source navigation, testing, contribution and project workflows, design decisions, active roadmaps, and deferred input-language references | -Pages use their primary audience for placement. A developer may consult a -maintainer design record, but that does not make governance material part of -the developer workflow. Cross-lane links should be exceptional and explain why -the reader is leaving the current lane. +Pages use their task and stability for placement within the contributor tree. +Implemented architecture, design proposals, roadmaps, and release procedures +remain separate topics, but they do not claim separate architectural +audiences. Cross-area links between user and contributor documentation should +explain why the reader is leaving the current task. ## Reading Order And Cross-Links @@ -86,7 +88,7 @@ metadata and a TODO section. Each page includes the behavior, warning, ownership fact, or limitation needed for its current task. A forward reference never defers a fact needed now. -README documentation lists, lane indexes, and explicit navigation menus are +README documentation lists, area indexes, and explicit navigation menus are exceptions because choosing a destination is their purpose. Same-page anchors and links to source or test evidence do not change documentation reading order. Contextual links to `user/reference/pyi-contracts/` are also allowed after a @@ -117,9 +119,8 @@ as `draft`. Production builds include a Markdown page only when: 1. its own front matter says `publication: reviewed`; 2. `docs/index.md` is reviewed; and -3. for a page in `user/`, `developer/`, or `maintainer/`, that lane's index is - also reviewed (`user/index.md`, `developer/index.md`, or - `maintainer/README.md`). +3. for a page in `user/` or `developer/`, that area's index is also reviewed + (`user/index.md` or `developer/index.md`). The publication hook removes every other Markdown page from the MkDocs file collection and navigation before rendering, so drafts do not enter generated @@ -167,15 +168,16 @@ docs/ troubleshooting/ developer/ index.md - contributing/ - source and workflow pages - maintainer/ - README.md - documentation-architecture.md + architecture.md + source-map.md + feature-to-code-map.md + testing-strategy.md + packages/ + concepts/ + workflows/ design/ - internal-architecture/ roadmap/ - CI and release policy + deferred/ javascripts/ code-copy.js stylesheets/ @@ -188,11 +190,11 @@ The repository-root `CHANGELOG.md` is the canonical release history. It lives beside `README.md` and `pyproject.toml` so GitHub and package users can find it without navigating the documentation website. -New active pages must be created in one of the three lanes. Website-only static +New active pages must be created in one of the two areas. Website-only static behavior and presentation assets live in `javascripts/` and `stylesheets/`. -Do not restore top-level topic directories or place maintainer rules beside the -website landing page. Historical `old_docs/` material is never eligible for -website publication. +Do not restore separate developer/maintainer architecture trees or place +contributor governance beside the website landing page. Historical +`old_docs/` material is never eligible for website publication. The Performance page keeps its explanatory text and reproduction workflow in reviewed Markdown. Result-dependent summary, table, and environment blocks are @@ -219,9 +221,9 @@ the order-specific suites remain in the uploaded artifact for auditability. ## Continuous Documentation Quality - Require metadata for every active page. -- Keep website navigation, repository routing, lane indexes, and physical lanes +- Keep website navigation, repository routing, area indexes, and physical areas synchronized. -- Reject draft pages and draft-gated lanes from published site navigation. +- Reject draft pages and draft-gated areas from published site navigation. - Require explicit publication metadata on every active page. - Check that User documentation does not link forward from instructional prose, except for the documented contextual `.pyi` contract references. diff --git a/docs/developer/workflows/quality-assurance.md b/docs/developer/workflows/quality-assurance.md new file mode 100644 index 000000000..4a7db95fb --- /dev/null +++ b/docs/developer/workflows/quality-assurance.md @@ -0,0 +1,148 @@ +--- +title: Quality Assurance +audience: developers, maintainers, contributors +prerequisites: repository checkout, QA dependencies +related: contributing.md, ../testing-strategy.md, ci.md +status: maintained +publication: draft +--- + +# Quality Assurance + +This page records the active quality stack and commands. Historical rollout +logs and completed tool-adoption checklists belong in Git history, not in the +current contributor workflow. + +## Install + +```bash +python3 -m pip install -e ".[qa]" +python3 tools/check_static_analysis_versions.py +``` + +## Active Cadence + +| Cadence | Evidence | +| --- | --- | +| Inner loop | Smallest owning pytest target and Ruff on changed code | +| Local pre-push | Blocking static analysis, focused documentation smoke, one compiled scalar-wrapper smoke, `tests/tools/`, and `tests/workflows/` | +| Pull request | Static analysis, compiler smoke, Python matrix, project coverage, real libraries, performance/docs, aggregate required check | +| Manual discovery | Deep Hypothesis fuzz profile and advisory complexity reports | +| Dependency change or annual review | Dependency vulnerability review | + +The one stable required ruleset context is: + +```text +Pull Request / Validation · all required checks +``` + +If the workflow or job display name changes, update the repository ruleset; +do not keep an alias job for the old name. + +## Focused And Documentation Checks + +Run the narrowest behavioral owner first: + +```bash +python3 -m pytest -q path/to/owning/tests +``` + +Documentation-only changes that do not alter Python, tests, build +configuration, or tooling use: + +```bash +python3 -m pytest -q tests/docs +git diff --check +``` + +When Python code, test logic, build behavior, or tools change, run the complete +blocking and advisory static suite: + +```bash +python3 -m ruff check . +python3 -m ruff format --check . +python3 tools/check_static_analysis_versions.py +python3 tools/check_codegen_complexity.py +python3 -m bandit -c pyproject.toml -r prik --severity-level medium --confidence-level medium +python3 -m vulture +python3 tools/check_radon_policy.py --base-ref auto +python3 -m radon cc prik -n C -s --total-average +python3 -m radon mi prik -s +``` + +Ruff, Bandit, Vulture, codegen complexity, version checks, and the changed-code +Radon policy are blocking. Full Radon reports are advisory. If automatic Radon +base detection lacks CI SHA metadata locally, rerun with `--base-ref main` and +report that fact. + +## Coverage And Test-Order Reproduction + +Do not run the complete coverage workflow for routine changes. When +investigating a CI coverage failure, mirror subprocess collection exactly: + +```bash +COVERAGE_PROCESS_START=pyproject.toml \ +PYTHONPATH=. \ +python3 -m coverage run -m pytest -q --randomly-seed=1 +python3 -m coverage combine +python3 -m coverage report +``` + +The blocking project coverage target is 90%. Codecov patch status is +informational, but new reachable behavior still needs focused tests. + +Reproduce an order-dependent failure with the seed from CI: + +```bash +python3 -m pytest -q --randomly-seed= +``` + +## Compiler And Property Evidence + +Run a configured alternate-compiler lane with: + +```bash +python3 tools/run_fortran_toolchain_lane.py --compiler=/path/to/ifx +python3 tools/run_fortran_toolchain_lane.py --compiler=/path/to/flang +``` + +`--plan` prints the selected tests without running them. CI currently pins +IFX/ICX 2026.1.1 and Flang/Clang 22.1.8 as evidence versions, not declared +minimum versions. + +Run property and deep fuzz profiles with: + +```bash +python3 -m pytest -q -m property --hypothesis-profile=ci +HYPOTHESIS_PROFILE=fuzz python3 -m pytest -q -m fuzz --hypothesis-show-statistics +``` + +Minimize an actionable fuzz failure and preserve it as a focused regression in +the owning feature/stage suite. + +## Tool Responsibilities + +| Tool | Role | +| --- | --- | +| pytest and coverage.py | Behavioral regression and project coverage | +| pytest-randomly | Stable-seed order-coupling detection | +| Hypothesis | Generated parser, semantic, and codegen invariants | +| Ruff | Linting, formatting, modernization, and bounded McCabe checks | +| Bandit | Medium-confidence/severity security boundary review | +| Vulture | Dead-code detection with narrow exclusions | +| Radon | Blocking changed-hotspot policy plus advisory project reports | +| GitHub Actions | Reproducible shared compiler, platform, library, benchmark, docs, and release evidence | + +Mutation testing and pre-commit are not part of the active stack. The tracked +`.githooks` pre-push hook is the supported local automation boundary. + +## Real Libraries And Verification Limits + +Ordinary local suites exclude `real_library`. BLAS, FFTPACK, and MINPACK may be +run through their documented example workflows. LAPACK wrapper tests remain a +GitHub Actions responsibility unless explicitly requested locally. + +GitHub Actions writes path-aware JUnit reports and prints failed pytest node +IDs at the end of failed matrix logs. The real-library lane builds and tests +the complete maintained BLAS, LAPACK, FFTPACK, and MINPACK examples using their +documented entrypoints. diff --git a/docs/maintainer/release-process.md b/docs/developer/workflows/release.md similarity index 83% rename from docs/maintainer/release-process.md rename to docs/developer/workflows/release.md index 7ab302e08..a38c3aee8 100644 --- a/docs/maintainer/release-process.md +++ b/docs/developer/workflows/release.md @@ -2,7 +2,7 @@ title: Release Process audience: maintainers prerequisites: CI/CD, changelog -related: ci-cd.md +related: ci.md, quality-assurance.md status: maintained publication: reviewed --- @@ -53,7 +53,7 @@ approval. Do not add a PyPI API token or password to GitHub secrets. 1. Choose a version that does not already exist on PyPI. 2. Set `[project].version` in `pyproject.toml`. 3. Move the user-visible entries from **Unreleased** into a versioned section - in the repository-root [`CHANGELOG.md`](../../CHANGELOG.md). + in the repository-root [`CHANGELOG.md`](../../../CHANGELOG.md). 4. Run the focused package checks and the repository's required static analysis. Let GitHub Actions run the complete cross-platform suite. 5. Merge the release preparation through the normal review process and wait @@ -63,13 +63,17 @@ Build the same artifacts locally when reviewing the release candidate: ```bash python3 -m pip install --upgrade build twine -python3 -m build -python3 -m twine check dist/* +python3 -m build --outdir .artifacts/dist +python3 -m twine check .artifacts/dist/* ``` -`dist/` must contain one source distribution and one universal wheel. The -source distribution must include the repository-root `CHANGELOG.md`. Install -the wheel in a fresh virtual environment and verify `prik --version`, +`.artifacts/dist/` must contain one source distribution and one universal +wheel. The hidden `.artifacts/` tree contains reproducible local and CI output; +only its ignore placeholder is maintained source. The repository-root +`setup.cfg` also directs setuptools' temporary `.egg-info` metadata into that +hidden tree. The source distribution must include the repository-root +`CHANGELOG.md`. Install the wheel in a fresh virtual environment and verify +`prik --version`, `prik.__version__`, `prik --help`, and `python -m prik --help` before creating the release. @@ -77,7 +81,7 @@ the release. Create a GitHub Release from the exact reviewed commit and use a tag matching the project version, such as `v0.1.0`. Use that version's section from -[`CHANGELOG.md`](../../CHANGELOG.md) as the release notes. Publishing the +[`CHANGELOG.md`](../../../CHANGELOG.md) as the release notes. Publishing the GitHub Release triggers `.github/workflows/publish-to-pypi.yml`. The workflow builds and checks the artifacts in an unprivileged job. A diff --git a/docs/maintainer/README.md b/docs/maintainer/README.md deleted file mode 100644 index 847262d02..000000000 --- a/docs/maintainer/README.md +++ /dev/null @@ -1,40 +0,0 @@ ---- -title: Maintainer Documentation -audience: maintainers -prerequisites: developer documentation -related: documentation-architecture.md, internal-architecture/index.md, roadmap/index.md -status: maintained -publication: draft ---- - -# Maintainer Documentation - -This lane is for project governance and long-term technical ownership. It is -not part of the user learning sequence or the ordinary contributor workflow. - -## Governance - -- [Documentation architecture](documentation-architecture.md) -- [CI/CD](ci-cd.md) -- [Release process](release-process.md) - -## Architecture And Decisions - -- [Design documents](design/index.md) -- [Internal architecture](internal-architecture/index.md) -- [Pipeline map](internal-architecture/pipeline-map.md) - -## Planning - -- [Roadmap](roadmap/index.md) -- [Language-first test suite and Fortran pipeline cleanup](roadmap/fortran-test-suite-cleanup-checklist.md) -- [Semantic `.pyi` wrapper checklist](roadmap/semantic-pyi-wrapper-checklist.md) -- [Native array handle checklist](roadmap/native-array-handle-checklist.md) -- [Documentation content checklist](roadmap/documentation-content-checklist.md) - -Implementation orientation, source maps, feature workflows, tests, and -contribution requirements remain in the separate -[Developer documentation](../developer/index.md) lane. - -The historical `docs/old_docs/` archive is retained for comparison only and is -excluded from the website and active navigation. diff --git a/docs/maintainer/design/code-generation.md b/docs/maintainer/design/code-generation.md deleted file mode 100644 index 2fd6b74ee..000000000 --- a/docs/maintainer/design/code-generation.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -title: Code Generation -audience: maintainers -prerequisites: semantic analysis -related: cpython-integration.md, runtime-model.md -status: planned-documentation -publication: draft ---- - -# Code Generation - -Reserved design page for lowering semantic IR into wrapper bridge and binding -artifacts. - -## TODO - -- TODO: Document supported code generation targets and deferred backend policy. -- TODO: Link dispatch tables, bridge generation, and binding generation to - internal architecture pages. diff --git a/docs/maintainer/design/cpython-integration.md b/docs/maintainer/design/cpython-integration.md deleted file mode 100644 index 3e0f792c8..000000000 --- a/docs/maintainer/design/cpython-integration.md +++ /dev/null @@ -1,27 +0,0 @@ ---- -# PRIK_C_DOCS: title: CPython Integration -title: Deferred Python Extension Integration -audience: maintainers -prerequisites: code generation -related: runtime-model.md, error-propagation-model.md -status: planned-documentation -publication: draft ---- - - - - - - - - diff --git a/docs/maintainer/design/error-propagation-model.md b/docs/maintainer/design/error-propagation-model.md deleted file mode 100644 index f43f33450..000000000 --- a/docs/maintainer/design/error-propagation-model.md +++ /dev/null @@ -1,22 +0,0 @@ ---- -title: Error Propagation Model -audience: maintainers -prerequisites: runtime model -related: ../../user/guide/error-handling.md, cpython-integration.md -status: planned-documentation -publication: draft ---- - -# Error Propagation Model - -Reserved design page for diagnostic reporting, stage-owned errors, compiler -failures, runtime exceptions, and callback exceptions. - -## TODO - -- TODO: Link diagnostics and Python exceptions to troubleshooting pages. - - diff --git a/docs/maintainer/design/index.md b/docs/maintainer/design/index.md deleted file mode 100644 index 92bb8afd1..000000000 --- a/docs/maintainer/design/index.md +++ /dev/null @@ -1,35 +0,0 @@ ---- -title: Design Documents -audience: maintainers -prerequisites: developer documentation, user contracts -related: ../internal-architecture/index.md, semantic-multilanguage-wrapper-runtime-architecture.md -status: planned-documentation -publication: draft ---- - -# Design Documents - -Design documents record long-term technical decisions for maintainers. They do -not by themselves establish native binding support. - -## Pages - -- [Wrapper design notes](wrapper-design-notes.md) -- [Semantic multilanguage wrapper runtime architecture](semantic-multilanguage-wrapper-runtime-architecture.md) -- [Overall architecture](overall-architecture.md) -- [Parser architecture](parser-architecture.md) -- [Semantic analysis](semantic-analysis.md) -- [Code generation](code-generation.md) -- [Runtime model](runtime-model.md) -- [Memory ownership model](memory-ownership-model.md) -- [Error propagation model](error-propagation-model.md) - - - -## TODO - -- TODO: Promote stable design explanations from existing notes into this tree. -- TODO: Keep design-only material clearly separated from supported user - behavior. diff --git a/docs/maintainer/design/memory-ownership-model.md b/docs/maintainer/design/memory-ownership-model.md deleted file mode 100644 index 0c451bb64..000000000 --- a/docs/maintainer/design/memory-ownership-model.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -title: Memory Ownership Model -audience: maintainers -prerequisites: runtime model -related: ../../user/guide/memory-management.md, error-propagation-model.md -status: planned-documentation -publication: draft ---- - -# Memory Ownership Model - -Reserved design page for owner categories, transfer modes, lifetime invariants, -and blocked unsafe cases. - -## TODO - -- TODO: Promote the ownership model from the wrapper guide into a design - document. -- TODO: Link every owner category to runtime examples and tests. diff --git a/docs/maintainer/design/overall-architecture.md b/docs/maintainer/design/overall-architecture.md deleted file mode 100644 index a7b0a095d..000000000 --- a/docs/maintainer/design/overall-architecture.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -title: Overall Architecture -audience: maintainers -prerequisites: documentation architecture -related: parser-architecture.md, code-generation.md -status: planned-documentation -publication: draft ---- - -# Overall Architecture - -Reserved design page for the end-to-end architecture from native source to -Python extension and semantic inspection outputs. - -## TODO - -- TODO: Add the complete pipeline diagram and ownership boundaries. -- TODO: Link each architectural stage to implementation and tests. diff --git a/docs/maintainer/design/parser-architecture.md b/docs/maintainer/design/parser-architecture.md deleted file mode 100644 index bdc7f3804..000000000 --- a/docs/maintainer/design/parser-architecture.md +++ /dev/null @@ -1,22 +0,0 @@ ---- -title: Parser Architecture -audience: maintainers -prerequisites: overall architecture -related: semantic-analysis.md, ../../developer/c-parser-reference.md, ../../developer/fortran-parser-reference.md -status: planned-documentation -publication: draft ---- - -# Parser Architecture - -Reserved design page for parser frontends, preprocessing, source facts, -diagnostics, and fixture strategy. - -## TODO - -- TODO: Document parser extension rules for new native constructs. - - diff --git a/docs/maintainer/design/runtime-model.md b/docs/maintainer/design/runtime-model.md deleted file mode 100644 index 7e4f32007..000000000 --- a/docs/maintainer/design/runtime-model.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -title: Runtime Model -audience: maintainers -prerequisites: CPython integration -related: memory-ownership-model.md, error-propagation-model.md -status: planned-documentation -publication: draft ---- - -# Runtime Model - -Reserved design page for runtime helper libraries, generated artifacts, native -calls, and wrapper execution. - -## TODO - -- TODO: Describe runtime helper responsibilities and generated artifact - boundaries. -- TODO: Document thread, callback, and OpenMP runtime considerations. diff --git a/docs/maintainer/design/semantic-analysis.md b/docs/maintainer/design/semantic-analysis.md deleted file mode 100644 index bfd3f7963..000000000 --- a/docs/maintainer/design/semantic-analysis.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -title: Semantic Analysis -audience: maintainers -prerequisites: parser architecture -related: code-generation.md, ../../user/reference/semantic-ir.md -status: planned-documentation -publication: draft ---- - -# Semantic Analysis - -Reserved design page for conversion from parser facts into language-neutral -semantic IR and wrapper-planning errors. - -## TODO - -- TODO: Document semantic passes, normalization policy, and error boundaries. -- TODO: Link semantic behavior to `.pyi` and error-handling references. diff --git a/docs/maintainer/internal-architecture/ast-design.md b/docs/maintainer/internal-architecture/ast-design.md deleted file mode 100644 index 72cc53bd4..000000000 --- a/docs/maintainer/internal-architecture/ast-design.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -title: AST Design -audience: maintainers -prerequisites: parser architecture -related: symbol-tables.md, type-system.md -status: planned-documentation -publication: draft ---- - -# AST Design - -Parser models and parsed semantic `.pyi` files may use syntax trees to preserve -source structure. Wrapper generation has no shared codegen AST: completed -semantic policy is projected into `WrapperPlan`, then the C binding and Fortran -bridge lower directly into their backend-specific source-syntax nodes. - -## TODO - -- TODO: Document AST ownership, source locations, and invariants. -- TODO: Link parser models, wrapper-plan records, and generated source-syntax - nodes to their tests. diff --git a/docs/maintainer/internal-architecture/dependency-analysis.md b/docs/maintainer/internal-architecture/dependency-analysis.md deleted file mode 100644 index 377094b5d..000000000 --- a/docs/maintainer/internal-architecture/dependency-analysis.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -title: Dependency Analysis -audience: maintainers -prerequisites: semantic passes -related: wrapper-generation-pipeline.md, symbol-tables.md -status: planned-documentation -publication: draft ---- - -# Dependency Analysis - -Reserved maintainer page for source ordering, module imports, native object -linking, and generated artifact dependencies. - -## TODO - -- TODO: Document dependency analysis for source-driven and `.pyi`-driven builds. -- TODO: Link multi-source build tests and limitations. diff --git a/docs/maintainer/internal-architecture/error-handling-pipeline.md b/docs/maintainer/internal-architecture/error-handling-pipeline.md deleted file mode 100644 index b011f7335..000000000 --- a/docs/maintainer/internal-architecture/error-handling-pipeline.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -title: Error Handling Pipeline -audience: maintainers -prerequisites: runtime layer, error propagation model -related: runtime-layer.md, ../../user/reference/diagnostic-codes.md -status: planned-documentation -publication: draft ---- - -# Error Handling Pipeline - -Reserved maintainer page for diagnostics, stage-owned errors, generated error -paths, Python exception state, and cleanup on failure. - -## TODO - -- TODO: Document error propagation from native callbacks through Python - exceptions. -- TODO: Link cleanup and ownership behavior for failure paths. diff --git a/docs/maintainer/internal-architecture/index.md b/docs/maintainer/internal-architecture/index.md deleted file mode 100644 index 5c9f5978a..000000000 --- a/docs/maintainer/internal-architecture/index.md +++ /dev/null @@ -1,31 +0,0 @@ ---- -title: Internal Architecture -audience: maintainers -prerequisites: design documents, developer guide -related: ../design/index.md, ../../developer/development-workflow.md -status: planned-documentation -publication: draft ---- - -# Internal Architecture - -Internal architecture pages are for maintainers who need implementation-level -details. They are separate from user guides and high-level design documents. - -## Pages - -- [Pipeline map](pipeline-map.md) -- [AST design](ast-design.md) -- [Symbol tables](symbol-tables.md) -- [Type system](type-system.md) -- [Semantic passes](semantic-passes.md) -- [Dependency analysis](dependency-analysis.md) -- [Wrapper generation pipeline](wrapper-generation-pipeline.md) -- [Runtime layer](runtime-layer.md) -- [Ownership tracking](ownership-tracking.md) -- [Error handling pipeline](error-handling-pipeline.md) - -## TODO - -- TODO: Fill these pages from implementation evidence and maintainer workflows. -- TODO: Keep volatile internals out of user-facing workflow pages. diff --git a/docs/maintainer/internal-architecture/ownership-tracking.md b/docs/maintainer/internal-architecture/ownership-tracking.md deleted file mode 100644 index b51523de5..000000000 --- a/docs/maintainer/internal-architecture/ownership-tracking.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -title: Ownership Tracking -audience: maintainers -prerequisites: runtime layer, memory ownership model -related: runtime-layer.md, error-handling-pipeline.md -status: planned-documentation -publication: draft ---- - -# Ownership Tracking - -Reserved maintainer page for ownership policy resolution, transfer actions, -destruction, borrowed views, and finalization. - -## TODO - -- TODO: Document ownership policy entrypoints and dispatch tables. -- TODO: Link each ownership action to generated code and runtime tests. diff --git a/docs/maintainer/internal-architecture/pipeline-map.md b/docs/maintainer/internal-architecture/pipeline-map.md deleted file mode 100644 index 120c3551b..000000000 --- a/docs/maintainer/internal-architecture/pipeline-map.md +++ /dev/null @@ -1,282 +0,0 @@ ---- -title: Pipeline Map -audience: maintainers -prerequisites: source map, overall architecture -related: ../../developer/source-map.md, wrapper-generation-pipeline.md, runtime-layer.md -status: maintained -publication: draft ---- - -# Pipeline Map - -This page is the source-code route through the current wrapper and inspection -pipelines. It complements the user-facing wrapper mechanism in -`docs/user/reference/fortran-wrapper.md` with the implementation files a maintainer should -open at each stage. - -## Source-Driven Fortran Wrapper Pipeline - - - -| Stage | Main source | Input | Output | Primary evidence | -| --- | --- | --- | --- | --- | -| CLI request | `prik/cli.py` | source paths and stage flags | selected stage or wrapper build options | `tests/fortran/command_line_interface/pipeline/` | -| Build orchestration | `prik/pipeline/build.py` | ordered Fortran sources or `.pyi` contracts plus explicit native artifacts | `WrapperBuildResult`, `NativeBuildPlan`, and generated artifact plan | wrapper build-mode tests | -| Preprocessing | `prik/pipeline/preprocessing.py` | source path, compiler config | preprocessed source and dependency facts | preprocessing tests | -| Parser project model | `prik/parsers/fortran/parser.py` | preprocessed Fortran source | parser project with modules, procedures, types, visibility | Fortran parser fixture tests | -| Target probes | `prik/probes/fortran_types.py` | semantic type requirements and compiler flags | resolved kind/storage facts | Fortran type probe tests | -| Semantic IR | `prik/semantics/fortran2ir.py` | parser project and target facts | `SemanticModule` objects | semantic Fortran tests | -| Semantic policy completion | `prik/semantics/policy_completion.py`, `prik/semantics/ownership.py` | full semantic modules with signatures and `.pyi` overrides | semantic modules annotated with every ownership, transfer, destruction, mutability, storage, accessor, and projection decision needed by wrapper generation | ownership-policy tests | -| Wrapper planning | `prik/codegen/planner.py`, `prik/codegen/plan.py` | policy-completed semantic modules | typed wrapper plans consuming completed decisions without a separate support-analysis traversal | `tests/fortran/infrastructure/codegen/`, wrapper tests | -| Direct bridge and binding lowering | `prik/codegen/fortran/bridge.py`, `prik/codegen/c/binding.py`, `prik/codegen/generator.py` | validated typed wrapper plans | Fortran, C, and header syntax nodes | `tests/fortran/infrastructure/codegen/`, wrapper tests | -| Wrapper and semantic-contract printing | `prik/codegen/printers/` | wrapper syntax nodes or semantic IR | wrapper source files or semantic `.pyi` text | printer, generated-contract, and wrapper artifact tests | -| Compile and link | `prik/compiling/`, `prik/pipeline/build.py` | dependency-batched native objects, generated bridge and binding objects, compiler-process limit, and ordered link inputs | shared library | wrapper runtime and build-mode tests | - - - -## Concept Ownership Rules - -The pipeline keeps separate concepts for contract facts, policy decisions, -generated implementation, and emitted source. Similar names across layers do -not mean those classes should be merged. - -The Python package layout follows those ownership boundaries: - -| Package | Owns | Must not become | -| --- | --- | --- | -| `prik/contracts/` | The public semantic `.pyi` vocabulary | A home for semantic conversion or runtime type mapping | -| `prik/types/` | Mappings from resolved semantic types to Python ecosystem types | A second semantic IR model | -| `prik/probes/` | Compiler-derived target facts and reports built from those facts | Semantic policy or build orchestration | -| `prik/pipeline/` | Source preprocessing, semantic `.pyi` loading, and end-to-end wrapper build orchestration | Parser models, semantic decisions, or compiler implementation details | -| `prik/runtime/` | Python objects used by generated extensions at execution time | Build-time semantic or codegen policy | -| `prik/utilities/` | Small domain-neutral mechanisms such as class visitor dispatch | A miscellaneous home for semantic or pipeline concepts | - -Semantic metadata and ownership policy remain in `prik/semantics/` even when -codegen consumes them. Downstream use does not turn semantic authority into -cross-cutting infrastructure. - -| Concept family | Owner | What belongs there | What must stay out | -| --- | --- | --- | --- | -| Parser facts | parser packages | Source syntax, native declaration structure, source locations, and parser diagnostics | Wrapper policy, Python API projection, generated names, and compile/link decisions | -| Semantic policy completion and ownership | `prik/semantics/policy_completion.py` and `prik/semantics/ownership.py` | Completed policy choices for ownership, lifetime, output projection, replacement, and ABI safety | Raw parser syntax, backend-specific statement trees, and hidden lowering-time policy decisions | -| Typed wrapper plan | `prik/codegen/plan.py` and `prik/codegen/planner.py` | A validated, backend-neutral implementation plan projected from completed semantic decisions | Source-contract authority, policy inference, and target-language statement details | -| Printers and compilation | `prik/codegen/printers/`, `prik/compiling/`, and wrapper orchestration | Text emission, generated artifact layout, compiler commands, native objects, libraries, include directories, and link inputs | Semantic support decisions and plan rewriting policy | - - - -Use these rules when adding a new notion: - -- Put it in semantic IR when the fact changes the user-visible or native - contract, must be preserved in `.pyi`, is needed for source-free wrapper - replay, or is required before policy completion can decide support. -- Put it in semantic policy completion or ownership policy when it is a safety decision rather - than a source fact: for example borrowed versus copied data, visible versus - hidden native outputs, replacement rules, destructor ownership, or unsupported - ABI combinations. If the decision depends on full signature context, complete - it in `policy_completion.py` before wrapper planning. -- Put it in compiling or wrapping when it describes build inputs or build - execution: sources, objects, libraries, library directories, include - directories, compiler flags, link items, binding support files, and generated - artifact paths. - - - -Merge or move concepts only when their invariants match: - -- Merge a shared object only when it has the same meaning and lifetime in every - layer and carries no generated implementation state. Small immutable value - objects such as identity, origin, scalar-kind descriptors, or naming-policy - results are candidates. -- Move a codegen concept into semantics only when it can be represented without - a generated body, temporary, scope, include, or target-language expression and - the fact is needed for `.pyi`, policy completion, or source-free replay. -- Move a semantic concept into a wrapper plan only when it does not change the public - contract, native contract, completed policy, or `.pyi` representation and exists only - to print or compile wrapper code. - - - -Examples: - -- `@bind` and a native procedure name belong to semantic identity. The bridge - symbol used to call it belongs to codegen naming and lowering. -- Python keyword avoidance for a public name, such as a native `def` routine, - belongs to naming policy. The chosen public spelling is stored where the - contract needs it, while target-specific helper symbols stay generated. -- Wrapper syntax nodes, body statements, temporaries, includes, and backend - datatypes stay out of `prik/semantics/models.py`. - - - -## Stage Maintenance Map - -| Stage family | First files to read | Source navigation owner | -| --- | --- | --- | -| CLI and output routing | `prik/cli.py`, parser CLI helpers | `docs/developer/source-map.md`, `docs/developer/feature-to-code-map.md` | -| Source loading and preprocessing | `prik/pipeline/preprocessing.py` | `docs/developer/source-map.md`, parser references | -| Editable semantic contracts | `prik/parsers/pyi/parser.py`, `prik/pipeline/pyi.py`, `prik/semantics/pyi2ir.py`, `prik/codegen/printers/pyi_printer.py` | `docs/user/reference/semantic-pyi-format.md` | -| Semantic and wrapper-planning errors | `prik/semantics/fortran2ir.py`, `prik/semantics/policy_completion.py`, `prik/semantics/wrapper_policy.py`, `prik/codegen/planner.py` | `docs/user/guide/error-handling.md` | -| Wrapper policy and lowering | `prik/semantics/policy_completion.py`, `prik/semantics/ownership.py`, `prik/codegen/planner.py`, `prik/codegen/generator.py` | `docs/user/reference/fortran-wrapper.md`, ownership docs | -| Native build | `prik/pipeline/build.py`, `prik/compiling/compilers.py`, `prik/compiling/native_support.py` | compiling package README and build-system docs | - - - -## Semantic `.pyi` Wrapper Pipeline - -Semantic `.pyi` builds reuse the wrapper backend but start from edited -contracts and explicit native artifacts instead of reparsing native source for -the Python API. - -```text -.pyi contract - -> prik/parsers/pyi/parser.py - -> prik/pipeline/pyi.py - -> prik/semantics/pyi2ir.py - -> prik/semantics/native_contract.py - -> prik/semantics/policy_completion.py - -> prik/codegen/planner.py - -> prik/codegen/generator.py - -> compile and link pipeline -``` - -The `.pyi` path must preserve native ABI facts in the semantic contract. Missing -native build inputs or contradictory contract facts fail before bridge emission -or native compilation. Ownership, transfer, and destruction policy is completed -from the full `.pyi` signature before planning; the wrapper planner and backend -generators consume that completed policy and must not invent a different one. - -## Shared Semantic Policy Boundary - - - - - - - -The completed decision is also the only semantic input to bridge and binding -behavior selection. Each backend owns an explicit dispatch table keyed by the -completed object kind and codegen action. A selected leaf method may construct -backend-local helper variables, but it must not choose ownership, writeback, -nullability, release responsibility, or `stack`/`heap`/`alias` placement for the -contract value. Missing dispatch combinations are errors; there is no datatype- -based policy fallback in bridge or binding generation. - -CLI source inspection uses a compact language dispatch table for the source -portion of this route: - -```text -pipeline = SOURCE_SEMANTIC_PIPELINES[language] -parsed = pipeline.parser(...) -semantic_modules = pipeline.converter_to_ir(parsed, ...) -semantic_modules -> semantic policy completion -> wrapper planning or lowering -``` - -Per-language parser/converter entries may still perform target-specific -preprocessing or ABI/kind probes, but ownership, transfer, destruction, -mutability, nullability, projection, and lifetime decisions must stay out of -those entries and flow through semantic policy completion after IR exists. - -## Inspection-Only Pipeline - -Inspection stages stop before wrapper code generation: - -```text -native source - -> parser facts - -> semantic IR - -> semantic .pyi -``` - - - -## Where Failures Should Happen - -| Failure type | Preferred owner | -| --- | --- | -| Source cannot be preprocessed | `prik/pipeline/preprocessing.py` | -| Source syntax cannot be represented by prik's parser model | parser package | -| Source facts cannot form a semantic contract | semantic conversion | -| Ownership, lifetime, ABI, projection, or wrapper support decision is unsafe | `prik/semantics/ownership.py` or policy completion | -| A completed policy is internally inconsistent while being projected | wrapper planner at the owner being projected | -| Native-language validity does not affect prik's contract | Fortran or C compiler | -| Generated code cannot represent a supported plan | bridge or binding generator with focused tests | -| Compiler/linker invocation is wrong | `prik/compiling/` or `prik/pipeline/build.py` | -| Python binding behavior is wrong | generated binding, native support, or ownership policy | diff --git a/docs/maintainer/internal-architecture/runtime-layer.md b/docs/maintainer/internal-architecture/runtime-layer.md deleted file mode 100644 index 4c9077630..000000000 --- a/docs/maintainer/internal-architecture/runtime-layer.md +++ /dev/null @@ -1,17 +0,0 @@ ---- -title: Runtime Layer -audience: maintainers -prerequisites: wrapper generation pipeline -related: ownership-tracking.md, error-handling-pipeline.md -status: planned-documentation -publication: draft ---- - -# Runtime Layer - -Reserved maintainer page for shared runtime helpers used by generated wrappers. - -## TODO - -- TODO: Document runtime helper responsibilities and native/Python boundaries. -- TODO: Link array, callback, and allocation helpers to tests. diff --git a/docs/maintainer/internal-architecture/semantic-passes.md b/docs/maintainer/internal-architecture/semantic-passes.md deleted file mode 100644 index 844c6dde3..000000000 --- a/docs/maintainer/internal-architecture/semantic-passes.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -title: Semantic Passes -audience: maintainers -prerequisites: type system, symbol tables -related: dependency-analysis.md, ../../user/reference/semantic-ir.md -status: planned-documentation -publication: draft ---- - -# Semantic Passes - -Reserved maintainer page for parser-to-IR conversion, validation, policy completion, -and `.pyi` round trips. - -## TODO - -- TODO: List semantic passes in execution order with owning modules. -- TODO: Document blocker policy and pass-specific tests. diff --git a/docs/maintainer/internal-architecture/symbol-tables.md b/docs/maintainer/internal-architecture/symbol-tables.md deleted file mode 100644 index 5dfe9222c..000000000 --- a/docs/maintainer/internal-architecture/symbol-tables.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -title: Symbol Tables -audience: maintainers -prerequisites: AST design -related: type-system.md, dependency-analysis.md -status: planned-documentation -publication: draft ---- - -# Symbol Tables - -Reserved maintainer page for symbol collection, scope lookup, visibility, and -name resolution. - -## TODO - -- TODO: Document symbol table data structures and update rules. -- TODO: Link visibility and collision policy to wrapper tests. diff --git a/docs/maintainer/internal-architecture/type-system.md b/docs/maintainer/internal-architecture/type-system.md deleted file mode 100644 index 6ff12b8d9..000000000 --- a/docs/maintainer/internal-architecture/type-system.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -title: Type System -audience: maintainers -prerequisites: AST design, semantic IR -related: semantic-passes.md, ownership-tracking.md -status: planned-documentation -publication: draft ---- - -# Type System - -Reserved maintainer page for native type facts, semantic datatypes, NumPy dtype -mapping, and target probing. - -## TODO - -- TODO: Document how compiler-probed native kinds become semantic and wrapper - types. -- TODO: Link datatype mappings to generated examples and tests. diff --git a/docs/maintainer/internal-architecture/wrapper-generation-pipeline.md b/docs/maintainer/internal-architecture/wrapper-generation-pipeline.md deleted file mode 100644 index 5d4bb331f..000000000 --- a/docs/maintainer/internal-architecture/wrapper-generation-pipeline.md +++ /dev/null @@ -1,220 +0,0 @@ ---- -title: Wrapper Generation Pipeline -audience: maintainers -prerequisites: semantic passes, code generation design -related: runtime-layer.md, ownership-tracking.md, ../roadmap/wrapper-plan-migration-checklist.md -status: maintained -publication: draft ---- - -# Wrapper Generation Pipeline - -This page describes the canonical wrapper-plan generation route. It covers the -completed scalar, string, array, native-handle, derived-type, class, callback, -module-state, generic, and build surfaces. - -## Architectural Boundary - -All semantic policy must be complete before wrapper planning begins. Post-IR -policy completion owns -object kind, ownership, transfer, destruction, mutability, writeback, -nullability, output projection, release responsibility, storage mode, getter -behavior, native setter assignment, and Python setter exposure. - -Planning projects those completed decisions into one editable `ModulePlan`. -Validation checks that the projections agree. Binding and bridge generation -then dispatch only from completed selectors into small named lowering methods; -they do not reconstruct policy from datatype, `intent`, shape, alias flags, or -local memory checks. - -Native-source `intent` may be consumed while importing a source declaration to -propose default Python argument/result positions. It is not retained in the -semantic `.pyi` or post-IR ownership context. The editable Python signature, -`Returns[...]` projection, and ordered native-call mapping are authoritative. -Bridge entry dummies omit `intent`, leaving their storage permissive; that -contract controls wrapper copy-in, copy-back, and returned values, while the -compiled native procedure's own interface controls native access. - -Within that contract, an explicit native-call list is exhaustive for native -dummy positions. Matching named `Returns[...]` items attach result positions to -visible `Arg(i)` entries automatically; direct function results remain the first -ordinary Python return item, while hidden native output dummies require explicit -`Return(...)` entries. Descriptor reassociation follows the same rule: -`Pointer(Arg(i))` without a projected return uses a call-local adapter and -discards reassociation, while a matching projected return requires storage that -can preserve association writeback. - -Native transport overrides also live on that mapping. Primitive `Arg(i)` is a -value handoff and `Addr(Arg(i))` selects call-local address handoff. Wrapped -derived `Arg(i)` is a typed reference handoff and `Value(Arg(i))` selects exact -typed value handoff. `Returns[...]` never selects either ABI; it only assigns a -Python result position and writeback expectation. A derived `Value(...)` slot -does not expose aggregate layout at the C boundary: C still supplies an opaque -address, the bridge reconstructs the exact native type, and the Fortran compiler -applies the explicit interface's `VALUE` semantics at the typed call. - -Standalone legacy externals use a completed declaration mode. Procedures whose -ABI is valid with an implicit interface, including classic BLAS/LAPACK -subroutines and scalar functions, lower to `external` declarations; optional, -descriptor-rich, polymorphic, or array-result procedures retain explicit -interfaces. The bridge dispatches this completed mode and does not reclassify -the signature. - -The public direct-generation boundary is: - -```python -complete_semantic_policies(module) -plan = WrapperPlanner().build(module) -artifacts = WrapperCodeGenerator().generate(plan) -``` - -`WrapperCodeGenerator.generate()` freezes the plan, runs the shared validator, -runs both backend preflight checks, lowers recursively to C and Fortran syntax -nodes, and asks the source printers to render those nodes. Build integration -compiles the rendered sources; it does not own datatype transfer policy. -Wrapper C/Fortran source printers and the semantic `.pyi` printer share -`prik/codegen/printers/`; no compatibility printer remains under the -legacy codegen package. - -Wrapper builds have no legacy route or fallback. An unsupported completed plan -fails with its exact owner path before either backend emits source. - -## Stable Tree and Datatype-Varying Records - -The shared plan has stable module, namespace, and function orchestration: - -```text -ModulePlan - binding: BindingModulePlan - bridge: BridgeModulePlan - namespaces: NamespacePlan ... - functions: FunctionPlan ... - binding: BindingFunctionPlan - bridge: BridgeFunctionPlan - arguments: ArgumentTransferPlan ... - binding: BindingArgumentPlan - bridge: BridgeArgumentPlan - native_call_slot: NativeCallSlotPlan - results: ResultPlan ... - binding: BindingResultPlan - bridge: BridgeResultPlan - native_call_slot: NativeCallSlotPlan | None - native_call_slots: NativeCallSlotPlan ... - lifecycle actions: LifecycleActionPlan ... - variables: ModuleVariablePlan ... -``` - -Most datatype-specific work belongs to `ArgumentTransferPlan` and -`ResultPlan`. Each is one transfer with explicit binding and bridge views. -`ModuleVariablePlan` is the other intentionally datatype-sensitive surface, -because getter, setter, and native assignment behavior depend on the stored -value. - -`FunctionPlan`, `NamespacePlan`, and `ModulePlan` remain orchestration records. -They own export names, call order, result order, runtime/GIL envelopes, and -aggregation, but not datatype policy. - -Python-facing documentation is also a plan projection. The shared docstring -builder consumes completed namespace, module-variable, class, overload, -argument, result, and lifecycle records and stores the rendered text on the -owning plan nodes. C method-table emission and generated Python class assembly -only attach that text; neither backend infers signatures, ownership, mutation, -nullability, or exception behavior while rendering source. - -`NativeCallSlotPlan` and `LifecycleActionPlan` are subordinate transfer -details. Native slots stay indexed on `FunctionPlan` because native ABI order -can interleave argument slots, result slots, literals, and helpers. Lifecycle -actions stay indexed there because copy-out, cleanup, and release order may -span several arguments and results or differ on failure. Argument and hidden -result slots are the same mutable records referenced from both their transfer -owner and the function-wide index; they are not duplicated policy. - -## One Repeatable Transfer Algorithm - -Use this sequence for scalars, strings, arrays, and future datatype families: - -1. Post-IR policy completion classifies the value with `ObjectKind` and - completes ownership, transfer, storage, nullability, mutability, projection, - barrier actions, data action, and any justified copy reason. -2. Wrapper policy records the backend-neutral transfer and the ordered native - slot. It must report a blocker instead of leaving a semantic choice for a - backend. -3. `WrapperPlanner` mechanically projects one `ArgumentTransferPlan` or - `ResultPlan`, adds symbolic handoff roles, and shares the corresponding - `NativeCallSlotPlan` reference. -4. The shared validator checks graph consistency and common invariants, then - dispatches by the completed `object_kind` to scalar, string, or ordinary- - array validation. -5. Backend preflight dispatches by the same completed kind and action selectors - and rejects combinations it cannot lower. -6. The binding lowers Python extraction or result construction. The bridge - lowers ABI declarations, representation conversion, the ordered native - call, and native result production. Both communicate through planned - symbolic roles. -7. Function-level orchestration applies status handling and ordered lifecycle - actions, aggregates Python results, and returns. Printers and build - integration remain generic. - -When adding a datatype, first extend semantic policy and its transfer record, -then add one named validator and one named lowering method per affected -backend. Do not add a parallel plan hierarchy or datatype branches to module, -namespace, or function traversal. Add a new typed action only when the existing -selectors cannot express a genuine semantic choice. - -## Selector Vocabulary - -The action axes are deliberately orthogonal: - -| Selector | Question answered | Examples | -| --- | --- | --- | -| `ObjectKind` | What kind of object follows this route? | `SCALAR`, `STRING`, `NUMPY_ARRAY` | -| `source_kind` | Where is a result produced? | `direct_return`, `hidden_output` | -| `PythonBarrierAction` | How does the binding cross the Python boundary? | `SCALAR_VALUE`, `STRING_VALUE`, `ARRAY_STORAGE` | -| `NativeBarrierAction` | What native ABI transport is used? | `PASS_VALUE`, `PASS_CALL_LOCAL_ADDRESS`, `PASS_ARRAY_BUFFER` | -| `CodegenAction` | What ownership or transfer operation occurs? | `DIRECT_VALUE`, `CALL_LOCAL_INPUT`, `COPY_IN_OUT`, `COPY_OUT` | -| `BridgeDataAction` | What happens to the representation in the bridge? | `DIRECT_TRANSFER`, `ASSOCIATE_VIEW`, `COPY_REPRESENTATION` | -| `WritebackPhase` | When does a lifecycle operation run? | native mutation, copy-out, cleanup, release | - -Hiddenness is not a transfer operation. A hidden scalar result therefore uses -`source_kind="hidden_output"` with `CodegenAction.DIRECT_VALUE`; hidden strings -and ordinary arrays use the same source kind with `CodegenAction.COPY_OUT`. - -`NativeBarrierAction.PASS_ARRAY_BUFFER` identifies the Phase 6 ordinary-array -data-buffer ABI. Its handoff plan carries data, rank, extents, strides, and -itemsize. `PASS_NATIVE_DESCRIPTOR` is reserved for Phase 7 persistent native -descriptors and handles. Neither backend may substitute one for the other. -Array handoff shapes are completed bridge extents; native source bounds are -temporary import facts and must not appear in semantic `.pyi` or become extent -dependencies. A source dimension such as `0:LDB-1` therefore completes to -extent `LDB`, while the native procedure keeps control of its own indexing -bounds. When `PASS_NATIVE_DESCRIPTOR` also carries -optional absence, the completed optional mode lowers a valid call-local -placeholder descriptor plus a separate presence role. This keeps the bridge -entry ABI valid while presence dispatch omits the native dummy. - -`DatatypeFamily` remains useful after object-kind dispatch for primitive -element spelling and conversion, such as integer versus real scalar types or -the element type of an ordinary array. It must not be used to rediscover -whether the transfer itself is a scalar, string, or array. - -## Maintainer Inspection and Acceptance - -Inspect the real records directly with normal Python prints. The primary path -is `complete_semantic_policies()` -> `WrapperPlanner.build()` -> -`WrapperCodeGenerator.generate()`. Generated artifacts from real passing -feature-local `tests/fortran/*/end_to_end/` cases are the behavioral -oracle; plan unit tests cover action and graph invariants. Production source -and semantic-`.pyi` builds both use this one path; unsupported completed policy -is an error before lowering, not a request to retry a legacy generator. - -A wrapper-generation change is acceptable when: - -- semantic decisions are complete before planning; -- datatype variation is confined to transfer, result, lifecycle, or - module-variable records and their named handlers; -- scalar, string, and array routes use the same planning and validation - sequence; -- binding and bridge consume the same shared roles and native-slot records; -- no backend infers policy or silently falls back to another action; -- focused plan tests, relevant wrapper runtime tests, documentation checks, and - static analysis pass. diff --git a/docs/maintainer/roadmap/index.md b/docs/maintainer/roadmap/index.md deleted file mode 100644 index 35a524786..000000000 --- a/docs/maintainer/roadmap/index.md +++ /dev/null @@ -1,38 +0,0 @@ ---- -title: Roadmap -audience: maintainers -prerequisites: user language support, developer documentation -related: ../../user/language-support/feature-matrix.md, fortran-test-suite-cleanup-checklist.md, wrapper-plan-migration-checklist.md, semantic-pyi-wrapper-checklist.md, native-array-handle-checklist.md, documentation-content-checklist.md -status: active-roadmap -publication: draft ---- - -# Roadmap - -This repository-only roadmap tracks implementation and documentation work for -maintainers. Public support status remains in User documentation. - -## Planned Features - -- [Wrapper plan migration checklist](wrapper-plan-migration-checklist.md) -- [Semantic `.pyi` wrapper checklist](semantic-pyi-wrapper-checklist.md) -- [Native array handle checklist](native-array-handle-checklist.md) -- [Documentation content checklist](documentation-content-checklist.md) -- TODO: Populate from accepted roadmap issues and maintained checklists. - -## In-Progress Features - -- [Language-first test suite and Fortran pipeline cleanup](fortran-test-suite-cleanup-checklist.md) - -## Future Ideas - -- TODO: Separate exploratory ideas from committed plans. - -## Long-Term Vision - -- TODO: Describe the long-term documentation and wrapper ecosystem goals. - -## TODO - -- TODO: Add tracking issue links when public issue tracking is available. -- TODO: Keep language support status synchronized with the feature matrix. diff --git a/docs/maintainer/roadmap/native-array-handle-checklist.md b/docs/maintainer/roadmap/native-array-handle-checklist.md deleted file mode 100644 index c8bc7dd76..000000000 --- a/docs/maintainer/roadmap/native-array-handle-checklist.md +++ /dev/null @@ -1,1074 +0,0 @@ ---- -title: Native Array Handle Checklist -audience: maintainers -prerequisites: semantic .pyi format, ownership policy, allocatables, pointers -related: index.md, ../../user/reference/semantic-pyi-format.md, ../../user/guide/allocatables.md, ../../user/guide/pointers.md -status: active-roadmap -publication: draft ---- - -# Native Array Handle Checklist - -This is the implementation and verification checklist for native Fortran array -descriptor handles: - -- `Allocatable[T[...]]` -- `Pointer[T[...]]` - -Use this page as the canonical checklist for this feature. The original prompts -are consolidated here; future implementation should use this page instead of -re-reading those prompts. - -The intended implementation is one shared native-array handle path with -descriptor-specific operations layered on top. Keep Allocatable and Pointer as -separate public contract types, but do not build two unrelated parser, policy, -runtime, bridge, or binding stacks. - -Scalar allocatable and pointer procedure projections are not part of this array -handle work. Scalars continue to use ordinary nullable Python values plus -`@native_call` descriptor projections such as `Allocatable(Arg(i))`, -`Pointer(Arg(i))`, `Allocatable(Return(...))`, and `Pointer(Return(...))`. -For scalar descriptor inputs, explicit `None` means a present but unallocated -allocatable descriptor or a present but unassociated pointer descriptor. Native -optional scalar descriptor dummies use the three-state scalar bridge path: -omission means `present(...)` false, explicit `None` means a present descriptor -with absent value state, and a value means a present descriptor with scalar -storage. Array handles keep a different rule: `Allocatable[T[...]] | None` and -`Pointer[T[...]] | None` mean optional absent handle only. - -## Core Contract Decisions - -- [x] `Allocatable[T[...]]` means a Python handle to a native allocatable array - descriptor. -- [x] `Pointer[T[...]]` means a Python handle to a native pointer array - descriptor. -- [x] `Allocatable[T[...]]` and `Pointer[T[...]]` are handles, not NumPy arrays. -- [x] `T[...]` remains the ordinary array data-buffer contract. -- [x] Passing a handle to an `Allocatable[T[...]]` or `Pointer[T[...]]` - parameter uses descriptor semantics. -- [x] Passing a handle to a `T[...]` parameter uses normal array-actual - semantics in the shared runtime handoff path. For native wrapper calls, pass - the handle's native array actual to the normal Fortran array dummy instead of - implicitly calling `.to_numpy()`; generated wrapper parameter integration is - tracked separately below. -- [x] Normal `T[...]` parameters require a valid array actual: allocatable - handles must be allocated, pointer handles must be associated, and - allocated/associated zero-length arrays remain valid. -- [x] Unallocated allocatable handles and unassociated pointer handles are - accepted only by descriptor-handle parameters such as `Allocatable[T[...]]` - and `Pointer[T[...]]`, where that state belongs inside the handle. -- [x] Plain NumPy arrays are rejected by the shared runtime descriptor-parameter - handoff for `Allocatable[T[...]]` and `Pointer[T[...]]` descriptor - parameters; generated wrapper parameter integration is tracked separately - below. -- [x] `| None` on a handle means the handle object itself may be absent for a - native optional dummy, not that a present handle is unallocated or - unassociated. -- [x] Unallocated allocatable state lives inside the allocatable handle: - `h.allocated is False` and `h.to_numpy() is None`. -- [x] Unassociated pointer state lives inside the pointer handle: - `p.associated is False` and `p.to_numpy() is None`. -- [x] `Annotated[T[...], Allocatable]` is not an active public allocatable-array - spelling after migration. -- [x] `Annotated[T[...], Pointer]` is not an active public pointer-array - spelling after migration. -- [x] `Snapshot[T]` is not an allocatable- or pointer-array extraction mode and - is no longer an active public contract. -- [x] Live native-array views, explicit user-requested NumPy copies, live - derived objects, and descriptor handles remain distinct concepts in docs, - diagnostics, runtime names, and tests. - -## Public `.pyi` Examples - -### Allocatable Handles - -```python -from prik.contracts import Allocatable, Float64, Int32 - -values: Allocatable[Float64[:]] - -class box: - values: Allocatable[Float64[:]] - -def resize(values: Allocatable[Float64[:]], n: Int32) -> None: ... - -def make_values(n: Int32) -> Allocatable[Float64[:]]: ... - -def maybe_optional( - values: Allocatable[Float64[:]] | None = ..., -) -> None: ... - -def scale(values: Float64[:]) -> None: ... -``` - -`scale()` is ordinary array-data semantics. It may accept an allocated -`Allocatable[Float64[:]]` at runtime by passing the handle's native array actual -to the normal Fortran array dummy. It does not receive an allocatable dummy -descriptor. - -### Pointer Handles - -```python -from prik.contracts import Float64, Int32, Pointer - -values: Pointer[Float64[:]] - -class box: - values: Pointer[Float64[:]] - -def reassociate( - values: Pointer[Float64[:]], - target: Pointer[Float64[:]], -) -> None: ... - -def maybe_optional( - values: Pointer[Float64[:]] | None = ..., -) -> None: ... - -def scale(values: Float64[:]) -> None: ... -``` - -`scale()` is ordinary array-data semantics. It may accept an associated -`Pointer[Float64[:]]` at runtime by passing the handle's native array actual to -the normal Fortran array dummy. It does not receive a pointer dummy descriptor. - -## Call Compatibility Model - -For a normal array data signature: - -```python -from prik.contracts import Allocatable, Float64, Pointer - -def f(x: Float64[:]) -> None: ... - -plain: Float64[:] -allocatable: Allocatable[Float64[:]] -pointer: Pointer[Float64[:]] - -f(plain) -f(allocatable) -f(pointer) -``` - -all three calls are valid when the runtime value can provide usable -`Float64[:]` data: - -- `f(plain)` passes the NumPy/data-buffer value directly; -- `f(allocatable)` verifies that the handle is allocated and compatible, then - passes the wrapped native allocatable array actual to the normal Fortran array - dummy; -- `f(pointer)` verifies that the handle is associated and compatible, then - passes the wrapped native pointer array actual to the normal Fortran array - dummy. - -This mirrors Fortran argument association: a non-allocatable array dummy can be -called with ordinary, allocatable, or pointer actual arrays when the actual -array is present/associated and otherwise valid. At the Python boundary this is -not allocatable-or-pointer dummy descriptor semantics; the callee sees a normal -array dummy. It also is not an implicit `.to_numpy()` conversion. - -If the user writes `f(allocatable.to_numpy())` or `f(pointer.to_numpy())`, that -is the explicit public extraction path. The result is treated like any other -ordinary array argument and must pass the normal validation rules, including -non-`None` state and any mutability requirements. - -For descriptor signatures: - -```python -def g(x: Allocatable[Float64[:]]) -> None: ... - -def h(x: Pointer[Float64[:]]) -> None: ... -``` - -`g()` requires an allocatable handle and `h()` requires a pointer handle. A -plain NumPy array is rejected because it has no native allocatable or pointer -descriptor to pass. - -Internally, model this as a handle with an array-data facet plus descriptor -facts, not as a global rewrite of the base array type. For example, an -`Allocatable[Float64[:]]` value should expose or carry: - -- array data type: `Float64[:]`; -- descriptor kind: `allocatable`; -- descriptor operations: allocation state, descriptor passing, deallocate, and - resize. - -A `Pointer[Float64[:]]` value should expose or carry: - -- array data type: `Float64[:]`; -- descriptor kind: `pointer`; -- descriptor operations: association state, descriptor passing, nullify, and - any policy-gated allocation/deallocation operations. -- default handle mode: `Pointer[T[...]]` must be usable without - `PointerPolicy(...)` for conservative handle creation, association - inspection, descriptor handoff, `nullify()` where legal, and supported - extraction operations. -- explicit policy mode: `PointerPolicy(...)` enables or requests behavior that - needs otherwise unprovable facts, such as allocation, deallocation, - reassociation, target lifetime, or unsafe ownership transfer. - -The implementation may use predicates or metadata equivalent to -`is_allocatable` and `is_pointer` on the specific handle semantic type, but -plain `Float64[:]` itself remains the data-buffer contract. Do not make a -normal array parameter infer descriptor semantics merely because a handle is -accepted as an array-like runtime value. - -## Recommended Implementation Order - -### 1. Documentation And Contract Sync - -Update the public docs first so the intended behavior is explicit before code -changes. - -- [x] Update `docs/user/guide/allocatables.md`. -- [x] Update `docs/user/guide/pointers.md`. -- [x] Update `docs/user/reference/semantic-pyi-format.md`. -- [x] Update memory-management or language-support pages if they describe - allocatable or pointer arrays as NumPy arrays, `ndarray | None`, metadata - annotations, or `Snapshot[T]`. -- [x] Document that `Allocatable[T[...]]` is a handle, not an ndarray. -- [x] Document that `Pointer[T[...]]` is a handle to pointer association state, - not an ndarray. -- [x] Document that `h.to_numpy()` returns a live view of the current allocation - or `None`, never an automatic detached copy. -- [x] Document that `p.to_numpy()` returns the current target view or `None`, - and can expose strided pointer targets when descriptor support is available. -- [x] Document that passing a handle to a handle parameter is descriptor - passing, while passing a handle to `T[...]` uses normal array-actual - semantics through the handle's native array-data facet. -- [x] Document call compatibility for `def f(x: T[...])`: ordinary arrays, - allocated allocatable handles, and associated pointer handles are accepted - as array actuals without implicit `.to_numpy()` conversion. -- [x] Document that normal `T[...]` parameters reject unallocated allocatable - handles and unassociated pointer handles because there is no valid array - actual to pass. Allocated or associated zero-length arrays are still valid. -- [x] Document that explicit `f(h.to_numpy())` is a separate user-requested - ndarray path and follows ordinary ndarray validation. -- [x] Document that parameters annotated as `Allocatable[T[...]]` or - `Pointer[T[...]]` pass native descriptors, so they require the corresponding - handle object. Ordinary arrays are accepted only by normal `T[...]` - parameters. -- [x] Document that plain ndarray inputs are rejected for descriptor-handle - parameters. -- [x] Document that module allocatables and pointer arrays expose handles, not - `ndarray | None` module attributes. -- [x] Document that derived allocatable and pointer fields expose handles. -- [x] Document that allocatable function results can return owned handles only - when prik creates stable owner storage. -- [x] Document that pointer handles do not imply target ownership. -- [x] Document that pointer `nullify()` is default, while pointer - `allocate()`, `deallocate()`, and `resize()` require explicit policy. -- [x] Document stale-view hazards after descriptor-changing operations, - reassociation, nullification, deallocation, or reallocation. -- [x] Remove active public examples of `Annotated[T[...], Allocatable]` and - `Annotated[T[...], Pointer]` for this feature. - -### 2. Public Contract Symbols, Parser, And Printer - -Implement the public contract wrappers once and parameterize by descriptor kind. - -- [x] Add `Allocatable[...]` as a real array-handle contract wrapper. -- [x] Add `Pointer[...]` as a real array-handle contract wrapper. -- [x] Parse `Allocatable[T[...]]` into a semantic representation for a native - allocatable array handle. -- [x] Parse `Pointer[T[...]]` into a semantic representation for a native - pointer array handle. -- [x] Parse optional absent callable-argument handles as - `Allocatable[T[...]] | None = ...` and `Pointer[T[...]] | None = ...`. -- [x] Reject `Allocatable[T[...]] | None` and `Pointer[T[...]] | None` outside - optional callable arguments. -- [x] Reject optional/defaulted callable-argument handles that omit the - explicit `| None` spelling. -- [x] Preserve normal `T[...]` type identity as array data semantics, not - descriptor semantics. -- [x] Do not interpret `Snapshot[T]` as a native-array descriptor wrapper. -- [x] Remove the obsolete public `Snapshot` contract, its semantic `.pyi` - parsing/printing, generated contracts, and recursive derived-object lowering. -- [x] Reject or fully migrate `Annotated[T[...], Allocatable]` from active - public contracts. -- [x] Reject or fully migrate `Annotated[T[...], Pointer]` from active public - contracts. -- [x] Keep metadata-only allocatable or pointer facts only as temporary internal - migration facts, not as accepted public syntax. -- [x] Print generated module allocatables as `Allocatable[T[...]]`. -- [x] Print generated derived allocatable fields as `Allocatable[T[...]]`. -- [x] Print allocatable descriptor arguments and supported handle results as - `Allocatable[T[...]]`. -- [x] Print generated module pointer arrays as `Pointer[T[...]]`. -- [x] Print generated derived pointer fields as `Pointer[T[...]]`. -- [x] Print pointer descriptor arguments and supported handle results as - `Pointer[T[...]]`. - -### 3. Semantic IR Representation - -Create one semantic representation family for native array handles with a -descriptor-kind field rather than separate unrelated models. - -- [x] Represent common native-array handle facts: - - descriptor kind: allocatable or pointer; - - element semantic type; - - dtype; - - rank; - - shape metadata when statically known; - - array data type/facet used when a handle is passed to a normal `T[...]` - parameter; - - string element length metadata when applicable; - - optional-absent handle state; - - source origin: module variable, derived field, argument, or result; - - native access path. -- [x] Keep handle semantic types distinct from normal array semantic types. -- [x] Store allocatable/pointer descriptor facts on the handle semantic type, - not as a global mutation of the plain array type. -- [x] Permit shared predicates or metadata equivalent to `is_allocatable` and - `is_pointer` on handle semantic types when that helps policy dispatch. -- [x] Keep each handle's base array data type available for ordinary `T[...]` - call compatibility. -- [x] Keep scalar descriptor projection metadata on the existing scalar - nullable-value path, not on array handle types. -- [x] Preserve `T[...]` arguments/results as normal array data in semantic IR - even when runtime may later accept a handle through data coercion. -- [x] Verify `Snapshot[T]` is absent from active semantic IR and remains - unrelated to native-array-handle extraction. - -### 4. Post-IR Policy Completion - -Complete every semantic decision before wrapper planning. Bridge and binding -generators must dispatch from these decisions rather than inferring policy from -datatype, intent, origin, dotted access shape, alias metadata, or local memory -checks. - -The completed decision is recorded as `NativeArrayHandlePolicy` metadata on -each handle declaration or result type. That policy records the descriptor kind, -handle origin/kind, owner retention, borrowed-vs-owned descriptor status, -getter/setter behavior, native assignment behavior, release responsibility, -target lifetime, generated destroy behavior, storage mode, optional -absent-handle state, `.to_numpy()` extraction policy, and descriptor operation -permissions. It also records whether the selected path requires pointer C -descriptor interop or owned-allocatable CFI storage, so build integration can -gate `ISO_Fortran_binding.h` from completed policy instead of raw datatype -checks. Wrapper planning must fail before lowering a native array handle that -is missing this completed policy. - -For pointer handles, plain `Pointer[T[...]]` gets a default conservative -operation table for descriptor association, `nullify()`, and unavailable -extraction reporting. `PointerPolicy(...)` adds facts for behavior that needs -an explicit contract. `allocate(shape)` requires an explicit reassociation -value that allows allocation, `deallocate()` requires an explicit deallocation -value, and `resize(shape)` requires both sides to opt into resize. When an -explicit pointer policy selects descriptor-view extraction, the completed policy -records the `pointer_c_descriptor` interop requirement even if another planning -blocker still prevents wrapper lowering. - -- [x] Add one completed native-array-handle policy decision shared by - Allocatable and Pointer. -- [x] Complete descriptor kind before lowering: allocatable or pointer. -- [x] Complete handle kind before lowering: - - `borrowed_module_descriptor`; - - `borrowed_field_descriptor`; - - `argument_descriptor`; - - `owned_result_descriptor`; - - `optional_absent_handle`; - - `unsupported`. -- [x] Complete ownership and lifetime retention before lowering. -- [x] Complete whether the Python handle is borrowed or owned before lowering. -- [x] Complete getter behavior before lowering. -- [x] Complete Python setter exposure, if any, before lowering. -- [x] Complete native setter assignment behavior, if any, before lowering. -- [x] Complete output projection/readback behavior before lowering. -- [x] Complete release responsibility and generated destroy behavior before - lowering. -- [x] Complete `.to_numpy()` policy before lowering: - - `borrowed_view`; - - `descriptor_view`; - - `contiguous_view`; - - `unsupported`. -- [x] Complete standard C-descriptor interop requirement before lowering: - `none`, `module_allocatable_c_descriptor`, or `pointer_c_descriptor`. -- [x] Complete nullability and optional-absent-handle behavior before lowering. -- [x] Complete contract-value storage mode before lowering: `stack`, `heap`, or - `alias`. -- [x] Keep descriptor-argument and optional-absent-handle policies semantically - complete before lowering; generated bridge descriptor pass-through dispatches - from this completed policy. -- [x] Fail wrapper planning with a clear diagnostic when descriptor ownership, target - lifetime, shape, addressability, release responsibility, or extraction policy - is incomplete. - -#### Allocatable Policy Items - -- [x] Complete allocated-state support. -- [x] Complete live-view mechanism independently from `Aliased`: direct - borrowed access where legal, otherwise standard descriptor access. -- [x] Give plain and `Aliased` module allocatable handles the same native-owned - borrowed lifetime, mutability, and live-view-or-`None` public behavior. -- [x] Block unsupported descriptor extraction explicitly instead of copying. -- [x] Complete `deallocate()` permission. -- [x] Complete `resize(shape)` permission. -- [x] Complete function-result ownership as wrapper-owned stable descriptor - storage. -- [x] Mark unsupported allocatable array forms as wrapper-planning errors rather than - silently falling back to NumPy-array copy behavior. - -#### Pointer Policy Items - -- [x] Complete association-state support. -- [x] Complete target lifetime policy. -- [x] Complete `to_numpy()` extraction policy: - - descriptor view; - - contiguous view; - - unsupported. -- [x] Select pointer `to_numpy()` policy from completed `PointerPolicy(...)` - facts before lowering: contiguous targets use `contiguous_view`, and - strided/general targets use `descriptor_view`. A copy-oriented pointer policy - may retain unrelated meaning, but must not make extraction copy. -- [x] Complete `nullify()` permission as the default pointer descriptor - operation. -- [x] Complete a default conservative handle profile for plain - `Pointer[T[...]]` without requiring `PointerPolicy(...)`. -- [x] Complete `allocate(shape)` permission only when explicit pointer policy - allows allocation through this pointer. -- [x] Complete `deallocate()` permission only when explicit pointer policy - allows deallocation through this pointer. -- [x] Add an explicit unsafe/user-responsibility deallocation policy value, - `unsafe_deallocate`, for callers who knowingly request deallocation without - prik-proven target ownership. -- [x] Complete `resize(shape)` permission only when explicit pointer policy - allows resize through this pointer. -- [x] Do not expose pointer `allocate()`, `deallocate()`, or `resize()` when - policy disallows those operations. -- [x] Treat pointer handle ownership as descriptor/association access by - default, not target ownership. -- [x] For pointer results, support `owned_result_descriptor` only when stable - owner storage and target lifetime are explicit; otherwise stop wrapper planning. - -### 5. Shared Runtime Handle Foundation - -Add or reuse one internal runtime base for both public handle classes. - -- [x] Implement or reuse `NativeArrayHandleBase`. -- [x] Put shared dtype metadata on the base. -- [x] Put shared rank metadata on the base. -- [x] Put shared shape-query dispatch on the base. -- [x] Put shared `to_numpy()` dispatch on the base. -- [x] Put shared owner/lifetime retention on the base. -- [x] Put shared generated ops table/accessor storage on the base. -- [x] Validate generated operation table names and callables when a runtime - handle is constructed. -- [x] Require generated runtime handles to provide the shared `shape` operation - at construction time. -- [x] Require generated runtime handles to provide an internal `array_actual` - operation for normal `T[...]` native array-actual handoff, distinct from - explicit public `to_numpy()` extraction. -- [x] Require generated runtime handles to provide an internal `descriptor` - operation for `Allocatable[T[...]]` and `Pointer[T[...]]` descriptor-parameter - handoff. -- [x] Reject generated `array_actual` or `descriptor` handoff operations that - return `None`, so present handles cannot collapse into optional absent-handle - state. -- [x] Require generated `array_actual` operations to return the internal typed - native-array handoff object carrying a non-null native data address. -- [x] Require generated `descriptor` operations to return either decoded - standard descriptor fields or a contiguous native data address that the - shared runtime normalizes into those fields. An unallocated or unassociated - descriptor may carry a null `base_addr`; that state remains distinct from an - absent optional handle. -- [x] Put shared borrowed-vs-owned descriptor kind on the base. -- [x] Validate shared runtime descriptor kind at handle construction, so - generated handles can only use `allocatable` or `pointer` descriptor tags. -- [x] Put shared owned-handle release state and finalizer dispatch on the base, - so generated owner-storage handles can call a generated `destroy` operation - exactly once. -- [x] Require generated owned-handle operation tables to provide a callable - `destroy` operation at construction time. -- [x] Run owned-handle destroy operations before marking the handle closed, so - generated destroy accessors can still read descriptor or owner state. -- [x] Mark owned handles closed after a generated destroy attempt even when - destroy reports an error, so finalizers cannot retry the same native release. -- [x] Leave room for optional generation or stale-view tracking later. -- [x] Add an internal runtime array-actual validation and handoff hook for - future normal `T[...]` handle inputs, without calling `to_numpy()`. -- [x] Add an internal runtime normal-array argument dispatcher that keeps - ordinary ndarray validation and native-handle array-actual handoff as - separate paths while sharing dtype, rank, shape, layout, and writeability - checks. -- [x] Add an internal runtime normal-array argument ABI packer that returns the - generated Bind-C array tuple fields from either an ndarray data pointer or a - native handle `array_actual` handoff: pointer address, optional runtime rank, - optional item size, extents, and optional upper bounds plus unit strides. -- [x] Normalize runtime layout validation for handle array-actual handoff with - the same supported `C` and `F` layout expectations used by the ndarray path. -- [x] Normalize runtime handle shapes as non-negative extents, rejecting - negative dimensions while preserving zero-length arrays as valid array - actuals. -- [x] Let the internal runtime normal-array argument dispatcher enforce native - byte order and alignment when generated binding policy requests those checks. -- [x] Add an internal runtime descriptor-parameter validation and handoff hook - for future `Allocatable[T[...]]` and `Pointer[T[...]]` binding inputs, - including optional absent-handle `None` mapping. -- [x] Validate descriptor-parameter handoff kind before mapping optional - absent-handle `None`, so unsupported descriptor kinds cannot silently pass. -- [x] Add an internal runtime descriptor-argument field packer that returns - validated `base_addr`, `elem_len`, `rank`, and per-dimension lower-bound, - extent, and stride-multiplier facts. Generated C uses those facts to establish - call-local `CFI_CDESC_T(rank)` storage for non-projected descriptor calls; - Python never supplies compiler-private descriptor storage. -- [x] Add a distinct direct standard-C-descriptor handoff for projected writable - handles. Owned allocatable handles pass their persistent `CFI_cdesc_t*` so - native allocation, deallocation, and shape changes update the same caller - handle instead of a discarded call-local descriptor copy. -- [x] Use a dedicated non-null runtime presence token for present optional - handle arguments, rather than reusing the descriptor handoff object as the - presence field. -- [x] Pack optional descriptor arguments with the same validated descriptor - facts plus a distinct presence token. Optional absent handles produce null - fact fields and a null presence token; present unallocated or unassociated - handles produce present descriptor facts whose `base_addr` may be null. -- [x] Carry the completed `.to_numpy()` extraction policy on the runtime - handle. -- [x] Remove detached-copy dispatch from the runtime handle; extraction-enabled - operations must supply live storage or standard descriptor facts. -- [x] Validate generated `.to_numpy()` operations return either a NumPy array or - `None` before applying borrowed-view, descriptor-view, or contiguous-view - policy. -- [x] Validate non-`None` `.to_numpy()` results against the handle's declared - dtype and rank before returning them to Python. -- [x] Short-circuit absent descriptor state before generated extraction, so an - unallocated allocatable handle or unassociated pointer handle returns `None` - from `to_numpy()` without relying on backend extraction code. -- [x] Reject generated `.to_numpy()` results that report `None` after the - handle has reported present descriptor state, so backend extraction cannot - collapse allocated or associated handles into absent state. -- [x] Require generated runtime handles with an extraction-enabled - `to_numpy_policy` to provide the generated `to_numpy` operation at - construction time; handles without extraction support must use - `to_numpy_policy="unsupported"`. -- [x] Enforce live contiguous-view and descriptor-view `.to_numpy()` policies - in the shared runtime handle without a copy fallback. -- [x] Implement `AllocatableArray` as a descriptor-specific subclass. -- [x] Require allocatable runtime handles to provide the generated `allocated` - operation at construction time. -- [x] Implement `PointerArray` as a descriptor-specific subclass. -- [x] Require pointer runtime handles to provide generated `associated` and - default `nullify` operations at construction time. - -Runtime handle support means the shared Python class enforces the completed -policy once generated operations provide access to current native storage. -Bridge generation for module variables, fields, arguments, results, and C -descriptor extraction remains tracked in the codegen and integration sections -below. - -#### Allocatable Runtime API - -- [x] `h.allocated -> bool` -- [x] `h.shape -> tuple[int, ...] | None` -- [x] `h.to_numpy() -> ndarray | None` -- [x] `h.deallocate()` -- [x] `h.resize(shape)` -- [x] `h.to_numpy()` returns `None` when unallocated. -- [x] `h.to_numpy()` returns a live mutable view for every supported allocated - handle, using direct or descriptor access as selected by completed policy. -- [x] Users can call `.copy()` on the returned NumPy array when they need - independent lifetime. -- [x] Existing views may become stale after descriptor-changing operations; - accessing stale views is unsupported and may crash. - -#### Pointer Runtime API - -- [x] `p.associated -> bool` -- [x] `p.shape -> tuple[int, ...] | None` -- [x] `p.to_numpy() -> ndarray | None` -- [x] `p.nullify()` -- [x] Optional, policy-gated `p.allocate(shape)` -- [x] Optional, policy-gated `p.deallocate()` -- [x] Optional, policy-gated `p.resize(shape)` -- [x] `p.to_numpy()` returns `None` when unassociated. -- [x] `p.to_numpy()` returns a live borrowed NumPy view when associated and - descriptor extraction is supported. -- [x] `p.to_numpy()` supports strided views when descriptor support is - available. -- [x] `p.to_numpy()` raises a clear error when descriptor extraction is - unavailable and no fallback is supported. -- [x] Old borrowed views are documented as stale after native code nullifies, - reassociates, deallocates, or otherwise changes the pointer target. - -### 6. IR-to-AST And Codegen Model - -Lower only completed policy decisions into named implementation methods. - -- [x] Use one completed array interop policy object for array-like bridge and - binding decisions. The policy selects a named ABI lane: - `data_buffer` for ordinary `T[...]` NumPy/data-pointer semantics, or - `descriptor` for `Allocatable[T[...]]` and `Pointer[T[...]]` descriptor - semantics. -- [x] Keep the generated implementation methods separate under that dispatcher: - the data-buffer lane emits the existing pointer/shape/stride ABI for normal - arrays, while the descriptor lane emits descriptor-handle ABI operations and - any gated TS 29113 reader code. -- [x] Add codegen model nodes or metadata for native-array handle creation. -- [x] Add codegen model nodes or metadata for generated native-array handle ops. -- [x] Route Allocatable and Pointer handles through the same lowering path with - descriptor-kind-specific operations. -- [x] Keep `@native_call Addr(Arg(i))` as data-address projection only. -- [x] Add native-array-handle bridge/binding dispatchers keyed by completed - descriptor kind and handle kind. -- [x] Route native-array module-variable bridge generation through completed - handle policy before ordinary module-variable array dispatch. -- [x] Route native-array derived-field bridge and binding generation through - completed handle policy before ordinary field array dispatch. -- [x] Route native-array function-result bridge and binding generation through - completed handle policy before ordinary array result dispatch. -- [x] Select descriptor passing from `Allocatable[T[...]]` or - `Pointer[T[...]]` plus completed policy, not from `Addr`. -- [x] Lower unsupported policy decisions to planning/codegen errors, not - fallback behavior. - -### 7. Bridge Generation - -Generate descriptor-access routines through the shared handle-ops shape, then -specialize operation bodies by descriptor kind. - -- [x] Block generated descriptor-handle accessors with explicit native-array - codegen blockers until the descriptor-access routines below exist. -- [x] Add the shared Bind-C/binding/runtime construction substrate for - generated handle objects: explicit operation-name maps, module-owner - retention, runtime factory creation, and pointer-address handoff wrapping. -- [x] Generate module allocatable handles as borrowed descriptor handles. -- [x] Create module allocatable handle objects at module initialization. -- [x] Store complete generated operation pointers/accessors for module - allocatable variables, including portable descriptor handoff and generated - `resize(shape)` operations. The operation table covers state, shape, - array-actual and descriptor-fact handoff, `.to_numpy()`, `deallocate()`, and - `resize(shape)` without exposing a compiler-private descriptor layout. -- [x] Do not move ownership out of the Fortran module for ordinary module - allocatable attribute reads. -- [x] Generate derived-field allocatable handles as borrowed descriptor handles. -- [x] Keep the parent wrapper object alive for derived-field allocatable - handles. -- [x] Generate field operations that access `parent%field`. -- [x] Generate owned allocatable function-result handles when policy supports - stable owner storage. -- [x] Use wrapper-owned standard C descriptor storage for allocatable results: - allocate persistent rank-specific `CFI_CDESC_T(rank)` storage and establish - it with allocatable attribute. Numeric function results populate a local - allocatable once, then transfer that allocation with `move_alloc`; generated - shape-changing operations use `CFI_allocate`. -- [x] Assign a supported numeric direct allocatable function result once into a - bridge-local allocatable, then `move_alloc` that allocation into the - allocatable `intent(out)` dummy backed by persistent CFI storage. Do not - generate a collector or a second intrinsic assignment. Rank-one, matrix, and - higher-rank results preserve allocated, zero-sized, and unallocated state. -- [x] Return a native pointer to owner storage for owned allocatable handles. -- [x] Generate destroy routines called by the Python handle finalizer for owned - allocatable handles. -- [x] Generate module pointer handles as borrowed descriptor handles. -- [x] Create module pointer handle objects at module initialization. -- [x] Store complete generated operation pointers/accessors for module pointer - variables, including portable descriptor handoff and policy-gated - `allocate(shape)`, `deallocate()`, and `resize(shape)` operations. The current - generated module operation table covers association state, shape, - pointer-address handoff wrappers, `nullify()`, and the policy-gated - shape-changing operations when completed policy enables them. Descriptor - handoff uses standard C descriptor facts rather than guessing a compiler - descriptor layout. -- [x] Do not transfer ownership of pointer targets for module pointer handles. -- [x] Generate derived-field pointer handles as borrowed descriptor handles. -- [x] Keep the parent wrapper object alive for derived-field pointer handles. -- [x] Generate field operations that access `parent%field`. -- [x] Route pointer descriptor-argument bridge and binding generation through - completed handle policy before ordinary array argument dispatch. -- [x] Model native descriptor-handle argument handoff as a dedicated Bind-C - descriptor tuple selected by bridge and binding descriptor-argument handlers - through completed output-projection policy. Non-projected calls establish - standard call-local descriptor storage from validated runtime facts. - Projected writable calls pass persistent standard C descriptor storage - directly so descriptor mutation remains attached to the caller handle. Both - paths add an explicit presence token only for optional absent handles. -- [x] Generate pointer descriptor-argument handoff for `Pointer[T[...]]` - parameters. -- [x] Route allocatable descriptor-argument bridge and binding generation - through completed handle policy before ordinary array argument dispatch. -- [x] Generate allocatable descriptor-argument handoff for - `Allocatable[T[...]]` parameters. -- [x] Do not guess compiler-specific descriptor layouts. Descriptor-based - interop must use the TS 29113 / Fortran 2018 C descriptor path or fail - wrapper planning with an explicit diagnostic. - -#### Pointer Descriptor Extraction - -This path is feature-gated. It may use TS 29113 / Fortran 2018 C descriptors -only when descriptor-view interop is selected, and it must not add a global -`ISO_Fortran_binding.h` requirement to wrappers that do not need descriptor -decoding. - -- [x] Use TS 29113 / Fortran 2018 C descriptors for general pointer-array - `to_numpy()` when this path is enabled. -- [x] Use `ISO_Fortran_binding.h` only for descriptor-based pointer interop - paths. -- [x] Do not require `ISO_Fortran_binding.h` globally. -- [x] In the shared runtime helper, build NumPy shape from decoded descriptor - `dim[i].extent` fields supplied by generated descriptor-interoperability - code. -- [x] In the shared runtime helper, build NumPy strides from decoded descriptor - `dim[i].sm` fields supplied by generated descriptor-interoperability code. -- [x] Let pointer `descriptor_view` extraction operations return decoded - descriptor fields as mappings or field-record objects, with the shared - runtime converting those fields into the NumPy view instead of requiring - every generated operation to call the helper. -- [x] Validate decoded pointer descriptor rank against the handle's declared - rank before constructing the NumPy view. -- [x] Reject decoded pointer descriptors with null `base_addr` after the handle - has reported associated state. -- [x] Support positive and negative descriptor stride multipliers in the shared - runtime descriptor-view helper by computing the full buffer window before - constructing the NumPy view. -- [x] In the shared runtime helper, read and validate decoded descriptor - `base_addr`, `elem_len`, `rank`, `dim[i].lower_bound`, `dim[i].extent`, and - `dim[i].sm` fields for pointer views. -- [x] Add a shared generated C/CPython descriptor-reader primitive that decodes - a `CFI_cdesc_t*` into the runtime descriptor-view mapping shape, without - exposing TS 29113 layout details in public Python APIs. -- [x] Generate code that reads TS 29113 descriptor `base_addr`, `elem_len`, - `rank`, `dim[i].lower_bound`, `dim[i].extent`, and `dim[i].sm` for pointer - descriptor-view operations in private generated CPython operation wrappers. -- [x] Support strided pointer targets, including negative strides, in the shared - runtime once generated descriptor-interoperability code supplies decoded - descriptor fields. -- [x] If descriptor support is unavailable, choose one explicit policy: - contiguous-only pointer views when shape/address are safely available, - explicitly implemented copy fallback, or a wrapper-planning failure with a - clear diagnostic. There is no deferred blocker payload. - -### 8. Python Binding Generation - -Keep descriptor-handle argument conversion separate from normal array data -conversion. For normal `T[...]` parameters, support both ordinary ndarray inputs -and native handle inputs, but do not implement handle inputs by implicitly -calling `.to_numpy()`. The handle path should route to a native array-actual -handoff when the wrapped call is native. - -- [x] Accept `AllocatableArray` objects for `Allocatable[T[...]]` parameters. -- [x] Reject plain NumPy arrays for `Allocatable[T[...]]` parameters. -- [x] Accept `PointerArray` objects for `Pointer[T[...]]` parameters. -- [x] Reject plain NumPy arrays for `Pointer[T[...]]` parameters. -- [x] Accept `None` for optional-absent handle parameters only when the `.pyi` - annotation includes `| None`. -- [x] Convert `None` optional handles into native absent optional dummies, not - into unallocated or unassociated handle objects. - The CPython binding layer now packs required and optional descriptor-handle - arguments through the runtime descriptor-argument helper, and bridge - generation passes descriptor dummies through the Bind-C descriptor tuple. -- [x] For normal `T[...]` parameters, accept ndarray inputs through the existing - array data path. -- [x] For concrete-rank numeric normal `T[...]` parameters in generated Bind-C - wrapper calls, accept allocated allocatable handles only by validating the - handle state and passing the wrapped native allocatable array actual to the - normal native array dummy. -- [x] For concrete-rank numeric normal `T[...]` parameters in generated Bind-C - wrapper calls, accept associated pointer handles only by validating the - handle state and passing the wrapped native pointer array actual to the normal - native array dummy. -- [x] Share dtype, rank, shape, layout, and mutability validation policy between - ndarray inputs and handle inputs for concrete-rank numeric `T[...]`, while - keeping the existing ndarray pointer/shape handoff and native-handle - array-actual handoff as separate generated implementation branches. -- [x] Treat explicit `h.to_numpy()` or `p.to_numpy()` results that are NumPy - arrays as ordinary ndarray input through the existing array-storage path. -- [x] Reject read-only arrays returned by explicit `h.to_numpy()` or - `p.to_numpy()` when the native dummy requires writable storage, reusing the - existing writable ndarray validation. -- [x] Add direct wrapper coverage showing explicit `h.to_numpy()` or - `p.to_numpy()` returning `None` is rejected as an ordinary ndarray argument - for non-nullable `T[...]` dummies. -- [x] Reject unallocated allocatable handles for concrete-rank numeric `T[...]` - unless nullable data-buffer behavior is explicitly implemented. -- [x] Reject unassociated pointer handles for concrete-rank numeric `T[...]` - unless nullable data-buffer behavior is explicitly implemented. -- [x] Reject unallocated allocatable handles for optional, assumed-rank, and - character `T[...]` unless nullable - data-buffer behavior is explicitly implemented. -- [x] Reject unassociated pointer handles for optional, assumed-rank, and - character `T[...]` unless nullable - data-buffer behavior is explicitly implemented. - -### 9. Compilation And Build Gating - -- [x] Require descriptor interop support only when a generated wrapper uses the - pointer C-descriptor path or persistent CFI owner storage for an allocatable - result. -- [x] Do not require `ISO_Fortran_binding.h` for allocatable-only builds that - contain no owned allocatable result handles. -- [x] Require `ISO_Fortran_binding.h` locally when owned allocatable result - handles use persistent `CFI_CDESC_T` storage. -- [x] Do not require `ISO_Fortran_binding.h` for pointer builds that do not use - descriptor-based pointer interop. -- [x] Collect native-array build requirements from completed handle policy - metadata, not from raw `Allocatable[...]` or `Pointer[...]` syntax. -- [x] Record native-array build requirements in replayable `.pyi` wrapper build - manifests. -- [x] Emit a clear planning or build diagnostic when pointer descriptor interop - is required but unavailable. - -## Test Checklist - -### Parser And Printer Tests - -- [x] Parse and print `Allocatable[Float64[:]]`. -- [x] Parse and print `Allocatable[String[:][:]]`. -- [x] Parse and print `Allocatable[Float64[:, :]]`. -- [x] Parse and print `Allocatable[Float64[:]] | None = ...`. -- [x] Reject `Allocatable[Float64[:]] | None` on non-argument declarations. -- [x] Parse and print `Pointer[Float64[:]]`. -- [x] Parse and print `Pointer[Float64[:, :]]`. -- [x] Parse and print `Pointer[String[8][:]]` if string arrays are supported. -- [x] Parse and print `Pointer[Float64[:]] | None = ...`. -- [x] Reject `Pointer[Float64[:]] | None` on non-argument declarations. -- [x] Verify normal `T[...]` type identity remains array data semantics even - when runtime can accept handles by data coercion. -- [x] Verify `Allocatable[T[...]]` and `Pointer[T[...]]` retain a base array - data type that matches the wrapped `T[...]` annotation. -- [x] Reject `Snapshot[T]` in active contracts. -- [x] Reject or migrate `Annotated[T[...], Allocatable]`. -- [x] Reject or migrate `Annotated[T[...], Pointer]`. -- [x] Verify no generated `.pyi` uses `Snapshot[T]`. -- [x] Verify no generated active `.pyi` uses public - `Annotated[T[...], Allocatable]` or `Annotated[T[...], Pointer]` for array - descriptor handles. - -### Semantic IR And Policy Tests - -- [x] Verify Allocatable and Pointer handles use the same semantic handle - representation family with distinct descriptor kinds. -- [x] Verify handle types are distinct from normal array data types. -- [x] Verify handle semantic types carry descriptor facts without mutating the - plain array semantic type. -- [x] Verify handle semantic types expose the base array data type used for - `T[...]` call compatibility. -- [x] Verify module-variable, derived-field, argument, result, and optional - absent handle origins complete policy before lowering. -- [x] Verify native array handle policy carries owner retention, target - lifetime, and generated destroy behavior before lowering. -- [x] Verify bridge/binding layers dispatch from completed policy decisions. -- [x] Verify incomplete ownership, lifetime, release, addressability, or - descriptor-extraction facts produce wrapper-planning errors. -- [x] Verify descriptor-handle arguments stop wrapper planning until generated handle - handoff exists instead of falling back to NumPy-array conversion. -- [x] Verify plain `Pointer[T[...]]` generates the default conservative handle - profile without requiring `PointerPolicy(...)`. -- [x] Verify missing owner/release facts block only ownership-changing pointer - operations, not association inspection, handle passing, or other safe default - handle operations. -- [x] Verify `Addr(Arg(i))` rejects `Allocatable[T[...]]` and - `Pointer[T[...]]` descriptor handles instead of acting as descriptor passing. -- [x] Verify pointer allocation/deallocation/resize permissions are absent or - blocked unless explicit pointer policy allows them. -- [x] Verify unsafe/user-responsibility deallocation is available only through - the explicit policy value and never by default. -- [x] Verify completed `PointerPolicy(...)` facts select `contiguous_view` or - `descriptor_view` before lowering, never an extraction-only copy action, and - only descriptor-view paths request pointer C-descriptor interop. -- [x] Verify pointer C-descriptor interop requirements produce an explicit - wrapper-planning error while that interop path is unavailable. - -### Shared Runtime Handle Tests - -- [x] Verify `AllocatableArray` and `PointerArray` use the same common - `to_numpy()`, `shape`, dtype, rank, owner-retention, and ops-table path. -- [x] Verify common shape metadata is reported consistently. -- [x] Verify common dtype metadata is reported consistently. -- [x] Verify handles keep required module, parent object, or owner storage alive. -- [x] Verify invalid generated operation tables fail at handle construction - before descriptor state or native handoff is queried. -- [x] Verify handles without the shared generated `shape`, `array_actual`, or - `descriptor` operations fail at construction. -- [x] Verify generated `array_actual` and `descriptor` handoff operations cannot - return `None`; optional absent handles are the only runtime path that maps to - absent descriptor fact fields. -- [x] Verify generated array-actual operations must return the internal typed - native-array handoff object with a non-null pointer address, rejecting - booleans, non-integers, zero addresses, negative addresses, and arbitrary - Python objects before generated bridge handoff. -- [x] Verify descriptor operations accept validated decoded standard descriptor - fields or a contiguous data address, preserve null `base_addr` as present - unallocated/unassociated state, and reject malformed field records. -- [x] Verify shared runtime handles reject unsupported descriptor-kind tags - before any generated operations are used. -- [x] Verify owned handles call generated destroy ops exactly once when closed - or finalized, and borrowed handles do not destroy native owner storage. -- [x] Verify owned handles without generated `destroy` operations fail at - construction instead of leaking through a suppressed finalizer error. -- [x] Verify owned-handle destroy operations can inspect live handle state - before the handle is marked closed. -- [x] Verify owned handles are marked closed after a failing destroy attempt, - preventing finalizer retries of the same generated release operation. -- [x] Verify generated `.to_numpy()` operations cannot return non-NumPy objects - from borrowed-view or contiguous-view policies, and cannot return non-NumPy - objects from descriptor-view policy unless the value is a decoded pointer - descriptor field mapping or field-record object. -- [x] Verify generated `.to_numpy()` arrays and decoded pointer descriptor - views must match the handle's declared dtype and rank. -- [x] Verify `to_numpy()` returns `None` for unallocated allocatable handles and - unassociated pointer handles before generated extraction or unsupported-policy - errors are reached. -- [x] Verify generated extraction cannot return `None`, or a decoded pointer - descriptor with null `base_addr`, after the handle has reported present - descriptor state. -- [x] Verify extraction-enabled handles without generated `to_numpy` fail at - construction, while unsupported extraction raises the completed-policy error. -- [x] Verify contiguous-view policy rejects non-contiguous arrays and never - copies; descriptor-view policy preserves validated shape and strides. -- [x] Verify the internal runtime array-actual hook rejects absent descriptor - state and uses generated handoff ops instead of `to_numpy()`. -- [x] Verify the internal runtime array-actual hook validates expected dtype, - rank, shape, layout, and writeability before generated handoff. -- [x] Verify handle array-actual layout validation normalizes `C` and `F` - expectations consistently with ndarray validation and rejects unsupported - layout names before generated handoff. -- [x] Verify the internal runtime normal-array argument dispatcher accepts - ordinary ndarrays through an ndarray path, accepts allocated/associated - handles through native array-actual handoff, and rejects unallocated or - unassociated handles without calling `to_numpy()`. -- [x] Verify the internal runtime normal-array argument ABI packer emits the - generated Bind-C array tuple shape for ndarray inputs and for - allocated/associated handle inputs without calling `to_numpy()`. -- [x] Verify the internal runtime normal-array argument dispatcher preserves - zero-length array actuals and rejects handle shapes with negative extents. -- [x] Verify the internal runtime normal-array argument dispatcher rejects - byte-swapped or unaligned ndarray inputs when generated binding policy - requests native byte order or alignment. -- [x] Verify the internal runtime descriptor-parameter hook accepts only the - matching handle class/kind, rejects ordinary arrays, validates expected dtype, - rank, and shape, and maps optional `None` to an absent native handle. -- [x] Verify optional absent-handle `None` still rejects unsupported descriptor - kinds before returning the native absent-handle sentinel. -- [x] Verify the internal runtime descriptor-argument field packer returns - `base_addr`, `elem_len`, `rank`, and each dimension's lower bound, extent, and - stride multiplier; maps optional absent-handle `None` to null fact fields; - and uses a distinct non-null presence token for present optional handles. -- [x] Verify projected writable handle arguments require a typed direct - standard-descriptor handoff and reject fact-only descriptors before native - mutation can detach the caller's handle state. -- [x] Verify CPython binding generation packs required and optional - descriptor-handle arguments through the runtime helper, dispatches - non-projected calls to standard call-local CFI storage, dispatches projected - writable calls to persistent descriptor storage, and passes the selected - descriptor pointer through the completed Bind-C tuple shape. -- [x] Verify allocatable handles without generated `allocated` fail at - construction. -- [x] Verify pointer handles without generated `associated` or `nullify` fail - at construction. - -### Allocatable Runtime Tests - -- [x] Module allocatable attribute is a handle object. -- [x] `h.allocated` updates after allocate, deallocate, and resize. -- [x] `h.shape` updates after allocate, deallocate, and resize. -- [x] `h.to_numpy()` returns `None` when unallocated. -- [x] Plain and `Aliased` allocated module handles both return mutable live - views, and mutating either view updates native module storage. -- [x] A fresh extraction follows allocation, deallocation, resize, and - reallocation state; an explicit `.copy()` remains independent. -- [x] Tests state the stale-view contract without dereferencing deliberately - stale storage. -- [x] Derived allocatable field is a handle object. -- [x] Derived-field handle keeps the parent wrapper alive. -- [x] Derived-field `deallocate()` operates on `parent%field`. -- [x] Derived-field `resize(shape)` operates on `parent%field`. -- [x] Allocatable function result returns an owned handle. -- [x] Owned result handle finalizer deallocates native owner storage. -- [x] Owned result handle `to_numpy()` works after the bridge returns. -- [x] `Allocatable[T[...]]` parameter accepts allocatable handles. -- [x] `Allocatable[T[...]]` parameter rejects plain ndarray. -- [x] `T[...]` parameter accepts ndarray. -- [x] Concrete-rank numeric `T[...]` parameter accepts allocated allocatable handle through native - array-actual handoff, without implicitly calling `h.to_numpy()`. -- [x] Concrete-rank numeric `T[...]` parameter applies the same dtype, rank, shape, layout, and - mutability validation policy to ndarray inputs and allocated allocatable - handles. -- [x] Explicit `T[...]` calls with `h.to_numpy()` follow the ordinary ndarray - path and reject `None` or read-only arrays when writable storage is required. -- [x] Concrete-rank numeric `T[...]` parameter rejects unallocated allocatable handle unless nullable - data-buffer behavior is explicitly supported. - -### Pointer Runtime Tests - -- [x] Module pointer attribute is a handle object. -- [x] `p.associated` reflects association state. -- [x] `p.shape` reflects association state and target shape. -- [x] `p.to_numpy()` returns `None` when unassociated. -- [x] `p.to_numpy()` returns a borrowed view when associated and supported. -- [x] `p.nullify()` disassociates the native pointer descriptor. -- [x] Derived pointer field is a handle object. -- [x] Derived-field pointer handle keeps the parent wrapper alive. -- [x] Derived-field pointer operations access `parent%field`. -- [x] Pointer associated with a slice returns a NumPy view with expected shape - and strides when C descriptors are available. -- [x] Runtime pointer handles raise a clear unavailable-operation error when - no descriptor-extraction `to_numpy()` operation is generated. -- [x] Runtime pointer descriptor-view extraction validates required decoded - TS29113 fields before constructing a NumPy view. -- [x] If C descriptors are unavailable, test the selected explicit behavior: - contiguous live view or clear wrapper-planning diagnostic, never a copy fallback. -- [x] Pointer `deallocate()` and `resize()` are absent or raise when policy - disallows them. -- [x] Pointer `allocate()`, `deallocate()`, and `resize()` work only when - explicit pointer policy allows them. -- [x] `Pointer[T[...]]` parameter accepts pointer handles. -- [x] `Pointer[T[...]]` parameter rejects plain ndarray. -- [x] Concrete-rank numeric `T[...]` parameter accepts an associated pointer - handle through native array-actual handoff when the target is contiguous, - without implicitly calling `p.to_numpy()`. -- [x] Concrete-rank numeric `T[...]` parameter applies the same dtype, rank, shape, layout, and - mutability validation policy to ndarray inputs and associated pointer - handles. -- [x] Reject noncontiguous pointer targets from the pointer/shape array-actual - handoff instead of silently treating their elements as contiguous. -- [x] Explicit `T[...]` calls with `p.to_numpy()` follow the ordinary ndarray - path and reject `None` or read-only arrays when writable storage is required. -- [x] Concrete-rank numeric `T[...]` parameter rejects unassociated pointer handle unless nullable - data-buffer behavior is explicitly supported. - -### Build Gating Tests - -- [x] Pointer descriptor interop includes or requires `ISO_Fortran_binding.h` - only when descriptor-based pointer interop is used. -- [x] Borrowed/argument-only allocatable builds do not require - `ISO_Fortran_binding.h`; owned allocatable result builds include it for their - persistent CFI storage path. -- [x] Builds that need unavailable pointer descriptor interop fail with a clear - diagnostic rather than guessing descriptor layout. - -### Documentation Regression Tests - -- [x] Public docs show `Allocatable[T[...]]` for allocatable array handles. -- [x] Public docs show `Pointer[T[...]]` for pointer array handles. -- [x] Public docs do not show `Annotated[T[...], Allocatable]` as the active - public spelling. -- [x] Public docs do not show `Annotated[T[...], Pointer]` as the active public - spelling. -- [x] Public docs do not show `Snapshot[T]` as an active array descriptor - contract. -- [x] Public docs explain `to_numpy()` as the explicit user-facing extraction - operation for both handle types, not the required internal implementation of - handle-to-native calls. - -## Completion Criteria - -The feature is complete only when all of these are true: - -- [x] Allocatable and Pointer array handles share one internal handle - foundation. -- [x] Public contract syntax is `Allocatable[T[...]]` and `Pointer[T[...]]`. -- [x] Active parser/printer paths reject or remove the old annotation and - snapshot forms. -- [x] Post-IR policy completes every handle decision before wrapper planning. -- [x] Bridge and binding generation dispatch from completed policy only. -- [x] Runtime handles expose the documented APIs and state transitions. -- [x] Pointer descriptor extraction never guesses compiler-specific descriptor - layout. -- [x] Build gating keeps descriptor interop requirements local to the paths that - need them. -- [x] Runtime tests cover module variables, derived fields, arguments, data - coercion, owned allocatable results, pointer association, pointer nullify, and - pointer policy gating. -- [x] Documentation and generated `.pyi` fixtures no longer present old public - forms as active contracts. diff --git a/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md b/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md deleted file mode 100644 index fefb44ae3..000000000 --- a/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md +++ /dev/null @@ -1,4869 +0,0 @@ ---- -title: Wrapper Plan Migration Checklist -audience: maintainers -prerequisites: pipeline map, semantic IR, ownership policy -related: ../internal-architecture/pipeline-map.md, ../../user/reference/semantic-ir.md, semantic-pyi-wrapper-checklist.md, index.md -status: active-roadmap -publication: draft ---- - -# Wrapper Plan Migration Checklist - -This file is the canonical implementation contract for wrapper-plan migration. -It replaces the generic semantic-IR wrapper lowering route one eligible module -at a time. The migration changes representation and generation organization; it -does not intentionally change the established Python, native ABI, ownership, or -build behavior of a migrated lane. - -## Canonical Pipeline - -```text -Semantic IR - -> post-IR policy completion - -> WrapperPlanner.build(module) - -> editable ModulePlan - -> WrapperCodeGenerator.generate(plan) - -> freeze and validate the received plan - -> recursively synthesize C binding nodes - -> recursively synthesize Fortran bridge nodes - -> print backend nodes - -> RenderedGeneratedWrapperArtifacts - -> existing build/link orchestration -``` - -There is no public or wrapper-domain representation between `ModulePlan` and -backend syntax nodes. `CModule`, `CHeader`, `CFunction`, `FortranModule`, and -`FortranFunction` are direct printer inputs, not another wrapper planning -stage. - -The public generation boundary is deliberately small: - -```python -complete_semantic_policies(module) -plan = WrapperPlanner().build(module) -artifacts = WrapperCodeGenerator().generate(plan) -``` - -`WrapperCodeGenerator.generate` accepts `ModulePlan` only. It does not accept -semantic modules, build a plan itself, select an alternate lowering route, or -retry a prior route after direct generation begins. - -Semantic `.pyi` generation remains outside this route. A semantic `.pyi` -contract can supply the semantic module consumed by planning, but planning does -not change `.pyi` emission. - -## One Shared Plan, Explicit Backend Views - -`ModulePlan` is one shared semantic-and-ABI contract. It is not a C plan joined -to a Fortran plan and it does not contain backend nodes or source text. - -Every owner that crosses or coordinates the boundary has binding and bridge -child plans in the same editable tree: - -```text -ModulePlan - binding: BindingModulePlan - bridge: BridgeModulePlan - functions: FunctionPlan ... - binding: BindingFunctionPlan - bridge: BridgeFunctionPlan - arguments: ArgumentTransferPlan ... - binding: BindingArgumentPlan - bridge: BridgeArgumentPlan - native_call_slot: NativeCallSlotPlan - transformations: TransformationPlan ... - results: ResultPlan ... - binding: BindingResultPlan - bridge: BridgeResultPlan - native_call_slot: NativeCallSlotPlan | None - transformations: TransformationPlan ... - lifecycle: LifecycleActionPlan ... - binding: BindingLifecyclePlan | None - bridge: BridgeLifecyclePlan | None - native_call_slots: ordered references to argument/result slots plus - function-owned literal or helper slots -``` - -`ArgumentTransferPlan` remains the only argument-owner record; do not add a -generic duplicate `ArgumentPlan`. Its backend-facing child plans are -deliberately distinct and directly editable: - -- `binding: BindingArgumentPlan` describes the Python input, its C conversion - action, and - the C handoff value/role it produces; -- `bridge: BridgeArgumentPlan` describes the C ABI slot, value-versus-address - convention, - native action, and Fortran value that the bridge consumes; -- `native_call_slot` records the exact native-call position and source; -- an argument or hidden result's `native_call_slot` is the same mutable record - referenced from `FunctionPlan.native_call_slots`, not a copied record that a - maintainer must edit twice; -- result and lifecycle records identify later producers, consumers, ordering, - and responsibility through their own binding and bridge views; -- native slots and lifecycle actions are subordinate transfer details, not - parallel datatype-policy systems. They remain indexed on `FunctionPlan` - because native ABI order and success/failure lifecycle order may span more - than one argument or result. A function-owned literal, status helper, or - other ABI slot may also have no single argument/result owner. - -The action vocabulary keeps source placement, data transfer, and native ABI -transport orthogonal. `ResultPlan.source_kind` says whether a result comes from -a `direct_return` or `hidden_output`; `CodegenAction` says how the value moves -or is owned (`DIRECT_VALUE`, `COPY_OUT`, `WRAPPER_INSTANCE`, and so on). A -hidden scalar therefore remains `DIRECT_VALUE`, while hidden strings and -ordinary arrays are `COPY_OUT`; hidden descriptor-owned objects use their -completed ownership action. `HIDDEN_OUTPUT` is not a codegen action because -hiddenness is a source location, not a transfer operation. - -Likewise, `NativeBarrierAction.PASS_ARRAY_BUFFER` means the Phase 6 data-buffer -ABI whose handoff plan carries data, rank, extents, strides, and itemsize. -`NativeBarrierAction.PASS_NATIVE_DESCRIPTOR` is reserved for the persistent -native descriptors and handles introduced in Phase 7. Neither backend may use -one action as a fallback for the other. - -`NativeBarrierAction.PASS_RAW_ADDRESS` is the third, deliberately narrower -array transport: one caller-supplied opaque address plus separately completed -pointee rank, shape, element type, and orientation facts. It does not authorize -NumPy extraction, a packed array-buffer ABI, or a native descriptor. Scalar, -fixed-string, and array raw addresses reuse this action and -`ArgumentHandoffMode.OPAQUE_ADDRESS`; their object kind then selects the named -bridge association method. Do not add datatype-specific raw-address actions. - -The binding input and bridge input may have different representations: a C -binding commonly receives `PyObject *`, produces a C scalar or address, and -the bridge then consumes a value or pointer according to its ABI slot. The plan -therefore records the producer/consumer handoff contract explicitly; it does -not assume identical types or actions. This keeps both backend plans in one -coherent editable tree while preventing them from drifting into disconnected -top-level plans. `CBindingGenerator` reads only binding views plus shared -handoff/order facts needed to create C nodes; `FortranBridgeGenerator` reads -only bridge views plus shared native-call order needed to create Fortran nodes. -Neither backend output is an input to the other backend. - -An owner may have only one active backend side when completed policy places the -behavior entirely in one backend, but that ownership must be explicit in the -plan rather than inferred during lowering. - -### Transformation Layer Ownership - -Every representation transformation has one explicit -`TransformationPlan.layer`: `BINDING` or `BRIDGE`. The record also names its -phase (`COPY_IN`, `NATIVE_MUTATION`, `COPY_OUT`, or `CLEANUP`), typed action, -source representation, target representation, and reason. These records are -subordinate to the `ArgumentTransferPlan` or `ResultPlan` that owns the value; -they are not a parallel datatype-policy hierarchy. - -Use the binding layer for transformations involving Python objects or NumPy -semantics: dtype/layout conversion, Python encoding/decoding, reference and -identity handling, copy-back into caller objects, and Python-owned temporary -cleanup. Use the bridge layer for transformations wholly between the ABI and a -native-language representation: Fortran character representation, native -descriptor/result materialization, derived native layout, or native-only -allocation and copy. - -One logical conversion and its inverse/cleanup must stay at one layer. If a -workflow genuinely needs both layers, policy completion records two distinct -transformations separated by a named intermediate ABI representation. A -backend consumes only transformations assigned to it and fails validation if -asked to lower an action owned by the other backend. Method location, datatype, -`intent`, and available local storage never select the transformation layer. -For `COPY_F`, copy-in, conditional copy-out, and cleanup are all binding-owned; -the bridge has no `COPY_F` transformation and reuses its ordinary ORDER_F -association path. - -### Editable Signature And Native Intent Boundary - -The semantic `.pyi` signature is authoritative for the Python-facing call -shape. The source converter may use native `intent` to propose the initial -generated signature, but that proposal is not backend policy. A user may -reorder visible Python arguments, keep a native output dummy as caller-supplied -storage, project it into a Python result, or introduce hidden bridge storage. -After the semantic contract is constructed or edited, completed native-call -slots must account for every required native position exactly once; stored -source `intent` must not silently override that mapping by hiding, exposing, -reordering, allocating, or projecting a Python value. - -Bridge dummies and backend-local variables use the most permissive declaration -that is compatible with the selected ABI, normally no `intent` or an internal -`intent(inout)`-equivalent writable local. The called native procedure enforces -its actual `intent(in)`, `intent(out)`, or `intent(inout)` contract. Required -interoperability attributes such as `value`, optional presence fields, standard -descriptor attributes, and true bridge-output parameters remain explicit ABI -facts; they are not permission for a backend to reconstruct the user-facing -signature from native `intent`. - -The plan includes every fact required for mechanical lowering: - -- owner path and plan-node kind; -- typed Python, native, and result actions; -- semantic datatype, datatype family, and precision/type facts; -- Python/native handoffs and bridge ABI slots; -- native-call slots in their exact order; -- result and output projection; -- ownership, transfer, destruction, mutability, writeback, release - responsibility, storage mode, nullability, and lifecycle ordering; and -- every typed lowering choice required by a supported backend. - -It contains decisions and facts, never generator method names. In particular, -there are no handler-name fields, handler records, or plan-owned handler -registries. The planner uses the completed policy actions already represented -by `PythonBarrierAction`, `NativeBarrierAction`, and `CodegenAction`; a new -typed action is added only when none of those can identify a necessary -mechanical behavior. Free-form string actions are forbidden. - -## Ownership Boundary - -Post-IR policy completion decides object kind, ownership, transfer, -destruction, mutability, writeback, nullability, output projection, release -responsibility, contract-value storage (`stack`, `heap`, or `alias`), getter -behavior, native setter assignment, Python setter exposure, ABI order, and -lifecycle order before planning starts. - -`WrapperPlanner` only projects those completed decisions and datatype facts -into readable editable records. It may traverse owners, preserve declared -order, assign stable owner paths, and wire already-decided producers to -consumers. It must not derive or replace policy from a datatype, `intent`, -decorator spelling, `is_alias`, dotted owner shape, local memory observation, -or a missing field. - -No code after planning may infer, replace, or override semantic policy. Backend -contexts may allocate temporary names and create declarations, error paths, -reference-count operations, and local cleanup statements after selecting a -typed lowering case. Those are emitted-code mechanics, not plan policy. - -`WrapperPlanner.build(module)` returns an editable `ModulePlan`. Maintainers -may edit its ordinary fields to inspect an experiment before generation. A -permanent behavior change belongs in the semantic contract and completed policy -rather than in a backend exception. - -## Generator-Owned Freezing and Validation - -Every consumer freezes the exact object it receives: - -```text -editable ModulePlan -- WrapperCodeGenerator --> frozen ModulePlan -editable backend modules -- source printers --> frozen backend modules -editable generated artifacts -- build integration --> frozen artifacts -``` - -At the start of `WrapperCodeGenerator.generate(plan)`, the generator must: - -1. recursively freeze that exact plan object; -2. run the complete binding/bridge plan-consistency validation on the final - edited plan; -3. ask each backend to preflight only its own implementation capability; and -4. recursively lower the validated plan into backend nodes before the printers - consume those nodes. - -Later mutation of the received plan raises `FrozenStageRecordError`. Backend -nodes remain editable until their printer consumes them. Generated artifacts -remain editable until `_build_rendered_wrapper_extension(...)` consumes them. - -`WrapperPlanner` does not validate its output. It mechanically projects an -editable plan, which may temporarily be inconsistent while a maintainer edits -it. `WrapperCodeGenerator` owns the private structured validation methods and -is the only validation consumer. There is no standalone validator class or -public validation operation. - -`WrapperCodeGenerator._validate_plan()` is the single plan-consistency gate. -It validates the complete binding/bridge graph after the editable plan has -been frozen and before either backend preflight or visitor runs. The gate stays -small by composing `_plan_diagnostics()` from typed private diagnostics for -namespaces, functions, arguments, results, lifecycle actions, and module -variable getter/setter action families. A cross-view invariant belongs to the -diagnostic for the lowest plan node that contains both views; for example, a -module-variable diagnostic validates Python setter exposure against its native -assignment and bridge setter role. - -`CBindingGenerator.require_supported()` and -`FortranBridgeGenerator.require_supported()` are later backend-local -capability preflights. They may reject a completed action, primitive type, -descriptor kind, or ABI combination that their own backend cannot implement, -but they do not establish whether binding and bridge views agree. Backend -visitors and `_lower_*` methods mechanically consume the decisions owned by -their view. Their exhaustive unmatched-action errors remain defensive -protection for direct backend use; the public generation path must report a -cross-view inconsistency from `_validate_plan()` first. No generator infers -consistency by reading the other backend's plan view. - -Structural validation preserves these invariants: - -- module getter actions and roles agree, and Python setter exposure agrees with - native assignment, bridge setter roles, descriptor kinds, and constant state; -- binding producer and bridge consumer roles agree; -- bridge ABI coverage, positions, and owner roles are complete; -- native-call slot coverage, exact ordering, hidden literals, and hidden - results agree, including hidden-result native and codegen actions; -- direct and hidden result producer/consumer roles agree; -- writeback, cleanup, and release actions use available source roles in their - declared order, and advertised roles exactly match their plan producers; -- positions and symbolic roles are neither duplicate nor missing; and -- external and bind-target requirements are complete. - -## Direct Recursive Lowering - -`WrapperCodeGenerator` owns two private backend visitors: - -```python -c_module, c_header = CBindingGenerator().visit(plan) -fortran_module = FortranBridgeGenerator().visit(plan) -``` - -They are private implementation organization inside direct generation, not -public stages. Both visitors recursively traverse the same plan tree and -return actual C or Fortran nodes (or tuples of actual nodes where a child needs -multiple declarations or statements). - -The recursive shape is: - -```text -ModulePlan - -> binding and bridge module contexts and backend nodes - -> NamespacePlan - -> directly owned FunctionPlan and ModuleVariablePlan records - -> binding/bridge argument transfers, result projection, lifecycle actions - -> complete backend function and namespace nodes - -> complete backend module node -``` - -An argument visitor returns the C or Fortran declarations/statements/parameters -needed for that backend. A result visitor returns the backend result nodes. -Lifecycle visitors return backend writeback, cleanup, or release nodes. Parent -visitors assemble these concrete child results directly into complete syntax -nodes. Do not introduce another wrapper-specific transport model. - -The public orchestration stays visibly direct: - -```python -class WrapperCodeGenerator: - def generate(self, plan: ModulePlan) -> RenderedGeneratedWrapperArtifacts: - plan.freeze() - self._validate_plan(plan) - self._c_generator.require_supported(plan) - self._fortran_generator.require_supported(plan) - - c_module, c_header = self._c_generator.visit(plan) - fortran_module = self._fortran_generator.visit(plan) - - c_source = self._c_printer.doprint(c_module) - c_header_source = self._c_printer.doprint(c_header) - fortran_source = self._fortran_printer.doprint(fortran_module) - return self._rendered_artifacts( - plan.owner_path, - c_source, - c_header_source, - fortran_source, - ) -``` - -The generator constructs `RenderedGeneratedWrapperArtifacts` directly from the -printed source plus artifact metadata. It does not duplicate native build plans, -compiler selection, link ordering, native-support installation, or compilation -policy; those remain in existing build/link orchestration. - -## Direct Lowering Methods - -Each backend visitor dispatches plan nodes by class through -`_visit_`. A visitor method then calls a typed -`_lower_` helper for each completed action family owned by that -backend. The helper uses an explicit, exhaustive action match and calls one -concrete `_lower__` implementation method. For example: - -```python -def _visit_ModuleVariablePlan(self, plan): - return ( - *self._lower_module_getter(plan), - *self._lower_module_setter(plan), - ) - -def _lower_module_getter(self, plan): - match plan.binding.getter_action: - case ModuleGetterAction.CONSTANT_VALUE: - return self._lower_module_getter_constant_value(plan) - case ModuleGetterAction.DIRECT_VALUE: - return self._lower_module_getter_direct_value(plan) - case ModuleGetterAction.NULLABLE_SNAPSHOT: - return self._lower_module_getter_nullable_snapshot(plan) - raise ValueError(...) -``` - -The C binding dispatches only from binding-owned actions, and the Fortran -bridge dispatches only from bridge-owned actions. In particular, native module -setter generation consumes the completed bridge assignment action rather than -the Python setter-exposure action. Post-IR policy completion records -`AssignmentMode.NONE` when no native setter is exposed and -`AssignmentMode.VALUE_COPY` for supported scalar value write-through; bridge -lowering does not reconstruct that choice from the Python setter action. -Backend support checks retain genuine ABI and capability validation; action -dispatch itself raises explicitly for every unsupported value, including an -unsupported alias assignment. - -Do not synthesize implementation method names, use `getattr` to execute -lowering, retain a fallback behavior, or store dispatcher names in the plan. -Do not create extra getter or setter plan nodes solely to gain more -`_visit_` methods. Both visitor and lowering methods return backend -syntax nodes; printers remain the only layer that renders those nodes as source -text. - -Primitive dtype spelling and converter differences live in the intentionally -scalar-specific `PrimitiveScalarTypeRegistry`; they do not duplicate control -flow methods or select semantic policy. - -Within policy, planning, support analysis, validation, and both backend -visitors, family-specific helpers stay in visibly labeled contiguous groups: -scalar helpers, string helpers, and ordinary-array helpers. Put a short section -comment above every such group so maintainers can find one datatype family -without scanning interleaved lowering methods. Generic orchestration remains -outside those groups and dispatches into them through the completed typed -actions. - -## Migration and Route Rules - -The legacy route remains the behavioral oracle until a lane has direct-plan -parity. Route selection is atomic per merged extension: a generation unit uses -either the direct wrapper-plan route or the legacy route. It never combines one -backend from one route with the other backend from the other route. - -The documented public contract is authoritative when it intentionally corrects -legacy behavior. In that case, use the legacy implementation to simplify the -mechanical ABI, conversion, ownership, and cleanup audit, improve the design -where the legacy path is unsafe or unnecessarily complex, and record every -intentional behavioral difference in focused tests. Do not preserve a known -legacy defect merely to obtain byte-for-byte or semantic parity. - -An unsupported owner may select the legacy route before planning. Once the plan -route is selected, planning, validation, lowering, printing, or compilation -failure fails the build; it must not fall back to legacy generation. - -Support reports and rollout gates keep scalar, string, and ordinary-array -input, optional, writeback, direct-result, and hidden-result lanes distinct. -Evidence for one datatype family must not make another family production -eligible accidentally. - -For each lane: - -1. replay an existing passing `tests/wrapper` case through the legacy route and - retain its generated artifacts; -2. record the relevant legacy source paths, ABI/call order, ownership and - cleanup behavior, artifact requirements, and runtime assertions; -3. complete every missing semantic decision before planning; -4. add the smallest required plan record and directly named lowering method; -5. produce the same complete artifact set through the direct route; -6. inspect differences, compile both routes, and run the existing assertions; -7. update checklist evidence only after direct-route parity is proven. - -Generated source is diagnostic evidence, not a byte-for-byte golden. Backend -temporary names and equivalent control flow may differ, but ABI, conversion, -ownership, cleanup, call order, and artifact requirements must remain proven. - -During this migration the full real-library BLAS/LAPACK wrapper corpus is -excluded locally and in CI until final cutover. General native-bundle coverage -remains active. - -## Staged Walkthrough - -`tools/wrapper_plan_staged_walkthrough.py` is the maintained hand-inspection -path. It shows only the source/contract entry, policy completion, plan creation, -a direct edit, direct generation, artifact inspection, build, and runtime use: - -```python -module = ... -complete_semantic_policies(module) - -plan = WrapperPlanner().build(module) -namespace = next(item for item in plan.namespaces if item.python_path == ()) -function = namespace.functions[0] -function.bridge.native_name = "SUB_R8" - -binding = CBindingGenerator() -bridge = FortranBridgeGenerator() -print(function.arguments[0].binding.optional_mode) -print(function.arguments[0].bridge.optional_mode) - -artifacts = WrapperCodeGenerator( - c_generator=binding, - fortran_generator=bridge, -).generate(plan) - -# inspect generated files -# build and run -``` - -It does not expose standalone validation. Printed plan inspection uses the -actual namespace, owner, and completed action records. The backend visitors -make the corresponding explicit action matches visible in their typed lowering -helpers. - -## Required Evidence - -Focused tests must prove: - -- `WrapperPlanner.build(module)` returns a directly mutable plan; -- direct edits to binding and bridge views change the relevant generated C and - Fortran source; -- `WrapperCodeGenerator.generate(plan)` freezes the exact consumed plan; -- module visitors recursively include generated function nodes; -- function visitors recursively include argument, result, and lifecycle nodes; -- directly named backend lowering methods cover every supported plan action; -- unsupported combinations fail explicitly; -- source printers freeze backend module nodes; -- generated artifacts remain editable until build consumption, which freezes - them; -- source and semantic-`.pyi` entries preserve compiled runtime parity; and -- backend lowering does not reconstruct semantic policy. - -Use package-export inspection and focused migration checks to prove removal of -obsolete internal representations; do not preserve tests whose only assertion -is that a removed API is absent. - -## Recovered Roadmap Scope - -The detailed migration queue below is retained from the original roadmap. The -obsolete Phase 0-2 emitter/fragment architecture is replaced by the simplified -direct-plan checklist later in this file; all later semantic lanes, matrix rows, -verification gates, and completion records remain explicit. - -## Existing Wrapper Suite As The Migration Queue - -`tests/wrapper` is the behavioral source and final acceptance suite for this -migration. Migrate its existing generation units one by one; do not create a -parallel wrapper suite or new native source fixtures merely to make the new -route easier to exercise. - -- Phase 0A adds a maintained migration matrix to this file covering every - Python test node under `tests/wrapper`. Each row records whether the test - generates a wrapper, the source/contract generation unit it uses, its - relevant feature lanes, and one status: - `not-applicable`, `deferred-real-library`, `legacy`, `dual-route`, or - `wrapper-plan`. -- Existing source files, contract fixtures, build helpers, runtime assertions, - failure assertions, and ABI assertions are reused as written whenever they - already cover the migrated behavior. Do not copy their behavior into a new - test with a smaller invented source. -- A new native source or contract fixture is allowed only when the audit proves - that accepted production behavior has no existing test. Record that coverage - gap and its owning semantic lane here before adding the fixture; migration - convenience is not sufficient justification. -- Whole-generation-unit routing still applies. An existing test moves to - `dual-route` only when every runtime-required feature in its module is - supported. If a nominally scalar fixture also contains results, strings, - arrays, decorators, module state, or classes, leave it on the legacy route - until those lanes are complete rather than carving out a narrower fixture. -- Dual-route parity reuses the same existing fixture and assertion function for - legacy and wrapper-plan builds through internal test orchestration. Do not - add a public route flag, duplicate the behavioral assertions, or require - byte-identical generated source. -- Once parity passes and production eligibility is widened, that existing test - moves to `wrapper-plan`. Keep deliberate legacy execution only in the - migration parity harness until final cutover. -- The final target is not merely that `tests/wrapper` passes. Every test in the - suite must be represented in the migration matrix, and every test that - generates a runtime wrapper must use the wrapper-plan route after cutover. - Tests that only inspect documentation, layout, parsing, or `.pyi` generation - may be `not-applicable` but must still pass. -- During active migration, ordinary pytest invocations exclude the full BLAS - and LAPACK example projects. Their dedicated lane owns library-scale - verification. General native-bundle tests remain active because they test - linker/build mechanics independently of the full corpora. - -### Wrapper Test Migration Matrix - -Matrix rows use pytest selector patterns. A row ending in `::*` covers every -collected test node in that Python file when all nodes share the same -generation classification. A row ending in `[*]` covers the parametrized nodes -for that test function. The structural layout test expands these selectors -against live `python3 -m pytest --collect-only -q tests/wrapper` output, so a -new wrapper test node must either match an existing row intentionally or add a -new row here before later implementation starts. - -Statuses have the meanings defined above: `legacy` still uses the current -`semantic_ir_to_codegen_ast()` route, `dual-route` runs the same generation -unit and runtime assertions through both implementations, `wrapper-plan` uses -only `WrapperPlan -> WrapperCodeGenerator`, `not-applicable` does not generate -a runtime wrapper, and `deferred-real-library` is reserved for the full BLAS -and LAPACK corpus until Phase 12. - -#### Current Wrapper Route Counts - -These are collected pytest-node counts, not matrix-row counts. The structural -layout test derives them from live `tests/wrapper` collection and fails if this -summary, the exhaustive matrix, and the test tree disagree. - -| Status | Collected nodes | -| --- | ---: | -| `wrapper-plan` | 370 | -| `dual-route` | 0 | -| `legacy` | 0 | -| `not-applicable` | 76 | -| `deferred-real-library` | 0 | - -#### Recorded Route Progression - -This history keeps phase movement visible instead of replacing the previous -snapshot with only the latest totals. Phase 2D moved all 17 dual-route nodes -and 44 legacy nodes to production plan routing, then added two parametrized -plan-route nodes. Phase 2E adds two scalar-only parity nodes, and Phase 2F adds -one isolated direct-return plus hidden-output scalar aggregation node. The -original mixed integration nodes retain their real array/string/object -blockers. - -| Proven checkpoint | `wrapper-plan` | `dual-route` | `legacy` | `not-applicable` | `deferred-real-library` | Total | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | -| Before Phase 2D | 0 | 17 | 178 | 95 | 2 | 292 | -| Phase 2D complete | 63 | 0 | 134 | 95 | 2 | 294 | -| Phase 2E scalar isolation | 65 | 0 | 134 | 95 | 2 | 296 | -| Phase 2F scalar result aggregation | 66 | 0 | 134 | 95 | 2 | 297 | -| Phase 5A required string values | 67 | 0 | 134 | 95 | 2 | 298 | -| Phase 5B fixed string results | 69 | 0 | 134 | 95 | 2 | 300 | -| Phase 5C fixed string writeback | 70 | 0 | 134 | 95 | 2 | 301 | -| Phase 5C assumed/optional string writeback | 71 | 0 | 134 | 95 | 2 | 302 | -| Phase 5D fixed string storage/raw addresses | 72 | 0 | 134 | 95 | 2 | 303 | -| Phase 5 production route reconciliation | 76 | 0 | 130 | 95 | 2 | 303 | -| Phase 6 ordinary arrays | 78 | 5 | 130 | 95 | 2 | 310 | -| Phase 6G raw array addresses | 80 | 5 | 130 | 95 | 2 | 312 | -| Phase 6 `COPY_F` representation copy | 81 | 5 | 130 | 95 | 2 | 313 | -| Phase 7 native handles/descriptors | 88 | 5 | 129 | 96 | 2 | 320 | -| Phase 7 production route reconciliation | 94 | 5 | 123 | 95 | 2 | 319 | -| Phase 8 scalar-derived object lifetimes | 106 | 5 | 123 | 95 | 2 | 331 | -| Phase 8 complete scalar-derived actual/dummy matrix | 213 | 5 | 123 | 95 | 2 | 438 | -| Phase 8H failure, qualified-type, and typed-value closure | 222 | 5 | 123 | 95 | 2 | 447 | -| Phase 11 cross-cutting suite completion | 344 | 0 | 0 | 95 | 2 | 441 | -| Phase 12 canonical cutover | 346 | 0 | 0 | 95 | 0 | 441 | - -Migration is complete only when `legacy`, `dual-route`, and -`deferred-real-library` are all zero. At that point every runtime-generating -node must be `wrapper-plan`; `not-applicable` may remain only for tests that do -not generate a wrapper. Until then, moving a node from `legacy` to `dual-route` -records proven parity, and moving it from `dual-route` to `wrapper-plan` -records final removal of its legacy execution. - -#### Complete Route Ledger - -For a `legacy` row, the feature-lane column identifies what still blocks the -new route. For `dual-route` and `wrapper-plan` rows, it identifies the behavior -already covered by the new generator. - -| Pytest selector | Generation unit | Feature lanes / blockers | Status | -| --- | --- | --- | --- | -| `tests/wrapper/fortran/arrays/test_array_contracts.py::*` | source/generated-.pyi parity or parametrized route | ordinary arrays | `wrapper-plan` | -| `tests/wrapper/fortran/arrays/test_array_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | -| `tests/wrapper/fortran/arrays/test_array_results.py::test_array_results_follow_data_buffer_and_descriptor_handle_contracts[*]` | production plan route in source/generated-.pyi parity modes | fixed/runtime-shape ordinary array results; owned allocatable descriptor results; namespace preservation | `wrapper-plan` | -| `tests/wrapper/fortran/arrays/test_array_results.py::test_maybe_unallocated_allocatable_result_preserves_absent_state` | edited semantic `.pyi` contract over the existing array-result native unit | `MaybeUnallocated` direct allocatable vector/matrix result annotations preserve allocated and unallocated result states without changing default always-allocated result handling | `wrapper-plan` | -| `tests/wrapper/fortran/arrays/test_array_results.py::test_ordinary_array_results_use_canonical_plan` | canonical production output-only plan route | fixed/runtime-shape ordinary array results; ranks one through fifteen; Fortran order; zero-sized results; allocation/copy/release failure paths | `wrapper-plan` | -| `tests/wrapper/fortran/arrays/test_array_results.py::test_owned_allocatable_results_preserve_handle_state` | canonical reduced owned-result contract | allocated and zero-sized wrapper-owned `CFI_CDESC_T` function-result handles; extraction and release | `wrapper-plan` | -| `tests/wrapper/fortran/arrays/test_assumed_rank_arrays.py::test_assumed_rank_arguments_dispatch_to_runtime_rank[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `wrapper-plan` | -| `tests/wrapper/fortran/arrays/test_assumed_rank_arrays.py::test_assumed_rank_bridge_dispatches_each_runtime_rank_argument[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `wrapper-plan` | -| `tests/wrapper/fortran/arrays/test_assumed_rank_arrays.py::test_assumed_rank_arrays_use_explicit_plan_branches` | reduced semantic `.pyi` entry over the existing assumed-rank native unit | runtime ranks one through fifteen; mutable storage; rank validation; native-handle actuals deferred to Phase 7 | `wrapper-plan` | -| `tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::test_rank2_contiguous_contract_requires_fortran_contiguous[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `wrapper-plan` | -| `tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::test_rank2_assumed_shape_accepts_fortran_ordered_strided_views[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `wrapper-plan` | -| `tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::test_rank2_assumed_shape_rejects_non_positive_strides[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `wrapper-plan` | -| `tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::test_rank2_explicit_shape_requires_fortran_contiguous[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `wrapper-plan` | -| `tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::test_rank3_contiguous_contract_requires_fortran_contiguous[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `wrapper-plan` | -| `tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::test_rank3_assumed_shape_accepts_fortran_ordered_strided_views[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `wrapper-plan` | -| `tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::test_dense_strided_and_projected_arrays_use_canonical_plan` | reduced semantic `.pyi` entry over the existing multidimensional native unit | dense/explicit extents; positive-strided views; zero-sized axes; projected output identity; native-handle actuals deferred to Phase 7 | `wrapper-plan` | -| `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_init_entry_uses_resolved_parent_name_from_inside_package` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | -| `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_output_name_override_replaces_entry_inference` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | -| `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_recursive_graph_preserves_module_and_symbol_aliases_and_ignores_support_imports` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | -| `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_recursive_graph_reports_cycles_before_codegen` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | -| `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_recursive_graph_reports_missing_relative_contract_before_native_validation` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | -| `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_source_build_preserves_modules_and_root_externals` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `wrapper-plan` | -| `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_complete_general_source_preserves_namespaces_through_canonical_plan[*]` | canonical production plan route | Python namespace hierarchy; native import aliases; scalar inputs/results; void calls | `wrapper-plan` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_entry_can_alias_one_module_procedure_at_the_root` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_entry_rejects_colliding_wildcard_exports` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_entry_wildcard_import_explicitly_flattens_module_leaf` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `wrapper-plan` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_generated_pyi_fixture_builds_from_native_object_without_source_reparse` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_generated_pyi_matches_checked_in_fixture` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_mixed_entry_exposes_externals_at_root_and_modules_as_children` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `wrapper-plan` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_module_leaf_can_be_the_entry_contract` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `wrapper-plan` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_module_variable_runtime_contract[*]` | source/generated-.pyi parity or parametrized route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `wrapper-plan` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_mutable_module_variable_default_initializes_native_storage` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `wrapper-plan` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_one_entry_preserves_multiple_native_module_namespaces` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_cli_accepts_exactly_one_entry_contract` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_cli_preserves_explicit_ordered_link_items` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_cli_requires_a_native_link_input` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_makefile_manifest_and_replay_workflows` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_manifest_records_pointer_descriptor_interop_requirements` | non-generating: manifest serialization unit | completed native-array build requirements and local standard-descriptor headers | `not-applicable` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_python_api_accepts_exactly_one_entry_contract` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_python_api_rejects_a_missing_native_artifact` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_python_api_rejects_invalid_address_contracts_before_codegen[*]` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_python_api_rejects_invalid_projection_before_codegen` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_python_api_rejects_python_suffix_as_semantic_contract` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_reduced_entry_generates_only_reachable_module_variable_bindings` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `wrapper-plan` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_scale_runtime_contract[*]` | source/generated-.pyi parity or parametrized route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_source_named_root_discovers_and_builds_module_leaf` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `wrapper-plan` | -| `tests/fortran/building_shared_library/end_to_end/test_source_build_modes.py::test_documented_readme_points_example_builds_and_imports` | direct wrapper/build route | build/compile/link orchestration; module namespace and derived-type inputs/results | `wrapper-plan` | -| `tests/fortran/building_shared_library/end_to_end/test_source_build_modes.py::test_fortran_wrapper_default_module_name_does_not_collide_with_root_function` | direct wrapper/build route | build/compile/link orchestration | `wrapper-plan` | -| `tests/fortran/building_shared_library/end_to_end/test_source_build_modes.py::test_fortran_wrapper_default_places_artifacts_in_invocation_directory` | direct wrapper/build route | build/compile/link orchestration | `wrapper-plan` | -| `tests/fortran/building_shared_library/end_to_end/test_source_build_modes.py::test_fortran_wrapper_out_dir_separates_abi_artifact_from_cli_alias` | direct wrapper/build route | build/compile/link orchestration | `wrapper-plan` | -| `tests/fortran/building_shared_library/end_to_end/test_source_build_modes.py::test_fortran_wrapper_out_names_importable_shared_library` | direct wrapper/build route | build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | -| `tests/fortran/building_shared_library/end_to_end/test_source_build_modes.py::test_generate_sources_cli_writes_wrapper_sources_without_native_outputs` | source-only wrapper generation route | build integration through the completed wrapper plan without compile/link execution | `wrapper-plan` | -| `tests/fortran/building_shared_library/end_to_end/test_source_build_modes.py::test_internal_preprocessing_mode_still_builds_importable_runtime_wrapper` | direct wrapper/build route | build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | -| `tests/fortran/building_shared_library/end_to_end/test_source_build_modes.py::test_native_link_plan_serializes_interleaved_item_kinds` | direct wrapper/build route | build/compile/link orchestration | `wrapper-plan` | -| `tests/fortran/building_shared_library/end_to_end/test_source_build_modes.py::test_source_build_reuses_native_plan_for_additional_compile_and_link_inputs` | source-only wrapper generation route | shared source/contract native build plan; implementation compiler flags; supplemental sources; objects; libraries; include and link directories; ordered link items | `wrapper-plan` | -| `tests/fortran/building_shared_library/end_to_end/test_source_build_modes.py::test_source_build_result_records_structured_native_plan` | direct wrapper/build route | build/compile/link orchestration | `wrapper-plan` | -| `tests/fortran/building_shared_library/end_to_end/test_source_build_modes.py::test_verbose_mode_prints_custom_wrapper_flags` | direct wrapper/build route | build/compile/link orchestration | `wrapper-plan` | -| `tests/fortran/building_shared_library/end_to_end/test_source_build_modes.py::test_verbose_mode_prints_full_direct_build_commands` | direct wrapper/build route | build/compile/link orchestration | `wrapper-plan` | -| `tests/fortran/building_shared_library/end_to_end/test_source_build_modes.py::test_wrapper_build_rejects_empty_source_list` | non-generating: validation/failure-path assertion | build/compile/link orchestration | `not-applicable` | -| `tests/fortran/building_shared_library/end_to_end/test_source_build_modes.py::test_wrapper_build_rejects_generation_verbose_combination[*]` | non-generating: validation/failure-path assertion | build/compile/link orchestration | `not-applicable` | -| `tests/fortran/building_shared_library/end_to_end/test_source_build_modes.py::test_wrapper_build_rejects_missing_source` | non-generating: validation/failure-path assertion | build/compile/link orchestration | `not-applicable` | -| `tests/wrapper/fortran/build_from_source/test_compiler_verbose.py::*` | direct wrapper/build route | build/compile/link orchestration | `wrapper-plan` | -| `tests/fortran/building_shared_library/end_to_end/test_runtime_compatibility.py::*` | direct wrapper/build route | build/compile/link orchestration | `wrapper-plan` | -| `tests/fortran/building_shared_library/pipeline/test_source_generated_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | -| `tests/fortran/callbacks/end_to_end/test_supported_callback_shapes.py::*` | source/generated-.pyi parity or parametrized route | callbacks/trampolines; derived types/object lifetimes | `wrapper-plan` | -| `tests/fortran/callbacks/end_to_end/test_array_callbacks.py::*` | source/generated-.pyi parity or parametrized route | callbacks/trampolines; ordinary arrays | `wrapper-plan` | -| `tests/fortran/callbacks/end_to_end/test_scalar_callbacks.py::*` | source/generated-.pyi parity or parametrized route | callbacks/trampolines; scalar inputs/results | `wrapper-plan` | -| `tests/fortran/callbacks/pipeline/test_generated_callback_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | -| `tests/wrapper/fortran/derived_types/test_borrowed_finalizers.py::*` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes; classes/methods/properties/overloads | `wrapper-plan` | -| `tests/wrapper/fortran/derived_types/test_constructors_and_finalizers.py::*` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes; classes/methods/properties/overloads | `wrapper-plan` | -| `tests/wrapper/fortran/derived_types/test_derived_layout.py::*` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes | `wrapper-plan` | -| `tests/wrapper/fortran/derived_types/test_derived_type_boundaries.py::*` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes | `wrapper-plan` | -| `tests/wrapper/fortran/derived_types/test_derived_type_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | -| `tests/wrapper/fortran/derived_types/test_derived_type_methods.py::*` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes; classes/methods/properties/overloads | `wrapper-plan` | -| `tests/wrapper/fortran/derived_types/test_inheritance.py::*` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes; classes/methods/properties/overloads | `wrapper-plan` | -| `tests/wrapper/fortran/derived_types/test_phase8_derived_plan.py::*` | reduced passing legacy/source artifacts compared with direct typed-plan generation; plain non-target module objects intentionally use the safer member-proxy correction described in Phase 8 | scalar derived arguments/results; optional and by-value inputs; projected identity; owned/borrowed lifecycle; plain/`Aliased` module objects; scalar/string/array/nested/native-handle fields; production routing | `wrapper-plan` | -| `tests/wrapper/fortran/derived_types/test_phase9_bound_constructors.py::*` | reduced direct-plan bound-constructor runtime and artifact proof | explicit bound construction; shared method plan; allocation and owner commit | `wrapper-plan` | -| `tests/wrapper/fortran/derived_types/test_scalar_derived_actual_dummy_matrix.py::*` | complete source/generated-contract and direct-plan proof over the canonical scalar-derived matrix fixture; replaces the former isolated descriptor rejection unit; final Phase 8H cross-suite verification remains a separate closure gate | all five actual declarations from module and wrapper origins; all six dummy forms; exact action/error selection; holder, scoped-address, allocation and pointer transactions; mixed multi-argument acquisition and reverse cleanup; distinct module-origin callbacks for qualified types from separate Fortran modules | `wrapper-plan` | -| `tests/wrapper/fortran/derived_types/test_pointers.py::test_caller_created_pointer_crosses_separately_built_extensions` | two independently built semantic-contract extensions | caller-created pointer descriptor identity; cross-extension validation and association | `wrapper-plan` | -| `tests/wrapper/fortran/derived_types/test_pointers.py::test_caller_created_pointer_handle_tracks_native_output_association` | direct semantic-contract wrapper/build route | caller-created pointer storage attachment; output association and descriptor operations | `wrapper-plan` | -| `tests/wrapper/fortran/derived_types/test_pointers.py::test_module_and_derived_pointer_handles_track_native_association[*]` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes; native handles/descriptors | `wrapper-plan` | -| `tests/wrapper/fortran/derived_types/test_pointers.py::test_module_native_array_handles_use_canonical_plan` | canonical reduced module-only contract | borrowed pointer/allocatable handles; descriptor calls; strided extraction; ordinary array actuals; operation permissions | `wrapper-plan` | -| `tests/wrapper/fortran/derived_types/test_pointers.py::test_pointer_array_results_use_owned_descriptors_without_owning_targets[*]` | source/generated-.pyi parity or parametrized route | owned pointer result descriptors; associated and unassociated state; borrowed target lifetime | `wrapper-plan` | -| `tests/wrapper/fortran/derived_types/test_pointers.py::test_pointer_descriptor_views_preserve_slice_shape_strides_and_parent_lifetime` | direct wrapper/build route | derived types/object lifetimes; native handles/descriptors | `wrapper-plan` | -| `tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py::test_editable_contract_can_use_native_order_arguments_without_native_call` | direct wrapper/build route | semantic .pyi generation/parsing; raw array addresses completed by Phase 6G; derived result remains Phase 8 | `wrapper-plan` | -| `tests/fortran/raw_addresses/end_to_end/test_raw_native_addresses.py::test_primitive_array_and_fixed_string_raw_addresses_share_one_native_build` | reduced edited semantic `.pyi` entries over existing scalar, vector, matrix, and fixed-string native routines | raw primitive, numeric-array, and fixed-string addresses; checked fixed-string storage; visible scalar-storage extents; rank one/two; default C and explicit Fortran orientation; mutation; integer-only conversion | `wrapper-plan` | -| `tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py::test_copy_f_preserves_logical_axes_through_binding_owned_temporary` | reduced edited semantic `.pyi` entries over the existing matrix native routine | explicit C-to-Fortran representation copy; native-input and inout calls; projected original identity; binding-owned copyback and cleanup | `wrapper-plan` | -| `tests/wrapper/fortran/edit_pyi_contracts/test_ownership_contracts.py::*` | direct wrapper/build route | semantic .pyi generation/parsing; native handles/descriptors; derived types/object lifetimes; optional/presence/writeback | `wrapper-plan` | -| `tests/wrapper/fortran/edit_pyi_contracts/test_policy_dispatch_contracts.py::test_contradictory_ownership_contract_fails_before_bridge_generation` | non-generating: policy validation before bridge generation | semantic .pyi generation/parsing; native handles/descriptors; derived types/object lifetimes; optional/presence/writeback | `not-applicable` | -| `tests/wrapper/fortran/edit_pyi_contracts/test_policy_dispatch_contracts.py::test_immutable_array_policy_copies_in_and_returns_replacement` | direct wrapper/build route | semantic .pyi generation/parsing; native handles/descriptors; derived types/object lifetimes; optional/presence/writeback | `wrapper-plan` | -| `tests/wrapper/fortran/edit_pyi_contracts/test_policy_dispatch_contracts.py::test_immutable_scalar_string_array_and_derived_policies_return_replacements` | direct wrapper/build route | semantic .pyi generation/parsing; native handles/descriptors; derived types/object lifetimes; optional/presence/writeback | `wrapper-plan` | -| `tests/wrapper/fortran/edit_pyi_contracts/test_surface_edit_contracts.py::*` | direct wrapper/build route | semantic .pyi generation/parsing; classes/methods/properties/overloads; naming/visibility/dispatch | `wrapper-plan` | -| `tests/wrapper/fortran/edit_pyi_contracts/test_visibility_contracts.py::*` | direct wrapper/build route | semantic .pyi generation/parsing; scalar module visibility and namespace projection | `wrapper-plan` | -| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_compact_blas_like_folder_generates_one_standalone_entry_and_preserves_separate_objects` | direct wrapper/build route | external symbols/native linkage; ordinary arrays | `wrapper-plan` | -| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_external_bind_renames_python_export_without_changing_native_call` | direct wrapper/build route | scalar external symbol; explicit bridge interface; renamed export | `wrapper-plan` | -| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_classic_external_bridge_uses_implicit_declaration_and_no_module_use` | direct wrapper/build route | scalar external symbol; implicit external declaration | `wrapper-plan` | -| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_fixed_form_standalone_procedure_runtime_parity[*]` | source/generated-.pyi parity or parametrized route | scalar external symbol; explicit bridge interface | `wrapper-plan` | -| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_free_form_standalone_procedure_runtime_parity[*]` | source/generated-.pyi parity or parametrized route | scalar external symbol; explicit bridge interface | `wrapper-plan` | -| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_generated_standalone_contracts_are_non_empty_root_fragments` | direct wrapper/build route | external symbols/native linkage | `wrapper-plan` | -| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_handwritten_c_order_flat_contract_passes_rank_preserving_bridge_view` | direct wrapper/build route | external symbols/native linkage; ordinary arrays | `wrapper-plan` | -| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_handwritten_fortran_order_flat_contract_flattens_the_final_python_axes` | direct wrapper/build route | external symbols/native linkage; flat arrays; scalar storage | `wrapper-plan` | -| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_standalone_allocatable_argument_accepts_a_caller_created_handle` | direct wrapper/build route | external symbols/native linkage; native allocatable descriptors | `wrapper-plan` | -| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_optional_flat_contracts_preserve_present_and_absent_calls` | direct wrapper/build route | optional/presence; F-order and C-order flat arrays | `wrapper-plan` | -| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_module_procedure_bridge_uses_native_module_scope` | direct wrapper/build route | external symbols/native linkage | `wrapper-plan` | -| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_namespace_imported_module_rejects_standalone_marker_before_codegen` | non-generating: validation/failure-path assertion | external symbols/native linkage | `not-applicable` | -| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_one_source_with_several_standalone_procedures_exports_each_at_root[*]` | source/generated-.pyi parity or parametrized route | scalar external symbols; explicit bridge interfaces | `wrapper-plan` | -| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_package_entry_rejects_non_standalone_root_declaration_before_codegen` | non-generating: validation/failure-path assertion | external symbols/native linkage | `not-applicable` | -| `tests/wrapper/fortran/function_calls/test_function_call_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | -| `tests/wrapper/fortran/function_calls/test_native_call_examples.py::test_native_call_examples_build_from_generated_pyi_and_native_shared_library` | direct wrapper/build route | native-call projections; ordinary arrays; strings; derived types/object lifetimes | `wrapper-plan` | -| `tests/wrapper/fortran/function_calls/test_native_call_examples.py::test_native_call_examples_cover_scalar_array_string_and_object_projection[*]` | source/generated-.pyi parity or parametrized route | native-call projections; ordinary arrays; strings; derived types/object lifetimes | `wrapper-plan` | -| `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_fixed_form_optional_arguments_drive_fortran_present_behavior[*]` | source/generated-.pyi parity or parametrized route | optional/presence; scalar inputs/results | `wrapper-plan` | -| `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_fixed_optional_scalar_plan_matches_all_presence_states` | canonical production plan route | optional/presence; scalar inputs/results; build/artifact integration | `wrapper-plan` | -| `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_allocatable_scalar_descriptor_distinguishes_omitted_none_and_value` | canonical production plan route | optional/presence; nullable scalar descriptor; build/artifact integration | `wrapper-plan` | -| `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_array_descriptors_preserve_presence_and_storage_state` | canonical reduced optional descriptor contract | omitted/`None` absence; present unallocated/unassociated and allocated/associated handle states; kind/dtype validation | `wrapper-plan` | -| `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_arguments_drive_fortran_present_behavior[*]` | source/generated-.pyi parity or parametrized route | optional/presence/writeback | `wrapper-plan` | -| `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_array_buffers_preserve_omission_and_identity` | reduced semantic `.pyi` entry over the existing optional native unit | omitted/`None`/present ordinary array storage; mutation; projected identity; native-handle actuals deferred to Phase 7 | `wrapper-plan` | -| `tests/wrapper/fortran/function_calls/test_scalar_writeback_plan.py::test_scalar_copy_in_out_returns_replacement` | canonical production plan route | scalar copy-in/native mutation/copy-out/cleanup; build/artifact integration | `wrapper-plan` | -| `tests/wrapper/fortran/function_calls/test_scalar_writeback_plan.py::test_source_generated_scalar_inout_contract_returns_replacement_and_keeps_namespace` | source/generated-.pyi parity | scalar replacement projection; namespace preservation; semantic .pyi generation/parsing | `wrapper-plan` | -| `tests/wrapper/fortran/function_calls/test_output_arguments.py::test_output_arguments_and_multiple_results_follow_python_projection_rules[*]` | source/generated-.pyi parity | mixed-type multiple-result aggregation; ordinary arrays; strings; derived types/object lifetimes; native handles/descriptors | `wrapper-plan` | -| `tests/wrapper/fortran/function_calls/test_output_arguments.py::test_hidden_ordinary_array_output_uses_canonical_plan` | canonical production output-only plan route | fixed/runtime-shape hidden ordinary array output; zero-sized output; allocation/copy failure paths | `wrapper-plan` | -| `tests/wrapper/fortran/layout_rules/test_wrapper_guide_layout.py::*` | non-generating: wrapper docs/test layout | test/docs layout | `not-applicable` | -| `tests/wrapper/fortran/module_state/test_allocatable_replacement.py::test_allocatable_inout_arrays_mutate_and_return_the_same_handle[*]` | source/generated-.pyi parity route | module variables/state; native handles/descriptors | `wrapper-plan` | -| `tests/wrapper/fortran/module_state/test_allocatable_replacement.py::test_allocatable_replacement_has_no_native_memory_errors[*]` | source/generated-.pyi parity route | module variables/state; native handles/descriptors | `wrapper-plan` | -| `tests/wrapper/fortran/module_state/test_allocatable_replacement.py::test_caller_created_allocatable_crosses_separately_built_extensions` | two independently built semantic-contract extensions | caller-created allocatable descriptor identity; cross-extension validation and mutation | `wrapper-plan` | -| `tests/wrapper/fortran/module_state/test_allocatable_replacement.py::test_projected_allocatable_descriptor_preserves_same_handle_identity` | canonical reduced owned-result plus projected-descriptor contract | direct persistent descriptor mutation; allocation/reallocation/deallocation; same-handle result identity | `wrapper-plan` | -| `tests/wrapper/fortran/module_state/test_allocatable_views.py::test_allocatable_module_fields_and_results_expose_lifetime_safe_handles[*]` | source/generated-.pyi parity with one mixed generation unit | derived class/field handles and parent retention remain Phase 8/9 blockers | `wrapper-plan` | -| `tests/wrapper/fortran/module_state/test_allocatable_views.py::test_maybe_unallocated_direct_allocatable_results_preserve_unallocated_state` | edited semantic `.pyi` contract over the existing allocatable module unit | `MaybeUnallocated` direct allocatable result annotation preserves the unallocated result state without changing default always-allocated result handling | `wrapper-plan` | -| `tests/wrapper/fortran/module_state/test_allocatable_views.py::test_scalar_descriptor_module_variables_return_copied_optional_values[*]` | production plan route in source/generated-.pyi parity modes | rank-zero allocatable/pointer arguments, writeback, results, and copied nullable module values | `wrapper-plan` | -| `tests/wrapper/fortran/module_state/test_allocatable_views.py::test_plain_allocatable_module_array_exposes_current_live_view[*]` | production plan route after the Phase 7 contract correction | plain and `Aliased` module handles return a current live view or `None`; explicit `.copy()` is independent and a fresh extraction follows current native state | `wrapper-plan` | -| `tests/wrapper/fortran/module_state/test_common_blocks.py::*` | source/generated-.pyi parity or parametrized route | scalar calls with internal common-block storage | `wrapper-plan` | -| `tests/wrapper/fortran/module_state/test_module_state.py::test_aliased_derived_module_object_borrows_native_state[*]` | source/generated-.pyi parity or parametrized route | module variables/state; derived types/object lifetimes | `wrapper-plan` | -| `tests/wrapper/fortran/module_state/test_module_state.py::test_scalar_module_variables_use_attributes_and_parameters_have_no_native_setter[*]` | source/generated-.pyi parity or parametrized route | module variables/state | `wrapper-plan` | -| `tests/wrapper/fortran/module_state/test_module_state_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | -| `tests/wrapper/fortran/module_state/test_scalar_module_variable_plan.py::test_whole_scalar_module_variable_behavior_uses_canonical_plan` | canonical production plan route | scalar inputs/results; scalar module variables/state; build/artifact integration | `wrapper-plan` | -| `tests/fortran/building_shared_library/end_to_end/test_multi_source_builds.py::test_makefile_mode_reproduces_multi_source_build` | direct wrapper/build route | build/compile/link orchestration; external symbols/native linkage; module variables/state | `wrapper-plan` | -| `tests/fortran/building_shared_library/end_to_end/test_multi_source_builds.py::test_multi_file_modules_build_one_merged_extension` | direct wrapper/build route | scalar multi-source build/link orchestration; module variables/state | `wrapper-plan` | -| `tests/fortran/building_shared_library/end_to_end/test_multi_source_builds.py::test_multi_file_standalone_procedures_build_one_merged_extension` | direct wrapper/build route | scalar multi-source external symbols and link orchestration | `wrapper-plan` | -| `tests/fortran/building_shared_library/end_to_end/test_multi_source_builds.py::test_generated_child_modules_are_importable_submodules` | direct wrapper/build route | generated child-module imports and namespace preservation | `wrapper-plan` | -| `tests/fortran/building_shared_library/end_to_end/test_multi_source_builds.py::test_multi_source_generated_contract_build_matches_source_runtime_and_link_order` | direct wrapper/build route | build/compile/link orchestration; external symbols/native linkage; module variables/state; semantic .pyi generation/parsing | `wrapper-plan` | -| `tests/fortran/building_shared_library/end_to_end/test_multi_source_builds.py::test_multi_source_modified_entry_preserves_modules_and_adds_documented_alias` | direct wrapper/build route | build/compile/link orchestration; external symbols/native linkage; module variables/state | `wrapper-plan` | -| `tests/fortran/building_shared_library/end_to_end/test_multi_source_builds.py::test_multi_source_pyi_out_writes_one_flat_combined_package` | direct wrapper/build route | build/compile/link orchestration; external symbols/native linkage; module variables/state; semantic .pyi generation/parsing | `wrapper-plan` | -| `tests/wrapper/fortran/naming/test_defined_operators.py::*` | source/generated-.pyi parity or parametrized route | naming/visibility/dispatch; classes/methods/properties/overloads; operators; generic dispatch | `wrapper-plan` | -| `tests/wrapper/fortran/naming/test_generic_interfaces.py::*` | source/generated-.pyi parity or parametrized route | naming/visibility/dispatch; classes/methods/properties/overloads; generic dispatch | `wrapper-plan` | -| `tests/wrapper/fortran/naming/test_naming_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | -| `tests/wrapper/fortran/naming/test_phase9_class_overloads.py::*` | reduced direct-plan constructor and method overload runtime proof | class-owned exact predicates; constructor ownership; no speculative calls | `wrapper-plan` | -| `tests/wrapper/fortran/naming/test_visibility_naming.py::test_strict_wrapper_names_reject_python_name_fixes` | direct wrapper/build route | naming/visibility/dispatch; classes/methods/properties/overloads | `wrapper-plan` | -| `tests/wrapper/fortran/naming/test_visibility_naming.py::test_visibility_and_default_python_name_fixing_policy[*]` | source/generated-.pyi parity or parametrized route | naming/visibility/dispatch; classes/methods/properties/overloads | `wrapper-plan` | -| `tests/fortran/building_shared_library/end_to_end/test_native_bundles.py::test_duplicate_native_definitions_report_linker_error` | direct wrapper/build route | scalar external symbols; linker failure propagation | `wrapper-plan` | -| `tests/fortran/building_shared_library/end_to_end/test_native_bundles.py::test_imported_contracts_resolve_from_one_archive_or_shared_library[*]` | source/generated-.pyi parity or parametrized route | external symbols/native linkage; build/compile/link orchestration | `wrapper-plan` | -| `tests/fortran/building_shared_library/end_to_end/test_native_bundles.py::test_incompatible_native_artifact_reports_linker_error` | non-generating: validation/failure-path assertion | external symbols/native linkage; build/compile/link orchestration | `not-applicable` | -| `tests/fortran/building_shared_library/end_to_end/test_native_bundles.py::test_missing_module_directory_reports_compile_error` | non-generating: validation/failure-path assertion | external symbols/native linkage; build/compile/link orchestration | `not-applicable` | -| `tests/fortran/building_shared_library/end_to_end/test_native_bundles.py::test_missing_symbol_reports_native_link_or_loader_error` | non-generating: validation/failure-path assertion | external symbols/native linkage; build/compile/link orchestration | `not-applicable` | -| `tests/fortran/building_shared_library/end_to_end/test_native_bundles.py::test_mixed_module_external_bundle_resolves_all_native_input_kinds` | direct wrapper/build route | scalar module/external symbols; ordered native inputs and library directories | `wrapper-plan` | -| `tests/fortran/building_shared_library/end_to_end/test_native_bundles.py::test_required_transitive_named_library_resolves_runtime_symbol` | direct wrapper/build route | scalar external symbol; transitive named library | `wrapper-plan` | -| `tests/fortran/building_shared_library/end_to_end/test_native_bundles.py::test_static_archive_dependency_order_resolves_transitive_library` | direct wrapper/build route | scalar external symbol; ordered archive linkage | `wrapper-plan` | -| `tests/fortran/building_shared_library/end_to_end/test_native_bundles.py::test_static_archive_groups_resolve_cyclic_archive_dependencies` | direct wrapper/build route | scalar external symbol; archive-group linkage | `wrapper-plan` | -| `tests/fortran/building_shared_library/end_to_end/test_native_bundles.py::test_unavailable_dependent_shared_library_reports_loader_error` | non-generating: validation/failure-path assertion | external symbols/native linkage; build/compile/link orchestration | `not-applicable` | -| `tests/wrapper/fortran/runtime_behavior/test_openmp_runtime.py::*` | direct wrapper/build route | runtime policies/errors/GIL; build/compile/link orchestration | `wrapper-plan` | -| `tests/wrapper/fortran/runtime_behavior/test_runtime_behavior_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | -| `tests/fortran/error_handling/end_to_end/test_status_projection.py::*` | edited-.pyi canonical production plan route with focused semantic and lowering evidence | runtime status projection, errors, cleanup, and GIL envelope | `wrapper-plan` | -| `tests/wrapper/fortran/runtime_behavior/test_runtime_recursion.py::*` | source/generated-.pyi parity or parametrized route | runtime policies/errors/GIL; scalar inputs/results | `wrapper-plan` | -| `tests/fortran/enumerations/end_to_end/test_enum_runtime.py::test_fortran_enums_preserve_integer_runtime_surface[*]` | source/generated-.pyi parity or parametrized route | scalar inputs/results; module variables/state | `wrapper-plan` | -| `tests/fortran/enumerations/semantics/test_enum_semantics.py::test_fortran_enums_preserve_values_in_generated_pyi_contract` | non-generating semantic and contract-emission evidence | scalar inputs/results; module constants/state | `not-applicable` | -| `tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py::*` | scalar-only copied native routines with deliberate legacy/direct-plan parity | primitive scalar kinds; value and `Addr(Arg(i))` inputs; hidden output; copy-in/copy-out; rank-zero storage; raw `Addr(T)`; native slot reordering; direct-plus-hidden result tuple assembly | `wrapper-plan` | -| `tests/wrapper/fortran/scalars/test_scalar_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | -| `tests/wrapper/fortran/scalars/test_scalar_kinds.py::*` | source/generated-.pyi parity or parametrized route | scalar inputs/results | `wrapper-plan` | -| `tests/wrapper/fortran/scalars/test_value_and_bind_c.py::*` | source/generated-.pyi parity or parametrized route | scalar inputs/results; native-call projections | `wrapper-plan` | -| `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_fmath_scalar_sources_use_canonical_wrapper_plan[*]` | canonical production plan route using the existing fixed- and free-form generation units | scalar inputs/results; native-call projections; Python namespaces; build/artifact integration | `wrapper-plan` | -| `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_f90_array_wrapper_distinguishes_contiguous_and_strided_contracts[*]` | source/generated-.pyi parity or parametrized route | scalar inputs/results; ordinary arrays | `wrapper-plan` | -| `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_f90_wrapper_pipeline_builds_importable_extension[*]` | source/generated-.pyi parity or parametrized route | scalar inputs/results | `wrapper-plan` | -| `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_fortran_array_wrapper_pipeline_matches_fmath_results_with_contiguous_arrays[*]` | source/generated-.pyi parity or parametrized route | scalar inputs/results; ordinary arrays | `wrapper-plan` | -| `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_fortran_wrapper_pipeline_builds_importable_extension[*]` | source/generated-.pyi parity or parametrized route | scalar inputs/results | `wrapper-plan` | -| `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_required_array_buffers_use_canonical_wrapper_plan` | reduced semantic `.pyi` entry over the existing `fmath_arrays_f90` native unit | required rank-one dense buffers; exact dtype/rank/order/alignment/writeability; zero length; native-handle actuals deferred to Phase 7 | `wrapper-plan` | -| `tests/wrapper/fortran/strings/test_character_arguments.py::test_edited_modern_string_contract_wraps_full_axis_spelling_set` | edited semantic `.pyi` contract | strings; fixed/assumed inputs; arrays; mutable string storage | `wrapper-plan` | -| `tests/wrapper/fortran/strings/test_character_arguments.py::test_legacy_fortran_character_arguments_and_results[*]` | production plan route from source/generated-.pyi parity | fixed-form strings; fixed/assumed inputs; fixed results | `wrapper-plan` | -| `tests/wrapper/fortran/strings/test_character_arguments.py::test_modern_fortran_character_arguments_and_results[*]` | source/generated-.pyi parity | strings; fixed/assumed inputs; fixed/deferred results; arrays; writeback | `wrapper-plan` | -| `tests/wrapper/fortran/strings/test_character_arguments.py::test_deferred_allocatable_string_results_use_canonical_plan` | canonical reduced scalar descriptor result contract | runtime length; nullable copy-out; UTF-8 data; allocation failure | `wrapper-plan` | -| `tests/wrapper/fortran/strings/test_character_arguments.py::test_deferred_character_array_handles_use_canonical_plan` | canonical reduced descriptor-result and projected-descriptor contract | hidden/direct owned deferred-character arrays; runtime `S3`/`S4`/`S5` width; projected identity; nullable rank-zero result | `wrapper-plan` | -| `tests/wrapper/fortran/strings/test_character_arguments.py::test_fixed_width_character_arrays_use_canonical_plan` | reduced semantic `.pyi` entry over the existing `fstrings_f90` native unit | fixed-width `NPY_STRING` array itemsize; rank/dtype/zero-size validation; native-handle actuals deferred to Phase 7 | `wrapper-plan` | -| `tests/wrapper/fortran/strings/test_character_arguments.py::test_raw_fixed_width_character_arrays_use_canonical_plan` | reduced edited semantic `.pyi` entry over the existing `fstrings_f90` native unit | raw fixed-width character array address; literal shape; element length; integer-only conversion | `wrapper-plan` | -| `tests/wrapper/fortran/strings/test_character_arguments.py::test_required_scalar_string_inputs_use_canonical_plan` | reduced semantic `.pyi` entry over the existing `fstrings_f90` native unit | required fixed/assumed scalar string inputs; default/kind-1/`c_char`; UTF-8 length and NUL validation; scalar results | `wrapper-plan` | -| `tests/wrapper/fortran/strings/test_character_arguments.py::test_fixed_string_results_use_canonical_plan` | reduced semantic `.pyi` entry over the existing `fstrings_f90` native unit | direct fixed string results; trailing blanks; default/`c_char`; allocation failure | `wrapper-plan` | -| `tests/wrapper/fortran/strings/test_character_edge_cases.py::test_fortran_character_edge_cases_follow_copy_in_copy_out_policy[*]` | production plan route from source/generated-.pyi parity | strings; fixed/assumed input/output; optional presence; Unicode/NUL handling | `wrapper-plan` | -| `tests/wrapper/fortran/strings/test_character_edge_cases.py::test_fixed_hidden_string_output_uses_canonical_plan` | reduced semantic `.pyi` entry over the existing `fcharacter_edges_f90` native unit | fixed hidden string output; trailing blanks; allocation failure | `wrapper-plan` | -| `tests/wrapper/fortran/strings/test_character_edge_cases.py::test_fixed_string_replacement_and_identity_use_canonical_plan` | reduced edited semantic `.pyi` entry over the existing `fcharacter_edges_f90` native unit | fixed immutable replacement and discarded identity; exact length; trailing blanks; allocation failure | `wrapper-plan` | -| `tests/wrapper/fortran/strings/test_character_edge_cases.py::test_assumed_and_optional_string_replacements_use_canonical_plan` | reduced semantic `.pyi` entry over the existing `fcharacter_edges_f90` native unit | assumed-length and optional immutable replacement; empty/omitted/`None`/concrete states; NUL rejection; concrete-only allocation failure | `wrapper-plan` | -| `tests/wrapper/fortran/strings/test_string_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | - -## Incremental Protocol - -For each lane: - -1. Select and run an existing passing `tests/wrapper` generation unit through - the legacy route. Retain and inspect its complete generated artifact set. -2. Trace the current lowering, binding, bridge, node/API-model, printer, - runtime-helper, and build paths that produced those artifacts, and record the - observed behavior and existing tests that make them the migration baseline. -3. Expand this checklist with the lane's exact scope, exclusions, source-path - baseline, required backend behavior, plan fields, and validation invariants. -4. Complete every policy field required by the lane in post-IR policy - completion; do not start its planner while semantic decisions remain - scattered or implicit. -5. Implement the lane's hierarchical plan records, planner visitors, - ABI/handoff specs, generator-owned structural checks, directly named backend - lowering methods, source-printer support, and support-report coverage. -6. Implement the minimum dependency-closed backend slice in - `prik.codegen`: copy small suitable pieces, rewrite oversized legacy - classes as minimal equivalents, and add only the intermediate tests required - by the contract above. -7. Generate the plan from policy-completed semantic IR, invoke the directly - named binding and bridge lowering methods, assemble complete backend modules, - and print complete internal artifacts. -8. Compare the new generated artifacts with the retained legacy artifacts and - explain every material difference before compilation. -9. Compile the internal artifacts before changing production route selection. -10. Run the same eligible existing fixtures and assertions through both routes - and compare compiled runtime behavior, failure paths, native-call mapping, - and artifact requirements. -11. Extend the whole-module support predicate so a generation unit uses the - wrapper-plan route only when all its elements belong to completed lanes. - Keep the old route for generation units containing unsupported lanes. -12. Update every affected `tests/wrapper` migration-matrix row and mark the lane - complete only when the reused parity tests pass and every intentional - difference from the baseline is separately documented. - -Do not start a later lane by guessing. Each lane must define the handoff specs -and consistency checks it needs. - -## Mandatory Expansion Gate For Broad Phases - -Phases 5 through 10 are roadmap envelopes, not complete implementation -checklists. Before implementation starts on one of them, update this file and -split that phase into dependency-ordered sub-lanes. The expansion must be based -on an audit of the live semantic models, completed policies, existing -bridge/binding behavior, decorators, and focused wrapper tests. - -Each expanded sub-lane must state: - -- the exact included and excluded semantic cases; -- the completed-policy fields it consumes and any decisions that still need to - move into post-IR policy completion; -- plan records, action keys, handoff specs, native-call slots, lifecycle phases, - and required generated artifacts; -- binding and bridge handler names and which backend-local helper values they - may create; -- validation invariants across Python input/result, binding handoff, bridge - handoff, native call, writeback, cleanup, ownership, and release; -- the whole-module support-predicate change that makes the sub-lane eligible; -- existing `tests/wrapper` nodes that cover the sub-lane, their migration-matrix - status changes, and dual-route parity evidence against the legacy route; -- the exact legacy source path and consumed behavior for every isolated - primitive, whether it is copied or rewritten, its minimal dependency closure, - baseline evidence, and a reason for every behavior with no legacy source; -- dependencies on earlier lanes and the legacy behavior that can be removed - when the sub-lane is complete. - -Do not mark a broad phase complete from its current envelope items. Mark its -expanded sub-lanes individually, then close the phase only after all live cases -in its audited support matrix are either migrated or explicitly removed from -the product contract. - -## Dependency-Ordered Checklist - -### Foundation and semantic authority - -- [x] Establish the isolated `prik.codegen` package boundary and - visitor infrastructure. -- [x] Complete the first primitive lane in general wrapper policy before - planning, including native-call order, result projection, ownership, and - lifecycle facts. -- [x] Build editable `ModulePlan`, `FunctionPlan`, transfer, result, ABI, - native-slot, and lifecycle records from completed wrapper policy. -- [x] Refactor each cross-boundary owner into explicit binding and bridge child - plans, including module/function/result/lifecycle scope as well as arguments. -- [x] Remove plan-owned method names and handler registries; add typed - datatype-family facts required by direct lowering. -- [x] Keep structural plan validation private to `WrapperCodeGenerator`, with - no planner-time validation or standalone validator class, and verify every - listed invariant after direct plan edits. - -### Direct generator boundary - -- [x] Change `WrapperCodeGenerator.generate` to consume only `ModulePlan`, - freeze it, validate it, validate lowering support, recursively generate - backend nodes, print them, and return artifacts directly. -- [x] Implement recursive `CBindingGenerator` synthesis of complete C modules, - headers, and functions from plan nodes. -- [x] Implement recursive `FortranBridgeGenerator` synthesis of complete - Fortran modules and functions from plan nodes. -- [x] Replace plan-selected method names with directly named backend lowering - methods selected by the visible `_lower_{subject}_{action.value}` rule. -- [x] Ensure backend node printers, artifact construction, and build - consumption retain their distinct freezing boundaries. - -### Scalar parity and route evidence - -- [x] Replay the existing scalar source and semantic-`.pyi` baseline through - the direct generator and compare generated artifacts and runtime behavior. -- [x] Update the staged walkthrough to use only plan editing and the public - generator boundary. -- [x] Retire superseded internal representations, their package exports, - orchestration, validation, documentation, and focused tests. -- [x] Run focused wrapper-codegen and pipeline tests; the walkthrough for both - supported entry choices where practical; `tests/wrapper` excluding LAPACK; - documentation checks; `git diff --check`; the required static-analysis suite; - and `tools/check_codegen_complexity.py`. - -## Phase 3 — Scalar Inout, Optional, And Descriptor-Like Scalars - -Scope: scalar copy-in/copy-out, optional arguments, present-but-null descriptor -values, and scalar allocatable/pointer descriptor boundaries. - -Phase 3 legacy replay audit: - -- `foptional_fixed.f` uses one nullable value pointer at the Bind-C ABI. - Omission and explicit `None` both pass a null pointer because both mean that - the ordinary optional dummy is absent; a concrete scalar uses call-local - storage, and the bridge branches on `c_associated(...)` before calling the - native function with or without the optional keyword. -- The existing optional allocatable-scalar contract uses two independent ABI - pointers. The value pointer is null for explicit `None`, while the presence - pointer is non-null for both `None` and a concrete value. Omission leaves both - null. The bridge therefore distinguishes absent, present-unallocated, and - present-with-value states without inferring presence from the value pointer. -- Immutable scalar replacement uses copy-in storage, native mutation of that - storage, copy-out to a new Python scalar, and scope-owned stack cleanup. The - caller's original NumPy scalar remains unchanged. The audit found no existing - runtime wrapper test for this primitive-scalar `Returns["argument", T]` - contract, so `test_scalar_writeback_plan.py` is the recorded coverage-gap - fixture for this lane. -- The first legacy replay of that coverage gap exposed a duplicate declaration - of the mutable scalar result. The legacy bridge now promotes the copy-in - temporary to the Bind-C function result and removes it from the ordinary - local-declaration set. Both routes compile, and incompatible Python values - fail with `TypeError` before the native call. Stack temporaries and local - allocatable descriptors require no explicit release action; their procedure - scope owns cleanup on normal return. - -The Phase 3 plan records optional mode, nullable value and presence handoffs, -and four ordered scalar replacement phases: `copy_in`, `native_mutation`, -`copy_out`, and `cleanup`. Generator preflight requires the complete phase set, -an existing source handoff, the correct binding/bridge owner for each phase, -and a Python result target for copy-out. Forced whole-module route selection -accepts these completed lanes after the dual-route evidence below. Automatic -production selection still remains on the legacy route under the independent -GIL parity deferral recorded for Phase 2D. - -- [x] Audit and record the legacy copy-in/out, optional presence, nullable - scalar descriptor, cleanup, and failure-path behavior for this lane. -- [x] Add or rewrite only the additional optional/descriptor nodes, API - primitives, local-state helpers, and printer cases required by this lane, - with baseline tests. -- [x] Represent copy-in, native mutation, copy-out, and cleanup as explicit - writeback phases. -- [x] Preserve the three-state optional rule: omitted argument, explicit `None`, - and present concrete value are distinct when the native ABI needs them. -- [x] Represent scalar descriptor presence tokens and nullable value handoffs in - the plan. -- [x] Validate that a writeback consumes an existing binding/bridge handoff and - writes to a Python-visible target or result slot. -- [x] Emit and print complete inout/optional/descriptor-capable modules - internally, then compile and compare both routes for all three presence - states, mutation, writeback, cleanup, ABI, and failures. -- [x] Widen whole-module route eligibility to this lane only after parity, and - complete it before moving arrays or handles to the plan path. - -## Phase 4 — Scalar Module Variables - -Scope: scalar module variables. Derived-type fields remain in Phases 8 and 9 -because their wrapper instance, owner, and property lifecycle must already be -represented before field access can use the plan route. - -Phase 4 legacy replay audit: - -- `fmodule_vars_f90.f90` establishes the ordinary scalar state contract. Its - legacy bridge emits value-returning getters and value-argument setters; - binding accessors run with the GIL held, aliases route to the same native - storage, deletion fails, and contract initializers call the native setter at - import. A `parameter` is instead copied into the Python module dictionary, so - rebinding it is local to that module object and never mutates native storage. - This whole source remains legacy because it also owns `rgb_color` and derived - module objects from later phases. -- The scalar subset of `fscalar_descriptors_f90` establishes nullable - allocatable and pointer reads. The legacy bridge returns null for absent - storage or allocates and copies one detached scalar; the binding converts the - copy, frees it, and rejects descriptor replacement. Its whole source cannot - migrate in Phase 4 because it also contains nullable snapshot-result forms - from a later lane. Allocation failure is deliberately injected with - `PRIK_WRAPPER_FAIL_ALLOC` and preserves the legacy null/`None` surface. -- No existing generation unit contained only the already completed scalar - function lanes plus every Phase 4 getter, setter, constant, descriptor, - initialization, reload, and failure behavior. The bounded - `test_scalar_module_variable_plan.py` whole-module fixture records that - coverage gap; it contains no strings, arrays, classes, or later-phase owner. - -The plan keeps only completed typed facts: Python names, getter and setter -actions, initializer or constant value, datatype family, native name/module, -native assignment, descriptor kind, and handoff roles. Both backends invoke -directly named lowering methods with matching subject/action suffixes wherever -their behavior is shared; datatype and descriptor facts stay method inputs. -The generator validates the complete frozen plan before either backend emits -anything, including binding/bridge getter agreement and the rule that a Python -write-through setter must have a compatible bridge setter role. Forced -whole-module selection now accepts `scalar-module-variables`; automatic -production selection remains independently deferred by the Phase 2D GIL gate. - -- [x] Audit and record the legacy scalar module-variable getter, setter, - rejected replacement, module initialization, and attribute-routing behavior. -- [x] Add or rewrite only the additional module/type nodes, getter/setter API - primitives, initialization nodes, and printer cases required by this lane, - with baseline tests. -- [x] Represent getter behavior, setter exposure, native setter assignment, and - rejected replacement behavior in module-variable plans. -- [x] Add binding actions for Python attribute get/set around scalar values. -- [x] Add bridge actions for scalar module-variable read/write. -- [x] Validate getter/setter pair consistency: a Python setter cannot exist - without a compatible bridge setter handoff. -- [x] Keep ordinary Python module-name rebinding semantics separate from native - module-variable storage. -- [x] Emit and print complete module-variable-capable modules internally, then - compile and compare both routes for get/set behavior, rejection paths, - initialization, cleanup, ABI, and generated artifacts. -- [x] Widen whole-module route eligibility to scalar module variables only after - that parity evidence passes. - -### Phase 3/4 whole-unit namespace correction - -The post-Phase 4 review found that scalar lowering itself matched the legacy -route, but the parity helper unwrapped a sole native child module before making -assertions. That hid a public-surface difference: the legacy route retained -Fortran modules as Python child namespaces while the plan route flattened their -members at the extension root. The old route-selection validation also accepted -colliding procedures from separate native modules and allowed the failure to -reach the Fortran compiler. - -This correction is part of the completed scalar foundation rather than a new -datatype lane: - -- [x] Add a concise `NamespacePlan` beneath `ModulePlan`; place functions and - variables in namespace nodes instead of flattening them into the module. -- [x] Complete Python export paths in post-IR export policy, including - namespace-local keyword and collision fixes, then mechanically group plan - owners by those paths without reconstructing namespace policy in either - backend. -- [x] Generate root, child, and nested Python modules while keeping native - module imports and generated bridge symbols unambiguous. -- [x] Support ordinary scalar subroutines with no projected result through the - existing native call plus Python `None` result path. -- [x] Reject duplicate Python exports and generated symbols before either - backend emits source. -- [x] Use one visible lowering naming rule in both backends: - `_lower_argument_`, `_lower_result_`, - `_lower_writeback_`, `_lower_module_getter_`, - and `_lower_module_setter_`. Do not store method-name strings - in the plan or hide these selections in backend dictionaries. -- [x] Remove scalar prefixes from general wrapper concepts, including the - function, argument, result, native-slot, lifecycle, node, and printer policy - surfaces. Retain scalar naming only for permanently scalar-specific ABI type - facts and actions. -- [x] Update the staged walkthrough to print the namespace tree, typed actions, - and the directly corresponding binding and bridge method names. -- [x] Compile both routes from the existing complete - `contract_mixed_module_external.f90`, `contract_import_graph.f90`, - `contract_multi_module.f90`, `contract_standalone_only.f90`, and - `contract_same_name.f90` fixtures; compare the real extension root and child - namespaces without `_sole_native_module` normalization. - -## Phase 2D — Native Call Runtime Envelope - -This is the next dependency-closed migration lane. Complete it before Phase 5 -so the already proven scalar generation units can move from temporary -`dual-route` evidence to production `wrapper-plan` routing instead of adding -more datatype lanes behind the same runtime gate. - -Scope: the binding-owned runtime envelope around an otherwise completed native -call. This phase includes default GIL release, explicit `@hold_gil`, and native -status/message projection through `@raises(...)`. Status projection is included -because the existing `fruntime_policy_f90` generation unit tests it together -with both GIL modes and whole-generation-unit routing cannot split that module. - -Excluded from this phase: - -- strings, arrays, descriptors, derived types, and callbacks, which remain in - their datatype or callback phases; -- callback re-entry and callback exception/abort behavior, which remain in - Phase 10; -- OpenMP array execution and Makefile-specific behavior, which remain blocked - by the array and cross-cutting build lanes; -- general Python exception translation that is not selected by a completed - native status policy. - -The final runtime oracle is -`tests/fortran/error_handling/end_to_end/test_status_projection.py`, -backed by focused semantic and lowering evidence in the same feature. It proves -that successful status returns produce the declared Python result, failing -status returns raise the selected exception with the native message, cleanup -completes on repeated failures, and emitted C places -`Py_BEGIN_ALLOW_THREADS` / `Py_END_ALLOW_THREADS` only around eligible native -calls. `test_recursive_native_runtime_calls` is the scalar regression unit to -run after the call envelope works. Do not use the OpenMP or callback fixtures -as the first parity unit. - -The plan and generators must follow these boundaries: - -- Post-IR policy completion owns `hold_gil` and the complete native status - error decision, including status source, message source, success value, and - Python exception kind. The planner only projects those completed facts. -- Keep the runtime facts concise and function-owned. Extend the existing - binding-facing function plan rather than adding a second function plan or a - backend dispatcher table. The bridge continues to lower native call slots - and result storage mechanically; it does not decide GIL or Python exception - policy. -- Argument parsing and conversion, Python result construction, status/message - conversion, exception creation, writeback, and Python-owned cleanup always - run with the GIL held. For the default policy, release the GIL immediately - before the bridge call and reacquire it immediately after that call. For - `hold_gil=True`, emit no release region. -- Perform status evaluation and raise the selected Python exception only after - the GIL has been reacquired. Validate before emission that every status and - message projection names an existing native result slot with a compatible - completed handoff. -- Use directly named lowering methods that follow the existing visible naming - rule. Do not infer runtime policy from result types, function names, emitted - locals, or the presence of status-like native arguments. - -The completed legacy audit found one binding-owned envelope in both oracle -builds. The legacy binding parsed and converted Python inputs with the GIL -held, emitted `Py_BEGIN_ALLOW_THREADS` immediately before the bridge call and -`Py_END_ALLOW_THREADS` immediately after it by default, and omitted both -macros for `@hold_gil`. Only after reacquiring the GIL did it convert hidden -status/message outputs, compare status with `success`, construct -`RuntimeError`, suppress those policy outputs from the declared Python result, -and decref converted result objects on both the failure and success paths. A -missing or incompatible status/message name was previously rediscovered from -raw decorator dictionaries and result datatypes in `ir2ast` and the legacy C -binding; Phase 2D moved that decision to typed post-IR completion and left the -legacy route as a dispatch consumer for rollback parity. - -The direct plan route now preserves that ordering with explicit released-call -and held-call lowering methods. Its fixed native message handoff is -bridge-owned null-terminated storage that the binding converts and frees after -the GIL is reacquired. Generated symbol spelling differs from the legacy -artifacts, but the same source and edited-`.pyi` concurrency, exception, -cleanup, and runtime assertions pass. Production cutover also reused the -existing rendered-artifact build path for inferred native module include -directories, native library directories, `.pyi` manifests, verbose timing, -and scalar external explicit interfaces; no legacy retry was added. - -- [x] Audit and record the exact legacy GIL release/hold region, status/message - projection, exception construction, result suppression, cleanup, and failure - behavior from both existing runtime-policy tests. -- [x] Complete the native status error decision in post-IR policy before - planning; retain the already completed `hold_gil` fact as its single source - of truth. -- [x] Extend the concise function plan with only the binding-facing runtime - facts needed for GIL and status-error lowering, and validate all referenced - native result slots before either backend emits source. -- [x] Add direct binding lowering for the released-call and held-call envelopes - plus post-call status projection. Keep the bridge call and result-slot - lowering on their existing paths. -- [x] Replay the semantic-`.pyi` runtime policy through the production wrapper - plan, with focused concurrency, exception, artifact, cleanup, and generated-C - assertions under `tests/fortran/error_handling/`. -- [x] Run `test_recursive_native_runtime_calls` through the wrapper-plan route - as the scalar recursion regression; leave OpenMP and callbacks in their - later lanes. -- [x] After dual-route parity passes, remove the blanket Phase 2D production - deferral. Send each whole generation unit through `wrapper-plan` only after - its feature lanes are complete; do not add fallback or per-function mixed - routing. -- [x] Move the eligible scalar matrix rows from `dual-route` or `legacy` to - `wrapper-plan`, update the live route counts, and prove their default builds - no longer invoke `semantic_ir_to_codegen_ast()`. -- [x] Finish this phase only when the production `wrapper-plan` count is - nonzero and the already completed scalar baseline no longer depends on the - legacy route outside deliberate rollback diagnostics. - -## Phase 2E — Scalar Boundary Completion and Test Isolation — Complete - -Complete the scalar public boundary before stopping this migration lane. This -phase does not begin strings or arrays. It separates scalar evidence from -mixed generation units so whole-unit routing cannot hide whether one scalar -policy is implemented. - -Scope: every supported primitive scalar kind; ordinary Python scalar values; -`Addr(Arg(i))` call-local address projection; projected scalar copy-in/copy-out; -caller-owned rank-zero NumPy storage spelled `T[()]`; caller-supplied integer -raw addresses spelled `Addr(T)`; and visible or hidden scalar `in`, `out`, and -`inout` behavior. For a mixed native fixture whose declarations cannot be -safely sliced, add a small distinctly named scalar-only native test routine -that preserves the policy decision under test. - -The boundary contract remains: - -- `T` accepts a Python/NumPy scalar value. When native code only reads it, the - wrapper converts into call-local storage. When native code writes through an - address projection and the contract projects `Returns["name", T]`, the - wrapper performs copy-in, native mutation, and copy-out to a replacement - Python scalar; the caller's immutable scalar object is not mutated. -- `T[()]` accepts a rank-zero NumPy array with exactly the declared dtype. The - wrapper validates caller storage and passes its data address; native `out` or - `inout` mutation remains visible in that same array and the Python call - returns `None` unless the contract declares another result. -- `Addr(T)` accepts an integer address such as `array.ctypes.data`. The wrapper - converts it to a raw pointer and forwards that same address without copying - or owning the pointee. Mutation is therefore observed through caller-owned - storage. -- `@native_call(...)` controls only native slot order and value/address/result - projection. It does not change which Python representation (`T`, `T[()]`, or - `Addr(T)`) the declared argument accepts. -- Use one necessary-copy rule. For interoperable scalar replacement, the - binding's converted C scalar is the copy-in storage and the bridge passes - that same storage directly to the native routine; after mutation the binding - converts it once to the Python replacement. `c_f_pointer` association for - `T[()]` or `Addr(T)` is not a data copy. A bridge-local data copy is allowed - only when the native representation actually changes, such as descriptor, - string-buffer, or ownership-snapshot construction. -- Enforce that rule with a completed `BridgeDataAction` on every argument, - result, and native-call output slot. `DIRECT_TRANSFER` reuses boundary - storage, `ASSOCIATE_VIEW` may create only a non-owning native view, - `COPY_REPRESENTATION` is the sole bridge data-copy permission and requires a - non-empty policy reason, and `BLOCKED` keeps the whole generation unit off - the plan route. A non-copying action carrying a copy reason is also invalid. - New array, string, or object support must complete this fact before route - eligibility is widened. - -Excluded: simultaneous multiple-result tuple assembly; rank-positive arrays; -strings including fixed status buffers except for already completed Phase 2D -status projection; derived types; callbacks; and any compatibility fallback to -the legacy generator. - -- [x] Record scalar-only tests separately from mixed array/string/derived - generation units in the route ledger; use copied minimal native routines - when fixture declarations are coupled. -- [x] Cover every primitive scalar kind exercised by the scalar runtime suite - through the direct registry and both binding/bridge generators. -- [x] Add direct named binding and bridge lowering for rank-zero numeric/logical - storage using the completed `SCALAR_STORAGE` and `PASS_STORAGE_ADDRESS` - decisions; validate dtype, rank zero, and writability before the native call. -- [x] Add direct named binding and bridge lowering for primitive raw addresses - using the completed `RAW_ADDRESS` and `PASS_RAW_ADDRESS` decisions; accept an - integer address and forward it without copy or ownership inference. -- [x] Prove isolated scalar input, hidden output, copy-in/copy-out `inout`, - caller-storage `out`/`inout`, and raw-address `out`/`inout` behavior through - compiled legacy/direct-plan parity where applicable. -- [x] Prove scalar copy-in/copy-out reuses one binding local and does not add a - redundant bridge-local value copy. -- [x] Prove plan validation rejects an unexplained bridge copy, a copy reason - on a non-copying path, and any still-blocked bridge data action. -- [x] Prove isolated `@native_call` argument mapping, including `Addr(Arg(i))` - and hidden `Return(...)` slots, without arrays determining route selection. -- [x] Move only proven scalar-only nodes to `wrapper-plan`, update collected - route counts, and leave the original mixed integration nodes on their real - datatype blockers. - -## Phase 2F — Multiple Scalar Result Assembly — Complete - -This is result aggregation, not another scalar boundary representation. The -first isolated oracle is the `with_scalar` policy from -`test_output_arguments.py`: one direct primitive scalar function return plus -one hidden primitive scalar output, assembled into a Python tuple in declared -result order. Keep it separate from arrays, strings, derived types, and native -handles before widening the plan route. - -For source-derived contracts, an ordinary non-descriptor `intent(out)` scalar -hidden by Python result projection still selects `PASS_CALL_LOCAL_ADDRESS` -even when no edited `.pyi` `Addr(...)` spelling exists. The hidden-result -projection is itself the completed semantic fact that requires writable -call-local native storage; the binding and bridge must not rediscover that ABI -rule. Rank-zero allocatable/pointer descriptor outputs retain the distinct -Phase 7H descriptor transport and are not rewritten as ordinary addresses. - -The completed representation is an ordered `FunctionWrapperPolicy.results` -tuple and an ordered `FunctionPlan.results` tuple. Each Python-visible result -has its own `ResultPolicy` and `ResultPlan`, including its binding consumer and -`result_position`. A direct native function return has -`source_kind="direct_return"` and no native-call slot. A hidden output has -`source_kind="hidden_output"` and references the exact same mutable -`NativeCallSlotPlan` stored in `FunctionPlan.native_call_slots`. The bridge -uses the sole direct result, when present, to select its function result and -passes every hidden result through its completed output-address slot. It does -not assemble Python results. - -After the native call, the binding converts each result from its completed -source role exactly once. One result is returned directly; two or more are -assembled into a Python tuple in ascending `result_position`. Tuple allocation, -reference transfer, and failure cleanup are binding-local emission details, -not semantic policy. Before either backend emits source, validation requires -result positions to cover `0..N-1` exactly once, at most one direct result, -every hidden result to share its function native-call slot, and every -non-status native output slot to have exactly one binding result consumer. -Phase 2F does not combine these consumers with projected argument writeback; -that broader aggregation remains blocked until it receives its own completed -policy. - -- [x] Add a scalar-only copied native routine and contract for a direct return - plus hidden scalar `Return(...)` slot. -- [x] Represent every Python result as an explicit binding consumer while - preserving the bridge's direct-return and output-address ABI roles. -- [x] Validate contiguous result positions and reject unclaimed outputs before - either backend emits source. -- [x] Prove compiled legacy/direct-plan parity, then update the route counts. - -## Phase 5 — Strings - -Scope: non-descriptor scalar character values, fixed-length strings, assumed- -length call inputs, immutable replacement, mutable rank-zero byte storage, and -raw fixed-length character addresses. Character arrays remain in Phases 6 and -7; allocatable or pointer scalar character values remain in Phase 7; character -fields remain in Phases 8 and 9; character callbacks remain in Phase 10. - -The legacy wrapper is the behavioral oracle for this phase. In particular, -`CPythonBindingGenerator._convert_python_string_value_argument()` and -`_convert_python_string_storage_argument()` define Python conversion, -validation, allocation, and writeback behavior, while -`FortranToCBridgeGenerator._build_string_argument()`, -`_build_string_storage_argument()`, `_convert_raw_string_argument()`, and -`_convert_string_result()` define the bridge representation. The public -contract and observable oracle are -`docs/user/reference/fortran-wrapper.md`, `docs/user/guide/data-types.md`, -`docs/user/reference/semantic-pyi-format.md`, -`tests/wrapper/fortran/strings/test_character_arguments.py`, and -`tests/wrapper/fortran/strings/test_character_edge_cases.py`. Direct-plan -lowering may use different temporary names or an equivalent internal C ABI, -but it must preserve the legacy Python behavior, native argument order, -character payload, length, ownership, cleanup, and result projection. - -Strings use the same completed-policy and planning pipeline as the other -rank-zero scalar families: - -```text -ArgumentPolicy -> ArgumentTransferPlan -> NativeCallSlotPlan -ResultPolicy -> ResultPlan -LifecyclePolicy -> LifecycleActionPlan -``` - -Do not add a parallel string plan hierarchy or plan-owned handler names. -Numeric and logical primitive families share registry-backed lowering because -their generated structure is the same. `DatatypeFamily.STRING` dispatches to -its own directly named binding and bridge lowering methods because character -conversion and ABI structure differ. The existing `STRING_VALUE`, -`STRING_STORAGE`, `PASS_CALL_LOCAL_ADDRESS`, `PASS_STORAGE_ADDRESS`, -`PASS_RAW_ADDRESS`, and generic codegen/lifecycle actions remain authoritative; -add a new typed action only if those completed actions cannot identify a real -semantic choice. - -Every string argument plan records the completed fixed positive character -length or the absence of a fixed length. The binding-to-bridge handoff records -both the payload address and encoded payload length when the bridge needs both; -this is an ABI fact in the existing argument transfer, not a new planning -stage. A fixed `String[n]` Python value must encode to exactly `n` bytes. A -plain `String` input carries its runtime UTF-8 byte length. Embedded NUL is -rejected before the native call. The bridge may copy bytes into Fortran -character storage only when `BridgeDataAction.COPY_REPRESENTATION` and its -non-empty policy reason were completed before planning. - -The phase is split into the following dependency-ordered sub-lanes. - -### Phase 5A — Required Read-Only String Values - -Included: required rank-zero `String[n]` and `String` Python `str` inputs; -default character, kind `1`, and `c_char`; fixed-length exact encoded-byte -validation; assumed-length runtime payload size; embedded-NUL rejection; and -primitive scalar or void results already supported by earlier phases. - -Excluded: writable inputs, projected replacement, optional strings, string -results, mutable `String[n][()]` storage, raw `Addr(String[n])`, arrays, -allocatable/deferred results, fields, and callbacks. - -Completed policy must provide `ObjectKind.STRING`, -`PythonBarrierAction.STRING_VALUE`, -`NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS`, -`CodegenAction.CALL_LOCAL_INPUT`, `StorageMode.STACK`, required presence, -`BridgeDataAction.COPY_REPRESENTATION`, and the reason that C UTF-8 bytes are -materialized as Fortran character storage. Planning projects those facts into -the ordinary argument/native-slot records. The C binding method -`_lower_argument_required_string_value()` validates `str`, extracts UTF-8 plus -byte length, rejects embedded NUL, and enforces a fixed length when present. -The Fortran bridge method `_lower_argument_required_string_value()` receives -the payload address and length, associates a byte view, copies it into one -backend-local character temporary, and passes that temporary in the completed -native-call position. - -Validation requires a string-value Python action, call-local-address native -action, character-buffer handoff, one matching payload-length role, required -presence, no projected result, and a justified representation copy. Whole-unit -eligibility widens only for generation units containing this lane plus already -completed scalar/result/runtime lanes. Replay uses the existing -`fstrings_f90` native object and contract package with a reduced entry that -exports only existing read-only scalar string procedures; both routes run the -same fixed/assumed-length, kind, NumPy-string-scalar, wrong-length, and embedded -NUL assertions. The mixed original string nodes remain `legacy` because their -units also contain string results, writable strings, arrays, and allocatables. - -- [x] Complete Phase 5A policy, ordinary plan projection, validation, named C - and Fortran lowering, reduced-entry dual-route runtime parity, support - predicate, and migration-ledger evidence. - -### Phase 5B — Fixed-Length String Results And Hidden Outputs - -Included: direct fixed-length scalar character results and fixed-length hidden -`intent(out)` results, including trailing blanks. The binding receives a -NUL-terminated C-owned copy, converts the full payload to a Python-owned -`str`, and releases the temporary exactly once. The bridge allocates and fills -that copy only through completed `COPY_REPRESENTATION` policy. Deferred-length -and nullable allocatable or pointer results remain in Phase 7 because their -runtime length and allocation state are descriptor lifecycle facts, not -scalar-string conversion facts. - -Both forms reuse the ordinary ordered `ResultPolicy -> ResultPlan` path and -record the fixed positive `character_length` on the result. A direct native -function result has `source_kind="direct_return"`, -`CodegenAction.COPY_OUT`, no native-call slot, and a bridge function result of -`type(c_ptr)`. The bridge first receives the native value in backend-local -`character(kind=c_char, len=n)` storage, then allocates `n + 1` bytes through -the existing `prik_malloc` interface, copies all `n` characters, appends -`c_null_char`, and returns the pointer. A hidden output has -`source_kind="hidden_output"`, `CodegenAction.COPY_OUT`, -`NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS`, and references the exact same -fixed-length `NativeCallSlotPlan` used by the function. Its existing output -slot receives native character storage, then performs the same justified -allocation and copy after the native call. - -In both cases, binding lowering checks for a null allocation, converts the -NUL-terminated UTF-8 payload with the same observable behavior as the legacy -`Py_BuildValue("s", ...)` path, frees the C allocation exactly once even when -Python conversion fails, and returns the Python-owned `str`. Phase 5B supports -exactly one Python-visible string result per function; mixed or multiple -string result aggregation remains blocked until cleanup of every unconverted -native allocation is explicitly planned. A function that combines a public -fixed string result with native status-error projection is blocked for the -same reason: the status failure path must not bypass the string allocation's -planned release. - -Validation requires a fixed positive length, `ObjectKind.STRING`, Python-owned -copy-return ownership, no Python barrier action, the source-appropriate -codegen/native action, `BridgeDataAction.COPY_REPRESENTATION` with the standard -fixed-string copy reason, and matching result/native-slot lengths for hidden -outputs. Direct results must not carry a native-call slot; hidden results must -share their function slot by identity. The C and Fortran backends dispatch -`DatatypeFamily.STRING` to `_lower_result_fixed_string()` methods instead of -the primitive scalar registry. - -Replay direct results from the existing `fstrings_f90` native object with a -reduced contract entry exporting `char_result_default`, -`char_result_c_char`, `string_result_fixed`, `string_result_padded`, and -`string_result_c_char`. Replay the hidden output from the existing -`fcharacter_edges_f90.make_out` unit through another reduced entry. Run the -same trailing-blank and returned-value assertions through legacy and direct -routes. The original mixed nodes remain `legacy` on deferred results, writable -strings, optionality, or arrays. - -- [x] Complete fixed-length direct and hidden string result policy, result-plan - length facts, allocation/failure cleanup, binding conversion, bridge copy, - validation, legacy/direct parity, support widening, and ledger updates. - -### Phase 5C — Immutable String Output And Inout Replacement - -Included: fixed and assumed-length Python `str` output/inout dummies, including -the pass-by-address mutable native call. Python strings remain immutable: the -binding creates mutable call-local storage; the bridge passes that storage to -the native dummy; a declared `Returns["name", String...]` consumer returns a -replacement string; identity form discards native mutation and returns `None`. -Fixed buffers retain their complete post-call contents and trailing blanks; -assumed-length buffers use the encoded input length. Optional omitted, -explicit-`None`, and concrete-value states are handled here after required -replacement works. - -The first Phase 5C slice is required fixed-length `String[n]` only. A projected -replacement consumes completed `ObjectKind.STRING`, Python-owned -`COPY_RETURN`, `PYTHON_REFCOUNT`, stack contract storage, -`PythonBarrierAction.STRING_VALUE`, -`NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS`, -`CodegenAction.COPY_IN_OUT`, native mutation, result projection, and -`BridgeDataAction.COPY_REPRESENTATION` with the fixed-string replacement copy -reason. The ordinary argument plan records those facts plus the fixed positive -character length, and its binding and bridge views both carry the completed -codegen action. The same mutable native-call slot is referenced throughout; -there is no second output slot. - -The binding validates the input exactly as Phase 5A does, allocates one -`n + 1` byte call-local buffer through `prik_malloc`, copies all `n` encoded -bytes, and appends NUL. Allocation failure raises `MemoryError` before native -execution. The bridge receives the mutable buffer and length, materializes -backend-local `character(kind=c_char, len=n)` storage, passes that storage to -the native dummy, then copies the complete post-call value back into the -binding buffer and restores the NUL terminator. After the call, binding -`_lower_writeback_string()` converts the replacement with the same -`Py_BuildValue("s", ...)` behavior as the legacy route and frees the call-local -buffer exactly once whether conversion succeeds or fails. - -The existing ordered lifecycle records remain authoritative: - -```text -COPY_IN (binding allocation and input copy) - -> NATIVE_MUTATION (bridge-local character call and copyback) - -> COPY_OUT (binding Python replacement conversion) - -> CLEANUP (binding call-local buffer release) -``` - -Validation requires the completed ownership/action facts, fixed result -position, one shared payload/length handoff, matching argument and native-slot -lengths/actions/copy reasons, exactly one complete lifecycle phase set, bridge -copyback ownership only for `COPY_IN_OUT`, and binding cleanup after conversion. -A replacement combined with native status-error projection stays blocked until -the status failure path also releases the mutable buffer. Multiple projected -results remain blocked by the existing single-writeback lane. - -A fixed identity contract uses the already-completed `CALL_LOCAL_INPUT` action, -the same call-local bridge character representation, no lifecycle result, and -returns `None`. Native writes affect only that temporary and are deliberately -discarded. Its binding buffer remains the borrowed read-only UTF-8 input because -the bridge never copies mutation back across the boundary. Assumed-length, -optional, mutable `String[n][()]`, and raw-address forms remain excluded from -this first slice. - -Replay both forms from the existing `fcharacter_edges_f90.fixed_inout` native -unit through a reduced edited contract that exports one projected replacement -and one identity spelling bound to the same native symbol. Run the same exact -length, trailing-blank, input-immutability, returned-value, and allocation -failure assertions through legacy and direct routes before widening the -whole-unit support predicate. - -The second Phase 5C slice keeps the same completed ownership, barrier, -representation-copy, and four-phase lifecycle records while removing the -compile-time-length restriction. For required assumed-length `String`, the -binding-recorded UTF-8 byte length is the native character length and the -replacement allocation size. A zero-byte input is valid: replacement owns a -one-byte NUL-only buffer, the bridge materializes a zero-length character -value, and binding returns the empty Python string after releasing the buffer. -Fixed strings still require the exact declared encoded length. - -Optional string values use the completed `OptionalMode.NULLABLE_VALUE`; they -do not invent a descriptor or reinterpret optionality as semantic nullability. -The binding ABI always carries the string payload pointer and runtime byte -length. Omitted and explicit `None` both send a null pointer with length zero, -so the bridge leaves the native optional dummy absent and a projected -replacement returns `None`. A concrete value is validated before native -execution, including embedded-NUL rejection and any fixed-length constraint. -Projected replacement allocates and owns the mutable `length + 1` buffer only -for that concrete value; identity form borrows the read-only payload and -discards native mutation exactly as the required identity path does. - -The bridge tests pointer association to choose the existing optional native -call branch. Only the present branch associates the payload, creates -`character(kind=c_char, len=runtime_length)` call-local storage, and invokes -the native optional dummy. Copyback is likewise guarded by pointer association, -so an absent optional never touches unassociated storage. Concrete projected -replacement restores the NUL terminator after copying all runtime-length -bytes. Binding then returns the concrete replacement and frees its allocation -exactly once, or returns `None` without calling `free` for the absent states. -Status-error combination and multiple projected replacements remain blocked -by the same explicit cleanup exclusions as the fixed required slice. - -Replay `assumed_inout` and `optional_inout` from the existing -`fcharacter_edges_f90` contract through a reduced entry module. Compare legacy -and direct routes for empty and non-empty assumed-length values, omitted, -explicit-`None`, and concrete optional states, input immutability, embedded-NUL -rejection before native execution, and allocator failure only for concrete -projected replacements. - -- [x] Complete fixed required replacement and discarded-identity policy, - writeback lifecycle, named lowering, cleanup, validation, parity, support - widening, and ledger updates. -- [x] Add assumed-length replacement and optional presence only after the fixed - required path is proven; preserve legacy empty-string and omitted/`None` - behavior and reject embedded NUL before native execution. - -### Phase 5D — Mutable Storage And Raw Fixed-Length Addresses - -Included: `String[n][()]` caller-owned rank-zero NumPy bytes storage and -`Addr(String[n])` caller-supplied integer addresses. The storage path validates -rank zero, dtype `S`, native byte order/alignment where applicable, and -writability before aliasing the caller buffer. The raw-address path does not -own or validate the pointee. Both use the declared fixed length; mutable -deferred-length scalar storage remains blocked. - -Both forms complete policy before planning and share no Phase 5C replacement -lifecycle. `String[n][()]` records `ObjectKind.STRING`, caller ownership, -`IN_PLACE`, caller destruction, alias contract/boundary storage, -`PythonBarrierAction.STRING_STORAGE`, -`NativeBarrierAction.PASS_STORAGE_ADDRESS`, `CodegenAction.IN_PLACE_ARGUMENT`, -native mutation, no result projection, and a fixed positive character length. -`Addr(String[n])` records the same caller ownership, in-place transfer, caller -destruction, mutation, and no result projection, but the contract value itself -uses stack storage while `PythonBarrierAction.RAW_ADDRESS` and -`NativeBarrierAction.PASS_RAW_ADDRESS` preserve the unsafe caller-supplied -address. The raw pointee is never adopted, released, sized, or validated by -prik. This corrects the pre-5D raw-string decision that incorrectly retained -immutable-string call-local ownership despite the completed raw-address -barriers. - -The ordinary argument plan carries the fixed length and uses -`ArgumentHandoffMode.OPAQUE_ADDRESS` for both forms. There is one pointer ABI -field and no runtime length field: the fixed character length comes only from -the completed plan. The binding storage handler accepts exactly a rank-zero -NumPy `NPY_STRING` array whose itemsize is `n`, requires alignment and -writability, and forwards `PyArray_DATA` without allocating or copying. -The raw handler accepts an integer and uses the existing `PyLong_AsVoidPtr` -path; it deliberately does not inspect the pointee, its allocation extent, or -its lifetime. - -The native character scalar is not directly C interoperable, so both forms -record `BridgeDataAction.COPY_REPRESENTATION` with a boundary-specific reason. -The bridge associates the incoming address with exactly `n` -`character(kind=c_char)` bytes, copies them into backend-local -`character(kind=c_char, len=n)` storage, invokes the native dummy, and copies -all `n` post-call bytes back. It does not append NUL, allocate, free, infer -ownership, or create a Python result. These helper locals are emitted-code -details selected by the completed storage/raw policy. - -Optional storage/address arguments and projected returns remain blocked in -this phase. `String[()]` and `Addr(String)` are rejected by the semantic `.pyi` -contract because the bridge has no fixed extent; arrays and callback storage -remain owned by their later lanes. Validation rejects edited plans with a -missing/nonpositive length, the wrong owner/transfer/destruction/storage mode, -an inconsistent barrier or handoff, a runtime length role, an unjustified copy -reason, missing mutation, or result projection before either backend lowers. - -Replay `fixed_inout_storage` and `fixed_inout_raw` from the existing -`fnative_call_examples_f90` edited contract through a reduced entry bound to -the same `fixed_inout` native routine. Compare legacy and direct routes for -complete eight-byte mutation, rank/dtype/itemsize/writability failures, raw -integer type rejection, and lack of Python return. Keep the existing mixed -native-order test on the legacy route because its array and derived-type -neighbors belong to later phases; add the reduced replay as a separate -wrapper-plan ledger node. - -- [x] Complete mutable string-storage and raw-address policy, address handoff, - bridge association/copyback mechanics, validation, legacy/direct parity, - support widening, and ledger updates. - -### Phase 5 Completion - -Descriptor-backed scalar character values are deliberately outside this -phase. A contract such as `String | None` with -`result=Allocatable(Return(...))` carries allocation state, runtime element -length, descriptor ownership, and native release responsibility. It must enter -the direct route only through Phase 7's shared allocatable/pointer descriptor -plan; it remains a rank-zero Python `str | None` result rather than a native -array handle. Phase 5 must not add a character-only descriptor ABI or cleanup -path. - -- [x] Expand the phase under the mandatory expansion gate from the live - policies, legacy binding/bridge implementation, public string contract, and - focused wrapper tests. -- [x] Validate fixed/runtime length sources, payload/length role agreement, - result ownership, writeback consumers, and cleanup responsibility before - either backend emits source. -- [x] Keep string behavior in directly named string lowering methods while - reusing the ordinary scalar planning records and lifecycle flow. -- [x] Finish Phase 5 only when all non-descriptor scalar string sub-lanes are - proven and every affected matrix row is either `wrapper-plan` or blocked by - a later array, descriptor, field, or callback lane recorded in the ledger. - -## Phase 6 — Ordinary Arrays - -Scope: NumPy data-buffer arrays that do not require native descriptor handles. - -The ordinary-array lane borrows or copies NumPy data buffers; it never creates -or consumes a persistent native descriptor handle. `Allocatable[T[...]]`, -`Pointer[T[...]]`, rank-zero allocatable/pointer scalars, and a native handle -used as the actual value for an ordinary array dummy all remain in Phase 7. -Caller-supplied `Addr(T[...])` storage is the distinct Phase 6G follow-up and -must complete before Phase 7. -Derived-type arrays remain in Phase 8, fields in Phases 8 and 9, and callback -arrays in Phase 10. The full BLAS/LAPACK generation unit remains deferred until -final cutover even when individual ordinary-array shapes become supported. - -Whole-generation-unit rollout preserves that Phase 7 boundary. Output-only -ordinary array results and hidden outputs may select the production plan route -now. A generation unit with an ordinary array actual remains on the legacy -route, even after its NumPy-buffer path has direct-route parity, because route -selection cannot know whether a caller will pass a NumPy array or a supported -native descriptor handle. Those reduced array-actual rows remain `dual-route` -with the native-handle caller contract recorded as their sole Phase 7 blocker; -the direct route is forced only by the internal parity harness. - -The public behavior is defined by the NumPy array contract in -`docs/user/reference/fortran-wrapper.md`, the array spelling and metadata rules in -`docs/user/reference/semantic-pyi-format.md`, and the existing array wrapper -tests. The legacy binding validates exact dtype, rank, every expressible -extent, native byte order, alignment, layout/stride requirements, and -writeability for mutable storage before the native call. It does not cast, -byte-swap, repair alignment, de-alias overlapping storage, or silently copy a -rejected layout. Read-only source `intent(in)` storage may remain read-only; -edited `.pyi` array storage is writable unless a completed policy says -otherwise. Zero-sized dimensions are valid when the rest of the contract is -valid. - -Every ordinary array remains in the existing -`ArgumentPolicy -> ArgumentTransferPlan -> NativeCallSlotPlan` or -`ResultPolicy -> ResultPlan` flow. An argument embeds one editable array -handoff spec containing element family, concrete or runtime rank, declared -shape expressions, axis modes, order, contiguity, itemsize when relevant, -writeability, and the exact ABI roles for data, extents, upper bounds, strides, -runtime rank, or itemsize. Policy completion selects -`PythonBarrierAction.ARRAY_STORAGE`, -`NativeBarrierAction.PASS_ARRAY_BUFFER`, and either -`BridgeDataAction.ASSOCIATE_VIEW` for caller storage or an explicit copy action -and reason. Planning must not reconstruct any of those choices from rank, -shape spelling, or datatype. The binding and bridge dispatch only to directly -named array implementation methods selected by those completed facts. - -The binding checks the NumPy object before extracting `PyArray_DATA`, shape, -and element strides. The bridge receives only the fields named by the handoff -spec, associates the pointer with the completed element type and extents, and -constructs a stride slice only when the plan explicitly allows it. C-oriented -flat storage reverses bridge association extents only when the completed order -requires it. Backend-local pointer views and slice expressions are emitted-code -details; dtype, rank, extent, order, stride acceptance, mutation, projection, -copy, and ownership are semantic policy. - -The phase is dependency-ordered as follows. - -### Phase 6A — Required Rank-One Contiguous Buffers - -Included: required concrete-rank-one ordinary arrays with dense contiguous -axes (`T[:]`) for the existing bool, integer, real, and complex primitive -families; caller-owned borrowed/in-place storage; scalar or void neighbors and -results already supported by earlier phases; native position reordering; and -zero-length buffers. The binding requires an exact NumPy dtype, rank one, -native byte order, alignment, contiguity, and writeability only when completed -ownership says native code mutates the storage. It forwards the data address -and runtime extent. The bridge creates one typed rank-one pointer view with -`c_f_pointer` and passes that view in the completed native-call position. - -This slice records `ArgumentHandoffMode.ARRAY_BUFFER` and -`BridgeDataAction.ASSOCIATE_VIEW`; it performs no allocation, element copy, -writeback action, release, or Python result projection. Explicit/fixed extent -expressions, `Flat`, multidimensional order, strided axes, optionality, -projected output identity, array results, character arrays, assumed rank, and -native-handle actuals remain in later sub-lanes. Replay one existing -`fmath_arrays_f90` contiguous routine through a reduced semantic `.pyi` entry -and compare legacy/direct behavior for mutation, dtype, rank, alignment, -byte-order, contiguity, writeability, zero length, and native argument order. - -- [x] Complete required rank-one contiguous array policy, editable handoff - spec, validation, named C/Fortran lowering, reduced legacy/direct parity, - support widening, and ledger evidence. - -Boolean arrays retain an exact one-byte NumPy boundary independently of native -language spelling. Semantic IR records compiler-measured native storage as -`Bool8`, `Bool16`, `Bool32`, or `Bool64`. Post-IR wrapper policy must distinguish -an exact `c_bool` view from an exact-kind representation copy and must record -copy-in and copy-out directions before planning. For copied arrays the bridge -owns the exact-kind temporary; copy-out both converts truth values and writes -canonical zero/one boundary bytes in one traversal. Binding code continues to -validate and forward only `NPY_BOOL` storage and must not infer native logical -kind from a semantic name. - -### Phase 6B — Declared Extents, Flat Storage, And Dense Rank - -Included: fixed and visible-symbol extent expressions, lower-bound-derived -extents, assumed-size `Flat`, ranks two through fifteen, `ORDER_F` and -`ORDER_C`, dense contiguous layout, and zero-sized axes. Shape expressions are -resolved against existing scalar handoff roles before backend emission; the -bridge association order follows the completed layout. Any expression that -cannot be represented by available roles remains blocked rather than being -recomputed in a backend. - -Declaration-expression normalization is shared across module variables, -derived fields, dummy arguments, and results. Generated `.pyi` uses Python -array properties (`a.size`, `a.shape[i]`, and `a.ndim`), while the completed -plan carries role-bound expressions that each backend only renders. - -Order is an exact-storage selector, not an implicit conversion selector. -`ORDER_F` preserves logical axes over Fortran-contiguous storage. `ORDER_C` -passes the original C-contiguous address and reverses bridge extents, so native -Fortran observes the transposed storage view. Preserving the same logical axes -while accepting the opposite layout uses explicit `COPY_F` metadata, never an -inference from order. The owning `ArgumentTransferPlan` records C source order, -F native order, copy-in, conditional copy-out, original-object projection, and -temporary cleanup. The binding performs both copy directions and owns the -NumPy temporary. The bridge receives the temporary through the unchanged -ORDER_F association path and performs neither half of this representation -conversion. - -The initial `COPY_F` lane includes required, concrete-rank, dense numeric -ndarray arguments. It excludes `Flat`, assumed-rank, strided, optional and -character arrays, native descriptor arguments, and handle actuals until each -has separate policy and parity evidence. - -`Flat` is one axis marker and never collapses a multidimensional plan: -`T[:, Flat]` remains rank two in Fortran order, while -`Annotated[T[Flat, :], ORDER_C]` is its C-order orientation. The bridge reverses -only the C-order association extents. For an external assumed-size interface, -an explicit prefix such as `T[3, Flat]` may lower to `a(3, *)`; a runtime-only -prefix uses the standards-valid sequence-associated `a(*)` declaration while -the bridge retains every runtime extent and the completed logical rank. - -- [x] Complete declared-shape evaluation, flat-storage orientation, - multidimensional dense handoff, validation, parity, and ledger evidence. -- [x] Complete explicit C-to-Fortran representation copies through `COPY_F`, - including native-input and inout calls through the same binding-owned copy - lifecycle, projected original identity, temporary cleanup, direct bridge - reuse, validation, and compiled parity. Native `intent` remains owned by the - called procedure and is not duplicated in the semantic `.pyi` or bridge - temporary. - -### Phase 6C — Positive-Strided Ordinary Views - -Included: `::` axes and bounded stride-aware axes, runtime upper bounds and -element strides, Fortran-oriented positive-stride slicing, contiguous views as -a valid special case, and degenerate zero-size strides. Negative, zero on an -addressable axis, incompatible C-oriented, broadcast, and otherwise invalid -layouts fail before the native call. No copy-to-contiguous fallback is inferred. - -- [x] Complete stride roles, upper bounds, positive-stride bridge slices, - layout validation, parity, and ledger evidence. - -### Phase 6D — Output Storage And Projected Identity - -Included: ordinary `intent(out)`/`intent(inout)` caller buffers and -`Returns["name", T[...]]` projections. Native code mutates the same validated -NumPy storage; the binding returns the original Python array object with one -owned reference rather than constructing a second array or copying elements. -Read-only output storage fails before the call. Multiple projections compose -with the existing ordered result aggregation only after every projected array -identity and failure-path reference is planned. - -- [x] Complete in-place output ownership, projected identity/reference - lifecycle, multiple-result aggregation, parity, and ledger evidence. - -### Phase 6E — Ordinary Array Results And Hidden Outputs - -Included: non-allocatable direct array results and hidden output arrays whose -shape and element ownership are fully expressible without persistent native -descriptors. The plan records the producer, every runtime extent, allocation -owner, copy or transfer action, Python NumPy construction, and release on -success and every failure path. Nullable allocatable/pointer results remain in -Phase 7. - -- [x] Complete ordinary result/hidden-output allocation, shape projection, - copy ownership, cleanup, parity, and ledger evidence. - -### Phase 6F — Optional, Assumed-Rank, And Character Buffers - -Included: ordinary optional NumPy arrays, numeric assumed-rank dispatch from -one through fifteen, and fixed-width NumPy bytes character arrays with planned -itemsize. Omitted ordinary optional arrays remain distinct from present -storage. Assumed-rank plans carry a runtime-rank role and validate the supported -range before bridge dispatch. Character arrays use exact `NPY_STRING` itemsize -and remain raw fixed-width bytes; deferred descriptor-backed character values -remain in Phase 7. Fixed-shape character array direct results and hidden -outputs reuse the Phase 6E copy-result path with their itemsize included in -NumPy dtype construction and bridge byte-count calculation. - -- [x] Complete optional presence, assumed-rank dispatch, character itemsize, - validation, parity, and ledger evidence. - -### Phase 6A-F Ordinary-Buffer Completion - -- [x] Expand the phase under the mandatory expansion gate from live semantic - array contracts, legacy binding/bridge lowering, public docs, and focused - wrapper tests. -- [x] Define array handoff specs for every supported data, rank, shape, stride, - order, itemsize, writeability, result, and lifecycle role. -- [x] Validate every completed array policy and handoff role before either - backend emits source. -- [x] Finish Phases 6A-F only when every ordinary-array buffer matrix row is - migrated or remains blocked solely by an explicitly later descriptor, - derived, field, callback, or deferred-real-library lane. - -### Phase 6G — Raw Array Addresses — Complete - -Implementation status: complete. Required raw array addresses now use the -shared completed policy, `ArgumentTransferPlan`, native slot, centralized -validation, and named binding/bridge lowering paths. The dependency-closed -numeric and fixed-character runtime rows have passed compiled legacy/direct -parity and moved to `wrapper-plan`. - -Scope: required Python-visible type-level raw-address array arguments such as -`Addr(Float64[n])`. The caller supplies one Python integer address, prik -forwards it as one opaque C address, and the bridge associates a typed native -array view using rank, shape, element type, and orientation facts completed -before `ir2ast.py`. There is no NumPy object, runtime handle, persistent native -descriptor, data copy, ownership transfer, or automatic release. - -This lane follows Phase 6 because its semantic object kind is -`ObjectKind.NUMPY_ARRAY` and its pointee layout reuses the array shape record. -It remains a distinct transport from an ordinary array buffer. The fixed -dispatch algorithm is: - -1. match `ObjectKind.NUMPY_ARRAY`; -2. match the completed Python barrier action; -3. lower `ARRAY_STORAGE` through the Phase 6A-F buffer path or `RAW_ADDRESS` - through Phase 6G; -4. require the matching native action, handoff mode, bridge data action, and - array-shape facts; and -5. fail validation rather than substituting the other transport. - -The same algorithm already separates scalar and string value, storage, and -raw-address forms. Phase 6G must extend that system; it must not add a parallel -raw-pointer planner, a datatype-based backend branch, or a special function or -module plan. - -#### Public Contract And Explicit Non-Scope - -The maintained public contract is already documented in -`docs/user/reference/semantic-pyi-format.md` and -`docs/user/guide/data-types.md`. Preserve it exactly: - -- `Addr(T[d1, ..., dr])` is depth one and has positive rank; -- the pointee dtype is primitive; -- every extent expression is resolved from literals and visible scalar - arguments or visible rank-zero scalar storage; -- the integer carries no dtype, rank, shape, order, alignment, bounds, - ownership, or lifetime metadata; -- prik cannot prove that the supplied address actually points to compatible, - sufficiently large, live storage; and -- edited semantic `.pyi` raw-address storage is mutable caller storage unless - a completed policy explicitly says otherwise. - -The initial compiled oracle is `Addr(Float64[n])`. Before declaring the lane -complete, audit every public primitive family already accepted by semantic -policy, including bool, integer, real, complex, and fixed-width character -array pointees. Add compiled coverage for a family only when an existing native -routine can prove it without broadening the public contract. A fixed scalar -`Addr(String[n])` remains the completed Phase 5D string path; a rank-positive -`Addr(String[k][n, ...])` is an array path and must carry both the fixed element -length and the resolved array shape. - -Explicitly excluded from this lane are: - -- scalar `Addr(T)`, already completed in Phase 2E; -- fixed scalar `Addr(String[n])`, already completed in Phase 5D; -- NumPy `T[...]` storage, already completed in Phases 6A-F; -- unresolved or assumed shapes such as `Addr(Float64[:])`, assumed rank, - assumed size, and stride-marker shapes; -- optional, nullable, projected, direct-result, and hidden-output raw addresses - unless a separate public-contract audit first proves their intended Python - ownership and absence/result behavior; -- wrapped/derived pointees, pointer graphs deeper than one, and callbacks; -- `Allocatable[T[...]]`, `Pointer[T[...]]`, runtime native handles, and C - descriptors, which belong to Phase 7; and -- any implicit conversion from an ndarray or runtime handle to its address. - -#### One Action Vocabulary, Three Array Transports - -| Contract | Object kind | Python action | Native action | Handoff mode | Bridge data action | -| --- | --- | --- | --- | --- | --- | -| NumPy `T[...]` | `NUMPY_ARRAY` | `ARRAY_STORAGE` | `PASS_ARRAY_BUFFER` | `ARRAY_BUFFER` | `ASSOCIATE_VIEW` | -| Raw `Addr(T[...])` | `NUMPY_ARRAY` | `RAW_ADDRESS` | `PASS_RAW_ADDRESS` | `OPAQUE_ADDRESS` | `ASSOCIATE_VIEW` | -| Native descriptor contract | completed handle kind | completed handle action | `PASS_NATIVE_DESCRIPTOR` | Phase 7 descriptor mode | completed Phase 7 action | - -`ASSOCIATE_VIEW` means the bridge creates a typed, non-owning view; it does not -mean that the Python binding extracted a NumPy buffer. The Python and native -barrier actions remain the authoritative distinction. Do not introduce names -such as `PASS_RAW_ARRAY`, `COPY_RAW_ARRAY`, or datatype-specific address -actions. - -The completed ownership/action tuple for the required mutable public form is: - -- `OwnershipOwner.CALLER`; -- `TransferMode.IN_PLACE`; -- `DestructionPolicy.CALLER`; -- `StorageMode.STACK` for the call-local pointer carrier, not for the pointee; -- `CodegenAction.IN_PLACE_ARGUMENT`; -- `PythonBarrierAction.RAW_ADDRESS`; -- `NativeBarrierAction.PASS_RAW_ADDRESS`; -- `ArgumentHandoffMode.OPAQUE_ADDRESS`; and -- `BridgeDataAction.ASSOCIATE_VIEW` with no copy reason. - -If a retained source-derived contract can be read-only, policy may instead -complete `CALL_LOCAL` / `CALL_LOCAL_INPUT` / `NONE` destruction. Both -backends must consume that completed tuple; neither may infer mutability from -the pointee type or raw-address spelling. A raw array never has copy-in, -copy-out, projected-identity, allocation, destruction, release, or lifecycle -actions in this lane. - -#### Required Policy And Plan Shape - -Keep the feature under the existing `ArgumentTransferPlan`: - -```text -ArgumentTransferPlan - object_kind = NUMPY_ARRAY - binding.python_action = RAW_ADDRESS - bridge.native_action = PASS_RAW_ADDRESS - bridge.handoff_mode = OPAQUE_ADDRESS - bridge.data_action = ASSOCIATE_VIEW - array = ArrayHandoffPlan - native_call_slot = the same referenced NativeCallSlotPlan -``` - -Do not add `RawArrayPlan`, a second native slot, or a raw-address lifecycle -owner. Generalize the existing completed `ArrayHandoffPolicy` and -`ArrayHandoffPlan` only enough to carry raw pointee layout: - -- concrete rank and one shape expression per axis; -- one `data_role` equal to the binding/bridge/native-slot address role; -- `extent_reference_roles` naming the existing visible scalar handoff roles - used by each shape expression; -- the completed orientation used for native pointer association; -- fixed character element length/itemsize when the pointee family is string; - and -- no binding-extracted runtime rank, extent, upper-bound, stride, or itemsize - ABI roles. - -For a raw address, the shape record describes the pointee view; it does not -describe fields packed by the binding. A visible `n` used by -`Addr(Float64[n])` already has its own `ArgumentTransferPlan` and native-call -slot. Reference that role rather than passing a duplicate array extent. A -literal extent requires no extra ABI field. The bridge resolves the shape -expression from those planned native role names. - -Post-IR policy completion must explicitly select multidimensional orientation -before planning. Preserve the current legacy interpretation, including its -default orientation, only after capturing a rank-two artifact/runtime oracle. -Do not leave `ir2ast.py`, a codegen-model `order` default, or the bridge's local -shape reversal to make that decision. - -#### Completed Direct-Plan Seams - -The implementation split completed array policy by Python barrier action, -selected `OPAQUE_ADDRESS` and `ASSOCIATE_VIEW` before lowering, projected raw -pointee layout into the shared array record, omitted packed NumPy-buffer roles, -and added named raw-address checks and association methods to both backends. -Ordinary-array buffer checks remain unchanged and fail closed; neither backend -substitutes one transport for another. - -#### Dependency-Ordered Implementation Slices - -##### Phase 6G1 — Complete Raw Array Policy - -- [x] Make the `NUMPY_ARRAY` boundary validator dispatch on - `PythonBarrierAction` and add a named raw-address branch with the exact - ownership/action tuple above. -- [x] Complete raw pointee rank, shape expressions and their visible-scalar - dependencies, primitive family, fixed character element length, and - orientation before `ir2ast.py`. -- [x] Complete `OPAQUE_ADDRESS` and `ASSOCIATE_VIEW` from the action pair; do - not infer either in a backend. -- [x] Keep unresolved dimensions, unsupported pointee families, optionality, - projection, nullability, and deeper pointer graphs blocked with owner-path - diagnostics. -- [x] Freeze current behavior for zero/negative extent expressions, zero or - negative integer addresses, and integer overflow against the public docs and - legacy conversion before changing any rule. If a rule changes, change it in - policy and public docs, not in one backend. - -Audit result: resolved zero and negative extent expressions remain accepted -without a positivity check; integer zero becomes a null pointer without a -conversion error; negative integers follow `PyLong_AsVoidPtr`; and pointer-size -overflow raises `OverflowError`. Public documentation now states that these are -unsafe caller responsibilities, and tests prove the conversion guard and -generated shape without dereferencing an invalid address. - -##### Phase 6G2 — Project And Validate The Shared Plan - -- [x] Populate the existing `ArgumentTransferPlan.array` and its shared - `NativeCallSlotPlan.array` with one identical raw pointee layout record. -- [x] Reuse the scalar/string address handoff role and - `ArgumentHandoffMode.OPAQUE_ADDRESS`; add no raw-array ABI action. -- [x] Resolve every shape symbol to an existing visible scalar role and reject - unavailable, cyclic, hidden, non-scalar, or result-only dependencies before - lowering. -- [x] Split central array diagnostics by the completed Python action so buffer - validation still requires packed extent/layout roles while raw validation - forbids them. -- [x] Add editable-plan tests that independently corrupt object kind, Python - action, native action, handoff mode, bridge data action, rank, shape, - reference roles, element family, character length, orientation, and native - slot identity. - -##### Phase 6G3 — Reuse Binding Raw-Address Extraction - -- [x] Reuse `_lower_argument_required_raw_address()` for the Python integer - check and `PyLong_AsVoidPtr` conversion. Scalar, string, and array raw - addresses should share this extraction code. -- [x] Emit one `void *` handoff value and no `PyArray_*`, dtype, rank, shape, - layout, writeability, or itemsize checks. -- [x] Keep object-kind-specific logic out of the conversion method; array - shape affects only validation, the bridge view, and native call. -- [x] Preserve the existing conversion rule under which integer zero produces - a null pointer without itself raising a Python conversion error. Prove that - rule without dereferencing the null pointer; runtime tests must never call - native code with an invalid test address. - -##### Phase 6G4 — Add Named Raw Array Bridge Association - -- [x] Add directly named raw-array declaration and association methods in the - array method group. Dispatch to them only for - `NUMPY_ARRAY` / `RAW_ADDRESS` / `PASS_RAW_ADDRESS` / - `OPAQUE_ADDRESS` / `ASSOCIATE_VIEW`. -- [x] Declare one `type(c_ptr), value` bridge parameter and one backend-local - typed pointer view. The local view is an emitted-code helper, not a new plan - owner. -- [x] Associate the view with `c_f_pointer` using only the planned shape and - orientation, then pass that view in the existing native-call slot position. -- [x] Preserve fixed character element length when the pointee is a character - array. Do not pass a runtime itemsize unless a future public contract - explicitly requires one. -- [x] Emit no copy, writeback, allocation, release, descriptor, or NumPy - mechanics. - -##### Phase 6G5 — Prove The Route Before Widening It - -- [x] Retain semantic conversion coverage in - `tests/fortran/semantic_pyi_format/semantics/test_calls_and_projections.py` - for round-trip, - visible extent sources, primitive pointees, and rejection of unresolved or - wrapped forms. -- [x] Add focused completed-policy tests for every authoritative action and - blocker, plus `array-raw-address-inputs` support classification. -- [x] Add `tests/fortran/raw_addresses/codegen/test_raw_array_lowering.py` for plan - shape, edits, validation, C nodes, Fortran nodes, native order, and the - absence of buffer/descriptor/lifecycle nodes. -- [x] Extract `fill_vector_raw` from - `test_editable_contract_can_use_native_order_arguments_without_native_call` - into a reduced legacy/direct-plan parity test. Cover mutation through a valid - `raw_vector.ctypes.data`, ndarray rejection, wrong Python types, a visible - rank-zero scalar extent, and the established native argument order. -- [x] Prove raw-array native argument reordering in the direct-plan generated - call test. The legacy AST route retains only a projection marker and is not - an oracle for reordered projection-slot lowering. -- [x] Add literal and arithmetic extent-role cases. Add a rank-two runtime - parity case before freezing default/explicit orientation. Add a fixed-width - character-array case if the public family audit retains that contract. -- [x] Keep the broad native-order test `legacy` until its derived-type owner is - migrated; only the reduced raw-array row may move to `wrapper-plan` here. -- [x] Run the focused policy/plan/backend tests, the relevant wrapper test, - documentation checks, wrapper-codegen complexity checker, and required - static-analysis suite before changing route support. - -#### Phase 6G Exit Gate - -- [x] Expand raw array addresses as the explicit next lane using the public - contract, completed semantic policy, legacy binding/bridge primitives, and - the existing compiled `Addr(Float64[n])` oracle. -- [x] Complete Phases 6G1 through 6G5 without changing the public raw-address - contract or introducing a parallel action vocabulary. -- [x] Prove that one maintainer algorithm—object kind, Python action, native - action, handoff mode, data action, then typed shape facts—covers ordinary and - raw arrays without backend inference. -- [x] Move only dependency-closed raw-array test rows after generated-artifact - comparison and compiled legacy/direct parity pass. -- [x] Begin Phase 7 only after this exit gate is complete. Phase 7 must consume - the established distinction among array buffers, raw addresses, and native - descriptors rather than revisiting it. - -## Phase 7 — Native Array Handles And Descriptors - -Implementation status: reopened for the view-only `to_numpy()` contract -correction. The previously completed direct Phase 7A-H slices remain evidence -for unaffected descriptor handoffs, but Phase 7 is not closed again until -plain and `Aliased` module-array handles both return a current live view or -`None` without an implicit copy and the final verification gate is rerun. -Every field, pointer-result, callback, and deferred-real-library exclusion -remains on its later blocker. - -Scope: migrate the existing native descriptor and runtime-handle contract into -the wrapper-plan path without redefining that public contract. The maintained -`native-array-handle-checklist.md` remains the feature-level behavioral oracle; -this section owns only its migration into completed wrapper policy, -`ArgumentTransferPlan`, `ResultPlan`, `ModuleVariablePlan`, subordinate native -slots and lifecycle actions, direct C/Fortran lowering, and production route -selection. - -The shared descriptor family includes: - -- rank-positive `Allocatable[T[...]]` and `Pointer[T[...]]` handle arguments; -- optional-absent array handles, where omission or `None` means the native - optional dummy is absent; -- projected writable descriptors whose mutation must remain attached to the - same caller handle; -- wrapper-owned allocatable array results and hidden outputs; -- borrowed module allocatable and pointer handles plus their generated operation - tables; -- native handles passed as actual values to ordinary `T[...]` dummies without - an implicit `.to_numpy()` call; -- build requirements for standard C descriptors; and -- the remaining rank-zero allocatable/pointer result cases, including nullable - deferred-length scalar character values, which return copied Python values - rather than native-array handle objects. - -Allocatable and Pointer remain separate public contract types but share one -plan and lowering structure. Descriptor kind selects only the operations that -genuinely differ: allocation state versus association state, allowed -shape-changing operations, target lifetime, extraction policy, and release. -Do not create independent allocatable and pointer planner hierarchies. - -For every rank-positive module handle, `to_numpy()` has one public result: -`None` for an unallocated/unassociated native object and a live NumPy view of -the current allocation/target otherwise. Plain and `Aliased` allocatable -module variables use the same behavior. `Aliased` remains semantic metadata -but never selects a detached copy. Users call `.copy()` explicitly for -independent storage; an old live view may become stale after native -deallocation, reallocation, nullification, or reassociation, and a fresh -`to_numpy()` call must inspect current native state. - -### Phase 7 Boundary And Explicit Non-Scope - -The following four boundaries must remain distinct: - -| Python contract | Planned Python input | Native transport | Owner phase | -| --- | --- | --- | --- | -| `T[...]` with a NumPy array | validated NumPy storage | `PASS_ARRAY_BUFFER` | Phase 6 | -| `T[...]` with an allocated/associated native handle actual | validated handle array-data facet | `PASS_ARRAY_BUFFER` | Phase 7A | -| `Allocatable[T[...]]` / `Pointer[T[...]]` | matching runtime handle object | `PASS_NATIVE_DESCRIPTOR` | Phase 7B onward | -| `Addr(T[n, ...])` | caller-supplied integer address | `PASS_RAW_ADDRESS` | Phase 6G prerequisite, not Phase 7 | - -`Addr(Float64[n])` is a supported public semantic `.pyi` contract when -every extent is a literal or an expression over visible scalar arguments or -rank-zero scalar storage. It accepts an integer such as `array.ctypes.data` and -forwards that address without ownership, dtype, alignment, lifetime, or bounds -validation. Parsing, policy completion, printing, and both compiled wrapper -routes support it through the completed -`RAW_ADDRESS` / `PASS_RAW_ADDRESS` selector pair. Do not misclassify this raw -pointer as a NumPy buffer, native handle, or C descriptor while maintaining -Phase 7. - -Other exclusions and dependencies are: - -- ordinary NumPy-only buffer extraction, shape, stride, output identity, and - copy-result behavior already completed in Phase 6; -- caller-supplied raw array addresses, completed separately by the Phase 6G - entry dependency; -- derived-type field attachment, class construction, parent-wrapper creation, - and property orchestration, which require Phases 8 and 9 even though the - shared native-handle plan must already be reusable by those later owners; -- scalar derived module-variable member access and argument compatibility, - which belong to Phase 8. Phase 7 descriptor machinery remains limited to - array handles; scalar derived module allocatables use the exact local - move-out/move-back route specified in Phase 8H and do not consume Phase 7 CFI - descriptor machinery. A failed scalar-object call handoff must not become a - module-access blocker; -- pointer results without completed stable owner storage and target lifetime; -- callback descriptor arguments or results, which remain in Phase 10; -- compiler-private descriptor layout inspection or copying; -- any implicit `.to_numpy()` conversion when a native handle is passed to an - ordinary array dummy; and -- the deferred BLAS/LAPACK generation unit until final cutover. - -### Existing Semantic Authority And Legacy Oracle - -Do not redesign the public feature while migrating it. Reuse these completed -sources of truth: - -- `prik/semantics/native_array_handles.py` defines - `NativeArrayHandlePolicy`, `ArrayInteropPolicy`, handle facts, descriptor - kinds, and completed build requirements. -- `prik/semantics/policy_completion.py` completes handle kind, origin, owner, - owner retention, descriptor ownership, getter/setter behavior, output - projection, release, target lifetime, destruction, extraction, interop, - nullability, storage mode, operations, and blockers before `ir2ast.py`. -- `prik/runtime/handles.py` owns the reusable runtime protocol, including - `_native_array_actual_argument_for_binding_positional`, - `_native_array_descriptor_argument_for_binding_positional`, and - `_native_array_descriptor_handoff_for_binding_positional`. Direct lowering - must call these helpers rather than duplicate their Python validation. -- `prik/codegen/bindings/c_to_python.py` is the legacy binding oracle. Its - `_ARRAY_INTEROP_POLICY_DISPATCHER`, `_NATIVE_ARRAY_HANDLE_DISPATCHER`, - descriptor-argument handlers, owned-result handlers, operation wrappers, and - descriptor reader define the currently passing C behavior. -- `prik/codegen/bridges/fortran_to_c.py` is the legacy bridge oracle. Its - corresponding dispatchers, descriptor-argument handlers, module/field - operation generators, and owned-allocatable result helpers define the - currently passing Fortran behavior. -- `prik/pipeline/build.py` already derives native-array build requirements from - completed semantic policy and records them in manifests. The wrapper plan - must carry and emit the matching artifact requirements without rediscovering - them from generated source text. - -The legacy generators are behavioral oracles, not dependencies of -`prik/codegen`. Reuse the runtime helpers and completed semantic -records directly. Rewrite the smallest equivalent node/lowering methods in the -direct generators; do not import legacy binding/bridge generator methods or -legacy codegen-model nodes into the wrapper-plan package. - -### Completed Direct-Plan Shape - -Wrapper policy now carries the completed native-handle and array-actual facts. -`ArgumentTransferPlan`, `ResultPlan`, and `ModuleVariablePlan` distinguish a -NumPy data-buffer transfer, a normal array dummy receiving a handle actual, and -a descriptor-handle transfer. Central validation fails closed when any typed -handoff, operation, role, ownership fact, or required header is inconsistent; -neither backend infers policy from datatype or `descriptor_boundary`. - -### Required Plan Shape - -Keep all descriptor-specific state subordinate to the existing datatype- -varying owners: - -```text -ArgumentTransferPlan - array: ArrayHandoffPlan | None - native_array_actual: NativeArrayActualPlan | None - native_array_handle: NativeArrayHandlePlan | None - handoff: NativeDescriptorHandoffPlan - native_call_slot: NativeCallSlotPlan - -ResultPlan - native_array_handle: NativeArrayHandlePlan | None - native_call_slot: NativeCallSlotPlan | None - -ModuleVariablePlan - native_array_handle: NativeArrayHandlePlan | None - -FunctionPlan - native_call_slots: shared ordered references - lifecycle actions: ordered handle materialization/release references -``` - -`NativeCallSlotPlan` and `LifecycleActionPlan` are not competing top-level -semantic owners. A native slot is the argument/result ABI facet shared by its -owning transfer plan, while lifecycle records are function-wide ordering -indexes back to argument/result roles. Descriptor ownership, release, and -operation policy stay under `ArgumentTransferPlan`, `ResultPlan`, or -`ModuleVariablePlan`. Backend-local CFI storage, copy buffers, and failure -cleanup remain inside the named lowerer selected by those plans. - -`NativeArrayActualPlan` is used only when an ordinary `T[...]` argument permits -a runtime native handle as another source for the existing array-buffer ABI. It -records the explicitly accepted Python source kinds and the shared dtype, rank, -shape, layout, writeability, native-byte-order, alignment, and ABI-role checks. -It never carries descriptor ownership or extraction policy. - -`NativeArrayHandlePlan` is one editable projection of the completed handle -policy. It must contain, using typed values rather than free-form backend -method names: - -- descriptor kind and handle kind; -- origin, owner, owner-retention mode, descriptor ownership, and borrowed state; -- element datatype family, dtype, rank, declared shape, order, and character - element length when applicable; -- getter behavior, Python setter exposure, and native setter assignment; -- output projection and same-handle identity requirements; -- release responsibility, target lifetime, destroy behavior, and storage mode; -- `.to_numpy()` extraction action and allowed generated operations; -- descriptor-interop requirement and required headers; -- nullability and optional-absent-handle behavior; and -- one `NativeDescriptorHandoffPlan` with its ABI form and symbolic roles. - -`NativeDescriptorHandoffPlan` must distinguish these typed ABI forms: - -- `FACT_PACKED_CALL_LOCAL`: a non-projected descriptor argument supplies - validated standard descriptor facts; the binding passes those fields and the - bridge establishes call-local standard C descriptor storage. -- `DIRECT_STANDARD_DESCRIPTOR`: a projected writable handle passes its - persistent standard-descriptor pointer so allocation, deallocation, - reassociation, and shape changes remain attached to that handle. -- `OWNED_RESULT_STORAGE`: an allocatable result is materialized into persistent - wrapper-owned CFI storage and later destroyed by the runtime handle. - -The handoff records the descriptor-pointer role when present, `base_addr`, -`elem_len`, runtime rank, per-axis lower-bound/extent/stride-multiplier roles, -an optional presence role, owner-storage role, and generated-operation roles. -The `NativeCallSlotPlan` and its owning argument or hidden result must reference -the same mutable handoff record; do not duplicate descriptor facts that a -maintainer would need to edit twice. - -Convert the current string-valued completed policy selectors into typed plan -enums or validate and translate them exactly once while building wrapper -policy. Backends must not match raw strings such as `argument_descriptor`, -`projected_handle`, or `pointer_c_descriptor` to choose behavior. - -### Consistent Action Vocabulary - -Reuse the existing orthogonal actions: - -| Case | `ObjectKind` | Python action | Native action | `CodegenAction` | Bridge data action | -| --- | --- | --- | --- | --- | --- | -| Ordinary array with ndarray or handle actual | `NUMPY_ARRAY` | `ARRAY_STORAGE`, with explicitly planned accepted sources | `PASS_ARRAY_BUFFER` | existing Phase 6 input/in-place action | `ASSOCIATE_VIEW` | -| Read-only descriptor handle argument | `NUMPY_ARRAY` | `WRAPPER_INSTANCE` | `PASS_NATIVE_DESCRIPTOR` | `CALL_LOCAL_INPUT` | `ASSOCIATE_VIEW` | -| Writable projected descriptor handle | `NUMPY_ARRAY` | `WRAPPER_INSTANCE` | `PASS_NATIVE_DESCRIPTOR` | `IN_PLACE_ARGUMENT` | `DIRECT_TRANSFER` | -| Owned allocatable handle result | `NUMPY_ARRAY` | `NONE` | `NONE` or hidden `PASS_NATIVE_DESCRIPTOR` | `WRAPPER_INSTANCE` | `COPY_REPRESENTATION` with an ownership-transfer reason | -| Borrowed module handle getter | `NUMPY_ARRAY` | module getter action `NATIVE_ARRAY_HANDLE` | operation-specific | `BORROWED_VIEW` | completed per operation | - -Using `WRAPPER_INSTANCE` for the Python handle is consistent with the existing -action axis: the binding validates and consumes a generated runtime wrapper -object, while `ObjectKind.NUMPY_ARRAY` still identifies its array semantic -family. Add a new Python action only if a proven backend operation cannot be -expressed by this existing pair. Add `ArgumentHandoffMode.NATIVE_DESCRIPTOR` -because descriptor tuples are a genuinely different binding-to-bridge ABI from -`ARRAY_BUFFER`; do not overload the Phase 6 mode. - -Keep rank-zero descriptor values on the scalar or string object-kind route. -Their result action creates a Python scalar/string or `None`, not -`WRAPPER_INSTANCE`, and they must not carry `NativeArrayHandlePlan`. - -### Cross-Backend Validation Invariants - -Before either backend emits source, `_validate_plan()` must reject every one of -these inconsistencies: - -- a descriptor plan whose completed `ObjectKind` is not `NUMPY_ARRAY`; -- `PASS_ARRAY_BUFFER` carrying descriptor ownership or CFI roles; -- `PASS_NATIVE_DESCRIPTOR` carrying ordinary data-buffer handoff roles without - a descriptor handoff; -- a disagreement among handle policy, interop ABI, descriptor kind, handle - kind, argument/result plan, and native-call slot; -- a required handle accepting `None`; -- an optional absent handle without a presence role, or a required handle with - one; -- collapsing optional absence into present-unallocated/present-unassociated - state: an absent handle has null fields and a null presence token, whereas a - present handle may have null `base_addr` but must have a non-null presence - token; -- fact-packed handoff for a projected writable descriptor, or direct persistent - descriptor handoff for a policy that does not permit descriptor mutation; -- direct descriptor handoff without a typed - `_NativeArrayDescriptorHandoff`-compatible runtime operation; -- descriptor dtype, rank, shape, element length, or per-axis field counts that - disagree with the declared handle data facet; -- pointer reassociation, allocation, deallocation, or resize without completed - `PointerPolicy` permission; -- a pointer result without stable owner storage and target lifetime; -- an owned result without wrapper ownership, heap/alias boundary storage, - destroy behavior, owner retention, or a failure-path release action; -- a borrowed module/field handle that claims to destroy native owner storage; -- descriptor-view extraction without its completed C-descriptor build - requirement; -- a C-descriptor header requirement on a generation unit whose completed plans - do not need that interop; and -- any semantic helper temporary represented by a fabricated - `OwnershipDecision`. Call-local CFI variables, decoded-dimension locals, - pointer views, status locals, and operation tables are backend-local emitted - storage inside the already selected method. - -### Phase 7A — Ordinary Array Dummies Accepting Native Handle Actuals - -Included: concrete-rank numeric `T[...]` arguments already supported by Phase 6 -when the runtime value is either a valid ndarray, an allocated allocatable -handle, or an associated pointer handle. The handle path validates the same -dtype, rank, shape, layout, writeability, byte-order, and alignment contract, -then calls the handle's internal `array_actual` operation and packs the existing -Phase 6 pointer/extent/stride ABI. It never calls `.to_numpy()` and never passes -the allocatable/pointer descriptor to the ordinary native dummy. - -Initially excluded: optional, assumed-rank, character, and unsupported -noncontiguous handle actuals. Audit each against live runtime-helper behavior -before widening this sub-lane; a rejected form must remain an explicit blocker, -not silently fall back to `.to_numpy()` or a raw address. - -Those exclusions remain visible as the uncompleted -`array-handle-actuals-excluded` rollout lane. Their direct Phase 6 ndarray -lowerers remain testable with a forced wrapper-plan route, but automatic -production selection stays on the legacy route until each corresponding handle -source has parity evidence. - -Legacy oracle: `CPythonBindingGenerator._native_array_actual_argument_body`, -the normal-array runtime helpers in `prik/runtime/handles.py`, and the existing -Phase 6 bridge array-buffer lowering. Reuse the runtime helpers and bridge ABI; -rewrite only the minimal direct binding call and source-kind branch. - -Plan and lowering requirements: - -- [x] Add `NativeArrayActualPlan` or equivalent accepted-source facts beneath - the existing ordinary `ArgumentTransferPlan`; keep - `PASS_ARRAY_BUFFER`, `ArgumentHandoffMode.ARRAY_BUFFER`, and - `ArrayHandoffPlan` unchanged. -- [x] Make the C binding's named ordinary-array input method call - `_native_array_actual_argument_for_binding_positional` with only planned - validation flags and ABI-field selections. -- [x] Keep the Fortran bridge on the exact Phase 6 array-buffer method; it must - not know whether Python supplied an ndarray or a handle. -- [x] Validate that handle actuals are allocated/associated, have a non-null - data address, and satisfy the same declared contract as ndarray inputs; - preserve allocated/associated zero-length arrays. -- [x] Add the `array-native-handle-actuals` support lane and remove the current - production gate on ordinary array actuals only after reduced compiled parity - proves both runtime source kinds and all rejection paths. -- [x] Reuse the normal-array calls in - `test_module_and_derived_pointer_handles_track_native_association` and - allocatable handle fixtures as the legacy baseline, but extract a class-free, - dependency-closed parity contract so Phase 8 does not determine this lane's - route. - -### Phase 7B — Required Read-Only Descriptor Handle Arguments - -Included: required, non-projected `Allocatable[T[...]]` and -`Pointer[T[...]]` arguments. The Python binding accepts only the matching -runtime handle class. A present unallocated allocatable or unassociated pointer -is still a present descriptor argument and may carry a null `base_addr`. - -The binding uses the existing descriptor runtime helper to obtain validated -standard descriptor facts. The bridge establishes rank-specific call-local CFI -storage from `base_addr`, `elem_len`, rank, and dimension records, then passes -the native allocatable or pointer dummy. This association is an emitted-code -view, not a semantic data copy. - -Legacy oracle: - -- binding `_bind_allocatable_descriptor_argument`, - `_bind_pointer_descriptor_argument`, and - `_bind_fact_packed_native_array_descriptor_argument`; -- bridge `_bridge_allocatable_descriptor_argument`, - `_bridge_pointer_descriptor_argument`, and - `_bridge_native_array_descriptor_argument`; and -- runtime `_native_array_descriptor_argument_for_binding_positional`. - -- [x] Carry the completed `NativeArrayHandlePolicy` and descriptor - `ArrayInteropPolicy` into `ArgumentPolicy`, `ArgumentTransferPlan`, and its - shared native slot. -- [x] Add `ArgumentHandoffMode.NATIVE_DESCRIPTOR` and a - `FACT_PACKED_CALL_LOCAL` descriptor handoff with exact symbolic roles. -- [x] Add directly named C and Fortran descriptor-input methods grouped under - the native-array-handle family; backend-local tuple items and CFI locals may - be created only inside those selected methods. -- [x] Validate matching handle class, descriptor kind, dtype, rank, declared - shape, and element length before the call. Reject ndarray inputs. -- [x] Add separate `allocatable-descriptor-inputs` and - `pointer-descriptor-inputs` support lanes after reduced descriptor-argument - parity passes. -- [x] Replay the descriptor calls in - `test_pointer_descriptor_views_preserve_slice_shape_strides_and_parent_lifetime` - and the allocatable descriptor fixtures through minimal class-free contracts; - retain the mixed original nodes as legacy until all their later owners migrate. - -### Phase 7C — Optional Absent Descriptor Handles - -Included: `Allocatable[T[...]] | None = ...` and -`Pointer[T[...]] | None = ...` callable arguments. Omission and explicit -`None` both mean native `present(...)` is false. A present handle remains -present even when its descriptor has absent allocation/association state. - -This is a two-level handle-presence contract, not the Phase 3 scalar descriptor -three-state value contract. Do not reuse the value pointer as the presence -token. The runtime helper already produces null fact fields plus null presence -for absence, and a distinct non-null token for every present handle. - -- [x] Project `optional_absent`, `nullable`, presence mode, and the dedicated - presence role from completed handle policy without inspecting the Python - object in planning or bridge code. -- [x] Generate both required and optional fact-packed descriptor calls through - the Phase 7B methods, adding only the planned presence ABI field and native - branch. -- [x] Validate required-versus-optional annotation, field count, presence role, - and the distinction between absent handle and present null `base_addr`. -- [x] Add `optional-native-array-handles` route coverage only after compiled - tests exercise omission, explicit `None`, present allocated/associated, - present unallocated/unassociated, wrong handle kind, and wrong dtype/rank. -- [x] Treat the lack of one isolated compiled optional array-handle fixture as - a coverage gap: create a reduced semantic `.pyi` entry over an existing - native optional descriptor routine instead of inventing behavior from the - runtime-only tests. - -### Phase 7D — Writable And Projected Descriptor Handles - -Included: descriptor arguments whose allocation, deallocation, resize, -reassociation, or nullification must remain visible through the same Python -handle, plus a matching projected result that returns that identical handle. -Allocatable mutation follows completed ownership. Writable pointer descriptor -mutation requires explicit `PointerPolicy` permissions and target-lifetime -facts. - -Fact-packed call-local descriptors are forbidden here because native mutation -would be discarded at return. The binding must request the handle's typed -persistent standard-descriptor pointer and the bridge must pass it directly. -Returning the projection increments/transfers the existing Python reference; it -does not construct a replacement handle or call `.to_numpy()`. - -The direct handoff requires generated persistent standard-descriptor storage. -Wrapper-owned result handles provide it. Borrowed module handles expose current -descriptor facts for read-only calls, but they are not accepted for projected -writable mutation because a reconstructed call-local descriptor would lose the -native descriptor update. - -Legacy oracle: - -- binding `_bind_direct_native_array_descriptor_argument` and - `_bind_projected_native_array_handle_result`; -- bridge descriptor argument dispatch with completed output projection; and -- runtime `_native_array_descriptor_handoff_for_binding_positional`. - -- [x] Add `DIRECT_STANDARD_DESCRIPTOR` handoff and same-handle result identity - to the owning `ArgumentTransferPlan`, shared native slot, `ResultPlan` or - lifecycle consumer, and function-wide result order. -- [x] Reuse `CodegenAction.IN_PLACE_ARGUMENT` and `DIRECT_TRANSFER`; do not add - a descriptor-copy action for same-handle mutation. -- [x] Plan success and failure reference handling so a projected handle is - returned exactly once and borrowed caller storage is never destroyed. -- [x] Validate operation permissions, descriptor ownership, target lifetime, - direct handoff type, result identity, and optional presence before emission. -- [x] Add `projected-native-array-handles` support only after - `test_allocatable_inout_arrays_mutate_and_return_the_same_handle` has a - reduced legacy/direct-plan parity replay covering allocation, reallocation, - deallocation, identity, wrong input types, and native-memory checks. -- [x] Keep writable pointer reassociation blocked unless the completed policy - proves every required permission and lifetime fact; never downgrade it to a - read-only fact-packed call. - -### Phase 7E — Owned Allocatable Results And Hidden Outputs - -Included: allocatable array direct function results and hidden output -descriptors whose completed policy selects `owned_result_descriptor`. -Direct array results preserve allocated, zero-sized, and unallocated state, -including matrices and higher-rank arrays. An allocatable output dummy may -validly remain unallocated and still returns a present `AllocatableArray` handle -whose state lives inside that handle. Pointer handle results remain blocked -until stable owner storage and target lifetime are -explicit. - -For a supported numeric direct allocatable function result, the bridge assigns -the native function expression once into a procedure-local allocatable and then -uses `move_alloc` to transfer its state into the allocatable `intent(out)` dummy -backed by persistent wrapper-owned `CFI_CDESC_T(rank)` storage. The move does -not copy the array payload and preserves an unallocated rank-one result. Do not -insert a collector helper or a second intrinsic assignment. Other -procedure-local storage remains permitted only when representation conversion -genuinely requires it, such as deferred-character byte materialization. The -binding constructs the complete generated operation table and Python handle -only after owner storage is valid. Ownership transfers to the handle exactly -once; every earlier failure path releases persistent storage and any genuinely -required bridge-local allocation. - -Character-element handles carry runtime `elem_len` and declared element-length -policy in the same descriptor record. Because a deferred character width is -unknown until the native result exists, the bridge first copies the bytes and -the binding then establishes and allocates persistent CFI storage with that -runtime width. This is a named lowering method under the same result handle -plan, not a separate string-result ownership hierarchy. - -Legacy oracle: - -- binding `_bind_owned_allocatable_result_handle`, owned-result operation - builders, `_bind_materialized_native_array_handle_result`, and destroy body; -- bridge `_bridge_owned_allocatable_result_handle` plus allocatable result - helper/copy logic; and -- the runtime handle factory and exactly-once `close()`/finalizer protocol. - -- [x] Attach one `NativeArrayHandlePlan` with `OWNED_RESULT_STORAGE` to direct - and hidden `ResultPlan` owners; hidden outputs share their exact descriptor - native slot. -- [x] Use `CodegenAction.WRAPPER_INSTANCE` and an explained - `COPY_REPRESENTATION` only for materialization into persistent owner storage; - source hiddenness remains `source_kind`, not a codegen action. -- [x] Record owner storage, materialization, handle construction, ownership - transfer, destroy behavior, and release responsibility under the result's - typed handle plan. Keep backend-local CFI allocation/copy/free nodes inside - the selected result lowerer rather than fabricating lifecycle policy records. -- [x] Require generated `shape`, `array_actual`, `descriptor`, extraction/state, - allowed mutation, and `destroy` operations before publishing the handle. -- [x] Validate CFI rank, dtype, element length, allocated state, owner - retention, release responsibility, destroy behavior, and all success/failure - paths before emission. -- [x] Add `owned-allocatable-results` and - `owned-allocatable-hidden-outputs` support lanes after reduced parity from - `test_array_results_follow_data_buffer_and_descriptor_handle_contracts`, - `test_output_arguments_and_multiple_results_follow_python_projection_rules`, - and `test_allocatable_module_fields_and_results_expose_lifetime_safe_handles`. -- [x] Keep pointer result tests on their explicit policy blocker; do not make - their matrix rows `wrapper-plan` merely because allocatable results pass. - -### Phase 7F — Borrowed Module Handles And Generated Operations - -Included: rank-positive allocatable and pointer module variables exposed as one -stable borrowed handle object at module initialization. Repeated attribute reads -return the same handle. Replacement assignment is rejected. The generated -operation table accesses current native state and includes only operations -allowed by completed policy. - -Allocatable operations include allocation state, shape, array actual, -descriptor handoff, extraction, deallocation, and resize where allowed. Pointer -operations include association state, shape, array actual, descriptor handoff, -nullification, extraction, and policy-gated allocation/deallocation/resize. -Borrowed module handles retain the Python module and never destroy native-owned -descriptor storage. - -Deferred-character handles also expose runtime `element_length`. Shape-only -`allocate` and `resize` operations are omitted because they cannot state the -new character width; native procedures that declare the width remain the -authoritative mutation path. - -Legacy oracle: the bridge's `_native_array_module_handle`, -`_native_array_module_handle_operations`, and operation-specific module methods; -the binding's `_bind_borrowed_native_array_module_handle`, operation wrappers, -and handle creation; and the current runtime handle factory. - -- [x] Add a native-handle getter action and one `NativeArrayHandlePlan` beneath - `ModuleVariablePlan`; keep Python attribute exposure and native operation - generation in its binding and bridge child views. -- [x] Plan operation roles and export names explicitly while leaving operation - call locals backend-local. Do not store generated method names in the plan. -- [x] Validate stable handle identity, module owner retention, rejected - replacement, descriptor kind, operation completeness, and borrowed/no-destroy - lifecycle. -- [x] Add `allocatable-module-handles` and `pointer-module-handles` support - lanes only after module-only reduced parity covers state changes, zero-length - state, extraction policy, operation permissions, stale-view behavior, and - module lifetime. -- [x] Use `test_allocatable_module_fields_and_results_expose_lifetime_safe_handles`, - `test_plain_allocatable_module_array_exposes_current_live_view`, - and the module portion of - `test_module_and_derived_pointer_handles_track_native_association` as legacy - oracles. Split out field/class assertions, which remain Phase 8/9 work. - -#### Phase 7F Contract Correction — View-Only Module Extraction - -The checked Phase 7F items above record the original migration slice; they do -not close this changed public contract. Complete this correction before Phase -8 implementation. - -- [x] Update public docs, maintainer docs, generated/checked semantic `.pyi` - evidence, and wrapper coverage rows to specify current live view or `None`, - explicit `.copy()`, and the unsupported stale-view window. -- [x] Complete plain and `Aliased` allocatable module arrays as native-owned - borrowed handles with the same extraction result. Keep addressability, - descriptor mechanism, owner retention, mutability, nullability, storage, - operation permissions, and release responsibility as separate completed - facts. -- [x] Remove `read_only_detached_copy` and extraction-only `copy_only` policy, - plan, runtime, binding, and bridge dispatch. Preserve only typed live-view - mechanisms such as contiguous or standard-descriptor views; unsupported - extraction fails instead of copying. -- [x] For a plain allocatable module array, add the completed standard- - descriptor module-state mechanism needed to inspect the current allocation - on each extraction. Keep it beneath `ModuleVariablePlan`; do not retain a - descriptor or data address as if it were permanently current. -- [x] Keep binding/bridge ownership explicit: the bridge exposes current native - descriptor facts without NumPy knowledge, and the binding validates - dtype/rank/shape/strides and creates the NumPy view with its handle owner as - the base. Native-handle argument handoff must not call `to_numpy()`. -- [x] Replace the obsolete read-only-copy test with source/generated-`.pyi` - parity covering plain and `Aliased` live mutation, allocated/unallocated and - associated/unassociated state, fresh extraction after state changes, - explicit-copy independence, stale-view documentation, parent/owned-result - retention, and contiguous/strided pointer views. -- [x] Rerun focused policy/plan/backend/runtime tests, documentation checks, - the wrapper suite excluding LAPACK, the wrapper-codegen complexity checker, - and the required static-analysis suite before closing Phase 7 again. - -### Phase 7G — Pointer Descriptor Extraction And Build Requirements - -Included: pointer `descriptor_view`, `contiguous_view`, and explicitly -unsupported extraction actions already selected by completed policy; standard -descriptor decoding; positive and negative strides; and local build/header -requirements. A `copy_only` `to_numpy()` action is obsolete and must not reach -the corrected plan. - -Descriptor views, the corrected plain allocatable module-state path, and -persistent allocatable owner storage require standard C descriptor support. -Generated code may read `CFI_cdesc_t` through `ISO_Fortran_binding.h` when the -completed plan requests it. It must never guess or expose a compiler-private -descriptor layout. Unsupported toolchains fail planning/build with the -completed owner path and requirement. - -- [x] Carry typed extraction and descriptor-interop actions plus required - headers into handle/module/result plans and rendered artifact metadata. -- [x] Reuse the runtime descriptor-view helper for shape, stride, buffer-window, - dtype, rank, and null-address validation; direct C lowering only decodes the - standard descriptor fields into its expected mapping. -- [x] Add directly named C descriptor-reader and operation-wrapper methods; - decoded dimension objects and mapping temporaries remain binding-local. -- [x] Validate that build requirements equal the union of completed plans, - appear in replayable manifests, and do not leak into wrappers that need only - ordinary buffers or non-CFI borrowed allocatable handles. -- [x] Replay - `test_pointer_descriptor_views_preserve_slice_shape_strides_and_parent_lifetime` - and `test_pyi_manifest_records_pointer_descriptor_interop_requirements`, plus - focused `tests/runtime/handles`, before enabling - `pointer-descriptor-extraction`. -- [x] Preserve the explicit planning failure when required C descriptor - support is unavailable; no contiguous-copy fallback may be inferred in the - backend. - -### Phase 7H — Remaining Rank-Zero Descriptor Results And Strings - -Phase 3 already owns ordinary and optional scalar descriptor inputs, including -omitted/present-null/present-value state. Phase 4 already owns nullable scalar -descriptor module reads as copied Python snapshots. Do not rebuild those paths -or turn rank-zero descriptors into runtime handle objects. - -Included here: direct scalar descriptor function results, hidden scalar -descriptor outputs, projected scalar descriptor readback, and allocatable or -pointer scalar character results with runtime/deferred length. The Python result -is `T | None` or `String | None`; absent allocation/association returns `None`. -An allocated/associated value is copied exactly once before the call-local or -native descriptor is released. Deferred-length strings use runtime element -length and preserve the existing encoding/byte contract. - -- [x] Add a subordinate scalar-descriptor handoff/result record to the existing - scalar or string `ArgumentTransferPlan`/`ResultPlan`; do not attach - `NativeArrayHandlePlan` or use `ObjectKind.NUMPY_ARRAY` for rank zero. -- [x] Complete result source, descriptor kind, presence, runtime element length, - copy action/reason, release owner, and failure cleanup in wrapper policy before - planning. -- [x] Reuse existing Phase 3 presence records and typed lifecycle ordering; - extend named scalar/string result lowering only for the descriptor producer - and copy/release steps. -- [x] Validate direct versus hidden descriptor source, nullable result spelling, - result ordering, runtime string length, null state, copy count, and cleanup on - conversion/status failure. -- [x] Add isolated legacy/direct parity for numeric allocatable and pointer - results and for `string_result_deferred` from - `test_modern_fortran_character_arguments_and_results`, including an absent - result and non-ASCII encoded data. -- [x] Keep pointer array results blocked even after pointer scalar values pass; - copied scalar readback does not prove array target lifetime. - -### Derived Fields Remain A Recorded Later Dependency - -The shared handle plan must be capable of recording -`borrowed_field_descriptor`, `owner_retention=parent_wrapper`, field operation -roles, and parent-owned destruction behavior. Do not add field/class traversal -or route eligibility in Phase 7. `BindCNativeArrayHandleProperty`, field -operation generation, and the field portions of allocatable/pointer tests remain -legacy oracles for Phases 8 and 9, where the owning wrapper instance and property -lifecycle exist in the plan. - -This boundary prevents Phase 7 from either duplicating future `FieldPlan` -ownership or falsely marking mixed module-and-field generation units supported. - -### Legacy Primitive Inventory And Rewrite Rule - -| Primitive | Legacy source | Direct-plan treatment | -| --- | --- | --- | -| Normal array handle actual | binding `_native_array_actual_argument_body`; runtime normal-array helpers | reuse runtime helper and Phase 6 bridge ABI; rewrite minimal binding nodes | -| Required/optional descriptor argument | binding descriptor argument helpers; bridge descriptor handlers | rewrite named direct methods around shared runtime packer and planned CFI roles | -| Projected writable descriptor | binding direct descriptor handler; bridge descriptor projection | rewrite direct pointer handoff and identity lifecycle; no fact-packed fallback | -| Owned allocatable result | binding owned-result/operation helpers; bridge allocatable result helper | rewrite minimal CFI owner-storage and result lifecycle nodes; assign once locally and transfer the allocation into the CFI-backed output dummy with `move_alloc` | -| Borrowed module handle | binding handle creation/operation wrappers; bridge module operations | rewrite under `ModuleVariablePlan`; reuse runtime factory | -| Pointer descriptor view | binding descriptor reader; runtime view helper | reuse runtime view helper; rewrite only standard-descriptor decoding nodes | -| Scalar descriptor result | legacy scalar descriptor/result conversion | extend existing scalar/string plan route; do not create an array handle | -| Build requirement | semantic `native_array_handle_build_requirements`; build manifest | reuse completed requirements and carry them through rendered artifacts | - -For every primitive, first retain generated legacy C/Fortran/header artifacts -from the cited passing wrapper test. Explain each material direct-plan artifact -difference before compilation. Copy a small legacy method only when it already -matches the direct node API and complexity limit; otherwise rewrite the minimal -equivalent. Do not copy legacy dispatcher classes, scope mutation machinery, or -datatype/policy inference. - -### Route And Test Migration Matrix For Phase 7 - -Mixed rows retain their later derived/field owners. The dependency-closed -Phase 7 rows were split, proved through both routes, and then recorded as -`wrapper-plan` in the complete ledger. - -| Existing node or group | Current role/status | Phase 7 owner and target | -| --- | --- | --- | -| `arrays/test_array_results.py::test_array_results_follow_data_buffer_and_descriptor_handle_contracts[*]` | ordinary/allocatable result generation unit; `wrapper-plan` | Phase 6 ordinary and Phase 7E allocatable results now share the production plan route | -| `../../fortran/pointers/pipeline/test_pointer_build_manifest.py::test_pyi_manifest_records_pointer_descriptor_interop_requirements` | non-generating manifest policy; `not-applicable` | Phase 7G plan/header union is covered by direct generated-artifact tests | -| `derived_types/test_pointers.py::test_module_and_derived_pointer_handles_track_native_association[*]` | module, normal-array actual, and field mix; `legacy` | split Phase 7A/7F module subsets; field subset remains Phase 8/9 | -| `derived_types/test_pointers.py::test_pointer_descriptor_views_preserve_slice_shape_strides_and_parent_lifetime` | module/field descriptor views; `legacy` | Phase 7B/7G module subset; field owner remains Phase 8/9 | -| `derived_types/test_pointers.py::test_pointer_array_results_use_owned_descriptors_without_owning_targets[*]` | owned pointer-result descriptor support; `wrapper-plan` | replaces the former owner-policy blocker after descriptor ownership and target lifetime became explicit | -| `edit_pyi_contracts/test_ownership_contracts.py::*` | module, field, result lifetime mix; `legacy` | Phase 7E/7F subsets; field/finalizer owners remain Phase 8/9 | -| `function_calls/test_optional_arguments.py::test_optional_allocatable_scalar_descriptor_distinguishes_omitted_none_and_value` | scalar baseline; `wrapper-plan` | reuse Phase 3 behavior; no status change | -| `function_calls/test_output_arguments.py::test_output_arguments_and_multiple_results_follow_python_projection_rules[*]` | mixed scalar/array/string/derived/allocatable outputs; `legacy` | Phase 7E reduced allocatable result; retain mixed row | -| `module_state/test_allocatable_replacement.py::*` | projected same-handle descriptor mutation plus a derived factory generation unit; `legacy` | Phase 7D reduced parity is `wrapper-plan`; the broad factory/class unit remains Phase 8/9 | -| `module_state/test_allocatable_views.py::test_allocatable_module_fields_and_results_expose_lifetime_safe_handles[*]` | module, result, and derived-field mix; `legacy` | field/class owner retention remains Phase 8/9 | -| `module_state/test_allocatable_views.py::test_scalar_descriptor_module_variables_return_copied_optional_values[*]` | scalar descriptor arguments/results/module state; `wrapper-plan` | source conversion records descriptor kind and argument/return reference before completed Phase 7H policy | -| `module_state/test_allocatable_views.py::test_plain_allocatable_module_array_exposes_current_live_view[*]` | corrected source/generated-`.pyi` production-plan evidence | proves Phase 7F plain/`Aliased` current live-view or `None` parity, native mutation, explicit-copy independence, and fresh extraction after state changes | -| `strings/test_character_arguments.py::test_modern_fortran_character_arguments_and_results[*]` | fixed strings plus deferred allocatable result; `legacy` | Phase 7H reduced deferred-result parity; retain mixed row as needed | -| `edit_pyi_contracts/test_native_order_contracts.py::test_editable_contract_can_use_native_order_arguments_without_native_call` | includes raw `Addr(Float64[n])` plus a derived result; `legacy` | raw-array subset is the completed Phase 6G prerequisite; derived subset remains Phase 8 | - -| Completed sub-lane | Dependency-closed compiled evidence | -| --- | --- | -| Phase 7A, 7B, 7F, and 7G | `derived_types/test_pointers.py::test_module_native_array_handles_use_canonical_plan` | -| Phase 7C | `function_calls/test_optional_arguments.py::test_optional_array_descriptors_preserve_presence_and_storage_state` | -| Phase 7D | `module_state/test_allocatable_replacement.py::test_projected_allocatable_descriptor_preserves_same_handle_identity` | -| Phase 7E numeric | `arrays/test_array_results.py::test_owned_allocatable_results_preserve_handle_state` | -| Phase 7E deferred character | `strings/test_character_arguments.py::test_deferred_character_array_handles_use_canonical_plan` | -| Phase 7H numeric | `scalars/test_scalar_boundary_plan.py::test_scalar_descriptor_results_copy_values_or_none_through_wrapper_plan_route` | -| Phase 7H deferred scalar character | `strings/test_character_arguments.py::test_deferred_allocatable_string_results_use_canonical_plan` and the nullable case in the deferred-character handle test | -| Phase 7H source/default projection | `module_state/test_allocatable_views.py::test_scalar_descriptor_module_variables_return_copied_optional_values[*]` | - -Required focused intermediate coverage includes: - -- completed semantic handle/interop policy projection tests; -- editable plan tests for every descriptor kind, handoff form, operation set, - ownership, optional presence state, and lifecycle edit; -- C and Fortran preflight rejection of mismatched or incomplete descriptor - plans; -- generated artifact assertions for standard descriptor fields, optional - presence, operation functions, owner storage, destroy paths, and local header - requirements; -- runtime helper tests under `tests/runtime/handles` without duplicating their - validation in wrapper-codegen tests; and -- compiled legacy/direct parity for each reduced sub-lane before any migration - ledger or production-route change. - -### Phase 7 Completion - -- [x] Expand Phase 7 under the mandatory gate using the live completed policy, - runtime handle implementation, legacy backends, build integration, public - contract, and focused wrapper tests. -- [x] Complete the shared typed handle, array-actual, and descriptor-handoff - plan records without adding a parallel top-level plan hierarchy. -- [x] Complete every missing semantic selector before `ir2ast.py`; remove - bridge-created `ArrayInteropPolicy` and fabricated semantic ownership choices. -- [x] Finish Phases 7A through 7H individually with legacy artifact capture, - direct lowering, validation, compiled parity, route evidence, and matrix - updates. -- [x] Preserve the completed Phase 6G raw-address boundary while keeping every - derived-field, pointer-result, callback, and deferred-real-library exclusion - on its explicit later blocker. -- [x] Complete the Phase 7F view-only correction for plain and `Aliased` - allocatable module arrays and remove every implicit-copy extraction path. -- [x] Rerun focused policy/plan/backend tests, relevant runtime-handle tests, - documentation checks, the wrapper suite excluding LAPACK, the wrapper-codegen - complexity checker, and the required static-analysis suite after the - correction. -- [x] Close Phase 7 only when every live non-field native-handle/descriptor case - is migrated or explicitly removed from the product contract, and no backend - infers descriptor policy or silently substitutes a data buffer, raw pointer, - `.to_numpy()` extraction, or copy fallback. - -Historical pre-correction evidence: 538 focused semantic/policy/plan/backend/ -runtime tests, 1,133 documentation and layout tests, and all 318 wrapper tests -outside the deferred BLAS/LAPACK file passed. The wrapper-codegen complexity -check, Ruff, formatting, Bandit, Vulture, whitespace, and explicit-base Radon -policy also passed. This evidence remains valid for unaffected sub-lanes but is -not closure evidence for the changed view-only extraction contract. Record a -new success signal after the Phase 7F correction. - -Post-correction closure evidence (2026-07-14): 214 focused runtime-handle, -policy, planning, lowering, legacy-dispatch, and Phase 7 direct-plan tests; -199 complete `tests/codegen` tests; 1,123 documentation tests; 317 -wrapper tests outside the shared real-library parameter plus the BLAS-only -parameter; and zero locally executed LAPACK tests all passed. The wrapper -complexity checker, Ruff lint/format, Bandit, Vulture, whitespace, and the -explicit-`origin/main` Radon policy passed. The required `--base-ref auto` -Radon invocation could not resolve a CI base SHA locally; the explicit base -rerun passed. Advisory full Radon complexity and maintainability reports were -also produced. - -Phase 7 was re-verified again with the final Phase 8 closure run on -2026-07-15: the 704-test semantic/policy/plan/backend regression batch, all 79 -runtime-handle tests, 1,123 documentation tests, and all 326 wrapper tests -outside LAPACK passed. No LAPACK test was run locally. - -## Phase 8 — Derived Types And Object Lifetimes - -Expansion status: complete. Implementation proceeds only after the Phase 7 -view-only correction is re-verified. - -Implementation status: reopened for the complete rank-zero scalar-derived -actual/dummy compatibility matrix in Phase 8H. The previous Phase 8A-I -evidence remains authoritative for unaffected fields and lifecycle paths, but -the old module-allocatable rejection, nonreassociating pointer-only path, -interoperable-only value restriction, and incomplete module-object call routes -are superseded. Phase 8 must not close again until direct, scoped-address, -wrapper-holder, module-transaction, pointer-input, and typed-value actions are -implemented without a fallback and re-verified with multi-argument calls. - -Scope: migrate scalar derived-type storage, arguments, results, borrowed -objects, and field handoffs into the wrapper-plan route. Phase 8 owns the -opaque native-instance substrate and the typed transfers that use it. Phase 9 -owns public constructors, methods, overloads, inheritance, and general -class-surface orchestration built on that substrate. Phase 8 owns public field -descriptors and their typed getters/setters because every live object origin, -including plain module proxies, needs the same readable and writable member -surface. - -Do not begin implementation while a Phase 7 native-array-handle correction is -open. In particular, Phase 8 must consume the final view-only `to_numpy()` -contract: an array-handle extraction is a live view or `None`, and an -independent array is obtained with an explicit `.copy()`. - -`Snapshot[T]` is no longer an active public contract. Plain and `Aliased` -rank-zero derived module variables both expose the normal live generated object -surface. Their lowering mechanisms remain distinct: `Aliased` proves a direct -address-backed borrow, while a plain declaration requires typed module-specific -bridge access and must not fabricate a native address. `Aliased` remains an -addressability and aliasing fact for raw-address legality, pointer association, -C-pointer policy, and direct derived-object handoff; it does not select -array-handle `to_numpy()` behavior. - -### Phase 8 Boundary And Explicit Non-Scope - -The first implementation slice is rank-zero, non-polymorphic derived values. -The runtime wrapper is opaque: the binding carries a native address, ownership -state, and an optional retained Python owner, while the bridge performs typed -native association, assignment, allocation, and destruction. The binding must -not depend on component offsets or reproduce native aggregate layout. - -The following surfaces are in Phase 8: - -- required and optional scalar derived arguments; -- visible `out` and `inout` wrappers whose identity remains caller-visible; -- hidden output and direct-function-result values materialized as - wrapper-owned instances; -- native `value` arguments for an exact rank-zero monomorphic derived type, - using a Fortran bridge-owned typed value copy rather than C-side layout - inference; the native type need not be `bind(C)` when the bridge imports its - exact definition; -- derived `parameter` and other explicit constant-value origins materialized - through the existing wrapper-owned immutable-value path, never as a fallback - for an ordinary mutable module object; -- plain rank-zero native module objects exposed as live module-backed proxies - through typed bridge operations; -- `Aliased` rank-zero native module objects exposed as live borrowed wrappers; -- borrowed nested component wrappers, their public field descriptors, and the - owner-retention facts required by those descriptors; -- Phase 7 allocatable/pointer field-handle plans attached to a derived owner; -- exact destruction, finalization, cleanup, and failure ownership for each of - those origins. - -The following remain outside Phase 8: - -- public default/keyword constructors, explicit `@bind(...)` constructors, - `tp_init`, methods, static methods, overload dispatch, Python inheritance, - and ordinary type-bound surface assembly; these remain Phase 9. Public field - descriptors, getters, and setters are Phase 8 and are not a Phase 9 blocker. - A generated semantic `.pyi` field constructor is therefore a whole-unit - Phase 9 blocker; only an opaque contract that suppresses default construction - may use the direct Phase 8 object route; -- scalar polymorphic dispatch and inheritance even where the legacy route - supports them; Phase 9 owns the class relationship needed to validate the - accepted runtime type set; -- callback-derived arguments and results, adapter procedures, and trampoline - ownership; these remain Phase 10; -- arrays of derived values, whose element layout, construction, destruction, - copy, and partial-failure behavior remain explicit planning errors; -- polymorphic results, mutable polymorphic arguments, `class(*)`, abstract - instantiation, deferred bindings, and allocatable/pointer polymorphic - scalars; -- polymorphic descriptor-backed scalars. Wrapper-owned allocatable and pointer - holders plus scalar derived module ordinary/`TARGET`/allocatable/pointer - variables are supported only by their explicit Phase 8H matrix rows. A - pointer holder owns its association container, never its target by default; - target retention and native release responsibility are completed separately - before lowering. Module allocation and pointer transactions use shared typed - holder addresses in interoperable callbacks, never CFI or a compiler-private - descriptor; -- any other derived origin that cannot use one of the explicit matrix rows. It - remains blocked rather than being silently turned into an address-backed - borrow or detached object; -- C-side aggregate casts, `ctypes` layout promises, compiler-private descriptor - inspection, or direct component offsets; -- ownership of targets reachable through pointer components. A containing - derived wrapper does not own such a target without completed pointer policy; - and -- detached whole-object snapshot classes or recursive member-copy graphs. They - are removed rather than retained as a compatibility path. - -### Public Representation And Lifetime Matrix - -Complete this matrix in post-IR policy before adding planner or backend code. -The rows are distinct origins, not datatype guesses made during lowering. - -| Surface | Python representation | Native handoff/storage | Owner and release | -| --- | --- | --- | --- | -| required `in` argument | existing wrapper instance | pass its opaque wrapper address and associate a typed native view for the call | wrapper remains owned by its existing Python object; call destroys nothing | -| required visible `inout` or caller-supplied `out` | same wrapper instance | pass the same address for native mutation | caller-visible wrapper retains identity; its normal wrapper finalizer remains the sole destroyer | -| optional argument, omitted or `None` | no wrapper instance | explicit absence token/branch; no fabricated native object | no allocation or cleanup | -| optional argument, present | validated wrapper instance | same typed address handoff as the required case | existing wrapper owner remains responsible | -| hidden output | new opaque wrapper object | allocate persistent wrapper-owned native storage before the call and pass its address | wrapper deallocator invokes native-aware destruction exactly once | -| direct function result | new opaque wrapper object | move or copy the native result before its temporary expires into persistent wrapper-owned storage | wrapper deallocator invokes native-aware destruction exactly once | -| constructor-created instance | Phase 9 only | Phase 9 must allocate through the same persistent wrapper-owned storage and native-aware destructor established here | explicitly blocked until Phase 9 class construction orchestration; no Phase 8 fallback constructor | -| native `value` input | existing wrapper instance | exact Fortran bridge passes the typed pointee to the native by-value slot; C never lays out or copies the aggregate | call-local native copy only; wrapper ownership is unchanged | -| plain rank-zero module variable | normal live generated object | module-specific typed getter/setter operations plus a synchronous scoped-address consumer when the object is passed to another procedure | native module owns storage; wrapper retains the module and never destroys storage; a temporary target/address cannot escape its consumer scope | -| `Aliased` or explicit `TARGET` rank-zero module variable | live borrowed wrapper | use `C_LOC` as the sole whole-object handoff; reconstruct the exact typed bridge view without copying | native module owns storage; wrapper never destroys it and rejects replacement | -| derived `parameter` or other explicit constant-value origin | wrapper-owned value copy with an immutable module binding | materialize the native value into persistent wrapper-owned storage | wrapper destroys only its materialized copy; no native module setter; normal writable fields modify only that independent copy | -| nested derived field | live borrowed child wrapper | address/alias of the component through the parent wrapper | child retains parent; child never destroys component storage | -| allocatable scalar derived module origin | nullable live module-backed proxy carrying its runtime origin | scoped-address consumer for payload-only calls; for an allocatable dummy, module-specific interoperable operations move between the module variable and a bridge-local shared typed holder addressed by `C_PTR` | native module owns storage before and after a transaction; successful move-out has exactly one reverse-order move-back; no descriptor crosses C | -| pointer scalar derived module origin | nullable live module-backed proxy carrying its runtime origin | current-target address for payload-only calls; for a pointer dummy, a bridge-local shared typed pointer holder receives the initial association and its address is passed to the module-specific restore operation | native module owns the pointer variable and, by default, its target; final association is restored exactly once after a normally returning native call | -| wrapper-owned allocatable scalar derived result | nullable live generated wrapper backed by one persistent typed allocatable holder per native type | result is moved into `holder%value`; ordinary, target, allocatable, allocatable-target, pointer-input, and value dummies use the explicit compatible matrix actions | each Python wrapper owns one target-capable holder and destroys it exactly once; allocation-state writeback preserves wrapper identity | -| wrapper-owned pointer scalar derived result | nullable live generated wrapper backed by one persistent typed pointer holder per native type | holder component stores current association and is passed directly to a compatible pointer dummy; payload-only calls use its associated target | wrapper owns and destroys only the holder; target ownership stays native unless completed policy retains a known wrapper/module target; destruction nullifies the component and never deallocates an unowned target | -| detached whole-object snapshot | removed | no recursive copy graph or snapshot helper is generated | no compatibility parser, lowering, or fallback; read the live object through normal fields instead | - -`Aliased` remains a public, language-neutral addressability/aliasing fact and -must survive parsing, semantic IR, and printing. For derived module objects it -distinguishes direct-address lowering from module-proxy lowering, not live -versus copied public behavior. It must never be reused to select live versus -copied native-array-handle extraction. - -### Existing Semantic Authority And Legacy Oracle - -Use the current implementation as an oracle, not as permission to preserve its -architecture: - -- `prik/semantics/ownership.py` already names `DERIVED_TYPE`, - `PASS_WRAPPER_ADDRESS`, `WRAPPER_INSTANCE`, and `BORROWED_VIEW`, and contains - the current argument/result/module/field owner defaults. Remove the obsolete - derived whole-object snapshot action without disturbing ordinary result - copies, scalar descriptor value copies, or explicit non-object uses of - `snapshot_copy` transfer policy. -- `prik/semantics/policy_completion.py` is the only allowed owner of origin, - ownership, transfer, destruction, mutability, nullability, projection, - release, storage, getter/setter, owner-retention, module-object handoff, and - field decisions. It must complete module-proxy policy for plain module - objects and direct-address borrowed policy for `Aliased` module objects. -- `prik/semantics/wrapper_policy.py` must gain a derived-specific policy branch. - Derived values must not continue through primitive-scalar blockers, - primitive result checks, or primitive bridge data-action selection. -- `prik/semantics/ir2ast.py` and the legacy generators remain the generated - artifact oracle. Direct lowering must not call `semantic_ir_to_codegen_ast()` - or reconstruct legacy codegen variables. -- `prik/codegen/bindings/c_to_python.py` contains the existing wrapper-instance - conversion, checked casts, owned/borrowed result construction, owner - retention, and allocator/destructor helpers. Remove recursive snapshot - construction rather than migrating it into the direct route. -- `prik/codegen/bridges/fortran_to_c.py` contains the existing typed wrapper - address conversion, native result materialization, borrowed field/module - access, native-aware destruction, and typed component getters/setters. Reuse - those live member-access mechanics as the artifact oracle while moving every - decision into typed plans. - -Capture complete legacy artifacts before each direct slice. Preserve observable -runtime behavior while replacing backend inference with completed typed plans. -Do not copy the broad legacy generator control flow into `codegen`. - -The existing wrapper tests decompose as follows: - -| Existing test or generation unit | Phase 8 oracle | Required split or later owner | -| --- | --- | --- | -| `function_calls/test_native_call_examples.py::test_native_call_examples_cover_scalar_array_string_and_object_projection[*]` | hidden derived result selected by `Return(...)` | add a reduced object-result entry; retain the mixed unit until every included lane is direct | -| `function_calls/test_output_arguments.py::test_output_arguments_and_multiple_results_follow_python_projection_rules[*]` | hidden derived output and mixed result aggregation | add a reduced derived-output entry; retain the broad unit until its complete tuple is direct | -| `edit_pyi_contracts/test_policy_dispatch_contracts.py::test_immutable_scalar_string_array_and_derived_policies_return_replacements` | edited projected derived replacement | isolate `make_point` as Phase 8 evidence; retain the mixed policy unit until whole-unit eligibility follows | -| `derived_types/test_derived_type_boundaries.py::test_scalar_derived_types_cross_procedure_boundaries[*]` | required input, in-place mutation, hidden/direct result, nested borrowed component | reduce first to result-created opaque objects passed back to `point_sum`/`move_point`; field descriptors and nested borrowing are Phase 8, while construction remains Phase 9 | -| `function_calls/test_optional_arguments.py::test_optional_arguments_drive_fortran_present_behavior[*]` | optional derived input and exact type/absence behavior | complete the optional transfer in Phase 8; the existing constructor-dependent broad runtime unit remains Phase 9 until it can route whole | -| `module_state/test_module_state.py::test_aliased_derived_module_object_borrows_native_state[*]` | native-owned borrowed module object and replacement rejection | use as the direct-address oracle; add a reduced plain-module proxy case with the same live field behavior; methods remain Phase 9 | -| `derived_types/test_borrowed_finalizers.py::test_borrowed_child_wrapper_never_finalizes_native_component[*]` | parent retention and exactly-once owner finalization | Phase 8 owns storage/lifetime plans and public field descriptors; constructor/method orchestration remains Phase 9 | -| `module_state/test_allocatable_views.py::test_allocatable_module_fields_and_results_expose_lifetime_safe_handles[*]` | Phase 7 field handle attached to a derived owner | reuse the existing `NativeArrayHandlePlan` and expose its public property in Phase 8 | -| `derived_types/test_pointers.py::test_pointer_descriptor_views_preserve_slice_shape_strides_and_parent_lifetime` | pointer field handle and parent lifetime | reuse Phase 7 descriptor extraction; do not move pointer target ownership into Phase 8 | -| `derived_types/test_derived_layout.py::test_bind_c_derived_types_use_accessors_and_fortran_value_copy[*]` | opaque `bind(C)` wrapper, field accessors, and typed native `value` copy | Phase 8 owns the handoff and field properties; constructor orchestration remains Phase 9 | -| former `module_state/contracts/fmodule_derived_snapshot_f90/` snapshot fixture | obsolete detached-object behavior | remove the `Snapshot[box]` fixture and snapshot-only runtime assertions; reuse the native unit only for reduced live module-proxy evidence where applicable | -| `derived_types/test_constructors_and_finalizers.py::*`, `derived_types/test_derived_type_methods.py::*`, and `derived_types/test_inheritance.py::*` | owned-instance finalizer and type facts may inform Phase 8 | production migration remains Phase 9 because the observable unit is constructor/method/property/inheritance owned | -| `tests/fortran/callbacks/end_to_end/test_supported_callback_shapes.py::*` | none | callback-derived transfers remain callback-owned after ordinary derived transfers are complete | - -The plain non-target module-object row has one recorded intentional correction: -the legacy whole-object getter attempts `c_loc` on storage without the required -addressability property and therefore has no passing whole-object artifact. -Phase 8 uses the real source declaration, the passing legacy typed component -getters/setters, and the passing `Aliased` direct-address behavior as its -mechanical oracles, then improves the plain path to a typed member proxy. The -compiled Phase 8 evidence asserts that this proxy never emits a fabricated -whole-object `c_loc` while its reads and writes remain live. - -### Mandatory Phase 8 Migration Algorithm - -Apply this same sequence to every dependency-closed sub-lane. A checked item in -a later step cannot compensate for an incomplete earlier step. - -1. Capture the complete generated artifacts and runtime assertions from one - real passing legacy/source case. Record which constructor/method or - callback assertions remain outside the reduced unit. -2. Complete object kind, origin, ownership, transfer, destruction, mutability, - nullability, projection, storage, release, owner retention, getter/setter, - native assignment, and any module-object mechanism before `ir2ast.py`. -3. Project those facts mechanically into `ArgumentTransferPlan`, `ResultPlan`, - or `ModuleVariablePlan`, with native slots and lifecycle actions remaining - subordinate references. -4. Validate type identity, roles, actions, owners, storage, result positions, - releases, and cross-backend handoffs before either backend emits source. -5. Lower through small named binding and bridge methods selected by typed - object kind and action. Backend-local temporaries remain implementation - details inside the already selected method. -6. Compare binding, bridge, header, and build artifacts with the captured - oracle and explain every intentional difference before compiling. -7. Add focused policy, plan-edit, validation, printer, backend, runtime, - documentation, and source/generated-`.pyi` parity tests. -8. Promote the reduced generation unit only after compiled legacy/direct parity - passes; otherwise retain one exact blocker without a fallback route. - -The maintainer trace is therefore always: - -```text -completed semantic facts - -> typed argument/result/module-variable plan - -> subordinate native slots and lifecycle actions - -> validation - -> binding and bridge lowering - -> generated artifacts - -> compiled runtime evidence -``` - -### Plan Shape And Stable Action Vocabulary - -Do not create a second function plan, a second result hierarchy, or a rendered -derived-plan layer. Extend the existing plan tree as follows: - -- add one explicit derived datatype family or equivalent non-primitive marker - so a derived semantic type never indexes the primitive scalar dtype maps; -- add a concise namespace-owned derived-type definition record containing - canonical semantic/native identity, native scope, Python exports, opaque - runtime type symbol, allocation role, destruction/finalization role, and the - minimal field identities needed by later field plans; -- add one `DerivedHandoffPlan`-style facet, analogous to `ArrayHandoffPlan`, to - `ArgumentTransferPlan`, `ResultPlan`, `ModuleVariablePlan`, and the owning - `NativeCallSlotPlan` only where that transfer needs it; -- give a derived `ModuleVariablePlan` one typed module-object access facet that - records the completed direct-address or opaque-callback mechanism, its - context/address roles, compiler capability, and module-lifetime owner. This - mechanism is subordinate to the module-variable policy and must not change - its public borrowed-wrapper facts; -- keep native slot order, symbolic roles, and ABI positions subordinate to the - owning argument or result transfer; -- represent result destruction, failed-construction cleanup, parent retention, - through transfer-owned `LifecycleActionPlan` records in function-wide - execution order; -- add field-handoff records beneath the owning derived-type definition. Do not - put their ownership decisions into a backend registry; and -- keep `FunctionPlan`, `ModulePlan`, namespace assembly, result ordering, GIL - envelope, and status-error behavior stable. - -Reuse the existing action vocabulary: - -- Python boundary: `WRAPPER_INSTANCE` for accepted live wrapper objects and - `NONE` for native-produced results; -- native boundary: `PASS_WRAPPER_ADDRESS` for opaque live objects and `NONE` - when the bridge itself owns result production; module-backed proxies use a - distinct typed module-origin handoff rather than a fabricated address; -- transfer/codegen: `CALL_LOCAL_INPUT`, `IN_PLACE_ARGUMENT`, - `IDENTITY_OUTPUT`, `WRAPPER_INSTANCE`, and `BORROWED_VIEW` according to the - completed matrix row; -- bridge data: `DIRECT_TRANSFER`, `ASSOCIATE_VIEW`, or - `COPY_REPRESENTATION`, with a completed copy reason only when a real native - representation copy occurs; and -- lifecycle: existing ordered copy-in/native-mutation/copy-out/cleanup phases, - extended only with a genuinely missing release phase/action rather than a - derived-only parallel lifecycle system. - -If one of these actions cannot express a required operation, document the -missing semantic distinction before adding exactly one typed action. Do not use -method-name strings, datatype conditionals, `intent`, `is_alias`, dotted-name -shape, or local temporary existence as hidden dispatch. - -### Binding, Bridge, And Validation Ownership - -| Layer | Owns | Must not own | -| --- | --- | --- | -| post-IR policy | origin, dynamic/static type allowance, owner, transfer, destruction, mutability, projection, nullability, storage, owner retention, getter/setter behavior, and blockers | emitted local names or source syntax | -| wrapper planner | mechanical projection into derived type/handoff facets, native roles, ordered results, and lifecycle indexes | new ownership or lifetime decisions | -| binding lowering | exact Python type checks, opaque wrapper address extraction, Python wrapper allocation, retained-owner references, result aggregation, and Python reference cleanup | native component layout, native assignment, or native finalization semantics | -| bridge lowering | typed association from opaque addresses, exact Fortran-owned `value` calls, native instance allocation/assignment, module/component access, and native-aware destruction/finalization | Python classes, C aggregate layout, reference counting, detached-copy fallback, or ownership inference | -| plan validation | matching type identity, roles, actions, owners, releases, result positions, and cross-backend handoffs before emission | fallback selection | - -Validation must reject at least: - -- a derived transfer without canonical type identity or an exported runtime - wrapper type; -- a wrapper-address slot whose binding and bridge roles or ABI positions differ; -- a primitive scalar action or datatype family applied to `DERIVED_TYPE`; -- a wrapper-owned result without persistent storage, allocator, destroy action, - or failure cleanup; -- a borrowed wrapper with a destroy action, or without its required native - module/parent owner retention; -- a call-local argument that schedules destruction of the caller's wrapper; -- a visible in-place argument projected as a replacement without completed - policy; -- a hidden output or direct result whose native temporary can escape by - address; -- a plain module proxy without complete typed member-path operations, or an - `Aliased` live module borrow without a completed direct-address handoff; -- binding and bridge module-object access roles that disagree; -- an obsolete `Snapshot` contract, recursive detached-copy action, or backend - fallback that manufactures a detached object; -- a derived array or unsupported polymorphic form entering scalar-derived - lowering; and -- any backend request to infer a class, owner, addressability, or release from - semantic datatype or `intent`. - -### Phase 8A — Contract, Origin, And Post-IR Policy Completion - -Complete the semantic contract before defining direct plan records. - -- [x] Inventory every live scalar derived origin from source and semantic - `.pyi`: constructor-created storage, wrapper-owned result, caller-supplied - argument, native module object, and nested component. -- [x] Introduce one typed completed origin/retention representation shared by - class-instance, argument, result, module-variable, and field policy. - Do not encode origins as ad hoc reason strings. -- [x] Keep generated and edited `.pyi` type identity stable across module - namespaces, imported derived types, renamed Python exports, and same-name - types from different native scopes. -- [x] Complete required, optional, visible `out`, visible `inout`, hidden - output, and direct-result ownership without treating `intent` as the final - Python signature. The editable signature and `@native_call(...)` projection - decide visibility and order; policy only ensures the native call is valid. -- [x] Complete wrapper-owned result storage and destruction, borrowed - module/field owner retention, native setter rejection, result projection, - and failure cleanup before `ir2ast.py`. -- [x] Preserve `Aliased` parsing, printing, and source-derived metadata. Use it - for a live derived-module borrow and direct-address legality, but never as a - native-array extraction mode. -- [x] Complete a plain ordinary module object as `owner=NATIVE`, - `transfer=BORROWED_VIEW`, native-owner destruction, module lifetime, module - owner retention, typed member-path access, and replacement rejection. Do not - claim or require a whole-object native address. -- [x] Complete an `Aliased` module object as `owner=NATIVE`, - `transfer=BORROWED_VIEW`, native-owner destruction, alias storage, module - owner retention, direct address acquisition, and replacement rejection. -- [x] Remove the obsolete public `Snapshot` keyword from `prik.contracts`, - parser, printer, generated `.pyi`, semantic IR, policy actions, legacy - generators, documentation, and fixtures. Do not remove unrelated explicit - copy-result or scalar descriptor value-copy policy. -- [x] Complete finite typed member-path traversal for plain module proxies. - Memoize derived type identities so recursive graphs do not expand forever; - require explicit pointer/allocatable association, ownership, and stale-child - policy at recursive descriptor-backed edges. -- [x] Remove the obsolete wrapper-owned pointer-result blocker. Complete a - persistent typed pointer-holder origin whose wrapper owns the holder but not - its target, then keep only arrays of derived values and unsupported - polymorphic forms on exact planning errors. Supported scalar module - allocatable/`TARGET`/pointer origins use only their explicit Phase 8H - actions. -- [x] Add focused parser, printer, source-conversion, ownership, accessor, - policy-completion and planning tests for every active matrix row and - blocker. Assert the deliberate module-proxy versus direct-address mechanism - distinction, their shared live public behavior, and that neither changes a - contained native handle's view-only extraction. - -### Phase 8B — Derived Plan Records And Preflight Validation - -- [x] Add the minimal namespace-owned opaque derived-type definition record and - derived handoff facets described above. Keep all per-call decisions in - `ArgumentTransferPlan`, `ResultPlan`, or `ModuleVariablePlan`. -- [x] Add an explicit derived datatype-family/type-reference representation so - documentation, roles, native slots, lifecycle records, and printers never - fall through primitive scalar maps. -- [x] Project class instance/self policies, native type identity, wrapper type - symbol, native scope, allocator/destroy roles, and finalizer requirements - mechanically from completed semantic policy. -- [x] Project optional presence, input/in-place/output action, native call - position, ownership, storage, owner retention, and result position into the - existing transfer records. -- [x] Share the exact `DerivedHandoffPlan` object with its owning - `NativeCallSlotPlan` where the array/handle lanes already share subordinate - facets; do not duplicate editable state. -- [x] Add recursive validation for the namespace type definitions, arguments, - results, module variables, module-object access facets, field facets, and - lifecycle indexes. -- [x] Make plan edits observable: changing a derived owner, action, type - identity, retained owner, or release must either change both backend - artifacts consistently or fail `_validate_plan()` before source emission. -- [x] Extend support analysis with precise derived lanes and blockers. Do not - remove the blanket class-owner blocker until the minimal opaque type surface - is direct and every remaining Phase 9 dependency is reported separately. -- [x] Add normal-print plan tests and direct generator preflight tests under - `tests/codegen/test_phase8_derived_types.py`. - -### Phase 8C — Minimal Opaque Wrapper Storage And Lifecycle - -This sub-lane creates the runtime substrate needed to return and pass opaque -objects. It does not implement public construction, fields, or methods by -itself; Phase 8F/H add the public field surface on this substrate. - -- [x] Emit one minimal runtime wrapper type per exported semantic derived type, - with an opaque native address, an owned/borrowed state, and an optional - retained Python owner. Keep the public constructor unavailable until Phase 9. -- [x] Generate bridge allocation and destruction helpers from completed type - policy. Native-aware destruction owns allocatable components and supported - finalization; the binding must not free native storage directly. -- [x] Ensure owned allocation, initialization, and result conversion failures - run native destruction and Python cleanup exactly once. -- [x] Ensure borrowed wrappers never run native destruction, including when - their retained owner is released through cyclic or delayed garbage - collection. -- [x] Register the minimal type in the correct exported namespace so result and - module-variable materialization use the same class identity in source and - generated-`.pyi` builds. -- [x] Keep wrapper struct/type declaration, allocation, owner retention, and - destruction methods grouped under a derived-type comment in the binding; - keep native allocate/associate/destroy helpers grouped likewise in the - bridge. -- [x] Add source-printer and artifact tests for owned, borrowed, failed - allocation, failed conversion, and exactly-once native destruction paths. - -### Phase 8D — Wrapper-Owned Hidden Outputs And Function Results - -- [x] Plan hidden `Return(...)` outputs and direct derived function results as - `WRAPPER_INSTANCE` results with persistent wrapper-owned native storage. -- [x] For hidden output, allocate the result wrapper before the native call and - pass its native address at the declared native slot. On failure, destroy it - before returning the Python error. -- [x] For a function result, move or copy the returned native value into - persistent wrapper-owned storage before the native temporary expires. Never - retain an address into a bridge local. -- [x] Preserve result order and mixed-result aggregation through the existing - `ResultPlan` and lifecycle sequence; do not special-case a derived result in - function/module orchestration. -- [x] Reuse the same result type object and destructor for direct results, - hidden outputs, and edited `Returns[...]` projections. -- [x] Add reduced legacy/direct compiled parity over the existing - `make_point` cases in `test_native_call_examples.py`, - `test_output_arguments.py`, and `test_derived_type_boundaries.py`, inspecting - result storage, slot order, allocation failure, and cleanup artifacts. -- [x] Promote only those reduced generation units after both source and - generated-`.pyi` routes return the correct opaque wrapper and finalization is - proved. Field-based assertions remain on Phase 8F/H until their typed member - operations are complete. - -### Phase 8E — Required, Optional, In-Place, And Caller-Supplied Outputs - -- [x] Accept only the exact completed wrapper type for a concrete derived - argument. Subclass acceptance belongs to completed Phase 9 polymorphic - policy, not normal Python `isinstance` convenience. -- [x] Extract the opaque native address in the binding and pass it through the - single planned role. The bridge associates the matching typed native pointer - and calls the native procedure without copying for ordinary reference - arguments. -- [x] Preserve the same Python wrapper identity for visible `inout` and - caller-supplied `out` arguments. Return it only when the edited projection - requests that sole result; otherwise return `None`. Keep a mixed direct or - hidden result plus visible derived writeback on an exact policy blocker until - general mixed result/writeback aggregation is completed; do not let the - direct route select it and then drop the wrapper identity. -- [x] Represent optional omission and explicit `None` as native absence. A - present wrapper follows the same typed handoff as a required input; no empty - wrapper or call-local default object may be fabricated. -- [x] Keep native slot order independent of normalized Python argument order - and preserve user edits to argument visibility and projection. -- [x] Keep an immutable visible derived replacement on its existing exact - blocker because no passing legacy contract defines its native copy and - finalization semantics. Existing hidden/direct derived outputs use the owned - result path completed in Phase 8D; do not mutate an immutable input or invent - a generic object copy merely to remove the blocker. -- [x] Add focused type-error, optional-presence, wrong-wrapper-class, - in-place-identity, caller-supplied-output, projection, and cleanup tests. -- [x] Add reduced compiled parity that creates a `point` through the Phase 8D - result path, passes it to `point_sum`, mutates it through `move_point`, and - observes the new value through another native call without requiring a - constructor; the follow-on Phase 8F evidence also observes public fields. - -### Phase 8F — Module Objects, Components, And Field Owners - -- [x] Plan every eligible plain rank-zero derived module variable as a - native-owned live module proxy with rejected replacement; plan every - supported `Aliased` equivalent as a native-owned direct-address borrowed - wrapper. Both retain the module and have no destroy action. -- [x] Preserve `Aliased` in generated semantic `.pyi` only when supplied by the - native/source contract. Prove its module-proxy-versus-direct-address lowering - meaning while separately proving that both are live and that it does not - affect any contained native handle's view-only extraction. -- [x] Repeated `Aliased` module reads may create separate Python wrappers, but - every wrapper must refer to the same native object and never claim ownership; - repeated plain reads may create separate proxies, but every proxy must - delegate to the same current native module object. -- [x] Plan a nested derived component as a borrowed wrapper whose retained - owner is the containing wrapper. Releasing the parent name must not destroy - the parent while a child wrapper remains live. -- [x] Ensure a borrowed child never invokes its own native finalizer; releasing - the final child/owner reference triggers the containing owned instance's - destruction exactly once. -- [x] Reuse the Phase 7 `NativeArrayHandlePlan` for allocatable/pointer fields, - changing only origin=`derived_field`, owner retention=`parent_wrapper`, and - the completed field operation roles. Do not create a derived-only handle. -- [x] Plan scalar, string, ordinary-array, nested-derived, and native-handle - field getter/setter handoffs beneath the owning type for both address-backed - and module-backed objects. Use typed bridge procedures rather than C layout - offsets. Phase 8 emits both the typed low-level operations and public property - descriptors, including setter exposure completed by semantic policy. -- [x] Traverse nested value components by finite member paths and type identity. - Memoize recursive type definitions; recursive pointer/allocatable edges use - their completed association and owner policy instead of unbounded flattening. -- [x] Preserve pointer-field target ownership and stale-view rules from - completed pointer policy; parent retention does not make the parent own an - external pointer target. -- [x] Add direct plan/backend lifetime tests, then reduced compiled evidence - for the distinct plain proxy and `Aliased` direct-address origins, plus the - borrowed-finalizer, allocatable-field, and pointer-field fixtures, without - promoting constructor or method surfaces that remain Phase 9. - -### Phase 8G — Exact Native `value` Copies And Opaque Layout - -- [x] Preserve `bind(C)`/`sequence`/ordinary derived-type facts and native - `value` transport through generated `Value(Arg(i))`, post-IR policy, and the - derived handoff plan. Do not store this per-call ABI choice on the annotated - Python type. -- [x] For every supported exact rank-zero monomorphic native `value` argument, - keep Python on the opaque wrapper contract. The Fortran bridge imports the - exact native type, reads the typed pointee, and performs the typed call. The - binding and C boundary never cast, lay out, or byte-copy the aggregate. -- [x] Remove the obsolete requirement that the native type itself be - interoperable. Ordinary, `sequence`, and `bind(C)` exact derived types use - the same Fortran-owned typed-value action; polymorphic or unresolved native - types remain exact blockers for type-identity reasons, not layout guesses. -- [x] Keep ordinary reference arguments and all component access on generated - bridge helpers even when a type is `bind(C)`; interoperability does not turn - fields into a public binary-layout promise. -- [x] Replace the obsolete unsupported-aggregate-layout assertions with - policy, plan, artifact, and compiled tests for ordinary, `sequence`, and - `bind(C)` exact typed value calls. Field-property assertions are Phase 8 - evidence; retain only constructor-dependent assertions in - `test_derived_layout.py` for Phase 9 production promotion. - -### Phase 8H — Direct-Address And Module-Proxy Object Access - -This sub-lane supplies the distinct lowering mechanisms for the two completed -module-object origins in Phase 8A/8F: direct address acquisition for an -`Aliased` live borrow, and typed live member access for a plain module proxy. - -- [x] Add one typed module-object access facet beneath `ModuleVariablePlan`. - Record `DIRECT_ADDRESS` or `MODULE_PROXY`, the native object type, member-path - operations, owner/release behavior, and failure behavior. Do not encode a - backend method name. -- [x] Use the direct path only when completed source/semantic facts make the - native address legal. The bridge exposes the opaque address mechanically; - the binding constructs the borrowed wrapper and retains its module owner. -- [x] For a plain module object, use typed per-field bridge getters/setters and - operations selected by the completed member graph. The binding constructs a - module-retaining proxy with no native destroy action; every read observes - current module state and every permitted write updates it. -- [x] Keep the initial direct-address and module-proxy paths rank-zero, - nonallocatable, nonpointer, noncoindexed, and nonpolymorphic; the explicit - descriptor-backed correction below adds only its named storage origins and - call actions. Record exact blockers for - unsupported type parameters, dynamic types, unresolved recursive pointer - ownership, or any member without a complete live operation. Do not switch - mechanisms as a fallback. -- [x] Validate direct-address roles or module-proxy member-operation coverage, - exported wrapper type identity, owner/release behavior, and - replacement rejection before either backend emits source. -- [x] Prove the `Aliased` address/lifetime premise and plain proxy live - read/write behavior in focused compiled source/generated-`.pyi` tests. -- [x] Remove the `Snapshot` contract name, metadata, recursive copy policy, - generated helper classes, documentation, and snapshot-only fixtures. Do not - retain a compatibility parser, printer, alias, or backend fallback. -- [x] Preserve ordinary result materialization, explicit constant-value - materialization, scalar descriptor value copies, ordinary array copy - results, and any unrelated active transfer action. Their copy semantics are - separate from removed whole-object snapshot behavior. -- [x] Replace the former plain-module snapshot fixture with source/generated- - `.pyi` parity and runtime evidence for live scalar, string, ordinary-array, - allocatable/pointer-handle, and nested-derived member paths, including - recursive-edge blockers and parent/module retention. - -#### Phase 8H Contract Correction — Complete Scalar-Derived Call Matrix - -This correction replaces every earlier isolated module-allocatable, stable -pointer-target, direct-address-only, and interoperable-value proposal with one -complete compatibility matrix. It covers exact rank-zero, monomorphic -`type(item)` objects. Phase 9 still owns `class(item)`, inheritance, and dynamic -dispatch; arrays of derived values remain outside this matrix. - -The actual declaration and its runtime origin are independent axes. The five -actual declaration forms are ordinary, `TARGET`, `ALLOCATABLE`, -`ALLOCATABLE,TARGET`, and `POINTER`; each can be module-owned or represented by -wrapper-owned storage where such storage is meaningful. The six native dummy -forms are: - -| Key | Exact native dummy | -| --- | --- | -| `O` | `type(item) :: arg` | -| `T` | `type(item), target :: arg` | -| `A` | `type(item), allocatable :: arg` | -| `AT` | `type(item), allocatable, target :: arg` | -| `P` | `type(item), pointer :: arg` | -| `V` | `type(item), value :: arg` | - -`OPTIONAL`, rank, and qualified type identity remain separate facts. Source -`INTENT` may propose the initial Python projection, but it is not a completed -matrix selector. For the `P` column, `Pointer(Arg(i))` without a matching -projected return selects a call-local pointer input adapter and discards native -reassociation. A matching `Returns[...]` selects association writeback and -therefore requires persistent pointer storage. prik never selects between these -paths from native `INTENT`. - -Use these completed action names. Parenthesized state requirements are runtime -preconditions, not alternative fallback actions: - -| Action | Meaning | -| --- | --- | -| `DIRECT_REFERENCE` | wrapper-owned or direct module address; reconstruct the exact typed object and pass it by reference | -| `SCOPED_REFERENCE` | originating module synchronously invokes a generic address consumer; the native call completes before the temporary target scope returns | -| `HOLDER_REFERENCE` | reconstruct a persistent typed holder and pass its component directly | -| `MODULE_ADDRESS` | originating module returns `C_LOC` for an explicit durable target | -| `ALLOCATABLE_HOLDER` | pass a persistent wrapper-owned allocatable holder component directly, including unallocated state | -| `MODULE_ALLOCATABLE_TRANSACTION` | move between the module variable and a bridge-local shared typed transaction holder through interoperable holder-address operations | -| `POINTEE_REFERENCE` | pass the current target of a pointer holder or module pointer to a nonpointer dummy | -| `POINTER_HOLDER` | pass a persistent wrapper-owned pointer holder component directly so association writeback updates the same holder | -| `MODULE_POINTER_TRANSACTION` | initialize one bridge-local typed pointer holder from the current target and restore its final association through an interoperable holder-address operation | -| `POINTER_INPUT_ADAPTER` | expose a payload through a call-local pointer carrier because the Python contract does not project pointer association writeback | -| `TYPED_VALUE_COPY` | the exact Fortran bridge passes the typed object into the native `VALUE` slot; C never copies aggregate bytes | -| `INCOMPATIBLE` | language-level storage mismatch; raise the specified `TypeError` and never enter native code | - -`[allocated]` means an allocated value is required. `[associated]` means an -associated pointer target is required. `A`, `AT`, and `P` descriptor calls -accept unallocated or disassociated state where the table does not carry one of -those preconditions. - -| Actual declaration | Origin | `O` | `T` | `A` | `AT` | `P` | `V` | -| --- | --- | --- | --- | --- | --- | --- | --- | -| `type(item) :: var` | non-module | `DIRECT_REFERENCE` | `DIRECT_REFERENCE` with call-scoped target | `INCOMPATIBLE` | `INCOMPATIBLE` | `POINTER_INPUT_ADAPTER` | `TYPED_VALUE_COPY` from direct reference | -| `type(item) :: var` | module proxy | `SCOPED_REFERENCE` | `SCOPED_REFERENCE` with call-scoped target | `INCOMPATIBLE` | `INCOMPATIBLE` | scoped `POINTER_INPUT_ADAPTER` | scoped `TYPED_VALUE_COPY` | -| `type(item), target :: var` | non-module | `DIRECT_REFERENCE` | `DIRECT_REFERENCE` with owner target lifetime | `INCOMPATIBLE` | `INCOMPATIBLE` | `POINTER_INPUT_ADAPTER` | direct `TYPED_VALUE_COPY` | -| `type(item), target :: var` | module | `MODULE_ADDRESS` | `MODULE_ADDRESS` with module target lifetime | `INCOMPATIBLE` | `INCOMPATIBLE` | `POINTER_INPUT_ADAPTER` | module-address `TYPED_VALUE_COPY` | -| `type(item), allocatable :: var` | non-module holder | `HOLDER_REFERENCE [allocated]` | `HOLDER_REFERENCE [allocated]` with holder target lifetime | `ALLOCATABLE_HOLDER` | `ALLOCATABLE_HOLDER` | holder `POINTER_INPUT_ADAPTER [allocated]` | holder `TYPED_VALUE_COPY [allocated]` | -| `type(item), allocatable :: var` | module | `SCOPED_REFERENCE [allocated]` | `SCOPED_REFERENCE [allocated]` with call-scoped target | `MODULE_ALLOCATABLE_TRANSACTION` | `MODULE_ALLOCATABLE_TRANSACTION` with call target | scoped `POINTER_INPUT_ADAPTER [allocated]` | scoped `TYPED_VALUE_COPY [allocated]` | -| `type(item), allocatable, target :: var` | non-module holder | `HOLDER_REFERENCE [allocated]` | `HOLDER_REFERENCE [allocated]` with holder target lifetime | `ALLOCATABLE_HOLDER` | `ALLOCATABLE_HOLDER` | holder `POINTER_INPUT_ADAPTER [allocated]` | holder `TYPED_VALUE_COPY [allocated]` | -| `type(item), allocatable, target :: var` | module | `MODULE_ADDRESS [allocated]` | `MODULE_ADDRESS` with module target lifetime | `MODULE_ALLOCATABLE_TRANSACTION` preserving target | `MODULE_ALLOCATABLE_TRANSACTION` preserving target | `POINTER_INPUT_ADAPTER [allocated]` | module-address `TYPED_VALUE_COPY [allocated]` | -| `type(item), pointer :: var` | non-module holder | `POINTEE_REFERENCE [associated]` | `POINTEE_REFERENCE [associated]` with retained target owner | `INCOMPATIBLE` | `INCOMPATIBLE` | `POINTER_HOLDER` | pointee `TYPED_VALUE_COPY [associated]` | -| `type(item), pointer :: var` | module | module `POINTEE_REFERENCE [associated]` | module `POINTEE_REFERENCE [associated]` with native target owner | `INCOMPATIBLE` | `INCOMPATIBLE` | `MODULE_POINTER_TRANSACTION` | module-pointee `TYPED_VALUE_COPY [associated]` | - -An `Aliased` ordinary module object follows `MODULE_ADDRESS` instead of -`SCOPED_REFERENCE`, but its original target-lifetime fact still controls whether -a native pointer may outlive the call. This does not change `Aliased` array-view -semantics. - -The matrix is exhaustive for this Phase 8 scope. Every cell becomes either one -completed action or one deliberate language-level error before lowering. No -backend may infer a different action from datatype, `intent`, module shape, -address presence, or local memory checks. - -The table's `P` entries show the non-projecting input-adapter form. When the -Python contract projects pointer association writeback, replace every -nonpointer `P` cell with `INCOMPATIBLE`; the two pointer-storage rows retain -`POINTER_HOLDER` and `MODULE_POINTER_TRANSACTION`. - -##### Shared Holder And Callback ABI - -Define these support types once per qualified native derived type and import -the same definitions in every producer, origin operation, and consumer: - -```fortran -type :: item_allocatable_holder - type(item), allocatable :: value -end type - -type :: item_pointer_holder - type(item), pointer :: value => null() -end type -``` - -A persistent wrapper-owned holder is allocated through a Fortran pointer and -its opaque holder address is stored by the Python wrapper. Its nonpointer -allocatable component is a targetable subobject of the persistent holder -target, so the same carrier supports both `A` and `AT`; do not invent a second -allocatable-target holder. - -Module allocation and pointer transactions use bridge-local holder objects -declared `TARGET`. The module-specific operations are interoperable -`BIND(C)` procedures taking only `type(C_PTR), value :: holder_address` plus -interoperable status/context values. Each operation reconstructs the exact -shared holder with `C_F_POINTER` and performs `MOVE_ALLOC` or pointer assignment -entirely in Fortran. The binding transports a typed function pointer and an -opaque holder address; no allocatable or pointer descriptor crosses C. - -The old proposal to pass `type(item), allocatable` or `type(item), pointer` -directly through a runtime C callback is removed as noninteroperable. The old -proposal to avoid a transaction holder for module allocation/pointer restore is -also removed. A bridge-local transaction holder is the portable carrier; it is -not a persistent replacement for the originating module variable. - -For a module allocatable transaction, the bridge performs the equivalent of: - -```fortran -type(item_allocatable_holder), target :: transaction - -status = move_out(c_loc(transaction)) -if (status == PRIK_STATUS_OK) then - call native_procedure(transaction%value) - restore_status = move_back(c_loc(transaction)) -end if -``` - -`move_out` executes `move_alloc(module_value, transaction%value)` and -`move_back` executes `move_alloc(transaction%value, module_value)`. A successful -move-out makes the module variable unavailable until restoration. When the -module actual has `TARGET`, both destinations preserve pointer association; -when it lacks `TARGET`, aliases created through a temporary target have only -call lifetime. - -For a module pointer transaction, the bridge initializes -`transaction%value` from the current `C_LOC`/`C_NULL_PTR`, passes that component -to the native pointer dummy, and invokes `restore_pointer(c_loc(transaction))`. -The origin reconstructs the pointer holder and executes -`module_pointer => transaction%value`. The final nullification, -reassociation, allocation, or deallocation is therefore visible in the module -pointer. - -Operation tables use typed C function-pointer fields; do not round-trip a -function pointer through `void *`. The proxy retains its originating extension -until every active scoped call or transaction has unwound. - -##### Pointer Target Ownership - -A pointer holder owns the holder and association variable, not its target. -Default scalar-derived pointer target ownership is native: holder destruction -nullifies the component and deallocates only the holder. It must never -deallocate an unowned target. When final association matches a known module, -parent, or wrapper-owned target, retain that owner in completed policy; an -otherwise durable native target retains the originating extension and remains -the native program's release responsibility. Native code that returns a pointer -to an expired local target violates the contract rather than creating an prik -fallback. - -This completed owner/release rule removes the old wrapper-owned pointer-result -blocker. Reassociation is supported, but it never silently transfers target -ownership to Python. Public documentation must warn that a native pointer saved -through a wrapper-owned target remains valid only while the wrapper and target -allocation remain alive. - -##### Multiple Scalar-Derived Arguments - -Do not generate `2**N` native call branches. Build one call context with one -slot per native argument and an ordered acquisition program: - -1. validate every Python wrapper, exact qualified type, storage capability, - allocation/association precondition, optional presence, and pointer-target - owner before entering any native origin operation; -2. retain all Python/module owners and acquire module transaction guards in a - deterministic total order; -3. deduplicate repeated actual identities so one module allocation or pointer - is checked out once and its holder/address can feed multiple native slots; -4. move out module allocatables in deterministic order, rolling back already - moved values in reverse order if a later acquisition fails; -5. initialize module pointer transaction holders; -6. enter all `SCOPED_REFERENCE` producers as a nested continuation chain, - storing each address in the context; and -7. invoke the native procedure exactly once after every slot is ready, then - unwind scoped producers, pointer restorations, allocation restorations, - guards, and retained owners in reverse order. - -If the same actual appears in multiple slots and any corresponding dummy may -define it while another slot references or defines it, reject the call before -checkout unless completed `INTENT` facts prove the aliasing legal. Read-only -duplicates share one acquisition. Never move the same module allocatable twice -or restore the same module pointer through independent locals. - -The generic scoped-address consumer ABI remains -`consumer(object_address, context) -> status`. The context carries all earlier -addresses, holders, ordinary arguments, result slots, and the first error. A -consumer never retains `object_address`; multiple module variables are handled -by nesting producers, not by generating one origin-module cross product per -native procedure. - -##### Error And Cleanup Contract - -Use one status protocol across scoped consumers and module transaction -operations. Do not raise a Python exception, `longjmp`, or unwind C++ through a -Fortran frame. Record status and any Python exception data in the call context, -return normally through every producer, complete cleanup, and only then raise -in the binding. - -- wrong qualified wrapper type, an incompatible matrix cell, or a known - reassociable pointer dummy receiving nonpointer storage raises `TypeError` - before native entry; -- a required ordinary/target/value/pointer-input actual whose allocatable is - unallocated or pointer is disassociated raises `ValueError` before native - entry; -- `A`, `AT`, and `P` descriptor calls preserve valid unallocated or - disassociated state and do not reinterpret it as optional omission; -- only an omitted Python argument or explicit `None` for an optional contract - selects native absence; a present empty handle never becomes omitted by - accident; -- an active recursive/concurrent transaction raises `RuntimeError` before the - affected origin changes state; -- every successful move-out has exactly one attempted move-back on every - normally returning path, and every native module-pointer call has exactly one - attempted association restore; -- cleanup continues in reverse order after the first restoration failure so - independent origins are not stranded; the first failure is reported with - later cleanup failures attached as context; -- a failed restoration leaves its origin guard poisoned instead of advertising - a usable proxy, and raises `RuntimeError` after all other cleanup attempts; -- conversion, result allocation, and Python-object creation that can fail are - completed before checkout where possible; failures after native return still - restore every transaction before propagating; and -- process termination, `ERROR STOP`, signals, or invalid native pointers are - not recoverable wrapper exceptions. The documentation must state that this - cleanup guarantee covers paths that return through the generated ABI. - -The per-origin guard must be thread-safe, or the binding must prove that the -GIL remains held for the complete transaction and that no callback re-entry is -possible. An unsynchronized Fortran `logical` is not a sufficient concurrency -guard. Internal synchronous address consumers are Phase 8 bridge machinery; -they do not expose the public callback semantics deferred to Phase 10. - -##### Implementation And Proof Checklist - -- [x] Preserve actual declaration attributes, module/non-module origin, - `TARGET` lifetime, allocatable/pointer state, exact type identity, and - pointer-dummy `INTENT` authority through parsing, semantic IR, and edited or - generated `.pyi` round trips. -- [x] Replace the former category/action-only contract with completed facets - capable of representing all six dummy forms and every action in the matrix. - `DerivedDummyCategory` remains the completed declared-form label and - `DerivedCallAction` remains the completed selected-action label; neither is - allowed to stand in for the lifetime, access, failure, cleanup, target-owner, - or release facets. The complete record includes `ALLOCATABLE,TARGET`, typed - value, target lifetime, pointer-input - validation, transaction cleanup, and target owner/release. Remove - `RUNTIME_POINTER_TARGET`, the module-allocatable incompatibility, and all old - fallback/rejection actions they made obsolete. -- [x] Complete every matrix decision in post-IR policy before `ir2ast.py`. - Binding and bridge generation only dispatch named actions; neither backend - inspects datatype, `intent`, module shape, address presence, or allocation - state to select a different mechanism. -- [x] Generate one shared allocatable holder and pointer holder per qualified - native type, with persistent create/destroy helpers and bridge-local - transaction use. Prove source/generated-`.pyi` bundles import the identical - holder definition and reject ABI/type mismatch before reconstruction. -- [x] Generate scoped-address producer operations for plain module objects and - non-`TARGET` allocated module allocatables, direct address operations for - durable module targets, move-out/move-back holder-address operations for - module allocatables, and current-target/restore holder-address operations for - module pointers. -- [x] Implement the ordered multi-argument acquisition/unwind program, - deduplicated origin identity, legal read-only aliasing, reverse rollback, - poisoned restoration failures, and a single final native invocation. -- [x] Implement the exact Python error mapping and optional/empty-state rules - above. Add injected failures before first acquisition, after one of several - acquisitions, during scoped nesting, after native return, and during each - cleanup category. -- [x] Remove the interoperable-`bind(C)` restriction from typed derived - `VALUE` calls. The Fortran bridge must perform the exact typed call without a - C aggregate cast, byte copy, layout promise, or detached-object fallback. -- [x] Support wrapper-owned pointer results with a persistent pointer holder, - native target ownership by default, explicit known-owner retention, direct - association writeback, and holder-only destruction. Remove the old blanket - target-ownership blocker rather than retaining it as a compatibility path. -- [x] Update public and maintainer documentation to teach the five actual - declarations, six dummy forms, complete matrix, direct versus scoped - address acquisition, holder and module transactions, `INTENT(IN)` pointer - exception, target lifetime, native pointer-target ownership, multi-argument - nesting, errors, and cleanup. Examples must show more than one scalar-derived - argument and link back to one canonical explanation instead of repeating - incomplete fragments. -- [x] Add one comprehensive native fixture at - `tests/data/fortran/wrapper/fscalar_derived_actual_dummy_matrix_f90.f90`, its - reduced source/generated contract under - `tests/wrapper/fortran/derived_types/contracts/fscalar_derived_actual_dummy_matrix_phase8/`, - focused policy/plan/artifact tests in - `tests/codegen/test_phase8_scalar_derived_actual_dummy_matrix.py`, and - compiled tests in - `tests/wrapper/fortran/derived_types/test_scalar_derived_actual_dummy_matrix.py`. - Replace the earlier proposed separate module-allocatable and - module-target/pointer fixtures; do not retain tests that assert their old - rejection paths. -- [x] Make that native fixture a complete Fortran module containing all five - module actual declarations, wrapper-owned ordinary/allocatable/pointer - producers, all six dummy forms, both pointer `INTENT(IN)` and reassociable - pointer procedures, two qualified native types with the same short name, - optional arguments, injected operation failures, and state-reset helpers. -- [x] Parameterize policy/plan tests over every matrix cell. Every legal cell - must select its one completed action; every incompatible cell must assert its - exact pre-native `TypeError`; allocated/unallocated and - associated/disassociated states must assert their exact `ValueError`, valid - descriptor call, or optional-absence behavior. -- [x] Compiled tests must exercise mixed calls containing several - scalar-derived arguments. Include at least: multiple nested scoped module - objects; two module allocatable transactions plus a module pointer - transaction; mixed direct, holder, scoped, allocatable, pointer, target, and - value slots in one native call; repeated read-only actual identity; rejected - writable duplicate identity; failure after the first of several checkouts; - reverse restoration; native deallocation/reallocation; pointer - nullification/reassociation/allocation/deallocation; and owner retention. - Phase 8 cannot close if the new compiled procedures test only one derived - argument at a time. -- [x] Run the portable ABI fixture with the supported GNU toolchain and every - available secondary compiler in the development environment. The proof must - cover scoped `C_FUNPTR` consumers, `C_PTR` transaction holders, - `C_F_PROCPOINTER`, holder targetability, target-preserving `MOVE_ALLOC`, and - the accepted-`INTENT(IN)`/rejected-reassociable pointer distinction. - -### Phase 8I — Production Routing, Regression, And Completion - -- [x] Add separate support-report lanes for derived inputs, optional derived - inputs, in-place derived arguments, wrapper-owned derived results, plain - module proxies, `Aliased` borrowed module objects, borrowed field owners, - and the exact typed-value slice. -- [x] Replace the isolated scalar-derived descriptor routes with one - dependency-closed actual/dummy-matrix lane only after every unchecked Phase - 8H row passes. It must cover direct and scoped references, target adapters, - allocatable and pointer holders, module allocation and association - transactions, exact typed values, and multi-argument acquisition/unwind. - No old call-incompatible, nonreassociating-only, or interoperability-only - compatibility route may remain selectable. -- [x] Add one deliberate legacy/direct parity node for every dependency-closed - Phase 8 lane and append it to the production rollout evidence only after its - generated artifacts and runtime behavior match. -- [x] Update the migration matrix row for each reduced unit. Keep broad units - containing constructors, methods, inheritance, or callbacks on - their explicit Phase 9/10 policy limitations until whole-generation-unit - planning is complete. -- [x] Treat source and generated-`.pyi` default field constructors as Phase 9 - class-surface blockers. Do not select the Phase 8 route merely because the - generated constructor was consumed into origin metadata rather than retained - as a semantic method; reduced opaque Phase 8 contracts must explicitly - suppress construction. -- [x] Prove an eligible opaque-derived generation unit selects the production - wrapper-plan route and no longer invokes `semantic_ir_to_codegen_ast()`. -- [x] Keep direct plan edits meaningful across both backends and preserve the - global no-fallback rule when a derived type, owner, release, or field action - is incomplete. -- [x] In every relevant planner, validator, binding generator, and bridge - generator, keep scalar, string, ordinary-array/native-handle, and - derived-type lowering methods in consistent groups with one short comment - above each group. Preserve typed object-kind/action matching; grouping must - not introduce datatype inference or a second dispatcher. -- [x] Run focused parser/printer, ownership/policy, plan/validation, - binding/bridge/printer, and runtime tests; relevant source/generated-`.pyi` - wrapper parity; and regressions for scalar, string, array, and Phase 7 handle - lanes. -- [x] Run the wrapper suite excluding the deferred LAPACK coverage, the wrapper - codegen complexity checker, documentation checks, whitespace check, and the - required static-analysis suite before closing implementation. -- [x] Run the comprehensive Phase 8 scalar-derived actual/dummy matrix policy, - artifact, and multi-argument compiled tests; all retained holder and Phase - 7/8 regressions; the wrapper suite excluding LAPACK; documentation and - whitespace checks; the wrapper complexity checker; and the required static - suite after the replacement route is implemented. -- [x] Close Phase 8 only when every supported rank-zero non-polymorphic derived - input/result/module transfer is direct, both plain module-proxy and `Aliased` - address-backed module-object paths are direct, live member operations and - recursive-edge policy are validated before emission, and - every remaining class-surface/callback/derived-array case has an exact Phase - 9/10 or unsupported-policy blocker. - -### Phase 8 Implementation Evidence - -- Post-IR origin, identity, handoff, ownership, field, lifecycle, and exact - blocker evidence lives in - `tests/codegen/test_phase8_derived_types.py`, with supporting parser, - printer, source-conversion, ownership, and planning suites named in - `tests/wrapper/CHECKLIST_COVERAGE.md`. -- Public-field validation is split into named completed-policy, descriptor, - typed object-kind, and setter checks so no single semantic-policy routine - becomes a second backend-style dispatcher. -- Compiled legacy/source and direct-plan evidence lives in - `tests/wrapper/fortran/derived_types/test_phase8_derived_plan.py`. It covers - required, optional, in-place, caller-supplied output, ordinary and `bind(C)` - typed native `value`, direct/hidden owned result, module-proxy, - direct-address module object, constant value, field, owner-retention, - allocation/cleanup artifact, and exactly-once finalization behavior. -- The former isolated scalar-derived descriptor evidence in - `tests/codegen/test_phase8_scalar_derived_descriptors.py`, - `tests/wrapper/fortran/derived_types/test_scalar_derived_descriptor_plan.py`, - and `tests/data/fortran/wrapper/fscalar_derived_descriptors_f90.f90` is - superseded by the comprehensive policy/artifact and compiled matrix files - named in Phase 8H. They cover all 60 declaration/dummy cells, empty states, - qualified same-short-name identities, `sequence` typed values, holder and - module transactions, multi-origin unwind, pointer target ownership, injected - cleanup failures, and the exact retained incompatibilities. No obsolete - module-allocatable rejection, stable-pointer-only, or wrapper-pointer-result - blocker remains as negative compatibility coverage. -- `prik/pipeline/build.py` registers the dependency-closed Phase 8 support - lanes and their passing production evidence. The automatic-route test - replaces `semantic_ir_to_codegen_ast()` with a failure sentinel and proves an - eligible opaque-derived unit never invokes it. -- Constructors, methods, properties beyond the completed field descriptors, - inheritance, and polymorphic class orchestration remain Phase 9. Public - callbacks remain Phase 10. Arrays of derived values, non-scalar holder member - operations, recursive value edges without completed descriptor policy, - unresolved imported types without an exact runtime definition, immutable visible - derived replacement, and mixed native result plus visible-writeback envelopes - carry exact unsupported-policy blockers instead of selecting a fallback. - Internal synchronous scoped-address consumers and module transactions are - Phase 8 implementation machinery, not deferred public callbacks. - -Historical Phase 8 closure evidence before the module-allocatable and -module-pointer restore redesigns (2026-07-15): all 39 focused Phase 8 -plan/compiled tests, the 711-test -cross-stage regression batch, all 79 runtime-handle tests, 1,133 documentation -and layout tests, and all 329 wrapper tests outside the deferred full -BLAS/LAPACK file passed. The wrapper complexity checker, Ruff lint/format, -Bandit, Vulture, whitespace, and explicit-`origin/main` Radon policy passed; -the advisory full Radon complexity and maintainability reports were also -produced. The required `--base-ref auto` Radon invocation could not resolve -CI-only base-SHA variables locally, and the explicit-base rerun passed. No -LAPACK test was run locally. This evidence does not close the reopened Phase 8H -rows. - -Final Phase 8H/I closure evidence (2026-07-15): the focused Phase 8 plus route- -ledger batch passed 180 tests; the affected cross-stage semantic, lowering, -runtime-handle, planner, and backend batch passed 611 tests; and the complete -wrapper suite outside the deferred combined BLAS/LAPACK file passed 445 tests. -Documentation checks passed 1,123 tests and whitespace validation passed. The -GNU toolchain compiled and ran the complete matrix suite; Intel `ifx` 2026.1.0 -compiled, linked, and ran the same generated ABI for `sequence` typed values, -mixed six-form input, module allocatable/pointer transactions, target-preserving -`MOVE_ALLOC`, and the accepted-input/rejected-reassociable pointer distinction. -The wrapper complexity checker, Ruff lint/format, Bandit, Vulture, explicit- -`origin/main` Radon policy, and advisory Radon complexity/maintainability runs -passed. The CI-only `--base-ref auto` Radon lookup was unavailable locally, so -the required explicit-base rerun was used. No LAPACK test was run locally. - -### Phase 8 Expansion Gate - -- [x] Inventory the live semantic contract, post-IR ownership policy, active - snapshot paths to remove, legacy binding/bridge paths, plan-route blockers, - public docs, checked `.pyi` fixtures, and real wrapper tests. -- [x] Separate origin, owned/borrowed lifetime, module address acquisition, - input/result/field/module-state use, destruction, owner retention, and - recursive member-path access into dependency-ordered Phase 8A-I sub-lanes. -- [x] Record the strict Phase 8/9/10 boundaries and identify reduced existing - native units that can prove opaque transfers without first migrating public - constructors, methods, inheritance, or callbacks. Public field descriptors - are part of Phase 8. -- [x] Begin Phase 8 implementation only from Phase 8A and keep every later - sub-lane blocked on its declared dependencies. - -## Phase 9 — Classes, Constructors, And Methods - -Expansion status: complete. Implementation status: complete. The direct class -path is covered by policy, plan-edit, artifact, compiled runtime, production -routing, and broad non-LAPACK wrapper-suite evidence below. - -Scope: generated Python class objects, namespace registration, default and -keyword constructors, explicit constructor bindings, constructor overloads, -instance and static methods, type-bound dispatch, method overloads, finalizer -attachment, inheritance, and the first supported scalar polymorphic input -dispatch. Phase 9 assembles those public class surfaces on the opaque storage, -field descriptors, handoffs, and lifetime rules completed in Phase 8. - -### Phase 9 Boundary And Explicit Non-Scope - -Phase 9 may compose completed Phase 8 records but must not revisit them. -Constructor and method policy may select how an instance is created or passed; -it may not change object origin, storage kind, field access, owner retention, -release, nullability, native setter assignment, or destruction. A class plan -references the namespace-owned `DerivedTypePlan` and its field plans rather -than copying or rendering them. - -The following surfaces are in Phase 9: - -- one generated Python type object for each public supported semantic class, - with stable native identity and explicit Python base identity; -- an explicitly present or deliberately absent public constructor surface; -- generated default/keyword field initialization for eligible public scalar - fields, including omitted-keyword preservation of native defaults; -- direct `@bind("native_name")` constructors and explicit constructor overload - candidates linked to concrete native procedures; -- passed-object type-bound instance methods, non-type-bound methods attached to - the class by the semantic contract, and supported `@staticmethod` methods; -- class-owned overload sets with exact candidate signatures and deterministic - runtime selection; -- owned-instance finalization through the Phase 8 destroy/release path and - borrowed-instance non-destruction; -- Python inheritance for supported Fortran extension types; and -- scalar, input-only polymorphic calls whose accepted runtime class set and - concrete native dispatch targets are fully enumerated before lowering. - -The following remain outside Phase 9: - -- callbacks, adapters, trampolines, and callable lifetime; these remain Phase - 10 even when a callback argument/result is a derived object; -- module-level generic/operator migration units that do not require a class - surface; those remain in Phase 11, although they may reuse the same overload - candidate and runtime-match vocabulary; -- arrays of derived or polymorphic values, elemental class dispatch, and - partial construction/destruction of array elements; -- polymorphic results, mutable polymorphic dummies, allocatable/pointer - polymorphic scalars, unlimited polymorphism, abstract instantiation, - deferred-binding execution, and runtime extension types not enumerated in - the semantic module; -- any unresolved Phase 8 storage blocker merely because a constructor or - method happens to use that type; Phase 9 reuses the completed allocatable and - pointer holders and must not invent a second storage path; -- generic constructor selection whose candidates are indistinguishable at the - Python boundary; and -- compatibility aliases, synthesized legacy entrypoints, string-built backend - method names, or a fallback from an incomplete class plan to legacy class - lowering. - -### Phase 9 Existing Oracle And Inventory - -The legacy route plus existing source/generated-`.pyi` runtime assertions are -the behavioral oracle. Capture complete binding, bridge, header, and runtime -evidence before each reduced direct-plan slice. Correct unsafe behavior only -when the documented contract says so; do not preserve legacy architecture. - -| Existing unit | Phase 9 behavior to preserve | Required reduced slice | -| --- | --- | --- | -| `derived_types/test_constructors_and_finalizers.py::test_fortran_default_constructor_keywords_and_finalization[*]` | default construction, keyword-only scalar fields, native defaults, invalid-call cleanup, and exactly-once finalization | default/keyword constructor plus owned destroy path | -| `derived_types/test_derived_type_methods.py::test_modern_fortran_derived_type_exposes_class_and_type_bound_methods[*]` | instance methods, explicit binding names, scalar arguments/results, class static factory, and Phase 7 handle fields | split `vector` methods from `vector_store` handle methods and static factory | -| `derived_types/test_borrowed_finalizers.py::test_borrowed_child_wrapper_never_finalizes_native_component[*]` | borrowed child retains parent; only the owned parent finalizes | class assembly over the completed Phase 8 borrowed-field owner path | -| `derived_types/test_derived_layout.py::test_bind_c_derived_types_use_accessors_and_fortran_value_copy[*]` | default class creation and methods coexist with opaque field access and typed native value copy | class surface only; Phase 8 retains layout and handoff ownership | -| `derived_types/test_inheritance.py::test_fortran_extension_types_generate_python_inheritance[*]` | Python subclass relationships, inherited field/method access, overridden methods, unbound base calls, and scalar polymorphic input dispatch | base/extension class graph first, polymorphic call second | -| `tests/fortran/derived_types/semantics/test_pyi_class_semantics.py` and `tests/fortran/generic_interfaces/semantics/test_pyi_overload_semantics.py` | generated versus bound constructors, removed constructors, direct constructor targets, explicit overload links, type-bound root targets, and invalid metadata diagnostics | semantic-policy fixtures before planner/backend work | -| `../../fortran/pyi_contracts/functions_and_classes/end_to_end/test_edited_class_surfaces.py` | edited contracts can remove constructors/methods/candidates and add explicit bindings without resurrecting source declarations | absence/export validation and source/generated/edited parity | -| `naming/test_defined_operators.py` and `naming/test_generic_interfaces.py` | exact candidate matching and Python export naming | reuse candidate-match vocabulary; broad module/generic units remain Phase 11 | - -Inventory these legacy owners without importing them into the direct package: - -- `prik/semantics/ir2ast.py` currently interprets constructor overloads, - passed-object positions, type-bound names, polymorphic variants, and class - insertion. Each semantic decision found there must move into post-IR class - policy before direct lowering. -- `prik/codegen/bindings/c_to_python.py` currently assembles type objects, - constructors, methods, overloads, properties, inheritance, module exports, - and finalizers. Reuse emitted behavior as the oracle, not its broad control - flow or method-name synthesis. -- `prik/codegen/bridges/fortran_to_c.py` currently supplies typed constructor - allocation, passed-object association, method calls, overload interfaces, - and finalization helpers. Direct bridge generation must consume completed - class/method actions and reuse Phase 8 native storage helpers. -- Generated semantic `.pyi` class declarations are a public contract. A - consumed default constructor still counts as a constructor surface and must - remain a whole-unit Phase 9 route requirement. - -### Phase 9 Plan Shape And Action Vocabulary - -Extend the existing namespace plan; do not introduce a rendered-class layer or -a second function plan. - -- Add one namespace-owned `ClassSurfacePlan` (name illustrative, not - prescriptive) that references exactly one `DerivedTypePlan`, its Python - exports, optional base-class identity, constructor plan, ordered methods, - ordered overload sets, type-object slots, and module-registration action. -- Add a `ConstructorPlan` with an explicit kind: `ABSENT`, - `DEFAULT_FIELDS`, `BOUND_PROCEDURE`, or `OVERLOAD_SET`. Record allocation - action, accepted Python parameters, native target/call slots, initialized - fields, omitted-field behavior, cleanup action, and success transition. -- Reuse `FunctionPlan` for each concrete method or constructor target. Add only - a class-call facet recording method kind, passed-object position, self - storage requirement, result attachment, public descriptor flags, and the - owning class identity. -- Add an `OverloadSetPlan` containing public export, overload kind, ordered - concrete candidate references, typed runtime predicates, ambiguity result, - no-match diagnostic, and selected native target. Candidate predicates use - exact dtype/rank/derived-class facts already completed by argument plans. -- Add an `InheritancePlan` containing canonical base identity, storage - compatibility, inherited/overridden method ownership, Python base type - symbol, and module initialization dependency order. -- Add a `PolymorphicDispatchPlan` only for supported scalar input calls. It - enumerates accepted concrete class identities and a concrete `FunctionPlan` - variant for each; it must not rediscover subclasses from runtime object names. -- Keep destructor selection on the referenced Phase 8 derived handoff/release - plan. Phase 9 records only which class slot invokes that existing action and - which constructor failure edges need cleanup. - -Stable semantic action names must describe behavior, not backend function -names. At minimum distinguish: - -- class registration: `CREATE_TYPE`, `SET_BASE`, `READY_TYPE`, `EXPORT_TYPE`; -- construction: `OMIT`, `ALLOCATE_DEFAULT`, `ALLOCATE_AND_ASSIGN_FIELDS`, - `CALL_BOUND_CONSTRUCTOR`, `DISPATCH_CONSTRUCTOR`, `REJECT_CONSTRUCTION`; -- method binding: `INSTANCE`, `STATIC`, and explicit unsupported class-method - policy until a real class-method contract exists; -- passed-object handoff: `WRAPPER_ADDRESS`, `BORROWED_ADDRESS`, or the exact - completed Phase 8 storage action; -- overload selection: `MATCH_EXACT`, `SELECT_CANDIDATE`, `NO_MATCH`, - `AMBIGUOUS`; and -- construction lifecycle: `ALLOCATE`, `INITIALIZE`, `COMMIT_OWNER`, - `CLEANUP_UNCOMMITTED`, `DESTROY_OWNED`. - -### Mandatory Phase 9 Migration Algorithm - -For every dependency-closed sub-lane: - -1. Capture one passing source/generated-`.pyi` legacy unit and its complete - class, binding, bridge, header, and runtime assertions. -2. Complete class export, constructor kind, method kind, passed-object policy, - overload candidates, inheritance, polymorphic accepted set, allocation, - commit, cleanup, and destruction before `ir2ast.py`. -3. Project those facts into the existing namespace, derived-type, function, - lifecycle, and native-slot plans plus the smallest class-specific facets. -4. Validate the complete class graph and every cross-backend symbolic role - before either backend emits source. -5. Lower through small named methods selected only by typed actions. Backend - local temporaries may implement a selected action but cannot choose policy. -6. Compare generated artifacts with the oracle and record intentional - differences before compiling. -7. Add focused policy, plan-edit, validation, printer, binding, bridge, - source/generated-`.pyi`, edited-contract, and compiled runtime tests. -8. Promote production routing only after the reduced unit passes direct-plan - runtime parity and no class-surface fallback remains. - -### Phase 9A — Semantic Class-Surface Completion - -- [x] Add completed post-IR policy records for public class identity, exports, - constructor kind, method kind, passed-object position, overload ownership, - base identity, type-object registration, and construction permissions. -- [x] Preserve explicit absence: an edited `.pyi` that removes `__init__`, a - method, or an overload candidate must produce an absent plan entry and cannot - resurrect source behavior. -- [x] Move any class-surface inference still in `ir2ast.py` into policy - completion. Wrapper planning must report the owner path and exact missing decision. -- [x] Add policy tests for generated, bound, removed, overloaded, inherited, - abstract, and invalid class surfaces before planner changes. - -### Phase 9B — Typed Class Plan And Validation - -- [x] Add the namespace-owned class, constructor, method-call, overload, and - inheritance plan facets described above, each referencing existing Phase 8 - type/field/lifetime plans rather than copying them. -- [x] Project Python/native names and export aliases once. Do not synthesize - backend method names from strings or recover native targets by scanning - emitted functions. -- [x] Validate unique class/type identity, base-before-derived order, one - constructor kind, method ownership, passed-object position, native call-slot - agreement, field-plan identity, lifecycle roles, and module export symbols. -- [x] Add direct plan-edit tests proving invalid constructor, method, base, - overload, or lifecycle references fail before emission in both backends. - -### Phase 9C — Class Creation And Module Registration - -- [x] Emit one Python type object per supported class, attach the completed - Phase 8 field descriptors, set the validated base type, ready the type, and - export every completed Python name in dependency order. -- [x] Keep opaque instance storage identical to Phase 8 wrapper storage. Class - assembly must not add C aggregate layout, component offsets, or a second - native owner field. -- [x] Attach the Phase 8 destruction action only to owning classes; borrowed - proxies and nested objects retain owners and never gain independent destroy - slots. -- [x] Add artifact and compiled reduced tests for an opaque constructible class, - an intentionally nonconstructible class, a borrowed child, and exact module - export identity. - -### Phase 9D — Default And Keyword Field Constructors - -- [x] Build constructor parameters only from fields explicitly eligible in - completed constructor policy. Preserve keyword-only behavior and native - default component initialization for omitted fields. -- [x] Allocate the Phase 8 persistent native instance first, apply validated - field assignments through existing field setter actions, then commit wrapper - ownership only after every step succeeds. -- [x] On parse, conversion, allocation, or field-assignment failure, clean up - the uncommitted native instance exactly once. Failed `tp_init` must not leak, - double-finalize, or expose a partially initialized wrapper. -- [x] Replay `fconstructors_f90` for default, partial, complete, positional, - unknown-keyword, native-default, and finalization-count assertions through - both source and generated-`.pyi` contracts. - -### Phase 9E — Explicit And Overloaded Constructors - -- [x] Represent direct `@bind("native_name")` construction as one constructor - action linked to a concrete function plan. It replaces, rather than wraps or - falls back to, the generated field constructor. -- [x] Represent constructor overloads as an explicit constructor-owned overload - set. Do not combine `@overload` and `@native_call`, and do not reinterpret a - normal method overload as `tp_init`. -- [x] Complete allocation-before-call versus native-produced-instance policy, - result attachment, owner commit, failure cleanup, and exactly-once release for - every candidate before lowering. -- [x] Reject indistinguishable candidates, missing targets, incompatible self - types, mixed constructor kinds, or ambiguous edited declarations during - policy or plan validation, never from candidate trial calls. -- [x] Add isolated semantic and compiled fixtures for direct bound construction, - two distinguishable constructor candidates, no-match, ambiguity, target - failure cleanup, and source/generated/edited-contract parity. - -### Phase 9F — Instance And Static Methods - -- [x] Lower passed-object instance methods from the completed self position and - Phase 8 handoff. Preserve native argument order when `self` is not the first - native slot. -- [x] Support explicit binding names and type-bound root-target metadata without - exporting the private concrete target as a duplicate module function. -- [x] Lower supported static methods without fabricating `self`; attach them to - the type object with their completed export and descriptor flags. -- [x] Reuse ordinary function argument/result plans for scalar, string, array, - handle, and derived transfers. A method cannot widen an unsupported ordinary - call lane. -- [x] Replay reduced `fclasses_f90` vector methods first, then `vector_store` - handle methods and static factory, with exact source/generated-`.pyi` runtime - and artifact parity. - -### Phase 9G — Class-Owned Overload Dispatch - -- [x] Complete ordered candidates and exact runtime predicates for each - class-owned overload set. Candidate selection may inspect only typed Python - argument facts named by the plan, never invoke candidates speculatively. -- [x] Reuse one overload matching vocabulary for constructors, methods, - operators, and later Phase 11 module generics while keeping their owners and - call actions distinct. -- [x] Detect indistinguishable signatures before emission and produce stable - no-match diagnostics listing the public overload and accepted signatures. -- [x] Validate native target, Python export, argument/result plans, passed-object - position, and overload kind across binding and bridge views. -- [x] Add focused method-overload tests for primitive kinds, ranks, derived - subclasses, keyword normalization, exact no-match, and ambiguity; keep broad - defined-operator/module-generic promotion in Phase 11. - -### Phase 9H — Finalization And Constructor Failure Safety - -- [x] Route normal owned-instance deallocation, constructor failure, and - native-constructor failure through the same Phase 8 destroy/release action, - guarded by an explicit uncommitted/committed lifecycle state. -- [x] Prove finalization occurs exactly once for successfully constructed - owners, once for native storage allocated before a rejected constructor call, - and never for borrowed children or native-owned module objects. -- [x] Prove child-to-parent retention survives method/property access and that - deleting the parent first delays only the parent's owning finalizer. -- [x] Replay `fconstructors_f90` and `fborrowed_finalizer_f90`, including forced - Python argument failures and repeated garbage collection. - -### Phase 9I — Inheritance And Scalar Polymorphic Input Dispatch - -- [x] Complete canonical base/extension relationships, storage compatibility, - inherited fields, inherited methods, overrides, Python base symbols, and - module initialization order before planning. -- [x] Construct base and derived wrappers with the same Phase 8 opaque storage - contract while preserving exact runtime type identity and safe unbound base - method calls on derived instances. -- [x] For each supported scalar input-only polymorphic dummy, enumerate the - accepted concrete class identities and one concrete native call variant per - identity. Reject unknown or abstract runtime classes before the native call. -- [x] Keep polymorphic results, mutable dummies, arrays, descriptor-backed - polymorphic scalars, unlimited polymorphism, and unenumerated extensions on - exact blockers; inheritance must not silently widen them. -- [x] Replay `finheritance_f90` for `issubclass`, `isinstance`, inherited field - access, override dispatch, unbound base calls, and base/circle/box - polymorphic inputs through source and generated-`.pyi` routes. - -### Phase 9J — Production Routing, Documentation, And Closure - -- [x] Add support-report lanes for class registration, default constructors, - bound constructors, constructor overloads, instance methods, static methods, - class overloads, finalizers, inheritance, and scalar polymorphic input. -- [x] Add one reduced compiled direct-plan node per dependency-closed lane, then - update its migration-matrix row only after artifact and runtime parity. -- [x] Prove eligible class units select the production wrapper-plan route and - never call `semantic_ir_to_codegen_ast()`; an unsupported class decision must - keep the whole generation unit on one exact blocker without partial fallback. -- [x] Synchronize constructor/method/inheritance user docs, semantic `.pyi` - reference, source map, feature matrix, subject README, and checklist coverage - with the implemented class contract. -- [x] Run focused policy/plan/backend tests, all affected existing class wrapper - nodes through source/generated-`.pyi` modes, the wrapper suite excluding - LAPACK, the wrapper complexity checker, documentation checks, whitespace, - and the required static-analysis suite. -- [x] Close Phase 9 only when every supported constructor/method/inheritance - unit routes directly, all Phase 8 field/storage/lifecycle decisions remain - unchanged, and every remaining callback, derived-array, polymorphic, or - ambiguous-overload case has an exact Phase 10/11 or unsupported-policy - blocker. - -Closure evidence (2026-07-16): focused semantic, lowering, routing, and direct -Phase 8-10 plan tests passed 184 tests after the final policy refactor. The -complete local wrapper suite excluding LAPACK passed 449 tests in source and -generated-contract modes. The wrapper complexity checker, Ruff lint/format, -Bandit, Vulture, explicit-`origin/main` Radon policy, and advisory Radon -complexity/maintainability commands passed. The CI-only `--base-ref auto` -Radon lookup could not resolve outside CI, so the required explicit-base run -was used. No LAPACK test was run locally. - -### Phase 9 Expansion Gate - -- [x] Inventory class creation/destruction, constructor categories, - instance/static/type-bound methods, overloads, inheritance/polymorphism, - decorator effects, module initialization, legacy owners, semantic fixtures, - and passing runtime oracles. -- [x] Define the Phase 8/9/10/11 ownership boundaries and keep all class - implementation rows unchecked. -- [x] Split implementation into dependency-ordered Phase 9A-J sub-lanes with - explicit policy, plan, validation, lowering, artifact, compiled parity, - production routing, documentation, and closure gates. - -## Phase 10 — Callbacks And Trampolines - -Expansion status: complete. Implementation status: complete. Immediate -callbacks are covered by focused policy/plan/artifact tests, existing compiled -runtime oracles, production routing, and broad non-LAPACK wrapper-suite -evidence below. - -Scope: immediate callback argument validation, call-scoped context lifetime, -external Fortran adapter procedures, C trampolines, scalar/string/array/derived -argument and result conversion, permissive reference writeback, same-thread -re-entry and GIL handling, callback cleanup, and the documented fatal error -boundary. - -### Phase 10 Boundary And Explicit Non-Scope - -Phase 10 composes ordinary call transfers completed in Phases 2-9 but does not -reinterpret them. A prototype is interface-facing: it describes the exact -procedure declaration that native Fortran uses, including argument order, -`In`/`Out`/`InOut` direction, value/reference transport, rank, shape, character -length, result representation, and procedure characteristics. Normal wrapper -projection and callback adapter projection remain distinct completed records. - -Named `@prototype` declarations are the single exact native-signature -authority. Annotation use selects a callback signature; call use selects a -directly callable standalone procedure entity. -`In(T)`, `Out(T)`, and `InOut(T)` preserve exact dummy direction, while -`Addr(T)` and `Value(T)` preserve transport independently. `@pure` preserves -the corresponding procedure characteristic. Prototypes are semantic-only -declarations and never become Python runtime exports. - -A pure prototype is not a supported Python callback signature. The callback -adapter calls the Python runtime and therefore cannot satisfy Fortran purity; -post-IR policy must block a prototype used both as a specification function and -as a callback before planning. - -Post-IR policy classifies each use as a callback, a standalone procedure entity, -or a module-procedure call. One shared prototype-signature plan owns the -generated `prik_` abstract-interface symbol and exact characteristics. Lowering -only declares callback adapters or concrete entities with -`procedure(prik_...)`; it does not reconstruct placement, purity, direction, -transport, or declaration mode. Direct prototype calls never fall back to an -implicit external declaration, and `@standalone` is rejected on a prototype as -redundant placement metadata. - -The supported callback contract is deliberately call-scoped: - -- the Python callable is validated and retained before the native call, placed - in one thread-local context stack for that callback site, and released after - the native call returns; -- nested callback-taking calls on the same entering Python thread are allowed; -- each C trampoline validates the entering thread, acquires the GIL, converts - completed adapter arguments, invokes the current Python callable, converts - or copies back results, releases the GIL, and returns to its Fortran adapter; -- `Value(T)` uses value conversion; scalar reference storage, fixed-length - character storage, arrays, and derived objects use permissive copy-in/out - storage already asserted by the runtime tests; and -- a Python exception, invalid callback return, missing context, or cross-thread - invocation prints the Python error and aborts the host process. The direct - path must not fabricate a fallback result or continue native execution. - -The following remain outside Phase 10: - -- stored callbacks, callback registration/unregistration, procedure-pointer - fields, callbacks invoked after the wrapped call, optional dummy procedures, - null procedure pointers, asynchronous callbacks, and cross-thread callback - execution; -- persistent callable ownership, callback teardown during object/library - destruction, and callback use as a synchronization mechanism; -- callbacks whose signature is incomplete, assumed-rank, has a runtime-only - character length, or otherwise lacks the exact ABI facts required by the - adapter and trampoline; -- callback-specific coercion, recovery, exception-result, or argument - reordering policies not present in the public contract or legacy tests; and -- module generic/operator orchestration that merely contains a callback-taking - candidate; its callback transfer may be reusable, but public generic routing - remains Phase 11. - -### Phase 10 Existing Oracle And Inventory - -The public callback guide/reference, generated semantic `.pyi` contracts, and -existing source/generated-`.pyi` runtime assertions are the behavioral oracle. - -| Existing unit | Phase 10 behavior to preserve | Required reduced slice | -| --- | --- | --- | -| `tests/fortran/callbacks/end_to_end/test_scalar_callbacks.py::test_immediate_scalar_dummy_procedure_calls_python_callback[*]` | scalar result/void callbacks, callable validation, balanced references, nested same-thread re-entry, held-GIL wrapper envelope, thread-local context, and fatal callback conversion failures | first context, trampoline, scalar-value, cleanup, and fatal-boundary slice | -| `tests/fortran/callbacks/end_to_end/test_array_callbacks.py::test_immediate_dummy_procedure_converts_array_arguments_and_results[*]` | writable array view, shaped array result, outer-output identity, and reference writeback | array argument/result slice | -| `tests/fortran/callbacks/end_to_end/test_supported_callback_shapes.py::test_immediate_callbacks_cover_all_supported_argument_shapes[*]` | scalar values, fixed strings, arrays, derived values, non-scalar reference writeback, and one combined call envelope | cross-kind and derived closure slice | -| `tests/fortran/callbacks/pipeline/test_generated_callback_contracts.py` | named prototypes, primitive value defaults, explicit primitive `Addr(T)` references, non-primitive `Value(T)` transport, shape, character storage, cross-module identity, and result annotations round-trip exactly | semantic-contract parity slice | -| `tests/fortran/callbacks/semantics/test_pyi_callback_semantics.py` | prototype declarations and references, primitive `Addr(T)`, non-primitive `Value(T)`, exact argument names used by shapes, and invalid prototype transport forms | policy completion before planner work | - -### Phase 10 Plan Shape And Action Vocabulary - -Extend the existing argument/function plans; do not add a second function plan -or embed a legacy AST. - -- Add one `CallbackHandoffPlan` facet to each callable argument. It records the - callable owner, call-scoped lifetime, context symbol, context stack action, - entering-thread rule, GIL rule, Fortran adapter symbol, C trampoline symbol, - ordered callback argument plans, optional result plan, and fatal-error - action. -- Add a `CallbackTransferPlan` for each callback argument/result containing the - semantic type identity, object kind, value/reference ABI, rank/shape/length - roles, Python barrier action, primitive-scalar value projection or non-scalar - reference writeback, borrowed-owner retention, and exact C ABI roles. -- Reuse ordinary scalar, string, array, and derived plan vocabulary where the - representation is identical. The native callback is the caller, so normal - Python-to-native argument projection cannot be silently reused in reverse. -- Add ordered function lifecycle phases `VALIDATE_CALLBACK`, `PUSH_CONTEXT`, - `ENTER_NATIVE`, `POP_CONTEXT`, and `RELEASE_CALLBACK`. Every failure edge - before native entry unwinds acquired references; the fatal trampoline edge - never returns. -- Keep backend-local adapter locals and temporary Python views inside the - selected implementation method. They are emitted-code details, not semantic - policy. - -Stable actions must describe behavior, not generated function names. At -minimum distinguish: - -- callable/context: `VALIDATE_CALLABLE`, `RETAIN_CALLABLE`, `PUSH_CONTEXT`, - `POP_CONTEXT`, `RELEASE_CALLABLE`; -- callback ABI: `VALUE`, `REFERENCE`, `DATA_AND_SHAPE`, - `DATA_AND_LENGTH`, and `DERIVED_ADDRESS`; -- adapter transfer: `COPY_IN`, `COPY_OUT`, `COPY_IN_OUT`, `BORROW_READ_ONLY`, - and `BORROW_WRITABLE`; -- trampoline runtime: `REQUIRE_ENTERING_THREAD`, `ACQUIRE_GIL`, `CALL_PYTHON`, - `RELEASE_GIL`, and `ABORT_WITH_PYTHON_ERROR`; and -- result handling: `RETURN_SCALAR`, `RETURN_ARRAY_ADDRESS`, - `RETURN_DERIVED_ADDRESS`, `RETURN_VOID`, and `REJECT_RESULT`. - -### Mandatory Phase 10 Migration Algorithm - -For every dependency-closed sub-lane: - -1. Capture the documented behavior, one passing source/generated-`.pyi` - legacy unit, and its callback-related binding, bridge, adapter, trampoline, - and runtime assertions. -2. Complete callable validity, signature order, ABI roles, reference - writeback/value isolation, - shape/length dependencies, result handling, context lifetime, thread/GIL - rules, cleanup, and fatal behavior before wrapper planning. -3. Project those facts into the existing function/argument/lifecycle plans plus - the smallest callback-specific facets. -4. Validate the binding, bridge, adapter, and trampoline role graph before - either backend emits source. -5. Lower through typed action dispatch and small named methods. Do not trial a - callback or infer shape/transport from emitted locals. -6. Compare emitted artifacts and behavior with the runtime oracle; document - any safety improvement before changing observable behavior. -7. Add focused policy, editable-plan, validation, binding, bridge, printer, - source/generated-`.pyi`, subprocess-failure, and compiled runtime tests. -8. Promote production routing only after the complete callback-taking - generation unit passes direct-plan parity with no callback fallback. - -### Phase 10A — Semantic Callback Completion - -- [x] Add completed post-IR callback records for callable signature order, - object kind, value/reference ABI, shape/length roles, result representation, - call scope, context lifetime, same-thread rule, GIL rule, cleanup, and - fatal-error behavior. -- [x] Preserve generated and edited named prototypes exactly. Reject an - incomplete prototype reference, invalid prototype `Addr`, optional procedure, - stored/procedure-pointer lifetime, unavailable mandatory native interface, - or unsupported result with the owner path and one exact reason. -- [x] Complete callback signature/result/ownership policy before wrapper - planning; lowering may only project the completed callback record. -- [x] Add policy/planning tests for primitive value defaults, explicit - primitive `Addr(T)` references, non-primitive `Value(T)`, - and retained unsupported forms before planner changes. - -### Phase 10B — Typed Callback Plan And Validation - -- [x] Add callback handoff, transfer, result, context, and lifecycle facets to - the existing function plan and reference ordinary datatype plans instead of - copying them. -- [x] Project adapter/trampoline symbols and ABI roles once. Do not synthesize - backend handler names or rediscover dimension/length dependencies from - emitted variables. -- [x] Validate unique callback sites, exact argument order, role availability, - transport/writeback, dtype/rank/shape/length agreement, derived type identity, - result compatibility, context balance, and validate/push/pop/release order. -- [x] Add direct plan-edit tests proving invalid callback roles, unbalanced - lifecycle, or cross-backend disagreement fail before emission. - -### Phase 10C — Context, Trampoline, GIL, And Scalar Values - -- [x] Emit one thread-local stack per callback site, callable validation and - strong-reference retention before native entry, reverse-order pop/release - after return, and cleanup on every ordinary pre-entry failure. -- [x] Emit one C trampoline and separately linked external Fortran adapter from - the completed ABI; validate the entering thread and context before Python - conversion. -- [x] Acquire/release the GIL inside the trampoline and keep the outer - callback-taking wrapper on the legacy-observed held-GIL envelope. -- [x] Lower void and scalar-value arguments/results first, then replay scalar - callback success, nested re-entry, non-callable rejection, and balanced - reference-count assertions in both build modes. - -### Phase 10D — Primitive Scalar Values And Fixed-String Storage - -- [x] Lower every primitive scalar callback argument as an owned NumPy scalar - value. `Value(T)` changes only the native ABI; scalar reference writeback is - unsupported and must be modeled as a callback result. -- [x] Lower fixed-string references as rank-zero fixed-width bytes storage with - exact length, padding, and writeback. The semantic annotation remains - `String[n]` and carries no native direction. -- [x] Reject runtime-length callback strings before emission; no adapter-local - inference may change the representation. -- [x] Replay the scalar-value and string-storage cases from the combined - callback fixture through source/generated-`.pyi` routes. - -### Phase 10E — Array Arguments And Results - -- [x] Lower array callback arguments from completed dtype, rank, shape, - ordering, contiguity, and alignment facts. Reference arrays expose writable - storage and copy back in adapter order. -- [x] Lower fixed-shape array results through one validated returned-address - ABI and assign them into the native adapter result. Reject incomplete shape - or unsupported ownership before emission. -- [x] Preserve output-array Python identity in the outer ordinary call and do - not add a detached-copy fallback. -- [x] Replay `fcallback_array_f90` plus the combined array-storage callback and - artifact assertions in both build modes. - -### Phase 10F — Derived Arguments And Results - -- [x] Reuse the exact Phase 8/9 type identity, opaque wrapper, owner-retention, - and destroy/release actions for callback-local derived wrappers. Do not - expose aggregate layout or introduce callback-specific storage ownership. -- [x] Borrow callback input wrappers only for the callback invocation; convert - supported callback results to the completed native result storage and - release temporary wrapper ownership exactly once. -- [x] Validate exact runtime class/type identity before using a returned - derived address. Polymorphic, descriptor-backed, or unsupported derived - callback forms retain exact blockers. -- [x] Replay `fcallback_derived_f90` and the combined derived callback after the - Phase 9 constructor/class route is green. - -### Phase 10G — Fatal Errors, Re-entry, And Cleanup - -- [x] Route Python exceptions, argument-call mismatch, invalid callback result, - missing context, and cross-thread entry through one - traceback-plus-`abort()` action. Never return a fabricated value. -- [x] Prove nested same-thread callback calls use stack discipline and restore - the previous callable/context after the inner call. -- [x] Prove ordinary validation or setup failures before native entry release - every retained reference, and successful calls leave the callable reference - count unchanged. -- [x] Run fatal cases in subprocesses for both source/generated-`.pyi` builds - and assert the documented traceback/error text plus nonzero termination. - -### Phase 10H — Production Routing, Documentation, And Closure - -- [x] Add support-report lanes for callback context, scalar value/storage, - fixed strings, arrays, derived values, result conversion, same-thread - re-entry, and fatal errors. -- [x] Add one reduced compiled direct-plan node per dependency-closed lane and - update its migration-matrix row only after artifact and runtime parity. -- [x] Prove eligible callback units select the production wrapper-plan route; - unsupported callback policy must keep the whole generation unit on one - exact blocker. -- [x] Synchronize callback guide/reference, semantic `.pyi` reference, feature - matrix, callback README, source map, and checklist coverage with the direct - implementation. -- [x] Run focused policy/plan/backend tests, every callback wrapper node in - source/generated-`.pyi` modes, the wrapper suite excluding LAPACK, the - wrapper complexity checker, documentation checks, whitespace, and the - required static-analysis suite. -- [x] Close Phase 10 only when every supported immediate callback unit routes - directly, no callback plan falls back after generation starts, and every - stored/optional/asynchronous/cross-thread or incomplete callback form has an - exact retained blocker. Stop before Phase 11. - -Closure evidence (2026-07-16): callback policy, editable-plan validation, -binding/bridge artifacts, scalar/string/array/derived conversion, nested -same-thread re-entry, reference cleanup, and subprocess fatal-boundary tests -all passed through the direct route. The same 184-test focused batch and -449-test non-LAPACK wrapper replay used for Phase 9 closure cover the complete -immediate-callback matrix. Required static checks passed with the explicit -Radon base noted above, and implementation stopped before Phase 11. - -### Phase 10 Expansion Gate - -- [x] Inventory the public callback contract, semantic prototype records, - legacy lowering/codegen owners, source/generated-`.pyi` runtime fixtures, - context lifetime, re-entry/GIL behavior, exception/abort behavior, and every - supported scalar/string/array/derived argument-result combination. -- [x] Define the Phase 9/10/11 boundary and retain explicit blockers for stored, - optional, asynchronous, cross-thread, incomplete-signature, and unsupported - callback forms. -- [x] Split implementation into dependency-ordered Phase 10A-H sub-lanes with - policy, typed plan, validation, lowering, compiled parity, production - routing, documentation, and closure gates. - -## Phase 11 — Cross-Cutting Wrapper Suite Completion - -Implementation status: complete. The pre-Phase-11 ledger contained 236 -wrapper-plan nodes, five dual-route array parity nodes, 113 passing legacy-route -nodes, 95 non-generating nodes, and two deferred real-library nodes. The final -forced-plan sweep passed 435 of 449 non-real-library nodes before obsolete -dual-route artifact assertions were removed; its two shared implementation -gaps were Fortran-ordered strided ndarray validation and static `nopass` method -dispatch, both now resolved through existing policy/runtime paths. - -The ordered output aggregator now combines direct and hidden native results -with visible scalar, string, array, and derived writeback. It converts each -value once in public result order and releases every earlier Python reference -if a later conversion or tuple allocation fails; the former single-result and -"native result plus writeback" blockers are removed. - -Scope: existing wrapper tests whose generation units combine completed semantic -lanes or exercise build and runtime behavior rather than introducing one new -datatype lane. - -Implement in these dependency-ordered waves: - -1. reconcile the five reduced array dual-route nodes and remove stale Phase 7 - exclusion bookkeeping where their completed actual-source policy now permits - production routing; -2. migrate source/semantic-`.pyi` build modes, edited contracts, external - symbols, multiple-source linkage, and independent native bundles through one - shared route and planner; -3. migrate mixed scalar/string/array/handle/derived/module/class generation - units without adding per-test or per-datatype fallback; -4. migrate naming, generic interfaces, defined operators, OpenMP/runtime policy, - and remaining public-surface orchestration; and -5. require the live nondeferred ledger to contain only `wrapper-plan` or - justified `not-applicable` nodes before Phase 12 begins. - -- [x] Reconcile every remaining `legacy` or `dual-route` matrix row by owning - test area: `build_from_source`, `build_from_pyi`, `edit_pyi_contracts`, - `external_routines`, `multiple_files`, `naming`, `runtime_behavior`, and the - full BLAS/LAPACK examples. -- [x] Group remaining rows into dependency-ordered waves by their actual - unsupported owner paths. Do not implement a broad test directory as one - special case and do not add per-test backend fallbacks. -- [x] For every newly discovered semantic or backend gap, expand the applicable - earlier lane or add an explicit sub-lane here, then follow the complete - policy -> plan -> backend -> emission -> compiled parity -> route sequence. -- [x] Prove source-driven and semantic-`.pyi`-driven builds use the same route - selector and wrapper planner while retaining their existing build assertions. -- [x] Prove edited-policy contracts, external symbols, multiple-source builds, - naming/generic interfaces, runtime policies, recursion, OpenMP, and real - library-independent native bundles preserve their existing assertions - through the wrapper-plan route. -- [x] Keep non-wrapper-generating tests, including layout and generated-`.pyi` - checks, marked `not-applicable` to route selection but passing in the same - suite. -- [x] Run every `tests/wrapper` test except - `test_real_blas_lapack.py` locally and in CI as the pre-cutover gate. -- [x] Finish this phase only when every nondeferred matrix row is either - `wrapper-plan` or justified `not-applicable`; no nondeferred row may remain - `legacy` or `dual-route`. BLAS/LAPACK rows remain - `deferred-real-library` until Phase 12. - -Closure evidence (2026-07-16): the Phase 11 ledger contains 344 canonical -wrapper-plan nodes, 95 justified non-generating nodes, two deferred -real-library nodes, and no legacy or dual-route node. The complete local -pre-cutover suite outside the shared BLAS/LAPACK file passed all 439 collected -tests. Mixed outputs use the ordered aggregator, Fortran-ordered strided array -validation reuses the shared array-actual runtime path, and static `nopass` -methods reuse the completed class invocation path; no per-test route or -backend fallback was added. - -## Phase 12 — Cutover And Removal - -Implementation status: complete. Local BLAS evidence is recorded below; -LAPACK execution remains intentionally CI-only. - -Local verification boundary: run the BLAS generation unit locally. Do not run -the LAPACK generation unit locally; make its wrapper-plan invocation runnable -in GitHub Actions and use that job for LAPACK parity and cutover evidence. - -External-interface parameter lists preserve native ABI order, while their -declarations may be topologically ordered from the plan's explicit array -extent-reference roles. This permits a later scalar extent dummy to be -declared before an earlier array dummy without reordering the native call. - -Cutover contract: source builds, semantic-`.pyi` builds, Makefile generation, -manifest replay, and strict-name validation all use completed policy -> -`WrapperPlan` -> `WrapperCodeGenerator`. The build API has no route selector, -rollback flag, or silent fallback; an unsupported owner path fails before any -backend or legacy lowering runs. - -- [x] Re-audit collected Python test nodes under `tests/wrapper` and reconcile - them with the migration matrix. No test may be missing from the matrix. -- [x] After every other migration row is complete, restore the full - `test_real_blas_lapack.py` run and any required native-cache preparation in - local opt-in verification and GitHub Actions. -- [x] Run BLAS locally through the canonical route using its existing contract, - import, ABI, and runtime assertions. Run the equivalent exact LAPACK node in - the dedicated GitHub Actions real-library matrix; do not run it locally. -- [x] Require every wrapper-generating test row to be `wrapper-plan`; no row - remains `legacy`, `dual-route`, or `deferred-real-library`. -- [x] Configure the complete `tests/wrapper` suite in CI with ordinary tests in - the main matrix and the full BLAS/LAPACK nodes in the cached real-library - matrix. -- [x] Confirm no wrapper build lane uses the old - `semantic_ir_to_codegen_ast()` path. The old lowering is no longer a supported - test owner and receives no focused compatibility coverage. -- [x] Remove route support tracking and fallback diagnostics; whole-generation - units now either validate and generate one plan or fail on exact owner-path - support diagnostics before emission. -- [x] Retain rollback only until the live ledger is reconciled, then remove it - in one cutover without compatibility flags or per-function fallback. -- [x] Do not move modified isolated nodes or printers back into the legacy - package during migration. After final cutover, remove the legacy package - pieces proven unused and keep `prik.codegen` as the canonical - generator rather than performing a second package rename. -- [x] Keep semantic `.pyi` emission under `prik.codegen.printers` and - retire focused tests of the old semantic AST, bridge, binding, and printer - implementation before deleting the legacy package. -- [x] Remove the temporary legacy route and its route diagnostics after every - live generation unit is supported; do not replace it with compatibility - shims or per-function fallback. -- [x] Remove migration-only dual-route orchestration after the complete existing - wrapper suite proves the wrapper-plan route and legacy rollback is no longer - supported. Keep the existing behavioral fixtures and assertions. -- [x] Keep source printers only for the remaining generated source fragments they - still own, or replace them with narrower emitters once the model layer is no - longer needed. - -Closure evidence (2026-07-16): the final live ledger contains 346 canonical -wrapper-plan nodes, 75 justified non-generating nodes, and zero legacy, -dual-route, or deferred nodes. The complete local suite outside the shared -real-library file passed 419 tests; the exact BLAS full-library node passed -locally; and the exact BLAS and LAPACK nodes are runnable as independent legs -of the cached GitHub Actions real-library matrix. LAPACK was intentionally not -run locally, so its runtime result remains CI evidence. Focused semantic and -compiled class/module policy tests passed 80 tests, all wrapper-codegen tests -passed 352 tests, and documentation plus structural layout checks passed 1,142 -tests. Ruff lint/format, Bandit, Vulture, the wrapper-codegen complexity check, -the Radon policy against explicit base `main`, advisory Radon complexity and -maintainability reports, and `git diff --check` all passed. - -## Verification - -- [x] Documentation changes run - `python3 -m pytest -q tests/docs` - and `git diff --check`. -- [x] Wrapper-plan code changes run the affected existing `tests/wrapper` nodes, - the minimal intermediate contract tests required above, and the required - static-analysis suite from `AGENTS.md`. -- [x] Wrapper-codegen implementation changes pass - `python3 tools/check_codegen_complexity.py` with no handler waiver. -- [x] Runtime wrapper tests cover every changed generated behavior. -- [x] Every migrated lane completed legacy-oracle comparison before cutover; - final tests now exercise only the canonical wrapper-plan route and retain the - existing behavior and ABI-relevant call assertions. -- [x] Structural dependency tests prove complete generator isolation: no - imports from `prik.codegen` to `prik.codegen` or in the reverse - direction. -- [x] BLAS and LAPACK full-library wrapper tests remained excluded locally and in - GitHub Actions throughout Phases 0-11. At the explicit Phase 12 gate, enable - BLAS locally and in GitHub Actions, enable LAPACK only in GitHub Actions, and - keep local LAPACK execution disabled. - -## Completion Record - -- [x] The final report for each lane names the plan actions added, the binding - and bridge handlers they dispatch to, and the handoff specs validated. -- [x] No unsupported wrapper lane uses old lowering/codegen; focused tests now - target completed policy, `WrapperPlan`, `WrapperCodeGenerator`, or compiled - public behavior rather than `ir2ast.py` and `prik.codegen` internals. -- [x] The final cutover report includes the completed `tests/wrapper` migration - matrix and confirms every wrapper-generating row uses the wrapper-plan route. -- [x] The final report includes focused verification commands and results. -- [x] The final report includes the changed-stage breakdown required by - `AGENTS.md` and names every test file added or updated with the behavior it - covers. - -## Post-Cutover Legacy Codegen Removal - -The legacy `prik.codegen` package, `prik/semantics/ir2ast.py`, and the obsolete -`prik/compiling/python_wrapper.py` pipeline are removed together. No alias, -fallback, compatibility import, or rejection-only test preserves that route. - -Required behavior remains with its current owner: completed semantic policy -tests for semantic decisions, `tests/codegen/` for plans and direct -source generation, and compiled `tests/wrapper/` cases for public Python -behavior and native ABI outcomes. Static-analysis baselines cover only source -that remains in the repository. -## Session Continuation Protocol - -The stable continuation prompt is: - -```text -Continue implementing the wrapper-plan migration checklist. -``` - -On continuation: read this checklist and `AGENTS.md`; inspect the dirty -worktree; choose the first unchecked dependency-closed item; replay the -existing passing wrapper test before extending a lane; implement code and tests -together; run required verification; and check items only from live evidence. -Do not reset unrelated user changes, infer missing policy in lowering, or use a -new fallback after direct plan generation starts. diff --git a/docs/old_docs/developper_guide.md b/docs/old_docs/developper_guide.md index 5c6ae2284..a93dc685e 100644 --- a/docs/old_docs/developper_guide.md +++ b/docs/old_docs/developper_guide.md @@ -204,10 +204,10 @@ implementation files. | Generated target datatype mapping examples | `prik/type_mapping_report.py` | `tests/tools/test_type_mapping_report.py`, `tests/tools/test_documentation_examples.py` | | Fortran to semantic IR | `prik/semantics/fortran2ir.py`, `prik/semantics/models.py` | `tests/semantics/test_fortran2ir.py` | | C to semantic IR | `prik/semantics/c2ir.py`, `prik/semantics/models.py` | `tests/semantics/test_c2ir.py` | -| `.pyi` printing | `prik/codegen/printers/pyi_printer.py` | `tests/semantics/test_pyi_printer.py`, `tests/semantics/test_pyi_printer_modern_example.py` | +| `.pyi` printing | `prik/printers/pyi.py` | `tests/semantics/test_pyi_printer.py`, `tests/semantics/test_pyi_printer_modern_example.py` | | `.pyi` loading/editing | `prik/pyi_parser/parser.py` | `tests/pyi/test_pyi_to_ir.py`, `tests/pyi/test_pyi_fixture_suite.py` | | Fortran wrapper orchestration | `prik/wrapping.py` | `tests/wrapper/fortran/native_build/test_build_modes.py`, `tests/wrapper/fortran/multi_source/test_multi_source_builds.py` | -| Wrapper planning and owner-local errors | `prik/codegen/planner.py` | `tests/codegen/` | +| Wrapper planning and owner-local errors | `prik/planning/planner.py` | `tests/codegen/` | | Semantic IR to codegen AST | `prik/semantics/ir2ast.py` | `tests/semantics/test_ir2ast.py`, `tests/wrapper/` | | Fortran-to-C bridge and CPython binding | `prik/codegen/bridges/fortran_to_c.py`, `prik/codegen/bindings/c_to_python.py` | `tests/wrapper/` subject suites | | Native compilation and binding support | `prik/compiling/`, `prik/binding_support/` | `tests/wrapper/fortran/native_build/test_runtime_abi.py`, `tests/wrapper/fortran/native_build/test_build_modes.py` | @@ -258,7 +258,7 @@ module-level function only to preserve an old internal call path. ### `.pyi` Contract Internals User-visible `.pyi` syntax is parsed by `prik/pyi_parser/parser.py` and printed -by `prik/codegen/printers/pyi_printer.py`. Both operate on `prik/semantics/models.py`. +by `prik/printers/pyi.py`. Both operate on `prik/semantics/models.py`. Important implementation rules: @@ -715,7 +715,7 @@ The main ownership boundaries are: - `prik/codegen/bridges/fortran_to_c.py`: Fortran-to-C ABI adaptation; - `prik/codegen/bindings/c_to_python.py`: Python argument/result conversion, reference handling, and CPython wrapper construction; -- `prik/codegen/printers/{fcode,ccode,cpythoncode}.py`: source rendering only; +- `prik/printers/{c,fortran,pyi}.py`: language source rendering only; - `prik/compiling/`: compiler commands and shared-library linking; and - `prik/binding_support/`: native binding support copied into each build. @@ -784,9 +784,9 @@ from `prik/semantics/models.py`. reserved home for local variables or local constants if a frontend later promotes them into semantic IR; local bindings are not emitted into `.pyi` or treated as wrapper interface items by default. -- `prik/codegen/printers/pyi_printer.py` emits editable user contracts. +- `prik/printers/pyi.py` emits editable user contracts. - `prik/pyi_parser/parser.py` loads edited contracts back into semantic IR. -- `prik/semantics/policy_completion.py` completes the decisions required for +- `prik/policy/completion.py` completes the decisions required for wrapping. Keep semantic IR stable where possible. If a parser change does not affect the @@ -990,7 +990,7 @@ Example target: add a new `Annotated[...]` metadata item or projection helper. 1. Add loader tests in `tests/pyi/test_pyi_to_ir.py`. 2. Update `prik/pyi_parser/parser.py`. 3. Add printer tests in `tests/semantics/test_pyi_printer.py`. -4. Update `prik/codegen/printers/pyi_printer.py`. +4. Update `prik/printers/pyi.py`. 5. Update semantic models in `prik/semantics/models.py` only if the IR needs a new field or constraint. 6. Update policy completion or wrapper planning if the syntax changes a diff --git a/docs/user/examples/recipes/compiler-preprocessing.md b/docs/user/examples/recipes/compiler-preprocessing.md index 9d19e9bbf..e0d20a810 100644 --- a/docs/user/examples/recipes/compiler-preprocessing.md +++ b/docs/user/examples/recipes/compiler-preprocessing.md @@ -2,7 +2,7 @@ title: Use Compiler Preprocessing Options audience: users, developers prerequisites: installation, native project compiler flags -related: ../../../developer/compiler-preprocessing.md, ../../../developer/c-parser-reference.md, ../../../developer/fortran-parser-reference.md +related: ../../../developer/packages/preprocessing.md, ../../../developer/deferred/c-parser.md, ../../../developer/packages/parsers.md status: maintained publication: draft --- @@ -49,5 +49,5 @@ PRIK_C_DOCS_END --> ## Next -- Read the [compiler preprocessing reference](../../../developer/compiler-preprocessing.md) +- Read the [preprocessing package guide](../../../developer/packages/preprocessing.md) for the pipeline model, adapters, diagnostics, and include-exposure policy. diff --git a/docs/user/examples/recipes/inspect-c-api.md b/docs/user/examples/recipes/inspect-c-api.md index d68b2f27b..a4681db0b 100644 --- a/docs/user/examples/recipes/inspect-c-api.md +++ b/docs/user/examples/recipes/inspect-c-api.md @@ -3,7 +3,7 @@ title: Deferred Native API Inspection audience: users, developers prerequisites: installation -related: ../../../developer/c-parser-reference.md +related: ../../../developer/deferred/c-parser.md status: maintained publication: draft --- diff --git a/docs/user/examples/recipes/use-python-inspection-apis.md b/docs/user/examples/recipes/use-python-inspection-apis.md index 4292c92d2..1951225fa 100644 --- a/docs/user/examples/recipes/use-python-inspection-apis.md +++ b/docs/user/examples/recipes/use-python-inspection-apis.md @@ -19,7 +19,7 @@ the shared CLI compiler preprocessing pipeline. ```python -from prik import parse_fortran_file +from prik.parsers.fortran import parse_fortran_file parsed = parse_fortran_file( "subroutine ping(n)\n" @@ -45,7 +45,7 @@ PRIK_C_DOCS_END --> | Generated reference pages for modules, functions, and classes | Partially supported | [Reference index](../reference/index.md) | [Source map](../../developer/source-map.md) | [Documentation reference checks](../../../tests/docs/test_reference_and_source_map.py), [semantic contract tests](../../../tests/fortran/semantic_pyi_format/semantics/test_calls_and_projections.py) | Maintained manual references exist for generated functions, modules, classes, and generated file contracts; automated reference inventory generation has not been selected. | diff --git a/docs/user/reference/configuration-files.md b/docs/user/reference/configuration-files.md index c3d36d7ba..6cd67f5f3 100644 --- a/docs/user/reference/configuration-files.md +++ b/docs/user/reference/configuration-files.md @@ -2,7 +2,7 @@ title: Configuration Files Reference audience: users, developers prerequisites: packaging, CLI commands -related: cli-commands.md, python-api.md, ../guide/building-shared-library.md, ../../developer/quality-assurance.md +related: cli-commands.md, python-api.md, ../guide/building-shared-library.md, ../../developer/workflows/quality-assurance.md status: maintained publication: draft --- @@ -107,7 +107,7 @@ The coverage contract is: When investigating coverage failures that involve subprocesses, run coverage with `COVERAGE_PROCESS_START=pyproject.toml`, combine parallel data, then report. The maintained workflow is documented in -[Quality Assurance](../../developer/quality-assurance.md#pytest-and-coveragepy). +[Quality Assurance](../../developer/workflows/quality-assurance.md#coverage-and-test-order-reproduction). ## `codecov.yml` @@ -121,11 +121,26 @@ Do not treat `pyproject.toml` as a user wrapper-build configuration file. Wrapper users select inputs through CLI flags, Python API arguments, semantic `.pyi` contracts, and generated manifests. +## `setup.cfg` + +`setup.cfg` contains only setuptools command-output placement. Its `egg_info` +section sends temporary package metadata to `.artifacts/` instead of creating +a visible `prik.egg-info/` directory in the repository root. Project metadata, +dependencies, package discovery, and tool configuration remain exclusively in +`pyproject.toml`; do not duplicate them here. + +The tracked `.artifacts/.gitignore` file makes the hidden output root available +in clean checkouts and source distributions while ignoring everything generated +beneath it. + ## `mkdocs.yml` `mkdocs.yml` is the documentation-site configuration. It sets `docs_dir: docs`, +sets the generated site output to the hidden `.artifacts/site/` directory, selects MkDocs' built-in Read the Docs theme, owns the complete intended -navigation tree, and loads the publication hook. The theme configuration keeps +navigation tree, and loads the publication hook. Generated documentation is +therefore kept out of the visible repository root while remaining available +for local inspection. The theme configuration keeps the sidebar expanded through four navigation levels. A local stylesheet keeps its scrollbar visible and draggable when the navigation is longer than the screen. The same stylesheet keeps the page body adjacent to the sidebar with a diff --git a/docs/user/reference/fortran-wrapper.md b/docs/user/reference/fortran-wrapper.md index c3813d890..17ec02da1 100644 --- a/docs/user/reference/fortran-wrapper.md +++ b/docs/user/reference/fortran-wrapper.md @@ -518,7 +518,7 @@ libraries. ```python from pathlib import Path -from prik import NativeBuildPlan, NativeLinkItem +from prik.pipeline.build import NativeBuildPlan, NativeLinkItem plan = NativeBuildPlan( link_items=( @@ -663,7 +663,7 @@ an independent lifetime. ### Policy Overrides In Semantic `.pyi` Files -Ownership decisions are centralized in `prik.semantics.ownership`. Semantic +Ownership decisions are centralized in `prik.policy.ownership`. Semantic lowering and both bridge layers consume that resolved decision; low-level printers do not invent ownership behavior. diff --git a/docs/user/reference/python-api.md b/docs/user/reference/python-api.md index f775e1a2a..983de8a38 100644 --- a/docs/user/reference/python-api.md +++ b/docs/user/reference/python-api.md @@ -2,16 +2,17 @@ title: Python API Reference audience: users, developers prerequisites: installation -related: cli-commands.md, ../../developer/development-workflow.md +related: cli-commands.md, fortran-wrapper.md, ../../developer/packages/index.md status: maintained publication: draft --- # Python API Reference -This page documents the checked public symbols exported from `prik.__all__`. -The names below are the supported import surface for callers that use prik as a -library. +`prik` is a small normal-user facade. It exposes the installed version and the +three ways to build a wrapper. It does not re-export parser models, semantic +conversion, compiler probes, runtime handles, plans, or CLI implementation. +Import those advanced tools from the package that owns them. ```python import prik @@ -19,228 +20,51 @@ import prik sorted(prik.__all__) ``` -## Package version +## Root API -| Symbol | Purpose | +| Symbol | Use it for | | --- | --- | -| `__version__` | Installed PRIK distribution version, read from package metadata. | +| `__version__` | Read the installed PRIK distribution version. | +| `build_fortran_extension` | Build an extension from Fortran source plus optional native-only inputs. | +| `build_pyi_extension` | Build an extension from semantic `.pyi` contracts plus explicit native implementation inputs. | +| `build_pyi_extension_from_manifest` | Replay a saved semantic-`.pyi` build manifest or generate its Makefile. | -`pyproject.toml` is the only file that declares the release version. The public -attribute does not maintain a second version constant: +For normal builds, import directly from the root: ```python -import prik +from prik import build_fortran_extension -assert prik.__version__ == "0.1.0" +result = build_fortran_extension("solver.f90", output_dir="build/solver") +module = result.import_module() ``` -## CLI entrypoint - -| Symbol | Purpose | -| --- | --- | -| `main` | Runs the `python3 -m prik` command-line interface. Prefer the CLI for shell workflows and the functions below for Python workflows. | - - - - - - - -## Fortran parser API - -| Symbol | Purpose | -| --- | --- | -| `parse_fortran_file` | Parses one Fortran file into a `FortranFile`. | -| `parse_fortran_project` | Parses multiple Fortran files into a `FortranProject`. | -| `FortranFile` | Parsed Fortran file model. | -| `FortranProject` | Parsed Fortran project model. | -| `FortranModule` | Parsed module model. | -| `FortranSubmodule` | Parsed submodule model. | -| `FortranProgram` | Parsed program model. | -| `FortranBlockData` | Parsed block-data unit model. | -| `FortranDerivedType` | Parsed derived-type model. | -| `FortranInterface` | Parsed interface model. | -| `FortranProcedureSignature` | Parsed function or subroutine signature model. | -| `FortranArgument` | Parsed procedure argument model. | -| `FortranParseError` | Error raised for Fortran parse failures. | - -## Semantic conversion API - -| Symbol | Purpose | -| --- | --- | -| `fortran_file_to_semantic_modules` | Converts a parsed Fortran file to semantic module models. | -| `fortran_project_to_semantic_modules` | Converts a parsed Fortran project to semantic module models. | -| `fortran_module_to_semantic_module` | Converts one parsed Fortran module to one semantic module. | -| `collect_semantic_compile_time_requirements` | Collects semantic values that must be known at compile time. | -| `resolve_semantic_compile_time_values` | Resolves collected compile-time requirements. | - - - -Semantic conversion is the boundary between parser models and wrapper-facing -contracts. Run the default wrapper build to complete policy and validate -whether the wrapper plan supports the contract. - -## Semantic `.pyi` contract API - -| Symbol | Purpose | -| --- | --- | -| `parse_pyi_text` | Parses semantic `.pyi` source text into Python AST. | -| `parse_pyi_file` | Loads and parses one semantic `.pyi` file into Python AST. | -| `convert_pyi_to_ir` | Converts parsed semantic `.pyi` AST to semantic IR. | -| `pyi_text_to_semantic_module` | Parses inline semantic `.pyi` text and converts it to semantic IR. | -| `pyi_file_to_semantic_module` | Converts one semantic `.pyi` file to semantic IR. | -| `pyi_paths_to_semantic_modules` | Converts semantic `.pyi` files or directories to semantic IR and reconciles imports. | - -Editable `.pyi` files are a contract surface. User-private declarations in a -`.pyi` file are distinct from source-private Fortran declarations omitted from -generated stubs. - -## Stub emission API - -| Symbol | Purpose | -| --- | --- | -| `emit_module_stubs` | Emits semantic Python `.pyi` text from semantic module models. | -| `opaque_dependency_modules` | Computes opaque dependency modules needed for emitted stubs. | - -## Native array handle API - -| Symbol | Purpose | -| --- | --- | -| `NativeArrayHandleBase` | Common runtime base for generated native array descriptor handles. | -| `AllocatableArray` | Runtime object for a native allocatable array descriptor. | -| `PointerArray` | Runtime object for a native pointer array descriptor. | - -Generated wrappers use these handle classes when an allocatable or pointer array -descriptor is exposed as a Python object. Users can test for these classes when -they need to distinguish descriptor handles from ordinary NumPy arrays. Borrowed -handles do not own native storage. Owned handles expose `close()` and `closed`; -their finalizer attempts generated owner-storage destruction at most once. - -`Allocatable[T[...]]()` creates an owned, initially unallocated -`AllocatableArray`. `Pointer[T[...]]()` creates an owned, initially -unassociated `PointerArray`. The dtype and rank come from the annotation. On -the first writable descriptor call, the generated wrapper attaches -compiler-compatible persistent storage to the same handle. Closing an -allocatable handle also releases any allocation it still owns; closing a -pointer handle releases only its descriptor, not an associated target. - -`p1.associate(p2)` makes `p1` refer to the same target as `p2`, or makes -`p1` unassociated when `p2` is unassociated. It replaces any current -association of `p1` without copying or deallocating target storage. - -Owned writable handles carry a versioned record defined by prik's bundled -native binding support. Separately built prik extensions can accept the same -handle without linking to each other when their prik handle ABI and Fortran -compiler/runtime ABIs are compatible. Each receiving wrapper validates the -record's version, size, descriptor kind, dtype, and rank before direct -descriptor use. - -These classes are array-only. Scalar `Allocatable[T]` and `Pointer[T]` -projections remain ordinary `T | None` values and never produce an -`AllocatableArray` or `PointerArray`. - -`to_numpy()` returns `None` when the descriptor is currently unallocated or -unassociated. Otherwise, it returns a live NumPy view of the current allocation -or pointer target and never an automatic detached copy. Results must match the -handle's declared dtype and rank. Contiguous-view policy rejects non-contiguous -storage, while descriptor-view extraction can expose positive or negative -strides when generated standard descriptor support is available. Unsupported -descriptor extraction fails explicitly. Reallocation, deallocation, pointer -reassociation, or nullification may make an older view stale; accessing a stale -view is unsupported and may crash. Call `.copy()` explicitly when independent -storage is required, and call `to_numpy()` again to inspect current state. - -When a generated wrapper accepts a handle for an ordinary `T[...]` argument, it -uses an internal native array-actual handoff rather than an implicit -`to_numpy()` call. Parameters annotated as `Allocatable[T[...]]` or -`Pointer[T[...]]` use descriptor handoff and require the matching handle class; -plain NumPy arrays are for ordinary array-data parameters. - -## Wrapper build API - -| Symbol | Purpose | -| --- | --- | -| `build_fortran_extension` | Builds a Python extension from semantic Fortran source inputs plus optional native-only sources, artifacts, compiler flags, include paths, libraries, and ordered link items. | -| `build_pyi_extension` | Builds a Python extension from semantic `.pyi` contracts plus explicit native artifacts. | -| `build_pyi_extension_from_manifest` | Replays a saved semantic `.pyi` wrapper build manifest, either building directly or regenerating `Makefile.prik`. | -| `WrapperBuildResult` | Result model returned by wrapper build functions; `import_module()` explicitly loads its built extension. | -| `NativeBuildPlan` | Structured native implementation compile/link plan attached to a wrapper build result. | -| `NativeCompilationUnit` | Native source compilation unit and produced object recorded in a native build plan. | -| `NativePrebuiltArtifact` | Caller-supplied native object, archive, or shared library recorded in a native build plan. | -| `NativeLinkItem` | One ordered object, archive, shared library, named library, or linker argument in a native link plan. | - -Fortran source wrapper builds own the normal source-to-extension workflow and -may augment their positional semantic sources with the same native compile and -link inputs used by contract builds. Semantic `.pyi` wrapper builds require at -least one explicit native implementation input such as native Fortran sources, -objects, libraries, or ordered link items. Inspect -`WrapperBuildResult.native_build_plan` when a caller needs the native -compilation units, produced objects, prebuilt artifacts, module/include -directories, library directories, or ordered native link items separately from -the semantic contract paths. Semantic `.pyi` build results also expose a -normalized replay `manifest`; Makefile mode writes that manifest to -`/prik-build.json` before generating `Makefile.prik`. - -When a program needs the generated extension immediately, call -`result.import_module()`. It loads `result.shared_library` under -`result.module_name` without changing `sys.path` and returns the imported -module. The method requires that the shared-library file already exists, so a -direct build can import at once and a Makefile result can import after `make` -has produced the extension. - -## Target type and NumPy helpers - -| Symbol | Purpose | -| --- | --- | -| `FortranTypeProbeError` | Error raised for Fortran type probing failures. | -| `FortranTypeProbeReport` | Report model for Fortran type probing. | -| `build_fortran_type_probe_source` | Builds the source used to probe Fortran type properties. | -| `fortran_type_probe_expressions` | Produces expressions used by the Fortran type probe. | -| `probe_fortran_type_expressions` | Runs Fortran type probes for selected expressions. | -| `evaluate_fortran_type_requirements` | Evaluates semantic requirements against a Fortran type probe report. | -| `SEMANTIC_DTYPE_TO_NUMPY_DTYPE` | Default semantic dtype to NumPy dtype map. | -| `semantic_dtype_to_numpy_dtype` | Maps one semantic dtype to a NumPy dtype. | -| `semantic_dtype_to_numpy_dtype_map` | Returns a semantic dtype to NumPy dtype mapping. | -| `semantic_type_to_numpy_dtype` | Maps one semantic type to a NumPy dtype. | -| `numpy_dtype_expression` | Returns the generated expression for a NumPy dtype. | - -These helpers are public because wrapper contracts need deterministic target -type and NumPy dtype mapping. The CLI type-probe flags are documented in -[CLI Commands Reference](cli-commands.md). - -## Current boundaries - -- Parser functions do not run CLI path expansion or command-line preprocessing - validation. -- Generated module, function, class, and configuration references document the - wrapper output surface; this page remains the maintained inventory for - `prik.__all__`. - - +The functions return `prik.pipeline.build.WrapperBuildResult`. Import result +models and native-build plan records from `prik.pipeline.build` only when you +need to inspect or construct those advanced values. + +## Advanced Package Imports + +| Need | Import from | Main entrypoints | +| --- | --- | --- | +| Fortran source facts and diagnostics | `prik.parsers.fortran` | `parse_fortran_file`, `parse_fortran_project`, `FortranParser`, parser models, `FortranParseError` | +| Raw semantic `.pyi` syntax | `prik.parsers.pyi` | `parse_pyi_text`, `parse_pyi_file` | +| Semantic conversion | `prik.semantics.fortran2ir` or `prik.semantics.pyi2ir` | Fortran conversion helpers or `convert_pyi_to_ir` | +| `.pyi` loading and stub emission | `prik.pipeline.pyi` | `pyi_*_to_semantic_module`, `emit_module_stubs` | +| Build records and results | `prik.pipeline.build` | `WrapperBuildResult`, `NativeBuildPlan`, `NativeCompilationUnit`, `NativePrebuiltArtifact`, `NativeLinkItem` | +| Target type probing | `prik.preprocessing.probes.fortran_types` | probe source, requirements, expressions, and report/error types | +| Runtime descriptor handles | `prik.runtime.handles` | `NativeArrayHandleBase`, `AllocatableArray`, `PointerArray` | +| Semantic `.pyi` vocabulary | `prik.contracts` | scalar, array, ownership, and native-call contract markers | +| CLI implementation | `prik.cli` | `main()`; shell users should run `python3 -m prik` instead | + +The [Fortran wrapper reference](fortran-wrapper.md) documents the normal build +functions. The [package guides](../../developer/packages/index.md) explain +advanced module responsibilities and their focused tests. + +## Current Boundaries + +- Root imports are intentionally small and do not load parser or semantic + implementation modules. +- A parser success is only a source fact. Semantic conversion, policy + completion, planning, and generation are separate stages. +- The C-input frontend is deferred from the published workflow. Its internal + parser package is not a root API. diff --git a/docs/user/reference/semantic-ir.md b/docs/user/reference/semantic-ir.md index ab33e4fbf..79699e4c6 100644 --- a/docs/user/reference/semantic-ir.md +++ b/docs/user/reference/semantic-ir.md @@ -296,7 +296,7 @@ PRIK_C_DOCS_END --> - `void` return -> `None`. - `_Bool` -> `Bool`. - All modeled primitive integer, real, and complex spellings consume supplied - `prik.probes.c_types` facts. Plain `char` signedness, integer widths, real + `prik.preprocessing.probes.c_types` facts. Plain `char` signedness, integer widths, real storage widths and precision metadata, and complex storage widths come from the selected compiler target. - `int` keeps semantic name `Int` while its concrete dtype follows the target. @@ -308,7 +308,7 @@ PRIK_C_DOCS_END --> - Local typedef chains are resolved when their parser model definitions are available. - `size_t` maps to `SizeT` without a target probe; supplied - `prik.probes.c_types` facts override standard typedefs with width-specific + `prik.preprocessing.probes.c_types` facts override standard typedefs with width-specific `Int*`, `UInt*`, or `Float*` semantic names. - Opaque standard-type probe facts such as `FILE` create named opaque semantic classes when referenced by converted declarations. @@ -533,7 +533,7 @@ The proposed target is Python wrappers for C libraries on a selected Linux ABI. Its primary design requirement is that a semantic `.pyi` file plus a compiled library be sufficient to generate a wrapper, with C header parsing used only as optional input generation. Related deferred policy is tracked in -[wrapper design notes](../../maintainer/design/wrapper-design-notes.md). +[wrapper design notes](../../developer/design/wrapper-design-notes.md). PRIK_C_DOCS_END -->