Skip to content

Add the programmatic Python API - #342

Merged
soaressgabriel merged 9 commits into
mainfrom
feat/341-python-api
Sep 22, 2026
Merged

soaressgabriel merged 9 commits into
mainfrom
feat/341-python-api

Conversation

@soaressgabriel

Copy link
Copy Markdown
Collaborator

Checklist

  • Have you followed the guidelines in our Contributing document?
  • Have you checked to ensure there aren't other open Pull Requests for the same update/change?
  • Added tests for changed code.
  • Updated documentation for changed code.

Description

  • New rubem.api, the public Python surface (__all__ = ["Model", "RunResult", "ConfigurationError"]):
    • Model.from_file(path, *, validate_input=True, base_dir=None) and Model.from_config(config, *, validate_input=True, base_dir=None); the latter takes the configuration document as a dictionary (legacy or format 1.0) or an already loaded ModelConfiguration, used as it is. Relative paths are anchored as the loader anchors them. The configuration property returns the loaded configuration.
    • model.run() runs the simulation in the current process (one framework per call, the library only logs) and returns a RunResult: output directory, raster members per variable and per enabled format in step order, time series tables per variable (CSV and/or TSS following the configured formats), metadata.json for a format 1.0 run, first and last step, elapsed seconds. The paths are enumerated from the configuration through the naming rules of the model, never by listing the directory, and are absolute.
    • model.run_isolated() runs the same simulation in a fresh spawned subprocess (ProcessPoolExecutor(max_workers=1, mp_context=spawn, max_tasks_per_child=1), shut down before returning); the configuration crosses as its document plus base_dir and the validation flag, the result as plain data; a ConfigurationError of the child reaches the caller as itself with its problems, any other exception propagates unchanged, and a subprocess that dies is reported as a RuntimeError.
    • Importing rubem.api does not need PCRaster or GDAL; loading or running without them raises ImportError with the guidance the command line prints (rubem/_deps.py now builds that message once for the CLI's SystemExit and the API's ImportError).
  • ConfigurationError gains __reduce__: it pickled as its message and could not be rebuilt on the other side of a process boundary (the constructor takes the problems); it now travels with its problems and its instance state.
  • Tests (tests/unit/api, 31): runs from a dictionary, a file, a loaded configuration and a format 1.0 document; the reported files equal the directory listing and the expected outputs; absolute paths with a relative output directory; the three time series format combinations and the zones aggregation names; two sequential isolated runs; the child is a spawned interpreter that starts without PCRaster and leaves no process behind; base_dir and the validation flag cross the boundary; a blocking configuration arrives as ConfigurationError with its problems, another failure as its own type, a killed child as RuntimeError; pickle round trip; rubem.api importable with PCRaster and GDAL blocked, every entry point raising the guidance while the command line still exits with the same text; the documented loader failures; an API surface snapshot (names, signatures, RunResult fields).
  • Docs: api.rst rewritten as "Python API" (quickstart, stability policy, process model and limitations, logging, module reference with rubem.api first and the exception documented); a Python snippet in the README usage section. Changelog entry under Added.

Related Issue

Motivation and context

  • The model could only be run from the command line or through internal classes, and nothing documented the process constraint (PCRaster keeps the clone and the raster memory process-wide) every tool running it repeatedly has to respect. The calibration command builds on Model.run() inside its workers; the architecture work that follows delegates through the same method, so the public surface stays put while the internals move.

Open for the maintainer, documented rather than decided: validate_input and base_dir are keyword-only; elapsed_seconds of an isolated run measures the simulation, not the interpreter start-up; zones_mapping.csv of an aggregated run is not a RunResult field; the log records of an isolated run stay in the subprocess (forwarding them to the caller would be a follow-up).

How has this been tested

  • python -m pytest -q -p no:cacheprovider --ignore=tests/integration/doc -n 4: 1032 passed, 1 skipped, on main at 176f204 (Linux, Python 3.13, PCRaster 4.4.2). The isolated runs of the tests are real spawned processes; mutation checks confirmed the new tests fail when the start method is changed to fork, when the paths are not made absolute and when base_dir does not cross the boundary. Every quickstart snippet of api.rst and the README snippet were executed against the synthetic dataset.
  • uvx ruff@0.16.4 format --check . and uvx ruff@0.16.4 check .: clean.
  • sphinx-build -W -b html doc/source with the documentation's Python 3.13 environment: no warnings.

Screenshots

  • N/A

ConfigurationError takes its problems in the constructor, but
BaseException pickles itself through args, which holds the formatted
message: unpickling called the constructor with a string and failed.
Rebuild the exception from its problems (pydantic models, picklable) and
carry the instance state, so it crosses a process boundary as itself.
rubem.api is the public surface: Model.from_file and Model.from_config
load a configuration file or document (or take a loaded configuration),
Model.run() runs the simulation in the current process and
Model.run_isolated() in a fresh spawned subprocess (one worker, one task,
shut down before returning), both returning a RunResult that enumerates
from the configuration the rasters, time series and metadata the run
wrote, with absolute paths. Importing the module needs neither PCRaster
nor GDAL; loading or running without them raises ImportError with the
guidance the command line prints, now built once in rubem._deps.
Rewrite the API page with a quickstart, the stability policy, the process
model and its limitations, the logging behaviour and the module reference,
and add a Python usage example to the README.
@codecov

codecov Bot commented Sep 16, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.13043% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.80%. Comparing base (9721f1c) to head (3be8af2).

Files with missing lines Patch % Lines
rubem/api.py 89.07% 9 Missing and 4 partials ⚠️
rubem/_deps.py 85.71% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #342      +/-   ##
==========================================
- Coverage   92.93%   92.80%   -0.13%     
==========================================
  Files          63       64       +1     
  Lines        4163     4297     +134     
  Branches      532      545      +13     
==========================================
+ Hits         3869     3988     +119     
- Misses        235      246      +11     
- Partials       59       63       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

os.kill(pid, 0) is not a portable liveness check: Windows has no signal 0
and raises "The parameter is incorrect" instead of ProcessLookupError,
and on any platform the probe could hit a reused pid. The worker of the
executor is a multiprocessing child of the test process, so an empty
active_children() already proves it was joined.
…s own document

ModelConfiguration kept the dictionary of the caller by reference and
Model.run_isolated() sent it to the subprocess, so editing the dictionary
to configure a second model redirected the isolated run of the first one,
parameters and output directory included. Model.run() was unaffected
because it reads the parsed settings.

The dictionary is now deep-copied when it is loaded, which also covers a
configuration loaded on its own and handed to Model afterwards.
Driving the API from an installed wheel showed four places where the text
and the behaviour had drifted apart:

- In-process runs on different grids do follow one another: every run sets
  the clone for itself. What cannot happen is two runs at the same time in
  one process. On different grids one of them fails; on the same grid they
  may all finish and write wrong results without any error. A test pins the
  sequential case.
- A script without the entry point guard, or read from standard input,
  kills the subprocess at start-up, and the RuntimeError blamed an outside
  kill or a native crash. The message now names both causes, and a test
  runs an unguarded script for real.
- An interrupt delivered to the calling process alone is raised only after
  the subprocess has finished the run.
- The newline of the PCRaster atexit hook is written once per isolated run,
  since the subprocess exits with every call.
Parallel runs need one process each, and the text named run_isolated() as
the only way to get it. A process pool of the caller whose workers call
Model.run() does the same and reuses its workers, so the interpreter
start-up is paid once per worker instead of once per run.

The API page gains a "Parallel runs" section with that example, what crosses
the boundary, the memory of a reused worker and why a thread pool is not a
substitute. A test runs six simulations that alternate between two grids
through a spawned pool of two workers and compares every file with the
in-process reference.
@soaressgabriel
soaressgabriel merged commit 601b262 into main Sep 22, 2026
17 checks passed
@soaressgabriel
soaressgabriel deleted the feat/341-python-api branch September 22, 2026 13:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add the programmatic Python API (rubem.api)

1 participant