Repository navigation
Add the programmatic Python API - #342
Merged
Merged
Conversation
ConfigurationError takes its problems in the constructor, but BaseException pickles itself through args, which holds the formatted message: unpickling called the constructor with a string and failed. Rebuild the exception from its problems (pydantic models, picklable) and carry the instance state, so it crosses a process boundary as itself.
rubem.api is the public surface: Model.from_file and Model.from_config load a configuration file or document (or take a loaded configuration), Model.run() runs the simulation in the current process and Model.run_isolated() in a fresh spawned subprocess (one worker, one task, shut down before returning), both returning a RunResult that enumerates from the configuration the rasters, time series and metadata the run wrote, with absolute paths. Importing the module needs neither PCRaster nor GDAL; loading or running without them raises ImportError with the guidance the command line prints, now built once in rubem._deps.
Rewrite the API page with a quickstart, the stability policy, the process model and its limitations, the logging behaviour and the module reference, and add a Python usage example to the README.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #342 +/- ##
==========================================
- Coverage 92.93% 92.80% -0.13%
==========================================
Files 63 64 +1
Lines 4163 4297 +134
Branches 532 545 +13
==========================================
+ Hits 3869 3988 +119
- Misses 235 246 +11
- Partials 59 63 +4 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
os.kill(pid, 0) is not a portable liveness check: Windows has no signal 0 and raises "The parameter is incorrect" instead of ProcessLookupError, and on any platform the probe could hit a reused pid. The worker of the executor is a multiprocessing child of the test process, so an empty active_children() already proves it was joined.
4 tasks done
…s own document ModelConfiguration kept the dictionary of the caller by reference and Model.run_isolated() sent it to the subprocess, so editing the dictionary to configure a second model redirected the isolated run of the first one, parameters and output directory included. Model.run() was unaffected because it reads the parsed settings. The dictionary is now deep-copied when it is loaded, which also covers a configuration loaded on its own and handed to Model afterwards.
Driving the API from an installed wheel showed four places where the text and the behaviour had drifted apart: - In-process runs on different grids do follow one another: every run sets the clone for itself. What cannot happen is two runs at the same time in one process. On different grids one of them fails; on the same grid they may all finish and write wrong results without any error. A test pins the sequential case. - A script without the entry point guard, or read from standard input, kills the subprocess at start-up, and the RuntimeError blamed an outside kill or a native crash. The message now names both causes, and a test runs an unguarded script for real. - An interrupt delivered to the calling process alone is raised only after the subprocess has finished the run. - The newline of the PCRaster atexit hook is written once per isolated run, since the subprocess exits with every call.
Parallel runs need one process each, and the text named run_isolated() as the only way to get it. A process pool of the caller whose workers call Model.run() does the same and reuses its workers, so the interpreter start-up is paid once per worker instead of once per run. The API page gains a "Parallel runs" section with that example, what crosses the boundary, the memory of a reused worker and why a thread pool is not a substitute. A test runs six simulations that alternate between two grids through a spawned pool of two workers and compares every file with the in-process reference.
4 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Checklist
Description
rubem.api, the public Python surface (__all__ = ["Model", "RunResult", "ConfigurationError"]):Model.from_file(path, *, validate_input=True, base_dir=None)andModel.from_config(config, *, validate_input=True, base_dir=None); the latter takes the configuration document as a dictionary (legacy or format 1.0) or an already loadedModelConfiguration, used as it is. Relative paths are anchored as the loader anchors them. Theconfigurationproperty returns the loaded configuration.model.run()runs the simulation in the current process (one framework per call, the library only logs) and returns aRunResult: output directory, raster members per variable and per enabled format in step order, time series tables per variable (CSV and/or TSS following the configured formats),metadata.jsonfor a format 1.0 run, first and last step, elapsed seconds. The paths are enumerated from the configuration through the naming rules of the model, never by listing the directory, and are absolute.model.run_isolated()runs the same simulation in a fresh spawned subprocess (ProcessPoolExecutor(max_workers=1, mp_context=spawn, max_tasks_per_child=1), shut down before returning); the configuration crosses as its document plusbase_dirand the validation flag, the result as plain data; aConfigurationErrorof the child reaches the caller as itself with its problems, any other exception propagates unchanged, and a subprocess that dies is reported as aRuntimeError.rubem.apidoes not need PCRaster or GDAL; loading or running without them raisesImportErrorwith the guidance the command line prints (rubem/_deps.pynow builds that message once for the CLI'sSystemExitand the API'sImportError).ConfigurationErrorgains__reduce__: it pickled as its message and could not be rebuilt on the other side of a process boundary (the constructor takes the problems); it now travels with its problems and its instance state.tests/unit/api, 31): runs from a dictionary, a file, a loaded configuration and a format 1.0 document; the reported files equal the directory listing and the expected outputs; absolute paths with a relative output directory; the three time series format combinations and the zones aggregation names; two sequential isolated runs; the child is a spawned interpreter that starts without PCRaster and leaves no process behind;base_dirand the validation flag cross the boundary; a blocking configuration arrives asConfigurationErrorwith its problems, another failure as its own type, a killed child asRuntimeError; pickle round trip;rubem.apiimportable with PCRaster and GDAL blocked, every entry point raising the guidance while the command line still exits with the same text; the documented loader failures; an API surface snapshot (names, signatures,RunResultfields).api.rstrewritten as "Python API" (quickstart, stability policy, process model and limitations, logging, module reference withrubem.apifirst and the exception documented); a Python snippet in the README usage section. Changelog entry under Added.Related Issue
Motivation and context
Model.run()inside its workers; the architecture work that follows delegates through the same method, so the public surface stays put while the internals move.Open for the maintainer, documented rather than decided:
validate_inputandbase_dirare keyword-only;elapsed_secondsof an isolated run measures the simulation, not the interpreter start-up;zones_mapping.csvof an aggregated run is not aRunResultfield; the log records of an isolated run stay in the subprocess (forwarding them to the caller would be a follow-up).How has this been tested
python -m pytest -q -p no:cacheprovider --ignore=tests/integration/doc -n 4: 1032 passed, 1 skipped, onmainat 176f204 (Linux, Python 3.13, PCRaster 4.4.2). The isolated runs of the tests are real spawned processes; mutation checks confirmed the new tests fail when the start method is changed tofork, when the paths are not made absolute and whenbase_dirdoes not cross the boundary. Every quickstart snippet ofapi.rstand the README snippet were executed against the synthetic dataset.uvx ruff@0.16.4 format --check .anduvx ruff@0.16.4 check .: clean.sphinx-build -W -b html doc/sourcewith the documentation's Python 3.13 environment: no warnings.Screenshots