diff --git a/CHANGELOG.md b/CHANGELOG.md index 3824a10..a6c5ac7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,24 +10,25 @@ and versions follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Added - **`uv.lock` is updated once a quarter.** Dependabot (`.github/dependabot.yml`) opens one pull request a quarter that moves the locked dependencies to the latest versions `pyproject.toml` allows. The tests run on it, so an update that breaks them is seen before it is merged. Between updates the versions stay as they are; `uv lock --upgrade` does the same by hand. - **A lock file, `uv.lock`, and uv as the way to install PET on Linux and macOS.** `pyproject.toml` names the dependencies but not their versions, so two installs a month apart could differ in numpy, scipy or anything below them, and a result could not be tied to the versions that produced it. `uv.lock` records every dependency, direct and indirect, at one version for each supported Python. `uv sync` installs exactly those into `.venv`, and CI now does the same (`uv sync --locked`, which fails if the lock no longer matches `pyproject.toml`). After changing the dependencies, run `uv lock` and commit the result; `uv lock --upgrade` moves them to their latest allowed versions. `pip install -e .` works as before and ignores the lock. Windows is not covered (`[tool.uv] environments`): it still installs with conda, see `docs/installation_windows.md`. + +## [0.3] - 2026-11-19 + +### Added - **`advance_on_failure` for the exterior penalty loop (`epf`).** A step that fails ends the run, whatever the constraint violation, so the penalty factor is only raised after an inner loop that ended on a tolerance or on its iteration limit. With `epf = {..., advance_on_failure = true}` a failed step ends only the current penalty iteration: the penalty is evaluated at the iterate (the `epf` dict otherwise holds that of the rejected trial point), and unless it is below `conv_crit` the factor is raised and the optimization goes on from the same point. On `benchmarks/popt/minires`, which has a row for either setting, this took EnOpt with EPF (starting at r = 10) from 1 feasible run of 10 to 8, for 30 % more simulations and a lower net present value, the price of the violation removed. The default is `false`: existing runs are unchanged. - **MiniRes takes well controls, so popt can optimize on it, and a benchmark does: `benchmarks/popt/minires`.** The MiniRes wrapper had no way to take controls: it required the permeability in the state, rejected every other key, and ran the wells as the config's records gave them. A new optional `controls` key of the simulator section, `{variable: {well, type = "rate" | "bhp", sign}}`, makes a state variable the schedule of a well: its values, whose number must divide the number of steps, are held constant over equally long intervals. With `controls` the permeability may be left out of the state (the `K` of `[simulator.model]` is then the reservoir's) and an `aux_input` is ignored. Without `controls` nothing changes: the same states are refused with the same messages, and `benchmarks/pipt/minires` reproduces its official numbers. The new benchmark poses a constrained optimization on the reservoir of `benchmarks/pipt/minires` with the well pattern turned round (four injectors on rate, a producer on pressure): the 32 injection rates of 8 intervals are chosen for net present value, under a cap on the field's water production (nonlinear, known only through the simulation) and a capacity for the total injection (linear). It runs `EnSQP`, `EnOpt` with the exterior penalty, with and without `advance_on_failure`, `EnOpt` with the constraints ignored and, as a reference, scipy's SLSQP with finite differences, on ten permeability fields, and writes `RESULTS.md` and a convergence figure. The data-assimilation benchmark moved from `benchmarks/minires` to `benchmarks/pipt/minires`, so that the two sit side by side under `pipt` and `popt`; its code and numbers are as they were. - **`EnSQP`, ensemble-based sequential quadratic programming for constrained optimization.** Until now a constraint could only be handled by the exterior penalty loop (`epf`), where the objective adds the penalty itself and the optimizer is re-run with a growing penalty factor. `EnSQP` takes equality and inequality constraints as such (`c_eq = 0`, `c_iq >= 0`) and estimates every derivative from one ensemble per iteration: the gradient of the objective, the Jacobian of the constraints and, optionally, the Hessian of the Lagrangian. Each iteration solves a quadratic program for the step and the Lagrange multipliers and backtracks on an l1 merit function, at EnOpt's cost of one ensemble plus the trial points. The objective returns the constraints with the objective, `{'f': ..., 'c_eq': ..., 'c_iq': ...}`, from the same simulation; `EnsembleOptimizationBase.function` keeps them (`stateC`, `enC`) and still returns the objective values, so every other optimizer runs such an objective unchanged, and `function_and_constraints` returns both. `GaussianEnsemble` gains `constraint_jacobian`, `lagrangian_hessian` and `sample_covariance`, all computed from the ensemble of the last `gradient` call; `gradient` and `hessian` give the same numbers as before. Options: `hessian = 'bfgs'` (default; Powell-damped BFGS on the ensemble gradients) or `'ensemble'` (the ensemble Hessian of the Lagrangian, made positive definite; its sampling error is as large as the Hessian until the members far outnumber the controls), and `scaling = 'sample'` (default) or `'cov'`: an ensemble gradient is a covariance times the gradient, and dividing by the covariance of the perturbations as sampled rather than as drawn makes the estimate exact for linear functions, so a linear constraint is met to round-off. With fewer members than controls the step is sought in the subspace the perturbations span, and the Hessian is updated from the curvature measured along the accepted step, since gradients from two different subspaces cannot be differenced. The quadratic programs are solved by a small dense active-set solver (`popt.optimization_methods.subroutines.qp.solve_qp`), so there is no new dependency. The docstring of `cost_functions.epf.epf` now states the sign convention its code has always had. - **A conda recipe for Windows.** `pip install -e .` cannot work on Windows: `graphite-maps` has no Windows build on PyPI, and the `scikit-sparse` it imports needs the CHOLMOD C library. `environment.yml` installs `scikit-sparse` from conda-forge, `graphite-maps` 0.0.11 from GitHub and PET itself in editable mode with one `conda env create -f environment.yml`; `docs/installation_windows.md` describes it. With that environment 583 of 589 non-slow tests pass on Windows; the six failures are five bit-for-bit comparisons in `test_subspace2.py` that differ in the last floating-point digit, and one test that expects a `/` path separator. -- **`GenOpt` is back**, on `OptimizerBase`, together with the `CMA` covariance-matrix adaptation it uses. Both were removed during the popt restructuring — `GenOpt` by `a6938529` with nothing put in its place, `cma.py` later as code with no remaining caller — while `src/popt/README.md` went on advertising the method. It draws from `GeneralizedEnsemble`'s marginals and advances the sampling distribution along with the controls: `theta` follows its own gradient at `alpha_theta`, and the correlation matrix follows `corr_adapt`, which is either a `CMA` instance or any callable. `GeneralizedEnsemble.mutation_gradient` and `.mutation_hessian` had survived the restructuring with no consumer; this is the consumer. `mutation_gradient` gains `return_ensembles=True`, returning `(nat_grad, {'gaussian': enZ, 'objective': enF})`, so the CMA path reuses the ensemble the gradient came from instead of drawing and simulating a second one. Two fixes over the version that was removed: `alpha_corr` is read from `alpha_corr` rather than from `alpha_theta`, so it is no longer silently ignored, and the optimizer no longer runs itself from its own constructor — call `run_optimization()` or `GenOpt.minimize(...)`, as with every other optimizer here. -- **`analysis = "subspace2"`**, the ensemble-transform IES of Raanes, Stordal & Evensen (2019), on ES-MDA, LM-EnRML and GN-EnRML. It solves for the `ne × ne` transform `W` directly, starting from `W = I`, and uses the analytic data covariance through `scale_data` rather than the ensemble representation `E Eᵀ` that `subspace` uses — so it takes no SVD and reads neither `energy` nor `iteration.energy`. It is exactly `margis` with the marginalised error scale `Ratio` fixed at 1, i.e. with the data uncertainty taken as known; `tests/assimilation/test_subspace2.py` pins that identity bit-for-bit. Ported from upstream 6f313d7, with two departures: the transform is initialised at iteration 0 rather than 1 (schemes here count from 0, so the reference version never initialises and dies on the first call), and the whitened observations are recomputed every call instead of cached on the first. ES-MDA redraws its observations and their scale at every assimilation step, so the cache would drive later steps with the first step's observations whitened by the first step's factor — the run still completes and the misfit still falls, which is why it would have gone unnoticed. On the characterisation case it is the difference between a posterior misfit of 175.7 and 201.1; the cached, lower number is the wrong one. The 39 existing golden arrays are unchanged. ### Fixed - **Adjoints with one gradient per parameter reach the analysis again.** A simulator may return each member's adjoint frame with `(datatype, parameter)` columns, one gradient per parameter in each cell. The frame path joined those gradients through `PETDataFrame._to_singlelevel_columns`; the `DataLayout`-based `_adjoint_array` that replaced it in #166 handed numpy the per-parameter Series as it was, so `forecast()` stopped with `ValueError: setting an array element with a sequence.` The gradients are now joined in column order, as before. Frames with one column per data type are unaffected. - **`autoadaloc` with `projection = "ensemble"` runs, and computes the update it describes.** The `approx` analysis built the projected anomalies as `X2 @ D_anom`, an `(nr, ne) @ (nd, ne)` product that raises `ValueError` unless `nd == ne`, and that would count the innovations twice even when it ran. They are now `V_r Σ_r X2`, the transform the non-localized branch already uses, so an all-ones taper gives the non-localized step exactly (#170). -- **A `LOCALIZATION` block that names no mode runs again.** Localization was selected by which keyword appeared in the block -- `autoadaloc`, `localanalysis`, `dist_loc`, a pickled mask file, or none of them for the parallel update -- and the rewrite replaced that with a required `name` key. Every config written before the rewrite therefore stopped at startup with `Localization config has no 'name'`, naming three strings its author had never seen, and `pet migrate` did not cover the block. The mode is now inferred from the keyword that used to select it, and an explicit `name` still wins. -- **A crashed simulation costs the optimizer its trial point, not the whole run.** `EnsembleOptimizationBase.function` raised `RuntimeError` whenever `calc_prediction` reported failure, so one control vector the simulator could not run ended the optimization and discarded every iteration before it. It now reports `inf`, which no backtracking step can improve on, so the point is rejected and the run continues from the last good iterate — the behaviour `765d637` removed. A crashed *single-point* evaluation additionally leaves `stateF` at its last good value: the gradient is `enF - repeat(stateF, nr)`, so writing `inf` there would have poisoned every later gradient with `inf`/`NaN` instead of rejecting one point. The separate abort when *every* member of a forecast fails is unchanged — there is nothing left to compute a gradient from. -- **A pickled localization file can be used again.** Those files hold plain dicts keyed by `(data_type, time, parameter)` — `taper_func`, `position`, `range` as `[radius, z_range]`, `anisotropi` as `[ratio, rotation]`, and `file` for the `import` taper. They were loaded and returned unconverted, so the first code to ask for `.taper` raised `AttributeError: 'dict' object has no attribute 'taper'` and no pickled mask file worked at all. They are now converted to `LocalizationEntry`, including older files that wrote `range` as the radius alone. -- **Distance localization builds its operator over the active cells.** `_resolve_mask` returned a mask spanning the whole grid while `_zero_mask` reduced to the active cells, so with an `actnum` a localized parameter contributed one row per grid cell and an unlocalized one a row per active cell. On a 1×10×10 field with 60 of 100 cells active and two parameters the operator came out with 160 rows where the state has 120, and nothing checked, so the mismatch surfaced far from its cause. An all-active `actnum` now gives exactly what passing none gives. -- **`autoadaloc = ` is no longer discarded.** The value is the number of noise standard deviations a correlation must clear -- `nstd` inside the old code -- and it was the value of the `autoadaloc` keyword itself. Only `cutoff` was read, so a config saying `autoadaloc = 2` silently ran at the default of 0.3: no error, a different taper, a different posterior. `cutoff`, `nstd` and `autoadaloc` are all accepted, `cutoff` first. The default stays 0.3; it was 1 before the rewrite, so a block that gave `autoadaloc` as a bare flag with no value tapers differently than it used to. ### Breaking changes - **The package version comes from the latest git tag.** `pyproject.toml` hard-coded `version = "1.0"` while the repository was tagged `v0.2`, so the two disagreed and `pet version` reported a number no release carried. setuptools-scm now derives it at build time: a tagged commit gets the tag (`v0.2` -> `0.2`), later commits a development version such as `0.3.dev4+g196c956`. Nothing creates tags automatically; release by pushing one. Installed versions therefore drop from `1.0` to `0.3.devN` until the next tag. CI checks out the full history so the tags are visible, and a build without git metadata falls back to `0.0.0`. + +## [0.2] - 2026-09-21 + +### Breaking changes - **`localanalysis` and the parallel update are refused while the config is read.** Both worked before the update schemes were restructured, and both need the per-subset observation machinery (`_ext_obs`, `current_state`, `pert_preddata`) that the rewrite replaced with a single `DataLayout` built once at setup. `localanalysis` previously reached a branch that left the posterior equal to the prior while still reporting a misfit, and is no longer a registered strategy; naming either now raises `ConfigError` explaining why and naming `autoadaloc` and `distance_loc` as the alternatives. Reimplementing them on the new contract is tracked separately; `analysis_tools.parallel_upd` is left in place for the dormant GIES schemes that still call it. - **`conv_crit` in the `epf` section means a penalty magnitude, not a state change.** The outer EPF loop used to stop once no control moved more than `conv_crit` relative to its previous value; it now stops once `mean(epf['penalty']) / epf['r']` falls below it. The old test asked the wrong question: it reported success whenever the inner optimizer stalled, however badly the constraints were still violated, and refused to finish while a single control kept jittering. The default is still `1e-5`, so a config written for the old criterion loads unchanged, but the number now carries the units of the objective rather than being dimensionless — check it against your penalty's scale. The objective must write `penalty` into the `epf` dict it is handed; one that does not now raises `KeyError` instead of silently converging on the step size. Ported from upstream 5358e07 and 4b8d878. - `max_iter` in the `iteration` section is the number of update iterations. It used to count the prior forecast as iteration 0, so `max_iter: 5` performed four updates; the same config now performs five. To keep an existing run as it was, lower `max_iter` by one. The run table, the convergence message and the `assimilation_result_{i}` files already numbered updates from 1 with the prior as 0, and are unchanged. @@ -467,6 +468,8 @@ and versions follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html). `SmcOpt`) override them exactly as before. ### Added +- **`GenOpt` is back**, on `OptimizerBase`, together with the `CMA` covariance-matrix adaptation it uses. Both were removed during the popt restructuring — `GenOpt` by `a6938529` with nothing put in its place, `cma.py` later as code with no remaining caller — while `src/popt/README.md` went on advertising the method. It draws from `GeneralizedEnsemble`'s marginals and advances the sampling distribution along with the controls: `theta` follows its own gradient at `alpha_theta`, and the correlation matrix follows `corr_adapt`, which is either a `CMA` instance or any callable. `GeneralizedEnsemble.mutation_gradient` and `.mutation_hessian` had survived the restructuring with no consumer; this is the consumer. `mutation_gradient` gains `return_ensembles=True`, returning `(nat_grad, {'gaussian': enZ, 'objective': enF})`, so the CMA path reuses the ensemble the gradient came from instead of drawing and simulating a second one. Two fixes over the version that was removed: `alpha_corr` is read from `alpha_corr` rather than from `alpha_theta`, so it is no longer silently ignored, and the optimizer no longer runs itself from its own constructor — call `run_optimization()` or `GenOpt.minimize(...)`, as with every other optimizer here. +- **`analysis = "subspace2"`**, the ensemble-transform IES of Raanes, Stordal & Evensen (2019), on ES-MDA, LM-EnRML and GN-EnRML. It solves for the `ne × ne` transform `W` directly, starting from `W = I`, and uses the analytic data covariance through `scale_data` rather than the ensemble representation `E Eᵀ` that `subspace` uses — so it takes no SVD and reads neither `energy` nor `iteration.energy`. It is exactly `margis` with the marginalised error scale `Ratio` fixed at 1, i.e. with the data uncertainty taken as known; `tests/assimilation/test_subspace2.py` pins that identity bit-for-bit. Ported from upstream 6f313d7, with two departures: the transform is initialised at iteration 0 rather than 1 (schemes here count from 0, so the reference version never initialises and dies on the first call), and the whitened observations are recomputed every call instead of cached on the first. ES-MDA redraws its observations and their scale at every assimilation step, so the cache would drive later steps with the first step's observations whitened by the first step's factor — the run still completes and the misfit still falls, which is why it would have gone unnoticed. On the characterisation case it is the difference between a posterior misfit of 175.7 and 201.1; the cached, lower number is the wrong one. The 39 existing golden arrays are unchanged. - Documentation: a configuration reference (`docs/configuration.md`) listing every key of the `dataassim`, `ensemble`, `optim` and `simulator` sections with meaning and default, and an architecture page (`docs/architecture.md`) describing the layers, the scheme, analysis and optimizer contracts, the data layouts, restart, random numbers, and where a new piece goes. Both are in the site navigation and linked from the README and the developer guide. `popt` exports its public API (`EnOpt`, `LineSearch`, `TrustRegion`, `SmcOpt`, `OptimizerBase`, `StepReport`, `GaussianEnsemble`, `GeneralizedEnsemble`). Every public function and class now has a docstring. - `misc.structures.DataLayout`: the order of the data vector, derived once from the observed frame (label-major, then data type, empty cells skipped). The ensemble builds it after scaling and exposes `obs_vector`, which the schemes now use in place of `data_df.to_matrix()`; the frame remains as the view (`DataLayout.to_frame`). First step of replacing frame flattening on the analysis path. - `seed` option in the ensemble config (`[ensemble] seed = 7` for pipt, `options['seed']` for popt). Every draw a run makes -- prior realisations, perturbed observations, outlier and crash replacement, the auto-adaptive localization's shuffle, popt's control perturbations -- now comes from the ensemble's `rng`: a private `numpy.random.RandomState(seed)` when a seed is given, so the run reproduces on its own and leaves NumPy's global state untouched; otherwise the global stream, exactly as before, so `np.random.seed(...)` before a run keeps working and every reference number is unchanged. The geostat sampler PET used for these draws is replicated draw for draw in `misc.sampling.gen_real`, which takes the stream as an argument; geostat remains a dependency for its covariance builder. @@ -524,6 +527,11 @@ and versions follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html). PIPT and POPT rather than duplicated. ### Fixed +- **A `LOCALIZATION` block that names no mode runs again.** Localization was selected by which keyword appeared in the block -- `autoadaloc`, `localanalysis`, `dist_loc`, a pickled mask file, or none of them for the parallel update -- and the rewrite replaced that with a required `name` key. Every config written before the rewrite therefore stopped at startup with `Localization config has no 'name'`, naming three strings its author had never seen, and `pet migrate` did not cover the block. The mode is now inferred from the keyword that used to select it, and an explicit `name` still wins. +- **A crashed simulation costs the optimizer its trial point, not the whole run.** `EnsembleOptimizationBase.function` raised `RuntimeError` whenever `calc_prediction` reported failure, so one control vector the simulator could not run ended the optimization and discarded every iteration before it. It now reports `inf`, which no backtracking step can improve on, so the point is rejected and the run continues from the last good iterate — the behaviour `765d637` removed. A crashed *single-point* evaluation additionally leaves `stateF` at its last good value: the gradient is `enF - repeat(stateF, nr)`, so writing `inf` there would have poisoned every later gradient with `inf`/`NaN` instead of rejecting one point. The separate abort when *every* member of a forecast fails is unchanged — there is nothing left to compute a gradient from. +- **A pickled localization file can be used again.** Those files hold plain dicts keyed by `(data_type, time, parameter)` — `taper_func`, `position`, `range` as `[radius, z_range]`, `anisotropi` as `[ratio, rotation]`, and `file` for the `import` taper. They were loaded and returned unconverted, so the first code to ask for `.taper` raised `AttributeError: 'dict' object has no attribute 'taper'` and no pickled mask file worked at all. They are now converted to `LocalizationEntry`, including older files that wrote `range` as the radius alone. +- **Distance localization builds its operator over the active cells.** `_resolve_mask` returned a mask spanning the whole grid while `_zero_mask` reduced to the active cells, so with an `actnum` a localized parameter contributed one row per grid cell and an unlocalized one a row per active cell. On a 1×10×10 field with 60 of 100 cells active and two parameters the operator came out with 160 rows where the state has 120, and nothing checked, so the mismatch surfaced far from its cause. An all-active `actnum` now gives exactly what passing none gives. +- **`autoadaloc = ` is no longer discarded.** The value is the number of noise standard deviations a correlation must clear -- `nstd` inside the old code -- and it was the value of the `autoadaloc` keyword itself. Only `cutoff` was read, so a config saying `autoadaloc = 2` silently ran at the default of 0.3: no error, a different taper, a different posterior. `cutoff`, `nstd` and `autoadaloc` are all accepted, `cutoff` first. The default stays 0.3; it was 1 before the rewrite, so a block that gave `autoadaloc` as a bare flag with no value tapers differently than it used to. - Flags written as strings were read by truthiness in several places, so `scale_data = "no"` in a config enabled scaling; every flag is a boolean after the boundary. - A NaN data variance for an observed cell was silently dropped when the covariance was assembled, leaving it one entry shorter than the observation vector; it is now reported with the cell. - The `scale` option of `[dataassim]` (multiply the predictions of named data types by a factor) never did anything: it iterated the characters of the column names. It now scales the named rows of the prediction matrix. @@ -980,3 +988,7 @@ and versions follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html). but likewise needs `subsurface.multphaseflow.opm.flow`, which is not a dependency of this repository, so neither notebook is executed by the docs build or CI. + +[Unreleased]: https://github.com/Python-Ensemble-Toolbox/PET/compare/v0.3...HEAD +[0.3]: https://github.com/Python-Ensemble-Toolbox/PET/compare/v0.2...v0.3 +[0.2]: https://github.com/Python-Ensemble-Toolbox/PET/compare/v0.1...v0.2