diff --git a/docs/guide/fba.md b/docs/guide/fba.md index c183227..ac0099b 100644 --- a/docs/guide/fba.md +++ b/docs/guide/fba.md @@ -317,4 +317,4 @@ The growth rate is unchanged and the total flux is the smallest that achieves it - [Getting started](getting-started.md), loading and inspecting a model. - [Growth media and conditions](media.md), what the exchange bounds mean and how to define a medium. -- [MATLAB vs Python](../raven3-vs-raven-toolbox.md), the full function mapping. +- [MATLAB vs Python](../matlab-vs-python.md), the full function mapping. diff --git a/docs/guide/getting-started.md b/docs/guide/getting-started.md index 72136f6..43241b0 100644 --- a/docs/guide/getting-started.md +++ b/docs/guide/getting-started.md @@ -272,5 +272,5 @@ An empty dictionary means the reaction balances. Anything listed is an element, ## See also - [Guide overview](index.md), the other pages, and the planned pages. -- [MATLAB vs Python](../raven3-vs-raven-toolbox.md), what each toolbox has, and where cobrapy takes over. +- [Comparison of RAVEN and raven-toolbox](../raven3-vs-raven-toolbox.md), what each toolbox has, and where cobrapy takes over. - [API reference](../api/index.md), every function in both toolboxes. diff --git a/docs/guide/init.md b/docs/guide/init.md index 7a83eb5..5954ebb 100644 --- a/docs/guide/init.md +++ b/docs/guide/init.md @@ -1,16 +1,16 @@ # Context-specific models with ftINIT -A genome-scale model describes what an organism can do. tINIT and ftINIT reduce it to what a particular sample (a tissue, a cell line, a condition) appears to be doing, from expression data plus a list of metabolic tasks that the result must be able to perform. +A genome-scale model describes what an organism can do. ftINIT reduces it to what a particular sample (a tissue, a cell line, a condition) appears to be doing, from expression data plus a list of metabolic tasks that the result must be able to perform. The legacy method tINIT, which ftINIT replaces, does the same for models that were built with it. The workflow uses Human-GEM and the RNA-seq data that Human-GEM provides, in both toolboxes. ## Choice of method -ftINIT is the recommended method, and the workflow here uses it. +ftINIT is the recommended method and replaces the legacy method tINIT. The workflow uses ftINIT. -Use tINIT (`getINITModel`) only to reproduce a model that was built with it. tINIT is supported and is not scheduled for removal. It issues a `RAVEN:legacyMethod` notice, which states that it is not intended for new work. Silence the notice with `warning('off','RAVEN:legacyMethod')`. +Use tINIT (`getINITModel`) to reproduce a model that was built with it. tINIT is supported and is not scheduled for removal. It issues a `RAVEN:legacyMethod` notice, which states that it is not intended for new work. Silence the notice with `warning('off','RAVEN:legacyMethod')`. -tINIT is MATLAB-only. RAVEN includes it for the models already built with it; raven-toolbox, an implementation without such an installed base, includes ftINIT only. A tINIT model has to be reproduced in MATLAB. +The legacy method tINIT is MATLAB-only. RAVEN includes it for the models already built with it; raven-toolbox, an implementation without such an installed base, includes ftINIT only. A tINIT model has to be reproduced in MATLAB. ## Relation between tINIT and ftINIT @@ -44,7 +44,7 @@ The same MATLAB workflow, as the model's own documentation presents it and inclu | `ftINIT` | `ftinit` | the staged extraction | | `scoreModel` | `score_reactions_from_genes` | gene scores → reaction scores | | no equivalent | `gene_scores_from_expression` | expression → gene scores | -| `runINIT`, `getINITModel` | no equivalent | tINIT, for reproducing models built with it | +| `runINIT`, `getINITModel` | no equivalent | the legacy tINIT method, for reproducing models built with it | | `removeLowScoreGenes` | `remove_low_score_genes` | prune negative-scoring genes from gene-protein-reaction rules (GPRs) | | `parseHPA`, `parseHPArna` | `parse_hpa`, `parse_hpa_rna` | read Human Protein Atlas (HPA) dumps | | no equivalent | `hpa_gene_scores`, `rna_gene_scores` | HPA levels or transcripts per million (TPM) to gene scores | diff --git a/docs/guide/io.md b/docs/guide/io.md index b88d6fe..c1613b5 100644 --- a/docs/guide/io.md +++ b/docs/guide/io.md @@ -266,4 +266,4 @@ print(sorted(p.relative_to(root).as_posix() for p in root.rglob("*") if p.is_fil - [Getting started](getting-started.md), what to do with the model once it is loaded. - [Guide overview](index.md), the other pages and the planned pages. -- [MATLAB vs Python](../raven3-vs-raven-toolbox.md), the full function mapping, including everything that resolves to cobrapy. +- [MATLAB vs Python](../matlab-vs-python.md), the full function mapping, including everything that resolves to cobrapy. diff --git a/docs/guide/media.md b/docs/guide/media.md index e243680..6536101 100644 --- a/docs/guide/media.md +++ b/docs/guide/media.md @@ -358,4 +358,4 @@ The values are uptake rates and not bounds, and the zeros are components that th - [Simulating growth with FBA](fba.md), the solve that uses the medium. - [Reading and writing models](io.md), keeping a model and its conditions in a repository. -- [MATLAB vs Python](../raven3-vs-raven-toolbox.md), the full function mapping. +- [MATLAB vs Python](../matlab-vs-python.md), the full function mapping. diff --git a/docs/guide/model-structure.md b/docs/guide/model-structure.md index 6bda0e8..17a83ef 100644 --- a/docs/guide/model-structure.md +++ b/docs/guide/model-structure.md @@ -327,4 +327,4 @@ RAVEN and the COBRA Toolbox use different field names for the same model. In MAT - [Getting started](getting-started.md), loading a model and inspecting it. - [Reading and writing models](io.md), where identifier prefixes come from. -- [MATLAB vs Python](../raven3-vs-raven-toolbox.md), the full function mapping. +- [MATLAB vs Python](../matlab-vs-python.md), the full function mapping. diff --git a/docs/installation/data-and-binaries.md b/docs/installation/data-and-binaries.md index 88ab3b2..97b41cf 100644 --- a/docs/installation/data-and-binaries.md +++ b/docs/installation/data-and-binaries.md @@ -49,7 +49,7 @@ The path `~/.cache` is used literally on every platform, and the platform conven The version is part of the path, so two KEGG releases coexist and a pinned run fetches the release it was pinned to when that release is missing from the cache. -In MATLAB the KEGG artefacts go to the directory `dataDir`, which `getKEGGModelForOrganism` requires as an argument. The executables are stored in `software/` inside the RAVEN installation directory on every platform. +In MATLAB the KEGG artefacts go to the directory `dataDir`, which `getKEGGModelForOrganism` requires as an argument. The executables are stored in `src/external/` (`blast+`, `diamond`, `hmmer`) inside the RAVEN installation directory on every platform. ## Tool resolution @@ -77,7 +77,7 @@ Downloading on first use puts the download inside the first run, which is not al downloadRavenBinaries ``` -`downloadRavenBinaries` fetches the executables that RAVEN needs into its `software/` directory. The KEGG artefacts are fetched separately, on the first `getKEGGModelForOrganism` call, into the `dataDir` given there. +`downloadRavenBinaries` fetches the executables that RAVEN needs into the `src/external/` directory of the RAVEN installation. The KEGG artefacts are fetched separately, on the first `getKEGGModelForOrganism` call, into the `dataDir` given there. ::: :::{tab-item} 🐍 Python :sync: python diff --git a/docs/installation/index.md b/docs/installation/index.md index 5f0f642..b887ea1 100644 --- a/docs/installation/index.md +++ b/docs/installation/index.md @@ -26,9 +26,9 @@ Both versions need a linear-programming solver. The options are the same for eit | Solver | Licence | Suited to | |---|---|---| -| Gurobi | Free academic | Genome-scale models, tINIT and ftINIT, gap-filling (recommended) | +| Gurobi | Free academic | Genome-scale models, ftINIT and the legacy tINIT method, gap-filling (recommended) | | GLPK | Open source | Small to medium models, getting started | -| SCIP | Open source | MILP problems (MATLAB) | +| SCIP | Open source | MILP problems (MATLAB); `setRavenSolver('scip')` downloads the binary on Windows only | In MATLAB, `setRavenSolver('gurobi')` selects the active solver. In raven-toolbox, the solver is configured through cobrapy (`cobra.Configuration`). diff --git a/docs/installation/raven.md b/docs/installation/raven.md index 185b202..dc8ba48 100644 --- a/docs/installation/raven.md +++ b/docs/installation/raven.md @@ -4,7 +4,7 @@ The RAVEN Toolbox runs in MATLAB and works independently. It does not require th ## Requirements -- MATLAB R2016b or later. R2016b is the oldest release that the code guards for. Continuous integration tests R2024b only. No additional MathWorks toolboxes are required. +- MATLAB, with no minimum release stated. The code has no check that rejects an older release. `readYAMLmodel` has a fallback for releases before R2020b, `parseTaskList` calls `readcell` (R2019a) for Excel task lists and `addIdentifierPrefix` calls `regexpPattern` (R2020b). Continuous integration tests R2024b only; older releases were not tested. No additional MathWorks toolboxes are required. - A linear-programming solver: [Gurobi](https://www.gurobi.com/) (free academic licence, recommended) or GLPK, bundled with RAVEN itself and needing no separate install. See [Choosing and configuring a solver](#choosing-a-solver-matlab). - RAVEN bundles `libSBML` and the GLPK mex files for Windows, macOS and Linux. `BLAST+`, `DIAMOND` and `HMMER` are not bundled: RAVEN fetches the build for the platform on first use, so the reconstruction functions need internet access the first time they run. `downloadRavenBinaries` fetches them ahead of time. [Download data and binaries](data-and-binaries.md) lists the full set, where it is cached and how to prepare an offline machine. @@ -16,7 +16,7 @@ The RAVEN Toolbox runs in MATLAB and works independently. It does not require th | Solver | LP | MILP | Relative speed | Licence | |---|---|---|---|---| | GLPK | yes | yes (RAVEN warns; slow on large problems) | 1x (baseline) | open source, bundled with RAVEN | -| SCIP | yes | yes | 0.33x | open source; bundled on Windows, a separate install on macOS or Linux | +| SCIP | yes | yes | 0.33x | open source; `setRavenSolver('scip')` downloads the MEX file on Windows (from the RAVEN v2.8.6 GitHub release), a separate install on macOS or Linux | | Gurobi | yes | yes | 4.2x | free academic licence | | COBRA Toolbox | yes | depends on the COBRA solver configured | depends on the COBRA solver configured | depends on the COBRA solver configured | diff --git a/docs/migrate.md b/docs/migrate.md index 7da30ac..e416adc 100644 --- a/docs/migrate.md +++ b/docs/migrate.md @@ -1,12 +1,11 @@ -# Migrate +# Migration guides -Two ways of moving between versions or languages, each answering a -different question. +Two pages compare toolboxes. One compares RAVEN 3 with RAVEN 2, the other RAVEN 3 with the Python package raven-toolbox. -| | | +| Page | Use | |---|---| -| [RAVEN 2 → RAVEN 3 (MATLAB)](raven3-migration.md) | Upgrading an existing MATLAB codebase: what was renamed, removed outright, or changed between RAVEN 2 and RAVEN 3. | -| [RAVEN 3 → raven-toolbox](raven3-vs-raven-toolbox.md) | Choosing between the two current implementations, or porting a MATLAB workflow to Python: what each toolbox has, and where they answer the same question differently. | +| [RAVEN 2 → RAVEN 3 (MATLAB)](raven3-migration.md) | Upgrading a MATLAB codebase: which functions were renamed or removed, and which behaviour differs between RAVEN 2 and RAVEN 3. | +| [RAVEN 3 → raven-toolbox](raven3-vs-raven-toolbox.md) | Choosing between the two current implementations or porting a MATLAB workflow to Python: what each toolbox has, and where they answer the same question differently. | ```{toctree} :hidden: diff --git a/docs/parameter-tuning/benchmarks.md b/docs/parameter-tuning/benchmarks.md index b9a3000..c81241d 100644 --- a/docs/parameter-tuning/benchmarks.md +++ b/docs/parameter-tuning/benchmarks.md @@ -1,27 +1,44 @@ # Benchmarks of `fseof`, `remove_genes` and other defaults -These benchmarks record parameters whose defaults were measured or checked without a study of their own. All were run with raven-toolbox commit `94f39407` on 2026-11-01: the effect of `flux_eps` on the targets of `fseof`, the handling of reactions without an enzyme in `remove_genes`, a sweep of three `fseof` parameters and a test of `connect_blocked_reactions.threshold`. A table lists literature and convention values with what supports each of them. +Six defaults of raven-toolbox were measured without a study of their own: `flux_eps` of `fseof`; `n_steps`, `max_fraction` and `correlation_threshold` of `fseof`; `blocked_reactions` of `remove_genes`; and `threshold` of `connect_blocked_reactions`. A table lists further defaults that rest on the literature or on a convention. All runs used raven-toolbox commit `94f39407` and were made on 2026-11-01. -The parameters with a measurement campaign of their own have studies listed under [Parameter tuning](index.md). [Tuned parameter defaults](../tuned-parameters.md) gives the current value and a one-line reason for every parameter of both implementations. +The measurements give no reason to change `flux_eps` from `1e-6`. The sweep does not select the other three `fseof` defaults, which follow [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10) (not checked against the paper), and the default of `blocked_reactions` gives the model that `removeGenes` of RAVEN gives by default. + +Parameters with a measurement campaign of their own have a study listed under [Parameter tuning](index.md). [Tuned parameter defaults](../tuned-parameters.md) gives the current value of every parameter of both implementations with a one-line reason. ## Set-up -The `fseof` benchmarks ran on iJO1366 (2,583 reactions) with the biomass reaction of the model as objective and `EX_succ_e` (succinate exchange) as target. In the `flux_eps` table the enforced flux ran from 1.5387 to 15.387 in 10 levels, and the unconstrained growth rate was 0.98237 h⁻¹. The other arguments were `n_steps` 10, `max_fraction` 0.9, `correlation_threshold` 0.9 and the default `equality=False`, which sets each enforced flux as the lower bound of the target reaction. +### `fseof` + +The `fseof` benchmarks ran on iJO1366 (2,583 reactions) with the biomass reaction of the model as the objective and `EX_succ_e` (succinate exchange) as the target. In the `flux_eps` runs the enforced flux ran from 1.5387 to 15.387 in 10 levels, and the unconstrained growth rate was 0.98237 h⁻¹. The other arguments were `n_steps` 10, `max_fraction` 0.9, `correlation_threshold` 0.9 and the default `equality=False`, which sets each enforced flux as the lower bound of the target reaction. + +At each enforced level `fseof` solves two problems, as `FSEOF` of RAVEN does. It first maximises the objective and then minimises the sum of fluxes with the objective held at 0.999999 of that optimum. A reaction is a target when four conditions hold: + +- its absolute flux exceeds `1e-10` at some level; +- the standard deviation of its absolute flux is above 0; +- the absolute correlation between its absolute flux and the enforced flux reaches `correlation_threshold`; +- the slope of the linear fit of its absolute flux against the enforced flux is not 0. -`fseof` solves each enforced level in two steps, as RAVEN's `FSEOF` does: it maximises the objective, then minimises the sum of fluxes with the objective held at 0.999999 of that optimum. A reaction is a target when its absolute flux exceeds `1e-10` at some level, the standard deviation of its absolute flux is above 0, the absolute correlation between its absolute flux and the enforced flux reaches `correlation_threshold`, and the slope of the linear fit of its absolute flux against the enforced flux is not 0. A target with a positive slope is an amplify target. A target with a negative slope is a knockout target when its last absolute flux is at most `1e-8`, and a knockdown target otherwise. `fseof` and RAVEN's `FSEOF` use these thresholds as fixed values. +A target with a positive slope is an amplify target. A target with a negative slope is a knockout target when its last absolute flux is at most `1e-8`, and a knockdown target otherwise. `fseof` and `FSEOF` of RAVEN use these thresholds as fixed values. -The `remove_genes` benchmark ran on e_coli_core (95 reactions) from the cobrapy distribution, with bounds as shipped, raven-toolbox commit `94f39407` and RAVEN commit `2d0630e3`. It deleted `b1779`, the only gene of GAPD (glyceraldehyde-3-phosphate dehydrogenase), with each setting of `blocked_reactions` in `remove_genes` and of `removeBlockedRxns` in RAVEN's `removeGenes`. The predicted growth is the optimal flux through the biomass reaction of the model, in h⁻¹. The unmodified model has 95 reactions and predicts a growth rate of 0.874 h⁻¹. Each call ran once. +The sweep changed one of `n_steps`, `max_fraction` and `correlation_threshold` at a time, with three runs per value on freshly read models. Every configuration gave the same counts and the same added and dropped targets in all three runs. The sweep reports two categories of target, and neither measures error: -The Python runs used Python 3.14.2, cobra 0.32.1, optlang 1.9.1 and Gurobi 13.0.3 (gurobipy wheel); MATLAB R2024b ran RAVEN with Gurobi 13.0.1 selected through `setRavenSolver('gurobi')`. The `flux_eps` runs were repeated three times per value, each on a freshly read model, and gave the same target set and labels every time. +1. **Added:** the target appears at the swept value and not in the default run, which is the set difference from the default. An added target is no evidence of error. +2. **Noise:** the standard deviation of the absolute flux of the target across the scan is below `1e-6`. -The sweep of `fseof` ran on iJO1366 with the settings above and Gurobi 13.0.3, changing one of `n_steps`, `max_fraction` and `correlation_threshold` at a time, with three runs per value on freshly read models. All three runs of every configuration gave the same counts and the same added and dropped targets. Two categories of target are reported, and neither measures error: +The default run contains 19 noise tracks. All are knockdown targets with a standard deviation of 5.08e-7, a slope of -1.15e-7 and a correlation of -1: `EGMEACPR`, `OGMEACPD`, `OPMEACPR`, `OPMEACPS`, `MALCOAMT`, `OGMEACPR`, `OGMEACPS`, `DM_amob_c`, `EX_meoh_e`, `AMAOTr`, `AOXSr2`, `BTS5`, `DBTS`, `EPMEACPR`, `I2FE2SR`, `MEOHtex`, `MEOHtrpp`, `OPMEACPD` and `PMEACPE` (biotin synthesis and methanol export). The classification keeps them, because it requires a standard deviation above 0. -1. **Added:** the target appears at the swept value and not in the default run (the set difference from the default). An added target is no evidence of error. -2. **Noise:** the standard deviation of its absolute flux across the scan is below `1e-6`. +### `remove_genes` -The default run contains 19 noise tracks, all knockdown targets with a standard deviation of 5.08e-7, a slope of -1.15e-7 and a correlation of -1: `EGMEACPR`, `OGMEACPD`, `OPMEACPR`, `OPMEACPS`, `MALCOAMT`, `OGMEACPR`, `OGMEACPS`, `DM_amob_c`, `EX_meoh_e`, `AMAOTr`, `AOXSr2`, `BTS5`, `DBTS`, `EPMEACPR`, `I2FE2SR`, `MEOHtex`, `MEOHtrpp`, `OPMEACPD` and `PMEACPE` (biotin synthesis and methanol export). The classification rule above keeps them, because it requires a standard deviation above 0. +The `remove_genes` benchmark ran on e_coli_core (95 reactions) from the cobrapy distribution, with the bounds as shipped, raven-toolbox commit `94f39407` and RAVEN commit `2d0630e3`. It deleted `b1779`, the only gene of GAPD (glyceraldehyde-3-phosphate dehydrogenase), with each setting of `blocked_reactions` in `remove_genes` and of `removeBlockedRxns` in `removeGenes` of RAVEN. The predicted growth is the optimal flux through the biomass reaction of the model, in h⁻¹. The unmodified model has 95 reactions and predicts a growth rate of 0.874 h⁻¹. Each call ran once. -The test of `connect_blocked_reactions.threshold` used a synthetic network: a draft with an import of A of at most 0.5, the reaction A to B, and B as a dead end, and a template with the exchange `EX_B`. The other arguments were at their defaults (`min_flux` 0.1). Each value ran once, with Gurobi 13.0.3. +### `connect_blocked_reactions` + +The test of `threshold` used a synthetic network. The draft has an import of A of at most 0.5, the reaction A to B, and B as a dead end, and the template has the exchange `EX_B`. The other arguments were at their defaults (`min_flux` 0.1). Each value ran once. + +### Software + +The Python runs used Python 3.14.2, cobra 0.32.1, optlang 1.9.1 and Gurobi 13.0.3 (gurobipy wheel). The MATLAB run used R2024b with RAVEN and Gurobi 13.0.1, selected through `setRavenSolver('gurobi')`. The `flux_eps` runs were repeated three times per value, each on a freshly read model. ## Results @@ -35,9 +52,11 @@ The test of `connect_blocked_reactions.threshold` used a synthetic network: a dr Of the 408 reactions, 406 are knockdown targets and 2 are knockout targets. Three runs per value gave the same set of targets. +`flux_eps` is the floor below which `fseof` treats the theoretical maximum of the target flux as zero, and the scan then raises an error. The classification of targets does not read `flux_eps`, and `FSEOF` of RAVEN has no such parameter. From the code, two values of `flux_eps` give different results if and only if the theoretical maximum of the target lies between them: `fseof` raises at the larger value and runs at the smaller. + ### `fseof` targets at other values of three parameters -iJO1366, target `EX_succ_e`. The last two columns list the targets that the swept value adds to the default set and drops from it. +The table lists iJO1366 with the target `EX_succ_e`. The last two columns give the targets that the swept value adds to the default set and drops from it. | Parameter | Value | Amplify targets | Knockdown and knockout targets | Added (amplify / knockdown and knockout) | Dropped (amplify / knockdown and knockout) | |---|---|---|---|---|---| @@ -49,7 +68,7 @@ iJO1366, target `EX_succ_e`. The last two columns list the targets that the swep | `correlation_threshold` | 0.7 | 22 | 416 | 4 / 8 (`FRD3`, `NADH18pp`, `ICL`, `MALS`; `EX_co2_e`, `CO2tex`, `CO2tpp`, `ICDHyr`, `AKGDH`, `PGI`, `TKT2`, `GLYCTO2`) | 0 / 0 | | `correlation_threshold` | 0.95 | 16 | 406 | 0 / 0 | 2 / 2 (`FUM`, `MDH`; `PDH`, `RPE`) | -Every swept value changes the target set. None of the added targets is a noise track: they are added, and the noise category does not apply to them. The four added amplify targets at `correlation_threshold` 0.7 have an absolute correlation of 0.71 to 0.72 in the default scan and a standard deviation of 0.62 to 4.4. Whether any added target is biologically wrong was not assessed. At `max_fraction` 0.5, `FBA3` and `PFK_3` change from knockout to knockdown targets; at 0.99, the 19 noise targets change from knockdown to knockout targets. +Every swept value changes the target set. None of the added targets is a noise track, because they are added relative to the default and the noise category does not apply to them. The four added amplify targets at `correlation_threshold` 0.7 have an absolute correlation of 0.71 to 0.72 in the default scan and a standard deviation of 0.62 to 4.4. At `max_fraction` 0.5, `FBA3` and `PFK_3` change from knockout to knockdown targets. At 0.99 the 19 noise targets change from knockdown to knockout targets. Whether any added target is biologically wrong was not assessed. ### `remove_genes` after deletion of `b1779` from e_coli_core @@ -63,9 +82,13 @@ Every swept value changes the target set. None of the added targets is a noise t | `removeGenes(model, genes, 'removeBlockedRxns', true)` | 94 | 0.000 | deleted | | unmodified model | 95 | 0.874 (MATLAB biomass flux 0.8739) | present | +`blocked_reactions` of `remove_genes` and `removeBlockedRxns` of `removeGenes` decide what happens to a reaction whose gene rule is emptied by a deletion. Both default settings keep such a reaction with bounds of zero, and the two gave the same model. After the deletion of `b1779` that model predicts no growth, because GAPD stays in the network without capacity to carry flux. With `"remove"` and with `removeBlockedRxns` true the model also predicted no growth. Both delete GAPD, which changes the number of reactions and the set of reaction identifiers. + +The setting `"keep"` leaves GAPD with an empty gene rule and unchanged bounds, so the model predicted the growth of the unmodified model, 0.874 h⁻¹, although the only gene of GAPD was deleted. The default setting predicted no growth after the same deletion, so the growth predicted with `"keep"` is probably a false positive in this model (the bounds are those shipped with e_coli_core). The docstring of `remove_genes` states no use for `"keep"`. The documentation names the editing of gene annotations without changing the network as a use, and the code gives no source for that use. + ### `connect_blocked_reactions` at four values of `threshold` -Synthetic network described in [Set-up](#set-up). +The draft network has an import of A, the reaction A to B and no exit for B, and the template supplies `EX_B`. | `threshold` | Gap filled | Reactions added | Maximum flux of the reaction A to B afterwards | |---|---|---|---| @@ -74,40 +97,32 @@ Synthetic network described in [Set-up](#set-up). | 0.5 | no | none | 0 | | 1.0 (default) | no | none | 0 | -At every value, `cannot_connect` also lists the reversible `EX_A`, which is blocked in the draft and is never a candidate. +At every value `cannot_connect` also lists the reversible `EX_A`, which is blocked in the draft and is never a candidate. ### Checked literature and convention values -Both implementations have the same value or behaviour for each of these parameters, and none was changed. The third column gives what supports the value; the three `fseof` rows and the `connect_blocked_reactions.threshold` row record a measurement. +Both implementations have the same value or behaviour for each of these parameters, and none was changed. The third column gives what supports the value. The three `fseof` rows and the `connect_blocked_reactions.threshold` row record a measurement, and the other rows were not run. | Parameter | Value | Basis | |---|---|---| -| `fseof.n_steps` | `10` | Value of [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10) (value not checked against the paper); on iJO1366, 5 steps gave 18 amplify and 410 knockdown and knockout targets and 20 steps gave 16 and 412, so the target set changes with `n_steps` | -| `fseof.max_fraction` | `0.9` | [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10) (value not checked against the paper); on iJO1366, 0.5 added 7 targets and dropped 4, and 0.99 gave the same counts as 0.9 with one target added and one dropped | -| `fseof.correlation_threshold` | `0.9` | [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10) (value not checked against the paper); on iJO1366, 0.7 added 4 amplify and 8 knockdown and knockout targets, and 0.95 dropped 2 and 2 | -| `connect_blocked_reactions.threshold` | `1.0` | Suited to models in the RAVEN convention, where exchange reactions have bounds of ±1000; on one synthetic network with a maximum flux of 0.5, values of 0.01 and 0.1 filled the gap and values of 0.5 and 1.0 did not (one run per value; the test shows that a reaction must carry more than the threshold, and no correctness comparison was made) | -| `connect_blocked_reactions.allow_net_production` | `False` | Keeps the steady-state balance `S @ v == 0`, the stricter test; `True` relaxes it to `S @ v >= 0` | +| `fseof.n_steps` | `10` | Value of [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10) (value not checked against the paper). On iJO1366, 5 steps gave 18 amplify and 410 knockdown and knockout targets and 20 steps gave 16 and 412, so the target set changes with `n_steps` | +| `fseof.max_fraction` | `0.9` | [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10) (value not checked against the paper). On iJO1366, 0.5 added 7 targets and dropped 4, and 0.99 gave the same counts as 0.9 with one target added and one dropped | +| `fseof.correlation_threshold` | `0.9` | [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10) (value not checked against the paper). On iJO1366, 0.7 added 4 amplify and 8 knockdown and knockout targets, and 0.95 dropped 2 and 2 | +| `connect_blocked_reactions.threshold` | `1.0` | Suited to models in the RAVEN convention, where exchange reactions have bounds of ±1000. On one synthetic network with a maximum flux of 0.5, values of 0.01 and 0.1 filled the gap and values of 0.5 and 1.0 did not (one run per value). The test shows that a reaction must carry more than the threshold, and no comparison of correctness was made | +| `connect_blocked_reactions.allow_net_production` | `False` | Keeps the steady-state balance `S @ v == 0`, the stricter test. `True` relaxes it to `S @ v >= 0` | | `fill_gaps_fast_lp.epsilon` | `1e-4` | Value of the fastGapFill paper, [Thiele et al. 2014](https://doi.org/10.1093/bioinformatics/btu321) | | `fill_gaps_milp.big_m` | `1000.0` | Matches the ±1000 bounds of the RAVEN convention | -| `check_tasks.close_boundaries` | `False` | Matches RAVEN's `checkTasks`, against which published task lists were curated; `True` closes the exchange reactions of the model so that the task defines the whole boundary | - -The three `fseof` rows and the `connect_blocked_reactions.threshold` row ran with commit `94f39407`. The other rows were not run. - -## `flux_eps` in `fseof` - -`flux_eps` is the floor below which `fseof` treats the theoretical maximum of the target flux as zero; the scan then raises an error. The classification of targets does not read `flux_eps`, and RAVEN's `FSEOF` has no such parameter. On iJO1366 with target `EX_succ_e`, the three values gave 18 amplify targets and 408 knockdown and knockout targets each. From the code, two values of `flux_eps` give different results only when the theoretical maximum of the target lies between them: `fseof` then raises at the larger value and runs at the smaller. - -## Reactions without an enzyme in `remove_genes` - -`blocked_reactions` in `remove_genes`, and `removeBlockedRxns` in RAVEN's `removeGenes`, decide what happens to a reaction whose gene rule is emptied by a deletion. By default both keep such a reaction with bounds of zero, and the two defaults gave the same model. After deletion of `b1779`, that model predicts no growth, because GAPD stays in the network with no capacity to carry flux. `"remove"` and `removeBlockedRxns` true also predicted no growth; they delete GAPD, which changes the number of reactions and the set of reaction identifiers. `"keep"` leaves GAPD with an empty gene rule and unchanged bounds, so the model predicted the growth of the unmodified model, 0.874 h⁻¹, although the only gene of GAPD was deleted. GAPD is essential for aerobic growth on glucose in e_coli_core, so the growth predicted with `"keep"` is a false positive. The docstring of `remove_genes` states no use for `"keep"`. Editing gene annotations without changing the network is a use given for it in the documentation, without a source in the code. +| `check_tasks.close_boundaries` | `False` | Matches `checkTasks` of RAVEN, against which published task lists were curated. `True` closes the exchange reactions of the model, which makes the task define the whole boundary | ## Choice of defaults -The measurement gives no reason to change `flux_eps` from `1e-6`. The sweep of `fseof` does not select the other three defaults: the target set changes with each swept value, by 2 to 12 targets, and no tested value leaves it unchanged. The defaults follow [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10) (not checked against the paper). The 19 noise tracks are in the target set at every setting. `blocked_reactions="constrain"` is the default of `remove_genes`; it gives the model that RAVEN's `removeGenes` gives by default. `"remove"` and `"keep"` are available on request. +The measurements give no reason to change `flux_eps` from `1e-6`. The sweep does not select the other three `fseof` defaults: the target set changes with each swept value by 2 to 12 targets, and no tested value leaves it unchanged. The defaults follow [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10) (not checked against the paper). The 19 noise tracks are in the target set at every setting. + +The default `blocked_reactions="constrain"` of `remove_genes` gives the model that `removeGenes` of RAVEN gives by default. The settings `"remove"` and `"keep"` are selected with the `blocked_reactions` argument. ## Limits -Each benchmark used one model: iJO1366 with one target for `fseof`, e_coli_core with one deleted gene for `remove_genes`, and a synthetic network for `connect_blocked_reactions`. Criterion 1 of the [evaluation methodology](index.md#evaluation-methodology) asks for a large and a small model. Gurobi was the sole solver. Whether the targets added by a swept value are biologically wrong was not assessed, and neither was the literature basis of the cited values. The rows `connect_blocked_reactions.allow_net_production`, `fill_gaps_fast_lp.epsilon`, `fill_gaps_milp.big_m` and `check_tasks.close_boundaries` rest on the literature or on a convention; no test was run for them. +Each benchmark used one model: iJO1366 with one target for `fseof`, e_coli_core with one deleted gene for `remove_genes`, and a synthetic network for `connect_blocked_reactions`. Criterion 1 of the [evaluation methodology](index.md#evaluation-methodology) asks for a large and a small model. Gurobi was the sole solver. Neither the biological correctness of the targets added by a swept value nor the literature basis of the cited values was assessed. The rows `connect_blocked_reactions.allow_net_production`, `fill_gaps_fast_lp.epsilon`, `fill_gaps_milp.big_m` and `check_tasks.close_boundaries` rest on the literature or on a convention, and no test was run for them. ## Source diff --git a/docs/parameter-tuning/studies/ftinit-determinism.md b/docs/parameter-tuning/studies/ftinit-determinism.md index a179400..ecb951e 100644 --- a/docs/parameter-tuning/studies/ftinit-determinism.md +++ b/docs/parameter-tuning/studies/ftinit-determinism.md @@ -1,12 +1,14 @@ # ftINIT reproducibility with `resolve_ties` and `prove_abs_gap` -This study builds Human-GEM models for one cell line with raven-toolbox's `ftinit` under three settings (default, `resolve_ties=True`, and `resolve_ties=True` with `prove_abs_gap`), varies the solver seed and the template, and records how many reactions and predicted essential genes change between builds and how long each build takes. The API documentation is in `raven_toolbox.init.ftinit`. +The study builds Human-GEM models for one cell line with the `ftinit` of raven-toolbox under three settings: the default, `resolve_ties=True`, and `resolve_ties=True` with `prove_abs_gap`. It varies the solver seed and the template, and records how many reactions and predicted essential genes change between builds and how long each build takes. The API documentation is in `raven_toolbox.init.ftinit`. -Two properties are measured. Determinism is that the same input, run again or with another seed, machine or solver build, gives the same model. Stability is that a similar input (a slightly edited template, or a comparable sample) gives a similar model. Both parameters are opt-in (`resolve_ties=False` and `prove_abs_gap=None` by default) and target determinism. In the measurements, the seed-to-seed difference fell from 110 reactions and 17 essential genes (default) to 41 reactions and 1 gene with `resolve_ties`, and to 1 reaction and 0 genes with `prove_abs_gap=1.0` added, at about 3.6 and 26 times the default build time. Neither parameter made the model stable under a change of the input: a small template edit moved a re-extraction by more reactions than a change of seed (156 against 110 at baseline, 59 against 41 with `resolve_ties`, 16 against 1 with `prove_abs_gap`) and by a similar number of essential genes (17 against 17, 1 against 1, 1 against 0). raven-toolbox has no parameter that anchors a re-extraction to a prior build. +Two properties are measured. Determinism means that the same input gives the same model when it is run again or with another seed, machine or solver build. Stability means that a similar input, such as a slightly edited template or a comparable sample, gives a similar model. Both parameters are opt-in (`resolve_ties=False` and `prove_abs_gap=None` by default) and target determinism. + +The seed-to-seed difference fell from 110 reactions and 17 essential genes (default) to 41 reactions and 1 gene with `resolve_ties`, and to 1 reaction and 0 genes with `prove_abs_gap=1.0` added. The build times were about 3.6 times (`resolve_ties`) and 26 times (`prove_abs_gap=1.0` added) the default build time. Neither parameter made the model stable under a change of the input. A small template edit moved a re-extraction by more reactions than a change of seed (156 against 110 at baseline, 59 against 41 with `resolve_ties`, 16 against 1 with `prove_abs_gap`) and by a similar number of essential genes (17 against 17, 1 against 1, 1 against 0). raven-toolbox has no parameter that anchors a re-extraction to a prior build. ## Main results -Each build in the next two tables is the full pipeline, including the task gap-fill, on Human-GEM and the DLD1 cell line (set-up in the next section). The seed-to-seed difference compares the build at seed 1234 with the build at seed 7. The repeat compares two builds at seed 1234 with nothing changed between them; it was run for two settings. For the baseline, the repeat (job A-base-rep) consists of two further builds (236.6 s and 230.9 s), separate from the seed 1234 build of the table (237.1 s). For `resolve_ties`, the repeat consists of two further builds (865 s and 861 s), separate from the seed 1234 build of the table (864 s). +Each build in the next two tables is the full pipeline, including the task gap-fill, on Human-GEM and the DLD1 cell line (the set-up is in the next section). The seed-to-seed difference compares the build at seed 1234 with the build at seed 7. The repeat compares two builds at seed 1234 with nothing changed between them, and it was run for two settings. For the baseline, the repeat (job A-base-rep) consists of two further builds (236.6 s and 230.9 s), separate from the seed 1234 build of the table (237.1 s). For `resolve_ties`, the repeat consists of two further builds (865 s and 861 s), separate from the seed 1234 build of the table (864 s). | Setting | Seed-to-seed difference, reactions | Seed-to-seed difference, essential genes | Essential genes (seed 1234 / seed 7) | Repeat difference at seed 1234, reactions / genes | Build time (seed 1234 / seed 7) | |---|---:|---:|---:|---:|---:| @@ -15,11 +17,11 @@ Each build in the next two tables is the full pipeline, including the task gap-f | `resolve_ties=True, prove_abs_gap=1.0` | 1 | 0 | 125 / 125 | not run | 5742 s / 6354 s | | `resolve_ties=True, prove_abs_gap=0.05` | 1 | 0 | 125 / 125 | not run | 5793 s / 6589 s | -The two `prove_abs_gap` values gave the same result. The build time grew from about 4 minutes (baseline) to about 14 minutes with `resolve_ties` (3.6 times) and about 100 minutes with `prove_abs_gap` added (26 times). +The two `prove_abs_gap` values gave the same result. The build time grew from about 4 minutes (baseline) to about 14 minutes with `resolve_ties` (3.6 times) and to about 100 minutes with `prove_abs_gap` added (26 times). -The two baseline builds at one seed differed by 2 reactions. Each baseline build had one run of the final step that stopped at its 5 s wall-clock limit; a run stopped at a time limit can return a different solution depending on machine load, and this is the presumed source of the 2 reactions (not tested). The two `resolve_ties` builds of the repeat had identical reaction sets. +The two baseline builds at one seed differed by 2 reactions. Each baseline build had one run of the final step that stopped at its 5 s wall-clock limit. A run stopped at a time limit can return a different solution depending on the machine load, and this is the presumed source of the 2 reactions (not tested). The two `resolve_ties` builds of the repeat had identical reaction sets. -The template edit removed 30 reactions that the reference builds did not use. They were drawn at random (seed 0) from the 2387 reactions of the prepared reference model that the build with `resolve_ties=True, prove_abs_gap=1.0` (seed 1234, 7754 reactions, 125 essential genes) did not keep and that are not task-essential. Of the 30, the `resolve_ties` builds kept 0 and the baseline build kept 1. Each setting was re-extracted from the edited template at seed 1234 and compared with its build from the unedited template: +The template edit removed 30 reactions that the reference builds did not use. They were drawn at random (seed 0) from the 2387 reactions of the prepared reference model that the build with `resolve_ties=True, prove_abs_gap=1.0` (seed 1234, 7754 reactions, 125 essential genes) did not keep and that are not task-essential. Of the 30, the `resolve_ties` builds kept 0 and the baseline build kept 1. Each setting was re-extracted from the edited template at seed 1234 and compared with its build from the unedited template. | Setting | Reaction difference | Essential-gene difference | Essential genes (unedited → edited) | Build time (edited) | |---|---:|---:|---:|---:| @@ -27,50 +29,27 @@ The template edit removed 30 reactions that the reference builds did not use. Th | `resolve_ties=True` | 59 | 1 | 114 → 115 | 638 s | | `resolve_ties=True, prove_abs_gap=1.0` | 16 | 1 | 125 → 126 | 4108 s | -The unedited builds hardly used the removed reactions. If the re-extraction changed only what the edit forces, the baseline model would lose the one removed reaction it kept and the `resolve_ties` models would lose none (inference, not tested). The baseline moved 156 reactions and 17 essential-gene calls; the two settings with `resolve_ties=True` moved 59 and 16 reactions and 1 gene each. +The unedited baseline build kept 1 of the 30 removed reactions and the unedited builds with `resolve_ties=True` kept none. If the re-extraction changed only what the edit forces, the baseline model would lose that reaction and the `resolve_ties` models would lose none (inference, not tested). The baseline moved 156 reactions and 17 essential-gene calls, and the two settings with `resolve_ties=True` moved 59 and 16 reactions and 1 gene each. ## Set-up -* Model and data: Human-GEM `v2.0.0-38`, Hart2015 DLD1 cell line, 57 essential tasks, series `1+1`. The `model/Human-GEM.yml` of that commit has 12877 reactions and 2848 genes. The preparation is the one of `estimateEssentialGenes.py` at that commit (removal of drug, amino-acid-triplet and duplicate complex I reactions), followed by raven-toolbox's own `prep_init_model`, which gives a reference model of 10141 reactions and a merged model of 6861 reactions (chains of linked reactions combined into one), with 206 task-essential reactions. This preparation differs from RAVEN's exported `raven_refModel.xml`, so absolute counts are not comparable to the [Human-GEM cell-type model validation against RAVEN](humangem-validation.md) run, which uses that export. -* Steps: series `1+1` runs two extraction steps, counted from 1 here; the debug log counts from 0. Step 1 decides the reactions with a gene association; step 2, the last step, adds the reactions without one and holds the reactions of step 1 on. -* Settings: `big_m=100` (the default is 1000), `mip_gap_abs=10` and `time_limit=1800` s, the values of `estimateEssentialGenes.py` at that commit. A named series such as `1+1` ignores `mip_gap` and `mip_gap_abs`; they apply to a step without a run schedule. -* Escalation: each solve stops at a mixed-integer programming (MIP) gap, the distance between the objective of the best solution found and the solver's bound on the optimum, either relative (divided by the objective) or absolute. Without `prove_abs_gap`, each step follows RAVEN's run schedule, here called the escalation. The first run uses a relative gap of 0.0004; each later run uses the larger of its relative gap and its absolute gap (10 or 20) divided by the objective, and the step stops as soon as the gap already reached meets the next target. `time_limit` caps the limit of each run: step 1 has runs of 120 s and 1800 s, the final step runs of 5 s, 120 s and 1800 s. -* With `prove_abs_gap`, each step is one solve proven to that absolute gap, with the 1800 s limit for the primary solve and for the two flux phases of `resolve_ties` (phases 2c and 2d), and half of it (`_TIE_BREAK_TIME_FRACTION = 0.5`) for its two on/off phases (phases 2a and 2b); the phases are defined in the section on the parameters. The task gap-fill uses a fixed 300 s per task in every setting. -* Solver: Gurobi through gurobipy, `Threads=1` unless stated, raven-toolbox at commit `94f39407`. Gurobi library 13.0.3 from the gurobipy wheel (most likely); the logs print no version. The builds ran as Slurm jobs on AMD EPYC 9354 nodes shared with other jobs, so wall-clock figures are approximate. Most jobs had 2 CPUs and the `Threads=4` jobs of the first-step probe had 4. Several builds ran in sequence in one job (the seeds of a setting, or the three builds of the stability test). -* Determinism probe: the extraction `seed` was varied (1234, 7, and further seeds where stated) with inputs, code, solver build and machine held fixed. A difference between builds then has two sources: the solver's choice among equal optima, measured where two builds reached the same objective value, and different objective values reached within the gap and time limits (see the location of the degeneracy and the sweep of `prove_abs_gap`). Differences between Gurobi versions were not measured. -* Metrics: the reaction difference is the symmetric difference of the kept-reaction sets (reactions kept by one build and not the other). The essential-gene difference is the symmetric difference of the sets of genes whose single deletion is predicted essential by `cobra.flux_analysis.find_essential_genes` at its default threshold. The gene difference is the result a user sees; the reaction difference is what the parameters act on. -* Debug log: all builds ran with `FTINIT_DEBUG=1`, which logs the status of each solve and of each tie-break phase. An incumbent is the best solution the solver holds when it stops; at a time limit it is not proven optimal. +- **Model and data.** Human-GEM `v2.0.0-38`, Hart2015 DLD1 cell line, 57 essential tasks, series `1+1`. The `model/Human-GEM.yml` of that commit has 12877 reactions and 2848 genes. The preparation is the one of `estimateEssentialGenes.py` at that commit (removal of the drug, amino-acid-triplet and duplicate complex I reactions), followed by `prep_init_model` of raven-toolbox. This gives a reference model of 10141 reactions and a merged model of 6861 reactions that contains 206 task-essential reactions. In the merged model, chains of linked reactions are combined into one reaction. The preparation differs from the exported `raven_refModel.xml` of RAVEN, so absolute counts are not comparable to the [Human-GEM cell-type model validation against RAVEN](humangem-validation.md), which uses that export. +- **Steps.** Series `1+1` runs two extraction steps, counted from 1 in this study; the debug log counts from 0. Step 1 decides the reactions with a gene association. Step 2, the last step, adds the reactions without one and holds the reactions of step 1 on. +- **Settings.** `big_m=100` (the default is 1000), `mip_gap_abs=10` and `time_limit=1800` s, the values of `estimateEssentialGenes.py` at that commit. A named series such as `1+1` ignores `mip_gap` and `mip_gap_abs`, which apply to a step without a run schedule. +- **Escalation.** Each solve stops at a mixed-integer programming (MIP) gap, the distance between the objective of the best solution found and the bound of the solver on the optimum, either relative (divided by the objective) or absolute. Without `prove_abs_gap`, each step follows the run schedule of RAVEN, called the escalation in this study. The first run uses a relative gap of 0.0004. Each later run uses the larger of its relative gap and its absolute gap (10 or 20) divided by the objective, and the step stops as soon as the gap already reached meets the next target. The setting `time_limit` caps the limit of each run: step 1 has runs of 120 s and 1800 s, and the final step runs of 5 s, 120 s and 1800 s. +- **`prove_abs_gap`.** Each step is one solve proven to that absolute gap. It has the 1800 s limit for the primary solve and for the two flux phases of `resolve_ties` (phases 2c and 2d), and half of it (`_TIE_BREAK_TIME_FRACTION = 0.5`) for its two on/off phases (phases 2a and 2b). The phases are defined in the section on the parameters. The task gap-fill uses a fixed 300 s per task in every setting. +- **Solver.** Gurobi through gurobipy, `Threads=1` unless stated, raven-toolbox at commit `94f39407`. The Gurobi library is 13.0.3 from the gurobipy wheel (most likely; the logs print no version). The builds ran as Slurm jobs on AMD EPYC 9354 nodes shared with other jobs, so wall-clock figures are approximate. Most jobs had 2 CPUs, and the `Threads=4` jobs of the first-step probe had 4. Several builds ran in sequence in one job: the seeds of a setting, or the three builds of the stability test. +- **Determinism probe.** The extraction `seed` was varied (1234, 7, and further seeds where stated) with inputs, code, solver build and machine held fixed. A difference between builds then has two sources: the choice of the solver among equal optima, measured where two builds reached the same objective value, and different objective values reached within the gap and time limits (see the location of the degeneracy and the sweep of `prove_abs_gap`). Differences between Gurobi versions were not measured. +- **Metrics.** The reaction difference is the symmetric difference of the kept-reaction sets, that is the reactions kept by one build and not by the other. The essential-gene difference is the symmetric difference of the sets of genes whose single deletion is predicted essential by `cobra.flux_analysis.find_essential_genes` at its default threshold. The gene difference is the result that a user sees, and the reaction difference is what the parameters act on. +- **Debug log.** All builds ran with `FTINIT_DEBUG=1`, which logs the status of each solve and of each tie-break phase. An incumbent is the best solution that the solver holds when it stops, and at a time limit it is not proven optimal. ## Degeneracy of the extraction MILP -The extraction mixed-integer linear program (MILP) has many optima of equal score. In the first step on Human-GEM and DLD1, 1547 merged reactions have a negative score, and these carry the 0/1 keep-or-drop variables. Of them, 98.3 % (1520) are in blocks of equal score. The largest block has 840 reactions at −5.0. The second has 420 reactions at −2.0, the default score of a reaction without gene association; across the five Hart2015 cell lines it has 419 to 421 members, so the expression data change its size by at most 2 members. Many reaction subsets reach the same optimum, and the solver returns one of them. - -`Threads=1` and a fixed `Seed` make that choice close to reproducible on one machine: the two baseline builds at one seed differed by 2 reactions. Another seed can select another member of the tied set, or end the solve at another objective value within the gap and time limits; both were seen in the first-step probe, and in either case the predicted essential genes change for reasons unrelated to the model or the data. Another Gurobi version or platform is expected to act the same way; that was not measured. - -## Parameters `resolve_ties` and `prove_abs_gap` - -`resolve_ties` adds a lexicographic phase 2 after the primary solve of each extraction step, with the score objective held at its optimum. The phases are, in order: - -* `phase2a-parsimony`: minimise the count of kept removable (negative-score) reactions. -* `phase2b-idrank`: among those, minimise the summed id rank of the kept reactions. A merged reaction is ranked by the smallest original id in its group (`rank_keys`), so the rank does not depend on which member names the merged reaction. -* `phase2c-minflux`: for that set of kept reactions, minimise the total absolute flux (gap 1e-6), so the fluxes that the next step reads for its forced directions and magnitudes do not depend on the solver's vertex. -* `phase2d-direction`: among the minimal-flux solutions, minimise the id-rank-weighted backward flux of the reactions whose flux can take either sign (gap 1e-6), so a flux that carries the same total either way gets a fixed direction. - -The last step skips `phase2c-minflux` and `phase2d-direction`, because no later step reads its fluxes. The task gap-fill uses the first two phases. Phases 2a and 2b get half of `time_limit` each; phases 2c and 2d get the full `time_limit`. - -`prove_abs_gap` proves each step to a fixed absolute MIP gap in a single solve. It replaces the escalation, whose final run, at an objective near zero, accepts any incumbent within its gap. - -A phase that does not prove its optimum has these outcomes: - -* A phase that stops at `time_limit` with an incumbent is adopted, and a `UserWarning` names it as an unproven incumbent. `phase2a-parsimony` and `phase2b-idrank` share one warning, which is raised when `phase2b-idrank` ends with an incumbent and not otherwise; `phase2c-minflux` and `phase2d-direction` each have their own. -* A flux phase (`phase2c-minflux`, `phase2d-direction`) that finds no solution warns with its name and keeps the fluxes of the phase before it. -* When `phase2a-parsimony` or `phase2b-idrank` finds no incumbent, the on/off tie-break of that solve is dropped without a warning: the primary solution is kept and the ties of that step are not resolved. A build without a warning can therefore hold an unresolved step. -* When the primary solve of a step stops at `time_limit`, all tie-break phases of that solve are skipped and the primary solution is kept. -* The escalation raises no warning either. In the four sweep runs with the escalation and `resolve_ties=True`, the step 1 objective ended 4.0 to 12.0 below the proven optimum; at least one of the 12 first-step runs without `resolve_ties` reached it (the location of the degeneracy). +The extraction mixed-integer linear program (MILP) has many optima of equal score. In the first step on Human-GEM and DLD1, 1547 merged reactions have a negative score, and these carry the 0/1 keep-or-drop variables. Of them, 98.3 % (1520) are in blocks of equal score. The largest block has 840 reactions at −5.0. The second has 420 reactions at −2.0, the default score of a reaction without gene association. Across the five Hart2015 cell lines it has 419 to 421 members, so the expression data change its size by at most 2 members. Many reaction subsets reach the same optimum, and the solver returns one of them. -For comparison, RAVEN's `ftINIT` with `resolveTies` skips the tie-break after a primary solve that stopped at the time limit, gives the count and id-rank phases half of `params.TimeLimit` each, skips the minimal-flux and direction phases in the last step, and names every unproven phase (`phase2a-parsimony`, `phase2b-idrank`, `phase2c-minflux`, `phase2d-direction`) in its "tie resolution did not converge" warning. In RAVEN's `src/INIT/ftINITInternalAlg.m`, the skip after a primary solve at the time limit is at line 483, the half limit at line 518 and the warning at line 662; `src/INIT/ftINIT.m` line 227 sets `pinFluxes` to false in the last step. RAVEN's `ftINIT` was not run in this study. +The settings `Threads=1` and a fixed `Seed` make that choice close to reproducible on one machine, as shown by the two baseline builds at one seed differed by 2 reactions. Another seed can select another member of the tied set, or end the solve at another objective value within the gap and time limits. Both were seen in the first-step probe, and in either case the predicted essential genes change for reasons unrelated to the model or the data. Another Gurobi version or platform is expected to act the same way, which was not measured. -## Location of the degeneracy +### Location of the degeneracy Six seeds were run through the first step alone (a one-step series, so the flux phases of `resolve_ties` do not run), with and without `resolve_ties`, at `Threads` 1 and 4. The mean pairwise difference is the reaction difference averaged over the 15 pairs of seeds. @@ -85,9 +64,28 @@ At the same seed, the baseline solutions at `Threads` 1 and 4 were 5 to 20 react Grouping the runs by objective value separates the tie-break from the primary solve. The 12 baseline runs (both thread settings) reached six objective values between 45887.113 and 45899.113. At a fixed objective value, the baseline returned 4 distinct solutions from 4 runs at 45892.113 (pairwise differences 6 to 60 reactions) and 4 from 4 at 45894.113 (35 to 53). With `resolve_ties`, the 4 runs at each of these two objective values returned one solution each. The 12 `resolve_ties` runs gave 5 distinct reaction sets, and the remaining differences came from runs whose primary solve reached different objective values. +## Parameters `resolve_ties` and `prove_abs_gap` + +The parameter `resolve_ties` adds a lexicographic phase 2 after the primary solve of each extraction step, with the score objective held at its optimum. Four phases follow in order, each held at the optimum of the one before. First, `phase2a-parsimony` minimises the count of kept removable (negative-score) reactions. Then `phase2b-idrank` minimises, among those, the summed id rank of the kept reactions; a merged reaction is ranked by the smallest original id in its group (`rank_keys`), so the rank does not depend on which member names the merged reaction. Then `phase2c-minflux` minimises, for that set of kept reactions, the total absolute flux (gap 1e-6), so the fluxes that the next step reads for its forced directions and magnitudes do not depend on the vertex of the solver. Finally `phase2d-direction` minimises, among the minimal-flux solutions, the id-rank-weighted backward flux of the reactions whose flux can take either sign (gap 1e-6), so a flux that carries the same total either way gets a fixed direction. + +The last step skips `phase2c-minflux` and `phase2d-direction`, because no later step reads its fluxes. The task gap-fill uses the first two phases. Phases 2a and 2b get half of `time_limit` each, and phases 2c and 2d get the full `time_limit`. + +The parameter `prove_abs_gap` proves each step to a fixed absolute MIP gap in a single solve. It replaces the escalation, whose final run, at an objective near zero, accepts any incumbent within its gap. + +A phase that does not prove its optimum has these outcomes: + +- A phase that stops at `time_limit` with an incumbent is adopted, and a `UserWarning` names it as an unproven incumbent. The phases `phase2a-parsimony` and `phase2b-idrank` share one warning, which is raised when `phase2b-idrank` ends with an incumbent and not otherwise. The phases `phase2c-minflux` and `phase2d-direction` each have their own. +- A flux phase (`phase2c-minflux`, `phase2d-direction`) that finds no solution warns with its name and keeps the fluxes of the phase before it. +- When `phase2a-parsimony` or `phase2b-idrank` finds no incumbent, the on/off tie-break of that solve is dropped without a warning. The primary solution is kept and the ties of that step are not resolved, so a build without a warning can hold an unresolved step. +- When the primary solve of a step stops at `time_limit`, all tie-break phases of that solve are skipped and the primary solution is kept. + +The escalation raises no warning either. In the four sweep runs with the escalation and `resolve_ties=True`, the step 1 objective ended 4.0 to 12.0 below the proven optimum. At least one of the 12 first-step runs without `resolve_ties` reached it (see the location of the degeneracy). + +For comparison, `ftINIT` of RAVEN with `resolveTies` skips the tie-break after a primary solve that stopped at the time limit, gives the count and id-rank phases half of `params.TimeLimit` each, and skips the minimal-flux and direction phases in the last step. Its warning "tie resolution did not converge" names every unproven phase (`phase2a-parsimony`, `phase2b-idrank`, `phase2c-minflux`, `phase2d-direction`). In `src/INIT/ftINITInternalAlg.m` of RAVEN, the skip after a primary solve at the time limit is at line 483, the half limit at line 518 and the warning at line 662, and `src/INIT/ftINIT.m` line 227 sets `pinFluxes` to false in the last step. `ftINIT` of RAVEN was not run in this study. + ## Effect of the `prove_abs_gap` value -The value was swept with `resolve_ties=True` on the two-step pipeline without gap-fill, at four seeds (1234, 7, 1, 2) per value. "`None`" is the escalation with `resolve_ties=True`. A step time includes the tie-break phases of that step. All 48 solves of step 1 and step 2 with a `prove_abs_gap` value ended with status `optimal`. +The value was swept with `resolve_ties=True` on the two-step pipeline without gap-fill, at four seeds (1234, 7, 1, 2) per value. The row `None` is the escalation with `resolve_ties=True`. A step time includes the tie-break phases of that step. All 48 solves of step 1 and step 2 with a `prove_abs_gap` value ended with status `optimal`. | `prove_abs_gap` | Step 1 objectives (4 seeds) | Step 2 objectives (4 seeds) | Step 1 time | Step 2 time | |---|---|---|---:|---:| @@ -99,15 +97,15 @@ The value was swept with `resolve_ties=True` on the two-step pipeline without ga | 0.1 | 45899.113 (4) | −613.872 (3), −611.872 | 3313 to 3650 s | 1757 to 2721 s | | 0.05 | 45899.113 (4) | −613.872 (3), −611.872 | 3317 to 3650 s | 1758 to 2734 s | -None of the four escalation runs reached the proven step 1 optimum of 45899.113; they ended 4.0 to 12.0 below it, without a warning. `prove_abs_gap=5.0` stopped 2.0 below it for one seed, and every value from 2.0 to 0.05 reached it for all four seeds. The step 2 objectives depend on the step 1 result, so they are comparable between rows where step 1 is equal. With a proven step 1, step 2 gave −613.872 for three seeds and −611.872 for seed 7. In step 1 of the `prove_abs_gap=1.0` builds, `phase2c-minflux` stopped at the time limit for 3 of the 4 seeds and was optimal for 1, and `phase2d-direction` stopped at the time limit for all 4, so the fluxes passed to step 2 were unproven incumbents; whether this causes the seed 7 difference was not tested. +None of the four escalation runs reached the proven step 1 optimum of 45899.113. They ended 4.0 to 12.0 below it, without a warning. The value `prove_abs_gap=5.0` stopped 2.0 below it for one seed, and every value from 2.0 to 0.05 reached it for all four seeds. The step 2 objectives depend on the step 1 result, so they are comparable between rows where step 1 is equal. With a proven step 1, step 2 gave −613.872 for three seeds and −611.872 for seed 7. In step 1 of the `prove_abs_gap=1.0` builds, `phase2c-minflux` stopped at the time limit for 3 of the 4 seeds and was optimal for 1, and `phase2d-direction` stopped at the time limit for all 4. The fluxes passed to step 2 were therefore unproven incumbents, and whether this causes the seed 7 difference was not tested. All values from 2.0 to 0.05 gave the same objectives and the same final reaction sets. A tighter gap added little run time: step 2 took up to 2734 s at 0.05 against 2500 s at 1.0, and a whole build took 5770 s against 5640 s on average. Per-phase times were not logged, so the split of step 2 between the proof and the tie-break is not known. -The final reaction sets of the sweep (without gap-fill) were: four distinct sets with the escalation, 37 to 56 reactions apart; three with `prove_abs_gap=5.0`; two with every value from 2.0 to 0.05, which differ by one reaction (seed 7 against the other three). +The final reaction sets of the sweep (without gap-fill) were four distinct sets with the escalation, 37 to 56 reactions apart, three with `prove_abs_gap=5.0`, and two with every value from 2.0 to 0.05, which differ by one reaction (seed 7 against the other three). ## Tie-break phases at genome scale -With a proven step 1 objective, the tie-break phases did not all prove within the time limit. The phase statuses of the eight tie-broken solves (two steps, four seeds) at `prove_abs_gap=1.0` were as follows; the other values from 0.05 to 5.0 gave the same counts. +With a proven step 1 objective, the tie-break phases did not all prove within the time limit. The table gives the phase statuses of the eight tie-broken solves (two steps, four seeds) at `prove_abs_gap=1.0`. The other values from 0.05 to 5.0 gave the same counts. | Phase | Optimal | Stopped at `time_limit` | |---|---:|---:| @@ -116,7 +114,7 @@ With a proven step 1 objective, the tie-break phases did not all prove within th | `phase2c-minflux` (step 1) | 1 | 3 | | `phase2d-direction` (step 1) | 0 | 4 | -Step 2 is the last step, so it skips the last two phases. The log of seed 1234: +Step 2 is the last step, so it skips the last two phases. The log of seed 1234 is: ``` [ftinit] final solve: obj=45899.11 status=optimal achieved_gap=2.8e-06 @@ -129,11 +127,11 @@ Step 2 is the last step, so it skips the last two phases. The log of seed 1234: [ftinit] tie-break phase2b-idrank: status=optimal obj=346311.0 ``` -The on/off tie-break of step 1 was proven for all four seeds, so the step 1 reaction set is fixed. `phase2c-minflux` reached the 1800 s limit for 3 of the 4 seeds and `phase2d-direction` for all 4, and each was adopted as an unproven incumbent. In the three seeds in which `phase2c-minflux` stopped at the limit, the two flux phases at the 1800 s limit take 3600 s, which accounts for most of the step 1 time of about 3650 s and for a build time of about 100 minutes. The flux that step 2 reads is therefore the incumbent of a time-limited solve. The step 2 objective differed by seed (−611.872 for seed 7, −613.872 for the other three) and the final sets by one reaction. The fluxes themselves were not compared between seeds; a seed-dependent flux is inferred from the step 2 objectives, not measured. +The on/off tie-break of step 1 was proven for all four seeds, so the step 1 reaction set is fixed. The phase `phase2c-minflux` reached the 1800 s limit for 3 of the 4 seeds and `phase2d-direction` for all 4, and each was adopted as an unproven incumbent. In the three seeds in which `phase2c-minflux` stopped at the limit, the two flux phases at the 1800 s limit take 3600 s, which accounts for most of the step 1 time of about 3650 s and for a build time of about 100 minutes. The flux that step 2 reads is therefore the incumbent of a time-limited solve. The step 2 objective differed by seed (−611.872 for seed 7, −613.872 for the other three) and the final sets by one reaction. The fluxes themselves were not compared between seeds, and a seed-dependent flux is inferred from the step 2 objectives and was not measured. -The escalation builds with `resolve_ties=True` also adopted phases at a time limit, here the limit of the escalation run (120 s in step 1, so 60 s for phases 2a and 2b). The full-pipeline build of seed 1234 adopted 3 phases (`phase2b-idrank`, `phase2c-minflux`, `phase2d-direction` of step 1) and the build of seed 7 adopted 2 (`phase2c-minflux`, `phase2d-direction` of step 1). In step 2 of both builds, `phase2a-parsimony` ended at the time limit with an incumbent and `phase2b-idrank` ended at the time limit with no incumbent (`A-rt-1234.log`, `A-rt-7.log`). `phase2b-idrank` then returns no solution, the tie-break of step 2 returns `False`, the primary solution of step 2 is kept, and no warning is raised. The reaction sets of these two builds rest on a resolved step 1 and an unresolved step 2. +The escalation builds with `resolve_ties=True` also adopted phases at a time limit, that is the limit of the escalation run (120 s in step 1, so 60 s for phases 2a and 2b). The full-pipeline build of seed 1234 adopted 3 phases (`phase2b-idrank`, `phase2c-minflux`, `phase2d-direction` of step 1) and the build of seed 7 adopted 2 (`phase2c-minflux`, `phase2d-direction` of step 1). In step 2 of both builds, `phase2a-parsimony` ended at the time limit with an incumbent and `phase2b-idrank` ended at the time limit with no incumbent (`A-rt-1234.log`, `A-rt-7.log`). The phase `phase2b-idrank` then returns no solution, the tie-break of step 2 returns `False`, the primary solution of step 2 is kept, and no warning is raised. The reaction sets of these two builds rest on a resolved step 1 and an unresolved step 2. -Counts of full builds (gap-fill included) with at least one unproven tie-break phase or a time-limited primary solve. The full builds of the main results are, per setting: baseline 5 (seeds 1234 and 7, the two builds of the repeat, the edited template), `resolve_ties=True` 5 (seeds 1234 and 7, the two builds of the repeat, the edited template), `prove_abs_gap` 5 (seeds 1234 and 7 at each of 1.0 and 0.05, the edited template at 1.0). The Builds column gives how many of them were counted. +The next table counts full builds (gap-fill included) with at least one unproven tie-break phase or a time-limited primary solve. The column Builds gives how many full builds of the setting were counted. | Setting | Builds | With an adopted phase at `time_limit` | With a primary solve at `time_limit` | With a tie-break dropped without a warning | |---|---:|---:|---:|---:| @@ -143,35 +141,35 @@ Counts of full builds (gap-fill included) with at least one unproven tie-break p Each row counts all builds of its setting, including the edited build at 1.0 for `prove_abs_gap`. The step 2 logs cited are those of seeds 1234 and 7. -The warnings cover the adopted phases and the flux phases without a solution. A build without a warning can hold an unresolved tie-break (a dropped `phase2a-parsimony` or `phase2b-idrank`) or an unproven escalation, and the debug log is the record of what each solve reached. In the sweep, all 24 builds with `prove_abs_gap` raised at least one warning. A larger `time_limit` is the setting that can let the phases prove; its effect was not measured. +The warnings cover the adopted phases and the flux phases without a solution. A build without a warning can hold an unresolved tie-break (a dropped `phase2a-parsimony` or `phase2b-idrank`) or an unproven escalation, and the debug log is the record of what each solve reached. In the sweep, all 24 builds with `prove_abs_gap` raised at least one warning. A larger `time_limit` is the setting that can let the phases prove, and its effect was not measured. ## Stability under a small template change The template-edit result is in the second table of the main results. The two settings with `resolve_ties=True` moved fewer reactions than the baseline, and for each setting the edit moved more reactions than a change of seed. Determinism does not give stability: a deterministic extractor returns the same model for the same input, and the edit changes the input. -Settings on identical inputs also differ. Baseline and `resolve_ties` at seed 1234 differ by 162 reactions and 17 essential-gene calls (125 against 114), with the same step 1 objective (45892.113) and different step 2 objectives (−657.240 for the baseline, −616.906 for `resolve_ties=True`). The step 1 difference is a different choice among solutions of equal score. The step 2 difference includes more than that choice, because step 2 reads the fluxes of step 1 and ends at a different objective. +Settings on identical inputs also differ. Baseline and `resolve_ties` at seed 1234 differ by 162 reactions and 17 essential-gene calls (125 against 114), with the same step 1 objective (45892.113) and different step 2 objectives (−657.240 for the baseline, −616.906 for `resolve_ties=True`). The step 1 difference is a different choice among solutions of equal score. The step 2 difference includes more than that choice, because step 2 reads the fluxes of step 1 and ends at another objective. The growth objective of the models did not indicate stability on this data: the unedited builds of one setting (`resolve_ties=True`) had 74.6 for seed 1234 and 83.1 for seed 7. -For curation of a template (fix one or a few reactions, re-extract, compare), the re-extraction difference contains the effect of the edit together with re-selection among tied solutions. Applying the same edit to the extracted model gives a control: the difference between that edited model and the unedited extraction contains the effect of the edit alone. No result for this control is reported. +For the curation of a template (fix one or a few reactions, re-extract, compare), the re-extraction difference contains the effect of the edit together with the re-selection among tied solutions. Applying the same edit to the extracted model gives a control: the difference between that edited model and the unedited extraction contains the effect of the edit alone. No result for this control is reported. ## Choice of settings -* `resolve_ties=True`: the repeat pair at seed 1234 had identical reaction sets, and the seed-to-seed difference fell from 17 genes to 1 on this data, at about 3.6 times the build time (4 to 14 minutes). Its warnings name the adopted phases that stopped at the time limit. A tie-break that finds no incumbent in phase 2a or 2b is dropped without a warning; the builds at seeds 1234 and 7 dropped the one of step 2, and the table of build counts gives 5 of 5 builds with a dropped step 2 tie-break. With the escalation, the tie-break runs again in each run of the schedule (`_solve_step` in `ftinit.py` and the run schedule in `steps.py`): the flux phases of step 1 get 120 s when step 1 stops after its first run and 1800 s when a second run follows (its 5000 s capped by `time_limit`). -* `prove_abs_gap=1.0` together with `resolve_ties=True` reduced the seed-to-seed difference from 41 to 1 reaction (1 to 0 genes) at 26 times the baseline time, about 100 minutes at `time_limit=1800`. The objectives of both steps are proven to 1.0 (in the four sweep runs with the escalation and `resolve_ties=True`, step 1 ended 4.0 to 12.0 below the optimum). In the four `prove_abs_gap=1.0` sweep builds (step 1 and step 2, no gap-fill; 4987 to 6183 s wall clock), step 1 took 3305 to 3650 s, 59.0 to 66.3 % of each build, mostly in the two flux phases, and step 2 another 1650 to 2500 s. A value from 2.0 to 0.05 gave the same models; 5.0 stopped 2.0 below the optimum for one seed. No build with `prove_abs_gap` and without `resolve_ties` was run. -* Gurobi versions were not compared; two baseline builds with everything held fixed differed by 2 reactions. +- **`resolve_ties=True`.** The repeat pair at seed 1234 had identical reaction sets, and the seed-to-seed difference fell from 17 genes to 1 on this data, at about 3.6 times the build time (4 to 14 minutes). Its warnings name the adopted phases that stopped at the time limit. A tie-break that finds no incumbent in phase 2a or 2b is dropped without a warning. The builds at seeds 1234 and 7 dropped the one of step 2, and the table of build counts gives 5 of 5 builds with a dropped step 2 tie-break. With the escalation, the tie-break runs again in each run of the schedule (`_solve_step` in `ftinit.py` and the run schedule in `steps.py`). The flux phases of step 1 get 120 s when step 1 stops after its first run and 1800 s when a second run follows (its 5000 s capped by `time_limit`). +- **`prove_abs_gap=1.0` with `resolve_ties=True`.** The seed-to-seed difference fell from 41 to 1 reaction (1 to 0 genes) at 26 times the baseline time, which is about 100 minutes at `time_limit=1800`. The objectives of both steps are proven to 1.0; in the four sweep runs with the escalation and `resolve_ties=True` step 1 ended 4.0 to 12.0 below the optimum. In the four `prove_abs_gap=1.0` sweep builds (step 1 and step 2, no gap-fill; 4987 to 6183 s wall clock), step 1 took 3305 to 3650 s, 59.0 to 66.3 % of each build, mostly in the two flux phases, and step 2 took another 1650 to 2500 s. A value from 2.0 to 0.05 gave the same models, and 5.0 stopped 2.0 below the optimum for one seed. No build with `prove_abs_gap` and without `resolve_ties` was run. +- **Gurobi versions.** They were not compared. Two baseline builds with everything held fixed differed by 2 reactions. ## Limits -The full-pipeline determinism table rests on one seed pair (seeds 1234 and 7), the first-step probe on six seeds and the sweep on four. The study used one cell line, one machine type, one Gurobi build and one `time_limit`, and one edit (30 unused reactions removed) for the stability measurement. Each figure comes from one build per seed and setting, except the repeat pairs at seed 1234, so the data give direction and rough magnitude and no error bars. +The determinism table of the full pipeline rests on one seed pair (seeds 1234 and 7), the first-step probe on six seeds and the sweep on four. The study used one cell line, one machine type, one Gurobi build and one `time_limit`, and one edit (30 unused reactions removed) for the stability measurement. Each figure comes from one build per seed and setting, except the repeat pairs at seed 1234, so the data give direction and rough magnitude and no error bars. -Differences between Gurobi versions, the failure that `resolve_ties` and `prove_abs_gap` target, were not measured; the seed probe stands in for them, and how well it represents them is not known. The stability edit removed 30 of 10141 reference reactions, of which the reference builds kept at most 1. Two comparable samples (for example, two expression profiles of one tissue) were not compared. +Differences between Gurobi versions, the failure that `resolve_ties` and `prove_abs_gap` target, were not measured. The seed probe stands in for them, and how well it represents them is not known. The stability edit removed 30 of 10141 reference reactions, of which the reference builds kept at most 1. Two comparable samples, for example two expression profiles of one tissue, were not compared. -Wall-clock figures come from jobs that shared nodes with other jobs and are approximate; the difference and objective columns are set comparisons and do not depend on load. The `prove_abs_gap` build times depend on the `time_limit` of 1800 s, because the flux phases of step 1 run to the limit; no other limit was measured. RAVEN's `ftINIT` was not run. +Wall-clock figures come from jobs that shared nodes with other jobs and are approximate. The difference and objective columns are set comparisons and do not depend on the load. The build times with `prove_abs_gap` depend on the `time_limit` of 1800 s, because the flux phases of step 1 run to the limit, and no other limit was measured. `ftINIT` of RAVEN was not run. ## Reproduction -Requires Gurobi and a Human-GEM checkout. The numbers come from `det/prep.py` (preparation of the template with the essential tasks, about 2000 s), `det/build.py` (one build or repeats of `ftinit` on DLD1 with `big_m=100`, `mip_gap_abs=10` and `time_limit=1800`), `det/select_edit.py` (the 30 removed reactions of the template edit, seed 0), `det/ties.py` (tied-score blocks) and the summaries `det/analyse.py` and `det/det_extra.py`. The scripts are in the raven-docs repository under `scripts/parameter_tuning/init/`. The campaign takes about 100 core-hours. The commands run through the Slurm wrappers as in the README of that folder, with `WORK`, `TOOLBOX_ROOT`, `HUMANGEM_ROOT`, `PY`, `DET_SCRIPTS` and `PREP_DIR` set: +The reproduction requires Gurobi and a Human-GEM checkout. The numbers come from `det/prep.py` (preparation of the template with the essential tasks, about 2000 s), `det/build.py` (one build or repeats of `ftinit` on DLD1 with `big_m=100`, `mip_gap_abs=10` and `time_limit=1800`), `det/select_edit.py` (the 30 removed reactions of the template edit, seed 0), `det/ties.py` (tied-score blocks) and the summaries `det/analyse.py` and `det/det_extra.py`. The scripts are in the raven-docs repository under `scripts/parameter_tuning/init/`. The campaign takes about 100 core-hours. The commands run through the Slurm wrappers as in the README of that folder, with `WORK`, `TOOLBOX_ROOT`, `HUMANGEM_ROOT`, `PY`, `DET_SCRIPTS` and `PREP_DIR` set: ```bash jid=$(sbatch --parsable -J det-prep -o $WORK/logs/prep.log $DET_SCRIPTS/prep.sbatch) @@ -182,7 +180,7 @@ python analyse.py $WORK/out python det_extra.py $WORK/out $WORK/logs ``` -The tables are computed from the outputs and logs by `analyse.py` and `det_extra.py`; the degeneracy counts are read from `ties.log`. `ties.py` and `select_edit.py` need the base preparation `prep-base.pkl`, which is not archived and takes about 2000 s to build. `det/build.py` produced the build numbers; series `1+1` ignores `mip_gap_abs`, which the call below passes as `mip_gap_abs=10.0`. +The tables are computed from the outputs and logs by `analyse.py` and `det_extra.py`, and the degeneracy counts are read from `ties.log`. The scripts `ties.py` and `select_edit.py` need the base preparation `prep-base.pkl`, which is not archived and takes about 2000 s to build. The script `det/build.py` produced the build numbers. The call below passes `mip_gap_abs=10.0`, which series `1+1` ignores. ```python from raven_toolbox.init import ftinit @@ -201,6 +199,6 @@ m1 = ftinit(prep_edited, rxn_scores=rxn_scores, resolve_ties=True, prove_abs_gap print(len({r.id for r in m0.reactions} ^ {r.id for r in m1.reactions})) ``` -`rxn_scores` and `gene_scores` come from `score_reactions_from_genes` and `gene_scores_from_expression` (original reaction id → score). `prep` comes from `prep_init_model(template, tasks, ...)`; `prep_edited` is built the same way from the edited template. +The inputs `rxn_scores` and `gene_scores` come from `score_reactions_from_genes` and `gene_scores_from_expression` (original reaction id → score). The input `prep` comes from `prep_init_model(template, tasks, ...)`, and `prep_edited` is built the same way from the edited template. -Set `FTINIT_DEBUG=1` to log the status of each solve and each tie-break phase, which shows whether a build rests on a proven optimum or an unproven incumbent. The `UserWarning` messages name the adopted phases at the time limit and miss a dropped tie-break. +Setting `FTINIT_DEBUG=1` logs the status of each solve and each tie-break phase, which shows whether a build rests on a proven optimum or an unproven incumbent. The `UserWarning` messages name the adopted phases at the time limit and miss a dropped tie-break. diff --git a/docs/parameter-tuning/studies/homology-cutoff-calibration.md b/docs/parameter-tuning/studies/homology-cutoff-calibration.md index bf8b2ac..28cdbd3 100644 --- a/docs/parameter-tuning/studies/homology-cutoff-calibration.md +++ b/docs/parameter-tuning/studies/homology-cutoff-calibration.md @@ -1,12 +1,14 @@ # Homology cut-off calibration -Homology-based reconstruction transfers a template reaction to a new organism when a gene of the new organism matches a template gene in a BLAST or DIAMOND search and the match passes three filters: `min_identity`, `min_align_len` and `max_evalue` (`minIde`, `minLen` and `maxE` in RAVEN's `getModelFromHomology`). The study aligned the proteomes of four organisms against *S. cerevisiae* once, applied each filter setting to the saved hits, and scored the surviving matches against two references that state which genes are counterparts. It records one score per organism and setting, the agreement between BLAST and DIAMOND, and their run times. +Homology-based reconstruction transfers a template reaction to a new organism when a gene of the new organism matches a template gene in a BLAST or DIAMOND search and the match passes three filters: `min_identity`, `min_align_len` and `max_evalue` (`minIde`, `minLen` and `maxE` in `getModelFromHomology` of RAVEN). The study aligned the proteomes of four organisms against *S. cerevisiae* once, applied each filter setting to the saved hits, and scored the surviving matches against two references that state which genes are counterparts. It records one score per organism and setting, the agreement between BLAST and DIAMOND, and their run times. + +The scores show the largest effect for `min_identity`, a smaller effect for `min_align_len` and an effect of `max_evalue` from 1e-50 downward, with a flat score from 1e-4 to 1e-30. The value `max_evalue` 1e-30 is kept from RAVEN; the Chosen defaults table gives the basis of every default. ## Set-up ### Organisms and alignment -*S. cerevisiae* S288C is the template. The four targets span a range of relatedness to it, because a setting that suits a close relative need not suit a distant one: +*S. cerevisiae* S288C is the template. The four targets span a range of relatedness to it, because a setting that suits a close relative need not suit a distant one. | Code | Organism | Relation to yeast | |---|---|---| @@ -15,21 +17,20 @@ Homology-based reconstruction transfers a template reaction to a new organism wh | `ani` | *A. nidulans* | distant | | `eco` | *E. coli* | most distant | -Proteomes come from UniProt reference proteomes. Each protein is relabelled with its KEGG gene name through UniProt's KEGG cross-reference for the exact organism code; proteins without one are dropped. Both references can then be applied to the same matches. +The proteomes come from the UniProt reference proteomes. Each protein is relabelled with its KEGG gene name through the KEGG cross-reference of UniProt for the exact organism code, and proteins without one are dropped. Both references can then be applied to the same matches. -Each target was aligned against the template in both directions, once with `run_blast` (aligner E-value `1e-4`) and once with `run_diamond` (aligner E-value `1e-3`). Every filter setting was applied to the same saved hit table, so each score comes from one deterministic calculation. A hit is kept when `evalue <= max_evalue`, `align_len >= min_align_len` and `identity >= min_identity`; `get_model_from_homology` requires `evalue < max_evalue`. The kept hits are collected into a set of (template gene, target gene) pairs, so a pair found in both directions, or by more than one hit, counts as one match. A hit whose two genes are not both judged by the reference is counted as unjudgeable and left out of the score. The scoring does not reduce hits to best hits or build a model. +Each target was aligned against the template in both directions, once with `run_blast` (aligner E-value `1e-4`) and once with `run_diamond` (aligner E-value `1e-3`). Every filter setting was applied to the same saved hit table, so each score comes from one deterministic calculation. A hit is kept when `evalue <= max_evalue`, `align_len >= min_align_len` and `identity >= min_identity`. The function `get_model_from_homology` requires `evalue < max_evalue`. The kept hits are collected into a set of (template gene, target gene) pairs, so a pair found in both directions, or by more than one hit, counts as one match. A hit whose two genes are not both judged by the reference is counted as unjudgeable and left out of the score. The scoring does not reduce hits to best hits and builds no model. The sweeps vary one setting at a time from the RAVEN 2.11.3 values `max_evalue=1e-30`, `min_align_len=200` and `min_identity=40`. The identity sweep therefore ran at an alignment length of 200, the length sweep at 40 % identity, and the E-value sweep at 40 % identity and length 200. ### References -The **KEGG reference** counts a match between a template gene and a target gene as correct when the two genes share at least one KO in KEGG's `organism_gene_ko` table (release 118). The table lists only KOs that are linked to a reaction. A match is judged only when both genes have such a KO; other matches are left out of the score. - -The **OMA reference** counts a match as correct when the [OMA browser](https://omabrowser.org/) lists the two genes as a pair of orthologs. OMA identifies proteins by UniProt identifiers, which are mapped to KEGG gene names on both sides; OMA pairs that cannot be mapped are dropped. A match is judged when both genes have a KEGG gene name, so the OMA reference covers the whole mapped proteome. +- **KEGG reference.** A match between a template gene and a target gene is correct when the two genes share at least one KO in the `organism_gene_ko` table of KEGG (release 118). The table lists only KOs that are linked to a reaction. A match is judged only when both genes have such a KO, and other matches are left out of the score. +- **OMA reference.** A match is correct when the [OMA browser](https://omabrowser.org/) lists the two genes as a pair of orthologs. OMA identifies proteins by UniProt identifiers, which are mapped to KEGG gene names on both sides, and OMA pairs that cannot be mapped are dropped. A match is judged when both genes have a KEGG gene name, so the OMA reference covers the whole mapped proteome. ### Score -For each setting, **precision** is the number of correct matches divided by all judged matches, and **recall** is the number of correct matches divided by all pairs in the reference. The score is F0.5, the weighted harmonic mean `1.25·P·R / (0.25·P + R)` of precision P and recall R. With β = 0.5 it weights precision twice as heavily as recall. The weighting is the authors' choice: a missing reaction can be added by gap-filling, while a wrongly transferred reaction is hard to notice and harder to remove. The reproduction command passes `--beta 0.5`, which is also the default of the script, so the scores in the tables are F0.5. All score tables are against the KEGG reference; their base rows agree with each other. +For each setting, the precision is the number of correct matches divided by all judged matches, and the recall is the number of correct matches divided by all pairs in the reference. The score is F0.5, the weighted harmonic mean `1.25·P·R / (0.25·P + R)` of precision P and recall R. With β = 0.5 it weights precision twice as heavily as recall. The weighting is the choice of the authors, because a missing reaction can be added by gap-filling, while a wrongly transferred reaction is hard to notice and harder to remove. The reproduction command passes `--beta 0.5`, which is also the default of the script, so the scores in the tables are F0.5. All score tables are against the KEGG reference, and their base rows agree with each other. ## Results @@ -44,9 +45,9 @@ For each setting, **precision** is the number of correct matches divided by all | 45 | 0.907 | 0.813 | 0.735 | 0.520 | | 50 | 0.882 | 0.736 | 0.640 | 0.323 | -Against the KEGG reference the highest score was at 40 % for `kla` and `eco` and at 35 % for `yli` and `ani`. Against the OMA reference with BLAST it was at 45 % for the three fungi and at 35 % for *E. coli*; with DIAMOND it was at 45 % for the fungi and at 30 % for *E. coli*. The OMA scores are not tabulated. With BLAST the optima of both references lie between 35 and 45 %; the only optimum below 35 % is *E. coli* against OMA with DIAMOND. +Against the KEGG reference the highest score was at 40 % for `kla` and `eco` and at 35 % for `yli` and `ani`. Against the OMA reference the highest score with BLAST was at 45 % for the three fungi and at 35 % for *E. coli*. With DIAMOND it was at 45 % for the fungi and at 30 % for *E. coli*. The OMA scores are not tabulated. With BLAST the optima of both references lie between 35 and 45 %, and the only optimum below 35 % is *E. coli* against OMA with DIAMOND. -Scored with F1, which weights precision and recall equally, the optimum with BLAST moves to 30 % for `kla` and `eco` and 25 % for `yli` and `ani`; with DIAMOND it is 30 % for `kla`, `ani` and `eco` and 25 % for `yli` (values not tabulated). The best identity therefore depends on how the score weights the two kinds of error. +Scored with F1, which weights precision and recall equally, the optimum with BLAST moves to 30 % for `kla` and `eco` and to 25 % for `yli` and `ani`. With DIAMOND it is 30 % for `kla`, `ani` and `eco` and 25 % for `yli` (values not tabulated). The best identity therefore depends on how the score weights the two kinds of error. ### Alignment length @@ -58,7 +59,7 @@ Scored with F1, which weights precision and recall equally, the optimum with BLA | 200 (RAVEN 2.11.3) | 0.923 | 0.860 | 0.804 | 0.598 | | 300 | 0.885 | 0.816 | 0.764 | 0.571 | -Lengths 50 and 100 gave identical scores and 150 was at most 0.003 lower. The score fell between 150 and 200, by 0.010 to 0.020 from 100 to 200. Compared with 200, a length of 100 raised recall by 3.0 (`kla`), 3.8 (`yli`), 2.2 (`ani`) and 3.1 (`eco`) percentage points, which is 3.9, 6.5, 4.3 and 10.2 % more correct matches, and changed precision by at most 0.6 percentage points (`yli`, 0.967 to 0.961; precision and recall values not tabulated). No OMA scores were reported for alignment length. +The lengths 50 and 100 gave identical scores, and 150 was at most 0.003 lower. The score fell between 150 and 200, by 0.010 to 0.020 from 100 to 200. Compared with a length of 200, a length of 100 raised the recall by 3.0 (`kla`), 3.8 (`yli`), 2.2 (`ani`) and 3.1 (`eco`) percentage points, which is 3.9, 6.5, 4.3 and 10.2 % more correct matches. It changed the precision by at most 0.6 percentage points (`yli`, 0.967 to 0.961; precision and recall values not tabulated). No OMA scores were reported for the alignment length. ### E-value @@ -72,7 +73,7 @@ The hits were collected at an aligner E-value of `1e-4`, so `max_evalue=1e-4` ke | 1e-50 | 0.923 | 0.858 | 0.801 | 0.592 | | 1e-100 | 0.903 | 0.809 | 0.743 | 0.535 | -The score was the same at 1e-4, 1e-10 and 1e-30 for all four organisms. At 1e-50 it was unchanged for `kla` and lower by 0.002 (`yli`), 0.003 (`ani`) and 0.006 (`eco`), and at 1e-100 it was lower by 0.020 to 0.063. The F1 values (BLAST) are 0.855 for `kla` from 1e-4 to 1e-50 and 0.811 at 1e-100, and 0.663, 0.658 and 0.563 for `ani` at 1e-30, 1e-50 and 1e-100. A hit with at least 40 % identity over at least 200 residues can therefore have an E-value above 1e-50 in `yli`, `ani` and `eco` (inference from the score change; the hit tables were not inspected for this). No OMA scores were reported for the E-value. +The score was the same at 1e-4, 1e-10 and 1e-30 for all four organisms. At 1e-50 it was unchanged for `kla` and lower by 0.002 (`yli`), 0.003 (`ani`) and 0.006 (`eco`), and at 1e-100 it was lower by 0.020 to 0.063. The F1 values (BLAST) are 0.855 for `kla` from 1e-4 to 1e-50 and 0.811 at 1e-100, and 0.663, 0.658 and 0.563 for `ani` at 1e-30, 1e-50 and 1e-100. A hit with at least 40 % identity over at least 200 residues can therefore have an E-value above 1e-50 in `yli`, `ani` and `eco`. This is an inference from the score change, and the hit tables were not inspected for it. No OMA scores were reported for the E-value. ### BLAST and DIAMOND @@ -83,39 +84,45 @@ The score was the same at 1e-4, 1e-10 and 1e-30 for all four organisms. At 1e-50 | `ani` | 35 | 35 | 0.814 vs 0.811 | | `eco` | 40 | 40 | 0.598 vs 0.593 | -Against the KEGG reference with F0.5, the best identity was the same with both aligners for every organism, and the scores at that identity differed by 0.002 to 0.005. DIAMOND ran with `--more-sensitive`, and the speed ratios below apply to that mode. DIAMOND returned 32,192 hits on `kla` against 54,513 for BLAST. After filtering, the two aligners agreed on between 87.9 and 92.3 % of the surviving matches, measured as the Jaccard index of the (template gene, target gene) pairs that survive the filters at the base setting (identity 40, length 200, E-value 1e-30): 92.3 % (`kla`), 88.9 % (`yli`), 87.9 % (`ani`) and 91.8 % (`eco`). That the hits missing from DIAMOND are weak matches the filters would discard is an inference from these two results. BLAST took 140, 152, 200 and 44 s for `kla`, `yli`, `ani` and `eco`, and DIAMOND 12, 11, 14 and 7 s; DIAMOND was 11.7 times faster on `kla` and 14.3 times on `ani` (4 threads, AMD EPYC 9354, DIAMOND with `--more-sensitive`). +Against the KEGG reference with F0.5, the best identity was the same with both aligners for every organism, and the scores at that identity differed by 0.002 to 0.005. DIAMOND ran with `--more-sensitive`, and the speed ratios below apply to that mode. + +DIAMOND returned 32,192 hits on `kla` against 54,513 for BLAST. After filtering, the two aligners agreed on 87.9 to 92.3 % of the surviving matches. The agreement is the Jaccard index of the (template gene, target gene) pairs that survive the filters at the base setting (identity 40, length 200, E-value 1e-30): 92.3 % (`kla`), 88.9 % (`yli`), 87.9 % (`ani`) and 91.8 % (`eco`). That the hits missing from DIAMOND are weak matches that the filters would discard is an inference from these two results. + +BLAST took 140, 152, 200 and 44 s for `kla`, `yli`, `ani` and `eco`, and DIAMOND took 12, 11, 14 and 7 s. DIAMOND was 11.7 times faster on `kla` and 14.3 times on `ani` (4 threads, AMD EPYC 9354, DIAMOND with `--more-sensitive`). ### Aligner settings Two more settings control the search that produces the hits. Neither was part of the sweeps. -`evalue` is the aligner's own cut-off: `1e-4` for `run_blast` and `1e-3` for `run_diamond`. Each matches the MATLAB side: `getBlast` passes `-evalue 10e-5`, which equals `1e-4`, and `getDiamond` passes none, so DIAMOND's own default applies. On *H. polymorpha* against *S. cerevisiae*, loosening the BLAST cut-off from `1e-5` to `1e-4` added 5,200 raw hits (58,702 against 53,502), 8.9 % of the table. The tables differ row by row: 5,314 rows are only in the `1e-4` table and 114 only in the `1e-5` table. Of the 5,314 rows, 4,372 have an E-value above `1e-5` and so fail `max_evalue=1e-30`, and 942 have an E-value at or below `1e-5` (the smallest is 3.1e-60; 4 are below `1e-30`). In the five examples of the 942 rows that were inspected, the rows are other alignments (HSPs) of gene pairs that the cut-off change reports differently; the other rows were not inspected. Whether the 4 rows below `1e-30` and the 114 rows found only at `1e-5` also pass the identity and length filters, and so could change a model, was not checked. The remaining rows fail `max_evalue=1e-30`, so for them the aligner cut-off changes the size of the hit table and not the model. +The aligner setting `evalue` is the cut-off of the aligner itself: `1e-4` for `run_blast` and `1e-3` for `run_diamond`. Each matches the MATLAB side, because `getBlast` passes `-evalue 10e-5`, which equals `1e-4`, and `getDiamond` passes none, so DIAMOND uses its built-in default. On *H. polymorpha* against *S. cerevisiae*, loosening the BLAST cut-off from `1e-5` to `1e-4` added 5,200 raw hits (58,702 against 53,502), which is 8.9 % of the table. The tables differ row by row: 5,314 rows are only in the `1e-4` table and 114 only in the `1e-5` table. + +Of the 5,314 rows, 4,372 have an E-value above `1e-5` and fail `max_evalue=1e-30`. The other 942 rows have an E-value at or below `1e-5`, the smallest being 3.1e-60, and 4 of them are below `1e-30`. The remaining 938 of these rows fail `max_evalue=1e-30` as well, so for 5,310 rows the aligner cut-off changes the size of the hit table and leaves the model unchanged. In the five examples of the 942 rows that were inspected, the rows are other alignments (HSPs) of gene pairs that the two tables report differently. The other rows were not inspected. Whether the 4 rows below `1e-30` and the 114 rows found only at `1e-5` also pass the identity and length filters, and so could change a model, was not checked. -`threads` defaults to every CPU the process may run on. The first 500 sequences of the *H. polymorpha* proteome returned the same 2,746 hits at one thread and at four, with identical hit tables, in 20.4 s and 5.9 s (3.5 times faster). On a full proteome pair, single-threaded BLASTP took 205 s (*H. polymorpha* against *S. cerevisiae*) and 218 s (the reverse direction). The aligner cut-offs `1e-4` and `1e-5` took 111 s and 108.5 s. The measurements used the SGD translated ORFs (6,735 sequences) as template and Linux BLAST+ 2.17.0 on an AMD EPYC 9354 node. +`threads` defaults to every CPU that the process may run on. The first 500 sequences of the *H. polymorpha* proteome returned the same 2,746 hits at one thread and at four, with identical hit tables, in 20.4 s and 5.9 s (3.5 times faster). On a full proteome pair, single-threaded BLASTP took 205 s (*H. polymorpha* against *S. cerevisiae*) and 218 s (the reverse direction). The aligner cut-offs `1e-4` and `1e-5` took 111 s and 108.5 s. The measurements used the translated ORFs of SGD (6,735 sequences) as template and Linux BLAST+ 2.17.0 on an AMD EPYC 9354 node. ## Chosen defaults | Setting | RAVEN 2.11.3 | Default (raven-toolbox and RAVEN 3) | Basis | |---|---|---|---| -| `min_identity` (`minIde`) | 40 | 40 | Over the sweep from 25 to 50 % the score changed by 0.062 (`kla`) to 0.275 (`eco`), against 0.047 to 0.055 over alignment lengths 50 to 300 and 0.020 to 0.063 over E-values 1e-4 to 1e-100; with BLAST the optima of both references lie between 35 and 45 %. | -| `min_align_len` (`minLen`) | 200 | 100 | 100 scored 0.010 to 0.020 higher than 200 against KEGG; 50 and 100 scored the same, and 100 is the stricter of the two. | -| `max_evalue` (`maxE`) | 1e-30 | 1e-30 | No change in score from 1e-4 to 1e-30 in any organism; 1e-50 lowered it by up to 0.006; kept at the RAVEN value. | +| `min_identity` (`minIde`) | 40 | 40 | Over the sweep from 25 to 50 % the score changed by 0.062 (`kla`) to 0.275 (`eco`), against 0.047 to 0.055 over alignment lengths 50 to 300 and 0.020 to 0.063 over E-values 1e-4 to 1e-100. With BLAST the optima of both references lie between 35 and 45 % | +| `min_align_len` (`minLen`) | 200 | 100 | The length 100 scored 0.010 to 0.020 higher than 200 against KEGG. The lengths 50 and 100 scored the same, and 100 is the stricter of the two | +| `max_evalue` (`maxE`) | 1e-30 | 1e-30 | No change in score from 1e-4 to 1e-30 in any organism, and 1e-50 lowered it by up to 0.006. The RAVEN value is kept | -The study supports one `min_identity` for BLAST and DIAMOND against the KEGG reference with F0.5, since both reached the same best identity for every organism. Against OMA the optimum for *E. coli* is 35 % with BLAST and 30 % with DIAMOND, and with F1 the optimum for `ani` is 25 % with BLAST and 30 % with DIAMOND. DIAMOND scores for alignment length and E-value are not tabulated here. No DIAMOND scores are tabulated for the length and E-value sweeps, whose base scores equal those of BLAST. +The study supports one `min_identity` for BLAST and DIAMOND against the KEGG reference with F0.5, because both reached the same best identity for every organism. Against OMA the optimum for *E. coli* is 35 % with BLAST and 30 % with DIAMOND, and with F1 the optimum for `ani` is 25 % with BLAST and 30 % with DIAMOND. The base rows of the alignment-length and E-value sweeps are the base row of the identity sweep (BLAST), and DIAMOND scores for those two sweeps are not tabulated. ## Limits -- KEGG's KO annotations are partly derived from sequence similarity, so the KEGG reference alone would partly measure agreement with the method under test. OMA infers orthology independently of BLAST, and its optima lie in the same range of 35 to 45 % identity. OMA scores were reported for identity only. +- The KO annotations of KEGG are partly derived from sequence similarity, so the KEGG reference alone would partly measure agreement with the method under test. OMA infers orthology independently of BLAST, and its optima lie in the same range of 35 to 45 % identity. OMA scores were reported for identity only. - The two references are not comparable in absolute terms. The KEGG reference judges only genes with a reaction-linked KO, while the OMA reference judges the whole mapped proteome and counts every match that OMA does not list as an ortholog pair as wrong. Paralogs probably lower the OMA score for this reason (inference). Only the shape of each column is comparable between the references. -- The OMA reference for *Y. lipolytica* is incomplete: 38 % of its OMA pairs could not be mapped to KEGG gene names. +- The OMA reference for *Y. lipolytica* is incomplete, because 38 % of its OMA pairs could not be mapped to KEGG gene names. - One template: everything was measured outward from *S. cerevisiae*. -- One setting was varied at a time from length 200 and identity 40 %. The identity optimum and the absence of an E-value effect up to 1e-30 were measured at length 200, and were not measured at the default length of 100. -- The scores count gene matches, not reactions in a finished model. A reaction transfers when any one of its genes matches, so the effect on a finished model is probably smaller (inference, not measured). -- Each score comes from one alignment per organism. Scoring a saved hit table is deterministic. DIAMOND returned 32,192 hits for `kla`. The alignment was run once, so the run-to-run spread of the hit table was not measured. +- One setting was varied at a time from length 200 and identity 40 %. The identity optimum and the absence of an E-value effect up to 1e-30 were measured at length 200 and not at the default length of 100. +- The scores count gene matches. A reaction transfers when any one of its genes matches, so the effect on a finished model is probably smaller (inference, not measured). +- Each score comes from one alignment per organism. Scoring a saved hit table is deterministic, and DIAMOND returned 32,192 hits for `kla`. The alignment was run once, so the run-to-run spread of the hit table was not measured. ## Curated-GEM reference -Scoring against curated genome-scale metabolic models (GEMs) of non-model fungi such as [hanpo-GEM](https://github.com/SysBioChalmers/hanpo-GEM) and [rhto-GEM](https://github.com/SysBioChalmers/rhto-GEM) is not part of the 2026-11-01 measurement, and the study reports no numbers for it. Both models were built with RAVEN's homology-based reconstruction, so a score against them would measure agreement with their own build settings and not correctness (inference). +Scoring against curated genome-scale metabolic models (GEMs) of non-model fungi, such as [hanpo-GEM](https://github.com/SysBioChalmers/hanpo-GEM) and [rhto-GEM](https://github.com/SysBioChalmers/rhto-GEM), is not part of the measurement of 2026-11-01, and the study reports no numbers for it. Both models were built with the homology-based reconstruction of RAVEN, so a score against them would measure agreement with their own build settings, which is no test of correctness (inference). ## Reproducing @@ -131,8 +138,8 @@ python homology_cutoff_kegg.py --out work score --beta 0.5 \ python agreement.py --work work --gene-ko work/refs/kegg118_organism_gene_ko_5org.tsv.gz --out agreement.tsv ``` -`fetch_references.py` downloads the release-118 gene→KO rows (from the `kegg118_core.tar.gz` release asset of raven-toolbox), the current KEGG REST tables and the SGD translated ORFs. `align --aligner diamond` and `score --aligner diamond` repeat the steps with DIAMOND, `score --reference oma` scores against the OMA reference and `score --beta 1` gives F1. The driver also scores identities of 20 and 60 %, which are not tabulated above. `agreement.py` computes the BLAST and DIAMOND agreement, `make_tables.py` collects the results, and `aligner_settings.py` runs the *H. polymorpha* measurements (thread test, aligner E-value `1e-4` against `1e-5`, single-thread full directions). `run_cached.py` wraps the driver, whose path is given in the environment variable `HOMOLOGY_SCRIPT`, and replaces its UniProt download with a disk cache, which allows scoring against OMA offline; it adds the `prefetch` subcommand and passes every other subcommand to the driver. The alignment output is saved, so re-scoring under a different weighting or against a different reference takes about two minutes. +`fetch_references.py` downloads the release-118 gene→KO rows (from the `kegg118_core.tar.gz` release asset of raven-toolbox), the current KEGG REST tables and the SGD translated ORFs. The commands `align --aligner diamond` and `score --aligner diamond` repeat the steps with DIAMOND, `score --reference oma` scores against the OMA reference, and `score --beta 1` gives F1. The driver also scores identities of 20 and 60 %, which are not tabulated above. The script `agreement.py` computes the BLAST and DIAMOND agreement, `make_tables.py` collects the results, and `aligner_settings.py` runs the *H. polymorpha* measurements (thread test, aligner E-value `1e-4` against `1e-5`, single-thread full directions). The script `run_cached.py` wraps the driver, whose path is given in the environment variable `HOMOLOGY_SCRIPT`, and replaces its UniProt download with a disk cache, which allows scoring against OMA offline. It adds the `prefetch` subcommand and passes every other subcommand to the driver. The alignment output is saved, so scoring again under another weighting or against another reference takes about two minutes. -The UniProt downloads, the OMA pairs, the KEGG tables and the SGD ORFs are in the archived input bundle `homology-inputs.tar.zst` (18.9 MB), and the BLAST and DIAMOND hit tables in `homology-hits.tar.zst` (4.0 MB). +The UniProt downloads, the OMA pairs, the KEGG tables and the SGD ORFs are in the archived input bundle `homology-inputs.tar.zst` (18.9 MB), and the BLAST and DIAMOND hit tables are in `homology-hits.tar.zst` (4.0 MB). -The scores come from the run of 2026-11-01 with raven-toolbox `94f3940`, BLAST+ 2.17.0, DIAMOND 2.1.17, the KEGG release-118 table and UniProt release 2026_03, with BLAST and DIAMOND at 4 threads on an AMD EPYC 9354 node. The *H. polymorpha* template is the SGD ORF file, and the thread test uses the first 500 sequences of `hanpo.faa`. The curated-GEM reference is not part of this run. +The scores come from the run of 2026-11-01 with raven-toolbox `94f3940`, BLAST+ 2.17.0, DIAMOND 2.1.17, the KEGG release-118 table and UniProt release 2026_03, with BLAST and DIAMOND at 4 threads on an AMD EPYC 9354 node. The *H. polymorpha* template is the SGD ORF file, and the thread test uses the first 500 sequences of `hanpo.faa`. diff --git a/docs/parameter-tuning/studies/humangem-validation.md b/docs/parameter-tuning/studies/humangem-validation.md index b88a6ec..d1c60cc 100644 --- a/docs/parameter-tuning/studies/humangem-validation.md +++ b/docs/parameter-tuning/studies/humangem-validation.md @@ -1,25 +1,38 @@ # Human-GEM cell-type model validation against RAVEN -raven-toolbox's ftINIT was run on Human-GEM with the RNA sequencing (RNA-seq) data of five cell lines from the Hart2015 dataset (DLD1, GBM, HCT116, HELA, RPE1), and each extracted cell-type model was compared with the model that MATLAB RAVEN's `ftINIT` extracted from the same input. The comparison records the reactions each model contains, the reactions the two models share and the reactions only one of them contains, and the time each step took. The question is whether raven-toolbox selects the same reactions as RAVEN. +The study asks whether the ftINIT of raven-toolbox selects the same reactions as the `ftINIT` of RAVEN. Both tools extracted a cell-type model from Human-GEM with the RNA sequencing (RNA-seq) data of five cell lines of the Hart2015 dataset (DLD1, GBM, HCT116, HELA, RPE1), and the models were compared by their reaction sets, together with the time of each step. + +With the task layer, the raven-toolbox models share a Jaccard index of 0.981 to 0.988 with the RAVEN models. Without the task layer the index is 0.964 to 0.982. Each cell line, variant and `big_m` value ran once, and the RAVEN result is task-constrained, so the first range is the like-for-like comparison. ## Set-up -Software: Human-GEM commit `38567ab8` (v2.1.1 plus 15 commits), RAVEN commit `2d0630e3` on MATLAB 2024b, raven-toolbox commit `94f3940`, and Gurobi for both tools, with 4 cores per run. The Python runs load the Gurobi library 13.0.3 from the gurobipy wheel (most likely; the logs print no version), and the MATLAB side links to the Gurobi module library 13.0.1 (from the linkage of the MATLAB interface; MATLAB prints no version). +### Software + +The software was Human-GEM commit `38567ab8` (v2.1.1 plus 15 commits), RAVEN commit `2d0630e3` on MATLAB 2024b, raven-toolbox commit `94f3940`, and Gurobi for both tools, with 4 cores per run. The Python runs load the Gurobi library 13.0.3 from the gurobipy wheel (most likely, because the logs print no version). The MATLAB side links to the library 13.0.1 of the Gurobi module, as the linkage of the MATLAB interface shows; MATLAB prints no version. + +### Template preparation + +Template preparation runs once per template model, before any cell line. -Template preparation is the step that runs once per template model, before any cell line. For RAVEN, the Human-GEM function `prepHumanModelForftINIT` removed the drug reactions, the amino-acid triplet reactions and the duplicate complex I reaction from Human-GEM, set the spontaneous and custom reaction lists, and prepared the model with the essential tasks of `metabolicTasks_Essential.txt`. RAVEN's prepared model, the model from which RAVEN extracts, has 9730 reactions. For raven-toolbox, `prep_init_model` ran on the same cleaned Human-GEM (11265 reactions). Every reaction in every raven-toolbox result is in RAVEN's prepared model, so the reaction identifiers of the two results are compared directly. +- **RAVEN.** The Human-GEM function `prepHumanModelForftINIT` removed the drug reactions, the amino-acid triplet reactions and the duplicate complex I reaction from Human-GEM, set the lists of spontaneous and custom reactions, and prepared the model with the essential tasks of `metabolicTasks_Essential.txt`. The prepared model, from which RAVEN extracts, has 9730 reactions. +- **raven-toolbox.** `prep_init_model` ran on the same cleaned Human-GEM (11265 reactions). Every reaction in every raven-toolbox result is in the prepared model of RAVEN, so the reaction identifiers of the two results are compared directly. -Scoring: gene scores are `5·ln(level/1)` of the transcripts per million (TPM) values in the Hart2015 file, clipped to [−5, 10], and reactions take their scores from the gene rules; a reaction without genes scores −2. raven-toolbox computes this with `gene_scores_from_expression` and `score_reactions_from_genes`, RAVEN with `transcrData` and `threshold=1`. +### Scoring and extraction -Extraction: both tools ran the series `1+1`, two staged mixed-integer linear program (MILP) steps. RAVEN ran `ftINIT` with its default settings. raven-toolbox ran `raven_toolbox.init.ftinit` with series `1+1`, `time_limit=600` and the default `big_m=1000`. A named series runs its own schedule of MILP runs in each step, so `mip_gap` and `mip_gap_abs` have no effect on these runs. +Gene scores are `5·ln(level/1)` of the transcripts per million (TPM) values in the Hart2015 file, clipped to [−5, 10]. Reactions take their scores from the gene rules, and a reaction without genes scores −2. raven-toolbox computes the scores with `gene_scores_from_expression` and `score_reactions_from_genes`, and RAVEN with `transcrData` and `threshold=1`. + +Both tools ran the series `1+1`, two staged mixed-integer linear program (MILP) steps. RAVEN ran `ftINIT` with its default settings. raven-toolbox ran `raven_toolbox.init.ftinit` with series `1+1`, `time_limit=600` and the default `big_m=1000`. A named series runs its own schedule of MILP runs in each step, so `mip_gap` and `mip_gap_abs` have no effect on these runs. raven-toolbox ran in two variants: -- **No-task**: the template is prepared without a task list, and the extraction uses the expression scores alone. -- **Task-constrained**: the template is prepared with the essential metabolic tasks, the reactions that a task needs are kept whatever their score, and `ftinit` adds reactions back until every task is feasible (its default `fill_gaps=True`). +- **No-task.** The template is prepared without a task list, and the extraction uses the expression scores alone. +- **Task-constrained.** The template is prepared with the essential metabolic tasks, the reactions that a task needs are kept whatever their score, and `ftinit` adds reactions back until every task is feasible (its default `fill_gaps=True`). The RAVEN result is task-constrained in this sense. A third set of no-task runs used `big_m=100`. -Measurements: the result of a run is its set of reaction identifiers. For each cell line the comparison gives the number of reactions in both sets (shared), in one set only, and the Jaccard index, the shared count divided by the size of the union. The comparison covers reactions only; genes, essential genes and task performance of the extracted models were not compared. Each cell line, variant and `big_m` value ran once. Times are wall-clock times of single runs on shared nodes. +### Measurements + +The result of a run is its set of reaction identifiers. For each cell line the comparison gives the number of reactions in both sets (shared), the number in one set only, and the Jaccard index, which is the shared count divided by the size of the union. The comparison covers reactions only. Genes, essential genes and the task performance of the extracted models were not compared. Each cell line, variant and `big_m` value ran once, and the times are wall-clock times of single runs on shared nodes. ## Results @@ -28,7 +41,7 @@ Measurements: the result of a run is its set of reaction identifiers. For each c | Step | Time | |------|-----:| | RAVEN `prepHumanModelForftINIT` (4-worker parallel pool, once per template) | 4320 s | -| raven-toolbox `prep_init_model`, no-task (cobrapy's default process count for flux variability analysis, once per template) | 1556 s | +| raven-toolbox `prep_init_model`, no-task (default process count of cobrapy for flux variability analysis, once per template) | 1556 s | | raven-toolbox `prep_init_model`, task-constrained (`cobra.Configuration().processes = 1`, once per template) | 1695 s | | RAVEN `ftINIT`, per cell line | 40 to 91 s | | raven-toolbox `ftinit`, no-task, per cell line | 81 to 91 s | @@ -76,46 +89,51 @@ Measurements: the result of a run is its set of reaction identifiers. For each c The no-task `ftinit` runs took 83 to 122 s at `big_m=100` and 81 to 91 s at `big_m=1000`. -## Agreement with the task layer +## Interpretation + +### Agreement with the task layer -The task-constrained variant shares Jaccard 0.981 to 0.988 with RAVEN across the five cell lines. Its reaction counts are within 0.2 % of RAVEN's, and the differences run in both directions in similar numbers: 46 to 70 reactions only in the raven-toolbox model and 43 to 71 only in the RAVEN model, out of ~7300. +The task-constrained variant agrees with RAVEN in every cell line. Its reaction counts are within 0.2 % of the counts of RAVEN. The differences run in both directions in similar numbers: 46 to 70 reactions only in the raven-toolbox model and 43 to 71 only in the RAVEN model, out of about 7300. -Neither tool was run twice on the same input, so the comparison does not separate alternate optima of the MILP from a difference between the two implementations. For scale, two raven-toolbox runs that differed only in the seed differed by 110 reactions in [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](ftinit-determinism.md). That measurement used a different set-up (Human-GEM v2.0.0-38, DLD1, `big_m=100`, an escalation schedule), so it indicates the order of magnitude and does not bound the spread here. +Neither tool was run twice on the same input, so the comparison does not separate alternate optima of the MILP from a difference between the two implementations. For scale, two raven-toolbox runs that differed only in the seed differed by 110 reactions in [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](ftinit-determinism.md). That measurement used another set-up (Human-GEM v2.0.0-38, DLD1, `big_m=100`, an escalation schedule), so it indicates the order of magnitude and does not bound the spread of this set-up. -## Agreement without the task layer +### Agreement without the task layer -The no-task variant shares Jaccard 0.964 to 0.982 with RAVEN. It keeps fewer reactions than RAVEN in every cell line: 0.3 to 0.5 % fewer for four cell lines and 1.6 % fewer for RPE1. The reactions only in the RAVEN model (76 to 186) outnumber those only in the raven-toolbox model (53 to 76). +The no-task variant keeps fewer reactions than RAVEN in every cell line: 0.3 to 0.5 % fewer for four cell lines and 1.6 % fewer for RPE1. The reactions only in the RAVEN model (76 to 186) outnumber those only in the raven-toolbox model (53 to 76). -The no-task variant omits the task layer that the RAVEN result includes. With the task layer, the reactions only in the RAVEN model fall to 43 to 71 per cell line. That the task layer accounts for most of the surplus is an inference from these counts; the reactions in the surplus were not checked against the task-essential reactions. +The no-task variant omits the task layer that the RAVEN result includes. With the task layer, the reactions only in the RAVEN model fall to 43 to 71 per cell line. That the task layer accounts for most of the surplus is an inference from these counts, and the reactions in the surplus were not checked against the task-essential reactions. -## Effect of `big_m` +### Effect of `big_m` -Lowering the cap on the per-reaction big-M from 1000 to 100 changed the Jaccard index against RAVEN by at most 0.003 in each cell line, one run per cell line and value. In every cell line the two raven-toolbox results shared a higher Jaccard index with each other (0.981 to 0.985) than either shared with RAVEN (0.964 to 0.982); for RPE1 the values were 0.984 between the two results against 0.964 and 0.967 with RAVEN. The cap therefore changed the reaction set (1 minus the Jaccard index between the two results, 0.015 to 0.019) by more than it changed the agreement with RAVEN (at most 0.003). At this resolution the two values agree with RAVEN equally well. +Lowering the cap on the per-reaction big-M from 1000 to 100 changed the Jaccard index against RAVEN by at most 0.003 in each cell line, with one run per cell line and value. In every cell line the two raven-toolbox results shared a higher Jaccard index with each other (0.981 to 0.985) than either shared with RAVEN (0.964 to 0.982). For RPE1 the values were 0.984 between the two results and 0.964 and 0.967 with RAVEN. The cap therefore changed the reaction set (1 minus the Jaccard index between the two results, 0.015 to 0.019) by more than it changed the agreement with RAVEN (at most 0.003). At this resolution the two values agree equally well with RAVEN. -## Run time +### Comparison of run times -After the template preparation, a task-constrained cell-type model took 146 to 381 s in raven-toolbox and 40 to 91 s in RAVEN; a no-task model took 81 to 91 s. The template preparation took 1556 s (26 min) and 1695 s (28 min) in raven-toolbox and 4320 s (72 min) in RAVEN. The preparations ran with different parallelism (see the run-time table), so the times compare whole runs and not the code alone. The [Reference run](../../guide/init.md#reference-run) of the guide Context-specific models with ftINIT reports a preparation of 47 min in MATLAB and 37 min in Python and an extraction of 797 s in MATLAB and 124 s in Python. That run used RAVEN's `prepINITModel` directly on Human-GEM `6812be2a` (12748 reactions) without the cleaning step, and series `1+0`; these four differences from this validation (preparation function, Human-GEM commit, cleaning step, series) were not separated, so the times are not comparable. +After the template preparation, a task-constrained cell-type model took 146 to 381 s in raven-toolbox and 40 to 91 s in RAVEN, and a no-task model took 81 to 91 s. The template preparation took 1556 s (26 min) and 1695 s (28 min) in raven-toolbox and 4320 s (72 min) in RAVEN. The preparations ran with different parallelism (see the run-time table), so the times compare whole runs, including the parallelism. + +The [Reference run](../../guide/init.md#reference-run) of the guide Context-specific models with ftINIT reports a preparation of 47 min in MATLAB and 37 min in Python and an extraction of 797 s in MATLAB and 124 s in Python. That run used `prepINITModel` of RAVEN directly on Human-GEM `6812be2a` (12748 reactions) without the cleaning step, and series `1+0`. These four differences from this validation (preparation function, Human-GEM commit, cleaning step and series) were not separated, so the times are not comparable. + +## Constraint construction + +Each steady-state balance of the extraction MILP is built from a flat list of terms and summed with `optlang.symbolics.add`, and the gap-fill uses the same construction. The `sum()` of Python simplifies the growing expression again at each term, and the balances of hub metabolites such as ATP, H⁺ and H₂O are long sums. ## Numerical settings shared with RAVEN raven-toolbox builds the extraction MILP with two numerical settings of RAVEN: -- Per-reaction big-M. The on-off constraints of a scored reaction use `max(min(ub, big_m), min(-lb, big_m), force_on)` as big-M, an infinite bound counting as `big_m`. The default cap `big_m=1000` is RAVEN's `bigMCap` in `ftINITInternalAlg`. The cap limits the flux a reaction can carry while it counts as on. -- Stoichiometric rescaling. `prep_init_model` applies RAVEN's `rescaleModelForINIT` to the merged model: each coefficient of a reaction is capped at 25 times the smallest coefficient of that reaction, and the reaction is then scaled to a mean absolute coefficient of 1. - -Which combinations of `big_m` and rescaling are feasible for the staged pipeline was measured on HCT116 in [ftINIT parameter calibration and input robustness](init-param-calibration.md): with rescaling, `big_m=25` was infeasible and `big_m=100` feasible; without rescaling, `big_m=1000` was feasible (157 s against 83 s with rescaling, Jaccard 0.969 between the two results) and `big_m=100` infeasible. +- **Per-reaction big-M.** The on-off constraints of a scored reaction use `max(min(ub, big_m), min(-lb, big_m), force_on)` as big-M, and an infinite bound counts as `big_m`. The default cap `big_m=1000` is `bigMCap` of RAVEN in `ftINITInternalAlg`. The cap limits the flux that a reaction can carry while it counts as on. +- **Stoichiometric rescaling.** `prep_init_model` applies `rescaleModelForINIT` of RAVEN to the merged model. Each coefficient of a reaction is capped at 25 times the smallest coefficient of that reaction, and the reaction is then scaled to a mean absolute coefficient of 1. -## Constraint construction +[ftINIT parameter calibration and input robustness](init-param-calibration.md) measured on HCT116 which combinations of `big_m` and rescaling are feasible for the staged pipeline. With rescaling, `big_m=25` was infeasible and `big_m=100` feasible. Without rescaling, `big_m=1000` was feasible (157 s against 83 s with rescaling, Jaccard index 0.969 between the two results) and `big_m=100` infeasible. -Each steady-state balance of the extraction MILP is built from a flat list of terms and summed with `optlang.symbolics.add`; the same construction is used in the gap-fill. Python's `sum()` re-simplifies the growing expression at each term, and the balances of hub metabolites such as ATP, H⁺ and H₂O are long sums. ## Use of the result -The agreement was measured with the code defaults `big_m=1000` (`_BIG_M` in `raven_toolbox/init/ftinit.py`, listed in [Tuned parameter defaults](../../tuned-parameters.md)) and rescaling on (`scale=True` in `prep_init_model`, with `max_stoich_diff=25` in `rescale_for_init`, both in `raven_toolbox/init/prep.py`). A raven-toolbox model agreed most closely with a RAVEN `ftINIT` model when `prep_init_model` received the same task list as RAVEN's preparation. +The agreement was measured with the code defaults `big_m=1000` (`_BIG_M` in `raven_toolbox/init/ftinit.py`, listed in [Tuned parameter defaults](../../tuned-parameters.md)) and rescaling on (`scale=True` in `prep_init_model`, with `max_stoich_diff=25` in `rescale_for_init`, both in `raven_toolbox/init/prep.py`). A raven-toolbox model agreed most closely with a RAVEN `ftINIT` model when `prep_init_model` received the same task list as the preparation of RAVEN. ## Limits -Each cell line, variant and `big_m` value ran once, without seed repeats, so the run-to-run spread of either tool was not measured and Jaccard differences of a few tenths of a percent are not resolved. The comparison covers reaction sets on one template and one dataset; gene content, essential genes and task performance were not compared. The timings come from shared nodes and are approximate. +Each cell line, variant and `big_m` value ran once, without repeats of the seed, so the run-to-run spread of either tool was not measured and Jaccard differences of a few tenths of a percent are not resolved. The comparison covers reaction sets on one template and one dataset. Gene content, essential genes and task performance were not compared. The timings come from shared nodes and are approximate. ## Reproduction diff --git a/docs/parameter-tuning/studies/init-param-calibration.md b/docs/parameter-tuning/studies/init-param-calibration.md index b04f1bc..29ee807 100644 --- a/docs/parameter-tuning/studies/init-param-calibration.md +++ b/docs/parameter-tuning/studies/init-param-calibration.md @@ -1,37 +1,45 @@ # ftINIT parameter calibration and input robustness -This study runs raven-toolbox's ftINIT on Human-GEM with the HCT116 expression data of the Hart2015 dataset and varies one or two settings per run series; the scaling runs and the full-pipeline runs vary scaling and `big_m` together. The calibration part varies the mixed-integer linear program (MILP) settings `mip_gap`, `big_m` and `force_on` and the stoichiometric scaling of the prepared model, and records the run time, the MILP objective and which reactions are kept. The robustness part degrades the expression input by dropout, noise and downsampling, extracts a model with the metabolic-task layer on, and records the model size, the essential tasks the model passes and its overlap with the model from clean data. Overlap is measured as the Jaccard index of two kept-reaction sets: the reactions in both divided by the reactions in either. +The study runs ftINIT of raven-toolbox on Human-GEM with the HCT116 expression data of the Hart2015 dataset and varies one or two settings per series of runs. The scaling runs and the full-pipeline runs vary the scaling and `big_m` together. The calibration part varies the settings `mip_gap`, `big_m` and `force_on` of the mixed-integer linear program (MILP) and the stoichiometric scaling of the prepared model, and records the run time, the MILP objective and the reactions that are kept. The robustness part degrades the expression input by dropout, noise and downsampling, extracts a model with the metabolic-task layer on, and records the size of the model, the essential tasks that it passes and its overlap with the model from clean data. -Steps are counted from 1 on the pages of this study; the run logs count from 0, so the logged `step 1` is the second step. +The overlap is the Jaccard index of two kept-reaction sets: the reactions in both divided by the reactions in either. Steps are counted from 1 on the pages of this study. The run logs count from 0, so the logged `step 1` is the second step. + +The main findings are: + +- In the first-step solve, every `big_m` from 10 to 1000 with every `max_stoich_diff` and with the unscaled model was feasible, and the time stayed between 23 and 30 s. +- In the full pipeline, `big_m=1000` was feasible with and without scaling (83 s and 157 s), `big_m=100` was feasible with scaling and infeasible without, and `big_m=25` was infeasible with scaling. +- Noise changed the model less than sparse input. At 70 % dropout no tested parameter recovered the expression evidence that dropout removes, which is an inference from four lever settings. +- In the first-step solve, `mip_gap` up to 0.01 left the Jaccard index at 0.99 or above and fell to 0.9649 and 0.9663 at 0.03 and 0.1, and `force_on` from 0.02 to 0.5 left the index at 0.987 or more. In the full pipeline `mip_gap` had no effect. +- The set-up has no repeats of the seed, so the seed-to-seed spread of the set-up is unknown. ## Set-up ### Software and inputs -The calibration runs used Human-GEM commit `38567ab8` (v2.1.1 plus 15 commits), the HCT116 column of the Hart2015 RNA sequencing file, raven-toolbox commit `94f3940` and Gurobi on 4 cores. Gurobi library 13.0.3 from the gurobipy wheel (most likely); the logs print no version. The template is the reference model of the [Human-GEM cell-type model validation against RAVEN](humangem-validation.md): Human-GEM without the drug reactions, the amino-acid triplet reactions and the duplicate complex I reaction, 11265 reactions. `prep_init_model` prepared it without a task list. Its scaled merged model, the model that the MILP receives, has 6657 reactions; the preparation took 26 min and is excluded from every time in the tables. +The calibration runs used Human-GEM commit `38567ab8` (v2.1.1 plus 15 commits), the HCT116 column of the Hart2015 RNA sequencing file, raven-toolbox commit `94f3940` and Gurobi on 4 cores. The Gurobi library is 13.0.3 from the gurobipy wheel (most likely; the logs print no version). The template is the reference model of the [Human-GEM cell-type model validation against RAVEN](humangem-validation.md): Human-GEM without the drug reactions, the amino-acid triplet reactions and the duplicate complex I reaction, with 11265 reactions. `prep_init_model` prepared it without a task list. Its scaled merged model, which is the model that the MILP receives, has 6657 reactions. The preparation took 26 min and is excluded from every time in the tables. The robustness runs used Human-GEM at a commit that is not recorded, the same HCT116 data, a template prepared with the essential metabolic tasks, and `big_m=100`. The two parts therefore have no common baseline, and their reaction counts and Jaccard indices cannot be compared with each other. -Gene scores are `5·ln(level/1)` of each expression level, clipped to [−5, 10] (see [Gene-scoring constants](#gene-scoring-constants)), and reaction scores follow from the gene rules (`score_reactions_from_genes`, default `no_gene_score=-2`). +Gene scores are `5·ln(level/1)` of each expression level, clipped to [−5, 10] (see [Gene-scoring constants](#gene-scoring-constants)). Reaction scores follow from the gene rules (`score_reactions_from_genes`, default `no_gene_score=-2`). ### Run types | Run type | What runs | Script and sweep | |---|---|---| | First-step solve | One `run_ftinit` call (one MILP, no run schedule) for the first step of series `1+1` on the scaled merged model, with a 900 s limit | `bigm/bm.py step0` | -| Scaling run | The same first-step solve on a model prepared without scaling, which `rescale_for_init` then rescales with the given `max_stoich_diff` (or leaves unscaled), with a 600 s limit and `mip_gap=0.001`. | `bigm/bm.py scale` | -| Full pipeline | One call of `ftinit` with series `1+1` and `time_limit=600` on the prepared model without tasks: both MILP steps, each with its own schedule of runs. | `bigm/bm.py full` | -| Robustness run | One call of `ftinit` with series `1+1` on the template prepared with tasks, including the gap-fill that adds reactions until every task is feasible, followed by `check_tasks` on the extracted model. | `scripts/analyze_init_robustness.py` (`MIP_GAP` 0.02, `TIME_LIMIT` 120), phases `gradient` and `levers` (not recoverable) | +| Scaling run | The same first-step solve on a model prepared without scaling, which `rescale_for_init` then rescales with the given `max_stoich_diff` or leaves unscaled, with a 600 s limit and `mip_gap=0.001` | `bigm/bm.py scale` | +| Full pipeline | One call of `ftinit` with series `1+1` and `time_limit=600` on the prepared model without tasks: both MILP steps, each with its own schedule of runs | `bigm/bm.py full` | +| Robustness run | One call of `ftinit` with series `1+1` on the template prepared with tasks, including the gap-fill that adds reactions until every task is feasible, followed by `check_tasks` on the extracted model | `scripts/analyze_init_robustness.py` (`MIP_GAP` 0.02, `TIME_LIMIT` 120), phases `gradient` and `levers` (not recoverable) | -A named series such as `1+1` ignores `mip_gap` and `mip_gap_abs`: every step runs its own schedule of MILP runs, with limits of 120 s, 5000 s and 5 s. `time_limit` lowers the limit of each scheduled run, so at `time_limit=600` it shortens the 5000 s runs only. `mip_gap` therefore acts on the first-step solve and not on the full pipeline. +A named series such as `1+1` ignores `mip_gap` and `mip_gap_abs`. Every step runs its own schedule of MILP runs, with limits of 120 s, 5000 s and 5 s. `time_limit` lowers the limit of each scheduled run, so at `time_limit=600` the 5000 s limit becomes 600 s and the 120 s and 5 s limits stay. `mip_gap` therefore acts on the first-step solve. -`big_m` is the cap on the per-reaction big-M of the on and off constraints. Each scored reaction gets `max(min(ub, big_m), min(-lb, big_m), force_on)`, and an infinite bound counts as `big_m`, so `big_m` limits the flux that a reaction can carry while it counts as on. `force_on` is the smallest flux at which a reaction counts as on. `max_stoich_diff` is the largest ratio between the stoichiometric coefficients of one reaction after `rescale_for_init` (RAVEN's `rescaleModelForINIT`). +`big_m` is the cap on the per-reaction big-M of the on and off constraints. Each scored reaction gets `max(min(ub, big_m), min(-lb, big_m), force_on)`, and an infinite bound counts as `big_m`, so `big_m` limits the flux that a reaction can carry while it counts as on. `force_on` is the smallest flux at which a reaction counts as on. `max_stoich_diff` is the largest ratio between the stoichiometric coefficients of one reaction after `rescale_for_init` (`rescaleModelForINIT` of RAVEN). The robustness runs degrade the expression vector of HCT116 in three ways, each with one random draw (seed 0): -- **Dropout**: a fraction of the genes in the data gets the level 0, which scores −5. -- **Noise**: every level is multiplied by `exp(N(0,σ))`, a log-normal factor with standard deviation σ on the natural-log scale. -- **Downsampling**: a fraction of the genes is removed from the data. Within a gene rule a missing gene is skipped, and a reaction whose genes are all missing gets `no_gene_score`. +- **Dropout:** a fraction of the genes in the data gets the level 0, which scores −5. +- **Noise:** every level is multiplied by `exp(N(0,σ))`, a log-normal factor with standard deviation σ on the natural-log scale. +- **Downsampling:** a fraction of the genes is removed from the data. Within a gene rule a missing gene is skipped, and a reaction whose genes are all missing gets `no_gene_score`. The lever runs repeat the 70 % dropout input with `no_gene_score` or `force_on` changed. @@ -41,27 +49,27 @@ The lever runs repeat the 70 % dropout input with `no_gene_score` or `force_on` |---|---| | time (s) | Wall-clock time of the `run_ftinit` call (first-step and scaling runs) or the `ftinit` call (full pipeline), including the construction of the MILP and excluding the preparation | | objective | MILP objective of the first-step solve | -| rel.obj.gap | (reference objective − objective) / reference objective; a positive value is a lower objective than the reference row | +| rel.obj.gap | (reference objective − objective) / reference objective. A positive value is a lower objective than the reference row | | `n_kept` | Reactions in the model that the full pipeline returns | | n_rxns | Reactions in the model that a robustness run returns, after the gap-fill | -| Jaccard | Jaccard index of two kept-reaction sets: the reactions in both divided by the reactions in either. First-step sets are merged-model reaction identifiers; full-pipeline and robustness sets are Human-GEM reaction identifiers | +| Jaccard | Jaccard index of two kept-reaction sets. First-step sets are merged-model reaction identifiers, and full-pipeline and robustness sets are Human-GEM reaction identifiers | | tasks pass, frac | Essential tasks that `check_tasks` reports as passed on the extracted model, as a count and as a fraction | -A MIP gap bounds the objective only. Two solutions within the gap can keep different reactions, so the reaction set is compared separately from the objective. +A MIP gap bounds the objective. Two solutions within the gap can keep different reactions, so the reaction set is compared separately from the objective. -Each table row is one run unless the row states repeats; this set-up has no runs that differ only in the seed. The only available measure of the seed-to-seed spread comes from a different set-up, in [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](ftinit-determinism.md): two baseline builds (Human-GEM v2.0.0-38, DLD1, `big_m=100`, escalation schedule, with tasks and gap-fill) that differed only in the seed differed by 110 reactions (symmetric difference of the kept-reaction sets). The two baseline builds kept 7779 and 7781 reactions, so the 110 reactions are 1.4 % of the model and correspond to a Jaccard index of 0.986. That pair compares two full pipelines with gap-fill; a first-step set is the result of one MILP step on merged-model identifiers without tasks, a different kind of object. Between the settings of the calibration the Jaccard index ranged from 0.9976 to 0.9649, a difference of 0.2 to 3.5 % from identical sets. +Each table row is one run unless the row states repeats. This set-up has no runs with the same settings and different seeds. The measure of the seed-to-seed spread comes from another set-up, in [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](ftinit-determinism.md). There, two baseline builds that had the same settings and different seeds (Human-GEM v2.0.0-38, DLD1, `big_m=100`, escalation schedule, with tasks and gap-fill) differed by 110 reactions, which is the symmetric difference of the kept-reaction sets. The two builds kept 7779 and 7781 reactions, so the 110 reactions are 1.4 % of the model and correspond to a Jaccard index of 0.986. That pair compares two full pipelines with gap-fill; a first-step set is the result of one MILP step on merged-model identifiers without tasks, which is another kind of object. Between the settings of the calibration the Jaccard index ranged from 0.9976 to 0.9649, a difference of 0.2 to 3.5 % from identical sets. ### Source of the numbers -The calibration tables come from `bigm/bm.py` with the subcommands `step0`, `scale` and `full`, summarised by `bigm/summ.py` and `bigm/compare.py` (prepared model `prep_notask.pkl`). The scripts are in the raven-docs repository under `scripts/parameter_tuning/init/`. `step0` runs a grid of 16 solves: `mip_gap` 0.0002 to 0.1, `big_m` 10 to 1000 and `force_on` 0.02 to 0.5, at `big_m=1000` for the `mip_gap` and `force_on` sweeps. `scale` runs the unscaled preparation rescaled with `max_stoich_diff` 10, 25, 50 and 100 or left unscaled, at `big_m` 1000 and 100. `full` runs one `ftinit` call per setting, with `--gap`, `--big-m`, `--scale` and `--noabs` (`mip_gap_abs=None`) options. +The calibration tables come from `bigm/bm.py` with the subcommands `step0`, `scale` and `full`, summarised by `bigm/summ.py` and `bigm/compare.py` (prepared model `prep_notask.pkl`). The scripts are in the raven-docs repository under `scripts/parameter_tuning/init/`. The subcommand `step0` runs a grid of 16 solves: `mip_gap` 0.0002 to 0.1, `big_m` 10 to 1000 and `force_on` 0.02 to 0.5, at `big_m=1000` for the `mip_gap` and `force_on` sweeps. The subcommand `scale` runs the unscaled preparation rescaled with `max_stoich_diff` 10, 25, 50 and 100 or left unscaled, at `big_m` 1000 and 100. The subcommand `full` runs one `ftinit` call per setting, with the options `--gap`, `--big-m`, `--scale` and `--noabs` (`mip_gap_abs=None`). -The first-step, scaling and full-pipeline tables are computed from the outputs by `compare.py`. The robustness tables (degraded expression input, lever runs and the 90 % dropout run) come from `scripts/analyze_init_robustness.py`. The script exists at the pinned commit; the state of the script and the inputs that produced the tables are not recoverable. +`compare.py` computes the tables of the first-step solve, the scaling runs and the full pipeline from the outputs. The robustness tables (degraded expression input, lever runs and the 90 % dropout run) come from `scripts/analyze_init_robustness.py`. The script exists at the pinned commit, and the state of the script and the inputs that produced the tables are not recoverable. ## Results ### First-step solve with different `mip_gap` values -`big_m=1000`, `force_on=0.1`. The reference is the tightest gap. +The settings are `big_m=1000` and `force_on=0.1`, and the reference is the tightest gap. | `mip_gap` | time (s) | objective | rel.obj.gap | Jaccard vs tightest | |--------:|---------:|----------:|------------:|--------------------:| @@ -74,7 +82,7 @@ The first-step, scaling and full-pipeline tables are computed from the outputs b ### First-step solve with different `big_m` values -`mip_gap=0.001`, `force_on=0.1`. The last two columns compare each set with the run at `big_m=1000` and with the run at `big_m=100`. +The settings are `mip_gap=0.001` and `force_on=0.1`. The last two columns compare each set with the run at `big_m=1000` and with the run at `big_m=100`. | `big_m` | time (s) | rel.obj.gap | Jaccard vs `big_m=1000` | Jaccard vs `big_m=100` | |------:|---------:|------------:|----------------------:|---------------------:| @@ -88,7 +96,7 @@ The first-step, scaling and full-pipeline tables are computed from the outputs b ### First-step solve with different `force_on` values -`mip_gap=0.001`, `big_m=1000`. The reference is the default 0.1. +The settings are `mip_gap=0.001` and `big_m=1000`, and the reference is the default 0.1. | `force_on` | time (s) | rel.obj.gap | Jaccard vs 0.1 | |---------:|---------:|------------:|---------------:| @@ -100,7 +108,7 @@ The first-step, scaling and full-pipeline tables are computed from the outputs b ### First-step solve with different stoichiometric scaling -Scaling runs at `mip_gap=0.001` and two `big_m` values. Every rel.obj.gap and Jaccard index in the table, including the `big_m=100` columns, is against the first row at `big_m=1000` (scale on, `max_stoich_diff=25`, abbreviated `msd`). +The scaling runs used `mip_gap=0.001` and two `big_m` values. Every rel.obj.gap and Jaccard index in the table, including the `big_m=100` columns, is against the first row at `big_m=1000` (scale on, `max_stoich_diff=25`, abbreviated `msd`). | config | time (s), `big_m=1000` | rel.obj.gap | Jaccard vs scaled `msd=25`, `big_m=1000` | time (s), `big_m=100` | rel.obj.gap | Jaccard vs scaled `msd=25`, `big_m=1000` | |--------|---------:|------------:|-------------------------:|---------:|------------:|-------------------------:| @@ -110,13 +118,13 @@ Scaling runs at `mip_gap=0.001` and two `big_m` values. Every rel.obj.gap and Ja | `msd=100` | 26 | +0.0048 | 0.980 | 25 | +0.0049 | 0.980 | | scale off | 30 | −0.0050 | 0.984 | 25 | −0.0044 | 0.983 | -The reference row of this table and the `mip_gap=0.001` row of the first-step `mip_gap` table have the same settings and come from different runs: the scaling run rescales an unscaled preparation with `rescale_for_init` and has a 600 s limit, the first-step run uses `prep(scale=True)` and a 900 s limit. Their objectives are 45539.6 and 45949.2, and their reaction sets share a Jaccard index of 0.983. +The reference row of this table and the `mip_gap=0.001` row of the first-step `mip_gap` table have the same settings and come from different runs. The scaling run rescales an unscaled preparation with `rescale_for_init` and has a 600 s limit, and the first-step run uses `prep(scale=True)` and a 900 s limit. Their objectives are 45539.6 and 45949.2, and their reaction sets share a Jaccard index of 0.983. ### Full pipeline with and without scaling -`time_limit=600`. +The runs used `time_limit=600`. -| config (full pipeline, `time_limit=600`) | result | +| config | result | |-----------------------------------|--------| | scale on, `big_m=1000` | feasible, 83 s | | scale off, `big_m=1000` | feasible, 157 s, 7264 reactions, Jaccard 0.969 vs scale on | @@ -127,7 +135,7 @@ An infeasible run stopped with `ftINIT MILP produced no usable solution (status: ### Full pipeline with different `big_m` values -Scaled preparation without tasks, series `1+1`, `time_limit=600s`. +The runs used a scaled preparation without tasks, series `1+1` and `time_limit=600s`. | config | time (s) | `n_kept` | Jaccard vs `big_m=1000` | |--------|---------:|-------:|---------------------:| @@ -138,11 +146,18 @@ Scaled preparation without tasks, series `1+1`, `time_limit=600s`. | `big_m=50` | 82 | 7325 | 0.982 | | `big_m=25` | 30 | infeasible | none | -The runs were started with different `mip_gap` values, which the series ignores, so runs at one `big_m` are repeats of one configuration: 0.001, 0.003 and 0.01 at `big_m=1000`, 0.001 and 0.003 at `big_m=100`, and 0.003 at the other caps. Further repeats ran with `mip_gap_abs=None` in place of the default `mip_gap_abs=10`: at `big_m=1000` three runs of 85, 84 and 81 s, which kept the same 7312 reactions, and at `big_m=250` one run of 114 s. A third run at `big_m=100` took 119 s (`mip_gap=0.001`, `mip_gap_abs=None`) and a second run at `big_m=50` took 81 s (`mip_gap=0.003`, `mip_gap_abs=None`). In total `big_m=1000` has six runs (81 to 85 s), `big_m=250` two, `big_m=100` three (117, 118 and 119 s) and `big_m=50` two (82 and 81 s); `big_m=500` and `big_m=25` ran once. +The runs were started with different `mip_gap` values, which the series ignores, so runs at one `big_m` are repeats of one configuration: 0.001, 0.003 and 0.01 at `big_m=1000`, 0.001 and 0.003 at `big_m=100`, and 0.003 at the other caps. Further repeats ran with `mip_gap_abs=None` in place of the default `mip_gap_abs=10`: + +- at `big_m=1000`, three runs of 85, 84 and 81 s, which kept the same 7312 reactions; +- at `big_m=250`, one run of 114 s; +- at `big_m=100`, a third run of 119 s (`mip_gap=0.001`, `mip_gap_abs=None`); +- at `big_m=50`, a second run of 81 s (`mip_gap=0.003`, `mip_gap_abs=None`). + +In total `big_m=1000` has six runs (81 to 85 s), `big_m=250` two, `big_m=100` three (117, 118 and 119 s) and `big_m=50` two (82 and 81 s). The caps `big_m=500` and `big_m=25` ran once. ### Degraded expression input -Robustness runs with the task layer on. The clean row is the reference of the Jaccard column. +The robustness runs had the task layer on, and the clean row is the reference of the Jaccard column. | input | n_rxns | tasks pass | frac | Jaccard vs clean | |-------|-------:|-----------:|-----:|-----------------:| @@ -158,7 +173,7 @@ One model from 90 % dropout input, outside these tables, ran for more than 75 mi ### Parameters at 70 % dropout -Lever runs on the 70 % dropout input. The default row is the dropout 70 % row of the previous table. +The lever runs used the 70 % dropout input. The default row is the dropout 70 % row of the previous table. | config | n_rxns | frac | Jaccard vs clean | |--------|-------:|-----:|-----------------:| @@ -167,31 +182,41 @@ Lever runs on the 70 % dropout input. The default row is the dropout 70 % row of | `no_gene_score=−0.5` | 5128 | 0.986 | 0.593 | | `force_on=0.2` | 5159 | 0.986 | 0.600 | -## Effect of `mip_gap` on a single solve +## Interpretation + +### Effect of `mip_gap` on a single solve -In the first-step solve the time was 24 to 25 s at every gap from 0.0002 to 0.1. At `mip_gap=0.001` the objective was within 0.0001 of the tightest gap and the Jaccard index was 0.9976. The Jaccard index stayed at 0.99 or above up to `mip_gap=0.01` and fell to 0.9649 and 0.9663 at 0.03 and 0.1. In the full pipeline the gap has no effect: the three `big_m=1000` runs at 0.001, 0.003 and 0.01 kept the same 7312 reactions. +In the first-step solve the time was 24 to 25 s at every gap from 0.0002 to 0.1. At `mip_gap=0.001` the objective was within 0.0001 of the tightest gap and the Jaccard index was 0.9976. The index stayed at 0.99 or above up to `mip_gap=0.01` and fell to 0.9649 and 0.9663 at 0.03 and 0.1. In the full pipeline the gap has no effect: the three `big_m=1000` runs at 0.001, 0.003 and 0.01 kept the same 7312 reactions. -## Effect of `big_m` and scaling +### Effect of `big_m` and scaling -In the first-step solve every `big_m` from 10 to 1000 was feasible, took 23 to 26 s and reached the objective of `big_m=1000` within 0.0003. Against the run at `big_m=1000` and against the run at `big_m=100`, the reaction sets of the other caps had Jaccard indices of 0.983 to 0.991. The first-step solve has no seed repeats, and the seed pair in [Set-up](#set-up) compares full pipelines with gap-fill, so it is no reference for these sets. In the first-step scaling runs, every `max_stoich_diff` from 10 to 100 and the unscaled model were feasible at both caps, with Jaccard indices of 0.980 to 0.991 against the scaled reference. +In the first-step solve every `big_m` from 10 to 1000 was feasible, took 23 to 26 s and reached the objective of `big_m=1000` within 0.0003. Against the run at `big_m=1000` and against the run at `big_m=100`, the reaction sets of the other caps had Jaccard indices of 0.983 to 0.991. The seed pair of the [Set-up](#set-up) is no reference for these sets. In the first-step scaling runs, every `max_stoich_diff` from 10 to 100 and the unscaled model were feasible at both caps, with Jaccard indices of 0.980 to 0.991 against the scaled reference. -The full pipeline separates the settings that the first-step solve does not. At `big_m=1000` it was feasible with and without scaling: 83 s with scaling, 157 s without, and the two models shared a Jaccard index of 0.969 (7312 and 7264 reactions). At `big_m=100` it was feasible with scaling (117 to 119 s) and infeasible without, and at `big_m=25` it was infeasible with scaling. With scaling, the caps from 50 to 1000 gave runs of 81 to 119 s with no trend in `big_m` and reaction sets that shared a Jaccard index of 0.978 to 0.982 with the default. The fastest run at a cap below 1000 (80.8 s at `big_m=50`, with `mip_gap_abs=None`) was as fast as the fastest `big_m=1000` run (81.1 s). Counted from the reaction sets (the rounded Jaccard values in the table give 148 and 163 for the first two), the sets at caps 500, 250, 100 and 50 differed from the `big_m=1000` set by 150, 165, 140 and 133 reactions. The one seed pair available, from the different set-up in [Set-up](#set-up), differed by 110 reactions, so every cap difference is 21 to 50 % larger than that single value; the seed-to-seed spread of this set-up is unknown. The three `big_m=1000` runs kept identical sets (Jaccard 1.000) and only the cap changed between rows, so the differences follow the cap; whether they are systematic or a different choice among near-equal optima was not tested. A lower cap also reduces the flux that a kept reaction can carry. +The full pipeline separates the settings that the first-step solve does not. -## Effect of `force_on` +- At `big_m=1000` it was feasible with and without scaling. It took 83 s with scaling and 157 s without, and the two models shared a Jaccard index of 0.969 (7312 and 7264 reactions). +- At `big_m=100` it was feasible with scaling (117 to 119 s) and infeasible without. +- At `big_m=25` it was infeasible with scaling. +- With scaling, the caps from 50 to 1000 gave runs of 81 to 119 s with no trend in `big_m`, and reaction sets that shared a Jaccard index of 0.978 to 0.982 with the default. +- The fastest run at a cap below 1000 (80.8 s at `big_m=50`, with `mip_gap_abs=None`) was as fast as the fastest `big_m=1000` run (81.1 s). -`force_on` changes the MILP itself, through the on and off constraints. From 0.02 to 0.5 the first-step solve took 24 to 25 s, the objective moved by at most 0.0002 and the reaction set shared a Jaccard index of 0.987 or more with the default 0.1. The full pipeline was not run with other `force_on` values. +Counted from the reaction sets, the sets at the caps 500, 250, 100 and 50 differed from the `big_m=1000` set by 150, 165, 140 and 133 reactions. The rounded Jaccard values in the table give 148 and 163 for the first two. Every cap difference is 21 to 50 % larger than the 110 reactions of the seed pair of the [Set-up](#set-up). The three `big_m=1000` runs kept identical sets (Jaccard 1.000) and only the cap changed between rows, so the differences follow the cap. Whether they are systematic or a different choice among near-equal optima was not tested. -## Effect of degraded input +### Effect of `force_on` -Noise changed the model less than sparse input. At σ=1.0 and σ=2.0 the model kept 7812 and 7768 reactions against 7777 for clean data, passed all 69 tasks and shared a Jaccard index of 0.952 and 0.919 with the clean model. Dropout of 50 % and 70 % reduced the model to 5968 and 5113 reactions (Jaccard 0.713 and 0.594), and downsampling of 50 % and 70 % to 6765 and 6123 reactions (Jaccard 0.815 and 0.728). At equal fractions dropout removed more reactions than downsampling. The difference matches the scores involved, −5 for a zero level against −2 for a reaction without measured genes, but this explanation was not tested. Under dropout and downsampling the model passed 67 or 68 of 69 tasks; the failing tasks and the cause of the failures were not examined. +The setting `force_on` changes the MILP itself, through the on and off constraints. From 0.02 to 0.5 the first-step solve took 24 to 25 s, the objective moved by at most 0.0002 and the reaction set shared a Jaccard index of 0.987 or more with the default 0.1. The full pipeline was not run with other `force_on` values. -At 70 % dropout no tested lever moved the model back towards the clean model: the Jaccard index stayed at 0.593 to 0.600, the size at 5110 to 5159 reactions and the task fraction at 0.986. Under dropout the dropped genes stay in the data with a zero level and score −5, so `no_gene_score` acts only on reactions without gene rules and on reactions whose genes are all absent from the data. The study concludes that no parameter recovers the expression evidence that dropout removes; this is an inference from the four lever settings, which do not include other parameters. The lever runs did not include downsampled input, so the effect of `no_gene_score` on missing genes was not measured. +### Effect of degraded input -In raven-toolbox commit `94f3940`, `ftinit` calls `fill_tasks` without a time limit, so the gap-fill uses the default of `fill_tasks`, 300 s per task (`_FILL_TIME_LIMIT`), and not the `time_limit` of the extraction. The time limit that the 90 % dropout run of more than 75 min used is not recorded. +Noise changed the model less than sparse input. At σ=1.0 and σ=2.0 the model kept 7812 and 7768 reactions against 7777 for clean data, passed all 69 tasks and shared a Jaccard index of 0.952 and 0.919 with the clean model. Dropout of 50 % and 70 % reduced the model to 5968 and 5113 reactions (Jaccard 0.713 and 0.594), and downsampling of 50 % and 70 % to 6765 and 6123 reactions (Jaccard 0.815 and 0.728). At equal fractions dropout removed more reactions than downsampling. The difference matches the scores involved, −5 for a zero level against −2 for a reaction without measured genes, but this explanation was not tested. Under dropout and downsampling the model passed 67 or 68 of 69 tasks, and the failing tasks and the cause of the failures were not examined. -## Choice of settings +At 70 % dropout no tested lever moved the model back towards the clean model. The Jaccard index stayed at 0.593 to 0.600, the size at 5110 to 5159 reactions and the task fraction at 0.986. Under dropout the dropped genes stay in the data with a zero level and score −5, so `no_gene_score` acts on reactions without gene rules and on reactions whose genes are all absent from the data. The conclusion that no parameter recovers the expression evidence that dropout removes is an inference from the four lever settings, which do not include other parameters. The lever runs did not include downsampled input, so the effect of `no_gene_score` on missing genes was not measured. -The defaults `big_m=1000`, `force_on=0.1` and scaling on with `max_stoich_diff=25` are RAVEN's values; every `big_m` from 50 to 1000 was feasible with scaling on, and `big_m=25` was infeasible. `big_m=1000` is the cap that RAVEN's `ftINITInternalAlg` uses (`bigMCap`). Scaling is not needed for feasibility at `big_m=1000`; in these single runs the pipeline took 83 s with scaling and 157 s without. In the first-step solve, `mip_gap=0.001` reached an objective within 0.0001 of the tightest gap; a named series ignores the setting. The full-pipeline runs used `time_limit=600` and the robustness script uses `time_limit=120`; no full-pipeline run took longer than 157 s in total. For noisy input the tested settings needed no change, and no tested setting compensated for dropout. +In raven-toolbox commit `94f3940`, `ftinit` calls `fill_tasks` without a time limit, so the gap-fill uses the default of `fill_tasks`, 300 s per task (`_FILL_TIME_LIMIT`), independent of the `time_limit` of the extraction. The time limit that the 90 % dropout run of more than 75 min used is not recorded. + +### Choice of settings + +The defaults `big_m=1000`, `force_on=0.1` and scaling on with `max_stoich_diff=25` are the values of RAVEN. Every `big_m` from 50 to 1000 was feasible with scaling on, and `big_m=25` was infeasible. The cap `big_m=1000` is the one that `ftINITInternalAlg` of RAVEN uses (`bigMCap`). Scaling is not needed for feasibility at `big_m=1000`, and in these single runs the pipeline took 83 s with scaling and 157 s without. In the first-step solve, `mip_gap=0.001` reached an objective within 0.0001 of the tightest gap, and a named series ignores the setting. The full-pipeline runs used `time_limit=600` and the robustness script uses `time_limit=120`. No full-pipeline run took longer than 157 s in total. ## Gene-scoring constants @@ -201,18 +226,22 @@ Expression values become gene scores through score = clip(factor × ln(level / reference), min_score, max_score) ``` -with `factor=5.0`, `max_score=10.0` and `min_score=-5.0`, the same in both implementations. The logarithm is natural (base e) in `init/score.py` and in RAVEN's `scoreModel`. The source of each implementation cites no paper for the three constants. A 2012 paper (Wang et al. 2012, [mCADRE](https://doi.org/10.1186/1752-0509-6-153)) uses categorical scoring and is not the origin. The constants are RAVEN's own values without a literature source. +with `factor=5.0`, `max_score=10.0` and `min_score=-5.0`, the same in both implementations. The logarithm is natural (base e) in `init/score.py` and in `scoreModel` of RAVEN. The source of each implementation cites no paper for the three constants. A 2012 paper (Wang et al. 2012, [mCADRE](https://doi.org/10.1186/1752-0509-6-153)) uses categorical scoring and did not introduce these constants. The constants are values of RAVEN itself without a literature source. ## Comparison with the solver benchmark -[ftINIT on different solvers](init-solver-benchmark.md) runs the same full pipeline with each solver interface. With Gurobi it took 92 s and kept 7312 reactions, against 83 s and 7312 reactions here. The benchmark time also includes loading the prepared model and scoring the reactions, and its limit is 900 s against 600 s here. A limit shortens only the 5000 s runs of the schedule, and no Gurobi run in either study ran for its overall limit. The warning `ftINIT step 1 hit the 600s time limit` appears 14 times in 13 full-pipeline and validation logs, and the Gurobi run of the solver benchmark has the same warning with 900 s. The logged `step 1` is the second step. The warning prints the `time_limit` argument, not the limit of the run that stopped; the stopped run is a shorter scheduled run, inferred from the total times of 83 s and 92 s, and the logs do not say which. One scheduled solve per run stopped at a limit. The Gurobi run of the solver benchmark at `big_m=100` kept 7324 reactions, the count of the two `big_m=100` runs here, whose reaction set shares a Jaccard index of 0.981 with the `big_m=1000` set. The solver comparison itself, with GLPK at 2182 s and a Jaccard index of 0.892 against Gurobi and the failure of HiGHS through the `scipy` interface, is on that page. +[ftINIT on different solvers](init-solver-benchmark.md) runs the same full pipeline with each solver interface. With Gurobi it took 92 s and kept 7312 reactions, against 83 s and 7312 reactions in this study. The benchmark time also includes loading the prepared model and scoring the reactions, and its limit is 900 s against 600 s in this study. A limit shortens the 5000 s runs of the schedule, and no Gurobi run in either study ran for its overall limit. + +The warning `ftINIT step 1 hit the 600s time limit` appears 14 times in 13 full-pipeline and validation logs, and the Gurobi run of the solver benchmark has the same warning with 900 s. The logged `step 1` is the second step. The warning prints the `time_limit` argument, which differs from the limit of the run that stopped. The stopped run is a shorter scheduled run, inferred from the total times of 83 s and 92 s, and the logs do not say which. One scheduled solve per run stopped at a limit. + +The Gurobi run of the solver benchmark at `big_m=100` kept 7324 reactions, the count of the two `big_m=100` runs of this study, whose reaction set shares a Jaccard index of 0.981 with the `big_m=1000` set. The solver comparison itself is on that page: GLPK at 2182 s with a Jaccard index of 0.892 against Gurobi, and the failure of HiGHS through the `scipy` interface. ## Limits - One model (Human-GEM), one cell line (HCT116) and one solver (Gurobi library 13.0.3 from the gurobipy wheel, most likely; the logs print no version). -- Most settings ran once; the repeats are listed with the full-pipeline table. This set-up has no seed repeats. The full-pipeline differences between caps are larger than the one seed pair available, which comes from a different set-up, so the seed-to-seed spread of this set-up is unknown. -- The robustness runs used a Human-GEM commit that is not recorded and `big_m=100`; the script at the pinned commit passes no `big_m`, so a run with it uses the default 1000. Each degradation is one random draw at one seed. -- The first-step solve shows no difference between caps or scaling settings that the full pipeline shows, so first-step results alone do not predict feasibility of the pipeline. +- Most settings ran once, and the repeats are listed with the full-pipeline table. This set-up has no repeats of the seed. The full-pipeline differences between caps are larger than the seed pair of the Set-up, so the seed-to-seed spread of this set-up is unknown. +- The robustness runs used a Human-GEM commit that is not recorded and `big_m=100`. The script at the pinned commit passes no `big_m`, so a run with it uses the default 1000. Each degradation is one random draw at one seed. +- The first-step solve shows no difference between caps or scaling settings that the full pipeline shows, so first-step results alone do not predict the feasibility of the pipeline. - The robustness tables have no run times, and the effect of `no_gene_score` on downsampled input was not measured. ## Reproduction @@ -230,4 +259,4 @@ python summ.py $WORK/out python compare.py --out $WORK/out --shared $WORK/shared --logs $WORK/logs ``` -The tags of the other full-pipeline runs are listed in the README: `--big-m` 25, 50, 250 and 500 at `--gap 0.003`, the `mip_gap` runs 0.001, 0.003 and 0.01 at `big_m=1000`, and the `--noabs` repeats. The unscaled preparation of `scale` takes about 26 min. The robustness tables have no reproduction command: the state of `analyze_init_robustness.py` and the inputs that produced them are not recoverable. +The tags of the other full-pipeline runs are listed in the README: `--big-m` 25, 50, 250 and 500 at `--gap 0.003`, the `mip_gap` runs 0.001, 0.003 and 0.01 at `big_m=1000`, and the `--noabs` repeats. The unscaled preparation of `scale` takes about 26 min. The robustness tables have no reproduction command, because the state of `analyze_init_robustness.py` and the inputs that produced them are not recoverable. diff --git a/docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md b/docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md index 7058f4c..b0d30be 100644 --- a/docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md +++ b/docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md @@ -1,12 +1,14 @@ # KEGG HMM cut-off calibration -KEGG-based reconstruction from sequence assigns each protein of a new organism to KEGG Orthology (KO) groups by searching the proteome against profile hidden Markov models (HMMs) of the KOs. The study ran this search on the proteomes of four organisms that KEGG annotates, assigned genes to KOs at a range of cut-offs and score ratios, and compared the result with KEGG's own gene→KO annotation of each organism. For each setting it records precision, recall and F1 of the gene→KO pairs, the fraction of the organism's annotated reactions that the prediction recovers and the number of predicted reactions outside the annotation. The results set the defaults `cutoff=1e-30` and `min_score_ratio_g=0.9` of `assign_kos` in place of the RAVEN 2.11.3 values `1e-50` and `0.8`. +KEGG-based reconstruction from sequence assigns each protein of a new organism to KEGG Orthology (KO) groups by searching the proteome against profile hidden Markov models (HMMs) of the KOs. The study ran this search on the proteomes of four organisms that KEGG annotates, assigned genes to KOs at a range of cut-offs and score ratios, and compared the result with the gene→KO annotation of KEGG for each organism. For each setting it records the precision, recall and F1 of the gene→KO pairs, the fraction of the annotated reactions of the organism that the prediction recovers, and the number of predicted reactions outside the annotation. + +The results set the defaults `cutoff=1e-30` and `min_score_ratio_g=0.9` of `assign_kos`, in place of the RAVEN 2.11.3 values `1e-50` and `0.8`. The defaults raise gene→KO recall by 0.01 to 0.09 and lower precision in three of the four organisms (by 0.01 to 0.07), and F1 changes by at most 0.02. ## Set-up ### Parameters under test -`assign_kos` (`reconstruction/kegg/query.py`, called by `get_kegg_model_from_sequences`) turns a KO×gene E-value matrix into gene→KO assignments in three steps. RAVEN's `getKEGGModelForOrganism` applies the same steps with the arguments `cutOff`, `minScoreRatioKO` and `minScoreRatioG`. +The function `assign_kos` (`reconstruction/kegg/query.py`, called by `get_kegg_model_from_sequences`) turns a KO×gene E-value matrix into gene→KO assignments in three steps. `getKEGGModelForOrganism` of RAVEN applies the same steps with the arguments `cutOff`, `minScoreRatioKO` and `minScoreRatioG`. 1. `cutoff` keeps hits with `evalue <= cutoff`. 2. `min_score_ratio_ko` drops, within a KO, genes whose `log(evalue)/log(best_evalue_in_KO) < min_score_ratio_ko`. @@ -16,16 +18,16 @@ KEGG-based reconstruction from sequence assigns each protein of a new organism t The percentile, cut-off, score-ratio and defaults tables come from one run on 2026-11-01 (raven-toolbox commit `94f3940`, HMMER 3.4, 16 threads on an AMD EPYC 9354 node) of the sources below. The timing and the true-reaction counts come from separate jobs. -- Tables: the kegg118 artefacts (`organism_gene_ko`, `ko_reaction` and the reference model) published as the `kegg118` release of toolbox-assets. -- HMM libraries: the published kegg118 libraries `kegg118_eukaryotes.hmm` (2698 profiles, 696,233,564 bytes) and `kegg118_prokaryotes.hmm` (4205 profiles, 838,505,885 bytes), built by the maintainers with `build_ko_hmm`. The sequences of each KO were clustered with CD-HIT at 90 % identity and aligned with MAFFT FFT-NS-2, or with MAFFT PartTree for KOs whose alignment exceeds a memory budget. -- Queries: the CDS genes of each organism from the KEGG REST lists, downloaded on 2026-11-01 (KEGG genes updated 2026-10-31, which is not KEGG release 118): 6021, 5010, 4288 and 476 sequences, of 6449, 5094, 4639 and 525 list entries. The sequences were not compared with those of KEGG release 118. The search is `hmmsearch`; `run_hmmsearch` in raven-toolbox fixes `-Z` to the number of profiles. Whether these E-values equal those of `hmmscan` against the same library was not checked. -- Each organism's hits were computed once, and every setting was applied to the same hit table. The differences between settings therefore contain no search-to-search variation. For `sce` the parsed hit tables were identical in four runs at 16, 12, 4 and 1 threads; the other organisms were searched once. +- **Tables.** The kegg118 artefacts (`organism_gene_ko`, `ko_reaction` and the reference model), published as the `kegg118` release of toolbox-assets. +- **HMM libraries.** The published kegg118 libraries `kegg118_eukaryotes.hmm` (2698 profiles, 696,233,564 bytes) and `kegg118_prokaryotes.hmm` (4205 profiles, 838,505,885 bytes), built by the maintainers with `build_ko_hmm`. The sequences of each KO were clustered with CD-HIT at 90 % identity and aligned with MAFFT FFT-NS-2, or with MAFFT PartTree for KOs whose alignment exceeds a memory budget. +- **Queries.** The CDS genes of each organism from the KEGG REST lists, downloaded on 2026-11-01 (KEGG genes updated 2026-10-31, which is not KEGG release 118): 6021, 5010, 4288 and 476 sequences, of 6449, 5094, 4639 and 525 list entries. The sequences were not compared with those of KEGG release 118. The search is `hmmsearch`, and `run_hmmsearch` in raven-toolbox fixes `-Z` to the number of profiles. Whether these E-values equal those of `hmmscan` against the same library was not checked. +- **Hit tables.** The hits of each organism were computed once, and every setting was applied to the same hit table, so the differences between settings contain no variation between searches. For `sce` the parsed hit tables were identical in four runs at 16, 12, 4 and 1 threads, and the other organisms were searched once. ### Reference and scores -The reference is the **KEGG annotation**: the organism's gene→KO pairs in the `organism_gene_ko` table of the kegg118 artefacts. The table lists the KOs linked to at least one reaction in `ko_reaction`, and the HMM libraries are built from the same table. The organism's **true reactions** are the reactions linked to its annotated KOs, limited to reactions that are in the kegg118 reference model (`--restrict-to-reference`). The limit is applied because reaction recall is measured against reactions that the draft can contain. +The reference is the KEGG annotation, which is the gene→KO pairs of the organism in the `organism_gene_ko` table of the kegg118 artefacts. The table lists the KOs linked to at least one reaction in `ko_reaction`, and the HMM libraries are built from the same table. The true reactions of the organism are the reactions linked to its annotated KOs, limited to reactions that are in the kegg118 reference model (`--restrict-to-reference`). The limit is applied because the reaction recall is measured against reactions that the draft can contain. -A predicted pair is the output of `assign_kos`, with the `organism:` prefix stripped from the gene identifier so that it matches the reference. Each predicted pair is either **matched** (present in the annotation) or **novel** (absent from it). +A predicted pair is the output of `assign_kos`, with the `organism:` prefix stripped from the gene identifier so that it matches the reference. Each predicted pair is either matched (present in the annotation) or novel (absent from it). | Score | Definition | |---|---| @@ -48,7 +50,7 @@ A novel pair counts as an error, although some novel pairs may be correct assign Without the limit to the reference model, the published `ko_reaction` table (7642 reactions, against 12,373 in the reference model) gives 1407, 1357, 1772 and 274 true reactions for the four organisms. The reaction columns of the tables below follow the limited definition. -`sce` and `eco` are model organisms; `cme` and `mge` are less studied. `mge` has the smallest proteome of the four and the weakest matched hits in the E-value table. +The organisms `sce` and `eco` are model organisms, and `cme` and `mge` are less studied. The organism `mge` has the smallest proteome of the four and the weakest matched hits in the E-value table. ## Results @@ -69,11 +71,11 @@ The table gives percentiles of `log10(E-value)` of the best hit per (gene, KO) p The median matched pair had an E-value between 1e-102 (`mge`) and 1e-153 (`sce`), and the weakest 1 % of matched pairs lay between 1e-15 and 1e-36. The median novel pair lay between 1e-4 and 1e-8. The weakest 1 % of matched pairs was 11 (`mge`) to 29 (`eco`) orders of magnitude stronger than the median novel pair. -The RAVEN 2.11.3 cut-off of `1e-50` falls inside the matched distribution: between p95 and p99 for `sce` and `eco`, between p90 and p95 for `cme`, and between p50 and p90 for `mge`. It therefore removes 1 to 5 % of the matched pairs of `sce` and `eco`, 5 to 10 % of `cme` and 10 to 50 % of `mge`. The default `1e-30` lies beyond p99 for `eco` (fewer than 1 % of the matched pairs removed) and between p95 and p99 for `sce`, `cme` and `mge`, and is 22 to 26 orders of magnitude stronger than the median novel pair. +The RAVEN 2.11.3 cut-off of `1e-50` falls inside the matched distribution: between p95 and p99 for `sce` and `eco`, between p90 and p95 for `cme`, and between p50 and p90 for `mge`. It therefore removes 1 to 5 % of the matched pairs of `sce` and `eco`, 5 to 10 % of `cme` and 10 to 50 % of `mge`. The default `1e-30` lies beyond p99 for `eco` (fewer than 1 % of the matched pairs removed) and between p95 and p99 for `sce`, `cme` and `mge`, and it is 22 to 26 orders of magnitude stronger than the median novel pair. ### Cut-off sweep -`min_score_ratio_ko = 0.3` and `min_score_ratio_g = 0.8` (the RAVEN 2.11.3 values) were fixed while `cutoff` varied. +The values `min_score_ratio_ko = 0.3` and `min_score_ratio_g = 0.8` (the RAVEN 2.11.3 values) were fixed while `cutoff` varied. #### `sce` @@ -119,11 +121,11 @@ The RAVEN 2.11.3 cut-off of `1e-50` falls inside the matched distribution: betwe | 1e-70 | 0.85 | 0.73 | 0.78 | 0.72 | 27 | | 1e-100 | 0.88 | 0.53 | 0.66 | 0.55 | 21 | -Gene→KO recall was between 0.95 and 0.99 from 1e-10 to 1e-30 in all four organisms and fell at stricter cut-offs. The fall was largest for `mge`, whose reaction recall dropped from 0.98 at 1e-30 to 0.88 at 1e-50 and 0.55 at 1e-100. Precision rose with every stricter cut-off in all four organisms, from 0.73 to 0.78 for `sce` between 1e-30 and 1e-50. With the ratios fixed at 0.3 and 0.8, F1 was highest at 1e-50 or 1e-70 in every organism. Novel predictions remained at the strictest cut-off: at 1e-100 the three larger proteomes had 61 to 137 novel reactions and a gene→KO precision of 0.80 to 0.85 (0.88 for `mge`). The study does not show whether these are wrong assignments, paralogs or genes that KEGG has not annotated. +Gene→KO recall was between 0.95 and 0.99 from 1e-10 to 1e-30 in all four organisms and fell at stricter cut-offs. The fall was largest for `mge`, whose reaction recall dropped from 0.98 at 1e-30 to 0.88 at 1e-50 and 0.55 at 1e-100. Precision rose with every stricter cut-off in all four organisms, for example from 0.73 to 0.78 for `sce` between 1e-30 and 1e-50. With the ratios fixed at 0.3 and 0.8, F1 was highest at 1e-50 or 1e-70 in every organism. Novel predictions remained at the strictest cut-off: at 1e-100 the three larger proteomes had 61 to 137 novel reactions and a gene→KO precision of 0.80 to 0.85 (0.88 for `mge`). The study does not show whether these are wrong assignments, paralogs or genes that KEGG has not annotated. ### Score-ratio sweep -`cutoff` was fixed at `1e-50`. The script scores every combination of `min_score_ratio_ko` (0.0, 0.3, 0.5) and `min_score_ratio_g` (0.5, 0.8, 0.95); the table holds the five combinations per organism that are reported. +The value `cutoff` was fixed at `1e-50`. The script scores every combination of `min_score_ratio_ko` (0.0, 0.3, 0.5) and `min_score_ratio_g` (0.5, 0.8, 0.95), and the table holds the five combinations per organism that are reported. | organism | ko ratio | g ratio | gKO prec | gKO rec | gKO F1 | |---|---|---|---|---|---| @@ -148,13 +150,13 @@ Gene→KO recall was between 0.95 and 0.99 from 1e-10 to 1e-30 in all four organ | `mge` | 0.3 | 0.80 | 0.77 | 0.85 | 0.81 | | `mge` | 0.5 | 0.80 | 0.77 | 0.85 | 0.81 | -At `min_score_ratio_g = 0.80`, raising `min_score_ratio_ko` from 0.0 to 0.5 raised precision by 0.02 (`sce`), 0.01 (`cme`), 0.03 (`eco`) and 0.00 (`mge`), and lowered recall by 0.01 for `cme` and `eco` and not at all for `sce` and `mge`. +At `min_score_ratio_g = 0.80`, raising `min_score_ratio_ko` from 0.0 to 0.5 raised the precision by 0.02 (`sce`), 0.01 (`cme`), 0.03 (`eco`) and 0.00 (`mge`), and lowered the recall by 0.01 for `cme` and `eco` and not at all for `sce` and `mge`. -Raising `min_score_ratio_g` from 0.80 to 0.95 (at `min_score_ratio_ko = 0.0`) raised precision by 0.06 (`sce`), 0.08 (`cme`), 0.10 (`eco`) and 0.05 (`mge`), and lowered recall by 0.00 to 0.02. At 0.50, precision was the lowest of the three values in every organism. The value 0.9 lies between the tested values 0.8 and 0.95 and was not swept. +Raising `min_score_ratio_g` from 0.80 to 0.95 (at `min_score_ratio_ko = 0.0`) raised the precision by 0.06 (`sce`), 0.08 (`cme`), 0.10 (`eco`) and 0.05 (`mge`), and lowered the recall by 0.00 to 0.02. At 0.50 the precision was the lowest of the three values in every organism. The value 0.9 lies between the tested values 0.8 and 0.95 and was not swept. ### Defaults compared with RAVEN 2.11.3 -The table compares the RAVEN 2.11.3 settings `(1e-50, 0.3, 0.8)` with the defaults `(1e-30, 0.3, 0.9)`, given as `(cutoff, min_score_ratio_ko, min_score_ratio_g)`. All values are scores of the saved hit tables; the RAVEN 2.11.3 values are the 1e-50 rows of the cut-off sweep. +The table compares the RAVEN 2.11.3 settings `(1e-50, 0.3, 0.8)` with the defaults `(1e-30, 0.3, 0.9)`, given as `(cutoff, min_score_ratio_ko, min_score_ratio_g)`. All values are scores of the saved hit tables, and the RAVEN 2.11.3 values are the 1e-50 rows of the cut-off sweep. | organism | gKO prec | gKO rec | gKO F1 | rxn rec | rxn novel | |---|---|---|---|---|---| @@ -163,11 +165,11 @@ The table compares the RAVEN 2.11.3 settings `(1e-50, 0.3, 0.8)` with the defaul | `eco` | 0.68 → 0.69 | 0.96 → 0.97 | 0.79 → 0.81 | 0.98 → 0.99 | 208 → 173 | | `mge` | 0.77 → 0.70 | 0.85 → 0.94 | 0.81 → 0.80 | 0.88 → 0.97 | 30 → 33 | -With the defaults, gene→KO recall rose by 0.09 for `mge` and by 0.01 to 0.03 for the other three organisms. Gene→KO precision fell by 0.01 (`sce`), 0.02 (`cme`) and 0.07 (`mge`), and rose by 0.01 for `eco`. F1 changed by at most 0.01 for `sce`, `cme` and `mge` and rose by 0.02 for `eco`. All scores are rounded to two decimals, so a difference of 0.01 lies within the rounding. The number of novel reactions fell for `sce`, `cme` and `eco` and rose from 30 to 33 for `mge`. +With the defaults, the gene→KO recall rose by 0.09 for `mge` and by 0.01 to 0.03 for the other three organisms. The gene→KO precision fell by 0.01 (`sce`), 0.02 (`cme`) and 0.07 (`mge`), and rose by 0.01 for `eco`. The F1 changed by at most 0.01 for `sce`, `cme` and `mge` and rose by 0.02 for `eco`. All scores are rounded to two decimals, so a difference of 0.01 lies within the rounding. The number of novel reactions fell for `sce`, `cme` and `eco` and rose from 30 to 33 for `mge`. ### Whole-model comparison of the two KEGG paths -*S. cerevisiae* was reconstructed in full by two paths, both with the RAVEN 2.11.3 settings `(1e-50, 0.3, 0.8)`. The annotation path takes the organism's genes and KOs from the `organism_gene_ko` table; the HMM path assigns KOs to the proteome with the HMM search. +*S. cerevisiae* was reconstructed in full by two paths, both with the RAVEN 2.11.3 settings `(1e-50, 0.3, 0.8)`. The annotation path takes the genes and KOs of the organism from the `organism_gene_ko` table, and the HMM path assigns KOs to the proteome with the HMM search. | Count | annotation path | HMM path | |---|---|---| @@ -175,33 +177,35 @@ With the defaults, gene→KO recall rose by 0.09 for `mge` and by 0.01 to 0.03 f | metabolites | 1502 | 1565 | | genes | 838 | 899 | -The HMM path contained 96.2 % of the reactions of the annotation path (1306 of 1357, Jaccard index 0.87) and 96.5 % of its genes (809 of 838, Jaccard index 0.87). The counts were measured with the published kegg118 tables and libraries. The search of `sce` against the eukaryote library is timed under [Settings without a sweep](#settings-without-a-sweep); the prokaryote library was not timed. +The HMM path contained 96.2 % of the reactions of the annotation path (1306 of 1357, Jaccard index 0.87) and 96.5 % of its genes (809 of 838, Jaccard index 0.87). The counts were measured with the published kegg118 tables and libraries. The search of `sce` against the eukaryote library is timed under [Settings without a sweep](#settings-without-a-sweep), and the prokaryote library was not timed. + +## Interpretation -## Meaning of the results +From 1e-10 to 1e-30 a stricter cut-off raised the precision while the gene→KO recall fell by at most 0.03 (from 0.98 or 0.99 to between 0.95 and 0.99). Beyond 1e-30 the cut-off traded recall for precision: it removed matched pairs together with novel pairs, and novel pairs remained at every cut-off tested. Across the whole sweep from 1e-10 to 1e-100 the precision rose by 0.28 (`sce`, 0.57 to 0.85) to 0.35 (`mge`). The RAVEN 2.11.3 value `1e-50` removes 10 to 50 % of the matched pairs of `mge`, the organism with the weakest matched hits, and 1 to 10 % in the other three. -From 1e-10 to 1e-30 a stricter cut-off raised precision while gene→KO recall fell by at most 0.03 (from 0.98 or 0.99 to between 0.95 and 0.99). Beyond 1e-30 the cut-off traded recall for precision: it removed matched pairs together with novel pairs, and novel pairs remained at every cut-off tested. Across the whole sweep from 1e-10 to 1e-100 precision rose by 0.28 (`sce`, 0.57 to 0.85) to 0.35 (`mge`). The RAVEN 2.11.3 value `1e-50` removes 10 to 50 % of the matched pairs of `mge`, the organism with the weakest matched hits, and 1 to 10 % in the other three. +At `cutoff = 1e-50`, raising `min_score_ratio_g` from 0.80 to 0.95 raised the precision by 0.05 to 0.10 for a recall loss of 0.00 to 0.02, and raising it from 0.50 to 0.80 raised the precision by 0.15 to 0.27 (`eco`, 0.40 to 0.67). At `min_score_ratio_g = 0.80` and `cutoff = 1e-50`, raising `min_score_ratio_ko` from 0.0 to 0.5 raised the precision by 0.00 to 0.03 and lowered the recall by 0.00 to 0.01. Its effect at other settings was not reported. -At `cutoff = 1e-50`, raising `min_score_ratio_g` from 0.80 to 0.95 raised precision by 0.05 to 0.10 for a recall loss of 0.00 to 0.02, and raising it from 0.50 to 0.80 raised precision by 0.15 to 0.27 (`eco`, 0.40 to 0.67). At `min_score_ratio_g = 0.80` and `cutoff = 1e-50`, raising `min_score_ratio_ko` from 0.0 to 0.5 raised precision by 0.00 to 0.03 and lowered recall by 0.00 to 0.01; its effect at other settings was not reported. The defaults move part of the filtering from the cut-off to `min_score_ratio_g`. On the four organisms the move gained recall, most for `mge`, at the cost of precision in three of them, and left F1 within 0.01 of the RAVEN 2.11.3 settings for three organisms and raised it by 0.02 for `eco`. +The defaults move part of the filtering from the cut-off to `min_score_ratio_g`. On the four organisms the move gained recall, most for `mge`, at the cost of precision in three of them. It left the F1 within 0.01 of the RAVEN 2.11.3 settings for three organisms and raised it by 0.02 for `eco`. ## Chosen defaults | parameter | RAVEN 2.11.3 | default (raven-toolbox and RAVEN 3) | basis | |---|---|---|---| -| `cutoff` (`cutOff`) | 1e-50 | 1e-30 | beyond matched p99 for `eco`, between p95 and p99 for `sce`, `cme` and `mge`; with `min_score_ratio_g` 0.9, `mge` recall 0.85 → 0.94 | -| `min_score_ratio_g` (`minScoreRatioG`) | 0.8 | 0.9 | from 0.80 to 0.95 it raised precision by 0.05 to 0.10 for a recall loss of 0.00 to 0.02; offsets part of the precision lost to the looser cut-off (inference, 0.9 not swept) | -| `min_score_ratio_ko` (`minScoreRatioKO`) | 0.3 | 0.3 | at `min_score_ratio_g` 0.80 and `cutoff` 1e-50, 0.0 to 0.5 raised precision by 0.00 to 0.03 and lowered recall by 0.00 to 0.01; kept at the RAVEN value | +| `cutoff` (`cutOff`) | 1e-50 | 1e-30 | Beyond matched p99 for `eco`, between p95 and p99 for `sce`, `cme` and `mge`. With `min_score_ratio_g` 0.9, the `mge` recall rises from 0.85 to 0.94 | +| `min_score_ratio_g` (`minScoreRatioG`) | 0.8 | 0.9 | From 0.80 to 0.95 it raised the precision by 0.05 to 0.10 for a recall loss of 0.00 to 0.02. It offsets part of the precision lost to the looser cut-off (inference, 0.9 not swept) | +| `min_score_ratio_ko` (`minScoreRatioKO`) | 0.3 | 0.3 | At `min_score_ratio_g` 0.80 and `cutoff` 1e-50, 0.0 to 0.5 raised the precision by 0.00 to 0.03 and lowered the recall by 0.00 to 0.01. Kept at the RAVEN value | -The defaults favour recall: the changed settings raised recall in every organism and lowered precision in three of the four. +The defaults favour recall: the changed settings raised the recall in every organism and lowered the precision in three of the four. ## Settings without a sweep -Two settings next to the cut-offs, `seq_identity` and the model-assembly flags, were not measured in the study. The reasons given for their values are not tested. `threads` was measured in one configuration. +Two settings next to the cut-offs, `seq_identity` and the model-assembly flags, were not measured in the study, and the reasons given for their values are not tested. The setting `threads` was measured in one configuration. -`threads` in `run_hmmsearch` and `build_ko_hmm` defaults to every CPU the process may run on. `hmmsearch` 3.4 of `sce` (6021 sequences) against `kegg118_eukaryotes.hmm` (2698 profiles) returned 10,293 hit rows for 2198 genes and 1998 KOs at 16, 12, 4 and 1 threads, and the parsed hit tables (KO, gene, E-value) were identical. One run per thread count took 89.7 s (16 threads), 88.6 s (12), 96.4 s (4) and 242.5 s (1), on an AMD EPYC 9354 node used exclusively (2026-11-01). Between 16 and 4 threads the time changes by less than 9 %. `build_ko_hmm` was not timed. +The setting `threads` in `run_hmmsearch` and `build_ko_hmm` defaults to every CPU that the process may run on. `hmmsearch` 3.4 of `sce` (6021 sequences) against `kegg118_eukaryotes.hmm` (2698 profiles) returned 10,293 hit rows for 2198 genes and 1998 KOs at 16, 12, 4 and 1 threads, and the parsed hit tables (KO, gene, E-value) were identical. One run per thread count took 89.7 s (16 threads), 88.6 s (12), 96.4 s (4) and 242.5 s (1), on an AMD EPYC 9354 node used exclusively (2026-11-01). Between 16 and 4 threads the time changes by less than 9 %. The function `build_ko_hmm` was not timed. -`seq_identity` in `build_ko_hmm` is `0.9`: the identity at which CD-HIT collapses near-duplicate sequences within a KO before the HMM is built; the clustering keeps a KO with many near-identical entries from giving an overfitted profile. It is CD-HIT's recommended value for proteins and the default of `seqIdentity` in RAVEN 2.11.3. `seq_identity=-1` skips the clustering. RAVEN 3's `getKEGGModelForOrganism` uses prebuilt HMM libraries and has no clustering step. +The setting `seq_identity` in `build_ko_hmm` is `0.9`. It is the identity at which CD-HIT collapses near-duplicate sequences within a KO before the HMM is built, and the clustering keeps a KO with many near-identical entries from giving an overfitted profile. It is the value that CD-HIT recommends for proteins and the default of `seqIdentity` in RAVEN 2.11.3. The value `seq_identity=-1` skips the clustering. The `getKEGGModelForOrganism` of RAVEN 3 uses prebuilt HMM libraries and has no clustering step. -The model-assembly flags decide which reactions a draft keeps, and have the same defaults as in RAVEN: +The model-assembly flags decide which reactions a draft keeps, and they have the same defaults as in RAVEN. | Flag | Default | What it admits | Reason given for the default | |---|---|---|---| @@ -212,9 +216,9 @@ The model-assembly flags decide which reactions a draft keeps, and have the same ## Limits -- All four organisms are in the set of sequences from which the libraries were built, so their own sequences contribute to their KO profiles. Recall on these organisms is therefore an upper bound for an organism outside KEGG (inference; a genome absent from KEGG was not tested). The comparison between settings is relative and gives no absolute accuracy. -- A novel pair that is a correct assignment KEGG has not made counts as an error, so the precision figures are a lower bound. The study does not measure how many novel pairs are correct. -- Four organisms, the KEGG tables of release 118 and one hit table per organism. For `sce` the parsed hit tables were identical in four runs at 16, 12, 4 and 1 threads; the other organisms were searched once. The scoring of a hit table by `assign_kos` involves no random step. +- All four organisms are in the set of sequences from which the libraries were built, so their own sequences contribute to their KO profiles. The recall on these organisms is therefore an upper bound for an organism outside KEGG (inference; a genome absent from KEGG was not tested). The comparison between settings is relative and gives no absolute accuracy. +- A novel pair that is a correct assignment that KEGG has not made counts as an error, so the precision figures are a lower bound. The study does not measure how many novel pairs are correct. +- The study used four organisms, the KEGG tables of release 118 and one hit table per organism. For `sce` the parsed hit tables were identical in four runs at 16, 12, 4 and 1 threads, and the other organisms were searched once. The scoring of a hit table by `assign_kos` involves no random step. - The score-ratio sweep used `cutoff = 1e-50` only, and `min_score_ratio_g = 0.9` was not part of either sweep. ## Reproduction @@ -229,8 +233,8 @@ The scripts are in [`scripts/parameter_tuning/kegg/`](https://github.com/CompMet | `compare_kegg_paths.py` | the whole-model comparison of the annotation and HMM paths | | `time_hmmsearch.py` | the timing and hit-table identity at several thread counts | -`analyze_hmm_cutoffs.py` is called once per organism with `--artefacts`, `--org`, `--proteome`, `--library` and `--threads`. `--save-tblout` writes the `hmmsearch --tblout` table, `--tblout` scores a saved table without searching, `--json` writes the results, and `--restrict-to-reference` limits `ko_reaction` to reactions of the reference model, which is the definition behind the reaction columns above. The gene→KO columns and the percentiles do not depend on that option. `compare_kegg_paths.py` is called with `--org`, `--proteome`, `--artefacts`, `--library`, `--domain`, `--threads` and `--settings`, for example `--settings 1e-50,0.3,0.8 1e-30,0.3,0.9`. +The script `analyze_hmm_cutoffs.py` is called once per organism with `--artefacts`, `--org`, `--proteome`, `--library` and `--threads`. The option `--save-tblout` writes the `hmmsearch --tblout` table, `--tblout` scores a saved table without searching, `--json` writes the results, and `--restrict-to-reference` limits `ko_reaction` to reactions of the reference model, which is the definition behind the reaction columns above. The gene→KO columns and the percentiles do not depend on that option. The script `compare_kegg_paths.py` is called with `--org`, `--proteome`, `--artefacts`, `--library`, `--domain`, `--threads` and `--settings`, for example `--settings 1e-50,0.3,0.8 1e-30,0.3,0.9`. -The proteomes (`kegg_rest/.faa`, `uniprot/.faa`, `counts.json`) are in the archived input bundle `kegg_proteomes_2026-11-01.tar.gz`, and the four `hmmsearch --tblout` tables in `kegg_hmmsearch_tblout_2026-11-01.tar.gz`. +The proteomes (`kegg_rest/.faa`, `uniprot/.faa`, `counts.json`) are in the archived input bundle `kegg_proteomes_2026-11-01.tar.gz`, and the four `hmmsearch --tblout` tables are in `kegg_hmmsearch_tblout_2026-11-01.tar.gz`. -The scores of the tables come from the run of 2026-11-01 with raven-toolbox commit `94f3940`, HMMER 3.4 and the published kegg118 tables and libraries, searched with 16 threads on an AMD EPYC 9354 node. Rescoring each saved hit table with `--tblout` gives the saved scores exactly. Not checked: whether the KEGG REST sequences of 2026-10-31 equal the sequences of KEGG release 118 (the sequence counts were compared, not the sequences), and whether `hmmsearch -Z` E-values equal `hmmscan` E-values on these inputs, because the binary bundle has no `hmmscan`. The UniProt proteomes in the bundle were downloaded and counted but not searched. +The scores of the tables come from the run of 2026-11-01 with raven-toolbox commit `94f3940`, HMMER 3.4 and the published kegg118 tables and libraries, searched with 16 threads on an AMD EPYC 9354 node. Scoring each saved hit table with `--tblout` reproduces the saved scores. Two points were not checked: whether the KEGG REST sequences of 2026-10-31 equal the sequences of KEGG release 118 (the sequence counts were compared, not the sequences), and whether `hmmsearch -Z` E-values equal `hmmscan` E-values on these inputs, because the binary bundle has no `hmmscan`. The UniProt proteomes in the bundle were downloaded and counted but not searched. diff --git a/docs/parameter-tuning/studies/sampling-convergence-calibration.md b/docs/parameter-tuning/studies/sampling-convergence-calibration.md index 8fe6e3c..dbab9ed 100644 --- a/docs/parameter-tuning/studies/sampling-convergence-calibration.md +++ b/docs/parameter-tuning/studies/sampling-convergence-calibration.md @@ -1,6 +1,8 @@ # Flux sampling convergence and settings -`random_sampling` in raven-toolbox draws flux vectors from a model with a Markov-chain sampler (`method='achr'` or `'chrr'`) or with random objectives (`method='random_objective'`). This study ran the two Markov-chain samplers on e_coli_core (95 reactions) and yeast-GEM, and recorded how strongly consecutive samples of one chain are correlated, whether independent chains reach the same flux distribution, and the wall time. It also records why two settings of the random-objective method differ from RAVEN MATLAB. +The function `random_sampling` of raven-toolbox draws flux vectors from a model with a Markov-chain sampler (`method='achr'` or `'chrr'`) or with random objectives (`method='random_objective'`). The study ran the two Markov-chain samplers on e_coli_core (95 reactions) and yeast-GEM and recorded three results: how strongly consecutive samples of one chain are correlated, whether independent chains reach the same flux distribution, and the wall time. It also records why two settings of the random-objective method differ from RAVEN in MATLAB. + +On e_coli_core, four chains of 300 samples agree at the R-hat threshold of 1.1 for 87 of 87 scored reactions, and the CHRR sampler was no better than ACHR there in single runs. On yeast-GEM, 50.7 % of the scored reactions exceed an R-hat of 1.1 for `n_samples=300` at the default `thinning=100`; the default `n_samples` of 1000 was not tested. No setting measured with R-hat or `ρ` gave converged samples. ## Terms @@ -10,26 +12,26 @@ - ACHR (Artificially Centered Hit-and-Run) takes its directions from warmup points and a running centre of the chain, and does not transform the polytope. raven-toolbox wraps `cobra.sampling.ACHRSampler`, which computes its warmup points with two linear programs (LPs) per reaction before the first step ([Random sampling](../../guide/sampling.md)). - CHRR (Coordinate Hit-and-Run with Rounding) first runs a flux variability analysis (FVA), fixes the reactions whose range is narrower than `fixed_width_tol`, reduces the polytope to a full-dimensional body through the nullspace of the equality constraints, and rounds it: it maps the body so that its maximum-volume inscribed ellipsoid (MVE) becomes a unit ball. The walk then moves along one coordinate per step. - `thinning` is the number of steps between two stored samples. An ACHR chain with `n_samples` stored samples therefore takes `thinning × n_samples` steps after its warmup points, and a CHRR chain takes `warmup + thinning × n_samples` steps after the rounding. -- `warmup` is the number of CHRR steps discarded before the first stored sample. `random_sampling` passes it to CHRR only; the ACHR runs ignore it. +- `warmup` is the number of CHRR steps discarded before the first stored sample. `random_sampling` passes it to CHRR only, and the ACHR runs ignore it. - The lag-1 autocorrelation `ρ` is the correlation between consecutive stored samples of one chain. - The effective sample size (ESS) is the number of independent samples that carry the same information as the correlated ones. Under the assumption that the chain is a first-order autoregressive process, `ESS = n × (1−ρ) / (1+ρ)` for `n` stored samples. - The R-hat of a reaction (Gelman-Rubin statistic) compares the flux variance between chains with the variance within chains. With `m` chains of `n` samples, `W` the mean within-chain variance and `B/n` the variance of the chain means, `R-hat = sqrt(((n−1)/n × W + B/n) / W)`. R-hat is 1.0 when the chains agree and grows when the distribution of a chain depends on where it started. The script reports the share of reactions above 1.1 and above 1.01, the stricter of the two thresholds. ## Set-up -Driver for the comparisons between chains: `scripts/analyze_sampling_convergence.py` in raven-toolbox. It starts `n_chains` calls of `random_sampling` with seeds 0 to `n_chains − 1`, one process per chain through `ProcessPoolExecutor`, because cobra models are not guaranteed to be safe for concurrent solves in threads. Each process reads the model from the SBML file. The script computes R-hat per reaction with the formula in [Terms](#terms). A reaction with the same constant flux in every chain has `W = 0` and no variance between the chain means, gets no R-hat, and is excluded as not scored. A reaction that is constant within each chain at different values in different chains has `W = 0` and an infinite R-hat, and is scored. +The driver for the comparisons between chains is `scripts/analyze_sampling_convergence.py` in raven-toolbox. It starts `n_chains` calls of `random_sampling` with seeds 0 to `n_chains − 1`, one process per chain through `ProcessPoolExecutor`, because cobra models are not guaranteed to be safe for concurrent solves in threads. Each process reads the model from the SBML file. The script computes R-hat per reaction with the formula in [Terms](#terms). A reaction with the same constant flux in every chain has `W = 0` and no variance between the chain means, gets no R-hat, and is excluded as not scored. A reaction that is constant within each chain at different values in different chains has `W = 0` and an infinite R-hat, and is scored. -Settings: `n_samples=300`, `thinning=100`, `warmup=1000` and 4 chains, unless a table states otherwise. `thinning` and `warmup` are at their defaults; the default `n_samples` is 1000. The solver was Gurobi, with 12 cores per job on AMD EPYC 9354 nodes. All figures are from runs on 2026-11-01 with Python 3.14.2, cobra 0.32.1, optlang 1.9.1, gurobipy 13.0.3 (Gurobi library 13.0.3) and raven-toolbox `94f3940`. Each configuration ran once, except e_coli_core ACHR, which was repeated; the single-chain runs also ran with seeds 1 to 3 (see [Autocorrelation within one chain](#autocorrelation-within-one-chain)). cobrapy's `OptGPSampler` was not tested, because `random_sampling` does not call it; its methods are `'achr'`, `'chrr'` and `'random_objective'`. +The settings were `n_samples=300`, `thinning=100`, `warmup=1000` and 4 chains, unless a table states otherwise. The values of `thinning` and `warmup` are the defaults, and the default `n_samples` is 1000. The solver was Gurobi, with 12 cores per job on AMD EPYC 9354 nodes. All figures are from runs on 2026-11-01 with Python 3.14.2, cobra 0.32.1, optlang 1.9.1, gurobipy 13.0.3 (Gurobi library 13.0.3) and raven-toolbox `94f3940`. Each configuration ran once, except e_coli_core ACHR, which was repeated. The single-chain runs also ran with seeds 1 to 3 (see [Autocorrelation within one chain](#autocorrelation-within-one-chain)). The `OptGPSampler` of cobrapy was not tested, because `random_sampling` does not call it, and its methods are `'achr'`, `'chrr'` and `'random_objective'`. -The models are two files of yeast-GEM and one of e_coli_core. The single-chain runs read the bundled `docs/data/yeast-GEM.xml` (yeastGEM_v9.1.0, 4,102 reactions). The comparisons between chains read yeast-GEM v9.1.1 (4,105 reactions), which differs from the develop branch at commit `ea756c2` only in the model identifier and two dates. e_coli_core is `textbook.xml.gz` of cobrapy 0.32.1 (model identifier `e_coli_core`). +The models are two files of yeast-GEM and one of e_coli_core. The single-chain runs read the bundled `docs/data/yeast-GEM.xml` (yeastGEM_v9.1.0, 4,102 reactions). The comparisons between chains read yeast-GEM v9.1.1 (4,105 reactions), which differs from the develop branch at commit `ea756c2` only in the model identifier and two dates. The model e_coli_core is `textbook.xml.gz` of cobrapy 0.32.1 (model identifier `e_coli_core`). -The wall time of a comparison between chains is the time from starting the processes to the return of the last chain: reading the model, the warmup and the sampling, without the R-hat computation. The wall time of a single-chain run is the time of reading the SBML file plus `random_sampling`. The `ρ` of a chain is the mean over the reactions with a flux range above `1e-6` in that chain of the lag-1 Pearson correlation of the reaction (`pandas.Series.autocorr(1)`); the median over the reactions is higher and is given in the text. The ESS uses the mean. +The wall time of a comparison between chains is the time from starting the processes to the return of the last chain: reading the model, the warmup and the sampling, without the R-hat computation. The wall time of a single-chain run is the time of reading the SBML file plus `random_sampling`. The `ρ` of a chain is the mean over the reactions with a flux range above `1e-6` in that chain of the lag-1 Pearson correlation of the reaction (`pandas.Series.autocorr(1)`). The median over the reactions is higher and is given in the text. The ESS uses the mean. ## Results ### Autocorrelation within one chain -One ACHR chain on yeast-GEM (4,102 reactions), `n_samples=300`, seed 0, at three values of `thinning`. The ESS column is computed from the formula in [Terms](#terms). +The runs used one ACHR chain on yeast-GEM (4,102 reactions), `n_samples=300`, seed 0, at three values of `thinning`. The ESS column is computed from the formula in [Terms](#terms). | `thinning` | Lag-1 autocorrelation | ESS of 300 stored samples | Wall time (s) | |---|---|---|---| @@ -47,15 +49,21 @@ The same settings with seeds 0 to 3 give the following `ρ`. | 100 | 0.9361 | 0.6523 | 0.9238 | 0.9286 | 0.652 to 0.936 | | 500 | 0.8251 | 0.6444 | 0.8013 | 0.7694 | 0.644 to 0.825 | -For every seed, `ρ` falls as `thinning` rises. Between seeds, `ρ` differs by up to 0.28, and the ESS of 300 stored samples ranges from 3.2 to 11.8 at `thinning=20`, from 9.9 to 63.1 at 100 and from 28.8 to 64.9 at 500. Seed 1 gives the lowest `ρ` at all three values; without it, the ranges are 0.972 to 0.979, 0.924 to 0.936 and 0.769 to 0.825. The four chains of the yeast-GEM comparison with 4,105 reactions (seeds 0 to 3, `thinning=100`) give a mean `ρ` of 0.951, 0.847, 0.947 and 0.911, so a low value for seed 1 appears on both model files. The spread between seeds is 0.10 on the 4,105-reaction chains (0.847 to 0.951) and 0.28 on the 4,102-reaction chains (0.652 to 0.936); seed 1 is the lowest in both. +For every seed, `ρ` falls as `thinning` rises. Between seeds, `ρ` differs by up to 0.28, and the ESS of 300 stored samples ranges from 3.2 to 11.8 at `thinning=20`, from 9.9 to 63.1 at 100 and from 28.8 to 64.9 at 500. Seed 1 gives the lowest `ρ` at all three values, and without it the ranges are 0.972 to 0.979, 0.924 to 0.936 and 0.769 to 0.825. The four chains of the yeast-GEM comparison with 4,105 reactions (seeds 0 to 3, `thinning=100`) give a mean `ρ` of 0.951, 0.847, 0.947 and 0.911, so a low value for seed 1 appears on both model files. The spread between seeds is 0.10 on the 4,105-reaction chains (0.847 to 0.951) and 0.28 on the 4,102-reaction chains (0.652 to 0.936), and seed 1 is the lowest in both. + +The value `n_samples` was not varied. With `ρ` fixed at its seed-0 value for the default `thinning=100`, the ESS formula is proportional to `n_samples`: about 10 from 300 stored samples, about 33 from the default 1000, and 100 needs about 3,000 stored samples. Building the ACHR warmup points alone took 185.9 s, 196.6 s and 195.4 s in the three runs, so the sampling steps took about 1 s, 5 s and 26 s for 6,000, 30,000 and 150,000 steps. + +The warmup is therefore about 95 % of the wall time at `thinning=100`, and the wall time rose from 189.5 s to 223.3 s (1.2-fold) while the number of steps rose 25-fold. + +With a fixed cost per run plus a cost per stored sample, 3,025 stored samples take about 242 s for one chain. Scaling the 203.6 s in proportion to 3,025 samples gives 2,053 s (about 34 min), an upper bound. -`n_samples` was not varied. With `ρ` fixed at its seed-0 value for the default `thinning=100`, the ESS formula is proportional to `n_samples`: ~10 from 300 stored samples, ~33 from the default 1000, and 100 needs ~3,000 stored samples. Building the ACHR warmup points alone took 185.9 s, 196.6 s and 195.4 s in the three runs, so the sampling steps took about 1 s, 5 s and 26 s for 6,000, 30,000 and 150,000 steps. The warmup is therefore ~95 % of the wall time at `thinning=100`, and the wall time rose from 189.5 s to 223.3 s (1.2-fold) while the number of steps rose 25-fold. With a fixed cost per run plus a cost per stored sample, 3,025 stored samples take ~242 s for one chain; scaling the 203.6 s in proportion to 3,025 samples gives 2,053 s (~34 min), an upper bound. The warmup time was measured on a second, fresh sampler after the run, so the split is approximate. +The warmup time was measured on a second, fresh sampler after the run, so the split is approximate. -For the choice of settings: within the measured range, a larger `thinning` gave more ESS per second of wall time (4.2 in 189.5 s at 20, 28.8 in 223.3 s at 500). That a larger `n_samples` at a fixed `thinning` raises ESS in proportion and leaves `ρ` unchanged follows from the first-order autoregressive formula; it was not measured. +Within the measured range, a larger `thinning` gave more ESS per second of wall time (4.2 in 189.5 s at 20, 28.8 in 223.3 s at 500). That a larger `n_samples` at a fixed `thinning` raises the ESS in proportion and leaves `ρ` unchanged follows from the first-order autoregressive formula, and it was not measured. ### Agreement between chains on e_coli_core -ACHR, 4 chains × 300 samples, 10.9 s wall time (5.4 s in a repeat). The repeat gave the same R-hat for every reaction, so on e_coli_core a chain is deterministic for a given seed and software. Reproducibility on yeast-GEM was not tested; the `random_objective` run on yeast-GEM is not reproducible (see [Other sampling settings](#other-sampling-settings)). The difference between 10.9 s and 5.4 s is unexplained. +The ACHR run used 4 chains × 300 samples and took 10.9 s wall time (5.4 s in a repeat). The repeat gave the same R-hat for every reaction, so on e_coli_core a chain is deterministic for a given seed and software. Reproducibility on yeast-GEM was not tested, and the `random_objective` run on yeast-GEM is not reproducible (see [Other sampling settings](#other-sampling-settings)). The difference between 10.9 s and 5.4 s is unexplained. | Measure | Value | |---|---:| @@ -66,7 +74,7 @@ ACHR, 4 chains × 300 samples, 10.9 s wall time (5.4 s in a repeat). The repeat | reactions with R-hat > 1.01 | 12 (13.8 %) | | reactions with R-hat > 1.1 | 0 (0.0 %) | -Reactions with the highest R-hat: +The reactions with the highest R-hat were: | Reaction | R-hat | |---|---:| @@ -81,11 +89,11 @@ Reactions with the highest R-hat: No reaction exceeded 1.1, and 12 of 87 scored reactions exceeded 1.01. The reaction with the highest R-hat is the succinate exchange `EX_succ_e`. -For the choice of settings: on a 95-reaction model, the default `thinning` with 300 samples per chain gave agreement at the 1.1 threshold for 87 of 87 scored reactions and at 1.01 for 75 of 87. Agreement at 1.01 for the remaining 12 needs more steps or a different method; CHRR did not remove it (see [CHRR on e_coli_core](#chrr-on-e_coli_core)). +On a 95-reaction model, the default `thinning` with 300 samples per chain gave agreement at the 1.1 threshold for 87 of 87 scored reactions and at 1.01 for 75 of 87. Agreement at 1.01 for the remaining 12 needs more steps or another method, and CHRR did not remove it (see [CHRR on e_coli_core](#chrr-on-e_coli_core)). ### Agreement between chains on yeast-GEM -ACHR, 4 chains × 300 samples, 572.3 s wall time (~9.5 min). +The ACHR run used 4 chains × 300 samples and took 572.3 s wall time (about 9.5 min). | Measure | Value | |---|---:| @@ -96,17 +104,17 @@ ACHR, 4 chains × 300 samples, 572.3 s wall time (~9.5 min). | reactions with R-hat > 1.01 | 3167 (95.2 %) | | reactions with R-hat > 1.1 | 1685 (50.7 %) | -Reactions with the highest R-hat: `r_4562` (18.90), `r_4358` (18.90), `r_0992` (5.18), `r_4249` (5.18), `r_4701` (5.18), `r_4621` (5.18), `r_1965` (5.09), `r_0917` (3.90), `r_0891` (3.90), `r_0918` (3.90). +The reactions with the highest R-hat were `r_4562` (18.90), `r_4358` (18.90), `r_0992` (5.18), `r_4249` (5.18), `r_4701` (5.18), `r_4621` (5.18), `r_1965` (5.09), `r_0917` (3.90), `r_0891` (3.90) and `r_0918` (3.90). -The median scored reaction was 1.1020, and one in two reactions (50.7 %) exceeded 1.1; 4.8 % passed 1.01. Each chain took 300 × 100 = 30,000 ACHR steps after its warmup points. The low ESS of one chain and the disagreement between chains are two separate measurements, and both show that these settings do not give a converged sample of yeast-GEM. +The median scored reaction was 1.1020, and one in two reactions (50.7 %) exceeded 1.1, and 4.8 % were at or below 1.01. Each chain took 300 × 100 = 30,000 ACHR steps after its warmup points. The low ESS of one chain and the disagreement between chains are two separate measurements, and both show that these settings do not give a converged sample of yeast-GEM. -The four parallel chains took 572.3 s, 2.8 times the 203.6 s of the single chain at the same settings. The two runs read different yeast-GEM files (4,105 and 4,102 reactions). Contention between the four Gurobi processes on the 12 cores is a possible cause of the difference; it was not measured. +The four parallel chains took 572.3 s, 2.8 times the 203.6 s of the single chain at the same settings. The two runs read different yeast-GEM files (4,105 and 4,102 reactions). Contention between the four Gurobi processes on the 12 cores is a possible cause of the difference, and it was not measured. -For the choice of settings: on yeast-GEM with `n_samples=300` and `thinning=100`, 50.7 % of the 3325 scored reactions had R-hat above 1.1. The default `n_samples=1000` (100,000 steps per chain) was not tested. +On yeast-GEM with `n_samples=300` and `thinning=100`, 50.7 % of the 3325 scored reactions had R-hat above 1.1. The default `n_samples=1000` (100,000 steps per chain) was not tested. ### Reallocating the ACHR step budget -The 30,000 steps per chain of the default run (`thinning=100, n_samples=300`), spread as `thinning=300, n_samples=100`. ACHR, 4 chains, yeast-GEM. +The 30,000 steps per chain of the default run (`thinning=100, n_samples=300`) were spread as `thinning=300, n_samples=100`. The runs used ACHR, 4 chains and yeast-GEM. | Measure | t=100, n=300 (default thinning) | t=300, n=100 (reallocated) | |---|---:|---:| @@ -118,13 +126,11 @@ The 30,000 steps per chain of the default run (`thinning=100, n_samples=300`), s | R-hat > 1.1 | 50.7 % | 50.5 % | | worst reaction | `r_4562`, `r_4358` (18.90) | `r_4562`, `r_4358` (21.91) | -Both runs scored 3325 reactions. The reallocation moved the median R-hat from 1.1020 to 1.1010 and the shares above the two thresholds by at most 1.7 percentage points, the reactions with the highest R-hat were the same, and the run took 612.4 s against 572.3 s (1.07 times). That the disagreement comes from how ACHR moves through this polytope, and not from the split of the steps, is an inference from this result; it was not tested at genome scale. - -For the choice of settings: at a fixed number of steps, moving steps from `n_samples` to `thinning` did not improve agreement between chains. A larger total number of steps was not tested with R-hat. +Both runs scored 3325 reactions. The reallocation moved the median R-hat from 1.1020 to 1.1010 and the shares above the two thresholds by at most 1.7 percentage points. The reactions with the highest R-hat were the same, and the run took 612.4 s against 572.3 s (1.07 times). That the disagreement comes from how ACHR moves through this polytope, and not from the split of the steps, is an inference from this result, and it was not tested on other models. At a fixed number of steps, moving steps from `n_samples` to `thinning` did not improve the agreement between chains, and a larger total number of steps was not tested with R-hat. ### CHRR on e_coli_core -4 chains × 300 samples on e_coli_core, with `method='achr'` and `method='chrr'`. +The runs used 4 chains × 300 samples on e_coli_core, with `method='achr'` and `method='chrr'`. | Measure | ACHR | CHRR | |---|---:|---:| @@ -136,37 +142,41 @@ For the choice of settings: at a fixed number of steps, moving steps from `n_sam | R-hat > 1.01 | 13.8 % (12) | 14.7 % (14) | | R-hat > 1.1 | 0.0 % | 0.0 % | -With either method, no reaction exceeded 1.1. CHRR did not agree better than ACHR on e_coli_core: its highest R-hat was `EX_succ_e` at 1.0377 against 1.0276 for ACHR, and 14.7 % of its reactions exceeded 1.01 against 13.8 %. CHRR took 66 times the wall time of ACHR (133 times that of the 5.4 s repeat), one run each. The percentages have different denominators, 87 and 95 scored reactions. The 8 reactions that ACHR left constant (all exactly 0.0) were scored with CHRR. CHRR computes every flux of a sample from the nullspace basis (`v0 + N x`); these 8 reactions vary within a chain by at most 1.6e-13, with means of the order of 1e-14, which is the size of floating-point rounding error in that product. +With either method, no reaction exceeded 1.1. CHRR did not agree better than ACHR on e_coli_core: its highest R-hat was `EX_succ_e` at 1.0377 against 1.0276 for ACHR, and 14.7 % of its reactions exceeded 1.01 against 13.8 %. CHRR took 66 times the wall time of ACHR (133 times that of the 5.4 s repeat), with one run each. The percentages have different denominators, 87 and 95 scored reactions. The 8 reactions that ACHR left constant (all exactly 0.0) were scored with CHRR. CHRR computes every flux of a sample from the nullspace basis (`v0 + N x`), and these 8 reactions vary within a chain by at most 1.6e-13, with means of the order of 1e-14, which is the size of the floating-point rounding error in that product. -For the choice of settings: on a 95-reaction model, ACHR and CHRR with the default `thinning` and 300 samples per chain both gave agreement at 1.1 for every reaction, and CHRR cost 66 times as much time (133 times against the repeat) without lowering the share above 1.01. +On a 95-reaction model, ACHR and CHRR with the default `thinning` and 300 samples per chain both gave agreement at 1.1 for every reaction, and CHRR cost 66 times as much time (133 times against the repeat) without lowering the share above 1.01. ### CHRR at genome scale -CHRR was not run on yeast-GEM with 4 chains × 300 samples. A probe with 2 chains × 20 samples (`thinning=100`) took 8147.9 s (~136 min), 14.2 times the 572.3 s of the full ACHR run, with a peak memory of about 31 GB of the 32 GB allocated. Each chain runs an FVA with its own worker pool; that the pools cause the memory use was not tested. yeast-GEM has 43 times the reactions of e_coli_core. +CHRR was not run on yeast-GEM with 4 chains × 300 samples. A probe with 2 chains × 20 samples (`thinning=100`) took 8147.9 s (about 136 min), 14.2 times the 572.3 s of the full ACHR run, with a peak memory of about 31 GB of the 32 GB allocated. Each chain runs an FVA with its own worker pool, and that the pools cause the memory use was not tested. The model yeast-GEM has 43 times the reactions of e_coli_core. -Each CHRR chain ran an FVA, a nullspace computation and the MVE rounding before its first step, and then 1,000 warmup steps and 2,000 sampling steps. The probe does not separate these parts. That the cost before the first step dominates, and that most of it is the MVE rounding, is an inference from the small number of steps; the parts were not timed. `random_sampling` computes this transform anew in every call, so each chain pays it. +Each CHRR chain ran an FVA, a nullspace computation and the MVE rounding before its first step, and then 1,000 warmup steps and 2,000 sampling steps. The probe does not separate these parts. That the cost before the first step dominates, and that most of it is the MVE rounding, is an inference from the small number of steps, and the parts were not timed. The function `random_sampling` computes this transform anew in every call, so each chain pays it. -The R-hat of the probe (median 5.7274, p90 165.18, max 1.899e9 for `r_4782`; all 4105 reactions scored, 97.3 % above 1.01 and 93.6 % above 1.1) does not show that CHRR converges worse than ACHR at genome scale. With 20 samples per chain, the within-chain variance `W` is estimated from too few samples to be stable: a reaction that shows almost no variance in 20 draws by chance, combined with any difference between the chain means (`B/n`), gives a large R-hat whatever its distribution. The probe cannot distinguish a CHRR that does not converge on yeast-GEM from an R-hat that needs more samples; a run with enough samples for a stable R-hat has the cost problem of the full run. +The R-hat of the probe (median 5.7274, p90 165.18, max 1.899e9 for `r_4782`; all 4105 reactions scored, 97.3 % above 1.01 and 93.6 % above 1.1) does not show that CHRR converges worse than ACHR at genome scale. With 20 samples per chain, the within-chain variance `W` is estimated from too few samples to be stable. A reaction that shows almost no variance in 20 draws by chance, combined with any difference between the chain means (`B/n`), gives a large R-hat whatever its distribution. The probe cannot distinguish a CHRR that does not converge on yeast-GEM from an R-hat that needs more samples, and a run with enough samples for a stable R-hat has the cost problem of the full run. -For the choice of settings: 20 samples per chain are too few for R-hat on yeast-GEM. Whether CHRR converges on yeast-GEM with 300 samples per chain is open. +Twenty samples per chain are too few for R-hat on yeast-GEM, and whether CHRR converges on yeast-GEM with 300 samples per chain is open. ## Choice of `thinning` and `n_samples` -The defaults are `method='achr'`, `thinning=100` and `n_samples=1000` ([Tuned parameter defaults](../../tuned-parameters.md)). CHRR is not the default: on e_coli_core, CHRR differed from ACHR by +0.0012 in median R-hat (1.0027 and 1.0039), by −0.0034 in p90 (1.0171 and 1.0137) and by +0.0101 in maximum (1.0276 and 1.0377), with one run each and no measured spread between seed sets, and CHRR took 66 to 133 times as long. On a model the size of e_coli_core, the default `thinning` with 300 samples per chain gave chains that agree at R-hat 1.1 for every reaction. On yeast-GEM, no setting tested gave converged samples: a `thinning` up to 500 left `ρ` at 0.825 for seed 0 (0.644 to 0.825 over four seeds), and moving the same 30,000 steps from `n_samples` to `thinning` moved the median R-hat from 1.1020 to 1.1010 and the shares above 1.01 and 1.1 by at most 1.7 percentage points. On yeast-GEM with `n_samples=300` and `thinning=100`, 50.7 % of the 3325 scored reactions had R-hat above 1.1. +The defaults are `method='achr'`, `thinning=100` and `n_samples=1000` ([Tuned parameter defaults](../../tuned-parameters.md)). ACHR is the default because on e_coli_core CHRR differed from ACHR by +0.0012 in median R-hat (1.0027 and 1.0039), by −0.0034 in p90 (1.0171 and 1.0137) and by +0.0101 in maximum (1.0276 and 1.0377), with one run each and no measured spread between sets of seeds, and CHRR took 66 to 133 times as long. + +On a model the size of e_coli_core, the default `thinning` with 300 samples per chain gave chains that agree at R-hat 1.1 for every scored reaction. On yeast-GEM, no setting measured with R-hat or `ρ` gave converged samples. A `thinning` up to 500 left `ρ` at 0.825 for seed 0 (0.644 to 0.825 over four seeds), and moving the same 30,000 steps from `n_samples` to `thinning` moved the median R-hat from 1.1020 to 1.1010 and the shares above 1.01 and 1.1 by at most 1.7 percentage points. With `n_samples=300` and `thinning=100`, 50.7 % of the 3325 scored reactions had R-hat above 1.1. -The `random_sampling` docstring recommends, for genome-scale models, a `thinning` of at least 1000 or a larger `n_samples`, `method='chrr'`, or an ESS or R-hat check. A `thinning` of 1000 or more was not measured. The driver script runs the R-hat check: it samples several chains with different seeds at given settings and reports the share of reactions above the R-hat thresholds, so a sampling result can be checked at the settings that produced it. +The docstring of `random_sampling` recommends, for genome-scale models, a `thinning` of at least 1000 or a larger `n_samples`, `method='chrr'`, or an ESS or R-hat check. A `thinning` of 1000 or more was not measured. The driver script runs the R-hat check: it samples several chains with different seeds at given settings and reports the share of reactions above the R-hat thresholds, so a sampling result can be checked at the settings that produced it. One change outside the parameters was identified and not pursued: CHRR recomputes its rounding transform in every call. The chains run in separate processes at the same time, so to save that cost the transform would have to be computed once per model and passed to every chain. The saving depends on the share of the rounding in the 136 min of the probe, which was not measured. ## Other sampling settings -Two settings of `method='random_objective'` differ from RAVEN MATLAB's `randomSampling` on purpose. +Two settings of `method='random_objective'` differ from `randomSampling` of RAVEN in MATLAB on purpose. + +The setting `replace_max_bound` (default `False`; `replaceBoundsWithInf` in MATLAB defaults to true) replaces the largest upper bound, the big-M value that stands for an unconstrained reaction, with infinity before sampling, and replaces the most negative lower bound, if it is below 0, with minus infinity. Models in the RAVEN convention use 1000 as the big-M for almost every reaction (4,083 of the 4,102 reactions of yeast-GEM). With `True`, the call fails after 9.6 s with `OptimizationError: Solver status is 'unbounded'`. A probe of 50 random objectives, drawn as the sampler draws them on the model with the bounds replaced, gave 49 optimal results and 1 unbounded result (one probe of 50 objectives). One unbounded objective stops all draws. The abort mechanism is read from the cobra code, where `Model.optimize` raises on the status `unbounded` (`raise_error=False` does not change this) and the retry loop of the sampler never sees the status. The sample at which the run aborted was not recorded. -`replace_max_bound` (default `False`; MATLAB's `replaceBoundsWithInf` defaults to true) replaces the largest upper bound, the big-M value that stands for an unconstrained reaction, with infinity before sampling, and replaces the most negative lower bound, if it is below 0, with minus infinity. RAVEN-convention models use 1000 as the big-M for almost every reaction (4,083 of the 4,102 reactions of yeast-GEM). With `True`, the call fails after 9.6 s with `OptimizationError: Solver status is 'unbounded'`. A probe of 50 random objectives, drawn as the sampler draws them on the model with the bounds replaced, gave 49 optimal results and 1 unbounded result (one probe of 50 objectives). One unbounded objective stops all draws: the abort mechanism is read from the cobra code, where `Model.optimize` raises on the status `unbounded` (`raise_error=False` does not change this) and the retry loop of the sampler never sees the status. The sample at which the run aborted was not recorded. With `False`, 200 samples completed (68.2 s with 12 processes, including the loopless FVA). Counting a flux as at the bound when `|v| ≥ 1000 × (1 − 1e-9)`, the 200 × 4102 sample matrix (820,400 entries) had 54 entries at the bound (0.0066 %) in one run, with 15 samples containing at least one such reaction and 23 reactions (0.56 % of 4102) at the bound in at least one sample. A second run had 269 entries (0.033 %), 16 samples (8.0 %) and 32 reactions (0.78 %). A share of 0.56 % of reactions at the bound occurred in one run and 0.78 % in another; the share of entries is 0.0066 to 0.033 %. +With `False`, 200 samples completed (68.2 s with 12 processes, including the loopless FVA). Counting a flux as at the bound when `|v| ≥ 1000 × (1 − 1e-9)`, the 200 × 4102 sample matrix (820,400 entries) had 54 entries at the bound (0.0066 %) in one run, with 15 samples containing at least one such reaction and 23 reactions (0.56 % of 4102) at the bound in at least one sample. A second run had 269 entries (0.033 %), 16 samples (8.0 %) and 32 reactions (0.78 %). The share of reactions at the bound was 0.56 % in one run and 0.78 % in the other, and the share of entries is 0.0066 to 0.033 %. -`random_objective` with a fixed seed is not reproducible on yeast-GEM: the loopless FVA returned 2307 and 2308 eligible reactions in two runs with the same seed. That a different eligible set shifts every random draw is an inference from these two counts. +The method `random_objective` with a fixed seed is not reproducible on yeast-GEM: the loopless FVA returned 2307 and 2308 eligible reactions in two runs with the same seed. That a different eligible set shifts every random draw is an inference from these two counts. -`loopless_good_reactions` (default `True`) selects which reactions are eligible as random objectives. MATLAB excludes every reaction whose FVA maximum is infinite or above 0.999 times the largest upper bound (999 on a model capped at 1000), and, when the smallest lower bound is negative, every reaction whose FVA minimum is minus infinity or below 0.999 times that bound; it treats proximity to the big-M as a sign of a thermodynamic loop. It also excludes reactions that cannot carry a non-zero flux and, in an ecModel, the `usage_prot_` reactions. That rule also excludes reactions that reach the bound for metabolic reasons. A loopless FVA excludes only the reactions that take part in infeasible cycles, at a higher cost. +The setting `loopless_good_reactions` (default `True`) selects which reactions are eligible as random objectives. MATLAB excludes every reaction whose FVA maximum is infinite or above 0.999 times the largest upper bound (999 on a model capped at 1000), and, when the smallest lower bound is negative, every reaction whose FVA minimum is minus infinity or below 0.999 times that bound. It treats proximity to the big-M as a sign of a thermodynamic loop. It also excludes reactions that cannot carry a non-zero flux and, in an ecModel, the `usage_prot_` reactions. That rule also excludes reactions that reach the bound for metabolic reasons. A loopless FVA excludes only the reactions that take part in infeasible cycles, at a higher cost. ## Limits @@ -193,4 +203,4 @@ python scripts/parameter_tuning/sampling/random_objective_bounds.py \ --n-proc 12 --n-probe 50 --out results/ ``` -The CHRR runs use the same command with `--method chrr`. The script caches the chains per model, method, `n_chains`, `n_samples`, `thinning` and `warmup` under `--out`; a run with a cached configuration computes R-hat from the stored chains without sampling. Another `--thinning` or `--n-samples` samples again and is cached separately. On yeast-GEM, four ACHR chains in parallel on 12 cores took 2.8 times the wall time of one chain at the same settings; the two runs read different yeast-GEM files and the cause of the difference was not measured (see [Agreement between chains on yeast-GEM](#agreement-between-chains-on-yeast-gem)). A sweep at genome scale can therefore take more time than one chain per configuration suggests. +The CHRR runs use the same command with `--method chrr`. The script caches the chains per model, method, `n_chains`, `n_samples`, `thinning` and `warmup` under `--out`, and a run with a cached configuration computes R-hat from the stored chains without sampling. Another `--thinning` or `--n-samples` samples again and is cached separately. On yeast-GEM, four ACHR chains in parallel on 12 cores took 2.8 times the wall time of one chain at the same settings. The two runs read different yeast-GEM files, and the cause of the difference was not measured (see [Agreement between chains on yeast-GEM](#agreement-between-chains-on-yeast-gem)). A sweep at genome scale can therefore take more time than one chain per configuration suggests. diff --git a/docs/raven3-migration.md b/docs/raven3-migration.md index 35990a0..f9c36bc 100644 --- a/docs/raven3-migration.md +++ b/docs/raven3-migration.md @@ -1,69 +1,54 @@ # RAVEN 2 to RAVEN 3 -RAVEN 3 changes the MATLAB toolbox in four areas: the folder structure with its removed and renamed functions, backward-incompatible behaviour, file formats and reconstruction pipelines, and new functionality with its dependencies. The calling convention and an upgrade checklist for users of RAVEN 2 follow. The page was compared with RAVEN 2.11.3. The comparison with the Python package is in [Comparison of RAVEN and raven-toolbox](raven3-vs-raven-toolbox.md). +RAVEN 3 is a breaking release. A script written for RAVEN 2 can stop with an error, or run and return a different model. The sections below list what to check first, give the calling convention for optional arguments and provide a checklist for the upgrade. Four detail pages hold the lists. -RAVEN 3 is a deliberate breaking release. The codebase was reorganised, low-value and duplicate functions were removed, several external dependencies were dropped and the calling convention for optional arguments changed in the 112 functions that use `parseRAVENargs`. The rationale is in the [v3 development review](https://github.com/CompMetEng/RAVEN/blob/develop/docs/v3_review.md) of the project. Most non-trivial RAVEN 2 scripts need to be checked; the core reconstruction and analysis behaviour is mostly the same. The changes that can affect a script are listed in [Backward-incompatible changes](raven3-migration/backward-incompatible.md#backward-incompatible-changes). +All statements about RAVEN 2 refer to release 2.11.3. The last RAVEN 2 release, 2.11.4, differs from 2.11.3 by bug fixes in `ftINIT`, `mergeLinear`, `replaceMets`, `checkInstallation`, `importModel`, `readYAMLmodel` and `optimizeProb`. The changelog of RAVEN 3 states its changes relative to RAVEN 2.9.3, so an entry in it can describe a difference that does not exist against 2.11.3. The comparison with the Python package is in [Comparison of RAVEN and raven-toolbox](raven3-vs-raven-toolbox.md). -:::{admonition} Changes to check first -:class: important +The codebase was reorganised, functions were removed (counts per folder are in [Structure, removals and renames](raven3-migration/structure-and-renames.md)), several external dependencies were dropped, and 112 functions read their optional arguments through `parseRAVENargs`. The rationale is in the [v3 development review](https://github.com/CompMetEng/RAVEN/blob/develop/docs/v3_review.md). Scripts that call reconstruction, KEGG, localisation or gene-deletion functions need a check. The changes that can affect any script are in [Backward-incompatible changes](raven3-migration/backward-incompatible.md#backward-incompatible-changes). -Two changes are most likely to affect existing scripts: +:::{admonition} Changes to check first +:class: warning -- The same homology-based or KEGG-based reconstruction call returns a different draft model, without an error (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)). -- WoLF PSORT-based localisation scores raise an error and have no in-toolbox migration path (see [Backward-incompatible changes](raven3-migration/backward-incompatible.md#backward-incompatible-changes)). +- A homology-based or KEGG-based reconstruction call returns a different draft model without an error (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)). +- A script that uses WoLF PSORT localisation scores raises an error (see [Backward-incompatible changes](raven3-migration/backward-incompatible.md#backward-incompatible-changes)). ::: ---- - (calling-convention)= -## Calling convention: positional or named arguments +## Calling convention -RAVEN 2 functions took optional arguments strictly positionally (`f(model, a, b, c)`). A later optional argument could be set only by specifying all optional arguments before it. +A RAVEN 2 function took its optional arguments by position, as in `f(model, a, b, c)`, and setting a later optional argument required every earlier one. A position N in these pages is the position among all arguments of the call, required ones included. In `f(model, a, b, c)` the model is position 1 and `a` is position 2. -RAVEN 3 introduces `utils/parseRAVENargs.m`, called by 112 functions. Required leading arguments stay explicit positional parameters; all optional arguments are collected into `varargin` and can be supplied in three interchangeable ways: +In RAVEN 3 the function `utils/parseRAVENargs.m` reads the arguments of 112 functions. The required leading arguments are positional parameters. The optional arguments are collected in `varargin` and can be given by position, by name or by both: ```matlab -removeReactions(model, rxnList, true, true, true) % positional (RAVEN 2 style, still works) +removeReactions(model, rxnList, true, true, true) % positional (RAVEN 2 style) removeReactions(model, rxnList, "removeUnusedGenes", true) % named removeReactions(model, rxnList, true, "removeUnusedComps", true) % hybrid ``` -The first string argument that matches a parameter name starts the name-value pairs. The arguments before it are positional, assigned in the order declared in the `parseRAVENargs` specification, and everything from it on is read as name-value pairs. If no argument matches, or the arguments from the first match on are not valid name-value pairs, all arguments are positional. - -Old positional calls keep working if the new parameter order still matches the old positional order. In 11 functions (`importModel`, `exportModel`, `exportToExcelFormat`, `removeReactions`, `ftINIT`, `fillGaps`, `fitTasks`, `getModelFromHomology`, `addIdentifierPrefix`, `removeIdentifierPrefix` and `exportForGit`) new optional parameters follow the RAVEN 2 ones, so old positional calls keep working unchanged. The old order is kept in the 11 functions checked above. A comparison of all 112 `parseRAVENargs` specifications with the RAVEN 2.11.3 signatures found these exceptions: +The first string argument that matches a parameter name, ignoring case, starts the name-value pairs. The arguments before it are positional and are assigned in the order of the `parseRAVENargs` specification of the function. When no argument matches, or the arguments from the first match on are not valid name-value pairs, all arguments are positional. More positional values than the specification has parameters raise `parseRAVENargs:tooManyArgs`. A positional argument whose value is a string equal to a parameter name has to be given in the name-value form. -- `getModelFromKEGG` and `getPhylDist` lost the leading `keggPath`, and `getKEGGModelForOrganism` lost `nSequences` and `seqIdentity`, so later positional arguments shift. -- `scoreModel` has new `isozymeScoring`, `complexScoring` and `dataPrecedence` parameters in the middle of the list. -- `writeYAMLmodel` had `(model, fileName, preserveQuotes, sortIds)` and has `(model, fileName, sortIds)`. -- `findGeneDeletions` lost `analysisType`, `refModel` and `oeFactor`. - -Each exception and its effect is a row in [Backward-incompatible changes](raven3-migration/backward-incompatible.md#backward-incompatible-changes). - -A positional argument that is itself a string matching a parameter name needs the name-value form to avoid ambiguity. - ---- +A positional call keeps its meaning where the parameter order is unchanged. This holds for `importModel`, `exportModel`, `exportToExcelFormat`, `removeReactions`, `addIdentifierPrefix`, `removeIdentifierPrefix` and `exportForGit`. The functions `ftINIT`, `fillGaps`, `fitTasks` and `getModelFromHomology` append new optional parameters after the RAVEN 2 ones, so their old calls also keep their meaning. A comparison of all 112 specifications with the RAVEN 2.11.3 signatures found six functions in which a positional call changes meaning or fails: `getModelFromKEGG`, `getPhylDist`, `getKEGGModelForOrganism`, `scoreModel`, `writeYAMLmodel` and `findGeneDeletions`. Each has a row in [Backward-incompatible changes](raven3-migration/backward-incompatible.md#backward-incompatible-changes). (upgrade-checklist)= ## Upgrade checklist -1. Update to RAVEN 3.0.0 (or later) and run `checkRaven` (replaces `checkInstallation`). The checks include the installation type, the MATLAB release, the libSBML version, the SBML import and export, the LP solvers, the BLAST+, DIAMOND and HMMER binaries and function names that overlap with other toolboxes. `checkRaven` downloads any of the three binaries that is missing (see [Toolbox dependencies and installation](#toolbox-deps)). -2. Search the scripts for the functions listed in [Removed functionality](raven3-migration/structure-and-renames.md#removed-functionality) and in [Renamed and merged functions](#renamed-merged-and-consolidated-functions), and update or remove those calls. -3. If scripts use `getModelFromKEGG`, `getKEGGModelForOrganism` or `getModelFromHomology`, the draft reconstruction can differ even from an unmodified call: default cutoffs, the KEGG data source and the homology best-hit criterion changed (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)). Rebuild and re-validate any KEGG-derived or homology-derived draft models. Positional calls of the KEGG functions also need a check, because the argument positions shifted (see [Backward-incompatible changes](raven3-migration/backward-incompatible.md#backward-incompatible-changes)). -4. If `.xlsx` curation templates or `importExcelModel` are used, convert the curation data to the `.tsv` format that `curateModelFromTables` expects; there is no drop-in Excel importer (see [Excel I/O](raven3-migration/formats-and-reconstruction.md#excel-io)). -5. If scripts use WoLF PSORT localisation scores (`getWoLFScores(...)` or `parseScores` with `'wolf'` or without a predictor), switch to DeepLoc 2, MULocDeep, COMPARTMENTS or UniProt. There is no in-toolbox migration path for existing WoLF PSORT output files (see [New functionality](#new-functionality)). -6. If workflows include metabolic-task gap-filling (`fitTasks`, `checkTasks`, `ftINIT`), re-run them and check the task pass and fail results. A fixed bug means that a task previously reported as passing can now fail (see [Gap-filling and context-specific model extraction](#ftinit)). -7. If model files are under version control (YAML or SBML), diffs can appear on the next export even without content changes (quoting style, EC-code annotation and ΔG fields; see [File formats and model I/O](raven3-migration/formats-and-reconstruction.md#file-formats-and-model-io)). -8. Re-run the test suite or validation pipeline of the project and compare the outputs (model size, growth rate and task results) with the RAVEN 2 baseline before using RAVEN 3 results in production. - ---- +1. Install RAVEN 3 and run `checkRaven`. It checks the installation type, the MATLAB release, the libSBML version, SBML import and export, the LP solvers, the BLAST+, DIAMOND and HMMER binaries, and function names that overlap with other toolboxes. Unless `checkBinaries` is false, `checkRaven` downloads any of the three binaries that is missing (see [Toolbox dependencies and installation](#toolbox-deps)). The function `checkInstallation` forwards to `checkRaven` and warns on every call. +2. Search the scripts for the functions in [Removed functionality](raven3-migration/structure-and-renames.md#removed-functionality) and in [Renamed and merged functions](#renamed-merged-and-consolidated-functions), and replace or delete those calls. The wrappers in `deprecated/` (`makeSomething`, `consumeSomething`, `canProduce`, `canConsume`) work and warn once per session, as do `getINITModel` and `runINIT`. +3. Rebuild and validate every draft model from `getModelFromKEGG`, `getKEGGModelForOrganism` or `getModelFromHomology`. The default cut-offs, the KEGG data source and the best-hit criterion differ, so an unchanged call gives a different draft (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)). Check the positional calls of the six functions in [Calling convention](#calling-convention). +4. Convert `.xlsx` curation templates to the `.tsv` format of `curateModelFromTables`. The function `importExcelModel` does not exist in RAVEN 3 (see [Excel I/O](raven3-migration/formats-and-reconstruction.md#excel-io)). +5. Replace WoLF PSORT scores (`getWoLFScores`, or `parseScores` with `'wolf'`) with DeepLoc 2, MULocDeep, COMPARTMENTS or UniProt. The default predictor of `parseScores` differs (see [Backward-incompatible changes](raven3-migration/backward-incompatible.md#backward-incompatible-changes)). +6. Run task-based gap-filling and context-specific model extraction (`fitTasks`, `checkTasks`, `ftINIT`) with RAVEN 3 and compare the task results. `simplifyModel` keeps the boundary constraint of a task on a metabolite without reactions, which RAVEN 2 removed, so the task check can give a different result in RAVEN 2 and RAVEN 3. The difference in the task check was not run (see [Gap-filling and context-specific model extraction](#ftinit)). The function `fitTasks` returns a third output, `failedTasks`. +7. Expect differences in YAML and SBML files on the next export of a model under version control. The changelog of RAVEN 3 lists changes in quoting, EC-code annotation and ΔG fields. These differences were not compared file by file (see [File formats and model I/O](raven3-migration/formats-and-reconstruction.md#file-formats-and-model-io)). +8. Compare the output of the validation pipeline of the model (model size, growth rate and task results) between RAVEN 2 and RAVEN 3 before using RAVEN 3 results. ## Detail pages | Page | Content | |---|---| -| [Structure, removals and renames](raven3-migration/structure-and-renames.md) | The folder reorganisation, every removed function and subsystem, and every renamed or merged function. | -| [Backward-incompatible changes](raven3-migration/backward-incompatible.md) | Changes that raise errors, changes that alter results without an error, changes to error and warning identifiers, changes that raise warnings and installation changes. | -| [File formats, reconstruction and GPR parsing](raven3-migration/formats-and-reconstruction.md) | SBML, Excel and YAML I/O; the KEGG and homology reconstruction pipelines; the new grRule parser. | +| [Structure, removals and renames](raven3-migration/structure-and-renames.md) | The folder reorganisation, every removed function and folder, and every renamed or merged function. | +| [Backward-incompatible changes](raven3-migration/backward-incompatible.md) | Changes that raise errors, changes that alter results without an error, changes to error and warning identifiers, changes that raise warnings, and installation changes. | +| [File formats, reconstruction and GPR parsing](raven3-migration/formats-and-reconstruction.md) | SBML, Excel and YAML I/O, the KEGG and homology reconstruction pipelines, and the grRule parser. | | [New functionality, dependencies and error handling](raven3-migration/new-and-environment.md) | Gap-filling and ftINIT changes, functions added in RAVEN 3, the toolbox dependency comparison, and error and warning handling. | ```{toctree} diff --git a/docs/raven3-migration/backward-incompatible.md b/docs/raven3-migration/backward-incompatible.md index 9e0209e..0603d8a 100644 --- a/docs/raven3-migration/backward-incompatible.md +++ b/docs/raven3-migration/backward-incompatible.md @@ -1,82 +1,87 @@ (backward-incompatible-changes)= # Backward-incompatible changes -The list covers the changes found by comparing RAVEN 2.11.3 with RAVEN 3. The changes are grouped into those that raise an error, those that give a different result without an error, those to error and warning identifiers, those that raise warnings and installation changes. +The changes below were found by comparing RAVEN 2.11.3 with RAVEN 3 in the code. They are grouped into changes that raise an error, changes that give a different result without an error, changes to error and warning identifiers, changes that raise warnings, and installation changes. The changelog of RAVEN 3 lists further bug fixes. Those whose effect was not checked are named after the table of corrected behaviour. ## Changes that raise errors | RAVEN 2 usage | RAVEN 3 behaviour | |---|---| -| `importExcelModel(...)` | Removed; use `curation/curateModelFromTables.m` (input format `.tsv`, not `.xlsx`) or convert the template to YAML or SBML and use `importModel` (see [Excel I/O](#excel-io)) | -| `importModel(...)` on SBML Level < 3, Level 3 without FBC v2, or any non-FBC SBML | Raises an error; only SBML Level 3 Version 1, FBC package version 2 is accepted (see [SBML I/O](#sbml-io)) | -| `exportToExcelFormat(model, fileName)` where `fileName` is a bare path (no `.xlsx`) | No fallback to tab-delimited text export; raises an error with the message "only export to xlsx format is supported" | -| `exportToTabDelimited(...)` | Removed (dead-code path) | -| `getWoLFScores(...)` | Removed; WoLF PSORT support is fully removed | -| `parseScores(..., 'wolf', ...)` | Raises an error on `'predictor','wolf'`, with a message pointing to `'deeploc'`, `'mulocdeep'`, `'compartments'` or `'uniprot'`; there is no in-toolbox migration path for existing WoLF PSORT output files | -| `qMOMA(...)`, `solveQP(...)` | Removed (the last dependency of RAVEN on `quadprog` from the Optimization Toolbox) | -| `findGeneDeletions(...)` with `analysisType`, `refModel` or `oeFactor` | Arguments removed together with `qMOMA` (MOMA-based over-expression analysis); only single and double gene deletion (`sgd`, `dgd`) by flux balance analysis (FBA) remains. More than the `'testType'` argument raises `parseRAVENargs:tooManyArgs` | -| `checkRxn(...)` | Removed, no direct successor; to debug why a reaction cannot carry flux, use `gapfilling/findLeakMetabolite.m`, `gapfilling/canExchange.m` or `gapfilling/checkProduction.m` | -| `dispEM(...)`, `followFluxes(...)`, `followChanged(...)`, `printModel(...)`, `mapCompartments(...)`, `getExpressionStructure(...)`, `getMetsInComp(...)` | All removed (see [Removed functionality](#removed-functionality) for the replacements, where one exists) | -| Anything under `external/metacyc/` | Subsystem removed: MetaCyc-based reconstruction; no in-toolbox replacement | -| Anything under `pathway/` or `plotting/` | Subsystems removed: all visualisation and plotting functionality (16 functions, such as `drawMap`, `drawPathway`, `markPathwayWithFluxes` and `setOmicDataToRxns`); no in-toolbox replacement, use the COBRA Toolbox or Escher | -| Anything under `legacy/` | Subsystem removed; no in-toolbox replacement (see [Removed functionality](#removed-functionality)) | -| `getGenesFromKEGG`, `getMetsFromKEGG`, `getRxnsFromKEGG`, `constructMultiFasta`, `getWSLpath`, `getBlastFromExcel` | Removed with the local KEGG FTP dump workflow (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)) | -| `getToolboxVersion(...)` | Removed as a standalone function; now part of an internal helper of `exportForGit` | +| `importExcelModel(...)` | Removed. Use `curation/curateModelFromTables.m` (input format `.tsv`). Alternatively convert the template to SBML and read it with `importModel`, or to YAML and read it with `readYAMLmodel` (see [Excel I/O](#excel-io)) | +| `importModel(...)` on SBML Level < 3, on Level 3 without FBC version 2, or on any SBML without FBC | Raises an error. The accepted format is SBML Level 3 Version 1 with FBC package version 2 (see [SBML I/O](#sbml-io)) | +| `exportToExcelFormat(model, fileName)` where `fileName` has no file name or no extension | RAVEN 2 exported tab-delimited text. RAVEN 3 raises the error "only export to xlsx format is supported" for every extension other than `.xlsx`, an empty one included. RAVEN 2 already raised this error for a file name with a different extension | +| `exportToTabDelimited(...)` | Removed | +| `getWoLFScores(...)` | Removed, with all WoLF PSORT support | +| `parseScores(..., 'wolf', ...)` | Raises an error whose message names `'deeploc'`, `'mulocdeep'`, `'compartments'` and `'uniprot'`. RAVEN 3 has no function that reads WoLF PSORT output files | +| `qMOMA(...)`, `solveQP(...)` | Removed. RAVEN 3 has no code that depends on `quadprog` of the Optimization Toolbox | +| `findGeneDeletions(...)` with `analysisType`, `refModel` or `oeFactor` | The arguments are removed together with `qMOMA` (MOMA-based over-expression analysis), and so are the test types `'sgo'` and `'dgo'`. Single and double gene deletion (`sgd`, `dgd`) by flux balance analysis (FBA) remain. More than the `'testType'` argument raises `parseRAVENargs:tooManyArgs` | +| `checkRxn(...)` | Removed without a direct successor. To find why a reaction cannot carry flux, use `gapfilling/findLeakMetabolite.m`, `gapfilling/canExchange.m` or `gapfilling/checkProduction.m` | +| `dispEM(...)`, `followFluxes(...)`, `followChanged(...)`, `printModel(...)`, `mapCompartments(...)`, `getExpressionStructure(...)`, `getMetsInComp(...)` | Removed. [Removed functionality](#removed-functionality) gives the replacements where one exists | +| Any function in `external/metacyc/` | Folder removed, with the 7 functions of the MetaCyc-based reconstruction. RAVEN 3 has no replacement | +| Any function in `pathway/` or `plotting/` | Folders removed, with 16 functions for visualisation and plotting, such as `drawMap`, `drawPathway`, `markPathwayWithFluxes` and `setOmicDataToRxns`. RAVEN 3 has no replacement; the COBRA Toolbox and Escher provide plotting | +| Any function in `legacy/` | Folder removed. RAVEN 3 has no replacement (see [Removed functionality](#removed-functionality)) | +| `getGenesFromKEGG`, `getMetsFromKEGG`, `getRxnsFromKEGG`, `constructMultiFasta`, `getWSLpath`, `getBlastFromExcel` | Removed (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)) | +| `getToolboxVersion(...)` | Removed as a function of its own. It is a local function of `exportForGit` | | `parallelPoolRAVEN(...)` | Renamed `parallelWorkersRAVEN`, with a different signature and output (see [Renamed and merged functions](#renamed-merged-and-consolidated-functions)) | -| `getModelFromKEGG(keggPath, keepSpontaneous, keepUndefinedStoich, keepIncomplete, keepGeneral)` | Raises `parseRAVENargs:tooManyArgs`, because `keggPath` is removed and the function accepts four optional arguments | -| `getKEGGModelForOrganism(...)` with 13 to 15 positional arguments | `nSequences` and `seqIdentity` are removed. 14 or 15 arguments raise `parseRAVENargs:tooManyArgs`; a non-empty `nSequences` (13th) is read as `globalModel` and raises an error | +| `getModelFromKEGG(keggPath, keepSpontaneous, keepUndefinedStoich, keepIncomplete, keepGeneral)` | Raises `parseRAVENargs:tooManyArgs`, because `keggPath` is removed and the function has four optional parameters | +| `getKEGGModelForOrganism(...)` with 13 to 15 positional arguments | The parameters `nSequences` and `seqIdentity` are removed. A call with 14 or 15 arguments raises `parseRAVENargs:tooManyArgs`. A non-empty 13th argument, which is `nSequences` in RAVEN 2, is read as `globalModel` | | `getPhylDist(keggPath, onlyInKingdom)` | Raises `parseRAVENargs:tooManyArgs`, because `keggPath` is removed | -| `writeYAMLmodel(model, fileName, preserveQuotes, sortIds)` with four arguments | Raises `parseRAVENargs:tooManyArgs`; RAVEN 3 has `(model, fileName, sortIds)` | -| `scoreModel(...)` with `'best'` as the seventh or eighth argument, or `hpaLevelScores` as the ninth | Raises `RAVEN:badInput` for `'best'`, which is not a valid value of `isozymeScoring` or `complexScoring`; a structure as the ninth argument raises an error, because that position now holds `multipleCellScoring` | +| `writeYAMLmodel(model, fileName, preserveQuotes, sortIds)` with four arguments | Raises `parseRAVENargs:tooManyArgs`. The RAVEN 3 signature is `(model, fileName, sortIds)` | +| `scoreModel(...)` with `'best'` at position 7, 8 or 9 among all arguments | Raises `RAVEN:badInput`. Positions 7, 8 and 9 are `isozymeScoring`, `complexScoring` and `multipleCellScoring`. The first two accept `'min'`, `'max'`, `'median'` and `'average'`, and the third accepts `'max'` and `'average'` | +| `scoreModel(...)` with `hpaLevelScores` at position 9 among all arguments | Raises an error. Position 9 is `multipleCellScoring`, and `hpaLevelScores` is position 10 | ## Changes that alter results without an error -The same call with the same input model gives different output. +The same call on the same model gives a different output. Positions count all arguments of the call, required ones included: in `scoreModel(model, hpaData, arrayData, tissue, celltype, ...)` the model is position 1 and `celltype` is position 5. -| Function | RAVEN 3 behaviour (default, unless noted) | +### Argument positions and defaults that change meaning + +| Function | RAVEN 3 behaviour | +|---|---| +| `getModelFromKEGG(keggPath)` to `getModelFromKEGG(keggPath, keepSpontaneous, keepUndefinedStoich[, keepIncomplete])` | A call with one to four positional arguments shifts. The first argument, `keggPath` in RAVEN 2, is read as `keepSpontaneous` and has no effect. Each later argument is read as the next parameter in the RAVEN 2 order, so a different class of reactions is removed | +| `getPhylDist(keggPath)` | `keggPath` is read as `'onlyInKingdom'` and has no effect. The function returns the full distance matrix | +| `writeYAMLmodel(model, fileName, true)` | The third argument is read as `sortIds`, so the IDs are sorted. RAVEN 2 read it as `'preserveQuotes'` | +| `scoreModel` | `isozymeScoring` and `complexScoring` replace the single `multipleGeneScoring` of RAVEN 2. By default (`complexScoring`) a reaction with an `and` rule gets the minimum over its genes. RAVEN 2 scored every reaction with the maximum over all its genes by default (`multipleGeneScoring` `'best'`) | +| `scoreModel(..., celltype, noGeneScore, 'average')` | `'average'` at position 7 among all arguments, which is `multipleGeneScoring` in RAVEN 2, is read as `isozymeScoring`. It does not apply to `and` rules, which use `complexScoring` | +| `scoreModel(..., celltype, noGeneScore, multipleGeneScoring, 'average')` | `'average'` at position 8 among all arguments, which is `multipleCellScoring` in RAVEN 2, is read as `complexScoring` without an error. The average runs over the genes of an `and` rule; RAVEN 2 averaged over the cell types | +| `scoreModel`, output `geneScores` | A gene that is absent from the data receives `NaN`. The RAVEN 2 documentation states `-Inf` | +| `scoreModel`, value names | RAVEN 2 accepted `'best'` for `multipleGeneScoring` and `multipleCellScoring`. RAVEN 3 uses `'max'` for `isozymeScoring`, `complexScoring` and `multipleCellScoring`. The parameter order is `celltype`, `noGeneScore`, `isozymeScoring`, `complexScoring`, `multipleCellScoring`, `hpaLevelScores` and `dataPrecedence` (default `'gene'`) | +| `parseScores(file)` | The default predictor is `'deeploc'`. RAVEN 2 used `'wolf'`. How a WoLF PSORT file is handled without a predictor was not run | +| `randomSampling(model, n)` | The default `method` is `'achr'`. RAVEN 2 ran the random-objective method, which `'randomObjective'` selects. The output `goodRxns` is empty for `'achr'` and `'chrr'`. The function has the parameters `nObjectives`, `seed`, `method`, `thinning` and `nBurnin` and a third output, `info` | +| `getModelFromHomology`, `strictness=3` | The best hit is the hit with the highest bitscore. RAVEN 2 selected the hit with the lowest E-value. No parameter selects the criterion (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)) | +| `getModelFromHomology`, `minLen` | The default is `100`. In RAVEN 2 it was `200` | +| `getKEGGModelForOrganism`, `cutOff` | The default is `10^-30`. In RAVEN 2 it was `10^-50` | +| `getKEGGModelForOrganism`, `minScoreRatioG` | The default is `0.9`. In RAVEN 2 it was `0.8` | + +### Corrected behaviour + +| Function | RAVEN 3 behaviour | |---|---| -| `getModelFromHomology`, `strictness=3` | Best-hit tie-break uses bitscore; RAVEN 2.11.3 used E-value, and no option restores that criterion (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)) | -| `getModelFromHomology`, `minLen` | Default lowered from `200` to `100` (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)) | -| `getModelFromHomology`, merging of gene-reaction rules (GPR) | Gene substitution is based on exact tokens; RAVEN 2 used regex or `strrep` (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)) | -| `getKEGGModelForOrganism`, `cutOff` | Default lowered from `10^-50` to `10^-30` (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)) | -| `getKEGGModelForOrganism`, `minScoreRatioG` | Default raised from `0.8` to `0.9` (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)) | -| `removeGenes`, `expandModel`, `addRxns` | RAVEN 3 parses `grRules` by exact gene-token matching; RAVEN 2 used substring or regex matching (see [Parsing of gene-reaction rules](#gpr-parsing)) | -| `findPotentialErrors` | RAVEN 3 parses the rule into a tree; the RAVEN 2.11.3 check inside `standardizeGrRules` searched for string patterns (see [Parsing of gene-reaction rules](#gpr-parsing)) | -| `changeGrRules(..., 'replace', false)` onto a gene-less reaction | Produces a valid grRule; RAVEN 2 produced an unparseable one (see [Parsing of gene-reaction rules](#gpr-parsing)) | -| `simplifyModel` (used by task-based gap-filling) | Keeps the boundary constraint of a task on a metabolite without reactions (see [Gap-filling and context-specific model extraction](#ftinit)) | -| `ftINITInternalAlg` | With `allowExcretion` true, RAVEN 3 constrains the metabolite balance to `S*v >= 0`; RAVEN 2 had the sense of the constraint inverted | -| `ravenCobraWrapper` | Gene-ID-to-MIRIAM matching handles gene IDs that contain regex metacharacters; RAVEN 2 mis-matched them (see [RAVEN⇄COBRA interoperability](#raven-cobra-interoperability)) | -| `mergeCompartments` | The merged metabolite is the copy already in compartment `s`, otherwise the copy whose ID sorts first (character code order), with every field of that copy | -| `simplifyModel` with `constrainReversible` on a model without a reversible reaction | This step changes nothing; RAVEN 2 raised an error, because `getAllowedBounds` took the empty reaction list for all reactions | -| `getModelFromKEGG(keggPath)` and `getModelFromKEGG(keggPath, keepSpontaneous)` | `keggPath` is read as `keepSpontaneous` and has no effect; the second argument is read as `keepUndefinedStoich`, so the wrong class of reactions is removed | -| `getPhylDist(keggPath)` | `keggPath` is read as `'onlyInKingdom'` and has no effect; the full distance matrix is returned | -| `writeYAMLmodel(model, fileName, true)` | The third argument is read as `sortIds`, so IDs are sorted; RAVEN 2.11.3 read it as `'preserveQuotes'` | -| `scoreModel` | Reactions with an `and` rule get the minimum over the genes by default (`complexScoring`); RAVEN 2.11.3 scored every reaction with the maximum over all genes | -| `scoreModel(..., celltype, noGeneScore, 'average')` | `'average'` as the seventh argument is read as `isozymeScoring`, so it no longer applies to `and` rules, which use `complexScoring` | -| `parseScores(file)` | The default predictor is `'deeploc'`; RAVEN 2.11.3 used `'wolf'`. WoLF PSORT output is parsed as DeepLoc output | -| `permuteModel`, `randomSampling`, `optimizeProb`, `getFluxZ`, `convertToIrrev`, `replaceMets`, `setExchangeBounds`, `fitParameters`, `guessComposition` | Each function had a bug in RAVEN 2; output differs only for models and inputs that triggered it | - -`removeGenes`, `expandModel` and `findPotentialErrors` parse the rule into a tree, which handles isozymes and complexes. `addRxns` only splits the rule on space-bounded `and` and `or` tokens and builds no tree. - -`simplifyModel` in RAVEN 2.11.3 dropped the boundary constraint of a task on a metabolite without reactions, so a task that RAVEN 2.11.3 reported as passing can fail in RAVEN 3. - -`mergeCompartments` in RAVEN 2.11.3 kept the copy that came first in `model.mets`. - -The `scoreModel` of RAVEN 3 takes `celltype`, `noGeneScore`, `isozymeScoring`, `complexScoring`, `multipleCellScoring`, `hpaLevelScores` and `dataPrecedence` (default `'gene'`) in this order. `multipleCellScoring` accepts `'max'` and `'average'`. - -None of the changes in this table is an API change. +| `getModelFromHomology`, merging of gene-reaction rules (GPR) | Gene substitution uses exact tokens. RAVEN 2 used regular expressions or `strrep` | +| `removeGenes`, `addRxns` | `removeGenes` evaluates the rule as a tree, so isozymes and complexes are handled. `addRxns` splits a rule at the space-bounded tokens `and` and `or`, so a gene ID such as `band1` stays whole, and it builds no tree. RAVEN 2 matched substrings | +| `expandModel` | The rule is parsed with `grRuleToDNF`. A rule that cannot be parsed (`RAVEN:badGrRule`) stays unchanged and its reaction is listed in `rxnToCheck`. A rule that expands to more than 10000 isozymes raises `RAVEN:grRuleTooComplex`. RAVEN 2 split the rule string at `' or '` and `' and '` (see [Parsing of gene-reaction rules](#gpr-parsing)) | +| `findPotentialErrors` | The changelog of RAVEN 3 states that the check uses a parse tree. The effect on results was not compared (see [Parsing of gene-reaction rules](#gpr-parsing)) | +| `changeGrRules(..., 'replace', false)` onto a reaction without genes | Produces a valid grRule. RAVEN 2 wrapped the new rule as `() or (X)`, which RAVEN 3 cannot parse | +| `simplifyModel` | Keeps the boundary constraint (`b`) of a task on a metabolite without reactions. RAVEN 2.11.3 removed such metabolites, so the task check can give a different result in RAVEN 3 (not run; see [Gap-filling and context-specific model extraction](#ftinit)) | +| `simplifyModel` with `constrainReversible` on a model without a reversible reaction | The step changes nothing. RAVEN 2 raised an error, because `getAllowedBounds` takes an empty reaction list as all reactions (the error is taken from the changelog and was not reproduced) | +| `ftINITInternalAlg` | With `allowExcretion` true, the metabolite balance is constrained to `S*v >= 0`. RAVEN 2 had the sense of the constraint inverted | +| `ravenCobraWrapper` | The conversion of `grRules` to the COBRA `rules` field handles gene IDs that contain regular-expression metacharacters. RAVEN 2 did not escape them (see [RAVEN⇄COBRA interoperability](#raven-cobra-interoperability)) | +| `mergeCompartments` | The merged metabolite is the copy in compartment `s` or, failing that, the copy whose ID sorts first in character code order, with every field of that copy. RAVEN 2 kept the copy that came first in `model.mets` | + +The changelog of RAVEN 3 lists a bug fix for each of `permuteModel`, `optimizeProb`, `getFluxZ`, `convertToIrrev`, `replaceMets`, `setExchangeBounds`, `fitParameters` and `guessComposition`. It lists further fixes that can change results, for example for `mergeLinear`, `mergeModels`, `closeModel`, `addTransport`, `getExchangeRxns`, `removeLowScoreGenes`, `getMinNrFluxes`, `reporterMetabolites`, `getElementalBalance`, `checkModelStruct` and `deleteUnusedGenes`. The effect of these fixes on results was not checked. ## Changes to error and warning identifiers -- `dispEM`, the universal error and warning function of RAVEN 2, always raised errors with an empty identifier (`MException('', ...)`), so no RAVEN 2 script could legitimately have caught on `err.identifier`. RAVEN 3 replaces `dispEM` with native `error()` and `warning()` calls. Calls that carry an identifier predominantly use `RAVEN:badInput` (220 calls) and `RAVEN:warning` (63 calls), plus other identifiers, for example `RAVEN:modelError`, `RAVEN:badGrRule`, `RAVEN:infeasibleModel`, `RAVEN:infeasible`, `RAVEN:sampling` and `RAVEN:grRuleTooComplex`. Not every call carries an identifier. -- `try/catch` blocks that match on message text keep working for messages whose text is unchanged from RAVEN 2. Scripts can match on `err.identifier` for the calls that carry one. +RAVEN 2 raised its errors through `dispEM` with an empty identifier (`MException('', ...)`), so no RAVEN 2 script could catch an error by `err.identifier`. RAVEN 3 replaces `dispEM` with `error()` and `warning()` calls. Not every call carries an identifier, and a script can match on `err.identifier` for the calls that do. `RAVEN:badInput` appears in 220 lines and `RAVEN:warning` in 63 lines of `src/` and the root `.m` files (non-comment lines that contain the identifier). Further identifiers in the `RAVEN:` namespace are `RAVEN:modelError`, `RAVEN:badGrRule`, `RAVEN:infeasibleModel`, `RAVEN:infeasible`, `RAVEN:sampling`, `RAVEN:grRuleTooComplex`, `RAVEN:deprecated`, `RAVEN:legacyMethod`, `RAVEN:notMainBranch`, `RAVEN:glpkMILP`, `RAVEN:fetchToolboxAsset:unverified`, `RAVEN:fetchToolboxAsset:checksum`, `RAVEN:fetchToolboxAsset:manifest`, `RAVEN:runSuppOrth:supportLv`, `RAVEN:runSuppOrth:failed`, `RAVEN:runSuppOrth:notFound` and `RAVEN:hitsFromSuppOrth:unmappedProteins`. Identifiers in other namespaces include `parseRAVENargs:tooManyArgs`, `parseRAVENargs:noSpec`, `getPhylDist:noData`, `renameModelGenes:*`, `setGAM:*` and `parseYAML:*`. The list is not exhaustive. + +Whether `try/catch` blocks that match on message text match was not compared message by message. ## Changes that raise warnings -- `getINITModel` and `runINIT` issue a one-time "legacy" notice that points at `ftINIT`; `warning('off','RAVEN:legacyMethod')` silences it. The notice is not a deprecation: the RAVEN documentation states that these functions "remain supported" indefinitely, to reproduce previously published Human-GEM-derived models. See [Gap-filling and context-specific model extraction](#ftinit). -- `makeSomething`, `consumeSomething`, `canProduce` and `canConsume` are in a new `deprecated/` folder as wrappers around `findLeakMetabolite` and `canExchange`. They issue a one-time `RAVEN:deprecated` warning that states they "will be removed in the next major release." The wrappers are scheduled for removal in the next major release, so calls should be migrated before then. See [Gap-filling and context-specific model extraction](#ftinit). -- `checkInstallation` is now `checkRaven`, moved from `installation/` to the repository root, with an unchanged signature (`[currVer, installType] = checkRaven(developMode, checkBinaries)`). Existing calls keep working: `checkInstallation` remains as a forwarding wrapper that passes every argument and output through, after issuing a `RAVEN:deprecated` warning that names `checkRaven`. The wrapper is scheduled for removal in the next major release, like the functions in `deprecated/`, so calls should be migrated before then. +- `getINITModel` and `runINIT`, the functions of the legacy tINIT method, issue a one-time notice with the identifier `RAVEN:legacyMethod` that points to `ftINIT`. `warning('off','RAVEN:legacyMethod')` silences it. The help text of both functions states that the function remains supported, although the changelog of RAVEN 3 marks both as deprecated (see [Gap-filling and context-specific model extraction](#ftinit)). +- `makeSomething`, `consumeSomething`, `canProduce` and `canConsume` are wrappers in the folder `deprecated/` around `findLeakMetabolite` and `canExchange`. They issue a one-time `RAVEN:deprecated` warning that they will be removed in the next major release. +- `checkInstallation` is `checkRaven` in RAVEN 3, moved from `installation/` to the repository root, with the same signature: `[currVer, installType] = checkRaven(developMode, checkBinaries)`. The wrapper `checkInstallation` forwards every argument and output to `checkRaven` and issues a `RAVEN:deprecated` warning that names `checkRaven` on every call. The wrapper will be removed in the next major release. ## Installation changes -The KEGG HMM libraries and the external command-line binaries (BLAST+, DIAMOND, HMMER, cd-hit, MAFFT and WoLF PSORT) are no longer committed to the repository. RAVEN 3 fetches the KEGG data and the BLAST+, DIAMOND and HMMER binaries from the [`toolbox-assets`](https://github.com/CompMetEng/toolbox-assets) repository the first time a function needs them, so internet access is required on first use as well as at clone or install time. What is fetched and where it is cached is described in [Download data and binaries](../installation/data-and-binaries.md). The air-gapped install and the full dependency comparison are in [Toolbox dependencies and installation](#toolbox-deps). +The repository of RAVEN 3 does not contain the KEGG HMM libraries, the external binaries (BLAST+, DIAMOND, HMMER, cd-hit, MAFFT, WoLF PSORT and SCIP) or Apache POI. The first time a function needs them, RAVEN 3 fetches the KEGG HMM libraries and reference tables and the BLAST+, DIAMOND and HMMER binaries from the [`toolbox-assets`](https://github.com/CompMetEng/toolbox-assets) repository, so the first use requires internet access. The binaries are stored in `src/external/`. The function `setRavenSolver` downloads SCIP for Windows from the RAVEN v2.8.6 GitHub release. An offline bundle of the binaries is available as a `*-binaries` RAVEN release. [Download data and binaries](../installation/data-and-binaries.md) describes what is fetched and where it is cached, and [Toolbox dependencies and installation](#toolbox-deps) covers the air-gapped install and the dependency comparison. diff --git a/docs/raven3-migration/formats-and-reconstruction.md b/docs/raven3-migration/formats-and-reconstruction.md index fbfed7f..bb67b92 100644 --- a/docs/raven3-migration/formats-and-reconstruction.md +++ b/docs/raven3-migration/formats-and-reconstruction.md @@ -1,6 +1,6 @@ # File formats, reconstruction and GPR parsing -Model file formats, draft reconstruction and the parsing of gene-reaction rules (`grRules`, abbreviated GPR) differ between RAVEN 2 and RAVEN 3. +Three areas differ between RAVEN 2 and RAVEN 3: the model file formats, the draft reconstruction from KEGG and from homology, and the parsing of gene-reaction rules (`grRules`, abbreviated GPR). Each section gives the changes that affect a script and the differences that show up as diffs in version-controlled model files. (file-formats-and-model-io)= ## File formats and model I/O @@ -11,39 +11,57 @@ Model file formats, draft reconstruction and the parsing of gene-reaction rules | Aspect | RAVEN 2 | RAVEN 3 | |---|---|---| | `exportModel` output format | SBML Level 3 Version 1, FBC package version 2 | Unchanged | -| `importModel` accepted input | Any SBML level and version; L2-style notes-based gene associations (`GENE_ASSOCIATION` and `PROTEIN_ASSOCIATION`) also read; FBC fields optional | Only SBML Level 3 Version 1, FBC package version 2; anything else raises an error | -| MIRIAM prefixes read on import | `urn:miriam:`, `http://identifiers.org/`, `https://identifiers.org/` | Unchanged | +| `importModel` accepted input | Any SBML level and version. The notes-based gene association `GENE_ASSOCIATION` of Level 2 files is read, and `SHORT NAME` is read for gene names. FBC fields are optional | SBML Level 3 Version 1 with FBC package version 2. Any other file raises the error `RAVEN:badInput` | +| MIRIAM prefixes read on import | `urn:miriam:`, `http://identifiers.org/` and `https://identifiers.org/` | `http://identifiers.org/` and `https://identifiers.org/` | -Older SBML files have to be converted before `importModel` accepts them, for example by re-exporting them from cobrapy or the COBRA Toolbox. +`importModel` of RAVEN 3 reads MIRIAM annotations (reaction EC codes, model and species annotations) with the prefixes `http://identifiers.org/` and `https://identifiers.org/`, and ignores `urn:miriam:` annotations. + +:::{admonition} Cross-references in older SBML files +:class: warning +An SBML file with `urn:miriam:` annotations is imported without an error, and its cross-references (for example `metMiriams`) are absent from the imported model. The `exportModel` function of RAVEN 2 writes `https://identifiers.org/` prefixes, so files written by RAVEN 2 are not affected. Files from other tools that use `urn:miriam:` can be converted to the `identifiers.org` form before the import. +::: + +A file in an older SBML level or version has to be converted before `importModel` accepts it. The error message suggests another tool or an older RAVEN version for the conversion. (excel-io)= ### Excel I/O | Aspect | RAVEN 2 | RAVEN 3 | |---|---|---| -| Excel engine | Apache POI (Java), bundled | Dependency-free OOXML writer (`io/writeExcel.m`) | -| Full-model `.xlsx` import | `importExcelModel.m` (`RXNS`, `METS`, `COMPS`, `GENES` and `MODEL` sheets → complete model) | Removed, no equivalent importer | -| Tabular curation of an existing model | No equivalent | `curation/curateModelFromTables.m` curates or extends a loaded model from separate `.tsv` files (metabolites by name and compartment, reactions by stoichiometry, genes by name); it is not an importer | -| `exportToExcelFormat` on a bare path (no `.xlsx`) | Fell back to tab-delimited text export | Raises an error, `.xlsx` only | +| Excel engine | Apache POI (Java), bundled with RAVEN | Dependency-free OOXML writer (`io/writeExcel.m`) | +| Import of a full model from `.xlsx` | `importExcelModel.m` builds a complete model from the sheets `RXNS`, `METS`, `COMPS`, `GENES` and `MODEL` | Removed | +| Tabular curation of an existing model | No equivalent | `curation/curateModelFromTables.m` curates or extends a loaded model from `.tsv` files: metabolites by name and compartment, reactions by stoichiometry, genes by name | +| `exportToExcelFormat` with a path that has no file name or extension | Falls back to tab-delimited text | Raises an error. The output is `.xlsx` | -A pipeline that depends on the RAVEN 2 Excel curation template can convert the template to the `.tsv` format that `curateModelFromTables` expects, or build the model with `importModel` or YAML and use `curateModelFromTables` for incremental edits. +A pipeline that depends on the RAVEN 2 Excel curation template has two options. The template can be converted to the `.tsv` format that `curateModelFromTables` expects. Alternatively the model is built with `importModel` or `readYAMLmodel`, and `curateModelFromTables` makes the later edits. +(yaml-io)= ### YAML I/O -`readYAMLmodel` and `writeYAMLmodel` are pure MATLAB in RAVEN 3 and were in RAVEN 2, so model YAML I/O has no Python dependency in either version. `io/parseYAML.m` is a new pure-MATLAB generic YAML parser, used only by `conditions/applyCondition.m` to read condition files. RAVEN 3 writes model YAML with the following differences, which appear as diffs in version-controlled model files: +`readYAMLmodel` and `writeYAMLmodel` are written in MATLAB in both versions, so the YAML I/O of a model has no Python dependency in either. The function `io/parseYAML.m` is a general YAML parser in MATLAB, which `conditions/applyCondition.m` uses to read condition files. + +`writeYAMLmodel` of RAVEN 3 takes name-value arguments and quotes strings by a fixed rule, in place of the RAVEN 2 option `preserveQuotes`. The output differs from the output of RAVEN 2 in the points below, and each of them appears as a diff in a version-controlled model file. | Aspect | RAVEN 2 | RAVEN 3 | |---|---|---| -| Quote style | Single quotes | Double quotes | -| Reaction EC codes | Top-level `eccodes` field only | Also written under `annotation: ec-code` | -| Gene-less model | `genes:` section omitted | `genes: []` written explicitly | -| Empty `gene_reaction_rule` | Written as an empty string | Omitted | -| `objective_coefficient` on import | Ignored | Honoured | +| Quoting | Every string has double quotes (default `preserveQuotes` true) | A string has double quotes where YAML needs them (reserved words, numbers, leading indicators, `: `). Other strings are bare | +| Indentation | List entries ` - !!omap` (four spaces), keys indented by six spaces | List entries ` - !!omap` (two spaces), keys indented by four spaces | +| Reaction EC codes | Top-level field `eccodes` | Under `annotation: ec-code`. The top-level key is not written | +| Metabolite SMILES | Top-level field | Under `annotation` as `smiles` | +| Reaction notes key | `rxnNotes` | `notes` | +| Gene-less model | `genes:` is omitted when `model.genes` is absent, and an empty field gives a bare `- genes:` | The line `- genes: []` | +| Document start | A `---` line, then `!!omap` | `!!omap` alone | +| Field order, metabolites | `formula`, then `charge` | `charge`, then `formula` | +| Field order, reactions | `rxnFrom` directly after `gene_reaction_rule`, and `references` before `subsystem` and `annotation` | `subsystem` and `annotation` first, then `references` and `rxnFrom` among the RAVEN-specific extras | +| Field order, genes | `protein` before `annotation` | `protein` after `annotation` | +| `objective_coefficient` on import | Sets the coefficient to 1 for any value | Reads the value | + +`readYAMLmodel` of RAVEN 3 accepts the top-level keys `eccodes`, `smiles` and `rxnNotes` as well as the forms `annotation` and `notes`. (raven-cobra-interoperability)= ### RAVEN⇄COBRA interoperability -`conversion/ravenCobraWrapper.m` keeps its signature (`newModel = ravenCobraWrapper(model)`). RAVEN 3 fixes two bugs that change the output: a stale variable that could reuse the wrong MIRIAM data during gene-annotation conversion, and a mis-match of gene IDs containing regex metacharacters (for example `ENSG00000141510.16`) during annotation conversion. +The function `conversion/ravenCobraWrapper.m` keeps its signature, `newModel = ravenCobraWrapper(model)`. RAVEN 3 fixes two bugs that change its output. One is a stale variable that could reuse the wrong MIRIAM data in the conversion of gene annotations. The other is a mismatch of gene IDs that contain regular-expression metacharacters, for example `ENSG00000141510.16`, when the `rules` field is built from `grRules`. --- @@ -54,54 +72,62 @@ A pipeline that depends on the RAVEN 2 Excel curation template can convert the t | Aspect | RAVEN 2 | RAVEN 3 | |---|---|---| -| Reaction and gene database source | Local KEGG FTP dump (licence required) | Prebuilt artefacts fetched on demand from releases of `toolbox-assets` | -| Functions that build the database | `getGenesFromKEGG`, `getMetsFromKEGG` and `getRxnsFromKEGG` parsed the dump into a cached `keggModel.mat` | `buildGlobalKEGGModel`, `buildGlobalGPR` and `readKEGGTable` assemble the artefacts | -| `getModelFromKEGG`'s `keggPath` argument | Required | Removed, not needed | -| Ortholog HMM library | One `.hmm` file per KEGG Orthology (KO) entry; one `hmmsearch` call per KO, via WSL on Windows | One concatenated per-domain library (`kegg118_eukaryotes.hmm` and `kegg118_prokaryotes.hmm`); one `hmmsearch` call against the whole proteome, no WSL | -| `dataDir` | Arbitrary local folder | Must be `kegg118_eukaryotes` or `kegg118_prokaryotes` | -| `nSequences`, `seqIdentity` (CD-HIT clustering controls) | Configurable | Removed, no local clustering step | -| `cutOff` default | `10^-50` | `10^-30` | -| `minScoreRatioG` default | `0.8` | `0.9` | -| Calling convention | Positional | Positional or named (`parseRAVENargs`); positions shift in `getModelFromKEGG`, `getKEGGModelForOrganism` and `getPhylDist` (see below the table) | +| Source of the reaction and gene database | Local KEGG FTP dump (licence required), read when the cached files `keggGenes.mat`, `keggMets.mat` and `keggRxns.mat` are absent | Prebuilt artefacts fetched on demand from releases of `toolbox-assets` | +| Functions that build the database | `getGenesFromKEGG`, `getMetsFromKEGG` and `getRxnsFromKEGG` | `buildGlobalKEGGModel`, `buildGlobalGPR` and `readKEGGTable` | +| Argument `keggPath` of `getModelFromKEGG` | Optional, default `external/kegg` | Removed | +| Ortholog HMM library | One `.hmm` file per KEGG Orthology (KO) entry and one `hmmsearch` call per KO, through the Windows Subsystem for Linux (WSL) on Windows | One concatenated library per domain (`kegg118_eukaryotes.hmm` and `kegg118_prokaryotes.hmm`), one `hmmsearch` call against the whole proteome, and no WSL | +| `dataDir` | Any local folder | For the homology method, which takes a FASTA file, the folder name must end in `kegg118_eukaryotes` or `kegg118_prokaryotes`. The annotation method ignores `dataDir` | +| `nSequences`, `seqIdentity` (CD-HIT clustering controls) | Configurable | Removed; there is no local clustering step | +| Default of `cutOff` | `10^-50` | `10^-30` | +| Default of `minScoreRatioG` | `0.8` | `0.9` | +| Calling convention | Positional | Positional or name-value (`parseRAVENargs`). A call with more positional arguments than the function defines raises `parseRAVENargs:tooManyArgs` | -The RAVEN 3 artefacts are a gene-free reference model plus KO, reaction and organism tables. In RAVEN 2 the per-KO `.hmm` files were either self-built with CD-HIT, MAFFT and `hmmbuild`, or downloaded. +The first call of `getModelFromKEGG` downloads the data, builds the global KEGG model and saves it as `src/reconstruction/kegg/keggModel.mat`. The source header states that the first build needs a few hundred MB of disk, and the size of the cached file was not checked. The function `getKEGGModelForOrganism` loads the global model, the KO model and the five flag vectors from the same file. The external binaries and the KEGG data come from pinned `toolbox-assets` tags and are not bundled (see [Toolbox dependencies and installation](#toolbox-deps)). -In RAVEN 3 the three functions take their optional arguments through `varargin`, in the order of RAVEN 2 minus the removed arguments. Positional calls written for RAVEN 2 (checked against v2.11.3) change as follows: +The three functions take their optional arguments through `varargin`, in the RAVEN 2 order without the removed arguments. A position counts all arguments of the call, required ones included. Positional calls written for RAVEN 2 (checked against 2.11.3) change as follows. -- `getModelFromKEGG(keggPath, keepSpontaneous, ...)` became `getModelFromKEGG(keepSpontaneous, ...)`. Each old positional argument is read as the parameter one position earlier. -- `getKEGGModelForOrganism` no longer has `nSequences` and `seqIdentity`, so `globalModel` moves from the 15th to the 13th argument. -- `getPhylDist(keggPath, onlyInKingdom)` became `getPhylDist(onlyInKingdom)`, because `keggPath` was removed. +- `getModelFromKEGG(keggPath, keepSpontaneous, ...)` becomes `getModelFromKEGG(keepSpontaneous, ...)`. An unchanged RAVEN 2 call shifts every later argument: the argument at position 2 is read as the parameter that RAVEN 2 has at position 3, and so on. +- `getKEGGModelForOrganism` has no `nSequences` and no `seqIdentity`, so `globalModel` moves from the 15th to the 13th argument. +- `getPhylDist(keggPath, onlyInKingdom)` becomes `getPhylDist(onlyInKingdom)`. ### Homology-based reconstruction (`getModelFromHomology`) | Aspect | RAVEN 2 | RAVEN 3 | |---|---|---| -| `strictness` parameter | `1`, `2` or `3` | Unchanged | -| Best-hit tie-break at `strictness=3` | E-value | Bitscore; no option restores the E-value criterion of RAVEN 2 | -| `minLen` default | `200` | `100` | -| GPR gene substitution when merging homology hits | Regex or `strrep` substitution (could corrupt IDs that were substrings of one another, or contained regex metacharacters) | Exact-token-based substitution | +| Parameter `strictness` | `1`, `2` or `3` | Unchanged | +| Best-hit criterion at `strictness=3` | E-value | Bitscore. No option restores the E-value criterion | +| Default of `minLen` | `200` | `100` | +| Gene substitution in GPRs when homology hits are merged | Substitution by regular expression or `strrep`, which corrupts IDs that are substrings of other IDs or contain regular-expression metacharacters | Substitution by exact token | +| Complexes, gene-free reactions and notes | A reaction with an incompletely mapped complex is flagged with the prefix `OLD_` | The options `complexPolicy` (default `'flag'`, the RAVEN 2 behaviour), `keepGeneFree` and `preserveNotes` | +| Gene IDs of the new organism | No restriction | Must not contain whitespace, parentheses, `\|` or `&` | -### Binaries and genome data +The functions `reconstruction/homology/runSuppOrth.m` and `reconstruction/homology/hitsFromSuppOrth.m` supply reconstruction input and have no RAVEN 2 counterpart. The homology method takes its hits from BLAST, DIAMOND or SuppOrth. -External binaries (BLAST+, DIAMOND, HMMER) and the KEGG data artefacts are fetched on demand from pinned `toolbox-assets` release tags and are not bundled with the repository; see [Toolbox dependencies and installation](#toolbox-deps). +### Genome data -`curation/downloadGenomeData.m`, `curation/getGeneData.m` and `curation/processProteinFastaFile.m` are new in RAVEN 3 and turn an NCBI genome accession (`GCF_` or `GCA_`) into a model-ready proteome and a gene ID mapping table. +The functions `curation/downloadGenomeData.m`, `curation/getGeneData.m` and `curation/processProteinFastaFile.m` turn an NCBI genome accession (`GCF_` or `GCA_`) into a proteome that is ready for modelling and a gene ID mapping table. RAVEN 2 has no counterpart. + +The content of the KEGG artefacts, a gene-free reference model plus KO, reaction and organism tables, was not checked against the `toolbox-assets` release. --- (gpr-parsing)= ## Parsing of gene-reaction rules -RAVEN 3 adds a gene-reaction rule tokeniser and parser. `utils/parseGrRule.m` builds a parse tree (`type`: `'gene'|'and'|'or'`, with `children`). `utils/grRuleToDNF.m` converts the tree to disjunctive normal form (DNF), RAVEN's canonical "OR of AND-clauses" shape for enzyme-complex and isozyme structure. `utils/isDnfGrRule.m` checks whether a rule is already in that form. `utils/grRuleToString.m` renders a tree back to a canonical string. Five functions use the parser and converters internally: `removeGenes`, `expandModel`, `findPotentialErrors`, `diffModels` and `curateModelFromTables`. The parser and converters are not applied automatically to a loaded model: a rule keeps its original text unless one of these functions is called on it. +RAVEN 3 parses gene-reaction rules into a tree. Five functions use the parser: `removeGenes`, `expandModel`, `findPotentialErrors`, `diffModels` and `curateModelFromTables`. The parser is not applied to a loaded model, so a rule keeps its text unless one of these functions changes it. | Function | RAVEN 2 parsing | RAVEN 3 parsing | |---|---|---| -| `removeGenes` | Substring match on gene ID (`"10"` also matched `"100"`) | Exact gene-token match | -| `expandModel` | Split on `' or '` (could drop an `AND`-ed subunit inside a nested rule) | Parse-tree-based isozyme expansion, preserves nested `AND` and `OR` | -| `findPotentialErrors` | Local function of `core/standardizeGrRules.m` (not callable); string-pattern check for `") and ("` and similar patterns (false positives on valid rules) | Function of its own; parse-tree-based non-DNF detection | -| `addRxns` (new gene IDs from a grRule) | Split on the substrings "and" and "or" anywhere (could split an ID such as `"band1"` at its "and") | Split only on space-bounded `' and '` and `' or '` tokens | -| `comparison/diffModels.m` | (not present) | grRules compared by logical DNF equality | +| `removeGenes` | Substring match on the gene ID (`"10"` also matches `"100"`) | Match on the exact gene token | +| `expandModel` | Split on `' or '` after the parentheses are removed. An `AND`-ed subunit in a nested rule can be dropped | Isozyme expansion from the parse tree, which keeps nested `AND` and `OR` | +| `findPotentialErrors` | Local function of `core/standardizeGrRules.m`. A string-pattern check for `") and ("` and similar patterns, with false positives on valid rules. Returns indices | A function of its own. Parse-tree detection of rules that are not in disjunctive normal form (DNF). `issues = findPotentialErrors(model)` returns a structure array with the fields `index`, `rxn`, `grRule` and `reason` | +| `diffModels` | Not present | Compares grRules. Two rules are equal when their DNF forms are equal after sorting of genes and clauses, without absorption: `A or (A and B)` differs from `A` | +| `curateModelFromTables` | Not present | Parses the rules of the table input | + +`addRxns` does not use the parser. It takes the new gene IDs of a grRule from `getGenesFromGrRules`, which splits at the space-bounded tokens `and`, `or`, `&` and `|`. RAVEN 2 split at the substrings "and" and "or" anywhere, so an ID such as `band1` was cut at "and". + +The parser functions are in `utils/`. `parseGrRule` builds a tree with `type` (`'gene'`, `'and'` or `'or'`) and `children`. `grRuleToDNF` converts the tree to DNF, an OR of AND clauses. `isDnfGrRule` checks whether a rule has that form, and `grRuleToString` renders a tree as a canonical string. -`parseGrRule` accepts `and` and `or` in any case and the symbols `&`, `&&`, `|` and `||`. Operators are recognised as whole words, so `G1&G2` is one gene ID. A malformed rule raises `RAVEN:badGrRule`. +`parseGrRule` accepts `and` and `or` in any case and the symbols `&`, `&&`, `|` and `||`. It recognises operators as whole words, so `G1&G2` is one gene ID. A malformed rule raises `RAVEN:badGrRule`. -`manipulation/standardizeGrRules.m` (which runs on import) does not build a parse tree. It converts ` AND ` and ` OR ` to lowercase and splits the rule on the space-bounded strings ` and ` and ` or `, so those are the two spellings of the operators it recognises. It calls `findPotentialErrors`, which parses the rules, and returns the indices of rules that cannot be parsed or are not in DNF (`indexes2check`). A direct call lists these rules in one warning; `importModel` calls the function with `embedded` set to `true`, which suppresses the warning. The function does not raise `RAVEN:badGrRule`. +The function `manipulation/standardizeGrRules.m`, which `importModel` calls, builds no parse tree. It converts ` AND ` and ` OR ` to lowercase and splits the rule at the space-bounded strings ` and ` and ` or `. It calls `findPotentialErrors` and returns the indices of rules that cannot be parsed or are not in DNF (`indexes2check`). A direct call lists these rules in one warning, and `importModel` passes `embedded` as `true`, which suppresses the warning. The function does not raise `RAVEN:badGrRule`. diff --git a/docs/raven3-migration/new-and-environment.md b/docs/raven3-migration/new-and-environment.md index 7755873..86023db 100644 --- a/docs/raven3-migration/new-and-environment.md +++ b/docs/raven3-migration/new-and-environment.md @@ -1,68 +1,82 @@ # New functionality, dependencies and error handling -Gap-filling, context-specific model extraction with `ftINIT`, new functions, MATLAB toolbox dependencies and error handling differ between RAVEN 2 and RAVEN 3. +RAVEN 3 changes gap-filling and the extraction of context-specific models with `ftINIT`, adds functions, depends on fewer MATLAB toolboxes, and handles errors in a different way. (ftinit)= ## Gap-filling and context-specific model extraction | Aspect | RAVEN 2 | RAVEN 3 | |---|---|---| -| `fillGaps` algorithm | One built-in mixed-integer linear programming (MILP) connectivity-maximisation strategy | The same strategy remains the default; the `algorithm` option adds `'fastLP'`, `'swiftLP'`, `'gapfillMILP'` and `'topological'` | -| `fillGaps` positional arguments | `allowNetProduction, useModelConstraints, supressWarnings, rxnScores` | Unchanged; new options are name-value-only | -| `ftINIT` tie-breaking | Depended on the solver seed | Optional `resolveTies` makes the answer less dependent on the solver seed | -| `ftINIT` MILP optimality proof | Escalating relative-gap search only | Optional `proveAbsGap` proves each stage to a fixed absolute gap in one solve | -| `ftINITInternalAlg` tie-break arguments | (not present) | `rankKeys` (14th argument) and `pinFluxes` (15th argument) | -| `getINITModel` and `runINIT` | Supported, no runtime notice | Supported, under `INIT/tINIT/`; one-time "legacy, use `ftINIT`" notice (see [Renamed and merged functions](#renamed-merged-and-consolidated-functions)) | -| `fitTasks` gap-fill mode | One behaviour (equivalent to `'merge'`) | `gapFillMode`: `'merge'` (default, same as in RAVEN 2) or `'preMerged'` | +| `fillGaps` algorithm | One built-in mixed-integer linear programming (MILP) strategy that maximises connectivity | The same strategy is the default (`algorithm` `'reference'`). The option `algorithm` adds `'fastLP'`, `'swiftLP'`, `'gapfillMILP'` and `'topological'` | +| `fillGaps` positional arguments | `allowNetProduction, useModelConstraints, supressWarnings, rxnScores` | The first four optional arguments are unchanged. `params` is the 7th and `algorithm` the 8th argument, followed by `epsilon`, `minGrowth` and `verbose`. All can also be given as name-value pairs | +| `ftINIT` tie-breaking | Depends on the solver seed, and the retries of the MILP set the seed to 1234. The dependence was not measured | The option `resolveTies` adds a best-effort selection among tied solutions. The resulting dependence on the seed was not measured | +| `ftINIT` proof of optimality | Retries with an increasing relative gap, which is derived from an absolute-gap setting and the last objective value | The option `proveAbsGap` proves each stage to a fixed absolute gap in one solve (`MIPGap` 0) | +| `ftINIT` positional arguments | `transcrData` to `verbose` are the 5th to 11th arguments | Unchanged. `resolveTies` and `proveAbsGap` follow `verbose` as the 12th and 13th arguments and are off by default | +| `fitTasks` gap-fill mode | One behaviour, equivalent to `'merge'` | `gapFillMode` is `'merge'` (default, as in RAVEN 2) or `'preMerged'` | | `fitTasks` outputs | `[outModel, addedRxns]` | `[outModel, addedRxns, failedTasks]` | -| Task requiring net production of a metabolite with no reactions | Boundary constraint dropped before gap-filling without a message; the task could be reported as passing incorrectly | Constraint preserved; the task is reported as failing if the reference model cannot produce the metabolite | -With `gapFillMode` set to `'preMerged'`, `fitTasks` expects a reference model that already contains the model's own reactions, so no merge is needed for each task, and `ftINITFillGaps` fills the gaps. +Positions in the migration pages count all arguments of the call, required ones included. The functions `getINITModel` and `runINIT` are described in [Renamed and merged functions](#renamed-merged-and-consolidated-functions). -The `algorithm` option of `fillGaps` selects among the following: +### Algorithms of `fillGaps` -- `'fastLP'` and `'swiftLP'`: FASTCORE-style and SWIFTCORE-style linear programming (LP) relaxation. -- `'gapfillMILP'`: growth-floor MILP with directionality repair. -- `'topological'`: breadth-first search producibility pre-screen, without model modification. +The option `algorithm` of `fillGaps` selects one of these algorithms. A value that is none of them runs the algorithm `'reference'`. -`fillGaps` calls `gapFillTopological` with the defaults of that function: the seeds are the metabolites of every reaction that has only substrates and a negative lower bound or only products and a positive upper bound, and the targets are the metabolites of the first objective reaction, substrates and products. `fillGaps` passes neither seeds nor targets, so a model without an objective reaction always raises `RAVEN:badInput` there. +- `'fastLP'` and `'swiftLP'` relax the problem to a linear program (LP) in the manner of FASTCORE and SWIFTCORE. +- `'gapfillMILP'` is a MILP with a growth floor. Reactions that are already in the draft model keep their bounds, so a reaction with the wrong direction is not changed. +- `'topological'` is a pre-screen of producibility by breadth-first search. It does not modify the model, requires a reference model and uses the first one. -With `resolveTies`, each `ftINIT` MILP step keeps the score optimum and then picks among the tied solutions in phases: the fewest kept removable reactions, the lowest summed ID rank of those (a merged reaction is ranked by the smallest original ID of its group), the fluxes with the least total absolute flux, and the fluxes with the least ID-rank-weighted backward flux. The arguments `resolveTies` and `proveAbsGap` are name-value-only and default to off; the positional arguments of `ftINIT` are unchanged. +The function `fillGaps` calls `gapFillTopological` with the defaults of that function. The seeds are the metabolites of the exchange reactions that `getExchangeRxns` classes as `'uptake'` or `'reverse'`. The targets are the metabolites of the first objective reaction, and a model without an objective reaction raises `RAVEN:badInput`. -In `ftINITInternalAlg`, `rankKeys` (default `model.rxns`) gives the key each reaction is ranked by in the ID-rank phases of `resolveTies`. Setting `pinFluxes` (default `true`) to `false` skips the two flux phases of `resolveTies`. `ftINIT` passes `false` in its last step. +### Gap-filling in `fitTasks` + +With `gapFillMode` set to `'preMerged'`, `fitTasks` expects a reference model that already contains the reactions of the model. No merge is needed for each task, and `ftINITFillGaps` fills the gaps. How `fitTasks` treats a task that requires net production of a metabolite without reactions was not checked. + +### Tie-breaking in `ftINIT` + +With `resolveTies`, each MILP step of `ftINIT` holds the score at its optimum and chooses among the tied solutions in phases, each with a time limit. The phases select, in this order: + +1. the fewest kept removable reactions; +2. the lowest summed ID rank of those reactions, where a merged reaction is ranked by the smallest original ID of its group; +3. the least total absolute flux; +4. the least backward flux, weighted by ID rank. + +The function `ftINITInternalAlg` has the arguments `resolveTies` (13th), `rankKeys` (14th, default `model.rxns`) and `pinFluxes` (15th, default `true`). With `pinFluxes` set to `false` the two flux phases are skipped, and `ftINIT` passes `false` in its last step. A step whose first solve stops at the time limit keeps that solution without the phases. The count phase and the rank phase get half of the step time limit each, and a phase that fails issues a warning. --- (new-functionality)= ## New functionality -RAVEN 3 adds the functions in the table; except where noted they do not change RAVEN 2 functions. +The table lists the functions that RAVEN 3 adds. The functions `randomSampling` and `parseScores` exist in RAVEN 2 and behave differently. -| Area | New functions | Added capability | +| Area | Functions | Capability | |---|---|---| | Flux sampling (`analysis/`) | `sampleACHR`, `sampleCHRR`, `sampleMaxVolEllipse`, `sampleChebyshevCenter`, `sampleWarmupPoints` | ACHR (hit-and-run) and CHRR (hit-and-run with rounding), two near-uniform Markov chain Monte Carlo (MCMC) samplers | -| Flux sampling (`analysis/`) | `randomSampling` (changed) | Becomes a dispatcher (`'method'` option) over the RAVEN 2 random-objective method (now `'randomObjective'`) and the two new samplers | -| Flux analysis and quality control (QC) | `compareFluxes` | Diffs flux vectors | +| Flux sampling (`analysis/`) | `randomSampling` (changed) | Dispatcher with the option `'method'`: `'achr'` (default), `'chrr'` and `'randomObjective'` (the RAVEN 2 method) | +| Flux analysis and quality control (QC) | `compareFluxes` | Compares flux vectors | | Flux analysis and QC | `getMinimalMedium` | Searches a minimal medium with a MILP | -| Flux analysis and QC | `traceFluxPath` | Traces the path of highest flux fraction between two reactions | +| Flux analysis and QC | `traceFluxPath` | Traces the path of the highest flux fraction between two reactions | | Flux analysis and QC | `walkFluxes` | Navigates the flux neighbourhood interactively | -| Flux analysis and QC | `modelSummary` | Prints a model and flux summary | -| Model comparison (`comparison/`) | `diffModels` | Diffs two models by ID: stoichiometry, bounds, objective, `grRules` (logical equality in disjunctive normal form, DNF), EC codes, metabolite properties | -| Manipulation | `findDuplicateRxns` | Standalone duplicate-stoichiometry query | -| Manipulation | `findPotentialErrors` | Function of its own (a local function of `core/standardizeGrRules.m` in RAVEN 2); parse-tree-based detection of `grRules` that are not in DNF and of rules that cannot be parsed (see [Parsing of gene-reaction rules](#gpr-parsing)) | -| Genome-based curation (`curation/`, new folder) | `downloadGenomeData`, `getGeneData`, `processProteinFastaFile` | Pipeline from an NCBI genome accession to a gene ID mapping table | -| Genome-based curation (`curation/`, new folder) | `renameModelGenes` | Bulk gene renaming in a model | -| Genome-based curation (`curation/`, new folder) | `curateModelFromTables` | Bulk curation of metabolites, reactions and genes from `.tsv` files against an existing model | -| Biomass composition (`biomass/`) | `getBiomassFractions`, `scaleBiomassFraction`, `scaleBiomassPseudoreaction`, `setGAM` | Organism-agnostic biomass-composition and growth-associated-maintenance (GAM) calibration; `tutorials/tutorial7.m` demonstrates the workflow | -| Localisation (`localization/`) | `assignCompartments`, `defaultCompartmentMap`, `getUniProtScores` | Deterministic MILP compartment assignment (alternative to the heuristic `predictLocalization`) | -| Localisation (`localization/`) | `parseScores` (changed) | Gains `'mulocdeep'`, `'compartments'` and `'uniprot'`; `'deeploc'` reads DeepLoc 2 output; the default predictor is `'deeploc'` (RAVEN 2: `'wolf'`); WoLF PSORT is removed (see [Backward-incompatible changes](#backward-incompatible-changes)) | -| Annotation (`annotation/`) | `assignSBOterms`, `loadDeltaGCSV`, `saveDeltaGCSV` | Organism-agnostic SBO term assignment; ΔG thermodynamic data CSV round-trip | -| Conditions (`conditions/`, new folder) | `applyCondition` | Applies a data-driven growth or media condition, given as a YAML file, to a model | -| Internal infrastructure | `ravenModelFields` | Central registry of every 1-D model field (entity type, default value), used internally by `permuteModel`, `removeReactions` and `curateModelFromTables` | - -`sampleCHRR` rounds the polytope before sampling, so mixing stays fast on strongly elongated polytopes, such as enzyme-constrained models, unlike `sampleACHR`. - -`diffModels` is separate from `compareMultipleModels`, the existing overlap-style comparison. `applyCondition` applies the parts of a condition in a fixed sequence: exchange bounds, optional cofactor-pseudoreaction edits, optional biomass-stoichiometry deltas and per-reaction bound overrides. +| Flux analysis and QC | `modelSummary` (`queries/`) | Prints a summary of a model and its fluxes | +| Model comparison (`comparison/`) | `diffModels` | Compares two models by ID: stoichiometry, bounds, objective, `grRules`, EC codes and metabolite properties. It is separate from the overlap comparison of `compareMultipleModels` | +| Manipulation | `findDuplicateRxns` | Query for reactions with the same stoichiometry | +| Genome-based curation (`curation/`) | `downloadGenomeData`, `getGeneData`, `processProteinFastaFile` | Pipeline from an NCBI genome accession to a gene ID mapping table | +| Genome-based curation (`curation/`) | `renameModelGenes` | Renames genes in bulk | +| Genome-based curation (`curation/`) | `curateModelFromTables` | Curates metabolites, reactions and genes of an existing model in bulk from `.tsv` files | +| Biomass composition (`biomass/`) | `getBiomassFractions`, `scaleBiomassFraction`, `scaleBiomassPseudoreaction`, `setGAM` | Calibration of the biomass composition and of the growth-associated maintenance (GAM). `tutorials/tutorial7.m` demonstrates the workflow | +| Localisation (`localization/`) | `assignCompartments`, `defaultCompartmentMap`, `getUniProtScores` | Deterministic assignment of compartments by MILP, as an alternative to the heuristic `predictLocalization` | +| Localisation (`localization/`) | `parseScores` (changed) | Adds `'mulocdeep'`, `'compartments'` and `'uniprot'`. The predictor `'deeploc'` reads DeepLoc 2 output and is the default (RAVEN 2: `'wolf'`). WoLF PSORT is removed (see [Backward-incompatible changes](#backward-incompatible-changes)) | +| Annotation (`annotation/`) | `assignSBOterms`, `loadDeltaGCSV`, `saveDeltaGCSV` | Assignment of SBO terms; CSV I/O of ΔG data | +| Conditions (`conditions/`) | `applyCondition` | Applies a condition from a YAML file to a model in a fixed order: exchange bounds, edits of cofactor pseudoreactions, changes of the biomass stoichiometry, and bound overrides per reaction | +| Infrastructure | `ravenModelFields` | Registry of every one-dimensional model field (entity type, default value), used by `permuteModel`, `removeReactions` and `curateModelFromTables` | +| Infrastructure | `progressReport`, `setRavenProgress` | Progress reporting with a selectable backend, for example a log file on compute clusters | +| Infrastructure | `splitProbForConditioning` (`solver/`) | Splits an optimisation problem to improve its conditioning | + +The call `randomSampling(model, n)` runs ACHR, where RAVEN 2 ran its own method. Adding `'method', 'randomObjective'` runs the RAVEN 2 method. The function also gains the outputs and options `info`, `seed`, `thinning` and `nBurnin`. + +The function documentation of `sampleCHRR` states that the polytope is rounded before sampling and that mixing stays fast on strongly elongated polytopes such as enzyme-constrained models. It states that `sampleACHR` does not round. The mixing speed was not measured. + +[Parsing of gene-reaction rules](#gpr-parsing) describes `findPotentialErrors`, the parser functions and the rule comparison of `diffModels`. [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction) describes the reconstruction and curation functions for KEGG, homology and genome data. --- @@ -71,26 +85,29 @@ RAVEN 3 adds the functions in the table; except where noted they do not change R | Aspect | RAVEN 2 | RAVEN 3 | |---|---|---| -| MATLAB version | No pinned or documented minimum (guidance: avoid very new MATLAB syntax, to accommodate older installs) | Tested in continuous integration with R2024b | -| Statistics Toolbox | Required (`combnk`) | Not required (`nchoosek`, base MATLAB) | -| Bioinformatics Toolbox | Required (`fastaread` and `fastawrite`) | Not required (`io/readFasta.m` and `io/writeFasta.m`) | -| Optimization Toolbox | Required for `qMOMA` and `solveQP` | Not required (both removed) | -| Excel I/O | Bundled Apache POI (Java) | Dependency-free OOXML writer | -| LP and MILP solver | Required (bundled `glpk`, or Gurobi, SCIP or COBRA) | Unchanged, still required | -| External binaries | Committed to the repository, available offline immediately after clone | BLAST+, DIAMOND and HMMER are fetched on demand on first use (per-platform ZIPs); an offline `*-binaries` release bundle is available for air-gapped installs | -| KEGG HMM libraries and reference data | Manual download or local build required | Fetched on demand, cached locally after first use | -| Installation self-check | `installation/checkInstallation.m` ran the full test suite to populate its pass/fail table | `checkRaven` in the repository root runs targeted checks and downloads missing binaries without a prompt (see [Renamed and merged functions](#renamed-merged-and-consolidated-functions)) | - -All downloads come from releases of `toolbox-assets`. KEGG data comes from the release tag that `keggDataVersion()` returns (`src/reconstruction/kegg/KEGG_VERSION.md`, tag `kegg118`). The binaries come from fixed tags set in `src/installation/downloadRavenBinaries.m`: `blast-2.17.0`, `diamond-2.1.17`, `hmmer-3.4.0` and, on Windows, `hmmer-3.3.2`. The checksums of downloaded files come from `manifest.json` of the `manifest-v3` release, read at download time. - -`checkRaven` runs a small import and export round-trip and a two-reaction LP per solver, without invoking the full test classes. - -A RAVEN 3 clone fetches binaries and data on demand, which requires network access. +| MATLAB version | No minimum stated. `readYAMLmodel` has a fallback for releases before R2020b (`readlines`) | No minimum stated, and the code has no check that rejects an older release. `readYAMLmodel` keeps the fallback for releases before R2020b, `parseTaskList` calls `readcell` (R2019a) for Excel task lists, and `addIdentifierPrefix` calls `regexpPattern` (R2020b). Continuous integration uses a single release, R2024b, and older releases were not tested | +| Statistics and Machine Learning Toolbox | Required by `compareMultipleModels` (`mdscale`, `pdist`, `squareform` and `tsne`, with an error without them) and by `compareRxnsGenesMetsComps` (`combnk`) | Optional in `compareMultipleModels`. Without it the projection and the clustering of the heatmap are skipped | +| Bioinformatics Toolbox | Required by `getKEGGModelForOrganism` when it builds missing per-KO alignments from a local KEGG dump (`fastaread` and `fastawrite`) | Not required (`io/readFasta.m` and `io/writeFasta.m`) | +| Optimization Toolbox | Required for `qMOMA` and `solveQP` (`quadprog`) | Not required, because both functions are removed | +| Parallel Computing Toolbox | Optional. The `parfor` loops, for example in `getAllowedBounds` and `randomSampling`, run serially without it | Optional. `parallelWorkersRAVEN` returns 0 workers when `runParallel` is false, which runs the `parfor` loops as plain loops | +| Excel I/O | Bundled Apache POI (Java) | OOXML writer without dependencies | +| LP and MILP solver | Required: bundled `glpk`, or Gurobi, SCIP or COBRA | Unchanged | +| External binaries | Committed to the repository (BLAST+, DIAMOND, HMMER) and available offline after cloning | Fetched on first use as ZIP files per platform into `src/external/`. For air-gapped installs the workflow `.github/workflows/release-bundle.yml` builds `RAVEN--full.zip` with the binaries of all platforms and attaches it to a published release. Whether a given release carries it was not checked | +| KEGG HMM libraries and reference data | Manual download or local build | Fetched on first use and cached | +| Installation self-check | `installation/checkInstallation.m` runs `importExportTests`, `solverTests` and the test files of BLAST+, DIAMOND, HMMER, CD-HIT and MAFFT to fill its table of passed and failed tests | `checkRaven` in the repository root imports `empty.xml`, exports SBML, imports and exports YAML, exports Excel, solves a two-reaction LP per solver and downloads missing binaries without a prompt. `checkInstallation` forwards to it (see [Renamed and merged functions](#renamed-merged-and-consolidated-functions)) | + +KEGG data, binaries and the checksum manifest come from releases of `toolbox-assets`. Other downloads have other sources. A RAVEN 3 clone needs network access for these downloads. + +- KEGG data comes from the release tag that `keggDataVersion()` returns (`src/reconstruction/kegg/KEGG_VERSION.md`, tag `kegg118`). +- The binaries come from the fixed tags in `src/installation/downloadRavenBinaries.m`: `blast-2.17.0`, `diamond-2.1.17`, `hmmer-3.4.0` and, on Windows, `hmmer-3.3.2`. +- The checksums of downloaded files come from `manifest.json` of the `manifest-v3` release, which is read at download time. +- `setRavenSolver('scip')` on Windows downloads `scip_mex_win.zip` from the RAVEN v2.8.6 release on GitHub into `src/external/scip`. The repository holds no SCIP files, and other platforms need a SCIP installation made by hand. +- `getUniProtScores` queries rest.uniprot.org, `downloadGenomeData` downloads from NCBI, and `checkRaven` reads the version file from GitHub. --- ## Error and warning handling -RAVEN 3 raises errors and issues warnings with native `error()` and `warning()` calls that carry identifiers, mostly `RAVEN:*`. The identifiers are additive, and message text is preserved for most existing checks, so `try/catch` blocks that match on message text keep working for those checks. [Backward-incompatible changes](#backward-incompatible-changes) gives the details of the `dispEM` replacement, and [Removed functionality](#removed-functionality) names the new `utils/ravenList.m` helper. +RAVEN 3 raises errors and issues warnings with `error()` and `warning()`. Outside `src/external` there are 411 `error` calls, counted as lines that contain `error(` and do not start with a comment. Of these, 301 carry an identifier, and 242 of those are in the `RAVEN:*` namespace. By the same rule there are 94 `warning` calls, of which 79 carry an identifier and 72 are in the `RAVEN:*` namespace. Whether the message texts of the RAVEN 2 checks are preserved was not checked. A `try/catch` block that matches message text can be replaced by a match on the identifier where one exists. [Backward-incompatible changes](#backward-incompatible-changes) describes the replacement of `dispEM`, and [Removed functionality](#removed-functionality) names the helper `utils/ravenList.m`. -RAVEN 3 adds an output argument to `queries/checkModelStruct.m`: `issues = checkModelStruct(model)` returns a structured array of the problems (`category`, `target` and `message`) for programmatic model QC and raises no error. Without an output argument it raises errors or issues warnings directly, as in RAVEN 2. +The call `issues = checkModelStruct(model)` returns a structure array of the problems (`category`, `target` and `message`) for programmatic model QC and raises no error. Without an output argument the function raises errors or issues warnings as in RAVEN 2. The options `throwErrors` and `trimWarnings`, which are positional in RAVEN 2, can also be given as name-value pairs. diff --git a/docs/raven3-migration/structure-and-renames.md b/docs/raven3-migration/structure-and-renames.md index 780acf1..8ac6570 100644 --- a/docs/raven3-migration/structure-and-renames.md +++ b/docs/raven3-migration/structure-and-renames.md @@ -1,91 +1,105 @@ # Structure, removals and renames -RAVEN 3 organises functions in folders named by function area, removes RAVEN 2 functions and subsystems, and renames or merges other functions. +RAVEN 3 sorts its functions into folders by function area. It removes 65 functions of RAVEN 2, merges or renames 8, and keeps 5 RAVEN 2 names as wrappers. ## Repository reorganisation -RAVEN 3 replaces the top-level layout of RAVEN 2 (`core/`, `external/`, `hpa/`, `io/`, `legacy/`, `pathway/`, `plotting/`, `solver/`, `struct_conversion/`, `installation/`, `utils/`) with folders named by function area: +The code of RAVEN 3 is in `src/`. RAVEN 2 kept its functions in the top-level folders `core/`, `external/`, `INIT/`, `hpa/`, `io/`, `legacy/`, `pathway/`, `plotting/`, `solver/`, `struct_conversion/`, `installation/` and `utils/`, and its bundled third-party code in `software/`. Other top-level folders moved as well: the third-party code of `software/` is in `src/external/`, `tutorial/` is `tutorials/`, `doc/` is `docs/`, and `testing/` is `tests/function_tests/`. -| RAVEN 3 folder | Contents | RAVEN 2 source | +| RAVEN 3 folder (under `src/`) | Contents | RAVEN 2 source | |---|---|---| -| `INIT/` | `ftINIT`, `INIT/tINIT/` for `getINITModel` and `runINIT`, and HPA parsing, expression parsing and gene scoring (`parseHPA`, `parseHPArna`, `scoreModel`) | `INIT/`, `hpa/` | -| `analysis/` | Analyses related to flux balance analysis (FBA): flux variability analysis (FVA), sampling, FSEOF, OptKnock, robustness, production envelope, flux comparison and tracing | `core/` | +| `INIT/` | `ftINIT`, `INIT/tINIT/` with `getINITModel` and `runINIT` (the legacy tINIT method), and the parsing of HPA and expression data and the scoring of genes (`parseHPA`, `parseHPArna`, `scoreModel`) | `INIT/`, `hpa/` | +| `analysis/` | Analyses based on flux balance analysis (FBA): flux variability analysis (FVA), sampling, FSEOF, OptKnock, robustness, production envelope, flux comparison and flux tracing | `core/` | | `annotation/` | MIRIAM editing, SBO term assignment, ΔG CSV I/O | `struct_conversion/` (and new functions) | -| `biomass/` | Biomass composition fitting and scaling, growth-associated maintenance (GAM) | `core/` (and new functions) | -| `comparison/` | Multi-model comparison, structured model diffing | `core/` (and new functions) | -| `conditions/` | Applying a data-driven growth or media condition to a model | New folder | +| `biomass/` | Fitting and scaling of the biomass composition, growth-associated maintenance (GAM) | `core/` (and new functions) | +| `comparison/` | Comparison of several models, structured model diffing | `core/` (and new functions) | +| `conditions/` | Applying a growth or media condition from data to a model | New folder | | `conversion/` | RAVEN⇄COBRA conversion, identifier prefixing, field ordering | `struct_conversion/` | -| `curation/` | Genome-annotation-driven gene and model curation | New folder | -| `external/` | Bundled third-party code (GLPKmex, libSBML, SCIP and m2html) and the download target of the BLAST+, DIAMOND and HMMER binaries | `external/` | -| `gapfilling/` | `fillGaps` and its pluggable algorithms, task fitting, leak-metabolite and exchange checks | `core/` (and new functions) | -| `installation/` | Path setup, on-demand binary and data download | `installation/` | -| `io/` | Model import and export (SBML, YAML, Excel), FASTA I/O | `io/` | -| `localization/` | Subcellular localisation prediction and scoring, compartment assignment | `external/` (and new functions) | -| `manipulation/` | Structural edits: add, remove and change reactions, metabolites and genes; merge, sort, simplify | `core/` | -| `queries/` | Non-mutating lookups: indexes, exchange and transport reactions, stoichiometry construction, model checking | `core/` | -| `reconstruction/` | KEGG- and homology-based draft reconstruction | `external/` | -| `solver/` | LP and MILP solving | `solver/` | -| `tasks/` | Metabolic task list parsing and checking | `core/` | -| `tests/function_tests/` (repository root, outside `src/`) | `matlab.unittest` suite | `testing/` | -| `utils/` | Generic helpers, including the new argument-parsing and registry infrastructure | `utils/` | +| `curation/` | Gene and model curation driven by a genome annotation | New folder | +| `external/` | Bundled third-party code (GLPKmex, libSBML and m2html) and the target folder `src/external/` for the downloaded BLAST+, DIAMOND and HMMER binaries. `src/external/scip/` holds no files, and `setRavenSolver('scip')` downloads the SCIP MEX file into it on Windows (see [Toolbox dependencies and installation](#toolbox-deps)) | `software/` | +| `gapfilling/` | `fillGaps` and its algorithms, task fitting, checks for leak metabolites and exchange | `core/`, `INIT/` (and new functions) | +| `installation/` | Path setup, download of binaries and data on demand | `installation/`, `core/` (`findRAVENroot`), `external/` (`updateDocumentation`) | +| `io/` | Model import and export (SBML, YAML, Excel), FASTA I/O | `io/` (and new functions) | +| `localization/` | Prediction and scoring of subcellular localisation, compartment assignment | `external/`, `core/` (`predictLocalization`) (and new functions) | +| `manipulation/` | Structural edits: adding, removing and changing reactions, metabolites and genes; merging, sorting and simplifying | `core/`, `io/` (`closeModel`, `sortIdentifiers`), `INIT/` (`removeLowScoreGenes`, `reverseRxns`) | +| `queries/` | Lookups that do not change the model: indexes, exchange and transport reactions, construction of stoichiometry, model checking | `core/` (and new functions) | +| `reconstruction/` | Draft reconstruction from KEGG and from homology | `external/`, `core/` (`getModelFromHomology`, `guessComposition`) | +| `solver/` | LP and MILP solving | `solver/`, `core/` (`getMinNrFluxes`) (and new functions) | +| `tasks/` | Parsing and checking of metabolic task lists | `core/` | +| `utils/` | General helpers, among them the argument parsing and the registry | `utils/`, `core/` (`convertCharArray`, `printOrange`) | | `deprecated/` | Wrappers for merged and renamed functions | New folder | +| `tests/function_tests/` (repository root, outside `src/`) | Class-based `matlab.unittest` suite. 23 of 25 classes derive from `RavenTestCase`, and `tBinaries` and `tParseRAVENargs` derive from `matlab.unittest.TestCase` | `testing/` (function-based tests built with `functiontests`) | -Removed without a successor folder: the subfolder `external/metacyc/` (MetaCyc reconstruction; `external/` itself remains), `pathway/` and `plotting/` (visualisation), `legacy/` (CellDesigner import and other pre-2.0 code), and the Excel-POI helpers of `struct_conversion/`. +The files `checkRaven.m` and `checkInstallation.m` (a wrapper) are in the repository root. -A script that calls RAVEN functions by name keeps working when the whole RAVEN folder tree, including subfolders, is on the MATLAB path (through `addRavenToUserPath` or `addpath(genpath(...))`), except for calls to removed functions and to merged functions without a wrapper. MATLAB resolves functions by name across the whole path, so moving a file to a new subfolder does not break the call. The reorganisation affects a script that (a) uses a hard-coded path to a specific `.m` file, (b) calls a removed function or a merged function without a wrapper, or (c) shadows a RAVEN function name with its own function. +Some code has no successor folder: the subfolder `external/metacyc/` with the MetaCyc function `external/combineMetaCycKEGGModels.m`, `pathway/` and `plotting/` (visualisation), `legacy/` (CellDesigner import and other pre-2.0 code), and the Excel-POI helpers of `io/` together with the Apache POI libraries in `software/apache-poi`. -The full mapping of moved files (RAVEN 2 path to RAVEN 3 path) is not reproduced here because it is large and rarely something that needs action in a script. +A script that calls RAVEN functions by name keeps working when the whole RAVEN folder tree, subfolders included, is on the MATLAB path (through `addRavenToUserPath` or `addpath(genpath(...))`), because MATLAB resolves functions by name across the path. The reorganisation affects a script in three cases: it uses a hard-coded path to a specific `.m` file, it calls a removed function or a merged function that has no wrapper, or it contains a function with the name of a RAVEN 3 function. `checkFunctionUniqueness` warns about the last case. Individual moved files are not mapped. --- (removed-functionality)= ## Removed functionality -RAVEN 3 removed the functions and subsystems in the table to make the codebase smaller. None of them has a call site left in RAVEN 3. +The table lists all 65 functions of RAVEN 2 that RAVEN 3 removes without a successor of the same name. The counts per RAVEN 2 folder are `pathway/` 6, `plotting/` 10, `external/` 12 (8 for MetaCyc, 4 others), `legacy/` 11, `io/` 12, `core/` 8, `utils/` 3, `solver/` 2 and `INIT/` 1. No function of RAVEN 3 calls a removed function. The name `getToolboxVersion` exists in RAVEN 3 as a local function of `exportForGit`. | Removed | Former purpose | Replacement | |---|---|---| -| `pathway/*`, `plotting/*` (16 functions, such as `drawMap`, `drawPathway`, `colorPathway`, `colorSubsystem`, `markPathwayWithFluxes`, `markPathwayWithExpression`, `setOmicDataToRxns` and `plotAdditionalInfo`) | Metabolic map and pathway visualisation | None in RAVEN 3. Alternatives are the COBRA Toolbox, Escher and export of flux and omics data for external plotting. | -| `external/metacyc/*` (7 functions: `getModelFromMetaCyc`, `getRxnsFromMetaCyc`, `getMetsFromMetaCyc`, `getEnzymesFromMetaCyc`, `addSpontaneousRxns`, `linkMetaCycKEGGRxns`, `combineMetaCycKEGGModels`) | MetaCyc-based draft reconstruction | None; KEGG- and homology-based reconstruction remain the supported reconstruction methods. | -| `legacy/*` (CellDesigner import, `changeGeneAssoc`, the bundled `xml_toolbox`) | Pre-2.0 compatibility code | None; considered dead. | -| `io/importExcelModel.m`, `io/loadWorkbook.m`, `io/loadSheet.m`, `io/writeSheet.m`, `io/SBMLFromExcel.m`, `io/addJavaPaths.m`, `io/startup.m` | Apache-POI-based Excel model I/O | `io/writeExcel.m` (dependency-free OOXML writer) for export; `curation/curateModelFromTables.m` for tabular curation. No full Excel-template import remains; see [Excel I/O](#excel-io). | -| `io/exportModelToSIF.m`, `io/exportToTabDelimited.m` | SIF graph export; tab-delimited model export | Dead or orphaned code with no callers. | -| `io/getFullPath.m`, `io/getMD5Hash.m`, `io/getToolboxVersion.m` | Path canonicalisation; file MD5 hashing; toolbox git-version lookup | Inlined where still needed (for example MD5 via Java's `MessageDigest` inside `getBlast`), or merged into `exportForGit`. | -| `core/dispEM.m` | Universal error and warning display | Native `warning()` and `error()` plus new `utils/ravenList.m` for formatted item lists. See [Backward-incompatible changes](#backward-incompatible-changes). | -| `core/followFluxes.m`, `core/followChanged.m` | Flux-change reporting relative to a reference solution | `queries/printFluxes.m` covers the cutoff filtering; nothing directly replaces the reference-flux comparison. | -| `core/printModel.m` | Printed reactions to screen or file | `queries/printModelStats.m` and `queries/printFluxes.m` cover related summary and flux printing (see the note below the table). | -| `core/getMetsInComp.m` | Returned metabolite indices in a given compartment | Inline it as `model.metComps == compIndex`. | -| `core/mapCompartments.m` | Merged and split the score columns of a localisation-score structure by compartment rules | Superseded by the `compartmentMap` option of `parseScores` and `getUniProtScores` (default `defaultCompartmentMap`); merged columns take the maximum score, and splitting is not supported. | -| `core/getExpressionStructure.m` | Loaded an expression-experiment structure from an ad hoc Excel format | Superseded by `INIT/parseHPA.m` and `INIT/parseHPArna.m` for the supported expression-data workflow. | -| `core/checkRxn.m` | Per-reaction reactant and product synthesisability debugging | No direct successor; use `gapfilling/findLeakMetabolite.m`, `gapfilling/canExchange.m` or `gapfilling/checkProduction.m`. | -| `external/getWoLFScores.m` | WoLF PSORT score parsing | None; see [Backward-incompatible changes](#backward-incompatible-changes). | -| `external/kegg/{getGenesFromKEGG,getMetsFromKEGG,getRxnsFromKEGG,constructMultiFasta,getWSLpath}.m`, `external/getBlastFromExcel.m` | Local-KEGG-FTP-dump parsing; per-KO multi-FASTA assembly; WSL path translation; Excel-based BLAST import | Superseded by the KEGG pipeline based on `toolbox-assets` artefacts. See [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction). | -| `solver/qMOMA.m`, `solver/solveQP.m` | MOMA-based analyses (quadratic programming) | Removed; RAVEN 3 has no replacement based on quadratic programming. See [Backward-incompatible changes](#backward-incompatible-changes) for the arguments removed from `findGeneDeletions`. | -| `testing/unit_tests/*.m` (script-based tests) | Ad hoc test scripts | `tests/function_tests/*.m` in the repository root, a `matlab.unittest`-class-based suite. Relevant to contributors only; `runRAVENtests` usage is unchanged for end users. | - -`printFluxes` prints the reactions of a flux vector with an absolute flux at or above `cutOffFlux` (default `10^-8`), one line per reaction; it prints exchange reactions only unless `onlyExchange` is `false`. A dump of all reactions of a model without fluxes has no direct replacement. +| `pathway/colorPathway`, `pathway/getPathwayDimensions`, `pathway/mapPathwayRxnNames`, `pathway/markPathwayWithExpression`, `pathway/markPathwayWithFluxes`, `pathway/trimPathway` (6 functions) | Handling and colouring of metabolic pathway maps | None in RAVEN 3. Alternatives are the COBRA Toolbox, Escher, and the export of flux and omics data for plotting elsewhere | +| `plotting/colorSubsystem`, `plotting/drawMap`, `plotting/drawPathway`, `plotting/getColorCodes`, `plotting/getObjectiveString`, `plotting/plotAdditionalInfo`, `plotting/plotLabels`, `plotting/setColorToMapRxns`, `plotting/setOmicDataToRxns`, `plotting/setTitle` (10 functions) | Drawing of metabolic maps and colouring with omics data | None, as above | +| `external/metacyc/{getModelFromMetaCyc,getRxnsFromMetaCyc,getMetsFromMetaCyc,getEnzymesFromMetaCyc,getMetaCycModelForOrganism,addSpontaneousRxns,linkMetaCycKEGGRxns}`, `external/combineMetaCycKEGGModels` (8 functions) | Draft reconstruction from MetaCyc | None. The reconstruction methods of RAVEN 3 are the KEGG-based and the homology-based method | +| `legacy/core/changeGeneAssoc`, `legacy/external/constructPathwayFromCelldesigner`, and the nine functions of the bundled `xml_toolbox` (`xml_format`, `xml_formatany`, `xml_help`, `xml_load`, `xml_parse`, `xml_parseany`, `xml_save`, `xml_strsplit`, `xml_unblank`) (11 functions) | CellDesigner import and pre-2.0 compatibility code | None | +| `io/importExcelModel`, `io/loadWorkbook`, `io/loadSheet`, `io/writeSheet`, `io/SBMLFromExcel`, `io/addJavaPaths`, `io/startup` (7 functions) | Excel model I/O through Apache POI | `io/writeExcel.m` (OOXML writer without dependencies) for export and `curation/curateModelFromTables.m` for tabular curation. RAVEN 3 cannot import a full Excel template (see [Excel I/O](#excel-io)) | +| `io/exportModelToSIF`, `io/exportToTabDelimited` | Export of a SIF graph; tab-delimited export of a model | None. The sole caller of `exportToTabDelimited` was the fallback of `exportToExcelFormat` for a path without file name, which is removed | +| `io/getFullPath`, `io/getMD5Hash`, `io/getToolboxVersion` | Canonical file path; MD5 hash of a file; git version of the toolbox | Inlined where needed: the MD5 hash through Java `MessageDigest` in `getBlast`, and the version as a local function of `exportForGit` | +| `core/dispEM` | Display of errors and warnings (388 lines call it in RAVEN 2, counted outside `doc/` and `testing/`) | `warning()` and `error()`, and `utils/ravenList.m` for formatted lists of items (see [Backward-incompatible changes](#backward-incompatible-changes)) | +| `core/followFluxes`, `core/followChanged` | Report of flux changes relative to a reference solution | `analysis/compareFluxes.m` compares two flux vectors of one model (reactions turned on or off, flipped and changed, with `cutoff` and `metaboliteList`) and replaces `core/followChanged`. `queries/printFluxes.m` prints the fluxes of one vector above a cut-off | +| `core/printModel` | Printing of reactions to the screen or to a file | `queries/printModelStats.m` and `queries/printFluxes.m` cover model summaries and the printing of fluxes (see below the table) | +| `core/getMetsInComp` | Indices of the metabolites in a compartment | `model.metComps == compIndex` | +| `core/mapCompartments` | Merging and splitting of the score columns of a localisation-score structure by compartment rules | The `compartmentMap` option of `parseScores` and `getUniProtScores`. The default map `defaultCompartmentMap` applies to `getUniProtScores` and to `parseScores` with the `'uniprot'` predictor. Elsewhere, without a map, every label stays unchanged. With a map, a label that the map does not contain is dropped together with its score column. Merged columns take the maximum score, and splitting is not supported | +| `core/getExpressionStructure` | Loading of an expression structure from an ad hoc Excel format | `INIT/parseHPA.m` and `INIT/parseHPArna.m` read Human Protein Atlas files. Expression tables from other sources have to be assembled by hand as the `arrayData` structure documented in `scoreModel` | +| `core/checkRxn` | Debugging of the synthesisability of one reaction | No direct successor. `gapfilling/findLeakMetabolite.m`, `gapfilling/canExchange.m` and `gapfilling/checkProduction.m` cover related checks | +| `external/getWoLFScores` | Parsing of WoLF PSORT scores | None. `parseScores` with `'wolf'` raises `RAVEN:badInput` (see [Backward-incompatible changes](#backward-incompatible-changes)) | +| `external/kegg/constructMultiFasta`, `external/kegg/getWSLpath`, `external/getBlastFromExcel` | Assembly of a multi-FASTA file per KO; translation of WSL paths; import of BLAST results from Excel | The KEGG pipeline based on the artefacts of `toolbox-assets` (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)) | +| `solver/qMOMA`, `solver/solveQP` | MOMA analyses and quadratic programming | None. RAVEN 3 has no quadratic programming. The arguments removed from `findGeneDeletions` are in [Backward-incompatible changes](#backward-incompatible-changes) | +| `utils/emptyOrLogicalScalar`, `utils/emptyOrTextOrCellOfText`, `utils/emptyOrTextScalar` | Validators of arguments | `utils/parseRAVENargs.m` | +| `INIT/getExprForRxnScore` | Reaction scores from expression data | A private helper of the test suite in `tests/function_tests/private/` | + +The function `printFluxes` prints the reactions of a flux vector whose absolute flux is at least `cutOffFlux` (default `10^-8`), one line per reaction. It prints exchange reactions unless `onlyExchange` is `false`. A listing of all reactions of a model without fluxes has no direct replacement. --- (renamed-merged-and-consolidated-functions)= ## Renamed and merged functions -RAVEN 3 introduces the `deprecated/` folder convention. In `deprecated/`, `makeSomething`, `consumeSomething`, `canProduce` and `canConsume` stay callable under their old names as wrappers that issue a one-time `RAVEN:deprecated` warning ("will be removed in the next major release") and forward to the new function. The other functions in the table have no wrapper, except `checkInstallation`, which has its own row. +The folder `deprecated/` holds `makeSomething`, `consumeSomething`, `canProduce` and `canConsume`. These wrappers keep the RAVEN 2 names callable, issue a `RAVEN:deprecated` warning once ("will be removed in the next major release") and forward to the RAVEN 3 function. The other functions in the table have no wrapper, except `checkInstallation`, which has a row of its own. | RAVEN 2 functions | RAVEN 3 function | Notes | |---|---|---| -| `makeSomething(model, ignoreMets, isNames, minNrFluxes, allowExcretion, params, ignoreIntBounds)` | `findLeakMetabolite(model, 'produce', ...)` | Argument order maps 1:1; the wrapper forwards `varargin` unchanged. | -| `consumeSomething(model, ignoreMets, isNames, minNrFluxes, params, ignoreIntBounds)` | `findLeakMetabolite(model, 'consume', ...)` | The old positional order of `consumeSomething` has no `allowExcretion` slot, so the wrapper remaps the arguments by name to keep `params` and `ignoreIntBounds` intact. | -| `canProduce(model, mets)` | `canExchange(model, 'produce', 'mets', mets)` | None. | -| `canConsume(model, mets)` | `canExchange(model, 'consume', 'mets', mets)` | None. | -| `compareRxnsGenesMetsComps` | Merged into `comparison/compareMultipleModels.m` | The reaction, gene, metabolite and compartment overlap output is one part of the output of `compareMultipleModels`; no separate function or wrapper remains. | -| `INIT/scoreComplexModel.m` | `INIT/scoreModel.m` | Merged; no wrapper remains. | -| `INIT/ftINITFillGapsMILP.m` | Merged into `solver/getMinNrFluxes.m` | Available through the `resolveTies` option of `getMinNrFluxes`; the `proveAbsGap` option belongs to `ftINIT`. | -| `INIT/ftINITFillGapsForAllTasks.m` | Merged into `gapfilling/ftINITFillGaps.m` and the `'preMerged'` gap-fill mode of `fitTasks` | None. | -| `getGenesFromKEGG`, `getMetsFromKEGG` and `getRxnsFromKEGG` | `reconstruction/kegg/readKEGGTable.m` and the `toolbox-assets` KEGG artefacts | The data source changed, so the signatures are not compatible; see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction). | -| `core/parallelPoolRAVEN.m` | `utils/parallelWorkersRAVEN.m` | Parallel Computing Toolbox pool setup, renamed and rewritten. `parallelWorkersRAVEN` returns a `parfor` worker count (`0` or `Inf`) and starts no pool; the old function returned `[ps, oldPoolAutoCreate]`. | -| `checkInstallation` | `checkRaven` | Moved from `installation/checkInstallation.m` to `checkRaven.m` in the repository root; the calling signature is identical. `checkInstallation` remains as a deprecated forwarding wrapper that issues a `RAVEN:deprecated` warning. | - -RAVEN 2 had both `INIT/scoreComplexModel.m` and `hpa/scoreModel.m`; RAVEN 3 merged them into one `scoreModel`, called with different settings from tINIT than from ftINIT. A RAVEN 2 call to `scoreModel` (`hpa/scoreModel.m`, checked against v2.11.3) changes: RAVEN 2 scored a reaction by the best or average score over all its genes (`multipleGeneScoring`), while RAVEN 3 applies `isozymeScoring` (default `'max'`) to the `or` parts and `complexScoring` (default `'min'`) to the `and` parts of the rule. A reaction with an `and` rule therefore receives a different default score. The positional arguments after `noGeneScore` changed: the 7th to 9th arguments of a RAVEN 2 call (`multipleGeneScoring`, `multipleCellScoring`, `hpaLevelScores`) are read as `isozymeScoring`, `complexScoring` and `multipleCellScoring`, `multipleCellScoring` accepts `'max'` and `'average'` (RAVEN 2: `'best'` and `'average'`), and `hpaLevelScores` becomes the 10th positional argument. RAVEN 3 adds `dataPrecedence` (default `'gene'`); see [Backward-incompatible changes](#backward-incompatible-changes). A call to `scoreComplexModel` has to move to `scoreModel` and pass the settings that reproduce the old behaviour (a single operator for both `and` and `or`, `dataPrecedence` `'reaction'`). - -`getINITModel` and `runINIT` (under `INIT/tINIT/`) are marked legacy but supported indefinitely; unlike the functions in `deprecated/`, they are not scheduled for removal. `utils/legacyMethodNotice.m` issues a one-time, silenceable `RAVEN:legacyMethod` warning that points to `ftINIT`. Use `ftINIT` for new work; keep `getINITModel` and `runINIT` only to reproduce models originally built with them. +| `makeSomething(model, ignoreMets, isNames, minNrFluxes, allowExcretion, params, ignoreIntBounds)` | `findLeakMetabolite(model, 'produce', ...)` | The argument order maps one to one, and the wrapper forwards `varargin` unchanged | +| `consumeSomething(model, ignoreMets, isNames, minNrFluxes, params, ignoreIntBounds)` | `findLeakMetabolite(model, 'consume', ...)` | The RAVEN 2 order has no position for `allowExcretion`, so the wrapper remaps the arguments by name to keep `params` and `ignoreIntBounds` | +| `canProduce(model, mets)` | `canExchange(model, 'produce', 'mets', mets)` | None | +| `canConsume(model, mets)` | `canExchange(model, 'consume', 'mets', mets)` | None | +| `compareRxnsGenesMetsComps` | `comparison/compareMultipleModels.m` | The overlap output of RAVEN 2 (reactions, genes, metabolites, EC codes, metabolite names, equations and unique equations) is part of the output of `compareMultipleModels`. No wrapper | +| `INIT/scoreComplexModel.m` | `INIT/scoreModel.m` | Merged, no wrapper. A call moves to `scoreModel` with the same positions and defaults for the first six optional arguments, and `dataPrecedence` is added as the 7th optional argument | +| `INIT/ftINITFillGapsMILP.m` | `solver/getMinNrFluxes.m` | Available through the option `resolveTies` of `getMinNrFluxes`, which exists for `formulation` `'reversible'`. The default `'irrev'` raises an error with `resolveTies`. The option `proveAbsGap` belongs to `ftINIT` | +| `INIT/ftINITFillGapsForAllTasks.m` | `gapfilling/ftINITFillGaps.m` and the gap-fill mode `'preMerged'` of `fitTasks` | None. `ftINITFillGaps` exists in RAVEN 2 as well | +| `getGenesFromKEGG`, `getMetsFromKEGG` and `getRxnsFromKEGG` | `reconstruction/kegg/readKEGGTable.m` (with `buildGlobalKEGGModel` and `buildGlobalGPR`) and the KEGG artefacts of `toolbox-assets` | The data source differs, so the signatures are not compatible (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)) | +| `core/parallelPoolRAVEN.m` | `utils/parallelWorkersRAVEN.m` | `parallelWorkersRAVEN` returns a worker count for `parfor` (`0` or `Inf`) and starts no pool. `parallelPoolRAVEN` returns `[ps, oldPoolAutoCreate]` | +| `installation/checkInstallation.m` | `checkRaven.m` in the repository root | The signature is identical. `checkInstallation.m` in the repository root forwards to `checkRaven` and issues a `RAVEN:deprecated` warning on every call | + +### Gene scoring with `scoreModel` + +RAVEN 2 has `INIT/scoreComplexModel.m` and `hpa/scoreModel.m`. RAVEN 3 has one `scoreModel`, which the legacy method tINIT and its replacement ftINIT call with different settings. A call written for `hpa/scoreModel.m` differs in these points (checked against 2.11.3): + +- RAVEN 2 scored a reaction by the best or the average score over all its genes (`multipleGeneScoring`, values `'best'` and `'average'`). RAVEN 3 applies `isozymeScoring` (default `'max'`) to the `or` parts and `complexScoring` (default `'min'`) to the `and` parts of the rule. A reaction with an `and` rule therefore gets a different default score. +- Positions count all arguments of the call. The 7th to 9th arguments of a RAVEN 2 call (`multipleGeneScoring`, `multipleCellScoring`, `hpaLevelScores`) are read as `isozymeScoring`, `complexScoring` and `multipleCellScoring`. `hpaLevelScores` is the 10th argument. +- `multipleCellScoring` accepts `'max'` and `'average'`. RAVEN 2 accepted `'best'` and `'average'`. +- `dataPrecedence` is an argument of RAVEN 3 with the default `'gene'`. +- A gene that is absent from the data receives `NaN` in `geneScores`, and the RAVEN 2 documentation states `-Inf`. `getINITModel` converts `NaN` back to `-Inf`. + +[Backward-incompatible changes](#backward-incompatible-changes) lists these differences as table rows. The function `getINITModel` passes `'max'` for `isozymeScoring` and for `complexScoring`, and `'reaction'` for `dataPrecedence`. Whether these settings reproduce the scores of the RAVEN 2 `hpa/scoreModel` was not checked, and the RAVEN 2 value `'average'` of `multipleGeneScoring` has no equivalent among them. + +### Legacy tINIT functions + +The method ftINIT replaces the legacy method tINIT. The functions `getINITModel` and `runINIT` (in `INIT/tINIT/`) implement tINIT, are marked legacy and are not scheduled for removal. `utils/legacyMethodNotice.m` issues a `RAVEN:legacyMethod` warning once, which can be silenced, and points to `ftINIT`. The method `ftINIT` is meant for new work, and `getINITModel` and `runINIT` reproduce models that were built with them. diff --git a/docs/raven3-vs-raven-toolbox.md b/docs/raven3-vs-raven-toolbox.md index f3c145c..9edefcb 100644 --- a/docs/raven3-vs-raven-toolbox.md +++ b/docs/raven3-vs-raven-toolbox.md @@ -1,58 +1,59 @@ # Comparison of RAVEN and raven-toolbox -RAVEN exists as two implementations: the MATLAB toolbox RAVEN and the Python package raven-toolbox. +RAVEN exists in two implementations: the MATLAB toolbox RAVEN 3 and the Python package raven-toolbox. The text below and the two detail pages help a reader who chooses between them or ports a workflow from one to the other. -- RAVEN is a MATLAB toolbox that runs without the COBRA Toolbox. -- raven-toolbox is a Python package built on [cobrapy](https://cobrapy.readthedocs.io/). A model is a `cobra.Model`. +- RAVEN runs in MATLAB without the COBRA Toolbox. +- raven-toolbox is built on [cobrapy](https://cobrapy.readthedocs.io/), and a model is a `cobra.Model`. -The implementations differ in function names, in functions that exist in one implementation only and in the results of certain functions (see [Differences in results between functions](raven3-vs-raven-toolbox/same-function-different-answer.md)). +The implementations differ in function names, in functions that a single one of them has, in the form of the results and in the results of some functions. [Features specific to each implementation](raven3-vs-raven-toolbox/what-each-has.md) covers the functions that exist in a single implementation. [Differences in results between functions](raven3-vs-raven-toolbox/same-function-different-answer.md) covers functions that exist in both and return different results. ## Choosing an implementation -The choice of an implementation depends on the conditions below. Each condition points to one implementation. +Each condition below favours the implementation named next to it. Both implementations provide homology and KEGG reconstruction, metabolic tasks, gap-filling, ftINIT, compartment assignment and model comparison. -| Implementation | Condition | +| Condition | Implementation | |---|---| -| raven-toolbox | The surrounding code is Python. | -| raven-toolbox | The model should be a `cobra.Model` for use with cobrapy tools. | -| raven-toolbox | Reproducible environments and continuous integration (CI) are required. | -| raven-toolbox | The KEGG artefacts must be built from a specified KEGG release. | -| RAVEN | The surrounding code is MATLAB. | -| RAVEN | A model built with tINIT must be reproduced. | -| RAVEN | Dynamic flux balance analysis (FBA) is required. | -| RAVEN | Conversion to the COBRA Toolbox structure is required. | -| Either | The functions both implementations share are needed: homology and KEGG reconstruction, metabolic tasks, gap-filling, ftINIT, compartment assignment and model comparison. | +| The surrounding code is Python | raven-toolbox | +| The model is used with cobrapy tools as a `cobra.Model` | raven-toolbox | +| The KEGG artefacts are built from a specified KEGG release (requires a KEGG FTP subscription) | raven-toolbox | +| The surrounding code is MATLAB | RAVEN | +| A model built with the legacy tINIT method, which ftINIT replaces, is reproduced | RAVEN | +| Dynamic flux balance analysis (FBA) is run with `runDynamicFBA` | RAVEN | +| A model is converted in memory to the COBRA Toolbox structure with `ravenCobraWrapper` | RAVEN | + +A model can also be exchanged by file. The function `read_matlab_model` of raven-toolbox reads a COBRA Toolbox `.mat` file as well as a RAVEN `.mat` file, and `write_matlab_model` writes the RAVEN layout. ## Name differences -The implementations differ in naming: RAVEN uses `camelCase` and raven-toolbox uses `snake_case`; many names are not the `snake_case` form of the RAVEN name. Examples: +RAVEN uses `camelCase` and raven-toolbox uses `snake_case`. The generated mapping pairs 103 RAVEN functions with a raven-toolbox function. The rule is: rows of the mapping that have a raven-toolbox function in the second column, with the cobrapy entries excluded. Of these 103, 48 have a name whose letters differ from the RAVEN name when case and the separator `_` are ignored. Examples: | RAVEN | raven-toolbox | |---|---| | `getBlast` | `run_blast` | | `addRxns` | `add_reactions_from_equations` | +| `addRxnsGenesMets` | `add_reactions_from_model` | | `fillGaps` | several functions (see [Gap-filling functions](raven3-vs-raven-toolbox/same-function-different-answer.md#gap-filling-functions)) | -The [MATLAB vs Python](matlab-vs-python.md) page, generated from the sources of both implementations at build time, lists the name in the other implementation. Some functions have no counterpart or map to cobrapy; the table records which functions cobrapy covers in place of raven-toolbox, and which are deliberately absent. +The function `add_reactions_from_equations` takes a sequence of mappings with the keys `id`, `equation`, `name`, `bounds`, `gene_reaction_rule` and `subsystem`. This corresponds to the `equations` form of the `rxnsToAdd` struct of `addRxns`. The form of `addRxns` with `mets` and `stoichCoeffs` has no counterpart, and the metadata fields of the struct, such as `eccodes`, `rxnMiriams` and `rxnNotes`, are not keys of the mapping. -## Detail pages +The page [MATLAB vs Python](matlab-vs-python.md) is generated at build time from the sources of both implementations. It has 132 rows for 130 documented RAVEN functions (`parseGrRule` and `setExchangeBounds` have two rows each): 55 matched automatically by name, 48 from a curated list and 29 covered by cobrapy. A further 42 RAVEN functions are listed as deliberately not ported and have no counterpart, so the page accounts for 172 RAVEN functions. -| Page | Content | -|---|---| -| [Features specific to each implementation](raven3-vs-raven-toolbox/what-each-has.md) | Features of raven-toolbox (model type, KEGG artefact generation, checksum registry) and of RAVEN (tINIT, COBRA Toolbox conversion, dynamic FBA, MATLAB-specific functions). | -| [Differences in results between functions](raven3-vs-raven-toolbox/same-function-different-answer.md) | Task-checking runtime, gap-filling functions, alternative optima of mixed-integer problems, elemental balance results and parsing of gene-reaction rules. | +## Form of results + +RAVEN functions return structs or several output arguments. Many raven-toolbox functions return dataclasses, for example `GapFillResult`, `TaskResult` and `ElementalBalance`, and others return a model or a list, for example `ftinit` and `add_reactions_from_equations`. Porting a call therefore changes how the result is read, in addition to the name. ## Solvers -The solver is set differently in each implementation. +The two implementations set the solver differently. | Aspect | RAVEN | raven-toolbox | |---|---|---| | Configuration | `setRavenSolver('gurobi')` | `cobra.Configuration().solver = 'gurobi'` | -| Recommended solver | Gurobi (free academic licence) | Gurobi (free academic licence) | -| Open-source option | GLPK (bundled with RAVEN) | GLPK (bundled with cobrapy) | +| Solvers | GLPK, Gurobi, SCIP and the COBRA Toolbox solvers. `setRavenSolver` downloads the SCIP binary for Windows alone | Any solver that cobrapy supports through optlang | +| Free licence | Gurobi (academic) | Gurobi (academic) | +| Bundled open-source solver | GLPK, bundled with RAVEN | GLPK, installed with cobrapy | -The mixed-integer linear programming (MILP) problems of ftINIT extraction can have several optima of equal objective value. For genome-scale MILP, the choice of solver changes which optimum is returned, and the seed, the thread count and wall-clock limits influence it as well (see [Alternative optima of mixed-integer problems](raven3-vs-raven-toolbox/same-function-different-answer.md#alternative-optima-of-mixed-integer-problems)). The solver matrix under [Choosing a solver](installation/index.md#choosing-a-solver) lists Gurobi for genome-scale models and GLPK for small to medium models. +[Choosing a solver](installation/index.md#choosing-a-solver) gives a recommendation by model size. In the measurements of the parameter-tuning studies, the ftINIT extraction of raven-toolbox returned different reaction sets for different solvers, seeds and thread counts. [Alternative optima of mixed-integer problems](raven3-vs-raven-toolbox/same-function-different-answer.md#alternative-optima-of-mixed-integer-problems) gives the figures. Whether the solutions have equal quality was not measured across solvers. ```{toctree} :hidden: diff --git a/docs/raven3-vs-raven-toolbox/same-function-different-answer.md b/docs/raven3-vs-raven-toolbox/same-function-different-answer.md index fa59ef6..818bcb5 100644 --- a/docs/raven3-vs-raven-toolbox/same-function-different-answer.md +++ b/docs/raven3-vs-raven-toolbox/same-function-different-answer.md @@ -1,52 +1,67 @@ # Differences in results between functions -The page lists functions that return a different result, return it in a different order, change the model differently or run for a different time in RAVEN and raven-toolbox, and two cases in which raven-toolbox differs from cobrapy. The list contains a function only after the difference is confirmed in the code of both functions compared. +A function of RAVEN and its counterpart in raven-toolbox can differ in the algorithm they choose, in the form of the result or in the value of the result. The sections below describe the differences found by reading the code of both implementations. Where a section says that a result was not checked, no run compared the two implementations. -The list is incomplete: a function that is not listed can still behave differently. +The list is incomplete, and a function that is not listed can behave differently. Candidates that were not compared include `random_sampling`, which replaces both `sampleACHR` and `sampleCHRR`; `remove_duplicate_reactions`, which replaces `contractModel`; and `simplify_model`, which corresponds to `simplifyModel`. -## Runtime of metabolic task checking +## Metabolic task checking -`checkTasks` in RAVEN and `check_tasks` in raven-toolbox return the same pass and fail results and differ in runtime. +The functions `checkTasks` and `check_tasks` implement the same task format and the same boundary semantics. The option `close_boundaries=True` of `check_tasks` corresponds to calling `checkTasks` on `closeModel(model)`. [Metabolic tasks](../guide/tasks.md) describes the options. Whether both functions return identical pass and fail results on a shared input was not checked. -`checkTasks` rebuilds the working model from the original for each task. `check_tasks` applies each task's constraints to one model inside a `with model:` block and reverts them afterwards. `check_tasks` saves and restores direct changes of mass-balance bounds, the one kind of edit that the context manager of cobrapy does not track. +`checkTasks` starts each task from the original model structure. `check_tasks` applies each task to one model inside a `with model:` block and reverts the changes afterwards. It also saves and restores direct edits of metabolite-constraint bounds. The runtime of the two functions was not compared. -At genome scale, copying the model for each task adds to the runtime of `checkTasks`, and `check_tasks` reuses a single model to avoid it. +The option `getEssential` of `checkTasks` has no counterpart option in `check_tasks`. In RAVEN, `checkTasks` calls `getEssentialRxns` on the model constrained by each task, and `find_task_essential_reactions` ports this step. It is the port of `checkTasks` with `getEssential`. ## Gap-filling functions -`fillGaps` in RAVEN selects one of several algorithms with the `algorithm` option. raven-toolbox splits them into four functions, so porting a `fillGaps` call requires choosing among the functions in this table: +The function `fillGaps` of RAVEN selects one of five algorithms with the option `algorithm`: `reference` (default), `fastLP`, `swiftLP`, `gapfillMILP` and `topological`. raven-toolbox has four functions for them, so porting a `fillGaps` call means choosing one function from the table. | Purpose | RAVEN algorithm | raven-toolbox function | |---|---|---| | Connecting blocked reactions against template models | `fillGaps` (default `'reference'`) | `connect_blocked_reactions` | | Fast filling of a large candidate set by linear programming (LP) | `fillGaps` with `'fastLP'` and `'swiftLP'` | `fill_gaps_fast_lp` (`variant="fast"` and `variant="swift"`) | | Filling by mixed-integer linear programming (MILP) with the fewest added reactions that restore growth | `fillGaps` with `'gapfillMILP'` | `fill_gaps_milp` | -| Finding the gaps without filling them | `fillGaps` with `'topological'` (`gapFillTopological`) | `analyse_topology` | +| Finding the gaps without filling them | `fillGaps` with `'topological'` | `analyse_topology` | -The choice among these functions changes the reactions added and the runtime. +The functions differ in three more points. -The RAVEN functions `canExchange`, `checkProduction` and `getAllSubGraphs` find gaps without filling them. raven-toolbox merges them into `gap_report`, the port of `gapReport`, and has no separate function for each of them. +- `fillGaps` with `useModelConstraints=true` fills so that the constraints of the model hold. `connect_blocked_reactions` has no such mode, and its documentation points to `cobra.flux_analysis.gapfill` for that use. The closest function for a growth target is `fill_gaps_milp`. +- `fillGaps` takes a cell array of template models, but the RAVEN functions `gapFillFastLP`, `gapFillMILP` and `gapFillTopological` use the first template and ignore the rest. The four raven-toolbox functions accept several templates. +- `fillGaps` returns `newConnected`, `cannotConnect`, `addedRxns`, `newModel` and `exitFlag`. raven-toolbox returns a `GapFillResult` from `connect_blocked_reactions`, a `FastLPResult` from `fill_gaps_fast_lp`, a `MILPGapFillResult` from `fill_gaps_milp` and a `TopologicalAnalysisResult` from `analyse_topology`. The count `newConnected` is not stable across runs (see [Gap-filling](../guide/gap-filling.md)). + +The RAVEN functions `canExchange`, `checkProduction` and `getAllSubGraphs` find gaps without filling them. raven-toolbox merges them into `gap_report`, the port of `gapReport`, and has no separate public function for each of them. ## Alternative optima of mixed-integer problems -ftINIT extraction, gap-filling and compartment assignment solve mixed-integer problems that routinely have several optima of equal objective value. Two runs can return different reaction sets that are both correct. Results can differ between implementations, between solvers and between repeat runs of one solver. +The extraction with ftINIT, the reference and MILP gap-filling algorithms, and the assignment of compartments (`assignCompartments` and `assign_compartments`) solve mixed-integer problems, which can have several optima with the same objective value. The LP algorithms were not measured for this, and the documentation of `swiftLP` in both implementations calls it stochastic. A solve that stops at its gap or time limit can also end at a different objective value. Two runs can therefore return different reaction sets, and no single exact result is expected. + +Four measurements quantify the differences. -The ftINIT reproducibility study measured raven-toolbox only (RAVEN's `ftINIT` was not run): runs of one solver also differ with the seed and the number of threads, and two repeated builds at one seed differed by 2 reactions; each of the two builds had one run of the final step that stopped at its 5 s wall-clock limit, the presumed and untested cause of the difference (see [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](../parameter-tuning/studies/ftinit-determinism.md)). No single exact result is expected. +- **Between implementations.** The ftINIT models of five cell lines, extracted from the same Human-GEM inputs with RAVEN and raven-toolbox, share Jaccard indices of 0.964 to 0.988 over their reaction sets (see [Human-GEM cell-type model validation against RAVEN](../parameter-tuning/studies/humangem-validation.md)). The template preparation differed between the two runs. +- **Between solvers.** In the raven-toolbox ftINIT benchmark on Human-GEM with one run per solver, Gurobi extracted 7312 reactions and GLPK 6758 (Jaccard index 0.892). Both main steps of GLPK stopped at the 900 s limit with an unproven incumbent, so the two models differ in objective value as well as in tied optima (see [ftINIT on different solvers](../parameter-tuning/studies/init-solver-benchmark.md)). RAVEN was not run on several solvers in that study. +- **Between repeated runs.** Two repeated raven-toolbox builds at one seed with the default settings differed by 2 reactions. In each build one run of the final step stopped at its 5 s wall-clock limit, which is the presumed and untested cause (see [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](../parameter-tuning/studies/ftinit-determinism.md)). The effects of seed, thread count and time limit were not measured for the `ftINIT` of RAVEN. +- **Between thread counts.** In the first ftINIT step of raven-toolbox without `resolve_ties`, with six seeds, the solutions at `Threads` 1 and 4 for the same seed were 5 to 20 reactions apart. In 5 of the 6 seeds the two runs reached different objective values (see [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](../parameter-tuning/studies/ftinit-determinism.md)). -Compare these outputs by the overlap of their reaction sets; do not test them for identity. +Compare such outputs by the overlap of the reaction sets, for example the Jaccard index. -`resolveTies` in RAVEN's `ftINIT` and `resolve_ties` in raven-toolbox's `ftinit` reduce the difference between runs for the ftINIT MILP. Each step then keeps the score optimum and picks among the tied solutions in phases: the fewest kept removable reactions, then the lowest summed ID rank of those, then the fluxes with the least total absolute flux, then the fluxes with the least ID-rank-weighted backward flux. The last step of `ftINIT` skips the two flux phases. +The option `resolveTies` of the `ftINIT` of RAVEN and `resolve_ties` of the `ftinit` of raven-toolbox are off by default in both. When on, each step keeps the score optimum and picks among the tied solutions in four phases, in this order: the fewest kept removable reactions, the lowest summed ID rank of those, the least total absolute flux and the least backward flux weighted by ID rank. The last step of `ftINIT` skips the two flux phases. A step that stops at its time limit keeps the answer of the solver. -For a given score optimum, the selected solution does not depend on the seed when every tie-break phase finishes within its time limit. It can differ between solvers, and a step that stops at its time limit keeps the solver's own answer. The primary solve can still reach a different score optimum for a different seed, so the final reaction sets can differ between seeds. Without `resolveTies` or `resolve_ties`, the ftINIT MILP can return any of the tied optima, and results differ between implementations, between solvers and between repeated runs of one solver. +In raven-toolbox, runs with `resolve_ties` at the same score optimum returned identical solutions across seeds. The primary solve can end at a different objective value for a different seed, within the gap and time limits, so the final reaction sets can differ. This was not measured for RAVEN. Without `resolveTies` or `resolve_ties`, the ftINIT MILP can return any of the tied optima. ## Elemental balance results -`getElementalBalance` in RAVEN and `get_elemental_balance` in raven-toolbox check whether each element (carbon, hydrogen, oxygen and so on) has the same count on both sides of a reaction. A reaction is unbalanced when the counts were computed and did not match, and the result is unknown when the counts cannot be computed. The two functions use different codes for these results. +The functions `getElementalBalance` in RAVEN and `get_elemental_balance` in raven-toolbox check whether each element has the same count on both sides of a reaction. A reaction is unbalanced when the counts were computed and do not match. The result is unknown when the counts cannot be computed. The two functions code these results differently. + +The function `getElementalBalance` returns the numeric field `balanceStructure.balanceStatus` per reaction: 1 for balanced, 0 for unbalanced, -1 for missing information (no formula, or a reaction without metabolites) and -2 for a formula that cannot be parsed. The function `get_elemental_balance` returns the strings `balanced`, `unbalanced` and `unknown`. It reports `unknown` for a missing formula, for a formula that cannot be parsed and for a reaction without metabolites, so the codes -1 and -2 of RAVEN are one result in raven-toolbox. + +The two parsers handle some formulas differently, so the results differ for some reactions. -`getElementalBalance` returns the numeric field `balanceStructure.balanceStatus` per reaction: 1 for balanced, 0 for unbalanced, -1 for missing information (no formula, or a reaction without metabolites) and -2 for a formula that cannot be parsed. `get_elemental_balance` returns the strings `balanced`, `unbalanced` and `unknown`. It reports `unknown` for a missing formula, for a formula that cobrapy cannot parse and for a reaction without metabolites, so the codes -1 and -2 of RAVEN are one result in raven-toolbox. For a reaction whose metabolites all have parseable formulas, the two functions return the same result. +- A polymer formula such as `(C5H8)n` is counted with n equal to 1 in RAVEN. raven-toolbox reports `unknown`. +- RAVEN uses the field `inchis` when `metFormulas` is absent. raven-toolbox does not. +- RAVEN reports a charge balance (`chargeStatus` and `chargeResidual`). `get_elemental_balance` reports elements. -cobrapy's `check_mass_balance` has no separate `unknown` result and raises an error for a formula that it cannot parse. For a reaction with a missing formula it treats the formula as empty and returns either an element imbalance or an empty result, so it gives the same kind of result as for a reaction with known formulas. raven-toolbox keeps its own function to retain the distinction. +The comparison used the code, and the functions were not run on a shared model. ## Parsing of gene-reaction rules -raven-toolbox and the JSON reader of cobrapy parse gene-reaction rules (the `grRules` field in RAVEN) differently. raven-toolbox reads them with `utils.parse_gr_rule` in `read_yaml_model`, `read_matlab_model`, `batch_curate` and the functions that copy or build rules, so `G1&G2` is one gene ID there. The JSON reader of cobrapy splits `G1&G2` into `G1` and `G2`. +The parsing of rules matters when a RAVEN user takes a model through cobrapy readers in Python. raven-toolbox parses gene-reaction rules (the field `grRules` in RAVEN) with `utils.parse_gr_rule`, which follows the grammar of `parseGrRule` of RAVEN. The function is used in `read_yaml_model`, `read_matlab_model`, `batch_curate` and the functions that copy or build rules, so `G1&G2` is one gene ID there. According to the changelog of raven-toolbox, the JSON reader of cobrapy splits `G1&G2` into `G1` and `G2`, and the SBML reader keeps it as one ID. This was not checked against cobrapy itself. diff --git a/docs/raven3-vs-raven-toolbox/what-each-has.md b/docs/raven3-vs-raven-toolbox/what-each-has.md index 76c5e85..2b027b2 100644 --- a/docs/raven3-vs-raven-toolbox/what-each-has.md +++ b/docs/raven3-vs-raven-toolbox/what-each-has.md @@ -1,41 +1,59 @@ # Features specific to each implementation -The page describes the features that exist in raven-toolbox only and the features that exist in RAVEN only. +Some features belong to raven-toolbox alone, and some belong to RAVEN alone. The sections below describe both groups. ## Features specific to raven-toolbox ### Model type -A raven-toolbox model is a `cobra.Model`. Every cobrapy function and cobrapy-based package applies to it without conversion. RAVEN uses a MATLAB struct as the model, and `ravenCobraWrapper` converts between that struct and the COBRA Toolbox format. +A raven-toolbox model is a `cobra.Model`. Every function of cobrapy and of cobrapy-based packages applies to it without conversion. RAVEN uses a MATLAB struct as the model. -cobrapy provides flux balance analysis (FBA), flux variability analysis (FVA), knockouts and media handling, in addition to the other functions that the mapping table marks as covered by cobrapy. The mapping table therefore gives a cobrapy entry in place of a raven-toolbox function for the corresponding RAVEN functions. +cobrapy provides flux balance analysis (FBA), flux variability analysis (FVA), knockouts and media handling. For the functions of RAVEN that cobrapy covers, the page [MATLAB vs Python](../matlab-vs-python.md) gives a cobrapy entry (see [Name differences](../raven3-vs-raven-toolbox.md#name-differences)). The function `setExchangeBounds` has two rows in the mapping: `set_exchange_bounds` in raven-toolbox and `cobra.Model.medium` in cobrapy. ### KEGG artefact generation -raven-toolbox builds the KEGG reference artefacts: it parses a KEGG release into reaction and compound tables, assembles the reference model, builds the per-KEGG Orthology (KO) FASTA sets and HMM libraries, and derives the phylogenetic distance matrix. RAVEN uses pre-built artefacts and does not produce them. +raven-toolbox builds the KEGG reference artefacts. It parses a KEGG release into tables of reactions and compounds, assembles the reference model with `build_reference_model`, builds the FASTA sets and HMM libraries per KEGG Orthology (KO) entry, and derives the phylogenetic distance matrix. The download of a KEGG release requires a KEGG FTP subscription, and the artefacts can be rebuilt from a specified KEGG release. -The artefacts can therefore be rebuilt from a specified KEGG release. +RAVEN loads published artefacts and does not produce them. The function `buildGlobalKEGGModel` joins a downloaded bundle onto the reference model, and the phylogenetic distance matrix is a file shipped with RAVEN (`keggPhylDist.mat`). In the mapping, `getModelFromKEGG` corresponds to `build_reference_model`, which loads in RAVEN and builds in raven-toolbox. ### Checksum registry for binaries and data -Both implementations fetch BLAST+, DIAMOND, HMMER and the KEGG artefacts on demand, do not bundle them, pin a release and check downloads against checksums (see [Download data and binaries](../installation/data-and-binaries.md)). +Both implementations fetch BLAST+, DIAMOND, HMMER and the KEGG tables on demand and do not bundle them. RAVEN bundles other components: the GLPK and libSBML MEX files and `keggPhylDist.mat`. [Download data and binaries](../installation/data-and-binaries.md) describes the download behaviour. -raven-toolbox has the checksums and the release of each dataset in a registry in its source, which ties each release to the assets it was tested with. RAVEN checks a download against the checksum of the published manifest. When the manifest cannot be read or has no entry for the asset, RAVEN issues a warning and keeps the file unchecked, and an already cached file is reused without a check. +The checksum handling differs. + +- raven-toolbox keeps the checksum and the release of each dataset and binary in a registry in its source (a version-pinned registry). A fresh download is checked against its checksum. A cached data file is reused without a check, unless `ensure_data_file` is called with `verify=True`, which checks it and downloads it anew on a mismatch. A cached binary is reused without a check. A JSON manifest can be merged into the registries with `raven_toolbox.manifest.load_into_registries`, which reads the path or URL in the environment variable `RAVEN_PYTHON_MANIFEST` when no source is passed. The resolvers load that variable by themselves while the registry is empty, so the call is needed with the filled default registries. +- RAVEN checks a download against the checksum in the published manifest. A mismatch deletes the file and raises an error. When the manifest cannot be read or has no entry for the asset, RAVEN issues a warning and keeps the file unchecked. A cached file is reused without a check. + +### Further functions + +Further features of raven-toolbox have no RAVEN counterpart in the generated mapping. Examples are `localization.relocate_reactions`, which places reactions in compartments chosen by a curator and repairs the model around that decision; `localization.triage_localization`; and the per-reaction confidence record of the `confidence` module, which scores the facets localization, equation and gene association and stores the scores in the reaction notes. ## Features specific to RAVEN -### tINIT +### Legacy tINIT method -`getINITModel` and `runINIT` are the original tINIT implementation. RAVEN keeps them for models built with them. raven-toolbox has no earlier tINIT models to reproduce, so it implements ftINIT only. A tINIT model has to be reproduced in MATLAB; see [Context-specific models with ftINIT](../guide/init.md). +The functions `getINITModel` and `runINIT` are the legacy tINIT method, which ftINIT replaces. RAVEN keeps them for models built with them. raven-toolbox implements ftINIT as its sole INIT variant, as a design decision. A model built with tINIT has to be reproduced in MATLAB (see [Context-specific models with ftINIT](../guide/init.md)). ### COBRA Toolbox conversion -`ravenCobraWrapper` converts between the RAVEN and COBRA Toolbox model structures. In raven-toolbox the model is a `cobra.Model`, so no conversion is needed and no equivalent exists. +The function `ravenCobraWrapper` converts a model in memory between the RAVEN structure and the COBRA Toolbox structure, and a round trip drops some fields. The model of raven-toolbox is a `cobra.Model`, so no in-memory conversion is needed and no equivalent exists. [Choosing an implementation](../raven3-vs-raven-toolbox.md#choosing-an-implementation) describes the exchange by file, and cobrapy writes the COBRA layout. ### Dynamic FBA -`runDynamicFBA` has no raven-toolbox counterpart by design. Python packages cover dynamic FBA ([dfba](https://pypi.org/project/dfba/), [reframed](https://pypi.org/project/reframed/), [mewpy](https://pypi.org/project/mewpy/)), and raven-toolbox does not reimplement it. +`runDynamicFBA` has no raven-toolbox counterpart by design. Python packages cover dynamic FBA: [dfba](https://pypi.org/project/dfba/), [reframed](https://pypi.org/project/reframed/) and [mewpy](https://pypi.org/project/mewpy/). The packages were not checked here. ### MATLAB-specific functions -RAVEN has MATLAB-specific infrastructure: path management (`addRavenToUserPath`, `findRAVENroot`), argument handling (`parseRAVENargs`, `convertCharArray`), progress and printing (`setRavenProgress`, `printOrange`) and the solver abstraction (`optimizeProb`, `setRavenSolver`). cobrapy's solver interface (optlang) takes the place of the solver abstraction. +RAVEN has MATLAB-specific infrastructure that raven-toolbox does not port: + +- path management (`addRavenToUserPath`, `findRAVENroot`); +- argument handling (`parseRAVENargs`, `convertCharArray`); +- progress and printing (`setRavenProgress`, `printOrange`); +- the solver abstraction (`optimizeProb`, `setRavenSolver`), which the solver interface of cobrapy (optlang) replaces. + +The generated mapping table lists the functions of its tracked categories. `addRavenToUserPath` and `findRAVENroot` belong to the installation folder, which is not tracked, so the table does not list them. The table "Deliberately not ported" lists further infrastructure functions, for example `checkSolution`, `permuteModel`, `parseYAML` and `deprecationWarning`. + +### Functions without a Python counterpart + +Some functions that raven-toolbox does not port are not infrastructure. Three have no replacement and can block a port: `getTransportRxns`, `runRobustnessAnalysis` and `runSimpleOptKnock`. The function `readFasta` has no public reader, and `findPotentialErrors` has no step that detects an invalid rule string, because the GPR class of cobrapy represents a parsed rule directly. For other functions the table gives a replacement: the `.annotation` dictionary of cobrapy for `editMiriam` and `extractMiriam`, the `cobra.Reaction` API for `reverseRxns`, and `model.summary()` for `printFluxes`. The table "Deliberately not ported" on the [MATLAB vs Python](../matlab-vs-python.md) page is the reference. diff --git a/docs/references.md b/docs/references.md index b230181..65ac284 100644 --- a/docs/references.md +++ b/docs/references.md @@ -31,9 +31,9 @@ > the National Academy of Sciences 120(6): e2217868120. > -### tINIT +### Legacy tINIT method -`runINIT` and `getINITModel` (RAVEN's MATLAB implementation of tINIT) build on the INIT method described in the following paper. tINIT has no citation of its own. +`runINIT` and `getINITModel` (the legacy tINIT method of RAVEN in MATLAB, which ftINIT replaces) build on the INIT method described in the following paper. tINIT has no citation of its own. > Agren R, Bordel S, Mardinoglu A, Pornputtapong N, Nookaew I, Nielsen J > (2012). **Reconstruction of genome-scale active metabolic networks for 69 diff --git a/docs/tuned-parameters.md b/docs/tuned-parameters.md index 9ee5fa8..29458e5 100644 --- a/docs/tuned-parameters.md +++ b/docs/tuned-parameters.md @@ -44,7 +44,7 @@ Full detail: [Benchmarks of `fseof`, `remove_genes` and other defaults](paramete RAVEN MATLAB: `runINIT`, `scoreModel`, `getINITModel`, `ftINIT`, `prepINITModel`, `ftINITInternalAlg`, `getINITSteps`. raven-toolbox: `score_reactions_from_genes`, `gene_scores_from_expression`, `ftinit`, `prep_init_model`, `run_ftinit`, `get_init_steps`, `classify_reactions`. -Only MATLAB has tINIT (`runINIT`, `getINITModel`); raven-toolbox implements ftINIT alone (see [Context-specific models with ftINIT](guide/init.md)). Rows measured across both algorithms say so. +Only MATLAB has the legacy tINIT method (`runINIT`, `getINITModel`), which ftINIT replaces; raven-toolbox implements ftINIT alone (see [Context-specific models with ftINIT](guide/init.md)). Rows measured across both algorithms say so. | Parameter | Default | How determined | |---|---|---| diff --git a/scripts/curated_pairs.yml b/scripts/curated_pairs.yml index c43f052..7e3d9b0 100644 --- a/scripts/curated_pairs.yml +++ b/scripts/curated_pairs.yml @@ -27,6 +27,7 @@ aliases: convertToIrrev: convert_to_irreversible copyToComps: copy_to_compartment findDuplicateRxns: find_duplicate_reactions + removeBadRxns: remove_bad_reactions removeMets: remove_metabolites simplifyModel: remove_dead_end_reactions sortModel: sort_identifiers @@ -251,11 +252,6 @@ not_ported: getTransportRxns: >- Classifies which reactions move a metabolite between compartments. raven-toolbox has no equivalent classifier yet. - removeBadRxns: >- - Iteratively removes reactions that let the model make or consume - something from nothing. raven-toolbox has no equivalent yet; see - find_leak_metabolite (mapped from makeSomething/consumeSomething) for - finding the same kind of issue. reverseRxns: >- Negates a reaction's stoichiometry and bounds to reverse its direction. Simple enough to do directly against a cobra.Reaction's own API; no diff --git a/scripts/known_names.txt b/scripts/known_names.txt index 489b150..77f5744 100644 --- a/scripts/known_names.txt +++ b/scripts/known_names.txt @@ -437,3 +437,45 @@ run_hmmscan # imported by analyze_hmm_cutoffs.py; not in query parse_hmmscan_tblout # imported by analyze_hmm_cutoffs.py; not in query.py cannot_connect # output of connect_blocked_reactions ml_prep # MATLAB script of the validation study +fastLP # value of the algorithm option of fillGaps +swiftLP # value of the algorithm option of fillGaps +gapfillMILP # value of the algorithm option of fillGaps +cannotConnect # output of fillGaps +newModel # output of fillGaps +exitFlag # output of fillGaps +chargeStatus # field of the getElementalBalance output +chargeResidual # field of the getElementalBalance output +# parameter and output names in the RAVEN 2 to RAVEN 3 migration pages +goodRxns +rxnToCheck +nObjectives +checkBinaries +failedTasks +getWoLFScores + +# RAVEN 2 identifiers, options and fields named in the migration pages (functions removed in RAVEN 3, options of RAVEN 2 or 3 functions, model fields) +complexPolicy +geneScores +getToolboxVersion +grRule +importExportTests +keepGeneFree +parallelPoolRAVEN +preserveNotes +preserveQuotes +rxnFrom +solverTests +trimWarnings +xml_format +xml_formatany +xml_help +xml_load +xml_parse +xml_parseany +xml_save +xml_strsplit +xml_unblank +# Test classes of RAVEN (tests/function_tests) and a MATLAB built-in +tBinaries +tParseRAVENargs +regexpPattern