From d374236d25b865aee89cf2141622101acc719b7b Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Sat, 31 Oct 2026 22:22:00 +0100 Subject: [PATCH 01/20] Cite localization/predictLocalization.m in the predict_localization time limit row --- docs/tuned-parameters.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tuned-parameters.md b/docs/tuned-parameters.md index a2affb4..b82f8ca 100644 --- a/docs/tuned-parameters.md +++ b/docs/tuned-parameters.md @@ -175,7 +175,7 @@ The [INIT solver benchmark](parameter-tuning/studies/init-solver-benchmark.md) m | `random_sampling.loopless_good_reactions` | `True` (loopless FVA) | heuristic (exclude FVA ≥ 999) | The two are different techniques. Loopless FVA excludes only reactions that take part in infeasible cycles, and the MATLAB heuristic also excludes reactions that legitimately reach capacity. Porting the loopless technique to MATLAB is a larger project than changing a default. | | `merge_models.match_by` | `'name'` | `'metNames'` | Same semantic field (metabolite display name) under a different field name, which results from the cobrapy and COBRA Toolbox schemas; the field name is not tunable. | | `run_diamond.evalue` | `1e-3` | `1e-3` (implicit) | Both agree at `1e-3`, matching the DIAMOND default. `getDiamond` passes no `-evalue` flag, so it inherits the DIAMOND default and does not follow the explicit `1e-4` of `getBlast`. `run_blast` and `run_diamond` deliberately stay unpaired: each follows the MATLAB call of its own aligner. | -| `predict_localization.time_limit` | `None` | `maxTime=15` (minutes) | Confirmed by reading `core/predictLocalization.m` (2026-08-28): MATLAB solves the problem with simulated annealing (`maxTime` is its search budget; more time generally improves the heuristic answer, with no optimality guarantee), and Python solves a deterministic MILP (`time_limit` is a solver cutoff; returns a proven-bounded incumbent). The same number has a different role in each, so the values are not unified. A tentative unification at `None` is not adopted, because it assumed that both sides solve the same kind of problem. | +| `predict_localization.time_limit` | `None` | `maxTime=15` (minutes) | Confirmed by reading `localization/predictLocalization.m` (2026-08-28): MATLAB solves the problem with simulated annealing (`maxTime` is its search budget; more time generally improves the heuristic answer, with no optimality guarantee), and Python solves a deterministic MILP (`time_limit` is a solver cutoff; returns a proven-bounded incumbent). The same number has a different role in each, so the values are not unified. A tentative unification at `None` is not adopted, because it assumed that both sides solve the same kind of problem. | | `predict_localization.default_compartment` | `'c'` | required arg (no default) | MATLAB has no default; Python's `'c'` is a convenience default for the near-universal correct choice and produces no output difference (a MATLAB user must already supply `'c'` explicitly in the common case). Optional, low priority: MATLAB could add the same default. | ## Changes needed in MATLAB RAVEN for parity From 24a7014ad200fb5df8ab7af67da657d10ce7a9ca Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Sat, 31 Oct 2026 22:26:53 +0100 Subject: [PATCH 02/20] Give boxes their title as an admonition title A note, warning or tip directive takes no title argument in Sphinx, so the title written after the directive name rendered as the first line of the box text under the generic header. The 38 titled boxes use the admonition directive with a class, which renders the title in the header. --- docs/api/index.md | 4 +++- docs/guide/biomass.md | 4 +++- docs/guide/building.md | 4 +++- docs/guide/combining.md | 12 +++++++++--- docs/guide/comparing.md | 8 ++++++-- docs/guide/curation.md | 8 ++++++-- docs/guide/deletions.md | 4 +++- docs/guide/editing.md | 4 +++- docs/guide/engineering.md | 4 +++- docs/guide/fba.md | 4 +++- docs/guide/fva.md | 4 +++- docs/guide/gap-filling.md | 4 +++- docs/guide/getting-started.md | 4 +++- docs/guide/homology.md | 8 ++++++-- docs/guide/init.md | 8 ++++++-- docs/guide/io.md | 4 +++- docs/guide/kegg.md | 4 +++- docs/guide/media.md | 4 +++- docs/guide/model-structure.md | 4 +++- docs/guide/quality-control.md | 8 ++++++-- docs/guide/sampling.md | 8 ++++++-- docs/guide/solvers.md | 4 +++- docs/guide/tasks.md | 4 +++- docs/installation/data-and-binaries.md | 4 +++- docs/installation/raven.md | 4 +++- docs/protocol/lipid-curation.md | 4 +++- docs/protocol/materials.md | 8 ++++++-- docs/protocol/reconstruction.md | 4 +++- docs/tuned-parameters.md | 4 +++- 29 files changed, 114 insertions(+), 38 deletions(-) diff --git a/docs/api/index.md b/docs/api/index.md index 4a23e43..6ce0818 100644 --- a/docs/api/index.md +++ b/docs/api/index.md @@ -44,7 +44,9 @@ in calling convention, not in what the functions compute: | Indexing | 1-based | 0-based | | Model object | RAVEN `model` struct | `cobra.Model` with RAVEN extensions | -:::{note} A note on the two docstring styles +:::{admonition} A note on the two docstring styles +:class: note + raven-toolbox's docstrings are NumPy-style, so they render as structured parameter/return tables. RAVEN's MATLAB help blocks use the toolbox's own indented convention, so they render as faithful help text rather than typed diff --git a/docs/guide/biomass.md b/docs/guide/biomass.md index d4151f4..c345880 100644 --- a/docs/guide/biomass.md +++ b/docs/guide/biomass.md @@ -247,7 +247,9 @@ Database cross-references are stored in MIRIAM form: a namespace and an identifi Thermodynamic data is handled in a separate file. `loadDeltaGCSV` and `load_delta_g_csv` read CSVs of standard Gibbs free energies onto the metabolites and reactions, and `saveDeltaGCSV` and `save_delta_g_csv` write it back out. Keeping it in CSV means a thermodynamics run can be versioned and reviewed separately from the model, and re-applied after the model changes. -:::{warning} Troubleshooting +:::{admonition} Troubleshooting +:class: warning + - A configuration that does not match the model. Components are matched on the pseudoreaction's name and not its id. A component whose pseudoreaction is missing contributes zero without failing, so a mistyped name shows as a total that is short. - The wrong mass strategy. Using `mw` where the convention is `mw_minus_2h` or `mw_minus_water` inflates every polymer by the mass of the bonds, and the total shows it: the total is not near 1. - Rescaling without balancing. Setting protein to a measured value and leaving the rest alone changes the total, and therefore rescales every growth rate the model produces. diff --git a/docs/guide/building.md b/docs/guide/building.md index 127c22d..ef2349e 100644 --- a/docs/guide/building.md +++ b/docs/guide/building.md @@ -234,7 +234,9 @@ HEX1 flux: 1000.00 ::: :::: -:::{warning} Troubleshooting +:::{admonition} Troubleshooting +:class: warning + - A typo creates a metabolite. The defaults differ: `add_reactions_from_equations` creates unrecognised metabolites and genes, while `addRxns` refuses them unless `allowNewMets` and `allowNewGenes` are set. So `m7` and `M7` become two different things in Python and an error in MATLAB. Set `allow_new_mets=False` and `allow_new_genes=False` once everything is defined. - No exchange reactions. Nothing can be taken up or secreted, so growth is zero, with no error to explain why. - A model with no external compartment. `add_boundary` cannot guess one, and RAVEN's `addExchangeRxns` adds exchanges for internal metabolites without warning, opening the model to uptake and secretion of intermediates. diff --git a/docs/guide/combining.md b/docs/guide/combining.md index c941ab8..da905e3 100644 --- a/docs/guide/combining.md +++ b/docs/guide/combining.md @@ -95,7 +95,9 @@ The asymmetry follows from the matching rules. Metabolites are matched on name a `mergeModels` takes `metParam` to choose what metabolites are matched on: `"metNames"`, the default used in the example, or `"mets"` to match on identifier. Matching on id is the right choice when both models come from the same database, where the ids mean the same thing and the names may not. -:::{warning} Merging matches on names, not identifiers +:::{admonition} Merging matches on names, not identifiers +:class: warning + Two models built from different databases usually share almost no metabolite names either, in which case merging produces a model with two disconnected halves in the same struct. Merging assumes that the naming is consistent; check the naming before merging. ::: @@ -134,7 +136,9 @@ after contracting: 53 rxns `simplifyModel` does little unless the flags say what to remove: every flag is off by default except `deleteUnconstrained`. `deleteMinMax` is the expensive one used in the example, minimising and maximising each reaction to find out whether it can carry flux; `deleteInaccessible` drops dead ends structurally, `deleteZeroInterval` drops reactions pinned to zero, and `deleteDuplicates` does what `contractModel` does. `reservedRxns` protects reactions from all of them. In Python the same choices are `delete_no_flux` and its siblings, with `open_exchanges` deciding whether the test runs on the medium as it stands or with the boundary opened first. -:::{warning} Which functions mutate +:::{admonition} Which functions mutate +:class: warning + Every RAVEN function here returns a new model struct and leaves its input unchanged. The Python functions differ: | Python | Behaviour | @@ -238,7 +242,9 @@ A reaction that holds a single metabolite after merging is kept or removed by th Collapsing compartments discards information. A model that distinguishes mitochondrial from cytosolic acetyl-CoA cannot be recovered from the flattened one, and the flattened model permits a pathway to run on a pool that no membrane separates. Flatten a copy, for a specific question, and keep the original. -:::{warning} Troubleshooting +:::{admonition} Troubleshooting +:class: warning + - Merging on inconsistent names. The match is on name and compartment. Different naming conventions give two disconnected networks in one model, and the merge raises no error; MATLAB only warns about the metabolite ids it renames. - Expecting de-duplication. Merging keeps every reaction from every source. Contract afterwards, and check what got collapsed. - Simplifying against the wrong condition. Reactions are removed because they cannot carry flux under the current bounds. Change the medium first. diff --git a/docs/guide/comparing.md b/docs/guide/comparing.md index 6a3be50..047f3b5 100644 --- a/docs/guide/comparing.md +++ b/docs/guide/comparing.md @@ -140,7 +140,9 @@ similarity: 0.945 ::: :::: -:::{warning} A high similarity does not mean the models agree +:::{admonition} A high similarity does not mean the models agree +:class: warning + These two models are 0.945 alike on the reaction set, and one of them is broken. The differences listed in [Entry-by-entry differences](#entry-by-entry-differences) are a doubled stoichiometric coefficient and a reaction made irreversible, changes that alter what the model predicts while barely moving a set-overlap score. Similarity is for grouping models, not for validating one. ::: @@ -189,7 +191,9 @@ RAVEN can also run this comparison. `compareMultipleModels` takes `funcCompare` The two tabs also show what `compareMultipleModels` and `compare_models` count differently. RAVEN's matrix has one row per combination of models, in-first-only, in-second-only, in-both, with `nElements` counting each; raven-toolbox's has one row per identifier with a column per model. Both answer the same question, but they are indexed in opposite directions. -:::{warning} Troubleshooting +:::{admonition} Troubleshooting +:class: warning + - Comparing on identifiers alone. Both functions match by id. Two models from different databases share few ids and look unrelated even when they describe the same metabolism; see [Combining and simplifying models](combining.md), where merging matches on names instead. - Reading similarity as quality. It measures overlap, not correctness. - Comparing one flux vector per model. An FBA solution is one of many optima, so two models, or two solvers on one model, can return different vectors with the same growth. Check that both solves succeeded and compare growth first, then ranges or tasks. `compareFluxes` and `compare_fluxes` compare two solutions of one model, so they need every reaction in both. diff --git a/docs/guide/curation.md b/docs/guide/curation.md index a6149a9..c1e7fe9 100644 --- a/docs/guide/curation.md +++ b/docs/guide/curation.md @@ -102,11 +102,15 @@ The prefixes are arguments: `metPrefix` and `rxnPrefix` in MATLAB, `met_id_prefi Adding a reaction needs both reaction tables. The row in the reactions table carries the bounds, the gene rule and the annotation; the rows in the coefficients table carry the stoichiometry, one per metabolite, negative for substrates and positive for products, linked by `rxnIdx`. Any metabolite named there that the model does not have is created, so the metabolite table is usually applied in the same pass. -:::{warning} Blank cells overwrite values +:::{admonition} Blank cells overwrite values +:class: warning + In both toolboxes an empty cell is an instruction and not an omission: by default it overwrites the model's existing value, so a table written to fix one charge erases the text annotations in every column that is present but blank in that row. Setting `overwriteWithEmpty` to false, or `overwrite_with_empty=False`, keeps the existing values instead. Empty `grRules` and bounds never overwrite, an empty charge or confidence score becomes 0 when another row has a value, and a charge, confidence-score or annotation (MIRIAM) column with no value at all is not written. In Python, an all-blank text column that is present still clears the stored values. For both toolboxes, include only the columns being curated, and keep the values that should survive in the file and do not rely on the blank, which keeps the value only when `overwriteWithEmpty` is false. ::: -:::{warning} Troubleshooting +:::{admonition} Troubleshooting +:class: warning + - Matching on names that do not match. A trailing space, a different capitalisation, or `name[comp]` written with the wrong compartment makes the row add a duplicate entity and not curate the intended one. - Reactions matched on stoichiometry. Two reactions with the same reactants and products, differing only in direction or bounds, are one reaction to the matcher. - The wrong id prefix. The default is `M_` and `R_`. On a model using `s_` and `r_` the additions are correct, but their ids do not look like anything else in the model. diff --git a/docs/guide/deletions.md b/docs/guide/deletions.md index 8d1cf19..11ef42d 100644 --- a/docs/guide/deletions.md +++ b/docs/guide/deletions.md @@ -258,7 +258,9 @@ The reference has to be computed before the knockout. Without `solution=`, cobra ::: :::: -:::{warning} Troubleshooting +:::{admonition} Troubleshooting +:class: warning + - Confusing gene and reaction knockouts. Deleting a gene silences a reaction only when the GPR says so. Isozymes mask single knockouts, which is what the double deletions test. - Comparing essentiality numbers from the two toolboxes. `getEssentialRxns` measures feasibility and `single_reaction_deletion` measures growth; they are different questions. - A knockout that looks lethal because the medium is wrong. Essentiality is relative to the medium and the objective. State both when reporting it; see [Growth media and conditions](media.md). diff --git a/docs/guide/editing.md b/docs/guide/editing.md index de21837..eac3b9a 100644 --- a/docs/guide/editing.md +++ b/docs/guide/editing.md @@ -267,7 +267,9 @@ PGI rule: 'YLR354C' `removeGenes` and `remove_genes` rewrite each affected GPR without the removed gene. A reaction whose last catalyst is removed stays in the model with its rule dropped and its bounds set to `(0, 0)`: this is the default of `remove_genes` (`blocked_reactions="constrain"`), while `removeGenes` does it when `removeBlockedRxns` is false, its default, and it is the bound state that `knock_out_model_genes` leaves. With `blocked_reactions="keep"` the blocked reactions keep their bounds; setting `removeBlockedRxns` to true, or `blocked_reactions="remove"`, deletes them instead, which turns a gene deletion into a reaction deletion; [Deletions and essentiality](deletions.md) covers when that is the question being asked. When at least one requested gene is in the model, genes that no remaining reaction used are removed as well, also when they were not requested; when none is in the model, no gene is removed. In `removeGenes`, `standardizeRules`, on by default, normalises the rewritten rules. -:::{warning} Troubleshooting +:::{admonition} Troubleshooting +:class: warning + - Editing `model.S` by hand. It leaves `model.rev`, the bounds and the gene matrix describing a different model than the matrix does. Use `changeRxns`. - Deleting a reaction and leaving its metabolites without a reaction. Without the orphan flags, metabolites that nothing produces or consumes remain in the model and appear as gaps. See [Quality control](quality-control.md). - Overwriting a GPR that was meant to be extended. `changeGrRules` and `change_gene_reaction_rules` replace by default. diff --git a/docs/guide/engineering.md b/docs/guide/engineering.md index b30eef9..ac91e93 100644 --- a/docs/guide/engineering.md +++ b/docs/guide/engineering.md @@ -210,7 +210,9 @@ Genes the model does not have and missing (NaN) p-values are dropped; a p-value Both toolboxes compute the background correction in closed form and report the same z-scores and one-sided p-values, sorted by z-score. RAVEN returns them as parallel arrays in a struct, raven-toolbox as one table per gene set. -:::{warning} Troubleshooting +:::{admonition} Troubleshooting +:class: warning + - Reading FSEOF as a prediction. It reports what the stoichiometry allows when product formation is forced, which is a hypothesis to test, not an expected yield. - A target list full of noise. With a lowered correlation threshold, alternative optima appear as targets. If a reaction's flux is not close to linear in the enforced product, its slope is not meaningful. - Forcing a product the model cannot make. If the target exchange cannot carry flux, every step of the scan is infeasible and there is nothing to correlate. Check it with a single solve first. diff --git a/docs/guide/fba.md b/docs/guide/fba.md index e0e1607..c183227 100644 --- a/docs/guide/fba.md +++ b/docs/guide/fba.md @@ -302,7 +302,9 @@ The growth rate is unchanged and the total flux is the smallest that achieves it ::: :::: -:::{warning} Troubleshooting +:::{admonition} Troubleshooting +:class: warning + - The status is `infeasible`: something is over-constrained, most often an uptake that is closed, or a bound set to the wrong sign. See [Growth media and conditions](media.md). - Growth is zero but the solve succeeded: the model is feasible and the optimum is zero, because a nutrient is missing or a gap blocks the biomass pseudoreaction. - The same model gives different flux distributions: this is expected, as alternative optima exist. Use pFBA, or compare ranges with flux variability analysis (FVA), not one solution. diff --git a/docs/guide/fva.md b/docs/guide/fva.md index 810e5a0..71c7539 100644 --- a/docs/guide/fva.md +++ b/docs/guide/fva.md @@ -235,7 +235,9 @@ All three agree here: this particular optimum happens to carry no loop flux, eve ::: :::: -:::{warning} Troubleshooting +:::{admonition} Troubleshooting +:class: warning + - `fraction_of_optimum` is not set: cobrapy defaults to `1.0`, ranges at the optimum. `getAllowedBounds` has no such notion and answers for the whole feasible space. The same call in the two toolboxes therefore asks different questions unless the call states which one is meant. - A flux with a wide range is reported: if the range at the optimum is wide, the number in a results table is one of many equally good answers. - A loop is mistaken for capacity: wide ranges on internal cycles are a property of the stoichiometry and not of the organism; see [Loops in flux ranges](#loops-in-flux-ranges). diff --git a/docs/guide/gap-filling.md b/docs/guide/gap-filling.md index 7391c7c..f345c80 100644 --- a/docs/guide/gap-filling.md +++ b/docs/guide/gap-filling.md @@ -310,7 +310,9 @@ See [Metabolic tasks](tasks.md) for the task list itself, and [Context-specific ::: :::: -:::{warning} Troubleshooting +:::{admonition} Troubleshooting +:class: warning + - Identifiers that do not line up. Gap-filling can only add what it can match. Two models built from different databases share almost no metabolite ids, and the result is a gap-filler that adds nothing, or adds duplicates. - Filling every gap. Some blocked reactions are blocked because the organism cannot perform them. Each addition is a claim about biology, and `fillGaps` makes hundreds of them in one call. - Net production hiding the problem. `allowNetProduction` makes more reactions look connectable by ignoring where the products go. It helps in early stages and misleads in later ones. diff --git a/docs/guide/getting-started.md b/docs/guide/getting-started.md index e11c789..72136f6 100644 --- a/docs/guide/getting-started.md +++ b/docs/guide/getting-started.md @@ -261,7 +261,9 @@ An empty dictionary means the reaction balances. Anything listed is an element, ::: :::: -:::{warning} Troubleshooting +:::{admonition} Troubleshooting +:class: warning + - A lookup fails. Identifiers are case-sensitive and carry the compartment suffix (`G6P_c`, not `G6P`). Neither toolbox returns a sentinel for a name it cannot find: `getIndexes` errors with `Could not find object 'X' in the model` and `get_by_id` raises `KeyError`, so a typo stops the script; nothing else is indexed without warning. - The model loads but nothing grows. In `smallYeast.yml` every uptake reaction is closed (`glcIN` has bounds `[0 0]`). Opening a medium is covered in [Growth media and conditions](media.md). - Gene identifiers differ between model and FASTA. Systematic names (`YBR196C`) and standard names (`PGI1`) are not interchangeable; RAVEN stores the systematic name as the identifier and the standard name as the gene name. diff --git a/docs/guide/homology.md b/docs/guide/homology.md index 080a173..4b49ffd 100644 --- a/docs/guide/homology.md +++ b/docs/guide/homology.md @@ -197,7 +197,9 @@ strict draft: 32 rxns, 46 genes Raising the two thresholds removes five reactions and eight genes. Whether that is an improvement depends on what the draft is for: a model that is curated by hand benefits from the extra candidates, since a wrong reaction is easier to spot than a missing one, while a model used directly for prediction is better with fewer and better-supported reactions. -:::{note} Reproducing a RAVEN 2 reconstruction +:::{admonition} Reproducing a RAVEN 2 reconstruction +:class: note + Some of these defaults differ from the values RAVEN 2 used, so the same script can produce a different draft under RAVEN 3. Set them explicitly, or see [Migrating from RAVEN 2](../raven3-migration.md). ::: @@ -248,7 +250,9 @@ None of the 37 reactions can carry flux. This is not a defect in the draft: with A homology draft is a set of claims about which reactions the organism has. Turning it into a model means giving it a medium ([Growth media and conditions](media.md)), closing the gaps that stop it producing biomass ([Gap-filling](gap-filling.md)), and checking it against what the organism is known to do ([Metabolic tasks](tasks.md)). The [GEM reconstruction protocol](../protocol/index.md) follows that path for *H. polymorpha* at full scale. -:::{warning} Troubleshooting +:::{admonition} Troubleshooting +:class: warning + - Identifiers that do not match. The FASTA headers must carry the same gene ids as the template model's `genes`. A mismatch produces a draft with no reactions and no error. - One template, one organism's biases. Every reaction in the draft comes from the template, so anything the template lacks the draft cannot have. Several templates, with `preferredOrder`, spread that risk. - Reading absence as evidence. A reaction left out means no acceptable hit was found, which does not show that the organism lacks the capability. Sequencing gaps, divergent sequences and short proteins all look the same here. diff --git a/docs/guide/init.md b/docs/guide/init.md index 453a238..c4b5ff8 100644 --- a/docs/guide/init.md +++ b/docs/guide/init.md @@ -135,7 +135,9 @@ Reading Human-GEM from the Systems Biology Markup Language (SBML) file alone tak ::: :::: -:::{note} Spontaneous reactions +:::{admonition} Spontaneous reactions +:class: note + Pass the list of spontaneous reactions. ftINIT keeps every reaction passed as spontaneous in every extracted model, whatever the data say. Other reactions without a gene rule are kept only where the optimisation needs them, so they can differ from sample to sample. In Human-GEM, `spontaneous` marks reactions that need no enzyme. That includes diffusion: transport between cytosol and nucleus through the nuclear pores, and gases and water (O2, CO2, H2O, NH3, H2O2, NO, H2S) crossing membranes. Leaving out the list lets ftINIT drop such transports in some samples. The model then uses other routes. For example, without the dCTP transport into the nucleus, deoxycytidine kinase (DCK) becomes essential for DNA synthesis, which is an artefact of the extraction and not biology. ::: @@ -381,7 +383,9 @@ print(sum(1 for result in report if result.passed), "of", len(report), "tasks pa Compare the result against the template and against the other samples: [Quality control](quality-control.md) covers the structural checks, and the Human-GEM guide's [GEM comparison](https://sysbiochalmers.github.io/Human-GEM-guide/gem_comparison/) covers comparing many extracted models at once. -:::{warning} Troubleshooting +:::{admonition} Troubleshooting +:class: warning + - Identifiers that do not match. Human-GEM uses ENSEMBL identifiers, and so does `Hart2015_RNAseq.txt`, so no mapping step appears in the workflow. Expression data with symbols or systematic names needs a mapping; score a few genes and check that they are not all at the floor before spending 37 to 47 minutes on the preparation. - Re-preparing per sample. The preparation depends only on the template and the tasks. Do it once, save it, and reuse it; preparation and extraction are separate, which makes that possible. - No Gurobi. Both extractions are mixed-integer problems; on Human-GEM, GLPK completed a benchmark extraction in 2182 s with a time-limited result (Jaccard 0.892 against Gurobi; Human-GEM `38567ab8`, no-task scaled prep, series `1+1`, `time_limit=900`, a different input from the one in this workflow). diff --git a/docs/guide/io.md b/docs/guide/io.md index 8d28e3a..b88d6fe 100644 --- a/docs/guide/io.md +++ b/docs/guide/io.md @@ -252,7 +252,9 @@ print(sorted(p.relative_to(root).as_posix() for p in root.rglob("*") if p.is_fil ::: :::: -:::{warning} Troubleshooting +:::{admonition} Troubleshooting +:class: warning + - The SBML writer refuses an identifier. SBML ids must start with a letter or underscore. Both toolboxes add a prefix to a whole field when any id in it does not, so `importModel` and `read_sbml_model` strip prefixes on the way back. - The model id differs between two copies of the same release. cobrapy decodes `__46__`-style escapes in reaction, metabolite and gene ids, but not in the model id, so yeast-GEM read from SBML reports `yeastGEM_v9__46__1__46__0`. Decode it with `re.sub(r"__(\d+)__", lambda m: chr(int(m.group(1))), model.id)`, or compare `model.name`. Do not match entities on annotations: some database ids are shared by differently named metabolites. - Excel export fails with a missing module. The Python side needs `openpyxl`; install `raven-toolbox[excel]`. diff --git a/docs/guide/kegg.md b/docs/guide/kegg.md index a7efec2..bbddbca 100644 --- a/docs/guide/kegg.md +++ b/docs/guide/kegg.md @@ -117,7 +117,9 @@ print(f"{len(strict.reactions)} rxns, {len(strict.genes)} genes " That is a change of nine reactions on this organism. The size of the effect differs per organism. Measure it for each organism. Unlike the other flags, `keepGeneral` / `keep_general` is off by default: general reactions are placeholders like "an alcohol + NAD+", and admitting them produces a network that appears to do far more than it can. -:::{warning} Troubleshooting +:::{admonition} Troubleshooting +:class: warning + - Expecting a model. No biomass reaction, no exchanges, no compartments, no medium. The draft is a reaction inventory with gene associations. - Trusting the KO assignment. A KO is an orthology group and not a demonstrated activity in the organism, and a partial HMM hit is weaker still. - Undefined stoichiometry. Reactions with `n` or `x` coefficients cannot be mass-balanced. They are kept by default because dropping them removes real chemistry, but they interfere with [Quality control](quality-control.md) later. diff --git a/docs/guide/media.md b/docs/guide/media.md index 02e4e67..e243680 100644 --- a/docs/guide/media.md +++ b/docs/guide/media.md @@ -344,7 +344,9 @@ The values are uptake rates and not bounds, and the zeros are components that th ::: :::: -:::{warning} Troubleshooting +:::{admonition} Troubleshooting +:class: warning + - `infeasible` right after setting a medium: something essential is closed. Reopen the medium one component at a time, or start from `minimal_medium` on the working model and compare. - A sign error from mixing conventions: for an exchange written `metabolite <=>`, as in yeast-GEM, uptake is a negative flux, but `model.medium` takes positive numbers. Both are correct; mixing them is not. - The model grows without a carbon source: usually a leak, where some reaction produces carbon from nothing. [Quality control](quality-control.md) covers the diagnosis, with `canExchange` and `analyse_topology`. diff --git a/docs/guide/model-structure.md b/docs/guide/model-structure.md index 17e4ea0..6bda0e8 100644 --- a/docs/guide/model-structure.md +++ b/docs/guide/model-structure.md @@ -313,7 +313,9 @@ same object: True RAVEN and the COBRA Toolbox use different field names for the same model. In MATLAB, `ravenCobraWrapper` converts a struct in either direction, inferring which direction from the fields it finds. There is no Python equivalent because there is nothing to convert: a raven-toolbox model is a `cobra.Model`, which every Python COBRA tool takes. -:::{warning} Troubleshooting +:::{admonition} Troubleshooting +:class: warning + - Fields drift out of alignment. In MATLAB, editing one reaction-length field by hand without editing the others leaves a model that looks fine and fails later, often far from the edit. `checkModelStruct` detects it. - A field ends up with mixed contents. `model.subSystems` takes strings or cell arrays of strings, but not both in the same model. Convert the whole field, not one entry. - `model.rev` disagrees with the bounds. RAVEN stores reversibility explicitly, so it can contradict `lb` and `ub` after a manual edit, and different functions consult different ones. Python cannot get into that state, because `reaction.reversibility` is computed from the bounds on every read. diff --git a/docs/guide/quality-control.md b/docs/guide/quality-control.md index 5008853..c6c351e 100644 --- a/docs/guide/quality-control.md +++ b/docs/guide/quality-control.md @@ -254,7 +254,9 @@ one leak: ['F6P_c', 'GP_c', 'PYR_c'] ::: :::: -:::{note} Outlets for by-products in leak tests +:::{admonition} Outlets for by-products in leak tests +:class: note + The leak-test functions assume every metabolite can be excreted while the test runs: `canExchange` and `gap_report` add an output reaction for each metabolite they check, and `findLeakMetabolite` and `find_leak_metabolite` take `allowExcretion` (`allow_excretion` in Python) as true by default. A hand-written loop has to add the outlets explicitly. Testing one demand reaction at a time, with no outlet for the by-products, reports zero leaks in this model instead of 33, because a leaking reaction is then blocked by its own stoichiometry. ::: @@ -308,7 +310,9 @@ Models differ (11 differences): `diffModels` compares two models entry by entry: which reactions are unique to each, and where a shared reaction differs in stoichiometry, bounds, objective coefficient, gene rule or metabolite properties. [Comparing models](comparing.md) uses the same function, here applied to a model and its last good version. -:::{warning} Troubleshooting +:::{admonition} Troubleshooting +:class: warning + - Judging a model by whether it solves. An unbalanced reaction makes it solve better. Balance first, then believe the growth rate. - Treating every unbalanced reaction as a bug. Exchange, demand, sink and biomass pseudo-reactions are unbalanced on purpose. - `unknown` instead of `unbalanced`. A metabolite with no formula, or a polymer formula cobrapy cannot parse, makes the balance undecidable, not wrong. Fill in the formula and check again. diff --git a/docs/guide/sampling.md b/docs/guide/sampling.md index a295cc9..584aa10 100644 --- a/docs/guide/sampling.md +++ b/docs/guide/sampling.md @@ -86,7 +86,9 @@ method: achr ::: :::: -:::{warning} Transposed layouts +:::{admonition} Transposed layouts +:class: warning + RAVEN returns reactions × samples; raven-toolbox returns a DataFrame of samples × reactions, the `cobra.sampling` layout. Every mean, histogram and correlation has to use the right axis, and a wrong axis gives no error, only numbers other than the intended ones. ::: @@ -248,7 +250,9 @@ Passing the list back as `good_reactions=` on a later call skips the FVA, the ex ::: :::: -:::{warning} Troubleshooting +:::{admonition} Troubleshooting +:class: warning + - No seed. The numbers change in every run. Set one, and report it with the solver: the warmup points come from LPs, so another solver (or, with GLPK, a solve of the same model before the sampling) gives another chain (see [First sample](#first-sample)). Report the distribution, not the draws. - Too little thinning. Consecutive MCMC steps are correlated, so only every 100th is recorded. That is enough for this model, not for a genome-scale one: on yeast-GEM, samples 100 steps apart are 0.93 correlated and independent chains disagree on most reactions (see the [sampling convergence study](../parameter-tuning/studies/sampling-convergence-calibration.md)). For results that are reported, run chains with different seeds and check that they agree. - Reading a mean as a prediction. The mean of a sample describes the feasible space and the sampler, not the cell. It is a feasible state, but it is located where most of the volume is: in [Distribution and range](#distribution-and-range) that puts mean growth below half the optimum. With `randomObjective` the samples are vertices, so a reaction can average 5 while every sample has it at 0 or 10. diff --git a/docs/guide/solvers.md b/docs/guide/solvers.md index 794f81c..8008530 100644 --- a/docs/guide/solvers.md +++ b/docs/guide/solvers.md @@ -200,7 +200,9 @@ Check the status before using a number. `slim_optimize` skips building a full so An infeasible model is over-constrained: the constraints as written have no solution. That is different from a feasible model whose optimum is zero, which is a statement about biology, not about the constraints. -:::{warning} Troubleshooting +:::{admonition} Troubleshooting +:class: warning + - A solver that is set but not installed. `setRavenSolver('gurobi')` stores the preference whether or not Gurobi is there; the failure appears at the next solve. `checkRaven` tests it directly. - A large MILP on GLPK. Both toolboxes send MILPs to GLPK, and RAVEN warns when it does. On a genome-scale model, gap-filling and ftINIT can run for hours on GLPK where Gurobi takes minutes; use Gurobi for those. - Reading fluxes after a non-optimal solve. `sol.stat` of `0` means a feasible point was found but not proven optimal, and `-2` means the parsimonious step failed after an optimal first solve. Both return numbers. diff --git a/docs/guide/tasks.md b/docs/guide/tasks.md index 4b48814..96113c8 100644 --- a/docs/guide/tasks.md +++ b/docs/guide/tasks.md @@ -223,7 +223,9 @@ from raven_toolbox.init import fill_tasks ::: :::: -:::{warning} Troubleshooting +:::{admonition} Troubleshooting +:class: warning + - A metabolite given by its identifier: task files address metabolites by name, not by identifier, as `name[compartment]`. A task that matches nothing without an error is a task that passes for the wrong reason. - Missing `SHOULD FAIL` tasks: the tasks that matter most are often the ones the model must not satisfy; without them, a leaking model passes everything. - The boundary a task ran against is unknown: with the model closed, a task defines its own inputs and does not inherit the model's medium, so a task passing says nothing about whether the model grows on the medium it has. Left open (the default in both toolboxes), the model's own exchanges take part, and a task can pass because of an uptake or secretion it never lists. diff --git a/docs/installation/data-and-binaries.md b/docs/installation/data-and-binaries.md index 9992414..88ab3b2 100644 --- a/docs/installation/data-and-binaries.md +++ b/docs/installation/data-and-binaries.md @@ -115,7 +115,9 @@ Every tool is invoked as a subprocess, so the availability of a native build for On Windows, homology reconstruction and the KEGG query path run natively, since they need only the first three tools. Building an HMM library needs MAFFT and CD-HIT, which have no Windows builds. Use Windows Subsystem for Linux 2 (WSL2) for that and keep the whole stack inside it, since raven-toolbox calls the resolved executable directly and does not translate paths between Windows and WSL. -:::{warning} Troubleshooting +:::{admonition} Troubleshooting +:class: warning + | Problem | Possible reason | Solution | |---|---|---| | A machine with no network | The first reconstruction needs to download files, which fails without a network. | Fetch on a connected machine, copy `~/.cache/raven_toolbox` and set `RAVEN_PYTHON_AUTOFETCH=0`. | diff --git a/docs/installation/raven.md b/docs/installation/raven.md index 797881f..185b202 100644 --- a/docs/installation/raven.md +++ b/docs/installation/raven.md @@ -51,7 +51,9 @@ setRavenSolver('cobra'); ## Installation -:::{warning} Installing 3.0.0b1 +:::{admonition} Installing 3.0.0b1 +:class: warning + This site documents RAVEN 3.0.0b1, a pre-release. The Add-Ons manager and the normal releases page of the Release download offer only the latest stable release (RAVEN 2.x), so neither method installs 3.0.0b1. Use the Clone with git tab, or the direct archive link in the Release download tab. ::: diff --git a/docs/protocol/lipid-curation.md b/docs/protocol/lipid-curation.md index fca8509..fed5030 100644 --- a/docs/protocol/lipid-curation.md +++ b/docs/protocol/lipid-curation.md @@ -2,7 +2,9 @@ After adjusting the biomass composition, it is useful to curate reactions already known to differ in the target organism. Using the **SLIME** formalism (Split Lipids Into Measurable Entities) to describe lipid metabolism requires curating the lipid reactions to match the lipids in the biomass reaction. -:::{note} Order of curation and gap-filling +:::{admonition} Order of curation and gap-filling +:class: note + This step can also be done after gap-filling, but the curation may influence the gap-filling result, so it is done first here. ::: diff --git a/docs/protocol/materials.md b/docs/protocol/materials.md index 0f6a775..38e81e9 100644 --- a/docs/protocol/materials.md +++ b/docs/protocol/materials.md @@ -34,7 +34,9 @@ Homology-based reconstruction requires files for both the target organism and th The target organism requires (1) a protein FASTA of all proteins in its genome, (2) a DNA FASTA for nucleotide ratios, and (3) a GenBank file for ribonucleotide ratios. Each template requires a protein FASTA and a model file in SBML format. -:::{important} Matching identifiers +:::{admonition} Matching identifiers +:class: important + For automatic gene matching, the protein FASTA and the model file of each template must use the same gene identifiers. ::: @@ -54,7 +56,9 @@ checkRaven See [Installing RAVEN](#verify) for what a successful run looks like. -:::{warning} Troubleshooting +:::{admonition} Troubleshooting +:class: warning + - Problem: `checkRaven` reports Fail for Export Excel format. - Possible reason: The protocol script reports that MATLAB's Text Analytics Toolbox can conflict with reading Excel files. - Solution: uninstall the Text Analytics Toolbox. For support, see the [RAVEN issues](https://github.com/CompMetEng/RAVEN/issues) page. diff --git a/docs/protocol/reconstruction.md b/docs/protocol/reconstruction.md index a6d422f..78f8e76 100644 --- a/docs/protocol/reconstruction.md +++ b/docs/protocol/reconstruction.md @@ -25,7 +25,9 @@ In this protocol a draft model is reconstructed for the industrially relevant me | [yeast-GEM](https://github.com/SysBioChalmers/yeast-GEM) | *Saccharomyces cerevisiae* (ascomycete) | Primary template | | [rhto-GEM](https://github.com/SysBioChalmers/rhto-GEM) | *Rhodotorula toruloides* (basidiomycete) | Complementary template | -:::{warning} Same identifier namespace +:::{admonition} Same identifier namespace +:class: warning + Multiple template models must use the same identifiers for metabolites, reactions and compartments. The yeast-GEM and rhto-GEM models used here share the same identifier style. ::: diff --git a/docs/tuned-parameters.md b/docs/tuned-parameters.md index b82f8ca..ba4e527 100644 --- a/docs/tuned-parameters.md +++ b/docs/tuned-parameters.md @@ -2,7 +2,9 @@ Every RAVEN parameter with a non-trivial default has been through one of three processes: matched to a literature value from the original paper of the method, measured empirically on real or synthetic models, or kept for consistency with an existing convention (the default of a widely used underlying tool, or a value already established elsewhere in RAVEN). The short version (current default and a one-line reason) is grouped by capability and covers both the MATLAB and Python (raven-toolbox) implementations. The linked studies give the full measurements and methodology. -:::{note} Differences between the two implementations +:::{admonition} Differences between the two implementations +:class: note + The MATLAB implementation of RAVEN dates from 2013 and the Python implementation (raven-toolbox) from 2026. The parameters re-measured in one implementation have been ported to the other (the homology and KEGG cut-offs). Un-measured defaults of both implementations were not assumed correct; where the two disagreed, the default was measured, with no deference to whichever implementation came first. Every row states the current value(s); where the two differ, both are given. [Cross-toolbox parity decisions](#cross-toolbox-parity-decisions) gives the full accounting, including the cases that are deliberately kept different because the two implementations rest on different solvers or algorithms. ::: From 68a74d9f56dc6f4eb065c2057f42add672e84583 Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Sat, 31 Oct 2026 23:13:48 +0100 Subject: [PATCH 03/20] Rewrite the parameter-tuning pages from the scripts and the code Each page states the question, the set-up, the results with their meaning, the choice of settings and the limits. Numbers, tables and commands are unchanged except where the script or the pinned code shows another value, and statements that no measurement supports are marked as inferences or removed. tuned-parameters.md, the guide and the comparison page agree with the studies, and every link into a study uses its title. --- docs/guide/engineering.md | 2 +- docs/guide/homology.md | 4 +- docs/guide/init.md | 6 +- docs/guide/kegg.md | 2 +- docs/guide/sampling.md | 2 +- docs/parameter-tuning/benchmarks.md | 83 +++++---- docs/parameter-tuning/index.md | 49 +++--- .../studies/ftinit-determinism.md | 166 +++++++++--------- .../studies/homology-cutoff-calibration.md | 98 ++++++----- .../studies/humangem-validation.md | 109 ++++++++---- .../studies/init-param-calibration.md | 154 ++++++++++------ .../studies/init-solver-benchmark.md | 57 +++--- .../studies/kegg-hmm-cutoff-calibration.md | 165 ++++++++++------- .../sampling-convergence-calibration.md | 155 +++++++++------- .../same-function-different-answer.md | 2 +- docs/tuned-parameters.md | 60 +++---- scripts/known_names.txt | 11 ++ 17 files changed, 665 insertions(+), 460 deletions(-) diff --git a/docs/guide/engineering.md b/docs/guide/engineering.md index ac91e93..ecc4523 100644 --- a/docs/guide/engineering.md +++ b/docs/guide/engineering.md @@ -95,7 +95,7 @@ Both tabs find 14 amplification targets here. `fseof` classifies the fluxes of a `fseof` returns the whole scan rather than only its conclusion. `scan` is the reactions by enforced-flux-level matrix of fluxes, `enforced` the levels themselves, and `targets` the classified per-reaction table sorted by score. `gene_targets` aggregates the same result to genes, which is the form a strain-design list usually takes. -`correlation_threshold` matches MATLAB's `corrThreshold`, and `flux_eps` is the tolerance of the check against the theoretical maximum flux (on iJO1366 with target `EX_succ_e` the counts are the same at `1e-8`, `1e-7` and `1e-6`, see [Parameter benchmarks](../parameter-tuning/benchmarks.md)). `min_target` sets the floor of the enforced range, by default one step (the top of the scan divided by `n_steps`); passing the target reaction's flux at the growth optimum starts the scan at the level the cell already has. +`correlation_threshold` matches MATLAB's `corrThreshold`, and `flux_eps` is the tolerance of the check against the theoretical maximum flux (on iJO1366 with target `EX_succ_e` the counts are the same at `1e-8`, `1e-7` and `1e-6`, see [Benchmarks of `fseof`, `remove_genes` and other defaults](../parameter-tuning/benchmarks.md)). `min_target` sets the floor of the enforced range, by default one step (the top of the scan divided by `n_steps`); passing the target reaction's flux at the growth optimum starts the scan at the level the cell already has. ::: :::: diff --git a/docs/guide/homology.md b/docs/guide/homology.md index 4b49ffd..d3aad75 100644 --- a/docs/guide/homology.md +++ b/docs/guide/homology.md @@ -106,7 +106,7 @@ Both call the same BLAST+ executables with the same parameters, so the hit count Without `threads`, `run_blast`, `run_diamond` and `run_supporth` use every CPU the process may run on. In MATLAB, `getBlast`, `getDiamond` and `runSuppOrth` use the cores assigned to MATLAB (`feature('numcores')`); `runSuppOrth` is the one of the three with a `threads` option. -`getDiamond` and `run_diamond` are drop-in alternatives that search with DIAMOND instead. DIAMOND indexes the database and searches in reduced amino-acid alphabets, which makes it one to two orders of magnitude faster on a full proteome pair (minutes against hours), and less sensitive for distant homologs, where the seeds it uses are less likely to match. For a template within the same genus the difference is small; for a template several hundred million years away, BLASTP finds pairs DIAMOND misses. +`getDiamond` and `run_diamond` are drop-in alternatives that search with DIAMOND instead. DIAMOND indexes the database and searches in reduced amino-acid alphabets, which made it 10.5 and 19.7 times faster than BLASTP on two full proteome pairs (34 s against 358 s and 40 s against 788 s in [Homology cut-off calibration](../parameter-tuning/studies/homology-cutoff-calibration.md)), and less sensitive for distant homologs, where the seeds it uses are less likely to match. For a template within the same genus the difference is small; for a template several hundred million years away, BLASTP finds pairs DIAMOND misses. `makeFakeBlastStructure` and `make_ortholog_hits` wrap orthology assignments from another source (OrthoFinder, OMA, a published table) in the structure the transfer step expects, so no search is run. @@ -159,7 +159,7 @@ Three cut-offs control which hits are accepted, and tightening any of them shrin `maxE` / `max_evalue` (`1e-30`) is the maximum BLAST E-value, the number of hits of at least this quality expected by chance in a database this size. The default is strict by BLAST standards, where `1e-5` is a common threshold, because transferring a reaction on a marginal hit adds a claim about metabolism that nothing downstream questions. -`minLen` / `min_align_len` (`100`) is the minimum aligned length in residues. It rejects hits that align well over a short conserved domain while the rest of the protein is unrelated. The value was measured against KEGG and OMA orthology assignments across four organisms: anything at or below 150 gave the same result, and higher values discarded genuine orthologs whose alignment was interrupted. +`minLen` / `min_align_len` (`100`) is the minimum aligned length in residues. It rejects hits that align well over a short conserved domain while the rest of the protein is unrelated. The value was measured against KEGG orthology assignments across four organisms ([Homology cut-off calibration](../parameter-tuning/studies/homology-cutoff-calibration.md)): 50 and 100 gave the same score, 150 a score at most 0.003 lower, and 200 and 300 lower scores, mainly through lost recall. `minIde` / `min_identity` (`40`) is the minimum percentage identity across the alignment. diff --git a/docs/guide/init.md b/docs/guide/init.md index c4b5ff8..ac878a3 100644 --- a/docs/guide/init.md +++ b/docs/guide/init.md @@ -67,11 +67,11 @@ git clone --depth=1 https://github.com/SysBioChalmers/Human-GEM.git | `data/metabolicTasks/metabolicTasks_Essential.txt` | 57 tasks the extracted model must pass | | `data/datasets/Hart2015_RNAseq.txt` | TPM for five cell lines: DLD1, GBM, HCT116, HELA, RPE1 | -Both extractions are mixed-integer problems. On Human-GEM, GLPK completed a benchmark extraction in 2182 s with a time-limited result (Jaccard 0.892 against Gurobi), so set Gurobi up first ([Solvers and configuration](solvers.md)). That benchmark used Human-GEM `38567ab8` (cleaned), the no-task scaled prep, series `1+1` and `time_limit=900`, a different input from the `6812be2a` template with task preparation used here. In raven-toolbox, without `time_limit`, the scheduled runs of a series keep their own limits (120 s and 5000 s, plus a 5 s first run in the last step of `1+1` and `2+1`; 10000 s for `full`, which runs once), for GLPK as for Gurobi. With `setRavenSolver('glpk')`, RAVEN does not pass the `TimeLimit` of the schedule to GLPK: `optimizeProb` sets a 1000 s limit on every GLPK MILP. +Both extractions are mixed-integer problems. On Human-GEM, GLPK completed a benchmark extraction in 2182 s with a time-limited result (Jaccard 0.892 against Gurobi), so set Gurobi up first ([Solvers and configuration](solvers.md)). That benchmark ([ftINIT on different solvers](../parameter-tuning/studies/init-solver-benchmark.md)) used Human-GEM `38567ab8` (cleaned), prepared without a task list and with scaled stoichiometry, series `1+1` and `time_limit=900`, a different input from the `6812be2a` template with task preparation used here. In raven-toolbox, without `time_limit`, the scheduled runs of a series keep their own limits (120 s and 5000 s, plus a 5 s first run in the last step of `1+1` and `2+1`; 10000 s for `full`, which runs once), for GLPK as for Gurobi. With `setRavenSolver('glpk')`, RAVEN does not pass the `TimeLimit` of the schedule to GLPK: `optimizeProb` sets a 1000 s limit on every GLPK MILP. ## Template preparation -The preparation finds the task-essential reactions, classifies every reaction into omics-independent categories, merges linear stretches, and rescales the stoichiometry to a narrower coefficient range, which is not needed for feasibility at the default `big_m` and shortens the run (83 s against 157 s in the [calibration](../parameter-tuning/studies/init-param-calibration.md)). It depends only on the template and the task list, and not on the expression data, so it is done once and reused for every sample. +The preparation finds the task-essential reactions, classifies every reaction into omics-independent categories, merges linear stretches, and rescales the stoichiometry to a narrower coefficient range, which is not needed for feasibility at the default `big_m` and shortens the run (83 s against 157 s in [ftINIT parameter calibration and input robustness](../parameter-tuning/studies/init-param-calibration.md)). It depends only on the template and the task list, and not on the expression data, so it is done once and reused for every sample. ::::{tab-set} :::{tab-item} Ⓜ️ MATLAB @@ -388,7 +388,7 @@ Compare the result against the template and against the other samples: [Quality - Identifiers that do not match. Human-GEM uses ENSEMBL identifiers, and so does `Hart2015_RNAseq.txt`, so no mapping step appears in the workflow. Expression data with symbols or systematic names needs a mapping; score a few genes and check that they are not all at the floor before spending 37 to 47 minutes on the preparation. - Re-preparing per sample. The preparation depends only on the template and the tasks. Do it once, save it, and reuse it; preparation and extraction are separate, which makes that possible. -- No Gurobi. Both extractions are mixed-integer problems; on Human-GEM, GLPK completed a benchmark extraction in 2182 s with a time-limited result (Jaccard 0.892 against Gurobi; Human-GEM `38567ab8`, no-task scaled prep, series `1+1`, `time_limit=900`, a different input from the one in this workflow). +- No Gurobi. Both extractions are mixed-integer problems; on Human-GEM, GLPK completed a benchmark extraction in 2182 s with a time-limited result (Jaccard 0.892 against Gurobi; Human-GEM `38567ab8` prepared without a task list and with scaled stoichiometry, series `1+1`, `time_limit=900`, a different input from the one in this workflow). - A model without capabilities assumed of it. Without a task list there is nothing to repair against, and the extraction is free to remove capabilities that were never checked. ::: diff --git a/docs/guide/kegg.md b/docs/guide/kegg.md index bbddbca..3c30914 100644 --- a/docs/guide/kegg.md +++ b/docs/guide/kegg.md @@ -85,7 +85,7 @@ This method is expensive. The eukaryotic HMM library alone is 129 MB compressed, With a `fastaFile`, the organism id only names the model; the search uses no phylogenetic distance and ignores `maxPhylDist`. -Three cut-offs shape the assignment. `cutOff` is the HMMER significance a hit needs at all. `minScoreRatioKO` drops genes that fit a KO far worse than the best gene in it, and `minScoreRatioG` drops KOs that fit a gene far worse than that gene's best KO; the two prune the same matrix from opposite sides. All three were measured in the [KEGG HMM cut-off study](../parameter-tuning/studies/kegg-hmm-cutoff-calibration.md); `cutOff` and `minScoreRatioG` changed from the RAVEN 2 values, while `minScoreRatioKO` proved inert and keeps its original value. +Three cut-offs shape the assignment. `cutOff` is the HMMER significance a hit needs at all. `minScoreRatioKO` drops genes that fit a KO far worse than the best gene in it, and `minScoreRatioG` drops KOs that fit a gene far worse than that gene's best KO; the two prune the same matrix from opposite sides. All three were measured in [KEGG HMM cut-off calibration](../parameter-tuning/studies/kegg-hmm-cutoff-calibration.md); `cutOff` and `minScoreRatioG` changed from the RAVEN 2 values. Raising `minScoreRatioKO` from 0.0 to 0.5 changed precision by 0.01 to 0.03 and recall not at all, and it keeps its original value of 0.3. ## Contents of a KEGG draft diff --git a/docs/guide/sampling.md b/docs/guide/sampling.md index 584aa10..42097d8 100644 --- a/docs/guide/sampling.md +++ b/docs/guide/sampling.md @@ -254,7 +254,7 @@ Passing the list back as `good_reactions=` on a later call skips the FVA, the ex :class: warning - No seed. The numbers change in every run. Set one, and report it with the solver: the warmup points come from LPs, so another solver (or, with GLPK, a solve of the same model before the sampling) gives another chain (see [First sample](#first-sample)). Report the distribution, not the draws. -- Too little thinning. Consecutive MCMC steps are correlated, so only every 100th is recorded. That is enough for this model, not for a genome-scale one: on yeast-GEM, samples 100 steps apart are 0.93 correlated and independent chains disagree on most reactions (see the [sampling convergence study](../parameter-tuning/studies/sampling-convergence-calibration.md)). For results that are reported, run chains with different seeds and check that they agree. +- Too little thinning. Consecutive MCMC steps are correlated, so only every 100th is recorded. That is enough for this model, not for a genome-scale one: on yeast-GEM, samples 100 steps apart are 0.93 correlated and independent chains disagree on most reactions (see [Flux sampling convergence and settings](../parameter-tuning/studies/sampling-convergence-calibration.md)). For results that are reported, run chains with different seeds and check that they agree. - Reading a mean as a prediction. The mean of a sample describes the feasible space and the sampler, not the cell. It is a feasible state, but it is located where most of the volume is: in [Distribution and range](#distribution-and-range) that puts mean growth below half the optimum. With `randomObjective` the samples are vertices, so a reaction can average 5 while every sample has it at 0 or 10. - Sampling a model that is wide open. With an unconstrained medium the space is enormous and the distribution says nothing. Constrain first. - Loops. They inflate the space being sampled, and every sample drawn inside a cycle is wasted. See [Flux variability](fva.md). diff --git a/docs/parameter-tuning/benchmarks.md b/docs/parameter-tuning/benchmarks.md index e88ca32..e6620db 100644 --- a/docs/parameter-tuning/benchmarks.md +++ b/docs/parameter-tuning/benchmarks.md @@ -1,55 +1,72 @@ -# Parameter benchmarks +# Benchmarks of `fseof`, `remove_genes` and other defaults -Parameters whose defaults were measured but did not need a study of their own. -The measurements were run on 2026-06-20 on yeast-GEM (4,102 reactions), iJO1366 (2,583) and e_coli_core (95); the `fseof` and `remove_genes` measurements in the next section were run on 2026-10-30. +These benchmarks record parameters whose defaults were measured or checked without a study of their own. Two of them were run with raven-toolbox commit `94f39407` on 2026-10-30: the effect of `flux_eps` on the targets of `fseof`, and the handling of reactions without an enzyme in `remove_genes`. A table lists literature and convention values, checked on 2026-06-20, with what supports each of them. -The parameters that had a dedicated campaign are in the studies listed under [Parameter tuning](index.md); the current value and one-line reason for every parameter on either side is in [Tuned parameter defaults](../tuned-parameters.md). +The parameters with a measurement campaign of their own have studies listed under [Parameter tuning](index.md). [Tuned parameter defaults](../tuned-parameters.md) gives the current value and a one-line reason for every parameter of both implementations. -## Measurements of `fseof` and `remove_genes` defaults +## Set-up -### `fseof` classifier thresholds and `flux_eps` +The `fseof` benchmark ran on iJO1366 (2,583 reactions) with the biomass reaction of the model as objective and `EX_succ_e` (succinate exchange) as target. The other arguments were `n_steps` 10, `max_fraction` 0.9, `correlation_threshold` 0.9 and the default `equality=False`, which sets each enforced flux as the lower bound of the target reaction. -`fseof` and RAVEN's `FSEOF` classify a reaction from its fluxes across the scan with the same fixed thresholds: a reaction is a target when some absolute flux exceeds `1e-10`, the standard deviation of its absolute flux is above 0 and the correlation reaches `correlation_threshold`; it is a knockout when the slope is negative and the last absolute flux is at most `1e-8`. `flux_eps` is the floor below which the theoretical maximum of the target flux is treated as zero, in which case the scan raises an error. It takes no part in the classification, and RAVEN's `FSEOF` has no `flux_eps` parameter. On iJO1366 with target `EX_succ_e`, `flux_eps` does not change the counts. +`fseof` solves each enforced level in two steps, as RAVEN's `FSEOF` does: it maximises the objective, then minimises the sum of fluxes with the objective held at 0.999999 of that optimum. A reaction is a target when its absolute flux exceeds `1e-10` at some level, the standard deviation of its absolute flux is above 0, the absolute correlation between its absolute flux and the enforced flux reaches `correlation_threshold`, and the slope of the linear fit of its absolute flux against the enforced flux is not 0. A target with a positive slope is an amplify target. A target with a negative slope is a knockout target when its last absolute flux is at most `1e-8`, and a knockdown target otherwise. `fseof` and RAVEN's `FSEOF` use these thresholds as fixed values. -Measured on iJO1366 (2,583 reactions) with the model's biomass reaction as objective, `EX_succ_e` as target, `n_steps` 10, `max_fraction` 0.9, `correlation_threshold` 0.9 and the default `equality=False`, with raven-toolbox commit `94f39407`, run on 2026-10-30: +The `remove_genes` benchmark ran on e_coli_core (95 reactions) from the cobrapy distribution, with bounds as shipped, Gurobi 13.0.1, raven-toolbox commit `94f39407` and RAVEN commit `2d0630e3`. It deleted `b1779`, the only gene of GAPD (glyceraldehyde-3-phosphate dehydrogenase), with each setting of `blocked_reactions` in `remove_genes` and of `removeBlockedRxns` in RAVEN's `removeGenes`. The predicted growth is the optimal flux through the biomass reaction of the model, in h⁻¹. The unmodified model has 95 reactions and predicts a growth rate of 0.874 h⁻¹. -| `flux_eps` | Amplified | Knocked down or out | +The checks of literature and convention values ran on 2026-06-20 with yeast-GEM (4,102 reactions), iJO1366 (2,583) and e_coli_core (95). The table names the model for the `fseof` rows only. + +## Results + +### `fseof` targets at three values of `flux_eps` + +| `flux_eps` | Amplify targets | Knockdown and knockout targets | |---|---|---| | `1e-8` | 18 | 408 | | `1e-7` | 18 | 408 | -| **`1e-6`** *(default)* | 18 | **408** | - -Of the 408 reactions, 406 are knockdown targets and 2 are knockout targets. The counts are the same at the three values on this model and target. Each level is solved in two steps with the objective held at 0.999999 of its optimum in the second. +| `1e-6` (default) | 18 | 408 | -### `remove_genes` handling of reactions without an enzyme +Of the 408 reactions, 406 are knockdown targets and 2 are knockout targets. -`blocked_reactions` in `remove_genes`, and `removeBlockedRxns` in RAVEN's `removeGenes`, decide what happens to a reaction whose gene rule is emptied by a deletion. Both tools constrain such a reaction by default. Deleting `b1779` from e_coli_core removes the only gene for GAPD, which is essential for aerobic growth on glucose. The unmodified model has 95 reactions and predicts a growth rate of 0.874. +### `remove_genes` after deletion of `b1779` from e_coli_core -| Call | Reactions left | Predicted growth | GAPD | +| Call | Reactions left | Predicted growth (h⁻¹) | GAPD | |---|---|---|---| -| `remove_genes(..., blocked_reactions="constrain")` *(default)* | 95 | 0.000 | kept, bounds `(0, 0)` | +| `remove_genes(..., blocked_reactions="constrain")` (default) | 95 | 0.000 | kept, bounds `(0, 0)` | | `remove_genes(..., blocked_reactions="remove")` | 94 | 0.000 | deleted | | `remove_genes(..., blocked_reactions="keep")` | 95 | 0.874 | kept, bounds unchanged | -| `removeGenes(model, genes)` *(`removeBlockedRxns` false, default)* | 95 | 0.000 | kept, bounds `[0 0]` | +| `removeGenes(model, genes)` (`removeBlockedRxns` false, default) | 95 | 0.000 | kept, bounds `[0 0]` | | `removeGenes(model, genes, 'removeBlockedRxns', true)` | 94 | 0.000 | deleted | -`"constrain"` and RAVEN's default give the same model. Both predict no growth after the deletion, because the reaction stays in the network with no capacity to carry flux. `"remove"` and `removeBlockedRxns` true also predict no growth and differ in that the reaction is deleted, which matters when the number of reactions or the reaction identifiers are used afterwards. `"keep"` leaves the reaction with an empty rule and unchanged bounds, so it runs unconstrained and the model predicts growth although the only enzyme of an essential step is gone; it is meant for editing gene annotations, where the network should not change as a result. +### Checked literature and convention values -Measured on e_coli_core from the cobrapy distribution (bounds as shipped), Gurobi 13.0.1, raven-toolbox commit `94f39407` and RAVEN commit `2d0630e3`, run on 2026-10-30. +Both implementations have the same value or behaviour for each of these parameters, and none was changed. The third column gives what supports the value; only the three `fseof` rows record a sweep. -## Checked literature and convention values +| Parameter | Value | Basis | +|---|---|---| +| `fseof.n_steps` | `10` | Value of [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10); 5 and 20 gave the same target set on iJO1366 | +| `fseof.max_fraction` | `0.9` | [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10); values below 0.9 admitted spurious targets on iJO1366, and 0.99 found no further targets | +| `fseof.correlation_threshold` | `0.9` | [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10); 0.7 added four spurious amplify targets on iJO1366 | +| `connect_blocked_reactions.threshold` | `1.0` | Correct for models in the RAVEN convention, where exchange reactions have bounds of ±1000; RAVEN's `fillGaps` uses a fixed threshold of 1 | +| `connect_blocked_reactions.allow_net_production` | `False` | Keeps the steady-state balance `S @ v == 0`, the stricter test; `True` relaxes it to `S @ v >= 0` | +| `fill_gaps_fast_lp.epsilon` | `1e-4` | Value of the fastGapFill paper, [Thiele et al. 2014](https://doi.org/10.1093/bioinformatics/btu321) | +| `fill_gaps_milp.big_m` | `1000.0` | Matches the ±1000 bounds of the RAVEN convention | +| `check_tasks.close_boundaries` | `False` | Matches RAVEN's `checkTasks`, against which published task lists were curated; `True` closes the exchange reactions of the model so that the task defines the whole boundary | -These values were swept and confirmed, and none was changed. Both implementations agree on each, so there is nothing to reconcile; the value of the check is that each number rests on a measurement. +The three `fseof` rows date from 2026-06-20 and were not repeated with commit `94f39407`. -| Parameter | Value | What the sweep showed | -|---|---|---| -| `fseof.n_steps` | `10` | Choi et al. 2010's value; 5 and 20 give the same target set on iJO1366 | -| `fseof.max_fraction` | `0.9` | Choi et al. 2010; below 0.9 admits spurious targets, 0.99 finds nothing extra | -| `fseof.correlation_threshold` | `0.9` | Choi et al. 2010; 0.7 adds four spurious amplification targets | -| `connect_blocked_reactions.eps` | `1.0` | Correct for RAVEN-convention models, where exchanges carry ±1000 bounds | -| `connect_blocked_reactions.allow_net_production` | `False` | The stricter question, and the one the function exists to ask | -| `fill_gaps_fast_lp.epsilon` | `1e-4` | The fastGapFill paper's value | -| `fill_gaps_milp.big_m` | `1000.0` | Matches RAVEN-convention bounds | -| `check_tasks.close_boundaries` | `False` | Matches RAVEN's `checkTasks`, which published task lists were curated against; `True` closes the model's own exchanges so that the task is the whole boundary | - -The `fseof.n_steps`, `fseof.max_fraction` and `fseof.correlation_threshold` rows were measured on 2026-06-20 and were not rerun for this page. +## `flux_eps` in `fseof` + +`flux_eps` is the floor below which `fseof` treats the theoretical maximum of the target flux as zero; the scan then raises an error. The classification of targets does not read `flux_eps`, and RAVEN's `FSEOF` has no such parameter. On iJO1366 with target `EX_succ_e`, the three values gave 18 amplify targets and 408 knockdown and knockout targets each. From the code, two values of `flux_eps` give different results only when the theoretical maximum of the target lies between them: `fseof` then raises at the larger value and runs at the smaller. + +## Reactions without an enzyme in `remove_genes` + +`blocked_reactions` in `remove_genes`, and `removeBlockedRxns` in RAVEN's `removeGenes`, decide what happens to a reaction whose gene rule is emptied by a deletion. By default both keep such a reaction with bounds of zero, and the two defaults gave the same model. After deletion of `b1779`, that model predicts no growth, because GAPD stays in the network with no capacity to carry flux. `"remove"` and `removeBlockedRxns` true also predicted no growth; they delete GAPD, which changes the number of reactions and the set of reaction identifiers. `"keep"` leaves GAPD with an empty gene rule and its original bounds, so the model predicted the growth of the unmodified model, 0.874 h⁻¹, although the only gene of GAPD was deleted. GAPD is essential for aerobic growth on glucose in e_coli_core, so the growth predicted with `"keep"` is a false positive. The docstring of `remove_genes` states no use for `"keep"`; a use suggested for it is editing gene annotations without changing the network (unverified, no source in the code). + +## Choice of defaults + +The measurement gives no reason to change `flux_eps` from `1e-6`. `blocked_reactions="constrain"` stays the default of `remove_genes`; it gives the model that RAVEN's `removeGenes` gives by default. `"remove"` and `"keep"` are available on request. + +## Limits + +Each benchmark used one model: iJO1366 with one target for `fseof`, and e_coli_core with one deleted gene for `remove_genes`. Criterion 1 of the [evaluation methodology](index.md#evaluation-methodology) asks for a large and a small model. The solver of the `fseof` runs and the number of runs per value are not recorded, and the criterion for a spurious target in the 2026-06-20 sweeps is not recorded. The rows of the checked values without a sweep rest on the literature or on a convention. + +raven-toolbox contains no script for these benchmarks. The unit tests `tests/test_analysis_fseof.py` and `tests/test_manipulation_remove.py` of raven-toolbox check, on small test networks, the classification thresholds of `fseof` and the three settings of `blocked_reactions`. diff --git a/docs/parameter-tuning/index.md b/docs/parameter-tuning/index.md index 64b57e0..e066345 100644 --- a/docs/parameter-tuning/index.md +++ b/docs/parameter-tuning/index.md @@ -1,21 +1,21 @@ # Parameter tuning -RAVEN, both the MATLAB toolbox and raven-toolbox (Python), ships many functions with numerical defaults: solver tolerances, cut-offs, iteration limits, literature constants. [Tuned parameter defaults](../tuned-parameters.md) gives the short version of every one of them: current value, one-line reason, grouped by capability. The detail behind that page comprises the methodology for evaluating a default, the full write-up for every parameter that had a dedicated measurement campaign, and the quicker per-function working notes (**benchmarks**) produced along the way. +RAVEN, both the MATLAB toolbox and raven-toolbox (Python), has many functions with numerical defaults: solver tolerances, cut-offs, iteration limits and literature constants. [Tuned parameter defaults](../tuned-parameters.md) lists each of them with its current value and a one-line reason, grouped by capability. The detail behind that list comprises the method for evaluating a default, a study for each parameter with a dedicated measurement campaign, and shorter per-function records (**benchmarks**) for parameters that needed no study of their own. ## Evaluation methodology -A default value is well-chosen when a user who does not read the docstring gets a correct and useful result for the most common case. +A default is well chosen when a user who does not read the docstring gets a correct result for the most common case. -**MATLAB and Python parity.** The defaults of both implementations were not systematically validated from the start. MATLAB RAVEN's defaults were often chosen by trial and error, copied from earlier tools, or never reconsidered, and raven-toolbox inherited a mix of ported MATLAB values and upstream (cobrapy) conventions. An existing default in either toolbox is a prior: it reflects years of practical use or a well-tested upstream library, and it is not necessarily correct. Where the two implementations differ, the response is to run both and measure; the value that came first is not assumed to be correct. +The defaults of the two implementations were not validated systematically when they were set. raven-toolbox took over a mix of ported MATLAB values and upstream (cobrapy) conventions. An existing default in either implementation is a starting point for the evaluation and is not assumed to be correct. Where the two implementations differ, both values are measured; the value that came first has no precedence. -The following criteria apply in rough priority order: +The criteria apply in this order of priority: -1. **Empirical correctness on real models.** Run the function with the candidate default on at least one large model (Yeast9, Human-GEM) and one small model (iJO1366 or similar). Compare the result against a known-good answer (a published reconstruction, a literature flux distribution, a validated gene-essentiality set). The default must produce a result that is meaningfully better than the alternatives tested, or at least no worse. -2. **Sensitivity envelope.** Vary the parameter by ±1 order of magnitude (or ±50 % for non-log-scale values) and measure the change in the result. If the output is insensitive across the range, the exact default value matters little; document that. If the output is highly sensitive, the default must lie in a plateau region (neither too loose nor too tight) and must be documented with the sensitivity profile. -3. **Literature anchor.** Algorithm-specific numerical parameters should match the value used in the original paper or the most-cited open-source implementation (cobrapy, COBRA Toolbox). Agreement with the literature is corroborating evidence and is not authority. -4. **Cross-implementation check.** Where a raven-toolbox function ports a MATLAB RAVEN function (or vice versa), note what the other side uses. A difference is to be investigated empirically and is not automatically a bug in either implementation. -5. **User expectation alignment.** Prefer values that match what a competent user would supply without thinking (for example `verbose=True` for a long-running MILP, `sort_ids=False` for round-trip-safe export). -6. **No undocumented `None` behaviour.** `None` or empty defaults are acceptable for optional features, but any resulting change to algorithmic behaviour must be documented in the fallback. +1. Empirical correctness on real models. The function runs with each candidate default on at least one large model (yeast-GEM or Human-GEM) and one small model (iJO1366 or similar). The result is compared with a known answer: a published reconstruction, a flux distribution from the literature or a validated gene-essentiality set. The chosen default gives a result that is better than the other candidates or no worse. +2. Sensitivity envelope. The parameter is varied by ±1 order of magnitude, or by ±50 % for a value on a linear scale, and the change in the result is measured. When the result does not change across the range, the exact value matters little, and the documentation says so. When the result changes steeply, the default lies on a plateau between too loose and too tight, and the documentation gives the sensitivity profile. +3. Literature anchor. A numerical parameter of an algorithm matches the value in the original paper or in the most-cited open-source implementation (cobrapy, COBRA Toolbox). Agreement with the literature corroborates a value and does not decide it. +4. Cross-implementation check. Where a raven-toolbox function ports a MATLAB RAVEN function, or the reverse, the documentation states the value of the other implementation. A difference is measured and is not assumed to be a bug in either implementation. +5. Alignment with user expectation. A default matches the value a user would supply without thinking about it, for example `verbose=True` for a long-running mixed-integer linear program (MILP) and `sort_ids=False` for an export that reads back unchanged. +6. Documented `None` behaviour. A default of `None` or empty is acceptable for an optional feature, and any change of algorithmic behaviour that follows from it is documented with the fallback. ### Evaluation workflow per parameter @@ -35,28 +35,35 @@ The following criteria apply in rough priority order: that supports it, and update the default and its docstring. ``` +Yeast9 in the checklist is version 9 of yeast-GEM. + ## Homology and KEGG reconstruction -| Study | Covers | +| Study | Parameters and measurement | |---|---| -| [Homology cut-off calibration](studies/homology-cutoff-calibration.md) | `min_align_len`, `min_identity`, `max_evalue` in homology-based reconstruction, measured against independent KEGG and OMA ortholog references across a 4-organism relatedness series | -| [KEGG HMM cut-off calibration](studies/kegg-hmm-cutoff-calibration.md) | `cutoff`, `min_score_ratio_ko`, `min_score_ratio_g` in KEGG HMM-based reconstruction, measured against real KEGG gene→KO annotations across 4 organisms | +| [Homology cut-off calibration](studies/homology-cutoff-calibration.md) | `min_align_len`, `min_identity` and `max_evalue` of homology-based reconstruction, measured against an independent KEGG ortholog reference for 4 organisms of increasing distance; `min_identity` was also scored against an OMA reference | +| [KEGG HMM cut-off calibration](studies/kegg-hmm-cutoff-calibration.md) | `cutoff`, `min_score_ratio_ko` and `min_score_ratio_g` of KEGG reconstruction with hidden Markov models (HMMs), measured against the KEGG Orthology (KO) annotations of the genes of 4 organisms | ## Context-specific extraction (ftINIT) -| Study | Covers | +| Study | Parameters and measurement | |---|---| -| [ftINIT parameter calibration](studies/init-param-calibration.md) | HCT116 on Human-GEM: `big_m`, `force_on`, stoichiometric scaling, and `mip_gap` in a single step (a named series ignores it) | -| [ftINIT reproducibility](studies/ftinit-determinism.md) | Reproducibility gained with `resolve_ties` and `prove_abs_gap` and the run time they cost on genome-scale Human-GEM | -| [Human-GEM validation vs MATLAB RAVEN](studies/humangem-validation.md) | raven-toolbox's ftINIT output validated against MATLAB RAVEN on Human-GEM across 5 cell lines (Jaccard 0.964–0.988) | +| [ftINIT parameter calibration and input robustness](studies/init-param-calibration.md) | `big_m`, `force_on`, stoichiometric scaling and `mip_gap` on Human-GEM with HCT116 expression; `mip_gap` acts on a single step, and a named series such as `1+1` ignores it | +| [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](studies/ftinit-determinism.md) | Reproducibility of the extracted model with `resolve_ties` and `prove_abs_gap`, and the run time they add on Human-GEM | +| [Human-GEM cell-type model validation against RAVEN](studies/humangem-validation.md) | ftINIT models of raven-toolbox compared with those of MATLAB RAVEN on Human-GEM for 5 cell lines (Jaccard index 0.964–0.988) | +| [ftINIT on different solvers](studies/init-solver-benchmark.md) | Benchmark of the completion, run time and extracted reactions of ftINIT on Human-GEM with each solver interface | -## Benchmarks +## Flux sampling -Per-function working notes are quicker parameter-by-parameter records produced alongside (and sometimes instead of) a full study, together with the master cross-toolbox to-do list. +| Study | Parameters and measurement | +|---|---| +| [Flux sampling convergence and settings](studies/sampling-convergence-calibration.md) | `thinning`, `n_samples` and convergence of artificially centered hit-and-run (ACHR) and coordinate hit-and-run with rounding (CHRR) sampling on e_coli_core and yeast-GEM, by autocorrelation and R-hat; two random-objective settings compared with RAVEN MATLAB | + +## Benchmarks -| Benchmark | Function(s) | +| Benchmark | Functions | |---|---| -| [Parameter benchmarks](benchmarks.md) | `fseof`, `remove_genes`, `connect_blocked_reactions`, `fill_gaps_*`, `check_tasks`, `find_task_essential_reactions` | +| [Benchmarks of `fseof`, `remove_genes` and other defaults](benchmarks.md) | `fseof`, `remove_genes`, `connect_blocked_reactions`, `fill_gaps_*`, `check_tasks` | ```{toctree} :hidden: diff --git a/docs/parameter-tuning/studies/ftinit-determinism.md b/docs/parameter-tuning/studies/ftinit-determinism.md index ddf3668..6d1de2c 100644 --- a/docs/parameter-tuning/studies/ftinit-determinism.md +++ b/docs/parameter-tuning/studies/ftinit-determinism.md @@ -1,94 +1,93 @@ # ftINIT reproducibility with `resolve_ties` and `prove_abs_gap` -This study measures how reproducible ftINIT is, what the opt-in raven-toolbox parameters `resolve_ties` and `prove_abs_gap` fix, and what they do not fix. The API documentation is in `raven_toolbox.init.ftinit`. +This study builds Human-GEM models for one cell line with raven-toolbox's `ftinit` under three settings (default, `resolve_ties=True`, and `resolve_ties=True` with `prove_abs_gap`), varies the solver seed and the template, and records how many reactions and predicted essential genes change between builds and how long each build takes. The API documentation is in `raven_toolbox.init.ftinit`. -Two properties are distinguished: +Two properties are measured. **Determinism** is that the same input, run again or with another seed, machine or solver build, gives the same model. **Stability** is that a similar input (a slightly edited template, or a comparable sample) gives a similar model. Both parameters are opt-in (`resolve_ties=False` and `prove_abs_gap=None` by default) and target determinism. In the measurements, the seed-to-seed difference fell from 110 reactions and 17 essential genes (default) to 41 reactions and 1 gene with `resolve_ties`, and to 1 reaction and 0 genes with `prove_abs_gap=1.0` added, at about 3.6 and 26 times the default build time. Neither parameter made the model stable under a change of the input: a small template edit moved a re-extraction by more reactions than a change of seed (156 against 110 at baseline, 59 against 41 with `resolve_ties`, 16 against 1 with `prove_abs_gap`) and by a similar number of essential genes (17 against 17, 1 against 1, 1 against 0). raven-toolbox has no parameter that anchors a re-extraction to a prior build. -* **Determinism**: same input, a different run, machine or solver build gives the same model. -* **Stability**: similar input (a curated template, a different but comparable sample) gives a similar model. +## Main results -`resolve_ties` and `prove_abs_gap` target determinism. Neither makes the model stable under a change of the input: a small template edit moved a re-extraction by more reactions than a change of seed (156 against 110 at baseline, 59 against 41 with `resolve_ties`, 16 against 1 with `prove_abs_gap`) and by a similar number of essential genes (17 against 17, 1 against 1, 1 against 0). raven-toolbox has no parameter that anchors a re-extraction to a prior build. +Each build in the next two tables is the full pipeline, including the task gap-fill, on Human-GEM and the DLD1 cell line (set-up in the next section). The seed-to-seed difference compares the build at seed 1234 with the build at seed 7. The repeat compares two builds at seed 1234 with nothing changed between them; it was run for two settings. For the baseline, the first build of the repeat is the seed 1234 build of the table (237 s) and the second took 231 s. For `resolve_ties`, the repeat consists of two further builds (865 s and 861 s), separate from the seed 1234 build of the table (864 s). -## Degeneracy of the extraction MILP +| Setting | Seed-to-seed difference, reactions | Seed-to-seed difference, essential genes | Essential genes (seed 1234 / seed 7) | Repeat difference at seed 1234, reactions / genes | Build time (seed 1234 / seed 7) | +|---|---:|---:|---:|---:|---:| +| baseline | 110 | 17 | 125 / 112 | 2 / 0 | 237 s / 228 s | +| `resolve_ties=True` | 41 | 1 | 114 / 113 | 0 / 0 | 864 s / 812 s | +| `resolve_ties=True, prove_abs_gap=1.0` | 1 | 0 | 125 / 125 | not run | 5742 s / 6354 s | +| `resolve_ties=True, prove_abs_gap=0.05` | 1 | 0 | 125 / 125 | not run | 5793 s / 6589 s | -The extraction MILP is degenerate. In the first step on Human-GEM/DLD1, 1547 merged reactions have a negative score (the true 0/1 keep-or-drop binaries), and 98.3 % of them (1520) are in blocks of equal score. The largest block has 840 reactions at −5.0; the second, 420 reactions at −2.0 (the default score of a reaction without gene association), has 419 to 421 members across the five Hart2015 cell lines, so the choice of expression data changes its size by at most 2 members. Many reaction subsets reach the same score optimum and the solver returns an arbitrary one. +The two `prove_abs_gap` values gave the same result. The build time grew from about 4 minutes (baseline) to about 14 minutes with `resolve_ties` (3.6 times) and about 100 minutes with `prove_abs_gap` added (26 times). -`Threads=1` and a fixed `Seed` make that choice close to reproducible, not unique: two baseline builds at one seed differ by 2 reactions, because runs that stop at a wall-clock limit (the 5 s run of the last step) can end at a different incumbent depending on machine load. A different Gurobi version, platform or seed selects a different member of the tied set, and the gene-essentiality predictions then change for reasons unrelated to the model or the data. +The two baseline builds at one seed differed by 2 reactions. Each baseline build had one run of the final step that stopped at its 5 s wall-clock limit; a run stopped at a time limit can return a different solution depending on machine load, and this is the presumed source of the 2 reactions (not tested). The two `resolve_ties` builds of the repeat had identical reaction sets. -Both parameters act on this tie directly: +The template edit removed 30 reactions that the reference builds did not use. They were drawn at random (seed 0) from the 2387 reactions of the prepared reference model that the build with `resolve_ties=True, prove_abs_gap=1.0` (seed 1234, 7754 reactions, 125 essential genes) did not keep and that are not task-essential. Of the 30, the `resolve_ties` builds kept 0 and the baseline build kept 1. Each setting was re-extracted from the edited template at seed 1234 and compared with its build from the unedited template: -* `resolve_ties`: a lexicographic phase 2 after the primary solve of each extraction step, with the score objective held at its optimum. The phases are, in order: - * `phase2a-parsimony`: minimise the count of kept removable (negative-score) reactions. - * `phase2b-idrank`: among those, minimise the summed id rank of the kept reactions. A merged reaction is ranked by the smallest original id in its group (`rank_keys`), so the rank does not depend on which member names the merged reaction. - * `phase2c-minflux`: for that on set, minimise the total absolute flux (gap 1e-6), so the fluxes that the next step reads for its forced directions and magnitudes do not depend on the solver's vertex. - * `phase2d-direction`: among the minimal-flux solutions, minimise the id-rank-weighted backward flux of the reactions whose flux can take either sign (gap 1e-6), so a flux that carries the same total either way gets a fixed direction. +| Setting | Reaction difference | Essential-gene difference | Essential genes (unedited → edited) | Build time (edited) | +|---|---:|---:|---:|---:| +| baseline | 156 | 17 (2 gains) | 125 → 112 | 194 s | +| `resolve_ties=True` | 59 | 1 | 114 → 115 | 638 s | +| `resolve_ties=True, prove_abs_gap=1.0` | 16 | 1 | 125 → 126 | 4108 s | - The last step skips `phase2c-minflux` and `phase2d-direction`, because no later step reads its fluxes. The task gap-fill uses the first two phases. Phases 2a and 2b get half of `time_limit` each; phases 2c and 2d get the full `time_limit`. -* `prove_abs_gap`: proves each step to a fixed absolute MIP gap in a single solve, replacing RAVEN's relative-gap escalation, whose final near-zero-objective run otherwise accepts an arbitrary within-gap incumbent. +The removed reactions were almost unused, so a re-extraction that followed only the edit would change next to nothing downstream. The baseline moved 156 reactions and 17 essential-gene calls; the two settings with `resolve_ties=True` moved 59 and 16 reactions and 1 gene each. -Both default to off (`resolve_ties=False`, `prove_abs_gap=None`). +## Set-up -A phase that does not prove its optimum has these outcomes: +* Model and data: Human-GEM `v2.0.0-38`, Hart2015 DLD1 cell line, 57 essential tasks, series `1+1`. The `model/Human-GEM.yml` of that commit has 12877 reactions and 2848 genes. The preparation is the one of `estimateEssentialGenes.py` at that commit (removal of drug, amino-acid-triplet and duplicate complex I reactions), followed by raven-toolbox's own `prep_init_model`, which gives a reference model of 10141 reactions and a merged model of 6861 reactions (chains of linked reactions combined into one), with 206 task-essential reactions. This preparation differs from RAVEN's exported `raven_refModel.xml`, so absolute counts are not comparable to the [Human-GEM cell-type model validation against RAVEN](humangem-validation.md) run, which uses that export. +* Steps: series `1+1` runs two extraction steps. Step 1 decides the reactions with a gene association; step 2, the last step, adds the reactions without one and holds the reactions of step 1 on. +* Settings: `big_m=100` (the default is 1000), `mip_gap_abs=10` and `time_limit=1800` s, the values of `estimateEssentialGenes.py` at that commit. A named series such as `1+1` ignores `mip_gap` and `mip_gap_abs`; they apply to a step without a run schedule. +* Escalation: each solve stops at a mixed-integer programming (MIP) gap, the distance between the objective of the best solution found and the solver's bound on the optimum, either relative (divided by the objective) or absolute. Without `prove_abs_gap`, each step follows RAVEN's run schedule, here called the **escalation**. The first run uses a relative gap of 0.0004; each later run uses the larger of its relative gap and its absolute gap (10 or 20) divided by the objective, and the step stops as soon as the gap already reached meets the next target. `time_limit` caps the limit of each run: step 1 has runs of 120 s and 1800 s, the final step runs of 5 s, 120 s and 1800 s. +* With `prove_abs_gap`, each step is one solve proven to that absolute gap, with the 1800 s limit for the primary solve and for the two flux phases of `resolve_ties` (phases 2c and 2d), and half of it (`_TIE_BREAK_TIME_FRACTION = 0.5`) for its two on/off phases (phases 2a and 2b); the phases are defined in the section on the parameters. The task gap-fill uses a fixed 300 s per task in every setting. +* Solver: Gurobi 13.0.1, `Threads=1` unless stated, raven-toolbox at commit `94f39407`. The builds ran as Slurm jobs on AMD EPYC 9354 nodes shared with other jobs, so wall-clock figures are approximate. Most jobs had 2 CPUs and the `Threads=4` jobs of the first-step probe had 4. Several builds ran in sequence in one job (the seeds of a setting, or the three builds of the stability test). +* Determinism probe: the extraction `seed` was varied (1234, 7, and further seeds where stated) with inputs, code, solver build and machine held fixed, so a difference between builds comes from the solver's choice among equal optima. Differences between Gurobi versions were not measured. +* Metrics: the reaction difference is the symmetric difference of the kept-reaction sets (reactions kept by one build and not the other). The essential-gene difference is the symmetric difference of the sets of genes whose single deletion is predicted essential by `cobra.flux_analysis.find_essential_genes` at its default threshold. The gene difference is the result a user sees; the reaction difference is what the parameters act on. +* Debug log: all builds ran with `FTINIT_DEBUG=1`, which logs the status of each solve and of each tie-break phase. An **incumbent** is the best solution the solver holds when it stops; at a time limit it is not proven optimal. -* A phase that stops at `time_limit` and holds an incumbent is adopted, and a `UserWarning` names it (an unproven incumbent): `phase2a-parsimony` and `phase2b-idrank` in one warning that they share, `phase2c-minflux` and `phase2d-direction` each in their own. The shared warning is raised when `phase2b-idrank` ends with an incumbent, and not otherwise. -* A flux phase (`phase2c-minflux`, `phase2d-direction`) that finds no solution warns with its name and keeps the fluxes of the phase before it. -* A phase `phase2a-parsimony` or `phase2b-idrank` that finds no incumbent drops the on/off tie-break of that solve without a warning: the primary solution is kept and the ties of that step are not resolved. The absence of a warning therefore does not mean that the tie-break completed. -* When the primary solve of a step stops at `time_limit`, all tie-break phases of that solve are skipped and the primary solution is kept. -* The baseline escalation (`resolve_ties=False`, `prove_abs_gap=None`) raises no warning either, and its step 1 objective ends 4.0 to 12.0 below the optimum. - -For comparison, RAVEN's `ftINIT` with `resolveTies` skips the tie-break after a primary solve that stopped at the time limit, gives the count and id-rank phases half of `params.TimeLimit` each, skips the minimal-flux and direction phases in the last step, and names every unproven phase (`phase2a-parsimony`, `phase2b-idrank`, `phase2c-minflux`, `phase2d-direction`) in its "tie resolution did not converge" warning (`ftINITInternalAlg.m`). +## Degeneracy of the extraction MILP -## Method +The extraction mixed-integer linear program (MILP) has many optima of equal score. In the first step on Human-GEM and DLD1, 1547 merged reactions have a negative score, and these carry the 0/1 keep-or-drop variables. Of them, 98.3 % (1520) are in blocks of equal score. The largest block has 840 reactions at −5.0. The second has 420 reactions at −2.0, the default score of a reaction without gene association; across the five Hart2015 cell lines it has 419 to 421 members, so the expression data change its size by at most 2 members. Many reaction subsets reach the same optimum, and the solver returns one of them. -* Model and data: Human-GEM `v2.0.0-38`, Hart2015 DLD1 cell line, 57 essential tasks, series `1+1`. The `model/Human-GEM.yml` of that commit has 12877 reactions and 2848 genes. The preparation is the one of `estimateEssentialGenes.py` at that commit (removal of drug, amino-acid-triplet and duplicate complex I reactions), followed by raven-toolbox's own `prep_init_model`: a reference model of 10141 reactions and a merged model of 6861 reactions, with 206 task-essential reactions. This is not RAVEN's exported `raven_refModel.xml`, so absolute counts are not comparable to the [Human-GEM validation](humangem-validation.md) run, which uses that export. -* Settings: `big_m=100`, `mip_gap_abs=10`, `time_limit=1800` (the values of `estimateEssentialGenes.py` at that commit). Without `prove_abs_gap`, each step runs RAVEN's escalation schedule, whose runs carry their own limits (step 1: 120 s for the first run; final step: 5 s, 120 s, then 1800 s). With `prove_abs_gap`, each primary solve and phases 2c and 2d use the 1800 s limit, phases 2a and 2b get half of it (`_TIE_BREAK_TIME_FRACTION = 0.5`), and the task gap-fill uses a fixed 300 s per task. -* Solver: Gurobi 13.0.1, `Threads=1` unless stated, raven-toolbox at commit `94f39407`. The builds ran as Slurm jobs on AMD EPYC 9354 nodes shared with other jobs, so wall-clock figures are approximate. Most jobs had 2 CPUs and the `Threads=4` jobs of the first-step probe had 4; several builds ran in sequence in one job (the seeds of a setting, or the three builds of the stability test). -* Determinism probe: the extraction `seed` is varied (1234, 7, and further seeds where noted) with everything else (inputs, code, solver build, machine) held fixed, so any difference between builds is the solver's tie-break among equal optima. Cross-version drift is not measured; see Limitations. -* Metrics: reaction seed-swing (symmetric difference of kept-reaction sets) and essential-gene seed-swing (symmetric difference of single-gene-deletion essential sets, via `cobra.flux_analysis.find_essential_genes` with its default threshold). The second is what matters for users; the first is what the parameters act on directly. -* Debug log: all builds ran with `FTINIT_DEBUG=1`, which logs the status of each solve and of each tie-break phase. +`Threads=1` and a fixed `Seed` make that choice close to reproducible on one machine: the two baseline builds at one seed differed by 2 reactions. Another seed selects another member of the tied set, and the predicted essential genes change for reasons unrelated to the model or the data. Another Gurobi version or platform is expected to act the same way; that was not measured. -## Results +## Parameters `resolve_ties` and `prove_abs_gap` -### Full pipeline at seeds 1234 and 7 +`resolve_ties` adds a lexicographic phase 2 after the primary solve of each extraction step, with the score objective held at its optimum. The phases are, in order: -Each build is the full pipeline with task gap-fill, one build per seed and setting. +* `phase2a-parsimony`: minimise the count of kept removable (negative-score) reactions. +* `phase2b-idrank`: among those, minimise the summed id rank of the kept reactions. A merged reaction is ranked by the smallest original id in its group (`rank_keys`), so the rank does not depend on which member names the merged reaction. +* `phase2c-minflux`: for that set of kept reactions, minimise the total absolute flux (gap 1e-6), so the fluxes that the next step reads for its forced directions and magnitudes do not depend on the solver's vertex. +* `phase2d-direction`: among the minimal-flux solutions, minimise the id-rank-weighted backward flux of the reactions whose flux can take either sign (gap 1e-6), so a flux that carries the same total either way gets a fixed direction. -| Config | Reaction swing | Gene swing | Essential genes | Build | -|---|---:|---:|---:|---:| -| baseline | 110 | 17 | 125 / 112 | 237 s / 228 s | -| `resolve_ties=True` | 41 | 1 | 114 / 113 | 864 s / 812 s | -| `resolve_ties=True, prove_abs_gap=1.0` | 1 | 0 | 125 / 125 | 5742 s / 6354 s | -| `resolve_ties=True, prove_abs_gap=0.05` | 1 | 0 | 125 / 125 | 5793 s / 6589 s | +The last step skips `phase2c-minflux` and `phase2d-direction`, because no later step reads its fluxes. The task gap-fill uses the first two phases. Phases 2a and 2b get half of `time_limit` each; phases 2c and 2d get the full `time_limit`. -The gene-level swing falls from 17 to 1 with `resolve_ties` and to 0 with `prove_abs_gap` added; the reaction-level swing falls from 110 to 41 to 1. The essential-gene counts of the two seeds are equal in the last two rows, and the two `prove_abs_gap` values give the same result. The runtime grows from about 4 minutes (baseline) to about 14 minutes (`resolve_ties`, 3.6 times) and about 100 minutes (`resolve_ties` with `prove_abs_gap`, 26 times). +`prove_abs_gap` proves each step to a fixed absolute MIP gap in a single solve. It replaces the escalation, whose final run, at an objective near zero, accepts any incumbent within its gap. -Repeated builds at a fixed seed (seed 1234, two builds per setting, nothing changed between them): +A phase that does not prove its optimum has these outcomes: -| Config | Reaction difference | Gene difference | Build | -|---|---:|---:|---:| -| baseline | 2 | 0 | 237 s / 231 s | -| `resolve_ties=True` | 0 | 0 | 865 s / 861 s | +* A phase that stops at `time_limit` with an incumbent is adopted, and a `UserWarning` names it as an unproven incumbent. `phase2a-parsimony` and `phase2b-idrank` share one warning, which is raised when `phase2b-idrank` ends with an incumbent and not otherwise; `phase2c-minflux` and `phase2d-direction` each have their own. +* A flux phase (`phase2c-minflux`, `phase2d-direction`) that finds no solution warns with its name and keeps the fluxes of the phase before it. +* When `phase2a-parsimony` or `phase2b-idrank` finds no incumbent, the on/off tie-break of that solve is dropped without a warning: the primary solution is kept and the ties of that step are not resolved. A build without a warning can therefore hold an unresolved step. +* When the primary solve of a step stops at `time_limit`, all tie-break phases of that solve are skipped and the primary solution is kept. +* The baseline escalation raises no warning either, and in this study its step 1 objective ended 4.0 to 12.0 below the optimum. -The baseline moves 2 reactions between two runs with nothing changed; the `resolve_ties` builds have identical reaction sets. +For comparison, RAVEN's `ftINIT` with `resolveTies` skips the tie-break after a primary solve that stopped at the time limit, gives the count and id-rank phases half of `params.TimeLimit` each, skips the minimal-flux and direction phases in the last step, and names every unproven phase (`phase2a-parsimony`, `phase2b-idrank`, `phase2c-minflux`, `phase2d-direction`) in its "tie resolution did not converge" warning (`ftINITInternalAlg.m`). RAVEN's `ftINIT` was not run in this study. -### Location of the degeneracy +## Location of the degeneracy -Six seeds through the first step alone (a one-step series, so the flux phases of `resolve_ties` do not run), with and without `resolve_ties`, and at `Threads` 1 and 4: +Six seeds were run through the first step alone (a one-step series, so the flux phases of `resolve_ties` do not run), with and without `resolve_ties`, at `Threads` 1 and 4. The mean pairwise difference is the reaction difference averaged over the 15 pairs of seeds. -| Config | Threads | Distinct solutions / 6 | Mean pairwise swing | Time per build | +| Setting | Threads | Distinct solutions / 6 | Mean pairwise difference, reactions | Time per build | |---|---:|---:|---:|---:| | baseline | 1 | 6 | 48.7 | 54 to 56 s | | baseline | 4 | 6 | 48.3 | 53 to 56 s | | `resolve_ties=True` | 1 | 4 | 7.4 | 71 to 124 s | | `resolve_ties=True` | 4 | 3 | 4.8 | 67 to 95 s | -At the same seed, the baseline solutions at `Threads` 1 and 4 are 5 to 20 reactions apart, and in 5 of the 6 seeds the two runs reach different objective values. The number of distinct solutions and the mean pairwise swing are similar at both thread counts (6 and 6 distinct, 48.7 and 48.3), so raising `Threads` did not remove the degeneracy in these runs. +At the same seed, the baseline solutions at `Threads` 1 and 4 were 5 to 20 reactions apart, and in 5 of the 6 seeds the two runs reached different objective values. The number of distinct solutions and the mean pairwise difference were similar at both thread counts (6 and 6 distinct, 48.7 and 48.3), so raising `Threads` did not remove the degeneracy in these runs. -Conditioning on the objective separates the tie-break from the primary solve. The 12 baseline runs (both thread settings) reached six objective values between 45887.113 and 45899.113. At a fixed objective value, the baseline returned 4 distinct solutions from 4 runs at 45892.113 (pairwise differences 6 to 60 reactions) and 4 from 4 at 45894.113 (35 to 53). With `resolve_ties`, the 4 runs at each of these two objective values returned one solution each. The 12 `resolve_ties` runs gave 5 distinct reaction sets, and the residual variation came from the primary solve reaching different objective values, not from the tie-break. +Grouping the runs by objective value separates the tie-break from the primary solve. The 12 baseline runs (both thread settings) reached six objective values between 45887.113 and 45899.113. At a fixed objective value, the baseline returned 4 distinct solutions from 4 runs at 45892.113 (pairwise differences 6 to 60 reactions) and 4 from 4 at 45894.113 (35 to 53). With `resolve_ties`, the 4 runs at each of these two objective values returned one solution each. The 12 `resolve_ties` runs gave 5 distinct reaction sets, and the remaining differences came from runs whose primary solve reached different objective values. -### Effect of `prove_abs_gap` +## Effect of the `prove_abs_gap` value -Sweeping the value with `resolve_ties=True` and the full two-step pipeline without gap-fill, four seeds (1234, 7, 1, 2) per value. "`None`" is the escalation with `resolve_ties=True`. The times are those of the whole step, including its tie-break phases. All 48 solves of step 1 and step 2 with a `prove_abs_gap` value ended with status `optimal`. +The value was swept with `resolve_ties=True` on the two-step pipeline without gap-fill, at four seeds (1234, 7, 1, 2) per value. "`None`" is the escalation with `resolve_ties=True`. A step time includes the tie-break phases of that step. All 48 solves of step 1 and step 2 with a `prove_abs_gap` value ended with status `optimal`. | `prove_abs_gap` | Step 1 objectives (4 seeds) | Step 2 objectives (4 seeds) | Step 1 time | Step 2 time | |---|---|---|---:|---:| @@ -100,15 +99,15 @@ Sweeping the value with `resolve_ties=True` and the full two-step pipeline witho | 0.1 | 45899.113 (4) | −613.872 (3), −611.872 | 3313 to 3650 s | 1757 to 2721 s | | 0.05 | 45899.113 (4) | −613.872 (3), −611.872 | 3317 to 3650 s | 1758 to 2734 s | -The escalation returns a step 1 solution 4.0 to 12.0 below the proven optimum 45899.113, without a warning: of its four runs, none reaches the optimum. `prove_abs_gap=5.0` stops 2.0 below it for one seed; every value from 2.0 to 0.05 reaches it for all four seeds. The step 2 objectives depend on the step 1 result, so they are comparable between rows where step 1 is equal: the four seeds with a proven step 1 give −613.872 for three seeds and −611.872 for seed 7 (`phase2c-minflux` and `phase2d-direction` of step 1 reach the limit, which leaves the flux read-out seed-dependent). +None of the four escalation runs reached the proven step 1 optimum of 45899.113; they ended 4.0 to 12.0 below it, without a warning. `prove_abs_gap=5.0` stopped 2.0 below it for one seed, and every value from 2.0 to 0.05 reached it for all four seeds. The step 2 objectives depend on the step 1 result, so they are comparable between rows where step 1 is equal. With a proven step 1, step 2 gave −613.872 for three seeds and −611.872 for seed 7. In step 1 of the `prove_abs_gap=1.0` builds, `phase2c-minflux` stopped at the time limit for 3 of the 4 seeds and was optimal for 1, and `phase2d-direction` stopped at the time limit for all 4, so the fluxes passed to step 2 were unproven incumbents; whether this causes the seed 7 difference was not tested. -All values from 2.0 to 0.05 give the same objectives and the same final reaction sets. Tighter gaps add little run time: step 2 takes up to 2734 s at 0.05 against 2500 s at 1.0, and the mean of a whole build is 5770 s against 5640 s. In step 1, `phase2c-minflux` and `phase2d-direction` run to the limit in most builds (3 of 4 seeds for `phase2c-minflux` and 4 of 4 for `phase2d-direction` at `prove_abs_gap=1.0`); per-phase times were not logged, so the split of step 2 between proof and tie-break is not measured. +All values from 2.0 to 0.05 gave the same objectives and the same final reaction sets. A tighter gap added little run time: step 2 took up to 2734 s at 0.05 against 2500 s at 1.0, and a whole build took 5770 s against 5640 s on average. Per-phase times were not logged, so the split of step 2 between the proof and the tie-break is not known. -Final reaction sets of the sweep (without gap-fill): the escalation gives four distinct sets, 37 to 56 reactions apart; `prove_abs_gap=5.0` gives three; every value from 2.0 to 0.05 gives two, which differ by one reaction (seed 7 against the other three). +The final reaction sets of the sweep (without gap-fill) were: four distinct sets with the escalation, 37 to 56 reactions apart; three with `prove_abs_gap=5.0`; two with every value from 2.0 to 0.05, which differ by one reaction (seed 7 against the other three). -### The limit of `resolve_ties` at genome scale +## Tie-break phases at genome scale -With a proven step 1 objective, the tie-break phases do not all prove at genome scale. The tie-break phase statuses of the eight tie-broken solves (two steps, four seeds) at `prove_abs_gap=1.0` (the other values 0.05 to 5.0 give the same counts): +With a proven step 1 objective, the tie-break phases did not all prove within the time limit. The phase statuses of the eight tie-broken solves (two steps, four seeds) at `prove_abs_gap=1.0` were as follows; the other values from 0.05 to 5.0 gave the same counts. | Phase | Optimal | Stopped at `time_limit` | |---|---:|---:| @@ -130,54 +129,47 @@ Step 2 is the last step, so it skips the last two phases. The log of seed 1234: [ftinit] tie-break phase2b-idrank: status=optimal obj=346311.0 ``` -The on/off tie-break of step 1 is proven in all four seeds, so the step 1 reaction set is fixed. `phase2c-minflux` reaches the 1800 s limit in 3 of the 4 seeds and `phase2d-direction` in all 4; each is adopted as an unproven incumbent. These two phases are the reason step 1 takes about 3650 s and a build about 100 minutes. The flux the next step reads is therefore the incumbent of a time-limited solve, and it differs by seed: the step 2 objective is −611.872 for seed 7 and −613.872 for the other three seeds, and the final sets differ by one reaction. +The on/off tie-break of step 1 was proven for all four seeds, so the step 1 reaction set is fixed. `phase2c-minflux` reached the 1800 s limit for 3 of the 4 seeds and `phase2d-direction` for all 4, and each was adopted as an unproven incumbent. Two flux phases at the 1800 s limit take 3600 s, which accounts for most of the step 1 time of about 3650 s and for a build time of about 100 minutes. The flux that step 2 reads is therefore the incumbent of a time-limited solve. The step 2 objective differed by seed (−611.872 for seed 7, −613.872 for the other three) and the final sets by one reaction. The fluxes themselves were not compared between seeds; a seed-dependent flux is inferred from the step 2 objectives, not measured. -The builds that use the escalation (`prove_abs_gap=None`) with `resolve_ties=True` also adopt phases at a time limit, here the limit of the escalation run. The build of seed 1234 in the full-pipeline table adopted 3 phases (`phase2b-idrank`, `phase2c-minflux`, `phase2d-direction` of step 1) and the build of seed 7 adopted 2 (`phase2c-minflux`, `phase2d-direction` of step 1). In step 2 of both builds `phase2a-parsimony` ended at the time limit with an incumbent and `phase2b-idrank` ended at the time limit with no incumbent (`A-rt-1234.log`, `A-rt-7.log`). `phase2b-idrank` then returns no solution, the tie-break of step 2 returns `False`, the primary solution of step 2 is kept, and no warning is raised. The reaction sets of these two builds therefore rest on a resolved step 1 and an unresolved step 2. +The escalation builds with `resolve_ties=True` also adopted phases at a time limit, here the limit of the escalation run (120 s in step 1, so 60 s for phases 2a and 2b). The full-pipeline build of seed 1234 adopted 3 phases (`phase2b-idrank`, `phase2c-minflux`, `phase2d-direction` of step 1) and the build of seed 7 adopted 2 (`phase2c-minflux`, `phase2d-direction` of step 1). In step 2 of both builds, `phase2a-parsimony` ended at the time limit with an incumbent and `phase2b-idrank` ended at the time limit with no incumbent (`A-rt-1234.log`, `A-rt-7.log`). `phase2b-idrank` then returns no solution, the tie-break of step 2 returns `False`, the primary solution of step 2 is kept, and no warning is raised. The reaction sets of these two builds rest on a resolved step 1 and an unresolved step 2. Counts of full builds (gap-fill included) with at least one unproven tie-break phase or a time-limited primary solve: -| Config | Builds | With an adopted phase at `time_limit` | With a primary solve at `time_limit` | With a tie-break dropped without a warning | +| Setting | Builds | With an adopted phase at `time_limit` | With a primary solve at `time_limit` | With a tie-break dropped without a warning | |---|---:|---:|---:|---:| | baseline | 4 | 0 (no tie-break) | 4 (one run of the final step) | not applicable | | `resolve_ties=True` | 4 | 4 | 4 (tie-break of that solve skipped) | 4 (step 2) | | `resolve_ties=True, prove_abs_gap=1.0` or `0.05` | 4 | 4 | 0 | 0 | -The phase statuses of a build are in the debug log. The warnings cover the adopted phases and the flux phases without a solution, so a build without a warning can still hold an unresolved tie-break (dropped `phase2a-parsimony` or `phase2b-idrank`) or an unproven escalation, and the debug log is the record of what each solve reached. In the sweep, all 24 builds with `prove_abs_gap` raised at least one warning. Raising `time_limit` is the available option; the effect of a larger limit is not measured. - -## Recommendation - -* `resolve_ties=True`: one repeat pair at a fixed seed (seed 1234) has identical reaction sets, and the seed-to-seed gene swing falls from 17 to 1 on this data, at about 3.6 times the runtime (4 to 14 minutes). Its warnings name the adopted phases that stopped at the time limit, but a tie-break that finds no incumbent in phase 2a or 2b is dropped without a warning (all four `resolve_ties=True` builds drop the one of step 2). With the escalation, the flux phases of step 1 run with the 120 s limit of that run. -* `prove_abs_gap=1.0` together with `resolve_ties=True`: the seed-to-seed swing falls to 1 reaction and 0 genes, and the objectives of both steps are proven to 1.0 (the escalation is 4.0 to 12.0 below the optimum in step 1). The build takes about 100 minutes at `time_limit=1800` (26 times the baseline), of which step 1 takes 3300 to 3650 s (59 to 66 % of the 4987 to 6183 s wall clock of the four `prove_abs_gap=1.0` sweep builds, which have step 1 plus step 2 and no gap-fill; the 5742 s and 6354 s of the full-pipeline table are the two builds with gap-fill), mostly in the two flux phases, and step 2 another 1650 to 2500 s. Use it when the swing of `resolve_ties` alone (41 reactions, 1 gene here) is too large and the runtime is acceptable. A value from 2.0 to 0.05 gives the same models; 5.0 can stop 2.0 below the optimum. The data contain no build with `prove_abs_gap` and without `resolve_ties`. -* Pinning the solver stack (raven-toolbox commit and `gurobipy` version) is the zero-cost option for run-to-run identity, and the parameters `resolve_ties` and `prove_abs_gap` do not change it. +The warnings cover the adopted phases and the flux phases without a solution. A build without a warning can hold an unresolved tie-break (a dropped `phase2a-parsimony` or `phase2b-idrank`) or an unproven escalation, and the debug log is the record of what each solve reached. In the sweep, all 24 builds with `prove_abs_gap` raised at least one warning. A larger `time_limit` is the setting that can let the phases prove; its effect was not measured. ## Stability under a small template change -Baseline and `resolve_ties` with seed 1234 on identical inputs differ by 162 reactions and 17 essential-gene calls (125 against 114), with the same step 1 objective (45892.113) and different step 2 objectives (−657.240 for the baseline, −616.906 for `resolve_ties=True`). The step 1 difference is a different pick among equally scored solutions; the step 2 difference includes more than tie selection, because step 2 reads the fluxes of step 1 and ends at a different objective. That is the order of magnitude by which a re-extraction can move even when nothing about the biology changed. +The template-edit result is in the second table of the main results. The two settings with `resolve_ties=True` moved fewer reactions than the baseline, and for each setting the edit moved more reactions than a change of seed. Determinism does not give stability: a deterministic extractor returns the same model for the same input, and the edit changes the input. -Two common workflows are affected: +Settings on identical inputs also differ. Baseline and `resolve_ties` at seed 1234 differ by 162 reactions and 17 essential-gene calls (125 against 114), with the same step 1 objective (45892.113) and different step 2 objectives (−657.240 for the baseline, −616.906 for `resolve_ties=True`). The step 1 difference is a different choice among solutions of equal score. The step 2 difference includes more than that choice, because step 2 reads the fluxes of step 1 and ends at a different objective. -* Curating the template: fix one or a few reactions, re-extract, and compare. The MILP re-solves globally and can move far more than the edit warrants, because the optimum is a discontinuous function of the input: genes flip that have nothing to do with the curation. A deterministic extractor does not remove this effect, because determinism is not continuity. -* Comparable but distinct samples: two RNA-seq samples from the same healthy tissue that differ by ordinary inter-patient variation can independently reselect within the same tied set and produce two models with reaction-level and gene-level differences that resemble biological differences but are not. +The growth objective of the models did not indicate stability on this data: the unedited builds of one setting (`resolve_ties=True`) had 74.6 for seed 1234 and 83.1 for seed 7. -The stability measurement on Human-GEM and DLD1 used an edit that removes reactions the reference builds almost never kept. 30 reactions were drawn at random (seed 0) from the 2387 reactions of the prepared reference model that the build with `resolve_ties=True, prove_abs_gap=1.0` (seed 1234, 7754 reactions, 125 essential genes) did not keep and that are not task-essential, and removed from the template. Of the 30, 0 were kept by the `resolve_ties` builds and 1 by the baseline build. Each setting was re-extracted from the edited template (seed 1234) and compared with the build of the same setting from the unedited template: +For curation of a template (fix one or a few reactions, re-extract, compare), the re-extraction difference contains the effect of the edit together with re-selection among tied solutions. Applying the same edit to the extracted model gives a control: the difference between that edited model and the unedited extraction contains the effect of the edit alone. No result for this control is reported. -| Setting | Reaction difference | Essential-gene drift | Essential genes | Build | -|---|---:|---:|---:|---:| -| baseline | 156 | 17 (2 gains) | 125 → 112 | 194 s | -| `resolve_ties=True` | 59 | 1 | 114 → 115 | 638 s | -| `resolve_ties=True, prove_abs_gap=1.0` | 16 | 1 | 125 → 126 | 4108 s | +## Choice of settings + +* `resolve_ties=True`: the repeat pair at seed 1234 had identical reaction sets, and the seed-to-seed difference fell from 17 genes to 1 on this data, at about 3.6 times the build time (4 to 14 minutes). Its warnings name the adopted phases that stopped at the time limit. A tie-break that finds no incumbent in phase 2a or 2b is dropped without a warning; all four `resolve_ties=True` builds dropped the one of step 2. With the escalation, the flux phases of step 1 run with the 120 s limit of that run. +* `prove_abs_gap=1.0` together with `resolve_ties=True`: the seed-to-seed difference fell to 1 reaction and 0 genes, and the objectives of both steps are proven to 1.0 (the escalation ended 4.0 to 12.0 below the optimum in step 1). The build takes about 100 minutes at `time_limit=1800`, 26 times the baseline. In the four `prove_abs_gap=1.0` sweep builds (step 1 and step 2, no gap-fill; 4987 to 6183 s wall clock), step 1 took 3300 to 3650 s (59 to 66 %), mostly in the two flux phases, and step 2 another 1650 to 2500 s. This setting fits when the difference left by `resolve_ties` alone (41 reactions and 1 gene here) is too large and the build time is acceptable. A value from 2.0 to 0.05 gave the same models; 5.0 stopped 2.0 below the optimum for one seed. No build with `prove_abs_gap` and without `resolve_ties` was run. +* Holding the raven-toolbox commit and the `gurobipy` version fixed removes the difference between solver builds at no cost in run time. `resolve_ties` and `prove_abs_gap` are independent of that choice. -The reference build with `resolve_ties` keeps none of the 30 removed reactions and the baseline reference build keeps 1, so a perfectly stable extractor would show next to no downstream change. The baseline re-extraction moves 156 reactions and 17 essential-gene calls; the tie-broken builds move 59 and 16 reactions and 1 gene each. The growth objective of the models is not a stable indicator on this data: the unedited builds of one setting (`resolve_ties=True`) have 74.6 for seed 1234 and 83.1 for seed 7. +## Limits -Applying the edit to the extracted model as well as to the template gives a control that separates a real effect from re-selection noise. Comparing an extraction of the edited template against the extracted baseline model with the same edit applied directly gives an exact, causal diff before and after for the reactions the edit touches; the re-extraction diff contains in addition the re-selection noise measured above. +The full-pipeline determinism table rests on one seed pair (seeds 1234 and 7), the first-step probe on six seeds and the sweep on four. The study used one cell line, one machine type, one Gurobi build and one `time_limit`, and one edit (30 unused reactions removed) for the stability measurement. Each figure comes from one build per seed and setting, except the repeat pairs at seed 1234, so the data give direction and rough magnitude and no error bars. -## Limitations +Differences between Gurobi versions, the failure that `resolve_ties` and `prove_abs_gap` target, were not measured; the seed probe stands in for them, and how well it represents them is not known. The stability edit touched almost nothing the reference builds used, which is a smaller change than a typical curation. Two comparable samples (for example, two expression profiles of one tissue) were not compared. -One cell line, one seed pair for the full-pipeline determinism table (six seeds for the first-step probe, four seeds for the sweep), one edit (30 unused reactions removed) for the stability measurement, one machine type, one Gurobi build, one `time_limit`. The data support direction and rough magnitude and give no error bars on any number. Cross-Gurobi-version drift (the failure mode that `resolve_ties` and `prove_abs_gap` target) is not measured; the seed probe is the substitute for it and is expected to understate it. The stability measurement used an edit chosen to touch almost nothing the reference builds used, which is gentler than a typical real curation. Wall-clock figures come from jobs that shared nodes with other jobs and are approximate; the swing and objective columns are set comparisons unaffected by that. The `prove_abs_gap` runtimes depend on the `time_limit` of 1800 s, because the flux phases of step 1 run to the limit; they were not measured with another limit. RAVEN's `ftINIT` was not run. +Wall-clock figures come from jobs that shared nodes with other jobs and are approximate; the difference and objective columns are set comparisons and do not depend on load. The `prove_abs_gap` build times depend on the `time_limit` of 1800 s, because the flux phases of step 1 run to the limit; no other limit was measured. RAVEN's `ftINIT` was not run. -## Reproducing +## Reproduction -Requires Gurobi and a Human-GEM checkout. The preparation of the Human-GEM template (`prep`) and the scores follow `code/test/estimateEssentialGenes.py` of Human-GEM. +Requires Gurobi and a Human-GEM checkout. The preparation of the Human-GEM template (`prep`) and the scores follow `code/test/estimateEssentialGenes.py` of Human-GEM. The example passes `mip_gap_abs=10.0`, which the series `1+1` ignores. ```python from raven_toolbox.init import ftinit @@ -198,4 +190,4 @@ print(len({r.id for r in m0.reactions} ^ {r.id for r in m1.reactions})) `rxn_scores` and `gene_scores` come from `score_reactions_from_genes` and `gene_scores_from_expression` (original reaction id → score). `prep` comes from `prep_init_model(template, tasks, ...)`; `prep_edited` is built the same way from the edited template. -Set `FTINIT_DEBUG=1` to log each solve's status and each tie-break phase, which shows whether a build rests on a proven optimum or an unproven incumbent. The `UserWarning` messages carry the same information. +Set `FTINIT_DEBUG=1` to log the status of each solve and each tie-break phase, which shows whether a build rests on a proven optimum or an unproven incumbent. The `UserWarning` messages name the adopted phases at the time limit and miss a dropped tie-break. diff --git a/docs/parameter-tuning/studies/homology-cutoff-calibration.md b/docs/parameter-tuning/studies/homology-cutoff-calibration.md index 4ec1a2d..6000ae1 100644 --- a/docs/parameter-tuning/studies/homology-cutoff-calibration.md +++ b/docs/parameter-tuning/studies/homology-cutoff-calibration.md @@ -1,35 +1,39 @@ # Homology cut-off calibration -Three settings decide which template reactions transfer to a new organism: `min_identity`, `min_align_len` and `max_evalue`. None of them had been measured in either toolbox before this study. +Homology-based reconstruction transfers a template reaction to a new organism when a gene of the new organism matches a template gene in a BLAST or DIAMOND search and the match passes three filters: `min_identity`, `min_align_len` and `max_evalue` (`minIde`, `minLen` and `maxE` in RAVEN's `getModelFromHomology`). The study aligned the proteomes of four organisms against *S. cerevisiae* once, applied each filter setting to the saved hits, and scored the surviving matches against two references that state which genes are counterparts. It records one score per organism and setting, the agreement between BLAST and DIAMOND, and their run times. -| Setting | Previous value | Current value | Reason | -|---|---|---|---| -| `min_identity` | 40 | 40 | Changes the result most; 40 is at the centre of the optimum range given by both references. | -| `min_align_len` | 200 (RAVEN 2.11.3) | 100 | 200 discarded real matches; at 100 the wrong-match rate moves by at most 0.6 percentage points. | -| `max_evalue` | 1e-30 | 1e-30 | No effect anywhere between 1e-4 and 1e-50. | - -`min_align_len` is 100 in raven-toolbox and in RAVEN 3 (`getModelFromHomology`'s `minLen`); RAVEN 2.11.3 used 200. - -## Test design - -Four organisms were reconstructed from an *S. cerevisiae* template at a range of settings. Each match that survived was assessed for whether the two genes are counterparts. +## Set-up -Two independent sources provide the reference: KEGG's gene annotations and OMA's ortholog assignments. Wrong matches count double, because a wrong reaction is worse than a missing one: gap-filling can add what is absent, while a wrong reaction is hard to notice and harder to remove. +### Organisms and alignment -The organisms span a range of relatedness, because a setting that suits a close relative need not suit a distant one: +*S. cerevisiae* S288C is the template. The four targets span a range of relatedness to it, because a setting that suits a close relative need not suit a distant one: | Code | Organism | Relation to yeast | |---|---|---| | `kla` | *K. lactis* | close | | `yli` | *Y. lipolytica* | moderate | | `ani` | *A. nidulans* | distant | -| `eco` | *E. coli* | very distant | +| `eco` | *E. coli* | most distant | + +Proteomes come from UniProt reference proteomes. Each protein is relabelled with its KEGG gene name through UniProt's KEGG cross-reference for the exact organism code; proteins without one are dropped. Both references can then be applied to the same matches. + +Each target was aligned against the template in both directions, once with `run_blast` (aligner E-value `1e-4`) and once with `run_diamond` (aligner E-value `1e-3`). Every filter setting was applied to the same saved hit table, so each score comes from one deterministic calculation. A hit is kept when `evalue <= max_evalue`, `align_len >= min_align_len` and `identity >= min_identity`; `get_model_from_homology` requires `evalue < max_evalue`. The kept hits are collected into a set of (template gene, target gene) pairs, so a pair found in both directions, or by more than one hit, counts as one match. A hit whose two genes are not both judged by the reference is counted as unjudgeable and left out of the score. The scoring does not reduce hits to best hits or build a model. + +The sweeps vary one setting at a time from the RAVEN 2.11.3 values `max_evalue=1e-30`, `min_align_len=200` and `min_identity=40`. The identity sweep therefore ran at an alignment length of 200, the length sweep at 40 % identity, and the E-value sweep at 40 % identity and length 200. + +### References -Proteomes come from UniProt, relabelled with KEGG gene names so both references can be applied to the same matches. +The **KEGG reference** counts a match between a template gene and a target gene as correct when the two genes share at least one KO in KEGG's `organism_gene_ko` table (release 118). The table lists only KOs that are linked to a reaction. A match is judged only when both genes have such a KO; other matches are left out of the score. -## Sequence identity +The **OMA reference** counts a match as correct when the [OMA browser](https://omabrowser.org/) lists the two genes as a pair of orthologs. OMA identifies proteins by UniProt identifiers, which are mapped to KEGG gene names on both sides; OMA pairs that cannot be mapped are dropped. A match is judged when both genes have a KEGG gene name, so the OMA reference covers the whole mapped proteome. -The scores are against KEGG annotations (higher is better); the name of the score is not stated. +### Score + +For each setting, **precision** is the number of correct matches divided by all judged matches, and **recall** is the number of correct matches divided by all pairs in the reference. The score is F0.5, the weighted harmonic mean `1.25·P·R / (0.25·P + R)` of precision P and recall R. With β = 0.5 it weights precision twice as heavily as recall. The weighting is the authors' choice: a missing reaction can be added by gap-filling, while a wrongly transferred reaction is hard to notice and harder to remove. The reproduction command passes `--beta 0.5`, which is also the default of the script, so the scores in the tables are F0.5. All score tables are against the KEGG reference; their base rows agree with each other. + +## Results + +### Sequence identity | identity | `kla` | `yli` | `ani` | `eco` | |---|---|---|---|---| @@ -40,13 +44,11 @@ The scores are against KEGG annotations (higher is better); the name of the scor | 45 | 0.907 | 0.813 | 0.735 | 0.520 | | 50 | 0.882 | 0.736 | 0.640 | 0.323 | -Against OMA the best value is 45 for the three fungi and 35 for *E. coli*. The two sources therefore put the optimum between 35 and 45, and 40 is at the centre of that range. Neither supports a looser setting. The default stays at 40. - -Counting a missing match as equally bad as a wrong one moves the recommendation to 25, a different answer from the same measurements. A recommendation about these settings that does not state how it weighs the two kinds of mistake cannot be interpreted. +Against the KEGG reference the highest score was at 40 % for `kla` and `eco` and at 35 % for `yli` and `ani`. Against the OMA reference it was at 45 % for the three fungi and at 35 % for *E. coli*; the OMA scores are not tabulated. The optima of both references lie between 35 and 45 %, and in neither reference is an optimum below 35 %. -## Alignment length +Scored with F1, which weights precision and recall equally, the optimum moves to 25 % (values not tabulated). The best identity therefore depends on how the score weights the two kinds of error. -The reference of this table is not labelled. Its 200 row (0.923, 0.860, 0.804, 0.598) equals the 40 row of the KEGG identity table, so it is scored against KEGG annotations. No OMA scores are reported for alignment length. +### Alignment length | length | `kla` | `yli` | `ani` | `eco` | |---|---|---|---|---| @@ -56,22 +58,20 @@ The reference of this table is not labelled. Its 200 row (0.923, 0.860, 0.804, 0 | 200 (RAVEN 2.11.3) | 0.923 | 0.860 | 0.804 | 0.598 | | 300 | 0.885 | 0.816 | 0.764 | 0.571 | -Every value at or below 150 performs the same; the loss appears between 150 and 200. Dropping to 100 recovers 3–4 % more real matches on every organism while the wrong-match rate moves by at most 0.6 of a percentage point. 50 and 100 are identical, so 100 was chosen as the less permissive of the two. +Lengths 50 and 100 gave identical scores and 150 was at most 0.003 lower. The score fell between 150 and 200, by 0.010 to 0.020 from 100 to 200. Compared with 200, a length of 100 raised recall by 3 to 4 points on every organism and lowered precision by at most 0.6 points (precision and recall values not tabulated). No OMA scores were reported for alignment length. -## E-value +### E-value -This table covers `kla` and `ani` only, and its reference is not labelled. Its 1e-4 … 1e-50 values (0.923, 0.804) equal the 40 row of the KEGG identity table, so it is scored against KEGG annotations. No OMA scores are reported for the e-value. +The E-value sweep was reported for `kla` and `ani` only. The hits were collected at an aligner E-value of `1e-4`, so `max_evalue=1e-4` keeps every hit. | `max_evalue` | `kla` | `ani` | |---|---|---| | 1e-4 … 1e-50 | 0.923 | 0.804 (identical throughout) | | 1e-100 | 0.877 | 0.703 | -The score does not change across the 46 orders of magnitude from 1e-4 to 1e-50. Identity and length have already removed whatever a looser e-value would have removed, so the e-value has no remaining effect. The score changes only at 1e-100, where good matches are discarded. The value stays at 1e-30 to match RAVEN. +The score was the same at 1e-4, 1e-10, 1e-30 and 1e-50, which spans 46 orders of magnitude, and fell at 1e-100. A likely reason (inference, not checked against the hit tables) is that every hit with at least 40 % identity over at least 200 residues already has an E-value below 1e-50. No OMA scores were reported for the E-value. -## Comparison of DIAMOND and BLAST - -Both toolboxes offer DIAMOND as the fast alternative and apply the same three settings to its output, which is safe only if the two aligners behave alike; the table shows that they do: +### BLAST and DIAMOND | Organism | best with BLAST | best with DIAMOND | score at that point | |---|---|---|---| @@ -80,31 +80,41 @@ Both toolboxes offer DIAMOND as the fast alternative and apply the same three se | `ani` | 35 | 35 | 0.814 vs 0.811 | | `eco` | 40 | 40 | 0.598 vs 0.593 | -DIAMOND finds about half as many matches (29,785 against 54,478 on `kla`), but almost all the missing ones are weak matches that these settings discard. After filtering, the two agree on 87–92 % of what survives, and DIAMOND runs 10–20× faster (34 s against 358 s on `kla`; 40 s against 788 s on `ani`). +The best identity was the same with both aligners for every organism, and the scores at that identity differed by 0.002 to 0.005. DIAMOND returned 29,785 hits on `kla` against 54,478 for BLAST. After filtering, the two aligners agreed on 87 to 92 % of the surviving matches. That the hits missing from DIAMOND are weak matches the filters would discard is an inference from these two results. DIAMOND took 34 s against 358 s for BLAST on `kla` and 40 s against 788 s on `ani`, 10.5 and 19.7 times faster (thread count not recorded). + +### Aligner settings -The data therefore support one shared set of defaults for both aligners, and DIAMOND is the faster option for large jobs. +Two more settings control the search that produces the hits. Neither was part of the sweeps. -## Search settings +`evalue` is the aligner's own cut-off: `1e-4` for `run_blast` and `1e-3` for `run_diamond`. Each matches the MATLAB side: `getBlast` passes `-evalue 10e-5`, which equals `1e-4`, and `getDiamond` passes none, so DIAMOND's own default applies. On *H. polymorpha* against *S. cerevisiae*, loosening the BLAST cut-off from `1e-5` to `1e-4` added 5,204 raw hits, 8.9 % of the table. Every added hit has an E-value above `1e-5` and so fails `max_evalue=1e-30`. As long as the aligner cut-off is looser than `max_evalue`, it changes the size of the hit table and not the model. -The three settings filter BLAST's output. Two more control the search that produces it, and neither affected the finished model. +`threads` defaults to every CPU the process may run on. A 500-query subset returned the same 2,469 hits at one thread and at four, in 45.2 s and 23.7 s. On a full proteome pair, single-threaded BLASTP took about 20 minutes per direction. -`evalue` is the aligner's own cut-off, `1e-4` for `run_blast` and `1e-3` for `run_diamond`, each matching what the MATLAB side passes, explicitly in `getBlast`'s case and by inheriting DIAMOND's own default in `getDiamond`'s. On *H. polymorpha* against *S. cerevisiae*, loosening from `1e-5` to `1e-4` admits 5,204 more raw hits, 8.9 % of the table. None of them reach the model: at `min_identity=40` and `max_evalue=1e-30` the post-BLAST filters discard essentially all of them. The setting decides the size of the intermediate hit table and nothing downstream of it, so there is no correctness argument for either value. +## Chosen defaults + +| Setting | RAVEN 2.11.3 | Default (raven-toolbox and RAVEN 3) | Basis | +|---|---|---|---| +| `min_identity` (`minIde`) | 40 | 40 | The setting with the largest effect; the optima of both references lie between 35 and 45 %. | +| `min_align_len` (`minLen`) | 200 | 100 | 100 scored 0.010 to 0.020 higher than 200 against KEGG; 50 and 100 scored the same, and 100 is the stricter of the two. | +| `max_evalue` (`maxE`) | 1e-30 | 1e-30 | No change in score from 1e-4 to 1e-50; kept at the RAVEN value. | -`threads` defaults to every CPU the process may run on. Both aligners score each query sequence independently, so the result does not depend on it: a 500-query subset returned an identical 2,469 hits at one thread and at four, in 45.2 s and 23.7 s. On a full proteome pair, single-threaded BLASTP takes about 20 minutes per direction. +The study supports one set of defaults for BLAST and DIAMOND, since both reached the same optimum. -## Limits of the measurements +## Limits -- KEGG's annotations rely on BLAST comparisons, so on their own they would partly measure agreement with the method being tested. OMA infers counterparts independently, so both references were used; they agree. -- The two sources are not directly comparable in absolute terms. KEGG only covers genes it has annotated (a few hundred well-studied ones per organism) while OMA covers whole proteomes and lists strict counterparts, so ordinary gene duplicates lower the measured score. Only the shape of each column is comparable between the two sources. -- The *Y. lipolytica* OMA row has the weakest reference: 38 % of its OMA pairs could not be matched to KEGG gene names, so that reference is incomplete. +- KEGG's KO annotations are partly derived from sequence similarity, so the KEGG reference alone would partly measure agreement with the method under test. OMA infers orthology independently of BLAST, and its optima lie in the same range of 35 to 45 % identity. OMA scores were reported for identity only. +- The two references are not comparable in absolute terms. The KEGG reference judges only genes with a reaction-linked KO, while the OMA reference judges the whole mapped proteome and counts every match that OMA does not list as an ortholog pair as wrong. Paralogs probably lower the OMA score for this reason (inference). Only the shape of each column is comparable between the references. +- The OMA reference for *Y. lipolytica* is incomplete: 38 % of its OMA pairs could not be mapped to KEGG gene names. - One template: everything was measured outward from *S. cerevisiae*. -- The measurements are on gene matches, not finished models: a reaction transfers if any one of its genes matches, so the effect on a finished model is smaller than these numbers suggest. +- One setting was varied at a time from length 200 and identity 40 %. The identity optimum and the absence of an E-value effect were measured at length 200, and were not measured at the default length of 100. +- The scores count gene matches, not reactions in a finished model. A reaction transfers when any one of its genes matches, so the effect on a finished model is probably smaller (inference, not measured). +- Each score comes from one alignment per organism; scoring is deterministic, so there is no run-to-run spread. ## Retired curated-GEM reference -One arm of this study scored candidate thresholds against curated non-model-organism GEMs (hanpo-GEM, rhto-GEM) built by RAVEN's own homology-based reconstruction. The arm was dropped, and the reason is recorded here because it generalises beyond this study. +A further part of the study scored candidate settings against two curated genome-scale metabolic models (GEMs) of non-model fungi, [hanpo-GEM](https://github.com/SysBioChalmers/hanpo-GEM) and [rhto-GEM](https://github.com/SysBioChalmers/rhto-GEM), which were built with RAVEN's homology-based reconstruction. This reference was dropped. -The diagnostic loosened the thresholds one step at a time and recorded what fraction of newly admitted reactions were already in the curated model. The rate held between 0.62 and 0.85 up to hanpo-GEM's actual build settings (`max_evalue=1e-30, min_align_len=150, min_identity=35`) and fell to 0.06 one step past them. The drop reflects the reference matching its own build settings: the curated model agrees with predictions made at its own build settings and disagrees with everything else, regardless of whether those other settings are better or worse. Optimising against it returns the build settings whether or not they are good, and it can measure self-consistency but not correctness. Most curated non-model fungal GEMs are RAVEN drafts built the same way, so this applies to that class as well as to these two models. +The test loosened the settings one step at a time and recorded which fraction of the newly admitted reactions was already in the curated model. The fraction stayed between 0.62 and 0.85 up to hanpo-GEM's own build settings (`max_evalue=1e-30, min_align_len=150, min_identity=35`) and fell to 0.06 one step beyond them. The inference is that the curated model agrees with predictions made at its own build settings and with nothing looser, whether or not those settings are good. A score against such a model returns its build settings and measures self-consistency, not correctness. The same applies to any curated model drafted with RAVEN's homology method (inference). ## Reproducing @@ -115,4 +125,4 @@ python scripts/homology_cutoff_kegg.py score --out work/ --beta 0.5 \ --gene-ko kegg118_organism_gene_ko.tsv.gz ``` -Aligning is the only slow part and its output is saved, so re-scoring (under a different weighting, or against a different reference) takes about two minutes. +`scripts/homology_cutoff_kegg.py` is in raven-toolbox. `align --aligner diamond` and `score --aligner diamond` repeat the steps with DIAMOND, and `score --reference oma` scores against the OMA reference. The script also scores identities of 20 and 60 %, which are not reported. The alignment times recorded for `kla` and `ani` are in [BLAST and DIAMOND](#blast-and-diamond). The alignment output is saved, so re-scoring under a different weighting or against a different reference takes about two minutes. The *H. polymorpha* and thread measurements of the aligner settings are not produced by this script. diff --git a/docs/parameter-tuning/studies/humangem-validation.md b/docs/parameter-tuning/studies/humangem-validation.md index db603c7..a4d9bb2 100644 --- a/docs/parameter-tuning/studies/humangem-validation.md +++ b/docs/parameter-tuning/studies/humangem-validation.md @@ -1,39 +1,42 @@ # Human-GEM cell-type model validation against RAVEN -Validation of raven-toolbox's ftINIT against MATLAB RAVEN on a genome-scale reconstruction (Human-GEM) using the Hart2015 RNA-seq dataset (5 cell lines: DLD1, GBM, HCT116, HELA, RPE1). The goal is functional equivalence: raven-toolbox and RAVEN should extract the same context-specific reaction sets from the same inputs. +raven-toolbox's ftINIT was run on Human-GEM with the RNA sequencing (RNA-seq) data of five cell lines from the Hart2015 dataset (DLD1, GBM, HCT116, HELA, RPE1), and each extracted cell-type model was compared with the model that MATLAB RAVEN's `ftINIT` extracted from the same input. The comparison records the reactions each model contains, the reactions the two models share and the reactions only one of them contains, and the time each step took. The question is whether raven-toolbox selects the same reactions as RAVEN. -## Method +## Set-up -* Versions. Human-GEM commit `38567ab8` (v2.1.1 plus 15 commits), RAVEN commit `2d0630e3` (MATLAB 2024b), raven-toolbox commit `94f3940`, Gurobi 13.0.1 for both tools, 4 cores per run. -* Template and inputs. RAVEN ran `prepHumanModelForftINIT` on Human-GEM (drug reactions, amino-acid triplet reactions and the duplicate complex I reaction removed, spontaneous and custom reaction lists set, essential tasks from `metabolicTasks_Essential.txt`), which gives a reference model of 9730 reactions. raven-toolbox ran `prep_init_model` on the same cleaned Human-GEM (11265 reactions). Every reaction in every raven-toolbox result is part of RAVEN's reference model, so the sets are compared directly. -* Scoring. Gene scores `5·ln(level/1)` clipped to [−5, 10] from the TPM values of the Hart2015 file, mapped to reactions through the GPR (`gene_scores_from_expression`, `score_reactions_from_genes` in raven-toolbox; `transcrData` with `threshold=1` in RAVEN), reactions without genes score −2. -* ftINIT. Series `1+1` (2 staged MILP steps). RAVEN: `ftINIT` with default settings. raven-toolbox: `raven_toolbox.init.ftinit` with series `1+1`, `time_limit=600` and the default `big_m=1000`. A named series runs its own schedule of MILP runs in each step, so `mip_gap` and `mip_gap_abs` have no effect on the full pipeline. -* Tasks. Two raven-toolbox variants: **no-task** (preparation without tasks, expression only) and **task-constrained** (essential metabolic tasks, task-essential reactions forced to be kept). RAVEN's reference is task-constrained. +Software: Human-GEM commit `38567ab8` (v2.1.1 plus 15 commits), RAVEN commit `2d0630e3` on MATLAB 2024b, raven-toolbox commit `94f3940`, and Gurobi 13.0.1 for both tools, with 4 cores per run. -## Construction choices that affect tractability +Template preparation is the step that runs once per template model, before any cell line. For RAVEN, the Human-GEM function `prepHumanModelForftINIT` removed the drug reactions, the amino-acid triplet reactions and the duplicate complex I reaction from Human-GEM, set the spontaneous and custom reaction lists, and prepared the model with the essential tasks of `metabolicTasks_Essential.txt`. Its reference model, the prepared model from which RAVEN extracts, has 9730 reactions. For raven-toolbox, `prep_init_model` ran on the same cleaned Human-GEM (11265 reactions). Every reaction in every raven-toolbox result is in RAVEN's reference model, so the reaction identifiers of the two results are compared directly. -1. Linear-time constraint construction. Building the steady-state balances with Python `sum()` re-canonicalises a growing sympy expression at each term; hub metabolites (ATP, H⁺ and H₂O in ~10³ reactions) give balances with 10³ terms or more, and summing 1500 terms with `sum` takes 59 s against 0.01 s with `optlang.symbolics.add`. raven-toolbox builds flat term lists once per reaction and sums them with `optlang.symbolics.add`. -2. Per-reaction big-M. The on/off indicator constraints of a scored reaction use `max(min(ub, big_m), min(-lb, big_m), force_on)` as big-M, an infinite bound counting as `big_m`, with `big_m=1000` as the default cap (RAVEN's `bigMCap` in `ftINITInternalAlg`). The cap limits the flux a kept reaction can carry. In the staged pipeline `big_m=25` is infeasible and `big_m=100` is feasible with the default scaling (see [init-param-calibration.md](init-param-calibration.md)). -3. Stoichiometric rescaling. `rescaleModelForINIT` (cap each reaction's coefficient dynamic range at 25×, normalise mean |coeff| to 1) is ported into `prep_init_model`. In the HCT116 runs of the [calibration study](init-param-calibration.md), the staged pipeline with `big_m=1000` is also feasible without it (157 s against 83 s with it, Jaccard 0.969 between the two results), and with `big_m=100` and no rescaling it is infeasible. +Scoring: gene scores are `5·ln(level/1)` of the transcripts per million (TPM) values in the Hart2015 file, clipped to [−5, 10], and reactions take their scores from the gene rules; a reaction without genes scores −2. raven-toolbox computes this with `gene_scores_from_expression` and `score_reactions_from_genes`, RAVEN with `transcrData` and `threshold=1`. + +Extraction: both tools ran the series `1+1`, two staged mixed-integer linear program (MILP) steps. RAVEN ran `ftINIT` with its default settings. raven-toolbox ran `raven_toolbox.init.ftinit` with series `1+1`, `time_limit=600` and the default `big_m=1000`. A named series runs its own schedule of MILP runs in each step, so `mip_gap` and `mip_gap_abs` have no effect on these runs. + +raven-toolbox ran in two variants: + +- **No-task**: the template is prepared without a task list, and the extraction uses the expression scores alone. +- **Task-constrained**: the template is prepared with the essential metabolic tasks, the reactions that a task needs are kept whatever their score, and `ftinit` adds reactions back until every task is feasible (its default `fill_gaps=True`). + +RAVEN's reference result is task-constrained in this sense. A third set of no-task runs used `big_m=100`. + +Measurements: the result of a run is its set of reaction identifiers. For each cell line the comparison gives the number of reactions in both sets (shared), in one set only, and the Jaccard index, the shared count divided by the size of the union. The comparison covers reactions only; genes, essential genes and task performance of the extracted models were not compared. Each cell line, variant and `big_m` value ran once. Times are wall-clock times of single runs on shared nodes. ## Results -### Run times (one run per cell line) +### Run times -| step | time | +| Step | Time | |------|-----:| | RAVEN `prepHumanModelForftINIT` (4-worker parallel pool, once per template) | 4320 s | -| raven-toolbox `prep_init_model`, no tasks (cobrapy's default process count for FVA, once per template) | 1556 s | -| raven-toolbox `prep_init_model`, with tasks (`cobra.Configuration().processes = 1`, once per template) | 1695 s | -| RAVEN `ftINIT` per cell line | 40 to 91 s | -| raven-toolbox `ftinit` no-task per cell line | 81 to 91 s | -| raven-toolbox `ftinit` task-constrained per cell line | 146 to 381 s | - -The preparation and `ftINIT` times differ from those in the [context-specific model guide](../../guide/init.md) (47 min in MATLAB and 37 min in Python for the preparation; `ftINIT` 797 s in MATLAB and 124 s in Python). The guide runs RAVEN's `prepINITModel` directly on Human-GEM `6812be2a` (12748 reactions, no cleaning) with series `1+0`; the validation runs `prepHumanModelForftINIT` (which removes drug and amino-acid triplet reactions first) on Human-GEM `38567ab8` with series `1+1`. The runs differ in function, template and series together, so the logs do not attribute the difference to one of them. +| raven-toolbox `prep_init_model`, no-task (cobrapy's default process count for flux variability analysis, once per template) | 1556 s | +| raven-toolbox `prep_init_model`, task-constrained (`cobra.Configuration().processes = 1`, once per template) | 1695 s | +| RAVEN `ftINIT`, per cell line | 40 to 91 s | +| raven-toolbox `ftinit`, no-task, per cell line | 81 to 91 s | +| raven-toolbox `ftinit`, task-constrained, per cell line | 146 to 381 s | -### Reaction counts +### Reactions in the extracted models -| cell line | RAVEN ftINIT | raven-toolbox ftINIT (no-task) | raven-toolbox ftINIT (task) | +| Cell line | RAVEN `ftINIT` | raven-toolbox, no-task | raven-toolbox, task-constrained | |-----------|-------------:|--------------------------:|-----------------------:| | DLD1 | 7339 | 7310 | 7348 | | GBM | 7278 | 7255 | 7281 | @@ -41,11 +44,9 @@ The preparation and `ftINIT` times differ from those in the [context-specific mo | HELA | 7417 | 7384 | 7432 | | RPE1 | 7166 | 7052 | 7165 | -The task-constrained counts are within 0.2 % of RAVEN's; the no-task counts are 0.3 to 0.5 % lower, and 1.6 % lower for RPE1. - -### Agreement of raven-toolbox (no-task) ftINIT with RAVEN ftINIT +### Agreement of the no-task variant with RAVEN -| cell line | shared | only raven-toolbox | only RAVEN | Jaccard | +| Cell line | Shared | Only raven-toolbox | Only RAVEN | Jaccard | |-----------|-------:|--------------:|-----------:|--------:| | DLD1 | 7255 | 55 | 84 | 0.981 | | GBM | 7202 | 53 | 76 | 0.982 | @@ -53,11 +54,9 @@ The task-constrained counts are within 0.2 % of RAVEN's; the no-task counts are | HELA | 7321 | 63 | 96 | 0.979 | | RPE1 | 6980 | 72 | 186 | 0.964 | -The no-task run is expression-only while RAVEN's reference is task-constrained; the surplus of reactions only in RAVEN (76 to 186) includes reactions that the tasks require. +### Agreement of the task-constrained variant with RAVEN -### Agreement of raven-toolbox (task-constrained) ftINIT with RAVEN ftINIT - -| cell line | shared | only raven-toolbox | only RAVEN | Jaccard | +| Cell line | Shared | Only raven-toolbox | Only RAVEN | Jaccard | |-----------|-------:|--------------:|-----------:|--------:| | DLD1 | 7286 | 62 | 53 | 0.984 | | GBM | 7235 | 46 | 43 | 0.988 | @@ -65,11 +64,9 @@ The no-task run is expression-only while RAVEN's reference is task-constrained; | HELA | 7366 | 66 | 51 | 0.984 | | RPE1 | 7095 | 70 | 71 | 0.981 | -With the essential metabolic tasks (the task list RAVEN uses) the agreement is Jaccard 0.981 to 0.988 and the differences are symmetric (only-raven-toolbox 46 to 70, only-RAVEN 43 to 71). The residual of 43 to 71 reactions each way out of ~7300 is of the same order as the seed-to-seed difference of 110 reactions in [ftinit-determinism.md](ftinit-determinism.md), which measures a different setup (Human-GEM v2.0.0-38, DLD1, `big_m=100`, escalation schedule). - -### Effect of `big_m` on the agreement (no-task, Jaccard against RAVEN) +### Agreement of the no-task variant at two `big_m` values -| cell line | `big_m=1000` (default) | `big_m=100` | Jaccard between the two raven-toolbox results | +| Cell line | Jaccard against RAVEN, `big_m=1000` (default) | Jaccard against RAVEN, `big_m=100` | Jaccard between the two raven-toolbox results | |-----------|---------------------:|----------:|----------------------------------------------:| | DLD1 | 0.981 | 0.979 | 0.984 | | GBM | 0.982 | 0.982 | 0.985 | @@ -77,14 +74,48 @@ With the essential metabolic tasks (the task list RAVEN uses) the agreement is J | HELA | 0.979 | 0.979 | 0.981 | | RPE1 | 0.964 | 0.967 | 0.984 | -Agreement with RAVEN is the same at the two values tried, `big_m=100` and `big_m=1000`, at the resolution of these runs (differences of at most 0.003, one run per cell line and value). The no-task run time at `big_m=100` is 83 to 122 s, at `big_m=1000` 81 to 91 s. +The no-task `ftinit` runs took 83 to 122 s at `big_m=100` and 81 to 91 s at `big_m=1000`. + +## Agreement with the task layer + +The task-constrained variant shares Jaccard 0.981 to 0.988 with RAVEN across the five cell lines. Its reaction counts are within 0.2 % of RAVEN's, and the differences run in both directions in similar numbers: 46 to 70 reactions only in the raven-toolbox model and 43 to 71 only in the RAVEN model, out of ~7300. + +Neither tool was run twice on the same input, so the comparison does not separate alternate optima of the MILP from a difference between the two implementations. For scale, two raven-toolbox runs that differed only in the seed differed by 110 reactions in [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](ftinit-determinism.md). That measurement used a different set-up (Human-GEM v2.0.0-38, DLD1, `big_m=100`, an escalation schedule), so it indicates the order of magnitude and does not bound the spread here. + +## Agreement without the task layer -## Conclusions +The no-task variant shares Jaccard 0.964 to 0.982 with RAVEN. It keeps fewer reactions than RAVEN in every cell line: 0.3 to 0.5 % fewer for four cell lines and 1.6 % fewer for RPE1. The reactions only in the RAVEN model (76 to 186) outnumber those only in the raven-toolbox model (53 to 76). -From the same cleaned Human-GEM, raven-toolbox reproduces RAVEN's ftINIT reaction selection to Jaccard 0.964 to 0.982 (no-task) and 0.981 to 0.988 (task-constrained) across five cell lines. In the task-constrained run, 43 to 71 reactions per cell line are in only one of the two results, in each direction. Neither tool has a repeat run in this data, so the numbers do not separate solver-level alternate optima from a difference between the implementations. +The no-task variant omits the task layer that RAVEN's reference applies. With the task layer, the reactions only in the RAVEN model fall to 43 to 71 per cell line. That the task layer accounts for most of the surplus is an inference from these counts; the reactions in the surplus were not checked against the task-essential reactions. -raven-toolbox uses RAVEN's numerical-conditioning choices (per-reaction big-M capped at 1000, `rescaleModelForINIT`) and uses `optlang.symbolics.add` in place of Python `sum()` in every MILP build (ftINIT and the gap-fill). With these, a task-constrained cell-line model builds in 146 to 381 s and a no-task one in 81 to 91 s after the once-per-template preparation (26 to 28 min), against 40 to 91 s for RAVEN after its 72 min preparation. +## Effect of `big_m` + +Lowering the cap on the per-reaction big-M from 1000 to 100 changed the Jaccard index against RAVEN by at most 0.003 in each cell line, one run per cell line and value. In every cell line the two raven-toolbox results shared a higher Jaccard index with each other (0.981 to 0.985) than either shared with RAVEN (0.964 to 0.982); for RPE1 the values were 0.984 between the two results against 0.964 and 0.967 with RAVEN. The cap therefore changed the reaction set (1 minus the Jaccard index between the two results, 0.015 to 0.019) by more than it changed the agreement with RAVEN (at most 0.003). At this resolution the two values agree with RAVEN equally well. + +## Run time + +After the template preparation, a task-constrained cell-type model took 146 to 381 s in raven-toolbox and 40 to 91 s in RAVEN; a no-task model took 81 to 91 s. The template preparation took 1556 s (26 min) and 1695 s (28 min) in raven-toolbox and 4320 s (72 min) in RAVEN. The preparations ran with different parallelism (see the run-time table), so the times compare whole runs and not the code alone. The [Reference run](../../guide/init.md#reference-run) of the guide Context-specific models with ftINIT reports a preparation of 47 min in MATLAB and 37 min in Python and an extraction of 797 s in MATLAB and 124 s in Python. That run used RAVEN's `prepINITModel` directly on Human-GEM `6812be2a` (12748 reactions) without the cleaning step, and series `1+0`; the three differences from this validation were not separated, so the times are not comparable. + +## Numerical settings shared with RAVEN + +raven-toolbox builds the extraction MILP with two numerical settings of RAVEN: + +- Per-reaction big-M. The on-off constraints of a scored reaction use `max(min(ub, big_m), min(-lb, big_m), force_on)` as big-M, an infinite bound counting as `big_m`. The default cap `big_m=1000` is RAVEN's `bigMCap` in `ftINITInternalAlg`. The cap limits the flux a reaction can carry while it counts as on. +- Stoichiometric rescaling. `prep_init_model` applies RAVEN's `rescaleModelForINIT` to the merged model: each coefficient of a reaction is capped at 25 times the smallest coefficient of that reaction, and the reaction is then scaled to a mean absolute coefficient of 1. +Which combinations of `big_m` and rescaling are feasible for the staged pipeline was measured on HCT116 in [ftINIT parameter calibration and input robustness](init-param-calibration.md): with rescaling, `big_m=25` was infeasible and `big_m=100` feasible; without rescaling, `big_m=1000` was feasible (157 s against 83 s with rescaling, Jaccard 0.969 between the two results) and `big_m=100` infeasible. + +## Constraint construction + +Each steady-state balance of the extraction MILP is built from a flat list of terms and summed with `optlang.symbolics.add`; the same construction is used in the gap-fill. Python's `sum()` re-simplifies the growing expression at each term, and the balances of hub metabolites such as ATP, H⁺ and H₂O have 10³ terms or more. In one timing, summing 1500 terms took 59 s with `sum` and 0.01 s with `optlang.symbolics.add`; the set-up of that timing is not recorded. + +## Use of the result + +The agreement was measured with the code defaults `big_m=1000` (`_BIG_M` in `raven_toolbox/init/ftinit.py`, listed in [Tuned parameter defaults](../../tuned-parameters.md)) and rescaling on (`scale=True` in `prep_init_model`, with `max_stoich_diff=25` in `rescale_for_init`, both in `raven_toolbox/init/prep.py`). A raven-toolbox model agreed most closely with a RAVEN `ftINIT` model when `prep_init_model` received the same task list as RAVEN's preparation. ## Limits -Each cell line and variant is one run, without seed repeats, so RAVEN's run-to-run spread was not measured and Jaccard differences of a few tenths of a percent are not resolved. The timings come from shared nodes and are approximate. The preparation times compare runs with different parallelism (see the run-time table). +Each cell line, variant and `big_m` value ran once, without seed repeats, so the run-to-run spread of either tool was not measured and Jaccard differences of a few tenths of a percent are not resolved. The comparison covers reaction sets on one template and one dataset; gene content, essential genes and task performance were not compared. The timings come from shared nodes and are approximate. + +## Reproduction + +The script that ran this validation is not part of the pinned raven-toolbox. Its outputs, kept in the working directory that the analysis scripts take as `--work`, are reused by later studies: the model exported from the RAVEN run as `raven_refModel.xml`, and the prepared Human-GEM of raven-toolbox as `rg_prep.pkl` (no-task) and `rg_prep_tasks.pkl` (task-constrained). `scripts/analyze_init_solvers.py` reads `rg_prep.pkl` and `scripts/analyze_init_robustness.py` reads `rg_prep_tasks.pkl`; both score reactions against `raven_refModel.xml`. `scripts/analyze_init_params.py` reads `raven_refModel.xml` and prepares its own models from it. diff --git a/docs/parameter-tuning/studies/init-param-calibration.md b/docs/parameter-tuning/studies/init-param-calibration.md index 30adccb..2d81732 100644 --- a/docs/parameter-tuning/studies/init-param-calibration.md +++ b/docs/parameter-tuning/studies/init-param-calibration.md @@ -1,19 +1,63 @@ # ftINIT parameter calibration and input robustness -The study covers the ftINIT parameters of raven-toolbox on a genome-scale model (Human-GEM, Hart2015, HCT116) in two parts: +This study runs raven-toolbox's ftINIT on Human-GEM with the HCT116 expression data of the Hart2015 dataset and varies one or two settings per run series; the scaling runs and the full-pipeline runs vary scaling and `big_m` together. The calibration part varies the mixed-integer linear program (MILP) settings `mip_gap`, `big_m` and `force_on` and the stoichiometric scaling of the prepared model, and records the run time, the MILP objective and which reactions are kept. The robustness part degrades the expression input by dropout, noise and downsampling, extracts a model with the metabolic-task layer on, and records the model size, the essential tasks the model passes and its overlap with the model from clean data. -1. Calibration: the parameter values with the best trade-off between speed and quality on clean data (`scripts/analyze_init_params.py`). -2. Robustness: the effect of degrading the transcriptomics input on the model, with the task layer always on (it is part of the pipeline and is not varied), and the parameters that keep the model functional and stable (`scripts/analyze_init_robustness.py`). +## Set-up -Both scripts are resumable and reusable on any model and dataset; the numbers are for HCT116. The clean-data calibration uses the per-reaction big-M cap `big_m` (Human-GEM `38567ab8`). The robustness measurements used `big_m=100` and an earlier Human-GEM state whose commit is not recorded, so the two sets of numbers do not share a baseline. "Jaccard" is the reaction-set overlap with the reference (tightest setting or clean data). For a model-extraction tool the reaction set is the product, and a MIP gap bounds only the objective, so set stability is tracked separately. +### Software and inputs ---- +The calibration runs used Human-GEM commit `38567ab8` (v2.1.1 plus 15 commits), the HCT116 column of the Hart2015 RNA sequencing file, raven-toolbox commit `94f3940` and Gurobi 13.0.1 on 4 cores. The template is the reference model of the [Human-GEM cell-type model validation against RAVEN](humangem-validation.md): Human-GEM without the drug reactions, the amino-acid triplet reactions and the duplicate complex I reaction, 11265 reactions. `prep_init_model` prepared it without a task list. Its scaled merged model, the model that the MILP receives, has 6657 reactions; the preparation took 26 min and is excluded from every time in the tables. -## Clean-data calibration +The robustness runs used an earlier Human-GEM state whose commit is not recorded, the same HCT116 data, a template prepared with the essential metabolic tasks, and `big_m=100`. The two parts therefore have no common baseline, and their reaction counts and Jaccard indices cannot be compared with each other. -The clean-data runs used Human-GEM commit `38567ab8` (v2.1.1 plus 15 commits), HCT116 column of Hart2015, raven-toolbox commit `94f3940`, Gurobi 13.0.1, 4 cores. The model after removing drug reactions, amino-acid triplet reactions and the duplicate complex I reaction has 11265 reactions; the scaled merged model that the MILP sees has 6657 reactions. `big_m` is the cap on the per-reaction big-M: each scored reaction gets `max(min(ub, big_m), min(-lb, big_m), force_on)`, an infinite bound counts as `big_m`, so `big_m` limits the flux that a reaction can carry while it counts as on. Each row is one run without seed repeats, so Jaccard differences of 1 to 2 % are not attributed to the parameter; this is the size of the spread between seeds in [ftinit-determinism.md](ftinit-determinism.md), which uses a different setup (Human-GEM v2.0.0-38, DLD1, `big_m=100`, escalation schedule). Times are the wall clock of the MILP call and of the whole `ftinit` call, without the model preparation (26 min for the scaled prep without tasks). +Gene scores are `5·ln(level/1)` of each expression level, clipped to [−5, 10] (see [Gene-scoring constants](#gene-scoring-constants)), and reaction scores follow from the gene rules (`score_reactions_from_genes`, default `no_gene_score=-2`). -### ftINIT MILP `mip_gap` (single step-0 solve, `big_m=1000`, `force_on=0.1`) +### Run types + +| Run type | What runs | Script and sweep | +|---|---|---| +| Step-0 solve | One `run_ftinit` call (one MILP, no run schedule) for the first step of series `1+1` on the scaled merged model, with a 900 s limit | `analyze_init_params.py`, sweep `ftinit_milp` | +| Scaling run | The same step-0 solve on a model prepared without scaling, which `rescale_for_init` then rescales with the given `max_stoich_diff` (or leaves unscaled), with a 600 s limit and `mip_gap=0.001`. | `analyze_init_params.py`, sweep `prep_scale` | +| Full pipeline | One call of `ftinit` with series `1+1` and `time_limit=600` on the prepared model without tasks: both MILP steps, each with its own schedule of runs. | `analyze_init_params.py`, sweep `ftinit_full` | +| Robustness run | One call of `ftinit` with series `1+1` on the template prepared with tasks, including the gap-fill that adds reactions until every task is feasible, followed by `check_tasks` on the extracted model. | `analyze_init_robustness.py`, phases `gradient` and `levers` | + +A named series such as `1+1` ignores `mip_gap` and `mip_gap_abs`: every step runs its own schedule of MILP runs, with limits of 120 s, 5000 s and 5 s. `time_limit` lowers the limit of each scheduled run, so at `time_limit=600` it shortens the 5000 s runs only. `mip_gap` therefore acts on the step-0 solve and not on the full pipeline. + +`big_m` is the cap on the per-reaction big-M of the on and off constraints. Each scored reaction gets `max(min(ub, big_m), min(-lb, big_m), force_on)`, and an infinite bound counts as `big_m`, so `big_m` limits the flux that a reaction can carry while it counts as on. `force_on` is the smallest flux at which a reaction counts as on. `max_stoich_diff` is the largest ratio between the stoichiometric coefficients of one reaction after `rescale_for_init` (RAVEN's `rescaleModelForINIT`). + +The robustness runs degrade the expression vector of HCT116 in three ways, each with one random draw (seed 0): + +- **Dropout**: a fraction of the genes in the data gets the level 0, which scores −5. +- **Noise**: every level is multiplied by `exp(N(0,σ))`, a log-normal factor with standard deviation σ on the natural-log scale. +- **Downsampling**: a fraction of the genes is removed from the data. Within a gene rule a missing gene is skipped, and a reaction whose genes are all missing gets `no_gene_score`. + +The lever runs repeat the 70 % dropout input with `no_gene_score` or `force_on` changed. + +### Measurements + +| Column | Meaning | +|---|---| +| time (s) | Wall-clock time of the `run_ftinit` call (step-0 and scaling runs) or the `ftinit` call (full pipeline), including the construction of the MILP and excluding the preparation | +| objective | MILP objective of the step-0 solve | +| rel.obj.gap | (reference objective − objective) / reference objective; a positive value is a lower objective than the reference row | +| `n_kept` | Reactions in the model that the full pipeline returns | +| n_rxns | Reactions in the model that a robustness run returns, after the gap-fill | +| Jaccard | Jaccard index of two kept-reaction sets: the reactions in both divided by the reactions in either. Step-0 sets are merged-model reaction identifiers; full-pipeline and robustness sets are Human-GEM reaction identifiers | +| tasks pass, frac | Essential tasks that `check_tasks` reports as passed on the extracted model, as a count and as a fraction | + +A MIP gap bounds the objective only. Two solutions within the gap can keep different reactions, so the reaction set is compared separately from the objective. + +Each table row is one run unless the row states repeats; this set-up has no runs that differ only in the seed. For scale, two raven-toolbox builds that differed only in the seed differed by 110 reactions (symmetric difference of the kept-reaction sets) in [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](ftinit-determinism.md), in a different set-up (Human-GEM v2.0.0-38, DLD1, `big_m=100`, escalation schedule, with tasks and gap-fill). That page gives 7754 reactions for one build of that set-up and not the sizes of the two seed builds; assuming builds of about 7300 to 7800 reactions, 110 reactions are about 1.5 % of the model and correspond to a Jaccard index of about 0.985. The differences between settings in this study, 1 to 2 % in the Jaccard index, are of the same order as that spread and are not attributed to a parameter. + +### Source of the numbers + +The calibration tables come from `analyze_init_params.py` with the sweeps `ftinit_milp`, `prep_scale` and `ftinit_full`, and the robustness tables from `analyze_init_robustness.py` with the phases `gradient` and `levers`. The calibration runs differ from the sweep grids of the pinned script. The script runs the step-0 `mip_gap` and `force_on` sweeps and the scaling runs at `big_m=100.0`; the tables report `big_m=1000` for the `mip_gap` and `force_on` sweeps and both caps for the scaling runs. The script's step-0 `big_m` grid is 1000, 250, 100, 50 and 25, and its full-pipeline grid is the default `big_m` (1000) at `mip_gap` 0.001, 0.003 and 0.01 and `big_m` 50 and 250 at `mip_gap=0.003`, all on the scaled preparation. Outside these grids the calibration ran `big_m` 10 and 500 in the step-0 solve; `big_m` 500, 100 and 25 and the unscaled preparation in the full pipeline; `big_m=1000` in the scaling runs; and the repeats with `mip_gap_abs=None`. The result files of the runs are not in the repository. + +## Results + +### Step-0 solve with different `mip_gap` values + +`big_m=1000`, `force_on=0.1`. The reference is the tightest gap. | `mip_gap` | time (s) | objective | rel.obj.gap | Jaccard vs tightest | |--------:|---------:|----------:|------------:|--------------------:| @@ -24,9 +68,9 @@ The clean-data runs used Human-GEM commit `38567ab8` (v2.1.1 plus 15 commits), H | 0.03 | 25 | 44705 | +0.0272 | 0.9649 | | 0.1 | 24 | 44647 | +0.0284 | 0.9663 | -The solve time is flat across the gap at 24 to 25 s. `mip_gap=0.001` is within 0.0001 of the tightest objective, and the reaction set drops below Jaccard 0.97 from 0.03. A named series such as `1+1` ignores `mip_gap` and `mip_gap_abs`, because every step runs its own schedule of MILP runs; the setting acts on a step without a schedule, such as this step-0 solve. +### Step-0 solve with different `big_m` values -### ftINIT MILP `big_m` (`mip_gap=0.001`, `force_on=0.1`) +`mip_gap=0.001`, `force_on=0.1`. The last two columns compare each set with the run at `big_m=1000` and with the run at `big_m=100`. | `big_m` | time (s) | rel.obj.gap | Jaccard vs `big_m=1000` | Jaccard vs `big_m=100` | |------:|---------:|------------:|----------------------:|---------------------:| @@ -38,9 +82,9 @@ The solve time is flat across the gap at 24 to 25 s. `mip_gap=0.001` is within 0 | 25 | 25 | −0.0003 | 0.987 | 0.991 | | 10 | 23 | −0.0002 | 0.988 | 0.990 | -In the single step-0 solve every `big_m` from 10 to 1000 was feasible, took the same time and reached the same objective within 0.0003. The reaction sets differ by 1 to 2 % between any two values, which is the size of the difference between equivalent solutions. The caps that matter appear in the staged pipeline. +### Step-0 solve with different `force_on` values -### ftINIT MILP `force_on` (`mip_gap=0.001`, `big_m=1000`) +`mip_gap=0.001`, `big_m=1000`. The reference is the default 0.1. | `force_on` | time (s) | rel.obj.gap | Jaccard vs 0.1 | |---------:|---------:|------------:|---------------:| @@ -50,9 +94,9 @@ In the single step-0 solve every `big_m` from 10 to 1000 was feasible, took the | 0.2 | 25 | −0.0002 | 0.992 | | 0.5 | 25 | −0.0000 | 0.987 | -`force_on` (minimum flux for a reaction to count as "on") changes the model through the on/off constraint; the reaction set varied little (Jaccard ≥0.987) and the objective moved little. The default is `force_on=0.1`. +### Step-0 solve with different stoichiometric scaling -### Prep scaling by `rescaleModelForINIT` (`max_stoich_diff`, scaling on or off; `mip_gap=0.001`, step-0 solve) +Scaling runs at `mip_gap=0.001` and two `big_m` values. Every rel.obj.gap and Jaccard index in the table, including the `big_m=100` columns, is against the first row at `big_m=1000` (scale on, `max_stoich_diff=25`, abbreviated `msd`). | config | time (s), `big_m=1000` | rel.obj.gap | Jaccard vs scaled `msd=25`, `big_m=1000` | time (s), `big_m=100` | rel.obj.gap | Jaccard vs scaled `msd=25`, `big_m=1000` | |--------|---------:|------------:|-------------------------:|---------:|------------:|-------------------------:| @@ -62,9 +106,11 @@ In the single step-0 solve every `big_m` from 10 to 1000 was feasible, took the | `msd=100` | 26 | +0.0048 | 0.980 | 25 | +0.0049 | 0.980 | | scale off | 30 | −0.0050 | 0.984 | 25 | −0.0044 | 0.983 | -The reference row (scale on, msd=25, `big_m=1000`) comes from different runs than the same-named row of the `mip_gap` and `big_m` tables. The scale runs prepare the model unscaled and then call `rescale_for_init` with a 600 s limit; the step-0 runs use `prep(scale=True)` with a 900 s limit. The two runs reach objectives of 45539.6 and 45949.2 and share Jaccard 0.983. Within this table every Jaccard value is against the first row, including the `big_m=100` columns. +The reference row of this table and the `mip_gap=0.001` row of the step-0 `mip_gap` table have the same settings and come from different runs: the scaling run rescales an unscaled preparation with `rescale_for_init` and has a 600 s limit, the step-0 run uses `prep(scale=True)` and a 900 s limit. Their objectives are 45539.6 and 45949.2, and their reaction sets share a Jaccard index of 0.983. -The single step-0 solve was feasible with and without scaling at both `big_m` values, and `max_stoich_diff` 10 to 100 stayed within Jaccard 0.98 of `max_stoich_diff=25`. In the full staged pipeline, scaling and the cap interact: +### Full pipeline with and without scaling + +`time_limit=600`. | config (full pipeline, `time_limit=600`) | result | |-----------------------------------|--------| @@ -73,11 +119,11 @@ The single step-0 solve was feasible with and without scaling at both `big_m` va | scale off, `big_m=100` | infeasible (no usable solution after 30 s) | | scale on, `big_m=25` | infeasible (no usable solution after 30 s) | -With the default `big_m=1000` the pipeline is feasible without scaling. With a cap of 100 and no scaling, or a cap of 25 with scaling, the run stops after 30 s with `ftINIT MILP produced no usable solution (status: infeasible)`. The recommended setting is scaling on (`max_stoich_diff=25`, RAVEN's setting): the pipeline took 83 s with scaling and 157 s without it in these single runs. It is not needed for feasibility at `big_m=1000`. +An infeasible run stopped with `ftINIT MILP produced no usable solution (status: infeasible)`. -The calibration gives `big_m=1000`, `force_on=0.1` and scaling on (`max_stoich_diff=25`); in a step-0 solve `mip_gap=0.001` is within 0.0001 of the tightest objective. A named series caps each scheduled run by its own limit (runs of 120 s, 5000 s and 5 s); `time_limit` lowers that cap, and the full-pipeline runs here used `time_limit=600`. +### Full pipeline with different `big_m` values -### ftINIT full pipeline (`ftinit`, series='1+1', no-task scaled prep, `time_limit=600s`) +Scaled preparation without tasks, series `1+1`, `time_limit=600s`. | config | time (s) | `n_kept` | Jaccard vs `big_m=1000` | |--------|---------:|-------:|---------------------:| @@ -88,17 +134,11 @@ The calibration gives `big_m=1000`, `force_on=0.1` and scaling on (`max_stoich_d | `big_m=50` | 82 | 7325 | 0.982 | | `big_m=25` | 30 | infeasible | none | -The runs were started with different `mip_gap` values: 0.001, 0.003 and 0.01 at `big_m=1000`, 0.001 and 0.003 at `big_m=100`, and 0.003 at the other caps. Rows that differ in `mip_gap` or `mip_gap_abs` are repeats of one configuration, because a named series such as `1+1` ignores both: every step runs its own schedule of MILP runs. The three `big_m=1000` runs give the same 7312 reactions in 83, 83 and 84 s with the default `mip_gap_abs=10`, and in 85, 84 and 81 s with `mip_gap_abs=None`. The `mip_gap` conclusions of this study rest on the step-0 table. - -`big_m` from 50 to 1000 gave feasible runs of 81 to 119 s without a trend and reaction sets that differ by 2 % from the default. Several caps have repeats: `big_m=1000` has six runs (81 to 85 s), `big_m=250` two (117 s, and 114 s without `mip_gap_abs`), `big_m=100` three (117, 118 and 119 s) and `big_m=50` two (82 s and 81 s); `big_m=500` and `big_m=25` are single runs. Below 50 the staged pipeline can become infeasible; 25 was infeasible here. The default is `big_m=1000`, the cap that RAVEN's `ftINITInternalAlg` uses (`bigMCap`). Lower caps reduce the flux that a kept reaction can carry and showed no measured gain. - ---- - -## Robustness to degraded transcriptomics (task layer always on) +The runs were started with different `mip_gap` values, which the series ignores, so runs at one `big_m` are repeats of one configuration: 0.001, 0.003 and 0.01 at `big_m=1000`, 0.001 and 0.003 at `big_m=100`, and 0.003 at the other caps. Further repeats ran with `mip_gap_abs=None` in place of the default `mip_gap_abs=10`: at `big_m=1000` three runs of 85, 84 and 81 s, which kept the same 7312 reactions, and at `big_m=250` one run of 114 s. A third run at `big_m=100` took 119 s and a second run at `big_m=50` took 81 s; the record of the runs does not state their settings beyond the cap. In total `big_m=1000` has six runs (81 to 85 s), `big_m=250` two, `big_m=100` three (117, 118 and 119 s) and `big_m=50` two (82 and 81 s); `big_m=500` and `big_m=25` ran once. -Measured with `big_m=100` (scaled prep) on an earlier Human-GEM state than the clean-data runs, HCT116, with the task layer on, so its reaction counts and Jaccard values are not comparable with the no-task numbers of the clean-data calibration. +### Degraded expression input -The metabolic-task and gap-fill layer is held fixed; only the expression input is degraded. `frac` is the fraction of the essential tasks that the extracted model performs (`check_tasks`); `Jaccard` is the reaction-set overlap with the clean-data model. +Robustness runs with the task layer on. The clean row is the reference of the Jaccard column. | input | n_rxns | tasks pass | frac | Jaccard vs clean | |-------|-------:|-----------:|-----:|-----------------:| @@ -110,18 +150,11 @@ The metabolic-task and gap-fill layer is held fixed; only the expression input i | downsample 50 % | 6765 | 68/69 | 0.986 | 0.815 | | downsample 70 % | 6123 | 68/69 | 0.986 | 0.728 | -Dropout sets genes to 0 (score −5), noise multiplies levels by `exp(N(0,σ))`, and downsampling drops genes (they receive `no_gene_score`). +One model from 90 % dropout input, outside these tables, ran for more than 75 min in the gap-fill MILP. -Findings: +### Parameters at 70 % dropout -* Multiplicative expression noise changed the model little (Jaccard 0.92–0.95, size stable, all tasks pass). Sparsity changed it far more: 50 % dropout reduced the reaction set to 0.71 Jaccard (7777→5968 reactions) and 70 % to 0.59. -* Sparsity shrinks the model toward the task-essential core. Missing or zeroed genes remove the expression evidence for a reaction, and the task layer adds back only what tasks require, so sparse input yields smaller, more generic models. Dropout (−5) shrank the model more than downsampling (−2). -* With the task layer, `frac` stayed ≥0.97 and dipped to 67–68/69 under heavy sparsity: the bounded gap-fill plus the post-hoc low-score-gene pruning occasionally leave 1–2 essential tasks unsatisfied. The lever sweep at 70 % dropout below tests whether `no_gene_score` or `force_on` recovers these tasks. -* Run time tracks damage. Dropout runs are slowest (more broken tasks, more gap-fill); noise runs take little run time. - -The gap-fill MILP must be bounded (`mip_gap`/`time_limit`). Unbounded, severe degradation (which breaks many tasks at once) makes it solve a hard min-cost MILP per broken task to proven optimality, observed to run >75 min for one 90 %-dropout model. With the bound it returns a near-optimal fill quickly. - -### Levers at 70 % dropout +Lever runs on the 70 % dropout input. The default row is the dropout 70 % row of the previous table. | config | n_rxns | frac | Jaccard vs clean | |--------|-------:|-----:|-----------------:| @@ -130,14 +163,31 @@ The gap-fill MILP must be bounded (`mip_gap`/`time_limit`). Unbounded, severe de | `no_gene_score=−0.5` | 5128 | 0.986 | 0.593 | | `force_on=0.2` | 5159 | 0.986 | 0.600 | -No lever recovered the drift: Jaccard stayed ~0.59 across all settings, for two reasons. +## Effect of `mip_gap` on a single solve + +In the step-0 solve the time was 24 to 25 s at every gap from 0.0002 to 0.1. At `mip_gap=0.001` the objective was within 0.0001 of the tightest gap and the Jaccard index was 0.9976. The Jaccard index stayed at 0.99 or above up to `mip_gap=0.01` and fell to 0.9649 and 0.9663 at 0.03 and 0.1. In the full pipeline the gap has no effect: the three `big_m=1000` runs at 0.001, 0.003 and 0.01 kept the same 7312 reactions. + +## Effect of `big_m` and scaling + +In the step-0 solve every `big_m` from 10 to 1000 was feasible, took 23 to 26 s and reached the objective of `big_m=1000` within 0.0003. Against the run at `big_m=1000` and against the run at `big_m=100`, the reaction sets of the other caps had Jaccard indices of 0.983 to 0.991, of the same order as the seed-to-seed spread of the different set-up in [Set-up](#set-up). In the step-0 scaling runs, every `max_stoich_diff` from 10 to 100 and the unscaled model were feasible at both caps, with Jaccard indices of 0.980 to 0.991 against the scaled reference. + +The full pipeline separates the settings that the step-0 solve does not. At `big_m=1000` it was feasible with and without scaling: 83 s with scaling, 157 s without, and the two models shared a Jaccard index of 0.969 (7312 and 7264 reactions). At `big_m=100` it was feasible with scaling (117 to 119 s) and infeasible without, and at `big_m=25` it was infeasible with scaling. With scaling, the caps from 50 to 1000 gave runs of 81 to 119 s with no trend in `big_m` and reaction sets that shared a Jaccard index of 0.978 to 0.982 with the default. No cap below 1000 ran faster than the 81 to 85 s of the `big_m=1000` runs. The differences in the reaction set are of the same order as the seed-to-seed spread of the different set-up (about 1.5 % of the reactions) and are not smaller than it; with no seed repeats in this set-up, they are not attributed to the cap. A lower cap also reduces the flux that a kept reaction can carry. + +## Effect of `force_on` + +`force_on` changes the MILP itself, through the on and off constraints. From 0.02 to 0.5 the step-0 solve took 24 to 25 s, the objective moved by at most 0.0002 and the reaction set shared a Jaccard index of 0.987 or more with the default 0.1. The full pipeline was not run with other `force_on` values. + +## Effect of degraded input + +Noise changed the model less than sparse input. At σ=1.0 and σ=2.0 the model kept 7812 and 7768 reactions against 7777 for clean data, passed all 69 tasks and shared a Jaccard index of 0.952 and 0.919 with the clean model. Dropout of 50 % and 70 % reduced the model to 5968 and 5113 reactions (Jaccard 0.713 and 0.594), and downsampling of 50 % and 70 % to 6765 and 6123 reactions (Jaccard 0.815 and 0.728). At equal fractions dropout removed more reactions than downsampling. The difference matches the scores involved, −5 for a zero level against −2 for a reaction without measured genes, but this explanation was not tested. Under dropout and downsampling the model passed 67 or 68 of 69 tasks; the failing tasks and the cause of the failures were not examined. + +At 70 % dropout no tested lever moved the model back towards the clean model: the Jaccard index stayed at 0.593 to 0.600, the size at 5110 to 5159 reactions and the task fraction at 0.986. Under dropout the dropped genes stay in the data with a zero level and score −5, so `no_gene_score` acts only on reactions without gene rules and on reactions whose genes are all absent from the data. The study concludes that no parameter recovers the expression evidence that dropout removes; this is an inference from the four lever settings, which do not include other parameters. The lever runs did not include downsampled input, so the effect of `no_gene_score` on missing genes was not measured. -* The information that dropout destroys is gone. No scoring or connectivity setting reconstructs the missing expression evidence, and no parameter setting compensates for sparse input. -* `no_gene_score` governs only reactions whose genes are absent from the data, which is the downsampling failure mode. Dropout leaves genes present with a zero level (scored −5), so `no_gene_score` has nothing to act on under dropout. For missing-data sparsity, `no_gene_score` is a lever: a less-negative value keeps more unmeasured reactions and grows the model back toward clean. +In raven-toolbox commit `94f3940`, `ftinit` calls `fill_tasks` without a time limit, so the gap-fill uses the default of `fill_tasks`, 300 s per task (`_FILL_TIME_LIMIT`), and not the `time_limit` of the extraction. The time limit that the 90 % dropout run of more than 75 min used is not recorded. -The robustness levers that matter are structural and not numeric: the task and gap-fill layer (functional regardless of input quality) and a bounded gap-fill MILP (tractable). For missing-gene sparsity, `no_gene_score` trades model size against confidence. For noise, no parameter change is needed. No parameter restores fidelity lost to dropout; the loss is a property of the data and not of the pipeline. +## Choice of settings ---- +The calibration supports the defaults `big_m=1000`, `force_on=0.1` and scaling on with `max_stoich_diff=25` (RAVEN's setting). `big_m=1000` is the cap that RAVEN's `ftINITInternalAlg` uses (`bigMCap`). Scaling is not needed for feasibility at `big_m=1000`; in these single runs the pipeline took 83 s with scaling and 157 s without. A step-0 solve or a step without a schedule at `mip_gap=0.001` reaches an objective within 0.0001 of the tightest gap; a named series ignores the setting. The full-pipeline runs used `time_limit=600` and the robustness script uses `time_limit=120`; no full-pipeline run took longer than 157 s in total. For noisy input the tested settings needed no change, and no tested setting compensated for dropout. ## Gene-scoring constants @@ -147,21 +197,25 @@ Expression values become gene scores through score = clip(factor × ln(level / reference), min_score, max_score) ``` -with `factor=5.0`, `max_score=10.0`, `min_score=-5.0` on both sides. The logarithm is natural (base e) in `init/score.py` and in RAVEN's `scoreModel`. +with `factor=5.0`, `max_score=10.0` and `min_score=-5.0`, the same in both implementations. The logarithm is natural (base e) in `init/score.py` and in RAVEN's `scoreModel`. The source of each implementation cites no paper for the three constants. The one candidate reference that has been suggested, a 2012 paper that describes mCADRE, uses categorical scoring, which differs from this continuous log-ratio, so it is not their origin. The constants are RAVEN's own values without a literature source. -The constants are unattributed. The source of each implementation cites no paper for them, and the one candidate reference that has been suggested (a 2012 paper describing mCADRE) uses categorical scoring, which differs from this continuous log-ratio, so it is not the origin. They are RAVEN's own numbers with no literature source. Both implementations agree, so there is nothing to reconcile; the values rest on convention, with no evidence behind them. +## Comparison with the solver benchmark -## Cross-solver portability +[ftINIT on different solvers](init-solver-benchmark.md) runs the same full pipeline with each solver interface. With Gurobi it took 92 s and kept 7312 reactions, against 83 s and 7312 reactions here. The benchmark time also includes loading the prepared model and scoring the reactions, and its limit is 900 s against 600 s here. A limit shortens only the 5000 s runs of the schedule, and no Gurobi run in either study reached its limit. Its Gurobi run at `big_m=100` kept 7324 reactions, the count of the two `big_m=100` runs here, whose reaction set shares a Jaccard index of 0.981 with the `big_m=1000` set. The solver comparison itself, with GLPK at 2182 s and a Jaccard index of 0.892 against Gurobi and the failure of HiGHS through the `scipy` interface, is on that page. -The genome-scale solver comparison (Gurobi, HiGHS, GLPK) is in [init-solver-benchmark.md](init-solver-benchmark.md), and CI parameterised over installed MILP backends is in [tests/test_init_solvers.py](https://github.com/CompMetEng/raven-toolbox/blob/develop/tests/test_init_solvers.py). For Human-GEM HCT116 with one run per solver, Gurobi completes the pipeline in 92 s. The 83 s of the full-pipeline table times the `ftinit` call alone, while the solver timing also includes loading the prep pickle and scoring the reactions, and its cap is 900 s against 600 s, which affects the 5000 s runs of the schedule, while the 120 s and 5 s runs stay below both caps. Neither run reaches its limit and both keep 7312 reactions; GLPK completes in 2182 s with time-limited incumbents (Jaccard 0.892 against Gurobi); HiGHS through optlang's `scipy` interface fails because it handles continuous variables only. Toy-scale correctness is portable (Gurobi and GLPK give identical verdicts on the unit-test networks), so local development works without a Gurobi licence. +## Limits ---- +- One model (Human-GEM), one cell line (HCT116) and one solver (Gurobi 13.0.1). +- Most settings ran once; the repeats are listed with the full-pipeline table. This set-up has no seed repeats, and differences of the order of the seed-to-seed spread of a different set-up are not attributed to a setting. +- The robustness runs used a Human-GEM state whose commit is not recorded and `big_m=100`; the pinned script passes no `big_m`, so a rerun uses the default 1000. Each degradation is one random draw at one seed. +- The step-0 solve shows no difference between caps or scaling settings that the full pipeline shows, so step-0 results alone do not predict feasibility of the pipeline. +- The robustness tables have no run times, and the effect of `no_gene_score` on downsampled input was not measured. -## Reproducing +## Reproduction ```bash python scripts/analyze_init_params.py --cell HCT116 --sweeps ftinit_milp,prep_scale,ftinit_full python scripts/analyze_init_robustness.py --cell HCT116 --algo ftinit ``` -Both reuse the cached Human-GEM preps from the validation run ([docs/humangem-validation.md](humangem-validation.md)) and are resumable. +Both scripts read the reference model `raven_refModel.xml` from the working folder of [Human-GEM cell-type model validation against RAVEN](humangem-validation.md) (`--work`, default `~/hgem_compare`) and keep their results in a pickle file, so a rerun skips finished settings. `analyze_init_params.py` prepares the template itself; `analyze_init_robustness.py` loads the prepared template with tasks, `rg_prep_tasks.pkl`, from the same folder. The grids of the pinned `analyze_init_params.py` are smaller than the set of runs in the tables (see [Source of the numbers](#source-of-the-numbers)). diff --git a/docs/parameter-tuning/studies/init-solver-benchmark.md b/docs/parameter-tuning/studies/init-solver-benchmark.md index 6d6a30c..6fcb910 100644 --- a/docs/parameter-tuning/studies/init-solver-benchmark.md +++ b/docs/parameter-tuning/studies/init-solver-benchmark.md @@ -1,33 +1,48 @@ -# Cross-solver ftINIT benchmark on Human-GEM and HCT116 +# ftINIT on different solvers -The same `ftinit()` call (no-task scaled prep; series `1+1`, which runs its own schedule of MILP runs in each step; `time_limit=900s`) was run with each MILP-capable optlang interface that the environment offers. The environment was Human-GEM commit `38567ab8`, raven-toolbox commit `94f3940`, Gurobi 13.0.1, cobra 0.32.1, 4 cores, one run per row. The CI-scale companion is `tests/test_init_solvers.py`. +ftINIT builds mixed-integer linear programs (MILPs) and passes them to a solver through optlang. This benchmark runs ftINIT on Human-GEM with each solver interface that the environment offered, and records whether the run completed, how long it took and which reactions the extracted model contains. -## Per-solver result +## Set-up -| solver | big_m | time (s) | status | n_rxns | -|--------|------:|---------:|--------|-------:| -| gurobi | 1000 (default) | 92 | ok | 7312 | -| gurobi | 100 | 128 | ok | 7324 | -| glpk | 1000 (default) | 2182 | ok; step 1 and step 2 each stopped at the 900 s time limit with an unproven gap | 6758 | -| scipy (HiGHS, continuous only) | 1000 (default) | 1521 | FAIL: `ValueError: Scipy solver only works with continuous variables. Please use another interface` | 0 | -| hybrid (HiGHS) | 1000 (default) | 10 | FAIL: `SolverNotFound: hybrid is not a valid solver interface. Pick one from glpk_exact, glpk, gurobi, scipy` | 0 | +Every run calls `ftinit()` with the same input: Human-GEM prepared without a task list, reaction scores from HCT116 expression, series `1+1` and `time_limit=900` s. The call also passes `mip_gap=0.001`, which a named series ignores. Software: Human-GEM commit `38567ab8`, raven-toolbox commit `94f3940`, Gurobi 13.0.1 and cobra 0.32.1, on 4 cores. Each setting ran once. -The warnings in the run log number the steps from 0 (`step 0`, `step 1`); this page numbers them from 1. +The time is the wall-clock time of the whole benchmark step: reading the reference and the prepared model, scoring the reactions and calling `ftinit()`. A failed run reports the time until the failure. -The GLPK result shares 6635 reactions with the Gurobi result at `big_m=1000` (123 only in GLPK, 677 only in Gurobi, Jaccard 0.892). The Gurobi runs at the two `big_m` values differ in time (92 s and 128 s) and share Jaccard 0.98 (see [init-param-calibration.md](init-param-calibration.md)). +## Results -## Findings +| Solver interface | big_m | Time (s) | Outcome | Reactions in the extracted model | +|---|---:|---:|---|---:| +| gurobi | 1000 (default) | 92 | completed | 7312 | +| gurobi | 100 | 128 | completed | 7324 | +| glpk | 1000 (default) | 2182 | completed; steps 1 and 2 each stopped at the 900 s limit, gap not proven | 6758 | +| scipy | 1000 (default) | 1521 | failed: `ValueError: Scipy solver only works with continuous variables. Please use another interface` | none | +| hybrid | 1000 (default) | 10 | failed: `SolverNotFound: hybrid is not a valid solver interface. Pick one from glpk_exact, glpk, gurobi, scipy` | none | -* Gurobi completes ftINIT on Human-GEM in 92 s for 7312 reactions with the default per-reaction big-M (capped at 1000), and in 128 s with every cap at 100. The 83 s of the full-pipeline table in the [calibration](init-param-calibration.md) times the `ftinit` call alone, while the 92 s here also include loading the prep pickle and scoring the reactions; the 900 s cap against 600 s affects the 5000 s runs of the schedule, while the 120 s and 5 s runs stay below both caps. A separate `run_ftinit` call for the first step alone at `mip_gap=0.001` with a 900 s limit takes 24 s; that call is not a share of the pipeline. -* GLPK completes the pipeline in this run: its MIP honours `configuration.timeout`, and both main steps stopped at the 900 s limit with an incumbent whose gap is unproven. The run takes 2182 s and its reaction set differs from the Gurobi one by Jaccard 0.892, so it is a time-limited incumbent that does not reproduce the optimum. -* HiGHS is available through the `scipy` interface of the installed optlang, which handles continuous variables only, so the ftINIT MILP cannot be built with it (failure after 1521 s). The `hybrid` interface was attempted and failed after about 10 s with `SolverNotFound`; it is not among the interfaces the installed optlang offers. +The run log counts the steps from 0; the table counts them from 1. -## Implications for solver choice +## Gurobi -* Gurobi completed genome-scale ftINIT without a step reaching the time limit in this benchmark. It finished in 92 s. GLPK finished in 2182 s with both steps stopped at the time limit and a smaller reaction set (6758 against 7312, Jaccard 0.892), and the HiGHS interface (`scipy`) is not usable because it takes continuous variables only. -* Toy and unit-test correctness is portable. `tests/test_init_solvers.py` runs the toy ftINIT networks on every installed MILP backend; the formulation is solver-independent, so local development and CI work without a Gurobi licence. +Gurobi completed in 92 s with the default big-M, which caps each reaction at 1000, and in 128 s with every cap at 100. No step reached the 900 s limit. The calibration study reports 83 s for the same call with a limit of 600 s; it times `ftinit()` alone, while the 92 s here also include reading and scoring. The two Gurobi models were not compared reaction by reaction. In the calibration study, with a time limit of 600 s, the reaction sets at `big_m` 100 and 1000 share a Jaccard index of 0.981 ([ftINIT parameter calibration and input robustness](init-param-calibration.md)). -## Reproducing +## GLPK + +GLPK completed the pipeline in 2182 s. Its MILP solver honours `configuration.timeout`, and both main steps stopped at the 900 s limit with an incumbent whose gap is not proven. The extracted model has 6758 reactions against 7312 for Gurobi: 6635 are shared, 123 occur only in the GLPK model and 677 only in the Gurobi model (Jaccard index 0.892). The GLPK model is the best solution found within 900 s per step. The benchmark does not show whether a longer limit would bring it closer to the Gurobi model. + +## Interfaces that failed + +HiGHS is reached through the `scipy` interface of optlang, which accepts continuous variables only and cannot hold the ftINIT MILP. The run failed with the error in the table after 1521 s. The benchmark selected optlang's `hybrid_interface` for HiGHS, but cobra 0.32.1 does not accept `hybrid` as a solver name; it accepts `glpk_exact`, `glpk`, `gurobi` and `scipy`. The failure came after 10 s, the time spent reading the reference model before the solver name was set. + +## Choice of solver + +Gurobi is the only solver that completed genome-scale ftINIT without reaching the time limit: 92 s against 2182 s for GLPK, which ended both main steps at the limit. HiGHS through `scipy` cannot be used. + +Small networks do not need Gurobi. `tests/test_init_solvers.py` runs ftINIT on toy networks with every installed MILP solver, so development and continuous integration need no Gurobi licence. + +## Limits + +Each setting ran once, on one model and one cell line. In a different set-up (the DLD1 cell line, `big_m=100`, with tasks and gap-filling), two Gurobi builds with different seeds differ by 110 reactions at baseline ([ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](ftinit-determinism.md)), so a difference of that size between solvers does not by itself indicate a solver effect. The 800 reactions that differ between GLPK and Gurobi (123 + 677) are larger. The `scipy` and `hybrid` rows record failures and give no solver timings. + +## Reproduction ```bash # CI parameterised tests (seconds, runs always): @@ -38,4 +53,4 @@ python scripts/analyze_init_solvers.py --cell HCT116 \ --doc docs/parameter-tuning/studies/init-solver-benchmark.md ``` -Both commands reuse the cached Human-GEM no-task prep from the validation run ([humangem-validation.md](humangem-validation.md)) and are resumable per solver. +The first command checks the toy networks and runs in seconds. The second runs the genome-scale benchmark, which takes minutes to hours. It resumes per solver and reuses the prepared Human-GEM of [Human-GEM cell-type model validation against RAVEN](humangem-validation.md). The script writes its report to the file given by `--doc`; give another path to keep the page. diff --git a/docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md b/docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md index 244095a..6183bda 100644 --- a/docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md +++ b/docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md @@ -1,29 +1,42 @@ -# KEGG HMM-query cut-off calibration +# KEGG HMM cut-off calibration -The measurements behind the default KO-assignment parameters in `reconstruction/kegg/query.py` (`assign_kos` / `get_kegg_model_from_sequences`, pipeline step 3b.5) and IMPROVEMENTS K15 are the evidence for the default `cutoff=1e-30`, in place of RAVEN 2.11.3's `1e-50`. +KEGG-based reconstruction from sequence assigns each protein of a new organism to KEGG Orthology (KO) groups by searching the proteome against profile hidden Markov models (HMMs) of the KOs. The study ran this search on the proteomes of four organisms that KEGG annotates, assigned genes to KOs at a range of cut-offs and score ratios, and compared the result with KEGG's own gene→KO annotation of each organism. For each setting it records precision, recall and F1 of the gene→KO pairs, the fraction of the organism's annotated reactions that the prediction recovers and the number of predicted reactions outside the annotation. The results set the defaults `cutoff=1e-30` and `min_score_ratio_g=0.9` of `assign_kos` in place of the RAVEN 2.11.3 values `1e-50` and `0.8`. -## What the parameters do +## Set-up -`assign_kos` turns an `hmmsearch` KO×gene E-value matrix into gene→KO assignments in three steps: +### Parameters under test + +`assign_kos` (`reconstruction/kegg/query.py`, called by `get_kegg_model_from_sequences`) turns a KO×gene E-value matrix into gene→KO assignments in three steps. RAVEN's `getKEGGModelForOrganism` applies the same steps with the arguments `cutOff`, `minScoreRatioKO` and `minScoreRatioG`. 1. `cutoff` keeps hits with `evalue <= cutoff`. 2. `min_score_ratio_ko` drops, within a KO, genes whose `log(evalue)/log(best_evalue_in_KO) < min_score_ratio_ko`. 3. `min_score_ratio_g` drops, within a gene, KOs whose `log(evalue)/log(best_evalue_for_gene) < min_score_ratio_g`. -## Method +### Inputs + +- KEGG release 118. +- HMM libraries: `prokaryotes.hmm` (831 MB) and `eukaryotes.hmm` (692 MB), built by the maintainers with `build_ko_hmm`. The sequences of each KO were clustered with CD-HIT at 90 % identity and aligned with MAFFT FFT-NS-2, or with MAFFT PartTree for KOs whose alignment exceeds a memory budget. +- Queries: the full proteome of each organism, extracted from KEGG `genes.pep`, searched with `hmmscan`. `run_hmmsearch` in raven-toolbox fixes `hmmsearch -Z` to the number of profiles; with that setting its E-values match those of `hmmscan` against the same library. +- Each organism's hits were computed once, and every setting was applied to the same hit table. The differences between settings therefore contain no search-to-search variation, and the run-to-run spread of the search itself was not measured. +- Thread count and hardware of the sweep runs were not recorded. + +### Reference and scores + +The reference is the **KEGG annotation**: the organism's gene→KO pairs in the `organism_gene_ko` table of the KEGG artefacts. The table lists only KOs linked to at least one reaction in `ko_reaction`, and the HMM libraries are built from the same table. The organism's **true reactions** are the reactions linked to its annotated KOs. -- Data: KEGG release 118. Libraries: the maintainer-built `prokaryotes.hmm` (831 MB) and `eukaryotes.hmm` (692 MB), 90 %-clustered, FFT-NS-2 and PartTree (K12). -- Queries: each organism's full proteome, extracted from `genes.pep`. -- Ground truth: the organism's KEGG gene→KO links, from the `organism_gene_ko` table (restricted, as the table is, to reaction-linked KOs). -- Prediction: `assign_kos` output, with the `organism:` prefix stripped from query gene IDs so they match the bare gene IDs in the ground truth. -- Metrics at gene→KO level: precision = |pred ∩ truth| / |pred|, recall = |pred ∩ truth| / |truth|, F1. At reaction level, `rxn_rec` is the fraction of the organism's true reactions recovered (KO→reaction via `ko_reaction`) and `rxn_novel` is the number of predicted reactions that are not in the annotation set. -- Reproduce with [`scripts/analyze_hmm_cutoffs.py`](https://github.com/CompMetEng/raven-toolbox/blob/develop/scripts/analyze_hmm_cutoffs.py). +A predicted pair is the output of `assign_kos`, with the `organism:` prefix stripped from the gene identifier so that it matches the reference. Each predicted pair is either **matched** (present in the annotation) or **novel** (absent from it). -### Limits of the validation +| Score | Definition | +|---|---| +| gKO prec | gene→KO precision: matched predicted pairs divided by all predicted pairs | +| gKO rec | gene→KO recall: matched predicted pairs divided by all annotated pairs | +| gKO F1 | harmonic mean of gene→KO precision and recall | +| rxn rec | true reactions linked to a predicted KO, divided by all true reactions | +| rxn novel | number of reactions linked to a predicted KO that are not true reactions | -All four organisms are present in the training set of the libraries, so their own sequences hit their KO profiles strongly and recall is an upper bound. The calibration is therefore relative: it shows how the parameters trade off and where the RAVEN 2.11.3 default sits relative to the signal, and it gives no absolute accuracy estimate. A genome absent from KEGG would be the next validation. The values of `rxn_novel` and "precision < 1" partly reflect legitimate homology that KEGG never annotated for that organism (paralogs, un-curated genes) and are not pure error, so the precision figures are a lower bound on the true precision. +A novel pair counts as an error, although some novel pairs may be correct assignments that KEGG has not made. -## Organisms +### Organisms | code | organism | library | proteome (seqs) | true gene→KO pairs | true reactions | |---|---|---|---|---|---| @@ -32,11 +45,13 @@ All four organisms are present in the training set of the libraries, so their ow | `eco` | *Escherichia coli* K-12 MG1655 | prok | 4288 | 1071 | 1548 | | `mge` | *Mycoplasmoides genitalium* G37 (minimal genome) | prok | 476 | 110 | 211 | -`sce` and `eco` are model organisms; `cme` and `mge` are lesser-studied, and `mge` is additionally a small, divergent genome. +`sce` and `eco` are model organisms; `cme` and `mge` are less studied. `mge` has the smallest proteome of the four and the weakest matched hits in the E-value table. -## E-value separation +## Results -`log10(E-value)` percentiles of the best hit per (gene, KO) pair, split by whether the pair is in the organism's annotation (**matched**) or not (**novel**). A smaller (more negative) value is a stronger hit. +### E-value distributions + +The table gives percentiles of `log10(E-value)` of the best hit per (gene, KO) pair, for all hits that `hmmscan` reported, split into matched and novel pairs. A more negative value is a stronger hit. The 99th percentile is the weakest 1 % of the pairs in a group. The table does not show the strongest novel pairs. | organism | group | n | p50 | p90 | p95 | p99 | |---|---|---|---|---|---|---| @@ -49,13 +64,15 @@ All four organisms are present in the training set of the libraries, so their ow | `mge` | matched | 110 | −100 | −42 | −35 | −15 | | `mge` | novel | 1904 | −4 | −2 | −1 | −0 | -Matched pairs clustered at E ≈ 1e-100…1e-155, and even their weakest 1 % lay at 1e-15…1e-36. Novel pairs clustered at ≈1e-8. The two are separated by ~20 orders of magnitude. RAVEN 2.11.3's `1e-50` lies inside the matched tail (between the matched p95 and p99 for `sce` and `eco`, between p90 and p95 for `cme`, and between p50 and p90 for `mge`), so it discards annotated pairs that score weakly while gaining nothing against the (far weaker) noise. +The median matched pair had an E-value between 1e-100 (`mge`) and 1e-155 (`sce`), and the weakest 1 % of matched pairs lay between 1e-15 and 1e-36. The median novel pair lay between 1e-4 and 1e-8. The weakest 1 % of matched pairs was 11 (`mge`) to 29 (`eco`) orders of magnitude stronger than the median novel pair. + +The RAVEN 2.11.3 cut-off of `1e-50` falls inside the matched distribution: between p95 and p99 for `sce` and `eco`, between p90 and p95 for `cme`, and between p50 and p90 for `mge`. It therefore removes 1 to 5 % of the matched pairs of `sce` and `eco`, 5 to 10 % of `cme` and 10 to 50 % of `mge`. The default `1e-30` lies beyond p99 for `sce` and `eco` and between p95 and p99 for `cme` and `mge`, and is 22 to 26 orders of magnitude stronger than the median novel pair. -## Cut-off sweep +### Cut-off sweep -`min_score_ratio_ko = 0.3` and `min_score_ratio_g = 0.8` were fixed. The tables give gene→KO precision, recall and F1 and reaction recovery against the annotation. +`min_score_ratio_ko = 0.3` and `min_score_ratio_g = 0.8` (the RAVEN 2.11.3 values) were fixed while `cutoff` varied. -### `sce` +#### `sce` | cutoff | gKO prec | gKO rec | gKO F1 | rxn rec | rxn novel | |---|---|---|---|---|---| @@ -66,7 +83,7 @@ Matched pairs clustered at E ≈ 1e-100…1e-155, and even their weakest 1 % lay | 1e-70 | 0.81 | 0.91 | 0.86 | 0.91 | 113 | | 1e-100 | 0.84 | 0.76 | 0.80 | 0.79 | 68 | -### `cme` +#### `cme` | cutoff | gKO prec | gKO rec | gKO F1 | rxn rec | rxn novel | |---|---|---|---|---|---| @@ -77,7 +94,7 @@ Matched pairs clustered at E ≈ 1e-100…1e-155, and even their weakest 1 % lay | 1e-70 | 0.75 | 0.85 | 0.80 | 0.87 | 223 | | 1e-100 | 0.80 | 0.70 | 0.75 | 0.71 | 136 | -### `eco` +#### `eco` | cutoff | gKO prec | gKO rec | gKO F1 | rxn rec | rxn novel | |---|---|---|---|---|---| @@ -88,7 +105,7 @@ Matched pairs clustered at E ≈ 1e-100…1e-155, and even their weakest 1 % lay | 1e-70 | 0.73 | 0.88 | 0.80 | 0.93 | 157 | | 1e-100 | 0.82 | 0.74 | 0.77 | 0.80 | 96 | -### `mge` +#### `mge` | cutoff | gKO prec | gKO rec | gKO F1 | rxn rec | rxn novel | |---|---|---|---|---|---| @@ -99,9 +116,11 @@ Matched pairs clustered at E ≈ 1e-100…1e-155, and even their weakest 1 % lay | 1e-70 | 0.85 | 0.73 | 0.78 | 0.73 | 27 | | 1e-100 | 0.87 | 0.50 | 0.64 | 0.47 | 21 | -Recall was flat and high from 1e-10 to ~1e-30, then fell as the cut-off removed part of the matched tail, gently for model organisms and sharply for the divergent `mge` (rxn recall 0.98 → 0.87 from 1e-30 → 1e-50, → 0.47 at 1e-100). The recall lost to a stricter cut-off is not noise rejection (the noise lies at 1e-8); it is annotated pairs. `rxn_novel` shrank with stricter cut-offs because strong un-annotated homologs were also removed. +Gene→KO recall stayed between 0.95 and 0.99 from 1e-10 to 1e-30 in all four organisms and fell at stricter cut-offs. The fall was largest for `mge`, whose reaction recall dropped from 0.98 at 1e-30 to 0.87 at 1e-50 and 0.47 at 1e-100. Precision rose with every stricter cut-off in all four organisms, from 0.72 to 0.78 for `sce` between 1e-30 and 1e-50. With the ratios fixed at 0.3 and 0.8, F1 was highest at 1e-50 or 1e-70 in every organism. Novel predictions remained at the strictest cut-off: at 1e-100 the three larger proteomes had 68 to 136 novel reactions and a gene→KO precision of 0.80 to 0.84 (0.87 for `mge`). The study does not show whether these are wrong assignments, paralogs or genes that KEGG has not annotated. -## Score-ratio sweep (`cutoff = 1e-50`) +### Score-ratio sweep + +`cutoff` was fixed at `1e-50`. The script scores every combination of `min_score_ratio_ko` (0.0, 0.3, 0.5) and `min_score_ratio_g` (0.5, 0.8, 0.95); the table holds the five combinations per organism that were reported. | organism | ko ratio | g ratio | gKO prec | gKO rec | gKO F1 | |---|---|---|---|---|---| @@ -126,53 +145,77 @@ Recall was flat and high from 1e-10 to ~1e-30, then fell as the cut-off removed | `mge` | 0.3 | 0.80 | 0.77 | 0.84 | 0.80 | | `mge` | 0.5 | 0.80 | 0.78 | 0.84 | 0.81 | -- `min_score_ratio_ko` is inert here: across all four organisms, varying it 0.0 → 0.3 → 0.5 left recall unchanged in the rows shown and changed precision by 0.01 to 0.03 (0.03 for `sce` and `eco`, 0.01 for `cme` and `mge` at `min_score_ratio_g` = 0.80). The script output holds the full grid of 0.0, 0.3 and 0.5 against the g-values; the table shows representative rows. -- `min_score_ratio_g` is the precision lever: 0.80 → 0.95 raised precision by ~0.05–0.10 for a recall loss of ~0.01–0.03. The value 0.50 is too loose: it gave the lowest precision of the tested values. +At `min_score_ratio_g = 0.80`, raising `min_score_ratio_ko` from 0.0 to 0.5 left recall unchanged and raised precision by 0.03 for `sce` and `eco` and by 0.01 for `cme` and `mge`. -## Chosen defaults and effect +Raising `min_score_ratio_g` from 0.80 to 0.95 (at `min_score_ratio_ko = 0.0`) raised precision by 0.07 (`sce`), 0.09 (`cme`), 0.10 (`eco`) and 0.05 (`mge`), and lowered recall by 0.01 to 0.03. At 0.50, precision was the lowest of the three values in every organism. The value 0.9 lies between the tested values 0.8 and 0.95 and was not swept. -| parameter | RAVEN 2.11.3 | default | rationale | -|---|---|---|---| -| `cutoff` | 1e-50 | 1e-30 | recovers the matched tail (especially divergent genomes); remains ~22 orders above the 1e-8 noise floor | -| `min_score_ratio_g` | 0.8 | 0.9 | the precision lever; offsets the looser cut-off | -| `min_score_ratio_ko` | 0.3 | 0.3 | inert in the measurements; retained for RAVEN parity | +### Defaults compared with RAVEN 2.11.3 -The RAVEN 2.11.3 settings `(1e-50, 0.3, 0.8)` compared with the defaults `(1e-30, 0.3, 0.9)` (`min_score_ratio_ko` 0.3 ≡ 0.0 here): +The table compares the RAVEN 2.11.3 settings `(1e-50, 0.3, 0.8)` with the defaults `(1e-30, 0.3, 0.9)`, given as `(cutoff, min_score_ratio_ko, min_score_ratio_g)`. For the RAVEN 2.11.3 settings the F1 column gives the measured value from the cut-off sweep; for the defaults, which were not part of either sweep, it is computed from the rounded precision and recall in the same row. -| organism | gKO prec | gKO rec | rxn rec | rxn novel | -|---|---|---|---|---| -| `sce` | 0.78 → 0.76 | 0.95 → 0.96 | 0.96 → 0.96 | 157 → 137 | -| `cme` | 0.70 → 0.67 | 0.93 → 0.96 | 0.94 → 0.97 | 307 → 305 | -| `eco` | 0.67 → 0.67 | 0.95 → 0.97 | 0.98 → 0.99 | 198 → 173 | -| `mge` | 0.77 → 0.69 | 0.84 → 0.94 | 0.87 → 0.97 | 29 → 35 | +| organism | gKO prec | gKO rec | gKO F1 (measured → computed) | rxn rec | rxn novel | +|---|---|---|---|---|---| +| `sce` | 0.78 → 0.76 | 0.95 → 0.96 | 0.86 → 0.85 | 0.96 → 0.96 | 157 → 137 | +| `cme` | 0.70 → 0.67 | 0.93 → 0.96 | 0.80 → 0.79 | 0.94 → 0.97 | 307 → 305 | +| `eco` | 0.67 → 0.67 | 0.95 → 0.97 | 0.78 → 0.79 | 0.98 → 0.99 | 198 → 173 | +| `mge` | 0.77 → 0.69 | 0.84 → 0.94 | 0.80 → 0.80 | 0.87 → 0.97 | 29 → 35 | -The divergent minimal genome gained ~10 points of gene→KO recall (the case the sequence path exists for). Recall rose by 0.01 to 0.03 for `sce`, `cme` and `eco`. Gene→KO precision fell by 0.02 for `sce`, 0.03 for `cme` and 0.08 for `mge`, and stayed at 0.67 for `eco`. `rxn_novel` fell for `sce` (157 → 137), `cme` (307 → 305) and `eco` (198 → 173) and rose for `mge` (29 → 35). +With the defaults, gene→KO recall rose by 0.10 for `mge` and by 0.01 to 0.03 for the other three organisms. Gene→KO precision fell by 0.02 (`sce`), 0.03 (`cme`) and 0.08 (`mge`), and stayed at 0.67 for `eco`. F1 changed by at most 0.01. The number of novel reactions fell for `sce`, `cme` and `eco` and rose from 29 to 35 for `mge`. -## Unmeasured settings of the query +### Whole-model comparison of the two KEGG paths -Three settings next to the cut-offs were not swept; the reasons for their values do not need a sweep. +*S. cerevisiae* was reconstructed in full by two paths, both with the RAVEN 2.11.3 settings `(1e-50, 0.3, 0.8)`. The annotation path takes the organism's genes and KOs from the `organism_gene_ko` table; the HMM path assigns KOs to the proteome with the HMM search. -`threads` in `run_hmmsearch` and `build_ko_hmm` defaults to every CPU the process may run on. HMMER's Viterbi search is deterministic across thread counts; E-value estimation can differ in the last floating-point places, far below any cut-off used here. Single-threaded search against the full KO library (more than 26,000 HMMs) takes 30 to 60 minutes per proteome against roughly five multi-threaded. +| Count | annotation path | HMM path | +|---|---|---| +| reactions | 1355 | 1461 | +| metabolites | 1501 | 1567 | +| genes | 835 | 896 | -`seq_identity` in `build_ko_hmm` is `0.9`, the identity at which CD-HIT collapses near-duplicate sequences within a KO before the HMM is built, so that a KO with many near-identical entries does not produce an overfitted profile. It is CD-HIT's own recommended value for proteins. `seq_identity=-1` skips the clustering. This step is Python-only: RAVEN 3 dropped its local clustering path when it moved to prebuilt HMM libraries, so there is no MATLAB value to compare with. +The HMM path contained 96.3 % of the reactions of the annotation path (1305 of 1355, Jaccard index 0.86) and 96.6 % of its genes (807 of 835, Jaccard index 0.87). `hmmscan` took ≈ 0.1 s per query against either library on 12 threads (yeast: 6021 queries in 633 s, about 10.5 minutes). -The model-assembly flags decide what a draft keeps, and match MATLAB: +## Meaning of the results -| Flag | Default | What it admits | -|---|---|---| -| `keep_spontaneous` | `True` | Reactions marked spontaneous in KEGG, which carry no gene rule. Excluding them breaks real routes. | -| `keep_undefined_stoich` | `True` | Reactions with variable stoichiometry (`n` subunits). They cannot be mass-balanced as written, but dropping them loses real chemistry. | -| `keep_incomplete` | `True` | Reactions whose enzyme set is only partly known. | -| `keep_general` | `False` | Overview-map reactions that lump many specific steps into one. Admitting them double-counts. | +From 1e-10 to 1e-30 a stricter cut-off raised precision while gene→KO recall fell by at most 0.03 (from 0.98 or 0.99 to between 0.95 and 0.98). Beyond 1e-30 the cut-off traded recall for precision: it removed matched pairs together with novel pairs, and novel pairs remained at every cut-off tested. Across the whole sweep from 1e-10 to 1e-100 precision rose by 0.27 (`sce`, 0.57 to 0.84) to 0.35 (`mge`). The RAVEN 2.11.3 value `1e-50` removes 10 to 50 % of the matched pairs of `mge`, the organism with the weakest matched hits, and 1 to 10 % in the other three. -## Whole-model cross-validation +At `cutoff = 1e-50`, raising `min_score_ratio_g` from 0.80 to 0.95 raised precision by 0.05 to 0.10 for a recall loss of 0.01 to 0.03, and raising it from 0.50 to 0.80 raised precision by 0.15 to 0.27 (`eco`, 0.39 to 0.66). `min_score_ratio_ko` changed precision by 0.01 to 0.03 and recall not at all. The defaults move part of the filtering from the cut-off to `min_score_ratio_g`. On the four organisms the move gained recall, most for `mge`, at the cost of precision in three of them, and left F1 within 0.01 of the RAVEN 2.11.3 settings. -Full reconstruction of *S. cerevisiae* by two paths with the settings `(1e-50, 0.3, 0.8)`: +## Chosen defaults -| | annotation path (3b.4) | HMM path (3b.5) | -|---|---|---| -| reactions | 1355 | 1461 | -| metabolites | 1501 | 1567 | -| genes | 835 | 896 | +| parameter | RAVEN 2.11.3 | default (raven-toolbox and RAVEN 3) | basis | +|---|---|---|---| +| `cutoff` (`cutOff`) | 1e-50 | 1e-30 | beyond matched p99 for `sce` and `eco`, between p95 and p99 for `cme` and `mge`; with the new `min_score_ratio_g`, `mge` recall 0.84 → 0.94 | +| `min_score_ratio_g` (`minScoreRatioG`) | 0.8 | 0.9 | from 0.80 to 0.95 it raised precision by 0.05 to 0.10 for a recall loss of 0.01 to 0.03; partly offsets the precision lost to the looser cut-off | +| `min_score_ratio_ko` (`minScoreRatioKO`) | 0.3 | 0.3 | changed precision by 0.01 to 0.03 and recall not at all; kept at the RAVEN value | + +The defaults favour recall: the changed settings raised recall in every organism and lowered precision in three of the four. + +## Settings without a sweep + +Three settings next to the cut-offs were not measured in the study. The reasons given for their values are not tested. + +`threads` in `run_hmmsearch` and `build_ko_hmm` defaults to every CPU the process may run on. The value rests on the assumption, not tested here, that HMMER's search gives the same hits at any thread count and that E-values differ at most in the last floating-point places. Single-threaded search against the full KO library (more than 26,000 HMMs) is given as 30 to 60 minutes per proteome, against roughly five minutes multi-threaded; no run with a recorded thread count and hardware supports either figure. The one timed run, `sce` against `eukaryotes.hmm` on 12 threads, took 633 s (about 10.5 minutes). + +`seq_identity` in `build_ko_hmm` is `0.9`: the identity at which CD-HIT collapses near-duplicate sequences within a KO before the HMM is built; the clustering keeps a KO with many near-identical entries from giving an overfitted profile. It is CD-HIT's recommended value for proteins and the default of `seqIdentity` in RAVEN 2.11.3. `seq_identity=-1` skips the clustering. RAVEN 3's `getKEGGModelForOrganism` uses prebuilt HMM libraries and has no clustering step. + +The model-assembly flags decide which reactions a draft keeps, and have the same defaults as in RAVEN: + +| Flag | Default | What it admits | Reason given for the default | +|---|---|---|---| +| `keep_spontaneous` | `True` | Reactions marked spontaneous in KEGG, which carry no gene rule. | Excluding them breaks pathways that need them. | +| `keep_undefined_stoich` | `True` | Reactions with variable stoichiometry (`n` subunits), which cannot be mass-balanced as written. | Dropping them removes real chemistry from the draft. | +| `keep_incomplete` | `True` | Reactions whose enzyme set is only partly known. | | +| `keep_general` | `False` | Overview-map reactions that lump many specific steps into one. | Admitting them counts the same conversion twice. | + +## Limits + +- All four organisms are in the set of sequences from which the libraries were built, so their own sequences contribute to their KO profiles. Recall on these organisms is therefore an upper bound for an organism outside KEGG (inference; a genome absent from KEGG was not tested). The comparison between settings is relative and gives no absolute accuracy. +- A novel pair that is a correct assignment KEGG has not made counts as an error, so the precision figures are a lower bound. The study does not measure how many novel pairs are correct. +- Four organisms, one KEGG release (118) and one hit table per organism. Each search ran once, so its run-to-run spread, and its dependence on the thread count, were not measured; the scoring of a hit table by `assign_kos` involves no random step. +- The score-ratio sweep used `cutoff = 1e-50` only, and `min_score_ratio_g = 0.9` was not part of either sweep. + +## Reproduction + +The sweeps are produced by [`scripts/analyze_hmm_cutoffs.py`](https://github.com/CompMetEng/raven-toolbox/blob/develop/scripts/analyze_hmm_cutoffs.py), one organism per call, with the KEGG artefacts, the organism's proteome and the HMM library (or a saved `hmmscan --tblout` file) as input. It prints the E-value percentiles, the cut-off sweep and the full score-ratio grid. The comparison of the defaults with RAVEN 2.11.3 and the whole-model comparison are not produced by this script. -Reaction recall 96.3 % (1305/1355 shared, Jaccard 0.86); gene recall 96.6 % (807/835 shared, Jaccard 0.87). The annotation path also uses the `organism_gene_ko.tsv.gz` artefact (K14). `hmmscan` throughput was ≈ 0.1 s/query against either library on 12 threads (yeast: 6021 queries in 633 s). +With raven-toolbox at commit `94f3940` the script does not run: it imports `run_hmmscan` and `parse_hmmscan_tblout` from `reconstruction/kegg/query.py`, which provides `run_hmmsearch` and `parse_hmmsearch_tblout` instead. Its cut-off sweep calls `assign_kos` with the default ratios, which are 0.3 and 0.9 at that commit, so it no longer reproduces the sweep at 0.3 and 0.8. diff --git a/docs/parameter-tuning/studies/sampling-convergence-calibration.md b/docs/parameter-tuning/studies/sampling-convergence-calibration.md index cf748ba..099cd92 100644 --- a/docs/parameter-tuning/studies/sampling-convergence-calibration.md +++ b/docs/parameter-tuning/studies/sampling-convergence-calibration.md @@ -1,38 +1,53 @@ # Flux sampling convergence and settings -ACHR samples at genome scale were assessed in two ways, by autocorrelation within one chain and by R-hat between chains, and two further sampling settings were tested. +`random_sampling` in raven-toolbox draws flux vectors from a model with a Markov-chain sampler (`method='achr'` or `'chrr'`) or with random objectives (`method='random_objective'`). This study ran the two Markov-chain samplers on e_coli_core (95 reactions) and yeast-GEM, and recorded how strongly consecutive samples of one chain are correlated, whether independent chains reach the same flux distribution, and the wall time. It also records why two settings of the random-objective method differ from RAVEN MATLAB. -## Autocorrelation within one chain +## Terms -`thinning=k` takes k random-walk steps between each stored sample; a larger k costs more compute and gives more independent samples. On yeast-GEM (`n_samples=300`, `warmup=1000`, Gurobi): +- The **flux polytope** is the set of flux vectors `v` with `S v = 0` and `lb <= v <= ub`, where `S` is the stoichiometric matrix and `lb` and `ub` are the reaction bounds. A **vertex** is a corner point of the polytope. +- A **Markov chain Monte Carlo (MCMC)** sampler is a random walk inside the polytope whose visited points, after enough steps, are distributed close to uniformly over it. One walk from one starting point and one random seed is a **chain**. +- A **hit-and-run** step chooses a direction, finds the line segment through the current point along that direction that lies inside the polytope, and moves to a uniformly random point on that segment. +- **ACHR** (Artificially Centered Hit-and-Run) takes its directions from warmup points and a running centre of the chain, and does not transform the polytope. raven-toolbox wraps `cobra.sampling.ACHRSampler`, which computes its warmup points with two linear programs (LPs) per reaction before the first step ([Flux sampling](../../guide/sampling.md)). +- **CHRR** (Coordinate Hit-and-Run with Rounding) first runs a flux variability analysis (FVA), fixes the reactions whose range is narrower than `fixed_width_tol`, reduces the polytope to a full-dimensional body through the nullspace of the equality constraints, and **rounds** it: it maps the body so that its maximum-volume inscribed ellipsoid (**MVE**) becomes a unit ball. The walk then moves along one coordinate per step. +- **`thinning`** is the number of steps between two stored samples. A chain with `n_samples` stored samples therefore takes `thinning × n_samples` steps after its start. +- **`warmup`** is the number of CHRR steps discarded before the first stored sample. `random_sampling` passes it to CHRR only; the ACHR runs ignore it. +- The **lag-1 autocorrelation** `ρ` is the correlation between consecutive stored samples of one chain. +- The **effective sample size (ESS)** is the number of independent samples that carry the same information as the correlated ones. Under the assumption that the chain is a first-order autoregressive process, `ESS = n × (1−ρ) / (1+ρ)` for `n` stored samples. +- The **R-hat** of a reaction (Gelman-Rubin statistic) compares the flux variance between chains with the variance within chains. With `m` chains of `n` samples, `W` the mean within-chain variance and `B/n` the variance of the chain means, `R-hat = sqrt(((n−1)/n × W + B/n) / W)`. R-hat is 1.0 when the chains agree and grows when the distribution of a chain depends on where it started. The script reports the share of reactions above 1.1 and above 1.01, the stricter of the two thresholds. -| `thinning` | Lag-1 autocorrelation | Wall time (s) | -|---|---|---| -| 20 | 0.973 | 660 | -| 100 (default) | 0.926 | 841 | -| 500 | 0.849 | 927 | +## Set-up -At 500, consecutive samples were 85 % correlated. The decay is too slow for the setting to fix: a fivefold increase from 20 to 100 removed 0.047 of the autocorrelation, and the next fivefold removed 0.077. Reaching roughly 0.3 (a commonly used approximate threshold for near-independence) would need thinning in the tens of thousands, which is weeks of compute for 1000 samples. +Driver for the comparisons between chains: [`scripts/analyze_sampling_convergence.py`](https://github.com/CompMetEng/raven-toolbox/blob/docs/parameter-defaults/scripts/analyze_sampling_convergence.py). It starts `n_chains` calls of `random_sampling` with seeds 0 to `n_chains − 1`, one process per chain through `ProcessPoolExecutor`, because cobra models are not guaranteed to be safe for concurrent solves in threads. Each process reads the model from the SBML file. The script computes R-hat per reaction with the formula in [Terms](#terms). A reaction with the same constant flux in every chain has `W = 0` and no variance between the chain means, gets no R-hat, and is excluded as **not scored**. A reaction that is constant within each chain at different values in different chains has `W = 0` and an infinite R-hat, and is scored. -Under an AR(1) assumption for the chain, the effective sample size is `n × (1−ρ) / (1+ρ)`. At the default, ρ=0.926 gives about 12 independent samples from 300 stored. Collecting 100 would need roughly 2,600 stored samples, about 7.5 hours. +Settings: `n_samples=300`, `thinning=100`, `warmup=1000` and 4 chains, unless a table states otherwise. `thinning` and `warmup` are at their defaults; the default `n_samples` is 1000. The solver was Gurobi, on a machine with 12 cores. Software versions were not recorded. Each configuration ran once. cobrapy's `OptGPSampler` was not tested, because `random_sampling` does not call it; its methods are `'achr'`, `'chrr'` and `'random_objective'`. -The behaviour is expected: ACHR's mixing time scales with the dimension of the polytope, and yeast-GEM's 4,102 reactions are 15–20 times the models cobrapy calibrated `thinning=100` against. The default equals the upstream value; at genome scale the number of samples needed changes, and the right setting does not. +The models are two files of yeast-GEM. The single-chain runs were made on a yeast-GEM with 4,102 reactions, the same reaction count as the bundled `docs/data/yeast-GEM.xml` (yeastGEM_v9.1.0); which file they read was not recorded. The comparisons between chains read a yeast-GEM file with 4105 reactions, the count that the script reports; its version was not recorded. -## Convergence between chains +The wall time of a comparison between chains is the time from starting the processes to the return of the last chain: reading the model, the warmup and the sampling, without the R-hat computation. What the single-chain wall times include, and how `ρ` was combined over the reactions, was not recorded. -Autocorrelation measures independence within a chain. It gives no information on whether that chain reached every part of the flux polytope or mixed well inside one sub-region. Judging that needs several independent chains from different starting points and a check that they agree, the Gelman-Rubin R-hat diagnostic. +## Results -For each reaction, R-hat compares between-chain variance to within-chain variance across `n_chains` independent `random_sampling` runs (different seeds). R-hat ≈ 1.0 means the chains agree; R-hat > 1.1 (the common convergence threshold) or > 1.01 (the stricter one used in published MCMC work) flags a reaction whose distribution depends on where its chain started. +### Autocorrelation within one chain -* Driver: [`scripts/analyze_sampling_convergence.py`](https://github.com/CompMetEng/raven-toolbox/blob/docs/parameter-defaults/scripts/analyze_sampling_convergence.py) -* Settings matched to the single-chain study for direct comparison: `n_samples=300`, `thinning=100`, `warmup=1000` (`thinning` and `warmup` at their defaults; the default `n_samples` is 1000). -* `n_chains=4`, run in parallel via `ProcessPoolExecutor` (one process per chain; cobra models are not guaranteed thread-safe for concurrent solves). +One ACHR chain on yeast-GEM (4,102 reactions), `n_samples=300`, at three values of `thinning`. The ESS column is computed from the formula in [Terms](#terms). -## e_coli_core (95 reactions) as methodology validation +| `thinning` | Lag-1 autocorrelation | ESS of 300 stored samples | Wall time (s) | +|---|---|---|---| +| 20 | 0.973 | ~4 | 660 | +| 100 (default) | 0.926 | ~12 | 841 | +| 500 | 0.849 | ~25 | 927 | -4 chains × 300 samples, 34.7 s wall (parallel). +At `thinning=500`, consecutive samples were 85 % correlated. The fivefold step from 20 to 100 lowered `ρ` by 0.047, and the next fivefold step lowered it by 0.077. Reaching a `ρ` of about 0.3, an approximate level for near-independent samples, needs a `thinning` above the measured range, and three points do not establish how `ρ` decays beyond 500. -| | value | +`n_samples` was not varied. With `ρ` fixed at its value for the default `thinning=100`, the ESS formula is proportional to `n_samples`: ~12 from 300 stored samples, ~38 from the default 1000, and 100 needs ~2,600 stored samples. Scaled in proportion from 841 s for 300 samples, 2,600 samples take ~7,300 s (~2 h). The wall time rose from 660 s to 927 s (1.4-fold) while the number of steps rose 25-fold, so most of the time of these runs did not depend on `thinning`. If the wall time is a fixed cost per run plus a cost per stored sample, the scaled estimate is an upper bound for one chain; the thinning sweep kept `n_samples` at 300 and cannot separate the two parts. + +For the choice of settings: within the measured range, a larger `thinning` gave more ESS per second of wall time (~4 in 660 s at 20, ~25 in 927 s at 500). That a larger `n_samples` at a fixed `thinning` raises ESS in proportion and leaves `ρ` unchanged follows from the first-order autoregressive formula; it was not measured. + +### Agreement between chains on e_coli_core + +ACHR, 4 chains × 300 samples, 34.7 s wall time. + +| Measure | Value | |---|---:| | reactions scored (non-constant) | 87 / 95 | | R-hat median | 1.0071 | @@ -41,9 +56,9 @@ For each reaction, R-hat compares between-chain variance to within-chain varianc | reactions with R-hat > 1.01 | 42 (48.3 %) | | reactions with R-hat > 1.1 | 1 (1.1 %) | -Worst-converged reactions: +Reactions with the highest R-hat: -| reaction | R-hat | +| Reaction | R-hat | |---|---:| | `EX_succ_e` | 1.3028 | | `EX_co2_e` | 1.0420 | @@ -52,13 +67,15 @@ Worst-converged reactions: | `EX_h_e` | 1.0383 | | `TKT2`, `RPE`, `TKT1`, `TALA`, `G6PDH2r` | 1.0369 (tied) | -On a 95-reaction textbook model, one reaction (`EX_succ_e`, succinate exchange, a byproduct or overflow pathway) exceeded the "not converged" threshold (R-hat 1.30) at the default settings, and nearly half the reactions failed the stricter 1.01 threshold. This failure mode differs from the one the single-chain ESS result measured (autocorrelation within a chain): independent chains reached measurably different distributions for a subset of reactions. The result is consistent with a byproduct-secretion pathway that is rarely favoured and is explored only if the random walk of a chain reaches that corner of the polytope. +One reaction, the succinate exchange `EX_succ_e` (R-hat 1.30), exceeded 1.1, and 42 of 87 scored reactions exceeded 1.01. The chains reached different distributions for that reaction, a failure that the autocorrelation of one chain cannot detect. -## yeast-GEM (4105 reactions, genome scale) +For the choice of settings: on a 95-reaction model, the default `thinning` with 300 samples per chain gave agreement at the 1.1 threshold for 86 of 87 scored reactions. Agreement at 1.01 needs more steps or a different method (see [CHRR on e_coli_core](#chrr-on-e_coli_core)). -4 chains × 300 samples, 2524.4 s wall (~42 min). The wall time of four parallel chains is longer than the time of one chain (841 s) by a factor of ~3: the four Gurobi processes evidently contended for resources on a 12-core machine and did not scale linearly. +### Agreement between chains on yeast-GEM -| | value | +ACHR, 4 chains × 300 samples, 2524.4 s wall time (~42 min). + +| Measure | Value | |---|---:| | reactions scored (non-constant) | 3364 / 4105 | | R-hat median | 1.1671 | @@ -67,19 +84,37 @@ On a 95-reaction textbook model, one reaction (`EX_succ_e`, succinate exchange, | reactions with R-hat > 1.01 | 3246 (96.5 %) | | reactions with R-hat > 1.1 | 2271 (67.5 %) | -Worst-converged reactions: `r_0318` (9.94), `r_0307` (9.65), `r_1690` (8.89), `r_1077` (8.59), `r_2625` (5.92), `r_1072` (4.21), `r_4015` (3.88), `r_2690` (3.43), `r_1113` (3.42), `r_1648` (3.42). +Reactions with the highest R-hat: `r_0318` (9.94), `r_0307` (9.65), `r_1690` (8.89), `r_1077` (8.59), `r_2625` (5.92), `r_1072` (4.21), `r_4015` (3.88), `r_2690` (3.43), `r_1113` (3.42), `r_1648` (3.42). -This is a materially worse picture than e_coli_core suggested, and worse than the single-chain ESS finding implied on its own. At genome scale, with `thinning=100`, `warmup=1000` and `n_samples=300`, the median reaction already exceeded the 1.1 "not converged" threshold: independent chains disagreed on where the flux distribution of a typical reaction lies. Two in three reactions failed even the loose threshold; only 3.5 % passed the strict one. +The median scored reaction exceeded 1.1: two in three reactions failed that threshold, and 3.5 % passed 1.01. Each chain took 300 × 100 = 30,000 ACHR steps after its warmup points. The low ESS of one chain and the disagreement between chains are two separate measurements, and both show that these settings do not give a converged sample of yeast-GEM. -The result is consistent with the single-chain result (ESS≈12 effective samples from 300 stored at these settings). Low ESS shows that samples are highly autocorrelated within a chain; R-hat shows that, at genome scale, the chains often had not reached the same distribution within 300×100=30,000 total ACHR steps. The two are independent lines of evidence that point in the same direction. +The four parallel chains took 2524 s, ~3 times the 841 s of the single chain at the same settings. The two runs read different yeast-GEM files (4105 and 4,102 reactions). Contention between the four Gurobi processes on the 12 cores is a possible cause of the difference; it was not measured. -Per-reaction flux ranges reported by `random_sampling` at default settings on a genome-scale model should not be trusted as converged for the majority of reactions. The `random_sampling` docstring states these numbers and recommends raising `thinning` (at least 1000) or `n_samples`, `method='chrr'`, or an ESS or R-hat check. +For the choice of settings: at the default `thinning` and `n_samples=300`, the flux distributions of most reactions in a genome-scale model depend on the seed. The default `n_samples=1000` (100,000 steps per chain) was not tested. -## `method='chrr'` on e_coli_core +### Reallocating the ACHR step budget -Same 4 chains × 300 samples on e_coli_core, with `method='chrr'` and with `'achr'`: +The 30,000 steps per chain of the default run (`thinning=100, n_samples=300`), spread as `thinning=300, n_samples=100`. ACHR, 4 chains, yeast-GEM. -| | ACHR | CHRR | +| Measure | t=100, n=300 (default thinning) | t=300, n=100 (reallocated) | +|---|---:|---:| +| wall time | 2524.4 s | 3451.0 s | +| R-hat median | 1.1671 | 1.1636 | +| R-hat p90 | 1.6414 | 1.6542 | +| R-hat max | 9.9416 | 9.8345 | +| R-hat > 1.01 | 96.5 % | 95.2 % | +| R-hat > 1.1 | 67.5 % | 67.2 % | +| worst reaction | `r_0318` (9.94) | `r_0318` (9.83) | + +The reallocation moved the median R-hat from 1.1671 to 1.1636 and the shares above the two thresholds by at most 1.3 percentage points, the reaction with the highest R-hat was the same, and the run took 57.5 min against 42 min. That the disagreement comes from how ACHR moves through this polytope, and not from the split of the steps, is an inference from this result and from the CHRR result on e_coli_core; it was not tested at genome scale. + +For the choice of settings: at a fixed number of steps, moving steps from `n_samples` to `thinning` did not improve agreement between chains. A larger total number of steps was not tested with R-hat. + +### CHRR on e_coli_core + +4 chains × 300 samples on e_coli_core, with `method='achr'` and `method='chrr'`. + +| Measure | ACHR | CHRR | |---|---:|---:| | wall time | 34.7 s | 702.0 s | | reactions scored | 87 | 95 (0 excluded as constant) | @@ -89,52 +124,44 @@ Same 4 chains × 300 samples on e_coli_core, with `method='chrr'` and with `'ach | R-hat > 1.01 | 48.3 % | 17.9 % | | R-hat > 1.1 | 1.1 % | 0.0 % | -CHRR converged on e_coli_core: the worst reaction (`ICDHyr`, 1.0248) did not reach the 1.1 threshold, against ACHR's `EX_succ_e` at 1.30. R-hat max fell from 1.30 to 1.02, and the share above 1.01 from 48.3 % to 17.9 %. +With CHRR, no reaction exceeded 1.1; the highest R-hat was `ICDHyr` at 1.0248, against `EX_succ_e` at 1.30 with ACHR. The share above 1.01 fell from 48.3 % to 17.9 %. CHRR took ~20 times the wall time of ACHR. The percentages have different denominators, 87 and 95 scored reactions. The 8 reactions that ACHR left constant were scored with CHRR. CHRR computes every flux of a sample from the nullspace basis (`v0 + N x`); that rounding error in this product makes those 8 reactions vary is an inference that was not checked. -CHRR was ~20x slower on a 95-reaction model. CHRR's up-front max-volume-ellipsoid (MVE) rounding step is the likely driver, and MVE computation typically scales worse than linearly with dimension, so the cost at yeast-GEM's 4102 reactions (43x more reactions than e_coli_core) could plausibly be anywhere from more than 20x to much more, and not less. At ACHR's measured 2524 s for 4 genome-scale chains, a proportional 20x would be ~14 hours, which was not attempted blind. The bounded probe below gives the genome-scale measurement. +For the choice of settings: on a 95-reaction model, CHRR with the default `thinning` and 300 samples per chain gave agreement at 1.1 for every reaction. -## Reallocating the ACHR step budget +### CHRR at genome scale -The total step budget equals the default configuration (`thinning × n_samples` = 30,000 either way) and is distributed differently: `thinning=300, n_samples=100` instead of `thinning=100, n_samples=300`. 4 chains, yeast-GEM. +CHRR was not run on yeast-GEM with 4 chains × 300 samples, because 20 times the 2524 s of the ACHR run gives ~14 h. A probe with 2 chains × 20 samples (`thinning=100`) took 4815.3 s (~80 min), 1.9 times the 2524 s of the full ACHR run. yeast-GEM has 43 times the reactions of e_coli_core. -| | default (t=100, n=300) | reallocated (t=300, n=100) | -|---|---:|---:| -| wall time | 2524.4 s | 3451.0 s | -| R-hat median | 1.1671 | 1.1636 | -| R-hat p90 | 1.6414 | 1.6542 | -| R-hat max | 9.9416 | 9.8345 | -| R-hat > 1.01 | 96.5 % | 95.2 % | -| R-hat > 1.1 | 67.5 % | 67.2 % | -| worst reaction | `r_0318` (9.94) | `r_0318` (9.83) | +Each CHRR chain ran an FVA, a nullspace computation and the MVE rounding before its first step, and then 1,000 warmup steps and 2,000 sampling steps. The probe does not separate these parts. That the cost before the first step dominates, and that most of it is the MVE rounding, is an inference from the small number of steps; the parts were not timed. `random_sampling` computes this transform anew in every call, so each chain pays it. -R-hat changed little: the worst reactions were the same, the rough ordering was the same, the overall failure rate was the same, and the run took longer (57.5 min against 42 min) despite equal total steps. This rules out more thinning within a fixed budget as a fix. If more thinning improved mixing, spending the same budget on longer gaps between fewer stored samples should have moved R-hat, and it did not move it meaningfully in either direction. The non-convergence looks structural to ACHR's mixing on this polytope and not a matter of an existing setting, consistent with CHRR (a different algorithm) fixing it on e_coli_core while this reallocation, which remains ACHR, does not. +The R-hat of the probe (median 5.16, p90 86, max in the billions) does not show that CHRR converges worse than ACHR at genome scale. With 20 samples per chain, the within-chain variance `W` is estimated from too few samples to be stable: a reaction that shows almost no variance in 20 draws by chance gives a large R-hat whatever its distribution. The probe cannot distinguish a CHRR that does not converge on yeast-GEM from an R-hat that needs more samples; a run with enough samples for a stable R-hat has the cost problem of the full run. -## CHRR at genome scale +For the choice of settings: 20 samples per chain are too few for R-hat on yeast-GEM. Whether CHRR converges on yeast-GEM with 300 samples per chain is open. -CHRR was not run at matching scale (4 chains × 300 samples), because the ~20x e_coli_core cost multiplier implies perhaps 14 hours. A small bounded probe (2 chains, 20 samples, same `thinning=100`) was run to obtain a genome-scale CHRR timing number before deciding whether to attempt a full run. +## Choice of `thinning` and `n_samples` -cobrapy's `OptGPSampler` was not a candidate here: `random_sampling` does not wire it in; its methods are `'achr'`, `'chrr'` and `'random_objective'`. +The defaults stay `method='achr'`, `thinning=100` and `n_samples=1000` ([Tuned parameter defaults](../../tuned-parameters.md)). On a model the size of e_coli_core, the default `thinning` with 300 samples per chain gave chains that agree at R-hat 1.1 for all but one reaction. On yeast-GEM, no setting tested gave converged samples: a `thinning` up to 500 left `ρ` at 0.849, and moving the same 30,000 steps from `n_samples` to `thinning` moved the median R-hat from 1.1671 to 1.1636 and the shares above 1.01 and 1.1 by at most 1.3 percentage points. Flux ranges from `random_sampling` with the default settings on a genome-scale model are unconverged for most reactions. -The probe took 4815.3 s (~80 min) for 2 chains × 20 samples. This gives the timing independent of sample-count effects: CHRR's cost at genome scale is dominated by a fixed per-chain cost (almost certainly the max-volume-ellipsoid rounding step, computed once before any samples are drawn) that does not shrink with a smaller sample request. The 20-sample probe cost wall time comparable to a full 300-sample ACHR run (2524 s). With the current implementation CHRR is not a cheap drop-in genome-scale fix, whatever its converged quality would be at a matching sample count. +The `random_sampling` docstring gives these numbers and recommends, for genome-scale models, a `thinning` of at least 1000 or a larger `n_samples`, `method='chrr'`, or an ESS or R-hat check. A `thinning` of 1000 or more was not measured. The driver script runs the R-hat check: it samples several chains with different seeds at given settings and reports the share of reactions above the R-hat thresholds, so a sampling result can be checked at the settings that produced it. -The R-hat computed from this probe (median 5.16, p90 86, max in the billions) is not evidence that CHRR converges worse than ACHR at genome scale. With only 20 samples per chain, the within-chain variance (R-hat's denominator) is estimated from too little data to be stable. A reaction that happens to show near-zero variance in 20 draws by chance, combined with any between-chain difference, produces a large ratio (the maximum was in the billions) that reflects the small denominator and not the flux distribution. At e_coli_core, 300 samples per chain gave stable R-hat throughout. This run cannot distinguish "CHRR does not work at genome scale" from "R-hat needs more than 20 samples to mean anything"; telling those apart would need a genome-scale CHRR run with enough samples for a stable R-hat, which has the same cost problem. +One change outside the parameters was identified and not pursued: CHRR recomputes its rounding transform in every call. The chains run in separate processes at the same time, so to save that cost the transform would have to be computed once per model and passed to every chain. The saving depends on the share of the rounding in the 80 min of the probe, which was not measured. -## Summary +## Other sampling settings -- Default settings are unconverged for most reactions at genome scale, confirmed by two independent lines of evidence (ESS and R-hat). -- Reallocating the same ACHR budget did not help, which rules out the cheapest possible fix. -- CHRR fixed the problem on a small model, but its genome-scale cost (a fixed ~80 min+ per chain before any samples are even drawn) makes it impractical as a drop-in fix with the current implementation. Whether CHRR would converge at genome scale given enough samples to trust the R-hat is open; it would need a run long enough to be informative, which has the same cost problem. -- No cheap, validated fix was identified. Users doing genome-scale flux sampling with `random_sampling`'s defaults should treat per-reaction flux ranges as unconverged for most reactions. The finding concerns the median reaction and not a minority of reactions. One unblock was identified but not pursued here: CHRR's rounding transform is recomputed from scratch per chain and call, and caching or reusing it across calls on the same model would remove the dominant fixed cost. This is an engineering change, and no parameter default achieves it. +Two settings of `method='random_objective'` differ from RAVEN MATLAB's `randomSampling` on purpose. -## Other sampling settings +`replace_max_bound` (default `False`; MATLAB's `replaceBoundsWithInf` defaults to true) replaces the largest upper bound, the **big-M** value that stands for an unconstrained reaction, with infinity before sampling, and replaces the most negative lower bound, if it is below 0, with minus infinity. RAVEN-convention models use 1000 as the big-M for almost every reaction (4,083 of the 4,102 reactions of yeast-GEM). With every such bound replaced, the random-objective LP becomes unbounded, because the objective can grow without limit through any reaction that is then unconstrained. With `False`, 200 samples completed, with 0.57 % of them at the bound. -Both settings differ from MATLAB, and in both cases the difference is deliberate. +`loopless_good_reactions` (default `True`) selects which reactions are eligible as random objectives. MATLAB excludes every reaction whose FVA maximum reaches 0.999 times the largest bound (999 on a model capped at 1000), which takes proximity to the big-M as a sign of a thermodynamic loop. That rule also excludes reactions that reach the bound for metabolic reasons. A loopless FVA excludes only the reactions that take part in infeasible cycles, at a higher cost. -`replace_max_bound` swaps big-M upper bounds for infinity before sampling, and applies only to `method='random_objective'`. RAVEN-convention models use 1000 as the big-M for almost every reaction (4,083 of yeast-GEM's 4,102), so replacing them all makes the random-objective LP unbounded: the objective can be driven to infinity through any unconstrained reaction. At `False`, 200 samples complete, with 0.57 % of them at the bound. MATLAB's `True` suits models where only a few reactions reach the big-M and those represent real capacity limits, which is rare. +## Limits -`loopless_good_reactions` selects which reactions are eligible as random objectives. MATLAB excludes any reaction whose FVA maximum reaches 999, treating proximity to the big-M as evidence of a thermodynamic loop. That also excludes reactions that reach capacity for metabolic reasons. Loopless FVA identifies only reactions that participate in infeasible cycles, at higher cost. +- Each configuration ran once. The spread of R-hat and `ρ` between repeated runs with other seeds was not measured, so differences of the size seen in the reallocation (0.0035 in the median) are within an unknown run-to-run spread. +- One small and one genome-scale model were tested. The single-chain runs and the comparisons between chains used two different yeast-GEM files, and software versions were not recorded. +- R-hat is computed in its original form, without split chains or rank normalisation. +- The default `n_samples=1000`, a `thinning` above 500, and CHRR on yeast-GEM with more than 20 samples per chain were not tested. -## Reproducing +## Reproduction ```bash python scripts/analyze_sampling_convergence.py \ @@ -142,6 +169,4 @@ python scripts/analyze_sampling_convergence.py \ --n-chains 4 --n-samples 300 --thinning 100 --warmup 1000 ``` -Results are cached per (model, n_chains, n_samples, thinning, warmup) config, so re-running with the same settings returns the cached result without sampling; changing `--thinning` or `--n-samples` re-runs and caches separately, so the R-hat against thinning trade-off can be explored without re-deriving cached chains. - -Four parallel chains took 2524 s, about 3x the 841 s of the single-chain run at the same settings. The four concurrent Gurobi processes on a 12-core machine evidently contended for resources and did not scale linearly; planning further sweeps at this scale should allow for that. +The CHRR runs use the same command with `--method chrr`. The script caches the chains per model, method, `n_chains`, `n_samples`, `thinning` and `warmup` under `--out`; a run with a cached configuration computes R-hat from the stored chains without sampling. Another `--thinning` or `--n-samples` samples again and is cached separately. Four chains in parallel on 12 cores took ~3 times the wall time of one chain, so a sweep at genome scale needs more time than one chain per configuration suggests. diff --git a/docs/raven3-vs-raven-toolbox/same-function-different-answer.md b/docs/raven3-vs-raven-toolbox/same-function-different-answer.md index 2af7156..fa59ef6 100644 --- a/docs/raven3-vs-raven-toolbox/same-function-different-answer.md +++ b/docs/raven3-vs-raven-toolbox/same-function-different-answer.md @@ -31,7 +31,7 @@ The RAVEN functions `canExchange`, `checkProduction` and `getAllSubGraphs` find ftINIT extraction, gap-filling and compartment assignment solve mixed-integer problems that routinely have several optima of equal objective value. Two runs can return different reaction sets that are both correct. Results can differ between implementations, between solvers and between repeat runs of one solver. -The ftINIT reproducibility study measured raven-toolbox only (RAVEN's `ftINIT` was not run): runs of one solver also differ with the seed and the number of threads, and two repeated builds at one seed differed by 2 reactions when a step stopped at its wall-clock limit (see [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](../parameter-tuning/studies/ftinit-determinism.md)). No single exact result is expected. +The ftINIT reproducibility study measured raven-toolbox only (RAVEN's `ftINIT` was not run): runs of one solver also differ with the seed and the number of threads, and two repeated builds at one seed differed by 2 reactions; each of the two builds had one run of the final step that stopped at its 5 s wall-clock limit, the presumed and untested cause of the difference (see [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](../parameter-tuning/studies/ftinit-determinism.md)). No single exact result is expected. Compare these outputs by the overlap of their reaction sets; do not test them for identity. diff --git a/docs/tuned-parameters.md b/docs/tuned-parameters.md index ba4e527..7b2fa6d 100644 --- a/docs/tuned-parameters.md +++ b/docs/tuned-parameters.md @@ -15,8 +15,8 @@ RAVEN MATLAB: `randomSampling` (`method='achr'|'chrr'|'random_objective'`), `sam | Parameter | Default | How determined | |---|---|---| | `method` | `'achr'` (Python); `'achr'` (RAVEN 3, `randomSampling`) | Literature: ACHR gives near-uniform interior sampling, and the random-objective method draws polytope vertices. All three methods are selectable in both implementations (`'randomObjective'` in MATLAB, `'random_objective'` in Python). RAVEN 2.11.3 had only the random-objective method. | -| `thinning` | `100` | Upstream convention (cobrapy's own ACHR default, inherited by RAVEN's Python side). Measured insufficient at genome scale: yeast-GEM gives only ~12 effective samples from 300 stored (ESS), and independent chains frequently disagree entirely (median Gelman-Rubin R-hat fails the convergence threshold). No cheap fix found; see the [convergence study](parameter-tuning/studies/sampling-convergence-calibration.md). | -| `warmup` | `1000` | Upstream convention (cobrapy). | +| `thinning` | `100` | Upstream convention (cobrapy's own ACHR default). Measured insufficient at genome scale: on yeast-GEM one ACHR chain had an effective sample size (ESS) of about 12 from 300 stored samples, and 4 chains disagreed (median Gelman-Rubin R-hat 1.1671, 67.5 % of scored reactions above 1.1); `thinning` up to 500 did not fix it. See [Flux sampling convergence and settings](parameter-tuning/studies/sampling-convergence-calibration.md). | +| `warmup` | `1000` | Used by `method='chrr'` only (the steps discarded before the first stored sample); the ACHR sampler of cobrapy has no such parameter. Same as `nBurnin` of RAVEN's `sampleCHRR` and `randomSampling`, which also applies to CHRR only. Not measured. | | `n_objectives` | `2` | Literature: [Bordel et al. 2010](https://doi.org/10.1371/journal.pcbi.1000859). | | `replace_max_bound` | `False` | Measured: applying the alternative (`True`) inside the Python and cobrapy solver stack makes the sampler unbounded on standard RAVEN-convention models (±1000 bounds). Kept different between the implementations by design, because of a solver-stack constraint. Whether MATLAB's own solver path handles it safely was not tested here. | | `loopless_good_reactions` | `True` | Loopless FVA excludes only reactions that take part in infeasible cycles, at higher cost; the alternative fixed-threshold (±999) heuristic also excludes reactions that legitimately reach capacity. The loopless technique is implemented in Python only; porting it is a larger project than changing a default. | @@ -25,7 +25,7 @@ RAVEN MATLAB: `randomSampling` (`method='achr'|'chrr'|'random_objective'`), `sam | `maxiter` (in `max_volume_ellipsoid`) | `150` | Literature: [Zhang & Gao 2003](https://doi.org/10.1137/S1052623401397230). | | `tol`, `reg` (in `max_volume_ellipsoid`) | `1e-6`, `1e-8` | Same in both implementations. | -Full detail: [sampling convergence study](parameter-tuning/studies/sampling-convergence-calibration.md) (the primary measurement) +Full detail: [Flux sampling convergence and settings](parameter-tuning/studies/sampling-convergence-calibration.md) (the primary measurement) ## FSEOF @@ -38,7 +38,7 @@ RAVEN MATLAB: `FSEOF`. raven-toolbox: `fseof`. | `correlation_threshold` | `0.9` | Literature: [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10); measured: 0.7 adds 4 spurious amplification targets on iJO1366. | | `flux_eps` | `1e-6` | Measured on iJO1366: the classification uses the fixed thresholds `1e-10` and `1e-8`, as RAVEN does; `flux_eps` gates the check against the theoretical maximum, and the target counts are identical at `1e-8`, `1e-7` and `1e-6`. | -Full detail: [Parameter benchmarks](parameter-tuning/benchmarks.md) +Full detail: [Benchmarks of `fseof`, `remove_genes` and other defaults](parameter-tuning/benchmarks.md) ## INIT / ftINIT @@ -49,9 +49,9 @@ Only MATLAB has tINIT (`runINIT`, `getINITModel`); raven-toolbox implements ftIN | Parameter | Default | How determined | |---|---|---| | `prod_weight` | `0.5` | Literature: [Ågren et al. 2012](https://doi.org/10.1371/journal.pcbi.1002518) (the original INIT paper). | -| `allow_excretion` | `False` | Measured zero effect at the default `prod_weight` across all three INIT entry points that existed at the time of measurement; raven-toolbox has dropped the two tINIT ones. | -| `mip_gap` | `None` in Python (solver default); `0.0004` in MATLAB | Acts on the step-0 solve and on any step without a schedule of MILP runs; a named series such as `1+1` ignores it, because every step runs its own schedule. Measured on Human-GEM (HCT116) in the step-0 solve: the time is flat at 24 to 25 s, the Jaccard index to the 0.001 result is about 0.99 at 0.01, and the Jaccard index to the tightest result falls below 0.97 from 0.03. The Python default `None` needs no change. | -| `time_limit` | `None` in Python; `5000` seconds in MATLAB | RAVEN's `ftINIT` sets the Gurobi `TimeLimit` of a MILP run without its own limit to 5000 seconds, and the step schedules of `getINITSteps` use runs of 120 s, 5000 s and 5 s. In Python, `ftinit` with a named series caps each scheduled run by its own limit and `time_limit` lowers that cap further; `time_limit=None` leaves an ad-hoc step or a `run_ftinit` call uncapped, but no scheduled run. Measured on Human-GEM HCT116 (single runs): single-step solves took 23 to 30 s and full pipelines 82 to 118 s. In the [robustness setup](parameter-tuning/studies/init-param-calibration.md) (degraded input, task layer on), an unbounded gap-fill MILP on one 90 %-dropout model ran longer than 75 minutes. `None` is the default, and the working range for hard or noisy input is 120 to 600 s per step. | +| `allow_excretion` | `False` | Reported to have no effect at the default `prod_weight` in the three INIT entry points that existed at the time; no measurement is recorded on this site. raven-toolbox has dropped the two tINIT ones. | +| `mip_gap` | `None` in Python (solver default); `0.0004` in MATLAB | Acts on the step-0 solve and on any step without a schedule of MILP runs; a named series such as `1+1` ignores it, because every step runs its own schedule. Measured on Human-GEM (HCT116) in the step-0 solve: the time is flat at 24 to 25 s, the Jaccard index to the result at the tightest gap (0.0002) is 0.9976 at 0.001 and 0.9908 at 0.01, and falls below 0.97 from 0.03. The Python default `None` needs no change. | +| `time_limit` | `None` in Python; `5000` seconds in MATLAB | RAVEN's `ftINIT` sets the Gurobi `TimeLimit` of a MILP run without its own limit to 5000 seconds, and the step schedules of `getINITSteps` use runs of 120 s, 5000 s and 5 s. In Python, `ftinit` with a named series caps each scheduled run by its own limit and `time_limit` lowers that cap further; `time_limit=None` leaves an ad-hoc step or a `run_ftinit` call uncapped, but no scheduled run. The task gap-fill of `ftinit` is a separate MILP with a fixed limit of 300 s per task and does not read `time_limit`. Measured on Human-GEM HCT116 (single runs): step-0 solves took 23 to 30 s, and full pipelines 81 to 119 s with scaling and 157 s without. In the robustness runs of [ftINIT parameter calibration and input robustness](parameter-tuning/studies/init-param-calibration.md) (degraded input, with tasks), the gap-fill of one 90 %-dropout model ran longer than 75 minutes; the time limit of that run is not recorded. `None` is the default. The study ran its full pipelines at `time_limit=600` and its robustness runs at 120 s, and the working range for hard or noisy input is 120 to 600 s per step. | | `series` | `'1+1'` | Literature: [Gustafsson et al. 2023](https://doi.org/10.1073/pnas.2217868120) (the ftINIT paper). | | `force_on` | `0.1` | RAVEN's original value; measured near-insensitive across the range 0.02–0.5. | | `big_m` | `1000.0` | The cap on each scored reaction's own big-M, as in `ftINITInternalAlg`: the big-M of a reaction is `max(min(ub, big_m), min(-lb, big_m), 0)`, raised to `force_on` where the "on" floor needs it, and an infinite bound counts as `big_m`. The cap limits the flux a reaction can carry when it is on (it is the largest big-M of a scored reaction in its on/off constraint) and sets no bound on the reaction in the model. | @@ -60,7 +60,7 @@ Only MATLAB has tINIT (`runINIT`, `getINITModel`); raven-toolbox implements ftIN | `factor`, `max_score`, `min_score` (in `gene_scores_from_expression`) | `5.0`, `10.0`, `-5.0` | RAVEN's own formula. An attribution to "Wang et al. 2012" was checked and could not be confirmed; no source in either implementation cites a paper for it. | | `score_reactions_from_genes` and `classify_reactions` arguments | various | Same in both implementations or standard practice. | -Full detail: [INIT parameter calibration study](parameter-tuning/studies/init-param-calibration.md) (the primary measurement, genome-scale Human-GEM) · [INIT solver benchmark](parameter-tuning/studies/init-solver-benchmark.md) · [ftINIT reproducibility](parameter-tuning/studies/ftinit-determinism.md) (`resolve_ties=True` gives identical reaction sets at a fixed seed and a seed-to-seed difference of 41 reactions and 1 gene; adding `prove_abs_gap=1.0` reduces it to 1 reaction and 0 genes at about 26 times the run time) · [Human-GEM validation against MATLAB RAVEN](parameter-tuning/studies/humangem-validation.md) (Jaccard 0.964–0.988 against RAVEN across 5 cell lines) +Full detail: [ftINIT parameter calibration and input robustness](parameter-tuning/studies/init-param-calibration.md) (the primary measurement, genome-scale Human-GEM) · [ftINIT on different solvers](parameter-tuning/studies/init-solver-benchmark.md) · [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](parameter-tuning/studies/ftinit-determinism.md) (`resolve_ties=True` gives identical reaction sets at a fixed seed and a seed-to-seed difference of 41 reactions and 1 gene; adding `prove_abs_gap=1.0` reduces it to 1 reaction and 0 genes at about 26 times the run time) · [Human-GEM cell-type model validation against RAVEN](parameter-tuning/studies/humangem-validation.md) (Jaccard index against RAVEN across 5 cell lines of 0.964 to 0.982 for models extracted without tasks and 0.981 to 0.988 for task-constrained models) ## Gap-filling @@ -69,11 +69,11 @@ RAVEN MATLAB: `fillGaps` (connectivity mode), ported as `connect_blocked_reactio | Parameter | Default | How determined | |---|---|---| | `epsilon` (in `fill_gaps_fast_lp`) | `0.0001` | Literature: [Thiele et al. 2014](https://doi.org/10.1093/bioinformatics/btu321) (the fastGapFill paper). | -| `eps` (in `connect_blocked_reactions`) | `1.0` | Measured on a synthetic supply-limited model; a reliable noise filter for RAVEN-convention (±1000-bound) models; lower it for tightly constrained exchanges. | +| `threshold` (in `connect_blocked_reactions`) | `1.0` | The flux a reaction must be able to exceed to count as unblocked. Same as the fixed threshold of 1 in RAVEN's `fillGaps`, and suited to models in the RAVEN convention, where exchange reactions have bounds of ±1000. Not measured. | | `penalty`, `allow_net_production` (in `connect_blocked_reactions`) | `1.0`, `False` | Measured or standard practice for reconstruction use (as against curation use). | | `big_m` (in `fill_gaps_milp`) | `1000.0` | Matches RAVEN's ±1000 bound convention. It caps an unbounded template reaction only when the merged model has no finite bound; otherwise the largest finite bound does. | -Full detail: [Parameter benchmarks](parameter-tuning/benchmarks.md) +Full detail: [Benchmarks of `fseof`, `remove_genes` and other defaults](parameter-tuning/benchmarks.md) ## Homology-based reconstruction @@ -81,17 +81,17 @@ RAVEN MATLAB: `getModelFromHomology`, `getBlast`, `getDiamond`. raven-toolbox: ` | Parameter | Default | How determined | |---|---|---| -| `evalue` (`run_blast`) | `1e-4` | Matches MATLAB's `getBlast` (hardcodes `-evalue 10e-5`); measured to have no effect on the final model, because the filters of `get_model_from_homology` (`max_evalue` `1e-30`) dominate downstream. | +| `evalue` (`run_blast`) | `1e-4` | Matches MATLAB's `getBlast` (hardcodes `-evalue 10e-5`). On *H. polymorpha* against *S. cerevisiae*, loosening it from `1e-5` to `1e-4` added 5,204 raw hits (8.9 % of the table), all of which fail the `max_evalue` filter (`1e-30`) of `get_model_from_homology`. | | `evalue` (`run_diamond`) | `1e-3` | Matches MATLAB's `getDiamond`, which passes no `-evalue` flag and so inherits DIAMOND's own default. The value is deliberately different from `run_blast`; each aligner follows its own MATLAB call. | -| `threads` | every CPU the process may run on (Python); the cores assigned to MATLAB (MATLAB) | Measured deterministic across thread counts; the change affects performance only, with a speedup of ~1.9–4×. Both sides use all available cores. | -| `max_evalue` | `1e-30` | Measured inert from `1e-4` to `1e-50`; identity and alignment length exclude whatever a looser e-value would admit. | -| `min_align_len` | `100`; RAVEN 2.11.3 used `200` | Measured against independent KEGG and OMA ortholog references across a 4-organism relatedness series: recovers 3–4 points of recall over `200` at a cost of ≤0.6 points of precision. RAVEN 3's `getModelFromHomology` uses `minLen` 100 too. | -| `min_identity` | `40` | Same study: the binding filter and the measured optimum, with a wrongly transferred reaction weighted as worse than a missing one. | +| `threads` | every CPU the process may run on (Python); the cores assigned to MATLAB (MATLAB) | A 500-query BLAST subset returned the same 2,469 hits at one thread and at four, in 45.2 s and 23.7 s (1.9 times faster). Both sides use all available cores. | +| `max_evalue` | `1e-30` | The score was identical from `1e-4` to `1e-50` on the two organisms tested (at identity 40 % and length 200) and fell at `1e-100`. That identity and alignment length already exclude whatever a looser E-value would admit is an inference, not checked against the hit tables. | +| `min_align_len` | `100`; RAVEN 2.11.3 used `200` | Measured against the KEGG ortholog reference for 4 organisms of increasing distance from *S. cerevisiae*: compared with `200`, a length of 100 raised the score (F0.5) by 0.010 to 0.020, with 3 to 4 points more recall and at most 0.6 points less precision; 50 and 100 gave the same score. No OMA scores were reported for length. RAVEN 3's `getModelFromHomology` uses `minLen` 100 too. | +| `min_identity` | `40` | Same study: the setting with the largest effect on the score. With a wrongly transferred reaction weighted as worse than a missing one (F0.5), the optimum was 35 or 40 % against KEGG and 35 or 45 % against OMA, depending on the organism. | | `bidirectional` | `True` | Same in both implementations. | An arm of the study that scored against curated GEMs (hanpo-GEM, rhto-GEM) built by the same reconstruction method was retired because the result was circular: agreement with the curated model peaked at exactly the build settings of that model and collapsed one step past them. -Full detail: [homology cut-off calibration study](parameter-tuning/studies/homology-cutoff-calibration.md) (the primary measurement, including the retired curated-GEM arm) +Full detail: [Homology cut-off calibration](parameter-tuning/studies/homology-cutoff-calibration.md) (the primary measurement, including the retired curated-GEM arm) ## KEGG-based reconstruction @@ -99,14 +99,14 @@ RAVEN MATLAB: `getKEGGModelForOrganism` (the exact internal name of the KO-assig | Parameter | Default | How determined | |---|---|---| -| `cutoff` (in `assign_kos`) | `1e-30`; RAVEN 2.11.3 used `1e-50` | Measured against real KEGG gene→KO annotations across 4 organisms of varying study depth: the tighter value lies inside the tail of real matches (~20 orders of magnitude above the noise floor) and discards correct hits without gain. RAVEN 3's `getKEGGModelForOrganism` uses `cutOff` `10^-30` too. | -| `min_score_ratio_g` (in `assign_kos`) | `0.9`; RAVEN 2.11.3 used `0.8` | Same study: the precision lever with an effect (0.8→0.95 raises precision by ~0.07–0.10 at a cost of ~0.02 recall). RAVEN 3's `getKEGGModelForOrganism` uses `minScoreRatioG` `0.9` too. | -| `min_score_ratio_ko` (in `assign_kos`) | `0.3` | Same study: measured inert (varying it changes precision and recall by ≤0.02); kept at the original value, since it does nothing either way. | +| `cutoff` (in `assign_kos`) | `1e-30`; RAVEN 2.11.3 used `1e-50` | Measured against real KEGG gene→KO annotations across 4 organisms of varying study depth: `1e-50` lies inside the E-value distribution of the matched pairs and removes 1 to 10 % of them in three organisms and 10 to 50 % in *M. genitalium*. Precision rose with every stricter cut-off (*S. cerevisiae* 0.72 at `1e-30`, 0.78 at `1e-50`) at a cost in recall. Together with `min_score_ratio_g` 0.9, the default raised recall in all four organisms and kept F1 within 0.01. RAVEN 3's `getKEGGModelForOrganism` uses `cutOff` `10^-30` too. | +| `min_score_ratio_g` (in `assign_kos`) | `0.9`; RAVEN 2.11.3 used `0.8` | Same study: from 0.80 to 0.95 it raised precision by 0.05 to 0.10 at a recall loss of 0.01 to 0.03, and from 0.50 to 0.80 by 0.15 to 0.27; 0.9 itself was not swept. RAVEN 3's `getKEGGModelForOrganism` uses `minScoreRatioG` `0.9` too. | +| `min_score_ratio_ko` (in `assign_kos`) | `0.3` | Same study: from 0.0 to 0.5 it changed precision by 0.01 to 0.03 and recall not at all; kept at the RAVEN value. | | `seq_identity` (in `build_ko_hmm`) | `0.9` | Literature: the CD-HIT recommendation for protein clustering. | | `threads` (`run_hmmsearch`, `build_ko_hmm`) | every CPU the process may run on | Affects performance only; HMMER is deterministic across thread counts (to within insignificant float noise). | | `keep_spontaneous`, `keep_undefined_stoich`, `keep_incomplete`, `keep_general` (`get_kegg_model_*`) | `True`, `True`, `True`, `False` | Same in both implementations or standard draft-reconstruction practice. | -Full detail: [KEGG HMM cut-off calibration study](parameter-tuning/studies/kegg-hmm-cutoff-calibration.md) (the primary measurement, 4 organisms against real KEGG annotations) +Full detail: [KEGG HMM cut-off calibration](parameter-tuning/studies/kegg-hmm-cutoff-calibration.md) (the primary measurement, 4 organisms against real KEGG annotations) ## Sub-cellular localisation @@ -134,7 +134,7 @@ RAVEN MATLAB: `removeGenes`, `mergeModels`, `addRxns`, `addTransport`, `simplify | `match_by` (in `merge_models`) | `'name'` (Python) and `'metNames'` (MATLAB) | Same semantic field (metabolite display name) under a different schema field name; no values disagree. | | `remove_orphans`, `mets_by`, `ignore_direction`, `reversible`, `only_to_existing` | various | Python additions and refinements; no direct MATLAB equivalent exists for comparison. | -Full detail: [Parameter benchmarks](parameter-tuning/benchmarks.md) +Full detail: [Benchmarks of `fseof`, `remove_genes` and other defaults](parameter-tuning/benchmarks.md) ## Tasks @@ -144,7 +144,7 @@ RAVEN MATLAB: `checkTasks`. raven-toolbox: `check_tasks`, `find_task_essential_r |---|---|---| | `close_boundaries` (both functions) | `False` | Matches RAVEN's `checkTasks`, which leaves the model's own exchange reactions open; published task lists were curated against that. `True` makes the task the whole boundary, as `closeModel` does in RAVEN. | -Full detail: [Parameter benchmarks](parameter-tuning/benchmarks.md) +Full detail: [Benchmarks of `fseof`, `remove_genes` and other defaults](parameter-tuning/benchmarks.md) ## Cross-toolbox parity decisions @@ -155,25 +155,25 @@ Policy: where Python and MATLAB RAVEN disagree on a default, one value is chosen | Parameter | Python now | MATLAB now | Unify at | Side that changes | Confidence | Evidence | |---|---|---|---|---|---|---| | `run_blast.evalue` | `1e-4` ✓ done | `1e-4` | `1e-4` | ~~Python~~ done | High | No downstream effect measured either way (`get_model_from_homology`'s `1e-30` dominates); matches `develop`'s independent PR #91 call | -| `get_model_from_homology.min_align_len` | `100` ✓ done | `100` ✓ done (`200` in RAVEN 2.11.3) | `100` | ~~Python and MATLAB~~ done | High | Directly measured: [homology cut-off calibration study](parameter-tuning/studies/homology-cutoff-calibration.md) | -| `assign_kos.cutoff` | `1e-30` ✓ done | `1e-30` ✓ done (`1e-50` in RAVEN 2.11.3) | `1e-30` | ~~MATLAB~~ done | High | Directly measured: [KEGG HMM cut-off calibration study](parameter-tuning/studies/kegg-hmm-cutoff-calibration.md) | +| `get_model_from_homology.min_align_len` | `100` ✓ done | `100` ✓ done (`200` in RAVEN 2.11.3) | `100` | ~~Python and MATLAB~~ done | High | Directly measured: [Homology cut-off calibration](parameter-tuning/studies/homology-cutoff-calibration.md) | +| `assign_kos.cutoff` | `1e-30` ✓ done | `1e-30` ✓ done (`1e-50` in RAVEN 2.11.3) | `1e-30` | ~~MATLAB~~ done | High | Directly measured: [KEGG HMM cut-off calibration](parameter-tuning/studies/kegg-hmm-cutoff-calibration.md) | | `assign_kos.min_score_ratio_g` | `0.9` ✓ done | `0.9` ✓ done (`0.8` in RAVEN 2.11.3) | `0.9` | ~~MATLAB~~ done | High | Directly measured, same study | -| `get_init_model.allow_excretion` | `False` ✓ done | `False` | `False` | ~~Python~~ done (2026-06-20) | High | Zero effect at default `prod_weight`, pure inconsistency; see `parameter-tuning/benchmarks.md` | +| `run_ftinit.allow_excretion` | `False` ✓ done | `False` | `False` | ~~Python~~ done (2026-06-20) | High | Zero effect at default `prod_weight`, pure inconsistency; the measurement is not recorded on a page of [Parameter tuning](parameter-tuning/index.md). Within `ftinit`, the step schedule sets the value (`allow_met_secr`) | ### Settings resolved without unification -The INIT parameters `mip_gap` and `time_limit` of `run_init` and `run_ftinit` are resolved by the genome-scale test with real expression data in the [INIT parameter calibration study](parameter-tuning/studies/init-param-calibration.md) (Human-GEM HCT116 and Hart2015), without matching the MATLAB numbers: +The INIT parameters `mip_gap` and `time_limit` of `run_init` and `run_ftinit` are resolved by the genome-scale test with real expression data in [ftINIT parameter calibration and input robustness](parameter-tuning/studies/init-param-calibration.md) (Human-GEM HCT116 and Hart2015), without matching the MATLAB numbers: - `mip_gap`: solve time is flat across the gap in the single step-0 solve (model build dominates), so a tight gap is nearly free there. A named series such as `1+1` ignores `mip_gap` and `mip_gap_abs`, because every step runs its own schedule of MILP runs, so the full pipeline gives the same result at 0.001, 0.003 and 0.01. Neither 0.001 nor 0.01 is MATLAB's `0.0004`. Keep `None` (Gurobi's own `~1e-4` default is at least as tight as the measured-good `0.001`). -- `time_limit`: RAVEN's `ftINIT` limits a MILP run to 5000 seconds when the run sets none, and its schedules use 120 s, 5000 s and 5 s runs. Python's `ftinit` with a named series caps each scheduled run by the same limits, and `time_limit` lowers the cap; `time_limit=None` leaves an ad-hoc step or a `run_ftinit` call uncapped, but no scheduled run. Measured step-0 solves took 23 to 30 s and full pipelines 82 to 118 s. In the robustness setup, one unbounded gap-fill MILP on a 90 %-dropout model ran longer than 75 minutes. The working values of the study are `120-600 s/step`. Keep `None` as the code default (correct for clean-data and single-step use, where no run-away has been observed); the docstring states the working values. +- `time_limit`: RAVEN's `ftINIT` limits a MILP run to 5000 seconds when the run sets none, and its schedules use 120 s, 5000 s and 5 s runs. Python's `ftinit` with a named series caps each scheduled run by the same limits, and `time_limit` lowers the cap; `time_limit=None` leaves an ad-hoc step or a `run_ftinit` call uncapped, but no scheduled run. Measured step-0 solves took 23 to 30 s, and full pipelines 81 to 119 s with scaling and 157 s without. In the robustness setup, the gap-fill of one 90 %-dropout model ran longer than 75 minutes; the gap-fill has its own fixed limit of 300 s per task and does not read `time_limit`. The study ran its full pipelines at 600 s and its robustness runs at 120 s. Keep `None` as the code default; the docstring of `run_ftinit` states the working values, `120-600 s/step`. -The [INIT solver benchmark](parameter-tuning/studies/init-solver-benchmark.md) measured GLPK honouring the 900 s limit: both steps of the full pipeline hit it, the run took 2182 s and kept 6758 reactions (Jaccard 0.892 against Gurobi). The `scipy` interface (HiGHS) fails because it supports continuous variables only. Gurobi is the backend the genome-scale numbers above were measured on, and the only one that completed without a step hitting the limit. +[ftINIT on different solvers](parameter-tuning/studies/init-solver-benchmark.md) measured GLPK honouring the 900 s limit: both steps of the full pipeline hit it, the run took 2182 s and kept 6758 reactions (Jaccard 0.892 against Gurobi). The `scipy` interface (HiGHS) fails because it supports continuous variables only. Gurobi is the backend the genome-scale numbers above were measured on, and the only one that completed without a step hitting the limit. ### Values kept different | Parameter | Python | MATLAB | Why | |---|---|---|---| -| `random_sampling.replace_max_bound` | `False` | `True` | Applying MATLAB's `True` inside cobrapy and optlang makes the sampler unbounded on standard RAVEN-convention models (measured; see `sampling.md`). Whether MATLAB's own solver path handles `True` safely was not tested here; in both cases the difference is a solver-stack constraint. | +| `random_sampling.replace_max_bound` | `False` | `True` | Applying MATLAB's `True` inside cobrapy and optlang makes the sampler unbounded on standard RAVEN-convention models (measured; see [Flux sampling convergence and settings](parameter-tuning/studies/sampling-convergence-calibration.md)). Whether MATLAB's own solver path handles `True` safely was not tested here; in both cases the difference is a solver-stack constraint. | | `random_sampling.loopless_good_reactions` | `True` (loopless FVA) | heuristic (exclude FVA ≥ 999) | The two are different techniques. Loopless FVA excludes only reactions that take part in infeasible cycles, and the MATLAB heuristic also excludes reactions that legitimately reach capacity. Porting the loopless technique to MATLAB is a larger project than changing a default. | | `merge_models.match_by` | `'name'` | `'metNames'` | Same semantic field (metabolite display name) under a different field name, which results from the cobrapy and COBRA Toolbox schemas; the field name is not tunable. | | `run_diamond.evalue` | `1e-3` | `1e-3` (implicit) | Both agree at `1e-3`, matching the DIAMOND default. `getDiamond` passes no `-evalue` flag, so it inherits the DIAMOND default and does not follow the explicit `1e-4` of `getBlast`. `run_blast` and `run_diamond` deliberately stay unpaired: each follows the MATLAB call of its own aligner. | @@ -194,7 +194,7 @@ The time-budget parameter of the localisation needs no port: `predictLocalizatio ## Open items -- Genome-scale ACHR sampling convergence has no cheap fix. At genome scale (yeast-GEM, default ACHR settings) the median reaction fails the R-hat>1.1 threshold. Neither follow-up fixes it cheaply: reallocating the same step budget onto a larger `thinning` makes no measurable difference, and `method='chrr'` converges on a small model, but its genome-scale cost (a fixed per-chain rounding step alone costing ~80 min in a minimal probe) makes it impractical as a drop-in fix at the time of the study. The [sampling convergence calibration study](parameter-tuning/studies/sampling-convergence-calibration.md) describes the investigation, including a possible future unblock (caching the CHRR rounding transform across calls, an engineering change and not a parameter default, so out of scope here). +- Genome-scale ACHR sampling convergence has no cheap fix. At genome scale (yeast-GEM, default ACHR settings) the median reaction fails the R-hat>1.1 threshold. Neither follow-up fixes it cheaply: reallocating the same step budget onto a larger `thinning` makes no measurable difference, and `method='chrr'` converges on a small model, but its genome-scale cost makes it impractical as a drop-in fix: a probe of 2 chains × 20 samples on yeast-GEM took 80 min in total, and the share of the rounding step in that time was not measured. [Flux sampling convergence and settings](parameter-tuning/studies/sampling-convergence-calibration.md) describes the investigation, including a possible future unblock (caching the CHRR rounding transform across calls, an engineering change and not a parameter default, so out of scope here). - The `gene_scores_from_expression` scoring constants (`factor=5.0`, `max_score=10.0`, `min_score=-5.0`) have no confirmed literature source on either side; see the INIT section. ## References diff --git a/scripts/known_names.txt b/scripts/known_names.txt index 7d649d0..5301bff 100644 --- a/scripts/known_names.txt +++ b/scripts/known_names.txt @@ -424,3 +424,14 @@ targetTotal # scaleBiomassPseudoreaction argument target_total # scale_biomass argument overwriteWithEmpty # curateModelFromTables argument new_met_prefix # change_reaction_equations argument + +# -- names of runs, options and arguments in the parameter-tuning studies -------- +ftinit_milp # run type of analyze_init_params.py +ftinit_full # run type of analyze_init_params.py +prep_scale # run type of analyze_init_params.py +fixed_width_tol # option of analyze_sampling_convergence.py +replaceBoundsWithInf # randomSampling argument +nBurnin # sampleCHRR and randomSampling argument +allow_met_secr # argument of the ftinit step schedule +run_hmmscan # imported by analyze_hmm_cutoffs.py; not in query.py +parse_hmmscan_tblout # imported by analyze_hmm_cutoffs.py; not in query.py From 019e4342a00f8646032edb03868d1a414b81c737 Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Sat, 31 Oct 2026 23:25:01 +0100 Subject: [PATCH 04/20] Correct the parameter-tuning pages after review Each statement of cause is marked as measured or inferred, derived numbers state their arithmetic and the runs they come from, and tuned-parameters.md, the guide pages and the index agree with the studies. --- docs/guide/homology.md | 2 +- docs/guide/init.md | 2 +- docs/guide/kegg.md | 4 +-- docs/parameter-tuning/benchmarks.md | 2 +- docs/parameter-tuning/index.md | 2 +- .../studies/ftinit-determinism.md | 18 ++++++----- .../studies/homology-cutoff-calibration.md | 4 +-- .../studies/humangem-validation.md | 10 +++--- .../studies/init-param-calibration.md | 12 +++---- .../studies/init-solver-benchmark.md | 8 ++--- .../studies/kegg-hmm-cutoff-calibration.md | 12 +++---- .../sampling-convergence-calibration.md | 10 +++--- docs/tuned-parameters.md | 32 +++++++++---------- 13 files changed, 60 insertions(+), 58 deletions(-) diff --git a/docs/guide/homology.md b/docs/guide/homology.md index d3aad75..da720a0 100644 --- a/docs/guide/homology.md +++ b/docs/guide/homology.md @@ -159,7 +159,7 @@ Three cut-offs control which hits are accepted, and tightening any of them shrin `maxE` / `max_evalue` (`1e-30`) is the maximum BLAST E-value, the number of hits of at least this quality expected by chance in a database this size. The default is strict by BLAST standards, where `1e-5` is a common threshold, because transferring a reaction on a marginal hit adds a claim about metabolism that nothing downstream questions. -`minLen` / `min_align_len` (`100`) is the minimum aligned length in residues. It rejects hits that align well over a short conserved domain while the rest of the protein is unrelated. The value was measured against KEGG orthology assignments across four organisms ([Homology cut-off calibration](../parameter-tuning/studies/homology-cutoff-calibration.md)): 50 and 100 gave the same score, 150 a score at most 0.003 lower, and 200 and 300 lower scores, mainly through lost recall. +`minLen` / `min_align_len` (`100`) is the minimum aligned length in residues. It rejects hits that align well over a short conserved domain while the rest of the protein is unrelated. The value was measured against KEGG orthology assignments across four organisms ([Homology cut-off calibration](../parameter-tuning/studies/homology-cutoff-calibration.md)): 50 and 100 gave the same score, 150 a score at most 0.003 lower, and 200 and 300 lower scores. At 200 the loss came mainly from recall; precision and recall were not reported for 300. `minIde` / `min_identity` (`40`) is the minimum percentage identity across the alignment. diff --git a/docs/guide/init.md b/docs/guide/init.md index ac878a3..bce80e9 100644 --- a/docs/guide/init.md +++ b/docs/guide/init.md @@ -71,7 +71,7 @@ Both extractions are mixed-integer problems. On Human-GEM, GLPK completed a benc ## Template preparation -The preparation finds the task-essential reactions, classifies every reaction into omics-independent categories, merges linear stretches, and rescales the stoichiometry to a narrower coefficient range, which is not needed for feasibility at the default `big_m` and shortens the run (83 s against 157 s in [ftINIT parameter calibration and input robustness](../parameter-tuning/studies/init-param-calibration.md)). It depends only on the template and the task list, and not on the expression data, so it is done once and reused for every sample. +The preparation finds the task-essential reactions, classifies every reaction into omics-independent categories, merges linear stretches, and rescales the stoichiometry to a narrower coefficient range, which is not needed for feasibility at the default `big_m`. In the single runs of [ftINIT parameter calibration and input robustness](../parameter-tuning/studies/init-param-calibration.md), the rescaling shortened the run (83 s against 157 s). It depends only on the template and the task list, and not on the expression data, so it is done once and reused for every sample. ::::{tab-set} :::{tab-item} Ⓜ️ MATLAB diff --git a/docs/guide/kegg.md b/docs/guide/kegg.md index 3c30914..8da87d8 100644 --- a/docs/guide/kegg.md +++ b/docs/guide/kegg.md @@ -81,11 +81,11 @@ hanpo_model = get_kegg_model_from_sequences_with_artefacts("hanpo.faa", domain=" ::: :::: -This method is expensive. The eukaryotic HMM library alone is 129 MB compressed, and one `hmmsearch` of a full proteome against all KO profiles takes tens of minutes to hours. It also needs HMMER, which has no bundle for every platform; [Download data and binaries](../installation/data-and-binaries.md) has the matrix. Without `threads`, `run_hmmsearch`, `get_kegg_model_from_sequences`, `build_ko_hmm` and `build_hmm_library` use every CPU the process may run on; `getKEGGModelForOrganism` runs `hmmsearch` on the cores assigned to MATLAB (`feature('numcores')`) and has no `threads` option. +This method is expensive. The eukaryotic HMM library alone is 129 MB compressed, and one timed `hmmsearch` of a proteome (*S. cerevisiae*, 6021 queries) against the eukaryotic library took about 10.5 minutes on 12 threads. It also needs HMMER, which has no bundle for every platform; [Download data and binaries](../installation/data-and-binaries.md) has the matrix. Without `threads`, `run_hmmsearch`, `get_kegg_model_from_sequences`, `build_ko_hmm` and `build_hmm_library` use every CPU the process may run on; `getKEGGModelForOrganism` runs `hmmsearch` on the cores assigned to MATLAB (`feature('numcores')`) and has no `threads` option. With a `fastaFile`, the organism id only names the model; the search uses no phylogenetic distance and ignores `maxPhylDist`. -Three cut-offs shape the assignment. `cutOff` is the HMMER significance a hit needs at all. `minScoreRatioKO` drops genes that fit a KO far worse than the best gene in it, and `minScoreRatioG` drops KOs that fit a gene far worse than that gene's best KO; the two prune the same matrix from opposite sides. All three were measured in [KEGG HMM cut-off calibration](../parameter-tuning/studies/kegg-hmm-cutoff-calibration.md); `cutOff` and `minScoreRatioG` changed from the RAVEN 2 values. Raising `minScoreRatioKO` from 0.0 to 0.5 changed precision by 0.01 to 0.03 and recall not at all, and it keeps its original value of 0.3. +Three cut-offs shape the assignment. `cutOff` is the HMMER significance a hit needs at all. `minScoreRatioKO` drops genes that fit a KO far worse than the best gene in it, and `minScoreRatioG` drops KOs that fit a gene far worse than that gene's best KO; the two prune the same matrix from opposite sides. All three were measured in [KEGG HMM cut-off calibration](../parameter-tuning/studies/kegg-hmm-cutoff-calibration.md); `cutOff` and `minScoreRatioG` changed from the RAVEN 2 values. At `minScoreRatioG` 0.8 and `cutOff` 1e-50, raising `minScoreRatioKO` from 0.0 to 0.5 changed precision by 0.01 to 0.03 and recall not at all, and it keeps its original value of 0.3. ## Contents of a KEGG draft diff --git a/docs/parameter-tuning/benchmarks.md b/docs/parameter-tuning/benchmarks.md index e6620db..fcc5eb1 100644 --- a/docs/parameter-tuning/benchmarks.md +++ b/docs/parameter-tuning/benchmarks.md @@ -59,7 +59,7 @@ The three `fseof` rows date from 2026-06-20 and were not repeated with commit `9 ## Reactions without an enzyme in `remove_genes` -`blocked_reactions` in `remove_genes`, and `removeBlockedRxns` in RAVEN's `removeGenes`, decide what happens to a reaction whose gene rule is emptied by a deletion. By default both keep such a reaction with bounds of zero, and the two defaults gave the same model. After deletion of `b1779`, that model predicts no growth, because GAPD stays in the network with no capacity to carry flux. `"remove"` and `removeBlockedRxns` true also predicted no growth; they delete GAPD, which changes the number of reactions and the set of reaction identifiers. `"keep"` leaves GAPD with an empty gene rule and its original bounds, so the model predicted the growth of the unmodified model, 0.874 h⁻¹, although the only gene of GAPD was deleted. GAPD is essential for aerobic growth on glucose in e_coli_core, so the growth predicted with `"keep"` is a false positive. The docstring of `remove_genes` states no use for `"keep"`; a use suggested for it is editing gene annotations without changing the network (unverified, no source in the code). +`blocked_reactions` in `remove_genes`, and `removeBlockedRxns` in RAVEN's `removeGenes`, decide what happens to a reaction whose gene rule is emptied by a deletion. By default both keep such a reaction with bounds of zero, and the two defaults gave the same model. After deletion of `b1779`, that model predicts no growth, because GAPD stays in the network with no capacity to carry flux. `"remove"` and `removeBlockedRxns` true also predicted no growth; they delete GAPD, which changes the number of reactions and the set of reaction identifiers. `"keep"` leaves GAPD with an empty gene rule and its original bounds, so the model predicted the growth of the unmodified model, 0.874 h⁻¹, although the only gene of GAPD was deleted. GAPD is essential for aerobic growth on glucose in e_coli_core, so the growth predicted with `"keep"` is a false positive. The docstring of `remove_genes` states no use for `"keep"`. Editing gene annotations without changing the network is a use given for it in the documentation, without a source in the code. ## Choice of defaults diff --git a/docs/parameter-tuning/index.md b/docs/parameter-tuning/index.md index e066345..c4d211a 100644 --- a/docs/parameter-tuning/index.md +++ b/docs/parameter-tuning/index.md @@ -41,7 +41,7 @@ Yeast9 in the checklist is version 9 of yeast-GEM. | Study | Parameters and measurement | |---|---| -| [Homology cut-off calibration](studies/homology-cutoff-calibration.md) | `min_align_len`, `min_identity` and `max_evalue` of homology-based reconstruction, measured against an independent KEGG ortholog reference for 4 organisms of increasing distance; `min_identity` was also scored against an OMA reference | +| [Homology cut-off calibration](studies/homology-cutoff-calibration.md) | `min_align_len`, `min_identity` and `max_evalue` of homology-based reconstruction, measured for 4 organisms of increasing distance against KEGG KO annotations (partly derived from sequence similarity); `min_identity` also against the OMA reference, independent of BLAST | | [KEGG HMM cut-off calibration](studies/kegg-hmm-cutoff-calibration.md) | `cutoff`, `min_score_ratio_ko` and `min_score_ratio_g` of KEGG reconstruction with hidden Markov models (HMMs), measured against the KEGG Orthology (KO) annotations of the genes of 4 organisms | ## Context-specific extraction (ftINIT) diff --git a/docs/parameter-tuning/studies/ftinit-determinism.md b/docs/parameter-tuning/studies/ftinit-determinism.md index 6d1de2c..3914b31 100644 --- a/docs/parameter-tuning/studies/ftinit-determinism.md +++ b/docs/parameter-tuning/studies/ftinit-determinism.md @@ -27,7 +27,7 @@ The template edit removed 30 reactions that the reference builds did not use. Th | `resolve_ties=True` | 59 | 1 | 114 → 115 | 638 s | | `resolve_ties=True, prove_abs_gap=1.0` | 16 | 1 | 125 → 126 | 4108 s | -The removed reactions were almost unused, so a re-extraction that followed only the edit would change next to nothing downstream. The baseline moved 156 reactions and 17 essential-gene calls; the two settings with `resolve_ties=True` moved 59 and 16 reactions and 1 gene each. +The unedited builds hardly used the removed reactions. If the re-extraction changed only what the edit forces, the baseline model would lose the one removed reaction it kept and the `resolve_ties` models would lose none (inference, not tested). The baseline moved 156 reactions and 17 essential-gene calls; the two settings with `resolve_ties=True` moved 59 and 16 reactions and 1 gene each. ## Set-up @@ -37,7 +37,7 @@ The removed reactions were almost unused, so a re-extraction that followed only * Escalation: each solve stops at a mixed-integer programming (MIP) gap, the distance between the objective of the best solution found and the solver's bound on the optimum, either relative (divided by the objective) or absolute. Without `prove_abs_gap`, each step follows RAVEN's run schedule, here called the **escalation**. The first run uses a relative gap of 0.0004; each later run uses the larger of its relative gap and its absolute gap (10 or 20) divided by the objective, and the step stops as soon as the gap already reached meets the next target. `time_limit` caps the limit of each run: step 1 has runs of 120 s and 1800 s, the final step runs of 5 s, 120 s and 1800 s. * With `prove_abs_gap`, each step is one solve proven to that absolute gap, with the 1800 s limit for the primary solve and for the two flux phases of `resolve_ties` (phases 2c and 2d), and half of it (`_TIE_BREAK_TIME_FRACTION = 0.5`) for its two on/off phases (phases 2a and 2b); the phases are defined in the section on the parameters. The task gap-fill uses a fixed 300 s per task in every setting. * Solver: Gurobi 13.0.1, `Threads=1` unless stated, raven-toolbox at commit `94f39407`. The builds ran as Slurm jobs on AMD EPYC 9354 nodes shared with other jobs, so wall-clock figures are approximate. Most jobs had 2 CPUs and the `Threads=4` jobs of the first-step probe had 4. Several builds ran in sequence in one job (the seeds of a setting, or the three builds of the stability test). -* Determinism probe: the extraction `seed` was varied (1234, 7, and further seeds where stated) with inputs, code, solver build and machine held fixed, so a difference between builds comes from the solver's choice among equal optima. Differences between Gurobi versions were not measured. +* Determinism probe: the extraction `seed` was varied (1234, 7, and further seeds where stated) with inputs, code, solver build and machine held fixed. A difference between builds then has two sources: the solver's choice among equal optima, measured where two builds reached the same objective value, and different objective values reached within the gap and time limits (see the location of the degeneracy and the sweep of `prove_abs_gap`). Differences between Gurobi versions were not measured. * Metrics: the reaction difference is the symmetric difference of the kept-reaction sets (reactions kept by one build and not the other). The essential-gene difference is the symmetric difference of the sets of genes whose single deletion is predicted essential by `cobra.flux_analysis.find_essential_genes` at its default threshold. The gene difference is the result a user sees; the reaction difference is what the parameters act on. * Debug log: all builds ran with `FTINIT_DEBUG=1`, which logs the status of each solve and of each tie-break phase. An **incumbent** is the best solution the solver holds when it stops; at a time limit it is not proven optimal. @@ -45,7 +45,7 @@ The removed reactions were almost unused, so a re-extraction that followed only The extraction mixed-integer linear program (MILP) has many optima of equal score. In the first step on Human-GEM and DLD1, 1547 merged reactions have a negative score, and these carry the 0/1 keep-or-drop variables. Of them, 98.3 % (1520) are in blocks of equal score. The largest block has 840 reactions at −5.0. The second has 420 reactions at −2.0, the default score of a reaction without gene association; across the five Hart2015 cell lines it has 419 to 421 members, so the expression data change its size by at most 2 members. Many reaction subsets reach the same optimum, and the solver returns one of them. -`Threads=1` and a fixed `Seed` make that choice close to reproducible on one machine: the two baseline builds at one seed differed by 2 reactions. Another seed selects another member of the tied set, and the predicted essential genes change for reasons unrelated to the model or the data. Another Gurobi version or platform is expected to act the same way; that was not measured. +`Threads=1` and a fixed `Seed` make that choice close to reproducible on one machine: the two baseline builds at one seed differed by 2 reactions. Another seed can select another member of the tied set, or end the solve at another objective value within the gap and time limits; both were seen in the first-step probe, and in either case the predicted essential genes change for reasons unrelated to the model or the data. Another Gurobi version or platform is expected to act the same way; that was not measured. ## Parameters `resolve_ties` and `prove_abs_gap` @@ -66,7 +66,7 @@ A phase that does not prove its optimum has these outcomes: * A flux phase (`phase2c-minflux`, `phase2d-direction`) that finds no solution warns with its name and keeps the fluxes of the phase before it. * When `phase2a-parsimony` or `phase2b-idrank` finds no incumbent, the on/off tie-break of that solve is dropped without a warning: the primary solution is kept and the ties of that step are not resolved. A build without a warning can therefore hold an unresolved step. * When the primary solve of a step stops at `time_limit`, all tie-break phases of that solve are skipped and the primary solution is kept. -* The baseline escalation raises no warning either, and in this study its step 1 objective ended 4.0 to 12.0 below the optimum. +* The escalation raises no warning either. In the four sweep runs with the escalation and `resolve_ties=True`, the step 1 objective ended 4.0 to 12.0 below the proven optimum; at least one of the 12 first-step runs without `resolve_ties` reached it (the location of the degeneracy). For comparison, RAVEN's `ftINIT` with `resolveTies` skips the tie-break after a primary solve that stopped at the time limit, gives the count and id-rank phases half of `params.TimeLimit` each, skips the minimal-flux and direction phases in the last step, and names every unproven phase (`phase2a-parsimony`, `phase2b-idrank`, `phase2c-minflux`, `phase2d-direction`) in its "tie resolution did not converge" warning (`ftINITInternalAlg.m`). RAVEN's `ftINIT` was not run in this study. @@ -133,7 +133,7 @@ The on/off tie-break of step 1 was proven for all four seeds, so the step 1 reac The escalation builds with `resolve_ties=True` also adopted phases at a time limit, here the limit of the escalation run (120 s in step 1, so 60 s for phases 2a and 2b). The full-pipeline build of seed 1234 adopted 3 phases (`phase2b-idrank`, `phase2c-minflux`, `phase2d-direction` of step 1) and the build of seed 7 adopted 2 (`phase2c-minflux`, `phase2d-direction` of step 1). In step 2 of both builds, `phase2a-parsimony` ended at the time limit with an incumbent and `phase2b-idrank` ended at the time limit with no incumbent (`A-rt-1234.log`, `A-rt-7.log`). `phase2b-idrank` then returns no solution, the tie-break of step 2 returns `False`, the primary solution of step 2 is kept, and no warning is raised. The reaction sets of these two builds rest on a resolved step 1 and an unresolved step 2. -Counts of full builds (gap-fill included) with at least one unproven tie-break phase or a time-limited primary solve: +Counts of full builds (gap-fill included) with at least one unproven tie-break phase or a time-limited primary solve. The full builds of the main results are, per setting: baseline 4 (seeds 1234 and 7, the second build of the repeat, the edited template), `resolve_ties=True` 5 (seeds 1234 and 7, the two builds of the repeat, the edited template), `prove_abs_gap` 5 (seeds 1234 and 7 at each of 1.0 and 0.05, the edited template at 1.0). The Builds column gives how many of them were counted. | Setting | Builds | With an adopted phase at `time_limit` | With a primary solve at `time_limit` | With a tie-break dropped without a warning | |---|---:|---:|---:|---:| @@ -141,6 +141,8 @@ Counts of full builds (gap-fill included) with at least one unproven tie-break p | `resolve_ties=True` | 4 | 4 | 4 (tie-break of that solve skipped) | 4 (step 2) | | `resolve_ties=True, prove_abs_gap=1.0` or `0.05` | 4 | 4 | 0 | 0 | +The baseline row counts all four baseline builds. The `prove_abs_gap` row counts the four unedited builds and leaves out the edited build at 1.0. The `resolve_ties=True` row counts four of the five builds; which build it leaves out is not recorded. The step 2 logs cited are those of seeds 1234 and 7. + The warnings cover the adopted phases and the flux phases without a solution. A build without a warning can hold an unresolved tie-break (a dropped `phase2a-parsimony` or `phase2b-idrank`) or an unproven escalation, and the debug log is the record of what each solve reached. In the sweep, all 24 builds with `prove_abs_gap` raised at least one warning. A larger `time_limit` is the setting that can let the phases prove; its effect was not measured. ## Stability under a small template change @@ -155,9 +157,9 @@ For curation of a template (fix one or a few reactions, re-extract, compare), th ## Choice of settings -* `resolve_ties=True`: the repeat pair at seed 1234 had identical reaction sets, and the seed-to-seed difference fell from 17 genes to 1 on this data, at about 3.6 times the build time (4 to 14 minutes). Its warnings name the adopted phases that stopped at the time limit. A tie-break that finds no incumbent in phase 2a or 2b is dropped without a warning; all four `resolve_ties=True` builds dropped the one of step 2. With the escalation, the flux phases of step 1 run with the 120 s limit of that run. -* `prove_abs_gap=1.0` together with `resolve_ties=True`: the seed-to-seed difference fell to 1 reaction and 0 genes, and the objectives of both steps are proven to 1.0 (the escalation ended 4.0 to 12.0 below the optimum in step 1). The build takes about 100 minutes at `time_limit=1800`, 26 times the baseline. In the four `prove_abs_gap=1.0` sweep builds (step 1 and step 2, no gap-fill; 4987 to 6183 s wall clock), step 1 took 3300 to 3650 s (59 to 66 %), mostly in the two flux phases, and step 2 another 1650 to 2500 s. This setting fits when the difference left by `resolve_ties` alone (41 reactions and 1 gene here) is too large and the build time is acceptable. A value from 2.0 to 0.05 gave the same models; 5.0 stopped 2.0 below the optimum for one seed. No build with `prove_abs_gap` and without `resolve_ties` was run. -* Holding the raven-toolbox commit and the `gurobipy` version fixed removes the difference between solver builds at no cost in run time. `resolve_ties` and `prove_abs_gap` are independent of that choice. +* `resolve_ties=True`: the repeat pair at seed 1234 had identical reaction sets, and the seed-to-seed difference fell from 17 genes to 1 on this data, at about 3.6 times the build time (4 to 14 minutes). Its warnings name the adopted phases that stopped at the time limit. A tie-break that finds no incumbent in phase 2a or 2b is dropped without a warning; the builds at seeds 1234 and 7 dropped the one of step 2, and the table of build counts gives 4 builds with a dropped step 2 tie-break. With the escalation, the tie-break runs again in each run of the schedule (`_solve_step` in `ftinit.py` and the run schedule in `steps.py`): the flux phases of step 1 get 120 s when step 1 stops after its first run and 1800 s when a second run follows (its 5000 s capped by `time_limit`). +* `prove_abs_gap=1.0` together with `resolve_ties=True`: the seed-to-seed difference fell to 1 reaction and 0 genes, and the objectives of both steps are proven to 1.0 (in the four sweep runs with the escalation and `resolve_ties=True`, step 1 ended 4.0 to 12.0 below the optimum). The build takes about 100 minutes at `time_limit=1800`, 26 times the baseline. In the four `prove_abs_gap=1.0` sweep builds (step 1 and step 2, no gap-fill; 4987 to 6183 s wall clock), step 1 took 3305 to 3650 s, mostly in the two flux phases, which is between 53 and 73 % of a build (bounds from the extreme step 1 and build times; the per-build shares were not recorded), and step 2 another 1650 to 2500 s. This setting fits when the difference left by `resolve_ties` alone (41 reactions and 1 gene here) is too large and the build time is acceptable. A value from 2.0 to 0.05 gave the same models; 5.0 stopped 2.0 below the optimum for one seed. No build with `prove_abs_gap` and without `resolve_ties` was run. +* Holding the raven-toolbox commit and the `gurobipy` version fixed is expected to remove the differences between solver builds, at no cost in run time. This was not measured: Gurobi versions were not compared, and two baseline builds with everything held fixed differed by 2 reactions. `resolve_ties` and `prove_abs_gap` are independent of that choice. ## Limits diff --git a/docs/parameter-tuning/studies/homology-cutoff-calibration.md b/docs/parameter-tuning/studies/homology-cutoff-calibration.md index 6000ae1..51f4031 100644 --- a/docs/parameter-tuning/studies/homology-cutoff-calibration.md +++ b/docs/parameter-tuning/studies/homology-cutoff-calibration.md @@ -58,7 +58,7 @@ Scored with F1, which weights precision and recall equally, the optimum moves to | 200 (RAVEN 2.11.3) | 0.923 | 0.860 | 0.804 | 0.598 | | 300 | 0.885 | 0.816 | 0.764 | 0.571 | -Lengths 50 and 100 gave identical scores and 150 was at most 0.003 lower. The score fell between 150 and 200, by 0.010 to 0.020 from 100 to 200. Compared with 200, a length of 100 raised recall by 3 to 4 points on every organism and lowered precision by at most 0.6 points (precision and recall values not tabulated). No OMA scores were reported for alignment length. +Lengths 50 and 100 gave identical scores and 150 was at most 0.003 lower. The score fell between 150 and 200, by 0.010 to 0.020 from 100 to 200. Compared with 200, a length of 100 gave 3 to 4 more recall on every organism and lowered precision by at most 0.6 percentage points (precision and recall values not tabulated). The unit of the 3 to 4 is not recorded: percentage points of recall, or per cent more correct matches. No OMA scores were reported for alignment length. ### E-value @@ -98,7 +98,7 @@ Two more settings control the search that produces the hits. Neither was part of | `min_align_len` (`minLen`) | 200 | 100 | 100 scored 0.010 to 0.020 higher than 200 against KEGG; 50 and 100 scored the same, and 100 is the stricter of the two. | | `max_evalue` (`maxE`) | 1e-30 | 1e-30 | No change in score from 1e-4 to 1e-50; kept at the RAVEN value. | -The study supports one set of defaults for BLAST and DIAMOND, since both reached the same optimum. +The study supports one `min_identity` for BLAST and DIAMOND, since both reached the same best identity for every organism. No DIAMOND scores were reported for the length and E-value sweeps, whose base scores equal those of BLAST. ## Limits diff --git a/docs/parameter-tuning/studies/humangem-validation.md b/docs/parameter-tuning/studies/humangem-validation.md index a4d9bb2..f8f2799 100644 --- a/docs/parameter-tuning/studies/humangem-validation.md +++ b/docs/parameter-tuning/studies/humangem-validation.md @@ -6,7 +6,7 @@ raven-toolbox's ftINIT was run on Human-GEM with the RNA sequencing (RNA-seq) da Software: Human-GEM commit `38567ab8` (v2.1.1 plus 15 commits), RAVEN commit `2d0630e3` on MATLAB 2024b, raven-toolbox commit `94f3940`, and Gurobi 13.0.1 for both tools, with 4 cores per run. -Template preparation is the step that runs once per template model, before any cell line. For RAVEN, the Human-GEM function `prepHumanModelForftINIT` removed the drug reactions, the amino-acid triplet reactions and the duplicate complex I reaction from Human-GEM, set the spontaneous and custom reaction lists, and prepared the model with the essential tasks of `metabolicTasks_Essential.txt`. Its reference model, the prepared model from which RAVEN extracts, has 9730 reactions. For raven-toolbox, `prep_init_model` ran on the same cleaned Human-GEM (11265 reactions). Every reaction in every raven-toolbox result is in RAVEN's reference model, so the reaction identifiers of the two results are compared directly. +Template preparation is the step that runs once per template model, before any cell line. For RAVEN, the Human-GEM function `prepHumanModelForftINIT` removed the drug reactions, the amino-acid triplet reactions and the duplicate complex I reaction from Human-GEM, set the spontaneous and custom reaction lists, and prepared the model with the essential tasks of `metabolicTasks_Essential.txt`. RAVEN's prepared model, the model from which RAVEN extracts, has 9730 reactions. For raven-toolbox, `prep_init_model` ran on the same cleaned Human-GEM (11265 reactions). Every reaction in every raven-toolbox result is in RAVEN's prepared model, so the reaction identifiers of the two results are compared directly. Scoring: gene scores are `5·ln(level/1)` of the transcripts per million (TPM) values in the Hart2015 file, clipped to [−5, 10], and reactions take their scores from the gene rules; a reaction without genes scores −2. raven-toolbox computes this with `gene_scores_from_expression` and `score_reactions_from_genes`, RAVEN with `transcrData` and `threshold=1`. @@ -17,7 +17,7 @@ raven-toolbox ran in two variants: - **No-task**: the template is prepared without a task list, and the extraction uses the expression scores alone. - **Task-constrained**: the template is prepared with the essential metabolic tasks, the reactions that a task needs are kept whatever their score, and `ftinit` adds reactions back until every task is feasible (its default `fill_gaps=True`). -RAVEN's reference result is task-constrained in this sense. A third set of no-task runs used `big_m=100`. +The RAVEN result is task-constrained in this sense. A third set of no-task runs used `big_m=100`. Measurements: the result of a run is its set of reaction identifiers. For each cell line the comparison gives the number of reactions in both sets (shared), in one set only, and the Jaccard index, the shared count divided by the size of the union. The comparison covers reactions only; genes, essential genes and task performance of the extracted models were not compared. Each cell line, variant and `big_m` value ran once. Times are wall-clock times of single runs on shared nodes. @@ -86,7 +86,7 @@ Neither tool was run twice on the same input, so the comparison does not separat The no-task variant shares Jaccard 0.964 to 0.982 with RAVEN. It keeps fewer reactions than RAVEN in every cell line: 0.3 to 0.5 % fewer for four cell lines and 1.6 % fewer for RPE1. The reactions only in the RAVEN model (76 to 186) outnumber those only in the raven-toolbox model (53 to 76). -The no-task variant omits the task layer that RAVEN's reference applies. With the task layer, the reactions only in the RAVEN model fall to 43 to 71 per cell line. That the task layer accounts for most of the surplus is an inference from these counts; the reactions in the surplus were not checked against the task-essential reactions. +The no-task variant omits the task layer that the RAVEN result includes. With the task layer, the reactions only in the RAVEN model fall to 43 to 71 per cell line. That the task layer accounts for most of the surplus is an inference from these counts; the reactions in the surplus were not checked against the task-essential reactions. ## Effect of `big_m` @@ -94,7 +94,7 @@ Lowering the cap on the per-reaction big-M from 1000 to 100 changed the Jaccard ## Run time -After the template preparation, a task-constrained cell-type model took 146 to 381 s in raven-toolbox and 40 to 91 s in RAVEN; a no-task model took 81 to 91 s. The template preparation took 1556 s (26 min) and 1695 s (28 min) in raven-toolbox and 4320 s (72 min) in RAVEN. The preparations ran with different parallelism (see the run-time table), so the times compare whole runs and not the code alone. The [Reference run](../../guide/init.md#reference-run) of the guide Context-specific models with ftINIT reports a preparation of 47 min in MATLAB and 37 min in Python and an extraction of 797 s in MATLAB and 124 s in Python. That run used RAVEN's `prepINITModel` directly on Human-GEM `6812be2a` (12748 reactions) without the cleaning step, and series `1+0`; the three differences from this validation were not separated, so the times are not comparable. +After the template preparation, a task-constrained cell-type model took 146 to 381 s in raven-toolbox and 40 to 91 s in RAVEN; a no-task model took 81 to 91 s. The template preparation took 1556 s (26 min) and 1695 s (28 min) in raven-toolbox and 4320 s (72 min) in RAVEN. The preparations ran with different parallelism (see the run-time table), so the times compare whole runs and not the code alone. The [Reference run](../../guide/init.md#reference-run) of the guide Context-specific models with ftINIT reports a preparation of 47 min in MATLAB and 37 min in Python and an extraction of 797 s in MATLAB and 124 s in Python. That run used RAVEN's `prepINITModel` directly on Human-GEM `6812be2a` (12748 reactions) without the cleaning step, and series `1+0`; these four differences from this validation (preparation function, Human-GEM commit, cleaning step, series) were not separated, so the times are not comparable. ## Numerical settings shared with RAVEN @@ -118,4 +118,4 @@ Each cell line, variant and `big_m` value ran once, without seed repeats, so the ## Reproduction -The script that ran this validation is not part of the pinned raven-toolbox. Its outputs, kept in the working directory that the analysis scripts take as `--work`, are reused by later studies: the model exported from the RAVEN run as `raven_refModel.xml`, and the prepared Human-GEM of raven-toolbox as `rg_prep.pkl` (no-task) and `rg_prep_tasks.pkl` (task-constrained). `scripts/analyze_init_solvers.py` reads `rg_prep.pkl` and `scripts/analyze_init_robustness.py` reads `rg_prep_tasks.pkl`; both score reactions against `raven_refModel.xml`. `scripts/analyze_init_params.py` reads `raven_refModel.xml` and prepares its own models from it. +The script that ran this validation is not part of the pinned raven-toolbox. Its outputs, kept in the working directory that the analysis scripts take as `--work`, are reused by later studies: a model exported from the RAVEN run as `raven_refModel.xml`, and the prepared Human-GEM of raven-toolbox as `rg_prep.pkl` (no-task) and `rg_prep_tasks.pkl` (task-constrained). `scripts/analyze_init_solvers.py` reads `rg_prep.pkl` and `scripts/analyze_init_robustness.py` reads `rg_prep_tasks.pkl`; both score reactions against `raven_refModel.xml`. `scripts/analyze_init_params.py` reads `raven_refModel.xml` and prepares its own models from it. diff --git a/docs/parameter-tuning/studies/init-param-calibration.md b/docs/parameter-tuning/studies/init-param-calibration.md index 2d81732..2d4417b 100644 --- a/docs/parameter-tuning/studies/init-param-calibration.md +++ b/docs/parameter-tuning/studies/init-param-calibration.md @@ -1,6 +1,6 @@ # ftINIT parameter calibration and input robustness -This study runs raven-toolbox's ftINIT on Human-GEM with the HCT116 expression data of the Hart2015 dataset and varies one or two settings per run series; the scaling runs and the full-pipeline runs vary scaling and `big_m` together. The calibration part varies the mixed-integer linear program (MILP) settings `mip_gap`, `big_m` and `force_on` and the stoichiometric scaling of the prepared model, and records the run time, the MILP objective and which reactions are kept. The robustness part degrades the expression input by dropout, noise and downsampling, extracts a model with the metabolic-task layer on, and records the model size, the essential tasks the model passes and its overlap with the model from clean data. +This study runs raven-toolbox's ftINIT on Human-GEM with the HCT116 expression data of the Hart2015 dataset and varies one or two settings per run series; the scaling runs and the full-pipeline runs vary scaling and `big_m` together. The calibration part varies the mixed-integer linear program (MILP) settings `mip_gap`, `big_m` and `force_on` and the stoichiometric scaling of the prepared model, and records the run time, the MILP objective and which reactions are kept. The robustness part degrades the expression input by dropout, noise and downsampling, extracts a model with the metabolic-task layer on, and records the model size, the essential tasks the model passes and its overlap with the model from clean data. Overlap is measured as the Jaccard index of two kept-reaction sets: the reactions in both divided by the reactions in either. ## Set-up @@ -47,7 +47,7 @@ The lever runs repeat the 70 % dropout input with `no_gene_score` or `force_on` A MIP gap bounds the objective only. Two solutions within the gap can keep different reactions, so the reaction set is compared separately from the objective. -Each table row is one run unless the row states repeats; this set-up has no runs that differ only in the seed. For scale, two raven-toolbox builds that differed only in the seed differed by 110 reactions (symmetric difference of the kept-reaction sets) in [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](ftinit-determinism.md), in a different set-up (Human-GEM v2.0.0-38, DLD1, `big_m=100`, escalation schedule, with tasks and gap-fill). That page gives 7754 reactions for one build of that set-up and not the sizes of the two seed builds; assuming builds of about 7300 to 7800 reactions, 110 reactions are about 1.5 % of the model and correspond to a Jaccard index of about 0.985. The differences between settings in this study, 1 to 2 % in the Jaccard index, are of the same order as that spread and are not attributed to a parameter. +Each table row is one run unless the row states repeats; this set-up has no runs that differ only in the seed. The only available measure of the seed-to-seed spread comes from a different set-up, in [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](ftinit-determinism.md): two baseline builds (Human-GEM v2.0.0-38, DLD1, `big_m=100`, escalation schedule, with tasks and gap-fill) that differed only in the seed differed by 110 reactions (symmetric difference of the kept-reaction sets). That page does not give the sizes of these two builds; the 7754 reactions it reports belong to a build with `resolve_ties=True` and `prove_abs_gap=1.0`. Assuming builds of about 7300 to 7800 reactions, 110 reactions are about 1.5 % of the model and correspond to a Jaccard index of about 0.985. That pair compares two full pipelines with gap-fill; a step-0 set is the result of one MILP step on merged-model identifiers without tasks, a different kind of object. Between the settings of the calibration the Jaccard index ranged from 0.9976 to 0.9649, a difference of 0.2 to 3.5 % from identical sets. ### Source of the numbers @@ -169,9 +169,9 @@ In the step-0 solve the time was 24 to 25 s at every gap from 0.0002 to 0.1. At ## Effect of `big_m` and scaling -In the step-0 solve every `big_m` from 10 to 1000 was feasible, took 23 to 26 s and reached the objective of `big_m=1000` within 0.0003. Against the run at `big_m=1000` and against the run at `big_m=100`, the reaction sets of the other caps had Jaccard indices of 0.983 to 0.991, of the same order as the seed-to-seed spread of the different set-up in [Set-up](#set-up). In the step-0 scaling runs, every `max_stoich_diff` from 10 to 100 and the unscaled model were feasible at both caps, with Jaccard indices of 0.980 to 0.991 against the scaled reference. +In the step-0 solve every `big_m` from 10 to 1000 was feasible, took 23 to 26 s and reached the objective of `big_m=1000` within 0.0003. Against the run at `big_m=1000` and against the run at `big_m=100`, the reaction sets of the other caps had Jaccard indices of 0.983 to 0.991. The step-0 solve has no seed repeats, and the seed pair in [Set-up](#set-up) compares full pipelines with gap-fill, so it is no reference for these sets. In the step-0 scaling runs, every `max_stoich_diff` from 10 to 100 and the unscaled model were feasible at both caps, with Jaccard indices of 0.980 to 0.991 against the scaled reference. -The full pipeline separates the settings that the step-0 solve does not. At `big_m=1000` it was feasible with and without scaling: 83 s with scaling, 157 s without, and the two models shared a Jaccard index of 0.969 (7312 and 7264 reactions). At `big_m=100` it was feasible with scaling (117 to 119 s) and infeasible without, and at `big_m=25` it was infeasible with scaling. With scaling, the caps from 50 to 1000 gave runs of 81 to 119 s with no trend in `big_m` and reaction sets that shared a Jaccard index of 0.978 to 0.982 with the default. No cap below 1000 ran faster than the 81 to 85 s of the `big_m=1000` runs. The differences in the reaction set are of the same order as the seed-to-seed spread of the different set-up (about 1.5 % of the reactions) and are not smaller than it; with no seed repeats in this set-up, they are not attributed to the cap. A lower cap also reduces the flux that a kept reaction can carry. +The full pipeline separates the settings that the step-0 solve does not. At `big_m=1000` it was feasible with and without scaling: 83 s with scaling, 157 s without, and the two models shared a Jaccard index of 0.969 (7312 and 7264 reactions). At `big_m=100` it was feasible with scaling (117 to 119 s) and infeasible without, and at `big_m=25` it was infeasible with scaling. With scaling, the caps from 50 to 1000 gave runs of 81 to 119 s with no trend in `big_m` and reaction sets that shared a Jaccard index of 0.978 to 0.982 with the default. No cap below 1000 ran faster than the 81 to 85 s of the `big_m=1000` runs. Derived from the sizes and Jaccard indices in the table (the sizes of two sets add up to the size of their union times 1 + J), the sets at caps 500, 250, 100 and 50 differed from the `big_m=1000` set by about 148, 163, 140 and 133 reactions. The one seed pair available, from the different set-up in [Set-up](#set-up), differed by 110 reactions, so every cap difference is about 20 to 50 % larger than that single value; the seed-to-seed spread of this set-up is unknown. The three `big_m=1000` runs kept identical sets (Jaccard 1.000) and only the cap changed between rows, so the differences follow the cap; whether they are systematic or a different choice among near-equal optima was not tested. A lower cap also reduces the flux that a kept reaction can carry. ## Effect of `force_on` @@ -187,7 +187,7 @@ In raven-toolbox commit `94f3940`, `ftinit` calls `fill_tasks` without a time li ## Choice of settings -The calibration supports the defaults `big_m=1000`, `force_on=0.1` and scaling on with `max_stoich_diff=25` (RAVEN's setting). `big_m=1000` is the cap that RAVEN's `ftINITInternalAlg` uses (`bigMCap`). Scaling is not needed for feasibility at `big_m=1000`; in these single runs the pipeline took 83 s with scaling and 157 s without. A step-0 solve or a step without a schedule at `mip_gap=0.001` reaches an objective within 0.0001 of the tightest gap; a named series ignores the setting. The full-pipeline runs used `time_limit=600` and the robustness script uses `time_limit=120`; no full-pipeline run took longer than 157 s in total. For noisy input the tested settings needed no change, and no tested setting compensated for dropout. +The calibration supports the defaults `big_m=1000`, `force_on=0.1` and scaling on with `max_stoich_diff=25` (RAVEN's setting). `big_m=1000` is the cap that RAVEN's `ftINITInternalAlg` uses (`bigMCap`). Scaling is not needed for feasibility at `big_m=1000`; in these single runs the pipeline took 83 s with scaling and 157 s without. In the step-0 solve, `mip_gap=0.001` reached an objective within 0.0001 of the tightest gap; a named series ignores the setting. The full-pipeline runs used `time_limit=600` and the robustness script uses `time_limit=120`; no full-pipeline run took longer than 157 s in total. For noisy input the tested settings needed no change, and no tested setting compensated for dropout. ## Gene-scoring constants @@ -206,7 +206,7 @@ with `factor=5.0`, `max_score=10.0` and `min_score=-5.0`, the same in both imple ## Limits - One model (Human-GEM), one cell line (HCT116) and one solver (Gurobi 13.0.1). -- Most settings ran once; the repeats are listed with the full-pipeline table. This set-up has no seed repeats, and differences of the order of the seed-to-seed spread of a different set-up are not attributed to a setting. +- Most settings ran once; the repeats are listed with the full-pipeline table. This set-up has no seed repeats. The full-pipeline differences between caps are larger than the one seed pair available, which comes from a different set-up, so the seed-to-seed spread of this set-up is unknown. - The robustness runs used a Human-GEM state whose commit is not recorded and `big_m=100`; the pinned script passes no `big_m`, so a rerun uses the default 1000. Each degradation is one random draw at one seed. - The step-0 solve shows no difference between caps or scaling settings that the full pipeline shows, so step-0 results alone do not predict feasibility of the pipeline. - The robustness tables have no run times, and the effect of `no_gene_score` on downsampled input was not measured. diff --git a/docs/parameter-tuning/studies/init-solver-benchmark.md b/docs/parameter-tuning/studies/init-solver-benchmark.md index 6fcb910..2af759f 100644 --- a/docs/parameter-tuning/studies/init-solver-benchmark.md +++ b/docs/parameter-tuning/studies/init-solver-benchmark.md @@ -1,10 +1,10 @@ # ftINIT on different solvers -ftINIT builds mixed-integer linear programs (MILPs) and passes them to a solver through optlang. This benchmark runs ftINIT on Human-GEM with each solver interface that the environment offered, and records whether the run completed, how long it took and which reactions the extracted model contains. +ftINIT builds mixed-integer linear programs (MILPs) and passes them to a solver through optlang. This benchmark runs ftINIT on Human-GEM with the gurobi, glpk, scipy and hybrid interfaces of optlang, and records whether the run completed, how long it took and which reactions the extracted model contains. ## Set-up -Every run calls `ftinit()` with the same input: Human-GEM prepared without a task list, reaction scores from HCT116 expression, series `1+1` and `time_limit=900` s. The call also passes `mip_gap=0.001`, which a named series ignores. Software: Human-GEM commit `38567ab8`, raven-toolbox commit `94f3940`, Gurobi 13.0.1 and cobra 0.32.1, on 4 cores. Each setting ran once. +Every run calls `ftinit()` with the same input: Human-GEM prepared without a task list and with scaled stoichiometry, reaction scores from HCT116 expression, series `1+1` and `time_limit=900` s. The call also passes `mip_gap=0.001`, which a named series ignores. Software: Human-GEM commit `38567ab8`, raven-toolbox commit `94f3940`, Gurobi 13.0.1 and cobra 0.32.1, on 4 cores. Each setting ran once. The time is the wall-clock time of the whole benchmark step: reading the reference and the prepared model, scoring the reactions and calling `ftinit()`. A failed run reports the time until the failure. @@ -30,7 +30,7 @@ GLPK completed the pipeline in 2182 s. Its MILP solver honours `configuration.ti ## Interfaces that failed -HiGHS is reached through the `scipy` interface of optlang, which accepts continuous variables only and cannot hold the ftINIT MILP. The run failed with the error in the table after 1521 s. The benchmark selected optlang's `hybrid_interface` for HiGHS, but cobra 0.32.1 does not accept `hybrid` as a solver name; it accepts `glpk_exact`, `glpk`, `gurobi` and `scipy`. The failure came after 10 s, the time spent reading the reference model before the solver name was set. +HiGHS is reached through the `scipy` interface of optlang, which accepts continuous variables only and cannot hold the ftINIT MILP. The run failed with the error in the table after 1521 s. The script selected optlang's `hybrid_interface` module for HiGHS, but `hybrid` was not among the solver names offered in this environment: the error lists `glpk_exact`, `glpk`, `gurobi` and `scipy`. `glpk_exact` was offered and not run. The failure came after 10 s, the time spent reading the reference model before the solver name was set. ## Choice of solver @@ -53,4 +53,4 @@ python scripts/analyze_init_solvers.py --cell HCT116 \ --doc docs/parameter-tuning/studies/init-solver-benchmark.md ``` -The first command checks the toy networks and runs in seconds. The second runs the genome-scale benchmark, which takes minutes to hours. It resumes per solver and reuses the prepared Human-GEM of [Human-GEM cell-type model validation against RAVEN](humangem-validation.md). The script writes its report to the file given by `--doc`; give another path to keep the page. +The first command checks the toy networks and runs in seconds. The second runs the genome-scale benchmark, which takes minutes to hours. It resumes per solver and reuses the prepared Human-GEM of [Human-GEM cell-type model validation against RAVEN](humangem-validation.md). The pinned script has no `big_m` option and runs the gurobi, hybrid and glpk interfaces only (`_INTERFACES` in `analyze_init_solvers.py`), so the command does not produce the `big_m=100` and `scipy` rows. The script writes its report to the file given by `--doc`; give another path to keep the page. diff --git a/docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md b/docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md index 6183bda..5d54a6e 100644 --- a/docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md +++ b/docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md @@ -160,7 +160,7 @@ The table compares the RAVEN 2.11.3 settings `(1e-50, 0.3, 0.8)` with the defaul | `eco` | 0.67 → 0.67 | 0.95 → 0.97 | 0.78 → 0.79 | 0.98 → 0.99 | 198 → 173 | | `mge` | 0.77 → 0.69 | 0.84 → 0.94 | 0.80 → 0.80 | 0.87 → 0.97 | 29 → 35 | -With the defaults, gene→KO recall rose by 0.10 for `mge` and by 0.01 to 0.03 for the other three organisms. Gene→KO precision fell by 0.02 (`sce`), 0.03 (`cme`) and 0.08 (`mge`), and stayed at 0.67 for `eco`. F1 changed by at most 0.01. The number of novel reactions fell for `sce`, `cme` and `eco` and rose from 29 to 35 for `mge`. +With the defaults, gene→KO recall rose by 0.10 for `mge` and by 0.01 to 0.03 for the other three organisms. Gene→KO precision fell by 0.02 (`sce`), 0.03 (`cme`) and 0.08 (`mge`), and stayed at 0.67 for `eco`. F1 changed by at most 0.01. The F1 of the defaults was computed from precision and recall rounded to two decimals, which alone moves it by up to 0.005, and the measured F1 is itself rounded to 0.01, so a difference of 0.01 is within the rounding. The number of novel reactions fell for `sce`, `cme` and `eco` and rose from 29 to 35 for `mge`. ### Whole-model comparison of the two KEGG paths @@ -172,21 +172,21 @@ With the defaults, gene→KO recall rose by 0.10 for `mge` and by 0.01 to 0.03 f | metabolites | 1501 | 1567 | | genes | 835 | 896 | -The HMM path contained 96.3 % of the reactions of the annotation path (1305 of 1355, Jaccard index 0.86) and 96.6 % of its genes (807 of 835, Jaccard index 0.87). `hmmscan` took ≈ 0.1 s per query against either library on 12 threads (yeast: 6021 queries in 633 s, about 10.5 minutes). +The HMM path contained 96.3 % of the reactions of the annotation path (1305 of 1355, Jaccard index 0.86) and 96.6 % of its genes (807 of 835, Jaccard index 0.87). The one timed search, `sce` against `eukaryotes.hmm` with `hmmscan` on 12 threads, took 633 s for 6021 queries (about 10.5 minutes, ≈ 0.1 s per query). The `prokaryotes.hmm` library was not timed. ## Meaning of the results From 1e-10 to 1e-30 a stricter cut-off raised precision while gene→KO recall fell by at most 0.03 (from 0.98 or 0.99 to between 0.95 and 0.98). Beyond 1e-30 the cut-off traded recall for precision: it removed matched pairs together with novel pairs, and novel pairs remained at every cut-off tested. Across the whole sweep from 1e-10 to 1e-100 precision rose by 0.27 (`sce`, 0.57 to 0.84) to 0.35 (`mge`). The RAVEN 2.11.3 value `1e-50` removes 10 to 50 % of the matched pairs of `mge`, the organism with the weakest matched hits, and 1 to 10 % in the other three. -At `cutoff = 1e-50`, raising `min_score_ratio_g` from 0.80 to 0.95 raised precision by 0.05 to 0.10 for a recall loss of 0.01 to 0.03, and raising it from 0.50 to 0.80 raised precision by 0.15 to 0.27 (`eco`, 0.39 to 0.66). `min_score_ratio_ko` changed precision by 0.01 to 0.03 and recall not at all. The defaults move part of the filtering from the cut-off to `min_score_ratio_g`. On the four organisms the move gained recall, most for `mge`, at the cost of precision in three of them, and left F1 within 0.01 of the RAVEN 2.11.3 settings. +At `cutoff = 1e-50`, raising `min_score_ratio_g` from 0.80 to 0.95 raised precision by 0.05 to 0.10 for a recall loss of 0.01 to 0.03, and raising it from 0.50 to 0.80 raised precision by 0.15 to 0.27 (`eco`, 0.39 to 0.66). At `min_score_ratio_g = 0.80` and `cutoff = 1e-50`, raising `min_score_ratio_ko` from 0.0 to 0.5 changed precision by 0.01 to 0.03 and recall not at all; its effect at other settings was not reported. The defaults move part of the filtering from the cut-off to `min_score_ratio_g`. On the four organisms the move gained recall, most for `mge`, at the cost of precision in three of them, and left F1 within 0.01 of the RAVEN 2.11.3 settings, a difference within the rounding of the computed F1. ## Chosen defaults | parameter | RAVEN 2.11.3 | default (raven-toolbox and RAVEN 3) | basis | |---|---|---|---| | `cutoff` (`cutOff`) | 1e-50 | 1e-30 | beyond matched p99 for `sce` and `eco`, between p95 and p99 for `cme` and `mge`; with the new `min_score_ratio_g`, `mge` recall 0.84 → 0.94 | -| `min_score_ratio_g` (`minScoreRatioG`) | 0.8 | 0.9 | from 0.80 to 0.95 it raised precision by 0.05 to 0.10 for a recall loss of 0.01 to 0.03; partly offsets the precision lost to the looser cut-off | -| `min_score_ratio_ko` (`minScoreRatioKO`) | 0.3 | 0.3 | changed precision by 0.01 to 0.03 and recall not at all; kept at the RAVEN value | +| `min_score_ratio_g` (`minScoreRatioG`) | 0.8 | 0.9 | from 0.80 to 0.95 it raised precision by 0.05 to 0.10 for a recall loss of 0.01 to 0.03; offsets part of the precision lost to the looser cut-off (inference, 0.9 not swept) | +| `min_score_ratio_ko` (`minScoreRatioKO`) | 0.3 | 0.3 | at `min_score_ratio_g` 0.80 and `cutoff` 1e-50, 0.0 to 0.5 changed precision by 0.01 to 0.03 and recall not at all; kept at the RAVEN value | The defaults favour recall: the changed settings raised recall in every organism and lowered precision in three of the four. @@ -194,7 +194,7 @@ The defaults favour recall: the changed settings raised recall in every organism Three settings next to the cut-offs were not measured in the study. The reasons given for their values are not tested. -`threads` in `run_hmmsearch` and `build_ko_hmm` defaults to every CPU the process may run on. The value rests on the assumption, not tested here, that HMMER's search gives the same hits at any thread count and that E-values differ at most in the last floating-point places. Single-threaded search against the full KO library (more than 26,000 HMMs) is given as 30 to 60 minutes per proteome, against roughly five minutes multi-threaded; no run with a recorded thread count and hardware supports either figure. The one timed run, `sce` against `eukaryotes.hmm` on 12 threads, took 633 s (about 10.5 minutes). +`threads` in `run_hmmsearch` and `build_ko_hmm` defaults to every CPU the process may run on. The value rests on the assumption, not tested here, that HMMER's search gives the same hits at any thread count and that E-values differ at most in the last floating-point places. The documentation gives single-threaded search against the full KO library (more than 26,000 HMMs) as 30 to 60 minutes per proteome, against roughly five minutes multi-threaded. The source of these figures is not recorded, and no run with a recorded thread count and hardware supports either of them. The one timed run, `sce` against `eukaryotes.hmm` on 12 threads, took 633 s (about 10.5 minutes). `seq_identity` in `build_ko_hmm` is `0.9`: the identity at which CD-HIT collapses near-duplicate sequences within a KO before the HMM is built; the clustering keeps a KO with many near-identical entries from giving an overfitted profile. It is CD-HIT's recommended value for proteins and the default of `seqIdentity` in RAVEN 2.11.3. `seq_identity=-1` skips the clustering. RAVEN 3's `getKEGGModelForOrganism` uses prebuilt HMM libraries and has no clustering step. diff --git a/docs/parameter-tuning/studies/sampling-convergence-calibration.md b/docs/parameter-tuning/studies/sampling-convergence-calibration.md index 099cd92..7da2b83 100644 --- a/docs/parameter-tuning/studies/sampling-convergence-calibration.md +++ b/docs/parameter-tuning/studies/sampling-convergence-calibration.md @@ -7,9 +7,9 @@ - The **flux polytope** is the set of flux vectors `v` with `S v = 0` and `lb <= v <= ub`, where `S` is the stoichiometric matrix and `lb` and `ub` are the reaction bounds. A **vertex** is a corner point of the polytope. - A **Markov chain Monte Carlo (MCMC)** sampler is a random walk inside the polytope whose visited points, after enough steps, are distributed close to uniformly over it. One walk from one starting point and one random seed is a **chain**. - A **hit-and-run** step chooses a direction, finds the line segment through the current point along that direction that lies inside the polytope, and moves to a uniformly random point on that segment. -- **ACHR** (Artificially Centered Hit-and-Run) takes its directions from warmup points and a running centre of the chain, and does not transform the polytope. raven-toolbox wraps `cobra.sampling.ACHRSampler`, which computes its warmup points with two linear programs (LPs) per reaction before the first step ([Flux sampling](../../guide/sampling.md)). +- **ACHR** (Artificially Centered Hit-and-Run) takes its directions from warmup points and a running centre of the chain, and does not transform the polytope. raven-toolbox wraps `cobra.sampling.ACHRSampler`, which computes its warmup points with two linear programs (LPs) per reaction before the first step ([Random sampling](../../guide/sampling.md)). - **CHRR** (Coordinate Hit-and-Run with Rounding) first runs a flux variability analysis (FVA), fixes the reactions whose range is narrower than `fixed_width_tol`, reduces the polytope to a full-dimensional body through the nullspace of the equality constraints, and **rounds** it: it maps the body so that its maximum-volume inscribed ellipsoid (**MVE**) becomes a unit ball. The walk then moves along one coordinate per step. -- **`thinning`** is the number of steps between two stored samples. A chain with `n_samples` stored samples therefore takes `thinning × n_samples` steps after its start. +- **`thinning`** is the number of steps between two stored samples. An ACHR chain with `n_samples` stored samples therefore takes `thinning × n_samples` steps after its warmup points, and a CHRR chain takes `warmup + thinning × n_samples` steps after the rounding. - **`warmup`** is the number of CHRR steps discarded before the first stored sample. `random_sampling` passes it to CHRR only; the ACHR runs ignore it. - The **lag-1 autocorrelation** `ρ` is the correlation between consecutive stored samples of one chain. - The **effective sample size (ESS)** is the number of independent samples that carry the same information as the correlated ones. Under the assumption that the chain is a first-order autoregressive process, `ESS = n × (1−ρ) / (1+ρ)` for `n` stored samples. @@ -134,7 +134,7 @@ CHRR was not run on yeast-GEM with 4 chains × 300 samples, because 20 times the Each CHRR chain ran an FVA, a nullspace computation and the MVE rounding before its first step, and then 1,000 warmup steps and 2,000 sampling steps. The probe does not separate these parts. That the cost before the first step dominates, and that most of it is the MVE rounding, is an inference from the small number of steps; the parts were not timed. `random_sampling` computes this transform anew in every call, so each chain pays it. -The R-hat of the probe (median 5.16, p90 86, max in the billions) does not show that CHRR converges worse than ACHR at genome scale. With 20 samples per chain, the within-chain variance `W` is estimated from too few samples to be stable: a reaction that shows almost no variance in 20 draws by chance gives a large R-hat whatever its distribution. The probe cannot distinguish a CHRR that does not converge on yeast-GEM from an R-hat that needs more samples; a run with enough samples for a stable R-hat has the cost problem of the full run. +The R-hat of the probe (median 5.16, p90 86, max in the billions) does not show that CHRR converges worse than ACHR at genome scale. With 20 samples per chain, the within-chain variance `W` is estimated from too few samples to be stable: a reaction that shows almost no variance in 20 draws by chance, combined with any difference between the chain means (`B/n`), gives a large R-hat whatever its distribution. The probe cannot distinguish a CHRR that does not converge on yeast-GEM from an R-hat that needs more samples; a run with enough samples for a stable R-hat has the cost problem of the full run. For the choice of settings: 20 samples per chain are too few for R-hat on yeast-GEM. Whether CHRR converges on yeast-GEM with 300 samples per chain is open. @@ -152,7 +152,7 @@ Two settings of `method='random_objective'` differ from RAVEN MATLAB's `randomSa `replace_max_bound` (default `False`; MATLAB's `replaceBoundsWithInf` defaults to true) replaces the largest upper bound, the **big-M** value that stands for an unconstrained reaction, with infinity before sampling, and replaces the most negative lower bound, if it is below 0, with minus infinity. RAVEN-convention models use 1000 as the big-M for almost every reaction (4,083 of the 4,102 reactions of yeast-GEM). With every such bound replaced, the random-objective LP becomes unbounded, because the objective can grow without limit through any reaction that is then unconstrained. With `False`, 200 samples completed, with 0.57 % of them at the bound. -`loopless_good_reactions` (default `True`) selects which reactions are eligible as random objectives. MATLAB excludes every reaction whose FVA maximum reaches 0.999 times the largest bound (999 on a model capped at 1000), which takes proximity to the big-M as a sign of a thermodynamic loop. That rule also excludes reactions that reach the bound for metabolic reasons. A loopless FVA excludes only the reactions that take part in infeasible cycles, at a higher cost. +`loopless_good_reactions` (default `True`) selects which reactions are eligible as random objectives. MATLAB excludes every reaction whose FVA maximum is infinite or above 0.999 times the largest upper bound (999 on a model capped at 1000), and, when the smallest lower bound is negative, every reaction whose FVA minimum is minus infinity or below 0.999 times that bound; it treats proximity to the big-M as a sign of a thermodynamic loop. It also excludes reactions that cannot carry a non-zero flux and, in an ecModel, the `usage_prot_` reactions. That rule also excludes reactions that reach the bound for metabolic reasons. A loopless FVA excludes only the reactions that take part in infeasible cycles, at a higher cost. ## Limits @@ -169,4 +169,4 @@ python scripts/analyze_sampling_convergence.py \ --n-chains 4 --n-samples 300 --thinning 100 --warmup 1000 ``` -The CHRR runs use the same command with `--method chrr`. The script caches the chains per model, method, `n_chains`, `n_samples`, `thinning` and `warmup` under `--out`; a run with a cached configuration computes R-hat from the stored chains without sampling. Another `--thinning` or `--n-samples` samples again and is cached separately. Four chains in parallel on 12 cores took ~3 times the wall time of one chain, so a sweep at genome scale needs more time than one chain per configuration suggests. +The CHRR runs use the same command with `--method chrr`. The script caches the chains per model, method, `n_chains`, `n_samples`, `thinning` and `warmup` under `--out`; a run with a cached configuration computes R-hat from the stored chains without sampling. Another `--thinning` or `--n-samples` samples again and is cached separately. On yeast-GEM, four ACHR chains in parallel on 12 cores took ~3 times the wall time of one chain at the same settings; the two runs read different yeast-GEM files and the cause of the difference was not measured (see [Agreement between chains on yeast-GEM](#agreement-between-chains-on-yeast-gem)). A sweep at genome scale can therefore take more time than one chain per configuration suggests. diff --git a/docs/tuned-parameters.md b/docs/tuned-parameters.md index 7b2fa6d..43634f2 100644 --- a/docs/tuned-parameters.md +++ b/docs/tuned-parameters.md @@ -15,10 +15,10 @@ RAVEN MATLAB: `randomSampling` (`method='achr'|'chrr'|'random_objective'`), `sam | Parameter | Default | How determined | |---|---|---| | `method` | `'achr'` (Python); `'achr'` (RAVEN 3, `randomSampling`) | Literature: ACHR gives near-uniform interior sampling, and the random-objective method draws polytope vertices. All three methods are selectable in both implementations (`'randomObjective'` in MATLAB, `'random_objective'` in Python). RAVEN 2.11.3 had only the random-objective method. | -| `thinning` | `100` | Upstream convention (cobrapy's own ACHR default). Measured insufficient at genome scale: on yeast-GEM one ACHR chain had an effective sample size (ESS) of about 12 from 300 stored samples, and 4 chains disagreed (median Gelman-Rubin R-hat 1.1671, 67.5 % of scored reactions above 1.1); `thinning` up to 500 did not fix it. See [Flux sampling convergence and settings](parameter-tuning/studies/sampling-convergence-calibration.md). | +| `thinning` | `100` | Upstream convention (cobrapy's own ACHR default). Measured insufficient at genome scale: on yeast-GEM one ACHR chain had an effective sample size (ESS) of about 12 from 300 stored samples, and 4 chains disagreed (median Gelman-Rubin R-hat 1.1671, 67.5 % of scored reactions above 1.1). At `thinning=500` the lag-1 autocorrelation of one chain was 0.849. Agreement between chains was tested at one other setting, `thinning=300` with `n_samples=100` (the same 30,000 steps per chain), and did not improve. See [Flux sampling convergence and settings](parameter-tuning/studies/sampling-convergence-calibration.md). | | `warmup` | `1000` | Used by `method='chrr'` only (the steps discarded before the first stored sample); the ACHR sampler of cobrapy has no such parameter. Same as `nBurnin` of RAVEN's `sampleCHRR` and `randomSampling`, which also applies to CHRR only. Not measured. | | `n_objectives` | `2` | Literature: [Bordel et al. 2010](https://doi.org/10.1371/journal.pcbi.1000859). | -| `replace_max_bound` | `False` | Measured: applying the alternative (`True`) inside the Python and cobrapy solver stack makes the sampler unbounded on standard RAVEN-convention models (±1000 bounds). Kept different between the implementations by design, because of a solver-stack constraint. Whether MATLAB's own solver path handles it safely was not tested here. | +| `replace_max_bound` | `False` | Used by `method='random_objective'`. The study gives the mechanism and records no run with `True`: RAVEN-convention models use 1000 as the bound of almost every reaction, and with every such bound replaced by infinity the random-objective LP becomes unbounded, because the objective can grow without limit through any reaction that is then unconstrained. With `False`, 200 samples completed. Kept different from MATLAB on purpose. Whether MATLAB's own solver path handles `True` was not tested here. | | `loopless_good_reactions` | `True` | Loopless FVA excludes only reactions that take part in infeasible cycles, at higher cost; the alternative fixed-threshold (±999) heuristic also excludes reactions that legitimately reach capacity. The loopless technique is implemented in Python only; porting it is a larger project than changing a default. | | `n_samples`, `seed`, `min_flux`, `max_attempts` | `1000`, `None`, `False`, `100` | Same in both implementations or no evidence favoured a different value. | | `flux_tol` (in `find_good_reactions`) | `1e-9` | Python-only helper (supports target selection of the random-objective method); no cross-implementation comparison. | @@ -49,14 +49,14 @@ Only MATLAB has tINIT (`runINIT`, `getINITModel`); raven-toolbox implements ftIN | Parameter | Default | How determined | |---|---|---| | `prod_weight` | `0.5` | Literature: [Ågren et al. 2012](https://doi.org/10.1371/journal.pcbi.1002518) (the original INIT paper). | -| `allow_excretion` | `False` | Reported to have no effect at the default `prod_weight` in the three INIT entry points that existed at the time; no measurement is recorded on this site. raven-toolbox has dropped the two tINIT ones. | +| `allow_excretion` | `False` | Recorded, without a measurement, as having no effect at the default `prod_weight` in the three INIT entry points that existed when the value was set; no measurement is recorded on this site. raven-toolbox has dropped the two tINIT ones. | | `mip_gap` | `None` in Python (solver default); `0.0004` in MATLAB | Acts on the step-0 solve and on any step without a schedule of MILP runs; a named series such as `1+1` ignores it, because every step runs its own schedule. Measured on Human-GEM (HCT116) in the step-0 solve: the time is flat at 24 to 25 s, the Jaccard index to the result at the tightest gap (0.0002) is 0.9976 at 0.001 and 0.9908 at 0.01, and falls below 0.97 from 0.03. The Python default `None` needs no change. | -| `time_limit` | `None` in Python; `5000` seconds in MATLAB | RAVEN's `ftINIT` sets the Gurobi `TimeLimit` of a MILP run without its own limit to 5000 seconds, and the step schedules of `getINITSteps` use runs of 120 s, 5000 s and 5 s. In Python, `ftinit` with a named series caps each scheduled run by its own limit and `time_limit` lowers that cap further; `time_limit=None` leaves an ad-hoc step or a `run_ftinit` call uncapped, but no scheduled run. The task gap-fill of `ftinit` is a separate MILP with a fixed limit of 300 s per task and does not read `time_limit`. Measured on Human-GEM HCT116 (single runs): step-0 solves took 23 to 30 s, and full pipelines 81 to 119 s with scaling and 157 s without. In the robustness runs of [ftINIT parameter calibration and input robustness](parameter-tuning/studies/init-param-calibration.md) (degraded input, with tasks), the gap-fill of one 90 %-dropout model ran longer than 75 minutes; the time limit of that run is not recorded. `None` is the default. The study ran its full pipelines at `time_limit=600` and its robustness runs at 120 s, and the working range for hard or noisy input is 120 to 600 s per step. | +| `time_limit` | `None` in Python; `5000` seconds in MATLAB | RAVEN's `ftINIT` sets the Gurobi `TimeLimit` of a MILP run without its own limit to 5000 seconds, and the step schedules of `getINITSteps` use runs of 120 s, 5000 s and 5 s. In Python, `ftinit` with a named series caps each scheduled run by its own limit and `time_limit` lowers that cap further; `time_limit=None` leaves an ad-hoc step or a `run_ftinit` call uncapped, but no scheduled run. The task gap-fill of `ftinit` is a separate MILP with a fixed limit of 300 s per task and does not read `time_limit`. Measured on Human-GEM HCT116 (single runs): step-0 solves took 23 to 30 s, and full pipelines 81 to 119 s with scaling and 157 s without. In the robustness runs of [ftINIT parameter calibration and input robustness](parameter-tuning/studies/init-param-calibration.md) (degraded input, with tasks), the gap-fill of one 90 %-dropout model ran longer than 75 minutes; the time limit of that run is not recorded. `None` is the default. The study ran its full pipelines at `time_limit=600`, and its robustness script sets `time_limit=120`; the limit of the recorded robustness runs is not stated. The docstring of `run_ftinit` recommends 120 to 600 s per step for hard or noisy input; no measurement on this site tests that range. | | `series` | `'1+1'` | Literature: [Gustafsson et al. 2023](https://doi.org/10.1073/pnas.2217868120) (the ftINIT paper). | | `force_on` | `0.1` | RAVEN's original value; measured near-insensitive across the range 0.02–0.5. | | `big_m` | `1000.0` | The cap on each scored reaction's own big-M, as in `ftINITInternalAlg`: the big-M of a reaction is `max(min(ub, big_m), min(-lb, big_m), 0)`, raised to `force_on` where the "on" floor needs it, and an infinite bound counts as `big_m`. The cap limits the flux a reaction can carry when it is on (it is the largest big-M of a scored reaction in its on/off constraint) and sets no bound on the reaction in the model. | | `eps` | `1.0` | Same in both implementations. | -| `resolve_ties`, `prove_abs_gap` | `False`, `None`, both opt-in; MATLAB has them as `resolveTies` and `proveAbsGap` at the same defaults | `resolve_ties` fixes the answer among equal-score optima in phases: it holds the score objective at its optimum, minimises the count of kept removable (negative-score) reactions, then among those minimises the summed id rank of the kept ones (a merged reaction is ranked by the smallest original id of its group), then takes the fluxes with the least total absolute flux (relative and absolute gap 1e-6), then those with the least id-rank-weighted backward flux. The last ftINIT step skips the last two phases, because no later step reads its fluxes. Measured on Human-GEM v2.0.0-38 (one cell line, seeds 1234 and 7): with the default escalation the seed-to-seed difference is 110 reactions and 17 predicted essential genes. `resolve_ties=True` gives identical reaction sets at a fixed seed and reduces the seed-to-seed difference to 41 reactions and 1 gene, at about 3.6 times the run time (about 14 min against 4 min at `time_limit=1800`). With `prove_abs_gap=1.0` as well, the difference is 1 reaction and 0 genes, at about 26 times the baseline run time (about 100 min), of which step 1 takes about 60 % (3300 to 3650 s), mostly in its two flux phases, and step 2 another 1650 to 2500 s; no build without `resolve_ties` was run with `prove_abs_gap`. The flux phases (`phase2c-minflux`, `phase2d-direction`) reach the time limit in most builds, and a warning names each phase adopted at the limit. A tie-break that finds no incumbent in phase 2a or 2b is dropped without a warning, and the baseline escalation raises no warning and ends 4 to 12 below the step 1 optimum, so a build without a warning is not necessarily proven or tie-broken. | +| `resolve_ties`, `prove_abs_gap` | `False`, `None`, both opt-in; MATLAB has them as `resolveTies` and `proveAbsGap` at the same defaults | `resolve_ties` fixes the answer among equal-score optima in phases: it holds the score objective at its optimum, minimises the count of kept removable (negative-score) reactions, then among those minimises the summed id rank of the kept ones (a merged reaction is ranked by the smallest original id of its group), then takes the fluxes with the least total absolute flux (relative and absolute gap 1e-6), then those with the least id-rank-weighted backward flux. The last ftINIT step skips the last two phases, because no later step reads its fluxes. Measured on Human-GEM v2.0.0-38 (one cell line, seeds 1234 and 7): with the default escalation the seed-to-seed difference is 110 reactions and 17 predicted essential genes. `resolve_ties=True` gives identical reaction sets at a fixed seed and reduces the seed-to-seed difference to 41 reactions and 1 gene, at about 3.6 times the run time (about 14 min against 4 min at `time_limit=1800`). With `prove_abs_gap=1.0` as well, the difference is 1 reaction and 0 genes, at about 26 times the baseline run time (about 100 min); in the four sweep builds with `prove_abs_gap=1.0`, step 1 took 3305 to 3650 s (53 to 73 % of a build), mostly in its two flux phases, and step 2 another 1650 to 2500 s; no build without `resolve_ties` was run with `prove_abs_gap`. The flux phases (`phase2c-minflux`, `phase2d-direction`) reach the time limit in most builds, and a warning names each phase adopted at the limit. A tie-break that finds no incumbent in phase 2a or 2b is dropped without a warning. The escalation raises no warning either: in the four `resolve_ties=True` runs without `prove_abs_gap` of the `prove_abs_gap` sweep, its step 1 objective ended 4.0 to 12.0 below the proven optimum. A build without a warning is therefore not necessarily proven or tie-broken. | | `factor`, `max_score`, `min_score` (in `gene_scores_from_expression`) | `5.0`, `10.0`, `-5.0` | RAVEN's own formula. An attribution to "Wang et al. 2012" was checked and could not be confirmed; no source in either implementation cites a paper for it. | | `score_reactions_from_genes` and `classify_reactions` arguments | various | Same in both implementations or standard practice. | @@ -70,7 +70,7 @@ RAVEN MATLAB: `fillGaps` (connectivity mode), ported as `connect_blocked_reactio |---|---|---| | `epsilon` (in `fill_gaps_fast_lp`) | `0.0001` | Literature: [Thiele et al. 2014](https://doi.org/10.1093/bioinformatics/btu321) (the fastGapFill paper). | | `threshold` (in `connect_blocked_reactions`) | `1.0` | The flux a reaction must be able to exceed to count as unblocked. Same as the fixed threshold of 1 in RAVEN's `fillGaps`, and suited to models in the RAVEN convention, where exchange reactions have bounds of ±1000. Not measured. | -| `penalty`, `allow_net_production` (in `connect_blocked_reactions`) | `1.0`, `False` | Measured or standard practice for reconstruction use (as against curation use). | +| `penalty`, `allow_net_production` (in `connect_blocked_reactions`) | `1.0`, `False` | No measurement recorded. `penalty=1.0` makes the MILP minimise the number of added reactions. `allow_net_production=False` keeps the steady-state balance `S @ v == 0`, the stricter test; `True` relaxes it to `S @ v >= 0`. | | `big_m` (in `fill_gaps_milp`) | `1000.0` | Matches RAVEN's ±1000 bound convention. It caps an unbounded template reaction only when the merged model has no finite bound; otherwise the largest finite bound does. | Full detail: [Benchmarks of `fseof`, `remove_genes` and other defaults](parameter-tuning/benchmarks.md) @@ -85,13 +85,13 @@ RAVEN MATLAB: `getModelFromHomology`, `getBlast`, `getDiamond`. raven-toolbox: ` | `evalue` (`run_diamond`) | `1e-3` | Matches MATLAB's `getDiamond`, which passes no `-evalue` flag and so inherits DIAMOND's own default. The value is deliberately different from `run_blast`; each aligner follows its own MATLAB call. | | `threads` | every CPU the process may run on (Python); the cores assigned to MATLAB (MATLAB) | A 500-query BLAST subset returned the same 2,469 hits at one thread and at four, in 45.2 s and 23.7 s (1.9 times faster). Both sides use all available cores. | | `max_evalue` | `1e-30` | The score was identical from `1e-4` to `1e-50` on the two organisms tested (at identity 40 % and length 200) and fell at `1e-100`. That identity and alignment length already exclude whatever a looser E-value would admit is an inference, not checked against the hit tables. | -| `min_align_len` | `100`; RAVEN 2.11.3 used `200` | Measured against the KEGG ortholog reference for 4 organisms of increasing distance from *S. cerevisiae*: compared with `200`, a length of 100 raised the score (F0.5) by 0.010 to 0.020, with 3 to 4 points more recall and at most 0.6 points less precision; 50 and 100 gave the same score. No OMA scores were reported for length. RAVEN 3's `getModelFromHomology` uses `minLen` 100 too. | +| `min_align_len` | `100`; RAVEN 2.11.3 used `200` | Measured against the KEGG ortholog reference for 4 organisms of increasing distance from *S. cerevisiae*: compared with `200`, a length of 100 raised the score (F0.5) by 0.010 to 0.020, with 3 to 4 more recall (percentage points of recall, or per cent more correct matches; the values are not tabulated and the unit is not recorded) and at most 0.6 percentage points less precision; 50 and 100 gave the same score. No OMA scores were reported for length. RAVEN 3's `getModelFromHomology` uses `minLen` 100 too. | | `min_identity` | `40` | Same study: the setting with the largest effect on the score. With a wrongly transferred reaction weighted as worse than a missing one (F0.5), the optimum was 35 or 40 % against KEGG and 35 or 45 % against OMA, depending on the organism. | | `bidirectional` | `True` | Same in both implementations. | -An arm of the study that scored against curated GEMs (hanpo-GEM, rhto-GEM) built by the same reconstruction method was retired because the result was circular: agreement with the curated model peaked at exactly the build settings of that model and collapsed one step past them. +A part of the study that scored against curated GEMs (hanpo-GEM, rhto-GEM) built by the same reconstruction method was retired because the result was circular: agreement with the curated model peaked at exactly the build settings of that model and collapsed one step past them. -Full detail: [Homology cut-off calibration](parameter-tuning/studies/homology-cutoff-calibration.md) (the primary measurement, including the retired curated-GEM arm) +Full detail: [Homology cut-off calibration](parameter-tuning/studies/homology-cutoff-calibration.md) (the primary measurement, including the curated-GEM comparison that the study does not use) ## KEGG-based reconstruction @@ -99,11 +99,11 @@ RAVEN MATLAB: `getKEGGModelForOrganism` (the exact internal name of the KO-assig | Parameter | Default | How determined | |---|---|---| -| `cutoff` (in `assign_kos`) | `1e-30`; RAVEN 2.11.3 used `1e-50` | Measured against real KEGG gene→KO annotations across 4 organisms of varying study depth: `1e-50` lies inside the E-value distribution of the matched pairs and removes 1 to 10 % of them in three organisms and 10 to 50 % in *M. genitalium*. Precision rose with every stricter cut-off (*S. cerevisiae* 0.72 at `1e-30`, 0.78 at `1e-50`) at a cost in recall. Together with `min_score_ratio_g` 0.9, the default raised recall in all four organisms and kept F1 within 0.01. RAVEN 3's `getKEGGModelForOrganism` uses `cutOff` `10^-30` too. | +| `cutoff` (in `assign_kos`) | `1e-30`; RAVEN 2.11.3 used `1e-50` | Measured against real KEGG gene→KO annotations across 4 organisms of varying study depth: `1e-50` lies inside the E-value distribution of the matched pairs and removes 1 to 10 % of them in three organisms and 10 to 50 % in *M. genitalium*. Precision rose with every stricter cut-off (*S. cerevisiae* 0.72 at `1e-30`, 0.78 at `1e-50`); recall fell by at most 0.03 down to `1e-30` and fell further at stricter cut-offs. Together with `min_score_ratio_g` 0.9, the default raised recall in all four organisms and kept F1 within 0.01. The F1 of the default was computed from precision and recall rounded to two decimals, which alone moves it by up to 0.005, so a difference of 0.01 is within the rounding. RAVEN 3's `getKEGGModelForOrganism` uses `cutOff` `10^-30` too. | | `min_score_ratio_g` (in `assign_kos`) | `0.9`; RAVEN 2.11.3 used `0.8` | Same study: from 0.80 to 0.95 it raised precision by 0.05 to 0.10 at a recall loss of 0.01 to 0.03, and from 0.50 to 0.80 by 0.15 to 0.27; 0.9 itself was not swept. RAVEN 3's `getKEGGModelForOrganism` uses `minScoreRatioG` `0.9` too. | -| `min_score_ratio_ko` (in `assign_kos`) | `0.3` | Same study: from 0.0 to 0.5 it changed precision by 0.01 to 0.03 and recall not at all; kept at the RAVEN value. | +| `min_score_ratio_ko` (in `assign_kos`) | `0.3` | Same study: at `min_score_ratio_g` 0.80 and `cutoff` `1e-50`, raising it from 0.0 to 0.5 changed precision by 0.01 to 0.03 and recall not at all; other combinations were not reported. Kept at the RAVEN value. | | `seq_identity` (in `build_ko_hmm`) | `0.9` | Literature: the CD-HIT recommendation for protein clustering. | -| `threads` (`run_hmmsearch`, `build_ko_hmm`) | every CPU the process may run on | Affects performance only; HMMER is deterministic across thread counts (to within insignificant float noise). | +| `threads` (`run_hmmsearch`, `build_ko_hmm`) | every CPU the process may run on | Not measured. The value assumes that HMMER gives the same hits at any thread count, with E-values that differ at most in the last floating-point places; [KEGG HMM cut-off calibration](parameter-tuning/studies/kegg-hmm-cutoff-calibration.md) did not test this. | | `keep_spontaneous`, `keep_undefined_stoich`, `keep_incomplete`, `keep_general` (`get_kegg_model_*`) | `True`, `True`, `True`, `False` | Same in both implementations or standard draft-reconstruction practice. | Full detail: [KEGG HMM cut-off calibration](parameter-tuning/studies/kegg-hmm-cutoff-calibration.md) (the primary measurement, 4 organisms against real KEGG annotations) @@ -158,14 +158,14 @@ Policy: where Python and MATLAB RAVEN disagree on a default, one value is chosen | `get_model_from_homology.min_align_len` | `100` ✓ done | `100` ✓ done (`200` in RAVEN 2.11.3) | `100` | ~~Python and MATLAB~~ done | High | Directly measured: [Homology cut-off calibration](parameter-tuning/studies/homology-cutoff-calibration.md) | | `assign_kos.cutoff` | `1e-30` ✓ done | `1e-30` ✓ done (`1e-50` in RAVEN 2.11.3) | `1e-30` | ~~MATLAB~~ done | High | Directly measured: [KEGG HMM cut-off calibration](parameter-tuning/studies/kegg-hmm-cutoff-calibration.md) | | `assign_kos.min_score_ratio_g` | `0.9` ✓ done | `0.9` ✓ done (`0.8` in RAVEN 2.11.3) | `0.9` | ~~MATLAB~~ done | High | Directly measured, same study | -| `run_ftinit.allow_excretion` | `False` ✓ done | `False` | `False` | ~~Python~~ done (2026-06-20) | High | Zero effect at default `prod_weight`, pure inconsistency; the measurement is not recorded on a page of [Parameter tuning](parameter-tuning/index.md). Within `ftinit`, the step schedule sets the value (`allow_met_secr`) | +| `run_ftinit.allow_excretion` | `False` ✓ done | `False` | `False` | ~~Python~~ done (2026-06-20) | High | Recorded, without a measurement, as having no effect at the default `prod_weight`; no measurement is recorded on a page of [Parameter tuning](parameter-tuning/index.md). Unified to remove the inconsistency. Within `ftinit`, the step schedule sets the value (`allow_met_secr`) | ### Settings resolved without unification The INIT parameters `mip_gap` and `time_limit` of `run_init` and `run_ftinit` are resolved by the genome-scale test with real expression data in [ftINIT parameter calibration and input robustness](parameter-tuning/studies/init-param-calibration.md) (Human-GEM HCT116 and Hart2015), without matching the MATLAB numbers: -- `mip_gap`: solve time is flat across the gap in the single step-0 solve (model build dominates), so a tight gap is nearly free there. A named series such as `1+1` ignores `mip_gap` and `mip_gap_abs`, because every step runs its own schedule of MILP runs, so the full pipeline gives the same result at 0.001, 0.003 and 0.01. Neither 0.001 nor 0.01 is MATLAB's `0.0004`. Keep `None` (Gurobi's own `~1e-4` default is at least as tight as the measured-good `0.001`). -- `time_limit`: RAVEN's `ftINIT` limits a MILP run to 5000 seconds when the run sets none, and its schedules use 120 s, 5000 s and 5 s runs. Python's `ftinit` with a named series caps each scheduled run by the same limits, and `time_limit` lowers the cap; `time_limit=None` leaves an ad-hoc step or a `run_ftinit` call uncapped, but no scheduled run. Measured step-0 solves took 23 to 30 s, and full pipelines 81 to 119 s with scaling and 157 s without. In the robustness setup, the gap-fill of one 90 %-dropout model ran longer than 75 minutes; the gap-fill has its own fixed limit of 300 s per task and does not read `time_limit`. The study ran its full pipelines at 600 s and its robustness runs at 120 s. Keep `None` as the code default; the docstring of `run_ftinit` states the working values, `120-600 s/step`. +- `mip_gap`: solve time is flat across the gap in the single step-0 solve, so a tight gap is nearly free there. The docstring of `run_ftinit` attributes the flat time to model construction; the study did not measure the share of model construction. A named series such as `1+1` ignores `mip_gap` and `mip_gap_abs`, because every step runs its own schedule of MILP runs, so the full pipeline gives the same result at 0.001, 0.003 and 0.01. Neither 0.001 nor 0.01 is MATLAB's `0.0004`. Keep `None` (Gurobi's own `~1e-4` default is at least as tight as the measured-good `0.001`). +- `time_limit`: RAVEN's `ftINIT` limits a MILP run to 5000 seconds when the run sets none, and its schedules use 120 s, 5000 s and 5 s runs. Python's `ftinit` with a named series caps each scheduled run by the same limits, and `time_limit` lowers the cap; `time_limit=None` leaves an ad-hoc step or a `run_ftinit` call uncapped, but no scheduled run. Measured step-0 solves took 23 to 30 s, and full pipelines 81 to 119 s with scaling and 157 s without. In the robustness setup, the gap-fill of one 90 %-dropout model ran longer than 75 minutes; the gap-fill has its own fixed limit of 300 s per task and does not read `time_limit`. The study ran its full pipelines at 600 s; its robustness script sets 120 s, and the limit of the recorded robustness runs is not stated. Keep `None` as the code default. The docstring of `run_ftinit` recommends `120-600 s/step`, a range with no measured basis on this site. [ftINIT on different solvers](parameter-tuning/studies/init-solver-benchmark.md) measured GLPK honouring the 900 s limit: both steps of the full pipeline hit it, the run took 2182 s and kept 6758 reactions (Jaccard 0.892 against Gurobi). The `scipy` interface (HiGHS) fails because it supports continuous variables only. Gurobi is the backend the genome-scale numbers above were measured on, and the only one that completed without a step hitting the limit. @@ -173,7 +173,7 @@ The INIT parameters `mip_gap` and `time_limit` of `run_init` and `run_ftinit` ar | Parameter | Python | MATLAB | Why | |---|---|---|---| -| `random_sampling.replace_max_bound` | `False` | `True` | Applying MATLAB's `True` inside cobrapy and optlang makes the sampler unbounded on standard RAVEN-convention models (measured; see [Flux sampling convergence and settings](parameter-tuning/studies/sampling-convergence-calibration.md)). Whether MATLAB's own solver path handles `True` safely was not tested here; in both cases the difference is a solver-stack constraint. | +| `random_sampling.replace_max_bound` | `False` | `True` | With every bound of 1000 replaced by infinity, the random-objective LP becomes unbounded, because the objective can grow without limit through any reaction that is then unconstrained ([Flux sampling convergence and settings](parameter-tuning/studies/sampling-convergence-calibration.md) gives this mechanism; it records no run with `True`, and its run with `False` completed 200 samples). Whether MATLAB's own solver path handles `True` was not tested here. | | `random_sampling.loopless_good_reactions` | `True` (loopless FVA) | heuristic (exclude FVA ≥ 999) | The two are different techniques. Loopless FVA excludes only reactions that take part in infeasible cycles, and the MATLAB heuristic also excludes reactions that legitimately reach capacity. Porting the loopless technique to MATLAB is a larger project than changing a default. | | `merge_models.match_by` | `'name'` | `'metNames'` | Same semantic field (metabolite display name) under a different field name, which results from the cobrapy and COBRA Toolbox schemas; the field name is not tunable. | | `run_diamond.evalue` | `1e-3` | `1e-3` (implicit) | Both agree at `1e-3`, matching the DIAMOND default. `getDiamond` passes no `-evalue` flag, so it inherits the DIAMOND default and does not follow the explicit `1e-4` of `getBlast`. `run_blast` and `run_diamond` deliberately stay unpaired: each follows the MATLAB call of its own aligner. | From 034f3dc064cdac23abd2925c81cb59bfa5af3d68 Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Sun, 1 Nov 2026 03:48:33 +0100 Subject: [PATCH 05/20] Correct the parameter-tuning studies against reruns and name their scripts The numbers of the init, benchmark, sampling, KEGG and homology studies are replaced by the values measured on 2026-11-01 with the scripts added to raven-toolbox under scripts/parameter_tuning/. Each study states its inputs, software and what is not reproducible. tuned-parameters.md and the guide pages follow the studies. --- docs/guide/homology.md | 4 +- docs/guide/init.md | 2 +- docs/guide/kegg.md | 8 +- docs/parameter-tuning/benchmarks.md | 84 ++++++-- .../studies/ftinit-determinism.md | 47 +++-- .../studies/homology-cutoff-calibration.md | 64 +++--- .../studies/humangem-validation.md | 20 +- .../studies/init-param-calibration.md | 74 ++++--- .../studies/init-solver-benchmark.md | 23 ++- .../studies/kegg-hmm-cutoff-calibration.md | 185 ++++++++++-------- .../sampling-convergence-calibration.md | 168 +++++++++------- docs/tuned-parameters.md | 59 +++--- scripts/known_names.txt | 2 + 13 files changed, 441 insertions(+), 299 deletions(-) diff --git a/docs/guide/homology.md b/docs/guide/homology.md index da720a0..efba081 100644 --- a/docs/guide/homology.md +++ b/docs/guide/homology.md @@ -106,7 +106,7 @@ Both call the same BLAST+ executables with the same parameters, so the hit count Without `threads`, `run_blast`, `run_diamond` and `run_supporth` use every CPU the process may run on. In MATLAB, `getBlast`, `getDiamond` and `runSuppOrth` use the cores assigned to MATLAB (`feature('numcores')`); `runSuppOrth` is the one of the three with a `threads` option. -`getDiamond` and `run_diamond` are drop-in alternatives that search with DIAMOND instead. DIAMOND indexes the database and searches in reduced amino-acid alphabets, which made it 10.5 and 19.7 times faster than BLASTP on two full proteome pairs (34 s against 358 s and 40 s against 788 s in [Homology cut-off calibration](../parameter-tuning/studies/homology-cutoff-calibration.md)), and less sensitive for distant homologs, where the seeds it uses are less likely to match. For a template within the same genus the difference is small; for a template several hundred million years away, BLASTP finds pairs DIAMOND misses. +`getDiamond` and `run_diamond` are drop-in alternatives that search with DIAMOND instead. DIAMOND ran with `--more-sensitive` (the `run_diamond` default) and was 11.7 and 14.3 times faster than BLASTP on two full proteome pairs (12 s against 140 s and 14 s against 200 s in [Homology cut-off calibration](../parameter-tuning/studies/homology-cutoff-calibration.md)). It returned fewer hits, 32,192 against 54,513 for *K. lactis* against *S. cerevisiae*. After the default filters (identity 40, length 200, E-value 1e-30) the two aligners kept 87.9 to 92.3 % of the same pairs on four targets (Jaccard index). `makeFakeBlastStructure` and `make_ortholog_hits` wrap orthology assignments from another source (OrthoFinder, OMA, a published table) in the structure the transfer step expects, so no search is run. @@ -157,7 +157,7 @@ The draft is smaller than the template because reactions whose genes have no acc Three cut-offs control which hits are accepted, and tightening any of them shrinks the draft. -`maxE` / `max_evalue` (`1e-30`) is the maximum BLAST E-value, the number of hits of at least this quality expected by chance in a database this size. The default is strict by BLAST standards, where `1e-5` is a common threshold, because transferring a reaction on a marginal hit adds a claim about metabolism that nothing downstream questions. +`maxE` / `max_evalue` (`1e-30`) is the maximum BLAST E-value, the number of hits of at least this quality expected by chance in a database this size. The default is strict by BLAST standards, where `1e-5` is a common threshold, because transferring a reaction on a marginal hit adds a claim about metabolism that nothing downstream questions. In four organisms the F0.5 score against KEGG was identical from `1e-4` to `1e-30`, was lower by up to 0.006 at `1e-50` and by 0.020 to 0.063 at `1e-100` ([Homology cut-off calibration](../parameter-tuning/studies/homology-cutoff-calibration.md)). `minLen` / `min_align_len` (`100`) is the minimum aligned length in residues. It rejects hits that align well over a short conserved domain while the rest of the protein is unrelated. The value was measured against KEGG orthology assignments across four organisms ([Homology cut-off calibration](../parameter-tuning/studies/homology-cutoff-calibration.md)): 50 and 100 gave the same score, 150 a score at most 0.003 lower, and 200 and 300 lower scores. At 200 the loss came mainly from recall; precision and recall were not reported for 300. diff --git a/docs/guide/init.md b/docs/guide/init.md index bce80e9..7a83eb5 100644 --- a/docs/guide/init.md +++ b/docs/guide/init.md @@ -30,7 +30,7 @@ What they share is general RAVEN machinery, nothing specific to the method: `che ## Reference run -The outputs in this workflow come from one run made by hand. Other guide pages are re-executed by the [Executed examples](index.md#executed-examples) workflow whenever a guide page, the example data, the runner, the workflow or a submodule changes, and once a week; the outputs here are not, because preparing Human-GEM took 47 minutes in MATLAB and 37 minutes in Python, and the MATLAB run produced a 174 MB artefact, which no documentation build should attempt. The numbers come from one real run (Human-GEM `main` at commit `6812be2a`, RAVEN commit `2d0630e3`, raven-toolbox commit `94f39407`, Gurobi 13.0.1, four cores, 2026-10-30) and are quoted with their wall-clock times for planning. +The outputs in this workflow come from one run made by hand. Other guide pages are re-executed by the [Executed examples](index.md#executed-examples) workflow whenever a guide page, the example data, the runner, the workflow or a submodule changes, and once a week; the outputs here are not, because preparing Human-GEM took 47 minutes in MATLAB and 37 minutes in Python, and the MATLAB run produced a 174 MB artefact, which no documentation build should attempt. The numbers come from one real run (Human-GEM `main` at commit `6812be2a`, RAVEN commit `2d0630e3`, raven-toolbox commit `94f39407`, Gurobi (MATLAB links the 13.0.1 module library; Python most likely loads the library of gurobipy 13.0.3), four cores, 2026-10-30) and are quoted with their wall-clock times for planning. The same MATLAB workflow, as the model's own documentation presents it and including the comparison of the extracted models, is in the [Human-GEM guide](https://sysbiochalmers.github.io/Human-GEM-guide/gem_extraction/), which is maintained alongside the model. diff --git a/docs/guide/kegg.md b/docs/guide/kegg.md index 8da87d8..47629c7 100644 --- a/docs/guide/kegg.md +++ b/docs/guide/kegg.md @@ -4,7 +4,7 @@ [Reconstruction from homology](homology.md) needs a template model of a related organism. KEGG needs none: its orthology groups (KOs) are already linked to reactions, so annotating a genome with KOs gives a draft directly. Which method applies depends on one thing: whether the organism is already in KEGG. -The documentation build does not run these examples. A KEGG reconstruction downloads tens of megabytes and takes minutes, so the page is not executed with the others. The Python numbers come from raven-toolbox 3.0.0b1 with the `kegg118` artefacts (about 2.5 minutes per call on a recent workstation, with or without the cache). +The documentation build does not run these examples. A KEGG reconstruction downloads tens of megabytes and takes minutes, so the page is not executed with the others. The Python numbers come from raven-toolbox 3.0.0b1 with the `kegg118` artefacts. ## Functions @@ -49,7 +49,7 @@ print(f"{len(model.reactions)} rxns, {len(model.metabolites)} mets, " 1357 rxns, 1502 mets, 838 genes ``` -The artefacts (a reference model and three tables, about 47 MB) are fetched from the `kegg118` raven-data release on first use and cached; see [Download data and binaries](../installation/data-and-binaries.md) for where, and how to fetch them ahead of a batch run. The first run takes a few minutes and later runs take much the same time, because assembling the draft from the tables is the slow part and the download is not. +The artefacts (a reference model and three tables, about 49 MB) are fetched from the `kegg118` raven-data release on first use and cached; see [Download data and binaries](../installation/data-and-binaries.md) for where, and how to fetch them ahead of a batch run. The first run takes a few minutes and later runs take much the same time, because assembling the draft from the tables is the slow part and the download is not. ::: :::: @@ -81,11 +81,11 @@ hanpo_model = get_kegg_model_from_sequences_with_artefacts("hanpo.faa", domain=" ::: :::: -This method is expensive. The eukaryotic HMM library alone is 129 MB compressed, and one timed `hmmsearch` of a proteome (*S. cerevisiae*, 6021 queries) against the eukaryotic library took about 10.5 minutes on 12 threads. It also needs HMMER, which has no bundle for every platform; [Download data and binaries](../installation/data-and-binaries.md) has the matrix. Without `threads`, `run_hmmsearch`, `get_kegg_model_from_sequences`, `build_ko_hmm` and `build_hmm_library` use every CPU the process may run on; `getKEGGModelForOrganism` runs `hmmsearch` on the cores assigned to MATLAB (`feature('numcores')`) and has no `threads` option. +This method is expensive. The eukaryotic HMM library alone is 135 MB compressed, and one timed `hmmsearch` of a proteome (*S. cerevisiae*, 6021 sequences) against the eukaryotic library (2698 profiles) took 89.7 s on 16 threads and 88.6 s on 12 threads (AMD EPYC 9354). It also needs HMMER, which has no bundle for every platform; [Download data and binaries](../installation/data-and-binaries.md) has the matrix. Without `threads`, `run_hmmsearch`, `get_kegg_model_from_sequences`, `build_ko_hmm` and `build_hmm_library` use every CPU the process may run on; `getKEGGModelForOrganism` runs `hmmsearch` on the cores assigned to MATLAB (`feature('numcores')`) and has no `threads` option. With a `fastaFile`, the organism id only names the model; the search uses no phylogenetic distance and ignores `maxPhylDist`. -Three cut-offs shape the assignment. `cutOff` is the HMMER significance a hit needs at all. `minScoreRatioKO` drops genes that fit a KO far worse than the best gene in it, and `minScoreRatioG` drops KOs that fit a gene far worse than that gene's best KO; the two prune the same matrix from opposite sides. All three were measured in [KEGG HMM cut-off calibration](../parameter-tuning/studies/kegg-hmm-cutoff-calibration.md); `cutOff` and `minScoreRatioG` changed from the RAVEN 2 values. At `minScoreRatioG` 0.8 and `cutOff` 1e-50, raising `minScoreRatioKO` from 0.0 to 0.5 changed precision by 0.01 to 0.03 and recall not at all, and it keeps its original value of 0.3. +Three cut-offs shape the assignment. `cutOff` is the HMMER significance a hit needs at all. `minScoreRatioKO` drops genes that fit a KO far worse than the best gene in it, and `minScoreRatioG` drops KOs that fit a gene far worse than that gene's best KO; the two prune the same matrix from opposite sides. All three were measured in [KEGG HMM cut-off calibration](../parameter-tuning/studies/kegg-hmm-cutoff-calibration.md); `cutOff` and `minScoreRatioG` changed from the RAVEN 2 values. At `minScoreRatioG` 0.8 and `cutOff` 1e-50, raising `minScoreRatioKO` from 0.0 to 0.5 raised precision by 0.00 to 0.03 and lowered recall by 0.00 to 0.01, and it keeps the RAVEN 2 value of 0.3. ## Contents of a KEGG draft diff --git a/docs/parameter-tuning/benchmarks.md b/docs/parameter-tuning/benchmarks.md index fcc5eb1..70d8473 100644 --- a/docs/parameter-tuning/benchmarks.md +++ b/docs/parameter-tuning/benchmarks.md @@ -1,18 +1,27 @@ # Benchmarks of `fseof`, `remove_genes` and other defaults -These benchmarks record parameters whose defaults were measured or checked without a study of their own. Two of them were run with raven-toolbox commit `94f39407` on 2026-10-30: the effect of `flux_eps` on the targets of `fseof`, and the handling of reactions without an enzyme in `remove_genes`. A table lists literature and convention values, checked on 2026-06-20, with what supports each of them. +These benchmarks record parameters whose defaults were measured or checked without a study of their own. All were run with raven-toolbox commit `94f39407` on 2026-11-01: the effect of `flux_eps` on the targets of `fseof`, the handling of reactions without an enzyme in `remove_genes`, a sweep of three `fseof` parameters and a test of `connect_blocked_reactions.threshold`. A table lists literature and convention values with what supports each of them. The parameters with a measurement campaign of their own have studies listed under [Parameter tuning](index.md). [Tuned parameter defaults](../tuned-parameters.md) gives the current value and a one-line reason for every parameter of both implementations. ## Set-up -The `fseof` benchmark ran on iJO1366 (2,583 reactions) with the biomass reaction of the model as objective and `EX_succ_e` (succinate exchange) as target. The other arguments were `n_steps` 10, `max_fraction` 0.9, `correlation_threshold` 0.9 and the default `equality=False`, which sets each enforced flux as the lower bound of the target reaction. +The `fseof` benchmarks ran on iJO1366 (2,583 reactions) with the biomass reaction of the model as objective and `EX_succ_e` (succinate exchange) as target. In the `flux_eps` table the enforced flux ran from 1.5387 to 15.387 in 10 levels, and the unconstrained growth rate was 0.98237 h⁻¹. The other arguments were `n_steps` 10, `max_fraction` 0.9, `correlation_threshold` 0.9 and the default `equality=False`, which sets each enforced flux as the lower bound of the target reaction. `fseof` solves each enforced level in two steps, as RAVEN's `FSEOF` does: it maximises the objective, then minimises the sum of fluxes with the objective held at 0.999999 of that optimum. A reaction is a target when its absolute flux exceeds `1e-10` at some level, the standard deviation of its absolute flux is above 0, the absolute correlation between its absolute flux and the enforced flux reaches `correlation_threshold`, and the slope of the linear fit of its absolute flux against the enforced flux is not 0. A target with a positive slope is an amplify target. A target with a negative slope is a knockout target when its last absolute flux is at most `1e-8`, and a knockdown target otherwise. `fseof` and RAVEN's `FSEOF` use these thresholds as fixed values. -The `remove_genes` benchmark ran on e_coli_core (95 reactions) from the cobrapy distribution, with bounds as shipped, Gurobi 13.0.1, raven-toolbox commit `94f39407` and RAVEN commit `2d0630e3`. It deleted `b1779`, the only gene of GAPD (glyceraldehyde-3-phosphate dehydrogenase), with each setting of `blocked_reactions` in `remove_genes` and of `removeBlockedRxns` in RAVEN's `removeGenes`. The predicted growth is the optimal flux through the biomass reaction of the model, in h⁻¹. The unmodified model has 95 reactions and predicts a growth rate of 0.874 h⁻¹. +The `remove_genes` benchmark ran on e_coli_core (95 reactions) from the cobrapy distribution, with bounds as shipped, raven-toolbox commit `94f39407` and RAVEN commit `2d0630e3`. It deleted `b1779`, the only gene of GAPD (glyceraldehyde-3-phosphate dehydrogenase), with each setting of `blocked_reactions` in `remove_genes` and of `removeBlockedRxns` in RAVEN's `removeGenes`. The predicted growth is the optimal flux through the biomass reaction of the model, in h⁻¹. The unmodified model has 95 reactions and predicts a growth rate of 0.874 h⁻¹. Each call ran once. -The checks of literature and convention values ran on 2026-06-20 with yeast-GEM (4,102 reactions), iJO1366 (2,583) and e_coli_core (95). The table names the model for the `fseof` rows only. +The Python runs used Python 3.14.2, cobra 0.32.1, optlang 1.9.1 and Gurobi 13.0.3 (gurobipy wheel); MATLAB R2024b ran RAVEN with Gurobi 13.0.1 selected through `setRavenSolver('gurobi')`. The `flux_eps` runs were repeated three times per value, each on a freshly read model, and gave the same target set and labels every time. + +The sweep of `fseof` ran on iJO1366 with the settings above and Gurobi 13.0.3, changing one of `n_steps`, `max_fraction` and `correlation_threshold` at a time, with three runs per value on freshly read models. All three runs of every configuration gave the same counts and the same added and dropped targets. Two categories of target are reported, and neither measures error: + +1. **Added:** the target appears at the swept value and not in the default run (the set difference from the default). An added target is no evidence of error. +2. **Noise:** the standard deviation of its absolute flux across the scan is below `1e-6`. + +The default run contains 19 noise tracks, all knockdown targets with a standard deviation of 5.08e-7, a slope of -1.15e-7 and a correlation of -1: `EGMEACPR`, `OGMEACPD`, `OPMEACPR`, `OPMEACPS`, `MALCOAMT`, `OGMEACPR`, `OGMEACPS`, `DM_amob_c`, `EX_meoh_e`, `AMAOTr`, `AOXSr2`, `BTS5`, `DBTS`, `EPMEACPR`, `I2FE2SR`, `MEOHtex`, `MEOHtrpp`, `OPMEACPD` and `PMEACPE` (biotin synthesis and methanol export). The classification rule above keeps them, because it requires a standard deviation above 0. + +The test of `connect_blocked_reactions.threshold` used a synthetic network: a draft with an import of A of at most 0.5, the reaction A to B, and B as a dead end, and a template with the exchange `EX_B`. The other arguments were at their defaults (`min_flux` 0.1). Each value ran once, with Gurobi 13.0.3. ## Results @@ -24,34 +33,65 @@ The checks of literature and convention values ran on 2026-06-20 with yeast-GEM | `1e-7` | 18 | 408 | | `1e-6` (default) | 18 | 408 | -Of the 408 reactions, 406 are knockdown targets and 2 are knockout targets. +Of the 408 reactions, 406 are knockdown targets and 2 are knockout targets. Three runs per value gave the same set of targets. + +### `fseof` targets at other values of three parameters + +iJO1366, target `EX_succ_e`. The last two columns list the targets that the swept value adds to the default set and drops from it. + +| Parameter | Value | Amplify targets | Knockdown and knockout targets | Added (amplify / knockdown and knockout) | Dropped (amplify / knockdown and knockout) | +|---|---|---|---|---|---| +| (defaults) | `n_steps` 10, `max_fraction` 0.9, `correlation_threshold` 0.9 | 18 | 408 | - | - | +| `n_steps` | 5 | 18 | 410 | 0 / 3 (`PGI`, `ACt2rpp`, `ACt4pp`) | 0 / 1 (`PDH`) | +| `n_steps` | 20 | 16 | 412 | 0 / 4 (`EX_co2_e`, `CO2tex`, `CO2tpp`, `GLYCTO2`) | 2 / 0 (`FUM`, `MDH`) | +| `max_fraction` | 0.5 | 15 | 414 | 1 / 6 (`SUCOAS`; `EX_co2_e`, `CO2tex`, `CO2tpp`, `ICDHyr`, `TKT2`, `AKGDH`) | 4 / 0 (`FUM`, `MDH`, `HEX7`, `XYLI2`) | +| `max_fraction` | 0.99 | 18 | 408 | 0 / 1 (`PGI`) | 0 / 1 (`PDH`) | +| `correlation_threshold` | 0.7 | 22 | 416 | 4 / 8 (`FRD3`, `NADH18pp`, `ICL`, `MALS`; `EX_co2_e`, `CO2tex`, `CO2tpp`, `ICDHyr`, `AKGDH`, `PGI`, `TKT2`, `GLYCTO2`) | 0 / 0 | +| `correlation_threshold` | 0.95 | 16 | 406 | 0 / 0 | 2 / 2 (`FUM`, `MDH`; `PDH`, `RPE`) | + +Every swept value changes the target set. None of the added targets is a noise track: they are added, and the noise category does not apply to them. The four added amplify targets at `correlation_threshold` 0.7 have an absolute correlation of 0.71 to 0.72 in the default scan and a standard deviation of 0.62 to 4.4. Whether any added target is biologically wrong was not assessed. At `max_fraction` 0.5, `FBA3` and `PFK_3` change from knockout to knockdown targets; at 0.99, the 19 noise targets change from knockdown to knockout targets. ### `remove_genes` after deletion of `b1779` from e_coli_core | Call | Reactions left | Predicted growth (h⁻¹) | GAPD | |---|---|---|---| -| `remove_genes(..., blocked_reactions="constrain")` (default) | 95 | 0.000 | kept, bounds `(0, 0)` | +| `remove_genes(..., blocked_reactions="constrain")` | 95 | 0.000 | kept, bounds `(0, 0)` | +| `remove_genes(...)` without the keyword (default) | 95 | 0.000 | kept, bounds `(0, 0)` | | `remove_genes(..., blocked_reactions="remove")` | 94 | 0.000 | deleted | -| `remove_genes(..., blocked_reactions="keep")` | 95 | 0.874 | kept, bounds unchanged | +| `remove_genes(..., blocked_reactions="keep")` | 95 | 0.874 | kept, bounds `(-1000, 1000)`, empty gene rule | | `removeGenes(model, genes)` (`removeBlockedRxns` false, default) | 95 | 0.000 | kept, bounds `[0 0]` | | `removeGenes(model, genes, 'removeBlockedRxns', true)` | 94 | 0.000 | deleted | +| unmodified model | 95 | 0.874 (MATLAB biomass flux 0.8739) | present | + +### `connect_blocked_reactions` at four values of `threshold` + +Synthetic network described in [Set-up](#set-up). + +| `threshold` | Gap filled | Reactions added | Maximum flux of the reaction A to B afterwards | +|---|---|---|---| +| 0.01 | yes | `EX_B` | 0.5 | +| 0.1 | yes | `EX_B` | 0.5 | +| 0.5 | no | none | 0 | +| 1.0 (default) | no | none | 0 | + +At every value, `cannot_connect` also lists the reversible `EX_A`, which is blocked in the draft and is never a candidate. ### Checked literature and convention values -Both implementations have the same value or behaviour for each of these parameters, and none was changed. The third column gives what supports the value; only the three `fseof` rows record a sweep. +Both implementations have the same value or behaviour for each of these parameters, and none was changed. The third column gives what supports the value; the three `fseof` rows and the `connect_blocked_reactions.threshold` row record a measurement. | Parameter | Value | Basis | |---|---|---| -| `fseof.n_steps` | `10` | Value of [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10); 5 and 20 gave the same target set on iJO1366 | -| `fseof.max_fraction` | `0.9` | [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10); values below 0.9 admitted spurious targets on iJO1366, and 0.99 found no further targets | -| `fseof.correlation_threshold` | `0.9` | [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10); 0.7 added four spurious amplify targets on iJO1366 | -| `connect_blocked_reactions.threshold` | `1.0` | Correct for models in the RAVEN convention, where exchange reactions have bounds of ±1000; RAVEN's `fillGaps` uses a fixed threshold of 1 | +| `fseof.n_steps` | `10` | Value of [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10) (value not checked against the paper); on iJO1366, 5 steps gave 18 amplify and 410 knockdown and knockout targets and 20 steps gave 16 and 412, so the target set changes with `n_steps` | +| `fseof.max_fraction` | `0.9` | [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10) (value not checked against the paper); on iJO1366, 0.5 added 7 targets and dropped 4, and 0.99 gave the same counts as 0.9 with one target added and one dropped | +| `fseof.correlation_threshold` | `0.9` | [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10) (value not checked against the paper); on iJO1366, 0.7 added 4 amplify and 8 knockdown and knockout targets, and 0.95 dropped 2 and 2 | +| `connect_blocked_reactions.threshold` | `1.0` | Suited to models in the RAVEN convention, where exchange reactions have bounds of ±1000; on one synthetic network with a maximum flux of 0.5, values of 0.01 and 0.1 filled the gap and values of 0.5 and 1.0 did not (one run per value; the test shows that a reaction must carry more than the threshold, and no correctness comparison was made) | | `connect_blocked_reactions.allow_net_production` | `False` | Keeps the steady-state balance `S @ v == 0`, the stricter test; `True` relaxes it to `S @ v >= 0` | | `fill_gaps_fast_lp.epsilon` | `1e-4` | Value of the fastGapFill paper, [Thiele et al. 2014](https://doi.org/10.1093/bioinformatics/btu321) | | `fill_gaps_milp.big_m` | `1000.0` | Matches the ±1000 bounds of the RAVEN convention | | `check_tasks.close_boundaries` | `False` | Matches RAVEN's `checkTasks`, against which published task lists were curated; `True` closes the exchange reactions of the model so that the task defines the whole boundary | -The three `fseof` rows date from 2026-06-20 and were not repeated with commit `94f39407`. +The three `fseof` rows and the `connect_blocked_reactions.threshold` row ran with commit `94f39407`. The other rows were not run. ## `flux_eps` in `fseof` @@ -59,14 +99,24 @@ The three `fseof` rows date from 2026-06-20 and were not repeated with commit `9 ## Reactions without an enzyme in `remove_genes` -`blocked_reactions` in `remove_genes`, and `removeBlockedRxns` in RAVEN's `removeGenes`, decide what happens to a reaction whose gene rule is emptied by a deletion. By default both keep such a reaction with bounds of zero, and the two defaults gave the same model. After deletion of `b1779`, that model predicts no growth, because GAPD stays in the network with no capacity to carry flux. `"remove"` and `removeBlockedRxns` true also predicted no growth; they delete GAPD, which changes the number of reactions and the set of reaction identifiers. `"keep"` leaves GAPD with an empty gene rule and its original bounds, so the model predicted the growth of the unmodified model, 0.874 h⁻¹, although the only gene of GAPD was deleted. GAPD is essential for aerobic growth on glucose in e_coli_core, so the growth predicted with `"keep"` is a false positive. The docstring of `remove_genes` states no use for `"keep"`. Editing gene annotations without changing the network is a use given for it in the documentation, without a source in the code. +`blocked_reactions` in `remove_genes`, and `removeBlockedRxns` in RAVEN's `removeGenes`, decide what happens to a reaction whose gene rule is emptied by a deletion. By default both keep such a reaction with bounds of zero, and the two defaults gave the same model. After deletion of `b1779`, that model predicts no growth, because GAPD stays in the network with no capacity to carry flux. `"remove"` and `removeBlockedRxns` true also predicted no growth; they delete GAPD, which changes the number of reactions and the set of reaction identifiers. `"keep"` leaves GAPD with an empty gene rule and unchanged bounds, so the model predicted the growth of the unmodified model, 0.874 h⁻¹, although the only gene of GAPD was deleted. GAPD is essential for aerobic growth on glucose in e_coli_core, so the growth predicted with `"keep"` is a false positive. The docstring of `remove_genes` states no use for `"keep"`. Editing gene annotations without changing the network is a use given for it in the documentation, without a source in the code. ## Choice of defaults -The measurement gives no reason to change `flux_eps` from `1e-6`. `blocked_reactions="constrain"` stays the default of `remove_genes`; it gives the model that RAVEN's `removeGenes` gives by default. `"remove"` and `"keep"` are available on request. +The measurement gives no reason to change `flux_eps` from `1e-6`. The sweep of `fseof` does not select the other three defaults: the target set changes with each swept value, by 2 to 12 targets, and no tested value leaves it unchanged. The defaults follow [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10) (not checked against the paper). The 19 noise tracks are in the target set at every setting. `blocked_reactions="constrain"` is the default of `remove_genes`; it gives the model that RAVEN's `removeGenes` gives by default. `"remove"` and `"keep"` are available on request. ## Limits -Each benchmark used one model: iJO1366 with one target for `fseof`, and e_coli_core with one deleted gene for `remove_genes`. Criterion 1 of the [evaluation methodology](index.md#evaluation-methodology) asks for a large and a small model. The solver of the `fseof` runs and the number of runs per value are not recorded, and the criterion for a spurious target in the 2026-06-20 sweeps is not recorded. The rows of the checked values without a sweep rest on the literature or on a convention. +Each benchmark used one model: iJO1366 with one target for `fseof`, e_coli_core with one deleted gene for `remove_genes`, and a synthetic network for `connect_blocked_reactions`. Criterion 1 of the [evaluation methodology](index.md#evaluation-methodology) asks for a large and a small model. Gurobi was the sole solver. Whether the targets added by a swept value are biologically wrong was not assessed, and neither was the literature basis of the cited values. The rows `connect_blocked_reactions.allow_net_production`, `fill_gaps_fast_lp.epsilon`, `fill_gaps_milp.big_m` and `check_tasks.close_boundaries` rest on the literature or on a convention; no test was run for them. + +## Source + +The scripts are in raven-toolbox under `scripts/parameter_tuning/benchmarks/`: `small.py` (`flux_eps` and `remove_genes`), `fseof_sweep.py` (the sweep of three parameters), `gapfill_threshold.py` (the test of `threshold`) and `mat_small.m` (`removeGenes` in RAVEN). For example: + +```bash +python scripts/parameter_tuning/benchmarks/small.py --repeats 3 +python scripts/parameter_tuning/benchmarks/fseof_sweep.py --repeats 3 +python scripts/parameter_tuning/benchmarks/gapfill_threshold.py +``` -raven-toolbox contains no script for these benchmarks. The unit tests `tests/test_analysis_fseof.py` and `tests/test_manipulation_remove.py` of raven-toolbox check, on small test networks, the classification thresholds of `fseof` and the three settings of `blocked_reactions`. +The unit tests `tests/test_analysis_fseof.py` and `tests/test_manipulation_remove.py` of raven-toolbox check, on small test networks, the classification thresholds of `fseof` and the three settings of `blocked_reactions`. diff --git a/docs/parameter-tuning/studies/ftinit-determinism.md b/docs/parameter-tuning/studies/ftinit-determinism.md index 3914b31..24eb43a 100644 --- a/docs/parameter-tuning/studies/ftinit-determinism.md +++ b/docs/parameter-tuning/studies/ftinit-determinism.md @@ -2,11 +2,11 @@ This study builds Human-GEM models for one cell line with raven-toolbox's `ftinit` under three settings (default, `resolve_ties=True`, and `resolve_ties=True` with `prove_abs_gap`), varies the solver seed and the template, and records how many reactions and predicted essential genes change between builds and how long each build takes. The API documentation is in `raven_toolbox.init.ftinit`. -Two properties are measured. **Determinism** is that the same input, run again or with another seed, machine or solver build, gives the same model. **Stability** is that a similar input (a slightly edited template, or a comparable sample) gives a similar model. Both parameters are opt-in (`resolve_ties=False` and `prove_abs_gap=None` by default) and target determinism. In the measurements, the seed-to-seed difference fell from 110 reactions and 17 essential genes (default) to 41 reactions and 1 gene with `resolve_ties`, and to 1 reaction and 0 genes with `prove_abs_gap=1.0` added, at about 3.6 and 26 times the default build time. Neither parameter made the model stable under a change of the input: a small template edit moved a re-extraction by more reactions than a change of seed (156 against 110 at baseline, 59 against 41 with `resolve_ties`, 16 against 1 with `prove_abs_gap`) and by a similar number of essential genes (17 against 17, 1 against 1, 1 against 0). raven-toolbox has no parameter that anchors a re-extraction to a prior build. +Two properties are measured. Determinism is that the same input, run again or with another seed, machine or solver build, gives the same model. Stability is that a similar input (a slightly edited template, or a comparable sample) gives a similar model. Both parameters are opt-in (`resolve_ties=False` and `prove_abs_gap=None` by default) and target determinism. In the measurements, the seed-to-seed difference fell from 110 reactions and 17 essential genes (default) to 41 reactions and 1 gene with `resolve_ties`, and to 1 reaction and 0 genes with `prove_abs_gap=1.0` added, at about 3.6 and 26 times the default build time. Neither parameter made the model stable under a change of the input: a small template edit moved a re-extraction by more reactions than a change of seed (156 against 110 at baseline, 59 against 41 with `resolve_ties`, 16 against 1 with `prove_abs_gap`) and by a similar number of essential genes (17 against 17, 1 against 1, 1 against 0). raven-toolbox has no parameter that anchors a re-extraction to a prior build. ## Main results -Each build in the next two tables is the full pipeline, including the task gap-fill, on Human-GEM and the DLD1 cell line (set-up in the next section). The seed-to-seed difference compares the build at seed 1234 with the build at seed 7. The repeat compares two builds at seed 1234 with nothing changed between them; it was run for two settings. For the baseline, the first build of the repeat is the seed 1234 build of the table (237 s) and the second took 231 s. For `resolve_ties`, the repeat consists of two further builds (865 s and 861 s), separate from the seed 1234 build of the table (864 s). +Each build in the next two tables is the full pipeline, including the task gap-fill, on Human-GEM and the DLD1 cell line (set-up in the next section). The seed-to-seed difference compares the build at seed 1234 with the build at seed 7. The repeat compares two builds at seed 1234 with nothing changed between them; it was run for two settings. For the baseline, the repeat (job A-base-rep) consists of two further builds (236.6 s and 230.9 s), separate from the seed 1234 build of the table (237.1 s). For `resolve_ties`, the repeat consists of two further builds (865 s and 861 s), separate from the seed 1234 build of the table (864 s). | Setting | Seed-to-seed difference, reactions | Seed-to-seed difference, essential genes | Essential genes (seed 1234 / seed 7) | Repeat difference at seed 1234, reactions / genes | Build time (seed 1234 / seed 7) | |---|---:|---:|---:|---:|---:| @@ -32,14 +32,14 @@ The unedited builds hardly used the removed reactions. If the re-extraction chan ## Set-up * Model and data: Human-GEM `v2.0.0-38`, Hart2015 DLD1 cell line, 57 essential tasks, series `1+1`. The `model/Human-GEM.yml` of that commit has 12877 reactions and 2848 genes. The preparation is the one of `estimateEssentialGenes.py` at that commit (removal of drug, amino-acid-triplet and duplicate complex I reactions), followed by raven-toolbox's own `prep_init_model`, which gives a reference model of 10141 reactions and a merged model of 6861 reactions (chains of linked reactions combined into one), with 206 task-essential reactions. This preparation differs from RAVEN's exported `raven_refModel.xml`, so absolute counts are not comparable to the [Human-GEM cell-type model validation against RAVEN](humangem-validation.md) run, which uses that export. -* Steps: series `1+1` runs two extraction steps. Step 1 decides the reactions with a gene association; step 2, the last step, adds the reactions without one and holds the reactions of step 1 on. +* Steps: series `1+1` runs two extraction steps, counted from 1 here; the debug log counts from 0. Step 1 decides the reactions with a gene association; step 2, the last step, adds the reactions without one and holds the reactions of step 1 on. * Settings: `big_m=100` (the default is 1000), `mip_gap_abs=10` and `time_limit=1800` s, the values of `estimateEssentialGenes.py` at that commit. A named series such as `1+1` ignores `mip_gap` and `mip_gap_abs`; they apply to a step without a run schedule. -* Escalation: each solve stops at a mixed-integer programming (MIP) gap, the distance between the objective of the best solution found and the solver's bound on the optimum, either relative (divided by the objective) or absolute. Without `prove_abs_gap`, each step follows RAVEN's run schedule, here called the **escalation**. The first run uses a relative gap of 0.0004; each later run uses the larger of its relative gap and its absolute gap (10 or 20) divided by the objective, and the step stops as soon as the gap already reached meets the next target. `time_limit` caps the limit of each run: step 1 has runs of 120 s and 1800 s, the final step runs of 5 s, 120 s and 1800 s. +* Escalation: each solve stops at a mixed-integer programming (MIP) gap, the distance between the objective of the best solution found and the solver's bound on the optimum, either relative (divided by the objective) or absolute. Without `prove_abs_gap`, each step follows RAVEN's run schedule, here called the escalation. The first run uses a relative gap of 0.0004; each later run uses the larger of its relative gap and its absolute gap (10 or 20) divided by the objective, and the step stops as soon as the gap already reached meets the next target. `time_limit` caps the limit of each run: step 1 has runs of 120 s and 1800 s, the final step runs of 5 s, 120 s and 1800 s. * With `prove_abs_gap`, each step is one solve proven to that absolute gap, with the 1800 s limit for the primary solve and for the two flux phases of `resolve_ties` (phases 2c and 2d), and half of it (`_TIE_BREAK_TIME_FRACTION = 0.5`) for its two on/off phases (phases 2a and 2b); the phases are defined in the section on the parameters. The task gap-fill uses a fixed 300 s per task in every setting. -* Solver: Gurobi 13.0.1, `Threads=1` unless stated, raven-toolbox at commit `94f39407`. The builds ran as Slurm jobs on AMD EPYC 9354 nodes shared with other jobs, so wall-clock figures are approximate. Most jobs had 2 CPUs and the `Threads=4` jobs of the first-step probe had 4. Several builds ran in sequence in one job (the seeds of a setting, or the three builds of the stability test). +* Solver: Gurobi through gurobipy, `Threads=1` unless stated, raven-toolbox at commit `94f39407`. Gurobi library 13.0.3 from the gurobipy wheel (most likely); the logs print no version. The builds ran as Slurm jobs on AMD EPYC 9354 nodes shared with other jobs, so wall-clock figures are approximate. Most jobs had 2 CPUs and the `Threads=4` jobs of the first-step probe had 4. Several builds ran in sequence in one job (the seeds of a setting, or the three builds of the stability test). * Determinism probe: the extraction `seed` was varied (1234, 7, and further seeds where stated) with inputs, code, solver build and machine held fixed. A difference between builds then has two sources: the solver's choice among equal optima, measured where two builds reached the same objective value, and different objective values reached within the gap and time limits (see the location of the degeneracy and the sweep of `prove_abs_gap`). Differences between Gurobi versions were not measured. * Metrics: the reaction difference is the symmetric difference of the kept-reaction sets (reactions kept by one build and not the other). The essential-gene difference is the symmetric difference of the sets of genes whose single deletion is predicted essential by `cobra.flux_analysis.find_essential_genes` at its default threshold. The gene difference is the result a user sees; the reaction difference is what the parameters act on. -* Debug log: all builds ran with `FTINIT_DEBUG=1`, which logs the status of each solve and of each tie-break phase. An **incumbent** is the best solution the solver holds when it stops; at a time limit it is not proven optimal. +* Debug log: all builds ran with `FTINIT_DEBUG=1`, which logs the status of each solve and of each tie-break phase. An incumbent is the best solution the solver holds when it stops; at a time limit it is not proven optimal. ## Degeneracy of the extraction MILP @@ -68,7 +68,7 @@ A phase that does not prove its optimum has these outcomes: * When the primary solve of a step stops at `time_limit`, all tie-break phases of that solve are skipped and the primary solution is kept. * The escalation raises no warning either. In the four sweep runs with the escalation and `resolve_ties=True`, the step 1 objective ended 4.0 to 12.0 below the proven optimum; at least one of the 12 first-step runs without `resolve_ties` reached it (the location of the degeneracy). -For comparison, RAVEN's `ftINIT` with `resolveTies` skips the tie-break after a primary solve that stopped at the time limit, gives the count and id-rank phases half of `params.TimeLimit` each, skips the minimal-flux and direction phases in the last step, and names every unproven phase (`phase2a-parsimony`, `phase2b-idrank`, `phase2c-minflux`, `phase2d-direction`) in its "tie resolution did not converge" warning (`ftINITInternalAlg.m`). RAVEN's `ftINIT` was not run in this study. +For comparison, RAVEN's `ftINIT` with `resolveTies` skips the tie-break after a primary solve that stopped at the time limit, gives the count and id-rank phases half of `params.TimeLimit` each, skips the minimal-flux and direction phases in the last step, and names every unproven phase (`phase2a-parsimony`, `phase2b-idrank`, `phase2c-minflux`, `phase2d-direction`) in its "tie resolution did not converge" warning. In RAVEN's `src/INIT/ftINITInternalAlg.m`, the skip after a primary solve at the time limit is at line 483, the half limit at line 518 and the warning at line 662; `src/INIT/ftINIT.m` line 227 sets `pinFluxes` to false in the last step. RAVEN's `ftINIT` was not run in this study. ## Location of the degeneracy @@ -129,19 +129,19 @@ Step 2 is the last step, so it skips the last two phases. The log of seed 1234: [ftinit] tie-break phase2b-idrank: status=optimal obj=346311.0 ``` -The on/off tie-break of step 1 was proven for all four seeds, so the step 1 reaction set is fixed. `phase2c-minflux` reached the 1800 s limit for 3 of the 4 seeds and `phase2d-direction` for all 4, and each was adopted as an unproven incumbent. Two flux phases at the 1800 s limit take 3600 s, which accounts for most of the step 1 time of about 3650 s and for a build time of about 100 minutes. The flux that step 2 reads is therefore the incumbent of a time-limited solve. The step 2 objective differed by seed (−611.872 for seed 7, −613.872 for the other three) and the final sets by one reaction. The fluxes themselves were not compared between seeds; a seed-dependent flux is inferred from the step 2 objectives, not measured. +The on/off tie-break of step 1 was proven for all four seeds, so the step 1 reaction set is fixed. `phase2c-minflux` reached the 1800 s limit for 3 of the 4 seeds and `phase2d-direction` for all 4, and each was adopted as an unproven incumbent. In the three seeds in which `phase2c-minflux` stopped at the limit, the two flux phases at the 1800 s limit take 3600 s, which accounts for most of the step 1 time of about 3650 s and for a build time of about 100 minutes. The flux that step 2 reads is therefore the incumbent of a time-limited solve. The step 2 objective differed by seed (−611.872 for seed 7, −613.872 for the other three) and the final sets by one reaction. The fluxes themselves were not compared between seeds; a seed-dependent flux is inferred from the step 2 objectives, not measured. The escalation builds with `resolve_ties=True` also adopted phases at a time limit, here the limit of the escalation run (120 s in step 1, so 60 s for phases 2a and 2b). The full-pipeline build of seed 1234 adopted 3 phases (`phase2b-idrank`, `phase2c-minflux`, `phase2d-direction` of step 1) and the build of seed 7 adopted 2 (`phase2c-minflux`, `phase2d-direction` of step 1). In step 2 of both builds, `phase2a-parsimony` ended at the time limit with an incumbent and `phase2b-idrank` ended at the time limit with no incumbent (`A-rt-1234.log`, `A-rt-7.log`). `phase2b-idrank` then returns no solution, the tie-break of step 2 returns `False`, the primary solution of step 2 is kept, and no warning is raised. The reaction sets of these two builds rest on a resolved step 1 and an unresolved step 2. -Counts of full builds (gap-fill included) with at least one unproven tie-break phase or a time-limited primary solve. The full builds of the main results are, per setting: baseline 4 (seeds 1234 and 7, the second build of the repeat, the edited template), `resolve_ties=True` 5 (seeds 1234 and 7, the two builds of the repeat, the edited template), `prove_abs_gap` 5 (seeds 1234 and 7 at each of 1.0 and 0.05, the edited template at 1.0). The Builds column gives how many of them were counted. +Counts of full builds (gap-fill included) with at least one unproven tie-break phase or a time-limited primary solve. The full builds of the main results are, per setting: baseline 5 (seeds 1234 and 7, the two builds of the repeat, the edited template), `resolve_ties=True` 5 (seeds 1234 and 7, the two builds of the repeat, the edited template), `prove_abs_gap` 5 (seeds 1234 and 7 at each of 1.0 and 0.05, the edited template at 1.0). The Builds column gives how many of them were counted. | Setting | Builds | With an adopted phase at `time_limit` | With a primary solve at `time_limit` | With a tie-break dropped without a warning | |---|---:|---:|---:|---:| -| baseline | 4 | 0 (no tie-break) | 4 (one run of the final step) | not applicable | -| `resolve_ties=True` | 4 | 4 | 4 (tie-break of that solve skipped) | 4 (step 2) | -| `resolve_ties=True, prove_abs_gap=1.0` or `0.05` | 4 | 4 | 0 | 0 | +| baseline | 5 | 0 (no tie-break) | 5 (one run of the final step) | not applicable | +| `resolve_ties=True` | 5 | 5 | 5 (tie-break of that solve skipped) | 5 (step 2) | +| `resolve_ties=True, prove_abs_gap=1.0` or `0.05` | 5 | 5 | 0 | 0 | -The baseline row counts all four baseline builds. The `prove_abs_gap` row counts the four unedited builds and leaves out the edited build at 1.0. The `resolve_ties=True` row counts four of the five builds; which build it leaves out is not recorded. The step 2 logs cited are those of seeds 1234 and 7. +Each row counts all builds of its setting, including the edited build at 1.0 for `prove_abs_gap`. The step 2 logs cited are those of seeds 1234 and 7. The warnings cover the adopted phases and the flux phases without a solution. A build without a warning can hold an unresolved tie-break (a dropped `phase2a-parsimony` or `phase2b-idrank`) or an unproven escalation, and the debug log is the record of what each solve reached. In the sweep, all 24 builds with `prove_abs_gap` raised at least one warning. A larger `time_limit` is the setting that can let the phases prove; its effect was not measured. @@ -157,21 +157,32 @@ For curation of a template (fix one or a few reactions, re-extract, compare), th ## Choice of settings -* `resolve_ties=True`: the repeat pair at seed 1234 had identical reaction sets, and the seed-to-seed difference fell from 17 genes to 1 on this data, at about 3.6 times the build time (4 to 14 minutes). Its warnings name the adopted phases that stopped at the time limit. A tie-break that finds no incumbent in phase 2a or 2b is dropped without a warning; the builds at seeds 1234 and 7 dropped the one of step 2, and the table of build counts gives 4 builds with a dropped step 2 tie-break. With the escalation, the tie-break runs again in each run of the schedule (`_solve_step` in `ftinit.py` and the run schedule in `steps.py`): the flux phases of step 1 get 120 s when step 1 stops after its first run and 1800 s when a second run follows (its 5000 s capped by `time_limit`). -* `prove_abs_gap=1.0` together with `resolve_ties=True`: the seed-to-seed difference fell to 1 reaction and 0 genes, and the objectives of both steps are proven to 1.0 (in the four sweep runs with the escalation and `resolve_ties=True`, step 1 ended 4.0 to 12.0 below the optimum). The build takes about 100 minutes at `time_limit=1800`, 26 times the baseline. In the four `prove_abs_gap=1.0` sweep builds (step 1 and step 2, no gap-fill; 4987 to 6183 s wall clock), step 1 took 3305 to 3650 s, mostly in the two flux phases, which is between 53 and 73 % of a build (bounds from the extreme step 1 and build times; the per-build shares were not recorded), and step 2 another 1650 to 2500 s. This setting fits when the difference left by `resolve_ties` alone (41 reactions and 1 gene here) is too large and the build time is acceptable. A value from 2.0 to 0.05 gave the same models; 5.0 stopped 2.0 below the optimum for one seed. No build with `prove_abs_gap` and without `resolve_ties` was run. -* Holding the raven-toolbox commit and the `gurobipy` version fixed is expected to remove the differences between solver builds, at no cost in run time. This was not measured: Gurobi versions were not compared, and two baseline builds with everything held fixed differed by 2 reactions. `resolve_ties` and `prove_abs_gap` are independent of that choice. +* `resolve_ties=True`: the repeat pair at seed 1234 had identical reaction sets, and the seed-to-seed difference fell from 17 genes to 1 on this data, at about 3.6 times the build time (4 to 14 minutes). Its warnings name the adopted phases that stopped at the time limit. A tie-break that finds no incumbent in phase 2a or 2b is dropped without a warning; the builds at seeds 1234 and 7 dropped the one of step 2, and the table of build counts gives 5 of 5 builds with a dropped step 2 tie-break. With the escalation, the tie-break runs again in each run of the schedule (`_solve_step` in `ftinit.py` and the run schedule in `steps.py`): the flux phases of step 1 get 120 s when step 1 stops after its first run and 1800 s when a second run follows (its 5000 s capped by `time_limit`). +* `prove_abs_gap=1.0` together with `resolve_ties=True` reduced the seed-to-seed difference from 41 to 1 reaction (1 to 0 genes) at 26 times the baseline time, about 100 minutes at `time_limit=1800`. The objectives of both steps are proven to 1.0 (in the four sweep runs with the escalation and `resolve_ties=True`, step 1 ended 4.0 to 12.0 below the optimum). In the four `prove_abs_gap=1.0` sweep builds (step 1 and step 2, no gap-fill; 4987 to 6183 s wall clock), step 1 took 3305 to 3650 s, 59.0 to 66.3 % of each build, mostly in the two flux phases, and step 2 another 1650 to 2500 s. A value from 2.0 to 0.05 gave the same models; 5.0 stopped 2.0 below the optimum for one seed. No build with `prove_abs_gap` and without `resolve_ties` was run. +* Gurobi versions were not compared; two baseline builds with everything held fixed differed by 2 reactions. ## Limits The full-pipeline determinism table rests on one seed pair (seeds 1234 and 7), the first-step probe on six seeds and the sweep on four. The study used one cell line, one machine type, one Gurobi build and one `time_limit`, and one edit (30 unused reactions removed) for the stability measurement. Each figure comes from one build per seed and setting, except the repeat pairs at seed 1234, so the data give direction and rough magnitude and no error bars. -Differences between Gurobi versions, the failure that `resolve_ties` and `prove_abs_gap` target, were not measured; the seed probe stands in for them, and how well it represents them is not known. The stability edit touched almost nothing the reference builds used, which is a smaller change than a typical curation. Two comparable samples (for example, two expression profiles of one tissue) were not compared. +Differences between Gurobi versions, the failure that `resolve_ties` and `prove_abs_gap` target, were not measured; the seed probe stands in for them, and how well it represents them is not known. The stability edit removed 30 of 10141 reference reactions, of which the reference builds kept at most 1. Two comparable samples (for example, two expression profiles of one tissue) were not compared. Wall-clock figures come from jobs that shared nodes with other jobs and are approximate; the difference and objective columns are set comparisons and do not depend on load. The `prove_abs_gap` build times depend on the `time_limit` of 1800 s, because the flux phases of step 1 run to the limit; no other limit was measured. RAVEN's `ftINIT` was not run. ## Reproduction -Requires Gurobi and a Human-GEM checkout. The preparation of the Human-GEM template (`prep`) and the scores follow `code/test/estimateEssentialGenes.py` of Human-GEM. The example passes `mip_gap_abs=10.0`, which the series `1+1` ignores. +Requires Gurobi and a Human-GEM checkout. The numbers come from `det/prep.py` (preparation of the template with the essential tasks, about 2000 s), `det/build.py` (one build or repeats of `ftinit` on DLD1 with `big_m=100`, `mip_gap_abs=10` and `time_limit=1800`), `det/select_edit.py` (the 30 removed reactions of the template edit, seed 0), `det/ties.py` (tied-score blocks) and the summaries `det/analyse.py` and `det/det_extra.py`. The scripts are in raven-toolbox under `scripts/parameter_tuning/init/`. The campaign takes about 100 core-hours. The commands run through the Slurm wrappers as in the README of that folder, with `WORK`, `TOOLBOX_ROOT`, `HUMANGEM_ROOT`, `PY`, `DET_SCRIPTS` and `PREP_DIR` set: + +```bash +jid=$(sbatch --parsable -J det-prep -o $WORK/logs/prep.log $DET_SCRIPTS/prep.sbatch) +bash $DET_SCRIPTS/submit_all.sh $jid +# after A-rtp1-1234 has finished: +sbatch -J det-edit -o $WORK/logs/edit.log $DET_SCRIPTS/edit.sbatch +python analyse.py $WORK/out +python det_extra.py $WORK/out $WORK/logs +``` + +The tables are computed from the outputs and logs by `analyse.py` and `det_extra.py`; the degeneracy counts are read from `ties.log`. `ties.py` and `select_edit.py` need the base preparation `prep-base.pkl`, which is not archived and takes about 2000 s to build. `det/build.py` produced the build numbers; series `1+1` ignores `mip_gap_abs`, which the call below passes as `mip_gap_abs=10.0`. ```python from raven_toolbox.init import ftinit diff --git a/docs/parameter-tuning/studies/homology-cutoff-calibration.md b/docs/parameter-tuning/studies/homology-cutoff-calibration.md index 51f4031..81dfdac 100644 --- a/docs/parameter-tuning/studies/homology-cutoff-calibration.md +++ b/docs/parameter-tuning/studies/homology-cutoff-calibration.md @@ -44,9 +44,9 @@ For each setting, **precision** is the number of correct matches divided by all | 45 | 0.907 | 0.813 | 0.735 | 0.520 | | 50 | 0.882 | 0.736 | 0.640 | 0.323 | -Against the KEGG reference the highest score was at 40 % for `kla` and `eco` and at 35 % for `yli` and `ani`. Against the OMA reference it was at 45 % for the three fungi and at 35 % for *E. coli*; the OMA scores are not tabulated. The optima of both references lie between 35 and 45 %, and in neither reference is an optimum below 35 %. +Against the KEGG reference the highest score was at 40 % for `kla` and `eco` and at 35 % for `yli` and `ani`. Against the OMA reference with BLAST it was at 45 % for the three fungi and at 35 % for *E. coli*; with DIAMOND it was at 45 % for the fungi and at 30 % for *E. coli*. The OMA scores are not tabulated. With BLAST the optima of both references lie between 35 and 45 %; the only optimum below 35 % is *E. coli* against OMA with DIAMOND. -Scored with F1, which weights precision and recall equally, the optimum moves to 25 % (values not tabulated). The best identity therefore depends on how the score weights the two kinds of error. +Scored with F1, which weights precision and recall equally, the optimum with BLAST moves to 30 % for `kla` and `eco` and 25 % for `yli` and `ani`; with DIAMOND it is 30 % for `kla`, `ani` and `eco` and 25 % for `yli` (values not tabulated). The best identity therefore depends on how the score weights the two kinds of error. ### Alignment length @@ -58,47 +58,50 @@ Scored with F1, which weights precision and recall equally, the optimum moves to | 200 (RAVEN 2.11.3) | 0.923 | 0.860 | 0.804 | 0.598 | | 300 | 0.885 | 0.816 | 0.764 | 0.571 | -Lengths 50 and 100 gave identical scores and 150 was at most 0.003 lower. The score fell between 150 and 200, by 0.010 to 0.020 from 100 to 200. Compared with 200, a length of 100 gave 3 to 4 more recall on every organism and lowered precision by at most 0.6 percentage points (precision and recall values not tabulated). The unit of the 3 to 4 is not recorded: percentage points of recall, or per cent more correct matches. No OMA scores were reported for alignment length. +Lengths 50 and 100 gave identical scores and 150 was at most 0.003 lower. The score fell between 150 and 200, by 0.010 to 0.020 from 100 to 200. Compared with 200, a length of 100 raised recall by 3.0 (`kla`), 3.8 (`yli`), 2.2 (`ani`) and 3.1 (`eco`) percentage points, which is 3.9, 6.5, 4.3 and 10.2 % more correct matches, and changed precision by at most 0.6 percentage points (`yli`, 0.967 to 0.961; precision and recall values not tabulated). No OMA scores were reported for alignment length. ### E-value -The E-value sweep was reported for `kla` and `ani` only. The hits were collected at an aligner E-value of `1e-4`, so `max_evalue=1e-4` keeps every hit. +The hits were collected at an aligner E-value of `1e-4`, so `max_evalue=1e-4` keeps every hit. -| `max_evalue` | `kla` | `ani` | -|---|---|---| -| 1e-4 … 1e-50 | 0.923 | 0.804 (identical throughout) | -| 1e-100 | 0.877 | 0.703 | +| `max_evalue` | `kla` | `yli` | `ani` | `eco` | +|---|---|---|---|---| +| 1e-4 | 0.923 | 0.860 | 0.804 | 0.598 | +| 1e-10 | 0.923 | 0.860 | 0.804 | 0.598 | +| 1e-30 | 0.923 | 0.860 | 0.804 | 0.598 | +| 1e-50 | 0.923 | 0.858 | 0.801 | 0.592 | +| 1e-100 | 0.903 | 0.809 | 0.743 | 0.535 | -The score was the same at 1e-4, 1e-10, 1e-30 and 1e-50, which spans 46 orders of magnitude, and fell at 1e-100. A likely reason (inference, not checked against the hit tables) is that every hit with at least 40 % identity over at least 200 residues already has an E-value below 1e-50. No OMA scores were reported for the E-value. +The score was the same at 1e-4, 1e-10 and 1e-30 for all four organisms. At 1e-50 it was unchanged for `kla` and lower by 0.002 (`yli`), 0.003 (`ani`) and 0.006 (`eco`), and at 1e-100 it was lower by 0.020 to 0.063. The F1 values (BLAST) are 0.855 for `kla` from 1e-4 to 1e-50 and 0.811 at 1e-100, and 0.663, 0.658 and 0.563 for `ani` at 1e-30, 1e-50 and 1e-100. A hit with at least 40 % identity over at least 200 residues can therefore have an E-value above 1e-50 in `yli`, `ani` and `eco` (inference from the score change; the hit tables were not inspected for this). No OMA scores were reported for the E-value. ### BLAST and DIAMOND | Organism | best with BLAST | best with DIAMOND | score at that point | |---|---|---|---| | `kla` | 40 | 40 | 0.923 vs 0.921 | -| `yli` | 35 | 35 | 0.868 vs 0.865 | +| `yli` | 35 | 35 | 0.868 vs 0.866 | | `ani` | 35 | 35 | 0.814 vs 0.811 | | `eco` | 40 | 40 | 0.598 vs 0.593 | -The best identity was the same with both aligners for every organism, and the scores at that identity differed by 0.002 to 0.005. DIAMOND returned 29,785 hits on `kla` against 54,478 for BLAST. After filtering, the two aligners agreed on 87 to 92 % of the surviving matches. That the hits missing from DIAMOND are weak matches the filters would discard is an inference from these two results. DIAMOND took 34 s against 358 s for BLAST on `kla` and 40 s against 788 s on `ani`, 10.5 and 19.7 times faster (thread count not recorded). +Against the KEGG reference with F0.5, the best identity was the same with both aligners for every organism, and the scores at that identity differed by 0.002 to 0.005. DIAMOND ran with `--more-sensitive`, and the speed ratios below apply to that mode. DIAMOND returned 32,192 hits on `kla` against 54,513 for BLAST. After filtering, the two aligners agreed on between 87.9 and 92.3 % of the surviving matches, measured as the Jaccard index of the (template gene, target gene) pairs that survive the filters at the base setting (identity 40, length 200, E-value 1e-30): 92.3 % (`kla`), 88.9 % (`yli`), 87.9 % (`ani`) and 91.8 % (`eco`). That the hits missing from DIAMOND are weak matches the filters would discard is an inference from these two results. BLAST took 140, 152, 200 and 44 s for `kla`, `yli`, `ani` and `eco`, and DIAMOND 12, 11, 14 and 7 s; DIAMOND was 11.7 times faster on `kla` and 14.3 times on `ani` (4 threads, AMD EPYC 9354, DIAMOND with `--more-sensitive`). ### Aligner settings Two more settings control the search that produces the hits. Neither was part of the sweeps. -`evalue` is the aligner's own cut-off: `1e-4` for `run_blast` and `1e-3` for `run_diamond`. Each matches the MATLAB side: `getBlast` passes `-evalue 10e-5`, which equals `1e-4`, and `getDiamond` passes none, so DIAMOND's own default applies. On *H. polymorpha* against *S. cerevisiae*, loosening the BLAST cut-off from `1e-5` to `1e-4` added 5,204 raw hits, 8.9 % of the table. Every added hit has an E-value above `1e-5` and so fails `max_evalue=1e-30`. As long as the aligner cut-off is looser than `max_evalue`, it changes the size of the hit table and not the model. +`evalue` is the aligner's own cut-off: `1e-4` for `run_blast` and `1e-3` for `run_diamond`. Each matches the MATLAB side: `getBlast` passes `-evalue 10e-5`, which equals `1e-4`, and `getDiamond` passes none, so DIAMOND's own default applies. On *H. polymorpha* against *S. cerevisiae*, loosening the BLAST cut-off from `1e-5` to `1e-4` added 5,200 raw hits (58,702 against 53,502), 8.9 % of the table. The tables differ row by row: 5,314 rows are only in the `1e-4` table and 114 only in the `1e-5` table. Of the 5,314 rows, 4,372 have an E-value above `1e-5` and so fail `max_evalue=1e-30`, and 942 have an E-value at or below `1e-5` (the smallest is 3.1e-60; 4 are below `1e-30`). In the five examples of the 942 rows that were inspected, the rows are other alignments (HSPs) of gene pairs that the cut-off change reports differently; the other rows were not inspected. Whether the 4 rows below `1e-30` and the 114 rows found only at `1e-5` also pass the identity and length filters, and so could change a model, was not checked. The remaining rows fail `max_evalue=1e-30`, so for them the aligner cut-off changes the size of the hit table and not the model. -`threads` defaults to every CPU the process may run on. A 500-query subset returned the same 2,469 hits at one thread and at four, in 45.2 s and 23.7 s. On a full proteome pair, single-threaded BLASTP took about 20 minutes per direction. +`threads` defaults to every CPU the process may run on. The first 500 sequences of the *H. polymorpha* proteome returned the same 2,746 hits at one thread and at four, with identical hit tables, in 20.4 s and 5.9 s (3.5 times faster). On a full proteome pair, single-threaded BLASTP took 205 s (*H. polymorpha* against *S. cerevisiae*) and 218 s (the reverse direction). The aligner cut-offs `1e-4` and `1e-5` took 111 s and 108.5 s. The measurements used the SGD translated ORFs (6,735 sequences) as template and Linux BLAST+ 2.17.0 on an AMD EPYC 9354 node. ## Chosen defaults | Setting | RAVEN 2.11.3 | Default (raven-toolbox and RAVEN 3) | Basis | |---|---|---|---| -| `min_identity` (`minIde`) | 40 | 40 | The setting with the largest effect; the optima of both references lie between 35 and 45 %. | +| `min_identity` (`minIde`) | 40 | 40 | Over the sweep from 25 to 50 % the score changed by 0.062 (`kla`) to 0.275 (`eco`), against 0.047 to 0.055 over alignment lengths 50 to 300 and 0.020 to 0.063 over E-values 1e-4 to 1e-100; with BLAST the optima of both references lie between 35 and 45 %. | | `min_align_len` (`minLen`) | 200 | 100 | 100 scored 0.010 to 0.020 higher than 200 against KEGG; 50 and 100 scored the same, and 100 is the stricter of the two. | -| `max_evalue` (`maxE`) | 1e-30 | 1e-30 | No change in score from 1e-4 to 1e-50; kept at the RAVEN value. | +| `max_evalue` (`maxE`) | 1e-30 | 1e-30 | No change in score from 1e-4 to 1e-30 in any organism; 1e-50 lowered it by up to 0.006; kept at the RAVEN value. | -The study supports one `min_identity` for BLAST and DIAMOND, since both reached the same best identity for every organism. No DIAMOND scores were reported for the length and E-value sweeps, whose base scores equal those of BLAST. +The study supports one `min_identity` for BLAST and DIAMOND against the KEGG reference with F0.5, since both reached the same best identity for every organism. Against OMA the optimum for *E. coli* is 35 % with BLAST and 30 % with DIAMOND, and with F1 the optimum for `ani` is 25 % with BLAST and 30 % with DIAMOND. DIAMOND scores for alignment length and E-value are not tabulated here. No DIAMOND scores are tabulated for the length and E-value sweeps, whose base scores equal those of BLAST. ## Limits @@ -106,23 +109,30 @@ The study supports one `min_identity` for BLAST and DIAMOND, since both reached - The two references are not comparable in absolute terms. The KEGG reference judges only genes with a reaction-linked KO, while the OMA reference judges the whole mapped proteome and counts every match that OMA does not list as an ortholog pair as wrong. Paralogs probably lower the OMA score for this reason (inference). Only the shape of each column is comparable between the references. - The OMA reference for *Y. lipolytica* is incomplete: 38 % of its OMA pairs could not be mapped to KEGG gene names. - One template: everything was measured outward from *S. cerevisiae*. -- One setting was varied at a time from length 200 and identity 40 %. The identity optimum and the absence of an E-value effect were measured at length 200, and were not measured at the default length of 100. +- One setting was varied at a time from length 200 and identity 40 %. The identity optimum and the absence of an E-value effect up to 1e-30 were measured at length 200, and were not measured at the default length of 100. - The scores count gene matches, not reactions in a finished model. A reaction transfers when any one of its genes matches, so the effect on a finished model is probably smaller (inference, not measured). -- Each score comes from one alignment per organism; scoring is deterministic, so there is no run-to-run spread. - -## Retired curated-GEM reference +- Each score comes from one alignment per organism. Scoring a saved hit table is deterministic. DIAMOND returned 32,192 hits for `kla`. The alignment was run once, so the run-to-run spread of the hit table was not measured. -A further part of the study scored candidate settings against two curated genome-scale metabolic models (GEMs) of non-model fungi, [hanpo-GEM](https://github.com/SysBioChalmers/hanpo-GEM) and [rhto-GEM](https://github.com/SysBioChalmers/rhto-GEM), which were built with RAVEN's homology-based reconstruction. This reference was dropped. +## Curated-GEM reference -The test loosened the settings one step at a time and recorded which fraction of the newly admitted reactions was already in the curated model. The fraction stayed between 0.62 and 0.85 up to hanpo-GEM's own build settings (`max_evalue=1e-30, min_align_len=150, min_identity=35`) and fell to 0.06 one step beyond them. The inference is that the curated model agrees with predictions made at its own build settings and with nothing looser, whether or not those settings are good. A score against such a model returns its build settings and measures self-consistency, not correctness. The same applies to any curated model drafted with RAVEN's homology method (inference). +Scoring against curated genome-scale metabolic models (GEMs) of non-model fungi such as [hanpo-GEM](https://github.com/SysBioChalmers/hanpo-GEM) and [rhto-GEM](https://github.com/SysBioChalmers/rhto-GEM) is not part of the 2026-11-01 measurement, and the study reports no numbers for it. Both models were built with RAVEN's homology-based reconstruction, so a score against them would measure agreement with their own build settings and not correctness (inference). ## Reproducing +The scripts are in [`scripts/parameter_tuning/homology/`](https://github.com/CompMetEng/raven-toolbox/tree/develop/scripts/parameter_tuning/homology) of raven-toolbox. The driver `homology_cutoff_kegg.py` takes `--out` before the subcommand: + ```bash -python scripts/homology_cutoff_kegg.py fetch --out work/ -python scripts/homology_cutoff_kegg.py align --out work/ -python scripts/homology_cutoff_kegg.py score --out work/ --beta 0.5 \ - --gene-ko kegg118_organism_gene_ko.tsv.gz +python homology_cutoff_kegg.py --out work fetch +python run_cached.py prefetch --out work +python fetch_references.py --out work/refs --kegg118-sha256 +python homology_cutoff_kegg.py --out work align +python homology_cutoff_kegg.py --out work score --beta 0.5 \ + --gene-ko work/refs/kegg118_organism_gene_ko_5org.tsv.gz +python agreement.py --work work --gene-ko work/refs/kegg118_organism_gene_ko_5org.tsv.gz --out agreement.tsv ``` -`scripts/homology_cutoff_kegg.py` is in raven-toolbox. `align --aligner diamond` and `score --aligner diamond` repeat the steps with DIAMOND, and `score --reference oma` scores against the OMA reference. The script also scores identities of 20 and 60 %, which are not reported. The alignment times recorded for `kla` and `ani` are in [BLAST and DIAMOND](#blast-and-diamond). The alignment output is saved, so re-scoring under a different weighting or against a different reference takes about two minutes. The *H. polymorpha* and thread measurements of the aligner settings are not produced by this script. +`fetch_references.py` downloads the release-118 gene→KO rows (from the `kegg118_core.tar.gz` release asset of raven-toolbox), the current KEGG REST tables and the SGD translated ORFs. `align --aligner diamond` and `score --aligner diamond` repeat the steps with DIAMOND, `score --reference oma` scores against the OMA reference and `score --beta 1` gives F1. The driver also scores identities of 20 and 60 %, which are not tabulated above. `agreement.py` computes the BLAST and DIAMOND agreement, `make_tables.py` collects the results, and `aligner_settings.py` runs the *H. polymorpha* measurements (thread test, aligner E-value `1e-4` against `1e-5`, single-thread full directions). `run_cached.py` wraps the driver, whose path is given in the environment variable `HOMOLOGY_SCRIPT`, and replaces its UniProt download with a disk cache, which allows scoring against OMA offline; it adds the `prefetch` subcommand and passes every other subcommand to the driver. The alignment output is saved, so re-scoring under a different weighting or against a different reference takes about two minutes. + +The UniProt downloads, the OMA pairs, the KEGG tables and the SGD ORFs are in the archived input bundle `homology-inputs.tar.zst` (18.9 MB), and the BLAST and DIAMOND hit tables in `homology-hits.tar.zst` (4.0 MB). + +The scores come from the run of 2026-11-01 with raven-toolbox `94f3940`, BLAST+ 2.17.0, DIAMOND 2.1.17, the KEGG release-118 table and UniProt release 2026_03, with BLAST and DIAMOND at 4 threads on an AMD EPYC 9354 node. The *H. polymorpha* template is the SGD ORF file, and the thread test uses the first 500 sequences of `hanpo.faa`. The curated-GEM reference is not part of this run. diff --git a/docs/parameter-tuning/studies/humangem-validation.md b/docs/parameter-tuning/studies/humangem-validation.md index f8f2799..7ae86a3 100644 --- a/docs/parameter-tuning/studies/humangem-validation.md +++ b/docs/parameter-tuning/studies/humangem-validation.md @@ -4,7 +4,7 @@ raven-toolbox's ftINIT was run on Human-GEM with the RNA sequencing (RNA-seq) da ## Set-up -Software: Human-GEM commit `38567ab8` (v2.1.1 plus 15 commits), RAVEN commit `2d0630e3` on MATLAB 2024b, raven-toolbox commit `94f3940`, and Gurobi 13.0.1 for both tools, with 4 cores per run. +Software: Human-GEM commit `38567ab8` (v2.1.1 plus 15 commits), RAVEN commit `2d0630e3` on MATLAB 2024b, raven-toolbox commit `94f3940`, and Gurobi for both tools, with 4 cores per run. The Python runs load the Gurobi library 13.0.3 from the gurobipy wheel (most likely; the logs print no version), and the MATLAB side links to the Gurobi module library 13.0.1 (from the linkage of the MATLAB interface; MATLAB prints no version). Template preparation is the step that runs once per template model, before any cell line. For RAVEN, the Human-GEM function `prepHumanModelForftINIT` removed the drug reactions, the amino-acid triplet reactions and the duplicate complex I reaction from Human-GEM, set the spontaneous and custom reaction lists, and prepared the model with the essential tasks of `metabolicTasks_Essential.txt`. RAVEN's prepared model, the model from which RAVEN extracts, has 9730 reactions. For raven-toolbox, `prep_init_model` ran on the same cleaned Human-GEM (11265 reactions). Every reaction in every raven-toolbox result is in RAVEN's prepared model, so the reaction identifiers of the two results are compared directly. @@ -102,11 +102,12 @@ raven-toolbox builds the extraction MILP with two numerical settings of RAVEN: - Per-reaction big-M. The on-off constraints of a scored reaction use `max(min(ub, big_m), min(-lb, big_m), force_on)` as big-M, an infinite bound counting as `big_m`. The default cap `big_m=1000` is RAVEN's `bigMCap` in `ftINITInternalAlg`. The cap limits the flux a reaction can carry while it counts as on. - Stoichiometric rescaling. `prep_init_model` applies RAVEN's `rescaleModelForINIT` to the merged model: each coefficient of a reaction is capped at 25 times the smallest coefficient of that reaction, and the reaction is then scaled to a mean absolute coefficient of 1. + Which combinations of `big_m` and rescaling are feasible for the staged pipeline was measured on HCT116 in [ftINIT parameter calibration and input robustness](init-param-calibration.md): with rescaling, `big_m=25` was infeasible and `big_m=100` feasible; without rescaling, `big_m=1000` was feasible (157 s against 83 s with rescaling, Jaccard 0.969 between the two results) and `big_m=100` infeasible. ## Constraint construction -Each steady-state balance of the extraction MILP is built from a flat list of terms and summed with `optlang.symbolics.add`; the same construction is used in the gap-fill. Python's `sum()` re-simplifies the growing expression at each term, and the balances of hub metabolites such as ATP, H⁺ and H₂O have 10³ terms or more. In one timing, summing 1500 terms took 59 s with `sum` and 0.01 s with `optlang.symbolics.add`; the set-up of that timing is not recorded. +Each steady-state balance of the extraction MILP is built from a flat list of terms and summed with `optlang.symbolics.add`; the same construction is used in the gap-fill. Python's `sum()` re-simplifies the growing expression at each term, and the balances of hub metabolites such as ATP, H⁺ and H₂O are long sums. ## Use of the result @@ -118,4 +119,17 @@ Each cell line, variant and `big_m` value ran once, without seed repeats, so the ## Reproduction -The script that ran this validation is not part of the pinned raven-toolbox. Its outputs, kept in the working directory that the analysis scripts take as `--work`, are reused by later studies: a model exported from the RAVEN run as `raven_refModel.xml`, and the prepared Human-GEM of raven-toolbox as `rg_prep.pkl` (no-task) and `rg_prep_tasks.pkl` (task-constrained). `scripts/analyze_init_solvers.py` reads `rg_prep.pkl` and `scripts/analyze_init_robustness.py` reads `rg_prep_tasks.pkl`; both score reactions against `raven_refModel.xml`. `scripts/analyze_init_params.py` reads `raven_refModel.xml` and prepares its own models from it. +The numbers come from `bigm/bm.py prep`, `preptask` and `val` (raven-toolbox side), `bigm/ml_prep.m` and `bigm/ml_cell.m` (RAVEN side) and `bigm/compare.py`, which prints every table of the study. The scripts are in raven-toolbox under `scripts/parameter_tuning/init/`. The commands run through the Slurm wrappers `job.sh` and `mljob.sh` as in the README of that folder, with `WORK`, `TOOLBOX_ROOT`, `RAVEN_ROOT`, `HUMANGEM_ROOT` and `PY` set: + +```bash +job.sh prep +job.sh preptask +mljob.sh ml_prep +for c in DLD1 GBM HCT116 HELA RPE1; do + job.sh val --cell $c + mljob.sh ml_cell $c +done +python compare.py --out $WORK/out --shared $WORK/shared --logs $WORK/logs +``` + +The preparations `prep`, `preptask` and `ml_prep` write the prepared models `prep_notask.pkl`, `prep_task.pkl` and `matlab_prepData.mat` to `shared/`. The tables are computed from the outputs by `compare.py`. diff --git a/docs/parameter-tuning/studies/init-param-calibration.md b/docs/parameter-tuning/studies/init-param-calibration.md index 2d4417b..14eb25d 100644 --- a/docs/parameter-tuning/studies/init-param-calibration.md +++ b/docs/parameter-tuning/studies/init-param-calibration.md @@ -2,13 +2,15 @@ This study runs raven-toolbox's ftINIT on Human-GEM with the HCT116 expression data of the Hart2015 dataset and varies one or two settings per run series; the scaling runs and the full-pipeline runs vary scaling and `big_m` together. The calibration part varies the mixed-integer linear program (MILP) settings `mip_gap`, `big_m` and `force_on` and the stoichiometric scaling of the prepared model, and records the run time, the MILP objective and which reactions are kept. The robustness part degrades the expression input by dropout, noise and downsampling, extracts a model with the metabolic-task layer on, and records the model size, the essential tasks the model passes and its overlap with the model from clean data. Overlap is measured as the Jaccard index of two kept-reaction sets: the reactions in both divided by the reactions in either. +Steps are counted from 1 on the pages of this study; the run logs count from 0, so the logged `step 1` is the second step. + ## Set-up ### Software and inputs -The calibration runs used Human-GEM commit `38567ab8` (v2.1.1 plus 15 commits), the HCT116 column of the Hart2015 RNA sequencing file, raven-toolbox commit `94f3940` and Gurobi 13.0.1 on 4 cores. The template is the reference model of the [Human-GEM cell-type model validation against RAVEN](humangem-validation.md): Human-GEM without the drug reactions, the amino-acid triplet reactions and the duplicate complex I reaction, 11265 reactions. `prep_init_model` prepared it without a task list. Its scaled merged model, the model that the MILP receives, has 6657 reactions; the preparation took 26 min and is excluded from every time in the tables. +The calibration runs used Human-GEM commit `38567ab8` (v2.1.1 plus 15 commits), the HCT116 column of the Hart2015 RNA sequencing file, raven-toolbox commit `94f3940` and Gurobi on 4 cores. Gurobi library 13.0.3 from the gurobipy wheel (most likely); the logs print no version. The template is the reference model of the [Human-GEM cell-type model validation against RAVEN](humangem-validation.md): Human-GEM without the drug reactions, the amino-acid triplet reactions and the duplicate complex I reaction, 11265 reactions. `prep_init_model` prepared it without a task list. Its scaled merged model, the model that the MILP receives, has 6657 reactions; the preparation took 26 min and is excluded from every time in the tables. -The robustness runs used an earlier Human-GEM state whose commit is not recorded, the same HCT116 data, a template prepared with the essential metabolic tasks, and `big_m=100`. The two parts therefore have no common baseline, and their reaction counts and Jaccard indices cannot be compared with each other. +The robustness runs used Human-GEM at a commit that is not recorded, the same HCT116 data, a template prepared with the essential metabolic tasks, and `big_m=100`. The two parts therefore have no common baseline, and their reaction counts and Jaccard indices cannot be compared with each other. Gene scores are `5·ln(level/1)` of each expression level, clipped to [−5, 10] (see [Gene-scoring constants](#gene-scoring-constants)), and reaction scores follow from the gene rules (`score_reactions_from_genes`, default `no_gene_score=-2`). @@ -16,12 +18,12 @@ Gene scores are `5·ln(level/1)` of each expression level, clipped to [−5, 10] | Run type | What runs | Script and sweep | |---|---|---| -| Step-0 solve | One `run_ftinit` call (one MILP, no run schedule) for the first step of series `1+1` on the scaled merged model, with a 900 s limit | `analyze_init_params.py`, sweep `ftinit_milp` | -| Scaling run | The same step-0 solve on a model prepared without scaling, which `rescale_for_init` then rescales with the given `max_stoich_diff` (or leaves unscaled), with a 600 s limit and `mip_gap=0.001`. | `analyze_init_params.py`, sweep `prep_scale` | -| Full pipeline | One call of `ftinit` with series `1+1` and `time_limit=600` on the prepared model without tasks: both MILP steps, each with its own schedule of runs. | `analyze_init_params.py`, sweep `ftinit_full` | -| Robustness run | One call of `ftinit` with series `1+1` on the template prepared with tasks, including the gap-fill that adds reactions until every task is feasible, followed by `check_tasks` on the extracted model. | `analyze_init_robustness.py`, phases `gradient` and `levers` | +| First-step solve | One `run_ftinit` call (one MILP, no run schedule) for the first step of series `1+1` on the scaled merged model, with a 900 s limit | `bigm/bm.py step0` | +| Scaling run | The same first-step solve on a model prepared without scaling, which `rescale_for_init` then rescales with the given `max_stoich_diff` (or leaves unscaled), with a 600 s limit and `mip_gap=0.001`. | `bigm/bm.py scale` | +| Full pipeline | One call of `ftinit` with series `1+1` and `time_limit=600` on the prepared model without tasks: both MILP steps, each with its own schedule of runs. | `bigm/bm.py full` | +| Robustness run | One call of `ftinit` with series `1+1` on the template prepared with tasks, including the gap-fill that adds reactions until every task is feasible, followed by `check_tasks` on the extracted model. | `scripts/analyze_init_robustness.py` (`MIP_GAP` 0.02, `TIME_LIMIT` 120), phases `gradient` and `levers` (not recoverable) | -A named series such as `1+1` ignores `mip_gap` and `mip_gap_abs`: every step runs its own schedule of MILP runs, with limits of 120 s, 5000 s and 5 s. `time_limit` lowers the limit of each scheduled run, so at `time_limit=600` it shortens the 5000 s runs only. `mip_gap` therefore acts on the step-0 solve and not on the full pipeline. +A named series such as `1+1` ignores `mip_gap` and `mip_gap_abs`: every step runs its own schedule of MILP runs, with limits of 120 s, 5000 s and 5 s. `time_limit` lowers the limit of each scheduled run, so at `time_limit=600` it shortens the 5000 s runs only. `mip_gap` therefore acts on the first-step solve and not on the full pipeline. `big_m` is the cap on the per-reaction big-M of the on and off constraints. Each scored reaction gets `max(min(ub, big_m), min(-lb, big_m), force_on)`, and an infinite bound counts as `big_m`, so `big_m` limits the flux that a reaction can carry while it counts as on. `force_on` is the smallest flux at which a reaction counts as on. `max_stoich_diff` is the largest ratio between the stoichiometric coefficients of one reaction after `rescale_for_init` (RAVEN's `rescaleModelForINIT`). @@ -37,25 +39,27 @@ The lever runs repeat the 70 % dropout input with `no_gene_score` or `force_on` | Column | Meaning | |---|---| -| time (s) | Wall-clock time of the `run_ftinit` call (step-0 and scaling runs) or the `ftinit` call (full pipeline), including the construction of the MILP and excluding the preparation | -| objective | MILP objective of the step-0 solve | +| time (s) | Wall-clock time of the `run_ftinit` call (first-step and scaling runs) or the `ftinit` call (full pipeline), including the construction of the MILP and excluding the preparation | +| objective | MILP objective of the first-step solve | | rel.obj.gap | (reference objective − objective) / reference objective; a positive value is a lower objective than the reference row | | `n_kept` | Reactions in the model that the full pipeline returns | | n_rxns | Reactions in the model that a robustness run returns, after the gap-fill | -| Jaccard | Jaccard index of two kept-reaction sets: the reactions in both divided by the reactions in either. Step-0 sets are merged-model reaction identifiers; full-pipeline and robustness sets are Human-GEM reaction identifiers | +| Jaccard | Jaccard index of two kept-reaction sets: the reactions in both divided by the reactions in either. First-step sets are merged-model reaction identifiers; full-pipeline and robustness sets are Human-GEM reaction identifiers | | tasks pass, frac | Essential tasks that `check_tasks` reports as passed on the extracted model, as a count and as a fraction | A MIP gap bounds the objective only. Two solutions within the gap can keep different reactions, so the reaction set is compared separately from the objective. -Each table row is one run unless the row states repeats; this set-up has no runs that differ only in the seed. The only available measure of the seed-to-seed spread comes from a different set-up, in [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](ftinit-determinism.md): two baseline builds (Human-GEM v2.0.0-38, DLD1, `big_m=100`, escalation schedule, with tasks and gap-fill) that differed only in the seed differed by 110 reactions (symmetric difference of the kept-reaction sets). That page does not give the sizes of these two builds; the 7754 reactions it reports belong to a build with `resolve_ties=True` and `prove_abs_gap=1.0`. Assuming builds of about 7300 to 7800 reactions, 110 reactions are about 1.5 % of the model and correspond to a Jaccard index of about 0.985. That pair compares two full pipelines with gap-fill; a step-0 set is the result of one MILP step on merged-model identifiers without tasks, a different kind of object. Between the settings of the calibration the Jaccard index ranged from 0.9976 to 0.9649, a difference of 0.2 to 3.5 % from identical sets. +Each table row is one run unless the row states repeats; this set-up has no runs that differ only in the seed. The only available measure of the seed-to-seed spread comes from a different set-up, in [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](ftinit-determinism.md): two baseline builds (Human-GEM v2.0.0-38, DLD1, `big_m=100`, escalation schedule, with tasks and gap-fill) that differed only in the seed differed by 110 reactions (symmetric difference of the kept-reaction sets). The two baseline builds kept 7779 and 7781 reactions, so the 110 reactions are 1.4 % of the model and correspond to a Jaccard index of 0.986. That pair compares two full pipelines with gap-fill; a first-step set is the result of one MILP step on merged-model identifiers without tasks, a different kind of object. Between the settings of the calibration the Jaccard index ranged from 0.9976 to 0.9649, a difference of 0.2 to 3.5 % from identical sets. ### Source of the numbers -The calibration tables come from `analyze_init_params.py` with the sweeps `ftinit_milp`, `prep_scale` and `ftinit_full`, and the robustness tables from `analyze_init_robustness.py` with the phases `gradient` and `levers`. The calibration runs differ from the sweep grids of the pinned script. The script runs the step-0 `mip_gap` and `force_on` sweeps and the scaling runs at `big_m=100.0`; the tables report `big_m=1000` for the `mip_gap` and `force_on` sweeps and both caps for the scaling runs. The script's step-0 `big_m` grid is 1000, 250, 100, 50 and 25, and its full-pipeline grid is the default `big_m` (1000) at `mip_gap` 0.001, 0.003 and 0.01 and `big_m` 50 and 250 at `mip_gap=0.003`, all on the scaled preparation. Outside these grids the calibration ran `big_m` 10 and 500 in the step-0 solve; `big_m` 500, 100 and 25 and the unscaled preparation in the full pipeline; `big_m=1000` in the scaling runs; and the repeats with `mip_gap_abs=None`. The result files of the runs are not in the repository. +The calibration tables come from `bigm/bm.py` with the subcommands `step0`, `scale` and `full`, summarised by `bigm/summ.py` and `bigm/compare.py` (prepared model `prep_notask.pkl`). The scripts are in raven-toolbox under `scripts/parameter_tuning/init/`. `step0` runs a grid of 16 solves: `mip_gap` 0.0002 to 0.1, `big_m` 10 to 1000 and `force_on` 0.02 to 0.5, at `big_m=1000` for the `mip_gap` and `force_on` sweeps. `scale` runs the unscaled preparation rescaled with `max_stoich_diff` 10, 25, 50 and 100 or left unscaled, at `big_m` 1000 and 100. `full` runs one `ftinit` call per setting, with `--gap`, `--big-m`, `--scale` and `--noabs` (`mip_gap_abs=None`) options. + +The first-step, scaling and full-pipeline tables are computed from the outputs by `compare.py`. The robustness tables (degraded expression input, lever runs and the 90 % dropout run) come from `scripts/analyze_init_robustness.py`. The script exists at the pinned commit; the state of the script and the inputs that produced the tables are not recoverable. ## Results -### Step-0 solve with different `mip_gap` values +### First-step solve with different `mip_gap` values `big_m=1000`, `force_on=0.1`. The reference is the tightest gap. @@ -68,7 +72,7 @@ The calibration tables come from `analyze_init_params.py` with the sweeps `ftini | 0.03 | 25 | 44705 | +0.0272 | 0.9649 | | 0.1 | 24 | 44647 | +0.0284 | 0.9663 | -### Step-0 solve with different `big_m` values +### First-step solve with different `big_m` values `mip_gap=0.001`, `force_on=0.1`. The last two columns compare each set with the run at `big_m=1000` and with the run at `big_m=100`. @@ -82,7 +86,7 @@ The calibration tables come from `analyze_init_params.py` with the sweeps `ftini | 25 | 25 | −0.0003 | 0.987 | 0.991 | | 10 | 23 | −0.0002 | 0.988 | 0.990 | -### Step-0 solve with different `force_on` values +### First-step solve with different `force_on` values `mip_gap=0.001`, `big_m=1000`. The reference is the default 0.1. @@ -94,7 +98,7 @@ The calibration tables come from `analyze_init_params.py` with the sweeps `ftini | 0.2 | 25 | −0.0002 | 0.992 | | 0.5 | 25 | −0.0000 | 0.987 | -### Step-0 solve with different stoichiometric scaling +### First-step solve with different stoichiometric scaling Scaling runs at `mip_gap=0.001` and two `big_m` values. Every rel.obj.gap and Jaccard index in the table, including the `big_m=100` columns, is against the first row at `big_m=1000` (scale on, `max_stoich_diff=25`, abbreviated `msd`). @@ -106,7 +110,7 @@ Scaling runs at `mip_gap=0.001` and two `big_m` values. Every rel.obj.gap and Ja | `msd=100` | 26 | +0.0048 | 0.980 | 25 | +0.0049 | 0.980 | | scale off | 30 | −0.0050 | 0.984 | 25 | −0.0044 | 0.983 | -The reference row of this table and the `mip_gap=0.001` row of the step-0 `mip_gap` table have the same settings and come from different runs: the scaling run rescales an unscaled preparation with `rescale_for_init` and has a 600 s limit, the step-0 run uses `prep(scale=True)` and a 900 s limit. Their objectives are 45539.6 and 45949.2, and their reaction sets share a Jaccard index of 0.983. +The reference row of this table and the `mip_gap=0.001` row of the first-step `mip_gap` table have the same settings and come from different runs: the scaling run rescales an unscaled preparation with `rescale_for_init` and has a 600 s limit, the first-step run uses `prep(scale=True)` and a 900 s limit. Their objectives are 45539.6 and 45949.2, and their reaction sets share a Jaccard index of 0.983. ### Full pipeline with and without scaling @@ -134,7 +138,7 @@ Scaled preparation without tasks, series `1+1`, `time_limit=600s`. | `big_m=50` | 82 | 7325 | 0.982 | | `big_m=25` | 30 | infeasible | none | -The runs were started with different `mip_gap` values, which the series ignores, so runs at one `big_m` are repeats of one configuration: 0.001, 0.003 and 0.01 at `big_m=1000`, 0.001 and 0.003 at `big_m=100`, and 0.003 at the other caps. Further repeats ran with `mip_gap_abs=None` in place of the default `mip_gap_abs=10`: at `big_m=1000` three runs of 85, 84 and 81 s, which kept the same 7312 reactions, and at `big_m=250` one run of 114 s. A third run at `big_m=100` took 119 s and a second run at `big_m=50` took 81 s; the record of the runs does not state their settings beyond the cap. In total `big_m=1000` has six runs (81 to 85 s), `big_m=250` two, `big_m=100` three (117, 118 and 119 s) and `big_m=50` two (82 and 81 s); `big_m=500` and `big_m=25` ran once. +The runs were started with different `mip_gap` values, which the series ignores, so runs at one `big_m` are repeats of one configuration: 0.001, 0.003 and 0.01 at `big_m=1000`, 0.001 and 0.003 at `big_m=100`, and 0.003 at the other caps. Further repeats ran with `mip_gap_abs=None` in place of the default `mip_gap_abs=10`: at `big_m=1000` three runs of 85, 84 and 81 s, which kept the same 7312 reactions, and at `big_m=250` one run of 114 s. A third run at `big_m=100` took 119 s (`mip_gap=0.001`, `mip_gap_abs=None`) and a second run at `big_m=50` took 81 s (`mip_gap=0.003`, `mip_gap_abs=None`). In total `big_m=1000` has six runs (81 to 85 s), `big_m=250` two, `big_m=100` three (117, 118 and 119 s) and `big_m=50` two (82 and 81 s); `big_m=500` and `big_m=25` ran once. ### Degraded expression input @@ -165,17 +169,17 @@ Lever runs on the 70 % dropout input. The default row is the dropout 70 % row of ## Effect of `mip_gap` on a single solve -In the step-0 solve the time was 24 to 25 s at every gap from 0.0002 to 0.1. At `mip_gap=0.001` the objective was within 0.0001 of the tightest gap and the Jaccard index was 0.9976. The Jaccard index stayed at 0.99 or above up to `mip_gap=0.01` and fell to 0.9649 and 0.9663 at 0.03 and 0.1. In the full pipeline the gap has no effect: the three `big_m=1000` runs at 0.001, 0.003 and 0.01 kept the same 7312 reactions. +In the first-step solve the time was 24 to 25 s at every gap from 0.0002 to 0.1. At `mip_gap=0.001` the objective was within 0.0001 of the tightest gap and the Jaccard index was 0.9976. The Jaccard index stayed at 0.99 or above up to `mip_gap=0.01` and fell to 0.9649 and 0.9663 at 0.03 and 0.1. In the full pipeline the gap has no effect: the three `big_m=1000` runs at 0.001, 0.003 and 0.01 kept the same 7312 reactions. ## Effect of `big_m` and scaling -In the step-0 solve every `big_m` from 10 to 1000 was feasible, took 23 to 26 s and reached the objective of `big_m=1000` within 0.0003. Against the run at `big_m=1000` and against the run at `big_m=100`, the reaction sets of the other caps had Jaccard indices of 0.983 to 0.991. The step-0 solve has no seed repeats, and the seed pair in [Set-up](#set-up) compares full pipelines with gap-fill, so it is no reference for these sets. In the step-0 scaling runs, every `max_stoich_diff` from 10 to 100 and the unscaled model were feasible at both caps, with Jaccard indices of 0.980 to 0.991 against the scaled reference. +In the first-step solve every `big_m` from 10 to 1000 was feasible, took 23 to 26 s and reached the objective of `big_m=1000` within 0.0003. Against the run at `big_m=1000` and against the run at `big_m=100`, the reaction sets of the other caps had Jaccard indices of 0.983 to 0.991. The first-step solve has no seed repeats, and the seed pair in [Set-up](#set-up) compares full pipelines with gap-fill, so it is no reference for these sets. In the first-step scaling runs, every `max_stoich_diff` from 10 to 100 and the unscaled model were feasible at both caps, with Jaccard indices of 0.980 to 0.991 against the scaled reference. -The full pipeline separates the settings that the step-0 solve does not. At `big_m=1000` it was feasible with and without scaling: 83 s with scaling, 157 s without, and the two models shared a Jaccard index of 0.969 (7312 and 7264 reactions). At `big_m=100` it was feasible with scaling (117 to 119 s) and infeasible without, and at `big_m=25` it was infeasible with scaling. With scaling, the caps from 50 to 1000 gave runs of 81 to 119 s with no trend in `big_m` and reaction sets that shared a Jaccard index of 0.978 to 0.982 with the default. No cap below 1000 ran faster than the 81 to 85 s of the `big_m=1000` runs. Derived from the sizes and Jaccard indices in the table (the sizes of two sets add up to the size of their union times 1 + J), the sets at caps 500, 250, 100 and 50 differed from the `big_m=1000` set by about 148, 163, 140 and 133 reactions. The one seed pair available, from the different set-up in [Set-up](#set-up), differed by 110 reactions, so every cap difference is about 20 to 50 % larger than that single value; the seed-to-seed spread of this set-up is unknown. The three `big_m=1000` runs kept identical sets (Jaccard 1.000) and only the cap changed between rows, so the differences follow the cap; whether they are systematic or a different choice among near-equal optima was not tested. A lower cap also reduces the flux that a kept reaction can carry. +The full pipeline separates the settings that the first-step solve does not. At `big_m=1000` it was feasible with and without scaling: 83 s with scaling, 157 s without, and the two models shared a Jaccard index of 0.969 (7312 and 7264 reactions). At `big_m=100` it was feasible with scaling (117 to 119 s) and infeasible without, and at `big_m=25` it was infeasible with scaling. With scaling, the caps from 50 to 1000 gave runs of 81 to 119 s with no trend in `big_m` and reaction sets that shared a Jaccard index of 0.978 to 0.982 with the default. The fastest run at a cap below 1000 (80.8 s at `big_m=50`, with `mip_gap_abs=None`) was as fast as the fastest `big_m=1000` run (81.1 s). Counted from the reaction sets (the rounded Jaccard values in the table give 148 and 163 for the first two), the sets at caps 500, 250, 100 and 50 differed from the `big_m=1000` set by 150, 165, 140 and 133 reactions. The one seed pair available, from the different set-up in [Set-up](#set-up), differed by 110 reactions, so every cap difference is 21 to 50 % larger than that single value; the seed-to-seed spread of this set-up is unknown. The three `big_m=1000` runs kept identical sets (Jaccard 1.000) and only the cap changed between rows, so the differences follow the cap; whether they are systematic or a different choice among near-equal optima was not tested. A lower cap also reduces the flux that a kept reaction can carry. ## Effect of `force_on` -`force_on` changes the MILP itself, through the on and off constraints. From 0.02 to 0.5 the step-0 solve took 24 to 25 s, the objective moved by at most 0.0002 and the reaction set shared a Jaccard index of 0.987 or more with the default 0.1. The full pipeline was not run with other `force_on` values. +`force_on` changes the MILP itself, through the on and off constraints. From 0.02 to 0.5 the first-step solve took 24 to 25 s, the objective moved by at most 0.0002 and the reaction set shared a Jaccard index of 0.987 or more with the default 0.1. The full pipeline was not run with other `force_on` values. ## Effect of degraded input @@ -187,7 +191,7 @@ In raven-toolbox commit `94f3940`, `ftinit` calls `fill_tasks` without a time li ## Choice of settings -The calibration supports the defaults `big_m=1000`, `force_on=0.1` and scaling on with `max_stoich_diff=25` (RAVEN's setting). `big_m=1000` is the cap that RAVEN's `ftINITInternalAlg` uses (`bigMCap`). Scaling is not needed for feasibility at `big_m=1000`; in these single runs the pipeline took 83 s with scaling and 157 s without. In the step-0 solve, `mip_gap=0.001` reached an objective within 0.0001 of the tightest gap; a named series ignores the setting. The full-pipeline runs used `time_limit=600` and the robustness script uses `time_limit=120`; no full-pipeline run took longer than 157 s in total. For noisy input the tested settings needed no change, and no tested setting compensated for dropout. +The defaults `big_m=1000`, `force_on=0.1` and scaling on with `max_stoich_diff=25` are RAVEN's values; every `big_m` from 50 to 1000 was feasible with scaling on, and `big_m=25` was infeasible. `big_m=1000` is the cap that RAVEN's `ftINITInternalAlg` uses (`bigMCap`). Scaling is not needed for feasibility at `big_m=1000`; in these single runs the pipeline took 83 s with scaling and 157 s without. In the first-step solve, `mip_gap=0.001` reached an objective within 0.0001 of the tightest gap; a named series ignores the setting. The full-pipeline runs used `time_limit=600` and the robustness script uses `time_limit=120`; no full-pipeline run took longer than 157 s in total. For noisy input the tested settings needed no change, and no tested setting compensated for dropout. ## Gene-scoring constants @@ -197,25 +201,33 @@ Expression values become gene scores through score = clip(factor × ln(level / reference), min_score, max_score) ``` -with `factor=5.0`, `max_score=10.0` and `min_score=-5.0`, the same in both implementations. The logarithm is natural (base e) in `init/score.py` and in RAVEN's `scoreModel`. The source of each implementation cites no paper for the three constants. The one candidate reference that has been suggested, a 2012 paper that describes mCADRE, uses categorical scoring, which differs from this continuous log-ratio, so it is not their origin. The constants are RAVEN's own values without a literature source. +with `factor=5.0`, `max_score=10.0` and `min_score=-5.0`, the same in both implementations. The logarithm is natural (base e) in `init/score.py` and in RAVEN's `scoreModel`. The source of each implementation cites no paper for the three constants. A 2012 paper (Wang et al. 2012, [mCADRE](https://doi.org/10.1186/1752-0509-6-153)) uses categorical scoring and is not the origin. The constants are RAVEN's own values without a literature source. ## Comparison with the solver benchmark -[ftINIT on different solvers](init-solver-benchmark.md) runs the same full pipeline with each solver interface. With Gurobi it took 92 s and kept 7312 reactions, against 83 s and 7312 reactions here. The benchmark time also includes loading the prepared model and scoring the reactions, and its limit is 900 s against 600 s here. A limit shortens only the 5000 s runs of the schedule, and no Gurobi run in either study reached its limit. Its Gurobi run at `big_m=100` kept 7324 reactions, the count of the two `big_m=100` runs here, whose reaction set shares a Jaccard index of 0.981 with the `big_m=1000` set. The solver comparison itself, with GLPK at 2182 s and a Jaccard index of 0.892 against Gurobi and the failure of HiGHS through the `scipy` interface, is on that page. +[ftINIT on different solvers](init-solver-benchmark.md) runs the same full pipeline with each solver interface. With Gurobi it took 92 s and kept 7312 reactions, against 83 s and 7312 reactions here. The benchmark time also includes loading the prepared model and scoring the reactions, and its limit is 900 s against 600 s here. A limit shortens only the 5000 s runs of the schedule, and no Gurobi run in either study ran for its overall limit. The warning `ftINIT step 1 hit the 600s time limit` appears 14 times in 13 full-pipeline and validation logs, and the Gurobi run of the solver benchmark has the same warning with 900 s. The logged `step 1` is the second step. The warning prints the `time_limit` argument, not the limit of the run that stopped; the stopped run is a shorter scheduled run, inferred from the total times of 83 s and 92 s, and the logs do not say which. One scheduled solve per run stopped at a limit. The Gurobi run of the solver benchmark at `big_m=100` kept 7324 reactions, the count of the two `big_m=100` runs here, whose reaction set shares a Jaccard index of 0.981 with the `big_m=1000` set. The solver comparison itself, with GLPK at 2182 s and a Jaccard index of 0.892 against Gurobi and the failure of HiGHS through the `scipy` interface, is on that page. ## Limits -- One model (Human-GEM), one cell line (HCT116) and one solver (Gurobi 13.0.1). +- One model (Human-GEM), one cell line (HCT116) and one solver (Gurobi library 13.0.3 from the gurobipy wheel, most likely; the logs print no version). - Most settings ran once; the repeats are listed with the full-pipeline table. This set-up has no seed repeats. The full-pipeline differences between caps are larger than the one seed pair available, which comes from a different set-up, so the seed-to-seed spread of this set-up is unknown. -- The robustness runs used a Human-GEM state whose commit is not recorded and `big_m=100`; the pinned script passes no `big_m`, so a rerun uses the default 1000. Each degradation is one random draw at one seed. -- The step-0 solve shows no difference between caps or scaling settings that the full pipeline shows, so step-0 results alone do not predict feasibility of the pipeline. +- The robustness runs used a Human-GEM commit that is not recorded and `big_m=100`; the script at the pinned commit passes no `big_m`, so a run with it uses the default 1000. Each degradation is one random draw at one seed. +- The first-step solve shows no difference between caps or scaling settings that the full pipeline shows, so first-step results alone do not predict feasibility of the pipeline. - The robustness tables have no run times, and the effect of `no_gene_score` on downsampled input was not measured. ## Reproduction +The commands run through the Slurm wrapper `job.sh` of `scripts/parameter_tuning/init/bigm/` as in the README of that folder, with `WORK`, `TOOLBOX_ROOT`, `HUMANGEM_ROOT` and `PY` set. The prepared models come first: + ```bash -python scripts/analyze_init_params.py --cell HCT116 --sweeps ftinit_milp,prep_scale,ftinit_full -python scripts/analyze_init_robustness.py --cell HCT116 --algo ftinit +job.sh prep +job.sh step0 --cell HCT116 +job.sh scale --cell HCT116 +job.sh full --gap 0.003 --big-m 1000 --tag g003-bm1000 +job.sh full --gap 0.003 --big-m 100 --scale off --tag g003-bm100-scaleoff +job.sh full --gap 0.001 --big-m 100 --noabs --tag noabs-g0.001-bm100 +python summ.py $WORK/out +python compare.py --out $WORK/out --shared $WORK/shared --logs $WORK/logs ``` -Both scripts read the reference model `raven_refModel.xml` from the working folder of [Human-GEM cell-type model validation against RAVEN](humangem-validation.md) (`--work`, default `~/hgem_compare`) and keep their results in a pickle file, so a rerun skips finished settings. `analyze_init_params.py` prepares the template itself; `analyze_init_robustness.py` loads the prepared template with tasks, `rg_prep_tasks.pkl`, from the same folder. The grids of the pinned `analyze_init_params.py` are smaller than the set of runs in the tables (see [Source of the numbers](#source-of-the-numbers)). +The tags of the other full-pipeline runs are listed in the README: `--big-m` 25, 50, 250 and 500 at `--gap 0.003`, the `mip_gap` runs 0.001, 0.003 and 0.01 at `big_m=1000`, and the `--noabs` repeats. The unscaled preparation of `scale` takes about 26 min. The robustness tables have no reproduction command: the state of `analyze_init_robustness.py` and the inputs that produced them are not recoverable. diff --git a/docs/parameter-tuning/studies/init-solver-benchmark.md b/docs/parameter-tuning/studies/init-solver-benchmark.md index 2af759f..49fcd1f 100644 --- a/docs/parameter-tuning/studies/init-solver-benchmark.md +++ b/docs/parameter-tuning/studies/init-solver-benchmark.md @@ -4,7 +4,7 @@ ftINIT builds mixed-integer linear programs (MILPs) and passes them to a solver ## Set-up -Every run calls `ftinit()` with the same input: Human-GEM prepared without a task list and with scaled stoichiometry, reaction scores from HCT116 expression, series `1+1` and `time_limit=900` s. The call also passes `mip_gap=0.001`, which a named series ignores. Software: Human-GEM commit `38567ab8`, raven-toolbox commit `94f3940`, Gurobi 13.0.1 and cobra 0.32.1, on 4 cores. Each setting ran once. +Every run calls `ftinit()` with the same input: Human-GEM prepared without a task list and with scaled stoichiometry, reaction scores from HCT116 expression, series `1+1` and `time_limit=900` s. The call also passes `mip_gap=0.001`, which a named series ignores. Software: Human-GEM commit `38567ab8`, raven-toolbox commit `94f3940`, gurobipy 13.0.3 and cobra 0.32.1, on 4 cores. Gurobi library 13.0.3 from the gurobipy wheel (most likely); the logs print no version. Each setting ran once. The time is the wall-clock time of the whole benchmark step: reading the reference and the prepared model, scoring the reactions and calling `ftinit()`. A failed run reports the time until the failure. @@ -22,7 +22,7 @@ The run log counts the steps from 0; the table counts them from 1. ## Gurobi -Gurobi completed in 92 s with the default big-M, which caps each reaction at 1000, and in 128 s with every cap at 100. No step reached the 900 s limit. The calibration study reports 83 s for the same call with a limit of 600 s; it times `ftinit()` alone, while the 92 s here also include reading and scoring. The two Gurobi models were not compared reaction by reaction. In the calibration study, with a time limit of 600 s, the reaction sets at `big_m` 100 and 1000 share a Jaccard index of 0.981 ([ftINIT parameter calibration and input robustness](init-param-calibration.md)). +Gurobi completed in 92 s with the default big-M, which caps each reaction at 1000, and in 128 s with every cap at 100. The log of the `big_m=1000` run carries the warning `ftINIT step 1 hit the 900s time limit with an unproven MIP gap (0.0030)`, although the run took 92 s in total, so one scheduled solve stopped at a limit. The logged `step 1` is the second step. The warning prints the `time_limit` argument, not the limit of the run that stopped. The stopped run is a shorter scheduled run, inferred from the total time; the log does not say which. No step ran for the full 900 s. The calibration study reports 83 s for the same call with a limit of 600 s; it times `ftinit()` alone, while the 92 s here also include reading and scoring. The reaction sets of the two Gurobi runs share a Jaccard index of 0.981, with 140 differing reactions. ## GLPK @@ -30,11 +30,11 @@ GLPK completed the pipeline in 2182 s. Its MILP solver honours `configuration.ti ## Interfaces that failed -HiGHS is reached through the `scipy` interface of optlang, which accepts continuous variables only and cannot hold the ftINIT MILP. The run failed with the error in the table after 1521 s. The script selected optlang's `hybrid_interface` module for HiGHS, but `hybrid` was not among the solver names offered in this environment: the error lists `glpk_exact`, `glpk`, `gurobi` and `scipy`. `glpk_exact` was offered and not run. The failure came after 10 s, the time spent reading the reference model before the solver name was set. +HiGHS is reached through the `scipy` interface of optlang, which accepts continuous variables only and cannot hold the ftINIT MILP. The run failed with the error in the table after 1521 s. `bm.py` sets the solver name `hybrid` for the `hybrid` row, and `hybrid` was not among the solver names offered in this environment: the error lists `glpk_exact`, `glpk`, `gurobi` and `scipy`. `glpk_exact` was offered and not run. The failure came after 10 s, the time spent loading the prepared-model pickle (reference model and prepared model) before the solver name was set. No HiGHS run exists in this benchmark. ## Choice of solver -Gurobi is the only solver that completed genome-scale ftINIT without reaching the time limit: 92 s against 2182 s for GLPK, which ended both main steps at the limit. HiGHS through `scipy` cannot be used. +With one run per solver, Gurobi completed in 92 s and 128 s; GLPK completed in 2182 s with both main steps stopped at the limit; scipy and hybrid failed. HiGHS through `scipy` cannot hold the ftINIT MILP. Small networks do not need Gurobi. `tests/test_init_solvers.py` runs ftINIT on toy networks with every installed MILP solver, so development and continuous integration need no Gurobi licence. @@ -42,15 +42,20 @@ Small networks do not need Gurobi. `tests/test_init_solvers.py` runs ftINIT on t Each setting ran once, on one model and one cell line. In a different set-up (the DLD1 cell line, `big_m=100`, with tasks and gap-filling), two Gurobi builds with different seeds differ by 110 reactions at baseline ([ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](ftinit-determinism.md)), so a difference of that size between solvers does not by itself indicate a solver effect. The 800 reactions that differ between GLPK and Gurobi (123 + 677) are larger. The `scipy` and `hybrid` rows record failures and give no solver timings. -## Reproduction +## Source of the numbers and reproduction + +The table comes from `bigm/bm.py solver` and the overlap figures from `bigm/compare.py`. The scripts are in raven-toolbox under `scripts/parameter_tuning/init/`. The CI test for toy networks is separate: ```bash # CI parameterised tests (seconds, runs always): python -m pytest tests/test_init_solvers.py -v -# Genome-scale benchmark (minutes-to-hours, manual): -python scripts/analyze_init_solvers.py --cell HCT116 \ - --doc docs/parameter-tuning/studies/init-solver-benchmark.md +# Genome-scale benchmark (needs the prepared model from job.sh prep): +job.sh prep +job.sh solver --solvers gurobi,hybrid +job.sh solver --solvers glpk +job.sh solver --solvers scipy +python compare.py --out $WORK/out --shared $WORK/shared --logs $WORK/logs ``` -The first command checks the toy networks and runs in seconds. The second runs the genome-scale benchmark, which takes minutes to hours. It resumes per solver and reuses the prepared Human-GEM of [Human-GEM cell-type model validation against RAVEN](humangem-validation.md). The pinned script has no `big_m` option and runs the gurobi, hybrid and glpk interfaces only (`_INTERFACES` in `analyze_init_solvers.py`), so the command does not produce the `big_m=100` and `scipy` rows. The script writes its report to the file given by `--doc`; give another path to keep the page. +The first command checks the toy networks and runs in seconds. The commands of the second block are the documented invocations; `job.sh` runs `bm.py` as a Slurm job, as in the README of the folder, with the environment variables `WORK`, `TOOLBOX_ROOT`, `HUMANGEM_ROOT` and `PY` set. The `big_m=100` Gurobi row is part of `bm.py solver`. The tables are computed from the outputs by `compare.py`. diff --git a/docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md b/docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md index 5d54a6e..9a0d865 100644 --- a/docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md +++ b/docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md @@ -14,15 +14,16 @@ KEGG-based reconstruction from sequence assigns each protein of a new organism t ### Inputs -- KEGG release 118. -- HMM libraries: `prokaryotes.hmm` (831 MB) and `eukaryotes.hmm` (692 MB), built by the maintainers with `build_ko_hmm`. The sequences of each KO were clustered with CD-HIT at 90 % identity and aligned with MAFFT FFT-NS-2, or with MAFFT PartTree for KOs whose alignment exceeds a memory budget. -- Queries: the full proteome of each organism, extracted from KEGG `genes.pep`, searched with `hmmscan`. `run_hmmsearch` in raven-toolbox fixes `hmmsearch -Z` to the number of profiles; with that setting its E-values match those of `hmmscan` against the same library. -- Each organism's hits were computed once, and every setting was applied to the same hit table. The differences between settings therefore contain no search-to-search variation, and the run-to-run spread of the search itself was not measured. -- Thread count and hardware of the sweep runs were not recorded. +The percentile, cut-off, score-ratio and defaults tables come from one run on 2026-11-01 (raven-toolbox commit `94f3940`, HMMER 3.4, 16 threads on an AMD EPYC 9354 node) of the sources below. The timing and the true-reaction counts come from separate jobs. + +- Tables: the kegg118 artefacts (`organism_gene_ko`, `ko_reaction` and the reference model) published as the `kegg118` release of toolbox-assets. +- HMM libraries: the published kegg118 libraries `kegg118_eukaryotes.hmm` (2698 profiles, 696,233,564 bytes) and `kegg118_prokaryotes.hmm` (4205 profiles, 838,505,885 bytes), built by the maintainers with `build_ko_hmm`. The sequences of each KO were clustered with CD-HIT at 90 % identity and aligned with MAFFT FFT-NS-2, or with MAFFT PartTree for KOs whose alignment exceeds a memory budget. +- Queries: the CDS genes of each organism from the KEGG REST lists, downloaded on 2026-11-01 (KEGG genes updated 2026-10-31, which is not KEGG release 118): 6021, 5010, 4288 and 476 sequences, of 6449, 5094, 4639 and 525 list entries. The sequences were not compared with those of KEGG release 118. The search is `hmmsearch`; `run_hmmsearch` in raven-toolbox fixes `-Z` to the number of profiles. Whether these E-values equal those of `hmmscan` against the same library was not checked. +- Each organism's hits were computed once, and every setting was applied to the same hit table. The differences between settings therefore contain no search-to-search variation. For `sce` the parsed hit tables were identical in four runs at 16, 12, 4 and 1 threads; the other organisms were searched once. ### Reference and scores -The reference is the **KEGG annotation**: the organism's gene→KO pairs in the `organism_gene_ko` table of the KEGG artefacts. The table lists only KOs linked to at least one reaction in `ko_reaction`, and the HMM libraries are built from the same table. The organism's **true reactions** are the reactions linked to its annotated KOs. +The reference is the **KEGG annotation**: the organism's gene→KO pairs in the `organism_gene_ko` table of the kegg118 artefacts. The table lists the KOs linked to at least one reaction in `ko_reaction`, and the HMM libraries are built from the same table. The organism's **true reactions** are the reactions linked to its annotated KOs, limited to reactions that are in the kegg118 reference model (`--restrict-to-reference`). The limit is applied because reaction recall is measured against reactions that the draft can contain. A predicted pair is the output of `assign_kos`, with the `organism:` prefix stripped from the gene identifier so that it matches the reference. Each predicted pair is either **matched** (present in the annotation) or **novel** (absent from it). @@ -40,33 +41,35 @@ A novel pair counts as an error, although some novel pairs may be correct assign | code | organism | library | proteome (seqs) | true gene→KO pairs | true reactions | |---|---|---|---|---|---| -| `sce` | *Saccharomyces cerevisiae* (budding yeast) | euk | 6021 | 841 | 1217 | -| `cme` | *Cyanidioschyzon merolae* (red alga) | euk | 5010 | 709 | 1157 | -| `eco` | *Escherichia coli* K-12 MG1655 | prok | 4288 | 1071 | 1548 | +| `sce` | *Saccharomyces cerevisiae* (budding yeast) | euk | 6021 | 844 | 1217 | +| `cme` | *Cyanidioschyzon merolae* (red alga) | euk | 5010 | 709 | 1156 | +| `eco` | *Escherichia coli* K-12 MG1655 | prok | 4288 | 1072 | 1548 | | `mge` | *Mycoplasmoides genitalium* G37 (minimal genome) | prok | 476 | 110 | 211 | +Without the limit to the reference model, the published `ko_reaction` table (7642 reactions, against 12,373 in the reference model) gives 1407, 1357, 1772 and 274 true reactions for the four organisms. The reaction columns of the tables below follow the limited definition. + `sce` and `eco` are model organisms; `cme` and `mge` are less studied. `mge` has the smallest proteome of the four and the weakest matched hits in the E-value table. ## Results ### E-value distributions -The table gives percentiles of `log10(E-value)` of the best hit per (gene, KO) pair, for all hits that `hmmscan` reported, split into matched and novel pairs. A more negative value is a stronger hit. The 99th percentile is the weakest 1 % of the pairs in a group. The table does not show the strongest novel pairs. +The table gives percentiles of `log10(E-value)` of the best hit per (gene, KO) pair, for all hits that `hmmsearch` reported, split into matched and novel pairs. A more negative value is a stronger hit. The 99th percentile is the weakest 1 % of the pairs in a group. The table does not show the strongest novel pairs. | organism | group | n | p50 | p90 | p95 | p99 | |---|---|---|---|---|---|---| -| `sce` | matched | 835 | −155 | −75 | −59 | −33 | -| `sce` | novel | 9467 | −8 | −2 | −0 | 1 | -| `cme` | matched | 704 | −133 | −63 | −47 | −25 | -| `cme` | novel | 10170 | −8 | −2 | −2 | 0 | -| `eco` | matched | 1070 | −142 | −69 | −57 | −36 | -| `eco` | novel | 27357 | −7 | −2 | −1 | −0 | -| `mge` | matched | 110 | −100 | −42 | −35 | −15 | -| `mge` | novel | 1904 | −4 | −2 | −1 | −0 | +| `sce` | matched | 842 | −153 | −73 | −57 | −25 | +| `sce` | novel | 9451 | −8 | −2 | 0 | 1 | +| `cme` | matched | 705 | −133 | −63 | −46 | −25 | +| `cme` | novel | 10189 | −8 | −2 | −2 | 0 | +| `eco` | matched | 1071 | −141 | −70 | −57 | −36 | +| `eco` | novel | 27352 | −7 | −2 | −1 | 0 | +| `mge` | matched | 110 | −102 | −43 | −35 | −15 | +| `mge` | novel | 1912 | −4 | −2 | −1 | 0 | -The median matched pair had an E-value between 1e-100 (`mge`) and 1e-155 (`sce`), and the weakest 1 % of matched pairs lay between 1e-15 and 1e-36. The median novel pair lay between 1e-4 and 1e-8. The weakest 1 % of matched pairs was 11 (`mge`) to 29 (`eco`) orders of magnitude stronger than the median novel pair. +The median matched pair had an E-value between 1e-102 (`mge`) and 1e-153 (`sce`), and the weakest 1 % of matched pairs lay between 1e-15 and 1e-36. The median novel pair lay between 1e-4 and 1e-8. The weakest 1 % of matched pairs was 11 (`mge`) to 29 (`eco`) orders of magnitude stronger than the median novel pair. -The RAVEN 2.11.3 cut-off of `1e-50` falls inside the matched distribution: between p95 and p99 for `sce` and `eco`, between p90 and p95 for `cme`, and between p50 and p90 for `mge`. It therefore removes 1 to 5 % of the matched pairs of `sce` and `eco`, 5 to 10 % of `cme` and 10 to 50 % of `mge`. The default `1e-30` lies beyond p99 for `sce` and `eco` and between p95 and p99 for `cme` and `mge`, and is 22 to 26 orders of magnitude stronger than the median novel pair. +The RAVEN 2.11.3 cut-off of `1e-50` falls inside the matched distribution: between p95 and p99 for `sce` and `eco`, between p90 and p95 for `cme`, and between p50 and p90 for `mge`. It therefore removes 1 to 5 % of the matched pairs of `sce` and `eco`, 5 to 10 % of `cme` and 10 to 50 % of `mge`. The default `1e-30` lies beyond p99 for `eco` (fewer than 1 % of the matched pairs removed) and between p95 and p99 for `sce`, `cme` and `mge`, and is 22 to 26 orders of magnitude stronger than the median novel pair. ### Cut-off sweep @@ -76,91 +79,91 @@ The RAVEN 2.11.3 cut-off of `1e-50` falls inside the matched distribution: betwe | cutoff | gKO prec | gKO rec | gKO F1 | rxn rec | rxn novel | |---|---|---|---|---|---| -| 1e-10 | 0.57 | 0.98 | 0.72 | 0.99 | 334 | -| 1e-20 | 0.65 | 0.98 | 0.78 | 0.97 | 283 | -| 1e-30 | 0.72 | 0.97 | 0.83 | 0.97 | 216 | -| 1e-50 | 0.78 | 0.95 | 0.86 | 0.96 | 157 | -| 1e-70 | 0.81 | 0.91 | 0.86 | 0.91 | 113 | -| 1e-100 | 0.84 | 0.76 | 0.80 | 0.79 | 68 | +| 1e-10 | 0.57 | 0.98 | 0.73 | 0.99 | 324 | +| 1e-20 | 0.66 | 0.98 | 0.79 | 0.99 | 274 | +| 1e-30 | 0.73 | 0.97 | 0.83 | 0.97 | 208 | +| 1e-50 | 0.78 | 0.95 | 0.86 | 0.96 | 151 | +| 1e-70 | 0.82 | 0.91 | 0.86 | 0.93 | 105 | +| 1e-100 | 0.85 | 0.76 | 0.80 | 0.79 | 61 | #### `cme` | cutoff | gKO prec | gKO rec | gKO F1 | rxn rec | rxn novel | |---|---|---|---|---|---| -| 1e-10 | 0.50 | 0.98 | 0.67 | 1.00 | 541 | -| 1e-20 | 0.57 | 0.98 | 0.72 | 1.00 | 421 | -| 1e-30 | 0.61 | 0.97 | 0.75 | 0.97 | 367 | -| 1e-50 | 0.70 | 0.93 | 0.80 | 0.94 | 307 | +| 1e-10 | 0.51 | 0.98 | 0.67 | 1.00 | 539 | +| 1e-20 | 0.56 | 0.98 | 0.71 | 1.00 | 426 | +| 1e-30 | 0.61 | 0.97 | 0.75 | 0.99 | 370 | +| 1e-50 | 0.69 | 0.93 | 0.79 | 0.93 | 310 | | 1e-70 | 0.75 | 0.85 | 0.80 | 0.87 | 223 | -| 1e-100 | 0.80 | 0.70 | 0.75 | 0.71 | 136 | +| 1e-100 | 0.80 | 0.70 | 0.75 | 0.71 | 137 | #### `eco` | cutoff | gKO prec | gKO rec | gKO F1 | rxn rec | rxn novel | |---|---|---|---|---|---| -| 1e-10 | 0.53 | 0.99 | 0.69 | 0.99 | 382 | -| 1e-20 | 0.57 | 0.99 | 0.73 | 0.99 | 300 | -| 1e-30 | 0.60 | 0.98 | 0.75 | 0.99 | 268 | -| 1e-50 | 0.67 | 0.95 | 0.78 | 0.98 | 198 | -| 1e-70 | 0.73 | 0.88 | 0.80 | 0.93 | 157 | -| 1e-100 | 0.82 | 0.74 | 0.77 | 0.80 | 96 | +| 1e-10 | 0.54 | 0.99 | 0.70 | 0.99 | 393 | +| 1e-20 | 0.58 | 0.99 | 0.73 | 0.99 | 308 | +| 1e-30 | 0.61 | 0.99 | 0.75 | 0.99 | 279 | +| 1e-50 | 0.68 | 0.96 | 0.79 | 0.98 | 208 | +| 1e-70 | 0.74 | 0.89 | 0.81 | 0.94 | 167 | +| 1e-100 | 0.82 | 0.75 | 0.78 | 0.81 | 111 | #### `mge` | cutoff | gKO prec | gKO rec | gKO F1 | rxn rec | rxn novel | |---|---|---|---|---|---| -| 1e-10 | 0.52 | 0.98 | 0.68 | 0.99 | 75 | -| 1e-20 | 0.62 | 0.96 | 0.75 | 0.98 | 51 | +| 1e-10 | 0.53 | 0.98 | 0.69 | 0.99 | 74 | +| 1e-20 | 0.62 | 0.96 | 0.76 | 0.98 | 51 | | 1e-30 | 0.65 | 0.95 | 0.77 | 0.98 | 39 | -| 1e-50 | 0.77 | 0.84 | 0.80 | 0.87 | 29 | -| 1e-70 | 0.85 | 0.73 | 0.78 | 0.73 | 27 | -| 1e-100 | 0.87 | 0.50 | 0.64 | 0.47 | 21 | +| 1e-50 | 0.77 | 0.85 | 0.81 | 0.88 | 30 | +| 1e-70 | 0.85 | 0.73 | 0.78 | 0.72 | 27 | +| 1e-100 | 0.88 | 0.53 | 0.66 | 0.55 | 21 | -Gene→KO recall stayed between 0.95 and 0.99 from 1e-10 to 1e-30 in all four organisms and fell at stricter cut-offs. The fall was largest for `mge`, whose reaction recall dropped from 0.98 at 1e-30 to 0.87 at 1e-50 and 0.47 at 1e-100. Precision rose with every stricter cut-off in all four organisms, from 0.72 to 0.78 for `sce` between 1e-30 and 1e-50. With the ratios fixed at 0.3 and 0.8, F1 was highest at 1e-50 or 1e-70 in every organism. Novel predictions remained at the strictest cut-off: at 1e-100 the three larger proteomes had 68 to 136 novel reactions and a gene→KO precision of 0.80 to 0.84 (0.87 for `mge`). The study does not show whether these are wrong assignments, paralogs or genes that KEGG has not annotated. +Gene→KO recall was between 0.95 and 0.99 from 1e-10 to 1e-30 in all four organisms and fell at stricter cut-offs. The fall was largest for `mge`, whose reaction recall dropped from 0.98 at 1e-30 to 0.88 at 1e-50 and 0.55 at 1e-100. Precision rose with every stricter cut-off in all four organisms, from 0.73 to 0.78 for `sce` between 1e-30 and 1e-50. With the ratios fixed at 0.3 and 0.8, F1 was highest at 1e-50 or 1e-70 in every organism. Novel predictions remained at the strictest cut-off: at 1e-100 the three larger proteomes had 61 to 137 novel reactions and a gene→KO precision of 0.80 to 0.85 (0.88 for `mge`). The study does not show whether these are wrong assignments, paralogs or genes that KEGG has not annotated. ### Score-ratio sweep -`cutoff` was fixed at `1e-50`. The script scores every combination of `min_score_ratio_ko` (0.0, 0.3, 0.5) and `min_score_ratio_g` (0.5, 0.8, 0.95); the table holds the five combinations per organism that were reported. +`cutoff` was fixed at `1e-50`. The script scores every combination of `min_score_ratio_ko` (0.0, 0.3, 0.5) and `min_score_ratio_g` (0.5, 0.8, 0.95); the table holds the five combinations per organism that are reported. | organism | ko ratio | g ratio | gKO prec | gKO rec | gKO F1 | |---|---|---|---|---|---| | `sce` | 0.0 | 0.50 | 0.61 | 0.96 | 0.74 | -| `sce` | 0.0 | 0.80 | 0.77 | 0.95 | 0.85 | +| `sce` | 0.0 | 0.80 | 0.78 | 0.95 | 0.86 | | `sce` | 0.0 | 0.95 | 0.84 | 0.93 | 0.88 | | `sce` | 0.3 | 0.80 | 0.78 | 0.95 | 0.86 | -| `sce` | 0.5 | 0.80 | 0.80 | 0.95 | 0.86 | -| `cme` | 0.0 | 0.50 | 0.53 | 0.94 | 0.68 | +| `sce` | 0.5 | 0.80 | 0.80 | 0.95 | 0.87 | +| `cme` | 0.0 | 0.50 | 0.53 | 0.93 | 0.68 | | `cme` | 0.0 | 0.80 | 0.69 | 0.93 | 0.79 | -| `cme` | 0.0 | 0.95 | 0.78 | 0.92 | 0.84 | -| `cme` | 0.3 | 0.80 | 0.70 | 0.93 | 0.80 | -| `cme` | 0.5 | 0.80 | 0.70 | 0.93 | 0.80 | -| `eco` | 0.0 | 0.50 | 0.39 | 0.96 | 0.56 | -| `eco` | 0.0 | 0.80 | 0.66 | 0.95 | 0.78 | -| `eco` | 0.0 | 0.95 | 0.76 | 0.94 | 0.84 | -| `eco` | 0.3 | 0.80 | 0.67 | 0.95 | 0.78 | -| `eco` | 0.5 | 0.80 | 0.69 | 0.95 | 0.80 | -| `mge` | 0.0 | 0.50 | 0.62 | 0.85 | 0.72 | -| `mge` | 0.0 | 0.80 | 0.77 | 0.84 | 0.80 | -| `mge` | 0.0 | 0.95 | 0.82 | 0.81 | 0.81 | -| `mge` | 0.3 | 0.80 | 0.77 | 0.84 | 0.80 | -| `mge` | 0.5 | 0.80 | 0.78 | 0.84 | 0.81 | - -At `min_score_ratio_g = 0.80`, raising `min_score_ratio_ko` from 0.0 to 0.5 left recall unchanged and raised precision by 0.03 for `sce` and `eco` and by 0.01 for `cme` and `mge`. - -Raising `min_score_ratio_g` from 0.80 to 0.95 (at `min_score_ratio_ko = 0.0`) raised precision by 0.07 (`sce`), 0.09 (`cme`), 0.10 (`eco`) and 0.05 (`mge`), and lowered recall by 0.01 to 0.03. At 0.50, precision was the lowest of the three values in every organism. The value 0.9 lies between the tested values 0.8 and 0.95 and was not swept. +| `cme` | 0.0 | 0.95 | 0.77 | 0.92 | 0.84 | +| `cme` | 0.3 | 0.80 | 0.69 | 0.93 | 0.79 | +| `cme` | 0.5 | 0.80 | 0.70 | 0.92 | 0.79 | +| `eco` | 0.0 | 0.50 | 0.40 | 0.96 | 0.56 | +| `eco` | 0.0 | 0.80 | 0.67 | 0.96 | 0.79 | +| `eco` | 0.0 | 0.95 | 0.77 | 0.94 | 0.85 | +| `eco` | 0.3 | 0.80 | 0.68 | 0.96 | 0.79 | +| `eco` | 0.5 | 0.80 | 0.70 | 0.95 | 0.81 | +| `mge` | 0.0 | 0.50 | 0.62 | 0.86 | 0.73 | +| `mge` | 0.0 | 0.80 | 0.77 | 0.85 | 0.81 | +| `mge` | 0.0 | 0.95 | 0.82 | 0.85 | 0.83 | +| `mge` | 0.3 | 0.80 | 0.77 | 0.85 | 0.81 | +| `mge` | 0.5 | 0.80 | 0.77 | 0.85 | 0.81 | + +At `min_score_ratio_g = 0.80`, raising `min_score_ratio_ko` from 0.0 to 0.5 raised precision by 0.02 (`sce`), 0.01 (`cme`), 0.03 (`eco`) and 0.00 (`mge`), and lowered recall by 0.01 for `cme` and `eco` and not at all for `sce` and `mge`. + +Raising `min_score_ratio_g` from 0.80 to 0.95 (at `min_score_ratio_ko = 0.0`) raised precision by 0.06 (`sce`), 0.08 (`cme`), 0.10 (`eco`) and 0.05 (`mge`), and lowered recall by 0.00 to 0.02. At 0.50, precision was the lowest of the three values in every organism. The value 0.9 lies between the tested values 0.8 and 0.95 and was not swept. ### Defaults compared with RAVEN 2.11.3 -The table compares the RAVEN 2.11.3 settings `(1e-50, 0.3, 0.8)` with the defaults `(1e-30, 0.3, 0.9)`, given as `(cutoff, min_score_ratio_ko, min_score_ratio_g)`. For the RAVEN 2.11.3 settings the F1 column gives the measured value from the cut-off sweep; for the defaults, which were not part of either sweep, it is computed from the rounded precision and recall in the same row. +The table compares the RAVEN 2.11.3 settings `(1e-50, 0.3, 0.8)` with the defaults `(1e-30, 0.3, 0.9)`, given as `(cutoff, min_score_ratio_ko, min_score_ratio_g)`. All values are scores of the saved hit tables; the RAVEN 2.11.3 values are the 1e-50 rows of the cut-off sweep. -| organism | gKO prec | gKO rec | gKO F1 (measured → computed) | rxn rec | rxn novel | +| organism | gKO prec | gKO rec | gKO F1 | rxn rec | rxn novel | |---|---|---|---|---|---| -| `sce` | 0.78 → 0.76 | 0.95 → 0.96 | 0.86 → 0.85 | 0.96 → 0.96 | 157 → 137 | -| `cme` | 0.70 → 0.67 | 0.93 → 0.96 | 0.80 → 0.79 | 0.94 → 0.97 | 307 → 305 | -| `eco` | 0.67 → 0.67 | 0.95 → 0.97 | 0.78 → 0.79 | 0.98 → 0.99 | 198 → 173 | -| `mge` | 0.77 → 0.69 | 0.84 → 0.94 | 0.80 → 0.80 | 0.87 → 0.97 | 29 → 35 | +| `sce` | 0.78 → 0.77 | 0.95 → 0.96 | 0.86 → 0.85 | 0.96 → 0.96 | 151 → 143 | +| `cme` | 0.69 → 0.67 | 0.93 → 0.96 | 0.79 → 0.79 | 0.93 → 0.99 | 310 → 306 | +| `eco` | 0.68 → 0.69 | 0.96 → 0.97 | 0.79 → 0.81 | 0.98 → 0.99 | 208 → 173 | +| `mge` | 0.77 → 0.70 | 0.85 → 0.94 | 0.81 → 0.80 | 0.88 → 0.97 | 30 → 33 | -With the defaults, gene→KO recall rose by 0.10 for `mge` and by 0.01 to 0.03 for the other three organisms. Gene→KO precision fell by 0.02 (`sce`), 0.03 (`cme`) and 0.08 (`mge`), and stayed at 0.67 for `eco`. F1 changed by at most 0.01. The F1 of the defaults was computed from precision and recall rounded to two decimals, which alone moves it by up to 0.005, and the measured F1 is itself rounded to 0.01, so a difference of 0.01 is within the rounding. The number of novel reactions fell for `sce`, `cme` and `eco` and rose from 29 to 35 for `mge`. +With the defaults, gene→KO recall rose by 0.09 for `mge` and by 0.01 to 0.03 for the other three organisms. Gene→KO precision fell by 0.01 (`sce`), 0.02 (`cme`) and 0.07 (`mge`), and rose by 0.01 for `eco`. F1 changed by at most 0.01 for `sce`, `cme` and `mge` and rose by 0.02 for `eco`. All scores are rounded to two decimals, so a difference of 0.01 lies within the rounding. The number of novel reactions fell for `sce`, `cme` and `eco` and rose from 30 to 33 for `mge`. ### Whole-model comparison of the two KEGG paths @@ -168,33 +171,33 @@ With the defaults, gene→KO recall rose by 0.10 for `mge` and by 0.01 to 0.03 f | Count | annotation path | HMM path | |---|---|---| -| reactions | 1355 | 1461 | -| metabolites | 1501 | 1567 | -| genes | 835 | 896 | +| reactions | 1357 | 1454 | +| metabolites | 1502 | 1565 | +| genes | 838 | 899 | -The HMM path contained 96.3 % of the reactions of the annotation path (1305 of 1355, Jaccard index 0.86) and 96.6 % of its genes (807 of 835, Jaccard index 0.87). The one timed search, `sce` against `eukaryotes.hmm` with `hmmscan` on 12 threads, took 633 s for 6021 queries (about 10.5 minutes, ≈ 0.1 s per query). The `prokaryotes.hmm` library was not timed. +The HMM path contained 96.2 % of the reactions of the annotation path (1306 of 1357, Jaccard index 0.87) and 96.5 % of its genes (809 of 838, Jaccard index 0.87). The counts were measured with the published kegg118 tables and libraries. The search of `sce` against the eukaryote library is timed under [Settings without a sweep](#settings-without-a-sweep); the prokaryote library was not timed. ## Meaning of the results -From 1e-10 to 1e-30 a stricter cut-off raised precision while gene→KO recall fell by at most 0.03 (from 0.98 or 0.99 to between 0.95 and 0.98). Beyond 1e-30 the cut-off traded recall for precision: it removed matched pairs together with novel pairs, and novel pairs remained at every cut-off tested. Across the whole sweep from 1e-10 to 1e-100 precision rose by 0.27 (`sce`, 0.57 to 0.84) to 0.35 (`mge`). The RAVEN 2.11.3 value `1e-50` removes 10 to 50 % of the matched pairs of `mge`, the organism with the weakest matched hits, and 1 to 10 % in the other three. +From 1e-10 to 1e-30 a stricter cut-off raised precision while gene→KO recall fell by at most 0.03 (from 0.98 or 0.99 to between 0.95 and 0.99). Beyond 1e-30 the cut-off traded recall for precision: it removed matched pairs together with novel pairs, and novel pairs remained at every cut-off tested. Across the whole sweep from 1e-10 to 1e-100 precision rose by 0.28 (`sce`, 0.57 to 0.85) to 0.35 (`mge`). The RAVEN 2.11.3 value `1e-50` removes 10 to 50 % of the matched pairs of `mge`, the organism with the weakest matched hits, and 1 to 10 % in the other three. -At `cutoff = 1e-50`, raising `min_score_ratio_g` from 0.80 to 0.95 raised precision by 0.05 to 0.10 for a recall loss of 0.01 to 0.03, and raising it from 0.50 to 0.80 raised precision by 0.15 to 0.27 (`eco`, 0.39 to 0.66). At `min_score_ratio_g = 0.80` and `cutoff = 1e-50`, raising `min_score_ratio_ko` from 0.0 to 0.5 changed precision by 0.01 to 0.03 and recall not at all; its effect at other settings was not reported. The defaults move part of the filtering from the cut-off to `min_score_ratio_g`. On the four organisms the move gained recall, most for `mge`, at the cost of precision in three of them, and left F1 within 0.01 of the RAVEN 2.11.3 settings, a difference within the rounding of the computed F1. +At `cutoff = 1e-50`, raising `min_score_ratio_g` from 0.80 to 0.95 raised precision by 0.05 to 0.10 for a recall loss of 0.00 to 0.02, and raising it from 0.50 to 0.80 raised precision by 0.15 to 0.27 (`eco`, 0.40 to 0.67). At `min_score_ratio_g = 0.80` and `cutoff = 1e-50`, raising `min_score_ratio_ko` from 0.0 to 0.5 raised precision by 0.00 to 0.03 and lowered recall by 0.00 to 0.01; its effect at other settings was not reported. The defaults move part of the filtering from the cut-off to `min_score_ratio_g`. On the four organisms the move gained recall, most for `mge`, at the cost of precision in three of them, and left F1 within 0.01 of the RAVEN 2.11.3 settings for three organisms and raised it by 0.02 for `eco`. ## Chosen defaults | parameter | RAVEN 2.11.3 | default (raven-toolbox and RAVEN 3) | basis | |---|---|---|---| -| `cutoff` (`cutOff`) | 1e-50 | 1e-30 | beyond matched p99 for `sce` and `eco`, between p95 and p99 for `cme` and `mge`; with the new `min_score_ratio_g`, `mge` recall 0.84 → 0.94 | -| `min_score_ratio_g` (`minScoreRatioG`) | 0.8 | 0.9 | from 0.80 to 0.95 it raised precision by 0.05 to 0.10 for a recall loss of 0.01 to 0.03; offsets part of the precision lost to the looser cut-off (inference, 0.9 not swept) | -| `min_score_ratio_ko` (`minScoreRatioKO`) | 0.3 | 0.3 | at `min_score_ratio_g` 0.80 and `cutoff` 1e-50, 0.0 to 0.5 changed precision by 0.01 to 0.03 and recall not at all; kept at the RAVEN value | +| `cutoff` (`cutOff`) | 1e-50 | 1e-30 | beyond matched p99 for `eco`, between p95 and p99 for `sce`, `cme` and `mge`; with `min_score_ratio_g` 0.9, `mge` recall 0.85 → 0.94 | +| `min_score_ratio_g` (`minScoreRatioG`) | 0.8 | 0.9 | from 0.80 to 0.95 it raised precision by 0.05 to 0.10 for a recall loss of 0.00 to 0.02; offsets part of the precision lost to the looser cut-off (inference, 0.9 not swept) | +| `min_score_ratio_ko` (`minScoreRatioKO`) | 0.3 | 0.3 | at `min_score_ratio_g` 0.80 and `cutoff` 1e-50, 0.0 to 0.5 raised precision by 0.00 to 0.03 and lowered recall by 0.00 to 0.01; kept at the RAVEN value | The defaults favour recall: the changed settings raised recall in every organism and lowered precision in three of the four. ## Settings without a sweep -Three settings next to the cut-offs were not measured in the study. The reasons given for their values are not tested. +Two settings next to the cut-offs, `seq_identity` and the model-assembly flags, were not measured in the study. The reasons given for their values are not tested. `threads` was measured in one configuration. -`threads` in `run_hmmsearch` and `build_ko_hmm` defaults to every CPU the process may run on. The value rests on the assumption, not tested here, that HMMER's search gives the same hits at any thread count and that E-values differ at most in the last floating-point places. The documentation gives single-threaded search against the full KO library (more than 26,000 HMMs) as 30 to 60 minutes per proteome, against roughly five minutes multi-threaded. The source of these figures is not recorded, and no run with a recorded thread count and hardware supports either of them. The one timed run, `sce` against `eukaryotes.hmm` on 12 threads, took 633 s (about 10.5 minutes). +`threads` in `run_hmmsearch` and `build_ko_hmm` defaults to every CPU the process may run on. `hmmsearch` 3.4 of `sce` (6021 sequences) against `kegg118_eukaryotes.hmm` (2698 profiles) returned 10,293 hit rows for 2198 genes and 1998 KOs at 16, 12, 4 and 1 threads, and the parsed hit tables (KO, gene, E-value) were identical. One run per thread count took 89.7 s (16 threads), 88.6 s (12), 96.4 s (4) and 242.5 s (1), on an AMD EPYC 9354 node used exclusively (2026-11-01). Between 16 and 4 threads the time changes by less than 9 %. `build_ko_hmm` was not timed. `seq_identity` in `build_ko_hmm` is `0.9`: the identity at which CD-HIT collapses near-duplicate sequences within a KO before the HMM is built; the clustering keeps a KO with many near-identical entries from giving an overfitted profile. It is CD-HIT's recommended value for proteins and the default of `seqIdentity` in RAVEN 2.11.3. `seq_identity=-1` skips the clustering. RAVEN 3's `getKEGGModelForOrganism` uses prebuilt HMM libraries and has no clustering step. @@ -211,11 +214,23 @@ The model-assembly flags decide which reactions a draft keeps, and have the same - All four organisms are in the set of sequences from which the libraries were built, so their own sequences contribute to their KO profiles. Recall on these organisms is therefore an upper bound for an organism outside KEGG (inference; a genome absent from KEGG was not tested). The comparison between settings is relative and gives no absolute accuracy. - A novel pair that is a correct assignment KEGG has not made counts as an error, so the precision figures are a lower bound. The study does not measure how many novel pairs are correct. -- Four organisms, one KEGG release (118) and one hit table per organism. Each search ran once, so its run-to-run spread, and its dependence on the thread count, were not measured; the scoring of a hit table by `assign_kos` involves no random step. +- Four organisms, the KEGG tables of release 118 and one hit table per organism. For `sce` the parsed hit tables were identical in four runs at 16, 12, 4 and 1 threads; the other organisms were searched once. The scoring of a hit table by `assign_kos` involves no random step. - The score-ratio sweep used `cutoff = 1e-50` only, and `min_score_ratio_g = 0.9` was not part of either sweep. ## Reproduction -The sweeps are produced by [`scripts/analyze_hmm_cutoffs.py`](https://github.com/CompMetEng/raven-toolbox/blob/develop/scripts/analyze_hmm_cutoffs.py), one organism per call, with the KEGG artefacts, the organism's proteome and the HMM library (or a saved `hmmscan --tblout` file) as input. It prints the E-value percentiles, the cut-off sweep and the full score-ratio grid. The comparison of the defaults with RAVEN 2.11.3 and the whole-model comparison are not produced by this script. +The scripts are in [`scripts/parameter_tuning/kegg/`](https://github.com/CompMetEng/raven-toolbox/tree/develop/scripts/parameter_tuning/kegg) of raven-toolbox. They search with `hmmsearch` from the toolbox binary bundle (`resolve_binary("hmmsearch")`) and take the published kegg118 tables and HMM libraries through `ensure_kegg_data()` and `ensure_kegg_hmm_library("eukaryotes")` or `("prokaryotes")`. + +| Script | Produces | +|---|---| +| `fetch_proteomes.py` | the proteomes from the KEGG REST lists and from UniProt | +| `analyze_hmm_cutoffs.py` | the organism header, E-value percentiles, cut-off sweep at (0.3, 0.8), the score-ratio grid and the defaults row (1e-30, 0.3, 0.9) | +| `check_true_reactions.py` | the true-reaction counts under both definitions | +| `compare_kegg_paths.py` | the whole-model comparison of the annotation and HMM paths | +| `time_hmmsearch.py` | the timing and hit-table identity at several thread counts | + +`analyze_hmm_cutoffs.py` is called once per organism with `--artefacts`, `--org`, `--proteome`, `--library` and `--threads`. `--save-tblout` writes the `hmmsearch --tblout` table, `--tblout` scores a saved table without searching, `--json` writes the results, and `--restrict-to-reference` limits `ko_reaction` to reactions of the reference model, which is the definition behind the reaction columns above. The gene→KO columns and the percentiles do not depend on that option. `compare_kegg_paths.py` is called with `--org`, `--proteome`, `--artefacts`, `--library`, `--domain`, `--threads` and `--settings`, for example `--settings 1e-50,0.3,0.8 1e-30,0.3,0.9`. + +The proteomes (`kegg_rest/.faa`, `uniprot/.faa`, `counts.json`) are in the archived input bundle `kegg_proteomes_2026-11-01.tar.gz`, and the four `hmmsearch --tblout` tables in `kegg_hmmsearch_tblout_2026-11-01.tar.gz`. -With raven-toolbox at commit `94f3940` the script does not run: it imports `run_hmmscan` and `parse_hmmscan_tblout` from `reconstruction/kegg/query.py`, which provides `run_hmmsearch` and `parse_hmmsearch_tblout` instead. Its cut-off sweep calls `assign_kos` with the default ratios, which are 0.3 and 0.9 at that commit, so it no longer reproduces the sweep at 0.3 and 0.8. +The scores of the tables come from the run of 2026-11-01 with raven-toolbox commit `94f3940`, HMMER 3.4 and the published kegg118 tables and libraries, searched with 16 threads on an AMD EPYC 9354 node. Rescoring each saved hit table with `--tblout` gives the saved scores exactly. Not checked: whether the KEGG REST sequences of 2026-10-31 equal the sequences of KEGG release 118 (the sequence counts were compared, not the sequences), and whether `hmmsearch -Z` E-values equal `hmmscan` E-values on these inputs, because the binary bundle has no `hmmscan`. The UniProt proteomes in the bundle were downloaded and counted but not searched. diff --git a/docs/parameter-tuning/studies/sampling-convergence-calibration.md b/docs/parameter-tuning/studies/sampling-convergence-calibration.md index 7da2b83..8fe6e3c 100644 --- a/docs/parameter-tuning/studies/sampling-convergence-calibration.md +++ b/docs/parameter-tuning/studies/sampling-convergence-calibration.md @@ -4,93 +4,105 @@ ## Terms -- The **flux polytope** is the set of flux vectors `v` with `S v = 0` and `lb <= v <= ub`, where `S` is the stoichiometric matrix and `lb` and `ub` are the reaction bounds. A **vertex** is a corner point of the polytope. -- A **Markov chain Monte Carlo (MCMC)** sampler is a random walk inside the polytope whose visited points, after enough steps, are distributed close to uniformly over it. One walk from one starting point and one random seed is a **chain**. -- A **hit-and-run** step chooses a direction, finds the line segment through the current point along that direction that lies inside the polytope, and moves to a uniformly random point on that segment. -- **ACHR** (Artificially Centered Hit-and-Run) takes its directions from warmup points and a running centre of the chain, and does not transform the polytope. raven-toolbox wraps `cobra.sampling.ACHRSampler`, which computes its warmup points with two linear programs (LPs) per reaction before the first step ([Random sampling](../../guide/sampling.md)). -- **CHRR** (Coordinate Hit-and-Run with Rounding) first runs a flux variability analysis (FVA), fixes the reactions whose range is narrower than `fixed_width_tol`, reduces the polytope to a full-dimensional body through the nullspace of the equality constraints, and **rounds** it: it maps the body so that its maximum-volume inscribed ellipsoid (**MVE**) becomes a unit ball. The walk then moves along one coordinate per step. -- **`thinning`** is the number of steps between two stored samples. An ACHR chain with `n_samples` stored samples therefore takes `thinning × n_samples` steps after its warmup points, and a CHRR chain takes `warmup + thinning × n_samples` steps after the rounding. -- **`warmup`** is the number of CHRR steps discarded before the first stored sample. `random_sampling` passes it to CHRR only; the ACHR runs ignore it. -- The **lag-1 autocorrelation** `ρ` is the correlation between consecutive stored samples of one chain. -- The **effective sample size (ESS)** is the number of independent samples that carry the same information as the correlated ones. Under the assumption that the chain is a first-order autoregressive process, `ESS = n × (1−ρ) / (1+ρ)` for `n` stored samples. -- The **R-hat** of a reaction (Gelman-Rubin statistic) compares the flux variance between chains with the variance within chains. With `m` chains of `n` samples, `W` the mean within-chain variance and `B/n` the variance of the chain means, `R-hat = sqrt(((n−1)/n × W + B/n) / W)`. R-hat is 1.0 when the chains agree and grows when the distribution of a chain depends on where it started. The script reports the share of reactions above 1.1 and above 1.01, the stricter of the two thresholds. +- The flux polytope is the set of flux vectors `v` with `S v = 0` and `lb <= v <= ub`, where `S` is the stoichiometric matrix and `lb` and `ub` are the reaction bounds. A vertex is a corner point of the polytope. +- A Markov chain Monte Carlo (MCMC) sampler is a random walk inside the polytope whose visited points, after enough steps, are distributed close to uniformly over it. One walk from one starting point and one random seed is a chain. +- A hit-and-run step chooses a direction, finds the line segment through the current point along that direction that lies inside the polytope, and moves to a uniformly random point on that segment. +- ACHR (Artificially Centered Hit-and-Run) takes its directions from warmup points and a running centre of the chain, and does not transform the polytope. raven-toolbox wraps `cobra.sampling.ACHRSampler`, which computes its warmup points with two linear programs (LPs) per reaction before the first step ([Random sampling](../../guide/sampling.md)). +- CHRR (Coordinate Hit-and-Run with Rounding) first runs a flux variability analysis (FVA), fixes the reactions whose range is narrower than `fixed_width_tol`, reduces the polytope to a full-dimensional body through the nullspace of the equality constraints, and rounds it: it maps the body so that its maximum-volume inscribed ellipsoid (MVE) becomes a unit ball. The walk then moves along one coordinate per step. +- `thinning` is the number of steps between two stored samples. An ACHR chain with `n_samples` stored samples therefore takes `thinning × n_samples` steps after its warmup points, and a CHRR chain takes `warmup + thinning × n_samples` steps after the rounding. +- `warmup` is the number of CHRR steps discarded before the first stored sample. `random_sampling` passes it to CHRR only; the ACHR runs ignore it. +- The lag-1 autocorrelation `ρ` is the correlation between consecutive stored samples of one chain. +- The effective sample size (ESS) is the number of independent samples that carry the same information as the correlated ones. Under the assumption that the chain is a first-order autoregressive process, `ESS = n × (1−ρ) / (1+ρ)` for `n` stored samples. +- The R-hat of a reaction (Gelman-Rubin statistic) compares the flux variance between chains with the variance within chains. With `m` chains of `n` samples, `W` the mean within-chain variance and `B/n` the variance of the chain means, `R-hat = sqrt(((n−1)/n × W + B/n) / W)`. R-hat is 1.0 when the chains agree and grows when the distribution of a chain depends on where it started. The script reports the share of reactions above 1.1 and above 1.01, the stricter of the two thresholds. ## Set-up -Driver for the comparisons between chains: [`scripts/analyze_sampling_convergence.py`](https://github.com/CompMetEng/raven-toolbox/blob/docs/parameter-defaults/scripts/analyze_sampling_convergence.py). It starts `n_chains` calls of `random_sampling` with seeds 0 to `n_chains − 1`, one process per chain through `ProcessPoolExecutor`, because cobra models are not guaranteed to be safe for concurrent solves in threads. Each process reads the model from the SBML file. The script computes R-hat per reaction with the formula in [Terms](#terms). A reaction with the same constant flux in every chain has `W = 0` and no variance between the chain means, gets no R-hat, and is excluded as **not scored**. A reaction that is constant within each chain at different values in different chains has `W = 0` and an infinite R-hat, and is scored. +Driver for the comparisons between chains: `scripts/analyze_sampling_convergence.py` in raven-toolbox. It starts `n_chains` calls of `random_sampling` with seeds 0 to `n_chains − 1`, one process per chain through `ProcessPoolExecutor`, because cobra models are not guaranteed to be safe for concurrent solves in threads. Each process reads the model from the SBML file. The script computes R-hat per reaction with the formula in [Terms](#terms). A reaction with the same constant flux in every chain has `W = 0` and no variance between the chain means, gets no R-hat, and is excluded as not scored. A reaction that is constant within each chain at different values in different chains has `W = 0` and an infinite R-hat, and is scored. -Settings: `n_samples=300`, `thinning=100`, `warmup=1000` and 4 chains, unless a table states otherwise. `thinning` and `warmup` are at their defaults; the default `n_samples` is 1000. The solver was Gurobi, on a machine with 12 cores. Software versions were not recorded. Each configuration ran once. cobrapy's `OptGPSampler` was not tested, because `random_sampling` does not call it; its methods are `'achr'`, `'chrr'` and `'random_objective'`. +Settings: `n_samples=300`, `thinning=100`, `warmup=1000` and 4 chains, unless a table states otherwise. `thinning` and `warmup` are at their defaults; the default `n_samples` is 1000. The solver was Gurobi, with 12 cores per job on AMD EPYC 9354 nodes. All figures are from runs on 2026-11-01 with Python 3.14.2, cobra 0.32.1, optlang 1.9.1, gurobipy 13.0.3 (Gurobi library 13.0.3) and raven-toolbox `94f3940`. Each configuration ran once, except e_coli_core ACHR, which was repeated; the single-chain runs also ran with seeds 1 to 3 (see [Autocorrelation within one chain](#autocorrelation-within-one-chain)). cobrapy's `OptGPSampler` was not tested, because `random_sampling` does not call it; its methods are `'achr'`, `'chrr'` and `'random_objective'`. -The models are two files of yeast-GEM. The single-chain runs were made on a yeast-GEM with 4,102 reactions, the same reaction count as the bundled `docs/data/yeast-GEM.xml` (yeastGEM_v9.1.0); which file they read was not recorded. The comparisons between chains read a yeast-GEM file with 4105 reactions, the count that the script reports; its version was not recorded. +The models are two files of yeast-GEM and one of e_coli_core. The single-chain runs read the bundled `docs/data/yeast-GEM.xml` (yeastGEM_v9.1.0, 4,102 reactions). The comparisons between chains read yeast-GEM v9.1.1 (4,105 reactions), which differs from the develop branch at commit `ea756c2` only in the model identifier and two dates. e_coli_core is `textbook.xml.gz` of cobrapy 0.32.1 (model identifier `e_coli_core`). -The wall time of a comparison between chains is the time from starting the processes to the return of the last chain: reading the model, the warmup and the sampling, without the R-hat computation. What the single-chain wall times include, and how `ρ` was combined over the reactions, was not recorded. +The wall time of a comparison between chains is the time from starting the processes to the return of the last chain: reading the model, the warmup and the sampling, without the R-hat computation. The wall time of a single-chain run is the time of reading the SBML file plus `random_sampling`. The `ρ` of a chain is the mean over the reactions with a flux range above `1e-6` in that chain of the lag-1 Pearson correlation of the reaction (`pandas.Series.autocorr(1)`); the median over the reactions is higher and is given in the text. The ESS uses the mean. ## Results ### Autocorrelation within one chain -One ACHR chain on yeast-GEM (4,102 reactions), `n_samples=300`, at three values of `thinning`. The ESS column is computed from the formula in [Terms](#terms). +One ACHR chain on yeast-GEM (4,102 reactions), `n_samples=300`, seed 0, at three values of `thinning`. The ESS column is computed from the formula in [Terms](#terms). | `thinning` | Lag-1 autocorrelation | ESS of 300 stored samples | Wall time (s) | |---|---|---|---| -| 20 | 0.973 | ~4 | 660 | -| 100 (default) | 0.926 | ~12 | 841 | -| 500 | 0.849 | ~25 | 927 | +| 20 | 0.9725 | 4.2 | 189.5 | +| 100 (default) | 0.9361 | 9.9 | 203.6 | +| 500 | 0.8251 | 28.8 | 223.3 | -At `thinning=500`, consecutive samples were 85 % correlated. The fivefold step from 20 to 100 lowered `ρ` by 0.047, and the next fivefold step lowered it by 0.077. Reaching a `ρ` of about 0.3, an approximate level for near-independent samples, needs a `thinning` above the measured range, and three points do not establish how `ρ` decays beyond 500. +The reactions with a flux range above `1e-6` numbered 2506, 2535 and 2575 of 4102. The median `ρ` over these reactions is 0.9829, 0.9519 and 0.8586. At `thinning=500`, consecutive samples were 83 % correlated. The fivefold step from 20 to 100 lowered `ρ` by 0.036, and the next fivefold step lowered it by 0.111. Three points do not establish how `ρ` decays beyond 500. -`n_samples` was not varied. With `ρ` fixed at its value for the default `thinning=100`, the ESS formula is proportional to `n_samples`: ~12 from 300 stored samples, ~38 from the default 1000, and 100 needs ~2,600 stored samples. Scaled in proportion from 841 s for 300 samples, 2,600 samples take ~7,300 s (~2 h). The wall time rose from 660 s to 927 s (1.4-fold) while the number of steps rose 25-fold, so most of the time of these runs did not depend on `thinning`. If the wall time is a fixed cost per run plus a cost per stored sample, the scaled estimate is an upper bound for one chain; the thinning sweep kept `n_samples` at 300 and cannot separate the two parts. +The same settings with seeds 0 to 3 give the following `ρ`. -For the choice of settings: within the measured range, a larger `thinning` gave more ESS per second of wall time (~4 in 660 s at 20, ~25 in 927 s at 500). That a larger `n_samples` at a fixed `thinning` raises ESS in proportion and leaves `ρ` unchanged follows from the first-order autoregressive formula; it was not measured. +| `thinning` | Seed 0 | Seed 1 | Seed 2 | Seed 3 | Range | +|---|---|---|---|---|---| +| 20 | 0.9725 | 0.9244 | 0.9790 | 0.9753 | 0.924 to 0.979 | +| 100 | 0.9361 | 0.6523 | 0.9238 | 0.9286 | 0.652 to 0.936 | +| 500 | 0.8251 | 0.6444 | 0.8013 | 0.7694 | 0.644 to 0.825 | + +For every seed, `ρ` falls as `thinning` rises. Between seeds, `ρ` differs by up to 0.28, and the ESS of 300 stored samples ranges from 3.2 to 11.8 at `thinning=20`, from 9.9 to 63.1 at 100 and from 28.8 to 64.9 at 500. Seed 1 gives the lowest `ρ` at all three values; without it, the ranges are 0.972 to 0.979, 0.924 to 0.936 and 0.769 to 0.825. The four chains of the yeast-GEM comparison with 4,105 reactions (seeds 0 to 3, `thinning=100`) give a mean `ρ` of 0.951, 0.847, 0.947 and 0.911, so a low value for seed 1 appears on both model files. The spread between seeds is 0.10 on the 4,105-reaction chains (0.847 to 0.951) and 0.28 on the 4,102-reaction chains (0.652 to 0.936); seed 1 is the lowest in both. + +`n_samples` was not varied. With `ρ` fixed at its seed-0 value for the default `thinning=100`, the ESS formula is proportional to `n_samples`: ~10 from 300 stored samples, ~33 from the default 1000, and 100 needs ~3,000 stored samples. Building the ACHR warmup points alone took 185.9 s, 196.6 s and 195.4 s in the three runs, so the sampling steps took about 1 s, 5 s and 26 s for 6,000, 30,000 and 150,000 steps. The warmup is therefore ~95 % of the wall time at `thinning=100`, and the wall time rose from 189.5 s to 223.3 s (1.2-fold) while the number of steps rose 25-fold. With a fixed cost per run plus a cost per stored sample, 3,025 stored samples take ~242 s for one chain; scaling the 203.6 s in proportion to 3,025 samples gives 2,053 s (~34 min), an upper bound. The warmup time was measured on a second, fresh sampler after the run, so the split is approximate. + +For the choice of settings: within the measured range, a larger `thinning` gave more ESS per second of wall time (4.2 in 189.5 s at 20, 28.8 in 223.3 s at 500). That a larger `n_samples` at a fixed `thinning` raises ESS in proportion and leaves `ρ` unchanged follows from the first-order autoregressive formula; it was not measured. ### Agreement between chains on e_coli_core -ACHR, 4 chains × 300 samples, 34.7 s wall time. +ACHR, 4 chains × 300 samples, 10.9 s wall time (5.4 s in a repeat). The repeat gave the same R-hat for every reaction, so on e_coli_core a chain is deterministic for a given seed and software. Reproducibility on yeast-GEM was not tested; the `random_objective` run on yeast-GEM is not reproducible (see [Other sampling settings](#other-sampling-settings)). The difference between 10.9 s and 5.4 s is unexplained. | Measure | Value | |---|---:| | reactions scored (non-constant) | 87 / 95 | -| R-hat median | 1.0071 | -| R-hat p90 | 1.0369 | -| R-hat max | 1.3028 | -| reactions with R-hat > 1.01 | 42 (48.3 %) | -| reactions with R-hat > 1.1 | 1 (1.1 %) | +| R-hat median | 1.0027 | +| R-hat p90 | 1.0171 | +| R-hat max | 1.0276 | +| reactions with R-hat > 1.01 | 12 (13.8 %) | +| reactions with R-hat > 1.1 | 0 (0.0 %) | Reactions with the highest R-hat: | Reaction | R-hat | |---|---:| -| `EX_succ_e` | 1.3028 | -| `EX_co2_e` | 1.0420 | -| `CO2t` | 1.0420 | -| `FUM` | 1.0390 | -| `EX_h_e` | 1.0383 | -| `TKT2`, `RPE`, `TKT1`, `TALA`, `G6PDH2r` | 1.0369 (tied) | +| `EX_succ_e` | 1.0276 | +| `FRD7` | 1.0266 | +| `SUCDi` | 1.0265 | +| `MALS`, `ICL` | 1.0208 | +| `SUCOAS`, `AKGDH` | 1.0203 | +| `ICDHyr` | 1.0193 | +| `ME1` | 1.0191 | +| `MDH` | 1.0157 | -One reaction, the succinate exchange `EX_succ_e` (R-hat 1.30), exceeded 1.1, and 42 of 87 scored reactions exceeded 1.01. The chains reached different distributions for that reaction, a failure that the autocorrelation of one chain cannot detect. +No reaction exceeded 1.1, and 12 of 87 scored reactions exceeded 1.01. The reaction with the highest R-hat is the succinate exchange `EX_succ_e`. -For the choice of settings: on a 95-reaction model, the default `thinning` with 300 samples per chain gave agreement at the 1.1 threshold for 86 of 87 scored reactions. Agreement at 1.01 needs more steps or a different method (see [CHRR on e_coli_core](#chrr-on-e_coli_core)). +For the choice of settings: on a 95-reaction model, the default `thinning` with 300 samples per chain gave agreement at the 1.1 threshold for 87 of 87 scored reactions and at 1.01 for 75 of 87. Agreement at 1.01 for the remaining 12 needs more steps or a different method; CHRR did not remove it (see [CHRR on e_coli_core](#chrr-on-e_coli_core)). ### Agreement between chains on yeast-GEM -ACHR, 4 chains × 300 samples, 2524.4 s wall time (~42 min). +ACHR, 4 chains × 300 samples, 572.3 s wall time (~9.5 min). | Measure | Value | |---|---:| -| reactions scored (non-constant) | 3364 / 4105 | -| R-hat median | 1.1671 | -| R-hat p90 | 1.6414 | -| R-hat max | 9.9416 | -| reactions with R-hat > 1.01 | 3246 (96.5 %) | -| reactions with R-hat > 1.1 | 2271 (67.5 %) | +| reactions scored (non-constant) | 3325 / 4105 | +| R-hat median | 1.1020 | +| R-hat p90 | 1.3793 | +| R-hat max | 18.8976 | +| reactions with R-hat > 1.01 | 3167 (95.2 %) | +| reactions with R-hat > 1.1 | 1685 (50.7 %) | -Reactions with the highest R-hat: `r_0318` (9.94), `r_0307` (9.65), `r_1690` (8.89), `r_1077` (8.59), `r_2625` (5.92), `r_1072` (4.21), `r_4015` (3.88), `r_2690` (3.43), `r_1113` (3.42), `r_1648` (3.42). +Reactions with the highest R-hat: `r_4562` (18.90), `r_4358` (18.90), `r_0992` (5.18), `r_4249` (5.18), `r_4701` (5.18), `r_4621` (5.18), `r_1965` (5.09), `r_0917` (3.90), `r_0891` (3.90), `r_0918` (3.90). -The median scored reaction exceeded 1.1: two in three reactions failed that threshold, and 3.5 % passed 1.01. Each chain took 300 × 100 = 30,000 ACHR steps after its warmup points. The low ESS of one chain and the disagreement between chains are two separate measurements, and both show that these settings do not give a converged sample of yeast-GEM. +The median scored reaction was 1.1020, and one in two reactions (50.7 %) exceeded 1.1; 4.8 % passed 1.01. Each chain took 300 × 100 = 30,000 ACHR steps after its warmup points. The low ESS of one chain and the disagreement between chains are two separate measurements, and both show that these settings do not give a converged sample of yeast-GEM. -The four parallel chains took 2524 s, ~3 times the 841 s of the single chain at the same settings. The two runs read different yeast-GEM files (4105 and 4,102 reactions). Contention between the four Gurobi processes on the 12 cores is a possible cause of the difference; it was not measured. +The four parallel chains took 572.3 s, 2.8 times the 203.6 s of the single chain at the same settings. The two runs read different yeast-GEM files (4,105 and 4,102 reactions). Contention between the four Gurobi processes on the 12 cores is a possible cause of the difference; it was not measured. -For the choice of settings: at the default `thinning` and `n_samples=300`, the flux distributions of most reactions in a genome-scale model depend on the seed. The default `n_samples=1000` (100,000 steps per chain) was not tested. +For the choice of settings: on yeast-GEM with `n_samples=300` and `thinning=100`, 50.7 % of the 3325 scored reactions had R-hat above 1.1. The default `n_samples=1000` (100,000 steps per chain) was not tested. ### Reallocating the ACHR step budget @@ -98,15 +110,15 @@ The 30,000 steps per chain of the default run (`thinning=100, n_samples=300`), s | Measure | t=100, n=300 (default thinning) | t=300, n=100 (reallocated) | |---|---:|---:| -| wall time | 2524.4 s | 3451.0 s | -| R-hat median | 1.1671 | 1.1636 | -| R-hat p90 | 1.6414 | 1.6542 | -| R-hat max | 9.9416 | 9.8345 | -| R-hat > 1.01 | 96.5 % | 95.2 % | -| R-hat > 1.1 | 67.5 % | 67.2 % | -| worst reaction | `r_0318` (9.94) | `r_0318` (9.83) | +| wall time | 572.3 s | 612.4 s | +| R-hat median | 1.1020 | 1.1010 | +| R-hat p90 | 1.3793 | 1.3778 | +| R-hat max | 18.8976 | 21.9072 | +| R-hat > 1.01 | 95.2 % | 93.5 % | +| R-hat > 1.1 | 50.7 % | 50.5 % | +| worst reaction | `r_4562`, `r_4358` (18.90) | `r_4562`, `r_4358` (21.91) | -The reallocation moved the median R-hat from 1.1671 to 1.1636 and the shares above the two thresholds by at most 1.3 percentage points, the reaction with the highest R-hat was the same, and the run took 57.5 min against 42 min. That the disagreement comes from how ACHR moves through this polytope, and not from the split of the steps, is an inference from this result and from the CHRR result on e_coli_core; it was not tested at genome scale. +Both runs scored 3325 reactions. The reallocation moved the median R-hat from 1.1020 to 1.1010 and the shares above the two thresholds by at most 1.7 percentage points, the reactions with the highest R-hat were the same, and the run took 612.4 s against 572.3 s (1.07 times). That the disagreement comes from how ACHR moves through this polytope, and not from the split of the steps, is an inference from this result; it was not tested at genome scale. For the choice of settings: at a fixed number of steps, moving steps from `n_samples` to `thinning` did not improve agreement between chains. A larger total number of steps was not tested with R-hat. @@ -116,57 +128,69 @@ For the choice of settings: at a fixed number of steps, moving steps from `n_sam | Measure | ACHR | CHRR | |---|---:|---:| -| wall time | 34.7 s | 702.0 s | +| wall time | 10.9 s | 716.9 s | | reactions scored | 87 | 95 (0 excluded as constant) | -| R-hat median | 1.0071 | 1.0049 | -| R-hat p90 | 1.0369 | 1.0146 | -| R-hat max | 1.3028 | 1.0248 | -| R-hat > 1.01 | 48.3 % | 17.9 % | -| R-hat > 1.1 | 1.1 % | 0.0 % | +| R-hat median | 1.0027 | 1.0039 | +| R-hat p90 | 1.0171 | 1.0137 | +| R-hat max | 1.0276 | 1.0377 | +| R-hat > 1.01 | 13.8 % (12) | 14.7 % (14) | +| R-hat > 1.1 | 0.0 % | 0.0 % | -With CHRR, no reaction exceeded 1.1; the highest R-hat was `ICDHyr` at 1.0248, against `EX_succ_e` at 1.30 with ACHR. The share above 1.01 fell from 48.3 % to 17.9 %. CHRR took ~20 times the wall time of ACHR. The percentages have different denominators, 87 and 95 scored reactions. The 8 reactions that ACHR left constant were scored with CHRR. CHRR computes every flux of a sample from the nullspace basis (`v0 + N x`); that rounding error in this product makes those 8 reactions vary is an inference that was not checked. +With either method, no reaction exceeded 1.1. CHRR did not agree better than ACHR on e_coli_core: its highest R-hat was `EX_succ_e` at 1.0377 against 1.0276 for ACHR, and 14.7 % of its reactions exceeded 1.01 against 13.8 %. CHRR took 66 times the wall time of ACHR (133 times that of the 5.4 s repeat), one run each. The percentages have different denominators, 87 and 95 scored reactions. The 8 reactions that ACHR left constant (all exactly 0.0) were scored with CHRR. CHRR computes every flux of a sample from the nullspace basis (`v0 + N x`); these 8 reactions vary within a chain by at most 1.6e-13, with means of the order of 1e-14, which is the size of floating-point rounding error in that product. -For the choice of settings: on a 95-reaction model, CHRR with the default `thinning` and 300 samples per chain gave agreement at 1.1 for every reaction. +For the choice of settings: on a 95-reaction model, ACHR and CHRR with the default `thinning` and 300 samples per chain both gave agreement at 1.1 for every reaction, and CHRR cost 66 times as much time (133 times against the repeat) without lowering the share above 1.01. ### CHRR at genome scale -CHRR was not run on yeast-GEM with 4 chains × 300 samples, because 20 times the 2524 s of the ACHR run gives ~14 h. A probe with 2 chains × 20 samples (`thinning=100`) took 4815.3 s (~80 min), 1.9 times the 2524 s of the full ACHR run. yeast-GEM has 43 times the reactions of e_coli_core. +CHRR was not run on yeast-GEM with 4 chains × 300 samples. A probe with 2 chains × 20 samples (`thinning=100`) took 8147.9 s (~136 min), 14.2 times the 572.3 s of the full ACHR run, with a peak memory of about 31 GB of the 32 GB allocated. Each chain runs an FVA with its own worker pool; that the pools cause the memory use was not tested. yeast-GEM has 43 times the reactions of e_coli_core. Each CHRR chain ran an FVA, a nullspace computation and the MVE rounding before its first step, and then 1,000 warmup steps and 2,000 sampling steps. The probe does not separate these parts. That the cost before the first step dominates, and that most of it is the MVE rounding, is an inference from the small number of steps; the parts were not timed. `random_sampling` computes this transform anew in every call, so each chain pays it. -The R-hat of the probe (median 5.16, p90 86, max in the billions) does not show that CHRR converges worse than ACHR at genome scale. With 20 samples per chain, the within-chain variance `W` is estimated from too few samples to be stable: a reaction that shows almost no variance in 20 draws by chance, combined with any difference between the chain means (`B/n`), gives a large R-hat whatever its distribution. The probe cannot distinguish a CHRR that does not converge on yeast-GEM from an R-hat that needs more samples; a run with enough samples for a stable R-hat has the cost problem of the full run. +The R-hat of the probe (median 5.7274, p90 165.18, max 1.899e9 for `r_4782`; all 4105 reactions scored, 97.3 % above 1.01 and 93.6 % above 1.1) does not show that CHRR converges worse than ACHR at genome scale. With 20 samples per chain, the within-chain variance `W` is estimated from too few samples to be stable: a reaction that shows almost no variance in 20 draws by chance, combined with any difference between the chain means (`B/n`), gives a large R-hat whatever its distribution. The probe cannot distinguish a CHRR that does not converge on yeast-GEM from an R-hat that needs more samples; a run with enough samples for a stable R-hat has the cost problem of the full run. For the choice of settings: 20 samples per chain are too few for R-hat on yeast-GEM. Whether CHRR converges on yeast-GEM with 300 samples per chain is open. ## Choice of `thinning` and `n_samples` -The defaults stay `method='achr'`, `thinning=100` and `n_samples=1000` ([Tuned parameter defaults](../../tuned-parameters.md)). On a model the size of e_coli_core, the default `thinning` with 300 samples per chain gave chains that agree at R-hat 1.1 for all but one reaction. On yeast-GEM, no setting tested gave converged samples: a `thinning` up to 500 left `ρ` at 0.849, and moving the same 30,000 steps from `n_samples` to `thinning` moved the median R-hat from 1.1671 to 1.1636 and the shares above 1.01 and 1.1 by at most 1.3 percentage points. Flux ranges from `random_sampling` with the default settings on a genome-scale model are unconverged for most reactions. +The defaults are `method='achr'`, `thinning=100` and `n_samples=1000` ([Tuned parameter defaults](../../tuned-parameters.md)). CHRR is not the default: on e_coli_core, CHRR differed from ACHR by +0.0012 in median R-hat (1.0027 and 1.0039), by −0.0034 in p90 (1.0171 and 1.0137) and by +0.0101 in maximum (1.0276 and 1.0377), with one run each and no measured spread between seed sets, and CHRR took 66 to 133 times as long. On a model the size of e_coli_core, the default `thinning` with 300 samples per chain gave chains that agree at R-hat 1.1 for every reaction. On yeast-GEM, no setting tested gave converged samples: a `thinning` up to 500 left `ρ` at 0.825 for seed 0 (0.644 to 0.825 over four seeds), and moving the same 30,000 steps from `n_samples` to `thinning` moved the median R-hat from 1.1020 to 1.1010 and the shares above 1.01 and 1.1 by at most 1.7 percentage points. On yeast-GEM with `n_samples=300` and `thinning=100`, 50.7 % of the 3325 scored reactions had R-hat above 1.1. -The `random_sampling` docstring gives these numbers and recommends, for genome-scale models, a `thinning` of at least 1000 or a larger `n_samples`, `method='chrr'`, or an ESS or R-hat check. A `thinning` of 1000 or more was not measured. The driver script runs the R-hat check: it samples several chains with different seeds at given settings and reports the share of reactions above the R-hat thresholds, so a sampling result can be checked at the settings that produced it. +The `random_sampling` docstring recommends, for genome-scale models, a `thinning` of at least 1000 or a larger `n_samples`, `method='chrr'`, or an ESS or R-hat check. A `thinning` of 1000 or more was not measured. The driver script runs the R-hat check: it samples several chains with different seeds at given settings and reports the share of reactions above the R-hat thresholds, so a sampling result can be checked at the settings that produced it. -One change outside the parameters was identified and not pursued: CHRR recomputes its rounding transform in every call. The chains run in separate processes at the same time, so to save that cost the transform would have to be computed once per model and passed to every chain. The saving depends on the share of the rounding in the 80 min of the probe, which was not measured. +One change outside the parameters was identified and not pursued: CHRR recomputes its rounding transform in every call. The chains run in separate processes at the same time, so to save that cost the transform would have to be computed once per model and passed to every chain. The saving depends on the share of the rounding in the 136 min of the probe, which was not measured. ## Other sampling settings Two settings of `method='random_objective'` differ from RAVEN MATLAB's `randomSampling` on purpose. -`replace_max_bound` (default `False`; MATLAB's `replaceBoundsWithInf` defaults to true) replaces the largest upper bound, the **big-M** value that stands for an unconstrained reaction, with infinity before sampling, and replaces the most negative lower bound, if it is below 0, with minus infinity. RAVEN-convention models use 1000 as the big-M for almost every reaction (4,083 of the 4,102 reactions of yeast-GEM). With every such bound replaced, the random-objective LP becomes unbounded, because the objective can grow without limit through any reaction that is then unconstrained. With `False`, 200 samples completed, with 0.57 % of them at the bound. +`replace_max_bound` (default `False`; MATLAB's `replaceBoundsWithInf` defaults to true) replaces the largest upper bound, the big-M value that stands for an unconstrained reaction, with infinity before sampling, and replaces the most negative lower bound, if it is below 0, with minus infinity. RAVEN-convention models use 1000 as the big-M for almost every reaction (4,083 of the 4,102 reactions of yeast-GEM). With `True`, the call fails after 9.6 s with `OptimizationError: Solver status is 'unbounded'`. A probe of 50 random objectives, drawn as the sampler draws them on the model with the bounds replaced, gave 49 optimal results and 1 unbounded result (one probe of 50 objectives). One unbounded objective stops all draws: the abort mechanism is read from the cobra code, where `Model.optimize` raises on the status `unbounded` (`raise_error=False` does not change this) and the retry loop of the sampler never sees the status. The sample at which the run aborted was not recorded. With `False`, 200 samples completed (68.2 s with 12 processes, including the loopless FVA). Counting a flux as at the bound when `|v| ≥ 1000 × (1 − 1e-9)`, the 200 × 4102 sample matrix (820,400 entries) had 54 entries at the bound (0.0066 %) in one run, with 15 samples containing at least one such reaction and 23 reactions (0.56 % of 4102) at the bound in at least one sample. A second run had 269 entries (0.033 %), 16 samples (8.0 %) and 32 reactions (0.78 %). A share of 0.56 % of reactions at the bound occurred in one run and 0.78 % in another; the share of entries is 0.0066 to 0.033 %. + +`random_objective` with a fixed seed is not reproducible on yeast-GEM: the loopless FVA returned 2307 and 2308 eligible reactions in two runs with the same seed. That a different eligible set shifts every random draw is an inference from these two counts. `loopless_good_reactions` (default `True`) selects which reactions are eligible as random objectives. MATLAB excludes every reaction whose FVA maximum is infinite or above 0.999 times the largest upper bound (999 on a model capped at 1000), and, when the smallest lower bound is negative, every reaction whose FVA minimum is minus infinity or below 0.999 times that bound; it treats proximity to the big-M as a sign of a thermodynamic loop. It also excludes reactions that cannot carry a non-zero flux and, in an ecModel, the `usage_prot_` reactions. That rule also excludes reactions that reach the bound for metabolic reasons. A loopless FVA excludes only the reactions that take part in infeasible cycles, at a higher cost. ## Limits -- Each configuration ran once. The spread of R-hat and `ρ` between repeated runs with other seeds was not measured, so differences of the size seen in the reallocation (0.0035 in the median) are within an unknown run-to-run spread. -- One small and one genome-scale model were tested. The single-chain runs and the comparisons between chains used two different yeast-GEM files, and software versions were not recorded. -- R-hat is computed in its original form, without split chains or rank normalisation. +- Each comparison between chains ran once, except the repeated e_coli_core ACHR run, with seeds 0 to `n_chains − 1`. The spread of R-hat between sets of seeds was not measured, so differences of the size seen in the reallocation (0.0010 in the median) are within an unknown spread. The spread of `ρ` between seeds was measured for the single chain and reaches 0.28. +- One small and one genome-scale model were tested. The single-chain runs and the comparisons between chains used two different yeast-GEM files (v9.1.0 and v9.1.1). +- The ACHR wall times and the e_coli_core R-hat values depend on the software versions and the load of the node. +- The wall times depend on the other jobs on the shared nodes and on the Gurobi thread count, which was left at its default and not logged. +- R-hat is computed in its basic form, without split chains or rank normalisation. - The default `n_samples=1000`, a `thinning` above 500, and CHRR on yeast-GEM with more than 20 samples per chain were not tested. ## Reproduction +The driver for the comparisons between chains is `scripts/analyze_sampling_convergence.py` in raven-toolbox. The other scripts are in `scripts/parameter_tuning/sampling/`: `summarize_chains.py`, `single_chain_autocorr.py` (autocorrelation of one chain), `random_objective_bounds.py` (`replace_max_bound`) and `posthoc_checks.py`. + ```bash python scripts/analyze_sampling_convergence.py \ --model /path/to/yeast-GEM.xml --out work/ \ --n-chains 4 --n-samples 300 --thinning 100 --warmup 1000 +python scripts/parameter_tuning/sampling/single_chain_autocorr.py \ + --model /path/to/yeast-GEM.xml --thinning 100 --n-samples 300 \ + --warmup 1000 --seed 0 --time-warmup --out results/ +python scripts/parameter_tuning/sampling/random_objective_bounds.py \ + --model /path/to/yeast-GEM.xml --n-samples 200 --seed 0 \ + --n-proc 12 --n-probe 50 --out results/ ``` -The CHRR runs use the same command with `--method chrr`. The script caches the chains per model, method, `n_chains`, `n_samples`, `thinning` and `warmup` under `--out`; a run with a cached configuration computes R-hat from the stored chains without sampling. Another `--thinning` or `--n-samples` samples again and is cached separately. On yeast-GEM, four ACHR chains in parallel on 12 cores took ~3 times the wall time of one chain at the same settings; the two runs read different yeast-GEM files and the cause of the difference was not measured (see [Agreement between chains on yeast-GEM](#agreement-between-chains-on-yeast-gem)). A sweep at genome scale can therefore take more time than one chain per configuration suggests. +The CHRR runs use the same command with `--method chrr`. The script caches the chains per model, method, `n_chains`, `n_samples`, `thinning` and `warmup` under `--out`; a run with a cached configuration computes R-hat from the stored chains without sampling. Another `--thinning` or `--n-samples` samples again and is cached separately. On yeast-GEM, four ACHR chains in parallel on 12 cores took 2.8 times the wall time of one chain at the same settings; the two runs read different yeast-GEM files and the cause of the difference was not measured (see [Agreement between chains on yeast-GEM](#agreement-between-chains-on-yeast-gem)). A sweep at genome scale can therefore take more time than one chain per configuration suggests. diff --git a/docs/tuned-parameters.md b/docs/tuned-parameters.md index 43634f2..9ee5fa8 100644 --- a/docs/tuned-parameters.md +++ b/docs/tuned-parameters.md @@ -15,10 +15,10 @@ RAVEN MATLAB: `randomSampling` (`method='achr'|'chrr'|'random_objective'`), `sam | Parameter | Default | How determined | |---|---|---| | `method` | `'achr'` (Python); `'achr'` (RAVEN 3, `randomSampling`) | Literature: ACHR gives near-uniform interior sampling, and the random-objective method draws polytope vertices. All three methods are selectable in both implementations (`'randomObjective'` in MATLAB, `'random_objective'` in Python). RAVEN 2.11.3 had only the random-objective method. | -| `thinning` | `100` | Upstream convention (cobrapy's own ACHR default). Measured insufficient at genome scale: on yeast-GEM one ACHR chain had an effective sample size (ESS) of about 12 from 300 stored samples, and 4 chains disagreed (median Gelman-Rubin R-hat 1.1671, 67.5 % of scored reactions above 1.1). At `thinning=500` the lag-1 autocorrelation of one chain was 0.849. Agreement between chains was tested at one other setting, `thinning=300` with `n_samples=100` (the same 30,000 steps per chain), and did not improve. See [Flux sampling convergence and settings](parameter-tuning/studies/sampling-convergence-calibration.md). | +| `thinning` | `100` | Upstream convention (cobrapy's own ACHR default). Measured insufficient at genome scale: on yeast-GEM one ACHR chain (seed 0) had an effective sample size (ESS) of about 10 from 300 stored samples (9.9 to 63.1 over four seeds), and 4 chains disagreed (median Gelman-Rubin R-hat 1.1020, 50.7 % of scored reactions above 1.1). At `thinning=500` the lag-1 autocorrelation of one chain was 0.825 for seed 0 (0.644 to 0.825 over four seeds, ESS 28.8 to 64.9). Agreement between chains was tested at one other setting, `thinning=300` with `n_samples=100` (the same 30,000 steps per chain), and did not improve. See [Flux sampling convergence and settings](parameter-tuning/studies/sampling-convergence-calibration.md). | | `warmup` | `1000` | Used by `method='chrr'` only (the steps discarded before the first stored sample); the ACHR sampler of cobrapy has no such parameter. Same as `nBurnin` of RAVEN's `sampleCHRR` and `randomSampling`, which also applies to CHRR only. Not measured. | | `n_objectives` | `2` | Literature: [Bordel et al. 2010](https://doi.org/10.1371/journal.pcbi.1000859). | -| `replace_max_bound` | `False` | Used by `method='random_objective'`. The study gives the mechanism and records no run with `True`: RAVEN-convention models use 1000 as the bound of almost every reaction, and with every such bound replaced by infinity the random-objective LP becomes unbounded, because the objective can grow without limit through any reaction that is then unconstrained. With `False`, 200 samples completed. Kept different from MATLAB on purpose. Whether MATLAB's own solver path handles `True` was not tested here. | +| `replace_max_bound` | `False` | Used by `method='random_objective'`. With `True` the run fails with an unbounded solver status: one probe of 50 objectives gave 49 optimal results and 1 unbounded result. From the cobra code, one unbounded objective stops all draws; the sample at which the run aborted was not recorded. With `False`, 200 samples completed. Kept different from MATLAB on purpose. Whether MATLAB's own solver path handles `True` was not tested. | | `loopless_good_reactions` | `True` | Loopless FVA excludes only reactions that take part in infeasible cycles, at higher cost; the alternative fixed-threshold (±999) heuristic also excludes reactions that legitimately reach capacity. The loopless technique is implemented in Python only; porting it is a larger project than changing a default. | | `n_samples`, `seed`, `min_flux`, `max_attempts` | `1000`, `None`, `False`, `100` | Same in both implementations or no evidence favoured a different value. | | `flux_tol` (in `find_good_reactions`) | `1e-9` | Python-only helper (supports target selection of the random-objective method); no cross-implementation comparison. | @@ -33,9 +33,9 @@ RAVEN MATLAB: `FSEOF`. raven-toolbox: `fseof`. | Parameter | Default | How determined | |---|---|---| -| `n_steps` | `10` | Literature: [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10). | -| `max_fraction` | `0.9` | Literature: [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10); measured: values below 0.9 produce spurious targets on iJO1366. | -| `correlation_threshold` | `0.9` | Literature: [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10); measured: 0.7 adds 4 spurious amplification targets on iJO1366. | +| `n_steps` | `10` | Literature: [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10); measured: on iJO1366, 5 and 20 steps change the target set by 4 and 6 targets ([Benchmarks](parameter-tuning/benchmarks.md)). | +| `max_fraction` | `0.9` | Literature: [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10); measured: on iJO1366, 0.5 and 0.99 change the target set by 11 and 2 targets, so the sweep does not select 0.9 ([Benchmarks](parameter-tuning/benchmarks.md)). | +| `correlation_threshold` | `0.9` | Literature: [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10); measured: on iJO1366, 0.7 adds 4 amplification and 8 knockdown or knockout targets and 0.95 drops 2 and 2 ([Benchmarks](parameter-tuning/benchmarks.md)). | | `flux_eps` | `1e-6` | Measured on iJO1366: the classification uses the fixed thresholds `1e-10` and `1e-8`, as RAVEN does; `flux_eps` gates the check against the theoretical maximum, and the target counts are identical at `1e-8`, `1e-7` and `1e-6`. | Full detail: [Benchmarks of `fseof`, `remove_genes` and other defaults](parameter-tuning/benchmarks.md) @@ -49,15 +49,15 @@ Only MATLAB has tINIT (`runINIT`, `getINITModel`); raven-toolbox implements ftIN | Parameter | Default | How determined | |---|---|---| | `prod_weight` | `0.5` | Literature: [Ågren et al. 2012](https://doi.org/10.1371/journal.pcbi.1002518) (the original INIT paper). | -| `allow_excretion` | `False` | Recorded, without a measurement, as having no effect at the default `prod_weight` in the three INIT entry points that existed when the value was set; no measurement is recorded on this site. raven-toolbox has dropped the two tINIT ones. | -| `mip_gap` | `None` in Python (solver default); `0.0004` in MATLAB | Acts on the step-0 solve and on any step without a schedule of MILP runs; a named series such as `1+1` ignores it, because every step runs its own schedule. Measured on Human-GEM (HCT116) in the step-0 solve: the time is flat at 24 to 25 s, the Jaccard index to the result at the tightest gap (0.0002) is 0.9976 at 0.001 and 0.9908 at 0.01, and falls below 0.97 from 0.03. The Python default `None` needs no change. | -| `time_limit` | `None` in Python; `5000` seconds in MATLAB | RAVEN's `ftINIT` sets the Gurobi `TimeLimit` of a MILP run without its own limit to 5000 seconds, and the step schedules of `getINITSteps` use runs of 120 s, 5000 s and 5 s. In Python, `ftinit` with a named series caps each scheduled run by its own limit and `time_limit` lowers that cap further; `time_limit=None` leaves an ad-hoc step or a `run_ftinit` call uncapped, but no scheduled run. The task gap-fill of `ftinit` is a separate MILP with a fixed limit of 300 s per task and does not read `time_limit`. Measured on Human-GEM HCT116 (single runs): step-0 solves took 23 to 30 s, and full pipelines 81 to 119 s with scaling and 157 s without. In the robustness runs of [ftINIT parameter calibration and input robustness](parameter-tuning/studies/init-param-calibration.md) (degraded input, with tasks), the gap-fill of one 90 %-dropout model ran longer than 75 minutes; the time limit of that run is not recorded. `None` is the default. The study ran its full pipelines at `time_limit=600`, and its robustness script sets `time_limit=120`; the limit of the recorded robustness runs is not stated. The docstring of `run_ftinit` recommends 120 to 600 s per step for hard or noisy input; no measurement on this site tests that range. | +| `allow_excretion` | `False` | Recorded, without a measurement, as having no effect at the default `prod_weight`; no measurement is recorded on this site. | +| `mip_gap` | `None` in Python (solver default); `0.0004` in MATLAB | Acts on the first-step solve and on any step without a schedule of MILP runs; a named series such as `1+1` ignores it, because every step runs its own schedule. Steps are counted from 1 here; the logs count from 0, so the logged `step 1` is the second step. Measured on Human-GEM (HCT116) in the first-step solve: the time is flat at 24 to 25 s, the Jaccard index to the result at the tightest gap (0.0002) is 0.9976 at 0.001 and 0.9908 at 0.01, and falls below 0.97 from 0.03. The Python default `None` needs no change. | +| `time_limit` | `None` in Python; `5000` seconds in MATLAB | RAVEN's `ftINIT` sets the Gurobi `TimeLimit` of a MILP run without its own limit to 5000 seconds, and the step schedules of `getINITSteps` use runs of 120 s, 5000 s and 5 s. In Python, `ftinit` with a named series caps each scheduled run by its own limit and `time_limit` lowers that cap further; `time_limit=None` leaves an ad-hoc step or a `run_ftinit` call uncapped, but no scheduled run. The task gap-fill of `ftinit` is a separate MILP with a fixed limit of 300 s per task and does not read `time_limit`. Measured on Human-GEM HCT116 (single runs): first-step solves took 23 to 30 s, and full pipelines 81 to 119 s with scaling and 157 s without. In the robustness runs of [ftINIT parameter calibration and input robustness](parameter-tuning/studies/init-param-calibration.md) (degraded input, with tasks), the gap-fill of one 90 %-dropout model ran longer than 75 minutes; the time limit of that run is not recorded. `None` is the default. The study ran its full pipelines at `time_limit=600`, and its robustness script sets `time_limit=120`; the limit of the recorded robustness runs is not stated. The docstring of `run_ftinit` recommends 120 to 600 s per step for hard or noisy input; no measurement on this site tests that range. | | `series` | `'1+1'` | Literature: [Gustafsson et al. 2023](https://doi.org/10.1073/pnas.2217868120) (the ftINIT paper). | -| `force_on` | `0.1` | RAVEN's original value; measured near-insensitive across the range 0.02–0.5. | +| `force_on` | `0.1` | RAVEN's value; measured near-insensitive across the range 0.02–0.5. | | `big_m` | `1000.0` | The cap on each scored reaction's own big-M, as in `ftINITInternalAlg`: the big-M of a reaction is `max(min(ub, big_m), min(-lb, big_m), 0)`, raised to `force_on` where the "on" floor needs it, and an infinite bound counts as `big_m`. The cap limits the flux a reaction can carry when it is on (it is the largest big-M of a scored reaction in its on/off constraint) and sets no bound on the reaction in the model. | | `eps` | `1.0` | Same in both implementations. | -| `resolve_ties`, `prove_abs_gap` | `False`, `None`, both opt-in; MATLAB has them as `resolveTies` and `proveAbsGap` at the same defaults | `resolve_ties` fixes the answer among equal-score optima in phases: it holds the score objective at its optimum, minimises the count of kept removable (negative-score) reactions, then among those minimises the summed id rank of the kept ones (a merged reaction is ranked by the smallest original id of its group), then takes the fluxes with the least total absolute flux (relative and absolute gap 1e-6), then those with the least id-rank-weighted backward flux. The last ftINIT step skips the last two phases, because no later step reads its fluxes. Measured on Human-GEM v2.0.0-38 (one cell line, seeds 1234 and 7): with the default escalation the seed-to-seed difference is 110 reactions and 17 predicted essential genes. `resolve_ties=True` gives identical reaction sets at a fixed seed and reduces the seed-to-seed difference to 41 reactions and 1 gene, at about 3.6 times the run time (about 14 min against 4 min at `time_limit=1800`). With `prove_abs_gap=1.0` as well, the difference is 1 reaction and 0 genes, at about 26 times the baseline run time (about 100 min); in the four sweep builds with `prove_abs_gap=1.0`, step 1 took 3305 to 3650 s (53 to 73 % of a build), mostly in its two flux phases, and step 2 another 1650 to 2500 s; no build without `resolve_ties` was run with `prove_abs_gap`. The flux phases (`phase2c-minflux`, `phase2d-direction`) reach the time limit in most builds, and a warning names each phase adopted at the limit. A tie-break that finds no incumbent in phase 2a or 2b is dropped without a warning. The escalation raises no warning either: in the four `resolve_ties=True` runs without `prove_abs_gap` of the `prove_abs_gap` sweep, its step 1 objective ended 4.0 to 12.0 below the proven optimum. A build without a warning is therefore not necessarily proven or tie-broken. | -| `factor`, `max_score`, `min_score` (in `gene_scores_from_expression`) | `5.0`, `10.0`, `-5.0` | RAVEN's own formula. An attribution to "Wang et al. 2012" was checked and could not be confirmed; no source in either implementation cites a paper for it. | +| `resolve_ties`, `prove_abs_gap` | `False`, `None`, both opt-in; MATLAB has them as `resolveTies` and `proveAbsGap` at the same defaults | `resolve_ties` fixes the answer among equal-score optima in phases: it holds the score objective at its optimum, minimises the count of kept removable (negative-score) reactions, then among those minimises the summed id rank of the kept ones (a merged reaction is ranked by the smallest original id of its group), then takes the fluxes with the least total absolute flux (relative and absolute gap 1e-6), then those with the least id-rank-weighted backward flux. The last ftINIT step skips the last two phases, because no later step reads its fluxes. Measured on Human-GEM v2.0.0-38 (one cell line, seeds 1234 and 7): with the default escalation the seed-to-seed difference is 110 reactions and 17 predicted essential genes. `resolve_ties=True` gives identical reaction sets at a fixed seed and reduces the seed-to-seed difference to 41 reactions and 1 gene, at about 3.6 times the run time (about 14 min against 4 min at `time_limit=1800`). With `prove_abs_gap=1.0` as well, the difference is 1 reaction and 0 genes, at about 26 times the baseline run time (about 100 min); in the four sweep builds with `prove_abs_gap=1.0`, step 1 took 3305 to 3650 s (59.0 to 66.3 % of a build), mostly in its two flux phases, and step 2 another 1650 to 2500 s; no build without `resolve_ties` was run with `prove_abs_gap`. The flux phases (`phase2c-minflux`, `phase2d-direction`) reach the time limit in most builds, and a warning names each phase adopted at the limit. A tie-break that finds no incumbent in phase 2a or 2b is dropped without a warning. The escalation raises no warning either: in the four `resolve_ties=True` runs without `prove_abs_gap` of the `prove_abs_gap` sweep, its step 1 objective ended 4.0 to 12.0 below the proven optimum. A build without a warning is therefore not necessarily proven or tie-broken. | +| `factor`, `max_score`, `min_score` (in `gene_scores_from_expression`) | `5.0`, `10.0`, `-5.0` | RAVEN's own formula. A 2012 paper (Wang et al. 2012, [mCADRE](https://doi.org/10.1186/1752-0509-6-153)) uses categorical scoring and is not the origin; no source in either implementation cites a paper for it. | | `score_reactions_from_genes` and `classify_reactions` arguments | various | Same in both implementations or standard practice. | Full detail: [ftINIT parameter calibration and input robustness](parameter-tuning/studies/init-param-calibration.md) (the primary measurement, genome-scale Human-GEM) · [ftINIT on different solvers](parameter-tuning/studies/init-solver-benchmark.md) · [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](parameter-tuning/studies/ftinit-determinism.md) (`resolve_ties=True` gives identical reaction sets at a fixed seed and a seed-to-seed difference of 41 reactions and 1 gene; adding `prove_abs_gap=1.0` reduces it to 1 reaction and 0 genes at about 26 times the run time) · [Human-GEM cell-type model validation against RAVEN](parameter-tuning/studies/humangem-validation.md) (Jaccard index against RAVEN across 5 cell lines of 0.964 to 0.982 for models extracted without tasks and 0.981 to 0.988 for task-constrained models) @@ -69,7 +69,7 @@ RAVEN MATLAB: `fillGaps` (connectivity mode), ported as `connect_blocked_reactio | Parameter | Default | How determined | |---|---|---| | `epsilon` (in `fill_gaps_fast_lp`) | `0.0001` | Literature: [Thiele et al. 2014](https://doi.org/10.1093/bioinformatics/btu321) (the fastGapFill paper). | -| `threshold` (in `connect_blocked_reactions`) | `1.0` | The flux a reaction must be able to exceed to count as unblocked. Same as the fixed threshold of 1 in RAVEN's `fillGaps`, and suited to models in the RAVEN convention, where exchange reactions have bounds of ±1000. Not measured. | +| `threshold` (in `connect_blocked_reactions`) | `1.0` | The flux a reaction must be able to exceed to count as unblocked. Same as the fixed threshold of 1 in RAVEN's `fillGaps`, and suited to models in the RAVEN convention, where exchange reactions have bounds of ±1000. Measured on one synthetic network with a maximum flux of 0.5, one run per value: 0.01 and 0.1 filled the gap, 0.5 and 1.0 did not ([Benchmarks](parameter-tuning/benchmarks.md)). | | `penalty`, `allow_net_production` (in `connect_blocked_reactions`) | `1.0`, `False` | No measurement recorded. `penalty=1.0` makes the MILP minimise the number of added reactions. `allow_net_production=False` keeps the steady-state balance `S @ v == 0`, the stricter test; `True` relaxes it to `S @ v >= 0`. | | `big_m` (in `fill_gaps_milp`) | `1000.0` | Matches RAVEN's ±1000 bound convention. It caps an unbounded template reaction only when the merged model has no finite bound; otherwise the largest finite bound does. | @@ -81,29 +81,28 @@ RAVEN MATLAB: `getModelFromHomology`, `getBlast`, `getDiamond`. raven-toolbox: ` | Parameter | Default | How determined | |---|---|---| -| `evalue` (`run_blast`) | `1e-4` | Matches MATLAB's `getBlast` (hardcodes `-evalue 10e-5`). On *H. polymorpha* against *S. cerevisiae*, loosening it from `1e-5` to `1e-4` added 5,204 raw hits (8.9 % of the table), all of which fail the `max_evalue` filter (`1e-30`) of `get_model_from_homology`. | +| `evalue` (`run_blast`) | `1e-4` | Matches MATLAB's `getBlast` (hardcodes `-evalue 10e-5`). On *H. polymorpha* against *S. cerevisiae*, loosening it from `1e-5` to `1e-4` added 5,200 raw hits (8.9 % of the table); 5,310 of the 5,314 rows found only at `1e-4` fail `max_evalue=1e-30` of `get_model_from_homology`; 4 have an E-value below `1e-30`. | | `evalue` (`run_diamond`) | `1e-3` | Matches MATLAB's `getDiamond`, which passes no `-evalue` flag and so inherits DIAMOND's own default. The value is deliberately different from `run_blast`; each aligner follows its own MATLAB call. | -| `threads` | every CPU the process may run on (Python); the cores assigned to MATLAB (MATLAB) | A 500-query BLAST subset returned the same 2,469 hits at one thread and at four, in 45.2 s and 23.7 s (1.9 times faster). Both sides use all available cores. | -| `max_evalue` | `1e-30` | The score was identical from `1e-4` to `1e-50` on the two organisms tested (at identity 40 % and length 200) and fell at `1e-100`. That identity and alignment length already exclude whatever a looser E-value would admit is an inference, not checked against the hit tables. | -| `min_align_len` | `100`; RAVEN 2.11.3 used `200` | Measured against the KEGG ortholog reference for 4 organisms of increasing distance from *S. cerevisiae*: compared with `200`, a length of 100 raised the score (F0.5) by 0.010 to 0.020, with 3 to 4 more recall (percentage points of recall, or per cent more correct matches; the values are not tabulated and the unit is not recorded) and at most 0.6 percentage points less precision; 50 and 100 gave the same score. No OMA scores were reported for length. RAVEN 3's `getModelFromHomology` uses `minLen` 100 too. | -| `min_identity` | `40` | Same study: the setting with the largest effect on the score. With a wrongly transferred reaction weighted as worse than a missing one (F0.5), the optimum was 35 or 40 % against KEGG and 35 or 45 % against OMA, depending on the organism. | +| `threads` | every CPU the process may run on (Python); the cores assigned to MATLAB (MATLAB) | A 500-query BLAST subset returned the same 2,746 hits at one thread and at four, in 20.4 s and 5.9 s (3.5 times faster; *H. polymorpha* against *S. cerevisiae*, BLAST+ 2.17.0, 2026-11-01). Both sides use all available cores. | +| `max_evalue` | `1e-30` | At identity 40 % and length 200 the score was identical from `1e-4` to `1e-30` in four organisms (BLAST, F0.5, KEGG reference). `1e-50` lowered it by 0.002 to 0.006 in three organisms and left it unchanged in *K. lactis*, and `1e-100` lowered it by 0.020 to 0.063. | +| `min_align_len` | `100`; RAVEN 2.11.3 used `200` | Measured against the KEGG ortholog reference for 4 organisms of increasing distance from *S. cerevisiae*: compared with `200`, a length of 100 raised the score (F0.5) by 0.010 to 0.020, with 2.2 to 3.8 percentage points more recall and at most 0.6 percentage points less precision; 50 and 100 gave the same score. No OMA scores were reported for length. RAVEN 3's `getModelFromHomology` uses `minLen` 100 too. | +| `min_identity` | `40` | Same study: over 25 to 50 % the score changed by 0.062 to 0.275, against 0.047 to 0.055 over alignment lengths 50 to 300 and 0.020 to 0.063 over E-values 1e-4 to 1e-100. With a wrongly transferred reaction weighted as worse than a missing one (F0.5, BLAST), the optimum was 35 or 40 % against KEGG and 35 or 45 % against OMA, depending on the organism; with DIAMOND against OMA the optimum for *E. coli* is 30 %. | | `bidirectional` | `True` | Same in both implementations. | -A part of the study that scored against curated GEMs (hanpo-GEM, rhto-GEM) built by the same reconstruction method was retired because the result was circular: agreement with the curated model peaked at exactly the build settings of that model and collapsed one step past them. - -Full detail: [Homology cut-off calibration](parameter-tuning/studies/homology-cutoff-calibration.md) (the primary measurement, including the curated-GEM comparison that the study does not use) +Scoring against curated GEMs (hanpo-GEM, rhto-GEM) built by the same reconstruction method is not part of the 2026-11-01 measurement. +Full detail: [Homology cut-off calibration](parameter-tuning/studies/homology-cutoff-calibration.md) ## KEGG-based reconstruction RAVEN MATLAB: `getKEGGModelForOrganism` (the exact internal name of the KO-assignment sub-step has not been independently confirmed). raven-toolbox: `assign_kos`, `run_hmmsearch`, `build_ko_hmm`, and the `get_kegg_model_*` family. | Parameter | Default | How determined | |---|---|---| -| `cutoff` (in `assign_kos`) | `1e-30`; RAVEN 2.11.3 used `1e-50` | Measured against real KEGG gene→KO annotations across 4 organisms of varying study depth: `1e-50` lies inside the E-value distribution of the matched pairs and removes 1 to 10 % of them in three organisms and 10 to 50 % in *M. genitalium*. Precision rose with every stricter cut-off (*S. cerevisiae* 0.72 at `1e-30`, 0.78 at `1e-50`); recall fell by at most 0.03 down to `1e-30` and fell further at stricter cut-offs. Together with `min_score_ratio_g` 0.9, the default raised recall in all four organisms and kept F1 within 0.01. The F1 of the default was computed from precision and recall rounded to two decimals, which alone moves it by up to 0.005, so a difference of 0.01 is within the rounding. RAVEN 3's `getKEGGModelForOrganism` uses `cutOff` `10^-30` too. | -| `min_score_ratio_g` (in `assign_kos`) | `0.9`; RAVEN 2.11.3 used `0.8` | Same study: from 0.80 to 0.95 it raised precision by 0.05 to 0.10 at a recall loss of 0.01 to 0.03, and from 0.50 to 0.80 by 0.15 to 0.27; 0.9 itself was not swept. RAVEN 3's `getKEGGModelForOrganism` uses `minScoreRatioG` `0.9` too. | -| `min_score_ratio_ko` (in `assign_kos`) | `0.3` | Same study: at `min_score_ratio_g` 0.80 and `cutoff` `1e-50`, raising it from 0.0 to 0.5 changed precision by 0.01 to 0.03 and recall not at all; other combinations were not reported. Kept at the RAVEN value. | +| `cutoff` (in `assign_kos`) | `1e-30`; RAVEN 2.11.3 used `1e-50` | Measured against real KEGG gene→KO annotations across 4 organisms of varying study depth: `1e-50` lies inside the E-value distribution of the matched pairs and removes 1 to 10 % of them in three organisms and 10 to 50 % in *M. genitalium*. Precision rose with every stricter cut-off (*S. cerevisiae* 0.73 at `1e-30`, 0.78 at `1e-50`); recall fell by at most 0.03 down to `1e-30` and fell further at stricter cut-offs. Together with `min_score_ratio_g` 0.9, the default raised recall in all four organisms and changed F1 by at most 0.01 in three organisms and raised it by 0.02 in *E. coli*. RAVEN 3's `getKEGGModelForOrganism` uses `cutOff` `10^-30` too. | +| `min_score_ratio_g` (in `assign_kos`) | `0.9`; RAVEN 2.11.3 used `0.8` | Same study: from 0.80 to 0.95 it raised precision by 0.05 to 0.10 at a recall loss of 0.00 to 0.02, and from 0.50 to 0.80 by 0.15 to 0.27; 0.9 itself was not swept. RAVEN 3's `getKEGGModelForOrganism` uses `minScoreRatioG` `0.9` too. | +| `min_score_ratio_ko` (in `assign_kos`) | `0.3` | Same study: at `min_score_ratio_g` 0.80 and `cutoff` `1e-50`, raising it from 0.0 to 0.5 raised precision by 0.00 to 0.03 and lowered recall by 0.00 to 0.01; other combinations were not reported. Kept at the RAVEN value. | | `seq_identity` (in `build_ko_hmm`) | `0.9` | Literature: the CD-HIT recommendation for protein clustering. | -| `threads` (`run_hmmsearch`, `build_ko_hmm`) | every CPU the process may run on | Not measured. The value assumes that HMMER gives the same hits at any thread count, with E-values that differ at most in the last floating-point places; [KEGG HMM cut-off calibration](parameter-tuning/studies/kegg-hmm-cutoff-calibration.md) did not test this. | +| `threads` (`run_hmmsearch`, `build_ko_hmm`) | every CPU the process may run on | `hmmsearch` on *S. cerevisiae* gave identical parsed hit tables on 16, 12, 4 and 1 threads, with run times of 89.7, 88.6, 96.4 and 242.5 s ([KEGG HMM cut-off calibration](parameter-tuning/studies/kegg-hmm-cutoff-calibration.md)). `build_ko_hmm` was not measured. | | `keep_spontaneous`, `keep_undefined_stoich`, `keep_incomplete`, `keep_general` (`get_kegg_model_*`) | `True`, `True`, `True`, `False` | Same in both implementations or standard draft-reconstruction practice. | Full detail: [KEGG HMM cut-off calibration](parameter-tuning/studies/kegg-hmm-cutoff-calibration.md) (the primary measurement, 4 organisms against real KEGG annotations) @@ -164,16 +163,16 @@ Policy: where Python and MATLAB RAVEN disagree on a default, one value is chosen The INIT parameters `mip_gap` and `time_limit` of `run_init` and `run_ftinit` are resolved by the genome-scale test with real expression data in [ftINIT parameter calibration and input robustness](parameter-tuning/studies/init-param-calibration.md) (Human-GEM HCT116 and Hart2015), without matching the MATLAB numbers: -- `mip_gap`: solve time is flat across the gap in the single step-0 solve, so a tight gap is nearly free there. The docstring of `run_ftinit` attributes the flat time to model construction; the study did not measure the share of model construction. A named series such as `1+1` ignores `mip_gap` and `mip_gap_abs`, because every step runs its own schedule of MILP runs, so the full pipeline gives the same result at 0.001, 0.003 and 0.01. Neither 0.001 nor 0.01 is MATLAB's `0.0004`. Keep `None` (Gurobi's own `~1e-4` default is at least as tight as the measured-good `0.001`). -- `time_limit`: RAVEN's `ftINIT` limits a MILP run to 5000 seconds when the run sets none, and its schedules use 120 s, 5000 s and 5 s runs. Python's `ftinit` with a named series caps each scheduled run by the same limits, and `time_limit` lowers the cap; `time_limit=None` leaves an ad-hoc step or a `run_ftinit` call uncapped, but no scheduled run. Measured step-0 solves took 23 to 30 s, and full pipelines 81 to 119 s with scaling and 157 s without. In the robustness setup, the gap-fill of one 90 %-dropout model ran longer than 75 minutes; the gap-fill has its own fixed limit of 300 s per task and does not read `time_limit`. The study ran its full pipelines at 600 s; its robustness script sets 120 s, and the limit of the recorded robustness runs is not stated. Keep `None` as the code default. The docstring of `run_ftinit` recommends `120-600 s/step`, a range with no measured basis on this site. +- `mip_gap`: solve time is flat across the gap in the single first-step solve, so a tight gap is nearly free there. The docstring of `run_ftinit` attributes the flat time to model construction; the study did not measure the share of model construction. A named series such as `1+1` ignores `mip_gap` and `mip_gap_abs`, because every step runs its own schedule of MILP runs, so the full pipeline gives the same result at 0.001, 0.003 and 0.01. Neither 0.001 nor 0.01 is MATLAB's `0.0004`. Keep `None` (Gurobi's own `~1e-4` default is at least as tight as the measured-good `0.001`). +- `time_limit`: RAVEN's `ftINIT` limits a MILP run to 5000 seconds when the run sets none, and its schedules use 120 s, 5000 s and 5 s runs. Python's `ftinit` with a named series caps each scheduled run by the same limits, and `time_limit` lowers the cap; `time_limit=None` leaves an ad-hoc step or a `run_ftinit` call uncapped, but no scheduled run. Measured first-step solves took 23 to 30 s, and full pipelines 81 to 119 s with scaling and 157 s without. In the robustness setup, the gap-fill of one 90 %-dropout model ran longer than 75 minutes; the gap-fill has its own fixed limit of 300 s per task and does not read `time_limit`. The study ran its full pipelines at 600 s; its robustness script sets 120 s, and the limit of the recorded robustness runs is not stated. Keep `None` as the code default. The docstring of `run_ftinit` recommends `120-600 s/step`, a range with no measured basis on this site. -[ftINIT on different solvers](parameter-tuning/studies/init-solver-benchmark.md) measured GLPK honouring the 900 s limit: both steps of the full pipeline hit it, the run took 2182 s and kept 6758 reactions (Jaccard 0.892 against Gurobi). The `scipy` interface (HiGHS) fails because it supports continuous variables only. Gurobi is the backend the genome-scale numbers above were measured on, and the only one that completed without a step hitting the limit. +[ftINIT on different solvers](parameter-tuning/studies/init-solver-benchmark.md) measured GLPK honouring the 900 s limit: both steps of the full pipeline hit it, the run took 2182 s and kept 6758 reactions (Jaccard 0.892 against Gurobi). The `scipy` interface (HiGHS) fails because it supports continuous variables only. Gurobi is the backend the genome-scale numbers above were measured on. It completed the run within the overall limit; one scheduled solve stopped at a limit (the warning in the `big_m=1000` run). ### Values kept different | Parameter | Python | MATLAB | Why | |---|---|---|---| -| `random_sampling.replace_max_bound` | `False` | `True` | With every bound of 1000 replaced by infinity, the random-objective LP becomes unbounded, because the objective can grow without limit through any reaction that is then unconstrained ([Flux sampling convergence and settings](parameter-tuning/studies/sampling-convergence-calibration.md) gives this mechanism; it records no run with `True`, and its run with `False` completed 200 samples). Whether MATLAB's own solver path handles `True` was not tested here. | +| `random_sampling.replace_max_bound` | `False` | `True` | With every bound of 1000 replaced by infinity, the random-objective LP becomes unbounded, because the objective can grow without limit through any reaction that is then unconstrained ([Flux sampling convergence and settings](parameter-tuning/studies/sampling-convergence-calibration.md) gives this mechanism, read from the cobra code; the run with `True` failed with an unbounded status, the sample of the abort was not recorded, and the run with `False` completed 200 samples). Whether MATLAB's own solver path handles `True` was not tested. | | `random_sampling.loopless_good_reactions` | `True` (loopless FVA) | heuristic (exclude FVA ≥ 999) | The two are different techniques. Loopless FVA excludes only reactions that take part in infeasible cycles, and the MATLAB heuristic also excludes reactions that legitimately reach capacity. Porting the loopless technique to MATLAB is a larger project than changing a default. | | `merge_models.match_by` | `'name'` | `'metNames'` | Same semantic field (metabolite display name) under a different field name, which results from the cobrapy and COBRA Toolbox schemas; the field name is not tunable. | | `run_diamond.evalue` | `1e-3` | `1e-3` (implicit) | Both agree at `1e-3`, matching the DIAMOND default. `getDiamond` passes no `-evalue` flag, so it inherits the DIAMOND default and does not follow the explicit `1e-4` of `getBlast`. `run_blast` and `run_diamond` deliberately stay unpaired: each follows the MATLAB call of its own aligner. | @@ -194,7 +193,7 @@ The time-budget parameter of the localisation needs no port: `predictLocalizatio ## Open items -- Genome-scale ACHR sampling convergence has no cheap fix. At genome scale (yeast-GEM, default ACHR settings) the median reaction fails the R-hat>1.1 threshold. Neither follow-up fixes it cheaply: reallocating the same step budget onto a larger `thinning` makes no measurable difference, and `method='chrr'` converges on a small model, but its genome-scale cost makes it impractical as a drop-in fix: a probe of 2 chains × 20 samples on yeast-GEM took 80 min in total, and the share of the rounding step in that time was not measured. [Flux sampling convergence and settings](parameter-tuning/studies/sampling-convergence-calibration.md) describes the investigation, including a possible future unblock (caching the CHRR rounding transform across calls, an engineering change and not a parameter default, so out of scope here). +- Genome-scale ACHR sampling convergence has no cheap fix. At genome scale (yeast-GEM, default ACHR settings) the median reaction fails the R-hat>1.1 threshold. Neither follow-up fixes it cheaply: reallocating the same step budget onto a larger `thinning` makes no measurable difference, and `method='chrr'` converges on a small model, but its genome-scale cost makes it impractical as a drop-in fix: a probe of 2 chains × 20 samples on yeast-GEM took 136 min (8147.9 s) in total, and the share of the rounding step in that time was not measured. [Flux sampling convergence and settings](parameter-tuning/studies/sampling-convergence-calibration.md) describes the investigation, including a possible future unblock (caching the CHRR rounding transform across calls, an engineering change and not a parameter default, so out of scope here). - The `gene_scores_from_expression` scoring constants (`factor=5.0`, `max_score=10.0`, `min_score=-5.0`) have no confirmed literature source on either side; see the INIT section. ## References @@ -228,4 +227,4 @@ The time-budget parameter of the localisation needs no port: `predictLocalizatio > filling in metabolic networks. Bioinformatics 30(17):2529–2531. > -Not listed: the `gene_scores_from_expression` constants (`factor`, `max_score`, `min_score`) were attributed to "Wang et al. 2012" in this documentation and in raven-toolbox's own documentation. The attribution was checked against the source of RAVEN and of raven-toolbox (neither cites a paper) and against the one 2012 Wang metabolic-modelling paper found (mCADRE, a categorical method unrelated to this continuous formula), and could not be confirmed. The scoring constants are treated as RAVEN's own choice with no confirmed literature source until a citation is found. +Not listed: the `gene_scores_from_expression` constants (`factor`, `max_score`, `min_score`). Neither the source of RAVEN nor that of raven-toolbox cites a paper for them. A 2012 paper (Wang et al. 2012, [mCADRE](https://doi.org/10.1186/1752-0509-6-153)) uses categorical scoring and is not the origin. The constants are RAVEN's own choice with no confirmed literature source. diff --git a/scripts/known_names.txt b/scripts/known_names.txt index 5301bff..489b150 100644 --- a/scripts/known_names.txt +++ b/scripts/known_names.txt @@ -435,3 +435,5 @@ nBurnin # sampleCHRR and randomSampling argument allow_met_secr # argument of the ftinit step schedule run_hmmscan # imported by analyze_hmm_cutoffs.py; not in query.py parse_hmmscan_tblout # imported by analyze_hmm_cutoffs.py; not in query.py +cannot_connect # output of connect_blocked_reactions +ml_prep # MATLAB script of the validation study From ae7560df6b488ae6f747dacae22df6de7e8074c7 Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Sun, 1 Nov 2026 10:10:21 +0100 Subject: [PATCH 06/20] Add the parameter-tuning study scripts and point the study pages to them --- docs/parameter-tuning/benchmarks.md | 2 +- .../studies/ftinit-determinism.md | 2 +- .../studies/homology-cutoff-calibration.md | 2 +- .../studies/humangem-validation.md | 2 +- .../studies/init-param-calibration.md | 2 +- .../studies/init-solver-benchmark.md | 2 +- .../studies/kegg-hmm-cutoff-calibration.md | 2 +- scripts/parameter_tuning/benchmarks/README.md | 44 ++ scripts/parameter_tuning/benchmarks/env_py.sh | 10 + .../benchmarks/fseof_sweep.py | 198 ++++++++ .../benchmarks/gapfill_threshold.py | 80 ++++ .../parameter_tuning/benchmarks/mat_small.m | 34 ++ .../benchmarks/sbatch_mat_small.sh | 14 + .../benchmarks/sbatch_small.sh | 10 + .../benchmarks/sbatch_sweep.sh | 11 + scripts/parameter_tuning/benchmarks/small.py | 147 ++++++ .../parameter_tuning/benchmarks/versions.py | 56 +++ scripts/parameter_tuning/homology/README.md | 71 +++ .../parameter_tuning/homology/agreement.py | 105 +++++ .../homology/aligner_settings.py | 189 ++++++++ .../homology/fetch_references.py | 143 ++++++ .../homology/job_align_score.sh | 54 +++ .../homology/job_aligner_settings.sh | 24 + .../parameter_tuning/homology/job_fetch.sh | 25 + .../parameter_tuning/homology/make_tables.py | 192 ++++++++ .../parameter_tuning/homology/run_cached.py | 102 ++++ scripts/parameter_tuning/init/README.md | 81 ++++ scripts/parameter_tuning/init/bigm/bm.py | 335 ++++++++++++++ scripts/parameter_tuning/init/bigm/compare.py | 438 ++++++++++++++++++ scripts/parameter_tuning/init/bigm/job.sh | 39 ++ scripts/parameter_tuning/init/bigm/ml_cell.m | 13 + .../parameter_tuning/init/bigm/ml_common.m | 15 + scripts/parameter_tuning/init/bigm/ml_prep.m | 14 + .../init/bigm/ml_restore_pref.m | 2 + scripts/parameter_tuning/init/bigm/mljob.sh | 38 ++ scripts/parameter_tuning/init/bigm/summ.py | 66 +++ scripts/parameter_tuning/init/det/analyse.py | 146 ++++++ scripts/parameter_tuning/init/det/build.py | 124 +++++ .../parameter_tuning/init/det/det_extra.py | 311 +++++++++++++ scripts/parameter_tuning/init/det/edit.sbatch | 15 + scripts/parameter_tuning/init/det/env.sh | 24 + scripts/parameter_tuning/init/det/job.sbatch | 12 + scripts/parameter_tuning/init/det/prep.py | 49 ++ scripts/parameter_tuning/init/det/prep.sbatch | 9 + .../parameter_tuning/init/det/select_edit.py | 28 ++ .../parameter_tuning/init/det/submit_all.sh | 29 ++ scripts/parameter_tuning/init/det/ties.py | 55 +++ scripts/parameter_tuning/kegg/README.md | 50 ++ .../kegg/analyze_hmm_cutoffs.py | 226 +++++++++ .../kegg/check_true_reactions.py | 79 ++++ .../kegg/compare_kegg_paths.py | 121 +++++ .../parameter_tuning/kegg/fetch_proteomes.py | 189 ++++++++ .../parameter_tuning/kegg/jobs/fetch.sbatch | 25 + .../parameter_tuning/kegg/jobs/rescore.sbatch | 29 ++ scripts/parameter_tuning/kegg/jobs/run.sbatch | 56 +++ .../parameter_tuning/kegg/jobs/timing.sbatch | 23 + .../parameter_tuning/kegg/time_hmmsearch.py | 61 +++ scripts/parameter_tuning/sampling/README.md | 48 ++ .../parameter_tuning/sampling/env_report.py | 86 ++++ scripts/parameter_tuning/sampling/jobs/env.sh | 16 + .../sampling/jobs/posthoc.sbatch | 11 + .../sampling/jobs/rhat.sbatch | 22 + .../parameter_tuning/sampling/jobs/s1.sbatch | 13 + .../sampling/jobs/s1_seeds.sbatch | 29 ++ .../parameter_tuning/sampling/jobs/s7.sbatch | 12 + .../sampling/posthoc_checks.py | 88 ++++ .../sampling/random_objective_bounds.py | 198 ++++++++ .../sampling/single_chain_autocorr.py | 162 +++++++ .../sampling/summarize_chains.py | 88 ++++ 69 files changed, 4991 insertions(+), 7 deletions(-) create mode 100644 scripts/parameter_tuning/benchmarks/README.md create mode 100644 scripts/parameter_tuning/benchmarks/env_py.sh create mode 100644 scripts/parameter_tuning/benchmarks/fseof_sweep.py create mode 100644 scripts/parameter_tuning/benchmarks/gapfill_threshold.py create mode 100644 scripts/parameter_tuning/benchmarks/mat_small.m create mode 100644 scripts/parameter_tuning/benchmarks/sbatch_mat_small.sh create mode 100644 scripts/parameter_tuning/benchmarks/sbatch_small.sh create mode 100644 scripts/parameter_tuning/benchmarks/sbatch_sweep.sh create mode 100644 scripts/parameter_tuning/benchmarks/small.py create mode 100644 scripts/parameter_tuning/benchmarks/versions.py create mode 100644 scripts/parameter_tuning/homology/README.md create mode 100644 scripts/parameter_tuning/homology/agreement.py create mode 100644 scripts/parameter_tuning/homology/aligner_settings.py create mode 100644 scripts/parameter_tuning/homology/fetch_references.py create mode 100644 scripts/parameter_tuning/homology/job_align_score.sh create mode 100644 scripts/parameter_tuning/homology/job_aligner_settings.sh create mode 100644 scripts/parameter_tuning/homology/job_fetch.sh create mode 100644 scripts/parameter_tuning/homology/make_tables.py create mode 100644 scripts/parameter_tuning/homology/run_cached.py create mode 100644 scripts/parameter_tuning/init/README.md create mode 100644 scripts/parameter_tuning/init/bigm/bm.py create mode 100644 scripts/parameter_tuning/init/bigm/compare.py create mode 100644 scripts/parameter_tuning/init/bigm/job.sh create mode 100644 scripts/parameter_tuning/init/bigm/ml_cell.m create mode 100644 scripts/parameter_tuning/init/bigm/ml_common.m create mode 100644 scripts/parameter_tuning/init/bigm/ml_prep.m create mode 100644 scripts/parameter_tuning/init/bigm/ml_restore_pref.m create mode 100755 scripts/parameter_tuning/init/bigm/mljob.sh create mode 100644 scripts/parameter_tuning/init/bigm/summ.py create mode 100644 scripts/parameter_tuning/init/det/analyse.py create mode 100644 scripts/parameter_tuning/init/det/build.py create mode 100644 scripts/parameter_tuning/init/det/det_extra.py create mode 100644 scripts/parameter_tuning/init/det/edit.sbatch create mode 100644 scripts/parameter_tuning/init/det/env.sh create mode 100644 scripts/parameter_tuning/init/det/job.sbatch create mode 100644 scripts/parameter_tuning/init/det/prep.py create mode 100644 scripts/parameter_tuning/init/det/prep.sbatch create mode 100644 scripts/parameter_tuning/init/det/select_edit.py create mode 100644 scripts/parameter_tuning/init/det/submit_all.sh create mode 100644 scripts/parameter_tuning/init/det/ties.py create mode 100644 scripts/parameter_tuning/kegg/README.md create mode 100644 scripts/parameter_tuning/kegg/analyze_hmm_cutoffs.py create mode 100644 scripts/parameter_tuning/kegg/check_true_reactions.py create mode 100644 scripts/parameter_tuning/kegg/compare_kegg_paths.py create mode 100644 scripts/parameter_tuning/kegg/fetch_proteomes.py create mode 100644 scripts/parameter_tuning/kegg/jobs/fetch.sbatch create mode 100644 scripts/parameter_tuning/kegg/jobs/rescore.sbatch create mode 100644 scripts/parameter_tuning/kegg/jobs/run.sbatch create mode 100644 scripts/parameter_tuning/kegg/jobs/timing.sbatch create mode 100644 scripts/parameter_tuning/kegg/time_hmmsearch.py create mode 100644 scripts/parameter_tuning/sampling/README.md create mode 100644 scripts/parameter_tuning/sampling/env_report.py create mode 100644 scripts/parameter_tuning/sampling/jobs/env.sh create mode 100644 scripts/parameter_tuning/sampling/jobs/posthoc.sbatch create mode 100644 scripts/parameter_tuning/sampling/jobs/rhat.sbatch create mode 100644 scripts/parameter_tuning/sampling/jobs/s1.sbatch create mode 100644 scripts/parameter_tuning/sampling/jobs/s1_seeds.sbatch create mode 100644 scripts/parameter_tuning/sampling/jobs/s7.sbatch create mode 100644 scripts/parameter_tuning/sampling/posthoc_checks.py create mode 100644 scripts/parameter_tuning/sampling/random_objective_bounds.py create mode 100644 scripts/parameter_tuning/sampling/single_chain_autocorr.py create mode 100644 scripts/parameter_tuning/sampling/summarize_chains.py diff --git a/docs/parameter-tuning/benchmarks.md b/docs/parameter-tuning/benchmarks.md index 70d8473..b9a3000 100644 --- a/docs/parameter-tuning/benchmarks.md +++ b/docs/parameter-tuning/benchmarks.md @@ -111,7 +111,7 @@ Each benchmark used one model: iJO1366 with one target for `fseof`, e_coli_core ## Source -The scripts are in raven-toolbox under `scripts/parameter_tuning/benchmarks/`: `small.py` (`flux_eps` and `remove_genes`), `fseof_sweep.py` (the sweep of three parameters), `gapfill_threshold.py` (the test of `threshold`) and `mat_small.m` (`removeGenes` in RAVEN). For example: +The scripts are in the raven-docs repository under `scripts/parameter_tuning/benchmarks/`: `small.py` (`flux_eps` and `remove_genes`), `fseof_sweep.py` (the sweep of three parameters), `gapfill_threshold.py` (the test of `threshold`) and `mat_small.m` (`removeGenes` in RAVEN). For example: ```bash python scripts/parameter_tuning/benchmarks/small.py --repeats 3 diff --git a/docs/parameter-tuning/studies/ftinit-determinism.md b/docs/parameter-tuning/studies/ftinit-determinism.md index 24eb43a..a179400 100644 --- a/docs/parameter-tuning/studies/ftinit-determinism.md +++ b/docs/parameter-tuning/studies/ftinit-determinism.md @@ -171,7 +171,7 @@ Wall-clock figures come from jobs that shared nodes with other jobs and are appr ## Reproduction -Requires Gurobi and a Human-GEM checkout. The numbers come from `det/prep.py` (preparation of the template with the essential tasks, about 2000 s), `det/build.py` (one build or repeats of `ftinit` on DLD1 with `big_m=100`, `mip_gap_abs=10` and `time_limit=1800`), `det/select_edit.py` (the 30 removed reactions of the template edit, seed 0), `det/ties.py` (tied-score blocks) and the summaries `det/analyse.py` and `det/det_extra.py`. The scripts are in raven-toolbox under `scripts/parameter_tuning/init/`. The campaign takes about 100 core-hours. The commands run through the Slurm wrappers as in the README of that folder, with `WORK`, `TOOLBOX_ROOT`, `HUMANGEM_ROOT`, `PY`, `DET_SCRIPTS` and `PREP_DIR` set: +Requires Gurobi and a Human-GEM checkout. The numbers come from `det/prep.py` (preparation of the template with the essential tasks, about 2000 s), `det/build.py` (one build or repeats of `ftinit` on DLD1 with `big_m=100`, `mip_gap_abs=10` and `time_limit=1800`), `det/select_edit.py` (the 30 removed reactions of the template edit, seed 0), `det/ties.py` (tied-score blocks) and the summaries `det/analyse.py` and `det/det_extra.py`. The scripts are in the raven-docs repository under `scripts/parameter_tuning/init/`. The campaign takes about 100 core-hours. The commands run through the Slurm wrappers as in the README of that folder, with `WORK`, `TOOLBOX_ROOT`, `HUMANGEM_ROOT`, `PY`, `DET_SCRIPTS` and `PREP_DIR` set: ```bash jid=$(sbatch --parsable -J det-prep -o $WORK/logs/prep.log $DET_SCRIPTS/prep.sbatch) diff --git a/docs/parameter-tuning/studies/homology-cutoff-calibration.md b/docs/parameter-tuning/studies/homology-cutoff-calibration.md index 81dfdac..bf8b2ac 100644 --- a/docs/parameter-tuning/studies/homology-cutoff-calibration.md +++ b/docs/parameter-tuning/studies/homology-cutoff-calibration.md @@ -119,7 +119,7 @@ Scoring against curated genome-scale metabolic models (GEMs) of non-model fungi ## Reproducing -The scripts are in [`scripts/parameter_tuning/homology/`](https://github.com/CompMetEng/raven-toolbox/tree/develop/scripts/parameter_tuning/homology) of raven-toolbox. The driver `homology_cutoff_kegg.py` takes `--out` before the subcommand: +The scripts are in [`scripts/parameter_tuning/homology/`](https://github.com/CompMetEng/raven-docs/tree/main/scripts/parameter_tuning/homology) of the raven-docs repository. The driver `homology_cutoff_kegg.py` takes `--out` before the subcommand: ```bash python homology_cutoff_kegg.py --out work fetch diff --git a/docs/parameter-tuning/studies/humangem-validation.md b/docs/parameter-tuning/studies/humangem-validation.md index 7ae86a3..b88a6ec 100644 --- a/docs/parameter-tuning/studies/humangem-validation.md +++ b/docs/parameter-tuning/studies/humangem-validation.md @@ -119,7 +119,7 @@ Each cell line, variant and `big_m` value ran once, without seed repeats, so the ## Reproduction -The numbers come from `bigm/bm.py prep`, `preptask` and `val` (raven-toolbox side), `bigm/ml_prep.m` and `bigm/ml_cell.m` (RAVEN side) and `bigm/compare.py`, which prints every table of the study. The scripts are in raven-toolbox under `scripts/parameter_tuning/init/`. The commands run through the Slurm wrappers `job.sh` and `mljob.sh` as in the README of that folder, with `WORK`, `TOOLBOX_ROOT`, `RAVEN_ROOT`, `HUMANGEM_ROOT` and `PY` set: +The numbers come from `bigm/bm.py prep`, `preptask` and `val` (raven-toolbox side), `bigm/ml_prep.m` and `bigm/ml_cell.m` (RAVEN side) and `bigm/compare.py`, which prints every table of the study. The scripts are in the raven-docs repository under `scripts/parameter_tuning/init/`. The commands run through the Slurm wrappers `job.sh` and `mljob.sh` as in the README of that folder, with `WORK`, `TOOLBOX_ROOT`, `RAVEN_ROOT`, `HUMANGEM_ROOT` and `PY` set: ```bash job.sh prep diff --git a/docs/parameter-tuning/studies/init-param-calibration.md b/docs/parameter-tuning/studies/init-param-calibration.md index 14eb25d..b04f1bc 100644 --- a/docs/parameter-tuning/studies/init-param-calibration.md +++ b/docs/parameter-tuning/studies/init-param-calibration.md @@ -53,7 +53,7 @@ Each table row is one run unless the row states repeats; this set-up has no runs ### Source of the numbers -The calibration tables come from `bigm/bm.py` with the subcommands `step0`, `scale` and `full`, summarised by `bigm/summ.py` and `bigm/compare.py` (prepared model `prep_notask.pkl`). The scripts are in raven-toolbox under `scripts/parameter_tuning/init/`. `step0` runs a grid of 16 solves: `mip_gap` 0.0002 to 0.1, `big_m` 10 to 1000 and `force_on` 0.02 to 0.5, at `big_m=1000` for the `mip_gap` and `force_on` sweeps. `scale` runs the unscaled preparation rescaled with `max_stoich_diff` 10, 25, 50 and 100 or left unscaled, at `big_m` 1000 and 100. `full` runs one `ftinit` call per setting, with `--gap`, `--big-m`, `--scale` and `--noabs` (`mip_gap_abs=None`) options. +The calibration tables come from `bigm/bm.py` with the subcommands `step0`, `scale` and `full`, summarised by `bigm/summ.py` and `bigm/compare.py` (prepared model `prep_notask.pkl`). The scripts are in the raven-docs repository under `scripts/parameter_tuning/init/`. `step0` runs a grid of 16 solves: `mip_gap` 0.0002 to 0.1, `big_m` 10 to 1000 and `force_on` 0.02 to 0.5, at `big_m=1000` for the `mip_gap` and `force_on` sweeps. `scale` runs the unscaled preparation rescaled with `max_stoich_diff` 10, 25, 50 and 100 or left unscaled, at `big_m` 1000 and 100. `full` runs one `ftinit` call per setting, with `--gap`, `--big-m`, `--scale` and `--noabs` (`mip_gap_abs=None`) options. The first-step, scaling and full-pipeline tables are computed from the outputs by `compare.py`. The robustness tables (degraded expression input, lever runs and the 90 % dropout run) come from `scripts/analyze_init_robustness.py`. The script exists at the pinned commit; the state of the script and the inputs that produced the tables are not recoverable. diff --git a/docs/parameter-tuning/studies/init-solver-benchmark.md b/docs/parameter-tuning/studies/init-solver-benchmark.md index 49fcd1f..823729b 100644 --- a/docs/parameter-tuning/studies/init-solver-benchmark.md +++ b/docs/parameter-tuning/studies/init-solver-benchmark.md @@ -44,7 +44,7 @@ Each setting ran once, on one model and one cell line. In a different set-up (th ## Source of the numbers and reproduction -The table comes from `bigm/bm.py solver` and the overlap figures from `bigm/compare.py`. The scripts are in raven-toolbox under `scripts/parameter_tuning/init/`. The CI test for toy networks is separate: +The table comes from `bigm/bm.py solver` and the overlap figures from `bigm/compare.py`. The scripts are in the raven-docs repository under `scripts/parameter_tuning/init/`. The CI test for toy networks is separate: ```bash # CI parameterised tests (seconds, runs always): diff --git a/docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md b/docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md index 9a0d865..7058f4c 100644 --- a/docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md +++ b/docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md @@ -219,7 +219,7 @@ The model-assembly flags decide which reactions a draft keeps, and have the same ## Reproduction -The scripts are in [`scripts/parameter_tuning/kegg/`](https://github.com/CompMetEng/raven-toolbox/tree/develop/scripts/parameter_tuning/kegg) of raven-toolbox. They search with `hmmsearch` from the toolbox binary bundle (`resolve_binary("hmmsearch")`) and take the published kegg118 tables and HMM libraries through `ensure_kegg_data()` and `ensure_kegg_hmm_library("eukaryotes")` or `("prokaryotes")`. +The scripts are in [`scripts/parameter_tuning/kegg/`](https://github.com/CompMetEng/raven-docs/tree/main/scripts/parameter_tuning/kegg) of the raven-docs repository. They search with `hmmsearch` from the toolbox binary bundle (`resolve_binary("hmmsearch")`) and take the published kegg118 tables and HMM libraries through `ensure_kegg_data()` and `ensure_kegg_hmm_library("eukaryotes")` or `("prokaryotes")`. | Script | Produces | |---|---| diff --git a/scripts/parameter_tuning/benchmarks/README.md b/scripts/parameter_tuning/benchmarks/README.md new file mode 100644 index 0000000..c66cd15 --- /dev/null +++ b/scripts/parameter_tuning/benchmarks/README.md @@ -0,0 +1,44 @@ +# Parameter benchmarks: fseof, remove_genes, connect_blocked_reactions + +Scripts for the benchmarks page of the parameter-tuning documentation: `fseof` at three values of `flux_eps` and a sweep of `n_steps`, `max_fraction` and `correlation_threshold` on iJO1366 (target `EX_succ_e`), `remove_genes` after deletion of `b1779` from e_coli_core, and `connect_blocked_reactions` `threshold` on a synthetic network. + +## Inputs and environment + +The models are the `iJO1366.xml.gz` and `textbook.xml.gz` (e_coli_core) files bundled with cobra 0.32.1. The SHA-256 of both is written into the output JSON files. Python runs need cobra, optlang and gurobipy; the MATLAB script needs RAVEN and Gurobi. + +| Variable | Meaning | +| --- | --- | +| `STUDY` | this folder | +| `RAVEN_TOOLBOX` | raven-toolbox checkout (`94f3940`); its `src` goes on `PYTHONPATH` | +| `BENCH_PY` | Python interpreter with cobra 0.32.1 and gurobipy | +| `R` | RAVEN checkout (`2d0630e`) | +| `GUROBI_HOME` | Gurobi installation that supplies the licence | +| `COBRA_DATA` | `cobra/data` folder of the Python environment | + +The module names in the wrappers are site-specific; other systems need an equivalent environment. + +## Commands + +```bash +sbatch --export=ALL -A -o small.log $STUDY/sbatch_small.sh +sbatch --export=ALL -A -o sweep.log $STUDY/sbatch_sweep.sh +sbatch --export=ALL -A -o mat_small.log $STUDY/sbatch_mat_small.sh +``` + +The wrappers run the following commands, with three repeats per configuration. `mat_small.m` reads the unzipped e_coli_core file from `$MODEL`. + +```bash +python small.py --out results/small.json --repeats 3 +python fseof_sweep.py --outdir results --repeats 3 +python gapfill_threshold.py --out results/gapfill_threshold.json +``` + +## Outputs + +| File | Content | +| --- | --- | +| `small.py` to `results/small.json` | `remove_genes` with `blocked_reactions` set to `constrain` (default), `remove` and `keep` and the unmodified model; `fseof` at `flux_eps` 1e-8, 1e-7 and 1e-6 with software versions and input checksums. | +| `fseof_sweep.py` to `results/sweep.json`, `results/sweep_summary.tsv`, `results/sweep_targets.tsv` | One row per configuration and repeat: target counts, targets added and dropped relative to the default run, and noise targets (flux standard deviation below 1e-6). `sweep_summary.tsv` is kept in this folder; the other two are regenerated by the run. | +| `gapfill_threshold.py` to `results/gapfill_threshold.json` | `connect_blocked_reactions` at `threshold` 0.01, 0.1, 0.5 and 1.0: draft with an import (at most 0.5), reaction A to B, dead-end B, template exchange `EX_B`. | +| `mat_small.m` | MATLAB `removeGenes` on e_coli_core with and without `removeBlockedRxns`, printed to the log. | +| `versions.py`, `env_py.sh` | Software versions written into the JSON files; environment set-up of the wrappers. | diff --git a/scripts/parameter_tuning/benchmarks/env_py.sh b/scripts/parameter_tuning/benchmarks/env_py.sh new file mode 100644 index 0000000..b7d5708 --- /dev/null +++ b/scripts/parameter_tuning/benchmarks/env_py.sh @@ -0,0 +1,10 @@ +# Environment for the Python benchmark scripts (sourced by the sbatch files). +# Required before sourcing: RAVEN_TOOLBOX (checkout of raven-toolbox at 94f3940) +# and BENCH_PY (python of the venv with cobra 0.32.1 and gurobipy), GUROBI_HOME (Gurobi 13.0.1 installation). +module purge +module load SciPy-bundle/2026.05-gfbf-2026.1 +export GUROBI_HOME=${GUROBI_HOME:?set GUROBI_HOME} +export GRB_LICENSE_FILE=$GUROBI_HOME/gurobi.lic +export LD_LIBRARY_PATH=$GUROBI_HOME/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH} +export PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 +export PYTHONPATH=$RAVEN_TOOLBOX/src diff --git a/scripts/parameter_tuning/benchmarks/fseof_sweep.py b/scripts/parameter_tuning/benchmarks/fseof_sweep.py new file mode 100644 index 0000000..bef18bc --- /dev/null +++ b/scripts/parameter_tuning/benchmarks/fseof_sweep.py @@ -0,0 +1,198 @@ +"""fseof parameter sweep on iJO1366. + +Runs ``raven_toolbox.analysis.fseof`` on iJO1366 (cobrapy ``iJO1366.xml.gz``, its own +biomass objective ``BIOMASS_Ec_iJO1366_core_53p95M``) with target ``EX_succ_e`` at the +defaults (n_steps 10, max_fraction 0.9, correlation_threshold 0.9, flux_eps 1e-6, +equality False) and with one parameter changed at a time: + +* n_steps 5, 20 +* max_fraction 0.5, 0.99 +* correlation_threshold 0.7, 0.95 + +Each configuration runs ``--repeats`` times, each on a freshly read model, with Gurobi. + +For each run it reports the counts per target type and compares the target set with the +first default run: targets added and dropped, by type. Two operational criteria mark +targets that may be spurious: + +* ``added``: a target found at the swept value and not with the defaults. +* ``noise``: a target whose absolute flux has a standard deviation (ddof 0) across the + scan below 1e-6. + +For each added target it also gives the correlation and standard deviation of the +same reaction's absolute flux in the first default scan. + +Outputs in ``--outdir``: sweep_summary.tsv (one row per run), sweep_targets.tsv +(targets of the first run of each configuration, with slope, r, std, in-default flag), +sweep.json (summary, versions, input checksum). + +Command line (see sbatch_sweep.sh): + $BENCH_PY fseof_sweep.py --outdir $OUT --repeats 3 +with PYTHONPATH=/src (env_py.sh). +""" + +import argparse +import gzip +import hashlib +import json +import os +import shutil +import sys +import time + +import cobra +import numpy as np +import pandas as pd +from cobra.io import read_sbml_model +from scipy.stats import linregress + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from versions import versions # noqa: E402 + +from raven_toolbox.analysis import fseof # noqa: E402 + +NOISE_STD = 1e-6 +TARGET = "EX_succ_e" +DEFAULT = {"n_steps": 10, "max_fraction": 0.9, "correlation_threshold": 0.9} +CONFIGS = [ + ("default", None, None), + ("n_steps", "n_steps", 5), + ("n_steps", "n_steps", 20), + ("max_fraction", "max_fraction", 0.5), + ("max_fraction", "max_fraction", 0.99), + ("correlation_threshold", "correlation_threshold", 0.7), + ("correlation_threshold", "correlation_threshold", 0.95), +] + +ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) +ap.add_argument("--outdir", required=True) +ap.add_argument("--repeats", type=int, default=3) +ap.add_argument("--tmp", default=os.environ.get("TMPDIR", "/tmp")) +args = ap.parse_args() +os.makedirs(args.outdir, exist_ok=True) + +src = os.path.join(os.path.dirname(cobra.__file__), "data", "iJO1366.xml.gz") +with open(src, "rb") as f: + sha = hashlib.sha256(f.read()).hexdigest() +sbml = os.path.join(args.tmp, "iJO1366.xml") +with gzip.open(src) as f, open(sbml, "wb") as o: + shutil.copyfileobj(f, o) + + +def annotate(r): + t = r.targets.copy() + scan = r.scan.loc[t.reaction, r.enforced].abs() + t["std_abs"] = scan.std(axis=1, ddof=0).to_numpy() + t["kdko"] = t.target_type.isin(["knockdown", "knockout"]) + return t + + +summary, target_rows, runs = [], [], {} +default_t = default_scan = None +for rep in range(1, args.repeats + 1): + for _label, key, value in CONFIGS: + m = read_sbml_model(sbml) + m.solver = "gurobi" + kw = dict(DEFAULT) + if key: + kw[key] = value + t0 = time.time() + r = fseof(m, TARGET, **kw) + secs = time.time() - t0 + t = annotate(r) + name = "default" if key is None else f"{key}={value}" + if key is None and rep == 1: + default_t, default_scan, default_enf = t, r.scan, np.asarray(r.enforced) + dset = dict(zip(default_t.reaction, default_t.target_type, strict=False)) + cur = dict(zip(t.reaction, t.target_type, strict=False)) + added = {k: v for k, v in cur.items() if k not in dset} + dropped = {k: v for k, v in dset.items() if k not in cur} + relabelled = {k: (dset[k], v) for k, v in cur.items() if k in dset and dset[k] != v} + noise = set(t.reaction[t.std_abs < NOISE_STD]) + c = t.target_type.value_counts() + row = { + "config": name, + "parameter": key or "default", + "value": value if key else "", + "rep": rep, + "n_levels": len(r.enforced), + "enforced_last": r.enforced[-1], + "amplify": int(c.get("amplify", 0)), + "knockdown": int(c.get("knockdown", 0)), + "knockout": int(c.get("knockout", 0)), + "added_amplify": sum(v == "amplify" for v in added.values()), + "added_kdko": sum(v != "amplify" for v in added.values()), + "dropped_amplify": sum(v == "amplify" for v in dropped.values()), + "dropped_kdko": sum(v != "amplify" for v in dropped.values()), + "relabelled": len(relabelled), + "noise_targets": len(noise), + "noise_amplify": int(((t.std_abs < NOISE_STD) & (t.target_type == "amplify")).sum()), + "added_noise": len(noise & set(added)), + "seconds": round(secs, 1), + } + row["kdko"] = row["knockdown"] + row["knockout"] + summary.append(row) + print(json.dumps(row)) + if rep == 1: + runs[name] = {"added": added, "dropped": dropped, "relabelled": relabelled} + for _, x in t.iterrows(): + target_rows.append( + { + "config": name, + "reaction": x.reaction, + "target_type": x.target_type, + "slope": x.slope, + "correlation": x.correlation, + "std_abs": x.std_abs, + "in_default": x.reaction in dset, + } + ) + for rid, ttype in added.items(): + a = default_scan.loc[rid, list(default_enf)].abs().to_numpy(dtype=float) + ok = a.std() > 0 + runs[name].setdefault("added_detail", []).append( + { + "reaction": rid, + "type": ttype, + "r_here": float(t.loc[t.reaction == rid, "correlation"].iloc[0]), + "std_here": float(t.loc[t.reaction == rid, "std_abs"].iloc[0]), + "r_default_scan": float(linregress(default_enf, a).rvalue) if ok else None, + "std_default_scan": float(a.std()), + } + ) + for k in ("added_detail",): + for d in runs[name].get(k, []): + print(" added", json.dumps(d)) + for rid, ttype in dropped.items(): + print(" dropped", rid, ttype) + +pd.DataFrame(summary).to_csv(os.path.join(args.outdir, "sweep_summary.tsv"), sep="\t", index=False) +pd.DataFrame(target_rows).to_csv( + os.path.join(args.outdir, "sweep_targets.tsv"), sep="\t", index=False, float_format="%.6g" +) +stable = {} +for row in summary: + stable.setdefault(row["config"], set()).add( + ( + row["amplify"], + row["knockdown"], + row["knockout"], + row["added_amplify"], + row["added_kdko"], + row["dropped_amplify"], + row["dropped_kdko"], + ) + ) +print("identical counts across repeats:", {k: len(v) == 1 for k, v in stable.items()}) +out = { + "input": {"path": src, "sha256": sha}, + "target": TARGET, + "defaults": DEFAULT, + "noise_std": NOISE_STD, + "summary": summary, + "rep1": runs, + "versions": versions(m), +} +with open(os.path.join(args.outdir, "sweep.json"), "w") as f: + json.dump(out, f, indent=1, default=str) +print("versions", json.dumps(out["versions"])) diff --git a/scripts/parameter_tuning/benchmarks/gapfill_threshold.py b/scripts/parameter_tuning/benchmarks/gapfill_threshold.py new file mode 100644 index 0000000..2b9a8f0 --- /dev/null +++ b/scripts/parameter_tuning/benchmarks/gapfill_threshold.py @@ -0,0 +1,80 @@ +"""connect_blocked_reactions threshold on a synthetic supply-limited gap. + +The synthetic model is: + +* draft: metabolites A[c] and B[c]; EX_A (A <=>, lower bound -0.5: uptake of at most + 0.5); R_AB (A -> B, irreversible, bounds 0..1000). B is a dead end, so R_AB is blocked. +* template: B[c] and EX_B (B ->, bounds 0..1000). + +``connect_blocked_reactions(draft, template, threshold=t)`` is run for t in 0.01, 0.1, 0.5, 1.0 (1.0 is the default) and the other +arguments at their defaults (min_flux 0.1, penalty 1.0, allow_net_production False). +The result lists the reactions added to fill the gap at each threshold. + +Solver: Gurobi. Output: log and the JSON file given by ``--out``. + +Command line (see sbatch_sweep.sh): + $BENCH_PY gapfill_threshold.py --out $OUT/gapfill_threshold.json +""" + +import argparse +import json +import os +import sys + +import cobra + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from versions import versions # noqa: E402 + +from raven_toolbox.gapfilling import connect_blocked_reactions # noqa: E402 + +ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) +ap.add_argument("--out", required=True) +args = ap.parse_args() + + +def met(mid, name): + return cobra.Metabolite(mid, name=name, compartment="c") + + +def draft(): + m = cobra.Model("draft") + m.solver = "gurobi" + a, b = met("A_c", "A"), met("B_c", "B") + ex = cobra.Reaction("EX_A", lower_bound=-0.5, upper_bound=1000) + ex.add_metabolites({a: -1}) + rab = cobra.Reaction("R_AB", lower_bound=0, upper_bound=1000) + rab.add_metabolites({a: -1, b: 1}) + m.add_reactions([ex, rab]) + return m + + +def template(): + t = cobra.Model("template") + t.solver = "gurobi" + exb = cobra.Reaction("EX_B", lower_bound=0, upper_bound=1000) + exb.add_metabolites({met("B_c", "B"): -1}) + t.add_reactions([exb]) + return t + + +rows = [] +for thr in (0.01, 0.1, 0.5, 1.0): + d = draft() + r = connect_blocked_reactions(d, template(), threshold=thr) + with r.model as fm: + fm.objective = "R_AB" + max_ab = fm.slim_optimize() + row = { + "threshold": thr, + "added": r.added_reactions, + "newly_connected": r.newly_connected, + "cannot_connect": r.cannot_connect, + "R_AB_max_after": max_ab, + } + print(json.dumps(row)) + rows.append(row) +out = {"rows": rows, "versions": versions(d)} +print("versions", json.dumps(out["versions"])) +with open(args.out, "w") as f: + json.dump(out, f, indent=1) diff --git a/scripts/parameter_tuning/benchmarks/mat_small.m b/scripts/parameter_tuning/benchmarks/mat_small.m new file mode 100644 index 0000000..978434e --- /dev/null +++ b/scripts/parameter_tuning/benchmarks/mat_small.m @@ -0,0 +1,34 @@ +% removeGenes on e_coli_core with RAVEN. +% +% Environment variables: R = RAVEN checkout (commit 2d0630e), MODEL = e_coli_core SBML +% (cobrapy textbook.xml.gz, gunzipped). Puts R on the path, sets the RAVEN/ravenPath +% preference to R and asserts findRAVENroot()==R, sets the solver to Gurobi, imports +% the model, solves it, then removeGenes(m,{'b1779'},'removeBlockedRxns',rb) for rb +% false and true, printing reactions left, growth and GAPD bounds. The previous +% ravenPath preference is restored at the end. +% Command line (see sbatch_mat_small.sh): +% matlab -nodisplay -nosplash -batch "run('/mat_small.m')" +R = getenv('R'); +hadPref = ispref('RAVEN','ravenPath'); +if hadPref, oldPref = getpref('RAVEN','ravenPath'); end +restoredefaultpath; addpath(genpath(R)); addpath(fullfile(getenv('EBROOTGUROBI'),'matlab')); +setpref('RAVEN','ravenPath',R); +root = findRAVENroot(); +assert(isequal(strip(root,'right',filesep), strip(R,'right',filesep)), 'root mismatch %s', root); +setRavenSolver('gurobi'); +fprintf('solver %s\n', getpref('RAVEN','solver','none')); +fprintf('MATLAB %s\n', version); +try, gv = gurobi_version(); disp(gv); catch, end +[~, c] = system(['git -C ' R ' rev-parse HEAD']); fprintf('RAVEN commit %s', c); +m = importModel(getenv('MODEL')); +fprintf('e_coli_core: %d rxns obj=%s\n', numel(m.rxns), strjoin(m.rxns(m.c~=0))); +sol = solveLP(m); fprintf('unmodified growth %.4f (printed as -sol.f)\n', -sol.f); +fprintf('unmodified growth %.4f (biomass flux)\n', sol.x(m.c~=0)); +for rb = [false true] + r = removeGenes(m, {'b1779'}, 'removeBlockedRxns', rb); + sol = solveLP(r); + i = find(strcmp(r.rxns,'GAPD')); + if isempty(i), bnd='removed'; else, bnd=sprintf('[%g %g]', r.lb(i), r.ub(i)); end + fprintf('removeGenes removeBlockedRxns=%d: %d rxns, growth %.4f, GAPD %s\n', rb, numel(r.rxns), -sol.f, bnd); +end +if hadPref, setpref('RAVEN','ravenPath',oldPref); else, rmpref('RAVEN','ravenPath'); end diff --git a/scripts/parameter_tuning/benchmarks/sbatch_mat_small.sh b/scripts/parameter_tuning/benchmarks/sbatch_mat_small.sh new file mode 100644 index 0000000..112c771 --- /dev/null +++ b/scripts/parameter_tuning/benchmarks/sbatch_mat_small.sh @@ -0,0 +1,14 @@ +#!/bin/bash +#SBATCH -c 2 +#SBATCH --mem=8G +#SBATCH -t 00:30:00 +#SBATCH -J bench_mat +# Submit with STUDY set to this folder (sbatch --export=ALL -A ). +# Needs R (RAVEN checkout) and COBRA_DATA (cobra/data folder with textbook.xml.gz). +module purge +module load MATLAB/2024b Gurobi/13.0.1-GCCcore-14.3.0 +echo "EBROOTGUROBI=$EBROOTGUROBI" +export MATLAB_PREFDIR=$TMPDIR/matprefs; mkdir -p $MATLAB_PREFDIR +export MODEL=$TMPDIR/e_coli_core.xml R +gunzip -c $COBRA_DATA/textbook.xml.gz > $MODEL +matlab -nodisplay -nosplash -batch "run('$STUDY/mat_small.m')" diff --git a/scripts/parameter_tuning/benchmarks/sbatch_small.sh b/scripts/parameter_tuning/benchmarks/sbatch_small.sh new file mode 100644 index 0000000..692e8c3 --- /dev/null +++ b/scripts/parameter_tuning/benchmarks/sbatch_small.sh @@ -0,0 +1,10 @@ +#!/bin/bash +#SBATCH -c 2 +#SBATCH --mem=8G +#SBATCH -t 01:00:00 +#SBATCH -J bench_small +# Submit with STUDY set to this folder (sbatch --export=ALL -A ). +# Needs RAVEN_TOOLBOX and BENCH_PY in the environment. +S=${STUDY:?set STUDY} +source $S/env_py.sh +$BENCH_PY $S/small.py --out $STUDY/results/small.json --repeats 3 diff --git a/scripts/parameter_tuning/benchmarks/sbatch_sweep.sh b/scripts/parameter_tuning/benchmarks/sbatch_sweep.sh new file mode 100644 index 0000000..dc08f12 --- /dev/null +++ b/scripts/parameter_tuning/benchmarks/sbatch_sweep.sh @@ -0,0 +1,11 @@ +#!/bin/bash +#SBATCH -c 2 +#SBATCH --mem=8G +#SBATCH -t 01:00:00 +#SBATCH -J bench_sweep +# Submit with STUDY set to this folder (sbatch --export=ALL -A ). +# Needs RAVEN_TOOLBOX and BENCH_PY in the environment. +S=${STUDY:?set STUDY} +source $S/env_py.sh +$BENCH_PY $S/fseof_sweep.py --outdir $STUDY/results --repeats 3 +$BENCH_PY $S/gapfill_threshold.py --out $STUDY/results/gapfill_threshold.json diff --git a/scripts/parameter_tuning/benchmarks/small.py b/scripts/parameter_tuning/benchmarks/small.py new file mode 100644 index 0000000..2527705 --- /dev/null +++ b/scripts/parameter_tuning/benchmarks/small.py @@ -0,0 +1,147 @@ +"""remove_genes on e_coli_core and fseof flux_eps on iJO1366. + +1. e_coli_core (cobrapy ``textbook.xml.gz``): delete gene b1779 (the only gene of GAPD) + with ``remove_genes`` for ``blocked_reactions`` in "constrain", "remove", "keep" and + with no keyword (default); print reactions left, ``slim_optimize()`` and GAPD bounds; + then the unmodified model. +2. iJO1366 (cobrapy ``iJO1366.xml.gz``, its own biomass objective): ``fseof`` with target + ``EX_succ_e`` at ``flux_eps`` 1e-8, 1e-7, 1e-6, other arguments at their defaults; + print the number of targets per ``target_type``. ``--repeats`` runs per value + (each on a freshly read model). + +Solver: Gurobi through optlang (``model.solver = "gurobi"``). + +Inputs: the SBML files in the ``cobra/data`` folder of the running cobra, unzipped to +``$TMPDIR``. Output: log on stdout and the JSON file given by ``--out``. + +Command line (see sbatch_small.sh): + $BENCH_PY small.py --out $OUT/small.json --repeats 3 +with PYTHONPATH=/src (env_py.sh). +""" + +import argparse +import gzip +import hashlib +import json +import os +import shutil +import sys +import time + +import cobra +from cobra.io import read_sbml_model + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from versions import versions # noqa: E402 + +from raven_toolbox.analysis import fseof # noqa: E402 +from raven_toolbox.manipulation import remove_genes # noqa: E402 + +ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) +ap.add_argument("--out", required=True, help="JSON result file") +ap.add_argument("--repeats", type=int, default=1, help="fseof runs per flux_eps value") +ap.add_argument("--tmp", default=os.environ.get("TMPDIR", "/tmp"), help="scratch folder") +args = ap.parse_args() + +data = os.path.join(os.path.dirname(cobra.__file__), "data") +tmp = args.tmp +res = {"inputs": {}, "remove_genes": [], "fseof": []} +for n in ("textbook", "iJO1366"): + src = f"{data}/{n}.xml.gz" + with open(src, "rb") as f: + res["inputs"][n] = {"path": src, "sha256": hashlib.sha256(f.read()).hexdigest()} + with gzip.open(src) as f, open(f"{tmp}/{n}.xml", "wb") as o: + shutil.copyfileobj(f, o) +shutil.copy(f"{tmp}/textbook.xml", f"{tmp}/e_coli_core.xml") + +print("=== remove_genes e_coli_core") +for mode in ("constrain", "remove", "keep", None): + m = read_sbml_model(f"{tmp}/e_coli_core.xml") + m.solver = "gurobi" + if mode is None: + remove_genes(m, "b1779") + mode = "default" + else: + try: + remove_genes(m, "b1779", blocked_reactions=mode) + except Exception as e: # noqa: BLE001 + print(mode, "ERR", e) + continue + g = m.slim_optimize() + gap = m.reactions.GAPD if "GAPD" in m.reactions else None + print( + mode, + "reactions", + len(m.reactions), + "growth %.4f" % g, # noqa: UP031 + "GAPD bounds", + gap.bounds if gap else "removed", + ) + res["remove_genes"].append( + { + "call": mode, + "reactions": len(m.reactions), + "growth": g, + "GAPD": list(gap.bounds) if gap else "removed", + "GAPD_gpr": gap.gene_reaction_rule if gap else None, + } + ) +m = read_sbml_model(f"{tmp}/e_coli_core.xml") +m.solver = "gurobi" +g = m.slim_optimize() +print("unmodified", len(m.reactions), "growth %.4f" % g) # noqa: UP031 +res["remove_genes"].append({"call": "unmodified", "reactions": len(m.reactions), "growth": g}) + +print("=== fseof iJO1366") +for rep in range(1, args.repeats + 1): + m = read_sbml_model(f"{tmp}/iJO1366.xml") + m.solver = "gurobi" + growth = m.slim_optimize() + print( + "rep", + rep, + "reactions", + len(m.reactions), + "objective", + m.objective.expression, + "growth", + growth, + ) + for eps in (1e-8, 1e-7, 1e-6): + t0 = time.time() + r = fseof(m, target_rxn="EX_succ_e", flux_eps=eps) + t = r.targets + counts = t.target_type.value_counts().to_dict() + print( + "rep", + rep, + "EX_succ_e flux_eps", + eps, + "n_targets", + len(t), + counts, + "enforced", + r.enforced[0], + r.enforced[-1], + "%.0fs" % (time.time() - t0), + ) + res["fseof"].append( + { + "rep": rep, + "flux_eps": eps, + "n_targets": len(t), + "counts": {k: int(v) for k, v in counts.items()}, + "enforced_first": r.enforced[0], + "enforced_last": r.enforced[-1], + "n_levels": len(r.enforced), + "targets": sorted(zip(t.reaction, t.target_type, strict=False)), + "growth_unconstrained": growth, + } + ) + +res["versions"] = versions(m) +print("versions", json.dumps(res["versions"])) +same = len({json.dumps(x["targets"]) for x in res["fseof"]}) == 1 +print("all fseof runs give the same target set and labels:", same) +with open(args.out, "w") as f: + json.dump(res, f, indent=1) diff --git a/scripts/parameter_tuning/benchmarks/versions.py b/scripts/parameter_tuning/benchmarks/versions.py new file mode 100644 index 0000000..f72c5db --- /dev/null +++ b/scripts/parameter_tuning/benchmarks/versions.py @@ -0,0 +1,56 @@ +"""Software versions of a benchmark run. + +Imported by small.py, fseof_sweep.py and gapfill_threshold.py. ``versions(model)`` +returns a dict with the Python, cobra, optlang, gurobipy and Gurobi library versions, +the optlang interface of ``model``, the raven-toolbox path and its git commit, and +the libgurobi shared objects mapped into the process (read from /proc/self/maps after +a solve, so it names the library that actually solved). +""" + +import os +import platform +import subprocess + + +def versions(model=None): + import cobra + import optlang + + import raven_toolbox + + out = { + "python": platform.python_version(), + "cobra": cobra.__version__, + "optlang": optlang.__version__, + "raven_toolbox_file": raven_toolbox.__file__, + } + src = os.path.dirname(os.path.dirname(os.path.dirname(raven_toolbox.__file__))) + try: + out["raven_toolbox_commit"] = subprocess.run( + ["git", "-C", src, "rev-parse", "HEAD"], capture_output=True, text=True + ).stdout.strip() + out["raven_toolbox_dirty"] = bool( + subprocess.run( + ["git", "-C", src, "status", "--porcelain", "--untracked-files=no"], + capture_output=True, + text=True, + ).stdout.strip() + ) + except OSError: + pass + try: + import gurobipy + + out["gurobipy"] = gurobipy.__version__ + out["gurobi_library"] = ".".join(map(str, gurobipy.gurobi.version())) + except Exception as e: # noqa: BLE001 + out["gurobipy"] = f"unavailable: {e}" + if model is not None: + out["solver_interface"] = model.solver.interface.__name__ + try: + with open("/proc/self/maps") as f: + libs = sorted({line.split()[-1] for line in f if "libgurobi" in line}) + out["libgurobi_mapped"] = libs + except OSError: + pass + return out diff --git a/scripts/parameter_tuning/homology/README.md b/scripts/parameter_tuning/homology/README.md new file mode 100644 index 0000000..150cf62 --- /dev/null +++ b/scripts/parameter_tuning/homology/README.md @@ -0,0 +1,71 @@ +# Homology cut-off calibration + +Scripts for the homology cut-off page of the parameter-tuning documentation. *S. cerevisiae* (`sce`) is the template against *K. lactis* (`kla`), *Y. lipolytica* (`yli`), *A. nidulans* (`ani`) and *E. coli* (`eco`). The gene-to-KO assignment is scored as F0.5 against the KEGG release 118 table, current KEGG and OMA orthologues over sequence identity, alignment length and E-value. BLAST and DIAMOND are compared, and the BLAST E-value and thread settings are tested on *H. polymorpha* against *S. cerevisiae*. The driver is `scripts/homology_cutoff_kegg.py` of raven-toolbox. + +## Inputs and environment + +| Input | Source | +| --- | --- | +| UniProt proteomes | UP000002311, UP000001300, UP000000560, UP000000625, UP000000598 (release 2026_03) | +| KEGG release 118 | `kegg118_core.tar.gz` of [CompMetEng/toolbox-assets](https://github.com/CompMetEng/toolbox-assets/releases/tag/kegg118), SHA-256 `520ba8bb09d6ca479ca1ab84ea70bf7c1d7f82d8f6406b717cee293b7c8609ca` | +| current KEGG | REST API (`link/ko`, restricted to KOs in `link/reaction/ko`, and all KOs) | +| OMA | `All.May2026`, `https://omabrowser.org/api/pairs//559292/` | +| SGD | `orf_trans_all.fasta.gz` of `sgd-archive.yeastgenome.org/sequence/S288C_reference/orf_protein` | +| *H. polymorpha* | `data/genomes/hanpo.faa` of the hanpo-GEM repository (5177 sequences) | + +`job_fetch.sh` downloads the inputs into `homology-inputs.tar.zst` (proteomes, UniProt cache, reference tables); `job_align_score.sh` writes the hit tables to `homology-hits.tar.zst`. Download manifests with URLs, sizes and SHA-256 are in `results/inputs/`. + +| Variable | Meaning | +| --- | --- | +| `STUDY` | this folder | +| `TOOLBOX_ROOT` | raven-toolbox checkout (`94f3940`) | +| `PY` | Python interpreter with pandas | +| `DATA` | folder of the two archives | +| `BINARIES_CACHE` | `raven_toolbox/binaries` cache with BLAST+ 2.17.0 and DIAMOND 2.1.17 | +| `HANPO_FAA` | `hanpo.faa` (aligner-settings job) | +| `HOMOLOGY_SCRIPT`, `HOMOLOGY_URL_CACHE`, `HOMOLOGY_OFFLINE` | read by `run_cached.py`: path of the driver, UniProt cache folder, offline switch (set by the jobs) | + +The module names in the job files are site-specific; other systems need an equivalent environment. + +## Commands + +```bash +export STUDY=... TOOLBOX_ROOT=... PY=... DATA=... BINARIES_CACHE=... HANPO_FAA=... +sbatch -A $STUDY/job_fetch.sh +sbatch -A $STUDY/job_align_score.sh +sbatch -A $STUDY/job_aligner_settings.sh +``` + +The jobs run the following, with `W` the extracted `homology-inputs/` folder, `REFS=$W/refs`, `K118=$REFS/kegg118_organism_gene_ko_5org.tsv.gz` and `RES` the results folder. `--out` goes before the subcommand. Each `score` run writes `hitlevel_results.json` into `$W`, which the job copies to `$RES/.json` with the name in the comment. + +```bash +python run_cached.py --out $W fetch +python run_cached.py prefetch --out $W # UniProt tables and OMA pairs +python fetch_references.py --out $REFS --kegg118-sha256 520ba8bb09d6ca479ca1ab84ea70bf7c1d7f82d8f6406b717cee293b7c8609ca +python run_cached.py --out $W align # BLAST, 4 threads, E-value 1e-4 +python run_cached.py --out $W align --aligner diamond # DIAMOND, 4 threads, E-value 1e-3, --more-sensitive +python run_cached.py --out $W score --beta 0.5 --gene-ko $K118 # kegg118_blast_b0.5 +python run_cached.py --out $W score --beta 0.5 --gene-ko $K118 --aligner diamond # kegg118_diamond_b0.5 +python run_cached.py --out $W score --beta 1 --gene-ko $K118 # kegg118_blast_b1 +python run_cached.py --out $W score --beta 1 --gene-ko $K118 --aligner diamond # kegg118_diamond_b1 +python run_cached.py --out $W score --beta 0.5 --gene-ko $K118 --reference oma # oma_blast_b0.5 +python run_cached.py --out $W score --beta 0.5 --gene-ko $K118 --reference oma --aligner diamond # oma_diamond_b0.5 +python run_cached.py --out $W score --beta 0.5 --gene-ko $REFS/keggrest_organism_gene_ko_rxn.tsv.gz # keggrest_rxn_blast_b0.5 +python run_cached.py --out $W score --beta 0.5 --gene-ko $REFS/keggrest_organism_gene_ko_rxn.tsv.gz --aligner diamond # keggrest_rxn_diamond_b0.5 +python run_cached.py --out $W score --beta 0.5 --gene-ko $REFS/keggrest_organism_gene_ko_all.tsv.gz # keggrest_all_blast_b0.5 (BLAST) +python agreement.py --work $W --gene-ko $K118 --out $RES/agreement.tsv +python aligner_settings.py --hanpo $HANPO_FAA --template \ + --bindir $BINARIES_CACHE/blast-2.17.0-linux-x86_64 --work $TMPDIR/as --out $RES/aligner_settings.json --threads 4 --single-thread-full +python make_tables.py --res $RES > $RES/tables.md +``` + +## Outputs + +| File | Content | +| --- | --- | +| `run_cached.py` | Runs the driver with its UniProt download function replaced by a disk cache, so that `score --reference oma` reads the UniProt tables offline. | +| `results/.json` | Driver output of each score run (reference, aligner, beta). | +| `results/all_rows.tsv`, `results/tables.md` | Every scored setting of every score file, and all tables of the page including identity 20 and 60, DIAMOND length and E-value rows, current-KEGG and OMA sweeps (`make_tables.py`). | +| `results/agreement.tsv` | Jaccard index of the surviving (template gene, target gene) pairs of BLAST and DIAMOND (`agreement.py`). | +| `results/aligner_settings.json`, `results/align_blast.txt`, `results/align_diamond.txt` | *H. polymorpha* hits at BLAST E-value 1e-4 and 1e-5 with a row-level difference, thread test (first 500 sequences at 1 and 4 threads), single-thread timing of the full directions; hit counts of the alignments. | +| `results/inputs/` | Download manifests, KEGG REST info and OMA version. | diff --git a/scripts/parameter_tuning/homology/agreement.py b/scripts/parameter_tuning/homology/agreement.py new file mode 100644 index 0000000..d319ba8 --- /dev/null +++ b/scripts/parameter_tuning/homology/agreement.py @@ -0,0 +1,105 @@ +#!/usr/bin/env python +"""BLAST/DIAMOND agreement on the matches that survive the homology filters. + +For each target organism, reads the saved BLAST and DIAMOND hit tables of +homology_cutoff_kegg.py (``hits_.csv``, ``hits_diamond_.csv``), +applies one filter setting (kept when evalue <= max_evalue, align_len >= +min_align_len, identity >= min_identity, as in the driver's ``score``), and +collapses the kept hits of both directions into a set of (template gene, target +gene) pairs. It then compares the BLAST set B and the DIAMOND set D: + + jaccard |B & D| / |B | D| + blast_in_dmnd |B & D| / |B| (share of BLAST matches DIAMOND also finds) + dmnd_in_blast |B & D| / |D| + +once over all surviving pairs and once over the pairs the KEGG reference judges +(both genes carry a KO in the gene-KO table), which are the pairs the scores use. + +Settings: the base (40 %, 200, 1e-30), the toolbox default (40 %, 100, 1e-30) +and each organism's best identity at length 200 (kla 40, yli 35, +ani 35, eco 40). + + python agreement.py --work $W --gene-ko kegg118_organism_gene_ko_5org.tsv.gz --out agreement.tsv +""" + +from __future__ import annotations + +import argparse +import collections +import gzip +import pathlib + +import pandas as pd + +TEMPLATE = "sce" +TARGETS = ("kla", "yli", "ani", "eco") +BEST_IDE = {"kla": 40, "yli": 35, "ani": 35, "eco": 40} + + +def pairs(hits: pd.DataFrame, ide: float, length: int, evalue: float) -> set[tuple[str, str]]: + kept = hits[(hits.evalue <= evalue) & (hits.align_len >= length) & (hits.identity >= ide)] + out = set() + for a, b, src in zip(kept.from_gene, kept.to_gene, kept.from_id, strict=True): + out.add((a, b) if src == TEMPLATE else (b, a)) + return out + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument("--work", type=pathlib.Path, required=True) + ap.add_argument("--gene-ko", type=pathlib.Path, required=True) + ap.add_argument("--out", type=pathlib.Path, required=True) + args = ap.parse_args() + + genes = collections.defaultdict(set) + with gzip.open(args.gene_ko, "rt", encoding="utf-8") as fh: + next(fh) + for line in fh: + org, gene, _ = line.rstrip("\n").split("\t") + genes[org].add(gene) + + rows = [] + for code in TARGETS: + blast = pd.read_csv( + args.work / f"hits_{code}.csv", dtype={"from_gene": str, "to_gene": str} + ) + dmnd = pd.read_csv( + args.work / f"hits_diamond_{code}.csv", dtype={"from_gene": str, "to_gene": str} + ) + settings = { + "base_40_200": (40, 200, 1e-30), + "default_40_100": (40, 100, 1e-30), + f"best_{BEST_IDE[code]}_200": (BEST_IDE[code], 200, 1e-30), + } + for label, (ide, length, ev) in settings.items(): + b, d = pairs(blast, ide, length, ev), pairs(dmnd, ide, length, ev) + for subset in ("all", "kegg_judged"): + if subset == "kegg_judged": + keep = lambda p: p[0] in genes[TEMPLATE] and p[1] in genes[code] # noqa: E731, B023 + bb, dd = {p for p in b if keep(p)}, {p for p in d if keep(p)} + else: + bb, dd = b, d + inter, union = len(bb & dd), len(bb | dd) + rows.append( + { + "organism": code, + "setting": label, + "pairs": subset, + "n_blast": len(bb), + "n_diamond": len(dd), + "both": inter, + "jaccard": round(inter / union, 4) if union else None, + "blast_in_dmnd": round(inter / len(bb), 4) if bb else None, + "dmnd_in_blast": round(inter / len(dd), 4) if dd else None, + "raw_hits_blast": len(blast), + "raw_hits_diamond": len(dmnd), + } + ) + df = pd.DataFrame(rows) + df.to_csv(args.out, sep="\t", index=False) + print(df.to_string(index=False)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/parameter_tuning/homology/aligner_settings.py b/scripts/parameter_tuning/homology/aligner_settings.py new file mode 100644 index 0000000..bdbb138 --- /dev/null +++ b/scripts/parameter_tuning/homology/aligner_settings.py @@ -0,0 +1,189 @@ +#!/usr/bin/env python +"""Aligner-settings measurements: BLAST thread test and aligner E-value 1e-4 vs 1e-5. + +Template proteome: SGD S288C ``orf_trans_all``. Target: hanpo-GEM ``data/genomes/hanpo.faa``. + +1. Thread test: the first N (500) sequences of hanpo.faa as queries against a + BLAST database of the template, ``blastp -evalue 1e-4``, once at 1 thread and + once at 4; reports hits, wall time and whether the two hit tables are identical. +2. Full direction at 1 thread (optional, --single-thread-full): both directions + of hanpo vs template with ``blastp -num_threads 1``, wall time per direction. +3. Aligner E-value: ``run_blast`` (bidirectional) at + evalue 1e-4 and 1e-5 with --threads; hits, time, the number of extra hits, their + share of the 1e-4 table and how many of them have an E-value above 1e-5. + + python aligner_settings.py --hanpo hanpo.faa --template orf_trans_all.fasta \\ + --bindir /blast-2.17.0-linux-x86_64 \\ + --work $TMPDIR/as --out aligner_settings.json --threads 4 --single-thread-full +""" + +from __future__ import annotations + +import argparse +import json +import pathlib +import subprocess +import time + +OUTFMT = "10 qseqid sseqid evalue pident length bitscore ppos" + + +def run(cmd: list[str]) -> str: + proc = subprocess.run(cmd, capture_output=True, text=True) + if proc.returncode != 0: + raise RuntimeError(f"{cmd}: {proc.stderr}") + return proc.stdout + + +def first_n(fasta: pathlib.Path, n: int, out: pathlib.Path) -> int: + count, lines = 0, [] + for line in fasta.read_text().splitlines(): + if line.startswith(">"): + count += 1 + if count > n: + break + lines.append(line) + out.write_text("\n".join(lines) + "\n") + return min(count, n) + + +def timed_blastp(blastp: str, query, db, evalue: str, threads: int) -> tuple[list[str], float]: + t0 = time.perf_counter() + out = run( + [ + blastp, + "-query", + str(query), + "-db", + str(db), + "-evalue", + evalue, + "-outfmt", + OUTFMT, + "-num_threads", + str(threads), + ] + ) + return out.splitlines(), time.perf_counter() - t0 + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument("--hanpo", type=pathlib.Path, required=True) + ap.add_argument("--template", type=pathlib.Path, required=True) + ap.add_argument("--bindir", type=pathlib.Path, required=True) + ap.add_argument("--work", type=pathlib.Path, required=True) + ap.add_argument("--out", type=pathlib.Path, required=True) + ap.add_argument("--subset", type=int, default=500) + ap.add_argument("--threads", type=int, default=4) + ap.add_argument("--single-thread-full", action="store_true") + args = ap.parse_args() + args.work.mkdir(parents=True, exist_ok=True) + blastp, makeblastdb = str(args.bindir / "blastp"), str(args.bindir / "makeblastdb") + res: dict = { + "blastp_version": run([blastp, "-version"]).strip(), + "n_hanpo": args.hanpo.read_text().count(">"), + "n_template": args.template.read_text().count(">"), + } + + db_t, db_h = args.work / "db_template", args.work / "db_hanpo" + run([makeblastdb, "-in", str(args.template), "-dbtype", "prot", "-out", str(db_t)]) + run([makeblastdb, "-in", str(args.hanpo), "-dbtype", "prot", "-out", str(db_h)]) + + sub = args.work / "subset.faa" + res["subset_queries"] = first_n(args.hanpo, args.subset, sub) + tt = {} + for th in (1, args.threads): + hits, secs = timed_blastp(blastp, sub, db_t, "1e-4", th) + tt[th] = (sorted(hits), secs) + print(f"subset, {th} thread(s): {len(hits)} hits, {secs:.1f} s", flush=True) + res["thread_test"] = { + "direction": "hanpo subset -> template db", + "evalue": "1e-4", + **{f"threads_{th}": {"hits": len(h), "seconds": round(s, 1)} for th, (h, s) in tt.items()}, + "identical_hit_tables": tt[1][0] == tt[args.threads][0], + } + + if args.single_thread_full: + full = {} + for label, query, db in ( + ("hanpo->template", args.hanpo, db_t), + ("template->hanpo", args.template, db_h), + ): + hits, secs = timed_blastp(blastp, query, db, "1e-4", 1) + full[label] = {"hits": len(hits), "seconds": round(secs, 1)} + print(f"full {label}, 1 thread: {len(hits)} hits, {secs:.1f} s", flush=True) + res["single_thread_full"] = full + + from raven_toolbox.reconstruction.homology import run_blast + + ev = {} + tables = {} + for e in (1e-4, 1e-5): + t0 = time.perf_counter() + hits = run_blast( + "hanpo", + args.hanpo, + ["sce"], + [args.template], + evalue=e, + threads=args.threads, + blastp=blastp, + makeblastdb=makeblastdb, + ) + secs = time.perf_counter() - t0 + tables[e] = hits + ev[f"{e:g}"] = {"hits": len(hits), "seconds": round(secs, 1)} + print(f"run_blast evalue {e:g}: {len(hits)} hits, {secs:.1f} s", flush=True) + extra = len(tables[1e-4]) - len(tables[1e-5]) + above = int((tables[1e-4].evalue > 1e-5).sum()) + import collections + + def rows(df): + return collections.Counter( + zip( + df.from_id, + df.from_gene, + df.to_gene, + df.evalue, + df.identity, + df.align_len, + df.bitscore, + strict=False, + ) + ) + + r4, r5 = rows(tables[1e-4]), rows(tables[1e-5]) + only4, only5 = r4 - r5, r5 - r4 + ev4 = [k[3] for k, n in only4.items() for _ in range(n)] + ev["row_diff"] = { + "rows_only_in_1e-4": sum(only4.values()), + "rows_only_in_1e-5": sum(only5.values()), + "only_in_1e-4_with_evalue_above_1e-5": sum(1 for e in ev4 if e > 1e-5), + "only_in_1e-4_with_evalue_at_most_1e-5": sum(1 for e in ev4 if e <= 1e-5), + "only_in_1e-4_min_evalue": min(ev4) if ev4 else None, + "only_in_1e-4_with_evalue_below_1e-30": sum(1 for e in ev4 if e < 1e-30), + "only_in_1e-5_examples": [list(map(str, k)) for k in list(only5)[:5]], + "only_in_1e-4_at_most_1e-5_examples": [list(map(str, k)) for k in only4 if k[3] <= 1e-5][ + :5 + ], + } + ev.update( + { + "extra_hits": extra, + "extra_share_of_1e-4_table": round(extra / len(tables[1e-4]), 4), + "hits_with_evalue_above_1e-5_in_1e-4_table": above, + "hits_passing_max_evalue_1e-30_among_those": int( + ((tables[1e-4].evalue > 1e-5) & (tables[1e-4].evalue < 1e-30)).sum() + ), + "threads": args.threads, + } + ) + res["aligner_evalue"] = ev + args.out.write_text(json.dumps(res, indent=2)) + print(json.dumps(res, indent=2)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/parameter_tuning/homology/fetch_references.py b/scripts/parameter_tuning/homology/fetch_references.py new file mode 100644 index 0000000..e8c49dd --- /dev/null +++ b/scripts/parameter_tuning/homology/fetch_references.py @@ -0,0 +1,143 @@ +#!/usr/bin/env python +"""Download the gene-KO references and the S. cerevisiae ORF translations. + +Writes into OUT: + kegg118_organism_gene_ko_5org.tsv.gz rows of KEGG release 118 ``organism_gene_ko`` + (from the raven-toolbox ``kegg118_core.tar.gz`` release asset, sha256 checked + against the toolbox registry) for sce, kla, yli, ani, eco; header kept. + kegg118_ko_check.json how many of those KOs are linked to a reaction in + ``ko_reaction`` of the same bundle. + keggrest_organism_gene_ko_all.tsv.gz current KEGG (REST ``link/ko/``), all KOs. + keggrest_organism_gene_ko_rxn.tsv.gz the same, only KOs listed by ``link/reaction/ko`` + (the reaction-linked subset, as in the release-118 table). + keggrest_info.txt output of ``info/kegg`` (release and date). + sgd_orf_trans_all.fasta.gz SGD S288C translated ORFs (template proteome). + oma_version.json OMA browser API ``/api/version/``. + manifest_references.json URL, UTC time, size and sha256 of every download. + +KEGG REST requests are sequential with a 1 s pause. + + python fetch_references.py --out $W/refs --kegg118-sha256 +""" + +from __future__ import annotations + +import argparse +import datetime +import gzip +import hashlib +import io +import json +import pathlib +import tarfile +import time +import urllib.request + +ORGS = ("sce", "kla", "yli", "ani", "eco") +KEGG118_URL = ( + "https://github.com/CompMetEng/toolbox-assets/releases/download/kegg118/kegg118_core.tar.gz" +) +SGD_URL = ( + "http://sgd-archive.yeastgenome.org/sequence/S288C_reference/orf_protein/orf_trans_all.fasta.gz" +) +OMA_VERSION_URL = "https://omabrowser.org/api/version/" + +manifest: list[dict] = [] + + +def get(url: str) -> bytes: + req = urllib.request.Request(url, headers={"User-Agent": "raven-toolbox-studies/1.0"}) + with urllib.request.urlopen(req, timeout=900) as fh: + data = fh.read() + manifest.append( + { + "url": url, + "fetched_utc": datetime.datetime.now(datetime.UTC).isoformat(), + "bytes": len(data), + "sha256": hashlib.sha256(data).hexdigest(), + } + ) + print(f"got {url}: {len(data)} bytes", flush=True) + return data + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument("--out", type=pathlib.Path, required=True) + ap.add_argument("--kegg118-sha256", required=True) + args = ap.parse_args() + out = args.out + out.mkdir(parents=True, exist_ok=True) + + # KEGG 118 release table + core = get(KEGG118_URL) + sha = hashlib.sha256(core).hexdigest() + if sha != args.kegg118_sha256: + raise SystemExit(f"kegg118_core sha256 {sha} != registry {args.kegg118_sha256}") + with tarfile.open(fileobj=io.BytesIO(core), mode="r:gz") as tar: + names = tar.getnames() + ogk_name = next(n for n in names if n.endswith("organism_gene_ko.tsv.gz")) + kr_name = next(n for n in names if n.endswith("ko_reaction.tsv.gz")) + ogk = gzip.decompress(tar.extractfile(ogk_name).read()).decode("utf-8").splitlines() + kr = gzip.decompress(tar.extractfile(kr_name).read()).decode("utf-8").splitlines() + header, kept = ogk[0], [line for line in ogk[1:] if line.split("\t", 1)[0] in ORGS] + with gzip.open(out / "kegg118_organism_gene_ko_5org.tsv.gz", "wt", encoding="utf-8") as fh: + fh.write(header + "\n" + "\n".join(kept) + "\n") + rxn_kos = {line.split("\t")[0] for line in kr[1:]} + all_kos_118 = {line.split("\t")[2] for line in ogk[1:]} + check = { + "members": names, + "organism_gene_ko_header": header, + "rows_all_orgs": len(ogk) - 1, + "organisms_in_table": len({line.split("\t", 1)[0] for line in ogk[1:]}), + "kos_in_table": len(all_kos_118), + "kos_in_ko_reaction": len(rxn_kos), + "kos_in_table_with_reaction": len(all_kos_118 & rxn_kos), + "example_rows": kept[:3], + "rows_per_org": {o: sum(1 for line in kept if line.startswith(o + "\t")) for o in ORGS}, + "genes_per_org": { + o: len({line.split("\t")[1] for line in kept if line.startswith(o + "\t")}) + for o in ORGS + }, + } + (out / "kegg118_ko_check.json").write_text(json.dumps(check, indent=2)) + del ogk, kr, core + + # Current KEGG via REST + (out / "keggrest_info.txt").write_bytes(get("https://rest.kegg.jp/info/kegg")) + time.sleep(1) + rxn_link = get("https://rest.kegg.jp/link/reaction/ko").decode().splitlines() + rest_rxn_kos = {line.split("\t")[0].removeprefix("ko:") for line in rxn_link if line} + rows = [] + for org in ORGS: + time.sleep(1) + for line in get(f"https://rest.kegg.jp/link/ko/{org}").decode().splitlines(): + if not line: + continue + gene, ko = line.split("\t") + rows.append((org, gene.split(":", 1)[1], ko.removeprefix("ko:"))) + for name, keep in (("all", lambda r: True), ("rxn", lambda r: r[2] in rest_rxn_kos)): + with gzip.open( + out / f"keggrest_organism_gene_ko_{name}.tsv.gz", "wt", encoding="utf-8" + ) as fh: + fh.write(header + "\n") + for r in rows: + if keep(r): + fh.write("\t".join(r) + "\n") + check_rest = { + "kos_with_reaction": len(rest_rxn_kos), + "rows_per_org_all": {o: sum(1 for r in rows if r[0] == o) for o in ORGS}, + "rows_per_org_rxn": { + o: sum(1 for r in rows if r[0] == o and r[2] in rest_rxn_kos) for o in ORGS + }, + } + (out / "keggrest_check.json").write_text(json.dumps(check_rest, indent=2)) + + (out / "sgd_orf_trans_all.fasta.gz").write_bytes(get(SGD_URL)) + (out / "oma_version.json").write_bytes(get(OMA_VERSION_URL)) + (out / "manifest_references.json").write_text(json.dumps(manifest, indent=2)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/parameter_tuning/homology/job_align_score.sh b/scripts/parameter_tuning/homology/job_align_score.sh new file mode 100644 index 0000000..76f548f --- /dev/null +++ b/scripts/parameter_tuning/homology/job_align_score.sh @@ -0,0 +1,54 @@ +#!/bin/bash +#SBATCH -J hom-align +#SBATCH -n 1 -c 4 +#SBATCH --mem=16G +#SBATCH -t 04:00:00 +# Align the four targets against S. cerevisiae with BLAST and DIAMOND (4 threads, +# the driver's default), score every variant, compute the BLAST/DIAMOND agreement +# and the tables. Inputs: homology-inputs.tar.zst (job_fetch.sh). +# Usage: sbatch job_align_score.sh +set -euo pipefail +module purge; module load SciPy-bundle/2026.05-gfbf-2026.1 +: "${PY:?set PY}" "${DATA:?set DATA}" "${TOOLBOX_ROOT:?set TOOLBOX_ROOT}" "${BINARIES_CACHE:?set BINARIES_CACHE}" +SD=${STUDY:?set STUDY} +RES=$STUDY/results +TB=$TOOLBOX_ROOT +export HOMOLOGY_SCRIPT=$TB/scripts/homology_cutoff_kegg.py +export PYTHONPATH=$TB/src +mkdir -p $RES +zstd -dc $DATA/homology-inputs.tar.zst | tar -C $TMPDIR -xf - +W=$TMPDIR/homology-inputs +export HOMOLOGY_URL_CACHE=$W/uniprot_cache HOMOLOGY_OFFLINE=1 +REFS=$W/refs +date -u; hostname; nproc; lscpu | grep -E "Model name|^CPU\(s\)|Thread|Socket" +echo "raven-toolbox $(git -C $TB rev-parse HEAD)" +$PY -c "import raven_toolbox, pandas, sys; print('raven_toolbox', getattr(raven_toolbox, '__version__', '?'), 'pandas', pandas.__version__, 'python', sys.version.split()[0])" +BIN=$BINARIES_CACHE +$BIN/blast-2.17.0-linux-x86_64/blastp -version; $BIN/diamond-2.1.17-linux-x86_64/diamond version +$PY -c "from raven_toolbox.binaries import resolve_binary as r; print(r('blastp'), r('diamond'))" + +$PY -I $SD/run_cached.py --out $W align 2>&1 | tee $RES/align_blast.txt +$PY -I $SD/run_cached.py --out $W align --aligner diamond 2>&1 | tee $RES/align_diamond.txt + +score() { # name, args... + local name=$1; shift + echo "== score $name: $*" + $PY -I $SD/run_cached.py --out $W score "$@" 2>&1 | tee $RES/score_$name.txt + cp $W/hitlevel_results.json $RES/$name.json +} +K118=$REFS/kegg118_organism_gene_ko_5org.tsv.gz +score kegg118_blast_b0.5 --beta 0.5 --gene-ko $K118 +score kegg118_diamond_b0.5 --beta 0.5 --gene-ko $K118 --aligner diamond +score kegg118_blast_b1 --beta 1 --gene-ko $K118 +score kegg118_diamond_b1 --beta 1 --gene-ko $K118 --aligner diamond +score oma_blast_b0.5 --beta 0.5 --gene-ko $K118 --reference oma +score oma_diamond_b0.5 --beta 0.5 --gene-ko $K118 --reference oma --aligner diamond +score keggrest_rxn_blast_b0.5 --beta 0.5 --gene-ko $REFS/keggrest_organism_gene_ko_rxn.tsv.gz +score keggrest_rxn_diamond_b0.5 --beta 0.5 --gene-ko $REFS/keggrest_organism_gene_ko_rxn.tsv.gz --aligner diamond +score keggrest_all_blast_b0.5 --beta 0.5 --gene-ko $REFS/keggrest_organism_gene_ko_all.tsv.gz + +$PY -I $SD/agreement.py --work $W --gene-ko $K118 --out $RES/agreement.tsv +$PY -I $SD/make_tables.py --res $RES > $RES/tables.md +tar -C $W -cf - $(cd $W && ls hits_*.csv) | zstd -19 -T4 -q -f -o $DATA/homology-hits.tar.zst +ls -la $DATA/homology-hits.tar.zst +date -u diff --git a/scripts/parameter_tuning/homology/job_aligner_settings.sh b/scripts/parameter_tuning/homology/job_aligner_settings.sh new file mode 100644 index 0000000..1401139 --- /dev/null +++ b/scripts/parameter_tuning/homology/job_aligner_settings.sh @@ -0,0 +1,24 @@ +#!/bin/bash +#SBATCH -J hom-asettings +#SBATCH -n 1 -c 4 +#SBATCH --mem=8G +#SBATCH -t 03:00:00 +# Thread test (500 hanpo queries, 1 vs 4 threads), single-thread full directions, +# and aligner E-value 1e-4 vs 1e-5 (run_blast, 4 threads), H. polymorpha vs +# S. cerevisiae (SGD orf_trans_all from homology-inputs.tar.zst). +# Usage: sbatch job_aligner_settings.sh +set -euo pipefail +module purge; module load SciPy-bundle/2026.05-gfbf-2026.1 +: "${PY:?set PY}" "${DATA:?set DATA}" "${TOOLBOX_ROOT:?set TOOLBOX_ROOT}" "${BINARIES_CACHE:?set BINARIES_CACHE}" "${HANPO_FAA:?set HANPO_FAA}" +SD=${STUDY:?set STUDY} +RES=$STUDY/results +export PYTHONPATH=$TOOLBOX_ROOT/src +mkdir -p $RES +zstd -dc $DATA/homology-inputs.tar.zst | tar -C $TMPDIR -xf - homology-inputs/refs/sgd_orf_trans_all.fasta.gz +zcat $TMPDIR/homology-inputs/refs/sgd_orf_trans_all.fasta.gz > $TMPDIR/sce_sgd.fasta +date -u; hostname; nproc; lscpu | grep -E "Model name|^CPU\(s\)" +sha256sum $HANPO_FAA +$PY -I $SD/aligner_settings.py --hanpo $HANPO_FAA \ + --template $TMPDIR/sce_sgd.fasta --bindir $BINARIES_CACHE/blast-2.17.0-linux-x86_64 \ + --work $TMPDIR/as --out $RES/aligner_settings.json --threads 4 --single-thread-full +date -u diff --git a/scripts/parameter_tuning/homology/job_fetch.sh b/scripts/parameter_tuning/homology/job_fetch.sh new file mode 100644 index 0000000..5702b73 --- /dev/null +++ b/scripts/parameter_tuning/homology/job_fetch.sh @@ -0,0 +1,25 @@ +#!/bin/bash +#SBATCH -J hom-fetch +#SBATCH -n 1 -c 1 +#SBATCH -t 02:00:00 +# Download all inputs of the homology cut-off study and pack them into one +# zstd tar under $DATA. Usage: sbatch job_fetch.sh +set -euo pipefail +module purge; module load SciPy-bundle/2026.05-gfbf-2026.1 +: "${PY:?set PY}" "${DATA:?set DATA}" "${TOOLBOX_ROOT:?set TOOLBOX_ROOT}" +SD=${STUDY:?set STUDY} +export HOMOLOGY_SCRIPT=$TOOLBOX_ROOT/scripts/homology_cutoff_kegg.py +export PYTHONPATH=$TOOLBOX_ROOT/src +W=$TMPDIR/homology-inputs +mkdir -p $W $DATA +export HOMOLOGY_URL_CACHE=$W/uniprot_cache +date -u; hostname +$PY -I $SD/run_cached.py --out $W fetch +$PY -I $SD/run_cached.py prefetch --out $W +$PY -I $SD/fetch_references.py --out $W/refs \ + --kegg118-sha256 520ba8bb09d6ca479ca1ab84ea70bf7c1d7f82d8f6406b717cee293b7c8609ca +for f in $W/proteomes/*.faa; do echo "$f $(grep -c '>' $f)"; done +ls -la $W $W/refs $W/uniprot_cache +tar -C $TMPDIR -cf - homology-inputs | zstd -19 -T1 -q -o $DATA/homology-inputs.tar.zst -f +ls -la $DATA/homology-inputs.tar.zst; sha256sum $DATA/homology-inputs.tar.zst +date -u diff --git a/scripts/parameter_tuning/homology/make_tables.py b/scripts/parameter_tuning/homology/make_tables.py new file mode 100644 index 0000000..224a0ae --- /dev/null +++ b/scripts/parameter_tuning/homology/make_tables.py @@ -0,0 +1,192 @@ +#!/usr/bin/env python +"""Turn the score files of a run into the tables of the study. + +Reads RES/.json (copies of the driver's ``hitlevel_results.json``, one per +score run: kegg118_blast_b0.5, kegg118_diamond_b0.5, kegg118_blast_b1, +kegg118_diamond_b1, oma_blast_b0.5, oma_diamond_b0.5, keggrest_rxn_blast_b0.5, +keggrest_rxn_diamond_b0.5, keggrest_all_blast_b0.5). +Writes RES/all_rows.tsv (every scored setting of every variant) and prints +Markdown tables. + + python make_tables.py --res results/ > results/tables.md +""" + +from __future__ import annotations + +import argparse +import json +import pathlib + +import pandas as pd + +ORGS = ("kla", "yli", "ani", "eco") +BASE = {"max_evalue": 1e-30, "min_align_len": 200, "min_identity": 40} + + +def load(res: pathlib.Path, name: str): + path = res / f"{name}.json" + return json.loads(path.read_text()) if path.is_file() else None + + +def row(data, code, **kw): + want = {**BASE, **kw} + return next(r for r in data[code]["rows"] if all(r[k] == v for k, v in want.items())) + + +def fmt(x): + return "n/a" if x is None else f"{x:.3f}" + + +def sweep_best(data, code, key="fbeta"): + rows = [r for r in data[code]["rows"] if r["min_align_len"] == 200 and r["max_evalue"] == 1e-30] + best = max(rows, key=lambda r: r[key]) + return best["min_identity"], best[key] + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument("--res", type=pathlib.Path, required=True) + args = ap.parse_args() + variants = [ + "kegg118_blast_b0.5", + "kegg118_diamond_b0.5", + "kegg118_blast_b1", + "kegg118_diamond_b1", + "oma_blast_b0.5", + "oma_diamond_b0.5", + "keggrest_rxn_blast_b0.5", + "keggrest_rxn_diamond_b0.5", + "keggrest_all_blast_b0.5", + ] + data = {v: load(args.res, v) for v in variants} + flat = [] + for v, d in data.items(): + if d: + for code, block in d.items(): + for r in block["rows"]: + flat.append( + { + "variant": v, + "organism": code, + "truth_pairs": block["truth_pairs"], + "n_hits": block["n_hits"], + **r, + } + ) + pd.DataFrame(flat).to_csv(args.res / "all_rows.tsv", sep="\t", index=False) + + k = data["kegg118_blast_b0.5"] + print("## Identity sweep, F0.5, KEGG 118 reference, BLAST\n") + print("| identity | " + " | ".join(f"`{o}`" for o in ORGS) + " |") + print("|---|---|---|---|---|") + for ide in (20, 25, 30, 35, 40, 45, 50, 60): + cells = [fmt(row(k, o, min_identity=ide)["fbeta"]) for o in ORGS] + print(f"| {ide} | " + " | ".join(cells) + " |") + + print("\n## Alignment-length sweep, F0.5, KEGG 118, BLAST\n") + print("| length | " + " | ".join(f"`{o}`" for o in ORGS) + " |") + print("|---|---|---|---|---|") + for ln in (50, 100, 150, 200, 300): + cells = [fmt(row(k, o, min_align_len=ln)["fbeta"]) for o in ORGS] + print(f"| {ln} | " + " | ".join(cells) + " |") + + print("\n### Precision and recall at length 200, 100 and 50, KEGG 118, BLAST\n") + print( + "| org | P 200 | R 200 | P 100 | R 100 | P 50 | R 50 | recall gain 200->100 (pp) | " + "TP gain 200->100 (%) |" + ) + print("|---|---|---|---|---|---|---|---|---|") + for o in ORGS: + a, b, c = row(k, o), row(k, o, min_align_len=100), row(k, o, min_align_len=50) + print( + f"| {o} | {a['precision']:.3f} | {a['recall']:.3f} | {b['precision']:.3f} | " + f"{b['recall']:.3f} | {c['precision']:.3f} | {c['recall']:.3f} | " + f"{100 * (b['recall'] - a['recall']):.1f} | {100 * (b['tp'] / a['tp'] - 1):.1f} |" + ) + + print("\n## E-value sweep, F0.5, KEGG 118, BLAST\n") + print("| max_evalue | " + " | ".join(f"`{o}`" for o in ORGS) + " |") + print("|---|---|---|---|---|") + for ev in (1e-4, 1e-10, 1e-30, 1e-50, 1e-100): + vals = [row(k, o, max_evalue=ev)["fbeta"] for o in ORGS] + print(f"| {ev:g} | " + " | ".join(fmt(v) for v in vals) + " |") + + dm = data["kegg118_diamond_b0.5"] + if dm: + print("\n## BLAST and DIAMOND, best identity (length 200, 1e-30), F0.5, KEGG 118\n") + print("| org | best BLAST | best DIAMOND | score BLAST vs DIAMOND at the BLAST optimum |") + print("|---|---|---|---|") + for o in ORGS: + ib, sb = sweep_best(k, o) + idd, sd = sweep_best(dm, o) + sd_at_b = row(dm, o, min_identity=ib)["fbeta"] + print( + f"| {o} | {ib} | {idd} | {sb:.3f} vs {sd_at_b:.3f}" + f"{'' if idd == ib else f' (DIAMOND best {sd:.3f})'} |" + ) + print("\nDIAMOND length and E-value rows:\n") + print("| setting | " + " | ".join(ORGS) + " |") + print("|---|---|---|---|---|") + for label, kw in [ + ("len 100", {"min_align_len": 100}), + ("len 50", {"min_align_len": 50}), + ("len 300", {"min_align_len": 300}), + ("1e-100", {"max_evalue": 1e-100}), + ("1e-4", {"max_evalue": 1e-4}), + ]: + print(f"| {label} | " + " | ".join(fmt(row(dm, o, **kw)["fbeta"]) for o in ORGS) + " |") + + for name, label in ( + ("kegg118_blast_b1", "F1, KEGG 118, BLAST"), + ("kegg118_diamond_b1", "F1, KEGG 118, DIAMOND"), + ("oma_blast_b0.5", "F0.5, OMA, BLAST"), + ("oma_diamond_b0.5", "F0.5, OMA, DIAMOND"), + ("keggrest_rxn_blast_b0.5", "F0.5, current KEGG (REST, reaction-linked KOs), BLAST"), + ("keggrest_rxn_diamond_b0.5", "F0.5, current KEGG (REST, reaction-linked KOs), DIAMOND"), + ("keggrest_all_blast_b0.5", "F0.5, current KEGG (REST, all KOs), BLAST"), + ): + d = data[name] + if not d: + print(f"\n## {label}: missing\n") + continue + print(f"\n## Identity sweep, {label}\n") + print("| identity | " + " | ".join(ORGS) + " |") + print("|---|---|---|---|---|") + for ide in (20, 25, 30, 35, 40, 45, 50, 60): + print( + f"| {ide} | " + + " | ".join(fmt(row(d, o, min_identity=ide)["fbeta"]) for o in ORGS) + + " |" + ) + print( + "| best identity (len 200) | " + + " | ".join(str(sweep_best(d, o)[0]) for o in ORGS) + + " |" + ) + print("| truth pairs | " + " | ".join(str(d[o]["truth_pairs"]) for o in ORGS) + " |") + if name.startswith("keggrest") or name.startswith("oma"): + print( + "| len 100 | " + + " | ".join(fmt(row(d, o, min_align_len=100)["fbeta"]) for o in ORGS) + + " |" + ) + print( + "| 1e-100 | " + + " | ".join(fmt(row(d, o, max_evalue=1e-100)["fbeta"]) for o in ORGS) + + " |" + ) + + print("\n## Counts at the base setting (40, 200, 1e-30), KEGG 118, BLAST\n") + print("| org | hits | truth pairs | judged | unjudgeable | tp | fp | P | R |") + print("|---|---|---|---|---|---|---|---|---|") + for o in ORGS: + r = row(k, o) + print( + f"| {o} | {k[o]['n_hits']} | {k[o]['truth_pairs']} | {r['judged']} | {r['unjudgeable']} | " + f"{r['tp']} | {r['fp']} | {r['precision']:.3f} | {r['recall']:.3f} |" + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/parameter_tuning/homology/run_cached.py b/scripts/parameter_tuning/homology/run_cached.py new file mode 100644 index 0000000..2f51d8c --- /dev/null +++ b/scripts/parameter_tuning/homology/run_cached.py @@ -0,0 +1,102 @@ +#!/usr/bin/env python +"""Run raven-toolbox's scripts/homology_cutoff_kegg.py with its UniProt downloads cached. + +The study driver (``fetch``, ``align``, ``score``) downloads UniProt proteomes and +UniProt-to-KEGG id tables through its ``_stream`` function, and ``score`` repeats +those downloads. This wrapper loads the driver unchanged and replaces ``_stream`` +with a disk cache, so every UniProt table is downloaded once, recorded (URL, time, +UniProt release headers) and reused offline by later jobs. + +Extra subcommand ``prefetch --out W`` wraps the driver: it calls the driver's +``_uniprot_keys`` for every proteome and its ``_oma_truth`` for every OMA target, so +that the UniProt tables and the OMA pairs (written to ``W/oma_pairs_.json``) +that ``score --reference oma`` needs are in the cache. Every other subcommand is +passed to the driver as is. + +Environment: + HOMOLOGY_SCRIPT path to the driver, scripts/homology_cutoff_kegg.py of the raven-toolbox checkout + HOMOLOGY_URL_CACHE directory of cached downloads (``index.jsonl`` lists them) + HOMOLOGY_OFFLINE=1 fail on a cache miss instead of downloading + +Command lines: + python run_cached.py --out $W fetch + python run_cached.py prefetch --out $W + python run_cached.py --out $W align [--aligner diamond] + python run_cached.py --out $W score --beta 0.5 --gene-ko K.tsv.gz [--aligner diamond] [--reference oma] +""" + +from __future__ import annotations + +import datetime +import hashlib +import importlib.util +import json +import os +import pathlib +import sys +import urllib.request + + +def _load_driver(): + spec = importlib.util.spec_from_file_location( + "homology_cutoff_kegg", os.environ["HOMOLOGY_SCRIPT"] + ) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +def _cached_stream(cache: pathlib.Path): + cache.mkdir(parents=True, exist_ok=True) + + def _stream(url: str) -> str: + path = cache / (hashlib.sha256(url.encode()).hexdigest()[:24] + ".txt") + if path.is_file(): + return path.read_text(encoding="utf-8") + if os.environ.get("HOMOLOGY_OFFLINE") == "1": + raise RuntimeError(f"not in cache: {url}") + with urllib.request.urlopen(url, timeout=900) as fh: + text = fh.read().decode("utf-8") + headers = { + k: v + for k, v in fh.headers.items() + if k.lower().startswith("x-uniprot") or k.lower() in ("date", "x-total-results") + } + path.write_text(text, encoding="utf-8") + with open(cache / "index.jsonl", "a", encoding="utf-8") as idx: + idx.write( + json.dumps( + { + "url": url, + "file": path.name, + "fetched_utc": datetime.datetime.now(datetime.UTC).isoformat(), + "headers": headers, + "lines": text.count("\n"), + } + ) + + "\n" + ) + print(f"downloaded {url} ({text.count(chr(10))} lines) {headers}", flush=True) + return text + + return _stream + + +def main(argv: list[str]) -> int: + mod = _load_driver() + mod._stream = _cached_stream(pathlib.Path(os.environ["HOMOLOGY_URL_CACHE"])) + if argv and argv[0] == "prefetch": + out = pathlib.Path(argv[argv.index("--out") + 1]) + out.mkdir(parents=True, exist_ok=True) + for code in mod.PROTEOMES: + keys = mod._uniprot_keys(code) + print(f"{code}: {len(keys)} UniProt keys with a KEGG id", flush=True) + for code in mod.OMA_TARGETS: + _, n_pairs, _ = mod._oma_truth(code, {}, {}, out) + print(f"{code}: {n_pairs} OMA pairs cached", flush=True) + return 0 + return mod.main(argv) + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/scripts/parameter_tuning/init/README.md b/scripts/parameter_tuning/init/README.md new file mode 100644 index 0000000..fbdd457 --- /dev/null +++ b/scripts/parameter_tuning/init/README.md @@ -0,0 +1,81 @@ +# ftINIT parameter studies + +Scripts for the ftINIT pages of the parameter-tuning documentation on Human-GEM: solver benchmark, parameter calibration (`mip_gap`, `big_m`, `force_on`, scaling), validation against RAVEN `ftINIT` on five cell lines, and determinism of the solution. `bigm/` holds the first three, `det/` the determinism study. + +## Inputs and environment + +All inputs except the prepared models come from a Human-GEM commit (`model/`, `data/datasets/`, `data/metabolicTasks/`, `code/test/`): `38567ab8` for `bigm/` and `1ea27d8b` (v2.0.0-38) for `det/`. The wrappers export the commit with `git archive` into `$TMPDIR/hg`. The preparations (`prep`, `preptask`, `ml_prep`, `det/prep.py`) take 26 to 72 min each and write the prepared models. + +| Variable | Meaning | +| --- | --- | +| `WORK` | work folder with `out/` (results), `shared/` (prepared models) and `logs/` | +| `TOOLBOX_ROOT` | raven-toolbox checkout (`94f3940`); its `src` goes on `PYTHONPATH` | +| `RAVEN_ROOT` | RAVEN checkout (`2d0630e`, MATLAB side) | +| `HUMANGEM_ROOT`, `HUMANGEM_COMMIT` | Human-GEM clone and commit | +| `PY` | Python interpreter with cobra, optlang and gurobipy | +| `BIGM_SCRIPTS`, `DET_SCRIPTS` | the `bigm/` and `det/` folders | +| `PREP_DIR` | `det/`: folder for `prep-base.pkl` and `prep-edit.pkl` on a shared file system (default `$WORK/prep`) | +| `GUROBI_HOME` | Gurobi 13.0.1 installation; the wrappers set `GRB_LICENSE_FILE` to its `gurobi.lic` | +| `MATLAB`, `MATLAB_PREFS` | `bigm/mljob.sh`: MATLAB R2024b executable; optional preference file to copy | + +The module names in the wrappers are site-specific; other systems need an equivalent environment. + +## Commands + +Calibration, solver benchmark and validation: + +```bash +export WORK=... TOOLBOX_ROOT=... RAVEN_ROOT=... HUMANGEM_ROOT=... PY=... GUROBI_HOME=... MATLAB=... BIGM_SCRIPTS=/bigm +J="sbatch -A -o $WORK/logs/%x-%j.out" +$J -J prep $BIGM_SCRIPTS/job.sh prep +$J -J preptask $BIGM_SCRIPTS/job.sh preptask +$J -J mlprep $BIGM_SCRIPTS/mljob.sh ml_prep +# after the preparations: +$J -J step0 $BIGM_SCRIPTS/job.sh step0 --cell HCT116 +$J -J scale $BIGM_SCRIPTS/job.sh scale --cell HCT116 +$J -J solver $BIGM_SCRIPTS/job.sh solver --solvers gurobi,hybrid +$J -J solv-glpk $BIGM_SCRIPTS/job.sh solver --solvers glpk +$J -J solv-scipy $BIGM_SCRIPTS/job.sh solver --solvers scipy +$J -J f-g001 $BIGM_SCRIPTS/job.sh full --gap 0.001 --big-m 1000 --tag g001-bm1000 +$J -J f-g003 $BIGM_SCRIPTS/job.sh full --gap 0.003 --big-m 1000 --tag g003-bm1000 +$J -J f-g01 $BIGM_SCRIPTS/job.sh full --gap 0.01 --big-m 1000 --tag g01-bm1000 +for b in 25 50 100 250 500; do $J -J f-bm$b $BIGM_SCRIPTS/job.sh full --gap 0.003 --big-m $b --tag g003-bm$b; done +$J -J f-g001-100 $BIGM_SCRIPTS/job.sh full --gap 0.001 --big-m 100 --tag g001-bm100 +$J -J f-off1000 $BIGM_SCRIPTS/job.sh full --gap 0.003 --big-m 1000 --scale off --tag g003-bm1000-scaleoff +$J -J f-off100 $BIGM_SCRIPTS/job.sh full --gap 0.003 --big-m 100 --scale off --tag g003-bm100-scaleoff +for g in 0.001 0.003 0.01; do $J -J fa-g$g $BIGM_SCRIPTS/job.sh full --gap $g --big-m 1000 --noabs --tag noabs-g$g-bm1000; done +$J -J fa-g001-100 $BIGM_SCRIPTS/job.sh full --gap 0.001 --big-m 100 --noabs --tag noabs-g0.001-bm100 +$J -J fa-g003-250 $BIGM_SCRIPTS/job.sh full --gap 0.003 --big-m 250 --noabs --tag noabs-g0.003-bm250 +$J -J fa-g003-50 $BIGM_SCRIPTS/job.sh full --gap 0.003 --big-m 50 --noabs --tag noabs-g0.003-bm50 +for c in DLD1 GBM HCT116 HELA RPE1; do + $J -J val-$c $BIGM_SCRIPTS/job.sh val --cell $c + $J -J ml-$c $BIGM_SCRIPTS/mljob.sh ml_cell $c +done +python $BIGM_SCRIPTS/summ.py $WORK/out +python $BIGM_SCRIPTS/compare.py --out $WORK/out --shared $WORK/shared --logs $WORK/logs --json tables.json +``` + +Determinism (about 100 core-hours at 2 CPUs per job): + +```bash +export WORK=... TOOLBOX_ROOT=... HUMANGEM_ROOT=... PY=... GUROBI_HOME=... DET_SCRIPTS=/det PREP_DIR= +export SBATCH_ACCOUNT= +mkdir -p $WORK/logs +jid=$(sbatch --parsable -A $SBATCH_ACCOUNT -J det-prep -o $WORK/logs/prep.log $DET_SCRIPTS/prep.sbatch) +bash $DET_SCRIPTS/submit_all.sh $jid +# after A-rtp1-1234 has finished: +sbatch -A $SBATCH_ACCOUNT -J det-edit -o $WORK/logs/edit.log $DET_SCRIPTS/edit.sbatch +sbatch -A $SBATCH_ACCOUNT -c 2 --mem=24G -o $WORK/logs/ties.log --wrap "source $DET_SCRIPTS/env.sh; \$PY $DET_SCRIPTS/ties.py \$P/prep-base.pkl" +python $DET_SCRIPTS/analyse.py $WORK/out +python $DET_SCRIPTS/det_extra.py $WORK/out $WORK/logs --json det_extra.json +``` + +## Outputs + +| File | Content | +| --- | --- | +| `bigm/bm.py` | Subcommands `prep`, `preptask`, `step0`, `scale`, `full`, `val`, `solver`; each writes `step0.json`, `scale.json`, `full-.json`, `val-.json` or `solver-.json` with the kept reaction sets into `$WORK/out`. | +| `bigm/ml_prep.m`, `bigm/ml_cell.m` | RAVEN `prepHumanModelForftINIT` and `ftINIT` for the validation; `ml_cell.m` writes `matlab-.txt`. | +| `bigm/summ.py`, `bigm/compare.py` | Summaries and all tables of the solver, calibration and validation studies. | +| `det/prep.py`, `det/build.py`, `det/select_edit.py`, `det/ties.py` | Template preparation, one or several `ftinit` builds (one JSON per build), the template edit, tied-score blocks. | +| `det/analyse.py`, `det/det_extra.py` | Tables of the determinism study from the build outputs and logs. | diff --git a/scripts/parameter_tuning/init/bigm/bm.py b/scripts/parameter_tuning/init/bigm/bm.py new file mode 100644 index 0000000..519154d --- /dev/null +++ b/scripts/parameter_tuning/init/bigm/bm.py @@ -0,0 +1,335 @@ +#!/usr/bin/env python3 +"""Driver of the ftINIT calibration, solver benchmark and validation runs (per-reaction big-M). + +Runs raven-toolbox ftINIT on the cleaned Human-GEM and writes one JSON file per +run series, with the kept reaction sets, into $WORK/out (prepared models into +$WORK/shared). + +Environment: + HUMANGEM_ROOT extracted Human-GEM tree (model/, data/, code/test/) at the + commit under study (38567ab8 for the October 2026 runs); + HG is accepted as the older name. + WORK work folder; OUT defaults to $WORK/out, SHARED to $WORK/shared. + OUT, SHARED override the two folders. + PYTHONPATH must contain $TOOLBOX_ROOT/src (raven-toolbox 94f3940). + +Subcommands (one per job, see job.sh): + prep cleaned Human-GEM -> prep_init_model(no tasks, scale=True) + -> SHARED/prep_notask.pkl, SHARED/ref_reactions.json + preptask same with metabolicTasks_Essential.txt -> SHARED/prep_task.pkl + step0 [--cell C] step-0 MILP grid (mip_gap, big_m, force_on) -> OUT/step0.json + scale [--cell C] unscaled prep, rescale_for_init(msd) x big_m 1000/100 -> OUT/scale.json + full --gap G --big-m B [--scale off] [--noabs] --tag T + one ftinit() call, series 1+1, time_limit 600 -> OUT/full-T.json + val --cell C no-task, task and no-task big_m=100 ftinit() -> OUT/val-C.json + solver --solvers a,b ftinit() per optlang interface, time_limit 900 + -> OUT/solver-a_b.json + +Command line: python bm.py [options] +""" + +import argparse +import json +import os +import pickle +import sys +import time +from pathlib import Path + +HG = Path(os.environ.get("HUMANGEM_ROOT") or os.environ["HG"]) +sys.path.insert(0, str(HG / "code" / "test")) +import cobra # noqa: E402 +import estimateEssentialGenes as E # noqa: E402 + +from raven_toolbox.init import ( # noqa: E402 + ftinit, + gene_scores_from_expression, # noqa: E402 + prep_init_model, + score_reactions_from_genes, +) +from raven_toolbox.init.ftinit import run_ftinit # noqa: E402 +from raven_toolbox.init.merge import group_rxn_scores # noqa: E402 +from raven_toolbox.init.prep import rescale_for_init # noqa: E402 +from raven_toolbox.init.steps import get_init_steps # noqa: E402 +from raven_toolbox.io import read_yaml_model # noqa: E402 +from raven_toolbox.tasks.tasklist import parse_task_list # noqa: E402 + +OUT = Path(os.environ.get("OUT") or Path(os.environ["WORK"]) / "out") +SHARED = Path( + os.environ.get("SHARED") + or (Path(os.environ["WORK"]) / "shared" if os.environ.get("WORK") else OUT) +) + + +def use_gurobi(m): + m.solver = "gurobi" + + +def clean_model(): + m = read_yaml_model(str(HG / "model" / "Human-GEM.yml")) + rm = set(E._drug_reactions(m)) | set(E._aa_triplet_reactions(m)) + rm |= {E.DUPLICATE_COMPLEX_I_RXN} & {r.id for r in m.reactions} + m.remove_reactions(sorted(rm), remove_orphans=True) + ids = {r.id for r in m.reactions} + spont = [r for r in E._load_spontaneous_reactions(E.REACTIONS_TSV) if r in ids] + custom = [r for r in E.CUSTOM_RXNS_TO_IGNORE if r in ids] + return m, spont, custom + + +def expr_for(cell): + tissues, expression = E._read_rnaseq(E.RNASEQ_FILE) + return expression[cell] + + +def scores_for(ref, cell): + g = gene_scores_from_expression(expr_for(cell), 1.0) + return g, score_reactions_from_genes(ref, g) + + +def jac(a, b): + a, b = set(a), set(b) + return len(a & b) / len(a | b) if a | b else 1.0 + + +def dump(name, obj): + (OUT / name).write_text(json.dumps(obj)) + + +def cmd_prep(a): + m, spont, custom = clean_model() + print("cleaned", len(m.reactions), "spont", len(spont), "custom", len(custom), flush=True) + (SHARED / "ref_reactions.json").write_text(json.dumps(sorted(r.id for r in m.reactions))) + t = time.time() + prep = prep_init_model(m, None, ext_comp="e", spontaneous=spont, custom=custom, scale=True) + print("prep notask s", time.time() - t, "min_model", len(prep.min_model.reactions), flush=True) + pickle.dump(prep, open(SHARED / "prep_notask.pkl", "wb")) + + +def cmd_preptask(a): + m, spont, custom = clean_model() + tasks = parse_task_list(E.ESSENTIAL_TASKS) + cobra.Configuration().processes = 1 + t = time.time() + prep = prep_init_model( + m, tasks, ext_comp="e", spontaneous=spont, custom=custom, scale=True, verbose=True + ) + print("prep task s", time.time() - t, "essential", len(prep.essential_rxns), flush=True) + pickle.dump(prep, open(SHARED / "prep_task.pkl", "wb")) + + +def load_prep(name): + p = pickle.load(open(SHARED / name, "rb")) + return p + + +def step0_inputs(prep, rxn_scores): + step = get_init_steps("1+1")[0] + to_zero = prep.masks.ignored(step.ignore_mask) + scores = group_rxn_scores( + prep.min_model, rxn_scores, prep.orig_rxn_ids, prep.group_ids, to_zero + ) + return step, scores + + +def run_step0(mm, scores, prep, step, **kw): + t = time.time() + try: + res = run_ftinit( + mm, + scores, + essential_rxns=set(prep.essential_rxns), + allow_excretion=step.allow_met_secr, + rem_pos_rev=step.pos_rev_off, + ignore_mets=step.mets_to_ignore, + **kw, + ) + return dict( + seconds=time.time() - t, + status="ok", + objective=res.objective, + n_kept=len(res.on_reactions), + reactions=sorted(res.on_reactions), + solver_status=res.status, + ) + except Exception as ex: # noqa: BLE001 + return dict( + seconds=time.time() - t, + status=f"FAIL:{type(ex).__name__}:{str(ex)[:100]}", + objective=None, + n_kept=0, + reactions=[], + ) + + +def cmd_step0(a): + prep = load_prep("prep_notask.pkl") + g, rs = scores_for(prep.ref_model, a.cell) + step, scores = step0_inputs(prep, rs) + mm = prep.min_model + res = {} + cfgs = [ + ("gap=%s" % x, dict(mip_gap=x, big_m=1000.0, force_on=0.1)) # noqa: UP031 + for x in (0.0002, 0.001, 0.003, 0.01, 0.03, 0.1) + ] + cfgs += [ + ("big_m=%d" % b, dict(mip_gap=0.001, big_m=float(b), force_on=0.1)) # noqa: UP031 + for b in (10, 25, 50, 100, 250, 500, 1000) + ] + cfgs += [ + ("force_on=%s" % f, dict(mip_gap=0.001, big_m=1000.0, force_on=f)) # noqa: UP031 + for f in (0.02, 0.05, 0.1, 0.2, 0.5) + ] + seen = set() + for label, kw in cfgs: + key = json.dumps(kw, sort_keys=True) + if key in seen: + continue + seen.add(key) + print("run", label, flush=True) + res[label] = run_step0(mm, scores, prep, step, time_limit=900, **kw) + res[label]["kw"] = kw + print(label, res[label]["seconds"], res[label]["status"], res[label]["n_kept"], flush=True) + dump("step0.json", res) + + +def cmd_scale(a): + m, spont, custom = clean_model() + base = prep_init_model(m, None, ext_comp="e", spontaneous=spont, custom=custom, scale=False) + g, rs = scores_for(base.ref_model, a.cell) + step, scores = step0_inputs(base, rs) + res = {} + for bm_ in (1000.0, 100.0): + for label, msd in ( + ("scale=on,msd=25", 25.0), + ("msd=10", 10.0), + ("msd=50", 50.0), + ("msd=100", 100.0), + ("scale=off", None), + ): + mm = base.min_model.copy() + if msd is not None: + rescale_for_init(mm, msd) + k = f"{label}|big_m={int(bm_)}" + res[k] = run_step0( + mm, scores, base, step, mip_gap=0.001, big_m=bm_, force_on=0.1, time_limit=600 + ) + print(k, res[k]["seconds"], res[k]["status"], res[k]["n_kept"], flush=True) + dump("scale.json", res) + + +def run_full(prep, g, rs, **kw): + t = time.time() + try: + out = ftinit(prep, rxn_scores=rs, gene_scores=g, series="1+1", **kw) + return dict( + seconds=time.time() - t, + status="ok", + n_kept=len(out.reactions), + reactions=sorted(x.id for x in out.reactions), + kw=kw, + ) + except Exception as ex: # noqa: BLE001 + return dict( + seconds=time.time() - t, + status=f"FAIL:{type(ex).__name__}:{str(ex)[:150]}", + n_kept=0, + reactions=[], + kw=kw, + ) + + +def cmd_full(a): + kw = dict(mip_gap=a.gap, big_m=a.big_m, time_limit=600) + if a.noabs: + kw["mip_gap_abs"] = None + if a.scale == "off": + m, spont, custom = clean_model() + prep = prep_init_model(m, None, ext_comp="e", spontaneous=spont, custom=custom, scale=False) + else: + prep = load_prep("prep_notask.pkl") + g, rs = scores_for(prep.ref_model, a.cell) + r = run_full(prep, g, rs, **kw) + r["scale"] = a.scale + print(a.tag, r["seconds"], r["status"], r["n_kept"], flush=True) + dump(f"full-{a.tag}.json", r) + + +def cmd_val(a): + res = {} + for variant, pname, bm_ in ( + ("notask", "prep_notask.pkl", 1000.0), + ("task", "prep_task.pkl", 1000.0), + ("notask100", "prep_notask.pkl", 100.0), + ): + prep = load_prep(pname) + g, rs = scores_for(prep.ref_model, a.cell) + res[variant] = run_full(prep, g, rs, mip_gap=0.001, big_m=bm_, time_limit=600) + print( + a.cell, + variant, + res[variant]["seconds"], + res[variant]["status"], + res[variant]["n_kept"], + flush=True, + ) + dump(f"val-{a.cell}.json", res) + + +def cmd_solver(a): + res = {} + for solver in a.solvers.split(","): + for bm_ in (1000.0, 100.0): + if solver != "gurobi" and bm_ != 1000.0: + continue + k = f"{solver}|big_m={int(bm_)}" + t = time.time() + try: + prep = load_prep("prep_notask.pkl") + prep.ref_model.solver = solver + prep.min_model.solver = solver + g, rs = scores_for(prep.ref_model, a.cell) + out = ftinit( + prep, + rxn_scores=rs, + gene_scores=g, + series="1+1", + mip_gap=0.001, + time_limit=900, + big_m=bm_, + ) + res[k] = dict( + seconds=time.time() - t, + status="ok", + n_kept=len(out.reactions), + reactions=sorted(x.id for x in out.reactions), + ) + except Exception as ex: # noqa: BLE001 + res[k] = dict( + seconds=time.time() - t, + status=f"FAIL:{type(ex).__name__}:{str(ex)[:150]}", + n_kept=0, + reactions=[], + ) + print(k, res[k]["seconds"], res[k]["status"], res[k]["n_kept"], flush=True) + dump(f"solver-{a.solvers.replace(',', '_')}.json", res) + + +def main(): + ap = argparse.ArgumentParser() + sp = ap.add_subparsers(dest="cmd", required=True) + for n in ("prep", "preptask", "step0", "scale", "full", "val", "solver"): + p = sp.add_parser(n) + p.add_argument("--cell", default="HCT116") + p.add_argument("--gap", type=float, default=0.001) + p.add_argument("--big-m", dest="big_m", type=float, default=1000.0) + p.add_argument("--scale", default="on") + p.add_argument("--tag", default="x") + p.add_argument("--noabs", action="store_true") + p.add_argument("--solvers", default="gurobi") + p.set_defaults(fn=globals()[f"cmd_{n}"]) + a = ap.parse_args() + a.fn(a) + + +if __name__ == "__main__": + main() diff --git a/scripts/parameter_tuning/init/bigm/compare.py b/scripts/parameter_tuning/init/bigm/compare.py new file mode 100644 index 0000000..08444ae --- /dev/null +++ b/scripts/parameter_tuning/init/bigm/compare.py @@ -0,0 +1,438 @@ +#!/usr/bin/env python3 +"""Tables of the solver benchmark, the parameter calibration and the Human-GEM validation. + +Reads the JSON outputs of bm.py (step0.json, scale.json, full-*.json, solver-*.json, +val-*.json), the RAVEN reaction lists matlab-.txt and, if given, the reference +list matlab_ref_reactions.txt and the job logs, and prints the tables of the solver benchmark, the +parameter calibration and the Human-GEM validation. Every derived count (shared, only, symmetric difference) comes from +the reaction sets, not from rounded Jaccard values. Also writes all values as JSON. + +Command line: + python compare.py --out $WORK/out [--shared $WORK/shared] [--logs $WORK/logs] [--json tables.json] +""" + +import argparse +import glob +import json +import os +import re + +CELLS = ("DLD1", "GBM", "HCT116", "HELA", "RPE1") + + +def jac(a, b): + a, b = set(a), set(b) + return len(a & b) / len(a | b) if a | b else 1.0 + + +def load(d, n): + p = os.path.join(d, n) + return json.load(open(p)) if os.path.exists(p) else None + + +def relgap(ref, obj): + return (ref - obj) / abs(ref) + + +def table(title, head, rows): + print(f"\n### {title}\n") + print("| " + " | ".join(head) + " |") + print("|" + "---|" * len(head)) + for r in rows: + print("| " + " | ".join(str(x) for x in r) + " |") + + +def f4(x): + return f"{x:+.4f}".replace("+-", "-") + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--out", required=True, help="bm.py output folder") + ap.add_argument("--shared", help="folder with matlab_ref_reactions.txt") + ap.add_argument("--logs", help="folder with the job logs (prep and RAVEN times)") + ap.add_argument("--json", help="write all values to this JSON file") + a = ap.parse_args() + O = a.out # noqa: E741 + R = {} + + # ---------------- solver benchmark ---------------- + solv = {} + for f in sorted(glob.glob(os.path.join(O, "solver-*.json"))): + solv.update(json.load(open(f))) + if solv: + rows = [] + for k, v in solv.items(): + name, bm = k.split("|") + rows.append( + [ + name, + bm.split("=")[1], + round(v["seconds"]), + v["status"][:110], + v["n_kept"] or "none", + ] + ) + table( + "Solver benchmark: results", + ["interface", "big_m", "time (s)", "outcome", "reactions"], + rows, + ) + R["solver_results"] = { + k: {x: y for x, y in v.items() if x != "reactions"} for k, v in solv.items() + } + g = set(solv.get("gurobi|big_m=1000", {}).get("reactions", [])) + g100 = set(solv.get("gurobi|big_m=100", {}).get("reactions", [])) + gl = set(solv.get("glpk|big_m=1000", {}).get("reactions", [])) + if g and gl: + ov = dict( + glpk=len(gl), + gurobi=len(g), + shared=len(g & gl), + only_glpk=len(gl - g), + only_gurobi=len(g - gl), + symdiff=len(g ^ gl), + jaccard=round(jac(g, gl), 4), + ) + R["solver_glpk_vs_gurobi"] = ov + print("\nGLPK vs Gurobi (big_m=1000):", ov) + if g and g100: + R["solver_gurobi_100_vs_1000"] = dict( + jaccard=round(jac(g, g100), 4), symdiff=len(g ^ g100) + ) + print("Gurobi big_m=100 vs 1000:", R["solver_gurobi_100_vs_1000"]) + + # ---------------- step 0 ---------------- + s = load(O, "step0.json") + if s: + ref = s["gap=0.0002"] + rows = [] + for k in ("gap=0.0002", "gap=0.001", "gap=0.003", "gap=0.01", "gap=0.03", "gap=0.1"): + v = s[k] + rows.append( + [ + k.split("=")[1], + round(v["seconds"]), + round(v["objective"]), + "ref" if k == "gap=0.0002" else f4(relgap(ref["objective"], v["objective"])), + "ref" if k == "gap=0.0002" else f"{jac(v['reactions'], ref['reactions']):.4f}", + ] + ) + table( + "Step-0 mip_gap (big_m=1000, force_on=0.1)", + ["mip_gap", "time (s)", "objective", "rel.obj.gap", "Jaccard vs tightest"], + rows, + ) + bm1000 = s[ + "gap=0.001" + ] # big_m=1000 at gap 0.001 is the same configuration and was run once + bm100 = s["big_m=100"] + rows = [] + for b in (1000, 500, 250, 100, 50, 25, 10): + v = bm1000 if b == 1000 else s[f"big_m={b}"] + rows.append( + [ + b, + round(v["seconds"]), + "ref" if b == 1000 else f4(relgap(bm1000["objective"], v["objective"])), + "ref" if b == 1000 else f"{jac(v['reactions'], bm1000['reactions']):.3f}", + "ref" if b == 100 else f"{jac(v['reactions'], bm100['reactions']):.3f}", + ] + ) + table( + "Step-0 big_m (mip_gap=0.001, force_on=0.1)", + ["big_m", "time (s)", "rel.obj.gap", "Jaccard vs big_m=1000", "Jaccard vs big_m=100"], + rows, + ) + rows = [] + for fo in ("0.1", "0.02", "0.05", "0.2", "0.5"): + v = bm1000 if fo == "0.1" else s[f"force_on={fo}"] + rows.append( + [ + fo, + round(v["seconds"]), + "ref" if fo == "0.1" else f4(relgap(bm1000["objective"], v["objective"])), + "ref" if fo == "0.1" else f"{jac(v['reactions'], bm1000['reactions']):.3f}", + ] + ) + table( + "Step-0 force_on (mip_gap=0.001, big_m=1000)", + ["force_on", "time (s)", "rel.obj.gap", "Jaccard vs 0.1"], + rows, + ) + R["step0"] = { + k: dict( + seconds=v["seconds"], + objective=v["objective"], + n_kept=v["n_kept"], + solver_status=v.get("solver_status"), + jac_vs_tightest=jac(v["reactions"], ref["reactions"]), + jac_vs_bm1000=jac(v["reactions"], bm1000["reactions"]), + jac_vs_bm100=jac(v["reactions"], bm100["reactions"]), + symdiff_vs_bm1000=len(set(v["reactions"]) ^ set(bm1000["reactions"])), + ) + for k, v in s.items() + } + js = [jac(s[k]["reactions"], ref["reactions"]) for k in s if k != "gap=0.0002"] + print( + f"\nStep-0 Jaccard range vs tightest over all settings: {min(js):.4f} to {max(js):.4f}" + ) + print( + "Step-0 statuses:", + sorted({v.get("solver_status") for v in s.values()}), + "times", + round(min(v["seconds"] for v in s.values()), 1), + "to", + round(max(v["seconds"] for v in s.values()), 1), + ) + + # ---------------- scaling ---------------- + sc = load(O, "scale.json") + if sc: + r0 = sc["scale=on,msd=25|big_m=1000"] + rows = [] + for lab, name in ( + ("scale=on,msd=25", "scale on, msd=25"), + ("msd=10", "msd=10"), + ("msd=50", "msd=50"), + ("msd=100", "msd=100"), + ("scale=off", "scale off"), + ): + row = [name] + for b in (1000, 100): + v = sc[f"{lab}|big_m={b}"] + isref = lab == "scale=on,msd=25" and b == 1000 + row += [ + round(v["seconds"]), + "ref" if isref else f4(relgap(r0["objective"], v["objective"])), + "ref" if isref else f"{jac(v['reactions'], r0['reactions']):.3f}", + ] + rows.append(row) + table( + "Step-0 scaling (mip_gap=0.001)", + [ + "config", + "time bm1000", + "rel.obj.gap", + "Jaccard", + "time bm100", + "rel.obj.gap", + "Jaccard", + ], + rows, + ) + R["scale"] = { + k: dict( + seconds=v["seconds"], + objective=v["objective"], + status=v["status"], + n_kept=v["n_kept"], + jac_vs_ref=jac(v["reactions"], r0["reactions"]), + ) + for k, v in sc.items() + } + if s: + rec = dict( + scale_ref_objective=r0["objective"], + step0_gap0001_objective=s["gap=0.001"]["objective"], + jaccard=jac(r0["reactions"], s["gap=0.001"]["reactions"]), + ) + R["scale_vs_step0"] = rec + print("\nReconciliation scale ref vs step0 gap=0.001:", rec) + + # ---------------- full pipeline ---------------- + full = { + os.path.basename(f)[5:-5]: json.load(open(f)) + for f in sorted(glob.glob(os.path.join(O, "full-*.json"))) + } + if full: + ref = set(full["g001-bm1000"]["reactions"]) if "g001-bm1000" in full else set() + rows = [] + R["full"] = {} + for t, v in sorted(full.items(), key=lambda x: (-x[1]["kw"]["big_m"], x[0])): + st = set(v["reactions"]) + kw = v["kw"] + d = dict( + seconds=v["seconds"], + status=v["status"], + n_kept=v["n_kept"], + kw=kw, + scale=v.get("scale"), + jac_vs_g001_bm1000=jac(st, ref) if st else None, + symdiff_vs_g001_bm1000=len(st ^ ref) if st else None, + ) + R["full"][t] = d + rows.append( + [ + t, + kw["big_m"], + kw["mip_gap"], + kw.get("mip_gap_abs", "default (10)"), + v.get("scale"), + round(v["seconds"]), + v["status"][:70], + v["n_kept"], + f"{d['jac_vs_g001_bm1000']:.4f}" if st else "none", + d["symdiff_vs_g001_bm1000"], + ] + ) + table( + "Full pipeline runs (time_limit=600)", + [ + "tag", + "big_m", + "mip_gap", + "mip_gap_abs", + "scale", + "time (s)", + "status", + "n_kept", + "Jaccard vs g001-bm1000", + "sym. diff vs g001-bm1000", + ], + rows, + ) + # identical sets among runs with one cap + groups = {} + for _t, v in full.items(): + if v["n_kept"] and v.get("scale") == "on": + groups.setdefault(v["kw"]["big_m"], []).append(frozenset(v["reactions"])) + print( + "\nDistinct reaction sets per cap (scaled):", + {int(b): f"{len(set(x))} of {len(x)}" for b, x in groups.items()}, + ) + # symmetric differences of the kept reaction sets, counted from the sets + print("\nCap differences vs big_m=1000") + R["cap_differences"] = {} + for b, tag in ( + (500, "g003-bm500"), + (250, "g003-bm250"), + (100, "g001-bm100"), + (50, "g003-bm50"), + ): + if tag not in full: + continue + st = set(full[tag]["reactions"]) + exact = len(st ^ ref) + R["cap_differences"][b] = dict( + tag=tag, + exact=exact, + ratio_to_110=round(exact / 110, 3), + ) + print( + f" big_m={b} ({tag}): {exact} differing reactions, ratio to 110 = {exact / 110:.2f}" + ) + off = full.get("g003-bm1000-scaleoff") + if off and off["n_kept"]: + print( + "scale off bm1000:", + off["n_kept"], + round(off["seconds"], 1), + f"Jaccard {jac(off['reactions'], ref):.3f}", + ) + + # ---------------- validation ---------------- + val = {c: load(O, f"val-{c}.json") for c in CELLS} + rav = {} + for c in CELLS: + p = os.path.join(O, f"matlab-{c}.txt") + if os.path.exists(p): + rav[c] = {x.strip() for x in open(p) if x.strip()} + if any(val.values()) and rav: + R["validation"] = {} + rows = [] + for c in CELLS: + if val[c] and c in rav: + rows.append([c, len(rav[c]), val[c]["notask"]["n_kept"], val[c]["task"]["n_kept"]]) + table( + "Reactions in the extracted models", + ["Cell line", "RAVEN ftINIT", "no-task", "task-constrained"], + rows, + ) + for var, title in (("notask", "No-task vs RAVEN"), ("task", "Task-constrained vs RAVEN")): + rows = [] + for c in CELLS: + if not (val[c] and c in rav): + continue + t = set(val[c][var]["reactions"]) + m = rav[c] + rows.append([c, len(t & m), len(t - m), len(m - t), f"{jac(t, m):.3f}"]) + R["validation"].setdefault(c, {})[var] = dict( + n=len(t), + raven=len(m), + shared=len(t & m), + only_toolbox=len(t - m), + only_raven=len(m - t), + jaccard=jac(t, m), + seconds=val[c][var]["seconds"], + ) + table( + title, ["Cell line", "Shared", "Only raven-toolbox", "Only RAVEN", "Jaccard"], rows + ) + rows = [] + for c in CELLS: + if not (val[c] and c in rav): + continue + t1, t2, m = ( + set(val[c]["notask"]["reactions"]), + set(val[c]["notask100"]["reactions"]), + rav[c], + ) + rows.append([c, f"{jac(t1, m):.3f}", f"{jac(t2, m):.3f}", f"{jac(t1, t2):.3f}"]) + R["validation"][c]["notask100"] = dict( + n=len(t2), + jaccard_raven=jac(t2, m), + jaccard_vs_notask=jac(t1, t2), + symdiff_vs_notask=len(t1 ^ t2), + seconds=val[c]["notask100"]["seconds"], + ) + table( + "No-task at two big_m values", + ["Cell line", "J vs RAVEN, 1000", "J vs RAVEN, 100", "J between the two"], + rows, + ) + for var in ("notask", "task", "notask100"): + ts = [val[c][var]["seconds"] for c in CELLS if val[c]] + print( + f"ftinit {var} seconds: {[round(x) for x in ts]} -> {round(min(ts))} to {round(max(ts))}" + ) + if a.shared and os.path.exists(os.path.join(a.shared, "matlab_ref_reactions.txt")): + mref = { + x.strip() + for x in open(os.path.join(a.shared, "matlab_ref_reactions.txt")) + if x.strip() + } + n_out = sum( + len(set(val[c][v]["reactions"]) - mref) + for c in CELLS + if val[c] + for v in ("notask", "task", "notask100") + ) + print( + f"RAVEN prepared model: {len(mref)} reactions; raven-toolbox reactions outside it (all runs): {n_out}" + ) + R["validation_ref"] = dict(matlab_ref=len(mref), toolbox_reactions_outside=n_out) + if a.logs: + times = {} + for f in glob.glob(os.path.join(a.logs, "*.out")): + for line in open(f, errors="replace"): + m = re.match(r"prep (notask|task) s ([\d.]+)", line) + if m: + times[f"toolbox prep {m.group(1)}"] = float(m.group(2)) + m = re.match(r"prep (\d+)s$", line.strip()) + if m: + times["RAVEN prepHumanModelForftINIT"] = float(m.group(1)) + m = re.match(r"ftINIT (\w+) (\d+)s rxns (\d+)", line.strip()) + if m: + times[f"RAVEN ftINIT {m.group(1)}"] = (float(m.group(2)), int(m.group(3))) + if times: + print("\nTimes from logs:") + for k in sorted(times): + print(" ", k, times[k]) + R["log_times"] = times + if a.json: + json.dump(R, open(a.json, "w"), indent=1, default=str) + + +if __name__ == "__main__": + main() diff --git a/scripts/parameter_tuning/init/bigm/job.sh b/scripts/parameter_tuning/init/bigm/job.sh new file mode 100644 index 0000000..2d86351 --- /dev/null +++ b/scripts/parameter_tuning/init/bigm/job.sh @@ -0,0 +1,39 @@ +#!/bin/bash +#SBATCH -c 4 +#SBATCH --mem=64G +#SBATCH -t 0-08:00:00 +# Slurm wrapper for bm.py (example; account and log path go on the sbatch line). +# +# Usage: +# export WORK=... TOOLBOX_ROOT=... HUMANGEM_ROOT=... PY=... BIGM_SCRIPTS= +# sbatch -A -J step0 -o $WORK/logs/%x-%j.out $BIGM_SCRIPTS/job.sh step0 --cell HCT116 +# +# Environment: +# WORK work folder; bm.py writes $WORK/out and reads/writes $WORK/shared +# TOOLBOX_ROOT raven-toolbox checkout (commit 94f3940); its src/ goes on PYTHONPATH +# HUMANGEM_ROOT Human-GEM git clone; HUMANGEM_COMMIT (default 38567ab8) is exported +# with git archive into $TMPDIR/hg, which bm.py receives as HUMANGEM_ROOT +# PY Python interpreter of the environment (cobra 0.32.1, optlang 1.9.1, gurobipy) +# GUROBI_HOME Gurobi 13.0.1 installation (required); GRB_LICENSE_FILE is +# always set to its gurobi.lic, whatever the calling shell exported +module purge +module load SciPy-bundle/2026.05-gfbf-2026.1 +: "${WORK:?set WORK}" "${TOOLBOX_ROOT:?set TOOLBOX_ROOT}" "${HUMANGEM_ROOT:?set HUMANGEM_ROOT}" +HUMANGEM_COMMIT=${HUMANGEM_COMMIT:-38567ab8} +PY=${PY:-python3} +export GUROBI_HOME=${GUROBI_HOME:?set GUROBI_HOME} +export GRB_LICENSE_FILE=$GUROBI_HOME/gurobi.lic +export LD_LIBRARY_PATH=$GUROBI_HOME/lib PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 +B=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +[ -f "$B/bm.py" ] || B=${BIGM_SCRIPTS:?set BIGM_SCRIPTS to the folder of bm.py} +HGREPO=$HUMANGEM_ROOT +export HUMANGEM_ROOT=$TMPDIR/hg OUT=${OUT:-$WORK/out} SHARED=${SHARED:-$WORK/shared} +mkdir -p "$OUT" "$SHARED" "$HUMANGEM_ROOT" +git -C "$HGREPO" archive "$HUMANGEM_COMMIT" model data/datasets data/metabolicTasks code/test | tar -x -C "$HUMANGEM_ROOT" +echo "host $(hostname) Human-GEM $HUMANGEM_COMMIT toolbox $(git -C "$TOOLBOX_ROOT" log --oneline -1)" +PYTHONPATH="$TOOLBOX_ROOT/src" "$PY" -c ' +import sys, cobra, optlang, gurobipy +print("python", sys.version.split()[0], "cobra", cobra.__version__, "optlang", optlang.__version__, + "gurobipy", gurobipy.__version__, "gurobi library", ".".join(map(str, gurobipy.gurobi.version()))) +print("libgurobi", sorted({l.split()[-1] for l in open("/proc/self/maps") if "libgurobi" in l}))' +PYTHONPATH="$TOOLBOX_ROOT/src" "$PY" "$B/bm.py" "$@" diff --git a/scripts/parameter_tuning/init/bigm/ml_cell.m b/scripts/parameter_tuning/init/bigm/ml_cell.m new file mode 100644 index 0000000..60a303c --- /dev/null +++ b/scripts/parameter_tuning/init/bigm/ml_cell.m @@ -0,0 +1,13 @@ +% RAVEN ftINIT for one Hart2015 cell line ($CELL) on $SHARED/matlab_prepData.mat, +% default settings (series 1+1), transcrData threshold 1. Writes the reaction ids to +% $OUT/matlab-.txt. Command line: sbatch ... mljob.sh ml_cell +run(fullfile(fileparts(mfilename('fullpath')),'ml_common.m')); +HGD = getenv('HUMANGEM_ROOT'); cell_ = getenv('CELL'); +s = load(fullfile(getenv('SHARED'),'matlab_prepData.mat')); prepData = s.prepData; +T = readtable(fullfile(HGD,'data','datasets','Hart2015_RNAseq.txt'),'Delimiter','\t','FileType','text'); +tr.genes = T.gene; tr.tissues = {cell_}; tr.celltypes = {cell_}; +tr.levels = T.(cell_); tr.threshold = 1; +tic; [model] = ftINIT(prepData, cell_, [], [], 'transcrData', tr); +fprintf('ftINIT %s %.0fs rxns %d\n', cell_, toc, numel(model.rxns)); +fid = fopen(fullfile(getenv('OUT'),['matlab-' cell_ '.txt']),'w'); fprintf(fid,'%s\n',model.rxns{:}); fclose(fid); +run(fullfile(fileparts(mfilename('fullpath')),'ml_restore_pref.m')); diff --git a/scripts/parameter_tuning/init/bigm/ml_common.m b/scripts/parameter_tuning/init/bigm/ml_common.m new file mode 100644 index 0000000..d50c6e6 --- /dev/null +++ b/scripts/parameter_tuning/init/bigm/ml_common.m @@ -0,0 +1,15 @@ +% Common MATLAB set-up of ml_prep.m and ml_cell.m: RAVEN checkout, Gurobi MATLAB +% interface, Human-GEM code. Environment: RAVEN_ROOT (RAVEN checkout, commit +% 2d0630e3), GUROBI_HOME (Gurobi installation), HUMANGEM_ROOT (extracted Human-GEM +% tree). The RAVEN/ravenPath preference is set to RAVEN_ROOT so that findRAVENroot +% returns this checkout and restored by ml_restore_pref.m at the end; +% mljob.sh also keeps the preferences in $TMPDIR. +RV = getenv('RAVEN_ROOT'); +GM = fullfile(getenv('GUROBI_HOME'), 'matlab'); +HGD = getenv('HUMANGEM_ROOT'); +restoredefaultpath; addpath(genpath(RV)); addpath(GM); +addpath(genpath(fullfile(HGD,'code'))); +if ispref('RAVEN','ravenPath'), oldRavenPath = getpref('RAVEN','ravenPath'); else, oldRavenPath = []; end +setpref('RAVEN','ravenPath',RV); +assert(strcmp(findRAVENroot(), RV) || strcmp([findRAVENroot() filesep], [RV filesep]), 'RAVEN root %s', findRAVENroot()); +setRavenSolver('gurobi'); diff --git a/scripts/parameter_tuning/init/bigm/ml_prep.m b/scripts/parameter_tuning/init/bigm/ml_prep.m new file mode 100644 index 0000000..ad72ff1 --- /dev/null +++ b/scripts/parameter_tuning/init/bigm/ml_prep.m @@ -0,0 +1,14 @@ +% RAVEN template preparation for the Human-GEM validation. +% Reads $HUMANGEM_ROOT/model/Human-GEM.yml, runs prepHumanModelForftINIT with the +% essential tasks and reactions.tsv, writes $SHARED/matlab_ref_reactions.txt and +% $SHARED/matlab_prepData.mat. Command line: sbatch ... mljob.sh ml_prep +run(fullfile(fileparts(mfilename('fullpath')),'ml_common.m')); +HGD = getenv('HUMANGEM_ROOT'); +tic; model = readYAMLmodel(fullfile(HGD,'model','Human-GEM.yml')); fprintf('read %d rxns %.0fs\n', numel(model.rxns), toc); +tic; +prepData = prepHumanModelForftINIT(model, false, fullfile(HGD,'data','metabolicTasks','metabolicTasks_Essential.txt'), fullfile(HGD,'model','reactions.tsv')); +fprintf('prep %.0fs\n', toc); +rm = prepData.refModel; fprintf('refModel rxns %d\n', numel(rm.rxns)); +fid = fopen(fullfile(getenv('SHARED'),'matlab_ref_reactions.txt'),'w'); fprintf(fid,'%s\n',rm.rxns{:}); fclose(fid); +save(fullfile(getenv('SHARED'),'matlab_prepData.mat'),'prepData','-v7.3'); +run(fullfile(fileparts(mfilename('fullpath')),'ml_restore_pref.m')); diff --git a/scripts/parameter_tuning/init/bigm/ml_restore_pref.m b/scripts/parameter_tuning/init/bigm/ml_restore_pref.m new file mode 100644 index 0000000..253fc2d --- /dev/null +++ b/scripts/parameter_tuning/init/bigm/ml_restore_pref.m @@ -0,0 +1,2 @@ +% Restore the RAVEN/ravenPath preference saved by ml_common.m. +if isempty(oldRavenPath), rmpref('RAVEN','ravenPath'); else, setpref('RAVEN','ravenPath',oldRavenPath); end diff --git a/scripts/parameter_tuning/init/bigm/mljob.sh b/scripts/parameter_tuning/init/bigm/mljob.sh new file mode 100755 index 0000000..d5522c1 --- /dev/null +++ b/scripts/parameter_tuning/init/bigm/mljob.sh @@ -0,0 +1,38 @@ +#!/bin/bash +#SBATCH -c 4 +#SBATCH --mem=64G +#SBATCH -t 1-00:00:00 +# Slurm wrapper for the MATLAB side of the Human-GEM validation (example). +# +# Usage: +# export WORK=... RAVEN_ROOT=... HUMANGEM_ROOT=... BIGM_SCRIPTS= +# sbatch -A -J mlprep -o $WORK/logs/%x-%j.out $BIGM_SCRIPTS/mljob.sh ml_prep +# sbatch -A -J ml-DLD1 -o $WORK/logs/%x-%j.out $BIGM_SCRIPTS/mljob.sh ml_cell DLD1 +# +# Environment: +# WORK work folder ($WORK/out, $WORK/shared) +# RAVEN_ROOT RAVEN checkout (commit 2d0630e3) +# HUMANGEM_ROOT Human-GEM git clone; HUMANGEM_COMMIT (default 38567ab8) is exported into +# $TMPDIR/hg, which the .m files receive as HUMANGEM_ROOT +# MATLAB MATLAB executable (R2024b) +# MATLAB_PREFS optional matlabprefs.mat copied into the temporary preference folder +# GUROBI_HOME Gurobi 13.0.1 installation (required); GRB_LICENSE_FILE is +# always set to its gurobi.lic, whatever the calling shell exported +module purge +: "${WORK:?set WORK}" "${RAVEN_ROOT:?set RAVEN_ROOT}" "${HUMANGEM_ROOT:?set HUMANGEM_ROOT}" +HUMANGEM_COMMIT=${HUMANGEM_COMMIT:-38567ab8} +B=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +[ -f "$B/ml_common.m" ] || B=${BIGM_SCRIPTS:?set BIGM_SCRIPTS to the folder of the .m files} +HGREPO=$HUMANGEM_ROOT +export HUMANGEM_ROOT=$TMPDIR/hg OUT=${OUT:-$WORK/out} SHARED=${SHARED:-$WORK/shared} CELL=$2 RAVEN_ROOT +export GUROBI_HOME=${GUROBI_HOME:?set GUROBI_HOME} +export GRB_LICENSE_FILE=$GUROBI_HOME/gurobi.lic +export LD_LIBRARY_PATH=$GUROBI_HOME/lib +mkdir -p "$OUT" "$SHARED" "$HUMANGEM_ROOT" +git -C "$HGREPO" archive "$HUMANGEM_COMMIT" model data/datasets data/metabolicTasks code | tar -x -C "$HUMANGEM_ROOT" +export MATLAB_PREFDIR=$TMPDIR/prefs; mkdir -p $MATLAB_PREFDIR +[ -n "$MATLAB_PREFS" ] && cp "$MATLAB_PREFS" $MATLAB_PREFDIR/ +cd $TMPDIR +echo "RAVEN $(git -C "$RAVEN_ROOT" log --oneline -1)" +MATLAB=${MATLAB:?set MATLAB to the MATLAB executable} +$MATLAB -batch "run('$B/$1.m')" diff --git a/scripts/parameter_tuning/init/bigm/summ.py b/scripts/parameter_tuning/init/bigm/summ.py new file mode 100644 index 0000000..4f60a4b --- /dev/null +++ b/scripts/parameter_tuning/init/bigm/summ.py @@ -0,0 +1,66 @@ +"""Print the step-0, scaling, full-pipeline and solver summaries of bm.py outputs. + +Reads OUT/step0.json, OUT/scale.json, OUT/full-*.json and OUT/solver-gurobi_hybrid.json. +Command line: python summ.py [OUT] (default: $WORK/out, else ./out) +""" + +import glob +import json +import os +import sys + +O = sys.argv[1] if len(sys.argv) > 1 else os.path.join(os.environ.get("WORK", "."), "out") # noqa: E741 + + +def J(a, b): + a, b = set(a), set(b) + return len(a & b) / len(a | b) if a | b else 1 + + +s = json.load(open(os.path.join(O, "step0.json"))) +ref = s["gap=0.0002"] +print("STEP0 ref gap0.0002 obj", ref["objective"]) +for k, v in s.items(): + print( + k, + round(v["seconds"]), + v["status"], + v["solver_status"] if "solver_status" in v else "", + v["objective"], + v["n_kept"], + "jac_vs_gap.0002", + round(J(v["reactions"], ref["reactions"]), 4), + "jac_bm100", + round(J(v["reactions"], s["big_m=100"]["reactions"]), 4), + "jac_bm1000", + round(J(v["reactions"], s["gap=0.001"]["reactions"]), 4), + "relgap", + round((ref["objective"] - v["objective"]) / abs(ref["objective"]), 4) + if v["objective"] + else None, + ) +sc = json.load(open(os.path.join(O, "scale.json"))) +print("SCALE") +for k, v in sc.items(): + print( + k, + round(v["seconds"]), + v["status"][:60], + v["objective"], + v["n_kept"], + round(J(v["reactions"], sc["scale=on,msd=25|big_m=1000"]["reactions"]), 4), + ) +print("FULL") +ref = json.load(open(os.path.join(O, "full-g001-bm1000.json")))["reactions"] +for f in sorted(glob.glob(os.path.join(O, "full-*.json"))): + v = json.load(open(f)) + print( + os.path.basename(f)[5:-5], + round(v["seconds"]), + v["status"][:50], + v["n_kept"], + round(J(v["reactions"], ref), 4), + ) +print("SOLVER") +for k, v in json.load(open(os.path.join(O, "solver-gurobi_hybrid.json"))).items(): + print(k, round(v["seconds"]), v["status"][:120], v["n_kept"]) diff --git a/scripts/parameter_tuning/init/det/analyse.py b/scripts/parameter_tuning/init/det/analyse.py new file mode 100644 index 0000000..38b88dc --- /dev/null +++ b/scripts/parameter_tuning/init/det/analyse.py @@ -0,0 +1,146 @@ +"""Print the tables of the determinism builds (A full pipeline, B first-step probe, C sweep). + +Reads /{A,B,C}-*.json written by build.py. +Command line: python analyse.py [] (default: $WORK/out) +""" + +import itertools +import json +import os +import statistics as st +import sys + +O = os.path.join(sys.argv[1] if len(sys.argv) > 1 else os.path.join(os.environ["WORK"], "out"), "") # noqa: E741 + + +def load(n): + p = O + n + ".json" + return json.load(open(p)) if os.path.exists(p) else None + + +def sd(a, b): + return len(set(a) ^ set(b)) + + +print("== A: full pipeline") +for cfg in ["base", "rt", "rtp1", "rtp005"]: + r = [load(f"A-{cfg}-{s}-s{s}") for s in (1234, 7)] + if None in r: + print(cfg, "missing") + continue + a, b = r[0][0], r[1][0] + print( + cfg, + "rxnswing", + sd(a["reactions"], b["reactions"]), + "geneswing", + sd(a["essential"], b["essential"]), + "ess", + len(a["essential"]), + len(b["essential"]), + "nrxn", + len(a["reactions"]), + len(b["reactions"]), + "wall", + a["wall_s"], + b["wall_s"], + "ess_s", + a["essential_s"], + "growth", + a["growth"], + b["growth"], + "warn", + len(a["warnings"]), + len(b["warnings"]), + ) +for cfg in ["base", "rt"]: + r = load(f"A-{cfg}-rep-s1234") + if r: + print( + "repeat", + cfg, + "rxn", + sd(r[0]["reactions"], r[1]["reactions"]), + "gene", + sd(r[0]["essential"], r[1]["essential"]), + "wall", + r[0]["wall_s"], + r[1]["wall_s"], + "nrxn", + len(r[0]["reactions"]), + len(r[1]["reactions"]), + ) +print( + "base vs rt seed 1234: rxn", + sd(load("A-base-1234-s1234")[0]["reactions"], load("A-rt-1234-s1234")[0]["reactions"]), + "gene", + sd(load("A-base-1234-s1234")[0]["essential"], load("A-rt-1234-s1234")[0]["essential"]), +) +print("== B: stage-1 probe") +for cfg in ["base-t1", "rt-t1", "base-t4", "rt-t4"]: + runs = [] + for s in (1234, 7, 1, 2, 3, 4): + r = load(f"B-{cfg}-s{s}") + if r: + runs.append(r[0]) + sets = [frozenset(x["reactions"]) for x in runs] + objs = [round(x["steps"][0]["objective"], 3) for x in runs] + pairs = [len(a ^ b) for a, b in itertools.combinations(sets, 2)] + print( + cfg, + "n", + len(runs), + "distinct", + len(set(sets)), + "objs", + sorted(set(objs)), + "meanpair", + round(st.mean(pairs), 1) if pairs else None, + "wall", + [x["wall_s"] for x in runs], + "status", + [x["steps"][0]["status"] for x in runs], + ) + # fixed objective + for o in sorted(set(objs)): + ss = {frozenset(x["reactions"]) for x, ob in zip(runs, objs, strict=False) if ob == o} + print(" obj", o, "n", objs.count(o), "distinct", len(ss)) +print("== C: sweep") +for g in ["none", "5.0", "2.0", "1.0", "0.5", "0.1", "0.05"]: + runs = [] + for s in (1234, 7, 1, 2): + r = load(f"C-{g}-{s}-s{s}") + if r: + runs.append(r[0]) + if not runs: + continue + for k in (0, 1): + objs = [round(x["steps"][k]["objective"], 3) for x in runs] + ss = {frozenset(x["reactions"]) for x in runs} if k == 1 else None + st1 = {frozenset(x["reactions"]) for x in runs} + print( + g, + "stage", + k + 1, + "n", + len(runs), + "objs", + sorted(set(objs)), + "status", + [x["steps"][k]["status"] for x in runs], + "secs", + [x["steps"][k]["seconds"] for x in runs], + "n_on", + [x["steps"][k]["n_on"] for x in runs], + ) + print( + g, + "final distinct sets", + len({frozenset(x["reactions"]) for x in runs}), + "wall", + [x["wall_s"] for x in runs], + "nrxn", + [len(x["reactions"]) for x in runs], + "warnings", + [len(x["warnings"]) for x in runs], + ) diff --git a/scripts/parameter_tuning/init/det/build.py b/scripts/parameter_tuning/init/det/build.py new file mode 100644 index 0000000..a3c656c --- /dev/null +++ b/scripts/parameter_tuning/init/det/build.py @@ -0,0 +1,124 @@ +"""One ftINIT build (full pipeline or first N steps) with step records. + +Scores the Hart2015 column --cell as Human-GEM 1ea27d8b estimateEssentialGenes.py +does, runs ftinit (series 1+1, first --nsteps steps, big_m=BIG_M 100, +mip_gap_abs=MIP_GAP_ABS 10, time_limit --tl) with --rt/--pag/--seed/--threads, +records objective, status, gap and time of every step (wrapping the private +_solve_step), the warnings, reaction and gene ids, growth and, with --essential 1, +the essential genes (find_essential_genes). Writes a JSON list (one entry per --repeat). +Command line: python build.py --prep P.pkl --out X.json [--cell DLD1] [--seed 1234] + [--threads 1] [--rt 0|1] [--pag V] [--nsteps 2] [--fill-gaps 1] [--tl 1800] + [--essential 0|1] [--repeat 1] +""" + +import argparse +import json +import pickle +import sys +import time +import warnings + +import cobra + +sys.setrecursionlimit(10000) +cobra.Configuration().processes = 1 +import raven_toolbox.init # noqa +from raven_toolbox.init import ( # noqa: E402 + ftinit, + get_init_steps, +) + +FT = sys.modules["raven_toolbox.init.ftinit"] +import estimateEssentialGenes as eeg # noqa: E402 + +ap = argparse.ArgumentParser() +ap.add_argument("--prep", required=True) +ap.add_argument("--out", required=True) +ap.add_argument("--cell", default="DLD1") +ap.add_argument("--seed", type=int, default=1234) +ap.add_argument("--threads", type=int, default=1) +ap.add_argument("--rt", type=int, default=0) +ap.add_argument("--pag", type=float) +ap.add_argument("--nsteps", type=int, default=2) +ap.add_argument("--fill-gaps", type=int, default=1) +ap.add_argument("--tl", type=float, default=1800.0) +ap.add_argument("--essential", type=int, default=0) +ap.add_argument("--repeat", type=int, default=1) +a = ap.parse_args() + +prep = pickle.load(open(a.prep, "rb")) +tissues, expr = eeg._read_rnaseq(eeg.RNASEQ_FILE) +gs = eeg.gene_scores_from_expression(expr[a.cell], eeg.EXPRESSION_THRESHOLD) +rs = eeg.score_reactions_from_genes(prep.ref_model, gs) + +records = [] +orig = FT._solve_step + + +def wrapped(*args, **kw): + t = time.time() + res = orig(*args, **kw) + records.append( + { + "step": len(records) + 1, + "objective": res.objective, + "status": res.status, + "achieved_gap": res.achieved_gap, + "n_on": len(res.on_reactions), + "n_kept_min": len(res.kept_reactions), + "seconds": round(time.time() - t, 1), + } + ) + print("STEPREC", json.dumps(records[-1]), flush=True) + return res + + +FT._solve_step = wrapped + +results = [] +for rep in range(a.repeat): + records.clear() + steps = get_init_steps("1+1")[: a.nsteps] + t0 = time.time() + with warnings.catch_warnings(record=True) as W: + warnings.simplefilter("always") + m = ftinit( + prep, + rxn_scores=rs, + gene_scores=gs, + steps=steps, + fill_gaps=bool(a.fill_gaps), + big_m=eeg.BIG_M, + mip_gap_abs=eeg.MIP_GAP_ABS, + time_limit=a.tl, + resolve_ties=bool(a.rt), + prove_abs_gap=a.pag, + seed=a.seed, + threads=a.threads, + verbose=True, + ) + wall = time.time() - t0 + out = { + "args": vars(a), + "rep": rep, + "wall_s": round(wall, 1), + "steps": list(records), + "warnings": [str(w.message)[:160] for w in W], + "reactions": sorted(r.id for r in m.reactions), + "genes": sorted(g.id for g in m.genes), + } + try: + out["growth"] = float(m.slim_optimize()) + except Exception as e: # noqa + out["growth"] = None + print("TIMING build", round(wall), "s rxns", len(m.reactions), "warnings", len(W), flush=True) + if a.essential: + from cobra.flux_analysis import find_essential_genes + + t1 = time.time() + ess = find_essential_genes(m, processes=1) + out["essential"] = sorted(g.id for g in ess) + out["essential_s"] = round(time.time() - t1) + print("essential", len(ess), flush=True) + results.append(out) + json.dump(results, open(a.out, "w")) diff --git a/scripts/parameter_tuning/init/det/det_extra.py b/scripts/parameter_tuning/init/det/det_extra.py new file mode 100644 index 0000000..8d202e3 --- /dev/null +++ b/scripts/parameter_tuning/init/det/det_extra.py @@ -0,0 +1,311 @@ +#!/usr/bin/env python3 +"""Figures of the determinism study that analyse.py does not print. + +From the build JSON files (): the template-edit table (E builds against the +unedited A builds), the edit-list reactions kept by the unedited builds, the pooled +first-step probe (objective values, distinct sets and pairwise ranges per objective, +Threads 1 against 4 at one seed), the distances of the sweep's final sets, the +build-time ratios. From the FTINIT_DEBUG logs (): the phase statuses of the +tie-broken solves per prove_abs_gap value and, per full build, whether it adopted a +phase at time_limit, had a primary solve at time_limit or dropped a tie-break (phase +2a or 2b without incumbent). + +Command line: python det_extra.py [--json det_extra.json] +""" + +import argparse +import itertools +import json +import os +import re +import statistics as st + +PROVEN = 45899.11297453767 + + +def load(o, n): + p = os.path.join(o, n + ".json") + return json.load(open(p)) if os.path.exists(p) else None + + +def sd(a, b): + return len(set(a) ^ set(b)) + + +def builds_in_log(path): + """Split an FTINIT_DEBUG log into builds, each a list of solve blocks.""" + builds, cur, solve = [], None, None + for line in open(path, errors="replace"): + line = line.strip() + m = re.match(r"\[ftinit\] step (\d+)/(\d+)", line) + if m and m.group(1) == "1": + cur = [] + builds.append(cur) + if line.startswith("[ftinit] final solve:"): + solve = { + "step_line": line, + "primary": re.search(r"status=(\w+)", line).group(1), + "phases": {}, + "skipped": False, + } + if cur is not None: + cur.append(solve) + elif line.startswith("[ftinit] tie-break skipped") and solve is not None: + solve["skipped"] = True + else: + m = re.match(r"\[ftinit\] tie-break (phase\w+)-\w+: status=(\w+) obj=(\S+)", line) + if m and solve is not None: + solve["phases"][m.group(1)] = (m.group(2), m.group(3)) + return builds + + +def classify(build): + adopted = primary_tl = dropped = False + for s in build: + if s["primary"] == "time_limit": + primary_tl = True + ph = s["phases"] + drop = any(ph.get(p, ("", ""))[1] == "inf" for p in ("phase2a", "phase2b")) + if drop: + dropped = True + for p, (stt, obj) in ph.items(): + if stt == "time_limit" and obj != "inf" and not (drop and p in ("phase2a", "phase2b")): + adopted = True + return dict( + adopted_phase_at_time_limit=adopted, + primary_at_time_limit=primary_tl, + tie_break_dropped=dropped, + ) + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("out") + ap.add_argument("logs") + ap.add_argument("--json") + a = ap.parse_args() + O, L, R = a.out, a.logs, {} # noqa: E741 + + # template edit + print("== Template edit (seed 1234)") + edit = ( + json.load(open(os.path.join(O, "edit-list.json"))) + if os.path.exists(os.path.join(O, "edit-list.json")) + else [] + ) + R["edit"] = {} + for cfg, ref in ( + ("base", "A-base-1234-s1234"), + ("rt", "A-rt-1234-s1234"), + ("rtp1", "A-rtp1-1234-s1234"), + ): + e, u = load(O, f"E-{cfg}-s1234"), load(O, ref) + if not (e and u): + continue + e, u = e[0], u[0] + gains = len(set(e["essential"]) - set(u["essential"])) + d = dict( + rxn_diff=sd(e["reactions"], u["reactions"]), + gene_diff=sd(e["essential"], u["essential"]), + gene_gains=gains, + ess_unedited=len(u["essential"]), + ess_edited=len(e["essential"]), + wall_edited=e["wall_s"], + n_edited=len(e["reactions"]), + n_unedited=len(u["reactions"]), + edit_reactions_kept_by_unedited=len(set(edit) & set(u["reactions"])), + ) + R["edit"][cfg] = d + print(cfg, d) + rtp1 = load(O, "A-rtp1-1234-s1234") + if rtp1: + print( + "A-rtp1-1234: reactions", + len(rtp1[0]["reactions"]), + "essential", + len(rtp1[0]["essential"]), + ) + print("edit list size", len(edit)) + + # full-pipeline extras + print("== Full pipeline extras") + A = { + n: load(O, n) + for n in ( + "A-base-1234-s1234", + "A-base-7-s7", + "A-rt-1234-s1234", + "A-rt-7-s7", + "A-rtp1-1234-s1234", + "A-rtp1-7-s7", + "A-rtp005-1234-s1234", + "A-rtp005-7-s7", + ) + } + for n, v in A.items(): + if v: + x = v[0] + print( + n, + "nrxn", + len(x["reactions"]), + "growth", + round(x["growth"], 1) if x["growth"] is not None else None, + "step objs", + [round(s["objective"], 3) for s in x["steps"]], + "wall", + x["wall_s"], + ) + b = [A["A-base-1234-s1234"][0]["wall_s"], A["A-base-7-s7"][0]["wall_s"]] + r = [A["A-rt-1234-s1234"][0]["wall_s"], A["A-rt-7-s7"][0]["wall_s"]] + p = [A["A-rtp1-1234-s1234"][0]["wall_s"], A["A-rtp1-7-s7"][0]["wall_s"]] + R["time_ratio"] = dict(rt=st.mean(r) / st.mean(b), rtp1=st.mean(p) / st.mean(b)) + print( + "build-time ratio vs baseline (means of seeds 1234 and 7): rt %.2f, rtp1 %.2f" # noqa: UP031 + % (R["time_ratio"]["rt"], R["time_ratio"]["rtp1"]) + ) + + # first-step probe pooled + print("== First-step probe, pooled") + runs = {} + for cfg in ("base-t1", "base-t4", "rt-t1", "rt-t4"): + for s in (1234, 7, 1, 2, 3, 4): + v = load(O, f"B-{cfg}-s{s}") + if v: + runs[(cfg, s)] = v[0] + R["probe"] = {} + for setting in ("base", "rt"): + rr = {k: v for k, v in runs.items() if k[0].startswith(setting + "-")} + objs = {k: round(v["steps"][0]["objective"], 3) for k, v in rr.items()} + sets = {k: frozenset(v["reactions"]) for k, v in rr.items()} + byobj = {} + for k, o in objs.items(): + byobj.setdefault(o, []).append(k) + info = {} + for o, ks in sorted(byobj.items()): + pw = [len(sets[x] ^ sets[y]) for x, y in itertools.combinations(ks, 2)] + info[o] = dict( + n=len(ks), + distinct=len({sets[k] for k in ks}), + pair_min=min(pw) if pw else None, + pair_max=max(pw) if pw else None, + ) + same_seed = { + s: dict( + diff=len(sets[(f"{setting}-t1", s)] ^ sets[(f"{setting}-t4", s)]), + same_obj=objs[(f"{setting}-t1", s)] == objs[(f"{setting}-t4", s)], + ) + for s in (1234, 7, 1, 2, 3, 4) + if (f"{setting}-t1", s) in sets and (f"{setting}-t4", s) in sets + } + walls = { + t: [ + rr[(f"{setting}-{t}", s)]["wall_s"] + for s in (1234, 7, 1, 2, 3, 4) + if (f"{setting}-{t}", s) in rr + ] + for t in ("t1", "t4") + } + R["probe"][setting] = dict( + n=len(rr), + distinct=len(set(sets.values())), + objectives=sorted(set(objs.values())), + by_objective=info, + t1_vs_t4_same_seed=same_seed, + wall_range={t: (min(w), max(w)) for t, w in walls.items() if w}, + ) + print(setting, json.dumps(R["probe"][setting], default=str)) + + # sweep final sets + print("== Sweep final sets and times") + R["sweep"] = {} + for g in ("none", "5.0", "2.0", "1.0", "0.5", "0.1", "0.05"): + rr = [load(O, f"C-{g}-{s}-s{s}") for s in (1234, 7, 1, 2)] + rr = [x[0] for x in rr if x] + if not rr: + continue + sets = [frozenset(x["reactions"]) for x in rr] + pw = [len(x ^ y) for x, y in itertools.combinations(sets, 2)] + d = dict( + distinct=len(set(sets)), + pair_min=min(pw), + pair_max=max(pw), + step1_below_proven=sorted(round(PROVEN - x["steps"][0]["objective"], 1) for x in rr), + step1_s=( + min(x["steps"][0]["seconds"] for x in rr), + max(x["steps"][0]["seconds"] for x in rr), + ), + step2_s=( + min(x["steps"][1]["seconds"] for x in rr), + max(x["steps"][1]["seconds"] for x in rr), + ), + wall=(min(x["wall_s"] for x in rr), max(x["wall_s"] for x in rr)), + wall_mean=round(st.mean(x["wall_s"] for x in rr), 1), + statuses=sorted({s["status"] for x in rr for s in x["steps"]}), + with_warning=sum(1 for x in rr if x["warnings"]), + ) + R["sweep"][g] = d + print(g, json.dumps(d)) + s10 = [x for x in (load(O, f"C-1.0-{s}-s{s}") for s in (1234, 7, 1, 2)) if x] + if s10: + sh = [x[0]["steps"][0]["seconds"] / x[0]["wall_s"] for x in s10] + print("pag=1.0 step-1 share of build per run: %.3f to %.3f" % (min(sh), max(sh))) # noqa: UP031 + R["sweep_step1_share_1.0"] = (min(sh), max(sh)) + + # phase statuses of the sweep logs + print("== Phase statuses of the tie-broken solves in the sweep logs") + R["phases"] = {} + for g in ("5.0", "2.0", "1.0", "0.5", "0.1", "0.05", "none"): + cnt = {} + for s in (1234, 7, 1, 2): + f = os.path.join(L, f"C-{g}-{s}.log") + if not os.path.exists(f): + continue + for bld in builds_in_log(f): + for _k, solve in enumerate(bld): + for ph, (stt, obj) in solve["phases"].items(): + key = f"{ph} {stt}" + (" no-incumbent" if obj == "inf" else "") + cnt[key] = cnt.get(key, 0) + 1 + R["phases"][g] = dict(sorted(cnt.items())) + print(g, dict(sorted(cnt.items()))) + + # build counts of the full builds + print("== Full builds: adopted phase / primary at time_limit / dropped tie-break") + R["build_counts"] = {} + logs = [ + ("baseline", "A-base-1234.log"), + ("baseline", "A-base-7.log"), + ("baseline", "A-base-rep.log"), + ("rt", "A-rt-1234.log"), + ("rt", "A-rt-7.log"), + ("rt", "A-rt-rep.log"), + ("pag", "A-rtp1-1234.log"), + ("pag", "A-rtp1-7.log"), + ("pag", "A-rtp005-1234.log"), + ("pag", "A-rtp005-7.log"), + ] + per = [] + for setting, f in logs: + p = os.path.join(L, f) + if os.path.exists(p): + for i, bld in enumerate(builds_in_log(p)): + per.append((setting, f"{f}#{i + 1}", classify(bld))) + ep = os.path.join(L, "edit.log") + if os.path.exists(ep): + for setting, bld in zip(("baseline", "rt", "pag"), builds_in_log(ep), strict=False): + per.append((setting, f"edit.log ({setting})", classify(bld))) + for setting, name, c in per: + print(f" {setting:8s} {name:24s} {c}") + for setting in ("baseline", "rt", "pag"): + cs = [c for s_, _, c in per if s_ == setting] + R["build_counts"][setting] = ( + dict(builds=len(cs), **{k: sum(c[k] for c in cs) for k in cs[0]}) if cs else {} + ) + print(setting, R["build_counts"][setting]) + R["build_counts_per_build"] = [dict(setting=s_, build=n, **c) for s_, n, c in per] + if a.json: + json.dump(R, open(a.json, "w"), indent=1, default=str) + + +if __name__ == "__main__": + main() diff --git a/scripts/parameter_tuning/init/det/edit.sbatch b/scripts/parameter_tuning/init/det/edit.sbatch new file mode 100644 index 0000000..086e2e6 --- /dev/null +++ b/scripts/parameter_tuning/init/det/edit.sbatch @@ -0,0 +1,15 @@ +#!/bin/bash +#SBATCH -c 2 +#SBATCH --mem=24G +#SBATCH -t 12:00:00 +# Template-edit arm: select 30 unused reactions, prepare the edited template, build +# E-base, E-rt and E-rtp1 at seed 1234. Needs $WORK/out/A-rtp1-1234-s1234.json. +# Usage: sbatch -A -J det-edit -o $WORK/logs/edit.log $DET_SCRIPTS/edit.sbatch +source "$DET_SCRIPTS/env.sh" +export FTINIT_DEBUG=1 +$PY "$DET_SCRIPTS/select_edit.py" "$P/prep-base.pkl" "$WORK/out/A-rtp1-1234-s1234.json" "$WORK/out/edit-list.json" +$PY "$DET_SCRIPTS/prep.py" --model "$HGT/model/Human-GEM.yml" --remove "$WORK/out/edit-list.json" --out "$P/prep-edit.pkl" 2>&1 | grep -v "^Read LP\|Reading time\|^: " +for cfg in "E-base --rt 0" "E-rt --rt 1" "E-rtp1 --rt 1 --pag 1.0"; do + set -- $cfg; n=$1; shift + $PY "$DET_SCRIPTS/build.py" --prep "$P/prep-edit.pkl" --seed 1234 --out "$WORK/out/$n-s1234.json" --essential 1 "$@" 2>&1 | grep -v "^Set parameter\|^Read LP\|Reading time\|^: \|^Warning for adding" +done diff --git a/scripts/parameter_tuning/init/det/env.sh b/scripts/parameter_tuning/init/det/env.sh new file mode 100644 index 0000000..311ff71 --- /dev/null +++ b/scripts/parameter_tuning/init/det/env.sh @@ -0,0 +1,24 @@ +# Environment of the ftINIT determinism jobs; sourced by prep.sbatch, job.sbatch and edit.sbatch. +# Required before sbatch (exported into the job): +# WORK work folder: $WORK/out (build JSON), $WORK/logs +# PREP_DIR folder for prep-base.pkl and prep-edit.pkl, on a file system all nodes see +# (default $WORK/prep) +# TOOLBOX_ROOT raven-toolbox checkout (commit 94f3940) +# HUMANGEM_ROOT Human-GEM git clone; HUMANGEM_COMMIT (default 1ea27d8b = v2.0.0-38) is +# exported into $TMPDIR/hg, whose code/test goes on PYTHONPATH +# DET_SCRIPTS folder of these scripts +# PY Python interpreter (default python3) +module purge +module load SciPy-bundle/2026.05-gfbf-2026.1 +export GUROBI_HOME=${GUROBI_HOME:?set GUROBI_HOME} +export GRB_LICENSE_FILE=$GUROBI_HOME/gurobi.lic +export LD_LIBRARY_PATH=$GUROBI_HOME/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH} +export PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 +: "${WORK:?set WORK}" "${TOOLBOX_ROOT:?set TOOLBOX_ROOT}" "${HUMANGEM_ROOT:?set HUMANGEM_ROOT}" "${DET_SCRIPTS:?set DET_SCRIPTS}" +HUMANGEM_COMMIT=${HUMANGEM_COMMIT:-1ea27d8b} +P=${PREP_DIR:-$WORK/prep} +PY=${PY:-python3} +HGT=$TMPDIR/hg +mkdir -p "$HGT" "$P" "$WORK/out" "$WORK/logs" +git -C "$HUMANGEM_ROOT" archive "$HUMANGEM_COMMIT" model/Human-GEM.yml model/reactions.tsv data/datasets/Hart2015_RNAseq.txt data/metabolicTasks code/test | tar -x -C "$HGT" +export PYTHONPATH=$TOOLBOX_ROOT/src:$HGT/code/test diff --git a/scripts/parameter_tuning/init/det/job.sbatch b/scripts/parameter_tuning/init/det/job.sbatch new file mode 100644 index 0000000..e4038b1 --- /dev/null +++ b/scripts/parameter_tuning/init/det/job.sbatch @@ -0,0 +1,12 @@ +#!/bin/bash +#SBATCH --mem=24G +#SBATCH -t 10:00:00 +# One or more determinism builds (build.py), one per seed, in sequence. +# Usage: sbatch -A -c N -J name -o log $DET_SCRIPTS/job.sbatch NAME "SEEDS" BUILD ARGS... +source "$DET_SCRIPTS/env.sh" +export FTINIT_DEBUG=1 +echo host $(hostname) $(lscpu | grep 'Model name' | sed 's/.*: *//') +NAME=$1; SEEDS=$2; shift 2 +for s in $SEEDS; do + $PY "$DET_SCRIPTS/build.py" --prep "$P/prep-base.pkl" --seed $s --out "$WORK/out/$NAME-s$s.json" "$@" 2>&1 | grep -v "^Set parameter\|^Read LP\|Reading time\|^: \|^Warning for adding" +done diff --git a/scripts/parameter_tuning/init/det/prep.py b/scripts/parameter_tuning/init/det/prep.py new file mode 100644 index 0000000..eef3343 --- /dev/null +++ b/scripts/parameter_tuning/init/det/prep.py @@ -0,0 +1,49 @@ +"""Build PrepData for the Human-GEM v2.0.0-38 export (optionally minus an edit list). + +Reads Human-GEM.yml, removes the reaction ids of --remove (JSON list), and runs +estimateEssentialGenes._prep_human_model_for_ftinit with the essential tasks. +Command line: python prep.py --model --out [--remove ] +""" + +import argparse +import json +import pickle +import sys +import time + +import cobra + +sys.setrecursionlimit(10000) +cobra.Configuration().processes = 1 +import estimateEssentialGenes as eeg # noqa: E402 + +from raven_toolbox.io import read_yaml_model # noqa: E402 +from raven_toolbox.tasks.tasklist import parse_task_list # noqa: E402 + +ap = argparse.ArgumentParser() +ap.add_argument("--model", required=True) +ap.add_argument("--out", required=True) +ap.add_argument("--remove") # json list of reaction ids to remove from the template +a = ap.parse_args() +t0 = time.time() +model = read_yaml_model(a.model) +print("model", len(model.reactions), len(model.genes), flush=True) +if a.remove: + ids = json.load(open(a.remove)) + model.remove_reactions(ids, remove_orphans=True) + print("removed", len(ids), flush=True) +tasks = parse_task_list(eeg.ESSENTIAL_TASKS) +prep = eeg._prep_human_model_for_ftinit(model, tasks) +print( + "ref", + len(prep.ref_model.reactions), + "merged", + len(prep.min_model.reactions), + "essential", + len(prep.essential_rxns), + "tasks", + len(prep.tasks), + flush=True, +) +pickle.dump(prep, open(a.out, "wb")) +print("TIMING prep", round(time.time() - t0), "s") diff --git a/scripts/parameter_tuning/init/det/prep.sbatch b/scripts/parameter_tuning/init/det/prep.sbatch new file mode 100644 index 0000000..ee1b48a --- /dev/null +++ b/scripts/parameter_tuning/init/det/prep.sbatch @@ -0,0 +1,9 @@ +#!/bin/bash +#SBATCH -c 2 +#SBATCH --mem=24G +#SBATCH -t 04:00:00 +# Base template preparation of the determinism study (about 2000 s). +# Usage: sbatch -A -J det-prep -o $WORK/logs/prep.log $DET_SCRIPTS/prep.sbatch +source "$DET_SCRIPTS/env.sh" +echo host $(hostname) +$PY "$DET_SCRIPTS/prep.py" --model "$HGT/model/Human-GEM.yml" --out "$P/prep-base.pkl" diff --git a/scripts/parameter_tuning/init/det/select_edit.py b/scripts/parameter_tuning/init/det/select_edit.py new file mode 100644 index 0000000..dd01274 --- /dev/null +++ b/scripts/parameter_tuning/init/det/select_edit.py @@ -0,0 +1,28 @@ +"""Draw the template edit: 30 reactions of the reference model not kept by a build. + +Candidates are prep.ref_model reactions absent from the first build in +and not task-essential; random.Random(0).shuffle, first 30, sorted. +Command line: python select_edit.py +""" + +import json +import pickle +import random +import sys + +import raven_toolbox.init # noqa + +prep = pickle.load(open(sys.argv[1], "rb")) +m0 = set(json.load(open(sys.argv[2]))[0]["reactions"]) +cands = sorted( + r.id + for r in prep.ref_model.reactions + if r.id not in m0 and r.id not in prep.essential_rxns and r.id.startswith("MAR0") is not None +) +cands = sorted( + r.id for r in prep.ref_model.reactions if r.id not in m0 and r.id not in prep.essential_rxns +) +random.Random(0).shuffle(cands) +pick = sorted(cands[:30]) +json.dump(pick, open(sys.argv[3], "w")) +print(len(cands), "candidates; picked", len(pick), pick) diff --git a/scripts/parameter_tuning/init/det/submit_all.sh b/scripts/parameter_tuning/init/det/submit_all.sh new file mode 100644 index 0000000..e44a37c --- /dev/null +++ b/scripts/parameter_tuning/init/det/submit_all.sh @@ -0,0 +1,29 @@ +#!/bin/bash +# Submit the A (full pipeline), B (first-step probe) and C (prove_abs_gap sweep) builds +# of the determinism study after the base preparation job. +# Usage: export WORK TOOLBOX_ROOT HUMANGEM_ROOT DET_SCRIPTS PY [PREP_DIR]; export SBATCH_ACCOUNT= +# bash submit_all.sh +# Then: sbatch edit.sbatch after A-rtp1-1234 has finished; python analyse.py $WORK/out +: "${WORK:?set WORK}" "${DET_SCRIPTS:?set DET_SCRIPTS}" "${SBATCH_ACCOUNT:?set SBATCH_ACCOUNT}" +mkdir -p "$WORK/logs" +DEP="--dependency=afterok:$1" +S() { n=$1; c=$2; seeds=$3; shift 3; sbatch $DEP -c $c -J det-$n -o $WORK/logs/$n.log $DET_SCRIPTS/job.sbatch $n "$seeds" "$@"; } +# A: full pipeline + gap fill + essential genes +for sd in 1234 7; do +S A-base-$sd 2 "$sd" --rt 0 --essential 1 +S A-rt-$sd 2 "$sd" --rt 1 --essential 1 +S A-rtp1-$sd 2 "$sd" --rt 1 --pag 1.0 --essential 1 +S A-rtp005-$sd 2 "$sd" --rt 1 --pag 0.05 --essential 1 +done +S A-base-rep 2 "1234" --rt 0 --essential 1 --repeat 2 +S A-rt-rep 2 "1234" --rt 1 --essential 1 --repeat 2 +# B: first-stage probe +S B-base-t1 2 "1234 7 1 2 3 4" --rt 0 --nsteps 1 --fill-gaps 0 +S B-rt-t1 2 "1234 7 1 2 3 4" --rt 1 --nsteps 1 --fill-gaps 0 +S B-base-t4 4 "1234 7 1 2 3 4" --rt 0 --nsteps 1 --fill-gaps 0 --threads 4 +S B-rt-t4 4 "1234 7 1 2 3 4" --rt 1 --nsteps 1 --fill-gaps 0 --threads 4 +# C: prove_abs_gap sweep (resolve_ties on), both stages, no gap fill +for sd in 1234 7 1 2; do +S C-none-$sd 2 "$sd" --rt 1 --fill-gaps 0 +for g in 5.0 2.0 1.0 0.5 0.1 0.05; do S C-$g-$sd 2 "$sd" --rt 1 --pag $g --fill-gaps 0; done +done diff --git a/scripts/parameter_tuning/init/det/ties.py b/scripts/parameter_tuning/init/det/ties.py new file mode 100644 index 0000000..bf5d123 --- /dev/null +++ b/scripts/parameter_tuning/init/det/ties.py @@ -0,0 +1,55 @@ +"""Tied-score blocks of the step-1 merged scores per Hart2015 cell line. + +Loads the base PrepData, scores every Hart2015 cell line as estimateEssentialGenes.py +does, applies ftinit's private _nudge_scores and group_rxn_scores with the step-1 +ignore mask, and prints the negative-score count, the share in tied blocks and the +three largest blocks. Needs the Human-GEM 1ea27d8b code/test on PYTHONPATH (env.sh). +Command line: python ties.py +""" + +import collections +import pickle +import sys + +import raven_toolbox.init # noqa +from raven_toolbox.init import get_init_steps +from raven_toolbox.init.merge import group_rxn_scores + +FT = sys.modules["raven_toolbox.init.ftinit"] +import estimateEssentialGenes as eeg # noqa: E402 + +prep = pickle.load(open(sys.argv[1], "rb")) +tissues, expr = eeg._read_rnaseq(eeg.RNASEQ_FILE) +step = get_init_steps("1+1")[0] +blocks = {} +for t in tissues: + gs = eeg.gene_scores_from_expression(expr[t], eeg.EXPRESSION_THRESHOLD) + rs = FT._nudge_scores(eeg.score_reactions_from_genes(prep.ref_model, gs)) + sc = group_rxn_scores( + prep.min_model, rs, prep.orig_rxn_ids, prep.group_ids, prep.masks.ignored(step.ignore_mask) + ) + neg = {r: s for r, s in sc.items() if s < 0} + c = collections.Counter(round(s, 6) for s in neg.values()) + top = c.most_common(3) + inblock = sum(v for k, v in c.items() if v > 1) + nogpr = sum(1 for r in prep.ref_model.reactions if not r.gene_reaction_rule) + print( + t, + "negative", + len(neg), + "in tied blocks", + inblock, + round(100 * inblock / len(neg), 1), + "top", + top, + flush=True, + ) + blocks[t] = top[0] +print( + "GPRless ref reactions", + nogpr, + "ref", + len(prep.ref_model.reactions), + "merged", + len(prep.min_model.reactions), +) diff --git a/scripts/parameter_tuning/kegg/README.md b/scripts/parameter_tuning/kegg/README.md new file mode 100644 index 0000000..2ed068b --- /dev/null +++ b/scripts/parameter_tuning/kegg/README.md @@ -0,0 +1,50 @@ +# KEGG HMM cut-off calibration + +Scripts for the KEGG HMM cut-off page of the parameter-tuning documentation. The gene-to-KO assignment of the `kegg118` HMM libraries is scored against the gene-to-KO and KO-to-reaction tables of the `kegg118` release for *S. cerevisiae* (`sce`), *C. merolae* (`cme`), *E. coli* (`eco`) and *M. genitalium* (`mge`), over E-value, `min_score_ratio_ko` and `min_score_ratio_g`. The annotation path and the HMM path are compared for whole models. + +## Inputs and environment + +The reference tables and HMM libraries are the `kegg118` release of [CompMetEng/toolbox-assets](https://github.com/CompMetEng/toolbox-assets/releases/tag/kegg118). `ensure_kegg_data()` and `ensure_kegg_hmm_library("eukaryotes" | "prokaryotes")` fetch them and check the SHA-256. The libraries hold 2698 (eukaryotes, used for `sce` and `cme`) and 4205 (prokaryotes, used for `eco` and `mge`) profiles. + +`fetch.sbatch` downloads the proteomes from the KEGG REST API (`list/` and `get//aaseq`) and UniProt, and packs them into `kegg_proteomes_.tar.gz`. It yields 6021 (`sce`), 5010 (`cme`), 4288 (`eco`) and 476 (`mge`) KEGG sequences. The UniProt proteomes are downloaded and not searched. `run.sbatch` packs the `hmmsearch --tblout` files into `kegg_hmmsearch_tblout_.tar.gz`, which `rescore.sbatch` and `timing.sbatch` read. + +| Variable | Meaning | +| --- | --- | +| `STUDY` | this folder | +| `TOOLBOX_ROOT` | raven-toolbox checkout (`94f3940`); its `src` goes on `PYTHONPATH` | +| `PY` | Python interpreter with cobra, optlang, pandas and numpy | +| `DATA` | folder of the two archives | +| `BINARIES_CACHE` | `raven_toolbox/binaries` cache with the `hmmsearch` 3.4 bundle | + +The module names in the job files are site-specific; other systems need an equivalent environment. + +## Commands + +```bash +export STUDY=... TOOLBOX_ROOT=... PY=... DATA=... BINARIES_CACHE=... +sbatch -A $STUDY/jobs/fetch.sbatch +sbatch -A $STUDY/jobs/run.sbatch # 16 cores, 48 GB +sbatch -A $STUDY/jobs/timing.sbatch # exclusive node, 16 cores +sbatch -A $STUDY/jobs/rescore.sbatch +``` + +For each organism `run.sbatch` calls the following, and `compare_kegg_paths.py` for `sce` and `mge`. + +```bash +python analyze_hmm_cutoffs.py --artefacts $ART --org --proteome kegg_rest/.faa \ + --library --threads 16 --save-tblout .tbl --json results/sweep_.json +python compare_kegg_paths.py --org --proteome kegg_rest/.faa --artefacts $ART \ + --library --domain --threads 16 --settings 1e-50,0.3,0.8 1e-30,0.3,0.9 +``` + +`rescore.sbatch` scores the saved hit tables again with `--tblout`, which gives identical sweeps, and with `--restrict-to-reference`. `timing.sbatch` runs `check_true_reactions.py` and `time_hmmsearch.py --threads 16 12 4 1`. + +## Outputs + +| File | Content | +| --- | --- | +| `analyze_hmm_cutoffs.py` | E-value percentiles of matched and novel hits, cut-off sweep at (0.3, 0.8) and (0.3, 0.9), score-ratio sweep and defaults row, as printed tables and JSON. By default every `ko_reaction` link counts as a true reaction; `--restrict-to-reference` counts links to reactions of the reference model. | +| `results/sweep_.json`, `results/sweep__refrxn.json` | Sweep output per organism, without and with `--restrict-to-reference`. | +| `compare_kegg_paths.py` to `results/wholemodel_.json` | Reactions, metabolites and genes of the two paths at each setting, and their overlap. | +| `check_true_reactions.py` to `results/true_reactions.json` | True reactions per organism under several definitions. | +| `time_hmmsearch.py` to `results/timing_sce_eukaryotes.json` | `hmmsearch` wall time at the given thread counts and a comparison of the parsed hit tables. | diff --git a/scripts/parameter_tuning/kegg/analyze_hmm_cutoffs.py b/scripts/parameter_tuning/kegg/analyze_hmm_cutoffs.py new file mode 100644 index 0000000..0877bc7 --- /dev/null +++ b/scripts/parameter_tuning/kegg/analyze_hmm_cutoffs.py @@ -0,0 +1,226 @@ +#!/usr/bin/env python3 +"""Cut-off sensitivity for the KEGG HMM query path. + +Cross-validates ``assign_kos`` against an organism's *real* KEGG gene→KO +annotation (from the ``organism_gene_ko`` table) and sweeps the E-value cut-off +and the two score-ratio filters. Produces the tables of +``docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md``. + +Usage +----- + python analyze_hmm_cutoffs.py \ + --artefacts \ + --proteome .faa \ + --org sce --library kegg118_eukaryotes.hmm --threads 16 + +``--proteome`` is the organism's protein FASTA (headers ``>org:gene ...``). +``--tblout`` may be given instead of ``--library`` to reuse a cached +``hmmsearch --tblout`` file (written by ``--save-tblout``). ``hmmsearch`` is +resolved by the toolbox (``RAVEN_PYTHON_HMMSEARCH``, PATH or the bundled binary) +when ``--library`` is used. ``--artefacts`` may hold the tables with or without a +version prefix (``kegg118_ko_reaction.tsv.gz``). ``--json`` writes all printed +numbers. ``--restrict-to-reference`` keeps only the ``ko_reaction`` links to +reactions of the reference model when the true and predicted reactions are counted. + +The cut-off sweep is printed twice: at (min_score_ratio_ko, min_score_ratio_g) = +(0.3, 0.8), the RAVEN 2.11.3 ratios, and at (0.3, 0.9), the defaults. + +Caveat: organisms present in the library's training set give an upper bound on +recall; the comparison is relative. +""" + +from __future__ import annotations + +import argparse +import json +import time +from pathlib import Path + +import numpy as np +import pandas as pd + +from raven_toolbox.reconstruction.kegg.parse import _resolve_artefact, read_kegg_table +from raven_toolbox.reconstruction.kegg.query import ( + assign_kos, + parse_hmmsearch_tblout, + run_hmmsearch, +) + +CUTOFFS = (1e-10, 1e-20, 1e-30, 1e-50, 1e-70, 1e-100) +KO_RATIOS = (0.0, 0.3, 0.5) +G_RATIOS = (0.5, 0.8, 0.95) +SWEEP_RATIOS = ((0.3, 0.8), (0.3, 0.9)) + + +def load_ko2rxn(artefacts: Path, restrict_to_reference: bool = False) -> dict[str, set[str]]: + tbl = read_kegg_table(_resolve_artefact(artefacts, "ko_reaction.tsv.gz")) + if restrict_to_reference: + from raven_toolbox.io.yaml import read_yaml_model + + ref = read_yaml_model(_resolve_artefact(artefacts, "reference_model.yml.gz")) + tbl = tbl[tbl["reaction"].isin({r.id for r in ref.reactions})] + ko2rxn: dict[str, set[str]] = {} + for ko, rxn in zip(tbl["ko"], tbl["reaction"], strict=True): + ko2rxn.setdefault(ko, set()).add(rxn) + return ko2rxn + + +def ground_truth(artefacts: Path, org: str, ko2rxn) -> tuple[set, set]: + ogk = read_kegg_table(_resolve_artefact(artefacts, "organism_gene_ko.tsv.gz")) + rows = ogk[ogk["organism"].str.lower() == org] + pairs = set(zip(rows["gene"], rows["ko"], strict=True)) + rxns = {r for _, ko in pairs for r in ko2rxn.get(ko, ())} + return pairs, rxns + + +def predicted_pairs(hits: pd.DataFrame, **kw) -> set: + out = set() + for ko, genes in assign_kos(hits, **kw).items(): + for g in genes: + out.add((g.split(":", 1)[1] if ":" in g else g, ko)) + return out + + +def prf(pred: set, truth: set) -> tuple[float, float, float]: + tp = len(pred & truth) + rec = tp / len(truth) if truth else 0.0 + prec = tp / len(pred) if pred else 0.0 + f1 = 2 * prec * rec / (prec + rec) if prec + rec else 0.0 + return prec, rec, f1 + + +def main(argv=None) -> None: + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument("--artefacts", type=Path, required=True) + ap.add_argument("--org", required=True, help="KEGG organism code, e.g. sce") + ap.add_argument("--proteome", type=Path, help="protein FASTA (headers >org:gene)") + ap.add_argument("--library", type=Path, help="HMM library flatfile for hmmsearch") + ap.add_argument("--tblout", type=Path, help="cached hmmsearch --tblout (skip hmmsearch)") + ap.add_argument("--save-tblout", type=Path, help="write the hmmsearch --tblout text here") + ap.add_argument("--json", type=Path, help="write the results as JSON here") + ap.add_argument("--threads", type=int, default=4) + ap.add_argument( + "--restrict-to-reference", + action="store_true", + help="keep only ko_reaction links to reactions of the reference model", + ) + args = ap.parse_args(argv) + result: dict = {"org": args.org.lower()} + + if args.tblout: + text = args.tblout.read_text() + elif args.library and args.proteome: + t0 = time.perf_counter() + text = run_hmmsearch(args.proteome, args.library, threads=args.threads) + secs = time.perf_counter() - t0 + result["search"] = { + "threads": args.threads, + "seconds": secs, + "library": str(args.library), + "proteome": str(args.proteome), + } + print(f"hmmsearch: {secs:.1f} s on {args.threads} threads") + else: + ap.error("give --tblout, or --library and --proteome") + if args.save_tblout: + args.save_tblout.write_text(text) + + org = args.org.lower() + hits = parse_hmmsearch_tblout(text) + hits = hits[hits["gene"].str.startswith(f"{org}:")].reset_index(drop=True) + ko2rxn = load_ko2rxn(args.artefacts, args.restrict_to_reference) + result["restrict_to_reference"] = args.restrict_to_reference + gt_pairs, gt_rxns = ground_truth(args.artefacts, org, ko2rxn) + if args.proteome: + result["proteome_seqs"] = sum( + 1 for line in args.proteome.read_text().splitlines() if line.startswith(">") + ) + result.update( + { + "hit_rows": len(hits), + "genes_with_hits": int(hits["gene"].nunique()), + "true_pairs": len(gt_pairs), + "true_reactions": len(gt_rxns), + } + ) + + print( + f"\n{'=' * 70}\n{org}: {hits['gene'].nunique()} query genes with hits, " + f"{len(gt_pairs)} true gene->KO pairs, {len(gt_rxns)} true reactions\n{'=' * 70}" + ) + + best: dict[tuple, float] = {} + for ko, gene, e in zip(hits["ko"], hits["gene"], hits["evalue"], strict=True): + key = (gene.split(":", 1)[1], ko) + if key not in best or e < best[key]: + best[key] = e + matched = np.array([e for k, e in best.items() if k in gt_pairs]) + novel = np.array([e for k, e in best.items() if k not in gt_pairs]) + + def logq(arr, q): + if not len(arr): + return float("nan") + v = np.quantile(arr, q) + return np.log10(v) if v > 0 else -300.0 + + print("\nlog10(E-value) percentiles [matched=in annotation, novel=not]:") + print(f" {'group':<8}{'n':>7}{'p50':>8}{'p90':>8}{'p95':>8}{'p99':>8}") + result["percentiles"] = {} + for name, arr in (("matched", matched), ("novel", novel)): + print( + f" {name:<8}{len(arr):>7}{logq(arr, 0.5):>8.0f}{logq(arr, 0.9):>8.0f}" + f"{logq(arr, 0.95):>8.0f}{logq(arr, 0.99):>8.0f}" + ) + result["percentiles"][name] = { + "n": len(arr), + **{f"p{int(q * 100)}": float(logq(arr, q)) for q in (0.5, 0.9, 0.95, 0.99)}, + } + + result["cutoff_sweep"] = [] + for rko, rg in SWEEP_RATIOS: + print(f"\ncutoff sweep (min_score_ratio_ko={rko}, min_score_ratio_g={rg}):") + print( + f" {'cutoff':>8}{'gKO_prec':>9}{'gKO_rec':>8}{'gKO_F1':>8}{'rxn_rec':>9}{'rxn_novel':>10}" + ) + for cutoff in CUTOFFS: + pred = predicted_pairs( + hits, cutoff=cutoff, min_score_ratio_ko=rko, min_score_ratio_g=rg + ) + prec, rec, f1 = prf(pred, gt_pairs) + pred_rxns = {r for _, ko in pred for r in ko2rxn.get(ko, ())} + rrec = len(pred_rxns & gt_rxns) / len(gt_rxns) if gt_rxns else 0.0 + print( + f" {cutoff:>8.0e}{prec:>9.2f}{rec:>8.2f}{f1:>8.2f}{rrec:>9.2f}" + f"{len(pred_rxns - gt_rxns):>10}" + ) + result["cutoff_sweep"].append( + { + "ko_ratio": rko, + "g_ratio": rg, + "cutoff": cutoff, + "pairs": len(pred), + "prec": prec, + "rec": rec, + "f1": f1, + "rxn_rec": rrec, + "rxn_novel": len(pred_rxns - gt_rxns), + } + ) + + print("\nratio sweep (cutoff=1e-50):") + print(f" {'ko_ratio':>9}{'g_ratio':>8}{'gKO_prec':>9}{'gKO_rec':>8}{'gKO_F1':>8}") + result["ratio_sweep"] = [] + for rko in KO_RATIOS: + for rg in G_RATIOS: + pred = predicted_pairs(hits, cutoff=1e-50, min_score_ratio_ko=rko, min_score_ratio_g=rg) + prec, rec, f1 = prf(pred, gt_pairs) + print(f" {rko:>9.1f}{rg:>8.2f}{prec:>9.2f}{rec:>8.2f}{f1:>8.2f}") + result["ratio_sweep"].append( + {"ko_ratio": rko, "g_ratio": rg, "prec": prec, "rec": rec, "f1": f1} + ) + if args.json: + args.json.write_text(json.dumps(result, indent=1)) + + +if __name__ == "__main__": + main() diff --git a/scripts/parameter_tuning/kegg/check_true_reactions.py b/scripts/parameter_tuning/kegg/check_true_reactions.py new file mode 100644 index 0000000..4bb1ccf --- /dev/null +++ b/scripts/parameter_tuning/kegg/check_true_reactions.py @@ -0,0 +1,79 @@ +#!/usr/bin/env python3 +"""Count each organism's "true reactions" of the KEGG artefacts under several definitions. + +``analyze_hmm_cutoffs.py`` defines the true reactions as every reaction that +``ko_reaction`` links to one of the organism's annotated KOs. This script prints that +count next to the counts restricted to reactions present in the reference model, and +further to reactions not flagged ``general`` in ``rxn_flags``, plus the numbers of +distinct KOs and gene→KO pairs. + +Example: + python check_true_reactions.py --artefacts $XDG_CACHE_HOME/raven_toolbox/data/kegg-kegg118 \ + --orgs sce cme eco mge --json true_reactions.json +""" + +from __future__ import annotations + +import argparse +import json +from pathlib import Path + +from raven_toolbox.io.yaml import read_yaml_model +from raven_toolbox.reconstruction.kegg.parse import _resolve_artefact, read_kegg_table + + +def main() -> None: + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument("--artefacts", type=Path, required=True) + ap.add_argument("--orgs", nargs="+", default=["sce", "cme", "eco", "mge"]) + ap.add_argument("--json", type=Path) + args = ap.parse_args() + a = args.artefacts + kr = read_kegg_table(_resolve_artefact(a, "ko_reaction.tsv.gz")) + ogk = read_kegg_table(_resolve_artefact(a, "organism_gene_ko.tsv.gz")) + flags = read_kegg_table(_resolve_artefact(a, "rxn_flags.tsv.gz")) + ref = read_yaml_model(_resolve_artefact(a, "reference_model.yml.gz")) + ref_rx = {r.id for r in ref.reactions} + print("rxn_flags columns:", list(flags.columns), "rows:", len(flags)) + general = set() + for col in flags.columns: + if "general" in col.lower(): + general = set( + flags.loc[flags[col].str.lower().isin(["1", "true", "yes"]), flags.columns[0]] + ) + if not general and "flag" in flags.columns: + general = set(flags.loc[flags["flag"].str.lower() == "general", flags.columns[0]]) + ko2rxn: dict[str, set[str]] = {} + for ko, rxn in zip(kr["ko"], kr["reaction"], strict=True): + ko2rxn.setdefault(ko, set()).add(rxn) + out = { + "ko_reaction_rows": len(kr), + "ko_reaction_kos": len(ko2rxn), + "ko_reaction_reactions": int(kr["reaction"].nunique()), + "reference_model_reactions": len(ref_rx), + "general_reactions": len(general), + "organisms": {}, + } + for org in args.orgs: + rows = ogk[ogk["organism"].str.lower() == org] + pairs = set(zip(rows["gene"], rows["ko"], strict=True)) + kos = {k for _, k in pairs} + rx = {r for k in kos for r in ko2rxn.get(k, ())} + d = { + "pairs": len(pairs), + "genes": int(rows["gene"].nunique()), + "kos": len(kos), + "kos_without_reaction": len(kos - set(ko2rxn)), + "true_reactions": len(rx), + "in_reference_model": len(rx & ref_rx), + "in_reference_not_general": len((rx & ref_rx) - general), + } + out["organisms"][org] = d + print(org, d, flush=True) + print({k: v for k, v in out.items() if k != "organisms"}) + if args.json: + args.json.write_text(json.dumps(out, indent=1)) + + +if __name__ == "__main__": + main() diff --git a/scripts/parameter_tuning/kegg/compare_kegg_paths.py b/scripts/parameter_tuning/kegg/compare_kegg_paths.py new file mode 100644 index 0000000..c1b835f --- /dev/null +++ b/scripts/parameter_tuning/kegg/compare_kegg_paths.py @@ -0,0 +1,121 @@ +#!/usr/bin/env python3 +"""Whole-model comparison of the two KEGG reconstruction paths for one organism. + +Builds the organism's draft model twice with raven-toolbox: + +- annotation path: ``get_kegg_model_for_organism_from_artefacts(org, artefacts)``, + genes and KOs from the ``organism_gene_ko`` table; +- HMM path: ``get_kegg_model_from_sequences_with_artefacts(proteome, artefacts, library, + cutoff, min_score_ratio_ko, min_score_ratio_g)``, KOs assigned by ``hmmsearch`` of the + KO library against the proteome. + +Both use the default keep flags (``keep_spontaneous``, ``keep_undefined_stoich``, +``keep_incomplete`` true, ``keep_general`` false). It reports the numbers of reactions, +metabolites and genes of each model, the shared reactions, metabolites and genes +(the ``org:`` prefix of the HMM-path gene ids is stripped) and their Jaccard indices, +and writes them as JSON. + +Inputs: ``--artefacts`` the directory of the KEGG core artefacts (version-prefixed +names allowed), ``--library`` the domain HMM library flatfile, ``--proteome`` a FASTA +with ``>org:gene`` headers. ``--settings`` takes one or more +``cutoff,ratio_ko,ratio_g`` triples; each runs one HMM-path build. + +Example: + python compare_kegg_paths.py --org sce --proteome sce.faa \ + --artefacts $XDG_CACHE_HOME/raven_toolbox/data/kegg-kegg118 \ + --library kegg118_eukaryotes.hmm --domain eukaryotes --threads 16 \ + --settings 1e-50,0.3,0.8 --json wholemodel_sce.json +""" + +from __future__ import annotations + +import argparse +import json +import time +from pathlib import Path + +from raven_toolbox.reconstruction.kegg.organism import get_kegg_model_for_organism_from_artefacts +from raven_toolbox.reconstruction.kegg.query import get_kegg_model_from_sequences_with_artefacts + + +def ids(model, strip=False): + rx = {r.id for r in model.reactions} + mets = {m.id for m in model.metabolites} + genes = {g.id.split(":", 1)[1] if strip and ":" in g.id else g.id for g in model.genes} + return rx, mets, genes + + +def jaccard(a, b): + return len(a & b) / len(a | b) if a | b else 0.0 + + +def main() -> None: + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument("--org", required=True) + ap.add_argument("--proteome", type=Path, required=True) + ap.add_argument("--artefacts", type=Path, required=True) + ap.add_argument("--library", type=Path, required=True) + ap.add_argument("--domain", default="eukaryotes") + ap.add_argument("--threads", type=int, default=4) + ap.add_argument("--settings", nargs="+", default=["1e-50,0.3,0.8"]) + ap.add_argument("--json", type=Path) + args = ap.parse_args() + + ann = get_kegg_model_for_organism_from_artefacts(args.org, args.artefacts) + a_rx, a_met, a_gene = ids(ann) + out = { + "org": args.org, + "annotation": {"reactions": len(a_rx), "metabolites": len(a_met), "genes": len(a_gene)}, + "hmm": [], + } + print( + f"{args.org} annotation path: {len(a_rx)} reactions, {len(a_met)} metabolites, " + f"{len(a_gene)} genes" + ) + for setting in args.settings: + cutoff, rko, rg = (float(x) for x in setting.split(",")) + t0 = time.perf_counter() + hmm = get_kegg_model_from_sequences_with_artefacts( + args.proteome, + args.artefacts, + args.library, + domain=args.domain, + model_id=args.org, + cutoff=cutoff, + min_score_ratio_ko=rko, + min_score_ratio_g=rg, + threads=args.threads, + ) + secs = time.perf_counter() - t0 + h_rx, h_met, h_gene = ids(hmm, strip=True) + row = { + "cutoff": cutoff, + "ko_ratio": rko, + "g_ratio": rg, + "seconds": secs, + "threads": args.threads, + "reactions": len(h_rx), + "metabolites": len(h_met), + "genes": len(h_gene), + "shared_reactions": len(a_rx & h_rx), + "shared_metabolites": len(a_met & h_met), + "shared_genes": len(a_gene & h_gene), + "jaccard_reactions": jaccard(a_rx, h_rx), + "jaccard_metabolites": jaccard(a_met, h_met), + "jaccard_genes": jaccard(a_gene, h_gene), + } + out["hmm"].append(row) + print( + f"{args.org} HMM path {setting}: {len(h_rx)} reactions, {len(h_met)} metabolites, " + f"{len(h_gene)} genes; shared reactions {row['shared_reactions']}/{len(a_rx)} " + f"(Jaccard {row['jaccard_reactions']:.2f}), shared genes " + f"{row['shared_genes']}/{len(a_gene)} (Jaccard {row['jaccard_genes']:.2f}); " + f"{secs:.0f} s", + flush=True, + ) + if args.json: + args.json.write_text(json.dumps(out, indent=1)) + + +if __name__ == "__main__": + main() diff --git a/scripts/parameter_tuning/kegg/fetch_proteomes.py b/scripts/parameter_tuning/kegg/fetch_proteomes.py new file mode 100644 index 0000000..d0a7160 --- /dev/null +++ b/scripts/parameter_tuning/kegg/fetch_proteomes.py @@ -0,0 +1,189 @@ +#!/usr/bin/env python3 +"""Fetch the proteomes of the KEGG HMM cut-off study from KEGG REST and UniProt. + +For each KEGG organism code (default sce, cme, eco, mge) this writes two protein +FASTA files with headers ``>org:gene``: + +- ``kegg_rest/.faa``: every gene of type CDS in ``https://rest.kegg.jp/list/``, + sequences from ``https://rest.kegg.jp/get/+...+/aaseq`` (10 ids per + request, one request at a time, at most about 3 requests per second); +- ``uniprot/.faa``: the UniProt reference proteome, relabelled to the KEGG + gene id from the UniProt KEGG cross-reference with the exact organism prefix + (entries without one are dropped), as ``homology_cutoff_kegg.py fetch`` does. + +It writes ``counts.json`` (sequence counts per source, request dates and URLs) and +packs everything into one ``.tar.gz`` given by ``--archive``. + +Command line used: + python -I fetch_proteomes.py --work $TMPDIR/prot \ + --archive $DATA/kegg_proteomes_2026-11-01.tar.gz +""" + +from __future__ import annotations + +import argparse +import datetime as dt +import json +import pathlib +import tarfile +import time +import urllib.error +import urllib.request + +UNIPROT = { + "sce": "UP000002311", + "cme": "UP000007014", + "eco": "UP000000625", + "mge": "UP000000807", +} +KEGG = "https://rest.kegg.jp" +_last = [0.0] + + +def get(url: str, *, pause: float = 0.35, tries: int = 5) -> str: + for attempt in range(tries): + wait = _last[0] + pause - time.time() + if wait > 0: + time.sleep(wait) + _last[0] = time.time() + try: + with urllib.request.urlopen(url, timeout=300) as fh: + return fh.read().decode("utf-8") + except (urllib.error.URLError, TimeoutError) as exc: + if attempt == tries - 1: + raise + print(f"retry {attempt + 1} for {url}: {exc}", flush=True) + time.sleep(5 * (attempt + 1)) + raise RuntimeError("unreachable") + + +def parse_fasta(text: str) -> dict[str, str]: + out, name, seq = {}, None, [] + for line in text.splitlines(): + if line.startswith(">"): + if name: + out[name] = "".join(seq) + name, seq = line[1:].split()[0], [] + elif name: + seq.append(line.strip()) + if name: + out[name] = "".join(seq) + return out + + +def write_fasta(path: pathlib.Path, seqs: dict[str, str]) -> None: + with open(path, "w", encoding="utf-8") as fh: + for name, seq in seqs.items(): + fh.write(f">{name}\n") + for i in range(0, len(seq), 60): + fh.write(seq[i : i + 60] + "\n") + + +def kegg_rest(org: str, out: pathlib.Path) -> dict: + listing = get(f"{KEGG}/list/{org}") + types: dict[str, int] = {} + cds = [] + for line in listing.splitlines(): + parts = line.split("\t") + if len(parts) < 2: + continue + types[parts[1]] = types.get(parts[1], 0) + 1 + if parts[1] == "CDS": + cds.append(parts[0] if ":" in parts[0] else f"{org}:{parts[0]}") + seqs: dict[str, str] = {} + for i in range(0, len(cds), 10): + chunk = cds[i : i + 10] + try: + text = get(f"{KEGG}/get/{'+'.join(chunk)}/aaseq") + except urllib.error.HTTPError as exc: # 404 if none of the ids has a sequence + print(f"{org}: HTTP {exc.code} for chunk {i}", flush=True) + continue + seqs.update(parse_fasta(text)) + if i % 500 == 0: + print(f"{org}: {len(seqs)}/{len(cds)}", flush=True) + write_fasta(out / f"{org}.faa", seqs) + return { + "list_entries": sum(types.values()), + "types": types, + "cds": len(cds), + "sequences": len(seqs), + "cds_without_sequence": len(set(cds) - set(seqs)), + } + + +def uniprot(org: str, out: pathlib.Path) -> dict: + upid = UNIPROT[org] + rows = get( + f"https://rest.uniprot.org/uniprotkb/stream?query=proteome:{upid}" + f"&format=tsv&fields=accession,xref_kegg", + pause=1, + ).splitlines()[1:] + mapping = {} + for line in rows: + parts = line.split("\t") + if len(parts) < 2: + continue + for entry in filter(None, (p.strip() for p in parts[1].split(";"))): + prefix, _, gene = entry.partition(":") + if prefix == org: + mapping[parts[0]] = f"{org}:{gene}" + break + fasta = get( + f"https://rest.uniprot.org/uniprotkb/stream?query=proteome:{upid}&format=fasta", pause=1 + ) + raw = parse_fasta(fasta) + seqs, dup = {}, 0 + for name, seq in raw.items(): + parts = name.split("|") + acc = parts[1] if len(parts) > 2 else parts[0] + gene = mapping.get(acc) + if gene is None: + continue + if gene in seqs: + dup += 1 + continue + seqs[gene] = seq + write_fasta(out / f"{org}.faa", seqs) + return { + "proteome": upid, + "entries": len(raw), + "mapped_unique_genes": len(seqs), + "duplicate_gene_ids": dup, + } + + +def main() -> None: + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument("--work", type=pathlib.Path, required=True) + ap.add_argument("--archive", type=pathlib.Path, required=True) + ap.add_argument("--orgs", nargs="+", default=["sce", "cme", "eco", "mge"]) + args = ap.parse_args() + (args.work / "kegg_rest").mkdir(parents=True, exist_ok=True) + (args.work / "uniprot").mkdir(parents=True, exist_ok=True) + info = { + "date_utc": dt.datetime.now(dt.UTC).isoformat(), + "kegg_info": get(f"{KEGG}/info/kegg").strip().splitlines(), + "urls": { + "kegg_list": f"{KEGG}/list/", + "kegg_aaseq": f"{KEGG}/get/+...(10 ids)/aaseq", + "uniprot_fasta": "https://rest.uniprot.org/uniprotkb/stream?query=proteome:&format=fasta", + "uniprot_xref": "https://rest.uniprot.org/uniprotkb/stream?query=proteome:&format=tsv&fields=accession,xref_kegg", + }, + "organisms": {}, + } + for org in args.orgs: + u = uniprot(org, args.work / "uniprot") + print(org, "uniprot", u, flush=True) + k = kegg_rest(org, args.work / "kegg_rest") + print(org, "kegg_rest", k, flush=True) + info["organisms"][org] = {"kegg_rest": k, "uniprot": u} + (args.work / "counts.json").write_text(json.dumps(info, indent=1)) + args.archive.parent.mkdir(parents=True, exist_ok=True) + with tarfile.open(args.archive, "w:gz") as tar: + for sub in ("counts.json", "kegg_rest", "uniprot"): + tar.add(args.work / sub, arcname=sub) + print(json.dumps(info, indent=1)) + + +if __name__ == "__main__": + main() diff --git a/scripts/parameter_tuning/kegg/jobs/fetch.sbatch b/scripts/parameter_tuning/kegg/jobs/fetch.sbatch new file mode 100644 index 0000000..1e06e38 --- /dev/null +++ b/scripts/parameter_tuning/kegg/jobs/fetch.sbatch @@ -0,0 +1,25 @@ +#!/bin/bash +#SBATCH -t 02:00:00 +#SBATCH -n 1 +#SBATCH -c 2 +#SBATCH -J kegg-fetch +set -euo pipefail +module purge; module load SciPy-bundle/2026.05-gfbf-2026.1 +: "${PY:?set PY}" "${DATA:?set DATA}" +D=${STUDY:?set STUDY} +export XDG_CACHE_HOME=$TMPDIR/cache +export PYTHONPATH=${TOOLBOX_ROOT:?set TOOLBOX_ROOT}/src +hostname; date -u +# peek at the core tables +$PY -c " +from raven_toolbox.data import ensure_kegg_data +from raven_toolbox.reconstruction.kegg.parse import read_kegg_table, _resolve_artefact +d=ensure_kegg_data(); import os; print(d, sorted(os.listdir(d))) +o=read_kegg_table(_resolve_artefact(d,'organism_gene_ko.tsv.gz')) +for org in ['sce','cme','eco','mge']: + r=o[o.organism.str.lower()==org]; print(org, len(r), r.gene.nunique(), r.head(3).values.tolist()) +" +$PY $D/fetch_proteomes.py --work $TMPDIR/prot \ + --archive $DATA/kegg_proteomes_2026-11-01.tar.gz +cp $TMPDIR/prot/counts.json $D/results/proteome_counts.json +date -u diff --git a/scripts/parameter_tuning/kegg/jobs/rescore.sbatch b/scripts/parameter_tuning/kegg/jobs/rescore.sbatch new file mode 100644 index 0000000..95d0a56 --- /dev/null +++ b/scripts/parameter_tuning/kegg/jobs/rescore.sbatch @@ -0,0 +1,29 @@ +#!/bin/bash +#SBATCH -t 00:40:00 +#SBATCH -n 1 +#SBATCH -c 2 +#SBATCH --mem=16G +#SBATCH -J kegg-rescore +set -euo pipefail +module purge; module load SciPy-bundle/2026.05-gfbf-2026.1 +: "${PY:?set PY}" "${DATA:?set DATA}" +D=${STUDY:?set STUDY} +export XDG_CACHE_HOME=$TMPDIR/cache PYTHONPATH=${TOOLBOX_ROOT:?set TOOLBOX_ROOT}/src +mkdir -p $TMPDIR/prot $TMPDIR/tbl +ART=$($PY -c "from raven_toolbox.data import ensure_kegg_data; print(ensure_kegg_data())") +tar xzf $DATA/kegg_proteomes_2026-11-01.tar.gz -C $TMPDIR/prot +tar xzf $DATA/kegg_hmmsearch_tblout_2026-11-01.tar.gz -C $TMPDIR/tbl +for org in sce cme eco mge; do + # same hit tables and all ko_reaction links: must equal the first run + $PY $D/analyze_hmm_cutoffs.py --artefacts $ART --org $org --proteome $TMPDIR/prot/kegg_rest/$org.faa \ + --tblout $TMPDIR/tbl/$org.tbl --json $TMPDIR/check_$org.json > /dev/null + $PY - $TMPDIR/check_$org.json $D/results/sweep_$org.json <<'PY' +import json, sys +a, b = (json.load(open(p)) for p in sys.argv[1:]) +for d in (a, b): + d.pop("search", None); d.pop("restrict_to_reference", None) +print(sys.argv[2], "rescored from saved tblout identical:", a == b) +PY + $PY $D/analyze_hmm_cutoffs.py --artefacts $ART --org $org --proteome $TMPDIR/prot/kegg_rest/$org.faa \ + --tblout $TMPDIR/tbl/$org.tbl --restrict-to-reference --json $D/results/sweep_${org}_refrxn.json +done diff --git a/scripts/parameter_tuning/kegg/jobs/run.sbatch b/scripts/parameter_tuning/kegg/jobs/run.sbatch new file mode 100644 index 0000000..9639199 --- /dev/null +++ b/scripts/parameter_tuning/kegg/jobs/run.sbatch @@ -0,0 +1,56 @@ +#!/bin/bash +#SBATCH -t 05:00:00 +#SBATCH -n 1 +#SBATCH -c 16 +#SBATCH --mem=48G +#SBATCH -J kegg-hmm +set -euo pipefail +module purge; module load SciPy-bundle/2026.05-gfbf-2026.1 +: "${PY:?set PY}" "${DATA:?set DATA}" +D=${STUDY:?set STUDY} +TB=${TOOLBOX_ROOT:?set TOOLBOX_ROOT} +export XDG_CACHE_HOME=$TMPDIR/cache +export PYTHONPATH=$TB/src +THREADS=${SLURM_CPUS_PER_TASK} +mkdir -p $XDG_CACHE_HOME/raven_toolbox $D/results +# hmmsearch 3.4.0 already fetched by the toolbox; expose its binaries cache in the job cache +ln -sfn "${BINARIES_CACHE:?set BINARIES_CACHE}" $XDG_CACHE_HOME/raven_toolbox/binaries +hostname; date -u; lscpu | grep -E "Model name|^CPU\(s\)"; echo "threads=$THREADS" +git -C $TB rev-parse HEAD +$PY -c "import sys,cobra,optlang,pandas,numpy; print(sys.version); print('cobra',cobra.__version__,'optlang',optlang.__version__,'pandas',pandas.__version__,'numpy',numpy.__version__)" +which hmmsearch || echo "hmmsearch not on PATH" +$PY - <<'PY' > $TMPDIR/paths.sh +import os, subprocess +from raven_toolbox.binaries import resolve_binary +from raven_toolbox.data import ensure_kegg_data, ensure_kegg_hmm_library +from raven_toolbox.reconstruction.kegg.query import _count_profiles +exe = resolve_binary("hmmsearch") +d = ensure_kegg_data() +libs = {dom: ensure_kegg_hmm_library(dom) for dom in ("eukaryotes", "prokaryotes")} +import sys +print("hmmsearch:", exe, subprocess.run([exe, "-h"], capture_output=True, text=True).stdout.splitlines()[1], file=sys.stderr) +for dom, p in libs.items(): + print(dom, p, os.path.getsize(p), "bytes", _count_profiles(p), "profiles", file=sys.stderr) +for f in sorted(os.listdir(d)): + print("artefact", f, os.path.getsize(os.path.join(d, f)), file=sys.stderr) +print(f"ART={d}\nEUK={libs['eukaryotes']}\nPROK={libs['prokaryotes']}") +PY +cat $TMPDIR/paths.sh; source $TMPDIR/paths.sh +mkdir -p $TMPDIR/prot $TMPDIR/tbl +tar xzf $DATA/kegg_proteomes_2026-11-01.tar.gz -C $TMPDIR/prot +for org in sce cme eco mge; do + case $org in sce|cme) LIB=$EUK;; *) LIB=$PROK;; esac + echo "### $org $(date -u)" + $PY $D/analyze_hmm_cutoffs.py --artefacts $ART --org $org \ + --proteome $TMPDIR/prot/kegg_rest/$org.faa --library $LIB --threads $THREADS \ + --save-tblout $TMPDIR/tbl/$org.tbl --json $D/results/sweep_$org.json +done +tar czf $DATA/kegg_hmmsearch_tblout_2026-11-01.tar.gz -C $TMPDIR/tbl . +for org in mge sce; do + case $org in sce) LIB=$EUK; DOM=eukaryotes;; *) LIB=$PROK; DOM=prokaryotes;; esac + echo "### whole model $org $(date -u)" + $PY $D/compare_kegg_paths.py --org $org --proteome $TMPDIR/prot/kegg_rest/$org.faa \ + --artefacts $ART --library $LIB --domain $DOM --threads $THREADS \ + --settings 1e-50,0.3,0.8 1e-30,0.3,0.9 --json $D/results/wholemodel_$org.json +done +date -u diff --git a/scripts/parameter_tuning/kegg/jobs/timing.sbatch b/scripts/parameter_tuning/kegg/jobs/timing.sbatch new file mode 100644 index 0000000..63b55ed --- /dev/null +++ b/scripts/parameter_tuning/kegg/jobs/timing.sbatch @@ -0,0 +1,23 @@ +#!/bin/bash +#SBATCH -t 02:00:00 +#SBATCH -n 1 +#SBATCH -c 16 +#SBATCH --mem=32G +#SBATCH --exclusive +#SBATCH -J kegg-time +set -euo pipefail +module purge; module load SciPy-bundle/2026.05-gfbf-2026.1 +: "${PY:?set PY}" "${DATA:?set DATA}" +D=${STUDY:?set STUDY} +export XDG_CACHE_HOME=$TMPDIR/cache PYTHONPATH=${TOOLBOX_ROOT:?set TOOLBOX_ROOT}/src +mkdir -p $XDG_CACHE_HOME/raven_toolbox $TMPDIR/prot +ln -sfn "${BINARIES_CACHE:?set BINARIES_CACHE}" $XDG_CACHE_HOME/raven_toolbox/binaries +hostname; date -u; lscpu | grep -E "Model name|^CPU\(s\)"; nproc +eval "$($PY -c " +from raven_toolbox.data import ensure_kegg_data, ensure_kegg_hmm_library +print(f'ART={ensure_kegg_data()}; EUK={ensure_kegg_hmm_library(\"eukaryotes\")}')")" +tar xzf $DATA/kegg_proteomes_2026-11-01.tar.gz -C $TMPDIR/prot +$PY $D/check_true_reactions.py --artefacts $ART --json $D/results/true_reactions.json +$PY $D/time_hmmsearch.py --proteome $TMPDIR/prot/kegg_rest/sce.faa --library $EUK \ + --threads 16 12 4 1 --json $D/results/timing_sce_eukaryotes.json +date -u diff --git a/scripts/parameter_tuning/kegg/time_hmmsearch.py b/scripts/parameter_tuning/kegg/time_hmmsearch.py new file mode 100644 index 0000000..600d962 --- /dev/null +++ b/scripts/parameter_tuning/kegg/time_hmmsearch.py @@ -0,0 +1,61 @@ +#!/usr/bin/env python3 +"""Time ``run_hmmsearch`` of one proteome against one KO HMM library at several thread counts. + +For each thread count it runs ``raven_toolbox.reconstruction.kegg.query.run_hmmsearch`` +once, reports the wall time, the number of hit rows, genes and KOs in the parsed +``--tblout``, and whether the parsed hit table (KO, gene, E-value) equals the one of +the first run. Writes the results as JSON. + +Example: + python time_hmmsearch.py --proteome sce.faa --library kegg118_eukaryotes.hmm \ + --threads 16 12 1 --json timing_sce_eukaryotes.json +""" + +from __future__ import annotations + +import argparse +import json +import time +from pathlib import Path + +from raven_toolbox.reconstruction.kegg.query import parse_hmmsearch_tblout, run_hmmsearch + + +def main() -> None: + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument("--proteome", type=Path, required=True) + ap.add_argument("--library", type=Path, required=True) + ap.add_argument("--threads", type=int, nargs="+", required=True) + ap.add_argument("--json", type=Path) + args = ap.parse_args() + nseq = sum(1 for line in args.proteome.read_text().splitlines() if line.startswith(">")) + rows, first = [], None + for t in args.threads: + t0 = time.perf_counter() + hits = parse_hmmsearch_tblout(run_hmmsearch(args.proteome, args.library, threads=t)) + secs = time.perf_counter() - t0 + key = sorted(zip(hits["ko"], hits["gene"], hits["evalue"], strict=False)) + same = None if first is None else key == first + first = key if first is None else first + row = { + "threads": t, + "seconds": secs, + "queries": nseq, + "hit_rows": len(hits), + "genes_with_hits": int(hits["gene"].nunique()), + "kos_with_hits": int(hits["ko"].nunique()), + "identical_to_first_run": same, + } + rows.append(row) + print(row, flush=True) + if args.json: + args.json.write_text( + json.dumps( + {"proteome": str(args.proteome), "library": str(args.library), "runs": rows}, + indent=1, + ) + ) + + +if __name__ == "__main__": + main() diff --git a/scripts/parameter_tuning/sampling/README.md b/scripts/parameter_tuning/sampling/README.md new file mode 100644 index 0000000..c1ab81f --- /dev/null +++ b/scripts/parameter_tuning/sampling/README.md @@ -0,0 +1,48 @@ +# Sampling convergence calibration + +Scripts for the sampling convergence page of the parameter-tuning documentation: autocorrelation within one ACHR chain on yeast-GEM at three thinning values (S1), agreement between chains (R-hat) for ACHR and CHRR on e_coli_core and yeast-GEM (S2 to S6), and `random_objective` with `replace_max_bound` on yeast-GEM (S7). The R-hat runs use `scripts/analyze_sampling_convergence.py` of raven-toolbox. + +## Inputs and environment + +| Variable | Meaning | +| --- | --- | +| `STUDY` | this folder | +| `RAVEN_TOOLBOX_DIR` | raven-toolbox checkout (`94f3940`) | +| `PY` | Python interpreter with cobra 0.32.1 and gurobipy | +| `DATA` | folder for chain pickles, sample matrices and logs | +| `Y4105` | yeast-GEM v9.1.1, 4105 reactions (S3, S4, S6) | +| `Y4102` | yeast-GEM v9.1.0, 4102 reactions (S1, S7) | +| `ECORE` | `cobra/data/textbook.xml.gz`, e_coli_core with 95 reactions (S2, S5) | +| `GUROBI_HOME` | Gurobi installation; its `gurobi.lic` is used | + +The module names in the job files are site-specific; other systems need an equivalent environment. Every job uses 12 cores (the post-hoc job uses 1). S6 needs about 32 GB. + +## Commands + +```bash +export STUDY=... RAVEN_TOOLBOX_DIR=... PY=... DATA=... Y4105=... Y4102=... ECORE=... GUROBI_HOME=... +J="sbatch -A " +W="--n-chains 4 --n-samples 300 --thinning 100 --warmup 1000" +$J --export=ALL,TAG=S2_ecore_achr_m4_n300_t100,MODELVAR=ECORE,ARGS="$W" $STUDY/jobs/rhat.sbatch +$J --export=ALL,TAG=S3_yeast4105_achr_m4_n300_t100,MODELVAR=Y4105,ARGS="$W" $STUDY/jobs/rhat.sbatch +$J --export=ALL,TAG=S4_yeast4105_achr_m4_n100_t300,MODELVAR=Y4105,ARGS="--n-chains 4 --n-samples 100 --thinning 300 --warmup 1000" $STUDY/jobs/rhat.sbatch +$J --export=ALL,TAG=S5_ecore_chrr_m4_n300_t100,MODELVAR=ECORE,ARGS="--method chrr $W" $STUDY/jobs/rhat.sbatch +$J --export=ALL,TAG=S6_yeast4105_chrr_m2_n20_t100,MODELVAR=Y4105,ARGS="--method chrr --n-chains 2 --n-samples 20 --thinning 100 --warmup 1000" --mem=32G $STUDY/jobs/rhat.sbatch +for t in 20 100 500; do $J --export=ALL,THIN=$t $STUDY/jobs/s1.sbatch; done +$J $STUDY/jobs/s1_seeds.sbatch +$J $STUDY/jobs/s7.sbatch +$J $STUDY/jobs/posthoc.sbatch # after S2, S5 and S7 +``` + +Chain seeds are 0 to m-1 (fixed in `analyze_sampling_convergence.py`). S1 and S7 use seed 0; `s1_seeds.sbatch` adds seeds 1 to 3. Logs and chain files go to `$DATA`, tables to `$STUDY/results`. + +## Outputs + +| File | Content | +| --- | --- | +| `jobs/rhat.sbatch` | Runs `analyze_sampling_convergence.py`, then `summarize_chains.py`, which writes `.json` (R-hat summary) and `_rhat.tsv` (R-hat per reaction, regenerated by the run). | +| `single_chain_autocorr.py` | S1: lag-1 autocorrelation per reaction over reactions with a flux range above 1e-6, effective sample size n (1 - rho) / (1 + rho), wall time; writes `s1_achr_t_n300_seed0.json` and a per-reaction table. | +| `results/s1_seed_spread.tsv` | Mean and median rho for seeds 0 to 3 at each thinning, written by `s1_seeds.sbatch`. | +| `random_objective_bounds.py` | S7: `random_objective` with `replace_max_bound=False` (samples and reactions at the bound) and `True` (abort, probe of random objectives for unbounded status); writes `s7_random_objective_n200_seed0.json`. | +| `posthoc_checks.py` | Reactions that ACHR leaves constant in S2 and S5 and their CHRR variation, and alternative denominators for S7; writes `results/posthoc.json`. | +| `env_report.py` | Prints Python, package, solver, CPU and model versions into every log. | diff --git a/scripts/parameter_tuning/sampling/env_report.py b/scripts/parameter_tuning/sampling/env_report.py new file mode 100644 index 0000000..7173fcb --- /dev/null +++ b/scripts/parameter_tuning/sampling/env_report.py @@ -0,0 +1,86 @@ +#!/usr/bin/env python3 +"""Print the software versions, the solver and the input file ids of a sampling run. + +Prints Python, cobra, optlang, gurobipy (and the Gurobi library version it +loads), numpy, pandas, scipy, the raven-toolbox commit and the solver that +cobra selects for each model, with the sha256, the SBML model id and the +reaction count of each model file given on the command line. + +Inputs: model files as arguments; RAVEN_TOOLBOX_DIR (the raven-toolbox +checkout whose src/ is on PYTHONPATH) for the commit. + +Command line used: + python env_report.py [ ...] +""" + +from __future__ import annotations + +import hashlib +import json +import os +import subprocess +import sys + + +def versions() -> dict: + import cobra + import numpy + import optlang + import pandas + import scipy + + out = { + "python": sys.version.split()[0], + "cobra": cobra.__version__, + "optlang": optlang.__version__, + "numpy": numpy.__version__, + "pandas": pandas.__version__, + "scipy": scipy.__version__, + } + try: + import gurobipy + + out["gurobipy"] = gurobipy.__version__ if hasattr(gurobipy, "__version__") else None + out["gurobi_library"] = ".".join(str(x) for x in gurobipy.gurobi.version()) + except Exception as exc: # noqa: BLE001 + out["gurobipy"] = f"unavailable: {exc}" + rt = os.environ.get("RAVEN_TOOLBOX_DIR") + if rt: + try: + out["raven_toolbox_commit"] = subprocess.run( + ["git", "-C", rt, "rev-parse", "HEAD"], capture_output=True, text=True, check=True + ).stdout.strip() + except Exception as exc: # noqa: BLE001 + out["raven_toolbox_commit"] = f"unknown: {exc}" + try: + with open("/proc/cpuinfo") as fh: + out["cpu_model"] = next( + line.split(":", 1)[1].strip() for line in fh if line.startswith("model name") + ) + except Exception: # noqa: BLE001 + pass + out["cpus_allowed"] = len(os.sched_getaffinity(0)) + return out + + +def model_info(path: str) -> dict: + import cobra + + h = hashlib.sha256() + with open(path, "rb") as fh: + for block in iter(lambda: fh.read(1 << 20), b""): + h.update(block) + model = cobra.io.read_sbml_model(path) + return { + "file": path, + "sha256": h.hexdigest(), + "model_id": model.id, + "n_reactions": len(model.reactions), + "n_metabolites": len(model.metabolites), + "solver_interface": model.solver.interface.__name__, + } + + +if __name__ == "__main__": + report = {"versions": versions(), "models": [model_info(p) for p in sys.argv[1:]]} + print(json.dumps(report, indent=2)) diff --git a/scripts/parameter_tuning/sampling/jobs/env.sh b/scripts/parameter_tuning/sampling/jobs/env.sh new file mode 100644 index 0000000..3a0d5d8 --- /dev/null +++ b/scripts/parameter_tuning/sampling/jobs/env.sh @@ -0,0 +1,16 @@ +# Environment of every job of the sampling study (sourced by the sbatch scripts). +# Required in the environment of sbatch: +# STUDY this study folder (scripts/parameter_tuning/sampling) +# RAVEN_TOOLBOX_DIR raven-toolbox checkout (commit 94f3940); its src/ goes on PYTHONPATH +# PY Python interpreter (cobra 0.32.1, optlang 1.9.1, gurobipy 13.0.3) +# DATA work folder for chain pickles and sample matrices +# Y4105, Y4102, ECORE yeast-GEM 9.1.1 (4105 reactions), yeast-GEM 9.1.0 (4102 reactions) and +# the cobra e_coli_core file (cobra/data/textbook.xml.gz) +# GUROBI_HOME Gurobi 13.0.1 installation; only its gurobi.lic is used +: "${STUDY:?}" "${RAVEN_TOOLBOX_DIR:?}" "${PY:?}" "${DATA:?}" "${Y4105:?}" "${Y4102:?}" "${ECORE:?}" +module purge +module load SciPy-bundle/2026.05-gfbf-2026.1 +export PYTHONPATH=$RAVEN_TOOLBOX_DIR/src +export GUROBI_HOME=${GUROBI_HOME:?set GUROBI_HOME} +export GRB_LICENSE_FILE=$GUROBI_HOME/gurobi.lic +export LD_LIBRARY_PATH=$GUROBI_HOME/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH} diff --git a/scripts/parameter_tuning/sampling/jobs/posthoc.sbatch b/scripts/parameter_tuning/sampling/jobs/posthoc.sbatch new file mode 100644 index 0000000..2dde54c --- /dev/null +++ b/scripts/parameter_tuning/sampling/jobs/posthoc.sbatch @@ -0,0 +1,11 @@ +#!/bin/bash +# Post-hoc checks on stored chains and S7 samples. +#SBATCH -N 1 +#SBATCH -c 1 +#SBATCH --mem=8G +set -euo pipefail +source "${STUDY:?set STUDY}/jobs/env.sh" +cd $STUDY +$PY $STUDY/posthoc_checks.py --achr-pickle $(ls $DATA/S2_ecore_achr_m4_n300_t100/chains_*.pkl) \ + --chrr-pickle $(ls $DATA/S5_ecore_chrr_m4_n300_t100/chains_*.pkl) --s7-npz $DATA/s7_false.npz \ + --model $Y4102 --out results/posthoc.json diff --git a/scripts/parameter_tuning/sampling/jobs/rhat.sbatch b/scripts/parameter_tuning/sampling/jobs/rhat.sbatch new file mode 100644 index 0000000..0b165eb --- /dev/null +++ b/scripts/parameter_tuning/sampling/jobs/rhat.sbatch @@ -0,0 +1,22 @@ +#!/bin/bash +# R-hat run with scripts/analyze_sampling_convergence.py of the raven-toolbox checkout. +# Variables (via --export): TAG, MODELVAR (name of the model variable in env.sh), ARGS (options of the script). +#SBATCH -N 1 +#SBATCH -c 12 +#SBATCH --mem=32G +set -euo pipefail +source "${STUDY:?set STUDY}/jobs/env.sh" +MODEL=${!MODELVAR} +OUT=$DATA/$TAG +mkdir -p $OUT $DATA/logs +cd $STUDY +echo "cpus $SLURM_CPUS_PER_TASK, start $(date -Is)" +$PY $STUDY/env_report.py $MODEL +CMD="$PY $RAVEN_TOOLBOX_DIR/scripts/analyze_sampling_convergence.py --model $MODEL --out $OUT $ARGS" +echo "CMD: $CMD" +$CMD | tee $DATA/logs/$TAG.analysis.txt +WALL=$(grep -o '[0-9.]*s wall' $DATA/logs/$TAG.analysis.txt | head -1 | tr -d 's wall') +PKL=$(ls $OUT/chains_*.pkl) +$PY $STUDY/summarize_chains.py --pickle $PKL --analysis-script $RAVEN_TOOLBOX_DIR/scripts/analyze_sampling_convergence.py \ + --tag $TAG --out results/ --wall-s ${WALL:-nan} > /dev/null +echo "end $(date -Is)" diff --git a/scripts/parameter_tuning/sampling/jobs/s1.sbatch b/scripts/parameter_tuning/sampling/jobs/s1.sbatch new file mode 100644 index 0000000..f565cc3 --- /dev/null +++ b/scripts/parameter_tuning/sampling/jobs/s1.sbatch @@ -0,0 +1,13 @@ +#!/bin/bash +# S1: one ACHR chain on the 4102-reaction yeast-GEM. Variable (via --export): THIN. +#SBATCH -N 1 +#SBATCH -c 12 +#SBATCH --mem=16G +set -euo pipefail +source "${STUDY:?set STUDY}/jobs/env.sh" +cd $STUDY +echo "cpus $SLURM_CPUS_PER_TASK, start $(date -Is)" +$PY $STUDY/env_report.py $Y4102 +$PY $STUDY/single_chain_autocorr.py --model $Y4102 --thinning $THIN --n-samples 300 --warmup 1000 --seed 0 \ + --time-warmup --out results/ --save-samples $DATA/s1_t${THIN}.npz +echo "end $(date -Is)" diff --git a/scripts/parameter_tuning/sampling/jobs/s1_seeds.sbatch b/scripts/parameter_tuning/sampling/jobs/s1_seeds.sbatch new file mode 100644 index 0000000..28d8bc3 --- /dev/null +++ b/scripts/parameter_tuning/sampling/jobs/s1_seeds.sbatch @@ -0,0 +1,29 @@ +#!/bin/bash +# S1 seed spread: one ACHR chain per seed 1..3 and thinning 20/100/500 on the 4102-reaction yeast-GEM, +# run sequentially; rho only (wall times are not comparable with the timed seed-0 jobs). +#SBATCH -N 1 +#SBATCH -c 12 +#SBATCH --mem=16G +set -euo pipefail +source "${STUDY:?set STUDY}/jobs/env.sh" +cd $STUDY +echo "cpus $SLURM_CPUS_PER_TASK, start $(date -Is)" +for seed in 1 2 3; do for t in 20 100 500; do + $PY $STUDY/single_chain_autocorr.py --model $Y4102 --thinning $t --n-samples 300 --warmup 1000 --seed $seed \ + --out $DATA/s1_seeds > /dev/null +done; done +$PY - <<'PY' +import json, glob, os +rows = [] +for f in sorted(glob.glob(os.environ["DATA"] + "/s1_seeds/*.json")) + sorted(glob.glob("results/s1_achr_t*_seed0.json")): + d = json.load(open(f)) + rows.append([d["thinning"], d["seed"], d["n_nonconstant"], d["rho_mean_pearson"], d["rho_median_pearson"], + d["rho_mean_acf"], d["rho_median_acf"], d["ess_from_mean_rho"], d["ess_from_median_rho"], d["t_total_s"]]) +rows.sort() +with open("results/s1_seed_spread.tsv", "w") as fh: + fh.write("thinning\tseed\tn_nonconstant\trho_mean_pearson\trho_median_pearson\trho_mean_acf\trho_median_acf\tess_from_mean_rho\tess_from_median_rho\tt_total_s\n") + for r in rows: + fh.write("\t".join(str(round(x, 4)) if isinstance(x, float) else str(x) for x in r) + "\n") +print(open("results/s1_seed_spread.tsv").read()) +PY +echo "end $(date -Is)" diff --git a/scripts/parameter_tuning/sampling/jobs/s7.sbatch b/scripts/parameter_tuning/sampling/jobs/s7.sbatch new file mode 100644 index 0000000..4a63115 --- /dev/null +++ b/scripts/parameter_tuning/sampling/jobs/s7.sbatch @@ -0,0 +1,12 @@ +#!/bin/bash +# S7: random_objective with replace_max_bound False and True on the 4102-reaction yeast-GEM. +#SBATCH -N 1 +#SBATCH -c 12 +#SBATCH --mem=32G +set -euo pipefail +source "${STUDY:?set STUDY}/jobs/env.sh" +cd $STUDY +echo "cpus $SLURM_CPUS_PER_TASK, start $(date -Is)" +$PY $STUDY/env_report.py $Y4102 +$PY $STUDY/random_objective_bounds.py --model $Y4102 --n-samples 200 --seed 0 --n-proc 12 --n-probe 50 --out results/ --save-samples $DATA/s7_false.npz +echo "end $(date -Is)" diff --git a/scripts/parameter_tuning/sampling/posthoc_checks.py b/scripts/parameter_tuning/sampling/posthoc_checks.py new file mode 100644 index 0000000..cf162f7 --- /dev/null +++ b/scripts/parameter_tuning/sampling/posthoc_checks.py @@ -0,0 +1,88 @@ +#!/usr/bin/env python3 +"""Two checks on stored results of the sampling study. + +1. ``--achr-pickle`` / ``--chrr-pickle`` (chain pickles of the e_coli_core ACHR + and CHRR runs): for each reaction that is constant in every ACHR chain (no + R-hat), the flux range within each CHRR chain and the mean, to see whether + CHRR's variation of these reactions is at rounding-error size. +2. ``--s7-npz`` (samples of ``random_objective_bounds.py --save-samples``) and + ``--bound``: the share of samples at the bound under further denominators + (entries of the good reactions, reactions at big-M, reactions that are + non-zero in some sample). + +Command line used: + python posthoc_checks.py --achr-pickle $DATA/S2_.../chains_*.pkl \ + --chrr-pickle $DATA/S5_.../chains_*.pkl --s7-npz $DATA/s7_false.npz \ + --model docs/data/yeast-GEM.xml --out results/posthoc.json +""" + +from __future__ import annotations + +import argparse +import json +import pickle +from pathlib import Path + +import numpy as np + + +def main() -> None: + p = argparse.ArgumentParser( + description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter + ) + p.add_argument("--achr-pickle", type=Path, required=True) + p.add_argument("--chrr-pickle", type=Path, required=True) + p.add_argument("--s7-npz", type=Path, required=True) + p.add_argument("--model", type=Path, required=True) + p.add_argument("--out", type=Path, required=True) + args = p.parse_args() + out: dict = {} + + achr = pickle.load(open(args.achr_pickle, "rb")) + chrr = pickle.load(open(args.chrr_pickle, "rb")) + a = np.stack([c.to_numpy() for c in achr]) + const = [rid for j, rid in enumerate(achr[0].columns) if (a[:, :, j].std(axis=1) == 0).all()] + rows = {} + for rid in const: + v = np.stack([c[rid].to_numpy() for c in chrr]) + rows[rid] = { + "achr_value": float(achr[0][rid].iloc[0]), + "chrr_mean": float(v.mean()), + "chrr_max_range_within_chain": float((v.max(axis=1) - v.min(axis=1)).max()), + } + out["ecore_achr_constant_reactions_in_chrr"] = rows + + import cobra + + model = cobra.io.read_sbml_model(str(args.model)) + d = np.load(args.s7_npz, allow_pickle=True) + x, rxns, good = d["samples"], list(d["reactions"]), set(d["good_reactions"]) + ub = np.array([model.reactions.get_by_id(r).upper_bound for r in rxns]) + bound = ub.max() + at = np.abs(x) >= bound * (1 - 1e-9) + gmask = np.array([r in good for r in rxns]) + nz = np.abs(x) >= 1e-9 + out["s7"] = { + "entries_at_bound": int(at.sum()), + "pct_of_good_reaction_entries": 100.0 * at[:, gmask].sum() / at[:, gmask].size, + "reactions_at_bound_any_sample": int(at.any(axis=0).sum()), + "pct_reactions_at_bound_of_bigM_ub": 100.0 + * at.any(axis=0).sum() + / int((ub == bound).sum()), + "pct_reactions_at_bound_of_good": 100.0 * at.any(axis=0).sum() / len(good), + "pct_reactions_at_bound_of_nonzero_any": 100.0 + * at.any(axis=0).sum() + / int(nz.any(axis=0).sum()), + "per_sample_pct_entries_at_bound_mean": float(100.0 * at.mean(axis=1).mean()), + "pct_of_nonzero_entries_in_good_reactions": 100.0 + * at[:, gmask].sum() + / max(1, nz[:, gmask].sum()), + } + args.out.parent.mkdir(parents=True, exist_ok=True) + with open(args.out, "w") as fh: + json.dump(out, fh, indent=2) + print(json.dumps(out, indent=2)) + + +if __name__ == "__main__": + main() diff --git a/scripts/parameter_tuning/sampling/random_objective_bounds.py b/scripts/parameter_tuning/sampling/random_objective_bounds.py new file mode 100644 index 0000000..ef1b68e --- /dev/null +++ b/scripts/parameter_tuning/sampling/random_objective_bounds.py @@ -0,0 +1,198 @@ +#!/usr/bin/env python3 +"""random_sampling(method="random_objective") with replace_max_bound False and True. + +1. Counts the big-M reactions of the model: reactions whose upper bound equals + the largest upper bound, reactions whose lower bound equals the most + negative lower bound, and reactions with either (the bounds that + ``replace_max_bound=True`` replaces with infinity). +2. Runs ``random_sampling(model, n_samples, method="random_objective", + seed=seed, replace_max_bound=False, n_proc=n_proc)`` and counts, with + ``|v| >= bound * (1 - 1e-9)`` as "at the bound" (bound = the largest upper + bound, 1000 in yeast-GEM): + - entries of the sample matrix (samples x reactions) at the bound, as count + and share of all entries and of the non-zero entries; + - samples (rows) with at least one reaction at the bound; + - reactions at the bound in at least one sample; + - reactions at zero (``|v| < 1e-9``) in every sample, and the median of the + per-reaction standard deviation. +3. Runs the same call with ``replace_max_bound=True``, ``n_proc=1`` and the + ``good_reactions`` of step 2 (they are computed on the finite bounds in + both cases, so passing them only skips the repeated loopless FVA), and + records whether it completes or the exception it raises. +4. Diagnostic for step 3: on a copy of the model with the bounds replaced as + ``replace_max_bound=True`` does, solves ``--n-probe`` random objectives drawn + as ``_draw_one_sample`` draws them (two reactions from ``good_reactions``, + weights U(0,1) with random sign) and counts the solver statuses. + +Inputs: the SBML model file; ``--out`` directory for the JSON summary; +optionally ``--save-samples`` for the sample matrix of step 2. + +Command line used: + python random_objective_bounds.py --model docs/data/yeast-GEM.xml \ + --n-samples 200 --seed 0 --n-proc 12 --n-probe 50 --out results/ \ + --save-samples $DATA/s7_false.npz +""" + +from __future__ import annotations + +import argparse +import collections +import json +import time +import traceback +from pathlib import Path + +import cobra +import numpy as np + + +def bigm_counts(model: cobra.Model) -> dict: + ubs = np.array([r.upper_bound for r in model.reactions]) + lbs = np.array([r.lower_bound for r in model.reactions]) + max_ub, min_lb = ubs.max(), lbs.min() + at_ub = ubs == max_ub + at_lb = (lbs == min_lb) if min_lb < 0 else np.zeros_like(at_ub) + return { + "n_reactions": int(len(ubs)), + "max_upper_bound": float(max_ub), + "min_lower_bound": float(min_lb), + "n_ub_at_max": int(at_ub.sum()), + "n_lb_at_min": int(at_lb.sum()), + "n_ub_or_lb_at_bigM": int((at_ub | at_lb).sum()), + "n_ub_and_lb_at_bigM": int((at_ub & at_lb).sum()), + } + + +def sample_counts(x: np.ndarray, bound: float) -> dict: + at = np.abs(x) >= bound * (1 - 1e-9) + nz = np.abs(x) >= 1e-9 + return { + "n_samples": int(x.shape[0]), + "n_entries": int(x.size), + "entries_at_bound": int(at.sum()), + "entries_at_bound_pct_of_all": 100.0 * at.sum() / x.size, + "nonzero_entries": int(nz.sum()), + "entries_at_bound_pct_of_nonzero": 100.0 * at.sum() / max(1, nz.sum()), + "samples_with_any_reaction_at_bound": int(at.any(axis=1).sum()), + "samples_with_any_reaction_at_bound_pct": 100.0 * at.any(axis=1).mean(), + "reactions_at_bound_in_any_sample": int(at.any(axis=0).sum()), + "reactions_at_bound_in_any_sample_pct_of_all_reactions": 100.0 * at.any(axis=0).mean(), + "reactions_nonzero_in_any_sample": int(nz.any(axis=0).sum()), + "reactions_always_zero": int((~nz).all(axis=0).sum()), + "median_per_reaction_std": float(np.median(x.std(axis=0, ddof=1))), + "max_abs_flux": float(np.abs(x).max()), + } + + +def main() -> None: + p = argparse.ArgumentParser( + description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter + ) + p.add_argument("--model", type=Path, required=True) + p.add_argument("--out", type=Path, required=True) + p.add_argument("--n-samples", type=int, default=200) + p.add_argument("--seed", type=int, default=0) + p.add_argument("--n-proc", type=int, default=1) + p.add_argument("--n-probe", type=int, default=50) + p.add_argument( + "--save-samples", + type=Path, + default=None, + help=".npz for the replace_max_bound=False samples", + ) + args = p.parse_args() + + from raven_toolbox.analysis.sampling import random_sampling + + args.out.mkdir(parents=True, exist_ok=True) + cobra.Configuration().processes = args.n_proc # FVA of find_good_reactions + model = cobra.io.read_sbml_model(str(args.model)) + summary: dict = { + "model": str(args.model), + "model_id": model.id, + "solver_interface": model.solver.interface.__name__, + "seed": args.seed, + "bigM": bigm_counts(model), + } + bound = summary["bigM"]["max_upper_bound"] + + t0 = time.time() + res = random_sampling( + model, + args.n_samples, + method="random_objective", + seed=args.seed, + replace_max_bound=False, + n_proc=args.n_proc, + ) + x = res.samples.to_numpy() + summary["false"] = { + "completed": True, + "wall_s": time.time() - t0, + "n_proc": args.n_proc, + "n_good_reactions": len(res.good_reactions), + **sample_counts(x, bound), + } + print(json.dumps(summary, indent=2), flush=True) + if args.save_samples: + np.savez_compressed( + args.save_samples, + samples=x, + reactions=np.array(res.samples.columns), + good_reactions=np.array(res.good_reactions), + ) + + t0 = time.time() + try: + res_t = random_sampling( + model, + args.n_samples, + method="random_objective", + seed=args.seed, + replace_max_bound=True, + good_reactions=res.good_reactions, + n_proc=1, + ) + summary["true"] = { + "completed": True, + "wall_s": time.time() - t0, + **sample_counts(res_t.samples.to_numpy(), bound), + } + except Exception as exc: # noqa: BLE001 + summary["true"] = { + "completed": False, + "wall_s": time.time() - t0, + "exception": type(exc).__name__, + "message": str(exc), + "traceback_tail": traceback.format_exc().splitlines()[-3:], + } + + m = model.copy() + max_ub = max(r.upper_bound for r in m.reactions) + min_lb = min(r.lower_bound for r in m.reactions) + for r in m.reactions: + if r.upper_bound == max_ub: + r.upper_bound = float("inf") + if min_lb < 0 and r.lower_bound == min_lb: + r.lower_bound = float("-inf") + rng = np.random.default_rng(args.seed) + objs = [m.reactions.get_by_id(r) for r in res.good_reactions] + statuses = collections.Counter() + for _ in range(args.n_probe): + chosen = rng.choice(len(objs), size=2, replace=False) + w = rng.random(2) * rng.choice((-1.0, 1.0), size=2) + m.objective = m.problem.Objective( + sum(wi * objs[j].flux_expression for j, wi in zip(chosen, w, strict=False)), + direction="max", + ) + statuses[m.solver.optimize()] += 1 + summary["true_probe_statuses"] = dict(statuses) + summary["true_probe_n"] = args.n_probe + + with open(args.out / f"s7_random_objective_n{args.n_samples}_seed{args.seed}.json", "w") as fh: + json.dump(summary, fh, indent=2) + print(json.dumps(summary, indent=2)) + + +if __name__ == "__main__": + main() diff --git a/scripts/parameter_tuning/sampling/single_chain_autocorr.py b/scripts/parameter_tuning/sampling/single_chain_autocorr.py new file mode 100644 index 0000000..20eea74 --- /dev/null +++ b/scripts/parameter_tuning/sampling/single_chain_autocorr.py @@ -0,0 +1,162 @@ +#!/usr/bin/env python3 +"""Lag-1 autocorrelation and ESS of one ACHR chain of raven-toolbox's random_sampling. + +Runs one chain ``random_sampling(model, n_samples, method="achr", seed=seed, +thinning=thinning, warmup=warmup)`` (``warmup`` is ignored by ACHR, it is passed +for compatibility with the other samplers), and computes per reaction the lag-1 autocorrelation of the +stored samples. The operational definitions are: + +* non-constant reaction: ``max - min`` of the reaction's flux over the chain is + larger than ``--const-tol`` (default 1e-6). Other reactions get no rho. +* ``rho_pearson``: Pearson correlation of ``x[0:n-1]`` with ``x[1:n]`` + (``pandas.Series.autocorr(lag=1)``). +* ``rho_acf``: the standard autocorrelation-function estimator + ``sum_t (x_t - m)(x_{t+1} - m) / sum_t (x_t - m)^2`` with the mean ``m`` of the + whole chain (as in ``statsmodels.tsa.stattools.acf``). +* the combined rho of the chain is the mean over non-constant reactions of + ``rho_pearson`` (the primary number); the median of ``rho_pearson``, the mean + and median of ``rho_acf`` are reported next to it. +* ESS uses the formula ``n (1 - rho) / (1 + rho)`` on the combined rho; + the median over reactions of the per-reaction ESS is also reported. + +Wall times: ``t_read`` (``read_sbml_model``), ``t_sample`` (the +``random_sampling`` call: ACHR warmup points plus the sampling) and their sum. +With ``--time-warmup`` the script afterwards builds +``cobra.sampling.ACHRSampler(model, thinning=thinning, seed=seed)`` once more +on a fresh copy of the model and times it, which is the warmup-point part of +``t_sample``. + +Inputs: the SBML model file; ``--out`` directory for the JSON summary and the +per-reaction TSV (and, with ``--save-samples``, the sample matrix as .npz). + +Command line used (one Slurm job per thinning): + python single_chain_autocorr.py --model docs/data/yeast-GEM.xml \ + --thinning 20 --n-samples 300 --warmup 1000 --seed 0 --time-warmup \ + --out results/ +""" + +from __future__ import annotations + +import argparse +import json +import time +from pathlib import Path + +import cobra +import numpy as np +import pandas as pd + + +def lag1_table(samples: pd.DataFrame, const_tol: float = 1e-6) -> pd.DataFrame: + """Per-reaction range, lag-1 autocorrelations and ESS of one chain (samples x reactions).""" + x = samples.to_numpy(dtype=float) + n = x.shape[0] + rng = x.max(axis=0) - x.min(axis=0) + nonconst = rng > const_tol + d = x - x.mean(axis=0) + with np.errstate(divide="ignore", invalid="ignore"): + acf = (d[:-1] * d[1:]).sum(axis=0) / (d * d).sum(axis=0) + a, b = x[:-1], x[1:] + da, db = a - a.mean(axis=0), b - b.mean(axis=0) + pear = (da * db).sum(axis=0) / np.sqrt((da * da).sum(axis=0) * (db * db).sum(axis=0)) + acf[~nonconst] = np.nan + pear[~nonconst] = np.nan + ess = n * (1 - pear) / (1 + pear) + return pd.DataFrame( + { + "range": rng, + "nonconstant": nonconst, + "rho_pearson": pear, + "rho_acf": acf, + "ess_pearson": ess, + }, + index=samples.columns, + ) + + +def summarize(tab: pd.DataFrame, n: int) -> dict: + p = tab["rho_pearson"].dropna() + a = tab["rho_acf"].dropna() + rho = float(p.mean()) + return { + "n_samples": n, + "n_reactions": int(len(tab)), + "n_nonconstant": int(tab["nonconstant"].sum()), + "rho_mean_pearson": rho, + "rho_median_pearson": float(p.median()), + "rho_mean_acf": float(a.mean()), + "rho_median_acf": float(a.median()), + "rho_p10_pearson": float(p.quantile(0.1)), + "rho_p90_pearson": float(p.quantile(0.9)), + "ess_from_mean_rho": n * (1 - rho) / (1 + rho), + "ess_from_median_rho": n * (1 - float(p.median())) / (1 + float(p.median())), + "ess_median_per_reaction": float(tab["ess_pearson"].dropna().median()), + } + + +def main() -> None: + p = argparse.ArgumentParser( + description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter + ) + p.add_argument("--model", type=Path, required=True) + p.add_argument("--out", type=Path, required=True) + p.add_argument("--thinning", type=int, required=True) + p.add_argument("--n-samples", type=int, default=300) + p.add_argument("--warmup", type=int, default=1000) + p.add_argument("--seed", type=int, default=0) + p.add_argument("--const-tol", type=float, default=1e-6) + p.add_argument("--time-warmup", action="store_true") + p.add_argument("--save-samples", type=Path, default=None) + args = p.parse_args() + + from raven_toolbox.analysis.sampling import random_sampling + + args.out.mkdir(parents=True, exist_ok=True) + t0 = time.time() + model = cobra.io.read_sbml_model(str(args.model)) + t1 = time.time() + res = random_sampling( + model, + args.n_samples, + method="achr", + seed=args.seed, + thinning=args.thinning, + warmup=args.warmup, + ) + t2 = time.time() + samples = res.samples + tab = lag1_table(samples, args.const_tol) + summary = { + "model": str(args.model), + "model_id": model.id, + "solver_interface": model.solver.interface.__name__, + "thinning": args.thinning, + "warmup_arg": args.warmup, + "seed": args.seed, + "n_warmup_points": getattr(res, "n_warmup", None), + "const_tol": args.const_tol, + "t_read_s": t1 - t0, + "t_sample_s": t2 - t1, + "t_total_s": t2 - t0, + **summarize(tab, samples.shape[0]), + } + if args.time_warmup: + from cobra.sampling import ACHRSampler + + m2 = cobra.io.read_sbml_model(str(args.model)) + t3 = time.time() + ACHRSampler(m2, thinning=args.thinning, seed=args.seed) + summary["t_achr_warmup_points_s"] = time.time() - t3 + stem = f"s1_achr_t{args.thinning}_n{args.n_samples}_seed{args.seed}" + tab.to_csv(args.out / f"{stem}_per_reaction.tsv", sep="\t", float_format="%.6g") + with open(args.out / f"{stem}.json", "w") as fh: + json.dump(summary, fh, indent=2) + if args.save_samples: + np.savez_compressed( + args.save_samples, samples=samples.to_numpy(), reactions=np.array(samples.columns) + ) + print(json.dumps(summary, indent=2)) + + +if __name__ == "__main__": + main() diff --git a/scripts/parameter_tuning/sampling/summarize_chains.py b/scripts/parameter_tuning/sampling/summarize_chains.py new file mode 100644 index 0000000..37f9962 --- /dev/null +++ b/scripts/parameter_tuning/sampling/summarize_chains.py @@ -0,0 +1,88 @@ +#!/usr/bin/env python3 +"""Tables from the chain pickle written by analyze_sampling_convergence.py. + +Loads ``chains_*.pkl`` (a list of sample DataFrames, one per chain, seeds +0..m-1), computes R-hat per reaction with ``_gelman_rubin`` imported from the +analysis script given by ``--analysis-script`` (the raven-toolbox +``scripts/analyze_sampling_convergence.py``), and writes: + +* ``_rhat.tsv``: R-hat per reaction (NaN = constant in every chain, not scored); +* ``.json``: the R-hat summary (reactions, scored, median, + p90, max, counts and shares above 1.01 and 1.1, worst 10) and, per chain, + the lag-1 autocorrelation summary of ``single_chain_autocorr.lag1_table`` + (same definitions as the single-chain runs). + +Inputs: the pickle, the analysis script, ``--tag`` and ``--out``. + +Command line used: + python summarize_chains.py --pickle /chains_yeast-GEM_achr_m4_n300_t100_w1000.pkl \ + --analysis-script $RAVEN_TOOLBOX_DIR/scripts/analyze_sampling_convergence.py \ + --tag s3_yeast_achr_m4_n300_t100 --out results/ +""" + +from __future__ import annotations + +import argparse +import importlib.util +import json +import pickle +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) +from single_chain_autocorr import lag1_table, summarize # noqa: E402 + + +def main() -> None: + p = argparse.ArgumentParser( + description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter + ) + p.add_argument("--pickle", type=Path, required=True) + p.add_argument("--analysis-script", type=Path, required=True) + p.add_argument("--tag", required=True) + p.add_argument("--out", type=Path, required=True) + p.add_argument( + "--wall-s", type=float, default=None, help="wall time printed by the analysis script" + ) + args = p.parse_args() + + spec = importlib.util.spec_from_file_location("asc", args.analysis_script) + asc = importlib.util.module_from_spec(spec) + spec.loader.exec_module(asc) + + with open(args.pickle, "rb") as fh: + chains = pickle.load(fh) + r = asc._gelman_rubin(chains) + scored = r.dropna() + finite = scored[scored != float("inf")] + worst = scored.sort_values(ascending=False).head(10) + out = { + "tag": args.tag, + "pickle": str(args.pickle), + "wall_s": args.wall_s, + "n_chains": len(chains), + "n_samples": int(chains[0].shape[0]), + "n_reactions": int(len(r)), + "n_constant": int(r.isna().sum()), + "n_scored": int(len(scored)), + "n_infinite": int((scored == float("inf")).sum()), + "median": float(scored.median()), + "p90": float(scored.quantile(0.9)), + "max": float(scored.max()), + "max_finite": float(finite.max()) if len(finite) else None, + "n_gt_1.01": int((scored > 1.01).sum()), + "pct_gt_1.01": 100.0 * float((scored > 1.01).mean()), + "n_gt_1.1": int((scored > 1.1).sum()), + "pct_gt_1.1": 100.0 * float((scored > 1.1).mean()), + "worst10": [[k, float(v)] for k, v in worst.items()], + "per_chain_lag1": [summarize(lag1_table(c), c.shape[0]) for c in chains], + } + args.out.mkdir(parents=True, exist_ok=True) + r.to_csv(args.out / f"{args.tag}_rhat.tsv", sep="\t", float_format="%.6g", header=True) + with open(args.out / f"{args.tag}.json", "w") as fh: + json.dump(out, fh, indent=2) + print(json.dumps(out, indent=2)) + + +if __name__ == "__main__": + main() From f9668119fb28fe781f03430afec0fae0cb75c2aa Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Sun, 1 Nov 2026 10:49:21 +0100 Subject: [PATCH 07/20] Update the documented sampling means to the pinned raven-toolbox --- docs/guide/sampling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guide/sampling.md b/docs/guide/sampling.md index 42097d8..95011b7 100644 --- a/docs/guide/sampling.md +++ b/docs/guide/sampling.md @@ -210,7 +210,7 @@ print(f"ethanol: free {result.samples['ethOUT'].mean():.2f}, " ``` ```text -ethanol: free 0.72, growth >= 90% 0.04 +ethanol: free 0.73, growth >= 90% 0.03 ``` ::: :::: From acf2e3f050463ec1c4c5e9585574afce2dbf6ced Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Sun, 1 Nov 2026 11:06:57 +0100 Subject: [PATCH 08/20] Rewrite the migration and comparison pages against the RAVEN 2.11.3, RAVEN 3 and raven-toolbox sources --- docs/guide/fba.md | 2 +- docs/guide/getting-started.md | 2 +- docs/guide/io.md | 2 +- docs/guide/media.md | 2 +- docs/guide/model-structure.md | 2 +- docs/installation/data-and-binaries.md | 4 +- docs/installation/index.md | 2 +- docs/installation/raven.md | 4 +- docs/migrate.md | 9 +- docs/raven3-migration.md | 57 +++------ .../raven3-migration/backward-incompatible.md | 117 +++++++++--------- .../formats-and-reconstruction.md | 106 ++++++++++------ docs/raven3-migration/new-and-environment.md | 98 +++++++-------- .../raven3-migration/structure-and-renames.md | 114 +++++++++-------- docs/raven3-vs-raven-toolbox.md | 42 ++++--- .../same-function-different-answer.md | 53 +++++--- docs/raven3-vs-raven-toolbox/what-each-has.md | 33 +++-- scripts/curated_pairs.yml | 6 +- scripts/known_names.txt | 42 +++++++ 19 files changed, 394 insertions(+), 303 deletions(-) diff --git a/docs/guide/fba.md b/docs/guide/fba.md index c183227..ac0099b 100644 --- a/docs/guide/fba.md +++ b/docs/guide/fba.md @@ -317,4 +317,4 @@ The growth rate is unchanged and the total flux is the smallest that achieves it - [Getting started](getting-started.md), loading and inspecting a model. - [Growth media and conditions](media.md), what the exchange bounds mean and how to define a medium. -- [MATLAB vs Python](../raven3-vs-raven-toolbox.md), the full function mapping. +- [MATLAB vs Python](../matlab-vs-python.md), the full function mapping. diff --git a/docs/guide/getting-started.md b/docs/guide/getting-started.md index 72136f6..43241b0 100644 --- a/docs/guide/getting-started.md +++ b/docs/guide/getting-started.md @@ -272,5 +272,5 @@ An empty dictionary means the reaction balances. Anything listed is an element, ## See also - [Guide overview](index.md), the other pages, and the planned pages. -- [MATLAB vs Python](../raven3-vs-raven-toolbox.md), what each toolbox has, and where cobrapy takes over. +- [Comparison of RAVEN and raven-toolbox](../raven3-vs-raven-toolbox.md), what each toolbox has, and where cobrapy takes over. - [API reference](../api/index.md), every function in both toolboxes. diff --git a/docs/guide/io.md b/docs/guide/io.md index b88d6fe..c1613b5 100644 --- a/docs/guide/io.md +++ b/docs/guide/io.md @@ -266,4 +266,4 @@ print(sorted(p.relative_to(root).as_posix() for p in root.rglob("*") if p.is_fil - [Getting started](getting-started.md), what to do with the model once it is loaded. - [Guide overview](index.md), the other pages and the planned pages. -- [MATLAB vs Python](../raven3-vs-raven-toolbox.md), the full function mapping, including everything that resolves to cobrapy. +- [MATLAB vs Python](../matlab-vs-python.md), the full function mapping, including everything that resolves to cobrapy. diff --git a/docs/guide/media.md b/docs/guide/media.md index e243680..6536101 100644 --- a/docs/guide/media.md +++ b/docs/guide/media.md @@ -358,4 +358,4 @@ The values are uptake rates and not bounds, and the zeros are components that th - [Simulating growth with FBA](fba.md), the solve that uses the medium. - [Reading and writing models](io.md), keeping a model and its conditions in a repository. -- [MATLAB vs Python](../raven3-vs-raven-toolbox.md), the full function mapping. +- [MATLAB vs Python](../matlab-vs-python.md), the full function mapping. diff --git a/docs/guide/model-structure.md b/docs/guide/model-structure.md index 6bda0e8..17a83ef 100644 --- a/docs/guide/model-structure.md +++ b/docs/guide/model-structure.md @@ -327,4 +327,4 @@ RAVEN and the COBRA Toolbox use different field names for the same model. In MAT - [Getting started](getting-started.md), loading a model and inspecting it. - [Reading and writing models](io.md), where identifier prefixes come from. -- [MATLAB vs Python](../raven3-vs-raven-toolbox.md), the full function mapping. +- [MATLAB vs Python](../matlab-vs-python.md), the full function mapping. diff --git a/docs/installation/data-and-binaries.md b/docs/installation/data-and-binaries.md index 88ab3b2..97b41cf 100644 --- a/docs/installation/data-and-binaries.md +++ b/docs/installation/data-and-binaries.md @@ -49,7 +49,7 @@ The path `~/.cache` is used literally on every platform, and the platform conven The version is part of the path, so two KEGG releases coexist and a pinned run fetches the release it was pinned to when that release is missing from the cache. -In MATLAB the KEGG artefacts go to the directory `dataDir`, which `getKEGGModelForOrganism` requires as an argument. The executables are stored in `software/` inside the RAVEN installation directory on every platform. +In MATLAB the KEGG artefacts go to the directory `dataDir`, which `getKEGGModelForOrganism` requires as an argument. The executables are stored in `src/external/` (`blast+`, `diamond`, `hmmer`) inside the RAVEN installation directory on every platform. ## Tool resolution @@ -77,7 +77,7 @@ Downloading on first use puts the download inside the first run, which is not al downloadRavenBinaries ``` -`downloadRavenBinaries` fetches the executables that RAVEN needs into its `software/` directory. The KEGG artefacts are fetched separately, on the first `getKEGGModelForOrganism` call, into the `dataDir` given there. +`downloadRavenBinaries` fetches the executables that RAVEN needs into the `src/external/` directory of the RAVEN installation. The KEGG artefacts are fetched separately, on the first `getKEGGModelForOrganism` call, into the `dataDir` given there. ::: :::{tab-item} 🐍 Python :sync: python diff --git a/docs/installation/index.md b/docs/installation/index.md index 5f0f642..351218a 100644 --- a/docs/installation/index.md +++ b/docs/installation/index.md @@ -28,7 +28,7 @@ Both versions need a linear-programming solver. The options are the same for eit |---|---|---| | Gurobi | Free academic | Genome-scale models, tINIT and ftINIT, gap-filling (recommended) | | GLPK | Open source | Small to medium models, getting started | -| SCIP | Open source | MILP problems (MATLAB) | +| SCIP | Open source | MILP problems (MATLAB); `setRavenSolver('scip')` downloads the binary on Windows only | In MATLAB, `setRavenSolver('gurobi')` selects the active solver. In raven-toolbox, the solver is configured through cobrapy (`cobra.Configuration`). diff --git a/docs/installation/raven.md b/docs/installation/raven.md index 185b202..dc8ba48 100644 --- a/docs/installation/raven.md +++ b/docs/installation/raven.md @@ -4,7 +4,7 @@ The RAVEN Toolbox runs in MATLAB and works independently. It does not require th ## Requirements -- MATLAB R2016b or later. R2016b is the oldest release that the code guards for. Continuous integration tests R2024b only. No additional MathWorks toolboxes are required. +- MATLAB, with no minimum release stated. The code has no check that rejects an older release. `readYAMLmodel` has a fallback for releases before R2020b, `parseTaskList` calls `readcell` (R2019a) for Excel task lists and `addIdentifierPrefix` calls `regexpPattern` (R2020b). Continuous integration tests R2024b only; older releases were not tested. No additional MathWorks toolboxes are required. - A linear-programming solver: [Gurobi](https://www.gurobi.com/) (free academic licence, recommended) or GLPK, bundled with RAVEN itself and needing no separate install. See [Choosing and configuring a solver](#choosing-a-solver-matlab). - RAVEN bundles `libSBML` and the GLPK mex files for Windows, macOS and Linux. `BLAST+`, `DIAMOND` and `HMMER` are not bundled: RAVEN fetches the build for the platform on first use, so the reconstruction functions need internet access the first time they run. `downloadRavenBinaries` fetches them ahead of time. [Download data and binaries](data-and-binaries.md) lists the full set, where it is cached and how to prepare an offline machine. @@ -16,7 +16,7 @@ The RAVEN Toolbox runs in MATLAB and works independently. It does not require th | Solver | LP | MILP | Relative speed | Licence | |---|---|---|---|---| | GLPK | yes | yes (RAVEN warns; slow on large problems) | 1x (baseline) | open source, bundled with RAVEN | -| SCIP | yes | yes | 0.33x | open source; bundled on Windows, a separate install on macOS or Linux | +| SCIP | yes | yes | 0.33x | open source; `setRavenSolver('scip')` downloads the MEX file on Windows (from the RAVEN v2.8.6 GitHub release), a separate install on macOS or Linux | | Gurobi | yes | yes | 4.2x | free academic licence | | COBRA Toolbox | yes | depends on the COBRA solver configured | depends on the COBRA solver configured | depends on the COBRA solver configured | diff --git a/docs/migrate.md b/docs/migrate.md index 7da30ac..6d6c48d 100644 --- a/docs/migrate.md +++ b/docs/migrate.md @@ -1,11 +1,10 @@ -# Migrate +# Migration guides -Two ways of moving between versions or languages, each answering a -different question. +Two pages describe differences between toolboxes. One compares RAVEN 3 with RAVEN 2, the other compares RAVEN 3 with the Python package raven-toolbox. -| | | +| Page | Use | |---|---| -| [RAVEN 2 → RAVEN 3 (MATLAB)](raven3-migration.md) | Upgrading an existing MATLAB codebase: what was renamed, removed outright, or changed between RAVEN 2 and RAVEN 3. | +| [RAVEN 2 → RAVEN 3 (MATLAB)](raven3-migration.md) | Upgrading an existing MATLAB codebase: which functions were renamed or removed, and which behaviour differs between RAVEN 2 and RAVEN 3. | | [RAVEN 3 → raven-toolbox](raven3-vs-raven-toolbox.md) | Choosing between the two current implementations, or porting a MATLAB workflow to Python: what each toolbox has, and where they answer the same question differently. | ```{toctree} diff --git a/docs/raven3-migration.md b/docs/raven3-migration.md index 35990a0..a3f859c 100644 --- a/docs/raven3-migration.md +++ b/docs/raven3-migration.md @@ -1,61 +1,44 @@ # RAVEN 2 to RAVEN 3 -RAVEN 3 changes the MATLAB toolbox in four areas: the folder structure with its removed and renamed functions, backward-incompatible behaviour, file formats and reconstruction pipelines, and new functionality with its dependencies. The calling convention and an upgrade checklist for users of RAVEN 2 follow. The page was compared with RAVEN 2.11.3. The comparison with the Python package is in [Comparison of RAVEN and raven-toolbox](raven3-vs-raven-toolbox.md). +The calling convention and an upgrade checklist for users of RAVEN 2 follow. Four detail pages follow the checklist. All statements on RAVEN 2 refer to RAVEN 2.11.3. The changelog of RAVEN 3 states its changes relative to RAVEN 2.9.3, so changelog entries may describe differences that are absent from the comparison with 2.11.3. The last RAVEN 2 release, 2.11.4, differs from 2.11.3 by bug fixes in `ftINIT`, `mergeLinear`, `replaceMets`, `checkInstallation`, `importModel`, `readYAMLmodel` and `optimizeProb`. The comparison with the Python package is in [Comparison of RAVEN and raven-toolbox](raven3-vs-raven-toolbox.md). -RAVEN 3 is a deliberate breaking release. The codebase was reorganised, low-value and duplicate functions were removed, several external dependencies were dropped and the calling convention for optional arguments changed in the 112 functions that use `parseRAVENargs`. The rationale is in the [v3 development review](https://github.com/CompMetEng/RAVEN/blob/develop/docs/v3_review.md) of the project. Most non-trivial RAVEN 2 scripts need to be checked; the core reconstruction and analysis behaviour is mostly the same. The changes that can affect a script are listed in [Backward-incompatible changes](raven3-migration/backward-incompatible.md#backward-incompatible-changes). +RAVEN 3 is a breaking release. The codebase was reorganised, functions were removed (counts per subsystem are in [Structure, removals and renames](raven3-migration/structure-and-renames.md)), several external dependencies were dropped and the calling convention for optional arguments changed in the 112 functions that use `parseRAVENargs`. The rationale is in the [v3 development review](https://github.com/CompMetEng/RAVEN/blob/develop/docs/v3_review.md) of the project. Scripts that use reconstruction, KEGG, localisation or gene-deletion functions need a check. The changes that can affect a script are listed in [Backward-incompatible changes](raven3-migration/backward-incompatible.md#backward-incompatible-changes). :::{admonition} Changes to check first -:class: important +:class: warning -Two changes are most likely to affect existing scripts: - -- The same homology-based or KEGG-based reconstruction call returns a different draft model, without an error (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)). -- WoLF PSORT-based localisation scores raise an error and have no in-toolbox migration path (see [Backward-incompatible changes](raven3-migration/backward-incompatible.md#backward-incompatible-changes)). +- The same homology-based or KEGG-based reconstruction call returns a different draft model without an error (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)). +- Scripts that use WoLF PSORT localisation scores raise an error (see [Backward-incompatible changes](raven3-migration/backward-incompatible.md#backward-incompatible-changes)). ::: ---- - (calling-convention)= -## Calling convention: positional or named arguments +## Calling convention -RAVEN 2 functions took optional arguments strictly positionally (`f(model, a, b, c)`). A later optional argument could be set only by specifying all optional arguments before it. +RAVEN 2 functions took optional arguments by position (`f(model, a, b, c)`). Setting a later optional argument required all earlier optional arguments to be given. In these pages a position N is the position among all arguments of the call, including the required ones: in `f(model, a, b, c)` the model is position 1 and `a` is position 2. -RAVEN 3 introduces `utils/parseRAVENargs.m`, called by 112 functions. Required leading arguments stay explicit positional parameters; all optional arguments are collected into `varargin` and can be supplied in three interchangeable ways: +RAVEN 3 has the function `utils/parseRAVENargs.m`, which 112 functions call. The required leading arguments are explicit positional parameters. All optional arguments are collected in `varargin` and can be given by position, by name or in a mix of both: ```matlab -removeReactions(model, rxnList, true, true, true) % positional (RAVEN 2 style, still works) +removeReactions(model, rxnList, true, true, true) % positional (RAVEN 2 style) removeReactions(model, rxnList, "removeUnusedGenes", true) % named removeReactions(model, rxnList, true, "removeUnusedComps", true) % hybrid ``` -The first string argument that matches a parameter name starts the name-value pairs. The arguments before it are positional, assigned in the order declared in the `parseRAVENargs` specification, and everything from it on is read as name-value pairs. If no argument matches, or the arguments from the first match on are not valid name-value pairs, all arguments are positional. - -Old positional calls keep working if the new parameter order still matches the old positional order. In 11 functions (`importModel`, `exportModel`, `exportToExcelFormat`, `removeReactions`, `ftINIT`, `fillGaps`, `fitTasks`, `getModelFromHomology`, `addIdentifierPrefix`, `removeIdentifierPrefix` and `exportForGit`) new optional parameters follow the RAVEN 2 ones, so old positional calls keep working unchanged. The old order is kept in the 11 functions checked above. A comparison of all 112 `parseRAVENargs` specifications with the RAVEN 2.11.3 signatures found these exceptions: - -- `getModelFromKEGG` and `getPhylDist` lost the leading `keggPath`, and `getKEGGModelForOrganism` lost `nSequences` and `seqIdentity`, so later positional arguments shift. -- `scoreModel` has new `isozymeScoring`, `complexScoring` and `dataPrecedence` parameters in the middle of the list. -- `writeYAMLmodel` had `(model, fileName, preserveQuotes, sortIds)` and has `(model, fileName, sortIds)`. -- `findGeneDeletions` lost `analysisType`, `refModel` and `oeFactor`. +The first string argument that matches a parameter name (case-insensitive) starts the name-value pairs. The arguments before it are positional and are assigned in the order of the `parseRAVENargs` specification of the function. If no argument matches, or the arguments from the first match on are not valid name-value pairs, all arguments are positional. More positional values than the specification has parameters raise `parseRAVENargs:tooManyArgs`. A positional argument that is a string equal to a parameter name has to be given in the name-value form. -Each exception and its effect is a row in [Backward-incompatible changes](raven3-migration/backward-incompatible.md#backward-incompatible-changes). - -A positional argument that is itself a string matching a parameter name needs the name-value form to avoid ambiguity. - ---- +An old positional call keeps its meaning when the order of the parameters is unchanged. `importModel`, `exportModel`, `exportToExcelFormat`, `removeReactions`, `addIdentifierPrefix`, `removeIdentifierPrefix` and `exportForGit` have the same parameters in the same order in RAVEN 2 and RAVEN 3. `ftINIT`, `fillGaps`, `fitTasks` and `getModelFromHomology` append new optional parameters after the RAVEN 2 ones. A comparison of all 112 specifications with the RAVEN 2.11.3 signatures found six functions where positional calls change meaning or fail: `getModelFromKEGG`, `getPhylDist`, `getKEGGModelForOrganism`, `scoreModel`, `writeYAMLmodel` and `findGeneDeletions`. Each of them has a row in [Backward-incompatible changes](raven3-migration/backward-incompatible.md#backward-incompatible-changes). (upgrade-checklist)= ## Upgrade checklist -1. Update to RAVEN 3.0.0 (or later) and run `checkRaven` (replaces `checkInstallation`). The checks include the installation type, the MATLAB release, the libSBML version, the SBML import and export, the LP solvers, the BLAST+, DIAMOND and HMMER binaries and function names that overlap with other toolboxes. `checkRaven` downloads any of the three binaries that is missing (see [Toolbox dependencies and installation](#toolbox-deps)). -2. Search the scripts for the functions listed in [Removed functionality](raven3-migration/structure-and-renames.md#removed-functionality) and in [Renamed and merged functions](#renamed-merged-and-consolidated-functions), and update or remove those calls. -3. If scripts use `getModelFromKEGG`, `getKEGGModelForOrganism` or `getModelFromHomology`, the draft reconstruction can differ even from an unmodified call: default cutoffs, the KEGG data source and the homology best-hit criterion changed (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)). Rebuild and re-validate any KEGG-derived or homology-derived draft models. Positional calls of the KEGG functions also need a check, because the argument positions shifted (see [Backward-incompatible changes](raven3-migration/backward-incompatible.md#backward-incompatible-changes)). -4. If `.xlsx` curation templates or `importExcelModel` are used, convert the curation data to the `.tsv` format that `curateModelFromTables` expects; there is no drop-in Excel importer (see [Excel I/O](raven3-migration/formats-and-reconstruction.md#excel-io)). -5. If scripts use WoLF PSORT localisation scores (`getWoLFScores(...)` or `parseScores` with `'wolf'` or without a predictor), switch to DeepLoc 2, MULocDeep, COMPARTMENTS or UniProt. There is no in-toolbox migration path for existing WoLF PSORT output files (see [New functionality](#new-functionality)). -6. If workflows include metabolic-task gap-filling (`fitTasks`, `checkTasks`, `ftINIT`), re-run them and check the task pass and fail results. A fixed bug means that a task previously reported as passing can now fail (see [Gap-filling and context-specific model extraction](#ftinit)). -7. If model files are under version control (YAML or SBML), diffs can appear on the next export even without content changes (quoting style, EC-code annotation and ΔG fields; see [File formats and model I/O](raven3-migration/formats-and-reconstruction.md#file-formats-and-model-io)). -8. Re-run the test suite or validation pipeline of the project and compare the outputs (model size, growth rate and task results) with the RAVEN 2 baseline before using RAVEN 3 results in production. - ---- +1. Install RAVEN 3 and run `checkRaven`. It checks the installation type, the MATLAB release, the libSBML version, SBML import and export, the LP solvers, the BLAST+, DIAMOND and HMMER binaries and function names that overlap with other toolboxes. Unless `checkBinaries` is false, `checkRaven` downloads any of the three binaries that is missing (see [Toolbox dependencies and installation](#toolbox-deps)). `checkInstallation` forwards to `checkRaven` and warns on every call. +2. Search the scripts for the functions listed in [Removed functionality](raven3-migration/structure-and-renames.md#removed-functionality) and in [Renamed and merged functions](#renamed-merged-and-consolidated-functions), and replace or remove those calls. The wrappers in `deprecated/` (`makeSomething`, `consumeSomething`, `canProduce`, `canConsume`) and `getINITModel` and `runINIT` work and warn once per session. +3. Rebuild and validate any draft model from `getModelFromKEGG`, `getKEGGModelForOrganism` or `getModelFromHomology`. Default cutoffs, the KEGG data source and the best-hit criterion differ, so an unmodified call gives a different draft (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)). Check positional calls of the six functions listed under [Calling convention](#calling-convention). +4. Convert `.xlsx` curation templates to the `.tsv` format of `curateModelFromTables`. `importExcelModel` does not exist in RAVEN 3 (see [Excel I/O](raven3-migration/formats-and-reconstruction.md#excel-io)). +5. Replace WoLF PSORT scores (`getWoLFScores`, or `parseScores` with `'wolf'`) with DeepLoc 2, MULocDeep, COMPARTMENTS or UniProt. The default predictor of `parseScores` differs (see [Backward-incompatible changes](raven3-migration/backward-incompatible.md#backward-incompatible-changes)). +6. Re-run task-based gap-filling and context-specific model extraction (`fitTasks`, `checkTasks`, `ftINIT`) and compare the task results. `simplifyModel` keeps the boundary constraint of a task on a metabolite without reactions, which RAVEN 2 removed, so the task check can give a different result in RAVEN 2 and RAVEN 3 (not run; see [Gap-filling and context-specific model extraction](#ftinit)). `fitTasks` returns a third output, `failedTasks`. +7. Expect differences in YAML and SBML files on the next export of a model under version control. The changelog of RAVEN 3 lists changes in quoting, EC-code annotation and ΔG fields; these differences were not compared file by file (see [File formats and model I/O](raven3-migration/formats-and-reconstruction.md#file-formats-and-model-io)). +8. Compare the output of the validation pipeline used for the model (model size, growth rate and task results) with the RAVEN 2 output before using RAVEN 3 results. ## Detail pages @@ -63,7 +46,7 @@ A positional argument that is itself a string matching a parameter name needs th |---|---| | [Structure, removals and renames](raven3-migration/structure-and-renames.md) | The folder reorganisation, every removed function and subsystem, and every renamed or merged function. | | [Backward-incompatible changes](raven3-migration/backward-incompatible.md) | Changes that raise errors, changes that alter results without an error, changes to error and warning identifiers, changes that raise warnings and installation changes. | -| [File formats, reconstruction and GPR parsing](raven3-migration/formats-and-reconstruction.md) | SBML, Excel and YAML I/O; the KEGG and homology reconstruction pipelines; the new grRule parser. | +| [File formats, reconstruction and GPR parsing](raven3-migration/formats-and-reconstruction.md) | SBML, Excel and YAML I/O; the KEGG and homology reconstruction pipelines; the grRule parser. | | [New functionality, dependencies and error handling](raven3-migration/new-and-environment.md) | Gap-filling and ftINIT changes, functions added in RAVEN 3, the toolbox dependency comparison, and error and warning handling. | ```{toctree} diff --git a/docs/raven3-migration/backward-incompatible.md b/docs/raven3-migration/backward-incompatible.md index 9e0209e..72e0810 100644 --- a/docs/raven3-migration/backward-incompatible.md +++ b/docs/raven3-migration/backward-incompatible.md @@ -1,82 +1,83 @@ (backward-incompatible-changes)= # Backward-incompatible changes -The list covers the changes found by comparing RAVEN 2.11.3 with RAVEN 3. The changes are grouped into those that raise an error, those that give a different result without an error, those to error and warning identifiers, those that raise warnings and installation changes. +The list covers the changes found by comparing RAVEN 2.11.3 with RAVEN 3 in the code. The changelog of RAVEN 3 lists further bug fixes, which are named in the last row of the table of corrected behaviour. The changes are grouped into those that raise an error, those that give a different result without an error, changes to error and warning identifiers, changes that raise warnings and installation changes. ## Changes that raise errors | RAVEN 2 usage | RAVEN 3 behaviour | |---|---| -| `importExcelModel(...)` | Removed; use `curation/curateModelFromTables.m` (input format `.tsv`, not `.xlsx`) or convert the template to YAML or SBML and use `importModel` (see [Excel I/O](#excel-io)) | -| `importModel(...)` on SBML Level < 3, Level 3 without FBC v2, or any non-FBC SBML | Raises an error; only SBML Level 3 Version 1, FBC package version 2 is accepted (see [SBML I/O](#sbml-io)) | -| `exportToExcelFormat(model, fileName)` where `fileName` is a bare path (no `.xlsx`) | No fallback to tab-delimited text export; raises an error with the message "only export to xlsx format is supported" | -| `exportToTabDelimited(...)` | Removed (dead-code path) | -| `getWoLFScores(...)` | Removed; WoLF PSORT support is fully removed | -| `parseScores(..., 'wolf', ...)` | Raises an error on `'predictor','wolf'`, with a message pointing to `'deeploc'`, `'mulocdeep'`, `'compartments'` or `'uniprot'`; there is no in-toolbox migration path for existing WoLF PSORT output files | -| `qMOMA(...)`, `solveQP(...)` | Removed (the last dependency of RAVEN on `quadprog` from the Optimization Toolbox) | -| `findGeneDeletions(...)` with `analysisType`, `refModel` or `oeFactor` | Arguments removed together with `qMOMA` (MOMA-based over-expression analysis); only single and double gene deletion (`sgd`, `dgd`) by flux balance analysis (FBA) remains. More than the `'testType'` argument raises `parseRAVENargs:tooManyArgs` | -| `checkRxn(...)` | Removed, no direct successor; to debug why a reaction cannot carry flux, use `gapfilling/findLeakMetabolite.m`, `gapfilling/canExchange.m` or `gapfilling/checkProduction.m` | -| `dispEM(...)`, `followFluxes(...)`, `followChanged(...)`, `printModel(...)`, `mapCompartments(...)`, `getExpressionStructure(...)`, `getMetsInComp(...)` | All removed (see [Removed functionality](#removed-functionality) for the replacements, where one exists) | -| Anything under `external/metacyc/` | Subsystem removed: MetaCyc-based reconstruction; no in-toolbox replacement | -| Anything under `pathway/` or `plotting/` | Subsystems removed: all visualisation and plotting functionality (16 functions, such as `drawMap`, `drawPathway`, `markPathwayWithFluxes` and `setOmicDataToRxns`); no in-toolbox replacement, use the COBRA Toolbox or Escher | -| Anything under `legacy/` | Subsystem removed; no in-toolbox replacement (see [Removed functionality](#removed-functionality)) | -| `getGenesFromKEGG`, `getMetsFromKEGG`, `getRxnsFromKEGG`, `constructMultiFasta`, `getWSLpath`, `getBlastFromExcel` | Removed with the local KEGG FTP dump workflow (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)) | -| `getToolboxVersion(...)` | Removed as a standalone function; now part of an internal helper of `exportForGit` | +| `importExcelModel(...)` | Removed. Use `curation/curateModelFromTables.m` (input format `.tsv`), or convert the template to SBML and use `importModel`, or to YAML and use `readYAMLmodel` (see [Excel I/O](#excel-io)) | +| `importModel(...)` on SBML Level < 3, Level 3 without FBC v2, or any non-FBC SBML | Raises an error. The accepted format is SBML Level 3 Version 1 with FBC package version 2 (see [SBML I/O](#sbml-io)) | +| `exportToExcelFormat(model, fileName)` where `fileName` has no file name or no extension | RAVEN 2 exported tab-delimited text. RAVEN 3 raises an error for every extension other than `.xlsx`, including an empty one, with the message "only export to xlsx format is supported". RAVEN 2 already raised this error for a file name with another extension | +| `exportToTabDelimited(...)` | Removed | +| `getWoLFScores(...)` | Removed. WoLF PSORT support is removed | +| `parseScores(..., 'wolf', ...)` | Raises an error with a message that names `'deeploc'`, `'mulocdeep'`, `'compartments'` and `'uniprot'`. RAVEN 3 has no function that reads WoLF PSORT output files | +| `qMOMA(...)`, `solveQP(...)` | Removed. RAVEN 3 has no code that depends on `quadprog` of the Optimization Toolbox | +| `findGeneDeletions(...)` with `analysisType`, `refModel` or `oeFactor` | The arguments are removed together with `qMOMA` (MOMA-based over-expression analysis). The test types `'sgo'` and `'dgo'` are removed. Single and double gene deletion (`sgd`, `dgd`) by flux balance analysis (FBA) remain. More than the `'testType'` argument raises `parseRAVENargs:tooManyArgs` | +| `checkRxn(...)` | Removed, no direct successor. To find why a reaction cannot carry flux, use `gapfilling/findLeakMetabolite.m`, `gapfilling/canExchange.m` or `gapfilling/checkProduction.m` | +| `dispEM(...)`, `followFluxes(...)`, `followChanged(...)`, `printModel(...)`, `mapCompartments(...)`, `getExpressionStructure(...)`, `getMetsInComp(...)` | Removed (see [Removed functionality](#removed-functionality) for the replacements, where one exists) | +| Anything under `external/metacyc/` | Subsystem removed (MetaCyc-based reconstruction, 7 functions). No replacement in RAVEN 3 | +| Anything under `pathway/` or `plotting/` | Subsystems removed: 16 functions for visualisation and plotting, such as `drawMap`, `drawPathway`, `markPathwayWithFluxes` and `setOmicDataToRxns`. No replacement in RAVEN 3; the COBRA Toolbox and Escher provide plotting | +| Anything under `legacy/` | Subsystem removed. No replacement in RAVEN 3 (see [Removed functionality](#removed-functionality)) | +| `getGenesFromKEGG`, `getMetsFromKEGG`, `getRxnsFromKEGG`, `constructMultiFasta`, `getWSLpath`, `getBlastFromExcel` | Removed (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)) | +| `getToolboxVersion(...)` | Removed as a standalone function. It is a local function of `exportForGit` | | `parallelPoolRAVEN(...)` | Renamed `parallelWorkersRAVEN`, with a different signature and output (see [Renamed and merged functions](#renamed-merged-and-consolidated-functions)) | -| `getModelFromKEGG(keggPath, keepSpontaneous, keepUndefinedStoich, keepIncomplete, keepGeneral)` | Raises `parseRAVENargs:tooManyArgs`, because `keggPath` is removed and the function accepts four optional arguments | -| `getKEGGModelForOrganism(...)` with 13 to 15 positional arguments | `nSequences` and `seqIdentity` are removed. 14 or 15 arguments raise `parseRAVENargs:tooManyArgs`; a non-empty `nSequences` (13th) is read as `globalModel` and raises an error | +| `getModelFromKEGG(keggPath, keepSpontaneous, keepUndefinedStoich, keepIncomplete, keepGeneral)` | Raises `parseRAVENargs:tooManyArgs`, because `keggPath` is removed and the function has four optional parameters | +| `getKEGGModelForOrganism(...)` with 13 to 15 positional arguments | The parameters `nSequences` and `seqIdentity` are removed. 14 or 15 arguments raise `parseRAVENargs:tooManyArgs`. A non-empty 13th argument (`nSequences` in RAVEN 2) is read as `globalModel` | | `getPhylDist(keggPath, onlyInKingdom)` | Raises `parseRAVENargs:tooManyArgs`, because `keggPath` is removed | -| `writeYAMLmodel(model, fileName, preserveQuotes, sortIds)` with four arguments | Raises `parseRAVENargs:tooManyArgs`; RAVEN 3 has `(model, fileName, sortIds)` | -| `scoreModel(...)` with `'best'` as the seventh or eighth argument, or `hpaLevelScores` as the ninth | Raises `RAVEN:badInput` for `'best'`, which is not a valid value of `isozymeScoring` or `complexScoring`; a structure as the ninth argument raises an error, because that position now holds `multipleCellScoring` | +| `writeYAMLmodel(model, fileName, preserveQuotes, sortIds)` with four arguments | Raises `parseRAVENargs:tooManyArgs`. RAVEN 3 has `(model, fileName, sortIds)` | +| `scoreModel(...)` with `'best'` at position 7, 8 or 9 among all arguments | Raises `RAVEN:badInput`. Positions 7, 8 and 9 are `isozymeScoring`, `complexScoring` and `multipleCellScoring`. The first two accept `'min'`, `'max'`, `'median'` and `'average'`, the third accepts `'max'` and `'average'` | +| `scoreModel(...)` with `hpaLevelScores` at position 9 among all arguments | Raises an error. Position 9 is `multipleCellScoring`, and `hpaLevelScores` is position 10 | ## Changes that alter results without an error -The same call with the same input model gives different output. +The same call with the same input model gives a different output. The changes fall into two groups: argument positions and defaults that change meaning, and corrected behaviour. Argument positions count all arguments of the call, including the required ones: in `scoreModel(model, hpaData, arrayData, tissue, celltype, ...)` the model is position 1 and `celltype` is position 5. -| Function | RAVEN 3 behaviour (default, unless noted) | -|---|---| -| `getModelFromHomology`, `strictness=3` | Best-hit tie-break uses bitscore; RAVEN 2.11.3 used E-value, and no option restores that criterion (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)) | -| `getModelFromHomology`, `minLen` | Default lowered from `200` to `100` (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)) | -| `getModelFromHomology`, merging of gene-reaction rules (GPR) | Gene substitution is based on exact tokens; RAVEN 2 used regex or `strrep` (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)) | -| `getKEGGModelForOrganism`, `cutOff` | Default lowered from `10^-50` to `10^-30` (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)) | -| `getKEGGModelForOrganism`, `minScoreRatioG` | Default raised from `0.8` to `0.9` (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)) | -| `removeGenes`, `expandModel`, `addRxns` | RAVEN 3 parses `grRules` by exact gene-token matching; RAVEN 2 used substring or regex matching (see [Parsing of gene-reaction rules](#gpr-parsing)) | -| `findPotentialErrors` | RAVEN 3 parses the rule into a tree; the RAVEN 2.11.3 check inside `standardizeGrRules` searched for string patterns (see [Parsing of gene-reaction rules](#gpr-parsing)) | -| `changeGrRules(..., 'replace', false)` onto a gene-less reaction | Produces a valid grRule; RAVEN 2 produced an unparseable one (see [Parsing of gene-reaction rules](#gpr-parsing)) | -| `simplifyModel` (used by task-based gap-filling) | Keeps the boundary constraint of a task on a metabolite without reactions (see [Gap-filling and context-specific model extraction](#ftinit)) | -| `ftINITInternalAlg` | With `allowExcretion` true, RAVEN 3 constrains the metabolite balance to `S*v >= 0`; RAVEN 2 had the sense of the constraint inverted | -| `ravenCobraWrapper` | Gene-ID-to-MIRIAM matching handles gene IDs that contain regex metacharacters; RAVEN 2 mis-matched them (see [RAVEN⇄COBRA interoperability](#raven-cobra-interoperability)) | -| `mergeCompartments` | The merged metabolite is the copy already in compartment `s`, otherwise the copy whose ID sorts first (character code order), with every field of that copy | -| `simplifyModel` with `constrainReversible` on a model without a reversible reaction | This step changes nothing; RAVEN 2 raised an error, because `getAllowedBounds` took the empty reaction list for all reactions | -| `getModelFromKEGG(keggPath)` and `getModelFromKEGG(keggPath, keepSpontaneous)` | `keggPath` is read as `keepSpontaneous` and has no effect; the second argument is read as `keepUndefinedStoich`, so the wrong class of reactions is removed | -| `getPhylDist(keggPath)` | `keggPath` is read as `'onlyInKingdom'` and has no effect; the full distance matrix is returned | -| `writeYAMLmodel(model, fileName, true)` | The third argument is read as `sortIds`, so IDs are sorted; RAVEN 2.11.3 read it as `'preserveQuotes'` | -| `scoreModel` | Reactions with an `and` rule get the minimum over the genes by default (`complexScoring`); RAVEN 2.11.3 scored every reaction with the maximum over all genes | -| `scoreModel(..., celltype, noGeneScore, 'average')` | `'average'` as the seventh argument is read as `isozymeScoring`, so it no longer applies to `and` rules, which use `complexScoring` | -| `parseScores(file)` | The default predictor is `'deeploc'`; RAVEN 2.11.3 used `'wolf'`. WoLF PSORT output is parsed as DeepLoc output | -| `permuteModel`, `randomSampling`, `optimizeProb`, `getFluxZ`, `convertToIrrev`, `replaceMets`, `setExchangeBounds`, `fitParameters`, `guessComposition` | Each function had a bug in RAVEN 2; output differs only for models and inputs that triggered it | - -`removeGenes`, `expandModel` and `findPotentialErrors` parse the rule into a tree, which handles isozymes and complexes. `addRxns` only splits the rule on space-bounded `and` and `or` tokens and builds no tree. - -`simplifyModel` in RAVEN 2.11.3 dropped the boundary constraint of a task on a metabolite without reactions, so a task that RAVEN 2.11.3 reported as passing can fail in RAVEN 3. - -`mergeCompartments` in RAVEN 2.11.3 kept the copy that came first in `model.mets`. +### Argument positions and defaults that change meaning -The `scoreModel` of RAVEN 3 takes `celltype`, `noGeneScore`, `isozymeScoring`, `complexScoring`, `multipleCellScoring`, `hpaLevelScores` and `dataPrecedence` (default `'gene'`) in this order. `multipleCellScoring` accepts `'max'` and `'average'`. - -None of the changes in this table is an API change. +| Function | RAVEN 3 behaviour | +|---|---| +| `getModelFromKEGG(keggPath)` to `getModelFromKEGG(keggPath, keepSpontaneous, keepUndefinedStoich[, keepIncomplete])` | Calls with one to four positional arguments shift. The first argument (`keggPath` in RAVEN 2) is read as `keepSpontaneous` and has no effect. Each later argument is read as the next parameter in the RAVEN 2 order, so a different class of reactions is removed | +| `getPhylDist(keggPath)` | `keggPath` is read as `'onlyInKingdom'` and has no effect. The full distance matrix is returned | +| `writeYAMLmodel(model, fileName, true)` | The third argument is read as `sortIds`, so IDs are sorted. RAVEN 2 read it as `'preserveQuotes'` | +| `scoreModel` | `isozymeScoring` and `complexScoring` replace the single `multipleGeneScoring` of RAVEN 2. Reactions with an `and` rule get the minimum over the genes by default (`complexScoring`). RAVEN 2 scored every reaction with the maximum over all genes by default (`multipleGeneScoring` `'best'`) | +| `scoreModel(..., celltype, noGeneScore, 'average')` | `'average'` at position 7 among all arguments (`multipleGeneScoring` in RAVEN 2) is read as `isozymeScoring`, so it does not apply to `and` rules, which use `complexScoring` | +| `scoreModel(..., celltype, noGeneScore, multipleGeneScoring, 'average')` | `'average'` at position 8 among all arguments (`multipleCellScoring` in RAVEN 2) is read as `complexScoring` without an error, so the average is taken over the genes of an `and` rule and no longer over cell types | +| `scoreModel` output `geneScores` | Genes that are absent from the data receive `NaN`. The RAVEN 2 documentation states `-Inf`. | +| `scoreModel`, value names | RAVEN 2 accepted `'best'` for `multipleGeneScoring` and `multipleCellScoring`. RAVEN 3 uses `'max'` for `isozymeScoring`, `complexScoring` and `multipleCellScoring`. The parameter order is `celltype`, `noGeneScore`, `isozymeScoring`, `complexScoring`, `multipleCellScoring`, `hpaLevelScores` and `dataPrecedence` (default `'gene'`) | +| `parseScores(file)` | The default predictor is `'deeploc'`. RAVEN 2 used `'wolf'`. How a WoLF PSORT file is handled without a predictor was not run | +| `getModelFromHomology`, `strictness=3` | The best hit is the hit with the highest bitscore. RAVEN 2 selected the hit with the lowest E-value. No parameter selects the criterion (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)) | +| `getModelFromHomology`, `minLen` | The default is `100`, in RAVEN 2 `200` | +| `getKEGGModelForOrganism`, `cutOff` | The default is `10^-30`, in RAVEN 2 `10^-50` | +| `getKEGGModelForOrganism`, `minScoreRatioG` | The default is `0.9`, in RAVEN 2 `0.8` | + +### Corrected behaviour + +| Function | RAVEN 3 behaviour | +|---|---| +| `getModelFromHomology`, merging of gene-reaction rules (GPR) | Gene substitution uses exact tokens. RAVEN 2 used regular expressions or `strrep` | +| `removeGenes`, `addRxns` | `removeGenes` evaluates the rule as a tree, so isozymes and complexes are handled. `addRxns` splits a rule at space-bounded `and` and `or` tokens, so a gene ID such as `band1` is not cut, and builds no tree. RAVEN 2 used substring matching. `expandModel` parses the rule with `grRuleToDNF`, leaves a rule that cannot be parsed (`RAVEN:badGrRule`) unchanged and lists the reaction in `rxnToCheck`, and raises `RAVEN:grRuleTooComplex` for a rule that expands to more than 10000 isozymes; RAVEN 2 split the rule string at `' or '` and `' and '` (see [Parsing of gene-reaction rules](#gpr-parsing)) | +| `findPotentialErrors` | The changelog of RAVEN 3 states that the check uses a parse tree. The effect on results was not compared (see [Parsing of gene-reaction rules](#gpr-parsing)) | +| `changeGrRules(..., 'replace', false)` onto a reaction without genes | Produces a valid grRule. RAVEN 2 wrapped the new rule as `() or (X)`, which RAVEN 3 cannot parse | +| `simplifyModel` | Keeps the boundary constraint (`b`) of a task on a metabolite without reactions. RAVEN 2.11.3 removed such metabolites, so the task check can give a different result in RAVEN 3 (not run; see [Gap-filling and context-specific model extraction](#ftinit)) | +| `simplifyModel` with `constrainReversible` on a model without a reversible reaction | The step changes nothing. RAVEN 2 raised an error, because `getAllowedBounds` takes an empty reaction list as all reactions (error taken from the changelog; not reproduced) | +| `ftINITInternalAlg` | With `allowExcretion` true, the metabolite balance is constrained to `S*v >= 0`. RAVEN 2 had the sense of the constraint inverted | +| `ravenCobraWrapper` | The conversion of `grRules` to the COBRA `rules` field handles gene IDs that contain regular-expression metacharacters. RAVEN 2 did not escape them (see [RAVEN⇄COBRA interoperability](#raven-cobra-interoperability)) | +| `mergeCompartments` | The merged metabolite is the copy in compartment `s`, otherwise the copy whose ID sorts first (character code order), with every field of that copy. RAVEN 2 kept the copy that came first in `model.mets` | +| `permuteModel`, `optimizeProb`, `getFluxZ`, `convertToIrrev`, `replaceMets`, `setExchangeBounds`, `fitParameters`, `guessComposition` | The changelog of RAVEN 3 lists a bug fix for each function. The effect on results was not checked. The default `method` of `randomSampling` is `'achr'`, so `randomSampling(model, n)` no longer runs the random-objective method of RAVEN 2 (`'randomObjective'` selects it); `goodRxns` is empty for `'achr'` and `'chrr'`; the function has the parameters `nObjectives`, `seed`, `method`, `thinning` and `nBurnin` and a third output `info`. The changelog lists further fixes that can change results, for example for `mergeLinear`, `mergeModels`, `closeModel`, `addTransport`, `getExchangeRxns`, `removeLowScoreGenes`, `getMinNrFluxes`, `reporterMetabolites`, `getElementalBalance`, `checkModelStruct` and `deleteUnusedGenes`; these were not checked | ## Changes to error and warning identifiers -- `dispEM`, the universal error and warning function of RAVEN 2, always raised errors with an empty identifier (`MException('', ...)`), so no RAVEN 2 script could legitimately have caught on `err.identifier`. RAVEN 3 replaces `dispEM` with native `error()` and `warning()` calls. Calls that carry an identifier predominantly use `RAVEN:badInput` (220 calls) and `RAVEN:warning` (63 calls), plus other identifiers, for example `RAVEN:modelError`, `RAVEN:badGrRule`, `RAVEN:infeasibleModel`, `RAVEN:infeasible`, `RAVEN:sampling` and `RAVEN:grRuleTooComplex`. Not every call carries an identifier. -- `try/catch` blocks that match on message text keep working for messages whose text is unchanged from RAVEN 2. Scripts can match on `err.identifier` for the calls that carry one. +- RAVEN 2 raised errors through `dispEM` with an empty identifier (`MException('', ...)`), so no RAVEN 2 script could catch on `err.identifier`. RAVEN 3 replaces `dispEM` with `error()` and `warning()` calls. `RAVEN:badInput` is used in 220 lines and `RAVEN:warning` in 63 lines of `src/` and the root `.m` files (non-comment lines that contain the identifier). Further identifiers are `RAVEN:modelError`, `RAVEN:badGrRule`, `RAVEN:infeasibleModel`, `RAVEN:infeasible`, `RAVEN:sampling`, `RAVEN:grRuleTooComplex`, `RAVEN:deprecated`, `RAVEN:legacyMethod`, `RAVEN:notMainBranch`, `RAVEN:glpkMILP`, `RAVEN:fetchToolboxAsset:unverified`, `RAVEN:fetchToolboxAsset:checksum`, `RAVEN:fetchToolboxAsset:manifest`, `RAVEN:runSuppOrth:supportLv`, `RAVEN:runSuppOrth:failed`, `RAVEN:runSuppOrth:notFound`, `RAVEN:hitsFromSuppOrth:unmappedProteins`, `parseRAVENargs:tooManyArgs`, `parseRAVENargs:noSpec` and `getPhylDist:noData`. The list is not exhaustive: identifiers in other namespaces, such as `renameModelGenes:*`, `setGAM:*` and `parseYAML:*`, also exist. Not every call carries an identifier. Scripts can match on `err.identifier` for the calls that carry one. +- Whether `try/catch` blocks that match on message text still match was not compared message by message. ## Changes that raise warnings -- `getINITModel` and `runINIT` issue a one-time "legacy" notice that points at `ftINIT`; `warning('off','RAVEN:legacyMethod')` silences it. The notice is not a deprecation: the RAVEN documentation states that these functions "remain supported" indefinitely, to reproduce previously published Human-GEM-derived models. See [Gap-filling and context-specific model extraction](#ftinit). -- `makeSomething`, `consumeSomething`, `canProduce` and `canConsume` are in a new `deprecated/` folder as wrappers around `findLeakMetabolite` and `canExchange`. They issue a one-time `RAVEN:deprecated` warning that states they "will be removed in the next major release." The wrappers are scheduled for removal in the next major release, so calls should be migrated before then. See [Gap-filling and context-specific model extraction](#ftinit). -- `checkInstallation` is now `checkRaven`, moved from `installation/` to the repository root, with an unchanged signature (`[currVer, installType] = checkRaven(developMode, checkBinaries)`). Existing calls keep working: `checkInstallation` remains as a forwarding wrapper that passes every argument and output through, after issuing a `RAVEN:deprecated` warning that names `checkRaven`. The wrapper is scheduled for removal in the next major release, like the functions in `deprecated/`, so calls should be migrated before then. +- `getINITModel` and `runINIT` issue a one-time notice with the identifier `RAVEN:legacyMethod` that points at `ftINIT`. `warning('off','RAVEN:legacyMethod')` silences it. The help text of both functions states that the function remains supported, although the changelog of RAVEN 3 marks both functions deprecated. See [Gap-filling and context-specific model extraction](#ftinit). +- `makeSomething`, `consumeSomething`, `canProduce` and `canConsume` are wrappers in the folder `deprecated/` around `findLeakMetabolite` and `canExchange`. They issue a one-time `RAVEN:deprecated` warning that states that they will be removed in the next major release. +- `checkInstallation` is `checkRaven` in RAVEN 3, moved from `installation/` to the repository root, with the same signature (`[currVer, installType] = checkRaven(developMode, checkBinaries)`). `checkInstallation` forwards every argument and output to `checkRaven` and issues a `RAVEN:deprecated` warning that names `checkRaven` on every call. The wrapper will be removed in the next major release. ## Installation changes -The KEGG HMM libraries and the external command-line binaries (BLAST+, DIAMOND, HMMER, cd-hit, MAFFT and WoLF PSORT) are no longer committed to the repository. RAVEN 3 fetches the KEGG data and the BLAST+, DIAMOND and HMMER binaries from the [`toolbox-assets`](https://github.com/CompMetEng/toolbox-assets) repository the first time a function needs them, so internet access is required on first use as well as at clone or install time. What is fetched and where it is cached is described in [Download data and binaries](../installation/data-and-binaries.md). The air-gapped install and the full dependency comparison are in [Toolbox dependencies and installation](#toolbox-deps). +RAVEN 3 does not contain the KEGG HMM libraries and the external binaries (BLAST+, DIAMOND, HMMER, cd-hit, MAFFT, WoLF PSORT and SCIP) or Apache POI in the repository. It fetches the KEGG HMM libraries and reference tables and the BLAST+, DIAMOND and HMMER binaries from the [`toolbox-assets`](https://github.com/CompMetEng/toolbox-assets) repository the first time a function needs them, so internet access is required on first use. The binaries are stored in `src/external/`. SCIP for Windows is downloaded by `setRavenSolver` from the RAVEN v2.8.6 GitHub release. An offline bundle of the binaries is available as a `*-binaries` RAVEN release. What is fetched and where it is cached is described in [Download data and binaries](../installation/data-and-binaries.md). The air-gapped install and the dependency comparison are in [Toolbox dependencies and installation](#toolbox-deps). diff --git a/docs/raven3-migration/formats-and-reconstruction.md b/docs/raven3-migration/formats-and-reconstruction.md index fbfed7f..9af0ef4 100644 --- a/docs/raven3-migration/formats-and-reconstruction.md +++ b/docs/raven3-migration/formats-and-reconstruction.md @@ -1,6 +1,6 @@ # File formats, reconstruction and GPR parsing -Model file formats, draft reconstruction and the parsing of gene-reaction rules (`grRules`, abbreviated GPR) differ between RAVEN 2 and RAVEN 3. +Model file formats, draft reconstruction and the parsing of gene-reaction rules (`grRules`, abbreviated GPR) differ between RAVEN 2 and RAVEN 3. The sections describe the changes that affect scripts and the differences that appear in version-controlled model files. (file-formats-and-model-io)= ## File formats and model I/O @@ -11,39 +11,57 @@ Model file formats, draft reconstruction and the parsing of gene-reaction rules | Aspect | RAVEN 2 | RAVEN 3 | |---|---|---| | `exportModel` output format | SBML Level 3 Version 1, FBC package version 2 | Unchanged | -| `importModel` accepted input | Any SBML level and version; L2-style notes-based gene associations (`GENE_ASSOCIATION` and `PROTEIN_ASSOCIATION`) also read; FBC fields optional | Only SBML Level 3 Version 1, FBC package version 2; anything else raises an error | -| MIRIAM prefixes read on import | `urn:miriam:`, `http://identifiers.org/`, `https://identifiers.org/` | Unchanged | +| `importModel` accepted input | Any SBML level and version; the notes-based gene association `GENE_ASSOCIATION` of Level 2 files is read, and `SHORT NAME` is read for gene names; FBC fields are optional | SBML Level 3 Version 1 with FBC package version 2; any other file raises the error `RAVEN:badInput` | +| MIRIAM prefixes read on import | `urn:miriam:`, `http://identifiers.org/` and `https://identifiers.org/` | `http://identifiers.org/` and `https://identifiers.org/` | -Older SBML files have to be converted before `importModel` accepts them, for example by re-exporting them from cobrapy or the COBRA Toolbox. +`importModel` of RAVEN 3 reads MIRIAM annotations (reaction EC codes, model and species annotations) with the `http://identifiers.org/` and `https://identifiers.org/` prefixes and ignores `urn:miriam:` annotations. + +:::{admonition} Cross-references in older SBML files +:class: warning +An SBML file that uses `urn:miriam:` annotations is imported without an error, and its cross-references (for example `metMiriams`) are absent from the imported model. RAVEN 2 writes `https://identifiers.org/` prefixes (`exportModel`), so its own files are not affected. Files from other tools that use `urn:miriam:` can be converted to the `identifiers.org` form before import. +::: + +A file in an older SBML level or version has to be converted before `importModel` accepts it. The error message suggests another tool or an older RAVEN version for the conversion. (excel-io)= ### Excel I/O | Aspect | RAVEN 2 | RAVEN 3 | |---|---|---| -| Excel engine | Apache POI (Java), bundled | Dependency-free OOXML writer (`io/writeExcel.m`) | -| Full-model `.xlsx` import | `importExcelModel.m` (`RXNS`, `METS`, `COMPS`, `GENES` and `MODEL` sheets → complete model) | Removed, no equivalent importer | -| Tabular curation of an existing model | No equivalent | `curation/curateModelFromTables.m` curates or extends a loaded model from separate `.tsv` files (metabolites by name and compartment, reactions by stoichiometry, genes by name); it is not an importer | -| `exportToExcelFormat` on a bare path (no `.xlsx`) | Fell back to tab-delimited text export | Raises an error, `.xlsx` only | +| Excel engine | Apache POI (Java), bundled with RAVEN | Dependency-free OOXML writer (`io/writeExcel.m`) | +| Full-model `.xlsx` import | `importExcelModel.m` (sheets `RXNS`, `METS`, `COMPS`, `GENES` and `MODEL`) builds a complete model | Removed | +| Tabular curation of an existing model | No equivalent | `curation/curateModelFromTables.m` curates or extends a loaded model from `.tsv` files (metabolites by name and compartment, reactions by stoichiometry, genes by name) | +| `exportToExcelFormat` with a path that has no file name or extension | Falls back to tab-delimited text export | Raises an error; the output is `.xlsx` | -A pipeline that depends on the RAVEN 2 Excel curation template can convert the template to the `.tsv` format that `curateModelFromTables` expects, or build the model with `importModel` or YAML and use `curateModelFromTables` for incremental edits. +A pipeline that depends on the RAVEN 2 Excel curation template can convert the template to the `.tsv` format that `curateModelFromTables` expects, or build the model with `importModel` or `readYAMLmodel` and use `curateModelFromTables` for incremental edits. +(yaml-io)= ### YAML I/O -`readYAMLmodel` and `writeYAMLmodel` are pure MATLAB in RAVEN 3 and were in RAVEN 2, so model YAML I/O has no Python dependency in either version. `io/parseYAML.m` is a new pure-MATLAB generic YAML parser, used only by `conditions/applyCondition.m` to read condition files. RAVEN 3 writes model YAML with the following differences, which appear as diffs in version-controlled model files: +`readYAMLmodel` and `writeYAMLmodel` are pure MATLAB in both versions, so model YAML I/O has no Python dependency in either. `io/parseYAML.m` is a generic pure-MATLAB YAML parser that `conditions/applyCondition.m` uses to read condition files. + +`writeYAMLmodel` of RAVEN 3 takes name-value arguments and quotes strings by a fixed rule in place of the RAVEN 2 option `preserveQuotes`. The output differs from RAVEN 2 in the following points, which appear as diffs in version-controlled model files. | Aspect | RAVEN 2 | RAVEN 3 | |---|---|---| -| Quote style | Single quotes | Double quotes | -| Reaction EC codes | Top-level `eccodes` field only | Also written under `annotation: ec-code` | -| Gene-less model | `genes:` section omitted | `genes: []` written explicitly | -| Empty `gene_reaction_rule` | Written as an empty string | Omitted | -| `objective_coefficient` on import | Ignored | Honoured | +| Quoting | Every string is quoted with double quotes (default `preserveQuotes` is true) | A string is quoted with double quotes where YAML needs it (reserved words, numbers, leading indicators, `: `); other strings are bare | +| Indentation | List entries ` - !!omap` (four spaces) with keys indented by six spaces | List entries ` - !!omap` (two spaces) with keys indented by four spaces | +| Reaction EC codes | Top-level `eccodes` field | Under `annotation: ec-code`; the top-level key is not written | +| Metabolite SMILES | Top-level field | Under `annotation` as `smiles` | +| Reaction notes key | `rxnNotes` | `notes` | +| Gene-less model | `genes:` is omitted when `model.genes` is absent; an empty field gives a bare `- genes:` | The line `- genes: []` | +| Document start | `---` line, then `!!omap` | `!!omap` only | +| Field order, metabolites | `formula`, then `charge` | `charge`, then `formula` | +| Field order, reactions | `rxnFrom` directly after `gene_reaction_rule`; `references` before `subsystem` and `annotation` | `subsystem` and `annotation` first; `references` and `rxnFrom` follow `annotation` among the RAVEN-specific extras | +| Field order, genes | `protein` before `annotation` | `protein` after `annotation` | +| `objective_coefficient` on import | Sets the coefficient to 1 for any value | The value is read | + +`readYAMLmodel` of RAVEN 3 accepts both the top-level `eccodes`, `smiles` and `rxnNotes` keys and the `annotation` and `notes` forms. (raven-cobra-interoperability)= ### RAVEN⇄COBRA interoperability -`conversion/ravenCobraWrapper.m` keeps its signature (`newModel = ravenCobraWrapper(model)`). RAVEN 3 fixes two bugs that change the output: a stale variable that could reuse the wrong MIRIAM data during gene-annotation conversion, and a mis-match of gene IDs containing regex metacharacters (for example `ENSG00000141510.16`) during annotation conversion. +`conversion/ravenCobraWrapper.m` keeps its signature (`newModel = ravenCobraWrapper(model)`). RAVEN 3 fixes two bugs that change the output: a stale variable that could reuse the wrong MIRIAM data during gene-annotation conversion, and a mismatch of gene IDs that contain regular-expression metacharacters (for example `ENSG00000141510.16`) when the `rules` field is built from `grRules`. --- @@ -54,54 +72,62 @@ A pipeline that depends on the RAVEN 2 Excel curation template can convert the t | Aspect | RAVEN 2 | RAVEN 3 | |---|---|---| -| Reaction and gene database source | Local KEGG FTP dump (licence required) | Prebuilt artefacts fetched on demand from releases of `toolbox-assets` | -| Functions that build the database | `getGenesFromKEGG`, `getMetsFromKEGG` and `getRxnsFromKEGG` parsed the dump into a cached `keggModel.mat` | `buildGlobalKEGGModel`, `buildGlobalGPR` and `readKEGGTable` assemble the artefacts | -| `getModelFromKEGG`'s `keggPath` argument | Required | Removed, not needed | -| Ortholog HMM library | One `.hmm` file per KEGG Orthology (KO) entry; one `hmmsearch` call per KO, via WSL on Windows | One concatenated per-domain library (`kegg118_eukaryotes.hmm` and `kegg118_prokaryotes.hmm`); one `hmmsearch` call against the whole proteome, no WSL | -| `dataDir` | Arbitrary local folder | Must be `kegg118_eukaryotes` or `kegg118_prokaryotes` | -| `nSequences`, `seqIdentity` (CD-HIT clustering controls) | Configurable | Removed, no local clustering step | +| Reaction and gene database source | Local KEGG FTP dump (licence required), read when the cached files `keggGenes.mat`, `keggMets.mat` and `keggRxns.mat` are absent | Prebuilt artefacts fetched on demand from releases of `toolbox-assets` | +| Functions that build the database | `getGenesFromKEGG`, `getMetsFromKEGG` and `getRxnsFromKEGG` | `buildGlobalKEGGModel`, `buildGlobalGPR` and `readKEGGTable` | +| `getModelFromKEGG` `keggPath` argument | Optional (default `external/kegg`) | Removed | +| Ortholog HMM library | One `.hmm` file per KEGG Orthology (KO) entry; one `hmmsearch` call per KO, through the Windows Subsystem for Linux (WSL) on Windows | One concatenated library per domain (`kegg118_eukaryotes.hmm` and `kegg118_prokaryotes.hmm`); one `hmmsearch` call against the whole proteome; no WSL | +| `dataDir` | Arbitrary local folder | For the homology method (a FASTA file is given), the folder name must end in `kegg118_eukaryotes` or `kegg118_prokaryotes`; the annotation method ignores `dataDir` | +| `nSequences`, `seqIdentity` (CD-HIT clustering controls) | Configurable | Removed; there is no local clustering step | | `cutOff` default | `10^-50` | `10^-30` | | `minScoreRatioG` default | `0.8` | `0.9` | -| Calling convention | Positional | Positional or named (`parseRAVENargs`); positions shift in `getModelFromKEGG`, `getKEGGModelForOrganism` and `getPhylDist` (see below the table) | +| Calling convention | Positional | Positional or name-value (`parseRAVENargs`); a call with more positional arguments than the function defines raises `parseRAVENargs:tooManyArgs` | -The RAVEN 3 artefacts are a gene-free reference model plus KO, reaction and organism tables. In RAVEN 2 the per-KO `.hmm` files were either self-built with CD-HIT, MAFFT and `hmmbuild`, or downloaded. +The first call of `getModelFromKEGG` downloads the data, builds the global KEGG model and saves it as `src/reconstruction/kegg/keggModel.mat` (the source header states that the first build needs a few hundred MB of disk; the size of the cached file was not checked). `getKEGGModelForOrganism` loads the global model, the KO model and the five flag vectors from the same file. External binaries and the KEGG data are fetched from pinned `toolbox-assets` tags and are not bundled; see [Toolbox dependencies and installation](#toolbox-deps). -In RAVEN 3 the three functions take their optional arguments through `varargin`, in the order of RAVEN 2 minus the removed arguments. Positional calls written for RAVEN 2 (checked against v2.11.3) change as follows: +The three functions take their optional arguments through `varargin`, in the order of RAVEN 2 minus the removed arguments. Positional calls written for RAVEN 2 (checked against v2.11.3) change as follows. -- `getModelFromKEGG(keggPath, keepSpontaneous, ...)` became `getModelFromKEGG(keepSpontaneous, ...)`. Each old positional argument is read as the parameter one position earlier. -- `getKEGGModelForOrganism` no longer has `nSequences` and `seqIdentity`, so `globalModel` moves from the 15th to the 13th argument. -- `getPhylDist(keggPath, onlyInKingdom)` became `getPhylDist(onlyInKingdom)`, because `keggPath` was removed. +- `getModelFromKEGG(keggPath, keepSpontaneous, ...)` becomes `getModelFromKEGG(keepSpontaneous, ...)`. An unchanged RAVEN 2 call shifts every later argument: the argument at position 2 is read as the parameter that RAVEN 2 takes at position 3, and so on. +- `getKEGGModelForOrganism` has no `nSequences` and `seqIdentity`, so `globalModel` moves from the 15th to the 13th argument among all arguments. In the migration pages, a position counts all arguments of the call, including the required ones. +- `getPhylDist(keggPath, onlyInKingdom)` becomes `getPhylDist(onlyInKingdom)`. ### Homology-based reconstruction (`getModelFromHomology`) | Aspect | RAVEN 2 | RAVEN 3 | |---|---|---| | `strictness` parameter | `1`, `2` or `3` | Unchanged | -| Best-hit tie-break at `strictness=3` | E-value | Bitscore; no option restores the E-value criterion of RAVEN 2 | +| Best-hit tie-break at `strictness=3` | E-value | Bitscore; no option restores the E-value criterion | | `minLen` default | `200` | `100` | -| GPR gene substitution when merging homology hits | Regex or `strrep` substitution (could corrupt IDs that were substrings of one another, or contained regex metacharacters) | Exact-token-based substitution | +| GPR gene substitution when merging homology hits | Regular-expression or `strrep` substitution, which corrupts IDs that are substrings of other IDs or contain regular-expression metacharacters | Exact-token substitution | +| Complex handling, gene-free reactions and notes | Reactions with incompletely mapped complexes are flagged with an `OLD_` prefix | Options `complexPolicy` (default `'flag'`, the RAVEN 2 behaviour), `keepGeneFree` and `preserveNotes` | +| Gene IDs of the new organism | No restriction | Must not contain whitespace, parentheses, `\|` or `&` | -### Binaries and genome data +`reconstruction/homology/runSuppOrth.m` and `reconstruction/homology/hitsFromSuppOrth.m` are reconstruction inputs without a RAVEN 2 counterpart. The homology method takes its hits from BLAST, DIAMOND or SuppOrth. -External binaries (BLAST+, DIAMOND, HMMER) and the KEGG data artefacts are fetched on demand from pinned `toolbox-assets` release tags and are not bundled with the repository; see [Toolbox dependencies and installation](#toolbox-deps). +### Genome data -`curation/downloadGenomeData.m`, `curation/getGeneData.m` and `curation/processProteinFastaFile.m` are new in RAVEN 3 and turn an NCBI genome accession (`GCF_` or `GCA_`) into a model-ready proteome and a gene ID mapping table. +`curation/downloadGenomeData.m`, `curation/getGeneData.m` and `curation/processProteinFastaFile.m` turn an NCBI genome accession (`GCF_` or `GCA_`) into a model-ready proteome and a gene ID mapping table. RAVEN 2 has no counterpart. + +The content of the KEGG artefacts (a gene-free reference model plus KO, reaction and organism tables) was not checked against the `toolbox-assets` release. --- (gpr-parsing)= ## Parsing of gene-reaction rules -RAVEN 3 adds a gene-reaction rule tokeniser and parser. `utils/parseGrRule.m` builds a parse tree (`type`: `'gene'|'and'|'or'`, with `children`). `utils/grRuleToDNF.m` converts the tree to disjunctive normal form (DNF), RAVEN's canonical "OR of AND-clauses" shape for enzyme-complex and isozyme structure. `utils/isDnfGrRule.m` checks whether a rule is already in that form. `utils/grRuleToString.m` renders a tree back to a canonical string. Five functions use the parser and converters internally: `removeGenes`, `expandModel`, `findPotentialErrors`, `diffModels` and `curateModelFromTables`. The parser and converters are not applied automatically to a loaded model: a rule keeps its original text unless one of these functions is called on it. +RAVEN 3 parses gene-reaction rules into a tree, and the five functions in the table use the parser (`removeGenes`, `expandModel`, `findPotentialErrors`, `diffModels` and `curateModelFromTables`). The parser is not applied to a loaded model: a rule keeps its original text unless one of these functions changes it. | Function | RAVEN 2 parsing | RAVEN 3 parsing | |---|---|---| -| `removeGenes` | Substring match on gene ID (`"10"` also matched `"100"`) | Exact gene-token match | -| `expandModel` | Split on `' or '` (could drop an `AND`-ed subunit inside a nested rule) | Parse-tree-based isozyme expansion, preserves nested `AND` and `OR` | -| `findPotentialErrors` | Local function of `core/standardizeGrRules.m` (not callable); string-pattern check for `") and ("` and similar patterns (false positives on valid rules) | Function of its own; parse-tree-based non-DNF detection | -| `addRxns` (new gene IDs from a grRule) | Split on the substrings "and" and "or" anywhere (could split an ID such as `"band1"` at its "and") | Split only on space-bounded `' and '` and `' or '` tokens | -| `comparison/diffModels.m` | (not present) | grRules compared by logical DNF equality | +| `removeGenes` | Substring match on gene ID (`"10"` also matches `"100"`) | Exact gene-token match | +| `expandModel` | Split on `' or '` after removing parentheses; an `AND`-ed subunit in a nested rule can be dropped | Parse-tree-based isozyme expansion that preserves nested `AND` and `OR` | +| `findPotentialErrors` | Local function of `core/standardizeGrRules.m`; string-pattern check for `") and ("` and similar patterns (false positives on valid rules); returns indices | Function of its own; parse-tree-based detection of rules that are not in disjunctive normal form (DNF); `issues = findPotentialErrors(model)` returns a structure array with the fields `index`, `rxn`, `grRule` and `reason` | +| `diffModels` | Not present | Compares grRules; two rules are equal when their DNF forms are equal after sorting of genes and clauses (no absorption: `A or (A and B)` differs from `A`) | +| `curateModelFromTables` | Not present | Parses rules of the table input | + +`addRxns` does not use the parser. It takes the new gene IDs of a grRule from `getGenesFromGrRules`, which splits on space-bounded `and`, `or`, `&` and `|`. RAVEN 2 split on the substrings "and" and "or" anywhere, so an ID such as `band1` was split at "and". + +The parser functions are in `utils/`. `parseGrRule` builds a tree with `type` (`'gene'`, `'and'` or `'or'`) and `children`. `grRuleToDNF` converts the tree to DNF, an OR of AND-clauses. `isDnfGrRule` checks whether a rule is in that form, and `grRuleToString` renders a tree as a canonical string. `parseGrRule` accepts `and` and `or` in any case and the symbols `&`, `&&`, `|` and `||`. Operators are recognised as whole words, so `G1&G2` is one gene ID. A malformed rule raises `RAVEN:badGrRule`. -`manipulation/standardizeGrRules.m` (which runs on import) does not build a parse tree. It converts ` AND ` and ` OR ` to lowercase and splits the rule on the space-bounded strings ` and ` and ` or `, so those are the two spellings of the operators it recognises. It calls `findPotentialErrors`, which parses the rules, and returns the indices of rules that cannot be parsed or are not in DNF (`indexes2check`). A direct call lists these rules in one warning; `importModel` calls the function with `embedded` set to `true`, which suppresses the warning. The function does not raise `RAVEN:badGrRule`. +`manipulation/standardizeGrRules.m` (called by `importModel`) builds no parse tree. It converts ` AND ` and ` OR ` to lowercase and splits the rule on the space-bounded strings ` and ` and ` or `. It calls `findPotentialErrors` and returns the indices of rules that cannot be parsed or are not in DNF (`indexes2check`). A direct call lists these rules in one warning; `importModel` passes `embedded` as `true`, which suppresses the warning. The function does not raise `RAVEN:badGrRule`. diff --git a/docs/raven3-migration/new-and-environment.md b/docs/raven3-migration/new-and-environment.md index 7755873..11aded1 100644 --- a/docs/raven3-migration/new-and-environment.md +++ b/docs/raven3-migration/new-and-environment.md @@ -1,68 +1,71 @@ # New functionality, dependencies and error handling -Gap-filling, context-specific model extraction with `ftINIT`, new functions, MATLAB toolbox dependencies and error handling differ between RAVEN 2 and RAVEN 3. +Gap-filling, context-specific model extraction with `ftINIT`, added functions, MATLAB toolbox dependencies and error handling differ between RAVEN 2 and RAVEN 3. (ftinit)= ## Gap-filling and context-specific model extraction | Aspect | RAVEN 2 | RAVEN 3 | |---|---|---| -| `fillGaps` algorithm | One built-in mixed-integer linear programming (MILP) connectivity-maximisation strategy | The same strategy remains the default; the `algorithm` option adds `'fastLP'`, `'swiftLP'`, `'gapfillMILP'` and `'topological'` | -| `fillGaps` positional arguments | `allowNetProduction, useModelConstraints, supressWarnings, rxnScores` | Unchanged; new options are name-value-only | -| `ftINIT` tie-breaking | Depended on the solver seed | Optional `resolveTies` makes the answer less dependent on the solver seed | -| `ftINIT` MILP optimality proof | Escalating relative-gap search only | Optional `proveAbsGap` proves each stage to a fixed absolute gap in one solve | -| `ftINITInternalAlg` tie-break arguments | (not present) | `rankKeys` (14th argument) and `pinFluxes` (15th argument) | -| `getINITModel` and `runINIT` | Supported, no runtime notice | Supported, under `INIT/tINIT/`; one-time "legacy, use `ftINIT`" notice (see [Renamed and merged functions](#renamed-merged-and-consolidated-functions)) | -| `fitTasks` gap-fill mode | One behaviour (equivalent to `'merge'`) | `gapFillMode`: `'merge'` (default, same as in RAVEN 2) or `'preMerged'` | +| `fillGaps` algorithm | One built-in mixed-integer linear programming (MILP) connectivity-maximisation strategy | The same strategy is the default (`algorithm` `'reference'`); the `algorithm` option adds `'fastLP'`, `'swiftLP'`, `'gapfillMILP'` and `'topological'` | +| `fillGaps` positional arguments | `allowNetProduction, useModelConstraints, supressWarnings, rxnScores` | The first four optional arguments are unchanged; `params` is the 7th and `algorithm` the 8th argument among all arguments, followed by `epsilon`, `minGrowth` and `verbose`; all can also be given as name-value pairs | +| `ftINIT` tie-breaking | Depends on the solver seed (the retries of the MILP set the seed to 1234); the dependence was not measured | The option `resolveTies` adds a best-effort selection among tied solutions (see below); the resulting dependence on the seed was not measured | +| `ftINIT` MILP optimality proof | Retries with an increasing relative gap, which is derived from an absolute-gap setting and the last objective value | The option `proveAbsGap` proves each stage to a fixed absolute gap in one solve (`MIPGap` 0) | +| `ftINIT` positional arguments | `transcrData` to `verbose` (5th to 11th argument among all arguments) | Unchanged; `resolveTies` and `proveAbsGap` follow `verbose` as the 12th and 13th argument among all arguments and are off by default | +| `fitTasks` gap-fill mode | One behaviour (equivalent to `'merge'`) | `gapFillMode`: `'merge'` (default, as in RAVEN 2) or `'preMerged'` | | `fitTasks` outputs | `[outModel, addedRxns]` | `[outModel, addedRxns, failedTasks]` | -| Task requiring net production of a metabolite with no reactions | Boundary constraint dropped before gap-filling without a message; the task could be reported as passing incorrectly | Constraint preserved; the task is reported as failing if the reference model cannot produce the metabolite | + +Argument positions in the migration pages count all arguments of the call, including the required ones. `getINITModel` and `runINIT` are described in [Renamed and merged functions](#renamed-merged-and-consolidated-functions). With `gapFillMode` set to `'preMerged'`, `fitTasks` expects a reference model that already contains the model's own reactions, so no merge is needed for each task, and `ftINITFillGaps` fills the gaps. -The `algorithm` option of `fillGaps` selects among the following: +How `fitTasks` treats a task that requires net production of a metabolite without reactions was not checked. -- `'fastLP'` and `'swiftLP'`: FASTCORE-style and SWIFTCORE-style linear programming (LP) relaxation. -- `'gapfillMILP'`: growth-floor MILP with directionality repair. -- `'topological'`: breadth-first search producibility pre-screen, without model modification. +The `algorithm` option of `fillGaps` selects among the following. -`fillGaps` calls `gapFillTopological` with the defaults of that function: the seeds are the metabolites of every reaction that has only substrates and a negative lower bound or only products and a positive upper bound, and the targets are the metabolites of the first objective reaction, substrates and products. `fillGaps` passes neither seeds nor targets, so a model without an objective reaction always raises `RAVEN:badInput` there. +- `'fastLP'` and `'swiftLP'`: FASTCORE-style and SWIFTCORE-style linear programming (LP) relaxation. +- `'gapfillMILP'`: growth-floor MILP. Reactions already in the draft model keep their bounds, so a reaction with the wrong direction is not changed. +- `'topological'`: breadth-first-search producibility pre-screen, without model modification. It requires a reference model and uses only the first one. -With `resolveTies`, each `ftINIT` MILP step keeps the score optimum and then picks among the tied solutions in phases: the fewest kept removable reactions, the lowest summed ID rank of those (a merged reaction is ranked by the smallest original ID of its group), the fluxes with the least total absolute flux, and the fluxes with the least ID-rank-weighted backward flux. The arguments `resolveTies` and `proveAbsGap` are name-value-only and default to off; the positional arguments of `ftINIT` are unchanged. +`fillGaps` calls `gapFillTopological` with the defaults of that function. The seeds are the metabolites of the exchange reactions that `getExchangeRxns` classes as `'uptake'` or `'reverse'`. The targets are the metabolites of the first objective reaction. A model without an objective reaction raises `RAVEN:badInput`. A value of `algorithm` that is none of the listed ones runs the `'reference'` algorithm. -In `ftINITInternalAlg`, `rankKeys` (default `model.rxns`) gives the key each reaction is ranked by in the ID-rank phases of `resolveTies`. Setting `pinFluxes` (default `true`) to `false` skips the two flux phases of `resolveTies`. `ftINIT` passes `false` in its last step. +With `resolveTies`, each `ftINIT` MILP step holds the score at its optimum and then chooses among the tied solutions in phases, each with a time limit: the fewest kept removable reactions, the lowest summed ID rank of those (a merged reaction is ranked by the smallest original ID of its group), the least total absolute flux, and the least ID-rank-weighted backward flux. `ftINITInternalAlg` has the arguments `resolveTies` (13th), `rankKeys` (14th, default `model.rxns`) and `pinFluxes` (15th, default `true`); `pinFluxes` set to `false` skips the two flux phases, and `ftINIT` passes `false` in its last step. A step whose first solve stops at the time limit keeps that solution without the phases, and the count and rank phases get half of the step time limit each; a phase that fails issues a warning. --- (new-functionality)= ## New functionality -RAVEN 3 adds the functions in the table; except where noted they do not change RAVEN 2 functions. +RAVEN 3 adds the functions in the table. `randomSampling` and `parseScores` are RAVEN 2 functions with changed behaviour. -| Area | New functions | Added capability | +| Area | Functions | Capability | |---|---|---| | Flux sampling (`analysis/`) | `sampleACHR`, `sampleCHRR`, `sampleMaxVolEllipse`, `sampleChebyshevCenter`, `sampleWarmupPoints` | ACHR (hit-and-run) and CHRR (hit-and-run with rounding), two near-uniform Markov chain Monte Carlo (MCMC) samplers | -| Flux sampling (`analysis/`) | `randomSampling` (changed) | Becomes a dispatcher (`'method'` option) over the RAVEN 2 random-objective method (now `'randomObjective'`) and the two new samplers | -| Flux analysis and quality control (QC) | `compareFluxes` | Diffs flux vectors | +| Flux sampling (`analysis/`) | `randomSampling` (changed) | Dispatcher with the option `'method'`: `'achr'` (default), `'chrr'` and `'randomObjective'` (the RAVEN 2 method) | +| Flux analysis and quality control (QC) | `compareFluxes` | Compares flux vectors | | Flux analysis and QC | `getMinimalMedium` | Searches a minimal medium with a MILP | | Flux analysis and QC | `traceFluxPath` | Traces the path of highest flux fraction between two reactions | | Flux analysis and QC | `walkFluxes` | Navigates the flux neighbourhood interactively | -| Flux analysis and QC | `modelSummary` | Prints a model and flux summary | -| Model comparison (`comparison/`) | `diffModels` | Diffs two models by ID: stoichiometry, bounds, objective, `grRules` (logical equality in disjunctive normal form, DNF), EC codes, metabolite properties | -| Manipulation | `findDuplicateRxns` | Standalone duplicate-stoichiometry query | -| Manipulation | `findPotentialErrors` | Function of its own (a local function of `core/standardizeGrRules.m` in RAVEN 2); parse-tree-based detection of `grRules` that are not in DNF and of rules that cannot be parsed (see [Parsing of gene-reaction rules](#gpr-parsing)) | -| Genome-based curation (`curation/`, new folder) | `downloadGenomeData`, `getGeneData`, `processProteinFastaFile` | Pipeline from an NCBI genome accession to a gene ID mapping table | -| Genome-based curation (`curation/`, new folder) | `renameModelGenes` | Bulk gene renaming in a model | -| Genome-based curation (`curation/`, new folder) | `curateModelFromTables` | Bulk curation of metabolites, reactions and genes from `.tsv` files against an existing model | -| Biomass composition (`biomass/`) | `getBiomassFractions`, `scaleBiomassFraction`, `scaleBiomassPseudoreaction`, `setGAM` | Organism-agnostic biomass-composition and growth-associated-maintenance (GAM) calibration; `tutorials/tutorial7.m` demonstrates the workflow | +| Flux analysis and QC | `modelSummary` (`queries/`) | Prints a model and flux summary | +| Model comparison (`comparison/`) | `diffModels` | Compares two models by ID: stoichiometry, bounds, objective, `grRules`, EC codes and metabolite properties; separate from the overlap comparison of `compareMultipleModels` | +| Manipulation | `findDuplicateRxns` | Duplicate-stoichiometry query | +| Genome-based curation (`curation/`) | `downloadGenomeData`, `getGeneData`, `processProteinFastaFile` | Pipeline from an NCBI genome accession to a gene ID mapping table | +| Genome-based curation (`curation/`) | `renameModelGenes` | Bulk gene renaming in a model | +| Genome-based curation (`curation/`) | `curateModelFromTables` | Bulk curation of metabolites, reactions and genes from `.tsv` files against an existing model | +| Biomass composition (`biomass/`) | `getBiomassFractions`, `scaleBiomassFraction`, `scaleBiomassPseudoreaction`, `setGAM` | Biomass-composition and growth-associated-maintenance (GAM) calibration; `tutorials/tutorial7.m` demonstrates the workflow | | Localisation (`localization/`) | `assignCompartments`, `defaultCompartmentMap`, `getUniProtScores` | Deterministic MILP compartment assignment (alternative to the heuristic `predictLocalization`) | -| Localisation (`localization/`) | `parseScores` (changed) | Gains `'mulocdeep'`, `'compartments'` and `'uniprot'`; `'deeploc'` reads DeepLoc 2 output; the default predictor is `'deeploc'` (RAVEN 2: `'wolf'`); WoLF PSORT is removed (see [Backward-incompatible changes](#backward-incompatible-changes)) | -| Annotation (`annotation/`) | `assignSBOterms`, `loadDeltaGCSV`, `saveDeltaGCSV` | Organism-agnostic SBO term assignment; ΔG thermodynamic data CSV round-trip | -| Conditions (`conditions/`, new folder) | `applyCondition` | Applies a data-driven growth or media condition, given as a YAML file, to a model | -| Internal infrastructure | `ravenModelFields` | Central registry of every 1-D model field (entity type, default value), used internally by `permuteModel`, `removeReactions` and `curateModelFromTables` | +| Localisation (`localization/`) | `parseScores` (changed) | Adds `'mulocdeep'`, `'compartments'` and `'uniprot'`; `'deeploc'` reads DeepLoc 2 output; the default predictor is `'deeploc'` (RAVEN 2: `'wolf'`); WoLF PSORT is removed (see [Backward-incompatible changes](#backward-incompatible-changes)) | +| Annotation (`annotation/`) | `assignSBOterms`, `loadDeltaGCSV`, `saveDeltaGCSV` | SBO term assignment; ΔG data CSV I/O | +| Conditions (`conditions/`) | `applyCondition` | Applies a condition given as a YAML file to a model, in a fixed order: exchange bounds, cofactor-pseudoreaction edits, biomass-stoichiometry deltas and per-reaction bound overrides | +| Infrastructure | `ravenModelFields` | Registry of every 1-D model field (entity type, default value), used by `permuteModel`, `removeReactions` and `curateModelFromTables` | +| Infrastructure | `progressReport`, `setRavenProgress` | Progress reporting with a selectable backend (for example a log for compute clusters) | +| Infrastructure | `splitProbForConditioning` (`solver/`) | Splitting of an optimisation problem for conditioning | -`sampleCHRR` rounds the polytope before sampling, so mixing stays fast on strongly elongated polytopes, such as enzyme-constrained models, unlike `sampleACHR`. +`randomSampling` changes the default: a RAVEN 2 call `randomSampling(model, n)` runs ACHR; adding `'method', 'randomObjective'` runs the RAVEN 2 method. `randomSampling` also gains the outputs and options `info`, `seed`, `thinning` and `nBurnin`. -`diffModels` is separate from `compareMultipleModels`, the existing overlap-style comparison. `applyCondition` applies the parts of a condition in a fixed sequence: exchange bounds, optional cofactor-pseudoreaction edits, optional biomass-stoichiometry deltas and per-reaction bound overrides. +The function documentation of `sampleCHRR` states that the polytope is rounded before sampling and mixing stays fast on strongly elongated polytopes such as enzyme-constrained models, and that `sampleACHR` does not round. The mixing speed was not measured. + +`findPotentialErrors`, the parser functions and `diffModels` rule comparison are described in [Parsing of gene-reaction rules](#gpr-parsing). The reconstruction and curation functions for KEGG, homology and genome data are described in [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction). --- @@ -71,26 +74,23 @@ RAVEN 3 adds the functions in the table; except where noted they do not change R | Aspect | RAVEN 2 | RAVEN 3 | |---|---|---| -| MATLAB version | No pinned or documented minimum (guidance: avoid very new MATLAB syntax, to accommodate older installs) | Tested in continuous integration with R2024b | -| Statistics Toolbox | Required (`combnk`) | Not required (`nchoosek`, base MATLAB) | -| Bioinformatics Toolbox | Required (`fastaread` and `fastawrite`) | Not required (`io/readFasta.m` and `io/writeFasta.m`) | -| Optimization Toolbox | Required for `qMOMA` and `solveQP` | Not required (both removed) | +| MATLAB version | No minimum stated; `readYAMLmodel` has a fallback for releases before R2020b (`readlines`) | No minimum stated; the code has no check that rejects an older release. `readYAMLmodel` keeps the fallback for releases before R2020b, `parseTaskList` calls `readcell` (R2019a) for Excel task lists and `addIdentifierPrefix` calls `regexpPattern` (R2020b). Continuous integration uses R2024b only. Older releases were not tested | +| Statistics and Machine Learning Toolbox | Required by `compareMultipleModels` (`mdscale`, `pdist`, `squareform` and `tsne`; an error without them) and by `compareRxnsGenesMetsComps` (`combnk`) | Optional in `compareMultipleModels`; without it the projection and heatmap clustering are skipped | +| Bioinformatics Toolbox | Required by `getKEGGModelForOrganism` when it builds missing per-KO alignments from a local KEGG dump (`fastaread` and `fastawrite`) | Not required (`io/readFasta.m` and `io/writeFasta.m`) | +| Optimization Toolbox | Required for `qMOMA` and `solveQP` (`quadprog`) | Not required (both removed) | +| Parallel Computing Toolbox | Optional; `parfor` loops (for example in `getAllowedBounds` and `randomSampling`) run serially without it | Optional; `parallelWorkersRAVEN` returns 0 workers when `runParallel` is false, which runs the `parfor` loops as plain loops | | Excel I/O | Bundled Apache POI (Java) | Dependency-free OOXML writer | -| LP and MILP solver | Required (bundled `glpk`, or Gurobi, SCIP or COBRA) | Unchanged, still required | -| External binaries | Committed to the repository, available offline immediately after clone | BLAST+, DIAMOND and HMMER are fetched on demand on first use (per-platform ZIPs); an offline `*-binaries` release bundle is available for air-gapped installs | -| KEGG HMM libraries and reference data | Manual download or local build required | Fetched on demand, cached locally after first use | -| Installation self-check | `installation/checkInstallation.m` ran the full test suite to populate its pass/fail table | `checkRaven` in the repository root runs targeted checks and downloads missing binaries without a prompt (see [Renamed and merged functions](#renamed-merged-and-consolidated-functions)) | - -All downloads come from releases of `toolbox-assets`. KEGG data comes from the release tag that `keggDataVersion()` returns (`src/reconstruction/kegg/KEGG_VERSION.md`, tag `kegg118`). The binaries come from fixed tags set in `src/installation/downloadRavenBinaries.m`: `blast-2.17.0`, `diamond-2.1.17`, `hmmer-3.4.0` and, on Windows, `hmmer-3.3.2`. The checksums of downloaded files come from `manifest.json` of the `manifest-v3` release, read at download time. - -`checkRaven` runs a small import and export round-trip and a two-reaction LP per solver, without invoking the full test classes. +| LP and MILP solver | Required (bundled `glpk`, or Gurobi, SCIP or COBRA) | Unchanged | +| External binaries | Committed to the repository (BLAST+, DIAMOND, HMMER); available offline after cloning | Fetched on first use as per-platform ZIPs into `src/external/`; for air-gapped installs the workflow `.github/workflows/release-bundle.yml` builds `RAVEN--full.zip` with all platform binaries and attaches it to a published release (whether a given release carries it was not checked) | +| KEGG HMM libraries and reference data | Manual download or local build | Fetched on first use and cached | +| Installation self-check | `installation/checkInstallation.m` runs `importExportTests`, `solverTests` and the BLAST+, DIAMOND, HMMER, CD-HIT and MAFFT test files to fill its table of passed and failed tests | `checkRaven` in the repository root imports `empty.xml`, exports SBML, imports and exports YAML, exports Excel, solves a two-reaction LP per solver and downloads missing binaries without a prompt; `checkInstallation` forwards to it (see [Renamed and merged functions](#renamed-merged-and-consolidated-functions)) | -A RAVEN 3 clone fetches binaries and data on demand, which requires network access. +KEGG data, binaries and the checksum manifest come from releases of `toolbox-assets`; other downloads have other sources (listed below). KEGG data comes from the release tag that `keggDataVersion()` returns (`src/reconstruction/kegg/KEGG_VERSION.md`, tag `kegg118`). The binaries come from fixed tags set in `src/installation/downloadRavenBinaries.m`: `blast-2.17.0`, `diamond-2.1.17`, `hmmer-3.4.0` and, on Windows, `hmmer-3.3.2`. The checksums of downloaded files come from `manifest.json` of the `manifest-v3` release, read at download time. `setRavenSolver('scip')` on Windows downloads `scip_mex_win.zip` from the RAVEN v2.8.6 release on GitHub into `src/external/scip`; the repository holds no SCIP files, and other platforms need a manually installed SCIP. `getUniProtScores` queries rest.uniprot.org, `downloadGenomeData` downloads from NCBI and `checkRaven` reads the version file from GitHub. A RAVEN 3 clone needs network access for these downloads. --- ## Error and warning handling -RAVEN 3 raises errors and issues warnings with native `error()` and `warning()` calls that carry identifiers, mostly `RAVEN:*`. The identifiers are additive, and message text is preserved for most existing checks, so `try/catch` blocks that match on message text keep working for those checks. [Backward-incompatible changes](#backward-incompatible-changes) gives the details of the `dispEM` replacement, and [Removed functionality](#removed-functionality) names the new `utils/ravenList.m` helper. +RAVEN 3 raises errors and issues warnings with native `error()` and `warning()` calls. Of 411 `error` calls outside `src/external` (lines that contain `error(` and do not start with a comment), 301 carry an identifier, 242 of them in the `RAVEN:*` namespace. Of 94 `warning` calls counted by the same rule, 79 carry an identifier, 72 of them in the `RAVEN:*` namespace. Whether message texts of RAVEN 2 checks are preserved was not checked; a `try/catch` block that matches message text can be replaced by a match on the identifier where one exists. [Backward-incompatible changes](#backward-incompatible-changes) describes the `dispEM` replacement, and [Removed functionality](#removed-functionality) names the `utils/ravenList.m` helper. -RAVEN 3 adds an output argument to `queries/checkModelStruct.m`: `issues = checkModelStruct(model)` returns a structured array of the problems (`category`, `target` and `message`) for programmatic model QC and raises no error. Without an output argument it raises errors or issues warnings directly, as in RAVEN 2. +`issues = checkModelStruct(model)` returns a structure array of the problems (`category`, `target` and `message`) for programmatic model QC and raises no error. Without an output argument the function raises errors or issues warnings as in RAVEN 2. The options `throwErrors` and `trimWarnings` (positional in RAVEN 2) can also be given as name-value pairs. diff --git a/docs/raven3-migration/structure-and-renames.md b/docs/raven3-migration/structure-and-renames.md index 780acf1..1e451ca 100644 --- a/docs/raven3-migration/structure-and-renames.md +++ b/docs/raven3-migration/structure-and-renames.md @@ -1,12 +1,12 @@ # Structure, removals and renames -RAVEN 3 organises functions in folders named by function area, removes RAVEN 2 functions and subsystems, and renames or merges other functions. +RAVEN 3 organises functions in folders named by function area, removes 65 RAVEN 2 functions, merges or renames 8 functions, and keeps 5 RAVEN 2 names as wrappers. ## Repository reorganisation -RAVEN 3 replaces the top-level layout of RAVEN 2 (`core/`, `external/`, `hpa/`, `io/`, `legacy/`, `pathway/`, `plotting/`, `solver/`, `struct_conversion/`, `installation/`, `utils/`) with folders named by function area: +The RAVEN 3 code is in `src/`. RAVEN 2 kept its functions in the top-level folders `core/`, `external/`, `INIT/`, `hpa/`, `io/`, `legacy/`, `pathway/`, `plotting/`, `solver/`, `struct_conversion/`, `installation/` and `utils/`, and its bundled third-party code in `software/`. Other top-level folders changed as well: `software/` third-party code is in `src/external/`, `tutorial/` is `tutorials/`, `doc/` is `docs/`, and `testing/` is `tests/function_tests/`. -| RAVEN 3 folder | Contents | RAVEN 2 source | +| RAVEN 3 folder (under `src/`) | Contents | RAVEN 2 source | |---|---|---| | `INIT/` | `ftINIT`, `INIT/tINIT/` for `getINITModel` and `runINIT`, and HPA parsing, expression parsing and gene scoring (`parseHPA`, `parseHPArna`, `scoreModel`) | `INIT/`, `hpa/` | | `analysis/` | Analyses related to flux balance analysis (FBA): flux variability analysis (FVA), sampling, FSEOF, OptKnock, robustness, production envelope, flux comparison and tracing | `core/` | @@ -16,76 +16,92 @@ RAVEN 3 replaces the top-level layout of RAVEN 2 (`core/`, `external/`, `hpa/`, | `conditions/` | Applying a data-driven growth or media condition to a model | New folder | | `conversion/` | RAVEN⇄COBRA conversion, identifier prefixing, field ordering | `struct_conversion/` | | `curation/` | Genome-annotation-driven gene and model curation | New folder | -| `external/` | Bundled third-party code (GLPKmex, libSBML, SCIP and m2html) and the download target of the BLAST+, DIAMOND and HMMER binaries | `external/` | -| `gapfilling/` | `fillGaps` and its pluggable algorithms, task fitting, leak-metabolite and exchange checks | `core/` (and new functions) | -| `installation/` | Path setup, on-demand binary and data download | `installation/` | -| `io/` | Model import and export (SBML, YAML, Excel), FASTA I/O | `io/` | -| `localization/` | Subcellular localisation prediction and scoring, compartment assignment | `external/` (and new functions) | -| `manipulation/` | Structural edits: add, remove and change reactions, metabolites and genes; merge, sort, simplify | `core/` | -| `queries/` | Non-mutating lookups: indexes, exchange and transport reactions, stoichiometry construction, model checking | `core/` | -| `reconstruction/` | KEGG- and homology-based draft reconstruction | `external/` | -| `solver/` | LP and MILP solving | `solver/` | +| `external/` | Bundled third-party code (GLPKmex, libSBML and m2html) and the download target of the BLAST+, DIAMOND and HMMER binaries (`src/external/`). `src/external/scip/` holds no files; `setRavenSolver('scip')` downloads the SCIP MEX file there on Windows (see [Toolbox dependencies and installation](#toolbox-deps)) | `software/` | +| `gapfilling/` | `fillGaps` and its pluggable algorithms, task fitting, leak-metabolite and exchange checks | `core/`, `INIT/` (and new functions) | +| `installation/` | Path setup, on-demand binary and data download | `installation/`, `core/` (`findRAVENroot`), `external/` (`updateDocumentation`) | +| `io/` | Model import and export (SBML, YAML, Excel), FASTA I/O | `io/` (and new functions) | +| `localization/` | Subcellular localisation prediction and scoring, compartment assignment | `external/`, `core/` (`predictLocalization`) (and new functions) | +| `manipulation/` | Structural edits: add, remove and change reactions, metabolites and genes; merge, sort, simplify | `core/`, `io/` (`closeModel`, `sortIdentifiers`), `INIT/` (`removeLowScoreGenes`, `reverseRxns`) | +| `queries/` | Non-mutating lookups: indexes, exchange and transport reactions, stoichiometry construction, model checking | `core/` (and new functions) | +| `reconstruction/` | KEGG- and homology-based draft reconstruction | `external/`, `core/` (`getModelFromHomology`, `guessComposition`) | +| `solver/` | LP and MILP solving | `solver/`, `core/` (`getMinNrFluxes`) (and new functions) | | `tasks/` | Metabolic task list parsing and checking | `core/` | -| `tests/function_tests/` (repository root, outside `src/`) | `matlab.unittest` suite | `testing/` | -| `utils/` | Generic helpers, including the new argument-parsing and registry infrastructure | `utils/` | +| `utils/` | Generic helpers, including the argument-parsing and registry infrastructure | `utils/`, `core/` (`convertCharArray`, `printOrange`) | | `deprecated/` | Wrappers for merged and renamed functions | New folder | +| `tests/function_tests/` (repository root, outside `src/`) | Class-based `matlab.unittest` suite (23 of 25 classes derive from `RavenTestCase`; `tBinaries` and `tParseRAVENargs` derive from `matlab.unittest.TestCase`) | `testing/` (function-based tests built with `functiontests`) | -Removed without a successor folder: the subfolder `external/metacyc/` (MetaCyc reconstruction; `external/` itself remains), `pathway/` and `plotting/` (visualisation), `legacy/` (CellDesigner import and other pre-2.0 code), and the Excel-POI helpers of `struct_conversion/`. +`checkRaven.m` and the wrapper `checkInstallation.m` are in the repository root. -A script that calls RAVEN functions by name keeps working when the whole RAVEN folder tree, including subfolders, is on the MATLAB path (through `addRavenToUserPath` or `addpath(genpath(...))`), except for calls to removed functions and to merged functions without a wrapper. MATLAB resolves functions by name across the whole path, so moving a file to a new subfolder does not break the call. The reorganisation affects a script that (a) uses a hard-coded path to a specific `.m` file, (b) calls a removed function or a merged function without a wrapper, or (c) shadows a RAVEN function name with its own function. +Removed without a successor folder: the subfolder `external/metacyc/` and the MetaCyc function `external/combineMetaCycKEGGModels.m`, `pathway/` and `plotting/` (visualisation), `legacy/` (CellDesigner import and other pre-2.0 code), and the Excel-POI helpers of `io/` together with the Apache POI libraries of `software/apache-poi`. -The full mapping of moved files (RAVEN 2 path to RAVEN 3 path) is not reproduced here because it is large and rarely something that needs action in a script. +A script that calls RAVEN functions by name works when the whole RAVEN folder tree, including subfolders, is on the MATLAB path (through `addRavenToUserPath` or `addpath(genpath(...))`), because MATLAB resolves functions by name across the path. The reorganisation affects a script that uses a hard-coded path to a specific `.m` file, calls a removed function or a merged function without a wrapper, or contains its own function with the same name as a RAVEN 3 function. `checkFunctionUniqueness` warns about the last case. + +The mapping of every moved file is not listed. --- (removed-functionality)= ## Removed functionality -RAVEN 3 removed the functions and subsystems in the table to make the codebase smaller. None of them has a call site left in RAVEN 3. +The table lists all 65 RAVEN 2 functions that RAVEN 3 removes without a successor of the same name. The numbers per RAVEN 2 folder are: `pathway/` 6, `plotting/` 10, `external/` 12 (8 MetaCyc, 4 others), `legacy/` 11, `io/` 12, `core/` 8, `utils/` 3, `solver/` 2 and `INIT/` 1. A function of RAVEN 3 calls no removed function; the name `getToolboxVersion` remains as a local function `getToolboxVersion` in `exportForGit`. | Removed | Former purpose | Replacement | |---|---|---| -| `pathway/*`, `plotting/*` (16 functions, such as `drawMap`, `drawPathway`, `colorPathway`, `colorSubsystem`, `markPathwayWithFluxes`, `markPathwayWithExpression`, `setOmicDataToRxns` and `plotAdditionalInfo`) | Metabolic map and pathway visualisation | None in RAVEN 3. Alternatives are the COBRA Toolbox, Escher and export of flux and omics data for external plotting. | -| `external/metacyc/*` (7 functions: `getModelFromMetaCyc`, `getRxnsFromMetaCyc`, `getMetsFromMetaCyc`, `getEnzymesFromMetaCyc`, `addSpontaneousRxns`, `linkMetaCycKEGGRxns`, `combineMetaCycKEGGModels`) | MetaCyc-based draft reconstruction | None; KEGG- and homology-based reconstruction remain the supported reconstruction methods. | -| `legacy/*` (CellDesigner import, `changeGeneAssoc`, the bundled `xml_toolbox`) | Pre-2.0 compatibility code | None; considered dead. | -| `io/importExcelModel.m`, `io/loadWorkbook.m`, `io/loadSheet.m`, `io/writeSheet.m`, `io/SBMLFromExcel.m`, `io/addJavaPaths.m`, `io/startup.m` | Apache-POI-based Excel model I/O | `io/writeExcel.m` (dependency-free OOXML writer) for export; `curation/curateModelFromTables.m` for tabular curation. No full Excel-template import remains; see [Excel I/O](#excel-io). | -| `io/exportModelToSIF.m`, `io/exportToTabDelimited.m` | SIF graph export; tab-delimited model export | Dead or orphaned code with no callers. | -| `io/getFullPath.m`, `io/getMD5Hash.m`, `io/getToolboxVersion.m` | Path canonicalisation; file MD5 hashing; toolbox git-version lookup | Inlined where still needed (for example MD5 via Java's `MessageDigest` inside `getBlast`), or merged into `exportForGit`. | -| `core/dispEM.m` | Universal error and warning display | Native `warning()` and `error()` plus new `utils/ravenList.m` for formatted item lists. See [Backward-incompatible changes](#backward-incompatible-changes). | -| `core/followFluxes.m`, `core/followChanged.m` | Flux-change reporting relative to a reference solution | `queries/printFluxes.m` covers the cutoff filtering; nothing directly replaces the reference-flux comparison. | -| `core/printModel.m` | Printed reactions to screen or file | `queries/printModelStats.m` and `queries/printFluxes.m` cover related summary and flux printing (see the note below the table). | -| `core/getMetsInComp.m` | Returned metabolite indices in a given compartment | Inline it as `model.metComps == compIndex`. | -| `core/mapCompartments.m` | Merged and split the score columns of a localisation-score structure by compartment rules | Superseded by the `compartmentMap` option of `parseScores` and `getUniProtScores` (default `defaultCompartmentMap`); merged columns take the maximum score, and splitting is not supported. | -| `core/getExpressionStructure.m` | Loaded an expression-experiment structure from an ad hoc Excel format | Superseded by `INIT/parseHPA.m` and `INIT/parseHPArna.m` for the supported expression-data workflow. | -| `core/checkRxn.m` | Per-reaction reactant and product synthesisability debugging | No direct successor; use `gapfilling/findLeakMetabolite.m`, `gapfilling/canExchange.m` or `gapfilling/checkProduction.m`. | -| `external/getWoLFScores.m` | WoLF PSORT score parsing | None; see [Backward-incompatible changes](#backward-incompatible-changes). | -| `external/kegg/{getGenesFromKEGG,getMetsFromKEGG,getRxnsFromKEGG,constructMultiFasta,getWSLpath}.m`, `external/getBlastFromExcel.m` | Local-KEGG-FTP-dump parsing; per-KO multi-FASTA assembly; WSL path translation; Excel-based BLAST import | Superseded by the KEGG pipeline based on `toolbox-assets` artefacts. See [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction). | -| `solver/qMOMA.m`, `solver/solveQP.m` | MOMA-based analyses (quadratic programming) | Removed; RAVEN 3 has no replacement based on quadratic programming. See [Backward-incompatible changes](#backward-incompatible-changes) for the arguments removed from `findGeneDeletions`. | -| `testing/unit_tests/*.m` (script-based tests) | Ad hoc test scripts | `tests/function_tests/*.m` in the repository root, a `matlab.unittest`-class-based suite. Relevant to contributors only; `runRAVENtests` usage is unchanged for end users. | - -`printFluxes` prints the reactions of a flux vector with an absolute flux at or above `cutOffFlux` (default `10^-8`), one line per reaction; it prints exchange reactions only unless `onlyExchange` is `false`. A dump of all reactions of a model without fluxes has no direct replacement. +| `pathway/colorPathway`, `pathway/getPathwayDimensions`, `pathway/mapPathwayRxnNames`, `pathway/markPathwayWithExpression`, `pathway/markPathwayWithFluxes`, `pathway/trimPathway` (6 functions) | Metabolic pathway map handling and colouring | None in RAVEN 3. Alternatives are the COBRA Toolbox, Escher and export of flux and omics data for external plotting. | +| `plotting/colorSubsystem`, `plotting/drawMap`, `plotting/drawPathway`, `plotting/getColorCodes`, `plotting/getObjectiveString`, `plotting/plotAdditionalInfo`, `plotting/plotLabels`, `plotting/setColorToMapRxns`, `plotting/setOmicDataToRxns`, `plotting/setTitle` (10 functions) | Metabolic map drawing and omics colouring | None, as above. | +| `external/metacyc/{getModelFromMetaCyc,getRxnsFromMetaCyc,getMetsFromMetaCyc,getEnzymesFromMetaCyc,getMetaCycModelForOrganism,addSpontaneousRxns,linkMetaCycKEGGRxns}`, `external/combineMetaCycKEGGModels` (8 functions) | MetaCyc-based draft reconstruction | None; KEGG- and homology-based reconstruction are the reconstruction methods of RAVEN 3. | +| `legacy/core/changeGeneAssoc`, `legacy/external/constructPathwayFromCelldesigner`, and the nine functions of the bundled `xml_toolbox` (`xml_format`, `xml_formatany`, `xml_help`, `xml_load`, `xml_parse`, `xml_parseany`, `xml_save`, `xml_strsplit`, `xml_unblank`) (11 functions) | CellDesigner import and pre-2.0 compatibility code | None. | +| `io/importExcelModel`, `io/loadWorkbook`, `io/loadSheet`, `io/writeSheet`, `io/SBMLFromExcel`, `io/addJavaPaths`, `io/startup` (7 functions) | Apache-POI-based Excel model I/O | `io/writeExcel.m` (dependency-free OOXML writer) for export; `curation/curateModelFromTables.m` for tabular curation. There is no full Excel-template import; see [Excel I/O](#excel-io). | +| `io/exportModelToSIF`, `io/exportToTabDelimited` | SIF graph export; tab-delimited model export | None. The sole caller of `exportToTabDelimited` was the bare-path fallback of `exportToExcelFormat`, which is removed. | +| `io/getFullPath`, `io/getMD5Hash`, `io/getToolboxVersion` | Path canonicalisation; file MD5 hashing; toolbox git-version lookup | Inlined where needed: the MD5 hash through Java `MessageDigest` in `getBlast`, the version lookup as a local function of `exportForGit`. | +| `core/dispEM` | Error and warning display (388 lines call it in RAVEN 2, counted outside `doc/` and `testing/`) | Native `warning()` and `error()`, and `utils/ravenList.m` for formatted item lists. See [Backward-incompatible changes](#backward-incompatible-changes). | +| `core/followFluxes`, `core/followChanged` | Flux-change reporting relative to a reference solution | `analysis/compareFluxes.m` compares two flux vectors of one model (reactions turned on or off, flipped and changed, with `cutoff` and `metaboliteList`), which replaces `core/followChanged`; `queries/printFluxes.m` prints the fluxes of one vector above a cutoff. | +| `core/printModel` | Printed reactions to screen or file | `queries/printModelStats.m` and `queries/printFluxes.m` cover model summaries and flux printing (see below the table). | +| `core/getMetsInComp` | Metabolite indices in a compartment | `model.metComps == compIndex` | +| `core/mapCompartments` | Merged and split the score columns of a localisation-score structure by compartment rules | The `compartmentMap` option of `parseScores` and `getUniProtScores`. The default map `defaultCompartmentMap` applies to `getUniProtScores` and to `parseScores` with the `'uniprot'` predictor; elsewhere, without a map, every label stays unchanged. With a map, a label that the map does not contain is dropped together with its score column. Merged columns take the maximum score; splitting is not supported. | +| `core/getExpressionStructure` | Loaded an expression structure from an ad hoc Excel format | `INIT/parseHPA.m` and `INIT/parseHPArna.m` read Human Protein Atlas files. Expression tables from other sources have to be assembled by hand as the `arrayData` structure documented in `scoreModel`. | +| `core/checkRxn` | Per-reaction synthesisability debugging | No direct successor; `gapfilling/findLeakMetabolite.m`, `gapfilling/canExchange.m` and `gapfilling/checkProduction.m` cover related checks. | +| `external/getWoLFScores` | WoLF PSORT score parsing | None; `parseScores` with `'wolf'` raises `RAVEN:badInput`. See [Backward-incompatible changes](#backward-incompatible-changes). | +| `external/kegg/constructMultiFasta`, `external/kegg/getWSLpath`, `external/getBlastFromExcel` | Per-KO multi-FASTA assembly; WSL path translation; Excel-based BLAST import | The KEGG pipeline based on `toolbox-assets` artefacts. See [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction). | +| `solver/qMOMA`, `solver/solveQP` | MOMA analyses and quadratic programming | None; RAVEN 3 has no quadratic programming. See [Backward-incompatible changes](#backward-incompatible-changes) for the arguments removed from `findGeneDeletions`. | +| `utils/emptyOrLogicalScalar`, `utils/emptyOrTextOrCellOfText`, `utils/emptyOrTextScalar` | Argument validators | `utils/parseRAVENargs.m` | +| `INIT/getExprForRxnScore` | Reaction scores from expression data | A private helper of the test suite in `tests/function_tests/private/` | + +`printFluxes` prints the reactions of a flux vector with an absolute flux at or above `cutOffFlux` (default `10^-8`), one line per reaction; it prints exchange reactions unless `onlyExchange` is `false`. A listing of all reactions of a model without fluxes has no direct replacement. --- (renamed-merged-and-consolidated-functions)= ## Renamed and merged functions -RAVEN 3 introduces the `deprecated/` folder convention. In `deprecated/`, `makeSomething`, `consumeSomething`, `canProduce` and `canConsume` stay callable under their old names as wrappers that issue a one-time `RAVEN:deprecated` warning ("will be removed in the next major release") and forward to the new function. The other functions in the table have no wrapper, except `checkInstallation`, which has its own row. +The `deprecated/` folder holds `makeSomething`, `consumeSomething`, `canProduce` and `canConsume`. These wrappers keep the RAVEN 2 names callable, issue a `RAVEN:deprecated` warning once ("will be removed in the next major release") and forward to the RAVEN 3 function. The other functions in the table have no wrapper, except `checkInstallation`, which has its own row. | RAVEN 2 functions | RAVEN 3 function | Notes | |---|---|---| -| `makeSomething(model, ignoreMets, isNames, minNrFluxes, allowExcretion, params, ignoreIntBounds)` | `findLeakMetabolite(model, 'produce', ...)` | Argument order maps 1:1; the wrapper forwards `varargin` unchanged. | -| `consumeSomething(model, ignoreMets, isNames, minNrFluxes, params, ignoreIntBounds)` | `findLeakMetabolite(model, 'consume', ...)` | The old positional order of `consumeSomething` has no `allowExcretion` slot, so the wrapper remaps the arguments by name to keep `params` and `ignoreIntBounds` intact. | +| `makeSomething(model, ignoreMets, isNames, minNrFluxes, allowExcretion, params, ignoreIntBounds)` | `findLeakMetabolite(model, 'produce', ...)` | The argument order maps one to one; the wrapper forwards `varargin` unchanged. | +| `consumeSomething(model, ignoreMets, isNames, minNrFluxes, params, ignoreIntBounds)` | `findLeakMetabolite(model, 'consume', ...)` | The RAVEN 2 argument order has no `allowExcretion` position, so the wrapper remaps the arguments by name to keep `params` and `ignoreIntBounds`. | | `canProduce(model, mets)` | `canExchange(model, 'produce', 'mets', mets)` | None. | | `canConsume(model, mets)` | `canExchange(model, 'consume', 'mets', mets)` | None. | -| `compareRxnsGenesMetsComps` | Merged into `comparison/compareMultipleModels.m` | The reaction, gene, metabolite and compartment overlap output is one part of the output of `compareMultipleModels`; no separate function or wrapper remains. | -| `INIT/scoreComplexModel.m` | `INIT/scoreModel.m` | Merged; no wrapper remains. | -| `INIT/ftINITFillGapsMILP.m` | Merged into `solver/getMinNrFluxes.m` | Available through the `resolveTies` option of `getMinNrFluxes`; the `proveAbsGap` option belongs to `ftINIT`. | -| `INIT/ftINITFillGapsForAllTasks.m` | Merged into `gapfilling/ftINITFillGaps.m` and the `'preMerged'` gap-fill mode of `fitTasks` | None. | -| `getGenesFromKEGG`, `getMetsFromKEGG` and `getRxnsFromKEGG` | `reconstruction/kegg/readKEGGTable.m` and the `toolbox-assets` KEGG artefacts | The data source changed, so the signatures are not compatible; see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction). | -| `core/parallelPoolRAVEN.m` | `utils/parallelWorkersRAVEN.m` | Parallel Computing Toolbox pool setup, renamed and rewritten. `parallelWorkersRAVEN` returns a `parfor` worker count (`0` or `Inf`) and starts no pool; the old function returned `[ps, oldPoolAutoCreate]`. | -| `checkInstallation` | `checkRaven` | Moved from `installation/checkInstallation.m` to `checkRaven.m` in the repository root; the calling signature is identical. `checkInstallation` remains as a deprecated forwarding wrapper that issues a `RAVEN:deprecated` warning. | +| `compareRxnsGenesMetsComps` | `comparison/compareMultipleModels.m` | The overlap output of RAVEN 2 (reactions, genes, metabolites, EC codes, metabolite names, equations and unique equations) is part of the output of `compareMultipleModels`. No wrapper. | +| `INIT/scoreComplexModel.m` | `INIT/scoreModel.m` | Merged; no wrapper. A call moves to `scoreModel` with the same positions and defaults for the first six optional arguments; `dataPrecedence` is added as the 7th optional argument. | +| `INIT/ftINITFillGapsMILP.m` | `solver/getMinNrFluxes.m` | Available through the `resolveTies` option of `getMinNrFluxes`, which is implemented for `formulation` `'reversible'` only (the default `'irrev'` raises an error with `resolveTies`); the `proveAbsGap` option belongs to `ftINIT`. | +| `INIT/ftINITFillGapsForAllTasks.m` | `gapfilling/ftINITFillGaps.m` and the `'preMerged'` gap-fill mode of `fitTasks` | None. `ftINITFillGaps` exists in RAVEN 2 as well. | +| `getGenesFromKEGG`, `getMetsFromKEGG` and `getRxnsFromKEGG` | `reconstruction/kegg/readKEGGTable.m` (with `buildGlobalKEGGModel` and `buildGlobalGPR`) and the `toolbox-assets` KEGG artefacts | The data source differs, so the signatures are not compatible; see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction). | +| `core/parallelPoolRAVEN.m` | `utils/parallelWorkersRAVEN.m` | `parallelWorkersRAVEN` returns a `parfor` worker count (`0` or `Inf`) and starts no pool; `parallelPoolRAVEN` returns `[ps, oldPoolAutoCreate]`. | +| `installation/checkInstallation.m` | `checkRaven.m` in the repository root | The calling signature is identical. `checkInstallation.m` in the repository root forwards to `checkRaven` and issues a `RAVEN:deprecated` warning on every call. | + +### Gene scoring with `scoreModel` + +RAVEN 2 has `INIT/scoreComplexModel.m` and `hpa/scoreModel.m`; RAVEN 3 has one `scoreModel`, which tINIT and ftINIT call with different settings. Differences for a call written for `hpa/scoreModel.m` (checked against v2.11.3): + +- RAVEN 2 scored a reaction by the best or average score over all its genes (`multipleGeneScoring`, values `'best'` and `'average'`). RAVEN 3 applies `isozymeScoring` (default `'max'`) to the `or` parts and `complexScoring` (default `'min'`) to the `and` parts of the rule. A reaction with an `and` rule receives a different default score. +- Positions count all arguments of the call. The 7th to 9th arguments of a RAVEN 2 call (`multipleGeneScoring`, `multipleCellScoring`, `hpaLevelScores`) are read as `isozymeScoring`, `complexScoring` and `multipleCellScoring`; `hpaLevelScores` is the 10th argument. +- `multipleCellScoring` accepts `'max'` and `'average'` (RAVEN 2: `'best'` and `'average'`). +- `dataPrecedence` is an argument of RAVEN 3 with the default `'gene'`. +- Genes that are absent from the data receive `NaN` in `geneScores`; the RAVEN 2 documentation states `-Inf`. `getINITModel` converts `NaN` back to `-Inf`. + +See [Backward-incompatible changes](#backward-incompatible-changes). `getINITModel` passes `'max'` for both `isozymeScoring` and `complexScoring` and `'reaction'` for `dataPrecedence`. Whether these settings reproduce the scores of the RAVEN 2 `hpa/scoreModel` was not checked; the RAVEN 2 value `'average'` for `multipleGeneScoring` has no equivalent among them. -RAVEN 2 had both `INIT/scoreComplexModel.m` and `hpa/scoreModel.m`; RAVEN 3 merged them into one `scoreModel`, called with different settings from tINIT than from ftINIT. A RAVEN 2 call to `scoreModel` (`hpa/scoreModel.m`, checked against v2.11.3) changes: RAVEN 2 scored a reaction by the best or average score over all its genes (`multipleGeneScoring`), while RAVEN 3 applies `isozymeScoring` (default `'max'`) to the `or` parts and `complexScoring` (default `'min'`) to the `and` parts of the rule. A reaction with an `and` rule therefore receives a different default score. The positional arguments after `noGeneScore` changed: the 7th to 9th arguments of a RAVEN 2 call (`multipleGeneScoring`, `multipleCellScoring`, `hpaLevelScores`) are read as `isozymeScoring`, `complexScoring` and `multipleCellScoring`, `multipleCellScoring` accepts `'max'` and `'average'` (RAVEN 2: `'best'` and `'average'`), and `hpaLevelScores` becomes the 10th positional argument. RAVEN 3 adds `dataPrecedence` (default `'gene'`); see [Backward-incompatible changes](#backward-incompatible-changes). A call to `scoreComplexModel` has to move to `scoreModel` and pass the settings that reproduce the old behaviour (a single operator for both `and` and `or`, `dataPrecedence` `'reaction'`). +### Legacy tINIT functions -`getINITModel` and `runINIT` (under `INIT/tINIT/`) are marked legacy but supported indefinitely; unlike the functions in `deprecated/`, they are not scheduled for removal. `utils/legacyMethodNotice.m` issues a one-time, silenceable `RAVEN:legacyMethod` warning that points to `ftINIT`. Use `ftINIT` for new work; keep `getINITModel` and `runINIT` only to reproduce models originally built with them. +`getINITModel` and `runINIT` (under `INIT/tINIT/`) are marked legacy and are not scheduled for removal. `utils/legacyMethodNotice.m` issues a `RAVEN:legacyMethod` warning once, which can be silenced, and points to `ftINIT`. `ftINIT` is the method for new work; `getINITModel` and `runINIT` reproduce models built with them. diff --git a/docs/raven3-vs-raven-toolbox.md b/docs/raven3-vs-raven-toolbox.md index f3c145c..c6db421 100644 --- a/docs/raven3-vs-raven-toolbox.md +++ b/docs/raven3-vs-raven-toolbox.md @@ -1,46 +1,47 @@ # Comparison of RAVEN and raven-toolbox -RAVEN exists as two implementations: the MATLAB toolbox RAVEN and the Python package raven-toolbox. +RAVEN exists as two implementations: the MATLAB toolbox RAVEN 3 and the Python package raven-toolbox. The pages below help a reader who chooses between them or ports a workflow from one to the other. - RAVEN is a MATLAB toolbox that runs without the COBRA Toolbox. - raven-toolbox is a Python package built on [cobrapy](https://cobrapy.readthedocs.io/). A model is a `cobra.Model`. -The implementations differ in function names, in functions that exist in one implementation only and in the results of certain functions (see [Differences in results between functions](raven3-vs-raven-toolbox/same-function-different-answer.md)). +The implementations differ in function names, in functions that exist in one implementation only, in the form of returned results and in the results of certain functions. Two detail pages cover these differences: [Features specific to each implementation](raven3-vs-raven-toolbox/what-each-has.md) and [Differences in results between functions](raven3-vs-raven-toolbox/same-function-different-answer.md). ## Choosing an implementation -The choice of an implementation depends on the conditions below. Each condition points to one implementation. +The first three rows of the table list conditions that favour raven-toolbox, the next four favour RAVEN, and the last row names the functions that both implementations have. | Implementation | Condition | |---|---| | raven-toolbox | The surrounding code is Python. | -| raven-toolbox | The model should be a `cobra.Model` for use with cobrapy tools. | -| raven-toolbox | Reproducible environments and continuous integration (CI) are required. | -| raven-toolbox | The KEGG artefacts must be built from a specified KEGG release. | +| raven-toolbox | The model is used with cobrapy tools as a `cobra.Model`. | +| raven-toolbox | The KEGG artefacts are built from a specified KEGG release (requires a KEGG FTP subscription). | | RAVEN | The surrounding code is MATLAB. | -| RAVEN | A model built with tINIT must be reproduced. | -| RAVEN | Dynamic flux balance analysis (FBA) is required. | -| RAVEN | Conversion to the COBRA Toolbox structure is required. | -| Either | The functions both implementations share are needed: homology and KEGG reconstruction, metabolic tasks, gap-filling, ftINIT, compartment assignment and model comparison. | +| RAVEN | A model built with tINIT is reproduced. | +| RAVEN | Dynamic flux balance analysis (FBA) is run with `runDynamicFBA`. | +| RAVEN | A model is converted in memory to the COBRA Toolbox structure with `ravenCobraWrapper`. | +| Either | Homology and KEGG reconstruction, metabolic tasks, gap-filling, ftINIT, compartment assignment and model comparison. | + +`read_matlab_model` of raven-toolbox reads a COBRA Toolbox `.mat` file as well as a RAVEN `.mat` file, and `write_matlab_model` writes the RAVEN layout, so a model can be exchanged by file between the implementations. ## Name differences -The implementations differ in naming: RAVEN uses `camelCase` and raven-toolbox uses `snake_case`; many names are not the `snake_case` form of the RAVEN name. Examples: +RAVEN uses `camelCase` and raven-toolbox uses `snake_case`. The generated mapping pairs 103 RAVEN functions with a raven-toolbox function (rule: rows of the mapping with a raven-toolbox function in the second column, cobrapy entries excluded). Of these, 48 have a name whose letters differ from the RAVEN name when case and the separator character `_` are ignored. Examples: | RAVEN | raven-toolbox | |---|---| | `getBlast` | `run_blast` | | `addRxns` | `add_reactions_from_equations` | +| `addRxnsGenesMets` | `add_reactions_from_model` | | `fillGaps` | several functions (see [Gap-filling functions](raven3-vs-raven-toolbox/same-function-different-answer.md#gap-filling-functions)) | -The [MATLAB vs Python](matlab-vs-python.md) page, generated from the sources of both implementations at build time, lists the name in the other implementation. Some functions have no counterpart or map to cobrapy; the table records which functions cobrapy covers in place of raven-toolbox, and which are deliberately absent. +`add_reactions_from_equations` takes a sequence of mappings with the keys `id`, `equation`, `name`, `bounds`, `gene_reaction_rule` and `subsystem`, which corresponds to the `equations` form of the `rxnsToAdd` struct of `addRxns`. The form of `addRxns` with `mets` and `stoichCoeffs` has no counterpart, and the metadata fields of the struct (such as `eccodes`, `rxnMiriams` and `rxnNotes`) are not keys of the mapping. -## Detail pages +The [MATLAB vs Python](matlab-vs-python.md) page is generated from the sources of both implementations at build time. It has 132 rows for 130 documented RAVEN functions (`parseGrRule` and `setExchangeBounds` have two rows each): 55 matched automatically by name, 48 from a curated list and 29 covered by cobrapy. A further 42 RAVEN functions are listed as deliberately not ported and have no counterpart, so the page accounts for 172 RAVEN functions. -| Page | Content | -|---|---| -| [Features specific to each implementation](raven3-vs-raven-toolbox/what-each-has.md) | Features of raven-toolbox (model type, KEGG artefact generation, checksum registry) and of RAVEN (tINIT, COBRA Toolbox conversion, dynamic FBA, MATLAB-specific functions). | -| [Differences in results between functions](raven3-vs-raven-toolbox/same-function-different-answer.md) | Task-checking runtime, gap-filling functions, alternative optima of mixed-integer problems, elemental balance results and parsing of gene-reaction rules. | +## Form of results + +RAVEN functions return structs or several output arguments. Many raven-toolbox functions return dataclasses, for example `GapFillResult`, `TaskResult` and `ElementalBalance`; others return a model or a list, for example `ftinit` and `add_reactions_from_equations`. Porting a call requires changing how the result is read, in addition to the name. ## Solvers @@ -49,10 +50,11 @@ The solver is set differently in each implementation. | Aspect | RAVEN | raven-toolbox | |---|---|---| | Configuration | `setRavenSolver('gurobi')` | `cobra.Configuration().solver = 'gurobi'` | -| Recommended solver | Gurobi (free academic licence) | Gurobi (free academic licence) | -| Open-source option | GLPK (bundled with RAVEN) | GLPK (bundled with cobrapy) | +| Solvers | GLPK, Gurobi, SCIP and the COBRA Toolbox solvers (`setRavenSolver` downloads the SCIP binary for Windows only) | Any solver that cobrapy supports through optlang | +| Free licence | Gurobi (academic) | Gurobi (academic) | +| Bundled open-source solver | GLPK (bundled with RAVEN) | GLPK (installed with cobrapy) | -The mixed-integer linear programming (MILP) problems of ftINIT extraction can have several optima of equal objective value. For genome-scale MILP, the choice of solver changes which optimum is returned, and the seed, the thread count and wall-clock limits influence it as well (see [Alternative optima of mixed-integer problems](raven3-vs-raven-toolbox/same-function-different-answer.md#alternative-optima-of-mixed-integer-problems)). The solver matrix under [Choosing a solver](installation/index.md#choosing-a-solver) lists Gurobi for genome-scale models and GLPK for small to medium models. +[Choosing a solver](installation/index.md#choosing-a-solver) gives the recommendation by model size. The ftINIT extraction of raven-toolbox returned different reaction sets for different solvers, seeds and thread counts in the measurements of the parameter-tuning studies; [Alternative optima of mixed-integer problems](raven3-vs-raven-toolbox/same-function-different-answer.md#alternative-optima-of-mixed-integer-problems) gives the figures. Whether the solutions are of equal quality was not measured across solvers. ```{toctree} :hidden: diff --git a/docs/raven3-vs-raven-toolbox/same-function-different-answer.md b/docs/raven3-vs-raven-toolbox/same-function-different-answer.md index fa59ef6..e5f7a40 100644 --- a/docs/raven3-vs-raven-toolbox/same-function-different-answer.md +++ b/docs/raven3-vs-raven-toolbox/same-function-different-answer.md @@ -1,52 +1,65 @@ # Differences in results between functions -The page lists functions that return a different result, return it in a different order, change the model differently or run for a different time in RAVEN and raven-toolbox, and two cases in which raven-toolbox differs from cobrapy. The list contains a function only after the difference is confirmed in the code of both functions compared. +The page lists functions that differ between RAVEN and raven-toolbox in the algorithm choice, in the form of the result or in the value of the result. The differences were found by reading the code of both implementations; where a section says a result was not checked, no run compared the two. The list is incomplete: a function that is not listed can behave differently. Candidates that were not compared include `random_sampling`, which replaces both `sampleACHR` and `sampleCHRR`, `remove_duplicate_reactions`, which replaces `contractModel`, and `simplify_model`, which corresponds to `simplifyModel`. -The list is incomplete: a function that is not listed can still behave differently. +## Metabolic task checking -## Runtime of metabolic task checking +`checkTasks` and `check_tasks` implement the same task format and the same boundary semantics. `close_boundaries=True` of `check_tasks` corresponds to calling `checkTasks` on `closeModel(model)`; [Metabolic tasks](../guide/tasks.md) describes the options. Identical pass and fail results on a shared input were not checked here. -`checkTasks` in RAVEN and `check_tasks` in raven-toolbox return the same pass and fail results and differ in runtime. +`checkTasks` starts each task from the original model structure. `check_tasks` applies each task to one model inside a `with model:` block and reverts the changes afterwards. It also saves and restores direct edits of metabolite-constraint bounds. The runtime of the two functions was not compared. -`checkTasks` rebuilds the working model from the original for each task. `check_tasks` applies each task's constraints to one model inside a `with model:` block and reverts them afterwards. `check_tasks` saves and restores direct changes of mass-balance bounds, the one kind of edit that the context manager of cobrapy does not track. - -At genome scale, copying the model for each task adds to the runtime of `checkTasks`, and `check_tasks` reuses a single model to avoid it. +The `getEssential` option of `checkTasks` has no option in `check_tasks`. In RAVEN, `checkTasks` calls `getEssentialRxns` on the model constrained by each task. The function `find_task_essential_reactions` ports this step, so it corresponds to `checkTasks` with `getEssential`, not to `getEssentialRxns` called on a model without a task. ## Gap-filling functions -`fillGaps` in RAVEN selects one of several algorithms with the `algorithm` option. raven-toolbox splits them into four functions, so porting a `fillGaps` call requires choosing among the functions in this table: +`fillGaps` in RAVEN selects one of five algorithms with the `algorithm` option: `reference` (default), `fastLP`, `swiftLP`, `gapfillMILP` and `topological`. raven-toolbox has four functions for them, so porting a `fillGaps` call requires choosing a function from this table: | Purpose | RAVEN algorithm | raven-toolbox function | |---|---|---| | Connecting blocked reactions against template models | `fillGaps` (default `'reference'`) | `connect_blocked_reactions` | | Fast filling of a large candidate set by linear programming (LP) | `fillGaps` with `'fastLP'` and `'swiftLP'` | `fill_gaps_fast_lp` (`variant="fast"` and `variant="swift"`) | | Filling by mixed-integer linear programming (MILP) with the fewest added reactions that restore growth | `fillGaps` with `'gapfillMILP'` | `fill_gaps_milp` | -| Finding the gaps without filling them | `fillGaps` with `'topological'` (`gapFillTopological`) | `analyse_topology` | +| Finding the gaps without filling them | `fillGaps` with `'topological'` | `analyse_topology` | + +Further differences: -The choice among these functions changes the reactions added and the runtime. +- `fillGaps` with `useModelConstraints=true` fills so that the constraints of the model hold. `connect_blocked_reactions` has no such mode; its documentation points to `cobra.flux_analysis.gapfill` for that use. `fill_gaps_milp` is the closest function for a growth target. +- `fillGaps` takes a cell array of template models, and the RAVEN functions `gapFillFastLP`, `gapFillMILP` and `gapFillTopological` use only the first template. The four raven-toolbox functions accept several templates. +- `fillGaps` returns `newConnected`, `cannotConnect`, `addedRxns`, `newModel` and `exitFlag`. raven-toolbox returns a `GapFillResult` from `connect_blocked_reactions`, a `FastLPResult` from `fill_gaps_fast_lp`, a `MILPGapFillResult` from `fill_gaps_milp` and a `TopologicalAnalysisResult` from `analyse_topology`. The count `newConnected` is not stable across runs; see [Gap-filling](../guide/gap-filling.md). -The RAVEN functions `canExchange`, `checkProduction` and `getAllSubGraphs` find gaps without filling them. raven-toolbox merges them into `gap_report`, the port of `gapReport`, and has no separate function for each of them. +The RAVEN functions `canExchange`, `checkProduction` and `getAllSubGraphs` find gaps without filling them. raven-toolbox merges them into `gap_report`, the port of `gapReport`, and has no separate public function for each of them. ## Alternative optima of mixed-integer problems -ftINIT extraction, gap-filling and compartment assignment solve mixed-integer problems that routinely have several optima of equal objective value. Two runs can return different reaction sets that are both correct. Results can differ between implementations, between solvers and between repeat runs of one solver. +ftINIT extraction, the reference and MILP gap-filling algorithms and compartment assignment (`assignCompartments` and `assign_compartments`) solve mixed-integer problems that can have several optima of equal objective value. The linear-programming algorithms were not measured for this, and the documentation of `swiftLP` in both implementations calls it stochastic. A solve that stops at its gap or time limit can also end at a different objective value. Two runs can return different reaction sets, and no single exact result is expected. + +Measurements of the differences: -The ftINIT reproducibility study measured raven-toolbox only (RAVEN's `ftINIT` was not run): runs of one solver also differ with the seed and the number of threads, and two repeated builds at one seed differed by 2 reactions; each of the two builds had one run of the final step that stopped at its 5 s wall-clock limit, the presumed and untested cause of the difference (see [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](../parameter-tuning/studies/ftinit-determinism.md)). No single exact result is expected. +- Between implementations: ftINIT models of five cell lines, extracted from the same Human-GEM inputs with RAVEN and raven-toolbox, share Jaccard indices of 0.964 to 0.988 over their reaction sets (see [Human-GEM cell-type model validation against RAVEN](../parameter-tuning/studies/humangem-validation.md)). The template preparation differed between the two runs. +- Between solvers: in the raven-toolbox ftINIT benchmark on Human-GEM with one run per solver, Gurobi extracted 7312 reactions and GLPK 6758 (Jaccard index 0.892). Both main steps of GLPK stopped at the 900 s limit with an unproven incumbent, so the two models differ in objective value as well as in tied optima (see [ftINIT on different solvers](../parameter-tuning/studies/init-solver-benchmark.md)). RAVEN was not run on several solvers in that study. +- Between repeated runs: two repeated raven-toolbox builds at one seed with the default settings differed by 2 reactions. Each build had one run of the final step that stopped at its 5 s wall-clock limit, the presumed and untested cause (see [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](../parameter-tuning/studies/ftinit-determinism.md)). RAVEN's `ftINIT` was not run for the effects of seed, thread count and time limit. +- Between thread counts: in raven-toolbox's first ftINIT step without `resolve_ties`, with six seeds, the solutions at `Threads` 1 and 4 for the same seed were 5 to 20 reactions apart, and in 5 of the 6 seeds the two runs reached different objective values (see [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](../parameter-tuning/studies/ftinit-determinism.md)). -Compare these outputs by the overlap of their reaction sets; do not test them for identity. +Compare outputs by the overlap of their reaction sets, for example with the Jaccard index. -`resolveTies` in RAVEN's `ftINIT` and `resolve_ties` in raven-toolbox's `ftinit` reduce the difference between runs for the ftINIT MILP. Each step then keeps the score optimum and picks among the tied solutions in phases: the fewest kept removable reactions, then the lowest summed ID rank of those, then the fluxes with the least total absolute flux, then the fluxes with the least ID-rank-weighted backward flux. The last step of `ftINIT` skips the two flux phases. +`resolveTies` in RAVEN's `ftINIT` and `resolve_ties` in raven-toolbox's `ftinit` are off by default in both. When on, each step keeps the score optimum and picks among the tied solutions in phases: the fewest kept removable reactions, then the lowest summed ID rank of those, then the least total absolute flux, then the least ID-rank-weighted backward flux. The last step of `ftINIT` skips the two flux phases. A step that stops at its time limit keeps the solver's own answer. -For a given score optimum, the selected solution does not depend on the seed when every tie-break phase finishes within its time limit. It can differ between solvers, and a step that stops at its time limit keeps the solver's own answer. The primary solve can still reach a different score optimum for a different seed, so the final reaction sets can differ between seeds. Without `resolveTies` or `resolve_ties`, the ftINIT MILP can return any of the tied optima, and results differ between implementations, between solvers and between repeated runs of one solver. +In raven-toolbox, runs with `resolve_ties` at the same score optimum returned identical solutions across seeds. The primary solve can still end at a different objective value for a different seed, within the gap and time limits, so the final reaction sets can differ. This was not measured for RAVEN. Without `resolveTies` or `resolve_ties`, the ftINIT MILP can return any of the tied optima. ## Elemental balance results -`getElementalBalance` in RAVEN and `get_elemental_balance` in raven-toolbox check whether each element (carbon, hydrogen, oxygen and so on) has the same count on both sides of a reaction. A reaction is unbalanced when the counts were computed and did not match, and the result is unknown when the counts cannot be computed. The two functions use different codes for these results. +`getElementalBalance` in RAVEN and `get_elemental_balance` in raven-toolbox check whether each element has the same count on both sides of a reaction. A reaction is unbalanced when the counts were computed and did not match, and the result is unknown when the counts cannot be computed. The two functions use different codes for these results. + +`getElementalBalance` returns the numeric field `balanceStructure.balanceStatus` per reaction: 1 for balanced, 0 for unbalanced, -1 for missing information (no formula, or a reaction without metabolites) and -2 for a formula that cannot be parsed. `get_elemental_balance` returns the strings `balanced`, `unbalanced` and `unknown`. It reports `unknown` for a missing formula, for an unparseable formula and for a reaction without metabolites, so the codes -1 and -2 of RAVEN are one result in raven-toolbox. + +The two parsers differ for some formulas, so the results differ for some reactions with formulas: -`getElementalBalance` returns the numeric field `balanceStructure.balanceStatus` per reaction: 1 for balanced, 0 for unbalanced, -1 for missing information (no formula, or a reaction without metabolites) and -2 for a formula that cannot be parsed. `get_elemental_balance` returns the strings `balanced`, `unbalanced` and `unknown`. It reports `unknown` for a missing formula, for a formula that cobrapy cannot parse and for a reaction without metabolites, so the codes -1 and -2 of RAVEN are one result in raven-toolbox. For a reaction whose metabolites all have parseable formulas, the two functions return the same result. +- A polymer formula such as `(C5H8)n` is counted with n equal to 1 in RAVEN. raven-toolbox reports `unknown`. +- RAVEN uses the `inchis` field when `metFormulas` is absent. raven-toolbox does not. +- RAVEN reports a charge balance (`chargeStatus` and `chargeResidual`). `get_elemental_balance` reports elements only. -cobrapy's `check_mass_balance` has no separate `unknown` result and raises an error for a formula that it cannot parse. For a reaction with a missing formula it treats the formula as empty and returns either an element imbalance or an empty result, so it gives the same kind of result as for a reaction with known formulas. raven-toolbox keeps its own function to retain the distinction. +The comparison used the code only; the functions were not run on a shared model. ## Parsing of gene-reaction rules -raven-toolbox and the JSON reader of cobrapy parse gene-reaction rules (the `grRules` field in RAVEN) differently. raven-toolbox reads them with `utils.parse_gr_rule` in `read_yaml_model`, `read_matlab_model`, `batch_curate` and the functions that copy or build rules, so `G1&G2` is one gene ID there. The JSON reader of cobrapy splits `G1&G2` into `G1` and `G2`. +The section applies when a RAVEN user takes a model through cobrapy readers in Python. raven-toolbox parses gene-reaction rules (the `grRules` field in RAVEN) with `utils.parse_gr_rule`, which follows the grammar of RAVEN's `parseGrRule`. The function is used in `read_yaml_model`, `read_matlab_model`, `batch_curate` and the functions that copy or build rules, so `G1&G2` is one gene ID there. According to the raven-toolbox changelog, the JSON reader of cobrapy splits `G1&G2` into `G1` and `G2`, and the SBML reader keeps it. This was not checked against cobrapy itself. diff --git a/docs/raven3-vs-raven-toolbox/what-each-has.md b/docs/raven3-vs-raven-toolbox/what-each-has.md index 76c5e85..abe6111 100644 --- a/docs/raven3-vs-raven-toolbox/what-each-has.md +++ b/docs/raven3-vs-raven-toolbox/what-each-has.md @@ -6,36 +6,49 @@ The page describes the features that exist in raven-toolbox only and the feature ### Model type -A raven-toolbox model is a `cobra.Model`. Every cobrapy function and cobrapy-based package applies to it without conversion. RAVEN uses a MATLAB struct as the model, and `ravenCobraWrapper` converts between that struct and the COBRA Toolbox format. +A raven-toolbox model is a `cobra.Model`. Every cobrapy function and cobrapy-based package applies to it without conversion. RAVEN uses a MATLAB struct as the model. -cobrapy provides flux balance analysis (FBA), flux variability analysis (FVA), knockouts and media handling, in addition to the other functions that the mapping table marks as covered by cobrapy. The mapping table therefore gives a cobrapy entry in place of a raven-toolbox function for the corresponding RAVEN functions. +cobrapy provides flux balance analysis (FBA), flux variability analysis (FVA), knockouts and media handling. The [MATLAB vs Python](../matlab-vs-python.md) page gives a cobrapy entry for the corresponding RAVEN functions where cobrapy covers them; [Name differences](../raven3-vs-raven-toolbox.md#name-differences) describes the page. `setExchangeBounds` has two rows in the mapping: `set_exchange_bounds` in raven-toolbox and `cobra.Model.medium` in cobrapy. ### KEGG artefact generation -raven-toolbox builds the KEGG reference artefacts: it parses a KEGG release into reaction and compound tables, assembles the reference model, builds the per-KEGG Orthology (KO) FASTA sets and HMM libraries, and derives the phylogenetic distance matrix. RAVEN uses pre-built artefacts and does not produce them. +raven-toolbox builds the KEGG reference artefacts: it parses a KEGG release into reaction and compound tables, assembles the reference model with `build_reference_model`, builds the per-KEGG Orthology (KO) FASTA sets and HMM libraries, and derives the phylogenetic distance matrix. The download of a KEGG release requires a KEGG FTP subscription. The artefacts can be rebuilt from a specified KEGG release. -The artefacts can therefore be rebuilt from a specified KEGG release. +RAVEN loads published artefacts and does not produce them. `buildGlobalKEGGModel` joins a downloaded bundle onto the reference model, and the phylogenetic distance matrix is a file shipped with RAVEN (`keggPhylDist.mat`). In the mapping, `getModelFromKEGG` corresponds to `build_reference_model`, which loads in RAVEN and builds in raven-toolbox. ### Checksum registry for binaries and data -Both implementations fetch BLAST+, DIAMOND, HMMER and the KEGG artefacts on demand, do not bundle them, pin a release and check downloads against checksums (see [Download data and binaries](../installation/data-and-binaries.md)). +BLAST+, DIAMOND, HMMER and the KEGG tables are fetched on demand in both implementations and are not bundled. RAVEN bundles other components: the GLPK and libSBML MEX files and `keggPhylDist.mat`. [Download data and binaries](../installation/data-and-binaries.md) describes the download behaviour. -raven-toolbox has the checksums and the release of each dataset in a registry in its source, which ties each release to the assets it was tested with. RAVEN checks a download against the checksum of the published manifest. When the manifest cannot be read or has no entry for the asset, RAVEN issues a warning and keeps the file unchecked, and an already cached file is reused without a check. +The checksum handling differs: + +- raven-toolbox keeps the checksum and release of each dataset and binary in a registry in its source (a version-pinned registry). A fresh download is checked against its checksum. A cached data file is reused without a check unless `ensure_data_file` is called with `verify=True`, which re-checks it and downloads it again on a mismatch; a cached binary is reused without a check. A JSON manifest can be merged into the registries with `raven_toolbox.manifest.load_into_registries`, which reads the path or URL in the environment variable `RAVEN_PYTHON_MANIFEST` when no source is passed. The resolvers load that variable by themselves only while the registry is empty, so with the filled default registries the call is needed. +- RAVEN checks a download against the checksum in the published manifest. A mismatch deletes the file and raises an error. When the manifest cannot be read or has no entry for the asset, RAVEN issues a warning and keeps the file unchecked. A cached file is reused without a check. + +### Further functions + +Further raven-toolbox features have no RAVEN counterpart in the generated mapping. Examples are `localization.relocate_reactions`, which places reactions in compartments chosen by a curator and repairs the model around the decision, `localization.triage_localization`, and the per-reaction confidence record of the `confidence` module, which scores the facets localization, equation and gene association and stores them in the reaction notes. ## Features specific to RAVEN ### tINIT -`getINITModel` and `runINIT` are the original tINIT implementation. RAVEN keeps them for models built with them. raven-toolbox has no earlier tINIT models to reproduce, so it implements ftINIT only. A tINIT model has to be reproduced in MATLAB; see [Context-specific models with ftINIT](../guide/init.md). +`getINITModel` and `runINIT` are the original tINIT implementation. RAVEN keeps them for models built with them. raven-toolbox implements ftINIT only, as a design decision. A tINIT model has to be reproduced in MATLAB; see [Context-specific models with ftINIT](../guide/init.md). ### COBRA Toolbox conversion -`ravenCobraWrapper` converts between the RAVEN and COBRA Toolbox model structures. In raven-toolbox the model is a `cobra.Model`, so no conversion is needed and no equivalent exists. +`ravenCobraWrapper` converts a model in memory between the RAVEN and COBRA Toolbox structures. A round trip drops some fields. In raven-toolbox the model is a `cobra.Model`, so no in-memory conversion is needed and no equivalent exists. File exchange is described in [Choosing an implementation](../raven3-vs-raven-toolbox.md#choosing-an-implementation); cobrapy writes the COBRA layout. ### Dynamic FBA -`runDynamicFBA` has no raven-toolbox counterpart by design. Python packages cover dynamic FBA ([dfba](https://pypi.org/project/dfba/), [reframed](https://pypi.org/project/reframed/), [mewpy](https://pypi.org/project/mewpy/)), and raven-toolbox does not reimplement it. +`runDynamicFBA` has no raven-toolbox counterpart by design. Python packages cover dynamic FBA ([dfba](https://pypi.org/project/dfba/), [reframed](https://pypi.org/project/reframed/), [mewpy](https://pypi.org/project/mewpy/)); the packages were not checked here. ### MATLAB-specific functions -RAVEN has MATLAB-specific infrastructure: path management (`addRavenToUserPath`, `findRAVENroot`), argument handling (`parseRAVENargs`, `convertCharArray`), progress and printing (`setRavenProgress`, `printOrange`) and the solver abstraction (`optimizeProb`, `setRavenSolver`). cobrapy's solver interface (optlang) takes the place of the solver abstraction. +RAVEN has MATLAB-specific infrastructure that raven-toolbox does not port: path management (`addRavenToUserPath`, `findRAVENroot`), argument handling (`parseRAVENargs`, `convertCharArray`), progress and printing (`setRavenProgress`, `printOrange`) and the solver abstraction (`optimizeProb`, `setRavenSolver`). cobrapy's solver interface (optlang) takes the place of the solver abstraction. + +The generated mapping table lists the functions in its tracked categories. `addRavenToUserPath` and `findRAVENroot` belong to the installation folder, which is not tracked, so the table does not list them. The "Deliberately not ported" table lists further infrastructure functions, for example `checkSolution`, `permuteModel`, `parseYAML` and `deprecationWarning`. + +### Functions without a Python counterpart + +Some RAVEN functions that raven-toolbox does not port are not infrastructure. `getTransportRxns`, `runRobustnessAnalysis` and `runSimpleOptKnock` have no replacement and can block a port. `readFasta` has no public reader, and `findPotentialErrors` has no step that detects an invalid rule string, because the cobrapy GPR class represents a parsed rule directly. The table gives other replacements: the `.annotation` dictionary of cobrapy for `editMiriam` and `extractMiriam`, the `cobra.Reaction` API for `reverseRxns` and `model.summary()` for `printFluxes`. The "Deliberately not ported" table of the [MATLAB vs Python](../matlab-vs-python.md) page is the reference. diff --git a/scripts/curated_pairs.yml b/scripts/curated_pairs.yml index c43f052..7e3d9b0 100644 --- a/scripts/curated_pairs.yml +++ b/scripts/curated_pairs.yml @@ -27,6 +27,7 @@ aliases: convertToIrrev: convert_to_irreversible copyToComps: copy_to_compartment findDuplicateRxns: find_duplicate_reactions + removeBadRxns: remove_bad_reactions removeMets: remove_metabolites simplifyModel: remove_dead_end_reactions sortModel: sort_identifiers @@ -251,11 +252,6 @@ not_ported: getTransportRxns: >- Classifies which reactions move a metabolite between compartments. raven-toolbox has no equivalent classifier yet. - removeBadRxns: >- - Iteratively removes reactions that let the model make or consume - something from nothing. raven-toolbox has no equivalent yet; see - find_leak_metabolite (mapped from makeSomething/consumeSomething) for - finding the same kind of issue. reverseRxns: >- Negates a reaction's stoichiometry and bounds to reverse its direction. Simple enough to do directly against a cobra.Reaction's own API; no diff --git a/scripts/known_names.txt b/scripts/known_names.txt index 489b150..77f5744 100644 --- a/scripts/known_names.txt +++ b/scripts/known_names.txt @@ -437,3 +437,45 @@ run_hmmscan # imported by analyze_hmm_cutoffs.py; not in query parse_hmmscan_tblout # imported by analyze_hmm_cutoffs.py; not in query.py cannot_connect # output of connect_blocked_reactions ml_prep # MATLAB script of the validation study +fastLP # value of the algorithm option of fillGaps +swiftLP # value of the algorithm option of fillGaps +gapfillMILP # value of the algorithm option of fillGaps +cannotConnect # output of fillGaps +newModel # output of fillGaps +exitFlag # output of fillGaps +chargeStatus # field of the getElementalBalance output +chargeResidual # field of the getElementalBalance output +# parameter and output names in the RAVEN 2 to RAVEN 3 migration pages +goodRxns +rxnToCheck +nObjectives +checkBinaries +failedTasks +getWoLFScores + +# RAVEN 2 identifiers, options and fields named in the migration pages (functions removed in RAVEN 3, options of RAVEN 2 or 3 functions, model fields) +complexPolicy +geneScores +getToolboxVersion +grRule +importExportTests +keepGeneFree +parallelPoolRAVEN +preserveNotes +preserveQuotes +rxnFrom +solverTests +trimWarnings +xml_format +xml_formatany +xml_help +xml_load +xml_parse +xml_parseany +xml_save +xml_strsplit +xml_unblank +# Test classes of RAVEN (tests/function_tests) and a MATLAB built-in +tBinaries +tParseRAVENargs +regexpPattern From a7645dceb11724a328727a27b00ef189a7f54b0c Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Sun, 1 Nov 2026 11:19:10 +0100 Subject: [PATCH 09/20] Name the removed MetaCyc, pathway, plotting and legacy code as folders --- docs/raven3-migration.md | 4 ++-- docs/raven3-migration/backward-incompatible.md | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/raven3-migration.md b/docs/raven3-migration.md index a3f859c..7d1bef5 100644 --- a/docs/raven3-migration.md +++ b/docs/raven3-migration.md @@ -2,7 +2,7 @@ The calling convention and an upgrade checklist for users of RAVEN 2 follow. Four detail pages follow the checklist. All statements on RAVEN 2 refer to RAVEN 2.11.3. The changelog of RAVEN 3 states its changes relative to RAVEN 2.9.3, so changelog entries may describe differences that are absent from the comparison with 2.11.3. The last RAVEN 2 release, 2.11.4, differs from 2.11.3 by bug fixes in `ftINIT`, `mergeLinear`, `replaceMets`, `checkInstallation`, `importModel`, `readYAMLmodel` and `optimizeProb`. The comparison with the Python package is in [Comparison of RAVEN and raven-toolbox](raven3-vs-raven-toolbox.md). -RAVEN 3 is a breaking release. The codebase was reorganised, functions were removed (counts per subsystem are in [Structure, removals and renames](raven3-migration/structure-and-renames.md)), several external dependencies were dropped and the calling convention for optional arguments changed in the 112 functions that use `parseRAVENargs`. The rationale is in the [v3 development review](https://github.com/CompMetEng/RAVEN/blob/develop/docs/v3_review.md) of the project. Scripts that use reconstruction, KEGG, localisation or gene-deletion functions need a check. The changes that can affect a script are listed in [Backward-incompatible changes](raven3-migration/backward-incompatible.md#backward-incompatible-changes). +RAVEN 3 is a breaking release. The codebase was reorganised, functions were removed (counts per folder are in [Structure, removals and renames](raven3-migration/structure-and-renames.md)), several external dependencies were dropped and the calling convention for optional arguments changed in the 112 functions that use `parseRAVENargs`. The rationale is in the [v3 development review](https://github.com/CompMetEng/RAVEN/blob/develop/docs/v3_review.md) of the project. Scripts that use reconstruction, KEGG, localisation or gene-deletion functions need a check. The changes that can affect a script are listed in [Backward-incompatible changes](raven3-migration/backward-incompatible.md#backward-incompatible-changes). :::{admonition} Changes to check first :class: warning @@ -44,7 +44,7 @@ An old positional call keeps its meaning when the order of the parameters is unc | Page | Content | |---|---| -| [Structure, removals and renames](raven3-migration/structure-and-renames.md) | The folder reorganisation, every removed function and subsystem, and every renamed or merged function. | +| [Structure, removals and renames](raven3-migration/structure-and-renames.md) | The folder reorganisation, every removed function and folder, and every renamed or merged function. | | [Backward-incompatible changes](raven3-migration/backward-incompatible.md) | Changes that raise errors, changes that alter results without an error, changes to error and warning identifiers, changes that raise warnings and installation changes. | | [File formats, reconstruction and GPR parsing](raven3-migration/formats-and-reconstruction.md) | SBML, Excel and YAML I/O; the KEGG and homology reconstruction pipelines; the grRule parser. | | [New functionality, dependencies and error handling](raven3-migration/new-and-environment.md) | Gap-filling and ftINIT changes, functions added in RAVEN 3, the toolbox dependency comparison, and error and warning handling. | diff --git a/docs/raven3-migration/backward-incompatible.md b/docs/raven3-migration/backward-incompatible.md index 72e0810..5e7ae63 100644 --- a/docs/raven3-migration/backward-incompatible.md +++ b/docs/raven3-migration/backward-incompatible.md @@ -17,9 +17,9 @@ The list covers the changes found by comparing RAVEN 2.11.3 with RAVEN 3 in the | `findGeneDeletions(...)` with `analysisType`, `refModel` or `oeFactor` | The arguments are removed together with `qMOMA` (MOMA-based over-expression analysis). The test types `'sgo'` and `'dgo'` are removed. Single and double gene deletion (`sgd`, `dgd`) by flux balance analysis (FBA) remain. More than the `'testType'` argument raises `parseRAVENargs:tooManyArgs` | | `checkRxn(...)` | Removed, no direct successor. To find why a reaction cannot carry flux, use `gapfilling/findLeakMetabolite.m`, `gapfilling/canExchange.m` or `gapfilling/checkProduction.m` | | `dispEM(...)`, `followFluxes(...)`, `followChanged(...)`, `printModel(...)`, `mapCompartments(...)`, `getExpressionStructure(...)`, `getMetsInComp(...)` | Removed (see [Removed functionality](#removed-functionality) for the replacements, where one exists) | -| Anything under `external/metacyc/` | Subsystem removed (MetaCyc-based reconstruction, 7 functions). No replacement in RAVEN 3 | -| Anything under `pathway/` or `plotting/` | Subsystems removed: 16 functions for visualisation and plotting, such as `drawMap`, `drawPathway`, `markPathwayWithFluxes` and `setOmicDataToRxns`. No replacement in RAVEN 3; the COBRA Toolbox and Escher provide plotting | -| Anything under `legacy/` | Subsystem removed. No replacement in RAVEN 3 (see [Removed functionality](#removed-functionality)) | +| Anything under `external/metacyc/` | Folder removed (MetaCyc-based reconstruction, 7 functions). No replacement in RAVEN 3 | +| Anything under `pathway/` or `plotting/` | Folders removed: 16 functions for visualisation and plotting, such as `drawMap`, `drawPathway`, `markPathwayWithFluxes` and `setOmicDataToRxns`. No replacement in RAVEN 3; the COBRA Toolbox and Escher provide plotting | +| Anything under `legacy/` | Folder removed. No replacement in RAVEN 3 (see [Removed functionality](#removed-functionality)) | | `getGenesFromKEGG`, `getMetsFromKEGG`, `getRxnsFromKEGG`, `constructMultiFasta`, `getWSLpath`, `getBlastFromExcel` | Removed (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)) | | `getToolboxVersion(...)` | Removed as a standalone function. It is a local function of `exportForGit` | | `parallelPoolRAVEN(...)` | Renamed `parallelWorkersRAVEN`, with a different signature and output (see [Renamed and merged functions](#renamed-merged-and-consolidated-functions)) | From fb8faba2572c676a684e86d91f4dcd9fe0c56333 Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Sun, 1 Nov 2026 11:28:01 +0100 Subject: [PATCH 10/20] Write the migration and comparison pages anew from their verified claims --- docs/migrate.md | 6 +- docs/raven3-migration.md | 38 +++--- .../raven3-migration/backward-incompatible.md | 81 ++++++------ .../formats-and-reconstruction.md | 100 +++++++------- docs/raven3-migration/new-and-environment.md | 109 +++++++++------- .../raven3-migration/structure-and-renames.md | 122 +++++++++--------- docs/raven3-vs-raven-toolbox.md | 45 ++++--- .../same-function-different-answer.md | 50 +++---- docs/raven3-vs-raven-toolbox/what-each-has.md | 35 ++--- 9 files changed, 309 insertions(+), 277 deletions(-) diff --git a/docs/migrate.md b/docs/migrate.md index 6d6c48d..e416adc 100644 --- a/docs/migrate.md +++ b/docs/migrate.md @@ -1,11 +1,11 @@ # Migration guides -Two pages describe differences between toolboxes. One compares RAVEN 3 with RAVEN 2, the other compares RAVEN 3 with the Python package raven-toolbox. +Two pages compare toolboxes. One compares RAVEN 3 with RAVEN 2, the other RAVEN 3 with the Python package raven-toolbox. | Page | Use | |---|---| -| [RAVEN 2 → RAVEN 3 (MATLAB)](raven3-migration.md) | Upgrading an existing MATLAB codebase: which functions were renamed or removed, and which behaviour differs between RAVEN 2 and RAVEN 3. | -| [RAVEN 3 → raven-toolbox](raven3-vs-raven-toolbox.md) | Choosing between the two current implementations, or porting a MATLAB workflow to Python: what each toolbox has, and where they answer the same question differently. | +| [RAVEN 2 → RAVEN 3 (MATLAB)](raven3-migration.md) | Upgrading a MATLAB codebase: which functions were renamed or removed, and which behaviour differs between RAVEN 2 and RAVEN 3. | +| [RAVEN 3 → raven-toolbox](raven3-vs-raven-toolbox.md) | Choosing between the two current implementations or porting a MATLAB workflow to Python: what each toolbox has, and where they answer the same question differently. | ```{toctree} :hidden: diff --git a/docs/raven3-migration.md b/docs/raven3-migration.md index 7d1bef5..675c2bf 100644 --- a/docs/raven3-migration.md +++ b/docs/raven3-migration.md @@ -1,22 +1,24 @@ # RAVEN 2 to RAVEN 3 -The calling convention and an upgrade checklist for users of RAVEN 2 follow. Four detail pages follow the checklist. All statements on RAVEN 2 refer to RAVEN 2.11.3. The changelog of RAVEN 3 states its changes relative to RAVEN 2.9.3, so changelog entries may describe differences that are absent from the comparison with 2.11.3. The last RAVEN 2 release, 2.11.4, differs from 2.11.3 by bug fixes in `ftINIT`, `mergeLinear`, `replaceMets`, `checkInstallation`, `importModel`, `readYAMLmodel` and `optimizeProb`. The comparison with the Python package is in [Comparison of RAVEN and raven-toolbox](raven3-vs-raven-toolbox.md). +RAVEN 3 is a breaking release. A script written for RAVEN 2 can stop with an error, or run and return a different model. The sections below list what to check first, give the calling convention for optional arguments and provide a checklist for the upgrade. Four detail pages hold the complete lists. -RAVEN 3 is a breaking release. The codebase was reorganised, functions were removed (counts per folder are in [Structure, removals and renames](raven3-migration/structure-and-renames.md)), several external dependencies were dropped and the calling convention for optional arguments changed in the 112 functions that use `parseRAVENargs`. The rationale is in the [v3 development review](https://github.com/CompMetEng/RAVEN/blob/develop/docs/v3_review.md) of the project. Scripts that use reconstruction, KEGG, localisation or gene-deletion functions need a check. The changes that can affect a script are listed in [Backward-incompatible changes](raven3-migration/backward-incompatible.md#backward-incompatible-changes). +All statements about RAVEN 2 refer to release 2.11.3. The last RAVEN 2 release, 2.11.4, differs from 2.11.3 by bug fixes in `ftINIT`, `mergeLinear`, `replaceMets`, `checkInstallation`, `importModel`, `readYAMLmodel` and `optimizeProb`. The changelog of RAVEN 3 states its changes relative to RAVEN 2.9.3, so an entry in it can describe a difference that does not exist against 2.11.3. The comparison with the Python package is in [Comparison of RAVEN and raven-toolbox](raven3-vs-raven-toolbox.md). + +The codebase was reorganised, functions were removed (counts per folder are in [Structure, removals and renames](raven3-migration/structure-and-renames.md)), several external dependencies were dropped, and 112 functions read their optional arguments through `parseRAVENargs`. The rationale is in the [v3 development review](https://github.com/CompMetEng/RAVEN/blob/develop/docs/v3_review.md). Scripts that call reconstruction, KEGG, localisation or gene-deletion functions are the most likely to be affected. The changes that can affect any script are in [Backward-incompatible changes](raven3-migration/backward-incompatible.md#backward-incompatible-changes). :::{admonition} Changes to check first :class: warning -- The same homology-based or KEGG-based reconstruction call returns a different draft model without an error (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)). -- Scripts that use WoLF PSORT localisation scores raise an error (see [Backward-incompatible changes](raven3-migration/backward-incompatible.md#backward-incompatible-changes)). +- A homology-based or KEGG-based reconstruction call returns a different draft model without an error (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)). +- A script that uses WoLF PSORT localisation scores raises an error (see [Backward-incompatible changes](raven3-migration/backward-incompatible.md#backward-incompatible-changes)). ::: (calling-convention)= ## Calling convention -RAVEN 2 functions took optional arguments by position (`f(model, a, b, c)`). Setting a later optional argument required all earlier optional arguments to be given. In these pages a position N is the position among all arguments of the call, including the required ones: in `f(model, a, b, c)` the model is position 1 and `a` is position 2. +A RAVEN 2 function took its optional arguments by position, as in `f(model, a, b, c)`, and setting a later optional argument required every earlier one. A position N in these pages is the position among all arguments of the call, required ones included. In `f(model, a, b, c)` the model is position 1 and `a` is position 2. -RAVEN 3 has the function `utils/parseRAVENargs.m`, which 112 functions call. The required leading arguments are explicit positional parameters. All optional arguments are collected in `varargin` and can be given by position, by name or in a mix of both: +In RAVEN 3 the function `utils/parseRAVENargs.m` reads the arguments of 112 functions. The required leading arguments are positional parameters. The optional arguments are collected in `varargin` and can be given by position, by name or by both: ```matlab removeReactions(model, rxnList, true, true, true) % positional (RAVEN 2 style) @@ -24,29 +26,29 @@ removeReactions(model, rxnList, "removeUnusedGenes", true) % named removeReactions(model, rxnList, true, "removeUnusedComps", true) % hybrid ``` -The first string argument that matches a parameter name (case-insensitive) starts the name-value pairs. The arguments before it are positional and are assigned in the order of the `parseRAVENargs` specification of the function. If no argument matches, or the arguments from the first match on are not valid name-value pairs, all arguments are positional. More positional values than the specification has parameters raise `parseRAVENargs:tooManyArgs`. A positional argument that is a string equal to a parameter name has to be given in the name-value form. +The first string argument that matches a parameter name, ignoring case, starts the name-value pairs. The arguments before it are positional and are assigned in the order of the `parseRAVENargs` specification of the function. When no argument matches, or the arguments from the first match on are not valid name-value pairs, all arguments are positional. More positional values than the specification has parameters raise `parseRAVENargs:tooManyArgs`. A positional argument whose value is a string equal to a parameter name has to be given in the name-value form. -An old positional call keeps its meaning when the order of the parameters is unchanged. `importModel`, `exportModel`, `exportToExcelFormat`, `removeReactions`, `addIdentifierPrefix`, `removeIdentifierPrefix` and `exportForGit` have the same parameters in the same order in RAVEN 2 and RAVEN 3. `ftINIT`, `fillGaps`, `fitTasks` and `getModelFromHomology` append new optional parameters after the RAVEN 2 ones. A comparison of all 112 specifications with the RAVEN 2.11.3 signatures found six functions where positional calls change meaning or fail: `getModelFromKEGG`, `getPhylDist`, `getKEGGModelForOrganism`, `scoreModel`, `writeYAMLmodel` and `findGeneDeletions`. Each of them has a row in [Backward-incompatible changes](raven3-migration/backward-incompatible.md#backward-incompatible-changes). +A positional call keeps its meaning where the parameter order is unchanged. This holds for `importModel`, `exportModel`, `exportToExcelFormat`, `removeReactions`, `addIdentifierPrefix`, `removeIdentifierPrefix` and `exportForGit`. The functions `ftINIT`, `fillGaps`, `fitTasks` and `getModelFromHomology` append new optional parameters after the RAVEN 2 ones, so their old calls also keep their meaning. A comparison of all 112 specifications with the RAVEN 2.11.3 signatures found six functions in which a positional call changes meaning or fails: `getModelFromKEGG`, `getPhylDist`, `getKEGGModelForOrganism`, `scoreModel`, `writeYAMLmodel` and `findGeneDeletions`. Each has a row in [Backward-incompatible changes](raven3-migration/backward-incompatible.md#backward-incompatible-changes). (upgrade-checklist)= ## Upgrade checklist -1. Install RAVEN 3 and run `checkRaven`. It checks the installation type, the MATLAB release, the libSBML version, SBML import and export, the LP solvers, the BLAST+, DIAMOND and HMMER binaries and function names that overlap with other toolboxes. Unless `checkBinaries` is false, `checkRaven` downloads any of the three binaries that is missing (see [Toolbox dependencies and installation](#toolbox-deps)). `checkInstallation` forwards to `checkRaven` and warns on every call. -2. Search the scripts for the functions listed in [Removed functionality](raven3-migration/structure-and-renames.md#removed-functionality) and in [Renamed and merged functions](#renamed-merged-and-consolidated-functions), and replace or remove those calls. The wrappers in `deprecated/` (`makeSomething`, `consumeSomething`, `canProduce`, `canConsume`) and `getINITModel` and `runINIT` work and warn once per session. -3. Rebuild and validate any draft model from `getModelFromKEGG`, `getKEGGModelForOrganism` or `getModelFromHomology`. Default cutoffs, the KEGG data source and the best-hit criterion differ, so an unmodified call gives a different draft (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)). Check positional calls of the six functions listed under [Calling convention](#calling-convention). -4. Convert `.xlsx` curation templates to the `.tsv` format of `curateModelFromTables`. `importExcelModel` does not exist in RAVEN 3 (see [Excel I/O](raven3-migration/formats-and-reconstruction.md#excel-io)). -5. Replace WoLF PSORT scores (`getWoLFScores`, or `parseScores` with `'wolf'`) with DeepLoc 2, MULocDeep, COMPARTMENTS or UniProt. The default predictor of `parseScores` differs (see [Backward-incompatible changes](raven3-migration/backward-incompatible.md#backward-incompatible-changes)). -6. Re-run task-based gap-filling and context-specific model extraction (`fitTasks`, `checkTasks`, `ftINIT`) and compare the task results. `simplifyModel` keeps the boundary constraint of a task on a metabolite without reactions, which RAVEN 2 removed, so the task check can give a different result in RAVEN 2 and RAVEN 3 (not run; see [Gap-filling and context-specific model extraction](#ftinit)). `fitTasks` returns a third output, `failedTasks`. -7. Expect differences in YAML and SBML files on the next export of a model under version control. The changelog of RAVEN 3 lists changes in quoting, EC-code annotation and ΔG fields; these differences were not compared file by file (see [File formats and model I/O](raven3-migration/formats-and-reconstruction.md#file-formats-and-model-io)). -8. Compare the output of the validation pipeline used for the model (model size, growth rate and task results) with the RAVEN 2 output before using RAVEN 3 results. +1. Install RAVEN 3 and run `checkRaven`. It checks the installation type, the MATLAB release, the libSBML version, SBML import and export, the LP solvers, the BLAST+, DIAMOND and HMMER binaries, and function names that overlap with other toolboxes. Unless `checkBinaries` is false, `checkRaven` downloads any of the three binaries that is missing (see [Toolbox dependencies and installation](#toolbox-deps)). The function `checkInstallation` forwards to `checkRaven` and warns on every call. +2. Search the scripts for the functions in [Removed functionality](raven3-migration/structure-and-renames.md#removed-functionality) and in [Renamed and merged functions](#renamed-merged-and-consolidated-functions), and replace or delete those calls. The wrappers in `deprecated/` (`makeSomething`, `consumeSomething`, `canProduce`, `canConsume`) work and warn once per session, as do `getINITModel` and `runINIT`. +3. Rebuild and validate every draft model from `getModelFromKEGG`, `getKEGGModelForOrganism` or `getModelFromHomology`. The default cut-offs, the KEGG data source and the best-hit criterion differ, so an unchanged call gives a different draft (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)). Check the positional calls of the six functions in [Calling convention](#calling-convention). +4. Convert `.xlsx` curation templates to the `.tsv` format of `curateModelFromTables`. The function `importExcelModel` does not exist in RAVEN 3 (see [Excel I/O](raven3-migration/formats-and-reconstruction.md#excel-io)). +5. Replace WoLF PSORT scores (`getWoLFScores`, or `parseScores` with `'wolf'`) with DeepLoc 2, MULocDeep, COMPARTMENTS or UniProt. The default predictor of `parseScores` also differs (see [Backward-incompatible changes](raven3-migration/backward-incompatible.md#backward-incompatible-changes)). +6. Run task-based gap-filling and context-specific model extraction (`fitTasks`, `checkTasks`, `ftINIT`) again and compare the task results. `simplifyModel` keeps the boundary constraint of a task on a metabolite without reactions, which RAVEN 2 removed, so the task check can give a different result in RAVEN 2 and RAVEN 3. This was not run (see [Gap-filling and context-specific model extraction](#ftinit)). The function `fitTasks` returns a third output, `failedTasks`. +7. Expect differences in YAML and SBML files on the next export of a model under version control. The changelog of RAVEN 3 lists changes in quoting, EC-code annotation and ΔG fields. These differences were not compared file by file (see [File formats and model I/O](raven3-migration/formats-and-reconstruction.md#file-formats-and-model-io)). +8. Compare the output of the validation pipeline of the model (model size, growth rate and task results) between RAVEN 2 and RAVEN 3 before using RAVEN 3 results. ## Detail pages | Page | Content | |---|---| | [Structure, removals and renames](raven3-migration/structure-and-renames.md) | The folder reorganisation, every removed function and folder, and every renamed or merged function. | -| [Backward-incompatible changes](raven3-migration/backward-incompatible.md) | Changes that raise errors, changes that alter results without an error, changes to error and warning identifiers, changes that raise warnings and installation changes. | -| [File formats, reconstruction and GPR parsing](raven3-migration/formats-and-reconstruction.md) | SBML, Excel and YAML I/O; the KEGG and homology reconstruction pipelines; the grRule parser. | +| [Backward-incompatible changes](raven3-migration/backward-incompatible.md) | Changes that raise errors, changes that alter results without an error, changes to error and warning identifiers, changes that raise warnings, and installation changes. | +| [File formats, reconstruction and GPR parsing](raven3-migration/formats-and-reconstruction.md) | SBML, Excel and YAML I/O, the KEGG and homology reconstruction pipelines, and the grRule parser. | | [New functionality, dependencies and error handling](raven3-migration/new-and-environment.md) | Gap-filling and ftINIT changes, functions added in RAVEN 3, the toolbox dependency comparison, and error and warning handling. | ```{toctree} diff --git a/docs/raven3-migration/backward-incompatible.md b/docs/raven3-migration/backward-incompatible.md index 5e7ae63..a1aa930 100644 --- a/docs/raven3-migration/backward-incompatible.md +++ b/docs/raven3-migration/backward-incompatible.md @@ -1,83 +1,90 @@ (backward-incompatible-changes)= # Backward-incompatible changes -The list covers the changes found by comparing RAVEN 2.11.3 with RAVEN 3 in the code. The changelog of RAVEN 3 lists further bug fixes, which are named in the last row of the table of corrected behaviour. The changes are grouped into those that raise an error, those that give a different result without an error, changes to error and warning identifiers, changes that raise warnings and installation changes. +The changes below were found by comparing RAVEN 2.11.3 with RAVEN 3 in the code. They are grouped into changes that raise an error, changes that give a different result without an error, changes to error and warning identifiers, changes that raise warnings, and installation changes. The changelog of RAVEN 3 lists further bug fixes. Those whose effect was not checked are named after the table of corrected behaviour. ## Changes that raise errors | RAVEN 2 usage | RAVEN 3 behaviour | |---|---| -| `importExcelModel(...)` | Removed. Use `curation/curateModelFromTables.m` (input format `.tsv`), or convert the template to SBML and use `importModel`, or to YAML and use `readYAMLmodel` (see [Excel I/O](#excel-io)) | -| `importModel(...)` on SBML Level < 3, Level 3 without FBC v2, or any non-FBC SBML | Raises an error. The accepted format is SBML Level 3 Version 1 with FBC package version 2 (see [SBML I/O](#sbml-io)) | -| `exportToExcelFormat(model, fileName)` where `fileName` has no file name or no extension | RAVEN 2 exported tab-delimited text. RAVEN 3 raises an error for every extension other than `.xlsx`, including an empty one, with the message "only export to xlsx format is supported". RAVEN 2 already raised this error for a file name with another extension | +| `importExcelModel(...)` | Removed. Use `curation/curateModelFromTables.m` (input format `.tsv`). Alternatively convert the template to SBML and read it with `importModel`, or to YAML and read it with `readYAMLmodel` (see [Excel I/O](#excel-io)) | +| `importModel(...)` on SBML Level < 3, on Level 3 without FBC version 2, or on any SBML without FBC | Raises an error. The accepted format is SBML Level 3 Version 1 with FBC package version 2 (see [SBML I/O](#sbml-io)) | +| `exportToExcelFormat(model, fileName)` where `fileName` has no file name or no extension | RAVEN 2 exported tab-delimited text. RAVEN 3 raises the error "only export to xlsx format is supported" for every extension other than `.xlsx`, an empty one included. RAVEN 2 already raised this error for a file name with another extension | | `exportToTabDelimited(...)` | Removed | -| `getWoLFScores(...)` | Removed. WoLF PSORT support is removed | -| `parseScores(..., 'wolf', ...)` | Raises an error with a message that names `'deeploc'`, `'mulocdeep'`, `'compartments'` and `'uniprot'`. RAVEN 3 has no function that reads WoLF PSORT output files | +| `getWoLFScores(...)` | Removed, with all WoLF PSORT support | +| `parseScores(..., 'wolf', ...)` | Raises an error whose message names `'deeploc'`, `'mulocdeep'`, `'compartments'` and `'uniprot'`. RAVEN 3 has no function that reads WoLF PSORT output files | | `qMOMA(...)`, `solveQP(...)` | Removed. RAVEN 3 has no code that depends on `quadprog` of the Optimization Toolbox | -| `findGeneDeletions(...)` with `analysisType`, `refModel` or `oeFactor` | The arguments are removed together with `qMOMA` (MOMA-based over-expression analysis). The test types `'sgo'` and `'dgo'` are removed. Single and double gene deletion (`sgd`, `dgd`) by flux balance analysis (FBA) remain. More than the `'testType'` argument raises `parseRAVENargs:tooManyArgs` | -| `checkRxn(...)` | Removed, no direct successor. To find why a reaction cannot carry flux, use `gapfilling/findLeakMetabolite.m`, `gapfilling/canExchange.m` or `gapfilling/checkProduction.m` | -| `dispEM(...)`, `followFluxes(...)`, `followChanged(...)`, `printModel(...)`, `mapCompartments(...)`, `getExpressionStructure(...)`, `getMetsInComp(...)` | Removed (see [Removed functionality](#removed-functionality) for the replacements, where one exists) | -| Anything under `external/metacyc/` | Folder removed (MetaCyc-based reconstruction, 7 functions). No replacement in RAVEN 3 | -| Anything under `pathway/` or `plotting/` | Folders removed: 16 functions for visualisation and plotting, such as `drawMap`, `drawPathway`, `markPathwayWithFluxes` and `setOmicDataToRxns`. No replacement in RAVEN 3; the COBRA Toolbox and Escher provide plotting | -| Anything under `legacy/` | Folder removed. No replacement in RAVEN 3 (see [Removed functionality](#removed-functionality)) | +| `findGeneDeletions(...)` with `analysisType`, `refModel` or `oeFactor` | The arguments are removed together with `qMOMA` (MOMA-based over-expression analysis), and so are the test types `'sgo'` and `'dgo'`. Single and double gene deletion (`sgd`, `dgd`) by flux balance analysis (FBA) remain. Any argument besides `'testType'` raises `parseRAVENargs:tooManyArgs` | +| `checkRxn(...)` | Removed without a direct successor. To find why a reaction cannot carry flux, use `gapfilling/findLeakMetabolite.m`, `gapfilling/canExchange.m` or `gapfilling/checkProduction.m` | +| `dispEM(...)`, `followFluxes(...)`, `followChanged(...)`, `printModel(...)`, `mapCompartments(...)`, `getExpressionStructure(...)`, `getMetsInComp(...)` | Removed. [Removed functionality](#removed-functionality) gives the replacements where one exists | +| Any function in `external/metacyc/` | Folder removed, with the 7 functions of the MetaCyc-based reconstruction. RAVEN 3 has no replacement | +| Any function in `pathway/` or `plotting/` | Folders removed, with 16 functions for visualisation and plotting, such as `drawMap`, `drawPathway`, `markPathwayWithFluxes` and `setOmicDataToRxns`. RAVEN 3 has no replacement; the COBRA Toolbox and Escher provide plotting | +| Any function in `legacy/` | Folder removed. RAVEN 3 has no replacement (see [Removed functionality](#removed-functionality)) | | `getGenesFromKEGG`, `getMetsFromKEGG`, `getRxnsFromKEGG`, `constructMultiFasta`, `getWSLpath`, `getBlastFromExcel` | Removed (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)) | -| `getToolboxVersion(...)` | Removed as a standalone function. It is a local function of `exportForGit` | -| `parallelPoolRAVEN(...)` | Renamed `parallelWorkersRAVEN`, with a different signature and output (see [Renamed and merged functions](#renamed-merged-and-consolidated-functions)) | +| `getToolboxVersion(...)` | Removed as a function of its own. It is a local function of `exportForGit` | +| `parallelPoolRAVEN(...)` | Renamed `parallelWorkersRAVEN`, with another signature and output (see [Renamed and merged functions](#renamed-merged-and-consolidated-functions)) | | `getModelFromKEGG(keggPath, keepSpontaneous, keepUndefinedStoich, keepIncomplete, keepGeneral)` | Raises `parseRAVENargs:tooManyArgs`, because `keggPath` is removed and the function has four optional parameters | -| `getKEGGModelForOrganism(...)` with 13 to 15 positional arguments | The parameters `nSequences` and `seqIdentity` are removed. 14 or 15 arguments raise `parseRAVENargs:tooManyArgs`. A non-empty 13th argument (`nSequences` in RAVEN 2) is read as `globalModel` | +| `getKEGGModelForOrganism(...)` with 13 to 15 positional arguments | The parameters `nSequences` and `seqIdentity` are removed. A call with 14 or 15 arguments raises `parseRAVENargs:tooManyArgs`. A non-empty 13th argument, which is `nSequences` in RAVEN 2, is read as `globalModel` | | `getPhylDist(keggPath, onlyInKingdom)` | Raises `parseRAVENargs:tooManyArgs`, because `keggPath` is removed | -| `writeYAMLmodel(model, fileName, preserveQuotes, sortIds)` with four arguments | Raises `parseRAVENargs:tooManyArgs`. RAVEN 3 has `(model, fileName, sortIds)` | -| `scoreModel(...)` with `'best'` at position 7, 8 or 9 among all arguments | Raises `RAVEN:badInput`. Positions 7, 8 and 9 are `isozymeScoring`, `complexScoring` and `multipleCellScoring`. The first two accept `'min'`, `'max'`, `'median'` and `'average'`, the third accepts `'max'` and `'average'` | +| `writeYAMLmodel(model, fileName, preserveQuotes, sortIds)` with four arguments | Raises `parseRAVENargs:tooManyArgs`. The RAVEN 3 signature is `(model, fileName, sortIds)` | +| `scoreModel(...)` with `'best'` at position 7, 8 or 9 among all arguments | Raises `RAVEN:badInput`. Positions 7, 8 and 9 are `isozymeScoring`, `complexScoring` and `multipleCellScoring`. The first two accept `'min'`, `'max'`, `'median'` and `'average'`, and the third accepts `'max'` and `'average'` | | `scoreModel(...)` with `hpaLevelScores` at position 9 among all arguments | Raises an error. Position 9 is `multipleCellScoring`, and `hpaLevelScores` is position 10 | ## Changes that alter results without an error -The same call with the same input model gives a different output. The changes fall into two groups: argument positions and defaults that change meaning, and corrected behaviour. Argument positions count all arguments of the call, including the required ones: in `scoreModel(model, hpaData, arrayData, tissue, celltype, ...)` the model is position 1 and `celltype` is position 5. +The same call on the same model gives another output. Positions count all arguments of the call, required ones included: in `scoreModel(model, hpaData, arrayData, tissue, celltype, ...)` the model is position 1 and `celltype` is position 5. ### Argument positions and defaults that change meaning | Function | RAVEN 3 behaviour | |---|---| -| `getModelFromKEGG(keggPath)` to `getModelFromKEGG(keggPath, keepSpontaneous, keepUndefinedStoich[, keepIncomplete])` | Calls with one to four positional arguments shift. The first argument (`keggPath` in RAVEN 2) is read as `keepSpontaneous` and has no effect. Each later argument is read as the next parameter in the RAVEN 2 order, so a different class of reactions is removed | -| `getPhylDist(keggPath)` | `keggPath` is read as `'onlyInKingdom'` and has no effect. The full distance matrix is returned | -| `writeYAMLmodel(model, fileName, true)` | The third argument is read as `sortIds`, so IDs are sorted. RAVEN 2 read it as `'preserveQuotes'` | -| `scoreModel` | `isozymeScoring` and `complexScoring` replace the single `multipleGeneScoring` of RAVEN 2. Reactions with an `and` rule get the minimum over the genes by default (`complexScoring`). RAVEN 2 scored every reaction with the maximum over all genes by default (`multipleGeneScoring` `'best'`) | -| `scoreModel(..., celltype, noGeneScore, 'average')` | `'average'` at position 7 among all arguments (`multipleGeneScoring` in RAVEN 2) is read as `isozymeScoring`, so it does not apply to `and` rules, which use `complexScoring` | -| `scoreModel(..., celltype, noGeneScore, multipleGeneScoring, 'average')` | `'average'` at position 8 among all arguments (`multipleCellScoring` in RAVEN 2) is read as `complexScoring` without an error, so the average is taken over the genes of an `and` rule and no longer over cell types | -| `scoreModel` output `geneScores` | Genes that are absent from the data receive `NaN`. The RAVEN 2 documentation states `-Inf`. | +| `getModelFromKEGG(keggPath)` to `getModelFromKEGG(keggPath, keepSpontaneous, keepUndefinedStoich[, keepIncomplete])` | A call with one to four positional arguments shifts. The first argument, `keggPath` in RAVEN 2, is read as `keepSpontaneous` and has no effect. Each later argument is read as the next parameter in the RAVEN 2 order, so another class of reactions is removed | +| `getPhylDist(keggPath)` | `keggPath` is read as `'onlyInKingdom'` and has no effect. The function returns the full distance matrix | +| `writeYAMLmodel(model, fileName, true)` | The third argument is read as `sortIds`, so the IDs are sorted. RAVEN 2 read it as `'preserveQuotes'` | +| `scoreModel` | `isozymeScoring` and `complexScoring` replace the single `multipleGeneScoring` of RAVEN 2. By default (`complexScoring`) a reaction with an `and` rule gets the minimum over its genes. RAVEN 2 scored every reaction with the maximum over all its genes by default (`multipleGeneScoring` `'best'`) | +| `scoreModel(..., celltype, noGeneScore, 'average')` | `'average'` at position 7 among all arguments, which is `multipleGeneScoring` in RAVEN 2, is read as `isozymeScoring`. It does not apply to `and` rules, which use `complexScoring` | +| `scoreModel(..., celltype, noGeneScore, multipleGeneScoring, 'average')` | `'average'` at position 8 among all arguments, which is `multipleCellScoring` in RAVEN 2, is read as `complexScoring` without an error. The average is then taken over the genes of an `and` rule and no longer over cell types | +| `scoreModel`, output `geneScores` | A gene that is absent from the data receives `NaN`. The RAVEN 2 documentation states `-Inf` | | `scoreModel`, value names | RAVEN 2 accepted `'best'` for `multipleGeneScoring` and `multipleCellScoring`. RAVEN 3 uses `'max'` for `isozymeScoring`, `complexScoring` and `multipleCellScoring`. The parameter order is `celltype`, `noGeneScore`, `isozymeScoring`, `complexScoring`, `multipleCellScoring`, `hpaLevelScores` and `dataPrecedence` (default `'gene'`) | | `parseScores(file)` | The default predictor is `'deeploc'`. RAVEN 2 used `'wolf'`. How a WoLF PSORT file is handled without a predictor was not run | +| `randomSampling(model, n)` | The default `method` is `'achr'`. RAVEN 2 ran the random-objective method, which `'randomObjective'` selects. The output `goodRxns` is empty for `'achr'` and `'chrr'`. The function has the parameters `nObjectives`, `seed`, `method`, `thinning` and `nBurnin` and a third output, `info` | | `getModelFromHomology`, `strictness=3` | The best hit is the hit with the highest bitscore. RAVEN 2 selected the hit with the lowest E-value. No parameter selects the criterion (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)) | -| `getModelFromHomology`, `minLen` | The default is `100`, in RAVEN 2 `200` | -| `getKEGGModelForOrganism`, `cutOff` | The default is `10^-30`, in RAVEN 2 `10^-50` | -| `getKEGGModelForOrganism`, `minScoreRatioG` | The default is `0.9`, in RAVEN 2 `0.8` | +| `getModelFromHomology`, `minLen` | The default is `100`. In RAVEN 2 it was `200` | +| `getKEGGModelForOrganism`, `cutOff` | The default is `10^-30`. In RAVEN 2 it was `10^-50` | +| `getKEGGModelForOrganism`, `minScoreRatioG` | The default is `0.9`. In RAVEN 2 it was `0.8` | ### Corrected behaviour | Function | RAVEN 3 behaviour | |---|---| | `getModelFromHomology`, merging of gene-reaction rules (GPR) | Gene substitution uses exact tokens. RAVEN 2 used regular expressions or `strrep` | -| `removeGenes`, `addRxns` | `removeGenes` evaluates the rule as a tree, so isozymes and complexes are handled. `addRxns` splits a rule at space-bounded `and` and `or` tokens, so a gene ID such as `band1` is not cut, and builds no tree. RAVEN 2 used substring matching. `expandModel` parses the rule with `grRuleToDNF`, leaves a rule that cannot be parsed (`RAVEN:badGrRule`) unchanged and lists the reaction in `rxnToCheck`, and raises `RAVEN:grRuleTooComplex` for a rule that expands to more than 10000 isozymes; RAVEN 2 split the rule string at `' or '` and `' and '` (see [Parsing of gene-reaction rules](#gpr-parsing)) | +| `removeGenes`, `addRxns` | `removeGenes` evaluates the rule as a tree, so isozymes and complexes are handled. `addRxns` splits a rule at the space-bounded tokens `and` and `or`, so a gene ID such as `band1` stays whole, and it builds no tree. RAVEN 2 matched substrings | +| `expandModel` | The rule is parsed with `grRuleToDNF`. A rule that cannot be parsed (`RAVEN:badGrRule`) stays unchanged and its reaction is listed in `rxnToCheck`. A rule that expands to more than 10000 isozymes raises `RAVEN:grRuleTooComplex`. RAVEN 2 split the rule string at `' or '` and `' and '` (see [Parsing of gene-reaction rules](#gpr-parsing)) | | `findPotentialErrors` | The changelog of RAVEN 3 states that the check uses a parse tree. The effect on results was not compared (see [Parsing of gene-reaction rules](#gpr-parsing)) | | `changeGrRules(..., 'replace', false)` onto a reaction without genes | Produces a valid grRule. RAVEN 2 wrapped the new rule as `() or (X)`, which RAVEN 3 cannot parse | | `simplifyModel` | Keeps the boundary constraint (`b`) of a task on a metabolite without reactions. RAVEN 2.11.3 removed such metabolites, so the task check can give a different result in RAVEN 3 (not run; see [Gap-filling and context-specific model extraction](#ftinit)) | -| `simplifyModel` with `constrainReversible` on a model without a reversible reaction | The step changes nothing. RAVEN 2 raised an error, because `getAllowedBounds` takes an empty reaction list as all reactions (error taken from the changelog; not reproduced) | +| `simplifyModel` with `constrainReversible` on a model without a reversible reaction | The step changes nothing. RAVEN 2 raised an error, because `getAllowedBounds` takes an empty reaction list as all reactions (the error is taken from the changelog and was not reproduced) | | `ftINITInternalAlg` | With `allowExcretion` true, the metabolite balance is constrained to `S*v >= 0`. RAVEN 2 had the sense of the constraint inverted | | `ravenCobraWrapper` | The conversion of `grRules` to the COBRA `rules` field handles gene IDs that contain regular-expression metacharacters. RAVEN 2 did not escape them (see [RAVEN⇄COBRA interoperability](#raven-cobra-interoperability)) | -| `mergeCompartments` | The merged metabolite is the copy in compartment `s`, otherwise the copy whose ID sorts first (character code order), with every field of that copy. RAVEN 2 kept the copy that came first in `model.mets` | -| `permuteModel`, `optimizeProb`, `getFluxZ`, `convertToIrrev`, `replaceMets`, `setExchangeBounds`, `fitParameters`, `guessComposition` | The changelog of RAVEN 3 lists a bug fix for each function. The effect on results was not checked. The default `method` of `randomSampling` is `'achr'`, so `randomSampling(model, n)` no longer runs the random-objective method of RAVEN 2 (`'randomObjective'` selects it); `goodRxns` is empty for `'achr'` and `'chrr'`; the function has the parameters `nObjectives`, `seed`, `method`, `thinning` and `nBurnin` and a third output `info`. The changelog lists further fixes that can change results, for example for `mergeLinear`, `mergeModels`, `closeModel`, `addTransport`, `getExchangeRxns`, `removeLowScoreGenes`, `getMinNrFluxes`, `reporterMetabolites`, `getElementalBalance`, `checkModelStruct` and `deleteUnusedGenes`; these were not checked | +| `mergeCompartments` | The merged metabolite is the copy in compartment `s` or, failing that, the copy whose ID sorts first in character code order, with every field of that copy. RAVEN 2 kept the copy that came first in `model.mets` | + +The changelog of RAVEN 3 lists a bug fix for each of `permuteModel`, `optimizeProb`, `getFluxZ`, `convertToIrrev`, `replaceMets`, `setExchangeBounds`, `fitParameters` and `guessComposition`. It lists further fixes that can change results, for example for `mergeLinear`, `mergeModels`, `closeModel`, `addTransport`, `getExchangeRxns`, `removeLowScoreGenes`, `getMinNrFluxes`, `reporterMetabolites`, `getElementalBalance`, `checkModelStruct` and `deleteUnusedGenes`. The effect of these fixes on results was not checked. ## Changes to error and warning identifiers -- RAVEN 2 raised errors through `dispEM` with an empty identifier (`MException('', ...)`), so no RAVEN 2 script could catch on `err.identifier`. RAVEN 3 replaces `dispEM` with `error()` and `warning()` calls. `RAVEN:badInput` is used in 220 lines and `RAVEN:warning` in 63 lines of `src/` and the root `.m` files (non-comment lines that contain the identifier). Further identifiers are `RAVEN:modelError`, `RAVEN:badGrRule`, `RAVEN:infeasibleModel`, `RAVEN:infeasible`, `RAVEN:sampling`, `RAVEN:grRuleTooComplex`, `RAVEN:deprecated`, `RAVEN:legacyMethod`, `RAVEN:notMainBranch`, `RAVEN:glpkMILP`, `RAVEN:fetchToolboxAsset:unverified`, `RAVEN:fetchToolboxAsset:checksum`, `RAVEN:fetchToolboxAsset:manifest`, `RAVEN:runSuppOrth:supportLv`, `RAVEN:runSuppOrth:failed`, `RAVEN:runSuppOrth:notFound`, `RAVEN:hitsFromSuppOrth:unmappedProteins`, `parseRAVENargs:tooManyArgs`, `parseRAVENargs:noSpec` and `getPhylDist:noData`. The list is not exhaustive: identifiers in other namespaces, such as `renameModelGenes:*`, `setGAM:*` and `parseYAML:*`, also exist. Not every call carries an identifier. Scripts can match on `err.identifier` for the calls that carry one. +RAVEN 2 raised its errors through `dispEM` with an empty identifier (`MException('', ...)`), so no RAVEN 2 script could catch an error by `err.identifier`. RAVEN 3 replaces `dispEM` with `error()` and `warning()` calls. Not every call carries an identifier, and a script can match on `err.identifier` for the calls that do. + +- `RAVEN:badInput` appears in 220 lines and `RAVEN:warning` in 63 lines of `src/` and the root `.m` files (non-comment lines that contain the identifier). +- Further identifiers in the `RAVEN:` namespace are `RAVEN:modelError`, `RAVEN:badGrRule`, `RAVEN:infeasibleModel`, `RAVEN:infeasible`, `RAVEN:sampling`, `RAVEN:grRuleTooComplex`, `RAVEN:deprecated`, `RAVEN:legacyMethod`, `RAVEN:notMainBranch`, `RAVEN:glpkMILP`, `RAVEN:fetchToolboxAsset:unverified`, `RAVEN:fetchToolboxAsset:checksum`, `RAVEN:fetchToolboxAsset:manifest`, `RAVEN:runSuppOrth:supportLv`, `RAVEN:runSuppOrth:failed`, `RAVEN:runSuppOrth:notFound` and `RAVEN:hitsFromSuppOrth:unmappedProteins`. +- Other namespaces include `parseRAVENargs:tooManyArgs`, `parseRAVENargs:noSpec`, `getPhylDist:noData`, `renameModelGenes:*`, `setGAM:*` and `parseYAML:*`. The list is not exhaustive. - Whether `try/catch` blocks that match on message text still match was not compared message by message. ## Changes that raise warnings -- `getINITModel` and `runINIT` issue a one-time notice with the identifier `RAVEN:legacyMethod` that points at `ftINIT`. `warning('off','RAVEN:legacyMethod')` silences it. The help text of both functions states that the function remains supported, although the changelog of RAVEN 3 marks both functions deprecated. See [Gap-filling and context-specific model extraction](#ftinit). -- `makeSomething`, `consumeSomething`, `canProduce` and `canConsume` are wrappers in the folder `deprecated/` around `findLeakMetabolite` and `canExchange`. They issue a one-time `RAVEN:deprecated` warning that states that they will be removed in the next major release. -- `checkInstallation` is `checkRaven` in RAVEN 3, moved from `installation/` to the repository root, with the same signature (`[currVer, installType] = checkRaven(developMode, checkBinaries)`). `checkInstallation` forwards every argument and output to `checkRaven` and issues a `RAVEN:deprecated` warning that names `checkRaven` on every call. The wrapper will be removed in the next major release. +- `getINITModel` and `runINIT` issue a one-time notice with the identifier `RAVEN:legacyMethod` that points to `ftINIT`. `warning('off','RAVEN:legacyMethod')` silences it. The help text of both functions states that the function remains supported, although the changelog of RAVEN 3 marks both as deprecated (see [Gap-filling and context-specific model extraction](#ftinit)). +- `makeSomething`, `consumeSomething`, `canProduce` and `canConsume` are wrappers in the folder `deprecated/` around `findLeakMetabolite` and `canExchange`. They issue a one-time `RAVEN:deprecated` warning that they will be removed in the next major release. +- `checkInstallation` is `checkRaven` in RAVEN 3, moved from `installation/` to the repository root, with the same signature: `[currVer, installType] = checkRaven(developMode, checkBinaries)`. The wrapper `checkInstallation` forwards every argument and output to `checkRaven` and issues a `RAVEN:deprecated` warning that names `checkRaven` on every call. The wrapper will be removed in the next major release. ## Installation changes -RAVEN 3 does not contain the KEGG HMM libraries and the external binaries (BLAST+, DIAMOND, HMMER, cd-hit, MAFFT, WoLF PSORT and SCIP) or Apache POI in the repository. It fetches the KEGG HMM libraries and reference tables and the BLAST+, DIAMOND and HMMER binaries from the [`toolbox-assets`](https://github.com/CompMetEng/toolbox-assets) repository the first time a function needs them, so internet access is required on first use. The binaries are stored in `src/external/`. SCIP for Windows is downloaded by `setRavenSolver` from the RAVEN v2.8.6 GitHub release. An offline bundle of the binaries is available as a `*-binaries` RAVEN release. What is fetched and where it is cached is described in [Download data and binaries](../installation/data-and-binaries.md). The air-gapped install and the dependency comparison are in [Toolbox dependencies and installation](#toolbox-deps). +The repository of RAVEN 3 does not contain the KEGG HMM libraries, the external binaries (BLAST+, DIAMOND, HMMER, cd-hit, MAFFT, WoLF PSORT and SCIP) or Apache POI. The first time a function needs them, RAVEN 3 fetches the KEGG HMM libraries and reference tables and the BLAST+, DIAMOND and HMMER binaries from the [`toolbox-assets`](https://github.com/CompMetEng/toolbox-assets) repository, so the first use requires internet access. The binaries are stored in `src/external/`. The function `setRavenSolver` downloads SCIP for Windows from the RAVEN v2.8.6 GitHub release. An offline bundle of the binaries is available as a `*-binaries` RAVEN release. [Download data and binaries](../installation/data-and-binaries.md) describes what is fetched and where it is cached, and [Toolbox dependencies and installation](#toolbox-deps) covers the air-gapped install and the dependency comparison. diff --git a/docs/raven3-migration/formats-and-reconstruction.md b/docs/raven3-migration/formats-and-reconstruction.md index 9af0ef4..c9865ef 100644 --- a/docs/raven3-migration/formats-and-reconstruction.md +++ b/docs/raven3-migration/formats-and-reconstruction.md @@ -1,6 +1,6 @@ # File formats, reconstruction and GPR parsing -Model file formats, draft reconstruction and the parsing of gene-reaction rules (`grRules`, abbreviated GPR) differ between RAVEN 2 and RAVEN 3. The sections describe the changes that affect scripts and the differences that appear in version-controlled model files. +Three areas differ between RAVEN 2 and RAVEN 3: the model file formats, the draft reconstruction from KEGG and from homology, and the parsing of gene-reaction rules (`grRules`, abbreviated GPR). Each section gives the changes that affect a script and the differences that show up as diffs in version-controlled model files. (file-formats-and-model-io)= ## File formats and model I/O @@ -11,17 +11,17 @@ Model file formats, draft reconstruction and the parsing of gene-reaction rules | Aspect | RAVEN 2 | RAVEN 3 | |---|---|---| | `exportModel` output format | SBML Level 3 Version 1, FBC package version 2 | Unchanged | -| `importModel` accepted input | Any SBML level and version; the notes-based gene association `GENE_ASSOCIATION` of Level 2 files is read, and `SHORT NAME` is read for gene names; FBC fields are optional | SBML Level 3 Version 1 with FBC package version 2; any other file raises the error `RAVEN:badInput` | +| `importModel` accepted input | Any SBML level and version. The notes-based gene association `GENE_ASSOCIATION` of Level 2 files is read, and `SHORT NAME` is read for gene names. FBC fields are optional | SBML Level 3 Version 1 with FBC package version 2. Any other file raises the error `RAVEN:badInput` | | MIRIAM prefixes read on import | `urn:miriam:`, `http://identifiers.org/` and `https://identifiers.org/` | `http://identifiers.org/` and `https://identifiers.org/` | -`importModel` of RAVEN 3 reads MIRIAM annotations (reaction EC codes, model and species annotations) with the `http://identifiers.org/` and `https://identifiers.org/` prefixes and ignores `urn:miriam:` annotations. +`importModel` of RAVEN 3 reads MIRIAM annotations of reaction EC codes, models and species with the prefixes `http://identifiers.org/` and `https://identifiers.org/`, and ignores `urn:miriam:` annotations. :::{admonition} Cross-references in older SBML files :class: warning -An SBML file that uses `urn:miriam:` annotations is imported without an error, and its cross-references (for example `metMiriams`) are absent from the imported model. RAVEN 2 writes `https://identifiers.org/` prefixes (`exportModel`), so its own files are not affected. Files from other tools that use `urn:miriam:` can be converted to the `identifiers.org` form before import. +An SBML file with `urn:miriam:` annotations is imported without an error, and its cross-references (for example `metMiriams`) are absent from the imported model. The `exportModel` function of RAVEN 2 writes `https://identifiers.org/` prefixes, so files written by RAVEN 2 are not affected. Files from other tools that use `urn:miriam:` can be converted to the `identifiers.org` form before the import. ::: -A file in an older SBML level or version has to be converted before `importModel` accepts it. The error message suggests another tool or an older RAVEN version for the conversion. +A file in an older SBML level or version has to be converted before `importModel` accepts it. The error message names another tool or an older RAVEN version for the conversion. (excel-io)= ### Excel I/O @@ -29,39 +29,39 @@ A file in an older SBML level or version has to be converted before `importModel | Aspect | RAVEN 2 | RAVEN 3 | |---|---|---| | Excel engine | Apache POI (Java), bundled with RAVEN | Dependency-free OOXML writer (`io/writeExcel.m`) | -| Full-model `.xlsx` import | `importExcelModel.m` (sheets `RXNS`, `METS`, `COMPS`, `GENES` and `MODEL`) builds a complete model | Removed | -| Tabular curation of an existing model | No equivalent | `curation/curateModelFromTables.m` curates or extends a loaded model from `.tsv` files (metabolites by name and compartment, reactions by stoichiometry, genes by name) | -| `exportToExcelFormat` with a path that has no file name or extension | Falls back to tab-delimited text export | Raises an error; the output is `.xlsx` | +| Import of a full model from `.xlsx` | `importExcelModel.m` builds a complete model from the sheets `RXNS`, `METS`, `COMPS`, `GENES` and `MODEL` | Removed | +| Tabular curation of an existing model | No equivalent | `curation/curateModelFromTables.m` curates or extends a loaded model from `.tsv` files: metabolites by name and compartment, reactions by stoichiometry, genes by name | +| `exportToExcelFormat` with a path that has no file name or extension | Falls back to tab-delimited text | Raises an error. The output is `.xlsx` | -A pipeline that depends on the RAVEN 2 Excel curation template can convert the template to the `.tsv` format that `curateModelFromTables` expects, or build the model with `importModel` or `readYAMLmodel` and use `curateModelFromTables` for incremental edits. +A pipeline that depends on the RAVEN 2 Excel curation template has two options. The template can be converted to the `.tsv` format that `curateModelFromTables` expects. Alternatively the model is built with `importModel` or `readYAMLmodel`, and `curateModelFromTables` makes the later edits. (yaml-io)= ### YAML I/O -`readYAMLmodel` and `writeYAMLmodel` are pure MATLAB in both versions, so model YAML I/O has no Python dependency in either. `io/parseYAML.m` is a generic pure-MATLAB YAML parser that `conditions/applyCondition.m` uses to read condition files. +`readYAMLmodel` and `writeYAMLmodel` are written in MATLAB in both versions, so the YAML I/O of a model has no Python dependency in either. The function `io/parseYAML.m` is a general YAML parser in MATLAB, which `conditions/applyCondition.m` uses to read condition files. -`writeYAMLmodel` of RAVEN 3 takes name-value arguments and quotes strings by a fixed rule in place of the RAVEN 2 option `preserveQuotes`. The output differs from RAVEN 2 in the following points, which appear as diffs in version-controlled model files. +`writeYAMLmodel` of RAVEN 3 takes name-value arguments and quotes strings by a fixed rule, in place of the RAVEN 2 option `preserveQuotes`. The output differs from the output of RAVEN 2 in the points below, and each of them appears as a diff in a version-controlled model file. | Aspect | RAVEN 2 | RAVEN 3 | |---|---|---| -| Quoting | Every string is quoted with double quotes (default `preserveQuotes` is true) | A string is quoted with double quotes where YAML needs it (reserved words, numbers, leading indicators, `: `); other strings are bare | -| Indentation | List entries ` - !!omap` (four spaces) with keys indented by six spaces | List entries ` - !!omap` (two spaces) with keys indented by four spaces | -| Reaction EC codes | Top-level `eccodes` field | Under `annotation: ec-code`; the top-level key is not written | +| Quoting | Every string has double quotes (default `preserveQuotes` true) | A string has double quotes where YAML needs them (reserved words, numbers, leading indicators, `: `). Other strings are bare | +| Indentation | List entries ` - !!omap` (four spaces), keys indented by six spaces | List entries ` - !!omap` (two spaces), keys indented by four spaces | +| Reaction EC codes | Top-level field `eccodes` | Under `annotation: ec-code`. The top-level key is not written | | Metabolite SMILES | Top-level field | Under `annotation` as `smiles` | | Reaction notes key | `rxnNotes` | `notes` | -| Gene-less model | `genes:` is omitted when `model.genes` is absent; an empty field gives a bare `- genes:` | The line `- genes: []` | -| Document start | `---` line, then `!!omap` | `!!omap` only | +| Gene-less model | `genes:` is omitted when `model.genes` is absent, and an empty field gives a bare `- genes:` | The line `- genes: []` | +| Document start | A `---` line, then `!!omap` | `!!omap` only | | Field order, metabolites | `formula`, then `charge` | `charge`, then `formula` | -| Field order, reactions | `rxnFrom` directly after `gene_reaction_rule`; `references` before `subsystem` and `annotation` | `subsystem` and `annotation` first; `references` and `rxnFrom` follow `annotation` among the RAVEN-specific extras | +| Field order, reactions | `rxnFrom` directly after `gene_reaction_rule`, and `references` before `subsystem` and `annotation` | `subsystem` and `annotation` first, then `references` and `rxnFrom` among the RAVEN-specific extras | | Field order, genes | `protein` before `annotation` | `protein` after `annotation` | -| `objective_coefficient` on import | Sets the coefficient to 1 for any value | The value is read | +| `objective_coefficient` on import | Sets the coefficient to 1 for any value | Reads the value | -`readYAMLmodel` of RAVEN 3 accepts both the top-level `eccodes`, `smiles` and `rxnNotes` keys and the `annotation` and `notes` forms. +`readYAMLmodel` of RAVEN 3 accepts the top-level keys `eccodes`, `smiles` and `rxnNotes` as well as the forms `annotation` and `notes`. (raven-cobra-interoperability)= ### RAVEN⇄COBRA interoperability -`conversion/ravenCobraWrapper.m` keeps its signature (`newModel = ravenCobraWrapper(model)`). RAVEN 3 fixes two bugs that change the output: a stale variable that could reuse the wrong MIRIAM data during gene-annotation conversion, and a mismatch of gene IDs that contain regular-expression metacharacters (for example `ENSG00000141510.16`) when the `rules` field is built from `grRules`. +The function `conversion/ravenCobraWrapper.m` keeps its signature, `newModel = ravenCobraWrapper(model)`. RAVEN 3 fixes two bugs that change its output. One is a stale variable that could reuse the wrong MIRIAM data in the conversion of gene annotations. The other is a mismatch of gene IDs that contain regular-expression metacharacters, for example `ENSG00000141510.16`, when the `rules` field is built from `grRules`. --- @@ -72,62 +72,62 @@ A pipeline that depends on the RAVEN 2 Excel curation template can convert the t | Aspect | RAVEN 2 | RAVEN 3 | |---|---|---| -| Reaction and gene database source | Local KEGG FTP dump (licence required), read when the cached files `keggGenes.mat`, `keggMets.mat` and `keggRxns.mat` are absent | Prebuilt artefacts fetched on demand from releases of `toolbox-assets` | +| Source of the reaction and gene database | Local KEGG FTP dump (licence required), read when the cached files `keggGenes.mat`, `keggMets.mat` and `keggRxns.mat` are absent | Prebuilt artefacts fetched on demand from releases of `toolbox-assets` | | Functions that build the database | `getGenesFromKEGG`, `getMetsFromKEGG` and `getRxnsFromKEGG` | `buildGlobalKEGGModel`, `buildGlobalGPR` and `readKEGGTable` | -| `getModelFromKEGG` `keggPath` argument | Optional (default `external/kegg`) | Removed | -| Ortholog HMM library | One `.hmm` file per KEGG Orthology (KO) entry; one `hmmsearch` call per KO, through the Windows Subsystem for Linux (WSL) on Windows | One concatenated library per domain (`kegg118_eukaryotes.hmm` and `kegg118_prokaryotes.hmm`); one `hmmsearch` call against the whole proteome; no WSL | -| `dataDir` | Arbitrary local folder | For the homology method (a FASTA file is given), the folder name must end in `kegg118_eukaryotes` or `kegg118_prokaryotes`; the annotation method ignores `dataDir` | -| `nSequences`, `seqIdentity` (CD-HIT clustering controls) | Configurable | Removed; there is no local clustering step | -| `cutOff` default | `10^-50` | `10^-30` | -| `minScoreRatioG` default | `0.8` | `0.9` | -| Calling convention | Positional | Positional or name-value (`parseRAVENargs`); a call with more positional arguments than the function defines raises `parseRAVENargs:tooManyArgs` | +| Argument `keggPath` of `getModelFromKEGG` | Optional, default `external/kegg` | Removed | +| Ortholog HMM library | One `.hmm` file per KEGG Orthology (KO) entry and one `hmmsearch` call per KO, through the Windows Subsystem for Linux (WSL) on Windows | One concatenated library per domain (`kegg118_eukaryotes.hmm` and `kegg118_prokaryotes.hmm`), one `hmmsearch` call against the whole proteome, and no WSL | +| `dataDir` | Any local folder | For the homology method, which takes a FASTA file, the folder name must end in `kegg118_eukaryotes` or `kegg118_prokaryotes`. The annotation method ignores `dataDir` | +| `nSequences`, `seqIdentity` (CD-HIT clustering controls) | Configurable | Removed, because there is no local clustering step | +| Default of `cutOff` | `10^-50` | `10^-30` | +| Default of `minScoreRatioG` | `0.8` | `0.9` | +| Calling convention | Positional | Positional or name-value (`parseRAVENargs`). A call with more positional arguments than the function defines raises `parseRAVENargs:tooManyArgs` | -The first call of `getModelFromKEGG` downloads the data, builds the global KEGG model and saves it as `src/reconstruction/kegg/keggModel.mat` (the source header states that the first build needs a few hundred MB of disk; the size of the cached file was not checked). `getKEGGModelForOrganism` loads the global model, the KO model and the five flag vectors from the same file. External binaries and the KEGG data are fetched from pinned `toolbox-assets` tags and are not bundled; see [Toolbox dependencies and installation](#toolbox-deps). +The first call of `getModelFromKEGG` downloads the data, builds the global KEGG model and saves it as `src/reconstruction/kegg/keggModel.mat`. The source header states that the first build needs a few hundred MB of disk, and the size of the cached file was not checked. The function `getKEGGModelForOrganism` loads the global model, the KO model and the five flag vectors from the same file. The external binaries and the KEGG data come from pinned `toolbox-assets` tags and are not bundled (see [Toolbox dependencies and installation](#toolbox-deps)). -The three functions take their optional arguments through `varargin`, in the order of RAVEN 2 minus the removed arguments. Positional calls written for RAVEN 2 (checked against v2.11.3) change as follows. +The three functions take their optional arguments through `varargin`, in the RAVEN 2 order without the removed arguments. A position counts all arguments of the call, required ones included. Positional calls written for RAVEN 2 (checked against 2.11.3) change as follows. -- `getModelFromKEGG(keggPath, keepSpontaneous, ...)` becomes `getModelFromKEGG(keepSpontaneous, ...)`. An unchanged RAVEN 2 call shifts every later argument: the argument at position 2 is read as the parameter that RAVEN 2 takes at position 3, and so on. -- `getKEGGModelForOrganism` has no `nSequences` and `seqIdentity`, so `globalModel` moves from the 15th to the 13th argument among all arguments. In the migration pages, a position counts all arguments of the call, including the required ones. +- `getModelFromKEGG(keggPath, keepSpontaneous, ...)` becomes `getModelFromKEGG(keepSpontaneous, ...)`. An unchanged RAVEN 2 call shifts every later argument: the argument at position 2 is read as the parameter that RAVEN 2 has at position 3, and so on. +- `getKEGGModelForOrganism` has no `nSequences` and no `seqIdentity`, so `globalModel` moves from the 15th to the 13th argument. - `getPhylDist(keggPath, onlyInKingdom)` becomes `getPhylDist(onlyInKingdom)`. ### Homology-based reconstruction (`getModelFromHomology`) | Aspect | RAVEN 2 | RAVEN 3 | |---|---|---| -| `strictness` parameter | `1`, `2` or `3` | Unchanged | -| Best-hit tie-break at `strictness=3` | E-value | Bitscore; no option restores the E-value criterion | -| `minLen` default | `200` | `100` | -| GPR gene substitution when merging homology hits | Regular-expression or `strrep` substitution, which corrupts IDs that are substrings of other IDs or contain regular-expression metacharacters | Exact-token substitution | -| Complex handling, gene-free reactions and notes | Reactions with incompletely mapped complexes are flagged with an `OLD_` prefix | Options `complexPolicy` (default `'flag'`, the RAVEN 2 behaviour), `keepGeneFree` and `preserveNotes` | +| Parameter `strictness` | `1`, `2` or `3` | Unchanged | +| Best-hit criterion at `strictness=3` | E-value | Bitscore. No option restores the E-value criterion | +| Default of `minLen` | `200` | `100` | +| Gene substitution in GPRs when homology hits are merged | Substitution by regular expression or `strrep`, which corrupts IDs that are substrings of other IDs or contain regular-expression metacharacters | Substitution by exact token | +| Complexes, gene-free reactions and notes | A reaction with an incompletely mapped complex is flagged with the prefix `OLD_` | The options `complexPolicy` (default `'flag'`, the RAVEN 2 behaviour), `keepGeneFree` and `preserveNotes` | | Gene IDs of the new organism | No restriction | Must not contain whitespace, parentheses, `\|` or `&` | -`reconstruction/homology/runSuppOrth.m` and `reconstruction/homology/hitsFromSuppOrth.m` are reconstruction inputs without a RAVEN 2 counterpart. The homology method takes its hits from BLAST, DIAMOND or SuppOrth. +The functions `reconstruction/homology/runSuppOrth.m` and `reconstruction/homology/hitsFromSuppOrth.m` supply reconstruction input and have no RAVEN 2 counterpart. The homology method takes its hits from BLAST, DIAMOND or SuppOrth. ### Genome data -`curation/downloadGenomeData.m`, `curation/getGeneData.m` and `curation/processProteinFastaFile.m` turn an NCBI genome accession (`GCF_` or `GCA_`) into a model-ready proteome and a gene ID mapping table. RAVEN 2 has no counterpart. +The functions `curation/downloadGenomeData.m`, `curation/getGeneData.m` and `curation/processProteinFastaFile.m` turn an NCBI genome accession (`GCF_` or `GCA_`) into a proteome that is ready for modelling and a gene ID mapping table. RAVEN 2 has no counterpart. -The content of the KEGG artefacts (a gene-free reference model plus KO, reaction and organism tables) was not checked against the `toolbox-assets` release. +The content of the KEGG artefacts, a gene-free reference model plus KO, reaction and organism tables, was not checked against the `toolbox-assets` release. --- (gpr-parsing)= ## Parsing of gene-reaction rules -RAVEN 3 parses gene-reaction rules into a tree, and the five functions in the table use the parser (`removeGenes`, `expandModel`, `findPotentialErrors`, `diffModels` and `curateModelFromTables`). The parser is not applied to a loaded model: a rule keeps its original text unless one of these functions changes it. +RAVEN 3 parses gene-reaction rules into a tree. Five functions use the parser: `removeGenes`, `expandModel`, `findPotentialErrors`, `diffModels` and `curateModelFromTables`. The parser is not applied to a loaded model, so a rule keeps its text unless one of these functions changes it. | Function | RAVEN 2 parsing | RAVEN 3 parsing | |---|---|---| -| `removeGenes` | Substring match on gene ID (`"10"` also matches `"100"`) | Exact gene-token match | -| `expandModel` | Split on `' or '` after removing parentheses; an `AND`-ed subunit in a nested rule can be dropped | Parse-tree-based isozyme expansion that preserves nested `AND` and `OR` | -| `findPotentialErrors` | Local function of `core/standardizeGrRules.m`; string-pattern check for `") and ("` and similar patterns (false positives on valid rules); returns indices | Function of its own; parse-tree-based detection of rules that are not in disjunctive normal form (DNF); `issues = findPotentialErrors(model)` returns a structure array with the fields `index`, `rxn`, `grRule` and `reason` | -| `diffModels` | Not present | Compares grRules; two rules are equal when their DNF forms are equal after sorting of genes and clauses (no absorption: `A or (A and B)` differs from `A`) | -| `curateModelFromTables` | Not present | Parses rules of the table input | +| `removeGenes` | Substring match on the gene ID (`"10"` also matches `"100"`) | Match on the exact gene token | +| `expandModel` | Split on `' or '` after the parentheses are removed. An `AND`-ed subunit in a nested rule can be dropped | Isozyme expansion from the parse tree, which keeps nested `AND` and `OR` | +| `findPotentialErrors` | Local function of `core/standardizeGrRules.m`. A string-pattern check for `") and ("` and similar patterns, with false positives on valid rules. Returns indices | A function of its own. Parse-tree detection of rules that are not in disjunctive normal form (DNF). `issues = findPotentialErrors(model)` returns a structure array with the fields `index`, `rxn`, `grRule` and `reason` | +| `diffModels` | Not present | Compares grRules. Two rules are equal when their DNF forms are equal after sorting of genes and clauses, without absorption: `A or (A and B)` differs from `A` | +| `curateModelFromTables` | Not present | Parses the rules of the table input | -`addRxns` does not use the parser. It takes the new gene IDs of a grRule from `getGenesFromGrRules`, which splits on space-bounded `and`, `or`, `&` and `|`. RAVEN 2 split on the substrings "and" and "or" anywhere, so an ID such as `band1` was split at "and". +The function `addRxns` does not use the parser. It takes the new gene IDs of a grRule from `getGenesFromGrRules`, which splits at the space-bounded tokens `and`, `or`, `&` and `|`. RAVEN 2 split at the substrings "and" and "or" anywhere, so an ID such as `band1` was cut at "and". -The parser functions are in `utils/`. `parseGrRule` builds a tree with `type` (`'gene'`, `'and'` or `'or'`) and `children`. `grRuleToDNF` converts the tree to DNF, an OR of AND-clauses. `isDnfGrRule` checks whether a rule is in that form, and `grRuleToString` renders a tree as a canonical string. +The parser functions are in `utils/`. The function `parseGrRule` builds a tree with `type` (`'gene'`, `'and'` or `'or'`) and `children`. The function `grRuleToDNF` converts the tree to DNF, an OR of AND clauses. The function `isDnfGrRule` checks whether a rule has that form, and `grRuleToString` renders a tree as a canonical string. -`parseGrRule` accepts `and` and `or` in any case and the symbols `&`, `&&`, `|` and `||`. Operators are recognised as whole words, so `G1&G2` is one gene ID. A malformed rule raises `RAVEN:badGrRule`. +`parseGrRule` accepts `and` and `or` in any case and the symbols `&`, `&&`, `|` and `||`. It recognises operators as whole words, so `G1&G2` is one gene ID. A malformed rule raises `RAVEN:badGrRule`. -`manipulation/standardizeGrRules.m` (called by `importModel`) builds no parse tree. It converts ` AND ` and ` OR ` to lowercase and splits the rule on the space-bounded strings ` and ` and ` or `. It calls `findPotentialErrors` and returns the indices of rules that cannot be parsed or are not in DNF (`indexes2check`). A direct call lists these rules in one warning; `importModel` passes `embedded` as `true`, which suppresses the warning. The function does not raise `RAVEN:badGrRule`. +The function `manipulation/standardizeGrRules.m`, which `importModel` calls, builds no parse tree. It converts ` AND ` and ` OR ` to lowercase and splits the rule at the space-bounded strings ` and ` and ` or `. It calls `findPotentialErrors` and returns the indices of rules that cannot be parsed or are not in DNF (`indexes2check`). A direct call lists these rules in one warning, and `importModel` passes `embedded` as `true`, which suppresses the warning. The function does not raise `RAVEN:badGrRule`. diff --git a/docs/raven3-migration/new-and-environment.md b/docs/raven3-migration/new-and-environment.md index 11aded1..fad25c3 100644 --- a/docs/raven3-migration/new-and-environment.md +++ b/docs/raven3-migration/new-and-environment.md @@ -1,42 +1,53 @@ # New functionality, dependencies and error handling -Gap-filling, context-specific model extraction with `ftINIT`, added functions, MATLAB toolbox dependencies and error handling differ between RAVEN 2 and RAVEN 3. +RAVEN 3 changes gap-filling and the extraction of context-specific models with `ftINIT`, adds functions, depends on fewer MATLAB toolboxes, and handles errors in another way. (ftinit)= ## Gap-filling and context-specific model extraction | Aspect | RAVEN 2 | RAVEN 3 | |---|---|---| -| `fillGaps` algorithm | One built-in mixed-integer linear programming (MILP) connectivity-maximisation strategy | The same strategy is the default (`algorithm` `'reference'`); the `algorithm` option adds `'fastLP'`, `'swiftLP'`, `'gapfillMILP'` and `'topological'` | -| `fillGaps` positional arguments | `allowNetProduction, useModelConstraints, supressWarnings, rxnScores` | The first four optional arguments are unchanged; `params` is the 7th and `algorithm` the 8th argument among all arguments, followed by `epsilon`, `minGrowth` and `verbose`; all can also be given as name-value pairs | -| `ftINIT` tie-breaking | Depends on the solver seed (the retries of the MILP set the seed to 1234); the dependence was not measured | The option `resolveTies` adds a best-effort selection among tied solutions (see below); the resulting dependence on the seed was not measured | -| `ftINIT` MILP optimality proof | Retries with an increasing relative gap, which is derived from an absolute-gap setting and the last objective value | The option `proveAbsGap` proves each stage to a fixed absolute gap in one solve (`MIPGap` 0) | -| `ftINIT` positional arguments | `transcrData` to `verbose` (5th to 11th argument among all arguments) | Unchanged; `resolveTies` and `proveAbsGap` follow `verbose` as the 12th and 13th argument among all arguments and are off by default | -| `fitTasks` gap-fill mode | One behaviour (equivalent to `'merge'`) | `gapFillMode`: `'merge'` (default, as in RAVEN 2) or `'preMerged'` | +| `fillGaps` algorithm | One built-in mixed-integer linear programming (MILP) strategy that maximises connectivity | The same strategy is the default (`algorithm` `'reference'`). The option `algorithm` adds `'fastLP'`, `'swiftLP'`, `'gapfillMILP'` and `'topological'` | +| `fillGaps` positional arguments | `allowNetProduction, useModelConstraints, supressWarnings, rxnScores` | The first four optional arguments are unchanged. `params` is the 7th and `algorithm` the 8th argument, followed by `epsilon`, `minGrowth` and `verbose`. All can also be given as name-value pairs | +| `ftINIT` tie-breaking | Depends on the solver seed, and the retries of the MILP set the seed to 1234. The dependence was not measured | The option `resolveTies` adds a best-effort selection among tied solutions. The resulting dependence on the seed was not measured | +| `ftINIT` proof of optimality | Retries with an increasing relative gap, which is derived from an absolute-gap setting and the last objective value | The option `proveAbsGap` proves each stage to a fixed absolute gap in one solve (`MIPGap` 0) | +| `ftINIT` positional arguments | `transcrData` to `verbose` are the 5th to 11th arguments | Unchanged. `resolveTies` and `proveAbsGap` follow `verbose` as the 12th and 13th arguments and are off by default | +| `fitTasks` gap-fill mode | One behaviour, equivalent to `'merge'` | `gapFillMode` is `'merge'` (default, as in RAVEN 2) or `'preMerged'` | | `fitTasks` outputs | `[outModel, addedRxns]` | `[outModel, addedRxns, failedTasks]` | -Argument positions in the migration pages count all arguments of the call, including the required ones. `getINITModel` and `runINIT` are described in [Renamed and merged functions](#renamed-merged-and-consolidated-functions). +Positions in the migration pages count all arguments of the call, required ones included. The functions `getINITModel` and `runINIT` are described in [Renamed and merged functions](#renamed-merged-and-consolidated-functions). -With `gapFillMode` set to `'preMerged'`, `fitTasks` expects a reference model that already contains the model's own reactions, so no merge is needed for each task, and `ftINITFillGaps` fills the gaps. +### Algorithms of `fillGaps` -How `fitTasks` treats a task that requires net production of a metabolite without reactions was not checked. +The option `algorithm` of `fillGaps` selects one of these algorithms. A value that is none of them runs the algorithm `'reference'`. -The `algorithm` option of `fillGaps` selects among the following. +- `'fastLP'` and `'swiftLP'` relax the problem to a linear program (LP) in the manner of FASTCORE and SWIFTCORE. +- `'gapfillMILP'` is a MILP with a growth floor. Reactions that are already in the draft model keep their bounds, so a reaction with the wrong direction is not changed. +- `'topological'` is a pre-screen of producibility by breadth-first search. It does not modify the model, requires a reference model and uses only the first one. -- `'fastLP'` and `'swiftLP'`: FASTCORE-style and SWIFTCORE-style linear programming (LP) relaxation. -- `'gapfillMILP'`: growth-floor MILP. Reactions already in the draft model keep their bounds, so a reaction with the wrong direction is not changed. -- `'topological'`: breadth-first-search producibility pre-screen, without model modification. It requires a reference model and uses only the first one. +The function `fillGaps` calls `gapFillTopological` with the defaults of that function. The seeds are the metabolites of the exchange reactions that `getExchangeRxns` classes as `'uptake'` or `'reverse'`. The targets are the metabolites of the first objective reaction, and a model without an objective reaction raises `RAVEN:badInput`. -`fillGaps` calls `gapFillTopological` with the defaults of that function. The seeds are the metabolites of the exchange reactions that `getExchangeRxns` classes as `'uptake'` or `'reverse'`. The targets are the metabolites of the first objective reaction. A model without an objective reaction raises `RAVEN:badInput`. A value of `algorithm` that is none of the listed ones runs the `'reference'` algorithm. +### Gap-filling in `fitTasks` -With `resolveTies`, each `ftINIT` MILP step holds the score at its optimum and then chooses among the tied solutions in phases, each with a time limit: the fewest kept removable reactions, the lowest summed ID rank of those (a merged reaction is ranked by the smallest original ID of its group), the least total absolute flux, and the least ID-rank-weighted backward flux. `ftINITInternalAlg` has the arguments `resolveTies` (13th), `rankKeys` (14th, default `model.rxns`) and `pinFluxes` (15th, default `true`); `pinFluxes` set to `false` skips the two flux phases, and `ftINIT` passes `false` in its last step. A step whose first solve stops at the time limit keeps that solution without the phases, and the count and rank phases get half of the step time limit each; a phase that fails issues a warning. +With `gapFillMode` set to `'preMerged'`, `fitTasks` expects a reference model that already contains the reactions of the model. No merge is then needed for each task, and `ftINITFillGaps` fills the gaps. How `fitTasks` treats a task that requires net production of a metabolite without reactions was not checked. + +### Tie-breaking in `ftINIT` + +With `resolveTies`, each MILP step of `ftINIT` holds the score at its optimum and then chooses among the tied solutions in phases, each with a time limit. The phases select, in this order: + +1. the fewest kept removable reactions; +2. the lowest summed ID rank of those reactions, where a merged reaction is ranked by the smallest original ID of its group; +3. the least total absolute flux; +4. the least backward flux, weighted by ID rank. + +The function `ftINITInternalAlg` has the arguments `resolveTies` (13th), `rankKeys` (14th, default `model.rxns`) and `pinFluxes` (15th, default `true`). With `pinFluxes` set to `false` the two flux phases are skipped, and `ftINIT` passes `false` in its last step. A step whose first solve stops at the time limit keeps that solution without the phases. The count phase and the rank phase get half of the step time limit each, and a phase that fails issues a warning. --- (new-functionality)= ## New functionality -RAVEN 3 adds the functions in the table. `randomSampling` and `parseScores` are RAVEN 2 functions with changed behaviour. +The table lists the functions that RAVEN 3 adds. The functions `randomSampling` and `parseScores` exist in RAVEN 2 and behave differently. | Area | Functions | Capability | |---|---|---| @@ -44,28 +55,28 @@ RAVEN 3 adds the functions in the table. `randomSampling` and `parseScores` are | Flux sampling (`analysis/`) | `randomSampling` (changed) | Dispatcher with the option `'method'`: `'achr'` (default), `'chrr'` and `'randomObjective'` (the RAVEN 2 method) | | Flux analysis and quality control (QC) | `compareFluxes` | Compares flux vectors | | Flux analysis and QC | `getMinimalMedium` | Searches a minimal medium with a MILP | -| Flux analysis and QC | `traceFluxPath` | Traces the path of highest flux fraction between two reactions | +| Flux analysis and QC | `traceFluxPath` | Traces the path of the highest flux fraction between two reactions | | Flux analysis and QC | `walkFluxes` | Navigates the flux neighbourhood interactively | -| Flux analysis and QC | `modelSummary` (`queries/`) | Prints a model and flux summary | -| Model comparison (`comparison/`) | `diffModels` | Compares two models by ID: stoichiometry, bounds, objective, `grRules`, EC codes and metabolite properties; separate from the overlap comparison of `compareMultipleModels` | -| Manipulation | `findDuplicateRxns` | Duplicate-stoichiometry query | +| Flux analysis and QC | `modelSummary` (`queries/`) | Prints a summary of a model and its fluxes | +| Model comparison (`comparison/`) | `diffModels` | Compares two models by ID: stoichiometry, bounds, objective, `grRules`, EC codes and metabolite properties. It is separate from the overlap comparison of `compareMultipleModels` | +| Manipulation | `findDuplicateRxns` | Query for reactions with the same stoichiometry | | Genome-based curation (`curation/`) | `downloadGenomeData`, `getGeneData`, `processProteinFastaFile` | Pipeline from an NCBI genome accession to a gene ID mapping table | -| Genome-based curation (`curation/`) | `renameModelGenes` | Bulk gene renaming in a model | -| Genome-based curation (`curation/`) | `curateModelFromTables` | Bulk curation of metabolites, reactions and genes from `.tsv` files against an existing model | -| Biomass composition (`biomass/`) | `getBiomassFractions`, `scaleBiomassFraction`, `scaleBiomassPseudoreaction`, `setGAM` | Biomass-composition and growth-associated-maintenance (GAM) calibration; `tutorials/tutorial7.m` demonstrates the workflow | -| Localisation (`localization/`) | `assignCompartments`, `defaultCompartmentMap`, `getUniProtScores` | Deterministic MILP compartment assignment (alternative to the heuristic `predictLocalization`) | -| Localisation (`localization/`) | `parseScores` (changed) | Adds `'mulocdeep'`, `'compartments'` and `'uniprot'`; `'deeploc'` reads DeepLoc 2 output; the default predictor is `'deeploc'` (RAVEN 2: `'wolf'`); WoLF PSORT is removed (see [Backward-incompatible changes](#backward-incompatible-changes)) | -| Annotation (`annotation/`) | `assignSBOterms`, `loadDeltaGCSV`, `saveDeltaGCSV` | SBO term assignment; ΔG data CSV I/O | -| Conditions (`conditions/`) | `applyCondition` | Applies a condition given as a YAML file to a model, in a fixed order: exchange bounds, cofactor-pseudoreaction edits, biomass-stoichiometry deltas and per-reaction bound overrides | -| Infrastructure | `ravenModelFields` | Registry of every 1-D model field (entity type, default value), used by `permuteModel`, `removeReactions` and `curateModelFromTables` | -| Infrastructure | `progressReport`, `setRavenProgress` | Progress reporting with a selectable backend (for example a log for compute clusters) | -| Infrastructure | `splitProbForConditioning` (`solver/`) | Splitting of an optimisation problem for conditioning | +| Genome-based curation (`curation/`) | `renameModelGenes` | Renames genes in bulk | +| Genome-based curation (`curation/`) | `curateModelFromTables` | Curates metabolites, reactions and genes of an existing model in bulk from `.tsv` files | +| Biomass composition (`biomass/`) | `getBiomassFractions`, `scaleBiomassFraction`, `scaleBiomassPseudoreaction`, `setGAM` | Calibration of the biomass composition and of the growth-associated maintenance (GAM). `tutorials/tutorial7.m` demonstrates the workflow | +| Localisation (`localization/`) | `assignCompartments`, `defaultCompartmentMap`, `getUniProtScores` | Deterministic assignment of compartments by MILP, as an alternative to the heuristic `predictLocalization` | +| Localisation (`localization/`) | `parseScores` (changed) | Adds `'mulocdeep'`, `'compartments'` and `'uniprot'`. The predictor `'deeploc'` reads DeepLoc 2 output and is the default (RAVEN 2: `'wolf'`). WoLF PSORT is removed (see [Backward-incompatible changes](#backward-incompatible-changes)) | +| Annotation (`annotation/`) | `assignSBOterms`, `loadDeltaGCSV`, `saveDeltaGCSV` | Assignment of SBO terms; CSV I/O of ΔG data | +| Conditions (`conditions/`) | `applyCondition` | Applies a condition from a YAML file to a model in a fixed order: exchange bounds, edits of cofactor pseudoreactions, changes of the biomass stoichiometry, and bound overrides per reaction | +| Infrastructure | `ravenModelFields` | Registry of every one-dimensional model field (entity type, default value), used by `permuteModel`, `removeReactions` and `curateModelFromTables` | +| Infrastructure | `progressReport`, `setRavenProgress` | Progress reporting with a selectable backend, for example a log file on compute clusters | +| Infrastructure | `splitProbForConditioning` (`solver/`) | Splits an optimisation problem to improve its conditioning | -`randomSampling` changes the default: a RAVEN 2 call `randomSampling(model, n)` runs ACHR; adding `'method', 'randomObjective'` runs the RAVEN 2 method. `randomSampling` also gains the outputs and options `info`, `seed`, `thinning` and `nBurnin`. +The call `randomSampling(model, n)` runs ACHR, where RAVEN 2 ran its own method. Adding `'method', 'randomObjective'` runs the RAVEN 2 method. The function also gains the outputs and options `info`, `seed`, `thinning` and `nBurnin`. -The function documentation of `sampleCHRR` states that the polytope is rounded before sampling and mixing stays fast on strongly elongated polytopes such as enzyme-constrained models, and that `sampleACHR` does not round. The mixing speed was not measured. +The function documentation of `sampleCHRR` states that the polytope is rounded before sampling and that mixing stays fast on strongly elongated polytopes such as enzyme-constrained models. It states that `sampleACHR` does not round. The mixing speed was not measured. -`findPotentialErrors`, the parser functions and `diffModels` rule comparison are described in [Parsing of gene-reaction rules](#gpr-parsing). The reconstruction and curation functions for KEGG, homology and genome data are described in [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction). +[Parsing of gene-reaction rules](#gpr-parsing) describes `findPotentialErrors`, the parser functions and the rule comparison of `diffModels`. [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction) describes the reconstruction and curation functions for KEGG, homology and genome data. --- @@ -74,23 +85,31 @@ The function documentation of `sampleCHRR` states that the polytope is rounded b | Aspect | RAVEN 2 | RAVEN 3 | |---|---|---| -| MATLAB version | No minimum stated; `readYAMLmodel` has a fallback for releases before R2020b (`readlines`) | No minimum stated; the code has no check that rejects an older release. `readYAMLmodel` keeps the fallback for releases before R2020b, `parseTaskList` calls `readcell` (R2019a) for Excel task lists and `addIdentifierPrefix` calls `regexpPattern` (R2020b). Continuous integration uses R2024b only. Older releases were not tested | -| Statistics and Machine Learning Toolbox | Required by `compareMultipleModels` (`mdscale`, `pdist`, `squareform` and `tsne`; an error without them) and by `compareRxnsGenesMetsComps` (`combnk`) | Optional in `compareMultipleModels`; without it the projection and heatmap clustering are skipped | +| MATLAB version | No minimum stated. `readYAMLmodel` has a fallback for releases before R2020b (`readlines`) | No minimum stated, and the code has no check that rejects an older release. `readYAMLmodel` keeps the fallback for releases before R2020b, `parseTaskList` calls `readcell` (R2019a) for Excel task lists, and `addIdentifierPrefix` calls `regexpPattern` (R2020b). Continuous integration uses R2024b only, and older releases were not tested | +| Statistics and Machine Learning Toolbox | Required by `compareMultipleModels` (`mdscale`, `pdist`, `squareform` and `tsne`, with an error without them) and by `compareRxnsGenesMetsComps` (`combnk`) | Optional in `compareMultipleModels`. Without it the projection and the clustering of the heatmap are skipped | | Bioinformatics Toolbox | Required by `getKEGGModelForOrganism` when it builds missing per-KO alignments from a local KEGG dump (`fastaread` and `fastawrite`) | Not required (`io/readFasta.m` and `io/writeFasta.m`) | -| Optimization Toolbox | Required for `qMOMA` and `solveQP` (`quadprog`) | Not required (both removed) | -| Parallel Computing Toolbox | Optional; `parfor` loops (for example in `getAllowedBounds` and `randomSampling`) run serially without it | Optional; `parallelWorkersRAVEN` returns 0 workers when `runParallel` is false, which runs the `parfor` loops as plain loops | -| Excel I/O | Bundled Apache POI (Java) | Dependency-free OOXML writer | -| LP and MILP solver | Required (bundled `glpk`, or Gurobi, SCIP or COBRA) | Unchanged | -| External binaries | Committed to the repository (BLAST+, DIAMOND, HMMER); available offline after cloning | Fetched on first use as per-platform ZIPs into `src/external/`; for air-gapped installs the workflow `.github/workflows/release-bundle.yml` builds `RAVEN--full.zip` with all platform binaries and attaches it to a published release (whether a given release carries it was not checked) | +| Optimization Toolbox | Required for `qMOMA` and `solveQP` (`quadprog`) | Not required, because both functions are removed | +| Parallel Computing Toolbox | Optional. The `parfor` loops, for example in `getAllowedBounds` and `randomSampling`, run serially without it | Optional. `parallelWorkersRAVEN` returns 0 workers when `runParallel` is false, which runs the `parfor` loops as plain loops | +| Excel I/O | Bundled Apache POI (Java) | OOXML writer without dependencies | +| LP and MILP solver | Required: bundled `glpk`, or Gurobi, SCIP or COBRA | Unchanged | +| External binaries | Committed to the repository (BLAST+, DIAMOND, HMMER) and available offline after cloning | Fetched on first use as ZIP files per platform into `src/external/`. For air-gapped installs the workflow `.github/workflows/release-bundle.yml` builds `RAVEN--full.zip` with the binaries of all platforms and attaches it to a published release. Whether a given release carries it was not checked | | KEGG HMM libraries and reference data | Manual download or local build | Fetched on first use and cached | -| Installation self-check | `installation/checkInstallation.m` runs `importExportTests`, `solverTests` and the BLAST+, DIAMOND, HMMER, CD-HIT and MAFFT test files to fill its table of passed and failed tests | `checkRaven` in the repository root imports `empty.xml`, exports SBML, imports and exports YAML, exports Excel, solves a two-reaction LP per solver and downloads missing binaries without a prompt; `checkInstallation` forwards to it (see [Renamed and merged functions](#renamed-merged-and-consolidated-functions)) | +| Installation self-check | `installation/checkInstallation.m` runs `importExportTests`, `solverTests` and the test files of BLAST+, DIAMOND, HMMER, CD-HIT and MAFFT to fill its table of passed and failed tests | `checkRaven` in the repository root imports `empty.xml`, exports SBML, imports and exports YAML, exports Excel, solves a two-reaction LP per solver and downloads missing binaries without a prompt. `checkInstallation` forwards to it (see [Renamed and merged functions](#renamed-merged-and-consolidated-functions)) | + +KEGG data, binaries and the checksum manifest come from releases of `toolbox-assets`. Other downloads have other sources. + +- KEGG data comes from the release tag that `keggDataVersion()` returns (`src/reconstruction/kegg/KEGG_VERSION.md`, tag `kegg118`). +- The binaries come from the fixed tags in `src/installation/downloadRavenBinaries.m`: `blast-2.17.0`, `diamond-2.1.17`, `hmmer-3.4.0` and, on Windows, `hmmer-3.3.2`. +- The checksums of downloaded files come from `manifest.json` of the `manifest-v3` release, which is read at download time. +- `setRavenSolver('scip')` on Windows downloads `scip_mex_win.zip` from the RAVEN v2.8.6 release on GitHub into `src/external/scip`. The repository holds no SCIP files, and other platforms need a SCIP installation made by hand. +- `getUniProtScores` queries rest.uniprot.org, `downloadGenomeData` downloads from NCBI, and `checkRaven` reads the version file from GitHub. -KEGG data, binaries and the checksum manifest come from releases of `toolbox-assets`; other downloads have other sources (listed below). KEGG data comes from the release tag that `keggDataVersion()` returns (`src/reconstruction/kegg/KEGG_VERSION.md`, tag `kegg118`). The binaries come from fixed tags set in `src/installation/downloadRavenBinaries.m`: `blast-2.17.0`, `diamond-2.1.17`, `hmmer-3.4.0` and, on Windows, `hmmer-3.3.2`. The checksums of downloaded files come from `manifest.json` of the `manifest-v3` release, read at download time. `setRavenSolver('scip')` on Windows downloads `scip_mex_win.zip` from the RAVEN v2.8.6 release on GitHub into `src/external/scip`; the repository holds no SCIP files, and other platforms need a manually installed SCIP. `getUniProtScores` queries rest.uniprot.org, `downloadGenomeData` downloads from NCBI and `checkRaven` reads the version file from GitHub. A RAVEN 3 clone needs network access for these downloads. +A RAVEN 3 clone needs network access for these downloads. --- ## Error and warning handling -RAVEN 3 raises errors and issues warnings with native `error()` and `warning()` calls. Of 411 `error` calls outside `src/external` (lines that contain `error(` and do not start with a comment), 301 carry an identifier, 242 of them in the `RAVEN:*` namespace. Of 94 `warning` calls counted by the same rule, 79 carry an identifier, 72 of them in the `RAVEN:*` namespace. Whether message texts of RAVEN 2 checks are preserved was not checked; a `try/catch` block that matches message text can be replaced by a match on the identifier where one exists. [Backward-incompatible changes](#backward-incompatible-changes) describes the `dispEM` replacement, and [Removed functionality](#removed-functionality) names the `utils/ravenList.m` helper. +RAVEN 3 raises errors and issues warnings with `error()` and `warning()`. Outside `src/external` there are 411 `error` calls, counted as lines that contain `error(` and do not start with a comment. Of these, 301 carry an identifier, and 242 of those are in the `RAVEN:*` namespace. By the same rule there are 94 `warning` calls, of which 79 carry an identifier and 72 are in the `RAVEN:*` namespace. Whether the message texts of the RAVEN 2 checks are preserved was not checked. A `try/catch` block that matches message text can be replaced by a match on the identifier where one exists. [Backward-incompatible changes](#backward-incompatible-changes) describes the replacement of `dispEM`, and [Removed functionality](#removed-functionality) names the helper `utils/ravenList.m`. -`issues = checkModelStruct(model)` returns a structure array of the problems (`category`, `target` and `message`) for programmatic model QC and raises no error. Without an output argument the function raises errors or issues warnings as in RAVEN 2. The options `throwErrors` and `trimWarnings` (positional in RAVEN 2) can also be given as name-value pairs. +The call `issues = checkModelStruct(model)` returns a structure array of the problems (`category`, `target` and `message`) for programmatic model QC and raises no error. Without an output argument the function raises errors or issues warnings as in RAVEN 2. The options `throwErrors` and `trimWarnings`, which are positional in RAVEN 2, can also be given as name-value pairs. diff --git a/docs/raven3-migration/structure-and-renames.md b/docs/raven3-migration/structure-and-renames.md index 1e451ca..fd9be2a 100644 --- a/docs/raven3-migration/structure-and-renames.md +++ b/docs/raven3-migration/structure-and-renames.md @@ -1,107 +1,105 @@ # Structure, removals and renames -RAVEN 3 organises functions in folders named by function area, removes 65 RAVEN 2 functions, merges or renames 8 functions, and keeps 5 RAVEN 2 names as wrappers. +RAVEN 3 sorts its functions into folders by function area. It removes 65 functions of RAVEN 2, merges or renames 8, and keeps 5 RAVEN 2 names as wrappers. ## Repository reorganisation -The RAVEN 3 code is in `src/`. RAVEN 2 kept its functions in the top-level folders `core/`, `external/`, `INIT/`, `hpa/`, `io/`, `legacy/`, `pathway/`, `plotting/`, `solver/`, `struct_conversion/`, `installation/` and `utils/`, and its bundled third-party code in `software/`. Other top-level folders changed as well: `software/` third-party code is in `src/external/`, `tutorial/` is `tutorials/`, `doc/` is `docs/`, and `testing/` is `tests/function_tests/`. +The code of RAVEN 3 is in `src/`. RAVEN 2 kept its functions in the top-level folders `core/`, `external/`, `INIT/`, `hpa/`, `io/`, `legacy/`, `pathway/`, `plotting/`, `solver/`, `struct_conversion/`, `installation/` and `utils/`, and its bundled third-party code in `software/`. Other top-level folders moved as well: the third-party code of `software/` is in `src/external/`, `tutorial/` is `tutorials/`, `doc/` is `docs/`, and `testing/` is `tests/function_tests/`. | RAVEN 3 folder (under `src/`) | Contents | RAVEN 2 source | |---|---|---| -| `INIT/` | `ftINIT`, `INIT/tINIT/` for `getINITModel` and `runINIT`, and HPA parsing, expression parsing and gene scoring (`parseHPA`, `parseHPArna`, `scoreModel`) | `INIT/`, `hpa/` | -| `analysis/` | Analyses related to flux balance analysis (FBA): flux variability analysis (FVA), sampling, FSEOF, OptKnock, robustness, production envelope, flux comparison and tracing | `core/` | +| `INIT/` | `ftINIT`, `INIT/tINIT/` with `getINITModel` and `runINIT`, and the parsing of HPA and expression data and the scoring of genes (`parseHPA`, `parseHPArna`, `scoreModel`) | `INIT/`, `hpa/` | +| `analysis/` | Analyses based on flux balance analysis (FBA): flux variability analysis (FVA), sampling, FSEOF, OptKnock, robustness, production envelope, flux comparison and flux tracing | `core/` | | `annotation/` | MIRIAM editing, SBO term assignment, ΔG CSV I/O | `struct_conversion/` (and new functions) | -| `biomass/` | Biomass composition fitting and scaling, growth-associated maintenance (GAM) | `core/` (and new functions) | -| `comparison/` | Multi-model comparison, structured model diffing | `core/` (and new functions) | -| `conditions/` | Applying a data-driven growth or media condition to a model | New folder | +| `biomass/` | Fitting and scaling of the biomass composition, growth-associated maintenance (GAM) | `core/` (and new functions) | +| `comparison/` | Comparison of several models, structured model diffing | `core/` (and new functions) | +| `conditions/` | Applying a growth or media condition from data to a model | New folder | | `conversion/` | RAVEN⇄COBRA conversion, identifier prefixing, field ordering | `struct_conversion/` | -| `curation/` | Genome-annotation-driven gene and model curation | New folder | -| `external/` | Bundled third-party code (GLPKmex, libSBML and m2html) and the download target of the BLAST+, DIAMOND and HMMER binaries (`src/external/`). `src/external/scip/` holds no files; `setRavenSolver('scip')` downloads the SCIP MEX file there on Windows (see [Toolbox dependencies and installation](#toolbox-deps)) | `software/` | -| `gapfilling/` | `fillGaps` and its pluggable algorithms, task fitting, leak-metabolite and exchange checks | `core/`, `INIT/` (and new functions) | -| `installation/` | Path setup, on-demand binary and data download | `installation/`, `core/` (`findRAVENroot`), `external/` (`updateDocumentation`) | +| `curation/` | Gene and model curation driven by a genome annotation | New folder | +| `external/` | Bundled third-party code (GLPKmex, libSBML and m2html) and the target folder `src/external/` for the downloaded BLAST+, DIAMOND and HMMER binaries. `src/external/scip/` holds no files, and `setRavenSolver('scip')` downloads the SCIP MEX file into it on Windows (see [Toolbox dependencies and installation](#toolbox-deps)) | `software/` | +| `gapfilling/` | `fillGaps` and its algorithms, task fitting, checks for leak metabolites and exchange | `core/`, `INIT/` (and new functions) | +| `installation/` | Path setup, download of binaries and data on demand | `installation/`, `core/` (`findRAVENroot`), `external/` (`updateDocumentation`) | | `io/` | Model import and export (SBML, YAML, Excel), FASTA I/O | `io/` (and new functions) | -| `localization/` | Subcellular localisation prediction and scoring, compartment assignment | `external/`, `core/` (`predictLocalization`) (and new functions) | -| `manipulation/` | Structural edits: add, remove and change reactions, metabolites and genes; merge, sort, simplify | `core/`, `io/` (`closeModel`, `sortIdentifiers`), `INIT/` (`removeLowScoreGenes`, `reverseRxns`) | -| `queries/` | Non-mutating lookups: indexes, exchange and transport reactions, stoichiometry construction, model checking | `core/` (and new functions) | -| `reconstruction/` | KEGG- and homology-based draft reconstruction | `external/`, `core/` (`getModelFromHomology`, `guessComposition`) | +| `localization/` | Prediction and scoring of subcellular localisation, compartment assignment | `external/`, `core/` (`predictLocalization`) (and new functions) | +| `manipulation/` | Structural edits: adding, removing and changing reactions, metabolites and genes; merging, sorting and simplifying | `core/`, `io/` (`closeModel`, `sortIdentifiers`), `INIT/` (`removeLowScoreGenes`, `reverseRxns`) | +| `queries/` | Lookups that do not change the model: indexes, exchange and transport reactions, construction of stoichiometry, model checking | `core/` (and new functions) | +| `reconstruction/` | Draft reconstruction from KEGG and from homology | `external/`, `core/` (`getModelFromHomology`, `guessComposition`) | | `solver/` | LP and MILP solving | `solver/`, `core/` (`getMinNrFluxes`) (and new functions) | -| `tasks/` | Metabolic task list parsing and checking | `core/` | -| `utils/` | Generic helpers, including the argument-parsing and registry infrastructure | `utils/`, `core/` (`convertCharArray`, `printOrange`) | +| `tasks/` | Parsing and checking of metabolic task lists | `core/` | +| `utils/` | General helpers, among them the argument parsing and the registry | `utils/`, `core/` (`convertCharArray`, `printOrange`) | | `deprecated/` | Wrappers for merged and renamed functions | New folder | -| `tests/function_tests/` (repository root, outside `src/`) | Class-based `matlab.unittest` suite (23 of 25 classes derive from `RavenTestCase`; `tBinaries` and `tParseRAVENargs` derive from `matlab.unittest.TestCase`) | `testing/` (function-based tests built with `functiontests`) | +| `tests/function_tests/` (repository root, outside `src/`) | Class-based `matlab.unittest` suite. 23 of 25 classes derive from `RavenTestCase`, and `tBinaries` and `tParseRAVENargs` derive from `matlab.unittest.TestCase` | `testing/` (function-based tests built with `functiontests`) | -`checkRaven.m` and the wrapper `checkInstallation.m` are in the repository root. +The files `checkRaven.m` and `checkInstallation.m` (a wrapper) are in the repository root. -Removed without a successor folder: the subfolder `external/metacyc/` and the MetaCyc function `external/combineMetaCycKEGGModels.m`, `pathway/` and `plotting/` (visualisation), `legacy/` (CellDesigner import and other pre-2.0 code), and the Excel-POI helpers of `io/` together with the Apache POI libraries of `software/apache-poi`. +Some code has no successor folder: the subfolder `external/metacyc/` with the MetaCyc function `external/combineMetaCycKEGGModels.m`, `pathway/` and `plotting/` (visualisation), `legacy/` (CellDesigner import and other code from before RAVEN 2.0), and the Excel-POI helpers of `io/` together with the Apache POI libraries in `software/apache-poi`. -A script that calls RAVEN functions by name works when the whole RAVEN folder tree, including subfolders, is on the MATLAB path (through `addRavenToUserPath` or `addpath(genpath(...))`), because MATLAB resolves functions by name across the path. The reorganisation affects a script that uses a hard-coded path to a specific `.m` file, calls a removed function or a merged function without a wrapper, or contains its own function with the same name as a RAVEN 3 function. `checkFunctionUniqueness` warns about the last case. - -The mapping of every moved file is not listed. +A script that calls RAVEN functions by name keeps working when the whole RAVEN folder tree, subfolders included, is on the MATLAB path (through `addRavenToUserPath` or `addpath(genpath(...))`), because MATLAB resolves functions by name across the path. The reorganisation affects a script in three cases: it uses a hard-coded path to a specific `.m` file, it calls a removed function or a merged function that has no wrapper, or it contains a function with the name of a RAVEN 3 function. `checkFunctionUniqueness` warns about the last case. --- (removed-functionality)= ## Removed functionality -The table lists all 65 RAVEN 2 functions that RAVEN 3 removes without a successor of the same name. The numbers per RAVEN 2 folder are: `pathway/` 6, `plotting/` 10, `external/` 12 (8 MetaCyc, 4 others), `legacy/` 11, `io/` 12, `core/` 8, `utils/` 3, `solver/` 2 and `INIT/` 1. A function of RAVEN 3 calls no removed function; the name `getToolboxVersion` remains as a local function `getToolboxVersion` in `exportForGit`. +The table lists all 65 functions of RAVEN 2 that RAVEN 3 removes without a successor of the same name. The counts per RAVEN 2 folder are `pathway/` 6, `plotting/` 10, `external/` 12 (8 for MetaCyc, 4 others), `legacy/` 11, `io/` 12, `core/` 8, `utils/` 3, `solver/` 2 and `INIT/` 1. No function of RAVEN 3 calls a removed function. The name `getToolboxVersion` survives only as a local function of `exportForGit`. | Removed | Former purpose | Replacement | |---|---|---| -| `pathway/colorPathway`, `pathway/getPathwayDimensions`, `pathway/mapPathwayRxnNames`, `pathway/markPathwayWithExpression`, `pathway/markPathwayWithFluxes`, `pathway/trimPathway` (6 functions) | Metabolic pathway map handling and colouring | None in RAVEN 3. Alternatives are the COBRA Toolbox, Escher and export of flux and omics data for external plotting. | -| `plotting/colorSubsystem`, `plotting/drawMap`, `plotting/drawPathway`, `plotting/getColorCodes`, `plotting/getObjectiveString`, `plotting/plotAdditionalInfo`, `plotting/plotLabels`, `plotting/setColorToMapRxns`, `plotting/setOmicDataToRxns`, `plotting/setTitle` (10 functions) | Metabolic map drawing and omics colouring | None, as above. | -| `external/metacyc/{getModelFromMetaCyc,getRxnsFromMetaCyc,getMetsFromMetaCyc,getEnzymesFromMetaCyc,getMetaCycModelForOrganism,addSpontaneousRxns,linkMetaCycKEGGRxns}`, `external/combineMetaCycKEGGModels` (8 functions) | MetaCyc-based draft reconstruction | None; KEGG- and homology-based reconstruction are the reconstruction methods of RAVEN 3. | -| `legacy/core/changeGeneAssoc`, `legacy/external/constructPathwayFromCelldesigner`, and the nine functions of the bundled `xml_toolbox` (`xml_format`, `xml_formatany`, `xml_help`, `xml_load`, `xml_parse`, `xml_parseany`, `xml_save`, `xml_strsplit`, `xml_unblank`) (11 functions) | CellDesigner import and pre-2.0 compatibility code | None. | -| `io/importExcelModel`, `io/loadWorkbook`, `io/loadSheet`, `io/writeSheet`, `io/SBMLFromExcel`, `io/addJavaPaths`, `io/startup` (7 functions) | Apache-POI-based Excel model I/O | `io/writeExcel.m` (dependency-free OOXML writer) for export; `curation/curateModelFromTables.m` for tabular curation. There is no full Excel-template import; see [Excel I/O](#excel-io). | -| `io/exportModelToSIF`, `io/exportToTabDelimited` | SIF graph export; tab-delimited model export | None. The sole caller of `exportToTabDelimited` was the bare-path fallback of `exportToExcelFormat`, which is removed. | -| `io/getFullPath`, `io/getMD5Hash`, `io/getToolboxVersion` | Path canonicalisation; file MD5 hashing; toolbox git-version lookup | Inlined where needed: the MD5 hash through Java `MessageDigest` in `getBlast`, the version lookup as a local function of `exportForGit`. | -| `core/dispEM` | Error and warning display (388 lines call it in RAVEN 2, counted outside `doc/` and `testing/`) | Native `warning()` and `error()`, and `utils/ravenList.m` for formatted item lists. See [Backward-incompatible changes](#backward-incompatible-changes). | -| `core/followFluxes`, `core/followChanged` | Flux-change reporting relative to a reference solution | `analysis/compareFluxes.m` compares two flux vectors of one model (reactions turned on or off, flipped and changed, with `cutoff` and `metaboliteList`), which replaces `core/followChanged`; `queries/printFluxes.m` prints the fluxes of one vector above a cutoff. | -| `core/printModel` | Printed reactions to screen or file | `queries/printModelStats.m` and `queries/printFluxes.m` cover model summaries and flux printing (see below the table). | -| `core/getMetsInComp` | Metabolite indices in a compartment | `model.metComps == compIndex` | -| `core/mapCompartments` | Merged and split the score columns of a localisation-score structure by compartment rules | The `compartmentMap` option of `parseScores` and `getUniProtScores`. The default map `defaultCompartmentMap` applies to `getUniProtScores` and to `parseScores` with the `'uniprot'` predictor; elsewhere, without a map, every label stays unchanged. With a map, a label that the map does not contain is dropped together with its score column. Merged columns take the maximum score; splitting is not supported. | -| `core/getExpressionStructure` | Loaded an expression structure from an ad hoc Excel format | `INIT/parseHPA.m` and `INIT/parseHPArna.m` read Human Protein Atlas files. Expression tables from other sources have to be assembled by hand as the `arrayData` structure documented in `scoreModel`. | -| `core/checkRxn` | Per-reaction synthesisability debugging | No direct successor; `gapfilling/findLeakMetabolite.m`, `gapfilling/canExchange.m` and `gapfilling/checkProduction.m` cover related checks. | -| `external/getWoLFScores` | WoLF PSORT score parsing | None; `parseScores` with `'wolf'` raises `RAVEN:badInput`. See [Backward-incompatible changes](#backward-incompatible-changes). | -| `external/kegg/constructMultiFasta`, `external/kegg/getWSLpath`, `external/getBlastFromExcel` | Per-KO multi-FASTA assembly; WSL path translation; Excel-based BLAST import | The KEGG pipeline based on `toolbox-assets` artefacts. See [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction). | -| `solver/qMOMA`, `solver/solveQP` | MOMA analyses and quadratic programming | None; RAVEN 3 has no quadratic programming. See [Backward-incompatible changes](#backward-incompatible-changes) for the arguments removed from `findGeneDeletions`. | -| `utils/emptyOrLogicalScalar`, `utils/emptyOrTextOrCellOfText`, `utils/emptyOrTextScalar` | Argument validators | `utils/parseRAVENargs.m` | +| `pathway/colorPathway`, `pathway/getPathwayDimensions`, `pathway/mapPathwayRxnNames`, `pathway/markPathwayWithExpression`, `pathway/markPathwayWithFluxes`, `pathway/trimPathway` (6 functions) | Handling and colouring of metabolic pathway maps | None in RAVEN 3. Alternatives are the COBRA Toolbox, Escher, and the export of flux and omics data for plotting elsewhere | +| `plotting/colorSubsystem`, `plotting/drawMap`, `plotting/drawPathway`, `plotting/getColorCodes`, `plotting/getObjectiveString`, `plotting/plotAdditionalInfo`, `plotting/plotLabels`, `plotting/setColorToMapRxns`, `plotting/setOmicDataToRxns`, `plotting/setTitle` (10 functions) | Drawing of metabolic maps and colouring with omics data | None, as above | +| `external/metacyc/{getModelFromMetaCyc,getRxnsFromMetaCyc,getMetsFromMetaCyc,getEnzymesFromMetaCyc,getMetaCycModelForOrganism,addSpontaneousRxns,linkMetaCycKEGGRxns}`, `external/combineMetaCycKEGGModels` (8 functions) | Draft reconstruction from MetaCyc | None. The reconstruction methods of RAVEN 3 are the KEGG-based and the homology-based method | +| `legacy/core/changeGeneAssoc`, `legacy/external/constructPathwayFromCelldesigner`, and the nine functions of the bundled `xml_toolbox` (`xml_format`, `xml_formatany`, `xml_help`, `xml_load`, `xml_parse`, `xml_parseany`, `xml_save`, `xml_strsplit`, `xml_unblank`) (11 functions) | CellDesigner import and compatibility code from before RAVEN 2.0 | None | +| `io/importExcelModel`, `io/loadWorkbook`, `io/loadSheet`, `io/writeSheet`, `io/SBMLFromExcel`, `io/addJavaPaths`, `io/startup` (7 functions) | Excel model I/O through Apache POI | `io/writeExcel.m` (OOXML writer without dependencies) for export and `curation/curateModelFromTables.m` for tabular curation. RAVEN 3 cannot import a full Excel template (see [Excel I/O](#excel-io)) | +| `io/exportModelToSIF`, `io/exportToTabDelimited` | Export of a SIF graph; tab-delimited export of a model | None. The only caller of `exportToTabDelimited` was the fallback of `exportToExcelFormat` for a path without file name, which is removed | +| `io/getFullPath`, `io/getMD5Hash`, `io/getToolboxVersion` | Canonical file path; MD5 hash of a file; git version of the toolbox | Inlined where needed: the MD5 hash through Java `MessageDigest` in `getBlast`, and the version as a local function of `exportForGit` | +| `core/dispEM` | Display of errors and warnings (388 lines call it in RAVEN 2, counted outside `doc/` and `testing/`) | `warning()` and `error()`, and `utils/ravenList.m` for formatted lists of items (see [Backward-incompatible changes](#backward-incompatible-changes)) | +| `core/followFluxes`, `core/followChanged` | Report of flux changes relative to a reference solution | `analysis/compareFluxes.m` compares two flux vectors of one model (reactions turned on or off, flipped and changed, with `cutoff` and `metaboliteList`) and replaces `core/followChanged`. `queries/printFluxes.m` prints the fluxes of one vector above a cut-off | +| `core/printModel` | Printing of reactions to the screen or to a file | `queries/printModelStats.m` and `queries/printFluxes.m` cover model summaries and the printing of fluxes (see below the table) | +| `core/getMetsInComp` | Indices of the metabolites in a compartment | `model.metComps == compIndex` | +| `core/mapCompartments` | Merging and splitting of the score columns of a localisation-score structure by compartment rules | The `compartmentMap` option of `parseScores` and `getUniProtScores`. The default map `defaultCompartmentMap` applies to `getUniProtScores` and to `parseScores` with the `'uniprot'` predictor. Without a map, every label stays unchanged. With a map, a label that the map does not contain is dropped together with its score column. Merged columns take the maximum score, and splitting is not supported | +| `core/getExpressionStructure` | Loading of an expression structure from an ad hoc Excel format | `INIT/parseHPA.m` and `INIT/parseHPArna.m` read Human Protein Atlas files. Expression tables from other sources have to be assembled by hand as the `arrayData` structure documented in `scoreModel` | +| `core/checkRxn` | Debugging of the synthesisability of one reaction | No direct successor. `gapfilling/findLeakMetabolite.m`, `gapfilling/canExchange.m` and `gapfilling/checkProduction.m` cover related checks | +| `external/getWoLFScores` | Parsing of WoLF PSORT scores | None. `parseScores` with `'wolf'` raises `RAVEN:badInput` (see [Backward-incompatible changes](#backward-incompatible-changes)) | +| `external/kegg/constructMultiFasta`, `external/kegg/getWSLpath`, `external/getBlastFromExcel` | Assembly of a multi-FASTA file per KO; translation of WSL paths; import of BLAST results from Excel | The KEGG pipeline based on the artefacts of `toolbox-assets` (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)) | +| `solver/qMOMA`, `solver/solveQP` | MOMA analyses and quadratic programming | None. RAVEN 3 has no quadratic programming. The arguments removed from `findGeneDeletions` are in [Backward-incompatible changes](#backward-incompatible-changes) | +| `utils/emptyOrLogicalScalar`, `utils/emptyOrTextOrCellOfText`, `utils/emptyOrTextScalar` | Validators of arguments | `utils/parseRAVENargs.m` | | `INIT/getExprForRxnScore` | Reaction scores from expression data | A private helper of the test suite in `tests/function_tests/private/` | -`printFluxes` prints the reactions of a flux vector with an absolute flux at or above `cutOffFlux` (default `10^-8`), one line per reaction; it prints exchange reactions unless `onlyExchange` is `false`. A listing of all reactions of a model without fluxes has no direct replacement. +The function `printFluxes` prints the reactions of a flux vector whose absolute flux is at least `cutOffFlux` (default `10^-8`), one line per reaction. It prints exchange reactions unless `onlyExchange` is `false`. A listing of all reactions of a model without fluxes has no direct replacement. --- (renamed-merged-and-consolidated-functions)= ## Renamed and merged functions -The `deprecated/` folder holds `makeSomething`, `consumeSomething`, `canProduce` and `canConsume`. These wrappers keep the RAVEN 2 names callable, issue a `RAVEN:deprecated` warning once ("will be removed in the next major release") and forward to the RAVEN 3 function. The other functions in the table have no wrapper, except `checkInstallation`, which has its own row. +The folder `deprecated/` holds `makeSomething`, `consumeSomething`, `canProduce` and `canConsume`. These wrappers keep the RAVEN 2 names callable, issue a `RAVEN:deprecated` warning once ("will be removed in the next major release") and forward to the RAVEN 3 function. The other functions in the table have no wrapper, except `checkInstallation`, which has a row of its own. | RAVEN 2 functions | RAVEN 3 function | Notes | |---|---|---| -| `makeSomething(model, ignoreMets, isNames, minNrFluxes, allowExcretion, params, ignoreIntBounds)` | `findLeakMetabolite(model, 'produce', ...)` | The argument order maps one to one; the wrapper forwards `varargin` unchanged. | -| `consumeSomething(model, ignoreMets, isNames, minNrFluxes, params, ignoreIntBounds)` | `findLeakMetabolite(model, 'consume', ...)` | The RAVEN 2 argument order has no `allowExcretion` position, so the wrapper remaps the arguments by name to keep `params` and `ignoreIntBounds`. | -| `canProduce(model, mets)` | `canExchange(model, 'produce', 'mets', mets)` | None. | -| `canConsume(model, mets)` | `canExchange(model, 'consume', 'mets', mets)` | None. | -| `compareRxnsGenesMetsComps` | `comparison/compareMultipleModels.m` | The overlap output of RAVEN 2 (reactions, genes, metabolites, EC codes, metabolite names, equations and unique equations) is part of the output of `compareMultipleModels`. No wrapper. | -| `INIT/scoreComplexModel.m` | `INIT/scoreModel.m` | Merged; no wrapper. A call moves to `scoreModel` with the same positions and defaults for the first six optional arguments; `dataPrecedence` is added as the 7th optional argument. | -| `INIT/ftINITFillGapsMILP.m` | `solver/getMinNrFluxes.m` | Available through the `resolveTies` option of `getMinNrFluxes`, which is implemented for `formulation` `'reversible'` only (the default `'irrev'` raises an error with `resolveTies`); the `proveAbsGap` option belongs to `ftINIT`. | -| `INIT/ftINITFillGapsForAllTasks.m` | `gapfilling/ftINITFillGaps.m` and the `'preMerged'` gap-fill mode of `fitTasks` | None. `ftINITFillGaps` exists in RAVEN 2 as well. | -| `getGenesFromKEGG`, `getMetsFromKEGG` and `getRxnsFromKEGG` | `reconstruction/kegg/readKEGGTable.m` (with `buildGlobalKEGGModel` and `buildGlobalGPR`) and the `toolbox-assets` KEGG artefacts | The data source differs, so the signatures are not compatible; see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction). | -| `core/parallelPoolRAVEN.m` | `utils/parallelWorkersRAVEN.m` | `parallelWorkersRAVEN` returns a `parfor` worker count (`0` or `Inf`) and starts no pool; `parallelPoolRAVEN` returns `[ps, oldPoolAutoCreate]`. | -| `installation/checkInstallation.m` | `checkRaven.m` in the repository root | The calling signature is identical. `checkInstallation.m` in the repository root forwards to `checkRaven` and issues a `RAVEN:deprecated` warning on every call. | +| `makeSomething(model, ignoreMets, isNames, minNrFluxes, allowExcretion, params, ignoreIntBounds)` | `findLeakMetabolite(model, 'produce', ...)` | The arguments map one to one, and the wrapper forwards `varargin` unchanged | +| `consumeSomething(model, ignoreMets, isNames, minNrFluxes, params, ignoreIntBounds)` | `findLeakMetabolite(model, 'consume', ...)` | The RAVEN 2 order has no position for `allowExcretion`, so the wrapper remaps the arguments by name to keep `params` and `ignoreIntBounds` | +| `canProduce(model, mets)` | `canExchange(model, 'produce', 'mets', mets)` | None | +| `canConsume(model, mets)` | `canExchange(model, 'consume', 'mets', mets)` | None | +| `compareRxnsGenesMetsComps` | `comparison/compareMultipleModels.m` | The overlap output of RAVEN 2 (reactions, genes, metabolites, EC codes, metabolite names, equations and unique equations) is part of the output of `compareMultipleModels`. No wrapper | +| `INIT/scoreComplexModel.m` | `INIT/scoreModel.m` | Merged, no wrapper. A call moves to `scoreModel` with the same positions and defaults for the first six optional arguments, and `dataPrecedence` is added as the 7th optional argument | +| `INIT/ftINITFillGapsMILP.m` | `solver/getMinNrFluxes.m` | Available through the option `resolveTies` of `getMinNrFluxes`, which exists for `formulation` `'reversible'` only. The default `'irrev'` raises an error with `resolveTies`. The option `proveAbsGap` belongs to `ftINIT` | +| `INIT/ftINITFillGapsForAllTasks.m` | `gapfilling/ftINITFillGaps.m` and the gap-fill mode `'preMerged'` of `fitTasks` | None. `ftINITFillGaps` exists in RAVEN 2 as well | +| `getGenesFromKEGG`, `getMetsFromKEGG` and `getRxnsFromKEGG` | `reconstruction/kegg/readKEGGTable.m` (with `buildGlobalKEGGModel` and `buildGlobalGPR`) and the KEGG artefacts of `toolbox-assets` | The data source differs, so the signatures are not compatible (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)) | +| `core/parallelPoolRAVEN.m` | `utils/parallelWorkersRAVEN.m` | `parallelWorkersRAVEN` returns a worker count for `parfor` (`0` or `Inf`) and starts no pool. `parallelPoolRAVEN` returns `[ps, oldPoolAutoCreate]` | +| `installation/checkInstallation.m` | `checkRaven.m` in the repository root | The signature is identical. `checkInstallation.m` in the repository root forwards to `checkRaven` and issues a `RAVEN:deprecated` warning on every call | ### Gene scoring with `scoreModel` -RAVEN 2 has `INIT/scoreComplexModel.m` and `hpa/scoreModel.m`; RAVEN 3 has one `scoreModel`, which tINIT and ftINIT call with different settings. Differences for a call written for `hpa/scoreModel.m` (checked against v2.11.3): +RAVEN 2 has `INIT/scoreComplexModel.m` and `hpa/scoreModel.m`. RAVEN 3 has one `scoreModel`, which tINIT and ftINIT call with different settings. A call written for `hpa/scoreModel.m` differs in these points (checked against 2.11.3): -- RAVEN 2 scored a reaction by the best or average score over all its genes (`multipleGeneScoring`, values `'best'` and `'average'`). RAVEN 3 applies `isozymeScoring` (default `'max'`) to the `or` parts and `complexScoring` (default `'min'`) to the `and` parts of the rule. A reaction with an `and` rule receives a different default score. -- Positions count all arguments of the call. The 7th to 9th arguments of a RAVEN 2 call (`multipleGeneScoring`, `multipleCellScoring`, `hpaLevelScores`) are read as `isozymeScoring`, `complexScoring` and `multipleCellScoring`; `hpaLevelScores` is the 10th argument. -- `multipleCellScoring` accepts `'max'` and `'average'` (RAVEN 2: `'best'` and `'average'`). +- RAVEN 2 scored a reaction by the best or the average score over all its genes (`multipleGeneScoring`, values `'best'` and `'average'`). RAVEN 3 applies `isozymeScoring` (default `'max'`) to the `or` parts and `complexScoring` (default `'min'`) to the `and` parts of the rule. A reaction with an `and` rule therefore gets another default score. +- Positions count all arguments of the call. The 7th to 9th arguments of a RAVEN 2 call (`multipleGeneScoring`, `multipleCellScoring`, `hpaLevelScores`) are read as `isozymeScoring`, `complexScoring` and `multipleCellScoring`. `hpaLevelScores` is the 10th argument. +- `multipleCellScoring` accepts `'max'` and `'average'`. RAVEN 2 accepted `'best'` and `'average'`. - `dataPrecedence` is an argument of RAVEN 3 with the default `'gene'`. -- Genes that are absent from the data receive `NaN` in `geneScores`; the RAVEN 2 documentation states `-Inf`. `getINITModel` converts `NaN` back to `-Inf`. +- A gene that is absent from the data receives `NaN` in `geneScores`, and the RAVEN 2 documentation states `-Inf`. `getINITModel` converts `NaN` back to `-Inf`. -See [Backward-incompatible changes](#backward-incompatible-changes). `getINITModel` passes `'max'` for both `isozymeScoring` and `complexScoring` and `'reaction'` for `dataPrecedence`. Whether these settings reproduce the scores of the RAVEN 2 `hpa/scoreModel` was not checked; the RAVEN 2 value `'average'` for `multipleGeneScoring` has no equivalent among them. +[Backward-incompatible changes](#backward-incompatible-changes) lists the same differences as table rows. The function `getINITModel` passes `'max'` for `isozymeScoring` and for `complexScoring`, and `'reaction'` for `dataPrecedence`. Whether these settings reproduce the scores of the RAVEN 2 `hpa/scoreModel` was not checked, and the RAVEN 2 value `'average'` of `multipleGeneScoring` has no equivalent among them. ### Legacy tINIT functions -`getINITModel` and `runINIT` (under `INIT/tINIT/`) are marked legacy and are not scheduled for removal. `utils/legacyMethodNotice.m` issues a `RAVEN:legacyMethod` warning once, which can be silenced, and points to `ftINIT`. `ftINIT` is the method for new work; `getINITModel` and `runINIT` reproduce models built with them. +The functions `getINITModel` and `runINIT` (in `INIT/tINIT/`) are marked legacy and are not scheduled for removal. `utils/legacyMethodNotice.m` issues a `RAVEN:legacyMethod` warning once, which can be silenced, and points to `ftINIT`. The method `ftINIT` is meant for new work, and `getINITModel` and `runINIT` reproduce models that were built with them. diff --git a/docs/raven3-vs-raven-toolbox.md b/docs/raven3-vs-raven-toolbox.md index c6db421..579955f 100644 --- a/docs/raven3-vs-raven-toolbox.md +++ b/docs/raven3-vs-raven-toolbox.md @@ -1,32 +1,31 @@ # Comparison of RAVEN and raven-toolbox -RAVEN exists as two implementations: the MATLAB toolbox RAVEN 3 and the Python package raven-toolbox. The pages below help a reader who chooses between them or ports a workflow from one to the other. +RAVEN exists in two implementations: the MATLAB toolbox RAVEN 3 and the Python package raven-toolbox. The text below and the two detail pages help a reader who chooses between them or ports a workflow from one to the other. -- RAVEN is a MATLAB toolbox that runs without the COBRA Toolbox. -- raven-toolbox is a Python package built on [cobrapy](https://cobrapy.readthedocs.io/). A model is a `cobra.Model`. +- RAVEN runs in MATLAB without the COBRA Toolbox. +- raven-toolbox is built on [cobrapy](https://cobrapy.readthedocs.io/), and a model is a `cobra.Model`. -The implementations differ in function names, in functions that exist in one implementation only, in the form of returned results and in the results of certain functions. Two detail pages cover these differences: [Features specific to each implementation](raven3-vs-raven-toolbox/what-each-has.md) and [Differences in results between functions](raven3-vs-raven-toolbox/same-function-different-answer.md). +The implementations differ in function names, in functions that only one of them has, in the form of the results and in the results of some functions. [Features specific to each implementation](raven3-vs-raven-toolbox/what-each-has.md) covers the functions that exist in one implementation only. [Differences in results between functions](raven3-vs-raven-toolbox/same-function-different-answer.md) covers functions that exist in both and return different results. ## Choosing an implementation -The first three rows of the table list conditions that favour raven-toolbox, the next four favour RAVEN, and the last row names the functions that both implementations have. +The conditions below favour one implementation. Both implementations provide homology and KEGG reconstruction, metabolic tasks, gap-filling, ftINIT, compartment assignment and model comparison. -| Implementation | Condition | +| Condition | Implementation | |---|---| -| raven-toolbox | The surrounding code is Python. | -| raven-toolbox | The model is used with cobrapy tools as a `cobra.Model`. | -| raven-toolbox | The KEGG artefacts are built from a specified KEGG release (requires a KEGG FTP subscription). | -| RAVEN | The surrounding code is MATLAB. | -| RAVEN | A model built with tINIT is reproduced. | -| RAVEN | Dynamic flux balance analysis (FBA) is run with `runDynamicFBA`. | -| RAVEN | A model is converted in memory to the COBRA Toolbox structure with `ravenCobraWrapper`. | -| Either | Homology and KEGG reconstruction, metabolic tasks, gap-filling, ftINIT, compartment assignment and model comparison. | +| The surrounding code is Python | raven-toolbox | +| The model is used with cobrapy tools as a `cobra.Model` | raven-toolbox | +| The KEGG artefacts are built from a specified KEGG release (requires a KEGG FTP subscription) | raven-toolbox | +| The surrounding code is MATLAB | RAVEN | +| A model built with tINIT is reproduced | RAVEN | +| Dynamic flux balance analysis (FBA) is run with `runDynamicFBA` | RAVEN | +| A model is converted in memory to the COBRA Toolbox structure with `ravenCobraWrapper` | RAVEN | -`read_matlab_model` of raven-toolbox reads a COBRA Toolbox `.mat` file as well as a RAVEN `.mat` file, and `write_matlab_model` writes the RAVEN layout, so a model can be exchanged by file between the implementations. +A model can also be exchanged by file. The function `read_matlab_model` of raven-toolbox reads a COBRA Toolbox `.mat` file as well as a RAVEN `.mat` file, and `write_matlab_model` writes the RAVEN layout. ## Name differences -RAVEN uses `camelCase` and raven-toolbox uses `snake_case`. The generated mapping pairs 103 RAVEN functions with a raven-toolbox function (rule: rows of the mapping with a raven-toolbox function in the second column, cobrapy entries excluded). Of these, 48 have a name whose letters differ from the RAVEN name when case and the separator character `_` are ignored. Examples: +RAVEN uses `camelCase` and raven-toolbox uses `snake_case`. The generated mapping pairs 103 RAVEN functions with a raven-toolbox function. The rule is: rows of the mapping that have a raven-toolbox function in the second column, with the cobrapy entries excluded. Of these 103, 48 have a name whose letters differ from the RAVEN name when case and the separator `_` are ignored. Examples: | RAVEN | raven-toolbox | |---|---| @@ -35,26 +34,26 @@ RAVEN uses `camelCase` and raven-toolbox uses `snake_case`. The generated mappin | `addRxnsGenesMets` | `add_reactions_from_model` | | `fillGaps` | several functions (see [Gap-filling functions](raven3-vs-raven-toolbox/same-function-different-answer.md#gap-filling-functions)) | -`add_reactions_from_equations` takes a sequence of mappings with the keys `id`, `equation`, `name`, `bounds`, `gene_reaction_rule` and `subsystem`, which corresponds to the `equations` form of the `rxnsToAdd` struct of `addRxns`. The form of `addRxns` with `mets` and `stoichCoeffs` has no counterpart, and the metadata fields of the struct (such as `eccodes`, `rxnMiriams` and `rxnNotes`) are not keys of the mapping. +The function `add_reactions_from_equations` takes a sequence of mappings with the keys `id`, `equation`, `name`, `bounds`, `gene_reaction_rule` and `subsystem`. This corresponds to the `equations` form of the `rxnsToAdd` struct of `addRxns`. The form of `addRxns` with `mets` and `stoichCoeffs` has no counterpart, and the metadata fields of the struct, such as `eccodes`, `rxnMiriams` and `rxnNotes`, are not keys of the mapping. -The [MATLAB vs Python](matlab-vs-python.md) page is generated from the sources of both implementations at build time. It has 132 rows for 130 documented RAVEN functions (`parseGrRule` and `setExchangeBounds` have two rows each): 55 matched automatically by name, 48 from a curated list and 29 covered by cobrapy. A further 42 RAVEN functions are listed as deliberately not ported and have no counterpart, so the page accounts for 172 RAVEN functions. +The page [MATLAB vs Python](matlab-vs-python.md) is generated at build time from the sources of both implementations. It has 132 rows for 130 documented RAVEN functions (`parseGrRule` and `setExchangeBounds` have two rows each): 55 matched automatically by name, 48 from a curated list and 29 covered by cobrapy. A further 42 RAVEN functions are listed as deliberately not ported and have no counterpart, so the page accounts for 172 RAVEN functions. ## Form of results -RAVEN functions return structs or several output arguments. Many raven-toolbox functions return dataclasses, for example `GapFillResult`, `TaskResult` and `ElementalBalance`; others return a model or a list, for example `ftinit` and `add_reactions_from_equations`. Porting a call requires changing how the result is read, in addition to the name. +RAVEN functions return structs or several output arguments. Many raven-toolbox functions return dataclasses, for example `GapFillResult`, `TaskResult` and `ElementalBalance`, and others return a model or a list, for example `ftinit` and `add_reactions_from_equations`. Porting a call therefore changes how the result is read, in addition to the name. ## Solvers -The solver is set differently in each implementation. +The two implementations set the solver differently. | Aspect | RAVEN | raven-toolbox | |---|---|---| | Configuration | `setRavenSolver('gurobi')` | `cobra.Configuration().solver = 'gurobi'` | -| Solvers | GLPK, Gurobi, SCIP and the COBRA Toolbox solvers (`setRavenSolver` downloads the SCIP binary for Windows only) | Any solver that cobrapy supports through optlang | +| Solvers | GLPK, Gurobi, SCIP and the COBRA Toolbox solvers. `setRavenSolver` downloads the SCIP binary for Windows only | Any solver that cobrapy supports through optlang | | Free licence | Gurobi (academic) | Gurobi (academic) | -| Bundled open-source solver | GLPK (bundled with RAVEN) | GLPK (installed with cobrapy) | +| Bundled open-source solver | GLPK, bundled with RAVEN | GLPK, installed with cobrapy | -[Choosing a solver](installation/index.md#choosing-a-solver) gives the recommendation by model size. The ftINIT extraction of raven-toolbox returned different reaction sets for different solvers, seeds and thread counts in the measurements of the parameter-tuning studies; [Alternative optima of mixed-integer problems](raven3-vs-raven-toolbox/same-function-different-answer.md#alternative-optima-of-mixed-integer-problems) gives the figures. Whether the solutions are of equal quality was not measured across solvers. +[Choosing a solver](installation/index.md#choosing-a-solver) gives a recommendation by model size. In the measurements of the parameter-tuning studies, the ftINIT extraction of raven-toolbox returned different reaction sets for different solvers, seeds and thread counts. [Alternative optima of mixed-integer problems](raven3-vs-raven-toolbox/same-function-different-answer.md#alternative-optima-of-mixed-integer-problems) gives the figures. Whether the solutions have equal quality was not measured across solvers. ```{toctree} :hidden: diff --git a/docs/raven3-vs-raven-toolbox/same-function-different-answer.md b/docs/raven3-vs-raven-toolbox/same-function-different-answer.md index e5f7a40..9912ab2 100644 --- a/docs/raven3-vs-raven-toolbox/same-function-different-answer.md +++ b/docs/raven3-vs-raven-toolbox/same-function-different-answer.md @@ -1,18 +1,20 @@ # Differences in results between functions -The page lists functions that differ between RAVEN and raven-toolbox in the algorithm choice, in the form of the result or in the value of the result. The differences were found by reading the code of both implementations; where a section says a result was not checked, no run compared the two. The list is incomplete: a function that is not listed can behave differently. Candidates that were not compared include `random_sampling`, which replaces both `sampleACHR` and `sampleCHRR`, `remove_duplicate_reactions`, which replaces `contractModel`, and `simplify_model`, which corresponds to `simplifyModel`. +A function of RAVEN and its counterpart in raven-toolbox can differ in the algorithm they choose, in the form of the result or in the value of the result. The sections below describe the differences found by reading the code of both implementations. Where a section says that a result was not checked, no run compared the two implementations. + +The list is incomplete, and a function that is not listed can behave differently. Three candidates were not compared: `random_sampling`, which replaces both `sampleACHR` and `sampleCHRR`; `remove_duplicate_reactions`, which replaces `contractModel`; and `simplify_model`, which corresponds to `simplifyModel`. ## Metabolic task checking -`checkTasks` and `check_tasks` implement the same task format and the same boundary semantics. `close_boundaries=True` of `check_tasks` corresponds to calling `checkTasks` on `closeModel(model)`; [Metabolic tasks](../guide/tasks.md) describes the options. Identical pass and fail results on a shared input were not checked here. +The functions `checkTasks` and `check_tasks` implement the same task format and the same boundary semantics. The option `close_boundaries=True` of `check_tasks` corresponds to calling `checkTasks` on `closeModel(model)`. [Metabolic tasks](../guide/tasks.md) describes the options. Whether both functions return identical pass and fail results on a shared input was not checked. -`checkTasks` starts each task from the original model structure. `check_tasks` applies each task to one model inside a `with model:` block and reverts the changes afterwards. It also saves and restores direct edits of metabolite-constraint bounds. The runtime of the two functions was not compared. +The function `checkTasks` starts each task from the original model structure. The function `check_tasks` applies each task to one model inside a `with model:` block and reverts the changes afterwards. It also saves and restores direct edits of metabolite-constraint bounds. The runtime of the two functions was not compared. -The `getEssential` option of `checkTasks` has no option in `check_tasks`. In RAVEN, `checkTasks` calls `getEssentialRxns` on the model constrained by each task. The function `find_task_essential_reactions` ports this step, so it corresponds to `checkTasks` with `getEssential`, not to `getEssentialRxns` called on a model without a task. +The option `getEssential` of `checkTasks` has no counterpart option in `check_tasks`. In RAVEN, `checkTasks` calls `getEssentialRxns` on the model constrained by each task, and `find_task_essential_reactions` ports this step. It therefore corresponds to `checkTasks` with `getEssential`, and not to `getEssentialRxns` called on a model without a task. ## Gap-filling functions -`fillGaps` in RAVEN selects one of five algorithms with the `algorithm` option: `reference` (default), `fastLP`, `swiftLP`, `gapfillMILP` and `topological`. raven-toolbox has four functions for them, so porting a `fillGaps` call requires choosing a function from this table: +The function `fillGaps` of RAVEN selects one of five algorithms with the option `algorithm`: `reference` (default), `fastLP`, `swiftLP`, `gapfillMILP` and `topological`. raven-toolbox has four functions for them, so porting a `fillGaps` call means choosing one function from the table. | Purpose | RAVEN algorithm | raven-toolbox function | |---|---|---| @@ -21,45 +23,45 @@ The `getEssential` option of `checkTasks` has no option in `check_tasks`. In RAV | Filling by mixed-integer linear programming (MILP) with the fewest added reactions that restore growth | `fillGaps` with `'gapfillMILP'` | `fill_gaps_milp` | | Finding the gaps without filling them | `fillGaps` with `'topological'` | `analyse_topology` | -Further differences: +The functions differ in three more points. -- `fillGaps` with `useModelConstraints=true` fills so that the constraints of the model hold. `connect_blocked_reactions` has no such mode; its documentation points to `cobra.flux_analysis.gapfill` for that use. `fill_gaps_milp` is the closest function for a growth target. -- `fillGaps` takes a cell array of template models, and the RAVEN functions `gapFillFastLP`, `gapFillMILP` and `gapFillTopological` use only the first template. The four raven-toolbox functions accept several templates. -- `fillGaps` returns `newConnected`, `cannotConnect`, `addedRxns`, `newModel` and `exitFlag`. raven-toolbox returns a `GapFillResult` from `connect_blocked_reactions`, a `FastLPResult` from `fill_gaps_fast_lp`, a `MILPGapFillResult` from `fill_gaps_milp` and a `TopologicalAnalysisResult` from `analyse_topology`. The count `newConnected` is not stable across runs; see [Gap-filling](../guide/gap-filling.md). +- `fillGaps` with `useModelConstraints=true` fills so that the constraints of the model hold. `connect_blocked_reactions` has no such mode, and its documentation points to `cobra.flux_analysis.gapfill` for that use. The closest function for a growth target is `fill_gaps_milp`. +- `fillGaps` takes a cell array of template models, but the RAVEN functions `gapFillFastLP`, `gapFillMILP` and `gapFillTopological` use only the first template. The four raven-toolbox functions accept several templates. +- `fillGaps` returns `newConnected`, `cannotConnect`, `addedRxns`, `newModel` and `exitFlag`. raven-toolbox returns a `GapFillResult` from `connect_blocked_reactions`, a `FastLPResult` from `fill_gaps_fast_lp`, a `MILPGapFillResult` from `fill_gaps_milp` and a `TopologicalAnalysisResult` from `analyse_topology`. The count `newConnected` is not stable across runs (see [Gap-filling](../guide/gap-filling.md)). The RAVEN functions `canExchange`, `checkProduction` and `getAllSubGraphs` find gaps without filling them. raven-toolbox merges them into `gap_report`, the port of `gapReport`, and has no separate public function for each of them. ## Alternative optima of mixed-integer problems -ftINIT extraction, the reference and MILP gap-filling algorithms and compartment assignment (`assignCompartments` and `assign_compartments`) solve mixed-integer problems that can have several optima of equal objective value. The linear-programming algorithms were not measured for this, and the documentation of `swiftLP` in both implementations calls it stochastic. A solve that stops at its gap or time limit can also end at a different objective value. Two runs can return different reaction sets, and no single exact result is expected. +The extraction with ftINIT, the reference and MILP gap-filling algorithms, and the assignment of compartments (`assignCompartments` and `assign_compartments`) solve mixed-integer problems, which can have several optima with the same objective value. The LP algorithms were not measured for this, and the documentation of `swiftLP` in both implementations calls it stochastic. A solve that stops at its gap or time limit can also end at another objective value. Two runs can therefore return different reaction sets, and no single exact result is expected. -Measurements of the differences: +Four measurements quantify the differences. -- Between implementations: ftINIT models of five cell lines, extracted from the same Human-GEM inputs with RAVEN and raven-toolbox, share Jaccard indices of 0.964 to 0.988 over their reaction sets (see [Human-GEM cell-type model validation against RAVEN](../parameter-tuning/studies/humangem-validation.md)). The template preparation differed between the two runs. -- Between solvers: in the raven-toolbox ftINIT benchmark on Human-GEM with one run per solver, Gurobi extracted 7312 reactions and GLPK 6758 (Jaccard index 0.892). Both main steps of GLPK stopped at the 900 s limit with an unproven incumbent, so the two models differ in objective value as well as in tied optima (see [ftINIT on different solvers](../parameter-tuning/studies/init-solver-benchmark.md)). RAVEN was not run on several solvers in that study. -- Between repeated runs: two repeated raven-toolbox builds at one seed with the default settings differed by 2 reactions. Each build had one run of the final step that stopped at its 5 s wall-clock limit, the presumed and untested cause (see [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](../parameter-tuning/studies/ftinit-determinism.md)). RAVEN's `ftINIT` was not run for the effects of seed, thread count and time limit. -- Between thread counts: in raven-toolbox's first ftINIT step without `resolve_ties`, with six seeds, the solutions at `Threads` 1 and 4 for the same seed were 5 to 20 reactions apart, and in 5 of the 6 seeds the two runs reached different objective values (see [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](../parameter-tuning/studies/ftinit-determinism.md)). +- **Between implementations.** The ftINIT models of five cell lines, extracted from the same Human-GEM inputs with RAVEN and raven-toolbox, share Jaccard indices of 0.964 to 0.988 over their reaction sets (see [Human-GEM cell-type model validation against RAVEN](../parameter-tuning/studies/humangem-validation.md)). The template preparation differed between the two runs. +- **Between solvers.** In the raven-toolbox ftINIT benchmark on Human-GEM with one run per solver, Gurobi extracted 7312 reactions and GLPK 6758 (Jaccard index 0.892). Both main steps of GLPK stopped at the 900 s limit with an unproven incumbent, so the two models differ in objective value as well as in tied optima (see [ftINIT on different solvers](../parameter-tuning/studies/init-solver-benchmark.md)). RAVEN was not run on several solvers in that study. +- **Between repeated runs.** Two repeated raven-toolbox builds at one seed with the default settings differed by 2 reactions. In each build one run of the final step stopped at its 5 s wall-clock limit, which is the presumed and untested cause (see [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](../parameter-tuning/studies/ftinit-determinism.md)). The effects of seed, thread count and time limit were not measured for the `ftINIT` of RAVEN. +- **Between thread counts.** In the first ftINIT step of raven-toolbox without `resolve_ties`, with six seeds, the solutions at `Threads` 1 and 4 for the same seed were 5 to 20 reactions apart. In 5 of the 6 seeds the two runs reached different objective values (see [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](../parameter-tuning/studies/ftinit-determinism.md)). -Compare outputs by the overlap of their reaction sets, for example with the Jaccard index. +The overlap of the reaction sets, for example the Jaccard index, is a measure for comparing such outputs. -`resolveTies` in RAVEN's `ftINIT` and `resolve_ties` in raven-toolbox's `ftinit` are off by default in both. When on, each step keeps the score optimum and picks among the tied solutions in phases: the fewest kept removable reactions, then the lowest summed ID rank of those, then the least total absolute flux, then the least ID-rank-weighted backward flux. The last step of `ftINIT` skips the two flux phases. A step that stops at its time limit keeps the solver's own answer. +The option `resolveTies` of the `ftINIT` of RAVEN and `resolve_ties` of the `ftinit` of raven-toolbox are off by default in both. When on, each step keeps the score optimum and picks among the tied solutions in phases: the fewest kept removable reactions, then the lowest summed ID rank of those, then the least total absolute flux, then the least backward flux weighted by ID rank. The last step of `ftINIT` skips the two flux phases. A step that stops at its time limit keeps the answer of the solver. -In raven-toolbox, runs with `resolve_ties` at the same score optimum returned identical solutions across seeds. The primary solve can still end at a different objective value for a different seed, within the gap and time limits, so the final reaction sets can differ. This was not measured for RAVEN. Without `resolveTies` or `resolve_ties`, the ftINIT MILP can return any of the tied optima. +In raven-toolbox, runs with `resolve_ties` at the same score optimum returned identical solutions across seeds. The primary solve can still end at another objective value for another seed, within the gap and time limits, so the final reaction sets can differ. This was not measured for RAVEN. Without `resolveTies` or `resolve_ties`, the ftINIT MILP can return any of the tied optima. ## Elemental balance results -`getElementalBalance` in RAVEN and `get_elemental_balance` in raven-toolbox check whether each element has the same count on both sides of a reaction. A reaction is unbalanced when the counts were computed and did not match, and the result is unknown when the counts cannot be computed. The two functions use different codes for these results. +The functions `getElementalBalance` in RAVEN and `get_elemental_balance` in raven-toolbox check whether each element has the same count on both sides of a reaction. A reaction is unbalanced when the counts were computed and do not match. The result is unknown when the counts cannot be computed. The two functions code these results differently. -`getElementalBalance` returns the numeric field `balanceStructure.balanceStatus` per reaction: 1 for balanced, 0 for unbalanced, -1 for missing information (no formula, or a reaction without metabolites) and -2 for a formula that cannot be parsed. `get_elemental_balance` returns the strings `balanced`, `unbalanced` and `unknown`. It reports `unknown` for a missing formula, for an unparseable formula and for a reaction without metabolites, so the codes -1 and -2 of RAVEN are one result in raven-toolbox. +The function `getElementalBalance` returns the numeric field `balanceStructure.balanceStatus` per reaction: 1 for balanced, 0 for unbalanced, -1 for missing information (no formula, or a reaction without metabolites) and -2 for a formula that cannot be parsed. The function `get_elemental_balance` returns the strings `balanced`, `unbalanced` and `unknown`. It reports `unknown` for a missing formula, for a formula that cannot be parsed and for a reaction without metabolites, so the codes -1 and -2 of RAVEN are one result in raven-toolbox. -The two parsers differ for some formulas, so the results differ for some reactions with formulas: +The two parsers handle some formulas differently, so the results differ for some reactions. - A polymer formula such as `(C5H8)n` is counted with n equal to 1 in RAVEN. raven-toolbox reports `unknown`. -- RAVEN uses the `inchis` field when `metFormulas` is absent. raven-toolbox does not. +- RAVEN uses the field `inchis` when `metFormulas` is absent. raven-toolbox does not. - RAVEN reports a charge balance (`chargeStatus` and `chargeResidual`). `get_elemental_balance` reports elements only. -The comparison used the code only; the functions were not run on a shared model. +The comparison used the code only, and the functions were not run on a shared model. ## Parsing of gene-reaction rules -The section applies when a RAVEN user takes a model through cobrapy readers in Python. raven-toolbox parses gene-reaction rules (the `grRules` field in RAVEN) with `utils.parse_gr_rule`, which follows the grammar of RAVEN's `parseGrRule`. The function is used in `read_yaml_model`, `read_matlab_model`, `batch_curate` and the functions that copy or build rules, so `G1&G2` is one gene ID there. According to the raven-toolbox changelog, the JSON reader of cobrapy splits `G1&G2` into `G1` and `G2`, and the SBML reader keeps it. This was not checked against cobrapy itself. +The parsing of rules matters when a RAVEN user takes a model through cobrapy readers in Python. raven-toolbox parses gene-reaction rules (the field `grRules` in RAVEN) with `utils.parse_gr_rule`, which follows the grammar of `parseGrRule` of RAVEN. The function is used in `read_yaml_model`, `read_matlab_model`, `batch_curate` and the functions that copy or build rules, so `G1&G2` is one gene ID there. According to the changelog of raven-toolbox, the JSON reader of cobrapy splits `G1&G2` into `G1` and `G2`, and the SBML reader keeps it as one ID. This was not checked against cobrapy itself. diff --git a/docs/raven3-vs-raven-toolbox/what-each-has.md b/docs/raven3-vs-raven-toolbox/what-each-has.md index abe6111..fe62b3f 100644 --- a/docs/raven3-vs-raven-toolbox/what-each-has.md +++ b/docs/raven3-vs-raven-toolbox/what-each-has.md @@ -1,54 +1,59 @@ # Features specific to each implementation -The page describes the features that exist in raven-toolbox only and the features that exist in RAVEN only. +Some features exist in raven-toolbox only, and some exist in RAVEN only. The sections below describe both groups. ## Features specific to raven-toolbox ### Model type -A raven-toolbox model is a `cobra.Model`. Every cobrapy function and cobrapy-based package applies to it without conversion. RAVEN uses a MATLAB struct as the model. +A raven-toolbox model is a `cobra.Model`. Every function of cobrapy and of cobrapy-based packages applies to it without conversion. RAVEN uses a MATLAB struct as the model. -cobrapy provides flux balance analysis (FBA), flux variability analysis (FVA), knockouts and media handling. The [MATLAB vs Python](../matlab-vs-python.md) page gives a cobrapy entry for the corresponding RAVEN functions where cobrapy covers them; [Name differences](../raven3-vs-raven-toolbox.md#name-differences) describes the page. `setExchangeBounds` has two rows in the mapping: `set_exchange_bounds` in raven-toolbox and `cobra.Model.medium` in cobrapy. +cobrapy provides flux balance analysis (FBA), flux variability analysis (FVA), knockouts and media handling. For the functions of RAVEN that cobrapy covers, the page [MATLAB vs Python](../matlab-vs-python.md) gives a cobrapy entry (see [Name differences](../raven3-vs-raven-toolbox.md#name-differences)). The function `setExchangeBounds` has two rows in the mapping: `set_exchange_bounds` in raven-toolbox and `cobra.Model.medium` in cobrapy. ### KEGG artefact generation -raven-toolbox builds the KEGG reference artefacts: it parses a KEGG release into reaction and compound tables, assembles the reference model with `build_reference_model`, builds the per-KEGG Orthology (KO) FASTA sets and HMM libraries, and derives the phylogenetic distance matrix. The download of a KEGG release requires a KEGG FTP subscription. The artefacts can be rebuilt from a specified KEGG release. +raven-toolbox builds the KEGG reference artefacts. It parses a KEGG release into tables of reactions and compounds, assembles the reference model with `build_reference_model`, builds the FASTA sets and HMM libraries per KEGG Orthology (KO) entry, and derives the phylogenetic distance matrix. The download of a KEGG release requires a KEGG FTP subscription, and the artefacts can be rebuilt from a specified KEGG release. -RAVEN loads published artefacts and does not produce them. `buildGlobalKEGGModel` joins a downloaded bundle onto the reference model, and the phylogenetic distance matrix is a file shipped with RAVEN (`keggPhylDist.mat`). In the mapping, `getModelFromKEGG` corresponds to `build_reference_model`, which loads in RAVEN and builds in raven-toolbox. +RAVEN loads published artefacts and does not produce them. The function `buildGlobalKEGGModel` joins a downloaded bundle onto the reference model, and the phylogenetic distance matrix is a file shipped with RAVEN (`keggPhylDist.mat`). In the mapping, `getModelFromKEGG` corresponds to `build_reference_model`, which loads in RAVEN and builds in raven-toolbox. ### Checksum registry for binaries and data -BLAST+, DIAMOND, HMMER and the KEGG tables are fetched on demand in both implementations and are not bundled. RAVEN bundles other components: the GLPK and libSBML MEX files and `keggPhylDist.mat`. [Download data and binaries](../installation/data-and-binaries.md) describes the download behaviour. +Both implementations fetch BLAST+, DIAMOND, HMMER and the KEGG tables on demand and do not bundle them. RAVEN bundles other components: the GLPK and libSBML MEX files and `keggPhylDist.mat`. [Download data and binaries](../installation/data-and-binaries.md) describes the download behaviour. -The checksum handling differs: +The checksum handling differs. -- raven-toolbox keeps the checksum and release of each dataset and binary in a registry in its source (a version-pinned registry). A fresh download is checked against its checksum. A cached data file is reused without a check unless `ensure_data_file` is called with `verify=True`, which re-checks it and downloads it again on a mismatch; a cached binary is reused without a check. A JSON manifest can be merged into the registries with `raven_toolbox.manifest.load_into_registries`, which reads the path or URL in the environment variable `RAVEN_PYTHON_MANIFEST` when no source is passed. The resolvers load that variable by themselves only while the registry is empty, so with the filled default registries the call is needed. +- raven-toolbox keeps the checksum and the release of each dataset and binary in a registry in its source (a version-pinned registry). A fresh download is checked against its checksum. A cached data file is reused without a check, unless `ensure_data_file` is called with `verify=True`, which checks it again and downloads it again on a mismatch. A cached binary is reused without a check. A JSON manifest can be merged into the registries with `raven_toolbox.manifest.load_into_registries`, which reads the path or URL in the environment variable `RAVEN_PYTHON_MANIFEST` when no source is passed. The resolvers load that variable by themselves only while the registry is empty, so the call is needed with the filled default registries. - RAVEN checks a download against the checksum in the published manifest. A mismatch deletes the file and raises an error. When the manifest cannot be read or has no entry for the asset, RAVEN issues a warning and keeps the file unchecked. A cached file is reused without a check. ### Further functions -Further raven-toolbox features have no RAVEN counterpart in the generated mapping. Examples are `localization.relocate_reactions`, which places reactions in compartments chosen by a curator and repairs the model around the decision, `localization.triage_localization`, and the per-reaction confidence record of the `confidence` module, which scores the facets localization, equation and gene association and stores them in the reaction notes. +Further features of raven-toolbox have no RAVEN counterpart in the generated mapping. Examples are `localization.relocate_reactions`, which places reactions in compartments chosen by a curator and repairs the model around that decision; `localization.triage_localization`; and the per-reaction confidence record of the `confidence` module, which scores the facets localization, equation and gene association and stores the scores in the reaction notes. ## Features specific to RAVEN ### tINIT -`getINITModel` and `runINIT` are the original tINIT implementation. RAVEN keeps them for models built with them. raven-toolbox implements ftINIT only, as a design decision. A tINIT model has to be reproduced in MATLAB; see [Context-specific models with ftINIT](../guide/init.md). +The functions `getINITModel` and `runINIT` are the original tINIT implementation, and RAVEN keeps them for models built with them. raven-toolbox implements ftINIT only, as a design decision. A tINIT model has to be reproduced in MATLAB (see [Context-specific models with ftINIT](../guide/init.md)). ### COBRA Toolbox conversion -`ravenCobraWrapper` converts a model in memory between the RAVEN and COBRA Toolbox structures. A round trip drops some fields. In raven-toolbox the model is a `cobra.Model`, so no in-memory conversion is needed and no equivalent exists. File exchange is described in [Choosing an implementation](../raven3-vs-raven-toolbox.md#choosing-an-implementation); cobrapy writes the COBRA layout. +The function `ravenCobraWrapper` converts a model in memory between the RAVEN structure and the COBRA Toolbox structure, and a round trip drops some fields. The model of raven-toolbox is a `cobra.Model`, so no in-memory conversion is needed and no equivalent exists. [Choosing an implementation](../raven3-vs-raven-toolbox.md#choosing-an-implementation) describes the exchange by file, and cobrapy writes the COBRA layout. ### Dynamic FBA -`runDynamicFBA` has no raven-toolbox counterpart by design. Python packages cover dynamic FBA ([dfba](https://pypi.org/project/dfba/), [reframed](https://pypi.org/project/reframed/), [mewpy](https://pypi.org/project/mewpy/)); the packages were not checked here. +`runDynamicFBA` has no raven-toolbox counterpart by design. Python packages cover dynamic FBA: [dfba](https://pypi.org/project/dfba/), [reframed](https://pypi.org/project/reframed/) and [mewpy](https://pypi.org/project/mewpy/). The packages were not checked here. ### MATLAB-specific functions -RAVEN has MATLAB-specific infrastructure that raven-toolbox does not port: path management (`addRavenToUserPath`, `findRAVENroot`), argument handling (`parseRAVENargs`, `convertCharArray`), progress and printing (`setRavenProgress`, `printOrange`) and the solver abstraction (`optimizeProb`, `setRavenSolver`). cobrapy's solver interface (optlang) takes the place of the solver abstraction. +RAVEN has MATLAB-specific infrastructure that raven-toolbox does not port: -The generated mapping table lists the functions in its tracked categories. `addRavenToUserPath` and `findRAVENroot` belong to the installation folder, which is not tracked, so the table does not list them. The "Deliberately not ported" table lists further infrastructure functions, for example `checkSolution`, `permuteModel`, `parseYAML` and `deprecationWarning`. +- path management (`addRavenToUserPath`, `findRAVENroot`); +- argument handling (`parseRAVENargs`, `convertCharArray`); +- progress and printing (`setRavenProgress`, `printOrange`); +- the solver abstraction (`optimizeProb`, `setRavenSolver`), which the solver interface of cobrapy (optlang) replaces. + +The generated mapping table lists the functions of its tracked categories. `addRavenToUserPath` and `findRAVENroot` belong to the installation folder, which is not tracked, so the table does not list them. The table "Deliberately not ported" lists further infrastructure functions, for example `checkSolution`, `permuteModel`, `parseYAML` and `deprecationWarning`. ### Functions without a Python counterpart -Some RAVEN functions that raven-toolbox does not port are not infrastructure. `getTransportRxns`, `runRobustnessAnalysis` and `runSimpleOptKnock` have no replacement and can block a port. `readFasta` has no public reader, and `findPotentialErrors` has no step that detects an invalid rule string, because the cobrapy GPR class represents a parsed rule directly. The table gives other replacements: the `.annotation` dictionary of cobrapy for `editMiriam` and `extractMiriam`, the `cobra.Reaction` API for `reverseRxns` and `model.summary()` for `printFluxes`. The "Deliberately not ported" table of the [MATLAB vs Python](../matlab-vs-python.md) page is the reference. +Some functions that raven-toolbox does not port are not infrastructure. Three have no replacement and can block a port: `getTransportRxns`, `runRobustnessAnalysis` and `runSimpleOptKnock`. The function `readFasta` has no public reader, and `findPotentialErrors` has no step that detects an invalid rule string, because the GPR class of cobrapy represents a parsed rule directly. For other functions the table gives a replacement: the `.annotation` dictionary of cobrapy for `editMiriam` and `extractMiriam`, the `cobra.Reaction` API for `reverseRxns`, and `model.summary()` for `printFluxes`. The table "Deliberately not ported" on the [MATLAB vs Python](../matlab-vs-python.md) page is the reference. From 597dbb1c0d1352acead402c8369af484a6987891 Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Sun, 1 Nov 2026 11:45:30 +0100 Subject: [PATCH 11/20] Write the benchmarks page anew --- docs/parameter-tuning/benchmarks.md | 79 ++++++++++++++++------------- 1 file changed, 45 insertions(+), 34 deletions(-) diff --git a/docs/parameter-tuning/benchmarks.md b/docs/parameter-tuning/benchmarks.md index b9a3000..dfc6e38 100644 --- a/docs/parameter-tuning/benchmarks.md +++ b/docs/parameter-tuning/benchmarks.md @@ -1,27 +1,42 @@ # Benchmarks of `fseof`, `remove_genes` and other defaults -These benchmarks record parameters whose defaults were measured or checked without a study of their own. All were run with raven-toolbox commit `94f39407` on 2026-11-01: the effect of `flux_eps` on the targets of `fseof`, the handling of reactions without an enzyme in `remove_genes`, a sweep of three `fseof` parameters and a test of `connect_blocked_reactions.threshold`. A table lists literature and convention values with what supports each of them. +Four defaults of raven-toolbox were measured without a study of their own: `flux_eps` of `fseof`, the sweep of `n_steps`, `max_fraction` and `correlation_threshold` of `fseof`, the handling of reactions without an enzyme in `remove_genes`, and `threshold` of `connect_blocked_reactions`. A table at the end lists further defaults that rest on the literature or on a convention. All runs used raven-toolbox commit `94f39407` and were made on 2026-11-01. -The parameters with a measurement campaign of their own have studies listed under [Parameter tuning](index.md). [Tuned parameter defaults](../tuned-parameters.md) gives the current value and a one-line reason for every parameter of both implementations. +Parameters with a measurement campaign of their own have a study listed under [Parameter tuning](index.md). [Tuned parameter defaults](../tuned-parameters.md) gives the current value of every parameter of both implementations with a one-line reason. ## Set-up -The `fseof` benchmarks ran on iJO1366 (2,583 reactions) with the biomass reaction of the model as objective and `EX_succ_e` (succinate exchange) as target. In the `flux_eps` table the enforced flux ran from 1.5387 to 15.387 in 10 levels, and the unconstrained growth rate was 0.98237 h⁻¹. The other arguments were `n_steps` 10, `max_fraction` 0.9, `correlation_threshold` 0.9 and the default `equality=False`, which sets each enforced flux as the lower bound of the target reaction. +### `fseof` -`fseof` solves each enforced level in two steps, as RAVEN's `FSEOF` does: it maximises the objective, then minimises the sum of fluxes with the objective held at 0.999999 of that optimum. A reaction is a target when its absolute flux exceeds `1e-10` at some level, the standard deviation of its absolute flux is above 0, the absolute correlation between its absolute flux and the enforced flux reaches `correlation_threshold`, and the slope of the linear fit of its absolute flux against the enforced flux is not 0. A target with a positive slope is an amplify target. A target with a negative slope is a knockout target when its last absolute flux is at most `1e-8`, and a knockdown target otherwise. `fseof` and RAVEN's `FSEOF` use these thresholds as fixed values. +The `fseof` benchmarks ran on iJO1366 (2,583 reactions) with the biomass reaction of the model as the objective and `EX_succ_e` (succinate exchange) as the target. The enforced flux ran from 1.5387 to 15.387 in 10 levels, and the unconstrained growth rate was 0.98237 h⁻¹. The other arguments were `n_steps` 10, `max_fraction` 0.9, `correlation_threshold` 0.9 and the default `equality=False`, which sets each enforced flux as the lower bound of the target reaction. -The `remove_genes` benchmark ran on e_coli_core (95 reactions) from the cobrapy distribution, with bounds as shipped, raven-toolbox commit `94f39407` and RAVEN commit `2d0630e3`. It deleted `b1779`, the only gene of GAPD (glyceraldehyde-3-phosphate dehydrogenase), with each setting of `blocked_reactions` in `remove_genes` and of `removeBlockedRxns` in RAVEN's `removeGenes`. The predicted growth is the optimal flux through the biomass reaction of the model, in h⁻¹. The unmodified model has 95 reactions and predicts a growth rate of 0.874 h⁻¹. Each call ran once. +At each enforced level `fseof` solves two problems, as `FSEOF` of RAVEN does. It first maximises the objective and then minimises the sum of fluxes with the objective held at 0.999999 of that optimum. A reaction is a target when four conditions hold: -The Python runs used Python 3.14.2, cobra 0.32.1, optlang 1.9.1 and Gurobi 13.0.3 (gurobipy wheel); MATLAB R2024b ran RAVEN with Gurobi 13.0.1 selected through `setRavenSolver('gurobi')`. The `flux_eps` runs were repeated three times per value, each on a freshly read model, and gave the same target set and labels every time. +- its absolute flux exceeds `1e-10` at some level; +- the standard deviation of its absolute flux is above 0; +- the absolute correlation between its absolute flux and the enforced flux reaches `correlation_threshold`; +- the slope of the linear fit of its absolute flux against the enforced flux is not 0. -The sweep of `fseof` ran on iJO1366 with the settings above and Gurobi 13.0.3, changing one of `n_steps`, `max_fraction` and `correlation_threshold` at a time, with three runs per value on freshly read models. All three runs of every configuration gave the same counts and the same added and dropped targets. Two categories of target are reported, and neither measures error: +A target with a positive slope is an amplify target. A target with a negative slope is a knockout target when its last absolute flux is at most `1e-8`, and a knockdown target otherwise. `fseof` and `FSEOF` of RAVEN use these thresholds as fixed values. -1. **Added:** the target appears at the swept value and not in the default run (the set difference from the default). An added target is no evidence of error. -2. **Noise:** the standard deviation of its absolute flux across the scan is below `1e-6`. +The sweep changed one of `n_steps`, `max_fraction` and `correlation_threshold` at a time, with three runs per value on freshly read models. Every configuration gave the same counts and the same added and dropped targets in all three runs. The sweep reports two categories of target, and neither measures error: -The default run contains 19 noise tracks, all knockdown targets with a standard deviation of 5.08e-7, a slope of -1.15e-7 and a correlation of -1: `EGMEACPR`, `OGMEACPD`, `OPMEACPR`, `OPMEACPS`, `MALCOAMT`, `OGMEACPR`, `OGMEACPS`, `DM_amob_c`, `EX_meoh_e`, `AMAOTr`, `AOXSr2`, `BTS5`, `DBTS`, `EPMEACPR`, `I2FE2SR`, `MEOHtex`, `MEOHtrpp`, `OPMEACPD` and `PMEACPE` (biotin synthesis and methanol export). The classification rule above keeps them, because it requires a standard deviation above 0. +1. **Added:** the target appears at the swept value and not in the default run, which is the set difference from the default. An added target is no evidence of error. +2. **Noise:** the standard deviation of the absolute flux of the target across the scan is below `1e-6`. -The test of `connect_blocked_reactions.threshold` used a synthetic network: a draft with an import of A of at most 0.5, the reaction A to B, and B as a dead end, and a template with the exchange `EX_B`. The other arguments were at their defaults (`min_flux` 0.1). Each value ran once, with Gurobi 13.0.3. +The default run contains 19 noise tracks. All are knockdown targets with a standard deviation of 5.08e-7, a slope of -1.15e-7 and a correlation of -1: `EGMEACPR`, `OGMEACPD`, `OPMEACPR`, `OPMEACPS`, `MALCOAMT`, `OGMEACPR`, `OGMEACPS`, `DM_amob_c`, `EX_meoh_e`, `AMAOTr`, `AOXSr2`, `BTS5`, `DBTS`, `EPMEACPR`, `I2FE2SR`, `MEOHtex`, `MEOHtrpp`, `OPMEACPD` and `PMEACPE` (biotin synthesis and methanol export). The classification keeps them, because it requires a standard deviation above 0. + +### `remove_genes` + +The `remove_genes` benchmark ran on e_coli_core (95 reactions) from the cobrapy distribution, with the bounds as shipped, raven-toolbox commit `94f39407` and RAVEN commit `2d0630e3`. It deleted `b1779`, the only gene of GAPD (glyceraldehyde-3-phosphate dehydrogenase), with each setting of `blocked_reactions` in `remove_genes` and of `removeBlockedRxns` in `removeGenes` of RAVEN. The predicted growth is the optimal flux through the biomass reaction of the model, in h⁻¹. The unmodified model has 95 reactions and predicts a growth rate of 0.874 h⁻¹. Each call ran once. + +### `connect_blocked_reactions` + +The test of `threshold` used a synthetic network. The draft has an import of A of at most 0.5, the reaction A to B, and B as a dead end, and the template has the exchange `EX_B`. The other arguments were at their defaults (`min_flux` 0.1). Each value ran once. + +### Software + +The Python runs used Python 3.14.2, cobra 0.32.1, optlang 1.9.1 and Gurobi 13.0.3 (gurobipy wheel). The MATLAB run used R2024b with RAVEN and Gurobi 13.0.1, selected through `setRavenSolver('gurobi')`. The `flux_eps` runs were repeated three times per value, each on a freshly read model, and gave the same target set and labels every time. ## Results @@ -35,9 +50,11 @@ The test of `connect_blocked_reactions.threshold` used a synthetic network: a dr Of the 408 reactions, 406 are knockdown targets and 2 are knockout targets. Three runs per value gave the same set of targets. +The parameter `flux_eps` is the floor below which `fseof` treats the theoretical maximum of the target flux as zero, and the scan then raises an error. The classification of targets does not read `flux_eps`, and `FSEOF` of RAVEN has no such parameter. From the code, two values of `flux_eps` give different results only when the theoretical maximum of the target lies between them: `fseof` raises at the larger value and runs at the smaller. + ### `fseof` targets at other values of three parameters -iJO1366, target `EX_succ_e`. The last two columns list the targets that the swept value adds to the default set and drops from it. +The table lists iJO1366 with the target `EX_succ_e`. The last two columns give the targets that the swept value adds to the default set and drops from it. | Parameter | Value | Amplify targets | Knockdown and knockout targets | Added (amplify / knockdown and knockout) | Dropped (amplify / knockdown and knockout) | |---|---|---|---|---|---| @@ -49,7 +66,7 @@ iJO1366, target `EX_succ_e`. The last two columns list the targets that the swep | `correlation_threshold` | 0.7 | 22 | 416 | 4 / 8 (`FRD3`, `NADH18pp`, `ICL`, `MALS`; `EX_co2_e`, `CO2tex`, `CO2tpp`, `ICDHyr`, `AKGDH`, `PGI`, `TKT2`, `GLYCTO2`) | 0 / 0 | | `correlation_threshold` | 0.95 | 16 | 406 | 0 / 0 | 2 / 2 (`FUM`, `MDH`; `PDH`, `RPE`) | -Every swept value changes the target set. None of the added targets is a noise track: they are added, and the noise category does not apply to them. The four added amplify targets at `correlation_threshold` 0.7 have an absolute correlation of 0.71 to 0.72 in the default scan and a standard deviation of 0.62 to 4.4. Whether any added target is biologically wrong was not assessed. At `max_fraction` 0.5, `FBA3` and `PFK_3` change from knockout to knockdown targets; at 0.99, the 19 noise targets change from knockdown to knockout targets. +Every swept value changes the target set. None of the added targets is a noise track, because they are added relative to the default and the noise category does not apply to them. The four added amplify targets at `correlation_threshold` 0.7 have an absolute correlation of 0.71 to 0.72 in the default scan and a standard deviation of 0.62 to 4.4. At `max_fraction` 0.5, `FBA3` and `PFK_3` change from knockout to knockdown targets. At 0.99 the 19 noise targets change from knockdown to knockout targets. Whether any added target is biologically wrong was not assessed. ### `remove_genes` after deletion of `b1779` from e_coli_core @@ -63,9 +80,11 @@ Every swept value changes the target set. None of the added targets is a noise t | `removeGenes(model, genes, 'removeBlockedRxns', true)` | 94 | 0.000 | deleted | | unmodified model | 95 | 0.874 (MATLAB biomass flux 0.8739) | present | +The parameter `blocked_reactions` of `remove_genes` and `removeBlockedRxns` of `removeGenes` decide what happens to a reaction whose gene rule is emptied by a deletion. Both defaults keep such a reaction with bounds of zero, and the two defaults gave the same model. After the deletion of `b1779` that model predicts no growth, because GAPD stays in the network without capacity to carry flux. The setting `"remove"` and `removeBlockedRxns` true also predicted no growth. They delete GAPD, which changes the number of reactions and the set of reaction identifiers. The setting `"keep"` leaves GAPD with an empty gene rule and unchanged bounds, so the model predicted the growth of the unmodified model, 0.874 h⁻¹, although the only gene of GAPD was deleted. GAPD is essential for aerobic growth on glucose in e_coli_core, so the growth predicted with `"keep"` is a false positive. The docstring of `remove_genes` states no use for `"keep"`, and the documentation names the editing of gene annotations without changing the network as a use, without a source in the code. + ### `connect_blocked_reactions` at four values of `threshold` -Synthetic network described in [Set-up](#set-up). +The network is the synthetic network of the set-up. | `threshold` | Gap filled | Reactions added | Maximum flux of the reaction A to B afterwards | |---|---|---|---| @@ -74,40 +93,32 @@ Synthetic network described in [Set-up](#set-up). | 0.5 | no | none | 0 | | 1.0 (default) | no | none | 0 | -At every value, `cannot_connect` also lists the reversible `EX_A`, which is blocked in the draft and is never a candidate. +At every value `cannot_connect` also lists the reversible `EX_A`, which is blocked in the draft and is never a candidate. ### Checked literature and convention values -Both implementations have the same value or behaviour for each of these parameters, and none was changed. The third column gives what supports the value; the three `fseof` rows and the `connect_blocked_reactions.threshold` row record a measurement. +Both implementations have the same value or behaviour for each of these parameters, and none was changed. The third column gives what supports the value. The three `fseof` rows and the `connect_blocked_reactions.threshold` row record a measurement, and the other rows were not run. | Parameter | Value | Basis | |---|---|---| -| `fseof.n_steps` | `10` | Value of [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10) (value not checked against the paper); on iJO1366, 5 steps gave 18 amplify and 410 knockdown and knockout targets and 20 steps gave 16 and 412, so the target set changes with `n_steps` | -| `fseof.max_fraction` | `0.9` | [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10) (value not checked against the paper); on iJO1366, 0.5 added 7 targets and dropped 4, and 0.99 gave the same counts as 0.9 with one target added and one dropped | -| `fseof.correlation_threshold` | `0.9` | [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10) (value not checked against the paper); on iJO1366, 0.7 added 4 amplify and 8 knockdown and knockout targets, and 0.95 dropped 2 and 2 | -| `connect_blocked_reactions.threshold` | `1.0` | Suited to models in the RAVEN convention, where exchange reactions have bounds of ±1000; on one synthetic network with a maximum flux of 0.5, values of 0.01 and 0.1 filled the gap and values of 0.5 and 1.0 did not (one run per value; the test shows that a reaction must carry more than the threshold, and no correctness comparison was made) | -| `connect_blocked_reactions.allow_net_production` | `False` | Keeps the steady-state balance `S @ v == 0`, the stricter test; `True` relaxes it to `S @ v >= 0` | +| `fseof.n_steps` | `10` | Value of [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10) (value not checked against the paper). On iJO1366, 5 steps gave 18 amplify and 410 knockdown and knockout targets and 20 steps gave 16 and 412, so the target set changes with `n_steps` | +| `fseof.max_fraction` | `0.9` | [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10) (value not checked against the paper). On iJO1366, 0.5 added 7 targets and dropped 4, and 0.99 gave the same counts as 0.9 with one target added and one dropped | +| `fseof.correlation_threshold` | `0.9` | [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10) (value not checked against the paper). On iJO1366, 0.7 added 4 amplify and 8 knockdown and knockout targets, and 0.95 dropped 2 and 2 | +| `connect_blocked_reactions.threshold` | `1.0` | Suited to models in the RAVEN convention, where exchange reactions have bounds of ±1000. On one synthetic network with a maximum flux of 0.5, values of 0.01 and 0.1 filled the gap and values of 0.5 and 1.0 did not (one run per value). The test shows that a reaction must carry more than the threshold, and no comparison of correctness was made | +| `connect_blocked_reactions.allow_net_production` | `False` | Keeps the steady-state balance `S @ v == 0`, the stricter test. `True` relaxes it to `S @ v >= 0` | | `fill_gaps_fast_lp.epsilon` | `1e-4` | Value of the fastGapFill paper, [Thiele et al. 2014](https://doi.org/10.1093/bioinformatics/btu321) | | `fill_gaps_milp.big_m` | `1000.0` | Matches the ±1000 bounds of the RAVEN convention | -| `check_tasks.close_boundaries` | `False` | Matches RAVEN's `checkTasks`, against which published task lists were curated; `True` closes the exchange reactions of the model so that the task defines the whole boundary | - -The three `fseof` rows and the `connect_blocked_reactions.threshold` row ran with commit `94f39407`. The other rows were not run. - -## `flux_eps` in `fseof` - -`flux_eps` is the floor below which `fseof` treats the theoretical maximum of the target flux as zero; the scan then raises an error. The classification of targets does not read `flux_eps`, and RAVEN's `FSEOF` has no such parameter. On iJO1366 with target `EX_succ_e`, the three values gave 18 amplify targets and 408 knockdown and knockout targets each. From the code, two values of `flux_eps` give different results only when the theoretical maximum of the target lies between them: `fseof` then raises at the larger value and runs at the smaller. - -## Reactions without an enzyme in `remove_genes` - -`blocked_reactions` in `remove_genes`, and `removeBlockedRxns` in RAVEN's `removeGenes`, decide what happens to a reaction whose gene rule is emptied by a deletion. By default both keep such a reaction with bounds of zero, and the two defaults gave the same model. After deletion of `b1779`, that model predicts no growth, because GAPD stays in the network with no capacity to carry flux. `"remove"` and `removeBlockedRxns` true also predicted no growth; they delete GAPD, which changes the number of reactions and the set of reaction identifiers. `"keep"` leaves GAPD with an empty gene rule and unchanged bounds, so the model predicted the growth of the unmodified model, 0.874 h⁻¹, although the only gene of GAPD was deleted. GAPD is essential for aerobic growth on glucose in e_coli_core, so the growth predicted with `"keep"` is a false positive. The docstring of `remove_genes` states no use for `"keep"`. Editing gene annotations without changing the network is a use given for it in the documentation, without a source in the code. +| `check_tasks.close_boundaries` | `False` | Matches `checkTasks` of RAVEN, against which published task lists were curated. `True` closes the exchange reactions of the model, so that the task defines the whole boundary | ## Choice of defaults -The measurement gives no reason to change `flux_eps` from `1e-6`. The sweep of `fseof` does not select the other three defaults: the target set changes with each swept value, by 2 to 12 targets, and no tested value leaves it unchanged. The defaults follow [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10) (not checked against the paper). The 19 noise tracks are in the target set at every setting. `blocked_reactions="constrain"` is the default of `remove_genes`; it gives the model that RAVEN's `removeGenes` gives by default. `"remove"` and `"keep"` are available on request. +The measurements give no reason to change `flux_eps` from `1e-6`. The sweep does not select the other three `fseof` defaults: the target set changes with each swept value by 2 to 12 targets, and no tested value leaves it unchanged. The defaults follow [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10) (not checked against the paper). The 19 noise tracks are in the target set at every setting. + +The default `blocked_reactions="constrain"` of `remove_genes` gives the model that `removeGenes` of RAVEN gives by default. The settings `"remove"` and `"keep"` are available on request. ## Limits -Each benchmark used one model: iJO1366 with one target for `fseof`, e_coli_core with one deleted gene for `remove_genes`, and a synthetic network for `connect_blocked_reactions`. Criterion 1 of the [evaluation methodology](index.md#evaluation-methodology) asks for a large and a small model. Gurobi was the sole solver. Whether the targets added by a swept value are biologically wrong was not assessed, and neither was the literature basis of the cited values. The rows `connect_blocked_reactions.allow_net_production`, `fill_gaps_fast_lp.epsilon`, `fill_gaps_milp.big_m` and `check_tasks.close_boundaries` rest on the literature or on a convention; no test was run for them. +Each benchmark used one model: iJO1366 with one target for `fseof`, e_coli_core with one deleted gene for `remove_genes`, and a synthetic network for `connect_blocked_reactions`. Criterion 1 of the [evaluation methodology](index.md#evaluation-methodology) asks for a large and a small model. Gurobi was the sole solver. Neither the biological correctness of the targets added by a swept value nor the literature basis of the cited values was assessed. The rows `connect_blocked_reactions.allow_net_production`, `fill_gaps_fast_lp.epsilon`, `fill_gaps_milp.big_m` and `check_tasks.close_boundaries` rest on the literature or on a convention, and no test was run for them. ## Source From eb936805c6b963e92f7a6af74111dcad0331fade Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Sun, 1 Nov 2026 11:46:02 +0100 Subject: [PATCH 12/20] Write the Human-GEM validation page anew --- .../studies/humangem-validation.md | 69 +++++++++++-------- 1 file changed, 42 insertions(+), 27 deletions(-) diff --git a/docs/parameter-tuning/studies/humangem-validation.md b/docs/parameter-tuning/studies/humangem-validation.md index b88a6ec..65fe53c 100644 --- a/docs/parameter-tuning/studies/humangem-validation.md +++ b/docs/parameter-tuning/studies/humangem-validation.md @@ -1,25 +1,38 @@ # Human-GEM cell-type model validation against RAVEN -raven-toolbox's ftINIT was run on Human-GEM with the RNA sequencing (RNA-seq) data of five cell lines from the Hart2015 dataset (DLD1, GBM, HCT116, HELA, RPE1), and each extracted cell-type model was compared with the model that MATLAB RAVEN's `ftINIT` extracted from the same input. The comparison records the reactions each model contains, the reactions the two models share and the reactions only one of them contains, and the time each step took. The question is whether raven-toolbox selects the same reactions as RAVEN. +The study asks whether the ftINIT of raven-toolbox selects the same reactions as the `ftINIT` of RAVEN. Both tools extracted a cell-type model from Human-GEM with the RNA sequencing (RNA-seq) data of five cell lines of the Hart2015 dataset (DLD1, GBM, HCT116, HELA, RPE1), and the models were compared by their reaction sets, together with the time of each step. + +With the task layer, the raven-toolbox models share a Jaccard index of 0.981 to 0.988 with the RAVEN models. Without the task layer the index is 0.964 to 0.982. ## Set-up -Software: Human-GEM commit `38567ab8` (v2.1.1 plus 15 commits), RAVEN commit `2d0630e3` on MATLAB 2024b, raven-toolbox commit `94f3940`, and Gurobi for both tools, with 4 cores per run. The Python runs load the Gurobi library 13.0.3 from the gurobipy wheel (most likely; the logs print no version), and the MATLAB side links to the Gurobi module library 13.0.1 (from the linkage of the MATLAB interface; MATLAB prints no version). +### Software + +The software was Human-GEM commit `38567ab8` (v2.1.1 plus 15 commits), RAVEN commit `2d0630e3` on MATLAB 2024b, raven-toolbox commit `94f3940`, and Gurobi for both tools, with 4 cores per run. The Python runs load the Gurobi library 13.0.3 from the gurobipy wheel (most likely, because the logs print no version). The MATLAB side links to the library 13.0.1 of the Gurobi module, as the linkage of the MATLAB interface shows; MATLAB prints no version. + +### Template preparation + +Template preparation runs once per template model, before any cell line. + +- **RAVEN.** The Human-GEM function `prepHumanModelForftINIT` removed the drug reactions, the amino-acid triplet reactions and the duplicate complex I reaction from Human-GEM, set the lists of spontaneous and custom reactions, and prepared the model with the essential tasks of `metabolicTasks_Essential.txt`. The prepared model, from which RAVEN extracts, has 9730 reactions. +- **raven-toolbox.** `prep_init_model` ran on the same cleaned Human-GEM (11265 reactions). Every reaction in every raven-toolbox result is in the prepared model of RAVEN, so the reaction identifiers of the two results are compared directly. -Template preparation is the step that runs once per template model, before any cell line. For RAVEN, the Human-GEM function `prepHumanModelForftINIT` removed the drug reactions, the amino-acid triplet reactions and the duplicate complex I reaction from Human-GEM, set the spontaneous and custom reaction lists, and prepared the model with the essential tasks of `metabolicTasks_Essential.txt`. RAVEN's prepared model, the model from which RAVEN extracts, has 9730 reactions. For raven-toolbox, `prep_init_model` ran on the same cleaned Human-GEM (11265 reactions). Every reaction in every raven-toolbox result is in RAVEN's prepared model, so the reaction identifiers of the two results are compared directly. +### Scoring and extraction -Scoring: gene scores are `5·ln(level/1)` of the transcripts per million (TPM) values in the Hart2015 file, clipped to [−5, 10], and reactions take their scores from the gene rules; a reaction without genes scores −2. raven-toolbox computes this with `gene_scores_from_expression` and `score_reactions_from_genes`, RAVEN with `transcrData` and `threshold=1`. +Gene scores are `5·ln(level/1)` of the transcripts per million (TPM) values in the Hart2015 file, clipped to [−5, 10]. Reactions take their scores from the gene rules, and a reaction without genes scores −2. raven-toolbox computes the scores with `gene_scores_from_expression` and `score_reactions_from_genes`, and RAVEN with `transcrData` and `threshold=1`. -Extraction: both tools ran the series `1+1`, two staged mixed-integer linear program (MILP) steps. RAVEN ran `ftINIT` with its default settings. raven-toolbox ran `raven_toolbox.init.ftinit` with series `1+1`, `time_limit=600` and the default `big_m=1000`. A named series runs its own schedule of MILP runs in each step, so `mip_gap` and `mip_gap_abs` have no effect on these runs. +Both tools ran the series `1+1`, two staged mixed-integer linear program (MILP) steps. RAVEN ran `ftINIT` with its default settings. raven-toolbox ran `raven_toolbox.init.ftinit` with series `1+1`, `time_limit=600` and the default `big_m=1000`. A named series runs its own schedule of MILP runs in each step, so `mip_gap` and `mip_gap_abs` have no effect on these runs. raven-toolbox ran in two variants: -- **No-task**: the template is prepared without a task list, and the extraction uses the expression scores alone. -- **Task-constrained**: the template is prepared with the essential metabolic tasks, the reactions that a task needs are kept whatever their score, and `ftinit` adds reactions back until every task is feasible (its default `fill_gaps=True`). +- **No-task.** The template is prepared without a task list, and the extraction uses the expression scores alone. +- **Task-constrained.** The template is prepared with the essential metabolic tasks, the reactions that a task needs are kept whatever their score, and `ftinit` adds reactions back until every task is feasible (its default `fill_gaps=True`). The RAVEN result is task-constrained in this sense. A third set of no-task runs used `big_m=100`. -Measurements: the result of a run is its set of reaction identifiers. For each cell line the comparison gives the number of reactions in both sets (shared), in one set only, and the Jaccard index, the shared count divided by the size of the union. The comparison covers reactions only; genes, essential genes and task performance of the extracted models were not compared. Each cell line, variant and `big_m` value ran once. Times are wall-clock times of single runs on shared nodes. +### Measurements + +The result of a run is its set of reaction identifiers. For each cell line the comparison gives the number of reactions in both sets (shared), the number in one set only, and the Jaccard index, which is the shared count divided by the size of the union. The comparison covers reactions only. Genes, essential genes and the task performance of the extracted models were not compared. Each cell line, variant and `big_m` value ran once, and the times are wall-clock times of single runs on shared nodes. ## Results @@ -28,7 +41,7 @@ Measurements: the result of a run is its set of reaction identifiers. For each c | Step | Time | |------|-----:| | RAVEN `prepHumanModelForftINIT` (4-worker parallel pool, once per template) | 4320 s | -| raven-toolbox `prep_init_model`, no-task (cobrapy's default process count for flux variability analysis, once per template) | 1556 s | +| raven-toolbox `prep_init_model`, no-task (default process count of cobrapy for flux variability analysis, once per template) | 1556 s | | raven-toolbox `prep_init_model`, task-constrained (`cobra.Configuration().processes = 1`, once per template) | 1695 s | | RAVEN `ftINIT`, per cell line | 40 to 91 s | | raven-toolbox `ftinit`, no-task, per cell line | 81 to 91 s | @@ -76,46 +89,48 @@ Measurements: the result of a run is its set of reaction identifiers. For each c The no-task `ftinit` runs took 83 to 122 s at `big_m=100` and 81 to 91 s at `big_m=1000`. -## Agreement with the task layer +## Interpretation + +### Agreement with the task layer -The task-constrained variant shares Jaccard 0.981 to 0.988 with RAVEN across the five cell lines. Its reaction counts are within 0.2 % of RAVEN's, and the differences run in both directions in similar numbers: 46 to 70 reactions only in the raven-toolbox model and 43 to 71 only in the RAVEN model, out of ~7300. +The task-constrained variant shares a Jaccard index of 0.981 to 0.988 with RAVEN across the five cell lines. Its reaction counts are within 0.2 % of the counts of RAVEN. The differences run in both directions in similar numbers: 46 to 70 reactions only in the raven-toolbox model and 43 to 71 only in the RAVEN model, out of about 7300. -Neither tool was run twice on the same input, so the comparison does not separate alternate optima of the MILP from a difference between the two implementations. For scale, two raven-toolbox runs that differed only in the seed differed by 110 reactions in [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](ftinit-determinism.md). That measurement used a different set-up (Human-GEM v2.0.0-38, DLD1, `big_m=100`, an escalation schedule), so it indicates the order of magnitude and does not bound the spread here. +Neither tool was run twice on the same input, so the comparison does not separate alternate optima of the MILP from a difference between the two implementations. For scale, two raven-toolbox runs that differed only in the seed differed by 110 reactions in [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](ftinit-determinism.md). That measurement used another set-up (Human-GEM v2.0.0-38, DLD1, `big_m=100`, an escalation schedule), so it indicates the order of magnitude and does not bound the spread here. -## Agreement without the task layer +### Agreement without the task layer -The no-task variant shares Jaccard 0.964 to 0.982 with RAVEN. It keeps fewer reactions than RAVEN in every cell line: 0.3 to 0.5 % fewer for four cell lines and 1.6 % fewer for RPE1. The reactions only in the RAVEN model (76 to 186) outnumber those only in the raven-toolbox model (53 to 76). +The no-task variant shares a Jaccard index of 0.964 to 0.982 with RAVEN. It keeps fewer reactions than RAVEN in every cell line: 0.3 to 0.5 % fewer for four cell lines and 1.6 % fewer for RPE1. The reactions only in the RAVEN model (76 to 186) outnumber those only in the raven-toolbox model (53 to 76). -The no-task variant omits the task layer that the RAVEN result includes. With the task layer, the reactions only in the RAVEN model fall to 43 to 71 per cell line. That the task layer accounts for most of the surplus is an inference from these counts; the reactions in the surplus were not checked against the task-essential reactions. +The no-task variant omits the task layer that the RAVEN result includes. With the task layer, the reactions only in the RAVEN model fall to 43 to 71 per cell line. That the task layer accounts for most of the surplus is an inference from these counts, and the reactions in the surplus were not checked against the task-essential reactions. -## Effect of `big_m` +### Effect of `big_m` -Lowering the cap on the per-reaction big-M from 1000 to 100 changed the Jaccard index against RAVEN by at most 0.003 in each cell line, one run per cell line and value. In every cell line the two raven-toolbox results shared a higher Jaccard index with each other (0.981 to 0.985) than either shared with RAVEN (0.964 to 0.982); for RPE1 the values were 0.984 between the two results against 0.964 and 0.967 with RAVEN. The cap therefore changed the reaction set (1 minus the Jaccard index between the two results, 0.015 to 0.019) by more than it changed the agreement with RAVEN (at most 0.003). At this resolution the two values agree with RAVEN equally well. +Lowering the cap on the per-reaction big-M from 1000 to 100 changed the Jaccard index against RAVEN by at most 0.003 in each cell line, with one run per cell line and value. In every cell line the two raven-toolbox results shared a higher Jaccard index with each other (0.981 to 0.985) than either shared with RAVEN (0.964 to 0.982). For RPE1 the values were 0.984 between the two results and 0.964 and 0.967 with RAVEN. The cap therefore changed the reaction set (1 minus the Jaccard index between the two results, 0.015 to 0.019) by more than it changed the agreement with RAVEN (at most 0.003). At this resolution the two values agree equally well with RAVEN. -## Run time +### Run time -After the template preparation, a task-constrained cell-type model took 146 to 381 s in raven-toolbox and 40 to 91 s in RAVEN; a no-task model took 81 to 91 s. The template preparation took 1556 s (26 min) and 1695 s (28 min) in raven-toolbox and 4320 s (72 min) in RAVEN. The preparations ran with different parallelism (see the run-time table), so the times compare whole runs and not the code alone. The [Reference run](../../guide/init.md#reference-run) of the guide Context-specific models with ftINIT reports a preparation of 47 min in MATLAB and 37 min in Python and an extraction of 797 s in MATLAB and 124 s in Python. That run used RAVEN's `prepINITModel` directly on Human-GEM `6812be2a` (12748 reactions) without the cleaning step, and series `1+0`; these four differences from this validation (preparation function, Human-GEM commit, cleaning step, series) were not separated, so the times are not comparable. +After the template preparation, a task-constrained cell-type model took 146 to 381 s in raven-toolbox and 40 to 91 s in RAVEN, and a no-task model took 81 to 91 s. The template preparation took 1556 s (26 min) and 1695 s (28 min) in raven-toolbox and 4320 s (72 min) in RAVEN. The preparations ran with different parallelism (see the run-time table), so the times compare whole runs and not the code alone. + +The [Reference run](../../guide/init.md#reference-run) of the guide Context-specific models with ftINIT reports a preparation of 47 min in MATLAB and 37 min in Python and an extraction of 797 s in MATLAB and 124 s in Python. That run used `prepINITModel` of RAVEN directly on Human-GEM `6812be2a` (12748 reactions) without the cleaning step, and series `1+0`. These four differences from this validation (preparation function, Human-GEM commit, cleaning step and series) were not separated, so the times are not comparable. ## Numerical settings shared with RAVEN raven-toolbox builds the extraction MILP with two numerical settings of RAVEN: -- Per-reaction big-M. The on-off constraints of a scored reaction use `max(min(ub, big_m), min(-lb, big_m), force_on)` as big-M, an infinite bound counting as `big_m`. The default cap `big_m=1000` is RAVEN's `bigMCap` in `ftINITInternalAlg`. The cap limits the flux a reaction can carry while it counts as on. -- Stoichiometric rescaling. `prep_init_model` applies RAVEN's `rescaleModelForINIT` to the merged model: each coefficient of a reaction is capped at 25 times the smallest coefficient of that reaction, and the reaction is then scaled to a mean absolute coefficient of 1. - -Which combinations of `big_m` and rescaling are feasible for the staged pipeline was measured on HCT116 in [ftINIT parameter calibration and input robustness](init-param-calibration.md): with rescaling, `big_m=25` was infeasible and `big_m=100` feasible; without rescaling, `big_m=1000` was feasible (157 s against 83 s with rescaling, Jaccard 0.969 between the two results) and `big_m=100` infeasible. +- **Per-reaction big-M.** The on-off constraints of a scored reaction use `max(min(ub, big_m), min(-lb, big_m), force_on)` as big-M, and an infinite bound counts as `big_m`. The default cap `big_m=1000` is `bigMCap` of RAVEN in `ftINITInternalAlg`. The cap limits the flux that a reaction can carry while it counts as on. +- **Stoichiometric rescaling.** `prep_init_model` applies `rescaleModelForINIT` of RAVEN to the merged model. Each coefficient of a reaction is capped at 25 times the smallest coefficient of that reaction, and the reaction is then scaled to a mean absolute coefficient of 1. -## Constraint construction +[ftINIT parameter calibration and input robustness](init-param-calibration.md) measured on HCT116 which combinations of `big_m` and rescaling are feasible for the staged pipeline. With rescaling, `big_m=25` was infeasible and `big_m=100` feasible. Without rescaling, `big_m=1000` was feasible (157 s against 83 s with rescaling, Jaccard index 0.969 between the two results) and `big_m=100` infeasible. -Each steady-state balance of the extraction MILP is built from a flat list of terms and summed with `optlang.symbolics.add`; the same construction is used in the gap-fill. Python's `sum()` re-simplifies the growing expression at each term, and the balances of hub metabolites such as ATP, H⁺ and H₂O are long sums. +Each steady-state balance of the extraction MILP is built from a flat list of terms and summed with `optlang.symbolics.add`, and the gap-fill uses the same construction. The `sum()` of Python simplifies the growing expression again at each term, and the balances of hub metabolites such as ATP, H⁺ and H₂O are long sums. ## Use of the result -The agreement was measured with the code defaults `big_m=1000` (`_BIG_M` in `raven_toolbox/init/ftinit.py`, listed in [Tuned parameter defaults](../../tuned-parameters.md)) and rescaling on (`scale=True` in `prep_init_model`, with `max_stoich_diff=25` in `rescale_for_init`, both in `raven_toolbox/init/prep.py`). A raven-toolbox model agreed most closely with a RAVEN `ftINIT` model when `prep_init_model` received the same task list as RAVEN's preparation. +The agreement was measured with the code defaults `big_m=1000` (`_BIG_M` in `raven_toolbox/init/ftinit.py`, listed in [Tuned parameter defaults](../../tuned-parameters.md))) and rescaling on (`scale=True` in `prep_init_model`, with `max_stoich_diff=25` in `rescale_for_init`, both in `raven_toolbox/init/prep.py`). A raven-toolbox model agreed most closely with a RAVEN `ftINIT` model when `prep_init_model` received the same task list as the preparation of RAVEN. ## Limits -Each cell line, variant and `big_m` value ran once, without seed repeats, so the run-to-run spread of either tool was not measured and Jaccard differences of a few tenths of a percent are not resolved. The comparison covers reaction sets on one template and one dataset; gene content, essential genes and task performance were not compared. The timings come from shared nodes and are approximate. +Each cell line, variant and `big_m` value ran once, without repeats of the seed, so the run-to-run spread of either tool was not measured and Jaccard differences of a few tenths of a percent are not resolved. The comparison covers reaction sets on one template and one dataset. Gene content, essential genes and task performance were not compared. The timings come from shared nodes and are approximate. ## Reproduction From 1af871639dba2bc21f7cb1dcee84533d6209451f Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Sun, 1 Nov 2026 11:46:40 +0100 Subject: [PATCH 13/20] Write the homology calibration page anew --- .../studies/homology-cutoff-calibration.md | 64 +++++++++++-------- 1 file changed, 37 insertions(+), 27 deletions(-) diff --git a/docs/parameter-tuning/studies/homology-cutoff-calibration.md b/docs/parameter-tuning/studies/homology-cutoff-calibration.md index bf8b2ac..db99e80 100644 --- a/docs/parameter-tuning/studies/homology-cutoff-calibration.md +++ b/docs/parameter-tuning/studies/homology-cutoff-calibration.md @@ -1,12 +1,14 @@ # Homology cut-off calibration -Homology-based reconstruction transfers a template reaction to a new organism when a gene of the new organism matches a template gene in a BLAST or DIAMOND search and the match passes three filters: `min_identity`, `min_align_len` and `max_evalue` (`minIde`, `minLen` and `maxE` in RAVEN's `getModelFromHomology`). The study aligned the proteomes of four organisms against *S. cerevisiae* once, applied each filter setting to the saved hits, and scored the surviving matches against two references that state which genes are counterparts. It records one score per organism and setting, the agreement between BLAST and DIAMOND, and their run times. +Homology-based reconstruction transfers a template reaction to a new organism when a gene of the new organism matches a template gene in a BLAST or DIAMOND search and the match passes three filters: `min_identity`, `min_align_len` and `max_evalue` (`minIde`, `minLen` and `maxE` in `getModelFromHomology` of RAVEN). The study aligned the proteomes of four organisms against *S. cerevisiae* once, applied each filter setting to the saved hits, and scored the surviving matches against two references that state which genes are counterparts. It records one score per organism and setting, the agreement between BLAST and DIAMOND, and their run times. + +The defaults that follow from the scores are `min_identity` 40, `min_align_len` 100 and `max_evalue` 1e-30. The identity has the largest effect on the score, the alignment length changes it less, and the E-value changes it only below 1e-30. ## Set-up ### Organisms and alignment -*S. cerevisiae* S288C is the template. The four targets span a range of relatedness to it, because a setting that suits a close relative need not suit a distant one: +*S. cerevisiae* S288C is the template. The four targets span a range of relatedness to it, because a setting that suits a close relative need not suit a distant one. | Code | Organism | Relation to yeast | |---|---|---| @@ -15,21 +17,20 @@ Homology-based reconstruction transfers a template reaction to a new organism wh | `ani` | *A. nidulans* | distant | | `eco` | *E. coli* | most distant | -Proteomes come from UniProt reference proteomes. Each protein is relabelled with its KEGG gene name through UniProt's KEGG cross-reference for the exact organism code; proteins without one are dropped. Both references can then be applied to the same matches. +The proteomes come from the UniProt reference proteomes. Each protein is relabelled with its KEGG gene name through the KEGG cross-reference of UniProt for the exact organism code, and proteins without one are dropped. Both references can then be applied to the same matches. -Each target was aligned against the template in both directions, once with `run_blast` (aligner E-value `1e-4`) and once with `run_diamond` (aligner E-value `1e-3`). Every filter setting was applied to the same saved hit table, so each score comes from one deterministic calculation. A hit is kept when `evalue <= max_evalue`, `align_len >= min_align_len` and `identity >= min_identity`; `get_model_from_homology` requires `evalue < max_evalue`. The kept hits are collected into a set of (template gene, target gene) pairs, so a pair found in both directions, or by more than one hit, counts as one match. A hit whose two genes are not both judged by the reference is counted as unjudgeable and left out of the score. The scoring does not reduce hits to best hits or build a model. +Each target was aligned against the template in both directions, once with `run_blast` (aligner E-value `1e-4`) and once with `run_diamond` (aligner E-value `1e-3`). Every filter setting was applied to the same saved hit table, so each score comes from one deterministic calculation. A hit is kept when `evalue <= max_evalue`, `align_len >= min_align_len` and `identity >= min_identity`, whereas `get_model_from_homology` requires `evalue < max_evalue`. The kept hits are collected into a set of (template gene, target gene) pairs, so a pair found in both directions, or by more than one hit, counts as one match. A hit whose two genes are not both judged by the reference is counted as unjudgeable and left out of the score. The scoring does not reduce hits to best hits and builds no model. The sweeps vary one setting at a time from the RAVEN 2.11.3 values `max_evalue=1e-30`, `min_align_len=200` and `min_identity=40`. The identity sweep therefore ran at an alignment length of 200, the length sweep at 40 % identity, and the E-value sweep at 40 % identity and length 200. ### References -The **KEGG reference** counts a match between a template gene and a target gene as correct when the two genes share at least one KO in KEGG's `organism_gene_ko` table (release 118). The table lists only KOs that are linked to a reaction. A match is judged only when both genes have such a KO; other matches are left out of the score. - -The **OMA reference** counts a match as correct when the [OMA browser](https://omabrowser.org/) lists the two genes as a pair of orthologs. OMA identifies proteins by UniProt identifiers, which are mapped to KEGG gene names on both sides; OMA pairs that cannot be mapped are dropped. A match is judged when both genes have a KEGG gene name, so the OMA reference covers the whole mapped proteome. +- **KEGG reference.** A match between a template gene and a target gene is correct when the two genes share at least one KO in the `organism_gene_ko` table of KEGG (release 118). The table lists only KOs that are linked to a reaction. A match is judged only when both genes have such a KO, and other matches are left out of the score. +- **OMA reference.** A match is correct when the [OMA browser](https://omabrowser.org/) lists the two genes as a pair of orthologs. OMA identifies proteins by UniProt identifiers, which are mapped to KEGG gene names on both sides, and OMA pairs that cannot be mapped are dropped. A match is judged when both genes have a KEGG gene name, so the OMA reference covers the whole mapped proteome. ### Score -For each setting, **precision** is the number of correct matches divided by all judged matches, and **recall** is the number of correct matches divided by all pairs in the reference. The score is F0.5, the weighted harmonic mean `1.25·P·R / (0.25·P + R)` of precision P and recall R. With β = 0.5 it weights precision twice as heavily as recall. The weighting is the authors' choice: a missing reaction can be added by gap-filling, while a wrongly transferred reaction is hard to notice and harder to remove. The reproduction command passes `--beta 0.5`, which is also the default of the script, so the scores in the tables are F0.5. All score tables are against the KEGG reference; their base rows agree with each other. +For each setting, the precision is the number of correct matches divided by all judged matches, and the recall is the number of correct matches divided by all pairs in the reference. The score is F0.5, the weighted harmonic mean `1.25·P·R / (0.25·P + R)` of precision P and recall R. With β = 0.5 it weights precision twice as heavily as recall. The weighting is the choice of the authors, because a missing reaction can be added by gap-filling, while a wrongly transferred reaction is hard to notice and harder to remove. The reproduction command passes `--beta 0.5`, which is also the default of the script, so the scores in the tables are F0.5. All score tables are against the KEGG reference, and their base rows agree with each other. ## Results @@ -44,9 +45,9 @@ For each setting, **precision** is the number of correct matches divided by all | 45 | 0.907 | 0.813 | 0.735 | 0.520 | | 50 | 0.882 | 0.736 | 0.640 | 0.323 | -Against the KEGG reference the highest score was at 40 % for `kla` and `eco` and at 35 % for `yli` and `ani`. Against the OMA reference with BLAST it was at 45 % for the three fungi and at 35 % for *E. coli*; with DIAMOND it was at 45 % for the fungi and at 30 % for *E. coli*. The OMA scores are not tabulated. With BLAST the optima of both references lie between 35 and 45 %; the only optimum below 35 % is *E. coli* against OMA with DIAMOND. +Against the KEGG reference the highest score was at 40 % for `kla` and `eco` and at 35 % for `yli` and `ani`. Against the OMA reference the highest score with BLAST was at 45 % for the three fungi and at 35 % for *E. coli*. With DIAMOND it was at 45 % for the fungi and at 30 % for *E. coli*. The OMA scores are not tabulated. With BLAST the optima of both references lie between 35 and 45 %, and the only optimum below 35 % is *E. coli* against OMA with DIAMOND. -Scored with F1, which weights precision and recall equally, the optimum with BLAST moves to 30 % for `kla` and `eco` and 25 % for `yli` and `ani`; with DIAMOND it is 30 % for `kla`, `ani` and `eco` and 25 % for `yli` (values not tabulated). The best identity therefore depends on how the score weights the two kinds of error. +Scored with F1, which weights precision and recall equally, the optimum with BLAST moves to 30 % for `kla` and `eco` and to 25 % for `yli` and `ani`. With DIAMOND it is 30 % for `kla`, `ani` and `eco` and 25 % for `yli` (values not tabulated). The best identity therefore depends on how the score weights the two kinds of error. ### Alignment length @@ -58,7 +59,7 @@ Scored with F1, which weights precision and recall equally, the optimum with BLA | 200 (RAVEN 2.11.3) | 0.923 | 0.860 | 0.804 | 0.598 | | 300 | 0.885 | 0.816 | 0.764 | 0.571 | -Lengths 50 and 100 gave identical scores and 150 was at most 0.003 lower. The score fell between 150 and 200, by 0.010 to 0.020 from 100 to 200. Compared with 200, a length of 100 raised recall by 3.0 (`kla`), 3.8 (`yli`), 2.2 (`ani`) and 3.1 (`eco`) percentage points, which is 3.9, 6.5, 4.3 and 10.2 % more correct matches, and changed precision by at most 0.6 percentage points (`yli`, 0.967 to 0.961; precision and recall values not tabulated). No OMA scores were reported for alignment length. +The lengths 50 and 100 gave identical scores, and 150 was at most 0.003 lower. The score fell between 150 and 200, by 0.010 to 0.020 from 100 to 200. Compared with a length of 200, a length of 100 raised the recall by 3.0 (`kla`), 3.8 (`yli`), 2.2 (`ani`) and 3.1 (`eco`) percentage points, which is 3.9, 6.5, 4.3 and 10.2 % more correct matches. It changed the precision by at most 0.6 percentage points (`yli`, 0.967 to 0.961; precision and recall values not tabulated). No OMA scores were reported for the alignment length. ### E-value @@ -72,7 +73,7 @@ The hits were collected at an aligner E-value of `1e-4`, so `max_evalue=1e-4` ke | 1e-50 | 0.923 | 0.858 | 0.801 | 0.592 | | 1e-100 | 0.903 | 0.809 | 0.743 | 0.535 | -The score was the same at 1e-4, 1e-10 and 1e-30 for all four organisms. At 1e-50 it was unchanged for `kla` and lower by 0.002 (`yli`), 0.003 (`ani`) and 0.006 (`eco`), and at 1e-100 it was lower by 0.020 to 0.063. The F1 values (BLAST) are 0.855 for `kla` from 1e-4 to 1e-50 and 0.811 at 1e-100, and 0.663, 0.658 and 0.563 for `ani` at 1e-30, 1e-50 and 1e-100. A hit with at least 40 % identity over at least 200 residues can therefore have an E-value above 1e-50 in `yli`, `ani` and `eco` (inference from the score change; the hit tables were not inspected for this). No OMA scores were reported for the E-value. +The score was the same at 1e-4, 1e-10 and 1e-30 for all four organisms. At 1e-50 it was unchanged for `kla` and lower by 0.002 (`yli`), 0.003 (`ani`) and 0.006 (`eco`), and at 1e-100 it was lower by 0.020 to 0.063. The F1 values (BLAST) are 0.855 for `kla` from 1e-4 to 1e-50 and 0.811 at 1e-100, and 0.663, 0.658 and 0.563 for `ani` at 1e-30, 1e-50 and 1e-100. A hit with at least 40 % identity over at least 200 residues can therefore have an E-value above 1e-50 in `yli`, `ani` and `eco`. This is an inference from the score change, and the hit tables were not inspected for it. No OMA scores were reported for the E-value. ### BLAST and DIAMOND @@ -83,39 +84,48 @@ The score was the same at 1e-4, 1e-10 and 1e-30 for all four organisms. At 1e-50 | `ani` | 35 | 35 | 0.814 vs 0.811 | | `eco` | 40 | 40 | 0.598 vs 0.593 | -Against the KEGG reference with F0.5, the best identity was the same with both aligners for every organism, and the scores at that identity differed by 0.002 to 0.005. DIAMOND ran with `--more-sensitive`, and the speed ratios below apply to that mode. DIAMOND returned 32,192 hits on `kla` against 54,513 for BLAST. After filtering, the two aligners agreed on between 87.9 and 92.3 % of the surviving matches, measured as the Jaccard index of the (template gene, target gene) pairs that survive the filters at the base setting (identity 40, length 200, E-value 1e-30): 92.3 % (`kla`), 88.9 % (`yli`), 87.9 % (`ani`) and 91.8 % (`eco`). That the hits missing from DIAMOND are weak matches the filters would discard is an inference from these two results. BLAST took 140, 152, 200 and 44 s for `kla`, `yli`, `ani` and `eco`, and DIAMOND 12, 11, 14 and 7 s; DIAMOND was 11.7 times faster on `kla` and 14.3 times on `ani` (4 threads, AMD EPYC 9354, DIAMOND with `--more-sensitive`). +Against the KEGG reference with F0.5, the best identity was the same with both aligners for every organism, and the scores at that identity differed by 0.002 to 0.005. DIAMOND ran with `--more-sensitive`, and the speed ratios below apply to that mode. + +DIAMOND returned 32,192 hits on `kla` against 54,513 for BLAST. After filtering, the two aligners agreed on 87.9 to 92.3 % of the surviving matches. The agreement is the Jaccard index of the (template gene, target gene) pairs that survive the filters at the base setting (identity 40, length 200, E-value 1e-30): 92.3 % (`kla`), 88.9 % (`yli`), 87.9 % (`ani`) and 91.8 % (`eco`). That the hits missing from DIAMOND are weak matches that the filters would discard is an inference from these two results. + +BLAST took 140, 152, 200 and 44 s for `kla`, `yli`, `ani` and `eco`, and DIAMOND took 12, 11, 14 and 7 s. DIAMOND was 11.7 times faster on `kla` and 14.3 times on `ani` (4 threads, AMD EPYC 9354, DIAMOND with `--more-sensitive`). ### Aligner settings Two more settings control the search that produces the hits. Neither was part of the sweeps. -`evalue` is the aligner's own cut-off: `1e-4` for `run_blast` and `1e-3` for `run_diamond`. Each matches the MATLAB side: `getBlast` passes `-evalue 10e-5`, which equals `1e-4`, and `getDiamond` passes none, so DIAMOND's own default applies. On *H. polymorpha* against *S. cerevisiae*, loosening the BLAST cut-off from `1e-5` to `1e-4` added 5,200 raw hits (58,702 against 53,502), 8.9 % of the table. The tables differ row by row: 5,314 rows are only in the `1e-4` table and 114 only in the `1e-5` table. Of the 5,314 rows, 4,372 have an E-value above `1e-5` and so fail `max_evalue=1e-30`, and 942 have an E-value at or below `1e-5` (the smallest is 3.1e-60; 4 are below `1e-30`). In the five examples of the 942 rows that were inspected, the rows are other alignments (HSPs) of gene pairs that the cut-off change reports differently; the other rows were not inspected. Whether the 4 rows below `1e-30` and the 114 rows found only at `1e-5` also pass the identity and length filters, and so could change a model, was not checked. The remaining rows fail `max_evalue=1e-30`, so for them the aligner cut-off changes the size of the hit table and not the model. +The aligner setting `evalue` is the own cut-off of the aligner: `1e-4` for `run_blast` and `1e-3` for `run_diamond`. Each matches the MATLAB side, because `getBlast` passes `-evalue 10e-5`, which equals `1e-4`, and `getDiamond` passes none, so the own default of DIAMOND applies. On *H. polymorpha* against *S. cerevisiae*, loosening the BLAST cut-off from `1e-5` to `1e-4` added 5,200 raw hits (58,702 against 53,502), which is 8.9 % of the table. The tables differ row by row: 5,314 rows are only in the `1e-4` table and 114 only in the `1e-5` table. + +- Of the 5,314 rows, 4,372 have an E-value above `1e-5` and fail `max_evalue=1e-30`. +- The other 942 rows have an E-value at or below `1e-5`, the smallest being 3.1e-60, and 4 of them are below `1e-30`. In the five examples of these 942 rows that were inspected, the rows are other alignments (HSPs) of gene pairs that the cut-off change reports differently. The other rows were not inspected. +- Whether the 4 rows below `1e-30` and the 114 rows found only at `1e-5` also pass the identity and length filters, and so could change a model, was not checked. +- The remaining rows fail `max_evalue=1e-30`, so for them the aligner cut-off changes the size of the hit table and not the model. -`threads` defaults to every CPU the process may run on. The first 500 sequences of the *H. polymorpha* proteome returned the same 2,746 hits at one thread and at four, with identical hit tables, in 20.4 s and 5.9 s (3.5 times faster). On a full proteome pair, single-threaded BLASTP took 205 s (*H. polymorpha* against *S. cerevisiae*) and 218 s (the reverse direction). The aligner cut-offs `1e-4` and `1e-5` took 111 s and 108.5 s. The measurements used the SGD translated ORFs (6,735 sequences) as template and Linux BLAST+ 2.17.0 on an AMD EPYC 9354 node. +The setting `threads` defaults to every CPU that the process may run on. The first 500 sequences of the *H. polymorpha* proteome returned the same 2,746 hits at one thread and at four, with identical hit tables, in 20.4 s and 5.9 s (3.5 times faster). On a full proteome pair, single-threaded BLASTP took 205 s (*H. polymorpha* against *S. cerevisiae*) and 218 s (the reverse direction). The aligner cut-offs `1e-4` and `1e-5` took 111 s and 108.5 s. The measurements used the translated ORFs of SGD (6,735 sequences) as template and Linux BLAST+ 2.17.0 on an AMD EPYC 9354 node. ## Chosen defaults | Setting | RAVEN 2.11.3 | Default (raven-toolbox and RAVEN 3) | Basis | |---|---|---|---| -| `min_identity` (`minIde`) | 40 | 40 | Over the sweep from 25 to 50 % the score changed by 0.062 (`kla`) to 0.275 (`eco`), against 0.047 to 0.055 over alignment lengths 50 to 300 and 0.020 to 0.063 over E-values 1e-4 to 1e-100; with BLAST the optima of both references lie between 35 and 45 %. | -| `min_align_len` (`minLen`) | 200 | 100 | 100 scored 0.010 to 0.020 higher than 200 against KEGG; 50 and 100 scored the same, and 100 is the stricter of the two. | -| `max_evalue` (`maxE`) | 1e-30 | 1e-30 | No change in score from 1e-4 to 1e-30 in any organism; 1e-50 lowered it by up to 0.006; kept at the RAVEN value. | +| `min_identity` (`minIde`) | 40 | 40 | Over the sweep from 25 to 50 % the score changed by 0.062 (`kla`) to 0.275 (`eco`), against 0.047 to 0.055 over alignment lengths 50 to 300 and 0.020 to 0.063 over E-values 1e-4 to 1e-100. With BLAST the optima of both references lie between 35 and 45 % | +| `min_align_len` (`minLen`) | 200 | 100 | The length 100 scored 0.010 to 0.020 higher than 200 against KEGG. The lengths 50 and 100 scored the same, and 100 is the stricter of the two | +| `max_evalue` (`maxE`) | 1e-30 | 1e-30 | No change in score from 1e-4 to 1e-30 in any organism, and 1e-50 lowered it by up to 0.006. The RAVEN value is kept | -The study supports one `min_identity` for BLAST and DIAMOND against the KEGG reference with F0.5, since both reached the same best identity for every organism. Against OMA the optimum for *E. coli* is 35 % with BLAST and 30 % with DIAMOND, and with F1 the optimum for `ani` is 25 % with BLAST and 30 % with DIAMOND. DIAMOND scores for alignment length and E-value are not tabulated here. No DIAMOND scores are tabulated for the length and E-value sweeps, whose base scores equal those of BLAST. +The study supports one `min_identity` for BLAST and DIAMOND against the KEGG reference with F0.5, because both reached the same best identity for every organism. Against OMA the optimum for *E. coli* is 35 % with BLAST and 30 % with DIAMOND, and with F1 the optimum for `ani` is 25 % with BLAST and 30 % with DIAMOND. The DIAMOND scores for the alignment length and the E-value are not tabulated, and their base scores equal those of BLAST. ## Limits -- KEGG's KO annotations are partly derived from sequence similarity, so the KEGG reference alone would partly measure agreement with the method under test. OMA infers orthology independently of BLAST, and its optima lie in the same range of 35 to 45 % identity. OMA scores were reported for identity only. +- The KO annotations of KEGG are partly derived from sequence similarity, so the KEGG reference alone would partly measure agreement with the method under test. OMA infers orthology independently of BLAST, and its optima lie in the same range of 35 to 45 % identity. OMA scores were reported for identity only. - The two references are not comparable in absolute terms. The KEGG reference judges only genes with a reaction-linked KO, while the OMA reference judges the whole mapped proteome and counts every match that OMA does not list as an ortholog pair as wrong. Paralogs probably lower the OMA score for this reason (inference). Only the shape of each column is comparable between the references. -- The OMA reference for *Y. lipolytica* is incomplete: 38 % of its OMA pairs could not be mapped to KEGG gene names. +- The OMA reference for *Y. lipolytica* is incomplete, because 38 % of its OMA pairs could not be mapped to KEGG gene names. - One template: everything was measured outward from *S. cerevisiae*. -- One setting was varied at a time from length 200 and identity 40 %. The identity optimum and the absence of an E-value effect up to 1e-30 were measured at length 200, and were not measured at the default length of 100. -- The scores count gene matches, not reactions in a finished model. A reaction transfers when any one of its genes matches, so the effect on a finished model is probably smaller (inference, not measured). -- Each score comes from one alignment per organism. Scoring a saved hit table is deterministic. DIAMOND returned 32,192 hits for `kla`. The alignment was run once, so the run-to-run spread of the hit table was not measured. +- One setting was varied at a time from length 200 and identity 40 %. The identity optimum and the absence of an E-value effect up to 1e-30 were measured at length 200 and not at the default length of 100. +- The scores count gene matches and not reactions in a finished model. A reaction transfers when any one of its genes matches, so the effect on a finished model is probably smaller (inference, not measured). +- Each score comes from one alignment per organism. Scoring a saved hit table is deterministic, and DIAMOND returned 32,192 hits for `kla`. The alignment was run once, so the run-to-run spread of the hit table was not measured. ## Curated-GEM reference -Scoring against curated genome-scale metabolic models (GEMs) of non-model fungi such as [hanpo-GEM](https://github.com/SysBioChalmers/hanpo-GEM) and [rhto-GEM](https://github.com/SysBioChalmers/rhto-GEM) is not part of the 2026-11-01 measurement, and the study reports no numbers for it. Both models were built with RAVEN's homology-based reconstruction, so a score against them would measure agreement with their own build settings and not correctness (inference). +Scoring against curated genome-scale metabolic models (GEMs) of non-model fungi, such as [hanpo-GEM](https://github.com/SysBioChalmers/hanpo-GEM) and [rhto-GEM](https://github.com/SysBioChalmers/rhto-GEM), is not part of the measurement of 2026-11-01, and the study reports no numbers for it. Both models were built with the homology-based reconstruction of RAVEN, so a score against them would measure agreement with their own build settings and not correctness (inference). ## Reproducing @@ -131,8 +141,8 @@ python homology_cutoff_kegg.py --out work score --beta 0.5 \ python agreement.py --work work --gene-ko work/refs/kegg118_organism_gene_ko_5org.tsv.gz --out agreement.tsv ``` -`fetch_references.py` downloads the release-118 gene→KO rows (from the `kegg118_core.tar.gz` release asset of raven-toolbox), the current KEGG REST tables and the SGD translated ORFs. `align --aligner diamond` and `score --aligner diamond` repeat the steps with DIAMOND, `score --reference oma` scores against the OMA reference and `score --beta 1` gives F1. The driver also scores identities of 20 and 60 %, which are not tabulated above. `agreement.py` computes the BLAST and DIAMOND agreement, `make_tables.py` collects the results, and `aligner_settings.py` runs the *H. polymorpha* measurements (thread test, aligner E-value `1e-4` against `1e-5`, single-thread full directions). `run_cached.py` wraps the driver, whose path is given in the environment variable `HOMOLOGY_SCRIPT`, and replaces its UniProt download with a disk cache, which allows scoring against OMA offline; it adds the `prefetch` subcommand and passes every other subcommand to the driver. The alignment output is saved, so re-scoring under a different weighting or against a different reference takes about two minutes. +`fetch_references.py` downloads the release-118 gene→KO rows (from the `kegg118_core.tar.gz` release asset of raven-toolbox), the current KEGG REST tables and the SGD translated ORFs. The commands `align --aligner diamond` and `score --aligner diamond` repeat the steps with DIAMOND, `score --reference oma` scores against the OMA reference, and `score --beta 1` gives F1. The driver also scores identities of 20 and 60 %, which are not tabulated above. The script `agreement.py` computes the BLAST and DIAMOND agreement, `make_tables.py` collects the results, and `aligner_settings.py` runs the *H. polymorpha* measurements (thread test, aligner E-value `1e-4` against `1e-5`, single-thread full directions). The script `run_cached.py` wraps the driver, whose path is given in the environment variable `HOMOLOGY_SCRIPT`, and replaces its UniProt download with a disk cache, which allows scoring against OMA offline. It adds the `prefetch` subcommand and passes every other subcommand to the driver. The alignment output is saved, so scoring again under another weighting or against another reference takes about two minutes. -The UniProt downloads, the OMA pairs, the KEGG tables and the SGD ORFs are in the archived input bundle `homology-inputs.tar.zst` (18.9 MB), and the BLAST and DIAMOND hit tables in `homology-hits.tar.zst` (4.0 MB). +The UniProt downloads, the OMA pairs, the KEGG tables and the SGD ORFs are in the archived input bundle `homology-inputs.tar.zst` (18.9 MB), and the BLAST and DIAMOND hit tables are in `homology-hits.tar.zst` (4.0 MB). The scores come from the run of 2026-11-01 with raven-toolbox `94f3940`, BLAST+ 2.17.0, DIAMOND 2.1.17, the KEGG release-118 table and UniProt release 2026_03, with BLAST and DIAMOND at 4 threads on an AMD EPYC 9354 node. The *H. polymorpha* template is the SGD ORF file, and the thread test uses the first 500 sequences of `hanpo.faa`. The curated-GEM reference is not part of this run. From 759ad604d5ead15f5ceaf7d36aa3ce7a110d7778 Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Sun, 1 Nov 2026 11:47:32 +0100 Subject: [PATCH 14/20] Write the ftINIT calibration page anew --- .../studies/init-param-calibration.md | 120 +++++++++++------- 1 file changed, 73 insertions(+), 47 deletions(-) diff --git a/docs/parameter-tuning/studies/init-param-calibration.md b/docs/parameter-tuning/studies/init-param-calibration.md index b04f1bc..a660109 100644 --- a/docs/parameter-tuning/studies/init-param-calibration.md +++ b/docs/parameter-tuning/studies/init-param-calibration.md @@ -1,37 +1,43 @@ # ftINIT parameter calibration and input robustness -This study runs raven-toolbox's ftINIT on Human-GEM with the HCT116 expression data of the Hart2015 dataset and varies one or two settings per run series; the scaling runs and the full-pipeline runs vary scaling and `big_m` together. The calibration part varies the mixed-integer linear program (MILP) settings `mip_gap`, `big_m` and `force_on` and the stoichiometric scaling of the prepared model, and records the run time, the MILP objective and which reactions are kept. The robustness part degrades the expression input by dropout, noise and downsampling, extracts a model with the metabolic-task layer on, and records the model size, the essential tasks the model passes and its overlap with the model from clean data. Overlap is measured as the Jaccard index of two kept-reaction sets: the reactions in both divided by the reactions in either. +The study runs the ftINIT of raven-toolbox on Human-GEM with the HCT116 expression data of the Hart2015 dataset and varies one or two settings per series of runs. The scaling runs and the full-pipeline runs vary the scaling and `big_m` together. The calibration part varies the settings `mip_gap`, `big_m` and `force_on` of the mixed-integer linear program (MILP) and the stoichiometric scaling of the prepared model, and records the run time, the MILP objective and the reactions that are kept. The robustness part degrades the expression input by dropout, noise and downsampling, extracts a model with the metabolic-task layer on, and records the size of the model, the essential tasks that it passes and its overlap with the model from clean data. -Steps are counted from 1 on the pages of this study; the run logs count from 0, so the logged `step 1` is the second step. +The overlap is the Jaccard index of two kept-reaction sets: the reactions in both divided by the reactions in either. Steps are counted from 1 on the pages of this study. The run logs count from 0, so the logged `step 1` is the second step. + +The main findings are: + +- In the first-step solve, every `big_m` from 10 to 1000 and every scaling setting was feasible, and the time stayed between 23 and 30 s. +- In the full pipeline, `big_m=1000` was feasible with and without scaling (83 s and 157 s), `big_m=100` was feasible with scaling only, and `big_m=25` was infeasible with scaling. +- Noise changed the model less than sparse input, and no tested parameter recovered the expression evidence that dropout removes. ## Set-up ### Software and inputs -The calibration runs used Human-GEM commit `38567ab8` (v2.1.1 plus 15 commits), the HCT116 column of the Hart2015 RNA sequencing file, raven-toolbox commit `94f3940` and Gurobi on 4 cores. Gurobi library 13.0.3 from the gurobipy wheel (most likely); the logs print no version. The template is the reference model of the [Human-GEM cell-type model validation against RAVEN](humangem-validation.md): Human-GEM without the drug reactions, the amino-acid triplet reactions and the duplicate complex I reaction, 11265 reactions. `prep_init_model` prepared it without a task list. Its scaled merged model, the model that the MILP receives, has 6657 reactions; the preparation took 26 min and is excluded from every time in the tables. +The calibration runs used Human-GEM commit `38567ab8` (v2.1.1 plus 15 commits), the HCT116 column of the Hart2015 RNA sequencing file, raven-toolbox commit `94f3940` and Gurobi on 4 cores. The Gurobi library is 13.0.3 from the gurobipy wheel (most likely, because the logs print no version). The template is the reference model of the [Human-GEM cell-type model validation against RAVEN](humangem-validation.md): Human-GEM without the drug reactions, the amino-acid triplet reactions and the duplicate complex I reaction, with 11265 reactions. `prep_init_model` prepared it without a task list. Its scaled merged model, which is the model that the MILP receives, has 6657 reactions. The preparation took 26 min and is excluded from every time in the tables. The robustness runs used Human-GEM at a commit that is not recorded, the same HCT116 data, a template prepared with the essential metabolic tasks, and `big_m=100`. The two parts therefore have no common baseline, and their reaction counts and Jaccard indices cannot be compared with each other. -Gene scores are `5·ln(level/1)` of each expression level, clipped to [−5, 10] (see [Gene-scoring constants](#gene-scoring-constants)), and reaction scores follow from the gene rules (`score_reactions_from_genes`, default `no_gene_score=-2`). +Gene scores are `5·ln(level/1)` of each expression level, clipped to [−5, 10] (see [Gene-scoring constants](#gene-scoring-constants)). Reaction scores follow from the gene rules (`score_reactions_from_genes`, default `no_gene_score=-2`). ### Run types | Run type | What runs | Script and sweep | |---|---|---| | First-step solve | One `run_ftinit` call (one MILP, no run schedule) for the first step of series `1+1` on the scaled merged model, with a 900 s limit | `bigm/bm.py step0` | -| Scaling run | The same first-step solve on a model prepared without scaling, which `rescale_for_init` then rescales with the given `max_stoich_diff` (or leaves unscaled), with a 600 s limit and `mip_gap=0.001`. | `bigm/bm.py scale` | -| Full pipeline | One call of `ftinit` with series `1+1` and `time_limit=600` on the prepared model without tasks: both MILP steps, each with its own schedule of runs. | `bigm/bm.py full` | -| Robustness run | One call of `ftinit` with series `1+1` on the template prepared with tasks, including the gap-fill that adds reactions until every task is feasible, followed by `check_tasks` on the extracted model. | `scripts/analyze_init_robustness.py` (`MIP_GAP` 0.02, `TIME_LIMIT` 120), phases `gradient` and `levers` (not recoverable) | +| Scaling run | The same first-step solve on a model prepared without scaling, which `rescale_for_init` then rescales with the given `max_stoich_diff` or leaves unscaled, with a 600 s limit and `mip_gap=0.001` | `bigm/bm.py scale` | +| Full pipeline | One call of `ftinit` with series `1+1` and `time_limit=600` on the prepared model without tasks: both MILP steps, each with its own schedule of runs | `bigm/bm.py full` | +| Robustness run | One call of `ftinit` with series `1+1` on the template prepared with tasks, including the gap-fill that adds reactions until every task is feasible, followed by `check_tasks` on the extracted model | `scripts/analyze_init_robustness.py` (`MIP_GAP` 0.02, `TIME_LIMIT` 120), phases `gradient` and `levers` (not recoverable) | -A named series such as `1+1` ignores `mip_gap` and `mip_gap_abs`: every step runs its own schedule of MILP runs, with limits of 120 s, 5000 s and 5 s. `time_limit` lowers the limit of each scheduled run, so at `time_limit=600` it shortens the 5000 s runs only. `mip_gap` therefore acts on the first-step solve and not on the full pipeline. +A named series such as `1+1` ignores `mip_gap` and `mip_gap_abs`. Every step runs its own schedule of MILP runs, with limits of 120 s, 5000 s and 5 s. The setting `time_limit` lowers the limit of each scheduled run, so at `time_limit=600` it shortens the 5000 s runs only. The setting `mip_gap` therefore acts on the first-step solve and not on the full pipeline. -`big_m` is the cap on the per-reaction big-M of the on and off constraints. Each scored reaction gets `max(min(ub, big_m), min(-lb, big_m), force_on)`, and an infinite bound counts as `big_m`, so `big_m` limits the flux that a reaction can carry while it counts as on. `force_on` is the smallest flux at which a reaction counts as on. `max_stoich_diff` is the largest ratio between the stoichiometric coefficients of one reaction after `rescale_for_init` (RAVEN's `rescaleModelForINIT`). +The setting `big_m` is the cap on the per-reaction big-M of the on and off constraints. Each scored reaction gets `max(min(ub, big_m), min(-lb, big_m), force_on)`, and an infinite bound counts as `big_m`, so `big_m` limits the flux that a reaction can carry while it counts as on. The setting `force_on` is the smallest flux at which a reaction counts as on. The setting `max_stoich_diff` is the largest ratio between the stoichiometric coefficients of one reaction after `rescale_for_init` (`rescaleModelForINIT` of RAVEN). The robustness runs degrade the expression vector of HCT116 in three ways, each with one random draw (seed 0): -- **Dropout**: a fraction of the genes in the data gets the level 0, which scores −5. -- **Noise**: every level is multiplied by `exp(N(0,σ))`, a log-normal factor with standard deviation σ on the natural-log scale. -- **Downsampling**: a fraction of the genes is removed from the data. Within a gene rule a missing gene is skipped, and a reaction whose genes are all missing gets `no_gene_score`. +- **Dropout:** a fraction of the genes in the data gets the level 0, which scores −5. +- **Noise:** every level is multiplied by `exp(N(0,σ))`, a log-normal factor with standard deviation σ on the natural-log scale. +- **Downsampling:** a fraction of the genes is removed from the data. Within a gene rule a missing gene is skipped, and a reaction whose genes are all missing gets `no_gene_score`. The lever runs repeat the 70 % dropout input with `no_gene_score` or `force_on` changed. @@ -41,27 +47,27 @@ The lever runs repeat the 70 % dropout input with `no_gene_score` or `force_on` |---|---| | time (s) | Wall-clock time of the `run_ftinit` call (first-step and scaling runs) or the `ftinit` call (full pipeline), including the construction of the MILP and excluding the preparation | | objective | MILP objective of the first-step solve | -| rel.obj.gap | (reference objective − objective) / reference objective; a positive value is a lower objective than the reference row | +| rel.obj.gap | (reference objective − objective) / reference objective. A positive value is a lower objective than the reference row | | `n_kept` | Reactions in the model that the full pipeline returns | | n_rxns | Reactions in the model that a robustness run returns, after the gap-fill | -| Jaccard | Jaccard index of two kept-reaction sets: the reactions in both divided by the reactions in either. First-step sets are merged-model reaction identifiers; full-pipeline and robustness sets are Human-GEM reaction identifiers | +| Jaccard | Jaccard index of two kept-reaction sets. First-step sets are merged-model reaction identifiers, and full-pipeline and robustness sets are Human-GEM reaction identifiers | | tasks pass, frac | Essential tasks that `check_tasks` reports as passed on the extracted model, as a count and as a fraction | A MIP gap bounds the objective only. Two solutions within the gap can keep different reactions, so the reaction set is compared separately from the objective. -Each table row is one run unless the row states repeats; this set-up has no runs that differ only in the seed. The only available measure of the seed-to-seed spread comes from a different set-up, in [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](ftinit-determinism.md): two baseline builds (Human-GEM v2.0.0-38, DLD1, `big_m=100`, escalation schedule, with tasks and gap-fill) that differed only in the seed differed by 110 reactions (symmetric difference of the kept-reaction sets). The two baseline builds kept 7779 and 7781 reactions, so the 110 reactions are 1.4 % of the model and correspond to a Jaccard index of 0.986. That pair compares two full pipelines with gap-fill; a first-step set is the result of one MILP step on merged-model identifiers without tasks, a different kind of object. Between the settings of the calibration the Jaccard index ranged from 0.9976 to 0.9649, a difference of 0.2 to 3.5 % from identical sets. +Each table row is one run unless the row states repeats. This set-up has no runs that differ only in the seed. The only measure of the seed-to-seed spread comes from another set-up, in [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](ftinit-determinism.md). There, two baseline builds that differed only in the seed (Human-GEM v2.0.0-38, DLD1, `big_m=100`, escalation schedule, with tasks and gap-fill) differed by 110 reactions, which is the symmetric difference of the kept-reaction sets. The two builds kept 7779 and 7781 reactions, so the 110 reactions are 1.4 % of the model and correspond to a Jaccard index of 0.986. That pair compares two full pipelines with gap-fill, whereas a first-step set is the result of one MILP step on merged-model identifiers without tasks, which is another kind of object. Between the settings of the calibration the Jaccard index ranged from 0.9976 to 0.9649, a difference of 0.2 to 3.5 % from identical sets. ### Source of the numbers -The calibration tables come from `bigm/bm.py` with the subcommands `step0`, `scale` and `full`, summarised by `bigm/summ.py` and `bigm/compare.py` (prepared model `prep_notask.pkl`). The scripts are in the raven-docs repository under `scripts/parameter_tuning/init/`. `step0` runs a grid of 16 solves: `mip_gap` 0.0002 to 0.1, `big_m` 10 to 1000 and `force_on` 0.02 to 0.5, at `big_m=1000` for the `mip_gap` and `force_on` sweeps. `scale` runs the unscaled preparation rescaled with `max_stoich_diff` 10, 25, 50 and 100 or left unscaled, at `big_m` 1000 and 100. `full` runs one `ftinit` call per setting, with `--gap`, `--big-m`, `--scale` and `--noabs` (`mip_gap_abs=None`) options. +The calibration tables come from `bigm/bm.py` with the subcommands `step0`, `scale` and `full`, summarised by `bigm/summ.py` and `bigm/compare.py` (prepared model `prep_notask.pkl`). The scripts are in the raven-docs repository under `scripts/parameter_tuning/init/`. The subcommand `step0` runs a grid of 16 solves: `mip_gap` 0.0002 to 0.1, `big_m` 10 to 1000 and `force_on` 0.02 to 0.5, at `big_m=1000` for the `mip_gap` and `force_on` sweeps. The subcommand `scale` runs the unscaled preparation rescaled with `max_stoich_diff` 10, 25, 50 and 100 or left unscaled, at `big_m` 1000 and 100. The subcommand `full` runs one `ftinit` call per setting, with the options `--gap`, `--big-m`, `--scale` and `--noabs` (`mip_gap_abs=None`). -The first-step, scaling and full-pipeline tables are computed from the outputs by `compare.py`. The robustness tables (degraded expression input, lever runs and the 90 % dropout run) come from `scripts/analyze_init_robustness.py`. The script exists at the pinned commit; the state of the script and the inputs that produced the tables are not recoverable. +`compare.py` computes the tables of the first-step solve, the scaling runs and the full pipeline from the outputs. The robustness tables (degraded expression input, lever runs and the 90 % dropout run) come from `scripts/analyze_init_robustness.py`. The script exists at the pinned commit, and the state of the script and the inputs that produced the tables are not recoverable. ## Results ### First-step solve with different `mip_gap` values -`big_m=1000`, `force_on=0.1`. The reference is the tightest gap. +The settings are `big_m=1000` and `force_on=0.1`, and the reference is the tightest gap. | `mip_gap` | time (s) | objective | rel.obj.gap | Jaccard vs tightest | |--------:|---------:|----------:|------------:|--------------------:| @@ -74,7 +80,7 @@ The first-step, scaling and full-pipeline tables are computed from the outputs b ### First-step solve with different `big_m` values -`mip_gap=0.001`, `force_on=0.1`. The last two columns compare each set with the run at `big_m=1000` and with the run at `big_m=100`. +The settings are `mip_gap=0.001` and `force_on=0.1`. The last two columns compare each set with the run at `big_m=1000` and with the run at `big_m=100`. | `big_m` | time (s) | rel.obj.gap | Jaccard vs `big_m=1000` | Jaccard vs `big_m=100` | |------:|---------:|------------:|----------------------:|---------------------:| @@ -88,7 +94,7 @@ The first-step, scaling and full-pipeline tables are computed from the outputs b ### First-step solve with different `force_on` values -`mip_gap=0.001`, `big_m=1000`. The reference is the default 0.1. +The settings are `mip_gap=0.001` and `big_m=1000`, and the reference is the default 0.1. | `force_on` | time (s) | rel.obj.gap | Jaccard vs 0.1 | |---------:|---------:|------------:|---------------:| @@ -100,7 +106,7 @@ The first-step, scaling and full-pipeline tables are computed from the outputs b ### First-step solve with different stoichiometric scaling -Scaling runs at `mip_gap=0.001` and two `big_m` values. Every rel.obj.gap and Jaccard index in the table, including the `big_m=100` columns, is against the first row at `big_m=1000` (scale on, `max_stoich_diff=25`, abbreviated `msd`). +The scaling runs used `mip_gap=0.001` and two `big_m` values. Every rel.obj.gap and Jaccard index in the table, including the `big_m=100` columns, is against the first row at `big_m=1000` (scale on, `max_stoich_diff=25`, abbreviated `msd`). | config | time (s), `big_m=1000` | rel.obj.gap | Jaccard vs scaled `msd=25`, `big_m=1000` | time (s), `big_m=100` | rel.obj.gap | Jaccard vs scaled `msd=25`, `big_m=1000` | |--------|---------:|------------:|-------------------------:|---------:|------------:|-------------------------:| @@ -110,13 +116,13 @@ Scaling runs at `mip_gap=0.001` and two `big_m` values. Every rel.obj.gap and Ja | `msd=100` | 26 | +0.0048 | 0.980 | 25 | +0.0049 | 0.980 | | scale off | 30 | −0.0050 | 0.984 | 25 | −0.0044 | 0.983 | -The reference row of this table and the `mip_gap=0.001` row of the first-step `mip_gap` table have the same settings and come from different runs: the scaling run rescales an unscaled preparation with `rescale_for_init` and has a 600 s limit, the first-step run uses `prep(scale=True)` and a 900 s limit. Their objectives are 45539.6 and 45949.2, and their reaction sets share a Jaccard index of 0.983. +The reference row of this table and the `mip_gap=0.001` row of the first-step `mip_gap` table have the same settings and come from different runs. The scaling run rescales an unscaled preparation with `rescale_for_init` and has a 600 s limit, whereas the first-step run uses `prep(scale=True)` and a 900 s limit. Their objectives are 45539.6 and 45949.2, and their reaction sets share a Jaccard index of 0.983. ### Full pipeline with and without scaling -`time_limit=600`. +The runs used `time_limit=600`. -| config (full pipeline, `time_limit=600`) | result | +| config | result | |-----------------------------------|--------| | scale on, `big_m=1000` | feasible, 83 s | | scale off, `big_m=1000` | feasible, 157 s, 7264 reactions, Jaccard 0.969 vs scale on | @@ -127,7 +133,7 @@ An infeasible run stopped with `ftINIT MILP produced no usable solution (status: ### Full pipeline with different `big_m` values -Scaled preparation without tasks, series `1+1`, `time_limit=600s`. +The runs used a scaled preparation without tasks, series `1+1` and `time_limit=600s`. | config | time (s) | `n_kept` | Jaccard vs `big_m=1000` | |--------|---------:|-------:|---------------------:| @@ -138,11 +144,18 @@ Scaled preparation without tasks, series `1+1`, `time_limit=600s`. | `big_m=50` | 82 | 7325 | 0.982 | | `big_m=25` | 30 | infeasible | none | -The runs were started with different `mip_gap` values, which the series ignores, so runs at one `big_m` are repeats of one configuration: 0.001, 0.003 and 0.01 at `big_m=1000`, 0.001 and 0.003 at `big_m=100`, and 0.003 at the other caps. Further repeats ran with `mip_gap_abs=None` in place of the default `mip_gap_abs=10`: at `big_m=1000` three runs of 85, 84 and 81 s, which kept the same 7312 reactions, and at `big_m=250` one run of 114 s. A third run at `big_m=100` took 119 s (`mip_gap=0.001`, `mip_gap_abs=None`) and a second run at `big_m=50` took 81 s (`mip_gap=0.003`, `mip_gap_abs=None`). In total `big_m=1000` has six runs (81 to 85 s), `big_m=250` two, `big_m=100` three (117, 118 and 119 s) and `big_m=50` two (82 and 81 s); `big_m=500` and `big_m=25` ran once. +The runs were started with different `mip_gap` values, which the series ignores, so runs at one `big_m` are repeats of one configuration: 0.001, 0.003 and 0.01 at `big_m=1000`, 0.001 and 0.003 at `big_m=100`, and 0.003 at the other caps. Further repeats ran with `mip_gap_abs=None` in place of the default `mip_gap_abs=10`: + +- at `big_m=1000`, three runs of 85, 84 and 81 s, which kept the same 7312 reactions; +- at `big_m=250`, one run of 114 s; +- at `big_m=100`, a third run of 119 s (`mip_gap=0.001`, `mip_gap_abs=None`); +- at `big_m=50`, a second run of 81 s (`mip_gap=0.003`, `mip_gap_abs=None`). + +In total `big_m=1000` has six runs (81 to 85 s), `big_m=250` two, `big_m=100` three (117, 118 and 119 s) and `big_m=50` two (82 and 81 s). The caps `big_m=500` and `big_m=25` ran once. ### Degraded expression input -Robustness runs with the task layer on. The clean row is the reference of the Jaccard column. +The robustness runs had the task layer on, and the clean row is the reference of the Jaccard column. | input | n_rxns | tasks pass | frac | Jaccard vs clean | |-------|-------:|-----------:|-----:|-----------------:| @@ -158,7 +171,7 @@ One model from 90 % dropout input, outside these tables, ran for more than 75 mi ### Parameters at 70 % dropout -Lever runs on the 70 % dropout input. The default row is the dropout 70 % row of the previous table. +The lever runs used the 70 % dropout input. The default row is the dropout 70 % row of the previous table. | config | n_rxns | frac | Jaccard vs clean | |--------|-------:|-----:|-----------------:| @@ -167,31 +180,40 @@ Lever runs on the 70 % dropout input. The default row is the dropout 70 % row of | `no_gene_score=−0.5` | 5128 | 0.986 | 0.593 | | `force_on=0.2` | 5159 | 0.986 | 0.600 | -## Effect of `mip_gap` on a single solve +## Interpretation + +### Effect of `mip_gap` on a single solve -In the first-step solve the time was 24 to 25 s at every gap from 0.0002 to 0.1. At `mip_gap=0.001` the objective was within 0.0001 of the tightest gap and the Jaccard index was 0.9976. The Jaccard index stayed at 0.99 or above up to `mip_gap=0.01` and fell to 0.9649 and 0.9663 at 0.03 and 0.1. In the full pipeline the gap has no effect: the three `big_m=1000` runs at 0.001, 0.003 and 0.01 kept the same 7312 reactions. +In the first-step solve the time was 24 to 25 s at every gap from 0.0002 to 0.1. At `mip_gap=0.001` the objective was within 0.0001 of the tightest gap and the Jaccard index was 0.9976. The index stayed at 0.99 or above up to `mip_gap=0.01` and fell to 0.9649 and 0.9663 at 0.03 and 0.1. In the full pipeline the gap has no effect: the three `big_m=1000` runs at 0.001, 0.003 and 0.01 kept the same 7312 reactions. -## Effect of `big_m` and scaling +### Effect of `big_m` and scaling -In the first-step solve every `big_m` from 10 to 1000 was feasible, took 23 to 26 s and reached the objective of `big_m=1000` within 0.0003. Against the run at `big_m=1000` and against the run at `big_m=100`, the reaction sets of the other caps had Jaccard indices of 0.983 to 0.991. The first-step solve has no seed repeats, and the seed pair in [Set-up](#set-up) compares full pipelines with gap-fill, so it is no reference for these sets. In the first-step scaling runs, every `max_stoich_diff` from 10 to 100 and the unscaled model were feasible at both caps, with Jaccard indices of 0.980 to 0.991 against the scaled reference. +In the first-step solve every `big_m` from 10 to 1000 was feasible, took 23 to 26 s and reached the objective of `big_m=1000` within 0.0003. Against the run at `big_m=1000` and against the run at `big_m=100`, the reaction sets of the other caps had Jaccard indices of 0.983 to 0.991. The first-step solve has no repeats of the seed, and the seed pair in [Set-up](#set-up) compares full pipelines with gap-fill, so it is no reference for these sets. In the first-step scaling runs, every `max_stoich_diff` from 10 to 100 and the unscaled model were feasible at both caps, with Jaccard indices of 0.980 to 0.991 against the scaled reference. -The full pipeline separates the settings that the first-step solve does not. At `big_m=1000` it was feasible with and without scaling: 83 s with scaling, 157 s without, and the two models shared a Jaccard index of 0.969 (7312 and 7264 reactions). At `big_m=100` it was feasible with scaling (117 to 119 s) and infeasible without, and at `big_m=25` it was infeasible with scaling. With scaling, the caps from 50 to 1000 gave runs of 81 to 119 s with no trend in `big_m` and reaction sets that shared a Jaccard index of 0.978 to 0.982 with the default. The fastest run at a cap below 1000 (80.8 s at `big_m=50`, with `mip_gap_abs=None`) was as fast as the fastest `big_m=1000` run (81.1 s). Counted from the reaction sets (the rounded Jaccard values in the table give 148 and 163 for the first two), the sets at caps 500, 250, 100 and 50 differed from the `big_m=1000` set by 150, 165, 140 and 133 reactions. The one seed pair available, from the different set-up in [Set-up](#set-up), differed by 110 reactions, so every cap difference is 21 to 50 % larger than that single value; the seed-to-seed spread of this set-up is unknown. The three `big_m=1000` runs kept identical sets (Jaccard 1.000) and only the cap changed between rows, so the differences follow the cap; whether they are systematic or a different choice among near-equal optima was not tested. A lower cap also reduces the flux that a kept reaction can carry. +The full pipeline separates the settings that the first-step solve does not. -## Effect of `force_on` +- At `big_m=1000` it was feasible with and without scaling. It took 83 s with scaling and 157 s without, and the two models shared a Jaccard index of 0.969 (7312 and 7264 reactions). +- At `big_m=100` it was feasible with scaling (117 to 119 s) and infeasible without. +- At `big_m=25` it was infeasible with scaling. +- With scaling, the caps from 50 to 1000 gave runs of 81 to 119 s with no trend in `big_m`, and reaction sets that shared a Jaccard index of 0.978 to 0.982 with the default. The fastest run at a cap below 1000 (80.8 s at `big_m=50`, with `mip_gap_abs=None`) was as fast as the fastest `big_m=1000` run (81.1 s). -`force_on` changes the MILP itself, through the on and off constraints. From 0.02 to 0.5 the first-step solve took 24 to 25 s, the objective moved by at most 0.0002 and the reaction set shared a Jaccard index of 0.987 or more with the default 0.1. The full pipeline was not run with other `force_on` values. +Counted from the reaction sets, the sets at the caps 500, 250, 100 and 50 differed from the `big_m=1000` set by 150, 165, 140 and 133 reactions. The rounded Jaccard values in the table give 148 and 163 for the first two. The one seed pair available, from the other set-up, differed by 110 reactions, so every cap difference is 21 to 50 % larger than that single value, and the seed-to-seed spread of this set-up is unknown. The three `big_m=1000` runs kept identical sets (Jaccard 1.000) and only the cap changed between rows, so the differences follow the cap. Whether they are systematic or a different choice among near-equal optima was not tested. A lower cap also reduces the flux that a kept reaction can carry. -## Effect of degraded input +### Effect of `force_on` -Noise changed the model less than sparse input. At σ=1.0 and σ=2.0 the model kept 7812 and 7768 reactions against 7777 for clean data, passed all 69 tasks and shared a Jaccard index of 0.952 and 0.919 with the clean model. Dropout of 50 % and 70 % reduced the model to 5968 and 5113 reactions (Jaccard 0.713 and 0.594), and downsampling of 50 % and 70 % to 6765 and 6123 reactions (Jaccard 0.815 and 0.728). At equal fractions dropout removed more reactions than downsampling. The difference matches the scores involved, −5 for a zero level against −2 for a reaction without measured genes, but this explanation was not tested. Under dropout and downsampling the model passed 67 or 68 of 69 tasks; the failing tasks and the cause of the failures were not examined. +The setting `force_on` changes the MILP itself, through the on and off constraints. From 0.02 to 0.5 the first-step solve took 24 to 25 s, the objective moved by at most 0.0002 and the reaction set shared a Jaccard index of 0.987 or more with the default 0.1. The full pipeline was not run with other `force_on` values. -At 70 % dropout no tested lever moved the model back towards the clean model: the Jaccard index stayed at 0.593 to 0.600, the size at 5110 to 5159 reactions and the task fraction at 0.986. Under dropout the dropped genes stay in the data with a zero level and score −5, so `no_gene_score` acts only on reactions without gene rules and on reactions whose genes are all absent from the data. The study concludes that no parameter recovers the expression evidence that dropout removes; this is an inference from the four lever settings, which do not include other parameters. The lever runs did not include downsampled input, so the effect of `no_gene_score` on missing genes was not measured. +### Effect of degraded input + +Noise changed the model less than sparse input. At σ=1.0 and σ=2.0 the model kept 7812 and 7768 reactions against 7777 for clean data, passed all 69 tasks and shared a Jaccard index of 0.952 and 0.919 with the clean model. Dropout of 50 % and 70 % reduced the model to 5968 and 5113 reactions (Jaccard 0.713 and 0.594), and downsampling of 50 % and 70 % to 6765 and 6123 reactions (Jaccard 0.815 and 0.728). At equal fractions dropout removed more reactions than downsampling. The difference matches the scores involved, −5 for a zero level against −2 for a reaction without measured genes, but this explanation was not tested. Under dropout and downsampling the model passed 67 or 68 of 69 tasks, and the failing tasks and the cause of the failures were not examined. + +At 70 % dropout no tested lever moved the model back towards the clean model. The Jaccard index stayed at 0.593 to 0.600, the size at 5110 to 5159 reactions and the task fraction at 0.986. Under dropout the dropped genes stay in the data with a zero level and score −5, so `no_gene_score` acts only on reactions without gene rules and on reactions whose genes are all absent from the data. The conclusion that no parameter recovers the expression evidence that dropout removes is an inference from the four lever settings, which do not include other parameters. The lever runs did not include downsampled input, so the effect of `no_gene_score` on missing genes was not measured. In raven-toolbox commit `94f3940`, `ftinit` calls `fill_tasks` without a time limit, so the gap-fill uses the default of `fill_tasks`, 300 s per task (`_FILL_TIME_LIMIT`), and not the `time_limit` of the extraction. The time limit that the 90 % dropout run of more than 75 min used is not recorded. -## Choice of settings +### Choice of settings -The defaults `big_m=1000`, `force_on=0.1` and scaling on with `max_stoich_diff=25` are RAVEN's values; every `big_m` from 50 to 1000 was feasible with scaling on, and `big_m=25` was infeasible. `big_m=1000` is the cap that RAVEN's `ftINITInternalAlg` uses (`bigMCap`). Scaling is not needed for feasibility at `big_m=1000`; in these single runs the pipeline took 83 s with scaling and 157 s without. In the first-step solve, `mip_gap=0.001` reached an objective within 0.0001 of the tightest gap; a named series ignores the setting. The full-pipeline runs used `time_limit=600` and the robustness script uses `time_limit=120`; no full-pipeline run took longer than 157 s in total. For noisy input the tested settings needed no change, and no tested setting compensated for dropout. +The defaults `big_m=1000`, `force_on=0.1` and scaling on with `max_stoich_diff=25` are the values of RAVEN. Every `big_m` from 50 to 1000 was feasible with scaling on, and `big_m=25` was infeasible. The cap `big_m=1000` is the one that `ftINITInternalAlg` of RAVEN uses (`bigMCap`). Scaling is not needed for feasibility at `big_m=1000`, and in these single runs the pipeline took 83 s with scaling and 157 s without. In the first-step solve, `mip_gap=0.001` reached an objective within 0.0001 of the tightest gap, and a named series ignores the setting. The full-pipeline runs used `time_limit=600` and the robustness script uses `time_limit=120`. No full-pipeline run took longer than 157 s in total. For noisy input the tested settings needed no change, and no tested setting compensated for dropout. ## Gene-scoring constants @@ -201,18 +223,22 @@ Expression values become gene scores through score = clip(factor × ln(level / reference), min_score, max_score) ``` -with `factor=5.0`, `max_score=10.0` and `min_score=-5.0`, the same in both implementations. The logarithm is natural (base e) in `init/score.py` and in RAVEN's `scoreModel`. The source of each implementation cites no paper for the three constants. A 2012 paper (Wang et al. 2012, [mCADRE](https://doi.org/10.1186/1752-0509-6-153)) uses categorical scoring and is not the origin. The constants are RAVEN's own values without a literature source. +with `factor=5.0`, `max_score=10.0` and `min_score=-5.0`, the same in both implementations. The logarithm is natural (base e) in `init/score.py` and in `scoreModel` of RAVEN. The source of each implementation cites no paper for the three constants. A 2012 paper (Wang et al. 2012, [mCADRE](https://doi.org/10.1186/1752-0509-6-153)) uses categorical scoring and is not the origin. The constants are values of RAVEN itself without a literature source. ## Comparison with the solver benchmark -[ftINIT on different solvers](init-solver-benchmark.md) runs the same full pipeline with each solver interface. With Gurobi it took 92 s and kept 7312 reactions, against 83 s and 7312 reactions here. The benchmark time also includes loading the prepared model and scoring the reactions, and its limit is 900 s against 600 s here. A limit shortens only the 5000 s runs of the schedule, and no Gurobi run in either study ran for its overall limit. The warning `ftINIT step 1 hit the 600s time limit` appears 14 times in 13 full-pipeline and validation logs, and the Gurobi run of the solver benchmark has the same warning with 900 s. The logged `step 1` is the second step. The warning prints the `time_limit` argument, not the limit of the run that stopped; the stopped run is a shorter scheduled run, inferred from the total times of 83 s and 92 s, and the logs do not say which. One scheduled solve per run stopped at a limit. The Gurobi run of the solver benchmark at `big_m=100` kept 7324 reactions, the count of the two `big_m=100` runs here, whose reaction set shares a Jaccard index of 0.981 with the `big_m=1000` set. The solver comparison itself, with GLPK at 2182 s and a Jaccard index of 0.892 against Gurobi and the failure of HiGHS through the `scipy` interface, is on that page. +[ftINIT on different solvers](init-solver-benchmark.md) runs the same full pipeline with each solver interface. With Gurobi it took 92 s and kept 7312 reactions, against 83 s and 7312 reactions here. The benchmark time also includes loading the prepared model and scoring the reactions, and its limit is 900 s against 600 s here. A limit shortens only the 5000 s runs of the schedule, and no Gurobi run in either study ran for its overall limit. + +The warning `ftINIT step 1 hit the 600s time limit` appears 14 times in 13 full-pipeline and validation logs, and the Gurobi run of the solver benchmark has the same warning with 900 s. The logged `step 1` is the second step. The warning prints the `time_limit` argument and not the limit of the run that stopped. The stopped run is a shorter scheduled run, inferred from the total times of 83 s and 92 s, and the logs do not say which. One scheduled solve per run stopped at a limit. + +The Gurobi run of the solver benchmark at `big_m=100` kept 7324 reactions, the count of the two `big_m=100` runs here, whose reaction set shares a Jaccard index of 0.981 with the `big_m=1000` set. The solver comparison itself is on that page: GLPK at 2182 s with a Jaccard index of 0.892 against Gurobi, and the failure of HiGHS through the `scipy` interface. ## Limits -- One model (Human-GEM), one cell line (HCT116) and one solver (Gurobi library 13.0.3 from the gurobipy wheel, most likely; the logs print no version). -- Most settings ran once; the repeats are listed with the full-pipeline table. This set-up has no seed repeats. The full-pipeline differences between caps are larger than the one seed pair available, which comes from a different set-up, so the seed-to-seed spread of this set-up is unknown. -- The robustness runs used a Human-GEM commit that is not recorded and `big_m=100`; the script at the pinned commit passes no `big_m`, so a run with it uses the default 1000. Each degradation is one random draw at one seed. -- The first-step solve shows no difference between caps or scaling settings that the full pipeline shows, so first-step results alone do not predict feasibility of the pipeline. +- One model (Human-GEM), one cell line (HCT116) and one solver (Gurobi library 13.0.3 from the gurobipy wheel, most likely, because the logs print no version). +- Most settings ran once, and the repeats are listed with the full-pipeline table. This set-up has no repeats of the seed. The full-pipeline differences between caps are larger than the one seed pair available, which comes from another set-up, so the seed-to-seed spread of this set-up is unknown. +- The robustness runs used a Human-GEM commit that is not recorded and `big_m=100`. The script at the pinned commit passes no `big_m`, so a run with it uses the default 1000. Each degradation is one random draw at one seed. +- The first-step solve shows no difference between caps or scaling settings that the full pipeline shows, so first-step results alone do not predict the feasibility of the pipeline. - The robustness tables have no run times, and the effect of `no_gene_score` on downsampled input was not measured. ## Reproduction @@ -230,4 +256,4 @@ python summ.py $WORK/out python compare.py --out $WORK/out --shared $WORK/shared --logs $WORK/logs ``` -The tags of the other full-pipeline runs are listed in the README: `--big-m` 25, 50, 250 and 500 at `--gap 0.003`, the `mip_gap` runs 0.001, 0.003 and 0.01 at `big_m=1000`, and the `--noabs` repeats. The unscaled preparation of `scale` takes about 26 min. The robustness tables have no reproduction command: the state of `analyze_init_robustness.py` and the inputs that produced them are not recoverable. +The tags of the other full-pipeline runs are listed in the README: `--big-m` 25, 50, 250 and 500 at `--gap 0.003`, the `mip_gap` runs 0.001, 0.003 and 0.01 at `big_m=1000`, and the `--noabs` repeats. The unscaled preparation of `scale` takes about 26 min. The robustness tables have no reproduction command, because the state of `analyze_init_robustness.py` and the inputs that produced them are not recoverable. From 84d9097e85f5d453cd73349158a4f28b031d33c8 Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Sun, 1 Nov 2026 11:48:29 +0100 Subject: [PATCH 15/20] Write the ftINIT reproducibility page anew --- .../studies/ftinit-determinism.md | 128 +++++++++--------- 1 file changed, 65 insertions(+), 63 deletions(-) diff --git a/docs/parameter-tuning/studies/ftinit-determinism.md b/docs/parameter-tuning/studies/ftinit-determinism.md index a179400..c55592e 100644 --- a/docs/parameter-tuning/studies/ftinit-determinism.md +++ b/docs/parameter-tuning/studies/ftinit-determinism.md @@ -1,12 +1,14 @@ # ftINIT reproducibility with `resolve_ties` and `prove_abs_gap` -This study builds Human-GEM models for one cell line with raven-toolbox's `ftinit` under three settings (default, `resolve_ties=True`, and `resolve_ties=True` with `prove_abs_gap`), varies the solver seed and the template, and records how many reactions and predicted essential genes change between builds and how long each build takes. The API documentation is in `raven_toolbox.init.ftinit`. +The study builds Human-GEM models for one cell line with the `ftinit` of raven-toolbox under three settings: the default, `resolve_ties=True`, and `resolve_ties=True` with `prove_abs_gap`. It varies the solver seed and the template, and records how many reactions and predicted essential genes change between builds and how long each build takes. The API documentation is in `raven_toolbox.init.ftinit`. -Two properties are measured. Determinism is that the same input, run again or with another seed, machine or solver build, gives the same model. Stability is that a similar input (a slightly edited template, or a comparable sample) gives a similar model. Both parameters are opt-in (`resolve_ties=False` and `prove_abs_gap=None` by default) and target determinism. In the measurements, the seed-to-seed difference fell from 110 reactions and 17 essential genes (default) to 41 reactions and 1 gene with `resolve_ties`, and to 1 reaction and 0 genes with `prove_abs_gap=1.0` added, at about 3.6 and 26 times the default build time. Neither parameter made the model stable under a change of the input: a small template edit moved a re-extraction by more reactions than a change of seed (156 against 110 at baseline, 59 against 41 with `resolve_ties`, 16 against 1 with `prove_abs_gap`) and by a similar number of essential genes (17 against 17, 1 against 1, 1 against 0). raven-toolbox has no parameter that anchors a re-extraction to a prior build. +Two properties are measured. Determinism means that the same input gives the same model when it is run again or with another seed, machine or solver build. Stability means that a similar input, such as a slightly edited template or a comparable sample, gives a similar model. Both parameters are opt-in (`resolve_ties=False` and `prove_abs_gap=None` by default) and target determinism. + +The seed-to-seed difference fell from 110 reactions and 17 essential genes (default) to 41 reactions and 1 gene with `resolve_ties`, and to 1 reaction and 0 genes with `prove_abs_gap=1.0` added. The build times were about 3.6 and 26 times the default build time. Neither parameter made the model stable under a change of the input. A small template edit moved a re-extraction by more reactions than a change of seed (156 against 110 at baseline, 59 against 41 with `resolve_ties`, 16 against 1 with `prove_abs_gap`) and by a similar number of essential genes (17 against 17, 1 against 1, 1 against 0). raven-toolbox has no parameter that anchors a re-extraction to a prior build. ## Main results -Each build in the next two tables is the full pipeline, including the task gap-fill, on Human-GEM and the DLD1 cell line (set-up in the next section). The seed-to-seed difference compares the build at seed 1234 with the build at seed 7. The repeat compares two builds at seed 1234 with nothing changed between them; it was run for two settings. For the baseline, the repeat (job A-base-rep) consists of two further builds (236.6 s and 230.9 s), separate from the seed 1234 build of the table (237.1 s). For `resolve_ties`, the repeat consists of two further builds (865 s and 861 s), separate from the seed 1234 build of the table (864 s). +Each build in the next two tables is the full pipeline, including the task gap-fill, on Human-GEM and the DLD1 cell line (the set-up is in the next section). The seed-to-seed difference compares the build at seed 1234 with the build at seed 7. The repeat compares two builds at seed 1234 with nothing changed between them, and it was run for two settings. For the baseline, the repeat (job A-base-rep) consists of two further builds (236.6 s and 230.9 s), separate from the seed 1234 build of the table (237.1 s). For `resolve_ties`, the repeat consists of two further builds (865 s and 861 s), separate from the seed 1234 build of the table (864 s). | Setting | Seed-to-seed difference, reactions | Seed-to-seed difference, essential genes | Essential genes (seed 1234 / seed 7) | Repeat difference at seed 1234, reactions / genes | Build time (seed 1234 / seed 7) | |---|---:|---:|---:|---:|---:| @@ -15,11 +17,11 @@ Each build in the next two tables is the full pipeline, including the task gap-f | `resolve_ties=True, prove_abs_gap=1.0` | 1 | 0 | 125 / 125 | not run | 5742 s / 6354 s | | `resolve_ties=True, prove_abs_gap=0.05` | 1 | 0 | 125 / 125 | not run | 5793 s / 6589 s | -The two `prove_abs_gap` values gave the same result. The build time grew from about 4 minutes (baseline) to about 14 minutes with `resolve_ties` (3.6 times) and about 100 minutes with `prove_abs_gap` added (26 times). +The two `prove_abs_gap` values gave the same result. The build time grew from about 4 minutes (baseline) to about 14 minutes with `resolve_ties` (3.6 times) and to about 100 minutes with `prove_abs_gap` added (26 times). -The two baseline builds at one seed differed by 2 reactions. Each baseline build had one run of the final step that stopped at its 5 s wall-clock limit; a run stopped at a time limit can return a different solution depending on machine load, and this is the presumed source of the 2 reactions (not tested). The two `resolve_ties` builds of the repeat had identical reaction sets. +The two baseline builds at one seed differed by 2 reactions. Each baseline build had one run of the final step that stopped at its 5 s wall-clock limit. A run stopped at a time limit can return a different solution depending on the machine load, and this is the presumed source of the 2 reactions (not tested). The two `resolve_ties` builds of the repeat had identical reaction sets. -The template edit removed 30 reactions that the reference builds did not use. They were drawn at random (seed 0) from the 2387 reactions of the prepared reference model that the build with `resolve_ties=True, prove_abs_gap=1.0` (seed 1234, 7754 reactions, 125 essential genes) did not keep and that are not task-essential. Of the 30, the `resolve_ties` builds kept 0 and the baseline build kept 1. Each setting was re-extracted from the edited template at seed 1234 and compared with its build from the unedited template: +The template edit removed 30 reactions that the reference builds did not use. They were drawn at random (seed 0) from the 2387 reactions of the prepared reference model that the build with `resolve_ties=True, prove_abs_gap=1.0` (seed 1234, 7754 reactions, 125 essential genes) did not keep and that are not task-essential. Of the 30, the `resolve_ties` builds kept 0 and the baseline build kept 1. Each setting was re-extracted from the edited template at seed 1234 and compared with its build from the unedited template. | Setting | Reaction difference | Essential-gene difference | Essential genes (unedited → edited) | Build time (edited) | |---|---:|---:|---:|---:| @@ -27,50 +29,27 @@ The template edit removed 30 reactions that the reference builds did not use. Th | `resolve_ties=True` | 59 | 1 | 114 → 115 | 638 s | | `resolve_ties=True, prove_abs_gap=1.0` | 16 | 1 | 125 → 126 | 4108 s | -The unedited builds hardly used the removed reactions. If the re-extraction changed only what the edit forces, the baseline model would lose the one removed reaction it kept and the `resolve_ties` models would lose none (inference, not tested). The baseline moved 156 reactions and 17 essential-gene calls; the two settings with `resolve_ties=True` moved 59 and 16 reactions and 1 gene each. +The unedited builds hardly used the removed reactions. If the re-extraction changed only what the edit forces, the baseline model would lose the one removed reaction it kept and the `resolve_ties` models would lose none (inference, not tested). The baseline moved 156 reactions and 17 essential-gene calls, and the two settings with `resolve_ties=True` moved 59 and 16 reactions and 1 gene each. ## Set-up -* Model and data: Human-GEM `v2.0.0-38`, Hart2015 DLD1 cell line, 57 essential tasks, series `1+1`. The `model/Human-GEM.yml` of that commit has 12877 reactions and 2848 genes. The preparation is the one of `estimateEssentialGenes.py` at that commit (removal of drug, amino-acid-triplet and duplicate complex I reactions), followed by raven-toolbox's own `prep_init_model`, which gives a reference model of 10141 reactions and a merged model of 6861 reactions (chains of linked reactions combined into one), with 206 task-essential reactions. This preparation differs from RAVEN's exported `raven_refModel.xml`, so absolute counts are not comparable to the [Human-GEM cell-type model validation against RAVEN](humangem-validation.md) run, which uses that export. -* Steps: series `1+1` runs two extraction steps, counted from 1 here; the debug log counts from 0. Step 1 decides the reactions with a gene association; step 2, the last step, adds the reactions without one and holds the reactions of step 1 on. -* Settings: `big_m=100` (the default is 1000), `mip_gap_abs=10` and `time_limit=1800` s, the values of `estimateEssentialGenes.py` at that commit. A named series such as `1+1` ignores `mip_gap` and `mip_gap_abs`; they apply to a step without a run schedule. -* Escalation: each solve stops at a mixed-integer programming (MIP) gap, the distance between the objective of the best solution found and the solver's bound on the optimum, either relative (divided by the objective) or absolute. Without `prove_abs_gap`, each step follows RAVEN's run schedule, here called the escalation. The first run uses a relative gap of 0.0004; each later run uses the larger of its relative gap and its absolute gap (10 or 20) divided by the objective, and the step stops as soon as the gap already reached meets the next target. `time_limit` caps the limit of each run: step 1 has runs of 120 s and 1800 s, the final step runs of 5 s, 120 s and 1800 s. -* With `prove_abs_gap`, each step is one solve proven to that absolute gap, with the 1800 s limit for the primary solve and for the two flux phases of `resolve_ties` (phases 2c and 2d), and half of it (`_TIE_BREAK_TIME_FRACTION = 0.5`) for its two on/off phases (phases 2a and 2b); the phases are defined in the section on the parameters. The task gap-fill uses a fixed 300 s per task in every setting. -* Solver: Gurobi through gurobipy, `Threads=1` unless stated, raven-toolbox at commit `94f39407`. Gurobi library 13.0.3 from the gurobipy wheel (most likely); the logs print no version. The builds ran as Slurm jobs on AMD EPYC 9354 nodes shared with other jobs, so wall-clock figures are approximate. Most jobs had 2 CPUs and the `Threads=4` jobs of the first-step probe had 4. Several builds ran in sequence in one job (the seeds of a setting, or the three builds of the stability test). -* Determinism probe: the extraction `seed` was varied (1234, 7, and further seeds where stated) with inputs, code, solver build and machine held fixed. A difference between builds then has two sources: the solver's choice among equal optima, measured where two builds reached the same objective value, and different objective values reached within the gap and time limits (see the location of the degeneracy and the sweep of `prove_abs_gap`). Differences between Gurobi versions were not measured. -* Metrics: the reaction difference is the symmetric difference of the kept-reaction sets (reactions kept by one build and not the other). The essential-gene difference is the symmetric difference of the sets of genes whose single deletion is predicted essential by `cobra.flux_analysis.find_essential_genes` at its default threshold. The gene difference is the result a user sees; the reaction difference is what the parameters act on. -* Debug log: all builds ran with `FTINIT_DEBUG=1`, which logs the status of each solve and of each tie-break phase. An incumbent is the best solution the solver holds when it stops; at a time limit it is not proven optimal. +- **Model and data.** Human-GEM `v2.0.0-38`, Hart2015 DLD1 cell line, 57 essential tasks, series `1+1`. The `model/Human-GEM.yml` of that commit has 12877 reactions and 2848 genes. The preparation is the one of `estimateEssentialGenes.py` at that commit (removal of the drug, amino-acid-triplet and duplicate complex I reactions), followed by `prep_init_model` of raven-toolbox. This gives a reference model of 10141 reactions and a merged model of 6861 reactions, in which chains of linked reactions are combined into one, with 206 task-essential reactions. The preparation differs from the exported `raven_refModel.xml` of RAVEN, so absolute counts are not comparable to the [Human-GEM cell-type model validation against RAVEN](humangem-validation.md), which uses that export. +- **Steps.** Series `1+1` runs two extraction steps, counted from 1 here; the debug log counts from 0. Step 1 decides the reactions with a gene association. Step 2, the last step, adds the reactions without one and holds the reactions of step 1 on. +- **Settings.** `big_m=100` (the default is 1000), `mip_gap_abs=10` and `time_limit=1800` s, the values of `estimateEssentialGenes.py` at that commit. A named series such as `1+1` ignores `mip_gap` and `mip_gap_abs`, which apply to a step without a run schedule. +- **Escalation.** Each solve stops at a mixed-integer programming (MIP) gap, the distance between the objective of the best solution found and the bound of the solver on the optimum, either relative (divided by the objective) or absolute. Without `prove_abs_gap`, each step follows the run schedule of RAVEN, called the escalation here. The first run uses a relative gap of 0.0004. Each later run uses the larger of its relative gap and its absolute gap (10 or 20) divided by the objective, and the step stops as soon as the gap already reached meets the next target. The setting `time_limit` caps the limit of each run: step 1 has runs of 120 s and 1800 s, and the final step runs of 5 s, 120 s and 1800 s. +- **`prove_abs_gap`.** Each step is one solve proven to that absolute gap. It has the 1800 s limit for the primary solve and for the two flux phases of `resolve_ties` (phases 2c and 2d), and half of it (`_TIE_BREAK_TIME_FRACTION = 0.5`) for its two on/off phases (phases 2a and 2b). The phases are defined in the section on the parameters. The task gap-fill uses a fixed 300 s per task in every setting. +- **Solver.** Gurobi through gurobipy, `Threads=1` unless stated, raven-toolbox at commit `94f39407`. The Gurobi library is 13.0.3 from the gurobipy wheel (most likely, because the logs print no version). The builds ran as Slurm jobs on AMD EPYC 9354 nodes shared with other jobs, so wall-clock figures are approximate. Most jobs had 2 CPUs, and the `Threads=4` jobs of the first-step probe had 4. Several builds ran in sequence in one job: the seeds of a setting, or the three builds of the stability test. +- **Determinism probe.** The extraction `seed` was varied (1234, 7, and further seeds where stated) with inputs, code, solver build and machine held fixed. A difference between builds then has two sources: the choice of the solver among equal optima, measured where two builds reached the same objective value, and different objective values reached within the gap and time limits (see the location of the degeneracy and the sweep of `prove_abs_gap`). Differences between Gurobi versions were not measured. +- **Metrics.** The reaction difference is the symmetric difference of the kept-reaction sets, that is the reactions kept by one build and not by the other. The essential-gene difference is the symmetric difference of the sets of genes whose single deletion is predicted essential by `cobra.flux_analysis.find_essential_genes` at its default threshold. The gene difference is the result that a user sees, and the reaction difference is what the parameters act on. +- **Debug log.** All builds ran with `FTINIT_DEBUG=1`, which logs the status of each solve and of each tie-break phase. An incumbent is the best solution that the solver holds when it stops, and at a time limit it is not proven optimal. ## Degeneracy of the extraction MILP -The extraction mixed-integer linear program (MILP) has many optima of equal score. In the first step on Human-GEM and DLD1, 1547 merged reactions have a negative score, and these carry the 0/1 keep-or-drop variables. Of them, 98.3 % (1520) are in blocks of equal score. The largest block has 840 reactions at −5.0. The second has 420 reactions at −2.0, the default score of a reaction without gene association; across the five Hart2015 cell lines it has 419 to 421 members, so the expression data change its size by at most 2 members. Many reaction subsets reach the same optimum, and the solver returns one of them. - -`Threads=1` and a fixed `Seed` make that choice close to reproducible on one machine: the two baseline builds at one seed differed by 2 reactions. Another seed can select another member of the tied set, or end the solve at another objective value within the gap and time limits; both were seen in the first-step probe, and in either case the predicted essential genes change for reasons unrelated to the model or the data. Another Gurobi version or platform is expected to act the same way; that was not measured. - -## Parameters `resolve_ties` and `prove_abs_gap` - -`resolve_ties` adds a lexicographic phase 2 after the primary solve of each extraction step, with the score objective held at its optimum. The phases are, in order: - -* `phase2a-parsimony`: minimise the count of kept removable (negative-score) reactions. -* `phase2b-idrank`: among those, minimise the summed id rank of the kept reactions. A merged reaction is ranked by the smallest original id in its group (`rank_keys`), so the rank does not depend on which member names the merged reaction. -* `phase2c-minflux`: for that set of kept reactions, minimise the total absolute flux (gap 1e-6), so the fluxes that the next step reads for its forced directions and magnitudes do not depend on the solver's vertex. -* `phase2d-direction`: among the minimal-flux solutions, minimise the id-rank-weighted backward flux of the reactions whose flux can take either sign (gap 1e-6), so a flux that carries the same total either way gets a fixed direction. - -The last step skips `phase2c-minflux` and `phase2d-direction`, because no later step reads its fluxes. The task gap-fill uses the first two phases. Phases 2a and 2b get half of `time_limit` each; phases 2c and 2d get the full `time_limit`. - -`prove_abs_gap` proves each step to a fixed absolute MIP gap in a single solve. It replaces the escalation, whose final run, at an objective near zero, accepts any incumbent within its gap. - -A phase that does not prove its optimum has these outcomes: - -* A phase that stops at `time_limit` with an incumbent is adopted, and a `UserWarning` names it as an unproven incumbent. `phase2a-parsimony` and `phase2b-idrank` share one warning, which is raised when `phase2b-idrank` ends with an incumbent and not otherwise; `phase2c-minflux` and `phase2d-direction` each have their own. -* A flux phase (`phase2c-minflux`, `phase2d-direction`) that finds no solution warns with its name and keeps the fluxes of the phase before it. -* When `phase2a-parsimony` or `phase2b-idrank` finds no incumbent, the on/off tie-break of that solve is dropped without a warning: the primary solution is kept and the ties of that step are not resolved. A build without a warning can therefore hold an unresolved step. -* When the primary solve of a step stops at `time_limit`, all tie-break phases of that solve are skipped and the primary solution is kept. -* The escalation raises no warning either. In the four sweep runs with the escalation and `resolve_ties=True`, the step 1 objective ended 4.0 to 12.0 below the proven optimum; at least one of the 12 first-step runs without `resolve_ties` reached it (the location of the degeneracy). +The extraction mixed-integer linear program (MILP) has many optima of equal score. In the first step on Human-GEM and DLD1, 1547 merged reactions have a negative score, and these carry the 0/1 keep-or-drop variables. Of them, 98.3 % (1520) are in blocks of equal score. The largest block has 840 reactions at −5.0. The second has 420 reactions at −2.0, the default score of a reaction without gene association. Across the five Hart2015 cell lines it has 419 to 421 members, so the expression data change its size by at most 2 members. Many reaction subsets reach the same optimum, and the solver returns one of them. -For comparison, RAVEN's `ftINIT` with `resolveTies` skips the tie-break after a primary solve that stopped at the time limit, gives the count and id-rank phases half of `params.TimeLimit` each, skips the minimal-flux and direction phases in the last step, and names every unproven phase (`phase2a-parsimony`, `phase2b-idrank`, `phase2c-minflux`, `phase2d-direction`) in its "tie resolution did not converge" warning. In RAVEN's `src/INIT/ftINITInternalAlg.m`, the skip after a primary solve at the time limit is at line 483, the half limit at line 518 and the warning at line 662; `src/INIT/ftINIT.m` line 227 sets `pinFluxes` to false in the last step. RAVEN's `ftINIT` was not run in this study. +The settings `Threads=1` and a fixed `Seed` make that choice close to reproducible on one machine, because the two baseline builds at one seed differed by 2 reactions. Another seed can select another member of the tied set, or end the solve at another objective value within the gap and time limits. Both were seen in the first-step probe, and in either case the predicted essential genes change for reasons unrelated to the model or the data. Another Gurobi version or platform is expected to act the same way, which was not measured. -## Location of the degeneracy +### Location of the degeneracy Six seeds were run through the first step alone (a one-step series, so the flux phases of `resolve_ties` do not run), with and without `resolve_ties`, at `Threads` 1 and 4. The mean pairwise difference is the reaction difference averaged over the 15 pairs of seeds. @@ -85,9 +64,32 @@ At the same seed, the baseline solutions at `Threads` 1 and 4 were 5 to 20 react Grouping the runs by objective value separates the tie-break from the primary solve. The 12 baseline runs (both thread settings) reached six objective values between 45887.113 and 45899.113. At a fixed objective value, the baseline returned 4 distinct solutions from 4 runs at 45892.113 (pairwise differences 6 to 60 reactions) and 4 from 4 at 45894.113 (35 to 53). With `resolve_ties`, the 4 runs at each of these two objective values returned one solution each. The 12 `resolve_ties` runs gave 5 distinct reaction sets, and the remaining differences came from runs whose primary solve reached different objective values. +## Parameters `resolve_ties` and `prove_abs_gap` + +The parameter `resolve_ties` adds a lexicographic phase 2 after the primary solve of each extraction step, with the score objective held at its optimum. The phases are, in order: + +- `phase2a-parsimony` minimises the count of kept removable (negative-score) reactions. +- `phase2b-idrank` minimises, among those, the summed id rank of the kept reactions. A merged reaction is ranked by the smallest original id in its group (`rank_keys`), so the rank does not depend on which member names the merged reaction. +- `phase2c-minflux` minimises, for that set of kept reactions, the total absolute flux (gap 1e-6), so the fluxes that the next step reads for its forced directions and magnitudes do not depend on the vertex of the solver. +- `phase2d-direction` minimises, among the minimal-flux solutions, the id-rank-weighted backward flux of the reactions whose flux can take either sign (gap 1e-6), so a flux that carries the same total either way gets a fixed direction. + +The last step skips `phase2c-minflux` and `phase2d-direction`, because no later step reads its fluxes. The task gap-fill uses the first two phases. Phases 2a and 2b get half of `time_limit` each, and phases 2c and 2d get the full `time_limit`. + +The parameter `prove_abs_gap` proves each step to a fixed absolute MIP gap in a single solve. It replaces the escalation, whose final run, at an objective near zero, accepts any incumbent within its gap. + +A phase that does not prove its optimum has these outcomes: + +- A phase that stops at `time_limit` with an incumbent is adopted, and a `UserWarning` names it as an unproven incumbent. The phases `phase2a-parsimony` and `phase2b-idrank` share one warning, which is raised when `phase2b-idrank` ends with an incumbent and not otherwise. The phases `phase2c-minflux` and `phase2d-direction` each have their own. +- A flux phase (`phase2c-minflux`, `phase2d-direction`) that finds no solution warns with its name and keeps the fluxes of the phase before it. +- When `phase2a-parsimony` or `phase2b-idrank` finds no incumbent, the on/off tie-break of that solve is dropped without a warning. The primary solution is kept and the ties of that step are not resolved, so a build without a warning can hold an unresolved step. +- When the primary solve of a step stops at `time_limit`, all tie-break phases of that solve are skipped and the primary solution is kept. +- The escalation raises no warning either. In the four sweep runs with the escalation and `resolve_ties=True`, the step 1 objective ended 4.0 to 12.0 below the proven optimum. At least one of the 12 first-step runs without `resolve_ties` reached it (see the location of the degeneracy). + +For comparison, `ftINIT` of RAVEN with `resolveTies` skips the tie-break after a primary solve that stopped at the time limit, gives the count and id-rank phases half of `params.TimeLimit` each, and skips the minimal-flux and direction phases in the last step. Its warning "tie resolution did not converge" names every unproven phase (`phase2a-parsimony`, `phase2b-idrank`, `phase2c-minflux`, `phase2d-direction`). In `src/INIT/ftINITInternalAlg.m` of RAVEN, the skip after a primary solve at the time limit is at line 483, the half limit at line 518 and the warning at line 662, and `src/INIT/ftINIT.m` line 227 sets `pinFluxes` to false in the last step. `ftINIT` of RAVEN was not run in this study. + ## Effect of the `prove_abs_gap` value -The value was swept with `resolve_ties=True` on the two-step pipeline without gap-fill, at four seeds (1234, 7, 1, 2) per value. "`None`" is the escalation with `resolve_ties=True`. A step time includes the tie-break phases of that step. All 48 solves of step 1 and step 2 with a `prove_abs_gap` value ended with status `optimal`. +The value was swept with `resolve_ties=True` on the two-step pipeline without gap-fill, at four seeds (1234, 7, 1, 2) per value. The row `None` is the escalation with `resolve_ties=True`. A step time includes the tie-break phases of that step. All 48 solves of step 1 and step 2 with a `prove_abs_gap` value ended with status `optimal`. | `prove_abs_gap` | Step 1 objectives (4 seeds) | Step 2 objectives (4 seeds) | Step 1 time | Step 2 time | |---|---|---|---:|---:| @@ -99,15 +101,15 @@ The value was swept with `resolve_ties=True` on the two-step pipeline without ga | 0.1 | 45899.113 (4) | −613.872 (3), −611.872 | 3313 to 3650 s | 1757 to 2721 s | | 0.05 | 45899.113 (4) | −613.872 (3), −611.872 | 3317 to 3650 s | 1758 to 2734 s | -None of the four escalation runs reached the proven step 1 optimum of 45899.113; they ended 4.0 to 12.0 below it, without a warning. `prove_abs_gap=5.0` stopped 2.0 below it for one seed, and every value from 2.0 to 0.05 reached it for all four seeds. The step 2 objectives depend on the step 1 result, so they are comparable between rows where step 1 is equal. With a proven step 1, step 2 gave −613.872 for three seeds and −611.872 for seed 7. In step 1 of the `prove_abs_gap=1.0` builds, `phase2c-minflux` stopped at the time limit for 3 of the 4 seeds and was optimal for 1, and `phase2d-direction` stopped at the time limit for all 4, so the fluxes passed to step 2 were unproven incumbents; whether this causes the seed 7 difference was not tested. +None of the four escalation runs reached the proven step 1 optimum of 45899.113. They ended 4.0 to 12.0 below it, without a warning. The value `prove_abs_gap=5.0` stopped 2.0 below it for one seed, and every value from 2.0 to 0.05 reached it for all four seeds. The step 2 objectives depend on the step 1 result, so they are comparable between rows where step 1 is equal. With a proven step 1, step 2 gave −613.872 for three seeds and −611.872 for seed 7. In step 1 of the `prove_abs_gap=1.0` builds, `phase2c-minflux` stopped at the time limit for 3 of the 4 seeds and was optimal for 1, and `phase2d-direction` stopped at the time limit for all 4. The fluxes passed to step 2 were therefore unproven incumbents, and whether this causes the seed 7 difference was not tested. All values from 2.0 to 0.05 gave the same objectives and the same final reaction sets. A tighter gap added little run time: step 2 took up to 2734 s at 0.05 against 2500 s at 1.0, and a whole build took 5770 s against 5640 s on average. Per-phase times were not logged, so the split of step 2 between the proof and the tie-break is not known. -The final reaction sets of the sweep (without gap-fill) were: four distinct sets with the escalation, 37 to 56 reactions apart; three with `prove_abs_gap=5.0`; two with every value from 2.0 to 0.05, which differ by one reaction (seed 7 against the other three). +The final reaction sets of the sweep (without gap-fill) were four distinct sets with the escalation, 37 to 56 reactions apart, three with `prove_abs_gap=5.0`, and two with every value from 2.0 to 0.05, which differ by one reaction (seed 7 against the other three). ## Tie-break phases at genome scale -With a proven step 1 objective, the tie-break phases did not all prove within the time limit. The phase statuses of the eight tie-broken solves (two steps, four seeds) at `prove_abs_gap=1.0` were as follows; the other values from 0.05 to 5.0 gave the same counts. +With a proven step 1 objective, the tie-break phases did not all prove within the time limit. The table gives the phase statuses of the eight tie-broken solves (two steps, four seeds) at `prove_abs_gap=1.0`. The other values from 0.05 to 5.0 gave the same counts. | Phase | Optimal | Stopped at `time_limit` | |---|---:|---:| @@ -116,7 +118,7 @@ With a proven step 1 objective, the tie-break phases did not all prove within th | `phase2c-minflux` (step 1) | 1 | 3 | | `phase2d-direction` (step 1) | 0 | 4 | -Step 2 is the last step, so it skips the last two phases. The log of seed 1234: +Step 2 is the last step, so it skips the last two phases. The log of seed 1234 is: ``` [ftinit] final solve: obj=45899.11 status=optimal achieved_gap=2.8e-06 @@ -129,11 +131,11 @@ Step 2 is the last step, so it skips the last two phases. The log of seed 1234: [ftinit] tie-break phase2b-idrank: status=optimal obj=346311.0 ``` -The on/off tie-break of step 1 was proven for all four seeds, so the step 1 reaction set is fixed. `phase2c-minflux` reached the 1800 s limit for 3 of the 4 seeds and `phase2d-direction` for all 4, and each was adopted as an unproven incumbent. In the three seeds in which `phase2c-minflux` stopped at the limit, the two flux phases at the 1800 s limit take 3600 s, which accounts for most of the step 1 time of about 3650 s and for a build time of about 100 minutes. The flux that step 2 reads is therefore the incumbent of a time-limited solve. The step 2 objective differed by seed (−611.872 for seed 7, −613.872 for the other three) and the final sets by one reaction. The fluxes themselves were not compared between seeds; a seed-dependent flux is inferred from the step 2 objectives, not measured. +The on/off tie-break of step 1 was proven for all four seeds, so the step 1 reaction set is fixed. The phase `phase2c-minflux` reached the 1800 s limit for 3 of the 4 seeds and `phase2d-direction` for all 4, and each was adopted as an unproven incumbent. In the three seeds in which `phase2c-minflux` stopped at the limit, the two flux phases at the 1800 s limit take 3600 s, which accounts for most of the step 1 time of about 3650 s and for a build time of about 100 minutes. The flux that step 2 reads is therefore the incumbent of a time-limited solve. The step 2 objective differed by seed (−611.872 for seed 7, −613.872 for the other three) and the final sets by one reaction. The fluxes themselves were not compared between seeds, and a seed-dependent flux is inferred from the step 2 objectives and not measured. -The escalation builds with `resolve_ties=True` also adopted phases at a time limit, here the limit of the escalation run (120 s in step 1, so 60 s for phases 2a and 2b). The full-pipeline build of seed 1234 adopted 3 phases (`phase2b-idrank`, `phase2c-minflux`, `phase2d-direction` of step 1) and the build of seed 7 adopted 2 (`phase2c-minflux`, `phase2d-direction` of step 1). In step 2 of both builds, `phase2a-parsimony` ended at the time limit with an incumbent and `phase2b-idrank` ended at the time limit with no incumbent (`A-rt-1234.log`, `A-rt-7.log`). `phase2b-idrank` then returns no solution, the tie-break of step 2 returns `False`, the primary solution of step 2 is kept, and no warning is raised. The reaction sets of these two builds rest on a resolved step 1 and an unresolved step 2. +The escalation builds with `resolve_ties=True` also adopted phases at a time limit, here the limit of the escalation run (120 s in step 1, so 60 s for phases 2a and 2b). The full-pipeline build of seed 1234 adopted 3 phases (`phase2b-idrank`, `phase2c-minflux`, `phase2d-direction` of step 1) and the build of seed 7 adopted 2 (`phase2c-minflux`, `phase2d-direction` of step 1). In step 2 of both builds, `phase2a-parsimony` ended at the time limit with an incumbent and `phase2b-idrank` ended at the time limit with no incumbent (`A-rt-1234.log`, `A-rt-7.log`). The phase `phase2b-idrank` then returns no solution, the tie-break of step 2 returns `False`, the primary solution of step 2 is kept, and no warning is raised. The reaction sets of these two builds rest on a resolved step 1 and an unresolved step 2. -Counts of full builds (gap-fill included) with at least one unproven tie-break phase or a time-limited primary solve. The full builds of the main results are, per setting: baseline 5 (seeds 1234 and 7, the two builds of the repeat, the edited template), `resolve_ties=True` 5 (seeds 1234 and 7, the two builds of the repeat, the edited template), `prove_abs_gap` 5 (seeds 1234 and 7 at each of 1.0 and 0.05, the edited template at 1.0). The Builds column gives how many of them were counted. +The next table counts full builds (gap-fill included) with at least one unproven tie-break phase or a time-limited primary solve. The full builds of the main results are, per setting: baseline 5 (seeds 1234 and 7, the two builds of the repeat, the edited template), `resolve_ties=True` 5 (seeds 1234 and 7, the two builds of the repeat, the edited template), and `prove_abs_gap` 5 (seeds 1234 and 7 at each of 1.0 and 0.05, the edited template at 1.0). The column Builds gives how many of them were counted. | Setting | Builds | With an adopted phase at `time_limit` | With a primary solve at `time_limit` | With a tie-break dropped without a warning | |---|---:|---:|---:|---:| @@ -143,35 +145,35 @@ Counts of full builds (gap-fill included) with at least one unproven tie-break p Each row counts all builds of its setting, including the edited build at 1.0 for `prove_abs_gap`. The step 2 logs cited are those of seeds 1234 and 7. -The warnings cover the adopted phases and the flux phases without a solution. A build without a warning can hold an unresolved tie-break (a dropped `phase2a-parsimony` or `phase2b-idrank`) or an unproven escalation, and the debug log is the record of what each solve reached. In the sweep, all 24 builds with `prove_abs_gap` raised at least one warning. A larger `time_limit` is the setting that can let the phases prove; its effect was not measured. +The warnings cover the adopted phases and the flux phases without a solution. A build without a warning can hold an unresolved tie-break (a dropped `phase2a-parsimony` or `phase2b-idrank`) or an unproven escalation, and the debug log is the record of what each solve reached. In the sweep, all 24 builds with `prove_abs_gap` raised at least one warning. A larger `time_limit` is the setting that can let the phases prove, and its effect was not measured. ## Stability under a small template change The template-edit result is in the second table of the main results. The two settings with `resolve_ties=True` moved fewer reactions than the baseline, and for each setting the edit moved more reactions than a change of seed. Determinism does not give stability: a deterministic extractor returns the same model for the same input, and the edit changes the input. -Settings on identical inputs also differ. Baseline and `resolve_ties` at seed 1234 differ by 162 reactions and 17 essential-gene calls (125 against 114), with the same step 1 objective (45892.113) and different step 2 objectives (−657.240 for the baseline, −616.906 for `resolve_ties=True`). The step 1 difference is a different choice among solutions of equal score. The step 2 difference includes more than that choice, because step 2 reads the fluxes of step 1 and ends at a different objective. +Settings on identical inputs also differ. Baseline and `resolve_ties` at seed 1234 differ by 162 reactions and 17 essential-gene calls (125 against 114), with the same step 1 objective (45892.113) and different step 2 objectives (−657.240 for the baseline, −616.906 for `resolve_ties=True`). The step 1 difference is a different choice among solutions of equal score. The step 2 difference includes more than that choice, because step 2 reads the fluxes of step 1 and ends at another objective. The growth objective of the models did not indicate stability on this data: the unedited builds of one setting (`resolve_ties=True`) had 74.6 for seed 1234 and 83.1 for seed 7. -For curation of a template (fix one or a few reactions, re-extract, compare), the re-extraction difference contains the effect of the edit together with re-selection among tied solutions. Applying the same edit to the extracted model gives a control: the difference between that edited model and the unedited extraction contains the effect of the edit alone. No result for this control is reported. +For the curation of a template (fix one or a few reactions, re-extract, compare), the re-extraction difference contains the effect of the edit together with the re-selection among tied solutions. Applying the same edit to the extracted model gives a control: the difference between that edited model and the unedited extraction contains the effect of the edit alone. No result for this control is reported. ## Choice of settings -* `resolve_ties=True`: the repeat pair at seed 1234 had identical reaction sets, and the seed-to-seed difference fell from 17 genes to 1 on this data, at about 3.6 times the build time (4 to 14 minutes). Its warnings name the adopted phases that stopped at the time limit. A tie-break that finds no incumbent in phase 2a or 2b is dropped without a warning; the builds at seeds 1234 and 7 dropped the one of step 2, and the table of build counts gives 5 of 5 builds with a dropped step 2 tie-break. With the escalation, the tie-break runs again in each run of the schedule (`_solve_step` in `ftinit.py` and the run schedule in `steps.py`): the flux phases of step 1 get 120 s when step 1 stops after its first run and 1800 s when a second run follows (its 5000 s capped by `time_limit`). -* `prove_abs_gap=1.0` together with `resolve_ties=True` reduced the seed-to-seed difference from 41 to 1 reaction (1 to 0 genes) at 26 times the baseline time, about 100 minutes at `time_limit=1800`. The objectives of both steps are proven to 1.0 (in the four sweep runs with the escalation and `resolve_ties=True`, step 1 ended 4.0 to 12.0 below the optimum). In the four `prove_abs_gap=1.0` sweep builds (step 1 and step 2, no gap-fill; 4987 to 6183 s wall clock), step 1 took 3305 to 3650 s, 59.0 to 66.3 % of each build, mostly in the two flux phases, and step 2 another 1650 to 2500 s. A value from 2.0 to 0.05 gave the same models; 5.0 stopped 2.0 below the optimum for one seed. No build with `prove_abs_gap` and without `resolve_ties` was run. -* Gurobi versions were not compared; two baseline builds with everything held fixed differed by 2 reactions. +- **`resolve_ties=True`.** The repeat pair at seed 1234 had identical reaction sets, and the seed-to-seed difference fell from 17 genes to 1 on this data, at about 3.6 times the build time (4 to 14 minutes). Its warnings name the adopted phases that stopped at the time limit. A tie-break that finds no incumbent in phase 2a or 2b is dropped without a warning. The builds at seeds 1234 and 7 dropped the one of step 2, and the table of build counts gives 5 of 5 builds with a dropped step 2 tie-break. With the escalation, the tie-break runs again in each run of the schedule (`_solve_step` in `ftinit.py` and the run schedule in `steps.py`). The flux phases of step 1 get 120 s when step 1 stops after its first run and 1800 s when a second run follows (its 5000 s capped by `time_limit`). +- **`prove_abs_gap=1.0` with `resolve_ties=True`.** The seed-to-seed difference fell from 41 to 1 reaction (1 to 0 genes) at 26 times the baseline time, which is about 100 minutes at `time_limit=1800`. The objectives of both steps are proven to 1.0, whereas in the four sweep runs with the escalation and `resolve_ties=True` step 1 ended 4.0 to 12.0 below the optimum. In the four `prove_abs_gap=1.0` sweep builds (step 1 and step 2, no gap-fill; 4987 to 6183 s wall clock), step 1 took 3305 to 3650 s, 59.0 to 66.3 % of each build, mostly in the two flux phases, and step 2 took another 1650 to 2500 s. A value from 2.0 to 0.05 gave the same models, and 5.0 stopped 2.0 below the optimum for one seed. No build with `prove_abs_gap` and without `resolve_ties` was run. +- **Gurobi versions.** They were not compared. Two baseline builds with everything held fixed differed by 2 reactions. ## Limits -The full-pipeline determinism table rests on one seed pair (seeds 1234 and 7), the first-step probe on six seeds and the sweep on four. The study used one cell line, one machine type, one Gurobi build and one `time_limit`, and one edit (30 unused reactions removed) for the stability measurement. Each figure comes from one build per seed and setting, except the repeat pairs at seed 1234, so the data give direction and rough magnitude and no error bars. +The determinism table of the full pipeline rests on one seed pair (seeds 1234 and 7), the first-step probe on six seeds and the sweep on four. The study used one cell line, one machine type, one Gurobi build and one `time_limit`, and one edit (30 unused reactions removed) for the stability measurement. Each figure comes from one build per seed and setting, except the repeat pairs at seed 1234, so the data give direction and rough magnitude and no error bars. -Differences between Gurobi versions, the failure that `resolve_ties` and `prove_abs_gap` target, were not measured; the seed probe stands in for them, and how well it represents them is not known. The stability edit removed 30 of 10141 reference reactions, of which the reference builds kept at most 1. Two comparable samples (for example, two expression profiles of one tissue) were not compared. +Differences between Gurobi versions, the failure that `resolve_ties` and `prove_abs_gap` target, were not measured. The seed probe stands in for them, and how well it represents them is not known. The stability edit removed 30 of 10141 reference reactions, of which the reference builds kept at most 1. Two comparable samples, for example two expression profiles of one tissue, were not compared. -Wall-clock figures come from jobs that shared nodes with other jobs and are approximate; the difference and objective columns are set comparisons and do not depend on load. The `prove_abs_gap` build times depend on the `time_limit` of 1800 s, because the flux phases of step 1 run to the limit; no other limit was measured. RAVEN's `ftINIT` was not run. +Wall-clock figures come from jobs that shared nodes with other jobs and are approximate. The difference and objective columns are set comparisons and do not depend on the load. The build times with `prove_abs_gap` depend on the `time_limit` of 1800 s, because the flux phases of step 1 run to the limit, and no other limit was measured. `ftINIT` of RAVEN was not run. ## Reproduction -Requires Gurobi and a Human-GEM checkout. The numbers come from `det/prep.py` (preparation of the template with the essential tasks, about 2000 s), `det/build.py` (one build or repeats of `ftinit` on DLD1 with `big_m=100`, `mip_gap_abs=10` and `time_limit=1800`), `det/select_edit.py` (the 30 removed reactions of the template edit, seed 0), `det/ties.py` (tied-score blocks) and the summaries `det/analyse.py` and `det/det_extra.py`. The scripts are in the raven-docs repository under `scripts/parameter_tuning/init/`. The campaign takes about 100 core-hours. The commands run through the Slurm wrappers as in the README of that folder, with `WORK`, `TOOLBOX_ROOT`, `HUMANGEM_ROOT`, `PY`, `DET_SCRIPTS` and `PREP_DIR` set: +The reproduction requires Gurobi and a Human-GEM checkout. The numbers come from `det/prep.py` (preparation of the template with the essential tasks, about 2000 s), `det/build.py` (one build or repeats of `ftinit` on DLD1 with `big_m=100`, `mip_gap_abs=10` and `time_limit=1800`), `det/select_edit.py` (the 30 removed reactions of the template edit, seed 0), `det/ties.py` (tied-score blocks) and the summaries `det/analyse.py` and `det/det_extra.py`. The scripts are in the raven-docs repository under `scripts/parameter_tuning/init/`. The campaign takes about 100 core-hours. The commands run through the Slurm wrappers as in the README of that folder, with `WORK`, `TOOLBOX_ROOT`, `HUMANGEM_ROOT`, `PY`, `DET_SCRIPTS` and `PREP_DIR` set: ```bash jid=$(sbatch --parsable -J det-prep -o $WORK/logs/prep.log $DET_SCRIPTS/prep.sbatch) @@ -182,7 +184,7 @@ python analyse.py $WORK/out python det_extra.py $WORK/out $WORK/logs ``` -The tables are computed from the outputs and logs by `analyse.py` and `det_extra.py`; the degeneracy counts are read from `ties.log`. `ties.py` and `select_edit.py` need the base preparation `prep-base.pkl`, which is not archived and takes about 2000 s to build. `det/build.py` produced the build numbers; series `1+1` ignores `mip_gap_abs`, which the call below passes as `mip_gap_abs=10.0`. +The tables are computed from the outputs and logs by `analyse.py` and `det_extra.py`, and the degeneracy counts are read from `ties.log`. The scripts `ties.py` and `select_edit.py` need the base preparation `prep-base.pkl`, which is not archived and takes about 2000 s to build. The script `det/build.py` produced the build numbers. Series `1+1` ignores `mip_gap_abs`, which the call below passes as `mip_gap_abs=10.0`. ```python from raven_toolbox.init import ftinit @@ -201,6 +203,6 @@ m1 = ftinit(prep_edited, rxn_scores=rxn_scores, resolve_ties=True, prove_abs_gap print(len({r.id for r in m0.reactions} ^ {r.id for r in m1.reactions})) ``` -`rxn_scores` and `gene_scores` come from `score_reactions_from_genes` and `gene_scores_from_expression` (original reaction id → score). `prep` comes from `prep_init_model(template, tasks, ...)`; `prep_edited` is built the same way from the edited template. +The inputs `rxn_scores` and `gene_scores` come from `score_reactions_from_genes` and `gene_scores_from_expression` (original reaction id → score). The input `prep` comes from `prep_init_model(template, tasks, ...)`, and `prep_edited` is built the same way from the edited template. -Set `FTINIT_DEBUG=1` to log the status of each solve and each tie-break phase, which shows whether a build rests on a proven optimum or an unproven incumbent. The `UserWarning` messages name the adopted phases at the time limit and miss a dropped tie-break. +Setting `FTINIT_DEBUG=1` logs the status of each solve and each tie-break phase, which shows whether a build rests on a proven optimum or an unproven incumbent. The `UserWarning` messages name the adopted phases at the time limit and miss a dropped tie-break. From ba3a209e9ad932286df47dfcf6f6f9398eb00884 Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Sun, 1 Nov 2026 11:49:21 +0100 Subject: [PATCH 16/20] Write the KEGG calibration page anew --- .../studies/kegg-hmm-cutoff-calibration.md | 74 ++++++++++--------- 1 file changed, 39 insertions(+), 35 deletions(-) diff --git a/docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md b/docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md index 7058f4c..95fbc06 100644 --- a/docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md +++ b/docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md @@ -1,12 +1,14 @@ # KEGG HMM cut-off calibration -KEGG-based reconstruction from sequence assigns each protein of a new organism to KEGG Orthology (KO) groups by searching the proteome against profile hidden Markov models (HMMs) of the KOs. The study ran this search on the proteomes of four organisms that KEGG annotates, assigned genes to KOs at a range of cut-offs and score ratios, and compared the result with KEGG's own gene→KO annotation of each organism. For each setting it records precision, recall and F1 of the gene→KO pairs, the fraction of the organism's annotated reactions that the prediction recovers and the number of predicted reactions outside the annotation. The results set the defaults `cutoff=1e-30` and `min_score_ratio_g=0.9` of `assign_kos` in place of the RAVEN 2.11.3 values `1e-50` and `0.8`. +KEGG-based reconstruction from sequence assigns each protein of a new organism to KEGG Orthology (KO) groups by searching the proteome against profile hidden Markov models (HMMs) of the KOs. The study ran this search on the proteomes of four organisms that KEGG annotates, assigned genes to KOs at a range of cut-offs and score ratios, and compared the result with the gene→KO annotation of KEGG for each organism. For each setting it records the precision, recall and F1 of the gene→KO pairs, the fraction of the annotated reactions of the organism that the prediction recovers, and the number of predicted reactions outside the annotation. + +The results set the defaults `cutoff=1e-30` and `min_score_ratio_g=0.9` of `assign_kos`, in place of the RAVEN 2.11.3 values `1e-50` and `0.8`. The defaults raise gene→KO recall by 0.01 to 0.09 and lower precision in three of the four organisms (by 0.01 to 0.07), and F1 changes by at most 0.02. ## Set-up ### Parameters under test -`assign_kos` (`reconstruction/kegg/query.py`, called by `get_kegg_model_from_sequences`) turns a KO×gene E-value matrix into gene→KO assignments in three steps. RAVEN's `getKEGGModelForOrganism` applies the same steps with the arguments `cutOff`, `minScoreRatioKO` and `minScoreRatioG`. +The function `assign_kos` (`reconstruction/kegg/query.py`, called by `get_kegg_model_from_sequences`) turns a KO×gene E-value matrix into gene→KO assignments in three steps. `getKEGGModelForOrganism` of RAVEN applies the same steps with the arguments `cutOff`, `minScoreRatioKO` and `minScoreRatioG`. 1. `cutoff` keeps hits with `evalue <= cutoff`. 2. `min_score_ratio_ko` drops, within a KO, genes whose `log(evalue)/log(best_evalue_in_KO) < min_score_ratio_ko`. @@ -16,14 +18,14 @@ KEGG-based reconstruction from sequence assigns each protein of a new organism t The percentile, cut-off, score-ratio and defaults tables come from one run on 2026-11-01 (raven-toolbox commit `94f3940`, HMMER 3.4, 16 threads on an AMD EPYC 9354 node) of the sources below. The timing and the true-reaction counts come from separate jobs. -- Tables: the kegg118 artefacts (`organism_gene_ko`, `ko_reaction` and the reference model) published as the `kegg118` release of toolbox-assets. -- HMM libraries: the published kegg118 libraries `kegg118_eukaryotes.hmm` (2698 profiles, 696,233,564 bytes) and `kegg118_prokaryotes.hmm` (4205 profiles, 838,505,885 bytes), built by the maintainers with `build_ko_hmm`. The sequences of each KO were clustered with CD-HIT at 90 % identity and aligned with MAFFT FFT-NS-2, or with MAFFT PartTree for KOs whose alignment exceeds a memory budget. -- Queries: the CDS genes of each organism from the KEGG REST lists, downloaded on 2026-11-01 (KEGG genes updated 2026-10-31, which is not KEGG release 118): 6021, 5010, 4288 and 476 sequences, of 6449, 5094, 4639 and 525 list entries. The sequences were not compared with those of KEGG release 118. The search is `hmmsearch`; `run_hmmsearch` in raven-toolbox fixes `-Z` to the number of profiles. Whether these E-values equal those of `hmmscan` against the same library was not checked. -- Each organism's hits were computed once, and every setting was applied to the same hit table. The differences between settings therefore contain no search-to-search variation. For `sce` the parsed hit tables were identical in four runs at 16, 12, 4 and 1 threads; the other organisms were searched once. +- **Tables.** The kegg118 artefacts (`organism_gene_ko`, `ko_reaction` and the reference model), published as the `kegg118` release of toolbox-assets. +- **HMM libraries.** The published kegg118 libraries `kegg118_eukaryotes.hmm` (2698 profiles, 696,233,564 bytes) and `kegg118_prokaryotes.hmm` (4205 profiles, 838,505,885 bytes), built by the maintainers with `build_ko_hmm`. The sequences of each KO were clustered with CD-HIT at 90 % identity and aligned with MAFFT FFT-NS-2, or with MAFFT PartTree for KOs whose alignment exceeds a memory budget. +- **Queries.** The CDS genes of each organism from the KEGG REST lists, downloaded on 2026-11-01 (KEGG genes updated 2026-10-31, which is not KEGG release 118): 6021, 5010, 4288 and 476 sequences, of 6449, 5094, 4639 and 525 list entries. The sequences were not compared with those of KEGG release 118. The search is `hmmsearch`, and `run_hmmsearch` in raven-toolbox fixes `-Z` to the number of profiles. Whether these E-values equal those of `hmmscan` against the same library was not checked. +- **Hit tables.** The hits of each organism were computed once, and every setting was applied to the same hit table, so the differences between settings contain no variation between searches. For `sce` the parsed hit tables were identical in four runs at 16, 12, 4 and 1 threads, and the other organisms were searched once. ### Reference and scores -The reference is the **KEGG annotation**: the organism's gene→KO pairs in the `organism_gene_ko` table of the kegg118 artefacts. The table lists the KOs linked to at least one reaction in `ko_reaction`, and the HMM libraries are built from the same table. The organism's **true reactions** are the reactions linked to its annotated KOs, limited to reactions that are in the kegg118 reference model (`--restrict-to-reference`). The limit is applied because reaction recall is measured against reactions that the draft can contain. +The reference is the **KEGG annotation**, which is the gene→KO pairs of the organism in the `organism_gene_ko` table of the kegg118 artefacts. The table lists the KOs linked to at least one reaction in `ko_reaction`, and the HMM libraries are built from the same table. The **true reactions** of the organism are the reactions linked to its annotated KOs, limited to reactions that are in the kegg118 reference model (`--restrict-to-reference`). The limit is applied because the reaction recall is measured against reactions that the draft can contain. A predicted pair is the output of `assign_kos`, with the `organism:` prefix stripped from the gene identifier so that it matches the reference. Each predicted pair is either **matched** (present in the annotation) or **novel** (absent from it). @@ -48,7 +50,7 @@ A novel pair counts as an error, although some novel pairs may be correct assign Without the limit to the reference model, the published `ko_reaction` table (7642 reactions, against 12,373 in the reference model) gives 1407, 1357, 1772 and 274 true reactions for the four organisms. The reaction columns of the tables below follow the limited definition. -`sce` and `eco` are model organisms; `cme` and `mge` are less studied. `mge` has the smallest proteome of the four and the weakest matched hits in the E-value table. +The organisms `sce` and `eco` are model organisms, and `cme` and `mge` are less studied. The organism `mge` has the smallest proteome of the four and the weakest matched hits in the E-value table. ## Results @@ -69,11 +71,11 @@ The table gives percentiles of `log10(E-value)` of the best hit per (gene, KO) p The median matched pair had an E-value between 1e-102 (`mge`) and 1e-153 (`sce`), and the weakest 1 % of matched pairs lay between 1e-15 and 1e-36. The median novel pair lay between 1e-4 and 1e-8. The weakest 1 % of matched pairs was 11 (`mge`) to 29 (`eco`) orders of magnitude stronger than the median novel pair. -The RAVEN 2.11.3 cut-off of `1e-50` falls inside the matched distribution: between p95 and p99 for `sce` and `eco`, between p90 and p95 for `cme`, and between p50 and p90 for `mge`. It therefore removes 1 to 5 % of the matched pairs of `sce` and `eco`, 5 to 10 % of `cme` and 10 to 50 % of `mge`. The default `1e-30` lies beyond p99 for `eco` (fewer than 1 % of the matched pairs removed) and between p95 and p99 for `sce`, `cme` and `mge`, and is 22 to 26 orders of magnitude stronger than the median novel pair. +The RAVEN 2.11.3 cut-off of `1e-50` falls inside the matched distribution: between p95 and p99 for `sce` and `eco`, between p90 and p95 for `cme`, and between p50 and p90 for `mge`. It therefore removes 1 to 5 % of the matched pairs of `sce` and `eco`, 5 to 10 % of `cme` and 10 to 50 % of `mge`. The default `1e-30` lies beyond p99 for `eco` (fewer than 1 % of the matched pairs removed) and between p95 and p99 for `sce`, `cme` and `mge`, and it is 22 to 26 orders of magnitude stronger than the median novel pair. ### Cut-off sweep -`min_score_ratio_ko = 0.3` and `min_score_ratio_g = 0.8` (the RAVEN 2.11.3 values) were fixed while `cutoff` varied. +The values `min_score_ratio_ko = 0.3` and `min_score_ratio_g = 0.8` (the RAVEN 2.11.3 values) were fixed while `cutoff` varied. #### `sce` @@ -119,11 +121,11 @@ The RAVEN 2.11.3 cut-off of `1e-50` falls inside the matched distribution: betwe | 1e-70 | 0.85 | 0.73 | 0.78 | 0.72 | 27 | | 1e-100 | 0.88 | 0.53 | 0.66 | 0.55 | 21 | -Gene→KO recall was between 0.95 and 0.99 from 1e-10 to 1e-30 in all four organisms and fell at stricter cut-offs. The fall was largest for `mge`, whose reaction recall dropped from 0.98 at 1e-30 to 0.88 at 1e-50 and 0.55 at 1e-100. Precision rose with every stricter cut-off in all four organisms, from 0.73 to 0.78 for `sce` between 1e-30 and 1e-50. With the ratios fixed at 0.3 and 0.8, F1 was highest at 1e-50 or 1e-70 in every organism. Novel predictions remained at the strictest cut-off: at 1e-100 the three larger proteomes had 61 to 137 novel reactions and a gene→KO precision of 0.80 to 0.85 (0.88 for `mge`). The study does not show whether these are wrong assignments, paralogs or genes that KEGG has not annotated. +Gene→KO recall was between 0.95 and 0.99 from 1e-10 to 1e-30 in all four organisms and fell at stricter cut-offs. The fall was largest for `mge`, whose reaction recall dropped from 0.98 at 1e-30 to 0.88 at 1e-50 and 0.55 at 1e-100. Precision rose with every stricter cut-off in all four organisms, for example from 0.73 to 0.78 for `sce` between 1e-30 and 1e-50. With the ratios fixed at 0.3 and 0.8, F1 was highest at 1e-50 or 1e-70 in every organism. Novel predictions remained at the strictest cut-off: at 1e-100 the three larger proteomes had 61 to 137 novel reactions and a gene→KO precision of 0.80 to 0.85 (0.88 for `mge`). The study does not show whether these are wrong assignments, paralogs or genes that KEGG has not annotated. ### Score-ratio sweep -`cutoff` was fixed at `1e-50`. The script scores every combination of `min_score_ratio_ko` (0.0, 0.3, 0.5) and `min_score_ratio_g` (0.5, 0.8, 0.95); the table holds the five combinations per organism that are reported. +The value `cutoff` was fixed at `1e-50`. The script scores every combination of `min_score_ratio_ko` (0.0, 0.3, 0.5) and `min_score_ratio_g` (0.5, 0.8, 0.95), and the table holds the five combinations per organism that are reported. | organism | ko ratio | g ratio | gKO prec | gKO rec | gKO F1 | |---|---|---|---|---|---| @@ -148,13 +150,13 @@ Gene→KO recall was between 0.95 and 0.99 from 1e-10 to 1e-30 in all four organ | `mge` | 0.3 | 0.80 | 0.77 | 0.85 | 0.81 | | `mge` | 0.5 | 0.80 | 0.77 | 0.85 | 0.81 | -At `min_score_ratio_g = 0.80`, raising `min_score_ratio_ko` from 0.0 to 0.5 raised precision by 0.02 (`sce`), 0.01 (`cme`), 0.03 (`eco`) and 0.00 (`mge`), and lowered recall by 0.01 for `cme` and `eco` and not at all for `sce` and `mge`. +At `min_score_ratio_g = 0.80`, raising `min_score_ratio_ko` from 0.0 to 0.5 raised the precision by 0.02 (`sce`), 0.01 (`cme`), 0.03 (`eco`) and 0.00 (`mge`), and lowered the recall by 0.01 for `cme` and `eco` and not at all for `sce` and `mge`. -Raising `min_score_ratio_g` from 0.80 to 0.95 (at `min_score_ratio_ko = 0.0`) raised precision by 0.06 (`sce`), 0.08 (`cme`), 0.10 (`eco`) and 0.05 (`mge`), and lowered recall by 0.00 to 0.02. At 0.50, precision was the lowest of the three values in every organism. The value 0.9 lies between the tested values 0.8 and 0.95 and was not swept. +Raising `min_score_ratio_g` from 0.80 to 0.95 (at `min_score_ratio_ko = 0.0`) raised the precision by 0.06 (`sce`), 0.08 (`cme`), 0.10 (`eco`) and 0.05 (`mge`), and lowered the recall by 0.00 to 0.02. At 0.50 the precision was the lowest of the three values in every organism. The value 0.9 lies between the tested values 0.8 and 0.95 and was not swept. ### Defaults compared with RAVEN 2.11.3 -The table compares the RAVEN 2.11.3 settings `(1e-50, 0.3, 0.8)` with the defaults `(1e-30, 0.3, 0.9)`, given as `(cutoff, min_score_ratio_ko, min_score_ratio_g)`. All values are scores of the saved hit tables; the RAVEN 2.11.3 values are the 1e-50 rows of the cut-off sweep. +The table compares the RAVEN 2.11.3 settings `(1e-50, 0.3, 0.8)` with the defaults `(1e-30, 0.3, 0.9)`, given as `(cutoff, min_score_ratio_ko, min_score_ratio_g)`. All values are scores of the saved hit tables, and the RAVEN 2.11.3 values are the 1e-50 rows of the cut-off sweep. | organism | gKO prec | gKO rec | gKO F1 | rxn rec | rxn novel | |---|---|---|---|---|---| @@ -163,11 +165,11 @@ The table compares the RAVEN 2.11.3 settings `(1e-50, 0.3, 0.8)` with the defaul | `eco` | 0.68 → 0.69 | 0.96 → 0.97 | 0.79 → 0.81 | 0.98 → 0.99 | 208 → 173 | | `mge` | 0.77 → 0.70 | 0.85 → 0.94 | 0.81 → 0.80 | 0.88 → 0.97 | 30 → 33 | -With the defaults, gene→KO recall rose by 0.09 for `mge` and by 0.01 to 0.03 for the other three organisms. Gene→KO precision fell by 0.01 (`sce`), 0.02 (`cme`) and 0.07 (`mge`), and rose by 0.01 for `eco`. F1 changed by at most 0.01 for `sce`, `cme` and `mge` and rose by 0.02 for `eco`. All scores are rounded to two decimals, so a difference of 0.01 lies within the rounding. The number of novel reactions fell for `sce`, `cme` and `eco` and rose from 30 to 33 for `mge`. +With the defaults, the gene→KO recall rose by 0.09 for `mge` and by 0.01 to 0.03 for the other three organisms. The gene→KO precision fell by 0.01 (`sce`), 0.02 (`cme`) and 0.07 (`mge`), and rose by 0.01 for `eco`. The F1 changed by at most 0.01 for `sce`, `cme` and `mge` and rose by 0.02 for `eco`. All scores are rounded to two decimals, so a difference of 0.01 lies within the rounding. The number of novel reactions fell for `sce`, `cme` and `eco` and rose from 30 to 33 for `mge`. ### Whole-model comparison of the two KEGG paths -*S. cerevisiae* was reconstructed in full by two paths, both with the RAVEN 2.11.3 settings `(1e-50, 0.3, 0.8)`. The annotation path takes the organism's genes and KOs from the `organism_gene_ko` table; the HMM path assigns KOs to the proteome with the HMM search. +*S. cerevisiae* was reconstructed in full by two paths, both with the RAVEN 2.11.3 settings `(1e-50, 0.3, 0.8)`. The annotation path takes the genes and KOs of the organism from the `organism_gene_ko` table, and the HMM path assigns KOs to the proteome with the HMM search. | Count | annotation path | HMM path | |---|---|---| @@ -175,33 +177,35 @@ With the defaults, gene→KO recall rose by 0.09 for `mge` and by 0.01 to 0.03 f | metabolites | 1502 | 1565 | | genes | 838 | 899 | -The HMM path contained 96.2 % of the reactions of the annotation path (1306 of 1357, Jaccard index 0.87) and 96.5 % of its genes (809 of 838, Jaccard index 0.87). The counts were measured with the published kegg118 tables and libraries. The search of `sce` against the eukaryote library is timed under [Settings without a sweep](#settings-without-a-sweep); the prokaryote library was not timed. +The HMM path contained 96.2 % of the reactions of the annotation path (1306 of 1357, Jaccard index 0.87) and 96.5 % of its genes (809 of 838, Jaccard index 0.87). The counts were measured with the published kegg118 tables and libraries. The search of `sce` against the eukaryote library is timed under [Settings without a sweep](#settings-without-a-sweep), and the prokaryote library was not timed. + +## Interpretation -## Meaning of the results +From 1e-10 to 1e-30 a stricter cut-off raised the precision while the gene→KO recall fell by at most 0.03 (from 0.98 or 0.99 to between 0.95 and 0.99). Beyond 1e-30 the cut-off traded recall for precision: it removed matched pairs together with novel pairs, and novel pairs remained at every cut-off tested. Across the whole sweep from 1e-10 to 1e-100 the precision rose by 0.28 (`sce`, 0.57 to 0.85) to 0.35 (`mge`). The RAVEN 2.11.3 value `1e-50` removes 10 to 50 % of the matched pairs of `mge`, the organism with the weakest matched hits, and 1 to 10 % in the other three. -From 1e-10 to 1e-30 a stricter cut-off raised precision while gene→KO recall fell by at most 0.03 (from 0.98 or 0.99 to between 0.95 and 0.99). Beyond 1e-30 the cut-off traded recall for precision: it removed matched pairs together with novel pairs, and novel pairs remained at every cut-off tested. Across the whole sweep from 1e-10 to 1e-100 precision rose by 0.28 (`sce`, 0.57 to 0.85) to 0.35 (`mge`). The RAVEN 2.11.3 value `1e-50` removes 10 to 50 % of the matched pairs of `mge`, the organism with the weakest matched hits, and 1 to 10 % in the other three. +At `cutoff = 1e-50`, raising `min_score_ratio_g` from 0.80 to 0.95 raised the precision by 0.05 to 0.10 for a recall loss of 0.00 to 0.02, and raising it from 0.50 to 0.80 raised the precision by 0.15 to 0.27 (`eco`, 0.40 to 0.67). At `min_score_ratio_g = 0.80` and `cutoff = 1e-50`, raising `min_score_ratio_ko` from 0.0 to 0.5 raised the precision by 0.00 to 0.03 and lowered the recall by 0.00 to 0.01. Its effect at other settings was not reported. -At `cutoff = 1e-50`, raising `min_score_ratio_g` from 0.80 to 0.95 raised precision by 0.05 to 0.10 for a recall loss of 0.00 to 0.02, and raising it from 0.50 to 0.80 raised precision by 0.15 to 0.27 (`eco`, 0.40 to 0.67). At `min_score_ratio_g = 0.80` and `cutoff = 1e-50`, raising `min_score_ratio_ko` from 0.0 to 0.5 raised precision by 0.00 to 0.03 and lowered recall by 0.00 to 0.01; its effect at other settings was not reported. The defaults move part of the filtering from the cut-off to `min_score_ratio_g`. On the four organisms the move gained recall, most for `mge`, at the cost of precision in three of them, and left F1 within 0.01 of the RAVEN 2.11.3 settings for three organisms and raised it by 0.02 for `eco`. +The defaults move part of the filtering from the cut-off to `min_score_ratio_g`. On the four organisms the move gained recall, most for `mge`, at the cost of precision in three of them. It left the F1 within 0.01 of the RAVEN 2.11.3 settings for three organisms and raised it by 0.02 for `eco`. ## Chosen defaults | parameter | RAVEN 2.11.3 | default (raven-toolbox and RAVEN 3) | basis | |---|---|---|---| -| `cutoff` (`cutOff`) | 1e-50 | 1e-30 | beyond matched p99 for `eco`, between p95 and p99 for `sce`, `cme` and `mge`; with `min_score_ratio_g` 0.9, `mge` recall 0.85 → 0.94 | -| `min_score_ratio_g` (`minScoreRatioG`) | 0.8 | 0.9 | from 0.80 to 0.95 it raised precision by 0.05 to 0.10 for a recall loss of 0.00 to 0.02; offsets part of the precision lost to the looser cut-off (inference, 0.9 not swept) | -| `min_score_ratio_ko` (`minScoreRatioKO`) | 0.3 | 0.3 | at `min_score_ratio_g` 0.80 and `cutoff` 1e-50, 0.0 to 0.5 raised precision by 0.00 to 0.03 and lowered recall by 0.00 to 0.01; kept at the RAVEN value | +| `cutoff` (`cutOff`) | 1e-50 | 1e-30 | Beyond matched p99 for `eco`, between p95 and p99 for `sce`, `cme` and `mge`. With `min_score_ratio_g` 0.9, the `mge` recall rises from 0.85 to 0.94 | +| `min_score_ratio_g` (`minScoreRatioG`) | 0.8 | 0.9 | From 0.80 to 0.95 it raised the precision by 0.05 to 0.10 for a recall loss of 0.00 to 0.02. It offsets part of the precision lost to the looser cut-off (inference, 0.9 not swept) | +| `min_score_ratio_ko` (`minScoreRatioKO`) | 0.3 | 0.3 | At `min_score_ratio_g` 0.80 and `cutoff` 1e-50, 0.0 to 0.5 raised the precision by 0.00 to 0.03 and lowered the recall by 0.00 to 0.01. Kept at the RAVEN value | -The defaults favour recall: the changed settings raised recall in every organism and lowered precision in three of the four. +The defaults favour recall: the changed settings raised the recall in every organism and lowered the precision in three of the four. ## Settings without a sweep -Two settings next to the cut-offs, `seq_identity` and the model-assembly flags, were not measured in the study. The reasons given for their values are not tested. `threads` was measured in one configuration. +Two settings next to the cut-offs, `seq_identity` and the model-assembly flags, were not measured in the study, and the reasons given for their values are not tested. The setting `threads` was measured in one configuration. -`threads` in `run_hmmsearch` and `build_ko_hmm` defaults to every CPU the process may run on. `hmmsearch` 3.4 of `sce` (6021 sequences) against `kegg118_eukaryotes.hmm` (2698 profiles) returned 10,293 hit rows for 2198 genes and 1998 KOs at 16, 12, 4 and 1 threads, and the parsed hit tables (KO, gene, E-value) were identical. One run per thread count took 89.7 s (16 threads), 88.6 s (12), 96.4 s (4) and 242.5 s (1), on an AMD EPYC 9354 node used exclusively (2026-11-01). Between 16 and 4 threads the time changes by less than 9 %. `build_ko_hmm` was not timed. +The setting `threads` in `run_hmmsearch` and `build_ko_hmm` defaults to every CPU that the process may run on. `hmmsearch` 3.4 of `sce` (6021 sequences) against `kegg118_eukaryotes.hmm` (2698 profiles) returned 10,293 hit rows for 2198 genes and 1998 KOs at 16, 12, 4 and 1 threads, and the parsed hit tables (KO, gene, E-value) were identical. One run per thread count took 89.7 s (16 threads), 88.6 s (12), 96.4 s (4) and 242.5 s (1), on an AMD EPYC 9354 node used exclusively (2026-11-01). Between 16 and 4 threads the time changes by less than 9 %. The function `build_ko_hmm` was not timed. -`seq_identity` in `build_ko_hmm` is `0.9`: the identity at which CD-HIT collapses near-duplicate sequences within a KO before the HMM is built; the clustering keeps a KO with many near-identical entries from giving an overfitted profile. It is CD-HIT's recommended value for proteins and the default of `seqIdentity` in RAVEN 2.11.3. `seq_identity=-1` skips the clustering. RAVEN 3's `getKEGGModelForOrganism` uses prebuilt HMM libraries and has no clustering step. +The setting `seq_identity` in `build_ko_hmm` is `0.9`. It is the identity at which CD-HIT collapses near-duplicate sequences within a KO before the HMM is built, and the clustering keeps a KO with many near-identical entries from giving an overfitted profile. It is the value that CD-HIT recommends for proteins and the default of `seqIdentity` in RAVEN 2.11.3. The value `seq_identity=-1` skips the clustering. The `getKEGGModelForOrganism` of RAVEN 3 uses prebuilt HMM libraries and has no clustering step. -The model-assembly flags decide which reactions a draft keeps, and have the same defaults as in RAVEN: +The model-assembly flags decide which reactions a draft keeps, and they have the same defaults as in RAVEN. | Flag | Default | What it admits | Reason given for the default | |---|---|---|---| @@ -212,9 +216,9 @@ The model-assembly flags decide which reactions a draft keeps, and have the same ## Limits -- All four organisms are in the set of sequences from which the libraries were built, so their own sequences contribute to their KO profiles. Recall on these organisms is therefore an upper bound for an organism outside KEGG (inference; a genome absent from KEGG was not tested). The comparison between settings is relative and gives no absolute accuracy. -- A novel pair that is a correct assignment KEGG has not made counts as an error, so the precision figures are a lower bound. The study does not measure how many novel pairs are correct. -- Four organisms, the KEGG tables of release 118 and one hit table per organism. For `sce` the parsed hit tables were identical in four runs at 16, 12, 4 and 1 threads; the other organisms were searched once. The scoring of a hit table by `assign_kos` involves no random step. +- All four organisms are in the set of sequences from which the libraries were built, so their own sequences contribute to their KO profiles. The recall on these organisms is therefore an upper bound for an organism outside KEGG (inference; a genome absent from KEGG was not tested). The comparison between settings is relative and gives no absolute accuracy. +- A novel pair that is a correct assignment that KEGG has not made counts as an error, so the precision figures are a lower bound. The study does not measure how many novel pairs are correct. +- The study used four organisms, the KEGG tables of release 118 and one hit table per organism. For `sce` the parsed hit tables were identical in four runs at 16, 12, 4 and 1 threads, and the other organisms were searched once. The scoring of a hit table by `assign_kos` involves no random step. - The score-ratio sweep used `cutoff = 1e-50` only, and `min_score_ratio_g = 0.9` was not part of either sweep. ## Reproduction @@ -229,8 +233,8 @@ The scripts are in [`scripts/parameter_tuning/kegg/`](https://github.com/CompMet | `compare_kegg_paths.py` | the whole-model comparison of the annotation and HMM paths | | `time_hmmsearch.py` | the timing and hit-table identity at several thread counts | -`analyze_hmm_cutoffs.py` is called once per organism with `--artefacts`, `--org`, `--proteome`, `--library` and `--threads`. `--save-tblout` writes the `hmmsearch --tblout` table, `--tblout` scores a saved table without searching, `--json` writes the results, and `--restrict-to-reference` limits `ko_reaction` to reactions of the reference model, which is the definition behind the reaction columns above. The gene→KO columns and the percentiles do not depend on that option. `compare_kegg_paths.py` is called with `--org`, `--proteome`, `--artefacts`, `--library`, `--domain`, `--threads` and `--settings`, for example `--settings 1e-50,0.3,0.8 1e-30,0.3,0.9`. +The script `analyze_hmm_cutoffs.py` is called once per organism with `--artefacts`, `--org`, `--proteome`, `--library` and `--threads`. The option `--save-tblout` writes the `hmmsearch --tblout` table, `--tblout` scores a saved table without searching, `--json` writes the results, and `--restrict-to-reference` limits `ko_reaction` to reactions of the reference model, which is the definition behind the reaction columns above. The gene→KO columns and the percentiles do not depend on that option. The script `compare_kegg_paths.py` is called with `--org`, `--proteome`, `--artefacts`, `--library`, `--domain`, `--threads` and `--settings`, for example `--settings 1e-50,0.3,0.8 1e-30,0.3,0.9`. -The proteomes (`kegg_rest/.faa`, `uniprot/.faa`, `counts.json`) are in the archived input bundle `kegg_proteomes_2026-11-01.tar.gz`, and the four `hmmsearch --tblout` tables in `kegg_hmmsearch_tblout_2026-11-01.tar.gz`. +The proteomes (`kegg_rest/.faa`, `uniprot/.faa`, `counts.json`) are in the archived input bundle `kegg_proteomes_2026-11-01.tar.gz`, and the four `hmmsearch --tblout` tables are in `kegg_hmmsearch_tblout_2026-11-01.tar.gz`. -The scores of the tables come from the run of 2026-11-01 with raven-toolbox commit `94f3940`, HMMER 3.4 and the published kegg118 tables and libraries, searched with 16 threads on an AMD EPYC 9354 node. Rescoring each saved hit table with `--tblout` gives the saved scores exactly. Not checked: whether the KEGG REST sequences of 2026-10-31 equal the sequences of KEGG release 118 (the sequence counts were compared, not the sequences), and whether `hmmsearch -Z` E-values equal `hmmscan` E-values on these inputs, because the binary bundle has no `hmmscan`. The UniProt proteomes in the bundle were downloaded and counted but not searched. +The scores of the tables come from the run of 2026-11-01 with raven-toolbox commit `94f3940`, HMMER 3.4 and the published kegg118 tables and libraries, searched with 16 threads on an AMD EPYC 9354 node. Scoring each saved hit table again with `--tblout` gives the saved scores exactly. Two points were not checked: whether the KEGG REST sequences of 2026-10-31 equal the sequences of KEGG release 118 (the sequence counts were compared, not the sequences), and whether `hmmsearch -Z` E-values equal `hmmscan` E-values on these inputs, because the binary bundle has no `hmmscan`. The UniProt proteomes in the bundle were downloaded and counted but not searched. From 6854456907ee04d060fa375c886e2423b182dd8f Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Sun, 1 Nov 2026 11:50:16 +0100 Subject: [PATCH 17/20] Write the sampling convergence page anew --- .../sampling-convergence-calibration.md | 76 ++++++++++--------- 1 file changed, 40 insertions(+), 36 deletions(-) diff --git a/docs/parameter-tuning/studies/sampling-convergence-calibration.md b/docs/parameter-tuning/studies/sampling-convergence-calibration.md index 8fe6e3c..93e4e67 100644 --- a/docs/parameter-tuning/studies/sampling-convergence-calibration.md +++ b/docs/parameter-tuning/studies/sampling-convergence-calibration.md @@ -1,6 +1,8 @@ # Flux sampling convergence and settings -`random_sampling` in raven-toolbox draws flux vectors from a model with a Markov-chain sampler (`method='achr'` or `'chrr'`) or with random objectives (`method='random_objective'`). This study ran the two Markov-chain samplers on e_coli_core (95 reactions) and yeast-GEM, and recorded how strongly consecutive samples of one chain are correlated, whether independent chains reach the same flux distribution, and the wall time. It also records why two settings of the random-objective method differ from RAVEN MATLAB. +The function `random_sampling` of raven-toolbox draws flux vectors from a model with a Markov-chain sampler (`method='achr'` or `'chrr'`) or with random objectives (`method='random_objective'`). The study ran the two Markov-chain samplers on e_coli_core (95 reactions) and yeast-GEM and recorded three results: how strongly consecutive samples of one chain are correlated, whether independent chains reach the same flux distribution, and the wall time. It also records why two settings of the random-objective method differ from RAVEN in MATLAB. + +On e_coli_core, four chains of 300 samples agree at the R-hat threshold of 1.1 for every reaction, and the CHRR sampler is no better than ACHR there. On yeast-GEM, 50.7 % of the scored reactions exceed an R-hat of 1.1 at the default `thinning`, and no tested setting gave converged samples. ## Terms @@ -10,26 +12,26 @@ - ACHR (Artificially Centered Hit-and-Run) takes its directions from warmup points and a running centre of the chain, and does not transform the polytope. raven-toolbox wraps `cobra.sampling.ACHRSampler`, which computes its warmup points with two linear programs (LPs) per reaction before the first step ([Random sampling](../../guide/sampling.md)). - CHRR (Coordinate Hit-and-Run with Rounding) first runs a flux variability analysis (FVA), fixes the reactions whose range is narrower than `fixed_width_tol`, reduces the polytope to a full-dimensional body through the nullspace of the equality constraints, and rounds it: it maps the body so that its maximum-volume inscribed ellipsoid (MVE) becomes a unit ball. The walk then moves along one coordinate per step. - `thinning` is the number of steps between two stored samples. An ACHR chain with `n_samples` stored samples therefore takes `thinning × n_samples` steps after its warmup points, and a CHRR chain takes `warmup + thinning × n_samples` steps after the rounding. -- `warmup` is the number of CHRR steps discarded before the first stored sample. `random_sampling` passes it to CHRR only; the ACHR runs ignore it. +- `warmup` is the number of CHRR steps discarded before the first stored sample. `random_sampling` passes it to CHRR only, and the ACHR runs ignore it. - The lag-1 autocorrelation `ρ` is the correlation between consecutive stored samples of one chain. - The effective sample size (ESS) is the number of independent samples that carry the same information as the correlated ones. Under the assumption that the chain is a first-order autoregressive process, `ESS = n × (1−ρ) / (1+ρ)` for `n` stored samples. - The R-hat of a reaction (Gelman-Rubin statistic) compares the flux variance between chains with the variance within chains. With `m` chains of `n` samples, `W` the mean within-chain variance and `B/n` the variance of the chain means, `R-hat = sqrt(((n−1)/n × W + B/n) / W)`. R-hat is 1.0 when the chains agree and grows when the distribution of a chain depends on where it started. The script reports the share of reactions above 1.1 and above 1.01, the stricter of the two thresholds. ## Set-up -Driver for the comparisons between chains: `scripts/analyze_sampling_convergence.py` in raven-toolbox. It starts `n_chains` calls of `random_sampling` with seeds 0 to `n_chains − 1`, one process per chain through `ProcessPoolExecutor`, because cobra models are not guaranteed to be safe for concurrent solves in threads. Each process reads the model from the SBML file. The script computes R-hat per reaction with the formula in [Terms](#terms). A reaction with the same constant flux in every chain has `W = 0` and no variance between the chain means, gets no R-hat, and is excluded as not scored. A reaction that is constant within each chain at different values in different chains has `W = 0` and an infinite R-hat, and is scored. +The driver for the comparisons between chains is `scripts/analyze_sampling_convergence.py` in raven-toolbox. It starts `n_chains` calls of `random_sampling` with seeds 0 to `n_chains − 1`, one process per chain through `ProcessPoolExecutor`, because cobra models are not guaranteed to be safe for concurrent solves in threads. Each process reads the model from the SBML file. The script computes R-hat per reaction with the formula in [Terms](#terms). A reaction with the same constant flux in every chain has `W = 0` and no variance between the chain means, gets no R-hat, and is excluded as not scored. A reaction that is constant within each chain at different values in different chains has `W = 0` and an infinite R-hat, and is scored. -Settings: `n_samples=300`, `thinning=100`, `warmup=1000` and 4 chains, unless a table states otherwise. `thinning` and `warmup` are at their defaults; the default `n_samples` is 1000. The solver was Gurobi, with 12 cores per job on AMD EPYC 9354 nodes. All figures are from runs on 2026-11-01 with Python 3.14.2, cobra 0.32.1, optlang 1.9.1, gurobipy 13.0.3 (Gurobi library 13.0.3) and raven-toolbox `94f3940`. Each configuration ran once, except e_coli_core ACHR, which was repeated; the single-chain runs also ran with seeds 1 to 3 (see [Autocorrelation within one chain](#autocorrelation-within-one-chain)). cobrapy's `OptGPSampler` was not tested, because `random_sampling` does not call it; its methods are `'achr'`, `'chrr'` and `'random_objective'`. +The settings were `n_samples=300`, `thinning=100`, `warmup=1000` and 4 chains, unless a table states otherwise. The values of `thinning` and `warmup` are the defaults, and the default `n_samples` is 1000. The solver was Gurobi, with 12 cores per job on AMD EPYC 9354 nodes. All figures are from runs on 2026-11-01 with Python 3.14.2, cobra 0.32.1, optlang 1.9.1, gurobipy 13.0.3 (Gurobi library 13.0.3) and raven-toolbox `94f3940`. Each configuration ran once, except e_coli_core ACHR, which was repeated. The single-chain runs also ran with seeds 1 to 3 (see [Autocorrelation within one chain](#autocorrelation-within-one-chain)). The `OptGPSampler` of cobrapy was not tested, because `random_sampling` does not call it, and its methods are `'achr'`, `'chrr'` and `'random_objective'`. -The models are two files of yeast-GEM and one of e_coli_core. The single-chain runs read the bundled `docs/data/yeast-GEM.xml` (yeastGEM_v9.1.0, 4,102 reactions). The comparisons between chains read yeast-GEM v9.1.1 (4,105 reactions), which differs from the develop branch at commit `ea756c2` only in the model identifier and two dates. e_coli_core is `textbook.xml.gz` of cobrapy 0.32.1 (model identifier `e_coli_core`). +The models are two files of yeast-GEM and one of e_coli_core. The single-chain runs read the bundled `docs/data/yeast-GEM.xml` (yeastGEM_v9.1.0, 4,102 reactions). The comparisons between chains read yeast-GEM v9.1.1 (4,105 reactions), which differs from the develop branch at commit `ea756c2` only in the model identifier and two dates. The model e_coli_core is `textbook.xml.gz` of cobrapy 0.32.1 (model identifier `e_coli_core`). -The wall time of a comparison between chains is the time from starting the processes to the return of the last chain: reading the model, the warmup and the sampling, without the R-hat computation. The wall time of a single-chain run is the time of reading the SBML file plus `random_sampling`. The `ρ` of a chain is the mean over the reactions with a flux range above `1e-6` in that chain of the lag-1 Pearson correlation of the reaction (`pandas.Series.autocorr(1)`); the median over the reactions is higher and is given in the text. The ESS uses the mean. +The wall time of a comparison between chains is the time from starting the processes to the return of the last chain: reading the model, the warmup and the sampling, without the R-hat computation. The wall time of a single-chain run is the time of reading the SBML file plus `random_sampling`. The `ρ` of a chain is the mean over the reactions with a flux range above `1e-6` in that chain of the lag-1 Pearson correlation of the reaction (`pandas.Series.autocorr(1)`). The median over the reactions is higher and is given in the text. The ESS uses the mean. ## Results ### Autocorrelation within one chain -One ACHR chain on yeast-GEM (4,102 reactions), `n_samples=300`, seed 0, at three values of `thinning`. The ESS column is computed from the formula in [Terms](#terms). +The runs used one ACHR chain on yeast-GEM (4,102 reactions), `n_samples=300`, seed 0, at three values of `thinning`. The ESS column is computed from the formula in [Terms](#terms). | `thinning` | Lag-1 autocorrelation | ESS of 300 stored samples | Wall time (s) | |---|---|---|---| @@ -47,15 +49,15 @@ The same settings with seeds 0 to 3 give the following `ρ`. | 100 | 0.9361 | 0.6523 | 0.9238 | 0.9286 | 0.652 to 0.936 | | 500 | 0.8251 | 0.6444 | 0.8013 | 0.7694 | 0.644 to 0.825 | -For every seed, `ρ` falls as `thinning` rises. Between seeds, `ρ` differs by up to 0.28, and the ESS of 300 stored samples ranges from 3.2 to 11.8 at `thinning=20`, from 9.9 to 63.1 at 100 and from 28.8 to 64.9 at 500. Seed 1 gives the lowest `ρ` at all three values; without it, the ranges are 0.972 to 0.979, 0.924 to 0.936 and 0.769 to 0.825. The four chains of the yeast-GEM comparison with 4,105 reactions (seeds 0 to 3, `thinning=100`) give a mean `ρ` of 0.951, 0.847, 0.947 and 0.911, so a low value for seed 1 appears on both model files. The spread between seeds is 0.10 on the 4,105-reaction chains (0.847 to 0.951) and 0.28 on the 4,102-reaction chains (0.652 to 0.936); seed 1 is the lowest in both. +For every seed, `ρ` falls as `thinning` rises. Between seeds, `ρ` differs by up to 0.28, and the ESS of 300 stored samples ranges from 3.2 to 11.8 at `thinning=20`, from 9.9 to 63.1 at 100 and from 28.8 to 64.9 at 500. Seed 1 gives the lowest `ρ` at all three values, and without it the ranges are 0.972 to 0.979, 0.924 to 0.936 and 0.769 to 0.825. The four chains of the yeast-GEM comparison with 4,105 reactions (seeds 0 to 3, `thinning=100`) give a mean `ρ` of 0.951, 0.847, 0.947 and 0.911, so a low value for seed 1 appears on both model files. The spread between seeds is 0.10 on the 4,105-reaction chains (0.847 to 0.951) and 0.28 on the 4,102-reaction chains (0.652 to 0.936), and seed 1 is the lowest in both. -`n_samples` was not varied. With `ρ` fixed at its seed-0 value for the default `thinning=100`, the ESS formula is proportional to `n_samples`: ~10 from 300 stored samples, ~33 from the default 1000, and 100 needs ~3,000 stored samples. Building the ACHR warmup points alone took 185.9 s, 196.6 s and 195.4 s in the three runs, so the sampling steps took about 1 s, 5 s and 26 s for 6,000, 30,000 and 150,000 steps. The warmup is therefore ~95 % of the wall time at `thinning=100`, and the wall time rose from 189.5 s to 223.3 s (1.2-fold) while the number of steps rose 25-fold. With a fixed cost per run plus a cost per stored sample, 3,025 stored samples take ~242 s for one chain; scaling the 203.6 s in proportion to 3,025 samples gives 2,053 s (~34 min), an upper bound. The warmup time was measured on a second, fresh sampler after the run, so the split is approximate. +The value `n_samples` was not varied. With `ρ` fixed at its seed-0 value for the default `thinning=100`, the ESS formula is proportional to `n_samples`: about 10 from 300 stored samples, about 33 from the default 1000, and 100 needs about 3,000 stored samples. Building the ACHR warmup points alone took 185.9 s, 196.6 s and 195.4 s in the three runs, so the sampling steps took about 1 s, 5 s and 26 s for 6,000, 30,000 and 150,000 steps. The warmup is therefore about 95 % of the wall time at `thinning=100`, and the wall time rose from 189.5 s to 223.3 s (1.2-fold) while the number of steps rose 25-fold. With a fixed cost per run plus a cost per stored sample, 3,025 stored samples take about 242 s for one chain. Scaling the 203.6 s in proportion to 3,025 samples gives 2,053 s (about 34 min), an upper bound. The warmup time was measured on a second, fresh sampler after the run, so the split is approximate. -For the choice of settings: within the measured range, a larger `thinning` gave more ESS per second of wall time (4.2 in 189.5 s at 20, 28.8 in 223.3 s at 500). That a larger `n_samples` at a fixed `thinning` raises ESS in proportion and leaves `ρ` unchanged follows from the first-order autoregressive formula; it was not measured. +Within the measured range, a larger `thinning` gave more ESS per second of wall time (4.2 in 189.5 s at 20, 28.8 in 223.3 s at 500). That a larger `n_samples` at a fixed `thinning` raises the ESS in proportion and leaves `ρ` unchanged follows from the first-order autoregressive formula, and it was not measured. ### Agreement between chains on e_coli_core -ACHR, 4 chains × 300 samples, 10.9 s wall time (5.4 s in a repeat). The repeat gave the same R-hat for every reaction, so on e_coli_core a chain is deterministic for a given seed and software. Reproducibility on yeast-GEM was not tested; the `random_objective` run on yeast-GEM is not reproducible (see [Other sampling settings](#other-sampling-settings)). The difference between 10.9 s and 5.4 s is unexplained. +The ACHR run used 4 chains × 300 samples and took 10.9 s wall time (5.4 s in a repeat). The repeat gave the same R-hat for every reaction, so on e_coli_core a chain is deterministic for a given seed and software. Reproducibility on yeast-GEM was not tested, and the `random_objective` run on yeast-GEM is not reproducible (see [Other sampling settings](#other-sampling-settings)). The difference between 10.9 s and 5.4 s is unexplained. | Measure | Value | |---|---:| @@ -66,7 +68,7 @@ ACHR, 4 chains × 300 samples, 10.9 s wall time (5.4 s in a repeat). The repeat | reactions with R-hat > 1.01 | 12 (13.8 %) | | reactions with R-hat > 1.1 | 0 (0.0 %) | -Reactions with the highest R-hat: +The reactions with the highest R-hat were: | Reaction | R-hat | |---|---:| @@ -81,11 +83,11 @@ Reactions with the highest R-hat: No reaction exceeded 1.1, and 12 of 87 scored reactions exceeded 1.01. The reaction with the highest R-hat is the succinate exchange `EX_succ_e`. -For the choice of settings: on a 95-reaction model, the default `thinning` with 300 samples per chain gave agreement at the 1.1 threshold for 87 of 87 scored reactions and at 1.01 for 75 of 87. Agreement at 1.01 for the remaining 12 needs more steps or a different method; CHRR did not remove it (see [CHRR on e_coli_core](#chrr-on-e_coli_core)). +On a 95-reaction model, the default `thinning` with 300 samples per chain gave agreement at the 1.1 threshold for 87 of 87 scored reactions and at 1.01 for 75 of 87. Agreement at 1.01 for the remaining 12 needs more steps or another method, and CHRR did not remove it (see [CHRR on e_coli_core](#chrr-on-e_coli_core)). ### Agreement between chains on yeast-GEM -ACHR, 4 chains × 300 samples, 572.3 s wall time (~9.5 min). +The ACHR run used 4 chains × 300 samples and took 572.3 s wall time (about 9.5 min). | Measure | Value | |---|---:| @@ -96,17 +98,17 @@ ACHR, 4 chains × 300 samples, 572.3 s wall time (~9.5 min). | reactions with R-hat > 1.01 | 3167 (95.2 %) | | reactions with R-hat > 1.1 | 1685 (50.7 %) | -Reactions with the highest R-hat: `r_4562` (18.90), `r_4358` (18.90), `r_0992` (5.18), `r_4249` (5.18), `r_4701` (5.18), `r_4621` (5.18), `r_1965` (5.09), `r_0917` (3.90), `r_0891` (3.90), `r_0918` (3.90). +The reactions with the highest R-hat were `r_4562` (18.90), `r_4358` (18.90), `r_0992` (5.18), `r_4249` (5.18), `r_4701` (5.18), `r_4621` (5.18), `r_1965` (5.09), `r_0917` (3.90), `r_0891` (3.90) and `r_0918` (3.90). -The median scored reaction was 1.1020, and one in two reactions (50.7 %) exceeded 1.1; 4.8 % passed 1.01. Each chain took 300 × 100 = 30,000 ACHR steps after its warmup points. The low ESS of one chain and the disagreement between chains are two separate measurements, and both show that these settings do not give a converged sample of yeast-GEM. +The median scored reaction was 1.1020, and one in two reactions (50.7 %) exceeded 1.1, whereas 4.8 % passed 1.01. Each chain took 300 × 100 = 30,000 ACHR steps after its warmup points. The low ESS of one chain and the disagreement between chains are two separate measurements, and both show that these settings do not give a converged sample of yeast-GEM. -The four parallel chains took 572.3 s, 2.8 times the 203.6 s of the single chain at the same settings. The two runs read different yeast-GEM files (4,105 and 4,102 reactions). Contention between the four Gurobi processes on the 12 cores is a possible cause of the difference; it was not measured. +The four parallel chains took 572.3 s, 2.8 times the 203.6 s of the single chain at the same settings. The two runs read different yeast-GEM files (4,105 and 4,102 reactions). Contention between the four Gurobi processes on the 12 cores is a possible cause of the difference, and it was not measured. -For the choice of settings: on yeast-GEM with `n_samples=300` and `thinning=100`, 50.7 % of the 3325 scored reactions had R-hat above 1.1. The default `n_samples=1000` (100,000 steps per chain) was not tested. +On yeast-GEM with `n_samples=300` and `thinning=100`, 50.7 % of the 3325 scored reactions had R-hat above 1.1. The default `n_samples=1000` (100,000 steps per chain) was not tested. ### Reallocating the ACHR step budget -The 30,000 steps per chain of the default run (`thinning=100, n_samples=300`), spread as `thinning=300, n_samples=100`. ACHR, 4 chains, yeast-GEM. +The 30,000 steps per chain of the default run (`thinning=100, n_samples=300`) were spread as `thinning=300, n_samples=100`. The runs used ACHR, 4 chains and yeast-GEM. | Measure | t=100, n=300 (default thinning) | t=300, n=100 (reallocated) | |---|---:|---:| @@ -118,13 +120,11 @@ The 30,000 steps per chain of the default run (`thinning=100, n_samples=300`), s | R-hat > 1.1 | 50.7 % | 50.5 % | | worst reaction | `r_4562`, `r_4358` (18.90) | `r_4562`, `r_4358` (21.91) | -Both runs scored 3325 reactions. The reallocation moved the median R-hat from 1.1020 to 1.1010 and the shares above the two thresholds by at most 1.7 percentage points, the reactions with the highest R-hat were the same, and the run took 612.4 s against 572.3 s (1.07 times). That the disagreement comes from how ACHR moves through this polytope, and not from the split of the steps, is an inference from this result; it was not tested at genome scale. - -For the choice of settings: at a fixed number of steps, moving steps from `n_samples` to `thinning` did not improve agreement between chains. A larger total number of steps was not tested with R-hat. +Both runs scored 3325 reactions. The reallocation moved the median R-hat from 1.1020 to 1.1010 and the shares above the two thresholds by at most 1.7 percentage points. The reactions with the highest R-hat were the same, and the run took 612.4 s against 572.3 s (1.07 times). That the disagreement comes from how ACHR moves through this polytope, and not from the split of the steps, is an inference from this result, and it was not tested at genome scale. At a fixed number of steps, moving steps from `n_samples` to `thinning` did not improve the agreement between chains, and a larger total number of steps was not tested with R-hat. ### CHRR on e_coli_core -4 chains × 300 samples on e_coli_core, with `method='achr'` and `method='chrr'`. +The runs used 4 chains × 300 samples on e_coli_core, with `method='achr'` and `method='chrr'`. | Measure | ACHR | CHRR | |---|---:|---:| @@ -136,37 +136,41 @@ For the choice of settings: at a fixed number of steps, moving steps from `n_sam | R-hat > 1.01 | 13.8 % (12) | 14.7 % (14) | | R-hat > 1.1 | 0.0 % | 0.0 % | -With either method, no reaction exceeded 1.1. CHRR did not agree better than ACHR on e_coli_core: its highest R-hat was `EX_succ_e` at 1.0377 against 1.0276 for ACHR, and 14.7 % of its reactions exceeded 1.01 against 13.8 %. CHRR took 66 times the wall time of ACHR (133 times that of the 5.4 s repeat), one run each. The percentages have different denominators, 87 and 95 scored reactions. The 8 reactions that ACHR left constant (all exactly 0.0) were scored with CHRR. CHRR computes every flux of a sample from the nullspace basis (`v0 + N x`); these 8 reactions vary within a chain by at most 1.6e-13, with means of the order of 1e-14, which is the size of floating-point rounding error in that product. +With either method, no reaction exceeded 1.1. CHRR did not agree better than ACHR on e_coli_core: its highest R-hat was `EX_succ_e` at 1.0377 against 1.0276 for ACHR, and 14.7 % of its reactions exceeded 1.01 against 13.8 %. CHRR took 66 times the wall time of ACHR (133 times that of the 5.4 s repeat), with one run each. The percentages have different denominators, 87 and 95 scored reactions. The 8 reactions that ACHR left constant (all exactly 0.0) were scored with CHRR. CHRR computes every flux of a sample from the nullspace basis (`v0 + N x`), and these 8 reactions vary within a chain by at most 1.6e-13, with means of the order of 1e-14, which is the size of the floating-point rounding error in that product. -For the choice of settings: on a 95-reaction model, ACHR and CHRR with the default `thinning` and 300 samples per chain both gave agreement at 1.1 for every reaction, and CHRR cost 66 times as much time (133 times against the repeat) without lowering the share above 1.01. +On a 95-reaction model, ACHR and CHRR with the default `thinning` and 300 samples per chain both gave agreement at 1.1 for every reaction, and CHRR cost 66 times as much time (133 times against the repeat) without lowering the share above 1.01. ### CHRR at genome scale -CHRR was not run on yeast-GEM with 4 chains × 300 samples. A probe with 2 chains × 20 samples (`thinning=100`) took 8147.9 s (~136 min), 14.2 times the 572.3 s of the full ACHR run, with a peak memory of about 31 GB of the 32 GB allocated. Each chain runs an FVA with its own worker pool; that the pools cause the memory use was not tested. yeast-GEM has 43 times the reactions of e_coli_core. +CHRR was not run on yeast-GEM with 4 chains × 300 samples. A probe with 2 chains × 20 samples (`thinning=100`) took 8147.9 s (about 136 min), 14.2 times the 572.3 s of the full ACHR run, with a peak memory of about 31 GB of the 32 GB allocated. Each chain runs an FVA with its own worker pool, and that the pools cause the memory use was not tested. The model yeast-GEM has 43 times the reactions of e_coli_core. -Each CHRR chain ran an FVA, a nullspace computation and the MVE rounding before its first step, and then 1,000 warmup steps and 2,000 sampling steps. The probe does not separate these parts. That the cost before the first step dominates, and that most of it is the MVE rounding, is an inference from the small number of steps; the parts were not timed. `random_sampling` computes this transform anew in every call, so each chain pays it. +Each CHRR chain ran an FVA, a nullspace computation and the MVE rounding before its first step, and then 1,000 warmup steps and 2,000 sampling steps. The probe does not separate these parts. That the cost before the first step dominates, and that most of it is the MVE rounding, is an inference from the small number of steps, and the parts were not timed. The function `random_sampling` computes this transform anew in every call, so each chain pays it. -The R-hat of the probe (median 5.7274, p90 165.18, max 1.899e9 for `r_4782`; all 4105 reactions scored, 97.3 % above 1.01 and 93.6 % above 1.1) does not show that CHRR converges worse than ACHR at genome scale. With 20 samples per chain, the within-chain variance `W` is estimated from too few samples to be stable: a reaction that shows almost no variance in 20 draws by chance, combined with any difference between the chain means (`B/n`), gives a large R-hat whatever its distribution. The probe cannot distinguish a CHRR that does not converge on yeast-GEM from an R-hat that needs more samples; a run with enough samples for a stable R-hat has the cost problem of the full run. +The R-hat of the probe (median 5.7274, p90 165.18, max 1.899e9 for `r_4782`; all 4105 reactions scored, 97.3 % above 1.01 and 93.6 % above 1.1) does not show that CHRR converges worse than ACHR at genome scale. With 20 samples per chain, the within-chain variance `W` is estimated from too few samples to be stable. A reaction that shows almost no variance in 20 draws by chance, combined with any difference between the chain means (`B/n`), gives a large R-hat whatever its distribution. The probe cannot distinguish a CHRR that does not converge on yeast-GEM from an R-hat that needs more samples, and a run with enough samples for a stable R-hat has the cost problem of the full run. -For the choice of settings: 20 samples per chain are too few for R-hat on yeast-GEM. Whether CHRR converges on yeast-GEM with 300 samples per chain is open. +Twenty samples per chain are too few for R-hat on yeast-GEM, and whether CHRR converges on yeast-GEM with 300 samples per chain is open. ## Choice of `thinning` and `n_samples` -The defaults are `method='achr'`, `thinning=100` and `n_samples=1000` ([Tuned parameter defaults](../../tuned-parameters.md)). CHRR is not the default: on e_coli_core, CHRR differed from ACHR by +0.0012 in median R-hat (1.0027 and 1.0039), by −0.0034 in p90 (1.0171 and 1.0137) and by +0.0101 in maximum (1.0276 and 1.0377), with one run each and no measured spread between seed sets, and CHRR took 66 to 133 times as long. On a model the size of e_coli_core, the default `thinning` with 300 samples per chain gave chains that agree at R-hat 1.1 for every reaction. On yeast-GEM, no setting tested gave converged samples: a `thinning` up to 500 left `ρ` at 0.825 for seed 0 (0.644 to 0.825 over four seeds), and moving the same 30,000 steps from `n_samples` to `thinning` moved the median R-hat from 1.1020 to 1.1010 and the shares above 1.01 and 1.1 by at most 1.7 percentage points. On yeast-GEM with `n_samples=300` and `thinning=100`, 50.7 % of the 3325 scored reactions had R-hat above 1.1. +The defaults are `method='achr'`, `thinning=100` and `n_samples=1000` ([Tuned parameter defaults](../../tuned-parameters.md)). CHRR is not the default. On e_coli_core, CHRR differed from ACHR by +0.0012 in median R-hat (1.0027 and 1.0039), by −0.0034 in p90 (1.0171 and 1.0137) and by +0.0101 in maximum (1.0276 and 1.0377), with one run each and no measured spread between sets of seeds, and CHRR took 66 to 133 times as long. -The `random_sampling` docstring recommends, for genome-scale models, a `thinning` of at least 1000 or a larger `n_samples`, `method='chrr'`, or an ESS or R-hat check. A `thinning` of 1000 or more was not measured. The driver script runs the R-hat check: it samples several chains with different seeds at given settings and reports the share of reactions above the R-hat thresholds, so a sampling result can be checked at the settings that produced it. +On a model the size of e_coli_core, the default `thinning` with 300 samples per chain gave chains that agree at R-hat 1.1 for every reaction. On yeast-GEM, no setting tested gave converged samples. A `thinning` up to 500 left `ρ` at 0.825 for seed 0 (0.644 to 0.825 over four seeds), and moving the same 30,000 steps from `n_samples` to `thinning` moved the median R-hat from 1.1020 to 1.1010 and the shares above 1.01 and 1.1 by at most 1.7 percentage points. With `n_samples=300` and `thinning=100`, 50.7 % of the 3325 scored reactions had R-hat above 1.1. + +The docstring of `random_sampling` recommends, for genome-scale models, a `thinning` of at least 1000 or a larger `n_samples`, `method='chrr'`, or an ESS or R-hat check. A `thinning` of 1000 or more was not measured. The driver script runs the R-hat check: it samples several chains with different seeds at given settings and reports the share of reactions above the R-hat thresholds, so a sampling result can be checked at the settings that produced it. One change outside the parameters was identified and not pursued: CHRR recomputes its rounding transform in every call. The chains run in separate processes at the same time, so to save that cost the transform would have to be computed once per model and passed to every chain. The saving depends on the share of the rounding in the 136 min of the probe, which was not measured. ## Other sampling settings -Two settings of `method='random_objective'` differ from RAVEN MATLAB's `randomSampling` on purpose. +Two settings of `method='random_objective'` differ from `randomSampling` of RAVEN in MATLAB on purpose. + +The setting `replace_max_bound` (default `False`; `replaceBoundsWithInf` in MATLAB defaults to true) replaces the largest upper bound, the big-M value that stands for an unconstrained reaction, with infinity before sampling, and replaces the most negative lower bound, if it is below 0, with minus infinity. Models in the RAVEN convention use 1000 as the big-M for almost every reaction (4,083 of the 4,102 reactions of yeast-GEM). With `True`, the call fails after 9.6 s with `OptimizationError: Solver status is 'unbounded'`. A probe of 50 random objectives, drawn as the sampler draws them on the model with the bounds replaced, gave 49 optimal results and 1 unbounded result (one probe of 50 objectives). One unbounded objective stops all draws. The abort mechanism is read from the cobra code, where `Model.optimize` raises on the status `unbounded` (`raise_error=False` does not change this) and the retry loop of the sampler never sees the status. The sample at which the run aborted was not recorded. -`replace_max_bound` (default `False`; MATLAB's `replaceBoundsWithInf` defaults to true) replaces the largest upper bound, the big-M value that stands for an unconstrained reaction, with infinity before sampling, and replaces the most negative lower bound, if it is below 0, with minus infinity. RAVEN-convention models use 1000 as the big-M for almost every reaction (4,083 of the 4,102 reactions of yeast-GEM). With `True`, the call fails after 9.6 s with `OptimizationError: Solver status is 'unbounded'`. A probe of 50 random objectives, drawn as the sampler draws them on the model with the bounds replaced, gave 49 optimal results and 1 unbounded result (one probe of 50 objectives). One unbounded objective stops all draws: the abort mechanism is read from the cobra code, where `Model.optimize` raises on the status `unbounded` (`raise_error=False` does not change this) and the retry loop of the sampler never sees the status. The sample at which the run aborted was not recorded. With `False`, 200 samples completed (68.2 s with 12 processes, including the loopless FVA). Counting a flux as at the bound when `|v| ≥ 1000 × (1 − 1e-9)`, the 200 × 4102 sample matrix (820,400 entries) had 54 entries at the bound (0.0066 %) in one run, with 15 samples containing at least one such reaction and 23 reactions (0.56 % of 4102) at the bound in at least one sample. A second run had 269 entries (0.033 %), 16 samples (8.0 %) and 32 reactions (0.78 %). A share of 0.56 % of reactions at the bound occurred in one run and 0.78 % in another; the share of entries is 0.0066 to 0.033 %. +With `False`, 200 samples completed (68.2 s with 12 processes, including the loopless FVA). Counting a flux as at the bound when `|v| ≥ 1000 × (1 − 1e-9)`, the 200 × 4102 sample matrix (820,400 entries) had 54 entries at the bound (0.0066 %) in one run, with 15 samples containing at least one such reaction and 23 reactions (0.56 % of 4102) at the bound in at least one sample. A second run had 269 entries (0.033 %), 16 samples (8.0 %) and 32 reactions (0.78 %). The share of reactions at the bound was 0.56 % in one run and 0.78 % in the other, and the share of entries is 0.0066 to 0.033 %. -`random_objective` with a fixed seed is not reproducible on yeast-GEM: the loopless FVA returned 2307 and 2308 eligible reactions in two runs with the same seed. That a different eligible set shifts every random draw is an inference from these two counts. +The method `random_objective` with a fixed seed is not reproducible on yeast-GEM: the loopless FVA returned 2307 and 2308 eligible reactions in two runs with the same seed. That a different eligible set shifts every random draw is an inference from these two counts. -`loopless_good_reactions` (default `True`) selects which reactions are eligible as random objectives. MATLAB excludes every reaction whose FVA maximum is infinite or above 0.999 times the largest upper bound (999 on a model capped at 1000), and, when the smallest lower bound is negative, every reaction whose FVA minimum is minus infinity or below 0.999 times that bound; it treats proximity to the big-M as a sign of a thermodynamic loop. It also excludes reactions that cannot carry a non-zero flux and, in an ecModel, the `usage_prot_` reactions. That rule also excludes reactions that reach the bound for metabolic reasons. A loopless FVA excludes only the reactions that take part in infeasible cycles, at a higher cost. +The setting `loopless_good_reactions` (default `True`) selects which reactions are eligible as random objectives. MATLAB excludes every reaction whose FVA maximum is infinite or above 0.999 times the largest upper bound (999 on a model capped at 1000), and, when the smallest lower bound is negative, every reaction whose FVA minimum is minus infinity or below 0.999 times that bound. It treats proximity to the big-M as a sign of a thermodynamic loop. It also excludes reactions that cannot carry a non-zero flux and, in an ecModel, the `usage_prot_` reactions. That rule also excludes reactions that reach the bound for metabolic reasons. A loopless FVA excludes only the reactions that take part in infeasible cycles, at a higher cost. ## Limits @@ -193,4 +197,4 @@ python scripts/parameter_tuning/sampling/random_objective_bounds.py \ --n-proc 12 --n-probe 50 --out results/ ``` -The CHRR runs use the same command with `--method chrr`. The script caches the chains per model, method, `n_chains`, `n_samples`, `thinning` and `warmup` under `--out`; a run with a cached configuration computes R-hat from the stored chains without sampling. Another `--thinning` or `--n-samples` samples again and is cached separately. On yeast-GEM, four ACHR chains in parallel on 12 cores took 2.8 times the wall time of one chain at the same settings; the two runs read different yeast-GEM files and the cause of the difference was not measured (see [Agreement between chains on yeast-GEM](#agreement-between-chains-on-yeast-gem)). A sweep at genome scale can therefore take more time than one chain per configuration suggests. +The CHRR runs use the same command with `--method chrr`. The script caches the chains per model, method, `n_chains`, `n_samples`, `thinning` and `warmup` under `--out`, and a run with a cached configuration computes R-hat from the stored chains without sampling. Another `--thinning` or `--n-samples` samples again and is cached separately. On yeast-GEM, four ACHR chains in parallel on 12 cores took 2.8 times the wall time of one chain at the same settings. The two runs read different yeast-GEM files, and the cause of the difference was not measured (see [Agreement between chains on yeast-GEM](#agreement-between-chains-on-yeast-gem)). A sweep at genome scale can therefore take more time than one chain per configuration suggests. From f598bfb0f31c61cf363ee82f49b9b3c08ba69041 Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Sun, 1 Nov 2026 11:50:38 +0100 Subject: [PATCH 18/20] Reword five sentences flagged by the style check --- docs/parameter-tuning/benchmarks.md | 2 +- docs/parameter-tuning/studies/init-param-calibration.md | 2 +- docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md | 2 +- .../studies/sampling-convergence-calibration.md | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/parameter-tuning/benchmarks.md b/docs/parameter-tuning/benchmarks.md index dfc6e38..223125d 100644 --- a/docs/parameter-tuning/benchmarks.md +++ b/docs/parameter-tuning/benchmarks.md @@ -108,7 +108,7 @@ Both implementations have the same value or behaviour for each of these paramete | `connect_blocked_reactions.allow_net_production` | `False` | Keeps the steady-state balance `S @ v == 0`, the stricter test. `True` relaxes it to `S @ v >= 0` | | `fill_gaps_fast_lp.epsilon` | `1e-4` | Value of the fastGapFill paper, [Thiele et al. 2014](https://doi.org/10.1093/bioinformatics/btu321) | | `fill_gaps_milp.big_m` | `1000.0` | Matches the ±1000 bounds of the RAVEN convention | -| `check_tasks.close_boundaries` | `False` | Matches `checkTasks` of RAVEN, against which published task lists were curated. `True` closes the exchange reactions of the model, so that the task defines the whole boundary | +| `check_tasks.close_boundaries` | `False` | Matches `checkTasks` of RAVEN, against which published task lists were curated. `True` closes the exchange reactions of the model, which makes the task define the whole boundary | ## Choice of defaults diff --git a/docs/parameter-tuning/studies/init-param-calibration.md b/docs/parameter-tuning/studies/init-param-calibration.md index a660109..8d674e9 100644 --- a/docs/parameter-tuning/studies/init-param-calibration.md +++ b/docs/parameter-tuning/studies/init-param-calibration.md @@ -223,7 +223,7 @@ Expression values become gene scores through score = clip(factor × ln(level / reference), min_score, max_score) ``` -with `factor=5.0`, `max_score=10.0` and `min_score=-5.0`, the same in both implementations. The logarithm is natural (base e) in `init/score.py` and in `scoreModel` of RAVEN. The source of each implementation cites no paper for the three constants. A 2012 paper (Wang et al. 2012, [mCADRE](https://doi.org/10.1186/1752-0509-6-153)) uses categorical scoring and is not the origin. The constants are values of RAVEN itself without a literature source. +with `factor=5.0`, `max_score=10.0` and `min_score=-5.0`, the same in both implementations. The logarithm is natural (base e) in `init/score.py` and in `scoreModel` of RAVEN. The source of each implementation cites no paper for the three constants. A 2012 paper (Wang et al. 2012, [mCADRE](https://doi.org/10.1186/1752-0509-6-153)) uses categorical scoring and does not give these constants. The constants are values of RAVEN itself without a literature source. ## Comparison with the solver benchmark diff --git a/docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md b/docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md index 95fbc06..9722654 100644 --- a/docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md +++ b/docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md @@ -237,4 +237,4 @@ The script `analyze_hmm_cutoffs.py` is called once per organism with `--artefact The proteomes (`kegg_rest/.faa`, `uniprot/.faa`, `counts.json`) are in the archived input bundle `kegg_proteomes_2026-11-01.tar.gz`, and the four `hmmsearch --tblout` tables are in `kegg_hmmsearch_tblout_2026-11-01.tar.gz`. -The scores of the tables come from the run of 2026-11-01 with raven-toolbox commit `94f3940`, HMMER 3.4 and the published kegg118 tables and libraries, searched with 16 threads on an AMD EPYC 9354 node. Scoring each saved hit table again with `--tblout` gives the saved scores exactly. Two points were not checked: whether the KEGG REST sequences of 2026-10-31 equal the sequences of KEGG release 118 (the sequence counts were compared, not the sequences), and whether `hmmsearch -Z` E-values equal `hmmscan` E-values on these inputs, because the binary bundle has no `hmmscan`. The UniProt proteomes in the bundle were downloaded and counted but not searched. +The scores of the tables come from the run of 2026-11-01 with raven-toolbox commit `94f3940`, HMMER 3.4 and the published kegg118 tables and libraries, searched with 16 threads on an AMD EPYC 9354 node. Scoring each saved hit table again with `--tblout` reproduces the saved scores. Two points were not checked: whether the KEGG REST sequences of 2026-10-31 equal the sequences of KEGG release 118 (the sequence counts were compared, not the sequences), and whether `hmmsearch -Z` E-values equal `hmmscan` E-values on these inputs, because the binary bundle has no `hmmscan`. The UniProt proteomes in the bundle were downloaded and counted but not searched. diff --git a/docs/parameter-tuning/studies/sampling-convergence-calibration.md b/docs/parameter-tuning/studies/sampling-convergence-calibration.md index 93e4e67..5ab0277 100644 --- a/docs/parameter-tuning/studies/sampling-convergence-calibration.md +++ b/docs/parameter-tuning/studies/sampling-convergence-calibration.md @@ -152,7 +152,7 @@ Twenty samples per chain are too few for R-hat on yeast-GEM, and whether CHRR co ## Choice of `thinning` and `n_samples` -The defaults are `method='achr'`, `thinning=100` and `n_samples=1000` ([Tuned parameter defaults](../../tuned-parameters.md)). CHRR is not the default. On e_coli_core, CHRR differed from ACHR by +0.0012 in median R-hat (1.0027 and 1.0039), by −0.0034 in p90 (1.0171 and 1.0137) and by +0.0101 in maximum (1.0276 and 1.0377), with one run each and no measured spread between sets of seeds, and CHRR took 66 to 133 times as long. +The defaults are `method='achr'`, `thinning=100` and `n_samples=1000` ([Tuned parameter defaults](../../tuned-parameters.md)). The default is ACHR. On e_coli_core, CHRR differed from ACHR by +0.0012 in median R-hat (1.0027 and 1.0039), by −0.0034 in p90 (1.0171 and 1.0137) and by +0.0101 in maximum (1.0276 and 1.0377), with one run each and no measured spread between sets of seeds, and CHRR took 66 to 133 times as long. On a model the size of e_coli_core, the default `thinning` with 300 samples per chain gave chains that agree at R-hat 1.1 for every reaction. On yeast-GEM, no setting tested gave converged samples. A `thinning` up to 500 left `ρ` at 0.825 for seed 0 (0.644 to 0.825 over four seeds), and moving the same 30,000 steps from `n_samples` to `thinning` moved the median R-hat from 1.1020 to 1.1010 and the shares above 1.01 and 1.1 by at most 1.7 percentage points. With `n_samples=300` and `thinning=100`, 50.7 % of the 3325 scored reactions had R-hat above 1.1. From 7bcc0a66ae99940b172b94c4d3de857a8bf946a9 Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Sun, 1 Nov 2026 11:56:20 +0100 Subject: [PATCH 19/20] Apply the review of the rewritten migration, comparison and parameter-tuning pages --- docs/parameter-tuning/benchmarks.md | 18 ++++--- .../studies/ftinit-determinism.md | 34 ++++++-------- .../studies/homology-cutoff-calibration.md | 21 ++++----- .../studies/humangem-validation.md | 19 ++++---- .../studies/init-param-calibration.md | 47 ++++++++++--------- .../studies/kegg-hmm-cutoff-calibration.md | 6 +-- .../sampling-convergence-calibration.md | 18 ++++--- docs/raven3-migration.md | 8 ++-- .../raven3-migration/backward-incompatible.md | 19 ++++---- .../formats-and-reconstruction.md | 12 ++--- docs/raven3-migration/new-and-environment.md | 14 +++--- .../raven3-migration/structure-and-renames.md | 20 ++++---- docs/raven3-vs-raven-toolbox.md | 6 +-- .../same-function-different-answer.md | 20 ++++---- docs/raven3-vs-raven-toolbox/what-each-has.md | 6 +-- 15 files changed, 136 insertions(+), 132 deletions(-) diff --git a/docs/parameter-tuning/benchmarks.md b/docs/parameter-tuning/benchmarks.md index 223125d..c81241d 100644 --- a/docs/parameter-tuning/benchmarks.md +++ b/docs/parameter-tuning/benchmarks.md @@ -1,6 +1,8 @@ # Benchmarks of `fseof`, `remove_genes` and other defaults -Four defaults of raven-toolbox were measured without a study of their own: `flux_eps` of `fseof`, the sweep of `n_steps`, `max_fraction` and `correlation_threshold` of `fseof`, the handling of reactions without an enzyme in `remove_genes`, and `threshold` of `connect_blocked_reactions`. A table at the end lists further defaults that rest on the literature or on a convention. All runs used raven-toolbox commit `94f39407` and were made on 2026-11-01. +Six defaults of raven-toolbox were measured without a study of their own: `flux_eps` of `fseof`; `n_steps`, `max_fraction` and `correlation_threshold` of `fseof`; `blocked_reactions` of `remove_genes`; and `threshold` of `connect_blocked_reactions`. A table lists further defaults that rest on the literature or on a convention. All runs used raven-toolbox commit `94f39407` and were made on 2026-11-01. + +The measurements give no reason to change `flux_eps` from `1e-6`. The sweep does not select the other three `fseof` defaults, which follow [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10) (not checked against the paper), and the default of `blocked_reactions` gives the model that `removeGenes` of RAVEN gives by default. Parameters with a measurement campaign of their own have a study listed under [Parameter tuning](index.md). [Tuned parameter defaults](../tuned-parameters.md) gives the current value of every parameter of both implementations with a one-line reason. @@ -8,7 +10,7 @@ Parameters with a measurement campaign of their own have a study listed under [P ### `fseof` -The `fseof` benchmarks ran on iJO1366 (2,583 reactions) with the biomass reaction of the model as the objective and `EX_succ_e` (succinate exchange) as the target. The enforced flux ran from 1.5387 to 15.387 in 10 levels, and the unconstrained growth rate was 0.98237 h⁻¹. The other arguments were `n_steps` 10, `max_fraction` 0.9, `correlation_threshold` 0.9 and the default `equality=False`, which sets each enforced flux as the lower bound of the target reaction. +The `fseof` benchmarks ran on iJO1366 (2,583 reactions) with the biomass reaction of the model as the objective and `EX_succ_e` (succinate exchange) as the target. In the `flux_eps` runs the enforced flux ran from 1.5387 to 15.387 in 10 levels, and the unconstrained growth rate was 0.98237 h⁻¹. The other arguments were `n_steps` 10, `max_fraction` 0.9, `correlation_threshold` 0.9 and the default `equality=False`, which sets each enforced flux as the lower bound of the target reaction. At each enforced level `fseof` solves two problems, as `FSEOF` of RAVEN does. It first maximises the objective and then minimises the sum of fluxes with the objective held at 0.999999 of that optimum. A reaction is a target when four conditions hold: @@ -36,7 +38,7 @@ The test of `threshold` used a synthetic network. The draft has an import of A o ### Software -The Python runs used Python 3.14.2, cobra 0.32.1, optlang 1.9.1 and Gurobi 13.0.3 (gurobipy wheel). The MATLAB run used R2024b with RAVEN and Gurobi 13.0.1, selected through `setRavenSolver('gurobi')`. The `flux_eps` runs were repeated three times per value, each on a freshly read model, and gave the same target set and labels every time. +The Python runs used Python 3.14.2, cobra 0.32.1, optlang 1.9.1 and Gurobi 13.0.3 (gurobipy wheel). The MATLAB run used R2024b with RAVEN and Gurobi 13.0.1, selected through `setRavenSolver('gurobi')`. The `flux_eps` runs were repeated three times per value, each on a freshly read model. ## Results @@ -50,7 +52,7 @@ The Python runs used Python 3.14.2, cobra 0.32.1, optlang 1.9.1 and Gurobi 13.0. Of the 408 reactions, 406 are knockdown targets and 2 are knockout targets. Three runs per value gave the same set of targets. -The parameter `flux_eps` is the floor below which `fseof` treats the theoretical maximum of the target flux as zero, and the scan then raises an error. The classification of targets does not read `flux_eps`, and `FSEOF` of RAVEN has no such parameter. From the code, two values of `flux_eps` give different results only when the theoretical maximum of the target lies between them: `fseof` raises at the larger value and runs at the smaller. +`flux_eps` is the floor below which `fseof` treats the theoretical maximum of the target flux as zero, and the scan then raises an error. The classification of targets does not read `flux_eps`, and `FSEOF` of RAVEN has no such parameter. From the code, two values of `flux_eps` give different results if and only if the theoretical maximum of the target lies between them: `fseof` raises at the larger value and runs at the smaller. ### `fseof` targets at other values of three parameters @@ -80,11 +82,13 @@ Every swept value changes the target set. None of the added targets is a noise t | `removeGenes(model, genes, 'removeBlockedRxns', true)` | 94 | 0.000 | deleted | | unmodified model | 95 | 0.874 (MATLAB biomass flux 0.8739) | present | -The parameter `blocked_reactions` of `remove_genes` and `removeBlockedRxns` of `removeGenes` decide what happens to a reaction whose gene rule is emptied by a deletion. Both defaults keep such a reaction with bounds of zero, and the two defaults gave the same model. After the deletion of `b1779` that model predicts no growth, because GAPD stays in the network without capacity to carry flux. The setting `"remove"` and `removeBlockedRxns` true also predicted no growth. They delete GAPD, which changes the number of reactions and the set of reaction identifiers. The setting `"keep"` leaves GAPD with an empty gene rule and unchanged bounds, so the model predicted the growth of the unmodified model, 0.874 h⁻¹, although the only gene of GAPD was deleted. GAPD is essential for aerobic growth on glucose in e_coli_core, so the growth predicted with `"keep"` is a false positive. The docstring of `remove_genes` states no use for `"keep"`, and the documentation names the editing of gene annotations without changing the network as a use, without a source in the code. +`blocked_reactions` of `remove_genes` and `removeBlockedRxns` of `removeGenes` decide what happens to a reaction whose gene rule is emptied by a deletion. Both default settings keep such a reaction with bounds of zero, and the two gave the same model. After the deletion of `b1779` that model predicts no growth, because GAPD stays in the network without capacity to carry flux. With `"remove"` and with `removeBlockedRxns` true the model also predicted no growth. Both delete GAPD, which changes the number of reactions and the set of reaction identifiers. + +The setting `"keep"` leaves GAPD with an empty gene rule and unchanged bounds, so the model predicted the growth of the unmodified model, 0.874 h⁻¹, although the only gene of GAPD was deleted. The default setting predicted no growth after the same deletion, so the growth predicted with `"keep"` is probably a false positive in this model (the bounds are those shipped with e_coli_core). The docstring of `remove_genes` states no use for `"keep"`. The documentation names the editing of gene annotations without changing the network as a use, and the code gives no source for that use. ### `connect_blocked_reactions` at four values of `threshold` -The network is the synthetic network of the set-up. +The draft network has an import of A, the reaction A to B and no exit for B, and the template supplies `EX_B`. | `threshold` | Gap filled | Reactions added | Maximum flux of the reaction A to B afterwards | |---|---|---|---| @@ -114,7 +118,7 @@ Both implementations have the same value or behaviour for each of these paramete The measurements give no reason to change `flux_eps` from `1e-6`. The sweep does not select the other three `fseof` defaults: the target set changes with each swept value by 2 to 12 targets, and no tested value leaves it unchanged. The defaults follow [Choi et al. 2010](https://doi.org/10.1128/AEM.00115-10) (not checked against the paper). The 19 noise tracks are in the target set at every setting. -The default `blocked_reactions="constrain"` of `remove_genes` gives the model that `removeGenes` of RAVEN gives by default. The settings `"remove"` and `"keep"` are available on request. +The default `blocked_reactions="constrain"` of `remove_genes` gives the model that `removeGenes` of RAVEN gives by default. The settings `"remove"` and `"keep"` are selected with the `blocked_reactions` argument. ## Limits diff --git a/docs/parameter-tuning/studies/ftinit-determinism.md b/docs/parameter-tuning/studies/ftinit-determinism.md index c55592e..ecb951e 100644 --- a/docs/parameter-tuning/studies/ftinit-determinism.md +++ b/docs/parameter-tuning/studies/ftinit-determinism.md @@ -4,7 +4,7 @@ The study builds Human-GEM models for one cell line with the `ftinit` of raven-t Two properties are measured. Determinism means that the same input gives the same model when it is run again or with another seed, machine or solver build. Stability means that a similar input, such as a slightly edited template or a comparable sample, gives a similar model. Both parameters are opt-in (`resolve_ties=False` and `prove_abs_gap=None` by default) and target determinism. -The seed-to-seed difference fell from 110 reactions and 17 essential genes (default) to 41 reactions and 1 gene with `resolve_ties`, and to 1 reaction and 0 genes with `prove_abs_gap=1.0` added. The build times were about 3.6 and 26 times the default build time. Neither parameter made the model stable under a change of the input. A small template edit moved a re-extraction by more reactions than a change of seed (156 against 110 at baseline, 59 against 41 with `resolve_ties`, 16 against 1 with `prove_abs_gap`) and by a similar number of essential genes (17 against 17, 1 against 1, 1 against 0). raven-toolbox has no parameter that anchors a re-extraction to a prior build. +The seed-to-seed difference fell from 110 reactions and 17 essential genes (default) to 41 reactions and 1 gene with `resolve_ties`, and to 1 reaction and 0 genes with `prove_abs_gap=1.0` added. The build times were about 3.6 times (`resolve_ties`) and 26 times (`prove_abs_gap=1.0` added) the default build time. Neither parameter made the model stable under a change of the input. A small template edit moved a re-extraction by more reactions than a change of seed (156 against 110 at baseline, 59 against 41 with `resolve_ties`, 16 against 1 with `prove_abs_gap`) and by a similar number of essential genes (17 against 17, 1 against 1, 1 against 0). raven-toolbox has no parameter that anchors a re-extraction to a prior build. ## Main results @@ -29,16 +29,16 @@ The template edit removed 30 reactions that the reference builds did not use. Th | `resolve_ties=True` | 59 | 1 | 114 → 115 | 638 s | | `resolve_ties=True, prove_abs_gap=1.0` | 16 | 1 | 125 → 126 | 4108 s | -The unedited builds hardly used the removed reactions. If the re-extraction changed only what the edit forces, the baseline model would lose the one removed reaction it kept and the `resolve_ties` models would lose none (inference, not tested). The baseline moved 156 reactions and 17 essential-gene calls, and the two settings with `resolve_ties=True` moved 59 and 16 reactions and 1 gene each. +The unedited baseline build kept 1 of the 30 removed reactions and the unedited builds with `resolve_ties=True` kept none. If the re-extraction changed only what the edit forces, the baseline model would lose that reaction and the `resolve_ties` models would lose none (inference, not tested). The baseline moved 156 reactions and 17 essential-gene calls, and the two settings with `resolve_ties=True` moved 59 and 16 reactions and 1 gene each. ## Set-up -- **Model and data.** Human-GEM `v2.0.0-38`, Hart2015 DLD1 cell line, 57 essential tasks, series `1+1`. The `model/Human-GEM.yml` of that commit has 12877 reactions and 2848 genes. The preparation is the one of `estimateEssentialGenes.py` at that commit (removal of the drug, amino-acid-triplet and duplicate complex I reactions), followed by `prep_init_model` of raven-toolbox. This gives a reference model of 10141 reactions and a merged model of 6861 reactions, in which chains of linked reactions are combined into one, with 206 task-essential reactions. The preparation differs from the exported `raven_refModel.xml` of RAVEN, so absolute counts are not comparable to the [Human-GEM cell-type model validation against RAVEN](humangem-validation.md), which uses that export. -- **Steps.** Series `1+1` runs two extraction steps, counted from 1 here; the debug log counts from 0. Step 1 decides the reactions with a gene association. Step 2, the last step, adds the reactions without one and holds the reactions of step 1 on. +- **Model and data.** Human-GEM `v2.0.0-38`, Hart2015 DLD1 cell line, 57 essential tasks, series `1+1`. The `model/Human-GEM.yml` of that commit has 12877 reactions and 2848 genes. The preparation is the one of `estimateEssentialGenes.py` at that commit (removal of the drug, amino-acid-triplet and duplicate complex I reactions), followed by `prep_init_model` of raven-toolbox. This gives a reference model of 10141 reactions and a merged model of 6861 reactions that contains 206 task-essential reactions. In the merged model, chains of linked reactions are combined into one reaction. The preparation differs from the exported `raven_refModel.xml` of RAVEN, so absolute counts are not comparable to the [Human-GEM cell-type model validation against RAVEN](humangem-validation.md), which uses that export. +- **Steps.** Series `1+1` runs two extraction steps, counted from 1 in this study; the debug log counts from 0. Step 1 decides the reactions with a gene association. Step 2, the last step, adds the reactions without one and holds the reactions of step 1 on. - **Settings.** `big_m=100` (the default is 1000), `mip_gap_abs=10` and `time_limit=1800` s, the values of `estimateEssentialGenes.py` at that commit. A named series such as `1+1` ignores `mip_gap` and `mip_gap_abs`, which apply to a step without a run schedule. -- **Escalation.** Each solve stops at a mixed-integer programming (MIP) gap, the distance between the objective of the best solution found and the bound of the solver on the optimum, either relative (divided by the objective) or absolute. Without `prove_abs_gap`, each step follows the run schedule of RAVEN, called the escalation here. The first run uses a relative gap of 0.0004. Each later run uses the larger of its relative gap and its absolute gap (10 or 20) divided by the objective, and the step stops as soon as the gap already reached meets the next target. The setting `time_limit` caps the limit of each run: step 1 has runs of 120 s and 1800 s, and the final step runs of 5 s, 120 s and 1800 s. +- **Escalation.** Each solve stops at a mixed-integer programming (MIP) gap, the distance between the objective of the best solution found and the bound of the solver on the optimum, either relative (divided by the objective) or absolute. Without `prove_abs_gap`, each step follows the run schedule of RAVEN, called the escalation in this study. The first run uses a relative gap of 0.0004. Each later run uses the larger of its relative gap and its absolute gap (10 or 20) divided by the objective, and the step stops as soon as the gap already reached meets the next target. The setting `time_limit` caps the limit of each run: step 1 has runs of 120 s and 1800 s, and the final step runs of 5 s, 120 s and 1800 s. - **`prove_abs_gap`.** Each step is one solve proven to that absolute gap. It has the 1800 s limit for the primary solve and for the two flux phases of `resolve_ties` (phases 2c and 2d), and half of it (`_TIE_BREAK_TIME_FRACTION = 0.5`) for its two on/off phases (phases 2a and 2b). The phases are defined in the section on the parameters. The task gap-fill uses a fixed 300 s per task in every setting. -- **Solver.** Gurobi through gurobipy, `Threads=1` unless stated, raven-toolbox at commit `94f39407`. The Gurobi library is 13.0.3 from the gurobipy wheel (most likely, because the logs print no version). The builds ran as Slurm jobs on AMD EPYC 9354 nodes shared with other jobs, so wall-clock figures are approximate. Most jobs had 2 CPUs, and the `Threads=4` jobs of the first-step probe had 4. Several builds ran in sequence in one job: the seeds of a setting, or the three builds of the stability test. +- **Solver.** Gurobi through gurobipy, `Threads=1` unless stated, raven-toolbox at commit `94f39407`. The Gurobi library is 13.0.3 from the gurobipy wheel (most likely; the logs print no version). The builds ran as Slurm jobs on AMD EPYC 9354 nodes shared with other jobs, so wall-clock figures are approximate. Most jobs had 2 CPUs, and the `Threads=4` jobs of the first-step probe had 4. Several builds ran in sequence in one job: the seeds of a setting, or the three builds of the stability test. - **Determinism probe.** The extraction `seed` was varied (1234, 7, and further seeds where stated) with inputs, code, solver build and machine held fixed. A difference between builds then has two sources: the choice of the solver among equal optima, measured where two builds reached the same objective value, and different objective values reached within the gap and time limits (see the location of the degeneracy and the sweep of `prove_abs_gap`). Differences between Gurobi versions were not measured. - **Metrics.** The reaction difference is the symmetric difference of the kept-reaction sets, that is the reactions kept by one build and not by the other. The essential-gene difference is the symmetric difference of the sets of genes whose single deletion is predicted essential by `cobra.flux_analysis.find_essential_genes` at its default threshold. The gene difference is the result that a user sees, and the reaction difference is what the parameters act on. - **Debug log.** All builds ran with `FTINIT_DEBUG=1`, which logs the status of each solve and of each tie-break phase. An incumbent is the best solution that the solver holds when it stops, and at a time limit it is not proven optimal. @@ -47,7 +47,7 @@ The unedited builds hardly used the removed reactions. If the re-extraction chan The extraction mixed-integer linear program (MILP) has many optima of equal score. In the first step on Human-GEM and DLD1, 1547 merged reactions have a negative score, and these carry the 0/1 keep-or-drop variables. Of them, 98.3 % (1520) are in blocks of equal score. The largest block has 840 reactions at −5.0. The second has 420 reactions at −2.0, the default score of a reaction without gene association. Across the five Hart2015 cell lines it has 419 to 421 members, so the expression data change its size by at most 2 members. Many reaction subsets reach the same optimum, and the solver returns one of them. -The settings `Threads=1` and a fixed `Seed` make that choice close to reproducible on one machine, because the two baseline builds at one seed differed by 2 reactions. Another seed can select another member of the tied set, or end the solve at another objective value within the gap and time limits. Both were seen in the first-step probe, and in either case the predicted essential genes change for reasons unrelated to the model or the data. Another Gurobi version or platform is expected to act the same way, which was not measured. +The settings `Threads=1` and a fixed `Seed` make that choice close to reproducible on one machine, as shown by the two baseline builds at one seed differed by 2 reactions. Another seed can select another member of the tied set, or end the solve at another objective value within the gap and time limits. Both were seen in the first-step probe, and in either case the predicted essential genes change for reasons unrelated to the model or the data. Another Gurobi version or platform is expected to act the same way, which was not measured. ### Location of the degeneracy @@ -66,12 +66,7 @@ Grouping the runs by objective value separates the tie-break from the primary so ## Parameters `resolve_ties` and `prove_abs_gap` -The parameter `resolve_ties` adds a lexicographic phase 2 after the primary solve of each extraction step, with the score objective held at its optimum. The phases are, in order: - -- `phase2a-parsimony` minimises the count of kept removable (negative-score) reactions. -- `phase2b-idrank` minimises, among those, the summed id rank of the kept reactions. A merged reaction is ranked by the smallest original id in its group (`rank_keys`), so the rank does not depend on which member names the merged reaction. -- `phase2c-minflux` minimises, for that set of kept reactions, the total absolute flux (gap 1e-6), so the fluxes that the next step reads for its forced directions and magnitudes do not depend on the vertex of the solver. -- `phase2d-direction` minimises, among the minimal-flux solutions, the id-rank-weighted backward flux of the reactions whose flux can take either sign (gap 1e-6), so a flux that carries the same total either way gets a fixed direction. +The parameter `resolve_ties` adds a lexicographic phase 2 after the primary solve of each extraction step, with the score objective held at its optimum. Four phases follow in order, each held at the optimum of the one before. First, `phase2a-parsimony` minimises the count of kept removable (negative-score) reactions. Then `phase2b-idrank` minimises, among those, the summed id rank of the kept reactions; a merged reaction is ranked by the smallest original id in its group (`rank_keys`), so the rank does not depend on which member names the merged reaction. Then `phase2c-minflux` minimises, for that set of kept reactions, the total absolute flux (gap 1e-6), so the fluxes that the next step reads for its forced directions and magnitudes do not depend on the vertex of the solver. Finally `phase2d-direction` minimises, among the minimal-flux solutions, the id-rank-weighted backward flux of the reactions whose flux can take either sign (gap 1e-6), so a flux that carries the same total either way gets a fixed direction. The last step skips `phase2c-minflux` and `phase2d-direction`, because no later step reads its fluxes. The task gap-fill uses the first two phases. Phases 2a and 2b get half of `time_limit` each, and phases 2c and 2d get the full `time_limit`. @@ -83,7 +78,8 @@ A phase that does not prove its optimum has these outcomes: - A flux phase (`phase2c-minflux`, `phase2d-direction`) that finds no solution warns with its name and keeps the fluxes of the phase before it. - When `phase2a-parsimony` or `phase2b-idrank` finds no incumbent, the on/off tie-break of that solve is dropped without a warning. The primary solution is kept and the ties of that step are not resolved, so a build without a warning can hold an unresolved step. - When the primary solve of a step stops at `time_limit`, all tie-break phases of that solve are skipped and the primary solution is kept. -- The escalation raises no warning either. In the four sweep runs with the escalation and `resolve_ties=True`, the step 1 objective ended 4.0 to 12.0 below the proven optimum. At least one of the 12 first-step runs without `resolve_ties` reached it (see the location of the degeneracy). + +The escalation raises no warning either. In the four sweep runs with the escalation and `resolve_ties=True`, the step 1 objective ended 4.0 to 12.0 below the proven optimum. At least one of the 12 first-step runs without `resolve_ties` reached it (see the location of the degeneracy). For comparison, `ftINIT` of RAVEN with `resolveTies` skips the tie-break after a primary solve that stopped at the time limit, gives the count and id-rank phases half of `params.TimeLimit` each, and skips the minimal-flux and direction phases in the last step. Its warning "tie resolution did not converge" names every unproven phase (`phase2a-parsimony`, `phase2b-idrank`, `phase2c-minflux`, `phase2d-direction`). In `src/INIT/ftINITInternalAlg.m` of RAVEN, the skip after a primary solve at the time limit is at line 483, the half limit at line 518 and the warning at line 662, and `src/INIT/ftINIT.m` line 227 sets `pinFluxes` to false in the last step. `ftINIT` of RAVEN was not run in this study. @@ -131,11 +127,11 @@ Step 2 is the last step, so it skips the last two phases. The log of seed 1234 i [ftinit] tie-break phase2b-idrank: status=optimal obj=346311.0 ``` -The on/off tie-break of step 1 was proven for all four seeds, so the step 1 reaction set is fixed. The phase `phase2c-minflux` reached the 1800 s limit for 3 of the 4 seeds and `phase2d-direction` for all 4, and each was adopted as an unproven incumbent. In the three seeds in which `phase2c-minflux` stopped at the limit, the two flux phases at the 1800 s limit take 3600 s, which accounts for most of the step 1 time of about 3650 s and for a build time of about 100 minutes. The flux that step 2 reads is therefore the incumbent of a time-limited solve. The step 2 objective differed by seed (−611.872 for seed 7, −613.872 for the other three) and the final sets by one reaction. The fluxes themselves were not compared between seeds, and a seed-dependent flux is inferred from the step 2 objectives and not measured. +The on/off tie-break of step 1 was proven for all four seeds, so the step 1 reaction set is fixed. The phase `phase2c-minflux` reached the 1800 s limit for 3 of the 4 seeds and `phase2d-direction` for all 4, and each was adopted as an unproven incumbent. In the three seeds in which `phase2c-minflux` stopped at the limit, the two flux phases at the 1800 s limit take 3600 s, which accounts for most of the step 1 time of about 3650 s and for a build time of about 100 minutes. The flux that step 2 reads is therefore the incumbent of a time-limited solve. The step 2 objective differed by seed (−611.872 for seed 7, −613.872 for the other three) and the final sets by one reaction. The fluxes themselves were not compared between seeds, and a seed-dependent flux is inferred from the step 2 objectives and was not measured. -The escalation builds with `resolve_ties=True` also adopted phases at a time limit, here the limit of the escalation run (120 s in step 1, so 60 s for phases 2a and 2b). The full-pipeline build of seed 1234 adopted 3 phases (`phase2b-idrank`, `phase2c-minflux`, `phase2d-direction` of step 1) and the build of seed 7 adopted 2 (`phase2c-minflux`, `phase2d-direction` of step 1). In step 2 of both builds, `phase2a-parsimony` ended at the time limit with an incumbent and `phase2b-idrank` ended at the time limit with no incumbent (`A-rt-1234.log`, `A-rt-7.log`). The phase `phase2b-idrank` then returns no solution, the tie-break of step 2 returns `False`, the primary solution of step 2 is kept, and no warning is raised. The reaction sets of these two builds rest on a resolved step 1 and an unresolved step 2. +The escalation builds with `resolve_ties=True` also adopted phases at a time limit, that is the limit of the escalation run (120 s in step 1, so 60 s for phases 2a and 2b). The full-pipeline build of seed 1234 adopted 3 phases (`phase2b-idrank`, `phase2c-minflux`, `phase2d-direction` of step 1) and the build of seed 7 adopted 2 (`phase2c-minflux`, `phase2d-direction` of step 1). In step 2 of both builds, `phase2a-parsimony` ended at the time limit with an incumbent and `phase2b-idrank` ended at the time limit with no incumbent (`A-rt-1234.log`, `A-rt-7.log`). The phase `phase2b-idrank` then returns no solution, the tie-break of step 2 returns `False`, the primary solution of step 2 is kept, and no warning is raised. The reaction sets of these two builds rest on a resolved step 1 and an unresolved step 2. -The next table counts full builds (gap-fill included) with at least one unproven tie-break phase or a time-limited primary solve. The full builds of the main results are, per setting: baseline 5 (seeds 1234 and 7, the two builds of the repeat, the edited template), `resolve_ties=True` 5 (seeds 1234 and 7, the two builds of the repeat, the edited template), and `prove_abs_gap` 5 (seeds 1234 and 7 at each of 1.0 and 0.05, the edited template at 1.0). The column Builds gives how many of them were counted. +The next table counts full builds (gap-fill included) with at least one unproven tie-break phase or a time-limited primary solve. The column Builds gives how many full builds of the setting were counted. | Setting | Builds | With an adopted phase at `time_limit` | With a primary solve at `time_limit` | With a tie-break dropped without a warning | |---|---:|---:|---:|---:| @@ -160,7 +156,7 @@ For the curation of a template (fix one or a few reactions, re-extract, compare) ## Choice of settings - **`resolve_ties=True`.** The repeat pair at seed 1234 had identical reaction sets, and the seed-to-seed difference fell from 17 genes to 1 on this data, at about 3.6 times the build time (4 to 14 minutes). Its warnings name the adopted phases that stopped at the time limit. A tie-break that finds no incumbent in phase 2a or 2b is dropped without a warning. The builds at seeds 1234 and 7 dropped the one of step 2, and the table of build counts gives 5 of 5 builds with a dropped step 2 tie-break. With the escalation, the tie-break runs again in each run of the schedule (`_solve_step` in `ftinit.py` and the run schedule in `steps.py`). The flux phases of step 1 get 120 s when step 1 stops after its first run and 1800 s when a second run follows (its 5000 s capped by `time_limit`). -- **`prove_abs_gap=1.0` with `resolve_ties=True`.** The seed-to-seed difference fell from 41 to 1 reaction (1 to 0 genes) at 26 times the baseline time, which is about 100 minutes at `time_limit=1800`. The objectives of both steps are proven to 1.0, whereas in the four sweep runs with the escalation and `resolve_ties=True` step 1 ended 4.0 to 12.0 below the optimum. In the four `prove_abs_gap=1.0` sweep builds (step 1 and step 2, no gap-fill; 4987 to 6183 s wall clock), step 1 took 3305 to 3650 s, 59.0 to 66.3 % of each build, mostly in the two flux phases, and step 2 took another 1650 to 2500 s. A value from 2.0 to 0.05 gave the same models, and 5.0 stopped 2.0 below the optimum for one seed. No build with `prove_abs_gap` and without `resolve_ties` was run. +- **`prove_abs_gap=1.0` with `resolve_ties=True`.** The seed-to-seed difference fell from 41 to 1 reaction (1 to 0 genes) at 26 times the baseline time, which is about 100 minutes at `time_limit=1800`. The objectives of both steps are proven to 1.0; in the four sweep runs with the escalation and `resolve_ties=True` step 1 ended 4.0 to 12.0 below the optimum. In the four `prove_abs_gap=1.0` sweep builds (step 1 and step 2, no gap-fill; 4987 to 6183 s wall clock), step 1 took 3305 to 3650 s, 59.0 to 66.3 % of each build, mostly in the two flux phases, and step 2 took another 1650 to 2500 s. A value from 2.0 to 0.05 gave the same models, and 5.0 stopped 2.0 below the optimum for one seed. No build with `prove_abs_gap` and without `resolve_ties` was run. - **Gurobi versions.** They were not compared. Two baseline builds with everything held fixed differed by 2 reactions. ## Limits @@ -184,7 +180,7 @@ python analyse.py $WORK/out python det_extra.py $WORK/out $WORK/logs ``` -The tables are computed from the outputs and logs by `analyse.py` and `det_extra.py`, and the degeneracy counts are read from `ties.log`. The scripts `ties.py` and `select_edit.py` need the base preparation `prep-base.pkl`, which is not archived and takes about 2000 s to build. The script `det/build.py` produced the build numbers. Series `1+1` ignores `mip_gap_abs`, which the call below passes as `mip_gap_abs=10.0`. +The tables are computed from the outputs and logs by `analyse.py` and `det_extra.py`, and the degeneracy counts are read from `ties.log`. The scripts `ties.py` and `select_edit.py` need the base preparation `prep-base.pkl`, which is not archived and takes about 2000 s to build. The script `det/build.py` produced the build numbers. The call below passes `mip_gap_abs=10.0`, which series `1+1` ignores. ```python from raven_toolbox.init import ftinit diff --git a/docs/parameter-tuning/studies/homology-cutoff-calibration.md b/docs/parameter-tuning/studies/homology-cutoff-calibration.md index db99e80..28cdbd3 100644 --- a/docs/parameter-tuning/studies/homology-cutoff-calibration.md +++ b/docs/parameter-tuning/studies/homology-cutoff-calibration.md @@ -2,7 +2,7 @@ Homology-based reconstruction transfers a template reaction to a new organism when a gene of the new organism matches a template gene in a BLAST or DIAMOND search and the match passes three filters: `min_identity`, `min_align_len` and `max_evalue` (`minIde`, `minLen` and `maxE` in `getModelFromHomology` of RAVEN). The study aligned the proteomes of four organisms against *S. cerevisiae* once, applied each filter setting to the saved hits, and scored the surviving matches against two references that state which genes are counterparts. It records one score per organism and setting, the agreement between BLAST and DIAMOND, and their run times. -The defaults that follow from the scores are `min_identity` 40, `min_align_len` 100 and `max_evalue` 1e-30. The identity has the largest effect on the score, the alignment length changes it less, and the E-value changes it only below 1e-30. +The scores show the largest effect for `min_identity`, a smaller effect for `min_align_len` and an effect of `max_evalue` from 1e-50 downward, with a flat score from 1e-4 to 1e-30. The value `max_evalue` 1e-30 is kept from RAVEN; the Chosen defaults table gives the basis of every default. ## Set-up @@ -19,7 +19,7 @@ The defaults that follow from the scores are `min_identity` 40, `min_align_len` The proteomes come from the UniProt reference proteomes. Each protein is relabelled with its KEGG gene name through the KEGG cross-reference of UniProt for the exact organism code, and proteins without one are dropped. Both references can then be applied to the same matches. -Each target was aligned against the template in both directions, once with `run_blast` (aligner E-value `1e-4`) and once with `run_diamond` (aligner E-value `1e-3`). Every filter setting was applied to the same saved hit table, so each score comes from one deterministic calculation. A hit is kept when `evalue <= max_evalue`, `align_len >= min_align_len` and `identity >= min_identity`, whereas `get_model_from_homology` requires `evalue < max_evalue`. The kept hits are collected into a set of (template gene, target gene) pairs, so a pair found in both directions, or by more than one hit, counts as one match. A hit whose two genes are not both judged by the reference is counted as unjudgeable and left out of the score. The scoring does not reduce hits to best hits and builds no model. +Each target was aligned against the template in both directions, once with `run_blast` (aligner E-value `1e-4`) and once with `run_diamond` (aligner E-value `1e-3`). Every filter setting was applied to the same saved hit table, so each score comes from one deterministic calculation. A hit is kept when `evalue <= max_evalue`, `align_len >= min_align_len` and `identity >= min_identity`. The function `get_model_from_homology` requires `evalue < max_evalue`. The kept hits are collected into a set of (template gene, target gene) pairs, so a pair found in both directions, or by more than one hit, counts as one match. A hit whose two genes are not both judged by the reference is counted as unjudgeable and left out of the score. The scoring does not reduce hits to best hits and builds no model. The sweeps vary one setting at a time from the RAVEN 2.11.3 values `max_evalue=1e-30`, `min_align_len=200` and `min_identity=40`. The identity sweep therefore ran at an alignment length of 200, the length sweep at 40 % identity, and the E-value sweep at 40 % identity and length 200. @@ -94,14 +94,11 @@ BLAST took 140, 152, 200 and 44 s for `kla`, `yli`, `ani` and `eco`, and DIAMOND Two more settings control the search that produces the hits. Neither was part of the sweeps. -The aligner setting `evalue` is the own cut-off of the aligner: `1e-4` for `run_blast` and `1e-3` for `run_diamond`. Each matches the MATLAB side, because `getBlast` passes `-evalue 10e-5`, which equals `1e-4`, and `getDiamond` passes none, so the own default of DIAMOND applies. On *H. polymorpha* against *S. cerevisiae*, loosening the BLAST cut-off from `1e-5` to `1e-4` added 5,200 raw hits (58,702 against 53,502), which is 8.9 % of the table. The tables differ row by row: 5,314 rows are only in the `1e-4` table and 114 only in the `1e-5` table. +The aligner setting `evalue` is the cut-off of the aligner itself: `1e-4` for `run_blast` and `1e-3` for `run_diamond`. Each matches the MATLAB side, because `getBlast` passes `-evalue 10e-5`, which equals `1e-4`, and `getDiamond` passes none, so DIAMOND uses its built-in default. On *H. polymorpha* against *S. cerevisiae*, loosening the BLAST cut-off from `1e-5` to `1e-4` added 5,200 raw hits (58,702 against 53,502), which is 8.9 % of the table. The tables differ row by row: 5,314 rows are only in the `1e-4` table and 114 only in the `1e-5` table. -- Of the 5,314 rows, 4,372 have an E-value above `1e-5` and fail `max_evalue=1e-30`. -- The other 942 rows have an E-value at or below `1e-5`, the smallest being 3.1e-60, and 4 of them are below `1e-30`. In the five examples of these 942 rows that were inspected, the rows are other alignments (HSPs) of gene pairs that the cut-off change reports differently. The other rows were not inspected. -- Whether the 4 rows below `1e-30` and the 114 rows found only at `1e-5` also pass the identity and length filters, and so could change a model, was not checked. -- The remaining rows fail `max_evalue=1e-30`, so for them the aligner cut-off changes the size of the hit table and not the model. +Of the 5,314 rows, 4,372 have an E-value above `1e-5` and fail `max_evalue=1e-30`. The other 942 rows have an E-value at or below `1e-5`, the smallest being 3.1e-60, and 4 of them are below `1e-30`. The remaining 938 of these rows fail `max_evalue=1e-30` as well, so for 5,310 rows the aligner cut-off changes the size of the hit table and leaves the model unchanged. In the five examples of the 942 rows that were inspected, the rows are other alignments (HSPs) of gene pairs that the two tables report differently. The other rows were not inspected. Whether the 4 rows below `1e-30` and the 114 rows found only at `1e-5` also pass the identity and length filters, and so could change a model, was not checked. -The setting `threads` defaults to every CPU that the process may run on. The first 500 sequences of the *H. polymorpha* proteome returned the same 2,746 hits at one thread and at four, with identical hit tables, in 20.4 s and 5.9 s (3.5 times faster). On a full proteome pair, single-threaded BLASTP took 205 s (*H. polymorpha* against *S. cerevisiae*) and 218 s (the reverse direction). The aligner cut-offs `1e-4` and `1e-5` took 111 s and 108.5 s. The measurements used the translated ORFs of SGD (6,735 sequences) as template and Linux BLAST+ 2.17.0 on an AMD EPYC 9354 node. +`threads` defaults to every CPU that the process may run on. The first 500 sequences of the *H. polymorpha* proteome returned the same 2,746 hits at one thread and at four, with identical hit tables, in 20.4 s and 5.9 s (3.5 times faster). On a full proteome pair, single-threaded BLASTP took 205 s (*H. polymorpha* against *S. cerevisiae*) and 218 s (the reverse direction). The aligner cut-offs `1e-4` and `1e-5` took 111 s and 108.5 s. The measurements used the translated ORFs of SGD (6,735 sequences) as template and Linux BLAST+ 2.17.0 on an AMD EPYC 9354 node. ## Chosen defaults @@ -111,7 +108,7 @@ The setting `threads` defaults to every CPU that the process may run on. The fir | `min_align_len` (`minLen`) | 200 | 100 | The length 100 scored 0.010 to 0.020 higher than 200 against KEGG. The lengths 50 and 100 scored the same, and 100 is the stricter of the two | | `max_evalue` (`maxE`) | 1e-30 | 1e-30 | No change in score from 1e-4 to 1e-30 in any organism, and 1e-50 lowered it by up to 0.006. The RAVEN value is kept | -The study supports one `min_identity` for BLAST and DIAMOND against the KEGG reference with F0.5, because both reached the same best identity for every organism. Against OMA the optimum for *E. coli* is 35 % with BLAST and 30 % with DIAMOND, and with F1 the optimum for `ani` is 25 % with BLAST and 30 % with DIAMOND. The DIAMOND scores for the alignment length and the E-value are not tabulated, and their base scores equal those of BLAST. +The study supports one `min_identity` for BLAST and DIAMOND against the KEGG reference with F0.5, because both reached the same best identity for every organism. Against OMA the optimum for *E. coli* is 35 % with BLAST and 30 % with DIAMOND, and with F1 the optimum for `ani` is 25 % with BLAST and 30 % with DIAMOND. The base rows of the alignment-length and E-value sweeps are the base row of the identity sweep (BLAST), and DIAMOND scores for those two sweeps are not tabulated. ## Limits @@ -120,12 +117,12 @@ The study supports one `min_identity` for BLAST and DIAMOND against the KEGG ref - The OMA reference for *Y. lipolytica* is incomplete, because 38 % of its OMA pairs could not be mapped to KEGG gene names. - One template: everything was measured outward from *S. cerevisiae*. - One setting was varied at a time from length 200 and identity 40 %. The identity optimum and the absence of an E-value effect up to 1e-30 were measured at length 200 and not at the default length of 100. -- The scores count gene matches and not reactions in a finished model. A reaction transfers when any one of its genes matches, so the effect on a finished model is probably smaller (inference, not measured). +- The scores count gene matches. A reaction transfers when any one of its genes matches, so the effect on a finished model is probably smaller (inference, not measured). - Each score comes from one alignment per organism. Scoring a saved hit table is deterministic, and DIAMOND returned 32,192 hits for `kla`. The alignment was run once, so the run-to-run spread of the hit table was not measured. ## Curated-GEM reference -Scoring against curated genome-scale metabolic models (GEMs) of non-model fungi, such as [hanpo-GEM](https://github.com/SysBioChalmers/hanpo-GEM) and [rhto-GEM](https://github.com/SysBioChalmers/rhto-GEM), is not part of the measurement of 2026-11-01, and the study reports no numbers for it. Both models were built with the homology-based reconstruction of RAVEN, so a score against them would measure agreement with their own build settings and not correctness (inference). +Scoring against curated genome-scale metabolic models (GEMs) of non-model fungi, such as [hanpo-GEM](https://github.com/SysBioChalmers/hanpo-GEM) and [rhto-GEM](https://github.com/SysBioChalmers/rhto-GEM), is not part of the measurement of 2026-11-01, and the study reports no numbers for it. Both models were built with the homology-based reconstruction of RAVEN, so a score against them would measure agreement with their own build settings, which is no test of correctness (inference). ## Reproducing @@ -145,4 +142,4 @@ python agreement.py --work work --gene-ko work/refs/kegg118_organism_gene_ko_5or The UniProt downloads, the OMA pairs, the KEGG tables and the SGD ORFs are in the archived input bundle `homology-inputs.tar.zst` (18.9 MB), and the BLAST and DIAMOND hit tables are in `homology-hits.tar.zst` (4.0 MB). -The scores come from the run of 2026-11-01 with raven-toolbox `94f3940`, BLAST+ 2.17.0, DIAMOND 2.1.17, the KEGG release-118 table and UniProt release 2026_03, with BLAST and DIAMOND at 4 threads on an AMD EPYC 9354 node. The *H. polymorpha* template is the SGD ORF file, and the thread test uses the first 500 sequences of `hanpo.faa`. The curated-GEM reference is not part of this run. +The scores come from the run of 2026-11-01 with raven-toolbox `94f3940`, BLAST+ 2.17.0, DIAMOND 2.1.17, the KEGG release-118 table and UniProt release 2026_03, with BLAST and DIAMOND at 4 threads on an AMD EPYC 9354 node. The *H. polymorpha* template is the SGD ORF file, and the thread test uses the first 500 sequences of `hanpo.faa`. diff --git a/docs/parameter-tuning/studies/humangem-validation.md b/docs/parameter-tuning/studies/humangem-validation.md index 65fe53c..d1c60cc 100644 --- a/docs/parameter-tuning/studies/humangem-validation.md +++ b/docs/parameter-tuning/studies/humangem-validation.md @@ -2,7 +2,7 @@ The study asks whether the ftINIT of raven-toolbox selects the same reactions as the `ftINIT` of RAVEN. Both tools extracted a cell-type model from Human-GEM with the RNA sequencing (RNA-seq) data of five cell lines of the Hart2015 dataset (DLD1, GBM, HCT116, HELA, RPE1), and the models were compared by their reaction sets, together with the time of each step. -With the task layer, the raven-toolbox models share a Jaccard index of 0.981 to 0.988 with the RAVEN models. Without the task layer the index is 0.964 to 0.982. +With the task layer, the raven-toolbox models share a Jaccard index of 0.981 to 0.988 with the RAVEN models. Without the task layer the index is 0.964 to 0.982. Each cell line, variant and `big_m` value ran once, and the RAVEN result is task-constrained, so the first range is the like-for-like comparison. ## Set-up @@ -93,13 +93,13 @@ The no-task `ftinit` runs took 83 to 122 s at `big_m=100` and 81 to 91 s at `big ### Agreement with the task layer -The task-constrained variant shares a Jaccard index of 0.981 to 0.988 with RAVEN across the five cell lines. Its reaction counts are within 0.2 % of the counts of RAVEN. The differences run in both directions in similar numbers: 46 to 70 reactions only in the raven-toolbox model and 43 to 71 only in the RAVEN model, out of about 7300. +The task-constrained variant agrees with RAVEN in every cell line. Its reaction counts are within 0.2 % of the counts of RAVEN. The differences run in both directions in similar numbers: 46 to 70 reactions only in the raven-toolbox model and 43 to 71 only in the RAVEN model, out of about 7300. -Neither tool was run twice on the same input, so the comparison does not separate alternate optima of the MILP from a difference between the two implementations. For scale, two raven-toolbox runs that differed only in the seed differed by 110 reactions in [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](ftinit-determinism.md). That measurement used another set-up (Human-GEM v2.0.0-38, DLD1, `big_m=100`, an escalation schedule), so it indicates the order of magnitude and does not bound the spread here. +Neither tool was run twice on the same input, so the comparison does not separate alternate optima of the MILP from a difference between the two implementations. For scale, two raven-toolbox runs that differed only in the seed differed by 110 reactions in [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](ftinit-determinism.md). That measurement used another set-up (Human-GEM v2.0.0-38, DLD1, `big_m=100`, an escalation schedule), so it indicates the order of magnitude and does not bound the spread of this set-up. ### Agreement without the task layer -The no-task variant shares a Jaccard index of 0.964 to 0.982 with RAVEN. It keeps fewer reactions than RAVEN in every cell line: 0.3 to 0.5 % fewer for four cell lines and 1.6 % fewer for RPE1. The reactions only in the RAVEN model (76 to 186) outnumber those only in the raven-toolbox model (53 to 76). +The no-task variant keeps fewer reactions than RAVEN in every cell line: 0.3 to 0.5 % fewer for four cell lines and 1.6 % fewer for RPE1. The reactions only in the RAVEN model (76 to 186) outnumber those only in the raven-toolbox model (53 to 76). The no-task variant omits the task layer that the RAVEN result includes. With the task layer, the reactions only in the RAVEN model fall to 43 to 71 per cell line. That the task layer accounts for most of the surplus is an inference from these counts, and the reactions in the surplus were not checked against the task-essential reactions. @@ -107,12 +107,16 @@ The no-task variant omits the task layer that the RAVEN result includes. With th Lowering the cap on the per-reaction big-M from 1000 to 100 changed the Jaccard index against RAVEN by at most 0.003 in each cell line, with one run per cell line and value. In every cell line the two raven-toolbox results shared a higher Jaccard index with each other (0.981 to 0.985) than either shared with RAVEN (0.964 to 0.982). For RPE1 the values were 0.984 between the two results and 0.964 and 0.967 with RAVEN. The cap therefore changed the reaction set (1 minus the Jaccard index between the two results, 0.015 to 0.019) by more than it changed the agreement with RAVEN (at most 0.003). At this resolution the two values agree equally well with RAVEN. -### Run time +### Comparison of run times -After the template preparation, a task-constrained cell-type model took 146 to 381 s in raven-toolbox and 40 to 91 s in RAVEN, and a no-task model took 81 to 91 s. The template preparation took 1556 s (26 min) and 1695 s (28 min) in raven-toolbox and 4320 s (72 min) in RAVEN. The preparations ran with different parallelism (see the run-time table), so the times compare whole runs and not the code alone. +After the template preparation, a task-constrained cell-type model took 146 to 381 s in raven-toolbox and 40 to 91 s in RAVEN, and a no-task model took 81 to 91 s. The template preparation took 1556 s (26 min) and 1695 s (28 min) in raven-toolbox and 4320 s (72 min) in RAVEN. The preparations ran with different parallelism (see the run-time table), so the times compare whole runs, including the parallelism. The [Reference run](../../guide/init.md#reference-run) of the guide Context-specific models with ftINIT reports a preparation of 47 min in MATLAB and 37 min in Python and an extraction of 797 s in MATLAB and 124 s in Python. That run used `prepINITModel` of RAVEN directly on Human-GEM `6812be2a` (12748 reactions) without the cleaning step, and series `1+0`. These four differences from this validation (preparation function, Human-GEM commit, cleaning step and series) were not separated, so the times are not comparable. +## Constraint construction + +Each steady-state balance of the extraction MILP is built from a flat list of terms and summed with `optlang.symbolics.add`, and the gap-fill uses the same construction. The `sum()` of Python simplifies the growing expression again at each term, and the balances of hub metabolites such as ATP, H⁺ and H₂O are long sums. + ## Numerical settings shared with RAVEN raven-toolbox builds the extraction MILP with two numerical settings of RAVEN: @@ -122,11 +126,10 @@ raven-toolbox builds the extraction MILP with two numerical settings of RAVEN: [ftINIT parameter calibration and input robustness](init-param-calibration.md) measured on HCT116 which combinations of `big_m` and rescaling are feasible for the staged pipeline. With rescaling, `big_m=25` was infeasible and `big_m=100` feasible. Without rescaling, `big_m=1000` was feasible (157 s against 83 s with rescaling, Jaccard index 0.969 between the two results) and `big_m=100` infeasible. -Each steady-state balance of the extraction MILP is built from a flat list of terms and summed with `optlang.symbolics.add`, and the gap-fill uses the same construction. The `sum()` of Python simplifies the growing expression again at each term, and the balances of hub metabolites such as ATP, H⁺ and H₂O are long sums. ## Use of the result -The agreement was measured with the code defaults `big_m=1000` (`_BIG_M` in `raven_toolbox/init/ftinit.py`, listed in [Tuned parameter defaults](../../tuned-parameters.md))) and rescaling on (`scale=True` in `prep_init_model`, with `max_stoich_diff=25` in `rescale_for_init`, both in `raven_toolbox/init/prep.py`). A raven-toolbox model agreed most closely with a RAVEN `ftINIT` model when `prep_init_model` received the same task list as the preparation of RAVEN. +The agreement was measured with the code defaults `big_m=1000` (`_BIG_M` in `raven_toolbox/init/ftinit.py`, listed in [Tuned parameter defaults](../../tuned-parameters.md)) and rescaling on (`scale=True` in `prep_init_model`, with `max_stoich_diff=25` in `rescale_for_init`, both in `raven_toolbox/init/prep.py`). A raven-toolbox model agreed most closely with a RAVEN `ftINIT` model when `prep_init_model` received the same task list as the preparation of RAVEN. ## Limits diff --git a/docs/parameter-tuning/studies/init-param-calibration.md b/docs/parameter-tuning/studies/init-param-calibration.md index 8d674e9..29ee807 100644 --- a/docs/parameter-tuning/studies/init-param-calibration.md +++ b/docs/parameter-tuning/studies/init-param-calibration.md @@ -1,20 +1,22 @@ # ftINIT parameter calibration and input robustness -The study runs the ftINIT of raven-toolbox on Human-GEM with the HCT116 expression data of the Hart2015 dataset and varies one or two settings per series of runs. The scaling runs and the full-pipeline runs vary the scaling and `big_m` together. The calibration part varies the settings `mip_gap`, `big_m` and `force_on` of the mixed-integer linear program (MILP) and the stoichiometric scaling of the prepared model, and records the run time, the MILP objective and the reactions that are kept. The robustness part degrades the expression input by dropout, noise and downsampling, extracts a model with the metabolic-task layer on, and records the size of the model, the essential tasks that it passes and its overlap with the model from clean data. +The study runs ftINIT of raven-toolbox on Human-GEM with the HCT116 expression data of the Hart2015 dataset and varies one or two settings per series of runs. The scaling runs and the full-pipeline runs vary the scaling and `big_m` together. The calibration part varies the settings `mip_gap`, `big_m` and `force_on` of the mixed-integer linear program (MILP) and the stoichiometric scaling of the prepared model, and records the run time, the MILP objective and the reactions that are kept. The robustness part degrades the expression input by dropout, noise and downsampling, extracts a model with the metabolic-task layer on, and records the size of the model, the essential tasks that it passes and its overlap with the model from clean data. The overlap is the Jaccard index of two kept-reaction sets: the reactions in both divided by the reactions in either. Steps are counted from 1 on the pages of this study. The run logs count from 0, so the logged `step 1` is the second step. The main findings are: -- In the first-step solve, every `big_m` from 10 to 1000 and every scaling setting was feasible, and the time stayed between 23 and 30 s. -- In the full pipeline, `big_m=1000` was feasible with and without scaling (83 s and 157 s), `big_m=100` was feasible with scaling only, and `big_m=25` was infeasible with scaling. -- Noise changed the model less than sparse input, and no tested parameter recovered the expression evidence that dropout removes. +- In the first-step solve, every `big_m` from 10 to 1000 with every `max_stoich_diff` and with the unscaled model was feasible, and the time stayed between 23 and 30 s. +- In the full pipeline, `big_m=1000` was feasible with and without scaling (83 s and 157 s), `big_m=100` was feasible with scaling and infeasible without, and `big_m=25` was infeasible with scaling. +- Noise changed the model less than sparse input. At 70 % dropout no tested parameter recovered the expression evidence that dropout removes, which is an inference from four lever settings. +- In the first-step solve, `mip_gap` up to 0.01 left the Jaccard index at 0.99 or above and fell to 0.9649 and 0.9663 at 0.03 and 0.1, and `force_on` from 0.02 to 0.5 left the index at 0.987 or more. In the full pipeline `mip_gap` had no effect. +- The set-up has no repeats of the seed, so the seed-to-seed spread of the set-up is unknown. ## Set-up ### Software and inputs -The calibration runs used Human-GEM commit `38567ab8` (v2.1.1 plus 15 commits), the HCT116 column of the Hart2015 RNA sequencing file, raven-toolbox commit `94f3940` and Gurobi on 4 cores. The Gurobi library is 13.0.3 from the gurobipy wheel (most likely, because the logs print no version). The template is the reference model of the [Human-GEM cell-type model validation against RAVEN](humangem-validation.md): Human-GEM without the drug reactions, the amino-acid triplet reactions and the duplicate complex I reaction, with 11265 reactions. `prep_init_model` prepared it without a task list. Its scaled merged model, which is the model that the MILP receives, has 6657 reactions. The preparation took 26 min and is excluded from every time in the tables. +The calibration runs used Human-GEM commit `38567ab8` (v2.1.1 plus 15 commits), the HCT116 column of the Hart2015 RNA sequencing file, raven-toolbox commit `94f3940` and Gurobi on 4 cores. The Gurobi library is 13.0.3 from the gurobipy wheel (most likely; the logs print no version). The template is the reference model of the [Human-GEM cell-type model validation against RAVEN](humangem-validation.md): Human-GEM without the drug reactions, the amino-acid triplet reactions and the duplicate complex I reaction, with 11265 reactions. `prep_init_model` prepared it without a task list. Its scaled merged model, which is the model that the MILP receives, has 6657 reactions. The preparation took 26 min and is excluded from every time in the tables. The robustness runs used Human-GEM at a commit that is not recorded, the same HCT116 data, a template prepared with the essential metabolic tasks, and `big_m=100`. The two parts therefore have no common baseline, and their reaction counts and Jaccard indices cannot be compared with each other. @@ -29,9 +31,9 @@ Gene scores are `5·ln(level/1)` of each expression level, clipped to [−5, 10] | Full pipeline | One call of `ftinit` with series `1+1` and `time_limit=600` on the prepared model without tasks: both MILP steps, each with its own schedule of runs | `bigm/bm.py full` | | Robustness run | One call of `ftinit` with series `1+1` on the template prepared with tasks, including the gap-fill that adds reactions until every task is feasible, followed by `check_tasks` on the extracted model | `scripts/analyze_init_robustness.py` (`MIP_GAP` 0.02, `TIME_LIMIT` 120), phases `gradient` and `levers` (not recoverable) | -A named series such as `1+1` ignores `mip_gap` and `mip_gap_abs`. Every step runs its own schedule of MILP runs, with limits of 120 s, 5000 s and 5 s. The setting `time_limit` lowers the limit of each scheduled run, so at `time_limit=600` it shortens the 5000 s runs only. The setting `mip_gap` therefore acts on the first-step solve and not on the full pipeline. +A named series such as `1+1` ignores `mip_gap` and `mip_gap_abs`. Every step runs its own schedule of MILP runs, with limits of 120 s, 5000 s and 5 s. `time_limit` lowers the limit of each scheduled run, so at `time_limit=600` the 5000 s limit becomes 600 s and the 120 s and 5 s limits stay. `mip_gap` therefore acts on the first-step solve. -The setting `big_m` is the cap on the per-reaction big-M of the on and off constraints. Each scored reaction gets `max(min(ub, big_m), min(-lb, big_m), force_on)`, and an infinite bound counts as `big_m`, so `big_m` limits the flux that a reaction can carry while it counts as on. The setting `force_on` is the smallest flux at which a reaction counts as on. The setting `max_stoich_diff` is the largest ratio between the stoichiometric coefficients of one reaction after `rescale_for_init` (`rescaleModelForINIT` of RAVEN). +`big_m` is the cap on the per-reaction big-M of the on and off constraints. Each scored reaction gets `max(min(ub, big_m), min(-lb, big_m), force_on)`, and an infinite bound counts as `big_m`, so `big_m` limits the flux that a reaction can carry while it counts as on. `force_on` is the smallest flux at which a reaction counts as on. `max_stoich_diff` is the largest ratio between the stoichiometric coefficients of one reaction after `rescale_for_init` (`rescaleModelForINIT` of RAVEN). The robustness runs degrade the expression vector of HCT116 in three ways, each with one random draw (seed 0): @@ -53,9 +55,9 @@ The lever runs repeat the 70 % dropout input with `no_gene_score` or `force_on` | Jaccard | Jaccard index of two kept-reaction sets. First-step sets are merged-model reaction identifiers, and full-pipeline and robustness sets are Human-GEM reaction identifiers | | tasks pass, frac | Essential tasks that `check_tasks` reports as passed on the extracted model, as a count and as a fraction | -A MIP gap bounds the objective only. Two solutions within the gap can keep different reactions, so the reaction set is compared separately from the objective. +A MIP gap bounds the objective. Two solutions within the gap can keep different reactions, so the reaction set is compared separately from the objective. -Each table row is one run unless the row states repeats. This set-up has no runs that differ only in the seed. The only measure of the seed-to-seed spread comes from another set-up, in [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](ftinit-determinism.md). There, two baseline builds that differed only in the seed (Human-GEM v2.0.0-38, DLD1, `big_m=100`, escalation schedule, with tasks and gap-fill) differed by 110 reactions, which is the symmetric difference of the kept-reaction sets. The two builds kept 7779 and 7781 reactions, so the 110 reactions are 1.4 % of the model and correspond to a Jaccard index of 0.986. That pair compares two full pipelines with gap-fill, whereas a first-step set is the result of one MILP step on merged-model identifiers without tasks, which is another kind of object. Between the settings of the calibration the Jaccard index ranged from 0.9976 to 0.9649, a difference of 0.2 to 3.5 % from identical sets. +Each table row is one run unless the row states repeats. This set-up has no runs with the same settings and different seeds. The measure of the seed-to-seed spread comes from another set-up, in [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](ftinit-determinism.md). There, two baseline builds that had the same settings and different seeds (Human-GEM v2.0.0-38, DLD1, `big_m=100`, escalation schedule, with tasks and gap-fill) differed by 110 reactions, which is the symmetric difference of the kept-reaction sets. The two builds kept 7779 and 7781 reactions, so the 110 reactions are 1.4 % of the model and correspond to a Jaccard index of 0.986. That pair compares two full pipelines with gap-fill; a first-step set is the result of one MILP step on merged-model identifiers without tasks, which is another kind of object. Between the settings of the calibration the Jaccard index ranged from 0.9976 to 0.9649, a difference of 0.2 to 3.5 % from identical sets. ### Source of the numbers @@ -116,7 +118,7 @@ The scaling runs used `mip_gap=0.001` and two `big_m` values. Every rel.obj.gap | `msd=100` | 26 | +0.0048 | 0.980 | 25 | +0.0049 | 0.980 | | scale off | 30 | −0.0050 | 0.984 | 25 | −0.0044 | 0.983 | -The reference row of this table and the `mip_gap=0.001` row of the first-step `mip_gap` table have the same settings and come from different runs. The scaling run rescales an unscaled preparation with `rescale_for_init` and has a 600 s limit, whereas the first-step run uses `prep(scale=True)` and a 900 s limit. Their objectives are 45539.6 and 45949.2, and their reaction sets share a Jaccard index of 0.983. +The reference row of this table and the `mip_gap=0.001` row of the first-step `mip_gap` table have the same settings and come from different runs. The scaling run rescales an unscaled preparation with `rescale_for_init` and has a 600 s limit, and the first-step run uses `prep(scale=True)` and a 900 s limit. Their objectives are 45539.6 and 45949.2, and their reaction sets share a Jaccard index of 0.983. ### Full pipeline with and without scaling @@ -188,16 +190,17 @@ In the first-step solve the time was 24 to 25 s at every gap from 0.0002 to 0.1. ### Effect of `big_m` and scaling -In the first-step solve every `big_m` from 10 to 1000 was feasible, took 23 to 26 s and reached the objective of `big_m=1000` within 0.0003. Against the run at `big_m=1000` and against the run at `big_m=100`, the reaction sets of the other caps had Jaccard indices of 0.983 to 0.991. The first-step solve has no repeats of the seed, and the seed pair in [Set-up](#set-up) compares full pipelines with gap-fill, so it is no reference for these sets. In the first-step scaling runs, every `max_stoich_diff` from 10 to 100 and the unscaled model were feasible at both caps, with Jaccard indices of 0.980 to 0.991 against the scaled reference. +In the first-step solve every `big_m` from 10 to 1000 was feasible, took 23 to 26 s and reached the objective of `big_m=1000` within 0.0003. Against the run at `big_m=1000` and against the run at `big_m=100`, the reaction sets of the other caps had Jaccard indices of 0.983 to 0.991. The seed pair of the [Set-up](#set-up) is no reference for these sets. In the first-step scaling runs, every `max_stoich_diff` from 10 to 100 and the unscaled model were feasible at both caps, with Jaccard indices of 0.980 to 0.991 against the scaled reference. The full pipeline separates the settings that the first-step solve does not. - At `big_m=1000` it was feasible with and without scaling. It took 83 s with scaling and 157 s without, and the two models shared a Jaccard index of 0.969 (7312 and 7264 reactions). - At `big_m=100` it was feasible with scaling (117 to 119 s) and infeasible without. - At `big_m=25` it was infeasible with scaling. -- With scaling, the caps from 50 to 1000 gave runs of 81 to 119 s with no trend in `big_m`, and reaction sets that shared a Jaccard index of 0.978 to 0.982 with the default. The fastest run at a cap below 1000 (80.8 s at `big_m=50`, with `mip_gap_abs=None`) was as fast as the fastest `big_m=1000` run (81.1 s). +- With scaling, the caps from 50 to 1000 gave runs of 81 to 119 s with no trend in `big_m`, and reaction sets that shared a Jaccard index of 0.978 to 0.982 with the default. +- The fastest run at a cap below 1000 (80.8 s at `big_m=50`, with `mip_gap_abs=None`) was as fast as the fastest `big_m=1000` run (81.1 s). -Counted from the reaction sets, the sets at the caps 500, 250, 100 and 50 differed from the `big_m=1000` set by 150, 165, 140 and 133 reactions. The rounded Jaccard values in the table give 148 and 163 for the first two. The one seed pair available, from the other set-up, differed by 110 reactions, so every cap difference is 21 to 50 % larger than that single value, and the seed-to-seed spread of this set-up is unknown. The three `big_m=1000` runs kept identical sets (Jaccard 1.000) and only the cap changed between rows, so the differences follow the cap. Whether they are systematic or a different choice among near-equal optima was not tested. A lower cap also reduces the flux that a kept reaction can carry. +Counted from the reaction sets, the sets at the caps 500, 250, 100 and 50 differed from the `big_m=1000` set by 150, 165, 140 and 133 reactions. The rounded Jaccard values in the table give 148 and 163 for the first two. Every cap difference is 21 to 50 % larger than the 110 reactions of the seed pair of the [Set-up](#set-up). The three `big_m=1000` runs kept identical sets (Jaccard 1.000) and only the cap changed between rows, so the differences follow the cap. Whether they are systematic or a different choice among near-equal optima was not tested. ### Effect of `force_on` @@ -207,13 +210,13 @@ The setting `force_on` changes the MILP itself, through the on and off constrain Noise changed the model less than sparse input. At σ=1.0 and σ=2.0 the model kept 7812 and 7768 reactions against 7777 for clean data, passed all 69 tasks and shared a Jaccard index of 0.952 and 0.919 with the clean model. Dropout of 50 % and 70 % reduced the model to 5968 and 5113 reactions (Jaccard 0.713 and 0.594), and downsampling of 50 % and 70 % to 6765 and 6123 reactions (Jaccard 0.815 and 0.728). At equal fractions dropout removed more reactions than downsampling. The difference matches the scores involved, −5 for a zero level against −2 for a reaction without measured genes, but this explanation was not tested. Under dropout and downsampling the model passed 67 or 68 of 69 tasks, and the failing tasks and the cause of the failures were not examined. -At 70 % dropout no tested lever moved the model back towards the clean model. The Jaccard index stayed at 0.593 to 0.600, the size at 5110 to 5159 reactions and the task fraction at 0.986. Under dropout the dropped genes stay in the data with a zero level and score −5, so `no_gene_score` acts only on reactions without gene rules and on reactions whose genes are all absent from the data. The conclusion that no parameter recovers the expression evidence that dropout removes is an inference from the four lever settings, which do not include other parameters. The lever runs did not include downsampled input, so the effect of `no_gene_score` on missing genes was not measured. +At 70 % dropout no tested lever moved the model back towards the clean model. The Jaccard index stayed at 0.593 to 0.600, the size at 5110 to 5159 reactions and the task fraction at 0.986. Under dropout the dropped genes stay in the data with a zero level and score −5, so `no_gene_score` acts on reactions without gene rules and on reactions whose genes are all absent from the data. The conclusion that no parameter recovers the expression evidence that dropout removes is an inference from the four lever settings, which do not include other parameters. The lever runs did not include downsampled input, so the effect of `no_gene_score` on missing genes was not measured. -In raven-toolbox commit `94f3940`, `ftinit` calls `fill_tasks` without a time limit, so the gap-fill uses the default of `fill_tasks`, 300 s per task (`_FILL_TIME_LIMIT`), and not the `time_limit` of the extraction. The time limit that the 90 % dropout run of more than 75 min used is not recorded. +In raven-toolbox commit `94f3940`, `ftinit` calls `fill_tasks` without a time limit, so the gap-fill uses the default of `fill_tasks`, 300 s per task (`_FILL_TIME_LIMIT`), independent of the `time_limit` of the extraction. The time limit that the 90 % dropout run of more than 75 min used is not recorded. ### Choice of settings -The defaults `big_m=1000`, `force_on=0.1` and scaling on with `max_stoich_diff=25` are the values of RAVEN. Every `big_m` from 50 to 1000 was feasible with scaling on, and `big_m=25` was infeasible. The cap `big_m=1000` is the one that `ftINITInternalAlg` of RAVEN uses (`bigMCap`). Scaling is not needed for feasibility at `big_m=1000`, and in these single runs the pipeline took 83 s with scaling and 157 s without. In the first-step solve, `mip_gap=0.001` reached an objective within 0.0001 of the tightest gap, and a named series ignores the setting. The full-pipeline runs used `time_limit=600` and the robustness script uses `time_limit=120`. No full-pipeline run took longer than 157 s in total. For noisy input the tested settings needed no change, and no tested setting compensated for dropout. +The defaults `big_m=1000`, `force_on=0.1` and scaling on with `max_stoich_diff=25` are the values of RAVEN. Every `big_m` from 50 to 1000 was feasible with scaling on, and `big_m=25` was infeasible. The cap `big_m=1000` is the one that `ftINITInternalAlg` of RAVEN uses (`bigMCap`). Scaling is not needed for feasibility at `big_m=1000`, and in these single runs the pipeline took 83 s with scaling and 157 s without. In the first-step solve, `mip_gap=0.001` reached an objective within 0.0001 of the tightest gap, and a named series ignores the setting. The full-pipeline runs used `time_limit=600` and the robustness script uses `time_limit=120`. No full-pipeline run took longer than 157 s in total. ## Gene-scoring constants @@ -223,20 +226,20 @@ Expression values become gene scores through score = clip(factor × ln(level / reference), min_score, max_score) ``` -with `factor=5.0`, `max_score=10.0` and `min_score=-5.0`, the same in both implementations. The logarithm is natural (base e) in `init/score.py` and in `scoreModel` of RAVEN. The source of each implementation cites no paper for the three constants. A 2012 paper (Wang et al. 2012, [mCADRE](https://doi.org/10.1186/1752-0509-6-153)) uses categorical scoring and does not give these constants. The constants are values of RAVEN itself without a literature source. +with `factor=5.0`, `max_score=10.0` and `min_score=-5.0`, the same in both implementations. The logarithm is natural (base e) in `init/score.py` and in `scoreModel` of RAVEN. The source of each implementation cites no paper for the three constants. A 2012 paper (Wang et al. 2012, [mCADRE](https://doi.org/10.1186/1752-0509-6-153)) uses categorical scoring and did not introduce these constants. The constants are values of RAVEN itself without a literature source. ## Comparison with the solver benchmark -[ftINIT on different solvers](init-solver-benchmark.md) runs the same full pipeline with each solver interface. With Gurobi it took 92 s and kept 7312 reactions, against 83 s and 7312 reactions here. The benchmark time also includes loading the prepared model and scoring the reactions, and its limit is 900 s against 600 s here. A limit shortens only the 5000 s runs of the schedule, and no Gurobi run in either study ran for its overall limit. +[ftINIT on different solvers](init-solver-benchmark.md) runs the same full pipeline with each solver interface. With Gurobi it took 92 s and kept 7312 reactions, against 83 s and 7312 reactions in this study. The benchmark time also includes loading the prepared model and scoring the reactions, and its limit is 900 s against 600 s in this study. A limit shortens the 5000 s runs of the schedule, and no Gurobi run in either study ran for its overall limit. -The warning `ftINIT step 1 hit the 600s time limit` appears 14 times in 13 full-pipeline and validation logs, and the Gurobi run of the solver benchmark has the same warning with 900 s. The logged `step 1` is the second step. The warning prints the `time_limit` argument and not the limit of the run that stopped. The stopped run is a shorter scheduled run, inferred from the total times of 83 s and 92 s, and the logs do not say which. One scheduled solve per run stopped at a limit. +The warning `ftINIT step 1 hit the 600s time limit` appears 14 times in 13 full-pipeline and validation logs, and the Gurobi run of the solver benchmark has the same warning with 900 s. The logged `step 1` is the second step. The warning prints the `time_limit` argument, which differs from the limit of the run that stopped. The stopped run is a shorter scheduled run, inferred from the total times of 83 s and 92 s, and the logs do not say which. One scheduled solve per run stopped at a limit. -The Gurobi run of the solver benchmark at `big_m=100` kept 7324 reactions, the count of the two `big_m=100` runs here, whose reaction set shares a Jaccard index of 0.981 with the `big_m=1000` set. The solver comparison itself is on that page: GLPK at 2182 s with a Jaccard index of 0.892 against Gurobi, and the failure of HiGHS through the `scipy` interface. +The Gurobi run of the solver benchmark at `big_m=100` kept 7324 reactions, the count of the two `big_m=100` runs of this study, whose reaction set shares a Jaccard index of 0.981 with the `big_m=1000` set. The solver comparison itself is on that page: GLPK at 2182 s with a Jaccard index of 0.892 against Gurobi, and the failure of HiGHS through the `scipy` interface. ## Limits -- One model (Human-GEM), one cell line (HCT116) and one solver (Gurobi library 13.0.3 from the gurobipy wheel, most likely, because the logs print no version). -- Most settings ran once, and the repeats are listed with the full-pipeline table. This set-up has no repeats of the seed. The full-pipeline differences between caps are larger than the one seed pair available, which comes from another set-up, so the seed-to-seed spread of this set-up is unknown. +- One model (Human-GEM), one cell line (HCT116) and one solver (Gurobi library 13.0.3 from the gurobipy wheel, most likely; the logs print no version). +- Most settings ran once, and the repeats are listed with the full-pipeline table. This set-up has no repeats of the seed. The full-pipeline differences between caps are larger than the seed pair of the Set-up, so the seed-to-seed spread of this set-up is unknown. - The robustness runs used a Human-GEM commit that is not recorded and `big_m=100`. The script at the pinned commit passes no `big_m`, so a run with it uses the default 1000. Each degradation is one random draw at one seed. - The first-step solve shows no difference between caps or scaling settings that the full pipeline shows, so first-step results alone do not predict the feasibility of the pipeline. - The robustness tables have no run times, and the effect of `no_gene_score` on downsampled input was not measured. diff --git a/docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md b/docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md index 9722654..b0d30be 100644 --- a/docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md +++ b/docs/parameter-tuning/studies/kegg-hmm-cutoff-calibration.md @@ -25,9 +25,9 @@ The percentile, cut-off, score-ratio and defaults tables come from one run on 20 ### Reference and scores -The reference is the **KEGG annotation**, which is the gene→KO pairs of the organism in the `organism_gene_ko` table of the kegg118 artefacts. The table lists the KOs linked to at least one reaction in `ko_reaction`, and the HMM libraries are built from the same table. The **true reactions** of the organism are the reactions linked to its annotated KOs, limited to reactions that are in the kegg118 reference model (`--restrict-to-reference`). The limit is applied because the reaction recall is measured against reactions that the draft can contain. +The reference is the KEGG annotation, which is the gene→KO pairs of the organism in the `organism_gene_ko` table of the kegg118 artefacts. The table lists the KOs linked to at least one reaction in `ko_reaction`, and the HMM libraries are built from the same table. The true reactions of the organism are the reactions linked to its annotated KOs, limited to reactions that are in the kegg118 reference model (`--restrict-to-reference`). The limit is applied because the reaction recall is measured against reactions that the draft can contain. -A predicted pair is the output of `assign_kos`, with the `organism:` prefix stripped from the gene identifier so that it matches the reference. Each predicted pair is either **matched** (present in the annotation) or **novel** (absent from it). +A predicted pair is the output of `assign_kos`, with the `organism:` prefix stripped from the gene identifier so that it matches the reference. Each predicted pair is either matched (present in the annotation) or novel (absent from it). | Score | Definition | |---|---| @@ -237,4 +237,4 @@ The script `analyze_hmm_cutoffs.py` is called once per organism with `--artefact The proteomes (`kegg_rest/.faa`, `uniprot/.faa`, `counts.json`) are in the archived input bundle `kegg_proteomes_2026-11-01.tar.gz`, and the four `hmmsearch --tblout` tables are in `kegg_hmmsearch_tblout_2026-11-01.tar.gz`. -The scores of the tables come from the run of 2026-11-01 with raven-toolbox commit `94f3940`, HMMER 3.4 and the published kegg118 tables and libraries, searched with 16 threads on an AMD EPYC 9354 node. Scoring each saved hit table again with `--tblout` reproduces the saved scores. Two points were not checked: whether the KEGG REST sequences of 2026-10-31 equal the sequences of KEGG release 118 (the sequence counts were compared, not the sequences), and whether `hmmsearch -Z` E-values equal `hmmscan` E-values on these inputs, because the binary bundle has no `hmmscan`. The UniProt proteomes in the bundle were downloaded and counted but not searched. +The scores of the tables come from the run of 2026-11-01 with raven-toolbox commit `94f3940`, HMMER 3.4 and the published kegg118 tables and libraries, searched with 16 threads on an AMD EPYC 9354 node. Scoring each saved hit table with `--tblout` reproduces the saved scores. Two points were not checked: whether the KEGG REST sequences of 2026-10-31 equal the sequences of KEGG release 118 (the sequence counts were compared, not the sequences), and whether `hmmsearch -Z` E-values equal `hmmscan` E-values on these inputs, because the binary bundle has no `hmmscan`. The UniProt proteomes in the bundle were downloaded and counted but not searched. diff --git a/docs/parameter-tuning/studies/sampling-convergence-calibration.md b/docs/parameter-tuning/studies/sampling-convergence-calibration.md index 5ab0277..dbab9ed 100644 --- a/docs/parameter-tuning/studies/sampling-convergence-calibration.md +++ b/docs/parameter-tuning/studies/sampling-convergence-calibration.md @@ -2,7 +2,7 @@ The function `random_sampling` of raven-toolbox draws flux vectors from a model with a Markov-chain sampler (`method='achr'` or `'chrr'`) or with random objectives (`method='random_objective'`). The study ran the two Markov-chain samplers on e_coli_core (95 reactions) and yeast-GEM and recorded three results: how strongly consecutive samples of one chain are correlated, whether independent chains reach the same flux distribution, and the wall time. It also records why two settings of the random-objective method differ from RAVEN in MATLAB. -On e_coli_core, four chains of 300 samples agree at the R-hat threshold of 1.1 for every reaction, and the CHRR sampler is no better than ACHR there. On yeast-GEM, 50.7 % of the scored reactions exceed an R-hat of 1.1 at the default `thinning`, and no tested setting gave converged samples. +On e_coli_core, four chains of 300 samples agree at the R-hat threshold of 1.1 for 87 of 87 scored reactions, and the CHRR sampler was no better than ACHR there in single runs. On yeast-GEM, 50.7 % of the scored reactions exceed an R-hat of 1.1 for `n_samples=300` at the default `thinning=100`; the default `n_samples` of 1000 was not tested. No setting measured with R-hat or `ρ` gave converged samples. ## Terms @@ -51,7 +51,13 @@ The same settings with seeds 0 to 3 give the following `ρ`. For every seed, `ρ` falls as `thinning` rises. Between seeds, `ρ` differs by up to 0.28, and the ESS of 300 stored samples ranges from 3.2 to 11.8 at `thinning=20`, from 9.9 to 63.1 at 100 and from 28.8 to 64.9 at 500. Seed 1 gives the lowest `ρ` at all three values, and without it the ranges are 0.972 to 0.979, 0.924 to 0.936 and 0.769 to 0.825. The four chains of the yeast-GEM comparison with 4,105 reactions (seeds 0 to 3, `thinning=100`) give a mean `ρ` of 0.951, 0.847, 0.947 and 0.911, so a low value for seed 1 appears on both model files. The spread between seeds is 0.10 on the 4,105-reaction chains (0.847 to 0.951) and 0.28 on the 4,102-reaction chains (0.652 to 0.936), and seed 1 is the lowest in both. -The value `n_samples` was not varied. With `ρ` fixed at its seed-0 value for the default `thinning=100`, the ESS formula is proportional to `n_samples`: about 10 from 300 stored samples, about 33 from the default 1000, and 100 needs about 3,000 stored samples. Building the ACHR warmup points alone took 185.9 s, 196.6 s and 195.4 s in the three runs, so the sampling steps took about 1 s, 5 s and 26 s for 6,000, 30,000 and 150,000 steps. The warmup is therefore about 95 % of the wall time at `thinning=100`, and the wall time rose from 189.5 s to 223.3 s (1.2-fold) while the number of steps rose 25-fold. With a fixed cost per run plus a cost per stored sample, 3,025 stored samples take about 242 s for one chain. Scaling the 203.6 s in proportion to 3,025 samples gives 2,053 s (about 34 min), an upper bound. The warmup time was measured on a second, fresh sampler after the run, so the split is approximate. +The value `n_samples` was not varied. With `ρ` fixed at its seed-0 value for the default `thinning=100`, the ESS formula is proportional to `n_samples`: about 10 from 300 stored samples, about 33 from the default 1000, and 100 needs about 3,000 stored samples. Building the ACHR warmup points alone took 185.9 s, 196.6 s and 195.4 s in the three runs, so the sampling steps took about 1 s, 5 s and 26 s for 6,000, 30,000 and 150,000 steps. + +The warmup is therefore about 95 % of the wall time at `thinning=100`, and the wall time rose from 189.5 s to 223.3 s (1.2-fold) while the number of steps rose 25-fold. + +With a fixed cost per run plus a cost per stored sample, 3,025 stored samples take about 242 s for one chain. Scaling the 203.6 s in proportion to 3,025 samples gives 2,053 s (about 34 min), an upper bound. + +The warmup time was measured on a second, fresh sampler after the run, so the split is approximate. Within the measured range, a larger `thinning` gave more ESS per second of wall time (4.2 in 189.5 s at 20, 28.8 in 223.3 s at 500). That a larger `n_samples` at a fixed `thinning` raises the ESS in proportion and leaves `ρ` unchanged follows from the first-order autoregressive formula, and it was not measured. @@ -100,7 +106,7 @@ The ACHR run used 4 chains × 300 samples and took 572.3 s wall time (about 9.5 The reactions with the highest R-hat were `r_4562` (18.90), `r_4358` (18.90), `r_0992` (5.18), `r_4249` (5.18), `r_4701` (5.18), `r_4621` (5.18), `r_1965` (5.09), `r_0917` (3.90), `r_0891` (3.90) and `r_0918` (3.90). -The median scored reaction was 1.1020, and one in two reactions (50.7 %) exceeded 1.1, whereas 4.8 % passed 1.01. Each chain took 300 × 100 = 30,000 ACHR steps after its warmup points. The low ESS of one chain and the disagreement between chains are two separate measurements, and both show that these settings do not give a converged sample of yeast-GEM. +The median scored reaction was 1.1020, and one in two reactions (50.7 %) exceeded 1.1, and 4.8 % were at or below 1.01. Each chain took 300 × 100 = 30,000 ACHR steps after its warmup points. The low ESS of one chain and the disagreement between chains are two separate measurements, and both show that these settings do not give a converged sample of yeast-GEM. The four parallel chains took 572.3 s, 2.8 times the 203.6 s of the single chain at the same settings. The two runs read different yeast-GEM files (4,105 and 4,102 reactions). Contention between the four Gurobi processes on the 12 cores is a possible cause of the difference, and it was not measured. @@ -120,7 +126,7 @@ The 30,000 steps per chain of the default run (`thinning=100, n_samples=300`) we | R-hat > 1.1 | 50.7 % | 50.5 % | | worst reaction | `r_4562`, `r_4358` (18.90) | `r_4562`, `r_4358` (21.91) | -Both runs scored 3325 reactions. The reallocation moved the median R-hat from 1.1020 to 1.1010 and the shares above the two thresholds by at most 1.7 percentage points. The reactions with the highest R-hat were the same, and the run took 612.4 s against 572.3 s (1.07 times). That the disagreement comes from how ACHR moves through this polytope, and not from the split of the steps, is an inference from this result, and it was not tested at genome scale. At a fixed number of steps, moving steps from `n_samples` to `thinning` did not improve the agreement between chains, and a larger total number of steps was not tested with R-hat. +Both runs scored 3325 reactions. The reallocation moved the median R-hat from 1.1020 to 1.1010 and the shares above the two thresholds by at most 1.7 percentage points. The reactions with the highest R-hat were the same, and the run took 612.4 s against 572.3 s (1.07 times). That the disagreement comes from how ACHR moves through this polytope, and not from the split of the steps, is an inference from this result, and it was not tested on other models. At a fixed number of steps, moving steps from `n_samples` to `thinning` did not improve the agreement between chains, and a larger total number of steps was not tested with R-hat. ### CHRR on e_coli_core @@ -152,9 +158,9 @@ Twenty samples per chain are too few for R-hat on yeast-GEM, and whether CHRR co ## Choice of `thinning` and `n_samples` -The defaults are `method='achr'`, `thinning=100` and `n_samples=1000` ([Tuned parameter defaults](../../tuned-parameters.md)). The default is ACHR. On e_coli_core, CHRR differed from ACHR by +0.0012 in median R-hat (1.0027 and 1.0039), by −0.0034 in p90 (1.0171 and 1.0137) and by +0.0101 in maximum (1.0276 and 1.0377), with one run each and no measured spread between sets of seeds, and CHRR took 66 to 133 times as long. +The defaults are `method='achr'`, `thinning=100` and `n_samples=1000` ([Tuned parameter defaults](../../tuned-parameters.md)). ACHR is the default because on e_coli_core CHRR differed from ACHR by +0.0012 in median R-hat (1.0027 and 1.0039), by −0.0034 in p90 (1.0171 and 1.0137) and by +0.0101 in maximum (1.0276 and 1.0377), with one run each and no measured spread between sets of seeds, and CHRR took 66 to 133 times as long. -On a model the size of e_coli_core, the default `thinning` with 300 samples per chain gave chains that agree at R-hat 1.1 for every reaction. On yeast-GEM, no setting tested gave converged samples. A `thinning` up to 500 left `ρ` at 0.825 for seed 0 (0.644 to 0.825 over four seeds), and moving the same 30,000 steps from `n_samples` to `thinning` moved the median R-hat from 1.1020 to 1.1010 and the shares above 1.01 and 1.1 by at most 1.7 percentage points. With `n_samples=300` and `thinning=100`, 50.7 % of the 3325 scored reactions had R-hat above 1.1. +On a model the size of e_coli_core, the default `thinning` with 300 samples per chain gave chains that agree at R-hat 1.1 for every scored reaction. On yeast-GEM, no setting measured with R-hat or `ρ` gave converged samples. A `thinning` up to 500 left `ρ` at 0.825 for seed 0 (0.644 to 0.825 over four seeds), and moving the same 30,000 steps from `n_samples` to `thinning` moved the median R-hat from 1.1020 to 1.1010 and the shares above 1.01 and 1.1 by at most 1.7 percentage points. With `n_samples=300` and `thinning=100`, 50.7 % of the 3325 scored reactions had R-hat above 1.1. The docstring of `random_sampling` recommends, for genome-scale models, a `thinning` of at least 1000 or a larger `n_samples`, `method='chrr'`, or an ESS or R-hat check. A `thinning` of 1000 or more was not measured. The driver script runs the R-hat check: it samples several chains with different seeds at given settings and reports the share of reactions above the R-hat thresholds, so a sampling result can be checked at the settings that produced it. diff --git a/docs/raven3-migration.md b/docs/raven3-migration.md index 675c2bf..f9c36bc 100644 --- a/docs/raven3-migration.md +++ b/docs/raven3-migration.md @@ -1,10 +1,10 @@ # RAVEN 2 to RAVEN 3 -RAVEN 3 is a breaking release. A script written for RAVEN 2 can stop with an error, or run and return a different model. The sections below list what to check first, give the calling convention for optional arguments and provide a checklist for the upgrade. Four detail pages hold the complete lists. +RAVEN 3 is a breaking release. A script written for RAVEN 2 can stop with an error, or run and return a different model. The sections below list what to check first, give the calling convention for optional arguments and provide a checklist for the upgrade. Four detail pages hold the lists. All statements about RAVEN 2 refer to release 2.11.3. The last RAVEN 2 release, 2.11.4, differs from 2.11.3 by bug fixes in `ftINIT`, `mergeLinear`, `replaceMets`, `checkInstallation`, `importModel`, `readYAMLmodel` and `optimizeProb`. The changelog of RAVEN 3 states its changes relative to RAVEN 2.9.3, so an entry in it can describe a difference that does not exist against 2.11.3. The comparison with the Python package is in [Comparison of RAVEN and raven-toolbox](raven3-vs-raven-toolbox.md). -The codebase was reorganised, functions were removed (counts per folder are in [Structure, removals and renames](raven3-migration/structure-and-renames.md)), several external dependencies were dropped, and 112 functions read their optional arguments through `parseRAVENargs`. The rationale is in the [v3 development review](https://github.com/CompMetEng/RAVEN/blob/develop/docs/v3_review.md). Scripts that call reconstruction, KEGG, localisation or gene-deletion functions are the most likely to be affected. The changes that can affect any script are in [Backward-incompatible changes](raven3-migration/backward-incompatible.md#backward-incompatible-changes). +The codebase was reorganised, functions were removed (counts per folder are in [Structure, removals and renames](raven3-migration/structure-and-renames.md)), several external dependencies were dropped, and 112 functions read their optional arguments through `parseRAVENargs`. The rationale is in the [v3 development review](https://github.com/CompMetEng/RAVEN/blob/develop/docs/v3_review.md). Scripts that call reconstruction, KEGG, localisation or gene-deletion functions need a check. The changes that can affect any script are in [Backward-incompatible changes](raven3-migration/backward-incompatible.md#backward-incompatible-changes). :::{admonition} Changes to check first :class: warning @@ -37,8 +37,8 @@ A positional call keeps its meaning where the parameter order is unchanged. This 2. Search the scripts for the functions in [Removed functionality](raven3-migration/structure-and-renames.md#removed-functionality) and in [Renamed and merged functions](#renamed-merged-and-consolidated-functions), and replace or delete those calls. The wrappers in `deprecated/` (`makeSomething`, `consumeSomething`, `canProduce`, `canConsume`) work and warn once per session, as do `getINITModel` and `runINIT`. 3. Rebuild and validate every draft model from `getModelFromKEGG`, `getKEGGModelForOrganism` or `getModelFromHomology`. The default cut-offs, the KEGG data source and the best-hit criterion differ, so an unchanged call gives a different draft (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)). Check the positional calls of the six functions in [Calling convention](#calling-convention). 4. Convert `.xlsx` curation templates to the `.tsv` format of `curateModelFromTables`. The function `importExcelModel` does not exist in RAVEN 3 (see [Excel I/O](raven3-migration/formats-and-reconstruction.md#excel-io)). -5. Replace WoLF PSORT scores (`getWoLFScores`, or `parseScores` with `'wolf'`) with DeepLoc 2, MULocDeep, COMPARTMENTS or UniProt. The default predictor of `parseScores` also differs (see [Backward-incompatible changes](raven3-migration/backward-incompatible.md#backward-incompatible-changes)). -6. Run task-based gap-filling and context-specific model extraction (`fitTasks`, `checkTasks`, `ftINIT`) again and compare the task results. `simplifyModel` keeps the boundary constraint of a task on a metabolite without reactions, which RAVEN 2 removed, so the task check can give a different result in RAVEN 2 and RAVEN 3. This was not run (see [Gap-filling and context-specific model extraction](#ftinit)). The function `fitTasks` returns a third output, `failedTasks`. +5. Replace WoLF PSORT scores (`getWoLFScores`, or `parseScores` with `'wolf'`) with DeepLoc 2, MULocDeep, COMPARTMENTS or UniProt. The default predictor of `parseScores` differs (see [Backward-incompatible changes](raven3-migration/backward-incompatible.md#backward-incompatible-changes)). +6. Run task-based gap-filling and context-specific model extraction (`fitTasks`, `checkTasks`, `ftINIT`) with RAVEN 3 and compare the task results. `simplifyModel` keeps the boundary constraint of a task on a metabolite without reactions, which RAVEN 2 removed, so the task check can give a different result in RAVEN 2 and RAVEN 3. The difference in the task check was not run (see [Gap-filling and context-specific model extraction](#ftinit)). The function `fitTasks` returns a third output, `failedTasks`. 7. Expect differences in YAML and SBML files on the next export of a model under version control. The changelog of RAVEN 3 lists changes in quoting, EC-code annotation and ΔG fields. These differences were not compared file by file (see [File formats and model I/O](raven3-migration/formats-and-reconstruction.md#file-formats-and-model-io)). 8. Compare the output of the validation pipeline of the model (model size, growth rate and task results) between RAVEN 2 and RAVEN 3 before using RAVEN 3 results. diff --git a/docs/raven3-migration/backward-incompatible.md b/docs/raven3-migration/backward-incompatible.md index a1aa930..8dff731 100644 --- a/docs/raven3-migration/backward-incompatible.md +++ b/docs/raven3-migration/backward-incompatible.md @@ -9,12 +9,12 @@ The changes below were found by comparing RAVEN 2.11.3 with RAVEN 3 in the code. |---|---| | `importExcelModel(...)` | Removed. Use `curation/curateModelFromTables.m` (input format `.tsv`). Alternatively convert the template to SBML and read it with `importModel`, or to YAML and read it with `readYAMLmodel` (see [Excel I/O](#excel-io)) | | `importModel(...)` on SBML Level < 3, on Level 3 without FBC version 2, or on any SBML without FBC | Raises an error. The accepted format is SBML Level 3 Version 1 with FBC package version 2 (see [SBML I/O](#sbml-io)) | -| `exportToExcelFormat(model, fileName)` where `fileName` has no file name or no extension | RAVEN 2 exported tab-delimited text. RAVEN 3 raises the error "only export to xlsx format is supported" for every extension other than `.xlsx`, an empty one included. RAVEN 2 already raised this error for a file name with another extension | +| `exportToExcelFormat(model, fileName)` where `fileName` has no file name or no extension | RAVEN 2 exported tab-delimited text. RAVEN 3 raises the error "only export to xlsx format is supported" for every extension other than `.xlsx`, an empty one included. RAVEN 2 already raised this error for a file name with a different extension | | `exportToTabDelimited(...)` | Removed | | `getWoLFScores(...)` | Removed, with all WoLF PSORT support | | `parseScores(..., 'wolf', ...)` | Raises an error whose message names `'deeploc'`, `'mulocdeep'`, `'compartments'` and `'uniprot'`. RAVEN 3 has no function that reads WoLF PSORT output files | | `qMOMA(...)`, `solveQP(...)` | Removed. RAVEN 3 has no code that depends on `quadprog` of the Optimization Toolbox | -| `findGeneDeletions(...)` with `analysisType`, `refModel` or `oeFactor` | The arguments are removed together with `qMOMA` (MOMA-based over-expression analysis), and so are the test types `'sgo'` and `'dgo'`. Single and double gene deletion (`sgd`, `dgd`) by flux balance analysis (FBA) remain. Any argument besides `'testType'` raises `parseRAVENargs:tooManyArgs` | +| `findGeneDeletions(...)` with `analysisType`, `refModel` or `oeFactor` | The arguments are removed together with `qMOMA` (MOMA-based over-expression analysis), and so are the test types `'sgo'` and `'dgo'`. Single and double gene deletion (`sgd`, `dgd`) by flux balance analysis (FBA) remain. More than the `'testType'` argument raises `parseRAVENargs:tooManyArgs` | | `checkRxn(...)` | Removed without a direct successor. To find why a reaction cannot carry flux, use `gapfilling/findLeakMetabolite.m`, `gapfilling/canExchange.m` or `gapfilling/checkProduction.m` | | `dispEM(...)`, `followFluxes(...)`, `followChanged(...)`, `printModel(...)`, `mapCompartments(...)`, `getExpressionStructure(...)`, `getMetsInComp(...)` | Removed. [Removed functionality](#removed-functionality) gives the replacements where one exists | | Any function in `external/metacyc/` | Folder removed, with the 7 functions of the MetaCyc-based reconstruction. RAVEN 3 has no replacement | @@ -22,7 +22,7 @@ The changes below were found by comparing RAVEN 2.11.3 with RAVEN 3 in the code. | Any function in `legacy/` | Folder removed. RAVEN 3 has no replacement (see [Removed functionality](#removed-functionality)) | | `getGenesFromKEGG`, `getMetsFromKEGG`, `getRxnsFromKEGG`, `constructMultiFasta`, `getWSLpath`, `getBlastFromExcel` | Removed (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)) | | `getToolboxVersion(...)` | Removed as a function of its own. It is a local function of `exportForGit` | -| `parallelPoolRAVEN(...)` | Renamed `parallelWorkersRAVEN`, with another signature and output (see [Renamed and merged functions](#renamed-merged-and-consolidated-functions)) | +| `parallelPoolRAVEN(...)` | Renamed `parallelWorkersRAVEN`, with a different signature and output (see [Renamed and merged functions](#renamed-merged-and-consolidated-functions)) | | `getModelFromKEGG(keggPath, keepSpontaneous, keepUndefinedStoich, keepIncomplete, keepGeneral)` | Raises `parseRAVENargs:tooManyArgs`, because `keggPath` is removed and the function has four optional parameters | | `getKEGGModelForOrganism(...)` with 13 to 15 positional arguments | The parameters `nSequences` and `seqIdentity` are removed. A call with 14 or 15 arguments raises `parseRAVENargs:tooManyArgs`. A non-empty 13th argument, which is `nSequences` in RAVEN 2, is read as `globalModel` | | `getPhylDist(keggPath, onlyInKingdom)` | Raises `parseRAVENargs:tooManyArgs`, because `keggPath` is removed | @@ -32,18 +32,18 @@ The changes below were found by comparing RAVEN 2.11.3 with RAVEN 3 in the code. ## Changes that alter results without an error -The same call on the same model gives another output. Positions count all arguments of the call, required ones included: in `scoreModel(model, hpaData, arrayData, tissue, celltype, ...)` the model is position 1 and `celltype` is position 5. +The same call on the same model gives a different output. Positions count all arguments of the call, required ones included: in `scoreModel(model, hpaData, arrayData, tissue, celltype, ...)` the model is position 1 and `celltype` is position 5. ### Argument positions and defaults that change meaning | Function | RAVEN 3 behaviour | |---|---| -| `getModelFromKEGG(keggPath)` to `getModelFromKEGG(keggPath, keepSpontaneous, keepUndefinedStoich[, keepIncomplete])` | A call with one to four positional arguments shifts. The first argument, `keggPath` in RAVEN 2, is read as `keepSpontaneous` and has no effect. Each later argument is read as the next parameter in the RAVEN 2 order, so another class of reactions is removed | +| `getModelFromKEGG(keggPath)` to `getModelFromKEGG(keggPath, keepSpontaneous, keepUndefinedStoich[, keepIncomplete])` | A call with one to four positional arguments shifts. The first argument, `keggPath` in RAVEN 2, is read as `keepSpontaneous` and has no effect. Each later argument is read as the next parameter in the RAVEN 2 order, so a different class of reactions is removed | | `getPhylDist(keggPath)` | `keggPath` is read as `'onlyInKingdom'` and has no effect. The function returns the full distance matrix | | `writeYAMLmodel(model, fileName, true)` | The third argument is read as `sortIds`, so the IDs are sorted. RAVEN 2 read it as `'preserveQuotes'` | | `scoreModel` | `isozymeScoring` and `complexScoring` replace the single `multipleGeneScoring` of RAVEN 2. By default (`complexScoring`) a reaction with an `and` rule gets the minimum over its genes. RAVEN 2 scored every reaction with the maximum over all its genes by default (`multipleGeneScoring` `'best'`) | | `scoreModel(..., celltype, noGeneScore, 'average')` | `'average'` at position 7 among all arguments, which is `multipleGeneScoring` in RAVEN 2, is read as `isozymeScoring`. It does not apply to `and` rules, which use `complexScoring` | -| `scoreModel(..., celltype, noGeneScore, multipleGeneScoring, 'average')` | `'average'` at position 8 among all arguments, which is `multipleCellScoring` in RAVEN 2, is read as `complexScoring` without an error. The average is then taken over the genes of an `and` rule and no longer over cell types | +| `scoreModel(..., celltype, noGeneScore, multipleGeneScoring, 'average')` | `'average'` at position 8 among all arguments, which is `multipleCellScoring` in RAVEN 2, is read as `complexScoring` without an error. The average runs over the genes of an `and` rule; RAVEN 2 averaged over the cell types | | `scoreModel`, output `geneScores` | A gene that is absent from the data receives `NaN`. The RAVEN 2 documentation states `-Inf` | | `scoreModel`, value names | RAVEN 2 accepted `'best'` for `multipleGeneScoring` and `multipleCellScoring`. RAVEN 3 uses `'max'` for `isozymeScoring`, `complexScoring` and `multipleCellScoring`. The parameter order is `celltype`, `noGeneScore`, `isozymeScoring`, `complexScoring`, `multipleCellScoring`, `hpaLevelScores` and `dataPrecedence` (default `'gene'`) | | `parseScores(file)` | The default predictor is `'deeploc'`. RAVEN 2 used `'wolf'`. How a WoLF PSORT file is handled without a predictor was not run | @@ -72,12 +72,9 @@ The changelog of RAVEN 3 lists a bug fix for each of `permuteModel`, `optimizePr ## Changes to error and warning identifiers -RAVEN 2 raised its errors through `dispEM` with an empty identifier (`MException('', ...)`), so no RAVEN 2 script could catch an error by `err.identifier`. RAVEN 3 replaces `dispEM` with `error()` and `warning()` calls. Not every call carries an identifier, and a script can match on `err.identifier` for the calls that do. +RAVEN 2 raised its errors through `dispEM` with an empty identifier (`MException('', ...)`), so no RAVEN 2 script could catch an error by `err.identifier`. RAVEN 3 replaces `dispEM` with `error()` and `warning()` calls. Not every call carries an identifier, and a script can match on `err.identifier` for the calls that do. `RAVEN:badInput` appears in 220 lines and `RAVEN:warning` in 63 lines of `src/` and the root `.m` files (non-comment lines that contain the identifier). Further identifiers in the `RAVEN:` namespace are `RAVEN:modelError`, `RAVEN:badGrRule`, `RAVEN:infeasibleModel`, `RAVEN:infeasible`, `RAVEN:sampling`, `RAVEN:grRuleTooComplex`, `RAVEN:deprecated`, `RAVEN:legacyMethod`, `RAVEN:notMainBranch`, `RAVEN:glpkMILP`, `RAVEN:fetchToolboxAsset:unverified`, `RAVEN:fetchToolboxAsset:checksum`, `RAVEN:fetchToolboxAsset:manifest`, `RAVEN:runSuppOrth:supportLv`, `RAVEN:runSuppOrth:failed`, `RAVEN:runSuppOrth:notFound` and `RAVEN:hitsFromSuppOrth:unmappedProteins`. Identifiers in other namespaces include `parseRAVENargs:tooManyArgs`, `parseRAVENargs:noSpec`, `getPhylDist:noData`, `renameModelGenes:*`, `setGAM:*` and `parseYAML:*`. The list is not exhaustive. -- `RAVEN:badInput` appears in 220 lines and `RAVEN:warning` in 63 lines of `src/` and the root `.m` files (non-comment lines that contain the identifier). -- Further identifiers in the `RAVEN:` namespace are `RAVEN:modelError`, `RAVEN:badGrRule`, `RAVEN:infeasibleModel`, `RAVEN:infeasible`, `RAVEN:sampling`, `RAVEN:grRuleTooComplex`, `RAVEN:deprecated`, `RAVEN:legacyMethod`, `RAVEN:notMainBranch`, `RAVEN:glpkMILP`, `RAVEN:fetchToolboxAsset:unverified`, `RAVEN:fetchToolboxAsset:checksum`, `RAVEN:fetchToolboxAsset:manifest`, `RAVEN:runSuppOrth:supportLv`, `RAVEN:runSuppOrth:failed`, `RAVEN:runSuppOrth:notFound` and `RAVEN:hitsFromSuppOrth:unmappedProteins`. -- Other namespaces include `parseRAVENargs:tooManyArgs`, `parseRAVENargs:noSpec`, `getPhylDist:noData`, `renameModelGenes:*`, `setGAM:*` and `parseYAML:*`. The list is not exhaustive. -- Whether `try/catch` blocks that match on message text still match was not compared message by message. +Whether `try/catch` blocks that match on message text match was not compared message by message. ## Changes that raise warnings diff --git a/docs/raven3-migration/formats-and-reconstruction.md b/docs/raven3-migration/formats-and-reconstruction.md index c9865ef..bb67b92 100644 --- a/docs/raven3-migration/formats-and-reconstruction.md +++ b/docs/raven3-migration/formats-and-reconstruction.md @@ -14,14 +14,14 @@ Three areas differ between RAVEN 2 and RAVEN 3: the model file formats, the draf | `importModel` accepted input | Any SBML level and version. The notes-based gene association `GENE_ASSOCIATION` of Level 2 files is read, and `SHORT NAME` is read for gene names. FBC fields are optional | SBML Level 3 Version 1 with FBC package version 2. Any other file raises the error `RAVEN:badInput` | | MIRIAM prefixes read on import | `urn:miriam:`, `http://identifiers.org/` and `https://identifiers.org/` | `http://identifiers.org/` and `https://identifiers.org/` | -`importModel` of RAVEN 3 reads MIRIAM annotations of reaction EC codes, models and species with the prefixes `http://identifiers.org/` and `https://identifiers.org/`, and ignores `urn:miriam:` annotations. +`importModel` of RAVEN 3 reads MIRIAM annotations (reaction EC codes, model and species annotations) with the prefixes `http://identifiers.org/` and `https://identifiers.org/`, and ignores `urn:miriam:` annotations. :::{admonition} Cross-references in older SBML files :class: warning An SBML file with `urn:miriam:` annotations is imported without an error, and its cross-references (for example `metMiriams`) are absent from the imported model. The `exportModel` function of RAVEN 2 writes `https://identifiers.org/` prefixes, so files written by RAVEN 2 are not affected. Files from other tools that use `urn:miriam:` can be converted to the `identifiers.org` form before the import. ::: -A file in an older SBML level or version has to be converted before `importModel` accepts it. The error message names another tool or an older RAVEN version for the conversion. +A file in an older SBML level or version has to be converted before `importModel` accepts it. The error message suggests another tool or an older RAVEN version for the conversion. (excel-io)= ### Excel I/O @@ -50,7 +50,7 @@ A pipeline that depends on the RAVEN 2 Excel curation template has two options. | Metabolite SMILES | Top-level field | Under `annotation` as `smiles` | | Reaction notes key | `rxnNotes` | `notes` | | Gene-less model | `genes:` is omitted when `model.genes` is absent, and an empty field gives a bare `- genes:` | The line `- genes: []` | -| Document start | A `---` line, then `!!omap` | `!!omap` only | +| Document start | A `---` line, then `!!omap` | `!!omap` alone | | Field order, metabolites | `formula`, then `charge` | `charge`, then `formula` | | Field order, reactions | `rxnFrom` directly after `gene_reaction_rule`, and `references` before `subsystem` and `annotation` | `subsystem` and `annotation` first, then `references` and `rxnFrom` among the RAVEN-specific extras | | Field order, genes | `protein` before `annotation` | `protein` after `annotation` | @@ -77,7 +77,7 @@ The function `conversion/ravenCobraWrapper.m` keeps its signature, `newModel = r | Argument `keggPath` of `getModelFromKEGG` | Optional, default `external/kegg` | Removed | | Ortholog HMM library | One `.hmm` file per KEGG Orthology (KO) entry and one `hmmsearch` call per KO, through the Windows Subsystem for Linux (WSL) on Windows | One concatenated library per domain (`kegg118_eukaryotes.hmm` and `kegg118_prokaryotes.hmm`), one `hmmsearch` call against the whole proteome, and no WSL | | `dataDir` | Any local folder | For the homology method, which takes a FASTA file, the folder name must end in `kegg118_eukaryotes` or `kegg118_prokaryotes`. The annotation method ignores `dataDir` | -| `nSequences`, `seqIdentity` (CD-HIT clustering controls) | Configurable | Removed, because there is no local clustering step | +| `nSequences`, `seqIdentity` (CD-HIT clustering controls) | Configurable | Removed; there is no local clustering step | | Default of `cutOff` | `10^-50` | `10^-30` | | Default of `minScoreRatioG` | `0.8` | `0.9` | | Calling convention | Positional | Positional or name-value (`parseRAVENargs`). A call with more positional arguments than the function defines raises `parseRAVENargs:tooManyArgs` | @@ -124,9 +124,9 @@ RAVEN 3 parses gene-reaction rules into a tree. Five functions use the parser: ` | `diffModels` | Not present | Compares grRules. Two rules are equal when their DNF forms are equal after sorting of genes and clauses, without absorption: `A or (A and B)` differs from `A` | | `curateModelFromTables` | Not present | Parses the rules of the table input | -The function `addRxns` does not use the parser. It takes the new gene IDs of a grRule from `getGenesFromGrRules`, which splits at the space-bounded tokens `and`, `or`, `&` and `|`. RAVEN 2 split at the substrings "and" and "or" anywhere, so an ID such as `band1` was cut at "and". +`addRxns` does not use the parser. It takes the new gene IDs of a grRule from `getGenesFromGrRules`, which splits at the space-bounded tokens `and`, `or`, `&` and `|`. RAVEN 2 split at the substrings "and" and "or" anywhere, so an ID such as `band1` was cut at "and". -The parser functions are in `utils/`. The function `parseGrRule` builds a tree with `type` (`'gene'`, `'and'` or `'or'`) and `children`. The function `grRuleToDNF` converts the tree to DNF, an OR of AND clauses. The function `isDnfGrRule` checks whether a rule has that form, and `grRuleToString` renders a tree as a canonical string. +The parser functions are in `utils/`. `parseGrRule` builds a tree with `type` (`'gene'`, `'and'` or `'or'`) and `children`. `grRuleToDNF` converts the tree to DNF, an OR of AND clauses. `isDnfGrRule` checks whether a rule has that form, and `grRuleToString` renders a tree as a canonical string. `parseGrRule` accepts `and` and `or` in any case and the symbols `&`, `&&`, `|` and `||`. It recognises operators as whole words, so `G1&G2` is one gene ID. A malformed rule raises `RAVEN:badGrRule`. diff --git a/docs/raven3-migration/new-and-environment.md b/docs/raven3-migration/new-and-environment.md index fad25c3..86023db 100644 --- a/docs/raven3-migration/new-and-environment.md +++ b/docs/raven3-migration/new-and-environment.md @@ -1,6 +1,6 @@ # New functionality, dependencies and error handling -RAVEN 3 changes gap-filling and the extraction of context-specific models with `ftINIT`, adds functions, depends on fewer MATLAB toolboxes, and handles errors in another way. +RAVEN 3 changes gap-filling and the extraction of context-specific models with `ftINIT`, adds functions, depends on fewer MATLAB toolboxes, and handles errors in a different way. (ftinit)= ## Gap-filling and context-specific model extraction @@ -23,17 +23,17 @@ The option `algorithm` of `fillGaps` selects one of these algorithms. A value th - `'fastLP'` and `'swiftLP'` relax the problem to a linear program (LP) in the manner of FASTCORE and SWIFTCORE. - `'gapfillMILP'` is a MILP with a growth floor. Reactions that are already in the draft model keep their bounds, so a reaction with the wrong direction is not changed. -- `'topological'` is a pre-screen of producibility by breadth-first search. It does not modify the model, requires a reference model and uses only the first one. +- `'topological'` is a pre-screen of producibility by breadth-first search. It does not modify the model, requires a reference model and uses the first one. The function `fillGaps` calls `gapFillTopological` with the defaults of that function. The seeds are the metabolites of the exchange reactions that `getExchangeRxns` classes as `'uptake'` or `'reverse'`. The targets are the metabolites of the first objective reaction, and a model without an objective reaction raises `RAVEN:badInput`. ### Gap-filling in `fitTasks` -With `gapFillMode` set to `'preMerged'`, `fitTasks` expects a reference model that already contains the reactions of the model. No merge is then needed for each task, and `ftINITFillGaps` fills the gaps. How `fitTasks` treats a task that requires net production of a metabolite without reactions was not checked. +With `gapFillMode` set to `'preMerged'`, `fitTasks` expects a reference model that already contains the reactions of the model. No merge is needed for each task, and `ftINITFillGaps` fills the gaps. How `fitTasks` treats a task that requires net production of a metabolite without reactions was not checked. ### Tie-breaking in `ftINIT` -With `resolveTies`, each MILP step of `ftINIT` holds the score at its optimum and then chooses among the tied solutions in phases, each with a time limit. The phases select, in this order: +With `resolveTies`, each MILP step of `ftINIT` holds the score at its optimum and chooses among the tied solutions in phases, each with a time limit. The phases select, in this order: 1. the fewest kept removable reactions; 2. the lowest summed ID rank of those reactions, where a merged reaction is ranked by the smallest original ID of its group; @@ -85,7 +85,7 @@ The function documentation of `sampleCHRR` states that the polytope is rounded b | Aspect | RAVEN 2 | RAVEN 3 | |---|---|---| -| MATLAB version | No minimum stated. `readYAMLmodel` has a fallback for releases before R2020b (`readlines`) | No minimum stated, and the code has no check that rejects an older release. `readYAMLmodel` keeps the fallback for releases before R2020b, `parseTaskList` calls `readcell` (R2019a) for Excel task lists, and `addIdentifierPrefix` calls `regexpPattern` (R2020b). Continuous integration uses R2024b only, and older releases were not tested | +| MATLAB version | No minimum stated. `readYAMLmodel` has a fallback for releases before R2020b (`readlines`) | No minimum stated, and the code has no check that rejects an older release. `readYAMLmodel` keeps the fallback for releases before R2020b, `parseTaskList` calls `readcell` (R2019a) for Excel task lists, and `addIdentifierPrefix` calls `regexpPattern` (R2020b). Continuous integration uses a single release, R2024b, and older releases were not tested | | Statistics and Machine Learning Toolbox | Required by `compareMultipleModels` (`mdscale`, `pdist`, `squareform` and `tsne`, with an error without them) and by `compareRxnsGenesMetsComps` (`combnk`) | Optional in `compareMultipleModels`. Without it the projection and the clustering of the heatmap are skipped | | Bioinformatics Toolbox | Required by `getKEGGModelForOrganism` when it builds missing per-KO alignments from a local KEGG dump (`fastaread` and `fastawrite`) | Not required (`io/readFasta.m` and `io/writeFasta.m`) | | Optimization Toolbox | Required for `qMOMA` and `solveQP` (`quadprog`) | Not required, because both functions are removed | @@ -96,7 +96,7 @@ The function documentation of `sampleCHRR` states that the polytope is rounded b | KEGG HMM libraries and reference data | Manual download or local build | Fetched on first use and cached | | Installation self-check | `installation/checkInstallation.m` runs `importExportTests`, `solverTests` and the test files of BLAST+, DIAMOND, HMMER, CD-HIT and MAFFT to fill its table of passed and failed tests | `checkRaven` in the repository root imports `empty.xml`, exports SBML, imports and exports YAML, exports Excel, solves a two-reaction LP per solver and downloads missing binaries without a prompt. `checkInstallation` forwards to it (see [Renamed and merged functions](#renamed-merged-and-consolidated-functions)) | -KEGG data, binaries and the checksum manifest come from releases of `toolbox-assets`. Other downloads have other sources. +KEGG data, binaries and the checksum manifest come from releases of `toolbox-assets`. Other downloads have other sources. A RAVEN 3 clone needs network access for these downloads. - KEGG data comes from the release tag that `keggDataVersion()` returns (`src/reconstruction/kegg/KEGG_VERSION.md`, tag `kegg118`). - The binaries come from the fixed tags in `src/installation/downloadRavenBinaries.m`: `blast-2.17.0`, `diamond-2.1.17`, `hmmer-3.4.0` and, on Windows, `hmmer-3.3.2`. @@ -104,8 +104,6 @@ KEGG data, binaries and the checksum manifest come from releases of `toolbox-ass - `setRavenSolver('scip')` on Windows downloads `scip_mex_win.zip` from the RAVEN v2.8.6 release on GitHub into `src/external/scip`. The repository holds no SCIP files, and other platforms need a SCIP installation made by hand. - `getUniProtScores` queries rest.uniprot.org, `downloadGenomeData` downloads from NCBI, and `checkRaven` reads the version file from GitHub. -A RAVEN 3 clone needs network access for these downloads. - --- ## Error and warning handling diff --git a/docs/raven3-migration/structure-and-renames.md b/docs/raven3-migration/structure-and-renames.md index fd9be2a..446304a 100644 --- a/docs/raven3-migration/structure-and-renames.md +++ b/docs/raven3-migration/structure-and-renames.md @@ -32,31 +32,31 @@ The code of RAVEN 3 is in `src/`. RAVEN 2 kept its functions in the top-level fo The files `checkRaven.m` and `checkInstallation.m` (a wrapper) are in the repository root. -Some code has no successor folder: the subfolder `external/metacyc/` with the MetaCyc function `external/combineMetaCycKEGGModels.m`, `pathway/` and `plotting/` (visualisation), `legacy/` (CellDesigner import and other code from before RAVEN 2.0), and the Excel-POI helpers of `io/` together with the Apache POI libraries in `software/apache-poi`. +Some code has no successor folder: the subfolder `external/metacyc/` with the MetaCyc function `external/combineMetaCycKEGGModels.m`, `pathway/` and `plotting/` (visualisation), `legacy/` (CellDesigner import and other pre-2.0 code), and the Excel-POI helpers of `io/` together with the Apache POI libraries in `software/apache-poi`. -A script that calls RAVEN functions by name keeps working when the whole RAVEN folder tree, subfolders included, is on the MATLAB path (through `addRavenToUserPath` or `addpath(genpath(...))`), because MATLAB resolves functions by name across the path. The reorganisation affects a script in three cases: it uses a hard-coded path to a specific `.m` file, it calls a removed function or a merged function that has no wrapper, or it contains a function with the name of a RAVEN 3 function. `checkFunctionUniqueness` warns about the last case. +A script that calls RAVEN functions by name keeps working when the whole RAVEN folder tree, subfolders included, is on the MATLAB path (through `addRavenToUserPath` or `addpath(genpath(...))`), because MATLAB resolves functions by name across the path. The reorganisation affects a script in three cases: it uses a hard-coded path to a specific `.m` file, it calls a removed function or a merged function that has no wrapper, or it contains a function with the name of a RAVEN 3 function. `checkFunctionUniqueness` warns about the last case. Individual moved files are not mapped. --- (removed-functionality)= ## Removed functionality -The table lists all 65 functions of RAVEN 2 that RAVEN 3 removes without a successor of the same name. The counts per RAVEN 2 folder are `pathway/` 6, `plotting/` 10, `external/` 12 (8 for MetaCyc, 4 others), `legacy/` 11, `io/` 12, `core/` 8, `utils/` 3, `solver/` 2 and `INIT/` 1. No function of RAVEN 3 calls a removed function. The name `getToolboxVersion` survives only as a local function of `exportForGit`. +The table lists all 65 functions of RAVEN 2 that RAVEN 3 removes without a successor of the same name. The counts per RAVEN 2 folder are `pathway/` 6, `plotting/` 10, `external/` 12 (8 for MetaCyc, 4 others), `legacy/` 11, `io/` 12, `core/` 8, `utils/` 3, `solver/` 2 and `INIT/` 1. No function of RAVEN 3 calls a removed function. The name `getToolboxVersion` exists in RAVEN 3 as a local function of `exportForGit`. | Removed | Former purpose | Replacement | |---|---|---| | `pathway/colorPathway`, `pathway/getPathwayDimensions`, `pathway/mapPathwayRxnNames`, `pathway/markPathwayWithExpression`, `pathway/markPathwayWithFluxes`, `pathway/trimPathway` (6 functions) | Handling and colouring of metabolic pathway maps | None in RAVEN 3. Alternatives are the COBRA Toolbox, Escher, and the export of flux and omics data for plotting elsewhere | | `plotting/colorSubsystem`, `plotting/drawMap`, `plotting/drawPathway`, `plotting/getColorCodes`, `plotting/getObjectiveString`, `plotting/plotAdditionalInfo`, `plotting/plotLabels`, `plotting/setColorToMapRxns`, `plotting/setOmicDataToRxns`, `plotting/setTitle` (10 functions) | Drawing of metabolic maps and colouring with omics data | None, as above | | `external/metacyc/{getModelFromMetaCyc,getRxnsFromMetaCyc,getMetsFromMetaCyc,getEnzymesFromMetaCyc,getMetaCycModelForOrganism,addSpontaneousRxns,linkMetaCycKEGGRxns}`, `external/combineMetaCycKEGGModels` (8 functions) | Draft reconstruction from MetaCyc | None. The reconstruction methods of RAVEN 3 are the KEGG-based and the homology-based method | -| `legacy/core/changeGeneAssoc`, `legacy/external/constructPathwayFromCelldesigner`, and the nine functions of the bundled `xml_toolbox` (`xml_format`, `xml_formatany`, `xml_help`, `xml_load`, `xml_parse`, `xml_parseany`, `xml_save`, `xml_strsplit`, `xml_unblank`) (11 functions) | CellDesigner import and compatibility code from before RAVEN 2.0 | None | +| `legacy/core/changeGeneAssoc`, `legacy/external/constructPathwayFromCelldesigner`, and the nine functions of the bundled `xml_toolbox` (`xml_format`, `xml_formatany`, `xml_help`, `xml_load`, `xml_parse`, `xml_parseany`, `xml_save`, `xml_strsplit`, `xml_unblank`) (11 functions) | CellDesigner import and pre-2.0 compatibility code | None | | `io/importExcelModel`, `io/loadWorkbook`, `io/loadSheet`, `io/writeSheet`, `io/SBMLFromExcel`, `io/addJavaPaths`, `io/startup` (7 functions) | Excel model I/O through Apache POI | `io/writeExcel.m` (OOXML writer without dependencies) for export and `curation/curateModelFromTables.m` for tabular curation. RAVEN 3 cannot import a full Excel template (see [Excel I/O](#excel-io)) | -| `io/exportModelToSIF`, `io/exportToTabDelimited` | Export of a SIF graph; tab-delimited export of a model | None. The only caller of `exportToTabDelimited` was the fallback of `exportToExcelFormat` for a path without file name, which is removed | +| `io/exportModelToSIF`, `io/exportToTabDelimited` | Export of a SIF graph; tab-delimited export of a model | None. The sole caller of `exportToTabDelimited` was the fallback of `exportToExcelFormat` for a path without file name, which is removed | | `io/getFullPath`, `io/getMD5Hash`, `io/getToolboxVersion` | Canonical file path; MD5 hash of a file; git version of the toolbox | Inlined where needed: the MD5 hash through Java `MessageDigest` in `getBlast`, and the version as a local function of `exportForGit` | | `core/dispEM` | Display of errors and warnings (388 lines call it in RAVEN 2, counted outside `doc/` and `testing/`) | `warning()` and `error()`, and `utils/ravenList.m` for formatted lists of items (see [Backward-incompatible changes](#backward-incompatible-changes)) | | `core/followFluxes`, `core/followChanged` | Report of flux changes relative to a reference solution | `analysis/compareFluxes.m` compares two flux vectors of one model (reactions turned on or off, flipped and changed, with `cutoff` and `metaboliteList`) and replaces `core/followChanged`. `queries/printFluxes.m` prints the fluxes of one vector above a cut-off | | `core/printModel` | Printing of reactions to the screen or to a file | `queries/printModelStats.m` and `queries/printFluxes.m` cover model summaries and the printing of fluxes (see below the table) | | `core/getMetsInComp` | Indices of the metabolites in a compartment | `model.metComps == compIndex` | -| `core/mapCompartments` | Merging and splitting of the score columns of a localisation-score structure by compartment rules | The `compartmentMap` option of `parseScores` and `getUniProtScores`. The default map `defaultCompartmentMap` applies to `getUniProtScores` and to `parseScores` with the `'uniprot'` predictor. Without a map, every label stays unchanged. With a map, a label that the map does not contain is dropped together with its score column. Merged columns take the maximum score, and splitting is not supported | +| `core/mapCompartments` | Merging and splitting of the score columns of a localisation-score structure by compartment rules | The `compartmentMap` option of `parseScores` and `getUniProtScores`. The default map `defaultCompartmentMap` applies to `getUniProtScores` and to `parseScores` with the `'uniprot'` predictor. Elsewhere, without a map, every label stays unchanged. With a map, a label that the map does not contain is dropped together with its score column. Merged columns take the maximum score, and splitting is not supported | | `core/getExpressionStructure` | Loading of an expression structure from an ad hoc Excel format | `INIT/parseHPA.m` and `INIT/parseHPArna.m` read Human Protein Atlas files. Expression tables from other sources have to be assembled by hand as the `arrayData` structure documented in `scoreModel` | | `core/checkRxn` | Debugging of the synthesisability of one reaction | No direct successor. `gapfilling/findLeakMetabolite.m`, `gapfilling/canExchange.m` and `gapfilling/checkProduction.m` cover related checks | | `external/getWoLFScores` | Parsing of WoLF PSORT scores | None. `parseScores` with `'wolf'` raises `RAVEN:badInput` (see [Backward-incompatible changes](#backward-incompatible-changes)) | @@ -76,13 +76,13 @@ The folder `deprecated/` holds `makeSomething`, `consumeSomething`, `canProduce` | RAVEN 2 functions | RAVEN 3 function | Notes | |---|---|---| -| `makeSomething(model, ignoreMets, isNames, minNrFluxes, allowExcretion, params, ignoreIntBounds)` | `findLeakMetabolite(model, 'produce', ...)` | The arguments map one to one, and the wrapper forwards `varargin` unchanged | +| `makeSomething(model, ignoreMets, isNames, minNrFluxes, allowExcretion, params, ignoreIntBounds)` | `findLeakMetabolite(model, 'produce', ...)` | The argument order maps one to one, and the wrapper forwards `varargin` unchanged | | `consumeSomething(model, ignoreMets, isNames, minNrFluxes, params, ignoreIntBounds)` | `findLeakMetabolite(model, 'consume', ...)` | The RAVEN 2 order has no position for `allowExcretion`, so the wrapper remaps the arguments by name to keep `params` and `ignoreIntBounds` | | `canProduce(model, mets)` | `canExchange(model, 'produce', 'mets', mets)` | None | | `canConsume(model, mets)` | `canExchange(model, 'consume', 'mets', mets)` | None | | `compareRxnsGenesMetsComps` | `comparison/compareMultipleModels.m` | The overlap output of RAVEN 2 (reactions, genes, metabolites, EC codes, metabolite names, equations and unique equations) is part of the output of `compareMultipleModels`. No wrapper | | `INIT/scoreComplexModel.m` | `INIT/scoreModel.m` | Merged, no wrapper. A call moves to `scoreModel` with the same positions and defaults for the first six optional arguments, and `dataPrecedence` is added as the 7th optional argument | -| `INIT/ftINITFillGapsMILP.m` | `solver/getMinNrFluxes.m` | Available through the option `resolveTies` of `getMinNrFluxes`, which exists for `formulation` `'reversible'` only. The default `'irrev'` raises an error with `resolveTies`. The option `proveAbsGap` belongs to `ftINIT` | +| `INIT/ftINITFillGapsMILP.m` | `solver/getMinNrFluxes.m` | Available through the option `resolveTies` of `getMinNrFluxes`, which exists for `formulation` `'reversible'`. The default `'irrev'` raises an error with `resolveTies`. The option `proveAbsGap` belongs to `ftINIT` | | `INIT/ftINITFillGapsForAllTasks.m` | `gapfilling/ftINITFillGaps.m` and the gap-fill mode `'preMerged'` of `fitTasks` | None. `ftINITFillGaps` exists in RAVEN 2 as well | | `getGenesFromKEGG`, `getMetsFromKEGG` and `getRxnsFromKEGG` | `reconstruction/kegg/readKEGGTable.m` (with `buildGlobalKEGGModel` and `buildGlobalGPR`) and the KEGG artefacts of `toolbox-assets` | The data source differs, so the signatures are not compatible (see [Reconstruction: KEGG and homology pipelines](#kegg-reconstruction)) | | `core/parallelPoolRAVEN.m` | `utils/parallelWorkersRAVEN.m` | `parallelWorkersRAVEN` returns a worker count for `parfor` (`0` or `Inf`) and starts no pool. `parallelPoolRAVEN` returns `[ps, oldPoolAutoCreate]` | @@ -92,13 +92,13 @@ The folder `deprecated/` holds `makeSomething`, `consumeSomething`, `canProduce` RAVEN 2 has `INIT/scoreComplexModel.m` and `hpa/scoreModel.m`. RAVEN 3 has one `scoreModel`, which tINIT and ftINIT call with different settings. A call written for `hpa/scoreModel.m` differs in these points (checked against 2.11.3): -- RAVEN 2 scored a reaction by the best or the average score over all its genes (`multipleGeneScoring`, values `'best'` and `'average'`). RAVEN 3 applies `isozymeScoring` (default `'max'`) to the `or` parts and `complexScoring` (default `'min'`) to the `and` parts of the rule. A reaction with an `and` rule therefore gets another default score. +- RAVEN 2 scored a reaction by the best or the average score over all its genes (`multipleGeneScoring`, values `'best'` and `'average'`). RAVEN 3 applies `isozymeScoring` (default `'max'`) to the `or` parts and `complexScoring` (default `'min'`) to the `and` parts of the rule. A reaction with an `and` rule therefore gets a different default score. - Positions count all arguments of the call. The 7th to 9th arguments of a RAVEN 2 call (`multipleGeneScoring`, `multipleCellScoring`, `hpaLevelScores`) are read as `isozymeScoring`, `complexScoring` and `multipleCellScoring`. `hpaLevelScores` is the 10th argument. - `multipleCellScoring` accepts `'max'` and `'average'`. RAVEN 2 accepted `'best'` and `'average'`. - `dataPrecedence` is an argument of RAVEN 3 with the default `'gene'`. - A gene that is absent from the data receives `NaN` in `geneScores`, and the RAVEN 2 documentation states `-Inf`. `getINITModel` converts `NaN` back to `-Inf`. -[Backward-incompatible changes](#backward-incompatible-changes) lists the same differences as table rows. The function `getINITModel` passes `'max'` for `isozymeScoring` and for `complexScoring`, and `'reaction'` for `dataPrecedence`. Whether these settings reproduce the scores of the RAVEN 2 `hpa/scoreModel` was not checked, and the RAVEN 2 value `'average'` of `multipleGeneScoring` has no equivalent among them. +[Backward-incompatible changes](#backward-incompatible-changes) lists these differences as table rows. The function `getINITModel` passes `'max'` for `isozymeScoring` and for `complexScoring`, and `'reaction'` for `dataPrecedence`. Whether these settings reproduce the scores of the RAVEN 2 `hpa/scoreModel` was not checked, and the RAVEN 2 value `'average'` of `multipleGeneScoring` has no equivalent among them. ### Legacy tINIT functions diff --git a/docs/raven3-vs-raven-toolbox.md b/docs/raven3-vs-raven-toolbox.md index 579955f..cc0b210 100644 --- a/docs/raven3-vs-raven-toolbox.md +++ b/docs/raven3-vs-raven-toolbox.md @@ -5,11 +5,11 @@ RAVEN exists in two implementations: the MATLAB toolbox RAVEN 3 and the Python p - RAVEN runs in MATLAB without the COBRA Toolbox. - raven-toolbox is built on [cobrapy](https://cobrapy.readthedocs.io/), and a model is a `cobra.Model`. -The implementations differ in function names, in functions that only one of them has, in the form of the results and in the results of some functions. [Features specific to each implementation](raven3-vs-raven-toolbox/what-each-has.md) covers the functions that exist in one implementation only. [Differences in results between functions](raven3-vs-raven-toolbox/same-function-different-answer.md) covers functions that exist in both and return different results. +The implementations differ in function names, in functions that a single one of them has, in the form of the results and in the results of some functions. [Features specific to each implementation](raven3-vs-raven-toolbox/what-each-has.md) covers the functions that exist in a single implementation. [Differences in results between functions](raven3-vs-raven-toolbox/same-function-different-answer.md) covers functions that exist in both and return different results. ## Choosing an implementation -The conditions below favour one implementation. Both implementations provide homology and KEGG reconstruction, metabolic tasks, gap-filling, ftINIT, compartment assignment and model comparison. +Each condition below favours the implementation named next to it. Both implementations provide homology and KEGG reconstruction, metabolic tasks, gap-filling, ftINIT, compartment assignment and model comparison. | Condition | Implementation | |---|---| @@ -49,7 +49,7 @@ The two implementations set the solver differently. | Aspect | RAVEN | raven-toolbox | |---|---|---| | Configuration | `setRavenSolver('gurobi')` | `cobra.Configuration().solver = 'gurobi'` | -| Solvers | GLPK, Gurobi, SCIP and the COBRA Toolbox solvers. `setRavenSolver` downloads the SCIP binary for Windows only | Any solver that cobrapy supports through optlang | +| Solvers | GLPK, Gurobi, SCIP and the COBRA Toolbox solvers. `setRavenSolver` downloads the SCIP binary for Windows alone | Any solver that cobrapy supports through optlang | | Free licence | Gurobi (academic) | Gurobi (academic) | | Bundled open-source solver | GLPK, bundled with RAVEN | GLPK, installed with cobrapy | diff --git a/docs/raven3-vs-raven-toolbox/same-function-different-answer.md b/docs/raven3-vs-raven-toolbox/same-function-different-answer.md index 9912ab2..818bcb5 100644 --- a/docs/raven3-vs-raven-toolbox/same-function-different-answer.md +++ b/docs/raven3-vs-raven-toolbox/same-function-different-answer.md @@ -2,15 +2,15 @@ A function of RAVEN and its counterpart in raven-toolbox can differ in the algorithm they choose, in the form of the result or in the value of the result. The sections below describe the differences found by reading the code of both implementations. Where a section says that a result was not checked, no run compared the two implementations. -The list is incomplete, and a function that is not listed can behave differently. Three candidates were not compared: `random_sampling`, which replaces both `sampleACHR` and `sampleCHRR`; `remove_duplicate_reactions`, which replaces `contractModel`; and `simplify_model`, which corresponds to `simplifyModel`. +The list is incomplete, and a function that is not listed can behave differently. Candidates that were not compared include `random_sampling`, which replaces both `sampleACHR` and `sampleCHRR`; `remove_duplicate_reactions`, which replaces `contractModel`; and `simplify_model`, which corresponds to `simplifyModel`. ## Metabolic task checking The functions `checkTasks` and `check_tasks` implement the same task format and the same boundary semantics. The option `close_boundaries=True` of `check_tasks` corresponds to calling `checkTasks` on `closeModel(model)`. [Metabolic tasks](../guide/tasks.md) describes the options. Whether both functions return identical pass and fail results on a shared input was not checked. -The function `checkTasks` starts each task from the original model structure. The function `check_tasks` applies each task to one model inside a `with model:` block and reverts the changes afterwards. It also saves and restores direct edits of metabolite-constraint bounds. The runtime of the two functions was not compared. +`checkTasks` starts each task from the original model structure. `check_tasks` applies each task to one model inside a `with model:` block and reverts the changes afterwards. It also saves and restores direct edits of metabolite-constraint bounds. The runtime of the two functions was not compared. -The option `getEssential` of `checkTasks` has no counterpart option in `check_tasks`. In RAVEN, `checkTasks` calls `getEssentialRxns` on the model constrained by each task, and `find_task_essential_reactions` ports this step. It therefore corresponds to `checkTasks` with `getEssential`, and not to `getEssentialRxns` called on a model without a task. +The option `getEssential` of `checkTasks` has no counterpart option in `check_tasks`. In RAVEN, `checkTasks` calls `getEssentialRxns` on the model constrained by each task, and `find_task_essential_reactions` ports this step. It is the port of `checkTasks` with `getEssential`. ## Gap-filling functions @@ -26,14 +26,14 @@ The function `fillGaps` of RAVEN selects one of five algorithms with the option The functions differ in three more points. - `fillGaps` with `useModelConstraints=true` fills so that the constraints of the model hold. `connect_blocked_reactions` has no such mode, and its documentation points to `cobra.flux_analysis.gapfill` for that use. The closest function for a growth target is `fill_gaps_milp`. -- `fillGaps` takes a cell array of template models, but the RAVEN functions `gapFillFastLP`, `gapFillMILP` and `gapFillTopological` use only the first template. The four raven-toolbox functions accept several templates. +- `fillGaps` takes a cell array of template models, but the RAVEN functions `gapFillFastLP`, `gapFillMILP` and `gapFillTopological` use the first template and ignore the rest. The four raven-toolbox functions accept several templates. - `fillGaps` returns `newConnected`, `cannotConnect`, `addedRxns`, `newModel` and `exitFlag`. raven-toolbox returns a `GapFillResult` from `connect_blocked_reactions`, a `FastLPResult` from `fill_gaps_fast_lp`, a `MILPGapFillResult` from `fill_gaps_milp` and a `TopologicalAnalysisResult` from `analyse_topology`. The count `newConnected` is not stable across runs (see [Gap-filling](../guide/gap-filling.md)). The RAVEN functions `canExchange`, `checkProduction` and `getAllSubGraphs` find gaps without filling them. raven-toolbox merges them into `gap_report`, the port of `gapReport`, and has no separate public function for each of them. ## Alternative optima of mixed-integer problems -The extraction with ftINIT, the reference and MILP gap-filling algorithms, and the assignment of compartments (`assignCompartments` and `assign_compartments`) solve mixed-integer problems, which can have several optima with the same objective value. The LP algorithms were not measured for this, and the documentation of `swiftLP` in both implementations calls it stochastic. A solve that stops at its gap or time limit can also end at another objective value. Two runs can therefore return different reaction sets, and no single exact result is expected. +The extraction with ftINIT, the reference and MILP gap-filling algorithms, and the assignment of compartments (`assignCompartments` and `assign_compartments`) solve mixed-integer problems, which can have several optima with the same objective value. The LP algorithms were not measured for this, and the documentation of `swiftLP` in both implementations calls it stochastic. A solve that stops at its gap or time limit can also end at a different objective value. Two runs can therefore return different reaction sets, and no single exact result is expected. Four measurements quantify the differences. @@ -42,11 +42,11 @@ Four measurements quantify the differences. - **Between repeated runs.** Two repeated raven-toolbox builds at one seed with the default settings differed by 2 reactions. In each build one run of the final step stopped at its 5 s wall-clock limit, which is the presumed and untested cause (see [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](../parameter-tuning/studies/ftinit-determinism.md)). The effects of seed, thread count and time limit were not measured for the `ftINIT` of RAVEN. - **Between thread counts.** In the first ftINIT step of raven-toolbox without `resolve_ties`, with six seeds, the solutions at `Threads` 1 and 4 for the same seed were 5 to 20 reactions apart. In 5 of the 6 seeds the two runs reached different objective values (see [ftINIT reproducibility with `resolve_ties` and `prove_abs_gap`](../parameter-tuning/studies/ftinit-determinism.md)). -The overlap of the reaction sets, for example the Jaccard index, is a measure for comparing such outputs. +Compare such outputs by the overlap of the reaction sets, for example the Jaccard index. -The option `resolveTies` of the `ftINIT` of RAVEN and `resolve_ties` of the `ftinit` of raven-toolbox are off by default in both. When on, each step keeps the score optimum and picks among the tied solutions in phases: the fewest kept removable reactions, then the lowest summed ID rank of those, then the least total absolute flux, then the least backward flux weighted by ID rank. The last step of `ftINIT` skips the two flux phases. A step that stops at its time limit keeps the answer of the solver. +The option `resolveTies` of the `ftINIT` of RAVEN and `resolve_ties` of the `ftinit` of raven-toolbox are off by default in both. When on, each step keeps the score optimum and picks among the tied solutions in four phases, in this order: the fewest kept removable reactions, the lowest summed ID rank of those, the least total absolute flux and the least backward flux weighted by ID rank. The last step of `ftINIT` skips the two flux phases. A step that stops at its time limit keeps the answer of the solver. -In raven-toolbox, runs with `resolve_ties` at the same score optimum returned identical solutions across seeds. The primary solve can still end at another objective value for another seed, within the gap and time limits, so the final reaction sets can differ. This was not measured for RAVEN. Without `resolveTies` or `resolve_ties`, the ftINIT MILP can return any of the tied optima. +In raven-toolbox, runs with `resolve_ties` at the same score optimum returned identical solutions across seeds. The primary solve can end at a different objective value for a different seed, within the gap and time limits, so the final reaction sets can differ. This was not measured for RAVEN. Without `resolveTies` or `resolve_ties`, the ftINIT MILP can return any of the tied optima. ## Elemental balance results @@ -58,9 +58,9 @@ The two parsers handle some formulas differently, so the results differ for some - A polymer formula such as `(C5H8)n` is counted with n equal to 1 in RAVEN. raven-toolbox reports `unknown`. - RAVEN uses the field `inchis` when `metFormulas` is absent. raven-toolbox does not. -- RAVEN reports a charge balance (`chargeStatus` and `chargeResidual`). `get_elemental_balance` reports elements only. +- RAVEN reports a charge balance (`chargeStatus` and `chargeResidual`). `get_elemental_balance` reports elements. -The comparison used the code only, and the functions were not run on a shared model. +The comparison used the code, and the functions were not run on a shared model. ## Parsing of gene-reaction rules diff --git a/docs/raven3-vs-raven-toolbox/what-each-has.md b/docs/raven3-vs-raven-toolbox/what-each-has.md index fe62b3f..3368c56 100644 --- a/docs/raven3-vs-raven-toolbox/what-each-has.md +++ b/docs/raven3-vs-raven-toolbox/what-each-has.md @@ -1,6 +1,6 @@ # Features specific to each implementation -Some features exist in raven-toolbox only, and some exist in RAVEN only. The sections below describe both groups. +Some features belong to raven-toolbox alone, and some belong to RAVEN alone. The sections below describe both groups. ## Features specific to raven-toolbox @@ -22,7 +22,7 @@ Both implementations fetch BLAST+, DIAMOND, HMMER and the KEGG tables on demand The checksum handling differs. -- raven-toolbox keeps the checksum and the release of each dataset and binary in a registry in its source (a version-pinned registry). A fresh download is checked against its checksum. A cached data file is reused without a check, unless `ensure_data_file` is called with `verify=True`, which checks it again and downloads it again on a mismatch. A cached binary is reused without a check. A JSON manifest can be merged into the registries with `raven_toolbox.manifest.load_into_registries`, which reads the path or URL in the environment variable `RAVEN_PYTHON_MANIFEST` when no source is passed. The resolvers load that variable by themselves only while the registry is empty, so the call is needed with the filled default registries. +- raven-toolbox keeps the checksum and the release of each dataset and binary in a registry in its source (a version-pinned registry). A fresh download is checked against its checksum. A cached data file is reused without a check, unless `ensure_data_file` is called with `verify=True`, which checks it and downloads it anew on a mismatch. A cached binary is reused without a check. A JSON manifest can be merged into the registries with `raven_toolbox.manifest.load_into_registries`, which reads the path or URL in the environment variable `RAVEN_PYTHON_MANIFEST` when no source is passed. The resolvers load that variable by themselves while the registry is empty, so the call is needed with the filled default registries. - RAVEN checks a download against the checksum in the published manifest. A mismatch deletes the file and raises an error. When the manifest cannot be read or has no entry for the asset, RAVEN issues a warning and keeps the file unchecked. A cached file is reused without a check. ### Further functions @@ -33,7 +33,7 @@ Further features of raven-toolbox have no RAVEN counterpart in the generated map ### tINIT -The functions `getINITModel` and `runINIT` are the original tINIT implementation, and RAVEN keeps them for models built with them. raven-toolbox implements ftINIT only, as a design decision. A tINIT model has to be reproduced in MATLAB (see [Context-specific models with ftINIT](../guide/init.md)). +The functions `getINITModel` and `runINIT` are the original tINIT implementation, and RAVEN keeps them for models built with them. raven-toolbox implements ftINIT as its sole INIT variant, as a design decision. A tINIT model has to be reproduced in MATLAB (see [Context-specific models with ftINIT](../guide/init.md)). ### COBRA Toolbox conversion From 64935ea9f39f592365a5aa178521da853ea916e3 Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Sun, 1 Nov 2026 12:22:38 +0100 Subject: [PATCH 20/20] Name tINIT as the legacy method that ftINIT replaces --- docs/guide/init.md | 10 +++++----- docs/installation/index.md | 2 +- docs/raven3-migration/backward-incompatible.md | 2 +- docs/raven3-migration/structure-and-renames.md | 6 +++--- docs/raven3-vs-raven-toolbox.md | 2 +- docs/raven3-vs-raven-toolbox/what-each-has.md | 4 ++-- docs/references.md | 4 ++-- docs/tuned-parameters.md | 2 +- 8 files changed, 16 insertions(+), 16 deletions(-) diff --git a/docs/guide/init.md b/docs/guide/init.md index 7a83eb5..5954ebb 100644 --- a/docs/guide/init.md +++ b/docs/guide/init.md @@ -1,16 +1,16 @@ # Context-specific models with ftINIT -A genome-scale model describes what an organism can do. tINIT and ftINIT reduce it to what a particular sample (a tissue, a cell line, a condition) appears to be doing, from expression data plus a list of metabolic tasks that the result must be able to perform. +A genome-scale model describes what an organism can do. ftINIT reduces it to what a particular sample (a tissue, a cell line, a condition) appears to be doing, from expression data plus a list of metabolic tasks that the result must be able to perform. The legacy method tINIT, which ftINIT replaces, does the same for models that were built with it. The workflow uses Human-GEM and the RNA-seq data that Human-GEM provides, in both toolboxes. ## Choice of method -ftINIT is the recommended method, and the workflow here uses it. +ftINIT is the recommended method and replaces the legacy method tINIT. The workflow uses ftINIT. -Use tINIT (`getINITModel`) only to reproduce a model that was built with it. tINIT is supported and is not scheduled for removal. It issues a `RAVEN:legacyMethod` notice, which states that it is not intended for new work. Silence the notice with `warning('off','RAVEN:legacyMethod')`. +Use tINIT (`getINITModel`) to reproduce a model that was built with it. tINIT is supported and is not scheduled for removal. It issues a `RAVEN:legacyMethod` notice, which states that it is not intended for new work. Silence the notice with `warning('off','RAVEN:legacyMethod')`. -tINIT is MATLAB-only. RAVEN includes it for the models already built with it; raven-toolbox, an implementation without such an installed base, includes ftINIT only. A tINIT model has to be reproduced in MATLAB. +The legacy method tINIT is MATLAB-only. RAVEN includes it for the models already built with it; raven-toolbox, an implementation without such an installed base, includes ftINIT only. A tINIT model has to be reproduced in MATLAB. ## Relation between tINIT and ftINIT @@ -44,7 +44,7 @@ The same MATLAB workflow, as the model's own documentation presents it and inclu | `ftINIT` | `ftinit` | the staged extraction | | `scoreModel` | `score_reactions_from_genes` | gene scores → reaction scores | | no equivalent | `gene_scores_from_expression` | expression → gene scores | -| `runINIT`, `getINITModel` | no equivalent | tINIT, for reproducing models built with it | +| `runINIT`, `getINITModel` | no equivalent | the legacy tINIT method, for reproducing models built with it | | `removeLowScoreGenes` | `remove_low_score_genes` | prune negative-scoring genes from gene-protein-reaction rules (GPRs) | | `parseHPA`, `parseHPArna` | `parse_hpa`, `parse_hpa_rna` | read Human Protein Atlas (HPA) dumps | | no equivalent | `hpa_gene_scores`, `rna_gene_scores` | HPA levels or transcripts per million (TPM) to gene scores | diff --git a/docs/installation/index.md b/docs/installation/index.md index 351218a..b887ea1 100644 --- a/docs/installation/index.md +++ b/docs/installation/index.md @@ -26,7 +26,7 @@ Both versions need a linear-programming solver. The options are the same for eit | Solver | Licence | Suited to | |---|---|---| -| Gurobi | Free academic | Genome-scale models, tINIT and ftINIT, gap-filling (recommended) | +| Gurobi | Free academic | Genome-scale models, ftINIT and the legacy tINIT method, gap-filling (recommended) | | GLPK | Open source | Small to medium models, getting started | | SCIP | Open source | MILP problems (MATLAB); `setRavenSolver('scip')` downloads the binary on Windows only | diff --git a/docs/raven3-migration/backward-incompatible.md b/docs/raven3-migration/backward-incompatible.md index 8dff731..0603d8a 100644 --- a/docs/raven3-migration/backward-incompatible.md +++ b/docs/raven3-migration/backward-incompatible.md @@ -78,7 +78,7 @@ Whether `try/catch` blocks that match on message text match was not compared mes ## Changes that raise warnings -- `getINITModel` and `runINIT` issue a one-time notice with the identifier `RAVEN:legacyMethod` that points to `ftINIT`. `warning('off','RAVEN:legacyMethod')` silences it. The help text of both functions states that the function remains supported, although the changelog of RAVEN 3 marks both as deprecated (see [Gap-filling and context-specific model extraction](#ftinit)). +- `getINITModel` and `runINIT`, the functions of the legacy tINIT method, issue a one-time notice with the identifier `RAVEN:legacyMethod` that points to `ftINIT`. `warning('off','RAVEN:legacyMethod')` silences it. The help text of both functions states that the function remains supported, although the changelog of RAVEN 3 marks both as deprecated (see [Gap-filling and context-specific model extraction](#ftinit)). - `makeSomething`, `consumeSomething`, `canProduce` and `canConsume` are wrappers in the folder `deprecated/` around `findLeakMetabolite` and `canExchange`. They issue a one-time `RAVEN:deprecated` warning that they will be removed in the next major release. - `checkInstallation` is `checkRaven` in RAVEN 3, moved from `installation/` to the repository root, with the same signature: `[currVer, installType] = checkRaven(developMode, checkBinaries)`. The wrapper `checkInstallation` forwards every argument and output to `checkRaven` and issues a `RAVEN:deprecated` warning that names `checkRaven` on every call. The wrapper will be removed in the next major release. diff --git a/docs/raven3-migration/structure-and-renames.md b/docs/raven3-migration/structure-and-renames.md index 446304a..8ac6570 100644 --- a/docs/raven3-migration/structure-and-renames.md +++ b/docs/raven3-migration/structure-and-renames.md @@ -8,7 +8,7 @@ The code of RAVEN 3 is in `src/`. RAVEN 2 kept its functions in the top-level fo | RAVEN 3 folder (under `src/`) | Contents | RAVEN 2 source | |---|---|---| -| `INIT/` | `ftINIT`, `INIT/tINIT/` with `getINITModel` and `runINIT`, and the parsing of HPA and expression data and the scoring of genes (`parseHPA`, `parseHPArna`, `scoreModel`) | `INIT/`, `hpa/` | +| `INIT/` | `ftINIT`, `INIT/tINIT/` with `getINITModel` and `runINIT` (the legacy tINIT method), and the parsing of HPA and expression data and the scoring of genes (`parseHPA`, `parseHPArna`, `scoreModel`) | `INIT/`, `hpa/` | | `analysis/` | Analyses based on flux balance analysis (FBA): flux variability analysis (FVA), sampling, FSEOF, OptKnock, robustness, production envelope, flux comparison and flux tracing | `core/` | | `annotation/` | MIRIAM editing, SBO term assignment, ΔG CSV I/O | `struct_conversion/` (and new functions) | | `biomass/` | Fitting and scaling of the biomass composition, growth-associated maintenance (GAM) | `core/` (and new functions) | @@ -90,7 +90,7 @@ The folder `deprecated/` holds `makeSomething`, `consumeSomething`, `canProduce` ### Gene scoring with `scoreModel` -RAVEN 2 has `INIT/scoreComplexModel.m` and `hpa/scoreModel.m`. RAVEN 3 has one `scoreModel`, which tINIT and ftINIT call with different settings. A call written for `hpa/scoreModel.m` differs in these points (checked against 2.11.3): +RAVEN 2 has `INIT/scoreComplexModel.m` and `hpa/scoreModel.m`. RAVEN 3 has one `scoreModel`, which the legacy method tINIT and its replacement ftINIT call with different settings. A call written for `hpa/scoreModel.m` differs in these points (checked against 2.11.3): - RAVEN 2 scored a reaction by the best or the average score over all its genes (`multipleGeneScoring`, values `'best'` and `'average'`). RAVEN 3 applies `isozymeScoring` (default `'max'`) to the `or` parts and `complexScoring` (default `'min'`) to the `and` parts of the rule. A reaction with an `and` rule therefore gets a different default score. - Positions count all arguments of the call. The 7th to 9th arguments of a RAVEN 2 call (`multipleGeneScoring`, `multipleCellScoring`, `hpaLevelScores`) are read as `isozymeScoring`, `complexScoring` and `multipleCellScoring`. `hpaLevelScores` is the 10th argument. @@ -102,4 +102,4 @@ RAVEN 2 has `INIT/scoreComplexModel.m` and `hpa/scoreModel.m`. RAVEN 3 has one ` ### Legacy tINIT functions -The functions `getINITModel` and `runINIT` (in `INIT/tINIT/`) are marked legacy and are not scheduled for removal. `utils/legacyMethodNotice.m` issues a `RAVEN:legacyMethod` warning once, which can be silenced, and points to `ftINIT`. The method `ftINIT` is meant for new work, and `getINITModel` and `runINIT` reproduce models that were built with them. +The method ftINIT replaces the legacy method tINIT. The functions `getINITModel` and `runINIT` (in `INIT/tINIT/`) implement tINIT, are marked legacy and are not scheduled for removal. `utils/legacyMethodNotice.m` issues a `RAVEN:legacyMethod` warning once, which can be silenced, and points to `ftINIT`. The method `ftINIT` is meant for new work, and `getINITModel` and `runINIT` reproduce models that were built with them. diff --git a/docs/raven3-vs-raven-toolbox.md b/docs/raven3-vs-raven-toolbox.md index cc0b210..9edefcb 100644 --- a/docs/raven3-vs-raven-toolbox.md +++ b/docs/raven3-vs-raven-toolbox.md @@ -17,7 +17,7 @@ Each condition below favours the implementation named next to it. Both implement | The model is used with cobrapy tools as a `cobra.Model` | raven-toolbox | | The KEGG artefacts are built from a specified KEGG release (requires a KEGG FTP subscription) | raven-toolbox | | The surrounding code is MATLAB | RAVEN | -| A model built with tINIT is reproduced | RAVEN | +| A model built with the legacy tINIT method, which ftINIT replaces, is reproduced | RAVEN | | Dynamic flux balance analysis (FBA) is run with `runDynamicFBA` | RAVEN | | A model is converted in memory to the COBRA Toolbox structure with `ravenCobraWrapper` | RAVEN | diff --git a/docs/raven3-vs-raven-toolbox/what-each-has.md b/docs/raven3-vs-raven-toolbox/what-each-has.md index 3368c56..2b027b2 100644 --- a/docs/raven3-vs-raven-toolbox/what-each-has.md +++ b/docs/raven3-vs-raven-toolbox/what-each-has.md @@ -31,9 +31,9 @@ Further features of raven-toolbox have no RAVEN counterpart in the generated map ## Features specific to RAVEN -### tINIT +### Legacy tINIT method -The functions `getINITModel` and `runINIT` are the original tINIT implementation, and RAVEN keeps them for models built with them. raven-toolbox implements ftINIT as its sole INIT variant, as a design decision. A tINIT model has to be reproduced in MATLAB (see [Context-specific models with ftINIT](../guide/init.md)). +The functions `getINITModel` and `runINIT` are the legacy tINIT method, which ftINIT replaces. RAVEN keeps them for models built with them. raven-toolbox implements ftINIT as its sole INIT variant, as a design decision. A model built with tINIT has to be reproduced in MATLAB (see [Context-specific models with ftINIT](../guide/init.md)). ### COBRA Toolbox conversion diff --git a/docs/references.md b/docs/references.md index b230181..65ac284 100644 --- a/docs/references.md +++ b/docs/references.md @@ -31,9 +31,9 @@ > the National Academy of Sciences 120(6): e2217868120. > -### tINIT +### Legacy tINIT method -`runINIT` and `getINITModel` (RAVEN's MATLAB implementation of tINIT) build on the INIT method described in the following paper. tINIT has no citation of its own. +`runINIT` and `getINITModel` (the legacy tINIT method of RAVEN in MATLAB, which ftINIT replaces) build on the INIT method described in the following paper. tINIT has no citation of its own. > Agren R, Bordel S, Mardinoglu A, Pornputtapong N, Nookaew I, Nielsen J > (2012). **Reconstruction of genome-scale active metabolic networks for 69 diff --git a/docs/tuned-parameters.md b/docs/tuned-parameters.md index 9ee5fa8..29458e5 100644 --- a/docs/tuned-parameters.md +++ b/docs/tuned-parameters.md @@ -44,7 +44,7 @@ Full detail: [Benchmarks of `fseof`, `remove_genes` and other defaults](paramete RAVEN MATLAB: `runINIT`, `scoreModel`, `getINITModel`, `ftINIT`, `prepINITModel`, `ftINITInternalAlg`, `getINITSteps`. raven-toolbox: `score_reactions_from_genes`, `gene_scores_from_expression`, `ftinit`, `prep_init_model`, `run_ftinit`, `get_init_steps`, `classify_reactions`. -Only MATLAB has tINIT (`runINIT`, `getINITModel`); raven-toolbox implements ftINIT alone (see [Context-specific models with ftINIT](guide/init.md)). Rows measured across both algorithms say so. +Only MATLAB has the legacy tINIT method (`runINIT`, `getINITModel`), which ftINIT replaces; raven-toolbox implements ftINIT alone (see [Context-specific models with ftINIT](guide/init.md)). Rows measured across both algorithms say so. | Parameter | Default | How determined | |---|---|---|