From aa8e87a70e99ca59dd1150e79cf352d3a16c0bf1 Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Tue, 3 Nov 2026 12:15:06 +0100 Subject: [PATCH 1/2] Document the kegg120 assets The KEGG guide, the download page, the migration pages and tutorial 5 name the kegg120 release. The draft of S. cerevisiae and the sizes of the assets are the measured values of kegg120. --- docs/guide/kegg.md | 24 +++++++++---------- docs/installation/data-and-binaries.md | 4 ++-- .../formats-and-reconstruction.md | 4 ++-- docs/raven3-migration/new-and-environment.md | 2 +- docs/tutorials/tutorial5.md | 4 ++-- 5 files changed, 19 insertions(+), 19 deletions(-) diff --git a/docs/guide/kegg.md b/docs/guide/kegg.md index 03f4a75..55157ce 100644 --- a/docs/guide/kegg.md +++ b/docs/guide/kegg.md @@ -4,7 +4,7 @@ [Reconstruction from homology](homology.md) needs a template model of a related organism. KEGG needs none, because its orthology groups (KOs) are linked to reactions, and annotating a genome with KOs gives a draft directly. The method depends on whether the organism is in KEGG. -The documentation build does not run these examples. A KEGG reconstruction downloads tens of megabytes and takes minutes. The Python numbers come from raven-toolbox 3.0.0b1 with the `kegg118` artefacts. +The documentation build does not run these examples. A KEGG reconstruction downloads tens of megabytes and takes minutes. The Python numbers come from raven-toolbox 3.0.0b2 with the `kegg120` artefacts. ## Functions @@ -25,7 +25,7 @@ When KEGG has the species (`sce` for *S. cerevisiae*), the assignments of genes ```matlab model = getKEGGModelForOrganism('sce', ... - 'dataDir', fullfile(tempdir, 'kegg118_eukaryotes')); + 'dataDir', fullfile(tempdir, 'kegg120_eukaryotes')); ``` Without a `fastaFile`, the reconstruction runs from the organism abbreviation alone. No sequence search takes place, and the HMM library is not fetched, because the download takes place when a FASTA file is supplied. The reconstruction needs the global KEGG model, which `getModelFromKEGG` supplies. On first use there is no `keggModel.mat` to load. The function downloads the published artefacts (the reference model without genes and the tables of KOs, reactions and organism genes), assembles them and caches the result. The first build takes a while and needs a few hundred MB of disk, and later calls load the cached file. @@ -46,10 +46,10 @@ print(f"{len(model.reactions)} rxns, {len(model.metabolites)} mets, " ``` ```text -1357 rxns, 1502 mets, 838 genes +1362 rxns, 1507 mets, 837 genes ``` -The artefacts (a reference model and three tables, about 49 MB) are fetched from the `kegg118` release of `toolbox-assets` on first use and cached. [Download data and binaries](../installation/data-and-binaries.md) gives the location and describes how to fetch them before a batch run. The first run takes a few minutes, and later runs take about as long, because assembling the draft from the tables is slow and the download is not. +The artefacts (a reference model and three tables, about 51 MB) are fetched from the `kegg120` release of `toolbox-assets` on first use and cached. [Download data and binaries](../installation/data-and-binaries.md) gives the location and describes how to fetch them before a batch run. The first run takes a few minutes, and later runs take about as long, because assembling the draft from the tables is slow and the download is not. ::: :::: @@ -64,7 +64,7 @@ For an organism that is not in KEGG, the KO assignments are made from sequence. ```matlab model = getKEGGModelForOrganism('hpo', ... 'fastaFile', 'hanpo.faa', ... - 'dataDir', fullfile(tempdir, 'kegg118_eukaryotes'), ... + 'dataDir', fullfile(tempdir, 'kegg120_eukaryotes'), ... 'outDir', fullfile(tempdir, 'hanpo_hmm')); ``` ::: @@ -81,7 +81,7 @@ hanpo_model = get_kegg_model_from_sequences_with_artefacts("hanpo.faa", domain=" ::: :::: -This method is expensive. The eukaryotic HMM library is 135 MB compressed. One timed `hmmsearch` of a proteome (*S. cerevisiae*, 6021 sequences) against the eukaryotic library (2698 profiles) took 89.7 s on 16 threads and 88.6 s on 12 threads (AMD EPYC 9354). The method also needs HMMER, which is not available as a bundle for every platform, and [Download data and binaries](../installation/data-and-binaries.md) has the matrix. Without `threads`, `run_hmmsearch`, `get_kegg_model_from_sequences`, `build_ko_hmm` and `build_hmm_library` use every CPU that the process can run on. `getKEGGModelForOrganism` runs `hmmsearch` on the cores assigned to MATLAB (`feature('numcores')`) and has no `threads` option. +This method is expensive. The eukaryotic HMM library is 140 MB compressed. A search of the proteome of *S. cerevisiae* (6021 sequences) against it takes about 1.5 minutes on 12 to 16 threads. The method also needs HMMER, which is not available as a bundle for every platform, and [Download data and binaries](../installation/data-and-binaries.md) has the matrix. Without `threads`, `run_hmmsearch`, `get_kegg_model_from_sequences`, `build_ko_hmm` and `build_hmm_library` use every CPU that the process can run on. `getKEGGModelForOrganism` runs `hmmsearch` on the cores assigned to MATLAB (`feature('numcores')`) and has no `threads` option. With a `fastaFile`, the organism identifier names the model. The search uses no phylogenetic distance and ignores `maxPhylDist`. @@ -92,9 +92,9 @@ Three cut-offs shape the assignment. `cutOff` is the HMMER significance that a h A KEGG draft has the size of a genome-scale model, and it differs from a working model in every other respect. ```text title="the sce draft, measured" -reactions: 1357 -metabolites: 1502 -genes: 838 +reactions: 1362 +metabolites: 1507 +genes: 837 compartments: 1 (everything is in 's') exchange rxns: 0 objective: none @@ -112,7 +112,7 @@ print(f"{len(strict.reactions)} rxns, {len(strict.genes)} genes " ``` ```text -1348 rxns, 827 genes (from 1357 and 838) +1353 rxns, 826 genes (from 1362 and 837) ``` This removes nine reactions from this organism, and the effect differs between organisms and should be measured for each. `keepGeneral` / `keep_general` is off by default, unlike the other flags. General reactions are placeholders such as "an alcohol + NAD+", and admitting them produces a network that appears to do more than it can. @@ -123,11 +123,11 @@ This removes nine reactions from this organism, and the effect differs between o - The draft is expected to be a model. It has no biomass reaction, no exchange reactions, no compartments and no medium. It is an inventory of reactions with gene associations. - The KO assignment is trusted. A KO is an orthology group, which does not demonstrate an activity in the organism, and a partial HMM hit is weaker. - The stoichiometry is undefined. Reactions with `n` or `x` coefficients cannot be mass-balanced. They are kept by default because removing them removes real chemistry, and they affect [Quality control](quality-control.md). -- The KEGG release changes. The artefacts are built from one KEGG release, `kegg118` here. A rebuild with another release gives a different model, so record the release. +- The KEGG release changes. The artefacts are built from one KEGG release, `kegg120` here. A rebuild with another release gives a different model, so record the release. ::: ## See also - [Reconstruction from homology](homology.md): the same goal with a template model in place of an orthology database. -- [Gap-filling](gap-filling.md): the step for the 1357 reactions of a draft without exchange reactions, none of which can carry flux. +- [Gap-filling](gap-filling.md): the step for the 1362 reactions of a draft without exchange reactions, none of which can carry flux. - [Legacy tutorial 5](../tutorials/tutorial5.md): the original KEGG reconstruction exercise from the RAVEN paper. diff --git a/docs/installation/data-and-binaries.md b/docs/installation/data-and-binaries.md index f7b6572..835275a 100644 --- a/docs/installation/data-and-binaries.md +++ b/docs/installation/data-and-binaries.md @@ -8,8 +8,8 @@ Both toolboxes read from the same [`toolbox-assets`](https://github.com/CompMetE | Artefact | Download size | Needed for | |---|---|---| -| KEGG core bundle (reference model and the tables of KOs, reactions and organism genes) | 49 MB | [Reconstruction from KEGG](../guide/kegg.md), both methods | -| KEGG HMM library, one per domain | 135 MB for eukaryotes, 155 MB for prokaryotes | a KEGG draft for an organism that is not in KEGG | +| KEGG core bundle (reference model and the tables of KOs, reactions and organism genes) | 51 MB | [Reconstruction from KEGG](../guide/kegg.md), both methods | +| KEGG HMM library, one per domain | 140 MB for eukaryotes, 159 MB for prokaryotes | a KEGG draft for an organism that is not in KEGG | | KEGG taxonomy | 0.2 MB | phylogenetic weighting of KO assignments | | BLAST+ (`blastp`, `makeblastdb`) | 13 to 35 MB per platform | [Reconstruction from homology](../guide/homology.md) | | DIAMOND | 3 to 9 MB per platform | the faster homology method | diff --git a/docs/raven3-migration/formats-and-reconstruction.md b/docs/raven3-migration/formats-and-reconstruction.md index bb67b92..64bade7 100644 --- a/docs/raven3-migration/formats-and-reconstruction.md +++ b/docs/raven3-migration/formats-and-reconstruction.md @@ -75,8 +75,8 @@ The function `conversion/ravenCobraWrapper.m` keeps its signature, `newModel = r | Source of the reaction and gene database | Local KEGG FTP dump (licence required), read when the cached files `keggGenes.mat`, `keggMets.mat` and `keggRxns.mat` are absent | Prebuilt artefacts fetched on demand from releases of `toolbox-assets` | | Functions that build the database | `getGenesFromKEGG`, `getMetsFromKEGG` and `getRxnsFromKEGG` | `buildGlobalKEGGModel`, `buildGlobalGPR` and `readKEGGTable` | | Argument `keggPath` of `getModelFromKEGG` | Optional, default `external/kegg` | Removed | -| Ortholog HMM library | One `.hmm` file per KEGG Orthology (KO) entry and one `hmmsearch` call per KO, through the Windows Subsystem for Linux (WSL) on Windows | One concatenated library per domain (`kegg118_eukaryotes.hmm` and `kegg118_prokaryotes.hmm`), one `hmmsearch` call against the whole proteome, and no WSL | -| `dataDir` | Any local folder | For the homology method, which takes a FASTA file, the folder name must end in `kegg118_eukaryotes` or `kegg118_prokaryotes`. The annotation method ignores `dataDir` | +| Ortholog HMM library | One `.hmm` file per KEGG Orthology (KO) entry and one `hmmsearch` call per KO, through the Windows Subsystem for Linux (WSL) on Windows | One concatenated library per domain (`kegg120_eukaryotes.hmm` and `kegg120_prokaryotes.hmm`), one `hmmsearch` call against the whole proteome, and no WSL | +| `dataDir` | Any local folder | For the homology method, which takes a FASTA file, the folder name must end in `kegg120_eukaryotes` or `kegg120_prokaryotes`. The annotation method ignores `dataDir` | | `nSequences`, `seqIdentity` (CD-HIT clustering controls) | Configurable | Removed; there is no local clustering step | | Default of `cutOff` | `10^-50` | `10^-30` | | Default of `minScoreRatioG` | `0.8` | `0.9` | diff --git a/docs/raven3-migration/new-and-environment.md b/docs/raven3-migration/new-and-environment.md index 86023db..85a9b9b 100644 --- a/docs/raven3-migration/new-and-environment.md +++ b/docs/raven3-migration/new-and-environment.md @@ -98,7 +98,7 @@ The function documentation of `sampleCHRR` states that the polytope is rounded b KEGG data, binaries and the checksum manifest come from releases of `toolbox-assets`. Other downloads have other sources. A RAVEN 3 clone needs network access for these downloads. -- KEGG data comes from the release tag that `keggDataVersion()` returns (`src/reconstruction/kegg/KEGG_VERSION.md`, tag `kegg118`). +- KEGG data comes from the release tag that `keggDataVersion()` returns (`src/reconstruction/kegg/KEGG_VERSION.md`, tag `kegg120`). - The binaries come from the fixed tags in `src/installation/downloadRavenBinaries.m`: `blast-2.17.0`, `diamond-2.1.17`, `hmmer-3.4.0` and, on Windows, `hmmer-3.3.2`. - The checksums of downloaded files come from `manifest.json` of the `manifest-v3` release, which is read at download time. - `setRavenSolver('scip')` on Windows downloads `scip_mex_win.zip` from the RAVEN v2.8.6 release on GitHub into `src/external/scip`. The repository holds no SCIP files, and other platforms need a SCIP installation made by hand. diff --git a/docs/tutorials/tutorial5.md b/docs/tutorials/tutorial5.md index 864a764..c402755 100644 --- a/docs/tutorials/tutorial5.md +++ b/docs/tutorials/tutorial5.md @@ -10,12 +10,12 @@ The de novo reconstruction searches large HMM databases. Building the model take ### 1. Reconstructing from KEGG -`getKEGGModelForOrganism` needs trained hidden Markov models (HMMs) for eukaryotes. When the library file `.hmm` or its `.gz` archive exists, for example after a manual download, it is used. Otherwise it is downloaded from the matching release of [toolbox-assets](https://github.com/CompMetEng/toolbox-assets). Here the archive `kegg118_eukaryotes` is selected for automatic download. +`getKEGGModelForOrganism` needs trained hidden Markov models (HMMs) for eukaryotes. When the library file `.hmm` or its `.gz` archive exists, for example after a manual download, it is used. Otherwise it is downloaded from the matching release of [toolbox-assets](https://github.com/CompMetEng/toolbox-assets). Here the archive `kegg120_eukaryotes` is selected for automatic download. The function then creates a model for *S. cerevisiae* from the FASTA file of the whole proteome (`sce.fa`). The parameters exclude general or unclear reactions and reactions with undefined stoichiometry. `help getKEGGModelForOrganism` describes the parameters. ```matlab -model=getKEGGModelForOrganism('sce','sce.fa','kegg118_eukaryotes','output',false,false,false,false,10^-30,0.8,0.3,-1); +model=getKEGGModelForOrganism('sce','sce.fa','kegg120_eukaryotes','output',false,false,false,false,10^-30,0.8,0.3,-1); disp(model); ``` From 5b425f7c146bf7b8d8e6030f2ad9ff5ed4d92aca Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Sun, 8 Nov 2026 11:58:24 +0100 Subject: [PATCH 2/2] Allow-list the kegg120 artefact names --- scripts/known_names.txt | 2 ++ 1 file changed, 2 insertions(+) diff --git a/scripts/known_names.txt b/scripts/known_names.txt index d214279..e3e8983 100644 --- a/scripts/known_names.txt +++ b/scripts/known_names.txt @@ -399,6 +399,8 @@ seqIdentity # getKEGGModelForOrganism argument, removed kegg118_eukaryotes # raven-data KEGG artefact name, not a function kegg118_prokaryotes # raven-data KEGG artefact name, not a function +kegg120_eukaryotes # toolbox-assets KEGG artefact name, not a function +kegg120_prokaryotes # toolbox-assets KEGG artefact name, not a function xml_toolbox # third-party library bundled by RAVEN 2's legacy/ # checkRaven.m sits at the RAVEN root, outside src/, which the index reads.