Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 12 additions & 12 deletions docs/guide/kegg.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

[Reconstruction from homology](homology.md) needs a template model of a related organism. KEGG needs none, because its orthology groups (KOs) are linked to reactions, and annotating a genome with KOs gives a draft directly. The method depends on whether the organism is in KEGG.

The documentation build does not run these examples. A KEGG reconstruction downloads tens of megabytes and takes minutes. The Python numbers come from raven-toolbox 3.0.0b1 with the `kegg118` artefacts.
The documentation build does not run these examples. A KEGG reconstruction downloads tens of megabytes and takes minutes. The Python numbers come from raven-toolbox 3.0.0b2 with the `kegg120` artefacts.

## Functions

Expand All @@ -25,7 +25,7 @@ When KEGG has the species (`sce` for *S. cerevisiae*), the assignments of genes

```matlab
model = getKEGGModelForOrganism('sce', ...
'dataDir', fullfile(tempdir, 'kegg118_eukaryotes'));
'dataDir', fullfile(tempdir, 'kegg120_eukaryotes'));
```

Without a `fastaFile`, the reconstruction runs from the organism abbreviation alone. No sequence search takes place, and the HMM library is not fetched, because the download takes place when a FASTA file is supplied. The reconstruction needs the global KEGG model, which `getModelFromKEGG` supplies. On first use there is no `keggModel.mat` to load. The function downloads the published artefacts (the reference model without genes and the tables of KOs, reactions and organism genes), assembles them and caches the result. The first build takes a while and needs a few hundred MB of disk, and later calls load the cached file.
Expand All @@ -46,10 +46,10 @@ print(f"{len(model.reactions)} rxns, {len(model.metabolites)} mets, "
```

```text
1357 rxns, 1502 mets, 838 genes
1362 rxns, 1507 mets, 837 genes
```

The artefacts (a reference model and three tables, about 49 MB) are fetched from the `kegg118` release of `toolbox-assets` on first use and cached. [Download data and binaries](../installation/data-and-binaries.md) gives the location and describes how to fetch them before a batch run. The first run takes a few minutes, and later runs take about as long, because assembling the draft from the tables is slow and the download is not.
The artefacts (a reference model and three tables, about 51 MB) are fetched from the `kegg120` release of `toolbox-assets` on first use and cached. [Download data and binaries](../installation/data-and-binaries.md) gives the location and describes how to fetch them before a batch run. The first run takes a few minutes, and later runs take about as long, because assembling the draft from the tables is slow and the download is not.
:::
::::

Expand All @@ -64,7 +64,7 @@ For an organism that is not in KEGG, the KO assignments are made from sequence.
```matlab
model = getKEGGModelForOrganism('hpo', ...
'fastaFile', 'hanpo.faa', ...
'dataDir', fullfile(tempdir, 'kegg118_eukaryotes'), ...
'dataDir', fullfile(tempdir, 'kegg120_eukaryotes'), ...
'outDir', fullfile(tempdir, 'hanpo_hmm'));
```
:::
Expand All @@ -81,7 +81,7 @@ hanpo_model = get_kegg_model_from_sequences_with_artefacts("hanpo.faa", domain="
:::
::::

This method is expensive. The eukaryotic HMM library is 135 MB compressed. One timed `hmmsearch` of a proteome (*S. cerevisiae*, 6021 sequences) against the eukaryotic library (2698 profiles) took 89.7 s on 16 threads and 88.6 s on 12 threads (AMD EPYC 9354). The method also needs HMMER, which is not available as a bundle for every platform, and [Download data and binaries](../installation/data-and-binaries.md) has the matrix. Without `threads`, `run_hmmsearch`, `get_kegg_model_from_sequences`, `build_ko_hmm` and `build_hmm_library` use every CPU that the process can run on. `getKEGGModelForOrganism` runs `hmmsearch` on the cores assigned to MATLAB (`feature('numcores')`) and has no `threads` option.
This method is expensive. The eukaryotic HMM library is 140 MB compressed. A search of the proteome of *S. cerevisiae* (6021 sequences) against it takes about 1.5 minutes on 12 to 16 threads. The method also needs HMMER, which is not available as a bundle for every platform, and [Download data and binaries](../installation/data-and-binaries.md) has the matrix. Without `threads`, `run_hmmsearch`, `get_kegg_model_from_sequences`, `build_ko_hmm` and `build_hmm_library` use every CPU that the process can run on. `getKEGGModelForOrganism` runs `hmmsearch` on the cores assigned to MATLAB (`feature('numcores')`) and has no `threads` option.

With a `fastaFile`, the organism identifier names the model. The search uses no phylogenetic distance and ignores `maxPhylDist`.

Expand All @@ -92,9 +92,9 @@ Three cut-offs shape the assignment. `cutOff` is the HMMER significance that a h
A KEGG draft has the size of a genome-scale model, and it differs from a working model in every other respect.

```text title="the sce draft, measured"
reactions: 1357
metabolites: 1502
genes: 838
reactions: 1362
metabolites: 1507
genes: 837
compartments: 1 (everything is in 's')
exchange rxns: 0
objective: none
Expand All @@ -112,7 +112,7 @@ print(f"{len(strict.reactions)} rxns, {len(strict.genes)} genes "
```

```text
1348 rxns, 827 genes (from 1357 and 838)
1353 rxns, 826 genes (from 1362 and 837)
```

This removes nine reactions from this organism, and the effect differs between organisms and should be measured for each. `keepGeneral` / `keep_general` is off by default, unlike the other flags. General reactions are placeholders such as "an alcohol + NAD+", and admitting them produces a network that appears to do more than it can.
Expand All @@ -123,11 +123,11 @@ This removes nine reactions from this organism, and the effect differs between o
- The draft is expected to be a model. It has no biomass reaction, no exchange reactions, no compartments and no medium. It is an inventory of reactions with gene associations.
- The KO assignment is trusted. A KO is an orthology group, which does not demonstrate an activity in the organism, and a partial HMM hit is weaker.
- The stoichiometry is undefined. Reactions with `n` or `x` coefficients cannot be mass-balanced. They are kept by default because removing them removes real chemistry, and they affect [Quality control](quality-control.md).
- The KEGG release changes. The artefacts are built from one KEGG release, `kegg118` here. A rebuild with another release gives a different model, so record the release.
- The KEGG release changes. The artefacts are built from one KEGG release, `kegg120` here. A rebuild with another release gives a different model, so record the release.
:::

## See also

- [Reconstruction from homology](homology.md): the same goal with a template model in place of an orthology database.
- [Gap-filling](gap-filling.md): the step for the 1357 reactions of a draft without exchange reactions, none of which can carry flux.
- [Gap-filling](gap-filling.md): the step for the 1362 reactions of a draft without exchange reactions, none of which can carry flux.
- [Legacy tutorial 5](../tutorials/tutorial5.md): the original KEGG reconstruction exercise from the RAVEN paper.
4 changes: 2 additions & 2 deletions docs/installation/data-and-binaries.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ Both toolboxes read from the same [`toolbox-assets`](https://github.com/CompMetE

| Artefact | Download size | Needed for |
|---|---|---|
| KEGG core bundle (reference model and the tables of KOs, reactions and organism genes) | 49 MB | [Reconstruction from KEGG](../guide/kegg.md), both methods |
| KEGG HMM library, one per domain | 135 MB for eukaryotes, 155 MB for prokaryotes | a KEGG draft for an organism that is not in KEGG |
| KEGG core bundle (reference model and the tables of KOs, reactions and organism genes) | 51 MB | [Reconstruction from KEGG](../guide/kegg.md), both methods |
| KEGG HMM library, one per domain | 140 MB for eukaryotes, 159 MB for prokaryotes | a KEGG draft for an organism that is not in KEGG |
| KEGG taxonomy | 0.2 MB | phylogenetic weighting of KO assignments |
| BLAST+ (`blastp`, `makeblastdb`) | 13 to 35 MB per platform | [Reconstruction from homology](../guide/homology.md) |
| DIAMOND | 3 to 9 MB per platform | the faster homology method |
Expand Down
4 changes: 2 additions & 2 deletions docs/raven3-migration/formats-and-reconstruction.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,8 +75,8 @@ The function `conversion/ravenCobraWrapper.m` keeps its signature, `newModel = r
| Source of the reaction and gene database | Local KEGG FTP dump (licence required), read when the cached files `keggGenes.mat`, `keggMets.mat` and `keggRxns.mat` are absent | Prebuilt artefacts fetched on demand from releases of `toolbox-assets` |
| Functions that build the database | `getGenesFromKEGG`, `getMetsFromKEGG` and `getRxnsFromKEGG` | `buildGlobalKEGGModel`, `buildGlobalGPR` and `readKEGGTable` |
| Argument `keggPath` of `getModelFromKEGG` | Optional, default `external/kegg` | Removed |
| Ortholog HMM library | One `.hmm` file per KEGG Orthology (KO) entry and one `hmmsearch` call per KO, through the Windows Subsystem for Linux (WSL) on Windows | One concatenated library per domain (`kegg118_eukaryotes.hmm` and `kegg118_prokaryotes.hmm`), one `hmmsearch` call against the whole proteome, and no WSL |
| `dataDir` | Any local folder | For the homology method, which takes a FASTA file, the folder name must end in `kegg118_eukaryotes` or `kegg118_prokaryotes`. The annotation method ignores `dataDir` |
| Ortholog HMM library | One `.hmm` file per KEGG Orthology (KO) entry and one `hmmsearch` call per KO, through the Windows Subsystem for Linux (WSL) on Windows | One concatenated library per domain (`kegg120_eukaryotes.hmm` and `kegg120_prokaryotes.hmm`), one `hmmsearch` call against the whole proteome, and no WSL |
| `dataDir` | Any local folder | For the homology method, which takes a FASTA file, the folder name must end in `kegg120_eukaryotes` or `kegg120_prokaryotes`. The annotation method ignores `dataDir` |
| `nSequences`, `seqIdentity` (CD-HIT clustering controls) | Configurable | Removed; there is no local clustering step |
| Default of `cutOff` | `10^-50` | `10^-30` |
| Default of `minScoreRatioG` | `0.8` | `0.9` |
Expand Down
2 changes: 1 addition & 1 deletion docs/raven3-migration/new-and-environment.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ The function documentation of `sampleCHRR` states that the polytope is rounded b

KEGG data, binaries and the checksum manifest come from releases of `toolbox-assets`. Other downloads have other sources. A RAVEN 3 clone needs network access for these downloads.

- KEGG data comes from the release tag that `keggDataVersion()` returns (`src/reconstruction/kegg/KEGG_VERSION.md`, tag `kegg118`).
- KEGG data comes from the release tag that `keggDataVersion()` returns (`src/reconstruction/kegg/KEGG_VERSION.md`, tag `kegg120`).
- The binaries come from the fixed tags in `src/installation/downloadRavenBinaries.m`: `blast-2.17.0`, `diamond-2.1.17`, `hmmer-3.4.0` and, on Windows, `hmmer-3.3.2`.
- The checksums of downloaded files come from `manifest.json` of the `manifest-v3` release, which is read at download time.
- `setRavenSolver('scip')` on Windows downloads `scip_mex_win.zip` from the RAVEN v2.8.6 release on GitHub into `src/external/scip`. The repository holds no SCIP files, and other platforms need a SCIP installation made by hand.
Expand Down
4 changes: 2 additions & 2 deletions docs/tutorials/tutorial5.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,12 @@ The de novo reconstruction searches large HMM databases. Building the model take

### 1. Reconstructing from KEGG

`getKEGGModelForOrganism` needs trained hidden Markov models (HMMs) for eukaryotes. When the library file `<dataDir>.hmm` or its `.gz` archive exists, for example after a manual download, it is used. Otherwise it is downloaded from the matching release of [toolbox-assets](https://github.com/CompMetEng/toolbox-assets). Here the archive `kegg118_eukaryotes` is selected for automatic download.
`getKEGGModelForOrganism` needs trained hidden Markov models (HMMs) for eukaryotes. When the library file `<dataDir>.hmm` or its `.gz` archive exists, for example after a manual download, it is used. Otherwise it is downloaded from the matching release of [toolbox-assets](https://github.com/CompMetEng/toolbox-assets). Here the archive `kegg120_eukaryotes` is selected for automatic download.

The function then creates a model for *S. cerevisiae* from the FASTA file of the whole proteome (`sce.fa`). The parameters exclude general or unclear reactions and reactions with undefined stoichiometry. `help getKEGGModelForOrganism` describes the parameters.

```matlab
model=getKEGGModelForOrganism('sce','sce.fa','kegg118_eukaryotes','output',false,false,false,false,10^-30,0.8,0.3,-1);
model=getKEGGModelForOrganism('sce','sce.fa','kegg120_eukaryotes','output',false,false,false,false,10^-30,0.8,0.3,-1);
disp(model);
```

Expand Down
2 changes: 2 additions & 0 deletions scripts/known_names.txt
Original file line number Diff line number Diff line change
Expand Up @@ -399,6 +399,8 @@ seqIdentity # getKEGGModelForOrganism argument, removed

kegg118_eukaryotes # raven-data KEGG artefact name, not a function
kegg118_prokaryotes # raven-data KEGG artefact name, not a function
kegg120_eukaryotes # toolbox-assets KEGG artefact name, not a function
kegg120_prokaryotes # toolbox-assets KEGG artefact name, not a function
xml_toolbox # third-party library bundled by RAVEN 2's legacy/

# checkRaven.m sits at the RAVEN root, outside src/, which the index reads.
Expand Down
Loading