diff --git a/docs/guide/homology.md b/docs/guide/homology.md index 4a29186..1942a8f 100644 --- a/docs/guide/homology.md +++ b/docs/guide/homology.md @@ -145,7 +145,7 @@ draft: 37 rxns, 49 mets, 54 genes ::: :::: -The transfer works reaction by reaction. For each reaction of the template, the genes in its gene rule are looked up in the hit table. A template gene is replaced by its accepted counterpart in the new organism, and the reaction is carried over when the rule can be satisfied after the substitution. A reaction that needs two subunits is kept with a placeholder gene (`OLD__`) for the missing subunit under the default `complexPolicy` (`complex_policy` in Python) of `flag`, and the policy `drop` removes it. A reaction with two isozymes is kept when either has a counterpart. The metabolites that a carried reaction needs are carried with it. The draft has 49 metabolites against 52 in the template, because the three metabolites that occur in dropped reactions alone are not created. +The transfer works reaction by reaction. For each reaction of the template, the genes in its gene rule are looked up in the hit table. A template gene is replaced by its accepted counterpart in the new organism, and the reaction is carried over when the rule can be satisfied after the substitution. A reaction that needs two subunits is kept with a placeholder gene (`OLD__`) for the missing subunit under the default `complexPolicy` (`complex_policy` in Python) of `flag`, and the policy `drop` removes each complex that has a missing subunit and removes the reaction when no complex remains. The rule `(A and B) or (C and D)` without a counterpart of `B` gives `C and D` under `drop`. A reaction with two isozymes is kept when either has a counterpart. The metabolites that a carried reaction needs are carried with it. The draft has 49 metabolites against 52 in the template, because the three metabolites that occur in dropped reactions alone are not created. The line `Standardizing grRules` shows RAVEN rewriting the rules of the template in a canonical form before the substitution. One rule written in two ways is then handled identically. diff --git a/docs/installation/updating-binaries.md b/docs/installation/updating-binaries.md index 326d3b5..aed3ebe 100644 --- a/docs/installation/updating-binaries.md +++ b/docs/installation/updating-binaries.md @@ -2,41 +2,41 @@ # Update the binary bundles -The BLAST+, DIAMOND and HMMER executables are published as one ZIP file per platform in [`toolbox-assets`](https://github.com/CompMetEng/toolbox-assets), and raven-toolbox resolves them at run time through `raven_toolbox.binaries`. [Download data and binaries](data-and-binaries.md) describes the resolution order, the `RAVEN_PYTHON_*` overrides and the `raven-toolbox-binaries` command for users. The sections below describe what a bundle contains and how the package maintainers publish a new one. Release tags are immutable, so a new version of a tool becomes a new tag, and every other tag stays as it is. +The BLAST+, DIAMOND and HMMER executables are published as one ZIP file per platform in [`toolbox-assets`](https://github.com/CompMetEng/toolbox-assets), and raven-toolbox resolves them at run time through `raven_toolbox.binaries`. [Download data and binaries](data-and-binaries.md) describes the resolution order, the `RAVEN_PYTHON_*` overrides and the `raven-toolbox-binaries` command for users. A new version of a tool is a new release tag, and every other tag stays as it is. -The documentation build does not run these commands. +The documentation build does not run the commands below. ## Contents of a bundle | Bundle | Included | Left out | |---|---|---| -| `diamond` | `diamond` | nothing; it is one static binary | -| `blast` | `blastp`, `makeblastdb` | `blastn`, `tblastn`, `psiblast`, `rpsblast`, `blast_formatter`, the `*_vdb` tools, `doc/`, `ChangeLog`, `README` and the other tools of the suite | +| `diamond` | `diamond` | nothing; the bundle is one executable | +| `blast` | `blastp`, `makeblastdb` | the other executables and the documentation of the suite | | `hmmer` | `hmmsearch` | the rest of the suite | -Both toolboxes invoke only `makeblastdb`, `blastp`, and the `makedb` and `blastp` subcommands of DIAMOND, so the same minimal set serves both. For BLAST+ the minimal set reduces the size of the bundle from hundreds of megabytes to a few. +A bundle holds the executables that the toolboxes invoke for a user: `makeblastdb` and `blastp` of BLAST+, the `makedb` and `blastp` subcommands of DIAMOND, and `hmmsearch` of HMMER. The build of the KEGG libraries needs `hmmbuild`, MAFFT and CD-HIT as well; they are not bundled, and [Update the KEGG assets](updating-kegg-assets.md) lists them. The published BLAST+ ZIP files are 13 to 35 MB, depending on the platform. ## File name and layout -A ZIP file is named `---.zip`. It is flat: the executables lie at the root beside the upstream `LICENSE`, without a nested `bin/` folder. A Windows ZIP carries the `.exe` files and the runtime DLLs that the build needs at the root, `nghttp2.dll` for BLAST+ and `cygwin1.dll` for HMMER, because the extraction is flat and the executable is looked up as `.exe`. +A ZIP file is named `---.zip`. It is flat: the executables lie at the root beside the upstream `LICENSE`, without a nested `bin/` folder. A Windows ZIP carries the `.exe` files and the runtime DLLs of the executables at the root, `nghttp2.dll` for BLAST+ and `cygwin1.dll` for HMMER, because the extraction is flat and the executable is looked up as `.exe`. ## Platforms and licences -The bundles are built for `linux-x86_64` first, then for `macos-arm64`, `macos-x86_64`, `linux-arm64` and `windows-x86_64`. A platform without a bundle is a supported state: `ensure_binary` raises an error that names the conda package to install. +`scripts/build_binary_bundles.py` builds the bundles for `linux-x86_64`, `macos-arm64` and `windows-x86_64`. The `macos-arm64` bundle serves Apple Silicon. On a platform without a bundle, `ensure_binary` raises an error that gives the `conda install -c bioconda` command for the executable. | Tool | Licence | Requirement for the ZIP | |---|---|---| | BLAST+ | produced by NCBI, public domain | the upstream `LICENSE`, for provenance | -| DIAMOND | GPL-3.0 | the licence text, the unmodified binary, and a separate asset outside the wheel | +| DIAMOND | GPL-3.0 | the licence text, the unmodified binary, and a separate asset that is not part of the raven-toolbox package | | HMMER | BSD-3-Clause | the upstream `LICENSE` | -The release body or a provenance file records, for each asset, the upstream URL, the upstream version, the upstream checksum and the published SHA-256 sum. +`PROVENANCE-binaries.txt` in the repository `toolbox-assets` records the source of the executables. It holds the content of the `PROVENANCE.txt` that the build script writes, and the publish command does not upload it. ## HMMER on Windows No native Windows build of HMMER 3.4 exists, because upstream targets POSIX systems. The published `windows-x86_64` asset is HMMER 3.3.2, repackaged from RAVEN v2.10.5, the last release that bundles a Cygwin-compiled Windows build. The file `cygwin1.dll` ships beside the `.exe`. -Searching profiles that HMMER 3.4 built with HMMER 3.3.2 is sound for the published libraries. They are concatenated ASCII files in the `HMMER3/f` format, which is unchanged from HMMER 3.1 to 3.4, and HMMER 3.4 is a maintenance release over 3.3.2 without a change of the protein scoring model. The bit scores and the calibrated KEGG cut-offs therefore carry over. This holds while the libraries stay in ASCII format. +The libraries that HMMER 3.4 builds are concatenated ASCII files in the `HMMER3/f` format, which is unchanged from HMMER 3.1 to 3.4, and HMMER 3.4 is a maintenance release over 3.3.2 without a change of the protein scoring model. HMMER 3.3.2 therefore gives the same bit scores on these libraries, and the calibrated KEGG cut-offs carry over. This holds while the libraries stay in ASCII format. ## Publish a bundle @@ -47,15 +47,19 @@ python scripts/build_binary_bundles.py python scripts/publish_to_toolbox_assets.py binaries --dir dist/binaries python scripts/make_registry_snippet.py manifest --manifest data/manifest.json \ --target binary --bundle blast --version 2.17.0 --provides blastp makeblastdb \ - --dir dist/binaries --tag blast-2.17.0 + --dir dist/binaries --tag blast-2.17.0 --description "" --license "" python scripts/make_registry_snippet.py sync ``` -The first command builds the ZIP files from the vetted binaries in the `software/` folder of RAVEN into `dist/binaries`, with checksums and provenance. The second creates the release tags in `toolbox-assets` and skips the assets that exist. The third computes the checksums with the helper that `ensure_binary` verifies with, and writes the entry of the bundle to `data/manifest.json`. A bundle with a licence that needs a statement takes `--license`, for example `--license GPL-3.0-only` for DIAMOND. The fourth regenerates the baked registries of raven-toolbox from the manifest. +The first command downloads the executables from the RAVEN release v2.10.5 and from a pinned commit of RAVEN on GitHub, and builds the ZIP files in `dist/binaries` with checksums. The second creates the release tags in `toolbox-assets` and skips the assets that exist. The third computes the checksums with the helper that `ensure_binary` verifies with, and writes the entry of the bundle to `data/manifest.json`. It runs once for each bundle, and it replaces the whole entry of the bundle, so `--description` and `--license` repeat the values of the existing entry (`GPL-3.0-only` for DIAMOND). It takes the ZIP files whose names start with `--`. The entry of `hmmer` in the published manifest lists the Windows file of version 3.3.2 beside the files of version 3.4.0. That platform is added to `data/manifest.json` by hand, with the URL and the checksum of the file. The fourth regenerates the baked registries of raven-toolbox from the manifest. -`data/manifest.json` is the single source of truth, and the registries `_DATA_REGISTRY` in `src/raven_toolbox/data.py` and `_REGISTRY` in `src/raven_toolbox/binaries.py` are never edited by hand. The manifest is published as the next `manifest-v` snapshot, as step 5 of [Update the KEGG assets](updating-kegg-assets.md) describes. +`data/manifest.json` is the source of truth, and the registries `_DATA_REGISTRY` in `src/raven_toolbox/data.py` and `_REGISTRY` in `src/raven_toolbox/binaries.py` are never edited by hand. The manifest is published as the next `manifest-v` snapshot, as step 5 of [Update the KEGG assets](updating-kegg-assets.md) describes. -A new tool needs no new provisioning code. A bundle entry with the right `provides` list and ZIP files that follow the layout above are enough for the wrappers to resolve the tool through `ensure_binary`. +## A new version or a new tool + +- **New version of a tool:** the versions, the source references and the `PROVIDES` table are values in `scripts/build_binary_bundles.py` (`SPECS`, `RAVEN3`, `V2105`, `PROVIDES`). `downloadRavenBinaries.m` of RAVEN names the tags and the asset names and needs the new tag. +- **New tool:** an entry in `_ENV_VARS` of `raven_toolbox.binaries` adds the environment variable that overrides the executable, and an entry in `BINARY_SETS` adds the executable to the downloads of the command `raven-toolbox-binaries`. +- **Both toolboxes** read the new manifest snapshot after the update: the manifest URL in `fetchToolboxAsset.m` and the baked registry of raven-toolbox. ## See also diff --git a/docs/installation/updating-kegg-assets.md b/docs/installation/updating-kegg-assets.md index 13991df..8a031d9 100644 --- a/docs/installation/updating-kegg-assets.md +++ b/docs/installation/updating-kegg-assets.md @@ -4,7 +4,7 @@ The KEGG reference data and the libraries of profile hidden Markov models (HMMs) are published as assets of one KEGG release in [`toolbox-assets`](https://github.com/CompMetEng/toolbox-assets). Both toolboxes download them, as described in [Download data and binaries](data-and-binaries.md). A new KEGG release needs a new set of assets. The package maintainers do this when they adopt a new KEGG release, and users of the toolboxes never do. -The documentation build does not run these commands. The download needs a KEGG subscription, and the build of the libraries takes hours. +The documentation build does not run the commands below. The download needs a KEGG subscription, and the build of the libraries is the slow step. ## Assets of one release @@ -24,7 +24,7 @@ Release tags are immutable. A release that is published under a name is not chan ## Requirements - A subscription to the bulk download of KEGG (`ftp.kegg.net`), which comes with a user name and a password. -- raven-toolbox from a checkout of its repository, installed as described in [Installing RAVEN](python.md). The commands run from the root of that checkout. +- raven-toolbox from a checkout of its repository, installed as described in [raven-toolbox (Python)](python.md). The commands run from the root of that checkout. - HMMER (`hmmbuild`), MAFFT and CD-HIT, on the `PATH` or pointed at with the environment variables `RAVEN_PYTHON_HMMBUILD`, `RAVEN_PYTHON_MAFFT` and `RAVEN_PYTHON_CDHIT`. MAFFT and CD-HIT have no native Windows build, so the build runs on Linux, macOS or inside WSL2. Keep the whole toolchain inside WSL2, because raven-toolbox calls the resolved executable directly and does not translate paths across the boundary. - The GitHub CLI `gh`, authenticated with write access to `toolbox-assets`. - Disk space for the dump and for the per-KO working files of the build, and enough memory for the largest alignments of MAFFT. MAFFT switches to a memory-light method for an alignment that is predicted to need more memory than the host has, and that method approximates the guide tree. @@ -49,7 +49,7 @@ from raven_toolbox.reconstruction.kegg import download_kegg_dump download_kegg_dump("keggdb", force=True, progress=False) ``` -The function fetches the reaction, compound, glycan and KO archives, the proteomes and the taxonomy file, extracts them and arranges the flat layout that the parser expects. Without `force=True`, it skips files that exist, so use it for a new release in a directory that holds an older dump. +The function fetches the reaction, compound, glycan and KO archives, the proteomes and the taxonomy file, extracts them and arranges the flat layout that the parser expects. Without `force=True`, the function keeps the files that exist. It deletes the archives after it extracts them and keeps the file `taxonomy` of an older dump. A new release needs `force=True` or an empty directory. ### 2. Build the assets @@ -58,7 +58,7 @@ python scripts/build_kegg_artefacts.py --keggdb keggdb --out artefacts \ --version kegg120 --hmms --threads 16 ``` -The script parses the dump into the reference model and the tables, builds the library of each domain from the sequences of the KOs, and arranges the four assets in `artefacts`. `--domains` restricts the libraries to one domain. +The script parses the dump into the reference model and the tables, builds the library of each domain from the sequences of the KOs, and arranges the four assets in `artefacts`. With `--hmms`, `--domains` restricts the libraries to the listed domains (`prokaryotes`, `eukaryotes`). The build of the libraries is the slow step. The script resumes: it skips a stage whose output exists, and it skips every KO whose profile exists in the working directory `artefacts/_hmms-`. Run the same command again after a failure. `--force` rebuilds everything, and a changed gene-to-KO table needs it, because a resumed run assumes the inputs of the first run. @@ -68,7 +68,7 @@ The working directory holds one FASTA file and one profile per KO. On a file sys - The four files of the table exist in `artefacts`, and their sizes are of the same order as the previous release (49 MB, 135 MB, 155 MB and 0.2 MB for `kegg118`). - The number of profiles in a library is the number of lines that start with `NAME` in the decompressed file. Compare it with the previous release. -- Reconstruct a draft for an organism that KEGG contains, for example *S. cerevisiae*, with `get_kegg_model_for_organism_from_artefacts`, and search its proteome against the eukaryotic library with `get_kegg_model_from_sequences_with_artefacts`. The assignment of the search should resemble the KEGG assignment of the organism. Both functions are described in [Reconstruction from KEGG](../guide/kegg.md). +- Extract `kegg_core.tar.gz` into a folder, and decompress `kegg_eukaryotes.hmm.gz`. Reconstruct a draft for an organism that KEGG contains, for example *S. cerevisiae*, with `get_kegg_model_for_organism_from_artefacts("sce", artefact_dir=)`, and search its proteome against the eukaryotic library with `get_kegg_model_from_sequences_with_artefacts(, artefact_dir=, library=)`. Without `artefact_dir` (and `library`), the functions fetch the release that the registry names. The assignment of the search should resemble the KEGG assignment of the organism. Both functions are described in [Reconstruction from KEGG](../guide/kegg.md). - Record the KEGG release and the versions of HMMER, MAFFT and CD-HIT of the build. ### 4. Publish the assets @@ -78,7 +78,7 @@ python scripts/publish_to_toolbox_assets.py --dry-run release --tag kegg120 --di python scripts/publish_to_toolbox_assets.py release --tag kegg120 --dir artefacts ``` -The first command prints the `gh` calls without running them. The second creates the release `kegg120` in `toolbox-assets` and uploads the four assets. A rerun uploads only the assets that are missing. +The first command prints the `gh` calls without running them. The second creates the release `kegg120` in `toolbox-assets` and uploads the four assets. A rerun uploads the assets that are missing and leaves the others as they are. ### 5. Update the manifest and the registry @@ -88,26 +88,26 @@ The manifest `data/manifest.json` is the single source of truth for the checksum python scripts/make_registry_snippet.py manifest --manifest data/manifest.json \ --target data --dataset kegg --version kegg120 --dir artefacts --tag kegg120 python scripts/make_registry_snippet.py sync -python scripts/publish_to_toolbox_assets.py release --tag manifest-v4 data/manifest.json +python scripts/publish_to_toolbox_assets.py release --tag manifest-v data/manifest.json ``` -The first command computes the URL, the SHA-256 sum and the size of each asset and writes them to the manifest. The second regenerates the baked registry of raven-toolbox from the manifest. The third publishes the manifest as a new snapshot. The number of the snapshot is the next number after the newest `manifest-v` release in `toolbox-assets`. +The first command computes the URL, the SHA-256 sum and the size of each asset and writes them to the manifest. It replaces the whole entry of the dataset, so `--description`, `--license` and `--source` repeat the values of the existing entry. The second regenerates the baked registry of raven-toolbox from the manifest. The third publishes the manifest as a new snapshot. The number of the snapshot is the next number after the newest `manifest-v` release in `toolbox-assets`. ### 6. Point the toolboxes to the new release - **raven-toolbox:** commit the changes to `data/manifest.json` and `src/raven_toolbox/data.py`, and open a pull request. raven-toolbox pins the manifest of its release, so users receive the new KEGG release with the next raven-toolbox release. The environment variable `RAVEN_PYTHON_MANIFEST` points an installed raven-toolbox to a newer manifest. -- **RAVEN:** write the new release name in the first line of `src/reconstruction/kegg/KEGG_VERSION.md`, which `keggDataVersion` reads, and change the manifest URL in `fetchToolboxAsset` to the new snapshot. RAVEN reads the manifest on each call. +- **RAVEN:** write the new release name in the first line of `src/reconstruction/kegg/KEGG_VERSION.md`, which `keggDataVersion` reads, and change the manifest URL in `fetchToolboxAsset` to the new snapshot. RAVEN reads the manifest once per session; the header comment of `fetchToolboxAsset.m` names the snapshot as well. - **raven-docs:** update the sizes in [Download data and binaries](data-and-binaries.md), and rerun the examples of [Reconstruction from KEGG](../guide/kegg.md) with the new release. That page quotes the release, the counts of the drafts and the timings. A new KEGG release changes the drafts. Adopt it in a release of the toolboxes after a review of the changes in the drafts of a few organisms against the previous release. ## Format of the assets -The tables are gzipped TSV and the reference model is gzipped YAML. The assets use gzip throughout, including the large `organism_gene_ko` table, because MATLAB reads gzip with its built-in `gunzip` and has no `xz` dependency. Both toolboxes share the files, so a format that only Python reads cheaply is excluded. The larger size is a cost of a download once per release. +The tables are gzipped TSV and the reference model is gzipped YAML. The assets use gzip throughout, including the large `organism_gene_ko` table, because MATLAB reads gzip with its built-in `gunzip` and has no `xz` dependency. Both toolboxes share the files, so a format that MATLAB reads at a high cost is excluded. The files are larger, and a user downloads them once per release. The table `organism_gene_ko` is sorted by `(organism, gene)`. The order places the gene identifiers of one organism next to each other, compresses better and matches the by-organism queries of both toolboxes. The sort is an external merge sort bounded by `chunk_rows`, so it stays within memory at the scale of KEGG. -Parquet would be smaller and typed, and MATLAB reads it natively since R2019a, but it needs the package `pyarrow` in Python. It becomes a candidate if the load time and not the size is the bottleneck, or if either toolbox starts to read columns repeatedly. SQLite is excluded, because MATLAB reads it only with the Database Toolbox, which removes the property of the same files in both languages without extra dependencies. +Parquet would be smaller and typed, and MATLAB reads it natively since R2019a, but it needs the package `pyarrow` in Python. It becomes a candidate if the load time is the bottleneck and the size is not, or if either toolbox starts to read columns repeatedly. SQLite is excluded, because MATLAB needs the Database Toolbox to read it, which removes the property of the same files in both languages without extra dependencies. ## Run the build on a cluster diff --git a/docs/yaml-format.md b/docs/yaml-format.md index 0c9b4a2..2d4c9d3 100644 --- a/docs/yaml-format.md +++ b/docs/yaml-format.md @@ -253,11 +253,11 @@ Enzyme-constrained models carry the EC layer in three additional top-level keys: The sections map to `model.ec` in RAVEN MATLAB and to `raven_toolbox.io.ec_data.EcData` (attached as `model.ec`) in raven-toolbox. cobrapy ignores them. -An ec reaction without enzymes is written with no `enzymes` entry, as GECKO 4 and geckopy 4.0.0b2 write it. GECKO and geckopy read a missing entry as a reaction without enzymes. They also read `- enzymes: !!omap` with nothing under the tag and `enzymes: !!omap []`, the forms that earlier releases wrote, as a reaction without enzymes. +An ec reaction without enzymes is written with no `enzymes` entry, as GECKO 4 and geckopy 4.0.0b2 write it. GECKO and geckopy read a missing entry as a reaction without enzymes, and they read `- enzymes: !!omap` with nothing under the tag and `enzymes: !!omap []` in the same way. -An ecModel file records its format and its writer in `metaData`: `gecko_format` is `4`, and `generator` names the release that wrote the file. `loadEcModel` of GECKO and `load_ec_model` of geckopy raise an error for a format above 4. GECKO upgrades an ecModel whose protein usage and pool reactions run in the GECKO 3 direction when it loads the file, with a warning. +An ecModel file records its format and its writer in `metaData`: `gecko_format` is `4`, and `generator` names the release that wrote the file. `loadEcModel` of GECKO and `load_ec_model` of geckopy raise an error for a format above 4. Both upgrade an ecModel whose protein usage and pool reactions run in the GECKO 3 direction, in memory and with a warning; the file stays as it is. -`load_ec_model` of geckopy reads a value in double quotes that contains double quotes, as RAVEN 2 wrote the notes of some ecModels, with the outer pair of quotes removed and the inner quotes kept. +A value in double quotes that contains double quotes is read with the outer pair of quotes removed and the inner quotes kept. RAVEN 2 ecModels carry the notes of some reactions in this form, and `read_yaml_model` of raven-toolbox, `readYAMLmodel` of RAVEN and `load_ec_model` of geckopy read it. The spelling `geckoLight` inside `metaData` is also accepted on reading. diff --git a/scripts/known_names.txt b/scripts/known_names.txt index d214279..a0cbfcf 100644 --- a/scripts/known_names.txt +++ b/scripts/known_names.txt @@ -487,3 +487,4 @@ chunk_rows # parameter of the sort of the organism_gene_ko tab loadEcModel # GECKO function load_ec_model # geckopy function gecko_format # key of the metaData of an ecModel file +artefact_dir # parameter of get_kegg_model_for_organism_from_artefacts