From fc16ea8630b36959c86652da6a8e2a9ece3f222f Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Thu, 5 Nov 2026 12:44:20 +0100 Subject: [PATCH 1/5] Describe the drop policy of the homology draft per complex The policy drop removes each complex with a missing subunit and the reaction when no complex remains. --- docs/guide/homology.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guide/homology.md b/docs/guide/homology.md index 4a29186..1942a8f 100644 --- a/docs/guide/homology.md +++ b/docs/guide/homology.md @@ -145,7 +145,7 @@ draft: 37 rxns, 49 mets, 54 genes ::: :::: -The transfer works reaction by reaction. For each reaction of the template, the genes in its gene rule are looked up in the hit table. A template gene is replaced by its accepted counterpart in the new organism, and the reaction is carried over when the rule can be satisfied after the substitution. A reaction that needs two subunits is kept with a placeholder gene (`OLD__`) for the missing subunit under the default `complexPolicy` (`complex_policy` in Python) of `flag`, and the policy `drop` removes it. A reaction with two isozymes is kept when either has a counterpart. The metabolites that a carried reaction needs are carried with it. The draft has 49 metabolites against 52 in the template, because the three metabolites that occur in dropped reactions alone are not created. +The transfer works reaction by reaction. For each reaction of the template, the genes in its gene rule are looked up in the hit table. A template gene is replaced by its accepted counterpart in the new organism, and the reaction is carried over when the rule can be satisfied after the substitution. A reaction that needs two subunits is kept with a placeholder gene (`OLD__`) for the missing subunit under the default `complexPolicy` (`complex_policy` in Python) of `flag`, and the policy `drop` removes each complex that has a missing subunit and removes the reaction when no complex remains. The rule `(A and B) or (C and D)` without a counterpart of `B` gives `C and D` under `drop`. A reaction with two isozymes is kept when either has a counterpart. The metabolites that a carried reaction needs are carried with it. The draft has 49 metabolites against 52 in the template, because the three metabolites that occur in dropped reactions alone are not created. The line `Standardizing grRules` shows RAVEN rewriting the rules of the template in a canonical form before the substitution. One rule written in two ways is then handled identically. From eea87ab4077997aa5ff77a8a5507d36f20c335a6 Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Fri, 6 Nov 2026 15:24:00 +0100 Subject: [PATCH 2/5] Rewrite the restriction words in the maintainer pages --- docs/installation/updating-binaries.md | 2 +- docs/installation/updating-kegg-assets.md | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/installation/updating-binaries.md b/docs/installation/updating-binaries.md index 326d3b5..ee30c8b 100644 --- a/docs/installation/updating-binaries.md +++ b/docs/installation/updating-binaries.md @@ -14,7 +14,7 @@ The documentation build does not run these commands. | `blast` | `blastp`, `makeblastdb` | `blastn`, `tblastn`, `psiblast`, `rpsblast`, `blast_formatter`, the `*_vdb` tools, `doc/`, `ChangeLog`, `README` and the other tools of the suite | | `hmmer` | `hmmsearch` | the rest of the suite | -Both toolboxes invoke only `makeblastdb`, `blastp`, and the `makedb` and `blastp` subcommands of DIAMOND, so the same minimal set serves both. For BLAST+ the minimal set reduces the size of the bundle from hundreds of megabytes to a few. +Both toolboxes invoke `makeblastdb`, `blastp`, and the `makedb` and `blastp` subcommands of DIAMOND, and no other tool, so the same minimal set serves both. For BLAST+ the minimal set reduces the size of the bundle from hundreds of megabytes to a few. ## File name and layout diff --git a/docs/installation/updating-kegg-assets.md b/docs/installation/updating-kegg-assets.md index 13991df..7ffd7de 100644 --- a/docs/installation/updating-kegg-assets.md +++ b/docs/installation/updating-kegg-assets.md @@ -78,7 +78,7 @@ python scripts/publish_to_toolbox_assets.py --dry-run release --tag kegg120 --di python scripts/publish_to_toolbox_assets.py release --tag kegg120 --dir artefacts ``` -The first command prints the `gh` calls without running them. The second creates the release `kegg120` in `toolbox-assets` and uploads the four assets. A rerun uploads only the assets that are missing. +The first command prints the `gh` calls without running them. The second creates the release `kegg120` in `toolbox-assets` and uploads the four assets. A rerun uploads the assets that are missing and leaves the others as they are. ### 5. Update the manifest and the registry @@ -103,11 +103,11 @@ A new KEGG release changes the drafts. Adopt it in a release of the toolboxes af ## Format of the assets -The tables are gzipped TSV and the reference model is gzipped YAML. The assets use gzip throughout, including the large `organism_gene_ko` table, because MATLAB reads gzip with its built-in `gunzip` and has no `xz` dependency. Both toolboxes share the files, so a format that only Python reads cheaply is excluded. The larger size is a cost of a download once per release. +The tables are gzipped TSV and the reference model is gzipped YAML. The assets use gzip throughout, including the large `organism_gene_ko` table, because MATLAB reads gzip with its built-in `gunzip` and has no `xz` dependency. Both toolboxes share the files, so a format that is cheap to read in Python alone is excluded. The larger size is a cost of a download once per release. The table `organism_gene_ko` is sorted by `(organism, gene)`. The order places the gene identifiers of one organism next to each other, compresses better and matches the by-organism queries of both toolboxes. The sort is an external merge sort bounded by `chunk_rows`, so it stays within memory at the scale of KEGG. -Parquet would be smaller and typed, and MATLAB reads it natively since R2019a, but it needs the package `pyarrow` in Python. It becomes a candidate if the load time and not the size is the bottleneck, or if either toolbox starts to read columns repeatedly. SQLite is excluded, because MATLAB reads it only with the Database Toolbox, which removes the property of the same files in both languages without extra dependencies. +Parquet would be smaller and typed, and MATLAB reads it natively since R2019a, but it needs the package `pyarrow` in Python. It becomes a candidate if the load time is the bottleneck and the size is not, or if either toolbox starts to read columns repeatedly. SQLite is excluded, because MATLAB reads it with the Database Toolbox alone, which removes the property of the same files in both languages without extra dependencies. ## Run the build on a cluster From 6dac89d17a2562ed4dcbf613da8d2237fc5d8f89 Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Fri, 6 Nov 2026 15:38:33 +0100 Subject: [PATCH 3/5] Correct the maintainer pages and the ecModel YAML paragraphs The binaries page names the bundled executables, the platforms that build_binary_bundles.py builds, the files that record the provenance, and the edits that a new version or a new tool needs. The KEGG page gives the arguments of the check functions and says that RAVEN reads the manifest once per session. The YAML paragraphs name the readers and state that the upgrade is in memory. --- docs/installation/updating-binaries.md | 28 +++++++++++++---------- docs/installation/updating-kegg-assets.md | 20 ++++++++-------- docs/yaml-format.md | 6 ++--- scripts/known_names.txt | 1 + 4 files changed, 30 insertions(+), 25 deletions(-) diff --git a/docs/installation/updating-binaries.md b/docs/installation/updating-binaries.md index ee30c8b..d935218 100644 --- a/docs/installation/updating-binaries.md +++ b/docs/installation/updating-binaries.md @@ -2,9 +2,9 @@ # Update the binary bundles -The BLAST+, DIAMOND and HMMER executables are published as one ZIP file per platform in [`toolbox-assets`](https://github.com/CompMetEng/toolbox-assets), and raven-toolbox resolves them at run time through `raven_toolbox.binaries`. [Download data and binaries](data-and-binaries.md) describes the resolution order, the `RAVEN_PYTHON_*` overrides and the `raven-toolbox-binaries` command for users. The sections below describe what a bundle contains and how the package maintainers publish a new one. Release tags are immutable, so a new version of a tool becomes a new tag, and every other tag stays as it is. +The BLAST+, DIAMOND and HMMER executables are published as one ZIP file per platform in [`toolbox-assets`](https://github.com/CompMetEng/toolbox-assets), and raven-toolbox resolves them at run time through `raven_toolbox.binaries`. [Download data and binaries](data-and-binaries.md) describes the resolution order, the `RAVEN_PYTHON_*` overrides and the `raven-toolbox-binaries` command for users. A new version of a tool is a new release tag, and every other tag stays as it is. -The documentation build does not run these commands. +The documentation build does not run the commands below. ## Contents of a bundle @@ -14,29 +14,29 @@ The documentation build does not run these commands. | `blast` | `blastp`, `makeblastdb` | `blastn`, `tblastn`, `psiblast`, `rpsblast`, `blast_formatter`, the `*_vdb` tools, `doc/`, `ChangeLog`, `README` and the other tools of the suite | | `hmmer` | `hmmsearch` | the rest of the suite | -Both toolboxes invoke `makeblastdb`, `blastp`, and the `makedb` and `blastp` subcommands of DIAMOND, and no other tool, so the same minimal set serves both. For BLAST+ the minimal set reduces the size of the bundle from hundreds of megabytes to a few. +A bundle holds the executables that the toolboxes invoke for a user: `makeblastdb` and `blastp` of BLAST+, the `makedb` and `blastp` subcommands of DIAMOND, and `hmmsearch` of HMMER. The build of the KEGG libraries needs `hmmbuild`, MAFFT and CD-HIT as well; they are not bundled, and [Update the KEGG assets](updating-kegg-assets.md) lists them. The published BLAST+ ZIP files are 13 to 35 MB, depending on the platform. ## File name and layout -A ZIP file is named `---.zip`. It is flat: the executables lie at the root beside the upstream `LICENSE`, without a nested `bin/` folder. A Windows ZIP carries the `.exe` files and the runtime DLLs that the build needs at the root, `nghttp2.dll` for BLAST+ and `cygwin1.dll` for HMMER, because the extraction is flat and the executable is looked up as `.exe`. +A ZIP file is named `---.zip`. It is flat: the executables lie at the root beside the upstream `LICENSE`, without a nested `bin/` folder. A Windows ZIP carries the `.exe` files and the runtime DLLs of the executables at the root, `nghttp2.dll` for BLAST+ and `cygwin1.dll` for HMMER, because the extraction is flat and the executable is looked up as `.exe`. ## Platforms and licences -The bundles are built for `linux-x86_64` first, then for `macos-arm64`, `macos-x86_64`, `linux-arm64` and `windows-x86_64`. A platform without a bundle is a supported state: `ensure_binary` raises an error that names the conda package to install. +`scripts/build_binary_bundles.py` builds the bundles for `linux-x86_64`, `macos-arm64` and `windows-x86_64`. The `macos-arm64` bundle serves Apple Silicon. On a platform without a bundle, `ensure_binary` raises an error that gives the `conda install -c bioconda` command for the executable. | Tool | Licence | Requirement for the ZIP | |---|---|---| | BLAST+ | produced by NCBI, public domain | the upstream `LICENSE`, for provenance | -| DIAMOND | GPL-3.0 | the licence text, the unmodified binary, and a separate asset outside the wheel | +| DIAMOND | GPL-3.0 | the licence text, the unmodified binary, and a separate asset that is not part of the raven-toolbox package | | HMMER | BSD-3-Clause | the upstream `LICENSE` | -The release body or a provenance file records, for each asset, the upstream URL, the upstream version, the upstream checksum and the published SHA-256 sum. +`PROVENANCE-binaries.txt` in the repository `toolbox-assets` records the source of the executables. ## HMMER on Windows No native Windows build of HMMER 3.4 exists, because upstream targets POSIX systems. The published `windows-x86_64` asset is HMMER 3.3.2, repackaged from RAVEN v2.10.5, the last release that bundles a Cygwin-compiled Windows build. The file `cygwin1.dll` ships beside the `.exe`. -Searching profiles that HMMER 3.4 built with HMMER 3.3.2 is sound for the published libraries. They are concatenated ASCII files in the `HMMER3/f` format, which is unchanged from HMMER 3.1 to 3.4, and HMMER 3.4 is a maintenance release over 3.3.2 without a change of the protein scoring model. The bit scores and the calibrated KEGG cut-offs therefore carry over. This holds while the libraries stay in ASCII format. +The libraries that HMMER 3.4 builds are concatenated ASCII files in the `HMMER3/f` format, which is unchanged from HMMER 3.1 to 3.4, and HMMER 3.4 is a maintenance release over 3.3.2 without a change of the protein scoring model. HMMER 3.3.2 therefore gives the same bit scores on these libraries, and the calibrated KEGG cut-offs carry over. This holds while the libraries stay in ASCII format. ## Publish a bundle @@ -47,15 +47,19 @@ python scripts/build_binary_bundles.py python scripts/publish_to_toolbox_assets.py binaries --dir dist/binaries python scripts/make_registry_snippet.py manifest --manifest data/manifest.json \ --target binary --bundle blast --version 2.17.0 --provides blastp makeblastdb \ - --dir dist/binaries --tag blast-2.17.0 + --dir dist/binaries --tag blast-2.17.0 --description "" --license "" python scripts/make_registry_snippet.py sync ``` -The first command builds the ZIP files from the vetted binaries in the `software/` folder of RAVEN into `dist/binaries`, with checksums and provenance. The second creates the release tags in `toolbox-assets` and skips the assets that exist. The third computes the checksums with the helper that `ensure_binary` verifies with, and writes the entry of the bundle to `data/manifest.json`. A bundle with a licence that needs a statement takes `--license`, for example `--license GPL-3.0-only` for DIAMOND. The fourth regenerates the baked registries of raven-toolbox from the manifest. +The first command downloads the executables from the RAVEN release v2.10.5 and from a pinned commit of RAVEN on GitHub, and builds the ZIP files in `dist/binaries` with checksums. The second creates the release tags in `toolbox-assets` and skips the assets that exist. The third computes the checksums with the helper that `ensure_binary` verifies with, and writes the entry of the bundle to `data/manifest.json`. It runs once for each bundle, and it replaces the whole entry of the bundle, so `--description` and `--license` repeat the values of the existing entry (`GPL-3.0-only` for DIAMOND). It takes the ZIP files whose names start with `--`, so a file with another version, such as the Windows file of HMMER 3.3.2, is not part of an entry for HMMER 3.4.0. The fourth regenerates the baked registries of raven-toolbox from the manifest. -`data/manifest.json` is the single source of truth, and the registries `_DATA_REGISTRY` in `src/raven_toolbox/data.py` and `_REGISTRY` in `src/raven_toolbox/binaries.py` are never edited by hand. The manifest is published as the next `manifest-v` snapshot, as step 5 of [Update the KEGG assets](updating-kegg-assets.md) describes. +`data/manifest.json` is the source of truth, and the registries `_DATA_REGISTRY` in `src/raven_toolbox/data.py` and `_REGISTRY` in `src/raven_toolbox/binaries.py` are never edited by hand. The manifest is published as the next `manifest-v` snapshot, as step 5 of [Update the KEGG assets](updating-kegg-assets.md) describes. -A new tool needs no new provisioning code. A bundle entry with the right `provides` list and ZIP files that follow the layout above are enough for the wrappers to resolve the tool through `ensure_binary`. +## A new version or a new tool + +- **New version of a tool:** the versions, the source references and the `PROVIDES` table are values in `scripts/build_binary_bundles.py` (`SPECS`, `RAVEN3`, `V2105`, `PROVIDES`). `downloadRavenBinaries.m` of RAVEN names the tags and the asset names and needs the new tag. +- **New tool:** `_ENV_VARS` in `raven_toolbox.binaries` names the environment variable that overrides the executable, and `BINARY_SETS` lists the executables that the command `raven-toolbox-binaries` downloads. Both need an entry for the tool. +- **Both toolboxes** read the new manifest snapshot after the update: the manifest URL in `fetchToolboxAsset.m` and the baked registry of raven-toolbox. ## See also diff --git a/docs/installation/updating-kegg-assets.md b/docs/installation/updating-kegg-assets.md index 7ffd7de..420edcb 100644 --- a/docs/installation/updating-kegg-assets.md +++ b/docs/installation/updating-kegg-assets.md @@ -4,7 +4,7 @@ The KEGG reference data and the libraries of profile hidden Markov models (HMMs) are published as assets of one KEGG release in [`toolbox-assets`](https://github.com/CompMetEng/toolbox-assets). Both toolboxes download them, as described in [Download data and binaries](data-and-binaries.md). A new KEGG release needs a new set of assets. The package maintainers do this when they adopt a new KEGG release, and users of the toolboxes never do. -The documentation build does not run these commands. The download needs a KEGG subscription, and the build of the libraries takes hours. +The documentation build does not run the commands below. The download needs a KEGG subscription, and the build of the libraries for `kegg120` took about 50 core-hours. ## Assets of one release @@ -24,7 +24,7 @@ Release tags are immutable. A release that is published under a name is not chan ## Requirements - A subscription to the bulk download of KEGG (`ftp.kegg.net`), which comes with a user name and a password. -- raven-toolbox from a checkout of its repository, installed as described in [Installing RAVEN](python.md). The commands run from the root of that checkout. +- raven-toolbox from a checkout of its repository, installed as described in [raven-toolbox (Python)](python.md). The commands run from the root of that checkout. - HMMER (`hmmbuild`), MAFFT and CD-HIT, on the `PATH` or pointed at with the environment variables `RAVEN_PYTHON_HMMBUILD`, `RAVEN_PYTHON_MAFFT` and `RAVEN_PYTHON_CDHIT`. MAFFT and CD-HIT have no native Windows build, so the build runs on Linux, macOS or inside WSL2. Keep the whole toolchain inside WSL2, because raven-toolbox calls the resolved executable directly and does not translate paths across the boundary. - The GitHub CLI `gh`, authenticated with write access to `toolbox-assets`. - Disk space for the dump and for the per-KO working files of the build, and enough memory for the largest alignments of MAFFT. MAFFT switches to a memory-light method for an alignment that is predicted to need more memory than the host has, and that method approximates the guide tree. @@ -49,7 +49,7 @@ from raven_toolbox.reconstruction.kegg import download_kegg_dump download_kegg_dump("keggdb", force=True, progress=False) ``` -The function fetches the reaction, compound, glycan and KO archives, the proteomes and the taxonomy file, extracts them and arranges the flat layout that the parser expects. Without `force=True`, it skips files that exist, so use it for a new release in a directory that holds an older dump. +The function fetches the reaction, compound, glycan and KO archives, the proteomes and the taxonomy file, extracts them and arranges the flat layout that the parser expects. Without `force=True`, the function keeps the files that exist. A new release in a directory that holds an older dump needs `force=True` to fetch the files again. ### 2. Build the assets @@ -58,7 +58,7 @@ python scripts/build_kegg_artefacts.py --keggdb keggdb --out artefacts \ --version kegg120 --hmms --threads 16 ``` -The script parses the dump into the reference model and the tables, builds the library of each domain from the sequences of the KOs, and arranges the four assets in `artefacts`. `--domains` restricts the libraries to one domain. +The script parses the dump into the reference model and the tables, builds the library of each domain from the sequences of the KOs, and arranges the four assets in `artefacts`. With `--hmms`, `--domains` restricts the libraries to the domains that follow it. The build of the libraries is the slow step. The script resumes: it skips a stage whose output exists, and it skips every KO whose profile exists in the working directory `artefacts/_hmms-`. Run the same command again after a failure. `--force` rebuilds everything, and a changed gene-to-KO table needs it, because a resumed run assumes the inputs of the first run. @@ -68,7 +68,7 @@ The working directory holds one FASTA file and one profile per KO. On a file sys - The four files of the table exist in `artefacts`, and their sizes are of the same order as the previous release (49 MB, 135 MB, 155 MB and 0.2 MB for `kegg118`). - The number of profiles in a library is the number of lines that start with `NAME` in the decompressed file. Compare it with the previous release. -- Reconstruct a draft for an organism that KEGG contains, for example *S. cerevisiae*, with `get_kegg_model_for_organism_from_artefacts`, and search its proteome against the eukaryotic library with `get_kegg_model_from_sequences_with_artefacts`. The assignment of the search should resemble the KEGG assignment of the organism. Both functions are described in [Reconstruction from KEGG](../guide/kegg.md). +- Extract `kegg_core.tar.gz` into a folder, and decompress `kegg_eukaryotes.hmm.gz`. Reconstruct a draft for an organism that KEGG contains, for example *S. cerevisiae*, with `get_kegg_model_for_organism_from_artefacts("sce", artefact_dir=)`, and search its proteome against the eukaryotic library with `get_kegg_model_from_sequences_with_artefacts(, artefact_dir=, library=, domain="eukaryotes")`. Without `artefact_dir` and `library`, the functions use the published release that the registry names. The assignment of the search should resemble the KEGG assignment of the organism. Both functions are described in [Reconstruction from KEGG](../guide/kegg.md). - Record the KEGG release and the versions of HMMER, MAFFT and CD-HIT of the build. ### 4. Publish the assets @@ -88,26 +88,26 @@ The manifest `data/manifest.json` is the single source of truth for the checksum python scripts/make_registry_snippet.py manifest --manifest data/manifest.json \ --target data --dataset kegg --version kegg120 --dir artefacts --tag kegg120 python scripts/make_registry_snippet.py sync -python scripts/publish_to_toolbox_assets.py release --tag manifest-v4 data/manifest.json +python scripts/publish_to_toolbox_assets.py release --tag manifest-v data/manifest.json ``` -The first command computes the URL, the SHA-256 sum and the size of each asset and writes them to the manifest. The second regenerates the baked registry of raven-toolbox from the manifest. The third publishes the manifest as a new snapshot. The number of the snapshot is the next number after the newest `manifest-v` release in `toolbox-assets`. +The first command computes the URL, the SHA-256 sum and the size of each asset and writes them to the manifest. It replaces the whole entry of the dataset, so `--description`, `--license` and `--source` repeat the values of the existing entry. The second regenerates the baked registry of raven-toolbox from the manifest. The third publishes the manifest as a new snapshot. The number of the snapshot is the next number after the newest `manifest-v` release in `toolbox-assets`. ### 6. Point the toolboxes to the new release - **raven-toolbox:** commit the changes to `data/manifest.json` and `src/raven_toolbox/data.py`, and open a pull request. raven-toolbox pins the manifest of its release, so users receive the new KEGG release with the next raven-toolbox release. The environment variable `RAVEN_PYTHON_MANIFEST` points an installed raven-toolbox to a newer manifest. -- **RAVEN:** write the new release name in the first line of `src/reconstruction/kegg/KEGG_VERSION.md`, which `keggDataVersion` reads, and change the manifest URL in `fetchToolboxAsset` to the new snapshot. RAVEN reads the manifest on each call. +- **RAVEN:** write the new release name in the first line of `src/reconstruction/kegg/KEGG_VERSION.md`, which `keggDataVersion` reads, and change the manifest URL in `fetchToolboxAsset` to the new snapshot. RAVEN reads the manifest once per session; the header comment of `fetchToolboxAsset.m` names the snapshot as well. - **raven-docs:** update the sizes in [Download data and binaries](data-and-binaries.md), and rerun the examples of [Reconstruction from KEGG](../guide/kegg.md) with the new release. That page quotes the release, the counts of the drafts and the timings. A new KEGG release changes the drafts. Adopt it in a release of the toolboxes after a review of the changes in the drafts of a few organisms against the previous release. ## Format of the assets -The tables are gzipped TSV and the reference model is gzipped YAML. The assets use gzip throughout, including the large `organism_gene_ko` table, because MATLAB reads gzip with its built-in `gunzip` and has no `xz` dependency. Both toolboxes share the files, so a format that is cheap to read in Python alone is excluded. The larger size is a cost of a download once per release. +The tables are gzipped TSV and the reference model is gzipped YAML. The assets use gzip throughout, including the large `organism_gene_ko` table, because MATLAB reads gzip with its built-in `gunzip` and has no `xz` dependency. Both toolboxes share the files, so a format that is cheap to read in Python alone is excluded. The files are larger, and a user downloads them once per release. The table `organism_gene_ko` is sorted by `(organism, gene)`. The order places the gene identifiers of one organism next to each other, compresses better and matches the by-organism queries of both toolboxes. The sort is an external merge sort bounded by `chunk_rows`, so it stays within memory at the scale of KEGG. -Parquet would be smaller and typed, and MATLAB reads it natively since R2019a, but it needs the package `pyarrow` in Python. It becomes a candidate if the load time is the bottleneck and the size is not, or if either toolbox starts to read columns repeatedly. SQLite is excluded, because MATLAB reads it with the Database Toolbox alone, which removes the property of the same files in both languages without extra dependencies. +Parquet would be smaller and typed, and MATLAB reads it natively since R2019a, but it needs the package `pyarrow` in Python. It becomes a candidate if the load time is the bottleneck and the size is not, or if either toolbox starts to read columns repeatedly. SQLite is excluded, because MATLAB needs the Database Toolbox to read it, which removes the property of the same files in both languages without extra dependencies. ## Run the build on a cluster diff --git a/docs/yaml-format.md b/docs/yaml-format.md index 0c9b4a2..9508bf2 100644 --- a/docs/yaml-format.md +++ b/docs/yaml-format.md @@ -253,11 +253,11 @@ Enzyme-constrained models carry the EC layer in three additional top-level keys: The sections map to `model.ec` in RAVEN MATLAB and to `raven_toolbox.io.ec_data.EcData` (attached as `model.ec`) in raven-toolbox. cobrapy ignores them. -An ec reaction without enzymes is written with no `enzymes` entry, as GECKO 4 and geckopy 4.0.0b2 write it. GECKO and geckopy read a missing entry as a reaction without enzymes. They also read `- enzymes: !!omap` with nothing under the tag and `enzymes: !!omap []`, the forms that earlier releases wrote, as a reaction without enzymes. +An ec reaction without enzymes is written with no `enzymes` entry, as GECKO 4 and geckopy 4.0.0b2 write it. GECKO and geckopy read a missing entry as a reaction without enzymes, and they read `- enzymes: !!omap` with nothing under the tag and `enzymes: !!omap []` in the same way. -An ecModel file records its format and its writer in `metaData`: `gecko_format` is `4`, and `generator` names the release that wrote the file. `loadEcModel` of GECKO and `load_ec_model` of geckopy raise an error for a format above 4. GECKO upgrades an ecModel whose protein usage and pool reactions run in the GECKO 3 direction when it loads the file, with a warning. +An ecModel file records its format and its writer in `metaData`: `gecko_format` is `4`, and `generator` names the release that wrote the file. `loadEcModel` of GECKO and `load_ec_model` of geckopy raise an error for a format above 4. Both upgrade an ecModel whose protein usage and pool reactions run in the GECKO 3 direction, in memory and with a warning; the file stays as it is. -`load_ec_model` of geckopy reads a value in double quotes that contains double quotes, as RAVEN 2 wrote the notes of some ecModels, with the outer pair of quotes removed and the inner quotes kept. +A value in double quotes that contains double quotes is read with the outer pair of quotes removed and the inner quotes kept. RAVEN 2 wrote the notes of some ecModels in this form, and `read_yaml_model` of raven-toolbox, `readYAMLmodel` of RAVEN and `load_ec_model` of geckopy read it. The spelling `geckoLight` inside `metaData` is also accepted on reading. diff --git a/scripts/known_names.txt b/scripts/known_names.txt index d214279..a0cbfcf 100644 --- a/scripts/known_names.txt +++ b/scripts/known_names.txt @@ -487,3 +487,4 @@ chunk_rows # parameter of the sort of the organism_gene_ko tab loadEcModel # GECKO function load_ec_model # geckopy function gecko_format # key of the metaData of an ecModel file +artefact_dir # parameter of get_kegg_model_for_organism_from_artefacts From a76656470c4de125f0bd732459b79dcb05f70825 Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Fri, 6 Nov 2026 15:45:23 +0100 Subject: [PATCH 4/5] Correct the HMMER entry, the force option and the check step on the maintainer pages --- docs/installation/updating-binaries.md | 10 +++++----- docs/installation/updating-kegg-assets.md | 10 +++++----- docs/yaml-format.md | 2 +- 3 files changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/installation/updating-binaries.md b/docs/installation/updating-binaries.md index d935218..b781e12 100644 --- a/docs/installation/updating-binaries.md +++ b/docs/installation/updating-binaries.md @@ -10,8 +10,8 @@ The documentation build does not run the commands below. | Bundle | Included | Left out | |---|---|---| -| `diamond` | `diamond` | nothing; it is one static binary | -| `blast` | `blastp`, `makeblastdb` | `blastn`, `tblastn`, `psiblast`, `rpsblast`, `blast_formatter`, the `*_vdb` tools, `doc/`, `ChangeLog`, `README` and the other tools of the suite | +| `diamond` | `diamond` | nothing; the bundle is one executable | +| `blast` | `blastp`, `makeblastdb` | the other executables and the documentation of the suite | | `hmmer` | `hmmsearch` | the rest of the suite | A bundle holds the executables that the toolboxes invoke for a user: `makeblastdb` and `blastp` of BLAST+, the `makedb` and `blastp` subcommands of DIAMOND, and `hmmsearch` of HMMER. The build of the KEGG libraries needs `hmmbuild`, MAFFT and CD-HIT as well; they are not bundled, and [Update the KEGG assets](updating-kegg-assets.md) lists them. The published BLAST+ ZIP files are 13 to 35 MB, depending on the platform. @@ -30,7 +30,7 @@ A ZIP file is named `---.zip`. It is flat: the execut | DIAMOND | GPL-3.0 | the licence text, the unmodified binary, and a separate asset that is not part of the raven-toolbox package | | HMMER | BSD-3-Clause | the upstream `LICENSE` | -`PROVENANCE-binaries.txt` in the repository `toolbox-assets` records the source of the executables. +`PROVENANCE-binaries.txt` in the repository `toolbox-assets` records the source of the executables. It holds the content of the `PROVENANCE.txt` that the build script writes, and the publish command does not upload it. ## HMMER on Windows @@ -51,14 +51,14 @@ python scripts/make_registry_snippet.py manifest --manifest data/manifest.json \ python scripts/make_registry_snippet.py sync ``` -The first command downloads the executables from the RAVEN release v2.10.5 and from a pinned commit of RAVEN on GitHub, and builds the ZIP files in `dist/binaries` with checksums. The second creates the release tags in `toolbox-assets` and skips the assets that exist. The third computes the checksums with the helper that `ensure_binary` verifies with, and writes the entry of the bundle to `data/manifest.json`. It runs once for each bundle, and it replaces the whole entry of the bundle, so `--description` and `--license` repeat the values of the existing entry (`GPL-3.0-only` for DIAMOND). It takes the ZIP files whose names start with `--`, so a file with another version, such as the Windows file of HMMER 3.3.2, is not part of an entry for HMMER 3.4.0. The fourth regenerates the baked registries of raven-toolbox from the manifest. +The first command downloads the executables from the RAVEN release v2.10.5 and from a pinned commit of RAVEN on GitHub, and builds the ZIP files in `dist/binaries` with checksums. The second creates the release tags in `toolbox-assets` and skips the assets that exist. The third computes the checksums with the helper that `ensure_binary` verifies with, and writes the entry of the bundle to `data/manifest.json`. It runs once for each bundle, and it replaces the whole entry of the bundle, so `--description` and `--license` repeat the values of the existing entry (`GPL-3.0-only` for DIAMOND). It takes the ZIP files whose names start with `--`. The entry of `hmmer` in the published manifest lists the Windows file of version 3.3.2 beside the files of version 3.4.0, so that platform is added to `data/manifest.json` by hand, with the URL and the checksum of the file. The fourth regenerates the baked registries of raven-toolbox from the manifest. `data/manifest.json` is the source of truth, and the registries `_DATA_REGISTRY` in `src/raven_toolbox/data.py` and `_REGISTRY` in `src/raven_toolbox/binaries.py` are never edited by hand. The manifest is published as the next `manifest-v` snapshot, as step 5 of [Update the KEGG assets](updating-kegg-assets.md) describes. ## A new version or a new tool - **New version of a tool:** the versions, the source references and the `PROVIDES` table are values in `scripts/build_binary_bundles.py` (`SPECS`, `RAVEN3`, `V2105`, `PROVIDES`). `downloadRavenBinaries.m` of RAVEN names the tags and the asset names and needs the new tag. -- **New tool:** `_ENV_VARS` in `raven_toolbox.binaries` names the environment variable that overrides the executable, and `BINARY_SETS` lists the executables that the command `raven-toolbox-binaries` downloads. Both need an entry for the tool. +- **New tool:** an entry in `_ENV_VARS` of `raven_toolbox.binaries` adds the environment variable that overrides the executable, and an entry in `BINARY_SETS` adds the executable to the downloads of the command `raven-toolbox-binaries`. - **Both toolboxes** read the new manifest snapshot after the update: the manifest URL in `fetchToolboxAsset.m` and the baked registry of raven-toolbox. ## See also diff --git a/docs/installation/updating-kegg-assets.md b/docs/installation/updating-kegg-assets.md index 420edcb..ca01b03 100644 --- a/docs/installation/updating-kegg-assets.md +++ b/docs/installation/updating-kegg-assets.md @@ -4,7 +4,7 @@ The KEGG reference data and the libraries of profile hidden Markov models (HMMs) are published as assets of one KEGG release in [`toolbox-assets`](https://github.com/CompMetEng/toolbox-assets). Both toolboxes download them, as described in [Download data and binaries](data-and-binaries.md). A new KEGG release needs a new set of assets. The package maintainers do this when they adopt a new KEGG release, and users of the toolboxes never do. -The documentation build does not run the commands below. The download needs a KEGG subscription, and the build of the libraries for `kegg120` took about 50 core-hours. +The documentation build does not run the commands below. The download needs a KEGG subscription, and the build of the libraries is the slow step. ## Assets of one release @@ -49,7 +49,7 @@ from raven_toolbox.reconstruction.kegg import download_kegg_dump download_kegg_dump("keggdb", force=True, progress=False) ``` -The function fetches the reaction, compound, glycan and KO archives, the proteomes and the taxonomy file, extracts them and arranges the flat layout that the parser expects. Without `force=True`, the function keeps the files that exist. A new release in a directory that holds an older dump needs `force=True` to fetch the files again. +The function fetches the reaction, compound, glycan and KO archives, the proteomes and the taxonomy file, extracts them and arranges the flat layout that the parser expects. Without `force=True`, the function keeps the files that exist. It deletes the archives after it extracts them, so the file `taxonomy` is the only file of an older dump that it keeps. A new release needs `force=True` or an empty directory. ### 2. Build the assets @@ -58,7 +58,7 @@ python scripts/build_kegg_artefacts.py --keggdb keggdb --out artefacts \ --version kegg120 --hmms --threads 16 ``` -The script parses the dump into the reference model and the tables, builds the library of each domain from the sequences of the KOs, and arranges the four assets in `artefacts`. With `--hmms`, `--domains` restricts the libraries to the domains that follow it. +The script parses the dump into the reference model and the tables, builds the library of each domain from the sequences of the KOs, and arranges the four assets in `artefacts`. With `--hmms`, `--domains` restricts the libraries to the listed domains (`prokaryotes`, `eukaryotes`). The build of the libraries is the slow step. The script resumes: it skips a stage whose output exists, and it skips every KO whose profile exists in the working directory `artefacts/_hmms-`. Run the same command again after a failure. `--force` rebuilds everything, and a changed gene-to-KO table needs it, because a resumed run assumes the inputs of the first run. @@ -68,7 +68,7 @@ The working directory holds one FASTA file and one profile per KO. On a file sys - The four files of the table exist in `artefacts`, and their sizes are of the same order as the previous release (49 MB, 135 MB, 155 MB and 0.2 MB for `kegg118`). - The number of profiles in a library is the number of lines that start with `NAME` in the decompressed file. Compare it with the previous release. -- Extract `kegg_core.tar.gz` into a folder, and decompress `kegg_eukaryotes.hmm.gz`. Reconstruct a draft for an organism that KEGG contains, for example *S. cerevisiae*, with `get_kegg_model_for_organism_from_artefacts("sce", artefact_dir=)`, and search its proteome against the eukaryotic library with `get_kegg_model_from_sequences_with_artefacts(, artefact_dir=, library=, domain="eukaryotes")`. Without `artefact_dir` and `library`, the functions use the published release that the registry names. The assignment of the search should resemble the KEGG assignment of the organism. Both functions are described in [Reconstruction from KEGG](../guide/kegg.md). +- Extract `kegg_core.tar.gz` into a folder, and decompress `kegg_eukaryotes.hmm.gz`. Reconstruct a draft for an organism that KEGG contains, for example *S. cerevisiae*, with `get_kegg_model_for_organism_from_artefacts("sce", artefact_dir=)`, and search its proteome against the eukaryotic library with `get_kegg_model_from_sequences_with_artefacts(, artefact_dir=, library=)`. Without `artefact_dir` (and `library`), the functions fetch the release that the registry names. The assignment of the search should resemble the KEGG assignment of the organism. Both functions are described in [Reconstruction from KEGG](../guide/kegg.md). - Record the KEGG release and the versions of HMMER, MAFFT and CD-HIT of the build. ### 4. Publish the assets @@ -103,7 +103,7 @@ A new KEGG release changes the drafts. Adopt it in a release of the toolboxes af ## Format of the assets -The tables are gzipped TSV and the reference model is gzipped YAML. The assets use gzip throughout, including the large `organism_gene_ko` table, because MATLAB reads gzip with its built-in `gunzip` and has no `xz` dependency. Both toolboxes share the files, so a format that is cheap to read in Python alone is excluded. The files are larger, and a user downloads them once per release. +The tables are gzipped TSV and the reference model is gzipped YAML. The assets use gzip throughout, including the large `organism_gene_ko` table, because MATLAB reads gzip with its built-in `gunzip` and has no `xz` dependency. Both toolboxes share the files, so a format that MATLAB reads at a high cost is excluded. The files are larger, and a user downloads them once per release. The table `organism_gene_ko` is sorted by `(organism, gene)`. The order places the gene identifiers of one organism next to each other, compresses better and matches the by-organism queries of both toolboxes. The sort is an external merge sort bounded by `chunk_rows`, so it stays within memory at the scale of KEGG. diff --git a/docs/yaml-format.md b/docs/yaml-format.md index 9508bf2..2d4c9d3 100644 --- a/docs/yaml-format.md +++ b/docs/yaml-format.md @@ -257,7 +257,7 @@ An ec reaction without enzymes is written with no `enzymes` entry, as GECKO 4 an An ecModel file records its format and its writer in `metaData`: `gecko_format` is `4`, and `generator` names the release that wrote the file. `loadEcModel` of GECKO and `load_ec_model` of geckopy raise an error for a format above 4. Both upgrade an ecModel whose protein usage and pool reactions run in the GECKO 3 direction, in memory and with a warning; the file stays as it is. -A value in double quotes that contains double quotes is read with the outer pair of quotes removed and the inner quotes kept. RAVEN 2 wrote the notes of some ecModels in this form, and `read_yaml_model` of raven-toolbox, `readYAMLmodel` of RAVEN and `load_ec_model` of geckopy read it. +A value in double quotes that contains double quotes is read with the outer pair of quotes removed and the inner quotes kept. RAVEN 2 ecModels carry the notes of some reactions in this form, and `read_yaml_model` of raven-toolbox, `readYAMLmodel` of RAVEN and `load_ec_model` of geckopy read it. The spelling `geckoLight` inside `metaData` is also accepted on reading. From 230e7b5d9a6b637c41543bce5eccb661106d0a78 Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Fri, 6 Nov 2026 15:45:40 +0100 Subject: [PATCH 5/5] Rewrite two sentences flagged by the style checker --- docs/installation/updating-binaries.md | 2 +- docs/installation/updating-kegg-assets.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/installation/updating-binaries.md b/docs/installation/updating-binaries.md index b781e12..aed3ebe 100644 --- a/docs/installation/updating-binaries.md +++ b/docs/installation/updating-binaries.md @@ -51,7 +51,7 @@ python scripts/make_registry_snippet.py manifest --manifest data/manifest.json \ python scripts/make_registry_snippet.py sync ``` -The first command downloads the executables from the RAVEN release v2.10.5 and from a pinned commit of RAVEN on GitHub, and builds the ZIP files in `dist/binaries` with checksums. The second creates the release tags in `toolbox-assets` and skips the assets that exist. The third computes the checksums with the helper that `ensure_binary` verifies with, and writes the entry of the bundle to `data/manifest.json`. It runs once for each bundle, and it replaces the whole entry of the bundle, so `--description` and `--license` repeat the values of the existing entry (`GPL-3.0-only` for DIAMOND). It takes the ZIP files whose names start with `--`. The entry of `hmmer` in the published manifest lists the Windows file of version 3.3.2 beside the files of version 3.4.0, so that platform is added to `data/manifest.json` by hand, with the URL and the checksum of the file. The fourth regenerates the baked registries of raven-toolbox from the manifest. +The first command downloads the executables from the RAVEN release v2.10.5 and from a pinned commit of RAVEN on GitHub, and builds the ZIP files in `dist/binaries` with checksums. The second creates the release tags in `toolbox-assets` and skips the assets that exist. The third computes the checksums with the helper that `ensure_binary` verifies with, and writes the entry of the bundle to `data/manifest.json`. It runs once for each bundle, and it replaces the whole entry of the bundle, so `--description` and `--license` repeat the values of the existing entry (`GPL-3.0-only` for DIAMOND). It takes the ZIP files whose names start with `--`. The entry of `hmmer` in the published manifest lists the Windows file of version 3.3.2 beside the files of version 3.4.0. That platform is added to `data/manifest.json` by hand, with the URL and the checksum of the file. The fourth regenerates the baked registries of raven-toolbox from the manifest. `data/manifest.json` is the source of truth, and the registries `_DATA_REGISTRY` in `src/raven_toolbox/data.py` and `_REGISTRY` in `src/raven_toolbox/binaries.py` are never edited by hand. The manifest is published as the next `manifest-v` snapshot, as step 5 of [Update the KEGG assets](updating-kegg-assets.md) describes. diff --git a/docs/installation/updating-kegg-assets.md b/docs/installation/updating-kegg-assets.md index ca01b03..8a031d9 100644 --- a/docs/installation/updating-kegg-assets.md +++ b/docs/installation/updating-kegg-assets.md @@ -49,7 +49,7 @@ from raven_toolbox.reconstruction.kegg import download_kegg_dump download_kegg_dump("keggdb", force=True, progress=False) ``` -The function fetches the reaction, compound, glycan and KO archives, the proteomes and the taxonomy file, extracts them and arranges the flat layout that the parser expects. Without `force=True`, the function keeps the files that exist. It deletes the archives after it extracts them, so the file `taxonomy` is the only file of an older dump that it keeps. A new release needs `force=True` or an empty directory. +The function fetches the reaction, compound, glycan and KO archives, the proteomes and the taxonomy file, extracts them and arranges the flat layout that the parser expects. Without `force=True`, the function keeps the files that exist. It deletes the archives after it extracts them and keeps the file `taxonomy` of an older dump. A new release needs `force=True` or an empty directory. ### 2. Build the assets