Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/guide/homology.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@ draft: 37 rxns, 49 mets, 54 genes
:::
::::

The transfer works reaction by reaction. For each reaction of the template, the genes in its gene rule are looked up in the hit table. A template gene is replaced by its accepted counterpart in the new organism, and the reaction is carried over when the rule can be satisfied after the substitution. A reaction that needs two subunits is kept with a placeholder gene (`OLD_<model>_<gene>`) for the missing subunit under the default `complexPolicy` (`complex_policy` in Python) of `flag`, and the policy `drop` removes it. A reaction with two isozymes is kept when either has a counterpart. The metabolites that a carried reaction needs are carried with it. The draft has 49 metabolites against 52 in the template, because the three metabolites that occur in dropped reactions alone are not created.
The transfer works reaction by reaction. For each reaction of the template, the genes in its gene rule are looked up in the hit table. A template gene is replaced by its accepted counterpart in the new organism, and the reaction is carried over when the rule can be satisfied after the substitution. A reaction that needs two subunits is kept with a placeholder gene (`OLD_<model>_<gene>`) for the missing subunit under the default `complexPolicy` (`complex_policy` in Python) of `flag`, and the policy `drop` removes each complex that has a missing subunit and removes the reaction when no complex remains. The rule `(A and B) or (C and D)` without a counterpart of `B` gives `C and D` under `drop`. A reaction with two isozymes is kept when either has a counterpart. The metabolites that a carried reaction needs are carried with it. The draft has 49 metabolites against 52 in the template, because the three metabolites that occur in dropped reactions alone are not created.

The line `Standardizing grRules` shows RAVEN rewriting the rules of the template in a canonical form before the substitution. One rule written in two ways is then handled identically.

Expand Down
32 changes: 18 additions & 14 deletions docs/installation/updating-binaries.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,41 +2,41 @@

# Update the binary bundles

The BLAST+, DIAMOND and HMMER executables are published as one ZIP file per platform in [`toolbox-assets`](https://github.com/CompMetEng/toolbox-assets), and raven-toolbox resolves them at run time through `raven_toolbox.binaries`. [Download data and binaries](data-and-binaries.md) describes the resolution order, the `RAVEN_PYTHON_*` overrides and the `raven-toolbox-binaries` command for users. The sections below describe what a bundle contains and how the package maintainers publish a new one. Release tags are immutable, so a new version of a tool becomes a new tag, and every other tag stays as it is.
The BLAST+, DIAMOND and HMMER executables are published as one ZIP file per platform in [`toolbox-assets`](https://github.com/CompMetEng/toolbox-assets), and raven-toolbox resolves them at run time through `raven_toolbox.binaries`. [Download data and binaries](data-and-binaries.md) describes the resolution order, the `RAVEN_PYTHON_*` overrides and the `raven-toolbox-binaries` command for users. A new version of a tool is a new release tag, and every other tag stays as it is.

The documentation build does not run these commands.
The documentation build does not run the commands below.

## Contents of a bundle

| Bundle | Included | Left out |
|---|---|---|
| `diamond` | `diamond` | nothing; it is one static binary |
| `blast` | `blastp`, `makeblastdb` | `blastn`, `tblastn`, `psiblast`, `rpsblast`, `blast_formatter`, the `*_vdb` tools, `doc/`, `ChangeLog`, `README` and the other tools of the suite |
| `diamond` | `diamond` | nothing; the bundle is one executable |
| `blast` | `blastp`, `makeblastdb` | the other executables and the documentation of the suite |
| `hmmer` | `hmmsearch` | the rest of the suite |

Both toolboxes invoke only `makeblastdb`, `blastp`, and the `makedb` and `blastp` subcommands of DIAMOND, so the same minimal set serves both. For BLAST+ the minimal set reduces the size of the bundle from hundreds of megabytes to a few.
A bundle holds the executables that the toolboxes invoke for a user: `makeblastdb` and `blastp` of BLAST+, the `makedb` and `blastp` subcommands of DIAMOND, and `hmmsearch` of HMMER. The build of the KEGG libraries needs `hmmbuild`, MAFFT and CD-HIT as well; they are not bundled, and [Update the KEGG assets](updating-kegg-assets.md) lists them. The published BLAST+ ZIP files are 13 to 35 MB, depending on the platform.

## File name and layout

A ZIP file is named `<bundle>-<version>-<os>-<arch>.zip`. It is flat: the executables lie at the root beside the upstream `LICENSE`, without a nested `bin/` folder. A Windows ZIP carries the `.exe` files and the runtime DLLs that the build needs at the root, `nghttp2.dll` for BLAST+ and `cygwin1.dll` for HMMER, because the extraction is flat and the executable is looked up as `<name>.exe`.
A ZIP file is named `<bundle>-<version>-<os>-<arch>.zip`. It is flat: the executables lie at the root beside the upstream `LICENSE`, without a nested `bin/` folder. A Windows ZIP carries the `.exe` files and the runtime DLLs of the executables at the root, `nghttp2.dll` for BLAST+ and `cygwin1.dll` for HMMER, because the extraction is flat and the executable is looked up as `<name>.exe`.

## Platforms and licences

The bundles are built for `linux-x86_64` first, then for `macos-arm64`, `macos-x86_64`, `linux-arm64` and `windows-x86_64`. A platform without a bundle is a supported state: `ensure_binary` raises an error that names the conda package to install.
`scripts/build_binary_bundles.py` builds the bundles for `linux-x86_64`, `macos-arm64` and `windows-x86_64`. The `macos-arm64` bundle serves Apple Silicon. On a platform without a bundle, `ensure_binary` raises an error that gives the `conda install -c bioconda` command for the executable.

| Tool | Licence | Requirement for the ZIP |
|---|---|---|
| BLAST+ | produced by NCBI, public domain | the upstream `LICENSE`, for provenance |
| DIAMOND | GPL-3.0 | the licence text, the unmodified binary, and a separate asset outside the wheel |
| DIAMOND | GPL-3.0 | the licence text, the unmodified binary, and a separate asset that is not part of the raven-toolbox package |
| HMMER | BSD-3-Clause | the upstream `LICENSE` |

The release body or a provenance file records, for each asset, the upstream URL, the upstream version, the upstream checksum and the published SHA-256 sum.
`PROVENANCE-binaries.txt` in the repository `toolbox-assets` records the source of the executables. It holds the content of the `PROVENANCE.txt` that the build script writes, and the publish command does not upload it.

## HMMER on Windows

No native Windows build of HMMER 3.4 exists, because upstream targets POSIX systems. The published `windows-x86_64` asset is HMMER 3.3.2, repackaged from RAVEN v2.10.5, the last release that bundles a Cygwin-compiled Windows build. The file `cygwin1.dll` ships beside the `.exe`.

Searching profiles that HMMER 3.4 built with HMMER 3.3.2 is sound for the published libraries. They are concatenated ASCII files in the `HMMER3/f` format, which is unchanged from HMMER 3.1 to 3.4, and HMMER 3.4 is a maintenance release over 3.3.2 without a change of the protein scoring model. The bit scores and the calibrated KEGG cut-offs therefore carry over. This holds while the libraries stay in ASCII format.
The libraries that HMMER 3.4 builds are concatenated ASCII files in the `HMMER3/f` format, which is unchanged from HMMER 3.1 to 3.4, and HMMER 3.4 is a maintenance release over 3.3.2 without a change of the protein scoring model. HMMER 3.3.2 therefore gives the same bit scores on these libraries, and the calibrated KEGG cut-offs carry over. This holds while the libraries stay in ASCII format.

## Publish a bundle

Expand All @@ -47,15 +47,19 @@ python scripts/build_binary_bundles.py
python scripts/publish_to_toolbox_assets.py binaries --dir dist/binaries
python scripts/make_registry_snippet.py manifest --manifest data/manifest.json \
--target binary --bundle blast --version 2.17.0 --provides blastp makeblastdb \
--dir dist/binaries --tag blast-2.17.0
--dir dist/binaries --tag blast-2.17.0 --description "<text>" --license "<licence>"
python scripts/make_registry_snippet.py sync
```

The first command builds the ZIP files from the vetted binaries in the `software/` folder of RAVEN into `dist/binaries`, with checksums and provenance. The second creates the release tags in `toolbox-assets` and skips the assets that exist. The third computes the checksums with the helper that `ensure_binary` verifies with, and writes the entry of the bundle to `data/manifest.json`. A bundle with a licence that needs a statement takes `--license`, for example `--license GPL-3.0-only` for DIAMOND. The fourth regenerates the baked registries of raven-toolbox from the manifest.
The first command downloads the executables from the RAVEN release v2.10.5 and from a pinned commit of RAVEN on GitHub, and builds the ZIP files in `dist/binaries` with checksums. The second creates the release tags in `toolbox-assets` and skips the assets that exist. The third computes the checksums with the helper that `ensure_binary` verifies with, and writes the entry of the bundle to `data/manifest.json`. It runs once for each bundle, and it replaces the whole entry of the bundle, so `--description` and `--license` repeat the values of the existing entry (`GPL-3.0-only` for DIAMOND). It takes the ZIP files whose names start with `<bundle>-<version>-`. The entry of `hmmer` in the published manifest lists the Windows file of version 3.3.2 beside the files of version 3.4.0. That platform is added to `data/manifest.json` by hand, with the URL and the checksum of the file. The fourth regenerates the baked registries of raven-toolbox from the manifest.

`data/manifest.json` is the single source of truth, and the registries `_DATA_REGISTRY` in `src/raven_toolbox/data.py` and `_REGISTRY` in `src/raven_toolbox/binaries.py` are never edited by hand. The manifest is published as the next `manifest-v<N>` snapshot, as step 5 of [Update the KEGG assets](updating-kegg-assets.md) describes.
`data/manifest.json` is the source of truth, and the registries `_DATA_REGISTRY` in `src/raven_toolbox/data.py` and `_REGISTRY` in `src/raven_toolbox/binaries.py` are never edited by hand. The manifest is published as the next `manifest-v<N>` snapshot, as step 5 of [Update the KEGG assets](updating-kegg-assets.md) describes.

A new tool needs no new provisioning code. A bundle entry with the right `provides` list and ZIP files that follow the layout above are enough for the wrappers to resolve the tool through `ensure_binary`.
## A new version or a new tool

- **New version of a tool:** the versions, the source references and the `PROVIDES` table are values in `scripts/build_binary_bundles.py` (`SPECS`, `RAVEN3`, `V2105`, `PROVIDES`). `downloadRavenBinaries.m` of RAVEN names the tags and the asset names and needs the new tag.
- **New tool:** an entry in `_ENV_VARS` of `raven_toolbox.binaries` adds the environment variable that overrides the executable, and an entry in `BINARY_SETS` adds the executable to the downloads of the command `raven-toolbox-binaries`.
- **Both toolboxes** read the new manifest snapshot after the update: the manifest URL in `fetchToolboxAsset.m` and the baked registry of raven-toolbox.

## See also

Expand Down
Loading
Loading