diff --git a/docs/installation/data-and-binaries.md b/docs/installation/data-and-binaries.md index 64df86f..0032ed5 100644 --- a/docs/installation/data-and-binaries.md +++ b/docs/installation/data-and-binaries.md @@ -133,3 +133,4 @@ On Windows, homology reconstruction and the KEGG query path run natively, becaus - [Reconstruction from homology](../guide/homology.md): the workflow that needs BLAST+ or DIAMOND. - [RAVEN (MATLAB)](raven.md) and [raven-toolbox (Python)](python.md). - [Update the KEGG assets](updating-kegg-assets.md): how the maintainers publish the assets of a new KEGG release. +- [Update the binary bundles](updating-binaries.md): how the maintainers publish the BLAST+, DIAMOND and HMMER bundles. diff --git a/docs/installation/index.md b/docs/installation/index.md index b989fa6..92d505f 100644 --- a/docs/installation/index.md +++ b/docs/installation/index.md @@ -39,4 +39,5 @@ raven python data-and-binaries updating-kegg-assets +updating-binaries ``` diff --git a/docs/installation/updating-binaries.md b/docs/installation/updating-binaries.md new file mode 100644 index 0000000..326d3b5 --- /dev/null +++ b/docs/installation/updating-binaries.md @@ -0,0 +1,63 @@ + + +# Update the binary bundles + +The BLAST+, DIAMOND and HMMER executables are published as one ZIP file per platform in [`toolbox-assets`](https://github.com/CompMetEng/toolbox-assets), and raven-toolbox resolves them at run time through `raven_toolbox.binaries`. [Download data and binaries](data-and-binaries.md) describes the resolution order, the `RAVEN_PYTHON_*` overrides and the `raven-toolbox-binaries` command for users. The sections below describe what a bundle contains and how the package maintainers publish a new one. Release tags are immutable, so a new version of a tool becomes a new tag, and every other tag stays as it is. + +The documentation build does not run these commands. + +## Contents of a bundle + +| Bundle | Included | Left out | +|---|---|---| +| `diamond` | `diamond` | nothing; it is one static binary | +| `blast` | `blastp`, `makeblastdb` | `blastn`, `tblastn`, `psiblast`, `rpsblast`, `blast_formatter`, the `*_vdb` tools, `doc/`, `ChangeLog`, `README` and the other tools of the suite | +| `hmmer` | `hmmsearch` | the rest of the suite | + +Both toolboxes invoke only `makeblastdb`, `blastp`, and the `makedb` and `blastp` subcommands of DIAMOND, so the same minimal set serves both. For BLAST+ the minimal set reduces the size of the bundle from hundreds of megabytes to a few. + +## File name and layout + +A ZIP file is named `---.zip`. It is flat: the executables lie at the root beside the upstream `LICENSE`, without a nested `bin/` folder. A Windows ZIP carries the `.exe` files and the runtime DLLs that the build needs at the root, `nghttp2.dll` for BLAST+ and `cygwin1.dll` for HMMER, because the extraction is flat and the executable is looked up as `.exe`. + +## Platforms and licences + +The bundles are built for `linux-x86_64` first, then for `macos-arm64`, `macos-x86_64`, `linux-arm64` and `windows-x86_64`. A platform without a bundle is a supported state: `ensure_binary` raises an error that names the conda package to install. + +| Tool | Licence | Requirement for the ZIP | +|---|---|---| +| BLAST+ | produced by NCBI, public domain | the upstream `LICENSE`, for provenance | +| DIAMOND | GPL-3.0 | the licence text, the unmodified binary, and a separate asset outside the wheel | +| HMMER | BSD-3-Clause | the upstream `LICENSE` | + +The release body or a provenance file records, for each asset, the upstream URL, the upstream version, the upstream checksum and the published SHA-256 sum. + +## HMMER on Windows + +No native Windows build of HMMER 3.4 exists, because upstream targets POSIX systems. The published `windows-x86_64` asset is HMMER 3.3.2, repackaged from RAVEN v2.10.5, the last release that bundles a Cygwin-compiled Windows build. The file `cygwin1.dll` ships beside the `.exe`. + +Searching profiles that HMMER 3.4 built with HMMER 3.3.2 is sound for the published libraries. They are concatenated ASCII files in the `HMMER3/f` format, which is unchanged from HMMER 3.1 to 3.4, and HMMER 3.4 is a maintenance release over 3.3.2 without a change of the protein scoring model. The bit scores and the calibrated KEGG cut-offs therefore carry over. This holds while the libraries stay in ASCII format. + +## Publish a bundle + +Run the commands from the root of a checkout of raven-toolbox. + +```bash +python scripts/build_binary_bundles.py +python scripts/publish_to_toolbox_assets.py binaries --dir dist/binaries +python scripts/make_registry_snippet.py manifest --manifest data/manifest.json \ + --target binary --bundle blast --version 2.17.0 --provides blastp makeblastdb \ + --dir dist/binaries --tag blast-2.17.0 +python scripts/make_registry_snippet.py sync +``` + +The first command builds the ZIP files from the vetted binaries in the `software/` folder of RAVEN into `dist/binaries`, with checksums and provenance. The second creates the release tags in `toolbox-assets` and skips the assets that exist. The third computes the checksums with the helper that `ensure_binary` verifies with, and writes the entry of the bundle to `data/manifest.json`. A bundle with a licence that needs a statement takes `--license`, for example `--license GPL-3.0-only` for DIAMOND. The fourth regenerates the baked registries of raven-toolbox from the manifest. + +`data/manifest.json` is the single source of truth, and the registries `_DATA_REGISTRY` in `src/raven_toolbox/data.py` and `_REGISTRY` in `src/raven_toolbox/binaries.py` are never edited by hand. The manifest is published as the next `manifest-v` snapshot, as step 5 of [Update the KEGG assets](updating-kegg-assets.md) describes. + +A new tool needs no new provisioning code. A bundle entry with the right `provides` list and ZIP files that follow the layout above are enough for the wrappers to resolve the tool through `ensure_binary`. + +## See also + +- [Download data and binaries](data-and-binaries.md): what a user downloads, and where the files are cached. +- [Update the KEGG assets](updating-kegg-assets.md): the same publication steps for the KEGG release. diff --git a/docs/installation/updating-kegg-assets.md b/docs/installation/updating-kegg-assets.md index 25c22f8..13991df 100644 --- a/docs/installation/updating-kegg-assets.md +++ b/docs/installation/updating-kegg-assets.md @@ -35,7 +35,7 @@ The downloader reads the KEGG credentials from `~/.netrc` and takes none on the machine ftp.kegg.net login YOUR_KEGG_USER password YOUR_KEGG_PASSWORD ``` -The host must be `ftp.kegg.net`, and Python's `netrc` parser refuses a file that others can read. To keep the credentials outside the home directory, pass `netrc_path=` to `download_kegg_dump`. +The host must be `ftp.kegg.net`, and Python's `netrc` parser refuses a file that others can read. To keep the credentials outside the home directory, pass `netrc_path=` to `download_kegg_dump`. A secret manager can pass the credentials at run time as `auth=("USER", "PASSWORD")`; a literal password never belongs in committed code. ## Update procedure @@ -101,6 +101,14 @@ The first command computes the URL, the SHA-256 sum and the size of each asset a A new KEGG release changes the drafts. Adopt it in a release of the toolboxes after a review of the changes in the drafts of a few organisms against the previous release. +## Format of the assets + +The tables are gzipped TSV and the reference model is gzipped YAML. The assets use gzip throughout, including the large `organism_gene_ko` table, because MATLAB reads gzip with its built-in `gunzip` and has no `xz` dependency. Both toolboxes share the files, so a format that only Python reads cheaply is excluded. The larger size is a cost of a download once per release. + +The table `organism_gene_ko` is sorted by `(organism, gene)`. The order places the gene identifiers of one organism next to each other, compresses better and matches the by-organism queries of both toolboxes. The sort is an external merge sort bounded by `chunk_rows`, so it stays within memory at the scale of KEGG. + +Parquet would be smaller and typed, and MATLAB reads it natively since R2019a, but it needs the package `pyarrow` in Python. It becomes a candidate if the load time and not the size is the bottleneck, or if either toolbox starts to read columns repeatedly. SQLite is excluded, because MATLAB reads it only with the Database Toolbox, which removes the property of the same files in both languages without extra dependencies. + ## Run the build on a cluster A batch job runs the build script with a time limit that covers the longest expected run. A resubmission of the same job continues where the previous job stopped. An example for Slurm: diff --git a/docs/yaml-format.md b/docs/yaml-format.md index 3154667..0c9b4a2 100644 --- a/docs/yaml-format.md +++ b/docs/yaml-format.md @@ -253,7 +253,11 @@ Enzyme-constrained models carry the EC layer in three additional top-level keys: The sections map to `model.ec` in RAVEN MATLAB and to `raven_toolbox.io.ec_data.EcData` (attached as `model.ec`) in raven-toolbox. cobrapy ignores them. -An `- enzymes: !!omap` line with nothing under the tag is read as a reaction without enzymes. +An ec reaction without enzymes is written with no `enzymes` entry, as GECKO 4 and geckopy 4.0.0b2 write it. GECKO and geckopy read a missing entry as a reaction without enzymes. They also read `- enzymes: !!omap` with nothing under the tag and `enzymes: !!omap []`, the forms that earlier releases wrote, as a reaction without enzymes. + +An ecModel file records its format and its writer in `metaData`: `gecko_format` is `4`, and `generator` names the release that wrote the file. `loadEcModel` of GECKO and `load_ec_model` of geckopy raise an error for a format above 4. GECKO upgrades an ecModel whose protein usage and pool reactions run in the GECKO 3 direction when it loads the file, with a warning. + +`load_ec_model` of geckopy reads a value in double quotes that contains double quotes, as RAVEN 2 wrote the notes of some ecModels, with the outer pair of quotes removed and the inner quotes kept. The spelling `geckoLight` inside `metaData` is also accepted on reading. diff --git a/scripts/known_names.txt b/scripts/known_names.txt index 53ea1be..d214279 100644 --- a/scripts/known_names.txt +++ b/scripts/known_names.txt @@ -482,3 +482,8 @@ regexpPattern removeUnusedMets complex_policy minNrFluxes +blast_formatter # executable of BLAST+ +chunk_rows # parameter of the sort of the organism_gene_ko table in raven_toolbox.reconstruction.kegg.parse +loadEcModel # GECKO function +load_ec_model # geckopy function +gecko_format # key of the metaData of an ecModel file