From 8e3adfd474124a8a4f296cd93fcae650241c8f9e Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Tue, 3 Nov 2026 20:37:33 +0100 Subject: [PATCH 1/3] Count page views with GoatCounter The pages load the GoatCounter script, which sets no cookies, and report to the counter of the site. --- docs/conf.py | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/docs/conf.py b/docs/conf.py index f765093..40070ca 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -51,7 +51,14 @@ html_static_path = ["_static"] html_css_files = ["custom.css"] # Remembers the language of the code tabs across pages, browser tabs and visits. -html_js_files = ["language-sync.js"] +html_js_files = [ + "language-sync.js", + # Page-view statistics of GoatCounter, which sets no cookies. + ( + "https://gc.zgo.at/count.js", + {"async": "async", "data-goatcounter": "https://raven-docs.goatcounter.com/count"}, + ), +] # The "view source" link points at this generated MyST file, not anything a # reader can act on the way an edit-on-GitHub link would. html_show_sourcelink = False From f1b1dd088f3bdb194c7b2335d5069afbe1560f09 Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Tue, 3 Nov 2026 20:46:41 +0100 Subject: [PATCH 2/3] Document the update of the KEGG assets (#115) Adds the page "Update the KEGG assets" to the Installation section, for the package maintainers. It describes how the four assets of a KEGG release (`kegg_core.tar.gz`, the two HMM libraries and the taxonomy file) are built from the bulk KEGG dump with `scripts/build_kegg_artefacts.py`, checked, published to `toolbox-assets`, entered in the manifest and registry, and adopted by raven-toolbox, RAVEN and raven-docs. The page is linked from [Download data and binaries](https://github.com/CompMetEng/raven-docs/blob/main/docs/installation/data-and-binaries.md). The examples are not run by the documentation build, because the download needs a KEGG subscription. --- docs/installation/data-and-binaries.md | 1 + docs/installation/index.md | 1 + docs/installation/updating-kegg-assets.md | 122 ++++++++++++++++++++++ 3 files changed, 124 insertions(+) create mode 100644 docs/installation/updating-kegg-assets.md diff --git a/docs/installation/data-and-binaries.md b/docs/installation/data-and-binaries.md index f7b6572..64df86f 100644 --- a/docs/installation/data-and-binaries.md +++ b/docs/installation/data-and-binaries.md @@ -132,3 +132,4 @@ On Windows, homology reconstruction and the KEGG query path run natively, becaus - [Reconstruction from KEGG](../guide/kegg.md): the workflow that fetches the KEGG artefacts. - [Reconstruction from homology](../guide/homology.md): the workflow that needs BLAST+ or DIAMOND. - [RAVEN (MATLAB)](raven.md) and [raven-toolbox (Python)](python.md). +- [Update the KEGG assets](updating-kegg-assets.md): how the maintainers publish the assets of a new KEGG release. diff --git a/docs/installation/index.md b/docs/installation/index.md index fb4da60..b989fa6 100644 --- a/docs/installation/index.md +++ b/docs/installation/index.md @@ -38,4 +38,5 @@ In MATLAB, `setRavenSolver('gurobi')` selects the active solver. In raven-toolbo raven python data-and-binaries +updating-kegg-assets ``` diff --git a/docs/installation/updating-kegg-assets.md b/docs/installation/updating-kegg-assets.md new file mode 100644 index 0000000..25c22f8 --- /dev/null +++ b/docs/installation/updating-kegg-assets.md @@ -0,0 +1,122 @@ + + +# Update the KEGG assets + +The KEGG reference data and the libraries of profile hidden Markov models (HMMs) are published as assets of one KEGG release in [`toolbox-assets`](https://github.com/CompMetEng/toolbox-assets). Both toolboxes download them, as described in [Download data and binaries](data-and-binaries.md). A new KEGG release needs a new set of assets. The package maintainers do this when they adopt a new KEGG release, and users of the toolboxes never do. + +The documentation build does not run these commands. The download needs a KEGG subscription, and the build of the libraries takes hours. + +## Assets of one release + +A release has the name `kegg`, with `N` the KEGG release number, and four assets that carry the name as a prefix. + +| Asset | Content | +|---|---| +| `kegg_core.tar.gz` | the gene-free reference model as gzipped YAML and the relational tables as gzipped TSV, among them `organism_gene_ko` | +| `kegg_eukaryotes.hmm.gz` | the profile HMM library of the eukaryotic organisms | +| `kegg_prokaryotes.hmm.gz` | the profile HMM library of the prokaryotic organisms | +| `kegg_taxonomy.gz` | the KEGG `taxonomy` file, which assigns each organism to a domain and which `PhylDist` reads | + +Each library holds one profile per KO in ASCII HMMER format, compressed with gzip. It is not indexed with `hmmpress`: `hmmsearch` reads the decompressed file, the gzip file is roughly ten times smaller than a binary index, MATLAB and Python read the same file, and the file stays readable across HMMER versions. The native Windows build of `hmmsearch` is an older version (3.3.2) than the Linux and macOS builds. + +Release tags are immutable. A release that is published under a name is not changed afterwards, and a correction becomes a new release. + +## Requirements + +- A subscription to the bulk download of KEGG (`ftp.kegg.net`), which comes with a user name and a password. +- raven-toolbox from a checkout of its repository, installed as described in [Installing RAVEN](python.md). The commands run from the root of that checkout. +- HMMER (`hmmbuild`), MAFFT and CD-HIT, on the `PATH` or pointed at with the environment variables `RAVEN_PYTHON_HMMBUILD`, `RAVEN_PYTHON_MAFFT` and `RAVEN_PYTHON_CDHIT`. MAFFT and CD-HIT have no native Windows build, so the build runs on Linux, macOS or inside WSL2. Keep the whole toolchain inside WSL2, because raven-toolbox calls the resolved executable directly and does not translate paths across the boundary. +- The GitHub CLI `gh`, authenticated with write access to `toolbox-assets`. +- Disk space for the dump and for the per-KO working files of the build, and enough memory for the largest alignments of MAFFT. MAFFT switches to a memory-light method for an alignment that is predicted to need more memory than the host has, and that method approximates the guide tree. + +The downloader reads the KEGG credentials from `~/.netrc` and takes none on the command line, so they stay out of the shell history and the process list. Create the file with mode 600 and add one line with your own credentials: + +```text +machine ftp.kegg.net login YOUR_KEGG_USER password YOUR_KEGG_PASSWORD +``` + +The host must be `ftp.kegg.net`, and Python's `netrc` parser refuses a file that others can read. To keep the credentials outside the home directory, pass `netrc_path=` to `download_kegg_dump`. + +## Update procedure + +The commands below use `kegg120`. Replace it with the name of the release. + +### 1. Download the KEGG dump + +```python +from raven_toolbox.reconstruction.kegg import download_kegg_dump + +download_kegg_dump("keggdb", force=True, progress=False) +``` + +The function fetches the reaction, compound, glycan and KO archives, the proteomes and the taxonomy file, extracts them and arranges the flat layout that the parser expects. Without `force=True`, it skips files that exist, so use it for a new release in a directory that holds an older dump. + +### 2. Build the assets + +```bash +python scripts/build_kegg_artefacts.py --keggdb keggdb --out artefacts \ + --version kegg120 --hmms --threads 16 +``` + +The script parses the dump into the reference model and the tables, builds the library of each domain from the sequences of the KOs, and arranges the four assets in `artefacts`. `--domains` restricts the libraries to one domain. + +The build of the libraries is the slow step. The script resumes: it skips a stage whose output exists, and it skips every KO whose profile exists in the working directory `artefacts/_hmms-`. Run the same command again after a failure. `--force` rebuilds everything, and a changed gene-to-KO table needs it, because a resumed run assumes the inputs of the first run. + +The working directory holds one FASTA file and one profile per KO. On a file system with a quota on the number of files, put `artefacts` on a file system without that quota. + +### 3. Check the assets + +- The four files of the table exist in `artefacts`, and their sizes are of the same order as the previous release (49 MB, 135 MB, 155 MB and 0.2 MB for `kegg118`). +- The number of profiles in a library is the number of lines that start with `NAME` in the decompressed file. Compare it with the previous release. +- Reconstruct a draft for an organism that KEGG contains, for example *S. cerevisiae*, with `get_kegg_model_for_organism_from_artefacts`, and search its proteome against the eukaryotic library with `get_kegg_model_from_sequences_with_artefacts`. The assignment of the search should resemble the KEGG assignment of the organism. Both functions are described in [Reconstruction from KEGG](../guide/kegg.md). +- Record the KEGG release and the versions of HMMER, MAFFT and CD-HIT of the build. + +### 4. Publish the assets + +```bash +python scripts/publish_to_toolbox_assets.py --dry-run release --tag kegg120 --dir artefacts +python scripts/publish_to_toolbox_assets.py release --tag kegg120 --dir artefacts +``` + +The first command prints the `gh` calls without running them. The second creates the release `kegg120` in `toolbox-assets` and uploads the four assets. A rerun uploads only the assets that are missing. + +### 5. Update the manifest and the registry + +The manifest `data/manifest.json` is the single source of truth for the checksums of the assets. Never edit the registry in `src/raven_toolbox/data.py` by hand. + +```bash +python scripts/make_registry_snippet.py manifest --manifest data/manifest.json \ + --target data --dataset kegg --version kegg120 --dir artefacts --tag kegg120 +python scripts/make_registry_snippet.py sync +python scripts/publish_to_toolbox_assets.py release --tag manifest-v4 data/manifest.json +``` + +The first command computes the URL, the SHA-256 sum and the size of each asset and writes them to the manifest. The second regenerates the baked registry of raven-toolbox from the manifest. The third publishes the manifest as a new snapshot. The number of the snapshot is the next number after the newest `manifest-v` release in `toolbox-assets`. + +### 6. Point the toolboxes to the new release + +- **raven-toolbox:** commit the changes to `data/manifest.json` and `src/raven_toolbox/data.py`, and open a pull request. raven-toolbox pins the manifest of its release, so users receive the new KEGG release with the next raven-toolbox release. The environment variable `RAVEN_PYTHON_MANIFEST` points an installed raven-toolbox to a newer manifest. +- **RAVEN:** write the new release name in the first line of `src/reconstruction/kegg/KEGG_VERSION.md`, which `keggDataVersion` reads, and change the manifest URL in `fetchToolboxAsset` to the new snapshot. RAVEN reads the manifest on each call. +- **raven-docs:** update the sizes in [Download data and binaries](data-and-binaries.md), and rerun the examples of [Reconstruction from KEGG](../guide/kegg.md) with the new release. That page quotes the release, the counts of the drafts and the timings. + +A new KEGG release changes the drafts. Adopt it in a release of the toolboxes after a review of the changes in the drafts of a few organisms against the previous release. + +## Run the build on a cluster + +A batch job runs the build script with a time limit that covers the longest expected run. A resubmission of the same job continues where the previous job stopped. An example for Slurm: + +```bash +#!/bin/bash +#SBATCH -A +#SBATCH -n 16 +#SBATCH -t 7-00:00:00 +#SBATCH -J kegg-assets + +python scripts/build_kegg_artefacts.py --keggdb keggdb --out artefacts \ + --version kegg120 --hmms --threads 16 +``` + +## See also + +- [Download data and binaries](data-and-binaries.md): what a user downloads, and where the files are cached. +- [Publishing versioned artefacts](https://github.com/CompMetEng/raven-gecko-parity/blob/develop/docs/publishing_artefacts.md): the convention for tags, manifests and checksums. From e15ebc6b04e249abb2b239257a55c764fb0ec6e8 Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Tue, 3 Nov 2026 20:46:52 +0100 Subject: [PATCH 3/3] Add the develop branch to the workflows (#117) `develop` is the default branch and documents the `develop` branches of RAVEN and raven-toolbox. The daily submodule update commits to `develop`, and the documentation build and the examples run on pushes to `develop` and `main`. `main` documents the released versions and receives a pull request from `develop` after a release of RAVEN and raven-toolbox. `DESIGN.md` and `CONTRIBUTING.md` describe the branches. --- .github/workflows/docs-build.yml | 2 +- .github/workflows/examples.yml | 2 +- .github/workflows/update-submodules.yml | 7 ++++--- CONTRIBUTING.md | 4 ++++ DESIGN.md | 4 ++-- 5 files changed, 12 insertions(+), 7 deletions(-) diff --git a/.github/workflows/docs-build.yml b/.github/workflows/docs-build.yml index 5e7d083..7c9bf57 100644 --- a/.github/workflows/docs-build.yml +++ b/.github/workflows/docs-build.yml @@ -6,7 +6,7 @@ name: Docs build on: push: - branches: [main] + branches: [main, develop] pull_request: schedule: - cron: "0 6 * * 1" diff --git a/.github/workflows/examples.yml b/.github/workflows/examples.yml index efc599c..02e73f1 100644 --- a/.github/workflows/examples.yml +++ b/.github/workflows/examples.yml @@ -10,7 +10,7 @@ name: Guide examples on: push: - branches: [main] + branches: [main, develop] paths: - "docs/guide/**" - "docs/data/**" diff --git a/.github/workflows/update-submodules.yml b/.github/workflows/update-submodules.yml index 369603b..1c3e6d8 100644 --- a/.github/workflows/update-submodules.yml +++ b/.github/workflows/update-submodules.yml @@ -1,7 +1,7 @@ name: Update submodules # Moves each submodule to the tip of the branch set in .gitmodules and commits -# the new pointers to main. +# the new pointers to develop. on: schedule: @@ -21,6 +21,7 @@ jobs: steps: - uses: actions/checkout@v7 with: + ref: develop submodules: true fetch-depth: 1 @@ -37,7 +38,7 @@ jobs: git checkout -B "$branch" FETCH_HEAD ' - - name: Commit to main + - name: Commit to develop run: | if git diff --quiet; then echo "No submodule updates." @@ -47,4 +48,4 @@ jobs: git config user.email "41898282+github-actions[bot]@users.noreply.github.com" git add -A git commit -m "chore: update submodules to latest tracked branches" - git push origin HEAD:main + git push origin HEAD:develop diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index efbcd8b..11a9341 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -12,6 +12,10 @@ The standard for this site is the `docs-site` plugin of [CompMetEng/skills](http 3. Established technical shorthand (a domain term, a standard abbreviation) stays. 4. No mention of AI tools or AI authorship in the repository: not in commits, pull requests, code comments or pages. Branch names do not contain a tool name. +## Branches + +`develop` is the default branch. It documents the `develop` branches of RAVEN and raven-toolbox, which the submodules track, and a daily workflow moves the submodule pointers to their tips. Pull requests target `develop`. `main` documents the released versions: a release of RAVEN and raven-toolbox is followed by a pull request from `develop` to `main` that is merged with a merge commit and pins the submodules to the released commits. + ## Building and checking ```bash diff --git a/DESIGN.md b/DESIGN.md index 9a110c7..5299988 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -3,7 +3,7 @@ > **This is a living document, and it is the source of intent.** It describes how > the RAVEN documentation site *should* be designed and what it *should* > contain; it is a set of instructions, not a description of the current state -> of `main`. When the site and this document disagree, treat this document as +> of `develop`. When the site and this document disagree, treat this document as > the target and bring the site in line (not the other way round). Keep it > updated as decisions are made; record new decisions in the decisions log and > move resolved items out of "open questions". @@ -15,7 +15,7 @@ > > ```bash > git fetch --all --prune -> git checkout main && git pull --ff-only +> git checkout develop && git pull --ff-only > git submodule sync --recursive > git submodule update --init --recursive > ```