Package the KEGG module graphs and module descriptions - #3
Merged
Conversation
I'm working to get the blimmp_nexflow code working, but it was failing to find any hits. This commit addresses that. In the manifest, `package_data` referenced `KEGG_Module_Graphs.zip`, a filename that no longer exists in the repo, and it didn't list `kegg_bacteria_modules.json`. Both files were therefore dropped by `pip install .`, so the published container had them in /app but not in site-packages, which is where module_detection.py looks. The consequence was silent. `modules_to_kos()` returned an empty dict, so the KO universe was empty, so the left join of observed annotations onto it discarded every hit. Runs exited 0 and wrote a structurally complete result in which the per-KO table had zero rows and all 340 modules scored 0.0 confidence. Verified against ghcr.io/traviswheelerlab/blimmp:0.1.2. Changes: - setup.py / MANIFEST.in: reference the real archive name and add the module descriptions. Globs instead of individual filenames so a future rename does not silently drop a file again. - module_detection.py: extract the graph archive via `ensure_module_graphs()`, which reuses an existing extraction, never deletes the source zip, and falls back to a writable cache dir (BLIMMP_CACHE_DIR) when the install tree is read-only. The previous code wrote into site-packages and removed the zip, which would have started failing under Singularity as soon as the zip was actually packaged. - module_detection.py: `validate_paths()` and an empty-KO-universe guard, so a mispackaged install aborts with a clear message instead of reporting every module absent. - Dockerfile: extract the graphs at build time, while the filesystem is still writable, and assert the data files made it into the installed package. - tests/smoke_test.py + CI: run BLIMMP on Examples/example.domtblout and assert a non-empty per-KO table, at least one non-zero module confidence, and populated module descriptions. Also runs against a read-only container. This commit deliberately leaves the version alone. Once it is merged to main, bump setup.py to 0.1.3 and tag v0.1.3 to trigger a corrected container build; setup.py has been stuck at 0.1.0 across the v0.1.1 and v0.1.2 tags. Verified on Python 3.10 (Examples/example.domtblout, sigma 1.0): before 0 per-KO rows, 0/340 modules with confidence, 0/340 with descriptions after 1980 per-KO rows, 219/340 with confidence, 323/340 with descriptions Implemented with help from Claude Opus 5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I'm working to get the blimmp_nexflow code working, but it was failing to find any hits. This PR addresses that. The issue is with the docker container being pulled by nf.
In the manifest,
package_datareferencedKEGG_Module_Graphs.zip, a filename that no longer exists in the repo, and it didn't listkegg_bacteria_modules.json. Both files were therefore dropped bypip install ., so the published container had them in /app but not in site-packages, which is where module_detection.py looks.Read details about the solution in the PR.
Once this is merged to main, bump setup.py to 0.1.3 and tag v0.1.3 to trigger a corrected container build; setup.py has been stuck at 0.1.0 across the v0.1.1 and v0.1.2 tags.