Skip to content

feat(stack): add build-list line numbers, module dumps and version-change.tsv - #10

Merged
HaoZeke merged 8 commits into
mainfrom
feat/stack-gap-migrations
Nov 3, 2026
Merged

HaoZeke merged 8 commits into
mainfrom
feat/stack-gap-migrations

Conversation

@HaoZeke

@HaoZeke HaoZeke commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Reproducer

eb-stack stack gap \
  --site tests/gap_routes/one-per-route.txt \
  --installed tests/gap_routes/installed.txt \
  --easyconfigs tests/gap_routes/easyconfigs \
  --generation 2026.1 \
  --exclude-file tests/gap_routes/site-only.tsv \
  --filter-deps zlib \
  --open-layer tests/gap_routes/open-layer.json \
  --open-easyconfigs tests/gap_routes/open-easyconfigs.json \
  --out-dir /tmp/gap-one-per-route

The fixture has seven site modules, one per route:

site=7 in-stack=1 filtered=1 open-layer=1 recipe=1 open-easyconfigs=1 no-recipe=1 excluded=1 out=/tmp/gap-one-per-route

What and why

U3, build-list and hooks lint.
stack provided now prints the one-based build-list line on every provided and unparsed finding.
The hook and toolchain findings already had it, so a failing list can be fixed without a manual search.

U4, live module dumps.
scripts/dump-module-trees.sh captures the module -t avail trees of the site, the older European Environment for Scientific Software Installations (EESSI) version and the target EESSI version.
Each tree comes from its own purged shell.
A non-interactive bash has no module function, so the helper sources $LMOD_PKG/init/bash, then $MODULESHOME/init/bash, and uses the first one that defines module.
Lmod's init isn't safe under set -eu, so the helper relaxes both around the source and checks for the function afterwards.
With neither init present it exits 2 and writes nothing.
Purge and load output for each tree goes to OUT_DIR/<tree>.module.log, so the inventory files hold only the avail listing.
A failed purge, a failed load, or a meta-module that module is-loaded doesn't report makes the helper exit 1, naming the tree and printing that log.
Before the first dump it deletes site.txt, eessi-older.txt and eessi-target.txt, so a failed run never leaves an earlier run's inventory for stack gap to read.
A half-written .tmp file is removed on exit.
The new tests/gap_routes/one-per-route.txt fixture gives exactly one row in each tab-separated route file.

U5, version-change.tsv.
stack gap now writes one row per old version.
Modules on the in-stack route are moved and name their EESSI version.
Excluded modules are site-only, and the rest stay gone until EESSI provides them.

This also fixes a malformed inline literal in the src/shard.rs doc comment.
Sphinx failed on it at shard.rst:70, because warnings count as errors here.

Before and after

eb-stack stack provided \
  --build-list tests/provided_fixtures/site.list \
  --installed tests/provided_fixtures/installed.txt \
  --baseline-lock tests/provided_fixtures/baseline.lock.json

Before (exit 1):

provided GROMACS-2026.0-foss-2026.1.eb name=GROMACS list=tests/provided_fixtures/site.list
unparsed not-a-version.eb list=tests/provided_fixtures/site.list

After (exit 1):

provided GROMACS-2026.0-foss-2026.1.eb name=GROMACS list=tests/provided_fixtures/site.list line=3
unparsed not-a-version.eb list=tests/provided_fixtures/site.list line=7

Before, stack gap wrote no version-change.tsv for the live-tree example in docs/orgmode/howto/module-gap.org, run with --out-dir /tmp/gap-mwe.
After:

name	old_version	eessi_version	status
CUDA	12.6.0	12.6.0	moved
FFTW	3.3.10	3.3.10	moved
LAMMPS	2023.0	-	gone
LAMMPS	2024.0	-	gone
MATLAB	2024b	-	site-only
QuantumESPRESSO	7.3	-	gone
binutils	2.42	-	gone
code-server	4.96.2	-	site-only

For the helper, the test helper_fails_with_the_lmod_error_when_a_load_fails stubs module so that module load EESSI/2026.06 fails, and leaves an old eessi-target.txt in the output folder.

The old helper sent the load error to /dev/null and stopped before mv, so by its code the old eessi-target.txt stayed behind for stack gap to read.
The new helper exits 1.
Its test checks that standard error names EESSI/2026.06 and carries the stub's error.
It also checks that no eessi-target.txt is left.
The script makes that run print the lines below, ending with the stub's error string:

site	/tmp/module-dumps/site.txt
older	/tmp/module-dumps/eessi-older.txt
dump-module-trees: target tree: module load EESSI/2026.06 failed
Lmod has detected the following error: no such module

With no module function and no Lmod init, the helper exits 2 before creating the output folder.
The test helper_exits_when_no_lmod_init_exists checks the exit code, the missing folder and the not initialized text.
The script prints this full message:

dump-module-trees: the module command is not initialized, and neither $LMOD_PKG/init/bash nor $MODULESHOME/init/bash exists

Tests

  • cargo test --locked --test module_tree_dump_script: 7 passed, 0 failed.
  • cargo test --locked --all-targets: 1449 passed, 0 failed, 10 ignored.
  • cargo fmt --check and cargo clippy --locked --all-targets -- -D warnings pass.
  • python3 scripts/gen-llms-txt.py --check passes, and the Sphinx build under -W passes.
  • CI pending.

The helper tests use a stub module command through BASH_ENV, MODULESHOME or LMOD_PKG.
They cover the init order, the missing init, a failed purge, a failed load, a module that isn't loaded afterwards, and stale inventories.

Repro

cargo fmt --check
cargo clippy --locked --all-targets -- -D warnings
cargo test --locked --all-targets
cargo package --locked
uvx prek run -a
pixi run -e docs docbld
scripts/check-doc-links.sh docs/build
python3 scripts/gen-llms-txt.py --check

On a real Lmod shell:

scripts/dump-module-trees.sh /tmp/module-dumps - EESSI/2025.06 EESSI/2026.06

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 682ea65. Configure here.

Comment thread scripts/dump-module-trees.sh Resolved
Comment thread scripts/dump-module-trees.sh Resolved
…ees.sh

A non-interactive bash has no `module` function.
The helper now sources `$LMOD_PKG/init/bash` or `$MODULESHOME/init/bash`, and exits 2 when neither exists.
It sources the init with errexit and nounset off, then checks for the function.
Purge and load output goes to a per-tree log instead of `/dev/null`.
A failed purge or load, or a module that `module is-loaded` does not report, exits 1 with the module name and that log.
The helper deletes earlier inventories before the first dump and writes none for a failed tree.
A purged base tree or a stale file therefore never passes as an older or target inventory.
The exit trap removes only the temporary file the helper is writing.

The test covers the `MODULESHOME` init, `LMOD_PKG` precedence over `MODULESHOME`, and a missing init.
It also covers a failed purge, a failed load that leaves no stale target, and a load that leaves the module unloaded.
@HaoZeke
HaoZeke merged commit a7c0fdd into main Nov 3, 2026
19 checks passed
@HaoZeke
HaoZeke deleted the feat/stack-gap-migrations branch November 4, 2026 09:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant