Skip to content

module-gap.org

Latest commit

 

History

History
180 lines (151 loc) · 7.73 KB

module-gap.org

File metadata and controls

180 lines (151 loc) · 7.73 KB
Raw

Route the modules a stack does not have

eb-stack stack gap takes the module names a site loads and says, for each one the stack does not have, which route closes it. The routes are tried in order. The first that applies wins.

RouteWhen
in-stackThe stack’s installed list has the name, ignoring case
excluded--exclude-file names it (licensed, vendor, conda, portal). Written to site-only.tsv
filtered--filter-deps names it, or EASYBUILD_FILTER_DEPS when the flag is absent
open-layerAn --open-layer pull request adds an easystack line of that name
recipeThe easyconfig trees have a recipe on this generation
open-easyconfigsAn --open-easyconfigs pull request adds a recipe at this generation
no-recipeNothing above applies. Written to bump.tsv. The newest easyconfig on any toolchain is the bump source

A toolchain whose version is the generation counts as on it, so lfoss-2026.1 is a 2026.1 recipe. So does a toolchain listed for that generation in data/eessi-generations.toml, such as llvm-compilers-21.1.8.

Capture the live inventories

The helper uses the shell’s module function. A non-interactive shell often has none, so the helper then sources $LMOD_PKG/init/bash, or $MODULESHOME/init/bash when the first is missing. It exits 2 when neither file exists. Its four arguments are the output directory, the site meta-module, the older EESSI module, and the target EESSI module. Use - for the site argument when the base module path is the live site tree.

scripts/dump-module-trees.sh \
  /tmp/module-dumps - EESSI/2025.06 EESSI/2026.06

Each inventory is captured in a separately purged subshell, so loading one EESSI version cannot add paths to the next dump. The files are /tmp/module-dumps/site.txt, eessi-older.txt, and eessi-target.txt. Pass them as --site, --older, and --installed, respectively. They are the raw module -t avail output; stack gap ignores path headers and default markers.

The purge and load output for each tree goes to a .module.log file next to its inventory, such as eessi-target.module.log. If the purge or the load fails, or module is-loaded does not report the module afterwards, the helper prints the module name and that log and exits 1. It writes no inventory for that tree, so stack gap never reads a purged base tree as an older or target inventory. Before the first dump, the helper deletes all three inventories from any earlier run, so a failed run leaves none of them behind.

Read the open-pull-request inventories

The two fetch-prs.sh examples produce the JSON accepted by --open-layer and --open-easyconfigs. Run each script in the clone named in its comment:

EB_STACK=$PWD
(cd "$EB_EESSI_SOFTWARE_LAYER" && \
  "$EB_STACK/examples/eessi-pr-overlap/fetch-prs.sh" \
    EESSI/software-layer 2026.06 /tmp/open-layer)
(cd "$EB_EASYCONFIGS_DEVELOP" && \
  "$EB_STACK/examples/easyconfigs-pr-overlap/fetch-prs.sh" \
    easybuilders/easybuild-easyconfigs /tmp/open-easyconfigs)

layer_args=()
for file in /tmp/open-layer/prs/*.json; do
  layer_args+=(--open-layer "$file")
done
easyconfig_args=()
for file in /tmp/open-easyconfigs/prs/*.json; do
  easyconfig_args+=(--open-easyconfigs "$file")
done

Add "${layer_args[@]}" and "${easyconfig_args[@]}" to the stack gap command. The fetch scripts only read pull requests. A closed pull request disappears from their rewritten prs/ directory on the next run.

Run the fixture

The nine names are in tests/gap_routes/site.txt. The easyconfig tree has BLAST+ on gompi-2026.1 and CP2K on foss-2025b. The installed list has FFTW. zlib is filtered. MATLAB is licensed. Pull request 1787 adds LAMMPS. Pull request 25001 adds QuantumESPRESSO at foss-2026.1 and OldTool at foss-2025b.

eb-stack stack gap \
  --site tests/gap_routes/site.txt \
  --installed tests/gap_routes/installed.txt \
  --easyconfigs tests/gap_routes/easyconfigs \
  --generation 2026.1 \
  --exclude-file tests/gap_routes/site-only.tsv \
  --filter-deps zlib \
  --open-layer tests/gap_routes/open-layer.json \
  --open-easyconfigs tests/gap_routes/open-easyconfigs.json \
  --out-dir /tmp/gap-demo

That command printed:

site=9 in-stack=1 filtered=1 open-layer=1 recipe=1 open-easyconfigs=1 no-recipe=3 excluded=1 out=/tmp/gap-demo

The process exited 0. /tmp/gap-demo/gap.tsv is every row:

name	route	recipe	newest_elsewhere	reason
BLAST+	recipe	BLAST+-2.17.0-gompi-2026.1.eb	-	-
CP2K	no-recipe	-	CP2K-2025.2-foss-2025b.eb	-
FFTW	in-stack	-	-	-
Julia	no-recipe	-	-	-
LAMMPS	open-layer	LAMMPS-2025.0-foss-2026.1.eb	-	software-layer#1787
MATLAB	excluded	-	-	licensed
OldTool	no-recipe	-	-	-
QuantumESPRESSO	open-easyconfigs	QuantumESPRESSO-7.4-foss-2026.1.eb	-	easyconfigs#25001
zlib	filtered	-	-	filtered to the compatibility layer

OldTool is in the easyconfigs pull request, and the file is foss-2025b, so it is not open-easyconfigs. Julia has no easyconfig at all. CP2K’s newest file is the bump source. BLAST+ is the only recipe row, and it is the only line of roots.txt, which is the input of stack batch --roots-file. bump-sources.txt names CP2K-2025.2-foss-2025b.eb.

Each route also has its own file: in-stack.tsv, site-only.tsv, filtered.tsv, open-layer.tsv, recipe.tsv, open-easyconfigs.tsv, and bump.tsv. An open-layer match is the easystack line the pull request adds, not a dependency that line would build.

version-change.tsv is the user migration table. It keeps the module versions exactly as the live dumps spell them, including a toolchain suffix. An in-stack name is moved and names the target EESSI version. An excluded name is site-only. Every other route is gone from the opened stack until that route lands. A name with two old versions has two rows; multiple target versions are comma-separated. The columns are name, old_version, eessi_version, and status.

eb_stack_gap writes the same files. open_layer and open_easyconfigs are arrays of the same pull-request JSON pr-overlap --pr reads.

A live module tree

--site, --older, and --installed also take a directory. The walk reads Name/version.lua and skips .modulerc.lua and default, so pass the modules root and not only modules/all. A file may be module -t avail or module spider -t output. --older is an EESSI version users load beside the site. Repeat --installed for the CPU tree and the accelerator compute-capability tree. --eessi-version 2026.06 drops an open software-layer easystack from another EESSI version. The pull-request match is still the filename that was added.

tests/gap_live is a site tree, an older tree, a CPU target, an accelerator target, and one module -t avail file. CUDA is only in the accelerator tree. FFTW is only in the CPU tree. Pull request 1700 adds LAMMPS to 2026.06. Pull request 1600 adds QuantumESPRESSO to 2025.06, and --eessi-version 2026.06 leaves that name as no-recipe.

eb-stack stack gap \
  --site tests/gap_live/site/modules \
  --site tests/gap_live/avail.txt \
  --older tests/gap_live/older/modules \
  --installed tests/gap_live/target-cpu/modules \
  --installed tests/gap_live/target-cc90/modules \
  --easyconfigs tests/gap_routes/easyconfigs \
  --generation 2026.1 \
  --eessi-version 2026.06 \
  --exclude-file tests/gap_live/exclude.tsv \
  --filter-deps binutils \
  --open-layer tests/gap_live/open-layer.json \
  --out-dir /tmp/gap-live

That command prints site=7 in-stack=2 filtered=1 open-layer=1 recipe=0 open-easyconfigs=0 no-recipe=1 excluded=2. The seven names are CUDA, FFTW, MATLAB, code-server, binutils, LAMMPS, and QuantumESPRESSO.