feat(stack): report site recipes that shadow an EESSI module

.github/workflows/ci_test.yml

@@ -180,6 +180,7 @@ jobs:
             --test generation_lint \
             --test stack_gap \
             --test stack_provided \
+            --test stack_shadow \
             --test stack_shard \
             --test hierarchy_universe \
             --test jenkins_build_list \

AGENTS.md

@@ -73,7 +73,7 @@ lint findings remain, and do not comment on the PR.
 ```text
 package  inspect | plan | bump
 recipe   easyblock | templates | hierarchy | check | lint | format
-stack    solve | sbom | order | easystack | sections | from-commit | gap | provided | batch | shard | arch | files | generation | cuda-cc | known-failures | oom | bot-classify
+stack    solve | sbom | order | easystack | sections | from-commit | gap | provided | shadow | batch | shard | arch | files | generation | cuda-cc | known-failures | oom | bot-classify
 target   list | doctor
 campaign run | status | finding claim | finding resolve
 repro    summary

CHANGELOG.md

@@ -6,6 +6,14 @@ All notable changes to this project are documented here.
 
 ### Added
 
+- `stack shadow` reports a site-layer recipe that shadows a module the
+  EESSI base stack already provides. `--base` is an Lmod tree or a module
+  listing, loaded with the same inventory reader as `stack gap`. The same
+  software version is `same`, another version of that name is `different`,
+  and a base name with no recorded version is `unversioned`. A name the
+  base does not have is not reported. The command exits 1 when any recipe
+  shadows, or when a recipe does not parse. `eb_stack_shadow` is the same
+  check.
 - `stack provided` includes the one-based build-list line on every
   `provided` and `unparsed` finding in its text report, as it already did for
   hook, toolchain, family-lock, and licensed findings.
@@ -52,6 +60,23 @@ All notable changes to this project are documented here.
   A line that cannot be resolved exits 1 and writes nothing.
   `eb_stack_shard` is the same command over MCP.
 
+### Documentation
+
+- How to split one Jenkins build list into shards that build at the
+  same time, with the shared dependencies in a prelude.
+- How to plan software-layer pull requests whose builds do not overlap,
+  including the second-version warning a batch records.
+- How to regroup a Jenkins build list under the section headers of a
+  previous list.
+- How to write a stack lock as the YAML `eb --easystack` reads.
+- How to read the easyblock class, the template values, and the
+  toolchain hierarchy EasyBuild derives.
+- How to report the newest stable upstream release of an EasyBuild
+  recipe, and why a listed version was kept or dropped.
+- How to read the reproduction scoreboard and hold it to a ratchet file.
+- How to report a site-layer recipe that shadows a module the EESSI
+  base stack already provides.
+
 ## 0.4.0 - 2026-11-01
 
 `stack solve` in 0.3.0 took the policy file as `--policy`. The flag is

docs/orgmode/explanation/sbom-blast-radius.org

@@ -989,7 +989,7 @@ table. ~reverse=0~ again, for the same reason as the CMakeMake run.
 
 * The same document over MCP
 
-The Model Context Protocol (MCP) catalog has 50 tools. ~tools/list~ prints
+The Model Context Protocol (MCP) catalog has 51 tools. ~tools/list~ prints
 their names:
 
 #+begin_src bash
@@ -1027,6 +1027,7 @@ eb_stack_easystack
 eb_package_latest
 eb_stack_gap
 eb_stack_provided
+eb_stack_shadow
 eb_stack_batch
 eb_stack_shard
 eb_stack_arch
@@ -1051,7 +1052,7 @@ eb_easyblock_reach
 eb_repro_summary
 #+end_src
 
-That is 50 tools.
+That is 51 tools.
 ~eb_stack_sbom~ writes the CycloneDX file ~stack sbom~ writes.
 Its ~tools/list~ description reads: "Emit CycloneDX from an existing stack lock, reading dependency edges and source checksums from the easyconfigs the lock names. With backport_list, also write the modules whose reach class justifies a backport."
 ~eb_stack_sbom~ reads the Cbc lock:

docs/orgmode/howto/package-latest.org

@@ -0,0 +1,49 @@
+#+TITLE: Report the newest stable upstream release
+#+OPTIONS: toc:t num:nil author:nil
+
+What it does: ~package latest~ reads each easyconfig's ~source_urls~ and
+asks that backend for versions. PyPI, GitHub, GitLab, the Comprehensive
+R Archive Network (CRAN), crates.io, and an HTTP directory listing are
+the backends. Every listed version is kept, or dropped with a reason: a
+draft, a pre-release, a yanked release, or a per-package override.
+~data/upstream-overrides.toml~ is the override file the binary carries.
+~--overrides~ files apply after it.
+
+Why you'd use it: ~package bump --latest~ moves a recipe to the version
+this command picks. Run the report first when you want to see what was
+dropped.
+
+* A directory that lists no version
+
+~tests/shadow_fixtures/easyconfigs/z/zlib/zlib-1.3.1-GCCcore-15.2.0.eb~
+points ~source_urls~ at ~https://example.org/~, which is not a file
+listing of zlib tarballs.
+
+#+begin_src bash
+eb-stack package latest \
+  --source tests/shadow_fixtures/easyconfigs/z/zlib/zlib-1.3.1-GCCcore-15.2.0.eb
+#+end_src
+
+That command printed:
+
+#+begin_example
+name=zlib current=1.3.1 newest=- state=unknown backend=directory:https://example.org/ reason="the backend listed no version of the recipe's source file"
+#+end_example
+
+Read the decision line left to right. ~current~ is the recipe version.
+~newest~ is the newest version that was kept, or ~-~ when none was.
+~state~ is ~current~ (the recipe is already that version), ~behind~,
+~pinned~ (an override names the version to keep), or ~unknown~. The
+~reason~ is why no version was chosen.
+
+A backend that does list versions adds one indented line per version,
+~kept~ or ~dropped (<reason>)~. A pre-release is dropped. ~--json~
+prints the same rows as one array. GitHub requests send ~GITHUB_TOKEN~
+or ~GH_TOKEN~ when it is set.
+
+A recipe whose ~source_urls~ name PyPI, such as
+~tests/sbom_recorded/rebuild/easybuild/easyconfigs/w/wrapt/wrapt-2.2.1-GCCcore-15.2.0.eb~,
+uses the PyPI backend. That command queries the network, and the newest
+stable release moves, so the page does not freeze a version number from
+one run. ~eb_package_latest~ is the same command. See
+[[file:../reference/cli.org][the CLI reference]].

docs/orgmode/howto/recipe-identity.org

@@ -0,0 +1,85 @@
+#+TITLE: Read the easyblock, templates, and hierarchy
+#+OPTIONS: toc:t num:nil author:nil
+
+What it does: three recipe commands answer questions EasyBuild decides
+before it builds. ~recipe easyblock~ prints the class name derived from
+a software name. ~recipe templates~ prints the template constants, or
+the values ~--name~ and ~--version~ resolve. ~recipe hierarchy~ prints
+the known toolchain hierarchies as JSON.
+
+Why you'd use it: a recipe that sets no ~easyblock~ is claiming the
+derived class exists. A template such as ~%(version_major_minor)s~ is a
+value, not a string to guess. A dependency's toolchain has to be a
+member of the parent's hierarchy.
+
+* The derived easyblock
+
+#+begin_src bash
+eb-stack recipe easyblock zlib
+eb-stack recipe easyblock GROMACS
+eb-stack recipe easyblock -- c++
+#+end_src
+
+Those commands printed:
+
+#+begin_example
+EB_zlib
+EB_GROMACS
+EB_c_plus__plus_
+#+end_example
+
+The third name is passed after ~--~ because it starts with punctuation.
+~--charmap~ prints the substitution table as JSON instead of one class
+name. The prefix is ~EB_~ and the table has 33 entries. The recorded
+table is ~tests/recipe_command_fixtures/easyblock-charmap.json~.
+
+* Resolved templates
+
+~--name~ and ~--version~ together print the values the derived-key
+rules produce. The toolchain in that pair is ~system~.
+
+#+begin_src bash
+eb-stack recipe templates --name zlib --version 1.3.1
+#+end_src
+
+That command printed:
+
+#+begin_example
+{
+  "arch": "x86_64",
+  "bitbucket_account": "zlib",
+  "github_account": "zlib",
+  "name": "zlib",
+  "nameletter": "z",
+  "nameletterlower": "z",
+  "namelower": "zlib",
+  "toolchain_name": "system",
+  "toolchain_version": "",
+  "version": "1.3.1",
+  "version_major": "1",
+  "version_major_minor": "1.3",
+  "version_major_minor_patch": "1.3.1",
+  "version_minor": "3",
+  "version_minor_patch": "3.1",
+  "version_patch": "1",
+  "versionsuffix": ""
+}
+#+end_example
+
+~--export~ prints the constant table and the derived-key rules instead
+of one resolved pair. ~arch~ follows the machine that runs the command.
+
+* Known hierarchies
+
+#+begin_src bash
+eb-stack recipe hierarchy
+#+end_src
+
+The document is the generations in
+~tests/recipe_command_fixtures/hierarchy.json~. The first parent is
+foss/2023b, and its members run from the system toolchain through
+GCCcore, GCC, gfbf, and gompi to foss, most minimal subtoolchain first.
+~--export~ adds the synthetic GCC-family rule. The plain document has
+no ~synthetic~ key. ~eb_recipe_easyblock~, ~eb_recipe_templates~, and
+~eb_recipe_hierarchy~ are the same commands. See
+[[file:../reference/cli.org][the CLI reference]].

docs/orgmode/howto/repro-summary.org

@@ -0,0 +1,70 @@
+#+TITLE: Read the reproduction scoreboard
+#+OPTIONS: toc:t num:nil author:nil
+
+What it does: ~repro summary~ reads every ~*.json~ case score under
+~--scores~ and prints the scoreboard. ~Score~ is ~EXACT~, ~SEMANTIC~, or
+~MATERIAL~. ~Allowance~ is how many hand-added lines the case still
+forgives. ~Residual~ is the raw diff lines left after that allowance.
+~--ratchet~ checks those allowance counts against a committed file.
+
+Why you'd use it: the reproduction suite writes one JSON file per case.
+The table is the grind's current position, and the ratchet fails the
+run when an allowance grows.
+
+* Two cases and a ratchet that holds
+
+The two files below are the shape ~repro summary~ reads. The GROMACS
+case still forgives one line. ScaFaCoS has an empty allowance and a
+semantic diff, so its residual in this file is 0.
+
+#+begin_src bash
+mkdir -p /tmp/repro-scores
+cat > /tmp/repro-scores/reproduces_gromacs.json <<'EOF'
+{
+  "case": "reproduces_gromacs",
+  "pull_request": null,
+  "source": "source.eb",
+  "target": "target.eb",
+  "score": "EXACT",
+  "allowance": ["    ('pybind11', '2.12.0'),"],
+  "stale_allowance": [],
+  "residual_lines": 0
+}
+EOF
+cat > /tmp/repro-scores/reproduces_scafacos.json <<'EOF'
+{
+  "case": "reproduces_scafacos",
+  "pull_request": null,
+  "source": "source.eb",
+  "target": "target.eb",
+  "score": "SEMANTIC",
+  "allowance": [],
+  "stale_allowance": [],
+  "residual_lines": 0
+}
+EOF
+cat > /tmp/repro-ratchet.json <<'EOF'
+{"note":"example","cases":{"reproduces_gromacs":1,"reproduces_scafacos":0},"total":1}
+EOF
+eb-stack repro summary --scores /tmp/repro-scores --ratchet /tmp/repro-ratchet.json
+#+end_src
+
+That command printed:
+
+#+begin_example
+| Case | PR | Score | Allowance | Residual |
+|---+---+---+---+---|
+| reproduces_gromacs | - | EXACT | 1 | 0 |
+| reproduces_scafacos | - | SEMANTIC | 0 | 0 |
+
+2 cases: 1 EXACT, 1 SEMANTIC, 0 MATERIAL; allowance total 1
+ratchet: 2 cases hold at 1
+#+end_example
+
+A pull request number prints as ~#N~. A case with none prints ~-~.
+~--ratchet~ exits nonzero when a case's allowance disagrees with the
+file, when a case is missing, or when ~total~ is not the sum of
+~cases~. ~--json~ prints the case records instead of the table. An
+empty ~--scores~ directory is an error. ~eb_repro_summary~ is the same
+command. See [[file:../explanation/fidelity.org][reproduction fidelity]]
+and [[file:../reference/cli.org][the CLI reference]].

docs/orgmode/howto/site-layer-shadow.org

@@ -0,0 +1,72 @@
+#+TITLE: Report a site recipe that shadows an EESSI module
+#+OPTIONS: toc:t num:nil author:nil
+
+What it does: ~stack shadow~ compares site-layer easyconfigs with the
+modules the European Environment for Scientific Software Installations
+(EESSI) base stack already provides. ~--base~ is loaded by the same
+reader ~stack gap~ uses for ~--installed~: a directory is an Lmod tree,
+and a file is ~module -t avail~, ~module spider -t~, or ~Name/version~.
+The comparison is the recipe's own name and software version. A match
+is ~same~. Another recorded version of that name is ~different~. A base
+name with no recorded version is ~unversioned~. A name the base does
+not have is the site layer and is not reported.
+
+Why you'd use it: a site module path searched before the EESSI path
+hides the base module of the same name. The site layer is supposed to
+be the software EESSI does not install. This check fails when a recipe
+would override one that it does.
+
+* The fixture layer
+
+~tests/shadow_fixtures/easyconfigs~ has four recipes. zlib 1.3.1 is the
+version in the base tree (~1.3.1-GCCcore-15.2.0~, the spelling EasyBuild
+uses for an installed module: software version, then ~-~, then the
+toolchain). FFTW 3.3.11 is not the base's 3.3.10. ~bare.txt~ names
+OnlyName and records no version. VASP is absent from the base. The file
+~default~ under zlib is not a module version.
+
+#+begin_src bash
+eb-stack stack shadow \
+  --easyconfigs tests/shadow_fixtures/easyconfigs \
+  --base tests/shadow_fixtures/base \
+  --base tests/shadow_fixtures/bare.txt
+#+end_src
+
+That command printed:
+
+#+begin_example
+same zlib-1.3.1-GCCcore-15.2.0.eb name=zlib version=1.3.1 base=1.3.1-GCCcore-15.2.0
+different FFTW-3.3.11-gompi-2023a.eb name=FFTW version=3.3.11 base=3.3.10-gompi-2023a
+unversioned OnlyName-1.2-GCCcore-15.2.0.eb name=OnlyName version=1.2 base=-
+same=1 different=1 unversioned=1 recipes=4 base=3
+#+end_example
+
+The process exited 1. Standard error was:
+
+#+begin_example
+error: 3 site-layer recipes shadow a module the EESSI base stack provides
+#+end_example
+
+Names match without regard to case. A recipe versionsuffix has to end
+the base module version, so a CUDA variant is not the same version as
+the plain build. ~--json~ prints ~hits~, ~same~, ~different~,
+~unversioned~, ~recipes~, and ~base_names~.
+
+* A recipe the base does not provide
+
+#+begin_src bash
+eb-stack stack shadow \
+  --easyconfigs tests/shadow_fixtures/easyconfigs/v/VASP/VASP-6.5.1-foss-2026.1.eb \
+  --base tests/shadow_fixtures/base \
+  --base tests/shadow_fixtures/bare.txt
+#+end_src
+
+That command printed:
+
+#+begin_example
+same=0 different=0 unversioned=0 recipes=1 base=3
+#+end_example
+
+The process exited 0. A recipe that does not parse exits 1 and names
+the file. ~eb_stack_shadow~ is the same check. See
+[[file:../reference/cli.org][the CLI reference]].