diff --git a/.github/workflows/ci_test.yml b/.github/workflows/ci_test.yml index 2a844467..76d5049f 100644 --- a/.github/workflows/ci_test.yml +++ b/.github/workflows/ci_test.yml @@ -180,6 +180,7 @@ jobs: --test generation_lint \ --test stack_gap \ --test stack_provided \ + --test stack_shadow \ --test stack_shard \ --test hierarchy_universe \ --test jenkins_build_list \ diff --git a/AGENTS.md b/AGENTS.md index f5c91e65..a1e115b1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -73,7 +73,7 @@ lint findings remain, and do not comment on the PR. ```text package inspect | plan | bump recipe easyblock | templates | hierarchy | check | lint | format -stack solve | sbom | order | easystack | sections | from-commit | gap | provided | batch | shard | arch | files | generation | cuda-cc | known-failures | oom | bot-classify +stack solve | sbom | order | easystack | sections | from-commit | gap | provided | shadow | batch | shard | arch | files | generation | cuda-cc | known-failures | oom | bot-classify target list | doctor campaign run | status | finding claim | finding resolve repro summary diff --git a/CHANGELOG.md b/CHANGELOG.md index 07b7aabb..4cb08ad8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,14 @@ All notable changes to this project are documented here. ### Added +- `stack shadow` reports a site-layer recipe that shadows a module the + EESSI base stack already provides. `--base` is an Lmod tree or a module + listing, loaded with the same inventory reader as `stack gap`. The same + software version is `same`, another version of that name is `different`, + and a base name with no recorded version is `unversioned`. A name the + base does not have is not reported. The command exits 1 when any recipe + shadows, or when a recipe does not parse. `eb_stack_shadow` is the same + check. - `stack provided` includes the one-based build-list line on every `provided` and `unparsed` finding in its text report, as it already did for hook, toolchain, family-lock, and licensed findings. @@ -52,6 +60,23 @@ All notable changes to this project are documented here. A line that cannot be resolved exits 1 and writes nothing. `eb_stack_shard` is the same command over MCP. +### Documentation + +- How to split one Jenkins build list into shards that build at the + same time, with the shared dependencies in a prelude. +- How to plan software-layer pull requests whose builds do not overlap, + including the second-version warning a batch records. +- How to regroup a Jenkins build list under the section headers of a + previous list. +- How to write a stack lock as the YAML `eb --easystack` reads. +- How to read the easyblock class, the template values, and the + toolchain hierarchy EasyBuild derives. +- How to report the newest stable upstream release of an EasyBuild + recipe, and why a listed version was kept or dropped. +- How to read the reproduction scoreboard and hold it to a ratchet file. +- How to report a site-layer recipe that shadows a module the EESSI + base stack already provides. + ## 0.4.0 - 2026-11-01 `stack solve` in 0.3.0 took the policy file as `--policy`. The flag is diff --git a/docs/orgmode/explanation/sbom-blast-radius.org b/docs/orgmode/explanation/sbom-blast-radius.org index 3efd40d8..6487242c 100644 --- a/docs/orgmode/explanation/sbom-blast-radius.org +++ b/docs/orgmode/explanation/sbom-blast-radius.org @@ -989,7 +989,7 @@ table. ~reverse=0~ again, for the same reason as the CMakeMake run. * The same document over MCP -The Model Context Protocol (MCP) catalog has 50 tools. ~tools/list~ prints +The Model Context Protocol (MCP) catalog has 51 tools. ~tools/list~ prints their names: #+begin_src bash @@ -1027,6 +1027,7 @@ eb_stack_easystack eb_package_latest eb_stack_gap eb_stack_provided +eb_stack_shadow eb_stack_batch eb_stack_shard eb_stack_arch @@ -1051,7 +1052,7 @@ eb_easyblock_reach eb_repro_summary #+end_src -That is 50 tools. +That is 51 tools. ~eb_stack_sbom~ writes the CycloneDX file ~stack sbom~ writes. Its ~tools/list~ description reads: "Emit CycloneDX from an existing stack lock, reading dependency edges and source checksums from the easyconfigs the lock names. With backport_list, also write the modules whose reach class justifies a backport." ~eb_stack_sbom~ reads the Cbc lock: diff --git a/docs/orgmode/howto/package-latest.org b/docs/orgmode/howto/package-latest.org new file mode 100644 index 00000000..99d817d2 --- /dev/null +++ b/docs/orgmode/howto/package-latest.org @@ -0,0 +1,49 @@ +#+TITLE: Report the newest stable upstream release +#+OPTIONS: toc:t num:nil author:nil + +What it does: ~package latest~ reads each easyconfig's ~source_urls~ and +asks that backend for versions. PyPI, GitHub, GitLab, the Comprehensive +R Archive Network (CRAN), crates.io, and an HTTP directory listing are +the backends. Every listed version is kept, or dropped with a reason: a +draft, a pre-release, a yanked release, or a per-package override. +~data/upstream-overrides.toml~ is the override file the binary carries. +~--overrides~ files apply after it. + +Why you'd use it: ~package bump --latest~ moves a recipe to the version +this command picks. Run the report first when you want to see what was +dropped. + +* A directory that lists no version + +~tests/shadow_fixtures/easyconfigs/z/zlib/zlib-1.3.1-GCCcore-15.2.0.eb~ +points ~source_urls~ at ~https://example.org/~, which is not a file +listing of zlib tarballs. + +#+begin_src bash +eb-stack package latest \ + --source tests/shadow_fixtures/easyconfigs/z/zlib/zlib-1.3.1-GCCcore-15.2.0.eb +#+end_src + +That command printed: + +#+begin_example +name=zlib current=1.3.1 newest=- state=unknown backend=directory:https://example.org/ reason="the backend listed no version of the recipe's source file" +#+end_example + +Read the decision line left to right. ~current~ is the recipe version. +~newest~ is the newest version that was kept, or ~-~ when none was. +~state~ is ~current~ (the recipe is already that version), ~behind~, +~pinned~ (an override names the version to keep), or ~unknown~. The +~reason~ is why no version was chosen. + +A backend that does list versions adds one indented line per version, +~kept~ or ~dropped ()~. A pre-release is dropped. ~--json~ +prints the same rows as one array. GitHub requests send ~GITHUB_TOKEN~ +or ~GH_TOKEN~ when it is set. + +A recipe whose ~source_urls~ name PyPI, such as +~tests/sbom_recorded/rebuild/easybuild/easyconfigs/w/wrapt/wrapt-2.2.1-GCCcore-15.2.0.eb~, +uses the PyPI backend. That command queries the network, and the newest +stable release moves, so the page does not freeze a version number from +one run. ~eb_package_latest~ is the same command. See +[[file:../reference/cli.org][the CLI reference]]. diff --git a/docs/orgmode/howto/recipe-identity.org b/docs/orgmode/howto/recipe-identity.org new file mode 100644 index 00000000..3810a961 --- /dev/null +++ b/docs/orgmode/howto/recipe-identity.org @@ -0,0 +1,85 @@ +#+TITLE: Read the easyblock, templates, and hierarchy +#+OPTIONS: toc:t num:nil author:nil + +What it does: three recipe commands answer questions EasyBuild decides +before it builds. ~recipe easyblock~ prints the class name derived from +a software name. ~recipe templates~ prints the template constants, or +the values ~--name~ and ~--version~ resolve. ~recipe hierarchy~ prints +the known toolchain hierarchies as JSON. + +Why you'd use it: a recipe that sets no ~easyblock~ is claiming the +derived class exists. A template such as ~%(version_major_minor)s~ is a +value, not a string to guess. A dependency's toolchain has to be a +member of the parent's hierarchy. + +* The derived easyblock + +#+begin_src bash +eb-stack recipe easyblock zlib +eb-stack recipe easyblock GROMACS +eb-stack recipe easyblock -- c++ +#+end_src + +Those commands printed: + +#+begin_example +EB_zlib +EB_GROMACS +EB_c_plus__plus_ +#+end_example + +The third name is passed after ~--~ because it starts with punctuation. +~--charmap~ prints the substitution table as JSON instead of one class +name. The prefix is ~EB_~ and the table has 33 entries. The recorded +table is ~tests/recipe_command_fixtures/easyblock-charmap.json~. + +* Resolved templates + +~--name~ and ~--version~ together print the values the derived-key +rules produce. The toolchain in that pair is ~system~. + +#+begin_src bash +eb-stack recipe templates --name zlib --version 1.3.1 +#+end_src + +That command printed: + +#+begin_example +{ + "arch": "x86_64", + "bitbucket_account": "zlib", + "github_account": "zlib", + "name": "zlib", + "nameletter": "z", + "nameletterlower": "z", + "namelower": "zlib", + "toolchain_name": "system", + "toolchain_version": "", + "version": "1.3.1", + "version_major": "1", + "version_major_minor": "1.3", + "version_major_minor_patch": "1.3.1", + "version_minor": "3", + "version_minor_patch": "3.1", + "version_patch": "1", + "versionsuffix": "" +} +#+end_example + +~--export~ prints the constant table and the derived-key rules instead +of one resolved pair. ~arch~ follows the machine that runs the command. + +* Known hierarchies + +#+begin_src bash +eb-stack recipe hierarchy +#+end_src + +The document is the generations in +~tests/recipe_command_fixtures/hierarchy.json~. The first parent is +foss/2023b, and its members run from the system toolchain through +GCCcore, GCC, gfbf, and gompi to foss, most minimal subtoolchain first. +~--export~ adds the synthetic GCC-family rule. The plain document has +no ~synthetic~ key. ~eb_recipe_easyblock~, ~eb_recipe_templates~, and +~eb_recipe_hierarchy~ are the same commands. See +[[file:../reference/cli.org][the CLI reference]]. diff --git a/docs/orgmode/howto/repro-summary.org b/docs/orgmode/howto/repro-summary.org new file mode 100644 index 00000000..4b5fba4d --- /dev/null +++ b/docs/orgmode/howto/repro-summary.org @@ -0,0 +1,70 @@ +#+TITLE: Read the reproduction scoreboard +#+OPTIONS: toc:t num:nil author:nil + +What it does: ~repro summary~ reads every ~*.json~ case score under +~--scores~ and prints the scoreboard. ~Score~ is ~EXACT~, ~SEMANTIC~, or +~MATERIAL~. ~Allowance~ is how many hand-added lines the case still +forgives. ~Residual~ is the raw diff lines left after that allowance. +~--ratchet~ checks those allowance counts against a committed file. + +Why you'd use it: the reproduction suite writes one JSON file per case. +The table is the grind's current position, and the ratchet fails the +run when an allowance grows. + +* Two cases and a ratchet that holds + +The two files below are the shape ~repro summary~ reads. The GROMACS +case still forgives one line. ScaFaCoS has an empty allowance and a +semantic diff, so its residual in this file is 0. + +#+begin_src bash +mkdir -p /tmp/repro-scores +cat > /tmp/repro-scores/reproduces_gromacs.json <<'EOF' +{ + "case": "reproduces_gromacs", + "pull_request": null, + "source": "source.eb", + "target": "target.eb", + "score": "EXACT", + "allowance": [" ('pybind11', '2.12.0'),"], + "stale_allowance": [], + "residual_lines": 0 +} +EOF +cat > /tmp/repro-scores/reproduces_scafacos.json <<'EOF' +{ + "case": "reproduces_scafacos", + "pull_request": null, + "source": "source.eb", + "target": "target.eb", + "score": "SEMANTIC", + "allowance": [], + "stale_allowance": [], + "residual_lines": 0 +} +EOF +cat > /tmp/repro-ratchet.json <<'EOF' +{"note":"example","cases":{"reproduces_gromacs":1,"reproduces_scafacos":0},"total":1} +EOF +eb-stack repro summary --scores /tmp/repro-scores --ratchet /tmp/repro-ratchet.json +#+end_src + +That command printed: + +#+begin_example +| Case | PR | Score | Allowance | Residual | +|---+---+---+---+---| +| reproduces_gromacs | - | EXACT | 1 | 0 | +| reproduces_scafacos | - | SEMANTIC | 0 | 0 | + +2 cases: 1 EXACT, 1 SEMANTIC, 0 MATERIAL; allowance total 1 +ratchet: 2 cases hold at 1 +#+end_example + +A pull request number prints as ~#N~. A case with none prints ~-~. +~--ratchet~ exits nonzero when a case's allowance disagrees with the +file, when a case is missing, or when ~total~ is not the sum of +~cases~. ~--json~ prints the case records instead of the table. An +empty ~--scores~ directory is an error. ~eb_repro_summary~ is the same +command. See [[file:../explanation/fidelity.org][reproduction fidelity]] +and [[file:../reference/cli.org][the CLI reference]]. diff --git a/docs/orgmode/howto/site-layer-shadow.org b/docs/orgmode/howto/site-layer-shadow.org new file mode 100644 index 00000000..a2c22e3d --- /dev/null +++ b/docs/orgmode/howto/site-layer-shadow.org @@ -0,0 +1,72 @@ +#+TITLE: Report a site recipe that shadows an EESSI module +#+OPTIONS: toc:t num:nil author:nil + +What it does: ~stack shadow~ compares site-layer easyconfigs with the +modules the European Environment for Scientific Software Installations +(EESSI) base stack already provides. ~--base~ is loaded by the same +reader ~stack gap~ uses for ~--installed~: a directory is an Lmod tree, +and a file is ~module -t avail~, ~module spider -t~, or ~Name/version~. +The comparison is the recipe's own name and software version. A match +is ~same~. Another recorded version of that name is ~different~. A base +name with no recorded version is ~unversioned~. A name the base does +not have is the site layer and is not reported. + +Why you'd use it: a site module path searched before the EESSI path +hides the base module of the same name. The site layer is supposed to +be the software EESSI does not install. This check fails when a recipe +would override one that it does. + +* The fixture layer + +~tests/shadow_fixtures/easyconfigs~ has four recipes. zlib 1.3.1 is the +version in the base tree (~1.3.1-GCCcore-15.2.0~, the spelling EasyBuild +uses for an installed module: software version, then ~-~, then the +toolchain). FFTW 3.3.11 is not the base's 3.3.10. ~bare.txt~ names +OnlyName and records no version. VASP is absent from the base. The file +~default~ under zlib is not a module version. + +#+begin_src bash +eb-stack stack shadow \ + --easyconfigs tests/shadow_fixtures/easyconfigs \ + --base tests/shadow_fixtures/base \ + --base tests/shadow_fixtures/bare.txt +#+end_src + +That command printed: + +#+begin_example +same zlib-1.3.1-GCCcore-15.2.0.eb name=zlib version=1.3.1 base=1.3.1-GCCcore-15.2.0 +different FFTW-3.3.11-gompi-2023a.eb name=FFTW version=3.3.11 base=3.3.10-gompi-2023a +unversioned OnlyName-1.2-GCCcore-15.2.0.eb name=OnlyName version=1.2 base=- +same=1 different=1 unversioned=1 recipes=4 base=3 +#+end_example + +The process exited 1. Standard error was: + +#+begin_example +error: 3 site-layer recipes shadow a module the EESSI base stack provides +#+end_example + +Names match without regard to case. A recipe versionsuffix has to end +the base module version, so a CUDA variant is not the same version as +the plain build. ~--json~ prints ~hits~, ~same~, ~different~, +~unversioned~, ~recipes~, and ~base_names~. + +* A recipe the base does not provide + +#+begin_src bash +eb-stack stack shadow \ + --easyconfigs tests/shadow_fixtures/easyconfigs/v/VASP/VASP-6.5.1-foss-2026.1.eb \ + --base tests/shadow_fixtures/base \ + --base tests/shadow_fixtures/bare.txt +#+end_src + +That command printed: + +#+begin_example +same=0 different=0 unversioned=0 recipes=1 base=3 +#+end_example + +The process exited 0. A recipe that does not parse exits 1 and names +the file. ~eb_stack_shadow~ is the same check. See +[[file:../reference/cli.org][the CLI reference]]. diff --git a/docs/orgmode/howto/stack-batch.org b/docs/orgmode/howto/stack-batch.org new file mode 100644 index 00000000..b8d0733b --- /dev/null +++ b/docs/orgmode/howto/stack-batch.org @@ -0,0 +1,69 @@ +#+TITLE: Plan software-layer pull requests that do not overlap +#+OPTIONS: toc:t num:nil author:nil + +What it does: ~stack batch~ plans European Environment for Scientific +Software Installations (EESSI) software-layer pull requests whose builds +do not overlap. A root's set is its closure over the robot path minus +the installed modules, the same set ~pr-overlap~ computes for one +easystack line. Roots whose sets meet go in one pull request. A root +whose set meets an open pull request waits for that pull request. The +rest are packed ~--size~ roots per pull request. + +Why you'd use it: ~stack gap~ writes ~roots.txt~ for the recipes the +generation can already build. Those roots should not all land in one +pull request, and two pull requests must not build the same module. + +* One root beside an installed version + +~tests/batch_versions_fixtures~ has GROMACS 2025.4 installed and a +recipe for GROMACS 2026.2 on foss/2026.1, generation 2026.1 of EESSI +2026.06. One root is one batch. The second version in that generation +is a warning, not a failed plan. + +#+begin_src bash +eb-stack stack batch \ + --root GROMACS-2026.2-foss-2026.1.eb \ + --easyconfigs tests/batch_versions_fixtures/easyconfigs \ + --installed tests/batch_versions_fixtures/installed-gromacs-2025.txt \ + --eessi-version 2026.06 \ + --easybuild-version 5.4.1 \ + --generation 2026.1 \ + --generated-at 2026-10-30T00:00:00Z \ + --out-dir /tmp/batches +#+end_src + +That command printed: + +#+begin_example +batches=1 roots=1 waits=0 unresolved=0 deployed=0 digest=cd1b5856e678c2d8fbd7bc563a861edf706a9afe7cece55596aaa892a235b080 out=/tmp/batches +#+end_example + +~/tmp/batches/plan.md~ names the batch and the warning: + +#+begin_example +## batch-01-gromacs + +{2026.06}[foss/2026.1] GROMACS 2026.2 + +1 module(s) to build. + +1 second-version warning(s); `pr-body.md` has the details. +#+end_example + +The batch directory holds the EESSI easystacks, a lock, a CycloneDX +software bill of materials, ~reproduce.sh~, ~batch.json~, and +~pr-body.md~. The pull-request body includes: + +#+begin_example +Second versions in one generation, from `eb-stack stack generation`: + +GROMACS has 2 versions in generation 2026.1: 2025.4 (already in the stack), 2026.2 (added: GROMACS-2026.2-foss-2026.1.eb); pass --allow-versions GROMACS@2026.1 to accept +#+end_example + +~--allow-versions GROMACS@2026.1~ accepts that pair. +~--generated-at~ fixes the time written into the plan, so the same +inputs give the same digest. ~--roots-file~ reads the ~roots.txt~ that +~stack gap~ writes. An entry the architecture cannot build exits 1 and +writes nothing. ~eb_stack_batch~ is the same command. See +[[file:generation-lint.org][the second-version check]] and +[[file:../reference/cli.org][the CLI reference]]. diff --git a/docs/orgmode/howto/stack-easystack.org b/docs/orgmode/howto/stack-easystack.org new file mode 100644 index 00000000..7a9191b7 --- /dev/null +++ b/docs/orgmode/howto/stack-easystack.org @@ -0,0 +1,58 @@ +#+TITLE: Write an easystack from a stack lock +#+OPTIONS: toc:t num:nil author:nil + +What it does: ~stack easystack~ writes a stack lock as the YAML +~eb --easystack~ reads. Entries are easyconfig filenames. ~--option~ +attaches per-file ~eb~ options, spelled without the leading dashes. +~--eessi-version~ writes the European Environment for Scientific +Software Installations (EESSI) software-layer layout instead of one +file: one easystack per generation, ~001-system~ for the system +toolchain, and ~accel/nvidia/~ for CUDA builds. + +Why you'd use it: ~stack solve~ writes a lock. The software layer and +~eb --easystack~ do not read that lock. This command is the file they +do read. + +* One lock, one file + +~tests/provided_fixtures/baseline.lock.json~ selects GROMACS 2025.2 on +foss/2026.1. + +#+begin_src bash +eb-stack stack easystack \ + --lock tests/provided_fixtures/baseline.lock.json \ + --out /tmp/stack.yml +#+end_src + +That command printed: + +#+begin_example +easystack=/tmp/stack.yml entries=1 of=1 +#+end_example + +~/tmp/stack.yml~ is: + +#+begin_example +easyconfigs: +- GROMACS-2025.2-foss-2026.1.eb +#+end_example + +~entries~ counts filenames in the YAML. ~of~ is the number of packages +in the lock. A package whose lock entry names no ~.eb~ file is left +out: EasyBuild cannot build a file that was never on disk. + +A file with options is a one-key mapping. Repeat ~--option~ as +~: