From 4b6a187f601dfeef40716942e01d555ac72708ec Mon Sep 17 00:00:00 2001 From: Rohit Goswami Date: Tue, 3 Nov 2026 12:54:31 +0000 Subject: [PATCH 1/9] feat(stack): report site recipes that shadow an EESSI module A site-layer recipe whose name is already in the base module inventory hides that module when the site path is searched first. The `stack shadow` command loads the inventory with the same reader `stack gap` uses and exits 1 when a recipe matches the software version, another version, or a bare name. --- .github/workflows/ci_test.yml | 1 + AGENTS.md | 2 +- CHANGELOG.md | 8 + .../orgmode/explanation/sbom-blast-radius.org | 5 +- docs/orgmode/reference/cli.org | 28 +- docs/orgmode/reference/formats.org | 16 + llms-full.txt | 49 +- src/build_order.rs | 19 +- src/lib.rs | 1 + src/main.rs | 41 ++ src/mcp.rs | 27 + src/shadow.rs | 575 ++++++++++++++++++ tests/mcp_stdio.rs | 3 +- tests/mcp_v1.rs | 5 +- tests/shadow_fixtures/bare.txt | 2 + .../modules/all/FFTW/3.3.10-gompi-2023a.lua | 1 + .../modules/all/zlib/1.3.1-GCCcore-15.2.0.lua | 1 + .../base/modules/all/zlib/default | 1 + tests/shadow_fixtures/broken/Broken-1.0.eb | 1 + .../f/FFTW/FFTW-3.3.11-gompi-2023a.eb | 15 + .../o/OnlyName/OnlyName-1.2-GCCcore-15.2.0.eb | 15 + .../v/VASP/VASP-6.5.1-foss-2026.1.eb | 15 + .../z/zlib/zlib-1.3.1-GCCcore-15.2.0.eb | 15 + tests/stack_shadow.rs | 174 ++++++ 24 files changed, 1010 insertions(+), 10 deletions(-) create mode 100644 src/shadow.rs create mode 100644 tests/shadow_fixtures/bare.txt create mode 100644 tests/shadow_fixtures/base/modules/all/FFTW/3.3.10-gompi-2023a.lua create mode 100644 tests/shadow_fixtures/base/modules/all/zlib/1.3.1-GCCcore-15.2.0.lua create mode 100644 tests/shadow_fixtures/base/modules/all/zlib/default create mode 100644 tests/shadow_fixtures/broken/Broken-1.0.eb create mode 100644 tests/shadow_fixtures/easyconfigs/f/FFTW/FFTW-3.3.11-gompi-2023a.eb create mode 100644 tests/shadow_fixtures/easyconfigs/o/OnlyName/OnlyName-1.2-GCCcore-15.2.0.eb create mode 100644 tests/shadow_fixtures/easyconfigs/v/VASP/VASP-6.5.1-foss-2026.1.eb create mode 100644 tests/shadow_fixtures/easyconfigs/z/zlib/zlib-1.3.1-GCCcore-15.2.0.eb create mode 100644 tests/stack_shadow.rs diff --git a/.github/workflows/ci_test.yml b/.github/workflows/ci_test.yml index 2a844467..76d5049f 100644 --- a/.github/workflows/ci_test.yml +++ b/.github/workflows/ci_test.yml @@ -180,6 +180,7 @@ jobs: --test generation_lint \ --test stack_gap \ --test stack_provided \ + --test stack_shadow \ --test stack_shard \ --test hierarchy_universe \ --test jenkins_build_list \ diff --git a/AGENTS.md b/AGENTS.md index f5c91e65..a1e115b1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -73,7 +73,7 @@ lint findings remain, and do not comment on the PR. ```text package inspect | plan | bump recipe easyblock | templates | hierarchy | check | lint | format -stack solve | sbom | order | easystack | sections | from-commit | gap | provided | batch | shard | arch | files | generation | cuda-cc | known-failures | oom | bot-classify +stack solve | sbom | order | easystack | sections | from-commit | gap | provided | shadow | batch | shard | arch | files | generation | cuda-cc | known-failures | oom | bot-classify target list | doctor campaign run | status | finding claim | finding resolve repro summary diff --git a/CHANGELOG.md b/CHANGELOG.md index 07b7aabb..0a721731 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,14 @@ All notable changes to this project are documented here. ### Added +- `stack shadow` reports a site-layer recipe that shadows a module the + EESSI base stack already provides. `--base` is an Lmod tree or a module + listing, loaded with the same inventory reader as `stack gap`. The same + software version is `same`, another version of that name is `different`, + and a base name with no recorded version is `unversioned`. A name the + base does not have is not reported. The command exits 1 when any recipe + shadows, or when a recipe does not parse. `eb_stack_shadow` is the same + check. - `stack provided` includes the one-based build-list line on every `provided` and `unparsed` finding in its text report, as it already did for hook, toolchain, family-lock, and licensed findings. diff --git a/docs/orgmode/explanation/sbom-blast-radius.org b/docs/orgmode/explanation/sbom-blast-radius.org index 3efd40d8..6487242c 100644 --- a/docs/orgmode/explanation/sbom-blast-radius.org +++ b/docs/orgmode/explanation/sbom-blast-radius.org @@ -989,7 +989,7 @@ table. ~reverse=0~ again, for the same reason as the CMakeMake run. * The same document over MCP -The Model Context Protocol (MCP) catalog has 50 tools. ~tools/list~ prints +The Model Context Protocol (MCP) catalog has 51 tools. ~tools/list~ prints their names: #+begin_src bash @@ -1027,6 +1027,7 @@ eb_stack_easystack eb_package_latest eb_stack_gap eb_stack_provided +eb_stack_shadow eb_stack_batch eb_stack_shard eb_stack_arch @@ -1051,7 +1052,7 @@ eb_easyblock_reach eb_repro_summary #+end_src -That is 50 tools. +That is 51 tools. ~eb_stack_sbom~ writes the CycloneDX file ~stack sbom~ writes. Its ~tools/list~ description reads: "Emit CycloneDX from an existing stack lock, reading dependency edges and source checksums from the easyconfigs the lock names. With backport_list, also write the modules whose reach class justifies a backport." ~eb_stack_sbom~ reads the Cbc lock: diff --git a/docs/orgmode/reference/cli.org b/docs/orgmode/reference/cli.org index 6f4e425d..6f942494 100644 --- a/docs/orgmode/reference/cli.org +++ b/docs/orgmode/reference/cli.org @@ -17,7 +17,7 @@ reachability, toolchain family, and the claim ladder are defined in | 1 | The work was attempted and failed: a missing or unreadable file, a document that does not parse, a check that found errors, a network fetch that failed, a doctor probe, or a campaign whose build failed | | 2 | The command line was rejected before that work: a missing or unknown flag (clap), or a flag value or combination the command does not accept | -A rejected value names the flag and the value. A missing file names the flag and the path. Status 2 is the same status clap already uses. A recipe that fails ~recipe lint~ or ~recipe check~ is status 1: the files were read and the check failed. ~stack from-commit~ is status 1 when any pin is ~released~, and status 0 when every pin is ~newer~ or ~unknown~ or the files name no pin. A file name with no ~eb-X.Y.Z~, and no ~--easybuild-version~, is status 2. ~stack arch~ is status 1 when an entry is unsupported or flagged, or when ~--probe-sources~ reports ~unreachable~ or ~arch-missing~ sources. ~stack files~ is status 1 when a patch or an easyconfig is missing, and status 0 when the only lines are ~no_url~ or ~unchecked~. ~stack generation~ is status 1 on any finding, and status 2 when an easystack path names no EESSI version and ~--eessi-version~ is absent. ~stack known-failures~ is status 0 when it prints warnings: they are advice for the pull request, not a refusal; with ~--strict~ it is status 1 when a warning remains, and status 0 when every hit is remedied. ~stack oom~ is status 0 when it warns: the warnings are advice, not a failed check. A ~--node~ value that is not ~CORES:MEMORY_MIB~ or ~CORES:MEMORY_MIB:TARGET~ is status 2, and so is a command with neither ~--easystack~ nor ~--node~. ~stack bot-classify~ is status 0 for every class, and status 2 when a path is a directory. ~stack batch~ is status 1 when the plan includes, or depends on, an entry the architecture cannot build. ~stack provided~ is status 1 when a build-list line names software the baseline already provides, names no package, uses a toolchain other than 2026.1 or GCCcore-15.2.0 below it, sets ~--hooks~, or belongs in a ~--family-lock~. A group-restricted warning is status 0 unless ~--strict~. Status 2 is when neither ~--installed~ nor ~--baseline-lock~ is set. ~stack shard~ is status 1 when a build-list line cannot be resolved, and writes nothing; ~--shards 0~ and a list with no lines are status 2. +A rejected value names the flag and the value. A missing file names the flag and the path. Status 2 is the same status clap already uses. A recipe that fails ~recipe lint~ or ~recipe check~ is status 1: the files were read and the check failed. ~stack from-commit~ is status 1 when any pin is ~released~, and status 0 when every pin is ~newer~ or ~unknown~ or the files name no pin. A file name with no ~eb-X.Y.Z~, and no ~--easybuild-version~, is status 2. ~stack arch~ is status 1 when an entry is unsupported or flagged, or when ~--probe-sources~ reports ~unreachable~ or ~arch-missing~ sources. ~stack files~ is status 1 when a patch or an easyconfig is missing, and status 0 when the only lines are ~no_url~ or ~unchecked~. ~stack generation~ is status 1 on any finding, and status 2 when an easystack path names no EESSI version and ~--eessi-version~ is absent. ~stack known-failures~ is status 0 when it prints warnings: they are advice for the pull request, not a refusal; with ~--strict~ it is status 1 when a warning remains, and status 0 when every hit is remedied. ~stack oom~ is status 0 when it warns: the warnings are advice, not a failed check. A ~--node~ value that is not ~CORES:MEMORY_MIB~ or ~CORES:MEMORY_MIB:TARGET~ is status 2, and so is a command with neither ~--easystack~ nor ~--node~. ~stack bot-classify~ is status 0 for every class, and status 2 when a path is a directory. ~stack batch~ is status 1 when the plan includes, or depends on, an entry the architecture cannot build. ~stack provided~ is status 1 when a build-list line names software the baseline already provides, names no package, uses a toolchain other than 2026.1 or GCCcore-15.2.0 below it, sets ~--hooks~, or belongs in a ~--family-lock~. A group-restricted warning is status 0 unless ~--strict~. Status 2 is when neither ~--installed~ nor ~--baseline-lock~ is set. ~stack shard~ is status 1 when a build-list line cannot be resolved, and writes nothing; ~--shards 0~ and a list with no lines are status 2. ~stack shadow~ is status 1 when a site-layer recipe shadows a base-stack module, or when a recipe does not parse, and status 0 when every recipe names software the base does not provide. ~--help~ for every command has an example and a description of every flag. A default that the command applies is printed in that help. A directory that receives several artifacts is ~--out-dir~. A single output file is ~--out~ (~--lock-out~ for the lock ~stack solve~ writes). ~--installed~, ~--easyconfigs~, ~--filter-deps~, and ~--generated-at~ use those names on every command that takes them. @@ -46,6 +46,7 @@ Package commands take the toolchain as ~--toolchain name/version~, the spelling | ~stack from-commit~ | Report ~from-commit~ pins already contained in an EasyBuild release | | ~stack gap~ | Route each site module the stack lacks, and write one TSV per route | | ~stack provided~ | Fail when a build list names software a baseline already provides | +| ~stack shadow~ | Report a site-layer recipe that shadows an EESSI base-stack module | | ~stack arch~ | Refuse a plan an architecture cannot build, and flag entries never installed there | | ~stack files~ | Find the patch and source files an easystack's easyconfigs need that no tree has | | ~stack generation~ | Refuse easystack entries outside their EESSI version's toolchain generations | @@ -462,6 +463,12 @@ eb-stack stack provided \ [--licensed NAME]... [--licensed-file NAMES.txt] \ [--format text|github|json] [--json] [--strict] +eb-stack stack shadow \ + --easyconfigs tests/shadow_fixtures/easyconfigs \ + --base tests/shadow_fixtures/base \ + --base tests/shadow_fixtures/bare.txt \ + [--json] + eb-stack stack batch \ (--root FILE.eb... | --roots-file ROOTS.txt) \ --easyconfigs TREE [--easyconfigs OVERLAY]... \ @@ -715,6 +722,25 @@ hit as ~provided name= list= line=~ and ends with ~json~. ~--json~ is the same as ~--format json~. See [[file:../howto/build-list-provided.org][fail a build list that names software the baseline provides]] and [[file:../howto/module-gap.org][route the modules a stack does not have]]. +~stack shadow~ reads each site-layer easyconfig and the EESSI base +inventory. ~--base~ is loaded by the same reader as ~stack gap +--installed~: a directory is an Lmod tree (~Name/version.lua~; ~default~ +and ~.modulerc.lua~ are not modules) and a file is ~module -t avail~, +~module spider -t~, or ~Name/version~. Repeated ~--base~ paths are merged. +A recipe whose software version equals a base version, or whose base +version continues past that software version with ~-~ and the toolchain, +is ~same~. EasyBuild's installed module version is that spelling: the +software version, then ~-~ and the toolchain, then the versionsuffix. +A recipe versionsuffix has to end the base version, so a CUDA variant +is not the same version as the plain build. Another recorded version of +the same name is ~different~. A base name with an empty version set is +~unversioned~. A name the base does not have is not reported. Stdout +lists each shadow as ~ name= version= +base=~ and ends with ~same= different= unversioned= recipes= +base=~. ~--json~ prints the same fields. The command exits 1 when any +shadow is found. A recipe that does not parse exits 1 and names the file. +See [[file:../howto/site-layer-shadow.org][report a site recipe that shadows an EESSI module]]. + ~stack batch~ plans software-layer pull requests whose builds do not overlap. A root's set is its closure over the robot path minus the installed modules, as ~pr-overlap~ computes it. Roots whose sets meet ship diff --git a/docs/orgmode/reference/formats.org b/docs/orgmode/reference/formats.org index 9c49c4ac..75d550aa 100644 --- a/docs/orgmode/reference/formats.org +++ b/docs/orgmode/reference/formats.org @@ -254,6 +254,22 @@ easyconfigs: ~stack order --easystack-out~ writes the build sequence. ~stack easystack~ writes lock order, which is sorted by package name. +* Site-layer shadow (~stack shadow~) + +One line per recipe whose name the ~--base~ inventory already has, then +a counts line. ~same~ is the recipe's software version. ~different~ is +another version of that name. ~unversioned~ is a base name with no +recorded version. ~base~ on a hit is the comma-separated module versions +that decided the kind, or ~-~ when none were recorded. ~--json~ prints +~hits~, ~same~, ~different~, ~unversioned~, ~recipes~, and ~base_names~. + +#+begin_src text +same zlib-1.3.1-GCCcore-15.2.0.eb name=zlib version=1.3.1 base=1.3.1-GCCcore-15.2.0 +different FFTW-3.3.11-gompi-2023a.eb name=FFTW version=3.3.11 base=3.3.10-gompi-2023a +unversioned OnlyName-1.2-GCCcore-15.2.0.eb name=OnlyName version=1.2 base=- +same=1 different=1 unversioned=1 recipes=4 base=3 +#+end_src + * Jenkins sections (~stack sections~) The same build-list lines as the input, regrouped under ~#### Title ####~ diff --git a/llms-full.txt b/llms-full.txt index b4a7afcc..f8bdafed 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -5286,7 +5286,7 @@ reachability, toolchain family, and the claim ladder are defined in | 1 | The work was attempted and failed: a missing or unreadable file, a document that does not parse, a check that found errors, a network fetch that failed, a doctor probe, or a campaign whose build failed | | 2 | The command line was rejected before that work: a missing or unknown flag (clap), or a flag value or combination the command does not accept | -A rejected value names the flag and the value. A missing file names the flag and the path. Status 2 is the same status clap already uses. A recipe that fails `recipe lint` or `recipe check` is status 1: the files were read and the check failed. `stack from-commit` is status 1 when any pin is `released`, and status 0 when every pin is `newer` or `unknown` or the files name no pin. A file name with no `eb-X.Y.Z`, and no `--easybuild-version`, is status 2. `stack arch` is status 1 when an entry is unsupported or flagged, or when `--probe-sources` reports `unreachable` or `arch-missing` sources. `stack files` is status 1 when a patch or an easyconfig is missing, and status 0 when the only lines are `no_url` or `unchecked`. `stack generation` is status 1 on any finding, and status 2 when an easystack path names no EESSI version and `--eessi-version` is absent. `stack known-failures` is status 0 when it prints warnings: they are advice for the pull request, not a refusal; with `--strict` it is status 1 when a warning remains, and status 0 when every hit is remedied. `stack oom` is status 0 when it warns: the warnings are advice, not a failed check. A `--node` value that is not `CORES:MEMORY_MIB` or `CORES:MEMORY_MIB:TARGET` is status 2, and so is a command with neither `--easystack` nor `--node`. `stack bot-classify` is status 0 for every class, and status 2 when a path is a directory. `stack batch` is status 1 when the plan includes, or depends on, an entry the architecture cannot build. `stack provided` is status 1 when a build-list line names software the baseline already provides, names no package, uses a toolchain other than 2026.1 or GCCcore-15.2.0 below it, sets `--hooks`, or belongs in a `--family-lock`. A group-restricted warning is status 0 unless `--strict`. Status 2 is when neither `--installed` nor `--baseline-lock` is set. `stack shard` is status 1 when a build-list line cannot be resolved, and writes nothing; `--shards 0` and a list with no lines are status 2. +A rejected value names the flag and the value. A missing file names the flag and the path. Status 2 is the same status clap already uses. A recipe that fails `recipe lint` or `recipe check` is status 1: the files were read and the check failed. `stack from-commit` is status 1 when any pin is `released`, and status 0 when every pin is `newer` or `unknown` or the files name no pin. A file name with no `eb-X.Y.Z`, and no `--easybuild-version`, is status 2. `stack arch` is status 1 when an entry is unsupported or flagged, or when `--probe-sources` reports `unreachable` or `arch-missing` sources. `stack files` is status 1 when a patch or an easyconfig is missing, and status 0 when the only lines are `no_url` or `unchecked`. `stack generation` is status 1 on any finding, and status 2 when an easystack path names no EESSI version and `--eessi-version` is absent. `stack known-failures` is status 0 when it prints warnings: they are advice for the pull request, not a refusal; with `--strict` it is status 1 when a warning remains, and status 0 when every hit is remedied. `stack oom` is status 0 when it warns: the warnings are advice, not a failed check. A `--node` value that is not `CORES:MEMORY_MIB` or `CORES:MEMORY_MIB:TARGET` is status 2, and so is a command with neither `--easystack` nor `--node`. `stack bot-classify` is status 0 for every class, and status 2 when a path is a directory. `stack batch` is status 1 when the plan includes, or depends on, an entry the architecture cannot build. `stack provided` is status 1 when a build-list line names software the baseline already provides, names no package, uses a toolchain other than 2026.1 or GCCcore-15.2.0 below it, sets `--hooks`, or belongs in a `--family-lock`. A group-restricted warning is status 0 unless `--strict`. Status 2 is when neither `--installed` nor `--baseline-lock` is set. `stack shard` is status 1 when a build-list line cannot be resolved, and writes nothing; `--shards 0` and a list with no lines are status 2. `stack shadow` is status 1 when a site-layer recipe shadows a base-stack module, or when a recipe does not parse, and status 0 when every recipe names software the base does not provide. `--help` for every command has an example and a description of every flag. A default that the command applies is printed in that help. A directory that receives several artifacts is `--out-dir`. A single output file is `--out` (`--lock-out` for the lock `stack solve` writes). `--installed`, `--easyconfigs`, `--filter-deps`, and `--generated-at` use those names on every command that takes them. @@ -5315,6 +5315,7 @@ Package commands take the toolchain as `--toolchain name/version`, the spelling | `stack from-commit` | Report `from-commit` pins already contained in an EasyBuild release | | `stack gap` | Route each site module the stack lacks, and write one TSV per route | | `stack provided` | Fail when a build list names software a baseline already provides | +| `stack shadow` | Report a site-layer recipe that shadows an EESSI base-stack module | | `stack arch` | Refuse a plan an architecture cannot build, and flag entries never installed there | | `stack files` | Find the patch and source files an easystack's easyconfigs need that no tree has | | `stack generation` | Refuse easystack entries outside their EESSI version's toolchain generations | @@ -5731,6 +5732,12 @@ eb-stack stack provided \ [--licensed NAME]... [--licensed-file NAMES.txt] \ [--format text|github|json] [--json] [--strict] +eb-stack stack shadow \ + --easyconfigs tests/shadow_fixtures/easyconfigs \ + --base tests/shadow_fixtures/base \ + --base tests/shadow_fixtures/bare.txt \ + [--json] + eb-stack stack batch \ (--root FILE.eb... | --roots-file ROOTS.txt) \ --easyconfigs TREE [--easyconfigs OVERLAY]... \ @@ -5984,6 +5991,25 @@ hit as `provided name= list= line=` and ends with `json`. `--json` is the same as `--format json`. See [fail a build list that names software the baseline provides](https://eb-stack.rgoswami.me/../howto/build-list-provided.html.md) and [route the modules a stack does not have](https://eb-stack.rgoswami.me/../howto/module-gap.html.md). +`stack shadow` reads each site-layer easyconfig and the EESSI base +inventory. `--base` is loaded by the same reader as ~stack gap +--installed`: a directory is an Lmod tree (`Name/version.lua`; `default~ +and `.modulerc.lua` are not modules) and a file is `module -t avail`, +`module spider -t`, or `Name/version`. Repeated `--base` paths are merged. +A recipe whose software version equals a base version, or whose base +version continues past that software version with `-` and the toolchain, +is `same`. EasyBuild's installed module version is that spelling: the +software version, then `-` and the toolchain, then the versionsuffix. +A recipe versionsuffix has to end the base version, so a CUDA variant +is not the same version as the plain build. Another recorded version of +the same name is `different`. A base name with an empty version set is +`unversioned`. A name the base does not have is not reported. Stdout +lists each shadow as ~ name` version` +base`~ and ends with ~same` different` unversioned` recipes= +base=`. `--json~ prints the same fields. The command exits 1 when any +shadow is found. A recipe that does not parse exits 1 and names the file. +See [report a site recipe that shadows an EESSI module](https://eb-stack.rgoswami.me/../howto/site-layer-shadow.html.md). + `stack batch` plans software-layer pull requests whose builds do not overlap. A root's set is its closure over the robot path minus the installed modules, as `pr-overlap` computes it. Roots whose sets meet ship @@ -8376,6 +8402,22 @@ easyconfigs: `stack order --easystack-out` writes the build sequence. `stack easystack` writes lock order, which is sorted by package name. +# Site-layer shadow (`stack shadow`) + +One line per recipe whose name the `--base` inventory already has, then +a counts line. `same` is the recipe's software version. `different` is +another version of that name. `unversioned` is a base name with no +recorded version. `base` on a hit is the comma-separated module versions +that decided the kind, or `-` when none were recorded. `--json` prints +`hits`, `same`, `different`, `unversioned`, `recipes`, and `base_names`. + +```text +same zlib-1.3.1-GCCcore-15.2.0.eb name=zlib version=1.3.1 base=1.3.1-GCCcore-15.2.0 +different FFTW-3.3.11-gompi-2023a.eb name=FFTW version=3.3.11 base=3.3.10-gompi-2023a +unversioned OnlyName-1.2-GCCcore-15.2.0.eb name=OnlyName version=1.2 base=- +same=1 different=1 unversioned=1 recipes=4 base=3 +``` + # Jenkins sections (`stack sections`) The same build-list lines as the input, regrouped under `#### Title ####` @@ -11048,7 +11090,7 @@ table. `reverse=0` again, for the same reason as the CMakeMake run. # The same document over MCP -The Model Context Protocol (MCP) catalog has 50 tools. `tools/list` prints +The Model Context Protocol (MCP) catalog has 51 tools. `tools/list` prints their names: ```bash @@ -11086,6 +11128,7 @@ eb_stack_easystack eb_package_latest eb_stack_gap eb_stack_provided +eb_stack_shadow eb_stack_batch eb_stack_shard eb_stack_arch @@ -11110,7 +11153,7 @@ eb_easyblock_reach eb_repro_summary ``` -That is 50 tools. +That is 51 tools. `eb_stack_sbom` writes the CycloneDX file `stack sbom` writes. Its `tools/list` description reads: "Emit CycloneDX from an existing stack lock, reading dependency edges and source checksums from the easyconfigs the lock names. With backport_list, also write the modules whose reach class justifies a backport." `eb_stack_sbom` reads the Cbc lock: diff --git a/src/build_order.rs b/src/build_order.rs index e0e552e1..b598ae0e 100644 --- a/src/build_order.rs +++ b/src/build_order.rs @@ -321,14 +321,22 @@ pub enum Choice { Oldest, } +/// Between the software version and the toolchain name, and between the +/// toolchain name and the toolchain version, in an installed module version. +pub(crate) const MODULE_VERSION_TOOLCHAIN_SEPARATOR: &str = "-"; + /// The part of a module name after the `/`: version, then the toolchain unless /// it is the system one, then the versionsuffix. +/// +/// [`installed_module_version`] is the same string. A system toolchain +/// contributes no toolchain segment. fn module_version(candidate: &Candidate) -> String { + let sep = MODULE_VERSION_TOOLCHAIN_SEPARATOR; let toolchain = if crate::hierarchy::is_system_toolchain(&candidate.toolchain) { String::new() } else { format!( - "-{}-{}", + "{sep}{}{sep}{}", candidate.toolchain.name, candidate.toolchain.version ) }; @@ -339,6 +347,15 @@ fn module_version(candidate: &Candidate) -> String { ) } +/// Module version EasyBuild installs for `candidate`. +/// +/// The software version, then the toolchain unless it is the system +/// toolchain, then the versionsuffix. `stack shadow` compares a base-stack +/// module version with this spelling. +pub(crate) fn installed_module_version(candidate: &Candidate) -> String { + module_version(candidate) +} + /// Whether a candidate satisfies one stated dependency. /// /// The rules are EasyBuild's: a version requirement, an optional toolchain that diff --git a/src/lib.rs b/src/lib.rs index b926b312..f41a5da0 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -78,6 +78,7 @@ pub mod repro_report; pub mod resolvo_provider; pub mod sbom; pub mod select; +pub mod shadow; pub mod shard; pub mod site_commands; pub mod source_fetch; diff --git a/src/main.rs b/src/main.rs index d66542ba..aa95d0e6 100644 --- a/src/main.rs +++ b/src/main.rs @@ -1255,6 +1255,31 @@ enum StackCommand { #[arg(long)] strict: bool, }, + /// Report site-layer recipes that shadow a module the EESSI base stack provides. + /// + /// `--base` is an Lmod tree or a module listing, loaded the same way + /// `stack gap` loads `--installed`. A recipe whose software version is + /// already in that inventory is `same`. Another version of a base name + /// is `different`. A base name with no recorded version is `unversioned`. + /// A name the base does not have is the site layer and is not reported. + /// Exits 1 when any recipe shadows. A recipe that does not parse exits 1 + /// and names the file. + #[command( + after_long_help = "Examples:\n eb-stack stack shadow --easyconfigs tests/shadow_fixtures/easyconfigs --base tests/shadow_fixtures/base --base tests/shadow_fixtures/bare.txt" + )] + Shadow { + /// Site-layer easyconfig tree, or one `.eb` file. Repeatable. + #[arg(long, required = true)] + easyconfigs: Vec, + /// EESSI base module tree or listing. A directory is walked. A file + /// is `module -t avail`, `module spider -t`, or `Name/version`. + /// Repeatable. + #[arg(long, required = true)] + base: Vec, + /// Print the report as JSON. + #[arg(long)] + json: bool, + }, /// Plan software-layer pull requests whose builds do not overlap. /// /// Each root's new-module set is its closure over the robot path minus the @@ -2959,6 +2984,22 @@ fn run_stack(command: StackCommand) -> Result<()> { } Ok(()) } + StackCommand::Shadow { + easyconfigs, + base, + json, + } => { + let report = eb_stack::shadow::run(&eb_stack::shadow::ShadowRequest { + easyconfigs, + base, + json, + })?; + print!("{}", report.stdout); + if let Some(message) = report.failure() { + bail!(message); + } + Ok(()) + } StackCommand::Batch(args) => { let BatchArgs { mut roots, diff --git a/src/mcp.rs b/src/mcp.rs index 854535e7..32e9bf24 100644 --- a/src/mcp.rs +++ b/src/mcp.rs @@ -419,6 +419,12 @@ fn tool_catalog() -> Vec { ("strict", "boolean"), ], ), + tool_with_optional( + "eb_stack_shadow", + "Report site-layer recipes that shadow a module the EESSI base stack already provides, as `stack shadow` does. base is an Lmod tree or a module listing, loaded the same way `stack gap` loads installed. A recipe whose software version is already in that inventory is same; another version of a base name is different; a base name with no recorded version is unversioned. A name the base does not have is not reported. A shadow, or a recipe that does not parse, is a tool error.", + &[("easyconfigs", "array_min1"), ("base", "array_min1")], + &[("json", "boolean")], + ), tool_with_optional( "eb_stack_batch", "Plan software-layer pull requests whose builds do not overlap, as `stack batch` does. Each batch directory gets the EESSI easystacks, a lock, a CycloneDX SBOM, reproduce.sh, batch.json and a pull-request body.", @@ -829,6 +835,7 @@ fn call_tool(name: &str, arguments: &Value) -> Result { "eb_package_latest" => package_latest_tool(arguments), "eb_stack_gap" => stack_gap_tool(arguments), "eb_stack_provided" => stack_provided_tool(arguments), + "eb_stack_shadow" => stack_shadow_tool(arguments), "eb_stack_batch" => stack_batch_tool(arguments), "eb_stack_shard" => stack_shard_tool(arguments), "eb_stack_arch" => stack_arch_tool(arguments), @@ -3085,6 +3092,26 @@ fn stack_gap_tool(arguments: &Value) -> Result { })) } +fn stack_shadow_tool(arguments: &Value) -> Result { + let report = crate::shadow::run(&crate::shadow::ShadowRequest { + easyconfigs: path_array(arguments, "easyconfigs")?, + base: path_array(arguments, "base")?, + json: optional_bool(arguments, "json"), + }) + .map_err(workflow_error)?; + if let Some(message) = report.failure() { + return Err(format!("{}\n{message}", report.stdout)); + } + Ok(json!({ + "text": report.stdout, + "same": report.count(crate::shadow::SHADOW_SAME), + "different": report.count(crate::shadow::SHADOW_DIFFERENT), + "unversioned": report.count(crate::shadow::SHADOW_UNVERSIONED), + "recipes": report.recipes, + "base": report.base_names, + })) +} + fn stack_provided_tool(arguments: &Value) -> Result { let json = optional_bool(arguments, "json"); let format = match optional_string(arguments, "format") { diff --git a/src/shadow.rs b/src/shadow.rs new file mode 100644 index 00000000..21cce280 --- /dev/null +++ b/src/shadow.rs @@ -0,0 +1,575 @@ +//! Report site-layer recipes that shadow a module the EESSI base stack provides. +//! +//! A site layer is the software EESSI does not install. When the site module +//! path is searched first, a recipe whose name is already a base-stack module +//! hides that module. `stack shadow` reads the site recipes and the base +//! inventory from [`crate::gap::inventory_from_module_source`], the loader +//! `stack gap` uses for `--installed`, and reports each shadow. +//! +//! EasyBuild's installed module version is the software version, then +//! [`crate::build_order::MODULE_VERSION_TOOLCHAIN_SEPARATOR`], then the +//! toolchain, then the versionsuffix. See +//! [`crate::build_order::installed_module_version`]. A base version is the +//! same software version when it equals the recipe version (plus its +//! versionsuffix) or continues with that separator. A recipe versionsuffix +//! also has to end the base version, so a CUDA variant does not count as the +//! same version as the plain build. A name the base lists with an empty +//! version set is `unversioned`: the loader records a bare name that way. + +use crate::build_order::{installed_module_version, MODULE_VERSION_TOOLCHAIN_SEPARATOR}; +use crate::cli_parse::usage; +use crate::domain::Candidate; +use crate::gap::{inventory_from_module_source, ModuleInventory}; +use anyhow::{bail, Context, Result}; +use serde::Serialize; +use std::collections::BTreeSet; +use std::path::{Path, PathBuf}; + +/// The recipe's software version is one the base stack already installs. +pub const SHADOW_SAME: &str = "same"; + +/// The base stack installs this name at a different software version. +pub const SHADOW_DIFFERENT: &str = "different"; + +/// The base stack lists this name and records no version. +pub const SHADOW_UNVERSIONED: &str = "unversioned"; + +/// Report order: an exact version, then another version, then a bare name. +pub const SHADOW_KINDS: &[&str] = &[SHADOW_SAME, SHADOW_DIFFERENT, SHADOW_UNVERSIONED]; + +/// Marker for a field with nothing to print. `gap::render_tsv` uses it too. +const ABSENT: &str = "-"; + +/// Arguments for `stack shadow`. +pub struct ShadowRequest { + /// Site-layer easyconfig trees, or individual `.eb` files. Repeatable. + pub easyconfigs: Vec, + /// EESSI base module trees or listings. Repeatable. Merged the way + /// `stack gap` merges repeated `--installed` paths. + pub base: Vec, + /// Print one JSON document instead of the text report. + pub json: bool, +} + +/// One site recipe whose name the base stack already provides. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub struct ShadowHit { + /// [`SHADOW_SAME`], [`SHADOW_DIFFERENT`], or [`SHADOW_UNVERSIONED`]. + pub kind: String, + /// Easyconfig path the recipe was parsed from. + pub path: String, + /// Package name, as the recipe spells it. + pub name: String, + /// Software version, without the versionsuffix. + pub version: String, + /// Recipe versionsuffix, including its leading separator when set. + #[serde(skip_serializing_if = "Option::is_none")] + pub versionsuffix: Option, + /// Base module versions that decided the kind. Matching versions for + /// [`SHADOW_SAME`]; every recorded version for [`SHADOW_DIFFERENT`]; + /// empty for [`SHADOW_UNVERSIONED`]. + pub base: Vec, +} + +/// What `stack shadow` found. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub struct ShadowReport { + /// Shadows, in [`SHADOW_KINDS`] order, then by name and path. + pub hits: Vec, + /// Site recipes that parsed. + pub recipes: usize, + /// Distinct base-stack names, matched without regard to case. + pub base_names: usize, + /// Text for stdout. JSON when the request asked for it. + #[serde(skip)] + pub stdout: String, +} + +impl ShadowReport { + /// How many hits have this kind. + pub fn count(&self, kind: &str) -> usize { + self.hits.iter().filter(|hit| hit.kind == kind).count() + } + + /// The stderr line when a recipe shadows the base, or `None` when none do. + pub fn failure(&self) -> Option { + let n = self.hits.len(); + if n == 0 { + return None; + } + Some(format!( + "{n} site-layer {} {} a module the EESSI base stack provides", + plural(n, "recipe", "recipes"), + plural(n, "shadows", "shadow"), + )) + } +} + +fn plural<'a>(n: usize, one: &'a str, many: &'a str) -> &'a str { + if n == 1 { + one + } else { + many + } +} + +fn kind_rank(kind: &str) -> usize { + SHADOW_KINDS + .iter() + .position(|name| *name == kind) + .unwrap_or(SHADOW_KINDS.len()) +} + +fn basename(path: &str) -> &str { + path.rsplit(['/', '\\']).next().unwrap_or(path) +} + +/// Whether `module_version` is the recipe's software version. +/// +/// The base string comes from [`inventory_from_module_source`], which keeps +/// the Lmod spelling, toolchain included. Equality covers a listing that +/// stored only the software version (and versionsuffix). A longer module +/// version matches when it continues with +/// [`MODULE_VERSION_TOOLCHAIN_SEPARATOR`], which is how +/// [`installed_module_version`] attaches the toolchain. A recipe +/// versionsuffix has to end that module version: the suffix is part of the +/// version token, as `Candidate` stores it (for example `-CUDA-12.6.0`). +pub fn same_software_version( + version: &str, + versionsuffix: Option<&str>, + module_version: &str, +) -> bool { + let suffix = versionsuffix.unwrap_or(""); + let token = format!("{version}{suffix}"); + if module_version == token { + return true; + } + let prefixed = format!("{version}{MODULE_VERSION_TOOLCHAIN_SEPARATOR}"); + module_version.starts_with(&prefixed) && (suffix.is_empty() || module_version.ends_with(suffix)) +} + +fn versions_for(inventory: &ModuleInventory, name: &str) -> Option> { + let mut found = false; + let mut versions = BTreeSet::new(); + for (key, set) in inventory { + if key.eq_ignore_ascii_case(name) { + found = true; + versions.extend(set.iter().cloned()); + } + } + found.then_some(versions) +} + +fn merge_inventory(into: &mut ModuleInventory, from: ModuleInventory) { + for (name, versions) in from { + let key = into + .keys() + .find(|key| key.eq_ignore_ascii_case(&name)) + .cloned(); + match key { + Some(key) => { + into.get_mut(&key) + .expect("key was just found in the inventory") + .extend(versions); + } + None => { + into.insert(name, versions); + } + } + } +} + +/// Load every `--base` path and merge them into one inventory. +/// +/// Each path goes through [`inventory_from_module_source`]. Names that +/// differ only by case share one key, the spelling of the first path that +/// named it. +pub fn load_base(paths: &[PathBuf]) -> Result { + if paths.is_empty() { + return Err(usage("pass at least one --base module tree")); + } + let mut inventory = ModuleInventory::new(); + for path in paths { + let loaded = inventory_from_module_source(path) + .with_context(|| format!("--base {}", path.display()))?; + merge_inventory(&mut inventory, loaded); + } + Ok(inventory) +} + +fn load_recipes(paths: &[PathBuf]) -> Result> { + if paths.is_empty() { + return Err(usage("pass at least one --easyconfigs tree")); + } + let mut directories: Vec<&Path> = Vec::new(); + let mut files = Vec::new(); + for path in paths { + if path.is_file() { + files.push(path); + } else { + directories.push(path.as_path()); + } + } + let mut candidates = Vec::new(); + if !directories.is_empty() { + let parsed = crate::parse_easyconfig_trees(&directories).map_err(|error| { + anyhow::Error::msg(error).context(format!( + "--easyconfigs {}", + directories + .iter() + .map(|path| path.display().to_string()) + .collect::>() + .join(" ") + )) + })?; + if !parsed.skipped.is_empty() { + let n = parsed.skipped.len(); + let lines = parsed + .skipped + .iter() + .map(|skipped| format!("{}: {}", skipped.path, skipped.error)) + .collect::>() + .join("\n"); + bail!( + "{n} site-layer {} did not parse:\n{lines}", + plural(n, "recipe", "recipes") + ); + } + candidates.extend(parsed.candidates); + } + for path in files { + let candidate = crate::parse_easyconfig_file(path).map_err(|error| { + anyhow::Error::msg(error).context(format!("--easyconfigs {}", path.display())) + })?; + candidates.push(candidate); + } + Ok(candidates) +} + +/// Classify site recipes against a base inventory. Names match without +/// regard to case. A name the inventory does not have is the site layer +/// and is left out. +pub fn classify(recipes: &[Candidate], base: &ModuleInventory) -> Vec { + let mut hits = Vec::new(); + for recipe in recipes { + let Some(recorded) = versions_for(base, &recipe.name) else { + continue; + }; + let installed = installed_module_version(recipe); + let matched: Vec = recorded + .iter() + .filter(|module_version| { + *module_version == &installed + || same_software_version( + &recipe.version, + recipe.versionsuffix.as_deref(), + module_version, + ) + }) + .cloned() + .collect(); + let (kind, base_versions) = if recorded.is_empty() { + (SHADOW_UNVERSIONED, Vec::new()) + } else if !matched.is_empty() { + (SHADOW_SAME, matched) + } else { + (SHADOW_DIFFERENT, recorded.into_iter().collect()) + }; + hits.push(ShadowHit { + kind: kind.to_string(), + path: recipe.easyconfig_path.clone(), + name: recipe.name.clone(), + version: recipe.version.clone(), + versionsuffix: recipe.versionsuffix.clone(), + base: base_versions, + }); + } + hits.sort_by(|left, right| { + kind_rank(&left.kind) + .cmp(&kind_rank(&right.kind)) + .then_with(|| left.name.to_lowercase().cmp(&right.name.to_lowercase())) + .then_with(|| left.version.cmp(&right.version)) + .then_with(|| left.path.cmp(&right.path)) + }); + hits +} + +fn version_token(hit: &ShadowHit) -> String { + format!( + "{}{}", + hit.version, + hit.versionsuffix.as_deref().unwrap_or("") + ) +} + +fn render_text(report: &ShadowReport) -> String { + let mut out = String::new(); + for hit in &report.hits { + let base = if hit.base.is_empty() { + ABSENT.to_string() + } else { + hit.base.join(",") + }; + out.push_str(&format!( + "{} {} name={} version={} base={base}\n", + hit.kind, + basename(&hit.path), + hit.name, + version_token(hit), + )); + } + out.push_str(&format!( + "same={} different={} unversioned={} recipes={} base={}\n", + report.count(SHADOW_SAME), + report.count(SHADOW_DIFFERENT), + report.count(SHADOW_UNVERSIONED), + report.recipes, + report.base_names, + )); + out +} + +fn render_json(report: &ShadowReport) -> String { + let mut value = serde_json::to_value(report).unwrap_or_default(); + if let Some(object) = value.as_object_mut() { + object.insert("same".into(), serde_json::json!(report.count(SHADOW_SAME))); + object.insert( + "different".into(), + serde_json::json!(report.count(SHADOW_DIFFERENT)), + ); + object.insert( + "unversioned".into(), + serde_json::json!(report.count(SHADOW_UNVERSIONED)), + ); + } + let mut text = serde_json::to_string_pretty(&value).unwrap_or_default(); + text.push('\n'); + text +} + +/// Read the site recipes and the base inventory, and report every shadow. +pub fn run(request: &ShadowRequest) -> Result { + let base = load_base(&request.base)?; + let recipes = load_recipes(&request.easyconfigs)?; + let hits = classify(&recipes, &base); + let mut report = ShadowReport { + hits, + recipes: recipes.len(), + base_names: base.len(), + stdout: String::new(), + }; + report.stdout = if request.json { + render_json(&report) + } else { + render_text(&report) + }; + Ok(report) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::domain::{Candidate, Toolchain}; + use std::collections::BTreeSet; + + fn recipe( + name: &str, + version: &str, + toolchain: (&str, &str), + versionsuffix: Option<&str>, + path: &str, + ) -> Candidate { + Candidate { + easyblock: None, + name: name.into(), + version: version.into(), + toolchain: Toolchain { + name: toolchain.0.into(), + version: toolchain.1.into(), + }, + versionsuffix: versionsuffix.map(str::to_string), + easyconfig_path: path.into(), + dependencies: Vec::new(), + builddependencies: Vec::new(), + exts_list: Vec::new(), + moduleclass: None, + } + } + + fn inventory(rows: &[(&str, &[&str])]) -> ModuleInventory { + let mut inventory = ModuleInventory::new(); + for (name, versions) in rows { + inventory.insert( + (*name).to_string(), + versions + .iter() + .map(|version| (*version).to_string()) + .collect(), + ); + } + inventory + } + + #[test] + fn the_installed_module_version_is_the_same_software_version() { + let zlib = recipe( + "zlib", + "1.3.1", + ("GCCcore", "15.2.0"), + None, + "zlib-1.3.1-GCCcore-15.2.0.eb", + ); + let full = installed_module_version(&zlib); + assert_eq!(full, "1.3.1-GCCcore-15.2.0"); + assert!(same_software_version("1.3.1", None, &full)); + let hits = classify(&[zlib], &inventory(&[("zlib", &[&full])])); + assert_eq!(hits.len(), 1); + assert_eq!(hits[0].kind, SHADOW_SAME); + assert_eq!(hits[0].base, vec![full]); + } + + #[test] + fn a_longer_version_does_not_match_a_shorter_one() { + assert!(!same_software_version("1.3", None, "1.3.1-GCCcore-15.2.0")); + assert!(same_software_version("1.3.1", None, "1.3.1")); + assert!(same_software_version("1.3.1", None, "1.3.1-GCCcore-15.2.0")); + } + + #[test] + fn a_versionsuffix_has_to_end_the_base_module_version() { + let suffix = "-CUDA-12.6.0"; + assert!(same_software_version( + "2025.1", + Some(suffix), + "2025.1-foss-2026.1-CUDA-12.6.0" + )); + assert!(!same_software_version( + "2025.1", + Some(suffix), + "2025.1-foss-2026.1" + )); + let lammps = recipe( + "LAMMPS", + "2025.1", + ("foss", "2026.1"), + Some(suffix), + "LAMMPS-2025.1-foss-2026.1-CUDA-12.6.0.eb", + ); + assert_eq!( + installed_module_version(&lammps), + "2025.1-foss-2026.1-CUDA-12.6.0" + ); + let hits = classify( + &[lammps], + &inventory(&[("LAMMPS", &["2025.1-foss-2026.1"])]), + ); + assert_eq!(hits[0].kind, SHADOW_DIFFERENT); + } + + #[test] + fn classify_reports_same_different_and_unversioned_and_skips_site_only() { + let recipes = vec![ + recipe( + "VASP", + "6.5.1", + ("foss", "2026.1"), + None, + "VASP-6.5.1-foss-2026.1.eb", + ), + recipe( + "FFTW", + "3.3.11", + ("gompi", "2023a"), + None, + "FFTW-3.3.11-gompi-2023a.eb", + ), + recipe( + "zlib", + "1.3.1", + ("GCCcore", "15.2.0"), + None, + "zlib-1.3.1-GCCcore-15.2.0.eb", + ), + recipe( + "OnlyName", + "1.2", + ("GCCcore", "15.2.0"), + None, + "OnlyName-1.2-GCCcore-15.2.0.eb", + ), + ]; + let base = inventory(&[ + ("Zlib", &["1.3.1-GCCcore-13.2.0"]), + ("FFTW", &["3.3.10-gompi-2023a"]), + ("OnlyName", &[]), + ]); + let hits = classify(&recipes, &base); + assert_eq!( + hits.iter().map(|hit| hit.kind.as_str()).collect::>(), + vec![SHADOW_SAME, SHADOW_DIFFERENT, SHADOW_UNVERSIONED] + ); + assert_eq!(hits[0].name, "zlib"); + assert_eq!(hits[1].name, "FFTW"); + assert_eq!(hits[1].base, vec!["3.3.10-gompi-2023a".to_string()]); + assert!(hits[2].base.is_empty()); + } + + #[test] + fn merged_base_paths_fold_case_and_versions() { + let mut into = inventory(&[("zlib", &["1.2.13-GCCcore-12.3.0"])]); + merge_inventory(&mut into, inventory(&[("Zlib", &["1.3.1-GCCcore-15.2.0"])])); + assert_eq!(into.len(), 1); + let versions = into.get("zlib").expect("first spelling"); + assert_eq!( + versions, + &BTreeSet::from([ + "1.2.13-GCCcore-12.3.0".to_string(), + "1.3.1-GCCcore-15.2.0".to_string(), + ]) + ); + } + + #[test] + fn the_text_report_names_each_shadow_and_the_failure() { + let recipes = vec![recipe( + "zlib", + "1.3.1", + ("GCCcore", "15.2.0"), + None, + "easyconfigs/z/zlib/zlib-1.3.1-GCCcore-15.2.0.eb", + )]; + let base = inventory(&[("zlib", &["1.3.1-GCCcore-15.2.0"])]); + let report = ShadowReport { + hits: classify(&recipes, &base), + recipes: recipes.len(), + base_names: base.len(), + stdout: String::new(), + }; + let text = render_text(&report); + assert_eq!( + text, + "\ +same zlib-1.3.1-GCCcore-15.2.0.eb name=zlib version=1.3.1 base=1.3.1-GCCcore-15.2.0 +same=1 different=0 unversioned=0 recipes=1 base=1 +" + ); + assert_eq!( + report.failure().as_deref(), + Some("1 site-layer recipe shadows a module the EESSI base stack provides") + ); + } + + #[test] + fn an_empty_site_layer_has_no_failure() { + let report = ShadowReport { + hits: Vec::new(), + recipes: 1, + base_names: 2, + stdout: String::new(), + }; + assert!(report.failure().is_none()); + assert!( + render_text(&report).ends_with("same=0 different=0 unversioned=0 recipes=1 base=2\n") + ); + } +} diff --git a/tests/mcp_stdio.rs b/tests/mcp_stdio.rs index bfbd18d6..763f8812 100644 --- a/tests/mcp_stdio.rs +++ b/tests/mcp_stdio.rs @@ -329,6 +329,7 @@ fn mcp_stdio_catalog_lists_the_overlap_and_reach_tools() { "eb_package_latest", "eb_stack_gap", "eb_stack_provided", + "eb_stack_shadow", "eb_stack_batch", "eb_stack_shard", "eb_stack_arch", @@ -347,7 +348,7 @@ fn mcp_stdio_catalog_lists_the_overlap_and_reach_tools() { ] { assert!(names.contains(&expected), "missing {expected}: {names:?}"); } - assert_eq!(names.len(), 50, "{names:?}"); + assert_eq!(names.len(), 51, "{names:?}"); } fn software_layer_prs() -> Vec { diff --git a/tests/mcp_v1.rs b/tests/mcp_v1.rs index 5a91c43d..bdbac047 100644 --- a/tests/mcp_v1.rs +++ b/tests/mcp_v1.rs @@ -145,6 +145,7 @@ fn mcp_catalog_matches_the_version_one_workflows() { "eb_package_latest", "eb_stack_gap", "eb_stack_provided", + "eb_stack_shadow", "eb_stack_batch", "eb_stack_shard", "eb_stack_arch", @@ -171,7 +172,7 @@ fn mcp_catalog_matches_the_version_one_workflows() { ] { assert!(names.contains(&expected), "missing {expected}: {names:?}"); } - assert_eq!(names.len(), 50, "unexpected MCP tools: {names:?}"); + assert_eq!(names.len(), 51, "unexpected MCP tools: {names:?}"); let inspect = response["result"]["tools"] .as_array() @@ -2552,6 +2553,7 @@ fn mcp_schemas_cover_cli_flags() { ("eb_stack_from_commit", "stack from-commit"), ("eb_stack_gap", "stack gap"), ("eb_stack_provided", "stack provided"), + ("eb_stack_shadow", "stack shadow"), ("eb_stack_batch", "stack batch"), ("eb_stack_shard", "stack shard"), ("eb_stack_arch", "stack arch"), @@ -3716,6 +3718,7 @@ fn every_cli_subcommand_has_an_mcp_tool() { ("eb_stack_from_commit", &["stack", "from-commit"]), ("eb_stack_gap", &["stack", "gap"]), ("eb_stack_provided", &["stack", "provided"]), + ("eb_stack_shadow", &["stack", "shadow"]), ("eb_stack_batch", &["stack", "batch"]), ("eb_stack_shard", &["stack", "shard"]), ("eb_stack_arch", &["stack", "arch"]), diff --git a/tests/shadow_fixtures/bare.txt b/tests/shadow_fixtures/bare.txt new file mode 100644 index 00000000..0d2849f9 --- /dev/null +++ b/tests/shadow_fixtures/bare.txt @@ -0,0 +1,2 @@ +# a base name with no version, as module -t avail can print one +OnlyName diff --git a/tests/shadow_fixtures/base/modules/all/FFTW/3.3.10-gompi-2023a.lua b/tests/shadow_fixtures/base/modules/all/FFTW/3.3.10-gompi-2023a.lua new file mode 100644 index 00000000..5324662e --- /dev/null +++ b/tests/shadow_fixtures/base/modules/all/FFTW/3.3.10-gompi-2023a.lua @@ -0,0 +1 @@ +help([[EESSI base module]]) diff --git a/tests/shadow_fixtures/base/modules/all/zlib/1.3.1-GCCcore-15.2.0.lua b/tests/shadow_fixtures/base/modules/all/zlib/1.3.1-GCCcore-15.2.0.lua new file mode 100644 index 00000000..5324662e --- /dev/null +++ b/tests/shadow_fixtures/base/modules/all/zlib/1.3.1-GCCcore-15.2.0.lua @@ -0,0 +1 @@ +help([[EESSI base module]]) diff --git a/tests/shadow_fixtures/base/modules/all/zlib/default b/tests/shadow_fixtures/base/modules/all/zlib/default new file mode 100644 index 00000000..f205c4e3 --- /dev/null +++ b/tests/shadow_fixtures/base/modules/all/zlib/default @@ -0,0 +1 @@ +1.3.1-GCCcore-15.2.0 diff --git a/tests/shadow_fixtures/broken/Broken-1.0.eb b/tests/shadow_fixtures/broken/Broken-1.0.eb new file mode 100644 index 00000000..2573d9c1 --- /dev/null +++ b/tests/shadow_fixtures/broken/Broken-1.0.eb @@ -0,0 +1 @@ +this is not an easyconfig diff --git a/tests/shadow_fixtures/easyconfigs/f/FFTW/FFTW-3.3.11-gompi-2023a.eb b/tests/shadow_fixtures/easyconfigs/f/FFTW/FFTW-3.3.11-gompi-2023a.eb new file mode 100644 index 00000000..114298c3 --- /dev/null +++ b/tests/shadow_fixtures/easyconfigs/f/FFTW/FFTW-3.3.11-gompi-2023a.eb @@ -0,0 +1,15 @@ +easyblock = 'ConfigureMake' + +name = 'FFTW' +version = '3.3.11' + +homepage = 'https://example.org/fftw' +description = "Site recipe at a different version from the EESSI module." + +toolchain = {'name': 'gompi', 'version': '2023a'} + +source_urls = ['https://example.org/'] +sources = [SOURCE_TAR_GZ] +checksums = ['0000000000000000000000000000000000000000000000000000000000000000'] + +moduleclass = 'numlib' diff --git a/tests/shadow_fixtures/easyconfigs/o/OnlyName/OnlyName-1.2-GCCcore-15.2.0.eb b/tests/shadow_fixtures/easyconfigs/o/OnlyName/OnlyName-1.2-GCCcore-15.2.0.eb new file mode 100644 index 00000000..7a10e744 --- /dev/null +++ b/tests/shadow_fixtures/easyconfigs/o/OnlyName/OnlyName-1.2-GCCcore-15.2.0.eb @@ -0,0 +1,15 @@ +easyblock = 'ConfigureMake' + +name = 'OnlyName' +version = '1.2' + +homepage = 'https://example.org/onlyname' +description = "The base listing names this software and records no version." + +toolchain = {'name': 'GCCcore', 'version': '15.2.0'} + +source_urls = ['https://example.org/'] +sources = [SOURCE_TAR_GZ] +checksums = ['0000000000000000000000000000000000000000000000000000000000000000'] + +moduleclass = 'tools' diff --git a/tests/shadow_fixtures/easyconfigs/v/VASP/VASP-6.5.1-foss-2026.1.eb b/tests/shadow_fixtures/easyconfigs/v/VASP/VASP-6.5.1-foss-2026.1.eb new file mode 100644 index 00000000..771a76a0 --- /dev/null +++ b/tests/shadow_fixtures/easyconfigs/v/VASP/VASP-6.5.1-foss-2026.1.eb @@ -0,0 +1,15 @@ +easyblock = 'MakeCp' + +name = 'VASP' +version = '6.5.1' + +homepage = 'https://example.org/vasp' +description = "Site-only recipe. The EESSI base inventory does not name it." + +toolchain = {'name': 'foss', 'version': '2026.1'} + +source_urls = ['https://example.org/'] +sources = [SOURCE_TAR_GZ] +checksums = ['0000000000000000000000000000000000000000000000000000000000000000'] + +moduleclass = 'chem' diff --git a/tests/shadow_fixtures/easyconfigs/z/zlib/zlib-1.3.1-GCCcore-15.2.0.eb b/tests/shadow_fixtures/easyconfigs/z/zlib/zlib-1.3.1-GCCcore-15.2.0.eb new file mode 100644 index 00000000..f60ece44 --- /dev/null +++ b/tests/shadow_fixtures/easyconfigs/z/zlib/zlib-1.3.1-GCCcore-15.2.0.eb @@ -0,0 +1,15 @@ +easyblock = 'ConfigureMake' + +name = 'zlib' +version = '1.3.1' + +homepage = 'https://example.org/zlib' +description = "Site recipe at the same version as the EESSI module." + +toolchain = {'name': 'GCCcore', 'version': '15.2.0'} + +source_urls = ['https://example.org/'] +sources = [SOURCE_TAR_GZ] +checksums = ['0000000000000000000000000000000000000000000000000000000000000000'] + +moduleclass = 'lib' diff --git a/tests/stack_shadow.rs b/tests/stack_shadow.rs new file mode 100644 index 00000000..f4aa1b74 --- /dev/null +++ b/tests/stack_shadow.rs @@ -0,0 +1,174 @@ +//! `stack shadow` against a site layer and an EESSI base module inventory. +//! +//! zlib is the same software version as the base module. FFTW is another +//! version of a base name. OnlyName is a bare base name. VASP is site-only. + +use eb_stack::mcp::handle_message; +use serde_json::json; +use std::path::PathBuf; +use std::process::Command; + +fn fixture(name: &str) -> String { + PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .join("tests/shadow_fixtures") + .join(name) + .to_str() + .unwrap() + .to_string() +} + +fn shadow(extra: &[&str]) -> std::process::Output { + let mut args = vec![ + "stack".to_string(), + "shadow".to_string(), + "--easyconfigs".to_string(), + fixture("easyconfigs"), + "--base".to_string(), + fixture("base"), + "--base".to_string(), + fixture("bare.txt"), + ]; + args.extend(extra.iter().map(|arg| (*arg).to_string())); + Command::new(env!("CARGO_BIN_EXE_eb-stack")) + .args(&args) + .output() + .expect("run eb-stack") +} + +const REPORT: &str = "\ +same zlib-1.3.1-GCCcore-15.2.0.eb name=zlib version=1.3.1 base=1.3.1-GCCcore-15.2.0 +different FFTW-3.3.11-gompi-2023a.eb name=FFTW version=3.3.11 base=3.3.10-gompi-2023a +unversioned OnlyName-1.2-GCCcore-15.2.0.eb name=OnlyName version=1.2 base=- +same=1 different=1 unversioned=1 recipes=4 base=3 +"; + +#[test] +fn a_site_layer_that_repeats_the_base_exits_1() { + let run = shadow(&[]); + let stdout = String::from_utf8(run.stdout).expect("utf8"); + let stderr = String::from_utf8(run.stderr).expect("utf8"); + assert_eq!(stdout, REPORT, "{stderr}"); + assert!( + stderr.contains("3 site-layer recipes shadow a module the EESSI base stack provides"), + "{stderr}" + ); + assert_eq!(run.status.code(), Some(1)); +} + +#[test] +fn a_site_only_recipe_exits_0() { + let run = Command::new(env!("CARGO_BIN_EXE_eb-stack")) + .args([ + "stack", + "shadow", + "--easyconfigs", + &fixture("easyconfigs/v/VASP/VASP-6.5.1-foss-2026.1.eb"), + "--base", + &fixture("base"), + "--base", + &fixture("bare.txt"), + ]) + .output() + .expect("run"); + let stdout = String::from_utf8(run.stdout).expect("utf8"); + assert!( + run.status.success(), + "{}", + String::from_utf8_lossy(&run.stderr) + ); + assert_eq!( + stdout, + "same=0 different=0 unversioned=0 recipes=1 base=3\n" + ); +} + +#[test] +fn json_reports_the_same_hits() { + let run = shadow(&["--json"]); + let stdout = String::from_utf8(run.stdout).expect("utf8"); + let value: serde_json::Value = serde_json::from_str(&stdout).expect(&stdout); + assert_eq!(value["same"], 1); + assert_eq!(value["different"], 1); + assert_eq!(value["unversioned"], 1); + assert_eq!(value["recipes"], 4); + assert_eq!(value["base_names"], 3); + assert_eq!(value["hits"][0]["kind"], "same"); + assert_eq!(value["hits"][0]["name"], "zlib"); + assert_eq!(run.status.code(), Some(1)); +} + +#[test] +fn an_unreadable_base_names_the_path() { + let missing = fixture("missing-base"); + let run = Command::new(env!("CARGO_BIN_EXE_eb-stack")) + .args([ + "stack", + "shadow", + "--easyconfigs", + &fixture("easyconfigs"), + "--base", + &missing, + ]) + .output() + .expect("run"); + let stderr = String::from_utf8(run.stderr).expect("utf8"); + assert!(stderr.contains(&missing), "{stderr}"); + assert_eq!(run.status.code(), Some(1)); +} + +#[test] +fn a_recipe_that_does_not_parse_names_the_file() { + let broken = fixture("broken"); + let run = Command::new(env!("CARGO_BIN_EXE_eb-stack")) + .args([ + "stack", + "shadow", + "--easyconfigs", + &broken, + "--base", + &fixture("base"), + ]) + .output() + .expect("run"); + let stderr = String::from_utf8(run.stderr).expect("utf8"); + assert!( + stderr.contains("Broken-1.0.eb") && stderr.contains("did not parse"), + "{stderr}" + ); + assert_eq!(run.status.code(), Some(1)); +} + +#[test] +fn a_missing_flag_is_status_2() { + let run = Command::new(env!("CARGO_BIN_EXE_eb-stack")) + .args(["stack", "shadow", "--easyconfigs", &fixture("easyconfigs")]) + .output() + .expect("run"); + assert_eq!(run.status.code(), Some(2)); +} + +#[test] +fn the_mcp_tool_returns_the_same_report() { + let response = handle_message(&json!({ + "jsonrpc": "2.0", + "id": 1, + "method": "tools/call", + "params": { + "name": "eb_stack_shadow", + "arguments": { + "easyconfigs": [fixture("easyconfigs")], + "base": [fixture("base"), fixture("bare.txt")] + } + } + })) + .expect("tools/call"); + assert_eq!(response["result"]["isError"], true, "{response}"); + let text = response["result"]["content"][0]["text"] + .as_str() + .expect("text"); + assert!(text.contains(REPORT.trim_end()), "{text}"); + assert!( + text.contains("3 site-layer recipes shadow a module the EESSI base stack provides"), + "{text}" + ); +} From 9f1aa8011165092042cc3c7cc3802d3c867d0a57 Mon Sep 17 00:00:00 2001 From: Rohit Goswami Date: Tue, 3 Nov 2026 12:55:01 +0000 Subject: [PATCH 2/9] docs(stack): show how to split a build list into shards The `stack shard` command has a CLI entry and no how-to. The page runs the fixture wave and quotes the prelude and the three shard files. It also says which modules are shared. --- CHANGELOG.md | 5 ++ docs/orgmode/howto/stack-shard.org | 76 ++++++++++++++++++++++++++++ docs/orgmode/index.org | 6 ++- llms-full.txt | 79 +++++++++++++++++++++++++++++- llms.txt | 1 + 5 files changed, 165 insertions(+), 2 deletions(-) create mode 100644 docs/orgmode/howto/stack-shard.org diff --git a/CHANGELOG.md b/CHANGELOG.md index 0a721731..24210b8d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -60,6 +60,11 @@ All notable changes to this project are documented here. A line that cannot be resolved exits 1 and writes nothing. `eb_stack_shard` is the same command over MCP. +### Documentation + +- How to split one Jenkins build list into shards that build at the + same time, with the shared dependencies in a prelude. + ## 0.4.0 - 2026-11-01 `stack solve` in 0.3.0 took the policy file as `--policy`. The flag is diff --git a/docs/orgmode/howto/stack-shard.org b/docs/orgmode/howto/stack-shard.org new file mode 100644 index 00000000..8a585c5d --- /dev/null +++ b/docs/orgmode/howto/stack-shard.org @@ -0,0 +1,76 @@ +#+TITLE: Split a Jenkins build list into shards +#+OPTIONS: toc:t num:nil author:nil + +What it does: ~stack shard~ takes one Jenkins build list for one node +type and splits it into shards that can build at the same time. A +line's set is its closure over ~--easyconfigs~ minus the modules that +are already installed. Two lines share a shard only when one line's +easyconfig is in the other's set. Every other module that two or more +shards need goes to ~prelude.list~, which builds first. + +Why you'd use it: a site build list for one CPU target is too long for +one job, and a naive split rebuilds HDF5 in every shard. The prelude +holds the shared dependencies once. + +* Split the fixture wave + +~tests/stack_shard_fixtures/wave1.list~ is four lines: OpenFOAM, +ParaView, PETSc, and Hypre. PETSc depends on Hypre, so those two lines +share a shard and Hypre is first. OpenFOAM and ParaView both need HDF5 +and Szip; OpenFOAM and PETSc both need MathLib. Those shared modules +are not lines of their own, so they go to the prelude. zlib is +installed through the module tree and CMake through the installed +list, so neither is built. + +#+begin_src bash +eb-stack stack shard \ + --build-list tests/stack_shard_fixtures/wave1.list \ + --easyconfigs tests/stack_shard_fixtures/easyconfigs \ + --installed tests/stack_shard_fixtures/installed.txt \ + --installed-modules-dir tests/stack_shard_fixtures/modules/all \ + --shards 4 \ + --generated-at 2026-11-03T00:00:00Z \ + --out-dir /tmp/shards +#+end_src + +That command printed: + +#+begin_example +shards=3 prelude=3 lines=4 +#+end_example + +~/tmp/shards/prelude.list~ is the shared dependencies, in dependency +order: + +#+begin_example +Szip-2.1.1-GCCcore-15.2.0.eb +HDF5-1.14.6-GCCcore-15.2.0.eb +MathLib-1.0-GCCcore-15.2.0.eb +#+end_example + +The three shard files keep the input lines verbatim. Hypre is a line +PETSc needs, so it is first in that shard: + +#+begin_example +Hypre-2.33.0-GCCcore-15.2.0.eb +PETSc-3.23.0-GCCcore-15.2.0.eb --include-easyblocks=/sw/easyblocks/petsc.py +#+end_example + +#+begin_example +ParaView-6.0.0-GCCcore-15.2.0.eb +#+end_example + +#+begin_example +OpenFOAM-13-GCCcore-15.2.0.eb --hooks=/sw/hooks/openfoam.py +#+end_example + +~--shards 4~ is an upper bound. Three groups exist, so three shard +files are written. ~plan.json~ records each line's closure, the shard +weights, and a SHA-256 of the inputs. ~--generated-at~ fixes the time +in that file. ~--weights~ is ~Name/VersionFull~ or the easyconfig +filename, a tab, then seconds. A module the file does not list weighs +~--default-weight~, which is 1. Build the prelude, then the shards. + +A line whose easyconfig is not on the robot path exits 1 and writes +nothing. ~eb_stack_shard~ is the same command. See +[[file:../reference/cli.org][the CLI reference]]. diff --git a/docs/orgmode/index.org b/docs/orgmode/index.org index 9d247070..33725021 100644 --- a/docs/orgmode/index.org +++ b/docs/orgmode/index.org @@ -79,6 +79,8 @@ Build from main for ``stack gap``, ``stack oom``, ``stack from-commit``, ``rebui *A plan entry likely to run out of memory in a bot job?* → :doc:`howto/oom-profile` *Which kind of FAILURE a bot job reported?* → :doc:`howto/bot-classify` + *Split one Jenkins build list into shards that build at once?* → :doc:`howto/stack-shard` + *Dependency closure of one change?* → :doc:`explanation/sbom-blast-radius` *CLI surface?* → :doc:`reference/cli` @@ -169,6 +171,7 @@ Build from main for ``stack gap``, ``stack oom``, ``stack from-commit``, ``rebui howto/arch-known-failures howto/oom-profile howto/bot-classify + howto/stack-shard .. toctree:: :maxdepth: 2 @@ -324,7 +327,8 @@ python3 -m json.tool /tmp/eon-inspect/package.sbom.cdx.json >/dev/null [[file:howto/cuda-compute-capabilities.org][check CUDA compute-capability mismatches in a plan]]. [[file:howto/arch-known-failures.org][warn about known per-architecture failures]]. [[file:howto/oom-profile.org][warn before a bot job runs out of memory]]. - [[file:howto/bot-classify.org][read an EESSI bot FAILURE]]. + [[file:howto/bot-classify.org][read an EESSI bot FAILURE]], + [[file:howto/stack-shard.org][split a Jenkins build list into shards]]. - Reference: [[file:reference/glossary.org][glossary]], [[file:reference/cli.org][CLI]], [[file:reference/rust-api.org][Rust API]], diff --git a/llms-full.txt b/llms-full.txt index f8bdafed..ccfaeffe 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -116,7 +116,8 @@ python3 -m json.tool /tmp/eon-inspect/package.sbom.cdx.json >/dev/null [check CUDA compute-capability mismatches in a plan](https://eb-stack.rgoswami.me/howto/cuda-compute-capabilities.html.md). [warn about known per-architecture failures](https://eb-stack.rgoswami.me/howto/arch-known-failures.html.md). [warn before a bot job runs out of memory](https://eb-stack.rgoswami.me/howto/oom-profile.html.md). - [read an EESSI bot FAILURE](https://eb-stack.rgoswami.me/howto/bot-classify.html.md). + [read an EESSI bot FAILURE](https://eb-stack.rgoswami.me/howto/bot-classify.html.md), + [split a Jenkins build list into shards](https://eb-stack.rgoswami.me/howto/stack-shard.html.md). - Reference: [glossary](https://eb-stack.rgoswami.me/reference/glossary.html.md), [CLI](https://eb-stack.rgoswami.me/reference/cli.html.md), [Rust API](https://eb-stack.rgoswami.me/reference/rust-api.html.md), @@ -5153,6 +5154,82 @@ text, the jobs and the counts. - A job that is still running has no `finished` row and is `unclassified`. +# Split a Jenkins build list into shards + +What it does: `stack shard` takes one Jenkins build list for one node +type and splits it into shards that can build at the same time. A +line's set is its closure over `--easyconfigs` minus the modules that +are already installed. Two lines share a shard only when one line's +easyconfig is in the other's set. Every other module that two or more +shards need goes to `prelude.list`, which builds first. + +Why you'd use it: a site build list for one CPU target is too long for +one job, and a naive split rebuilds HDF5 in every shard. The prelude +holds the shared dependencies once. + +# Split the fixture wave + +`tests/stack_shard_fixtures/wave1.list` is four lines: OpenFOAM, +ParaView, PETSc, and Hypre. PETSc depends on Hypre, so those two lines +share a shard and Hypre is first. OpenFOAM and ParaView both need HDF5 +and Szip; OpenFOAM and PETSc both need MathLib. Those shared modules +are not lines of their own, so they go to the prelude. zlib is +installed through the module tree and CMake through the installed +list, so neither is built. + +```bash +eb-stack stack shard \ + --build-list tests/stack_shard_fixtures/wave1.list \ + --easyconfigs tests/stack_shard_fixtures/easyconfigs \ + --installed tests/stack_shard_fixtures/installed.txt \ + --installed-modules-dir tests/stack_shard_fixtures/modules/all \ + --shards 4 \ + --generated-at 2026-11-03T00:00:00Z \ + --out-dir /tmp/shards +``` + +That command printed: + +``` +shards=3 prelude=3 lines=4 +``` + +`/tmp/shards/prelude.list` is the shared dependencies, in dependency +order: + +``` +Szip-2.1.1-GCCcore-15.2.0.eb +HDF5-1.14.6-GCCcore-15.2.0.eb +MathLib-1.0-GCCcore-15.2.0.eb +``` + +The three shard files keep the input lines verbatim. Hypre is a line +PETSc needs, so it is first in that shard: + +``` +Hypre-2.33.0-GCCcore-15.2.0.eb +PETSc-3.23.0-GCCcore-15.2.0.eb --include-easyblocks=/sw/easyblocks/petsc.py +``` + +``` +ParaView-6.0.0-GCCcore-15.2.0.eb +``` + +``` +OpenFOAM-13-GCCcore-15.2.0.eb --hooks=/sw/hooks/openfoam.py +``` + +`--shards 4` is an upper bound. Three groups exist, so three shard +files are written. `plan.json` records each line's closure, the shard +weights, and a SHA-256 of the inputs. `--generated-at` fixes the time +in that file. `--weights` is `Name/VersionFull` or the easyconfig +filename, a tab, then seconds. A module the file does not list weighs +`--default-weight`, which is 1. Build the prelude, then the shards. + +A line whose easyconfig is not on the robot path exits 1 and writes +nothing. `eb_stack_shard` is the same command. See +[the CLI reference](https://eb-stack.rgoswami.me/../reference/cli.html.md). + # Glossary Names shared by the tutorial, the how-to guides, and the reference. diff --git a/llms.txt b/llms.txt index 49ead137..8cb888c2 100644 --- a/llms.txt +++ b/llms.txt @@ -38,6 +38,7 @@ The binary is `eb-stack`. The crate is `eb_stack`. Rust 1.88 or newer builds it - [Warn about known per-architecture failures](https://eb-stack.rgoswami.me/howto/arch-known-failures.html.md): What it does: stack known-failures reads an easystack as a plan, closes its dependencies over --easyconfigs, and warns about every entry a bot run has already failed on one of the architectures you name. - [Warn before a bot job runs out of memory](https://eb-stack.rgoswami.me/howto/oom-profile.html.md): What it does: stack oom reads an easystack as a plan. - [Read an EESSI bot FAILURE](https://eb-stack.rgoswami.me/howto/bot-classify.html.md): What it does: stack bot-classify reads what an EESSI software-layer bot job left behind and says which kind of FAILURE it was: a real failure, ERROR-only noise, an empty tarball, or infra. +- [Split a Jenkins build list into shards](https://eb-stack.rgoswami.me/howto/stack-shard.html.md): What it does: stack shard takes one Jenkins build list for one node type and splits it into shards that can build at the same time. ## Reference From 9b72c20f7ca93cd3da4fcc47c045fc75172a2735 Mon Sep 17 00:00:00 2001 From: Rohit Goswami Date: Tue, 3 Nov 2026 12:55:18 +0000 Subject: [PATCH 3/9] docs(stack): show how to plan non-overlapping batches The `stack batch` command was described from the other stack pages and had no how-to of its own. The page runs the GROMACS fixture and quotes the second-version warning the batch records. --- CHANGELOG.md | 2 + docs/orgmode/howto/stack-batch.org | 69 ++++++++++++++++++++++++++++ docs/orgmode/index.org | 6 ++- llms-full.txt | 72 +++++++++++++++++++++++++++++- llms.txt | 1 + 5 files changed, 148 insertions(+), 2 deletions(-) create mode 100644 docs/orgmode/howto/stack-batch.org diff --git a/CHANGELOG.md b/CHANGELOG.md index 24210b8d..cc0b91b4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -64,6 +64,8 @@ All notable changes to this project are documented here. - How to split one Jenkins build list into shards that build at the same time, with the shared dependencies in a prelude. +- How to plan software-layer pull requests whose builds do not overlap, + including the second-version warning a batch records. ## 0.4.0 - 2026-11-01 diff --git a/docs/orgmode/howto/stack-batch.org b/docs/orgmode/howto/stack-batch.org new file mode 100644 index 00000000..b8d0733b --- /dev/null +++ b/docs/orgmode/howto/stack-batch.org @@ -0,0 +1,69 @@ +#+TITLE: Plan software-layer pull requests that do not overlap +#+OPTIONS: toc:t num:nil author:nil + +What it does: ~stack batch~ plans European Environment for Scientific +Software Installations (EESSI) software-layer pull requests whose builds +do not overlap. A root's set is its closure over the robot path minus +the installed modules, the same set ~pr-overlap~ computes for one +easystack line. Roots whose sets meet go in one pull request. A root +whose set meets an open pull request waits for that pull request. The +rest are packed ~--size~ roots per pull request. + +Why you'd use it: ~stack gap~ writes ~roots.txt~ for the recipes the +generation can already build. Those roots should not all land in one +pull request, and two pull requests must not build the same module. + +* One root beside an installed version + +~tests/batch_versions_fixtures~ has GROMACS 2025.4 installed and a +recipe for GROMACS 2026.2 on foss/2026.1, generation 2026.1 of EESSI +2026.06. One root is one batch. The second version in that generation +is a warning, not a failed plan. + +#+begin_src bash +eb-stack stack batch \ + --root GROMACS-2026.2-foss-2026.1.eb \ + --easyconfigs tests/batch_versions_fixtures/easyconfigs \ + --installed tests/batch_versions_fixtures/installed-gromacs-2025.txt \ + --eessi-version 2026.06 \ + --easybuild-version 5.4.1 \ + --generation 2026.1 \ + --generated-at 2026-10-30T00:00:00Z \ + --out-dir /tmp/batches +#+end_src + +That command printed: + +#+begin_example +batches=1 roots=1 waits=0 unresolved=0 deployed=0 digest=cd1b5856e678c2d8fbd7bc563a861edf706a9afe7cece55596aaa892a235b080 out=/tmp/batches +#+end_example + +~/tmp/batches/plan.md~ names the batch and the warning: + +#+begin_example +## batch-01-gromacs + +{2026.06}[foss/2026.1] GROMACS 2026.2 + +1 module(s) to build. + +1 second-version warning(s); `pr-body.md` has the details. +#+end_example + +The batch directory holds the EESSI easystacks, a lock, a CycloneDX +software bill of materials, ~reproduce.sh~, ~batch.json~, and +~pr-body.md~. The pull-request body includes: + +#+begin_example +Second versions in one generation, from `eb-stack stack generation`: + +GROMACS has 2 versions in generation 2026.1: 2025.4 (already in the stack), 2026.2 (added: GROMACS-2026.2-foss-2026.1.eb); pass --allow-versions GROMACS@2026.1 to accept +#+end_example + +~--allow-versions GROMACS@2026.1~ accepts that pair. +~--generated-at~ fixes the time written into the plan, so the same +inputs give the same digest. ~--roots-file~ reads the ~roots.txt~ that +~stack gap~ writes. An entry the architecture cannot build exits 1 and +writes nothing. ~eb_stack_batch~ is the same command. See +[[file:generation-lint.org][the second-version check]] and +[[file:../reference/cli.org][the CLI reference]]. diff --git a/docs/orgmode/index.org b/docs/orgmode/index.org index 33725021..26e8e309 100644 --- a/docs/orgmode/index.org +++ b/docs/orgmode/index.org @@ -81,6 +81,8 @@ Build from main for ``stack gap``, ``stack oom``, ``stack from-commit``, ``rebui *Split one Jenkins build list into shards that build at once?* → :doc:`howto/stack-shard` + *Plan software-layer pull requests that do not overlap?* → :doc:`howto/stack-batch` + *Dependency closure of one change?* → :doc:`explanation/sbom-blast-radius` *CLI surface?* → :doc:`reference/cli` @@ -172,6 +174,7 @@ Build from main for ``stack gap``, ``stack oom``, ``stack from-commit``, ``rebui howto/oom-profile howto/bot-classify howto/stack-shard + howto/stack-batch .. toctree:: :maxdepth: 2 @@ -328,7 +331,8 @@ python3 -m json.tool /tmp/eon-inspect/package.sbom.cdx.json >/dev/null [[file:howto/arch-known-failures.org][warn about known per-architecture failures]]. [[file:howto/oom-profile.org][warn before a bot job runs out of memory]]. [[file:howto/bot-classify.org][read an EESSI bot FAILURE]], - [[file:howto/stack-shard.org][split a Jenkins build list into shards]]. + [[file:howto/stack-shard.org][split a Jenkins build list into shards]], + [[file:howto/stack-batch.org][plan software-layer pull requests that do not overlap]]. - Reference: [[file:reference/glossary.org][glossary]], [[file:reference/cli.org][CLI]], [[file:reference/rust-api.org][Rust API]], diff --git a/llms-full.txt b/llms-full.txt index ccfaeffe..95598ec3 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -117,7 +117,8 @@ python3 -m json.tool /tmp/eon-inspect/package.sbom.cdx.json >/dev/null [warn about known per-architecture failures](https://eb-stack.rgoswami.me/howto/arch-known-failures.html.md). [warn before a bot job runs out of memory](https://eb-stack.rgoswami.me/howto/oom-profile.html.md). [read an EESSI bot FAILURE](https://eb-stack.rgoswami.me/howto/bot-classify.html.md), - [split a Jenkins build list into shards](https://eb-stack.rgoswami.me/howto/stack-shard.html.md). + [split a Jenkins build list into shards](https://eb-stack.rgoswami.me/howto/stack-shard.html.md), + [plan software-layer pull requests that do not overlap](https://eb-stack.rgoswami.me/howto/stack-batch.html.md). - Reference: [glossary](https://eb-stack.rgoswami.me/reference/glossary.html.md), [CLI](https://eb-stack.rgoswami.me/reference/cli.html.md), [Rust API](https://eb-stack.rgoswami.me/reference/rust-api.html.md), @@ -5230,6 +5231,75 @@ A line whose easyconfig is not on the robot path exits 1 and writes nothing. `eb_stack_shard` is the same command. See [the CLI reference](https://eb-stack.rgoswami.me/../reference/cli.html.md). +# Plan software-layer pull requests that do not overlap + +What it does: `stack batch` plans European Environment for Scientific +Software Installations (EESSI) software-layer pull requests whose builds +do not overlap. A root's set is its closure over the robot path minus +the installed modules, the same set `pr-overlap` computes for one +easystack line. Roots whose sets meet go in one pull request. A root +whose set meets an open pull request waits for that pull request. The +rest are packed `--size` roots per pull request. + +Why you'd use it: `stack gap` writes `roots.txt` for the recipes the +generation can already build. Those roots should not all land in one +pull request, and two pull requests must not build the same module. + +# One root beside an installed version + +`tests/batch_versions_fixtures` has GROMACS 2025.4 installed and a +recipe for GROMACS 2026.2 on foss/2026.1, generation 2026.1 of EESSI +2026.06. One root is one batch. The second version in that generation +is a warning, not a failed plan. + +```bash +eb-stack stack batch \ + --root GROMACS-2026.2-foss-2026.1.eb \ + --easyconfigs tests/batch_versions_fixtures/easyconfigs \ + --installed tests/batch_versions_fixtures/installed-gromacs-2025.txt \ + --eessi-version 2026.06 \ + --easybuild-version 5.4.1 \ + --generation 2026.1 \ + --generated-at 2026-10-30T00:00:00Z \ + --out-dir /tmp/batches +``` + +That command printed: + +``` +batches=1 roots=1 waits=0 unresolved=0 deployed=0 digest=cd1b5856e678c2d8fbd7bc563a861edf706a9afe7cece55596aaa892a235b080 out=/tmp/batches +``` + +`/tmp/batches/plan.md` names the batch and the warning: + +``` +## batch-01-gromacs + +{2026.06}[foss/2026.1] GROMACS 2026.2 + +1 module(s) to build. + +1 second-version warning(s); `pr-body.md` has the details. +``` + +The batch directory holds the EESSI easystacks, a lock, a CycloneDX +software bill of materials, `reproduce.sh`, `batch.json`, and +`pr-body.md`. The pull-request body includes: + +``` +Second versions in one generation, from `eb-stack stack generation`: + +GROMACS has 2 versions in generation 2026.1: 2025.4 (already in the stack), 2026.2 (added: GROMACS-2026.2-foss-2026.1.eb); pass --allow-versions GROMACS@2026.1 to accept +``` + +`--allow-versions GROMACS@2026.1` accepts that pair. +`--generated-at` fixes the time written into the plan, so the same +inputs give the same digest. `--roots-file` reads the `roots.txt` that +`stack gap` writes. An entry the architecture cannot build exits 1 and +writes nothing. `eb_stack_batch` is the same command. See +[the second-version check](https://eb-stack.rgoswami.me/generation-lint.html.md) and +[the CLI reference](https://eb-stack.rgoswami.me/../reference/cli.html.md). + # Glossary Names shared by the tutorial, the how-to guides, and the reference. diff --git a/llms.txt b/llms.txt index 8cb888c2..3b501500 100644 --- a/llms.txt +++ b/llms.txt @@ -39,6 +39,7 @@ The binary is `eb-stack`. The crate is `eb_stack`. Rust 1.88 or newer builds it - [Warn before a bot job runs out of memory](https://eb-stack.rgoswami.me/howto/oom-profile.html.md): What it does: stack oom reads an easystack as a plan. - [Read an EESSI bot FAILURE](https://eb-stack.rgoswami.me/howto/bot-classify.html.md): What it does: stack bot-classify reads what an EESSI software-layer bot job left behind and says which kind of FAILURE it was: a real failure, ERROR-only noise, an empty tarball, or infra. - [Split a Jenkins build list into shards](https://eb-stack.rgoswami.me/howto/stack-shard.html.md): What it does: stack shard takes one Jenkins build list for one node type and splits it into shards that can build at the same time. +- [Plan software-layer pull requests that do not overlap](https://eb-stack.rgoswami.me/howto/stack-batch.html.md): What it does: stack batch plans European Environment for Scientific Software Installations (EESSI) software-layer pull requests whose builds do not overlap. ## Reference From da8ee6deae95c4a6295dacf0f7ff6e99237122e7 Mon Sep 17 00:00:00 2001 From: Rohit Goswami Date: Tue, 3 Nov 2026 12:55:32 +0000 Subject: [PATCH 4/9] docs(stack): show how to regroup a Jenkins build list The `stack sections` command had a reference section and no how-to. The page runs the recorded previous and next lists and quotes the regrouped file. --- CHANGELOG.md | 2 + docs/orgmode/howto/stack-sections.org | 50 +++++++++++++++++++++++++ docs/orgmode/index.org | 6 ++- llms-full.txt | 53 ++++++++++++++++++++++++++- llms.txt | 1 + 5 files changed, 110 insertions(+), 2 deletions(-) create mode 100644 docs/orgmode/howto/stack-sections.org diff --git a/CHANGELOG.md b/CHANGELOG.md index cc0b91b4..7d092031 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -66,6 +66,8 @@ All notable changes to this project are documented here. same time, with the shared dependencies in a prelude. - How to plan software-layer pull requests whose builds do not overlap, including the second-version warning a batch records. +- How to regroup a Jenkins build list under the section headers of a + previous list. ## 0.4.0 - 2026-11-01 diff --git a/docs/orgmode/howto/stack-sections.org b/docs/orgmode/howto/stack-sections.org new file mode 100644 index 00000000..c26168d7 --- /dev/null +++ b/docs/orgmode/howto/stack-sections.org @@ -0,0 +1,50 @@ +#+TITLE: Regroup a Jenkins build list under the previous sections +#+OPTIONS: toc:t num:nil author:nil + +What it does: ~stack sections~ reads a Jenkins build list and writes it +again under the ~#### Title ####~ headers of a previous list. Each +entry keeps its flags and whether it is commented out. A name no +previous section files goes under ~--unfiled~. + +Why you'd use it: a solved build list is one flat file. The site's +previous list already files each package under MPI, Chemistry, and the +other headings the operators read. The regrouped list is that order. + +* Regroup the recorded lists + +~tests/jenkins_sections/previous.list~ has MPI and Chemistry. +~tests/jenkins_sections/next.list~ is the next generation, with CP2K +commented out and not named by either previous section. + +#+begin_src bash +eb-stack stack sections tests/jenkins_sections/next.list \ + --from tests/jenkins_sections/previous.list \ + --out /tmp/grouped.list +#+end_src + +That command printed: + +#+begin_example +list=/tmp/grouped.list sections=3 of=3 +#+end_example + +~/tmp/grouped.list~ is: + +#+begin_example +# next generation + +#### MPI #### +# OpenMPI-5.0.10-GCC-15.2.0.eb --hooks=/sw/h.py + +#### Chemistry #### +# GROMACS-2025.0-foss-2025b.eb + +#### New in this generation #### +# CP2K-2026.1-foss-2025b.eb +#+end_example + +~of=3~ is the previous list's two sections plus the unfiled header. +The default unfiled title is ~New in this generation~. The comment +block before the first entry stays on top. OpenMPI keeps +~--hooks=/sw/h.py~. ~eb_stack_sections~ is the same command. See +[[file:../reference/cli.org][the CLI reference]]. diff --git a/docs/orgmode/index.org b/docs/orgmode/index.org index 26e8e309..edf1a8ad 100644 --- a/docs/orgmode/index.org +++ b/docs/orgmode/index.org @@ -83,6 +83,8 @@ Build from main for ``stack gap``, ``stack oom``, ``stack from-commit``, ``rebui *Plan software-layer pull requests that do not overlap?* → :doc:`howto/stack-batch` + *Regroup a Jenkins build list under the previous sections?* → :doc:`howto/stack-sections` + *Dependency closure of one change?* → :doc:`explanation/sbom-blast-radius` *CLI surface?* → :doc:`reference/cli` @@ -175,6 +177,7 @@ Build from main for ``stack gap``, ``stack oom``, ``stack from-commit``, ``rebui howto/bot-classify howto/stack-shard howto/stack-batch + howto/stack-sections .. toctree:: :maxdepth: 2 @@ -332,7 +335,8 @@ python3 -m json.tool /tmp/eon-inspect/package.sbom.cdx.json >/dev/null [[file:howto/oom-profile.org][warn before a bot job runs out of memory]]. [[file:howto/bot-classify.org][read an EESSI bot FAILURE]], [[file:howto/stack-shard.org][split a Jenkins build list into shards]], - [[file:howto/stack-batch.org][plan software-layer pull requests that do not overlap]]. + [[file:howto/stack-batch.org][plan software-layer pull requests that do not overlap]], + [[file:howto/stack-sections.org][regroup a Jenkins build list under the previous sections]]. - Reference: [[file:reference/glossary.org][glossary]], [[file:reference/cli.org][CLI]], [[file:reference/rust-api.org][Rust API]], diff --git a/llms-full.txt b/llms-full.txt index 95598ec3..ad393e8e 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -118,7 +118,8 @@ python3 -m json.tool /tmp/eon-inspect/package.sbom.cdx.json >/dev/null [warn before a bot job runs out of memory](https://eb-stack.rgoswami.me/howto/oom-profile.html.md). [read an EESSI bot FAILURE](https://eb-stack.rgoswami.me/howto/bot-classify.html.md), [split a Jenkins build list into shards](https://eb-stack.rgoswami.me/howto/stack-shard.html.md), - [plan software-layer pull requests that do not overlap](https://eb-stack.rgoswami.me/howto/stack-batch.html.md). + [plan software-layer pull requests that do not overlap](https://eb-stack.rgoswami.me/howto/stack-batch.html.md), + [regroup a Jenkins build list under the previous sections](https://eb-stack.rgoswami.me/howto/stack-sections.html.md). - Reference: [glossary](https://eb-stack.rgoswami.me/reference/glossary.html.md), [CLI](https://eb-stack.rgoswami.me/reference/cli.html.md), [Rust API](https://eb-stack.rgoswami.me/reference/rust-api.html.md), @@ -5300,6 +5301,56 @@ writes nothing. `eb_stack_batch` is the same command. See [the second-version check](https://eb-stack.rgoswami.me/generation-lint.html.md) and [the CLI reference](https://eb-stack.rgoswami.me/../reference/cli.html.md). +# Regroup a Jenkins build list under the previous sections + +What it does: `stack sections` reads a Jenkins build list and writes it +again under the `#### Title ####` headers of a previous list. Each +entry keeps its flags and whether it is commented out. A name no +previous section files goes under `--unfiled`. + +Why you'd use it: a solved build list is one flat file. The site's +previous list already files each package under MPI, Chemistry, and the +other headings the operators read. The regrouped list is that order. + +# Regroup the recorded lists + +`tests/jenkins_sections/previous.list` has MPI and Chemistry. +`tests/jenkins_sections/next.list` is the next generation, with CP2K +commented out and not named by either previous section. + +```bash +eb-stack stack sections tests/jenkins_sections/next.list \ + --from tests/jenkins_sections/previous.list \ + --out /tmp/grouped.list +``` + +That command printed: + +``` +list=/tmp/grouped.list sections=3 of=3 +``` + +`/tmp/grouped.list` is: + +``` +# next generation + +#### MPI #### +# OpenMPI-5.0.10-GCC-15.2.0.eb --hooks=/sw/h.py + +#### Chemistry #### +# GROMACS-2025.0-foss-2025b.eb + +#### New in this generation #### +# CP2K-2026.1-foss-2025b.eb +``` + +`of=3` is the previous list's two sections plus the unfiled header. +The default unfiled title is `New in this generation`. The comment +block before the first entry stays on top. OpenMPI keeps +`--hooks=/sw/h.py`. `eb_stack_sections` is the same command. See +[the CLI reference](https://eb-stack.rgoswami.me/../reference/cli.html.md). + # Glossary Names shared by the tutorial, the how-to guides, and the reference. diff --git a/llms.txt b/llms.txt index 3b501500..2d894c55 100644 --- a/llms.txt +++ b/llms.txt @@ -40,6 +40,7 @@ The binary is `eb-stack`. The crate is `eb_stack`. Rust 1.88 or newer builds it - [Read an EESSI bot FAILURE](https://eb-stack.rgoswami.me/howto/bot-classify.html.md): What it does: stack bot-classify reads what an EESSI software-layer bot job left behind and says which kind of FAILURE it was: a real failure, ERROR-only noise, an empty tarball, or infra. - [Split a Jenkins build list into shards](https://eb-stack.rgoswami.me/howto/stack-shard.html.md): What it does: stack shard takes one Jenkins build list for one node type and splits it into shards that can build at the same time. - [Plan software-layer pull requests that do not overlap](https://eb-stack.rgoswami.me/howto/stack-batch.html.md): What it does: stack batch plans European Environment for Scientific Software Installations (EESSI) software-layer pull requests whose builds do not overlap. +- [Regroup a Jenkins build list under the previous sections](https://eb-stack.rgoswami.me/howto/stack-sections.html.md): What it does: stack sections reads a Jenkins build list and writes it again under the #### Title #### headers of a previous list. ## Reference From 591d6ca64066240edc1c02da0be68d88a5e31d18 Mon Sep 17 00:00:00 2001 From: Rohit Goswami Date: Tue, 3 Nov 2026 12:55:48 +0000 Subject: [PATCH 5/9] docs(stack): show how to write an easystack from a lock The `stack easystack` command was in the command reference and the format page, and had no how-to. The page writes the baseline lock and shows the option spelling. --- CHANGELOG.md | 1 + docs/orgmode/howto/stack-easystack.org | 58 ++++++++++++++++++++++++ docs/orgmode/index.org | 6 ++- llms-full.txt | 61 +++++++++++++++++++++++++- llms.txt | 1 + 5 files changed, 125 insertions(+), 2 deletions(-) create mode 100644 docs/orgmode/howto/stack-easystack.org diff --git a/CHANGELOG.md b/CHANGELOG.md index 7d092031..43355421 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -68,6 +68,7 @@ All notable changes to this project are documented here. including the second-version warning a batch records. - How to regroup a Jenkins build list under the section headers of a previous list. +- How to write a stack lock as the YAML `eb --easystack` reads. ## 0.4.0 - 2026-11-01 diff --git a/docs/orgmode/howto/stack-easystack.org b/docs/orgmode/howto/stack-easystack.org new file mode 100644 index 00000000..7a9191b7 --- /dev/null +++ b/docs/orgmode/howto/stack-easystack.org @@ -0,0 +1,58 @@ +#+TITLE: Write an easystack from a stack lock +#+OPTIONS: toc:t num:nil author:nil + +What it does: ~stack easystack~ writes a stack lock as the YAML +~eb --easystack~ reads. Entries are easyconfig filenames. ~--option~ +attaches per-file ~eb~ options, spelled without the leading dashes. +~--eessi-version~ writes the European Environment for Scientific +Software Installations (EESSI) software-layer layout instead of one +file: one easystack per generation, ~001-system~ for the system +toolchain, and ~accel/nvidia/~ for CUDA builds. + +Why you'd use it: ~stack solve~ writes a lock. The software layer and +~eb --easystack~ do not read that lock. This command is the file they +do read. + +* One lock, one file + +~tests/provided_fixtures/baseline.lock.json~ selects GROMACS 2025.2 on +foss/2026.1. + +#+begin_src bash +eb-stack stack easystack \ + --lock tests/provided_fixtures/baseline.lock.json \ + --out /tmp/stack.yml +#+end_src + +That command printed: + +#+begin_example +easystack=/tmp/stack.yml entries=1 of=1 +#+end_example + +~/tmp/stack.yml~ is: + +#+begin_example +easyconfigs: +- GROMACS-2025.2-foss-2026.1.eb +#+end_example + +~entries~ counts filenames in the YAML. ~of~ is the number of packages +in the lock. A package whose lock entry names no ~.eb~ file is left +out: EasyBuild cannot build a file that was never on disk. + +A file with options is a one-key mapping. Repeat ~--option~ as +~: