diff --git a/CHANGELOG.md b/CHANGELOG.md index c3894758..5c6eb21c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,12 @@ All notable changes to this project are documented here. under `opaque`. Any other unresolved line is still an error and nothing is written. A list with 22 `--from-commit` lines and seven site recipes failed whole before. +- `examples/easyconfigs-pr-overlap/fetch-prs.sh` and + `examples/eessi-pr-overlap/fetch-prs.sh` request each page of open pull + requests and of changed files, 100 at a time. The open list does not + stop at 500. A changed-file list that stops at GitHub's cap of 3000 is + kept: the pull request is still written, with `file_list_truncated` + set, and the record notes `truncated at GitHub's limit`. - `stack order` exits 1 when runtime requirements of one name at one toolchain have no version that satisfies every one of them. The error names each requester and the version it resolved. Eigen 3.4.1 required @@ -29,6 +35,19 @@ All notable changes to this project are documented here. ### Added +- `stack order --installed` stops the build order at modules already on + the machine. A file is a module listing and a directory is an Lmod + tree, read the same way as `stack gap --installed`. The version is the + full module version. A listed module is left out of the order, and so + are the dependencies that were installed with it. `--filter-deps` + still drops dependencies first. The modules that remain are chosen + with the synthetic GCC-family rule `recipe hierarchy --export` + prints. Stdout reports `installed=` as how many modules the walk + stopped at. `eb_stack_order` takes the same `installed` list. +- The Model Context Protocol manual lists one tool for every CLI + command. `docs/orgmode/reference/mcp.org` names each tool, including + the recipe and easyblock tools that have no command. A test fails + when a subcommand's tool is absent from that page. - `stack shadow` reports a site-layer recipe that shadows a module the EESSI base stack already provides. `--base` is an Lmod tree or a module listing, loaded with the same inventory reader as `stack gap`. The same @@ -270,6 +289,9 @@ include `stack gap`, `stack oom`, `stack from-commit`, `rebuild-set`, `source probe:`. With no `--arch` all three are still probed. With one family there is nothing to compare with, so a source it cannot fetch is `unreachable` rather than `arch-missing`, and still exits 1. + Two families compare only with each other: a miss is `arch-missing` + when the other named family fetched that source slot, and + `unreachable` when neither did. - `classify_build_failure` classes `No space left on device` and `Disk quota exceeded` as `resource`. A full disk in an EasyBuild log was `compile` before. diff --git a/README.md b/README.md index ff1ac8d9..838f2e75 100644 --- a/README.md +++ b/README.md @@ -402,7 +402,8 @@ for `pr-overlap`, `rebuild-set`, `component-reach`, and `easyblock-reach`. `eb-stack mcp --workspace ROOT` (or `EB_STACK_WORKSPACE`) is the root for the recipe file tools. -- `eb_package_inspect`, `eb_package_plan`, `eb_package_bump`; +- `eb_package_inspect`, `eb_package_plan`, `eb_package_bump`, + `eb_package_latest`; - `eb_recipe_read`, `eb_recipe_list`, `eb_recipe_edit`, `eb_recipe_write`, `eb_recipe_grep`, `eb_recipe_apply_suggestion`, `eb_recipe_diff`, `eb_dep_versions`; @@ -411,7 +412,11 @@ recipe file tools. - `eb_module_lint`, `eb_easyblock_dry_run`, `eb_build_log`, `eb_build_test` (easyblocks; `eb_build_test` only with `EB_STACK_ALLOW_BUILD=1`); - `eb_stack_solve`, `eb_stack_sbom`, `eb_stack_order`, - `eb_stack_easystack`, `eb_stack_sections`, `eb_stack_from_commit`; + `eb_stack_easystack`, `eb_stack_gap`, `eb_stack_provided`, + `eb_stack_shadow`, `eb_stack_batch`, `eb_stack_shard`, + `eb_stack_arch`, `eb_stack_files`, `eb_stack_generation`, + `eb_stack_cuda_cc`, `eb_stack_known_failures`, `eb_stack_oom`, + `eb_stack_bot_classify`, `eb_stack_sections`, `eb_stack_from_commit`; - `eb_component_reach` (same query as `component-reach`, including a purl or a CVE id); - `eb_target_list`, `eb_target_doctor`; - `eb_campaign_run`, `eb_campaign_status`; diff --git a/docs/orgmode/howto/arch-unsupported.org b/docs/orgmode/howto/arch-unsupported.org index 01374e1d..b44ed802 100644 --- a/docs/orgmode/howto/arch-unsupported.org +++ b/docs/orgmode/howto/arch-unsupported.org @@ -199,4 +199,40 @@ same source slot resolves to a file that exists on another architecture. ~--arch x86_64~ on this fixture probes only the x86_64 wheel, which the server has, and exits 0. +** Two architectures + +What it does: ~--arch x86_64 --arch aarch64 --probe-sources~ probes +those two families and no other. A target such as ~x86_64/amd/zen4~ +is still the family before the slash. Naming a family and one of its +targets, ~--arch aarch64 --arch aarch64/neoverse_v1~, probes that +family once. The order of the flags does not change the report. + +Why you'd use it: two bot targets failed the fetch and a third was +not in the run. The comparison is between the families that were +named. A family left off the command is not fetched, and it does not +turn ~arch-missing~ into ~unreachable~ or the other way around. + +#+begin_src bash +eb-stack stack arch \ + --arch x86_64 \ + --arch aarch64 \ + --easystack tests/source_probe_fixtures/easystacks/kaleido.yml \ + --easyconfigs tests/source_probe_fixtures/easyconfigs \ + --probe-sources +#+end_src + +On the same copy and server it prints: + +#+begin_example +Kaleido-0.2.1-GCCcore-15.2.0.eb source kaleido-0.2.1-manylinux2014-aarch64.whl is arch-missing on aarch64: __BASE_URL__/kaleido-0.2.1-manylinux2014-aarch64.whl (attempt 1/3 HEAD=status:404 RANGE=status:404) +parse_failed=0 unsupported=0 flagged=0 source_unreachable=0 source_arch_missing=1 +#+end_example + +How to read it: the x86_64 wheel is on the server, so the aarch64 +wheel is ~arch-missing~. riscv64 is not named and is not probed. The +same command with ~--arch aarch64 --arch riscv64~ prints both wheels +as ~unreachable~, with ~source_unreachable=2~ and +~source_arch_missing=0~, because neither probed family fetched the +source slot. The x86_64 wheel is not consulted. + See [[file:../reference/cli.org][the command-line reference]]. diff --git a/docs/orgmode/howto/easyconfigs-pr-overlap.org b/docs/orgmode/howto/easyconfigs-pr-overlap.org index 197881b8..86b82edc 100644 --- a/docs/orgmode/howto/easyconfigs-pr-overlap.org +++ b/docs/orgmode/howto/easyconfigs-pr-overlap.org @@ -138,7 +138,11 @@ writes one JSON file per open PR that changes an easyconfig. Run it from a clone of easybuild-easyconfigs whose ~origin~ is that repository, with ~gh~ authenticated for public reads. Set ~EB_STACK_CHECKOUT~ to the eb-stack checkout. The script reads base -text from ~develop~ and does not check a PR out. +text from ~develop~ and does not check a PR out. Open pull requests +are read 100 per page, with no stop at 500. Changed files are read +100 per page through GitHub's cap of 3000. A pull request whose file +list stops there is still written, with ~file_list_truncated~ set. +~pr-overlap~ notes ~truncated at GitHub's limit~. #+begin_src bash : "${EB_STACK_CHECKOUT:?set this to an eb-stack checkout}" diff --git a/docs/orgmode/howto/pr-overlap.org b/docs/orgmode/howto/pr-overlap.org index 662c1d10..d692db1b 100644 --- a/docs/orgmode/howto/pr-overlap.org +++ b/docs/orgmode/howto/pr-overlap.org @@ -153,9 +153,12 @@ EasyBuild version these easystacks name. differ from that version. ~examples/eessi-pr-overlap/fetch-prs.sh~ reads every open PR and writes one JSON file per PR that changes an easystack for that EESSI version. -It pages through the GitHub pulls API 100 at a time with no upper -limit, so a repository with more open PRs than one listing returns is -still read in full. It fetches each PR head into the clone as +It pages through the GitHub pulls API 100 at a time with no stop at +500, so a repository with more open PRs than one listing returns is +still read in full. Changed files are read 100 per page through +GitHub's cap of 3000. A pull request whose file list stops there is +still written, with ~file_list_truncated~ set, and the record notes +~truncated at GitHub's limit~. It fetches each PR head into the clone as ~refs/remotes/pr/N~ and does not check it out. ~--eessi-version~ defaults to ~2026.06~. diff --git a/docs/orgmode/howto/solve-lock.org b/docs/orgmode/howto/solve-lock.org index 3296e290..d26edf3d 100644 --- a/docs/orgmode/howto/solve-lock.org +++ b/docs/orgmode/howto/solve-lock.org @@ -118,6 +118,25 @@ eb-stack stack order \ requirement. The hash file is one ~ ~ line per build. See [[file:../reference/cli.org][the CLI reference]]. +~--installed~ stops the walk at modules the site already has, the way +the robot does. ~--filter-deps~ still drops dependencies before that +walk. The two compose: a filtered dependency is gone, and a listed +module is a leaf whose own dependencies are not ordered. + +#+begin_src bash +eb-stack stack order \ + --easyconfigs fixtures/gromacs_2025_to_next/easyconfigs \ + --root GROMACS \ + --installed fixtures/stack_order_installed/installed.txt \ + --filter-deps FFTW \ + --out /tmp/build-order.txt +#+end_src + +The order file has no ~OpenBLAS~ path and no ~FFTW~ path. ~GROMACS~ is +still in it. Standard output ends with ~installed=1~ and then +~order=/tmp/build-order.txt builds=~. ~installed=1~ is OpenBLAS. FFTW +was filtered, so it is not a leaf the walk stopped at. + * One lock per toolchain family A foss solve only sees the foss hierarchy. diff --git a/docs/orgmode/reference/cli.org b/docs/orgmode/reference/cli.org index f0c858c2..860a75da 100644 --- a/docs/orgmode/reference/cli.org +++ b/docs/orgmode/reference/cli.org @@ -432,6 +432,7 @@ eb-stack stack order \ --easyconfigs TREE [--easyconfigs OVERLAY]... \ --root NAME [--root NAME==VERSION]... \ [--filter-deps NAME,NAME...] \ + [--installed LIST]... \ [--out /tmp/build-order.txt] \ [--oldest] \ [--easystack-out /tmp/order.yml] \ @@ -644,7 +645,20 @@ not repin a recipe. See [[file:../howto/solve-lock.org][Order every pinned build ~name==version~ pins one build. ~--filter-deps~ drops each named dependency before that order is built, together with the subtree under it, using the same matching as ~stack solve~; when the flag is absent -the command reads ~EASYBUILD_FILTER_DEPS~. ~--oldest~ takes the oldest +the command reads ~EASYBUILD_FILTER_DEPS~. ~--installed~ repeats. A file +is ~module -t avail~, ~module spider -t~, or ~Name/version~, and a +directory is an Lmod tree, both read the same way as +~stack gap --installed~. The version is the full module version, +~1.3.1-GCCcore-15.2.0~, the spelling under ~modules/all~. A system +toolchain contributes no toolchain segment. A listed module is a leaf: +it stays out of the order, and so do the dependencies that were +installed with it. A dependency the tree cannot resolve is still an +error. The walk that remains uses the synthetic GCC-family rule +~recipe hierarchy --export~ prints, so an unpinned dependency of a +~GCC~ or ~GCCcore~ recipe stays in that family. ~--filter-deps~ is +applied first, so a filtered dependency is not counted as a leaf. The +line ~installed=~ just before ~order=~ is how many distinct modules +the walk stopped at. ~--oldest~ takes the oldest admissible version of an unpinned requirement, and the default is the newest. ~--out~ is one easyconfig path per line (default ~build-order.txt~). ~--easystack-out~ writes that same sequence as an @@ -863,6 +877,9 @@ fallback, on each family ~--arch~ names, or on ~x86_64~, ~aarch64~, and ~riscv64~ with no ~--arch~. A source no probed family can fetch is ~unreachable~. A source another probed family can fetch is ~arch-missing~, so with one ~--arch~ a missing source is ~unreachable~. +Two ~--arch~ values compare only that pair: a miss is ~arch-missing~ +when the other named family fetched the same source slot, and +~unreachable~ when neither did. ~--probe-retries~ and ~--probe-timeout-s~ tune that network check. [[file:../howto/arch-unsupported.org][Refuse a plan an architecture cannot build]] @@ -1751,7 +1768,8 @@ plus sandboxed recipe and easyblock tools: (easyblocks; ~eb_build_test~ only with ~EB_STACK_ALLOW_BUILD=1~); - ~eb_stack_solve~, ~eb_stack_sbom~, ~eb_stack_order~, ~eb_stack_easystack~, ~eb_stack_gap~, ~eb_stack_provided~, - ~eb_stack_batch~, ~eb_stack_shard~, ~eb_stack_arch~, ~eb_stack_files~, + ~eb_stack_shadow~, ~eb_stack_batch~, ~eb_stack_shard~, + ~eb_stack_arch~, ~eb_stack_files~, ~eb_stack_generation~, ~eb_stack_cuda_cc~, ~eb_stack_known_failures~, ~eb_stack_oom~, ~eb_stack_bot_classify~, ~eb_stack_sections~, ~eb_stack_from_commit~; @@ -1792,8 +1810,12 @@ table and the derived-key rules. ~eb_recipe_hierarchy~ takes ~export~. The plain call prints the fixture generations. ~export~ true adds the synthetic GCC-family rule, the same split as ~recipe hierarchy --export~. ~eb_stack_order~ requires ~easyconfigs~ and ~root~, and takes ~out~, -~oldest~, ~easystack_out~, ~hashes_out~, and ~filter_deps~. +~oldest~, ~easystack_out~, ~hashes_out~, ~filter_deps~, and ~installed~. ~filter_deps~ drops the same dependencies ~--filter-deps~ drops. +~installed~ is the module list or Lmod tree ~--installed~ reads. A +module it names is left out of the order. The result field ~installed~ +is how many modules the walk stopped at, and is null when the argument +is omitted. ~eb_stack_easystack~ requires ~lock~ and takes ~out~ and ~option~, each option a ~: