fix(examples): page pull request lists past 500 and the file cap

CHANGELOG.md

@@ -15,6 +15,12 @@ All notable changes to this project are documented here.
   under `opaque`. Any other unresolved line is still an error and
   nothing is written. A list with 22 `--from-commit` lines and seven
   site recipes failed whole before.
+- `examples/easyconfigs-pr-overlap/fetch-prs.sh` and
+  `examples/eessi-pr-overlap/fetch-prs.sh` request each page of open pull
+  requests and of changed files, 100 at a time. The open list does not
+  stop at 500. A changed-file list that stops at GitHub's cap of 3000 is
+  kept: the pull request is still written, with `file_list_truncated`
+  set, and the record notes `truncated at GitHub's limit`.
 - `stack order` exits 1 when runtime requirements of one name at one
   toolchain have no version that satisfies every one of them. The error
   names each requester and the version it resolved. Eigen 3.4.1 required
@@ -29,6 +35,19 @@ All notable changes to this project are documented here.
 
 ### Added
 
+- `stack order --installed` stops the build order at modules already on
+  the machine. A file is a module listing and a directory is an Lmod
+  tree, read the same way as `stack gap --installed`. The version is the
+  full module version. A listed module is left out of the order, and so
+  are the dependencies that were installed with it. `--filter-deps`
+  still drops dependencies first. The modules that remain are chosen
+  with the synthetic GCC-family rule `recipe hierarchy --export`
+  prints. Stdout reports `installed=` as how many modules the walk
+  stopped at. `eb_stack_order` takes the same `installed` list.
+- The Model Context Protocol manual lists one tool for every CLI
+  command. `docs/orgmode/reference/mcp.org` names each tool, including
+  the recipe and easyblock tools that have no command. A test fails
+  when a subcommand's tool is absent from that page.
 - `stack shadow` reports a site-layer recipe that shadows a module the
   EESSI base stack already provides. `--base` is an Lmod tree or a module
   listing, loaded with the same inventory reader as `stack gap`. The same
@@ -270,6 +289,9 @@ include `stack gap`, `stack oom`, `stack from-commit`, `rebuild-set`,
   `source probe:`. With no `--arch` all three are still probed. With one
   family there is nothing to compare with, so a source it cannot fetch
   is `unreachable` rather than `arch-missing`, and still exits 1.
+  Two families compare only with each other: a miss is `arch-missing`
+  when the other named family fetched that source slot, and
+  `unreachable` when neither did.
 - `classify_build_failure` classes `No space left on device` and
   `Disk quota exceeded` as `resource`. A full disk in an EasyBuild log was
   `compile` before.

README.md

@@ -402,7 +402,8 @@ for `pr-overlap`, `rebuild-set`, `component-reach`, and `easyblock-reach`.
 `eb-stack mcp --workspace ROOT` (or `EB_STACK_WORKSPACE`) is the root for the
 recipe file tools.
 
-- `eb_package_inspect`, `eb_package_plan`, `eb_package_bump`;
+- `eb_package_inspect`, `eb_package_plan`, `eb_package_bump`,
+  `eb_package_latest`;
 - `eb_recipe_read`, `eb_recipe_list`, `eb_recipe_edit`, `eb_recipe_write`,
   `eb_recipe_grep`, `eb_recipe_apply_suggestion`, `eb_recipe_diff`,
   `eb_dep_versions`;
@@ -411,7 +412,11 @@ recipe file tools.
 - `eb_module_lint`, `eb_easyblock_dry_run`, `eb_build_log`, `eb_build_test`
   (easyblocks; `eb_build_test` only with `EB_STACK_ALLOW_BUILD=1`);
 - `eb_stack_solve`, `eb_stack_sbom`, `eb_stack_order`,
-  `eb_stack_easystack`, `eb_stack_sections`, `eb_stack_from_commit`;
+  `eb_stack_easystack`, `eb_stack_gap`, `eb_stack_provided`,
+  `eb_stack_shadow`, `eb_stack_batch`, `eb_stack_shard`,
+  `eb_stack_arch`, `eb_stack_files`, `eb_stack_generation`,
+  `eb_stack_cuda_cc`, `eb_stack_known_failures`, `eb_stack_oom`,
+  `eb_stack_bot_classify`, `eb_stack_sections`, `eb_stack_from_commit`;
 - `eb_component_reach` (same query as `component-reach`, including a purl or a CVE id);
 - `eb_target_list`, `eb_target_doctor`;
 - `eb_campaign_run`, `eb_campaign_status`;

docs/orgmode/howto/arch-unsupported.org

@@ -199,4 +199,40 @@ same source slot resolves to a file that exists on another
 architecture. ~--arch x86_64~ on this fixture probes only the x86_64
 wheel, which the server has, and exits 0.
 
+** Two architectures
+
+What it does: ~--arch x86_64 --arch aarch64 --probe-sources~ probes
+those two families and no other. A target such as ~x86_64/amd/zen4~
+is still the family before the slash. Naming a family and one of its
+targets, ~--arch aarch64 --arch aarch64/neoverse_v1~, probes that
+family once. The order of the flags does not change the report.
+
+Why you'd use it: two bot targets failed the fetch and a third was
+not in the run. The comparison is between the families that were
+named. A family left off the command is not fetched, and it does not
+turn ~arch-missing~ into ~unreachable~ or the other way around.
+
+#+begin_src bash
+eb-stack stack arch \
+  --arch x86_64 \
+  --arch aarch64 \
+  --easystack tests/source_probe_fixtures/easystacks/kaleido.yml \
+  --easyconfigs tests/source_probe_fixtures/easyconfigs \
+  --probe-sources
+#+end_src
+
+On the same copy and server it prints:
+
+#+begin_example
+Kaleido-0.2.1-GCCcore-15.2.0.eb source kaleido-0.2.1-manylinux2014-aarch64.whl is arch-missing on aarch64: __BASE_URL__/kaleido-0.2.1-manylinux2014-aarch64.whl (attempt 1/3 HEAD=status:404 RANGE=status:404)
+parse_failed=0 unsupported=0 flagged=0 source_unreachable=0 source_arch_missing=1
+#+end_example
+
+How to read it: the x86_64 wheel is on the server, so the aarch64
+wheel is ~arch-missing~. riscv64 is not named and is not probed. The
+same command with ~--arch aarch64 --arch riscv64~ prints both wheels
+as ~unreachable~, with ~source_unreachable=2~ and
+~source_arch_missing=0~, because neither probed family fetched the
+source slot. The x86_64 wheel is not consulted.
+
 See [[file:../reference/cli.org][the command-line reference]].

docs/orgmode/howto/easyconfigs-pr-overlap.org

@@ -138,7 +138,11 @@ writes one JSON file per open PR that changes an easyconfig. Run it
 from a clone of easybuild-easyconfigs whose ~origin~ is that
 repository, with ~gh~ authenticated for public reads. Set
 ~EB_STACK_CHECKOUT~ to the eb-stack checkout. The script reads base
-text from ~develop~ and does not check a PR out.
+text from ~develop~ and does not check a PR out. Open pull requests
+are read 100 per page, with no stop at 500. Changed files are read
+100 per page through GitHub's cap of 3000. A pull request whose file
+list stops there is still written, with ~file_list_truncated~ set.
+~pr-overlap~ notes ~truncated at GitHub's limit~.
 
 #+begin_src bash
 : "${EB_STACK_CHECKOUT:?set this to an eb-stack checkout}"

docs/orgmode/howto/pr-overlap.org

@@ -153,9 +153,12 @@ EasyBuild version these easystacks name.
 differ from that version.
 ~examples/eessi-pr-overlap/fetch-prs.sh~ reads every open PR and writes
 one JSON file per PR that changes an easystack for that EESSI version.
-It pages through the GitHub pulls API 100 at a time with no upper
-limit, so a repository with more open PRs than one listing returns is
-still read in full. It fetches each PR head into the clone as
+It pages through the GitHub pulls API 100 at a time with no stop at
+500, so a repository with more open PRs than one listing returns is
+still read in full. Changed files are read 100 per page through
+GitHub's cap of 3000. A pull request whose file list stops there is
+still written, with ~file_list_truncated~ set, and the record notes
+~truncated at GitHub's limit~. It fetches each PR head into the clone as
 ~refs/remotes/pr/N~ and does not check it out. ~--eessi-version~
 defaults to ~2026.06~.
 

docs/orgmode/howto/solve-lock.org

@@ -118,6 +118,25 @@ eb-stack stack order \
 requirement. The hash file is one ~<sha256>  <module>~ line per build.
 See [[file:../reference/cli.org][the CLI reference]].
 
+~--installed~ stops the walk at modules the site already has, the way
+the robot does. ~--filter-deps~ still drops dependencies before that
+walk. The two compose: a filtered dependency is gone, and a listed
+module is a leaf whose own dependencies are not ordered.
+
+#+begin_src bash
+eb-stack stack order \
+  --easyconfigs fixtures/gromacs_2025_to_next/easyconfigs \
+  --root GROMACS \
+  --installed fixtures/stack_order_installed/installed.txt \
+  --filter-deps FFTW \
+  --out /tmp/build-order.txt
+#+end_src
+
+The order file has no ~OpenBLAS~ path and no ~FFTW~ path. ~GROMACS~ is
+still in it. Standard output ends with ~installed=1~ and then
+~order=/tmp/build-order.txt builds=~. ~installed=1~ is OpenBLAS. FFTW
+was filtered, so it is not a leaf the walk stopped at.
+
 * One lock per toolchain family
 
 A foss solve only sees the foss hierarchy.

docs/orgmode/reference/cli.org

@@ -432,6 +432,7 @@ eb-stack stack order \
   --easyconfigs TREE [--easyconfigs OVERLAY]... \
   --root NAME [--root NAME==VERSION]... \
   [--filter-deps NAME,NAME...] \
+  [--installed LIST]... \
   [--out /tmp/build-order.txt] \
   [--oldest] \
   [--easystack-out /tmp/order.yml] \
@@ -644,7 +645,20 @@ not repin a recipe. See [[file:../howto/solve-lock.org][Order every pinned build
 ~name==version~ pins one build. ~--filter-deps~ drops each named
 dependency before that order is built, together with the subtree under
 it, using the same matching as ~stack solve~; when the flag is absent
-the command reads ~EASYBUILD_FILTER_DEPS~. ~--oldest~ takes the oldest
+the command reads ~EASYBUILD_FILTER_DEPS~. ~--installed~ repeats. A file
+is ~module -t avail~, ~module spider -t~, or ~Name/version~, and a
+directory is an Lmod tree, both read the same way as
+~stack gap --installed~. The version is the full module version,
+~1.3.1-GCCcore-15.2.0~, the spelling under ~modules/all~. A system
+toolchain contributes no toolchain segment. A listed module is a leaf:
+it stays out of the order, and so do the dependencies that were
+installed with it. A dependency the tree cannot resolve is still an
+error. The walk that remains uses the synthetic GCC-family rule
+~recipe hierarchy --export~ prints, so an unpinned dependency of a
+~GCC~ or ~GCCcore~ recipe stays in that family. ~--filter-deps~ is
+applied first, so a filtered dependency is not counted as a leaf. The
+line ~installed=<N>~ just before ~order=~ is how many distinct modules
+the walk stopped at. ~--oldest~ takes the oldest
 admissible version of an unpinned requirement, and the default is the
 newest. ~--out~ is one easyconfig path per line (default
 ~build-order.txt~). ~--easystack-out~ writes that same sequence as an
@@ -863,6 +877,9 @@ fallback, on each family ~--arch~ names, or on ~x86_64~, ~aarch64~,
 and ~riscv64~ with no ~--arch~. A source no probed family can fetch is
 ~unreachable~. A source another probed family can fetch is
 ~arch-missing~, so with one ~--arch~ a missing source is ~unreachable~.
+Two ~--arch~ values compare only that pair: a miss is ~arch-missing~
+when the other named family fetched the same source slot, and
+~unreachable~ when neither did.
 ~--probe-retries~ and ~--probe-timeout-s~ tune that network check.
 
 [[file:../howto/arch-unsupported.org][Refuse a plan an architecture cannot build]]
@@ -1751,7 +1768,8 @@ plus sandboxed recipe and easyblock tools:
   (easyblocks; ~eb_build_test~ only with ~EB_STACK_ALLOW_BUILD=1~);
 - ~eb_stack_solve~, ~eb_stack_sbom~, ~eb_stack_order~,
   ~eb_stack_easystack~, ~eb_stack_gap~, ~eb_stack_provided~,
-  ~eb_stack_batch~, ~eb_stack_shard~, ~eb_stack_arch~, ~eb_stack_files~,
+  ~eb_stack_shadow~, ~eb_stack_batch~, ~eb_stack_shard~,
+  ~eb_stack_arch~, ~eb_stack_files~,
   ~eb_stack_generation~, ~eb_stack_cuda_cc~, ~eb_stack_known_failures~,
   ~eb_stack_oom~, ~eb_stack_bot_classify~, ~eb_stack_sections~,
   ~eb_stack_from_commit~;
@@ -1792,8 +1810,12 @@ table and the derived-key rules. ~eb_recipe_hierarchy~ takes ~export~.
 The plain call prints the fixture generations. ~export~ true adds the
 synthetic GCC-family rule, the same split as ~recipe hierarchy --export~.
 ~eb_stack_order~ requires ~easyconfigs~ and ~root~, and takes ~out~,
-~oldest~, ~easystack_out~, ~hashes_out~, and ~filter_deps~.
+~oldest~, ~easystack_out~, ~hashes_out~, ~filter_deps~, and ~installed~.
 ~filter_deps~ drops the same dependencies ~--filter-deps~ drops.
+~installed~ is the module list or Lmod tree ~--installed~ reads. A
+module it names is left out of the order. The result field ~installed~
+is how many modules the walk stopped at, and is null when the argument
+is omitted.
 ~eb_stack_easystack~ requires ~lock~ and takes ~out~ and ~option~, each
 option a ~<file.eb>:<option>=<value>~ string, and ~eessi_version~,
 ~easybuild_version~, ~generation~, and ~out_dir~ for the software-layer
@@ -1809,13 +1831,16 @@ requires ~easystack~ and takes ~easyconfigs_repo~, ~easyblocks_repo~,
 ~licensed_file~, ~json~, ~format~, and ~strict~. A hit, an unparsed
 line, a wrong toolchain, a ~--hooks~ flag, or a family-lock name is a
 tool error. A licensed warning is not, unless ~strict~ is true. The
-text is the same report as ~stack provided~. ~eb_stack_arch~ requires
-~easystack~ and takes
+text is the same report as ~stack provided~. ~eb_stack_shadow~
+requires ~easyconfigs~ and ~base~, and takes ~json~. It returns the
+same report as ~stack shadow~. A shadow, or a recipe that does not
+parse, is a tool error. ~eb_stack_arch~ requires ~easystack~ and takes
 ~arch~, ~easyconfigs~, ~installed~, ~additional_arch~, ~unsupported~,
 ~probe_sources~, ~probe_retries~, and ~probe_timeout_s~. It returns the
 same report as ~stack arch~, with ~parse_failed~ and ~source_findings~,
 and a hit is a tool error. ~probe_sources~ probes the families ~arch~
-names, as ~--arch~ does. ~eb_stack_files~ requires ~easystack~ and takes
+names, as ~--arch~ does. Two names are compared only with each other.
+~eb_stack_files~ requires ~easystack~ and takes
 ~easyconfigs~, ~commit_tree~, ~easyconfigs_repo~, ~sourcepath~, and
 ~path_map~. It returns the same report as ~stack files~, and a missing
 file is a tool error. ~eb_stack_cuda_cc~ requires ~easystack~ and ~easyconfigs~,

docs/orgmode/reference/formats.org

@@ -219,16 +219,19 @@ solver selection, not a scan of installed software.
 
 One easyconfig path per line, in dependency order. A name EasyBuild
 installs twice lists both paths: another toolchain, a system bootstrap
-pair, or a build-only tool. ~--oldest~ changes which unpinned version is
-chosen; it does not change the file shape. When runtime requirements of
+pair, or a build-only tool. ~--installed~ leaves a listed module, and
+the dependencies that were installed with it, out of this file.
+~--oldest~ changes which unpinned version is chosen; it does not change
+the file shape. When runtime requirements of
 one name at one toolchain have no version that satisfies every one of
 them, the command writes no file and exits 1. Standard error names each
 requester and the version it resolved. The file is not a place where a
 pin is rewritten.
 
 Stdout names a co-installable repeated build as
 ~multiple builds of <name>: <ver-tc>, ...~ and ends with
-~order=<path> builds=<N>~.
+~order=<path> builds=<N>~. When ~--installed~ is set, the line before
+that is ~installed=<N>~, how many listed modules the walk stopped at.
 
 * Input hashes (~--hashes-out~)
 

docs/orgmode/reference/mcp.org

@@ -88,7 +88,71 @@ absolute path there.
 
 * Tools
 
-~tools/list~ returns the workflows of ~package~, ~recipe~, ~stack~,
-~target~, ~campaign~, and ~repro~, plus ~pr-overlap~, ~rebuild-set~,
-~component-reach~, and ~easyblock-reach~. ~eb_build_test~ runs only
-when ~EB_STACK_ALLOW_BUILD=1~.
+The catalog has 51 tools. ~tools/list~ returns one tool for each CLI
+command in the table, and the recipe and easyblock tools that have no
+command. ~mcp~ publishes that catalog and has no tool of its own. Each
+command's tool calls the same runner as that command. The ~text~ field
+of a successful result is that command's stdout. Each argument is a
+name in the schema ~tools/list~ returns. The
+[[file:cli.org][CLI reference]] records the flags.
+
+| Command | Tool |
+|-
+| ~package inspect~ | ~eb_package_inspect~ |
+| ~package plan~ | ~eb_package_plan~ |
+| ~package bump~ | ~eb_package_bump~ |
+| ~package latest~ | ~eb_package_latest~ |
+| ~recipe easyblock~ | ~eb_recipe_easyblock~ |
+| ~recipe templates~ | ~eb_recipe_templates~ |
+| ~recipe hierarchy~ | ~eb_recipe_hierarchy~ |
+| ~recipe check~ | ~eb_recipe_check~ |
+| ~recipe lint~ | ~eb_recipe_lint~ |
+| ~recipe format~ | ~eb_recipe_format~ |
+| ~stack solve~ | ~eb_stack_solve~ |
+| ~stack sbom~ | ~eb_stack_sbom~ |
+| ~stack order~ | ~eb_stack_order~ |
+| ~stack gap~ | ~eb_stack_gap~ |
+| ~stack provided~ | ~eb_stack_provided~ |
+| ~stack shadow~ | ~eb_stack_shadow~ |
+| ~stack batch~ | ~eb_stack_batch~ |
+| ~stack shard~ | ~eb_stack_shard~ |
+| ~stack easystack~ | ~eb_stack_easystack~ |
+| ~stack sections~ | ~eb_stack_sections~ |
+| ~stack from-commit~ | ~eb_stack_from_commit~ |
+| ~stack arch~ | ~eb_stack_arch~ |
+| ~stack files~ | ~eb_stack_files~ |
+| ~stack generation~ | ~eb_stack_generation~ |
+| ~stack cuda-cc~ | ~eb_stack_cuda_cc~ |
+| ~stack known-failures~ | ~eb_stack_known_failures~ |
+| ~stack oom~ | ~eb_stack_oom~ |
+| ~stack bot-classify~ | ~eb_stack_bot_classify~ |
+| ~target list~ | ~eb_target_list~ |
+| ~target doctor~ | ~eb_target_doctor~ |
+| ~campaign run~ | ~eb_campaign_run~ |
+| ~campaign status~ | ~eb_campaign_status~ |
+| ~campaign finding claim~ | ~eb_campaign_finding_claim~ |
+| ~campaign finding resolve~ | ~eb_campaign_finding_resolve~ |
+| ~repro summary~ | ~eb_repro_summary~ |
+| ~easyblock-reach~ | ~eb_easyblock_reach~ |
+| ~pr-overlap~ | ~eb_pr_overlap~ |
+| ~rebuild-set~ | ~eb_rebuild_set~ |
+| ~component-reach~ | ~eb_component_reach~ |
+
+These tools edit files or run EasyBuild. They are not CLI commands.
+
+| Tool | Role |
+|-
+| ~eb_recipe_read~ | Read one UTF-8 file inside the workspace root |
+| ~eb_recipe_list~ | List one directory inside the workspace root |
+| ~eb_recipe_edit~ | Replace an exact string in a workspace file |
+| ~eb_recipe_apply_suggestion~ | Apply a review suggestion to the lines it is anchored on |
+| ~eb_recipe_diff~ | Show what changed in each recipe since the server first saw it |
+| ~eb_recipe_write~ | Create or overwrite a UTF-8 file inside the workspace root |
+| ~eb_recipe_grep~ | Search workspace files for a literal, or a regular expression |
+| ~eb_dep_versions~ | List easyconfig versions of one dependency in a local tree |
+| ~eb_module_lint~ | Check Python modules that ship with recipes |
+| ~eb_easyblock_dry_run~ | Run ~eb -x~ and return the commands the easyblock generates |
+| ~eb_build_log~ | Read an EasyBuild log by step |
+| ~eb_build_test~ | Run a build |
+
+~eb_build_test~ runs only when ~EB_STACK_ALLOW_BUILD=1~.