Skip to content

fix(examples): page pull request lists past 500 and the file cap - #18

Merged
HaoZeke merged 4 commits into
mainfrom
fix/fetch-prs-page-past-caps
Nov 3, 2026
Merged

HaoZeke merged 4 commits into
mainfrom
fix/fetch-prs-page-past-caps

Conversation

@HaoZeke

@HaoZeke HaoZeke commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Page open pull requests and changed files

examples/easyconfigs-pr-overlap/fetch-prs.sh and examples/eessi-pr-overlap/fetch-prs.sh read open pull requests and changed files one page at a time.

What was wrong

Both scripts passed the open-list URL and the changed-file URL to one gh api --paginate call.
The changed-file call exits non-zero when GitHub answers 422 at its cap of 3000 files.
The scripts kept the filenames already printed and did not record that the list was cut, so a capped pull request was treated as a complete file list.
The tests used 102 open pull requests, so a walk that stopped at 500 still passed.

What changed

Each script requests pages of 100 itself.

  • Open pull requests continue until a page is short.
    The walk does not stop at 500.
  • Changed files continue until a page is short.
    After 3000 names, one further page is probed.
    HTTP 422 that names the cap, or another non-empty page, sets file_list_truncated.
    Names past 3000 are not kept.
    The pull request is still written.
    pr-overlap notes truncated at GitHub's limit.
  • Each script names the page size (page_size=100), the file cap (file_cap=3000, also used in the 422 message match), and the page limits (open_page_limit=1000, file_page_limit=40) at the top.

Before and after

Before, the fixture stopped short of pull request 501 (page 6 of a 100-wide list).
A file list that hit the cap left no file_list_truncated flag.

After, a fixture of 501 open pull requests prints open pull requests: 501 and writes prs/501.json for the pull request that changes an easyconfig or a 2026.06 easystack.
A changed file at position 501 (page 6) is kept, with no truncation flag.
A list that answers 422 on the page after 3000 files exits 0, keeps the pull request, and sets file_list_truncated to true.

Tests

cargo +stable fmt --all --check exited 0.
cargo +stable clippy --locked --all-targets -- -D warnings exited 0.
cargo +stable test --locked --all-targets exited 0.
cargo +stable test --locked --test fetch_prs_script was 10 passed, 0 failed.
shellcheck on both scripts exited 0.

Reproduce

cargo +stable test --locked --test fetch_prs_script \
  easyconfigs_fetch_reads_open_pull_requests_past_five_hundred \
  eessi_fetch_reads_open_pull_requests_past_five_hundred \
  easyconfigs_fetch_pages_files_past_five_hundred_and_keeps_the_cap \
  eessi_fetch_pages_files_past_five_hundred_and_keeps_the_cap

Note

Medium Risk
Changes build-order graph walking and source-probe classification; PR-fetch scripts alter which PRs and file lists downstream overlap sees when repos are large or capped.

Overview
Adds stack order --installed (and eb_stack_order installed) so build orders skip modules already on the site—using the same module list/Lmod inventory as stack gap --installed—and omit dependencies that were installed with them. --filter-deps still runs first; stdout reports installed=N before order=.

Reworks examples/easyconfigs-pr-overlap/fetch-prs.sh and examples/eessi-pr-overlap/fetch-prs.sh to page open PRs and changed files 100 at a time (no 500 stop). When GitHub caps a PR at 3000 files, the PR JSON is still written with file_list_truncated, with tests for 501 open PRs and deep file lists.

stack arch --probe-sources with two --arch values now compares only that pair (arch-missing vs unreachable); docs and tests cover the Kaleido fixture.

Expands MCP/README/CHANGELOG with a 51-tool command↔tool table in mcp.org and a test that every CLI subcommand (and listed tool name) appears there.

Reviewed by Cursor Bugbot for commit 9be50b1. Bugbot is set up for automated code reviews on this repo. Configure here.

mcp.org named the command groups and left each tool name to tools/list.
eb_stack_shadow was missing from the CLI reference.
The README left out the stack report tools and package latest.
The manual now names every CLI command's tool and the recipe and easyblock tools.
A test fails when a subcommand's tool is absent from that page.
`stack order` walked every dependency the roots pin, including modules already on the machine.
`--installed` now stops that walk at each listed module and leaves that module and its own dependencies out of the order.
`--filter-deps` still drops dependencies first, and the modules that remain follow the synthetic GCC-family hierarchy.
A single `--arch` was the only probe the suite locked.
Two flags were free to consult the third default family again: a wheel neither named family can fetch would come back `arch-missing` when that third family can fetch it, and `unreachable` only for one `--arch`.

The new test probes x86_64 with aarch64, and aarch64 with riscv64, on the Kaleido fixture.
The first pair reports the aarch64 wheel as `arch-missing` and does not mention riscv64.
The second reports both wheels as `unreachable`.
Flag order, a CPU target, and a family named twice stay on that same set.
`eb_stack_arch` returns the same lines.
The how-to quotes the pair.
Three texts now say that two names are compared only with each other: the command reference, the `--probe-sources` help, and the tool description.
fetch-prs.sh handed the open pull request list and the changed-file list to one `gh api --paginate` call.
GitHub answers 422 at its 3000-file cap, and the file call then exits non-zero.
The scripts kept the names already printed and did not record that the list was cut.
Nothing in the tests used more than 102 open pull requests, so a walk that stopped at 500 still passed.

Both scripts now request each page of 100.
The open list runs until a short page.
A changed-file list that stops at 3000 keeps the pull request and sets file_list_truncated, which the record notes as truncated at GitHub's limit.
The page size, the file cap, and both page limits are named variables at the top of each script.
@HaoZeke
HaoZeke merged commit 273da4b into main Nov 3, 2026
19 checks passed
@HaoZeke
HaoZeke deleted the fix/fetch-prs-page-past-caps branch November 4, 2026 09:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant