Skip to content

Follow pagination when listing GitHub names - #2228

Draft
bIackr0se wants to merge 1 commit into
fsspec:masterfrom
bIackr0se:fix/github-api-pagination
Draft

bIackr0se wants to merge 1 commit into
fsspec:masterfrom
bIackr0se:fix/github-api-pagination

Conversation

@bIackr0se

Copy link
Copy Markdown

GithubFileSystem.tags, branches and repos() read only the first GitHub API page. When results span several pages, valid names are missing.

Follow the response's rel="next" link until no next page remains. Each request keeps the existing timeout and instance authentication. A later-page HTTP error raises instead of returning a partial list.

Observed against fsspec/filesystem_spec on 2026-10-04:

fs = GithubFileSystem(
    org="fsspec", repo="filesystem_spec",
    sha="778f95612ca60da45ccfb14c22b9ee409d67f1ee",
)
len(fs.tags)
# Before: 30
# After:  103, matching all four API pages

Verification:

  • Nine regression cases fail before the fix; all 17 pagination tests pass after it. They cover three-page traversal, exact continuation URLs, authentication, timeouts, empty results, refs and later-page errors.
  • Pagination, GitHub path and filesystem specification tests: 247 passed, 67 skipped, 1 expected failure on macOS with Python 3.14.7.
  • A wheel installed outside the checkout passes three-page listings and failure paths through a local HTTP server. Against the public API it discovers all 103 tags, then selects 2023.9.2 from a later page and reads its README, matching the raw file byte for byte.
  • Repository pre-commit hooks pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant