Skip to content

Respect exclusive read bounds in MMapCache - #2219

Open
hansu650 wants to merge 1 commit into
fsspec:masterfrom
hansu650:fix/mmap-read-bounds
Open

hansu650 wants to merge 1 commit into
fsspec:masterfrom
hansu650:fix/mmap-read-bounds

Conversation

@hansu650

@hansu650 hansu650 commented Oct 2, 2026

Copy link
Copy Markdown

Summary

MMapCache._fetch() treated the exclusive end offset as an additional block and did not clamp oversized reads to the file size. This could mark nonexistent blocks as cached or, after the final block was already cached, send an empty/inverted range to the backend. For example, with a 10-byte file and 4-byte blocks, reading byte 8 followed by _fetch(0, 100) requested (12, 10). The same problem is reachable through AbstractBufferedFile.read() with cache_type="mmap".

  • Clamp end to EOF and calculate the final block from end - 1, after the existing empty-range guard.
  • Keep the existing range coalescing, retry bookkeeping, and single/multi-fetcher paths unchanged.
  • Add 42 regression/control cases for exclusive block boundaries, aligned/partial EOF, cold/warm caches, holes, empty reads, repeated reads, cache statistics, and buffered-file reads; add a changelog entry.

This is the mmap counterpart of the bounds handling previously fixed for the other block caches, not a change to their prefetch policy.

Validation

Windows / dedicated Conda Python 3.13.15 environment, based on upstream af73c654:

  • New cases before the fix: 25 failed, 17 passed; all 42 pass after the fix.
  • Cache and cache-thread suites: 239 passed. MMapCache._fetch has 100% statement and branch coverage.
  • Broader local/core/cached-filesystem integration: 1,379 passed, 172 skipped, 10 xfailed, 2 failed. Both failures are existing Windows-invalid | filename tests; both were independently reproduced on an unmodified upstream export (211 other selected cases passed). A Windows HTTP-fixture teardown warning also reproduces upstream. No existing assertions or skip rules were changed.
  • Repository-wide Ruff check/format, changed-file pre-commit hooks, strict Sphinx build, and dependency consistency passed.
  • Clean Git-export sdist/wheel build and Twine checks passed; the installed wheel passed the complete cache module (236 passed) with its import path verified outside the source tree.

Full optional/cloud/Docker/downstream and other Python/OS CI runs are not claimed.

Prepared with OpenAI Codex assistance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant