fix: hf.revision property to pin HuggingFace Hub reads - #3609
Conversation
huggingface_hub.HfFileSystem resolves revision per call rather than as a filesystem-wide default, so hf.revision is threaded through as a read-time kwarg on FsspecInputFile. Writes and deletes always target the repository's default branch, since a pinned revision can be an immutable tag or commit.
kszucs
changed the title
hf.revision property to pin HuggingFace Hub reads
… test FsspecInputFile.exists() reimplemented exists()-via-info() with a try/except catching only FileNotFoundError, unlike fsspec's own lexists()/exists() which swallow any exception. Delegate to self._fs.exists() directly instead, which already forwards kwargs to info() with the correct broad exception handling. Also add an HF_TOKEN-gated integration test against a real (temporary) Hugging Face dataset repo, demonstrating the concrete problem this property solves: without hf.revision, reads always follow the repo's moving default branch, so a file "written" at one commit silently returns different content once someone pushes a new commit to the same path. Minor: use a _HF_SCHEMES frozenset in _get_fs_kwargs for consistency with the existing _ADLS_SCHEMES dispatch pattern, and de-duplicate/strengthen two tests that asserted only on the private _fs_kwargs attribute.
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
|
Superseded by #3890 (rebased on current main, plus revision support for writes/deletes and precedence for revisions embedded in the location) — GitHub wouldn't let this one be reopened. |
1 participant
Rationale for this change
HuggingFace Hub (
hf://) storage support already works viafsspec'sHfFileSystem, but has no way to pin reads to a specific revision (branch/tag/commit) for reproducibility, unlike iceberg-rust'sHF_REVISION.HfFileSystemresolvesrevisionper call rather than as a filesystem-wide default, sohf.revisionis threaded through as a read-time kwarg onFsspecInputFile. Writes and deletes intentionally always target the repository's default branch, since a pinned revision can be an immutable tag or commit and isn't a valid write target.Are these changes tested?
Yes: unit tests covering revision forwarding on reads (
len(),.exists(),.open()), confirming writes don't receive the revision kwarg, the no-revision-set default, and non-hfschemes being unaffected.Are there any user-facing changes?
Yes: new
hf.revisiontable/catalog property, documented inconfiguration.md.