fix(datasets): use fully qualified HF dataset ids (#1262) [cherry-pick to v0.3.0] - #1263
Merged
Merged
Conversation
`huggingface_hub` now requires repository ids in `namespace/name` form, so the legacy bare canonical ids no longer resolve. The default calibration dataset was still requesting `glue`, which fails with: ``` HfUriError: Invalid HF URI 'hf://datasets/glue@.../.huggingface.yaml'. Repository id must be 'namespace/name', got 'glue'. ``` This never surfaced as a hard failure. When `TextDataset` construction fails, `universal_calib_dataset` silently falls back to `RandomDataset`, so `winml quantize` still reported success while calibrating on random noise instead of real text. That silently degrades quantization accuracy for every text task routed through `TextDataset` (`text-classification`, `feature-extraction`, `fill-mask`, `sentence-similarity`, `next-sentence-prediction`, `zero-shot-classification`). The fix is to use fully qualified dataset ids: - `glue` -> `nyu-mll/glue` (calibration default in `datasets/text.py`) - `tweet_eval` -> `cardiffnlp/tweet_eval` (e2e testset, recipe configs, and the matching baseline cache keys) The baseline cache keys are derived from `dataset_config["path"]`, so renaming the path required renaming the keys to keep the cached entries reachable. The recorded metrics are unchanged, so no baselines needed recomputing. Everything else is example paths in help text and docstrings. Verified with `winml export` followed by `winml quantize` on `philschmid/tiny-bert-sst2-distilled` and `cross-encoder/nli-deberta-v3-small`. Before the fix both runs logged the `HfUriError` and fell back to `RandomDataset`; after the fix both load real MRPC sentence pairs. This also covers the zero-shot problem reported in #1261. `zero-shot-classification` routes through the same `TextDataset`, and MRPC is natively a sentence-pair dataset, so calibration now gets the premise/hypothesis style input those models expect. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-picks #1262 into the v0.3.0 release branch.
#1262 landed on main after
release/v0.3.0was cut, so it is not in the release branch. Its parent commit is the release branch tip (16152fd4), so this is a clean pick with no conflicts — the diff here is byte-identical to #1262.Original: fix(datasets): use fully qualified HF dataset ids (#1262) by @zhenchaoni