Skip to content

fix(datasets): use fully qualified HF dataset ids - #1262

Merged
zhenchaoni merged 1 commit into
mainfrom
zhenchaoni-fix-hf-dataset-namespace
Jul 31, 2026
Merged

fix(datasets): use fully qualified HF dataset ids#1262
zhenchaoni merged 1 commit into
mainfrom
zhenchaoni-fix-hf-dataset-namespace

Conversation

@zhenchaoni

Copy link
Copy Markdown
Member

huggingface_hub now requires repository ids in namespace/name form, so the legacy bare canonical ids no longer resolve. The default calibration dataset was still requesting glue, which fails with:

HfUriError: Invalid HF URI 'hf://datasets/glue@.../.huggingface.yaml'.
Repository id must be 'namespace/name', got 'glue'.

This never surfaced as a hard failure. When TextDataset construction fails, universal_calib_dataset silently falls back to RandomDataset, so winml quantize still reported success while calibrating on random noise instead of real text. That silently degrades quantization accuracy for every text task routed through TextDataset (text-classification, feature-extraction, fill-mask, sentence-similarity, next-sentence-prediction, zero-shot-classification).

The fix is to use fully qualified dataset ids:

  • glue -> nyu-mll/glue (calibration default in datasets/text.py)
  • tweet_eval -> cardiffnlp/tweet_eval (e2e testset, recipe configs, and the matching baseline cache keys)

The baseline cache keys are derived from dataset_config["path"], so renaming the path required renaming the keys to keep the cached entries reachable. The recorded metrics are unchanged, so no baselines needed recomputing. Everything else is example paths in help text and docstrings.

Verified with winml export followed by winml quantize on philschmid/tiny-bert-sst2-distilled and cross-encoder/nli-deberta-v3-small. Before the fix both runs logged the HfUriError and fell back to RandomDataset; after the fix both load real MRPC sentence pairs.

This also covers the zero-shot problem reported in #1261. zero-shot-classification routes through the same TextDataset, and MRPC is natively a sentence-pair dataset, so calibration now gets the premise/hypothesis style input those models expect.

Bare canonical dataset ids are no longer resolvable by huggingface_hub,
which now requires a 'namespace/name' repository id. Loading 'glue'
raises HfUriError, so the calibration dataset silently fell back to
RandomDataset for every text task.

Use 'nyu-mll/glue' and 'cardiffnlp/tweet_eval' instead.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@zhenchaoni
zhenchaoni requested a review from a team as a code owner July 31, 2026 05:27
@zhenchaoni
zhenchaoni merged commit 31bb863 into main Jul 31, 2026
9 checks passed
@zhenchaoni
zhenchaoni deleted the zhenchaoni-fix-hf-dataset-namespace branch July 31, 2026 05:41
DingmaomaoBJTU pushed a commit that referenced this pull request Jul 31, 2026
…k to v0.3.0] (#1263)

Cherry-picks #1262 into the v0.3.0 release branch.

#1262 landed on main after `release/v0.3.0` was cut, so it is not in the
release branch. Its parent commit is the release branch tip
(`16152fd4`), so this is a clean pick with no conflicts — the diff here
is byte-identical to #1262.

**Original:** fix(datasets): use fully qualified HF dataset ids (#1262)
by @zhenchaoni

Co-authored-by: Zhenchao Ni <zhenni@microsoft.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants