Skip to content

Fix zero-shot calibration defaults - #1261

Open
DingmaomaoBJTU wants to merge 37 commits into
mainfrom
dingmaomaobjtu-run-e2e-tests
Open

Fix zero-shot calibration defaults#1261
DingmaomaoBJTU wants to merge 37 commits into
mainfrom
dingmaomaobjtu-run-e2e-tests

Conversation

@DingmaomaoBJTU

@DingmaomaoBJTU DingmaomaoBJTU commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

Summary

Fix clean-cache QNN zero-shot-classification E2E instability by making zero-shot calibration task-specific instead of reusing the generic text-classification calibration default.

Root cause

Static quantization creates a default calibration reader when callers do not provide custom calibration data. Before this change, zero-shot-classification was mapped to TextDataset, whose default dataset is glue/mrpc.

On clean agents with the current HF stack, load_dataset("glue", "mrpc") can fail while resolving the repo metadata path:

Invalid HF URI 'hf://datasets/glue@.../.huggingface.yaml'. Repository id must be 'namespace/name', got 'glue'.

That failure caused calibration to fall back to RandomDataset. The random token inputs are not representative for an NLI-style zero-shot classifier, so the QNN quantized artifact could produce a low F1 score in the E2E test.

Fix

  • Add ZeroShotClassificationDataset for calibration.
  • Map zero-shot-classification to this specialized dataset instead of TextDataset.
  • Use the same built-in zero-shot defaults as evaluation: fancyzhx/ag_news, test split, candidate labels World,Sports,Business,Sci/Tech, and hypothesis template This text is about {}.
  • Expand each source text into premise/hypothesis pairs before tokenization, matching the input distribution expected by zero-shot/NLI models.
  • Leave other task mappings and generic TextDataset behavior unchanged.

Validation

  • uv run ruff check --fix src\winml\modelkit\datasets\zero_shot_classification.py src\winml\modelkit\datasets\__init__.py tests\integration\datasets\test_zero_shot_classification.py
  • uv run pytest tests\integration\datasets\test_zero_shot_classification.py tests\unit\datasets\test_random_dataset.py -vv
  • Cleared glue HF cache and the cross-encoder/nli-deberta-v3-small WinML artifact cache, then ran:
    uv run pytest -m e2e tests\e2e\test_eval_e2e.py::TestEvalPerTask::test_zero_shot_classification -vv -s --tb=short

github-actions Bot and others added 30 commits July 28, 2026 19:04
Parse wrapped winml build artifact paths so successful builds are not misreported as export failures, and classify QNN backend graph-finalization rejections as unsupported perf skips instead of relying on registry target blacklists.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Temporarily switch HF-style attention configs to eager only while torch.onnx.export runs, then restore the original attention implementation. This keeps transformers 5 exports compatible with QNN by avoiding SDPA mask guard graphs.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Move the eager-attention export helper into transformers_compat so transformers 5 export compatibility behavior stays in one module while preserving lazy import behavior.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Treat Windows native access violation exits from winml build as non-fatal only when the completed ONNX artifact can be resolved from the build output. This preserves real build failures while allowing QNN eval jobs to continue after interpreter teardown crashes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Document the generic export compatibility policy design for EP/device-targeted export overrides and portable default target merging.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add a task-by-task implementation plan for the EP/device export compatibility policy.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Capture perf target explicitness before EP resolution so the implementation plan preserves portable export defaults.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move export compatibility target derivation into build config generation so command paths no longer pass export-policy-specific target state. Preserve portable-default semantics separately from resolved runtime EP/device handles and round-trip resolved empty compatibility configs.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use the resolved runtime EP/device for quantization and compile policy while preserving the original request/default target for export compatibility policy.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Split build CLI runtime targets from export policy targets, restore nested attention configs in topology order, and narrow export compatibility targets.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
github-actions Bot and others added 7 commits July 30, 2026 14:24
Use resolved runtime targets for composite component config generation while preserving the original request for export compatibility policy.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use a task-specific zero-shot calibration dataset that expands AG News text into NLI premise/hypothesis pairs instead of reusing TextDataset's glue/mrpc default.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@DingmaomaoBJTU
DingmaomaoBJTU requested a review from a team as a code owner July 31, 2026 03:21
zhenchaoni added a commit that referenced this pull request Jul 31, 2026
`huggingface_hub` now requires repository ids in `namespace/name` form,
so the legacy bare canonical ids no longer resolve. The default
calibration dataset was still requesting `glue`, which fails with:

```
HfUriError: Invalid HF URI 'hf://datasets/glue@.../.huggingface.yaml'.
Repository id must be 'namespace/name', got 'glue'.
```

This never surfaced as a hard failure. When `TextDataset` construction
fails, `universal_calib_dataset` silently falls back to `RandomDataset`,
so `winml quantize` still reported success while calibrating on random
noise instead of real text. That silently degrades quantization accuracy
for every text task routed through `TextDataset` (`text-classification`,
`feature-extraction`, `fill-mask`, `sentence-similarity`,
`next-sentence-prediction`, `zero-shot-classification`).

The fix is to use fully qualified dataset ids:

- `glue` -> `nyu-mll/glue` (calibration default in `datasets/text.py`)
- `tweet_eval` -> `cardiffnlp/tweet_eval` (e2e testset, recipe configs,
and the matching baseline cache keys)

The baseline cache keys are derived from `dataset_config["path"]`, so
renaming the path required renaming the keys to keep the cached entries
reachable. The recorded metrics are unchanged, so no baselines needed
recomputing. Everything else is example paths in help text and
docstrings.

Verified with `winml export` followed by `winml quantize` on
`philschmid/tiny-bert-sst2-distilled` and
`cross-encoder/nli-deberta-v3-small`. Before the fix both runs logged
the `HfUriError` and fell back to `RandomDataset`; after the fix both
load real MRPC sentence pairs.

This also covers the zero-shot problem reported in #1261.
`zero-shot-classification` routes through the same `TextDataset`, and
MRPC is natively a sentence-pair dataset, so calibration now gets the
premise/hypothesis style input those models expect.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants