Fix zero-shot calibration defaults - #1261
Open
DingmaomaoBJTU wants to merge 37 commits into
Open
Conversation
Parse wrapped winml build artifact paths so successful builds are not misreported as export failures, and classify QNN backend graph-finalization rejections as unsupported perf skips instead of relying on registry target blacklists. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Temporarily switch HF-style attention configs to eager only while torch.onnx.export runs, then restore the original attention implementation. This keeps transformers 5 exports compatible with QNN by avoiding SDPA mask guard graphs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Move the eager-attention export helper into transformers_compat so transformers 5 export compatibility behavior stays in one module while preserving lazy import behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Treat Windows native access violation exits from winml build as non-fatal only when the completed ONNX artifact can be resolved from the build output. This preserves real build failures while allowing QNN eval jobs to continue after interpreter teardown crashes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Document the generic export compatibility policy design for EP/device-targeted export overrides and portable default target merging. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add a task-by-task implementation plan for the EP/device export compatibility policy. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Capture perf target explicitness before EP resolution so the implementation plan preserves portable export defaults. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move export compatibility target derivation into build config generation so command paths no longer pass export-policy-specific target state. Preserve portable-default semantics separately from resolved runtime EP/device handles and round-trip resolved empty compatibility configs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use the resolved runtime EP/device for quantization and compile policy while preserving the original request/default target for export compatibility policy. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Split build CLI runtime targets from export policy targets, restore nested attention configs in topology order, and narrow export compatibility targets. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use resolved runtime targets for composite component config generation while preserving the original request for export compatibility policy. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This reverts commit ddca869.
Use a task-specific zero-shot calibration dataset that expands AG News text into NLI premise/hypothesis pairs instead of reusing TextDataset's glue/mrpc default. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
zhenchaoni
added a commit
that referenced
this pull request
Jul 31, 2026
`huggingface_hub` now requires repository ids in `namespace/name` form, so the legacy bare canonical ids no longer resolve. The default calibration dataset was still requesting `glue`, which fails with: ``` HfUriError: Invalid HF URI 'hf://datasets/glue@.../.huggingface.yaml'. Repository id must be 'namespace/name', got 'glue'. ``` This never surfaced as a hard failure. When `TextDataset` construction fails, `universal_calib_dataset` silently falls back to `RandomDataset`, so `winml quantize` still reported success while calibrating on random noise instead of real text. That silently degrades quantization accuracy for every text task routed through `TextDataset` (`text-classification`, `feature-extraction`, `fill-mask`, `sentence-similarity`, `next-sentence-prediction`, `zero-shot-classification`). The fix is to use fully qualified dataset ids: - `glue` -> `nyu-mll/glue` (calibration default in `datasets/text.py`) - `tweet_eval` -> `cardiffnlp/tweet_eval` (e2e testset, recipe configs, and the matching baseline cache keys) The baseline cache keys are derived from `dataset_config["path"]`, so renaming the path required renaming the keys to keep the cached entries reachable. The recorded metrics are unchanged, so no baselines needed recomputing. Everything else is example paths in help text and docstrings. Verified with `winml export` followed by `winml quantize` on `philschmid/tiny-bert-sst2-distilled` and `cross-encoder/nli-deberta-v3-small`. Before the fix both runs logged the `HfUriError` and fell back to `RandomDataset`; after the fix both load real MRPC sentence pairs. This also covers the zero-shot problem reported in #1261. `zero-shot-classification` routes through the same `TextDataset`, and MRPC is natively a sentence-pair dataset, so calibration now gets the premise/hypothesis style input those models expect. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix clean-cache QNN zero-shot-classification E2E instability by making zero-shot calibration task-specific instead of reusing the generic text-classification calibration default.
Root cause
Static quantization creates a default calibration reader when callers do not provide custom calibration data. Before this change,
zero-shot-classificationwas mapped toTextDataset, whose default dataset isglue/mrpc.On clean agents with the current HF stack,
load_dataset("glue", "mrpc")can fail while resolving the repo metadata path:That failure caused calibration to fall back to
RandomDataset. The random token inputs are not representative for an NLI-style zero-shot classifier, so the QNN quantized artifact could produce a low F1 score in the E2E test.Fix
ZeroShotClassificationDatasetfor calibration.zero-shot-classificationto this specialized dataset instead ofTextDataset.fancyzhx/ag_news,testsplit, candidate labelsWorld,Sports,Business,Sci/Tech, and hypothesis templateThis text is about {}.TextDatasetbehavior unchanged.Validation
uv run ruff check --fix src\winml\modelkit\datasets\zero_shot_classification.py src\winml\modelkit\datasets\__init__.py tests\integration\datasets\test_zero_shot_classification.pyuv run pytest tests\integration\datasets\test_zero_shot_classification.py tests\unit\datasets\test_random_dataset.py -vvglueHF cache and thecross-encoder/nli-deberta-v3-smallWinML artifact cache, then ran:uv run pytest -m e2e tests\e2e\test_eval_e2e.py::TestEvalPerTask::test_zero_shot_classification -vv -s --tb=short