feat(eval): add audio classification evaluation and refresh Gustking deepfake recipes

[ssss141414]

Summary

This L2 contribution adds generalized audio-classification Eval support and refreshes the existing FP32 and FP16 CPU recipes for Gustking/wav2vec2-large-xlsr-deepfake-audio-classification. It enables bounded, semantically checked real-versus-fake audio evaluation while preserving exact recipe tuple identities. Candidate c76c0eff5c99dd9fddfbd550e19dd91d3776a058 reached Goal L3 PASS with full coverage and no deferred tuples; the shipped Outcome remains L2.

Model metadata

What the model does

This checkpoint accepts speech waveforms and emits two sequence-classification logits interpreted as real human speech or fake/synthetic speech for deepfake-audio detection. Evidence: the pinned config declares Wav2Vec2ForSequenceClassification and id2label={0: real, 1: fake}; the pinned model card states deepfake audio classification and reports ASVspoof2019 evaluation; and the graph is input_values[1,16000] -> logits[1,2]. Confidence: verified.

Primary user stories

• A user supplies a speech recording to obtain real-versus-fake logits for screening synthetic or manipulated audio. Evidence: pinned model-card purpose and real/fake label map. Confidence: verified.

Supported tasks

• audio-classification across checkpoint, Transformers, and WinML support surfaces. Evidence: Hub pipeline_tag=audio-classification, Transformers AutoModelForAudioClassification, and WinML inspect resolving audio-classification with Wav2Vec2OnnxConfig. Confidence: verified.

Model architecture

Wav2Vec2ForSequenceClassification
|-- Wav2Vec2Model
|   |-- Convolutional feature encoder x7 (channels 512)
|   |-- Feature projection (512 -> 1024)
|   `-- Stable-LayerNorm encoder stack x24
|       |-- Self-attention (16 heads, hidden 1024)
|       `-- Feed-forward (1024 -> 4096 -> 1024, GELU)
|-- Projector (1024 -> 256)
|-- Temporal mean pooling
`-- Classifier (256 -> 2: real, fake)

• Source/confidence: pinned checkpoint Gustking/wav2vec2-large-xlsr-deepfake-audio-classification@f7050b586236dc910d1157f430def2d0647b02b4, pinned config dimensions, Transformers Wav2Vec2ForSequenceClassification source, and hierarchy metadata (409 modules, 315,701,634 parameters, 118 traced modules); verified.

Validation and support evidence

Baseline

• Authoritative base is main at be3e59dd412d4918c5a852aa4d8f0207c34aaf6f, WinML 0.0.1.dev0. Main advanced from 8d631f6f1e5db26a04e8c13045280408c6f984dc; Planner issued a fail-closed PARTIAL-RERUN because src/winml/modelkit/__init__.py changed ONNX Runtime initialization and its paired test was deleted.
• Planner reused only the frozen model profile, inspect/config diagnosis, dataset selection and semantics, and E/G/O scope. The previous baseline measured build PASS in 156.0 s for Wav2Vec2ForSequenceClassification, 315.7M parameters, with FLOAT input_values[1,16000] -> logits[1,2]; perf mean 139.492 ms, p50 134.010 ms, throughput 7.17 samples/s, total RSS delta 91.05 MB; and pinned-PyTorch parity cosine 0.9999999985074244, maximum absolute error 0.000011314637959003448, top-1 match. These baseline measurements remain explicitly bound to the prior main evidence commit and are not presented as current-candidate results.
• Auto-config resolved audio-classification, AutoModelForAudioClassification, and transformers_attention=eager. Optimum 2 had no vendor Wav2Vec2 registrations until WinML registration; verdict WINML-ONLY.
• Baseline Eval was CLI-BLOCKED before dataset loading because audio-classification was absent from WinML Eval. Baseline Analyze was CLI-BLOCKED because runtime rule Parquets were absent; an independent graph census found 873 nodes / 14 op types, dominated by Add=227, MatMul=194, Transpose=111, Reshape=96, and Mul=88.
• The candidate patch remained equivalent after rebase, but package-level runtime reach invalidated build, perf, parity, Eval, Analyze, and quality evidence. Tester freshly reran that complete closure on c76c0eff5c99dd9fddfbd550e19dd91d3776a058; no superseded-candidate measurement is used below.

Goal

• Effort: L2.
• Committed Goal ceiling: L3, marching through build, perf, pinned-PyTorch numeric parity, and bounded task-metric functional smoke.
• Outcome: L2, comprising generalized task-family evaluator support, regression coverage, and target recipe refreshes.
• Success required both CPU/cpu precisions to pass L0 build, L1 perf, and L2 parity, plus one final-candidate FP32 CPU L3 functional smoke with exact dataset/checkpoint semantics and bounded fan-out. No replacement ceiling was issued.

Outcome

• Shipped tier: L2. Highest Goal verdict: L3 PASS. Coverage: full. Deferred tuples: none.
• Candidate: c76c0eff5c99dd9fddfbd550e19dd91d3776a058; tree c2beedf4dee41579cd4323f43934be761daca897; base be3e59dd412d4918c5a852aa4d8f0207c34aaf6f.
• Shared generalized support is isolated in droppable commit bf781055f334676ba8dc6af6981347d48b866792; target-only recipe refresh and recipe regression coverage are isolated in c76c0eff5c99dd9fddfbd550e19dd91d3776a058.
• Refreshed Learner findings wav2vec2-027..030 capture artifact/architecture facts, CPU precision tradeoffs, exact bounded real/fake semantics, and static Analyze interpretation. They are published separately in Lane A draft PR Save op columns to json files. #312 at head 8d010b1dbc646b41aabf006a57b15b231bbba32a.
• Draft PRs feat(eval): add audio classification evaluation and music genre recipes #1388 and feat(eval): add audio classification evaluation and MMS language ID recipes #1389 were open, draft, and unmerged at the last authoritative check. This branch is independently based on current main and does not stack on either. The two-commit split makes overlap removal mechanical: bf781055 owns shared evaluator support, while c76c0eff owns only the Model14 recipes and recipe regression. If either PR merges before shipment or review, stop; Planner must rebase and compare the merged implementation, drop or reconcile bf781055, retain/rebase the target-only commit only when still needed, and route every impacted stage back to Tester before this body may be used.

Per-EP/device/precision results and Functional smoke Eval

Tier	EP / Device	Precision	Verdict	Mean	p50	Throughput	RAM delta
L0	CPUExecutionProvider / cpu	fp32	PASS	-	-	-	-
L0	CPUExecutionProvider / cpu	fp16	PASS	-	-	-	-
L1	CPUExecutionProvider / cpu	fp32	PASS	148.523 ms	152.364 ms	6.73 samples/s	+91.23 MB
L1	CPUExecutionProvider / cpu	fp16	PASS	162.778 ms	163.186 ms	6.14 samples/s	+43.45 MB
L2	CPUExecutionProvider / cpu	fp32	PASS	-	-	-	-
L2	CPUExecutionProvider / cpu	fp16	PASS	-	-	-	-
L3	CPUExecutionProvider / cpu	fp32	PASS	-	-	-	-

L0 artifact and graph signatures

• FP32: ONNX 293,409 bytes plus external data 1,262,801,920 bytes; initializers FLOAT=425 and INT64=11. Input input_values FLOAT [1,16000]; output logits FLOAT [1,2]; CPUExecutionProvider output [1,2] was finite.
• FP16: ONNX 293,716 bytes plus external data 631,400,448 bytes; initializers FLOAT16=425 and INT64=11; FP16/FP32 external-data ratio 0.4999995945524061. I/O remains FP32: input_values FLOAT [1,16000] and logits FLOAT [1,2]; CPUExecutionProvider output [1,2] was finite.

L1 memory details

• FP32: RSS baseline 1689.14 MB; after compile 1689.45 MB; after inference and checkpoint peak 1780.38 MB; model-load delta 0.30 MB; inference delta 90.93 MB; total delta 91.23 MB. Local/shared VRAM baseline, after compile, after inference, checkpoint peak, and all load/inference/total deltas were 0.0 MB.
• FP16: RSS baseline 1746.40 MB; after compile 1746.70 MB; after inference and checkpoint peak 1789.86 MB; model-load delta 0.30 MB; inference delta 43.15 MB; total delta 43.45 MB. Local/shared VRAM baseline, after compile, after inference, checkpoint peak, and all load/inference/total deltas were 0.0 MB.

L2 numeric parity

• FP32 PASS: cosine 0.9999999983582369; maximum absolute error 0.000011951662600040436; reference logits [0.008207489736378193, -0.007818397134542465]; ONNX logits [0.008219093084335327, -0.007830348797142506]; reference and ONNX class 0 (real). Bounds: cosine >= 0.99999, maximum absolute error <= 0.001, predicted-class match required. Provider: CPUExecutionProvider; input input_values [1,16000] tensor(float).
• FP16 PASS: cosine 0.9999545270207585; maximum absolute error 0.002589728683233261; reference logits [0.008207489736378193, -0.007818397134542465]; ONNX logits [0.010719520039856434, -0.010408125817775726]; reference and ONNX class 0 (real). Bounds: cosine >= 0.999, maximum absolute error <= 0.02, predicted-class match required. Provider: CPUExecutionProvider; input input_values [1,16000] tensor(float).

Functional smoke Eval

FP32 CPU Eval PASS on candidate c76c0eff5c99dd9fddfbd550e19dd91d3776a058 using public, ungated CC-BY-4.0 dataset garystafford/deepfake-audio-detection@fcf5344bb7f82b54b6b932291326d29750ef1e82, config default, split train, with deterministic first usable row per ClassLabel. Dataset and checkpoint labels matched exactly as raw scalar/ClassLabel IDs: 0=real, 1=fake; media came from embedded FLAC bytes/path and was decoded with SoundFile rather than inferred from filenames.

The selected real row was yt_0000_p2_part_167.flac: 477,353 bytes, 44,100 Hz, stereo, 230,951 frames, 5.236984126984127 seconds. The selected fake row was el_0001_c_part_002.flac: 131,470 bytes, 16,000 Hz, mono, 57,792 frames, 3.612 seconds.

Accounting was exact: scan 934, eligible 1,866, selected 2, processed 2, inference windows 4, truncated 2, rejected 0. Each utterance was converted to mono, resampled to 16,000 Hz, normalized through the feature extractor, capped at 2 seconds / 32,000 samples, split into at most two 16,000-sample windows, and reduced once per utterance by arithmetic mean of window logits; total inference was capped at 4 windows. Targets and predictions were real -> real once and fake -> fake once. Accuracy was 1.0 and macro-F1 was 1.0, with 2/2 represented classes and class coverage 1.0.

This is functional-smoke evidence of end-to-end operability only. Two selected utterances are not representative accuracy or benchmark-quality evidence, and no Eval accuracy claim is made for FP16 or another EP.

Quality gates

• Focused: 15 passed in 23.02 s.
• Affected Eval/recipe partitions: 678 passed, 1 warning in 64.63 s (0:01:04).
• Analyze: 1,529 passed, 45 skipped in 334.89 s (0:05:34).
• Models: 1,541 passed, 7 skipped, 1 xfailed in 63.26 s (0:01:03).
• Optim: 876 passed, 16 skipped, 1 xfailed in 54.88 s.
• Commands/config/build/compiler/session/Eval: 3,697 passed, 9 skipped, 2 warnings in 343.79 s (0:05:43).
• Remaining core/ONNX/cache/utils/helpers/sysinfo/inspect/optracing/serve/regression/CLI: 964 passed, 2 skipped, 1 deselected, 1 warning in 639.74 s (0:10:39).
• Partition marker exclusions: not e2e and not npu and not gpu.
• Mypy baseline parent and candidate each reported Success: no issues found in 442 source files; both also noted unused pyproject sections for onnxruntime_genai.*, openvino.*, tests, and tests.*.
• Ruff: PASS. License: PASS.

Delta

Recipe refresh

Both existing exact-tuple recipe files change only JSON pointer /export/compatibility/transformers_attention from null to "eager":

• examples/recipes/Gustking_wav2vec2-large-xlsr-deepfake-audio-classification/cpu/cpu/audio-classification_fp16_config.json.
• examples/recipes/Gustking_wav2vec2-large-xlsr-deepfake-audio-classification/cpu/cpu/audio-classification_fp32_config.json.

The delta is reducibility-consistent with the charter. Recipe authority was verified without recipe-owned CLI overrides. The production examples/recipes/README.md remains untouched.

Complete changed paths

Droppable shared evaluator commit bf781055f334676ba8dc6af6981347d48b866792:

• src/winml/modelkit/commands/eval.py
• src/winml/modelkit/eval/__init__.py
• src/winml/modelkit/eval/audio_classification_evaluator.py
• src/winml/modelkit/eval/config.py
• src/winml/modelkit/eval/evaluate.py
• src/winml/modelkit/utils/eval_utils.py
• tests/unit/eval/test_audio_classification_evaluator.py

Target-only commit c76c0eff5c99dd9fddfbd550e19dd91d3776a058:

• examples/recipes/Gustking_wav2vec2-large-xlsr-deepfake-audio-classification/cpu/cpu/audio-classification_fp16_config.json
• examples/recipes/Gustking_wav2vec2-large-xlsr-deepfake-audio-classification/cpu/cpu/audio-classification_fp32_config.json
• tests/unit/recipes/test_cpu_recipes.py

No other-model recipe, recipe README, or checkpoint-hardcoded behavior is included.

Bug fix explanation

1. User-visible symptom and trigger: baseline winml eval --task audio-classification failed before dataset loading because the task was absent from the Eval registry; an explicit audio dataset and a standard single-logits audio classifier were sufficient to trigger the missing path.
2. Root cause: Eval had no audio-classification registry/schema entry, raw-media adapter, waveform preprocessing/window aggregation, exact target mapping, prediction decoder, or classification metric path. Existing evaluators could not safely reinterpret raw audio rows or preserve their label semantics.
3. Changed symbols and mechanism: WinMLAudioClassificationEvaluator, _AudioModelAdapter, DatasetConfig.max_duration_seconds, _EVALUATOR_REGISTRY, and TASK_SCHEMAS add raw bytes/path decoding, mono conversion, rational resampling, feature-extractor normalization, bounded static windows, utterance-mean logits, exact scalar/ClassLabel mapping, predictions, accuracy/macro-F1, errors, and accounting.
4. General/data-driven rule: activation is derived from task=audio-classification, one static audio input, and one rank-2 [batch,classes] logits output. Dataset/checkpoint label namespaces must match exactly. No checkpoint ID, filename, model-specific recipe, or positional-label inference controls evaluator behavior.
5. Compatibility and blast radius: existing Eval tasks, schemas, mappings, and metrics remain unchanged; audio evaluation requires an explicit semantically compatible dataset and introduces no universal default. Runtime/decode errors still fail, malformed rows are counted, zero processed rows fails closed, and the only intentional public change is support for explicit-dataset audio classification. Existing Gustking tuple identities remain unchanged except for derived attention compatibility metadata.
6. Regression evidence: focused audio tests passed 15/15; affected Eval/recipe tests passed 678 with 1 warning; all listed quality partitions passed; both FP32/FP16 build, perf, and parity tuples passed; and the FP32 bounded functional smoke processed exact raw real/fake rows with 1.0 accuracy and macro-F1.

Analyze summary - component level and op level

Analyze completed with ANALYZE-PARTIAL-SUCCESS and exit code 1 for each artifact because four requested EPs had no shipped rule data. This is static rule compatibility analysis, not accelerator runtime execution.

Component-level summary

Artifact	Architecture coverage	Mapping	Actionable EP findings
FP32	convolutional feature encoder; feature projection; 24-layer stable-LayerNorm encoder; projector; temporal mean pool; real/fake classifier	747/747 mapped, 0 unmapped	None in rule-backed data
FP16	same regions	748/749 mapped, 1 unmapped (graph_input_cast0)	No rule-backed op issue; one input-Cast component-mapping gap

Mapping confidence is mapped. FP16 has 749 graph operators but 748 mapped component nodes: its classifier region includes one Cast, while the separate graph-input Cast remains unmapped. That gap is retained rather than promoted to full component coverage.

Op-level summary

Artifact	Graph	Dominant operators	Rule-backed EP roll-up
FP32	747 operators / 13 unique types	Reshape 244; Gemm 147; Transpose 111; LayerNormalization 57; Add 49	NvTensorRTRTX, QNN, and OpenVINO: all 13 types supported; no partial, unsupported, or unknown types
FP16	749 operators / 14 unique types	Reshape 244; Gemm 147; Transpose 111; LayerNormalization 57; Add 49; Cast 2	NvTensorRTRTX, QNN, and OpenVINO: all 14 types supported; no partial, unsupported, or unknown types

The complete FP32 inventory is Unsqueeze 1, Conv 8, Transpose 111, LayerNormalization 57, Gelu 32, Reshape 244, Gemm 147, Slice 1, Add 49, MatMul 48, Mul 24, Softmax 24, and ReduceMean 1. FP16 adds Cast 2.

CUDAExecutionProvider, MIGraphXExecutionProvider, TensorrtExecutionProvider, and DmlExecutionProvider have no check results because no rule data is shipped for them. These are absent-rule results, not unsupported operators, and they provide no runtime-support claim.

Reproduce commands

$OUT='temp/gustking-wav2vec2-deepfake'
python -m winml.modelkit.cli build -c examples/recipes/Gustking_wav2vec2-large-xlsr-deepfake-audio-classification/cpu/cpu/audio-classification_fp32_config.json -m Gustking/wav2vec2-large-xlsr-deepfake-audio-classification -o $OUT/fp32
python -m winml.modelkit.cli build -c examples/recipes/Gustking_wav2vec2-large-xlsr-deepfake-audio-classification/cpu/cpu/audio-classification_fp16_config.json -m Gustking/wav2vec2-large-xlsr-deepfake-audio-classification -o $OUT/fp16
python -m winml.modelkit.cli perf -m $OUT/fp32/model.onnx --ep cpu --device cpu --iterations 10 --warmup 3
python -m winml.modelkit.cli perf -m $OUT/fp16/model.onnx --ep cpu --device cpu --iterations 10 --warmup 3
python -m winml.modelkit.cli analyze --model $OUT/fp32/model.onnx --ep all --output $OUT/analyze-fp32.json
python -m winml.modelkit.cli analyze --model $OUT/fp16/model.onnx --ep all --output $OUT/analyze-fp16.json
python -m winml.modelkit.cli eval -m $OUT/fp32/model.onnx --model-id Gustking/wav2vec2-large-xlsr-deepfake-audio-classification --dataset garystafford/deepfake-audio-detection --dataset-name default --dataset-revision fcf5344bb7f82b54b6b932291326d29750ef1e82 --task audio-classification --device cpu --ep cpu --samples 2 --max-duration-seconds 2 --split train --no-shuffle --output $OUT/eval-fp32.json

[ssss141414]

APPROVE

No blocking findings.

Reviewed exact candidate c76c0eff5c99dd9fddfbd550e19dd91d3776a058 (tree c2beedf4dee41579cd4323f43934be761daca897, base be3e59dd412d4918c5a852aa4d8f0207c34aaf6f) and all 10 changed files line by line. The generalized audio-classification evaluator/config/CLI support is data-driven and sound: raw bytes/path/virtual-archive decoding, mono conversion, resampling, bounded duration/windowing, mean-logit aggregation, exact label mapping, accuracy/macro-F1, and fail-closed sample accounting are covered. The two target recipes match their exact CPU evidence. No model/dataset hardcoding, other-model recipes, production README, scratch paths, or --no-optimize changes are present.

Fresh evidence and independent reruns agree: L0 is input_values [1,16000] FLOAT -> logits [1,2] FLOAT for both artifacts; FP32 has 425 FLOAT initializers and 1,262,801,920-byte external data, while FP16 has 425 FLOAT16 initializers, preserves FP32 I/O, and has 631,400,448-byte external data. L1 is FP32 148.523 ms / p50 152.364 ms / 6.73 samples/s / 91.23 MB and FP16 162.778 ms / p50 163.186 ms / 6.14 samples/s / 43.45 MB. Independent L2 reproduced cosine/max-abs/class 0.9999999984 / 1.19516626e-05 / 0 and 0.9999545270 / 0.0025897287 / 0. Independent pinned real-media L3 reproduced 1,866 eligible, 2 selected/processed, 4 windows, 2 truncated, 0 rejected, exact real/fake predictions, and accuracy/macro-F1 1.0/1.0 as functional smoke only.

Analyze is valid partial success with expected exit 1: FP32 747/747 mapped and 13 op types; FP16 748/749 mapped with only graph_input_cast0, 749 operators, and 14 types. NvTensorRTRTX, QNN, and OpenVINO rule sets are complete; CUDA, MIGraphX, TensorRT, and DML rule data are absent and are not presented as runtime coverage. Tester quality is exact: focused 15, eval/recipes 678, analyze 1529, models 1541, optim 876, commands 3697, remaining 964; license/Ruff/mypy pass. Reviewer reran focused 15, full Ruff, mypy (442 files), and the affected commands partition (3697 passed, 9 skipped).

Commit topology is genuinely separable: bf781055f334676ba8dc6af6981347d48b866792 contains shared evaluator support and c76c0eff5c99dd9fddfbd550e19dd91d3776a058 contains only the target recipes plus their recipe regression. Stable patch IDs and range-diff exactly match the old candidate. PRs #1388/#1389 remain OPEN drafts. If either merges first, the shared recipe-test edit conflicts, so route Planner for dedupe/rebase and impacted Tester rerun; do not approve this SHA after such a main move.

PR #1390 is OPEN draft by ssss141414, base main, exact branch/head, mergeable, exact sealed body length 18261, exact 10 files, and label model-scale-by-skill. Enumerated 0 line comments, 0 conversation comments, 0 reviews, 0 review threads, and 0 open threads.

Exact-head GitHub checks:

• Analyze (Python): COMPLETED/SUCCESS
• CodeQL: COMPLETED/SUCCESS
• license/cla: COMPLETED/SUCCESS
• lint: COMPLETED/SUCCESS
• test (analyze): COMPLETED/SUCCESS
• test (commands): COMPLETED/SUCCESS
• test (models): COMPLETED/SUCCESS
• test (optim): COMPLETED/SUCCESS
• test (remaining): COMPLETED/SUCCESS

Coverage is full: verified CPU/cpu/fp32 and CPU/cpu/fp16; deferred tuples: none.

Disposition: terminal skill-level APPROVE. Keep the PR draft. This is an ordinary conversation comment, not a GitHub Review state.