Feature gap
winml eval does not register automatic-speech-recognition, so valid Wav2Vec2 CTC artifacts cannot be evaluated with a task metric through the CLI.
Reproduction
- Model:
MahmoudAshraf/mms-300m-1130-forced-aligner
- Task:
automatic-speech-recognition
- Artifacts: verified CPU FP32 and FP16 ONNX models with
input_values [1,16000] -> logits [1,49,31]
Run either:
winml eval --schema --task automatic-speech-recognition
or:
winml eval -m <model.onnx> --model-id MahmoudAshraf/mms-300m-1130-forced-aligner --task automatic-speech-recognition --device cpu --ep cpu
Both are rejected because ASR is absent from the supported-task registry.
Requested support
Add an ASR evaluator/schema with raw-audio preprocessing and a defensible transcript/audio dataset contract. For CTC checkpoints, expose an appropriate metric such as WER; forced-alignment checkpoints may additionally need alignment/timestamp quality evaluation.
The referenced model card only says “forced alignment dataset” and does not identify a dataset repository, subset, split, or revision, so callers must be able to provide an explicit evaluation dataset rather than silently substituting one.
Feature gap
winml evaldoes not registerautomatic-speech-recognition, so valid Wav2Vec2 CTC artifacts cannot be evaluated with a task metric through the CLI.Reproduction
MahmoudAshraf/mms-300m-1130-forced-alignerautomatic-speech-recognitioninput_values [1,16000] -> logits [1,49,31]Run either:
winml eval --schema --task automatic-speech-recognitionor:
winml eval -m <model.onnx> --model-id MahmoudAshraf/mms-300m-1130-forced-aligner --task automatic-speech-recognition --device cpu --ep cpuBoth are rejected because ASR is absent from the supported-task registry.
Requested support
Add an ASR evaluator/schema with raw-audio preprocessing and a defensible transcript/audio dataset contract. For CTC checkpoints, expose an appropriate metric such as WER; forced-alignment checkpoints may additionally need alignment/timestamp quality evaluation.
The referenced model card only says “forced alignment dataset” and does not identify a dataset repository, subset, split, or revision, so callers must be able to provide an explicit evaluation dataset rather than silently substituting one.