The CI ASR benchmark scores identical hypothesis text differently between runs.
Evidence from the asr-results artifacts of two runs on adjacent heads (#908 run 34606750813, #910 run 34630144506; the two heads differ only in code the ladder never exercises on these files):
| record |
hypothesis (byte-identical in both artifacts) |
#908 run |
#910 run |
test-other v3 1688-142285-0002.flac |
You don't mean that you thought me so silly. |
WER 0%, CER 0% |
WER 10%, CER 2.9% |
Reference: YOU DON'T MEAN THAT YOU THOUGHT ME SO SILLY. The other 99 records (v2/v3 × clean/other) are identical between the runs. That single record moves the v3 test-other average from 1.19% to 1.59% on a 25-file subset. Locally the same file scores 0% in eight separate processes on the #910 build. Earlier runs of the same #908 head also reported v2 test-other at 1.40% and 1.56%.
AsrBenchmark computes WER from the same asrResult.text it stores as the hypothesis, so the difference is inside the scoring path. TextNormalizer.normalize (CLI) applies several replacement tables by iterating Swift dictionaries (contractions, britishToAmerican, abbreviations, numberWords) with replacingOccurrences, and the tables contain overlapping keys (n't and 't, for instance), so the result of a normalization can depend on dictionary iteration order, which Swift randomizes per process; a 1-character CER on a sentence with don't fits that shape. Not yet reproduced locally, so the mechanism is a suspicion; the run-to-run drift on identical code is the fact.
Suggested fix: iterate the replacement tables in a fixed order (sorted keys, or arrays of pairs ordered longest-first), and add a unit test that normalizes a set of sentences under several SWIFT_DETERMINISTIC_HASHING seeds or across processes. Until then, benchmark rows can move by a whole file's worth on a 25-file subset without a code change.
The CI ASR benchmark scores identical hypothesis text differently between runs.
Evidence from the
asr-resultsartifacts of two runs on adjacent heads (#908 run 34606750813, #910 run 34630144506; the two heads differ only in code the ladder never exercises on these files):test-otherv31688-142285-0002.flacYou don't mean that you thought me so silly.Reference:
YOU DON'T MEAN THAT YOU THOUGHT ME SO SILLY. The other 99 records (v2/v3 × clean/other) are identical between the runs. That single record moves the v3 test-other average from 1.19% to 1.59% on a 25-file subset. Locally the same file scores 0% in eight separate processes on the #910 build. Earlier runs of the same #908 head also reported v2 test-other at 1.40% and 1.56%.AsrBenchmarkcomputes WER from the sameasrResult.textit stores as the hypothesis, so the difference is inside the scoring path.TextNormalizer.normalize(CLI) applies several replacement tables by iterating Swift dictionaries (contractions,britishToAmerican,abbreviations,numberWords) withreplacingOccurrences, and the tables contain overlapping keys (n'tand't, for instance), so the result of a normalization can depend on dictionary iteration order, which Swift randomizes per process; a 1-character CER on a sentence withdon'tfits that shape. Not yet reproduced locally, so the mechanism is a suspicion; the run-to-run drift on identical code is the fact.Suggested fix: iterate the replacement tables in a fixed order (sorted keys, or arrays of pairs ordered longest-first), and add a unit test that normalizes a set of sentences under several
SWIFT_DETERMINISTIC_HASHINGseeds or across processes. Until then, benchmark rows can move by a whole file's worth on a 25-file subset without a code change.