[WIP] Add benchmarking tool to compare tool witha a baseline

[fedeizzo]

What does this PR do?

This PR tries to update the existing warmbench script to have an experiment -> validation loop between two versions of audio.cpp.

./tools/bench.py warmbench \
  --family qwen3_tts \
  --backend vulkan \
  --baseline-bin /path/to/baseline/build/bin/audiocpp_cli

An example of the output can be found at the end of the pr description (I'm working on a PR with some vulkan improvements).

⚠️ this PR is not ready yet, I used a lot AI to quickly iterate on some ideas and validate them.

@0xShug0 what do you think? can we integrate this in the main repo once the code is properly written?

📊 Benchmark Report: qwen3_tts (vulkan)

🧪 Validation Status

Test	Status	Details
Baseline Output Valid	✅ Pass
New Output Valid	✅ Pass
Parity (Request 0)	❌ Fail	mismatch:samples
Parity (Request 1)	❌ Fail	mismatch:samples

⏱️ Performance Metrics

Metric	Baseline	New	Diff	% Change
qwen3_tts.prefill_build_ms	0.54	0.55	+0.01	🔴 +2.13%
qwen3_tts.speaker_encoder.graph.build_ms	0.26	0.21	-0.05	🟢 -20.95%
qwen3_tts.speech_decoder.graph.build_ms	404.90	415.54	+10.65	🔴 +2.63%
qwen3_tts.speech_decoder.graph.compute_ms	8261.19	2854.28	-5406.92	🟢 -65.45%
qwen3_tts.speech_decoder.graph.rebuilds	3.00	3.00	+0.00	+0.00%
qwen3_tts.speech_decoder.input_upload_ms	0.27	0.32	+0.05	🔴 +18.65%
qwen3_tts.speech_decoder.output_read_ms	1.55	1.48	-0.07	🟢 -4.28%
qwen3_tts.speech_decoder.total_ms	8668.55	3272.27	-5396.28	🟢 -62.25%
qwen3_tts.speech_decoder_ms	8668.74	3272.45	-5396.29	🟢 -62.25%
qwen3_tts.speech_tokenizer_encoder.graph.build_ms	0.34	0.33	-0.02	🟢 -5.55%
qwen3_tts.synthesize_wall_ms	9158.21	6115.80	-3042.41	🟢 -33.22%
qwen3_tts.talker.cached_step.graph.compute_ms	3468.76	2640.39	-828.38	🟢 -23.88%
qwen3_tts.talker.cached_step.input_upload_ms	43.56	39.23	-4.33	🟢 -9.95%
qwen3_tts.talker.cached_step.kv_copy_ms	0.00	0.00	N/A	N/A
qwen3_tts.talker.cached_step.mask_upload_ms	6.69	5.86	-0.83	🟢 -12.42%
qwen3_tts.talker.cached_step.output_read_ms	20.85	16.56	-4.29	🟢 -20.56%
qwen3_tts.talker.cached_step_build_ms	20.87	13.94	-6.93	🟢 -33.21%
qwen3_tts.talker.cached_step_ms	3540.53	2702.54	-837.99	🟢 -23.67%
qwen3_tts.talker.code_predictor.graph.compute_ms	7822.50	7345.50	-477.00	🟢 -6.10%
qwen3_tts.talker.code_predictor.input_upload_ms	736.06	671.91	-64.15	🟢 -8.72%
qwen3_tts.talker.code_predictor.output_read_ms	156.66	134.53	-22.12	🟢 -14.12%
qwen3_tts.talker.code_predictor_build_ms	8.15	6.58	-1.58	🟢 -19.36%
qwen3_tts.talker.code_predictor_ms	9123.48	8541.36	-582.12	🟢 -6.38%
qwen3_tts.talker.frame_embed_ms	2.75	2.18	-0.57	🟢 -20.80%
qwen3_tts.talker.import_prefill_state_ms	43.42	129.37	+85.95	🔴 +197.94%
qwen3_tts.talker.prefill.graph.build_ms	49.53	49.72	+0.19	🔴 +0.38%
qwen3_tts.talker.prefill_ms	227.67	222.39	-5.28	🟢 -2.32%
qwen3_tts.talker.processor_ms	28.89	27.12	-1.77	🟢 -6.13%
qwen3_tts.talker.prompt_state_ms	483.43	481.82	-1.61	🟢 -0.33%
qwen3_tts.talker.total_ms	13498.82	12149.79	-1349.03	🟢 -9.99%
qwen3_tts.talker_ms	13502.59	12152.69	-1349.90	🟢 -10.00%
qwen3_tts.voice_prompt_ms	3018.49	3040.10	+21.61	🔴 +0.72%
session.wall_ms	25190.82	18466.23	-6724.58	🟢 -26.69%

[fedeizzo]

Another example of massive improvement

📊 Benchmark Report: voxcpm2 (vulkan)

🧪 Validation Status

Test	Status	Details
Baseline Output Valid	✅ Pass
New Output Valid	✅ Pass
Parity (Request 0)	✅ Pass	ok
Parity (Request 1)	✅ Pass	ok

⏱️ Performance Metrics

Metric	Baseline	New	Diff	% Change
session.wall_ms	31068.68	20454.04	-10614.64	🟢 -34.17%
voxcpm2.audiovae.decoder.graph.build_ms	0.84	7.45	+6.61	🔴 +785.78%
voxcpm2.audiovae.decoder.graph.rebuilt	1.00	1.00	+0.00	+0.00%
voxcpm2.audiovae.decoder.graph.reused	95.00	95.00	+0.00	+0.00%
voxcpm2.audiovae.decoder.latent_capacity	384.00	384.00	+0.00	+0.00%
voxcpm2.audiovae_decoder_ms	11238.63	1608.07	-9630.55	🟢 -85.69%
voxcpm2.generator_ms	19830.03	18845.94	-984.09	🟢 -4.96%
voxcpm2.generator_streaming_callbacks_ms	31068.65	20454.01	-10614.64	🟢 -34.17%
voxcpm2.streaming_chunks	96.00	96.00	+0.00	+0.00%

[fedeizzo]

@0xShug0 some massive gains for vulkan should come for free if external/ggml is updated, at least by looking at similar code in llama.cpp (which has a newer version). Do you plan to bump it?

[0xShug0]

@fedeizzo This is super useful! The current repo already has a path test tool that does something similar: it compares session wall time and output parity against a given run using collected logs. I’d suggest extending the path test compare tool with a more detailed timing breakdown instead of using warmbench, which will be deprecated soon.
https://github.com/0xShug0/audio.cpp/blob/main/tools/audiocpp_cli/compare_audiocpp_cli_path_results.py.

For GGML update, I have did my test a few days ago #63 (comment). My decision was not to change GGML. Of course, the RTF can be misleading as a session wall time of 10 ms versus 11 ms already translates to about a 10% difference in RTF. The reason for using RTF is that changing GGML also produced WAVs with different numbers of frames.

(RTF/session wall deltas that are likely runtime noise are filtered)

v16 -- v.0.16.0
v17 -- v.0.17.0

v16 CUDA vs vendor

Family	Requests	Vendor RTF	v16 CUDA RTF	RTF Delta	Result
miocodec	1	0.0305	0.0243	-20.5%	Win
qwen3_forced_aligner	1	0.0129	0.0106	-18.3%	Win
stable_audio	1	0.0266	0.0231	-13.2%	Win
moss_tts_nano	1	0.0815	0.0739	-9.3%	Win
citrinet_asr	1	0.0133	0.0141	+6.1%	Loss
sortformer_diar	1	0.0034	0.0036	+6.2%	Loss
pocket_tts	3	0.0116-0.0178	0.0121-0.0199	+6.3%	Loss
marblenet_vad	1	0.0062	0.0067	+7.7%	Loss
vevo2	11	route mix	route mix	+18.7%	Loss

v17 CUDA vs vendor

Family	Requests	Vendor RTF	v17 CUDA RTF	RTF Delta	Result
citrinet_asr	1	0.0133	0.0084	-37.1%	Win
marblenet_vad	1	0.0062	0.0051	-18.7%	Win
qwen3_forced_aligner	1	0.0129	0.0110	-14.8%	Win
stable_audio	1	0.0266	0.0231	-13.2%	Win
moss_tts_nano	1	0.0815	0.0751	-7.9%	Win
hviske_asr	4	0.0068-0.0191	0.0058-0.0186	-5.7%	Win
pocket_tts	3	0.0116-0.0178	0.0120-0.0202	+6.3%	Loss
omnivoice	1	0.0472	0.0503	+6.5%	Loss
nemotron_asr	1	0.0162	0.0176	+8.8%	Loss
miocodec	1	0.0305	0.0350	+14.4%	Loss
sortformer_diar	1	0.0034	0.0050	+46.4%	Loss

v16 Vulkan vs vendor

Family	Requests	Vendor RTF	v16 Vulkan RTF	RTF Delta	Result
marblenet_vad	1	0.0062	0.0007	-88.2%	Win
citrinet_asr	1	0.0133	0.0025	-81.0%	Win
qwen3_forced_aligner	1	0.0129	0.0055	-57.6%	Win
sortformer_diar	1	0.0034	0.0022	-34.4%	Win
seed_vc	1	0.0511	0.0400	-21.7%	Win
mel_band_roformer	1	0.0670	0.0593	-11.4%	Win
supertonic	2	0.0108-0.0314	0.0155-0.0187	-7.7%	Win
vibevoice_asr	1	0.0847	0.0903	+6.7%	Loss
heartmula	1	0.3092	0.3483	+12.6%	Loss
stable_audio	1	0.0266	0.0300	+12.7%	Loss
chatterbox	1	0.1804	0.2069	+14.7%	Loss
higgs_audio_stt	3	0.0034-0.0211	0.0056-0.0194	+19.0%	Loss
qwen3_tts	3	0.1448-0.2000	0.1831-0.2465	+23.8%	Loss
vevo2	11	route mix	route mix	+40.0%	Loss
hviske_asr	4	0.0068-0.0191	0.0074-0.0274	+32.5%	Loss
moss_tts_nano	1	0.0815	0.1079	+32.3%	Loss
omnivoice	1	0.0472	0.0627	+32.9%	Loss
moss_tts_local	3	0.2084-0.4617	0.2860-0.7078	+36.5%	Loss
index_tts2	1	0.4362	0.6220	+42.6%	Loss
htdemucs	1	0.0163	0.0291	+79.2%	Loss
vibevoice	1	0.1261	0.2542	+101.7%	Loss
pocket_tts	3	0.0116-0.0178	0.0220-0.0446	+108.9%	Loss
voxcpm2	1	0.1739	0.4089	+135.1%	Loss
irodori_tts	2	0.0629-0.0678	0.1466-0.1871	+153.5%	Loss

v17 Vulkan vs vendor

Family	Requests	Vendor RTF	v17 Vulkan RTF	RTF Delta	Result
marblenet_vad	1	0.0062	0.0010	-83.4%	Win
citrinet_asr	1	0.0133	0.0026	-80.8%	Win
qwen3_forced_aligner	1	0.0129	0.0055	-57.9%	Win
sortformer_diar	1	0.0034	0.0023	-31.0%	Win
seed_vc	1	0.0511	0.0389	-24.0%	Win
mel_band_roformer	1	0.0670	0.0595	-11.2%	Win
stable_audio	1	0.0266	0.0283	+6.4%	Loss
chatterbox	1	0.1804	0.2043	+13.3%	Loss
heartmula	1	0.3092	0.3497	+13.1%	Loss
higgs_audio_stt	3	0.0034-0.0211	0.0056-0.0172	+13.6%	Loss
qwen3_tts	3	0.1448-0.2000	0.1847-0.2389	+23.3%	Loss
miocodec	1	0.0305	0.0386	+26.5%	Loss
vevo2	11	route mix	route mix	+50.3%	Loss
moss_tts_nano	1	0.0815	0.1084	+33.0%	Loss
omnivoice	1	0.0472	0.0633	+34.2%	Loss
index_tts2	1	0.4362	0.5913	+35.6%	Loss
qwen3_asr	1	0.0112	0.0152	+36.1%	Loss
moss_tts_local	3	0.2084-0.4617	0.2851-0.7089	+36.4%	Loss
hviske_asr	4	0.0068-0.0191	0.0075-0.0273	+39.6%	Loss
htdemucs	1	0.0163	0.0290	+78.3%	Loss
vibevoice	1	0.1261	0.2522	+100.1%	Loss
pocket_tts	3	0.0116-0.0178	0.0224-0.0446	+111.2%	Loss
voxcpm2	1	0.1739	0.4287	+146.5%	Loss
irodori_tts	2	0.0629-0.0678	0.1795-0.1815	+176.3%	Loss

v16 CUDA vs vendor

Family	Requests	Vendor wall ms	v16 CUDA wall ms	Timing Delta	Result
miocodec	1	158.9	126.3	-20.5%	Win
qwen3_forced_aligner	1	182.2	148.8	-18.3%	Win
stable_audio	5	272.5-799.2	241.0-693.7	-12.4%	Win
qwen3_tts	9	651.0-2835.2	665.7-2258.1	-11.7%	Win
qwen3_asr	4	19.3-393.1	19.2-364.7	-6.5%	Win
citrinet_asr	1	79.1	83.9	+6.1%	Loss
sortformer_diar	1	47.8	50.8	+6.2%	Loss
marblenet_vad	1	87.4	94.2	+7.7%	Loss
vevo2	11	569.9-1944.5	697.6-3713.0	+18.7%	Loss

v17 CUDA vs vendor

Family	Requests	Vendor wall ms	v17 CUDA wall ms	Timing Delta	Result
citrinet_asr	1	79.1	49.8	-37.1%	Win
marblenet_vad	1	87.4	71.1	-18.7%	Win
qwen3_forced_aligner	1	182.2	155.2	-14.8%	Win
stable_audio	5	272.5-799.2	237.2-693.9	-11.1%	Win
qwen3_tts	9	651.0-2835.2	660.0-2940.0	-7.8%	Win
hviske_asr	4	121.2-308.5	113.5-300.3	-5.7%	Win
nemotron_asr	4	29.2-178.9	30.4-194.7	+7.0%	Loss
miocodec	1	158.9	181.8	+14.4%	Loss
qwen3_asr	4	19.3-393.1	24.1-717.9	+19.3%	Loss
sortformer_diar	1	47.8	70.0	+46.4%	Loss

v16 Vulkan vs vendor

Family	Requests	Vendor wall ms	v16 Vulkan wall ms	Timing Delta	Result
marblenet_vad	1	87.4	10.3	-88.2%	Win
citrinet_asr	1	79.1	15.0	-81.0%	Win
qwen3_forced_aligner	1	182.2	77.2	-57.6%	Win
sortformer_diar	1	47.8	31.4	-34.4%	Win
seed_vc	4	637.5-24672.7	498.9-20757.1	-15.3%	Win
moss_tts_local	6	1292.8-62397.1	1784.3-5074.0	-14.7%	Win
mel_band_roformer	1	1004.4	889.7	-11.4%	Win
supertonic	2	62.7-188.1	89.8-111.9	-7.7%	Win
heartmula	1	18577.6	20923.1	+12.6%	Loss
chatterbox	2	1226.5-1778.7	1406.9-2065.0	+15.4%	Loss
qwen3_tts	9	651.0-2835.2	747.2-3534.6	+18.6%	Loss
ace_step	8	2123.0-11074.2	2116.2-14069.0	+25.7%	Loss
omnivoice	2	195.9-325.6	260.3-411.6	+29.6%	Loss
higgs_audio_stt	12	48.3-297.4	71.3-273.5	+31.7%	Loss
hviske_asr	4	121.2-308.5	146.5-442.0	+32.5%	Loss
qwen3_asr	4	19.3-393.1	64.0-318.5	+37.8%	Loss
vevo2	11	569.9-1944.5	774.6-3227.4	+41.5%	Loss
index_tts2	2	2000.2-3023.7	3011.3-4355.8	+47.3%	Loss
htdemucs	1	2925.8	5244.3	+79.2%	Loss
nemotron_asr	4	29.2-178.9	76.7-185.1	+99.3%	Loss
moss_tts_nano	5	564.2-1683.9	834.0-4754.6	+102.6%	Loss
vibevoice	1	6722.9	13762.4	+104.7%	Loss
pocket_tts	3	46.9-136.4	78.3-392.5	+108.9%	Loss
irodori_tts	5	263.5-506.1	707.7-1189.7	+162.6%	Loss
voxcpm2	3	1038.9-1502.8	2446.6-4577.2	+175.6%	Loss

v17 Vulkan vs vendor

Family	Requests	Vendor wall ms	v17 Vulkan wall ms	Timing Delta	Result
marblenet_vad	1	87.4	14.5	-83.4%	Win
citrinet_asr	1	79.1	15.2	-80.8%	Win
qwen3_forced_aligner	1	182.2	76.7	-57.9%	Win
sortformer_diar	1	47.8	33.0	-31.0%	Win
seed_vc	4	637.5-24672.7	484.6-20641.2	-14.7%	Win
moss_tts_local	6	1292.8-62397.1	1799.6-5744.2	-13.7%	Win
mel_band_roformer	1	1004.4	892.0	-11.2%	Win
stable_audio	5	272.5-799.2	220.7-1130.6	+9.2%	Loss
qwen3_tts	9	651.0-2835.2	752.6-3324.1	+9.5%	Loss
heartmula	1	18577.6	21012.0	+13.1%	Loss
vevo2	11	569.9-1944.5	779.3-2607.4	+23.2%	Loss
miocodec	1	158.9	200.9	+26.5%	Loss
ace_step	8	2123.0-11074.2	2052.8-18242.1	+29.6%	Loss
higgs_audio_stt	12	48.3-297.4	72.3-242.4	+30.2%	Loss
chatterbox	2	1226.5-1778.7	1389.3-2707.8	+31.3%	Loss
omnivoice	2	195.9-325.6	262.8-462.4	+38.0%	Loss
hviske_asr	4	121.2-308.5	148.2-440.5	+39.6%	Loss
index_tts2	2	2000.2-3023.7	2862.7-4368.2	+43.8%	Loss
qwen3_asr	4	19.3-393.1	64.1-390.9	+44.2%	Loss
nemotron_asr	4	29.2-178.9	57.2-183.8	+59.0%	Loss
htdemucs	1	2925.8	5215.7	+78.3%	Loss
moss_tts_nano	5	564.2-1683.9	831.4-4141.9	+98.1%	Loss
vibevoice	1	6722.9	13654.5	+103.1%	Loss
pocket_tts	3	46.9-136.4	79.6-392.3	+111.2%	Loss
irodori_tts	5	263.5-506.1	684.0-1466.4	+171.1%	Loss
voxcpm2	3	1038.9-1502.8	2514.1-4389.2	+178.6%	Loss

[0xShug0]

@fedeizzo For Vulkan optimization, I’d prefer to keep it scoped to Vulkan and specific models, and make the changes additive. There is already a unit test for comparing different GGML lowerings: #48 (comment). Some lowerings work better on Vulkan but are much slower on CUDA in my tests. Since the current priority is still CUDA performance, I'd avoid any changes that could cause a CUDA performance regression.