Speed up voxtral_realtime streaming and stop EOS from ending a stream#116
Conversation
A streaming step advances 80 ms of audio, so it has to cost under 80 ms to track a realtime source. It cost 133 ms/step. Instrument the streaming path first, then optimize against the measurements. Correctness first. process_one_stream_chunk set stream_reached_eos_ on an EOS token and the consuming loop tested it, while nothing outside reset() ever cleared it, so the first EOS halted transcription permanently; once halted the buffer-trim block became unreachable while audio kept being appended. The reference gives EOS no special treatment: the sampled token feeds back unconditionally and only the audio source running dry ends the stream. Special ids emit no text, so the transcript is unchanged. The offline generate() EOS stop is retained. Removing utterance resets is also what makes the fixed-size decode cache below a requirement rather than a nicety. The session also invoked its own event sink and returned the same event to the caller, which forwards it, so every partial was delivered twice, over the CLI and the server's SSE deltas alike. Several chunks can be consumed per process_audio_chunk call, so the sink is the route that yields a partial per step; the returned event now drops what the sink already delivered instead. Granularity is unchanged, the duplicates are gone. Throughput: - Feed the KV caches to flash attention as strided views in both static attention builders instead of materializing a transpose. This was copying the whole K and V cache per layer per token and was the single largest cost. - Hold the encoder's sliding-window KV cache in F16. It is never exported, so this is local, and Metal's flash attention has a much faster F16 path than the halved read traffic alone predicts. - Accept q4_0/q4_k/q5_k/q6_k for the encoder and decoder weights. The published q4_k GGUF avoids the load-time requantization these otherwise cost. - Pin the streaming decode cache to a fixed size (default 1024 steps, ~82 s) built once at begin_stream, so a long session never stalls on a cache-growth rebuild. - Add stream_batch_tokens=N: one encoder forward covers N audio tokens while the decoder still steps once per token. The encoder is dominated by fixed per-forward work, so N=4 more than halves it, at the price of delaying each partial by up to N*80 ms. Default 1, so nothing changes unless asked for. - Fold the conv-cache update into the encoder graph, dropping a per-step async copy and a GPU sync. Measured on an M3 Air (Metal, q8_0), transcripts identical to the baseline: baseline 133.2 ms/step 1.67x default 78.2 0.98x batch_tokens=4 60.4 0.76x Sustained over a 7-minute stream those become 87.9 and 73.9 ms/step; a short clip run immediately after still measured 87.9, so the gap is the fanless machine staying warm, not per-session drift. Budget for the sustained figure. Verified: partial sequence and final transcript identical to the baseline once the baseline's duplicates are collapsed; both utterances of a file with a 3 s pause transcribe on both builds; offline output identical at 186 tokens with the decoder 52.2 -> 45.9 ms/token; a 418 s stream stays coherent across five cache wraparounds with exactly one decode-graph build; CPU backend agrees with Metal. ctest 24/27, matching upstream (gguf_tensor_source_test and scaled_dot_product_attention_test fail on main too, audio_dsp_test passes standalone). CUDA is unmeasured. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
@patrickvonplaten Thank you for the PR and I will test it later. Interestingly, I already backported the new FlashPreserveViews optimization pattern, but the improvement in Voxtral ASR was within the noise level, so I decided not to keep the change. It may need to be combined with the other changes, or it may show larger gains under different settings. |
|
By far the biggest win is doing the F16 encoder KV cache. This seems to give the biggest speed up. What's quite nice is that with the above optimizations Voxtral Streaming now runs well below 1.0 Realtime on MacBook M* series with metal. This PR is quite heavily aided by code agents, but it should touch mostly only Voxtral specific code. Happy to adapt the PR in any way / making it smaller. If you believe certain optimizations are not worth it and clutter the code too much - let's leave them out. Want to upload some q8 and q4 checkpoints on: https://huggingface.co/mistral-experimental to make it easy to use. Lemme know your thoughts |
|
I’d be happy to host the models there. I’ve already submitted a request to join Mistral Experimental. |
|
Also what's important is that EOS should not end the generation. Since voxstral Realtime should in theory run "forever" - a.k.a. as long as audio is streamed in - I think we should not add logic that makes it end with EOS but only have the following two cases:
For more reference here:
This should also be fixed in this PR |
Great approved you - thanks! |
|
I sadly currently don't have access to CUDA - so it would probably be good to test this PR on CUDA as well |
|
Yes I will test on CUDA and Vulkan. |
|
Looks like solid improvements. Merged. Thanks! Case: A = main baseline, B = PR default, C = PR opt-in batch mode.
Streaming Server
|
|
Nice to see, thanks for the benchmarking and merging! |
2 participants
Extracted from #112, which mixes two unrelated things: live stdin/PCM capture (CLI plumbing) and streaming throughput (engine). This is only the second. It touches
src/models/voxtral_realtime/andinclude/engine/models/voxtral_realtime/plus docs and one new test — noapp/changes, no new CLI flags. #112 keeps the stdin work and will need a rebase if this lands.Verification here is offline streaming transcription:
--mode streaming --audio <file>. No live source involved.Throughput
A streaming step always advances 80 ms of audio, so it has to cost under 80 ms to track a realtime source. Measured on an M3 Air (Metal, q8_0,
assets/resources/sample_16k.wav), every configuration producing the same transcript:10cb78destream_batch_tokens=4Sustained numbers are worse, and that is the number to budget against. Over
speech.wav(418 s, 5219 steps) the default measured 87.9 ms/step andstream_batch_tokens=4measured 73.9. A short clip run immediately after the long one still measured 87.9 rather than dropping back to 78.2, so this is the fanless machine staying warm, not something that resets between sessions. Practically: the default is borderline at realtime on this hardware andstream_batch_tokens=4is what clears it.Offline mode benefits too, since the attention change applies to its decode path: 52.2 → 45.9 ms/token, transcript identical at 186 tokens.
What changed
EOS no longer ends a stream.
process_one_stream_chunksetstream_reached_eos_, the consuming loop tested it, and nothing outsidereset()ever cleared it — the first EOS halted transcription permanently, and once halted the buffer-trim block became unreachable whileprocess_audio_chunkkept appending. The vLLM reference gives EOS no special treatment: the sampled token feeds back unconditionally and only the audio queue's sentinel ends the stream.tokenizer_.is_stream_text_tokenalready returns false for special ids, so no text is emitted for them and the transcript is unchanged. The offlinegenerate()EOS stop is retained. Removing utterance resets is what turns the fixed-size decode cache below from a nicety into a requirement.This is a latent bug on the files I have — EOS never fires on them — so the justification is matching the reference, not a reproduced failure.
One event per partial, without coarsening them. The session invoked its own
stream_event_sink_and returned the event thatapp/streaming/streaming.cppforwards, so every partial was delivered twice, over the CLI and the server's SSEtranscript.text.deltaalike. Worth flagging for reviewers: simply dropping the session-side sink call — the obvious fix, and the one #112 currently has — is not equivalent. Several chunks can be consumed perprocess_audio_chunkcall, so the sink is the route that yields a partial per step while the return value yields one per caller chunk; dropping the sink silently coarsened partials from 33 distinct to 25 on a 14 s clip. The returned event now drops what the sink already delivered instead, which is thenemotron_asrpattern. Measured: 59 raw events → 34, with the distinct sequence identical to the baseline's.Strided cache views instead of a materialized transpose (
text_decoder.cpp,audio_encoder.cpp) — the single largest win. The old code transposed andggml_conted the entire K and V cache per layer per token.FlashGroupedViewKVandFlashPreserveViewsalready exist in the framework and are used byfish_audio,higgs_audio_tts,vibevoice_asr,qwen3_ttsandoutetts, so this is a lowering-enum change plus dropping fourggml_contcalls.F16 encoder KV cache (
audio_encoder.cpp). The sliding-window cache is never exported, so there is noTransformerKVStateboundary to convert — just the tensor type and a relaxedset_audio_kv_rowsvalidation, sinceggml_set_rowsconverts f32 rows on write.4-bit weight types (
text_decoder.cpp,audio_encoder.cpp).normalize_weight_storagewidened to acceptq4_0/q4_k/q5_k/q6_k.BackendWeightStorealready falls back to F32 for rows that are not whole blocks (the AdaRMS 32-wide projection, the 3-wide conv kernels), so no per-tensor special-casing. Requesting one of these requantizes at load (~3 min from the shipped q8_0), so the docs point at the publishedq4_kGGUF package instead. Encoder staysnative: it is not bandwidth-bound in streaming mode.Fixed-size streaming decode cache (
text_decoder.cpp,session.cpp).voxtral_realtime.stream_decode_cache_steps, default 1024 (~82 s). Pinning bothsliding_windowandmin_cache_stepsto the cap makestarget_cache_stepsconstant, so the graph is built once atbegin_stream. The 418 s run does exactly one decode-graph build across 5219 steps and stays coherent across five ring wraparounds.Encoder batching (
frontend.cpp,session.cpp,text_decoder.cpp).voxtral_realtime.stream_batch_tokens=N, default 1. One encoder forward covers N tokens (8Nfeature frames →4Nencoder steps → N audio tokens, which falls out of the existing conv arithmetic); the decoder still steps once per token, each consuming its own row via a new explicitaudio_row. Opt-in because it delays each partial by up toN*80 ms.Conv-cache update folded into the encoder graph (
audio_encoder.cpp), dropping a per-steptensor_copy_asyncpair and a GPU sync. Worth ~0.2% of throughput — it is in for the removed sync, not the speed.Instrumentation — per-step frontend/encoder/decoder wall time plus upload/compute/readback/sample splits. This is what redirected the work: the encoder turned out to be ~98% graph compute rather than dispatch-bound, and the F16 cache gained ~10 ms/step where halved read traffic alone predicts ~2, so Metal is taking a genuinely faster F16 flash-attention path.
Tests
New hermetic
voxtral_realtime_stream_chunking_testbuilds aVoxtralRealtimeFrontendfrom a default config — no model, weights or backend — and pins the batched chunk arithmetic:steady_stream_chunk_samples(N) == N*1280 + 400, advance linear in N, defaults unchanged at1680/1280,8Nfeature frames, agreement between the fresh-STFT and cached-prefix routes, and rejection of non-positive token counts.ctest -E asr_standalone_gguf_testis 24/27.gguf_tensor_source_testandscaled_dot_product_attention_testalso fail on10cb78de(verified against a clean upstream build);audio_dsp_testpasses standalone and flakes only under parallel load.Verified
--backend cpuagrees with Metal on both builds, so the F16 cache and view-KV lowerings are not Metal-only.🤖 Generated with Claude Code