[WIP] Improve streaming

[patrickvonplaten]

Work in progress to improve streaming.

[patrickvonplaten]

Status report

Still WIP and draft. Summary of what is in the branch, what was measured, and what was
deliberately left out.

What this changes

1. Streaming input is actually streaming. run_streaming_task previously required a fully
materialized request.audio_input and sliced it into chunks, so --mode streaming only ever
replayed a file that was already in memory. This introduces an AudioChunkStream (a format plus a
pull-based reader) and routes both input paths through it — the file path is expressed as a
reader over the buffer — so the live and buffered routes cannot diverge in behaviour. The
three-argument run_streaming_task is kept as a wrapper, so app/server/runtime.cpp and every
other caller are untouched.

The CLI gains --audio -, reading raw interleaved PCM from stdin, with --input-format
(s16le/f32le), --input-rate and --input-channels. Decoding is explicitly little-endian and
reuses the WAV reader's /32768 scaling, so raw PCM and a WAV of the same audio decode to
identical samples. --audio - requires --mode streaming, and both that guard and an unsupported
--input-format fail before the model loads.

2. EOS no longer terminates a stream. process_one_stream_chunk set stream_reached_eos_ on
an EOS token, the consuming loop tested that flag, and nothing outside reset() ever cleared it —
so the first EOS halted transcription permanently for the rest of the session. Once halted, the
loop body never ran, so the buffer-trim block became unreachable while process_audio_chunk kept
appending, growing the rolling buffer without bound. EOS now closes an utterance: the text is
banked into a completed_text_ accumulator, the per-utterance decoder and audio caches restart,
and only the audio source running dry ends the loop.

Worth being precise here: this is a latent bug. I originally diagnosed it as the cause of live
capture stopping after a few seconds, and that was wrong — measured against the pre-fix binary, a
25 s file containing 3 s of silence processed 306 of ~312 expected steps and transcribed both
utterances, so EOS never fired. The real cause is throughput (below). The fix is still correct and
worth having; it just is not what fixes live capture.

3. One event per chunk. The session invoked its own stream_event_sink_ and returned the
event that the driver forwards, so every partial was delivered twice. This affected the file path
and the server's SSE transcript.text.delta stream identically. Only the returned event is used
now.

4. Terminal output. Partials carry the full transcript so far, so appending each one printed a
pyramid of growing copies. On a TTY a single line is now rewritten in place; redirected output
keeps one line per update so pipes and logs stay parseable.

Throughput: measured, not addressed

This is the main open issue and the reason live microphone capture is not usable yet.

Each steady step advances audio_length_per_tok * hop_length = 1280 samples = 80 ms of audio,
so a step must cost under 80 ms to keep up. Measured on an M3, Metal, q8_0:

	measured	per unit
Whole streaming run	131 steps / 17 896 ms	136.6 ms/step — 1.71x too slow
Text decoder	7 765 ms / 148 tokens	52.5 ms/token
Audio encoder (batched, offline)	745 ms / 187 tokens	4.0 ms/token
Frontend (STFT/mel)	2.3 ms total	negligible

Two independent bottlenecks:

• The decoder is memory-bandwidth bound and already near hardware limit. 5.1 GB of q8_0
  weights read per token against roughly 100 GB/s gives a ~51 ms/token floor, matching the measured
  52.5 ms. That alone is 66% of the budget.
• The encoder is dispatch-latency bound, not compute bound. ~84 ms/token in streaming versus
  4.0 ms/token batched — a ~20x penalty — because it runs a full 32-layer forward to produce a
  single audio token, so fixed per-kernel launch overhead dominates. Graph rebuilds are ruled
  out: audio_encoder.stream.graph_build_ms appears twice at ~1 ms and
  text_decoder.decode_cache_steps once for the entire run.

Caveat on confidence: the ~84 ms encoder figure is derived by subtraction (136.6 total minus
52.5 decoder), not measured directly — there is no per-step instrumentation in the streaming path.
Two timing_log_scalar calls in process_one_stream_chunk would settle it in one run.

The consequence for live capture: the consumer runs at ~0.6x realtime while a capture device
produces at exactly 1x, so the pipe buffer (64 KiB, about 2 s at 16 kHz mono) fills and the capture
tool blocks or drops frames. Directly observed: a feeder pacing 25 s of audio at 1x needed 47.9 s
to finish writing. The lag is also monotonic — every 80 ms of audio costs a full step whether it
contains speech or silence, so pausing does not let the model catch up.

Likely paths to realtime, neither attempted here: a q4 GGUF roughly halves the decoder (~52 ->
~27 ms/token, landing near ~112 ms/step, still short on its own), and multi-token encoder batching
attacks the dispatch overhead but requires lifting the audio_embeddings.tokens == 1 constraint in
the decoder and adds N x 80 ms of latency.

Tried and removed

A --live flag (warm the model with silence to build graphs and compile backend pipelines, then
discard audio buffered during startup) was implemented and then removed. It worked as designed —
66 Metal pipeline compilations moved off the live path, and it dropped exactly 2.048 s of stale
audio, precisely the pipe buffer — but it only removes a one-time startup offset, which is lost
again within seconds to the monotonic drift above. Not worth the surface area on app/cli/main.cpp
for the benefit. The commit was dropped rather than reverted, so it is not in this history.

Testing

tests/unittests/test_streaming_audio_input.cpp is new and hermetic (no model required): a
recording fake session proves the stdin source and the buffer source drive the session
chunk-for-chunk identically, plus chunk sizing and drain-to-EOF, little-endian decoding, trailing
partial-frame truncation, empty streams, and format validation.

End-to-end, on an M3 with Metal:

• stdin and file input produce byte-identical output — all 41 lines before the duplicate-event
  fix, 17 after, every incremental partial matching, not just the final transcript
• live behaviour confirmed under paced input: partials arrive progressively while audio is still
  being fed
• in-place TTY rendering confirmed on a real pty (16 re-renders on one line, no partial_text=
  labels, summary on a fresh row)
• a two-utterance file transcribes across a 3 s pause

Not covered by tests: the EOS-boundary and duplicate-emission fixes both live inside
VoxtralRealtimeSession, which needs the full model to instantiate, so the fake-session harness
cannot reach them. Evidence for both is the end-to-end runs above.

[patrickvonplaten]

Commands for benchmarking and verification

Everything below assumes these two, adjusted for your machine and backend
(--backend cuda / --backend cpu substitute cleanly for --backend metal):

BIN=./build/macos-metal-release/bin/audiocpp_cli
MODEL=path/to/voxtral-mini-4b-realtime-2602-q8_0.gguf

Build

scripts/build_metal.sh --with-tests --target audiocpp_cli

The unit test needs ENGINE_BUILD_TESTS=ON; if the cache was configured without it, the test
targets silently do not exist:

cmake -S . -B build/macos-metal-release -DENGINE_BUILD_TESTS=ON
cmake --build build/macos-metal-release --target audiocpp_cli streaming_audio_input_test

Tests

# Hermetic, no model required, ~2 s
./build/macos-metal-release/bin/streaming_audio_input_test

# Full suite. asr_standalone_gguf_test does not compile on main and is excluded.
cd build/macos-metal-release && ctest -E "asr_standalone_gguf_test"

Expect 25/27. gguf_tensor_source_test and scaled_dot_product_attention_test fail on main too.
audio_dsp_test and supertonic_vector_convnext_exp_test flake under parallel load — re-run them
individually before treating either as a regression:

ctest -R "supertonic_vector_convnext_exp_test"

The correctness bar: stdin must equal file

This is the check that matters most for this PR. Streaming from a file and streaming the same audio
as raw PCM over stdin must produce byte-identical output — every incremental partial, not just
the final transcript.

$BIN --task asr --family voxtral_realtime --model $MODEL --backend metal \
  --mode streaming --audio speech.wav 2>/dev/null \
  | grep -E '^(partial_text|text_output)=' > /tmp/from_file.txt

python3 -c "
import sys, wave
w = wave.open('speech.wav')
sys.stdout.buffer.write(w.readframes(w.getnframes()))
" | $BIN --task asr --family voxtral_realtime --model $MODEL --backend metal \
      --mode streaming --audio - 2>/dev/null \
  | grep -E '^(partial_text|text_output)=' > /tmp/from_stdin.txt

diff /tmp/from_file.txt /tmp/from_stdin.txt && echo IDENTICAL

speech.wav must be 16 kHz mono s16 for the raw-PCM side to match the defaults; otherwise pass
--input-rate / --input-channels / --input-format to match your file.

Throughput: is it realtime?

The single number that matters. Each steady step advances
audio_length_per_tok * hop_length / sample_rate = 8 * 160 / 16000 = 80 ms of audio, so a
step must cost under 80 ms to keep up.

$BIN --task asr --family voxtral_realtime --model $MODEL --backend metal \
  --mode streaming --audio speech.wav --log 2>&1 \
  | grep -E "stream\.(steps_processed|wall_ms)"

voxtral_realtime.session.stream.steps_processed 131
voxtral_realtime.session.stream.wall_ms 17895.927

wall_ms / steps_processed is your ms/step; divide by 80 for the realtime factor. The run above is
136.6 ms/step = 1.71x too slow. Under 1.0x means live capture can keep up.

Where the time goes

The streaming path has no per-step breakdown, so run the same audio offline to attribute cost
between the encoder and the decoder:

$BIN --task asr --family voxtral_realtime --model $MODEL --backend metal \
  --audio speech.wav --log 2>&1 \
  | grep -E "session\.(frontend_ms|audio_encoder_ms)|text_decoder\.(step_total_ms|generated_tokens)|session\.audio_tokens"

voxtral_realtime.session.frontend_ms 2.281959
voxtral_realtime.session.audio_encoder_ms 745.419458      # / audio_tokens 187 = 4.0 ms/token
voxtral_realtime.text_decoder.step_total_ms 7764.752414   # / generated_tokens 148 = 52.5 ms/token

The decoder figure is directly comparable to the streaming path. The encoder figure is batched,
so comparing it against ms/step - decoder_ms exposes the per-dispatch penalty of producing one
audio token per forward pass (~4.0 vs ~84 ms/token here).

Also worth checking that graph rebuilds are not polluting a measurement — these should appear once
or twice for a whole run, not per step:

... --log 2>&1 | grep -cE "graph_build_ms|decode_cache_steps"

Live behaviour

Feed at 1x wall-clock and timestamp each partial. Partials should appear progressively while audio
is still being written, not all at the end:

python3 -c "
import sys, time, wave
w = wave.open('speech.wav')
t0 = time.time()
while True:
    f = w.readframes(1600)          # 100 ms at 16 kHz
    if not f: break
    sys.stdout.buffer.write(f); sys.stdout.buffer.flush()
    time.sleep(0.1)
sys.stderr.write(f'[feeder] finished writing at {time.time()-t0:.1f}s\n')
" | $BIN --task asr --family voxtral_realtime --model $MODEL --backend metal \
      --mode streaming --audio - 2>/dev/null \
  | python3 -c "
import sys, time
t0 = time.time()
for l in sys.stdin:
    if l.startswith(('partial_text=', 'text_output=')):
        print(f'[{time.time()-t0:6.2f}s] {l.rstrip()[:78]}')
"

The [feeder] finished writing line is the backpressure detector: for 25 s of audio it should
report ~25 s. It reported 47.9 s here, meaning writes blocked because the consumer runs slower
than realtime.

Multi-utterance / EOS boundary

Builds a file with a pause in the middle, which exercises the EOS-as-utterance-boundary path.
Both utterances must appear in the final transcript:

python3 -c "
import wave
src = wave.open('speech.wav'); n, sr = src.getnframes(), src.getframerate()
pcm = src.readframes(n)
out = wave.open('/tmp/two_utterances.wav','wb')
out.setnchannels(1); out.setsampwidth(2); out.setframerate(sr)
out.writeframes(pcm + b'\x00\x00' * (sr * 3) + pcm)
out.close()
"
$BIN --task asr --family voxtral_realtime --model $MODEL --backend metal \
  --mode streaming --audio /tmp/two_utterances.wav 2>/dev/null | tail -1

Step count is a useful cross-check: a 25 s file should report ~312 steps (25 s / 80 ms). A much
lower count means the stream stopped early.

Real microphone

ffmpeg -f avfoundation -i ":0" -t 30 -ar 16000 -ac 1 -f s16le -loglevel warning - 2>ffmpeg.log \
 | $BIN --task asr --family voxtral_realtime --model $MODEL --backend metal --mode streaming --audio -

Use -t rather than Ctrl-C so ffmpeg exits cleanly, stdin reaches EOF, and the CLI finalizes and
prints text_output=. Keep -loglevel warning, not error — buffer-full/frame-dropped warnings in
ffmpeg.log are how you tell whether audio was lost at the capture layer because the consumer
could not keep up, versus merely delivered late. On Linux substitute -f alsa, on Windows
-f dshow.

Terminal rendering

Partials rewrite one line in place on a TTY and print one line per update when redirected. script
does not reliably allocate a pty in every environment; this checks it directly:

python3 -c "
import os
pid, fd = os.forkpty()
if pid == 0:
    os.execv('$BIN', ['audiocpp_cli','--task','asr','--family','voxtral_realtime',
      '--model','$MODEL','--backend','metal','--mode','streaming','--audio','speech.wav'])
data = b''
while True:
    try: d = os.read(fd, 65536)
    except OSError: break
    if not d: break
    data += d
os.waitpid(pid, 0)
print('carriage returns:', data.count(b'\r'), '| partial_text= labels:', data.count(b'partial_text='))
"

Expect many carriage returns and zero partial_text= labels on a pty; the reverse when piped.

[patrickvonplaten]

Implementation plan: get streaming to realtime on M3 (hand-off)

This comment is a self-contained implementation plan for whoever picks this branch up next.
It is based on a full read of the streaming path plus a verification pass against the
reference vLLM implementation, and it supersedes one claim in the status report above: the
EOS-as-utterance-boundary behavior this PR introduced is wrong and should be removed
(details in Phase 0). Everything else in the status report stands.

What the investigation found

Architecture (Kyutai-style delayed streams; one decoder step per 80 ms, always):

• Frontend: 16 kHz → STFT(400/160) → 128-mel. One token = 1280 samples = 80 ms. ~2.3 ms
  total per run — negligible.
• Audio encoder (src/models/voxtral_realtime/audio_encoder.cpp): conv(s1)+conv(s2), then
  32 transformer layers, d=1280, MHA 32 heads×64, MLP 5120, sliding window 750 encoder steps
  (~0.87 B params ≈ 0.89 GB q8_0). With downsample_factor 4, the streaming graph runs the
  full 32-layer stack over just 4 encoder steps to emit one 3072-d audio token per 80 ms.
• Text decoder (text_decoder.cpp): 26 layers, d=3072, GQA 32q/8kv heads×128, MLP 9216,
  vocab 131072 with tied lm_head (~3.4 B params ≈ 3.6 GB q8_0), AdaRMS time-conditioning on
  num_delay_tokens=6 (~480 ms lookahead). The audio embedding is added to the previous
  token's embedding each step. Steady state runs a 1-token DecodeStepGraph against a
  bounded static f32 KV cache that grows in power-of-2 buckets (256→…→8192), each growth
  being a full graph rebuild + KV export/import through CPU.

Cost model (M3 ≈ 90–100 GB/s shared bandwidth; measured 136.6 ms/step = 52.5 decoder

• ~84 everything else):

• Decoder, 52.5 ms/token — bandwidth-bound: ~3.6 GB of q8_0 weights read per token
  ≈ 40 ms floor. Two hidden extras that grow over time:
  (a) build_text_attention_static (text_decoder.cpp:392–397) transposes and ggml_conts
  the entire K and V cache every layer, every token — ~2.5 ms at cache 512, but
  ~3.5 GB/token of pure copy traffic at the full 8192 window;
  (b) every cache-growth bucket rebuilds the graph and round-trips all KV state through CPU
  f32 — a >80 ms stall in the middle of a live stream.
• Encoder path, ~84 ms/step (derived by subtraction — instrument before trusting the
  split): ~0.89 GB weight reads (~10 ms floor) + transpose+ggml_cont of the full 750-step
  f32 K/V caches every layer (audio_encoder.cpp:284–287; ~1.6 GB/step of copies) + f32
  flash-attn reads (~0.4 GB) + per-node dispatch overhead across ~900 graph nodes + 3 GPU
  syncs per step (encoder compute, conv-cache copy_async, decoder).
• Verified in the vendored ggml: Metal flash attention supports f32 K/V (no CPU fallback;
  FLASH_ATTN_EXT case in ggml-metal-device.m), and Metal SET_ROWS supports f32→f16
  destinations. The repo already ships the no-copy attention pattern:
  flash_attention_from_grouped_heads_view_kv
  (src/framework/modules/transformers/qwen_decoder.cpp:173), used on Metal by
  higgs_audio_tts/outetts.
• Structural constraints: the 80 ms cadence is fixed by the model — silence costs the same
  as speech, and there is no speculative shortcut for a 1-token-per-chunk stream. The
  fanless M3 Air throttles under sustained load, so target ≤60 ms/step, not 79.

Phase 0 — correctness: remove the EOS special-casing (match the vLLM reference)

The reference implementation gives EOS no special treatment whatsoever; the stream ends
only when audio input ends:

• vllm/model_executor/models/voxtral_realtime.py —
  VoxtralRealtimeBuffer.get_input_stream() terminates only on the audio-queue None
  sentinel; feed_tokens feeds all_outputs[-1:] (whatever token was sampled, EOS/pad
  included) back as the next text-stream token, unconditionally.
• vllm/entrypoints/speech_to_text/realtime/connection.py — _run_generation uses
  SamplingParams(temperature=0.0, max_tokens=realtime_max_tokens) with
  realtime_max_tokens = 1 (interfaces.py): exactly one token per 80 ms segment, no EOS
  filtering in the feedback loop; special tokens just produce empty text deltas.

Change in session.cpp: delete the text_decoder_.is_eos(token) →
begin_new_stream_segment() branch in process_one_stream_chunk, delete
begin_new_stream_segment() and completed_text_, and decode straight through EOS — the
token feeds back as previous_stream_token_ like any other, and
tokenizer_.is_stream_text_token() already returns false for special ids (EOS=2,
STREAMING_PAD=32) so no text is emitted for them. The loop's only exit remains "audio source
ran dry". Keep the EOS stop in the offline generate() path (offline mode legitimately
stops at EOS). Re-verify: the two-utterance file (3 s pause) must transcribe both utterances
via continuous decoding, and the stdin-vs-file byte-identical check must still pass.

Consequence: with no utterance resets, decoder KV state is continuous for the whole session
— which upgrades Phase 2 item 5 (fixed-size cache) from nice-to-have to required.

Phase 1 — instrument the streaming path (settle the ~84 ms attribution)

Accumulate per-step frontend / encoder / decoder wall-time in
VoxtralRealtimeSession::process_one_stream_chunk, emit sums + counts in finalize()
(same style as stream.steps_processed); add upload/compute/readback timers inside
StreamingAudioGraph::run and stream_step. Re-baseline with the benchmark commands in the
comment above. Gate every later phase on these numbers.

Phase 2 — low-risk throughput wins (est. ~136 → ~75–90 ms/step)

1. 4-bit decoder weights (biggest single lever; ~52 → ~25–30 ms/token): widen
   normalize_weight_storage in text_decoder.cpp and audio_encoder.cpp (currently
   native/f32/f16/bf16/q8_0 only) to also accept Q4_0/Q4_K/Q5_K/Q6_K. The framework
   already supports these end-to-end — TensorStorageType, option parsing, and on-load
   requantization in tensor_source.cpp — so
   --opt voxtral_realtime.text_decoder_weight_type=q4_k then transcodes the existing q8_0
   GGUF at load. Gate on transcript quality vs q8_0; if requant-from-q8_0 degrades, convert
   a proper q4_K GGUF from the HF bf16 checkpoint with audiocpp_gguf (docs/gguf.md). Keep
   the encoder at q8_0 initially; try q6_k for it later.
2. Drop the full-cache transpose+cont in both static attention builders: pass strided
   views of the cache directly to ggml_flash_attn_ext, copying the
   flash_attention_from_grouped_heads_view_kv pattern (qwen_decoder.cpp:173 — ggml_cont
   on q only). Sites: build_text_attention_static (text_decoder.cpp:392–397) and
   build_audio_attention_streaming_static (audio_encoder.cpp:284–287). Removes
   ~1.6 GB/step of encoder copy traffic and makes decoder cost independent of cache fill.
3. F16 KV caches in StreamingAudioCache and DecodeStepGraph: Metal SET_ROWS
   writes f32→f16 natively and flash attention prefers f16 K/V. Halves the remaining cache
   traffic. Convert at the TransformerKVState (f32 vectors) import/export boundary.
4. Fold the conv-cache update into the encoder graph (in-graph ggml_cpy into the
   cache tensors, as qwen_decoder.cpp:722 does) — drops the
   ggml_backend_tensor_copy_async + second ggml_backend_synchronize per step.
5. Fixed-size streaming decode cache (required after Phase 0): new option
   voxtral_realtime.stream_decode_cache_steps (default 1024 ≈ 82 s effective window; the
   ring cache_slot = absolute_end % cache_steps logic in
   bounded_static_kv_decode.cpp already handles wraparound). Pre-build once at
   begin_stream → no growth-rebuild stalls mid-stream, bounded flash-attn cost. Tradeoff:
   effective attention window shrinks from 8192 to the cap.

Phase 3 — encoder batching option (amortize dispatch; target ≤60 ms/step)

--opt voxtral_realtime.stream_batch_tokens=N (default 1 = today's behavior and latency).
The session accumulates N×80 ms, runs one encoder forward with feature_frames = 8N
(graph node count is constant, so per-token dispatch cost divides by N), then N sequential
stream_step calls, each fed its own embedding row (replace the
audio_embeddings.tokens == 1 guard with per-call row selection). Adds N×80 ms latency,
hence opt-in; N=2–4 recommended on M3 Air. Phase 0 removed EOS resets, so there is no
mid-batch reset edge case.

Phase 4 — only if still short after measuring Phases 2–3

• Pipeline encoder ‖ decoder on two Metal backend instances/command queues (the
  components already have separate weight stores): step cost → max(encoder, decoder)
  instead of their sum.
• VAD-gated catch-up (src/models/silero_vad exists in-repo): when behind wall-clock,
  drop silent chunks so lag recovers during pauses. Changes output semantics — strictly
  opt-in.

Verification (run per phase)

• Throughput: --mode streaming --audio speech.wav --log, read
  stream.wall_ms / stream.steps_processed; must trend to <80 ms/step, target ≤60.
  Build: scripts/build_metal.sh --with-tests --target audiocpp_cli.
• Correctness: stdin-vs-file byte-identical partial_text= diff (commands in the
  benchmark comment above); two-utterance pause file; ctest -E asr_standalone_gguf_test
  stays at the 25/27 baseline; streaming_audio_input_test green.
• Quality (quantization): q8_0 vs q4_k transcripts on the test wavs must match or
  differ trivially; otherwise escalate to q5_k/q6_k or convert from bf16.
• Live: paced-feeder backpressure test (feeder must finish in ~audio duration), a real
  ffmpeg -f avfoundation mic run, and a ≥5-minute run to observe fanless-throttling
  behavior and confirm the absence of cache-growth stalls.

[patrickvonplaten]

Streaming is now faster than realtime: 138.9 → 39.6 ms/step

All of this landed in a single commit, 01ce5c8 ("Make voxtral_realtime streaming faster than realtime on Metal"), which implements the hand-off plan above. Every number below is measured on an M3 (Metal, 25 s clip) and every configuration produced a byte-identical transcript to the q8_0 baseline.

Config	ms/step	Realtime	Encoder	Decoder
Baseline (0b49f04)	138.9	1.74x ✗	80.6	55.9
New default	78.5	0.98x	29.9	48.5
stream_batch_tokens=4	61.4	0.77x	12.8	48.5
+ text_decoder_weight_type=q4_k	39.6	0.49x	12.8	26.8

The practical outcome: a paced feeder writes 25 s of audio in 33.9 s (was 47.9 s), and session lag is now flat at the one-time model load (~9 s) instead of growing monotonically.

What changed, in 01ce5c8

Phase 0 — remove the EOS utterance boundary (session.cpp). The vLLM reference gives EOS no special treatment: the sampled token feeds back unconditionally and only the audio queue's sentinel ends the stream. Deleted the is_eos → begin_new_stream_segment() branch along with completed_text_. The offline generate() EOS stop is retained. The two-utterance file now transcribes both utterances through continuous decoding.

Phase 1 — instrument first (all four files). Per-step frontend/encoder/decoder wall time plus upload/compute/readback/sample splits. This paid for itself immediately and redirected the rest of the work — see "corrections" below.

Phase 2.2 — strided cache views instead of a materialized transpose (text_decoder.cpp, audio_encoder.cpp). The single biggest win: 138.9 → 91.3 ms/step. FlashGroupedViewKV and FlashPreserveViews already existed in the framework, so this was a lowering-enum change plus dropping four ggml_cont calls, not new plumbing.

Phase 2.3 — F16 encoder KV cache (audio_encoder.cpp). Encoder 42 → 31.5 ms/step. Simpler than the plan assumed: the encoder's sliding-window cache is never exported, so there is no TransformerKVState boundary to convert — just the tensor type and a relaxed set_audio_kv_rows validation, since ggml_set_rows converts f32 rows on write.

Phase 2.1 — 4-bit decoder weights (text_decoder.cpp, audio_encoder.cpp). Widened normalize_weight_storage to accept q4_0/q4_k/q5_k/q6_k. BackendWeightStore already falls back to F32 for rows that are not whole blocks (the AdaRMS 32-wide projection, the 3-wide conv kernels), so no per-tensor special-casing was needed. Decoder 49 → 27 ms/step, transcript-identical.

Phase 2.5 — fixed-size streaming decode cache (text_decoder.cpp, session.cpp). New voxtral_realtime.stream_decode_cache_steps (default 1024 ≈ 82 s). Pinning both sliding_window and min_cache_steps to the cap makes target_cache_steps constant, so the graph is built once at begin_stream. Required now that Phase 0 removed utterance resets. A 7-minute run does exactly one decode-graph build across 5217 steps, and the slot ring wraps correctly past 1024.

Phase 3 — encoder batching (frontend.cpp, session.cpp, text_decoder.cpp). New voxtral_realtime.stream_batch_tokens=N (default 1). One encoder forward covers N tokens (8N feature frames → 4N encoder steps → N audio tokens, which falls out of the existing conv arithmetic); the decoder still steps once per token, each consuming its own embedding row via a new explicit audio_row. N=4 takes the encoder 30 → 13 ms/step, delaying each partial by up to N*80 ms, hence opt-in.

Phase 2.4 — conv-cache update folded into the encoder graph (audio_encoder.cpp). Drops a per-step tensor_copy_async pair and a GPU sync. The encoder body is expanded before the ggml_cpy nodes so the backend's node ordering keeps the write from clobbering the current chunk's own left context.

Corrections to the plan

Three of the plan's cost estimates were wrong, and the Phase 1 instrumentation is what caught them:

• The encoder was 98% pure graph compute, not per-dispatch overhead plus GPU syncs. Upload, conv-copy and readback together were 0.4 ms/step, so Phase 2.4 was worth ~0.2% rather than being a meaningful lever. It is still in, but for the removed sync, not throughput.
• F16 KV caches were the surprise winner, not the modest Phase 2 item they looked like. Halved read traffic alone predicts ~2 ms; the measured gain was ~10 ms/step, so Metal is taking a genuinely faster F16 flash-attention path.
• Quantizing the encoder makes it slower (37.9 → 45.0 ms/step). It is not bandwidth-bound in streaming mode, so dequantization cost dominates. Left at native, and documented as such.

Also worth flagging for anyone following the benchmarking comment: the session-option flag is --session-option, not --opt. An unrecognized --opt is silently ignored, which produced one misleading "q4_k barely helps" measurement before I noticed.

The one caveat

q4_k costs ~174 s of startup. Requantizing the shipped q8_0 GGUF means dequantizing to F32 and requantizing ~3.4 B parameters single-threaded on the CPU, before the first partial appears — during which a live producer blocks. Steady state is genuinely 0.49x, so it is right for files and long-lived servers, but live capture wants a pre-converted q4_K GGUF.

I could not produce one here: audiocpp_gguf discovers sidecars from disk, and this GGUF has them embedded, so GGUF→GGUF conversion fails on missing config.json/tekken.json. Converting from the HF bf16 checkpoint (as the plan suggested) needs a download I did not start. That is the main follow-up — with it, 0.49x becomes available without the startup penalty.

Verification

• Throughput/thermal: 7-minute sustained run holds 65.3 ms/step with one decode-graph build; a 25 s burst immediately afterwards measured 61.8 ms/step, so the gap is sustained-load behaviour, not thermal drift. stream_decode_cache_steps 256 vs 1024 measured 65.7 vs 65.3 over that file — the knob does not meaningfully trade throughput for context, so the default keeps the longer window.
• Correctness: stdin-vs-file byte-identical for both default and batched configs; two-utterance pause file transcribes both utterances; long-run transcript stays coherent well past the 82 s ring wraparound.
• Tests: ctest -E asr_standalone_gguf_test at 25/27, matching the documented baseline (gguf_tensor_source_test and scaled_dot_product_attention_test fail on main; audio_dsp_test flaked once under parallel load and passes standalone). streaming_audio_input_test green. Offline mode unaffected at 48.4 ms/token.

docs/asr.md is updated: the stale 137 ms throughput note now carries the measured figures, a streaming session-options table, and the requantization warning.

[patrickvonplaten]

Command to test:

ffmpeg -f avfoundation -i ":0" -ar 16000 -ac 1 -f s16le -loglevel warning - 2>/tmp/ffmpeg.log \
  | /Users/patrickprivate/audio.cpp/build/macos-metal-release/bin/audiocpp_cli \
      --task asr --family voxtral_realtime \
      --model /Users/patrickprivate/models/Voxtral-Mini-4B-Realtime-2602-GGUF/voxtral-mini-4b-realtime-2602-q8_0.gguf \
      --backend metal --mode streaming --audio - \
      --session-option voxtral_realtime.stream_batch_tokens=4

[patrickvonplaten]

Rebased onto main (7f3a73c).

The performance work that made up the tail of this branch landed separately as #116, so this PR is now a single commit: the live stdin input.

What the rebase dropped

• Make voxtral_realtime streaming faster than realtime on Metal — that commit is Speed up voxtral_realtime streaming and stop EOS from ending a stream #116. Every file it touched (audio_encoder.{h,cpp}, frontend.{h,cpp}, text_decoder.{h,cpp}) is now byte-identical to main, so it replayed as empty.
• Note that streaming lag does not recover during pauses — the note reached main through Speed up voxtral_realtime streaming and stop EOS from ending a stream #116's throughput section, word for word.
• This branch's EOS handling in session.{h,cpp}. It banked each utterance into completed_text_ and restarted the per-utterance frontend/encoder state on EOS. Speed up voxtral_realtime streaming and stop EOS from ending a stream #116 replaced that with the reference behaviour — EOS gets no special treatment, the sampled token feeds back unconditionally, and only the audio source running dry ends the stream — which is also what makes the fixed-size decode cache a requirement rather than a nicety. main's version is kept unchanged; session.cpp and session.h now carry no changes from this PR at all.
• This branch's "remove the event sink" dedupe, in favour of main's sink call plus partial_text.reset(). Same deduplication, but main emits one partial per step rather than one per process_audio_chunk call.
• The stale throughput note in docs/asr.md ("137 ms/step, roughly 1.7x too slow"), which Speed up voxtral_realtime streaming and stop EOS from ending a stream #116's optimizations obsoleted. main's measured table stays, with the stdin section above it.

What is left

11 files, +665/−49, entirely app-layer and docs:

• AudioChunkStream in app/streaming/ — a format declared up front plus a pull-based reader. Both input routes go through it, with the file path expressed as a reader over the buffer, so live and buffered input cannot diverge.
• --audio - in the CLI, with --input-format / --input-rate / --input-channels for the format a live stream carries no header for.
• In-place transcript rendering on a TTY; one flushed line per update when redirected.
• docs/asr.md, docs/usage.md, and streaming_audio_input_test.

Verification (M3 Air, Metal, q8_0)

• assets/resources/sample_16k.wav through the file path and through ffmpeg … -f s16le - | produce byte-identical output apart from the audio_input=stdin banner — 34 partials either way.
• Partial delivery matches a CLI built from main exactly: 34 partials, growing one token at a time, and the same single trailing duplicate at finalize (pre-existing on main, not introduced here).
• ctest 29/32. gguf_tensor_source_test and scaled_dot_product_attention_test fail on main too, and asr_standalone_gguf_test does not compile there — tests/unittests/test_asr_standalone_gguf.cpp calls load_hviske_assets while the header declares load_hviske_asr_assets. The three tests/moss_tts_local/codec_*_parity.cpp targets are likewise broken on main. None of these are touched by this branch.

Still WIP

Throughput against a live source is the open question. With #116's optimizations a step costs ~78 ms cool and ~88 ms sustained against an 80 ms budget, so an open-ended capture still backs up on this machine; stream_batch_tokens=4 brings it to ~74 ms sustained at the price of delaying each partial. CUDA is unmeasured.

🤖 Generated with Claude Code

[patrickvonplaten]

Carved the stdin streaming input out into #118, which is reviewable on its own.

This PR was doing two unrelated things. #118 takes the first — AudioChunkStream, the raw PCM reader, and the --audio - surface. What stays here is the second: in-place terminal rendering of partials. That change is orthogonal to where the audio comes from — partials each carry the full transcript, so the "pyramid" it fixes is already visible on main today with --audio file.wav --mode streaming.

Keeping it separate also keeps this out of #118:

#define isatty _isatty
#define fileno _fileno

Redefining libc names as macros at file scope in app/cli/main.cpp, a 951-line translation unit every CLI task flows through, deserves its own review rather than riding along with a feature.

One piece did move to #118 rather than stay here: flushing the partial_text= line. That is functional, not cosmetic — without it a piped consumer sees nothing until the stdio buffer fills, which defeats the point of a live source. Only the \r rewriting remains a rendering concern.

I'll rebase this branch on #118 once that lands, leaving just the rendering change.