ASR Kroko community

[mirek190]

Summary

Adds native support for the public free Kroko Community ASR packages.

• native Kaldi-compatible filterbank, Conv2dSubsampling/ConvNeXt frontend, 19-layer Zipformer2, stateless RNN-T predictor/joiner, and greedy decoder
• offline and stateful streaming inference with partial transcripts
• word timestamps derived from RNN-T emission frames
• dynamic 64-L (141/128) and 128-L (269/256) package layouts
• German, English, Spanish, French, Italian, Hebrew, Dutch, Portuguese, Swedish, and Turkish packages
• conversion from free Kroko .data packages to the audio.cpp safetensors layout
• movable standalone Q8 GGUF with embedded config, tokens, and package spec
• current split model_specs/ runtime layout and model_specs_v1/ catalog metadata
• CLI, server, model-manager, probes, parity tooling, documentation, and validation report

Kroko packages are single-language. The original .data package is needed only to create the safetensors/GGUF model. Running a standalone GGUF does not require the .data file or external sidecars.

Build

Tested on Windows 11, Ryzen 9 7950X3D, RTX 3090 24 GiB, CUDA 12.4, MSVC 2022, and CMake 3.29.2.

$cuda = "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4"
$env:CUDA_PATH = $cuda
$env:CUDAToolkit_ROOT = $cuda
$env:CUDACXX = Join-Path $cuda "bin\nvcc.exe"
$env:NVCC_PREPEND_FLAGS = "-allow-unsupported-compiler"

.\scripts\build_windows.ps1 -Preset windows-cuda-release -ConfigureOnly
cmake -S . -B build\windows-cuda-release -DENGINE_BUILD_WARMBENCH=ON -DCMAKE_CUDA_COMPILER="$env:CUDACXX"
cmake --build build\windows-cuda-release --config Release --target audiocpp_cli audiocpp_server audiocpp_gguf kroko_asr_streaming_probe kroko_asr_encoder_probe -j 8

Conversion and run

Converter/model-manager package id: kroko_asr_community_converted.

.\venv\Scripts\python.exe tools\model_manager.py install `
  kroko_asr_community_converted `
  --source-file ..\models\Kroko-ASR\Kroko-SV-Community-64-L-Streaming-001.data `
  --models-root ..\models\Kroko-ASR --overwrite

build\windows-cuda-release\bin\audiocpp_gguf.exe `
  --input ..\models\Kroko-ASR\Kroko-SV-Community-64-L-Native\model.safetensors `
  --root ..\models\Kroko-ASR\Kroko-SV-Community-64-L-Native `
  --family kroko_asr --type q8_0 --overwrite `
  --output ..\models\Kroko-ASR\Kroko-SV-Community-64-L-Native\Kroko-SV-Community-64-L-Q8.gguf

build\windows-cuda-release\bin\audiocpp_cli.exe `
  --task asr --mode streaming --family kroko_asr `
  --model ..\models\Kroko-ASR\Kroko-SV-Community-64-L-Native\Kroko-SV-Community-64-L-Q8.gguf `
  --backend cuda --audio ..\outputs\kroko_multilingual_samples\sv.wav `
  --language sv --text-out ..\outputs\kroko_sv_q8_cuda.txt `
  --words-out ..\outputs\kroko_sv_q8_cuda_words.json --log

Original implementation and GGUF comparison

The original comparison uses sherpa-onnx 1.13.4 with the source Kroko ONNX graph and greedy RNN-T decoder.

Across ten public 64-L language packages:

• all ten audio.cpp transcripts exactly match the original implementation
• reference WER is 0 for every request
• word counts match for every request
• worst word-start timestamp difference is 0.305 us
• native/ONNX encoder-boundary cosine is 0.99957-0.99988
• native audio.cpp CPU RTF is 0.0875-0.0972 (about 10-11x real time)
• optimized ONNX Runtime CPU RTF is 0.0182-0.0222, 4.3-4.7x faster for these short requests

Same 4.968-second Swedish request:

Path	Backend	Wall time	RTF	Quality result
Original Kroko ONNX	CPU	107.267 ms	0.0216	Reference
audio.cpp native safetensors	CPU	455.093 ms	0.0916	Exact transcript; 0 reference WER
audio.cpp standalone Q8 GGUF	CUDA	462.556 ms	0.0931	Transcript and word JSON byte-identical to native safetensors

The 167,754,176-byte Q8 GGUF embeds config.json, tokens.txt, and its package spec. It runs directly from a random file path without the source .data package or sidecars.

Streaming, server, and memory

A 49.680-second Swedish sample produced byte-identical offline and streaming transcript/timestamp artifacts:

Mode	Session	RTF	Encoder chunks	Peak retained waveform
Streaming	3981.768 ms	0.0801	40	38,560 samples
Offline	3936.170 ms	0.0792	40	n/a

The stream accepted 794,880 samples while retaining only 38,560 values at peak.

Two Q8 CUDA requests through the long-lived server returned the exact reference transcript:

Request	Wall	TTFT
1	579.880 ms	189.824 ms
2	491.622 ms	111.753 ms

After request 2 the server used 619.5 MiB RSS and 1478.1 MiB private memory. Reliable per-process CUDA VRAM is unavailable through Windows WDDM.

Validation

.\venv\Scripts\python.exe webui\test_catalog_sync.py
.\venv\Scripts\python.exe tools\check_loader_catalog_sync.py --self-test
.\venv\Scripts\python.exe tools\check_loader_catalog_sync.py

Results:

webui catalog: 6 tests passed
loader self-test: 2 tests passed
active_loaders=37 commented_loaders=0 catalog_packages=48
ok: installable catalog families match registered loaders

Backends tested:

• CPU: all ten languages, ONNX parity, timestamps, 64-L/128-L probes, offline/streaming long-audio equality
• CUDA: standalone Q8 CLI streaming and two-request server session
• Vulkan: not tested
• Metal: not tested

The complete exact commands, per-request language table, generated artifact paths, hashes, boundary metrics, server configuration, timing, RSS, and known limitations are in docs/reports/kroko_asr_validation.md.

GGUF models to test

https://huggingface.co/mirek190/audio.cpp/tree/main/Audio%20to%20text%20(ASR)/Kroko

[0xShug0]

You're on fire 🔥!

[mirek190]

Added commit 19feb21 (Optimize Kroko ASR inference).

What changed

• Compute the 512-value joiner tanh activation once per encoder frame instead of once for every vocabulary row.
• Evaluate independent vocabulary projections in parallel while preserving each row's accumulation order and the serial argmax.
• Batch graph input, recurrent-state, and output transfers asynchronously at graph boundaries.
• Reuse the padded feature-chunk scratch buffer.
• Decode directly from the valid encoder-output prefix.

I also tested keeping recurrent state fully device-resident. That version was not retained because it shifted several token emissions by one 40 ms frame. The committed host-state path preserves exact transcript and timestamp parity.

Performance

Test system: Ryzen 9 7950X3D and RTX 3090.

• Ten-language CPU matrix: 1.804?1.957x faster, arithmetic mean 1.881x, with 0.0448?0.0531 RTF.
• Seven standalone Q8 CUDA CLI requests: 223.328?244.833 ms, median 232.341 ms, 0.0468 RTF.
• Compared with the earlier 462.556 ms Q8 CUDA validation result: 1.991x faster.
• Long 49.68-second input:
  • streaming CPU: 3981.768 -> 2051.994 ms
  • offline CPU: 3936.170 -> 1984.566 ms
• Warm server request: 491.622 -> 138.293 ms wall time, with 125.561 ms measured inside the session.

Parity and validation

• All ten languages still exactly match sherpa-onnx greedy transcripts: WER 0.
• Worst word-start timestamp delta remains 0.305 microseconds.
• Seven Q8 outputs, the 128-L English package, and long-audio offline/streaming text and word JSON remain byte-identical.
• Windows CPU and CUDA builds pass.
• 64-L/128-L boundary probes, WebUI catalog tests, and loader/catalog synchronization checks pass.

The full commands and per-request results are recorded in docs/reports/kroko_asr_validation.md.

[mirek190]

Implemented the remaining Kroko decoding and streaming work in commit f1fe367.

What changed

• Added modified RNN-T beam search with 1-64 active paths while keeping greedy search as the backward-compatible default.
• Added blank-penalty support.
• Added natural-text hotwords with slash/newline phrase separation and configurable per-token context score. The tokenizer works directly from the packaged token table, so the public Kroko package does not need an external SentencePiece model.
• Added the upstream three-rule endpoint detector and speech-segment output.
• Endpoint resets preserve Zipformer/subsampling state and each hypothesis's final two predictor tokens, while resetting the output segment and hotword graph, matching the sherpa-onnx high-level reset behavior.
• Reworked non-16-kHz streaming input to use bounded incremental mono resampling instead of retaining the complete source waveform.
• Added a bounded predictor-output cache for modified beam search.
• Exposed all controls through loader metadata, model_specs_v1, CLI request options, JSON server requests, and the catalog-driven UI surface.
• Extended the sherpa-onnx reference runner and updated the Kroko usage and validation documentation.

Request options

• decoding_method=greedy_search|modified_beam_search
• max_active_paths=1..64
• blank_penalty=
• hotwords=phrase/phrase
• hotwords_score=
• enable_endpoint=true|false
• rule1_min_trailing_silence
• rule2_min_trailing_silence
• rule3_min_utterance_length

The kroko_asr.* namespaced aliases are also accepted. Hotwords require modified beam search.

Validation

• Windows CUDA release build passed for audiocpp_cli, audiocpp_server, audiocpp_gguf, and all four Kroko probes.
• Loader/catalog self-tests passed: 37 active loaders and 48 catalog packages.
• The complete ten-language CPU matrix remains exact against sherpa-onnx greedy search: identical transcripts, zero reference WER, matching word counts, and only sub-microsecond float representation differences on the same 40 ms timestamp grid.
• max_active_paths=1 is byte-identical to greedy output, including word JSON.
• Four-path modified beam search on EN_3 matches the sherpa-onnx transcript exactly.
• Hotword bias was verified with deliberately strong scores on acoustically competing paths; e.g. tomorrow changes test numbers to tomorrow numbers.
• EN_2 + 3 seconds silence + EN_3 produced two contiguous endpoint segments: [0,81920] and [81920,165974]. sherpa-onnx produced the same words and endpoint split; audio.cpp normalizes the boundary punctuation from security , If to security, If.
• A 48 kHz stereo streaming request is byte-identical to offline output for transcript and word JSON. Peak normalized waveform storage was 48,000 values and peak resampler-source storage was one 48,000-value input chunk for 79,872 total normalized samples.
• Standalone Q8 GGUF passed CUDA streaming with modified beam, blank penalty, endpointing, and movable embedded sidecars.
• The server JSON route forwarded decoding_method, max_active_paths, blank_penalty, and enable_endpoint correctly. The final Q8/CUDA server request completed in 284.322 ms wall time.
• The existing 128-L tensor-boundary parity remains unchanged: embedding cosine 0.999959055 / 0.999957586 and final encoder cosine 0.999573775 / 0.999731910 for chunks 1 / 2.

CPU performance and parity safeguards

A focused five-run EN_3 CPU test measured 203.664 ms mean native session time versus 88.704 ms for optimized ONNX Runtime, a 2.296x ratio. Native execution remains roughly 20x faster than real time on this machine.

I also tested keeping recurrent state resident and retaining allocator-managed graph constants between calls. Both shortcuts changed decoded tokens, so they were rejected. The committed path preserves the explicit transfers required for transcript and timestamp parity.

Intentionally unchanged / known limitations

• Multi-package automatic language routing is intentionally not part of this update; each package remains single-language.
• Conversion remains limited to public packages declaring free=true.
• Native CPU remains slower than optimized ONNX Runtime.
• Vulkan and Metal still need contributor testing.

[mirek190]

Added commit 34fe021 to align Kroko with the normalized v1 option contract.

followup to #123 (comment)

• The shared beam-width concept is now published as num_beams, matching the framework contract and allowing central integer validation.
• The loader, model_specs_v1, usage documentation, server example, and validation commands all advertise num_beams.
• Existing max_active_paths, kroko_asr.max_active_paths, and namespaced kroko_asr.num_beams requests remain accepted as compatibility aliases.
• Kroko-specific controls such as decoding_method, blank_penalty, hotwords, and endpoint rules remain local unprefixed request names because the framework currently has no shared contracts for those concepts.
• Supplying conflicting normalized and legacy values is rejected by the shared option resolver.

Validation: JSON/model-manager checks pass, loader/catalog synchronization passes (37 active loaders, 48 packages), git diff --check passes, and the complete CPU engine_runtime target builds. No decoder math or output behavior changed.

[0xShug0]

@mirek190 I've become the ruthless button-clicker and merged all three of your PRs. 😄

[mirek190]

@mirek190 I've become the ruthless button-clicker and merged all three of your PRs. 😄

Hehe

[mirek190]

GGUF models to test

https://huggingface.co/mirek190/audio.cpp/tree/main/Audio%20to%20text%20(ASR)/Kroko

[0xShug0]

@mirek190 Please check #128. The example is Confucius4-TTS.