In progress: Add community_models/parakeet_tdt: NVIDIA Parakeet-TDT 0.6B v3 ASR#111
Draft
dleiferives wants to merge 12 commits into
Draft
In progress: Add community_models/parakeet_tdt: NVIDIA Parakeet-TDT 0.6B v3 ASR#111dleiferives wants to merge 12 commits into
dleiferives wants to merge 12 commits into
Conversation
Initial port of nvidia/parakeet-tdt-0.6b-v3 — a 600M-param
FastConformer-TDT
ASR model supporting 25 European languages with auto-detection.
Files under community_models/parakeet_tdt/:
- assets: config parsing (Transformers-format), tokenizer loading
- weights: NeMo safetensors key mapping, BN fold for 24-layer
depthwise conv
- frontend: Mel filterbank frontend (16kHz, 128 bins, preemphasis=0.97)
- encoder: 24-layer FastConformer, NeMo-style Conv2D subsampling,
relative position attention with untied biases, xscaling
- decoder: 2-layer LSTM predictor + joint (joint_enc(1024→640),
decoder_projector(640→640), joint_head(640→8198))
- session: offline session with prepare/run
- loader: IVoiceModelLoader, family="parakeet_tdt"
- headers: include/engine/community_models/parakeet_tdt/
Wiring: model_specs/parakeet_tdt.json, registry.cpp registration,
CMakeLists.txt engine_runtime sources + parakeet_warm_bench target.
The model loads and runs end-to-end on both CPU and CUDA (GTX 1650:
~180ms vs ~1570ms CPU). Encoder graph builds and produces 93-frame
1024-dim output. Decoder produces all-blank output due to a suspected
safetensors reader bug returning wrong data for tensors at offsets
beyond ~40 MB — BN statistics and depthwise conv weights in the
48 MB region read different values than direct HTTP range fetches,
while embedding weights at 1.6 MB read correctly.
…tend The C++ frontend computed log-mel features (preemphasis -> STFT -> mel filterbank -> log) but never normalized them, emitting raw log-mel values (mean ~-11.8, std ~4.1). NeMo's AudioToMelSpectrogramPreprocessor for this model is configured with normalize: "per_feature", which subtracts the per-mel-bin mean and divides by the per-mel-bin unbiased std (computed over the valid, unpadded time frames only, with a 1e-5 epsilon added to the std) before the features ever reach the encoder. Skipping this step fed the subsampling conv stack input at roughly the wrong scale by orders of magnitude, and that error compounded through the conv/linear layers, producing runaway-magnitude activations by the time they reached the transformer encoder layers. Confirmed against NeMo's own normalize_batch() (features.py): with the fix, our mel features match NeMo's bit-for-bit in direction (cosine similarity 1.0) and to 7 decimal places in scale, and the subsampling stage's linear output now matches NeMo's to 4 significant figures (previously off by ~500x). This does not fully fix decoding on its own — a separate bug remains in the transformer encoder layers — but it eliminates a major, independently-verified source of divergence from the reference model. Bing Bong
The FastConformer encoder layer's depthwise conv was padded causally (all padding on the left, via pad_causal_1d(x, kernel-1)), which is wrong for this model's full-context (non-streaming) configuration. NeMo's ConformerConvolution.depthwise_conv is a CausalConv1D, but for the full-context encoder it is constructed with an explicit integer padding=conv_context_size=(kernel_size-1)//2, and CausalConv1D treats an int padding argument as SYMMETRIC left/right padding, not causal-only left padding (only a `None` or explicit [left, right] argument produces asymmetric/causal padding, and neither is used here). Padding causally instead shifted every frame's receptive field, corrupting the residual stream through all 24 encoder layers. Isolated a single encoder layer's conv module against hook-captured NeMo reference activations (same weights, same input) to localize this: with causal padding the module's output was nearly uncorrelated with NeMo's (cosine similarity 0.22); switching to symmetric padding raised that to 0.92, uniformly across all 93 frames (not concentrated at the sequence edges, which rules out an off-by-one/boundary-only explanation and points at the padding mode itself). Combined with the prior frontend normalization fix, the model now transcribes the reference test clip correctly end to end: "Well, I don't wish to see it any more, observed Phoebe, turning away her eyes. It is certainly very like the old portrait." — matching the NeMo reference transcription exactly. Bing Bong
Two follow-ups from the recent Parakeet TDT correctness review
(tmp/TODO.md items 1-2):
- decoder.cpp: remove an unconditional fprintf(stderr, "DECODE_DIAG ...")
that fired on every single decode() call (frame 0 of every request),
left over from the earlier blank-output debugging session. It also
shadowed the outer `cfg` local with a redundant re-declaration.
- assets.cpp: stop silently swallowing all processor_config.json parse
errors behind a bare `catch (...)` that only restored the preemphasis
default, leaving sample_rate/feature_size/n_fft/win_length/hop_length
silently on their struct defaults if anything about that file failed to
parse. Now: skip parsing (and use defaults) only when the file is
genuinely absent (resources.has_file(...) == false); if it's present but
fails to parse or doesn't match the expected schema, that exception now
propagates instead of being swallowed. Verified against three cases:
file present and well-formed (unchanged behavior), file missing entirely
(still falls back to defaults, though in practice resource-bundle file
matching already makes this case unreachable today), and file present
but malformed (previously silent, now throws
"missing required json key: feature_extractor" as expected).
Rebuilt and reran parakeet_warm_bench against the reference clip: stderr
is now clean of DECODE_DIAG output and the transcription is unchanged
("Well, I don't wish to see it any more, observed Phoebe, turning away
her eyes. It is certainly very like the old portrait.").
Bing Bong
Follow-up from the recent Parakeet TDT correctness review (tmp/TODO.md item 3, "Tier 1"). Neither of the two real bugs fixed in 4a10c48/3bf2d12 (missing frontend normalization, wrong conv padding mode) crashed or produced an obviously malformed result — the model loaded, ran, and produced plausible-shaped tensors throughout — so nothing in the existing test suite would have caught either regression. This adds the cheapest thing that actually would have: run the full offline pipeline against a fixed clip and assert the decoded text matches the known-correct NeMo reference transcription exactly. - tests/parakeet_tdt/assets/2086-149220-0033.wav: a ~7.44s LibriSpeech test-clean clip (CC BY 4.0), with provenance documented in the accompanying assets/README.md. - tests/parakeet_tdt/test_golden_transcription.cpp: loads the model, runs it against that clip, and asserts the output text is exactly "Well, I don't wish to see it any more, observed Phoebe, turning away her eyes. It is certainly very like the old portrait." — verified against the actual NeMo reference model output for this clip, not just eyeballed. Requires the real (multi-GB) model weights, which aren't present in a normal checkout/CI build; the test detects this at startup and exits 125 (SKIP) rather than failing when the model or fixture audio is missing. - CMakeLists.txt: wires the test into the existing ENGINE_BUILD_TESTS/ ctest suite via the add_engine_unittest() helper, with SKIP_RETURN_CODE 125 configured so a missing-model environment shows as skipped rather than failed. This is not a substitute for numerical layer-by-layer parity comparison against NeMo (the "Tier 2" harness described in tmp/todo-03-add-parity-regression-test.md, not yet built) — it only catches regressions large enough to flip the final decoded text for this one clip. It's deliberately cheap enough to run on every change, unlike a NeMo-based numerical comparison which needs a NeMo install on top of the model weights this test already needs. Verified: `ctest -R parakeet_golden_transcription_test` passes (~7.4s on CPU) against the current model checkout, and the skip path was confirmed independently by running the binary against a nonexistent model path (exits 125 as expected). Bing Bong
The depthwise conv in build_fastconformer_conv_module was built with use_bias=false, silently dropping weights.conv_dw_bias — the batch-norm bias term fold_bn() computes and bakes into the depthwise conv (-running_mean*scale + bn_bias). The scale/weight side of the BN fold was applied correctly; only the additive bias term was being discarded. This did not produce obviously wrong output — greedy/argmax decoding turned out to be robust enough to this that the golden-clip transcription was already correct without it — which is exactly why it went unnoticed until a numerical (not just text-output) comparison against the real NeMo model was built. Isolating the depthwise conv in a minimal single-op graph and comparing against a hand-computed reference showed every output value differed from the correct one by exactly the same constant offset, equal in magnitude to the folded bias for that channel — a very clean signature once found, but invisible from the top-level transcription alone. Also exports build_encoder_layer (previously anonymous-namespace) via encoder.h so a layer can be built and numerically verified in isolation without duplicating the real production graph-building code — this is what made isolating the bug to this one line tractable instead of only being visible as an accumulated ~0.78 cosine similarity drift by the final of 24 layers. Verified: chaining 24 isolated single-layer builds — each fed the previous one's real output, exactly like the production encoder does — now stays at cosine similarity >= 0.999999 against the real NeMo model's per-layer output through every one of the 24 layers (previously this drifted down substantially by the later layers). Golden-clip transcription remains correct on both CPU and CUDA. Bing Bong
Adds the "Tier 2" numerical layer-by-layer comparison harness alongside the existing golden-transcription test: a NeMo reference dump script, a C++ dump tool built off the real production entry points (frontend extraction, full encoder, and one isolated encoder layer via the newly exported build_encoder_layer), and a numpy-only comparator that checks cosine similarity and relative scale per stage. This tool is what found and confirmed the encoder conv-module bias bug fixed in 837382e — the golden-transcription test alone did not catch it, since greedy decoding was robust enough to that particular corruption not to change the final transcribed text on the one test clip. Comparing actual activations, not just final decoded text, is a meaningfully different and complementary check. - tests/parakeet_tdt/parity/dump_nemo_reference.py: runs the real NeMo model with forward hooks, dumps mel features, pos_emb, every encoder layer's output, and the final encoder output as .npy files. - tests/parakeet_tdt/parity/npy_io.h: minimal float32 .npy reader/writer (just enough for this tool; not a general-purpose npy library). - tests/parakeet_tdt/parity/dump_cpp_reference.cpp: mirrors the NeMo dump using the real, unmodified C++ production entry points for mel features and the full encoder, plus one isolated single-layer build for granular comparison. Deliberately does not splice extra debug-output taps into the shared production graph — see the file's top comment for why (an extra "dead end" output tensor on that specific cached multi-thousand- node graph read back incorrect values during the original encoder debugging session, for reasons never fully root-caused; small isolated graphs and the unmodified production entry points both proved reliable and are what this tool sticks to). - tests/parakeet_tdt/parity/compare_parity.py: numpy-only comparator, runnable without NeMo/torch installed. - CMakeLists.txt: wires parakeet_parity_dump as a buildable tool under ENGINE_BUILD_TESTS (not registered as an add_test — it's a manual/ pre-release check requiring a NeMo install on the Python side, not a CI gate). Bing Bong
The parakeet_tdt loader was registered and working but had no installable package entry (tools/check_loader_catalog_sync.py warned about this) and wasn't listed anywhere in README.md or docs/community_models/models.md. - tools/model_manager.py: add a parakeet_tdt ModelPackage entry. The repo (nvidia/parakeet-tdt-0.6b-v3) hosts both a Transformers-compatible safetensors checkpoint and a NeMo .nemo archive side by side; the package uses include_prefixes to fetch only the four files this loader actually reads (config.json, processor_config.json, tokenizer.json, model.safetensors), not the redundant multi-GB .nemo file. Verified the entry resolves against the real repo: `python3 tools/model_manager.py install parakeet_tdt` began fetching the correct files (config.json matched the existing local file's exact byte size; model.safetensors started downloading from the right repo) before being stopped partway through given the file size. - docs/model_manager.md: add the package row to the recommended package table (required by the sync checker). - docs/community_models/parakeet_tdt.md: new per-model doc following the existing community-model doc convention — build/run commands, what was validated and how (golden-transcription test + numerical parity harness), the two bugs found and fixed while getting there, and current known limitations (offline only, no streaming yet). - README.md, docs/community_models/models.md: add the community-models table row. python3 tools/check_loader_catalog_sync.py now passes with no warnings. Bing Bong
Investigated adding NeMo cache-aware streaming support (causal conv with a cross-chunk cache_last_time cache, chunked-limited attention with a cache_last_channel KV cache) before writing any code, and found the actual checkpoint doesn't support it in a meaningful way: - nvidia/parakeet-tdt-0.6b-v3's encoder config has att_context_style="regular" and att_context_size=[-1, -1] — unlimited, fully bidirectional attention, not NeMo's "chunked_limited" streaming style. - Calling NeMo's own encoder.setup_streaming_params() on this checkpoint doesn't error, but produces a degenerate configuration (~5.8 second chunks, a ~10000-frame attention cache) that wouldn't provide the latency or bounded-memory benefit real streaming is for. - NVIDIA's own maintainers, asked directly about streaming with this model on its Hugging Face discussion page, pointed users at a different, dedicated cache-aware streaming architecture rather than confirming this checkpoint for the purpose. Building a genuine chunked/causal streaming path against a model trained with an unrestricted attention receptive field risks silently degrading accuracy for a mode that wouldn't provide much practical benefit even working correctly. This needs a different, purpose-trained checkpoint (att_context_style="chunked_limited"), not more implementation effort against this one. This framework already has real streaming ASR: the nemotron_asr family targets nvidia/nemotron-3.5-asr-streaming-0.6b, a same-size-class NVIDIA checkpoint actually trained cache-aware with configurable chunk sizes down to 80ms, and already implements IStreamingVoiceTaskSession. Pointed to it from parakeet_tdt's known-limitations doc for anyone who lands here specifically wanting streaming. Bing Bong
pad_causal_1d, pad_causal_2d, and causal_conv_output_dim were leftover, never-called scaffolding from an earlier attempt at streaming support (compiler already flagged all three as unused). Confirmed streaming isn't a viable follow-up for this specific checkpoint (see the preceding commit) before removing them, so this isn't losing groundwork for later — it's dead code for a direction that doesn't apply to this model. The one function that was actually load-bearing, pad_symmetric_1d (the fix for the encoder conv module's padding mode), is unaffected. Verified: rebuilds warning-free, golden-transcription test and the numerical parity harness both still pass unchanged. Bing Bong
parakeet_warm_bench.cpp set ENGINE_TIMING_ENABLED/ENGINE_TIMING_FILE as
environment variables, but the logging subsystem doesn't read those env
vars anywhere in the codebase — every other *_warm_bench.cpp tool
(nemotron_asr, vibevoice_asr, qwen3_tts, etc.) wires up timing by calling
engine::debug::configure_logging(LoggingConfig{true, timing_path}) directly.
parakeet_warm_bench never did this, so its --timing-file output has been
silently empty since the tool was written — the frontend/encoder/decoder
breakdown was never actually available, only the top-level wall-clock
number.
Also fixed two stale/wrong keys in ordered_keys(): "parakeet.encoder_ms"
never matched what encoder.cpp actually logs
(debug::timing_log_scalar("parakeet_tdt.encoder_ms", ...) — note the
_tdt), and "parakeet.pre_encode_ms" is never emitted anywhere at all
(dead reference, presumably left over from a metric that was renamed or
removed). Replaced with the correct "parakeet_tdt.encoder_ms" and added
"parakeet_tdt.encoder.graph.compute_ms", which is what actually turned
out to matter for performance work (see the following commits): encoder
graph compute is ~93-96% of total wall time in every configuration
measured.
Bing Bong
…ights Compared audio.cpp against the real NeMo/PyTorch reference on the same machine (see docs/community_models/parakeet_tdt.md's new Performance section for the full numbers and methodology). With the timing breakdown now actually working (previous commit), encoder graph compute was clearly ~93-96% of total wall time in every configuration, so that's where optimization effort actually matters — not the frontend or decoder. Two changes, both using infrastructure that already existed (no new inference code): - Thread count: this machine is 6 physical / 12 logical cores; 8 threads (an arbitrary default) oversubscribes, 6 is consistently fastest, 12 is consistently worst. Core-count-dependent, not a universal number — sweep it on your own hardware. - parakeet_tdt.matmul_weight_type=q8_0 (an existing but previously untested session option): ggml's CPU backend has heavily hand-tuned Q8_0 dot-product kernels; measured faster than F16/BF16 here, not just faster than F32. Net: CPU 1247.5ms -> 830.9ms (1.60x), CUDA 167.4ms -> 132.1ms (1.27x), transcription unchanged in every configuration. CPU with q8_0 is now ~3% faster than PyTorch's own CPU number on this clip. Quantified the accuracy cost via the numerical parity harness rather than just trusting "the transcription still matched": enc_out cosine similarity vs the NeMo reference drops from 0.972258 (native F32) to 0.968028 (q8_0) — comparable in size to the float32 accumulation noise already present between an isolated single-layer graph and the full 24-layer production graph, not a cliff, but a real, measured, non-zero cost. Only validated against one clip, so this is documented as a strongly-recommended opt-in session option, not changed as the default — that would need broader validation first. - dump_cpp_reference.cpp: add --matmul-weight-type so the parity harness can numerically compare any weight storage type against the NeMo reference, which is what produced the 0.972258 -> 0.968028 numbers above. - docs/community_models/parakeet_tdt.md: new Performance section. - tests/parakeet_tdt/parity/README.md: document the new flag. Bing Bong
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Initial port of nvidia/parakeet-tdt-0.6b-v3 — a 600M-param FastConformer-TDT
ASR model supporting 25 European languages with auto-detection.
Files under community_models/parakeet_tdt/:
depthwise conv
relative position attention with untied biases, xscaling
decoder_projector(640→640), joint_head(640→8198))
Wiring: model_specs/parakeet_tdt.json, registry.cpp registration, CMakeLists.txt engine_runtime sources + parakeet_warm_bench target.
The model loads and runs end-to-end on both CPU and CUDA (GTX 1650: ~180ms vs ~1570ms CPU). Encoder graph builds and produces 93-frame 1024-dim output. Decoder produces all-blank output due to a suspected safetensors reader bug returning wrong data for tensors at offsets beyond ~40 MB — BN statistics and depthwise conv weights in the 48 MB region read different values than direct HTTP range fetches, while embedding weights at 1.6 MB read correctly.