Skip to content

feat: Parakeet-TDT-0.6B-v3 as a fast CPU transcription backend for short audio - #29

Merged
eksrha merged 2 commits into
mainfrom
feat/parakeet-asr-backend
Nov 14, 2026
Merged

eksrha merged 2 commits into
mainfrom
feat/parakeet-asr-backend

Conversation

@eksrha

@eksrha eksrha commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

Closes #28

What

  • Server deploy/parakeet/server.py: OpenAI-compatible POST /v1/audio/transcriptions on onnx-asr 0.12.0 + ONNX Runtime 1.32 (CPU). onnx-asr was chosen over sherpa-onnx because its only dependency is numpy. Decoding and resampling to 16 kHz mono with PyAV (FFmpeg bundled in the wheel): wav, mp3, ogg/opus, webm/opus, mp4/m4a. Formats json, text, verbose_json (segments split at sentence punctuation, words with start/end/confidence), srt, vtt. Audio above 30 s is cut at a quiet point and transcribed in chunks; above MAX_AUDIO_S (120) it is rejected with 413. MAX_CONCURRENT + MAX_QUEUE, then 429 with Retry-After. Warmup before /health reports ready. Logs carry sizes and timings, never audio or text.
  • Image deploy/Containerfile.parakeet: Chainguard wolfi-base (digest-pinned) with Python 3.13 (the public Chainguard Python image is on 3.15, which has no ONNX Runtime wheels yet). Model files pinned by revision and SHA-256, baked in (no runtime download, no pickle). CC-BY-4.0 attribution in /opt/model/NOTICE and the image description. 1.08 GB.
  • Router: /v1/audio/transcriptions goes to Parakeet only when it is enabled and the multipart model field is parakeet-tdt-0.6b-v3 (a provider prefix is accepted). Everything else, including translations, goes to Whisper unchanged. The body is buffered only up to the model part.
  • Chart: optional parakeet Deployment, Service and NetworkPolicy (ingress only from the router, no egress), soft anti-affinity to the embedding pods, read-only root filesystem, enabled: false by default.
  • Release/CI: relctl component parakeet (image parakeet:tdt-0.6b-v3-<CalVer>), CI jobs for the server tests and the chart rendering test.

Measurements (container, AVX2 host)

Clip 2 CPUs 4 CPUs
5 s 0.59 s
10 s 0.84 s
12.7 s 1.53 s 1.06 s
32.9 s 3.91 s 2.62 s
66 s (3 chunks) 7.72 s

Peak memory 1.6 GiB (2 CPUs, 66 s). WER on two synthetic German clips with domain terms: 11.4 % (12.7 s) and 11.3 % (32.9 s); Whisper large-v3-turbo with a domain prompt: 14.3 % and 8.3 % on the same clips.

Tests

  • go test -race ./..., golangci-lint: routing by model id (model before/after the file, prefix, unknown id, disabled backend, translations), body passed through byte for byte
  • pytest deploy/parakeet (19 tests, fake model, real decoding incl. webm/opus, ogg/opus, m4a, mp3)
  • scripts/test-chart.sh

Open points

  • A chunk cut can split a word in continuous speech without pauses.
  • Parakeet has no prompt or hotword input; domain terms must be corrected after transcription.
  • Kubelet probes must be allowed by the CNI despite the NetworkPolicy (most CNIs allow node traffic).

…r short audio

A second OpenAI-compatible transcription backend (onnx-asr on ONNX Runtime,
int8 model baked into a Chainguard Python image). The router sends
/v1/audio/transcriptions to it only when the multipart model field names
parakeet-tdt-0.6b-v3; everything else, including translations, stays on
Whisper. The chart adds an optional parakeet Deployment, Service and a
NetworkPolicy that admits only the router (disabled by default).

Refs #28
@github-actions

Copy link
Copy Markdown
Possible relctl commands for this Pull Request

Version scheme: calver. The version of the release is the date of the release (YYYYMM.D.N, N counts the releases of that day). Branch prefixes and comment commands have no effect.

Need more help?

Have a look at my repo

This message was created by relctl and can be disabled by the env variable RELCTL_SILENT=true

…load is parsed

- router: read at most 1 MiB to find the model field, then stream the rest;
  a later or missing field goes to Whisper
- parakeet: admission middleware answers 429 and 413 before Starlette spools
  the upload; decode from the spooled file without a second copy; 504 after
  REQUEST_TIMEOUT_S; OpenAI-style error bodies
- image: hash-pinned requirements installed with --require-hashes
- chart: refuse to render when concurrent uploads can exceed the /tmp volume;
  request timeout; note on kubelet probes and the NetworkPolicy

Refs #28
@eksrha

eksrha commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

Review follow-up:

  • Router reads at most 1 MiB to find model, then streams the rest; a missing or late field goes to Whisper with the body unchanged (tests for a 2 MiB file before model, a 3 MiB body without model, several model fields).
  • Parakeet: an ASGI middleware answers 429 and 413 before the upload is spooled (Content-Length, or counting chunked bodies); decoding reads the spooled upload directly; 504 after REQUEST_TIMEOUT_S, checked while decoding and between chunks; OpenAI-style error bodies. Tests use real back-pressure (two blocked requests, third gets 429) and a chunked 4 MiB upload.
  • Chart refuses to render when (maxConcurrent + maxQueue) x maxUploadMB does not fit into tmpSizeMB; probe/CNI note in values.
  • Requirements are hash-pinned (pip-compile --generate-hashes) and installed with --require-hashes.
  • Container re-run (2 CPUs): 12.7 s clip 1.55 s, 32.9 s clip 4.06 s, 66 s clip 7.8 s, peak memory 1.71 GiB (tmpfs upload included).

@eksrha
eksrha marked this pull request as ready for review November 14, 2026 09:22
@eksrha
eksrha merged commit f50c25a into main Nov 14, 2026
5 checks passed
@eksrha
eksrha deleted the feat/parakeet-asr-backend branch November 14, 2026 09:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fast CPU transcription backend for short audio (Parakeet-TDT-0.6B-v3)

1 participant