diff --git a/AGENTS.md b/AGENTS.md
index b521405..14a4165 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -101,7 +101,7 @@ Key details:
### Whisper (Standalone)
-Model is baked directly into the image at build time. No init container, no PVC needed. The image runs `deploy/whisper/server.py`, which speaks the OpenAI audio API itself (the router forwards paths unchanged). `/health` does not queue behind inference. Pin `av` (see Containerfile) — newer PyAV majors break faster-whisper. Each request logs one JSON line (`whisper_request`, timings, RTF; never prompt/transcript text). `WHISPER_INITIAL_PROMPT` / `WHISPER_HOTWORDS` are server-side defaults (see README). Tests: `pytest deploy/whisper` (stubbed model).
+Model is baked directly into the image at build time. No init container, no PVC needed. The image runs `deploy/whisper/server.py`, which speaks the OpenAI audio API itself (the router forwards paths unchanged). `/health` does not queue behind inference. Pin `av` (see Containerfile) — newer PyAV majors break faster-whisper. Each request logs one JSON line (`whisper_request`, timings, RTF; never prompt/transcript text). `WHISPER_INITIAL_PROMPT` / `WHISPER_HOTWORDS` are server-side defaults (see docs/tuning.md). Tests: `pytest deploy/whisper` (stubbed model).
### Router (Go Proxy)
@@ -137,7 +137,7 @@ make docker/whisper # whisper
## Tech Stack & Constraints
-- **Go 1.24+** — no generics workarounds needed
+- **Go 1.26** (see `go.mod`)
- **`net/http/httptest`** for all tests — no external test frameworks (no testify, no gomock)
- **`go.uber.org/zap`** for structured logging
- **`github.com/prometheus/client_golang`** for metrics
@@ -228,7 +228,7 @@ It exposes `/health`, `/v1/models`, and the backend-specific inference endpoints
- `promauto` metrics panic on double-registration. If you add a new metric, add it to `internal/metrics/metrics.go` as a package-level var, not inside a function.
- `health.Checker` must be initialised with `health.New(cfg.EnabledBackends(), log)` — passing `cfg.ActiveBackends()` will include disabled backends and skew the health response.
- The `statusWriter` wrapper in `proxy.go` must set `status = 200` lazily (in `Write`) to handle the case where `WriteHeader` is never called explicitly.
-- TEI does **not** support reranking — the reranker uses FlagEmbedding, not TEI.
+- Reranking runs on TEI (ONNX, see `docs/ADR-001-tei-for-reranking.md`). `deploy/Containerfile.reranker-server` and `deploy/reranker/` are the legacy FlagEmbedding server and are not released any more.
- Some CSI volumes don't support timestamp preservation — always use `rsync -rlgoDv`, never `rsync -a`.
- Pod `fsGroup: 1000` is required for init containers to write to PVCs (CSI provisions volumes as root).
@@ -250,8 +250,11 @@ It exposes `/health`, `/v1/models`, and the backend-specific inference endpoints
| `EMBEDDING_URL` | — | Base URL for TEI embedding |
| `EMBEDDING_TIMEOUT` | `60s` | Per-request timeout |
| `RERANKER_ENABLED` | `false` | Enable reranker backend |
-| `RERANKER_URL` | — | Base URL for FlagEmbedding reranker |
+| `RERANKER_URL` | — | Base URL for the TEI reranker |
| `RERANKER_TIMEOUT` | `30s` | Per-request timeout |
| `WHISPER_ENABLED` | `false` | Enable Whisper backend |
| `WHISPER_URL` | — | Base URL for Whisper |
| `WHISPER_TIMEOUT` | `300s` | Per-request timeout |
+| `EMBEDDING_MAX_CONCURRENT` / `RERANKER_MAX_CONCURRENT` / `WHISPER_MAX_CONCURRENT` | `20` / `20` / `8` | Concurrency bound per backend; above it the router answers `503` + `Retry-After` |
+
+User-facing documentation: `README.md`, `docs/configuration.md`, `docs/tuning.md`.
diff --git a/README.md b/README.md
index bf115c8..d508a68 100644
--- a/README.md
+++ b/README.md
@@ -1,124 +1,150 @@
-# inference-stack
-
-[](https://github.com/layer87-labs/inference-stack/actions/workflows/ci.yml)
-[](LICENSE)
-
-Unified CPU-based inference stack — serving embeddings, reranking and transcription behind a single OpenAI-compatible API.
+
+
+
-## Components
+
+
+
+
+
+
-| Component | Runtime | Model | Purpose |
-| -------------------- | ---------------------------------------------------------------------- | ----------------------- | --------------------------------------- |
-| **inference-router** | Go | — | Unified OpenAI-compatible reverse proxy |
-| **embedding** | [TEI](https://github.com/huggingface/text-embeddings-inference) (Rust) | BAAI/bge-m3 | Dense + sparse embeddings |
-| **reranker** | [TEI](https://github.com/huggingface/text-embeddings-inference) (Rust, ONNX) | mmarco-mMiniLMv2-L12-H384-v1 (default), bge-reranker-v2-m3 (option) | Cross-encoder reranking |
-| **whisper** | [faster-whisper](https://github.com/SYSTRAN/faster-whisper) (Python) | large-v3-turbo | Audio transcription (ASR) |
+# inference-stack
-All backends are disabled by default and enabled selectively via Helm values or env vars.
+**inference-stack** is a small OpenAI-compatible router in front of CPU-only inference backends. One endpoint
+gives you **embeddings** (`BAAI/bge-m3` via [TEI](https://github.com/huggingface/text-embeddings-inference)),
+**reranking** (a cross-encoder via TEI/ONNX) and **speech-to-text** (Whisper `large-v3-turbo` via
+[faster-whisper](https://github.com/SYSTRAN/faster-whisper)). It runs on plain CPU and RAM. No GPU needed.
+
+**Why it exists.** Search, RAG and transcription should not need a GPU budget or a pile of glue code. We
+([Layer87](https://github.com/layer87-labs)) needed exactly this in our own applications, built it, run it ourselves
+and open-sourced it. It ships with hardened container images, a Helm chart and signed releases.
+
+## Highlights
+
+- **One API, three capabilities.** `/v1/embeddings`, `/v1/rerank`, `/v1/audio/transcriptions`, plus a merged `/v1/models`
+ that also reports the backends' request limits.
+- **CPU only.** Sized and measured on CPU; the numbers (latency per rerank request, memory, real-time factor) are in
+ [docs/tuning.md](docs/tuning.md).
+- **Backends are optional.** All are off by default; enable only what you need.
+- **Backpressure instead of pile-ups.** Per-backend concurrency bound; above it the router answers `503` with
+ `Retry-After` right away ([details](docs/configuration.md#backpressure)).
+- **Observable.** Prometheus metrics on a separate port, JSON logs, optional `ServiceMonitor` and `NetworkPolicy`.
+- **Signed images.** Released images are signed keyless with [cosign](https://github.com/sigstore/cosign) via GitHub
+ OIDC ([verify](#security)).
+- **Hardened by default.** Non-root containers, no privilege escalation, all capabilities dropped, `RuntimeDefault`
+ seccomp, executable-stack flag cleared on the ONNX Runtime libraries of the TEI base image.
+- **Offline at runtime.** Models are baked into images (`HF_HUB_OFFLINE=1`); model volumes are mounted read-only.
+- **Pinned build inputs.** The Whisper image pins its base image by digest, all Python dependencies, the model
+ revision and the model file's SHA-256; the build fails if any of them changes.
+- **Per-component CalVer releases.** Router, TEI images, models, Whisper and the Helm chart are released
+ independently with [relctl](https://github.com/layer87-labs/relctl).
-## Quick Start
+## Architecture
-### Local Development
+```
+ ┌──────────────────────────────────────┐
+ your app ──HTTP──▶ │ inference-router (Go, :8080) │ ─▶ :9090/metrics
+ OpenAI-style API └───────┬───────────────┬──────────────┘
+ │ │ │
+ /v1/embeddings /v1/rerank /v1/audio/transcriptions
+ ▼ ▼ ▼
+ ┌────────────┐ ┌────────────┐ ┌──────────────────┐
+ │ TEI │ │ TEI (ONNX) │ │ faster-whisper │
+ │ bge-m3 │ │ cross-enc. │ │ large-v3-turbo │
+ └─────▲──────┘ └─────▲──────┘ └──────────────────┘
+ │ /model (ro) │ /model (ro) model baked into image
+ model-init (rsync) model-init (rsync)
+```
-```bash
-# Run unit tests
-make test
+The router is a pure reverse proxy: it routes by path, bounds concurrency, probes backend health and exports
+metrics. It contains no model logic. The TEI backends read their model from a volume that an init container
+fills from a model baked into its image (idempotent, `rsync --ignore-existing`); Whisper has the model in the
+image itself.
-# Full end-to-end test (starts mock backends + router)
-make test-local
-```
+## Quick start
-### Kubernetes (Helm)
+**Try it locally in a minute** (mock backends, no models, needs Go):
```bash
-helm install inference-stack ./deploy/helm \
- --namespace ai \
- --set embedding.enabled=true \
- --set reranker.enabled=true \
- --set router.enabled=true
+make test-local # builds, starts mock backends + router, runs the end-to-end checks
```
-## API
-
-All endpoints are OpenAI-compatible:
+Or start the pieces by hand and call the API yourself:
```bash
-# Embeddings
-curl http://localhost:8080/v1/embeddings \
- -H "Content-Type: application/json" \
- -d '{"model":"BAAI/bge-m3","input":"Hello world"}'
-
-# Reranking
-curl http://localhost:8080/v1/rerank \
- -H "Content-Type: application/json" \
- -d '{"query":"What is Go?","documents":["Go is a language","Python is a language"]}'
-
-# Transcription
-curl http://localhost:8080/v1/audio/transcriptions \
- -F model=whisper-large-v3-turbo \
- -F file=@audio.mp3
+make build
+MOCK_TYPE=embedding MOCK_PORT=9101 build/bin/mock-backend &
+EMBEDDING_ENABLED=true EMBEDDING_URL=http://localhost:9101 build/bin/inference-router &
-# Model list (merged from all enabled backends)
-curl http://localhost:8080/v1/models
+curl http://localhost:8080/v1/embeddings -H 'Content-Type: application/json' \
+ -d '{"model":"BAAI/bge-m3","input":"Hello world"}'
```
-`/v1/models` also reports the request limits of the embedding and reranker
-backends (`limits.max_batch_tokens`, `limits.max_client_batch_size`,
-`limits.max_input_length`), read from the running backend, so clients can size
-batches without trial and error. The model id it advertises
-(`BAAI/bge-m3`) can be sent back as `model` on `/v1/embeddings`; the router
-strips it before forwarding, because TEI only accepts its internal model path
-or no model at all.
+**Run it for real on Kubernetes** with the Helm chart from the OCI registry. Pick a chart version from the
+[releases](https://github.com/layer87-labs/inference-stack/releases) (tags look like `chart/`):
-## Architecture
+```yaml
+# values.yaml: everything is off by default, enable what you need
+router: { enabled: true }
+embedding: { enabled: true }
+reranker: { enabled: true }
+whisper: { enabled: true }
+```
-### Init Container Pattern (Embedding + Reranker)
+```bash
+helm install stack oci://ghcr.io/layer87-labs/charts/inference-stack \
+ --version --namespace ai --create-namespace -f values.yaml
-```
-┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
-│ model-init │────▶│ /model (PVC) │────▶│ main container │
-│ (bakes model) │ │ shared volume │ │ (reads model) │
-└─────────────────┘ └──────────────────┘ └─────────────────┘
+kubectl -n ai port-forward svc/stack-inference-stack-router 8080:8080
```
-1. **Init container** copies baked model to PVC via `rsync --ignore-existing`
-2. **Main container** reads model from PVC (mounted read-only)
-3. PVC persists model across restarts (idempotent — no re-copy on restart)
+The first start takes a while: the init containers copy the models to the volume. Sizing per backend (CPU, memory,
+batch tokens) is in [docs/tuning.md](docs/tuning.md); all chart options are commented in
+[`deploy/helm/values.yaml`](deploy/helm/values.yaml).
-### Whisper (Standalone)
+## API
-Model baked into image at build time. No init container or PVC needed. The
-image runs `deploy/whisper/server.py` (faster-whisper behind a small FastAPI
-app) which implements `POST /v1/audio/transcriptions` and
-`/v1/audio/translations` natively, so the router forwards requests unchanged.
-`/health` is answered independently of inference and stays responsive while a
-transcription is running.
+| Method and path | Backend | Notes |
+| --- | --- | --- |
+| `POST /v1/embeddings` | embedding | OpenAI-style. The advertised model id `BAAI/bge-m3` may be sent as `model`. |
+| `POST /embed`, `POST /embed_sparse` | embedding | TEI native (dense and sparse) |
+| `POST /v1/rerank`, `POST /rerank` | reranker | `/v1/rerank` is rewritten to TEI's `/rerank` |
+| `POST /v1/audio/transcriptions` | whisper | multipart; `json`, `text`, `verbose_json`, `srt`, `vtt` |
+| `POST /v1/audio/translations` | whisper | exposed, but `large-v3-turbo` is not trained for translation |
+| `GET /v1/models` | all enabled | merged list incl. `limits` of embedding and reranker |
+| `GET /healthz`, `GET /readyz` | router | per-backend status, always HTTP 200 (`ok` or `degraded`) |
+| `GET :9090/metrics` | router | Prometheus |
-### Router (Go Proxy)
+```bash
+curl http://localhost:8080/v1/rerank -H 'Content-Type: application/json' \
+ -d '{"query":"What is Go?","texts":["Go is a language","Python is a language"]}'
-Pure reverse proxy — routes requests to backends based on path prefix. Zero model logic.
+curl http://localhost:8080/v1/audio/transcriptions -F model=whisper-large-v3-turbo -F file=@audio.mp3
+```
-## Container Images
+A disabled backend answers `503 backend_disabled`; an unreachable one `502 upstream_error`; one at its
+concurrency bound `503 backend_overloaded` with `Retry-After`.
-All images are published to `ghcr.io/layer87-labs/`:
+## Components and images
-| Image | Description |
-| --------------------- | -------------------------------------- |
-| `inference-router` | Go reverse proxy |
-| `tei-base` | Hardened TEI base (patchelf, non-root) |
-| `tei-runtime` | TEI runtime (model via volume) |
-| `tei-model-init` | BGE-M3 model baked, copies to volume |
-| `tei-reranker-model-init` | Reranker model as ONNX (default: mmarco-mMiniLMv2-L12-H384-v1; option: BGE-reranker-v2-m3) |
-| `reranker-server` | FlagEmbedding HTTP server |
-| `whisper` | Whisper ASR server, model baked in |
+| Component | Image(s) in `ghcr.io/layer87-labs/` | Purpose |
+| --- | --- | --- |
+| router | `inference-router` | Go reverse proxy on a minimal static base image (no shell) |
+| tei | `tei-base`, `tei-runtime` | Hardened TEI CPU base and runtime (model via volume) |
+| embedding-model | `tei-model-init` | `BAAI/bge-m3` baked in, copies to the volume |
+| reranker-model | `tei-reranker-model-init` | Reranker as ONNX: `cross-encoder/mmarco-mMiniLMv2-L12-H384-v1` (default) or `BAAI/bge-reranker-v2-m3` |
+| whisper | `whisper` | faster-whisper server, `large-v3-turbo` baked in |
+| chart | `charts/inference-stack` (OCI) | Helm chart |
-All images run as non-root with no privilege escalation.
+Why the default reranker is the small model: on 4 CPUs it needs 0.76 to 0.85 s for 20 documents of 800
+characters, against 8.3 to 11.9 s for `bge-reranker-v2-m3`. Method and quality numbers are in
+[docs/tuning.md](docs/tuning.md#reranker-model-choice).
-### Verify image signatures
+## Security
-Images built by the release workflows are signed with
-[cosign](https://github.com/sigstore/cosign) keyless via GitHub OIDC. Verify by
-digest against the exact workflow identity:
+Container images from the release pipeline are signed keyless with cosign. Verify by digest against the exact
+workflow identity:
```bash
IMAGE=ghcr.io/layer87-labs/inference-router
@@ -130,290 +156,33 @@ cosign verify \
"$IMAGE@$DIGEST"
```
-`release.yml` signs `inference-router`, `tei-base` and `tei-runtime`.
-`release-models.yml` signs `tei-model-init`, `tei-reranker-model-init` and
-`whisper`; for those, use the identity
-`https://github.com/layer87-labs/inference-stack/.github/workflows/release-models.yml@refs/heads/main`
-(for manual runs, the ref is the branch the workflow was started from).
-Images published before signing was added are unsigned.
+Model and Whisper images published earlier were signed by the separate `release-models.yml` workflow (same
+command, that workflow file as identity). Images published before signing was added are
+unsigned. The Helm chart is not signed.
-## Build
+Runtime posture of the chart defaults: non-root users (uid 65532 for the router, 1000 for the rest),
+`allowPrivilegeEscalation: false`, `capabilities.drop: [ALL]`, `seccompProfile: RuntimeDefault`, a read-only root
+filesystem for the router, read-only model mounts, and no outbound network calls needed at runtime. An optional
+`NetworkPolicy` restricts the router.
-```bash
-# Go binary
-make build
+## Documentation
-# All container images
-make docker
-
-# Push to registry
-make push REGISTRY=ghcr.io/layer87-labs VERSION=202611.3.1
-```
+- [docs/configuration.md](docs/configuration.md): environment variables, backpressure, health endpoints, metrics
+- [docs/tuning.md](docs/tuning.md): CPU sizing, reranker model choice and measurements, Whisper options, scheduling
+- [docs/ADR-001-tei-for-reranking.md](docs/ADR-001-tei-for-reranking.md): why reranking runs on TEI
+- [AGENTS.md](AGENTS.md): repository layout, build and test commands, release components
-## Releases
-
-The router, TEI images, embedding model, reranker models, Whisper and the Helm chart are released
-independently as [relctl](https://github.com/layer87-labs/relctl) components with CalVer versions
-(`YYYYMM.D.N`, tag `/`). Only the parts whose files changed are built on a merge to
-`main`. See `AGENTS.md` for the component table and image tag formats.
-
-## Metrics
-
-Prometheus metrics on `:9090/metrics`:
-
-- `inference_router_requests_total` — total requests by backend/path/status
-- `inference_router_request_duration_seconds` — latency histogram
-- `inference_router_active_requests` — in-flight requests
-- `inference_router_backend_up` — backend health (0/1)
-- `inference_router_upstream_errors_total` — upstream error counts
-
-## CPU Tuning
-
-### Embedding (TEI + BGE-M3)
-
-| `--max-batch-tokens` | RSS (fp32) | Recommendation |
-| -------------------- | ---------- | -------------------- |
-| 4096 | 3-6Gi | **CPU default** |
-| 8192 | 6-10Gi | May OOM on 8Gi limit |
-| 16384 | 10-16Gi | GPU-only |
-
-### Reranker model choice
-
-The reranker model is baked into the `tei-reranker-model-init` image and
-selected by its tag (`reranker.modelInit.image.tag`). Both models run on the
-same TEI runtime with the ONNX Runtime backend.
-
-Measured with TEI `cpu-1.9.4`, limited to 4 CPUs, `--max-batch-tokens 2048`,
-one `/rerank` request with one query and 20 documents of 800 characters each
-(German/English mix), 4 runs per model. The CPU was an older AVX2 machine, so
-only the ratios carry over.
-
-| Model | Format | Parameters | Latency (20 docs) |
-| ------------------------------------------------------ | ------------------------- | ---------- | ----------------- |
-| `cross-encoder/mmarco-mMiniLMv2-L12-H384-v1` (default) | fp32 ONNX | 118M | 0.76 to 0.85 s |
-| `BAAI/bge-reranker-v2-m3` | fp32 ONNX | 568M | 8.3 to 11.9 s |
-| `BAAI/bge-reranker-v2-m3` | int8 ONNX | 568M | 7.9 to 8.2 s |
-| `Alibaba-NLP/gte-multilingual-reranker-base` | safetensors (candle path) | n/a | 11.1 to 12.1 s |
-
-That is about 0.04 s per document pair for the default against roughly 0.4 to
-0.5 s for `bge-reranker-v2-m3`, so reranking 40 candidates with the larger
-model does not fit an interactive latency budget on 4 CPUs.
-
-Quality, on a downstream 75-question German/English relevance set (hybrid
-retrieval, rerank the top 20 candidates, 800 characters per document):
-
-| Setup | hit@1 | hit@5 | hit@10 |
-| ------------------------------------------ | ----- | ----- | ------ |
-| No rerank | 77% | 95% | 96% |
-| `BAAI/bge-reranker-v2-m3` | 92% | 96% | 96% |
-| `cross-encoder/mmarco-mMiniLMv2-L12-H384-v1` | 92% | 96% | 96% |
-
-This set is small and saturated (one question is 1.3 points), its documents
-were short, and the embeddings used for candidate retrieval were a simple
-stand-in, so it cannot separate the two models. Real-world German technical
-text may favour the larger model; measure on your own data if quality matters
-more than latency.
-
-#### Sizing the request for the default model
-
-Latency is dominated by the number of passages and their length, and one
-request already uses all CPUs, so concurrent requests queue. Measured with the
-default model on a 4-vCPU pod (TEI `cpu-1.9.4`, `--max-batch-tokens 2048`),
-75 queries, passages of title plus text cut to the given length (800
-characters are about 190 tokens, 500 about 120), p95 latency of one `/rerank`
-call:
-
-| Passages | 800 chars | 600 chars | 500 chars | 400 chars |
-| -------- | --------- | --------- | --------- | --------- |
-| 40 | 3.2 s | 2.5 s | 2.1 s | 1.8 s |
-| 30 | 2.4 s | 1.9 s | 1.7 s | 1.4 s |
-| 20 | 1.7 s | 1.3 s | 1.1 s | 1.0 s |
-| 15 | 1.3 s | 1.0 s | 0.9 s | 0.7 s |
-
-With four such requests in flight at once, p95 is about 3 to 4 times higher
-(for example 3.0 s for 15 passages of 500 characters, 6.4 s for 40 of 400).
-If the budget is about 1.5 s per request, send at most 15 to 20 passages of
-about 500 characters, and scale out (`reranker.replicaCount`) or add CPU for
-concurrent callers. Reranking a short list also bounds what it can gain: it can
-only reorder the passages it receives.
-
-Memory: the working set stayed at about 1.4 GiB, including four concurrent
-requests of 40 passages, so the chart defaults to a 1.5 GiB request and a 2 GiB
-limit. Inputs longer than the model's 512-token limit are truncated by default
-(`truncate: true`, or the TEI default when the field is omitted); with
-`truncate: false` the request fails with HTTP 422.
-
-The model repository also ships quantised ONNX variants (for example
-`onnx/model_qint8_avx512_vnni.onnx`, `onnx/model_quint8_avx2.onnx`). They are
-not used by this chart and were not measured; they are CPU-specific and
-would need to be baked as `onnx/model.onnx` for TEI.
-
-Licence: `cross-encoder/mmarco-mMiniLMv2-L12-H384-v1` is Apache-2.0,
-`BAAI/bge-reranker-v2-m3` is Apache-2.0 as well (as tagged on the Hugging Face hub); check the model card of
-anything else you select.
-
-To use the higher-quality model (more CPU or a GPU), build or pull the matching
-init image and set:
+## Contributing
-```yaml
-reranker:
- modelName: "BAAI/bge-reranker-v2-m3"
- modelInit:
- image:
- tag: "bge-reranker-v2-m3-" # e.g. bge-reranker-v2-m3-202611.3.1, see the published tags
-```
-
-The init step copies with `--ignore-existing`, so when you change the model on
-an existing release, delete the reranker model PVC (or install with
-`reranker.persistence.enabled=false`) so the old model files are not reused.
-`BAAI/bge-reranker-v2-m3` needs `requests: 4Gi` / `limits: 8Gi` memory with
-`--max-batch-tokens 2048`; the default model needs far less.
-
-To build an init image yourself: `make docker/tei-reranker-model-init
-RERANKER_MODEL_ID=`. If the model repository ships
-`onnx/model.onnx` it is used as-is; otherwise the model is exported with
-Optimum. Only XLM-RoBERTa-style sequence-classification models that TEI
-supports will work.
-
-### Whisper (faster-whisper + large-v3-turbo, int8)
-
-Measured on CPU: ~2.0 GiB RSS peak; roughly real-time speed on 2 threads (a
-3.7 min recording took 3.3 min), faster with more threads.
-
-- `whisper.cpuThreads` must match the CPU limit (CTranslate2 sizes itself by
- the node's cores, not by the cgroup limit).
-- Transcriptions run one at a time; further requests queue in the server. Keep
- the router's `WHISPER_TIMEOUT` (default `300s`) above queue time + runtime,
- e.g. via `router.extraEnv`.
-- Upload limit: `whisper.maxUploadMB` (default 100). Formats: anything FFmpeg
- decodes (wav, mp3, m4a, ogg, flac, webm, mp4, ...).
-- `response_format`: `json` (default), `text`, `verbose_json`, `srt`, `vtt`.
-- Set `whisper.language` (e.g. `de`) to skip language auto-detection; a
- request's own `language` field wins.
-- Vocabulary hints, two mechanisms (both optional, both off by default):
- - `prompt` (form field, OpenAI-compatible) becomes faster-whisper's
- `initial_prompt`. Set a server-side default with `whisper.initialPrompt`
- (`WHISPER_INITIAL_PROMPT`) for clients that send no prompt (e.g. an STT
- integration that only sends `model` and `language`). A request prompt is
- **appended** to the default (``): Whisper weighs the end
- of the prompt most, and an over-long prompt is cut from the front, so the
- request-specific text survives. Whisper reads at most 223 tokens
- (`max_length // 2 - 1`); faster-whisper keeps the last 223 and drops the
- rest. The server additionally caps the combined text at 2000 characters
- (tail kept).
- - `hotwords` (form field, extension; free text such as
- `Kubernetes Grafana Postgres`) is passed to faster-whisper's `hotwords`.
- Server default: `whisper.hotwords` (`WHISPER_HOTWORDS`). A request's
- `hotwords` **replaces** the default (no merging). faster-whisper puts the
- hotwords in front of the previous-text prompt; it ignores them only when
- `prefix` is set, which this server never sets, so `prompt` and `hotwords`
- work together (checked in `faster_whisper/transcribe.py`, `get_prompt`,
- version 1.2.1). Both count against the same 223-token window of their own.
- - Neutral example for `whisper.initialPrompt`:
- `Release notes for Kubernetes, PostgreSQL and Grafana.`
- - Hints raise the odds of a spelling; they do not guarantee it. Keep them
- short and spell terms the way they should appear.
-- Repetition loops on long audio: the sequential decoder conditions every
- window on the previous text and can get stuck repeating itself. The server
- therefore (1) keeps `condition_on_previous_text` off by default
- (`whisper.conditionOnPreviousText`), (2) sends audio longer than
- `whisper.batchThresholdSeconds` (default 35 s, `WHISPER_BATCH_THRESHOLD_S`)
- through faster-whisper's `BatchedInferencePipeline` (VAD-cut chunks decoded
- independently, `whisper.batchSize` = 8, `WHISPER_BATCH_SIZE`), and (3) uses
- faster-whisper's temperature fallback (0.0, 0.2, ... 1.0 with its default
- compression-ratio, log-prob and no-speech thresholds) on the sequential
- path. The batched path always uses VAD and only the first temperature (the
- library does not fall back there). A request's explicit `temperature` form
- field (including `0`) is used as given, without fallback; clients that always
- send `temperature=0` therefore get no fallback on the sequential path.
- Measured locally (FLEURS de, int8, 4 threads, beam 1, VAD): three 55-88 s
- clips WER 3.1 % instead of 42 % (one clip looped to 256 instead of 120
- words) and 56 s of audio in 17.8 s instead of 27.2 s; 40 single clips
- 5.6 % vs 5.2 % (noise); no speed difference on short clips.
-- Warmup: after loading the model the server runs a 2 s synthetic clip through
- VAD and decoder (`WHISPER_WARMUP`, default `true`) so the first real request
- does not pay the initialisation cost. `/health` returns 503 until model and
- warmup are done; a failed warmup is logged and does not block startup.
-- Model source and pins: the image bakes `large-v3-turbo` from
- `dropbox-dash/faster-whisper-large-v3-turbo` at a fixed commit
- (`WHISPER_MODEL_REVISION` in `deploy/Containerfile.whisper`) and verifies
- `model.bin` against its SHA-256; the build fails if either changes. This is
- the repository faster-whisper's `large-v3-turbo` alias
- (`mobiuslabsgmbh/faster-whisper-large-v3-turbo`) redirects to (HTTP 307);
- `Systran/faster-whisper-large-v3-turbo` answers HTTP 401. Model license: MIT.
- Python dependencies are pinned in `deploy/whisper/requirements.txt` (`av`
- must stay on 16.x, newer majors break faster-whisper 1.2.1) and the base
- image by digest.
-- Per-request log line (JSON after the prefix `whisper_request`), e.g.
- `whisper_request {"status":"ok","path":"standard","audio_s":9.0,"prep_s":0.4,"infer_s":7.6,"total_s":8.1,"rtf":0.889,...}`:
- audio duration (`audio_s`, `audio_after_vad_s`), upload, queue wait, decode +
- VAD (`prep_s`) and inference (`infer_s`) time, real-time factor
- (`rtf = (prep_s + infer_s) / audio_s`), model, language, `beam_size`, `vad`, `path` (`standard|batched`),
- `temperature_mode` (`fallback|fixed`),
- file size, and `prompt_chars` / `hotwords_chars` / `prompt_source`
- (`none|default|request|both`). Prompt, hotwords and transcript text are
- never logged.
-- `/v1/audio/translations` (to English) is exposed, but large-v3-turbo is not
- trained for translation; use a non-turbo model for that.
-
-### Scheduling (`affinity`, `topologySpreadConstraints`)
-
-`embedding` and `whisper` accept optional `affinity` and
-`topologySpreadConstraints` (empty by default, so the rendered manifests do not
-change). With `embedding.replicaCount > 1` the replicas can land on one node,
-and then the PodDisruptionBudget does not protect against a node failure.
-Example, soft spreading that never blocks scheduling:
-
-```yaml
-embedding:
- replicaCount: 2
- affinity:
- podAntiAffinity:
- preferredDuringSchedulingIgnoredDuringExecution:
- - weight: 100
- podAffinityTerm:
- topologyKey: kubernetes.io/hostname
- labelSelector:
- matchLabels:
- app.kubernetes.io/component: embedding
- topologySpreadConstraints:
- - maxSkew: 1
- topologyKey: kubernetes.io/hostname
- whenUnsatisfiable: ScheduleAnyway
- labelSelector:
- matchLabels:
- app.kubernetes.io/component: embedding
+```bash
+make test # go test -race ./...
+make lint # golangci-lint
+make test-local # end-to-end test against mock backends
```
-## Environment Variables
-
-| Variable | Default | Description |
-| ---------------------- | ------- | ----------------------------- |
-| `ROUTER_ADDR` | `:8080` | Main listen address |
-| `ROUTER_METRICS_ADDR` | `:9090` | Prometheus metrics address |
-| `ROUTER_READ_TIMEOUT` | `120s` | HTTP read timeout |
-| `ROUTER_WRITE_TIMEOUT` | `300s` | HTTP write timeout |
-| `EMBEDDING_ENABLED` | `false` | Enable embedding backend |
-| `EMBEDDING_URL` | — | Base URL for TEI embedding |
-| `EMBEDDING_TIMEOUT` | `60s` | Per-request timeout |
-| `RERANKER_ENABLED` | `false` | Enable reranker backend |
-| `RERANKER_URL` | — | Base URL for reranker |
-| `RERANKER_TIMEOUT` | `30s` | Per-request timeout |
-| `WHISPER_ENABLED` | `false` | Enable Whisper backend |
-| `WHISPER_URL` | — | Base URL for Whisper |
-| `WHISPER_TIMEOUT` | `300s` | Per-request timeout |
-| `LOG_LEVEL` | `info` | `debug`/`info`/`warn`/`error` |
-| `LOG_FORMAT` | `json` | `json`/`console` |
-
-## Security
-
-- All containers run as non-root (uid 1000 or 65532)
-- `allowPrivilegeEscalation: false`
-- `capabilities: drop: [ALL]`
-- ONNX Runtime exec-stack flag cleared via patchelf
-- No outbound network calls at runtime (`HF_HUB_OFFLINE=1`)
-- Router image uses distroless base (no shell)
+Pull requests are welcome. Releases are cut automatically from `main`
+per component, see [AGENTS.md](AGENTS.md#container-image-versioning).
## License
-[Apache License 2.0](LICENSE)
+[Apache License 2.0](LICENSE). The models have their own licenses: check the model cards of the models you use.
diff --git a/assets/banner.png b/assets/banner.png
new file mode 100644
index 0000000..7eda2ef
Binary files /dev/null and b/assets/banner.png differ
diff --git a/assets/icon-tile.svg b/assets/icon-tile.svg
new file mode 100644
index 0000000..88d9af1
--- /dev/null
+++ b/assets/icon-tile.svg
@@ -0,0 +1,2 @@
+
diff --git a/assets/icon.svg b/assets/icon.svg
new file mode 100644
index 0000000..f817758
--- /dev/null
+++ b/assets/icon.svg
@@ -0,0 +1,2 @@
+
diff --git a/assets/lockup-dark.svg b/assets/lockup-dark.svg
new file mode 100644
index 0000000..e024b62
--- /dev/null
+++ b/assets/lockup-dark.svg
@@ -0,0 +1,2 @@
+
diff --git a/assets/lockup-light.svg b/assets/lockup-light.svg
new file mode 100644
index 0000000..09e7bae
--- /dev/null
+++ b/assets/lockup-light.svg
@@ -0,0 +1,2 @@
+
diff --git a/assets/logo.svg b/assets/logo.svg
new file mode 100644
index 0000000..b862611
--- /dev/null
+++ b/assets/logo.svg
@@ -0,0 +1,2 @@
+
diff --git a/docs/configuration.md b/docs/configuration.md
new file mode 100644
index 0000000..8c8968f
--- /dev/null
+++ b/docs/configuration.md
@@ -0,0 +1,60 @@
+# Configuration
+
+The router is configured exclusively through environment variables (`internal/config/config.go`). With the
+Helm chart the backend variables are set for you; add or override variables with `router.extraEnv`.
+
+At least one backend must be enabled, and every enabled backend needs its `*_URL`, otherwise the router exits
+at startup with a config error.
+
+| Variable | Default | Description |
+| --- | --- | --- |
+| `ROUTER_ADDR` | `:8080` | Main listen address |
+| `ROUTER_METRICS_ADDR` | `:9090` | Prometheus metrics address (`/metrics`) |
+| `ROUTER_READ_TIMEOUT` | `120s` | HTTP read timeout |
+| `ROUTER_WRITE_TIMEOUT` | `300s` | HTTP write timeout |
+| `ROUTER_IDLE_TIMEOUT` | `60s` | HTTP idle timeout |
+| `ROUTER_MAX_REQUEST_SIZE` | `104857600` | Maximum request body in bytes (100 MB) |
+| `LOG_LEVEL` | `info` | `debug`, `info`, `warn` or `error` |
+| `LOG_FORMAT` | `json` | `json` or `console` |
+| `EMBEDDING_ENABLED` | `false` | Enable the embedding backend |
+| `EMBEDDING_URL` | none | Base URL of the TEI embedding server |
+| `EMBEDDING_TIMEOUT` | `60s` | Per-request timeout |
+| `EMBEDDING_MAX_CONCURRENT` | `20` | Requests forwarded at once; more get `503` + `Retry-After` |
+| `RERANKER_ENABLED` | `false` | Enable the reranker backend |
+| `RERANKER_URL` | none | Base URL of the TEI reranker server |
+| `RERANKER_TIMEOUT` | `30s` | Per-request timeout |
+| `RERANKER_MAX_CONCURRENT` | `20` | Requests forwarded at once; more get `503` + `Retry-After` |
+| `WHISPER_ENABLED` | `false` | Enable the Whisper backend |
+| `WHISPER_URL` | none | Base URL of the Whisper server |
+| `WHISPER_TIMEOUT` | `300s` | Per-request timeout |
+| `WHISPER_MAX_CONCURRENT` | `8` | Requests forwarded at once; more get `503` + `Retry-After` |
+
+A value of `0` or below for a `*_MAX_CONCURRENT` variable disables the bound for that backend.
+
+## Backpressure
+
+When a backend is at its concurrency bound the router answers immediately with
+`503 Service Unavailable`, a `Retry-After: 3` header and `{"error":"backend_overloaded", ...}` instead of
+queuing the request. Clients should back off and retry. Each rejection increments
+`inference_router_rejected_total{backend}`.
+
+## Health endpoints
+
+| Path | Port | Behaviour |
+| --- | --- | --- |
+| `/healthz`, `/readyz` | 8080 | Backends are probed (`GET /health`) every 15 seconds. The answer is always HTTP 200; the JSON body says `ok` or `degraded` and lists each backend as `ok`, `unhealthy` or `disabled`. A failing backend does not take the router out of rotation. |
+| `/health` | 8080 | Router liveness only: `{"status":"ok"}` |
+| `/metrics` | 9090 | Prometheus metrics |
+
+## Metrics
+
+| Metric | Type | Labels |
+| --- | --- | --- |
+| `inference_router_requests_total` | counter | `backend`, `path`, `status` |
+| `inference_router_request_duration_seconds` | histogram | `backend`, `path` |
+| `inference_router_request_size_bytes` | histogram | `backend` |
+| `inference_router_response_size_bytes` | histogram | `backend` |
+| `inference_router_active_requests` | gauge | `backend` |
+| `inference_router_backend_up` | gauge | `backend` |
+| `inference_router_upstream_errors_total` | counter | `backend`, `error_type` |
+| `inference_router_rejected_total` | counter | `backend` |
diff --git a/docs/tuning.md b/docs/tuning.md
new file mode 100644
index 0000000..183d1f2
--- /dev/null
+++ b/docs/tuning.md
@@ -0,0 +1,220 @@
+# CPU tuning and sizing
+
+Measurements and settings for running the stack on plain CPU. The Helm values referenced here are in
+[`deploy/helm/values.yaml`](../deploy/helm/values.yaml); see also [Configuration](configuration.md).
+
+## Embedding (TEI + BGE-M3)
+
+| `--max-batch-tokens` | RSS (fp32) | Recommendation |
+| -------------------- | ---------- | -------------------- |
+| 4096 | 3-6Gi | **CPU default** |
+| 8192 | 6-10Gi | May OOM on 8Gi limit |
+| 16384 | 10-16Gi | GPU-only |
+
+## Reranker model choice
+
+The reranker model is baked into the `tei-reranker-model-init` image and
+selected by its tag (`reranker.modelInit.image.tag`). Both models run on the
+same TEI runtime with the ONNX Runtime backend.
+
+Measured with TEI `cpu-1.9.4`, limited to 4 CPUs, `--max-batch-tokens 2048`,
+one `/rerank` request with one query and 20 documents of 800 characters each
+(German/English mix), 4 runs per model. The CPU was an older AVX2 machine, so
+only the ratios carry over.
+
+| Model | Format | Parameters | Latency (20 docs) |
+| ------------------------------------------------------ | ------------------------- | ---------- | ----------------- |
+| `cross-encoder/mmarco-mMiniLMv2-L12-H384-v1` (default) | fp32 ONNX | 118M | 0.76 to 0.85 s |
+| `BAAI/bge-reranker-v2-m3` | fp32 ONNX | 568M | 8.3 to 11.9 s |
+| `BAAI/bge-reranker-v2-m3` | int8 ONNX | 568M | 7.9 to 8.2 s |
+| `Alibaba-NLP/gte-multilingual-reranker-base` | safetensors (candle path) | n/a | 11.1 to 12.1 s |
+
+That is about 0.04 s per document pair for the default against roughly 0.4 to
+0.5 s for `bge-reranker-v2-m3`, so reranking 40 candidates with the larger
+model does not fit an interactive latency budget on 4 CPUs.
+
+Quality, on a downstream 75-question German/English relevance set (hybrid
+retrieval, rerank the top 20 candidates, 800 characters per document):
+
+| Setup | hit@1 | hit@5 | hit@10 |
+| ------------------------------------------ | ----- | ----- | ------ |
+| No rerank | 77% | 95% | 96% |
+| `BAAI/bge-reranker-v2-m3` | 92% | 96% | 96% |
+| `cross-encoder/mmarco-mMiniLMv2-L12-H384-v1` | 92% | 96% | 96% |
+
+This set is small and saturated (one question is 1.3 points), its documents
+were short, and the embeddings used for candidate retrieval were a simple
+stand-in, so it cannot separate the two models. Real-world German technical
+text may favour the larger model; measure on your own data if quality matters
+more than latency.
+
+### Sizing the request for the default model
+
+Latency is dominated by the number of passages and their length, and one
+request already uses all CPUs, so concurrent requests queue. Measured with the
+default model on a 4-vCPU pod (TEI `cpu-1.9.4`, `--max-batch-tokens 2048`),
+75 queries, passages of title plus text cut to the given length (800
+characters are about 190 tokens, 500 about 120), p95 latency of one `/rerank`
+call:
+
+| Passages | 800 chars | 600 chars | 500 chars | 400 chars |
+| -------- | --------- | --------- | --------- | --------- |
+| 40 | 3.2 s | 2.5 s | 2.1 s | 1.8 s |
+| 30 | 2.4 s | 1.9 s | 1.7 s | 1.4 s |
+| 20 | 1.7 s | 1.3 s | 1.1 s | 1.0 s |
+| 15 | 1.3 s | 1.0 s | 0.9 s | 0.7 s |
+
+With four such requests in flight at once, p95 is about 3 to 4 times higher
+(for example 3.0 s for 15 passages of 500 characters, 6.4 s for 40 of 400).
+If the budget is about 1.5 s per request, send at most 15 to 20 passages of
+about 500 characters, and scale out (`reranker.replicaCount`) or add CPU for
+concurrent callers. Reranking a short list also bounds what it can gain: it can
+only reorder the passages it receives.
+
+Memory: the working set stayed at about 1.4 GiB, including four concurrent
+requests of 40 passages, so the chart defaults to a 1.5 GiB request and a 2 GiB
+limit. Inputs longer than the model's 512-token limit are truncated by default
+(`truncate: true`, or the TEI default when the field is omitted); with
+`truncate: false` the request fails with HTTP 422.
+
+The model repository also ships quantised ONNX variants (for example
+`onnx/model_qint8_avx512_vnni.onnx`, `onnx/model_quint8_avx2.onnx`). They are
+not used by this chart and were not measured; they are CPU-specific and
+would need to be baked as `onnx/model.onnx` for TEI.
+
+Licence: `cross-encoder/mmarco-mMiniLMv2-L12-H384-v1` is Apache-2.0,
+`BAAI/bge-reranker-v2-m3` is Apache-2.0 as well (as tagged on the Hugging Face hub); check the model card of
+anything else you select.
+
+To use the higher-quality model (more CPU or a GPU), build or pull the matching
+init image and set:
+
+```yaml
+reranker:
+ modelName: "BAAI/bge-reranker-v2-m3"
+ modelInit:
+ image:
+ tag: "bge-reranker-v2-m3-" # e.g. bge-reranker-v2-m3-202611.3.1, see the published tags
+```
+
+The init step copies with `--ignore-existing`, so when you change the model on
+an existing release, delete the reranker model PVC (or install with
+`reranker.persistence.enabled=false`) so the old model files are not reused.
+`BAAI/bge-reranker-v2-m3` needs `requests: 4Gi` / `limits: 8Gi` memory with
+`--max-batch-tokens 2048`; the default model needs far less.
+
+To build an init image yourself: `make docker/tei-reranker-model-init
+RERANKER_MODEL_ID=`. If the model repository ships
+`onnx/model.onnx` it is used as-is; otherwise the model is exported with
+Optimum. Only XLM-RoBERTa-style sequence-classification models that TEI
+supports will work.
+
+## Whisper (faster-whisper + large-v3-turbo, int8)
+
+Measured on CPU: ~2.0 GiB RSS peak; roughly real-time speed on 2 threads (a
+3.7 min recording took 3.3 min), faster with more threads.
+
+- `whisper.cpuThreads` must match the CPU limit (CTranslate2 sizes itself by
+ the node's cores, not by the cgroup limit).
+- Transcriptions run one at a time; further requests queue in the server. Keep
+ the router's `WHISPER_TIMEOUT` (default `300s`) above queue time + runtime,
+ e.g. via `router.extraEnv`.
+- Upload limit: `whisper.maxUploadMB` (default 100). Formats: anything FFmpeg
+ decodes (wav, mp3, m4a, ogg, flac, webm, mp4, ...).
+- `response_format`: `json` (default), `text`, `verbose_json`, `srt`, `vtt`.
+- Set `whisper.language` (e.g. `de`) to skip language auto-detection; a
+ request's own `language` field wins.
+- Vocabulary hints, two mechanisms (both optional, both off by default):
+ - `prompt` (form field, OpenAI-compatible) becomes faster-whisper's
+ `initial_prompt`. Set a server-side default with `whisper.initialPrompt`
+ (`WHISPER_INITIAL_PROMPT`) for clients that send no prompt (e.g. an STT
+ integration that only sends `model` and `language`). A request prompt is
+ **appended** to the default (``): Whisper weighs the end
+ of the prompt most, and an over-long prompt is cut from the front, so the
+ request-specific text survives. Whisper reads at most 223 tokens
+ (`max_length // 2 - 1`); faster-whisper keeps the last 223 and drops the
+ rest. The server additionally caps the combined text at 2000 characters
+ (tail kept).
+ - `hotwords` (form field, extension; free text such as
+ `Kubernetes Grafana Postgres`) is passed to faster-whisper's `hotwords`.
+ Server default: `whisper.hotwords` (`WHISPER_HOTWORDS`). A request's
+ `hotwords` **replaces** the default (no merging). faster-whisper puts the
+ hotwords in front of the previous-text prompt; it ignores them only when
+ `prefix` is set, which this server never sets, so `prompt` and `hotwords`
+ work together (checked in `faster_whisper/transcribe.py`, `get_prompt`,
+ version 1.2.1). Both count against the same 223-token window of their own.
+ - Neutral example for `whisper.initialPrompt`:
+ `Release notes for Kubernetes, PostgreSQL and Grafana.`
+ - Hints raise the odds of a spelling; they do not guarantee it. Keep them
+ short and spell terms the way they should appear.
+- Repetition loops on long audio: the sequential decoder conditions every
+ window on the previous text and can get stuck repeating itself. The server
+ therefore (1) keeps `condition_on_previous_text` off by default
+ (`whisper.conditionOnPreviousText`), (2) sends audio longer than
+ `whisper.batchThresholdSeconds` (default 35 s, `WHISPER_BATCH_THRESHOLD_S`)
+ through faster-whisper's `BatchedInferencePipeline` (VAD-cut chunks decoded
+ independently, `whisper.batchSize` = 8, `WHISPER_BATCH_SIZE`), and (3) uses
+ faster-whisper's temperature fallback (0.0, 0.2, ... 1.0 with its default
+ compression-ratio, log-prob and no-speech thresholds) on the sequential
+ path. The batched path always uses VAD and only the first temperature (the
+ library does not fall back there). A request's explicit `temperature` form
+ field (including `0`) is used as given, without fallback; clients that always
+ send `temperature=0` therefore get no fallback on the sequential path.
+ Measured locally (FLEURS de, int8, 4 threads, beam 1, VAD): three 55-88 s
+ clips WER 3.1 % instead of 42 % (one clip looped to 256 instead of 120
+ words) and 56 s of audio in 17.8 s instead of 27.2 s; 40 single clips
+ 5.6 % vs 5.2 % (noise); no speed difference on short clips.
+- Warmup: after loading the model the server runs a 2 s synthetic clip through
+ VAD and decoder (`WHISPER_WARMUP`, default `true`) so the first real request
+ does not pay the initialisation cost. `/health` returns 503 until model and
+ warmup are done; a failed warmup is logged and does not block startup.
+- Model source and pins: the image bakes `large-v3-turbo` from
+ `dropbox-dash/faster-whisper-large-v3-turbo` at a fixed commit
+ (`WHISPER_MODEL_REVISION` in `deploy/Containerfile.whisper`) and verifies
+ `model.bin` against its SHA-256; the build fails if either changes. This is
+ the repository faster-whisper's `large-v3-turbo` alias
+ (`mobiuslabsgmbh/faster-whisper-large-v3-turbo`) redirects to (HTTP 307);
+ `Systran/faster-whisper-large-v3-turbo` answers HTTP 401. Model license: MIT.
+ Python dependencies are pinned in `deploy/whisper/requirements.txt` (`av`
+ must stay on 16.x, newer majors break faster-whisper 1.2.1) and the base
+ image by digest.
+- Per-request log line (JSON after the prefix `whisper_request`), e.g.
+ `whisper_request {"status":"ok","path":"standard","audio_s":9.0,"prep_s":0.4,"infer_s":7.6,"total_s":8.1,"rtf":0.889,...}`:
+ audio duration (`audio_s`, `audio_after_vad_s`), upload, queue wait, decode +
+ VAD (`prep_s`) and inference (`infer_s`) time, real-time factor
+ (`rtf = (prep_s + infer_s) / audio_s`), model, language, `beam_size`, `vad`, `path` (`standard|batched`),
+ `temperature_mode` (`fallback|fixed`),
+ file size, and `prompt_chars` / `hotwords_chars` / `prompt_source`
+ (`none|default|request|both`). Prompt, hotwords and transcript text are
+ never logged.
+- `/v1/audio/translations` (to English) is exposed, but large-v3-turbo is not
+ trained for translation; use a non-turbo model for that.
+
+## Scheduling (`affinity`, `topologySpreadConstraints`)
+
+`embedding` and `whisper` accept optional `affinity` and
+`topologySpreadConstraints` (empty by default, so the rendered manifests do not
+change). With `embedding.replicaCount > 1` the replicas can land on one node,
+and then the PodDisruptionBudget does not protect against a node failure.
+Example, soft spreading that never blocks scheduling:
+
+```yaml
+embedding:
+ replicaCount: 2
+ affinity:
+ podAntiAffinity:
+ preferredDuringSchedulingIgnoredDuringExecution:
+ - weight: 100
+ podAffinityTerm:
+ topologyKey: kubernetes.io/hostname
+ labelSelector:
+ matchLabels:
+ app.kubernetes.io/component: embedding
+ topologySpreadConstraints:
+ - maxSkew: 1
+ topologyKey: kubernetes.io/hostname
+ whenUnsatisfiable: ScheduleAnyway
+ labelSelector:
+ matchLabels:
+ app.kubernetes.io/component: embedding
+```