diff --git a/AGENTS.md b/AGENTS.md index b521405..14a4165 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -101,7 +101,7 @@ Key details: ### Whisper (Standalone) -Model is baked directly into the image at build time. No init container, no PVC needed. The image runs `deploy/whisper/server.py`, which speaks the OpenAI audio API itself (the router forwards paths unchanged). `/health` does not queue behind inference. Pin `av` (see Containerfile) — newer PyAV majors break faster-whisper. Each request logs one JSON line (`whisper_request`, timings, RTF; never prompt/transcript text). `WHISPER_INITIAL_PROMPT` / `WHISPER_HOTWORDS` are server-side defaults (see README). Tests: `pytest deploy/whisper` (stubbed model). +Model is baked directly into the image at build time. No init container, no PVC needed. The image runs `deploy/whisper/server.py`, which speaks the OpenAI audio API itself (the router forwards paths unchanged). `/health` does not queue behind inference. Pin `av` (see Containerfile) — newer PyAV majors break faster-whisper. Each request logs one JSON line (`whisper_request`, timings, RTF; never prompt/transcript text). `WHISPER_INITIAL_PROMPT` / `WHISPER_HOTWORDS` are server-side defaults (see docs/tuning.md). Tests: `pytest deploy/whisper` (stubbed model). ### Router (Go Proxy) @@ -137,7 +137,7 @@ make docker/whisper # whisper ## Tech Stack & Constraints -- **Go 1.24+** — no generics workarounds needed +- **Go 1.26** (see `go.mod`) - **`net/http/httptest`** for all tests — no external test frameworks (no testify, no gomock) - **`go.uber.org/zap`** for structured logging - **`github.com/prometheus/client_golang`** for metrics @@ -228,7 +228,7 @@ It exposes `/health`, `/v1/models`, and the backend-specific inference endpoints - `promauto` metrics panic on double-registration. If you add a new metric, add it to `internal/metrics/metrics.go` as a package-level var, not inside a function. - `health.Checker` must be initialised with `health.New(cfg.EnabledBackends(), log)` — passing `cfg.ActiveBackends()` will include disabled backends and skew the health response. - The `statusWriter` wrapper in `proxy.go` must set `status = 200` lazily (in `Write`) to handle the case where `WriteHeader` is never called explicitly. -- TEI does **not** support reranking — the reranker uses FlagEmbedding, not TEI. +- Reranking runs on TEI (ONNX, see `docs/ADR-001-tei-for-reranking.md`). `deploy/Containerfile.reranker-server` and `deploy/reranker/` are the legacy FlagEmbedding server and are not released any more. - Some CSI volumes don't support timestamp preservation — always use `rsync -rlgoDv`, never `rsync -a`. - Pod `fsGroup: 1000` is required for init containers to write to PVCs (CSI provisions volumes as root). @@ -250,8 +250,11 @@ It exposes `/health`, `/v1/models`, and the backend-specific inference endpoints | `EMBEDDING_URL` | — | Base URL for TEI embedding | | `EMBEDDING_TIMEOUT` | `60s` | Per-request timeout | | `RERANKER_ENABLED` | `false` | Enable reranker backend | -| `RERANKER_URL` | — | Base URL for FlagEmbedding reranker | +| `RERANKER_URL` | — | Base URL for the TEI reranker | | `RERANKER_TIMEOUT` | `30s` | Per-request timeout | | `WHISPER_ENABLED` | `false` | Enable Whisper backend | | `WHISPER_URL` | — | Base URL for Whisper | | `WHISPER_TIMEOUT` | `300s` | Per-request timeout | +| `EMBEDDING_MAX_CONCURRENT` / `RERANKER_MAX_CONCURRENT` / `WHISPER_MAX_CONCURRENT` | `20` / `20` / `8` | Concurrency bound per backend; above it the router answers `503` + `Retry-After` | + +User-facing documentation: `README.md`, `docs/configuration.md`, `docs/tuning.md`. diff --git a/README.md b/README.md index bf115c8..d508a68 100644 --- a/README.md +++ b/README.md @@ -1,124 +1,150 @@ -# inference-stack - -[![Go Tests](https://github.com/layer87-labs/inference-stack/actions/workflows/ci.yml/badge.svg)](https://github.com/layer87-labs/inference-stack/actions/workflows/ci.yml) -[![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE) - -Unified CPU-based inference stack — serving embeddings, reranking and transcription behind a single OpenAI-compatible API. +

+ inference-stack: embeddings, reranking and speech-to-text on plain CPU +

-## Components +

+ Latest router release + CI + Go version + License: Apache-2.0 +

-| Component | Runtime | Model | Purpose | -| -------------------- | ---------------------------------------------------------------------- | ----------------------- | --------------------------------------- | -| **inference-router** | Go | — | Unified OpenAI-compatible reverse proxy | -| **embedding** | [TEI](https://github.com/huggingface/text-embeddings-inference) (Rust) | BAAI/bge-m3 | Dense + sparse embeddings | -| **reranker** | [TEI](https://github.com/huggingface/text-embeddings-inference) (Rust, ONNX) | mmarco-mMiniLMv2-L12-H384-v1 (default), bge-reranker-v2-m3 (option) | Cross-encoder reranking | -| **whisper** | [faster-whisper](https://github.com/SYSTRAN/faster-whisper) (Python) | large-v3-turbo | Audio transcription (ASR) | +# inference-stack -All backends are disabled by default and enabled selectively via Helm values or env vars. +**inference-stack** is a small OpenAI-compatible router in front of CPU-only inference backends. One endpoint +gives you **embeddings** (`BAAI/bge-m3` via [TEI](https://github.com/huggingface/text-embeddings-inference)), +**reranking** (a cross-encoder via TEI/ONNX) and **speech-to-text** (Whisper `large-v3-turbo` via +[faster-whisper](https://github.com/SYSTRAN/faster-whisper)). It runs on plain CPU and RAM. No GPU needed. + +**Why it exists.** Search, RAG and transcription should not need a GPU budget or a pile of glue code. We +([Layer87](https://github.com/layer87-labs)) needed exactly this in our own applications, built it, run it ourselves +and open-sourced it. It ships with hardened container images, a Helm chart and signed releases. + +## Highlights + +- **One API, three capabilities.** `/v1/embeddings`, `/v1/rerank`, `/v1/audio/transcriptions`, plus a merged `/v1/models` + that also reports the backends' request limits. +- **CPU only.** Sized and measured on CPU; the numbers (latency per rerank request, memory, real-time factor) are in + [docs/tuning.md](docs/tuning.md). +- **Backends are optional.** All are off by default; enable only what you need. +- **Backpressure instead of pile-ups.** Per-backend concurrency bound; above it the router answers `503` with + `Retry-After` right away ([details](docs/configuration.md#backpressure)). +- **Observable.** Prometheus metrics on a separate port, JSON logs, optional `ServiceMonitor` and `NetworkPolicy`. +- **Signed images.** Released images are signed keyless with [cosign](https://github.com/sigstore/cosign) via GitHub + OIDC ([verify](#security)). +- **Hardened by default.** Non-root containers, no privilege escalation, all capabilities dropped, `RuntimeDefault` + seccomp, executable-stack flag cleared on the ONNX Runtime libraries of the TEI base image. +- **Offline at runtime.** Models are baked into images (`HF_HUB_OFFLINE=1`); model volumes are mounted read-only. +- **Pinned build inputs.** The Whisper image pins its base image by digest, all Python dependencies, the model + revision and the model file's SHA-256; the build fails if any of them changes. +- **Per-component CalVer releases.** Router, TEI images, models, Whisper and the Helm chart are released + independently with [relctl](https://github.com/layer87-labs/relctl). -## Quick Start +## Architecture -### Local Development +``` + ┌──────────────────────────────────────┐ + your app ──HTTP──▶ │ inference-router (Go, :8080) │ ─▶ :9090/metrics + OpenAI-style API └───────┬───────────────┬──────────────┘ + │ │ │ + /v1/embeddings /v1/rerank /v1/audio/transcriptions + ▼ ▼ ▼ + ┌────────────┐ ┌────────────┐ ┌──────────────────┐ + │ TEI │ │ TEI (ONNX) │ │ faster-whisper │ + │ bge-m3 │ │ cross-enc. │ │ large-v3-turbo │ + └─────▲──────┘ └─────▲──────┘ └──────────────────┘ + │ /model (ro) │ /model (ro) model baked into image + model-init (rsync) model-init (rsync) +``` -```bash -# Run unit tests -make test +The router is a pure reverse proxy: it routes by path, bounds concurrency, probes backend health and exports +metrics. It contains no model logic. The TEI backends read their model from a volume that an init container +fills from a model baked into its image (idempotent, `rsync --ignore-existing`); Whisper has the model in the +image itself. -# Full end-to-end test (starts mock backends + router) -make test-local -``` +## Quick start -### Kubernetes (Helm) +**Try it locally in a minute** (mock backends, no models, needs Go): ```bash -helm install inference-stack ./deploy/helm \ - --namespace ai \ - --set embedding.enabled=true \ - --set reranker.enabled=true \ - --set router.enabled=true +make test-local # builds, starts mock backends + router, runs the end-to-end checks ``` -## API - -All endpoints are OpenAI-compatible: +Or start the pieces by hand and call the API yourself: ```bash -# Embeddings -curl http://localhost:8080/v1/embeddings \ - -H "Content-Type: application/json" \ - -d '{"model":"BAAI/bge-m3","input":"Hello world"}' - -# Reranking -curl http://localhost:8080/v1/rerank \ - -H "Content-Type: application/json" \ - -d '{"query":"What is Go?","documents":["Go is a language","Python is a language"]}' - -# Transcription -curl http://localhost:8080/v1/audio/transcriptions \ - -F model=whisper-large-v3-turbo \ - -F file=@audio.mp3 +make build +MOCK_TYPE=embedding MOCK_PORT=9101 build/bin/mock-backend & +EMBEDDING_ENABLED=true EMBEDDING_URL=http://localhost:9101 build/bin/inference-router & -# Model list (merged from all enabled backends) -curl http://localhost:8080/v1/models +curl http://localhost:8080/v1/embeddings -H 'Content-Type: application/json' \ + -d '{"model":"BAAI/bge-m3","input":"Hello world"}' ``` -`/v1/models` also reports the request limits of the embedding and reranker -backends (`limits.max_batch_tokens`, `limits.max_client_batch_size`, -`limits.max_input_length`), read from the running backend, so clients can size -batches without trial and error. The model id it advertises -(`BAAI/bge-m3`) can be sent back as `model` on `/v1/embeddings`; the router -strips it before forwarding, because TEI only accepts its internal model path -or no model at all. +**Run it for real on Kubernetes** with the Helm chart from the OCI registry. Pick a chart version from the +[releases](https://github.com/layer87-labs/inference-stack/releases) (tags look like `chart/`): -## Architecture +```yaml +# values.yaml: everything is off by default, enable what you need +router: { enabled: true } +embedding: { enabled: true } +reranker: { enabled: true } +whisper: { enabled: true } +``` -### Init Container Pattern (Embedding + Reranker) +```bash +helm install stack oci://ghcr.io/layer87-labs/charts/inference-stack \ + --version --namespace ai --create-namespace -f values.yaml -``` -┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐ -│ model-init │────▶│ /model (PVC) │────▶│ main container │ -│ (bakes model) │ │ shared volume │ │ (reads model) │ -└─────────────────┘ └──────────────────┘ └─────────────────┘ +kubectl -n ai port-forward svc/stack-inference-stack-router 8080:8080 ``` -1. **Init container** copies baked model to PVC via `rsync --ignore-existing` -2. **Main container** reads model from PVC (mounted read-only) -3. PVC persists model across restarts (idempotent — no re-copy on restart) +The first start takes a while: the init containers copy the models to the volume. Sizing per backend (CPU, memory, +batch tokens) is in [docs/tuning.md](docs/tuning.md); all chart options are commented in +[`deploy/helm/values.yaml`](deploy/helm/values.yaml). -### Whisper (Standalone) +## API -Model baked into image at build time. No init container or PVC needed. The -image runs `deploy/whisper/server.py` (faster-whisper behind a small FastAPI -app) which implements `POST /v1/audio/transcriptions` and -`/v1/audio/translations` natively, so the router forwards requests unchanged. -`/health` is answered independently of inference and stays responsive while a -transcription is running. +| Method and path | Backend | Notes | +| --- | --- | --- | +| `POST /v1/embeddings` | embedding | OpenAI-style. The advertised model id `BAAI/bge-m3` may be sent as `model`. | +| `POST /embed`, `POST /embed_sparse` | embedding | TEI native (dense and sparse) | +| `POST /v1/rerank`, `POST /rerank` | reranker | `/v1/rerank` is rewritten to TEI's `/rerank` | +| `POST /v1/audio/transcriptions` | whisper | multipart; `json`, `text`, `verbose_json`, `srt`, `vtt` | +| `POST /v1/audio/translations` | whisper | exposed, but `large-v3-turbo` is not trained for translation | +| `GET /v1/models` | all enabled | merged list incl. `limits` of embedding and reranker | +| `GET /healthz`, `GET /readyz` | router | per-backend status, always HTTP 200 (`ok` or `degraded`) | +| `GET :9090/metrics` | router | Prometheus | -### Router (Go Proxy) +```bash +curl http://localhost:8080/v1/rerank -H 'Content-Type: application/json' \ + -d '{"query":"What is Go?","texts":["Go is a language","Python is a language"]}' -Pure reverse proxy — routes requests to backends based on path prefix. Zero model logic. +curl http://localhost:8080/v1/audio/transcriptions -F model=whisper-large-v3-turbo -F file=@audio.mp3 +``` -## Container Images +A disabled backend answers `503 backend_disabled`; an unreachable one `502 upstream_error`; one at its +concurrency bound `503 backend_overloaded` with `Retry-After`. -All images are published to `ghcr.io/layer87-labs/`: +## Components and images -| Image | Description | -| --------------------- | -------------------------------------- | -| `inference-router` | Go reverse proxy | -| `tei-base` | Hardened TEI base (patchelf, non-root) | -| `tei-runtime` | TEI runtime (model via volume) | -| `tei-model-init` | BGE-M3 model baked, copies to volume | -| `tei-reranker-model-init` | Reranker model as ONNX (default: mmarco-mMiniLMv2-L12-H384-v1; option: BGE-reranker-v2-m3) | -| `reranker-server` | FlagEmbedding HTTP server | -| `whisper` | Whisper ASR server, model baked in | +| Component | Image(s) in `ghcr.io/layer87-labs/` | Purpose | +| --- | --- | --- | +| router | `inference-router` | Go reverse proxy on a minimal static base image (no shell) | +| tei | `tei-base`, `tei-runtime` | Hardened TEI CPU base and runtime (model via volume) | +| embedding-model | `tei-model-init` | `BAAI/bge-m3` baked in, copies to the volume | +| reranker-model | `tei-reranker-model-init` | Reranker as ONNX: `cross-encoder/mmarco-mMiniLMv2-L12-H384-v1` (default) or `BAAI/bge-reranker-v2-m3` | +| whisper | `whisper` | faster-whisper server, `large-v3-turbo` baked in | +| chart | `charts/inference-stack` (OCI) | Helm chart | -All images run as non-root with no privilege escalation. +Why the default reranker is the small model: on 4 CPUs it needs 0.76 to 0.85 s for 20 documents of 800 +characters, against 8.3 to 11.9 s for `bge-reranker-v2-m3`. Method and quality numbers are in +[docs/tuning.md](docs/tuning.md#reranker-model-choice). -### Verify image signatures +## Security -Images built by the release workflows are signed with -[cosign](https://github.com/sigstore/cosign) keyless via GitHub OIDC. Verify by -digest against the exact workflow identity: +Container images from the release pipeline are signed keyless with cosign. Verify by digest against the exact +workflow identity: ```bash IMAGE=ghcr.io/layer87-labs/inference-router @@ -130,290 +156,33 @@ cosign verify \ "$IMAGE@$DIGEST" ``` -`release.yml` signs `inference-router`, `tei-base` and `tei-runtime`. -`release-models.yml` signs `tei-model-init`, `tei-reranker-model-init` and -`whisper`; for those, use the identity -`https://github.com/layer87-labs/inference-stack/.github/workflows/release-models.yml@refs/heads/main` -(for manual runs, the ref is the branch the workflow was started from). -Images published before signing was added are unsigned. +Model and Whisper images published earlier were signed by the separate `release-models.yml` workflow (same +command, that workflow file as identity). Images published before signing was added are +unsigned. The Helm chart is not signed. -## Build +Runtime posture of the chart defaults: non-root users (uid 65532 for the router, 1000 for the rest), +`allowPrivilegeEscalation: false`, `capabilities.drop: [ALL]`, `seccompProfile: RuntimeDefault`, a read-only root +filesystem for the router, read-only model mounts, and no outbound network calls needed at runtime. An optional +`NetworkPolicy` restricts the router. -```bash -# Go binary -make build +## Documentation -# All container images -make docker - -# Push to registry -make push REGISTRY=ghcr.io/layer87-labs VERSION=202611.3.1 -``` +- [docs/configuration.md](docs/configuration.md): environment variables, backpressure, health endpoints, metrics +- [docs/tuning.md](docs/tuning.md): CPU sizing, reranker model choice and measurements, Whisper options, scheduling +- [docs/ADR-001-tei-for-reranking.md](docs/ADR-001-tei-for-reranking.md): why reranking runs on TEI +- [AGENTS.md](AGENTS.md): repository layout, build and test commands, release components -## Releases - -The router, TEI images, embedding model, reranker models, Whisper and the Helm chart are released -independently as [relctl](https://github.com/layer87-labs/relctl) components with CalVer versions -(`YYYYMM.D.N`, tag `/`). Only the parts whose files changed are built on a merge to -`main`. See `AGENTS.md` for the component table and image tag formats. - -## Metrics - -Prometheus metrics on `:9090/metrics`: - -- `inference_router_requests_total` — total requests by backend/path/status -- `inference_router_request_duration_seconds` — latency histogram -- `inference_router_active_requests` — in-flight requests -- `inference_router_backend_up` — backend health (0/1) -- `inference_router_upstream_errors_total` — upstream error counts - -## CPU Tuning - -### Embedding (TEI + BGE-M3) - -| `--max-batch-tokens` | RSS (fp32) | Recommendation | -| -------------------- | ---------- | -------------------- | -| 4096 | 3-6Gi | **CPU default** | -| 8192 | 6-10Gi | May OOM on 8Gi limit | -| 16384 | 10-16Gi | GPU-only | - -### Reranker model choice - -The reranker model is baked into the `tei-reranker-model-init` image and -selected by its tag (`reranker.modelInit.image.tag`). Both models run on the -same TEI runtime with the ONNX Runtime backend. - -Measured with TEI `cpu-1.9.4`, limited to 4 CPUs, `--max-batch-tokens 2048`, -one `/rerank` request with one query and 20 documents of 800 characters each -(German/English mix), 4 runs per model. The CPU was an older AVX2 machine, so -only the ratios carry over. - -| Model | Format | Parameters | Latency (20 docs) | -| ------------------------------------------------------ | ------------------------- | ---------- | ----------------- | -| `cross-encoder/mmarco-mMiniLMv2-L12-H384-v1` (default) | fp32 ONNX | 118M | 0.76 to 0.85 s | -| `BAAI/bge-reranker-v2-m3` | fp32 ONNX | 568M | 8.3 to 11.9 s | -| `BAAI/bge-reranker-v2-m3` | int8 ONNX | 568M | 7.9 to 8.2 s | -| `Alibaba-NLP/gte-multilingual-reranker-base` | safetensors (candle path) | n/a | 11.1 to 12.1 s | - -That is about 0.04 s per document pair for the default against roughly 0.4 to -0.5 s for `bge-reranker-v2-m3`, so reranking 40 candidates with the larger -model does not fit an interactive latency budget on 4 CPUs. - -Quality, on a downstream 75-question German/English relevance set (hybrid -retrieval, rerank the top 20 candidates, 800 characters per document): - -| Setup | hit@1 | hit@5 | hit@10 | -| ------------------------------------------ | ----- | ----- | ------ | -| No rerank | 77% | 95% | 96% | -| `BAAI/bge-reranker-v2-m3` | 92% | 96% | 96% | -| `cross-encoder/mmarco-mMiniLMv2-L12-H384-v1` | 92% | 96% | 96% | - -This set is small and saturated (one question is 1.3 points), its documents -were short, and the embeddings used for candidate retrieval were a simple -stand-in, so it cannot separate the two models. Real-world German technical -text may favour the larger model; measure on your own data if quality matters -more than latency. - -#### Sizing the request for the default model - -Latency is dominated by the number of passages and their length, and one -request already uses all CPUs, so concurrent requests queue. Measured with the -default model on a 4-vCPU pod (TEI `cpu-1.9.4`, `--max-batch-tokens 2048`), -75 queries, passages of title plus text cut to the given length (800 -characters are about 190 tokens, 500 about 120), p95 latency of one `/rerank` -call: - -| Passages | 800 chars | 600 chars | 500 chars | 400 chars | -| -------- | --------- | --------- | --------- | --------- | -| 40 | 3.2 s | 2.5 s | 2.1 s | 1.8 s | -| 30 | 2.4 s | 1.9 s | 1.7 s | 1.4 s | -| 20 | 1.7 s | 1.3 s | 1.1 s | 1.0 s | -| 15 | 1.3 s | 1.0 s | 0.9 s | 0.7 s | - -With four such requests in flight at once, p95 is about 3 to 4 times higher -(for example 3.0 s for 15 passages of 500 characters, 6.4 s for 40 of 400). -If the budget is about 1.5 s per request, send at most 15 to 20 passages of -about 500 characters, and scale out (`reranker.replicaCount`) or add CPU for -concurrent callers. Reranking a short list also bounds what it can gain: it can -only reorder the passages it receives. - -Memory: the working set stayed at about 1.4 GiB, including four concurrent -requests of 40 passages, so the chart defaults to a 1.5 GiB request and a 2 GiB -limit. Inputs longer than the model's 512-token limit are truncated by default -(`truncate: true`, or the TEI default when the field is omitted); with -`truncate: false` the request fails with HTTP 422. - -The model repository also ships quantised ONNX variants (for example -`onnx/model_qint8_avx512_vnni.onnx`, `onnx/model_quint8_avx2.onnx`). They are -not used by this chart and were not measured; they are CPU-specific and -would need to be baked as `onnx/model.onnx` for TEI. - -Licence: `cross-encoder/mmarco-mMiniLMv2-L12-H384-v1` is Apache-2.0, -`BAAI/bge-reranker-v2-m3` is Apache-2.0 as well (as tagged on the Hugging Face hub); check the model card of -anything else you select. - -To use the higher-quality model (more CPU or a GPU), build or pull the matching -init image and set: +## Contributing -```yaml -reranker: - modelName: "BAAI/bge-reranker-v2-m3" - modelInit: - image: - tag: "bge-reranker-v2-m3-" # e.g. bge-reranker-v2-m3-202611.3.1, see the published tags -``` - -The init step copies with `--ignore-existing`, so when you change the model on -an existing release, delete the reranker model PVC (or install with -`reranker.persistence.enabled=false`) so the old model files are not reused. -`BAAI/bge-reranker-v2-m3` needs `requests: 4Gi` / `limits: 8Gi` memory with -`--max-batch-tokens 2048`; the default model needs far less. - -To build an init image yourself: `make docker/tei-reranker-model-init -RERANKER_MODEL_ID=`. If the model repository ships -`onnx/model.onnx` it is used as-is; otherwise the model is exported with -Optimum. Only XLM-RoBERTa-style sequence-classification models that TEI -supports will work. - -### Whisper (faster-whisper + large-v3-turbo, int8) - -Measured on CPU: ~2.0 GiB RSS peak; roughly real-time speed on 2 threads (a -3.7 min recording took 3.3 min), faster with more threads. - -- `whisper.cpuThreads` must match the CPU limit (CTranslate2 sizes itself by - the node's cores, not by the cgroup limit). -- Transcriptions run one at a time; further requests queue in the server. Keep - the router's `WHISPER_TIMEOUT` (default `300s`) above queue time + runtime, - e.g. via `router.extraEnv`. -- Upload limit: `whisper.maxUploadMB` (default 100). Formats: anything FFmpeg - decodes (wav, mp3, m4a, ogg, flac, webm, mp4, ...). -- `response_format`: `json` (default), `text`, `verbose_json`, `srt`, `vtt`. -- Set `whisper.language` (e.g. `de`) to skip language auto-detection; a - request's own `language` field wins. -- Vocabulary hints, two mechanisms (both optional, both off by default): - - `prompt` (form field, OpenAI-compatible) becomes faster-whisper's - `initial_prompt`. Set a server-side default with `whisper.initialPrompt` - (`WHISPER_INITIAL_PROMPT`) for clients that send no prompt (e.g. an STT - integration that only sends `model` and `language`). A request prompt is - **appended** to the default (` `): Whisper weighs the end - of the prompt most, and an over-long prompt is cut from the front, so the - request-specific text survives. Whisper reads at most 223 tokens - (`max_length // 2 - 1`); faster-whisper keeps the last 223 and drops the - rest. The server additionally caps the combined text at 2000 characters - (tail kept). - - `hotwords` (form field, extension; free text such as - `Kubernetes Grafana Postgres`) is passed to faster-whisper's `hotwords`. - Server default: `whisper.hotwords` (`WHISPER_HOTWORDS`). A request's - `hotwords` **replaces** the default (no merging). faster-whisper puts the - hotwords in front of the previous-text prompt; it ignores them only when - `prefix` is set, which this server never sets, so `prompt` and `hotwords` - work together (checked in `faster_whisper/transcribe.py`, `get_prompt`, - version 1.2.1). Both count against the same 223-token window of their own. - - Neutral example for `whisper.initialPrompt`: - `Release notes for Kubernetes, PostgreSQL and Grafana.` - - Hints raise the odds of a spelling; they do not guarantee it. Keep them - short and spell terms the way they should appear. -- Repetition loops on long audio: the sequential decoder conditions every - window on the previous text and can get stuck repeating itself. The server - therefore (1) keeps `condition_on_previous_text` off by default - (`whisper.conditionOnPreviousText`), (2) sends audio longer than - `whisper.batchThresholdSeconds` (default 35 s, `WHISPER_BATCH_THRESHOLD_S`) - through faster-whisper's `BatchedInferencePipeline` (VAD-cut chunks decoded - independently, `whisper.batchSize` = 8, `WHISPER_BATCH_SIZE`), and (3) uses - faster-whisper's temperature fallback (0.0, 0.2, ... 1.0 with its default - compression-ratio, log-prob and no-speech thresholds) on the sequential - path. The batched path always uses VAD and only the first temperature (the - library does not fall back there). A request's explicit `temperature` form - field (including `0`) is used as given, without fallback; clients that always - send `temperature=0` therefore get no fallback on the sequential path. - Measured locally (FLEURS de, int8, 4 threads, beam 1, VAD): three 55-88 s - clips WER 3.1 % instead of 42 % (one clip looped to 256 instead of 120 - words) and 56 s of audio in 17.8 s instead of 27.2 s; 40 single clips - 5.6 % vs 5.2 % (noise); no speed difference on short clips. -- Warmup: after loading the model the server runs a 2 s synthetic clip through - VAD and decoder (`WHISPER_WARMUP`, default `true`) so the first real request - does not pay the initialisation cost. `/health` returns 503 until model and - warmup are done; a failed warmup is logged and does not block startup. -- Model source and pins: the image bakes `large-v3-turbo` from - `dropbox-dash/faster-whisper-large-v3-turbo` at a fixed commit - (`WHISPER_MODEL_REVISION` in `deploy/Containerfile.whisper`) and verifies - `model.bin` against its SHA-256; the build fails if either changes. This is - the repository faster-whisper's `large-v3-turbo` alias - (`mobiuslabsgmbh/faster-whisper-large-v3-turbo`) redirects to (HTTP 307); - `Systran/faster-whisper-large-v3-turbo` answers HTTP 401. Model license: MIT. - Python dependencies are pinned in `deploy/whisper/requirements.txt` (`av` - must stay on 16.x, newer majors break faster-whisper 1.2.1) and the base - image by digest. -- Per-request log line (JSON after the prefix `whisper_request`), e.g. - `whisper_request {"status":"ok","path":"standard","audio_s":9.0,"prep_s":0.4,"infer_s":7.6,"total_s":8.1,"rtf":0.889,...}`: - audio duration (`audio_s`, `audio_after_vad_s`), upload, queue wait, decode + - VAD (`prep_s`) and inference (`infer_s`) time, real-time factor - (`rtf = (prep_s + infer_s) / audio_s`), model, language, `beam_size`, `vad`, `path` (`standard|batched`), - `temperature_mode` (`fallback|fixed`), - file size, and `prompt_chars` / `hotwords_chars` / `prompt_source` - (`none|default|request|both`). Prompt, hotwords and transcript text are - never logged. -- `/v1/audio/translations` (to English) is exposed, but large-v3-turbo is not - trained for translation; use a non-turbo model for that. - -### Scheduling (`affinity`, `topologySpreadConstraints`) - -`embedding` and `whisper` accept optional `affinity` and -`topologySpreadConstraints` (empty by default, so the rendered manifests do not -change). With `embedding.replicaCount > 1` the replicas can land on one node, -and then the PodDisruptionBudget does not protect against a node failure. -Example, soft spreading that never blocks scheduling: - -```yaml -embedding: - replicaCount: 2 - affinity: - podAntiAffinity: - preferredDuringSchedulingIgnoredDuringExecution: - - weight: 100 - podAffinityTerm: - topologyKey: kubernetes.io/hostname - labelSelector: - matchLabels: - app.kubernetes.io/component: embedding - topologySpreadConstraints: - - maxSkew: 1 - topologyKey: kubernetes.io/hostname - whenUnsatisfiable: ScheduleAnyway - labelSelector: - matchLabels: - app.kubernetes.io/component: embedding +```bash +make test # go test -race ./... +make lint # golangci-lint +make test-local # end-to-end test against mock backends ``` -## Environment Variables - -| Variable | Default | Description | -| ---------------------- | ------- | ----------------------------- | -| `ROUTER_ADDR` | `:8080` | Main listen address | -| `ROUTER_METRICS_ADDR` | `:9090` | Prometheus metrics address | -| `ROUTER_READ_TIMEOUT` | `120s` | HTTP read timeout | -| `ROUTER_WRITE_TIMEOUT` | `300s` | HTTP write timeout | -| `EMBEDDING_ENABLED` | `false` | Enable embedding backend | -| `EMBEDDING_URL` | — | Base URL for TEI embedding | -| `EMBEDDING_TIMEOUT` | `60s` | Per-request timeout | -| `RERANKER_ENABLED` | `false` | Enable reranker backend | -| `RERANKER_URL` | — | Base URL for reranker | -| `RERANKER_TIMEOUT` | `30s` | Per-request timeout | -| `WHISPER_ENABLED` | `false` | Enable Whisper backend | -| `WHISPER_URL` | — | Base URL for Whisper | -| `WHISPER_TIMEOUT` | `300s` | Per-request timeout | -| `LOG_LEVEL` | `info` | `debug`/`info`/`warn`/`error` | -| `LOG_FORMAT` | `json` | `json`/`console` | - -## Security - -- All containers run as non-root (uid 1000 or 65532) -- `allowPrivilegeEscalation: false` -- `capabilities: drop: [ALL]` -- ONNX Runtime exec-stack flag cleared via patchelf -- No outbound network calls at runtime (`HF_HUB_OFFLINE=1`) -- Router image uses distroless base (no shell) +Pull requests are welcome. Releases are cut automatically from `main` +per component, see [AGENTS.md](AGENTS.md#container-image-versioning). ## License -[Apache License 2.0](LICENSE) +[Apache License 2.0](LICENSE). The models have their own licenses: check the model cards of the models you use. diff --git a/assets/banner.png b/assets/banner.png new file mode 100644 index 0000000..7eda2ef Binary files /dev/null and b/assets/banner.png differ diff --git a/assets/icon-tile.svg b/assets/icon-tile.svg new file mode 100644 index 0000000..88d9af1 --- /dev/null +++ b/assets/icon-tile.svg @@ -0,0 +1,2 @@ + + diff --git a/assets/icon.svg b/assets/icon.svg new file mode 100644 index 0000000..f817758 --- /dev/null +++ b/assets/icon.svg @@ -0,0 +1,2 @@ + + diff --git a/assets/lockup-dark.svg b/assets/lockup-dark.svg new file mode 100644 index 0000000..e024b62 --- /dev/null +++ b/assets/lockup-dark.svg @@ -0,0 +1,2 @@ + + diff --git a/assets/lockup-light.svg b/assets/lockup-light.svg new file mode 100644 index 0000000..09e7bae --- /dev/null +++ b/assets/lockup-light.svg @@ -0,0 +1,2 @@ + + diff --git a/assets/logo.svg b/assets/logo.svg new file mode 100644 index 0000000..b862611 --- /dev/null +++ b/assets/logo.svg @@ -0,0 +1,2 @@ + + diff --git a/docs/configuration.md b/docs/configuration.md new file mode 100644 index 0000000..8c8968f --- /dev/null +++ b/docs/configuration.md @@ -0,0 +1,60 @@ +# Configuration + +The router is configured exclusively through environment variables (`internal/config/config.go`). With the +Helm chart the backend variables are set for you; add or override variables with `router.extraEnv`. + +At least one backend must be enabled, and every enabled backend needs its `*_URL`, otherwise the router exits +at startup with a config error. + +| Variable | Default | Description | +| --- | --- | --- | +| `ROUTER_ADDR` | `:8080` | Main listen address | +| `ROUTER_METRICS_ADDR` | `:9090` | Prometheus metrics address (`/metrics`) | +| `ROUTER_READ_TIMEOUT` | `120s` | HTTP read timeout | +| `ROUTER_WRITE_TIMEOUT` | `300s` | HTTP write timeout | +| `ROUTER_IDLE_TIMEOUT` | `60s` | HTTP idle timeout | +| `ROUTER_MAX_REQUEST_SIZE` | `104857600` | Maximum request body in bytes (100 MB) | +| `LOG_LEVEL` | `info` | `debug`, `info`, `warn` or `error` | +| `LOG_FORMAT` | `json` | `json` or `console` | +| `EMBEDDING_ENABLED` | `false` | Enable the embedding backend | +| `EMBEDDING_URL` | none | Base URL of the TEI embedding server | +| `EMBEDDING_TIMEOUT` | `60s` | Per-request timeout | +| `EMBEDDING_MAX_CONCURRENT` | `20` | Requests forwarded at once; more get `503` + `Retry-After` | +| `RERANKER_ENABLED` | `false` | Enable the reranker backend | +| `RERANKER_URL` | none | Base URL of the TEI reranker server | +| `RERANKER_TIMEOUT` | `30s` | Per-request timeout | +| `RERANKER_MAX_CONCURRENT` | `20` | Requests forwarded at once; more get `503` + `Retry-After` | +| `WHISPER_ENABLED` | `false` | Enable the Whisper backend | +| `WHISPER_URL` | none | Base URL of the Whisper server | +| `WHISPER_TIMEOUT` | `300s` | Per-request timeout | +| `WHISPER_MAX_CONCURRENT` | `8` | Requests forwarded at once; more get `503` + `Retry-After` | + +A value of `0` or below for a `*_MAX_CONCURRENT` variable disables the bound for that backend. + +## Backpressure + +When a backend is at its concurrency bound the router answers immediately with +`503 Service Unavailable`, a `Retry-After: 3` header and `{"error":"backend_overloaded", ...}` instead of +queuing the request. Clients should back off and retry. Each rejection increments +`inference_router_rejected_total{backend}`. + +## Health endpoints + +| Path | Port | Behaviour | +| --- | --- | --- | +| `/healthz`, `/readyz` | 8080 | Backends are probed (`GET /health`) every 15 seconds. The answer is always HTTP 200; the JSON body says `ok` or `degraded` and lists each backend as `ok`, `unhealthy` or `disabled`. A failing backend does not take the router out of rotation. | +| `/health` | 8080 | Router liveness only: `{"status":"ok"}` | +| `/metrics` | 9090 | Prometheus metrics | + +## Metrics + +| Metric | Type | Labels | +| --- | --- | --- | +| `inference_router_requests_total` | counter | `backend`, `path`, `status` | +| `inference_router_request_duration_seconds` | histogram | `backend`, `path` | +| `inference_router_request_size_bytes` | histogram | `backend` | +| `inference_router_response_size_bytes` | histogram | `backend` | +| `inference_router_active_requests` | gauge | `backend` | +| `inference_router_backend_up` | gauge | `backend` | +| `inference_router_upstream_errors_total` | counter | `backend`, `error_type` | +| `inference_router_rejected_total` | counter | `backend` | diff --git a/docs/tuning.md b/docs/tuning.md new file mode 100644 index 0000000..183d1f2 --- /dev/null +++ b/docs/tuning.md @@ -0,0 +1,220 @@ +# CPU tuning and sizing + +Measurements and settings for running the stack on plain CPU. The Helm values referenced here are in +[`deploy/helm/values.yaml`](../deploy/helm/values.yaml); see also [Configuration](configuration.md). + +## Embedding (TEI + BGE-M3) + +| `--max-batch-tokens` | RSS (fp32) | Recommendation | +| -------------------- | ---------- | -------------------- | +| 4096 | 3-6Gi | **CPU default** | +| 8192 | 6-10Gi | May OOM on 8Gi limit | +| 16384 | 10-16Gi | GPU-only | + +## Reranker model choice + +The reranker model is baked into the `tei-reranker-model-init` image and +selected by its tag (`reranker.modelInit.image.tag`). Both models run on the +same TEI runtime with the ONNX Runtime backend. + +Measured with TEI `cpu-1.9.4`, limited to 4 CPUs, `--max-batch-tokens 2048`, +one `/rerank` request with one query and 20 documents of 800 characters each +(German/English mix), 4 runs per model. The CPU was an older AVX2 machine, so +only the ratios carry over. + +| Model | Format | Parameters | Latency (20 docs) | +| ------------------------------------------------------ | ------------------------- | ---------- | ----------------- | +| `cross-encoder/mmarco-mMiniLMv2-L12-H384-v1` (default) | fp32 ONNX | 118M | 0.76 to 0.85 s | +| `BAAI/bge-reranker-v2-m3` | fp32 ONNX | 568M | 8.3 to 11.9 s | +| `BAAI/bge-reranker-v2-m3` | int8 ONNX | 568M | 7.9 to 8.2 s | +| `Alibaba-NLP/gte-multilingual-reranker-base` | safetensors (candle path) | n/a | 11.1 to 12.1 s | + +That is about 0.04 s per document pair for the default against roughly 0.4 to +0.5 s for `bge-reranker-v2-m3`, so reranking 40 candidates with the larger +model does not fit an interactive latency budget on 4 CPUs. + +Quality, on a downstream 75-question German/English relevance set (hybrid +retrieval, rerank the top 20 candidates, 800 characters per document): + +| Setup | hit@1 | hit@5 | hit@10 | +| ------------------------------------------ | ----- | ----- | ------ | +| No rerank | 77% | 95% | 96% | +| `BAAI/bge-reranker-v2-m3` | 92% | 96% | 96% | +| `cross-encoder/mmarco-mMiniLMv2-L12-H384-v1` | 92% | 96% | 96% | + +This set is small and saturated (one question is 1.3 points), its documents +were short, and the embeddings used for candidate retrieval were a simple +stand-in, so it cannot separate the two models. Real-world German technical +text may favour the larger model; measure on your own data if quality matters +more than latency. + +### Sizing the request for the default model + +Latency is dominated by the number of passages and their length, and one +request already uses all CPUs, so concurrent requests queue. Measured with the +default model on a 4-vCPU pod (TEI `cpu-1.9.4`, `--max-batch-tokens 2048`), +75 queries, passages of title plus text cut to the given length (800 +characters are about 190 tokens, 500 about 120), p95 latency of one `/rerank` +call: + +| Passages | 800 chars | 600 chars | 500 chars | 400 chars | +| -------- | --------- | --------- | --------- | --------- | +| 40 | 3.2 s | 2.5 s | 2.1 s | 1.8 s | +| 30 | 2.4 s | 1.9 s | 1.7 s | 1.4 s | +| 20 | 1.7 s | 1.3 s | 1.1 s | 1.0 s | +| 15 | 1.3 s | 1.0 s | 0.9 s | 0.7 s | + +With four such requests in flight at once, p95 is about 3 to 4 times higher +(for example 3.0 s for 15 passages of 500 characters, 6.4 s for 40 of 400). +If the budget is about 1.5 s per request, send at most 15 to 20 passages of +about 500 characters, and scale out (`reranker.replicaCount`) or add CPU for +concurrent callers. Reranking a short list also bounds what it can gain: it can +only reorder the passages it receives. + +Memory: the working set stayed at about 1.4 GiB, including four concurrent +requests of 40 passages, so the chart defaults to a 1.5 GiB request and a 2 GiB +limit. Inputs longer than the model's 512-token limit are truncated by default +(`truncate: true`, or the TEI default when the field is omitted); with +`truncate: false` the request fails with HTTP 422. + +The model repository also ships quantised ONNX variants (for example +`onnx/model_qint8_avx512_vnni.onnx`, `onnx/model_quint8_avx2.onnx`). They are +not used by this chart and were not measured; they are CPU-specific and +would need to be baked as `onnx/model.onnx` for TEI. + +Licence: `cross-encoder/mmarco-mMiniLMv2-L12-H384-v1` is Apache-2.0, +`BAAI/bge-reranker-v2-m3` is Apache-2.0 as well (as tagged on the Hugging Face hub); check the model card of +anything else you select. + +To use the higher-quality model (more CPU or a GPU), build or pull the matching +init image and set: + +```yaml +reranker: + modelName: "BAAI/bge-reranker-v2-m3" + modelInit: + image: + tag: "bge-reranker-v2-m3-" # e.g. bge-reranker-v2-m3-202611.3.1, see the published tags +``` + +The init step copies with `--ignore-existing`, so when you change the model on +an existing release, delete the reranker model PVC (or install with +`reranker.persistence.enabled=false`) so the old model files are not reused. +`BAAI/bge-reranker-v2-m3` needs `requests: 4Gi` / `limits: 8Gi` memory with +`--max-batch-tokens 2048`; the default model needs far less. + +To build an init image yourself: `make docker/tei-reranker-model-init +RERANKER_MODEL_ID=`. If the model repository ships +`onnx/model.onnx` it is used as-is; otherwise the model is exported with +Optimum. Only XLM-RoBERTa-style sequence-classification models that TEI +supports will work. + +## Whisper (faster-whisper + large-v3-turbo, int8) + +Measured on CPU: ~2.0 GiB RSS peak; roughly real-time speed on 2 threads (a +3.7 min recording took 3.3 min), faster with more threads. + +- `whisper.cpuThreads` must match the CPU limit (CTranslate2 sizes itself by + the node's cores, not by the cgroup limit). +- Transcriptions run one at a time; further requests queue in the server. Keep + the router's `WHISPER_TIMEOUT` (default `300s`) above queue time + runtime, + e.g. via `router.extraEnv`. +- Upload limit: `whisper.maxUploadMB` (default 100). Formats: anything FFmpeg + decodes (wav, mp3, m4a, ogg, flac, webm, mp4, ...). +- `response_format`: `json` (default), `text`, `verbose_json`, `srt`, `vtt`. +- Set `whisper.language` (e.g. `de`) to skip language auto-detection; a + request's own `language` field wins. +- Vocabulary hints, two mechanisms (both optional, both off by default): + - `prompt` (form field, OpenAI-compatible) becomes faster-whisper's + `initial_prompt`. Set a server-side default with `whisper.initialPrompt` + (`WHISPER_INITIAL_PROMPT`) for clients that send no prompt (e.g. an STT + integration that only sends `model` and `language`). A request prompt is + **appended** to the default (` `): Whisper weighs the end + of the prompt most, and an over-long prompt is cut from the front, so the + request-specific text survives. Whisper reads at most 223 tokens + (`max_length // 2 - 1`); faster-whisper keeps the last 223 and drops the + rest. The server additionally caps the combined text at 2000 characters + (tail kept). + - `hotwords` (form field, extension; free text such as + `Kubernetes Grafana Postgres`) is passed to faster-whisper's `hotwords`. + Server default: `whisper.hotwords` (`WHISPER_HOTWORDS`). A request's + `hotwords` **replaces** the default (no merging). faster-whisper puts the + hotwords in front of the previous-text prompt; it ignores them only when + `prefix` is set, which this server never sets, so `prompt` and `hotwords` + work together (checked in `faster_whisper/transcribe.py`, `get_prompt`, + version 1.2.1). Both count against the same 223-token window of their own. + - Neutral example for `whisper.initialPrompt`: + `Release notes for Kubernetes, PostgreSQL and Grafana.` + - Hints raise the odds of a spelling; they do not guarantee it. Keep them + short and spell terms the way they should appear. +- Repetition loops on long audio: the sequential decoder conditions every + window on the previous text and can get stuck repeating itself. The server + therefore (1) keeps `condition_on_previous_text` off by default + (`whisper.conditionOnPreviousText`), (2) sends audio longer than + `whisper.batchThresholdSeconds` (default 35 s, `WHISPER_BATCH_THRESHOLD_S`) + through faster-whisper's `BatchedInferencePipeline` (VAD-cut chunks decoded + independently, `whisper.batchSize` = 8, `WHISPER_BATCH_SIZE`), and (3) uses + faster-whisper's temperature fallback (0.0, 0.2, ... 1.0 with its default + compression-ratio, log-prob and no-speech thresholds) on the sequential + path. The batched path always uses VAD and only the first temperature (the + library does not fall back there). A request's explicit `temperature` form + field (including `0`) is used as given, without fallback; clients that always + send `temperature=0` therefore get no fallback on the sequential path. + Measured locally (FLEURS de, int8, 4 threads, beam 1, VAD): three 55-88 s + clips WER 3.1 % instead of 42 % (one clip looped to 256 instead of 120 + words) and 56 s of audio in 17.8 s instead of 27.2 s; 40 single clips + 5.6 % vs 5.2 % (noise); no speed difference on short clips. +- Warmup: after loading the model the server runs a 2 s synthetic clip through + VAD and decoder (`WHISPER_WARMUP`, default `true`) so the first real request + does not pay the initialisation cost. `/health` returns 503 until model and + warmup are done; a failed warmup is logged and does not block startup. +- Model source and pins: the image bakes `large-v3-turbo` from + `dropbox-dash/faster-whisper-large-v3-turbo` at a fixed commit + (`WHISPER_MODEL_REVISION` in `deploy/Containerfile.whisper`) and verifies + `model.bin` against its SHA-256; the build fails if either changes. This is + the repository faster-whisper's `large-v3-turbo` alias + (`mobiuslabsgmbh/faster-whisper-large-v3-turbo`) redirects to (HTTP 307); + `Systran/faster-whisper-large-v3-turbo` answers HTTP 401. Model license: MIT. + Python dependencies are pinned in `deploy/whisper/requirements.txt` (`av` + must stay on 16.x, newer majors break faster-whisper 1.2.1) and the base + image by digest. +- Per-request log line (JSON after the prefix `whisper_request`), e.g. + `whisper_request {"status":"ok","path":"standard","audio_s":9.0,"prep_s":0.4,"infer_s":7.6,"total_s":8.1,"rtf":0.889,...}`: + audio duration (`audio_s`, `audio_after_vad_s`), upload, queue wait, decode + + VAD (`prep_s`) and inference (`infer_s`) time, real-time factor + (`rtf = (prep_s + infer_s) / audio_s`), model, language, `beam_size`, `vad`, `path` (`standard|batched`), + `temperature_mode` (`fallback|fixed`), + file size, and `prompt_chars` / `hotwords_chars` / `prompt_source` + (`none|default|request|both`). Prompt, hotwords and transcript text are + never logged. +- `/v1/audio/translations` (to English) is exposed, but large-v3-turbo is not + trained for translation; use a non-turbo model for that. + +## Scheduling (`affinity`, `topologySpreadConstraints`) + +`embedding` and `whisper` accept optional `affinity` and +`topologySpreadConstraints` (empty by default, so the rendered manifests do not +change). With `embedding.replicaCount > 1` the replicas can land on one node, +and then the PodDisruptionBudget does not protect against a node failure. +Example, soft spreading that never blocks scheduling: + +```yaml +embedding: + replicaCount: 2 + affinity: + podAntiAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 100 + podAffinityTerm: + topologyKey: kubernetes.io/hostname + labelSelector: + matchLabels: + app.kubernetes.io/component: embedding + topologySpreadConstraints: + - maxSkew: 1 + topologyKey: kubernetes.io/hostname + whenUnsatisfiable: ScheduleAnyway + labelSelector: + matchLabels: + app.kubernetes.io/component: embedding +```