Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 7 additions & 4 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ Key details:

### Whisper (Standalone)

Model is baked directly into the image at build time. No init container, no PVC needed. The image runs `deploy/whisper/server.py`, which speaks the OpenAI audio API itself (the router forwards paths unchanged). `/health` does not queue behind inference. Pin `av` (see Containerfile) — newer PyAV majors break faster-whisper. Each request logs one JSON line (`whisper_request`, timings, RTF; never prompt/transcript text). `WHISPER_INITIAL_PROMPT` / `WHISPER_HOTWORDS` are server-side defaults (see README). Tests: `pytest deploy/whisper` (stubbed model).
Model is baked directly into the image at build time. No init container, no PVC needed. The image runs `deploy/whisper/server.py`, which speaks the OpenAI audio API itself (the router forwards paths unchanged). `/health` does not queue behind inference. Pin `av` (see Containerfile) — newer PyAV majors break faster-whisper. Each request logs one JSON line (`whisper_request`, timings, RTF; never prompt/transcript text). `WHISPER_INITIAL_PROMPT` / `WHISPER_HOTWORDS` are server-side defaults (see docs/tuning.md). Tests: `pytest deploy/whisper` (stubbed model).

### Router (Go Proxy)

Expand Down Expand Up @@ -137,7 +137,7 @@ make docker/whisper # whisper

## Tech Stack & Constraints

- **Go 1.24+** — no generics workarounds needed
- **Go 1.26** (see `go.mod`)
- **`net/http/httptest`** for all tests — no external test frameworks (no testify, no gomock)
- **`go.uber.org/zap`** for structured logging
- **`github.com/prometheus/client_golang`** for metrics
Expand Down Expand Up @@ -228,7 +228,7 @@ It exposes `/health`, `/v1/models`, and the backend-specific inference endpoints
- `promauto` metrics panic on double-registration. If you add a new metric, add it to `internal/metrics/metrics.go` as a package-level var, not inside a function.
- `health.Checker` must be initialised with `health.New(cfg.EnabledBackends(), log)` — passing `cfg.ActiveBackends()` will include disabled backends and skew the health response.
- The `statusWriter` wrapper in `proxy.go` must set `status = 200` lazily (in `Write`) to handle the case where `WriteHeader` is never called explicitly.
- TEI does **not** support reranking — the reranker uses FlagEmbedding, not TEI.
- Reranking runs on TEI (ONNX, see `docs/ADR-001-tei-for-reranking.md`). `deploy/Containerfile.reranker-server` and `deploy/reranker/` are the legacy FlagEmbedding server and are not released any more.
- Some CSI volumes don't support timestamp preservation — always use `rsync -rlgoDv`, never `rsync -a`.
- Pod `fsGroup: 1000` is required for init containers to write to PVCs (CSI provisions volumes as root).

Expand All @@ -250,8 +250,11 @@ It exposes `/health`, `/v1/models`, and the backend-specific inference endpoints
| `EMBEDDING_URL` | — | Base URL for TEI embedding |
| `EMBEDDING_TIMEOUT` | `60s` | Per-request timeout |
| `RERANKER_ENABLED` | `false` | Enable reranker backend |
| `RERANKER_URL` | — | Base URL for FlagEmbedding reranker |
| `RERANKER_URL` | — | Base URL for the TEI reranker |
| `RERANKER_TIMEOUT` | `30s` | Per-request timeout |
| `WHISPER_ENABLED` | `false` | Enable Whisper backend |
| `WHISPER_URL` | — | Base URL for Whisper |
| `WHISPER_TIMEOUT` | `300s` | Per-request timeout |
| `EMBEDDING_MAX_CONCURRENT` / `RERANKER_MAX_CONCURRENT` / `WHISPER_MAX_CONCURRENT` | `20` / `20` / `8` | Concurrency bound per backend; above it the router answers `503` + `Retry-After` |

User-facing documentation: `README.md`, `docs/configuration.md`, `docs/tuning.md`.
501 changes: 135 additions & 366 deletions README.md

Large diffs are not rendered by default.

Binary file added assets/banner.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 2 additions & 0 deletions assets/icon-tile.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 2 additions & 0 deletions assets/icon.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 2 additions & 0 deletions assets/lockup-dark.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 2 additions & 0 deletions assets/lockup-light.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 2 additions & 0 deletions assets/logo.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
60 changes: 60 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# Configuration

The router is configured exclusively through environment variables (`internal/config/config.go`). With the
Helm chart the backend variables are set for you; add or override variables with `router.extraEnv`.

At least one backend must be enabled, and every enabled backend needs its `*_URL`, otherwise the router exits
at startup with a config error.

| Variable | Default | Description |
| --- | --- | --- |
| `ROUTER_ADDR` | `:8080` | Main listen address |
| `ROUTER_METRICS_ADDR` | `:9090` | Prometheus metrics address (`/metrics`) |
| `ROUTER_READ_TIMEOUT` | `120s` | HTTP read timeout |
| `ROUTER_WRITE_TIMEOUT` | `300s` | HTTP write timeout |
| `ROUTER_IDLE_TIMEOUT` | `60s` | HTTP idle timeout |
| `ROUTER_MAX_REQUEST_SIZE` | `104857600` | Maximum request body in bytes (100 MB) |
| `LOG_LEVEL` | `info` | `debug`, `info`, `warn` or `error` |
| `LOG_FORMAT` | `json` | `json` or `console` |
| `EMBEDDING_ENABLED` | `false` | Enable the embedding backend |
| `EMBEDDING_URL` | none | Base URL of the TEI embedding server |
| `EMBEDDING_TIMEOUT` | `60s` | Per-request timeout |
| `EMBEDDING_MAX_CONCURRENT` | `20` | Requests forwarded at once; more get `503` + `Retry-After` |
| `RERANKER_ENABLED` | `false` | Enable the reranker backend |
| `RERANKER_URL` | none | Base URL of the TEI reranker server |
| `RERANKER_TIMEOUT` | `30s` | Per-request timeout |
| `RERANKER_MAX_CONCURRENT` | `20` | Requests forwarded at once; more get `503` + `Retry-After` |
| `WHISPER_ENABLED` | `false` | Enable the Whisper backend |
| `WHISPER_URL` | none | Base URL of the Whisper server |
| `WHISPER_TIMEOUT` | `300s` | Per-request timeout |
| `WHISPER_MAX_CONCURRENT` | `8` | Requests forwarded at once; more get `503` + `Retry-After` |

A value of `0` or below for a `*_MAX_CONCURRENT` variable disables the bound for that backend.

## Backpressure

When a backend is at its concurrency bound the router answers immediately with
`503 Service Unavailable`, a `Retry-After: 3` header and `{"error":"backend_overloaded", ...}` instead of
queuing the request. Clients should back off and retry. Each rejection increments
`inference_router_rejected_total{backend}`.

## Health endpoints

| Path | Port | Behaviour |
| --- | --- | --- |
| `/healthz`, `/readyz` | 8080 | Backends are probed (`GET <url>/health`) every 15 seconds. The answer is always HTTP 200; the JSON body says `ok` or `degraded` and lists each backend as `ok`, `unhealthy` or `disabled`. A failing backend does not take the router out of rotation. |
| `/health` | 8080 | Router liveness only: `{"status":"ok"}` |
| `/metrics` | 9090 | Prometheus metrics |

## Metrics

| Metric | Type | Labels |
| --- | --- | --- |
| `inference_router_requests_total` | counter | `backend`, `path`, `status` |
| `inference_router_request_duration_seconds` | histogram | `backend`, `path` |
| `inference_router_request_size_bytes` | histogram | `backend` |
| `inference_router_response_size_bytes` | histogram | `backend` |
| `inference_router_active_requests` | gauge | `backend` |
| `inference_router_backend_up` | gauge | `backend` |
| `inference_router_upstream_errors_total` | counter | `backend`, `error_type` |
| `inference_router_rejected_total` | counter | `backend` |
Loading
Loading