Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Use it when you want to:

`localmodelproxy` listens on `127.0.0.1` by default and exposes:

- `GET /v1/models` — always returns the configured models list
- `GET /v1/models` — always returns the configured models list. The interactive TUI's **Models** diagnostic tab separately queries upstream `/models` endpoints and compares their responses with this configuration.
- `POST /v1/chat/completions` — validated against configured model IDs and forwarded to the matching backend

The proxy selects a backend by inspecting the request model, validates it against configured model IDs (returning a 400 `model_not_found` error if not found), forwards the request transparently, rewrites model aliases when configured, applies the backend's authentication method, and aggregates token usage and optional cost from the response.
Expand Down
21 changes: 20 additions & 1 deletion docs/USAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,13 +92,32 @@ The app uses the TUI when stdout is an interactive terminal and `--headless` is

### TUI Navigation

When the Test tab is enabled (the default), the TUI has two views accessible via the **Tab** key:
The TUI has the following views, accessible with **Tab**, **←/→**, or **h/l**:

- **Stats** – Displays per-model statistics and recent requests (the default view).
- **Models** – Queries every upstream backend's `/models` endpoint and compares the response with the configured models.
- **Test** – Lists all configured models. Use **↑/↓** to select a model and **Enter** to send a test request. The response is displayed inline.

Test requests go through the proxy like any other request, so they count towards stats and token usage.

### Model Discovery Diagnostics

Opening the **Models** tab calls the OpenAI-compatible `/models` endpoint on each configured upstream backend. The request uses the backend's configured base URL, authentication, HTTP client, and TLS settings. It does not compare against the proxy's own `GET /v1/models` endpoint, because that endpoint is generated from the config itself.

The combined list distinguishes the source and consistency of every model:

| Color | Status | Meaning |
|-------|--------|---------|
| Green | `MATCH` | A configured model's effective upstream ID was returned by that backend. |
| Red | `MISSING` | The backend responded successfully but did not return the configured upstream ID. |
| Cyan | `UNCONFIGURED` | The backend returned a model that is not explicitly present in its config. |
| Green | `ALLOWED` | A response model is accepted by a backend configured with `models: all`. |
| Yellow | `UNKNOWN` | Discovery failed, so whether the configured model exists cannot be determined. |

With color enabled, row colors carry the status and the legend shows colored descriptions without repeating color names. When `NO_COLOR` is set, the list adds a textual **Status** column instead.

For configured aliases, matching uses `upstream_id`; when it is omitted, matching uses the local `id`. Use **↑/↓** to select a row and **Enter** to open its detail page. The detail page includes the complete JSON object returned for that model, including provider-specific fields. Use **Esc** or **Backspace** to return to the list, and **R** to query all backends again. Backend-level HTTP, authentication, parsing, and connection failures are shown above the list.

## Backends

Each backend declares where requests go, how they authenticate, and which models they serve.
Expand Down
124 changes: 124 additions & 0 deletions internal/proxy/models.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
package proxy

import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"net/url"
"sort"
"sync"
)

// DiscoveredModel is one model entry returned by an upstream /models endpoint.
// Details retains the complete response object so diagnostics are not limited to
// the standard OpenAI fields.
type DiscoveredModel struct {
ID string
Details json.RawMessage
}

// ModelDiscovery describes the result of querying one configured backend.
type ModelDiscovery struct {
Backend string
URL string
StatusCode int
Models []DiscoveredModel
Err error
}

// DiscoverModels queries every configured backend's OpenAI-compatible /models
// endpoint using the same HTTP and authentication configuration as proxy calls.
// A failed backend is returned alongside successful results so the TUI can show
// a complete diagnostic picture.
func (p *Proxy) DiscoverModels(ctx context.Context) []ModelDiscovery {
results := make([]ModelDiscovery, len(p.backends))
var wg sync.WaitGroup
for i, backend := range p.backends {
wg.Add(1)
go func() {
defer wg.Done()
results[i] = p.discoverBackendModels(ctx, backend)
}()
}
wg.Wait()
return results
}

func (p *Proxy) discoverBackendModels(ctx context.Context, backend *resolvedBackend) ModelDiscovery {
result := ModelDiscovery{Backend: backend.cfg.Name}
modelsURL, err := buildUpstreamURL(backend.base, &url.URL{Path: "/v1/models"})
if err != nil {
result.Err = fmt.Errorf("build models URL: %w", err)
return result
}
result.URL = modelsURL

req, err := http.NewRequestWithContext(ctx, http.MethodGet, modelsURL, nil)
if err != nil {
result.Err = fmt.Errorf("create models request: %w", err)
return result
}
req.Header.Set("Accept", "application/json")
token, err := backend.tokens.Token(ctx)
if err != nil {
result.Err = fmt.Errorf("authenticate: %w", err)
return result
}
if token != "" {
req.Header.Set("Authorization", "Bearer "+token)
}

resp, err := backend.client.Do(req)
if err != nil {
result.Err = fmt.Errorf("request models: %w", err)
return result
}
defer resp.Body.Close()
result.StatusCode = resp.StatusCode
body, err := io.ReadAll(io.LimitReader(resp.Body, captureLimit+1))
if err != nil {
result.Err = fmt.Errorf("read models response: %w", err)
return result
}
if len(body) > captureLimit {
result.Err = fmt.Errorf("models response exceeds %d byte limit", captureLimit)
return result
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
result.Err = fmt.Errorf("HTTP %d: %s", resp.StatusCode, truncateForError(string(body), 200))
return result
}

var response struct {
Data []json.RawMessage `json:"data"`
}
if err := json.Unmarshal(body, &response); err != nil {
result.Err = fmt.Errorf("parse models response: %w", err)
return result
}
for _, raw := range response.Data {
var entry struct {
ID string `json:"id"`
}
if err := json.Unmarshal(raw, &entry); err != nil {
result.Err = fmt.Errorf("parse model entry: %w", err)
return result
}
if entry.ID == "" {
result.Err = fmt.Errorf("models response contains an entry without an id")
return result
}
result.Models = append(result.Models, DiscoveredModel{ID: entry.ID, Details: raw})
}
sort.Slice(result.Models, func(i, j int) bool { return result.Models[i].ID < result.Models[j].ID })
return result
}

func truncateForError(value string, limit int) string {
if len(value) <= limit {
return value
}
return value[:limit] + "..."
}
44 changes: 44 additions & 0 deletions internal/proxy/proxy_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,50 @@ func TestModelsEmpty(t *testing.T) {
}
}

func TestDiscoverModelsCallsUpstreamWithAuthenticationAndPreservesDetails(t *testing.T) {
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/v1/models" {
t.Errorf("unexpected path %q", r.URL.Path)
}
if got := r.Header.Get("Authorization"); got != "Bearer diagnostic-token" {
t.Errorf("unexpected authorization %q", got)
}
w.Header().Set("Content-Type", "application/json")
_, _ = io.WriteString(w, `{"object":"list","data":[{"id":"z-model","owned_by":"owner"},{"id":"a-model","custom":{"size":7}}]}`)
}))
defer upstream.Close()

cfg := testConfig([]config.Model{{ID: "a-model"}})
cfg.Backends[0].BaseURL = upstream.URL + "/v1"
handler := mustNew(t, Options{Config: cfg, TokenProvider: StaticTokenProvider("diagnostic-token"), Metrics: NewMetrics()})
results := handler.DiscoverModels(context.Background())

if len(results) != 1 || results[0].Err != nil {
t.Fatalf("unexpected discovery result: %#v", results)
}
if results[0].StatusCode != http.StatusOK || len(results[0].Models) != 2 {
t.Fatalf("unexpected discovery response: %#v", results[0])
}
if results[0].Models[0].ID != "a-model" || !strings.Contains(string(results[0].Models[0].Details), `"size":7`) {
t.Fatalf("model details were not sorted/preserved: %#v", results[0].Models)
}
}

func TestDiscoverModelsReportsBackendHTTPError(t *testing.T) {
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
http.Error(w, "not available", http.StatusServiceUnavailable)
}))
defer upstream.Close()
cfg := testConfig([]config.Model{{ID: "missing"}})
cfg.Backends[0].BaseURL = upstream.URL + "/v1"
handler := mustNew(t, Options{Config: cfg, TokenProvider: StaticTokenProvider("token"), Metrics: NewMetrics()})

results := handler.DiscoverModels(context.Background())
if len(results) != 1 || results[0].Err == nil || !strings.Contains(results[0].Err.Error(), "HTTP 503") {
t.Fatalf("expected per-backend HTTP error, got %#v", results)
}
}

func TestChatCompletionsMissingModel(t *testing.T) {
handler := mustNew(t, Options{
Config: testConfig([]config.Model{{ID: "google/gemini-2.5-flash"}}),
Expand Down
Loading
Loading