diff --git a/CHANGELOG.md b/CHANGELOG.md index 19964f1..c720319 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,11 @@ has a matching GitHub release whose body comes from its section below. ### Added +- The harbor's SWITCH tab loads one llama-swap model: a paired device taps LOAD on a model that is not loaded, and + the model shows loading until llama-swap reports it ready. The gateway's new `POST /v1/berth/load + {"engine","model"}` does it by asking llama-swap for the model's `/upstream/MODEL/health`, since llama-swap has + no load call of its own. The board's models carry `loading`, and its engines their `type`. + - The harbor has a SWITCH tab: each card with its state, holders and cordon, and every engine with its state and models. A device paired with `berth pair` can switch the card to an engine or stop the engine that holds it, from a phone. A pinned holder shows PINNED and has no stop. On a machine with several cards a picker diff --git a/docs/gateway.md b/docs/gateway.md index 0c52de2..dcc005c 100644 --- a/docs/gateway.md +++ b/docs/gateway.md @@ -266,6 +266,7 @@ name the engine: ```sh curl -s http://127.0.0.1:9799/v1/berth/switch -H "Authorization: Bearer $(berth key)" -d '{"engine":"strata"}' curl -s http://127.0.0.1:9799/v1/berth/stop -H "Authorization: Bearer $(berth key)" -d '{"engine":"strata"}' +curl -s http://127.0.0.1:9799/v1/berth/load -H "Authorization: Bearer $(berth key)" -d '{"engine":"llama-swap","model":"qwen3.6-35b-a3b"}' ``` - **Switch** answers `202` once the start is under way, and starts the engine under a lease the same way a request @@ -273,11 +274,17 @@ curl -s http://127.0.0.1:9799/v1/berth/stop -H "Authorization: Bearer $(berth - **Stop** answers `202` with the leases berthd asked to give the card back, the same drain as `berth stop`. A pinned holder (a game, a render, `berth hold`) is a `409`: Berth never stops it from the network. An engine that is not running under a lease is a `409` too, and berthd not answering is a `503`. +- **Load** answers `202` once llama-swap is asked for the model, by a `GET` of `/upstream/MODEL/health` on it: + llama-swap has no load call, and any request there starts the model. It takes an engine of `type = "llama-swap"` + and a model that engine lists; another engine is a `409` and an unlisted model a `404`. A second load of the same + model while one is under way joins it. `GET /v1/berth/board` is the node in one read, for a board or a script: the card (who holds it, and whether it is free, held, draining, in a VM or failing its probe), then every engine in `berth.toml` with its state (up, offline, waiting, starting, failed), whether it can be switched to (`launch`) or stopped (`stoppable`), and its models with a -`loaded` flag from the engine's own reader. `cards` lists every card the node governs, each with its name +`loaded` flag from the engine's own reader (for llama-swap, its `/running` read with the board, so a load never reads cold +between llama-swap's answer and berthd's next read) and a `loading` flag while a load is under way or the engine reports the +model starting. Each engine also carries its `type`. `cards` lists every card the node governs, each with its name on the node (`card`, such as `gpu1`), its base URL on this gateway (`base`, `/cards/gpu1/v1`), `cordoned` when `berth cordon` closed it, and the holders on that card alone. Each engine names the cards its entry runs on (`cards`), so a board for several cards can put each engine under its card. diff --git a/docs/harbor.md b/docs/harbor.md index 7decc52..7063734 100644 --- a/docs/harbor.md +++ b/docs/harbor.md @@ -168,11 +168,20 @@ A paired device (see [Pair a device](#pair-a-device)) also gets the actions: for a request to that engine's model. - **STOP** asks the engine that holds the card to give it back at its next request boundary, as `berth stop` does. +- **LOAD**, on a llama-swap model that is not loaded, asks llama-swap for that model. The model + shows *loading* until llama-swap reports it ready. llama-swap's model command runs under + `berth run`, so the card is leased and its holder drained as for any other start. - A pinned holder, such as a game or `berth hold`, shows **PINNED** and has no STOP. - **PAIR A PHONE** shows a QR code that pairs another phone. -The board and both actions are the gateway's (`GET /v1/berth/board`, `POST /v1/berth/switch` and -`POST /v1/berth/stop`, see [the gateway](gateway.md)). The harbor sends them with the node's API key, +On a machine with several cards, an engine runs on the card its entry names (`gpu = "gpu1"`), or +on the first card when it names none. SWITCH starts it there and drains only that card's holder; +engines on the other cards keep running. To run one engine on either card, give it one entry per +card, such as `comfyui-gpu0` and `comfyui-gpu1`, each with its own `gpu` and port. The picker then +shows each under its card. + +The board and the actions are the gateway's (`GET /v1/berth/board`, `POST /v1/berth/switch`, +`POST /v1/berth/stop` and `POST /v1/berth/load`, see [the gateway](gateway.md)). The harbor sends them with the node's API key, so `berth gateway` must be running. `berth harbor --gateway URL` names the gateway when it is not on `[gateway] listen`, and `--key-file` names the key when it is not in the default place. The harbor reads the key and never creates it. Without a key, the tab says so and offers no actions. diff --git a/internal/gateway/board.go b/internal/gateway/board.go index f63b99c..95da52a 100644 --- a/internal/gateway/board.go +++ b/internal/gateway/board.go @@ -4,9 +4,12 @@ package gateway import ( + "context" "encoding/json" + "maps" "net/http" "os" + "strings" "github.com/T-Crypt/berth/internal/node" "github.com/T-Crypt/berth/internal/proto" @@ -50,32 +53,77 @@ type boardEngine struct { type boardModel struct { ID string `json:"id"` Loaded bool `json:"loaded,omitempty"` // the engine's reader reports it loaded + // Loading: a switchboard load is under way, or the engine reports the + // model starting. + Loading bool `json:"loading,omitempty"` } func (g *Gateway) board(w http.ResponseWriter, r *http.Request) { status := g.nodeStatus() rows := g.guiRows(r.Context(), status) listed := map[string][]string{} + live := map[string]map[string]string{} for i := range g.Engines { e := &g.Engines[i] if rows[i].State == "up" { listed[e.Name] = g.listed(r.Context(), e.URL) + if e.Type == "llama-swap" { + if states := g.swapRunning(r.Context(), e.URL); states != nil { + live[e.Name] = states + } + } } if len(listed[e.Name]) == 0 { listed[e.Name] = e.Serves } } host, _ := os.Hostname() + g.mu.Lock() + inFlight := maps.Clone(g.loads) + g.mu.Unlock() w.Header().Set("Content-Type", "application/json") w.Header().Set("Cache-Control", "no-store") - _ = json.NewEncoder(w).Encode(boardOf(host, status, rows, listed)) + _ = json.NewEncoder(w).Encode(boardOf(host, status, rows, listed, live, inFlight)) +} + +// swapRunning is llama-swap's own /running, model -> state, read at once: +// berthd's reader reads it every few seconds, and a load the board shows +// must not read cold again in the gap between llama-swap's answer and that +// read. nil when llama-swap does not answer. +func (g *Gateway) swapRunning(ctx context.Context, base string) map[string]string { + req, err := http.NewRequestWithContext(ctx, http.MethodGet, strings.TrimRight(base, "/")+"/running", nil) + if err != nil { + return nil + } + resp, err := g.client().Do(req) + if err != nil { + return nil + } + defer resp.Body.Close() + var r struct { + Running []struct { + Model string `json:"model"` + State string `json:"state"` + } `json:"running"` + } + if resp.StatusCode != http.StatusOK || json.NewDecoder(resp.Body).Decode(&r) != nil { + return nil + } + states := map[string]string{} + for _, m := range r.Running { + states[m.Model] = m.State + } + return states } // boardOf assembles the board from berthd's status (nil = not answering), -// the engines' rows in config order and each engine's model list. -func boardOf(host string, status *proto.Status, rows []guiRow, listed map[string][]string) boardNode { +// the engines' rows in config order, each engine's model list, the model +// states llama-swap reported just now (over berthd's older read) and the +// switchboard loads under way. +func boardOf(host string, status *proto.Status, rows []guiRow, listed map[string][]string, live map[string]map[string]string, inFlight map[loadKey]bool) boardNode { n := boardNode{Name: host, Reachable: status != nil, Cards: []boardCard{}, Engines: make([]boardEngine, 0, len(rows))} loaded := map[string]map[string]bool{} + starting := map[string]map[string]bool{} if status != nil { for _, gp := range status.GPUs { if !gp.Scheduled { @@ -105,11 +153,20 @@ func boardOf(host string, status *proto.Status, rows []guiRow, listed map[string } for _, e := range status.Engines { loaded[e.Name] = map[string]bool{} + starting[e.Name] = map[string]bool{} for _, m := range e.Models { loaded[e.Name][m.Name] = m.State == "loaded" || m.State == "ready" + starting[e.Name][m.Name] = m.State == "starting" } } } + for name, states := range live { + loaded[name], starting[name] = map[string]bool{}, map[string]bool{} + for m, s := range states { + loaded[name][m] = s == "ready" + starting[name][m] = s == "starting" + } + } for _, row := range rows { be := boardEngine{guiRow: row, Models: []boardModel{}} if status != nil { @@ -120,7 +177,9 @@ func boardOf(host string, status *proto.Status, rows []guiRow, listed map[string } } for _, id := range listed[row.Name] { - be.Models = append(be.Models, boardModel{ID: id, Loaded: loaded[row.Name][id]}) + l := loaded[row.Name][id] + be.Models = append(be.Models, boardModel{ID: id, Loaded: l, + Loading: !l && (starting[row.Name][id] || inFlight[loadKey{row.Name, id}])}) } n.Engines = append(n.Engines, be) } diff --git a/internal/gateway/board_test.go b/internal/gateway/board_test.go index 173ccf5..7765b28 100644 --- a/internal/gateway/board_test.go +++ b/internal/gateway/board_test.go @@ -99,21 +99,32 @@ func TestBoardOf(t *testing.T) { card string stoppable []bool loaded []boardModel + inFlight map[loadKey]bool + live map[string]map[string]string }{ - {"unreachable", nil, "", []bool{false, false}, []boardModel{{ID: "qwen"}, {ID: "gemma"}}}, + {"unreachable", nil, "", []bool{false, false}, []boardModel{{ID: "qwen"}, {ID: "gemma"}}, nil, nil}, {"held, one model ready", &proto.Status{GPUs: gpu, Leases: []proto.Lease{{Engine: "llama-swap", Group: "qwen", State: "active"}}, Engines: []proto.Engine{{Name: "llama-swap", Models: []proto.Model{{Name: "qwen", State: "ready"}, {Name: "gemma", State: "starting"}}}}}, - "held", []bool{true, false}, []boardModel{{ID: "qwen", Loaded: true}, {ID: "gemma"}}}, + "held", []bool{true, false}, []boardModel{{ID: "qwen", Loaded: true}, {ID: "gemma", Loading: true}}, nil, nil}, + {"a switchboard load under way", &proto.Status{GPUs: gpu}, + "free", []bool{false, false}, []boardModel{{ID: "qwen"}, {ID: "gemma", Loading: true}}, map[loadKey]bool{{"llama-swap", "gemma"}: true}, nil}, + {"a load under way for a model already ready", &proto.Status{GPUs: gpu, + Engines: []proto.Engine{{Name: "llama-swap", Models: []proto.Model{{Name: "qwen", State: "ready"}}}}}, + "free", []bool{false, false}, []boardModel{{ID: "qwen", Loaded: true}, {ID: "gemma"}}, map[loadKey]bool{{"llama-swap", "qwen"}: true}, nil}, + {"llama-swap's own read is newer than berthd's", &proto.Status{GPUs: gpu, + Engines: []proto.Engine{{Name: "llama-swap", Models: []proto.Model{{Name: "gemma", State: "starting"}}}}}, + "free", []bool{false, false}, []boardModel{{ID: "qwen", Loading: true}, {ID: "gemma", Loaded: true}}, nil, + map[string]map[string]string{"llama-swap": {"gemma": "ready", "qwen": "starting"}}}, {"pinned game", &proto.Status{GPUs: gpu, Leases: []proto.Lease{{Engine: "game", State: "active", Pinned: true}}}, - "held", []bool{false, false}, []boardModel{{ID: "qwen"}, {ID: "gemma"}}}, + "held", []bool{false, false}, []boardModel{{ID: "qwen"}, {ID: "gemma"}}, nil, nil}, {"draining", &proto.Status{GPUs: gpu, Leases: []proto.Lease{{Engine: "llama-swap", State: "draining"}}}, - "draining", []bool{false, false}, []boardModel{{ID: "qwen"}, {ID: "gemma"}}}, + "draining", []bool{false, false}, []boardModel{{ID: "qwen"}, {ID: "gemma"}}, nil, nil}, {"in a VM", &proto.Status{GPUs: gpu, Passthrough: &proto.Passthrough{State: proto.CardPassthrough}}, - "passthrough", []bool{false, false}, []boardModel{{ID: "qwen"}, {ID: "gemma"}}}, + "passthrough", []bool{false, false}, []boardModel{{ID: "qwen"}, {ID: "gemma"}}, nil, nil}, } { t.Run(c.name, func(t *testing.T) { - n := boardOf("box", c.node, rows, listed) + n := boardOf("box", c.node, rows, listed, c.live, c.inFlight) if n.Reachable != (c.node != nil) { t.Fatalf("reachable = %v", n.Reachable) } diff --git a/internal/gateway/cards_test.go b/internal/gateway/cards_test.go index a1548f8..5382745 100644 --- a/internal/gateway/cards_test.go +++ b/internal/gateway/cards_test.go @@ -84,7 +84,7 @@ func TestBackgroundEngineStartsOnItsCardWhileAnotherIsHeld(t *testing.T) { }) holders := map[string][]string{} st := n.Status() - for _, c := range boardOf("box", &st, nil, nil).Cards { + for _, c := range boardOf("box", &st, nil, nil, nil, nil).Cards { for _, h := range c.Holders { holders[c.Card] = append(holders[c.Card], h.Name) } diff --git a/internal/gateway/gateway.go b/internal/gateway/gateway.go index e2e21ac..f232717 100644 --- a/internal/gateway/gateway.go +++ b/internal/gateway/gateway.go @@ -91,6 +91,7 @@ type Gateway struct { failed map[string]*failure // engines whose last start died before answering lists map[string]cached // engine URL -> its /v1/models, briefly cached wakes map[string]*wake // the wake page's last start, by engine name + loads map[loadKey]bool // the models a load from the switchboard is under way for adoptOnce sync.Once stateMu sync.Mutex @@ -220,6 +221,7 @@ func (g *Gateway) Handler() http.Handler { mux.HandleFunc("GET /v1/berth/board", g.board) mux.HandleFunc("POST /v1/berth/switch", g.apiSwitch) mux.HandleFunc("POST /v1/berth/stop", g.apiStop) + mux.HandleFunc("POST /v1/berth/load", g.apiLoad) for _, p := range []string{"/v1/chat/completions", "/v1/completions", "/v1/embeddings"} { mux.HandleFunc("POST "+p, g.forward) mux.HandleFunc("POST /cards/{card}"+p, g.onCard(g.forward)) diff --git a/internal/gateway/gui.go b/internal/gateway/gui.go index 6332294..88668c3 100644 --- a/internal/gateway/gui.go +++ b/internal/gateway/gui.go @@ -138,6 +138,7 @@ func (g *Gateway) probe(ctx context.Context, e *config.Engine) bool { // only) and what it is doing on the card right now. type guiRow struct { Name string `json:"name"` + Type string `json:"type"` // the engine's type in berth.toml; llama-swap rows can load a model GUI string `json:"gui,omitempty"` // the engine's web UI; empty = No UI API string `json:"api,omitempty"` // the engine's own address, shown when there is no web UI Launch bool `json:"launch"` // true when the engine has a start command @@ -154,7 +155,7 @@ type guiRow struct { // guiRowFor is the static part of an engine's row: what it is. The state // fields are filled in once, per request. func guiRowFor(e *config.Engine) guiRow { - row := guiRow{Name: e.Name, GUI: e.GUI, Launch: len(e.Start) > 0} + row := guiRow{Name: e.Name, Type: e.Type, GUI: e.GUI, Launch: len(e.Start) > 0} if e.GUI == "" { row.API = e.URL } diff --git a/internal/gateway/load.go b/internal/gateway/load.go new file mode 100644 index 0000000..67bf07b --- /dev/null +++ b/internal/gateway/load.go @@ -0,0 +1,109 @@ +// Copyright 2026 Trevin Tindall (T-Crypt) +// SPDX-License-Identifier: Apache-2.0 + +package gateway + +import ( + "context" + "encoding/json" + "fmt" + "io" + "net/http" + "net/url" + "slices" + "strings" +) + +// The switchboard's third action: load one model behind llama-swap. Berth +// does not swap models itself; llama-swap does, and has no load call of its +// own. Any request under /upstream/MODEL/ makes it start that model, so a +// load is a GET of the model's /health there. llama-swap's model command runs +// under berth run, so the card is leased the usual way. + +type loadKey struct{ engine, model string } + +// apiLoad is POST /v1/berth/load: 202 once the load is under way. The board +// shows the model loading until llama-swap reports it ready. +func (g *Gateway) apiLoad(w http.ResponseWriter, r *http.Request) { + var req struct { + Engine string `json:"engine"` + Model string `json:"model"` + } + body, err := io.ReadAll(io.LimitReader(r.Body, 1<<16)) + if err != nil || json.Unmarshal(body, &req) != nil || req.Engine == "" || req.Model == "" { + apiError(w, http.StatusBadRequest, `the body must be JSON like {"engine":"NAME","model":"MODEL"}`) + return + } + e := g.engineByName(req.Engine) + switch { + case e == nil: + apiError(w, http.StatusNotFound, fmt.Sprintf("no engine %q in berth.toml", req.Engine)) + return + case e.Type != "llama-swap": + apiError(w, http.StatusConflict, req.Engine+" is not llama-swap; only llama-swap loads one model on request") + return + case !slices.Contains(g.listed(r.Context(), e.URL), req.Model): + apiError(w, http.StatusNotFound, fmt.Sprintf("%s does not list a model %q", req.Engine, req.Model)) + return + } + g.load(e.Name, e.URL, req.Model) + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(http.StatusAccepted) + _ = json.NewEncoder(w).Encode(struct { + Engine string `json:"engine"` + Model string `json:"model"` + }{e.Name, req.Model}) +} + +// load asks llama-swap for the model in the background; a second ask while +// one is under way joins it. +func (g *Gateway) load(engine, base, model string) { + k := loadKey{engine, model} + g.mu.Lock() + if g.loads == nil { + g.loads = map[loadKey]bool{} + } + if g.loads[k] { + g.mu.Unlock() + return + } + g.loads[k] = true + g.mu.Unlock() + g.log().Info("loading a model", "engine", engine, "model", model) + go func() { + defer func() { + g.mu.Lock() + delete(g.loads, k) + g.mu.Unlock() + }() + // The load outlives the request that asked for it, bounded by the + // gateway's ready. + ctx, cancel := context.WithTimeout(context.Background(), g.ready()) + defer cancel() + u := strings.TrimRight(base, "/") + "/upstream/" + escapeSegments(model) + "/health" + req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil) + if err == nil { + var resp *http.Response + if resp, err = http.DefaultClient.Do(req); err == nil { + resp.Body.Close() + if resp.StatusCode >= 300 { + err = fmt.Errorf("llama-swap answered %s", resp.Status) + } + } + } + if err != nil { + g.log().Warn("load failed", "engine", engine, "model", model, "err", err) + } + }() +} + +// escapeSegments escapes a model ID for a URL path but keeps its slashes: +// llama-swap finds the model in /upstream/ by its path segments, and an ID +// such as org/model must reach it as two of them. +func escapeSegments(id string) string { + parts := strings.Split(id, "/") + for i, p := range parts { + parts[i] = url.PathEscape(p) + } + return strings.Join(parts, "/") +} diff --git a/internal/gateway/load_test.go b/internal/gateway/load_test.go new file mode 100644 index 0000000..14542fc --- /dev/null +++ b/internal/gateway/load_test.go @@ -0,0 +1,166 @@ +// Copyright 2026 Trevin Tindall (T-Crypt) +// SPDX-License-Identifier: Apache-2.0 + +//go:build unix + +package gateway + +import ( + "encoding/json" + "io" + "log/slog" + "net/http" + "net/http/httptest" + "strings" + "sync" + "testing" + "time" + + "github.com/T-Crypt/berth/internal/config" +) + +// fakeSwap is llama-swap as the load sees it: a model list, and an upstream +// /health that holds until the test lets the model come up. +type fakeSwap struct { + mu sync.Mutex + paths []string // every /upstream request's escaped path + release chan struct{} + status int + state string // the loaded model's state in /running; "" = not running +} + +func (f *fakeSwap) ServeHTTP(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/v1/models": + _, _ = io.WriteString(w, `{"data":[{"id":"qwen"},{"id":"org/gemma 4"}]}`) + return + case "/running": + f.mu.Lock() + state := f.state + f.mu.Unlock() + if state == "" { + _, _ = io.WriteString(w, `{"running":[]}`) + } else { + _, _ = io.WriteString(w, `{"running":[{"model":"org/gemma 4","state":"`+state+`"}]}`) + } + return + } + f.mu.Lock() + f.paths = append(f.paths, r.URL.EscapedPath()) + f.state = "starting" + f.mu.Unlock() + <-f.release + f.mu.Lock() + f.state = "ready" + f.mu.Unlock() + w.WriteHeader(f.status) +} + +func (f *fakeSwap) seen() []string { + f.mu.Lock() + defer f.mu.Unlock() + return append([]string(nil), f.paths...) +} + +func loadGateway(t *testing.T, status int) (*httptest.Server, *fakeSwap) { + t.Helper() + f := &fakeSwap{release: make(chan struct{}), status: status} + swap := httptest.NewServer(f) + t.Cleanup(func() { close(f.release); swap.Close() }) + g := &Gateway{Ready: 10 * time.Second, Key: testKey, Log: slog.New(slog.NewTextHandler(io.Discard, nil)), + Engines: []config.Engine{ + {Name: "llama-swap", Type: "llama-swap", URL: swap.URL}, + {Name: "strata", Type: "strata", URL: "http://127.0.0.1:1"}, + }} + srv := httptest.NewServer(g.Handler()) + t.Cleanup(srv.Close) + return srv, f +} + +func modelLoading(t *testing.T, url, model string) bool { + t.Helper() + for _, e := range readBoard(t, url).Engines { + for _, m := range e.Models { + if e.Name == "llama-swap" && m.ID == model { + return m.Loading + } + } + } + t.Fatalf("no model %q on the board", model) + return false +} + +func modelLoaded(t *testing.T, url, model string) bool { + t.Helper() + for _, e := range readBoard(t, url).Engines { + for _, m := range e.Models { + if e.Name == "llama-swap" && m.ID == model { + return m.Loaded + } + } + } + return false +} + +func TestLoadRefusals(t *testing.T) { + srv, f := loadGateway(t, http.StatusOK) + for _, c := range []struct { + name, body string + want int + msg string + }{ + {"no model", `{"engine":"llama-swap"}`, 400, `the body must be JSON like {"engine":"NAME","model":"MODEL"}`}, + {"unknown engine", `{"engine":"nope","model":"qwen"}`, 404, `no engine "nope" in berth.toml`}, + {"not llama-swap", `{"engine":"strata","model":"qwen"}`, 409, "strata is not llama-swap; only llama-swap loads one model on request"}, + {"unlisted model", `{"engine":"llama-swap","model":"llama"}`, 404, `llama-swap does not list a model "llama"`}, + } { + t.Run(c.name, func(t *testing.T) { + resp := call(t, "POST", srv.URL+"/v1/berth/load", c.body) + var e struct { + Error struct{ Message string } `json:"error"` + } + _ = json.NewDecoder(resp.Body).Decode(&e) + resp.Body.Close() + if resp.StatusCode != c.want || e.Error.Message != c.msg { + t.Fatalf("= %d %q, want %d %q", resp.StatusCode, e.Error.Message, c.want, c.msg) + } + }) + } + if got := f.seen(); len(got) != 0 { + t.Fatalf("a refused load reached llama-swap: %q", got) + } +} + +// A load answers before the model is up, shows on the board until +// llama-swap answers, and a second ask joins the first. +func TestLoadThroughUpstream(t *testing.T) { + srv, f := loadGateway(t, http.StatusOK) + if e := readBoard(t, srv.URL).Engines; e[0].Type != "llama-swap" || e[1].Type != "strata" { + t.Fatalf("board types = %q, %q", e[0].Type, e[1].Type) + } + for range 2 { + resp := call(t, "POST", srv.URL+"/v1/berth/load", `{"engine":"llama-swap","model":"org/gemma 4"}`) + b, _ := io.ReadAll(resp.Body) + resp.Body.Close() + if resp.StatusCode != http.StatusAccepted || strings.TrimSpace(string(b)) != `{"engine":"llama-swap","model":"org/gemma 4"}` { + t.Fatalf("load = %d %s", resp.StatusCode, b) + } + } + until(t, "llama-swap to be asked", func() bool { return len(f.seen()) == 1 }) + if got := f.seen(); got[0] != "/upstream/org/gemma%204/health" { + t.Fatalf("llama-swap was asked for %q", got[0]) + } + if !modelLoading(t, srv.URL, "org/gemma 4") || modelLoading(t, srv.URL, "qwen") { + t.Fatal("the board does not show the load under way") + } + f.release <- struct{}{} + // Once llama-swap answers, the board reads the model loaded at once, not + // cold until berthd's reader catches up. + until(t, "the load to end", func() bool { return !modelLoading(t, srv.URL, "org/gemma 4") }) + if !modelLoaded(t, srv.URL, "org/gemma 4") { + t.Fatal("the model read cold between llama-swap's answer and berthd's next read") + } + if got := f.seen(); len(got) != 1 { + t.Fatalf("two asks loaded twice: %q", got) + } +} diff --git a/internal/harbor/harbor.go b/internal/harbor/harbor.go index c93f1a2..ac1afc9 100644 --- a/internal/harbor/harbor.go +++ b/internal/harbor/harbor.go @@ -131,6 +131,7 @@ func (h *Harbor) Handler() http.Handler { mux.HandleFunc("GET /v1/board", h.boardRead) mux.HandleFunc("POST /v1/board/switch", h.boardAction("switch")) mux.HandleFunc("POST /v1/board/stop", h.boardAction("stop")) + mux.HandleFunc("POST /v1/board/load", h.boardAction("load")) mux.HandleFunc("POST /v1/pair/code", h.pairCode) return h.guard(mux) } @@ -189,7 +190,7 @@ func (h *Harbor) guard(next http.Handler) http.Handler { } // writePaths are the only paths the harbor takes a POST on. -var writePaths = map[string]bool{"/pair": true, "/v1/board/switch": true, "/v1/board/stop": true, "/v1/pair/code": true} +var writePaths = map[string]bool{"/pair": true, "/v1/board/switch": true, "/v1/board/stop": true, "/v1/board/load": true, "/v1/pair/code": true} // mode tells the page which guard this harbor runs under. It is a harbor // fact, stable for the process lifetime, so the page fetches it once: unlike diff --git a/internal/harbor/switchboard.go b/internal/harbor/switchboard.go index 83b6ea4..6ff5f5e 100644 --- a/internal/harbor/switchboard.go +++ b/internal/harbor/switchboard.go @@ -45,9 +45,9 @@ func (h *Harbor) boardRead(w http.ResponseWriter, r *http.Request) { h.toGateway(w, r, http.MethodGet, "/v1/berth/board", nil) } -// boardAction is POST /v1/board/switch and /v1/board/stop. The body is the -// gateway's own, {"engine":"NAME"}, checked here so a bad one never leaves -// the harbor. +// boardAction is POST /v1/board/switch, /v1/board/stop and /v1/board/load. +// The body is the gateway's own, {"engine":"NAME"}, plus "model" for a load, +// checked here so a bad one never leaves the harbor. func (h *Harbor) boardAction(action string) http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { if !h.writeOK(w, r) { @@ -55,12 +55,21 @@ func (h *Harbor) boardAction(action string) http.HandlerFunc { } var req struct { Engine string `json:"engine"` + Model string `json:"model,omitempty"` } body, err := io.ReadAll(io.LimitReader(r.Body, 1<<16)) - if err != nil || json.Unmarshal(body, &req) != nil || req.Engine == "" { - apiError(w, http.StatusBadRequest, "bad_body", `the body must be JSON like {"engine":"NAME"}`) + load := action == "load" + if err != nil || json.Unmarshal(body, &req) != nil || req.Engine == "" || load && req.Model == "" { + want := `{"engine":"NAME"}` + if load { + want = `{"engine":"NAME","model":"MODEL"}` + } + apiError(w, http.StatusBadRequest, "bad_body", "the body must be JSON like "+want) return } + if !load { + req.Model = "" + } out, _ := json.Marshal(req) h.toGateway(w, r, http.MethodPost, "/v1/berth/"+action, out) } diff --git a/internal/harbor/switchboard_test.go b/internal/harbor/switchboard_test.go index 938b198..b20ecad 100644 --- a/internal/harbor/switchboard_test.go +++ b/internal/harbor/switchboard_test.go @@ -42,7 +42,7 @@ func (f *fakeGateway) ServeHTTP(w http.ResponseWriter, r *http.Request) { switch r.URL.Path { case "/v1/berth/board": _, _ = io.WriteString(w, `{"name":"aphotic","reachable":true}`) - case "/v1/berth/switch": + case "/v1/berth/switch", "/v1/berth/load": w.WriteHeader(http.StatusAccepted) _, _ = io.WriteString(w, `{"state":"waiting"}`) case "/v1/berth/stop": @@ -144,6 +144,8 @@ func TestBoardActionGates(t *testing.T) { {"not paired", "/v1/board/switch", `{"engine":"x"}`, with("Cookie", ""), 401, "not_paired"}, {"forged token", "/v1/board/stop", `{"engine":"x"}`, with("Cookie", cookieName+"=forged"), 401, "not_paired"}, {"bad body", "/v1/board/switch", `{"engine":""}`, ok, 400, "bad_body"}, + {"load without a model", "/v1/board/load", `{"engine":"llama-swap"}`, ok, 400, "bad_body"}, + {"load not paired", "/v1/board/load", `{"engine":"llama-swap","model":"qwen"}`, with("Cookie", ""), 401, "not_paired"}, {"pair code unpaired", "/v1/pair/code", ``, with("Cookie", ""), 401, "not_paired"}, } for _, tc := range cases { @@ -172,6 +174,15 @@ func TestBoardActionForwards(t *testing.T) { if got := gw.last(); got != `POST /v1/berth/switch Bearer `+testKey+` {"engine":"strata"}` { t.Fatalf("gateway saw %q", got) } + // A load carries its model; a switch never passes one on. + code, _ = call(t, "POST", base+"/v1/board/load", `{"engine":"llama-swap","model":"org/qwen"}`, hdr) + if got := gw.last(); code != http.StatusAccepted || got != `POST /v1/berth/load Bearer `+testKey+` {"engine":"llama-swap","model":"org/qwen"}` { + t.Fatalf("load = %d, gateway saw %q", code, got) + } + _, _ = call(t, "POST", base+"/v1/board/switch", `{"engine":"strata","model":"x"}`, hdr) + if got := gw.last(); got != `POST /v1/berth/switch Bearer `+testKey+` {"engine":"strata"}` { + t.Fatalf("switch passed a model on: %q", got) + } // A bearer token works for scripts as the cookie does for a browser. gw.mu.Lock() gw.stop = http.StatusConflict diff --git a/internal/harbor/web/demo-board.json b/internal/harbor/web/demo-board.json index 37771ac..320daea 100644 --- a/internal/harbor/web/demo-board.json +++ b/internal/harbor/web/demo-board.json @@ -11,8 +11,15 @@ "total": 25757220864, "used": 22548578304, "holders": [ - { "name": "llama-swap", "state": "active" }, - { "name": "blender", "state": "active", "pinned": true } + { + "name": "llama-swap", + "state": "active" + }, + { + "name": "blender", + "state": "active", + "pinned": true + } ] }, { @@ -30,58 +37,94 @@ "engines": [ { "name": "llama-swap", + "type": "llama-swap", "api": "http://127.0.0.1:9090", "launch": true, "state": "up", "message": "llama-swap is ready", - "cards": ["gpu0"], + "cards": [ + "gpu0" + ], "stoppable": true, "models": [ - { "id": "qwen3.6-35b-a3b", "loaded": true }, - { "id": "gemma-4-27b" }, - { "id": "qwen3.6-coder-30b" } + { + "id": "qwen3.6-35b-a3b", + "loaded": true + }, + { + "id": "gemma-4-27b", + "loading": true + }, + { + "id": "qwen3.6-coder-30b" + } ] }, { "name": "blender", + "type": "custom", "launch": false, "state": "up", "message": "blender holds the card under a pinned lease", - "cards": ["gpu0"], + "cards": [ + "gpu0" + ], "stoppable": false, "models": [] }, { "name": "comfyui", + "type": "comfyui", "gui": "http://127.0.0.1:8188/", "launch": true, "state": "offline", "message": "comfyui is offline; the card is in use by llama-swap. Start asks it to release the card once its current request finishes", - "card": [{ "name": "llama-swap", "state": "active" }], - "cards": ["gpu0"], + "card": [ + { + "name": "llama-swap", + "state": "active" + } + ], + "cards": [ + "gpu0" + ], "stoppable": false, "models": [] }, { "name": "strata", + "type": "strata", "api": "http://127.0.0.1:8080", "launch": true, "state": "starting", "message": "strata is loading its weights", - "cards": ["gpu1"], + "cards": [ + "gpu1" + ], "stoppable": false, - "models": [{ "id": "qwen3.8-flash-next" }] + "models": [ + { + "id": "qwen3.8-flash-next" + } + ] }, { "name": "acestep", + "type": "custom", "api": "http://127.0.0.1:8001", "launch": true, "state": "failed", "message": "acestep did not come up", "detail": "exit status 1: CUDA out of memory", - "cards": ["gpu1"], + "cards": [ + "gpu1" + ], "stoppable": false, - "models": [{ "id": "acestep-1.5" }] + "models": [ + { + "id": "acestep-1.5" + } + ] } ] } diff --git a/internal/harbor/web/harbor.css b/internal/harbor/web/harbor.css index e1b4d37..1a91cf0 100644 --- a/internal/harbor/web/harbor.css +++ b/internal/harbor/web/harbor.css @@ -1049,9 +1049,13 @@ body.ready #panel-engines > :nth-child(n+2) { animation-delay: 60ms; } .sw-paired { font-family: var(--mono); font-size: var(--fs-small); color: var(--dim); display: flex; flex-wrap: wrap; align-items: center; justify-content: space-between; gap: var(--s2); margin-bottom: var(--s3); } #sw-note > .empty-note { margin-bottom: var(--s3); } .sw-model-list { list-style: none; margin: var(--s1) 0 0; padding: 0; display: flex; flex-direction: column; gap: 2px; color: var(--dim); } -.sw-model-list li { display: flex; align-items: center; gap: var(--s2); overflow-wrap: anywhere; } +.sw-model-list li { display: flex; flex-wrap: wrap; align-items: center; gap: var(--s2); overflow-wrap: anywhere; } .sw-model-list li.loaded { color: var(--ink-strong); } .sw-loaded { color: var(--green); } +.sw-loading { color: var(--amber); } +.sw-model-id { flex: 1 1 auto; min-width: 0; overflow-wrap: anywhere; } +.sw-model-list .sw-load { min-height: var(--touch); min-width: 72px; } +.sw-model-list .sw-error { flex-basis: 100%; } dialog#pair-dlg { border: 1px solid var(--hair-2); padding: 0; background: var(--panel); color: var(--ink); border-radius: 8px; box-shadow: var(--shadow-pop); max-width: 360px; width: calc(100vw - 48px); } dialog#pair-dlg::backdrop { background: var(--scrim); backdrop-filter: blur(3px); } .pair-body { padding: var(--s3); display: flex; flex-direction: column; gap: var(--s2); font-family: var(--mono); font-size: var(--fs-small); } diff --git a/internal/harbor/web/switch.js b/internal/harbor/web/switch.js index d1fb533..8d597e5 100644 --- a/internal/harbor/web/switch.js +++ b/internal/harbor/web/switch.js @@ -1,6 +1,6 @@ // The SWITCH tab: who holds each card, and for a paired device the -// switchboard's two actions, switch the card to an engine or stop the one -// that holds it. The board and the actions are the gateway's; the harbor +// switchboard's actions: switch the card to an engine, stop the one that +// holds it, or load one llama-swap model. The board and the actions are the gateway's; the harbor // forwards them. API strings only ever reach textContent. import { $, el, fmtGiB, siEl } from "./util.js"; import { showDialog, toast } from "./ui.js"; @@ -42,26 +42,27 @@ export function initSwitch({ demo }) { } } - async function act(action, engine) { - if (demo) { toast("demo: " + action + " " + engine + " is not sent", "info"); return; } - busy.add(engine); - errors.delete(engine); + async function act(action, engine, model) { + if (demo) { toast("demo: " + action + " " + (model || engine) + " is not sent", "info"); return; } + const key = model ? engine + "\u0000" + model : engine; + busy.add(key); + errors.delete(key); paint(); try { const res = await fetch("/v1/board/" + action, { method: "POST", headers: { "X-Berth": "1", "Content-Type": "application/json", accept: "application/json" }, - body: JSON.stringify({ engine }), + body: JSON.stringify(model ? { engine, model } : { engine }), }); if (!res.ok) { const j = await res.json().catch(() => ({})); - errors.set(engine, j.message || (j.error && j.error.message) || res.statusText); + errors.set(key, j.message || (j.error && j.error.message) || res.statusText); if (res.status === 401) paired = false; } } catch (e) { - errors.set(engine, e.message); + errors.set(key, e.message); } - busy.delete(engine); + busy.delete(key); await read(); } @@ -139,7 +140,7 @@ export function initSwitch({ demo }) { const models = e.models || []; if (models.length) actions.append(modelsToggle(e.name, models.length)); if (actions.childElementCount) row.append(actions); - if (models.length && open.has(e.name)) row.append(modelList(models)); + if (models.length && open.has(e.name)) row.append(modelList(e, models)); return row; } @@ -160,16 +161,32 @@ export function initSwitch({ demo }) { return b; } - function modelList(models) { + // A llama-swap row loads one model on request; the board marks it loading + // until llama-swap reports it ready. + function modelList(e, models) { const ul = el("ul", "sw-model-list"); for (const m of models) { - const li = el("li", m.loaded ? "loaded" : null, siEl(m.loaded ? "dot" : "ring"), m.id); - if (m.loaded) li.append(el("span", "sw-loaded", " loaded")); + const li = el("li", m.loaded ? "loaded" : null, siEl(m.loaded ? "dot" : m.loading ? "half" : "ring"), el("span", "sw-model-id", m.id)); + if (m.loaded) li.append(el("span", "sw-loaded", "loaded")); + else if (m.loading) li.append(el("span", "sw-loading", "loading")); + else if (paired && e.type === "llama-swap") li.append(loadButton(e.name, m.id)); + const err = errors.get(e.name + "\u0000" + m.id); + if (err) li.append(el("span", "sw-error", err)); ul.append(li); } return ul; } + function loadButton(engine, model) { + const key = engine + "\u0000" + model; + const b = el("button", "btn sw-btn sw-load", busy.has(key) ? "…" : "LOAD"); + b.type = "button"; + b.disabled = busy.has(key); + b.setAttribute("aria-label", "load " + model); + b.addEventListener("click", () => act("load", engine, model)); + return b; + } + function pairButton() { const b = el("button", "btn sw-btn", "PAIR A PHONE"); b.type = "button";