feat(harbor): load a llama-swap model from the SWITCH tab

CHANGELOG.md

@@ -8,6 +8,11 @@ has a matching GitHub release whose body comes from its section below.
 
 ### Added
 
+- The harbor's SWITCH tab loads one llama-swap model: a paired device taps LOAD on a model that is not loaded, and
+  the model shows loading until llama-swap reports it ready. The gateway's new `POST /v1/berth/load
+  {"engine","model"}` does it by asking llama-swap for the model's `/upstream/MODEL/health`, since llama-swap has
+  no load call of its own. The board's models carry `loading`, and its engines their `type`.
+
 - The harbor has a SWITCH tab: each card with its state, holders and cordon, and every engine with its state
   and models. A device paired with `berth pair` can switch the card to an engine or stop the engine that holds
   it, from a phone. A pinned holder shows PINNED and has no stop. On a machine with several cards a picker

docs/gateway.md

@@ -266,18 +266,25 @@ name the engine:
 ```sh
 curl -s http://127.0.0.1:9799/v1/berth/switch -H "Authorization: Bearer $(berth key)" -d '{"engine":"strata"}'
 curl -s http://127.0.0.1:9799/v1/berth/stop   -H "Authorization: Bearer $(berth key)" -d '{"engine":"strata"}'
+curl -s http://127.0.0.1:9799/v1/berth/load   -H "Authorization: Bearer $(berth key)" -d '{"engine":"llama-swap","model":"qwen3.6-35b-a3b"}'
 ```
 
 - **Switch** answers `202` once the start is under way, and starts the engine under a lease the same way a request
   would: the holder drains at its boundary first. It takes any engine with a `start` command; another is a `404`.
 - **Stop** answers `202` with the leases berthd asked to give the card back, the same drain as `berth stop`. A
   pinned holder (a game, a render, `berth hold`) is a `409`: Berth never stops it from the network. An engine that is
   not running under a lease is a `409` too, and berthd not answering is a `503`.
+- **Load** answers `202` once llama-swap is asked for the model, by a `GET` of `/upstream/MODEL/health` on it:
+  llama-swap has no load call, and any request there starts the model. It takes an engine of `type = "llama-swap"`
+  and a model that engine lists; another engine is a `409` and an unlisted model a `404`. A second load of the same
+  model while one is under way joins it.
 
 `GET /v1/berth/board` is the node in one read, for a board or a script: the card (who holds it, and whether it is
 free, held, draining, in a VM or failing its probe), then every engine in `berth.toml` with its state (up, offline,
 waiting, starting, failed), whether it can be switched to (`launch`) or stopped (`stoppable`), and its models with a
-`loaded` flag from the engine's own reader. `cards` lists every card the node governs, each with its name
+`loaded` flag from the engine's own reader (for llama-swap, its `/running` read with the board, so a load never reads cold
+between llama-swap's answer and berthd's next read) and a `loading` flag while a load is under way or the engine reports the
+model starting. Each engine also carries its `type`. `cards` lists every card the node governs, each with its name
 on the node (`card`, such as `gpu1`), its base URL on this gateway (`base`, `/cards/gpu1/v1`), `cordoned` when
 `berth cordon` closed it, and the holders on that card alone. Each engine names the cards its entry runs on
 (`cards`), so a board for several cards can put each engine under its card.

docs/harbor.md

@@ -168,11 +168,20 @@ A paired device (see [Pair a device](#pair-a-device)) also gets the actions:
   for a request to that engine's model.
 - **STOP** asks the engine that holds the card to give it back at its next request boundary,
   as `berth stop` does.
+- **LOAD**, on a llama-swap model that is not loaded, asks llama-swap for that model. The model
+  shows *loading* until llama-swap reports it ready. llama-swap's model command runs under
+  `berth run`, so the card is leased and its holder drained as for any other start.
 - A pinned holder, such as a game or `berth hold`, shows **PINNED** and has no STOP.
 - **PAIR A PHONE** shows a QR code that pairs another phone.
 
-The board and both actions are the gateway's (`GET /v1/berth/board`, `POST /v1/berth/switch` and
-`POST /v1/berth/stop`, see [the gateway](gateway.md)). The harbor sends them with the node's API key,
+On a machine with several cards, an engine runs on the card its entry names (`gpu = "gpu1"`), or
+on the first card when it names none. SWITCH starts it there and drains only that card's holder;
+engines on the other cards keep running. To run one engine on either card, give it one entry per
+card, such as `comfyui-gpu0` and `comfyui-gpu1`, each with its own `gpu` and port. The picker then
+shows each under its card.
+
+The board and the actions are the gateway's (`GET /v1/berth/board`, `POST /v1/berth/switch`,
+`POST /v1/berth/stop` and `POST /v1/berth/load`, see [the gateway](gateway.md)). The harbor sends them with the node's API key,
 so `berth gateway` must be running. `berth harbor --gateway URL` names the gateway when it is not on
 `[gateway] listen`, and `--key-file` names the key when it is not in the default place. The harbor
 reads the key and never creates it. Without a key, the tab says so and offers no actions.

internal/gateway/board.go

@@ -4,9 +4,12 @@
 package gateway
 
 import (
+	"context"
 	"encoding/json"
+	"maps"
 	"net/http"
 	"os"
+	"strings"
 
 	"github.com/T-Crypt/berth/internal/node"
 	"github.com/T-Crypt/berth/internal/proto"
@@ -50,32 +53,77 @@ type boardEngine struct {
 type boardModel struct {
 	ID     string `json:"id"`
 	Loaded bool   `json:"loaded,omitempty"` // the engine's reader reports it loaded
+	// Loading: a switchboard load is under way, or the engine reports the
+	// model starting.
+	Loading bool `json:"loading,omitempty"`
 }
 
 func (g *Gateway) board(w http.ResponseWriter, r *http.Request) {
 	status := g.nodeStatus()
 	rows := g.guiRows(r.Context(), status)
 	listed := map[string][]string{}
+	live := map[string]map[string]string{}
 	for i := range g.Engines {
 		e := &g.Engines[i]
 		if rows[i].State == "up" {
 			listed[e.Name] = g.listed(r.Context(), e.URL)
+			if e.Type == "llama-swap" {
+				if states := g.swapRunning(r.Context(), e.URL); states != nil {
+					live[e.Name] = states
+				}
+			}
 		}
 		if len(listed[e.Name]) == 0 {
 			listed[e.Name] = e.Serves
 		}
 	}
 	host, _ := os.Hostname()
+	g.mu.Lock()
+	inFlight := maps.Clone(g.loads)
+	g.mu.Unlock()
 	w.Header().Set("Content-Type", "application/json")
 	w.Header().Set("Cache-Control", "no-store")
-	_ = json.NewEncoder(w).Encode(boardOf(host, status, rows, listed))
+	_ = json.NewEncoder(w).Encode(boardOf(host, status, rows, listed, live, inFlight))
+}
+
+// swapRunning is llama-swap's own /running, model -> state, read at once:
+// berthd's reader reads it every few seconds, and a load the board shows
+// must not read cold again in the gap between llama-swap's answer and that
+// read. nil when llama-swap does not answer.
+func (g *Gateway) swapRunning(ctx context.Context, base string) map[string]string {
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, strings.TrimRight(base, "/")+"/running", nil)
+	if err != nil {
+		return nil
+	}
+	resp, err := g.client().Do(req)
+	if err != nil {
+		return nil
+	}
+	defer resp.Body.Close()
+	var r struct {
+		Running []struct {
+			Model string `json:"model"`
+			State string `json:"state"`
+		} `json:"running"`
+	}
+	if resp.StatusCode != http.StatusOK || json.NewDecoder(resp.Body).Decode(&r) != nil {
+		return nil
+	}
+	states := map[string]string{}
+	for _, m := range r.Running {
+		states[m.Model] = m.State
+	}
+	return states
 }
 
 // boardOf assembles the board from berthd's status (nil = not answering),
-// the engines' rows in config order and each engine's model list.
-func boardOf(host string, status *proto.Status, rows []guiRow, listed map[string][]string) boardNode {
+// the engines' rows in config order, each engine's model list, the model
+// states llama-swap reported just now (over berthd's older read) and the
+// switchboard loads under way.
+func boardOf(host string, status *proto.Status, rows []guiRow, listed map[string][]string, live map[string]map[string]string, inFlight map[loadKey]bool) boardNode {
 	n := boardNode{Name: host, Reachable: status != nil, Cards: []boardCard{}, Engines: make([]boardEngine, 0, len(rows))}
 	loaded := map[string]map[string]bool{}
+	starting := map[string]map[string]bool{}
 	if status != nil {
 		for _, gp := range status.GPUs {
 			if !gp.Scheduled {
@@ -105,11 +153,20 @@ func boardOf(host string, status *proto.Status, rows []guiRow, listed map[string
 		}
 		for _, e := range status.Engines {
 			loaded[e.Name] = map[string]bool{}
+			starting[e.Name] = map[string]bool{}
 			for _, m := range e.Models {
 				loaded[e.Name][m.Name] = m.State == "loaded" || m.State == "ready"
+				starting[e.Name][m.Name] = m.State == "starting"
 			}
 		}
 	}
+	for name, states := range live {
+		loaded[name], starting[name] = map[string]bool{}, map[string]bool{}
+		for m, s := range states {
+			loaded[name][m] = s == "ready"
+			starting[name][m] = s == "starting"
+		}
+	}
 	for _, row := range rows {
 		be := boardEngine{guiRow: row, Models: []boardModel{}}
 		if status != nil {
@@ -120,7 +177,9 @@ func boardOf(host string, status *proto.Status, rows []guiRow, listed map[string
 			}
 		}
 		for _, id := range listed[row.Name] {
-			be.Models = append(be.Models, boardModel{ID: id, Loaded: loaded[row.Name][id]})
+			l := loaded[row.Name][id]
+			be.Models = append(be.Models, boardModel{ID: id, Loaded: l,
+				Loading: !l && (starting[row.Name][id] || inFlight[loadKey{row.Name, id}])})
 		}
 		n.Engines = append(n.Engines, be)
 	}

internal/gateway/board_test.go

@@ -99,21 +99,32 @@ func TestBoardOf(t *testing.T) {
 		card      string
 		stoppable []bool
 		loaded    []boardModel
+		inFlight  map[loadKey]bool
+		live      map[string]map[string]string
 	}{
-		{"unreachable", nil, "", []bool{false, false}, []boardModel{{ID: "qwen"}, {ID: "gemma"}}},
+		{"unreachable", nil, "", []bool{false, false}, []boardModel{{ID: "qwen"}, {ID: "gemma"}}, nil, nil},
 		{"held, one model ready", &proto.Status{GPUs: gpu,
 			Leases:  []proto.Lease{{Engine: "llama-swap", Group: "qwen", State: "active"}},
 			Engines: []proto.Engine{{Name: "llama-swap", Models: []proto.Model{{Name: "qwen", State: "ready"}, {Name: "gemma", State: "starting"}}}}},
-			"held", []bool{true, false}, []boardModel{{ID: "qwen", Loaded: true}, {ID: "gemma"}}},
+			"held", []bool{true, false}, []boardModel{{ID: "qwen", Loaded: true}, {ID: "gemma", Loading: true}}, nil, nil},
+		{"a switchboard load under way", &proto.Status{GPUs: gpu},
+			"free", []bool{false, false}, []boardModel{{ID: "qwen"}, {ID: "gemma", Loading: true}}, map[loadKey]bool{{"llama-swap", "gemma"}: true}, nil},
+		{"a load under way for a model already ready", &proto.Status{GPUs: gpu,
+			Engines: []proto.Engine{{Name: "llama-swap", Models: []proto.Model{{Name: "qwen", State: "ready"}}}}},
+			"free", []bool{false, false}, []boardModel{{ID: "qwen", Loaded: true}, {ID: "gemma"}}, map[loadKey]bool{{"llama-swap", "qwen"}: true}, nil},
+		{"llama-swap's own read is newer than berthd's", &proto.Status{GPUs: gpu,
+			Engines: []proto.Engine{{Name: "llama-swap", Models: []proto.Model{{Name: "gemma", State: "starting"}}}}},
+			"free", []bool{false, false}, []boardModel{{ID: "qwen", Loading: true}, {ID: "gemma", Loaded: true}}, nil,
+			map[string]map[string]string{"llama-swap": {"gemma": "ready", "qwen": "starting"}}},
 		{"pinned game", &proto.Status{GPUs: gpu, Leases: []proto.Lease{{Engine: "game", State: "active", Pinned: true}}},
-			"held", []bool{false, false}, []boardModel{{ID: "qwen"}, {ID: "gemma"}}},
+			"held", []bool{false, false}, []boardModel{{ID: "qwen"}, {ID: "gemma"}}, nil, nil},
 		{"draining", &proto.Status{GPUs: gpu, Leases: []proto.Lease{{Engine: "llama-swap", State: "draining"}}},
-			"draining", []bool{false, false}, []boardModel{{ID: "qwen"}, {ID: "gemma"}}},
+			"draining", []bool{false, false}, []boardModel{{ID: "qwen"}, {ID: "gemma"}}, nil, nil},
 		{"in a VM", &proto.Status{GPUs: gpu, Passthrough: &proto.Passthrough{State: proto.CardPassthrough}},
-			"passthrough", []bool{false, false}, []boardModel{{ID: "qwen"}, {ID: "gemma"}}},
+			"passthrough", []bool{false, false}, []boardModel{{ID: "qwen"}, {ID: "gemma"}}, nil, nil},
 	} {
 		t.Run(c.name, func(t *testing.T) {
-			n := boardOf("box", c.node, rows, listed)
+			n := boardOf("box", c.node, rows, listed, c.live, c.inFlight)
 			if n.Reachable != (c.node != nil) {
 				t.Fatalf("reachable = %v", n.Reachable)
 			}

internal/gateway/cards_test.go

@@ -84,7 +84,7 @@ func TestBackgroundEngineStartsOnItsCardWhileAnotherIsHeld(t *testing.T) {
 	})
 	holders := map[string][]string{}
 	st := n.Status()
-	for _, c := range boardOf("box", &st, nil, nil).Cards {
+	for _, c := range boardOf("box", &st, nil, nil, nil, nil).Cards {
 		for _, h := range c.Holders {
 			holders[c.Card] = append(holders[c.Card], h.Name)
 		}

internal/gateway/gateway.go

@@ -91,6 +91,7 @@ type Gateway struct {
 	failed     map[string]*failure // engines whose last start died before answering
 	lists      map[string]cached   // engine URL -> its /v1/models, briefly cached
 	wakes      map[string]*wake    // the wake page's last start, by engine name
+	loads      map[loadKey]bool    // the models a load from the switchboard is under way for
 
 	adoptOnce sync.Once
 	stateMu   sync.Mutex
@@ -220,6 +221,7 @@ func (g *Gateway) Handler() http.Handler {
 	mux.HandleFunc("GET /v1/berth/board", g.board)
 	mux.HandleFunc("POST /v1/berth/switch", g.apiSwitch)
 	mux.HandleFunc("POST /v1/berth/stop", g.apiStop)
+	mux.HandleFunc("POST /v1/berth/load", g.apiLoad)
 	for _, p := range []string{"/v1/chat/completions", "/v1/completions", "/v1/embeddings"} {
 		mux.HandleFunc("POST "+p, g.forward)
 		mux.HandleFunc("POST /cards/{card}"+p, g.onCard(g.forward))

internal/gateway/gui.go

@@ -138,6 +138,7 @@ func (g *Gateway) probe(ctx context.Context, e *config.Engine) bool {
 // only) and what it is doing on the card right now.
 type guiRow struct {
 	Name    string    `json:"name"`
+	Type    string    `json:"type"`          // the engine's type in berth.toml; llama-swap rows can load a model
 	GUI     string    `json:"gui,omitempty"` // the engine's web UI; empty = No UI
 	API     string    `json:"api,omitempty"` // the engine's own address, shown when there is no web UI
 	Launch  bool      `json:"launch"`        // true when the engine has a start command
@@ -154,7 +155,7 @@ type guiRow struct {
 // guiRowFor is the static part of an engine's row: what it is. The state
 // fields are filled in once, per request.
 func guiRowFor(e *config.Engine) guiRow {
-	row := guiRow{Name: e.Name, GUI: e.GUI, Launch: len(e.Start) > 0}
+	row := guiRow{Name: e.Name, Type: e.Type, GUI: e.GUI, Launch: len(e.Start) > 0}
 	if e.GUI == "" {
 		row.API = e.URL
 	}

internal/gateway/load.go

@@ -0,0 +1,109 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package gateway
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+	"net/url"
+	"slices"
+	"strings"
+)
+
+// The switchboard's third action: load one model behind llama-swap. Berth
+// does not swap models itself; llama-swap does, and has no load call of its
+// own. Any request under /upstream/MODEL/ makes it start that model, so a
+// load is a GET of the model's /health there. llama-swap's model command runs
+// under berth run, so the card is leased the usual way.
+
+type loadKey struct{ engine, model string }
+
+// apiLoad is POST /v1/berth/load: 202 once the load is under way. The board
+// shows the model loading until llama-swap reports it ready.
+func (g *Gateway) apiLoad(w http.ResponseWriter, r *http.Request) {
+	var req struct {
+		Engine string `json:"engine"`
+		Model  string `json:"model"`
+	}
+	body, err := io.ReadAll(io.LimitReader(r.Body, 1<<16))
+	if err != nil || json.Unmarshal(body, &req) != nil || req.Engine == "" || req.Model == "" {
+		apiError(w, http.StatusBadRequest, `the body must be JSON like {"engine":"NAME","model":"MODEL"}`)
+		return
+	}
+	e := g.engineByName(req.Engine)
+	switch {
+	case e == nil:
+		apiError(w, http.StatusNotFound, fmt.Sprintf("no engine %q in berth.toml", req.Engine))
+		return
+	case e.Type != "llama-swap":
+		apiError(w, http.StatusConflict, req.Engine+" is not llama-swap; only llama-swap loads one model on request")
+		return
+	case !slices.Contains(g.listed(r.Context(), e.URL), req.Model):
+		apiError(w, http.StatusNotFound, fmt.Sprintf("%s does not list a model %q", req.Engine, req.Model))
+		return
+	}
+	g.load(e.Name, e.URL, req.Model)
+	w.Header().Set("Content-Type", "application/json")
+	w.WriteHeader(http.StatusAccepted)
+	_ = json.NewEncoder(w).Encode(struct {
+		Engine string `json:"engine"`
+		Model  string `json:"model"`
+	}{e.Name, req.Model})
+}
+
+// load asks llama-swap for the model in the background; a second ask while
+// one is under way joins it.
+func (g *Gateway) load(engine, base, model string) {
+	k := loadKey{engine, model}
+	g.mu.Lock()
+	if g.loads == nil {
+		g.loads = map[loadKey]bool{}
+	}
+	if g.loads[k] {
+		g.mu.Unlock()
+		return
+	}
+	g.loads[k] = true
+	g.mu.Unlock()
+	g.log().Info("loading a model", "engine", engine, "model", model)
+	go func() {
+		defer func() {
+			g.mu.Lock()
+			delete(g.loads, k)
+			g.mu.Unlock()
+		}()
+		// The load outlives the request that asked for it, bounded by the
+		// gateway's ready.
+		ctx, cancel := context.WithTimeout(context.Background(), g.ready())
+		defer cancel()
+		u := strings.TrimRight(base, "/") + "/upstream/" + escapeSegments(model) + "/health"
+		req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
+		if err == nil {
+			var resp *http.Response
+			if resp, err = http.DefaultClient.Do(req); err == nil {
+				resp.Body.Close()
+				if resp.StatusCode >= 300 {
+					err = fmt.Errorf("llama-swap answered %s", resp.Status)
+				}
+			}
+		}
+		if err != nil {
+			g.log().Warn("load failed", "engine", engine, "model", model, "err", err)
+		}
+	}()
+}
+
+// escapeSegments escapes a model ID for a URL path but keeps its slashes:
+// llama-swap finds the model in /upstream/ by its path segments, and an ID
+// such as org/model must reach it as two of them.
+func escapeSegments(id string) string {
+	parts := strings.Split(id, "/")
+	for i, p := range parts {
+		parts[i] = url.PathEscape(p)
+	}
+	return strings.Join(parts, "/")
+}