feat: berth cordon and uncordon a card

CHANGELOG.md

@@ -8,6 +8,10 @@ has a matching GitHub release whose body comes from its section below.
 
 ### Added
 
+- `berth cordon CARD` stops new grants on a card, holds aside, while the node's other cards serve; `berth uncordon
+  CARD` lifts it. What holds the card keeps running. Waiters say the card is cordoned, `berth status` marks it
+  `CORDONED`, and the gateway treats it as out of reach: no background engine starts there and a ComfyUI job for it
+  queues. A berthd restart clears the cordon.
 - Each card has its own base URL on the gateway, `/cards/CARD/v1`, named by the card's name or PCI address. A
   request through it only reaches engines on that card, and `GET /cards/CARD/v1/models` lists only theirs. One
   model can now be served on several cards, one engine entry per card, and `berth config check` still refuses two

docs/gateway.md

@@ -317,11 +317,13 @@ Otherwise the gateway takes the card that costs least to reach, in this order:
 1. the card where the model's engine is parked in RAM;
 2. a free card: no lease, waiter, stowaway or active claim, and not in a VM;
 3. a card whose holder drains the usual way;
-4. a card a VM or a pinned lease (a game, a hold) keeps, where the request waits.
+4. a card a VM, a pinned lease (a game, a hold) or a cordon keeps, where the request waits.
 
 Two cards of the same rank go to the entry that comes first in `berth.toml`.
 
 - A card the node does not have answers `404` with the node's cards.
+- A cordoned card ([Cordon a card](leasing.md#cordon-a-card)) still takes requests through its URL, and they wait
+  in line.
 - A model no engine on that card serves answers `404` naming the card.
 - An engine berthd finds that has no entry in `berth.toml` belongs to no card here, so only the plain `/v1` URL
   reaches it.

docs/interfaces.md

@@ -1,7 +1,7 @@
 # Interfaces
 
 - **CLI:** one command tree with one output contract. `berth status`, `berth events -f`,
-  `berth run`, `berth hold`, `berth stop`, `berth notify`, `berth claims`, `berth pull`, `berth verify`, `berth models`, `berth gateway`,
+  `berth run`, `berth hold`, `berth stop`, `berth cordon|uncordon`, `berth notify`, `berth claims`, `berth pull`, `berth verify`, `berth models`, `berth gateway`,
   `berth wire llama-swap`, `berth engine check`, `berth config check`, `berth config show`,
   `berth export`, `berth key`, `berth skill install|show`, `berth update`, `berth version`,
   and for the node itself `berth daemon`, `berth harbor`, `berth service install|uninstall|status`,

docs/leasing.md

@@ -126,6 +126,20 @@
 - A hold cannot free memory held by something outside Berth: that memory is still reported,
   as contested.
 
+## Cordon a card
+
+On a node with several cards, `berth cordon gpu1` stops new grants on that card while the others keep serving, for
+maintenance or before a VM takes it. Name the card by its name or its PCI address.
+
+- What holds the card keeps running. A cordon drains nothing: `berth stop ENGINE` does that.
+- Waiters stay in line, and their reason says the card is cordoned. A hold (`berth hold`, a game claim) is still
+  granted.
+- The gateway treats the card as out of reach. A plain `/v1` request goes to another card that serves its model,
+  a background engine for the card does not start, and a ComfyUI job for it queues.
+- `berth status` marks the card `CORDONED`, and `berth events` records `card.cordoned` and `card.uncordoned`.
+- `berth uncordon gpu1` lets new work in, and the card's waiters are scheduled at once. A berthd restart also
+  clears the cordon.
+
 ## Training runs
 
 A training run that loses its cards loses everything since its last checkpoint. Run it under one pinned lease:

docs/roadmap.md

@@ -111,11 +111,11 @@ v0.5 lets a client choose the card on a node with several. Each slice ships on i
 
 1. **One base URL per card**, merged in [#278](https://github.com/T-Crypt/berth/pull/278). `/cards/CARD/v1` reaches only
    the engines on that card, and one model can be served on several cards, one engine entry per card.
-2. **Placement for a plain `/v1` request.** When several cards serve the model, the gateway picks one: the card
-   where it already runs, then one where it is parked, then a free card, then one whose holder drains, and last a
-   card a VM or a game keeps.
-3. **Cordon a card.** A cordoned card takes no new grants until it is uncordoned, so it can be drained for
-   maintenance or a VM while the others serve.
+2. **Placement for a plain `/v1` request**, merged in [#280](https://github.com/T-Crypt/berth/pull/280). When
+   several cards serve the model, the gateway picks one: the card where it already runs, then one where it is
+   parked, then a free card, then one whose holder drains, and last a card a VM, a game or a cordon keeps.
+3. **Cordon a card.** `berth cordon gpu1` stops new grants on the card, holds aside, until `berth uncordon`, so it
+   can be drained for maintenance or a VM while the others serve.
 4. **A card picker on the board.** The harbor and `GET /v1/berth/board` name the card each engine can run on.
 
 ## Also queued

internal/broker/broker.go

@@ -479,6 +479,7 @@ type Broker struct {
 	saveFailed bool
 
 	verdicts map[string]verdict // loser engine key -> the juggle it lost
+	cordoned bool               // berth cordon: grant only holds
 	// tracked is the claim watcher's last scan: a process (and every child)
 	// a track claim names. Its card memory is the claim's, never a stowaway.
 	tracked    map[int]string
@@ -1337,6 +1338,9 @@ func (c cardBudget) avail() uint64   { return sat(c.ceiling(), c.held) }
 // evaluate decides whether w fits now. If not, it returns why and which
 // holders to drain; no victims means draining would not help, so w waits.
 func (b *Broker) evaluate(w *lease, now time.Time) (bool, string, []*lease) {
+	if b.cordoned && w.prio != Hold {
+		return false, "the card is cordoned; berth uncordon lets new work in", nil
+	}
 	// A resumed lease is still in b.leases while it waits: its own
 	// residual is the copy it comes back to, never a holder it waits on.
 	hs := slices.DeleteFunc(b.holders(), func(h *lease) bool { return h == w })
@@ -1817,3 +1821,22 @@ func (b *Broker) abandon(l *lease, why string) error {
 	b.emit(proto.Event{Kind: proto.EvRefused, Lease: l.id, Engine: l.engine, Detail: why})
 	return fmt.Errorf("%w: %s", ErrRefused, why)
 }
+
+// Cordon stops new grants on the card but holds (on), or lets them in again
+// (off). Holders keep the card; a cordon drains nothing. A change goes on
+// the event stream, and an uncordon schedules the queue at once.
+func (b *Broker) Cordon(on bool) {
+	b.mu.Lock()
+	defer b.mu.Unlock()
+	if b.cordoned == on {
+		return
+	}
+	b.cordoned = on
+	kind := proto.EvUncordoned
+	if on {
+		kind = proto.EvCordoned
+	}
+	b.log.Info("cordon", "card", b.cfg.Card, "on", on)
+	b.emit(proto.Event{Kind: kind})
+	b.schedule(b.now())
+}

internal/broker/cordon_test.go

@@ -0,0 +1,39 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package broker
+
+import (
+	"strings"
+	"testing"
+
+	"github.com/T-Crypt/berth/internal/procs"
+)
+
+// A cordoned card grants nothing new but a hold, keeps its holder, and
+// says why in the waiter's reason; uncordoning lets the waiter in.
+func TestCordonStopsNewGrantsUntilUncordoned(t *testing.T) {
+	b, _ := startTurnBroker(t, procs.Map{})
+	holder := must(t, acquire(b, Request{Engine: "llm", Priority: Batch}))
+	b.Cordon(true)
+	if st := b.Status(); len(st.GPUs) == 0 || !st.GPUs[0].Cordoned {
+		t.Fatalf("status does not show the cordon: %+v", st.GPUs)
+	}
+	holder.Release()
+
+	w := acquireW(b, Request{Engine: "image", Priority: Foreground})
+	pending(t, w.ch)
+	if r := firstReason(t, w); !strings.Contains(r, "cordoned") {
+		t.Errorf("reason = %q, want the cordon named", r)
+	}
+	hold := must(t, acquire(b, Request{Engine: "game", Priority: Hold}))
+	hold.Release()
+	pending(t, w.ch)
+
+	b.Cordon(false)
+	b.Tick(t.Context())
+	must(t, w.ch)
+	if st := b.Status(); st.GPUs[0].Cordoned {
+		t.Error("status still shows the cordon")
+	}
+}

internal/broker/status.go

@@ -66,7 +66,7 @@ func (b *Broker) Status() proto.Status {
 			Util: d.Util, Temp: d.Temp, Power: d.Power, PowerLimit: d.PowerLimit, Fan: d.Fan,
 			ClockSM: d.ClockSM, ClockMem: d.ClockMem, PCIeGen: d.PCIeGen, PCIeWidth: d.PCIeWidth}
 		if d.Index == b.snap.Index {
-			g.Scheduled, g.Baseline = true, b.snap.Baseline()
+			g.Scheduled, g.Baseline, g.Cordoned = true, b.snap.Baseline(), b.cordoned
 			g.UnobservedPIDs = append([]int(nil), b.snap.Unobserved...)
 			g.GTTBaseline = b.snap.GTTBaseline()
 		}

internal/cli/commands.go

@@ -520,6 +520,28 @@ unpairs every device.`,
 		Formats: []Format{FormatHuman, FormatJSON, FormatYAML},
 		Run:     runPair,
 	},
+	{
+		Name:    "cordon",
+		Summary: "stop new grants on a card; what holds it keeps running",
+		Args:    "CARD [--socket P] [--format F]",
+		Help: `Stop new grants on the named card, by its name (gpu1) or PCI address, for
+maintenance or before a VM takes it, while the node's other cards keep
+serving. Waiters stay in line with a reason that names the cordon, and
+the gateway sends a plain /v1 request to another card that serves its
+model. A hold (berth hold, a game claim) is still granted. What holds the
+card keeps running: berth stop ENGINE drains it. The cordon lasts until
+berth uncordon or until berthd restarts.`,
+		Formats: []Format{FormatHuman, FormatJSON, FormatYAML},
+		Run:     runCordon,
+	},
+	{
+		Name:    "uncordon",
+		Summary: "let new work get a cordoned card again",
+		Args:    "CARD [--socket P] [--format F]",
+		Help:    `Lift berth cordon from the named card. Its waiters are scheduled at once.`,
+		Formats: []Format{FormatHuman, FormatJSON, FormatYAML},
+		Run:     runUncordon,
+	},
 	{
 		Name:    "stop",
 		Summary: "stop an engine on purpose: its leases drain through their own wrappers",

internal/cli/cordon.go

@@ -0,0 +1,68 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package cli
+
+// berth cordon and berth uncordon: a person stops new grants on a card,
+// for maintenance or before a VM takes it, while the other cards serve.
+
+import (
+	"strings"
+
+	"github.com/T-Crypt/berth/internal/proto"
+	"github.com/T-Crypt/berth/internal/server"
+)
+
+func runCordon(c *Context) int   { return cordonCmd(c, proto.OpCordon) }
+func runUncordon(c *Context) int { return cordonCmd(c, proto.OpUncordon) }
+
+func cordonCmd(c *Context, op string) int {
+	fs := c.FlagSet()
+	sock := fs.String("socket", defaultSocket(), "berthd socket ($BERTH_SOCKET)")
+	if code, done := c.ParseInterspersed(fs); done {
+		return code
+	}
+	if len(c.Args) != 1 {
+		return c.fail(Failf("berth %s takes one card", op).WithHint("usage: berth %s CARD (berth status names the cards)", op))
+	}
+	conn, raw, err := server.Dial(*sock)
+	if err != nil {
+		return c.fail(Systemf("berthd unreachable on %s: %v", *sock, err))
+	}
+	defer raw.Close()
+	if err := conn.Send(proto.Msg{Op: op, Card: c.Args[0]}); err != nil {
+		return c.fail(Systemf("berthd on %s: %v", *sock, err))
+	}
+	m, err := conn.Recv()
+	if err != nil {
+		return c.fail(Systemf("no %s reply from berthd: %v", op, err))
+	}
+	if m.Op == proto.OpRefused {
+		return c.fail(Failf("%s", m.Reason))
+	}
+	return c.Write(cordonView{op: op, cards: m.Cards}, nil)
+}
+
+type cordonView struct {
+	op    string
+	cards []string
+}
+
+func (v cordonView) Data() any {
+	return struct {
+		Cordoned bool     `json:"cordoned"`
+		Cards    []string `json:"cards"`
+	}{v.op == proto.OpCordon, v.cards}
+}
+
+func (v cordonView) Human(c *Context, ok bool) error {
+	names := strings.Join(v.cards, ", ")
+	if v.op == proto.OpUncordon {
+		c.Printf("%s uncordoned: new work gets the card again\n", names)
+		return nil
+	}
+	c.Printf("%s cordoned: no new grants but holds\n", names)
+	c.Bullet("what holds it keeps running; berth stop ENGINE drains it")
+	c.Bullet("the cordon lasts until berth uncordon or a berthd restart")
+	return nil
+}

internal/cli/cordon_test.go

@@ -0,0 +1,50 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package cli
+
+import (
+	"encoding/json"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/T-Crypt/berth/internal/proto"
+)
+
+// berth cordon and uncordon against a live daemon: the report names the
+// card, status shows the cordon, and an unknown card fails with exit 1.
+func TestCordonAgainstADaemon(t *testing.T) {
+	sock, _, _ := startStopDaemon(t, time.Second)
+	cordoned := func() bool {
+		code, out, _ := runCLI(t, "status", "--socket", sock, "--format", "json")
+		if code != 0 {
+			t.Fatalf("status exit %d", code)
+		}
+		var st proto.Status
+		if err := json.Unmarshal([]byte(out), &st); err != nil {
+			t.Fatal(err)
+		}
+		return len(st.GPUs) > 0 && st.GPUs[0].Cordoned
+	}
+
+	code, out, errOut := runCLI(t, "cordon", "gpu0", "--socket", sock)
+	if code != 0 || !strings.Contains(out, "gpu0") {
+		t.Fatalf("cordon: exit %d, out %q, err %q", code, out, errOut)
+	}
+	if !cordoned() {
+		t.Error("status does not show gpu0 cordoned")
+	}
+	if _, out, _ := runCLI(t, "status", "--socket", sock); !strings.Contains(out, "CORDONED") {
+		t.Errorf("berth status does not say CORDONED:\n%s", out)
+	}
+	if code, out, errOut = runCLI(t, "uncordon", "gpu0", "--socket", sock); code != 0 || !strings.Contains(out, "gpu0") {
+		t.Fatalf("uncordon: exit %d, out %q, err %q", code, out, errOut)
+	}
+	if cordoned() {
+		t.Error("status still shows gpu0 cordoned")
+	}
+	if code, _, errOut = runCLI(t, "cordon", "gpu7", "--socket", sock); code != ExitFail || !strings.Contains(errOut, "no card gpu7") {
+		t.Errorf("unknown card: exit %d, err %q, want exit %d naming the card", code, errOut, ExitFail)
+	}
+}

internal/cli/status.go

@@ -51,6 +51,9 @@ func (v statusView) Human(c *Context, ok bool) error {
 		role := "inventory only"
 		if g.Scheduled {
 			role = "leased by berth, baseline " + server.MiB(g.Baseline)
+			if g.Cordoned {
+				role = c.Style.Err("CORDONED") + ", holds only; " + role
+			}
 		}
 		c.Printf("gpu%d    %s: %s used of %s, %s free (%s, read %s ago)\n", g.Index, g.Name,
 			server.MiB(g.Used), server.MiB(g.Total), server.MiB(g.Free), role, age(g.Age))

internal/gateway/background.go

@@ -31,7 +31,7 @@ const DefaultIdleAfter = 60 * time.Second
 // beside it is the collision Berth exists to prevent.
 func cardFree(st *proto.Status) bool {
 	if st == nil || st.Passthrough != nil || st.ProbeErr != "" ||
-		len(st.Leases) > 0 || len(st.Waiting) > 0 || len(st.Squatters) > 0 {
+		len(st.Leases) > 0 || len(st.Waiting) > 0 || len(st.Squatters) > 0 || cordoned(st) {
 		return false
 	}
 	// Status lists every claim rule; only one matching right now is in use.